Claude Code. Any Model. The most powerful AI coding agent now speaks every language.
fix(recovery): make the budget, the chain guard and the log real in the arm nobody tested
Jack Rudenko09eb4deb75d2
diff --git a/ai-docs/architecture/network-recovery.md b/ai-docs/architecture/network-recovery.mdindex d4d79e0b..16221e58 100644--- a/ai-docs/architecture/network-recovery.md+++ b/ai-docs/architecture/network-recovery.md@@ -47,7 +47,32 @@ The other two ceilings the derivation clears:| ceiling | value | where ||---|---|---|| Bun's per-request idle timeout | 255 s | `proxy-server.ts`'s `idleTimeout` |-| the unclamped attempt 1 | up to 45 s | reserved out of the budget, never clamped (NFR-1) |+| attempt 1 | the remaining budget | `deadlineClamp`, armed before the primary fetch |++### The deadline is only a deadline if attempt 1 is inside it — and twice it was not++Both halves of this were found by a black-box contract test, and neither is visible to a suite built+on refused loopback ports, because a 1 ms attempt cannot cross a deadline in the middle of itself.++**The zero point was set by the first READER, not by the request.** `inboundStartedAtPerf` memoises+on `c.req.raw`, and on the fetch path the first caller was `tier1DeadlineAt(c)` *inside the catch* —+after attempt 1 had already failed. Measured against the unrouted `192.0.2.1` with+`API_TIMEOUT_MS=60000` (a 30 s budget): attempt 1 spent macOS's 75 s connect timeout, the budget then+started counting from t+75 s, and **the client was answered at 108 s**. The budget had not been+exceeded — it had never been applied. `handle()` now stamps the zero point on entry.++**Attempt 1 had no ceiling at all.** `DEADLINE_MARGIN_MS`'s 30 s was supposed to reserve room for its+tail; the OS's connect timeout is 75 s, so the reserve is wrong by 45 s for a slow connect and wrong+by more than the whole budget whenever `API_TIMEOUT_MS ≤ 105 s`. It is now bounded by `deadlineAt` —+and by `deadlineAt` specifically, **not** by `PER_ATTEMPT_CONNECT_CAP_MS`:++- 45 s is a plausible time-to-first-byte for a thinking model behind a long prompt, and this ceiling+ sits on a possibly-HEALTHY call. Clamping it at 45 s would re-open RISK-6's first leak — a latency+ event reclassified as "the host is unreachable". At the default settings the ceiling is 270 s,+ which no provider's first byte approaches, and it shortens only as the user's own knob does;+- firing AT the deadline is what makes it safe: there is by construction no budget left when the+ abort is classified, so `shouldSkipTier1`'s `no-budget` gate fires and the request answers. **This+ ceiling can never cause a re-issue.** An earlier firing time would not have that property.Bun's 255 s stops binding because `c.env.timeout(c.req.raw, 0)` **works** — measured, a requestgiven `0` survived **6×** its `idleTimeout` where the un-disarmed control died at 1.6×. That call is@@ -146,6 +171,47 @@ a duration quoted verbatim in a user-facing message to this one.On a tier-2 rejoin the ladder does **not** reset. The target has been down for the whole of it, andrestarting at 5 s would hammer a dead host harder the longer the outage lasted.+### …but a carried rung may be TRUNCATED once, or the re-ask retries nothing++The ladder index belongs to the **episode**; the deadline belongs to the **socket**, and a re-ask+gets a fresh — possibly much shorter — one. Once the carried rung outgrew that deadline, the+no-budget gate fired on the first iteration and the request answered **in 8 ms having re-issued+nothing**. Measured at the upstream socket with `API_TIMEOUT_MS=60000`:++| `API_TIMEOUT_MS` | deadline | request 1 | request 2 (before) | request 2 (now) |+|---|---|---|---|---|+| 120 000 | 90 s | 0/5/15/45 s | 0/60 s | unchanged |+| 60 000 | 30 s | 0/5/15 s | **one attempt, 8 ms** | 0/27 s |+| 40 000 | 10 s | 0/5 s | 0/10 s | unchanged |++And it was permanent: with no banner the answer is a 400, which Claude Code does not re-ask, so+recovery was over for that endpoint for the rest of the outage. Invisible at the 270 s default, where+the terminal 60 s rung always fits — it bites every `API_TIMEOUT_MS ≤ 90 s`, a value+`docs/advanced/environment.md` invites.++So a waiter that has **not yet re-issued anything** shortens the wait to the budget it actually has+(floor `MIN_ATTEMPT_SLOT_MS`), makes its one attempt, and only then hands off. The condition is+`requestRetries === 0`, which is why request 1's shape above is untouched: by the time ITS rung stops+fitting it has re-issued two or three times already. It is not a hammer — at most one truncated rung+per request, and the gap between two client re-asks is the client's own backoff (up to 38.4 s), not+ours.++### Four places watch the client's abort, and a mutation that survived does not make them redundant++Disabling all three abort paths in `transient-retry.ts` — the loop-top check, `untilAborted`, and+`mergeSignalIntoInit`'s threading — left client-disconnect handling working. The surviving path is+`coordinator.ts`'s `waitForNextAttempt`, which registers its own listener on the same signal; against+a refused loopback port the ladder spends ~100% of its wall clock parked in that wait, so the wait+answered. Each covers a window no other one does, and a slow connect inverts the arithmetic (45 s of+attempt to 5 s of wait):++| # | site | the window only it covers |+|---|---|---|+| 1 | the loop's `if (ctx.signal.aborted)` | between an attempt settling and the next wait starting; and the first iteration, before any wait exists |+| 2 | `coordinator.waitForNextAttempt` | during the wait — and it is the only one that can DETACH the waiter from the shared episode timer |+| 3 | `untilAborted` | during an attempt whose operation cannot take a signal at all: `refreshAuth()`/`getHeaders()` accept no arguments, so on the auth path this is the only unwind there is |+| 4 | `mergeSignalIntoInit` | inside `fetch`, so the SOCKET closes. The other three unwind the waiter and leave the connect running |+### `PER_ATTEMPT_CONNECT_CAP_MS` is 45 s because 75 s collides with the OSmacOS gives up on its own TCP connect at **75.004 s** (measured against the unrouted `192.0.2.1`; a@@ -229,6 +295,30 @@ So the marker is a **header**, `x-claudish-recovery: 1`, checked in two independ`handle()` returns a marked response VERBATIM before reading the body (defeating the combining), and`isRetryableError` returns `false` on it **above** the quota match (defeating the wording).+### One marker was not enough: it was minted on the 503 arm ONLY++Which made chain-safety a property of **which arm answered**. Exhaustion answers 400 whenever no+banner is attached — `-p`, `--no-recovery`, CI, any machine without magmux, i.e. *every headless+run* — and that 400 carried no marker at all, so the wording check decided again. A black-box test+reproduced it from outside: candidate 1 unreachable at a host named `quota-exceeded-…`, and+**candidate 2's own socket served the client**. `insufficient-credits` and `rate-limit` in the same+position did not reproduce it, so the variable really was the word — and the error text quotes the+endpoint host and full URL, so the word is user-influenced.++The 400 arm now carries its own marker, `x-claudish-connection-error: 1`, and `isRetryableError`+asks one predicate — `isClaudishConnectionVerdict` — that is true of either. The invariant is+stated in terms of the request rather than of the response:++> **A connection-failure response from this feature never advances the fallback chain — regardless+> of its status, its message, or whether a banner was attached.**++It is deliberately a SECOND header rather than the same one on both arms, because the two facts+differ and one is load-bearing elsewhere: `x-claudish-recovery` means "the retry was handed back",+and `FallbackHandler` returns anything wearing it **verbatim** so `formatCombinedError` cannot demote+its 503. A 400 has nothing handed back and must stay foldable into the combined chain error, which is+where the user reads what every candidate did. Round 1's quota mutation tested the 503 path and+passed; nobody tested the 400 path.+> **Moving the marker check below the quota match — present, but LATE — is a live billing bug.**> It is mutation-covered as one, and the mutation that kills it is the quota-wording variant> specifically. A test whose 503 body says nothing about quota cannot see the defect at all.@@ -399,6 +489,24 @@ would be a phantom distorting request counts, success rate and latency percentilstream that is supposed to answer the question. Episode lifecycle is a `[Recovery]` structural logline instead.+### `[Recovery]` is OUTPUT, not a debug aid — and for a while it was neither++The lifecycle lines were on `log()`, which writes to two FILES: the `--debug` log and the always-on+structural log under `~/.claudish/logs/`. That satisfies "it is written down" and fails the thing it+was written down for. A hold runs for up to the derived deadline (~4.5 minutes at the default) and in+a run with no pane — which is *every headless run* — the user saw nothing at all while it did. A+silent multi-minute hold with the reason legible nowhere is the state §3 calls "strictly worse than+the bug this feature exists to remove"; a file the user does not know to open is not a surface.++They now go through `logRecovery()` → `logStderr`, which is why the noise objection does not bite:+in an interactive session `logStderr` routes to `DiagOutput` (a file) rather than to the client's+TUI, and in quiet mode it is suppressed — so the lines land on a terminal exactly where a terminal is+the only surface there is. Both log files still receive them.++The line is drawn at the **episode's lifecycle** — opened, rejoined, each attempt, each wait, handed+off, closed, recovered, exhausted: the six-to-ten lines from which a reader can reconstruct the+ladder. Pane connect/disconnect, socket paths, a skipped ladder and `client_gone` stay on `log()`.+### The trap: `eventToLogRecord` is a hand-written allowlistA field added to the `StatsEvent` interface and populated by `stats.ts` but **not pushed in@@ -503,10 +611,23 @@ MCP server process dying outright.## 8. What a healthy request pays-Nothing, and the guarantee is **placement**, not a flag: the entire retry apparatus is constructed-inside the `catch`, after classification has already returned non-null. The primary fetch expression-— including its `enqueueRequest` ternary — is byte-identical to what it was before recovery existed.-That is verifiable by reading the diff rather than by trusting a predicate.+**Three statements, and the guarantee around them is still placement rather than a flag**: the entire+retry apparatus — episode, ladder, waiters, pane — is constructed inside the `catch`, after+classification has already returned non-null. The `enqueueRequest` ternary that issues attempt 1 is+byte-identical to what it was before recovery existed. What a healthy request now pays, outside that+catch, is exactly:++| | cost | why it cannot be moved into the catch |+|---|---|---|+| `inboundStartedAtPerf(c)` at `handle()` entry | one clock read, one `WeakMap.set` | the deadline's zero point must be the request's start; asking for it later is what produced the 108 s answer in §1 |+| `tier1DeadlineAt(c)` before the fetch | arithmetic over one env read | the ceiling has to exist before the call it bounds |+| `deadlineClamp(deadlineAt)` | one `AbortController`, one timer, one `AbortSignal.any` | same |++The timer is disarmed in a `finally` the instant the call settles — before a byte of the body is+read — so it can only ever bound connect-and-headers, never cut a response already arriving. This+replaces an earlier, stronger claim ("nothing, and the primary fetch expression is byte-identical"):+that claim was true and the feature was not, because a budget nothing enforces on attempt 1 is not a+budget.The re-issue goes through that **same ternary**. Six transports implement `enqueueRequest`, and whatthey implement is not decoration: a bounded 429 loop with `Retry-After`, a served-set model-fallbackdiff --git a/docs/advanced/environment.md b/docs/advanced/environment.mdindex d9c6b29d..63ea502a 100644--- a/docs/advanced/environment.md+++ b/docs/advanced/environment.md@@ -153,7 +153,11 @@ The deadline is **derived from `API_TIMEOUT_MS`**, not fixed:min(API_TIMEOUT_MS, 300s) − 30s # ~270s at the 360s default```-Claude Code aborts its own request at `API_TIMEOUT_MS` (360 000 ms by default), so a hold longer than that would be cut off by the client mid-wait. If you have shortened `API_TIMEOUT_MS`, claudish shortens the hold to match and logs a `[Recovery]` line saying so.+Claude Code aborts its own request at `API_TIMEOUT_MS` (360 000 ms by default), so a hold longer than that would be cut off by the client mid-wait. If you have shortened `API_TIMEOUT_MS`, claudish shortens the hold to match and says so with a `[Recovery]` line.++The deadline is the ceiling on the **whole** attempt, including the first one: a connect that never completes is abandoned when the budget runs out rather than waiting out the operating system's own connect timeout (75 s on macOS), which on a shortened `API_TIMEOUT_MS` would answer you after the client had already given up.++**Where the `[Recovery]` lines go.** A recovery episode reports itself on **stderr** — opened, each attempt, each wait and how it ended — with no `--debug` anywhere, because a request can be held for minutes and a hold nobody can see is worse than a fast failure. In an interactive session they are routed to the session log instead of the terminal, so they cannot corrupt Claude Code's display; `~/.claudish/logs/` keeps them either way.### `CLAUDISH_RECOVERY_UI`diff --git a/packages/cli/src/handlers/composed-handler.ts b/packages/cli/src/handlers/composed-handler.tsindex 53e7b623..c846570d 100644--- a/packages/cli/src/handlers/composed-handler.ts+++ b/packages/cli/src/handlers/composed-handler.ts@@ -33,7 +33,14 @@ import { resolveQuotaAdapter } from "../auth/quota/registry.js";import { PLAN_POLL_INTERVAL_MS } from "../auth/quota/types.js";import type { BehaviorEngine, BehaviorSession } from "../behavior/index.js";import { getBehaviorEngine } from "../behavior/index.js";-import { getLogLevel, log, logStderr, logStructured, truncateContent } from "../logger.js";+import {+ getLogLevel,+ log,+ logRecovery,+ logStderr,+ logStructured,+ truncateContent,+} from "../logger.js";import { GeminiThoughtSignatureMiddleware, MiddlewareManager } from "../middleware/index.js";import { deepMergeParams } from "../model-params.js";import { describeSiblingKeys, getProviderByName } from "../providers/provider-definitions.js";@@ -75,7 +82,7 @@ import { sniffDevinStreamHead } from "./shared/devin-stream-head-sniffer.js";import { hasActionableLink, hasModelUnsupportedWording } from "./shared/model-unsupported.js";import { filterIdentity } from "./shared/openai-compat.js";import { hasPlanLimitWording, isQuotaExhaustionError } from "./shared/quota-exhaustion.js";-import { recoveryHoldHeaders } from "./shared/recovery-marker.js";+import { connectionFaultHeaders, recoveryHoldHeaders } from "./shared/recovery-marker.js";import {CONTEXT_OVERFLOW_PHRASE,isContextOverflowError,@@ -92,6 +99,8 @@ import { TokenTracker, type UsageCacheDetail } from "./shared/token-tracker.js";import {type ConnectionErrorInfo,MIN_ATTEMPT_SLOT_MS,+ deadlineClamp,+ inboundStartedAtPerf,mergeSignalIntoInit,refreshDeadlineAt,tier1DeadlineAt,@@ -512,7 +521,25 @@ export class ComposedHandler implements ModelHandler {} catch {// Stats must never crash claudish}- return c.json(wrapAnthropicError(400, msg, "connection_error"), 400 as any);+ // The chain-safety marker, on THIS arm as well as on the 503.+ //+ // It is the third argument rather than a `c.header()` call, because `c` is+ // SHARED across every candidate in a fallback chain and a sticky header set+ // here would ride out on `formatCombinedError`'s response — the same+ // invariant `respondRecoveryHold` obeys by building a standalone Response.+ // Hono's `newResponse` applies this bag to the returned Response only; it+ // never touches the context's own header store.+ //+ // Without it, chain-safety existed exactly where a banner did. A headless+ // exhaustion answered an unmarked 400, `isRetryableError` fell through to+ // its status-agnostic quota-wording match, and a host whose NAME contained+ // "quota" advanced the chain onto a metered provider mid-outage. See+ // `recovery-marker.ts`.+ return c.json(+ wrapAnthropicError(400, msg, "connection_error"),+ 400 as any,+ connectionFaultHeaders()+ );}/**@@ -783,7 +810,7 @@ export class ComposedHandler implements ModelHandler {ctx.recovery.outcome = result.kind === "exhausted" ? "handoff" : "gave_up";if (result.kind === "exhausted" && leased) {- log(+ logRecovery(`[Recovery] ${this.provider.displayName} exhausted after ${result.attempts} attempts ` +`in ${result.recoveryMs}ms (episode ${result.episodeId}, ui_lease=true) — ` +"handing the retry back to the client (503, tier 2)"@@ -800,7 +827,7 @@ export class ComposedHandler implements ModelHandler {};}- log(+ logRecovery(`[Recovery] ${this.provider.displayName} exhausted after ${result.attempts} attempts ` +`in ${result.recoveryMs}ms (episode ${result.episodeId}, outcome ${result.kind}, ` +`ui_lease=${leased}) — answering connection_error`@@ -851,6 +878,23 @@ export class ComposedHandler implements ModelHandler {async handle(c: Context, payload: any): Promise<Response> {const startTime = performance.now();+ // Stamp the recovery deadline's ZERO POINT here, and not wherever something+ // first asks for it.+ //+ // `inboundStartedAtPerf` memoises on `c.req.raw`, so before this line the+ // zero point was set by the FIRST caller — which on the fetch path is+ // `tier1DeadlineAt(c)` inside the `catch`, i.e. AFTER attempt 1 has already+ // failed. MEASURED against an unrouted address: attempt 1 burned macOS's+ // 75 s connect timeout, the deadline's clock then started from there, and a+ // request with a 30 s budget answered at 108 s — the budget was not+ // exceeded, it was never applied. A ladder cannot bound a request whose+ // clock starts after the overrun it was supposed to bound.+ //+ // One `performance.now()` and one `WeakMap.set` per request; nothing else+ // on the healthy path reads it. A later candidate in a fallback chain finds+ // the value already there and inherits it, which is the documented intent —+ // the deadline belongs to the inbound request, not to a candidate.+ inboundStartedAtPerf(c);// latency_ms = time-to-first-byte (from request send to successful response).// Captured here so it is available to the post-stream stats callback below.let latencyMs = 0;@@ -1261,19 +1305,42 @@ export class ComposedHandler implements ModelHandler {// Merge provider-specific fetch options (e.g., undici dispatcher, abort signal)const requestInit = this.provider.getRequestInit?.() || {};++ // THE REQUEST'S OWN CEILING — the one thing outside the `catch` that this+ // feature adds to a healthy request, and the reason it is here rather than+ // in the ladder: an unclamped attempt 1 could outlive the whole derived+ // deadline (measured: 75 s of macOS connect against a 30 s budget, answered+ // at 108 s), which makes the budget a suggestion. It is absolute, it fires+ // at `deadlineAt` and never earlier, and it is disarmed the instant the+ // call settles — see `deadlineClamp` for why both of those are what keep it+ // from turning a slow answer into a re-issued one.+ const deadlineAt = tier1DeadlineAt(c);+ const clamp = deadlineClamp(deadlineAt);const doFetch = () =>- fetch(endpoint, {- method: "POST",- headers,- body: serialized?.body ?? JSON.stringify(requestPayload),- ...requestInit,- });+ fetch(+ endpoint,+ mergeSignalIntoInit(+ {+ method: "POST",+ headers,+ body: serialized?.body ?? JSON.stringify(requestPayload),+ ...requestInit,+ },+ clamp.signal+ )+ );let response: Response;try {- response = this.provider.enqueueRequest- ? await this.provider.enqueueRequest(doFetch)- : await doFetch();+ try {+ response = this.provider.enqueueRequest+ ? await this.provider.enqueueRequest(doFetch)+ : await doFetch();+ } finally {+ // Before a byte of the body is read, so the ceiling can only ever bound+ // connect-and-headers — never cut a response already arriving.+ clamp.disarm();+ }} catch (error: any) {// A failure to even REACH the provider (DNS can't resolve, connection// refused, host unreachable) is a LOCAL network problem, not an upstream@@ -1333,7 +1400,8 @@ export class ComposedHandler implements ModelHandler {const outcome = await this.recoverConnection<Response>(c, error, conn, endpoint, reissue, {startTime,- deadlineAt: tier1DeadlineAt(c),+ // The SAME value the primary attempt was bounded by, not a fresh read.+ deadlineAt,fallbackMeta,site: "fetch",recovery: recoveryTally,diff --git a/packages/cli/src/handlers/fallback-handler.ts b/packages/cli/src/handlers/fallback-handler.tsindex 69899eee..343a0139 100644--- a/packages/cli/src/handlers/fallback-handler.ts+++ b/packages/cli/src/handlers/fallback-handler.ts@@ -14,7 +14,7 @@ import { logStderr } from "../logger.js";import { ComposedHandler } from "./composed-handler.js";import { extractUpstreamStatus } from "./shared/anthropic-error.js";import { hasQuotaExhaustionWording } from "./shared/quota-exhaustion.js";-import { hasRecoveryMarker, isRecoveryHoldResponse } from "./shared/recovery-marker.js";+import { isClaudishConnectionVerdict, isRecoveryHoldResponse } from "./shared/recovery-marker.js";import type { ModelHandler } from "./types.js";export interface FallbackCandidate {@@ -213,23 +213,25 @@ export function isRetryableError(provider?: string,headers?: Headers): boolean {- // ── CLAUDISH'S OWN RECOVERY 503 IS NEVER A REASON TO ADVANCE ───────────────+ // ── CLAUDISH'S OWN "CANNOT REACH THE HOST" IS NEVER A REASON TO ADVANCE ────//// FIRST, above the quota wording check, and the ORDER is the whole point.// The next statement is `hasQuotaExhaustionWording(errorBody)`, which is// deliberately status-agnostic and whose phrase list contains the bare- // substring "quota" — so a recovery 503 that merely mentioned one would be- // read as a spent subscription and walk the user onto metered billing in the- // middle of a network outage.+ // substring "quota" — so one of our own connection verdicts that merely+ // mentioned one would be read as a spent subscription and walk the user onto+ // metered billing in the middle of a network outage. The error text quotes+ // the endpoint HOST and URL, so "merely mentioned" is not hypothetical: a+ // host named `quota-…` reproduced it, and only that word did.//- // `handle()` already returns a marked response verbatim before this function- // is consulted, so in the shipped call graph this branch is unreachable.- // It is kept anyway, and it is not belt-and-braces decoration: it makes the- // guarantee a property of the DECISION rather than of one call site, so a- // future refactor that moves, inlines or reorders the check in `handle()`- // cannot silently re-open the billing hole. Two independent mechanisms, one- // marker.- if (hasRecoveryMarker(headers)) return false;+ // BOTH arms, via one predicate. The 503 handoff carries `x-claudish-recovery`+ // and is already returned verbatim by `handle()` above, so for that arm this+ // branch is a second independent mechanism. The 400 arm — every headless run,+ // where no banner can exist — carries `x-claudish-connection-error`, and for+ // IT this branch is the only mechanism there is. Asking the combined+ // predicate rather than either half is what stops chain-safety from being a+ // property of which arm happened to answer.+ if (isClaudishConnectionVerdict(headers)) return false;// A spent subscription allowance is retryable AT THE CHAIN LEVEL: this// provider cannot serve, but the next one can.diff --git a/packages/cli/src/handlers/shared/recovery-marker.ts b/packages/cli/src/handlers/shared/recovery-marker.tsindex 648288bb..75177c33 100644--- a/packages/cli/src/handlers/shared/recovery-marker.ts+++ b/packages/cli/src/handlers/shared/recovery-marker.ts@@ -51,7 +51,10 @@* the marker, so a forged one on a 200 changes nothing.** Keep both properties true. If a future edit returns an upstream `Response`- * object on a non-ok path, the marker must be stripped there.+ * object on a non-ok path, the markers must be stripped there.+ *+ * Both markers below rest on exactly this argument, and both are minted at a+ * `c.json(...)` call whose headers Hono builds from nothing.*//** The header name. Referenced by name in no other file — import this. */@@ -60,6 +63,49 @@ export const RECOVERY_MARKER_HEADER = "x-claudish-recovery";/** The only value that counts. Anything else is not our response. */export const RECOVERY_MARKER_VALUE = "1";+/**+ * The SECOND marker, and the reason it exists rather than the first one being+ * reused.+ *+ * ── THE DEFECT IT CLOSES (found by a black-box contract test, F-2) ──────────+ *+ * `x-claudish-recovery` is minted on ONE arm: the 503 handoff, which requires a+ * valid UI lease. When a hold exhausts with no banner — `-p`, `--no-recovery`,+ * CI, any machine without magmux, i.e. EVERY headless run — the answer is a 400+ * `connection_error` and there was no marker at all. `isRetryableError` then+ * fell through to its first real statement, `hasQuotaExhaustionWording`, whose+ * phrase list contains the bare substring `"quota"` — and the error text of a+ * connection failure quotes the ENDPOINT HOST AND URL verbatim. A host named+ * `quota-exceeded…` (or any URL containing the word) was therefore read as a+ * spent subscription, and the chain ADVANCED off a possibly-subscription+ * candidate onto a metered one in the middle of a network outage. Measured:+ * candidate 2's own socket served the client; `insufficient-credits` and+ * `rate-limit` in the same position did not reproduce it, so the variable+ * really was the word.+ *+ * The guarantee the two markers hold together, and the shape of it matters:+ *+ * > **A connection-failure response from this feature never advances the+ * > fallback chain — regardless of its STATUS, its MESSAGE, or whether a banner+ * > was attached.**+ *+ * Chain-safety must not be a property of the arm that happened to answer. The+ * 503 arm's marker made it structural in one arm only, which is worse than+ * obvious to review: the guard reads as present, and it is present, but exactly+ * where the money is at stake — a run with no surface — it is not minted.+ *+ * ── WHY NOT SIMPLY MINT `x-claudish-recovery` ON BOTH ARMS ──────────────────+ *+ * Because the two facts are different, and one of them is load-bearing+ * elsewhere. `x-claudish-recovery` means "the retry was HANDED BACK; re-POST+ * and you will rejoin the same episode" — `FallbackHandler` returns such a+ * response VERBATIM, before reading the body, so `formatCombinedError` cannot+ * demote its 503 to a terminal 400. A 400 has nothing handed back and must+ * still be foldable into a combined chain error, which is where the user reads+ * what every candidate did. Same guarantee, two facts, two headers.+ */+export const CONNECTION_FAULT_HEADER = "x-claudish-connection-error";+/*** The headers a tier-1 exhaustion 503 carries.*@@ -75,6 +121,21 @@ export function recoveryHoldHeaders(): Record<string, string> {};}+/**+ * The headers claudish's own "could not reach the host" 400 carries.+ *+ * Deliberately NOT `recoveryHoldHeaders()`: no `x-should-retry`, because+ * nothing was handed back, and no `x-claudish-recovery`, because that header+ * means "handed back" and `FallbackHandler` returns anything wearing it+ * verbatim — which would silently drop the other candidates' reasons out of a+ * combined chain error.+ *+ * `content-type` is left to the caller's `c.json`, which sets it.+ */+export function connectionFaultHeaders(): Record<string, string> {+ return { [CONNECTION_FAULT_HEADER]: RECOVERY_MARKER_VALUE };+}+/** Does this header bag carry the marker? Tolerates `undefined` for callers that have none. */export function hasRecoveryMarker(headers: Headers | undefined | null): boolean {try {@@ -85,6 +146,27 @@ export function hasRecoveryMarker(headers: Headers | undefined | null): boolean}}+/** Does this header bag carry the connection-fault marker? */+export function hasConnectionFaultMarker(headers: Headers | undefined | null): boolean {+ try {+ return headers?.get(CONNECTION_FAULT_HEADER) === RECOVERY_MARKER_VALUE;+ } catch {+ return false;+ }+}++/**+ * Is this claudish's own verdict that it could not reach the host — on EITHER+ * arm?+ *+ * The one predicate the chain-safety rule is written in terms of, so the rule+ * cannot again be true of one arm and false of the other. A caller asking "may+ * I advance past this?" must ask THIS, never one of the two halves.+ */+export function isClaudishConnectionVerdict(headers: Headers | undefined | null): boolean {+ return hasRecoveryMarker(headers) || hasConnectionFaultMarker(headers);+}+/** Is this response claudish's own recovery-exhaustion 503? */export function isRecoveryHoldResponse(response: { headers?: Headers } | null | undefineddiff --git a/packages/cli/src/handlers/shared/transient-retry.ts b/packages/cli/src/handlers/shared/transient-retry.tsindex 9cc7bb36..5068100e 100644--- a/packages/cli/src/handlers/shared/transient-retry.ts+++ b/packages/cli/src/handlers/shared/transient-retry.ts@@ -25,7 +25,7 @@*/import type { Context } from "hono";-import { log } from "../../logger.js";+import { log, logRecovery } from "../../logger.js";import { recoveryClock } from "../../recovery/clock.js";import { type EpisodeHandle, joinEpisode } from "../../recovery/coordinator.js";import { logDeadlineIfShortened, resolveTier1DeadlineMs } from "../../recovery/settings.js";@@ -108,6 +108,17 @@ export function refreshDeadlineAt(deadlineAt: number): number {* The residual is real and wanted: a candidate that recovers late and then* returns a retryable status leaves candidate 2 with only the remaining budget* rather than a fresh one. That is the point.+ *+ * ── THE ZERO POINT IS STAMPED BY `handle()`, NOT BY THE FIRST READER ────────+ *+ * Memoisation makes this function's value "whenever somebody first asked", and+ * on the fetch path the first asker used to be `tier1DeadlineAt(c)` inside the+ * `catch` — after attempt 1 had already failed. MEASURED against an unrouted+ * address: attempt 1 spent macOS's 75 s connect timeout, the 30 s budget then+ * started counting from t+75 s, and the client was answered at 108 s. The+ * budget had not been exceeded; it had never been applied. `handle()` now+ * stamps it on entry, which is why a caller must never assume this is cheap to+ * call late — it is only correct to call it EARLY.*/const INBOUND_START = new WeakMap<Request, number>();@@ -121,11 +132,91 @@ export function inboundStartedAtPerf(c: Context): number {return t;}-/** The absolute tier-1 deadline for this inbound request, PROCESS ms. */+/**+ * The absolute tier-1 deadline for this inbound request, PROCESS ms.+ *+ * PURE, and it has to stay pure: it is now read BEFORE the primary fetch (to+ * arm the request ceiling below), so anything it announced would be announced+ * on healthy requests too. The shortened-hold notice moved to+ * `noteDeadlineForEpisode`, which runs only once an episode really opens.+ */export function tier1DeadlineAt(c: Context): number {- const budget = resolveTier1DeadlineMs();- logDeadlineIfShortened(budget);- return inboundStartedAtPerf(c) + budget;+ return inboundStartedAtPerf(c) + resolveTier1DeadlineMs();+}++/**+ * Announce a hold the environment shortened — from the ladder, once per+ * process.+ *+ * Separated from `tier1DeadlineAt` because that function is now on the healthy+ * path. A user who has set `API_TIMEOUT_MS=60000` and never has an outage must+ * not be told about a hold that never happened.+ */+export function noteDeadlineForEpisode(): void {+ logDeadlineIfShortened(resolveTier1DeadlineMs());+}++/** An armed ceiling on ONE outbound call. `signal` is absent when there is none. */+export interface DeadlineClamp {+ /** Compose into the fetch init with `mergeSignalIntoInit`. */+ signal?: AbortSignal;+ /** Call the INSTANT the call settles. A live body must never be cut. */+ disarm(): void;+}++/**+ * Bound one outbound call by an ABSOLUTE deadline — the ceiling that makes the+ * derived budget authoritative for the FIRST attempt as well as for the ladder.+ *+ * ── WHY ATTEMPT 1 NEEDED A CEILING AT ALL ───────────────────────────────────+ *+ * The ladder's attempts were already clamped (`PER_ATTEMPT_CONNECT_CAP_MS`),+ * but attempt 1 — the byte-identical primary fetch — was not, on the reasoning+ * that `DEADLINE_MARGIN_MS` (30 s) reserves room for its tail. macOS's own TCP+ * connect timeout is **75 s**, so the reserve is wrong by 45 s the moment the+ * fault is a SLOW connect rather than a refusal, and it is wrong by more than+ * the entire budget whenever `API_TIMEOUT_MS ≤ 105 s`. A single connect then+ * outlived the whole request deadline and the client (which gives up at+ * `API_TIMEOUT_MS`) was gone before we answered. Every refusal-based test is+ * blind to this: a 1 ms attempt cannot cross a deadline in the middle of+ * itself.+ *+ * ── WHY THE DEADLINE AND NOT `PER_ATTEMPT_CONNECT_CAP_MS` ───────────────────+ *+ * This ceiling is on a possibly-HEALTHY call, and 45 s is a plausible+ * time-to-first-byte for a thinking model behind a long prompt. Clamping the+ * primary fetch at 45 s would convert an ordinary LATENCY event into "the host+ * is unreachable" — the exact leak past RISK-6's boundary that this design+ * already found and closed once. At the default settings this ceiling is 270 s,+ * which no provider's first byte approaches, and it shortens only as the user's+ * own `API_TIMEOUT_MS` does.+ *+ * ── AND WHY FIRING AT THE DEADLINE IS WHAT MAKES IT SAFE ────────────────────+ *+ * Because the abort lands AT `deadlineAt`, there is by construction no budget+ * left when it is classified: `shouldSkipTier1`'s `no-budget` gate fires and+ * the request answers. So this ceiling can never cause a RE-ISSUE, and cannot+ * put a slow-but-alive inference on the ladder to be billed twice. An earlier+ * firing time would not have that property — keep it absolute.+ */+export function deadlineClamp(deadlineAt: number): DeadlineClamp {+ const clock = recoveryClock();+ const remaining = deadlineAt - clock.now();+ // NaN-safe: an unparseable budget leaves the call unbounded, which is+ // pre-recovery behaviour rather than an instant abort.+ if (!(remaining > 0)) return { disarm: () => {} };+ const ac = new AbortController();+ const timer = clock.setTimeout(+ // `markOwnTimeout` is what lets this be classified at all — the discriminator+ // is the signal's ORIGIN, never the name, so a transport's own inference+ // ceiling keeps its pre-recovery route out.+ () => ac.abort(markOwnTimeout(new DOMException("request deadline", "TimeoutError"))),+ remaining+ );+ return {+ signal: ac.signal,+ disarm: () => clock.clearTimeout(timer),+ };}export interface ConnectionErrorInfo {@@ -348,6 +439,10 @@ export async function withConnectionRetry<T>(let error: unknown = first;let endpoint = ctx.resolveEndpoint(first);+ // A shortened hold is announced HERE — once a hold is really happening —+ // rather than by `tier1DeadlineAt`, which the healthy path now calls.+ noteDeadlineForEpisode();+const handle: EpisodeHandle = joinEpisode({providerName: ctx.providerName,providerDisplayName: ctx.providerDisplayName,@@ -377,17 +472,75 @@ export async function withConnectionRetry<T>(handle.recordAttemptResult(conn.code, false);for (;;) {+ // ── FOUR ABORT PATHS, AND WHY NONE OF THEM IS REDUNDANT ───────────────+ //+ // A black-box mutation (M6) disabled the three in THIS file — this check,+ // `untilAborted`, and `mergeSignalIntoInit`'s threading — and the client+ // disconnect was still handled. It was not proof of redundancy: the+ // surviving path is `coordinator.ts`'s `waitForNextAttempt`, which+ // registers its own `abort` listener on the same signal, and a ladder+ // against a REFUSED loopback port spends ~100% of its wall clock parked+ // in that wait. The mutation test aborted during a wait, so the wait's+ // own listener answered it.+ //+ // Each covers a window no other one does:+ //+ // 1. HERE — an abort that landed while an attempt was settling, or+ // before any wait has happened at all (first iteration). Nothing+ // else is listening in that gap.+ // 2. `coordinator.waitForNextAttempt` — during the wait. Also the only+ // one that can DETACH the waiter from the shared episode timer.+ // 3. `untilAborted` — during an attempt whose operation structurally+ // cannot take a signal: `refreshAuth()`/`getHeaders()` accept no+ // arguments, so for the auth path this is the ONLY unwind there is.+ // 4. `mergeSignalIntoInit` — inside `fetch`, so the SOCKET closes+ // rather than being abandoned. The other three unwind the waiter and+ // leave the connect running.+ //+ // A slow connect inverts the mutation's arithmetic: 45 s of attempt to+ // 5 s of wait means (3) and (4) do nearly all the work and (2) almost+ // none. Deleting any of them leaves a window that only shows up under the+ // fault class the loopback suite cannot produce.if (ctx.signal.aborted) {return { kind: "client_gone", ...tally() };}- const delay = connectionRetryDelayMs(handle.ladderIndex());- const now = clock.now();- // The check bounds the ATTEMPT THAT FOLLOWS the sleep, not just the- // sleep. Bounding only the sleep is how a budget stops being a ceiling.- if (now + delay + MIN_ATTEMPT_SLOT_MS > ctx.deadlineAt) {- handle.handoff();- return { kind: "exhausted", conn, error, endpoint, ...tally() };+ // The budget this waiter has left for a WAIT: its own deadline, less the+ // slot the attempt after the wait needs. The check bounds the ATTEMPT+ // THAT FOLLOWS the sleep, not just the sleep — bounding only the sleep is+ // how a budget stops being a ceiling.+ const waitBudget = ctx.deadlineAt - clock.now() - MIN_ATTEMPT_SLOT_MS;+ let delay = connectionRetryDelayMs(handle.ladderIndex());++ if (delay > waitBudget) {+ // ── A CARRIED RUNG MAY BE TRUNCATED, ONCE, BY A REQUEST THAT HAS NOT+ // RE-ISSUED ANYTHING YET ──────────────────────────────────────────+ //+ // The ladder index belongs to the EPISODE and deliberately does not+ // reset when the client re-asks (§2: restarting at 5 s would hammer a+ // dead host harder the longer the outage lasted). But the deadline+ // belongs to the SOCKET, and a re-ask gets a fresh, possibly SHORTER+ // one. Once the carried rung outgrew that deadline — 60 s carried into+ // a 30 s budget — this gate fired on the very first iteration and the+ // request answered in 8 ms having re-issued NOTHING. Measured at the+ // upstream socket: with `API_TIMEOUT_MS=60000`, request 2 of an outage+ // made one connect and was done, permanently, because the 400 it+ // answered is not a status Claude Code re-asks. Invisible at the 270 s+ // default, where the terminal 60 s rung always fits; it bites every+ // `API_TIMEOUT_MS ≤ 90 s`, a value the docs invite.+ //+ // So a request that has not yet re-issued once spends what it has+ // instead of handing back an empty turn: the wait shrinks to the+ // budget, one attempt is made, and only then does it hand off. It is+ // not a hammer — the floor is `MIN_ATTEMPT_SLOT_MS`, it happens at most+ // once per request, and the gap between two client re-asks is the+ // client's own backoff (up to 38.4 s), not ours.+ const mayTruncate = requestRetries === 0 && waitBudget >= MIN_ATTEMPT_SLOT_MS;+ if (!mayTruncate) {+ handle.handoff();+ return { kind: "exhausted", conn, error, endpoint, ...tally() };+ }+ delay = waitBudget;}let waited: { kind: "attempt" } | { kind: "gave_up" };@@ -427,7 +580,7 @@ export async function withConnectionRetry<T>(requestRetries++;const value = await untilAborted(op(merged), merged);handle.recordAttemptResult("ok", true);- log(+ logRecovery(`[Recovery] ${ctx.providerDisplayName} recovered after ${handle.attempts()} attempts ` +`in ${handle.recoveryMs()}ms (episode ${handle.episodeId})`);diff --git a/packages/cli/src/logger.ts b/packages/cli/src/logger.tsindex df61b230..511f1007 100644--- a/packages/cli/src/logger.ts+++ b/packages/cli/src/logger.ts@@ -319,6 +319,36 @@ export function logStderr(message: string): void {log(message); // always write to debug log}+/**+ * A network-recovery episode's own account of itself: opened, each attempt,+ * each wait, the outcome.+ *+ * ── WHY THIS IS STRUCTURAL OUTPUT AND NOT A DEBUG AID ───────────────────────+ *+ * These lines were on `log()`, which writes to two FILES — the `--debug` log+ * and the always-on structural log under `~/.claudish/logs/`. That satisfies+ * "it is written down" and fails the thing it was written down for. Recovery+ * holds an inbound request for up to the derived deadline (~4.5 minutes at the+ * default), and in a run with no pane — `-p`, CI, a pipe, any machine without+ * magmux, which is EVERY headless run — nothing at all reaches the user while+ * it does. A silent multi-minute hold with the reason legible nowhere is the+ * state this feature's own design calls "strictly worse than the bug it exists+ * to remove"; a file the user does not know to open is not a surface.+ *+ * The noise objection is real and is already answered by the channel: in an+ * interactive session `logStderr` routes to `DiagOutput` (a file) rather than+ * to the client's TUI, and in quiet mode it is suppressed. So the lines land on+ * a terminal exactly where a terminal is the only surface there is, which is+ * the case that needed them.+ *+ * Fine-grained plumbing — pane connect/disconnect, socket paths, a skipped+ * ladder — stays on `log()`. This is for the episode's LIFECYCLE only: the six+ * or so lines from which a reader can reconstruct the ladder.+ */+export function logRecovery(message: string): void {+ logStderr(message);+}+/*** Set the DiagOutput instance. When set, logStderr() routes to it* instead of stderr. This replaces the stderrQuiet mechanism fordiff --git a/packages/cli/src/recovery-contract/child-proxy-run.ts b/packages/cli/src/recovery-contract/child-proxy-run.tsnew file mode 100644index 00000000..2fdb09ef--- /dev/null+++ b/packages/cli/src/recovery-contract/child-proxy-run.ts@@ -0,0 +1,93 @@+/**+ * A real proxy in a REAL CHILD PROCESS, for the observations that cannot be+ * made in-process:+ *+ * • which FILE DESCRIPTOR a diagnostic took. An in-process+ * `process.stderr.write` monkeypatch cannot tell stdout from stderr — it+ * intercepts the call, not the fd. This project has a terminal-isolation+ * module precisely because a stray write corrupted a client's TUI.+ * • whether the process EXITS after a recovery episode, which is invisible to+ * a test running inside it: a live 60s timer keeps the loop alive.+ *+ * Usage (all arguments positional):+ * bun child-proxy-run.ts <mode> <endpointUrl> <apiTimeoutMs> [abortAfterMs]+ * mode: "exhaust" | "abort"+ *+ * stdout carries exactly one JSON result line, and nothing else may appear+ * there. Everything diagnostic belongs on stderr.+ */++import { mkdtempSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";+import { setConfigFileOverride } from "../config-override.js";+import { createProxyServer } from "../proxy-server.js";++const [mode, endpointUrl, apiTimeoutMs, abortAfterMs] = process.argv.slice(2);++// A distinctive secret: nothing the proxy prints may ever contain it.+const SECRET_KEY = "sk-blackbox-SECRET-VALUE-9f3c1d";++process.env.API_TIMEOUT_MS = apiTimeoutMs ?? "40000";+delete process.env.CLAUDISH_RECOVERY;++const dir = mkdtempSync(join(tmpdir(), "recovery-child-"));+const configPath = join(dir, "config.json");+writeFileSync(+ configPath,+ JSON.stringify({+ customEndpoints: {+ "ep-child": {+ kind: "simple",+ url: endpointUrl,+ format: "openai",+ apiKey: SECRET_KEY,+ models: ["m-child"],+ },+ },+ }),+ "utf8"+);+setConfigFileOverride(configPath);++const proxy = await createProxyServer(0, undefined, undefined, false, undefined, undefined, {+ quiet: false,+});++const body = JSON.stringify({+ model: "ep-child@m-child",+ max_tokens: 32,+ messages: [{ role: "user", content: "hello" }],+});++const controller = new AbortController();+if (mode === "abort") {+ setTimeout(() => controller.abort(), Number(abortAfterMs ?? 7000));+}++const started = Date.now();+let status = 0;+let text = "";+let aborted = false;+try {+ const res = await fetch(`${proxy.url}/v1/messages`, {+ method: "POST",+ headers: { "content-type": "application/json" },+ body,+ signal: controller.signal,+ });+ status = res.status;+ text = await res.text();+} catch {+ aborted = true;+}++const requestMs = Date.now() - started;+const shutdownStarted = Date.now();+await proxy.shutdown();+const shutdownMs = Date.now() - shutdownStarted;++process.stdout.write(+ `${JSON.stringify({ mode, status, aborted, requestMs, shutdownMs, bodyPrefix: text.slice(0, 200) })}\n`+);+process.exit(0);diff --git a/packages/cli/src/recovery-contract/fixtures.ts b/packages/cli/src/recovery-contract/fixtures.tsnew file mode 100644index 00000000..3a94d5ac--- /dev/null+++ b/packages/cli/src/recovery-contract/fixtures.ts@@ -0,0 +1,561 @@+/**+ * Black-box fault fixtures for the network-recovery contract tests.+ *+ * Written from the requirements + the published contract only. Nothing in this+ * file imports, names, or reaches into the recovery implementation: every+ * observation is made either at the PROXY'S OWN URL (a real `fetch`, so the+ * status asserted is the one that survived any internal remap) or at an+ * UPSTREAM SOCKET we own (so an attempt is counted because a TCP connection+ * arrived, never because a log line said so).+ *+ * The two ground rules this file exists to enforce:+ * 1. attempts are counted at the upstream socket, not from `[Recovery]` logs;+ * 2. statuses are read from `Response.status` of a fetch against `proxy.url`.+ */++import { mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";+import { setConfigFileOverride } from "../config-override.js";+import { createProxyServer } from "../proxy-server.js";+import type { ProxyServer } from "../types.js";++/** A single upstream connection observed at a fault fixture. */+export interface ObservedConnection {+ /** ms since the fixture's `t0` (set when the fixture was created). */+ at: number;+ /** The raw request body bytes this connection delivered ("" if it was reset first). */+ body: string;+ /** Raw request head (request line + headers), lowercased header names. */+ headers: Record<string, string>;+}++export interface RawFixture {+ readonly port: number;+ /** Connections in arrival order. */+ readonly connections: ObservedConnection[];+ /** ms offsets of each connection relative to the FIRST connection. */+ offsetsFromFirst(): number[];+ /** Gaps between consecutive connections, in ms. */+ gaps(): number[];+ /** Reset the recorded connections (keeps the server listening). */+ reset(): void;+ stop(): void;+}++export interface RawFixtureOptions {+ /**+ * 1-based attempt index from which the fixture answers with a real HTTP 200.+ * Attempts before it are dropped. `Infinity` (the default) never answers.+ */+ serveFromAttempt?: number;+ /** Text the served SSE answer carries. */+ sseText?: string;+ /**+ * "onOpen" — reset the socket the instant it is accepted (fastest; the+ * request body is never seen, so attempts are timed but not read).+ * "afterBody" — read the whole request first, record it, then reset. This is+ * what makes a body-replay assertion possible.+ */+ resetWhen?: "onOpen" | "afterBody";+ /** Delay (ms) before answering, once the fixture has decided to answer. */+ respondDelayMs?: number;+}++/**+ * An OpenAI-shaped streaming answer. The proxy always talks `stream:true`+ * upstream (measured against a live fixture), so a non-stream JSON body comes+ * back with empty content — this is the shape a fixture must produce.+ */+export function openAiSse(text: string): string {+ const chunk = (obj: unknown) => `data: ${JSON.stringify(obj)}\n\n`;+ return (+ chunk({+ id: "chatcmpl-fixture",+ object: "chat.completion.chunk",+ created: 1,+ model: "fixture",+ choices: [{ index: 0, delta: { role: "assistant", content: text }, finish_reason: null }],+ }) ++ chunk({+ id: "chatcmpl-fixture",+ object: "chat.completion.chunk",+ created: 1,+ model: "fixture",+ choices: [{ index: 0, delta: {}, finish_reason: "stop" }],+ usage: { prompt_tokens: 5, completion_tokens: 3, total_tokens: 8 },+ }) ++ "data: [DONE]\n\n"+ );+}++interface SocketLike {+ write(data: string): unknown;+ flush(): unknown;+ end(): unknown;+ terminate?: () => void;+}++/** Write a canned HTTP response after `delayMs`, tolerating a peer that left. */+function answerLater(sock: SocketLike, body: string, delayMs: number): void {+ setTimeout(() => {+ try {+ sock.write(body);+ sock.flush();+ sock.end();+ } catch {+ /* peer gone */+ }+ }, delayMs);+}++/**+ * Parse one complete HTTP/1.1 request out of the bytes so far.+ * Returns null while the head or the declared body is still incomplete.+ */+function parseHttpRequest(buf: Buffer): { headers: Record<string, string>; body: string } | null {+ const raw = buf.toString("latin1");+ const headEnd = raw.indexOf("\r\n\r\n");+ if (headEnd === -1) return null;++ const head = raw.slice(0, headEnd);+ const headers: Record<string, string> = {};+ for (const line of head.split("\r\n").slice(1)) {+ const i = line.indexOf(":");+ if (i > 0) headers[line.slice(0, i).trim().toLowerCase()] = line.slice(i + 1).trim();+ }++ const headLen = Buffer.byteLength(head, "latin1") + 4;+ const bodyBytes = buf.length - headLen;+ const declared = headers["content-length"];+ if (declared !== undefined && bodyBytes < Number(declared)) return null;+ if (declared === undefined && headers["transfer-encoding"]?.includes("chunked")) {+ if (!raw.endsWith("0\r\n\r\n")) return null;+ }+ return { headers, body: buf.subarray(headLen).toString("utf8") };+}++function httpOk(body: string): string {+ return [+ "HTTP/1.1 200 OK",+ "Content-Type: text/event-stream",+ "Cache-Control: no-cache",+ `Content-Length: ${Buffer.byteLength(body)}`,+ "Connection: close",+ "",+ body,+ ].join("\r\n");+}++/**+ * A raw TCP fixture on a FIXED port.+ *+ * Every accepted connection is one upstream attempt, timestamped at accept+ * time. This is the instrument for anything that asserts the ladder's SHAPE:+ * a refused port cannot count and cannot be slow.+ */+export function startRawFixture(port: number, options: RawFixtureOptions = {}): RawFixture {+ const {+ serveFromAttempt = Number.POSITIVE_INFINITY,+ sseText = "RAW-FIXTURE-ANSWER",+ resetWhen = "onOpen",+ respondDelayMs = 0,+ } = options;++ const connections: ObservedConnection[] = [];+ let t0 = Date.now();++ interface PerSocket {+ index: number;+ buf: Buffer;+ recorded: boolean;+ }++ const server = Bun.listen<PerSocket>({+ hostname: "127.0.0.1",+ port,+ socket: {+ open(sock) {+ const index = connections.length + 1;+ sock.data = { index, buf: Buffer.alloc(0), recorded: false };+ if (resetWhen === "onOpen") {+ connections.push({ at: Date.now() - t0, body: "", headers: {} });+ if (index >= serveFromAttempt) {+ answerLater(sock, httpOk(openAiSse(sseText)), respondDelayMs);+ } else {+ hardClose(sock);+ }+ }+ },+ data(sock, chunk) {+ if (resetWhen === "onOpen") return;+ const state = sock.data;+ if (state.recorded) return;+ state.buf = Buffer.concat([state.buf, chunk]);+ const parsed = parseHttpRequest(state.buf);+ if (parsed === null) return;++ state.recorded = true;+ connections.push({ at: Date.now() - t0, body: parsed.body, headers: parsed.headers });+ if (state.index >= serveFromAttempt) {+ answerLater(sock, httpOk(openAiSse(sseText)), respondDelayMs);+ } else {+ hardClose(sock);+ }+ },+ close() {},+ error() {},+ drain() {},+ },+ });++ return {+ port: server.port,+ connections,+ offsetsFromFirst() {+ const first = connections[0];+ if (first === undefined) return [];+ return connections.map((c) => c.at - first.at);+ },+ gaps() {+ const offsets = this.offsetsFromFirst();+ return offsets.slice(1).map((o, i) => o - (offsets[i] ?? 0));+ },+ reset() {+ connections.length = 0;+ t0 = Date.now();+ },+ stop() {+ try {+ server.stop(true);+ } catch {+ /* already stopped */+ }+ },+ };+}++function hardClose(sock: { terminate?: () => void; end: () => void }): void {+ try {+ if (typeof sock.terminate === "function") sock.terminate();+ else sock.end();+ } catch {+ /* already gone */+ }+}++/** A healthy upstream that records what it received. */+export interface HealthyFixture {+ readonly port: number;+ readonly requests: { at: number; body: string; authorization: string | null; path: string }[];+ stop(): void;+}++export function startHealthyFixture(text = "HEALTHY-FIXTURE-ANSWER", port = 0): HealthyFixture {+ const requests: HealthyFixture["requests"] = [];+ const t0 = Date.now();+ const server = Bun.serve({+ port,+ async fetch(req) {+ requests.push({+ at: Date.now() - t0,+ body: await req.text(),+ authorization: req.headers.get("authorization"),+ path: new URL(req.url).pathname,+ });+ return new Response(openAiSse(text), {+ headers: { "content-type": "text/event-stream" },+ });+ },+ });+ return {+ port: server.port as number,+ requests,+ stop() {+ try {+ server.stop(true);+ } catch {+ /* already stopped */+ }+ },+ };+}++/** An upstream that is reachable and answers a real HTTP error status. */+export function startStatusFixture(status: number, port = 0) {+ let hits = 0;+ const server = Bun.serve({+ port,+ fetch() {+ hits += 1;+ return new Response(JSON.stringify({ error: { message: "fixture-upstream-error" } }), {+ status,+ headers: { "content-type": "application/json" },+ });+ },+ });+ return {+ port: server.port as number,+ get hits() {+ return hits;+ },+ stop() {+ try {+ server.stop(true);+ } catch {+ /* already stopped */+ }+ },+ };+}++/**+ * Reserve `count` ports by binding them all at once and then releasing them.+ * Binding simultaneously guarantees the numbers differ from each other.+ *+ * NEVER port 1: it is privileged, and a recovery fixture has to be able to+ * bind the same port again later.+ */+export function reservePorts(count: number): number[] {+ const servers = Array.from({ length: count }, () =>+ Bun.serve({ port: 0, fetch: () => new Response("reserved") })+ );+ const ports = servers.map((s) => s.port as number);+ for (const s of servers) s.stop(true);+ return ports;+}++/**+ * P-1: prove the "refused" fault is really refused, immediately before the run.+ *+ * A sibling test that grabbed the port turns a refusal into a live server and+ * inverts the scenario silently. Fails loudly — never skips.+ */+export async function assertRefused(port: number): Promise<void> {+ try {+ const res = await fetch(`http://127.0.0.1:${port}/`, {+ signal: AbortSignal.timeout(1000),+ });+ throw new Error(+ `PRECONDITION P-1 FAILED: 127.0.0.1:${port} answered HTTP ${res.status}; ` ++ "the refusal fault is not in force (a sibling test probably took the port)."+ );+ } catch (err) {+ const msg = err instanceof Error ? err.message : String(err);+ if (msg.startsWith("PRECONDITION")) throw err;+ // Any connect-level rejection is what we wanted.+ }+}++export interface EndpointSpec {+ name: string;+ port: number;+ model: string;+ apiKey?: string;+ /** Override the URL entirely (for DNS / TEST-NET faults). */+ url?: string;+}++export function endpointsConfig(specs: EndpointSpec[]): Record<string, unknown> {+ const customEndpoints: Record<string, unknown> = {};+ for (const spec of specs) {+ customEndpoints[spec.name] = {+ kind: "simple",+ url: spec.url ?? `http://127.0.0.1:${spec.port}/v1`,+ format: "openai",+ apiKey: spec.apiKey ?? `test-key-${spec.name}`,+ models: [spec.model],+ };+ }+ return { customEndpoints };+}++/**+ * A pool of identical raw fault fixtures, one endpoint each.+ *+ * MEASURED, and the reason this exists: a recovery EPISODE is keyed per+ * endpoint and survives the request that opened it — a second request to the+ * same dead endpoint continues the ladder at the next rung rather than+ * restarting at 5s. So two arms of a paired experiment that share an endpoint+ * are not comparable: the later arm inherits the earlier arm's rung. Every arm+ * takes a FRESH endpoint from this pool, which also makes the file+ * order-independent.+ */+export interface FixturePool {+ specs: EndpointSpec[];+ /** Hand out an endpoint/fixture pair nobody else in this file has used. */+ next(): { model: string; fixture: RawFixture; name: string };+ stopAll(): void;+}++export function createRawFixturePool(+ count: number,+ prefix: string,+ options: RawFixtureOptions = {}+): FixturePool {+ const ports = reservePorts(count);+ const fixtures = ports.map((p) => startRawFixture(p, options));+ const specs: EndpointSpec[] = ports.map((p, i) => ({+ name: `${prefix}-${i + 1}`,+ port: p,+ model: `m-${i + 1}`,+ }));+ let cursor = 0;+ return {+ specs,+ next() {+ if (cursor >= specs.length) {+ throw new Error(+ `fixture pool "${prefix}" exhausted after ${specs.length} arms — ` ++ "raise the pool size rather than reusing an endpoint (episodes are per-endpoint)."+ );+ }+ const spec = specs[cursor] as EndpointSpec;+ const fixture = fixtures[cursor] as RawFixture;+ cursor += 1;+ return { model: `${spec.name}@${spec.model}`, fixture, name: spec.name };+ },+ stopAll() {+ for (const f of fixtures) f.stop();+ },+ };+}++export interface ProxyHandle {+ proxy: ProxyServer;+ url: string;+ /** Everything the proxy wrote to stderr while this handle was live. */+ stderr(): string;+ shutdown(): Promise<void>;+}++export interface StartProxyOptions {+ model?: string;+ modelChain?: string[];+ quiet?: boolean;+ captureStderr?: boolean;+}++/**+ * Start a real proxy against a temp config file, exactly as `proxy-server.test.ts`+ * does — the config-file override, never a write to the developer's real config.+ */+export async function startProxy(+ config: Record<string, unknown>,+ options: StartProxyOptions = {}+): Promise<ProxyHandle> {+ const dir = mkdtempSync(join(tmpdir(), "recovery-contract-"));+ const configPath = join(dir, "config.json");+ writeFileSync(configPath, JSON.stringify(config), "utf8");+ setConfigFileOverride(configPath);++ const captured: string[] = [];+ const realWrite = process.stderr.write.bind(process.stderr);+ if (options.captureStderr) {+ process.stderr.write = ((chunk: string | Uint8Array) => {+ captured.push(typeof chunk === "string" ? chunk : Buffer.from(chunk).toString("utf8"));+ return true;+ }) as typeof process.stderr.write;+ }++ let proxy: ProxyServer;+ try {+ proxy = await createProxyServer(0, undefined, options.model, false, undefined, undefined, {+ quiet: options.quiet ?? true,+ modelChain: options.modelChain,+ });+ } catch (err) {+ if (options.captureStderr) process.stderr.write = realWrite;+ setConfigFileOverride(null);+ rmSync(dir, { recursive: true, force: true });+ throw err;+ }++ return {+ proxy,+ url: proxy.url,+ stderr: () => captured.join(""),+ async shutdown() {+ if (options.captureStderr) process.stderr.write = realWrite;+ try {+ await proxy.shutdown();+ } finally {+ setConfigFileOverride(null);+ rmSync(dir, { recursive: true, force: true });+ }+ },+ };+}++export interface MessageRequest {+ model: string;+ stream?: boolean;+ headers?: Record<string, string>;+ content?: unknown;+ signal?: AbortSignal;+ maxTokens?: number;+}++/** POST /v1/messages at the proxy's PUBLIC url, and time it. */+export async function postMessage(+ proxyUrl: string,+ req: MessageRequest+): Promise<{ status: number; body: string; headers: Headers; elapsedMs: number }> {+ const started = Date.now();+ const res = await fetch(`${proxyUrl}/v1/messages`, {+ method: "POST",+ headers: { "content-type": "application/json", ...(req.headers ?? {}) },+ body: JSON.stringify({+ model: req.model,+ max_tokens: req.maxTokens ?? 64,+ ...(req.stream ? { stream: true } : {}),+ messages: [{ role: "user", content: req.content ?? "hello" }],+ }),+ signal: req.signal,+ });+ const body = await res.text();+ return { status: res.status, body, headers: res.headers, elapsedMs: Date.now() - started };+}++/**+ * The deadline the contract DERIVES from `API_TIMEOUT_MS`:+ * min(API_TIMEOUT_MS, 300_000) − 30_000+ * (docs/advanced/environment.md). Computed here from the published formula so+ * a test can never hardcode 270_000.+ */+export function derivedDeadlineMs(apiTimeoutMs: number): number {+ return Math.min(apiTimeoutMs, 300_000) - 30_000;+}++/** The published ladder: 5s → 10s → 30s → 60s → 60s → … (FR-2). */+export const LADDER_MS = [5_000, 10_000, 30_000, 60_000, 60_000, 60_000] as const;++/**+ * Attempt start offsets the published ladder puts inside `deadlineMs`,+ * measured from the first attempt at t=0. Derived, never hardcoded.+ */+export function scheduledAttemptOffsets(deadlineMs: number): number[] {+ const offsets = [0];+ let t = 0;+ for (let i = 0; ; i += 1) {+ const gap = LADDER_MS[Math.min(i, LADDER_MS.length - 1)];+ t += gap;+ if (t >= deadlineMs) break;+ offsets.push(t);+ }+ return offsets;+}++/** Restores env vars a test set, including deleting ones that were absent. */+export function envSnapshot(names: string[]): () => void {+ const saved = new Map<string, string | undefined>();+ for (const n of names) saved.set(n, process.env[n]);+ return () => {+ for (const [n, v] of saved) {+ if (v === undefined) delete process.env[n];+ else process.env[n] = v;+ }+ };+}++export const sleep = (ms: number) => new Promise<void>((r) => setTimeout(r, ms));diff --git a/packages/cli/src/recovery-contract/recovery-chain-blackbox.test.ts b/packages/cli/src/recovery-contract/recovery-chain-blackbox.test.tsnew file mode 100644index 00000000..e812d717--- /dev/null+++ b/packages/cli/src/recovery-contract/recovery-chain-blackbox.test.ts@@ -0,0 +1,178 @@+/**+ * CONTRACT POINT 6 — a connection failure must NEVER advance the provider+ * chain. This is the criterion that separates a fix from a bill: a chain that+ * advances during an outage can move a subscription user onto metered billing.+ *+ * Trap T-12 is the whole design of this file. "No advance" asserted on a chain+ * that COULD NOT have advanced — one candidate, or a second candidate with no+ * credential — is vacuous and passes on any build at all. So:+ * • the chain here is genuinely multi-candidate (two custom endpoints, both+ * constructible, both credentialled);+ * • the POSITIVE CONTROL in the same file drives the same shape of chain into+ * an advance and proves candidate 2 is reachable and would have served;+ * • "did it advance?" is answered at CANDIDATE 2's OWN SOCKET — a request+ * count on a server we own — not by grepping the proxy's log for+ * `[Fallback]`, which is written by the code under test.+ */++import { afterEach, describe, expect, test } from "bun:test";+import {+ type HealthyFixture,+ type ProxyHandle,+ type RawFixture,+ endpointsConfig,+ envSnapshot,+ postMessage,+ reservePorts,+ startHealthyFixture,+ startProxy,+ startRawFixture,+ startStatusFixture,+} from "./fixtures.js";++let restoreEnv: (() => void) | undefined;+let proxy: ProxyHandle | undefined;+const stoppables: { stop(): void }[] = [];++afterEach(async () => {+ await proxy?.shutdown();+ proxy = undefined;+ while (stoppables.length > 0) stoppables.pop()?.stop();+ restoreEnv?.();+ restoreEnv = undefined;+});++const SECOND = "SECOND-CANDIDATE-ANSWER";++function setEnv(apiTimeoutMs: number): void {+ restoreEnv = envSnapshot(["API_TIMEOUT_MS", "CLAUDISH_RECOVERY"]);+ process.env.API_TIMEOUT_MS = String(apiTimeoutMs);+ delete process.env.CLAUDISH_RECOVERY;+}++async function startChain(opts: {+ firstUrl?: string;+ firstPort?: number;+ secondPort: number;+}): Promise<ProxyHandle> {+ const config = endpointsConfig([+ {+ name: "ep-one",+ port: opts.firstPort ?? 0,+ model: "m-one",+ ...(opts.firstUrl ? { url: opts.firstUrl } : {}),+ },+ { name: "ep-two", port: opts.secondPort, model: "m-two" },+ ]);+ const handle = await startProxy(config, {+ model: "ep-one@m-one",+ modelChain: ["ep-one@m-one", "ep-two@m-two"],+ quiet: false,+ captureStderr: true,+ });+ proxy = handle;+ return handle;+}++describe("CP-6 — a connection failure never switches providers", () => {+ test("POSITIVE CONTROL: the same chain DOES advance on a retryable upstream status", async () => {+ setEnv(40_000);+ const [firstPort, secondPort] = reservePorts(2);+ const first = startStatusFixture(429, firstPort);+ const second: HealthyFixture = startHealthyFixture(SECOND, secondPort);+ stoppables.push(first, second);++ const handle = await startChain({ firstPort, secondPort });+ const result = await postMessage(handle.url, { model: "ep-one@m-one" });++ expect(result.status).toBe(200);+ expect(result.body).toContain(SECOND);+ // Candidate 2 was reachable, credentialled and willing — asserted at its+ // own socket. Without this, the no-advance test below is vacuous.+ expect(second.requests.length).toBe(1);+ expect(first.hits).toBeGreaterThanOrEqual(1);+ expect(handle.stderr()).toContain("trying next provider");+ }, 60_000);++ test("an unreachable candidate 1 retries in place and candidate 2 is never contacted", async () => {+ setEnv(40_000);+ const [firstPort, secondPort] = reservePorts(2);+ const first: RawFixture = startRawFixture(firstPort);+ const second: HealthyFixture = startHealthyFixture(SECOND, secondPort);+ stoppables.push(first, second);++ const handle = await startChain({ firstPort, secondPort });+ const result = await postMessage(handle.url, { model: "ep-one@m-one" });++ // The ladder ran on candidate 1 …+ expect(first.connections.length).toBeGreaterThanOrEqual(2);+ // … and candidate 2 never saw a byte. Counted at candidate 2's socket.+ expect(second.requests.length).toBe(0);+ expect(result.body).not.toContain(SECOND);+ expect(JSON.parse(result.body).error.type).toBe("connection_error");+ expect(handle.stderr()).not.toContain("trying next provider");+ }, 60_000);++ test("CONTROL: a neutral DNS failure retries in place and does not advance", async () => {+ // Isolates the variable for the wording test below: the fault kind is the+ // same (name never resolves), only the words in it differ.+ setEnv(40_000);+ const [secondPort] = reservePorts(1);+ const second: HealthyFixture = startHealthyFixture(SECOND, secondPort);+ stoppables.push(second);++ const handle = await startChain({+ firstUrl: "https://no-such-host-blackbox.invalid/v1",+ secondPort,+ });+ const result = await postMessage(handle.url, { model: "ep-one@m-one" });++ expect(second.requests.length).toBe(0);+ expect(JSON.parse(result.body).error.type).toBe("connection_error");+ expect(result.elapsedMs).toBeGreaterThan(4_000); // the ladder really ran+ }, 60_000);++ test("a connection error whose HOST reads like a billing failure still does not advance", async () => {+ // The wording trap, induced from outside: the error text genuinely+ // contains "quota" because the unreachable host is called that. A+ // classifier that matches on message text rather than on the failure's+ // kind advances the chain here and nowhere else.+ setEnv(40_000);+ const [secondPort] = reservePorts(1);+ const second: HealthyFixture = startHealthyFixture(SECOND, secondPort);+ stoppables.push(second);++ // MEASURED while writing this file: of `quota`, `insufficient-credits`+ // and `rate-limit`, only the substring "quota" flips the outcome — the+ // other two behave exactly like the neutral control above. So the+ // variable really is the WORD, not the fault.+ const badHost = "quota-exceeded-insufficient-credits.invalid";+ // P-2: a hijacking resolver would turn this into an ordinary HTTP+ // failure and silently invert the scenario. Fail loudly, never skip.+ let resolved = false;+ try {+ await fetch(`https://${badHost}/v1`, { signal: AbortSignal.timeout(3_000) });+ resolved = true;+ } catch {+ resolved = false;+ }+ expect(+ resolved,+ `PRECONDITION P-2 FAILED: ${badHost} answered. This machine's resolver hijacks .invalid, ` ++ "so the DNS fault is not in force."+ ).toBe(false);++ const handle = await startChain({+ firstUrl: `https://${badHost}/v1`,+ secondPort,+ });+ const result = await postMessage(handle.url, { model: "ep-one@m-one" });++ expect(second.requests.length).toBe(0);+ expect(result.body).not.toContain(SECOND);+ expect(handle.stderr()).not.toContain("trying next provider");+ const err = JSON.parse(result.body).error;+ expect(err.type).toBe("connection_error");+ expect(err.message.toLowerCase()).not.toContain("billing");+ }, 60_000);+});diff --git a/packages/cli/src/recovery-contract/recovery-completes-blackbox.test.ts b/packages/cli/src/recovery-contract/recovery-completes-blackbox.test.tsnew file mode 100644index 00000000..e40d86c1--- /dev/null+++ b/packages/cli/src/recovery-contract/recovery-completes-blackbox.test.ts@@ -0,0 +1,172 @@+/**+ * CONTRACT POINT 3 — when an attempt succeeds mid-ladder, the request completes+ * normally.+ *+ * REAL TIME, deliberately (trap T-10): a fake clock proves the policy and hides+ * the mechanism. A body that was consumed by attempt 1 and replayed empty, a+ * credential dropped on the retry, a timer that fires after the response was+ * already written — none of those are visible without real sockets and real+ * timers.+ *+ * Trap T-3 is the one this file exists for: every "it retried at 5s/10s/30s"+ * assertion in the suite passes on a feature that can never actually finish.+ */++import { afterAll, beforeAll, describe, expect, test } from "bun:test";+import {+ type ProxyHandle,+ type RawFixture,+ endpointsConfig,+ envSnapshot,+ postMessage,+ reservePorts,+ startProxy,+ startRawFixture,+} from "./fixtures.js";++const API_TIMEOUT_MS = 60_000; // ⇒ 30s deadline ⇒ attempts at 0s, 5s, 15s++let restoreEnv: () => void;+let proxy: ProxyHandle;+const fixtures: RawFixture[] = [];++/** One endpoint per scenario: an episode is per-endpoint and outlives its request. */+const SCENARIOS = [+ "recovers", // resets attempt 1, serves attempt 2+ "body", // same, with a large distinctive body+ "stream", // same, client asked for stream:true+ "never", // never serves — the exhaustion arm+] as const;++const ports: Record<string, number> = {};+const fixtureOf: Record<string, RawFixture> = {};++beforeAll(async () => {+ restoreEnv = envSnapshot(["API_TIMEOUT_MS", "CLAUDISH_RECOVERY"]);+ process.env.API_TIMEOUT_MS = String(API_TIMEOUT_MS);+ delete process.env.CLAUDISH_RECOVERY;++ const reserved = reservePorts(SCENARIOS.length);+ SCENARIOS.forEach((name, i) => {+ ports[name] = reserved[i];+ const fx = startRawFixture(reserved[i], {+ resetWhen: "afterBody",+ serveFromAttempt: name === "never" ? Number.POSITIVE_INFINITY : 2,+ sseText: `RECOVERED-ANSWER-${name.toUpperCase()}`,+ });+ fixtures.push(fx);+ fixtureOf[name] = fx;+ });++ proxy = await startProxy(+ endpointsConfig(+ SCENARIOS.map((name) => ({ name: `ep-${name}`, port: ports[name], model: `m-${name}` }))+ )+ );+}, 60_000);++afterAll(async () => {+ await proxy?.shutdown();+ for (const f of fixtures) f.stop();+ restoreEnv?.();+});++describe("CP-3 — recovery completes the turn", () => {+ test("the upstream comes back mid-ladder and the client gets 200 with the real answer", async () => {+ const fixture = fixtureOf.recovers;+ const result = await postMessage(proxy.url, { model: "ep-recovers@m-recovers" });++ expect(result.status).toBe(200);+ expect(result.body).toContain("RECOVERED-ANSWER-RECOVERS");+ // It really did fail first: ≥2 upstream connections, and the answer+ // arrived no earlier than the second rung.+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);+ expect(result.elapsedMs).toBeGreaterThan(4_000);++ const parsed = JSON.parse(result.body);+ expect(parsed.type).toBe("message");+ expect(parsed.content[0].text).toContain("RECOVERED-ANSWER-RECOVERS");+ }, 60_000);++ test("the request BODY is replayed intact on the successful attempt", async () => {+ // ~200 KB of distinctive payload plus a tool_result block. A body consumed+ // by attempt 1 and replayed empty satisfies every timing assertion in this+ // suite and delivers garbage to the model.+ const filler = "REPLAY-MARKER-".repeat(14_000); // ≈ 196 KB+ const content = [+ { type: "text", text: `head ${filler} tail` },+ {+ type: "tool_result",+ tool_use_id: "toolu_blackbox_1",+ content: "TOOL-RESULT-SENTINEL",+ },+ ];++ const fixture = fixtureOf.body;+ const result = await postMessage(proxy.url, {+ model: "ep-body@m-body",+ content,+ });++ expect(result.status).toBe(200);+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);++ const first = fixture.connections[0].body;+ const success = fixture.connections[fixture.connections.length - 1].body;++ expect(first.length).toBeGreaterThan(150_000);+ expect(success.length).toBeGreaterThan(150_000);+ expect(success).toContain("TOOL-RESULT-SENTINEL");+ expect(success.split("REPLAY-MARKER-").length - 1).toBe(14_000);+ // The retry must send the SAME request, not a truncated or re-encoded one.+ expect(success).toBe(first);+ }, 90_000);++ test("the credential travels with the retry", async () => {+ // A retry that drops the Authorization header turns an outage into a 401+ // storm the moment the network returns.+ const fixture = fixtureOf.body;+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);+ const first = fixture.connections[0].headers.authorization;+ const success = fixture.connections[fixture.connections.length - 1].headers.authorization;+ expect(first).toBe("Bearer test-key-ep-body");+ expect(success).toBe(first);+ }, 20_000);++ test("a recovered streaming turn is a well-formed Claude SSE stream", async () => {+ const fixture = fixtureOf.stream;+ const result = await postMessage(proxy.url, {+ model: "ep-stream@m-stream",+ stream: true,+ });++ expect(result.status).toBe(200);+ expect(result.headers.get("content-type")).toContain("text/event-stream");+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);++ const events = result.body+ .split("\n")+ .filter((l) => l.startsWith("event: "))+ .map((l) => l.slice(7).trim());++ expect(events.filter((e) => e === "message_start").length).toBe(1);+ expect(events).toContain("message_stop");+ expect(result.body).toContain("RECOVERED-ANSWER-STREAM");+ // Nothing may precede message_start that a strict SSE client would choke+ // on: the first event of the stream is the message_start itself.+ expect(events[0]).toBe("message_start");+ }, 60_000);++ test("an upstream that never returns produces exactly one exhaustion answer", async () => {+ const fixture = fixtureOf.never;+ const result = await postMessage(proxy.url, { model: "ep-never@m-never" });++ expect(result.status).toBeGreaterThanOrEqual(400);+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);++ const attemptsAtAnswer = fixture.connections.length;+ // No late attempt and no second response after the client was answered.+ await new Promise((r) => setTimeout(r, 3_000));+ expect(fixture.connections.length).toBe(attemptsAtAnswer);+ }, 90_000);+});diff --git a/packages/cli/src/recovery-contract/recovery-deadline-blackbox.test.ts b/packages/cli/src/recovery-contract/recovery-deadline-blackbox.test.tsnew file mode 100644index 00000000..e5eac1b6--- /dev/null+++ b/packages/cli/src/recovery-contract/recovery-deadline-blackbox.test.ts@@ -0,0 +1,160 @@+/**+ * CONTRACT POINT 2 — the recovery deadline is DERIVED from `API_TIMEOUT_MS`,+ * never a constant that happens to equal the default.+ *+ * `min(API_TIMEOUT_MS, 300s) − 30s` (docs/advanced/environment.md). The+ * expected attempt schedule is COMPUTED from that formula and the published+ * ladder (FR-2, 5/10/30/60…), so no attempt count and no handoff time is+ * hardcoded anywhere in this file.+ *+ * Two traps live here, and both need more than one value in ONE process:+ * T-4 a deadline that is a constant equal to the default passes any+ * single-value run;+ * T-5 an env var read once at module load passes whichever value ran first.+ * So this file runs 40s → 60s → 40s and asserts each on its own merits; a+ * module-load snapshot yields the same schedule three times and fails twice.+ */++import { afterEach, describe, expect, test } from "bun:test";+import {+ type ProxyHandle,+ createRawFixturePool,+ derivedDeadlineMs,+ endpointsConfig,+ envSnapshot,+ postMessage,+ scheduledAttemptOffsets,+ sleep,+ startProxy,+} from "./fixtures.js";++let restoreEnv: (() => void) | undefined;+let liveProxy: ProxyHandle | undefined;+let livePool: ReturnType<typeof createRawFixturePool> | undefined;++afterEach(async () => {+ await liveProxy?.shutdown();+ liveProxy = undefined;+ livePool?.stopAll();+ livePool = undefined;+ restoreEnv?.();+ restoreEnv = undefined;+});++interface LadderRun {+ offsets: number[];+ gaps: number[];+ status: number;+ elapsedMs: number;+ body: string;+}++/**+ * Run one full ladder to exhaustion under `apiTimeoutMs`.+ *+ * A fresh proxy per value, so the test does not accidentally pin WHEN the+ * implementation is allowed to read the variable — only that the value it+ * ends up using is the one the environment stated.+ */+async function runLadderUnder(apiTimeoutMs: number): Promise<LadderRun> {+ restoreEnv = envSnapshot(["API_TIMEOUT_MS", "CLAUDISH_RECOVERY"]);+ process.env.API_TIMEOUT_MS = String(apiTimeoutMs);+ delete process.env.CLAUDISH_RECOVERY;++ const pool = createRawFixturePool(1, "ep-deadline");+ livePool = pool;+ const proxy = await startProxy(endpointsConfig(pool.specs));+ liveProxy = proxy;++ const { model, fixture } = pool.next();+ const result = await postMessage(proxy.url, { model });+ return {+ offsets: fixture.offsetsFromFirst(),+ gaps: fixture.gaps(),+ status: result.status,+ elapsedMs: result.elapsedMs,+ body: result.body,+ };+}++function expectSchedule(run: LadderRun, apiTimeoutMs: number): void {+ const deadline = derivedDeadlineMs(apiTimeoutMs);+ const expected = scheduledAttemptOffsets(deadline);++ expect(run.offsets.length).toBe(expected.length);+ for (const [i, want] of expected.entries()) {+ expect(Math.abs(run.offsets[i] - want)).toBeLessThan(1_500);+ }+ // The published ladder, read off the wire: 5s then 10s then 30s …+ for (const [i, gap] of run.gaps.entries()) {+ const want = expected[i + 1] - expected[i];+ expect(Math.abs(gap - want)).toBeLessThan(1_000);+ }+ // Nothing may start after the derived deadline, and the client must be+ // answered inside it (measured from inbound request start).+ expect(Math.max(...run.offsets)).toBeLessThan(deadline);+ expect(run.elapsedMs).toBeLessThan(deadline + 5_000);+}++describe("CP-2 — the deadline derives from API_TIMEOUT_MS", () => {+ test("API_TIMEOUT_MS=40000 ⇒ the schedule the derived 10s deadline admits", async () => {+ const run = await runLadderUnder(40_000);+ expectSchedule(run, 40_000);+ expect(run.status).toBe(400);+ }, 60_000);++ test("API_TIMEOUT_MS=60000 ⇒ a LONGER schedule in the same process (anti-hardcode)", async () => {+ const run = await runLadderUnder(60_000);+ expectSchedule(run, 60_000);+ // The point of the pairing: this run must contain strictly more attempts+ // than the 40s run above. A constant 270s deadline never gets here (the+ // per-test timeout kills it); a constant anything gives both runs the+ // same count.+ expect(run.offsets.length).toBeGreaterThan(+ scheduledAttemptOffsets(derivedDeadlineMs(40_000)).length+ );+ }, 90_000);++ test("back to API_TIMEOUT_MS=40000 in the same process ⇒ the short schedule returns", async () => {+ // T-5: an implementation that snapshots the env at module load passes the+ // first of these three and fails whichever ran second and third.+ const run = await runLadderUnder(40_000);+ expectSchedule(run, 40_000);+ }, 60_000);++ test("a garbled API_TIMEOUT_MS falls back to a working deadline rather than NaN", async () => {+ restoreEnv = envSnapshot(["API_TIMEOUT_MS", "CLAUDISH_RECOVERY"]);+ process.env.API_TIMEOUT_MS = "abc";+ delete process.env.CLAUDISH_RECOVERY;++ const pool = createRawFixturePool(1, "ep-garbled");+ livePool = pool;+ const proxy = await startProxy(endpointsConfig(pool.specs));+ liveProxy = proxy;++ const { model, fixture } = pool.next();+ const controller = new AbortController();+ const pending = postMessage(proxy.url, { model, signal: controller.signal }).catch(() => null);++ // A NaN deadline collapses to "no budget" and answers instantly; a+ // negative one does the same. Either way the second rung never arrives.+ const started = Date.now();+ while (Date.now() - started < 12_000 && fixture.connections.length < 2) await sleep(50);+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);+ expect(Math.abs(fixture.gaps()[0] - 5_000)).toBeLessThan(1_000);+ controller.abort();+ await pending;+ }, 40_000);++ test("an API_TIMEOUT_MS below one rung still answers, bounded, without crashing", async () => {+ // A-1: the contract states no floor for `min(API_TIMEOUT_MS,300s)−30s`,+ // so the only thing assertable here is that a sub-rung deadline neither+ // hangs nor produces a negative budget. The floor itself stays untestable+ // until it is pinned.+ const run = await runLadderUnder(31_000);+ expect(run.status).toBe(400);+ expect(JSON.parse(run.body).error.type).toBe("connection_error");+ expect(run.offsets.length).toBeGreaterThanOrEqual(1);+ expect(run.elapsedMs).toBeLessThan(30_000);+ }, 60_000);+});diff --git a/packages/cli/src/recovery-contract/recovery-disconnect-blackbox.test.ts b/packages/cli/src/recovery-contract/recovery-disconnect-blackbox.test.tsnew file mode 100644index 00000000..dda910e1--- /dev/null+++ b/packages/cli/src/recovery-contract/recovery-disconnect-blackbox.test.ts@@ -0,0 +1,147 @@+/**+ * CONTRACT POINT 7 — a client disconnect stops the retry promptly.+ *+ * REAL TIME ONLY. A fake clock cannot see a timer that was never cancelled,+ * because the timer only fires when time really passes.+ *+ * Trap T-11 is the one this file is built around: aborting BEFORE any attempt+ * has happened makes "it stopped retrying" trivially true. So the abort here+ * happens only after the fixture has observed ≥2 attempts, and it lands inside+ * a KNOWN wait rung — the third attempt is due at a time we can name, and the+ * assertion is that it never arrives.+ */++import { afterAll, beforeAll, describe, expect, test } from "bun:test";+import {+ type FixturePool,+ type ProxyHandle,+ type RawFixture,+ createRawFixturePool,+ endpointsConfig,+ envSnapshot,+ postMessage,+ sleep,+ startProxy,+} from "./fixtures.js";++const API_TIMEOUT_MS = 60_000; // ⇒ 30s deadline ⇒ attempts at 0s, 5s, 15s++let restoreEnv: () => void;+let proxy: ProxyHandle;+let pool: FixturePool;++beforeAll(async () => {+ restoreEnv = envSnapshot(["API_TIMEOUT_MS", "CLAUDISH_RECOVERY"]);+ process.env.API_TIMEOUT_MS = String(API_TIMEOUT_MS);+ delete process.env.CLAUDISH_RECOVERY;++ pool = createRawFixturePool(6, "ep-abort");+ proxy = await startProxy(endpointsConfig(pool.specs));+}, 60_000);++afterAll(async () => {+ await proxy?.shutdown();+ pool?.stopAll();+ restoreEnv?.();+});++async function waitForAttempts(fixture: RawFixture, n: number, budgetMs: number): Promise<void> {+ const started = Date.now();+ while (Date.now() - started < budgetMs) {+ if (fixture.connections.length >= n) return;+ await sleep(25);+ }+ throw new Error(+ `fixture saw only ${fixture.connections.length} attempts in ${budgetMs}ms; expected ${n}`+ );+}++describe("CP-7 — client disconnect stops the ladder", () => {+ test("no further upstream attempt arrives after the client goes away mid-rung", async () => {+ const { model, fixture } = pool.next();+ const controller = new AbortController();+ const pending = postMessage(proxy.url, { model, signal: controller.signal }).catch(+ (e: unknown) => e+ );++ // Let the ladder demonstrably run: attempts at t≈0 and t≈5s.+ await waitForAttempts(fixture, 2, 12_000);+ const attemptsAtAbort = fixture.connections.length;+ expect(attemptsAtAbort).toBeGreaterThanOrEqual(2);++ // Abort INSIDE the 10s wait rung. The next attempt is due at t≈15s.+ await sleep(1_500);+ controller.abort();+ await pending;++ // Watch well past the moment attempt 3 was due.+ await sleep(12_000);+ expect(fixture.connections.length).toBe(attemptsAtAbort);+ }, 60_000);++ test("NEGATIVE CONTROL: with nobody aborting, the third attempt does arrive", async () => {+ // Without this, the assertion above ("no new connection after the abort")+ // is satisfied by any build that simply stops after two attempts — the+ // abort would be doing none of the work. Same fixture shape, same rung,+ // the only difference is that nothing aborts.+ const { model, fixture } = pool.next();+ const controller = new AbortController();+ const pending = postMessage(proxy.url, { model, signal: controller.signal }).catch(+ (e: unknown) => e+ );+ await waitForAttempts(fixture, 3, 25_000);+ expect(fixture.connections.length).toBeGreaterThanOrEqual(3);+ controller.abort();+ await pending;+ }, 45_000);++ test("aborting one request does not stop another request's ladder", async () => {+ // Guards a single shared coordinator: A's abort must not cancel B.+ const a = pool.next();+ const b = pool.next();++ const controllerA = new AbortController();+ const pendingA = postMessage(proxy.url, {+ model: a.model,+ signal: controllerA.signal,+ }).catch((e: unknown) => e);+ const pendingB = postMessage(proxy.url, { model: b.model });++ await waitForAttempts(a.fixture, 2, 12_000);+ await waitForAttempts(b.fixture, 2, 12_000);++ const aAtAbort = a.fixture.connections.length;+ controllerA.abort();+ await pendingA;++ const resultB = await pendingB;+ expect(resultB.status).toBeGreaterThanOrEqual(400);+ // B ran its full schedule …+ expect(b.fixture.connections.length).toBeGreaterThanOrEqual(3);+ // … while A stopped where it was.+ expect(a.fixture.connections.length).toBe(aAtAbort);+ }, 90_000);++ test("shutdown after an aborted ladder resolves promptly", async () => {+ // NFR-2: a live 60s timer keeps the loop alive and is invisible from+ // inside the request path.+ const ownPool = createRawFixturePool(1, "ep-shutdown");+ const ownProxy = await startProxy(endpointsConfig(ownPool.specs));+ try {+ const { model, fixture } = ownPool.next();+ const controller = new AbortController();+ const pending = postMessage(ownProxy.url, { model, signal: controller.signal }).catch(+ (e: unknown) => e+ );+ await waitForAttempts(fixture, 2, 12_000);+ controller.abort();+ await pending;++ const t0 = Date.now();+ await ownProxy.shutdown();+ expect(Date.now() - t0).toBeLessThan(1_000);+ } finally {+ ownPool.stopAll();+ }+ }, 60_000);+});diff --git a/packages/cli/src/recovery-contract/recovery-exhaustion-blackbox.test.ts b/packages/cli/src/recovery-contract/recovery-exhaustion-blackbox.test.tsnew file mode 100644index 00000000..5c883b30--- /dev/null+++ b/packages/cli/src/recovery-contract/recovery-exhaustion-blackbox.test.ts@@ -0,0 +1,140 @@+/**+ * CONTRACT POINT 4 — at exhaustion the client gets 503 when a recovery banner+ * is confirmed on screen, and 400 when it is not.+ *+ * ONLY THE NEGATIVE ARM IS HERE, and that is a finding rather than an omission:+ * "a banner is confirmed on screen" has no published way to be SET from outside+ * the proxy independently of the switch that also enables the ladder, so the+ * 503 arm and the 400 arm cannot both be produced in one experiment (trap+ * T-13). See test-results.md — it needs a documented pane-attach seam or a PTY+ * end-to-end run.+ *+ * Every status below is read from `Response.status` of a fetch against the+ * proxy's own URL. This project's documented failure mode is that a terminal+ * error is remapped to 400 on the way out, so a status asserted anywhere+ * upstream of the socket is measuring a different thing than the user gets+ * (trap T-6).+ */++import { afterAll, beforeAll, describe, expect, test } from "bun:test";+import {+ type FixturePool,+ type ProxyHandle,+ createRawFixturePool,+ endpointsConfig,+ envSnapshot,+ postMessage,+ reservePorts,+ startProxy,+ startRawFixture,+ startStatusFixture,+} from "./fixtures.js";++const API_TIMEOUT_MS = 40_000; // ⇒ 10s deadline ⇒ attempts at 0s, 5s++let restoreEnv: () => void;+let proxy: ProxyHandle;+let pool: FixturePool;++beforeAll(async () => {+ restoreEnv = envSnapshot(["API_TIMEOUT_MS", "CLAUDISH_RECOVERY", "CLAUDISH_RECOVERY_UI"]);+ process.env.API_TIMEOUT_MS = String(API_TIMEOUT_MS);+ delete process.env.CLAUDISH_RECOVERY;+ delete process.env.CLAUDISH_RECOVERY_UI; // UI on by default; no pane can attach here++ pool = createRawFixturePool(3, "ep-exhaust");+ proxy = await startProxy(endpointsConfig(pool.specs));+}, 60_000);++afterAll(async () => {+ await proxy?.shutdown();+ pool?.stopAll();+ restoreEnv?.();+});++describe("CP-4 — exhaustion with no banner on screen", () => {+ test("answers 400 connection_error naming provider, host and reason, with no recovery headers", async () => {+ const { model, fixture, name } = pool.next();+ const result = await postMessage(proxy.url, { model });++ // The ladder really ran first (otherwise this is just the off-switch).+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);++ expect(result.status).toBe(400);+ const err = JSON.parse(result.body).error;+ expect(err.type).toBe("connection_error");+ // provider, host and the reason, in the words the product already uses.+ expect(err.message.toLowerCase()).toContain(name.toLowerCase());+ expect(err.message).toContain(`127.0.0.1:${fixture.port}`);+ expect(err.message.length).toBeGreaterThan(20);++ // A 503 handed back for retry carries these; a 400 must not.+ expect(result.headers.get("x-claudish-recovery")).toBeNull();+ expect(result.headers.get("x-should-retry")).toBeNull();+ }, 60_000);++ test("the streaming path leaves the client something it can act on", async () => {+ // A-6: with headers already committed a 503 is unsendable, so the+ // contract is satisfiable two ways. Both are acceptable; a half-written+ // stream with no terminal error event is not — that is the shape that+ // silently truncates a turn.+ const { model, fixture } = pool.next();+ const result = await postMessage(proxy.url, { model, stream: true });+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);++ const actionable =+ result.status >= 400 ||+ (result.status === 200 && /event:\s*error/.test(result.body)) ||+ /"type"\s*:\s*"error"/.test(result.body);+ expect(+ actionable,+ `streaming exhaustion produced status ${result.status} with body ${result.body.slice(0, 300)}`+ ).toBe(true);+ }, 60_000);++ test("a recovery exhaustion survives an earlier candidate's auth failure in the chain", async () => {+ // C-14: an earlier candidate's non-connection failure must not turn the+ // recovery outcome into a combined terminal error that hides the reason+ // — the client must still be told the network is down, not that it is+ // unauthenticated.+ const [authPort] = reservePorts(1);+ const auth401 = startStatusFixture(401, authPort);+ const [deadPort] = reservePorts(1);+ const dead = startRawFixture(deadPort);+ let chainProxy: ProxyHandle | undefined;+ try {+ chainProxy = await startProxy(+ endpointsConfig([+ { name: "ep-auth", port: authPort, model: "m-auth" },+ { name: "ep-down", port: deadPort, model: "m-down" },+ ]),+ {+ model: "ep-auth@m-auth",+ modelChain: ["ep-auth@m-auth", "ep-down@m-down"],+ }+ );++ const result = await postMessage(chainProxy.url, { model: "ep-auth@m-auth" });++ expect(auth401.hits).toBeGreaterThanOrEqual(1);+ expect(dead.connections.length).toBeGreaterThanOrEqual(2); // candidate 2 laddered+ const err = JSON.parse(result.body).error;+ // The contract pins the STATUS for this case (C-14), not the error+ // type, so the type is asserted only negatively: the user must not be+ // told they have an auth problem while the network is down.+ expect(result.status).not.toBe(401);+ expect(result.status).toBeGreaterThanOrEqual(400);+ expect(err.type).not.toBe("authentication_error");+ // The network reason survives into the body the client can read. On+ // the chain path it lands in `error.attempts[]` rather than+ // `error.message`, so assert against the body the client receives+ // rather than against one field of it.+ expect(result.body).toContain("Cannot reach");+ expect(result.body).toContain(`127.0.0.1:${dead.port}`);+ } finally {+ await chainProxy?.shutdown();+ auth401.stop();+ dead.stop();+ }+ }, 90_000);+});diff --git a/packages/cli/src/recovery-contract/recovery-ladder-blackbox.test.ts b/packages/cli/src/recovery-contract/recovery-ladder-blackbox.test.tsnew file mode 100644index 00000000..d372d72c--- /dev/null+++ b/packages/cli/src/recovery-contract/recovery-ladder-blackbox.test.ts@@ -0,0 +1,285 @@+/**+ * CONTRACT POINT 1 (a connection failure no longer answers immediately),+ * CONTRACT POINT 5 (`x-claudish-no-recovery` skips the ladder) and the+ * env-var arm of CONTRACT POINT 8.+ *+ * BLACK BOX. Every attempt is counted at an upstream socket we own; every+ * status is read from `Response.status` of a fetch against the proxy's own URL,+ * i.e. AFTER any internal status remap.+ *+ * Trap T-7 is the reason this file is shaped the way it is: an "immediate 400"+ * assertion passes just as well on a build where the whole feature is off. So+ * every fast-fail scenario here is PAIRED with an enabled control run against+ * an identical fixture in the same process, and the control asserts a second+ * upstream CONNECTION, not merely a slower answer.+ *+ * Every arm takes its own endpoint from the pool: an episode is per-endpoint+ * and outlives the request that opened it, so a shared endpoint would let the+ * previous arm's rung explain the next arm's result.+ */++import { afterAll, beforeAll, describe, expect, test } from "bun:test";+import {+ type FixturePool,+ type ProxyHandle,+ assertRefused,+ createRawFixturePool,+ endpointsConfig,+ envSnapshot,+ postMessage,+ reservePorts,+ scheduledAttemptOffsets,+ sleep,+ startHealthyFixture,+ startProxy,+ startStatusFixture,+} from "./fixtures.js";++const API_TIMEOUT_MS = 60_000; // ⇒ derived deadline 30s ⇒ attempts at 0s, 5s, 15s+const DEADLINE_MS = 30_000;++let restoreEnv: () => void;+let proxy: ProxyHandle;+let pool: FixturePool;+let healthy: ReturnType<typeof startHealthyFixture>;+let upstream500: ReturnType<typeof startStatusFixture>;+let deadPort: number;++beforeAll(async () => {+ restoreEnv = envSnapshot(["API_TIMEOUT_MS", "CLAUDISH_RECOVERY", "CLAUDISH_RECOVERY_UI"]);+ process.env.API_TIMEOUT_MS = String(API_TIMEOUT_MS);+ delete process.env.CLAUDISH_RECOVERY;++ const [healthyPort, errPort, refusedPort] = reservePorts(3);+ deadPort = refusedPort;++ pool = createRawFixturePool(8, "ep-rst");+ healthy = startHealthyFixture("HEALTHY-FIXTURE-ANSWER", healthyPort);+ upstream500 = startStatusFixture(500, errPort);++ proxy = await startProxy(+ endpointsConfig([+ ...pool.specs,+ { name: "ep-live", port: healthyPort, model: "m-live" },+ { name: "ep-500", port: errPort, model: "m-500" },+ { name: "ep-refused", port: refusedPort, model: "m-refused" },+ ])+ );++ // Warm the proxy (catalog/registry latching) so the "<2s" negatives measure+ // the request path rather than one-time startup work.+ const warm = await postMessage(proxy.url, { model: "ep-live@m-live" });+ expect(warm.status).toBe(200);+}, 60_000);++afterAll(async () => {+ await proxy?.shutdown();+ pool?.stopAll();+ healthy?.stop();+ upstream500?.stop();+ restoreEnv?.();+});++/**+ * Issue a request against a fresh endpoint and resolve as soon as the fixture+ * has seen `n` connections, then abort. An "is the ladder really on?" control+ * therefore costs one rung rather than a whole deadline.+ */+async function ladderReaches(+ n: number,+ headers?: Record<string, string>,+ budgetMs = 20_000+): Promise<{ connections: number; answeredBeforeNthAttempt: boolean }> {+ const { model, fixture } = pool.next();+ const controller = new AbortController();+ let settled = false;+ const pending = postMessage(proxy.url, { model, headers, signal: controller.signal })+ .then(() => {+ settled = true;+ })+ .catch(() => {+ settled = true;+ });++ const started = Date.now();+ while (Date.now() - started < budgetMs) {+ if (fixture.connections.length >= n || settled) break;+ await sleep(50);+ }+ const connections = fixture.connections.length;+ const answeredBeforeNthAttempt = settled && connections < n;+ controller.abort();+ await pending;+ return { connections, answeredBeforeNthAttempt };+}++describe("CP-1 — a connection failure enters a retry ladder instead of answering", () => {+ test("the upstream socket records the published ladder, truncated by the derived deadline", async () => {+ const { model, fixture } = pool.next();+ let answered = false;+ const request = postMessage(proxy.url, { model }).then((r) => {+ answered = true;+ return r;+ });++ // CP1-01: nothing has been answered while the first rung is still waiting.+ await sleep(2_000);+ expect(answered).toBe(false);++ const result = await request;++ // Counted at the socket, never from a [Recovery] line (trap T-9).+ const expectedOffsets = scheduledAttemptOffsets(DEADLINE_MS); // [0, 5000, 15000]+ expect(fixture.connections.length).toBe(expectedOffsets.length);++ const offsets = fixture.offsetsFromFirst();+ for (const [i, expected] of expectedOffsets.entries()) {+ expect(Math.abs(offsets[i] - expected)).toBeLessThan(1_500);+ }++ // No attempt may start after the derived deadline.+ expect(Math.max(...offsets)).toBeLessThan(DEADLINE_MS);++ // The answer is read at the proxy's public URL, after any remap (T-6).+ expect(result.status).not.toBe(200);+ expect(result.elapsedMs).toBeGreaterThan(15_000);+ }, 60_000);++ test("a client re-ask during the same outage is still retried, not answered instantly", async () => {+ // The shipped design hands the retry back to the client at the deadline+ // and the client re-enters the SAME episode (requirements, AMENDMENTS+ // table, FR-3/FR-4). A re-ask that fast-fails would end the recovery the+ // amendment promises to continue across requests.+ const { model, fixture } = pool.next();+ const first = await postMessage(proxy.url, { model });+ expect(first.status).not.toBe(200);+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);++ fixture.reset();+ const reask = await postMessage(proxy.url, { model });+ expect(reask.status).not.toBe(200);+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);+ }, 90_000);++ test("NEGATIVE: a reachable upstream that answers an error is answered at once, with no ladder", async () => {+ const before = upstream500.hits;+ const result = await postMessage(proxy.url, { model: "ep-500@m-500" });+ expect(result.elapsedMs).toBeLessThan(2_000);+ expect(upstream500.hits).toBe(before + 1);+ expect(result.status).toBeGreaterThanOrEqual(400);+ }, 20_000);++ test("NEGATIVE: a healthy upstream answers 200 with the fixture's own text", async () => {+ const result = await postMessage(proxy.url, { model: "ep-live@m-live" });+ expect(result.status).toBe(200);+ expect(result.body).toContain("HEALTHY-FIXTURE-ANSWER");+ expect(result.elapsedMs).toBeLessThan(2_000);+ }, 20_000);++ test("NEGATIVE: /v1/models and /v1/messages/count_tokens never enter the ladder", async () => {+ await assertRefused(deadPort);++ const t0 = Date.now();+ const models = await fetch(`${proxy.url}/v1/models`);+ await models.text();+ expect(models.status).toBe(200);+ expect(Date.now() - t0).toBeLessThan(2_000);++ const t1 = Date.now();+ const count = await fetch(`${proxy.url}/v1/messages/count_tokens`, {+ method: "POST",+ headers: { "content-type": "application/json" },+ body: JSON.stringify({+ model: "ep-refused@m-refused",+ max_tokens: 16,+ messages: [{ role: "user", content: "hello" }],+ }),+ });+ await count.text();+ expect(Date.now() - t1).toBeLessThan(2_000);+ }, 20_000);+});++describe("CP-5 — x-claudish-no-recovery: 1 skips the ladder", () => {+ test("with the header: one upstream attempt, a connection_error in under a second", async () => {+ const { model, fixture } = pool.next();+ const result = await postMessage(proxy.url, {+ model,+ headers: { "x-claudish-no-recovery": "1" },+ });++ expect(result.elapsedMs).toBeLessThan(1_000);+ expect(result.status).toBe(400);+ expect(JSON.parse(result.body).error.type).toBe("connection_error");+ // The decisive half: the ladder never ran at the socket.+ expect(fixture.connections.length).toBe(1);+ }, 20_000);++ test("PAIRED CONTROL: identical fixture, same process, header removed ⇒ a second attempt arrives", async () => {+ const control = await ladderReaches(2);+ expect(control.answeredBeforeNthAttempt).toBe(false);+ expect(control.connections).toBeGreaterThanOrEqual(2);+ }, 30_000);++ test("the header is honoured case-insensitively", async () => {+ const { model, fixture } = pool.next();+ const result = await postMessage(proxy.url, {+ model,+ headers: { "X-Claudish-No-Recovery": "1" },+ });+ expect(result.elapsedMs).toBeLessThan(1_000);+ expect(result.status).toBe(400);+ expect(fixture.connections.length).toBe(1);+ }, 20_000);++ test("the skip also holds on the streaming path", async () => {+ const { model, fixture } = pool.next();+ const result = await postMessage(proxy.url, {+ model,+ stream: true,+ headers: { "x-claudish-no-recovery": "1" },+ });+ expect(result.elapsedMs).toBeLessThan(2_000);+ expect(fixture.connections.length).toBe(1);+ }, 20_000);+});++describe("CP-8 — the environment switch, with its paired enabled control", () => {+ test("CLAUDISH_RECOVERY=0 ⇒ one attempt and an immediate error", async () => {+ const { model, fixture } = pool.next();+ process.env.CLAUDISH_RECOVERY = "0";+ try {+ const result = await postMessage(proxy.url, { model });+ expect(result.elapsedMs).toBeLessThan(2_000);+ expect(result.status).toBe(400);+ expect(JSON.parse(result.body).error.type).toBe("connection_error");+ expect(fixture.connections.length).toBe(1);+ } finally {+ delete process.env.CLAUDISH_RECOVERY;+ }+ }, 20_000);++ test("PAIRED CONTROL: the switch removed, same process ⇒ the ladder runs again", async () => {+ delete process.env.CLAUDISH_RECOVERY;+ const control = await ladderReaches(2);+ expect(control.answeredBeforeNthAttempt).toBe(false);+ expect(control.connections).toBeGreaterThanOrEqual(2);+ }, 30_000);++ test("the switch does not change the healthy path's answer", async () => {+ process.env.CLAUDISH_RECOVERY = "0";+ let off: Awaited<ReturnType<typeof postMessage>>;+ try {+ off = await postMessage(proxy.url, { model: "ep-live@m-live" });+ } finally {+ delete process.env.CLAUDISH_RECOVERY;+ }+ expect(off.status).toBe(200);+ expect(off.body).toContain("HEALTHY-FIXTURE-ANSWER");++ const on = await postMessage(proxy.url, { model: "ep-live@m-live" });+ expect(on.status).toBe(200);+ expect(on.body).toContain("HEALTHY-FIXTURE-ANSWER");+ expect(JSON.parse(on.body).content).toEqual(JSON.parse(off.body).content);+ }, 20_000);+});diff --git a/packages/cli/src/recovery-contract/recovery-process-blackbox.test.ts b/packages/cli/src/recovery-contract/recovery-process-blackbox.test.tsnew file mode 100644index 00000000..1c1e37b2--- /dev/null+++ b/packages/cli/src/recovery-contract/recovery-process-blackbox.test.ts@@ -0,0 +1,169 @@+/**+ * CONTRACT POINT 10 (`[Recovery]` log lines, with no `--debug` anywhere) and+ * the process-level half of CONTRACT POINT 7 (stop promptly, and actually+ * exit).+ *+ * A SPAWNED CHILD with separated pipes, deliberately. An in-process+ * `process.stderr.write` monkeypatch intercepts the CALL, not the file+ * descriptor, so it cannot answer "did this byte go to stdout?" — which is the+ * only question that matters when the client on the other end is a TUI. And a+ * process that never exits is invisible to a test running inside it.+ */++import { describe, expect, test } from "bun:test";+import { reservePorts, sleep, startRawFixture } from "./fixtures.js";++const CHILD = new URL("./child-proxy-run.ts", import.meta.url).pathname;+const SECRET_KEY = "sk-blackbox-SECRET-VALUE-9f3c1d"; // the key the child configures++interface ChildRun {+ stdout: string;+ stderr: string;+ exitCode: number | null;+ lifetimeMs: number;+ result: Record<string, unknown>;+}++async function runChild(+ mode: "exhaust" | "abort",+ endpointUrl: string,+ apiTimeoutMs: number,+ abortAfterMs?: number+): Promise<ChildRun> {+ const started = Date.now();+ const child = Bun.spawn(+ [+ "bun",+ "run",+ CHILD,+ mode,+ endpointUrl,+ String(apiTimeoutMs),+ ...(abortAfterMs ? [String(abortAfterMs)] : []),+ ],+ { stdout: "pipe", stderr: "pipe", stdin: "ignore" }+ );+ const [stdout, stderr] = await Promise.all([+ new Response(child.stdout).text(),+ new Response(child.stderr).text(),+ ]);+ const exitCode = await child.exited;+ const lifetimeMs = Date.now() - started;+ const line = stdout.trim().split("\n").at(-1) ?? "{}";+ let result: Record<string, unknown> = {};+ try {+ result = JSON.parse(line);+ } catch {+ /* asserted by the caller */+ }+ return { stdout, stderr, exitCode, lifetimeMs, result };+}++describe("CP-10 — recovery diagnostics in a real child process", () => {+ test("a recovery episode is reported in the captured output with no --debug anywhere", async () => {+ const [port] = reservePorts(1);+ const fixture = startRawFixture(port);+ try {+ const run = await runChild("exhaust", `http://127.0.0.1:${port}/v1`, 40_000);+ // The ladder really ran: counted at our socket, not in the log.+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);+ expect(run.result.status).toBe(400);++ // Contract point 10: the retry activity is legible without a debug flag.+ // The docs make the same promise for a shortened API_TIMEOUT_MS:+ // "claudish shortens the hold to match and logs a [Recovery] line+ // saying so" (docs/advanced/environment.md).+ //+ // MEASURED at HEAD 8ed0f13: the lines exist and are rich, but they go+ // to the DEBUG LOG FILE (logs/claudish_*.log, which only a --debug run+ // writes). With no debug flag a whole recovery episode leaves nothing+ // on either stream but the final `[claudish] Error:` line.+ expect(+ run.stderr,+ `neither stream carried a [Recovery] line.\nstdout=${run.stdout}\nstderr=${run.stderr}`+ ).toContain("[Recovery]");+ } finally {+ fixture.stop();+ }+ }, 90_000);++ test("no diagnostic byte lands on stdout — the client's protocol stream stays clean", async () => {+ const [port] = reservePorts(1);+ const fixture = startRawFixture(port);+ try {+ const run = await runChild("exhaust", `http://127.0.0.1:${port}/v1`, 40_000);+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);++ // stdout carries the child's single JSON result line and nothing else.+ const stdoutLines = run.stdout.split("\n").filter((l) => l.trim().length > 0);+ expect(stdoutLines.length).toBe(1);+ expect(run.stdout).not.toContain("[Recovery]");+ expect(run.stdout).not.toContain("[claudish]");+ } finally {+ fixture.stop();+ }+ }, 90_000);++ test("no credential appears in anything the process printed", async () => {+ const [port] = reservePorts(1);+ const fixture = startRawFixture(port);+ try {+ const run = await runChild("exhaust", `http://127.0.0.1:${port}/v1`, 40_000);+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);+ // Positive control: the run really did print the failing endpoint, so+ // "no secret" is not passing because nothing was printed at all.+ expect(run.stderr + run.stdout).toContain(`127.0.0.1:${port}`);+ expect(run.stdout).not.toContain(SECRET_KEY);+ expect(run.stderr).not.toContain(SECRET_KEY);+ } finally {+ fixture.stop();+ }+ }, 90_000);+});++describe("CP-7 — the process itself lets go", () => {+ test("after an aborted ladder the child shuts down and exits promptly", async () => {+ const [port] = reservePorts(1);+ const fixture = startRawFixture(port);+ try {+ // 12s, not 7s: the child's FIRST upstream attempt lands ~2.5s after the+ // request starts (catalog/registry warm-up inside the request path), so+ // a 7s abort can land before attempt 2 and make "it stopped retrying"+ // trivially true — trap T-11.+ const abortAfterMs = 12_000;+ const run = await runChild("abort", `http://127.0.0.1:${port}/v1`, 120_000, abortAfterMs);++ expect(run.result.aborted).toBe(true);+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2); // it was mid-ladder+ expect(run.exitCode).toBe(0);+ expect(Number(run.result.shutdownMs)).toBeLessThan(1_000);+ // Startup + the abort delay + a generous margin. A retained 60s timer+ // blows straight through this.+ expect(run.lifetimeMs).toBeLessThan(abortAfterMs + 10_000);+ } finally {+ fixture.stop();+ }+ }, 90_000);++ test("SIGTERM mid-ladder ends the process within two seconds", async () => {+ const [port] = reservePorts(1);+ const fixture = startRawFixture(port);+ const child = Bun.spawn(+ ["bun", "run", CHILD, "exhaust", `http://127.0.0.1:${port}/v1`, "120000"],+ { stdout: "pipe", stderr: "pipe", stdin: "ignore" }+ );+ try {+ const deadline = Date.now() + 30_000;+ while (Date.now() < deadline && fixture.connections.length < 2) await sleep(50);+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);++ const killedAt = Date.now();+ child.kill("SIGTERM");+ await child.exited;+ expect(Date.now() - killedAt).toBeLessThan(2_000);+ } finally {+ child.kill("SIGKILL");+ fixture.stop();+ }+ }, 90_000);+});diff --git a/packages/cli/src/recovery-contract/recovery-slow-fault-blackbox.test.ts b/packages/cli/src/recovery-contract/recovery-slow-fault-blackbox.test.tsnew file mode 100644index 00000000..19562a93--- /dev/null+++ b/packages/cli/src/recovery-contract/recovery-slow-fault-blackbox.test.ts@@ -0,0 +1,128 @@+/**+ * The SLOW-FAULT arm. Trap T-1, made executable.+ *+ * Every loopback refusal in this suite fails in about one millisecond, and a+ * ~1ms fault is the one kind that cannot exercise anything that decays DURING+ * an attempt: a per-attempt connect cap, a budget measured from inbound request+ * start, or a liveness signal that a long silent attempt would let expire. The+ * class of the originating incident — `unreachable`/`ETIMEDOUT` — is measured+ * in tens of seconds, not milliseconds, and the validation contract's own+ * C-18 exists because a suite built entirely on refusals is blind to it.+ *+ * Fault: F-SLOWCONNECT — `192.0.2.1` (TEST-NET-1, RFC 5737, unrouted), so the+ * TCP connect burns the OS timeout instead of being refused. Local, no+ * privileges, no network manipulation.+ *+ * REAL TIME. A fake clock makes every assertion in this file vacuous.+ */++import { afterAll, beforeAll, describe, expect, test } from "bun:test";+import {+ type ProxyHandle,+ derivedDeadlineMs,+ endpointsConfig,+ envSnapshot,+ postMessage,+ sleep,+ startProxy,+} from "./fixtures.js";++const API_TIMEOUT_MS = 60_000; // ⇒ 30s derived deadline+const DEADLINE_MS = derivedDeadlineMs(API_TIMEOUT_MS);++let restoreEnv: () => void;+let proxy: ProxyHandle;++beforeAll(async () => {+ restoreEnv = envSnapshot(["API_TIMEOUT_MS", "CLAUDISH_RECOVERY"]);+ process.env.API_TIMEOUT_MS = String(API_TIMEOUT_MS);+ delete process.env.CLAUDISH_RECOVERY;++ proxy = await startProxy(+ endpointsConfig([{ name: "ep-slow", port: 0, model: "m-slow", url: "https://192.0.2.1/v1" }])+ );+}, 60_000);++afterAll(async () => {+ await proxy?.shutdown();+ restoreEnv?.();+}, 60_000);++describe("a fault that is SLOW rather than instant", () => {+ test("PRECONDITION P-3: a connect to TEST-NET-1 is still pending after 3s on this machine", async () => {+ // A captive portal or an aggressive corporate resolver answers 192.0.2.1,+ // which would silently turn the slow fault into a fast one. Fail loudly+ // with the measured duration; never skip.+ const started = Date.now();+ let outcome = "answered";+ try {+ await fetch("https://192.0.2.1/v1", { signal: AbortSignal.timeout(3_000) });+ } catch (err) {+ outcome = String(err);+ }+ const elapsed = Date.now() - started;+ expect(+ elapsed,+ `PRECONDITION P-3 FAILED: 192.0.2.1 settled in ${elapsed}ms (${outcome}). ` ++ "Something on this network answers TEST-NET-1, so the slow-connect fault is not in force."+ ).toBeGreaterThanOrEqual(2_800);+ expect(outcome).not.toBe("answered");+ }, 20_000);++ test("the request is held past the early rungs — a slow fault is not answered early", async () => {+ const controller = new AbortController();+ let answered = false;+ const request = postMessage(proxy.url, {+ model: "ep-slow@m-slow",+ signal: controller.signal,+ })+ .then(() => {+ answered = true;+ })+ .catch(() => {+ answered = true;+ });++ await sleep(12_000);+ expect(answered).toBe(false);+ controller.abort();+ await request;+ }, 40_000);++ test("the derived deadline bounds a SLOW attempt, not just a fast one", async () => {+ // Contract point 2: the budget is measured from inbound request start and+ // the client is answered inside it. An implementation that checks the+ // deadline only BETWEEN attempts satisfies every refusal-based test in+ // this suite and overruns here, because one attempt outlives the whole+ // budget. C-18 calls the missing piece PER_ATTEMPT_CONNECT_CAP_MS.+ const controller = new AbortController();+ let settledAt: number | null = null;+ const started = Date.now();+ const request = postMessage(proxy.url, {+ model: "ep-slow@m-slow",+ signal: controller.signal,+ })+ .then(() => {+ settledAt = Date.now() - started;+ })+ .catch(() => {+ settledAt = Date.now() - started;+ });++ const budget = DEADLINE_MS + 10_000;+ const watchUntil = Date.now() + budget;+ while (Date.now() < watchUntil && settledAt === null) await sleep(250);+ // Snapshot BEFORE aborting: a value recorded after the abort would be the+ // abort's own doing, not the proxy answering.+ const settledOnItsOwn = settledAt;+ controller.abort();+ await request;++ expect(+ settledOnItsOwn,+ `the client was still waiting ${budget}ms into a ${DEADLINE_MS}ms budget — ` ++ "one slow connect outlived the whole deadline"+ ).not.toBeNull();+ expect(settledOnItsOwn as unknown as number).toBeLessThan(budget);+ }, 90_000);+});diff --git a/packages/cli/src/recovery-contract/recovery-switches-blackbox.test.ts b/packages/cli/src/recovery-contract/recovery-switches-blackbox.test.tsnew file mode 100644index 00000000..34ed6a59--- /dev/null+++ b/packages/cli/src/recovery-contract/recovery-switches-blackbox.test.ts@@ -0,0 +1,174 @@+/**+ * CONTRACT POINT 8 — three switches (CLI flag, env var, config field) and their+ * precedence: flag > env > project config > global config > default(on).+ *+ * The env arm lives in recovery-ladder-blackbox.test.ts (with its own paired+ * control). This file covers the two CONFIG scopes, their precedence against+ * the env var, and the invariant that costs a config field its life:+ * a `ClaudishProfileConfig` field missing from `loadConfig`'s allowlist+ * survives on disk until the first global save and is then silently dropped.+ * Every other switch test passes right up to that save.+ *+ * Each arm takes a fresh endpoint: a recovery episode is per-endpoint and+ * outlives the request that opened it.+ */++import { afterAll, afterEach, beforeAll, describe, expect, test } from "bun:test";+import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";+import { setConfigFileOverride } from "../config-override.js";+import { loadConfig, saveConfig } from "../profile-config.js";+import {+ type FixturePool,+ type ProxyHandle,+ createRawFixturePool,+ endpointsConfig,+ envSnapshot,+ postMessage,+ sleep,+ startProxy,+} from "./fixtures.js";++const API_TIMEOUT_MS = 40_000; // ⇒ 10s deadline ⇒ attempts at 0s, 5s++let restoreEnv: () => void;+let proxy: ProxyHandle;+let pool: FixturePool;+let projectDir: string;+let originalCwd: string;++beforeAll(async () => {+ restoreEnv = envSnapshot(["API_TIMEOUT_MS", "CLAUDISH_RECOVERY", "CLAUDISH_RECOVERY_UI"]);+ process.env.API_TIMEOUT_MS = String(API_TIMEOUT_MS);+ delete process.env.CLAUDISH_RECOVERY;++ originalCwd = process.cwd();+ projectDir = mkdtempSync(join(tmpdir(), "recovery-project-"));++ pool = createRawFixturePool(8, "ep-switch");+ proxy = await startProxy(endpointsConfig(pool.specs));++ // Warm-up: the FIRST request through a fresh proxy pays one-time catalog and+ // registry latching (~3s), which would otherwise be charged to whichever+ // "answers immediately" assertion happened to run first. The header keeps the+ // warm-up itself out of the ladder.+ const warm = pool.next();+ await postMessage(proxy.url, {+ model: warm.model,+ headers: { "x-claudish-no-recovery": "1" },+ });+}, 60_000);++afterEach(() => {+ process.chdir(originalCwd);+ rmSync(join(projectDir, ".claudish.json"), { force: true });+ delete process.env.CLAUDISH_RECOVERY;+});++afterAll(async () => {+ process.chdir(originalCwd);+ await proxy?.shutdown();+ pool?.stopAll();+ rmSync(projectDir, { recursive: true, force: true });+ restoreEnv?.();+});++function writeProjectConfig(value: unknown): void {+ writeFileSync(join(projectDir, ".claudish.json"), JSON.stringify(value), "utf8");+ process.chdir(projectDir);+}++/** Did the ladder run? Answered at the fixture's socket, with a bounded wait. */+async function ladderRan(headers?: Record<string, string>): Promise<{+ attempts: number;+ elapsedMs: number;+ status: number;+}> {+ const { model, fixture } = pool.next();+ const result = await postMessage(proxy.url, { model, headers });+ return {+ attempts: fixture.connections.length,+ elapsedMs: result.elapsedMs,+ status: result.status,+ };+}++describe("CP-8 — the config switches", () => {+ test("project config recovery.enabled=false ⇒ one attempt, immediate answer", async () => {+ writeProjectConfig({ recovery: { enabled: false } });+ const run = await ladderRan();+ expect(run.attempts).toBe(1);+ expect(run.elapsedMs).toBeLessThan(2_000);+ expect(run.status).toBe(400);+ }, 30_000);++ test("PAIRED CONTROL: the same project file with the field removed ⇒ the ladder runs", async () => {+ writeProjectConfig({ someUnrelatedKey: true });+ const run = await ladderRan();+ expect(run.attempts).toBeGreaterThanOrEqual(2);+ expect(run.elapsedMs).toBeGreaterThan(4_000);+ }, 30_000);++ test("an explicit project ON beats a project-scope absence and keeps the ladder", async () => {+ writeProjectConfig({ recovery: { enabled: true } });+ const run = await ladderRan();+ expect(run.attempts).toBeGreaterThanOrEqual(2);+ }, 30_000);++ test("the env var beats the project config", async () => {+ // Documented precedence: flag > env > project > global > default(on).+ writeProjectConfig({ recovery: { enabled: true } });+ process.env.CLAUDISH_RECOVERY = "0";+ const run = await ladderRan();+ expect(run.attempts).toBe(1);+ expect(run.elapsedMs).toBeLessThan(2_000);+ }, 30_000);++ test("an empty object in the project scope is 'no opinion', not 'off'", async () => {+ writeProjectConfig({ recovery: {} });+ const run = await ladderRan();+ expect(run.attempts).toBeGreaterThanOrEqual(2);+ }, 30_000);++ test("a garbage value does not silently disable recovery", async () => {+ writeProjectConfig({ recovery: { enabled: "maybe" } });+ const run = await ladderRan();+ expect(run.attempts).toBeGreaterThanOrEqual(2);+ }, 30_000);+});++describe("CP-8 — the switch survives a config save", () => {+ test("recovery and recoveryUi are still on disk after load → save", async () => {+ // The recorded failure mode: a field absent from loadConfig's allowlist+ // lives on disk until the first global save and is then dropped with no+ // error. Every behavioural switch test above passes right up to that save.+ const dir = mkdtempSync(join(tmpdir(), "recovery-save-"));+ const file = join(dir, "config.json");+ writeFileSync(+ file,+ JSON.stringify({+ recovery: { enabled: false },+ recoveryUi: { enabled: false },+ stats: { enabled: false },+ }),+ "utf8"+ );+ setConfigFileOverride(file);+ try {+ const loaded = loadConfig();+ expect(loaded.recovery).toEqual({ enabled: false });+ expect(loaded.recoveryUi).toEqual({ enabled: false });++ saveConfig(loaded);+ await sleep(10);++ const onDisk = JSON.parse(readFileSync(file, "utf8"));+ expect(onDisk.recovery).toEqual({ enabled: false });+ expect(onDisk.recoveryUi).toEqual({ enabled: false });+ } finally {+ setConfigFileOverride(null);+ rmSync(dir, { recursive: true, force: true });+ }+ }, 20_000);+});diff --git a/packages/cli/src/recovery/coordinator.ts b/packages/cli/src/recovery/coordinator.tsindex 91f921ca..5d7b3d70 100644--- a/packages/cli/src/recovery/coordinator.ts+++ b/packages/cli/src/recovery/coordinator.ts@@ -29,7 +29,7 @@import { randomUUID } from "node:crypto";import { type ConnectionErrorKind, isLoopback } from "../handlers/shared/connection-error.js";-import { log } from "../logger.js";+import { log, logRecovery } from "../logger.js";import { recoveryClock } from "./clock.js";import {RECOVERY_PROTOCOL_VERSION,@@ -364,7 +364,7 @@ export function joinEpisode(seed: EpisodeSeed): EpisodeHandle {ep.clientRetries++;ep.state = "attempting";ep.maxTier = 2;- log(+ logRecovery(`[Recovery] ${ep.providerDisplayName} rejoined episode ${ep.episodeId} ` +`(client retry ${ep.clientRetries}, ladder ${ep.ladderIndex}, attempts ${ep.attempts})`);@@ -410,7 +410,7 @@ export function joinEpisode(seed: EpisodeSeed): EpisodeHandle {};episodes.set(key, ep);byId.set(ep.episodeId, ep);- log(+ logRecovery(`[Recovery] episode ${ep.episodeId} opened for ${ep.providerDisplayName} at ${ep.endpoint} ` +`(${ep.kind}/${ep.code ?? "?"})`);@@ -508,7 +508,7 @@ export function joinEpisode(seed: EpisodeSeed): EpisodeHandle {episode.advancedForRound = episode.round;episode.ladderIndex++;}- log(+ logRecovery(`[Recovery] ${episode.providerDisplayName} attempt ${episode.attempts} failed: ${outcome} ` +`(episode ${episode.episodeId}, ladder ${episode.ladderIndex})`);@@ -554,7 +554,7 @@ export function joinEpisode(seed: EpisodeSeed): EpisodeHandle {}, EPISODE_GRACE_MS);c.unref?.(timer);episode.graceTimer = timer;- log(+ logRecovery(`[Recovery] episode ${episode.episodeId} handed off after ${Math.round(c.now() - episode.startedAtPerf)}ms, ${episode.attempts} attempts — holding ${EPISODE_GRACE_MS / 1000}s for a client retry`@@ -580,7 +580,7 @@ function armAttemptTimer(episode: Episode, delay: number): void {episode.round++;for (const w of [...episode.waiters]) w.wake?.({ kind: "attempt" });}, delay);- log(+ logRecovery(`[Recovery] ${episode.providerDisplayName} waiting ${delay / 1000}s before attempt ` +`${episode.attempts + 1} (episode ${episode.episodeId}, ${episode.waiters.size} waiting)`);@@ -610,7 +610,7 @@ function closeEpisode(episode: Episode, outcome: RecoveryOutcome): void {if (episode.state === "abandoned") {for (const w of [...episode.waiters]) w.wake?.({ kind: "gave_up" });}- log(+ logRecovery(`[Recovery] episode ${episode.episodeId} closed: ${outcome} after ` +`${Math.round(clock.now() - episode.startedAtPerf)}ms and ${episode.attempts} attempts`);@@ -678,7 +678,7 @@ export function noteTargetReachable(providerName: string, endpoint: string): voifor (const [key, ep] of [...episodes]) {if (key !== exact && !key.startsWith(prefix)) continue;if (ep.state !== "handoff") continue;- log(+ logRecovery(`[Recovery] ${ep.providerDisplayName} answered on the client's own retry — ` +`closing episode ${ep.episodeId} instead of waiting out its grace`);diff --git a/packages/cli/src/stats.ts b/packages/cli/src/stats.tsindex 2ab2c23c..0244d871 100644--- a/packages/cli/src/stats.ts+++ b/packages/cli/src/stats.ts@@ -127,6 +127,28 @@ export function initStats(_config: ClaudishConfig): void {}}+/**+ * Tests only: drop the one-shot init latch.+ *+ * `initStats` is idempotent by design — it must be, since several entry points+ * call it — and the latch is PROCESS state that a test file shares with every+ * sibling. Bun runs a suite's files in one process, so whichever file+ * initialised stats first decided `statsEnabled` for all of them: a later file+ * that enables stats and asserts on recorded events silently asserts nothing,+ * because `recordStats`'s first line returns on the stale `initialized` flag.+ * A no-op assertion is the one test failure mode that never turns red.+ *+ * Same convention as `__resetEndpointDiagnosticsForTests` and+ * `__resetPredefinedStateForTests`: exported, `__`-prefixed, never called by+ * production code.+ */+export function __resetStatsForTests(): void {+ initialized = false;+ statsEnabled = false;+ claudishVersion = "";+ installMethod = "unknown";+}+/*** Record a stats event. Fast exit if disabled.* Buffers to memory via appendEvent() — non-blocking.