Skip to content

Claude Code. Any Model. The most powerful AI coding agent now speaks every language.

fix(recovery): make the budget, the chain guard and the log real in the arm nobody tested

Jack Rudenko09eb4deb75d2
diff --git a/ai-docs/architecture/network-recovery.md b/ai-docs/architecture/network-recovery.md
index d4d79e0b..16221e58 100644
--- a/ai-docs/architecture/network-recovery.md
+++ b/ai-docs/architecture/network-recovery.md
@@ -47,7 +47,32 @@ The other two ceilings the derivation clears:
| ceiling | value | where |
|---|---|---|
| Bun's per-request idle timeout | 255 s | `proxy-server.ts`'s `idleTimeout` |
-| the unclamped attempt 1 | up to 45 s | reserved out of the budget, never clamped (NFR-1) |
+| attempt 1 | the remaining budget | `deadlineClamp`, armed before the primary fetch |
+
+### The deadline is only a deadline if attempt 1 is inside it — and twice it was not
+
+Both halves of this were found by a black-box contract test, and neither is visible to a suite built
+on refused loopback ports, because a 1 ms attempt cannot cross a deadline in the middle of itself.
+
+**The zero point was set by the first READER, not by the request.** `inboundStartedAtPerf` memoises
+on `c.req.raw`, and on the fetch path the first caller was `tier1DeadlineAt(c)` *inside the catch* —
+after attempt 1 had already failed. Measured against the unrouted `192.0.2.1` with
+`API_TIMEOUT_MS=60000` (a 30 s budget): attempt 1 spent macOS's 75 s connect timeout, the budget then
+started counting from t+75 s, and **the client was answered at 108 s**. The budget had not been
+exceeded — it had never been applied. `handle()` now stamps the zero point on entry.
+
+**Attempt 1 had no ceiling at all.** `DEADLINE_MARGIN_MS`'s 30 s was supposed to reserve room for its
+tail; the OS's connect timeout is 75 s, so the reserve is wrong by 45 s for a slow connect and wrong
+by more than the whole budget whenever `API_TIMEOUT_MS ≤ 105 s`. It is now bounded by `deadlineAt` —
+and by `deadlineAt` specifically, **not** by `PER_ATTEMPT_CONNECT_CAP_MS`:
+
+- 45 s is a plausible time-to-first-byte for a thinking model behind a long prompt, and this ceiling
+ sits on a possibly-HEALTHY call. Clamping it at 45 s would re-open RISK-6's first leak — a latency
+ event reclassified as "the host is unreachable". At the default settings the ceiling is 270 s,
+ which no provider's first byte approaches, and it shortens only as the user's own knob does;
+- firing AT the deadline is what makes it safe: there is by construction no budget left when the
+ abort is classified, so `shouldSkipTier1`'s `no-budget` gate fires and the request answers. **This
+ ceiling can never cause a re-issue.** An earlier firing time would not have that property.
Bun's 255 s stops binding because `c.env.timeout(c.req.raw, 0)` **works** — measured, a request
given `0` survived **6×** its `idleTimeout` where the un-disarmed control died at 1.6×. That call is
@@ -146,6 +171,47 @@ a duration quoted verbatim in a user-facing message to this one.
On a tier-2 rejoin the ladder does **not** reset. The target has been down for the whole of it, and
restarting at 5 s would hammer a dead host harder the longer the outage lasted.
+### …but a carried rung may be TRUNCATED once, or the re-ask retries nothing
+
+The ladder index belongs to the **episode**; the deadline belongs to the **socket**, and a re-ask
+gets a fresh — possibly much shorter — one. Once the carried rung outgrew that deadline, the
+no-budget gate fired on the first iteration and the request answered **in 8 ms having re-issued
+nothing**. Measured at the upstream socket with `API_TIMEOUT_MS=60000`:
+
+| `API_TIMEOUT_MS` | deadline | request 1 | request 2 (before) | request 2 (now) |
+|---|---|---|---|---|
+| 120 000 | 90 s | 0/5/15/45 s | 0/60 s | unchanged |
+| 60 000 | 30 s | 0/5/15 s | **one attempt, 8 ms** | 0/27 s |
+| 40 000 | 10 s | 0/5 s | 0/10 s | unchanged |
+
+And it was permanent: with no banner the answer is a 400, which Claude Code does not re-ask, so
+recovery was over for that endpoint for the rest of the outage. Invisible at the 270 s default, where
+the terminal 60 s rung always fits — it bites every `API_TIMEOUT_MS ≤ 90 s`, a value
+`docs/advanced/environment.md` invites.
+
+So a waiter that has **not yet re-issued anything** shortens the wait to the budget it actually has
+(floor `MIN_ATTEMPT_SLOT_MS`), makes its one attempt, and only then hands off. The condition is
+`requestRetries === 0`, which is why request 1's shape above is untouched: by the time ITS rung stops
+fitting it has re-issued two or three times already. It is not a hammer — at most one truncated rung
+per request, and the gap between two client re-asks is the client's own backoff (up to 38.4 s), not
+ours.
+
+### Four places watch the client's abort, and a mutation that survived does not make them redundant
+
+Disabling all three abort paths in `transient-retry.ts` — the loop-top check, `untilAborted`, and
+`mergeSignalIntoInit`'s threading — left client-disconnect handling working. The surviving path is
+`coordinator.ts`'s `waitForNextAttempt`, which registers its own listener on the same signal; against
+a refused loopback port the ladder spends ~100% of its wall clock parked in that wait, so the wait
+answered. Each covers a window no other one does, and a slow connect inverts the arithmetic (45 s of
+attempt to 5 s of wait):
+
+| # | site | the window only it covers |
+|---|---|---|
+| 1 | the loop's `if (ctx.signal.aborted)` | between an attempt settling and the next wait starting; and the first iteration, before any wait exists |
+| 2 | `coordinator.waitForNextAttempt` | during the wait — and it is the only one that can DETACH the waiter from the shared episode timer |
+| 3 | `untilAborted` | during an attempt whose operation cannot take a signal at all: `refreshAuth()`/`getHeaders()` accept no arguments, so on the auth path this is the only unwind there is |
+| 4 | `mergeSignalIntoInit` | inside `fetch`, so the SOCKET closes. The other three unwind the waiter and leave the connect running |
+
### `PER_ATTEMPT_CONNECT_CAP_MS` is 45 s because 75 s collides with the OS
macOS gives up on its own TCP connect at **75.004 s** (measured against the unrouted `192.0.2.1`; a
@@ -229,6 +295,30 @@ So the marker is a **header**, `x-claudish-recovery: 1`, checked in two independ
`handle()` returns a marked response VERBATIM before reading the body (defeating the combining), and
`isRetryableError` returns `false` on it **above** the quota match (defeating the wording).
+### One marker was not enough: it was minted on the 503 arm ONLY
+
+Which made chain-safety a property of **which arm answered**. Exhaustion answers 400 whenever no
+banner is attached — `-p`, `--no-recovery`, CI, any machine without magmux, i.e. *every headless
+run* — and that 400 carried no marker at all, so the wording check decided again. A black-box test
+reproduced it from outside: candidate 1 unreachable at a host named `quota-exceeded-…`, and
+**candidate 2's own socket served the client**. `insufficient-credits` and `rate-limit` in the same
+position did not reproduce it, so the variable really was the word — and the error text quotes the
+endpoint host and full URL, so the word is user-influenced.
+
+The 400 arm now carries its own marker, `x-claudish-connection-error: 1`, and `isRetryableError`
+asks one predicate — `isClaudishConnectionVerdict` — that is true of either. The invariant is
+stated in terms of the request rather than of the response:
+
+> **A connection-failure response from this feature never advances the fallback chain — regardless
+> of its status, its message, or whether a banner was attached.**
+
+It is deliberately a SECOND header rather than the same one on both arms, because the two facts
+differ and one is load-bearing elsewhere: `x-claudish-recovery` means "the retry was handed back",
+and `FallbackHandler` returns anything wearing it **verbatim** so `formatCombinedError` cannot demote
+its 503. A 400 has nothing handed back and must stay foldable into the combined chain error, which is
+where the user reads what every candidate did. Round 1's quota mutation tested the 503 path and
+passed; nobody tested the 400 path.
+
> **Moving the marker check below the quota match — present, but LATE — is a live billing bug.**
> It is mutation-covered as one, and the mutation that kills it is the quota-wording variant
> specifically. A test whose 503 body says nothing about quota cannot see the defect at all.
@@ -399,6 +489,24 @@ would be a phantom distorting request counts, success rate and latency percentil
stream that is supposed to answer the question. Episode lifecycle is a `[Recovery]` structural log
line instead.
+### `[Recovery]` is OUTPUT, not a debug aid — and for a while it was neither
+
+The lifecycle lines were on `log()`, which writes to two FILES: the `--debug` log and the always-on
+structural log under `~/.claudish/logs/`. That satisfies "it is written down" and fails the thing it
+was written down for. A hold runs for up to the derived deadline (~4.5 minutes at the default) and in
+a run with no pane — which is *every headless run* — the user saw nothing at all while it did. A
+silent multi-minute hold with the reason legible nowhere is the state §3 calls "strictly worse than
+the bug this feature exists to remove"; a file the user does not know to open is not a surface.
+
+They now go through `logRecovery()` → `logStderr`, which is why the noise objection does not bite:
+in an interactive session `logStderr` routes to `DiagOutput` (a file) rather than to the client's
+TUI, and in quiet mode it is suppressed — so the lines land on a terminal exactly where a terminal is
+the only surface there is. Both log files still receive them.
+
+The line is drawn at the **episode's lifecycle** — opened, rejoined, each attempt, each wait, handed
+off, closed, recovered, exhausted: the six-to-ten lines from which a reader can reconstruct the
+ladder. Pane connect/disconnect, socket paths, a skipped ladder and `client_gone` stay on `log()`.
+
### The trap: `eventToLogRecord` is a hand-written allowlist
A field added to the `StatsEvent` interface and populated by `stats.ts` but **not pushed in
@@ -503,10 +611,23 @@ MCP server process dying outright.
## 8. What a healthy request pays
-Nothing, and the guarantee is **placement**, not a flag: the entire retry apparatus is constructed
-inside the `catch`, after classification has already returned non-null. The primary fetch expression
-— including its `enqueueRequest` ternary — is byte-identical to what it was before recovery existed.
-That is verifiable by reading the diff rather than by trusting a predicate.
+**Three statements, and the guarantee around them is still placement rather than a flag**: the entire
+retry apparatus — episode, ladder, waiters, pane — is constructed inside the `catch`, after
+classification has already returned non-null. The `enqueueRequest` ternary that issues attempt 1 is
+byte-identical to what it was before recovery existed. What a healthy request now pays, outside that
+catch, is exactly:
+
+| | cost | why it cannot be moved into the catch |
+|---|---|---|
+| `inboundStartedAtPerf(c)` at `handle()` entry | one clock read, one `WeakMap.set` | the deadline's zero point must be the request's start; asking for it later is what produced the 108 s answer in §1 |
+| `tier1DeadlineAt(c)` before the fetch | arithmetic over one env read | the ceiling has to exist before the call it bounds |
+| `deadlineClamp(deadlineAt)` | one `AbortController`, one timer, one `AbortSignal.any` | same |
+
+The timer is disarmed in a `finally` the instant the call settles — before a byte of the body is
+read — so it can only ever bound connect-and-headers, never cut a response already arriving. This
+replaces an earlier, stronger claim ("nothing, and the primary fetch expression is byte-identical"):
+that claim was true and the feature was not, because a budget nothing enforces on attempt 1 is not a
+budget.
The re-issue goes through that **same ternary**. Six transports implement `enqueueRequest`, and what
they implement is not decoration: a bounded 429 loop with `Retry-After`, a served-set model-fallback
diff --git a/docs/advanced/environment.md b/docs/advanced/environment.md
index d9c6b29d..63ea502a 100644
--- a/docs/advanced/environment.md
+++ b/docs/advanced/environment.md
@@ -153,7 +153,11 @@ The deadline is **derived from `API_TIMEOUT_MS`**, not fixed:
min(API_TIMEOUT_MS, 300s) − 30s # ~270s at the 360s default
```
-Claude Code aborts its own request at `API_TIMEOUT_MS` (360 000 ms by default), so a hold longer than that would be cut off by the client mid-wait. If you have shortened `API_TIMEOUT_MS`, claudish shortens the hold to match and logs a `[Recovery]` line saying so.
+Claude Code aborts its own request at `API_TIMEOUT_MS` (360 000 ms by default), so a hold longer than that would be cut off by the client mid-wait. If you have shortened `API_TIMEOUT_MS`, claudish shortens the hold to match and says so with a `[Recovery]` line.
+
+The deadline is the ceiling on the **whole** attempt, including the first one: a connect that never completes is abandoned when the budget runs out rather than waiting out the operating system's own connect timeout (75 s on macOS), which on a shortened `API_TIMEOUT_MS` would answer you after the client had already given up.
+
+**Where the `[Recovery]` lines go.** A recovery episode reports itself on **stderr** — opened, each attempt, each wait and how it ended — with no `--debug` anywhere, because a request can be held for minutes and a hold nobody can see is worse than a fast failure. In an interactive session they are routed to the session log instead of the terminal, so they cannot corrupt Claude Code's display; `~/.claudish/logs/` keeps them either way.
### `CLAUDISH_RECOVERY_UI`
diff --git a/packages/cli/src/handlers/composed-handler.ts b/packages/cli/src/handlers/composed-handler.ts
index 53e7b623..c846570d 100644
--- a/packages/cli/src/handlers/composed-handler.ts
+++ b/packages/cli/src/handlers/composed-handler.ts
@@ -33,7 +33,14 @@ import { resolveQuotaAdapter } from "../auth/quota/registry.js";
import { PLAN_POLL_INTERVAL_MS } from "../auth/quota/types.js";
import type { BehaviorEngine, BehaviorSession } from "../behavior/index.js";
import { getBehaviorEngine } from "../behavior/index.js";
-import { getLogLevel, log, logStderr, logStructured, truncateContent } from "../logger.js";
+import {
+ getLogLevel,
+ log,
+ logRecovery,
+ logStderr,
+ logStructured,
+ truncateContent,
+} from "../logger.js";
import { GeminiThoughtSignatureMiddleware, MiddlewareManager } from "../middleware/index.js";
import { deepMergeParams } from "../model-params.js";
import { describeSiblingKeys, getProviderByName } from "../providers/provider-definitions.js";
@@ -75,7 +82,7 @@ import { sniffDevinStreamHead } from "./shared/devin-stream-head-sniffer.js";
import { hasActionableLink, hasModelUnsupportedWording } from "./shared/model-unsupported.js";
import { filterIdentity } from "./shared/openai-compat.js";
import { hasPlanLimitWording, isQuotaExhaustionError } from "./shared/quota-exhaustion.js";
-import { recoveryHoldHeaders } from "./shared/recovery-marker.js";
+import { connectionFaultHeaders, recoveryHoldHeaders } from "./shared/recovery-marker.js";
import {
CONTEXT_OVERFLOW_PHRASE,
isContextOverflowError,
@@ -92,6 +99,8 @@ import { TokenTracker, type UsageCacheDetail } from "./shared/token-tracker.js";
import {
type ConnectionErrorInfo,
MIN_ATTEMPT_SLOT_MS,
+ deadlineClamp,
+ inboundStartedAtPerf,
mergeSignalIntoInit,
refreshDeadlineAt,
tier1DeadlineAt,
@@ -512,7 +521,25 @@ export class ComposedHandler implements ModelHandler {
} catch {
// Stats must never crash claudish
}
- return c.json(wrapAnthropicError(400, msg, "connection_error"), 400 as any);
+ // The chain-safety marker, on THIS arm as well as on the 503.
+ //
+ // It is the third argument rather than a `c.header()` call, because `c` is
+ // SHARED across every candidate in a fallback chain and a sticky header set
+ // here would ride out on `formatCombinedError`'s response — the same
+ // invariant `respondRecoveryHold` obeys by building a standalone Response.
+ // Hono's `newResponse` applies this bag to the returned Response only; it
+ // never touches the context's own header store.
+ //
+ // Without it, chain-safety existed exactly where a banner did. A headless
+ // exhaustion answered an unmarked 400, `isRetryableError` fell through to
+ // its status-agnostic quota-wording match, and a host whose NAME contained
+ // "quota" advanced the chain onto a metered provider mid-outage. See
+ // `recovery-marker.ts`.
+ return c.json(
+ wrapAnthropicError(400, msg, "connection_error"),
+ 400 as any,
+ connectionFaultHeaders()
+ );
}
/**
@@ -783,7 +810,7 @@ export class ComposedHandler implements ModelHandler {
ctx.recovery.outcome = result.kind === "exhausted" ? "handoff" : "gave_up";
if (result.kind === "exhausted" && leased) {
- log(
+ logRecovery(
`[Recovery] ${this.provider.displayName} exhausted after ${result.attempts} attempts ` +
`in ${result.recoveryMs}ms (episode ${result.episodeId}, ui_lease=true) — ` +
"handing the retry back to the client (503, tier 2)"
@@ -800,7 +827,7 @@ export class ComposedHandler implements ModelHandler {
};
}
- log(
+ logRecovery(
`[Recovery] ${this.provider.displayName} exhausted after ${result.attempts} attempts ` +
`in ${result.recoveryMs}ms (episode ${result.episodeId}, outcome ${result.kind}, ` +
`ui_lease=${leased}) — answering connection_error`
@@ -851,6 +878,23 @@ export class ComposedHandler implements ModelHandler {
async handle(c: Context, payload: any): Promise<Response> {
const startTime = performance.now();
+ // Stamp the recovery deadline's ZERO POINT here, and not wherever something
+ // first asks for it.
+ //
+ // `inboundStartedAtPerf` memoises on `c.req.raw`, so before this line the
+ // zero point was set by the FIRST caller — which on the fetch path is
+ // `tier1DeadlineAt(c)` inside the `catch`, i.e. AFTER attempt 1 has already
+ // failed. MEASURED against an unrouted address: attempt 1 burned macOS's
+ // 75 s connect timeout, the deadline's clock then started from there, and a
+ // request with a 30 s budget answered at 108 s — the budget was not
+ // exceeded, it was never applied. A ladder cannot bound a request whose
+ // clock starts after the overrun it was supposed to bound.
+ //
+ // One `performance.now()` and one `WeakMap.set` per request; nothing else
+ // on the healthy path reads it. A later candidate in a fallback chain finds
+ // the value already there and inherits it, which is the documented intent —
+ // the deadline belongs to the inbound request, not to a candidate.
+ inboundStartedAtPerf(c);
// latency_ms = time-to-first-byte (from request send to successful response).
// Captured here so it is available to the post-stream stats callback below.
let latencyMs = 0;
@@ -1261,19 +1305,42 @@ export class ComposedHandler implements ModelHandler {
// Merge provider-specific fetch options (e.g., undici dispatcher, abort signal)
const requestInit = this.provider.getRequestInit?.() || {};
+
+ // THE REQUEST'S OWN CEILING — the one thing outside the `catch` that this
+ // feature adds to a healthy request, and the reason it is here rather than
+ // in the ladder: an unclamped attempt 1 could outlive the whole derived
+ // deadline (measured: 75 s of macOS connect against a 30 s budget, answered
+ // at 108 s), which makes the budget a suggestion. It is absolute, it fires
+ // at `deadlineAt` and never earlier, and it is disarmed the instant the
+ // call settles — see `deadlineClamp` for why both of those are what keep it
+ // from turning a slow answer into a re-issued one.
+ const deadlineAt = tier1DeadlineAt(c);
+ const clamp = deadlineClamp(deadlineAt);
const doFetch = () =>
- fetch(endpoint, {
- method: "POST",
- headers,
- body: serialized?.body ?? JSON.stringify(requestPayload),
- ...requestInit,
- });
+ fetch(
+ endpoint,
+ mergeSignalIntoInit(
+ {
+ method: "POST",
+ headers,
+ body: serialized?.body ?? JSON.stringify(requestPayload),
+ ...requestInit,
+ },
+ clamp.signal
+ )
+ );
let response: Response;
try {
- response = this.provider.enqueueRequest
- ? await this.provider.enqueueRequest(doFetch)
- : await doFetch();
+ try {
+ response = this.provider.enqueueRequest
+ ? await this.provider.enqueueRequest(doFetch)
+ : await doFetch();
+ } finally {
+ // Before a byte of the body is read, so the ceiling can only ever bound
+ // connect-and-headers — never cut a response already arriving.
+ clamp.disarm();
+ }
} catch (error: any) {
// A failure to even REACH the provider (DNS can't resolve, connection
// refused, host unreachable) is a LOCAL network problem, not an upstream
@@ -1333,7 +1400,8 @@ export class ComposedHandler implements ModelHandler {
const outcome = await this.recoverConnection<Response>(c, error, conn, endpoint, reissue, {
startTime,
- deadlineAt: tier1DeadlineAt(c),
+ // The SAME value the primary attempt was bounded by, not a fresh read.
+ deadlineAt,
fallbackMeta,
site: "fetch",
recovery: recoveryTally,
diff --git a/packages/cli/src/handlers/fallback-handler.ts b/packages/cli/src/handlers/fallback-handler.ts
index 69899eee..343a0139 100644
--- a/packages/cli/src/handlers/fallback-handler.ts
+++ b/packages/cli/src/handlers/fallback-handler.ts
@@ -14,7 +14,7 @@ import { logStderr } from "../logger.js";
import { ComposedHandler } from "./composed-handler.js";
import { extractUpstreamStatus } from "./shared/anthropic-error.js";
import { hasQuotaExhaustionWording } from "./shared/quota-exhaustion.js";
-import { hasRecoveryMarker, isRecoveryHoldResponse } from "./shared/recovery-marker.js";
+import { isClaudishConnectionVerdict, isRecoveryHoldResponse } from "./shared/recovery-marker.js";
import type { ModelHandler } from "./types.js";
export interface FallbackCandidate {
@@ -213,23 +213,25 @@ export function isRetryableError(
provider?: string,
headers?: Headers
): boolean {
- // ── CLAUDISH'S OWN RECOVERY 503 IS NEVER A REASON TO ADVANCE ───────────────
+ // ── CLAUDISH'S OWN "CANNOT REACH THE HOST" IS NEVER A REASON TO ADVANCE ────
//
// FIRST, above the quota wording check, and the ORDER is the whole point.
// The next statement is `hasQuotaExhaustionWording(errorBody)`, which is
// deliberately status-agnostic and whose phrase list contains the bare
- // substring "quota" — so a recovery 503 that merely mentioned one would be
- // read as a spent subscription and walk the user onto metered billing in the
- // middle of a network outage.
+ // substring "quota" — so one of our own connection verdicts that merely
+ // mentioned one would be read as a spent subscription and walk the user onto
+ // metered billing in the middle of a network outage. The error text quotes
+ // the endpoint HOST and URL, so "merely mentioned" is not hypothetical: a
+ // host named `quota-…` reproduced it, and only that word did.
//
- // `handle()` already returns a marked response verbatim before this function
- // is consulted, so in the shipped call graph this branch is unreachable.
- // It is kept anyway, and it is not belt-and-braces decoration: it makes the
- // guarantee a property of the DECISION rather than of one call site, so a
- // future refactor that moves, inlines or reorders the check in `handle()`
- // cannot silently re-open the billing hole. Two independent mechanisms, one
- // marker.
- if (hasRecoveryMarker(headers)) return false;
+ // BOTH arms, via one predicate. The 503 handoff carries `x-claudish-recovery`
+ // and is already returned verbatim by `handle()` above, so for that arm this
+ // branch is a second independent mechanism. The 400 arm — every headless run,
+ // where no banner can exist — carries `x-claudish-connection-error`, and for
+ // IT this branch is the only mechanism there is. Asking the combined
+ // predicate rather than either half is what stops chain-safety from being a
+ // property of which arm happened to answer.
+ if (isClaudishConnectionVerdict(headers)) return false;
// A spent subscription allowance is retryable AT THE CHAIN LEVEL: this
// provider cannot serve, but the next one can.
diff --git a/packages/cli/src/handlers/shared/recovery-marker.ts b/packages/cli/src/handlers/shared/recovery-marker.ts
index 648288bb..75177c33 100644
--- a/packages/cli/src/handlers/shared/recovery-marker.ts
+++ b/packages/cli/src/handlers/shared/recovery-marker.ts
@@ -51,7 +51,10 @@
* the marker, so a forged one on a 200 changes nothing.
*
* Keep both properties true. If a future edit returns an upstream `Response`
- * object on a non-ok path, the marker must be stripped there.
+ * object on a non-ok path, the markers must be stripped there.
+ *
+ * Both markers below rest on exactly this argument, and both are minted at a
+ * `c.json(...)` call whose headers Hono builds from nothing.
*/
/** The header name. Referenced by name in no other file — import this. */
@@ -60,6 +63,49 @@ export const RECOVERY_MARKER_HEADER = "x-claudish-recovery";
/** The only value that counts. Anything else is not our response. */
export const RECOVERY_MARKER_VALUE = "1";
+/**
+ * The SECOND marker, and the reason it exists rather than the first one being
+ * reused.
+ *
+ * ── THE DEFECT IT CLOSES (found by a black-box contract test, F-2) ──────────
+ *
+ * `x-claudish-recovery` is minted on ONE arm: the 503 handoff, which requires a
+ * valid UI lease. When a hold exhausts with no banner — `-p`, `--no-recovery`,
+ * CI, any machine without magmux, i.e. EVERY headless run — the answer is a 400
+ * `connection_error` and there was no marker at all. `isRetryableError` then
+ * fell through to its first real statement, `hasQuotaExhaustionWording`, whose
+ * phrase list contains the bare substring `"quota"` — and the error text of a
+ * connection failure quotes the ENDPOINT HOST AND URL verbatim. A host named
+ * `quota-exceeded…` (or any URL containing the word) was therefore read as a
+ * spent subscription, and the chain ADVANCED off a possibly-subscription
+ * candidate onto a metered one in the middle of a network outage. Measured:
+ * candidate 2's own socket served the client; `insufficient-credits` and
+ * `rate-limit` in the same position did not reproduce it, so the variable
+ * really was the word.
+ *
+ * The guarantee the two markers hold together, and the shape of it matters:
+ *
+ * > **A connection-failure response from this feature never advances the
+ * > fallback chain — regardless of its STATUS, its MESSAGE, or whether a banner
+ * > was attached.**
+ *
+ * Chain-safety must not be a property of the arm that happened to answer. The
+ * 503 arm's marker made it structural in one arm only, which is worse than
+ * obvious to review: the guard reads as present, and it is present, but exactly
+ * where the money is at stake — a run with no surface — it is not minted.
+ *
+ * ── WHY NOT SIMPLY MINT `x-claudish-recovery` ON BOTH ARMS ──────────────────
+ *
+ * Because the two facts are different, and one of them is load-bearing
+ * elsewhere. `x-claudish-recovery` means "the retry was HANDED BACK; re-POST
+ * and you will rejoin the same episode" — `FallbackHandler` returns such a
+ * response VERBATIM, before reading the body, so `formatCombinedError` cannot
+ * demote its 503 to a terminal 400. A 400 has nothing handed back and must
+ * still be foldable into a combined chain error, which is where the user reads
+ * what every candidate did. Same guarantee, two facts, two headers.
+ */
+export const CONNECTION_FAULT_HEADER = "x-claudish-connection-error";
+
/**
* The headers a tier-1 exhaustion 503 carries.
*
@@ -75,6 +121,21 @@ export function recoveryHoldHeaders(): Record<string, string> {
};
}
+/**
+ * The headers claudish's own "could not reach the host" 400 carries.
+ *
+ * Deliberately NOT `recoveryHoldHeaders()`: no `x-should-retry`, because
+ * nothing was handed back, and no `x-claudish-recovery`, because that header
+ * means "handed back" and `FallbackHandler` returns anything wearing it
+ * verbatim — which would silently drop the other candidates' reasons out of a
+ * combined chain error.
+ *
+ * `content-type` is left to the caller's `c.json`, which sets it.
+ */
+export function connectionFaultHeaders(): Record<string, string> {
+ return { [CONNECTION_FAULT_HEADER]: RECOVERY_MARKER_VALUE };
+}
+
/** Does this header bag carry the marker? Tolerates `undefined` for callers that have none. */
export function hasRecoveryMarker(headers: Headers | undefined | null): boolean {
try {
@@ -85,6 +146,27 @@ export function hasRecoveryMarker(headers: Headers | undefined | null): boolean
}
}
+/** Does this header bag carry the connection-fault marker? */
+export function hasConnectionFaultMarker(headers: Headers | undefined | null): boolean {
+ try {
+ return headers?.get(CONNECTION_FAULT_HEADER) === RECOVERY_MARKER_VALUE;
+ } catch {
+ return false;
+ }
+}
+
+/**
+ * Is this claudish's own verdict that it could not reach the host — on EITHER
+ * arm?
+ *
+ * The one predicate the chain-safety rule is written in terms of, so the rule
+ * cannot again be true of one arm and false of the other. A caller asking "may
+ * I advance past this?" must ask THIS, never one of the two halves.
+ */
+export function isClaudishConnectionVerdict(headers: Headers | undefined | null): boolean {
+ return hasRecoveryMarker(headers) || hasConnectionFaultMarker(headers);
+}
+
/** Is this response claudish's own recovery-exhaustion 503? */
export function isRecoveryHoldResponse(
response: { headers?: Headers } | null | undefined
diff --git a/packages/cli/src/handlers/shared/transient-retry.ts b/packages/cli/src/handlers/shared/transient-retry.ts
index 9cc7bb36..5068100e 100644
--- a/packages/cli/src/handlers/shared/transient-retry.ts
+++ b/packages/cli/src/handlers/shared/transient-retry.ts
@@ -25,7 +25,7 @@
*/
import type { Context } from "hono";
-import { log } from "../../logger.js";
+import { log, logRecovery } from "../../logger.js";
import { recoveryClock } from "../../recovery/clock.js";
import { type EpisodeHandle, joinEpisode } from "../../recovery/coordinator.js";
import { logDeadlineIfShortened, resolveTier1DeadlineMs } from "../../recovery/settings.js";
@@ -108,6 +108,17 @@ export function refreshDeadlineAt(deadlineAt: number): number {
* The residual is real and wanted: a candidate that recovers late and then
* returns a retryable status leaves candidate 2 with only the remaining budget
* rather than a fresh one. That is the point.
+ *
+ * ── THE ZERO POINT IS STAMPED BY `handle()`, NOT BY THE FIRST READER ────────
+ *
+ * Memoisation makes this function's value "whenever somebody first asked", and
+ * on the fetch path the first asker used to be `tier1DeadlineAt(c)` inside the
+ * `catch` — after attempt 1 had already failed. MEASURED against an unrouted
+ * address: attempt 1 spent macOS's 75 s connect timeout, the 30 s budget then
+ * started counting from t+75 s, and the client was answered at 108 s. The
+ * budget had not been exceeded; it had never been applied. `handle()` now
+ * stamps it on entry, which is why a caller must never assume this is cheap to
+ * call late — it is only correct to call it EARLY.
*/
const INBOUND_START = new WeakMap<Request, number>();
@@ -121,11 +132,91 @@ export function inboundStartedAtPerf(c: Context): number {
return t;
}
-/** The absolute tier-1 deadline for this inbound request, PROCESS ms. */
+/**
+ * The absolute tier-1 deadline for this inbound request, PROCESS ms.
+ *
+ * PURE, and it has to stay pure: it is now read BEFORE the primary fetch (to
+ * arm the request ceiling below), so anything it announced would be announced
+ * on healthy requests too. The shortened-hold notice moved to
+ * `noteDeadlineForEpisode`, which runs only once an episode really opens.
+ */
export function tier1DeadlineAt(c: Context): number {
- const budget = resolveTier1DeadlineMs();
- logDeadlineIfShortened(budget);
- return inboundStartedAtPerf(c) + budget;
+ return inboundStartedAtPerf(c) + resolveTier1DeadlineMs();
+}
+
+/**
+ * Announce a hold the environment shortened — from the ladder, once per
+ * process.
+ *
+ * Separated from `tier1DeadlineAt` because that function is now on the healthy
+ * path. A user who has set `API_TIMEOUT_MS=60000` and never has an outage must
+ * not be told about a hold that never happened.
+ */
+export function noteDeadlineForEpisode(): void {
+ logDeadlineIfShortened(resolveTier1DeadlineMs());
+}
+
+/** An armed ceiling on ONE outbound call. `signal` is absent when there is none. */
+export interface DeadlineClamp {
+ /** Compose into the fetch init with `mergeSignalIntoInit`. */
+ signal?: AbortSignal;
+ /** Call the INSTANT the call settles. A live body must never be cut. */
+ disarm(): void;
+}
+
+/**
+ * Bound one outbound call by an ABSOLUTE deadline — the ceiling that makes the
+ * derived budget authoritative for the FIRST attempt as well as for the ladder.
+ *
+ * ── WHY ATTEMPT 1 NEEDED A CEILING AT ALL ───────────────────────────────────
+ *
+ * The ladder's attempts were already clamped (`PER_ATTEMPT_CONNECT_CAP_MS`),
+ * but attempt 1 — the byte-identical primary fetch — was not, on the reasoning
+ * that `DEADLINE_MARGIN_MS` (30 s) reserves room for its tail. macOS's own TCP
+ * connect timeout is **75 s**, so the reserve is wrong by 45 s the moment the
+ * fault is a SLOW connect rather than a refusal, and it is wrong by more than
+ * the entire budget whenever `API_TIMEOUT_MS ≤ 105 s`. A single connect then
+ * outlived the whole request deadline and the client (which gives up at
+ * `API_TIMEOUT_MS`) was gone before we answered. Every refusal-based test is
+ * blind to this: a 1 ms attempt cannot cross a deadline in the middle of
+ * itself.
+ *
+ * ── WHY THE DEADLINE AND NOT `PER_ATTEMPT_CONNECT_CAP_MS` ───────────────────
+ *
+ * This ceiling is on a possibly-HEALTHY call, and 45 s is a plausible
+ * time-to-first-byte for a thinking model behind a long prompt. Clamping the
+ * primary fetch at 45 s would convert an ordinary LATENCY event into "the host
+ * is unreachable" — the exact leak past RISK-6's boundary that this design
+ * already found and closed once. At the default settings this ceiling is 270 s,
+ * which no provider's first byte approaches, and it shortens only as the user's
+ * own `API_TIMEOUT_MS` does.
+ *
+ * ── AND WHY FIRING AT THE DEADLINE IS WHAT MAKES IT SAFE ────────────────────
+ *
+ * Because the abort lands AT `deadlineAt`, there is by construction no budget
+ * left when it is classified: `shouldSkipTier1`'s `no-budget` gate fires and
+ * the request answers. So this ceiling can never cause a RE-ISSUE, and cannot
+ * put a slow-but-alive inference on the ladder to be billed twice. An earlier
+ * firing time would not have that property — keep it absolute.
+ */
+export function deadlineClamp(deadlineAt: number): DeadlineClamp {
+ const clock = recoveryClock();
+ const remaining = deadlineAt - clock.now();
+ // NaN-safe: an unparseable budget leaves the call unbounded, which is
+ // pre-recovery behaviour rather than an instant abort.
+ if (!(remaining > 0)) return { disarm: () => {} };
+ const ac = new AbortController();
+ const timer = clock.setTimeout(
+ // `markOwnTimeout` is what lets this be classified at all — the discriminator
+ // is the signal's ORIGIN, never the name, so a transport's own inference
+ // ceiling keeps its pre-recovery route out.
+ () => ac.abort(markOwnTimeout(new DOMException("request deadline", "TimeoutError"))),
+ remaining
+ );
+ return {
+ signal: ac.signal,
+ disarm: () => clock.clearTimeout(timer),
+ };
}
export interface ConnectionErrorInfo {
@@ -348,6 +439,10 @@ export async function withConnectionRetry<T>(
let error: unknown = first;
let endpoint = ctx.resolveEndpoint(first);
+ // A shortened hold is announced HERE — once a hold is really happening —
+ // rather than by `tier1DeadlineAt`, which the healthy path now calls.
+ noteDeadlineForEpisode();
+
const handle: EpisodeHandle = joinEpisode({
providerName: ctx.providerName,
providerDisplayName: ctx.providerDisplayName,
@@ -377,17 +472,75 @@ export async function withConnectionRetry<T>(
handle.recordAttemptResult(conn.code, false);
for (;;) {
+ // ── FOUR ABORT PATHS, AND WHY NONE OF THEM IS REDUNDANT ───────────────
+ //
+ // A black-box mutation (M6) disabled the three in THIS file — this check,
+ // `untilAborted`, and `mergeSignalIntoInit`'s threading — and the client
+ // disconnect was still handled. It was not proof of redundancy: the
+ // surviving path is `coordinator.ts`'s `waitForNextAttempt`, which
+ // registers its own `abort` listener on the same signal, and a ladder
+ // against a REFUSED loopback port spends ~100% of its wall clock parked
+ // in that wait. The mutation test aborted during a wait, so the wait's
+ // own listener answered it.
+ //
+ // Each covers a window no other one does:
+ //
+ // 1. HERE — an abort that landed while an attempt was settling, or
+ // before any wait has happened at all (first iteration). Nothing
+ // else is listening in that gap.
+ // 2. `coordinator.waitForNextAttempt` — during the wait. Also the only
+ // one that can DETACH the waiter from the shared episode timer.
+ // 3. `untilAborted` — during an attempt whose operation structurally
+ // cannot take a signal: `refreshAuth()`/`getHeaders()` accept no
+ // arguments, so for the auth path this is the ONLY unwind there is.
+ // 4. `mergeSignalIntoInit` — inside `fetch`, so the SOCKET closes
+ // rather than being abandoned. The other three unwind the waiter and
+ // leave the connect running.
+ //
+ // A slow connect inverts the mutation's arithmetic: 45 s of attempt to
+ // 5 s of wait means (3) and (4) do nearly all the work and (2) almost
+ // none. Deleting any of them leaves a window that only shows up under the
+ // fault class the loopback suite cannot produce.
if (ctx.signal.aborted) {
return { kind: "client_gone", ...tally() };
}
- const delay = connectionRetryDelayMs(handle.ladderIndex());
- const now = clock.now();
- // The check bounds the ATTEMPT THAT FOLLOWS the sleep, not just the
- // sleep. Bounding only the sleep is how a budget stops being a ceiling.
- if (now + delay + MIN_ATTEMPT_SLOT_MS > ctx.deadlineAt) {
- handle.handoff();
- return { kind: "exhausted", conn, error, endpoint, ...tally() };
+ // The budget this waiter has left for a WAIT: its own deadline, less the
+ // slot the attempt after the wait needs. The check bounds the ATTEMPT
+ // THAT FOLLOWS the sleep, not just the sleep — bounding only the sleep is
+ // how a budget stops being a ceiling.
+ const waitBudget = ctx.deadlineAt - clock.now() - MIN_ATTEMPT_SLOT_MS;
+ let delay = connectionRetryDelayMs(handle.ladderIndex());
+
+ if (delay > waitBudget) {
+ // ── A CARRIED RUNG MAY BE TRUNCATED, ONCE, BY A REQUEST THAT HAS NOT
+ // RE-ISSUED ANYTHING YET ──────────────────────────────────────────
+ //
+ // The ladder index belongs to the EPISODE and deliberately does not
+ // reset when the client re-asks (§2: restarting at 5 s would hammer a
+ // dead host harder the longer the outage lasted). But the deadline
+ // belongs to the SOCKET, and a re-ask gets a fresh, possibly SHORTER
+ // one. Once the carried rung outgrew that deadline — 60 s carried into
+ // a 30 s budget — this gate fired on the very first iteration and the
+ // request answered in 8 ms having re-issued NOTHING. Measured at the
+ // upstream socket: with `API_TIMEOUT_MS=60000`, request 2 of an outage
+ // made one connect and was done, permanently, because the 400 it
+ // answered is not a status Claude Code re-asks. Invisible at the 270 s
+ // default, where the terminal 60 s rung always fits; it bites every
+ // `API_TIMEOUT_MS ≤ 90 s`, a value the docs invite.
+ //
+ // So a request that has not yet re-issued once spends what it has
+ // instead of handing back an empty turn: the wait shrinks to the
+ // budget, one attempt is made, and only then does it hand off. It is
+ // not a hammer — the floor is `MIN_ATTEMPT_SLOT_MS`, it happens at most
+ // once per request, and the gap between two client re-asks is the
+ // client's own backoff (up to 38.4 s), not ours.
+ const mayTruncate = requestRetries === 0 && waitBudget >= MIN_ATTEMPT_SLOT_MS;
+ if (!mayTruncate) {
+ handle.handoff();
+ return { kind: "exhausted", conn, error, endpoint, ...tally() };
+ }
+ delay = waitBudget;
}
let waited: { kind: "attempt" } | { kind: "gave_up" };
@@ -427,7 +580,7 @@ export async function withConnectionRetry<T>(
requestRetries++;
const value = await untilAborted(op(merged), merged);
handle.recordAttemptResult("ok", true);
- log(
+ logRecovery(
`[Recovery] ${ctx.providerDisplayName} recovered after ${handle.attempts()} attempts ` +
`in ${handle.recoveryMs()}ms (episode ${handle.episodeId})`
);
diff --git a/packages/cli/src/logger.ts b/packages/cli/src/logger.ts
index df61b230..511f1007 100644
--- a/packages/cli/src/logger.ts
+++ b/packages/cli/src/logger.ts
@@ -319,6 +319,36 @@ export function logStderr(message: string): void {
log(message); // always write to debug log
}
+/**
+ * A network-recovery episode's own account of itself: opened, each attempt,
+ * each wait, the outcome.
+ *
+ * ── WHY THIS IS STRUCTURAL OUTPUT AND NOT A DEBUG AID ───────────────────────
+ *
+ * These lines were on `log()`, which writes to two FILES — the `--debug` log
+ * and the always-on structural log under `~/.claudish/logs/`. That satisfies
+ * "it is written down" and fails the thing it was written down for. Recovery
+ * holds an inbound request for up to the derived deadline (~4.5 minutes at the
+ * default), and in a run with no pane — `-p`, CI, a pipe, any machine without
+ * magmux, which is EVERY headless run — nothing at all reaches the user while
+ * it does. A silent multi-minute hold with the reason legible nowhere is the
+ * state this feature's own design calls "strictly worse than the bug it exists
+ * to remove"; a file the user does not know to open is not a surface.
+ *
+ * The noise objection is real and is already answered by the channel: in an
+ * interactive session `logStderr` routes to `DiagOutput` (a file) rather than
+ * to the client's TUI, and in quiet mode it is suppressed. So the lines land on
+ * a terminal exactly where a terminal is the only surface there is, which is
+ * the case that needed them.
+ *
+ * Fine-grained plumbing — pane connect/disconnect, socket paths, a skipped
+ * ladder — stays on `log()`. This is for the episode's LIFECYCLE only: the six
+ * or so lines from which a reader can reconstruct the ladder.
+ */
+export function logRecovery(message: string): void {
+ logStderr(message);
+}
+
/**
* Set the DiagOutput instance. When set, logStderr() routes to it
* instead of stderr. This replaces the stderrQuiet mechanism for
diff --git a/packages/cli/src/recovery-contract/child-proxy-run.ts b/packages/cli/src/recovery-contract/child-proxy-run.ts
new file mode 100644
index 00000000..2fdb09ef
--- /dev/null
+++ b/packages/cli/src/recovery-contract/child-proxy-run.ts
@@ -0,0 +1,93 @@
+/**
+ * A real proxy in a REAL CHILD PROCESS, for the observations that cannot be
+ * made in-process:
+ *
+ * • which FILE DESCRIPTOR a diagnostic took. An in-process
+ * `process.stderr.write` monkeypatch cannot tell stdout from stderr — it
+ * intercepts the call, not the fd. This project has a terminal-isolation
+ * module precisely because a stray write corrupted a client's TUI.
+ * • whether the process EXITS after a recovery episode, which is invisible to
+ * a test running inside it: a live 60s timer keeps the loop alive.
+ *
+ * Usage (all arguments positional):
+ * bun child-proxy-run.ts <mode> <endpointUrl> <apiTimeoutMs> [abortAfterMs]
+ * mode: "exhaust" | "abort"
+ *
+ * stdout carries exactly one JSON result line, and nothing else may appear
+ * there. Everything diagnostic belongs on stderr.
+ */
+
+import { mkdtempSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { setConfigFileOverride } from "../config-override.js";
+import { createProxyServer } from "../proxy-server.js";
+
+const [mode, endpointUrl, apiTimeoutMs, abortAfterMs] = process.argv.slice(2);
+
+// A distinctive secret: nothing the proxy prints may ever contain it.
+const SECRET_KEY = "sk-blackbox-SECRET-VALUE-9f3c1d";
+
+process.env.API_TIMEOUT_MS = apiTimeoutMs ?? "40000";
+delete process.env.CLAUDISH_RECOVERY;
+
+const dir = mkdtempSync(join(tmpdir(), "recovery-child-"));
+const configPath = join(dir, "config.json");
+writeFileSync(
+ configPath,
+ JSON.stringify({
+ customEndpoints: {
+ "ep-child": {
+ kind: "simple",
+ url: endpointUrl,
+ format: "openai",
+ apiKey: SECRET_KEY,
+ models: ["m-child"],
+ },
+ },
+ }),
+ "utf8"
+);
+setConfigFileOverride(configPath);
+
+const proxy = await createProxyServer(0, undefined, undefined, false, undefined, undefined, {
+ quiet: false,
+});
+
+const body = JSON.stringify({
+ model: "ep-child@m-child",
+ max_tokens: 32,
+ messages: [{ role: "user", content: "hello" }],
+});
+
+const controller = new AbortController();
+if (mode === "abort") {
+ setTimeout(() => controller.abort(), Number(abortAfterMs ?? 7000));
+}
+
+const started = Date.now();
+let status = 0;
+let text = "";
+let aborted = false;
+try {
+ const res = await fetch(`${proxy.url}/v1/messages`, {
+ method: "POST",
+ headers: { "content-type": "application/json" },
+ body,
+ signal: controller.signal,
+ });
+ status = res.status;
+ text = await res.text();
+} catch {
+ aborted = true;
+}
+
+const requestMs = Date.now() - started;
+const shutdownStarted = Date.now();
+await proxy.shutdown();
+const shutdownMs = Date.now() - shutdownStarted;
+
+process.stdout.write(
+ `${JSON.stringify({ mode, status, aborted, requestMs, shutdownMs, bodyPrefix: text.slice(0, 200) })}\n`
+);
+process.exit(0);
diff --git a/packages/cli/src/recovery-contract/fixtures.ts b/packages/cli/src/recovery-contract/fixtures.ts
new file mode 100644
index 00000000..3a94d5ac
--- /dev/null
+++ b/packages/cli/src/recovery-contract/fixtures.ts
@@ -0,0 +1,561 @@
+/**
+ * Black-box fault fixtures for the network-recovery contract tests.
+ *
+ * Written from the requirements + the published contract only. Nothing in this
+ * file imports, names, or reaches into the recovery implementation: every
+ * observation is made either at the PROXY'S OWN URL (a real `fetch`, so the
+ * status asserted is the one that survived any internal remap) or at an
+ * UPSTREAM SOCKET we own (so an attempt is counted because a TCP connection
+ * arrived, never because a log line said so).
+ *
+ * The two ground rules this file exists to enforce:
+ * 1. attempts are counted at the upstream socket, not from `[Recovery]` logs;
+ * 2. statuses are read from `Response.status` of a fetch against `proxy.url`.
+ */
+
+import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { setConfigFileOverride } from "../config-override.js";
+import { createProxyServer } from "../proxy-server.js";
+import type { ProxyServer } from "../types.js";
+
+/** A single upstream connection observed at a fault fixture. */
+export interface ObservedConnection {
+ /** ms since the fixture's `t0` (set when the fixture was created). */
+ at: number;
+ /** The raw request body bytes this connection delivered ("" if it was reset first). */
+ body: string;
+ /** Raw request head (request line + headers), lowercased header names. */
+ headers: Record<string, string>;
+}
+
+export interface RawFixture {
+ readonly port: number;
+ /** Connections in arrival order. */
+ readonly connections: ObservedConnection[];
+ /** ms offsets of each connection relative to the FIRST connection. */
+ offsetsFromFirst(): number[];
+ /** Gaps between consecutive connections, in ms. */
+ gaps(): number[];
+ /** Reset the recorded connections (keeps the server listening). */
+ reset(): void;
+ stop(): void;
+}
+
+export interface RawFixtureOptions {
+ /**
+ * 1-based attempt index from which the fixture answers with a real HTTP 200.
+ * Attempts before it are dropped. `Infinity` (the default) never answers.
+ */
+ serveFromAttempt?: number;
+ /** Text the served SSE answer carries. */
+ sseText?: string;
+ /**
+ * "onOpen" — reset the socket the instant it is accepted (fastest; the
+ * request body is never seen, so attempts are timed but not read).
+ * "afterBody" — read the whole request first, record it, then reset. This is
+ * what makes a body-replay assertion possible.
+ */
+ resetWhen?: "onOpen" | "afterBody";
+ /** Delay (ms) before answering, once the fixture has decided to answer. */
+ respondDelayMs?: number;
+}
+
+/**
+ * An OpenAI-shaped streaming answer. The proxy always talks `stream:true`
+ * upstream (measured against a live fixture), so a non-stream JSON body comes
+ * back with empty content — this is the shape a fixture must produce.
+ */
+export function openAiSse(text: string): string {
+ const chunk = (obj: unknown) => `data: ${JSON.stringify(obj)}\n\n`;
+ return (
+ chunk({
+ id: "chatcmpl-fixture",
+ object: "chat.completion.chunk",
+ created: 1,
+ model: "fixture",
+ choices: [{ index: 0, delta: { role: "assistant", content: text }, finish_reason: null }],
+ }) +
+ chunk({
+ id: "chatcmpl-fixture",
+ object: "chat.completion.chunk",
+ created: 1,
+ model: "fixture",
+ choices: [{ index: 0, delta: {}, finish_reason: "stop" }],
+ usage: { prompt_tokens: 5, completion_tokens: 3, total_tokens: 8 },
+ }) +
+ "data: [DONE]\n\n"
+ );
+}
+
+interface SocketLike {
+ write(data: string): unknown;
+ flush(): unknown;
+ end(): unknown;
+ terminate?: () => void;
+}
+
+/** Write a canned HTTP response after `delayMs`, tolerating a peer that left. */
+function answerLater(sock: SocketLike, body: string, delayMs: number): void {
+ setTimeout(() => {
+ try {
+ sock.write(body);
+ sock.flush();
+ sock.end();
+ } catch {
+ /* peer gone */
+ }
+ }, delayMs);
+}
+
+/**
+ * Parse one complete HTTP/1.1 request out of the bytes so far.
+ * Returns null while the head or the declared body is still incomplete.
+ */
+function parseHttpRequest(buf: Buffer): { headers: Record<string, string>; body: string } | null {
+ const raw = buf.toString("latin1");
+ const headEnd = raw.indexOf("\r\n\r\n");
+ if (headEnd === -1) return null;
+
+ const head = raw.slice(0, headEnd);
+ const headers: Record<string, string> = {};
+ for (const line of head.split("\r\n").slice(1)) {
+ const i = line.indexOf(":");
+ if (i > 0) headers[line.slice(0, i).trim().toLowerCase()] = line.slice(i + 1).trim();
+ }
+
+ const headLen = Buffer.byteLength(head, "latin1") + 4;
+ const bodyBytes = buf.length - headLen;
+ const declared = headers["content-length"];
+ if (declared !== undefined && bodyBytes < Number(declared)) return null;
+ if (declared === undefined && headers["transfer-encoding"]?.includes("chunked")) {
+ if (!raw.endsWith("0\r\n\r\n")) return null;
+ }
+ return { headers, body: buf.subarray(headLen).toString("utf8") };
+}
+
+function httpOk(body: string): string {
+ return [
+ "HTTP/1.1 200 OK",
+ "Content-Type: text/event-stream",
+ "Cache-Control: no-cache",
+ `Content-Length: ${Buffer.byteLength(body)}`,
+ "Connection: close",
+ "",
+ body,
+ ].join("\r\n");
+}
+
+/**
+ * A raw TCP fixture on a FIXED port.
+ *
+ * Every accepted connection is one upstream attempt, timestamped at accept
+ * time. This is the instrument for anything that asserts the ladder's SHAPE:
+ * a refused port cannot count and cannot be slow.
+ */
+export function startRawFixture(port: number, options: RawFixtureOptions = {}): RawFixture {
+ const {
+ serveFromAttempt = Number.POSITIVE_INFINITY,
+ sseText = "RAW-FIXTURE-ANSWER",
+ resetWhen = "onOpen",
+ respondDelayMs = 0,
+ } = options;
+
+ const connections: ObservedConnection[] = [];
+ let t0 = Date.now();
+
+ interface PerSocket {
+ index: number;
+ buf: Buffer;
+ recorded: boolean;
+ }
+
+ const server = Bun.listen<PerSocket>({
+ hostname: "127.0.0.1",
+ port,
+ socket: {
+ open(sock) {
+ const index = connections.length + 1;
+ sock.data = { index, buf: Buffer.alloc(0), recorded: false };
+ if (resetWhen === "onOpen") {
+ connections.push({ at: Date.now() - t0, body: "", headers: {} });
+ if (index >= serveFromAttempt) {
+ answerLater(sock, httpOk(openAiSse(sseText)), respondDelayMs);
+ } else {
+ hardClose(sock);
+ }
+ }
+ },
+ data(sock, chunk) {
+ if (resetWhen === "onOpen") return;
+ const state = sock.data;
+ if (state.recorded) return;
+ state.buf = Buffer.concat([state.buf, chunk]);
+ const parsed = parseHttpRequest(state.buf);
+ if (parsed === null) return;
+
+ state.recorded = true;
+ connections.push({ at: Date.now() - t0, body: parsed.body, headers: parsed.headers });
+ if (state.index >= serveFromAttempt) {
+ answerLater(sock, httpOk(openAiSse(sseText)), respondDelayMs);
+ } else {
+ hardClose(sock);
+ }
+ },
+ close() {},
+ error() {},
+ drain() {},
+ },
+ });
+
+ return {
+ port: server.port,
+ connections,
+ offsetsFromFirst() {
+ const first = connections[0];
+ if (first === undefined) return [];
+ return connections.map((c) => c.at - first.at);
+ },
+ gaps() {
+ const offsets = this.offsetsFromFirst();
+ return offsets.slice(1).map((o, i) => o - (offsets[i] ?? 0));
+ },
+ reset() {
+ connections.length = 0;
+ t0 = Date.now();
+ },
+ stop() {
+ try {
+ server.stop(true);
+ } catch {
+ /* already stopped */
+ }
+ },
+ };
+}
+
+function hardClose(sock: { terminate?: () => void; end: () => void }): void {
+ try {
+ if (typeof sock.terminate === "function") sock.terminate();
+ else sock.end();
+ } catch {
+ /* already gone */
+ }
+}
+
+/** A healthy upstream that records what it received. */
+export interface HealthyFixture {
+ readonly port: number;
+ readonly requests: { at: number; body: string; authorization: string | null; path: string }[];
+ stop(): void;
+}
+
+export function startHealthyFixture(text = "HEALTHY-FIXTURE-ANSWER", port = 0): HealthyFixture {
+ const requests: HealthyFixture["requests"] = [];
+ const t0 = Date.now();
+ const server = Bun.serve({
+ port,
+ async fetch(req) {
+ requests.push({
+ at: Date.now() - t0,
+ body: await req.text(),
+ authorization: req.headers.get("authorization"),
+ path: new URL(req.url).pathname,
+ });
+ return new Response(openAiSse(text), {
+ headers: { "content-type": "text/event-stream" },
+ });
+ },
+ });
+ return {
+ port: server.port as number,
+ requests,
+ stop() {
+ try {
+ server.stop(true);
+ } catch {
+ /* already stopped */
+ }
+ },
+ };
+}
+
+/** An upstream that is reachable and answers a real HTTP error status. */
+export function startStatusFixture(status: number, port = 0) {
+ let hits = 0;
+ const server = Bun.serve({
+ port,
+ fetch() {
+ hits += 1;
+ return new Response(JSON.stringify({ error: { message: "fixture-upstream-error" } }), {
+ status,
+ headers: { "content-type": "application/json" },
+ });
+ },
+ });
+ return {
+ port: server.port as number,
+ get hits() {
+ return hits;
+ },
+ stop() {
+ try {
+ server.stop(true);
+ } catch {
+ /* already stopped */
+ }
+ },
+ };
+}
+
+/**
+ * Reserve `count` ports by binding them all at once and then releasing them.
+ * Binding simultaneously guarantees the numbers differ from each other.
+ *
+ * NEVER port 1: it is privileged, and a recovery fixture has to be able to
+ * bind the same port again later.
+ */
+export function reservePorts(count: number): number[] {
+ const servers = Array.from({ length: count }, () =>
+ Bun.serve({ port: 0, fetch: () => new Response("reserved") })
+ );
+ const ports = servers.map((s) => s.port as number);
+ for (const s of servers) s.stop(true);
+ return ports;
+}
+
+/**
+ * P-1: prove the "refused" fault is really refused, immediately before the run.
+ *
+ * A sibling test that grabbed the port turns a refusal into a live server and
+ * inverts the scenario silently. Fails loudly — never skips.
+ */
+export async function assertRefused(port: number): Promise<void> {
+ try {
+ const res = await fetch(`http://127.0.0.1:${port}/`, {
+ signal: AbortSignal.timeout(1000),
+ });
+ throw new Error(
+ `PRECONDITION P-1 FAILED: 127.0.0.1:${port} answered HTTP ${res.status}; ` +
+ "the refusal fault is not in force (a sibling test probably took the port)."
+ );
+ } catch (err) {
+ const msg = err instanceof Error ? err.message : String(err);
+ if (msg.startsWith("PRECONDITION")) throw err;
+ // Any connect-level rejection is what we wanted.
+ }
+}
+
+export interface EndpointSpec {
+ name: string;
+ port: number;
+ model: string;
+ apiKey?: string;
+ /** Override the URL entirely (for DNS / TEST-NET faults). */
+ url?: string;
+}
+
+export function endpointsConfig(specs: EndpointSpec[]): Record<string, unknown> {
+ const customEndpoints: Record<string, unknown> = {};
+ for (const spec of specs) {
+ customEndpoints[spec.name] = {
+ kind: "simple",
+ url: spec.url ?? `http://127.0.0.1:${spec.port}/v1`,
+ format: "openai",
+ apiKey: spec.apiKey ?? `test-key-${spec.name}`,
+ models: [spec.model],
+ };
+ }
+ return { customEndpoints };
+}
+
+/**
+ * A pool of identical raw fault fixtures, one endpoint each.
+ *
+ * MEASURED, and the reason this exists: a recovery EPISODE is keyed per
+ * endpoint and survives the request that opened it — a second request to the
+ * same dead endpoint continues the ladder at the next rung rather than
+ * restarting at 5s. So two arms of a paired experiment that share an endpoint
+ * are not comparable: the later arm inherits the earlier arm's rung. Every arm
+ * takes a FRESH endpoint from this pool, which also makes the file
+ * order-independent.
+ */
+export interface FixturePool {
+ specs: EndpointSpec[];
+ /** Hand out an endpoint/fixture pair nobody else in this file has used. */
+ next(): { model: string; fixture: RawFixture; name: string };
+ stopAll(): void;
+}
+
+export function createRawFixturePool(
+ count: number,
+ prefix: string,
+ options: RawFixtureOptions = {}
+): FixturePool {
+ const ports = reservePorts(count);
+ const fixtures = ports.map((p) => startRawFixture(p, options));
+ const specs: EndpointSpec[] = ports.map((p, i) => ({
+ name: `${prefix}-${i + 1}`,
+ port: p,
+ model: `m-${i + 1}`,
+ }));
+ let cursor = 0;
+ return {
+ specs,
+ next() {
+ if (cursor >= specs.length) {
+ throw new Error(
+ `fixture pool "${prefix}" exhausted after ${specs.length} arms — ` +
+ "raise the pool size rather than reusing an endpoint (episodes are per-endpoint)."
+ );
+ }
+ const spec = specs[cursor] as EndpointSpec;
+ const fixture = fixtures[cursor] as RawFixture;
+ cursor += 1;
+ return { model: `${spec.name}@${spec.model}`, fixture, name: spec.name };
+ },
+ stopAll() {
+ for (const f of fixtures) f.stop();
+ },
+ };
+}
+
+export interface ProxyHandle {
+ proxy: ProxyServer;
+ url: string;
+ /** Everything the proxy wrote to stderr while this handle was live. */
+ stderr(): string;
+ shutdown(): Promise<void>;
+}
+
+export interface StartProxyOptions {
+ model?: string;
+ modelChain?: string[];
+ quiet?: boolean;
+ captureStderr?: boolean;
+}
+
+/**
+ * Start a real proxy against a temp config file, exactly as `proxy-server.test.ts`
+ * does — the config-file override, never a write to the developer's real config.
+ */
+export async function startProxy(
+ config: Record<string, unknown>,
+ options: StartProxyOptions = {}
+): Promise<ProxyHandle> {
+ const dir = mkdtempSync(join(tmpdir(), "recovery-contract-"));
+ const configPath = join(dir, "config.json");
+ writeFileSync(configPath, JSON.stringify(config), "utf8");
+ setConfigFileOverride(configPath);
+
+ const captured: string[] = [];
+ const realWrite = process.stderr.write.bind(process.stderr);
+ if (options.captureStderr) {
+ process.stderr.write = ((chunk: string | Uint8Array) => {
+ captured.push(typeof chunk === "string" ? chunk : Buffer.from(chunk).toString("utf8"));
+ return true;
+ }) as typeof process.stderr.write;
+ }
+
+ let proxy: ProxyServer;
+ try {
+ proxy = await createProxyServer(0, undefined, options.model, false, undefined, undefined, {
+ quiet: options.quiet ?? true,
+ modelChain: options.modelChain,
+ });
+ } catch (err) {
+ if (options.captureStderr) process.stderr.write = realWrite;
+ setConfigFileOverride(null);
+ rmSync(dir, { recursive: true, force: true });
+ throw err;
+ }
+
+ return {
+ proxy,
+ url: proxy.url,
+ stderr: () => captured.join(""),
+ async shutdown() {
+ if (options.captureStderr) process.stderr.write = realWrite;
+ try {
+ await proxy.shutdown();
+ } finally {
+ setConfigFileOverride(null);
+ rmSync(dir, { recursive: true, force: true });
+ }
+ },
+ };
+}
+
+export interface MessageRequest {
+ model: string;
+ stream?: boolean;
+ headers?: Record<string, string>;
+ content?: unknown;
+ signal?: AbortSignal;
+ maxTokens?: number;
+}
+
+/** POST /v1/messages at the proxy's PUBLIC url, and time it. */
+export async function postMessage(
+ proxyUrl: string,
+ req: MessageRequest
+): Promise<{ status: number; body: string; headers: Headers; elapsedMs: number }> {
+ const started = Date.now();
+ const res = await fetch(`${proxyUrl}/v1/messages`, {
+ method: "POST",
+ headers: { "content-type": "application/json", ...(req.headers ?? {}) },
+ body: JSON.stringify({
+ model: req.model,
+ max_tokens: req.maxTokens ?? 64,
+ ...(req.stream ? { stream: true } : {}),
+ messages: [{ role: "user", content: req.content ?? "hello" }],
+ }),
+ signal: req.signal,
+ });
+ const body = await res.text();
+ return { status: res.status, body, headers: res.headers, elapsedMs: Date.now() - started };
+}
+
+/**
+ * The deadline the contract DERIVES from `API_TIMEOUT_MS`:
+ * min(API_TIMEOUT_MS, 300_000) − 30_000
+ * (docs/advanced/environment.md). Computed here from the published formula so
+ * a test can never hardcode 270_000.
+ */
+export function derivedDeadlineMs(apiTimeoutMs: number): number {
+ return Math.min(apiTimeoutMs, 300_000) - 30_000;
+}
+
+/** The published ladder: 5s → 10s → 30s → 60s → 60s → … (FR-2). */
+export const LADDER_MS = [5_000, 10_000, 30_000, 60_000, 60_000, 60_000] as const;
+
+/**
+ * Attempt start offsets the published ladder puts inside `deadlineMs`,
+ * measured from the first attempt at t=0. Derived, never hardcoded.
+ */
+export function scheduledAttemptOffsets(deadlineMs: number): number[] {
+ const offsets = [0];
+ let t = 0;
+ for (let i = 0; ; i += 1) {
+ const gap = LADDER_MS[Math.min(i, LADDER_MS.length - 1)];
+ t += gap;
+ if (t >= deadlineMs) break;
+ offsets.push(t);
+ }
+ return offsets;
+}
+
+/** Restores env vars a test set, including deleting ones that were absent. */
+export function envSnapshot(names: string[]): () => void {
+ const saved = new Map<string, string | undefined>();
+ for (const n of names) saved.set(n, process.env[n]);
+ return () => {
+ for (const [n, v] of saved) {
+ if (v === undefined) delete process.env[n];
+ else process.env[n] = v;
+ }
+ };
+}
+
+export const sleep = (ms: number) => new Promise<void>((r) => setTimeout(r, ms));
diff --git a/packages/cli/src/recovery-contract/recovery-chain-blackbox.test.ts b/packages/cli/src/recovery-contract/recovery-chain-blackbox.test.ts
new file mode 100644
index 00000000..e812d717
--- /dev/null
+++ b/packages/cli/src/recovery-contract/recovery-chain-blackbox.test.ts
@@ -0,0 +1,178 @@
+/**
+ * CONTRACT POINT 6 — a connection failure must NEVER advance the provider
+ * chain. This is the criterion that separates a fix from a bill: a chain that
+ * advances during an outage can move a subscription user onto metered billing.
+ *
+ * Trap T-12 is the whole design of this file. "No advance" asserted on a chain
+ * that COULD NOT have advanced — one candidate, or a second candidate with no
+ * credential — is vacuous and passes on any build at all. So:
+ * • the chain here is genuinely multi-candidate (two custom endpoints, both
+ * constructible, both credentialled);
+ * • the POSITIVE CONTROL in the same file drives the same shape of chain into
+ * an advance and proves candidate 2 is reachable and would have served;
+ * • "did it advance?" is answered at CANDIDATE 2's OWN SOCKET — a request
+ * count on a server we own — not by grepping the proxy's log for
+ * `[Fallback]`, which is written by the code under test.
+ */
+
+import { afterEach, describe, expect, test } from "bun:test";
+import {
+ type HealthyFixture,
+ type ProxyHandle,
+ type RawFixture,
+ endpointsConfig,
+ envSnapshot,
+ postMessage,
+ reservePorts,
+ startHealthyFixture,
+ startProxy,
+ startRawFixture,
+ startStatusFixture,
+} from "./fixtures.js";
+
+let restoreEnv: (() => void) | undefined;
+let proxy: ProxyHandle | undefined;
+const stoppables: { stop(): void }[] = [];
+
+afterEach(async () => {
+ await proxy?.shutdown();
+ proxy = undefined;
+ while (stoppables.length > 0) stoppables.pop()?.stop();
+ restoreEnv?.();
+ restoreEnv = undefined;
+});
+
+const SECOND = "SECOND-CANDIDATE-ANSWER";
+
+function setEnv(apiTimeoutMs: number): void {
+ restoreEnv = envSnapshot(["API_TIMEOUT_MS", "CLAUDISH_RECOVERY"]);
+ process.env.API_TIMEOUT_MS = String(apiTimeoutMs);
+ delete process.env.CLAUDISH_RECOVERY;
+}
+
+async function startChain(opts: {
+ firstUrl?: string;
+ firstPort?: number;
+ secondPort: number;
+}): Promise<ProxyHandle> {
+ const config = endpointsConfig([
+ {
+ name: "ep-one",
+ port: opts.firstPort ?? 0,
+ model: "m-one",
+ ...(opts.firstUrl ? { url: opts.firstUrl } : {}),
+ },
+ { name: "ep-two", port: opts.secondPort, model: "m-two" },
+ ]);
+ const handle = await startProxy(config, {
+ model: "ep-one@m-one",
+ modelChain: ["ep-one@m-one", "ep-two@m-two"],
+ quiet: false,
+ captureStderr: true,
+ });
+ proxy = handle;
+ return handle;
+}
+
+describe("CP-6 — a connection failure never switches providers", () => {
+ test("POSITIVE CONTROL: the same chain DOES advance on a retryable upstream status", async () => {
+ setEnv(40_000);
+ const [firstPort, secondPort] = reservePorts(2);
+ const first = startStatusFixture(429, firstPort);
+ const second: HealthyFixture = startHealthyFixture(SECOND, secondPort);
+ stoppables.push(first, second);
+
+ const handle = await startChain({ firstPort, secondPort });
+ const result = await postMessage(handle.url, { model: "ep-one@m-one" });
+
+ expect(result.status).toBe(200);
+ expect(result.body).toContain(SECOND);
+ // Candidate 2 was reachable, credentialled and willing — asserted at its
+ // own socket. Without this, the no-advance test below is vacuous.
+ expect(second.requests.length).toBe(1);
+ expect(first.hits).toBeGreaterThanOrEqual(1);
+ expect(handle.stderr()).toContain("trying next provider");
+ }, 60_000);
+
+ test("an unreachable candidate 1 retries in place and candidate 2 is never contacted", async () => {
+ setEnv(40_000);
+ const [firstPort, secondPort] = reservePorts(2);
+ const first: RawFixture = startRawFixture(firstPort);
+ const second: HealthyFixture = startHealthyFixture(SECOND, secondPort);
+ stoppables.push(first, second);
+
+ const handle = await startChain({ firstPort, secondPort });
+ const result = await postMessage(handle.url, { model: "ep-one@m-one" });
+
+ // The ladder ran on candidate 1 …
+ expect(first.connections.length).toBeGreaterThanOrEqual(2);
+ // … and candidate 2 never saw a byte. Counted at candidate 2's socket.
+ expect(second.requests.length).toBe(0);
+ expect(result.body).not.toContain(SECOND);
+ expect(JSON.parse(result.body).error.type).toBe("connection_error");
+ expect(handle.stderr()).not.toContain("trying next provider");
+ }, 60_000);
+
+ test("CONTROL: a neutral DNS failure retries in place and does not advance", async () => {
+ // Isolates the variable for the wording test below: the fault kind is the
+ // same (name never resolves), only the words in it differ.
+ setEnv(40_000);
+ const [secondPort] = reservePorts(1);
+ const second: HealthyFixture = startHealthyFixture(SECOND, secondPort);
+ stoppables.push(second);
+
+ const handle = await startChain({
+ firstUrl: "https://no-such-host-blackbox.invalid/v1",
+ secondPort,
+ });
+ const result = await postMessage(handle.url, { model: "ep-one@m-one" });
+
+ expect(second.requests.length).toBe(0);
+ expect(JSON.parse(result.body).error.type).toBe("connection_error");
+ expect(result.elapsedMs).toBeGreaterThan(4_000); // the ladder really ran
+ }, 60_000);
+
+ test("a connection error whose HOST reads like a billing failure still does not advance", async () => {
+ // The wording trap, induced from outside: the error text genuinely
+ // contains "quota" because the unreachable host is called that. A
+ // classifier that matches on message text rather than on the failure's
+ // kind advances the chain here and nowhere else.
+ setEnv(40_000);
+ const [secondPort] = reservePorts(1);
+ const second: HealthyFixture = startHealthyFixture(SECOND, secondPort);
+ stoppables.push(second);
+
+ // MEASURED while writing this file: of `quota`, `insufficient-credits`
+ // and `rate-limit`, only the substring "quota" flips the outcome — the
+ // other two behave exactly like the neutral control above. So the
+ // variable really is the WORD, not the fault.
+ const badHost = "quota-exceeded-insufficient-credits.invalid";
+ // P-2: a hijacking resolver would turn this into an ordinary HTTP
+ // failure and silently invert the scenario. Fail loudly, never skip.
+ let resolved = false;
+ try {
+ await fetch(`https://${badHost}/v1`, { signal: AbortSignal.timeout(3_000) });
+ resolved = true;
+ } catch {
+ resolved = false;
+ }
+ expect(
+ resolved,
+ `PRECONDITION P-2 FAILED: ${badHost} answered. This machine's resolver hijacks .invalid, ` +
+ "so the DNS fault is not in force."
+ ).toBe(false);
+
+ const handle = await startChain({
+ firstUrl: `https://${badHost}/v1`,
+ secondPort,
+ });
+ const result = await postMessage(handle.url, { model: "ep-one@m-one" });
+
+ expect(second.requests.length).toBe(0);
+ expect(result.body).not.toContain(SECOND);
+ expect(handle.stderr()).not.toContain("trying next provider");
+ const err = JSON.parse(result.body).error;
+ expect(err.type).toBe("connection_error");
+ expect(err.message.toLowerCase()).not.toContain("billing");
+ }, 60_000);
+});
diff --git a/packages/cli/src/recovery-contract/recovery-completes-blackbox.test.ts b/packages/cli/src/recovery-contract/recovery-completes-blackbox.test.ts
new file mode 100644
index 00000000..e40d86c1
--- /dev/null
+++ b/packages/cli/src/recovery-contract/recovery-completes-blackbox.test.ts
@@ -0,0 +1,172 @@
+/**
+ * CONTRACT POINT 3 — when an attempt succeeds mid-ladder, the request completes
+ * normally.
+ *
+ * REAL TIME, deliberately (trap T-10): a fake clock proves the policy and hides
+ * the mechanism. A body that was consumed by attempt 1 and replayed empty, a
+ * credential dropped on the retry, a timer that fires after the response was
+ * already written — none of those are visible without real sockets and real
+ * timers.
+ *
+ * Trap T-3 is the one this file exists for: every "it retried at 5s/10s/30s"
+ * assertion in the suite passes on a feature that can never actually finish.
+ */
+
+import { afterAll, beforeAll, describe, expect, test } from "bun:test";
+import {
+ type ProxyHandle,
+ type RawFixture,
+ endpointsConfig,
+ envSnapshot,
+ postMessage,
+ reservePorts,
+ startProxy,
+ startRawFixture,
+} from "./fixtures.js";
+
+const API_TIMEOUT_MS = 60_000; // ⇒ 30s deadline ⇒ attempts at 0s, 5s, 15s
+
+let restoreEnv: () => void;
+let proxy: ProxyHandle;
+const fixtures: RawFixture[] = [];
+
+/** One endpoint per scenario: an episode is per-endpoint and outlives its request. */
+const SCENARIOS = [
+ "recovers", // resets attempt 1, serves attempt 2
+ "body", // same, with a large distinctive body
+ "stream", // same, client asked for stream:true
+ "never", // never serves — the exhaustion arm
+] as const;
+
+const ports: Record<string, number> = {};
+const fixtureOf: Record<string, RawFixture> = {};
+
+beforeAll(async () => {
+ restoreEnv = envSnapshot(["API_TIMEOUT_MS", "CLAUDISH_RECOVERY"]);
+ process.env.API_TIMEOUT_MS = String(API_TIMEOUT_MS);
+ delete process.env.CLAUDISH_RECOVERY;
+
+ const reserved = reservePorts(SCENARIOS.length);
+ SCENARIOS.forEach((name, i) => {
+ ports[name] = reserved[i];
+ const fx = startRawFixture(reserved[i], {
+ resetWhen: "afterBody",
+ serveFromAttempt: name === "never" ? Number.POSITIVE_INFINITY : 2,
+ sseText: `RECOVERED-ANSWER-${name.toUpperCase()}`,
+ });
+ fixtures.push(fx);
+ fixtureOf[name] = fx;
+ });
+
+ proxy = await startProxy(
+ endpointsConfig(
+ SCENARIOS.map((name) => ({ name: `ep-${name}`, port: ports[name], model: `m-${name}` }))
+ )
+ );
+}, 60_000);
+
+afterAll(async () => {
+ await proxy?.shutdown();
+ for (const f of fixtures) f.stop();
+ restoreEnv?.();
+});
+
+describe("CP-3 — recovery completes the turn", () => {
+ test("the upstream comes back mid-ladder and the client gets 200 with the real answer", async () => {
+ const fixture = fixtureOf.recovers;
+ const result = await postMessage(proxy.url, { model: "ep-recovers@m-recovers" });
+
+ expect(result.status).toBe(200);
+ expect(result.body).toContain("RECOVERED-ANSWER-RECOVERS");
+ // It really did fail first: ≥2 upstream connections, and the answer
+ // arrived no earlier than the second rung.
+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);
+ expect(result.elapsedMs).toBeGreaterThan(4_000);
+
+ const parsed = JSON.parse(result.body);
+ expect(parsed.type).toBe("message");
+ expect(parsed.content[0].text).toContain("RECOVERED-ANSWER-RECOVERS");
+ }, 60_000);
+
+ test("the request BODY is replayed intact on the successful attempt", async () => {
+ // ~200 KB of distinctive payload plus a tool_result block. A body consumed
+ // by attempt 1 and replayed empty satisfies every timing assertion in this
+ // suite and delivers garbage to the model.
+ const filler = "REPLAY-MARKER-".repeat(14_000); // ≈ 196 KB
+ const content = [
+ { type: "text", text: `head ${filler} tail` },
+ {
+ type: "tool_result",
+ tool_use_id: "toolu_blackbox_1",
+ content: "TOOL-RESULT-SENTINEL",
+ },
+ ];
+
+ const fixture = fixtureOf.body;
+ const result = await postMessage(proxy.url, {
+ model: "ep-body@m-body",
+ content,
+ });
+
+ expect(result.status).toBe(200);
+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);
+
+ const first = fixture.connections[0].body;
+ const success = fixture.connections[fixture.connections.length - 1].body;
+
+ expect(first.length).toBeGreaterThan(150_000);
+ expect(success.length).toBeGreaterThan(150_000);
+ expect(success).toContain("TOOL-RESULT-SENTINEL");
+ expect(success.split("REPLAY-MARKER-").length - 1).toBe(14_000);
+ // The retry must send the SAME request, not a truncated or re-encoded one.
+ expect(success).toBe(first);
+ }, 90_000);
+
+ test("the credential travels with the retry", async () => {
+ // A retry that drops the Authorization header turns an outage into a 401
+ // storm the moment the network returns.
+ const fixture = fixtureOf.body;
+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);
+ const first = fixture.connections[0].headers.authorization;
+ const success = fixture.connections[fixture.connections.length - 1].headers.authorization;
+ expect(first).toBe("Bearer test-key-ep-body");
+ expect(success).toBe(first);
+ }, 20_000);
+
+ test("a recovered streaming turn is a well-formed Claude SSE stream", async () => {
+ const fixture = fixtureOf.stream;
+ const result = await postMessage(proxy.url, {
+ model: "ep-stream@m-stream",
+ stream: true,
+ });
+
+ expect(result.status).toBe(200);
+ expect(result.headers.get("content-type")).toContain("text/event-stream");
+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);
+
+ const events = result.body
+ .split("\n")
+ .filter((l) => l.startsWith("event: "))
+ .map((l) => l.slice(7).trim());
+
+ expect(events.filter((e) => e === "message_start").length).toBe(1);
+ expect(events).toContain("message_stop");
+ expect(result.body).toContain("RECOVERED-ANSWER-STREAM");
+ // Nothing may precede message_start that a strict SSE client would choke
+ // on: the first event of the stream is the message_start itself.
+ expect(events[0]).toBe("message_start");
+ }, 60_000);
+
+ test("an upstream that never returns produces exactly one exhaustion answer", async () => {
+ const fixture = fixtureOf.never;
+ const result = await postMessage(proxy.url, { model: "ep-never@m-never" });
+
+ expect(result.status).toBeGreaterThanOrEqual(400);
+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);
+
+ const attemptsAtAnswer = fixture.connections.length;
+ // No late attempt and no second response after the client was answered.
+ await new Promise((r) => setTimeout(r, 3_000));
+ expect(fixture.connections.length).toBe(attemptsAtAnswer);
+ }, 90_000);
+});
diff --git a/packages/cli/src/recovery-contract/recovery-deadline-blackbox.test.ts b/packages/cli/src/recovery-contract/recovery-deadline-blackbox.test.ts
new file mode 100644
index 00000000..e5eac1b6
--- /dev/null
+++ b/packages/cli/src/recovery-contract/recovery-deadline-blackbox.test.ts
@@ -0,0 +1,160 @@
+/**
+ * CONTRACT POINT 2 — the recovery deadline is DERIVED from `API_TIMEOUT_MS`,
+ * never a constant that happens to equal the default.
+ *
+ * `min(API_TIMEOUT_MS, 300s) − 30s` (docs/advanced/environment.md). The
+ * expected attempt schedule is COMPUTED from that formula and the published
+ * ladder (FR-2, 5/10/30/60…), so no attempt count and no handoff time is
+ * hardcoded anywhere in this file.
+ *
+ * Two traps live here, and both need more than one value in ONE process:
+ * T-4 a deadline that is a constant equal to the default passes any
+ * single-value run;
+ * T-5 an env var read once at module load passes whichever value ran first.
+ * So this file runs 40s → 60s → 40s and asserts each on its own merits; a
+ * module-load snapshot yields the same schedule three times and fails twice.
+ */
+
+import { afterEach, describe, expect, test } from "bun:test";
+import {
+ type ProxyHandle,
+ createRawFixturePool,
+ derivedDeadlineMs,
+ endpointsConfig,
+ envSnapshot,
+ postMessage,
+ scheduledAttemptOffsets,
+ sleep,
+ startProxy,
+} from "./fixtures.js";
+
+let restoreEnv: (() => void) | undefined;
+let liveProxy: ProxyHandle | undefined;
+let livePool: ReturnType<typeof createRawFixturePool> | undefined;
+
+afterEach(async () => {
+ await liveProxy?.shutdown();
+ liveProxy = undefined;
+ livePool?.stopAll();
+ livePool = undefined;
+ restoreEnv?.();
+ restoreEnv = undefined;
+});
+
+interface LadderRun {
+ offsets: number[];
+ gaps: number[];
+ status: number;
+ elapsedMs: number;
+ body: string;
+}
+
+/**
+ * Run one full ladder to exhaustion under `apiTimeoutMs`.
+ *
+ * A fresh proxy per value, so the test does not accidentally pin WHEN the
+ * implementation is allowed to read the variable — only that the value it
+ * ends up using is the one the environment stated.
+ */
+async function runLadderUnder(apiTimeoutMs: number): Promise<LadderRun> {
+ restoreEnv = envSnapshot(["API_TIMEOUT_MS", "CLAUDISH_RECOVERY"]);
+ process.env.API_TIMEOUT_MS = String(apiTimeoutMs);
+ delete process.env.CLAUDISH_RECOVERY;
+
+ const pool = createRawFixturePool(1, "ep-deadline");
+ livePool = pool;
+ const proxy = await startProxy(endpointsConfig(pool.specs));
+ liveProxy = proxy;
+
+ const { model, fixture } = pool.next();
+ const result = await postMessage(proxy.url, { model });
+ return {
+ offsets: fixture.offsetsFromFirst(),
+ gaps: fixture.gaps(),
+ status: result.status,
+ elapsedMs: result.elapsedMs,
+ body: result.body,
+ };
+}
+
+function expectSchedule(run: LadderRun, apiTimeoutMs: number): void {
+ const deadline = derivedDeadlineMs(apiTimeoutMs);
+ const expected = scheduledAttemptOffsets(deadline);
+
+ expect(run.offsets.length).toBe(expected.length);
+ for (const [i, want] of expected.entries()) {
+ expect(Math.abs(run.offsets[i] - want)).toBeLessThan(1_500);
+ }
+ // The published ladder, read off the wire: 5s then 10s then 30s …
+ for (const [i, gap] of run.gaps.entries()) {
+ const want = expected[i + 1] - expected[i];
+ expect(Math.abs(gap - want)).toBeLessThan(1_000);
+ }
+ // Nothing may start after the derived deadline, and the client must be
+ // answered inside it (measured from inbound request start).
+ expect(Math.max(...run.offsets)).toBeLessThan(deadline);
+ expect(run.elapsedMs).toBeLessThan(deadline + 5_000);
+}
+
+describe("CP-2 — the deadline derives from API_TIMEOUT_MS", () => {
+ test("API_TIMEOUT_MS=40000 ⇒ the schedule the derived 10s deadline admits", async () => {
+ const run = await runLadderUnder(40_000);
+ expectSchedule(run, 40_000);
+ expect(run.status).toBe(400);
+ }, 60_000);
+
+ test("API_TIMEOUT_MS=60000 ⇒ a LONGER schedule in the same process (anti-hardcode)", async () => {
+ const run = await runLadderUnder(60_000);
+ expectSchedule(run, 60_000);
+ // The point of the pairing: this run must contain strictly more attempts
+ // than the 40s run above. A constant 270s deadline never gets here (the
+ // per-test timeout kills it); a constant anything gives both runs the
+ // same count.
+ expect(run.offsets.length).toBeGreaterThan(
+ scheduledAttemptOffsets(derivedDeadlineMs(40_000)).length
+ );
+ }, 90_000);
+
+ test("back to API_TIMEOUT_MS=40000 in the same process ⇒ the short schedule returns", async () => {
+ // T-5: an implementation that snapshots the env at module load passes the
+ // first of these three and fails whichever ran second and third.
+ const run = await runLadderUnder(40_000);
+ expectSchedule(run, 40_000);
+ }, 60_000);
+
+ test("a garbled API_TIMEOUT_MS falls back to a working deadline rather than NaN", async () => {
+ restoreEnv = envSnapshot(["API_TIMEOUT_MS", "CLAUDISH_RECOVERY"]);
+ process.env.API_TIMEOUT_MS = "abc";
+ delete process.env.CLAUDISH_RECOVERY;
+
+ const pool = createRawFixturePool(1, "ep-garbled");
+ livePool = pool;
+ const proxy = await startProxy(endpointsConfig(pool.specs));
+ liveProxy = proxy;
+
+ const { model, fixture } = pool.next();
+ const controller = new AbortController();
+ const pending = postMessage(proxy.url, { model, signal: controller.signal }).catch(() => null);
+
+ // A NaN deadline collapses to "no budget" and answers instantly; a
+ // negative one does the same. Either way the second rung never arrives.
+ const started = Date.now();
+ while (Date.now() - started < 12_000 && fixture.connections.length < 2) await sleep(50);
+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);
+ expect(Math.abs(fixture.gaps()[0] - 5_000)).toBeLessThan(1_000);
+ controller.abort();
+ await pending;
+ }, 40_000);
+
+ test("an API_TIMEOUT_MS below one rung still answers, bounded, without crashing", async () => {
+ // A-1: the contract states no floor for `min(API_TIMEOUT_MS,300s)−30s`,
+ // so the only thing assertable here is that a sub-rung deadline neither
+ // hangs nor produces a negative budget. The floor itself stays untestable
+ // until it is pinned.
+ const run = await runLadderUnder(31_000);
+ expect(run.status).toBe(400);
+ expect(JSON.parse(run.body).error.type).toBe("connection_error");
+ expect(run.offsets.length).toBeGreaterThanOrEqual(1);
+ expect(run.elapsedMs).toBeLessThan(30_000);
+ }, 60_000);
+});
diff --git a/packages/cli/src/recovery-contract/recovery-disconnect-blackbox.test.ts b/packages/cli/src/recovery-contract/recovery-disconnect-blackbox.test.ts
new file mode 100644
index 00000000..dda910e1
--- /dev/null
+++ b/packages/cli/src/recovery-contract/recovery-disconnect-blackbox.test.ts
@@ -0,0 +1,147 @@
+/**
+ * CONTRACT POINT 7 — a client disconnect stops the retry promptly.
+ *
+ * REAL TIME ONLY. A fake clock cannot see a timer that was never cancelled,
+ * because the timer only fires when time really passes.
+ *
+ * Trap T-11 is the one this file is built around: aborting BEFORE any attempt
+ * has happened makes "it stopped retrying" trivially true. So the abort here
+ * happens only after the fixture has observed ≥2 attempts, and it lands inside
+ * a KNOWN wait rung — the third attempt is due at a time we can name, and the
+ * assertion is that it never arrives.
+ */
+
+import { afterAll, beforeAll, describe, expect, test } from "bun:test";
+import {
+ type FixturePool,
+ type ProxyHandle,
+ type RawFixture,
+ createRawFixturePool,
+ endpointsConfig,
+ envSnapshot,
+ postMessage,
+ sleep,
+ startProxy,
+} from "./fixtures.js";
+
+const API_TIMEOUT_MS = 60_000; // ⇒ 30s deadline ⇒ attempts at 0s, 5s, 15s
+
+let restoreEnv: () => void;
+let proxy: ProxyHandle;
+let pool: FixturePool;
+
+beforeAll(async () => {
+ restoreEnv = envSnapshot(["API_TIMEOUT_MS", "CLAUDISH_RECOVERY"]);
+ process.env.API_TIMEOUT_MS = String(API_TIMEOUT_MS);
+ delete process.env.CLAUDISH_RECOVERY;
+
+ pool = createRawFixturePool(6, "ep-abort");
+ proxy = await startProxy(endpointsConfig(pool.specs));
+}, 60_000);
+
+afterAll(async () => {
+ await proxy?.shutdown();
+ pool?.stopAll();
+ restoreEnv?.();
+});
+
+async function waitForAttempts(fixture: RawFixture, n: number, budgetMs: number): Promise<void> {
+ const started = Date.now();
+ while (Date.now() - started < budgetMs) {
+ if (fixture.connections.length >= n) return;
+ await sleep(25);
+ }
+ throw new Error(
+ `fixture saw only ${fixture.connections.length} attempts in ${budgetMs}ms; expected ${n}`
+ );
+}
+
+describe("CP-7 — client disconnect stops the ladder", () => {
+ test("no further upstream attempt arrives after the client goes away mid-rung", async () => {
+ const { model, fixture } = pool.next();
+ const controller = new AbortController();
+ const pending = postMessage(proxy.url, { model, signal: controller.signal }).catch(
+ (e: unknown) => e
+ );
+
+ // Let the ladder demonstrably run: attempts at t≈0 and t≈5s.
+ await waitForAttempts(fixture, 2, 12_000);
+ const attemptsAtAbort = fixture.connections.length;
+ expect(attemptsAtAbort).toBeGreaterThanOrEqual(2);
+
+ // Abort INSIDE the 10s wait rung. The next attempt is due at t≈15s.
+ await sleep(1_500);
+ controller.abort();
+ await pending;
+
+ // Watch well past the moment attempt 3 was due.
+ await sleep(12_000);
+ expect(fixture.connections.length).toBe(attemptsAtAbort);
+ }, 60_000);
+
+ test("NEGATIVE CONTROL: with nobody aborting, the third attempt does arrive", async () => {
+ // Without this, the assertion above ("no new connection after the abort")
+ // is satisfied by any build that simply stops after two attempts — the
+ // abort would be doing none of the work. Same fixture shape, same rung,
+ // the only difference is that nothing aborts.
+ const { model, fixture } = pool.next();
+ const controller = new AbortController();
+ const pending = postMessage(proxy.url, { model, signal: controller.signal }).catch(
+ (e: unknown) => e
+ );
+ await waitForAttempts(fixture, 3, 25_000);
+ expect(fixture.connections.length).toBeGreaterThanOrEqual(3);
+ controller.abort();
+ await pending;
+ }, 45_000);
+
+ test("aborting one request does not stop another request's ladder", async () => {
+ // Guards a single shared coordinator: A's abort must not cancel B.
+ const a = pool.next();
+ const b = pool.next();
+
+ const controllerA = new AbortController();
+ const pendingA = postMessage(proxy.url, {
+ model: a.model,
+ signal: controllerA.signal,
+ }).catch((e: unknown) => e);
+ const pendingB = postMessage(proxy.url, { model: b.model });
+
+ await waitForAttempts(a.fixture, 2, 12_000);
+ await waitForAttempts(b.fixture, 2, 12_000);
+
+ const aAtAbort = a.fixture.connections.length;
+ controllerA.abort();
+ await pendingA;
+
+ const resultB = await pendingB;
+ expect(resultB.status).toBeGreaterThanOrEqual(400);
+ // B ran its full schedule …
+ expect(b.fixture.connections.length).toBeGreaterThanOrEqual(3);
+ // … while A stopped where it was.
+ expect(a.fixture.connections.length).toBe(aAtAbort);
+ }, 90_000);
+
+ test("shutdown after an aborted ladder resolves promptly", async () => {
+ // NFR-2: a live 60s timer keeps the loop alive and is invisible from
+ // inside the request path.
+ const ownPool = createRawFixturePool(1, "ep-shutdown");
+ const ownProxy = await startProxy(endpointsConfig(ownPool.specs));
+ try {
+ const { model, fixture } = ownPool.next();
+ const controller = new AbortController();
+ const pending = postMessage(ownProxy.url, { model, signal: controller.signal }).catch(
+ (e: unknown) => e
+ );
+ await waitForAttempts(fixture, 2, 12_000);
+ controller.abort();
+ await pending;
+
+ const t0 = Date.now();
+ await ownProxy.shutdown();
+ expect(Date.now() - t0).toBeLessThan(1_000);
+ } finally {
+ ownPool.stopAll();
+ }
+ }, 60_000);
+});
diff --git a/packages/cli/src/recovery-contract/recovery-exhaustion-blackbox.test.ts b/packages/cli/src/recovery-contract/recovery-exhaustion-blackbox.test.ts
new file mode 100644
index 00000000..5c883b30
--- /dev/null
+++ b/packages/cli/src/recovery-contract/recovery-exhaustion-blackbox.test.ts
@@ -0,0 +1,140 @@
+/**
+ * CONTRACT POINT 4 — at exhaustion the client gets 503 when a recovery banner
+ * is confirmed on screen, and 400 when it is not.
+ *
+ * ONLY THE NEGATIVE ARM IS HERE, and that is a finding rather than an omission:
+ * "a banner is confirmed on screen" has no published way to be SET from outside
+ * the proxy independently of the switch that also enables the ladder, so the
+ * 503 arm and the 400 arm cannot both be produced in one experiment (trap
+ * T-13). See test-results.md — it needs a documented pane-attach seam or a PTY
+ * end-to-end run.
+ *
+ * Every status below is read from `Response.status` of a fetch against the
+ * proxy's own URL. This project's documented failure mode is that a terminal
+ * error is remapped to 400 on the way out, so a status asserted anywhere
+ * upstream of the socket is measuring a different thing than the user gets
+ * (trap T-6).
+ */
+
+import { afterAll, beforeAll, describe, expect, test } from "bun:test";
+import {
+ type FixturePool,
+ type ProxyHandle,
+ createRawFixturePool,
+ endpointsConfig,
+ envSnapshot,
+ postMessage,
+ reservePorts,
+ startProxy,
+ startRawFixture,
+ startStatusFixture,
+} from "./fixtures.js";
+
+const API_TIMEOUT_MS = 40_000; // ⇒ 10s deadline ⇒ attempts at 0s, 5s
+
+let restoreEnv: () => void;
+let proxy: ProxyHandle;
+let pool: FixturePool;
+
+beforeAll(async () => {
+ restoreEnv = envSnapshot(["API_TIMEOUT_MS", "CLAUDISH_RECOVERY", "CLAUDISH_RECOVERY_UI"]);
+ process.env.API_TIMEOUT_MS = String(API_TIMEOUT_MS);
+ delete process.env.CLAUDISH_RECOVERY;
+ delete process.env.CLAUDISH_RECOVERY_UI; // UI on by default; no pane can attach here
+
+ pool = createRawFixturePool(3, "ep-exhaust");
+ proxy = await startProxy(endpointsConfig(pool.specs));
+}, 60_000);
+
+afterAll(async () => {
+ await proxy?.shutdown();
+ pool?.stopAll();
+ restoreEnv?.();
+});
+
+describe("CP-4 — exhaustion with no banner on screen", () => {
+ test("answers 400 connection_error naming provider, host and reason, with no recovery headers", async () => {
+ const { model, fixture, name } = pool.next();
+ const result = await postMessage(proxy.url, { model });
+
+ // The ladder really ran first (otherwise this is just the off-switch).
+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);
+
+ expect(result.status).toBe(400);
+ const err = JSON.parse(result.body).error;
+ expect(err.type).toBe("connection_error");
+ // provider, host and the reason, in the words the product already uses.
+ expect(err.message.toLowerCase()).toContain(name.toLowerCase());
+ expect(err.message).toContain(`127.0.0.1:${fixture.port}`);
+ expect(err.message.length).toBeGreaterThan(20);
+
+ // A 503 handed back for retry carries these; a 400 must not.
+ expect(result.headers.get("x-claudish-recovery")).toBeNull();
+ expect(result.headers.get("x-should-retry")).toBeNull();
+ }, 60_000);
+
+ test("the streaming path leaves the client something it can act on", async () => {
+ // A-6: with headers already committed a 503 is unsendable, so the
+ // contract is satisfiable two ways. Both are acceptable; a half-written
+ // stream with no terminal error event is not — that is the shape that
+ // silently truncates a turn.
+ const { model, fixture } = pool.next();
+ const result = await postMessage(proxy.url, { model, stream: true });
+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);
+
+ const actionable =
+ result.status >= 400 ||
+ (result.status === 200 && /event:\s*error/.test(result.body)) ||
+ /"type"\s*:\s*"error"/.test(result.body);
+ expect(
+ actionable,
+ `streaming exhaustion produced status ${result.status} with body ${result.body.slice(0, 300)}`
+ ).toBe(true);
+ }, 60_000);
+
+ test("a recovery exhaustion survives an earlier candidate's auth failure in the chain", async () => {
+ // C-14: an earlier candidate's non-connection failure must not turn the
+ // recovery outcome into a combined terminal error that hides the reason
+ // — the client must still be told the network is down, not that it is
+ // unauthenticated.
+ const [authPort] = reservePorts(1);
+ const auth401 = startStatusFixture(401, authPort);
+ const [deadPort] = reservePorts(1);
+ const dead = startRawFixture(deadPort);
+ let chainProxy: ProxyHandle | undefined;
+ try {
+ chainProxy = await startProxy(
+ endpointsConfig([
+ { name: "ep-auth", port: authPort, model: "m-auth" },
+ { name: "ep-down", port: deadPort, model: "m-down" },
+ ]),
+ {
+ model: "ep-auth@m-auth",
+ modelChain: ["ep-auth@m-auth", "ep-down@m-down"],
+ }
+ );
+
+ const result = await postMessage(chainProxy.url, { model: "ep-auth@m-auth" });
+
+ expect(auth401.hits).toBeGreaterThanOrEqual(1);
+ expect(dead.connections.length).toBeGreaterThanOrEqual(2); // candidate 2 laddered
+ const err = JSON.parse(result.body).error;
+ // The contract pins the STATUS for this case (C-14), not the error
+ // type, so the type is asserted only negatively: the user must not be
+ // told they have an auth problem while the network is down.
+ expect(result.status).not.toBe(401);
+ expect(result.status).toBeGreaterThanOrEqual(400);
+ expect(err.type).not.toBe("authentication_error");
+ // The network reason survives into the body the client can read. On
+ // the chain path it lands in `error.attempts[]` rather than
+ // `error.message`, so assert against the body the client receives
+ // rather than against one field of it.
+ expect(result.body).toContain("Cannot reach");
+ expect(result.body).toContain(`127.0.0.1:${dead.port}`);
+ } finally {
+ await chainProxy?.shutdown();
+ auth401.stop();
+ dead.stop();
+ }
+ }, 90_000);
+});
diff --git a/packages/cli/src/recovery-contract/recovery-ladder-blackbox.test.ts b/packages/cli/src/recovery-contract/recovery-ladder-blackbox.test.ts
new file mode 100644
index 00000000..d372d72c
--- /dev/null
+++ b/packages/cli/src/recovery-contract/recovery-ladder-blackbox.test.ts
@@ -0,0 +1,285 @@
+/**
+ * CONTRACT POINT 1 (a connection failure no longer answers immediately),
+ * CONTRACT POINT 5 (`x-claudish-no-recovery` skips the ladder) and the
+ * env-var arm of CONTRACT POINT 8.
+ *
+ * BLACK BOX. Every attempt is counted at an upstream socket we own; every
+ * status is read from `Response.status` of a fetch against the proxy's own URL,
+ * i.e. AFTER any internal status remap.
+ *
+ * Trap T-7 is the reason this file is shaped the way it is: an "immediate 400"
+ * assertion passes just as well on a build where the whole feature is off. So
+ * every fast-fail scenario here is PAIRED with an enabled control run against
+ * an identical fixture in the same process, and the control asserts a second
+ * upstream CONNECTION, not merely a slower answer.
+ *
+ * Every arm takes its own endpoint from the pool: an episode is per-endpoint
+ * and outlives the request that opened it, so a shared endpoint would let the
+ * previous arm's rung explain the next arm's result.
+ */
+
+import { afterAll, beforeAll, describe, expect, test } from "bun:test";
+import {
+ type FixturePool,
+ type ProxyHandle,
+ assertRefused,
+ createRawFixturePool,
+ endpointsConfig,
+ envSnapshot,
+ postMessage,
+ reservePorts,
+ scheduledAttemptOffsets,
+ sleep,
+ startHealthyFixture,
+ startProxy,
+ startStatusFixture,
+} from "./fixtures.js";
+
+const API_TIMEOUT_MS = 60_000; // ⇒ derived deadline 30s ⇒ attempts at 0s, 5s, 15s
+const DEADLINE_MS = 30_000;
+
+let restoreEnv: () => void;
+let proxy: ProxyHandle;
+let pool: FixturePool;
+let healthy: ReturnType<typeof startHealthyFixture>;
+let upstream500: ReturnType<typeof startStatusFixture>;
+let deadPort: number;
+
+beforeAll(async () => {
+ restoreEnv = envSnapshot(["API_TIMEOUT_MS", "CLAUDISH_RECOVERY", "CLAUDISH_RECOVERY_UI"]);
+ process.env.API_TIMEOUT_MS = String(API_TIMEOUT_MS);
+ delete process.env.CLAUDISH_RECOVERY;
+
+ const [healthyPort, errPort, refusedPort] = reservePorts(3);
+ deadPort = refusedPort;
+
+ pool = createRawFixturePool(8, "ep-rst");
+ healthy = startHealthyFixture("HEALTHY-FIXTURE-ANSWER", healthyPort);
+ upstream500 = startStatusFixture(500, errPort);
+
+ proxy = await startProxy(
+ endpointsConfig([
+ ...pool.specs,
+ { name: "ep-live", port: healthyPort, model: "m-live" },
+ { name: "ep-500", port: errPort, model: "m-500" },
+ { name: "ep-refused", port: refusedPort, model: "m-refused" },
+ ])
+ );
+
+ // Warm the proxy (catalog/registry latching) so the "<2s" negatives measure
+ // the request path rather than one-time startup work.
+ const warm = await postMessage(proxy.url, { model: "ep-live@m-live" });
+ expect(warm.status).toBe(200);
+}, 60_000);
+
+afterAll(async () => {
+ await proxy?.shutdown();
+ pool?.stopAll();
+ healthy?.stop();
+ upstream500?.stop();
+ restoreEnv?.();
+});
+
+/**
+ * Issue a request against a fresh endpoint and resolve as soon as the fixture
+ * has seen `n` connections, then abort. An "is the ladder really on?" control
+ * therefore costs one rung rather than a whole deadline.
+ */
+async function ladderReaches(
+ n: number,
+ headers?: Record<string, string>,
+ budgetMs = 20_000
+): Promise<{ connections: number; answeredBeforeNthAttempt: boolean }> {
+ const { model, fixture } = pool.next();
+ const controller = new AbortController();
+ let settled = false;
+ const pending = postMessage(proxy.url, { model, headers, signal: controller.signal })
+ .then(() => {
+ settled = true;
+ })
+ .catch(() => {
+ settled = true;
+ });
+
+ const started = Date.now();
+ while (Date.now() - started < budgetMs) {
+ if (fixture.connections.length >= n || settled) break;
+ await sleep(50);
+ }
+ const connections = fixture.connections.length;
+ const answeredBeforeNthAttempt = settled && connections < n;
+ controller.abort();
+ await pending;
+ return { connections, answeredBeforeNthAttempt };
+}
+
+describe("CP-1 — a connection failure enters a retry ladder instead of answering", () => {
+ test("the upstream socket records the published ladder, truncated by the derived deadline", async () => {
+ const { model, fixture } = pool.next();
+ let answered = false;
+ const request = postMessage(proxy.url, { model }).then((r) => {
+ answered = true;
+ return r;
+ });
+
+ // CP1-01: nothing has been answered while the first rung is still waiting.
+ await sleep(2_000);
+ expect(answered).toBe(false);
+
+ const result = await request;
+
+ // Counted at the socket, never from a [Recovery] line (trap T-9).
+ const expectedOffsets = scheduledAttemptOffsets(DEADLINE_MS); // [0, 5000, 15000]
+ expect(fixture.connections.length).toBe(expectedOffsets.length);
+
+ const offsets = fixture.offsetsFromFirst();
+ for (const [i, expected] of expectedOffsets.entries()) {
+ expect(Math.abs(offsets[i] - expected)).toBeLessThan(1_500);
+ }
+
+ // No attempt may start after the derived deadline.
+ expect(Math.max(...offsets)).toBeLessThan(DEADLINE_MS);
+
+ // The answer is read at the proxy's public URL, after any remap (T-6).
+ expect(result.status).not.toBe(200);
+ expect(result.elapsedMs).toBeGreaterThan(15_000);
+ }, 60_000);
+
+ test("a client re-ask during the same outage is still retried, not answered instantly", async () => {
+ // The shipped design hands the retry back to the client at the deadline
+ // and the client re-enters the SAME episode (requirements, AMENDMENTS
+ // table, FR-3/FR-4). A re-ask that fast-fails would end the recovery the
+ // amendment promises to continue across requests.
+ const { model, fixture } = pool.next();
+ const first = await postMessage(proxy.url, { model });
+ expect(first.status).not.toBe(200);
+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);
+
+ fixture.reset();
+ const reask = await postMessage(proxy.url, { model });
+ expect(reask.status).not.toBe(200);
+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);
+ }, 90_000);
+
+ test("NEGATIVE: a reachable upstream that answers an error is answered at once, with no ladder", async () => {
+ const before = upstream500.hits;
+ const result = await postMessage(proxy.url, { model: "ep-500@m-500" });
+ expect(result.elapsedMs).toBeLessThan(2_000);
+ expect(upstream500.hits).toBe(before + 1);
+ expect(result.status).toBeGreaterThanOrEqual(400);
+ }, 20_000);
+
+ test("NEGATIVE: a healthy upstream answers 200 with the fixture's own text", async () => {
+ const result = await postMessage(proxy.url, { model: "ep-live@m-live" });
+ expect(result.status).toBe(200);
+ expect(result.body).toContain("HEALTHY-FIXTURE-ANSWER");
+ expect(result.elapsedMs).toBeLessThan(2_000);
+ }, 20_000);
+
+ test("NEGATIVE: /v1/models and /v1/messages/count_tokens never enter the ladder", async () => {
+ await assertRefused(deadPort);
+
+ const t0 = Date.now();
+ const models = await fetch(`${proxy.url}/v1/models`);
+ await models.text();
+ expect(models.status).toBe(200);
+ expect(Date.now() - t0).toBeLessThan(2_000);
+
+ const t1 = Date.now();
+ const count = await fetch(`${proxy.url}/v1/messages/count_tokens`, {
+ method: "POST",
+ headers: { "content-type": "application/json" },
+ body: JSON.stringify({
+ model: "ep-refused@m-refused",
+ max_tokens: 16,
+ messages: [{ role: "user", content: "hello" }],
+ }),
+ });
+ await count.text();
+ expect(Date.now() - t1).toBeLessThan(2_000);
+ }, 20_000);
+});
+
+describe("CP-5 — x-claudish-no-recovery: 1 skips the ladder", () => {
+ test("with the header: one upstream attempt, a connection_error in under a second", async () => {
+ const { model, fixture } = pool.next();
+ const result = await postMessage(proxy.url, {
+ model,
+ headers: { "x-claudish-no-recovery": "1" },
+ });
+
+ expect(result.elapsedMs).toBeLessThan(1_000);
+ expect(result.status).toBe(400);
+ expect(JSON.parse(result.body).error.type).toBe("connection_error");
+ // The decisive half: the ladder never ran at the socket.
+ expect(fixture.connections.length).toBe(1);
+ }, 20_000);
+
+ test("PAIRED CONTROL: identical fixture, same process, header removed ⇒ a second attempt arrives", async () => {
+ const control = await ladderReaches(2);
+ expect(control.answeredBeforeNthAttempt).toBe(false);
+ expect(control.connections).toBeGreaterThanOrEqual(2);
+ }, 30_000);
+
+ test("the header is honoured case-insensitively", async () => {
+ const { model, fixture } = pool.next();
+ const result = await postMessage(proxy.url, {
+ model,
+ headers: { "X-Claudish-No-Recovery": "1" },
+ });
+ expect(result.elapsedMs).toBeLessThan(1_000);
+ expect(result.status).toBe(400);
+ expect(fixture.connections.length).toBe(1);
+ }, 20_000);
+
+ test("the skip also holds on the streaming path", async () => {
+ const { model, fixture } = pool.next();
+ const result = await postMessage(proxy.url, {
+ model,
+ stream: true,
+ headers: { "x-claudish-no-recovery": "1" },
+ });
+ expect(result.elapsedMs).toBeLessThan(2_000);
+ expect(fixture.connections.length).toBe(1);
+ }, 20_000);
+});
+
+describe("CP-8 — the environment switch, with its paired enabled control", () => {
+ test("CLAUDISH_RECOVERY=0 ⇒ one attempt and an immediate error", async () => {
+ const { model, fixture } = pool.next();
+ process.env.CLAUDISH_RECOVERY = "0";
+ try {
+ const result = await postMessage(proxy.url, { model });
+ expect(result.elapsedMs).toBeLessThan(2_000);
+ expect(result.status).toBe(400);
+ expect(JSON.parse(result.body).error.type).toBe("connection_error");
+ expect(fixture.connections.length).toBe(1);
+ } finally {
+ delete process.env.CLAUDISH_RECOVERY;
+ }
+ }, 20_000);
+
+ test("PAIRED CONTROL: the switch removed, same process ⇒ the ladder runs again", async () => {
+ delete process.env.CLAUDISH_RECOVERY;
+ const control = await ladderReaches(2);
+ expect(control.answeredBeforeNthAttempt).toBe(false);
+ expect(control.connections).toBeGreaterThanOrEqual(2);
+ }, 30_000);
+
+ test("the switch does not change the healthy path's answer", async () => {
+ process.env.CLAUDISH_RECOVERY = "0";
+ let off: Awaited<ReturnType<typeof postMessage>>;
+ try {
+ off = await postMessage(proxy.url, { model: "ep-live@m-live" });
+ } finally {
+ delete process.env.CLAUDISH_RECOVERY;
+ }
+ expect(off.status).toBe(200);
+ expect(off.body).toContain("HEALTHY-FIXTURE-ANSWER");
+
+ const on = await postMessage(proxy.url, { model: "ep-live@m-live" });
+ expect(on.status).toBe(200);
+ expect(on.body).toContain("HEALTHY-FIXTURE-ANSWER");
+ expect(JSON.parse(on.body).content).toEqual(JSON.parse(off.body).content);
+ }, 20_000);
+});
diff --git a/packages/cli/src/recovery-contract/recovery-process-blackbox.test.ts b/packages/cli/src/recovery-contract/recovery-process-blackbox.test.ts
new file mode 100644
index 00000000..1c1e37b2
--- /dev/null
+++ b/packages/cli/src/recovery-contract/recovery-process-blackbox.test.ts
@@ -0,0 +1,169 @@
+/**
+ * CONTRACT POINT 10 (`[Recovery]` log lines, with no `--debug` anywhere) and
+ * the process-level half of CONTRACT POINT 7 (stop promptly, and actually
+ * exit).
+ *
+ * A SPAWNED CHILD with separated pipes, deliberately. An in-process
+ * `process.stderr.write` monkeypatch intercepts the CALL, not the file
+ * descriptor, so it cannot answer "did this byte go to stdout?" — which is the
+ * only question that matters when the client on the other end is a TUI. And a
+ * process that never exits is invisible to a test running inside it.
+ */
+
+import { describe, expect, test } from "bun:test";
+import { reservePorts, sleep, startRawFixture } from "./fixtures.js";
+
+const CHILD = new URL("./child-proxy-run.ts", import.meta.url).pathname;
+const SECRET_KEY = "sk-blackbox-SECRET-VALUE-9f3c1d"; // the key the child configures
+
+interface ChildRun {
+ stdout: string;
+ stderr: string;
+ exitCode: number | null;
+ lifetimeMs: number;
+ result: Record<string, unknown>;
+}
+
+async function runChild(
+ mode: "exhaust" | "abort",
+ endpointUrl: string,
+ apiTimeoutMs: number,
+ abortAfterMs?: number
+): Promise<ChildRun> {
+ const started = Date.now();
+ const child = Bun.spawn(
+ [
+ "bun",
+ "run",
+ CHILD,
+ mode,
+ endpointUrl,
+ String(apiTimeoutMs),
+ ...(abortAfterMs ? [String(abortAfterMs)] : []),
+ ],
+ { stdout: "pipe", stderr: "pipe", stdin: "ignore" }
+ );
+ const [stdout, stderr] = await Promise.all([
+ new Response(child.stdout).text(),
+ new Response(child.stderr).text(),
+ ]);
+ const exitCode = await child.exited;
+ const lifetimeMs = Date.now() - started;
+ const line = stdout.trim().split("\n").at(-1) ?? "{}";
+ let result: Record<string, unknown> = {};
+ try {
+ result = JSON.parse(line);
+ } catch {
+ /* asserted by the caller */
+ }
+ return { stdout, stderr, exitCode, lifetimeMs, result };
+}
+
+describe("CP-10 — recovery diagnostics in a real child process", () => {
+ test("a recovery episode is reported in the captured output with no --debug anywhere", async () => {
+ const [port] = reservePorts(1);
+ const fixture = startRawFixture(port);
+ try {
+ const run = await runChild("exhaust", `http://127.0.0.1:${port}/v1`, 40_000);
+ // The ladder really ran: counted at our socket, not in the log.
+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);
+ expect(run.result.status).toBe(400);
+
+ // Contract point 10: the retry activity is legible without a debug flag.
+ // The docs make the same promise for a shortened API_TIMEOUT_MS:
+ // "claudish shortens the hold to match and logs a [Recovery] line
+ // saying so" (docs/advanced/environment.md).
+ //
+ // MEASURED at HEAD 8ed0f13: the lines exist and are rich, but they go
+ // to the DEBUG LOG FILE (logs/claudish_*.log, which only a --debug run
+ // writes). With no debug flag a whole recovery episode leaves nothing
+ // on either stream but the final `[claudish] Error:` line.
+ expect(
+ run.stderr,
+ `neither stream carried a [Recovery] line.\nstdout=${run.stdout}\nstderr=${run.stderr}`
+ ).toContain("[Recovery]");
+ } finally {
+ fixture.stop();
+ }
+ }, 90_000);
+
+ test("no diagnostic byte lands on stdout — the client's protocol stream stays clean", async () => {
+ const [port] = reservePorts(1);
+ const fixture = startRawFixture(port);
+ try {
+ const run = await runChild("exhaust", `http://127.0.0.1:${port}/v1`, 40_000);
+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);
+
+ // stdout carries the child's single JSON result line and nothing else.
+ const stdoutLines = run.stdout.split("\n").filter((l) => l.trim().length > 0);
+ expect(stdoutLines.length).toBe(1);
+ expect(run.stdout).not.toContain("[Recovery]");
+ expect(run.stdout).not.toContain("[claudish]");
+ } finally {
+ fixture.stop();
+ }
+ }, 90_000);
+
+ test("no credential appears in anything the process printed", async () => {
+ const [port] = reservePorts(1);
+ const fixture = startRawFixture(port);
+ try {
+ const run = await runChild("exhaust", `http://127.0.0.1:${port}/v1`, 40_000);
+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);
+ // Positive control: the run really did print the failing endpoint, so
+ // "no secret" is not passing because nothing was printed at all.
+ expect(run.stderr + run.stdout).toContain(`127.0.0.1:${port}`);
+ expect(run.stdout).not.toContain(SECRET_KEY);
+ expect(run.stderr).not.toContain(SECRET_KEY);
+ } finally {
+ fixture.stop();
+ }
+ }, 90_000);
+});
+
+describe("CP-7 — the process itself lets go", () => {
+ test("after an aborted ladder the child shuts down and exits promptly", async () => {
+ const [port] = reservePorts(1);
+ const fixture = startRawFixture(port);
+ try {
+ // 12s, not 7s: the child's FIRST upstream attempt lands ~2.5s after the
+ // request starts (catalog/registry warm-up inside the request path), so
+ // a 7s abort can land before attempt 2 and make "it stopped retrying"
+ // trivially true — trap T-11.
+ const abortAfterMs = 12_000;
+ const run = await runChild("abort", `http://127.0.0.1:${port}/v1`, 120_000, abortAfterMs);
+
+ expect(run.result.aborted).toBe(true);
+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2); // it was mid-ladder
+ expect(run.exitCode).toBe(0);
+ expect(Number(run.result.shutdownMs)).toBeLessThan(1_000);
+ // Startup + the abort delay + a generous margin. A retained 60s timer
+ // blows straight through this.
+ expect(run.lifetimeMs).toBeLessThan(abortAfterMs + 10_000);
+ } finally {
+ fixture.stop();
+ }
+ }, 90_000);
+
+ test("SIGTERM mid-ladder ends the process within two seconds", async () => {
+ const [port] = reservePorts(1);
+ const fixture = startRawFixture(port);
+ const child = Bun.spawn(
+ ["bun", "run", CHILD, "exhaust", `http://127.0.0.1:${port}/v1`, "120000"],
+ { stdout: "pipe", stderr: "pipe", stdin: "ignore" }
+ );
+ try {
+ const deadline = Date.now() + 30_000;
+ while (Date.now() < deadline && fixture.connections.length < 2) await sleep(50);
+ expect(fixture.connections.length).toBeGreaterThanOrEqual(2);
+
+ const killedAt = Date.now();
+ child.kill("SIGTERM");
+ await child.exited;
+ expect(Date.now() - killedAt).toBeLessThan(2_000);
+ } finally {
+ child.kill("SIGKILL");
+ fixture.stop();
+ }
+ }, 90_000);
+});
diff --git a/packages/cli/src/recovery-contract/recovery-slow-fault-blackbox.test.ts b/packages/cli/src/recovery-contract/recovery-slow-fault-blackbox.test.ts
new file mode 100644
index 00000000..19562a93
--- /dev/null
+++ b/packages/cli/src/recovery-contract/recovery-slow-fault-blackbox.test.ts
@@ -0,0 +1,128 @@
+/**
+ * The SLOW-FAULT arm. Trap T-1, made executable.
+ *
+ * Every loopback refusal in this suite fails in about one millisecond, and a
+ * ~1ms fault is the one kind that cannot exercise anything that decays DURING
+ * an attempt: a per-attempt connect cap, a budget measured from inbound request
+ * start, or a liveness signal that a long silent attempt would let expire. The
+ * class of the originating incident — `unreachable`/`ETIMEDOUT` — is measured
+ * in tens of seconds, not milliseconds, and the validation contract's own
+ * C-18 exists because a suite built entirely on refusals is blind to it.
+ *
+ * Fault: F-SLOWCONNECT — `192.0.2.1` (TEST-NET-1, RFC 5737, unrouted), so the
+ * TCP connect burns the OS timeout instead of being refused. Local, no
+ * privileges, no network manipulation.
+ *
+ * REAL TIME. A fake clock makes every assertion in this file vacuous.
+ */
+
+import { afterAll, beforeAll, describe, expect, test } from "bun:test";
+import {
+ type ProxyHandle,
+ derivedDeadlineMs,
+ endpointsConfig,
+ envSnapshot,
+ postMessage,
+ sleep,
+ startProxy,
+} from "./fixtures.js";
+
+const API_TIMEOUT_MS = 60_000; // ⇒ 30s derived deadline
+const DEADLINE_MS = derivedDeadlineMs(API_TIMEOUT_MS);
+
+let restoreEnv: () => void;
+let proxy: ProxyHandle;
+
+beforeAll(async () => {
+ restoreEnv = envSnapshot(["API_TIMEOUT_MS", "CLAUDISH_RECOVERY"]);
+ process.env.API_TIMEOUT_MS = String(API_TIMEOUT_MS);
+ delete process.env.CLAUDISH_RECOVERY;
+
+ proxy = await startProxy(
+ endpointsConfig([{ name: "ep-slow", port: 0, model: "m-slow", url: "https://192.0.2.1/v1" }])
+ );
+}, 60_000);
+
+afterAll(async () => {
+ await proxy?.shutdown();
+ restoreEnv?.();
+}, 60_000);
+
+describe("a fault that is SLOW rather than instant", () => {
+ test("PRECONDITION P-3: a connect to TEST-NET-1 is still pending after 3s on this machine", async () => {
+ // A captive portal or an aggressive corporate resolver answers 192.0.2.1,
+ // which would silently turn the slow fault into a fast one. Fail loudly
+ // with the measured duration; never skip.
+ const started = Date.now();
+ let outcome = "answered";
+ try {
+ await fetch("https://192.0.2.1/v1", { signal: AbortSignal.timeout(3_000) });
+ } catch (err) {
+ outcome = String(err);
+ }
+ const elapsed = Date.now() - started;
+ expect(
+ elapsed,
+ `PRECONDITION P-3 FAILED: 192.0.2.1 settled in ${elapsed}ms (${outcome}). ` +
+ "Something on this network answers TEST-NET-1, so the slow-connect fault is not in force."
+ ).toBeGreaterThanOrEqual(2_800);
+ expect(outcome).not.toBe("answered");
+ }, 20_000);
+
+ test("the request is held past the early rungs — a slow fault is not answered early", async () => {
+ const controller = new AbortController();
+ let answered = false;
+ const request = postMessage(proxy.url, {
+ model: "ep-slow@m-slow",
+ signal: controller.signal,
+ })
+ .then(() => {
+ answered = true;
+ })
+ .catch(() => {
+ answered = true;
+ });
+
+ await sleep(12_000);
+ expect(answered).toBe(false);
+ controller.abort();
+ await request;
+ }, 40_000);
+
+ test("the derived deadline bounds a SLOW attempt, not just a fast one", async () => {
+ // Contract point 2: the budget is measured from inbound request start and
+ // the client is answered inside it. An implementation that checks the
+ // deadline only BETWEEN attempts satisfies every refusal-based test in
+ // this suite and overruns here, because one attempt outlives the whole
+ // budget. C-18 calls the missing piece PER_ATTEMPT_CONNECT_CAP_MS.
+ const controller = new AbortController();
+ let settledAt: number | null = null;
+ const started = Date.now();
+ const request = postMessage(proxy.url, {
+ model: "ep-slow@m-slow",
+ signal: controller.signal,
+ })
+ .then(() => {
+ settledAt = Date.now() - started;
+ })
+ .catch(() => {
+ settledAt = Date.now() - started;
+ });
+
+ const budget = DEADLINE_MS + 10_000;
+ const watchUntil = Date.now() + budget;
+ while (Date.now() < watchUntil && settledAt === null) await sleep(250);
+ // Snapshot BEFORE aborting: a value recorded after the abort would be the
+ // abort's own doing, not the proxy answering.
+ const settledOnItsOwn = settledAt;
+ controller.abort();
+ await request;
+
+ expect(
+ settledOnItsOwn,
+ `the client was still waiting ${budget}ms into a ${DEADLINE_MS}ms budget — ` +
+ "one slow connect outlived the whole deadline"
+ ).not.toBeNull();
+ expect(settledOnItsOwn as unknown as number).toBeLessThan(budget);
+ }, 90_000);
+});
diff --git a/packages/cli/src/recovery-contract/recovery-switches-blackbox.test.ts b/packages/cli/src/recovery-contract/recovery-switches-blackbox.test.ts
new file mode 100644
index 00000000..34ed6a59
--- /dev/null
+++ b/packages/cli/src/recovery-contract/recovery-switches-blackbox.test.ts
@@ -0,0 +1,174 @@
+/**
+ * CONTRACT POINT 8 — three switches (CLI flag, env var, config field) and their
+ * precedence: flag > env > project config > global config > default(on).
+ *
+ * The env arm lives in recovery-ladder-blackbox.test.ts (with its own paired
+ * control). This file covers the two CONFIG scopes, their precedence against
+ * the env var, and the invariant that costs a config field its life:
+ * a `ClaudishProfileConfig` field missing from `loadConfig`'s allowlist
+ * survives on disk until the first global save and is then silently dropped.
+ * Every other switch test passes right up to that save.
+ *
+ * Each arm takes a fresh endpoint: a recovery episode is per-endpoint and
+ * outlives the request that opened it.
+ */
+
+import { afterAll, afterEach, beforeAll, describe, expect, test } from "bun:test";
+import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { setConfigFileOverride } from "../config-override.js";
+import { loadConfig, saveConfig } from "../profile-config.js";
+import {
+ type FixturePool,
+ type ProxyHandle,
+ createRawFixturePool,
+ endpointsConfig,
+ envSnapshot,
+ postMessage,
+ sleep,
+ startProxy,
+} from "./fixtures.js";
+
+const API_TIMEOUT_MS = 40_000; // ⇒ 10s deadline ⇒ attempts at 0s, 5s
+
+let restoreEnv: () => void;
+let proxy: ProxyHandle;
+let pool: FixturePool;
+let projectDir: string;
+let originalCwd: string;
+
+beforeAll(async () => {
+ restoreEnv = envSnapshot(["API_TIMEOUT_MS", "CLAUDISH_RECOVERY", "CLAUDISH_RECOVERY_UI"]);
+ process.env.API_TIMEOUT_MS = String(API_TIMEOUT_MS);
+ delete process.env.CLAUDISH_RECOVERY;
+
+ originalCwd = process.cwd();
+ projectDir = mkdtempSync(join(tmpdir(), "recovery-project-"));
+
+ pool = createRawFixturePool(8, "ep-switch");
+ proxy = await startProxy(endpointsConfig(pool.specs));
+
+ // Warm-up: the FIRST request through a fresh proxy pays one-time catalog and
+ // registry latching (~3s), which would otherwise be charged to whichever
+ // "answers immediately" assertion happened to run first. The header keeps the
+ // warm-up itself out of the ladder.
+ const warm = pool.next();
+ await postMessage(proxy.url, {
+ model: warm.model,
+ headers: { "x-claudish-no-recovery": "1" },
+ });
+}, 60_000);
+
+afterEach(() => {
+ process.chdir(originalCwd);
+ rmSync(join(projectDir, ".claudish.json"), { force: true });
+ delete process.env.CLAUDISH_RECOVERY;
+});
+
+afterAll(async () => {
+ process.chdir(originalCwd);
+ await proxy?.shutdown();
+ pool?.stopAll();
+ rmSync(projectDir, { recursive: true, force: true });
+ restoreEnv?.();
+});
+
+function writeProjectConfig(value: unknown): void {
+ writeFileSync(join(projectDir, ".claudish.json"), JSON.stringify(value), "utf8");
+ process.chdir(projectDir);
+}
+
+/** Did the ladder run? Answered at the fixture's socket, with a bounded wait. */
+async function ladderRan(headers?: Record<string, string>): Promise<{
+ attempts: number;
+ elapsedMs: number;
+ status: number;
+}> {
+ const { model, fixture } = pool.next();
+ const result = await postMessage(proxy.url, { model, headers });
+ return {
+ attempts: fixture.connections.length,
+ elapsedMs: result.elapsedMs,
+ status: result.status,
+ };
+}
+
+describe("CP-8 — the config switches", () => {
+ test("project config recovery.enabled=false ⇒ one attempt, immediate answer", async () => {
+ writeProjectConfig({ recovery: { enabled: false } });
+ const run = await ladderRan();
+ expect(run.attempts).toBe(1);
+ expect(run.elapsedMs).toBeLessThan(2_000);
+ expect(run.status).toBe(400);
+ }, 30_000);
+
+ test("PAIRED CONTROL: the same project file with the field removed ⇒ the ladder runs", async () => {
+ writeProjectConfig({ someUnrelatedKey: true });
+ const run = await ladderRan();
+ expect(run.attempts).toBeGreaterThanOrEqual(2);
+ expect(run.elapsedMs).toBeGreaterThan(4_000);
+ }, 30_000);
+
+ test("an explicit project ON beats a project-scope absence and keeps the ladder", async () => {
+ writeProjectConfig({ recovery: { enabled: true } });
+ const run = await ladderRan();
+ expect(run.attempts).toBeGreaterThanOrEqual(2);
+ }, 30_000);
+
+ test("the env var beats the project config", async () => {
+ // Documented precedence: flag > env > project > global > default(on).
+ writeProjectConfig({ recovery: { enabled: true } });
+ process.env.CLAUDISH_RECOVERY = "0";
+ const run = await ladderRan();
+ expect(run.attempts).toBe(1);
+ expect(run.elapsedMs).toBeLessThan(2_000);
+ }, 30_000);
+
+ test("an empty object in the project scope is 'no opinion', not 'off'", async () => {
+ writeProjectConfig({ recovery: {} });
+ const run = await ladderRan();
+ expect(run.attempts).toBeGreaterThanOrEqual(2);
+ }, 30_000);
+
+ test("a garbage value does not silently disable recovery", async () => {
+ writeProjectConfig({ recovery: { enabled: "maybe" } });
+ const run = await ladderRan();
+ expect(run.attempts).toBeGreaterThanOrEqual(2);
+ }, 30_000);
+});
+
+describe("CP-8 — the switch survives a config save", () => {
+ test("recovery and recoveryUi are still on disk after load → save", async () => {
+ // The recorded failure mode: a field absent from loadConfig's allowlist
+ // lives on disk until the first global save and is then dropped with no
+ // error. Every behavioural switch test above passes right up to that save.
+ const dir = mkdtempSync(join(tmpdir(), "recovery-save-"));
+ const file = join(dir, "config.json");
+ writeFileSync(
+ file,
+ JSON.stringify({
+ recovery: { enabled: false },
+ recoveryUi: { enabled: false },
+ stats: { enabled: false },
+ }),
+ "utf8"
+ );
+ setConfigFileOverride(file);
+ try {
+ const loaded = loadConfig();
+ expect(loaded.recovery).toEqual({ enabled: false });
+ expect(loaded.recoveryUi).toEqual({ enabled: false });
+
+ saveConfig(loaded);
+ await sleep(10);
+
+ const onDisk = JSON.parse(readFileSync(file, "utf8"));
+ expect(onDisk.recovery).toEqual({ enabled: false });
+ expect(onDisk.recoveryUi).toEqual({ enabled: false });
+ } finally {
+ setConfigFileOverride(null);
+ rmSync(dir, { recursive: true, force: true });
+ }
+ }, 20_000);
+});
diff --git a/packages/cli/src/recovery/coordinator.ts b/packages/cli/src/recovery/coordinator.ts
index 91f921ca..5d7b3d70 100644
--- a/packages/cli/src/recovery/coordinator.ts
+++ b/packages/cli/src/recovery/coordinator.ts
@@ -29,7 +29,7 @@
import { randomUUID } from "node:crypto";
import { type ConnectionErrorKind, isLoopback } from "../handlers/shared/connection-error.js";
-import { log } from "../logger.js";
+import { log, logRecovery } from "../logger.js";
import { recoveryClock } from "./clock.js";
import {
RECOVERY_PROTOCOL_VERSION,
@@ -364,7 +364,7 @@ export function joinEpisode(seed: EpisodeSeed): EpisodeHandle {
ep.clientRetries++;
ep.state = "attempting";
ep.maxTier = 2;
- log(
+ logRecovery(
`[Recovery] ${ep.providerDisplayName} rejoined episode ${ep.episodeId} ` +
`(client retry ${ep.clientRetries}, ladder ${ep.ladderIndex}, attempts ${ep.attempts})`
);
@@ -410,7 +410,7 @@ export function joinEpisode(seed: EpisodeSeed): EpisodeHandle {
};
episodes.set(key, ep);
byId.set(ep.episodeId, ep);
- log(
+ logRecovery(
`[Recovery] episode ${ep.episodeId} opened for ${ep.providerDisplayName} at ${ep.endpoint} ` +
`(${ep.kind}/${ep.code ?? "?"})`
);
@@ -508,7 +508,7 @@ export function joinEpisode(seed: EpisodeSeed): EpisodeHandle {
episode.advancedForRound = episode.round;
episode.ladderIndex++;
}
- log(
+ logRecovery(
`[Recovery] ${episode.providerDisplayName} attempt ${episode.attempts} failed: ${outcome} ` +
`(episode ${episode.episodeId}, ladder ${episode.ladderIndex})`
);
@@ -554,7 +554,7 @@ export function joinEpisode(seed: EpisodeSeed): EpisodeHandle {
}, EPISODE_GRACE_MS);
c.unref?.(timer);
episode.graceTimer = timer;
- log(
+ logRecovery(
`[Recovery] episode ${episode.episodeId} handed off after ${Math.round(
c.now() - episode.startedAtPerf
)}ms, ${episode.attempts} attempts — holding ${EPISODE_GRACE_MS / 1000}s for a client retry`
@@ -580,7 +580,7 @@ function armAttemptTimer(episode: Episode, delay: number): void {
episode.round++;
for (const w of [...episode.waiters]) w.wake?.({ kind: "attempt" });
}, delay);
- log(
+ logRecovery(
`[Recovery] ${episode.providerDisplayName} waiting ${delay / 1000}s before attempt ` +
`${episode.attempts + 1} (episode ${episode.episodeId}, ${episode.waiters.size} waiting)`
);
@@ -610,7 +610,7 @@ function closeEpisode(episode: Episode, outcome: RecoveryOutcome): void {
if (episode.state === "abandoned") {
for (const w of [...episode.waiters]) w.wake?.({ kind: "gave_up" });
}
- log(
+ logRecovery(
`[Recovery] episode ${episode.episodeId} closed: ${outcome} after ` +
`${Math.round(clock.now() - episode.startedAtPerf)}ms and ${episode.attempts} attempts`
);
@@ -678,7 +678,7 @@ export function noteTargetReachable(providerName: string, endpoint: string): voi
for (const [key, ep] of [...episodes]) {
if (key !== exact && !key.startsWith(prefix)) continue;
if (ep.state !== "handoff") continue;
- log(
+ logRecovery(
`[Recovery] ${ep.providerDisplayName} answered on the client's own retry — ` +
`closing episode ${ep.episodeId} instead of waiting out its grace`
);
diff --git a/packages/cli/src/stats.ts b/packages/cli/src/stats.ts
index 2ab2c23c..0244d871 100644
--- a/packages/cli/src/stats.ts
+++ b/packages/cli/src/stats.ts
@@ -127,6 +127,28 @@ export function initStats(_config: ClaudishConfig): void {
}
}
+/**
+ * Tests only: drop the one-shot init latch.
+ *
+ * `initStats` is idempotent by design — it must be, since several entry points
+ * call it — and the latch is PROCESS state that a test file shares with every
+ * sibling. Bun runs a suite's files in one process, so whichever file
+ * initialised stats first decided `statsEnabled` for all of them: a later file
+ * that enables stats and asserts on recorded events silently asserts nothing,
+ * because `recordStats`'s first line returns on the stale `initialized` flag.
+ * A no-op assertion is the one test failure mode that never turns red.
+ *
+ * Same convention as `__resetEndpointDiagnosticsForTests` and
+ * `__resetPredefinedStateForTests`: exported, `__`-prefixed, never called by
+ * production code.
+ */
+export function __resetStatsForTests(): void {
+ initialized = false;
+ statsEnabled = false;
+ claudishVersion = "";
+ installMethod = "unknown";
+}
+
/**
* Record a stats event. Fast exit if disabled.
* Buffers to memory via appendEvent() — non-blocking.