Skip to content

Claude Code. Any Model. The most powerful AI coding agent now speaks every language.

feat(recovery): count the outages, and write down why the shape is what it is

Jack Rudenkoca5b4eb92a15
diff --git a/CLAUDE.md b/CLAUDE.md
index 6c28a187..9d2de308 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -11,6 +11,9 @@ lives in `ROADMAP.md`.
- `routing.md` — `provider@model` syntax, every provider prefix, catalog-gathered route candidates and their tier order, `defaultProvider` as the fallback position, the derived picker provider list, `SUBSCRIPTION_PROVIDERS`, local models
- `adapters.md` — Layers 1–3, stream parsers, error classification and retry, the 400 remap, why Gemini tool schemas need `items` on every array
+- `network-recovery.md` — the two-tier connection hold: the DERIVED deadline (`API_TIMEOUT_MS`, not a
+ watchdog), why tier 2 answers 503 and never 429, why the pane lease is heartbeat-driven, the
+ five auth sites
- `behavior-layer.md` — Layer 4, the harness-conformance supervisor
- `advisor.md` — `--advisor` for any main model: independent of `--monitor`, decorator on the routed handler, ids by tool name, retained session state, stub paths S1-S10, Claude Code's gates, metered panel billing; read before editing advisor, decorator, monitor-launch or native-auth code
- `providers/devin.md`, `providers/grok-subscription.md`, `providers/antigravity.md`, `providers/qwen-alibaba.md` — one per reverse-engineered provider
@@ -113,6 +116,8 @@ negative case: if you are about to type one of those words, stop and use the lef
- `gk@` is the Grok SUBSCRIPTION; `grok@`/`xai@` is the metered `x-ai`. `moonshot-cn@` is a different service from `moonshot@`.
- A bare Claude name (`claude-opus-5`, `opus`, `internal`) must never reach `route()`: `native-anthropic` has no credential store, so the credential filter drops it and the chain degrades to OpenRouter. Check `nativeRouteFor()` (`providers/native-route.ts`) FIRST, as the proxy does — `preflight` and the TUI probe once didn't and told agents to drop their own subscription's models. Native routes also cannot be probed from outside a session (the handler forwards the inbound Claude Code header); report them as a third outcome, never as success or failure.
- A new `ClaudishProfileConfig` field MUST be added to `loadConfig`'s allowlist in `profile-config.ts`; otherwise it survives on disk until the first global save and is then dropped. Bit `onepasswordEnvironments`, then `keychain`.
+- A new `StatsEvent` field MUST also be pushed in `eventToLogRecord` — `stats-otlp.ts` is a HAND-WRITTEN attribute allowlist, so a field wired only into the interface and `stats.ts` is typed, buffered to `~/.claudish/stats-buffer.json`, and never sent. Nothing errors; the number is just missing from every dashboard, a quarter later. `stats-otlp.test.ts`'s table is `satisfies Record<OptionalStatsKey, …>`, so adding the field breaks compilation until it is listed there too — keep that.
+- A connection failure can arrive wearing an AUTH status code, at FIVE sites in `composed-handler.ts` (`refreshAuth`, `forceRefreshAuth`, the parameter-recovery re-fetch, `getHeaders`, the primary fetch). 401 is retryable to `FallbackHandler`, so answering it for a network fault walks a subscription user onto metered billing mid-outage; an unclassified THROW is the same bug one layer out (`{status: 0}` + an unconditional advance, with no cost warning). Classify first or rethrow unchanged — never invent a status. `getHeaders()` is the non-obvious one: for `gk@` it is the request's first network touch, and refresh-conditional, so it is rare rather than safe.
## Commands
diff --git a/ai-docs/architecture/README.md b/ai-docs/architecture/README.md
index f87213f5..bfb57f0f 100644
--- a/ai-docs/architecture/README.md
+++ b/ai-docs/architecture/README.md
@@ -27,6 +27,7 @@ three earlier write-ups this material cites were lost.
| [`onepassword.md`](onepassword.md) | SDK-only secret resolution, the FOUR causes of an identical denial error, the cross-process handshake lock, parent-side route pinning, and the config TUI tab | touch `providers/onepassword*.ts`, credential pre-hydration, or the 1Password TUI tab |
| [`keychain.md`](keychain.md) | The macOS Keychain backend: the 17ms-vs-28ms measurement that split presence from values, the two `security` behaviours that are traps (hex reads, a "secure" write that stores nothing and exits 0), and why no new `CredentialSource` member | touch `providers/keychain.ts`, `keychain-source.ts`, or the Providers tab's key write/delete |
| [`adapters.md`](adapters.md) | Layers 1–3 (FormatConverter / ModelTranslator / ProviderTransport), stream-parser selection, errors that ride an HTTP 200, and the `upstream_status` remap with its downstream readers | add a provider, model translator, or stream parser; or change error classification and retry |
+| [`network-recovery.md`](network-recovery.md) | The two-tier connection hold: why the deadline is DERIVED from `API_TIMEOUT_MS` (measured 359.607 s, a default and not a watchdog), why the tier-2 status is 503 and never 429, why the pane's lease is heartbeat-driven and not frame-driven, and the five sites where a network failure wore an auth status code | touch the retry ladder, `recovery/`, the tier-2 status, the recovery pane, or any `x-claudish-recovery` reader |
| [`behavior-layer.md`](behavior-layer.md) | Layer 4, the harness-convention SUPERVISOR: the rule engine, the four hooks, the 115-call plan-mode measurement it was built on, and opt-in telemetry | write a `behavior/` rule, or change harness-conformance handling |
| [`advisor.md`](advisor.md) | `--advisor` for any main model: why it is independent of `--monitor` yet keeps four of its launch bits, the decorator around the ROUTED handler, id capture by tool name (the `toolu_` prefix that made it a no-op on every foreign provider), the tee and retained session-keyed state, the ten stub paths and advice origin, Claude Code's advisor gates, metered panel billing, and the inherited-placeholder 401 | touch `--advisor`, `advisor-decorator.ts`, `native-handler-advisor.ts`, `advisor-startup.ts`, monitor-gated launch code, or native auth in `claude-runner.ts` |
| [`headless-vs-interactive.md`](headless-vs-interactive.md) | Why `-p` is NOT interactive-without-a-TTY: `--agent` validation is SILENTLY skipped under `--input-format stream-json` (measured, exit 0, no stream frame, `result: success`), upstream closed it as not planned, and why magmux is in the dependency tree | drive Claude Code headlessly, add a child-spawn flag, or assume a headless run behaves like an interactive one |
diff --git a/ai-docs/architecture/adapters.md b/ai-docs/architecture/adapters.md
index 7af0fa2a..a07dc9c3 100644
--- a/ai-docs/architecture/adapters.md
+++ b/ai-docs/architecture/adapters.md
@@ -251,7 +251,11 @@ Every retry hook in claudish keys off the HTTP **status** (`anthropic-compat.ts`
- **Terminal** in-stream errors (`context_length_exceeded`, `invalid_request_error`) are NOT retried — they keep the existing inline-text treatment, which is the actionable path for them.
- **Anything else** (any content event) → `clean`, and the consumed bytes are **replayed byte-identically** so the real parser sees an unchanged stream.
-This is the one place the 400-not-503 doctrine (`composed-handler.ts` ~line 461) is deliberately inverted. That rule exists because a 503 makes Claude Code show "API error · Retrying · attempt N/10" with the real reason buried — correct for **terminal** faults, where retrying is theatre. An upstream overload is the opposite: genuinely transient, and the retry banner is the appropriate behaviour because retrying is the actual remedy. Terminal → 400 inline; transient-after-our-own-retries → 503.
+This is **one of two** places the 400-not-503 doctrine (`composed-handler.ts`'s `respondConnectionError`) is deliberately inverted. That rule exists because a 503 makes Claude Code show "API error · Retrying · attempt N/10" with the real reason buried — correct for **terminal** faults, where retrying is theatre. An upstream overload is the opposite: genuinely transient, and the retry banner is the appropriate behaviour because retrying is the actual remedy. Terminal → 400 inline; transient-after-our-own-retries → 503.
+
+The second inversion is network recovery's tier-2 handoff, which earns the same 503 by a different argument — not "the banner is the remedy" but "a pane is painting the reason, so the status no longer has to carry it". See [`network-recovery.md`](network-recovery.md).
+
+> This reference was `~line 461` until v9.2.x, then really `:665-675`, and is now inside a named method. **Cite the symbol, not the line.** Three separate readers have been sent to the wrong part of this 2,000-line file by a number that was correct when it was written.
**Trade-off to know:** sniffing withholds response headers until the first decisive event, capped by `DEFAULT_SNIFF_BUDGET_MS` (12s, chosen above the 0.85s–7.7s error latencies observed in the real log). On a healthy xhigh-reasoning turn that delays `message_start` by however long the model thinks before its first output item. No content is lost or reordered — the client shows a spinner either way — but time-to-first-byte is genuinely later than before. Past the budget claudish flushes and degrades gracefully to the inline-text path.
diff --git a/ai-docs/architecture/network-recovery.md b/ai-docs/architecture/network-recovery.md
new file mode 100644
index 00000000..dd5792ee
--- /dev/null
+++ b/ai-docs/architecture/network-recovery.md
@@ -0,0 +1,458 @@
+> Two-tier connection recovery: the in-request retry ladder, the magmux recovery pane, and the
+> 503 handoff that lets Claude Code's own retry loop carry an outage the socket cannot.
+>
+> Indexed in [`README.md`](./README.md). Status-code neighbours: [`adapters.md`](adapters.md).
+
+# Network recovery — why "retry forever" is two tiers and not one loop
+
+A transport-unreachable failure used to end a turn. `classifyConnectionError` returned non-null,
+`respondConnectionError` answered `400 connection_error`, and the transcript carried a dead turn the
+user had to retype. VPN flaps, a Tailscale MagicDNS outage, a stopped `ollama serve`, a laptop lid
+closing between two tool calls — all of them, terminal.
+
+The fix is **not** an unbounded retry loop, and this document exists mostly to record *why not*,
+because the reasons are measurements that the source cannot show you.
+
+---
+
+## 1. The client's clock is the real constraint, and it is a KNOB
+
+The obvious design is to hold the inbound request open and retry until the network returns. Phase 0
+measured what actually happens when you do.
+
+**Claude Code aborts the connection at 359.607 s** — reproduced at 359.4 / 360.1 / 360.1 across two
+processes, against a custom `ANTHROPIC_BASE_URL`. The important part is what that number *is*:
+
+> It is **not** a fixed watchdog. It is `API_TIMEOUT_MS`'s default of 360 000 ms. Setting
+> `API_TIMEOUT_MS=20000` produced six aborts at **exactly 20.000 s**.
+
+That single fact decides the shape of the feature twice.
+
+1. **A hold has a hard ceiling.** Whatever we do, the socket dies at the client's timeout. "Retry
+ forever inside one request" is not implementable — not as a trade-off, as an impossibility.
+2. **The deadline must be DERIVED, never written down.** A user with `API_TIMEOUT_MS=60000` gets a
+ client that gives up at 60 s. A hardcoded 270 s deadline means *every* episode on that machine
+ ends in `client_gone` and recovery never engages — silently, and only on their machine. So:
+
+```
+TIER1_DEADLINE_MS = max(FLOOR, min(API_TIMEOUT_MS ?? 360_000, 300_000) − 30_000)
+```
+
+`recovery/settings.ts` owns that computation and logs a `[Recovery]` line whenever the environment
+shortens the budget, so a short hold is visible rather than mysterious. This is CLAUDE.md's
+*"a default is a rule, never a pinned id"* arriving in the time domain.
+
+The other two ceilings the derivation clears:
+
+| ceiling | value | where |
+|---|---|---|
+| Bun's per-request idle timeout | 255 s | `proxy-server.ts`'s `idleTimeout` |
+| the unclamped attempt 1 | up to 45 s | reserved out of the budget, never clamped (NFR-1) |
+
+Bun's 255 s stops binding because `c.env.timeout(c.req.raw, 0)` **works** — measured, a request
+given `0` survived **6×** its `idleTimeout` where the un-disarmed control died at 1.6×. That call is
+made inside the `catch`, never on a healthy request.
+
+### Claude Code's retry budget is a COUNT, not a duration
+
+The second tier hands the retry back to the client. What the client will then spend is:
+
+- **~11 attempts** on its default budget, spanning **~174 s** for a 503 chain (backoff caps at
+ 38.4 s per gap, and `retry-after` is honoured verbatim);
+- **~300 attempts** with `CLAUDE_CODE_RETRY_WATCHDOG=1`, which claudish sets when the recovery UI is
+ enabled — reaching **~a day** of unattended recovery on a hung connection.
+
+It is a count, not a clock. That is why `EPISODE_GRACE_MS` is 120 s: the client's worst gap is
+38.4 s, so 120 s is 3.1× inside it and a returning client always finds its episode still open.
+
+**The watchdog is a deliberate, informed acceptance of RISK-6** — see §7.
+
+---
+
+## 2. The two tiers, and the episode that makes them one thing
+
+```
+POST /v1/messages ──► ComposedHandler.handle() startTime = performance.now()
+ attempt 1 = today's expression, BYTE-IDENTICAL ──► success
+ │ throws
+ ▼
+ classifyConnectionError() ── null ──► rethrow, unchanged
+ │ non-null
+ ▼
+ shouldSkipTier1()? probe header · recovery disabled · no budget left
+ │ no
+ ┌───────────────────▼──────────────────────────────────────────┐
+ │ TIER 1 — in-request, PRE-FLUSH, deadline derived from §1 │
+ │ joinEpisode(key) ONE episode per target, N waiters │
+ │ 5 → 10 → 30 → 60 → 60 … on a SHARED clock │
+ │ re-issue THROUGH provider.enqueueRequest, re-classified │
+ │ success ─► return the upstream Response ────────────────────┼──►
+ └───────────────────┬──────────────────────────────────────────┘
+ │ deadline reached
+ ┌───────────────────▼──────────────────────────────────────────┐
+ │ TIER 2 — hand the retry back │
+ │ uiLeaseValid(episodeId) ? 503 overloaded_error │
+ │ + x-should-retry: true │
+ │ + x-claudish-recovery: 1 │
+ │ : 400 connection_error (as before) │
+ │ the episode stays OPEN for EPISODE_GRACE_MS │
+ └───────────────────┬──────────────────────────────────────────┘
+ │ Claude Code's own retry re-POSTs and JOINS
+ │ the same episode — one continuous banner
+```
+
+**The episode is what makes two tiers read as one recovery.** It is a process-level singleton keyed
+by `${provider}|${host}`. A handoff leaves it open; the client's re-issued request finds it,
+increments `clientRetries`, and continues the *same* attempt counter, ladder position and banner —
+while getting a *fresh per-request deadline*, because the deadline belongs to the socket and the
+socket is new.
+
+Three lifetimes, and none of them is the state machine. Conflating them was the largest defect of
+the superseded design:
+
+| lifetime | scope | owner | ends when |
+|---|---|---|---|
+| the **deadline** | one inbound request | the waiter | its own `deadlineAt` passes — *that waiter alone* answers and leaves |
+| the **ladder** | one episode (`key`) | the coordinator | the last waiter leaves, or an attempt succeeds |
+| the **lease** | one pane × one episode | the UI manager | `UI_LEASE_MS` after the last heartbeat naming that episode |
+
+When one waiter's deadline moved the *shared* episode into `handoff` — a state with no timer — every
+other parked waiter silently stopped being retried and answered having attempted nothing. The
+visible symptom was a banner reading `waiters: 4` with one of them actually being retried.
+
+### The measured ladder
+
+Real clock, real refused loopback socket, through `ComposedHandler.handle()`:
+
+| attempt | fires at | gap |
+|---|---|---|
+| 1 | t+0 | — |
+| 2 | t+5.002 s | 5002 ms |
+| 3 | t+15.004 s | 10002 ms |
+| 4 | t+45.009 s | 30005 ms |
+| 5 | t+105.010 s | 60001 ms |
+| 6 | t+165.011 s | 60001 ms |
+| 7 | t+225.012 s | 60001 ms |
+| — | handoff at t+225.018 s, deadline 270 s | |
+
+18 ms of drift over 225 seconds. **The last array element repeats and the DEADLINE ends the
+schedule** — which inverts `STREAM_RETRY_DELAYS_MS`, where the array running out *is* the budget.
+The two schedules are deliberately separate: merging them would couple the sniffer's 12 s budget and
+a duration quoted verbatim in a user-facing message to this one.
+
+On a tier-2 rejoin the ladder does **not** reset. The target has been down for the whole of it, and
+restarting at 5 s would hammer a dead host harder the longer the outage lasted.
+
+### `PER_ATTEMPT_CONNECT_CAP_MS` is 45 s because 75 s collides with the OS
+
+macOS gives up on its own TCP connect at **75.004 s** (measured against the unrouted `192.0.2.1`; a
+refused loopback port is 5 ms). A 75 000 ms clamp races the operating system by **4 ms** and is
+decided differently on different runs, which makes every assertion keyed on "the attempt ended at
+the clamp" flaky by construction. 45 s clears the collision by 30 seconds and always wins.
+
+**Slow and fast connect failures are indistinguishable BY CODE on macOS.** `192.0.2.1:443` reports
+`ECONNREFUSED` / `ConnectionRefused` — the same code a loopback port refuses instantly with. Only
+elapsed time separates them, so nothing may branch on the code to tell them apart. `isLoopback(endpoint)`
+is the only honest discriminator, and it reads the address, not the error.
+
+---
+
+## 3. The status table — and why 503 and not 429
+
+| condition | status | headers | why |
+|---|---|---|---|
+| recovered inside tier 1 | the upstream's own | — | the caller carries on as if the first attempt had worked |
+| exhausted, **a lease is valid** | **503** `overloaded_error` | `x-should-retry: true`, `x-claudish-recovery: 1` | the reason is legible on the pane, so the retry may be handed back |
+| exhausted, **no lease** | **400** `connection_error` | — | nothing can display the reason, so it must ride the status |
+| `[q] give up` | **400** `connection_error` | — | guarded on `result.kind`, never on the lease — see below |
+| client disconnected | 499 (unread) | — | the socket is already gone |
+| unclassifiable throw on any attempt | rethrown unchanged | — | recovery must not widen what "transient" means |
+
+### 429 would have been a live billing bug
+
+`fallback-handler.ts` contains, verbatim:
+
+```ts
+// Rate limited — per-provider limit, a different provider may have capacity
+if (status === 429) return true;
+```
+
+A 429 from the exhausted network path makes `FallbackHandler` **advance to the next provider**.
+During a network outage the next provider is unreachable for the same reason, so the chain burns
+every candidate — each paying its own multi-minute budget — and per CLAUDE.md's standing invariant,
+an advance off a `SUBSCRIPTION_PROVIDERS` candidate onto a metered one quotes real money for a fault
+no provider caused.
+
+503 does not have this problem: `isRetryableError` has **no 503 branch**. It is the status the house
+already owns for "transient after our own retries, do not switch the user's provider" — the stream
+sniffer's exhaustion arm uses it for exactly that reason (`adapters.md`).
+
+529 was considered and rejected: it is absent from `exhaustedChainStatus`'s transient set and would
+add a status this codebase has never carried, re-opening every `status ===` under `handlers/`.
+
+### Chain-safety is STRUCTURAL, not wording-based — and ORDER is the property
+
+A 503 stopping the chain is not enough, twice over:
+
+- **`isRetryableError`'s FIRST statement is `hasQuotaExhaustionWording(errorBody)`**, deliberately
+ status-agnostic (it exists *because* of the 400 remap), and its phrase list contains the **bare
+ substring `"quota"`**. A 503 whose *message* happened to carry that word advanced the chain and
+ spent the user's money.
+- **`exhaustedChainStatus` could demote it.** With an earlier candidate already failed, a
+ non-retryable response goes to `formatCombinedError`, whose status is 503 only if *every*
+ accumulated error is transient. One earlier auth or 404 failure turns our 503 into a terminal 400
+ and the client never re-POSTs.
+
+So the marker is a **header**, `x-claudish-recovery: 1`, checked in two independent places:
+`handle()` returns a marked response VERBATIM before reading the body (defeating the combining), and
+`isRetryableError` returns `false` on it **above** the quota match (defeating the wording).
+
+> **Moving the marker check below the quota match — present, but LATE — is a live billing bug.**
+> It is mutation-covered as one, and the mutation that kills it is the quota-wording variant
+> specifically. A test whose 503 body says nothing about quota cannot see the defect at all.
+
+The marker cannot be forged: every non-ok exit from `ComposedHandler` is `c.json(...)`, which builds
+headers from nothing, and the only path that copies upstream headers verbatim
+(`stream-head-sniffer.ts`'s `replayResponse()`) runs after `!response.ok` has already returned.
+**If a future edit ever returns an upstream `Response` object on a non-ok path, the marker must be
+stripped there.**
+
+### `[q] give up` is guarded on the OUTCOME, never on the lease
+
+A lease is by definition valid at the moment the give-up key is pressed — the pane that took the
+keystroke is alive. A lease-only test would answer a retryable 503, Claude Code would immediately
+re-ask, and the give-up key would be a no-op with a banner still on screen.
+
+---
+
+## 4. The lease — why the heartbeat must not be driven by frames
+
+The generalisable rule the two exhaustion arms encode:
+
+> **A retryable status is permissible exactly when claudish still has a surface on which the reason
+> is legible.** Absent such a surface, the reason must ride the status, which means 400.
+
+The gate is not a boolean and not a latch. A latch set on `hello` and never cleared is true after
+`[q] give up`, after a killed pane, after a dead magmux and after a socket EOF — and the exhaustion
+arm then answers 503 with the reason visible nowhere, *which is strictly worse than the bug this
+feature exists to remove*. It is a **lease**:
+
+```
+uiLeaseValid(episodeId)
+ ⇔ the proxy itself opened a pane and holds the `open_pane` reply
+ ∧ some connected client has HEARTBEATED `ack` naming THIS episode within UI_LEASE_MS
+```
+
+Every clause works. *Naming this episode* makes it episode-scoped, so two concurrent episodes cannot
+borrow one pane's legitimacy. *`ack`, not `hello`* makes it **post-render** — a process can send
+`hello` before painting anything. *Within `UI_LEASE_MS`* makes it self-clearing on EOF, crash and
+freeze, with no cleanup path to forget. *The proxy opened the pane* means a forged same-uid client
+cannot manufacture the forbidden 503-with-no-banner state.
+
+**"Heartbeated", not "renewed by frame receipts" — this is the single most consequential word here,
+and the superseded design had it wrong.** Frames were emitted only while the episode was `waiting`.
+There is no tick during `attempting`. A real `unreachable` connect takes 20–75 s (`192.0.2.1`
+measured at **75 005 ms**), which is 2×–7.5× `UI_LEASE_MS` — so the lease expired *while the banner
+was alive and painted*, and the exhaustion arm answered 400 for exactly the failure class that
+motivated the feature.
+
+The lease is a statement about **the renderer being alive and painting this episode**, so it is
+renewed by the renderer on its own `UI_HEARTBEAT_MS` timer and by nothing else. The proxy's
+`FRAME_TICK_MS` tick — now emitted in *every* live state, not only `waiting` — is then free to serve
+the banner rather than the gate.
+
+**Every loopback-only test passed over this bug.** A refused loopback connect resolves in ~1 ms, so
+the ladder is all `waiting` and frames never stop. The regression is only visible with a slow
+connect, which is why the unit test fakes a 45-second attempt (4.5 lease windows, **not one frame
+emitted**) and the integration test runs a real pane against a 20-second lease clock.
+
+No transition in the episode's table touches the lease, and the lease never causes a transition. The
+retry loop is not the pane's business, and the pane's liveness is not the deadline's business.
+
+---
+
+## 5. A connection failure can wear an AUTH status code — at five separate sites
+
+This is the half of the work that shipped on its own merits and would have been worth doing with no
+recovery ladder at all.
+
+`fallback-handler.ts` reads **401 as retryable**. So a network outage during a token refresh did not
+fail the request — it **advanced the chain**, moving a subscription user onto a per-token candidate
+mid-outage, for a fault that had nothing to do with their credentials. Reproduced:
+
+```
+[Fallback] Sakana Fugu failed (HTTP 401), trying next provider...
+[Fallback] Metered Fallback succeeded after 1 failed attempt(s)
+```
+
+An *unclassified throw* leaving `handle()` is the same bug one layer out: it lands in
+`fallback-handler.ts`'s catch, which pushes `{ status: 0 }` and advances **unconditionally** —
+without even the per-token cost warning, which sits on the non-throwing branch. On a single-candidate
+route no `FallbackHandler` exists, so it lands in `proxy-server.ts` as a bare 500 instead.
+
+The inventory — every outbound call in `ComposedHandler` that can throw without a `Response`:
+
+| # | site | before | after |
+|---|---|---|---|
+| 1 | `refreshAuth()` catch | classified **never** — unconditional **401** | classify FIRST, before `err.terminal` |
+| 2 | `forceRefreshAuth()` catch (wraps the 401-retry `fetch` too) | classified **never** — unconditional **401** | classify FIRST |
+| 3 | parameter-recovery re-fetch | **no `try` at all** — escaped `handle()` → `{status: 0}` + advance | wrapped; classified → 400, unclassified → rethrown |
+| 4 | `getHeaders()` | **outside every `try`** — escaped `handle()`, or a bare 500 | wrapped; classified → 400, unclassified → rethrown |
+| 5 | primary `fetch` / `enqueueRequest` | already classified — the control | unchanged, refactored onto the shared helper |
+
+**Site 4 is worth a paragraph.** `getHeaders()` looks like a pure accessor and is not one. For `gk@`
+it is the request's *first* network touch: `grok-subscription.ts` → `grok-credential.ts` →
+`resolveGrokAccessToken()` → `fetch(auth.x.ai/oauth2/token)`. The touch is **refresh-conditional**,
+which makes the escape rare — not safe. Rarity is what kept it alive through three reviews.
+
+The rule all five now obey: **classify first, or rethrow unchanged; none may invent a status.** There
+is exactly one way to answer "claudish could not reach the host" — `respondConnectionError()` — and
+it answers 400 `connection_error`, never 401 and never a bare 500.
+
+Local transports needed evidence preservation before any of this could reach them: `local.ts` used
+to swallow its connect error in `catch (e: any) { log(…) }` and throw a bare
+`Error("Cannot connect to Ollama at …")`. `findConnectionCode` found no `.code`, no `.cause` and no
+message match, so `classifyConnectionError` returned **null** and a stopped Ollama — the single most
+common local failure — never reached tier 1 at all. Same class of fix in `vertex-oauth.ts`
+(discarded `code`), `openai.ts` (neither `code` nor `cause` on its error classes) and
+`antigravity.ts` (the banner must name the *auth* host, not the eventual inference endpoint).
+
+`healthChecked` latches on **success only**, so a retried `refreshAuth()` re-probes instead of
+returning a false success to attempt two.
+
+---
+
+## 6. Stats — and the one step whose omission is silent
+
+Five optional fields on `StatsEvent`, absent on every healthy request:
+
+| field | scope | what it answers |
+|---|---|---|
+| `retry_attempts` | **request** | how many times *this request* re-issued against the same provider. Not `fallback_attempts`, which counts different providers |
+| `recovery_ms` | **request** | how much of this record's `latency_ms` was backoff and failed connects |
+| `recovery_episode_id` | episode | the correlation key — `count(distinct …)` is the only honest answer to "how often" |
+| `recovery_client_retry` | episode | which tier-2 re-entry this request is; 0 = the original |
+| `recovery_outcome` | episode | `recovered` / `handoff` / `client_gone` / `gave_up` |
+
+**`latency_ms` continues to INCLUDE the waits**, per the standing decision in `adapters.md` ("the
+honest figure is time-to-usable-response"). `recovery_ms` exists so the resulting skew is explicable
+rather than mysterious: `latency_ms: 246_000, recovery_ms: 201_900` reads as a 44-second turn behind
+a three-and-a-half-minute outage. Without the second field, the same record reads as a four-minute
+model.
+
+**Why two scopes.** The episode's cumulative counters cannot stand in for the per-request ones: N
+concurrent requests share one episode, so each would report the sum of all of them, and a tier-2
+re-entry inherits counters from a request that already recorded its own. Summing `retry_attempts`
+across an outage would multiply it by the number of waiters and again by the number of re-entries.
+The episode-scoped figures are what the *banner* quotes, because the banner is about the outage.
+
+**The cardinality rule.** `stats-otlp.ts` says verbatim "one per LLM request". A request that hits
+both an auth-path episode and a fetch-path episode carries the id of the episode that **decided its
+status** — the later one. An auth episode that *recovered* emits no record of its own: its attempts
+fold into the same request's `retry_attempts` and its waits into the same `recovery_ms`. The
+coordinator emits **no lifecycle records at all**; a `grace_expired` event with no request behind it
+would be a phantom distorting request counts, success rate and latency percentiles in the very
+stream that is supposed to answer the question. Episode lifecycle is a `[Recovery]` structural log
+line instead.
+
+### The trap: `eventToLogRecord` is a hand-written allowlist
+
+A field added to the `StatsEvent` interface and populated by `stats.ts` but **not pushed in
+`eventToLogRecord`** is typed, type-checked, buffered, and written to
+`~/.claudish/stats-buffer.json` — and never leaves the machine. Nothing throws. The number is simply
+absent from every dashboard built to read it, and the loss surfaces a quarter later when someone
+asks a question the data cannot answer.
+
+The guard is in `stats-otlp.test.ts` and has two layers:
+
+1. `satisfies Record<OptionalStatsKey, …>` makes its table **exhaustive over `StatsEvent`'s optional
+ fields at COMPILE TIME**. Add an optional field to the interface and the test file stops
+ compiling until it is listed.
+2. The table then drives emit-when-set and omit-when-unset per field, so listing a field without
+ pushing it is red.
+
+Layer 1 is the load-bearing one — it is what stops this being a checklist someone has to remember to
+read. Both directions are mutation-proven: deleting the `recovery_episode_id` push, and swapping one
+`!== undefined` for a truthiness check (which drops `retry_attempts: 0`, the record that says "an
+episode existed and this request added nothing to it").
+
+`retryAttempted` on the error report is now **derived**, not hand-set. It had been a literal `false`
+on the connection path since before the ladder existed, so every recovered-then-failed outage was
+reported as a first-and-only attempt.
+
+---
+
+## 7. Risks accepted, explicitly
+
+**RISK-6 — duplicate charges on a re-issue.** A retry after `ECONNRESET`/`EPIPE` may re-run
+inference the provider already started billing. The user accepted this at the scale of our own
+ladder, and then accepted it again, with the arithmetic in front of them, at the watchdog's scale:
+
+| | watchdog dropped | **watchdog restored (chosen)** |
+|---|---|---|
+| client retry budget | ~11 attempts | ~300 attempts |
+| unattended reach, hung connection | ~66 min | **~a day** |
+| worst-case retry attempts per turn | ~46 | **~2,100** |
+| RISK-6 exposure | as originally accepted | **≈30× that** |
+
+`recovery_episode_id` is what makes the exposure *attributable* — without it, attribution breaks
+exactly across a tier-2 handoff, which is the highest-exposure path.
+
+**RISK-7 — a CI run holding a dead endpoint for the full deadline.** `--no-recovery` /
+`CLAUDISH_RECOVERY=0` restores today's immediate 400 everywhere, byte for byte. `--no-recovery-ui` /
+`CLAUDISH_RECOVERY_UI=0` keeps the retries and drops only the pane (and with it the 503 arm, since
+the lease can never be valid).
+
+**There is NO loopback carve-out, and its absence is a decision.** An earlier design skipped the
+ladder for a refused loopback address on the reasoning that a stopped local server will not start
+itself. It was cut for two reasons. The narrow one: the predicate was a regression generator — every
+revision of it either fired universally (skipping recovery everywhere) or fired never, and both
+times the defect was invisible because the *same rule* decided the tests. The broad one, which is
+the real one, in the user's words: *"why are we talking about ollama at all? we are not building a
+solution for ollama, it is a general one."* A user who restarts their server mid-ladder gets their
+turn back, exactly as a user whose VPN reconnects does.
+
+**The probe path must fail fast.** Every probe POSTs through `ComposedHandler`, so without a gate
+`--probe` and the config TUI's Test All would each hold an unreachable link open for the full
+deadline and then misreport it as a `timeout` — breaking the two tools that exist to diagnose this
+exact fault, and multiplying a Test All run by the probe timeout per unreachable link.
+`probe-live.ts` sends `x-claudish-no-recovery: 1` and `shouldSkipTier1` honours it.
+
+**The MCP path is deadline-bounded only.** `mcp-server.ts` passes `method`, `headers` and `body` but
+**no `signal`**, so when Claude Code cancels the MCP *tool call*, the proxy socket stays open,
+`c.req.raw.signal` never fires, and the ladder runs its full deadline. `client_gone` covers only the
+MCP server process dying outright.
+
+---
+
+## 8. What a healthy request pays
+
+Nothing, and the guarantee is **placement**, not a flag: the entire retry apparatus is constructed
+inside the `catch`, after classification has already returned non-null. The primary fetch expression
+— including its `enqueueRequest` ternary — is byte-identical to what it was before recovery existed.
+That is verifiable by reading the diff rather than by trusting a predicate.
+
+The re-issue goes through that **same ternary**. Six transports implement `enqueueRequest`, and what
+they implement is not decoration: a bounded 429 loop with `Retry-After`, a served-set model-fallback
+chain, and the local concurrency gate that stops `[email protected]:3` running four inferences at once.
+Skipping it would make the attempt that finally *connects* behave differently from the one that
+failed — and at the moment a network returns, N woken waiters would stampede unqueued into a
+provider that has just come back.
+
+The one measured cost is launch: wrapping the session in a magmux pane adds **+107 ms median**
+(login shell +47, magmux +60; worst wrapped sample 718 ms), which is 4.7× inside the 500 ms bar the
+design set for itself. Default-on survives on that evidence.
+
+---
+
+## 9. Evidence
+
+| report | what it establishes |
+|---|---|
+| [`network-recovery-phase0-measurements.md`](../reports/network-recovery-phase0-measurements.md) | the 359.607 s client abort and its `API_TIMEOUT_MS` cause; `c.env.timeout(req,0)`; `Request.signal` firing in 0.49–0.86 ms; the 75.004 s macOS connect; magmux's +107 ms |
+| [`network-recovery-phase2-verification-20260911.md`](../reports/network-recovery-phase2-verification-20260911.md) | the measured 5/10/30/60/60/60 ladder with 18 ms drift over 225 s; the four modules that shipped with no tests |
+| [`network-recovery-phase3-pane-20260911.md`](../reports/network-recovery-phase3-pane-20260911.md) | the lease, the frame-driven-heartbeat CRITICAL, and the two tests that can fail on it |
+| [`network-recovery-phase4-status-flip-20260911.md`](../reports/network-recovery-phase4-status-flip-20260911.md) | the 503 flip through a real interactive session; the chain-safety mutations, including the quota-wording one |
+
+Auth-path detail — the five-site inventory with per-site line references and the reproduced
+`[Fallback]` advance — is in the Phase-1 half of the same body of work, summarised in §5 above.
diff --git a/docs/advanced/environment.md b/docs/advanced/environment.md
index c127b673..d9c6b29d 100644
--- a/docs/advanced/environment.md
+++ b/docs/advanced/environment.md
@@ -135,6 +135,40 @@ export CLAUDISH_PORT='3456'
Useful when you need a predictable port for firewall rules or debugging.
+### `CLAUDISH_RECOVERY`
+
+Master switch for network recovery. Default **on**.
+
+```bash
+export CLAUDISH_RECOVERY=0 # or: claudish --no-recovery
+```
+
+When claudish cannot reach a provider — DNS unresolved, connection refused, host unreachable — it no longer fails the turn immediately. It retries on a `5s → 10s → 30s → 60s → 60s…` ladder for up to a derived deadline (see below), and if the network comes back the turn continues as if nothing happened.
+
+**Turn it off in CI and in scripted `-p` runs.** With recovery on, a genuinely dead endpoint — a wrong host, a firewalled port, a service you forgot to start — holds the request for the whole deadline before reporting. Off restores the previous behaviour exactly: an immediate error with the same message.
+
+The deadline is **derived from `API_TIMEOUT_MS`**, not fixed:
+
+```
+min(API_TIMEOUT_MS, 300s) − 30s # ~270s at the 360s default
+```
+
+Claude Code aborts its own request at `API_TIMEOUT_MS` (360 000 ms by default), so a hold longer than that would be cut off by the client mid-wait. If you have shortened `API_TIMEOUT_MS`, claudish shortens the hold to match and logs a `[Recovery]` line saying so.
+
+### `CLAUDISH_RECOVERY_UI`
+
+Whether claudish may open a magmux pane showing the recovery banner. Default **on**.
+
+```bash
+export CLAUDISH_RECOVERY_UI=0 # or: claudish --no-recovery-ui
+```
+
+The banner names the provider, the host, the reason, the attempt count and a live countdown, with `[r] try now` and `[q] give up`. Turning it off keeps the retries and drops only the surface — and with it the ability to hand a retry back to Claude Code, since claudish will not ask a client to retry a failure it cannot show you.
+
+Independent of `CLAUDISH_RECOVERY`: the UI switch never decides whether a retry happens, only whether it is visible.
+
+Full rationale — the measurements behind the deadline, and why the handoff status is 503 — is in `ai-docs/architecture/network-recovery.md`.
+
---
## Read-Only Variables
@@ -164,6 +198,9 @@ CLAUDISH_MODEL_SUBAGENT=minimax/minimax-m2
# Fixed port (optional)
# CLAUDISH_PORT=3456
+
+# CI: fail fast on an unreachable provider instead of retrying for ~270s
+# CLAUDISH_RECOVERY=0
```
---
diff --git a/docs/usage/magmux.md b/docs/usage/magmux.md
index 48f88472..0b43ae5b 100644
--- a/docs/usage/magmux.md
+++ b/docs/usage/magmux.md
@@ -65,9 +65,11 @@ What happens:
1. Claudish creates a session directory with anonymized model IDs
2. Generates a gridfile (one command per pane)
3. Launches magmux with the grid layout
-4. Polls for completion and updates the status bar every 500ms
+4. Connects to magmux's IPC socket as a client and collects its lifecycle events
-The status bar shows live progress:
+Claudish does **not** poll. Magmux owns the entire pane lifecycle — idle detection, the `DONE`/`FAIL` overlays, the green/red tints, the status bar, and auto-exit — and pushes `snapshot`, `exit`, `results` and `shutdown` events over that socket. Claudish reads them and treats the final `results` event as the authoritative per-pane state.
+
+The status bar shows live progress, redrawn by magmux's own render loop:
```
claudish team 3 done 32s complete ctrl-g q to quit
@@ -251,7 +253,7 @@ The C MTM binary still ships in the repo (`packages/cli/native/mtm/`) as a fallb
### Status bar shows stale data in grid mode
-**Cause**: The claudish poller writes the status bar file every 500ms. Brief delays between model completion and status bar update are normal.
+**Cause**: Magmux redraws the status bar from its own render loop, counting done and running panes. A model that has finished producing output but whose process has not yet exited still counts as running, so a brief lag between the last line of output and the counter moving is normal.
**Fix**: Wait a moment. The final status always reflects the true state after all models finish.
diff --git a/packages/cli/src/handlers/composed-handler.ts b/packages/cli/src/handlers/composed-handler.ts
index 0d2a62ef..17049fdb 100644
--- a/packages/cli/src/handlers/composed-handler.ts
+++ b/packages/cli/src/handlers/composed-handler.ts
@@ -39,7 +39,12 @@ import { deepMergeParams } from "../model-params.js";
import { describeSiblingKeys, getProviderByName } from "../providers/provider-definitions.js";
import { isTerminal429 } from "../providers/transport/openai.js";
import { recoveryClock } from "../recovery/clock.js";
-import { episodeCount, noteTargetReachable, uiLeaseValid } from "../recovery/coordinator.js";
+import {
+ type RecoveryOutcome,
+ episodeCount,
+ noteTargetReachable,
+ uiLeaseValid,
+} from "../recovery/coordinator.js";
import { resolveRecoveryEnabled } from "../recovery/settings.js";
import {
type OpenAIImageBlock,
@@ -48,6 +53,7 @@ import {
} from "../services/vision-proxy.js";
import { type SessionEventRegistry, extractSessionId } from "../session-events/index.js";
import { applyProInjection } from "../session-events/pro-injection.js";
+import type { StatsEvent } from "../stats-otlp.js";
import { recordStats } from "../stats.js";
import { classifyError, reportError } from "../telemetry.js";
import { transformOpenAIToClaude } from "../transform.js";
@@ -100,6 +106,28 @@ import { captureUpstreamError } from "./shared/upstream-error-capture.js";
*/
const STREAM_RETRY_DELAYS_MS = [3_000, 15_000, 30_000];
+/**
+ * One request's running account of what network recovery cost it. Mutable, and
+ * mutated only by `recoverConnection`.
+ *
+ * Every field is REQUEST-scoped even though the episode behind them is not.
+ * See `ComposedHandler.newRecoveryTally` for why that distinction is the whole
+ * point, and `transient-retry.ts`'s `ConnectionRetryTally` for where the
+ * request-scoped figures come from.
+ */
+interface RecoveryTally {
+ /** Re-issues this request made. Not counting the failure that started it. */
+ retries: number;
+ /** Ms of this request's own latency spent inside the ladder. */
+ recoveryMs: number;
+ /** Set once an episode exists. Its presence is what makes the rest real. */
+ episodeId?: string;
+ /** Which tier-2 re-entry this request is. 0 = the original. */
+ clientRetry: number;
+ /** How the episode ended for this request, once it has. */
+ outcome?: RecoveryOutcome;
+}
+
function extractAuthHeaders(c: Context): VisionProxyAuthHeaders {
const headers = c.req.header();
const auth: VisionProxyAuthHeaders = {};
@@ -348,6 +376,50 @@ export class ComposedHandler implements ModelHandler {
}
}
+ /**
+ * What one REQUEST did inside the retry ladder, accumulated across every site
+ * in that request that could enter it.
+ *
+ * One per `handle()` call, never on `this` — a `ComposedHandler` instance is
+ * reused for the life of the process and serves concurrent requests, so an
+ * instance field here would report one request's outage on another request's
+ * record. The cost of getting that wrong is not a wrong number in a log: it
+ * is a wrong number in the only stream that can answer "how often does this
+ * happen", which is what the feature was measured by.
+ *
+ * ACCUMULATION IS PER-REQUEST AND LAST-EPISODE-WINS. A request can hit an
+ * auth-path episode and then a fetch-path episode. Attempts and waits ADD
+ * (they are both this request's own time, and there is exactly one record per
+ * request), while `episodeId` and `clientRetry` are overwritten — so the id
+ * that survives is the one belonging to the episode that DECIDED THE STATUS,
+ * which is the only reading under which a scalar correlation id is
+ * well-defined.
+ */
+ private static newRecoveryTally(): RecoveryTally {
+ return { retries: 0, recoveryMs: 0, clientRetry: 0 };
+ }
+
+ /**
+ * The recovery half of a stats event — spread into every `recordStats` on a
+ * path a recovery could precede.
+ *
+ * Empty unless an episode actually happened, which is what keeps a healthy
+ * request's record byte-identical to what it was before this feature existed.
+ * `retry_attempts: 0` IS emitted when an episode existed and this request
+ * added no re-issue to it; that is a different fact from "no recovery", and
+ * only the presence of `recovery_episode_id` distinguishes them.
+ */
+ private recoveryStats(tally: RecoveryTally | undefined): Partial<StatsEvent> {
+ if (!tally?.episodeId) return {};
+ return {
+ retry_attempts: tally.retries,
+ recovery_ms: tally.recoveryMs,
+ recovery_episode_id: tally.episodeId,
+ recovery_client_retry: tally.clientRetry,
+ ...(tally.outcome ? { recovery_outcome: tally.outcome } : {}),
+ };
+ }
+
/**
* The ONE response shape for "claudish could not reach the host".
*
@@ -378,7 +450,18 @@ export class ComposedHandler implements ModelHandler {
ctx: {
startTime: number;
fallbackMeta?: { chain: string[]; attempts: number };
- retryAttempted?: boolean;
+ /**
+ * This request's ladder account, if it ran one. Absent means the ladder
+ * was skipped, which is today's immediate-400 behaviour.
+ */
+ recovery?: RecoveryTally;
+ /**
+ * A retry this request made BEFORE the ladder — the parameter-recovery
+ * re-fetch and the forced-auth re-fetch both are one. OR'd with the
+ * ladder's own count rather than replaced by it, because both are true
+ * answers to "did claudish retry before reporting this".
+ */
+ retriedBeforeLadder?: boolean;
authType?: "api-key" | "oauth" | "none";
/** Where the failure happened, for the debug log only. */
site?: string;
@@ -397,7 +480,12 @@ export class ComposedHandler implements ModelHandler {
modelId: this.targetModel,
httpStatus: undefined,
isStreaming: false,
- retryAttempted: ctx.retryAttempted ?? false,
+ // DERIVED, never hand-set. This field had been a literal `false` on this
+ // path since before the ladder existed, so every recovered-then-failed
+ // outage was reported as a first-and-only attempt. It is true exactly
+ // when this request re-issued the operation at least once, from either
+ // source.
+ retryAttempted: (ctx.retriedBeforeLadder ?? false) || (ctx.recovery?.retries ?? 0) > 0,
isInteractive: this.isInteractive,
...(ctx.authType ? { authType: ctx.authType } : {}),
});
@@ -419,6 +507,7 @@ export class ComposedHandler implements ModelHandler {
fallback_chain: ctx.fallbackMeta?.chain,
fallback_attempts: ctx.fallbackMeta?.attempts,
invocation_mode: this.options.invocationMode ?? "auto-route",
+ ...this.recoveryStats(ctx.recovery),
});
} catch {
// Stats must never crash claudish
@@ -469,10 +558,15 @@ export class ComposedHandler implements ModelHandler {
conn: { kind: ConnectionErrorKind; code: string },
endpoint: string,
ctx: {
+ /** EPISODE-scoped, and deliberately so: these two are what the SENTENCE
+ * quotes, and the sentence is about the outage rather than about this
+ * socket. The request-scoped figures the stats record are in
+ * `recovery`. */
startTime: number;
attempts: number;
recoveryMs: number;
fallbackMeta?: { chain: string[]; attempts: number };
+ recovery?: RecoveryTally;
}
): Response {
const reason = buildConnectionErrorMessage(conn.kind, this.provider.displayName, endpoint);
@@ -495,6 +589,7 @@ export class ComposedHandler implements ModelHandler {
fallback_chain: ctx.fallbackMeta?.chain,
fallback_attempts: ctx.fallbackMeta?.attempts,
invocation_mode: this.options.invocationMode ?? "auto-route",
+ ...this.recoveryStats(ctx.recovery),
});
} catch {
// Stats must never crash claudish
@@ -579,9 +674,16 @@ export class ComposedHandler implements ModelHandler {
startTime: number;
deadlineAt: number;
fallbackMeta?: { chain: string[]; attempts: number };
- retryAttempted?: boolean;
authType?: "api-key" | "oauth" | "none";
site: string;
+ /** True at the two sites that are THEMSELVES a retry. See the field of
+ * the same name on `respondConnectionError`. */
+ retriedBeforeLadder?: boolean;
+ /**
+ * THIS REQUEST'S tally, created once per `handle()` and threaded through
+ * every site that can enter the ladder. Mutated here and nowhere else.
+ */
+ recovery: RecoveryTally;
}
): Promise<{ kind: "value"; value: T } | { kind: "respond"; response: Response }> {
const skip = this.shouldSkipTier1(c, ctx.deadlineAt);
@@ -620,15 +722,33 @@ export class ComposedHandler implements ModelHandler {
signal: this.clientSignal(c),
});
+ // Fold this ladder run into the request's account BEFORE any arm below can
+ // return. A request that recovers at the auth site and then exhausts at the
+ // fetch site must report BOTH sets of attempts and BOTH waits — they were
+ // all its own latency — under the id of the episode that decided its
+ // status, which is the later one. Both halves of that are this statement.
+ ctx.recovery.retries += result.requestRetries;
+ ctx.recovery.recoveryMs += result.requestRecoveryMs;
+ ctx.recovery.episodeId = result.episodeId;
+ ctx.recovery.clientRetry = result.clientRetry;
+
switch (result.kind) {
case "ok":
+ ctx.recovery.outcome = "recovered";
return { kind: "value", value: result.value };
case "threw":
// Classification runs on every attempt. Something that is not a
// connection failure must keep its own route out of here.
throw result.error;
case "client_gone":
- this.recordClientGone(ctx.startTime, ctx.fallbackMeta, result.episodeId, result.attempts);
+ ctx.recovery.outcome = "client_gone";
+ this.recordClientGone(
+ ctx.startTime,
+ ctx.fallbackMeta,
+ result.episodeId,
+ result.attempts,
+ ctx.recovery
+ );
// Nobody reads this. It exists so Hono has an object to return for a
// socket that is already gone.
return { kind: "respond", response: new Response(null, { status: 499 }) };
@@ -653,6 +773,8 @@ export class ComposedHandler implements ModelHandler {
// by definition alive), so a lease-only test would answer a retryable
// 503 and Claude Code would immediately re-ask — turning the give-up
// key into a no-op with a banner still on screen.
+ ctx.recovery.outcome = result.kind === "exhausted" ? "handoff" : "gave_up";
+
if (result.kind === "exhausted" && leased) {
log(
`[Recovery] ${this.provider.displayName} exhausted after ${result.attempts} attempts ` +
@@ -666,6 +788,7 @@ export class ComposedHandler implements ModelHandler {
attempts: result.attempts,
recoveryMs: result.recoveryMs,
fallbackMeta: ctx.fallbackMeta,
+ recovery: ctx.recovery,
}),
};
}
@@ -677,10 +800,7 @@ export class ComposedHandler implements ModelHandler {
);
return {
kind: "respond",
- response: this.respondConnectionError(c, result.error, result.conn, result.endpoint, {
- ...ctx,
- retryAttempted: true,
- }),
+ response: this.respondConnectionError(c, result.error, result.conn, result.endpoint, ctx),
};
}
}
@@ -691,7 +811,8 @@ export class ComposedHandler implements ModelHandler {
startTime: number,
fallbackMeta: { chain: string[]; attempts: number } | undefined,
episodeId: string,
- attempts: number
+ attempts: number,
+ recovery?: RecoveryTally
): void {
log(
`[Recovery] client disconnected after ${attempts} attempts (episode ${episodeId}) — ` +
@@ -714,6 +835,7 @@ export class ComposedHandler implements ModelHandler {
fallback_chain: fallbackMeta?.chain,
fallback_attempts: fallbackMeta?.attempts,
invocation_mode: this.options.invocationMode ?? "auto-route",
+ ...this.recoveryStats(recovery),
});
} catch {
// Stats must never crash claudish
@@ -729,6 +851,11 @@ export class ComposedHandler implements ModelHandler {
// Used in all stats recording paths so a single event carries complete info.
const fallbackMeta = this.pendingFallbackMeta;
this.pendingFallbackMeta = undefined;
+ // This request's network-recovery account. A local, like `fallbackMeta` and
+ // for the same reason: the handler instance outlives the request and serves
+ // several at once. It stays all-zero and contributes NOTHING to any stats
+ // record unless a classified connection failure actually occurs.
+ const recoveryTally = ComposedHandler.newRecoveryTally();
// 1. Transform incoming Claude-format request
const { claudeRequest, droppedParams } = transformOpenAIToClaude(payload);
@@ -997,6 +1124,7 @@ export class ComposedHandler implements ModelHandler {
fallbackMeta,
authType: "oauth",
site: "refreshAuth",
+ recovery: recoveryTally,
}
);
if (outcome.kind === "respond") return outcome.response;
@@ -1106,6 +1234,7 @@ export class ComposedHandler implements ModelHandler {
fallbackMeta,
authType: "oauth",
site: "getHeaders",
+ recovery: recoveryTally,
}
);
if (outcome.kind === "respond") return outcome.response;
@@ -1186,6 +1315,7 @@ export class ComposedHandler implements ModelHandler {
deadlineAt: tier1DeadlineAt(c),
fallbackMeta,
site: "fetch",
+ recovery: recoveryTally,
});
if (outcome.kind === "respond") return outcome.response;
response = outcome.value;
@@ -1319,8 +1449,9 @@ export class ComposedHandler implements ModelHandler {
// there is no unclamped attempt left ahead of it to reserve for.
deadlineAt: tier1DeadlineAt(c),
fallbackMeta,
- retryAttempted: true,
+ retriedBeforeLadder: true,
site: "parameter-recovery",
+ recovery: recoveryTally,
}
);
if (outcome.kind === "respond") return outcome.response;
@@ -1414,6 +1545,7 @@ export class ComposedHandler implements ModelHandler {
fallback_chain: fallbackMeta?.chain,
fallback_attempts: fallbackMeta?.attempts,
invocation_mode: this.options.invocationMode ?? "auto-route",
+ ...this.recoveryStats(recoveryTally),
});
} catch {
// Stats must never crash claudish
@@ -1447,9 +1579,10 @@ export class ComposedHandler implements ModelHandler {
startTime,
deadlineAt: tier1DeadlineAt(c),
fallbackMeta,
- retryAttempted: true,
+ retriedBeforeLadder: true,
authType: "oauth",
site: "forceRefreshAuth",
+ recovery: recoveryTally,
}
);
if (outcome.kind === "respond") return outcome.response;
@@ -1492,6 +1625,7 @@ export class ComposedHandler implements ModelHandler {
fallback_chain: fallbackMeta?.chain,
fallback_attempts: fallbackMeta?.attempts,
invocation_mode: this.options.invocationMode ?? "auto-route",
+ ...this.recoveryStats(recoveryTally),
});
} catch {
// Stats must never crash claudish
@@ -1594,6 +1728,7 @@ export class ComposedHandler implements ModelHandler {
fallback_chain: fallbackMeta?.chain,
fallback_attempts: fallbackMeta?.attempts,
invocation_mode: this.options.invocationMode ?? "auto-route",
+ ...this.recoveryStats(recoveryTally),
});
} catch {
// Stats must never crash claudish
@@ -1681,6 +1816,7 @@ export class ComposedHandler implements ModelHandler {
fallback_chain: fallbackMeta?.chain,
fallback_attempts: fallbackMeta?.attempts,
invocation_mode: this.options.invocationMode ?? "auto-route",
+ ...this.recoveryStats(recoveryTally),
});
} catch {
// Stats must never crash claudish
@@ -1739,6 +1875,7 @@ export class ComposedHandler implements ModelHandler {
fallback_chain: fallbackMeta?.chain,
fallback_attempts: fallbackMeta?.attempts,
invocation_mode: this.options.invocationMode ?? "auto-route",
+ ...this.recoveryStats(recoveryTally),
});
} catch {
// Stats must never crash claudish
@@ -1793,6 +1930,7 @@ export class ComposedHandler implements ModelHandler {
fallback_chain: fallbackMeta?.chain,
fallback_attempts: fallbackMeta?.attempts,
invocation_mode: this.options.invocationMode ?? "auto-route",
+ ...this.recoveryStats(recoveryTally),
});
} catch {
// Stats must never crash claudish
diff --git a/packages/cli/src/handlers/shared/transient-retry.ts b/packages/cli/src/handlers/shared/transient-retry.ts
index 911f5244..5c6043ba 100644
--- a/packages/cli/src/handlers/shared/transient-retry.ts
+++ b/packages/cli/src/handlers/shared/transient-retry.ts
@@ -169,35 +169,64 @@ export interface ConnectionRetryContext {
signal: AbortSignal;
}
+/**
+ * The counters every outcome carries, in TWO scopes, because the two answer
+ * different questions and conflating them mis-reports both.
+ *
+ * EPISODE scope (`attempts`, `recoveryMs`) is what the human reads: one banner
+ * saying "attempt 9 · 4m 12s in recovery" across every waiter and both tiers,
+ * which is the honest account of the OUTAGE.
+ *
+ * REQUEST scope (`requestRetries`, `requestRecoveryMs`) is what stats record,
+ * because a stats event is one-per-request by definition. The episode figures
+ * cannot stand in for these: N concurrent requests share one episode, so each
+ * would report the sum of all of them, and a tier-2 re-entry inherits counters
+ * from a request that already recorded its own — so summing `retry_attempts`
+ * over an outage would multiply it by the number of waiters and again by the
+ * number of re-entries.
+ */
+interface ConnectionRetryTally {
+ /** Episode-scoped: every attempt by every waiter, across both tiers. */
+ attempts: number;
+ /** Episode-scoped: ms since the episode opened. */
+ recoveryMs: number;
+ /**
+ * THIS request's RE-ISSUES — the operation run again, not counting the
+ * failure that brought us here. 0 is meaningful and reachable: an episode
+ * existed and this request added nothing to it because its own budget was
+ * already spent. That is what `retry_attempts` records.
+ */
+ requestRetries: number;
+ /** THIS request's own ms inside the ladder — `recovery_ms`. */
+ requestRecoveryMs: number;
+ /** Which tier-2 re-entry this is. 0 = the request that opened the episode. */
+ clientRetry: number;
+ episodeId: string;
+}
+
export type ConnectionRetryResult<T> =
- | { kind: "ok"; value: T; attempts: number; recoveryMs: number; episodeId: string }
+ | ({ kind: "ok"; value: T } & ConnectionRetryTally)
/**
* NOTE the absence of a `uiLeased` field. Whether a surface is painting this
* episode is LIVE and must be read at the moment the status is chosen; a
* boolean on this result IS a cache of it, and by the time a caller
* destructures the result it can already be stale.
*/
- | {
+ | ({
kind: "exhausted";
conn: ConnectionErrorInfo;
error: unknown;
endpoint: string;
- attempts: number;
- recoveryMs: number;
- episodeId: string;
- }
- | {
+ } & ConnectionRetryTally)
+ | ({
kind: "gave_up";
conn: ConnectionErrorInfo;
error: unknown;
endpoint: string;
- attempts: number;
- recoveryMs: number;
- episodeId: string;
- }
- | { kind: "client_gone"; attempts: number; recoveryMs: number; episodeId: string }
+ } & ConnectionRetryTally)
+ | ({ kind: "client_gone" } & ConnectionRetryTally)
/** A later attempt threw something that is NOT a connection error. Not ours. */
- | { kind: "threw"; error: unknown; attempts: number; recoveryMs: number; episodeId: string };
+ | ({ kind: "threw"; error: unknown } & ConnectionRetryTally);
/** True when the rejection is the inbound client going away, not our own clamp. */
function isClientAbort(signal: AbortSignal, err: unknown): boolean {
@@ -225,12 +254,29 @@ export async function withConnectionRetry<T>(
ctx: ConnectionRetryContext
): Promise<ConnectionRetryResult<T>> {
const clock = recoveryClock();
+ // PROCESS ms at which THIS request entered the ladder. `recovery_ms` is
+ // measured from here rather than from the episode, because an episode can be
+ // older than the request that joined it — a tier-2 re-entry inherits one —
+ // and a `recovery_ms` larger than the same record's `latency_ms` is a figure
+ // no reader can make sense of.
+ const enteredAtPerf = clock.now();
+ let requestRetries = 0;
+
const firstConn = classifyConnectionError(first);
if (!firstConn) {
// The caller is contractually required to classify before calling. Being
// defensive here costs nothing and keeps a future caller's mistake from
// silently widening what "transient" means.
- return { kind: "threw", error: first, attempts: 0, recoveryMs: 0, episodeId: "" };
+ return {
+ kind: "threw",
+ error: first,
+ attempts: 0,
+ recoveryMs: 0,
+ requestRetries: 0,
+ requestRecoveryMs: 0,
+ clientRetry: 0,
+ episodeId: "",
+ };
}
let conn: ConnectionErrorInfo = firstConn;
@@ -247,19 +293,27 @@ export async function withConnectionRetry<T>(
deadlineAt: ctx.deadlineAt,
});
- const finish = <R extends ConnectionRetryResult<T>>(r: R): R => r;
+ /**
+ * The counters, snapshotted at the instant an outcome is returned. Every
+ * `return` in this function goes through it, so a new counter is added in one
+ * place and cannot be forgotten on one arm — which is how five hand-written
+ * copies of the same four fields drift.
+ */
+ const tally = () => ({
+ attempts: handle.attempts(),
+ recoveryMs: handle.recoveryMs(),
+ requestRetries,
+ requestRecoveryMs: Math.round(clock.now() - enteredAtPerf),
+ clientRetry: handle.clientRetries(),
+ episodeId: handle.episodeId,
+ });
try {
handle.recordAttemptResult(conn.code, false);
for (;;) {
if (ctx.signal.aborted) {
- return finish({
- kind: "client_gone",
- attempts: handle.attempts(),
- recoveryMs: handle.recoveryMs(),
- episodeId: handle.episodeId,
- });
+ return { kind: "client_gone", ...tally() };
}
const delay = connectionRetryDelayMs(handle.ladderIndex());
@@ -268,15 +322,7 @@ export async function withConnectionRetry<T>(
// sleep. Bounding only the sleep is how a budget stops being a ceiling.
if (now + delay + MIN_ATTEMPT_SLOT_MS > ctx.deadlineAt) {
handle.handoff();
- return finish({
- kind: "exhausted",
- conn,
- error,
- endpoint,
- attempts: handle.attempts(),
- recoveryMs: handle.recoveryMs(),
- episodeId: handle.episodeId,
- });
+ return { kind: "exhausted", conn, error, endpoint, ...tally() };
}
let waited: { kind: "attempt" } | { kind: "gave_up" };
@@ -285,26 +331,13 @@ export async function withConnectionRetry<T>(
} catch (err) {
if (isClientAbort(ctx.signal, err)) {
log(`[Recovery] client gone during wait (episode ${handle.episodeId})`);
- return finish({
- kind: "client_gone",
- attempts: handle.attempts(),
- recoveryMs: handle.recoveryMs(),
- episodeId: handle.episodeId,
- });
+ return { kind: "client_gone", ...tally() };
}
throw err;
}
if (waited.kind === "gave_up") {
- return finish({
- kind: "gave_up",
- conn,
- error,
- endpoint,
- attempts: handle.attempts(),
- recoveryMs: handle.recoveryMs(),
- episodeId: handle.episodeId,
- });
+ return { kind: "gave_up", conn, error, endpoint, ...tally() };
}
// Per-attempt clamp. Cleared the moment the operation settles, so it can
@@ -322,28 +355,18 @@ export async function withConnectionRetry<T>(
const merged = AbortSignal.any([ctx.signal, attemptAc.signal]);
try {
+ requestRetries++;
const value = await op(merged);
handle.recordAttemptResult("ok", true);
log(
`[Recovery] ${ctx.providerDisplayName} recovered after ${handle.attempts()} attempts ` +
`in ${handle.recoveryMs()}ms (episode ${handle.episodeId})`
);
- return finish({
- kind: "ok",
- value,
- attempts: handle.attempts(),
- recoveryMs: handle.recoveryMs(),
- episodeId: handle.episodeId,
- });
+ return { kind: "ok", value, ...tally() };
} catch (err) {
if (isClientAbort(ctx.signal, err)) {
log(`[Recovery] client gone during attempt (episode ${handle.episodeId})`);
- return finish({
- kind: "client_gone",
- attempts: handle.attempts(),
- recoveryMs: handle.recoveryMs(),
- episodeId: handle.episodeId,
- });
+ return { kind: "client_gone", ...tally() };
}
// Re-classify EVERY attempt. Our own clamp lands here as a
// `TimeoutError`, which classification now recognises for the same
@@ -351,13 +374,7 @@ export async function withConnectionRetry<T>(
// out of a host we could not reach in time.
const next = classifyConnectionError(err);
if (!next) {
- return finish({
- kind: "threw",
- error: err,
- attempts: handle.attempts(),
- recoveryMs: handle.recoveryMs(),
- episodeId: handle.episodeId,
- });
+ return { kind: "threw", error: err, ...tally() };
}
conn = next;
error = err;
diff --git a/packages/cli/src/providers/probe-live.ts b/packages/cli/src/providers/probe-live.ts
index 5f11e5bb..ab394661 100644
--- a/packages/cli/src/providers/probe-live.ts
+++ b/packages/cli/src/providers/probe-live.ts
@@ -398,10 +398,23 @@ function extractErrorType(body: string): string | undefined {
export function classifyHttpError(status: number, body: string, latencyMs: number): ProbeResult {
const lowered = body.toLowerCase();
// A proxy-side connection failure (DNS unresolved, connection refused, host
- // unreachable) is tagged by composed-handler as a 503 connection_error. That's
- // a LOCAL network problem, not an upstream server error — classify it as
- // network-error so the row reads "network error" with the proxy's actionable
- // message ("check your network/DNS") instead of a generic "server error · 503".
+ // unreachable) is tagged by composed-handler as a **400** connection_error —
+ // NOT a 503, whatever this comment said until v9.2.x. That matters twice
+ // over. The status is 400 because a 503 makes Claude Code retry with the
+ // reason buried behind its own banner, and because `isRetryableError` would
+ // otherwise walk the fallback chain. So the test below keys off the `type`,
+ // which is status-agnostic and therefore survives both that choice and the
+ // terminal-error remap that shares the same 400.
+ //
+ // Probes never reach the retry ladder at all: `probeLink` sends
+ // `x-claudish-no-recovery: 1` and `shouldSkipTier1` honours it, so an
+ // unreachable provider still answers here in milliseconds instead of holding
+ // for the full recovery deadline and misreporting as a timeout.
+ //
+ // It is a LOCAL network problem, not an upstream server error — classify it
+ // as network-error so the row reads "network error" with the proxy's
+ // actionable message ("check your network/DNS") instead of a generic
+ // "server error".
if (extractErrorType(body) === "connection_error") {
return {
state: "network-error",
diff --git a/packages/cli/src/recovery/coordinator.ts b/packages/cli/src/recovery/coordinator.ts
index 4876c849..84ce467d 100644
--- a/packages/cli/src/recovery/coordinator.ts
+++ b/packages/cli/src/recovery/coordinator.ts
@@ -298,6 +298,14 @@ export interface EpisodeHandle {
attempts(): number;
/** Ms since the episode opened — not since this request started. */
recoveryMs(): number;
+ /**
+ * How many times a client has re-entered this episode after a tier-2
+ * handoff. 0 for the request that opened it.
+ *
+ * Read for stats, so that a record can say WHICH re-entry it was rather than
+ * leaving five records of one outage indistinguishable from five outages.
+ */
+ clientRetries(): number;
ladderIndex(): number;
/** LIVE. Never cache the result. */
uiLeaseValid(): boolean;
@@ -441,6 +449,7 @@ export function joinEpisode(seed: EpisodeSeed): EpisodeHandle {
key: episode.key,
attempts: () => episode.attempts,
recoveryMs: () => Math.round(recoveryClock().now() - episode.startedAtPerf),
+ clientRetries: () => episode.clientRetries,
ladderIndex: () => episode.ladderIndex,
uiLeaseValid: () => uiLeaseValid(episode.episodeId),
diff --git a/packages/cli/src/recovery/settings.ts b/packages/cli/src/recovery/settings.ts
index f0373466..3aa33b9a 100644
--- a/packages/cli/src/recovery/settings.ts
+++ b/packages/cli/src/recovery/settings.ts
@@ -60,16 +60,18 @@ export function resolveRecoveryEnabled(): boolean {
* May claudish own a surface on which a recovery episode's reason is legible?
* Default true.
*
- * NOTHING READS THIS YET, and that is expected. It exists now for two reasons.
- * The first is that the status decision it will gate — whether an exhausted
- * episode may answer a RETRYABLE status instead of an inline error — is a later
- * phase, and landing the switch early means that phase changes one branch
- * rather than four files. The second is the CLAUDE.md invariant it trips: a new
- * `ClaudishProfileConfig` field that is not in `loadConfig`'s allowlist AND
- * does not have a scoped reader survives on disk until the first global save
- * and is then dropped, silently. That trap has caught this codebase twice, so
- * the field is landed with its round-trip test the moment it is defined rather
- * than the moment it is first consumed.
+ * READ BY `claude-runner.ts`, which decides whether to wrap the session in a
+ * magmux pane. It is the USER'S half of the gate; the other half is
+ * `uiLeaseValid()`, which asks whether a renderer is painting THIS episode right
+ * now. Both must hold before an exhausted episode may answer a retryable 503 —
+ * the user has to have allowed a surface, and a surface has to actually exist.
+ *
+ * It was landed one phase before its first reader, deliberately, because of the
+ * CLAUDE.md invariant it trips: a new `ClaudishProfileConfig` field that is not
+ * in `loadConfig`'s allowlist AND has no scoped reader survives on disk until
+ * the first global save and is then dropped, silently. That trap has caught this
+ * codebase twice, so the field got its round-trip test the moment it was
+ * defined rather than the moment it was first consumed.
*
* It is a question about the USER'S CONFIGURATION and about nothing else. It
* must never be conflated with "can this process open a pane right now", which
diff --git a/packages/cli/src/stats-otlp.test.ts b/packages/cli/src/stats-otlp.test.ts
index a0dfa0be..bbcd31c9 100644
--- a/packages/cli/src/stats-otlp.test.ts
+++ b/packages/cli/src/stats-otlp.test.ts
@@ -216,3 +216,128 @@ describe("formatOtlpBatch", () => {
expect(result.resourceLogs).toEqual([]);
});
});
+
+/**
+ * THE GUARD FOR THE ONE SILENT FAILURE IN `stats-otlp.ts`.
+ *
+ * `eventToLogRecord` is a hand-written attribute allowlist. A field added to
+ * `StatsEvent` and populated by `stats.ts` but NOT pushed there is typed,
+ * type-checked, buffered and written to `~/.claudish/stats-buffer.json` — and
+ * never leaves the machine. Nothing throws. The field is simply missing from
+ * every dashboard built to read it, and the loss is discovered a quarter later,
+ * when someone asks a question the data cannot answer.
+ *
+ * Two layers, and the first is the durable one:
+ *
+ * 1. `satisfies Record<OptionalStatsKey, …>` makes the table below EXHAUSTIVE
+ * over `StatsEvent`'s optional fields AT COMPILE TIME. Add an optional field
+ * to the interface and this file stops compiling until it is listed here.
+ * 2. The table then drives emit-when-set and omit-when-unset for every entry,
+ * so listing a field without pushing it in `eventToLogRecord` is red.
+ *
+ * Together, a new optional field cannot reach `main` without its push. Layer 1
+ * is what stops this from being a checklist someone has to remember to read.
+ */
+type OptionalStatsKey = {
+ [K in keyof StatsEvent]-?: undefined extends StatsEvent[K] ? K : never;
+}[keyof StatsEvent];
+
+const OPTIONAL_WIRE = {
+ error_class: {
+ attribute: "llm.error_class",
+ sample: "network",
+ value: { stringValue: "network" },
+ },
+ error_code: {
+ attribute: "llm.error_code",
+ sample: "ECONNREFUSED",
+ value: { stringValue: "ECONNREFUSED" },
+ },
+ fallback_chain: {
+ attribute: "llm.fallback_chain",
+ sample: ["litellm", "openrouter"],
+ value: { arrayValue: { values: [{ stringValue: "litellm" }, { stringValue: "openrouter" }] } },
+ },
+ fallback_attempts: {
+ attribute: "llm.fallback_attempts",
+ sample: 2,
+ value: { intValue: "2" },
+ },
+ retry_attempts: {
+ attribute: "llm.retry_attempts",
+ sample: 5,
+ value: { intValue: "5" },
+ },
+ recovery_ms: {
+ attribute: "llm.recovery_ms",
+ sample: 201_900,
+ value: { intValue: "201900" },
+ },
+ recovery_episode_id: {
+ attribute: "llm.recovery_episode_id",
+ sample: "3f2b9c14-5d6e-4a71-9b0c-8e2d4f6a1b33",
+ value: { stringValue: "3f2b9c14-5d6e-4a71-9b0c-8e2d4f6a1b33" },
+ },
+ recovery_client_retry: {
+ attribute: "llm.recovery_client_retry",
+ sample: 3,
+ value: { intValue: "3" },
+ },
+ recovery_outcome: {
+ attribute: "llm.recovery_outcome",
+ sample: "handoff",
+ value: { stringValue: "handoff" },
+ },
+} satisfies Record<OptionalStatsKey, { attribute: string; sample: unknown; value: unknown }>;
+
+const OPTIONAL_ENTRIES = Object.entries(OPTIONAL_WIRE) as Array<
+ [OptionalStatsKey, { attribute: string; sample: unknown; value: unknown }]
+>;
+
+describe("eventToLogRecord — every optional field reaches the wire", () => {
+ for (const [field, spec] of OPTIONAL_ENTRIES) {
+ it(`emits ${spec.attribute} when ${field} is set`, () => {
+ const record = eventToLogRecord({ ...SAMPLE_EVENT, [field]: spec.sample } as StatsEvent);
+ const attr = record.attributes.find((a) => a.key === spec.attribute);
+ expect(attr).toBeDefined();
+ expect(attr?.value).toEqual(spec.value as any);
+ });
+
+ it(`omits ${spec.attribute} when ${field} is unset`, () => {
+ // Deleting rather than assigning undefined: `{ ...e, x: undefined }` leaves
+ // the KEY present, and the absent-key shape is the one production emits.
+ const bare = { ...SAMPLE_EVENT } as Record<string, unknown>;
+ delete bare[field];
+ const record = eventToLogRecord(bare as unknown as StatsEvent);
+ expect(record.attributes.some((a) => a.key === spec.attribute)).toBe(false);
+ });
+ }
+
+ it("emits retry_attempts: 0 — zero is a fact, not an absence", () => {
+ // The episode existed and this request added no re-issue to it. A truthiness
+ // guard (`if (event.retry_attempts)`) instead of `!== undefined` deletes this
+ // record's only evidence that recovery ran at all, and deletes it precisely
+ // on the requests that were handed off with no budget left.
+ const record = eventToLogRecord({
+ ...SAMPLE_EVENT,
+ retry_attempts: 0,
+ recovery_ms: 0,
+ recovery_client_retry: 0,
+ recovery_episode_id: "3f2b9c14-5d6e-4a71-9b0c-8e2d4f6a1b33",
+ });
+ expect(record.attributes.find((a) => a.key === "llm.retry_attempts")?.value).toEqual({
+ intValue: "0",
+ } as any);
+ expect(record.attributes.find((a) => a.key === "llm.recovery_client_retry")?.value).toEqual({
+ intValue: "0",
+ } as any);
+ });
+
+ it("a healthy request carries no recovery attributes at all", () => {
+ // The other half of NFR-1: a machine that never loses its network sends the
+ // same bytes it sent before this feature existed.
+ const keys = eventToLogRecord(SAMPLE_EVENT).attributes.map((a) => a.key);
+ expect(keys.filter((k) => k.startsWith("llm.recovery_"))).toEqual([]);
+ expect(keys).not.toContain("llm.retry_attempts");
+ });
+});
diff --git a/packages/cli/src/stats-otlp.ts b/packages/cli/src/stats-otlp.ts
index 1a197d83..70a2ece6 100644
--- a/packages/cli/src/stats-otlp.ts
+++ b/packages/cli/src/stats-otlp.ts
@@ -7,8 +7,20 @@
* Wire format: OTLP JSON Logs
* Signal type: LogRecord per request
* Namespace: llm.* for custom attributes, standard OTel for resource/HTTP
+ *
+ * ── THE ONE TRAP IN THIS FILE ────────────────────────────────────────────────
+ *
+ * `eventToLogRecord` below is a HAND-WRITTEN ALLOWLIST. A field added to
+ * `StatsEvent` and populated by `stats.ts` but not pushed there is typed,
+ * type-checked, buffered, and written to `~/.claudish/stats-buffer.json` — and
+ * never leaves the machine. Nothing fails; the number is simply absent from
+ * every dashboard that was built to read it. Add the push, and add a case to
+ * `stats-otlp.test.ts`'s emit/omit table, which exists to make the omission
+ * cost a red test instead of a silent quarter.
*/
+import type { RecoveryOutcome } from "./recovery/coordinator.js";
+
// ─── Interfaces ───────────────────────────────────────────────────────────────
/**
@@ -46,6 +58,61 @@ export interface StatsEvent {
fallback_chain?: string[]; // provider names tried, in order
fallback_attempts?: number; // how many failed before success
+ // ── Network recovery ────────────────────────────────────────────────────────
+ // Present only on a request that hit a classified connection failure. On every
+ // healthy request all five are absent, and `eventToLogRecord` emits nothing
+ // for them — so a machine that never loses its network sends exactly the
+ // bytes it sent before this feature existed.
+
+ /**
+ * Connect retries THIS REQUEST made against the SAME provider.
+ *
+ * NOT `fallback_attempts`, which counts how many DIFFERENT providers were
+ * tried. The two answer opposite questions and a chain that never advanced
+ * can still have retried thirty times.
+ *
+ * Per-request, not per-episode: one episode can serve several concurrent
+ * requests and can outlive any of them, so the episode's cumulative count
+ * would over-report every request in it. `recovery_episode_id` is what joins
+ * them back together.
+ */
+ retry_attempts?: number;
+
+ /**
+ * Ms of this request's own `latency_ms` spent inside the retry ladder —
+ * backoff waits plus failed connects.
+ *
+ * `latency_ms` deliberately still INCLUDES that time (`adapters.md`: "the
+ * honest figure is time-to-usable-response"). This field is what makes the
+ * resulting skew explicable rather than mysterious: `latency_ms: 246_000,
+ * recovery_ms: 201_900` reads as a 44-second turn behind a three-and-a-half
+ * minute outage. Without it, the same record reads as a four-minute model.
+ */
+ recovery_ms?: number;
+
+ /**
+ * The episode this request belonged to — the correlation id, and the only
+ * field that makes "how often does recovery happen" answerable.
+ *
+ * One outage is one episode and may span several requests (each tier-2
+ * handoff is a new request re-entering the same episode), so counting
+ * records over-counts outages by the retry factor.
+ * `count(distinct recovery_episode_id)` is the honest figure.
+ *
+ * A request that hits BOTH an auth-path episode and a fetch-path episode
+ * carries the id of the one that DECIDED ITS STATUS. An auth episode that
+ * recovered has no record of its own: its attempts fold into this request's
+ * `retry_attempts` and its waits into this request's `recovery_ms`, because
+ * there is exactly one record per request.
+ */
+ recovery_episode_id?: string;
+
+ /** Which tier-2 re-entry this request is. 0 = the original request. */
+ recovery_client_retry?: number;
+
+ /** How the episode ended for this request. */
+ recovery_outcome?: RecoveryOutcome;
+
// Invocation
invocation_mode: string; // "profile" | "explicit-model" | "auto-route" | "env-var" | "model-map"
@@ -219,6 +286,25 @@ export function eventToLogRecord(event: StatsEvent): OtlpLogRecord {
attributes.push(intAttr("llm.fallback_attempts", event.fallback_attempts));
}
+ // Optional network-recovery fields. THIS IS THE STEP WHOSE OMISSION IS
+ // SILENT — see the file header. Each is emitted only when set, so a healthy
+ // request's record is byte-identical to what it was before recovery existed.
+ if (event.retry_attempts !== undefined) {
+ attributes.push(intAttr("llm.retry_attempts", event.retry_attempts));
+ }
+ if (event.recovery_ms !== undefined) {
+ attributes.push(intAttr("llm.recovery_ms", event.recovery_ms));
+ }
+ if (event.recovery_episode_id !== undefined) {
+ attributes.push(stringAttr("llm.recovery_episode_id", event.recovery_episode_id));
+ }
+ if (event.recovery_client_retry !== undefined) {
+ attributes.push(intAttr("llm.recovery_client_retry", event.recovery_client_retry));
+ }
+ if (event.recovery_outcome !== undefined) {
+ attributes.push(stringAttr("llm.recovery_outcome", event.recovery_outcome));
+ }
+
return {
timeUnixNano,
severityNumber: 9, // INFO
diff --git a/packages/cli/src/stats.ts b/packages/cli/src/stats.ts
index 80665ae2..2ab2c23c 100644
--- a/packages/cli/src/stats.ts
+++ b/packages/cli/src/stats.ts
@@ -181,6 +181,19 @@ export function recordStats(partial: Partial<StatsEvent>): void {
if (partial.fallback_chain !== undefined) event.fallback_chain = partial.fallback_chain;
if (partial.fallback_attempts !== undefined)
event.fallback_attempts = partial.fallback_attempts;
+ // Network recovery. Absent on every healthy request, and absent means
+ // absent — a `?? 0` here would put `retry_attempts: 0` on every record
+ // ever sent and make "did this turn retry" unanswerable without a join.
+ // Copying them here is only half the job: `eventToLogRecord` in
+ // stats-otlp.ts is a hand-written allowlist, and a field that reaches this
+ // block but not that one is buffered to disk and never sent.
+ if (partial.retry_attempts !== undefined) event.retry_attempts = partial.retry_attempts;
+ if (partial.recovery_ms !== undefined) event.recovery_ms = partial.recovery_ms;
+ if (partial.recovery_episode_id !== undefined)
+ event.recovery_episode_id = partial.recovery_episode_id;
+ if (partial.recovery_client_retry !== undefined)
+ event.recovery_client_retry = partial.recovery_client_retry;
+ if (partial.recovery_outcome !== undefined) event.recovery_outcome = partial.recovery_outcome;
appendEvent(event);