Claude Code. Any Model. The most powerful AI coding agent now speaks every language.
feat(recovery): count the outages, and write down why the shape is what it is
Jack Rudenkoca5b4eb92a15
diff --git a/CLAUDE.md b/CLAUDE.mdindex 6c28a187..9d2de308 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -11,6 +11,9 @@ lives in `ROADMAP.md`.- `routing.md` — `provider@model` syntax, every provider prefix, catalog-gathered route candidates and their tier order, `defaultProvider` as the fallback position, the derived picker provider list, `SUBSCRIPTION_PROVIDERS`, local models- `adapters.md` — Layers 1–3, stream parsers, error classification and retry, the 400 remap, why Gemini tool schemas need `items` on every array+- `network-recovery.md` — the two-tier connection hold: the DERIVED deadline (`API_TIMEOUT_MS`, not a+ watchdog), why tier 2 answers 503 and never 429, why the pane lease is heartbeat-driven, the+ five auth sites- `behavior-layer.md` — Layer 4, the harness-conformance supervisor- `advisor.md` — `--advisor` for any main model: independent of `--monitor`, decorator on the routed handler, ids by tool name, retained session state, stub paths S1-S10, Claude Code's gates, metered panel billing; read before editing advisor, decorator, monitor-launch or native-auth code- `providers/devin.md`, `providers/grok-subscription.md`, `providers/antigravity.md`, `providers/qwen-alibaba.md` — one per reverse-engineered provider@@ -113,6 +116,8 @@ negative case: if you are about to type one of those words, stop and use the lef- `gk@` is the Grok SUBSCRIPTION; `grok@`/`xai@` is the metered `x-ai`. `moonshot-cn@` is a different service from `moonshot@`.- A bare Claude name (`claude-opus-5`, `opus`, `internal`) must never reach `route()`: `native-anthropic` has no credential store, so the credential filter drops it and the chain degrades to OpenRouter. Check `nativeRouteFor()` (`providers/native-route.ts`) FIRST, as the proxy does — `preflight` and the TUI probe once didn't and told agents to drop their own subscription's models. Native routes also cannot be probed from outside a session (the handler forwards the inbound Claude Code header); report them as a third outcome, never as success or failure.- A new `ClaudishProfileConfig` field MUST be added to `loadConfig`'s allowlist in `profile-config.ts`; otherwise it survives on disk until the first global save and is then dropped. Bit `onepasswordEnvironments`, then `keychain`.+- A new `StatsEvent` field MUST also be pushed in `eventToLogRecord` — `stats-otlp.ts` is a HAND-WRITTEN attribute allowlist, so a field wired only into the interface and `stats.ts` is typed, buffered to `~/.claudish/stats-buffer.json`, and never sent. Nothing errors; the number is just missing from every dashboard, a quarter later. `stats-otlp.test.ts`'s table is `satisfies Record<OptionalStatsKey, …>`, so adding the field breaks compilation until it is listed there too — keep that.+- A connection failure can arrive wearing an AUTH status code, at FIVE sites in `composed-handler.ts` (`refreshAuth`, `forceRefreshAuth`, the parameter-recovery re-fetch, `getHeaders`, the primary fetch). 401 is retryable to `FallbackHandler`, so answering it for a network fault walks a subscription user onto metered billing mid-outage; an unclassified THROW is the same bug one layer out (`{status: 0}` + an unconditional advance, with no cost warning). Classify first or rethrow unchanged — never invent a status. `getHeaders()` is the non-obvious one: for `gk@` it is the request's first network touch, and refresh-conditional, so it is rare rather than safe.## Commandsdiff --git a/ai-docs/architecture/README.md b/ai-docs/architecture/README.mdindex f87213f5..bfb57f0f 100644--- a/ai-docs/architecture/README.md+++ b/ai-docs/architecture/README.md@@ -27,6 +27,7 @@ three earlier write-ups this material cites were lost.| [`onepassword.md`](onepassword.md) | SDK-only secret resolution, the FOUR causes of an identical denial error, the cross-process handshake lock, parent-side route pinning, and the config TUI tab | touch `providers/onepassword*.ts`, credential pre-hydration, or the 1Password TUI tab || [`keychain.md`](keychain.md) | The macOS Keychain backend: the 17ms-vs-28ms measurement that split presence from values, the two `security` behaviours that are traps (hex reads, a "secure" write that stores nothing and exits 0), and why no new `CredentialSource` member | touch `providers/keychain.ts`, `keychain-source.ts`, or the Providers tab's key write/delete || [`adapters.md`](adapters.md) | Layers 1–3 (FormatConverter / ModelTranslator / ProviderTransport), stream-parser selection, errors that ride an HTTP 200, and the `upstream_status` remap with its downstream readers | add a provider, model translator, or stream parser; or change error classification and retry |+| [`network-recovery.md`](network-recovery.md) | The two-tier connection hold: why the deadline is DERIVED from `API_TIMEOUT_MS` (measured 359.607 s, a default and not a watchdog), why the tier-2 status is 503 and never 429, why the pane's lease is heartbeat-driven and not frame-driven, and the five sites where a network failure wore an auth status code | touch the retry ladder, `recovery/`, the tier-2 status, the recovery pane, or any `x-claudish-recovery` reader || [`behavior-layer.md`](behavior-layer.md) | Layer 4, the harness-convention SUPERVISOR: the rule engine, the four hooks, the 115-call plan-mode measurement it was built on, and opt-in telemetry | write a `behavior/` rule, or change harness-conformance handling || [`advisor.md`](advisor.md) | `--advisor` for any main model: why it is independent of `--monitor` yet keeps four of its launch bits, the decorator around the ROUTED handler, id capture by tool name (the `toolu_` prefix that made it a no-op on every foreign provider), the tee and retained session-keyed state, the ten stub paths and advice origin, Claude Code's advisor gates, metered panel billing, and the inherited-placeholder 401 | touch `--advisor`, `advisor-decorator.ts`, `native-handler-advisor.ts`, `advisor-startup.ts`, monitor-gated launch code, or native auth in `claude-runner.ts` || [`headless-vs-interactive.md`](headless-vs-interactive.md) | Why `-p` is NOT interactive-without-a-TTY: `--agent` validation is SILENTLY skipped under `--input-format stream-json` (measured, exit 0, no stream frame, `result: success`), upstream closed it as not planned, and why magmux is in the dependency tree | drive Claude Code headlessly, add a child-spawn flag, or assume a headless run behaves like an interactive one |diff --git a/ai-docs/architecture/adapters.md b/ai-docs/architecture/adapters.mdindex 7af0fa2a..a07dc9c3 100644--- a/ai-docs/architecture/adapters.md+++ b/ai-docs/architecture/adapters.md@@ -251,7 +251,11 @@ Every retry hook in claudish keys off the HTTP **status** (`anthropic-compat.ts`- **Terminal** in-stream errors (`context_length_exceeded`, `invalid_request_error`) are NOT retried — they keep the existing inline-text treatment, which is the actionable path for them.- **Anything else** (any content event) → `clean`, and the consumed bytes are **replayed byte-identically** so the real parser sees an unchanged stream.-This is the one place the 400-not-503 doctrine (`composed-handler.ts` ~line 461) is deliberately inverted. That rule exists because a 503 makes Claude Code show "API error · Retrying · attempt N/10" with the real reason buried — correct for **terminal** faults, where retrying is theatre. An upstream overload is the opposite: genuinely transient, and the retry banner is the appropriate behaviour because retrying is the actual remedy. Terminal → 400 inline; transient-after-our-own-retries → 503.+This is **one of two** places the 400-not-503 doctrine (`composed-handler.ts`'s `respondConnectionError`) is deliberately inverted. That rule exists because a 503 makes Claude Code show "API error · Retrying · attempt N/10" with the real reason buried — correct for **terminal** faults, where retrying is theatre. An upstream overload is the opposite: genuinely transient, and the retry banner is the appropriate behaviour because retrying is the actual remedy. Terminal → 400 inline; transient-after-our-own-retries → 503.++The second inversion is network recovery's tier-2 handoff, which earns the same 503 by a different argument — not "the banner is the remedy" but "a pane is painting the reason, so the status no longer has to carry it". See [`network-recovery.md`](network-recovery.md).++> This reference was `~line 461` until v9.2.x, then really `:665-675`, and is now inside a named method. **Cite the symbol, not the line.** Three separate readers have been sent to the wrong part of this 2,000-line file by a number that was correct when it was written.**Trade-off to know:** sniffing withholds response headers until the first decisive event, capped by `DEFAULT_SNIFF_BUDGET_MS` (12s, chosen above the 0.85s–7.7s error latencies observed in the real log). On a healthy xhigh-reasoning turn that delays `message_start` by however long the model thinks before its first output item. No content is lost or reordered — the client shows a spinner either way — but time-to-first-byte is genuinely later than before. Past the budget claudish flushes and degrades gracefully to the inline-text path.diff --git a/ai-docs/architecture/network-recovery.md b/ai-docs/architecture/network-recovery.mdnew file mode 100644index 00000000..dd5792ee--- /dev/null+++ b/ai-docs/architecture/network-recovery.md@@ -0,0 +1,458 @@+> Two-tier connection recovery: the in-request retry ladder, the magmux recovery pane, and the+> 503 handoff that lets Claude Code's own retry loop carry an outage the socket cannot.+>+> Indexed in [`README.md`](./README.md). Status-code neighbours: [`adapters.md`](adapters.md).++# Network recovery — why "retry forever" is two tiers and not one loop++A transport-unreachable failure used to end a turn. `classifyConnectionError` returned non-null,+`respondConnectionError` answered `400 connection_error`, and the transcript carried a dead turn the+user had to retype. VPN flaps, a Tailscale MagicDNS outage, a stopped `ollama serve`, a laptop lid+closing between two tool calls — all of them, terminal.++The fix is **not** an unbounded retry loop, and this document exists mostly to record *why not*,+because the reasons are measurements that the source cannot show you.++---++## 1. The client's clock is the real constraint, and it is a KNOB++The obvious design is to hold the inbound request open and retry until the network returns. Phase 0+measured what actually happens when you do.++**Claude Code aborts the connection at 359.607 s** — reproduced at 359.4 / 360.1 / 360.1 across two+processes, against a custom `ANTHROPIC_BASE_URL`. The important part is what that number *is*:++> It is **not** a fixed watchdog. It is `API_TIMEOUT_MS`'s default of 360 000 ms. Setting+> `API_TIMEOUT_MS=20000` produced six aborts at **exactly 20.000 s**.++That single fact decides the shape of the feature twice.++1. **A hold has a hard ceiling.** Whatever we do, the socket dies at the client's timeout. "Retry+ forever inside one request" is not implementable — not as a trade-off, as an impossibility.+2. **The deadline must be DERIVED, never written down.** A user with `API_TIMEOUT_MS=60000` gets a+ client that gives up at 60 s. A hardcoded 270 s deadline means *every* episode on that machine+ ends in `client_gone` and recovery never engages — silently, and only on their machine. So:++```+TIER1_DEADLINE_MS = max(FLOOR, min(API_TIMEOUT_MS ?? 360_000, 300_000) − 30_000)+```++`recovery/settings.ts` owns that computation and logs a `[Recovery]` line whenever the environment+shortens the budget, so a short hold is visible rather than mysterious. This is CLAUDE.md's+*"a default is a rule, never a pinned id"* arriving in the time domain.++The other two ceilings the derivation clears:++| ceiling | value | where |+|---|---|---|+| Bun's per-request idle timeout | 255 s | `proxy-server.ts`'s `idleTimeout` |+| the unclamped attempt 1 | up to 45 s | reserved out of the budget, never clamped (NFR-1) |++Bun's 255 s stops binding because `c.env.timeout(c.req.raw, 0)` **works** — measured, a request+given `0` survived **6×** its `idleTimeout` where the un-disarmed control died at 1.6×. That call is+made inside the `catch`, never on a healthy request.++### Claude Code's retry budget is a COUNT, not a duration++The second tier hands the retry back to the client. What the client will then spend is:++- **~11 attempts** on its default budget, spanning **~174 s** for a 503 chain (backoff caps at+ 38.4 s per gap, and `retry-after` is honoured verbatim);+- **~300 attempts** with `CLAUDE_CODE_RETRY_WATCHDOG=1`, which claudish sets when the recovery UI is+ enabled — reaching **~a day** of unattended recovery on a hung connection.++It is a count, not a clock. That is why `EPISODE_GRACE_MS` is 120 s: the client's worst gap is+38.4 s, so 120 s is 3.1× inside it and a returning client always finds its episode still open.++**The watchdog is a deliberate, informed acceptance of RISK-6** — see §7.++---++## 2. The two tiers, and the episode that makes them one thing++```+POST /v1/messages ──► ComposedHandler.handle() startTime = performance.now()+ attempt 1 = today's expression, BYTE-IDENTICAL ──► success+ │ throws+ ▼+ classifyConnectionError() ── null ──► rethrow, unchanged+ │ non-null+ ▼+ shouldSkipTier1()? probe header · recovery disabled · no budget left+ │ no+ ┌───────────────────▼──────────────────────────────────────────┐+ │ TIER 1 — in-request, PRE-FLUSH, deadline derived from §1 │+ │ joinEpisode(key) ONE episode per target, N waiters │+ │ 5 → 10 → 30 → 60 → 60 … on a SHARED clock │+ │ re-issue THROUGH provider.enqueueRequest, re-classified │+ │ success ─► return the upstream Response ────────────────────┼──►+ └───────────────────┬──────────────────────────────────────────┘+ │ deadline reached+ ┌───────────────────▼──────────────────────────────────────────┐+ │ TIER 2 — hand the retry back │+ │ uiLeaseValid(episodeId) ? 503 overloaded_error │+ │ + x-should-retry: true │+ │ + x-claudish-recovery: 1 │+ │ : 400 connection_error (as before) │+ │ the episode stays OPEN for EPISODE_GRACE_MS │+ └───────────────────┬──────────────────────────────────────────┘+ │ Claude Code's own retry re-POSTs and JOINS+ │ the same episode — one continuous banner+```++**The episode is what makes two tiers read as one recovery.** It is a process-level singleton keyed+by `${provider}|${host}`. A handoff leaves it open; the client's re-issued request finds it,+increments `clientRetries`, and continues the *same* attempt counter, ladder position and banner —+while getting a *fresh per-request deadline*, because the deadline belongs to the socket and the+socket is new.++Three lifetimes, and none of them is the state machine. Conflating them was the largest defect of+the superseded design:++| lifetime | scope | owner | ends when |+|---|---|---|---|+| the **deadline** | one inbound request | the waiter | its own `deadlineAt` passes — *that waiter alone* answers and leaves |+| the **ladder** | one episode (`key`) | the coordinator | the last waiter leaves, or an attempt succeeds |+| the **lease** | one pane × one episode | the UI manager | `UI_LEASE_MS` after the last heartbeat naming that episode |++When one waiter's deadline moved the *shared* episode into `handoff` — a state with no timer — every+other parked waiter silently stopped being retried and answered having attempted nothing. The+visible symptom was a banner reading `waiters: 4` with one of them actually being retried.++### The measured ladder++Real clock, real refused loopback socket, through `ComposedHandler.handle()`:++| attempt | fires at | gap |+|---|---|---|+| 1 | t+0 | — |+| 2 | t+5.002 s | 5002 ms |+| 3 | t+15.004 s | 10002 ms |+| 4 | t+45.009 s | 30005 ms |+| 5 | t+105.010 s | 60001 ms |+| 6 | t+165.011 s | 60001 ms |+| 7 | t+225.012 s | 60001 ms |+| — | handoff at t+225.018 s, deadline 270 s | |++18 ms of drift over 225 seconds. **The last array element repeats and the DEADLINE ends the+schedule** — which inverts `STREAM_RETRY_DELAYS_MS`, where the array running out *is* the budget.+The two schedules are deliberately separate: merging them would couple the sniffer's 12 s budget and+a duration quoted verbatim in a user-facing message to this one.++On a tier-2 rejoin the ladder does **not** reset. The target has been down for the whole of it, and+restarting at 5 s would hammer a dead host harder the longer the outage lasted.++### `PER_ATTEMPT_CONNECT_CAP_MS` is 45 s because 75 s collides with the OS++macOS gives up on its own TCP connect at **75.004 s** (measured against the unrouted `192.0.2.1`; a+refused loopback port is 5 ms). A 75 000 ms clamp races the operating system by **4 ms** and is+decided differently on different runs, which makes every assertion keyed on "the attempt ended at+the clamp" flaky by construction. 45 s clears the collision by 30 seconds and always wins.++**Slow and fast connect failures are indistinguishable BY CODE on macOS.** `192.0.2.1:443` reports+`ECONNREFUSED` / `ConnectionRefused` — the same code a loopback port refuses instantly with. Only+elapsed time separates them, so nothing may branch on the code to tell them apart. `isLoopback(endpoint)`+is the only honest discriminator, and it reads the address, not the error.++---++## 3. The status table — and why 503 and not 429++| condition | status | headers | why |+|---|---|---|---|+| recovered inside tier 1 | the upstream's own | — | the caller carries on as if the first attempt had worked |+| exhausted, **a lease is valid** | **503** `overloaded_error` | `x-should-retry: true`, `x-claudish-recovery: 1` | the reason is legible on the pane, so the retry may be handed back |+| exhausted, **no lease** | **400** `connection_error` | — | nothing can display the reason, so it must ride the status |+| `[q] give up` | **400** `connection_error` | — | guarded on `result.kind`, never on the lease — see below |+| client disconnected | 499 (unread) | — | the socket is already gone |+| unclassifiable throw on any attempt | rethrown unchanged | — | recovery must not widen what "transient" means |++### 429 would have been a live billing bug++`fallback-handler.ts` contains, verbatim:++```ts+// Rate limited — per-provider limit, a different provider may have capacity+if (status === 429) return true;+```++A 429 from the exhausted network path makes `FallbackHandler` **advance to the next provider**.+During a network outage the next provider is unreachable for the same reason, so the chain burns+every candidate — each paying its own multi-minute budget — and per CLAUDE.md's standing invariant,+an advance off a `SUBSCRIPTION_PROVIDERS` candidate onto a metered one quotes real money for a fault+no provider caused.++503 does not have this problem: `isRetryableError` has **no 503 branch**. It is the status the house+already owns for "transient after our own retries, do not switch the user's provider" — the stream+sniffer's exhaustion arm uses it for exactly that reason (`adapters.md`).++529 was considered and rejected: it is absent from `exhaustedChainStatus`'s transient set and would+add a status this codebase has never carried, re-opening every `status ===` under `handlers/`.++### Chain-safety is STRUCTURAL, not wording-based — and ORDER is the property++A 503 stopping the chain is not enough, twice over:++- **`isRetryableError`'s FIRST statement is `hasQuotaExhaustionWording(errorBody)`**, deliberately+ status-agnostic (it exists *because* of the 400 remap), and its phrase list contains the **bare+ substring `"quota"`**. A 503 whose *message* happened to carry that word advanced the chain and+ spent the user's money.+- **`exhaustedChainStatus` could demote it.** With an earlier candidate already failed, a+ non-retryable response goes to `formatCombinedError`, whose status is 503 only if *every*+ accumulated error is transient. One earlier auth or 404 failure turns our 503 into a terminal 400+ and the client never re-POSTs.++So the marker is a **header**, `x-claudish-recovery: 1`, checked in two independent places:+`handle()` returns a marked response VERBATIM before reading the body (defeating the combining), and+`isRetryableError` returns `false` on it **above** the quota match (defeating the wording).++> **Moving the marker check below the quota match — present, but LATE — is a live billing bug.**+> It is mutation-covered as one, and the mutation that kills it is the quota-wording variant+> specifically. A test whose 503 body says nothing about quota cannot see the defect at all.++The marker cannot be forged: every non-ok exit from `ComposedHandler` is `c.json(...)`, which builds+headers from nothing, and the only path that copies upstream headers verbatim+(`stream-head-sniffer.ts`'s `replayResponse()`) runs after `!response.ok` has already returned.+**If a future edit ever returns an upstream `Response` object on a non-ok path, the marker must be+stripped there.**++### `[q] give up` is guarded on the OUTCOME, never on the lease++A lease is by definition valid at the moment the give-up key is pressed — the pane that took the+keystroke is alive. A lease-only test would answer a retryable 503, Claude Code would immediately+re-ask, and the give-up key would be a no-op with a banner still on screen.++---++## 4. The lease — why the heartbeat must not be driven by frames++The generalisable rule the two exhaustion arms encode:++> **A retryable status is permissible exactly when claudish still has a surface on which the reason+> is legible.** Absent such a surface, the reason must ride the status, which means 400.++The gate is not a boolean and not a latch. A latch set on `hello` and never cleared is true after+`[q] give up`, after a killed pane, after a dead magmux and after a socket EOF — and the exhaustion+arm then answers 503 with the reason visible nowhere, *which is strictly worse than the bug this+feature exists to remove*. It is a **lease**:++```+uiLeaseValid(episodeId)+ ⇔ the proxy itself opened a pane and holds the `open_pane` reply+ ∧ some connected client has HEARTBEATED `ack` naming THIS episode within UI_LEASE_MS+```++Every clause works. *Naming this episode* makes it episode-scoped, so two concurrent episodes cannot+borrow one pane's legitimacy. *`ack`, not `hello`* makes it **post-render** — a process can send+`hello` before painting anything. *Within `UI_LEASE_MS`* makes it self-clearing on EOF, crash and+freeze, with no cleanup path to forget. *The proxy opened the pane* means a forged same-uid client+cannot manufacture the forbidden 503-with-no-banner state.++**"Heartbeated", not "renewed by frame receipts" — this is the single most consequential word here,+and the superseded design had it wrong.** Frames were emitted only while the episode was `waiting`.+There is no tick during `attempting`. A real `unreachable` connect takes 20–75 s (`192.0.2.1`+measured at **75 005 ms**), which is 2×–7.5× `UI_LEASE_MS` — so the lease expired *while the banner+was alive and painted*, and the exhaustion arm answered 400 for exactly the failure class that+motivated the feature.++The lease is a statement about **the renderer being alive and painting this episode**, so it is+renewed by the renderer on its own `UI_HEARTBEAT_MS` timer and by nothing else. The proxy's+`FRAME_TICK_MS` tick — now emitted in *every* live state, not only `waiting` — is then free to serve+the banner rather than the gate.++**Every loopback-only test passed over this bug.** A refused loopback connect resolves in ~1 ms, so+the ladder is all `waiting` and frames never stop. The regression is only visible with a slow+connect, which is why the unit test fakes a 45-second attempt (4.5 lease windows, **not one frame+emitted**) and the integration test runs a real pane against a 20-second lease clock.++No transition in the episode's table touches the lease, and the lease never causes a transition. The+retry loop is not the pane's business, and the pane's liveness is not the deadline's business.++---++## 5. A connection failure can wear an AUTH status code — at five separate sites++This is the half of the work that shipped on its own merits and would have been worth doing with no+recovery ladder at all.++`fallback-handler.ts` reads **401 as retryable**. So a network outage during a token refresh did not+fail the request — it **advanced the chain**, moving a subscription user onto a per-token candidate+mid-outage, for a fault that had nothing to do with their credentials. Reproduced:++```+[Fallback] Sakana Fugu failed (HTTP 401), trying next provider...+[Fallback] Metered Fallback succeeded after 1 failed attempt(s)+```++An *unclassified throw* leaving `handle()` is the same bug one layer out: it lands in+`fallback-handler.ts`'s catch, which pushes `{ status: 0 }` and advances **unconditionally** —+without even the per-token cost warning, which sits on the non-throwing branch. On a single-candidate+route no `FallbackHandler` exists, so it lands in `proxy-server.ts` as a bare 500 instead.++The inventory — every outbound call in `ComposedHandler` that can throw without a `Response`:++| # | site | before | after |+|---|---|---|---|+| 1 | `refreshAuth()` catch | classified **never** — unconditional **401** | classify FIRST, before `err.terminal` |+| 2 | `forceRefreshAuth()` catch (wraps the 401-retry `fetch` too) | classified **never** — unconditional **401** | classify FIRST |+| 3 | parameter-recovery re-fetch | **no `try` at all** — escaped `handle()` → `{status: 0}` + advance | wrapped; classified → 400, unclassified → rethrown |+| 4 | `getHeaders()` | **outside every `try`** — escaped `handle()`, or a bare 500 | wrapped; classified → 400, unclassified → rethrown |+| 5 | primary `fetch` / `enqueueRequest` | already classified — the control | unchanged, refactored onto the shared helper |++**Site 4 is worth a paragraph.** `getHeaders()` looks like a pure accessor and is not one. For `gk@`+it is the request's *first* network touch: `grok-subscription.ts` → `grok-credential.ts` →+`resolveGrokAccessToken()` → `fetch(auth.x.ai/oauth2/token)`. The touch is **refresh-conditional**,+which makes the escape rare — not safe. Rarity is what kept it alive through three reviews.++The rule all five now obey: **classify first, or rethrow unchanged; none may invent a status.** There+is exactly one way to answer "claudish could not reach the host" — `respondConnectionError()` — and+it answers 400 `connection_error`, never 401 and never a bare 500.++Local transports needed evidence preservation before any of this could reach them: `local.ts` used+to swallow its connect error in `catch (e: any) { log(…) }` and throw a bare+`Error("Cannot connect to Ollama at …")`. `findConnectionCode` found no `.code`, no `.cause` and no+message match, so `classifyConnectionError` returned **null** and a stopped Ollama — the single most+common local failure — never reached tier 1 at all. Same class of fix in `vertex-oauth.ts`+(discarded `code`), `openai.ts` (neither `code` nor `cause` on its error classes) and+`antigravity.ts` (the banner must name the *auth* host, not the eventual inference endpoint).++`healthChecked` latches on **success only**, so a retried `refreshAuth()` re-probes instead of+returning a false success to attempt two.++---++## 6. Stats — and the one step whose omission is silent++Five optional fields on `StatsEvent`, absent on every healthy request:++| field | scope | what it answers |+|---|---|---|+| `retry_attempts` | **request** | how many times *this request* re-issued against the same provider. Not `fallback_attempts`, which counts different providers |+| `recovery_ms` | **request** | how much of this record's `latency_ms` was backoff and failed connects |+| `recovery_episode_id` | episode | the correlation key — `count(distinct …)` is the only honest answer to "how often" |+| `recovery_client_retry` | episode | which tier-2 re-entry this request is; 0 = the original |+| `recovery_outcome` | episode | `recovered` / `handoff` / `client_gone` / `gave_up` |++**`latency_ms` continues to INCLUDE the waits**, per the standing decision in `adapters.md` ("the+honest figure is time-to-usable-response"). `recovery_ms` exists so the resulting skew is explicable+rather than mysterious: `latency_ms: 246_000, recovery_ms: 201_900` reads as a 44-second turn behind+a three-and-a-half-minute outage. Without the second field, the same record reads as a four-minute+model.++**Why two scopes.** The episode's cumulative counters cannot stand in for the per-request ones: N+concurrent requests share one episode, so each would report the sum of all of them, and a tier-2+re-entry inherits counters from a request that already recorded its own. Summing `retry_attempts`+across an outage would multiply it by the number of waiters and again by the number of re-entries.+The episode-scoped figures are what the *banner* quotes, because the banner is about the outage.++**The cardinality rule.** `stats-otlp.ts` says verbatim "one per LLM request". A request that hits+both an auth-path episode and a fetch-path episode carries the id of the episode that **decided its+status** — the later one. An auth episode that *recovered* emits no record of its own: its attempts+fold into the same request's `retry_attempts` and its waits into the same `recovery_ms`. The+coordinator emits **no lifecycle records at all**; a `grace_expired` event with no request behind it+would be a phantom distorting request counts, success rate and latency percentiles in the very+stream that is supposed to answer the question. Episode lifecycle is a `[Recovery]` structural log+line instead.++### The trap: `eventToLogRecord` is a hand-written allowlist++A field added to the `StatsEvent` interface and populated by `stats.ts` but **not pushed in+`eventToLogRecord`** is typed, type-checked, buffered, and written to+`~/.claudish/stats-buffer.json` — and never leaves the machine. Nothing throws. The number is simply+absent from every dashboard built to read it, and the loss surfaces a quarter later when someone+asks a question the data cannot answer.++The guard is in `stats-otlp.test.ts` and has two layers:++1. `satisfies Record<OptionalStatsKey, …>` makes its table **exhaustive over `StatsEvent`'s optional+ fields at COMPILE TIME**. Add an optional field to the interface and the test file stops+ compiling until it is listed.+2. The table then drives emit-when-set and omit-when-unset per field, so listing a field without+ pushing it is red.++Layer 1 is the load-bearing one — it is what stops this being a checklist someone has to remember to+read. Both directions are mutation-proven: deleting the `recovery_episode_id` push, and swapping one+`!== undefined` for a truthiness check (which drops `retry_attempts: 0`, the record that says "an+episode existed and this request added nothing to it").++`retryAttempted` on the error report is now **derived**, not hand-set. It had been a literal `false`+on the connection path since before the ladder existed, so every recovered-then-failed outage was+reported as a first-and-only attempt.++---++## 7. Risks accepted, explicitly++**RISK-6 — duplicate charges on a re-issue.** A retry after `ECONNRESET`/`EPIPE` may re-run+inference the provider already started billing. The user accepted this at the scale of our own+ladder, and then accepted it again, with the arithmetic in front of them, at the watchdog's scale:++| | watchdog dropped | **watchdog restored (chosen)** |+|---|---|---|+| client retry budget | ~11 attempts | ~300 attempts |+| unattended reach, hung connection | ~66 min | **~a day** |+| worst-case retry attempts per turn | ~46 | **~2,100** |+| RISK-6 exposure | as originally accepted | **≈30× that** |++`recovery_episode_id` is what makes the exposure *attributable* — without it, attribution breaks+exactly across a tier-2 handoff, which is the highest-exposure path.++**RISK-7 — a CI run holding a dead endpoint for the full deadline.** `--no-recovery` /+`CLAUDISH_RECOVERY=0` restores today's immediate 400 everywhere, byte for byte. `--no-recovery-ui` /+`CLAUDISH_RECOVERY_UI=0` keeps the retries and drops only the pane (and with it the 503 arm, since+the lease can never be valid).++**There is NO loopback carve-out, and its absence is a decision.** An earlier design skipped the+ladder for a refused loopback address on the reasoning that a stopped local server will not start+itself. It was cut for two reasons. The narrow one: the predicate was a regression generator — every+revision of it either fired universally (skipping recovery everywhere) or fired never, and both+times the defect was invisible because the *same rule* decided the tests. The broad one, which is+the real one, in the user's words: *"why are we talking about ollama at all? we are not building a+solution for ollama, it is a general one."* A user who restarts their server mid-ladder gets their+turn back, exactly as a user whose VPN reconnects does.++**The probe path must fail fast.** Every probe POSTs through `ComposedHandler`, so without a gate+`--probe` and the config TUI's Test All would each hold an unreachable link open for the full+deadline and then misreport it as a `timeout` — breaking the two tools that exist to diagnose this+exact fault, and multiplying a Test All run by the probe timeout per unreachable link.+`probe-live.ts` sends `x-claudish-no-recovery: 1` and `shouldSkipTier1` honours it.++**The MCP path is deadline-bounded only.** `mcp-server.ts` passes `method`, `headers` and `body` but+**no `signal`**, so when Claude Code cancels the MCP *tool call*, the proxy socket stays open,+`c.req.raw.signal` never fires, and the ladder runs its full deadline. `client_gone` covers only the+MCP server process dying outright.++---++## 8. What a healthy request pays++Nothing, and the guarantee is **placement**, not a flag: the entire retry apparatus is constructed+inside the `catch`, after classification has already returned non-null. The primary fetch expression+— including its `enqueueRequest` ternary — is byte-identical to what it was before recovery existed.+That is verifiable by reading the diff rather than by trusting a predicate.++The re-issue goes through that **same ternary**. Six transports implement `enqueueRequest`, and what+they implement is not decoration: a bounded 429 loop with `Retry-After`, a served-set model-fallback+chain, and the local concurrency gate that stops `[email protected]:3` running four inferences at once.+Skipping it would make the attempt that finally *connects* behave differently from the one that+failed — and at the moment a network returns, N woken waiters would stampede unqueued into a+provider that has just come back.++The one measured cost is launch: wrapping the session in a magmux pane adds **+107 ms median**+(login shell +47, magmux +60; worst wrapped sample 718 ms), which is 4.7× inside the 500 ms bar the+design set for itself. Default-on survives on that evidence.++---++## 9. Evidence++| report | what it establishes |+|---|---|+| [`network-recovery-phase0-measurements.md`](../reports/network-recovery-phase0-measurements.md) | the 359.607 s client abort and its `API_TIMEOUT_MS` cause; `c.env.timeout(req,0)`; `Request.signal` firing in 0.49–0.86 ms; the 75.004 s macOS connect; magmux's +107 ms |+| [`network-recovery-phase2-verification-20260911.md`](../reports/network-recovery-phase2-verification-20260911.md) | the measured 5/10/30/60/60/60 ladder with 18 ms drift over 225 s; the four modules that shipped with no tests |+| [`network-recovery-phase3-pane-20260911.md`](../reports/network-recovery-phase3-pane-20260911.md) | the lease, the frame-driven-heartbeat CRITICAL, and the two tests that can fail on it |+| [`network-recovery-phase4-status-flip-20260911.md`](../reports/network-recovery-phase4-status-flip-20260911.md) | the 503 flip through a real interactive session; the chain-safety mutations, including the quota-wording one |++Auth-path detail — the five-site inventory with per-site line references and the reproduced+`[Fallback]` advance — is in the Phase-1 half of the same body of work, summarised in §5 above.diff --git a/docs/advanced/environment.md b/docs/advanced/environment.mdindex c127b673..d9c6b29d 100644--- a/docs/advanced/environment.md+++ b/docs/advanced/environment.md@@ -135,6 +135,40 @@ export CLAUDISH_PORT='3456'Useful when you need a predictable port for firewall rules or debugging.+### `CLAUDISH_RECOVERY`++Master switch for network recovery. Default **on**.++```bash+export CLAUDISH_RECOVERY=0 # or: claudish --no-recovery+```++When claudish cannot reach a provider — DNS unresolved, connection refused, host unreachable — it no longer fails the turn immediately. It retries on a `5s → 10s → 30s → 60s → 60s…` ladder for up to a derived deadline (see below), and if the network comes back the turn continues as if nothing happened.++**Turn it off in CI and in scripted `-p` runs.** With recovery on, a genuinely dead endpoint — a wrong host, a firewalled port, a service you forgot to start — holds the request for the whole deadline before reporting. Off restores the previous behaviour exactly: an immediate error with the same message.++The deadline is **derived from `API_TIMEOUT_MS`**, not fixed:++```+min(API_TIMEOUT_MS, 300s) − 30s # ~270s at the 360s default+```++Claude Code aborts its own request at `API_TIMEOUT_MS` (360 000 ms by default), so a hold longer than that would be cut off by the client mid-wait. If you have shortened `API_TIMEOUT_MS`, claudish shortens the hold to match and logs a `[Recovery]` line saying so.++### `CLAUDISH_RECOVERY_UI`++Whether claudish may open a magmux pane showing the recovery banner. Default **on**.++```bash+export CLAUDISH_RECOVERY_UI=0 # or: claudish --no-recovery-ui+```++The banner names the provider, the host, the reason, the attempt count and a live countdown, with `[r] try now` and `[q] give up`. Turning it off keeps the retries and drops only the surface — and with it the ability to hand a retry back to Claude Code, since claudish will not ask a client to retry a failure it cannot show you.++Independent of `CLAUDISH_RECOVERY`: the UI switch never decides whether a retry happens, only whether it is visible.++Full rationale — the measurements behind the deadline, and why the handoff status is 503 — is in `ai-docs/architecture/network-recovery.md`.+---## Read-Only Variables@@ -164,6 +198,9 @@ CLAUDISH_MODEL_SUBAGENT=minimax/minimax-m2# Fixed port (optional)# CLAUDISH_PORT=3456++# CI: fail fast on an unreachable provider instead of retrying for ~270s+# CLAUDISH_RECOVERY=0```---diff --git a/docs/usage/magmux.md b/docs/usage/magmux.mdindex 48f88472..0b43ae5b 100644--- a/docs/usage/magmux.md+++ b/docs/usage/magmux.md@@ -65,9 +65,11 @@ What happens:1. Claudish creates a session directory with anonymized model IDs2. Generates a gridfile (one command per pane)3. Launches magmux with the grid layout-4. Polls for completion and updates the status bar every 500ms+4. Connects to magmux's IPC socket as a client and collects its lifecycle events-The status bar shows live progress:+Claudish does **not** poll. Magmux owns the entire pane lifecycle — idle detection, the `DONE`/`FAIL` overlays, the green/red tints, the status bar, and auto-exit — and pushes `snapshot`, `exit`, `results` and `shutdown` events over that socket. Claudish reads them and treats the final `results` event as the authoritative per-pane state.++The status bar shows live progress, redrawn by magmux's own render loop:```claudish team 3 done 32s complete ctrl-g q to quit@@ -251,7 +253,7 @@ The C MTM binary still ships in the repo (`packages/cli/native/mtm/`) as a fallb### Status bar shows stale data in grid mode-**Cause**: The claudish poller writes the status bar file every 500ms. Brief delays between model completion and status bar update are normal.+**Cause**: Magmux redraws the status bar from its own render loop, counting done and running panes. A model that has finished producing output but whose process has not yet exited still counts as running, so a brief lag between the last line of output and the counter moving is normal.**Fix**: Wait a moment. The final status always reflects the true state after all models finish.diff --git a/packages/cli/src/handlers/composed-handler.ts b/packages/cli/src/handlers/composed-handler.tsindex 0d2a62ef..17049fdb 100644--- a/packages/cli/src/handlers/composed-handler.ts+++ b/packages/cli/src/handlers/composed-handler.ts@@ -39,7 +39,12 @@ import { deepMergeParams } from "../model-params.js";import { describeSiblingKeys, getProviderByName } from "../providers/provider-definitions.js";import { isTerminal429 } from "../providers/transport/openai.js";import { recoveryClock } from "../recovery/clock.js";-import { episodeCount, noteTargetReachable, uiLeaseValid } from "../recovery/coordinator.js";+import {+ type RecoveryOutcome,+ episodeCount,+ noteTargetReachable,+ uiLeaseValid,+} from "../recovery/coordinator.js";import { resolveRecoveryEnabled } from "../recovery/settings.js";import {type OpenAIImageBlock,@@ -48,6 +53,7 @@ import {} from "../services/vision-proxy.js";import { type SessionEventRegistry, extractSessionId } from "../session-events/index.js";import { applyProInjection } from "../session-events/pro-injection.js";+import type { StatsEvent } from "../stats-otlp.js";import { recordStats } from "../stats.js";import { classifyError, reportError } from "../telemetry.js";import { transformOpenAIToClaude } from "../transform.js";@@ -100,6 +106,28 @@ import { captureUpstreamError } from "./shared/upstream-error-capture.js";*/const STREAM_RETRY_DELAYS_MS = [3_000, 15_000, 30_000];+/**+ * One request's running account of what network recovery cost it. Mutable, and+ * mutated only by `recoverConnection`.+ *+ * Every field is REQUEST-scoped even though the episode behind them is not.+ * See `ComposedHandler.newRecoveryTally` for why that distinction is the whole+ * point, and `transient-retry.ts`'s `ConnectionRetryTally` for where the+ * request-scoped figures come from.+ */+interface RecoveryTally {+ /** Re-issues this request made. Not counting the failure that started it. */+ retries: number;+ /** Ms of this request's own latency spent inside the ladder. */+ recoveryMs: number;+ /** Set once an episode exists. Its presence is what makes the rest real. */+ episodeId?: string;+ /** Which tier-2 re-entry this request is. 0 = the original. */+ clientRetry: number;+ /** How the episode ended for this request, once it has. */+ outcome?: RecoveryOutcome;+}+function extractAuthHeaders(c: Context): VisionProxyAuthHeaders {const headers = c.req.header();const auth: VisionProxyAuthHeaders = {};@@ -348,6 +376,50 @@ export class ComposedHandler implements ModelHandler {}}+ /**+ * What one REQUEST did inside the retry ladder, accumulated across every site+ * in that request that could enter it.+ *+ * One per `handle()` call, never on `this` — a `ComposedHandler` instance is+ * reused for the life of the process and serves concurrent requests, so an+ * instance field here would report one request's outage on another request's+ * record. The cost of getting that wrong is not a wrong number in a log: it+ * is a wrong number in the only stream that can answer "how often does this+ * happen", which is what the feature was measured by.+ *+ * ACCUMULATION IS PER-REQUEST AND LAST-EPISODE-WINS. A request can hit an+ * auth-path episode and then a fetch-path episode. Attempts and waits ADD+ * (they are both this request's own time, and there is exactly one record per+ * request), while `episodeId` and `clientRetry` are overwritten — so the id+ * that survives is the one belonging to the episode that DECIDED THE STATUS,+ * which is the only reading under which a scalar correlation id is+ * well-defined.+ */+ private static newRecoveryTally(): RecoveryTally {+ return { retries: 0, recoveryMs: 0, clientRetry: 0 };+ }++ /**+ * The recovery half of a stats event — spread into every `recordStats` on a+ * path a recovery could precede.+ *+ * Empty unless an episode actually happened, which is what keeps a healthy+ * request's record byte-identical to what it was before this feature existed.+ * `retry_attempts: 0` IS emitted when an episode existed and this request+ * added no re-issue to it; that is a different fact from "no recovery", and+ * only the presence of `recovery_episode_id` distinguishes them.+ */+ private recoveryStats(tally: RecoveryTally | undefined): Partial<StatsEvent> {+ if (!tally?.episodeId) return {};+ return {+ retry_attempts: tally.retries,+ recovery_ms: tally.recoveryMs,+ recovery_episode_id: tally.episodeId,+ recovery_client_retry: tally.clientRetry,+ ...(tally.outcome ? { recovery_outcome: tally.outcome } : {}),+ };+ }+/*** The ONE response shape for "claudish could not reach the host".*@@ -378,7 +450,18 @@ export class ComposedHandler implements ModelHandler {ctx: {startTime: number;fallbackMeta?: { chain: string[]; attempts: number };- retryAttempted?: boolean;+ /**+ * This request's ladder account, if it ran one. Absent means the ladder+ * was skipped, which is today's immediate-400 behaviour.+ */+ recovery?: RecoveryTally;+ /**+ * A retry this request made BEFORE the ladder — the parameter-recovery+ * re-fetch and the forced-auth re-fetch both are one. OR'd with the+ * ladder's own count rather than replaced by it, because both are true+ * answers to "did claudish retry before reporting this".+ */+ retriedBeforeLadder?: boolean;authType?: "api-key" | "oauth" | "none";/** Where the failure happened, for the debug log only. */site?: string;@@ -397,7 +480,12 @@ export class ComposedHandler implements ModelHandler {modelId: this.targetModel,httpStatus: undefined,isStreaming: false,- retryAttempted: ctx.retryAttempted ?? false,+ // DERIVED, never hand-set. This field had been a literal `false` on this+ // path since before the ladder existed, so every recovered-then-failed+ // outage was reported as a first-and-only attempt. It is true exactly+ // when this request re-issued the operation at least once, from either+ // source.+ retryAttempted: (ctx.retriedBeforeLadder ?? false) || (ctx.recovery?.retries ?? 0) > 0,isInteractive: this.isInteractive,...(ctx.authType ? { authType: ctx.authType } : {}),});@@ -419,6 +507,7 @@ export class ComposedHandler implements ModelHandler {fallback_chain: ctx.fallbackMeta?.chain,fallback_attempts: ctx.fallbackMeta?.attempts,invocation_mode: this.options.invocationMode ?? "auto-route",+ ...this.recoveryStats(ctx.recovery),});} catch {// Stats must never crash claudish@@ -469,10 +558,15 @@ export class ComposedHandler implements ModelHandler {conn: { kind: ConnectionErrorKind; code: string },endpoint: string,ctx: {+ /** EPISODE-scoped, and deliberately so: these two are what the SENTENCE+ * quotes, and the sentence is about the outage rather than about this+ * socket. The request-scoped figures the stats record are in+ * `recovery`. */startTime: number;attempts: number;recoveryMs: number;fallbackMeta?: { chain: string[]; attempts: number };+ recovery?: RecoveryTally;}): Response {const reason = buildConnectionErrorMessage(conn.kind, this.provider.displayName, endpoint);@@ -495,6 +589,7 @@ export class ComposedHandler implements ModelHandler {fallback_chain: ctx.fallbackMeta?.chain,fallback_attempts: ctx.fallbackMeta?.attempts,invocation_mode: this.options.invocationMode ?? "auto-route",+ ...this.recoveryStats(ctx.recovery),});} catch {// Stats must never crash claudish@@ -579,9 +674,16 @@ export class ComposedHandler implements ModelHandler {startTime: number;deadlineAt: number;fallbackMeta?: { chain: string[]; attempts: number };- retryAttempted?: boolean;authType?: "api-key" | "oauth" | "none";site: string;+ /** True at the two sites that are THEMSELVES a retry. See the field of+ * the same name on `respondConnectionError`. */+ retriedBeforeLadder?: boolean;+ /**+ * THIS REQUEST'S tally, created once per `handle()` and threaded through+ * every site that can enter the ladder. Mutated here and nowhere else.+ */+ recovery: RecoveryTally;}): Promise<{ kind: "value"; value: T } | { kind: "respond"; response: Response }> {const skip = this.shouldSkipTier1(c, ctx.deadlineAt);@@ -620,15 +722,33 @@ export class ComposedHandler implements ModelHandler {signal: this.clientSignal(c),});+ // Fold this ladder run into the request's account BEFORE any arm below can+ // return. A request that recovers at the auth site and then exhausts at the+ // fetch site must report BOTH sets of attempts and BOTH waits — they were+ // all its own latency — under the id of the episode that decided its+ // status, which is the later one. Both halves of that are this statement.+ ctx.recovery.retries += result.requestRetries;+ ctx.recovery.recoveryMs += result.requestRecoveryMs;+ ctx.recovery.episodeId = result.episodeId;+ ctx.recovery.clientRetry = result.clientRetry;+switch (result.kind) {case "ok":+ ctx.recovery.outcome = "recovered";return { kind: "value", value: result.value };case "threw":// Classification runs on every attempt. Something that is not a// connection failure must keep its own route out of here.throw result.error;case "client_gone":- this.recordClientGone(ctx.startTime, ctx.fallbackMeta, result.episodeId, result.attempts);+ ctx.recovery.outcome = "client_gone";+ this.recordClientGone(+ ctx.startTime,+ ctx.fallbackMeta,+ result.episodeId,+ result.attempts,+ ctx.recovery+ );// Nobody reads this. It exists so Hono has an object to return for a// socket that is already gone.return { kind: "respond", response: new Response(null, { status: 499 }) };@@ -653,6 +773,8 @@ export class ComposedHandler implements ModelHandler {// by definition alive), so a lease-only test would answer a retryable// 503 and Claude Code would immediately re-ask — turning the give-up// key into a no-op with a banner still on screen.+ ctx.recovery.outcome = result.kind === "exhausted" ? "handoff" : "gave_up";+if (result.kind === "exhausted" && leased) {log(`[Recovery] ${this.provider.displayName} exhausted after ${result.attempts} attempts ` +@@ -666,6 +788,7 @@ export class ComposedHandler implements ModelHandler {attempts: result.attempts,recoveryMs: result.recoveryMs,fallbackMeta: ctx.fallbackMeta,+ recovery: ctx.recovery,}),};}@@ -677,10 +800,7 @@ export class ComposedHandler implements ModelHandler {);return {kind: "respond",- response: this.respondConnectionError(c, result.error, result.conn, result.endpoint, {- ...ctx,- retryAttempted: true,- }),+ response: this.respondConnectionError(c, result.error, result.conn, result.endpoint, ctx),};}}@@ -691,7 +811,8 @@ export class ComposedHandler implements ModelHandler {startTime: number,fallbackMeta: { chain: string[]; attempts: number } | undefined,episodeId: string,- attempts: number+ attempts: number,+ recovery?: RecoveryTally): void {log(`[Recovery] client disconnected after ${attempts} attempts (episode ${episodeId}) — ` +@@ -714,6 +835,7 @@ export class ComposedHandler implements ModelHandler {fallback_chain: fallbackMeta?.chain,fallback_attempts: fallbackMeta?.attempts,invocation_mode: this.options.invocationMode ?? "auto-route",+ ...this.recoveryStats(recovery),});} catch {// Stats must never crash claudish@@ -729,6 +851,11 @@ export class ComposedHandler implements ModelHandler {// Used in all stats recording paths so a single event carries complete info.const fallbackMeta = this.pendingFallbackMeta;this.pendingFallbackMeta = undefined;+ // This request's network-recovery account. A local, like `fallbackMeta` and+ // for the same reason: the handler instance outlives the request and serves+ // several at once. It stays all-zero and contributes NOTHING to any stats+ // record unless a classified connection failure actually occurs.+ const recoveryTally = ComposedHandler.newRecoveryTally();// 1. Transform incoming Claude-format requestconst { claudeRequest, droppedParams } = transformOpenAIToClaude(payload);@@ -997,6 +1124,7 @@ export class ComposedHandler implements ModelHandler {fallbackMeta,authType: "oauth",site: "refreshAuth",+ recovery: recoveryTally,});if (outcome.kind === "respond") return outcome.response;@@ -1106,6 +1234,7 @@ export class ComposedHandler implements ModelHandler {fallbackMeta,authType: "oauth",site: "getHeaders",+ recovery: recoveryTally,});if (outcome.kind === "respond") return outcome.response;@@ -1186,6 +1315,7 @@ export class ComposedHandler implements ModelHandler {deadlineAt: tier1DeadlineAt(c),fallbackMeta,site: "fetch",+ recovery: recoveryTally,});if (outcome.kind === "respond") return outcome.response;response = outcome.value;@@ -1319,8 +1449,9 @@ export class ComposedHandler implements ModelHandler {// there is no unclamped attempt left ahead of it to reserve for.deadlineAt: tier1DeadlineAt(c),fallbackMeta,- retryAttempted: true,+ retriedBeforeLadder: true,site: "parameter-recovery",+ recovery: recoveryTally,});if (outcome.kind === "respond") return outcome.response;@@ -1414,6 +1545,7 @@ export class ComposedHandler implements ModelHandler {fallback_chain: fallbackMeta?.chain,fallback_attempts: fallbackMeta?.attempts,invocation_mode: this.options.invocationMode ?? "auto-route",+ ...this.recoveryStats(recoveryTally),});} catch {// Stats must never crash claudish@@ -1447,9 +1579,10 @@ export class ComposedHandler implements ModelHandler {startTime,deadlineAt: tier1DeadlineAt(c),fallbackMeta,- retryAttempted: true,+ retriedBeforeLadder: true,authType: "oauth",site: "forceRefreshAuth",+ recovery: recoveryTally,});if (outcome.kind === "respond") return outcome.response;@@ -1492,6 +1625,7 @@ export class ComposedHandler implements ModelHandler {fallback_chain: fallbackMeta?.chain,fallback_attempts: fallbackMeta?.attempts,invocation_mode: this.options.invocationMode ?? "auto-route",+ ...this.recoveryStats(recoveryTally),});} catch {// Stats must never crash claudish@@ -1594,6 +1728,7 @@ export class ComposedHandler implements ModelHandler {fallback_chain: fallbackMeta?.chain,fallback_attempts: fallbackMeta?.attempts,invocation_mode: this.options.invocationMode ?? "auto-route",+ ...this.recoveryStats(recoveryTally),});} catch {// Stats must never crash claudish@@ -1681,6 +1816,7 @@ export class ComposedHandler implements ModelHandler {fallback_chain: fallbackMeta?.chain,fallback_attempts: fallbackMeta?.attempts,invocation_mode: this.options.invocationMode ?? "auto-route",+ ...this.recoveryStats(recoveryTally),});} catch {// Stats must never crash claudish@@ -1739,6 +1875,7 @@ export class ComposedHandler implements ModelHandler {fallback_chain: fallbackMeta?.chain,fallback_attempts: fallbackMeta?.attempts,invocation_mode: this.options.invocationMode ?? "auto-route",+ ...this.recoveryStats(recoveryTally),});} catch {// Stats must never crash claudish@@ -1793,6 +1930,7 @@ export class ComposedHandler implements ModelHandler {fallback_chain: fallbackMeta?.chain,fallback_attempts: fallbackMeta?.attempts,invocation_mode: this.options.invocationMode ?? "auto-route",+ ...this.recoveryStats(recoveryTally),});} catch {// Stats must never crash claudishdiff --git a/packages/cli/src/handlers/shared/transient-retry.ts b/packages/cli/src/handlers/shared/transient-retry.tsindex 911f5244..5c6043ba 100644--- a/packages/cli/src/handlers/shared/transient-retry.ts+++ b/packages/cli/src/handlers/shared/transient-retry.ts@@ -169,35 +169,64 @@ export interface ConnectionRetryContext {signal: AbortSignal;}+/**+ * The counters every outcome carries, in TWO scopes, because the two answer+ * different questions and conflating them mis-reports both.+ *+ * EPISODE scope (`attempts`, `recoveryMs`) is what the human reads: one banner+ * saying "attempt 9 · 4m 12s in recovery" across every waiter and both tiers,+ * which is the honest account of the OUTAGE.+ *+ * REQUEST scope (`requestRetries`, `requestRecoveryMs`) is what stats record,+ * because a stats event is one-per-request by definition. The episode figures+ * cannot stand in for these: N concurrent requests share one episode, so each+ * would report the sum of all of them, and a tier-2 re-entry inherits counters+ * from a request that already recorded its own — so summing `retry_attempts`+ * over an outage would multiply it by the number of waiters and again by the+ * number of re-entries.+ */+interface ConnectionRetryTally {+ /** Episode-scoped: every attempt by every waiter, across both tiers. */+ attempts: number;+ /** Episode-scoped: ms since the episode opened. */+ recoveryMs: number;+ /**+ * THIS request's RE-ISSUES — the operation run again, not counting the+ * failure that brought us here. 0 is meaningful and reachable: an episode+ * existed and this request added nothing to it because its own budget was+ * already spent. That is what `retry_attempts` records.+ */+ requestRetries: number;+ /** THIS request's own ms inside the ladder — `recovery_ms`. */+ requestRecoveryMs: number;+ /** Which tier-2 re-entry this is. 0 = the request that opened the episode. */+ clientRetry: number;+ episodeId: string;+}+export type ConnectionRetryResult<T> =- | { kind: "ok"; value: T; attempts: number; recoveryMs: number; episodeId: string }+ | ({ kind: "ok"; value: T } & ConnectionRetryTally)/*** NOTE the absence of a `uiLeased` field. Whether a surface is painting this* episode is LIVE and must be read at the moment the status is chosen; a* boolean on this result IS a cache of it, and by the time a caller* destructures the result it can already be stale.*/- | {+ | ({kind: "exhausted";conn: ConnectionErrorInfo;error: unknown;endpoint: string;- attempts: number;- recoveryMs: number;- episodeId: string;- }- | {+ } & ConnectionRetryTally)+ | ({kind: "gave_up";conn: ConnectionErrorInfo;error: unknown;endpoint: string;- attempts: number;- recoveryMs: number;- episodeId: string;- }- | { kind: "client_gone"; attempts: number; recoveryMs: number; episodeId: string }+ } & ConnectionRetryTally)+ | ({ kind: "client_gone" } & ConnectionRetryTally)/** A later attempt threw something that is NOT a connection error. Not ours. */- | { kind: "threw"; error: unknown; attempts: number; recoveryMs: number; episodeId: string };+ | ({ kind: "threw"; error: unknown } & ConnectionRetryTally);/** True when the rejection is the inbound client going away, not our own clamp. */function isClientAbort(signal: AbortSignal, err: unknown): boolean {@@ -225,12 +254,29 @@ export async function withConnectionRetry<T>(ctx: ConnectionRetryContext): Promise<ConnectionRetryResult<T>> {const clock = recoveryClock();+ // PROCESS ms at which THIS request entered the ladder. `recovery_ms` is+ // measured from here rather than from the episode, because an episode can be+ // older than the request that joined it — a tier-2 re-entry inherits one —+ // and a `recovery_ms` larger than the same record's `latency_ms` is a figure+ // no reader can make sense of.+ const enteredAtPerf = clock.now();+ let requestRetries = 0;+const firstConn = classifyConnectionError(first);if (!firstConn) {// The caller is contractually required to classify before calling. Being// defensive here costs nothing and keeps a future caller's mistake from// silently widening what "transient" means.- return { kind: "threw", error: first, attempts: 0, recoveryMs: 0, episodeId: "" };+ return {+ kind: "threw",+ error: first,+ attempts: 0,+ recoveryMs: 0,+ requestRetries: 0,+ requestRecoveryMs: 0,+ clientRetry: 0,+ episodeId: "",+ };}let conn: ConnectionErrorInfo = firstConn;@@ -247,19 +293,27 @@ export async function withConnectionRetry<T>(deadlineAt: ctx.deadlineAt,});- const finish = <R extends ConnectionRetryResult<T>>(r: R): R => r;+ /**+ * The counters, snapshotted at the instant an outcome is returned. Every+ * `return` in this function goes through it, so a new counter is added in one+ * place and cannot be forgotten on one arm — which is how five hand-written+ * copies of the same four fields drift.+ */+ const tally = () => ({+ attempts: handle.attempts(),+ recoveryMs: handle.recoveryMs(),+ requestRetries,+ requestRecoveryMs: Math.round(clock.now() - enteredAtPerf),+ clientRetry: handle.clientRetries(),+ episodeId: handle.episodeId,+ });try {handle.recordAttemptResult(conn.code, false);for (;;) {if (ctx.signal.aborted) {- return finish({- kind: "client_gone",- attempts: handle.attempts(),- recoveryMs: handle.recoveryMs(),- episodeId: handle.episodeId,- });+ return { kind: "client_gone", ...tally() };}const delay = connectionRetryDelayMs(handle.ladderIndex());@@ -268,15 +322,7 @@ export async function withConnectionRetry<T>(// sleep. Bounding only the sleep is how a budget stops being a ceiling.if (now + delay + MIN_ATTEMPT_SLOT_MS > ctx.deadlineAt) {handle.handoff();- return finish({- kind: "exhausted",- conn,- error,- endpoint,- attempts: handle.attempts(),- recoveryMs: handle.recoveryMs(),- episodeId: handle.episodeId,- });+ return { kind: "exhausted", conn, error, endpoint, ...tally() };}let waited: { kind: "attempt" } | { kind: "gave_up" };@@ -285,26 +331,13 @@ export async function withConnectionRetry<T>(} catch (err) {if (isClientAbort(ctx.signal, err)) {log(`[Recovery] client gone during wait (episode ${handle.episodeId})`);- return finish({- kind: "client_gone",- attempts: handle.attempts(),- recoveryMs: handle.recoveryMs(),- episodeId: handle.episodeId,- });+ return { kind: "client_gone", ...tally() };}throw err;}if (waited.kind === "gave_up") {- return finish({- kind: "gave_up",- conn,- error,- endpoint,- attempts: handle.attempts(),- recoveryMs: handle.recoveryMs(),- episodeId: handle.episodeId,- });+ return { kind: "gave_up", conn, error, endpoint, ...tally() };}// Per-attempt clamp. Cleared the moment the operation settles, so it can@@ -322,28 +355,18 @@ export async function withConnectionRetry<T>(const merged = AbortSignal.any([ctx.signal, attemptAc.signal]);try {+ requestRetries++;const value = await op(merged);handle.recordAttemptResult("ok", true);log(`[Recovery] ${ctx.providerDisplayName} recovered after ${handle.attempts()} attempts ` +`in ${handle.recoveryMs()}ms (episode ${handle.episodeId})`);- return finish({- kind: "ok",- value,- attempts: handle.attempts(),- recoveryMs: handle.recoveryMs(),- episodeId: handle.episodeId,- });+ return { kind: "ok", value, ...tally() };} catch (err) {if (isClientAbort(ctx.signal, err)) {log(`[Recovery] client gone during attempt (episode ${handle.episodeId})`);- return finish({- kind: "client_gone",- attempts: handle.attempts(),- recoveryMs: handle.recoveryMs(),- episodeId: handle.episodeId,- });+ return { kind: "client_gone", ...tally() };}// Re-classify EVERY attempt. Our own clamp lands here as a// `TimeoutError`, which classification now recognises for the same@@ -351,13 +374,7 @@ export async function withConnectionRetry<T>(// out of a host we could not reach in time.const next = classifyConnectionError(err);if (!next) {- return finish({- kind: "threw",- error: err,- attempts: handle.attempts(),- recoveryMs: handle.recoveryMs(),- episodeId: handle.episodeId,- });+ return { kind: "threw", error: err, ...tally() };}conn = next;error = err;diff --git a/packages/cli/src/providers/probe-live.ts b/packages/cli/src/providers/probe-live.tsindex 5f11e5bb..ab394661 100644--- a/packages/cli/src/providers/probe-live.ts+++ b/packages/cli/src/providers/probe-live.ts@@ -398,10 +398,23 @@ function extractErrorType(body: string): string | undefined {export function classifyHttpError(status: number, body: string, latencyMs: number): ProbeResult {const lowered = body.toLowerCase();// A proxy-side connection failure (DNS unresolved, connection refused, host- // unreachable) is tagged by composed-handler as a 503 connection_error. That's- // a LOCAL network problem, not an upstream server error — classify it as- // network-error so the row reads "network error" with the proxy's actionable- // message ("check your network/DNS") instead of a generic "server error · 503".+ // unreachable) is tagged by composed-handler as a **400** connection_error —+ // NOT a 503, whatever this comment said until v9.2.x. That matters twice+ // over. The status is 400 because a 503 makes Claude Code retry with the+ // reason buried behind its own banner, and because `isRetryableError` would+ // otherwise walk the fallback chain. So the test below keys off the `type`,+ // which is status-agnostic and therefore survives both that choice and the+ // terminal-error remap that shares the same 400.+ //+ // Probes never reach the retry ladder at all: `probeLink` sends+ // `x-claudish-no-recovery: 1` and `shouldSkipTier1` honours it, so an+ // unreachable provider still answers here in milliseconds instead of holding+ // for the full recovery deadline and misreporting as a timeout.+ //+ // It is a LOCAL network problem, not an upstream server error — classify it+ // as network-error so the row reads "network error" with the proxy's+ // actionable message ("check your network/DNS") instead of a generic+ // "server error".if (extractErrorType(body) === "connection_error") {return {state: "network-error",diff --git a/packages/cli/src/recovery/coordinator.ts b/packages/cli/src/recovery/coordinator.tsindex 4876c849..84ce467d 100644--- a/packages/cli/src/recovery/coordinator.ts+++ b/packages/cli/src/recovery/coordinator.ts@@ -298,6 +298,14 @@ export interface EpisodeHandle {attempts(): number;/** Ms since the episode opened — not since this request started. */recoveryMs(): number;+ /**+ * How many times a client has re-entered this episode after a tier-2+ * handoff. 0 for the request that opened it.+ *+ * Read for stats, so that a record can say WHICH re-entry it was rather than+ * leaving five records of one outage indistinguishable from five outages.+ */+ clientRetries(): number;ladderIndex(): number;/** LIVE. Never cache the result. */uiLeaseValid(): boolean;@@ -441,6 +449,7 @@ export function joinEpisode(seed: EpisodeSeed): EpisodeHandle {key: episode.key,attempts: () => episode.attempts,recoveryMs: () => Math.round(recoveryClock().now() - episode.startedAtPerf),+ clientRetries: () => episode.clientRetries,ladderIndex: () => episode.ladderIndex,uiLeaseValid: () => uiLeaseValid(episode.episodeId),diff --git a/packages/cli/src/recovery/settings.ts b/packages/cli/src/recovery/settings.tsindex f0373466..3aa33b9a 100644--- a/packages/cli/src/recovery/settings.ts+++ b/packages/cli/src/recovery/settings.ts@@ -60,16 +60,18 @@ export function resolveRecoveryEnabled(): boolean {* May claudish own a surface on which a recovery episode's reason is legible?* Default true.*- * NOTHING READS THIS YET, and that is expected. It exists now for two reasons.- * The first is that the status decision it will gate — whether an exhausted- * episode may answer a RETRYABLE status instead of an inline error — is a later- * phase, and landing the switch early means that phase changes one branch- * rather than four files. The second is the CLAUDE.md invariant it trips: a new- * `ClaudishProfileConfig` field that is not in `loadConfig`'s allowlist AND- * does not have a scoped reader survives on disk until the first global save- * and is then dropped, silently. That trap has caught this codebase twice, so- * the field is landed with its round-trip test the moment it is defined rather- * than the moment it is first consumed.+ * READ BY `claude-runner.ts`, which decides whether to wrap the session in a+ * magmux pane. It is the USER'S half of the gate; the other half is+ * `uiLeaseValid()`, which asks whether a renderer is painting THIS episode right+ * now. Both must hold before an exhausted episode may answer a retryable 503 —+ * the user has to have allowed a surface, and a surface has to actually exist.+ *+ * It was landed one phase before its first reader, deliberately, because of the+ * CLAUDE.md invariant it trips: a new `ClaudishProfileConfig` field that is not+ * in `loadConfig`'s allowlist AND has no scoped reader survives on disk until+ * the first global save and is then dropped, silently. That trap has caught this+ * codebase twice, so the field got its round-trip test the moment it was+ * defined rather than the moment it was first consumed.** It is a question about the USER'S CONFIGURATION and about nothing else. It* must never be conflated with "can this process open a pane right now", whichdiff --git a/packages/cli/src/stats-otlp.test.ts b/packages/cli/src/stats-otlp.test.tsindex a0dfa0be..bbcd31c9 100644--- a/packages/cli/src/stats-otlp.test.ts+++ b/packages/cli/src/stats-otlp.test.ts@@ -216,3 +216,128 @@ describe("formatOtlpBatch", () => {expect(result.resourceLogs).toEqual([]);});});++/**+ * THE GUARD FOR THE ONE SILENT FAILURE IN `stats-otlp.ts`.+ *+ * `eventToLogRecord` is a hand-written attribute allowlist. A field added to+ * `StatsEvent` and populated by `stats.ts` but NOT pushed there is typed,+ * type-checked, buffered and written to `~/.claudish/stats-buffer.json` — and+ * never leaves the machine. Nothing throws. The field is simply missing from+ * every dashboard built to read it, and the loss is discovered a quarter later,+ * when someone asks a question the data cannot answer.+ *+ * Two layers, and the first is the durable one:+ *+ * 1. `satisfies Record<OptionalStatsKey, …>` makes the table below EXHAUSTIVE+ * over `StatsEvent`'s optional fields AT COMPILE TIME. Add an optional field+ * to the interface and this file stops compiling until it is listed here.+ * 2. The table then drives emit-when-set and omit-when-unset for every entry,+ * so listing a field without pushing it in `eventToLogRecord` is red.+ *+ * Together, a new optional field cannot reach `main` without its push. Layer 1+ * is what stops this from being a checklist someone has to remember to read.+ */+type OptionalStatsKey = {+ [K in keyof StatsEvent]-?: undefined extends StatsEvent[K] ? K : never;+}[keyof StatsEvent];++const OPTIONAL_WIRE = {+ error_class: {+ attribute: "llm.error_class",+ sample: "network",+ value: { stringValue: "network" },+ },+ error_code: {+ attribute: "llm.error_code",+ sample: "ECONNREFUSED",+ value: { stringValue: "ECONNREFUSED" },+ },+ fallback_chain: {+ attribute: "llm.fallback_chain",+ sample: ["litellm", "openrouter"],+ value: { arrayValue: { values: [{ stringValue: "litellm" }, { stringValue: "openrouter" }] } },+ },+ fallback_attempts: {+ attribute: "llm.fallback_attempts",+ sample: 2,+ value: { intValue: "2" },+ },+ retry_attempts: {+ attribute: "llm.retry_attempts",+ sample: 5,+ value: { intValue: "5" },+ },+ recovery_ms: {+ attribute: "llm.recovery_ms",+ sample: 201_900,+ value: { intValue: "201900" },+ },+ recovery_episode_id: {+ attribute: "llm.recovery_episode_id",+ sample: "3f2b9c14-5d6e-4a71-9b0c-8e2d4f6a1b33",+ value: { stringValue: "3f2b9c14-5d6e-4a71-9b0c-8e2d4f6a1b33" },+ },+ recovery_client_retry: {+ attribute: "llm.recovery_client_retry",+ sample: 3,+ value: { intValue: "3" },+ },+ recovery_outcome: {+ attribute: "llm.recovery_outcome",+ sample: "handoff",+ value: { stringValue: "handoff" },+ },+} satisfies Record<OptionalStatsKey, { attribute: string; sample: unknown; value: unknown }>;++const OPTIONAL_ENTRIES = Object.entries(OPTIONAL_WIRE) as Array<+ [OptionalStatsKey, { attribute: string; sample: unknown; value: unknown }]+>;++describe("eventToLogRecord — every optional field reaches the wire", () => {+ for (const [field, spec] of OPTIONAL_ENTRIES) {+ it(`emits ${spec.attribute} when ${field} is set`, () => {+ const record = eventToLogRecord({ ...SAMPLE_EVENT, [field]: spec.sample } as StatsEvent);+ const attr = record.attributes.find((a) => a.key === spec.attribute);+ expect(attr).toBeDefined();+ expect(attr?.value).toEqual(spec.value as any);+ });++ it(`omits ${spec.attribute} when ${field} is unset`, () => {+ // Deleting rather than assigning undefined: `{ ...e, x: undefined }` leaves+ // the KEY present, and the absent-key shape is the one production emits.+ const bare = { ...SAMPLE_EVENT } as Record<string, unknown>;+ delete bare[field];+ const record = eventToLogRecord(bare as unknown as StatsEvent);+ expect(record.attributes.some((a) => a.key === spec.attribute)).toBe(false);+ });+ }++ it("emits retry_attempts: 0 — zero is a fact, not an absence", () => {+ // The episode existed and this request added no re-issue to it. A truthiness+ // guard (`if (event.retry_attempts)`) instead of `!== undefined` deletes this+ // record's only evidence that recovery ran at all, and deletes it precisely+ // on the requests that were handed off with no budget left.+ const record = eventToLogRecord({+ ...SAMPLE_EVENT,+ retry_attempts: 0,+ recovery_ms: 0,+ recovery_client_retry: 0,+ recovery_episode_id: "3f2b9c14-5d6e-4a71-9b0c-8e2d4f6a1b33",+ });+ expect(record.attributes.find((a) => a.key === "llm.retry_attempts")?.value).toEqual({+ intValue: "0",+ } as any);+ expect(record.attributes.find((a) => a.key === "llm.recovery_client_retry")?.value).toEqual({+ intValue: "0",+ } as any);+ });++ it("a healthy request carries no recovery attributes at all", () => {+ // The other half of NFR-1: a machine that never loses its network sends the+ // same bytes it sent before this feature existed.+ const keys = eventToLogRecord(SAMPLE_EVENT).attributes.map((a) => a.key);+ expect(keys.filter((k) => k.startsWith("llm.recovery_"))).toEqual([]);+ expect(keys).not.toContain("llm.retry_attempts");+ });+});diff --git a/packages/cli/src/stats-otlp.ts b/packages/cli/src/stats-otlp.tsindex 1a197d83..70a2ece6 100644--- a/packages/cli/src/stats-otlp.ts+++ b/packages/cli/src/stats-otlp.ts@@ -7,8 +7,20 @@* Wire format: OTLP JSON Logs* Signal type: LogRecord per request* Namespace: llm.* for custom attributes, standard OTel for resource/HTTP+ *+ * ── THE ONE TRAP IN THIS FILE ────────────────────────────────────────────────+ *+ * `eventToLogRecord` below is a HAND-WRITTEN ALLOWLIST. A field added to+ * `StatsEvent` and populated by `stats.ts` but not pushed there is typed,+ * type-checked, buffered, and written to `~/.claudish/stats-buffer.json` — and+ * never leaves the machine. Nothing fails; the number is simply absent from+ * every dashboard that was built to read it. Add the push, and add a case to+ * `stats-otlp.test.ts`'s emit/omit table, which exists to make the omission+ * cost a red test instead of a silent quarter.*/+import type { RecoveryOutcome } from "./recovery/coordinator.js";+// ─── Interfaces ───────────────────────────────────────────────────────────────/**@@ -46,6 +58,61 @@ export interface StatsEvent {fallback_chain?: string[]; // provider names tried, in orderfallback_attempts?: number; // how many failed before success+ // ── Network recovery ────────────────────────────────────────────────────────+ // Present only on a request that hit a classified connection failure. On every+ // healthy request all five are absent, and `eventToLogRecord` emits nothing+ // for them — so a machine that never loses its network sends exactly the+ // bytes it sent before this feature existed.++ /**+ * Connect retries THIS REQUEST made against the SAME provider.+ *+ * NOT `fallback_attempts`, which counts how many DIFFERENT providers were+ * tried. The two answer opposite questions and a chain that never advanced+ * can still have retried thirty times.+ *+ * Per-request, not per-episode: one episode can serve several concurrent+ * requests and can outlive any of them, so the episode's cumulative count+ * would over-report every request in it. `recovery_episode_id` is what joins+ * them back together.+ */+ retry_attempts?: number;++ /**+ * Ms of this request's own `latency_ms` spent inside the retry ladder —+ * backoff waits plus failed connects.+ *+ * `latency_ms` deliberately still INCLUDES that time (`adapters.md`: "the+ * honest figure is time-to-usable-response"). This field is what makes the+ * resulting skew explicable rather than mysterious: `latency_ms: 246_000,+ * recovery_ms: 201_900` reads as a 44-second turn behind a three-and-a-half+ * minute outage. Without it, the same record reads as a four-minute model.+ */+ recovery_ms?: number;++ /**+ * The episode this request belonged to — the correlation id, and the only+ * field that makes "how often does recovery happen" answerable.+ *+ * One outage is one episode and may span several requests (each tier-2+ * handoff is a new request re-entering the same episode), so counting+ * records over-counts outages by the retry factor.+ * `count(distinct recovery_episode_id)` is the honest figure.+ *+ * A request that hits BOTH an auth-path episode and a fetch-path episode+ * carries the id of the one that DECIDED ITS STATUS. An auth episode that+ * recovered has no record of its own: its attempts fold into this request's+ * `retry_attempts` and its waits into this request's `recovery_ms`, because+ * there is exactly one record per request.+ */+ recovery_episode_id?: string;++ /** Which tier-2 re-entry this request is. 0 = the original request. */+ recovery_client_retry?: number;++ /** How the episode ended for this request. */+ recovery_outcome?: RecoveryOutcome;+// Invocationinvocation_mode: string; // "profile" | "explicit-model" | "auto-route" | "env-var" | "model-map"@@ -219,6 +286,25 @@ export function eventToLogRecord(event: StatsEvent): OtlpLogRecord {attributes.push(intAttr("llm.fallback_attempts", event.fallback_attempts));}+ // Optional network-recovery fields. THIS IS THE STEP WHOSE OMISSION IS+ // SILENT — see the file header. Each is emitted only when set, so a healthy+ // request's record is byte-identical to what it was before recovery existed.+ if (event.retry_attempts !== undefined) {+ attributes.push(intAttr("llm.retry_attempts", event.retry_attempts));+ }+ if (event.recovery_ms !== undefined) {+ attributes.push(intAttr("llm.recovery_ms", event.recovery_ms));+ }+ if (event.recovery_episode_id !== undefined) {+ attributes.push(stringAttr("llm.recovery_episode_id", event.recovery_episode_id));+ }+ if (event.recovery_client_retry !== undefined) {+ attributes.push(intAttr("llm.recovery_client_retry", event.recovery_client_retry));+ }+ if (event.recovery_outcome !== undefined) {+ attributes.push(stringAttr("llm.recovery_outcome", event.recovery_outcome));+ }+return {timeUnixNano,severityNumber: 9, // INFOdiff --git a/packages/cli/src/stats.ts b/packages/cli/src/stats.tsindex 80665ae2..2ab2c23c 100644--- a/packages/cli/src/stats.ts+++ b/packages/cli/src/stats.ts@@ -181,6 +181,19 @@ export function recordStats(partial: Partial<StatsEvent>): void {if (partial.fallback_chain !== undefined) event.fallback_chain = partial.fallback_chain;if (partial.fallback_attempts !== undefined)event.fallback_attempts = partial.fallback_attempts;+ // Network recovery. Absent on every healthy request, and absent means+ // absent — a `?? 0` here would put `retry_attempts: 0` on every record+ // ever sent and make "did this turn retry" unanswerable without a join.+ // Copying them here is only half the job: `eventToLogRecord` in+ // stats-otlp.ts is a hand-written allowlist, and a field that reaches this+ // block but not that one is buffered to disk and never sent.+ if (partial.retry_attempts !== undefined) event.retry_attempts = partial.retry_attempts;+ if (partial.recovery_ms !== undefined) event.recovery_ms = partial.recovery_ms;+ if (partial.recovery_episode_id !== undefined)+ event.recovery_episode_id = partial.recovery_episode_id;+ if (partial.recovery_client_retry !== undefined)+ event.recovery_client_retry = partial.recovery_client_retry;+ if (partial.recovery_outcome !== undefined) event.recovery_outcome = partial.recovery_outcome;appendEvent(event);