Claude Code. Any Model. The most powerful AI coding agent now speaks every language.
docs(recovery): record the overlay banner, verified live through teardown
Jack Rudenkof3e262eca64f
diff --git a/CLAUDE.md b/CLAUDE.mdindex 9d2de308..7f8e15ae 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -12,8 +12,8 @@ lives in `ROADMAP.md`.- `routing.md` — `provider@model` syntax, every provider prefix, catalog-gathered route candidates and their tier order, `defaultProvider` as the fallback position, the derived picker provider list, `SUBSCRIPTION_PROVIDERS`, local models- `adapters.md` — Layers 1–3, stream parsers, error classification and retry, the 400 remap, why Gemini tool schemas need `items` on every array- `network-recovery.md` — the two-tier connection hold: the DERIVED deadline (`API_TIMEOUT_MS`, not a- watchdog), why tier 2 answers 503 and never 429, why the pane lease is heartbeat-driven, the- five auth sites+ watchdog), why tier 2 answers 503 and never 429, why the banner is magmux's `overlay` and never a+ pane of ours (closing a pane wedged Claude Code), the five auth sites- `behavior-layer.md` — Layer 4, the harness-conformance supervisor- `advisor.md` — `--advisor` for any main model: independent of `--monitor`, decorator on the routed handler, ids by tool name, retained session state, stub paths S1-S10, Claude Code's gates, metered panel billing; read before editing advisor, decorator, monitor-launch or native-auth code- `providers/devin.md`, `providers/grok-subscription.md`, `providers/antigravity.md`, `providers/qwen-alibaba.md` — one per reverse-engineered providerdiff --git a/ai-docs/architecture/README.md b/ai-docs/architecture/README.mdindex bfb57f0f..8b142114 100644--- a/ai-docs/architecture/README.md+++ b/ai-docs/architecture/README.md@@ -27,7 +27,7 @@ three earlier write-ups this material cites were lost.| [`onepassword.md`](onepassword.md) | SDK-only secret resolution, the FOUR causes of an identical denial error, the cross-process handshake lock, parent-side route pinning, and the config TUI tab | touch `providers/onepassword*.ts`, credential pre-hydration, or the 1Password TUI tab || [`keychain.md`](keychain.md) | The macOS Keychain backend: the 17ms-vs-28ms measurement that split presence from values, the two `security` behaviours that are traps (hex reads, a "secure" write that stores nothing and exits 0), and why no new `CredentialSource` member | touch `providers/keychain.ts`, `keychain-source.ts`, or the Providers tab's key write/delete || [`adapters.md`](adapters.md) | Layers 1–3 (FormatConverter / ModelTranslator / ProviderTransport), stream-parser selection, errors that ride an HTTP 200, and the `upstream_status` remap with its downstream readers | add a provider, model translator, or stream parser; or change error classification and retry |-| [`network-recovery.md`](network-recovery.md) | The two-tier connection hold: why the deadline is DERIVED from `API_TIMEOUT_MS` (measured 359.607 s, a default and not a watchdog), why the tier-2 status is 503 and never 429, why the pane's lease is heartbeat-driven and not frame-driven, and the five sites where a network failure wore an auth status code | touch the retry ladder, `recovery/`, the tier-2 status, the recovery pane, or any `x-claudish-recovery` reader |+| [`network-recovery.md`](network-recovery.md) | The two-tier connection hold: why the deadline is DERIVED from `API_TIMEOUT_MS` (measured 359.607 s, a default and not a watchdog), why the tier-2 status is 503 and never 429, why the banner is magmux's `overlay` and never a pane of ours (closing a pane wedged Claude Code), why its lease is renewed on its own tick and not by ladder events, and the five sites where a network failure wore an auth status code | touch the retry ladder, `recovery/`, the tier-2 status, the recovery banner, or any `x-claudish-recovery` reader || [`behavior-layer.md`](behavior-layer.md) | Layer 4, the harness-convention SUPERVISOR: the rule engine, the four hooks, the 115-call plan-mode measurement it was built on, and opt-in telemetry | write a `behavior/` rule, or change harness-conformance handling || [`advisor.md`](advisor.md) | `--advisor` for any main model: why it is independent of `--monitor` yet keeps four of its launch bits, the decorator around the ROUTED handler, id capture by tool name (the `toolu_` prefix that made it a no-op on every foreign provider), the tee and retained session-keyed state, the ten stub paths and advice origin, Claude Code's advisor gates, metered panel billing, and the inherited-placeholder 401 | touch `--advisor`, `advisor-decorator.ts`, `native-handler-advisor.ts`, `advisor-startup.ts`, monitor-gated launch code, or native auth in `claude-runner.ts` || [`headless-vs-interactive.md`](headless-vs-interactive.md) | Why `-p` is NOT interactive-without-a-TTY: `--agent` validation is SILENTLY skipped under `--input-format stream-json` (measured, exit 0, no stream frame, `result: success`), upstream closed it as not planned, and why magmux is in the dependency tree | drive Claude Code headlessly, add a child-spawn flag, or assume a headless run behaves like an interactive one |diff --git a/ai-docs/architecture/network-recovery.md b/ai-docs/architecture/network-recovery.mdindex e76145ee..bd39dba6 100644--- a/ai-docs/architecture/network-recovery.md+++ b/ai-docs/architecture/network-recovery.md@@ -1,5 +1,6 @@-> Two-tier connection recovery: the in-request retry ladder, the magmux recovery pane, and the-> 503 handoff that lets Claude Code's own retry loop carry an outage the socket cannot.+> Two-tier connection recovery: the in-request retry ladder, the banner drawn with magmux's own+> `overlay`, and the 503 handoff that lets Claude Code's own retry loop carry an outage the socket+> cannot.>> Indexed in [`README.md`](./README.md). Status-code neighbours: [`adapters.md`](adapters.md).@@ -85,7 +86,8 @@ The second tier hands the retry back to the client. What the client will then sp- **~11 attempts** on its default budget, spanning **~174 s** for a 503 chain (backoff caps at38.4 s per gap, and `retry-after` is honoured verbatim);- **~300 attempts** with `CLAUDE_CODE_RETRY_WATCHDOG=1`, which claudish sets only when recovery is- enabled, the UI is allowed, AND this launch can actually obtain a pane — reaching **~a day** of+ enabled, the UI is allowed, AND this launch runs inside a magmux it can draw a banner on —+ reaching **~a day** ofunattended recovery on a hung connection. All three gates, because the watchdog amplifies EVERY503 the session sees and is worth its cost only where a surface can exist (§7).@@ -107,7 +109,7 @@ POST /v1/messages ──► ComposedHandler.handle() startTime = performance.n│ non-null▼shouldSkipTier1()? probe header · recovery disabled ·- user pressed [q] · no budget left+ no budget left│ no┌───────────────────▼──────────────────────────────────────────┐│ TIER 1 — in-request, PRE-FLUSH, deadline derived from §1 │@@ -142,7 +144,7 @@ the superseded design:|---|---|---|---|| the **deadline** | one inbound request | the waiter | its own `deadlineAt` passes — *that waiter alone* answers and leaves || the **ladder** | one episode (`key`) | the coordinator | the last waiter leaves, or an attempt succeeds |-| the **lease** | one pane × one episode | the UI manager | `UI_LEASE_MS` after the last heartbeat naming that episode |+| the **lease** | one episode | the banner (`magmux-ui.ts`) | `UI_LEASE_MS` after magmux last acknowledged an overlay write painting that episode |When one waiter's deadline moved the *shared* episode into `handoff` — a state with no timer — everyother parked waiter silently stopped being retried and answered having attempted nothing. The@@ -231,30 +233,25 @@ is the only honest discriminator, and it reads the address, not the error.| condition | status | headers | why ||---|---|---|---|| recovered inside tier 1 | the upstream's own | — | the caller carries on as if the first attempt had worked |-| exhausted, **a lease is valid** | **503** `overloaded_error` | `x-should-retry: true`, `x-claudish-recovery: 1` | the reason is legible on the pane, so the retry may be handed back |+| exhausted, **a lease is valid** | **503** `overloaded_error` | `x-should-retry: true`, `x-claudish-recovery: 1` | the reason is legible on the banner, so the retry may be handed back || exhausted, **no lease** | **400** `connection_error` | — | nothing can display the reason, so it must ride the status |-| `[q] give up` | **400** `connection_error` | — | guarded on `result.kind`, never on the lease — see below |-| **a request arriving within 60 s of `[q]`** | **400** `connection_error` | — | the ladder is SKIPPED outright: the user said stop |-| client disconnected | 499 (unread) | — | the socket is already gone |+| client disconnected (Esc in Claude Code) | 499 (unread) | — | the socket is already gone || unclassifiable throw on any attempt | rethrown unchanged | — | recovery must not widen what "transient" means |-### `[q] give up` suppresses the HOLD, not only the banner+### There are no keys, and that is a trade-`bye` originally did two things, both inside the UI manager: it suppressed re-opening the pane for-60 s, and it called `giveUpAll()` — which iterates the episodes alive *at that instant*. Nothing-recorded that the user had asked recovery to stop, and `shouldSkipTier1` had no gate that could-notice. So the next request against the same dead target opened a NEW episode, found the pane-suppressed, got no pane and no lease, and then held its socket for the full ~4.5-minute deadline-before answering 400 — **a long hold with no surface, which is the state this design calls "strictly-worse than the bug this feature exists to remove", reached from the one affordance whose entire-purpose is to end it.** Before recovery existed those requests failed in milliseconds.+The first banner was a pane of claudish's own, with `[r] try now` and `[q] give up`. Closing that+pane wedged Claude Code's renderer (§4), so the banner became magmux's `overlay`, which magmux draws+itself and which therefore cannot read a keystroke. Both keys went with it, and so did the code only+they reached: the coordinator's `tryNow` and `giveUpAll`, and a process-level "the user gave up"+fact that `shouldSkipTier1` read as a skip reason.-It is not an edge case. Claude Code issues concurrent requests during an outage (main loop, title-model, subagents), so a request arriving inside the suppression window is the EXPECTED one.--The fact is therefore process-level — `recovery/settings.ts`'s `recoveryGiveUpActive()` — and is read-by **both** `shouldSkipTier1` (the hold) and `ensureRecoveryUi` (the surface). One number, one-window, two readers. A control that appears to stop something and does not is worse than no control.+What is left is enough. The ladder retries on its own, so `[r]` only ever collapsed a wait. Esc in+Claude Code aborts the held request, which the handler already treats as a client disconnect — the+waiter leaves and, if it was the last, the episode closes. What Esc does NOT do is what `[q]` did:+suppress the hold for the NEXT request. That request holds again, with the banner explaining why.+Status-line control and key forwarding are requested of magmux in its+`ai-docs/feature-request-status-line.md`; if that lands, the keys can come back.### 429 would have been a live billing bug@@ -329,66 +326,78 @@ headers from nothing, and the only path that copies upstream headers verbatim**If a future edit ever returns an upstream `Response` object on a non-ok path, the marker must bestripped there.**-### `[q] give up` is guarded on the OUTCOME, never on the lease--A lease is by definition valid at the moment the give-up key is pressed — the pane that took the-keystroke is alive. A lease-only test would answer a retryable 503, Claude Code would immediately-re-ask, and the give-up key would be a no-op with a banner still on screen.-----## 4. The lease — why the heartbeat must not be driven by frames+## 4. The banner and its leaseThe generalisable rule the two exhaustion arms encode:> **A retryable status is permissible exactly when claudish still has a surface on which the reason> is legible.** Absent such a surface, the reason must ride the status, which means 400.-The gate is not a boolean and not a latch. A latch set on `hello` and never cleared is true after-`[q] give up`, after a killed pane, after a dead magmux and after a socket EOF — and the exhaustion-arm then answers 503 with the reason visible nowhere, *which is strictly worse than the bug this-feature exists to remove*. It is a **lease**:+### The banner is magmux's `overlay`, because a pane of our own wedged Claude Code++The first banner was a pane: `open_pane` split Claude Code's pane at the start of an outage and+`close_pane` removed it after. Opening a stacked pane shrinks Claude Code; closing it grows Claude+Code back — and Claude Code 2.1.272 does not survive the grow. Six rows of its bottom chrome collapse+onto one line and the input box is destroyed, and nothing done from outside repairs it: Escape, a+keystroke, `Ctrl-L`, `SIGWINCH` at the same size, and a full shrink-then-grow cycle were each+measured and each failed. magmux is not at fault — the survivor's pty is resized to exactly the full+terminal — and the reflow is the trigger. It shipped through eight phases of validation because every+capture was taken while the pane was still open+(`reports/network-recovery-pane-close-wedges-host-tui-20260915.md`).++`overlay` draws a styled box OVER a pane and changes no layout, so there is no reflow to survive.+Measured on the replacement, in a 214×29 pane: Claude Code's `❯` composer sat on row 26 during the+outage, under the recovered banner, and after the overlay cleared, and typed text landed in it+(`reports/network-recovery-overlay-validation/`). It also retired a pane process, an NDJSON socket,+a wire protocol and a cross-process lock, because magmux is the renderer now — and in a+`team --grid`, every claudish draws on its own pane instead of one slot winning the only banner.++magmux's status bar stays hidden (`--no-status`). A controller cannot show it on demand over the+socket — only the `Ctrl-G s` key can — which is what claudish's request to magmux asks to change.++### The lease, for a renderer claudish does not own++The gate is not a boolean and not a latch. A latch set once and never cleared stays true after a dead+magmux and after a socket EOF, and the exhaustion arm then answers 503 with the reason visible+nowhere, *which is strictly worse than the bug this feature exists to remove*. It is a **lease**:```uiLeaseValid(episodeId)- ⇔ the proxy itself opened a pane and holds the `open_pane` reply- ∧ some connected client has HEARTBEATED `ack` naming THIS episode within UI_LEASE_MS+ ⇔ claudish holds a magmux control connection and knows its pane+ ∧ magmux ACKNOWLEDGED an overlay write painting THIS episode within UI_LEASE_MS```-Every clause works. *Naming this episode* makes it episode-scoped, so two concurrent episodes cannot-borrow one pane's legitimacy. *`ack`, not `hello`* makes it **post-render** — a process can send-`hello` before painting anything. *Within `UI_LEASE_MS`* makes it self-clearing on EOF, crash and-freeze, with no cleanup path to forget. *The proxy opened the pane* means a forged same-uid client-cannot manufacture the forbidden 503-with-no-banner state.--**"Heartbeated", not "renewed by frame receipts" — this is the single most consequential word here,-and the superseded design had it wrong.** Frames were emitted only while the episode was `waiting`.-There is no tick during `attempting`. A real `unreachable` connect takes 20–75 s (`192.0.2.1`-measured at **75 005 ms**), which is 2×–7.5× `UI_LEASE_MS` — so the lease expired *while the banner-was alive and painted*, and the exhaustion arm answered 400 for exactly the failure class that-motivated the feature.--The lease is a statement about **the renderer being alive and painting this episode**, so it is-renewed by the renderer on its own `UI_HEARTBEAT_MS` timer and by nothing else. The proxy's-`FRAME_TICK_MS` tick — now emitted in *every* live state, not only `waiting` — is then free to serve-the banner rather than the gate.--**Every loopback-only test passed over this bug.** A refused loopback connect resolves in ~1 ms, so-the ladder is all `waiting` and frames never stop. The regression is only visible with a slow-connect, which is why the unit test fakes a 45-second attempt (4.5 lease windows, **not one frame-emitted**) and the integration test runs a real pane against a 20-second lease clock.+*Painting this episode* makes it episode-scoped, so two concurrent episodes cannot borrow one+banner's legitimacy. *Acknowledged*, not *sent*: magmux replies only to a message that carries an+`id`, and a reply means it accepted the text for a pane it is drawing. *Within `UI_LEASE_MS`* makes+it self-clearing when magmux dies or stops answering, with no cleanup path to forget. A closed control+socket drops every lease at once.++**The renewal must not depend on the ladder — this is the single most consequential property here,+and a superseded design had it wrong.** Frames were once emitted only while the episode was+`waiting`, with no tick during `attempting`. A real `unreachable` connect takes 20–75 s (`192.0.2.1`+measured at **75 005 ms**), 2×–7.5× `UI_LEASE_MS`, so the lease expired *while the banner was alive+and painted*, and the exhaustion arm answered 400 for exactly the failure class that motivated the+feature. The banner therefore redraws on its own `FRAME_TICK_MS` (1 s) timer in every live state, and+each acknowledged write renews the lease. **Every loopback-only test passed over the original bug**: a+refused loopback connect resolves in ~1 ms, so the ladder is all `waiting` and frames never stopped.++**The overlay is re-asserted every tick, never written once.** magmux writes the same overlay from+its own Claude Code state tracker — `CtrlError` paints `✗ …` over whatever is there+(`magmux/mux/mux.go`) — so a banner written once can be replaced mid-outage. The countdown needs a+write per second anyway, and that write restores ours within a second.**The residual forbidden window is exactly `UI_LEASE_MS` wide, and that is a property of a lease-rather than a gap in it.** A renderer killed 8.7 s before exhaustion still reads-`lastAckAgoMs: 9124, valid: true`, so that exhaustion answers 503 with nothing on screen — measured-in validation, and correct: self-clearing on crash with no cleanup path to forget is the whole reason-the gate is a lease. The cost is bounded and self-repairing (the client re-asks, finds no pane, and-that request answers 400), which is why 10 s is affordable. Anyone tightening the window should move-`UI_LEASE_MS`, not add a second liveness test beside it — and must keep `UI_HEARTBEAT_MS` well under-whatever they choose.+rather than a gap in it.** If magmux dies just after acknowledging a write, the lease reads valid for+up to 10 s with nothing on screen, and an exhaustion in that window answers 503. The cost is bounded+and self-repairing — the client re-asks, finds no lease, and that request answers 400 — which is why+10 s is affordable. Anyone tightening the window should move `UI_LEASE_MS`, not add a second liveness+test beside it, and must keep `FRAME_TICK_MS` well under whatever they choose.No transition in the episode's table touches the lease, and the lease never causes a transition. The-retry loop is not the pane's business, and the pane's liveness is not the deadline's business.+retry loop is not the banner's business, and the banner's liveness is not the deadline's business.---@@ -468,7 +477,7 @@ returning a false success to attempt two.— in the banner, in the log and in the episode key — as `api.x.ai`. Grok's refresh wrapper nowattaches it, as `local.ts` already did.- **`noteTargetReachable` closes by PROVIDER, not by host.** Because an auth episode can be keyed on- a different host than the one a later success reaches, a strict key lookup missed it and the pane+ a different host than the one a later success reaches, a strict key lookup missed it and the bannerpainted "waiting for Claude Code to retry" over a working session for the full 120 s grace. Arequest that reached the model endpoint had to authenticate first, so it has proved every host ittouched is answering — and a `handoff` episode holds no waiters, so nothing is closed from under@@ -514,7 +523,7 @@ line instead.The lifecycle lines were on `log()`, which writes to two FILES: the `--debug` log and the always-onstructural log under `~/.claudish/logs/`. That satisfies "it is written down" and fails the thing itwas written down for. A hold runs for up to the derived deadline (~4.5 minutes at the default) and in-a run with no pane — which is *every headless run* — the user saw nothing at all while it did. A+a run with no banner — which is *every headless run* — the user saw nothing at all while it did. Asilent multi-minute hold with the reason legible nowhere is the state §3 calls "strictly worse thanthe bug this feature exists to remove"; a file the user does not know to open is not a surface.@@ -525,7 +534,8 @@ the only surface there is. Both log files still receive them.The line is drawn at the **episode's lifecycle** — opened, rejoined, each attempt, each wait, handedoff, closed, recovered, exhausted: the six-to-ten lines from which a reader can reconstruct the-ladder. Pane connect/disconnect, socket paths, a skipped ladder and `client_gone` stay on `log()`.+ladder. The banner's magmux connection and pane lookup, a skipped ladder and `client_gone` stay on+`log()`.### The trap: `eventToLogRecord` is a hand-written allowlist@@ -605,7 +615,7 @@ primary fetch, and spread into every re-issue. A transport that returns a one-sh`AbortSignal.timeout` therefore poisoned the whole ladder the moment it fired:`AbortSignal.any([fired, live])` is ALREADY ABORTED, so every later attempt rejected without openinga socket. For `vx@` that made tier 1 silently dead past t+30 s — the request held the full deadline-making **zero** real connect attempts while the log and the pane counted attempts that never left the+making **zero** real connect attempts while the log and the banner counted attempts that never left theprocess. `mergeSignalIntoInit` now also drops an `own` that is already aborted, as the belt behindthat brace.@@ -614,7 +624,7 @@ that brace.`recovery-disabled` on the first classified failure, and `recoverySurfaceAllowed()` — which is`resolveRecoveryEnabled() && resolveRecoveryUi()` — drops the magmux wrap, the recovery-UI installand `CLAUDE_CODE_RETRY_WATCHDOG` together. `--no-recovery-ui` / `CLAUDISH_RECOVERY_UI=0` keeps the-retries and drops only the pane (and with it the 503 arm, since the lease can never be valid).+retries and drops only the banner (and with it the 503 arm, since the lease can never be valid).**This paragraph used to claim "byte for byte", and that claim was wrong twice.** It is recordedhere rather than quietly corrected, because both halves were read as true by reviewers:@@ -623,8 +633,8 @@ here rather than quietly corrected, because both halves were read as true by rev`resolveRecoveryUi()` alone, so `--no-recovery` still started `magmux --id claudish-<pid>` —observed in the OS process table on a real launch. The switch that turns the feature off leftbehind its most user-visible cost (magmux's ring replaces the emulator's native scrollback,- RISK-4) plus ~89 ms of launch and a generated launcher script, for a pane that- `recovery-disabled` guarantees can never open. Fixed; the four-arm live matrix is+ RISK-4) plus ~89 ms of launch and a generated launcher script, for a banner that+ `recovery-disabled` guarantees can never draw. Fixed; the four-arm live matrix is`validation/c8/c8-no-recovery-wrap-live-fixed.txt`, and a source guard in`recovery/settings.test.ts` fails if either call site is reverted.2. **It is still not literally byte-for-byte about the RESPONSE**, and the two differences are@@ -648,7 +658,7 @@ here rather than quietly corrected, because both halves were read as true by revSo the honest statement is: **the opt-out restores the pre-recovery OUTCOME (an immediate 400`connection_error`, the same status and the same bytes of message) and the pre-recovery LAUNCH- (no wrap, no pane, no watchdog); it does not restore the missing marker header, and it does not+ (no wrap, no banner, no watchdog); it does not restore the missing marker header, and it does notrestore an unbounded first attempt.** Both exceptions make the opt-out path strictly betterthan what it replaced.@@ -677,7 +687,7 @@ MCP server process dying outright.## 8. What a healthy request pays**Three statements, and the guarantee around them is still placement rather than a flag**: the entire-retry apparatus — episode, ladder, waiters, pane — is constructed inside the `catch`, after+retry apparatus — episode, ladder, waiters, banner — is constructed inside the `catch`, afterclassification has already returned non-null. The `enqueueRequest` ternary that issues attempt 1 isbyte-identical to what it was before recovery existed. What a healthy request now pays, outside thatcatch, is exactly:@@ -713,7 +723,9 @@ design set for itself. Default-on survives on that evidence.|---|---|| [`network-recovery-phase0-measurements.md`](../reports/network-recovery-phase0-measurements.md) | the 359.607 s client abort and its `API_TIMEOUT_MS` cause; `c.env.timeout(req,0)`; `Request.signal` firing in 0.49–0.86 ms; the 75.004 s macOS connect; magmux's +107 ms || [`network-recovery-phase2-verification-20260911.md`](../reports/network-recovery-phase2-verification-20260911.md) | the measured 5/10/30/60/60/60 ladder with 18 ms drift over 225 s; the four modules that shipped with no tests |-| [`network-recovery-phase3-pane-20260911.md`](../reports/network-recovery-phase3-pane-20260911.md) | the lease, the frame-driven-heartbeat CRITICAL, and the two tests that can fail on it |+| [`network-recovery-phase3-pane-20260911.md`](../reports/network-recovery-phase3-pane-20260911.md) | SUPERSEDED design (the recovery pane). Still the record of the lease and the frame-driven-heartbeat CRITICAL |+| [`network-recovery-pane-close-wedges-host-tui-20260915.md`](../reports/network-recovery-pane-close-wedges-host-tui-20260915.md) | why the pane went: closing it wedged Claude Code's renderer, measured row by row, and why eight phases of validation missed it |+| [`network-recovery-overlay-validation/`](../reports/network-recovery-overlay-validation/) | the overlay banner, live: captures during the outage, at recovery and after teardown, with the recovery log and the forwarder traffic || [`network-recovery-phase4-status-flip-20260911.md`](../reports/network-recovery-phase4-status-flip-20260911.md) | the 503 flip through a real interactive session; the chain-safety mutations, including the quota-wording one || [`network-recovery-phase7-validation-20260912.md`](../reports/network-recovery-phase7-validation-20260912.md) | all nineteen acceptance criteria against HEAD: C-17 through a real stopped `ollama serve` with the probes COUNTED per attempt; the `--no-recovery` wrap regression and its four-arm live matrix; the §7 "byte for byte" comparison against a `3c1fa26` control; and three findings — **`API_TIMEOUT_MS ≤ 75 000` disables the auth-path ladder outright**, the 503-without-banner window is `UI_LEASE_MS` wide, and Bun's `os.homedir()` ignores `$HOME` |diff --git a/ai-docs/reports/network-recovery-overlay-validation/capture-teardown.ts b/ai-docs/reports/network-recovery-overlay-validation/capture-teardown.tsnew file mode 100644index 00000000..46275ab7--- /dev/null+++ b/ai-docs/reports/network-recovery-overlay-validation/capture-teardown.ts@@ -0,0 +1,55 @@+/**+ * Capture the demo pane at the moment of recovery and again after the overlay+ * clears — the state the first version never looked at.+ */+import { readdirSync, readFileSync, writeFileSync } from "node:fs";+import { join } from "node:path";++const PANE = process.argv[2] ?? "%273";+const DEMO = "/private/tmp/claude-501/-Users-jack-mag-claudish--claude-worktrees-recover/5ea44449-884c-45a6-99c5-67b0ee7d3998/scratchpad/demo";+const LOGS = join(DEMO, "work/logs");+const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));++function capture(): string {+ const out = Bun.spawnSync(["tmux", "capture-pane", "-p", "-t", PANE]).stdout.toString();+ return out+ .split("\n")+ .map((l, i) => `${String(i + 1).padStart(3)} ${l}`)+ .join("\n");+}++function log(): string {+ const f = readdirSync(LOGS).find((n) => n.endsWith(".log"));+ return f ? readFileSync(join(LOGS, f), "utf-8") : "";+}++const deadline = Date.now() + 240_000;+while (!/closed: recovered/.test(log())) {+ if (Date.now() > deadline) {+ console.log("TIMEOUT: never recovered");+ process.exit(1);+ }+ await sleep(250);+}++await sleep(400); // the success banner is written asynchronously after the close+const t0 = new Date().toISOString();+const atRecovery = capture();+await sleep(6_000); // OUTCOME_LINGER_MS is 4 000+const t1 = new Date().toISOString();+const afterTeardown = capture();+const size = Bun.spawnSync(["tmux", "display-message", "-p", "-t", PANE, "#{pane_width}x#{pane_height}"])+ .stdout.toString()+ .trim();++const report = [+ `=== AT RECOVERY ${t0} ===`,+ atRecovery,+ "",+ `=== AFTER TEARDOWN ${t1} (6 s later; linger is 4 s) ===`,+ afterTeardown,+ "",+ `pane size: ${size}`,+].join("\n");+writeFileSync(join(DEMO, "teardown-capture.txt"), report);+console.log(report);diff --git a/ai-docs/reports/network-recovery-overlay-validation/recovery-lines.txt b/ai-docs/reports/network-recovery-overlay-validation/recovery-lines.txtnew file mode 100644index 00000000..844fc70f--- /dev/null+++ b/ai-docs/reports/network-recovery-overlay-validation/recovery-lines.txt@@ -0,0 +1,15 @@+[2026-09-23T03:20:57.354Z] [Recovery] episode fdd99463-0ffa-4bc9-b87f-2fbe2b8be6a7 opened for Recov at http://127.0.0.1:54341/v1/chat/completions (refused/ConnectionRefused)+[2026-09-23T03:20:57.354Z] [Recovery] Recov attempt 1 failed: ConnectionRefused (episode fdd99463-0ffa-4bc9-b87f-2fbe2b8be6a7, ladder 0)+[2026-09-23T03:20:57.355Z] [Recovery] Recov waiting 5s before attempt 2 (episode fdd99463-0ffa-4bc9-b87f-2fbe2b8be6a7, 1 waiting)+[2026-09-23T03:20:57.355Z] [Recovery] Recov attempt 2 failed: ConnectionRefused (episode fdd99463-0ffa-4bc9-b87f-2fbe2b8be6a7, ladder 0)+[2026-09-23T03:20:57.355Z] [Recovery] banner will draw on magmux pane 0 (magmux launched by claudish)+[2026-09-23T03:20:57.355Z] [Recovery] banner painted for episode fdd99463-0ffa-4bc9-b87f-2fbe2b8be6a7 on magmux pane 0 — lease granted+[2026-09-23T03:21:02.358Z] [Recovery] Recov attempt 3 failed: ConnectionRefused (episode fdd99463-0ffa-4bc9-b87f-2fbe2b8be6a7, ladder 1)+[2026-09-23T03:21:02.358Z] [Recovery] Recov waiting 10s before attempt 4 (episode fdd99463-0ffa-4bc9-b87f-2fbe2b8be6a7, 2 waiting)+[2026-09-23T03:21:02.358Z] [Recovery] Recov attempt 4 failed: ConnectionRefused (episode fdd99463-0ffa-4bc9-b87f-2fbe2b8be6a7, ladder 1)+[2026-09-23T03:21:12.361Z] [Recovery] Recov attempt 5 failed: ConnectionRefused (episode fdd99463-0ffa-4bc9-b87f-2fbe2b8be6a7, ladder 2)+[2026-09-23T03:21:12.361Z] [Recovery] Recov waiting 30s before attempt 6 (episode fdd99463-0ffa-4bc9-b87f-2fbe2b8be6a7, 2 waiting)+[2026-09-23T03:21:12.361Z] [Recovery] Recov attempt 6 failed: ConnectionRefused (episode fdd99463-0ffa-4bc9-b87f-2fbe2b8be6a7, ladder 2)+[2026-09-23T03:21:42.475Z] [Recovery] Recov recovered after 7 attempts in 45120ms (episode fdd99463-0ffa-4bc9-b87f-2fbe2b8be6a7)+[2026-09-23T03:21:57.672Z] [Recovery] Recov recovered after 8 attempts in 60316ms (episode fdd99463-0ffa-4bc9-b87f-2fbe2b8be6a7)+[2026-09-23T03:21:57.672Z] [Recovery] episode fdd99463-0ffa-4bc9-b87f-2fbe2b8be6a7 closed: recovered after 60318ms and 8 attemptsdiff --git a/ai-docs/reports/network-recovery-overlay-validation/teardown-capture.txt b/ai-docs/reports/network-recovery-overlay-validation/teardown-capture.txtnew file mode 100644index 00000000..d8ee8b1a--- /dev/null+++ b/ai-docs/reports/network-recovery-overlay-validation/teardown-capture.txt@@ -0,0 +1,65 @@+=== AT RECOVERY 2026-09-23T03:21:58.371Z ===+ 1+ 2 ▐▛███▛█ Claude Code v2.1.280+ 3 ▝▜██████▀ recov@recov-model with xhigh effort · API Usage Billing+ 4 ▝▝ ▝▝ /private/tmp/claude-501/-Users-jack-mag-claudish--claude-worktrees-recover/5ea44449-884c-45a6-99c5-67b0ee7d3998/scratchpad/demo/work+ 5+ 6+ 7 ❯ Reply with exactly one word: banana+ 8+ 9 ∴ User just wants a simple greeting response. I should just reply that.+ 10+ 11 ⏺ Hi+ 12+ 13 ╭─────────────────────────────╮+ 14 │ ✓ NETWORK · Recov recovered │+ 15 │ 7 attempts over 1m 00s │+ 16 ╰─────────────────────────────╯+ 17+ 18+ 19+ 20+ 21+ 22+ 23 ✶ Creating… (running Stop hooks… 1/2 · 1m 1s · ↓ 15 tokens · thinking with xhigh effort)+ 24 32045 tokens+ 25 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────+ 26 ❯+ 27 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────+ 28 * recov@recov-model | I | $0.00 | 0s | 320M | ░░░░░░ 0% • $0.000 • N/A+ 29 -- INSERT -- ⏵⏵ bypass permissions on (shift+tab to cycle) · ← 3 agents+ 30++=== AFTER TEARDOWN 2026-09-23T03:22:04.384Z (6 s later; linger is 4 s) ===+ 1+ 2 ▐▛███▛█ Claude Code v2.1.280+ 3 ▝▜██████▀ recov@recov-model with xhigh effort · API Usage Billing+ 4 ▝▝ ▝▝ /private/tmp/claude-501/-Users-jack-mag-claudish--claude-worktrees-recover/5ea44449-884c-45a6-99c5-67b0ee7d3998/scratchpad/demo/work+ 5+ 6+ 7 ❯ Reply with exactly one word: banana+ 8+ 9 ∴ User just wants a simple greeting response. I should just reply that.+ 10+ 11 ⏺ Hi+ 12+ 13 ✻ Cogitated for 1m 2s · done 1:21 pm+ 14+ 15+ 16+ 17+ 18+ 19+ 20+ 21+ 22+ 23+ 24 32045 tokens+ 25 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────+ 26 ❯+ 27 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────+ 28 * recov@recov-model | I | $0.16 | 1m1s | 364M | ░░░░░░ 16% • Recov • ~$0.038 • 32k tokens+ 29 -- INSERT -- ⏵⏵ bypass permissions on (shift+tab to cycle) · ← 3 agents+ 30++pane size: 214x29\ No newline at end of filediff --git a/ai-docs/reports/network-recovery-pane-close-wedges-host-tui-20260915.md b/ai-docs/reports/network-recovery-pane-close-wedges-host-tui-20260915.mdindex 0baa640d..15d2baa0 100644--- a/ai-docs/reports/network-recovery-pane-close-wedges-host-tui-20260915.md+++ b/ai-docs/reports/network-recovery-pane-close-wedges-host-tui-20260915.md@@ -1,8 +1,11 @@# Closing the recovery pane wedges Claude Code's renderer**Found**: 2026-09-15, by the user, looking at a real session after the live demo.-**Status**: OPEN. Blocks the UI half of the network-recovery feature.-**Branch**: `worktree-recover` at `8aaa270`.+**Status**: RESOLVED 2026-09-23. The pane is gone: the banner is magmux's own `overlay` on the+pane Claude Code already occupies, so nothing reflows (§9). Verified live in a 214×29 pane — the+`❯` composer sat on row 26 during the outage, at the recovered banner, and after the overlay+cleared, and typed text landed in it — see `network-recovery-overlay-validation/`.+**Branch**: `worktree-recover`, found at `8aaa270`, fixed in `b26d0a76`.---diff --git a/packages/cli/src/recovery/coordinator.ts b/packages/cli/src/recovery/coordinator.tsindex 77fddf22..bc5ea025 100644--- a/packages/cli/src/recovery/coordinator.ts+++ b/packages/cli/src/recovery/coordinator.ts@@ -171,8 +171,11 @@ export function describeEpisode(episodeId: string): Readonly<Record<string, unknexport interface RecoveryUiHooks {/** A new episode exists. The manager starts drawing the banner. */onEpisodeOpened(episodeId: string): void;- /** An episode reached a terminal state. */- onEpisodeClosed(episodeId: string, outcome: RecoveryOutcome): void;+ /**+ * An episode reached a terminal state. `attempts` is the FINAL count, the+ * winning attempt included; the last frame the banner drew predates it.+ */+ onEpisodeClosed(episodeId: string, outcome: RecoveryOutcome, attempts: number): void;/** Is a renderer painting this episode RIGHT NOW? */leaseValid(episodeId: string): boolean;}@@ -594,7 +597,7 @@ function closeEpisode(episode: Episode, outcome: RecoveryOutcome): void {);if (uiHooks) {try {- uiHooks.onEpisodeClosed(episode.episodeId, outcome);+ uiHooks.onEpisodeClosed(episode.episodeId, outcome, episode.attempts);} catch {/* see onEpisodeOpened — the UI never breaks recovery */}diff --git a/packages/cli/src/recovery/magmux-ui.ts b/packages/cli/src/recovery/magmux-ui.tsindex ea9bb542..cdb7e0dd 100644--- a/packages/cli/src/recovery/magmux-ui.ts+++ b/packages/cli/src/recovery/magmux-ui.ts@@ -392,20 +392,26 @@ export function bannerText(frame: RecoveryEpisodeFrame, nowMs: number): string {lines.push(...wrap(frame.reason, OVERLAY_WRAP_COLS));lines.push(`${frame.host} · ${frame.code ?? frame.kind}${frame.loopback ? " · local" : ""}`);const elapsed = formatElapsed(nowMs - frame.startedAtMs);+ // An episode exists only because an attempt already failed, but the banner is+ // first drawn the instant the episode opens — before the coordinator has+ // counted that attempt or registered the request holding it. Measured: the+ // first frame read `attempt 0` and `0 requests held` for up to a second.+ const attempt = Math.max(1, frame.attempts);if (frame.state === "handoff") {lines.push(`handed back to Claude Code · waiting for its retry · ${elapsed} in recovery`);} else if (frame.nextAttemptAtMs === null) {- lines.push(`connecting to ${frame.host}… · attempt ${frame.attempts} · ${elapsed} in recovery`);+ lines.push(`connecting to ${frame.host}… · attempt ${attempt} · ${elapsed} in recovery`);} else {const secs = Math.max(0, Math.ceil((frame.nextAttemptAtMs - nowMs) / 1000));- lines.push(`next attempt in ${secs}s · attempt ${frame.attempts} · ${elapsed} in recovery`);+ lines.push(`next attempt in ${secs}s · attempt ${attempt} · ${elapsed} in recovery`);}- const held = `${frame.waiters} request${frame.waiters === 1 ? "" : "s"} held`;- const more =- frame.otherEpisodes > 0- ? ` · +${frame.otherEpisodes} more outage${frame.otherEpisodes === 1 ? "" : "s"}`- : "";- lines.push(`${held}${more}`);+ const counts: string[] = [];+ if (frame.waiters > 0)+ counts.push(`${frame.waiters} request${frame.waiters === 1 ? "" : "s"} held`);+ if (frame.otherEpisodes > 0) {+ counts.push(`+${frame.otherEpisodes} more outage${frame.otherEpisodes === 1 ? "" : "s"}`);+ }+ if (counts.length > 0) lines.push(counts.join(" · "));lines.push("Esc in Claude Code stops the turn");return lines.join("\n");}@@ -521,7 +527,7 @@ function onEpisodeOpened(episodeId: string): void {* One episode ended. The coordinator has already removed it from the live set,* so `renderableEpisodeFrame()` here answers "is another outage still live?".*/-function onEpisodeClosed(episodeId: string, outcome: RecoveryOutcome): void {+function onEpisodeClosed(episodeId: string, outcome: RecoveryOutcome, attempts: number): void {state.leases.delete(episodeId);if (renderableEpisodeFrame() !== null) return; // the tick keeps drawing the other onestopTick();@@ -533,7 +539,9 @@ function onEpisodeClosed(episodeId: string, outcome: RecoveryOutcome): void {void clearOverlay();return;}- void writeOverlay(recoveredText(last, Date.now()), "success", "green");+ // The final count, not the last painted frame's: that frame predates the+ // attempt that succeeded. Measured: the banner read 7 while the log said 8.+ void writeOverlay(recoveredText({ ...last, attempts }, Date.now()), "success", "green");cancelLinger();const clock = recoveryClock();const timer = clock.setTimeout(() => {