Claude Code. Any Model. The most powerful AI coding agent now speaks every language.
feat(recovery): give the retry ladder a face — the magmux recovery pane
Jack Rudenkofce758eab2ca
diff --git a/ai-docs/reports/network-recovery-phase3-pane-20260911.md b/ai-docs/reports/network-recovery-phase3-pane-20260911.mdnew file mode 100644index 00000000..6e6afa96--- /dev/null+++ b/ai-docs/reports/network-recovery-phase3-pane-20260911.md@@ -0,0 +1,328 @@+# Network recovery, design-Phase 3 — the recovery pane, measured++> Promoted out of the gitignored session directory deliberately, exactly as+> `network-recovery-phase0-measurements.md` and `network-recovery-phase2-verification-20260911.md`+> were. CLAUDE.md records three write-ups already lost because they lived in `ai-docs/sessions/`,+> which does not survive a fresh clone or `git worktree remove`.+>+> Session: `dev-feature-network-recovery-20260910-110514-0ad0a5a0` · measured 2026-09-11 on+> macOS 25.6.0, bun 1.4.0, `magmux 0.11.0 (8ce5ef8)`. Baseline commit `343e35e`.++Phase 2 shipped the tier-1 retry ladder. Phase 3 gives it a face: a magmux pane claudish owns, a+versioned NDJSON link between the two processes, and the **lease** that will decide, one phase from+now, whether an exhausted episode may answer a retryable status. **Nothing here changes a status.**+The lease is computed and logged only — shipping the 503 flip before the banner exists would+recreate the buried-reason bug this feature exists to kill.++Suite: **3271 → 3331 pass**, 18 → 18 skip, 2 → 2 fail (+60 tests, zero new failures). The two are the+pre-existing `displayWidth` Unicode-oracle cases in `tui/viz/color.test.ts`, red on clean `main`+under bun 1.4.0 while CI pins 1.3.10. `typecheck` clean. **`bun run lint` now exits 0, which it did+not at `343e35e`.**++One run in the middle of this work showed a third failure — `fallback-handler.test.ts`'s+`kimi-k2.5 without prefix` at exactly 30 001 ms, a live-API call timing out under load. It passes+alone and passed on the next full run; it is the documented contention-flake class, not a+regression.++---++## 1. Two uncertainties, resolved before any production code++### U-4 — `open_pane`'s `split` is a 50/50 split, not a strip++One live `open_pane` against a headless magmux at `COLUMNS=120 LINES=40`, `list` before and after:++```+before: {"pane":0,"cols":120,"rows":40,"state":"running"} ← the whole terminal+after: {"pane":0,"cols":120,"rows":20,"state":"running"} ← HALVED+ {"pane":1,"control":true,"hidden":true,"state":"panel"}+ {"pane":2,"cols":120,"rows":19,"state":"running","cmd":"/bin/zsh -l -c …"}+```++There is no strip option in the protocol. **The architecture's stated fallback is taken: accept+50/50 for the outage's duration.** Claude Code handles SIGWINCH, and the pane is closed again when+the last episode ends.++Two further facts from the same capture, both load-bearing:++1. **The opened pane is index 2, not 1.** magmux's control panel occupies index 1 — its own help+ says "magmux's own panes are appended after them, whether or not `-c` was passed". The index is+ read from the `open_pane` reply (`{"id":1,"ok":true,"result":{"pane":2}}`) and never assumed.+2. **`cmd` really does run through the login shell** — `"/bin/zsh -l -c …"`, verbatim. RISK-5 is not+ hypothetical, so the launcher script is required rather than defensive.++### U-6 — magmux answers OSC 11, so theme detection works inside the pane++A probe run AS a pane command, in a headless magmux with no terminal behind it at all:++```+stdin.isTTY=true stdout.isTTY=true+TERM=screen-256color TERM_THEME=light MAGMUX_THEME=light COLORFGBG=15;0+MAGMUX_SOCK=/tmp/magmux-osc11probe.sock+RESULT: reply="\x1b]11;rgb:efef/f1f1/f5f5\x07" len=24+```++magmux synthesises the reply itself, so the bounded OSC 11 query in `theme-mode.ts` resolves inside+a pane — and outranks the `COLORFGBG=15;0` it also inherits, which would have said *dark*: the exact+lie `theming.md` demoted `COLORFGBG` for. Safe either way, because the banner's fill is deliberately+theme-independent (self-contained mid-dark red + bright-white ink, the `team-grid.ts` precedent), but+now known rather than assumed. `MAGMUX_SOCK` is confirmed exported to children.++---++## 2. The pane, live — C-3, C-4, C-5++Everything below `installRecoveryUi` in the harness is production code: the real coordinator, the+real UI manager, the real socket server, the real `claudish recovery-pane` process launched by+`open_pane`, inside a real magmux. The harness supplies only the two ends — a genuinely refused+endpoint and a stand-in for the Claude Code pane.++**On capture.** magmux's control socket has no screenshot command — `snapshot`, `screenshot` and+`capture` were each tried live and none returns pane text — and this session had no access to the+terminal plugin's `mux` MCP tools. So magmux was given a real PTY and its byte stream replayed+through a small VT grid (`phase3/vt.ts`), which is how a terminal produces a screen in the first+place. What follows is that grid.++### C-3 — the red banner, with provider, host and reason++```+CLAUDE CODE (stand-in) — this pane is the session+────────────────────────────────────────────────────────────────────────────────────────────────────+ ██ NETWORK · Live-probe refused+ Cannot connect to Live-probe at http://127.0.0.1:53999/v1/messages. Make sure the server is+ running.+ host 127.0.0.1:53999 · ConnectionRefused · local+ next attempt in 2s · attempt 1 · 3s in recovery+ last: ConnectionRefused · 1 request held+ click here or Ctrl-G Tab, then [r] try now · [q] give up+```++The sentence is `buildConnectionErrorMessage`'s own, verbatim — the same words the inline 400 would+have carried. If those two ever diverge, the user is being told two different stories about one+fault.++### C-4 — the countdown is live++Two captures four seconds apart, during the 30-second rung:++```+ next attempt in 28s · attempt 3 · 17s in recovery ← A+ next attempt in 24s · attempt 3 · 21s in recovery ← B, +4s+```++### C-5 — `[r]` collapses the wait++`focus` the recovery pane over the control socket, then one `r` into magmux's PTY:++```+[Recovery] attempt 4 (manual) for episode b6475bcd-…+[Recovery] Live-probe attempt 4 failed: ConnectionRefused (episode b6475bcd-…, ladder 3)+```++Keypress → attempt: **1 ms**, against a scheduled wait that still had **23 s** to run. The banner+then shows the next rung, `next attempt in 57s`, and a transient `retrying now…`.++The full episode log for that run, which also shows the pane attaching and the lease being granted:++```+[Recovery] episode b6475bcd-… opened for Live-probe at http://127.0.0.1:53999/v1/messages (refused/ConnectionRefused)+[Recovery] Live-probe attempt 1 failed: ConnectionRefused (…, ladder 0)+[Recovery] Live-probe waiting 5s before attempt 2 (…, 1 waiting)+[Recovery] socket listening at /tmp/claudish-recovery-27799446dda0a9a875cee1f0/r.sock+[Recovery] recovery pane opened (magmux pane 2) at /tmp/claudish-recovery-…/r.sock+[Recovery] pane connected (pid 68812, protocol 1)+[Recovery] pane is painting episode b6475bcd-… — lease granted (paneOpen=true)+[Recovery] Live-probe attempt 2 failed: ConnectionRefused (…, ladder 1)+[Recovery] Live-probe waiting 10s before attempt 3 (…, 1 waiting)+[Recovery] Live-probe attempt 3 failed: ConnectionRefused (…, ladder 2)+[Recovery] Live-probe waiting 30s before attempt 4 (…, 1 waiting)+[Recovery] attempt 4 (manual) for episode b6475bcd-…+[Recovery] Live-probe attempt 4 failed: ConnectionRefused (…, ladder 3)+[Recovery] Live-probe waiting 60s before attempt 5 (…, 1 waiting)+[Recovery] client gone during wait (episode b6475bcd-…)+[Recovery] episode b6475bcd-… closed: client_gone after 25412ms and 4 attempts+```++**The pane opens 4 ms after the episode does, and is painting 115 ms after that.** The socket path+carries 96 bits of randomness rather than the proxy port, which is printed at startup.++---++## 3. C-7 — the healthy request path, against `343e35e`++Measured where the request path lives: `ComposedHandler.handle()` against a REACHABLE upstream — a+real `Bun.serve` speaking real SSE over a real socket — timed to the first byte of the response+body. 40 samples each, after 10 warm-up calls. The `343e35e` column is a second worktree checked out+at that commit, running the same benchmark file.++| | median | p90 | min |+|---|---|---|---|+| `343e35e` (the phase-2 baseline) | **0.618 ms** | 5.047 ms | 0.367 ms |+| phase 3, recovery UI not installed | **0.627 ms** | 1.835 ms | 0.347 ms |+| phase 3, recovery UI installed | **0.448 ms** | 3.423 ms | 0.307 ms |++Δ median vs the baseline: **+0.009 ms** and **−0.170 ms** — both inside the run-to-run spread. That+is what the structure predicts: the only thing phase 3 adds to a request is one `uiLeaseValid()` call+**inside the exhaustion arm of a catch block**, which a healthy request never enters.++**What this does NOT measure: a real provider.** No local model server was running and the sandbox+had no credentials, so the upstream is a stub. The stub exercises the whole handler — adapters,+stream parser, response construction — and differs from a real provider only in network latency,+which is common to both columns.++## 4. C-16 / C-7's separate half — the magmux launch overhead++Spawn to the child's first frame, wrapped versus direct, 7 samples each, with a stand-in for Claude+Code so the number is the wrapper's cost and nothing else:++| | median | p90 | min | max |+|---|---|---|---|---|+| direct (login shell only) | 39.4 ms | 55.4 ms | 33.8 ms | 55.4 ms |+| wrapped in magmux | 136.2 ms | 179.4 ms | 131.4 ms | 179.4 ms |++**Δ median: 96.8 ms.** The bar is 500 ms; above 1 s revisits default-on. Phase 0 measured +107 ms+median on the same machine for the same thing, so this reproduces to within 10 ms.++**magmux's own exit status does not carry the pane's** — measured: pane 0 exited 42 while magmux+stayed alive waiting for a second pane. It does announce the death on the control socket+(`{"type":"exit","pane":0,"exitCode":42}`), so that event is the authority for Claude Code's exit+code, and it is also what lets the recovery pane be closed the instant Claude Code is gone rather+than after its linger — which matters because `-w` waits for EVERY pane.++---++## 5. C-8 — the precedence matrix, one real process per row++`flag > env > project > global > true`, nine rows, a fresh process each (the flag layer is module+state and the config readers hit real files, so two rows in one process would test the leftovers of+the first). `HOME` is redirected per row, so nothing reads or writes the user's own configuration.+Each row reports both what the switch resolves to and what it causes — whether a magmux wrap plan is+built — because a switch is only meaningful through its effect.++| flag | env | project | global | `resolveRecoveryUi()` | magmux wrap built | expected |+|---|---|---|---|---|---|---|+| — | — | — | — | **true** | true | true |+| — | — | — | `{"enabled":false}` | **false** | false | false |+| — | — | — | `{"enabled":true}` | **true** | true | true |+| — | — | `{}` | `{"enabled":false}` | **false** | false | false |+| — | — | `{"enabled":false}` | `{"enabled":true}` | **false** | false | false |+| — | `0` | `{"enabled":true}` | — | **false** | false | false |+| — | `1` | — | `{"enabled":false}` | **true** | true | true |+| `--recovery-ui` | `0` | `{"enabled":false}` | `{"enabled":false}` | **true** | true | true |+| `--no-recovery-ui` | `1` | — | — | **false** | false | false |++All nine pass. Row 4 is the one worth having: a project file carrying `"recoveryUi": {}` means **no+opinion** and falls through to the global scope rather than reading as `true`. That is why the config+field is an object rather than a bare boolean, and it is invisible to any test that only writes+`true`/`false`.++---++## 6. The lease — the round-2 CRITICAL, and the test that can fail on it++The rule: *a retryable status is permissible exactly while claudish still has a surface on which the+reason is legible.* The superseded design renewed that lease from **frame receipts**, and frames+ticked only while the ladder was `waiting`. A connect against an unreachable host takes 20–75 s+(measured on this machine: `192.0.2.1` = 75 005 ms), during which the proxy has nothing new to say —+so the lease died ten seconds into every slow attempt, and the exhaustion arm, which reads it+immediately after an attempt returns, saw it false for the **entire failure class the feature was+built for**, with a live painted banner on screen. Every test passed, because every fault in the+suite was a ~1 ms loopback refusal.++What ships instead:++```+uiLeaseValid(id) ⇔ paneOpen (WE opened it, and hold the open_pane reply)+ ∧ some client heartbeated `ack` naming THIS episode within UI_LEASE_MS+```++- **granted by an `ack` sent AFTER PAINTING**, not on frame receipt;+- **renewed by the renderer's own 1 Hz timer**, independent of frame arrival;+- **revoked** by `bye`, by socket EOF, by a killed or frozen pane — all through one expiry path;+- **`paneOpen` is required**, so a forged same-uid client cannot manufacture the forbidden state.++Two tests exist specifically to fail on the old behaviour:++1. *unit, fake clock* — 45 seconds of attempt, 4.5 lease windows, **not one frame emitted**; the+ renderer heartbeats on its own timer and the lease holds throughout.+2. *end to end, real socket, real renderer* — the server sends exactly ONE frame (the replay) and+ then nothing, which is what a hung connect looks like on the wire. The real+ `claudish recovery-pane` still emits ≥3 acks and the lease is valid with the lease clock 20 s past+ that only frame. **Mutating the renderer back to ack-per-frame fails this test.**++Also pinned: the lease expires exactly `UI_LEASE_MS` after the last heartbeat; it is per-episode, so+heartbeating one grants nothing to another; switching the painted episode lets the old lease lapse+with no extra protocol; and `[q]` closes the pane so that no later heartbeat can resurrect anything.++---++## 7. Mutation proof — 21 properties, 21 killed++Each mutation reverts ONE property, runs the tests that claim to pin it, and restores the file **by+copy — never with git**, because the stash stack and the index are shared with the main checkout and+every sibling worktree.++Killed: replay-on-connect · the tick in every live state · the frame version gate · `hello` parsing+at the wrong version · the 0600 socket / 0700 directory · unlinking both on close · the countdown+reading the live clock · **the palette read at RENDER time** · `attempting` showing the connect+rather than a frozen countdown · the dead-proxy notice threshold · `q` sending `bye` · `r` sending+`retry_now` · clipping to the pane width · **only the deltas reaching disk** · the launcher script's+0600 mode · **exporting AFTER the profile** · stripping `CLAUDECODE` / `CLAUDE_CODE_CHILD_SESSION` ·+the no-nesting gate · the interactive + TTY gate · `--id` / `--no-idle-done` / `-w` / `--no-status` ·+`MAGMUX_SCROLLBACK` never overriding the user. Plus the four lease mutations above, run separately.++One near-miss worth recording: `writeFileSync(..., { mode: 0o600 })` and the `chmodSync` that follows+it are **both** present, and only the chmod is observable — `mode:` is subject to the umask, so+mutating it alone leaves the final mode correct. `mode:` still earns its place (it closes the window+between creation and chmod during which the file would be world-readable), but no assertion on a+finished file can see a race, so it is documented rather than pinned.++---++## 8. Three things found while building, none of them in the plan++### 8.1 A `team --grid` slot would have wrapped a second magmux inside its own pane++`team --grid --mode interactive` launches one `claudish --model X -i` per pane. Each of those IS+interactive with a real TTY, so the wrapper's stated gate (`interactive && stdout is a TTY && the+binary resolves`) is satisfied in every slot — and each would have started its own magmux **inside+the pane it was already running in**. N nested multiplexers, each with its own control socket and its+own idea of the layout.++magmux exports `MAGMUX_SOCK` to its children precisely so a child can tell. The wrapper now returns+null when it is set, and `claude-runner` installs the recovery UI against that **ambient** socket+instead. The cross-process `O_EXCL` lock then keeps the grid to ONE banner: the winner serves it, the+losers still retry and still recover, hold no lease, and answer inline at exhaustion — which is+exactly what the biconditional requires.++### 8.2 The pane command must be built from `process.argv`, and the first attempt proved why++`recoveryPaneCommandLine()` builds `<runtime> <script> recovery-pane --socket <path>` from+`process.argv`, which is correct in production because `argv[1]` IS claudish's entry point. The first+live run had `argv[1]` pointing at the harness, so the pane **re-launched the harness** — it printed+"waiting for magmux…" into the banner's pane and left a directory of captures named `recovery-pane/`+in the repo root. The failure mode is precisely the one the builder exists to avoid (a pane that is+not this build), and it is loud rather than silent, which is the good news.++### 8.3 A split pane is half the terminal, so every line is clipped to its width++Following from U-4: the banner is rendered into a pane that may be 40 columns wide. Every line now+goes through one clipping step that steps OVER the escapes — `String.length` counts an SGR sequence+as eight or nine characters and would cut a line mid-escape, leaving the rest of the pane painted in+whatever colour the fragment started. Below 64 columns the key hint sheds the part a user can work+out (where to click) and keeps the part they cannot (which keys exist).++---++## 9. What is NOT verified, and why++- **C-7 against a real provider.** No local model server was running and the sandbox has no+ credentials; the measurement above uses a stub upstream through the real handler. The stub differs+ from a provider only in network latency, which is common to both columns.+- **C-16's Ctrl-C and exit-code propagation in a real terminal.** The exit-code path is implemented+ from the measured `{"type":"exit","pane":0,"exitCode":42}` event and unit-tested at the plan level,+ but driving a real Ctrl-C into a real attached terminal needs a TTY this session does not have.+ The `[Claude Code] Exited…` structural log line is unchanged code on an unchanged path.+- **C-6, C-12, C-14, C-15, C-17, C-18, C-19.** Not in this phase's scope. C-18's lease half is+ covered by §6 — the lease is logged at exhaustion (`ui_lease=…`) rather than acted on, one phase+ before it can cost anything.diff --git a/packages/cli/src/claude-runner.ts b/packages/cli/src/claude-runner.tsindex 1bba166b..e6b9a36e 100644--- a/packages/cli/src/claude-runner.ts+++ b/packages/cli/src/claude-runner.ts@@ -17,6 +17,7 @@ import { isatty } from "node:tty";import { lookupModelForProvider } from "./adapters/model-catalog.js";import { classifierPassthroughEnabled } from "./classifier-passthrough.js";import { ENV } from "./config.js";+import { planMagmuxWrap } from "./launcher/magmux-wrapper.js";// Aliased: runClaudeWithProxy declares its own local `log` (a quiet-aware// console printer), and an unaliased import would be shadowed inside it.import { log as debugLog, logStderr } from "./logger.js";@@ -25,6 +26,8 @@ import { discoverContextWindow } from "./providers/model-discovery.js";import { parseModelSpec } from "./providers/model-parser.js";import { getProviderByName } from "./providers/provider-definitions.js";import { route } from "./providers/routing-rules.js";+import { installRecoveryUi, shutdownRecoveryUi } from "./recovery/magmux-ui.js";+import { resolveRecoveryUi } from "./recovery/settings.js";import { setClaudeCodeRunning } from "./telemetry.js";import { beginTerminalIsolation } from "./terminal-isolation.js";import { getThemeMode } from "./theme/theme-mode.js";@@ -1932,11 +1935,66 @@ export async function runClaudeWithProxy(? ["inherit", "inherit", "pipe"]: "inherit";- const proc = spawn(spawnCommand, claudeArgs, {- env,- stdio,- shell: needsShell,- });+ // Wrap the spawn in a magmux pane when — and ONLY when — the session is+ // interactive, stdout is a real TTY, and the binary resolves. That is the+ // whole gate: `-p`, `--stdin`, `serve`, the MCP server and the `team --grid`+ // panes take the untouched path below, and a machine without magmux launches+ // exactly as it did before this feature existed.+ //+ // The pane is what earns claudish the right to WAIT on a network fault+ // instead of killing the turn: it is the only surface claudish owns during+ // an interactive session, and a retryable status is only honest while the+ // reason is legible somewhere.+ //+ // `resolveRecoveryUi()` is the user's half of the gate — flag > env > project+ // > global > true. It decides whether claudish may own a SURFACE; it never+ // decides whether a retry happens. Off means no pane, therefore no lease,+ // therefore an inline error at exhaustion — which is the 503-vs-400 decision+ // made structurally rather than by a second predicate somewhere else.+ const wrap = resolveRecoveryUi()+ ? planMagmuxWrap({+ claudeBinary,+ claudeArgs,+ childEnv: env,+ cwd: process.cwd(),+ interactive: Boolean(config.interactive),+ stdoutIsTty: Boolean(process.stdout.isTTY),+ })+ : null;++ const proc = wrap+ ? spawn(wrap.command, wrap.args, { env: wrap.env, stdio, shell: false })+ : spawn(spawnCommand, claudeArgs, {+ env,+ stdio,+ shell: needsShell,+ });++ if (wrap) {+ wrap.watch(proc);+ // Hand the control socket to the proxy as an EXPLICIT input. It cannot be+ // discovered: claudish is magmux's parent and `MAGMUX_SOCK` is exported+ // downward only, so the `--id` path — fixed before magmux starts — is the+ // only way the parent can ever know where to send `open_pane`.+ installRecoveryUi(wrap.controlSocket);+ wrap.onClaudeExit(() => {+ // Claude Code is gone, so nothing can consume a recovery any more. Take+ // the pane down at once rather than after its linger: `-w` waits for+ // EVERY pane, so a lingering banner would hold the whole session open+ // after the user had already quit.+ void shutdownRecoveryUi();+ });+ } else if (resolveRecoveryUi() && config.interactive && process.env.MAGMUX_SOCK) {+ // Already inside someone else's magmux — `team --grid --mode interactive`,+ // or a user who launched claudish in a pane by hand. There is nothing to+ // wrap, but there IS a multiplexer to ask for a pane, so the recovery UI+ // installs against the ambient control socket. In a grid this runs in N+ // processes at once; the cross-process `O_EXCL` lock decides which one+ // serves the banner, and the losers still retry and still recover — they+ // simply hold no lease and answer inline at exhaustion, which is exactly+ // what "a retryable status only while the reason is legible" requires.+ installRecoveryUi(null);+ }if (filterChildStderr && proc.stderr) {relayChildStderr(proc.stderr);@@ -1988,12 +2046,25 @@ export async function runClaudeWithProxy(}>((resolve) => {proc.on("exit", (code, signal) => {setClaudeCodeRunning(false);+ // Under the wrapper the process that just exited is MAGMUX, and magmux's+ // status does not carry the pane's — measured: pane 0 exited 42 while+ // magmux stayed alive waiting for a second pane. The pane's own exit is+ // announced on the control socket, and that announcement is the+ // authority for Claude Code's exit code. Falling back to magmux's is+ // right for the case where magmux itself died first.+ const paneExit = wrap?.paneExitCode() ?? null;resolve({- exitCode: signal ? 128 + (SIGNAL_EXIT_NUMBERS[signal] ?? 0) : (code ?? 1),+ exitCode:+ paneExit !== null+ ? paneExit+ : signal+ ? 128 + (SIGNAL_EXIT_NUMBERS[signal] ?? 0)+ : (code ?? 1),exitSignal: signal,});});});+ wrap?.cleanup();// The only durable record of WHY the run ended. The always-on session log// carries proxy traffic, so a session that died before its first request leftdiff --git a/packages/cli/src/cli.ts b/packages/cli/src/cli.tsindex 1b73a2bd..028cbe91 100644--- a/packages/cli/src/cli.ts+++ b/packages/cli/src/cli.ts@@ -69,6 +69,7 @@ import {loadRoutingRules,matchRoutingRule,} from "./providers/routing-rules.js";+import { setRecoveryFlagOverrides } from "./recovery/settings.js";import { cliAnsi } from "./theme/ansi.js";import type { ClaudishConfig } from "./types.js";import { VERSION } from "./version.js";@@ -338,6 +339,19 @@ export async function parseArgs(args: string[]): Promise<ClaudishConfig> {if (config.logLevel === "info") {config.logLevel = "debug";}+ } else if (arg === "--recovery-ui" || arg === "--no-recovery-ui") {+ // Whether claudish may own a surface — a magmux pane — on which a network+ // outage's reason is legible. The flag layer of the precedence chain+ // `flag > env > project > global > true` that `recovery/settings.ts`+ // implements; the same shape `--debug-claudish` uses two branches up.+ setRecoveryFlagOverrides({ recoveryUi: arg === "--recovery-ui" });+ } else if (arg === "--recovery" || arg === "--no-recovery") {+ // The master switch for the retry ladder ITSELF, independent of the UI.+ // Off restores the pre-recovery behaviour everywhere, byte for byte, which+ // is what makes it the documented CI switch: a scripted `-p` against a+ // dead endpoint goes back to failing in milliseconds instead of holding+ // the request for the whole tier-1 deadline.+ setRecoveryFlagOverrides({ recovery: arg === "--recovery" });} else if (arg === "--no-debug-claudish") {// Escape hatch when debug is globally enabled (config.json / CLAUDISH_DEBUG):// turn the debug file log off for this single run.diff --git a/packages/cli/src/handlers/composed-handler.ts b/packages/cli/src/handlers/composed-handler.tsindex 4b3fb181..fc6d8bfa 100644--- a/packages/cli/src/handlers/composed-handler.ts+++ b/packages/cli/src/handlers/composed-handler.ts@@ -39,6 +39,7 @@ import { deepMergeParams } from "../model-params.js";import { describeSiblingKeys, getProviderByName } from "../providers/provider-definitions.js";import { isTerminal429 } from "../providers/transport/openai.js";import { recoveryClock } from "../recovery/clock.js";+import { uiLeaseValid } from "../recovery/coordinator.js";import { resolveRecoveryEnabled } from "../recovery/settings.js";import {type OpenAIImageBlock,@@ -548,10 +549,21 @@ export class ComposedHandler implements ModelHandler {// socket that is already gone.return { kind: "respond", response: new Response(null, { status: 499 }) };default: {+ // THE LEASE IS READ HERE AND NOWHERE ELSE, at the instant the status is+ // chosen — never cached off the result object, which would be stale by+ // the time it was destructured.+ //+ // In THIS phase it only goes into the log. That is deliberate: shipping+ // the 503 flip before the banner exists recreates the buried-reason bug+ // this whole feature exists to kill. What the log buys now is the+ // ability to prove, one phase early, that the lease is still valid at+ // exhaustion after a 45-second attempt — the defect that made every+ // loopback-only test pass while the feature did not work.+ const leased = uiLeaseValid(result.episodeId);log(`[Recovery] ${this.provider.displayName} exhausted after ${result.attempts} attempts ` +- `in ${result.recoveryMs}ms (episode ${result.episodeId}, outcome ${result.kind}) — ` +- "answering connection_error"+ `in ${result.recoveryMs}ms (episode ${result.episodeId}, outcome ${result.kind}, ` ++ `ui_lease=${leased}) — answering connection_error`);return {kind: "respond",diff --git a/packages/cli/src/index.ts b/packages/cli/src/index.tsindex 37c6317f..d702af4b 100644--- a/packages/cli/src/index.ts+++ b/packages/cli/src/index.ts@@ -52,15 +52,25 @@ function classifyStartupKind(): string {"quota","usage",]);- if ((first && management.has(first)) || argv.includes("--mcp") || first === "serve") {+ if (+ (first && management.has(first)) ||+ argv.includes("--mcp") ||+ first === "serve" ||+ first === "recovery-pane"+ ) {return "other";}return "run";}process.on("exit", () => {const argv = process.argv.slice(2);+ const first = argv.find((a) => !a.startsWith("-"));+ // `recovery-pane` joins the long-running exclusions for the same reason+ // `serve` and `--mcp` are there: it lives for as long as the outage does, so+ // an at-exit total is its lifetime rather than anyone's startup, and it would+ // pollute the metrics with a number that measures a network failure.const longRunningServer =- argv.includes("--mcp") || argv.find((a) => !a.startsWith("-")) === "serve";+ argv.includes("--mcp") || first === "serve" || first === "recovery-pane";if (longRunningServer) return;finalizeStartupTrace(classifyStartupKind(), { quiet: true });});@@ -318,6 +328,11 @@ const isStatsCommand = firstPositional === "stats";const isConfigCommand = firstPositional === "config";// Serve subcommand: claudish serve --port <n> --models <path> (Claude Desktop redirect gateway)const isServeCommand = firstPositional === "serve";+// Recovery pane: claudish recovery-pane --socket <path>. Launched BY claudish+// into a magmux pane, never by a user — but it must be routed here all the+// same, because an unrouted subcommand does not error: it falls through to the+// default path and becomes a catalog search for the literal string.+const isRecoveryPaneCommand = firstPositional === "recovery-pane";// Providers subcommand: claudish providers --json (credential presence, no key material)const isProvidersCommand = firstPositional === "providers";// Keychain subcommand: claudish keychain status|list|import|set|rm|enable|disable@@ -373,6 +388,17 @@ if (isMcpMode) {process.exit(1);}));+} else if (isRecoveryPaneCommand) {+ // The network-recovery banner. Its own process, its own PTY, its own stdin —+ // which is the entire reason the recovery UI is a pane: keys typed into it+ // reach THIS process rather than the Claude Code session beside it.+ const paneArgIndex = args.indexOf("recovery-pane");+ import("./recovery/pane-app.js").then((m) =>+ m.recoveryPaneCommand(args.slice(paneArgIndex + 1)).catch((e) => {+ console.error(`[claudish recovery-pane] ${e instanceof Error ? e.message : String(e)}`);+ process.exit(1);+ })+ );} else if (isBehaviorCommand) {// Layer 4 introspection: claudish behavior rules|corpus. Read-only unless// --write is passed to `corpus`.diff --git a/packages/cli/src/launcher/magmux-binary.ts b/packages/cli/src/launcher/magmux-binary.tsnew file mode 100644index 00000000..5757784a--- /dev/null+++ b/packages/cli/src/launcher/magmux-binary.ts@@ -0,0 +1,74 @@+/**+ * Where magmux lives, in the three shapes claudish ships in.+ *+ * Extracted from `team-grid.ts` rather than copied, because a second roster is+ * a second thing to forget: `team --grid` and the recovery wrapper must agree+ * about which binary is running, or a machine where one of them finds magmux+ * and the other does not produces a feature that works in one surface and is+ * silently absent from the other.+ */++import { execSync } from "node:child_process";+import { existsSync } from "node:fs";+import { dirname, join } from "node:path";+import { fileURLToPath } from "node:url";++/**+ * Find the magmux binary, or null.+ *+ * Priority:+ * 1. Bundled (`native/magmux-<platform>-<arch>`)+ * 2. The platform-specific npm package (npm installs only the matching one)+ * 3. `magmux` in PATH (e.g. Homebrew)+ */+export function findMagmuxBinaryOrNull(): string | null {+ const thisFile = fileURLToPath(import.meta.url);+ const thisDir = dirname(thisFile);+ // `launcher/` is one level below `src/`, which is itself below the package+ // root — so two levels up from here is where `native/` sits in a build.+ const pkgRoots = [join(thisDir, ".."), join(thisDir, "..", "..")];+ const platform = process.platform;+ const arch = process.arch;++ for (const pkgRoot of pkgRoots) {+ const bundled = join(pkgRoot, "native", `magmux-${platform}-${arch}`);+ if (existsSync(bundled)) return bundled;+ }++ try {+ const pkgName = `@claudish/magmux-${platform}-${arch}`;+ let searchDir = pkgRoots[0] as string;+ for (let i = 0; i < 5; i++) {+ const candidate = join(searchDir, "node_modules", pkgName, "bin", "magmux");+ if (existsSync(candidate)) return candidate;+ const parent = dirname(searchDir);+ if (parent === searchDir) break;+ searchDir = parent;+ }+ } catch {+ /* not installed */+ }++ try {+ const result = execSync("which magmux", { encoding: "utf-8" }).trim();+ if (result) return result;+ } catch {+ /* not in PATH */+ }++ return null;+}++/**+ * The same lookup, but fatal.+ *+ * `team --grid` cannot run at all without magmux, so it wants the throw. The+ * recovery wrapper wants the null: a missing multiplexer there means the+ * session launches exactly as it did before this feature existed, which is the+ * correct degradation and not an error.+ */+export function findMagmuxBinary(): string {+ const found = findMagmuxBinaryOrNull();+ if (found) return found;+ throw new Error("magmux not found. Install it:\n brew install MadAppGang/tap/magmux");+}diff --git a/packages/cli/src/launcher/magmux-wrapper.test.ts b/packages/cli/src/launcher/magmux-wrapper.test.tsnew file mode 100644index 00000000..a7e05d3b--- /dev/null+++ b/packages/cli/src/launcher/magmux-wrapper.test.ts@@ -0,0 +1,240 @@+/**+ * The launch wrapper.+ *+ * Two of these assertions are about SECURITY (only deltas reach the disk, and+ * the file is 0600) and two are about a contract that fails silently if it+ * breaks (the exports come after the login shell, and the strip list is+ * applied). The rest pin the gate: an implementation that wrapped `-p`, a+ * non-TTY, or a machine without magmux would break surfaces that have nothing+ * to do with this feature.+ */++import { afterEach, describe, expect, test } from "bun:test";+import { mkdtempSync, readFileSync, rmSync, statSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";+import {+ STRIPPED_CHILD_VARS,+ buildLauncherScript,+ envDeltas,+ planMagmuxWrap,+} from "./magmux-wrapper.js";++const roots: string[] = [];+function scratch(): string {+ const d = mkdtempSync(join(tmpdir(), "claudish-wrap-test-"));+ roots.push(d);+ return d;+}++afterEach(() => {+ for (const d of roots.splice(0)) rmSync(d, { recursive: true, force: true });+});++const BASE = {+ claudeBinary: "/usr/local/bin/claude",+ claudeArgs: ["--dangerously-skip-permissions"],+ interactive: true,+ stdoutIsTty: true,+ magmuxBinary: "/opt/homebrew/bin/magmux",+ pid: 4242,+};++describe("envDeltas", () => {+ test("carries ONLY what claudish changed — never the inherited environment", () => {+ // The difference between a delta and a dump is whether the user's exported+ // secrets end up in a file on disk.+ const parent = { PATH: "/usr/bin", HOME: "/Users/jack", OPENAI_API_KEY: "sk-secret" };+ const child = { ...parent, ANTHROPIC_BASE_URL: "http://127.0.0.1:8787" };+ const { set, unset } = envDeltas(parent, child);+ expect(set).toEqual([["ANTHROPIC_BASE_URL", "http://127.0.0.1:8787"]]);+ expect(set.map(([k]) => k)).not.toContain("OPENAI_API_KEY");+ expect(set.map(([k]) => k)).not.toContain("PATH");+ expect(unset).toEqual([...STRIPPED_CHILD_VARS]);+ });++ test("a CHANGED value is a delta; an identical one is not", () => {+ const parent = { A: "1", B: "2" };+ const child = { A: "1", B: "changed" };+ expect(envDeltas(parent, child).set).toEqual([["B", "changed"]]);+ });++ test("a variable claudish deleted is unset, not silently inherited", () => {+ // Monitor mode deletes ANTHROPIC_API_KEY so Claude Code uses its own+ // credentials. If the login shell re-inherited it, the session would bill+ // the API instead of the subscription.+ const parent = { ANTHROPIC_API_KEY: "sk-real" };+ const child = {};+ expect(envDeltas(parent, child).unset).toContain("ANTHROPIC_API_KEY");+ });++ test("CLAUDECODE and CLAUDE_CODE_CHILD_SESSION are always unset, never exported", () => {+ // Inherited, the pair turns transcript saving off and degrades magmux's own+ // controller to terminal-idle heuristics. `claude-runner.ts` spreads+ // `...process.env` and strips only the first.+ const parent = { CLAUDECODE: "1", CLAUDE_CODE_CHILD_SESSION: "1" };+ const child = { CLAUDECODE: "1", CLAUDE_CODE_CHILD_SESSION: "1", X: "y" };+ const { set, unset } = envDeltas(parent, child);+ expect(set.map(([k]) => k)).not.toContain("CLAUDECODE");+ expect(set.map(([k]) => k)).not.toContain("CLAUDE_CODE_CHILD_SESSION");+ expect(unset).toContain("CLAUDECODE");+ expect(unset).toContain("CLAUDE_CODE_CHILD_SESSION");+ });+});++describe("buildLauncherScript", () => {+ test("exports come AFTER the profile and immediately before exec — that is the point", () => {+ // RISK-5, measured rather than assumed: a live `list` reply shows magmux+ // runs a pane command as `/bin/zsh -l -c …`. A user's `.zprofile` therefore+ // runs between claudish and Claude Code, and claudish's whole contract with+ // Claude Code — starting with ANTHROPIC_BASE_URL — travels in the+ // environment. Re-asserting after the profile is what survives it.+ const script = buildLauncherScript("/usr/local/bin/claude", ["-i"], {+ set: [["ANTHROPIC_BASE_URL", "http://127.0.0.1:8787"]],+ unset: ["CLAUDECODE"],+ });+ const lines = script.trim().split("\n");+ const exportAt = lines.findIndex((l) => l.startsWith("export ANTHROPIC_BASE_URL="));+ const execAt = lines.findIndex((l) => l.startsWith("exec "));+ expect(exportAt).toBeGreaterThan(-1);+ expect(execAt).toBe(lines.length - 1);+ expect(exportAt).toBeLessThan(execAt);+ });++ test("values and arguments are single-quoted, so a space or a quote cannot escape", () => {+ const script = buildLauncherScript("/path with space/claude", ["it's", "--flag"], {+ set: [["WEIRD", "a'b c"]],+ unset: [],+ });+ expect(script).toContain(`export WEIRD='a'\\''b c'`);+ expect(script).toContain(`exec '/path with space/claude' 'it'\\''s' '--flag'`);+ });+});++describe("planMagmuxWrap", () => {+ test("returns null for a non-interactive run — `-p` and `--stdin` are untouched", () => {+ expect(+ planMagmuxWrap({ ...BASE, interactive: false, childEnv: {}, tmpRoot: scratch() })+ ).toBeNull();+ });++ test("returns null when stdout is not a TTY", () => {+ expect(+ planMagmuxWrap({ ...BASE, stdoutIsTty: false, childEnv: {}, tmpRoot: scratch() })+ ).toBeNull();+ });++ test("returns null INSIDE an existing magmux — a grid pane must not nest one", () => {+ // `team --grid --mode interactive` runs one `claudish -i` per pane, each+ // interactive with a real TTY. Without this, every slot would wrap itself+ // in a second magmux inside the pane it is already running in. magmux+ // exports MAGMUX_SOCK to its children precisely so a child can tell.+ expect(+ planMagmuxWrap({+ ...BASE,+ parentEnv: { MAGMUX_SOCK: "/tmp/magmux-someone-else.sock" },+ childEnv: {},+ tmpRoot: scratch(),+ })+ ).toBeNull();+ });++ test("returns null when magmux is missing — the session launches exactly as before", () => {+ // The correct degradation, and it needs no version check: without a+ // multiplexer nothing can open a pane, so no lease is ever granted and the+ // feature is simply absent.+ expect(+ planMagmuxWrap({ ...BASE, magmuxBinary: null, childEnv: {}, tmpRoot: scratch() })+ ).toBeNull();+ });++ test("the control socket path is DERIVED from --id, so it is known before magmux starts", () => {+ // claudish is magmux's parent and `MAGMUX_SOCK` is exported downward only,+ // so a pid-derived socket could never be discovered by the parent. `--id`+ // is what makes the recovery UI reachable at all.+ const plan = planMagmuxWrap({ ...BASE, childEnv: {}, tmpRoot: scratch() });+ expect(plan).not.toBeNull();+ expect(plan?.controlSocket).toBe("/tmp/magmux-claudish-4242.sock");+ expect(plan?.args).toContain("--id");+ expect(plan?.args[plan.args.indexOf("--id") + 1]).toBe("claudish-4242");+ });++ test("passes --no-idle-done, -w and --no-status", () => {+ const plan = planMagmuxWrap({ ...BASE, childEnv: {}, tmpRoot: scratch() });+ // `--no-idle-done` is the fix for the `-w` trap: an idle pane is a resting+ // session, not a finished one, and with the flag `-w` waits for the process+ // to exit rather than for a turn to end.+ expect(plan?.args).toContain("--no-idle-done");+ expect(plan?.args).toContain("-w");+ expect(plan?.args).toContain("--no-status");+ });++ test("the pane command SOURCES the generated script", () => {+ const plan = planMagmuxWrap({ ...BASE, childEnv: {}, tmpRoot: scratch() });+ const cmd = plan?.args[plan.args.length - 1] as string;+ expect(cmd.startsWith(". '")).toBe(true);+ expect(cmd).toContain(plan?.scriptPath as string);+ });++ test("the script is 0600 inside a 0700 directory and carries only the deltas", () => {+ const root = scratch();+ const plan = planMagmuxWrap({+ ...BASE,+ parentEnv: { PATH: "/usr/bin", OPENAI_API_KEY: "sk-inherited-secret" },+ childEnv: {+ PATH: "/usr/bin",+ OPENAI_API_KEY: "sk-inherited-secret",+ ANTHROPIC_BASE_URL: "http://127.0.0.1:8787",+ },+ tmpRoot: root,+ });+ const script = readFileSync(plan?.scriptPath as string, "utf-8");+ expect(script).toContain("export ANTHROPIC_BASE_URL='http://127.0.0.1:8787'");+ // The inherited secret is NOT written to disk. This is the assertion that+ // makes "only the deltas" a property rather than a comment.+ expect(script).not.toContain("sk-inherited-secret");+ expect(script).not.toContain("PATH=");+ expect(statSync(plan?.scriptPath as string).mode & 0o777).toBe(0o600);+ expect(statSync(join(root, "claudish-launch-4242")).mode & 0o777).toBe(0o700);+ });++ test("MAGMUX_SCROLLBACK defaults to 10000 but never overrides the user's own", () => {+ // RISK-4: the pane replaces the emulator's native scrollback with magmux's+ // ring. A bigger ring does not give selection back, but it stops the+ // session's history from being the thing that is lost.+ const a = planMagmuxWrap({ ...BASE, childEnv: {}, tmpRoot: scratch() });+ expect(a?.env.MAGMUX_SCROLLBACK).toBe("10000");+ const b = planMagmuxWrap({+ ...BASE,+ childEnv: { MAGMUX_SCROLLBACK: "500" },+ tmpRoot: scratch(),+ });+ expect(b?.env.MAGMUX_SCROLLBACK).toBe("500");+ });++ test("the magmux process itself never inherits the stripped variables", () => {+ const plan = planMagmuxWrap({+ ...BASE,+ childEnv: { CLAUDECODE: "1", CLAUDE_CODE_CHILD_SESSION: "1", KEEP: "yes" },+ tmpRoot: scratch(),+ });+ expect(plan?.env.CLAUDECODE).toBeUndefined();+ expect(plan?.env.CLAUDE_CODE_CHILD_SESSION).toBeUndefined();+ expect(plan?.env.KEEP).toBe("yes");+ });++ test("cleanup removes the script and its directory", () => {+ const root = scratch();+ const plan = planMagmuxWrap({ ...BASE, childEnv: {}, tmpRoot: root });+ expect(statSync(plan?.scriptPath as string).isFile()).toBe(true);+ plan?.cleanup();+ expect(() => statSync(plan?.scriptPath as string)).toThrow();+ });++ test("the pane exit code starts unknown — magmux's own status is not the child's", () => {+ // Measured: pane 0 exited 42 while magmux stayed alive waiting for a second+ // pane, so the exit code has to come off the control socket's `exit` event.+ const plan = planMagmuxWrap({ ...BASE, childEnv: {}, tmpRoot: scratch() });+ expect(plan?.paneExitCode()).toBeNull();+ });+});diff --git a/packages/cli/src/launcher/magmux-wrapper.ts b/packages/cli/src/launcher/magmux-wrapper.tsnew file mode 100644index 00000000..a1c6da55--- /dev/null+++ b/packages/cli/src/launcher/magmux-wrapper.ts@@ -0,0 +1,302 @@+/**+ * Launch Claude Code inside a magmux pane, so claudish owns a surface it can+ * draw a recovery banner on.+ *+ * There is no other surface. The status line repaints on Claude Code's+ * schedule and has no input channel; stderr is firewalled precisely because a+ * stray write corrupted the TUI; an OSC-9 notification cannot render a+ * countdown; a second terminal window is platform-specific and hostile; and+ * allocating our own PTY is writing a multiplexer. A pane is the cheapest thing+ * claudish can own.+ *+ * APPLIES ONLY WHEN `interactive === true` AND stdout is a TTY AND the binary+ * resolves. `-p`, `--stdin`, `serve`, the MCP server and the `team --grid`+ * panes are untouched, by construction rather than by a flag.+ *+ * ── RISK-5, and why the launcher script exists ──────────────────────────────+ * magmux runs a pane command through the LOGIN SHELL. Measured, verbatim from+ * a live `list` reply: `"cmd": "/bin/zsh -l -c bash -c '…'"`. So a user's+ * `.zprofile` runs between claudish and Claude Code, and claudish's entire+ * contract with Claude Code travels in the environment — `ANTHROPIC_BASE_URL`+ * is the whole proxy. A profile that sets any of those variables would silently+ * take the session off the proxy.+ *+ * The mitigation is a generated script that re-exports ONLY claudish's own+ * DELTAS immediately before `exec`. Only the deltas: writing the full+ * environment to disk would put every secret the user has exported into a file.+ *+ * ── C-16, and why the exit code comes off the socket ────────────────────────+ * magmux's own exit status does not carry the pane's — measured: pane 0 exited+ * 42 while magmux stayed alive waiting for a second pane. It does announce the+ * death on the control socket (`{"type":"exit","pane":0,"exitCode":42}`), so+ * that event is the authority for the child's exit code, and it is also what+ * lets the recovery pane be closed the instant Claude Code is gone rather than+ * after its linger.+ */++import type { ChildProcess } from "node:child_process";+import { chmodSync, existsSync, mkdirSync, rmSync, writeFileSync } from "node:fs";+import { type Socket, connect as netConnect } from "node:net";+import { join } from "node:path";+import { findMagmuxBinaryOrNull } from "./magmux-binary.js";++/**+ * Variables that must NOT reach the child.+ *+ * `claude-runner.ts` spreads `...process.env` and deletes `CLAUDECODE` but not+ * `CLAUDE_CODE_CHILD_SESSION`; inherited, the pair turns transcript saving off+ * and degrades magmux's own controller to terminal-idle heuristics. Stripping+ * them here covers both the magmux process and the login shell, which is a+ * second place they could have re-entered from.+ */+export const STRIPPED_CHILD_VARS = ["CLAUDECODE", "CLAUDE_CODE_CHILD_SESSION"] as const;++export interface MagmuxWrapInput {+ claudeBinary: string;+ claudeArgs: string[];+ /** The environment `claude-runner` built for Claude Code. */+ childEnv: Record<string, string>;+ /** What the login shell will start from. Defaults to `process.env`. */+ parentEnv?: NodeJS.ProcessEnv;+ cwd?: string;+ interactive: boolean;+ stdoutIsTty: boolean;+ /** Overridable so a test does not need magmux installed. */+ magmuxBinary?: string | null;+ /** Overridable so a test does not write into the real temp directory. */+ tmpRoot?: string;+ pid?: number;+}++export interface MagmuxWrapPlan {+ command: string;+ args: string[];+ env: Record<string, string>;+ /** `/tmp/magmux-claudish-<pid>.sock` — known BEFORE magmux starts. */+ controlSocket: string;+ scriptPath: string;+ /** Start watching the control socket. Call once, after spawning. */+ watch(proc: ChildProcess): void;+ /** Claude Code's own exit code, once magmux has announced it. */+ paneExitCode(): number | null;+ /** Called when pane 0 dies, before magmux itself exits. */+ onClaudeExit(fn: (exitCode: number | null) => void): void;+ cleanup(): void;+}++/**+ * What claudish added to, changed in, or removed from the environment.+ *+ * The delta — not the whole environment — is what the launcher script carries,+ * and the difference is whether a user's API keys end up in a file on disk.+ */+export function envDeltas(+ parentEnv: NodeJS.ProcessEnv,+ childEnv: Record<string, string>+): { set: Array<[string, string]>; unset: string[] } {+ const set: Array<[string, string]> = [];+ const unset: string[] = [];+ for (const [k, v] of Object.entries(childEnv)) {+ if (STRIPPED_CHILD_VARS.includes(k as (typeof STRIPPED_CHILD_VARS)[number])) continue;+ if (parentEnv[k] !== v) set.push([k, v]);+ }+ for (const k of Object.keys(parentEnv)) {+ if (!(k in childEnv)) unset.push(k);+ }+ for (const k of STRIPPED_CHILD_VARS) {+ if (!unset.includes(k)) unset.push(k);+ }+ return { set, unset };+}++/** POSIX single-quoting. The only escaping a `'…'` string ever needs. */+function shq(s: string): string {+ return `'${s.replace(/'/g, "'\\''")}'`;+}++/**+ * The script magmux's login shell sources, then `exec`s.+ *+ * Pure, so the exact bytes that will land on disk can be asserted in a test —+ * which matters because the thing being asserted is a security property (only+ * deltas) and a correctness property (the exports come AFTER the profile).+ */+export function buildLauncherScript(+ claudeBinary: string,+ claudeArgs: string[],+ deltas: { set: Array<[string, string]>; unset: string[] }+): string {+ const lines = [+ "#!/bin/sh",+ "# Generated by claudish. Re-asserts claudish's OWN environment deltas after",+ "# the login shell has run the user's profile, then hands the terminal to",+ "# Claude Code. Only deltas are written here: the full environment would put",+ "# every exported secret into a file.",+ ];+ for (const k of deltas.unset) lines.push(`unset ${k}`);+ for (const [k, v] of deltas.set) lines.push(`export ${k}=${shq(v)}`);+ lines.push(`exec ${shq(claudeBinary)} ${claudeArgs.map(shq).join(" ")}`);+ return `${lines.join("\n")}\n`;+}++/**+ * Build the plan, or null when this launch must not be wrapped.+ *+ * Returns null — never throws — for every "not applicable" case, because a+ * missing multiplexer means the session should launch exactly as it did before+ * this feature existed.+ */+export function planMagmuxWrap(input: MagmuxWrapInput): MagmuxWrapPlan | null {+ if (!input.interactive || !input.stdoutIsTty) return null;+ // ALREADY INSIDE A MULTIPLEXER — do not nest one.+ //+ // `team --grid --mode interactive` launches one `claudish --model X -i` per+ // pane, and each of those is interactive with a real TTY, so without this it+ // would wrap ITSELF in a second magmux inside the grid pane it is already+ // running in — N nested multiplexers, each with its own control socket and+ // its own idea of the layout. magmux exports `MAGMUX_SOCK` to its children+ // precisely so a child can tell, and a process that finds it does not need a+ // new magmux: it already has one it can ask for a pane. The recovery UI is+ // installed against that AMBIENT socket instead, and the cross-process+ // `O_EXCL` lock in `magmux-ui.ts` is what keeps the grid to ONE banner rather+ // than N.+ if ((input.parentEnv ?? process.env).MAGMUX_SOCK) return null;+ const magmux = input.magmuxBinary === undefined ? findMagmuxBinaryOrNull() : input.magmuxBinary;+ if (!magmux) return null;++ const pid = input.pid ?? process.pid;+ const id = `claudish-${pid}`;+ const controlSocket = `/tmp/magmux-${id}.sock`;++ // 0700 directory, 0600 file. The script carries claudish's deltas, which on+ // the proxy path include the key Claude Code authenticates to us with.+ const tmpRoot = input.tmpRoot ?? "/tmp";+ const scriptDir = join(tmpRoot, `claudish-launch-${pid}`);+ mkdirSync(scriptDir, { mode: 0o700, recursive: true });+ // Named for Claude Code on purpose: magmux attaches its controller by+ // spotting `claude`/`claudish` in the pane's command.+ const scriptPath = join(scriptDir, "claude-launch.sh");++ const parentEnv = input.parentEnv ?? process.env;+ const deltas = envDeltas(parentEnv, input.childEnv);+ writeFileSync(scriptPath, buildLauncherScript(input.claudeBinary, input.claudeArgs, deltas), {+ mode: 0o600,+ });+ try {+ chmodSync(scriptPath, 0o600);+ } catch {+ /* the 0700 directory is the real gate */+ }++ const env: Record<string, string> = { ...input.childEnv };+ for (const k of STRIPPED_CHILD_VARS) delete env[k];+ // RISK-4: the pane replaces the emulator's native scrollback with magmux's+ // ring, which is the single most user-visible cost of wrapping by default.+ // A bigger ring does not give the emulator's selection back, but it does stop+ // the session's history from being the thing that is lost.+ if (!env.MAGMUX_SCROLLBACK) env.MAGMUX_SCROLLBACK = "10000";++ const args = [+ "--id",+ id,+ // An idle pane is a resting session, not a finished one — and with this+ // flag `-w` waits for the process to exit rather than for a turn to end,+ // which is the exact trap `headless-vs-interactive.md` records.+ "--no-idle-done",+ "-w",+ "--no-status",+ "-e",+ // Sourced, not executed: `exec` then replaces the login shell itself, so+ // the pane's process IS Claude Code rather than a shell holding it.+ `. ${shq(scriptPath)}`,+ ];++ let paneExit: number | null = null;+ const exitListeners: Array<(code: number | null) => void> = [];+ let control: Socket | null = null;++ const cleanup = () => {+ try {+ control?.destroy();+ } catch {+ /* already gone */+ }+ control = null;+ try {+ rmSync(scriptDir, { recursive: true, force: true });+ } catch {+ /* already gone */+ }+ try {+ // magmux's startup reaper only removes PID-named sockets whose owner it+ // can prove is dead, so a `--id` socket is never reaped for us.+ if (existsSync(controlSocket)) rmSync(controlSocket, { force: true });+ } catch {+ /* already gone */+ }+ };++ return {+ command: magmux,+ args,+ env,+ controlSocket,+ scriptPath,+ paneExitCode: () => paneExit,+ onClaudeExit(fn) {+ exitListeners.push(fn);+ },+ watch(proc: ChildProcess) {+ let stopped = false;+ const attempt = async () => {+ for (let i = 0; i < 60 && !stopped; i++) {+ if (existsSync(controlSocket)) {+ try {+ const s = await new Promise<Socket>((resolve, reject) => {+ const sock = netConnect(controlSocket);+ sock.once("connect", () => resolve(sock));+ sock.once("error", reject);+ });+ control = s;+ let buf = "";+ s.on("data", (chunk: Buffer) => {+ buf += chunk.toString("utf-8");+ let nl = buf.indexOf("\n");+ while (nl >= 0) {+ const line = buf.slice(0, nl).trim();+ buf = buf.slice(nl + 1);+ nl = buf.indexOf("\n");+ if (!line) continue;+ try {+ const evt = JSON.parse(line) as Record<string, unknown>;+ // Pane 0 is Claude Code: `-e` panes get ids 0..N-1 in+ // argument order and magmux's own panes are appended after.+ if (evt.type === "exit" && evt.pane === 0) {+ paneExit = typeof evt.exitCode === "number" ? evt.exitCode : null;+ for (const fn of exitListeners) fn(paneExit);+ }+ } catch {+ /* not ours */+ }+ }+ });+ s.on("error", () => {+ control = null;+ });+ return;+ } catch {+ /* magmux binds the socket a moment after the file appears */+ }+ }+ await new Promise((r) => setTimeout(r, 50));+ }+ };+ proc.once("exit", () => {+ stopped = true;+ cleanup();+ });+ void attempt();+ },+ cleanup,+ };+}diff --git a/packages/cli/src/recovery/coordinator.ts b/packages/cli/src/recovery/coordinator.tsindex 5f384bab..1bec8921 100644--- a/packages/cli/src/recovery/coordinator.ts+++ b/packages/cli/src/recovery/coordinator.ts@@ -19,17 +19,23 @@* A waiter past its own deadline answers for itself and leaves.* - the LADDER belongs to the episode. It ends when the last waiter leaves* or an attempt succeeds — never because one waiter ran out of time.- * - the PANE LEASE belongs to a renderer painting this episode. Nothing in- * this file grants one yet; see `uiLeaseValid`.+ * - the PANE LEASE belongs to a renderer painting this episode. It is granted+ * and revoked in `magmux-ui.ts`, which registers itself here; this file+ * only ever ASKS, via `uiLeaseValid`, and never caches the answer.** Letting one field mean all three is what made a four-waiter episode show* `waiters: 4` while exactly one of them was still being retried.*/import { randomUUID } from "node:crypto";-import type { ConnectionErrorKind } from "../handlers/shared/connection-error.js";+import { type ConnectionErrorKind, isLoopback } from "../handlers/shared/connection-error.js";import { log } from "../logger.js";import { recoveryClock } from "./clock.js";+import {+ RECOVERY_PROTOCOL_VERSION,+ type RecoveryEpisodeFrame,+ type RecoveryFrameState,+} from "./types.js";export type RecoveryState = "attempting" | "waiting" | "recovered" | "handoff" | "abandoned";@@ -95,7 +101,11 @@ interface Episode {state: RecoveryState;startedAtPerf: number;startedAtMs: number;- /** Set when a pane is first ASKED for. Null for the whole of this phase. */+ /**+ * Set when a pane is first ASKED for — including when the answer is an+ * immediate no, because asking and being told no is still asking. Null on+ * every surface that cannot open one at all.+ */paneRequestedAtPerf: number | null;ladderIndex: number;attempts: number;@@ -151,18 +161,134 @@ export function describeEpisode(episodeId: string): Readonly<Record<string, unkn};}+/**+ * What the UI manager (`magmux-ui.ts`) plugs into the coordinator.+ *+ * A REGISTRATION SEAM RATHER THAN AN IMPORT, and not for testability. The UI+ * manager reads episode state (to build a frame) and drives the ladder (`[r]`,+ * `[q]`), so it must import this module; a direct import back the other way is+ * a cycle. More importantly it keeps the dependency HONEST in the other+ * direction: nothing is installed in `-p`, in `serve`, in the MCP server or in+ * the test suite, so those paths cannot open a pane, cannot hold a lease, and+ * behave exactly as they did before this phase — by construction rather than by+ * a flag someone has to remember to check.+ */+export interface RecoveryUiHooks {+ /** A new episode exists. The manager decides whether to ask for a pane. */+ onEpisodeOpened(episodeId: string): void;+ /** An episode reached a terminal state. */+ onEpisodeClosed(episodeId: string, outcome: RecoveryOutcome): void;+ /** Is a renderer painting this episode RIGHT NOW? */+ leaseValid(episodeId: string): boolean;+}++let uiHooks: RecoveryUiHooks | null = null;++/** Install (or, with null, remove) the recovery UI. Process-scoped. */+export function registerRecoveryUi(hooks: RecoveryUiHooks | null): void {+ uiHooks = hooks;+}+/*** Is a renderer currently painting this episode?*- * SEAM, deliberately inert in this phase. There is no socket, no pane and no- * heartbeat yet, so the honest answer is false and the exhaustion arm keeps- * answering today's 400. It is a function rather than a field because the- * answer is LIVE — a lease read once and cached into a boolean is stale by the- * time the status is chosen, which is the whole reason the previous design's- * `uiAttached` latch was wrong.+ * A FUNCTION AND NOT A FIELD, because the answer is LIVE. A lease read once and+ * cached into a boolean is already stale by the time a status is chosen from+ * it — which is precisely how the superseded `uiAttached` latch could report a+ * banner that had been killed, closed or frozen minutes earlier.+ *+ * FALSE whenever no UI is installed, which is every non-interactive surface.*/-export function uiLeaseValid(_episodeId: string): boolean {- return false;+export function uiLeaseValid(episodeId: string): boolean {+ if (!uiHooks) return false;+ try {+ return uiHooks.leaseValid(episodeId);+ } catch {+ // A throwing lease probe must read as "no surface", never as "yes". The+ // asymmetry is the point: false costs a legible inline error, true costs+ // the turn.+ return false;+ }+}++/** Record that a pane was ASKED for. Asking and being told no is still asking. */+export function markPaneRequested(episodeId: string): void {+ const ep = byId.get(episodeId);+ if (!ep || ep.paneRequestedAtPerf !== null) return;+ ep.paneRequestedAtPerf = recoveryClock().now();+}++/** Live episodes, most waiters first, tie-broken by earliest start. */+function liveEpisodes(): Episode[] {+ const live = [...episodes.values()].filter(+ (e) => e.state === "attempting" || e.state === "waiting" || e.state === "handoff"+ );+ live.sort((a, b) => b.waiters.size - a.waiters.size || a.startedAtMs - b.startedAtMs);+ return live;+}++/**+ * The episode a renderer should paint, as a wire frame — or null.+ *+ * Concurrent episodes are ORDINARY: the main loop's provider and the small+ * title model's provider can be unreachable at the same moment. Which one wins+ * is not load-bearing, because the lease is per-episode — a non-rendered+ * episode simply holds no lease, and at exhaustion answers the inline error,+ * which is exactly what "a retryable status is permissible only while the+ * reason is legible" demands.+ *+ * THE UNIT CONVERSION HAPPENS HERE AND NOWHERE ELSE. Intervals inside this+ * module are process ms; everything on the wire is epoch ms, because the pane+ * is a different process with a different `performance.now()` origin and would+ * render a process-ms instant as an arbitrary number with no error anywhere.+ */+export function renderableEpisodeFrame(): RecoveryEpisodeFrame | null {+ const live = liveEpisodes();+ const ep = live[0];+ if (!ep) return null;+ const clock = recoveryClock();+ const perfNow = clock.now();+ const epochNow = Date.now();+ const nextAttemptAtMs =+ ep.nextAttemptAtPerf === null ? null : epochNow + (ep.nextAttemptAtPerf - perfNow);+ return {+ v: RECOVERY_PROTOCOL_VERSION,+ type: "episode",+ episodeId: ep.episodeId,+ state: ep.state as RecoveryFrameState,+ tier: ep.maxTier,+ providerDisplayName: ep.providerDisplayName,+ host: ep.host,+ endpoint: ep.endpoint,+ kind: ep.kind,+ code: ep.code,+ loopback: ep.loopback,+ reason: ep.reason,+ attempts: ep.attempts,+ clientRetries: ep.clientRetries,+ startedAtMs: ep.startedAtMs,+ nextAttemptAtMs,+ lastOutcome: ep.lastOutcome,+ waiters: ep.waiters.size,+ otherEpisodes: live.length - 1,+ };+}++/** How many attempts this episode has made. For the manual-retry log line. */+export function attemptsSoFar(episodeId: string): number {+ return byId.get(episodeId)?.attempts ?? 0;+}++/** Is this episode still live? Used by the UI manager's lease check. */+export function episodeIsLive(episodeId: string): boolean {+ const ep = byId.get(episodeId);+ if (!ep) return false;+ return ep.state === "attempting" || ep.state === "waiting" || ep.state === "handoff";+}++/** `[q] give up` — the user said stop, so every live episode stops. */+export function giveUpAll(): void {+ for (const ep of liveEpisodes()) giveUp(ep.episodeId);}export interface EpisodeHandle {@@ -247,7 +373,11 @@ export function joinEpisode(seed: EpisodeSeed): EpisodeHandle {endpoint: seed.endpoint,host,kind: seed.kind,- loopback: false,+ // Decided ONCE, at creation, from the endpoint — never from the error+ // code. On macOS an unrouted REMOTE address reports `ECONNREFUSED` after+ // 75 s, the same code a loopback port refuses in 5 ms, so the code cannot+ // tell them apart and only the address can.+ loopback: isLoopback(seed.endpoint),code: seed.code,reason: seed.reason,state: "attempting",@@ -279,6 +409,18 @@ export function joinEpisode(seed: EpisodeSeed): EpisodeHandle {`[Recovery] episode ${ep.episodeId} opened for ${ep.providerDisplayName} at ${ep.endpoint} ` +`(${ep.kind}/${ep.code ?? "?"})`);+ // Ask for a surface. Nothing is installed unless this process was launched+ // through the magmux wrapper, so on every other surface this is a no-op —+ // which is why `-p`, `serve` and the suite are untouched.+ if (uiHooks) {+ try {+ uiHooks.onEpisodeOpened(ep.episodeId);+ } catch (err) {+ // A UI failure must never break recovery. The banner is what EARNS the+ // wait; it is not what performs it.+ log(`[Recovery] UI could not open for episode ${ep.episodeId}: ${String(err)}`);+ }+ }}const episode = ep;@@ -466,6 +608,13 @@ function closeEpisode(episode: Episode, outcome: RecoveryOutcome): void {`[Recovery] episode ${episode.episodeId} closed: ${outcome} after ` +`${Math.round(clock.now() - episode.startedAtPerf)}ms and ${episode.attempts} attempts`);+ if (uiHooks) {+ try {+ uiHooks.onEpisodeClosed(episode.episodeId, outcome);+ } catch {+ /* see onEpisodeOpened — the UI never breaks recovery */+ }+ }}/**@@ -482,12 +631,22 @@ export function tryNow(episodeId: string): void {for (const w of [...ep.waiters]) w.wake?.({ kind: "attempt" });}-/** Wake every waiter of every episode with `gave_up`. The `[q]` key, later. */+/**+ * `[q] give up` for ONE episode: every waiter answers today's inline error and+ * the episode is over.+ *+ * `gaveUp` latches before the close so a waiter that has not parked yet — one+ * between attempts at the instant the key was pressed — still reads the answer+ * rather than parking on a timer that no longer exists. The user said stop, so+ * stop, and say why inline: this is the missing half of an affordance the+ * superseded design labelled "give up" and gave no effect at all.+ */export function giveUp(episodeId: string): void {const ep = byId.get(episodeId);if (!ep) return;ep.gaveUp = true;clearAttemptTimer(ep);+ closeEpisode(ep, "gave_up");for (const w of [...ep.waiters]) w.wake?.({ kind: "gave_up" });}diff --git a/packages/cli/src/recovery/lease.test.ts b/packages/cli/src/recovery/lease.test.tsnew file mode 100644index 00000000..59b203b0--- /dev/null+++ b/packages/cli/src/recovery/lease.test.ts@@ -0,0 +1,252 @@+/**+ * The UI lease — the round-2 CRITICAL, and the reason this file exists at all.+ *+ * The lease is what will decide, one phase from now, whether an exhausted+ * episode may answer a RETRYABLE status instead of an inline error. The rule it+ * encodes is: *a retryable status is permissible exactly while claudish still+ * has a surface on which the reason is legible.*+ *+ * The superseded design renewed that lease from FRAME RECEIPTS, and frames only+ * ticked while the ladder was `waiting`. A connect against an unreachable host+ * takes 20–75 s (measured on this machine: `192.0.2.1` = 75 005 ms), during+ * which the proxy has nothing new to say — so the lease died ten seconds into+ * every slow attempt, and the exhaustion arm, which reads it immediately after+ * an attempt returns, saw it false for the ENTIRE failure class the feature was+ * built for, with a live painted banner on screen.+ *+ * And every test passed, because every fault in the suite was a ~1 ms loopback+ * refusal. These are the tests that can fail on it.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { EventEmitter } from "node:events";+import { resetRecoveryClock, setRecoveryClock } from "./clock.js";+import {+ __handleCommandForTests,+ __resetUiStateForTests,+ __setPaneOpenForTests,+ describeLease,+ uiLeaseValid,+} from "./magmux-ui.js";+import { runRecoveryPane } from "./pane-app.js";+import {+ type RecoverySocketServer,+ createRecoverySocketDir,+ recoverySocketPathIn,+ startRecoverySocketServer,+} from "./socket-server.js";+import { FakeClock } from "./test-helpers/fake-clock.js";+import {+ RECOVERY_PROTOCOL_VERSION,+ type RecoveryEpisodeFrame,+ UI_HEARTBEAT_MS,+ UI_LEASE_MS,+} from "./types.js";++const EPISODE: RecoveryEpisodeFrame = {+ v: RECOVERY_PROTOCOL_VERSION,+ type: "episode",+ episodeId: "ep-slow",+ // `attempting`, not `waiting`: this is the state the proxy sits in for the+ // whole of a 45-second connect, and the state in which the old design went+ // silent.+ state: "attempting",+ tier: 1,+ providerDisplayName: "Openai-codex",+ host: "chatgpt.com",+ endpoint: "https://chatgpt.com/backend-api/codex/responses",+ kind: "unreachable",+ code: "ConnectionClosed",+ loopback: false,+ reason: "Cannot reach Openai-codex at https://chatgpt.com/backend-api/codex/responses.",+ attempts: 4,+ clientRetries: 0,+ startedAtMs: 1_700_000_000_000,+ nextAttemptAtMs: null,+ lastOutcome: "ConnectionClosed after 45001ms",+ waiters: 1,+ otherEpisodes: 0,+};++let clock: FakeClock;++beforeEach(() => {+ clock = new FakeClock(0);+ setRecoveryClock(clock);+ __resetUiStateForTests();+});++afterEach(() => {+ __resetUiStateForTests();+ resetRecoveryClock();+});++const ack = (episodeId: string) =>+ __handleCommandForTests({+ v: RECOVERY_PROTOCOL_VERSION,+ type: "ack",+ episodeId,+ paintedAt: Date.now(),+ });++describe("the UI lease (unit, fake clock)", () => {+ test("SURVIVES an attempt far longer than UI_LEASE_MS while the renderer heartbeats", async () => {+ // THE round-2 CRITICAL, expressed as an assertion. 45 seconds of attempt —+ // 4.5 leases — with NOT ONE FRAME emitted, because the proxy has nothing to+ // say while a connect hangs. The renderer's own heartbeat is the only thing+ // renewing anything.+ __setPaneOpenForTests(true);+ ack("ep-slow");++ for (let elapsed = 0; elapsed < 45_000; elapsed += UI_HEARTBEAT_MS) {+ await clock.advance(UI_HEARTBEAT_MS);+ ack("ep-slow"); // the renderer, on ITS timer, not on a frame+ expect(uiLeaseValid("ep-slow")).toBe(true);+ }++ expect(clock.now()).toBeGreaterThanOrEqual(45_000);+ expect(uiLeaseValid("ep-slow")).toBe(true);+ });++ test("expires UI_LEASE_MS after the last heartbeat — a frozen renderer holds nothing", async () => {+ __setPaneOpenForTests(true);+ ack("ep-slow");+ await clock.advance(UI_LEASE_MS);+ expect(uiLeaseValid("ep-slow")).toBe(true); // exactly at the boundary+ await clock.advance(1);+ expect(uiLeaseValid("ep-slow")).toBe(false);+ });++ test("a forged client cannot grant itself a lease: paneOpen is required", () => {+ // The socket has no same-uid boundary and cannot have one. What is defended+ // is the CONSEQUENCE: the one forbidden state (a retryable status with no+ // banner) additionally requires that WE opened a pane and hold the reply,+ // which is a fact no client can cause.+ __setPaneOpenForTests(false);+ ack("ep-slow");+ expect(uiLeaseValid("ep-slow")).toBe(false);+ expect(describeLease("ep-slow").lastAckAgoMs).toBe(0); // it DID heartbeat…+ expect(describeLease("ep-slow").paneOpen).toBe(false); // …and it still gets nothing+ });++ test("the lease is per-EPISODE: heartbeating one does not lease another", () => {+ __setPaneOpenForTests(true);+ ack("ep-slow");+ expect(uiLeaseValid("ep-slow")).toBe(true);+ expect(uiLeaseValid("ep-other")).toBe(false);+ });++ test("switching the painted episode lets the old lease lapse with no extra protocol", async () => {+ __setPaneOpenForTests(true);+ ack("ep-slow");+ for (let i = 0; i < 12; i++) {+ await clock.advance(UI_HEARTBEAT_MS);+ ack("ep-other"); // the renderer moved on+ }+ expect(uiLeaseValid("ep-other")).toBe(true);+ expect(uiLeaseValid("ep-slow")).toBe(false);+ });++ test("`bye` revokes every lease, and no later heartbeat can resurrect one", () => {+ // `[q] give up` is the affordance the superseded design labelled+ // "give up" and gave no server-side effect at all — so a user could close+ // the banner while the machine kept retrying and eventually answered a+ // retryable status with nothing on screen. That is the design's OWN stated+ // worst case, reached through a key that says "give up".+ __setPaneOpenForTests(true);+ ack("ep-slow");+ ack("ep-other");+ expect(uiLeaseValid("ep-slow")).toBe(true);++ __handleCommandForTests({ v: RECOVERY_PROTOCOL_VERSION, type: "bye", reason: "user_quit" });++ expect(uiLeaseValid("ep-slow")).toBe(false);+ expect(uiLeaseValid("ep-other")).toBe(false);+ // The pane is genuinely gone, not merely un-leased…+ expect(describeLease("ep-slow").paneOpen).toBe(false);+ // …so a client that keeps heartbeating into the closed socket gets nothing.+ // Without this, `bye` would be undone by the very next tick of a renderer+ // that had not noticed it was closed.+ ack("ep-slow");+ expect(uiLeaseValid("ep-slow")).toBe(false);+ });+});++// ─── The same property, end to end, over a real socket ───────────────────────++class FakeStdin extends EventEmitter {+ rawMode = false;+ setRawMode(on: boolean) {+ this.rawMode = on;+ return this;+ }+ resume() {+ return this;+ }+ pause() {+ return this;+ }+}++class FakeStdout extends EventEmitter {+ columns = 100;+ chunks: string[] = [];+ write(s: string) {+ this.chunks.push(s);+ return true;+ }+}++describe("the UI lease (end to end, real socket, real renderer)", () => {+ test("the REAL pane heartbeats through total frame silence, and the lease holds", async () => {+ // One frame in the whole run — the replay — and then nothing, which is what+ // a hung connect looks like on the wire. If the renderer ever goes back to+ // acking per received frame, the ack count collapses to 1 and the lease is+ // dead by fake-t=10s. That is the defect this test exists to fail on.+ const path = recoverySocketPathIn(createRecoverySocketDir());+ let framesSent = 0;+ let acks = 0;++ let server: RecoverySocketServer | null = null;+ const stdin = new FakeStdin();+ const stdout = new FakeStdout();++ server = await startRecoverySocketServer({+ socketPath: path,+ snapshot: () => {+ framesSent++;+ return EPISODE;+ },+ onCommand: (cmd) => {+ if (cmd.type === "ack") acks++;+ __handleCommandForTests(cmd);+ },+ tickMs: 0, // the proxy says NOTHING after the replay+ });++ __setPaneOpenForTests(true);+ const paneDone = runRecoveryPane({+ socketPath: path,+ stdin: stdin as unknown as NodeJS.ReadStream,+ stdout: stdout as unknown as NodeJS.WriteStream,+ });++ // ~4 real heartbeat intervals, with the LEASE clock advancing 5 s each time+ // — so the episode is 20 s old in lease terms while the socket has carried+ // exactly one frame.+ for (let i = 0; i < 4; i++) {+ await new Promise((r) => setTimeout(r, UI_HEARTBEAT_MS + 120));+ await clock.advance(5_000);+ expect(uiLeaseValid("ep-slow")).toBe(true);+ }++ expect(framesSent).toBe(1);+ expect(acks).toBeGreaterThanOrEqual(3);+ expect(clock.now()).toBeGreaterThan(UI_LEASE_MS);+ expect(uiLeaseValid("ep-slow")).toBe(true);++ stdin.emit("data", Buffer.from("q"));+ await paneDone;+ await server.close();+ }, 15_000);+});diff --git a/packages/cli/src/recovery/magmux-ui.ts b/packages/cli/src/recovery/magmux-ui.tsnew file mode 100644index 00000000..f09bd8bd--- /dev/null+++ b/packages/cli/src/recovery/magmux-ui.ts@@ -0,0 +1,568 @@+/**+ * The recovery UI manager — the single owner of the socket, the pane, the+ * `paneOpen` fact, the post-`bye` suppression window, the cross-process lock+ * and the lease.+ *+ * OWNERSHIP IS PROCESS-SCOPED; ONLY EPISODES ARE EPISODE-SCOPED. There is ONE+ * pane and it multiplexes every episode, so "a pane exists" is a property of+ * this module and never of an episode. An earlier design hung `paneOpened` on+ * each episode, which meant the second concurrent episode could never be leased+ * — only the first one ever received an `open_pane` reply — and that closing+ * one episode tore down a socket a different episode's renderer was reading.+ * `releaseEpisodeUi` therefore emits one episode's `closed` frame and NOTHING+ * else; the socket and the pane come down only when no episode is live at all.+ *+ * THE LEASE IS THE WHOLE POINT OF THIS FILE. It is not a latch, not a config+ * read and not "a client connected":+ *+ * uiLeaseValid(id) ⇔ paneOpen (WE opened it, and hold the reply)+ * ∧ some client heartbeated `ack` naming THIS episode+ * within UI_LEASE_MS+ *+ * Both clauses are load-bearing. `paneOpen` is what a forged same-uid client+ * cannot manufacture, so it cannot talk the proxy into the one forbidden state+ * (a retryable status with no banner on screen). The heartbeat window is what+ * makes the lease self-clearing on EOF, on a killed pane, on a frozen renderer+ * and on `[q]`, with no cleanup path anyone has to remember.+ *+ * AND THE HEARTBEAT IS THE RENDERER'S, NOT OURS. The pane sends `ack` on its+ * own 1 Hz timer for as long as it is painting an episode, whether or not a+ * frame arrived. Renewing from frame receipts would tie the renderer's apparent+ * liveness to OUR emission cadence — and we have nothing to say for the 20–75 s+ * an unreachable connect takes (measured: `192.0.2.1` = 75 005 ms). The lease+ * would then read false at exhaustion for precisely the failure class this+ * feature exists for, with a live painted banner on screen, and every+ * loopback-only test would still pass.+ *+ * NOTHING HERE CHANGES A STATUS IN THIS PHASE. The lease is computed and+ * logged; the 503-vs-400 flip is the next one. Shipping the flip before the+ * banner exists recreates the buried-reason bug the feature is meant to kill.+ */++import { closeSync, existsSync, openSync, readFileSync, unlinkSync, writeSync } from "node:fs";+import { type Socket, connect as netConnect } from "node:net";+import { basename, join } from "node:path";+import { log } from "../logger.js";+import { recoveryClock } from "./clock.js";+import {+ type RecoveryOutcome,+ type RecoveryUiHooks,+ attemptsSoFar,+ episodeIsLive,+ giveUpAll,+ markPaneRequested,+ registerRecoveryUi,+ renderableEpisodeFrame,+ tryNow,+} from "./coordinator.js";+import {+ type RecoverySocketServer,+ cleanupRecoverySocket,+ createRecoverySocketDir,+ recoverySocketPathIn,+ startRecoverySocketServer,+} from "./socket-server.js";+import {+ PANE_LINGER_MS,+ RECOVERY_PROTOCOL_VERSION,+ type RecoveryCommand,+ UI_LEASE_MS,+} from "./types.js";++/** After `[q]`, do not re-open a pane for this long. */+export const UI_SUPPRESS_AFTER_BYE_MS = 60_000;++// ─── The control endpoint ────────────────────────────────────────────────────++let explicitControlSock: string | null = null;++/**+ * The magmux control socket, handed in by the launch wrapper.+ *+ * AN EXPLICIT INPUT, NOT AN ENV LOOKUP, and that is the only thing that makes+ * this work at all: claudish is magmux's PARENT, and `MAGMUX_SOCK` is exported+ * DOWNWARD, so the parent can never learn a pid-derived path. The wrapper+ * generates `--id claudish-<pid>`, which fixes the path before magmux starts,+ * and hands it here.+ */+export function setMagmuxControlSocket(sock: string | null): void {+ explicitControlSock = sock;+}++/**+ * Where can this process send `open_pane` right now?+ *+ * Order: the path the wrapper handed us, then an ambient `MAGMUX_SOCK` (the+ * already-inside-a-pane case, e.g. `team --grid --mode interactive`), then null.+ *+ * IT ANSWERS A LAUNCH-ORDER QUESTION, NOT A USER PREFERENCE, and only this+ * module may read it. Null means "this process cannot open a pane" — true in+ * `-p`, in `serve`, in the MCP server, in every unwrapped launch and in the+ * whole test suite. It must never gate whether a RETRY happens: keying the+ * retry ladder on it made recovery skip universally in every phase before the+ * wrapper existed, twice, and both times the same rule decided the tests.+ */+export function resolveMagmuxControl(): { sock: string } | null {+ if (explicitControlSock && existsSync(explicitControlSock)) return { sock: explicitControlSock };+ const ambient = process.env.MAGMUX_SOCK;+ if (ambient && existsSync(ambient)) return { sock: ambient };+ return null;+}++// ─── Manager state ───────────────────────────────────────────────────────────++interface ManagerState {+ server: RecoverySocketServer | null;+ socketPath: string | null;+ control: Socket | null;+ /** True only once magmux has REPLIED to our `open_pane`. */+ paneOpen: boolean;+ /** magmux's own index for our pane. Read from the reply — never assumed. */+ paneIndex: number | null;+ /** episodeId → PROCESS ms of the last heartbeat naming it. */+ leases: Map<string, number>;+ /** PROCESS ms before which no pane may be opened (post-`bye`). */+ suppressUntilPerf: number;+ /** Guards the async open against re-entry. */+ opening: boolean;+ lockPath: string | null;+ lingerTimer: unknown | null;+ nextControlId: number;+}++const state: ManagerState = {+ server: null,+ socketPath: null,+ control: null,+ paneOpen: false,+ paneIndex: null,+ leases: new Map(),+ suppressUntilPerf: 0,+ opening: false,+ lockPath: null,+ lingerTimer: null,+ nextControlId: 1,+};++let installed = false;++// ─── The lease ───────────────────────────────────────────────────────────────++/**+ * Is a renderer painting this episode right now?+ *+ * Read LIVE at the moment a status is chosen. Never cached: by the time a+ * caller destructures a boolean off a result object, the pane it describes may+ * have been killed.+ */+export function uiLeaseValid(episodeId: string): boolean {+ if (!state.paneOpen) return false;+ const last = state.leases.get(episodeId);+ if (last === undefined) return false;+ return recoveryClock().now() - last <= UI_LEASE_MS;+}++/** Diagnostics for the log and for tests. Never used to decide anything. */+export function describeLease(episodeId: string): {+ paneOpen: boolean;+ lastAckAgoMs: number | null;+ valid: boolean;+} {+ const last = state.leases.get(episodeId);+ return {+ paneOpen: state.paneOpen,+ lastAckAgoMs: last === undefined ? null : Math.round(recoveryClock().now() - last),+ valid: uiLeaseValid(episodeId),+ };+}++// ─── The magmux control connection ───────────────────────────────────────────++async function connectControl(sock: string): Promise<Socket | null> {+ try {+ return await new Promise<Socket>((resolve, reject) => {+ const s = netConnect(sock);+ s.once("connect", () => resolve(s));+ s.once("error", reject);+ });+ } catch {+ return null;+ }+}++/** Send one control message and wait for the `reply` carrying the same id. */+function request(+ socket: Socket,+ msg: Record<string, unknown>,+ timeoutMs = 3_000+): Promise<Record<string, unknown> | null> {+ const id = state.nextControlId++;+ return new Promise((resolve) => {+ let buf = "";+ let settled = false;+ const finish = (v: Record<string, unknown> | null) => {+ if (settled) return;+ settled = true;+ socket.removeListener("data", onData);+ clearTimeout(timer);+ resolve(v);+ };+ const onData = (chunk: Buffer) => {+ buf += chunk.toString("utf-8");+ let nl = buf.indexOf("\n");+ while (nl >= 0) {+ const line = buf.slice(0, nl).trim();+ buf = buf.slice(nl + 1);+ nl = buf.indexOf("\n");+ if (!line) continue;+ try {+ const evt = JSON.parse(line) as Record<string, unknown>;+ if (evt.type === "reply" && evt.id === id) finish(evt);+ } catch {+ /* magmux emits live events on the same stream; ignore what is not ours */+ }+ }+ };+ const timer = setTimeout(() => finish(null), timeoutMs);+ (timer as { unref?: () => void }).unref?.();+ socket.on("data", onData);+ try {+ socket.write(`${JSON.stringify({ ...msg, id })}\n`);+ } catch {+ finish(null);+ }+ });+}++// ─── The cross-process single-pane lock ──────────────────────────────────────++/**+ * ONE recovery pane per magmux, not N.+ *+ * `team --grid --mode interactive` launches one claudish per pane and magmux+ * exports the same `MAGMUX_SOCK` to all of them, so one outage hits N processes+ * at once and a merely process-level singleton would open N panes into one+ * grid. The losers still run the ladder — they retry and they recover — they+ * just never open a pane, hold no lease, and answer the inline error at+ * exhaustion. That is honest: one banner names one slot's provider, and every+ * slot still recovers.+ */+function acquirePaneLock(controlSock: string): string | null {+ const lockPath = join("/tmp", `claudish-recovery-${basename(controlSock)}.lock`);+ for (let attempt = 0; attempt < 2; attempt++) {+ try {+ const fd = openSync(lockPath, "wx", 0o600);+ writeSync(fd, String(process.pid));+ closeSync(fd);+ return lockPath;+ } catch {+ // Held — by a live peer, or by debris from one that was SIGKILLed.+ try {+ const owner = Number(readFileSync(lockPath, "utf-8").trim());+ if (Number.isFinite(owner) && owner > 0) {+ try {+ process.kill(owner, 0);+ return null; // a live peer owns the banner+ } catch {+ unlinkSync(lockPath); // the owner is gone; the lock is debris+ continue;+ }+ }+ unlinkSync(lockPath);+ } catch {+ return null;+ }+ }+ }+ return null;+}++function releasePaneLock(): void {+ if (!state.lockPath) return;+ try {+ if (existsSync(state.lockPath)) {+ const owner = Number(readFileSync(state.lockPath, "utf-8").trim());+ if (owner === process.pid) unlinkSync(state.lockPath);+ }+ } catch {+ /* best effort */+ }+ state.lockPath = null;+}++// ─── The pane command line ───────────────────────────────────────────────────++/**+ * How to run `claudish recovery-pane` from wherever this build lives.+ *+ * Built from `process.argv` rather than from the name `claudish`, because the+ * process that must start is THIS build: a global install mid-update, a `bun+ * run src/index.ts` in a worktree and an npm-installed bundle are three+ * different files, and a pane from a different build is the version-skew case+ * the protocol has to print a notice for. `claudish` on PATH stays as the+ * fallback for a packaging shape neither argv entry describes.+ */+export function recoveryPaneCommandLine(socketPath: string): string {+ const script = process.argv[1];+ const q = (s: string) => `'${s.replace(/'/g, "'\\''")}'`;+ if (script && (script.endsWith(".ts") || script.endsWith(".js") || script.endsWith(".cjs"))) {+ return `${q(process.execPath)} ${q(script)} recovery-pane --socket ${q(socketPath)}`;+ }+ return `claudish recovery-pane --socket ${q(socketPath)}`;+}++// ─── Opening and closing the pane ────────────────────────────────────────────++function onCommand(cmd: RecoveryCommand): void {+ switch (cmd.type) {+ case "hello":+ log(`[Recovery] pane connected (pid ${cmd.pid}, protocol ${cmd.protocol})`);+ break;+ case "ack": {+ // THE LEASE RENEWAL. Only here, and only from a renderer that painted.+ const first = !state.leases.has(cmd.episodeId);+ state.leases.set(cmd.episodeId, recoveryClock().now());+ if (first) {+ log(+ `[Recovery] pane is painting episode ${cmd.episodeId} — lease granted ` ++ `(paneOpen=${state.paneOpen})`+ );+ }+ break;+ }+ case "retry_now":+ if (episodeIsLive(cmd.episodeId)) {+ const n = attemptsSoFar(cmd.episodeId) + 1;+ log(`[Recovery] attempt ${n} (manual) for episode ${cmd.episodeId}`);+ tryNow(cmd.episodeId);+ }+ break;+ case "bye":+ log(`[Recovery] pane said bye (${cmd.reason}) — giving up on every live episode`);+ state.suppressUntilPerf = recoveryClock().now() + UI_SUPPRESS_AFTER_BYE_MS;+ giveUpAll();+ // The pane the user just closed must not reappear two seconds later, and+ // closing it is ALSO what revokes every lease: `teardown()` drops+ // `paneOpen`, and without that clause no heartbeat can grant anything.+ // One revocation path, not two — a second one is a second thing to+ // forget, and the superseded design's bug was exactly a revocation path+ // that existed on paper and cleared nothing.+ void closePane();+ break;+ }+}++async function openPane(): Promise<void> {+ const control = resolveMagmuxControl();+ if (!control) {+ log("[Recovery] no magmux control endpoint — no pane, recovery continues without a banner");+ return;+ }+ const lock = acquirePaneLock(control.sock);+ if (!lock) {+ log("[Recovery] another claudish already owns the recovery pane for this magmux");+ return;+ }+ state.lockPath = lock;++ const dir = createRecoverySocketDir();+ const socketPath = recoverySocketPathIn(dir);+ state.socketPath = socketPath;+ state.server = await startRecoverySocketServer({+ socketPath,+ snapshot: () => renderableEpisodeFrame(),+ onCommand,+ onDisconnect: () => {+ // EOF revokes nothing by itself — the lease expires on its own within+ // UI_LEASE_MS, and having exactly one expiry path means there is no+ // second one to forget. The retry loop is not the pane's business.+ log("[Recovery] pane disconnected");+ },+ });++ const socket = await connectControl(control.sock);+ if (!socket) {+ log(`[Recovery] could not reach magmux at ${control.sock}`);+ await teardown();+ return;+ }+ state.control = socket;+ socket.on("error", () => {+ state.paneOpen = false;+ });+ socket.on("close", () => {+ state.paneOpen = false;+ });++ const reply = await request(socket, {+ type: "open_pane",+ cmd: recoveryPaneCommandLine(socketPath),+ cwd: process.cwd(),+ split: "vertical",+ });+ if (!reply || reply.ok !== true) {+ log(`[Recovery] magmux refused open_pane: ${JSON.stringify(reply)}`);+ await teardown();+ return;+ }+ const result = reply.result as { pane?: number } | undefined;+ // Read the index back. magmux's own control panel occupies an index, so the+ // first pane an agent opens is NOT necessarily 1 — measured: 2.+ state.paneIndex = typeof result?.pane === "number" ? result.pane : null;+ state.paneOpen = true;+ log(`[Recovery] recovery pane opened (magmux pane ${state.paneIndex}) at ${socketPath}`);+}++async function closePane(): Promise<void> {+ if (state.control && state.paneIndex !== null) {+ await request(state.control, { type: "close_pane", pane: state.paneIndex, force: true }, 1_000);+ }+ await teardown();+}++async function teardown(): Promise<void> {+ state.paneOpen = false;+ state.paneIndex = null;+ state.leases.clear();+ if (state.control) {+ try {+ state.control.end();+ } catch {+ /* already gone */+ }+ state.control = null;+ }+ if (state.server) {+ await state.server.close();+ state.server = null;+ } else if (state.socketPath) {+ cleanupRecoverySocket(state.socketPath);+ }+ state.socketPath = null;+ releasePaneLock();+}++// ─── The hooks ───────────────────────────────────────────────────────────────++/**+ * Ask for a surface for this episode. Idempotent, and never awaited by the+ * retry loop — a pane that takes 200 ms to appear must not delay the first+ * retry by 200 ms.+ */+export function ensureRecoveryUi(episodeId: string): void {+ // Asking and being told no is still asking: the timestamp is what a later+ // phase's grace rule measures from, and it must be set even when the answer+ // is an immediate no.+ markPaneRequested(episodeId);+ if (state.lingerTimer !== null) {+ recoveryClock().clearTimeout(state.lingerTimer);+ state.lingerTimer = null;+ }+ if (state.paneOpen || state.opening) return;+ if (recoveryClock().now() < state.suppressUntilPerf) {+ log("[Recovery] pane suppressed — the user pressed [q] less than a minute ago");+ return;+ }+ state.opening = true;+ void openPane()+ .catch((err) => {+ log(`[Recovery] pane could not be opened: ${String(err)}`);+ })+ .finally(() => {+ state.opening = false;+ });+}++/**+ * One episode ended. Tell the renderer, and NOTHING else — a concurrent+ * episode may still be painting through the same socket and the same pane.+ */+export function releaseEpisodeUi(episodeId: string, outcome: RecoveryOutcome): void {+ state.leases.delete(episodeId);+ state.server?.broadcast({+ v: RECOVERY_PROTOCOL_VERSION,+ type: "closed",+ episodeId,+ outcome,+ });+ if (renderableEpisodeFrame() !== null) return; // something else is still live+ // Nothing left to paint. Linger, so the pane does not flap open and shut on+ // every client-retry cycle, and so the last thing that happened stays+ // readable for someone who looked away.+ if (state.lingerTimer !== null) recoveryClock().clearTimeout(state.lingerTimer);+ const clock = recoveryClock();+ const timer = clock.setTimeout(() => {+ state.lingerTimer = null;+ if (renderableEpisodeFrame() !== null) return;+ void closePane();+ }, PANE_LINGER_MS);+ clock.unref?.(timer);+ state.lingerTimer = timer;+}++const hooks: RecoveryUiHooks = {+ onEpisodeOpened: (episodeId) => ensureRecoveryUi(episodeId),+ onEpisodeClosed: (episodeId, outcome) => releaseEpisodeUi(episodeId, outcome),+ leaseValid: (episodeId) => uiLeaseValid(episodeId),+};++/**+ * Install the recovery UI into the coordinator.+ *+ * Called from exactly one place — the magmux launch wrapper, once it knows the+ * control socket. Every other entry point leaves it uninstalled, so `-p`,+ * `--stdin`, `serve`, the MCP server and the test suite cannot open a pane and+ * cannot hold a lease, without any of them having to opt out.+ */+export function installRecoveryUi(controlSock: string | null): void {+ setMagmuxControlSocket(controlSock);+ if (installed) return;+ installed = true;+ registerRecoveryUi(hooks);+ process.on("exit", () => {+ releasePaneLock();+ if (state.socketPath) cleanupRecoverySocket(state.socketPath);+ });+}++/** Tear everything down. Proxy shutdown, and tests. */+export async function shutdownRecoveryUi(): Promise<void> {+ if (state.lingerTimer !== null) {+ recoveryClock().clearTimeout(state.lingerTimer);+ state.lingerTimer = null;+ }+ await closePane();+ registerRecoveryUi(null);+ installed = false;+ state.suppressUntilPerf = 0;+ explicitControlSock = null;+}++/** Tests only: the manager is process state shared with every sibling test. */+export function __setPaneOpenForTests(open: boolean, socketPath?: string): void {+ state.paneOpen = open;+ if (socketPath !== undefined) state.socketPath = socketPath;+}++/** Tests only: feed a command as though a pane had sent it. */+export function __handleCommandForTests(cmd: RecoveryCommand): void {+ onCommand(cmd);+}++/** Tests only: forget every lease and every suppression. */+export function __resetUiStateForTests(): void {+ state.leases.clear();+ state.paneOpen = false;+ state.paneIndex = null;+ state.suppressUntilPerf = 0;+ state.opening = false;+ state.socketPath = null;+ state.server = null;+ state.control = null;+ if (state.lingerTimer !== null) {+ recoveryClock().clearTimeout(state.lingerTimer);+ state.lingerTimer = null;+ }+ releasePaneLock();+ explicitControlSock = null;+}diff --git a/packages/cli/src/recovery/pane-app.test.ts b/packages/cli/src/recovery/pane-app.test.tsnew file mode 100644index 00000000..7b92cc52--- /dev/null+++ b/packages/cli/src/recovery/pane-app.test.ts@@ -0,0 +1,342 @@+/**+ * The recovery banner.+ *+ * `renderPane` is pure, which is why it is the thing asserted on: the live+ * evidence (a capture of the real pane in a real magmux) and these tests then+ * look at the SAME function rather than at two implementations that drift.+ */++import { afterEach, describe, expect, test } from "bun:test";+import { EventEmitter } from "node:events";+import { resetThemeModeForTests, setThemeMode } from "../theme/theme-mode.js";+import {+ type PaneViewState,+ formatDuration,+ parsePaneArgs,+ renderPane,+ runRecoveryPane,+ stripAnsi,+ wrapText,+} from "./pane-app.js";+import {+ type RecoverySocketServer,+ createRecoverySocketDir,+ recoverySocketPathIn,+ startRecoverySocketServer,+} from "./socket-server.js";+import {+ PROXY_SILENT_MS,+ RECOVERY_PROTOCOL_VERSION,+ type RecoveryCommand,+ type RecoveryEpisodeFrame,+} from "./types.js";++const T0 = 1_700_000_000_000;++const EPISODE: RecoveryEpisodeFrame = {+ v: RECOVERY_PROTOCOL_VERSION,+ type: "episode",+ episodeId: "ep-1",+ state: "waiting",+ tier: 1,+ providerDisplayName: "Openai-codex",+ host: "chatgpt.com",+ endpoint: "https://chatgpt.com/backend-api/codex/responses",+ kind: "unreachable",+ code: "ConnectionClosed",+ loopback: false,+ reason:+ "Cannot reach Openai-codex at https://chatgpt.com/backend-api/codex/responses. Check your network connection.",+ attempts: 5,+ clientRetries: 0,+ startedAtMs: T0,+ nextAttemptAtMs: T0 + 132_000,+ lastOutcome: "ConnectionClosed after 11ms",+ waiters: 2,+ otherEpisodes: 0,+};++const view = (over: Partial<PaneViewState> = {}): PaneViewState => ({+ frame: EPISODE,+ closed: null,+ nowMs: T0 + 120_000,+ lastFrameAtMs: T0 + 120_000,+ connected: true,+ protocolMismatch: false,+ notice: null,+ ...over,+});++/** Escapes are the medium, not the message — strip them to read the banner. */+/** The banner's text, without the paint. Shared with the renderer on purpose:+ * a second escape-stripper in the test would be a second thing to get wrong. */+const plain = stripAnsi;++afterEach(() => {+ // `setThemeMode` flips a PROCESS-GLOBAL palette and Bun runs sibling test+ // files in one process. Never leave it flipped.+ resetThemeModeForTests();+});++describe("renderPane", () => {+ test("names the provider, the host and the reason — C-3's three facts", () => {+ const out = plain(renderPane(view(), 100));+ expect(out).toContain("Openai-codex");+ expect(out).toContain("chatgpt.com");+ // The reason is `buildConnectionErrorMessage`'s OWN sentence, not a+ // paraphrase: if the banner and the inline error ever disagree, the user is+ // being told two different stories about one fault.+ expect(out).toContain("Cannot reach Openai-codex at");+ // The sentence is wrapped to the pane, so read it as one flow.+ expect(out.replace(/\s+/g, " ")).toContain("Check your network connection.");+ expect(out).toContain("ConnectionClosed");+ });++ test("offers both keys and does not promise focus it will not steal", () => {+ const out = plain(renderPane(view(), 100));+ expect(out).toContain("[r]");+ expect(out).toContain("[q]");+ // magmux forwards clicks to focus a pane. Stealing focus would swallow+ // keystrokes the user was queueing into Claude Code, which accepts messages+ // while a turn is in flight — a worse failure than one extra click.+ expect(out).toContain("click here");+ });++ test("the countdown DECREASES between two renders — C-4, as a unit", () => {+ const early = plain(renderPane(view({ nowMs: T0 + 70_000 }), 100));+ const later = plain(renderPane(view({ nowMs: T0 + 73_000 }), 100));+ expect(early).toContain("next attempt in 1m 02s");+ expect(later).toContain("next attempt in 59s");+ expect(early).not.toBe(later);+ });++ test("a countdown that has gone past reads 0s, never a negative", () => {+ expect(plain(renderPane(view({ nowMs: T0 + 200_000 }), 100))).toContain("next attempt in 0s");+ });++ test("`attempting` shows what we are doing, not a frozen countdown", () => {+ // A 45-second connect with a countdown stuck at the same number reads as a+ // hang. This line is why the proxy ticks in every live state.+ const out = plain(+ renderPane(view({ frame: { ...EPISODE, state: "attempting", nextAttemptAtMs: null } }), 100)+ );+ expect(out).toContain("connecting to chatgpt.com");+ expect(out).not.toContain("next attempt in");+ });++ test("`handoff` says the client is expected back, which is the truth", () => {+ const out = plain(+ renderPane(+ view({+ frame: { ...EPISODE, state: "handoff", nextAttemptAtMs: null, tier: 2, clientRetries: 2 },+ }),+ 100+ )+ );+ expect(out).toContain("waiting for Claude Code to retry");+ expect(out).toContain("client retry 2");+ });++ test("counters: attempts, elapsed, last outcome, held requests, other episodes", () => {+ const out = plain(renderPane(view({ frame: { ...EPISODE, otherEpisodes: 1 } }), 100));+ expect(out).toContain("attempt 5");+ expect(out).toContain("2m 00s in recovery");+ expect(out).toContain("last: ConnectionClosed after 11ms");+ expect(out).toContain("2 requests held");+ expect(out).toContain("+1 more");+ });++ test("a loopback endpoint is marked local", () => {+ const out = plain(renderPane(view({ frame: { ...EPISODE, loopback: true } }), 100));+ expect(out).toContain("local");+ });++ test("the dead-proxy notice appears only after real silence", () => {+ // C-18 asserts this notice is ABSENT for a whole slow-connect run, so it+ // must key on silence longer than the proxy's own tick and nothing else.+ const quiet = view({ lastFrameAtMs: T0 + 120_000 - (PROXY_SILENT_MS - 500) });+ expect(plain(renderPane(quiet, 100))).not.toContain("has not reported");+ const dead = view({ lastFrameAtMs: T0 + 120_000 - (PROXY_SILENT_MS + 500) });+ expect(plain(renderPane(dead, 100))).toContain("has not reported");+ });++ test("a closed episode says how it ended and that the pane will go by itself", () => {+ const ok = plain(+ renderPane(+ view({ closed: { v: 1, type: "closed", episodeId: "ep-1", outcome: "recovered" } }),+ 100+ )+ );+ expect(ok).toContain("recovered");+ expect(ok).toContain("this pane closes on its own");+ const bad = plain(+ renderPane(+ view({ closed: { v: 1, type: "closed", episodeId: "ep-1", outcome: "gave_up" } }),+ 100+ )+ );+ expect(bad).toContain("gave up");+ });++ test("version skew prints a notice instead of rendering fields it cannot read", () => {+ const out = plain(renderPane(view({ protocolMismatch: true }), 100));+ expect(out).toContain("version skew");+ expect(out).not.toContain("Openai-codex");+ });++ test("before the first frame it says what it is waiting for", () => {+ expect(plain(renderPane(view({ frame: null }), 100))).toContain("waiting for the proxy");+ expect(plain(renderPane(view({ frame: null, connected: false }), 100))).toContain(+ "connecting to the claudish proxy"+ );+ });++ test("every line fits a 40-column pane — a split pane is HALF the terminal", () => {+ // Measured on magmux 0.11.0: `split:"vertical"` took pane 0 from 40 rows to+ // 20 and gave the new pane 19. A user with a narrow window gets a narrow+ // pane, and a banner that runs off it is a banner that cannot be read.+ const narrow = renderPane(view(), 40)+ .split("\n")+ .map((l) => plain(l));+ for (const line of narrow) expect(line.length).toBeLessThanOrEqual(40);+ // …and the keys survive the squeeze, because they are the part a user+ // cannot work out for themselves.+ const joined = narrow.join(" ");+ expect(joined).toContain("[r]");+ expect(joined).toContain("[q]");+ });++ // ── THE THEMING INVARIANT ────────────────────────────────────────────────+ test("the palette is read at RENDER time, not at module load", () => {+ // This module was imported at the top of this file, long before the line+ // below runs — which is exactly the production ordering: command modules+ // are imported before theme detection completes. A module-level+ // `const C = cliAnsi()` would snapshot the dark palette here and never+ // change again. That bug class has been found six times in this codebase.+ setThemeMode("dark");+ const dark = renderPane(view(), 100);+ setThemeMode("light");+ const light = renderPane(view(), 100);+ expect(light).not.toBe(dark);+ expect(light).toContain("\x1b[38;2;"); // truecolor: the LIGHT palette+ expect(dark).not.toContain("\x1b[38;2;"); // classic 16-colour: dark/unknown+ });++ test("the red banner fill is the SAME in both themes — it is theme-independent", () => {+ // The one element that must be legible whatever happens does not depend on+ // detection succeeding: a self-contained mid-dark fill with bright-white+ // ink, the pairing `team-grid.ts` established.+ const FILL = "\x1b[1;97;48;2;160;50;70m";+ setThemeMode("dark");+ expect(renderPane(view(), 100)).toContain(FILL);+ setThemeMode("light");+ expect(renderPane(view(), 100)).toContain(FILL);+ });+});++describe("formatDuration / wrapText / parsePaneArgs", () => {+ test("durations", () => {+ expect(formatDuration(0)).toBe("0s");+ expect(formatDuration(-5_000)).toBe("0s");+ expect(formatDuration(47_000)).toBe("47s");+ expect(formatDuration(132_000)).toBe("2m 12s");+ });++ test("wrapping never drops or splits a word", () => {+ const words = "alpha beta gamma delta epsilon".split(" ");+ const lines = wrapText(words.join(" "), 12);+ expect(lines.join(" ").split(/\s+/)).toEqual(words);+ for (const l of lines) expect(l.length).toBeLessThanOrEqual(12);+ });++ test("--socket parses in both spellings, and is required", () => {+ expect(parsePaneArgs(["--socket", "/tmp/x/r.sock"]).socketPath).toBe("/tmp/x/r.sock");+ expect(parsePaneArgs(["--socket=/tmp/y/r.sock"]).socketPath).toBe("/tmp/y/r.sock");+ expect(parsePaneArgs([]).socketPath).toBeNull();+ });+});++// ─── Keys, over a real socket ────────────────────────────────────────────────++class FakeStdin extends EventEmitter {+ setRawMode() {+ return this;+ }+ resume() {+ return this;+ }+ pause() {+ return this;+ }+}+class FakeStdout extends EventEmitter {+ columns = 100;+ chunks: string[] = [];+ write(s: string) {+ this.chunks.push(s);+ return true;+ }+}++describe("the pane's two keys", () => {+ test("`r`, `R` and Enter each ask for an attempt now; `q` gives up and exits", async () => {+ const path = recoverySocketPathIn(createRecoverySocketDir());+ const received: RecoveryCommand[] = [];+ let server: RecoverySocketServer | null = null;+ server = await startRecoverySocketServer({+ socketPath: path,+ snapshot: () => EPISODE,+ onCommand: (cmd) => received.push(cmd),+ tickMs: 0,+ });++ const stdin = new FakeStdin();+ const stdout = new FakeStdout();+ const done = runRecoveryPane({+ socketPath: path,+ stdin: stdin as unknown as NodeJS.ReadStream,+ stdout: stdout as unknown as NodeJS.WriteStream,+ });+ await new Promise((r) => setTimeout(r, 150));++ for (const key of ["r", "R", "\r"]) stdin.emit("data", Buffer.from(key));+ await new Promise((r) => setTimeout(r, 120));+ expect(received.filter((c) => c.type === "retry_now")).toHaveLength(3);+ expect(+ received+ .filter((c) => c.type === "retry_now")+ .every((c) => "episodeId" in c && c.episodeId === "ep-1")+ ).toBe(true);++ stdin.emit("data", Buffer.from("q"));+ expect(await done).toBe(0);+ await new Promise((r) => setTimeout(r, 80));+ expect(received.filter((c) => c.type === "bye")).toHaveLength(1);+ await server.close();+ }, 15_000);++ test("the banner is painted from the REPLAYED frame, with no tick at all", async () => {+ const path = recoverySocketPathIn(createRecoverySocketDir());+ let server: RecoverySocketServer | null = null;+ server = await startRecoverySocketServer({+ socketPath: path,+ snapshot: () => EPISODE,+ onCommand: () => {},+ tickMs: 0,+ });+ const stdin = new FakeStdin();+ const stdout = new FakeStdout();+ const done = runRecoveryPane({+ socketPath: path,+ stdin: stdin as unknown as NodeJS.ReadStream,+ stdout: stdout as unknown as NodeJS.WriteStream,+ });+ await new Promise((r) => setTimeout(r, 150));+ const painted = plain(stdout.chunks.join(""));+ expect(painted).toContain("Openai-codex");+ expect(painted).toContain("chatgpt.com");+ stdin.emit("data", Buffer.from("q"));+ await done;+ await server.close();+ }, 15_000);+});diff --git a/packages/cli/src/recovery/pane-app.ts b/packages/cli/src/recovery/pane-app.tsnew file mode 100644index 00000000..ff95f971--- /dev/null+++ b/packages/cli/src/recovery/pane-app.ts@@ -0,0 +1,489 @@+/**+ * `claudish recovery-pane` — the five-line banner that earns claudish the right+ * to wait.+ *+ * PLAIN ANSI WITH RAW STDIN, NOT OpenTUI. This process exists to paint five+ * lines and read two keys. claudish already boots OpenTUI three times (config+ * TUI, probe TUI, resume picker) and a fourth boot for a banner is+ * disproportionate — it would pay a React reconciler and a renderer handshake+ * on the critical path of a user who is, by construction, already having a bad+ * minute. `team-grid.ts`'s self-contained banner is the established precedent+ * in this repo for exactly this shape.+ *+ * THEMING. `cliAnsi()` is called INSIDE `renderPane`, never at module load:+ * command modules are imported before theme detection runs, so a module-level+ * `const C = cliAnsi()` snapshots the pre-detection (dark) palette forever.+ * That bug class has been found six times in this codebase. The banner FILL is+ * additionally theme-independent by construction — a self-contained mid-dark+ * red with bright-white ink, the same pairing `team-grid.ts` uses — so the one+ * element that must be legible no matter what does not depend on detection+ * succeeding at all. (Measured: magmux DOES answer OSC 11 inside a pane, so+ * detection works; the banner does not rely on it.)+ *+ * WHY A PANE AND NOT AN OVERLAY. Keys typed here reach THIS process's own PTY,+ * not Claude Code's. That is the whole reason the recovery UI is a pane: it+ * dissolves the "how does a keystroke reach the proxy" question instead of+ * working around it. Focus is deliberately NOT stolen — Claude Code accepts+ * messages while a turn is in flight, and capturing those keystrokes into our+ * renderer would be a worse failure than one extra click.+ */++import { cliAnsi } from "../theme/ansi.js";+import { type RecoverySocketClient, connectRecoverySocket } from "./socket-client.js";+import {+ PANE_LINGER_MS,+ PROXY_SILENT_MS,+ RECOVERY_PROTOCOL_VERSION,+ type RecoveryClosedFrame,+ type RecoveryEpisodeFrame,+ UI_HEARTBEAT_MS,+} from "./types.js";++export interface PaneViewState {+ /** The episode being painted, or null before the first frame. */+ frame: RecoveryEpisodeFrame | null;+ /** Set once the episode ends; the pane lingers on this rather than vanishing. */+ closed: RecoveryClosedFrame | null;+ /** EPOCH ms — the same clock `nextAttemptAtMs` and `startedAtMs` are in. */+ nowMs: number;+ /** EPOCH ms of the last frame of any kind, or null. */+ lastFrameAtMs: number | null;+ connected: boolean;+ protocolMismatch: boolean;+ /** A transient line, e.g. after `[r]`. */+ notice: string | null;+}++function pad(text: string, width: number): string {+ return text.length >= width ? text.slice(0, width) : text + " ".repeat(width - text.length);+}++/**+ * Truncate to a VISIBLE width, stepping over the escapes.+ *+ * `String.length` counts an SGR escape as eight or nine characters and would+ * cut a line in the middle of one, leaving the rest of the pane painted in+ * whatever colour the fragment happened to start. A split pane is HALF the+ * terminal — measured: `split:"vertical"` took pane 0 from 40 rows to 20 — so+ * "the pane is always wide" is not an assumption this can make.+ */+/** The SGR sequence starting at `i` (`ESC [ … m`), or null. */+function matchSgr(text: string, i: number): string | null {+ if (text[i] !== "\x1b" || text[i + 1] !== "[") return null;+ let j = i + 2;+ while (j < text.length) {+ const c = text.charCodeAt(j);+ const isParam = (c >= 0x30 && c <= 0x39) || c === 0x3b; // 0-9 ;+ if (!isParam) break;+ j++;+ }+ return text[j] === "m" ? text.slice(i, j + 1) : null;+}++/**+ * Drop every SGR sequence, leaving the text.+ *+ * Exported for the tests and for the evidence harness: assertions are about+ * what a person reads, and a capture full of escapes cannot be read by either.+ */+export function stripAnsi(text: string): string {+ let out = "";+ for (let i = 0; i < text.length; i++) {+ const m = matchSgr(text, i);+ if (m) {+ i += m.length - 1;+ continue;+ }+ out += text[i];+ }+ return out;+}++export function clipAnsi(text: string, width: number): string {+ let out = "";+ let visible = 0;+ for (let i = 0; i < text.length; i++) {+ const ch = text[i] as string;+ if (ch === "\x1b") {+ // ESC is the subject matter here, so the scan is written by hand rather+ // than as a regex: a regex literal containing the control character trips+ // `noControlCharactersInRegex`, and suppressing a lint rule to parse the+ // one thing this function exists to parse is worse than four lines of+ // explicit scanning.+ const m = matchSgr(text, i);+ if (m) {+ out += m;+ i += m.length - 1;+ continue;+ }+ }+ if (visible >= width) return `${out}\x1b[0m`;+ out += ch;+ visible++;+ }+ return out;+}++/** `2m 12s`, `47s`, `0s`. Never negative — a countdown that went past reads `0s`. */+export function formatDuration(ms: number): string {+ const total = Math.max(0, Math.round(ms / 1000));+ if (total < 60) return `${total}s`;+ const m = Math.floor(total / 60);+ const s = total % 60;+ return `${m}m ${String(s).padStart(2, "0")}s`;+}++/** Greedy wrap. The reason sentence is the one thing that must stay readable. */+export function wrapText(text: string, width: number): string[] {+ if (width <= 0) return [text];+ const words = text.split(/\s+/).filter(Boolean);+ const lines: string[] = [];+ let line = "";+ for (const w of words) {+ if (!line) {+ line = w;+ } else if (line.length + 1 + w.length <= width) {+ line += ` ${w}`;+ } else {+ lines.push(line);+ line = w;+ }+ }+ if (line) lines.push(line);+ return lines.length > 0 ? lines : [""];+}++/**+ * Build the banner. PURE — no clock, no terminal, no escape-code state.+ *+ * Pure because it is the only part of this process worth asserting on in a+ * unit test, and because the live evidence (a capture of the real pane) and the+ * test then look at the same function rather than at two implementations that+ * could drift.+ */+export function renderPane(state: PaneViewState, width: number): string {+ // RENDER TIME. Never at module load.+ const C = cliAnsi();+ const w = Math.max(28, Math.min(width || 80, 200));+ const RED_FILL = "\x1b[1;97;48;2;160;50;70m";+ const GREEN_FILL = "\x1b[1;97;48;2;30;120;70m";+ const lines: string[] = [];++ // Every return goes through `finish`, so there is ONE place the pane's width+ // is enforced and no later branch can forget it.+ const finish = (ls: string[]) => ls.map((l) => clipAnsi(l, w)).join("\n");++ if (state.protocolMismatch) {+ return finish([+ `${RED_FILL}${pad(" ██ CLAUDISH RECOVERY · version skew", w)}\x1b[0m`,+ `${C.GRAY} This pane speaks protocol v${RECOVERY_PROTOCOL_VERSION}; the proxy does not.${C.RESET}`,+ `${C.GRAY} Restart the session so both halves are the same build.${C.RESET}`,+ "",+ ]);+ }++ const ep = state.frame;+ const closed = state.closed;++ if (closed) {+ const ok = closed.outcome === "recovered";+ const fill = ok ? GREEN_FILL : RED_FILL;+ const title = ok+ ? ` ██ NETWORK · ${ep?.providerDisplayName ?? "provider"} recovered`+ : ` ██ NETWORK · recovery ended (${closed.outcome.replace(/_/g, " ")})`;+ lines.push(`${fill}${pad(title, w)}\x1b[0m`);+ if (ep) {+ lines.push(+ `${C.GRAY} ${pad(`${ep.attempts} attempts over ${formatDuration(state.nowMs - ep.startedAtMs)}`, w - 2)}${C.RESET}`+ );+ }+ lines.push(`${C.GRAY} this pane closes on its own${C.RESET}`);+ return finish(lines);+ }++ if (!ep) {+ lines.push(`${RED_FILL}${pad(" ██ NETWORK · claudish is retrying", w)}\x1b[0m`);+ lines.push(+ `${C.GRAY} ${state.connected ? "waiting for the proxy to describe the failure…" : "connecting to the claudish proxy…"}${C.RESET}`+ );+ return finish(lines);+ }++ // 1 — the red line. Provider and the failure kind, in the fill that reads on+ // any page.+ const kindWord =+ ep.kind === "dns" ? "DNS failure" : ep.kind === "refused" ? "refused" : "unreachable";+ lines.push(`${RED_FILL}${pad(` ██ NETWORK · ${ep.providerDisplayName} ${kindWord}`, w)}\x1b[0m`);++ // 2 — the reason, in `buildConnectionErrorMessage`'s OWN words. The same+ // sentence the inline 400 would have carried; if the two ever diverge the+ // user is being told two different stories about one fault.+ for (const l of wrapText(ep.reason, w - 4)) lines.push(`${C.STRONG} ${l}${C.RESET}`);++ // 3 — host and code, as discrete tokens. The sentence above contains the host+ // too, but embedded in prose; this line is what a capture can be grepped for.+ lines.push(+ `${C.GRAY} host ${C.RESET}${C.CYAN}${ep.host}${C.RESET}${C.GRAY} · ${ep.code ?? "no code"}${+ ep.loopback ? " · local" : ""+ }${C.RESET}`+ );++ // 4 — the live line. A countdown while waiting; what we are doing instead+ // while an attempt is actually in flight, because a frozen countdown during a+ // 45-second connect reads as a hang.+ const elapsed = formatDuration(state.nowMs - ep.startedAtMs);+ const when =+ ep.state === "waiting" && ep.nextAttemptAtMs !== null+ ? `${C.YELLOW}next attempt in ${formatDuration(ep.nextAttemptAtMs - state.nowMs)}${C.RESET}`+ : ep.state === "handoff"+ ? `${C.YELLOW}waiting for Claude Code to retry${C.RESET}`+ : `${C.YELLOW}connecting to ${ep.host}…${C.RESET}`;+ const tierNote = ep.tier === 2 ? ` · client retry ${ep.clientRetries}` : "";+ lines.push(+ ` ${when}${C.GRAY} · attempt ${ep.attempts} · ${elapsed} in recovery${tierNote}${C.RESET}`+ );++ // 5 — last outcome and the other-episode count.+ const others = ep.otherEpisodes > 0 ? ` · +${ep.otherEpisodes} more` : "";+ const waiters = ep.waiters === 1 ? "1 request held" : `${ep.waiters} requests held`;+ lines.push(`${C.GRAY} last: ${ep.lastOutcome} · ${waiters}${others}${C.RESET}`);++ // 6 — the keys. The click instruction is not decoration: magmux forwards+ // mouse clicks to focus a pane, and focus is not stolen — so "click, then+ // press" is the actual interaction and the banner has to say so.+ //+ // A `split` pane is HALF the terminal (measured: 40 rows became 20 and 19),+ // and a user with a narrow window gets a narrow pane. Below 64 columns the+ // full sentence wraps or is cut, so the hint sheds the part a user can work+ // out — where to click — and keeps the part they cannot: which keys exist.+ lines.push(+ w < 64+ ? `${C.GRAY} ${C.RESET}${C.BOLD}[r]${C.RESET} try now${C.GRAY} · ${C.RESET}${C.BOLD}[q]${C.RESET} give up`+ : `${C.GRAY} click here or Ctrl-G Tab, then ${C.RESET}${C.BOLD}[r]${C.RESET} try now${C.GRAY} · ${C.RESET}${C.BOLD}[q]${C.RESET} give up`+ );++ if (state.notice) lines.push(`${C.GREEN} ${state.notice}${C.RESET}`);++ // The dead-proxy notice. Frame silence is the heuristic, which is exactly why+ // the proxy ticks in EVERY live state — a 45-second connect used to look+ // identical to a dead proxy.+ if (+ state.lastFrameAtMs !== null &&+ state.nowMs - state.lastFrameAtMs > PROXY_SILENT_MS &&+ state.connected+ ) {+ lines.push(+ `${C.RED} the claudish proxy has not reported in ${formatDuration(state.nowMs - state.lastFrameAtMs)}${C.RESET}`+ );+ }+ if (!state.connected) {+ lines.push(`${C.RED} the claudish proxy closed this connection${C.RESET}`);+ }++ return finish(lines);+}++export interface RecoveryPaneOptions {+ socketPath: string;+ stdin?: NodeJS.ReadStream;+ stdout?: NodeJS.WriteStream;+ /** EPOCH ms. Injectable so a test can drive the countdown deterministically. */+ now?: () => number;+ /** Stop after this long with no live episode. Tests pass something small. */+ lingerMs?: number;+}++/** Parse `--socket <path>` out of a recovery-pane argv slice. */+export function parsePaneArgs(argv: string[]): { socketPath: string | null } {+ for (let i = 0; i < argv.length; i++) {+ const a = argv[i];+ if (a === "--socket" && argv[i + 1]) return { socketPath: argv[i + 1] as string };+ if (a?.startsWith("--socket=")) return { socketPath: a.slice("--socket=".length) };+ }+ return { socketPath: null };+}++/**+ * Run the pane until the user quits, the episode ends and the linger expires,+ * or the proxy goes away. Resolves with the process exit code.+ */+export async function runRecoveryPane(opts: RecoveryPaneOptions): Promise<number> {+ const stdin = opts.stdin ?? process.stdin;+ const stdout = opts.stdout ?? process.stdout;+ const now = opts.now ?? (() => Date.now());+ const lingerMs = opts.lingerMs ?? PANE_LINGER_MS;++ const state: PaneViewState = {+ frame: null,+ closed: null,+ nowMs: now(),+ lastFrameAtMs: null,+ connected: true,+ protocolMismatch: false,+ notice: null,+ };++ let noticeUntilMs = 0;+ let closedAtMs: number | null = null;+ /** The episode this renderer is heartbeating. Exactly one at a time. */+ let paintedEpisodeId: string | null = null;+ let client: RecoverySocketClient | null = null;++ const paint = () => {+ state.nowMs = now();+ if (state.notice && state.nowMs > noticeUntilMs) state.notice = null;+ const width = stdout.columns ?? 80;+ const body = renderPane(state, width);+ // Home, repaint each line clearing to EOL, then clear the rest. No+ // alternate screen: the pane's content stays in magmux's scrollback, which+ // is where a user looks after the fact to find out what happened.+ const painted = body.split("\n").join("\x1b[K\n");+ stdout.write(`\x1b[H${painted}\x1b[K\x1b[J`);++ // THE LEASE. Sent after painting, and on this timer rather than on frame+ // arrival — the proxy has nothing to say for the 20–75 s an unreachable+ // connect takes, and a lease renewed by frames would expire in exactly the+ // failure class this feature exists for.+ if (paintedEpisodeId && client?.connected) {+ client.send({+ v: RECOVERY_PROTOCOL_VERSION,+ type: "ack",+ episodeId: paintedEpisodeId,+ paintedAt: state.nowMs,+ });+ }+ };++ stdout.write("\x1b[?25l"); // hide the cursor; there is nothing to type into+ const restore = () => {+ try {+ stdout.write("\x1b[?25h\n");+ stdin.setRawMode?.(false);+ stdin.pause();+ } catch {+ /* the terminal went away first */+ }+ };++ client = await connectRecoverySocket({+ socketPath: opts.socketPath,+ onFrame: (frame) => {+ state.lastFrameAtMs = now();+ if (frame.type === "closed") {+ if (frame.outcome === "protocol_mismatch") {+ state.protocolMismatch = true;+ paint();+ return;+ }+ if (!state.frame || frame.episodeId === state.frame.episodeId) {+ state.closed = frame;+ closedAtMs = now();+ // Stop heartbeating: this episode is over, and a lease that outlived+ // its episode is the latch this design replaced.+ paintedEpisodeId = null;+ }+ paint();+ return;+ }+ // A new episode supersedes the old one. The renderer heartbeats exactly+ // one episode; the previous one's lease lapses on its own within+ // UI_LEASE_MS with no extra protocol.+ state.frame = frame;+ state.closed = null;+ closedAtMs = null;+ paintedEpisodeId = frame.episodeId;+ paint();+ },+ onClose: () => {+ state.connected = false;+ paint();+ },+ });++ if (!client) {+ state.connected = false;+ paint();+ restore();+ return 1;+ }++ let done: (code: number) => void = () => {};+ const finished = new Promise<number>((resolve) => {+ done = resolve;+ });++ // Raw stdin: the pane's two keys must not need a newline, and the terminal+ // must not echo them into the banner.+ try {+ stdin.setRawMode?.(true);+ } catch {+ /* not a TTY — the banner still renders, the keys simply do nothing */+ }+ // Outside the try: a stdin that cannot go raw can still deliver bytes, and+ // resume() is also what keeps this process's event loop alive.+ stdin.resume();+ stdin.on("data", (buf: Buffer) => {+ const s = buf.toString("utf-8");+ for (const ch of s) {+ if (ch === "\x03") {+ // Ctrl-C in this pane means the same as [q]: the user asked to stop.+ client?.send({ v: RECOVERY_PROTOCOL_VERSION, type: "bye", reason: "user_quit" });+ restore();+ done(0);+ return;+ }+ if (ch === "q" || ch === "Q") {+ client?.send({ v: RECOVERY_PROTOCOL_VERSION, type: "bye", reason: "user_quit" });+ restore();+ done(0);+ return;+ }+ if (ch === "r" || ch === "R" || ch === "\r" || ch === "\n") {+ if (state.frame && client?.connected) {+ client.send({+ v: RECOVERY_PROTOCOL_VERSION,+ type: "retry_now",+ episodeId: state.frame.episodeId,+ });+ state.notice = "retrying now…";+ noticeUntilMs = now() + 3_000;+ paint();+ }+ }+ }+ });++ // ONE timer drives both the countdown and the heartbeat, at UI_HEARTBEAT_MS.+ // They are the same act — "I am still painting this episode" — and splitting+ // them into two timers would let a frozen renderer keep its lease.+ const ticker = setInterval(() => {+ paint();+ if (closedAtMs !== null && now() - closedAtMs > lingerMs) {+ restore();+ done(0);+ }+ if (!state.connected && client && !client.connected) {+ // The proxy is gone. Linger briefly so the last state is readable, then+ // leave rather than sitting on a dead socket forever.+ if (closedAtMs === null) closedAtMs = now();+ }+ }, UI_HEARTBEAT_MS);+ ticker.unref?.();++ paint();+ const code = await finished;+ clearInterval(ticker);+ client.close();+ return code;+}++/** The `claudish recovery-pane` entry point. */+export async function recoveryPaneCommand(argv: string[]): Promise<void> {+ const { socketPath } = parsePaneArgs(argv);+ if (!socketPath) {+ console.error("Usage: claudish recovery-pane --socket <path>");+ process.exit(2);+ }+ const code = await runRecoveryPane({ socketPath });+ process.exit(code);+}diff --git a/packages/cli/src/recovery/socket-client.ts b/packages/cli/src/recovery/socket-client.tsnew file mode 100644index 00000000..14e8683d--- /dev/null+++ b/packages/cli/src/recovery/socket-client.ts@@ -0,0 +1,130 @@+/**+ * The pane half of the recovery link.+ *+ * Shaped after `team-grid.ts`'s `subscribeToMagmux` — the same NDJSON framing,+ * the same tolerance of malformed lines, and above all the same CONNECT-RETRY+ * LOOP, because the socket appears asynchronously relative to the process that+ * dials it. There the race is magmux creating its socket after spawn; here it+ * is the pane starting ~100 ms after `open_pane` while the proxy is still+ * binding, or — in the normal case — long after. Either way a single+ * `connect()` that fails is indistinguishable from a proxy that is not coming,+ * and the difference is the whole user experience.+ */++import { existsSync } from "node:fs";+import { type Socket, connect as netConnect } from "node:net";+import {+ RECOVERY_PROTOCOL_VERSION,+ type RecoveryCommand,+ type RecoveryFrame,+ encodeLine,+ parseFrame,+} from "./types.js";++export interface RecoverySocketClientOptions {+ socketPath: string;+ onFrame: (frame: RecoveryFrame) => void;+ /** The proxy hung up, or was never there. */+ onClose?: () => void;+ /** Total connect attempts before giving up. 40 × 50 ms ≈ 2 s, as in team-grid. */+ attempts?: number;+ retryDelayMs?: number;+}++export interface RecoverySocketClient {+ send(cmd: RecoveryCommand): void;+ close(): void;+ readonly connected: boolean;+}++/**+ * Dial the recovery socket, retrying while it does not yet exist.+ *+ * Resolves null when every attempt failed — the caller decides what to print.+ * It never throws: a pane that crashes on a missing socket replaces a banner+ * with a stack trace in the middle of the user's screen.+ */+export async function connectRecoverySocket(+ opts: RecoverySocketClientOptions+): Promise<RecoverySocketClient | null> {+ const attempts = opts.attempts ?? 40;+ const delay = opts.retryDelayMs ?? 50;+ let socket: Socket | null = null;++ for (let attempt = 0; attempt < attempts; attempt++) {+ if (existsSync(opts.socketPath)) {+ try {+ socket = await new Promise<Socket>((resolve, reject) => {+ const s = netConnect(opts.socketPath);+ s.once("connect", () => resolve(s));+ s.once("error", reject);+ });+ break;+ } catch {+ /* not ready yet — the server binds after mkdir, so the file can exist+ for a few microseconds before anything is listening on it */+ }+ }+ await new Promise((r) => setTimeout(r, delay));+ }++ if (!socket) return null;++ const live = socket;+ let open = true;+ let buf = "";++ live.on("data", (chunk: Buffer) => {+ buf += chunk.toString("utf-8");+ if (buf.length > 256 * 1024) buf = buf.slice(-4096);+ let nl = buf.indexOf("\n");+ while (nl >= 0) {+ const line = buf.slice(0, nl).trim();+ buf = buf.slice(nl + 1);+ nl = buf.indexOf("\n");+ if (!line) continue;+ const frame = parseFrame(line);+ if (frame) opts.onFrame(frame);+ }+ });++ const gone = () => {+ if (!open) return;+ open = false;+ opts.onClose?.();+ };+ live.on("error", gone);+ live.on("close", gone);+ live.on("end", gone);++ const client: RecoverySocketClient = {+ get connected() {+ return open;+ },+ send(cmd: RecoveryCommand) {+ if (!open || live.destroyed) return;+ try {+ live.write(encodeLine(cmd));+ } catch {+ /* the proxy went away; `gone` will fire */+ }+ },+ close() {+ open = false;+ try {+ live.end();+ } catch {+ /* already gone */+ }+ },+ };++ client.send({+ v: RECOVERY_PROTOCOL_VERSION,+ type: "hello",+ protocol: RECOVERY_PROTOCOL_VERSION,+ pid: process.pid,+ });++ return client;+}diff --git a/packages/cli/src/recovery/socket-server.test.ts b/packages/cli/src/recovery/socket-server.test.tsnew file mode 100644index 00000000..fa7d5aaf--- /dev/null+++ b/packages/cli/src/recovery/socket-server.test.ts@@ -0,0 +1,234 @@+/**+ * The recovery socket, over a REAL unix socket with REAL clients.+ *+ * No `mock.module()` anywhere: this repo's standing rule is that mocking shared+ * infrastructure bleeds across sibling test files in Bun's module registry, and+ * `node:net` is about as shared as infrastructure gets. The socket is cheap+ * enough to use for real.+ */++import { afterEach, describe, expect, test } from "bun:test";+import { existsSync, statSync } from "node:fs";+import { connect } from "node:net";+import { dirname } from "node:path";+import {+ type RecoverySocketServer,+ createRecoverySocketDir,+ recoverySocketPathIn,+ startRecoverySocketServer,+} from "./socket-server.js";+import {+ RECOVERY_PROTOCOL_VERSION,+ type RecoveryCommand,+ type RecoveryEpisodeFrame,+ type RecoveryFrame,+ encodeLine,+ parseFrame,+} from "./types.js";++const EPISODE: RecoveryEpisodeFrame = {+ v: RECOVERY_PROTOCOL_VERSION,+ type: "episode",+ episodeId: "ep-replay",+ state: "attempting",+ tier: 1,+ providerDisplayName: "Openai-codex",+ host: "chatgpt.com",+ endpoint: "https://chatgpt.com/backend-api/codex/responses",+ kind: "unreachable",+ code: "ConnectionClosed",+ loopback: false,+ reason: "Cannot reach Openai-codex at https://chatgpt.com/backend-api/codex/responses.",+ attempts: 3,+ clientRetries: 0,+ startedAtMs: 1_700_000_000_000,+ nextAttemptAtMs: null,+ lastOutcome: "ConnectionClosed after 11ms",+ waiters: 1,+ otherEpisodes: 0,+};++/** A raw NDJSON client, so the test exercises the same bytes the pane does. */+function rawClient(path: string): Promise<{+ frames: RecoveryFrame[];+ send: (line: string) => void;+ end: () => void;+}> {+ return new Promise((resolve, reject) => {+ const s = connect(path);+ const frames: RecoveryFrame[] = [];+ let buf = "";+ s.on("data", (chunk: Buffer) => {+ buf += chunk.toString();+ let nl = buf.indexOf("\n");+ while (nl >= 0) {+ const line = buf.slice(0, nl);+ buf = buf.slice(nl + 1);+ nl = buf.indexOf("\n");+ const f = parseFrame(line);+ if (f) frames.push(f);+ }+ });+ s.once("connect", () =>+ resolve({+ frames,+ send: (line: string) => s.write(line),+ end: () => s.destroy(),+ })+ );+ s.once("error", reject);+ });+}++const settle = (ms = 60) => new Promise((r) => setTimeout(r, ms));++let server: RecoverySocketServer | null = null;++afterEach(async () => {+ await server?.close();+ server = null;+});++describe("recovery socket server", () => {+ test("replays the current episode to a client that connects LATE", async () => {+ // The pane starts ~100ms after `open_pane`, which is always after the+ // episode began. Without a replay its first paint is blank, and that blank+ // is the first thing a user ever sees of this feature.+ const path = recoverySocketPathIn(createRecoverySocketDir());+ server = await startRecoverySocketServer({+ socketPath: path,+ snapshot: () => EPISODE,+ onCommand: () => {},+ tickMs: 0,+ });++ const client = await rawClient(path);+ await settle();+ expect(client.frames).toHaveLength(1);+ expect(client.frames[0]).toEqual(EPISODE);+ client.end();+ });++ test("ticks in EVERY live state, including `attempting` with no state change", async () => {+ // The state below never changes and no broadcast is ever called. Frames+ // must arrive anyway: a 45-second connect is 45 seconds of silence+ // otherwise, which looks identical to a dead proxy to the pane's own+ // liveness heuristic.+ const path = recoverySocketPathIn(createRecoverySocketDir());+ server = await startRecoverySocketServer({+ socketPath: path,+ snapshot: () => ({ ...EPISODE, state: "attempting", nextAttemptAtMs: null }),+ onCommand: () => {},+ tickMs: 25,+ });+ const client = await rawClient(path);+ await settle(180);+ // 1 replay + several ticks.+ expect(client.frames.length).toBeGreaterThanOrEqual(4);+ expect(client.frames.every((f) => f.type === "episode" && f.state === "attempting")).toBe(true);+ client.end();+ });++ test("a null snapshot suppresses the replay and the tick rather than sending junk", async () => {+ const path = recoverySocketPathIn(createRecoverySocketDir());+ server = await startRecoverySocketServer({+ socketPath: path,+ snapshot: () => null,+ onCommand: () => {},+ tickMs: 25,+ });+ const client = await rawClient(path);+ await settle(120);+ expect(client.frames).toHaveLength(0);+ client.end();+ });++ test("routes ack, retry_now and bye; ignores malformed and skewed lines", async () => {+ const path = recoverySocketPathIn(createRecoverySocketDir());+ const seen: RecoveryCommand[] = [];+ server = await startRecoverySocketServer({+ socketPath: path,+ snapshot: () => null,+ onCommand: (cmd) => seen.push(cmd),+ tickMs: 0,+ });+ const client = await rawClient(path);+ client.send(+ encodeLine({ v: 1, type: "ack", episodeId: "ep-replay", paintedAt: 1 }) ++ encodeLine({ v: 1, type: "retry_now", episodeId: "ep-replay" }) ++ "{bad json}\n" ++ `${JSON.stringify({ v: 9, type: "ack", episodeId: "ep-replay" })}\n` ++ encodeLine({ v: 1, type: "bye", reason: "user_quit" })+ );+ await settle();+ expect(seen.map((c) => c.type)).toEqual(["ack", "retry_now", "bye"]);+ client.end();+ });++ test("a client on the WRONG protocol is told so and dropped", async () => {+ const path = recoverySocketPathIn(createRecoverySocketDir());+ const seen: RecoveryCommand[] = [];+ server = await startRecoverySocketServer({+ socketPath: path,+ snapshot: () => null,+ onCommand: (cmd) => seen.push(cmd),+ tickMs: 0,+ });+ const client = await rawClient(path);+ client.send(`${JSON.stringify({ v: 9, type: "hello", protocol: 9, pid: 1 })}\n`);+ await settle();+ expect(client.frames).toHaveLength(1);+ expect(client.frames[0]).toMatchObject({ type: "closed", outcome: "protocol_mismatch" });+ // And the command is NOT delivered — a build we cannot talk to must not be+ // able to drive the ladder.+ expect(seen).toHaveLength(0);+ });++ test("the directory is 0700, the socket 0600, and both are gone after close", async () => {+ const dir = createRecoverySocketDir();+ const path = recoverySocketPathIn(dir);+ server = await startRecoverySocketServer({+ socketPath: path,+ snapshot: () => null,+ onCommand: () => {},+ tickMs: 0,+ });+ expect(statSync(dirname(path)).mode & 0o777).toBe(0o700);+ expect(statSync(path).mode & 0o777).toBe(0o600);+ await server.close();+ server = null;+ expect(existsSync(path)).toBe(false);+ expect(existsSync(dir)).toBe(false);+ });++ test("two directories from two calls never collide", async () => {+ const a = createRecoverySocketDir();+ const b = createRecoverySocketDir();+ expect(a).not.toBe(b);+ // 24 hex characters of randomness — not the proxy port, which is printed at+ // startup and would make the path guessable by something not looking for us.+ expect(a).toMatch(/^\/tmp\/claudish-recovery-[0-9a-f]{24}$/);+ });++ test("disconnects are reported, and broadcasting afterwards does not throw", async () => {+ const path = recoverySocketPathIn(createRecoverySocketDir());+ let gone = 0;+ server = await startRecoverySocketServer({+ socketPath: path,+ snapshot: () => null,+ onCommand: () => {},+ onDisconnect: () => {+ gone++;+ },+ tickMs: 0,+ });+ const client = await rawClient(path);+ await settle();+ expect(server.clientCount()).toBe(1);+ client.end();+ await settle();+ expect(gone).toBe(1);+ expect(server.clientCount()).toBe(0);+ server.broadcast(EPISODE); // must be a no-op, not a throw+ });+});diff --git a/packages/cli/src/recovery/socket-server.ts b/packages/cli/src/recovery/socket-server.tsnew file mode 100644index 00000000..91374b1b--- /dev/null+++ b/packages/cli/src/recovery/socket-server.ts@@ -0,0 +1,266 @@+/**+ * The proxy half of the recovery link: an NDJSON server on a unix socket.+ *+ * WHY A SOCKET AND NOT THE MAGMUX CHANNEL. magmux's `send` writes KEYSTROKES+ * into a pane, and the receiving program reads them as user input. Using it as+ * a data bus would make the renderer parse one stream for both proxy state and+ * the user's `[r]`/`[q]` — two protocols on one fd, with this repo's own+ * paste-detection trap waiting at the bottom of it.+ *+ * WHY NOT A FILE. No poller exists to read one (`docs/usage/magmux.md`'s 500 ms+ * status-bar poller was removed in `168c814` and the docs were never+ * corrected), building one buys 500 ms granularity and whole-file rewrite+ * races, and decisively there is no upstream path for `[r] try now`.+ *+ * WHY `/tmp` AND NOT `~/.claudish`. macOS `sun_path` is 104 bytes, `$TMPDIR` on+ * macOS is `/var/folders/<28 chars>/T/`, and `~` is longer still for many+ * users. `/tmp` is also magmux's own convention for its control socket.+ *+ * WHAT THIS DEFENDS AND WHAT IT DOES NOT. The directory is 0700 with an+ * unguessable name and the socket is 0600, which excludes other uids. It is NOT+ * a boundary against a SAME-UID process and no unix socket can be: that process+ * can read our argv, read the pane's argv, or attach a debugger. Saying+ * otherwise would be theatre. What is defended is the consequence — a forged+ * client cannot manufacture the one forbidden state (a retryable status with no+ * banner), because the lease additionally requires that the proxy ITSELF opened+ * a pane, which is a fact no client can cause.+ */++import { randomBytes } from "node:crypto";+import { chmodSync, existsSync, mkdirSync, rmSync, unlinkSync } from "node:fs";+import { type Server, type Socket, createServer } from "node:net";+import { dirname, join } from "node:path";+import { log } from "../logger.js";+import {+ FRAME_TICK_MS,+ RECOVERY_PROTOCOL_VERSION,+ type RecoveryCommand,+ type RecoveryEpisodeFrame,+ type RecoveryFrame,+ encodeLine,+ parseCommand,+} from "./types.js";++export interface RecoverySocketClientHandle {+ readonly id: number;+ /** The pid the client claimed in its `hello`, once it has sent one. */+ pid: number | null;+ send(frame: RecoveryFrame): void;+ close(): void;+}++export interface RecoverySocketServerOptions {+ socketPath: string;+ /**+ * The frame a NEWLY CONNECTED client must see immediately, and the frame the+ * tick broadcasts. Null means there is nothing live to paint.+ *+ * REPLAY-ON-CONNECT is what makes a late pane correct rather than blank. The+ * pane starts roughly 100 ms after `open_pane` and the episode began before+ * that; without a replay its first paint would be empty until the next tick,+ * which is the first thing a user sees of this feature.+ */+ snapshot: () => RecoveryEpisodeFrame | null;+ onCommand: (cmd: RecoveryCommand, client: RecoverySocketClientHandle) => void;+ onDisconnect?: (client: RecoverySocketClientHandle) => void;+ /**+ * Broadcast cadence. A tick fires in EVERY live state, not only while+ * `waiting`: the banner has to say "connecting…" during a 45 s attempt, and+ * the pane's dead-proxy heuristic is frame silence, so a quiet `attempting`+ * phase would make a healthy proxy look dead to its own renderer.+ */+ tickMs?: number;+}++export interface RecoverySocketServer {+ readonly path: string;+ broadcast(frame: RecoveryFrame): void;+ clientCount(): number;+ close(): Promise<void>;+}++/**+ * A private directory for one process's recovery socket.+ *+ * The name carries 96 bits of randomness rather than the proxy port, which is+ * printed at startup and would make the path guessable — and while a same-uid+ * process can find it anyway, an unguessable name costs one line and removes+ * the only version of this that is trivially findable by something that is NOT+ * looking for us.+ */+export function createRecoverySocketDir(): string {+ const dir = join("/tmp", `claudish-recovery-${randomBytes(12).toString("hex")}`);+ mkdirSync(dir, { mode: 0o700, recursive: true });+ return dir;+}++/** The socket path inside a directory from `createRecoverySocketDir()`. */+export function recoverySocketPathIn(dir: string): string {+ return join(dir, "r.sock");+}++/** Remove the socket and, if we made it, its private directory. Never throws. */+export function cleanupRecoverySocket(socketPath: string): void {+ try {+ if (existsSync(socketPath)) unlinkSync(socketPath);+ } catch {+ /* already gone */+ }+ try {+ const dir = dirname(socketPath);+ if (dir.startsWith("/tmp/claudish-recovery-")) rmSync(dir, { recursive: true, force: true });+ } catch {+ /* already gone */+ }+}++export async function startRecoverySocketServer(+ opts: RecoverySocketServerOptions+): Promise<RecoverySocketServer> {+ const { socketPath } = opts;+ const tickMs = opts.tickMs ?? FRAME_TICK_MS;+ const clients = new Map<number, { socket: Socket; handle: RecoverySocketClientHandle }>();+ let nextId = 1;+ let closed = false;++ // A stale socket file from a SIGKILLed predecessor makes `listen` fail with+ // EADDRINUSE even though nothing is listening. The directory name is unique+ // per process, so anything already at this path is debris by construction.+ try {+ if (existsSync(socketPath)) unlinkSync(socketPath);+ } catch {+ /* best effort */+ }++ const server: Server = createServer((socket: Socket) => {+ if (closed) {+ socket.destroy();+ return;+ }+ const id = nextId++;+ const handle: RecoverySocketClientHandle = {+ id,+ pid: null,+ send(frame: RecoveryFrame) {+ if (socket.destroyed) return;+ try {+ socket.write(encodeLine(frame));+ } catch {+ /* the pane went away mid-write; the close handler cleans up */+ }+ },+ close() {+ try {+ socket.end();+ } catch {+ /* already gone */+ }+ },+ };+ clients.set(id, { socket, handle });++ let buf = "";+ socket.on("data", (chunk: Buffer) => {+ buf += chunk.toString("utf-8");+ // Bound the buffer. A client that never sends a newline must not be able+ // to grow the proxy's heap without limit.+ if (buf.length > 64 * 1024) buf = buf.slice(-1024);+ let nl = buf.indexOf("\n");+ while (nl >= 0) {+ const line = buf.slice(0, nl).trim();+ buf = buf.slice(nl + 1);+ nl = buf.indexOf("\n");+ if (!line) continue;+ const cmd = parseCommand(line);+ if (!cmd) continue;+ if (cmd.type === "hello") {+ handle.pid = typeof cmd.pid === "number" ? cmd.pid : null;+ if (cmd.protocol !== RECOVERY_PROTOCOL_VERSION) {+ // Tell it, then drop it. A renderer that cannot read our frames must+ // print one line about version skew rather than paint garbage.+ handle.send({+ v: RECOVERY_PROTOCOL_VERSION,+ type: "closed",+ episodeId: "",+ outcome: "protocol_mismatch",+ });+ handle.close();+ return;+ }+ }+ opts.onCommand(cmd, handle);+ }+ });++ const gone = () => {+ if (!clients.delete(id)) return;+ opts.onDisconnect?.(handle);+ };+ socket.on("error", gone);+ socket.on("close", gone);+ socket.on("end", gone);++ // REPLAY. Before anything else this client could possibly ask for.+ const current = opts.snapshot();+ if (current) handle.send(current);+ });++ await new Promise<void>((resolve, reject) => {+ server.once("error", reject);+ server.listen(socketPath, () => {+ server.removeListener("error", reject);+ resolve();+ });+ });++ try {+ chmodSync(socketPath, 0o600);+ } catch {+ /* the 0700 directory is the real gate; a chmod failure does not open it */+ }++ let ticker: ReturnType<typeof setInterval> | null = null;+ if (tickMs > 0) {+ ticker = setInterval(() => {+ if (closed || clients.size === 0) return;+ const frame = opts.snapshot();+ if (!frame) return;+ for (const { handle } of clients.values()) handle.send(frame);+ }, tickMs);+ // The proxy must still be able to exit. A recovery ticker that holds the+ // event loop open would turn every recovered outage into a hung process.+ ticker.unref?.();+ }++ const onExit = () => {+ cleanupRecoverySocket(socketPath);+ };+ process.on("exit", onExit);++ log(`[Recovery] socket listening at ${socketPath}`);++ return {+ path: socketPath,+ broadcast(frame: RecoveryFrame) {+ if (closed) return;+ for (const { handle } of clients.values()) handle.send(frame);+ },+ clientCount: () => clients.size,+ async close() {+ if (closed) return;+ closed = true;+ if (ticker) clearInterval(ticker);+ for (const { socket } of clients.values()) {+ try {+ socket.destroy();+ } catch {+ /* already gone */+ }+ }+ clients.clear();+ await new Promise<void>((resolve) => server.close(() => resolve()));+ process.removeListener("exit", onExit);+ cleanupRecoverySocket(socketPath);+ },+ };+}diff --git a/packages/cli/src/recovery/types.test.ts b/packages/cli/src/recovery/types.test.tsnew file mode 100644index 00000000..6a2f8378--- /dev/null+++ b/packages/cli/src/recovery/types.test.ts@@ -0,0 +1,101 @@+/**+ * The wire contract.+ *+ * Worth its own file because both processes depend on it and NEITHER can+ * detect a breakage in the other: a proxy that emits a frame the pane silently+ * drops shows a blank banner, and a pane that sends a command the proxy silently+ * drops holds no lease. Both failures are silent by construction, so the+ * parsers' refusals are what has to be pinned.+ */++import { describe, expect, test } from "bun:test";+import {+ RECOVERY_PROTOCOL_VERSION,+ type RecoveryEpisodeFrame,+ UI_HEARTBEAT_MS,+ UI_LEASE_MS,+ encodeLine,+ parseCommand,+ parseFrame,+} from "./types.js";++const frame = (over: Partial<RecoveryEpisodeFrame> = {}): RecoveryEpisodeFrame => ({+ v: RECOVERY_PROTOCOL_VERSION,+ type: "episode",+ episodeId: "ep-1",+ state: "waiting",+ tier: 1,+ providerDisplayName: "Openai-codex",+ host: "chatgpt.com",+ endpoint: "https://chatgpt.com/backend-api/codex/responses",+ kind: "unreachable",+ code: "ConnectionClosed",+ loopback: false,+ reason: "Cannot reach Openai-codex at https://chatgpt.com/backend-api/codex/responses.",+ attempts: 5,+ clientRetries: 0,+ startedAtMs: 1_789_234_567_890,+ nextAttemptAtMs: 1_789_234_627_890,+ lastOutcome: "ConnectionClosed after 11ms",+ waiters: 2,+ otherEpisodes: 0,+ ...over,+});++describe("recovery wire contract", () => {+ test("an episode frame round-trips through one NDJSON line", () => {+ const line = encodeLine(frame());+ expect(line.endsWith("\n")).toBe(true);+ expect(line.indexOf("\n")).toBe(line.length - 1); // exactly one, at the end+ expect(parseFrame(line)).toEqual(frame());+ });++ test("a frame from a DIFFERENT protocol version is refused, not guessed at", () => {+ const skewed = encodeLine({ ...frame(), v: 2 });+ expect(parseFrame(skewed)).toBeNull();+ });++ test("malformed JSON and unknown types parse to null rather than throwing", () => {+ expect(parseFrame("{not json")).toBeNull();+ expect(parseFrame("null")).toBeNull();+ expect(parseFrame(JSON.stringify({ v: 1, type: "wat", episodeId: "x" }))).toBeNull();+ expect(parseFrame(JSON.stringify({ v: 1, type: "episode" }))).toBeNull(); // no episodeId+ });++ test("`hello` parses even at the WRONG version — the mismatch must be answerable", () => {+ // The server cannot tell a skewed client it is skewed if the one message it+ // needs to answer is the one it drops.+ const cmd = parseCommand(JSON.stringify({ v: 99, type: "hello", protocol: 99, pid: 7 }));+ expect(cmd).not.toBeNull();+ expect(cmd?.type).toBe("hello");+ });++ test("every OTHER command is version-gated", () => {+ expect(parseCommand(JSON.stringify({ v: 2, type: "ack", episodeId: "ep-1" }))).toBeNull();+ expect(parseCommand(JSON.stringify({ v: 2, type: "retry_now", episodeId: "ep-1" }))).toBeNull();+ expect(parseCommand(JSON.stringify({ v: 2, type: "bye", reason: "user_quit" }))).toBeNull();+ });++ test("ack, retry_now and bye parse at the right version", () => {+ expect(+ parseCommand(JSON.stringify({ v: 1, type: "ack", episodeId: "ep-1", paintedAt: 5 }))+ ).toEqual({ v: 1, type: "ack", episodeId: "ep-1", paintedAt: 5 });+ expect(parseCommand(JSON.stringify({ v: 1, type: "retry_now", episodeId: "ep-1" }))).toEqual({+ v: 1,+ type: "retry_now",+ episodeId: "ep-1",+ });+ expect(parseCommand(JSON.stringify({ v: 1, type: "bye", reason: "user_quit" }))).toEqual({+ v: 1,+ type: "bye",+ reason: "user_quit",+ });+ });++ test("the heartbeat leaves an order of magnitude of headroom under the lease", () => {+ // Nine consecutive dropped heartbeats must be survivable. If these two ever+ // come within 2x of each other, one slow paint revokes a live banner's+ // lease and the exhaustion arm answers as though nothing were on screen.+ expect(UI_LEASE_MS / UI_HEARTBEAT_MS).toBeGreaterThanOrEqual(10);+ });+});diff --git a/packages/cli/src/recovery/types.ts b/packages/cli/src/recovery/types.tsnew file mode 100644index 00000000..790f2999--- /dev/null+++ b/packages/cli/src/recovery/types.ts@@ -0,0 +1,228 @@+/**+ * The recovery wire contract — the ONE module both processes import.+ *+ * It has no imports of its own, deliberately. The proxy links it into a Hono+ * request path that has already loaded the whole adapter stack; the pane links+ * it into a process whose entire job is to paint five lines and whose startup+ * cost is on the critical path of a user already having a bad minute. Anything+ * this file imported would be paid for twice, and the second time for nothing.+ *+ * FRAMES ARE FAT ON PURPOSE. Every `episode` frame carries the whole banner —+ * provider, host, endpoint, reason sentence, counters, the next-attempt instant+ * — rather than a delta or an id to look up. The pane has nothing to call back+ * to: there is no request/response direction from renderer to proxy beyond the+ * three commands below, and adding one would mean the banner could be blank+ * while a round trip was in flight. A frame is ~400 bytes at 1 Hz.+ *+ * EVERY TIMESTAMP ON THE WIRE IS EPOCH MS (`Date.now()`). The two processes+ * have different `performance.now()` origins, so a process-ms value would+ * render as an arbitrary number in the pane with no error anywhere. The rule+ * that makes this checkable: a field crossing this boundary is named `…AtMs`+ * and is epoch; a field used in an interval comparison is named `…AtPerf` and+ * never leaves its process.+ */++/**+ * Bumped when a frame or command changes shape incompatibly.+ *+ * Both processes are the same binary in practice, but not always: a+ * `team --grid` launches N claudish processes that may be different builds+ * against one magmux, and a global install can be mid-update. A mismatched+ * client is told so and prints one line, instead of rendering a banner with+ * fields it does not understand.+ */+export const RECOVERY_PROTOCOL_VERSION = 1;++/** The live states. `recovered`/`abandoned` arrive as a `closed` frame instead. */+export type RecoveryFrameState = "attempting" | "waiting" | "handoff";++export interface RecoveryEpisodeFrame {+ v: number;+ type: "episode";+ episodeId: string;+ state: RecoveryFrameState;+ /** DISPLAY ONLY — the highest tier any waiter in this episode has reached. */+ tier: 1 | 2;+ providerDisplayName: string;+ /** `chatgpt.com` — the host alone, for the banner's first line. */+ host: string;+ /** The endpoint that actually failed; the AUTH host on an auth-path failure. */+ endpoint: string;+ kind: "dns" | "refused" | "unreachable";+ code: string | null;+ loopback: boolean;+ /** `buildConnectionErrorMessage(...)` verbatim — the SAME sentence the inline 400 carries. */+ reason: string;+ attempts: number;+ clientRetries: number;+ /** EPOCH ms. */+ startedAtMs: number;+ /** EPOCH ms, or null while an attempt is actually in flight. */+ nextAttemptAtMs: number | null;+ lastOutcome: string;+ waiters: number;+ /** How many OTHER live episodes exist, so the banner can say `+N more`. */+ otherEpisodes: number;+}++export type RecoveryClosedOutcome =+ | "recovered"+ | "handoff"+ | "client_gone"+ | "gave_up"+ | "grace_expired"+ | "shutdown"+ | "protocol_mismatch";++export interface RecoveryClosedFrame {+ v: number;+ type: "closed";+ episodeId: string;+ outcome: RecoveryClosedOutcome;+}++export type RecoveryFrame = RecoveryEpisodeFrame | RecoveryClosedFrame;++/** Version negotiation only. It grants NOTHING — see `ack`. */+export interface RecoveryHelloCommand {+ v: number;+ type: "hello";+ protocol: number;+ pid: number;+}++/**+ * The lease heartbeat, and the single most consequential message in the+ * protocol.+ *+ * It is sent AFTER PAINTING, and then once per heartbeat interval for as long+ * as the renderer is still painting THIS `episodeId` — whether or not a frame+ * arrived in the meantime. It is not a frame receipt. Renewing the lease from+ * frame arrivals ties the renderer's apparent liveness to the PROXY's emission+ * cadence, and a connect against an unreachable host takes 20–75 s (measured:+ * `192.0.2.1` = 75 005 ms) during which the proxy has nothing new to say. The+ * lease would then be invalid at exhaustion for exactly the failure class this+ * feature exists for, with a live, painted banner on screen.+ */+export interface RecoveryAckCommand {+ v: number;+ type: "ack";+ episodeId: string;+ /** EPOCH ms of the paint this ack attests to. */+ paintedAt: number;+}++/** `[r]` — collapse the current wait. Ignored unless it names the live episode. */+export interface RecoveryRetryNowCommand {+ v: number;+ type: "retry_now";+ episodeId: string;+}++/** `[q]` — the user said stop. Every live episode gives up and answers inline. */+export interface RecoveryByeCommand {+ v: number;+ type: "bye";+ reason: string;+}++export type RecoveryCommand =+ | RecoveryHelloCommand+ | RecoveryAckCommand+ | RecoveryRetryNowCommand+ | RecoveryByeCommand;++/** One NDJSON line, newline included. */+export function encodeLine(value: RecoveryFrame | RecoveryCommand): string {+ return `${JSON.stringify(value)}\n`;+}++function isRecord(value: unknown): value is Record<string, unknown> {+ return typeof value === "object" && value !== null;+}++/**+ * Parse one NDJSON line into a frame, or null.+ *+ * Null for malformed JSON, for an unknown `type`, and for a `v` this build does+ * not speak — a reader that guesses at an unknown shape is how a version skew+ * becomes a rendering bug instead of a one-line notice.+ */+export function parseFrame(line: string): RecoveryFrame | null {+ let parsed: unknown;+ try {+ parsed = JSON.parse(line);+ } catch {+ return null;+ }+ if (!isRecord(parsed)) return null;+ if (parsed.v !== RECOVERY_PROTOCOL_VERSION) return null;+ if (parsed.type === "episode" && typeof parsed.episodeId === "string") {+ return parsed as unknown as RecoveryEpisodeFrame;+ }+ if (parsed.type === "closed" && typeof parsed.episodeId === "string") {+ return parsed as unknown as RecoveryClosedFrame;+ }+ return null;+}++/**+ * Parse one NDJSON line into a command.+ *+ * A `hello` with the WRONG protocol still parses — the server has to be able to+ * tell a mismatched client so, and it cannot do that if the message it needs to+ * answer is the one it drops. Every other command is version-gated.+ */+export function parseCommand(line: string): RecoveryCommand | null {+ let parsed: unknown;+ try {+ parsed = JSON.parse(line);+ } catch {+ return null;+ }+ if (!isRecord(parsed)) return null;+ if (parsed.type === "hello" && typeof parsed.protocol === "number") {+ return parsed as unknown as RecoveryHelloCommand;+ }+ if (parsed.v !== RECOVERY_PROTOCOL_VERSION) return null;+ if (parsed.type === "ack" && typeof parsed.episodeId === "string") {+ return parsed as unknown as RecoveryAckCommand;+ }+ if (parsed.type === "retry_now" && typeof parsed.episodeId === "string") {+ return parsed as unknown as RecoveryRetryNowCommand;+ }+ if (parsed.type === "bye") {+ return { v: RECOVERY_PROTOCOL_VERSION, type: "bye", reason: String(parsed.reason ?? "") };+ }+ return null;+}++// ─── Timings shared by both processes ────────────────────────────────────────++/**+ * A lease lives this long after the last heartbeat naming its episode.+ *+ * NEVER compare this against a per-attempt connect cap. They measure unrelated+ * things — one the renderer's liveness, the other a network ceiling — and the+ * whole fix for the round-2 CRITICAL is that they are independent.+ */+export const UI_LEASE_MS = 10_000;++/** The pane heartbeats this often while painting. 10× headroom under the lease. */+export const UI_HEARTBEAT_MS = 1_000;++/** The proxy emits an `episode` frame this often in EVERY live state. */+export const FRAME_TICK_MS = 1_000;++/**+ * No frame for this long and the pane says so.+ *+ * Strictly greater than `FRAME_TICK_MS` by enough to survive a scheduler hiccup+ * and a slow paint: the notice means "the proxy is gone", and saying that while+ * a healthy proxy is mid-connect is the symptom the every-live-state tick+ * exists to prevent.+ */+export const PROXY_SILENT_MS = 6_000;++/** Keep the pane up this long after the last episode ends, so it cannot flap. */+export const PANE_LINGER_MS = 30_000;diff --git a/packages/cli/src/team-grid.ts b/packages/cli/src/team-grid.tsindex f8b472d0..23d81306 100644--- a/packages/cli/src/team-grid.ts+++ b/packages/cli/src/team-grid.ts@@ -1,10 +1,9 @@import { spawn } from "node:child_process";-import { execSync } from "node:child_process";import { existsSync, readFileSync, writeFileSync } from "node:fs";import { type Socket, connect as netConnect } from "node:net";-import { dirname, join } from "node:path";+import { join } from "node:path";import { setTimeout as wait } from "node:timers/promises";-import { fileURLToPath } from "node:url";+import { findMagmuxBinary } from "./launcher/magmux-binary.js";import { loadConfig, loadLocalConfig } from "./profile-config.js";import { parseModelSpec } from "./providers/model-parser.js";import {@@ -175,53 +174,6 @@ function buildPaneHeader(model: string, prompt: string, bg: string): string {return lines.join(" ");}-// ─── Multiplexer Binary Detection ────────────────────────────────────────────--/**- * Find the magmux binary. Priority:- * 1. Bundled magmux (native/magmux-<platform>-<arch>)- * 2. Platform-specific npm package (@claudish/magmux-<platform>-<arch>)- * 3. magmux in PATH (e.g. via Homebrew)- */-function findMagmuxBinary(): string {- const thisFile = fileURLToPath(import.meta.url);- const thisDir = dirname(thisFile);- const pkgRoot = join(thisDir, "..");- const platform = process.platform;- const arch = process.arch;-- // 1. Bundled magmux (native/magmux-<platform>-<arch>)- const bundledMagmux = join(pkgRoot, "native", `magmux-${platform}-${arch}`);- if (existsSync(bundledMagmux)) return bundledMagmux;-- // 2. Platform-specific npm package (@claudish/magmux-<platform>-<arch>)- // npm installs only the matching platform's optional dep- try {- const pkgName = `@claudish/magmux-${platform}-${arch}`;- // Walk up from this file to find node_modules- let searchDir = pkgRoot;- for (let i = 0; i < 5; i++) {- const candidate = join(searchDir, "node_modules", pkgName, "bin", "magmux");- if (existsSync(candidate)) return candidate;- const parent = dirname(searchDir);- if (parent === searchDir) break;- searchDir = parent;- }- } catch {- /* not installed */- }-- // 3. magmux in PATH- try {- const result = execSync("which magmux", { encoding: "utf-8" }).trim();- if (result) return result;- } catch {- /* not in PATH */- }-- throw new Error("magmux not found. Install it:\n brew install MadAppGang/tap/magmux");-}-// ─── Magmux Event Protocol ───────────────────────────────────────────────────//// magmux pushes events over its Unix socket. We care about: