Node-based terminal manager for AI coding agents — tmux-backed terminals and parallel agent sessions as draggable nodes on an infinite pan/zoom canvas. macOS, Linux, and a browser Server Edition.
TypeScript adhdagent-orchestrationai-agentscanvasclaude-codecodexcoding-agentsdeveloper-toolselectrongeminilinuxmacosmultiplexeropencodeparallel-agentsterminalterminal-emulatorterminal-multiplexertmuxworkspace-manager
fix(ssh): re-verify the reverse hook tunnel on a reused master
enes03564e497f63
diff --git a/CLAUDE.md b/CLAUDE.mdindex 3b2337c04..9375bdb71 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -1584,6 +1584,30 @@ else, and its context links must keep classifying across restarts).remote installers at once; a second repair mechanism would be exactly the duplicated rule thisfile warns about. It strips only OUR handler out of a definition, so a hook a user hand-mergedbeside ours survives.+- **A reused ControlMaster is not evidence of a live hook tunnel** (issue #735 — remote sessions+ stuck on **Unknown**, no completion notifications, no unread dots). `connect()`'s reuse branch+ returned the cached `hookEndpointPath` whenever `ssh -O check` answered, on the written-down+ assumption that *"a master that answered `-O check` never lost its tunnel"*. That is false, and+ the mechanism is **our own self-heal**: `childArgs` uses `ControlMaster=auto` + `ControlPersist`+ precisely so a dead master is rebuilt by the next child command (a status poll, a mirror push, a+ remote git call) on the same ControlPath. The rebuilt master answers `-O check` — and carries no+ `-R`, because `RemoteHooks.setup()` is the only caller of `hookForwardArgs` and it runs only on+ the branch where a master has just come up. The 45 s watchdog then parks on the reuse branch+ forever. Nothing reports it: the project says `connected`, terminals work, the mirror pushes, and+ only the hook POSTs die — into a socket file that still EXISTS with nobody listening.+ **MEASURED on the host that prompted the fix**: 10 per-project hook sockets on disk, exactly ONE+ with a listener (`ss -lxp`); the dead project's socket answered `curl` exit 7 while that same+ project's status mirror was being written the same second; **107 of 128** live `nodeterm-rmt`+ tmux sessions were pinned to that dead endpoint. Sessions are pinned for life+ (`new-session -A -e …` — tmux ignores `-e` on an existing session), so every one of them stayed+ dark until the app restarted. The reuse branch now probes the tunnel (`RemoteHooks.tunnelAlive`,+ one `curl` over the already-multiplexed master) and re-runs the idempotent `setup()` when it does+ not answer, firing `onTunnelVerified` so the working agents resync. **The retry is backed off**+ (`tunnel-repair.ts`, pure + tested): the FIRST failure repairs immediately — that is the common+ case — while a host that can never forward (`AllowStreamLocalForwarding no`, no `curl`, a `$HOME`+ the validator refuses) settles at one attempt per 15 minutes instead of rewriting every agent's+ hook config every 45 s. A missing spec answers "not alive" rather than "unknown": nothing of ours+ is bound, which is a tunnel that cannot deliver.- **Per-node hook identity** (`src/core/agents/node-auth-*.ts`, `node-token-*.ts`,`node-identity-policy.ts` — full write-up in **`docs/node-identity.md`**) — the shared bearer proves"a session on this machine", never *which* session, so every node also gets a capability deriveddiff --git a/src/main/remote-ssh/remote-hooks.ts b/src/main/remote-ssh/remote-hooks.tsindex 57fd4d1ae..1163794ff 100644--- a/src/main/remote-ssh/remote-hooks.ts+++ b/src/main/remote-ssh/remote-hooks.ts@@ -110,6 +110,26 @@ export class RemoteHooks {constructor(private r: RemoteRunner) {}+ /**+ * Is THIS project's reverse hook tunnel still answering? (issue #735)+ *+ * The connect() reuse branch used to assume a master that answered `-O check` still carried its+ * `-R` forward. It does not, and the mechanism is our own self-heal: `childArgs` uses+ * `ControlMaster=auto` + `ControlPersist` precisely so that when a master dies the next child+ * command rebuilds one on the same ControlPath. That rebuilt master answers `-O check` — but+ * nothing ever handed it `ssh -O forward -R`, because `setup()` is the only caller of+ * `hookForwardArgs` and it runs only on the "master just came up" branch.+ *+ * A missing spec answers FALSE: it means nothing of ours is bound for this project in this app+ * run, which is a tunnel that cannot deliver, not an unknown. The caller repairs by re-running+ * `setup()`, which is idempotent and re-verifies end-to-end.+ */+ async tunnelAlive(projectId: string, conn: SshConnection, controlPath: string, token: string): Promise<boolean> {+ const spec = this.specs.get(projectId)+ if (!spec || !token) return false+ return this.verifyTunnel(conn, controlPath, spec.sock, token)+ }+async setup(projectId: string,conn: SshConnection,diff --git a/src/main/remote-ssh/ssh-project.test.ts b/src/main/remote-ssh/ssh-project.test.tsindex 04318bb52..c0107ed6b 100644--- a/src/main/remote-ssh/ssh-project.test.ts+++ b/src/main/remote-ssh/ssh-project.test.ts@@ -1345,14 +1345,16 @@ describe('SshProjectManager', () => {* connect() takes the ordinary fresh-master path — a genuine establish. */function makeVerifiedMgr(onTunnelVerified: (projectId: string, controlPath: string, conn: SshConnection) => void,- httpCode = '204'+ /** A THUNK is what lets a test kill the tunnel between two connects — see the repair tests. */+ httpCode: string | (() => string) = '204') {vi.spyOn(fs, 'mkdir').mockResolvedValue(undefined as never)vi.spyOn(fs, 'stat').mockRejectedValue(Object.assign(new Error('ENOENT'), { code: 'ENOENT' }))const run = vi.fn(async (args: string[]) => {const j = args.join(' ')if (j.includes('$HOME')) return { code: 0, stdout: '/home/u' }- if (j.includes('%{http_code}')) return { code: 0, stdout: httpCode }+ if (j.includes('%{http_code}'))+ return { code: 0, stdout: typeof httpCode === 'function' ? httpCode() : httpCode }return { code: 0, stdout: '' }})return new SshProjectManager({@@ -1390,12 +1392,73 @@ describe('SshProjectManager', () => {expect(homeAtHookTime).toBe('/home/u')})- it('does NOT fire on the reuse branch — a live master never lost its tunnel', async () => {+ it('does NOT fire on the reuse branch while the tunnel still ANSWERS', async () => {+ // A healthy reuse must stay exactly as cheap and as quiet as it was: one `-O check`, one+ // tunnel probe, no re-install and no resync.const onTunnelVerified = vi.fn()const mgr = makeVerifiedMgr(onTunnelVerified)await mgr.connect('p1', conn, '/remote/cwd')onTunnelVerified.mockClear()- await mgr.connect('p1', conn, '/remote/cwd') // `-O check` answers → early return+ await mgr.connect('p1', conn, '/remote/cwd')+ expect(onTunnelVerified).not.toHaveBeenCalled()+ })++ it('REPAIRS a reused master whose tunnel stopped answering, and resyncs (issue #735)', async () => {+ // The pin this replaces asserted "a live master never lost its tunnel", which is false and was+ // the bug: `ControlMaster=auto` + `ControlPersist` mean the next child command rebuilds a dead+ // master on the same ControlPath, and the rebuilt one carries no `-R`. It answers `-O check`+ // all the same, so the watchdog parked on this branch forever while every remote hook POST+ // vanished into a socket file with no listener — nodes stuck on "Unknown", no notifications.+ //+ // `deadProbes` models the real sequence exactly: the establish is clean, then the tunnel dies+ // under a master that still answers `-O check`, the liveness probe finds it, and the rebuild+ // binds a fresh `-R` which DOES verify. Arming it after the establish matters — the establish+ // verifies too, and a queue primed up front would be eaten by that instead.+ const onTunnelVerified = vi.fn()+ let deadProbes = 0+ const mgr = makeVerifiedMgr(onTunnelVerified, () => (deadProbes-- > 0 ? '000' : '204'))+ await mgr.connect('p1', conn, '/remote/cwd')+ onTunnelVerified.mockClear()+ deadProbes = 1+ await mgr.connect('p1', conn, '/remote/cwd')+ expect(onTunnelVerified).toHaveBeenCalledWith('p1', controlPathFor('p1'), conn)+ })++ it('rebinds the forward on repair — the endpoint is re-advertised, not merely re-probed', async () => {+ // The whole failure is a master with no `-R`, so a repair that did not call `-O forward`+ // would leave every hook POST dying exactly as before while reporting success.+ let deadProbes = 0+ const mgr = makeVerifiedMgr(vi.fn(), () => (deadProbes-- > 0 ? '000' : '204'))+ await mgr.connect('p1', conn, '/remote/cwd')+ deadProbes = 1+ const forwards = () =>+ (mgr as unknown as { r: { run: ReturnType<typeof vi.fn> } }).r.run.mock.calls.filter(+ (c: unknown[]) => (c[0] as string[]).includes('forward')+ ).length+ const afterEstablish = forwards()+ await mgr.connect('p1', conn, '/remote/cwd')+ expect(forwards()).toBeGreaterThan(afterEstablish)+ })++ it('backs off a host that can never forward instead of re-installing every tick', async () => {+ // `setup()` rewrites the managed hook into every agent's config on the host, and the watchdog+ // reuses every 45 s. On a host where the tunnel can never bind (sshd+ // `AllowStreamLocalForwarding no`, no curl) an unthrottled repair would rewrite those files+ // forever. The FIRST failure still repairs immediately — that is the case this exists for.+ const onTunnelVerified = vi.fn()+ const mgr = makeVerifiedMgr(onTunnelVerified, () => '000')+ const forwards = () =>+ (mgr as unknown as { r: { run: ReturnType<typeof vi.fn> } }).r.run.mock.calls.filter(+ (c: unknown[]) => (c[0] as string[]).includes('forward')+ ).length+ await mgr.connect('p1', conn, '/remote/cwd')+ const afterEstablish = forwards()+ await mgr.connect('p1', conn, '/remote/cwd') // failure #1 → repair attempted+ const afterFirstRepair = forwards()+ expect(afterFirstRepair).toBeGreaterThan(afterEstablish)+ await mgr.connect('p1', conn, '/remote/cwd') // inside the backoff window → no second rebuild+ await mgr.connect('p1', conn, '/remote/cwd')+ expect(forwards()).toBe(afterFirstRepair)expect(onTunnelVerified).not.toHaveBeenCalled()})diff --git a/src/main/remote-ssh/ssh-project.ts b/src/main/remote-ssh/ssh-project.tsindex e2421a678..35dbf75be 100644--- a/src/main/remote-ssh/ssh-project.ts+++ b/src/main/remote-ssh/ssh-project.ts@@ -42,6 +42,12 @@ import {} from '../../core/remote-ssh/control-master'import { claudeVersionProbeCommand, parseClaudeVersionProbe } from '../../core/remote-ssh/claude-version-probe'import { RemoteHooks } from './remote-hooks'+import {+ recordTunnelRepair,+ shouldAttemptTunnelRepair,+ type TunnelRepairState+} from './tunnel-repair'+import { hookServer } from '../../core/agents/hook-server'import {nodeIdsForCanvas,@@ -122,10 +128,13 @@ interface Runners {/** The last SSH connection just went away through a user-facing disconnect. Production schedules* the app-private agent's shutdown, which is what "forget the key" actually means. */onIdle?: () => void- /** A project's reverse hook tunnel was just VERIFIED on a freshly established master. Production- * resyncs that project's working agents: hook events lost while the tunnel was down are gone for- * good, so a node can be stranded at `working` until the 20-minute stale sweep. Deliberately not- * called on the reuse branch — a master that answered `-O check` never lost its tunnel. The+ /** A project's reverse hook tunnel was just VERIFIED — on a freshly established master, or by the+ * reuse branch's repair. Production resyncs that project's working agents: hook events lost while+ * the tunnel was down are gone for good, so a node can be stranded at `working` until the+ * 20-minute stale sweep. This used to say it was "deliberately not called on the reuse branch —+ * a master that answered `-O check` never lost its tunnel", which was FALSE and was issue #735:+ * `ControlMaster=auto` means a dead master is rebuilt by the next child command, and the rebuilt+ * one carries no `-R`. It answers `-O check` all the same. The* `conn` rides along because the resync builds its own remote commands (the host's tmux session* list, a pane probe) and the alternative — looking the connection back up by control path —* would add a public accessor for a fact this call site already holds. */@@ -429,6 +438,64 @@ export class SshProjectManager {* respawned, 'reconnecting' status so the renderer flow engages). Interval is unref'd so it* never holds the process open; an empty conns map makes a tick a no-op.*/+ /** Consecutive failed tunnel repairs per project, so a host that can never forward is not+ * re-installed on every watchdog tick. See `tunnel-repair.ts` for why the first one is free. */+ private tunnelRepair = new Map<string, TunnelRepairState>()++ /**+ * Re-verify a REUSED master's reverse hook tunnel, and rebuild it if it stopped answering+ * (issue #735 — remote sessions stuck on "Unknown", no notifications, no unread dots).+ *+ * `-O check` answering is not evidence the tunnel is alive, and the reason is our own self-heal:+ * `childArgs` uses `ControlMaster=auto` + `ControlPersist` on purpose, so when a master dies the+ * next child command (a status poll, a mirror push, a remote git call) rebuilds a background+ * master on the same ControlPath. That rebuilt master answers `-O check` — and carries no `-R`,+ * because `setup()` is the only caller of `hookForwardArgs` and it runs only on the branch where+ * a master has just come up. The watchdog then lands on the reuse branch forever.+ *+ * Nothing reports this. The project says `connected`, terminals work, the mirror pushes; only+ * the hook POSTs die, into a socket file that still exists with nobody listening. MEASURED on+ * the host that prompted the fix: 107 of 128 live remote tmux sessions pinned to an endpoint+ * whose socket answered `curl` exit 7, while that project's status mirror was being written the+ * same second.+ *+ * Best-effort throughout, exactly like the establish path: a failed probe or a failed rebuild+ * leaves the connection alone and the project keeps working without hooks. It must never cost+ * the user a connection that is already up.+ */+ private async repairHookTunnelIfDead(projectId: string, existing: Conn): Promise<void> {+ try {+ const hook = this.r.getHook()+ // No hook server yet ⇒ nothing to point at, and `setup()` would refuse anyway.+ if (!hook?.port || !hook.token) return+ if (await this.remoteHooks.tunnelAlive(projectId, existing.conn, existing.controlPath, hook.token)) {+ this.tunnelRepair.delete(projectId)+ return+ }+ const now = Date.now()+ if (!shouldAttemptTunnelRepair(this.tunnelRepair.get(projectId), now)) return+ const res = await this.remoteHooks.setup(projectId, existing.conn, existing.controlPath, hook)+ this.tunnelRepair.set(projectId, recordTunnelRepair(this.tunnelRepair.get(projectId), !!res, now))+ if (!res) return+ // Ownership re-check: `setup()` is several round-trips, and a disconnect + reconnect inside+ // that window means this entry is no longer the live one — the same rule the establish path+ // applies before rebuilding a master. Writing the endpoint onto a superseded entry would+ // point sessions at a socket belonging to a connection nobody holds.+ if (this.conns.get(projectId) !== existing) return+ existing.hookEndpointPath = res.endpointPath+ // Same contract as the establish path: hook events lost while the tunnel was down are gone+ // for good, so the working agents need a resync. Fire-and-forget behind a catch — a repair+ // job must never surface to the user as a dead SSH project.+ try {+ this.r.onTunnelVerified?.(projectId, existing.controlPath, existing.conn)+ } catch {+ // undecided changes nothing: the stale sweep remains the backstop+ }+ } catch {+ // fail-open: the reuse returns whatever it already had, exactly as before this repair existed+ }+ }+startWatchdog(intervalMs = MASTER_WATCHDOG_MS): void {if (this.watchdog) returnthis.watchdog = setInterval(() => {@@ -577,6 +644,8 @@ export class SshProjectManager {// Keep the remote git cwd current even on an idempotent reuse (the folder may have changed).// Guard against a later connect without remoteCwd clearing a known cwd.existing.remoteCwd = remoteCwd ?? existing.remoteCwd+ // ...and CHECK THE TUNNEL, because `-O check` says nothing about it (issue #735).+ await this.repairHookTunnelIfDead(projectId, existing)return {controlPath: existing.controlPath,hookEndpointPath: existing.hookEndpointPath,diff --git a/src/main/remote-ssh/tunnel-repair.test.ts b/src/main/remote-ssh/tunnel-repair.test.tsnew file mode 100644index 000000000..c47db1bb6--- /dev/null+++ b/src/main/remote-ssh/tunnel-repair.test.ts@@ -0,0 +1,78 @@+import { describe, expect, it } from 'vitest'+import {+ TUNNEL_REPAIR_DELAYS_MS,+ recordTunnelRepair,+ shouldAttemptTunnelRepair,+ tunnelRepairDelayMs+} from './tunnel-repair'++describe('tunnelRepairDelayMs', () => {+ it('owes nothing before the first failure', () => {+ expect(tunnelRepairDelayMs(0)).toBe(0)+ expect(tunnelRepairDelayMs(-1)).toBe(0)+ })++ it('grows with consecutive failures and then CLAMPS', () => {+ expect(tunnelRepairDelayMs(1)).toBe(TUNNEL_REPAIR_DELAYS_MS[0])+ expect(tunnelRepairDelayMs(2)).toBe(TUNNEL_REPAIR_DELAYS_MS[1])+ expect(tunnelRepairDelayMs(3)).toBe(TUNNEL_REPAIR_DELAYS_MS[2])+ // A host that will never forward settles at the last delay rather than growing without bound —+ // it must keep trying (sshd config changes, curl gets installed), just not every 45 s.+ expect(tunnelRepairDelayMs(99)).toBe(TUNNEL_REPAIR_DELAYS_MS[TUNNEL_REPAIR_DELAYS_MS.length - 1])+ })++ it('is monotonic — a later failure never waits LESS than an earlier one', () => {+ for (let n = 1; n < 10; n++) {+ expect(tunnelRepairDelayMs(n + 1)).toBeGreaterThanOrEqual(tunnelRepairDelayMs(n))+ }+ })+})++describe('shouldAttemptTunnelRepair', () => {+ it('the FIRST repair is free', () => {+ // The asymmetry is the point. A needless repair costs one idempotent re-install; NOT repairing+ // costs a project whose agents report nothing for as long as it stays connected — on the host+ // that prompted this, 107 of 128 live sessions.+ expect(shouldAttemptTunnelRepair(undefined, 1_000)).toBe(true)+ expect(shouldAttemptTunnelRepair({ failures: 0, lastAttemptAt: 1_000 }, 1_000)).toBe(true)+ })++ it('refuses inside the backoff window and allows once it has elapsed', () => {+ const state = { failures: 1, lastAttemptAt: 1_000 }+ expect(shouldAttemptTunnelRepair(state, 1_000)).toBe(false)+ expect(shouldAttemptTunnelRepair(state, 1_000 + TUNNEL_REPAIR_DELAYS_MS[0] - 1)).toBe(false)+ expect(shouldAttemptTunnelRepair(state, 1_000 + TUNNEL_REPAIR_DELAYS_MS[0])).toBe(true)+ })++ it('a longer streak waits longer', () => {+ const justPastFirst = 1_000 + TUNNEL_REPAIR_DELAYS_MS[0]+ expect(shouldAttemptTunnelRepair({ failures: 1, lastAttemptAt: 1_000 }, justPastFirst)).toBe(true)+ expect(shouldAttemptTunnelRepair({ failures: 2, lastAttemptAt: 1_000 }, justPastFirst)).toBe(false)+ })++ it('a clock that went BACKWARDS refuses rather than repairing in a loop', () => {+ // NTP steps and sleep/wake both do this. Refusing is the safe direction: the window simply+ // takes longer to expire, where the other direction is an unthrottled re-install.+ expect(shouldAttemptTunnelRepair({ failures: 1, lastAttemptAt: 10_000 }, 1_000)).toBe(false)+ })+})++describe('recordTunnelRepair', () => {+ it('counts consecutive failures', () => {+ let s = recordTunnelRepair(undefined, false, 100)+ expect(s).toEqual({ failures: 1, lastAttemptAt: 100 })+ s = recordTunnelRepair(s, false, 200)+ expect(s).toEqual({ failures: 2, lastAttemptAt: 200 })+ })++ it('a SUCCESS resets the streak, so the next outage repairs immediately', () => {+ const failed = recordTunnelRepair(recordTunnelRepair(undefined, false, 100), false, 200)+ const ok = recordTunnelRepair(failed, true, 300)+ expect(ok.failures).toBe(0)+ expect(shouldAttemptTunnelRepair(ok, 300)).toBe(true)+ })++ it('stamps the ATTEMPT time, not the outcome time — the window starts when we spent the work', () => {+ expect(recordTunnelRepair(undefined, false, 4_242).lastAttemptAt).toBe(4_242)+ })+})diff --git a/src/main/remote-ssh/tunnel-repair.ts b/src/main/remote-ssh/tunnel-repair.tsnew file mode 100644index 000000000..c26b207fb--- /dev/null+++ b/src/main/remote-ssh/tunnel-repair.ts@@ -0,0 +1,61 @@+/**+ * When may the connect() reuse branch spend a full `RemoteHooks.setup()` on repairing a reverse+ * hook tunnel that stopped answering? (issue #735)+ *+ * Pure, because the decision is the whole risk. A dead tunnel is silent — the project reports+ * `connected`, terminals work, the mirror pushes — so the repair has to be automatic; but+ * `setup()` re-installs the managed hook into every agent's config on the host, and the watchdog+ * calls connect() every 45 s per project. On a host that can never bind one (sshd with+ * `AllowStreamLocalForwarding no`, no `curl`, a `$HOME` the validator refuses) an unthrottled+ * retry would rewrite those config files forever, for a tunnel that is never coming.+ *+ * So: the FIRST failure repairs immediately — that is the case this exists for, and it is the+ * common one (a master died, a child rebuilt it, and the rebuilt one carries no `-R`) — and each+ * consecutive failure backs off, capped. Success resets. Nothing here decides whether the tunnel+ * is dead; it only decides whether we are allowed to try again yet.+ */++/** Backoff after consecutive FAILED repairs. The first entry is the delay after failure #1. */+export const TUNNEL_REPAIR_DELAYS_MS = [60_000, 300_000, 900_000] as const++export interface TunnelRepairState {+ /** Consecutive failed repairs. Reset to 0 by a success. */+ failures: number+ /** When the last repair was ATTEMPTED (not when it finished). */+ lastAttemptAt: number+}++/**+ * The delay owed after `failures` consecutive failures. Clamped to the last entry, so a host that+ * will never forward settles at one attempt per 15 minutes rather than one per watchdog tick.+ */+export function tunnelRepairDelayMs(failures: number): number {+ if (failures <= 0) return 0+ const i = Math.min(failures, TUNNEL_REPAIR_DELAYS_MS.length) - 1+ return TUNNEL_REPAIR_DELAYS_MS[i]+}++/**+ * May we attempt a repair now?+ *+ * `undefined` state = nothing has failed yet for this project ⇒ yes, immediately. That is the+ * deliberate asymmetry: the cost of a needless repair is one idempotent re-install, while the cost+ * of NOT repairing is a project whose agents report nothing at all for as long as it stays+ * connected — which on the host that prompted this was 107 of 128 live sessions.+ */+export function shouldAttemptTunnelRepair(+ state: TunnelRepairState | undefined,+ now: number+): boolean {+ if (!state || state.failures <= 0) return true+ return now - state.lastAttemptAt >= tunnelRepairDelayMs(state.failures)+}++/** Fold an attempt's outcome into the state a later `shouldAttemptTunnelRepair` will read. */+export function recordTunnelRepair(+ state: TunnelRepairState | undefined,+ ok: boolean,+ now: number+): TunnelRepairState {+ return { failures: ok ? 0 : (state?.failures ?? 0) + 1, lastAttemptAt: now }+}