Node-based terminal manager for AI coding agents — tmux-backed terminals and parallel agent sessions as draggable nodes on an infinite pan/zoom canvas. macOS, Linux, and a browser Server Edition.
TypeScript adhdagent-orchestrationai-agentscanvasclaude-codecodexcoding-agentsdeveloper-toolselectrongeminilinuxmacosmultiplexeropencodeparallel-agentsterminalterminal-emulatorterminal-multiplexertmuxworkspace-manager
fix: say the owning connection is unreachable, not "permanent"
review8b0009eca660
diff --git a/CLAUDE.md b/CLAUDE.mdindex 03d430d24..22ac8054a 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -2510,6 +2510,17 @@ command-bearing opens; this does not add a human-confirm dialog or change mobilelegacy callers retain existing discovery; actual owning-endpoint refusals remain final.Skipped foreign candidates do not consume the three-network-attempt budget. Hook eventdelivery retains its existing independent failover policy.+ **Measured on an SSH host (2026-09-28/29):** the desktop slept, the session's tunnel socket+ stayed on disk with no listener, and the walk reached an unrelated Server Edition whose+ `control-unsupported-on-this-edition … permanent … do not retry` (and, for context reads,+ "No linked nodes") was true about that server and false about the session; the tunnel came+ back minutes later. When a foreign candidate was skipped and no owner answered, the shims now+ print `FOREIGN_ENDPOINT_HINT` — the owning connection is unreachable, the state is temporary,+ the usual cause for an SSH project is the tunnel — and all four agent-facing bodies quote its+ lead via `ownerUnreachableGuidanceLines`, because their other refusal lines rightly say "do not+ retry". `src/server/control-owner-tunnel-down.test.ts` rebuilds that host under real `/bin/sh`+ with the real Server Edition handlers as the foreign endpoint; removing the owner guard+ reproduces the incident's exact refusal line.- **Every generated sh client walks the SAME endpoint failover** (`nt_candidates`/`nt_adopt`,`core/agents/hook-endpoint-failover-sh.ts`) — issue #445, the endpoint-level twin of #384: asession is pinned for life to the endpoint PATH it got at tmux creation, so an appdiff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex 9042b1e8a..cd3e622bf 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -1055,3 +1055,5 @@ Control and linked-context fallback must keep a known node identity on its ownininstance. A foreign server being reachable is not evidence that it owns the canvas. Match thecandidate's own advertised node token before sending; never borrow a token from a global directoryfor that check. Keep real owning-endpoint refusals final and legacy hook delivery unchanged.+When no owner answers, say so as a temporary state (`FOREIGN_ENDPOINT_HINT`); never let another+instance's "permanent, do not retry" stand in for a dropped tunnel.diff --git a/src/core/agents/hook-endpoint-failover-sh.ts b/src/core/agents/hook-endpoint-failover-sh.tsindex 218ef88de..cd38afa08 100644--- a/src/core/agents/hook-endpoint-failover-sh.ts+++ b/src/core/agents/hook-endpoint-failover-sh.ts@@ -140,5 +140,35 @@ nt_adopt_for_node() {}`+/**+ * What the shims print when the walk skipped at least one foreign endpoint and nothing that owns+ * this node answered. Measured on an SSH host (2026-09-28/29): the desktop slept, its reverse+ * tunnel's socket stayed on disk with no listener, and the walk used to reach an unrelated Server+ * Edition whose reply — "permanent on this host … do not retry" — was true about that server and+ * false about this session. An agent that reads a permanent refusal stops for good; the tunnel came+ * back minutes later. So this sentence says the three things that are actually known: the OWNER is+ * unreachable, the other endpoints were not asked, and the state is temporary. The tunnel is named+ * as the usual cause for an SSH project, not asserted: the shim cannot see why a socket is silent.+ *+ * `OWNER_UNREACHABLE_LEAD` is quoted verbatim by the agent-facing bodies (see+ * `ownerUnreachableGuidanceLines`), so the docs and the script cannot drift apart.+ */+export const OWNER_UNREACHABLE_LEAD = 'The nodeterm connection that owns this node is unreachable.'+export const FOREIGN_ENDPOINT_HINT =- 'Unrelated endpoints were skipped because they do not advertise this node\'s current identity. The owning nodeterm connection must be available; another local server cannot control this canvas.'+ `${OWNER_UNREACHABLE_LEAD} Other nodeterm endpoints on this machine were skipped: they do not own ` ++ 'this node, so their answer would describe a different canvas. This is temporary — for an SSH ' ++ "project it usually means the desktop's reverse tunnel is down (the desktop is asleep, offline or " ++ 'reconnecting). Retry the same command after it reconnects.'++/** The agent-facing half of FOREIGN_ENDPOINT_HINT, rendered into all four bodies (canvas skill ++ * instructions block, context skill + instructions block). Without it an agent has only the+ * bodies' other refusal lines to go on, and several of those correctly say "do not retry". */+export function ownerUnreachableGuidanceLines(): string[] {+ return [+ `Owner unreachable: if a call fails with "${OWNER_UNREACHABLE_LEAD.replace(/\.$/, '')}", the`,+ 'nodeterm app that owns this session is not answering right now; for an SSH project that usually',+ "means the desktop's reverse tunnel is down (asleep, offline or reconnecting). This is temporary:",+ 'retry the same command later rather than giving up.'+ ]+}diff --git a/src/core/canvas-control-core.ts b/src/core/canvas-control-core.tsindex 71e505a64..02df72aa1 100644--- a/src/core/canvas-control-core.ts+++ b/src/core/canvas-control-core.ts@@ -8,7 +8,8 @@ import {HOOK_ENDPOINT_FALLBACK_SH,OWNED_ENDPOINT_FALLBACK_SH,FOREIGN_ENDPOINT_HINT,- STALE_ENDPOINT_HINT+ STALE_ENDPOINT_HINT,+ ownerUnreachableGuidanceLines} from './agents/hook-endpoint-failover-sh'import { codexSandboxGuidanceLines } from './context-link-core'import { NODE_TOKEN_READ_SH } from './agents/node-token-sh'@@ -677,6 +678,8 @@ export function buildCanvasControlInstructions(shimPath: string): string {'',...browserGuidanceLines(),'',+ ...ownerUnreachableGuidanceLines(),+ '',...codexSandboxGuidanceLines(CONTROL_UNREACHABLE_MSG),'','Orchestration ("Build with Nodeterm orchestration"): first decide what is genuinely',@@ -1245,6 +1248,8 @@ Notes:toggle the kanban view. They need no confirmation.- If the CLI says canvas control is unavailable, you are not in a controllable nodeterm session — do not retry.+${ownerUnreachableGuidanceLines().join('\n')}+${codexSandboxGuidanceLines(CONTROL_UNREACHABLE_MSG).join('\n')}To orchestrate a team: decide the roles + a concrete starting prompt for each, then onediff --git a/src/core/context-link-core.test.ts b/src/core/context-link-core.test.tsindex 2daca8e88..3797151b3 100644--- a/src/core/context-link-core.test.ts+++ b/src/core/context-link-core.test.ts@@ -11,6 +11,7 @@ import {CONTEXT_UNREACHABLE_MSG} from './context-link-core'import { CODEX_SANDBOX_BLOCKED_LINE } from './agents/hook-sandbox-hint-sh'+import { FOREIGN_ENDPOINT_HINT, OWNER_UNREACHABLE_LEAD } from './agents/hook-endpoint-failover-sh'describe('buildLinkDoc', () => {it('enriches each link with tmux name, injected transcript path, and cwd', () => {@@ -221,6 +222,17 @@ describe('buildLinkedContextInstructions', () => {}})+ // Same pin as canvas-control-core.test.ts: the context shim printed the foreign instance's+ // "No linked nodes" on 2026-09-28/29, and the bodies tell an agent not to retry THAT. The+ // owner-unreachable sentence that replaces it must be taught as temporary, in its own words.+ it('both agent-facing texts teach the owner-unreachable failure as temporary', () => {+ expect(CONTEXT_SHIM_SCRIPT).toContain(`echo "${FOREIGN_ENDPOINT_HINT}" >&2`)+ for (const body of [buildContextLinkSkillBody('/x/context.sh'), buildLinkedContextInstructions('/x/context.sh')]) {+ expect(body).toContain(OWNER_UNREACHABLE_LEAD.replace(/\.$/, ''))+ expect(body).toMatch(/This is temporary: *\n?retry the same *\n?command later/)+ }+ })+it('the shim embeds the sandbox hint, keeping the generic sentence for the non-sandbox case', () => {expect(CONTEXT_SHIM_SCRIPT).toContain('nt_codex_sandbox_hint() {')expect(CONTEXT_SHIM_SCRIPT).toContain('nt_codex_sandbox_hint && exit 1')diff --git a/src/core/context-link-core.ts b/src/core/context-link-core.tsindex a3be8fa78..9f76e8ed7 100644--- a/src/core/context-link-core.ts+++ b/src/core/context-link-core.ts@@ -9,7 +9,8 @@ import {HOOK_ENDPOINT_FALLBACK_SH,OWNED_ENDPOINT_FALLBACK_SH,FOREIGN_ENDPOINT_HINT,- STALE_ENDPOINT_HINT+ STALE_ENDPOINT_HINT,+ ownerUnreachableGuidanceLines} from './agents/hook-endpoint-failover-sh'import { NODE_TOKEN_READ_SH } from './agents/node-token-sh'import { codexThreadIdentityResolverSh } from './codex-thread-identity-sh'@@ -100,6 +101,8 @@ export function buildLinkedContextInstructions(shimPath: string): string {'Only meaningful inside nodeterm (NODETERM_NODE_ID set) with a linked edge. If the CLI','says "Not a nodeterm session" or "No linked nodes", there is nothing to read — do not retry.','',+ ...ownerUnreachableGuidanceLines(),+ '',...codexSandboxGuidanceLines(CONTEXT_UNREACHABLE_MSG)].join('\n')}@@ -362,6 +365,8 @@ since it was first linked).from \`list\`. If the CLI says "Not a nodeterm session" or "No linked nodes", there is nothingto read — do not retry.+${ownerUnreachableGuidanceLines().join('\n')}+${codexSandboxGuidanceLines(CONTEXT_UNREACHABLE_MSG).join('\n')}`}diff --git a/src/main/canvas-control-core.test.ts b/src/main/canvas-control-core.test.tsindex 0b270f626..e1a01823d 100644--- a/src/main/canvas-control-core.test.ts+++ b/src/main/canvas-control-core.test.ts@@ -13,6 +13,7 @@ import {CODEX_SANDBOX_RETRY_LINE} from '../core/agents/hook-sandbox-hint-sh'import { RETRYABLE } from '../core/agents/agent-message-decide'+import { FOREIGN_ENDPOINT_HINT, OWNER_UNREACHABLE_LEAD } from '../core/agents/hook-endpoint-failover-sh'import { PROJECT_TARGETABLE_VERBS } from '../core/project-grants'import { DRY_RUN_VERBS } from '../shared/control-verbs'import {@@ -402,6 +403,20 @@ describe('parseControlRequest', () => {}})+ // The owner-unreachable sentence (2026-09-28/29): the shim prints it when the failover skipped+ // an endpoint that does not own this node and nothing that does answered. Both bodies must+ // teach it as TEMPORARY, quoting the shim's own lead, because the bodies' other refusal lines+ // correctly say "do not retry" and an agent would otherwise file this one with them. The+ // runtime shape is proven under real /bin/sh in src/server/control-owner-tunnel-down.test.ts.+ it('both agent-facing texts teach the owner-unreachable failure as temporary', () => {+ expect(FOREIGN_ENDPOINT_HINT.startsWith(OWNER_UNREACHABLE_LEAD)).toBe(true)+ expect(CONTROL_SHIM_SCRIPT).toContain(`echo "${FOREIGN_ENDPOINT_HINT}" >&2`)+ for (const body of [buildCanvasSkillBody('/x/shim.sh'), buildCanvasControlInstructions('/tmp/nodeterm.sh')]) {+ expect(body).toContain(OWNER_UNREACHABLE_LEAD.replace(/\.$/, ''))+ expect(body).toMatch(/This is temporary: *\n?retry the same *\n?command later/)+ }+ })+it('the shim embeds the sandbox hint and its call sites in both failure tails', () => {// The fragment (one definition)...expect(CONTROL_SHIM_SCRIPT).toContain('nt_codex_sandbox_hint() {')diff --git a/src/server/control-owner-tunnel-down.test.ts b/src/server/control-owner-tunnel-down.test.tsnew file mode 100644index 000000000..a0c03223a--- /dev/null+++ b/src/server/control-owner-tunnel-down.test.ts@@ -0,0 +1,239 @@+// The 2026-09-28/29 incident, rebuilt as a fixture and run for real: the generated canvas-control+// and context-link shims, under the real /bin/sh, on an SSH host whose desktop has gone to sleep.+//+// What the host looked like, measured at the time:+// - the session's endpoint file named a per-project reverse-tunnel socket under ~/.nodeterm that+// still EXISTED but had no listener (the sshd that held it was gone with the connection);+// - its siblings — every other project's tunnel from the same desktop — were dead the same way;+// - an unrelated Server Edition ran on the same host, and its endpoint file sat in the first slot+// the failover walk tries.+// The walk reached that Server Edition, which answered `control-unsupported-on-this-edition: …+// This is permanent on this host, not a temporary failure — do not retry.` for every control verb+// and "No linked nodes" for every context read. Both answers were true about the Server Edition+// and false about this session, and the first one tells an agent to stop for good over a tunnel+// that came back minutes later.+//+// The foreign endpoint here is not a stub: it is the real HookServer, started under a+// `.nodeterm-server` data dir so it writes its own endpoint file exactly where a Server Edition+// does, answering with the handlers src/server/index.ts and src/server/context-link.ts install.+// The first test proves that, so the rest cannot pass by pointing at an endpoint that says nothing.+import { describe, it, expect, beforeAll, afterAll } from 'vitest'+import { execFile, spawn } from 'node:child_process'+import { createServer, type Server } from 'node:http'+import fs from 'node:fs'+import os from 'node:os'+import path from 'node:path'+import { promisify } from 'node:util'+import { CONTROL_SHIM_SCRIPT, CONTROL_UNREACHABLE_MSG } from '../core/canvas-control-core'+import { CONTEXT_SHIM_SCRIPT, CONTEXT_UNREACHABLE_MSG } from '../core/context-link-core'+import { FOREIGN_ENDPOINT_HINT } from '../core/agents/hook-endpoint-failover-sh'+import { hookServer } from '../core/agents/hook-server'+import { handleContextLinkRequest } from '../core/context-link'+import { initPlatform, resetPlatformForTests } from '../core/platform'+import { fakePlatform } from '../core/platform-fake'+import {+ CONTROL_UNSUPPORTED_ERROR,+ CONTROL_UNSUPPORTED_SENTENCE,+ serverEditionControlHandler+} from './control-unsupported'++const run = promisify(execFile)+const NODE_ID = 'node-desk-b'+const OWNER_TOKEN = 'owner-kid.owner-mac'++let dir = ''+let home = ''+let primaryEndpoint = ''+let primarySock = ''+let foreignEndpoint = ''+let owner: Server | null = null+const ownerRequests: { url: string; nodeToken: string | undefined }[] = []+const foreignControl: string[] = []+const foreignContext: string[] = []++/** A unix socket FILE with nobody listening: bind it in a child and SIGKILL the child, so nothing+ * runs the unlink a clean close would. That is what a dead sshd leaves behind. */+async function staleSocket(p: string): Promise<void> {+ const child = spawn(+ process.execPath,+ ['-e', "require('net').createServer().listen(process.argv[1], () => process.stdout.write('up'))", p],+ { stdio: ['ignore', 'pipe', 'inherit'] }+ )+ await new Promise<void>((resolve) => child.stdout!.once('data', () => resolve()))+ await new Promise<void>((resolve) => {+ child.once('exit', () => resolve())+ child.kill('SIGKILL')+ })+ if (!fs.statSync(p).isSocket()) throw new Error(`fixture: ${p} is not a stale socket`)+}++function tunnelEndpoint(file: string, sock: string, withTokenDir = true): void {+ fs.writeFileSync(+ file,+ `NODETERM_HOOK_SOCK='${sock}'\nNODETERM_HOOK_TOKEN='desktop-bearer'\n` ++ (withTokenDir+ ? `NODETERM_HOOK_VERSION='2'\nNODETERM_NODE_TOKEN_DIR='${path.join(home, '.nodeterm/node-tokens')}'\n`+ : 'NODETERM_HOOK_VERSION=1\n')+ )+}++beforeAll(async () => {+ // Short on purpose: socket paths must fit sun_path (104 bytes on macOS).+ dir = fs.mkdtempSync(path.join(os.tmpdir(), 'nt-td-'))+ home = path.join(dir, 'h')+ const desktop = path.join(home, '.nodeterm')+ fs.mkdirSync(path.join(desktop, 'node-tokens'), { recursive: true })+ fs.writeFileSync(path.join(desktop, 'node-tokens', NODE_ID), `${OWNER_TOKEN}\n`)++ // The session's own tunnel, and two siblings from the same desktop: one current, one pre-v2+ // (no advertised token dir, so the walk derives `<dir>/node-tokens`). All three are dead.+ primarySock = path.join(desktop, 'hook-a.sock')+ await staleSocket(primarySock)+ primaryEndpoint = path.join(desktop, 'hook-endpoint-project-a-e1b3.env')+ tunnelEndpoint(primaryEndpoint, primarySock)+ await staleSocket(path.join(desktop, 'hook-b.sock'))+ tunnelEndpoint(path.join(desktop, 'hook-endpoint-project-b.env'), path.join(desktop, 'hook-b.sock'))+ await staleSocket(path.join(desktop, 'hook-c.sock'))+ tunnelEndpoint(path.join(desktop, 'hook-endpoint-project-c.env'), path.join(desktop, 'hook-c.sock'), false)++ // The unrelated Server Edition, booted by the real code under its real data dir name.+ resetPlatformForTests()+ initPlatform(fakePlatform({ userDataDir: path.join(home, '.nodeterm-server') }))+ await hookServer.start()+ hookServer.setControlHandler(async (req) => {+ foreignControl.push(req.verb)+ return serverEditionControlHandler(req)+ })+ hookServer.setContextLinkHandler(async (req) => {+ foreignContext.push(req.verb)+ return handleContextLinkRequest(req)+ })+ foreignEndpoint = path.join(home, '.nodeterm-server', 'hook-endpoint.env')+ const advertised = fs.readFileSync(foreignEndpoint, 'utf8')+ // The shape measured on the host: a port, a token dir, and (off Windows) a unix socket — all of+ // it inside this fixture, never the developer's own ~/.nodeterm.+ expect(advertised).toContain(`NODETERM_NODE_TOKEN_DIR='${path.join(home, '.nodeterm-server/node-tokens')}'`)+ const sock = /NODETERM_HOOK_SOCK='([^']+)'/.exec(advertised)?.[1]+ if (sock) expect(sock.startsWith(dir)).toBe(true)+})++afterAll(async () => {+ hookServer.stop()+ if (owner) await new Promise<void>((r) => owner!.close(() => r()))+ fs.rmSync(dir, { recursive: true, force: true })+})++function shim(script: string, name: string): string {+ const p = path.join(dir, name)+ if (!fs.existsSync(p)) fs.writeFileSync(p, script, { mode: 0o755 })+ return p+}++/** Run a shim the way an agent on the SSH host does, with the env its tmux session was born with. */+async function call(+ script: string,+ name: string,+ args: string[],+ endpoint = primaryEndpoint+): Promise<{ code: number; stdout: string; stderr: string }> {+ try {+ const { stdout, stderr } = await run('/bin/sh', [shim(script, name), ...args], {+ timeout: 15_000,+ env: {+ PATH: process.env.PATH ?? '',+ HOME: home,+ NODETERM_CANVAS_CONTROL: '1',+ NODETERM_NODE_ID: NODE_ID,+ NODETERM_HOOK_ENDPOINT: endpoint+ }+ })+ return { code: 0, stdout, stderr }+ } catch (e) {+ const err = e as { code?: number; stdout?: string; stderr?: string }+ return { code: err.code ?? -1, stdout: err.stdout ?? '', stderr: err.stderr ?? '' }+ }+}++const control = (args: string[], endpoint?: string) => call(CONTROL_SHIM_SCRIPT, 'nodeterm.sh', args, endpoint)+const context = (args: string[], endpoint?: string) => call(CONTEXT_SHIM_SCRIPT, 'context.sh', args, endpoint)++/** The claims that made the incident expensive. None of them may reach an agent whose own+ * connection is merely down. */+function expectNoForeignVerdict(r: { stdout: string; stderr: string }): void {+ const all = `${r.stdout}\n${r.stderr}`+ expect(all).not.toContain(CONTROL_UNSUPPORTED_ERROR)+ expect(all).not.toContain(CONTROL_UNSUPPORTED_SENTENCE)+ expect(all).not.toMatch(/permanent/i)+ expect(all).not.toMatch(/do not retry/i)+ expect(all).not.toMatch(/Server Edition/)+ expect(all).not.toMatch(/No linked nodes/)+}++/** The sentence that replaces them: the owning connection is unreachable, it is temporary, and+ * the thing to wait for is the desktop's tunnel coming back. */+function expectOwnerUnreachable(stderr: string): void {+ expect(stderr).toContain(FOREIGN_ENDPOINT_HINT)+ expect(stderr).toMatch(/owns this node is unreachable/)+ expect(stderr).toMatch(/temporary/i)+ expect(stderr).toMatch(/tunnel/)+ expect(stderr).toMatch(/reconnect/)+}++describe.skipIf(process.platform === 'win32')('owning desktop tunnel down, unrelated Server Edition up', () => {+ it('the fixture is the incident: reached directly, that Server Edition answers the permanent refusal', async () => {+ const ctl = await control(['list'], foreignEndpoint)+ expect(ctl.code).toBe(1)+ expect(ctl.stderr).toContain(CONTROL_UNSUPPORTED_SENTENCE)+ const ctx = await context(['list'], foreignEndpoint)+ expect(ctx.stdout).toContain('No linked nodes')+ // Reset: everything below must reach it zero times.+ foreignControl.length = 0+ foreignContext.length = 0+ })++ it('control: skips the foreign endpoint and says the owning connection is down, not that control is unsupported', async () => {+ for (const args of [['list'], ['close', '--node', 'finished-a,finished-b']]) {+ const r = await control(args)+ expect(r.code).toBe(1)+ expectOwnerUnreachable(r.stderr)+ expect(r.stderr).toContain(CONTROL_UNREACHABLE_MSG)+ expectNoForeignVerdict(r)+ }+ expect(foreignControl).toEqual([])+ })++ it('context: skips the foreign endpoint instead of reporting its "No linked nodes"', async () => {+ const r = await context(['list'])+ expect(r.code).toBe(1)+ expectOwnerUnreachable(r.stderr)+ expect(r.stderr).toContain(CONTEXT_UNREACHABLE_MSG)+ expectNoForeignVerdict(r)+ expect(foreignContext).toEqual([])+ })++ // The advice has to be true: once the desktop reconnects (sshd rebinds the same socket path),+ // the SAME command, retried unchanged, reaches the owner and presents the owner's capability.+ it('retrying after the tunnel comes back reaches the owning desktop', async () => {+ fs.unlinkSync(primarySock)+ owner = createServer((req, res) => {+ const tok = req.headers['x-nodeterm-node-token']+ ownerRequests.push({ url: req.url ?? '', nodeToken: Array.isArray(tok) ? tok[0] : tok })+ req.resume()+ req.on('end', () => {+ res.writeHead(200, { 'content-type': 'text/plain' })+ res.end('owner reply')+ })+ })+ await new Promise<void>((r) => owner!.listen(primarySock, r))++ const ctl = await control(['close', '--node', 'finished-a,finished-b'])+ expect(ctl).toMatchObject({ code: 0, stdout: 'owner reply' })+ const ctx = await context(['list'])+ expect(ctx).toMatchObject({ code: 0, stdout: 'owner reply' })++ expect(ownerRequests.map((q) => q.url)).toEqual(['/control/close', '/context-link/list'])+ expect(ownerRequests.every((q) => q.nodeToken === OWNER_TOKEN)).toBe(true)+ expect(foreignControl).toEqual([])+ expect(foreignContext).toEqual([])+ })+})