Skip to content

Node-based terminal manager for AI coding agents — tmux-backed terminals and parallel agent sessions as draggable nodes on an infinite pan/zoom canvas. macOS, Linux, and a browser Server Edition.

fix: bound the endpoint walk and take the owner token from the primary only

review9fad4769f763
diff --git a/CLAUDE.md b/CLAUDE.md
index 7b981b41d..56252a2c7 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -2502,25 +2502,43 @@ command-bearing opens; this does not add a human-confirm dialog or change mobile
the launcher and the prelude, pass `npm run typecheck` and every unit test, and ship INERT.
`main/codex-identity-record-wiring.test.ts` pins it at source level, the same remedy
`hook-verified-parity.test.ts` uses for the same class of hole.
- - Control/context endpoint discovery retains a known node capability. A dead Desktop SSH tunnel
- must not redirect a command to a local Server Edition that has no matching token for that node:
- its unsupported-edition response describes the wrong instance. The two shims use
- `nt_adopt_for_node` and read only the candidate's advertised token directory (or adjacent
- `node-tokens` for old endpoint files), never borrow a global token for a candidate. Unknown
- legacy callers retain existing discovery; actual owning-endpoint refusals remain final.
- Skipped foreign candidates do not consume the three-network-attempt budget. Hook event
- delivery retains its existing independent failover policy.
+ - Control/context endpoint discovery keeps a known node capability as a **routing rule, not an
+ ownership proof**. A dead Desktop SSH tunnel must not redirect a command to a local Server
+ Edition: its unsupported-edition response describes the wrong instance. The two shims use
+ `nt_adopt_for_node` (`core/agents/hook-endpoint-failover-sh.ts`). The reference value is read
+ from the PRIMARY endpoint's own token dir only — the one it advertises, else the adjacent
+ `node-tokens` — never from the global search `nt_read_node_token` walks (that search exists to
+ PRESENT a capability, #384; as a reference it let a Server Edition that opened the same
+ project.json supply the "owner's" token whenever the desktop's token write had failed). Once
+ that dir EXISTS, a candidate must hold the same value in its own dir, "both hold nothing"
+ included; only a session with no such dir at all keeps legacy discovery. What a match shows is
+ that the candidate reads the same token file for this node — on an SSH host that file is shared
+ per unix ACCOUNT (`remote-hooks.ts`, KNOWN LIMITATION), so two desktops driving one account are
+ indistinguishable here, and the receiving server still authorizes every request. Actual
+ owning-endpoint refusals remain final. Skipped foreign candidates do not consume the
+ three-attempt budget, and a skipped candidate restores the previous endpoint vars (the codex
+ sandbox hint names `$NODETERM_HOOK_SOCK` as the socket to allow). Hook event delivery retains
+ its existing independent failover policy.
+ **Every FALLBACK candidate is probed before the real POST** (`nt_probe_endpoint`: `/hook/verify`,
+ 204 on the bearer alone on every server build, `--connect-timeout 0.5 --max-time 1.5`). A reverse
+ tunnel whose sshd outlived the desktop's connection ACCEPTS and never answers, and once the
+ foreign Server Edition stopped absorbing the walk, a call posted straight into such a socket
+ hung. The bound is on the probe only: the primary is never probed and every real POST stays
+ unbounded, because a confirm-gated verb waits for a human (see "two canvases cannot raise two
+ dialogs" below).
**Measured on an SSH host (2026-09-28/29):** the desktop slept, the session's tunnel socket
stayed on disk with no listener, and the walk reached an unrelated Server Edition whose
`control-unsupported-on-this-edition … permanent … do not retry` (and, for context reads,
"No linked nodes") was true about that server and false about the session; the tunnel came
back minutes later. When a foreign candidate was skipped and no owner answered, the shims now
print `FOREIGN_ENDPOINT_HINT` — the owning connection is unreachable, the state is temporary,
- the usual cause for an SSH project is the tunnel — and all four agent-facing bodies quote its
- lead via `ownerUnreachableGuidanceLines`, because their other refusal lines rightly say "do not
- retry". `src/server/control-owner-tunnel-down.test.ts` rebuilds that host under real `/bin/sh`
- with the real Server Edition handlers as the foreign endpoint; removing the owner guard
- reproduces the incident's exact refusal line.
+ the usual cause for an SSH project is the tunnel — INSTEAD OF `STALE_ENDPOINT_HINT`, so a failure
+ carries one retry advice, not two. All four agent-facing bodies quote its lead via
+ `ownerUnreachableGuidanceLines`, because their other refusal lines rightly say "do not retry".
+ `src/server/control-owner-tunnel-down.test.ts` rebuilds that host under real `/bin/sh` with the
+ real Server Edition handlers as the foreign endpoint; `src/core/owned-endpoint-walk.test.ts`
+ pins the probe (hanging sockets), the owner reference, the single advice and the restore, each
+ mutation-checked.
- **Every generated sh client walks the SAME endpoint failover** (`nt_candidates`/`nt_adopt`,
`core/agents/hook-endpoint-failover-sh.ts`) — issue #445, the endpoint-level twin of #384: a
session is pinned for life to the endpoint PATH it got at tmux creation, so an app
@@ -3238,6 +3256,8 @@ command-bearing opens; this does not add a human-confirm dialog or change mobile
`nt_reached()` is true — failover fires only on a dead transport (`000`/empty). If a future change
gives that curl a timeout, this paragraph stops being true: a confirm-gated verb would then fail
over mid-wait and a second instance WOULD open a second dialog for the same logical request.
+ (The walk's liveness probe IS bounded, and does not break this: it runs only against a FALLBACK
+ candidate, only after the primary's transport already failed, and before any POST to it.)
**Grouping verbs** (`group` / `ungroup` / `move` / `arrange` / `align`): `group` wraps **sibling**
objects — nodes or frames — into a new frame in their shared container (a mixed-container set, or
an ancestor plus its descendant, is refused with that reason); `ungroup --group <id>` dissolves a
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index cd3e622bf..41b92226f 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -1051,9 +1051,13 @@ a transmitted text request is uncertain delivery, never a pre-paste refusal; sho
warning. SessionStart idle rescue is scoped to that same nonempty session and agent identity,
and a foreign idle must not broadcast fresh state proof to the renderer.
-Control and linked-context fallback must keep a known node identity on its owning nodeterm
-instance. A foreign server being reachable is not evidence that it owns the canvas. Match the
-candidate's own advertised node token before sending; never borrow a token from a global directory
-for that check. Keep real owning-endpoint refusals final and legacy hook delivery unchanged.
-When no owner answers, say so as a temporary state (`FOREIGN_ENDPOINT_HINT`); never let another
-instance's "permanent, do not retry" stand in for a dropped tunnel.
+Control and linked-context fallback must keep a known node identity on the endpoint family the
+session was born on. A foreign server being reachable is not evidence that it owns the canvas —
+and a matching token is not proof either (on an SSH host the token dir is shared per unix
+account); it is a routing rule, and the server still authorizes. Take the reference token from the
+primary endpoint's own dir, never from a global directory, and match each candidate's own dir
+against it. Probe a fallback candidate (bounded) before posting to it, but never put a timeout on
+the real POST: a confirm-gated verb waits for a human. Keep real owning-endpoint refusals final and
+legacy hook delivery unchanged. When no owner answers, say so once, as a temporary state
+(`FOREIGN_ENDPOINT_HINT`); never let another instance's "permanent, do not retry" stand in for a
+dropped tunnel.
diff --git a/src/core/agents/hook-endpoint-failover-sh.ts b/src/core/agents/hook-endpoint-failover-sh.ts
index cd38afa08..dea9bdb01 100644
--- a/src/core/agents/hook-endpoint-failover-sh.ts
+++ b/src/core/agents/hook-endpoint-failover-sh.ts
@@ -117,25 +117,107 @@ export const HOOK_ENDPOINT_FALLBACK_SH = [
/** Control/context requests must retain a known node identity across endpoint discovery.
- * Hook event failover deliberately keeps its existing policy. This is client-side routing,
- * not authorization: the receiving server still checks every bearer, node and verb. */
+ * Hook event failover deliberately keeps its existing policy. This is client-side ROUTING, not
+ * authorization and not proof of ownership: the receiving server still checks every bearer, node
+ * and verb. What a token match shows is only that the candidate reads its token for this node from
+ * the same place, with the same content, as the endpoint this session was born on. On an SSH host
+ * that place is shared per unix account (`remote-hooks.ts`, KNOWN LIMITATION: two desktops driving
+ * the same account overwrite each other's files), so it narrows the walk to "the same family of
+ * endpoints", which is what keeps an unrelated local Server Edition from answering for a desktop
+ * node; it does not tell two such desktops apart.
+ *
+ * Runs at the point it is spliced in, while the endpoint vars are still the PRIMARY's (both shims
+ * place it after the primary endpoint file is sourced and before the first POST).
+ *
+ * - THE OWNER TOKEN is read from the primary's own dir only — the one it advertises, else
+ * `<dir of the endpoint file>/node-tokens` — never from the global search `nt_read_node_token`
+ * walks. That search exists to PRESENT a capability (issue #384) and is kept for that; as an
+ * ownership reference it let a Server Edition that opened the same project.json (same node ids)
+ * supply the "owner's" token whenever the desktop's own token write had failed, and then pass
+ * the owner check against itself.
+ * - OWNER MODE needs that dir to EXIST. Then a candidate must hold exactly the same value in its
+ * own dir — including both holding nothing, which is what a failed token write looks like on
+ * every endpoint of the owning family. No such dir at all (a session with no endpoint file, a
+ * pre-token layout) is the only unknown-owner case, and keeps the legacy walk: adopt, re-read.
+ * - A FALLBACK candidate is probed before it is posted to (`nt_probe_endpoint`). The real POST has
+ * no --max-time on purpose — a confirm-gated verb waits for a human, and a client-side timeout
+ * would fail over mid-wait and let a second instance raise a second dialog — but a reverse-
+ * tunnel socket whose sshd outlived the desktop's connection (the Mac asleep) ACCEPTS and never
+ * answers, so posting straight into one hung the call. The primary is never probed: its POST is
+ * the question, and its genuine answer, however slow, stays final.
+ * - A skipped candidate leaves no endpoint behind (`nt_restore_endpoint`): the vars go back to
+ * the last endpoint this node could use, which is what the codex-sandbox hint names as the
+ * socket to allow. */
export const OWNED_ENDPOINT_FALLBACK_SH = `
-# A candidate must advertise this node's SAME capability. Do not use the global token search:
-# that could borrow the desktop token while posting to an unrelated headless server.
+# nt_owned_token_in <endpoint-file>: print the token the CURRENTLY SOURCED endpoint keeps for this
+# node, from the dir it advertises or else the one beside its file. Fails (status 1) when neither
+# names an existing directory — the unknown-owner case. Never the global dirs (see above).
+nt_owned_token_in() {
+ nt_otd="$NODETERM_NODE_TOKEN_DIR"
+ [ -n "$nt_otd" ] || nt_otd=$(nt_token_dir_beside "$1")
+ [ -n "$nt_otd" ] && [ -d "$nt_otd" ] || return 1
+ head -n 1 "$nt_otd/$NODETERM_NODE_ID" 2>/dev/null || :
+ return 0
+}
+nt_owner_known=""
+nt_owner_node_token=""
+nt_skipped_foreign_endpoint=""
+if nt_owner_node_token=$(nt_owned_token_in "$NODETERM_HOOK_ENDPOINT"); then
+ nt_owner_known=1
+fi
+
+nt_restore_endpoint() {
+ NODETERM_HOOK_SOCK="$nt_prev_sock"
+ NODETERM_HOOK_PORT="$nt_prev_port"
+ NODETERM_HOOK_TOKEN="$nt_prev_token"
+ NODETERM_NODE_TOKEN_DIR="$nt_prev_dir"
+ nt_node_token="$nt_prev_node_token"
+}
+
nt_adopt_for_node() {
- nt_adopt "$1" || return 1
- if [ -z "$nt_owner_node_token" ]; then
+ nt_prev_sock="$NODETERM_HOOK_SOCK"
+ nt_prev_port="$NODETERM_HOOK_PORT"
+ nt_prev_token="$NODETERM_HOOK_TOKEN"
+ nt_prev_dir="$NODETERM_NODE_TOKEN_DIR"
+ nt_prev_node_token="$nt_node_token"
+ if ! nt_adopt "$1"; then
+ nt_restore_endpoint
+ return 1
+ fi
+ if [ -z "$nt_owner_known" ]; then
nt_read_node_token "$1"
return 0
fi
- nt_node_token=""
- nt_owner_dir="$NODETERM_NODE_TOKEN_DIR"
- [ -n "$nt_owner_dir" ] || nt_owner_dir=$(nt_token_dir_beside "$1")
- if [ -n "$nt_owner_dir" ]; then
- nt_node_token=$(head -n 1 "$nt_owner_dir/$NODETERM_NODE_ID" 2>/dev/null) || nt_node_token=""
+ nt_cand_token=$(nt_owned_token_in "$1") || nt_cand_token=""
+ if [ "$nt_cand_token" = "$nt_owner_node_token" ]; then
+ nt_node_token="$nt_cand_token"
+ return 0
fi
- [ -n "$nt_node_token" ] && [ "$nt_node_token" = "$nt_owner_node_token" ] && return 0
nt_skipped_foreign_endpoint=1
+ nt_restore_endpoint
+ return 1
+}
+
+# Bounded liveness probe for an ADOPTED fallback candidate. /hook/verify answers 204 on the bearer
+# alone on every server build (the desktop's own tunnel probe used it before /verify existed, and
+# the server keeps it answering for that reason) and 421 for a bearer it does not own. On failure
+# nt_code is set as the POST would have left it (421, or 000 for a dead transport) so the final
+# diagnosis reads the same; the candidate stays adopted, since it is this node's own endpoint.
+nt_probe_endpoint() {
+ nt_pc=""
+ if [ -n "$NODETERM_HOOK_SOCK" ]; then
+ nt_pc=$(nt_hook_headers |
+ curl -s -o /dev/null -w '%{http_code}' -X POST --config - --connect-timeout 0.5 --max-time 1.5 --unix-socket "$NODETERM_HOOK_SOCK" "http://localhost/hook/verify" --data '' 2>/dev/null)
+ elif [ -n "$NODETERM_HOOK_PORT" ]; then
+ nt_pc=$(nt_hook_headers |
+ curl -s -o /dev/null -w '%{http_code}' -X POST --config - --connect-timeout 0.5 --max-time 1.5 "http://127.0.0.1:$NODETERM_HOOK_PORT/hook/verify" --data '' 2>/dev/null)
+ else
+ nt_code=""
+ return 1
+ fi
+ nt_had_transport=1
+ [ "$nt_pc" = "204" ] && return 0
+ if [ "$nt_pc" = "421" ]; then nt_code="421"; else nt_code="000"; fi
return 1
}
`
@@ -156,8 +238,8 @@ nt_adopt_for_node() {
export const OWNER_UNREACHABLE_LEAD = 'The nodeterm connection that owns this node is unreachable.'
export const FOREIGN_ENDPOINT_HINT =
- `${OWNER_UNREACHABLE_LEAD} Other nodeterm endpoints on this machine were skipped: they do not own ` +
- 'this node, so their answer would describe a different canvas. This is temporary — for an SSH ' +
+ `${OWNER_UNREACHABLE_LEAD} Other nodeterm endpoints on this machine were skipped: they keep a ` +
+ 'different identity for this node, so their answer would describe a different canvas. This is temporary — for an SSH ' +
"project it usually means the desktop's reverse tunnel is down (the desktop is asleep, offline or " +
'reconnecting). Retry the same command after it reconnects.'
diff --git a/src/core/canvas-control-core.ts b/src/core/canvas-control-core.ts
index 02df72aa1..7f33e4bab 100644
--- a/src/core/canvas-control-core.ts
+++ b/src/core/canvas-control-core.ts
@@ -761,8 +761,6 @@ fi
# Missing everywhere leaves it empty, which the server reads as legacy — the request still goes.
${NODE_TOKEN_READ_SH}
nt_read_node_token
-nt_owner_node_token="$nt_node_token"
-nt_skipped_foreign_endpoint=""
${HOOK_CURL_HEADERS_SH}
@@ -894,6 +892,7 @@ if ! nt_reached && { [ "$nt_code" = "421" ] || [ -z "$CODEX_SANDBOX_NETWORK_DISA
[ "$nt_n" -lt "$nt_fallback_max" ] || break
nt_adopt_for_node "$nt_ep" || continue
nt_n=$((nt_n + 1))
+ nt_probe_endpoint || continue
nt_control_post "$@"
nt_reached && break
done <<NT_CANDIDATES
@@ -919,7 +918,9 @@ if [ -z "$nt_code" ] || [ "$nt_code" = "000" ]; then
echo "nodeterm control endpoint unavailable." >&2
else
nt_codex_sandbox_hint || echo "${CONTROL_UNREACHABLE_MSG}" >&2
- if [ -z "$CODEX_SANDBOX_NETWORK_DISABLED" ]; then
+ # One piece of advice per failure: when the walk skipped a foreign endpoint, the owner-unreachable
+ # sentence above already says what happened and when to retry.
+ if [ -z "$CODEX_SANDBOX_NETWORK_DISABLED" ] && [ -z "$nt_skipped_foreign_endpoint" ]; then
echo "${STALE_ENDPOINT_HINT}" >&2
fi
fi
diff --git a/src/core/context-link-core.ts b/src/core/context-link-core.ts
index 9f76e8ed7..c337152a8 100644
--- a/src/core/context-link-core.ts
+++ b/src/core/context-link-core.ts
@@ -200,8 +200,6 @@ fi
# falls back to the standard locations rather than reading as \`legacy\` forever (issue #384).
${NODE_TOKEN_READ_SH}
nt_read_node_token
-nt_owner_node_token="$nt_node_token"
-nt_skipped_foreign_endpoint=""
${HOOK_CURL_HEADERS_SH}
@@ -279,6 +277,7 @@ if ! nt_reached && { [ "$nt_code" = "421" ] || [ -z "$CODEX_SANDBOX_NETWORK_DISA
[ "$nt_n" -lt "$nt_fallback_max" ] || break
nt_adopt_for_node "$nt_ep" || continue
nt_n=$((nt_n + 1))
+ nt_probe_endpoint || continue
nt_ctx_post "$@"
nt_reached && break
done <<NT_CANDIDATES
@@ -309,7 +308,9 @@ fi
echo "${CONTEXT_UNREACHABLE_MSG}" >&2
# A transport WAS advertised and nothing answered, primary or fallback: name the stale endpoint so
# the agent stops relinking a healthy canvas (the link is fine; the app behind it is not there).
-if [ -z "$nt_code" ] || [ "$nt_code" = "000" ]; then
+# One piece of advice per failure: when the walk skipped a foreign endpoint, the owner-unreachable
+# sentence above already says what happened and when to retry.
+if { [ -z "$nt_code" ] || [ "$nt_code" = "000" ]; } && [ -z "$nt_skipped_foreign_endpoint" ]; then
echo "${STALE_ENDPOINT_HINT}" >&2
fi
exit 1
diff --git a/src/core/context-link.cli.test.ts b/src/core/context-link.cli.test.ts
index bd29ee85e..5bbf05073 100644
--- a/src/core/context-link.cli.test.ts
+++ b/src/core/context-link.cli.test.ts
@@ -423,7 +423,11 @@ describe('context-link shim keeps credentials off curl\'s command line', () => {
it('names no credential header in the generated source at all', () => {
expect(CONTEXT_SHIM_SCRIPT).not.toContain('-H "X-Nodeterm-Hook-Token')
expect(CONTEXT_SHIM_SCRIPT).not.toContain('-H "X-Nodeterm-Node-Token')
- expect((CONTEXT_SHIM_SCRIPT.match(/--config -/g) ?? []).length).toBe(2)
+ // Every curl call site reads its headers from stdin: the POST over each transport, and the
+ // fallback liveness probe over each transport (hook-endpoint-failover-sh.ts).
+ const calls = CONTEXT_SHIM_SCRIPT.match(/\bcurl -s/g) ?? []
+ expect(calls).toHaveLength(4)
+ expect((CONTEXT_SHIM_SCRIPT.match(/--config -/g) ?? []).length).toBe(calls.length)
})
it('over TCP: neither token is in argv, both arrive on stdin and reach the server', async () => {
diff --git a/src/core/owned-endpoint-walk.test.ts b/src/core/owned-endpoint-walk.test.ts
new file mode 100644
index 000000000..08d158bc4
--- /dev/null
+++ b/src/core/owned-endpoint-walk.test.ts
@@ -0,0 +1,383 @@
+// The owned-endpoint walk (hook-endpoint-failover-sh.ts), exercised for real: both generated shims
+// under the real /bin/sh, against fixture endpoints that implement the one server contract the walk
+// relies on — `/hook/verify` answers 204 for the endpoint's own bearer and 421 for any other, the
+// way HookServer does. Follow-ups to PR #911 (the 2026-09-28/29 incident, see
+// src/server/control-owner-tunnel-down.test.ts for the incident itself):
+//
+// 1. A reverse-tunnel socket whose sshd outlived the desktop's connection (the Mac asleep) ACCEPTS
+// and never answers. Once the foreign Server Edition stopped absorbing the walk, the walk posted
+// straight into such sockets — and the control POST has no --max-time on purpose (a confirm-
+// gated verb waits for a human), so the call hung. Fallback candidates are now probed first,
+// with a bound; the primary's POST, and a probed candidate's POST, stay unbounded.
+// 2. The owner token came from the global dir search, so a Server Edition holding a token for the
+// same node id (the same project.json open in both) passed as the owner when the desktop's own
+// token write had failed.
+// 3. The stale-endpoint hint ("retry once") printed after the owner-unreachable sentence ("retry
+// after it reconnects"): two pieces of advice for one failure.
+// 4. A skipped foreign candidate left its SOCK behind, so the macOS codex-sandbox hint could name
+// the foreign socket.
+import { describe, it, expect, afterEach } from 'vitest'
+import { execFile, spawn } from 'node:child_process'
+import { createServer as createHttpServer, type IncomingMessage } from 'node:http'
+import { createServer as createNetServer, type Server, type Socket } from 'node:net'
+import fs from 'node:fs'
+import os from 'node:os'
+import path from 'node:path'
+import { CONTROL_SHIM_SCRIPT } from './canvas-control-core'
+import { CONTEXT_SHIM_SCRIPT } from './context-link-core'
+import { FOREIGN_ENDPOINT_HINT, STALE_ENDPOINT_HINT } from './agents/hook-endpoint-failover-sh'
+
+const NODE = 'n1'
+const OWNER_TOKEN = 'owner-kid.owner-mac'
+const SE_REFUSAL =
+ 'control-unsupported-on-this-edition: Canvas control is not available on the nodeterm Server Edition. ' +
+ 'This is permanent on this host, not a temporary failure — do not retry.'
+
+const cleanups: (() => Promise<void> | void)[] = []
+afterEach(async () => {
+ for (const c of cleanups.splice(0).reverse()) await c()
+})
+
+function tmp(): string {
+ // Short: every socket below must fit sun_path (104 bytes on macOS).
+ const d = fs.mkdtempSync(path.join(os.tmpdir(), 'nt-ow-'))
+ cleanups.push(() => fs.rmSync(d, { recursive: true, force: true }))
+ return d
+}
+
+interface Recorded { url: string; nodeToken?: string }
+
+/** An endpoint that behaves like HookServer on the two routes the walk uses. */
+async function endpoint(
+ sock: string,
+ bearer: string,
+ post: { status: number; body: string; delayMs?: number }
+): Promise<{ probes: Recorded[]; posts: Recorded[] }> {
+ const probes: Recorded[] = []
+ const posts: Recorded[] = []
+ const server = createHttpServer((req: IncomingMessage, res) => {
+ const tok = req.headers['x-nodeterm-node-token']
+ const rec = { url: req.url ?? '', nodeToken: Array.isArray(tok) ? tok[0] : tok }
+ req.resume()
+ req.on('end', () => {
+ if (req.headers['x-nodeterm-hook-token'] !== bearer) {
+ res.writeHead(421)
+ res.end('hook-endpoint-wrong-owner\n')
+ return
+ }
+ if (req.url === '/hook/verify' || req.url === '/verify') {
+ probes.push(rec)
+ res.writeHead(204)
+ res.end()
+ return
+ }
+ posts.push(rec)
+ setTimeout(() => {
+ res.writeHead(post.status, { 'content-type': 'text/plain' })
+ res.end(post.body)
+ }, post.delayMs ?? 0)
+ })
+ })
+ await new Promise<void>((r) => server.listen(sock, r))
+ cleanups.push(() => new Promise<void>((r) => server.close(() => r())))
+ return { probes, posts }
+}
+
+/** What an sshd-held reverse-tunnel socket does while the desktop is asleep: accept, never answer. */
+async function hangingSocket(sock: string): Promise<{ data: () => string }> {
+ let data = ''
+ const conns = new Set<Socket>()
+ const server: Server = createNetServer((c) => {
+ conns.add(c)
+ c.on('data', (d) => (data += d.toString()))
+ c.on('error', () => {})
+ })
+ await new Promise<void>((r) => server.listen(sock, r))
+ cleanups.push(
+ () =>
+ new Promise<void>((r) => {
+ for (const c of conns) c.destroy()
+ server.close(() => r())
+ })
+ )
+ return { data: () => data }
+}
+
+/** A socket FILE with no listener — what a dead sshd leaves (a SIGKILLed holder never unlinks). */
+async function staleSocket(sock: string): Promise<void> {
+ const child = spawn(
+ process.execPath,
+ ['-e', "require('net').createServer().listen(process.argv[1], () => process.stdout.write('up'))", sock],
+ { stdio: ['ignore', 'pipe', 'inherit'] }
+ )
+ await new Promise<void>((r) => child.stdout!.once('data', () => r()))
+ await new Promise<void>((r) => {
+ child.once('exit', () => r())
+ child.kill('SIGKILL')
+ })
+}
+
+function endpointFile(file: string, sock: string, bearer: string, tokenDir?: string): void {
+ fs.mkdirSync(path.dirname(file), { recursive: true })
+ fs.writeFileSync(
+ file,
+ `NODETERM_HOOK_SOCK='${sock}'\nNODETERM_HOOK_TOKEN='${bearer}'\nNODETERM_HOOK_VERSION='2'\n` +
+ (tokenDir ? `NODETERM_NODE_TOKEN_DIR='${tokenDir}'\n` : '')
+ )
+}
+
+/** Force the tunnel walk's `ls -t` order: larger `ageSec` = older = tried later. */
+function age(file: string, ageSec: number): void {
+ const t = Date.now() / 1000 - ageSec
+ fs.utimesSync(file, t, t)
+}
+
+interface Run { code: number; stdout: string; stderr: string; ms: number }
+
+async function shim(
+ d: string,
+ which: 'control' | 'context',
+ args: string[],
+ env: Record<string, string>
+): Promise<Run> {
+ const file = path.join(d, `${which}.sh`)
+ fs.writeFileSync(file, which === 'control' ? CONTROL_SHIM_SCRIPT : CONTEXT_SHIM_SCRIPT, { mode: 0o755 })
+ const started = Date.now()
+ return new Promise((resolve) => {
+ execFile(
+ '/bin/sh',
+ [file, ...args],
+ {
+ timeout: 12_000,
+ env: {
+ PATH: process.env.PATH ?? '',
+ NODETERM_CANVAS_CONTROL: '1',
+ NODETERM_NODE_ID: NODE,
+ ...env
+ }
+ },
+ (err, stdout, stderr) => {
+ const code = err ? (typeof err.code === 'number' ? err.code : -1) : 0
+ resolve({ code, stdout, stderr, ms: Date.now() - started })
+ }
+ )
+ })
+}
+
+/** The SSH-host layout: the session's own tunnel (dead) + the desktop's token dir. */
+async function sshHost(opts: { ownerToken?: boolean } = {}): Promise<{
+ d: string
+ home: string
+ tokens: string
+ primary: string
+ env: Record<string, string>
+}> {
+ const d = tmp()
+ const home = path.join(d, 'h')
+ const tokens = path.join(home, '.nodeterm', 'node-tokens')
+ fs.mkdirSync(tokens, { recursive: true })
+ // Another node's token: the dir is real and in use, whatever this node's own file says.
+ fs.writeFileSync(path.join(tokens, 'someone-else'), 'other\n')
+ if (opts.ownerToken !== false) fs.writeFileSync(path.join(tokens, NODE), `${OWNER_TOKEN}\n`)
+ const sock = path.join(home, '.nodeterm', 'a.sock')
+ await staleSocket(sock)
+ const primary = path.join(home, '.nodeterm', 'hook-endpoint-pa.env')
+ endpointFile(primary, sock, 'desk-bearer', tokens)
+ return { d, home, tokens, primary, env: { HOME: home, NODETERM_HOOK_ENDPOINT: primary } }
+}
+
+/** An unrelated local Server Edition in the first slot the walk tries. */
+async function serverEdition(home: string, opts: { tokenForNode?: string } = {}) {
+ const dir = path.join(home, '.nodeterm-server')
+ const tokens = path.join(dir, 'node-tokens')
+ fs.mkdirSync(tokens, { recursive: true })
+ if (opts.tokenForNode) fs.writeFileSync(path.join(tokens, NODE), `${opts.tokenForNode}\n`)
+ const sock = path.join(dir, 's.sock')
+ const ep = await endpoint(sock, 'se-bearer', { status: 400, body: SE_REFUSAL })
+ endpointFile(path.join(dir, 'hook-endpoint.env'), sock, 'se-bearer', tokens)
+ return ep
+}
+
+const skipWin = describe.skipIf(process.platform === 'win32')
+// A hang is what item 1 is about: give every case room to FAIL (the shim's own 12 s exec bound)
+// rather than letting vitest's 5 s default report a timeout that looks like any other flake.
+const T = 20_000
+
+skipWin('1. a fallback tunnel that accepts and never answers cannot hang the call', () => {
+ it.each(['control', 'context'] as const)(
+ '%s: probes past a hanging sibling and reaches the live owner behind it',
+ async (which) => {
+ const host = await sshHost()
+ const hang = await hangingSocket(path.join(host.home, '.nodeterm', 'b.sock'))
+ const pb = path.join(host.home, '.nodeterm', 'hook-endpoint-pb.env')
+ endpointFile(pb, path.join(host.home, '.nodeterm', 'b.sock'), 'desk-bearer', host.tokens)
+ age(pb, 10)
+ const live = await endpoint(path.join(host.home, '.nodeterm', 'c.sock'), 'desk-bearer', {
+ status: 200,
+ body: 'owner reply'
+ })
+ const pc = path.join(host.home, '.nodeterm', 'hook-endpoint-pc.env')
+ endpointFile(pc, path.join(host.home, '.nodeterm', 'c.sock'), 'desk-bearer', host.tokens)
+ age(pc, 100)
+
+ const r = await shim(host.d, which, ['list'], host.env)
+ expect(r).toMatchObject({ code: 0, stdout: 'owner reply' })
+ expect(r.ms).toBeLessThan(6000)
+ // The hanging socket saw a probe and nothing else: no verb was ever parked inside it.
+ expect(hang.data()).toContain('/hook/verify')
+ expect(hang.data()).not.toMatch(/\/control\/|\/context-link\//)
+ expect(live.posts.map((p) => p.nodeToken)).toEqual([OWNER_TOKEN])
+ },
+ T
+ )
+
+ it("last night's shape with sshd still holding every sibling: fails in bounded time, as owner-unreachable", async () => {
+ const host = await sshHost()
+ const se = await serverEdition(host.home)
+ const hangs = []
+ for (const [i, n] of ['b', 'c', 'd'].entries()) {
+ const sock = path.join(host.home, '.nodeterm', `${n}.sock`)
+ hangs.push(await hangingSocket(sock))
+ const f = path.join(host.home, '.nodeterm', `hook-endpoint-p${n}.env`)
+ endpointFile(f, sock, 'desk-bearer', host.tokens)
+ age(f, 10 * (i + 1))
+ }
+ const [ctl, ctx] = await Promise.all([
+ shim(host.d, 'control', ['close', '--node', 'a,b'], host.env),
+ shim(host.d, 'context', ['list'], host.env)
+ ])
+ for (const r of [ctl, ctx]) {
+ expect(r.code).toBe(1)
+ expect(r.ms).toBeLessThan(9000)
+ expect(r.stderr).toContain(FOREIGN_ENDPOINT_HINT)
+ // 3. ONE piece of advice: the owner-unreachable sentence, not also "retry once".
+ expect(r.stderr).not.toContain(STALE_ENDPOINT_HINT)
+ expect(r.stderr).not.toMatch(/retry once/i)
+ expect(r.stderr).not.toMatch(/permanent|do not retry/i)
+ }
+ expect(se.probes).toEqual([])
+ expect(se.posts).toEqual([])
+ for (const h of hangs) expect(h.data()).not.toMatch(/\/control\/|\/context-link\//)
+ }, T)
+
+ // The bound is on the PROBE only. A real POST — the primary's, or a probed candidate's — may be a
+ // confirm-gated verb waiting for a human (up to CONTROL_REQUEST_TIMEOUT_MS), and a client-side
+ // timeout there would fail over mid-wait and let a second instance raise a second dialog.
+ it("the primary's slow answer is waited for and stays final", async () => {
+ const d = tmp()
+ const home = path.join(d, 'h')
+ const tokens = path.join(home, '.nodeterm', 'node-tokens')
+ fs.mkdirSync(tokens, { recursive: true })
+ fs.writeFileSync(path.join(tokens, NODE), `${OWNER_TOKEN}\n`)
+ const primary = await endpoint(path.join(home, '.nodeterm', 'a.sock'), 'desk-bearer', {
+ status: 200,
+ body: 'confirmed after a while',
+ delayMs: 2500
+ })
+ const primaryFile = path.join(home, '.nodeterm', 'hook-endpoint-pa.env')
+ endpointFile(primaryFile, path.join(home, '.nodeterm', 'a.sock'), 'desk-bearer', tokens)
+ const sibling = await endpoint(path.join(home, '.nodeterm', 'b.sock'), 'desk-bearer', {
+ status: 200,
+ body: 'sibling'
+ })
+ endpointFile(path.join(home, '.nodeterm', 'hook-endpoint-pb.env'), path.join(home, '.nodeterm', 'b.sock'), 'desk-bearer', tokens)
+
+ const r = await shim(d, 'control', ['close', '--node', 'a'], { HOME: home, NODETERM_HOOK_ENDPOINT: primaryFile })
+ expect(r).toMatchObject({ code: 0, stdout: 'confirmed after a while' })
+ expect(primary.probes).toEqual([]) // the primary is never probed: its POST is the question
+ expect(sibling.probes).toEqual([])
+ expect(sibling.posts).toEqual([])
+ }, T)
+
+ it("a probed fallback owner's slow answer is waited for too", async () => {
+ const host = await sshHost()
+ const live = await endpoint(path.join(host.home, '.nodeterm', 'b.sock'), 'desk-bearer', {
+ status: 200,
+ body: 'confirmed via sibling',
+ delayMs: 2500
+ })
+ endpointFile(path.join(host.home, '.nodeterm', 'hook-endpoint-pb.env'), path.join(host.home, '.nodeterm', 'b.sock'), 'desk-bearer', host.tokens)
+ const r = await shim(host.d, 'control', ['close', '--node', 'a'], host.env)
+ expect(r).toMatchObject({ code: 0, stdout: 'confirmed via sibling' })
+ expect(live.probes).toHaveLength(1)
+ expect(live.posts).toHaveLength(1)
+ }, T)
+})
+
+skipWin("2. the owner token comes from the primary's own dir, never the global search", () => {
+ // The desktop's remote token write failed, so the dir the session's endpoint advertises exists
+ // but holds no file for this node — while a local Server Edition that opened the same
+ // project.json holds one for the same node id. The global search used to adopt that token as
+ // "the owner's", and the Server Edition then passed the owner check.
+ it.each(['control', 'context'] as const)(
+ '%s: a Server Edition holding a token for the same node id is not the owner',
+ async (which) => {
+ const host = await sshHost({ ownerToken: false })
+ const se = await serverEdition(host.home, { tokenForNode: 'se-kid.se-mac' })
+ const sibling = await endpoint(path.join(host.home, '.nodeterm', 'b.sock'), 'desk-bearer', {
+ status: 200,
+ body: 'owner reply'
+ })
+ endpointFile(path.join(host.home, '.nodeterm', 'hook-endpoint-pb.env'), path.join(host.home, '.nodeterm', 'b.sock'), 'desk-bearer', host.tokens)
+
+ const r = await shim(host.d, which, ['list'], host.env)
+ expect(r).toMatchObject({ code: 0, stdout: 'owner reply' })
+ expect(se.probes).toEqual([])
+ expect(se.posts).toEqual([])
+ // The owner's dir holds nothing for this node, so nothing is presented — never the SE's token.
+ expect(sibling.posts.map((p) => p.nodeToken)).toEqual([undefined])
+ },
+ T
+ )
+
+ it('with no owning endpoint alive, says so instead of relaying the Server Edition refusal', async () => {
+ const host = await sshHost({ ownerToken: false })
+ const se = await serverEdition(host.home, { tokenForNode: 'se-kid.se-mac' })
+ const r = await shim(host.d, 'control', ['list'], host.env)
+ expect(r.code).toBe(1)
+ expect(r.stderr).toContain(FOREIGN_ENDPOINT_HINT)
+ expect(r.stderr).not.toContain('control-unsupported-on-this-edition')
+ expect(se.posts).toEqual([])
+ }, T)
+})
+
+skipWin('4. a skipped foreign candidate leaves no endpoint behind', () => {
+ // Codex's macOS sandbox hint names $NODETERM_HOOK_SOCK as the socket to allow. After the walk it
+ // must be an endpoint this node could actually use — never one that was skipped as foreign.
+ it('the sandbox hint names the owned socket, not the foreign one adopted last', async () => {
+ const host = await sshHost()
+ // Primary answers 421 (the one failure a sandboxed walk may act on).
+ const primarySock = path.join(host.home, '.nodeterm', 'p.sock')
+ await endpoint(primarySock, 'the-real-bearer', { status: 200, body: 'unreachable' })
+ endpointFile(host.primary, primarySock, 'stale-bearer', host.tokens)
+ // Owned candidate, dead; then a foreign one, later in the walk's fixed local order.
+ const ownedSock = path.join(host.home, 'o.sock')
+ await staleSocket(ownedSock)
+ endpointFile(path.join(host.home, '.config', 'node-terminal', 'hook-endpoint.env'), ownedSock, 'desk-bearer', host.tokens)
+ const foreignTokens = path.join(host.home, 'ft')
+ fs.mkdirSync(foreignTokens)
+ const foreignSock = path.join(host.home, 'f.sock')
+ const foreign = await endpoint(foreignSock, 'f-bearer', { status: 200, body: 'foreign' })
+ endpointFile(
+ path.join(host.home, 'Library', 'Application Support', 'node-terminal', 'hook-endpoint.env'),
+ foreignSock,
+ 'f-bearer',
+ foreignTokens
+ )
+ // The hint's socket line is macOS-only; pretend.
+ const bin = path.join(host.d, 'bin')
+ fs.mkdirSync(bin)
+ fs.writeFileSync(path.join(bin, 'uname'), '#!/bin/sh\necho Darwin\n', { mode: 0o755 })
+
+ const r = await shim(host.d, 'control', ['list'], {
+ ...host.env,
+ PATH: `${bin}:${process.env.PATH ?? ''}`,
+ CODEX_SANDBOX_NETWORK_DISABLED: '1'
+ })
+ expect(r.code).toBe(1)
+ expect(r.stderr).toContain(`"${ownedSock}"`)
+ expect(r.stderr).not.toContain(foreignSock)
+ expect(foreign.probes).toEqual([])
+ expect(foreign.posts).toEqual([])
+ }, T)
+})
diff --git a/src/main/canvas-control-shim.test.ts b/src/main/canvas-control-shim.test.ts
index b0318e06e..4c104847e 100644
--- a/src/main/canvas-control-shim.test.ts
+++ b/src/main/canvas-control-shim.test.ts
@@ -512,7 +512,11 @@ describe('canvas-control shim keeps credentials off curl\'s command line', () =>
it('names no credential header in the generated source at all', () => {
expect(CONTROL_SHIM_SCRIPT).not.toContain('-H "X-Nodeterm-Hook-Token')
expect(CONTROL_SHIM_SCRIPT).not.toContain('-H "X-Nodeterm-Node-Token')
- expect((CONTROL_SHIM_SCRIPT.match(/--config -/g) ?? []).length).toBe(2)
+ // Every curl call site reads its headers from stdin: the POST over each transport, and the
+ // fallback liveness probe over each transport (hook-endpoint-failover-sh.ts).
+ const calls = CONTROL_SHIM_SCRIPT.match(/\bcurl -s/g) ?? []
+ expect(calls).toHaveLength(4)
+ expect((CONTROL_SHIM_SCRIPT.match(/--config -/g) ?? []).length).toBe(calls.length)
})
it('over TCP: neither token is in argv, both arrive on stdin and reach the server', async () => {
diff --git a/src/main/control-owner-failover.test.ts b/src/main/control-owner-failover.test.ts
index 55a64f5fb..0418dcd65 100644
--- a/src/main/control-owner-failover.test.ts
+++ b/src/main/control-owner-failover.test.ts
@@ -18,17 +18,25 @@ afterEach(async () => {
await Promise.all(servers.splice(0).map(s => new Promise<void>(r => s.close(() => r()))))
for (const dir of dirs.splice(0)) fs.rmSync(dir, { recursive: true, force: true })
})
+// Like HookServer, a fixture answers the walk's liveness probe (`/hook/verify`) with 204 — kept out
+// of `requests`, which records only the verbs a client actually delivered.
async function endpoint(status: number, message: string) {
const requests: { url: string; body: string; nodeToken: string | string[] | undefined }[] = []
+ const probes: string[] = []
const server = createServer(async (req, res) => {
let body = ''
for await (const chunk of req) body += chunk
+ if (req.url === '/hook/verify') {
+ probes.push(req.url)
+ res.writeHead(204); res.end()
+ return
+ }
requests.push({ url: req.url!, body, nodeToken: req.headers['x-nodeterm-node-token'] })
res.writeHead(status); res.end(message)
})
servers.push(server)
await new Promise<void>(r => server.listen(0, '127.0.0.1', r))
- return { port: (server.address() as { port: number }).port, requests }
+ return { port: (server.address() as { port: number }).port, requests, probes }
}
const clients = [
{ name: 'control', script: CONTROL_SHIM_SCRIPT, args: ['close', '--node', 'finished-a,finished-b'], route: '/control/close' },
@@ -42,6 +50,7 @@ describe.skipIf(process.platform === 'win32')('node identity survives endpoint f
const result = await fixture.call()
expect(result.stdout).toBe('owner reply')
expect(fixture.foreign.requests).toEqual([])
+ expect(fixture.foreign.probes).toEqual([])
expect(fixture.owner.requests).toHaveLength(1)
expect(fixture.owner.requests[0].nodeToken).toBe('owner-capability')
if (client.name === 'control') {
@@ -62,6 +71,7 @@ describe.skipIf(process.platform === 'win32')('node identity survives endpoint f
.replace(/^NODETERM_NODE_TOKEN_DIR=.*\n/m, ''))
expect((await fixture.call()).stdout).toBe('owner reply')
expect(fixture.foreign.requests).toEqual([])
+ expect(fixture.foreign.probes).toEqual([])
expect(fixture.owner.requests).toHaveLength(1)
})
it(`${client.name}: reports the missing owning connection, not a foreign edition refusal`, async () => {
@@ -73,6 +83,7 @@ describe.skipIf(process.platform === 'win32')('node identity survives endpoint f
expect(error.stderr).not.toContain('control-unsupported-on-this-edition')
expect(error.stderr).not.toContain('owner-capability')
expect(fixture.foreign.requests).toEqual([])
+ expect(fixture.foreign.probes).toEqual([])
expect(fixture.owner.requests).toEqual([])
})
it.each([400, 403])(`${client.name}: keeps an owning endpoint's %s refusal final`, async status => {
@@ -83,6 +94,7 @@ describe.skipIf(process.platform === 'win32')('node identity survives endpoint f
expect(error.stderr).not.toContain(FOREIGN_ENDPOINT_HINT)
expect(fixture.owner.requests).toHaveLength(1)
expect(fixture.foreign.requests).toEqual([])
+ expect(fixture.foreign.probes).toEqual([])
})
}
})