The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.
TypeScript 3d-modelingagent-frameworkagent-harnessai-agentscadclaude-codecodexcoding-agentsdeveloper-toolsesp32game-developmentmcp-servermulti-agentopen-hardwareopencodeopenharnesspcbroboticsterminaltmux
harnessd: the relay link under stress (round 34): no replayed news, no silent loss after a restart (#802)
Deec2118be67220
diff --git a/cli/e2e/harness/fakeBackend.ts b/cli/e2e/harness/fakeBackend.tsindex 44cc58eb7..842de953d 100644--- a/cli/e2e/harness/fakeBackend.ts+++ b/cli/e2e/harness/fakeBackend.ts@@ -11,9 +11,22 @@* that machine's node, an untagged one to the machine selected; a reply to that commander comes back* up tagged with the machine it is from, which is how the device knows which session opens it.*+ * - `/api/web-ws`: a phone's (or a browser's) socket, as the real hub's web clients: it starts with+ * `connected {userId}`, `machine_select` binds it to one machine (`connected {machineId}`, then that+ * machine's `node_status`), and from then on what it sends goes down to the machine's node under its+ * own connection id. A broadcast from the node reaches every web client of that machine unless it is+ * `webEligible: false`; a frame targeted at a connection reaches that one only. A web client leaving+ * is `__client_disconnected` to the node, and the node coming and going is `node_status {online}` to+ * the machine's web clients, as the real backend's node role publishes them.+ ** Like the real relay it is blind: the machines seal everything between them end to end, and a frame* for a machine whose node is not connected is simply never answered, as the real hub's would not be.* Everything else is answered 404. Requests are recorded in `seen`, so a test can say what was asked.+ *+ * And it can misbehave, as a relay under stress does: every socket dropped at once, down for a while,+ * restarted on the same port, seconds of latency on what it relays, garbage or an old sealed frame+ * injected towards a machine or a phone. What each web client sent is kept, sealed, so a test can+ * replay it; so is what reached each one.*/import { randomUUID } from 'node:crypto'import { createServer, type IncomingMessage, type Server } from 'node:http'@@ -30,6 +43,13 @@ export interface FakeMachine {type Frame = { type?: string; machineId?: string; agentId?: string; payload?: Record<string, unknown>; [key: string]: unknown }+/** One web socket (a phone, a browser): bound to at most one machine at a time, under one connection id. */+interface WebClient {+ ws: WebSocket+ connId: string+ machineId: string | null+}+/** One device socket: a commander on every machine of the account, one connection id per machine. */interface Device {ws: WebSocket@@ -46,8 +66,19 @@ export class FakeBackend {private readonly nodes = new Map<string, WebSocket>()private readonly devices = new Set<Device>()private readonly connOwners = new Map<string, { device: Device; machineId: string }>()+ private readonly webClients = new Set<WebClient>()+ /** The tokens a phone or a browser of this account signs in to the web socket with. */+ private readonly userTokens = new Set<string>()private readonly wss = new WebSocketServer({ noServer: true, handleProtocols: (protocols) => [...protocols][0] ?? false })private generation = 0+ /** Down: every new connection is refused, as a backend that is not there. */+ private down = false+ /** How long everything relayed between a machine and its web clients is held, ms. */+ latencyMs = 0+ /** What each web client sent towards its machine, as sent (sealed), by connection id. */+ readonly webSent = new Map<string, Frame[]>()+ /** What reached each web client from its machine, as relayed (sealed), by connection id. */+ readonly webReceived = new Map<string, Frame[]>()private constructor(private readonly server: Server, readonly port: number) {}@@ -61,6 +92,7 @@ export class FakeBackend {backend.seen.push(`${req.method} ${req.url}`)const machine = backend.byToken(req)res.setHeader('content-type', 'application/json')+ if (backend.down) { res.statusCode = 503; res.end(json({ success: false, error: { code: 'UNAVAILABLE' } })); return }if (!machine) { res.statusCode = 401; res.end(json({ success: false, error: { code: 'UNAUTHORIZED' } })); return }if (req.method === 'GET' && req.url?.split('?')[0] === '/api/machines') {res.end(json({ success: true, data: { machines: backend.rows() } }))@@ -98,6 +130,72 @@ export class FakeBackend {return [...this.devices].filter((device) => device.ws.readyState === WebSocket.OPEN).length}+ /** A phone or a browser of this account: the token it opens the web socket with. */+ addUser(token: string): void {+ this.userTokens.add(token)+ }++ /** The web clients bound to a machine now, by connection id. */+ webConnections(machineId: string): string[] {+ return [...this.webClients].filter((client) => client.machineId === machineId && client.ws.readyState === WebSocket.OPEN).map((client) => client.connId)+ }++ // ── a relay under stress ────────────────────────────────────────────────────────────────────++ /** Every socket dropped at once, with no close frame: what a relay going away looks like to both ends. */+ dropAll(): void {+ for (const client of this.wss.clients) client.terminate()+ }++ /** Down: every socket dropped and every new connection refused, until `comeUp`. */+ goDown(): void {+ this.down = true+ this.dropAll()+ }++ comeUp(): void {+ this.down = false+ }++ /** The backend process restarting: everything it knew about connections goes, nothing answers for+ * `gapMs`, and it listens again on the same port. */+ async restart(gapMs: number): Promise<void> {+ this.dropAll()+ this.nodes.clear()+ this.devices.clear()+ this.connOwners.clear()+ this.webClients.clear()+ await new Promise<void>((done) => this.server.close(() => done()))+ await new Promise((done) => setTimeout(done, gapMs))+ await new Promise<void>((done, fail) => {+ this.server.once('error', fail)+ this.server.listen(this.port, '127.0.0.1', () => { this.server.off('error', fail); done() })+ })+ }++ /** A frame (or raw bytes) down to a machine's node as if a web client had sent it under `connId`:+ * what a relay replaying or corrupting traffic does. False when the node is not connected. */+ injectDown(machineId: string, connId: string, frame: Frame | string): boolean {+ const ws = this.nodes.get(machineId)+ if (ws?.readyState !== WebSocket.OPEN) return false+ ws.send(typeof frame === 'string' ? frame : json({ t: 'down', connId, frame }))+ return true+ }++ /** A frame (or raw bytes) to one web client as if its machine had sent it. False when it is gone. */+ injectUp(connId: string, frame: Frame | string): boolean {+ const client = [...this.webClients].find((candidate) => candidate.connId === connId)+ if (client?.ws.readyState !== WebSocket.OPEN) return false+ client.ws.send(typeof frame === 'string' ? frame : json(frame))+ return true+ }++ /** Later by `latencyMs`, in order: equal delays fire in the order they were set. */+ private relay(run: () => void): void {+ if (this.latencyMs > 0) setTimeout(run, this.latencyMs)+ else run()+ }+async close(): Promise<void> {for (const client of this.wss.clients) client.terminate()await new Promise<void>((done) => this.wss.close(() => done()))@@ -119,7 +217,11 @@ export class FakeBackend {const bearer = /^Bearer (.+)$/.exec(String(req.headers.authorization ?? ''))?.[1]const offered = String(req.headers['sec-websocket-protocol'] ?? '').split(',')[0]?.trim()const token = bearer ?? offered- return token ? [...this.machines.values()].find((m) => m.token === token) : undefined+ if (!token) return undefined+ const machine = [...this.machines.values()].find((m) => m.token === token)+ // A phone's token is the account's, not a machine's: it may open the web socket and nothing else.+ if (!machine && this.userTokens.has(token)) return { machineId: '', computerId: '', name: 'phone', token }+ return machine}private upgrade(req: IncomingMessage, socket: Duplex, head: Buffer): void {@@ -129,7 +231,13 @@ export class FakeBackend {const refuse = (status: number, text: string): void => {socket.end(`HTTP/1.1 ${status} ${text}\r\nConnection: close\r\nContent-Length: 0\r\n\r\n`)}+ if (this.down) { refuse(503, 'Service Unavailable'); return }if (!machine) { refuse(401, 'Unauthorized'); return }+ if (url.pathname === '/api/web-ws') {+ this.wss.handleUpgrade(req, socket, head, (ws) => this.attachWeb(ws))+ return+ }+ if (!machine.machineId) { refuse(403, 'Forbidden'); return }if (url.pathname === '/api/adapter-ws') {// The node is the machine that signed in, on the computer it says it is.if (url.searchParams.get('computer') !== machine.computerId) { refuse(403, 'Forbidden'); return }@@ -151,21 +259,30 @@ export class FakeBackend {// What the real backend says first: the machine's name, and how many commanders are watching it.this.sendNode(machine.machineId, '', { type: 'machine_meta', payload: { name: machine.name } })this.sendClients(machine.machineId)+ this.toWebClients(machine.machineId, { type: 'node_status', payload: { online: true } })ws.on('message', (raw, binary) => { if (!binary) this.fromNode(machine.machineId, raw) })- ws.on('close', () => { if (this.nodes.get(machine.machineId) === ws) this.nodes.delete(machine.machineId) })+ ws.on('close', () => {+ if (this.nodes.get(machine.machineId) !== ws) return+ this.nodes.delete(machine.machineId)+ this.toWebClients(machine.machineId, { type: 'node_status', payload: { online: false, reason: 'node offline' } })+ })}private fromNode(machineId: string, raw: RawData): void {- let envelope: { t?: string; targetConnId?: string; commanderEligible?: boolean; frame?: Frame }+ let envelope: { t?: string; targetConnId?: string; commanderEligible?: boolean; webEligible?: boolean; frame?: Frame }try { envelope = JSON.parse(raw.toString()) } catch { return }if (envelope.t !== 'up' || !envelope.frame) returnconst tagged = { ...envelope.frame, machineId }if (envelope.targetConnId) {const owner = this.connOwners.get(envelope.targetConnId)if (owner?.machineId === machineId) this.sendDevice(owner.device, tagged)+ const web = [...this.webClients].find((client) => client.connId === envelope.targetConnId && client.machineId === machineId)+ if (web) this.toWeb(web, envelope.frame)return}if (envelope.commanderEligible) for (const device of this.devices) this.sendDevice(device, tagged)+ // The real hub: web clients get every broadcast that is not device-only.+ if (envelope.webEligible !== false) this.toWebClients(machineId, envelope.frame)}private sendNode(machineId: string, connId: string, frame: Frame): boolean {@@ -183,6 +300,57 @@ export class FakeBackend {})}+ // ── web clients (phones, browsers) ───────────────────────────────────────────────────────────++ private attachWeb(ws: WebSocket): void {+ const client: WebClient = { ws, connId: `web:${randomUUID()}`, machineId: null }+ this.webClients.add(client)+ ws.send(json({ type: 'connected', payload: { userId: 'e2e-user' } }))+ ws.on('message', (raw, binary) => { if (!binary) this.fromWeb(client, raw) })+ ws.on('close', () => {+ this.webClients.delete(client)+ const machineId = client.machineId+ if (machineId) this.relay(() => this.sendNode(machineId, client.connId, { type: '__client_disconnected', payload: {} }))+ })+ }++ private fromWeb(client: WebClient, raw: RawData): void {+ let frame: Frame+ try { frame = JSON.parse(raw.toString()) as Frame } catch { return }+ const type = frame.type ?? ''+ // The backend's own frames are never a client's to send, as the real hub refuses them.+ if (type.startsWith('__')) return+ if (type === 'machine_select') {+ const machineId = String(frame.payload?.machineId ?? '')+ if (!this.machines.has(machineId)) {+ client.ws.send(json({ type: 'machine_select_error', payload: { machineId, error: 'NOT_YOUR_MACHINE' } }))+ return+ }+ client.machineId = machineId+ client.ws.send(json({ type: 'connected', payload: { machineId, p2p: null } }))+ client.ws.send(json({ type: 'node_status', payload: { online: this.nodeUp(machineId) } }))+ return+ }+ const machineId = client.machineId+ if (!machineId) return+ const sent = this.webSent.get(client.connId) ?? []+ sent.push(frame)+ this.webSent.set(client.connId, sent)+ // Never answered when the node is not there, as the real hub's would not be.+ this.relay(() => this.sendNode(machineId, client.connId, frame))+ }++ private toWebClients(machineId: string, frame: Frame): void {+ for (const client of this.webClients) if (client.machineId === machineId) this.toWeb(client, frame)+ }++ private toWeb(client: WebClient, frame: Frame): void {+ const received = this.webReceived.get(client.connId) ?? []+ received.push(frame)+ this.webReceived.set(client.connId, received)+ this.relay(() => { if (client.ws.readyState === WebSocket.OPEN) client.ws.send(json(frame)) })+ }+// ── devices ──────────────────────────────────────────────────────────────────────────────────private attachDevice(ws: WebSocket): void {diff --git a/cli/e2e/harness/fleet.ts b/cli/e2e/harness/fleet.tsindex 9a99e3605..569e34dfc 100644--- a/cli/e2e/harness/fleet.ts+++ b/cli/e2e/harness/fleet.ts@@ -85,6 +85,59 @@ async function writeE2ee(daemon: IsolatedDaemon, files: Record<string, unknown>)for (const [name, value] of Object.entries(files)) await writeFile(join(dir, name), JSON.stringify(value, null, 2), { mode: 0o600 })}+/** A daemon for one machine of the account, signed in to the fake backend and nothing else. */+async function signedInDaemon(backend: FakeBackend, spec: FakeMachine, env: Record<string, string> = {}): Promise<IsolatedDaemon> {+ const daemon = await IsolatedDaemon.create({ env: { ...remoteSettings(backend), ADAPTER_COMPUTER_ID: spec.computerId, ...env } })+ // The daemon inherits the runner's environment, where a registry or a proxy may be named. None of+ // them is anywhere a fleet daemon may go.+ for (const [key, value] of Object.entries(daemon.env)) {+ const host = serverHost(value)+ if (host && !loopback(host)) delete daemon.env[key]+ }+ assertLocalOnly(daemon.env, backend)+ await signIn(daemon, spec)+ return daemon+}++export interface PhoneMachine {+ backend: FakeBackend+ machine: FleetMachine+ /** A phone of the account, paired with the machine: its key is trusted there as a web client. */+ phone: { identity: Identity; token: string }+ close(): Promise<void>+}++/** One signed-in machine and a phone paired with it, on the fake backend: what a phone reaches the+ * machine through, end to end encrypted and relayed. */+export async function startPhoneMachine(options: { env?: Record<string, string> } = {}): Promise<PhoneMachine> {+ const backend = await FakeBackend.start()+ const spec = { machineId: 'a1'.repeat(16), computerId: 'e2e-computer-0000-0000-00000000000a', name: 'machine-a', token: 'e2e-token-machine-a', identity: newIdentity() }+ backend.addMachine(spec)+ const phone = { identity: newIdentity(), token: 'e2e-token-phone' }+ backend.addUser(phone.token)+ let daemon: IsolatedDaemon | null = null+ try {+ daemon = await signedInDaemon(backend, spec, options.env)+ await writeE2ee(daemon, {+ 'identity.json': { priv: b64e(spec.identity.priv), pub: b64e(spec.identity.pub) },+ 'paired.json': [{ identityPub: b64e(phone.identity.pub), label: 'phone', pairedAt: Date.now(), role: 'web' }],+ })+ await daemon.start()+ const started = daemon+ return {+ backend, machine: { ...spec, daemon: started }, phone,+ async close() {+ await started.close().catch(() => {})+ await backend.close()+ },+ }+ } catch (error) {+ await daemon?.close().catch(() => {})+ await backend.close()+ throw error+ }+}+export async function startFleet(options: FleetOptions = {}): Promise<Fleet> {const backend = await FakeBackend.start()const specs: Array<FakeMachine & { identity: Identity }> = [@@ -100,16 +153,8 @@ export async function startFleet(options: FleetOptions = {}): Promise<Fleet> {const own: DaemonOptions['env'] = index === 0? { ...(dial ? { CABLE_DISABLE: 'false', HARNESSD_TEST_DIAL_PORT: dial.path } : {}), ...options.envA }: { ...options.envB }- const daemon = await IsolatedDaemon.create({ env: { ...remoteSettings(backend), ADAPTER_COMPUTER_ID: spec.computerId, ...own } })+ const daemon = await signedInDaemon(backend, spec, own)created.push(daemon)- // The daemon inherits the runner's environment, where a registry or a proxy may be named. None of- // them is anywhere a fleet daemon may go.- for (const [key, value] of Object.entries(daemon.env)) {- const host = serverHost(value)- if (host && !loopback(host)) delete daemon.env[key]- }- assertLocalOnly(daemon.env, backend)- await signIn(daemon, spec)machines.push({ ...spec, daemon })}const [a, b] = machinesdiff --git a/cli/e2e/harness/relayPhone.ts b/cli/e2e/harness/relayPhone.tsnew file mode 100644index 000000000..dfbc14dc0--- /dev/null+++ b/cli/e2e/harness/relayPhone.ts@@ -0,0 +1,197 @@+/**+ * A phone, as the relay sees one: a web socket to the (fake) backend's `/api/web-ws`, bound to one+ * machine with `machine_select`, and an end-to-end encrypted session with that machine's daemon opened+ * through the relay (`e2e_hello` / `e2e_welcome`). The crypto is the CLI's own client role+ * (src/lib/e2ee/relayClient.ts, what this daemon runs when it relays the desktop to another machine), and+ * the behaviour around it is that relay client's (src/lib/remoteRelay.ts):+ * - a frame that does not open under the session is dropped, never shown;+ * - `node_status {online: false}` retires the session (the daemon's next process has other keys);+ * - nothing is sent without a ready session: a send then says it did not go (`false`), and a request+ * fails at once, so a message can be refused visibly but is never queued into nowhere.+ *+ * Everything it opened is kept in `frames`, in arrival order, across sessions.+ */+import { randomUUID } from 'node:crypto'+import { WebSocket } from 'ws'+import type { Identity } from '../../src/lib/e2ee/core.js'+import { RelaySessionCrypto } from '../../src/lib/e2ee/relayClient.js'+import type { FakeBackend } from './fakeBackend.js'++export type PhoneFrame = { type: string; payload?: Record<string, any>; agentId?: string; replay?: boolean; [key: string]: unknown }++export interface RelayPhoneOptions {+ backend: FakeBackend+ machineId: string+ identity: Identity+ /** The machine's identity key, pinned when the phone was paired: the welcome must be signed by it. */+ machinePub: Uint8Array+ token: string+ /** Keep the socket and the session through `node_status {online:false}`, as the phone app does+ * (mobile app_state.dart `_applyNodeStatus`), instead of retiring them as the CLI's relay client does. */+ keepSessionWhenOffline?: boolean+}++export class RelayPhone {+ /** Every frame opened, in arrival order, with the session it came on. */+ readonly frames: Array<PhoneFrame & { session: number }> = []+ /** Every sealed frame sent, as it left: what a relay could replay. */+ readonly sealed: PhoneFrame[] = []+ /** Frames the session refused to open: sealed for another session, replayed, or garbage. */+ dropped = 0+ /** How many sessions it has opened. */+ sessions = 0+ connId: string | null = null+ private ws: WebSocket | null = null+ private crypto: RelaySessionCrypto | null = null+ private waiters: Array<{ test: (frame: PhoneFrame) => boolean; done: (frame: PhoneFrame) => void }> = []+ private pending = new Map<string, { resolve: (payload: Record<string, any>) => void; reject: (error: Error) => void }>()+ /** The machine's broadcasts opened so far, by epoch, carried from session to session as the CLI's+ * relay pool carries them (src/lib/remoteRelay.ts `groupSeen`). */+ private readonly groupSeen = new Map<string, number>()++ constructor(private readonly options: RelayPhoneOptions) {}++ get ready(): boolean {+ return this.ws?.readyState === WebSocket.OPEN && this.crypto?.ready === true+ }++ /** Dial, bind to the machine and open a session. Rejects when any step fails or takes too long. */+ async open(ms = 20_000): Promise<void> {+ this.retire()+ const { backend, machineId, identity, machinePub, token } = this.options+ const ws = new WebSocket(`${backend.wsUrl}/api/web-ws?autonomousEnv=prod`, [token])+ ws.on('error', () => { /* 'close' follows */ })+ const crypto = new RelaySessionCrypto({ machineId, selfIdentity: identity, peerPub: machinePub, groupSeen: this.groupSeen })+ this.ws = ws+ this.crypto = crypto+ await new Promise<void>((resolve, reject) => {+ const timer = setTimeout(() => { fail(new Error('the phone could not open a session in time')) }, ms)+ const fail = (error: Error): void => { clearTimeout(timer); ws.terminate(); reject(error) }+ ws.on('message', (raw, binary) => {+ if (binary) return+ let frame: PhoneFrame+ try { frame = JSON.parse(raw.toString()) as PhoneFrame } catch { this.dropped++; return }+ if (!crypto.ready) {+ if (frame.type === 'connected' && frame.payload?.machineId === machineId) {+ ws.send(JSON.stringify(crypto.helloFrame()))+ } else if (frame.type === 'machine_select_error') {+ fail(new Error(`machine_select refused: ${frame.payload?.error}`))+ } else if (frame.type === 'e2e_denied') {+ fail(new Error(`the daemon denied the phone: ${frame.payload?.reason}`))+ } else if (frame.type === 'e2e_welcome') {+ if (!crypto.handleWelcome(frame.payload ?? {})) { fail(new Error('the welcome did not open')); return }+ clearTimeout(timer)+ this.sessions+++ resolve()+ }+ return+ }+ this.receive(ws, crypto, frame)+ })+ ws.on('open', () => ws.send(JSON.stringify({ type: 'machine_select', payload: { machineId } })))+ ws.on('close', () => {+ fail(new Error('the relay closed the phone\'s socket'))+ if (this.ws === ws) this.retire()+ })+ })+ this.connId = backend.webConnections(machineId).at(-1) ?? null+ }++ private receive(ws: WebSocket, crypto: RelaySessionCrypto, frame: PhoneFrame): void {+ if (frame.type === 'node_status') {+ this.record({ ...frame }, this.sessions)+ // As the CLI's relay client does: the daemon's next process knows nothing of these keys.+ if (frame.payload?.online === false && !this.options.keepSessionWhenOffline) ws.close(1012, 'remote machine disconnected')+ return+ }+ if (frame.type === 'e2e_session_unknown') {+ // The daemon has no session for this connection any more: what was sent on it was not taken.+ this.record({ ...frame }, this.sessions)+ ws.close(1012, 'remote session gone')+ return+ }+ if (frame.type === 'e2e_rekey') { crypto.handleRekey(frame.payload ?? {}); return }+ if (frame.type === 'e2e_welcome' || frame.type === 'connected') return+ const opened = crypto.unwrapIncoming(frame) as PhoneFrame | null+ // A payload sealed for another session, replayed, or not sealed at all where it must be: dropped.+ if (!opened || opened.payload?.__e2e) { this.dropped++; return }+ const requestId = opened.payload?.requestId+ if (typeof requestId === 'string' && opened.type.endsWith('_result') && this.pending.has(requestId)) {+ const waiting = this.pending.get(requestId)!+ this.pending.delete(requestId)+ waiting.resolve(opened.payload ?? {})+ }+ this.record(opened, this.sessions)+ }++ private record(frame: PhoneFrame, session: number): void {+ this.frames.push({ ...frame, session })+ for (const waiter of [...this.waiters]) {+ if (waiter.test(frame)) { this.waiters.splice(this.waiters.indexOf(waiter), 1); waiter.done(frame) }+ }+ }++ /** Let go of the socket and the session; anything waiting on them fails now, visibly. */+ private retire(): void {+ const ws = this.ws+ this.ws = null+ this.crypto = null+ this.connId = null+ for (const [id, waiting] of this.pending) { this.pending.delete(id); waiting.reject(new Error('the phone\'s session ended')) }+ if (ws && ws.readyState === WebSocket.OPEN) ws.close()+ }++ /** A sealed frame with no answer expected (a message). False when there is no session to send it on. */+ send(type: string, payload: Record<string, unknown>): boolean {+ if (!this.ready) return false+ const sealed = this.crypto!.wrapOutgoing({ type, payload }) as PhoneFrame+ this.sealed.push(sealed)+ this.ws!.send(JSON.stringify(sealed))+ return true+ }++ /** A sealed request and its sealed answer. Fails at once with no session, or when the session ends. */+ request(type: string, payload: Record<string, unknown> = {}, ms = 30_000): Promise<Record<string, any>> {+ if (!this.ready) return Promise.reject(new Error(`no session to ask ${type} on`))+ const requestId = randomUUID()+ return new Promise((resolve, reject) => {+ const timer = setTimeout(() => {+ this.pending.delete(requestId)+ reject(new Error(`${type} was not answered within ${ms}ms`))+ }, ms)+ this.pending.set(requestId, {+ resolve: (answer) => { clearTimeout(timer); resolve(answer) },+ reject: (error) => { clearTimeout(timer); reject(error) },+ })+ this.send(type, { ...payload, requestId })+ })+ }++ /** The first frame, past or future, that passes `test`. */+ waitFor(test: (frame: PhoneFrame) => boolean, ms = 30_000, what = 'a frame', since = 0): Promise<PhoneFrame> {+ const seen = this.frames.slice(since).find(test)+ if (seen) return Promise.resolve(seen)+ return new Promise((resolve, reject) => {+ const timer = setTimeout(() => {+ this.waiters = this.waiters.filter((waiter) => waiter.done !== done)+ reject(new Error(`the phone waited ${ms}ms for ${what}; last frames: ${JSON.stringify(this.frames.slice(-8).map((f) => f.type))}`))+ }, ms)+ const done = (frame: PhoneFrame) => { clearTimeout(timer); resolve(frame) }+ this.waiters.push({ test, done })+ })+ }++ /** Only frames that arrive after this call. */+ next(test: (frame: PhoneFrame) => boolean, ms = 30_000, what = 'a frame'): Promise<PhoneFrame> {+ return this.waitFor(test, ms, what, this.frames.length)+ }++ /** The socket gone with no close frame, as a phone losing its network. */+ drop(): void {+ this.ws?.terminate()+ }++ close(): void {+ this.retire()+ }+}diff --git a/cli/e2e/relay.e2e.ts b/cli/e2e/relay.e2e.tsnew file mode 100644index 000000000..84756ba29--- /dev/null+++ b/cli/e2e/relay.e2e.ts@@ -0,0 +1,307 @@+/**+ * The relay link under stress, as a phone sees it (round 34). A signed-in daemon on the fake backend+ * (harness/fakeBackend.ts), a phone paired with it reaching it through the fake relay with an end-to-end+ * encrypted session (harness/relayPhone.ts, the CLI's own relay client role), and the desktop window on+ * the machine itself. Then the relay misbehaves:+ * - it drops mid-turn and comes back;+ * - the backend restarts;+ * - it is slow, by seconds;+ * - it sends garbage, or replays an old sealed frame;+ * - and the daemon restarts while the phone is connected.+ *+ * What correct is, from the code:+ * - The phone resyncs after every reconnect: it opens a new session and re-reads the conversation+ * (`session_get`). Every message is in that history once.+ * - Nothing the daemon sends after that re-read announces, as live news, a turn already in it: a phone+ * would show it twice (notify a turn that ended long ago, open a turn that is over).+ * - A message the phone tries while it has no session is refused at once and never appears. One sent+ * with a session appears once.+ * - The relay is not trusted: garbage costs it nothing but the frame, a replayed sealed frame never+ * becomes a second turn, and a frame sealed for a dead session opens nowhere.+ *+ * Nothing here reaches beyond this machine: the daemon's every server setting is the fake backend's,+ * and `assertLocalOnly` refuses one that is not.+ */+import { mkdirSync } from 'node:fs'+import { join } from 'node:path'+import { afterEach, describe, expect, it, onTestFailed } from 'vitest'+import { LocalClient, type Frame } from './harness/client.js'+import { until, type IsolatedDaemon } from './harness/daemon.js'+import { startPhoneMachine, type PhoneMachine } from './harness/fleet.js'+import { RelayPhone, type PhoneFrame } from './harness/relayPhone.js'++const row = async (client: LocalClient, agentId: string) =>+ ((await client.request<{ agents: Array<Record<string, any>> }>('agents_list', { includeStopped: true }, 30_000)).agents)+ .find((agent) => agent.id === agentId)++/** A Claude Code agent started from the desktop window, bound to its conversation. */+async function boundAgent(daemon: IsolatedDaemon, desk: LocalClient, name: string): Promise<{ agentId: string; sessionId: string }> {+ const cwd = join(daemon.projectsDir, name)+ mkdirSync(cwd, { recursive: true })+ const created = await desk.request('agent_create', { engine: 'claude', cwd, bypassPermission: true }, 60_000)+ expect(created.error, JSON.stringify(created)).toBeUndefined()+ const agentId: string = created.agent.id+ const sessionId = await until('the agent to bind its conversation', async () => (await row(desk, agentId))?.sessionId || null, 45_000, 500)+ return { agentId, sessionId }+}++const isTurn = (type: string, agentId: string) => (frame: Frame | PhoneFrame) => frame.type === type && frame.agentId === agentId++/** A turn from the desktop window, to its end. */+async function deskTurn(desk: LocalClient, agentId: string, content: string): Promise<void> {+ const ended = desk.next(isTurn('turn_ended', agentId), 45_000, `turn_ended (${content})`)+ desk.send('message', { agentId, content })+ await ended+}++/** What the phone shows of a conversation after it re-reads it: each message asked, oldest first. */+async function history(phone: RelayPhone, sessionId: string): Promise<string[]> {+ const page = await phone.request('session_get', { sessionId, limit: 500 }, 30_000)+ expect(page.error, JSON.stringify(page)).toBeUndefined()+ return (page.events as Array<{ type: string; payload: Record<string, any> }>)+ .filter((event) => event.type === 'user_message')+ .map((event) => String(event.payload.content ?? event.payload.text ?? ''))+}++/** Each message once, in this order, and nothing else. */+const once = (seen: string[], expected: string[]) => {+ for (const message of expected) expect(seen.filter((m) => m === message), `"${message}" in ${JSON.stringify(seen)}`).toHaveLength(1)+}++/** The live turn news the phone got after frame `from`: what it would show as happening now. */+const liveNews = (phone: RelayPhone, agentId: string, from: number) => phone.frames.slice(from)+ .filter((frame) => (frame.type === 'turn_started' || frame.type === 'turn_ended') && frame.agentId === agentId && !frame.replay && !frame.payload?.replay)++describe('the relay link under stress, as a phone sees it', () => {+ let world: PhoneMachine | undefined+ let phone: RelayPhone | undefined+ let desk: LocalClient | undefined+ afterEach(async () => {+ phone?.close()+ desk?.close()+ await world?.close()+ world = undefined; phone = undefined; desk = undefined+ })++ const fresh = async (env: Record<string, string> = {}) => {+ const w = await startPhoneMachine({ env })+ world = w+ const { backend, machine } = w+ onTestFailed(() => {+ console.log(`---- daemon log\n${machine.daemon.log().split('\n').slice(-150).join('\n')}`)+ console.log(`---- fake backend saw\n${backend.seen.slice(-40).join('\n')}`)+ console.log(`---- the phone opened\n${phone?.frames.map((f) => `${f.session}:${f.type}${f.replay || f.payload?.replay ? '(replay)' : ''}`).join(' ') ?? '(no phone)'}`)+ })+ await until('the machine to be connected to the backend', () => backend.nodeUp(machine.machineId) || null, 30_000)+ desk = await LocalClient.connect(machine.daemon, { machineId: machine.machineId })+ const agent = await boundAgent(machine.daemon, desk, 'on-a')+ phone = new RelayPhone({ backend, machineId: machine.machineId, identity: w.phone.identity, machinePub: machine.identity.pub, token: w.phone.token })+ await phone.open()+ return { w, backend, machine, desk, phone, ...agent }+ }+ /** The phone back: once the machine is on the relay again, a new session, retried until it opens. */+ const reopen = async (w: PhoneMachine, p: RelayPhone) => {+ await until('the machine to be back on the relay', () => w.backend.nodeUp(w.machine.machineId) || null, 60_000, 200)+ await until('the phone to open a new session', async () => { await p.open(10_000); return true }, 60_000, 500)+ }++ it('a turn from the phone runs, and its news reaches the phone once, sealed', async () => {+ const { phone, agentId, sessionId, backend, machine } = await fresh()+ const started = phone.next(isTurn('turn_started', agentId), 30_000, 'turn_started on the phone')+ const ended = phone.next(isTurn('turn_ended', agentId), 30_000, 'turn_ended on the phone')+ expect(phone.send('message', { agentId, content: 'from the phone' })).toBe(true)+ expect((await started).payload?.userMessage).toBe('from the phone')+ await ended+ once(await history(phone, sessionId), ['from the phone'])+ // The relay saw only sealed payloads both ways.+ const relayed = [...(backend.webSent.get(phone.connId!) ?? []), ...(backend.webReceived.get(phone.connId!) ?? [])]+ expect(relayed.filter((f) => f.type === 'message' || f.type === 'turn_started').every((f) => (f.payload as any)?.__e2e)).toBe(true)+ expect(JSON.stringify(relayed)).not.toContain('from the phone')+ expect(machine.daemon.log()).not.toMatch(/getaddrinfo|ENOTFOUND|autonomous\.ai/)+ })++ it('the relay drops mid-turn and comes back: no turn lost or shown twice, and a message tried meanwhile is refused visibly', async () => {+ const { w, phone, desk, agentId, sessionId, backend } = await fresh()+ const started = phone.next(isTurn('turn_started', agentId), 30_000, 'the slow turn starting')+ expect(phone.send('message', { agentId, content: '!slow 4000 mid-turn' })).toBe(true)+ await started+ // A message the relay is still holding (it is slow for a moment) when it goes: both its ends drop,+ // and nothing connects for a while.+ const deskSawEnd = desk.next(isTurn('turn_ended', agentId), 30_000, 'the slow turn ending')+ backend.latencyMs = 1_500+ expect(phone.send('message', { agentId, content: 'in flight when the relay drops' })).toBe(true)+ backend.goDown()+ backend.latencyMs = 0+ await until('the phone to know it has no session', () => !phone.ready || null, 10_000, 50)+ // Tried while down: refused, visibly, right away.+ expect(phone.send('message', { agentId, content: 'tried while the relay is down' })).toBe(false)+ await expect(phone.request('agents_list')).rejects.toThrow(/no session/)+ // The turn ends meanwhile, and the desktop window starts and finishes another one.+ await deskSawEnd+ await deskTurn(desk, agentId, 'from the desk while the relay is down')+ backend.comeUp()+ await reopen(w, phone)+ const resynced = phone.frames.length+ const asked = await history(phone, sessionId)+ once(asked, ['!slow 4000 mid-turn', 'from the desk while the relay is down'])+ expect(asked).not.toContain('tried while the relay is down')+ // Lost in the relay: the daemon never had it. No message is acknowledged end to end, so reading the+ // conversation again is the only way the phone learns it was not delivered (listed in the report).+ expect(asked).not.toContain('in flight when the relay drops')+ // A while later: nothing announced as live what the phone already read.+ await new Promise((done) => setTimeout(done, 3_000))+ expect(liveNews(phone, agentId, resynced)).toEqual([])+ // And the link works: a new turn from the phone, its news once.+ const next = phone.next(isTurn('turn_ended', agentId), 30_000, 'a turn after the relay came back')+ expect(phone.send('message', { agentId, content: 'after the relay came back' })).toBe(true)+ await next+ once(await history(phone, sessionId), ['!slow 4000 mid-turn', 'from the desk while the relay is down', 'after the relay came back'])+ expect(liveNews(phone, agentId, resynced).filter((f) => f.type === 'turn_ended')).toHaveLength(1)+ })++ it('the backend restarts: the daemon and the phone come back to it, and nothing is lost or doubled', async () => {+ const { w, phone, desk, agentId, sessionId, backend, machine } = await fresh()+ const ended = phone.next(isTurn('turn_ended', agentId), 30_000, 'a turn before the restart')+ expect(phone.send('message', { agentId, content: 'before the restart' })).toBe(true)+ await ended+ const restarting = backend.restart(3_000)+ await until('the phone to know it has no session', () => !phone.ready || null, 10_000, 50)+ await deskTurn(desk, agentId, 'from the desk during the restart')+ await restarting+ await reopen(w, phone)+ const resynced = phone.frames.length+ once(await history(phone, sessionId), ['before the restart', 'from the desk during the restart'])+ await new Promise((done) => setTimeout(done, 3_000))+ expect(liveNews(phone, agentId, resynced)).toEqual([])+ const after = phone.next(isTurn('turn_ended', agentId), 30_000, 'a turn after the restart')+ expect(phone.send('message', { agentId, content: 'after the restart' })).toBe(true)+ await after+ expect(machine.daemon.coresStarted()).toBe(1)+ })++ it('the relay is slow by seconds: the session opens, requests are answered, and turns arrive in order, once', async () => {+ const w = await startPhoneMachine()+ world = w+ const { backend, machine } = w+ onTestFailed(() => { console.log(`---- daemon log\n${machine.daemon.log().split('\n').slice(-120).join('\n')}`) })+ await until('the machine to be connected to the backend', () => backend.nodeUp(machine.machineId) || null, 30_000)+ desk = await LocalClient.connect(machine.daemon, { machineId: machine.machineId })+ const { agentId, sessionId } = await boundAgent(machine.daemon, desk, 'slow-relay')+ backend.latencyMs = 2_500+ phone = new RelayPhone({ backend, machineId: machine.machineId, identity: w.phone.identity, machinePub: machine.identity.pub, token: w.phone.token })+ await phone.open(30_000)+ const listed = await phone.request('agents_list', {}, 30_000)+ expect(listed.agents.map((agent: Record<string, any>) => agent.id)).toContain(agentId)+ const from = phone.frames.length+ for (const content of ['one', 'two', 'three']) {+ const ended = phone.next(isTurn('turn_ended', agentId), 45_000, `turn_ended (${content})`)+ expect(phone.send('message', { agentId, content })).toBe(true)+ await ended+ }+ const starts = phone.frames.slice(from).filter((f) => f.type === 'turn_started' && f.agentId === agentId).map((f) => f.payload?.userMessage)+ expect(starts).toEqual(['one', 'two', 'three'])+ expect(phone.frames.slice(from).filter((f) => f.type === 'turn_ended' && f.agentId === agentId)).toHaveLength(3)+ once(await history(phone, sessionId), ['one', 'two', 'three'])+ expect(backend.nodeUp(machine.machineId)).toBe(true)+ })++ it('the relay sends garbage or replays an old sealed frame: no second turn, no crash, the link goes on', async () => {+ const { w, phone, desk, agentId, sessionId, backend, machine } = await fresh()+ const ended = phone.next(isTurn('turn_ended', agentId), 30_000, 'the original turn')+ expect(phone.send('message', { agentId, content: 'said once' })).toBe(true)+ await ended+ const original = phone.sealed.find((frame) => frame.type === 'message')!+ const connId = phone.connId!+ // Garbage towards the daemon, on the phone's connection and on the backend's own address.+ for (const garbage of ['not json', '{"t":"down"}', '{"t":"down","connId":"x","frame":null}', JSON.stringify({ t: 'down', connId, frame: { type: 'message' } })]) {+ expect(backend.injectDown(machine.machineId, connId, garbage)).toBe(true)+ }+ backend.injectDown(machine.machineId, connId, { type: 'message', payload: { agentId, content: 'unsealed from the relay' } })+ backend.injectDown(machine.machineId, connId, { type: 'message', payload: { __e2e: { v: 1, k: 'p', n: 'x', ct: 7 } } })+ backend.injectDown(machine.machineId, connId, { type: 'message', payload: { __e2e: { v: 1, k: 'p', n: 999, ct: 'AAAA' } } })+ backend.injectDown(machine.machineId, connId, { type: '__clients', payload: { commander: 9 } })+ backend.injectDown(machine.machineId, connId, { type: 'machine_meta', payload: { name: 'renamed by the relay' } })+ backend.injectDown(machine.machineId, connId, { type: 'e2e_hello', payload: { identityPub: 'AAAA', ephPub: 'AAAA', sig: 'AAAA' } })+ // The phone's own sealed message, replayed on its connection: the session has seen that counter.+ for (let n = 0; n < 3; n++) backend.injectDown(machine.machineId, connId, original)+ // Garbage and an old sealed broadcast towards the phone: dropped there, never shown.+ const capturedStart = (backend.webReceived.get(connId) ?? []).find((frame) => frame.type === 'turn_started')!+ const droppedBefore = phone.dropped+ backend.injectUp(connId, 'not json either')+ backend.injectUp(connId, capturedStart)+ await until('the phone to drop what the relay injected', () => phone.dropped >= droppedBefore + 2 || null, 10_000, 100)+ // Nothing became a turn: the conversation still has the message once.+ await new Promise((done) => setTimeout(done, 3_000))+ once(await history(phone, sessionId), ['said once'])+ expect(await history(phone, sessionId)).not.toContain('unsealed from the relay')+ // Unsealed client frames are refused, and the backend's own frames on a client's address ignored.+ expect(machine.daemon.log()).toContain('refusing plaintext "message" from relay')+ expect(machine.daemon.log()).toContain('ignoring "machine_meta" from relay')+ expect(machine.daemon.log()).toContain('ignoring "__clients" from relay')+ // The phone comes back on a new connection; its old sealed message replayed there opens nowhere.+ phone.drop()+ await reopen(w, phone)+ backend.injectDown(machine.machineId, phone.connId!, original)+ // And a broadcast sealed before, replayed to the new session, is not shown as news.+ const resynced = phone.frames.length+ backend.injectUp(phone.connId!, capturedStart)+ await new Promise((done) => setTimeout(done, 3_000))+ once(await history(phone, sessionId), ['said once'])+ expect(liveNews(phone, agentId, resynced), 'an old sealed turn_started replayed to a new session').toEqual([])+ // The link goes on, and the daemon never restarted.+ const after = phone.next(isTurn('turn_ended', agentId), 30_000, 'a turn after the garbage')+ expect(phone.send('message', { agentId, content: 'after the garbage' })).toBe(true)+ await after+ expect(machine.daemon.coresStarted()).toBe(1)+ expect(await row(desk, agentId)).toMatchObject({ id: agentId })+ })++ it('the daemon restarts under a phone that keeps its socket, as the phone app does: what it sends on its dead session is refused visibly', async () => {+ const { w, machine, agentId, sessionId } = await fresh()+ // Its own phone: one that keeps its socket and session through the machine going offline.+ phone!.close()+ const keeper = new RelayPhone({ backend: w.backend, machineId: machine.machineId, identity: w.phone.identity, machinePub: machine.identity.pub, token: w.phone.token, keepSessionWhenOffline: true })+ phone = keeper+ await keeper.open()+ desk!.close()+ const back = keeper.next((frame) => frame.type === 'node_status' && frame.payload?.online === true, 60_000, 'node_status online on the phone')+ await machine.daemon.restart()+ await back+ expect(keeper.ready).toBe(true)+ // Its session is from the daemon's last process. A message sent on it must not vanish silently, and+ // a request must not wait out its timeout: the daemon says the session is gone.+ const told = keeper.next((frame) => frame.type === 'e2e_session_unknown', 10_000, 'the daemon saying the session is gone')+ expect(keeper.send('message', { agentId, content: 'on a session the daemon no longer has' })).toBe(true)+ const refused = await told+ expect(refused.payload).toMatchObject({ refused: { type: 'message' } })+ expect(keeper.ready).toBe(false)+ // The phone opens a new session and sends again: delivered, once.+ await reopen(w, keeper)+ const ended = keeper.next(isTurn('turn_ended', agentId), 45_000, 'the message sent again')+ expect(keeper.send('message', { agentId, content: 'sent again on a new session' })).toBe(true)+ await ended+ const asked = await history(keeper, sessionId)+ expect(asked).not.toContain('on a session the daemon no longer has')+ once(asked, ['sent again on a new session'])+ })++ it('the daemon restarts while the phone is connected: the phone is told, opens a new session, and nothing is lost or doubled', async () => {+ const { w, phone, desk, agentId, sessionId, machine } = await fresh()+ const ended = phone.next(isTurn('turn_ended', agentId), 30_000, 'a turn before the restart')+ expect(phone.send('message', { agentId, content: 'before the daemon restarts' })).toBe(true)+ await ended+ desk.close()+ const offline = phone.next((frame) => frame.type === 'node_status' && frame.payload?.online === false, 30_000, 'node_status offline on the phone')+ await machine.daemon.restart()+ await offline+ expect(phone.ready).toBe(false)+ await reopen(w, phone)+ const resynced = phone.frames.length+ once(await history(phone, sessionId), ['before the daemon restarts'])+ const after = phone.next(isTurn('turn_ended', agentId), 45_000, 'a turn after the daemon restarted')+ expect(phone.send('message', { agentId, content: 'after the daemon restarted' })).toBe(true)+ await after+ once(await history(phone, sessionId), ['before the daemon restarts', 'after the daemon restarted'])+ expect(liveNews(phone, agentId, resynced).filter((f) => f.type === 'turn_ended')).toHaveLength(1)+ })+})diff --git a/cli/src/backendSocket.spec.ts b/cli/src/backendSocket.spec.tsindex 87402223e..e60cc6978 100644--- a/cli/src/backendSocket.spec.ts+++ b/cli/src/backendSocket.spec.ts@@ -148,6 +148,30 @@ describe('safe session close RPC', () => {describe('reviewed permanent deletion RPCs', () => {afterEach(() => vi.restoreAllMocks())+ it('tells a relayed client whose sealed frame finds no session that its session is gone, and only then', async () => {+ const socket = new BackendSocket('fixture')+ const internals = socket as any+ const sent = () => internals.queue.map((item: { msg: unknown }) => item.msg)+ const sealed = { type: 'message', payload: { __e2e: { v: 1, k: 'p', n: 3, ct: 'fixture' } } }+ await internals.dispatchDown(sealed, 'web:gone')+ expect(sent()).toContainEqual({ t: 'up', targetConnId: 'web:gone', frame: { type: 'e2e_session_unknown', payload: { refused: { type: 'message', n: 3 } } } })+ // A frame that does not open on a live session is dropped without a word.+ const before = sent().length+ vi.spyOn(internals.e2ee, 'sessionGone').mockReturnValue(null)+ await internals.dispatchDown(sealed, 'web:live')+ expect(sent()).toHaveLength(before)+ })++ it('tells a relayed client whose terminal bytes find no session that its session is gone', async () => {+ const socket = new BackendSocket('fixture')+ const internals = socket as any+ const { sealTerminalBinary } = await import('./lib/terminalBinary.js')+ const raw = sealTerminalBinary(new Uint8Array(32).fill(7), 9, { kind: 1, streamId: '00112233-4455-6677-8899-aabbccddeeff', seq: 1, bytes: new TextEncoder().encode('ls\r'), compressed: false })!+ internals.enqueueTerminalBinary('web:gone', raw)+ await internals.downChains.get('web:gone')+ expect(internals.queue.map((item: { msg: unknown }) => item.msg)).toContainEqual({ t: 'up', targetConnId: 'web:gone', frame: { type: 'e2e_session_unknown', payload: { refused: { type: 'terminal_binary', kind: 1, n: 9 } } } })+ })+it.each(['agent_purge', 'agent_worktree_delete'])('%s requires an owner, explicit identity and a review token', async type => {const socket = new BackendSocket('fixture')const internals = socket as any, frames: any[] = []diff --git a/cli/src/backendSocket.ts b/cli/src/backendSocket.tsindex 785813400..5e09eee20 100644--- a/cli/src/backendSocket.ts+++ b/cli/src/backendSocket.ts@@ -1414,6 +1414,7 @@ export class BackendSocket {.then(async () => {const clear = this.e2ee.unwrapTerminalBinary(connId, raw)if (clear) await this.terminalStreams?.handleBinary(connId, clear)+ else { const gone = this.e2ee.terminalSessionGone(connId, raw); if (gone) this.sendTo(connId, gone) }}).catch((err) => {console.error('[backend] binary terminal dispatch failed:', err instanceof Error ? err.message : err)@@ -1594,7 +1595,8 @@ export class BackendSocket {}} else if (wrapped) {const dec = this.e2ee.unwrapDown(connId, frame)- if (!dec) return+ // Sealed for a session this process never had: told, rather than dropped without a word.+ if (!dec) { const gone = this.e2ee.sessionGone(connId, frame); if (gone) this.sendTo(connId, gone); return }frame = dec} else {console.warn(`[backend] refusing plaintext ${logSafeType(type)} from ${from} — E2EE required`)diff --git a/cli/src/lib/e2ee/manager.test.ts b/cli/src/lib/e2ee/manager.test.tsindex 6a43e8cab..c8db6ee23 100644--- a/cli/src/lib/e2ee/manager.test.ts+++ b/cli/src/lib/e2ee/manager.test.ts@@ -611,3 +611,69 @@ describe('E2eeManager persistent remote-password pairing', () => {expect(takeLast('e2e_pw_pair_result').payload).toMatchObject({ ok: false, error: 'NO_REMOTE_PASSWORD' })})})++describe('a sealed frame for a session this process never had', () => {+ const sealed = { type: 'message', payload: { __e2e: { v: 1, k: 'p', n: 7, ct: 'AAAA' } } }++ it('tells the client its session is gone, naming the frame, at most once a second per connection', () => {+ let now = 1_000+ const mgr = new makeManager({ machineId: AGENT, sendTo: () => {}, isConnected: () => true }, () => now)+ expect(mgr.sessionGone('conn-old', sealed)).toEqual({ type: 'e2e_session_unknown', payload: { refused: { type: 'message', n: 7 } } })+ expect(mgr.sessionGone('conn-old', sealed)).toBeNull()+ now += 999+ expect(mgr.sessionGone('conn-old', sealed)).toBeNull()+ now += 1+ expect(mgr.sessionGone('conn-old', sealed)).not.toBeNull()+ // Another connection is its own; a frame with no counter to name is named by its type.+ expect(mgr.sessionGone('conn-other', { type: 'agents_list', payload: { __e2e: { n: 'x' } } })).toEqual({ type: 'e2e_session_unknown', payload: { refused: { type: 'agents_list' } } })+ expect(mgr.sessionGone('conn-bare', { type: 'agents_list' })).toEqual({ type: 'e2e_session_unknown', payload: { refused: { type: 'agents_list' } } })+ })++ it('says nothing to a connection that has a session: a frame that does not open there is the relay\'s', () => {+ const h = machine()+ const peer = new WebPeer()+ h.mgr.trustPeer({ pub: C.b64e(peer.identity.pub), label: 'phone' })+ h.mgr.handleFrame('live', peer.hello())+ expect(h.mgr.hasSession('live')).toBe(true)+ expect(h.mgr.sessionGone('live', sealed)).toBeNull()+ })++ it('remembers a bounded number of the connections it told, the oldest let go first', () => {+ const mgr = new makeManager({ machineId: AGENT, sendTo: () => {}, isConnected: () => true }, () => 5_000)+ for (let n = 0; n <= 256; n++) expect(mgr.sessionGone(`conn-${n}`, sealed)).not.toBeNull()+ // The first was let go, so it is told again at once; the last is still remembered.+ expect(mgr.sessionGone('conn-0', sealed)).not.toBeNull()+ expect(mgr.sessionGone('conn-256', sealed)).toBeNull()+ })+})++describe('terminal bytes sealed for a session this process never had', () => {+ const keystrokes = async (counter: number) => {+ const { sealTerminalBinary } = await import('../terminalBinary.js')+ return sealTerminalBinary(new Uint8Array(32).fill(9), counter, { kind: 1, streamId: '00112233-4455-6677-8899-aabbccddeeff', seq: 1, bytes: new TextEncoder().encode('ls\r'), compressed: false })!+ }++ it('tells the client its session is gone, naming the bytes by their clear kind and counter', async () => {+ const mgr = new makeManager({ machineId: AGENT, sendTo: () => {}, isConnected: () => true }, () => 1_000)+ expect(mgr.terminalSessionGone('conn-old', await keystrokes(5))).toEqual({ type: 'e2e_session_unknown', payload: { refused: { type: 'terminal_binary', kind: 1, n: 5 } } })+ // Bytes that are not a terminal frame at all are named by what they claim to be.+ expect(mgr.terminalSessionGone('conn-garbled', new Uint8Array([1, 2, 3]))).toEqual({ type: 'e2e_session_unknown', payload: { refused: { type: 'terminal_binary' } } })+ })++ it('shares the once-a-second allowance per connection with sealed frames', async () => {+ let now = 1_000+ const mgr = new makeManager({ machineId: AGENT, sendTo: () => {}, isConnected: () => true }, () => now)+ expect(mgr.sessionGone('conn-old', { type: 'message', payload: { __e2e: { n: 1 } } })).not.toBeNull()+ expect(mgr.terminalSessionGone('conn-old', await keystrokes(2))).toBeNull()+ now += 1_000+ expect(mgr.terminalSessionGone('conn-old', await keystrokes(3))).not.toBeNull()+ })++ it('says nothing to a connection that has a session', async () => {+ const h = machine()+ const peer = new WebPeer()+ h.mgr.trustPeer({ pub: C.b64e(peer.identity.pub), label: 'phone' })+ h.mgr.handleFrame('live', peer.hello())+ expect(h.mgr.terminalSessionGone('live', await keystrokes(4))).toBeNull()+ })+})diff --git a/cli/src/lib/e2ee/manager.ts b/cli/src/lib/e2ee/manager.tsindex dd63184fc..95ebea374 100644--- a/cli/src/lib/e2ee/manager.ts+++ b/cli/src/lib/e2ee/manager.ts@@ -73,6 +73,10 @@ interface Session {terminalC2sRecv: ReplayWindow}+/** A client told its session is gone is told again no sooner than this, and only so many are+ * remembered: a relay injecting sealed frames for made-up connections gets a bounded echo. */+const SESSION_GONE_EVERY_MS = 1_000+const SESSION_GONE_REMEMBERED = 256const PAIR_TTL_MS = 60_000const ROUND_TIMEOUT_MS = 15_000const RATE_WINDOW_MS = 5 * 60_000@@ -149,6 +153,7 @@ export class E2eeManager {private sessions = new Map<string, Session>()private attempts: number[] = [] // timestamps of FAILED pairings — anti online-guessing rate limitprivate pwSlots = new Map<string, PwPairSlot>() // connId -> in-progress password-PAKE attempt+ private sessionGoneTold = new Map<string, number>() // connId -> when it was last told its session is goneprivate now: () => number/** Local dashboard port (loopback) — surfaced to the web in e2e_status so it can link there to* approve pairing. Not sensitive (a localhost port); set by the adapter after the hook server binds. */@@ -416,6 +421,42 @@ export class E2eeManager {return { ...frame, payload: plain }}+ /**+ * What to tell a client whose sealed frame found no session here, or null when it has one (a frame+ * that does not open on a live session is the relay's or a replay, and is dropped without a word).+ *+ * No session most often means this machine's daemon restarted since the client's hello while the relay+ * kept the client's socket: the phone app keeps it through `node_status` offline (mobile+ * app_state.dart `_applyNodeStatus`). Its frames then reached a daemon that could not open them, and+ * were dropped silently: a message vanished, and a request waited out its timeout before the app+ * dialled again (round 34, `e2e/relay.e2e.ts`). Told, the client opens a new session at once and+ * knows the frame was not taken. The frame is named by its outer type and envelope counter, which the+ * relay already sees; nothing sealed is echoed. At most once a second per connection.+ */+ sessionGone(connId: string, frame: Frame): Frame | null {+ const counter = (frame.payload as { __e2e?: { n?: unknown } } | undefined)?.__e2e?.n+ return this.tellSessionGone(connId, { type: frame.type, ...(typeof counter === 'number' ? { n: counter } : {}) })+ }++ /** The same, for binary terminal bytes (keystrokes, a paste) sealed for a session this process never+ * had: those vanished silently too. Named by the frame's clear header, its kind and counter, which+ * the relay already reads; the same once-a-second allowance per connection as a sealed frame. */+ terminalSessionGone(connId: string, raw: Uint8Array): Frame | null {+ const envelope = parseTerminalBinaryEnvelope(raw)+ return this.tellSessionGone(connId, { type: 'terminal_binary', ...(envelope ? { kind: envelope.kind, n: envelope.counter } : {}) })+ }++ private tellSessionGone(connId: string, refused: Record<string, unknown>): Frame | null {+ if (this.sessions.has(connId)) return null+ const now = this.now()+ const last = this.sessionGoneTold.get(connId)+ if (last !== undefined && now - last < SESSION_GONE_EVERY_MS) return null+ this.sessionGoneTold.delete(connId)+ this.sessionGoneTold.set(connId, now)+ if (this.sessionGoneTold.size > SESSION_GONE_REMEMBERED) this.sessionGoneTold.delete(this.sessionGoneTold.keys().next().value!)+ return { type: 'e2e_session_unknown', payload: { refused } }+ }+// ── e2e_* frame handling (returns true if consumed) ──────────────────────────────────────────────handleFrame(connId: string, frame: Frame): boolean {diff --git a/cli/src/lib/e2ee/relayClient.ts b/cli/src/lib/e2ee/relayClient.tsindex cebbf1ba4..f3f418b8d 100644--- a/cli/src/lib/e2ee/relayClient.ts+++ b/cli/src/lib/e2ee/relayClient.ts@@ -22,8 +22,21 @@ export interface RelayCryptoDeps {machineId: stringselfIdentity: C.IdentitypeerPub: Uint8Array+ /**+ * The highest group-key counter this client has opened from the machine, per epoch, carried from one+ * of its sessions to the next. A broadcast is sealed under the machine's group key, which every+ * session of one machine process shares, so a session's own record, starting empty, let a relay+ * replay any broadcast the machine had sent since it started (a turn starting, an agent's frame) to+ * the client's next session, which opened it as news (round 34, `e2e/relay.e2e.ts`). Within an epoch+ * broadcasts only count up, so a later session never needs an older one. Never shared between two+ * sessions open at once: each connection is sent every broadcast, under the same counters.+ */+ groupSeen?: Map<string, number>}+/** The machine's epochs a client remembers counters for; one per process it ran, and per key rotation. */+export const GROUP_EPOCHS_KEPT = 16+export class RelaySessionCrypto {private readonly eph = C.newEphemeral()private c2s: Uint8Array | null = null@@ -39,9 +52,11 @@ export class RelaySessionCrypto {private p2pVersion = 0private viewerVersion = 0private strict = false- private groupRecv = new Map<string, number>() // epoch -> highest counter seen+ private readonly groupRecv: Map<string, number> // epoch -> highest counter seen- constructor(private readonly deps: RelayCryptoDeps) {}+ constructor(private readonly deps: RelayCryptoDeps) {+ this.groupRecv = deps.groupSeen ?? new Map()+ }get ready(): boolean { return this.c2s !== null && this.s2c !== null }get terminalP2pVersion(): number { return this.p2pVersion }@@ -134,6 +149,8 @@ export class RelaySessionCrypto {const plain = C.unwrapPayload(this.groupKey, env, type, frame.dbSessionId as string | undefined)if (plain === null) return nullthis.groupRecv.set(env.epoch, env.n)+ // The epochs of the machine's earlier processes: a few kept, the first seen let go first.+ if (this.groupRecv.size > GROUP_EPOCHS_KEPT) this.groupRecv.delete(this.groupRecv.keys().next().value!)return { ...frame, payload: plain }}if (!this.s2c) return nulldiff --git a/cli/src/lib/e2ee/relayLink.spec.ts b/cli/src/lib/e2ee/relayLink.spec.tsindex 629a94546..46d0ab995 100644--- a/cli/src/lib/e2ee/relayLink.spec.ts+++ b/cli/src/lib/e2ee/relayLink.spec.ts@@ -855,3 +855,148 @@ describe('RemoteRelayPool closes the upstream socket when the handshake fails',expect(closeCode).toBe(1000)})})++describe('a client\'s sessions with one machine never open the same broadcast twice', () => {+ let E2eeStore: typeof import('./store.js')['E2eeStore']+ let GROUP_EPOCHS_KEPT: number+ const fakeAuth = { accessToken: async () => 'unused-in-this-fake' } as unknown as import('../authSession.js').AuthSessionManager+ beforeAll(async () => {+ E2eeStore = (await import('./store.js')).E2eeStore+ GROUP_EPOCHS_KEPT = (await import('./relayClient.js')).GROUP_EPOCHS_KEPT+ })++ /** A machine process: its own group key and epoch, trusting `client` as a web client. */+ const machine = (client: import('./core.js').Identity) => {+ const inbox: Frame[] = []+ const manager = new E2eeManagerCtor({ machineId: MACHINE_ID, sendTo: (_connId, frame) => inbox.push(frame), isConnected: () => true })+ manager.trustPeer({ pub: C.b64e(client.pub), label: 'phone' })+ return { manager, inbox }+ }+ /** One session of `client` with that machine, on `connId`, carrying `groupSeen` over when given. */+ const session = (m: ReturnType<typeof machine>, client: import('./core.js').Identity, connId: string, groupSeen?: Map<string, number>) => {+ const crypto = new RelaySessionCrypto({ machineId: MACHINE_ID, selfIdentity: client, peerPub: new E2eeStore().getIdentity().pub, ...(groupSeen ? { groupSeen } : {}) })+ m.manager.handleFrame(connId, crypto.helloFrame())+ expect(crypto.handleWelcome(m.inbox.at(-1)!.payload as Record<string, unknown>)).toBe(true)+ return crypto+ }++ it('a broadcast the relay replays into the next session is not opened again, and a new one is', () => {+ const client = C.newIdentity()+ const m = machine(client)+ const groupSeen = new Map<string, number>()+ const first = session(m, client, 'conn-1', groupSeen)+ const started = m.manager.wrapUp({ type: 'turn_started', agentId: 'a', payload: { userMessage: 'once' } })+ expect(first.unwrapIncoming(started)?.payload).toEqual({ userMessage: 'once' })+ const second = session(m, client, 'conn-2', groupSeen)+ expect(second.unwrapIncoming(started)).toBeNull()+ const ended = m.manager.wrapUp({ type: 'turn_ended', agentId: 'a', payload: {} })+ expect(second.unwrapIncoming(ended)?.payload).toEqual({})+ // A session with no record carried over opens whatever this process has sealed since it started:+ // why the relay pool carries one from each of its sessions to the next.+ expect(session(m, client, 'conn-3').unwrapIncoming(started)?.payload).toEqual({ userMessage: 'once' })+ })++ it('remembers a bounded number of the machine\'s epochs, the first seen let go first', () => {+ const client = C.newIdentity()+ const groupSeen = new Map<string, number>()+ const epochs: string[] = []+ for (let n = 0; n <= GROUP_EPOCHS_KEPT; n++) {+ // A new process of the machine each time: a new group key and epoch.+ const m = machine(client)+ const sealed = m.manager.wrapUp({ type: 'turn_ended', agentId: 'a', payload: {} })+ epochs.push((sealed.payload as { __e2e: { epoch: string } }).__e2e.epoch)+ expect(session(m, client, `conn-${n}`, groupSeen).unwrapIncoming(sealed)).not.toBeNull()+ }+ expect(groupSeen.size).toBe(GROUP_EPOCHS_KEPT)+ expect(groupSeen.has(epochs[0])).toBe(false)+ expect(groupSeen.has(epochs.at(-1)!)).toBe(true)+ })++ it('RemoteRelayPool carries the record from a session to its next: a replayed broadcast is not forwarded', async () => {+ const wss = new WebSocketServer({ port: 0 })+ const sockets = new Map<string, import('ws').WebSocket>()+ const client = C.newIdentity()+ const manager = new E2eeManagerCtor({ machineId: MACHINE_ID, isConnected: () => true, sendTo: (id, frame) => sockets.get(id)?.send(JSON.stringify(frame)) })+ manager.trustPeer({ pub: C.b64e(client.pub), label: 'desktop' })+ let connections = 0+ wss.on('connection', (ws) => {+ const id = `conn-${++connections}`+ sockets.set(id, ws)+ ws.on('message', (raw) => {+ const frame = JSON.parse(raw.toString()) as Frame+ if (frame.type === 'machine_select') ws.send(JSON.stringify({ type: 'connected', payload: { machineId: MACHINE_ID } }))+ else if (String(frame.type).startsWith('e2e_')) manager.handleFrame(id, frame)+ })+ })+ const base = `ws://127.0.0.1:${(wss.address() as AddressInfo).port}`+ const peers = new MachinePeerStore()+ peers.pin(MACHINE_ID, C.b64e(new E2eeStore().getIdentity().pub), 'test')+ const pool = new RemoteRelayPool(fakeAuth, base, client, peers, { p2p: false })+ const inbox: Frame[] = []+ const sink = { sendFrame: (frame: Frame) => { inbox.push(frame); return true }, sendBinary: () => true }+ const select = { type: 'machine_select', payload: { machineId: MACHINE_ID } }+ const arrived = async (type: string, count: number) => {+ for (let n = 0; n < 100 && inbox.filter((frame) => frame.type === type).length < count; n++) await new Promise((resolve) => setTimeout(resolve, 20))+ }+ try {+ await pool.acquire(MACHINE_ID, 'prod', select, sink, () => {})+ const started = manager.wrapUp({ type: 'turn_started', agentId: 'a', payload: { userMessage: 'once' } })+ sockets.get('conn-1')!.send(JSON.stringify(started))+ await arrived('turn_started', 1)+ pool.invalidate(MACHINE_ID)+ await pool.acquire(MACHINE_ID, 'prod', select, sink, () => {})+ // The relay replays the old broadcast into the new session, then the machine says something new.+ sockets.get('conn-2')!.send(JSON.stringify(started))+ sockets.get('conn-2')!.send(JSON.stringify(manager.wrapUp({ type: 'turn_ended', agentId: 'a', payload: {} })))+ await arrived('turn_ended', 1)+ expect(inbox.filter((frame) => frame.type === 'turn_started')).toHaveLength(1)+ expect(inbox.filter((frame) => frame.type === 'turn_ended')).toHaveLength(1)+ } finally {+ pool.invalidate(MACHINE_ID)+ for (const ws of wss.clients) ws.terminate()+ await new Promise<void>((resolve) => wss.close(() => resolve()))+ }+ })+})++describe('RemoteRelayPool retires a session the machine no longer has', () => {+ const fakeAuth = { accessToken: async () => 'unused-in-this-fake' } as unknown as import('../authSession.js').AuthSessionManager++ it('e2e_session_unknown closes the local connection as for node_status offline, keeps the pin, and the next acquire dials a new session', async () => {+ const E2eeStore = (await import('./store.js')).E2eeStore+ const wss = new WebSocketServer({ port: 0 })+ const sockets = new Map<string, import('ws').WebSocket>()+ const client = C.newIdentity()+ const manager = new E2eeManagerCtor({ machineId: MACHINE_ID, isConnected: () => true, sendTo: (id, frame) => sockets.get(id)?.send(JSON.stringify(frame)) })+ manager.trustPeer({ pub: C.b64e(client.pub), label: 'desktop' })+ let connections = 0+ wss.on('connection', (ws) => {+ const id = `conn-${++connections}`+ sockets.set(id, ws)+ ws.on('message', (raw) => {+ const frame = JSON.parse(raw.toString()) as Frame+ if (frame.type === 'machine_select') ws.send(JSON.stringify({ type: 'connected', payload: { machineId: MACHINE_ID } }))+ else if (String(frame.type).startsWith('e2e_')) manager.handleFrame(id, frame)+ })+ })+ const peers = new MachinePeerStore()+ peers.pin(MACHINE_ID, C.b64e(new E2eeStore().getIdentity().pub), 'test')+ const pool = new RemoteRelayPool(fakeAuth, `ws://127.0.0.1:${(wss.address() as AddressInfo).port}`, client, peers, { p2p: false })+ const select = { type: 'machine_select', payload: { machineId: MACHINE_ID } }+ const sink = { sendFrame: () => true, sendBinary: () => true }+ try {+ const closed = new Promise<number>((resolve) => { void pool.acquire(MACHINE_ID, 'prod', select, sink, (code) => resolve(code)) })+ for (let n = 0; n < 100 && !manager.hasSession('conn-1'); n++) await new Promise((resolve) => setTimeout(resolve, 20))+ sockets.get('conn-1')!.send(JSON.stringify({ type: 'e2e_session_unknown', payload: { refused: { type: 'message', n: 4 } } }))+ expect(await closed).toBe(1012)+ expect(peers.get(MACHINE_ID)).toBeDefined()+ await pool.acquire(MACHINE_ID, 'prod', select, sink, () => {})+ expect(connections).toBe(2)+ expect(manager.hasSession('conn-2')).toBe(true)+ } finally {+ pool.invalidate(MACHINE_ID)+ for (const ws of wss.clients) ws.terminate()+ await new Promise<void>((resolve) => wss.close(() => resolve()))+ }+ })+})diff --git a/cli/src/lib/remoteRelay.ts b/cli/src/lib/remoteRelay.tsindex 3cefab2a3..a05c64516 100644--- a/cli/src/lib/remoteRelay.ts+++ b/cli/src/lib/remoteRelay.ts@@ -279,6 +279,10 @@ export class RemoteRelayPool {/** Warm background pools per machine, each holding one lingering session nobody is attached to —* see acquireIsolated(). A pool is either here (idle) or in a client's hands, never both. */private readonly idleIsolated = new Map<string, RemoteRelayPool[]>()+ /** Per machine, the group counters this pool's sessions opened, carried to its next session so a+ * relay cannot replay an old broadcast into it (relayClient.ts `groupSeen`). This pool's alone: the+ * background pools are clients of their own, sent every broadcast as this one is. */+ private readonly groupSeen = new Map<string, Map<string, number>>()constructor(private readonly auth: AuthSessionManager,@@ -458,7 +462,9 @@ export class RemoteRelayPool {// this sink turns every later emit — including the one terminate() raises while CONNECTING — into a// plain 'close', which is what the owners actually clean up on.ws.on('error', () => { /* handled via 'close' */ })- const crypto = new RelaySessionCrypto({ machineId, selfIdentity: this.selfIdentity, peerPub: b64d(peer.pub) })+ let groupSeen = this.groupSeen.get(machineId)+ if (!groupSeen) { groupSeen = new Map(); this.groupSeen.set(machineId, groupSeen) }+ const crypto = new RelaySessionCrypto({ machineId, selfIdentity: this.selfIdentity, peerPub: b64d(peer.pub), groupSeen })const entry: Entry = {ws,crypto,@@ -620,6 +626,14 @@ export class RemoteRelayPool {try { ws.close(1012, 'remote machine disconnected') } catch { ws.terminate() }return}+ if (frame.type === 'e2e_session_unknown') {+ // The machine's daemon has no session for this connection: it restarted while the relay kept+ // our socket, and nothing sent on these keys will open there. Retired as for node_status+ // offline, so the next select opens a new session. The pin stays: nothing was revoked.+ if (this.entries.get(machineId) === entry) this.entries.delete(machineId)+ try { ws.close(1012, 'remote session gone') } catch { ws.terminate() }+ return+ }if (frame.type === 'e2e_rekey') { crypto.handleRekey((frame.payload ?? {}) as Record<string, unknown>); return }if (frame.type === 'e2e_denied') {// Mid-session revoke (e.g. `harness unpair` run on the peer while this relay was alreadydiff --git a/docs/design/2026-10-03-harnessd.md b/docs/design/2026-10-03-harnessd.mdindex f6e3b7dfc..671d5f2cb 100644--- a/docs/design/2026-10-03-harnessd.md+++ b/docs/design/2026-10-03-harnessd.md@@ -917,6 +917,24 @@ Found while mapping, and fixed on their own rather than inside a move:- A stop does not wait on it forever.No defect was found.+ - The thirty-fourth round (`e2e/relay.e2e.ts`): the relay link under stress, as a phone sees it. A+ phone is paired with a signed-in daemon and reaches it through the fake backend's blind relay+ (`e2e/harness/fakeBackend.ts`, `e2e/harness/relayPhone.ts`, the CLI's own relay client role).+ The relay drops mid-turn and comes back, the backend restarts, the relay is slow by seconds, it+ sends garbage or replays old sealed frames, and the daemon restarts while the phone is connected.+ Every message is in the phone's re-read history once, nothing announces a turn it has already+ read, and a message tried with no session is refused at once. Found and fixed:+ - A broadcast the relay replayed into a client's next session opened as news, because the replay+ record for the machine's group key was the session's. The CLI's relay client now carries it from+ one session to the next (`lib/e2ee/relayClient.ts`, `lib/remoteRelay.ts`).+ - A frame a client sealed for a session the daemon no longer had was dropped without a word: a+ message vanished, and a request waited out its timeout. That happens when the daemon restarted+ while the relay kept the client's socket, as the phone app does. The daemon now answers+ `e2e_session_unknown`, naming the frame (terminal bytes by their clear kind and counter), and the+ CLI's relay client opens a new session on it.++ Left, larger: no message is acknowledged end to end, so one lost inside the relay is found only by+ reading the conversation again; and the phone app has both of the above to learn.- Every request the apps send is answered: all 39 of the desktop's, all 21 of the phone's and all 20of hn's, checked against the daemon's sources.- The desktop's `lessons` is answered `UNSUPPORTED` on purpose, for a feature not yet built.