The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.
TypeScript 3d-modelingagent-frameworkagent-harnessai-agentscadclaude-codecodexcoding-agentsdeveloper-toolsesp32game-developmentmcp-servermulti-agentopen-hardwareopencodeopenharnesspcbroboticsterminaltmux
perf(backend): a viewer's terminal frames go straight to its connection (#2413)
0xken157b07e79d0a42
diff --git a/backend/src/config/env.ts b/backend/src/config/env.tsindex 874bf30db..1fdc90686 100644--- a/backend/src/config/env.ts+++ b/backend/src/config/env.ts@@ -39,6 +39,10 @@ const envSchema = z.object({// What each process keeps of the Swarms it read, by their stamps in Redis (superTabs/cache.ts). 'off' is the kill switch:// every read goes to Mongo again, while writes still bump the stamps, so processes that keep theirs on stay right.SWARM_READ_CACHE: z.enum(['on', 'off']).default('on'),+ // A frame an adapter sends for one viewer goes to the process holding that viewer (lib/terminalRoute.ts), found by its+ // record in Redis, instead of to every process with a viewer of the machine. 'off' is the kill switch: such frames go+ // through the machine's channels again; a viewer on the adapter's own process is still served directly.+ TERMINAL_POD_ROUTES: z.enum(['on', 'off']).default('on'),// Its size per process, counted as the JSON size of what it holds.SWARM_CACHE_MB: z.string().default('64').transform(Number),// The worker's hourly move of finished deliveries older than 14 days into their messages' receiptsdiff --git a/backend/src/lib/adapterRoutes.test.ts b/backend/src/lib/adapterRoutes.test.tsnew file mode 100644index 000000000..8c0ddd87f--- /dev/null+++ b/backend/src/lib/adapterRoutes.test.ts@@ -0,0 +1,120 @@+import { EventEmitter } from 'node:events'+import { beforeEach, describe, expect, it, vi } from 'vitest'++// The machine socket sends each frame for one viewer through that viewer's route (lib/terminalRoute.ts), on the real+// handler; the harness is adapterShortTurns.test.ts's, with the routes faked.+const m = vi.hoisted(() => ({+ auth: vi.fn(), upgrade: vi.fn(), resolve: vi.fn(), publishUp: vi.fn(), recordTurn: vi.fn(), nodeRole: vi.fn(), warn: vi.fn(), info: vi.fn(),+}))+vi.mock('./ssoAuth.js', async original => ({ ...await original<typeof import('./ssoAuth.js')>(), authenticateAccessToken: m.auth }))+vi.mock('./wsServer.js', async original => ({ ...await original<typeof import('./wsServer.js')>(), createWss: () => ({ handleUpgrade: m.upgrade }) }))+vi.mock('../services/MachineService.js', () => ({ machineService: { resolveOrCreateForComputer: m.resolve } }))+vi.mock('./prisma.js', () => ({+ prisma: { machine: { update: vi.fn(async () => ({})), findUnique: vi.fn(async () => null) }, user: { findUnique: vi.fn(async () => null) } },+ machineAlive: { OR: [{ deletedAt: null }, { deletedAt: { isSet: false } }] },+}))+vi.mock('./bus.js', () => ({+ publishUp: m.publishUp, claimMachineOwner: vi.fn(async () => true), releaseMachineOwner: vi.fn(), publishDeviceE2eePair: vi.fn(),+ publishTerminalUp: vi.fn(), subscribeTerminalDown: vi.fn(async () => () => {}), setMachineAppState: vi.fn(), clearMachineAppState: vi.fn(),+}))+vi.mock('./adapterAccountPushes.js', () => ({ relayAccountPushes: vi.fn(async () => () => {}) }))+vi.mock('./deviceKeyLog.js', () => ({ appendDeviceKey: vi.fn(), touchDeviceKey: vi.fn() }))+vi.mock('./nodeRole.js', () => ({ attachNodeRole: m.nodeRole, PRESENCE_TTL_SEC: 30 }))+vi.mock('./hub.js', () => ({+ CLIENTS_FRAME: '__clients', CLIENTS_DIRTY: '__clients_dirty', LIVENESS_TICK_MS: 5_000, CLIENT_IDLE_DEADLINE_MS: 75_000,+ DEVICE_IDLE_DEADLINE_MS: 35_000, CLIENT_HEARTBEAT_MS: 25_000, deliverTerminalUpLocal: vi.fn(), deliverUpLocal: vi.fn(),+ pushDeviceRevoked: vi.fn(), sendClientsControl: vi.fn(), recomputeAndSendClients: vi.fn(), attachHubClient: vi.fn(),+ trackSocketLiveness: () => () => {},+}))+vi.mock('./dailyTracking.js', async original => ({ ...await original<typeof import('./dailyTracking.js')>(),+ touchMachineOnlineDay: vi.fn(async () => {}), touchUserOnlineDay: vi.fn(async () => {}), touchClientOnlineDay: vi.fn(async () => {}),+ recordTurnStarted: m.recordTurn }))+const routes = vi.hoisted(() => ({ hop: vi.fn(), targeted: vi.fn(), close: vi.fn() }))+vi.mock('./terminalRoute.js', () => ({ createConnectionRoutes: vi.fn(() => routes) }))+vi.mock('./p2pSignaling.js', async original => ({ ...await original<typeof import('./p2pSignaling.js')>(),+ terminalP2pPolicy: () => ({ enabled: true, protocolVersion: 1, stunUrls: ['stun:example'], openWaitMs: 1_000 }) }))+vi.mock('../utils/logger.js', () => ({ logger: { warn: m.warn, info: m.info, error: vi.fn(), debug: vi.fn() } }))++import { handleAdapterUpgrade } from './adapterWs.js'+import { encodeTerminalHop, TerminalBinaryKind, TerminalHopDirection } from './terminalBinary.js'++const flush = async () => { for (let i = 0; i < 400; i++) await Promise.resolve() }++class Socket extends EventEmitter {+ readyState = 1; bufferedAmount = 0+ send = vi.fn()+ close = vi.fn((code?: number, reason?: string) => { this.readyState = 3; this.emit('close', code, Buffer.from(reason ?? '')) })+ terminate = vi.fn(() => this.close())+ frame(body: unknown) { this.emit('message', Buffer.from(JSON.stringify(body)), false) }+}++async function machineSocket(query = ''): Promise<Socket> {+ const ws = new Socket()+ m.upgrade.mockImplementation((_r, _s, _h, done) => done(ws))+ handleAdapterUpgrade({ url: `/api/adapter-ws?computer=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa&label=Mac&autonomousEnv=prod${query}`,+ headers: { 'sec-websocket-protocol': 'access-token' } } as never,+ { write: vi.fn(), destroy: vi.fn(), destroyed: false, on: vi.fn(), once: vi.fn() } as never, Buffer.alloc(0))+ await flush()+ return ws+}+const sealed = (k: 'p' | 'g' = 'p') => ({ __e2e: { v: 1, n: 1, k, ct: 'x' } })+const viewer = '0f0e0d0c-0b0a-4908-8706-050403020100'++beforeEach(() => {+ vi.clearAllMocks()+ m.auth.mockResolvedValue({ sub: 'owner', email: '[email protected]', role: 'user', autonomousEnv: 'prod' })+ m.resolve.mockResolvedValue({ machine: { machineId: 'routed-mac', userId: 'owner', name: 'Mac', billingStatus: 'not_required', sessionKind: null }, created: false })+ m.publishUp.mockResolvedValue(1)+ m.recordTurn.mockResolvedValue(undefined)+ m.nodeRole.mockResolvedValue({ touch: vi.fn(), detach: vi.fn(async () => {}) })+})++describe('a machine socket and its viewers\' frames', () => {+ it('delivers an untargeted frame to this process\'s viewers before a targeted frame sent after it, and publishes it for the others', async () => {+ const hub = await import('./hub.js')+ const ws = await machineSocket()+ ws.frame({ t: 'up', frame: { type: 'agent_synced', payload: { __e2e: 'y' } } })+ ws.frame({ t: 'up', targetConnId: 'c1', frame: { type: 'rpc_result', payload: { __e2e: 'z' } } })+ await flush()+ const local = vi.mocked(hub.deliverUpLocal)+ expect(local).toHaveBeenCalledWith('routed-mac', { webEligible: true, commanderEligible: false, frame: { type: 'agent_synced', payload: { __e2e: 'y' } } })+ expect(local.mock.invocationCallOrder[0]).toBeLessThan(routes.targeted.mock.invocationCallOrder[0]!)+ // The Redis copy carries this process's id, which this process's own subscription skips.+ expect(m.publishUp.mock.calls[0]![1]).toMatchObject({ frame: { type: 'agent_synced' }, originPid: expect.any(String) })+ })++ it('publishes a frame whose target is not a plausible connection id as before, without a route', async () => {+ const ws = await machineSocket()+ ws.frame({ t: 'up', targetConnId: 'x'.repeat(200), frame: { type: 'rpc_result', payload: { __e2e: 'z' } } })+ await flush()+ expect(routes.targeted).not.toHaveBeenCalled()+ expect(m.publishUp.mock.calls[0]![1]).toMatchObject({ targetConnId: 'x'.repeat(200) })+ })++ it('routes a hop, a terminal frame, a p2p frame and a targeted reply by their connection, and publishes the rest as before', async () => {+ const ws = await machineSocket()+ const frame = Buffer.alloc(36)+ frame.write('HTRM'); frame[4] = 3; frame[5] = TerminalBinaryKind.output; frame.writeUInt32BE(16, 16)+ const packet = encodeTerminalHop(TerminalHopDirection.up, viewer, frame)!+ ws.emit('message', packet, true)+ ws.frame({ t: 'up', targetConnId: viewer, frame: { type: 'terminal_output', payload: sealed() } })+ ws.frame({ t: 'up', targetConnId: viewer, frame: { type: 'p2p_answer', payload: sealed() } })+ ws.frame({ t: 'up', targetConnId: 'c2', frame: { type: 'e2e_welcome', payload: { x: 1 } } })+ ws.frame({ t: 'up', frame: { type: 'text_delta', payload: { __e2e: 'y' } } })+ // A terminal frame that fails today's checks (not sealed) is still dropped before any route.+ ws.frame({ t: 'up', targetConnId: viewer, frame: { type: 'terminal_output', payload: { plain: true } } })+ await flush()+ expect(routes.hop).toHaveBeenCalledTimes(1)+ expect(Buffer.from(routes.hop.mock.calls[0]![0] as Uint8Array)).toEqual(packet)+ expect(routes.hop.mock.calls[0]![1]).toBe(viewer)+ expect(routes.targeted.mock.calls.map(([msg]) => msg)).toEqual([+ { webEligible: true, commanderEligible: false, targetConnId: viewer, targetKind: 'web', frame: { type: 'terminal_output', payload: sealed() } },+ { webEligible: true, commanderEligible: false, targetConnId: viewer, targetKind: 'web', frame: { type: 'p2p_answer', payload: sealed() } },+ { webEligible: true, commanderEligible: false, targetConnId: 'c2', frame: { type: 'e2e_welcome', payload: { x: 1 } } },+ ])+ expect(m.publishUp.mock.calls.map(([, msg]) => msg)).toEqual([+ { webEligible: true, commanderEligible: false, frame: { type: 'text_delta', payload: { __e2e: 'y' } }, originPid: expect.any(String) }])+ ws.emit('close', 1000, Buffer.alloc(0))+ expect(routes.close).toHaveBeenCalled()+ })+})diff --git a/backend/src/lib/adapterWs.ts b/backend/src/lib/adapterWs.tsindex 7e1da75ab..dd0a87c48 100644--- a/backend/src/lib/adapterWs.ts+++ b/backend/src/lib/adapterWs.ts@@ -24,17 +24,18 @@ import { createWss, upgradeStatusText, WS_LIMITS } from './wsServer.js'import { prisma } from './prisma.js'import {publishUp, claimMachineOwner, releaseMachineOwner, publishDeviceE2eePair,- publishTerminalUp, subscribeTerminalDown,+ subscribeTerminalDown,setMachineAppState, clearMachineAppState,} from './bus.js'-import { trackSocketLiveness } from './hub.js'+import { deliverUpLocal, trackSocketLiveness } from './hub.js'+import { PROCESS_ID } from './registry.js'import { guardedSend, guardedSendJson } from './wsSend.js'import { trackSocket } from './wsAttribution.js'import { attachNodeRole, PRESENCE_TTL_SEC } from './nodeRole.js'import { presenceTracker, presenceWriteDue, recordTurnStarted, touchClientOnlineDay, touchMachineOnlineDay, touchUserOnlineDay, TUI_SURFACES, type ClientSurface, type PresenceWriteState } from './dailyTracking.js'import { countryCodeFromHeaders } from './clientGeo.js'import { recordCreatedAgent, recordDeletedAgent } from './agentTracker.js'-import type { Frame } from './tunnel.js'+import type { Frame, UpBusMsg } from './tunnel.js'import { logger } from '../utils/logger.js'import { utcDayKey } from '../types/analytics.js'import { machineBillingAllowsDataPlane } from './billingState.js'@@ -67,6 +68,8 @@ import {} from './p2pSignaling.js'import { ShortTurnGuard } from './shortTurnGuard.js'import { markCpuKind } from './selfProfile.js'+import { createConnectionRoutes } from './terminalRoute.js'+import { recordRelay } from './relayStats.js'const wss = createWss(WS_LIMITS.adapter, { echoFirstProtocol: true })@@ -412,6 +415,17 @@ async function attachAdapter(ws: WebSocket, machineId: string, userId: string, c// deadline instead, which is the same fix already applied to the hub's own sockets.const releaseLiveness = trackSocketLiveness(ws, 'adapter')+ // Frames for one viewer go straight to it when it is on this process, else to the process holding it+ // (lib/terminalRoute.ts); a connection keeps the route its first frame took for this socket's life.+ const routes = createConnectionRoutes(machineId)+ // A frame for one connection takes its route; a target that is not a plausible connection id (the daemon sets it) is+ // published as before, never used as a route or Redis key.+ const sendTargeted = (msg: UpBusMsg & { targetConnId: unknown }): void => {+ const target = msg.targetConnId+ if (typeof target === 'string' && target.length > 0 && target.length <= 128) routes.targeted({ ...msg, targetConnId: target })+ else void publishUp(machineId, msg as UpBusMsg)+ }+onMessage = (raw: RawData, isBinary: boolean): void => {if (isBinary) {const hop = decodeTerminalHop(new Uint8Array(raw as Buffer))@@ -424,7 +438,7 @@ async function attachAdapter(ws: WebSocket, machineId: string, userId: string, c: hop.kind === TerminalBinaryKind.keyframe ? 'terminal_keyframe' : 'terminal_sync',hop.clientFrame.length,)) return- void publishTerminalUp(machineId, Buffer.from(raw as Buffer))+ routes.hop(Buffer.from(raw as Buffer), hop.connId)return}let env: AdapterFrame@@ -440,7 +454,7 @@ async function attachAdapter(ws: WebSocket, machineId: string, userId: string, cconst bytes = terminalFrameBytes(env.frame)if (!env.targetConnId || bytes > TERMINAL_ENVELOPE_MAX_BYTES|| !isEncryptedTerminalFrame(env.frame) || !terminalRate.allow(terminalType, bytes)) return- void publishUp(machineId, {+ sendTargeted({webEligible: true,commanderEligible: false,targetConnId: env.targetConnId,@@ -455,7 +469,7 @@ async function attachAdapter(ws: WebSocket, machineId: string, userId: string, c|| !terminalP2pPolicy(userId, machineId).enabled|| bytes > P2P_SIGNAL_MAX_BYTES || !isEncryptedP2pFrame(env.frame)|| !p2pSignalRate.allow(bytes)) return- void publishUp(machineId, {+ sendTargeted({webEligible: true,commanderEligible: false,targetConnId: env.targetConnId,@@ -562,14 +576,19 @@ async function attachAdapter(ws: WebSocket, machineId: string, userId: string, c}if (env.webEligible === false && env.commanderEligible !== true && !env.targetConnId) return}- // No originPid: the hub's Redis loopback must deliver to this instance's own web clients too- // (there's no manager-ws local fast-path on this route).- void publishUp(machineId, {+ const up = {webEligible: env.webEligible !== false, // default true (plain event/RPC frames)commanderEligible: env.commanderEligible === true, // devices get only opted-in frames- ...(env.targetConnId ? { targetConnId: env.targetConnId } : {}), // E2EE per-client pushframe: env.frame,- })+ }+ // An E2EE per-client push (pairing, welcome, an RPC reply) takes its connection's route, so it keeps its order with+ // that connection's terminal frames.+ if (env.targetConnId) { sendTargeted({ ...up, targetConnId: env.targetConnId }); return }+ // Every client of the machine: this process's own first, then the rest through Redis, whose copy this process+ // skips (originPid), as manager-ws does. A local viewer's targeted frames are delivered here at once, so its+ // untargeted ones must be too, or an RPC reply would overtake the event the daemon sent before it (review 2026-10-31).+ deliverUpLocal(machineId, up)+ void publishUp(machineId, { ...up, originPid: PROCESS_ID }).then(n => recordRelay('up broadcast', 'published', n))}}// Drain whatever arrived while the role was being attached, in the order it was sent.@@ -578,6 +597,7 @@ async function attachAdapter(ws: WebSocket, machineId: string, userId: string, cconst cleanup = (): void => {releaseLiveness()+ routes.close()terminalDownUnsub()accountPushesUnsub()swarmPushesUnsub()diff --git a/backend/src/lib/bus.ts b/backend/src/lib/bus.tsindex 014c7bbe6..9d1b668b6 100644--- a/backend/src/lib/bus.ts+++ b/backend/src/lib/bus.ts@@ -274,6 +274,69 @@ export function publishTerminalDown(machineId: string, payload: Uint8Array): Proreturn safePublish(terminalDownChannel(machineId), Buffer.from(payload))}+// ── per-process channels for frames addressed to one viewer (lib/terminalRoute.ts) ─────────────────++/** How long a viewer's record lives without its process renewing it (hub heartbeat every 25 s). */+export const CONN_OWNER_TTL_MS = 60_000+const connOwnerKey = (connId: string): string => `conn:${connId}`+const podHopChannel = (processId: string): string => `termpod:${processId}`+const podJsonChannel = (processId: string): string => `uppod:${processId}`++/** Records that these viewer connections live on `processId`, in one pipeline. A failure leaves them unrecorded: frames+ * for them then go through the machine's channels, as before records existed. */+export async function recordConnOwners(connIds: readonly string[], processId: string): Promise<void> {+ if (connIds.length === 0) return+ const p = pub.pipeline()+ for (const connId of connIds) p.set(connOwnerKey(connId), processId, 'PX', CONN_OWNER_TTL_MS)+ try { await p.exec() } catch (err) { logger.debug('[bus] conn owner record failed', { error: String(err) }) }+}+export async function forgetConnOwner(connId: string): Promise<void> {+ try { await pub.del(connOwnerKey(connId)) } catch { /* expires on its own */ }+}+/** The process holding this viewer connection, or null when none is recorded (or Redis cannot say). */+export async function connOwner(connId: string): Promise<string | null> {+ try { return await pub.get(connOwnerKey(connId)) } catch { return null }+}++/** A terminal hop for one process: the machine id (2-byte length + UTF-8) before the hop, so the receiver delivers only+ * to that machine's viewer with the hop's connection id. */+export function encodePodHop(machineId: string, packet: Uint8Array): Buffer {+ const id = Buffer.from(machineId, 'utf8')+ const out = Buffer.allocUnsafe(2 + id.length + packet.length)+ out.writeUInt16BE(id.length, 0)+ id.copy(out, 2)+ out.set(packet, 2 + id.length)+ return out+}+export function decodePodHop(payload: Buffer): { machineId: string; packet: Uint8Array } | null {+ if (payload.length < 2) return null+ const length = payload.readUInt16BE(0)+ if (payload.length < 2 + length) return null+ return { machineId: payload.toString('utf8', 2, 2 + length), packet: payload.subarray(2 + length) }+}+export function publishPodHop(processId: string, machineId: string, packet: Uint8Array): Promise<number> {+ return safePublish(podHopChannel(processId), encodePodHop(machineId, packet))+}+export function publishPodJson(processId: string, machineId: string, msg: UpBusMsg): Promise<number> {+ return safePublish(podJsonChannel(processId), JSON.stringify({ machineId, msg }))+}+/** This process's own channels: hops on the terminal subscriber connection, JSON frames on the main one, the same+ * connections the machine's channels use, so a viewer's frames keep their order on each. */+export async function subscribePod(processId: string, on: { hop(machineId: string, packet: Uint8Array): void; json(machineId: string, msg: UpBusMsg): void }): Promise<() => void> {+ const subscribed = await Promise.allSettled([+ addBinarySub(podHopChannel(processId), payload => { const hop = decodePodHop(payload); if (hop) on.hop(hop.machineId, hop.packet) }),+ // Only a frame for one connection comes here; anything else would reach every viewer of the machine on this process.+ addSub(podJsonChannel(processId), message => {+ const { machineId, msg } = (message ?? {}) as { machineId?: unknown; msg?: UpBusMsg }+ if (typeof machineId === 'string' && msg && typeof msg.targetConnId === 'string') on.json(machineId, msg)+ }),+ ])+ const stops = subscribed.flatMap(each => each.status === 'fulfilled' ? [each.value] : [])+ const failed = subscribed.find((each): each is PromiseRejectedResult => each.status === 'rejected')+ if (failed) { for (const stop of stops) stop(); throw failed.reason }+ return () => { for (const stop of stops) stop() }+}+// ── up / down data channels ──────────────────────────────────────────────────────────────────────export function subscribeUp(machineId: string, cb: (msg: UpBusMsg) => void): Promise<() => void> {diff --git a/backend/src/lib/hub.ts b/backend/src/lib/hub.tsindex b9ae7ef45..0573573ee 100644--- a/backend/src/lib/hub.ts+++ b/backend/src/lib/hub.ts@@ -10,12 +10,15 @@ import { WebSocket } from 'ws'import { randomUUID } from 'crypto'import {bumpCommanderJoinGeneration,+ forgetConnOwner,getAgentClientState,publishDown,publishTerminalDown,publishUp,setAgentClientCount,setAgentClientCountsBatch,+ recordConnOwners,+ subscribePod,subscribeUp,subscribeTerminalUp,} from './bus.js'@@ -309,6 +312,8 @@ function clientHeartbeatTick(): void {// for every local agent, not 2 round trips each.const rows = registry.localAgentIds().map((machineId) => ({ machineId, ...registry.clientCounts(machineId) }))void setAgentClientCountsBatch(INSTANCE_ID, rows)+ // Each viewer's record (CONN_OWNER_TTL_MS, 60 s) outlives two missed beats; a crashed process's records expire.+ if (podRoutes) void recordConnOwners(registry.allClients().map(c => c.connId), registry.PROCESS_ID)// B_m periodic resync: re-emit the summed `__clients` for every agent whose control socket lives// here. Field EXPIRY (a client-holding instance crashed — its fields age out of the hash after 45s)// has no attach/detach event to poke a recompute, so without this the node keeps the ghost count@@ -326,6 +331,20 @@ setTimeout(() => {setInterval(clientHeartbeatTick, CLIENT_HEARTBEAT_MS).unref?.()}, Math.floor(Math.random() * 5_000)).unref?.()+/** Whether this process takes frames addressed to its viewers on its own channels (lib/terminalRoute.ts): set once by+ * startPodRoutes, with TERMINAL_POD_ROUTES on. Off, no viewer is recorded and adapters publish as before. */+let podRoutes = false+export const podRoutesOn = (): boolean => podRoutes++/** Records every viewer this process holds, subscribes to its own channels, and keeps both up to date (attach, detach,+ * the client heartbeat). Called once by server.ts. Returns its stop. */+export async function startPodRoutes(): Promise<() => void> {+ const unsubscribe = await subscribePod(registry.PROCESS_ID, { hop: deliverTerminalUpLocal, json: deliverUpLocal })+ podRoutes = true+ void recordConnOwners(registry.allClients().map(c => c.connId), registry.PROCESS_ID)+ return () => { podRoutes = false; unsubscribe() }+}+/** Register a client socket into the hub. The caller still owns the socket's 'message' handler. */export function attachHubClient(socket: WebSocket, machineId: string, kind: ClientKind, opts: { guest?: boolean } = {}): HubClient {const connId = randomUUID()@@ -334,6 +353,7 @@ export function attachHubClient(socket: WebSocket, machineId: string, kind: Clie// touch __hubDeadlineMs: a device sets its own (shorter) deadline before any machine attaches.wireLiveness(socket)const first = registry.addClient({ connId, machineId, kind, socket, ...(opts.guest ? { guest: true } : {}) })+ if (podRoutes) void recordConnOwners([connId], registry.PROCESS_ID)if (first) {void subscribeUp(machineId, (msg) => {@@ -412,6 +432,7 @@ export function attachHubClient(socket: WebSocket, machineId: string, kind: Clie* than inside the HubClient closure. */function detachConn(machineId: string, connId: string): void {const last = registry.removeClient(machineId, connId)+ if (podRoutes) void forgetConnOwner(connId)if (last) {const unsub = upSubs.get(machineId)if (unsub) { unsub(); upSubs.delete(machineId) }diff --git a/backend/src/lib/relayStats.test.ts b/backend/src/lib/relayStats.test.tsnew file mode 100644index 000000000..53d191465--- /dev/null+++ b/backend/src/lib/relayStats.test.ts@@ -0,0 +1,21 @@+import { afterEach, expect, it } from 'vitest'+import { recordRelay, startRelayStats } from './relayStats.js'++let stop = () => {}+afterEach(() => stop())++it('counts local and published frames and the receivers each publish reached, per kind, and logs nothing in a quiet window', async () => {+ const logs: Array<Record<string, unknown>> = []+ stop = startRelayStats({ everyMs: 50, log: (_m, c) => logs.push(c) })+ recordRelay('termup', 'local')+ recordRelay('termup', 'local')+ recordRelay('termup', 'published', 2)+ recordRelay('up targeted', 'published', 1)+ while (!logs.length) await new Promise(resolve => setTimeout(resolve, 10))+ expect(logs[0]!.kinds).toEqual({+ termup: { local: 2, published: 1, receivers: 2 },+ 'up targeted': { local: 0, published: 1, receivers: 1 },+ })+ await new Promise(resolve => setTimeout(resolve, 120))+ expect(logs).toHaveLength(1)+})diff --git a/backend/src/lib/relayStats.ts b/backend/src/lib/relayStats.tsnew file mode 100644index 000000000..45b72373e--- /dev/null+++ b/backend/src/lib/relayStats.ts@@ -0,0 +1,31 @@+/**+ * How each frame an adapter sends for one viewer reaches it, logged every minute (`terminal relay`): delivered on this+ * process, or published, and how many subscriptions each publish reached (Redis answers that). A frame for one viewer+ * that wakes several processes is work for nothing (spec 2026-10-31-terminal-relay-direct). Counts only, never an id.+ */+import { logger } from '../utils/logger.js'++export type RelayKind = 'termup' | 'up targeted' | 'up broadcast' | 'pod hop' | 'pod json'+interface Counts { local: number; published: number; receivers: number }++let kinds = new Map<RelayKind, Counts>()++export function recordRelay(kind: RelayKind, path: 'local' | 'published', receivers = 0): void {+ let counts = kinds.get(kind)+ if (!counts) kinds.set(kind, counts = { local: 0, published: 0, receivers: 0 })+ if (path === 'local') counts.local+++ else { counts.published++; counts.receivers += receivers }+}++export function startRelayStats({ everyMs = 60_000, log = (message: string, context: Record<string, unknown>) => logger.info(message, context) }:+ { everyMs?: number; log?: (message: string, context: Record<string, unknown>) => void } = {}): () => void {+ let since = Date.now()+ const timer = setInterval(() => {+ const now = Date.now(), window = kinds+ kinds = new Map()+ if (window.size) log('terminal relay', { windowMs: now - since, kinds: Object.fromEntries(window) })+ since = now+ }, everyMs)+ timer.unref()+ return () => clearInterval(timer)+}diff --git a/backend/src/lib/terminalRoute.test.ts b/backend/src/lib/terminalRoute.test.tsnew file mode 100644index 000000000..e655efeb4--- /dev/null+++ b/backend/src/lib/terminalRoute.test.ts@@ -0,0 +1,227 @@+import { beforeEach, describe, expect, it, vi } from 'vitest'+import type { WebSocket } from 'ws'+import { encodeTerminalHop, TerminalBinaryKind, TerminalHopDirection } from './terminalBinary.js'++const bus = vi.hoisted(() => ({+ publishUp: vi.fn(async (..._args: unknown[]) => 1),+ publishTerminalUp: vi.fn(async (..._args: unknown[]) => 1),+ publishPodHop: vi.fn(async (..._args: unknown[]) => 1),+ publishPodJson: vi.fn(async (..._args: unknown[]) => 1),+ connOwner: vi.fn(async (_connId: string): Promise<string | null> => null),+ recordConnOwners: vi.fn(async () => undefined),+ forgetConnOwner: vi.fn(async () => undefined),+ subscribePod: vi.fn(async () => () => undefined),+ subscribeUp: async () => () => undefined,+ subscribeTerminalUp: async () => () => undefined,+ publishDown: async () => 1,+ publishTerminalDown: async () => 1,+ setAgentClientCount: async () => undefined,+ setAgentClientCountsBatch: async () => undefined,+ getAgentClientState: async () => ({ totals: { ui: 0, commander: 0, commanderActive: 0 }, commanderJoinGeneration: 1 }),+ bumpCommanderJoinGeneration: async () => 1,+}))+vi.mock('./bus.js', () => bus)+vi.mock('./providerLink.js', () => ({ routeDown: async (_m: string, _d: unknown, fallback: () => Promise<number>) => fallback() }))+vi.mock('../utils/logger.js', () => ({ logger: { warn: vi.fn(), info: vi.fn(), error: vi.fn(), debug: vi.fn() } }))++const { attachHubClient, startPodRoutes } = await import('./hub.js')+const { createConnectionRoutes, MAX_ROUTES } = await import('./terminalRoute.js')+const registry = await import('./registry.js')++const socket = (out: unknown[]): WebSocket => ({ readyState: 1, bufferedAmount: 0, send: (data: unknown) => out.push(data),+ close: vi.fn(), terminate: vi.fn(), on: () => undefined }) as unknown as WebSocket+const clientFrame = (byte: number) => {+ const frame = Buffer.alloc(36)+ frame.write('HTRM'); frame[4] = 3; frame[5] = TerminalBinaryKind.output; frame.writeUInt32BE(16, 16); frame[35] = byte+ return frame+}+const hop = (connId: string, byte: number) => encodeTerminalHop(TerminalHopDirection.up, connId, clientFrame(byte))!+const json = (connId: string, n: number) => ({ webEligible: true, commanderEligible: false, targetConnId: connId, targetKind: 'web' as const,+ frame: { type: 'terminal_output', payload: { n } } })+const flush = () => new Promise(resolve => setImmediate(resolve))+const remoteConn = () => '0f0e0d0c-0b0a-4908-8706-050403020100'+let machine = 0+const machineId = () => `route-${Date.now()}-${machine++}`++beforeEach(() => { for (const fn of Object.values(bus)) if (vi.isMockFunction(fn)) fn.mockClear(); bus.connOwner.mockImplementation(async () => null) })++describe('pod routes off (Phase 1 alone)', () => {+ it('delivers a hop and a targeted JSON frame to a local web connection without publishing, in order', () => {+ const id = machineId(), out: unknown[] = []+ const web = attachHubClient(socket(out), id, 'web')+ const routes = createConnectionRoutes(id)+ routes.hop(hop(web.connId, 1), web.connId)+ routes.targeted(json(web.connId, 2))+ routes.hop(hop(web.connId, 3), web.connId)+ expect(out).toHaveLength(3)+ expect(Buffer.from(out[0] as Uint8Array)).toEqual(clientFrame(1))+ expect(out[1]).toBe(JSON.stringify(json(web.connId, 2).frame))+ expect(Buffer.from(out[2] as Uint8Array)).toEqual(clientFrame(3))+ expect(bus.publishTerminalUp).not.toHaveBeenCalled()+ expect(bus.publishUp).not.toHaveBeenCalled()+ web.detach()+ })++ it('publishes for a connection that is not on this process, as before', async () => {+ const id = machineId(), routes = createConnectionRoutes(id), conn = remoteConn()+ const packet = hop(conn, 1)+ routes.hop(packet, conn)+ routes.targeted(json(conn, 2))+ expect(bus.publishTerminalUp).toHaveBeenCalledWith(id, packet)+ expect(bus.publishUp).toHaveBeenCalledWith(id, json(conn, 2))+ expect(bus.connOwner).not.toHaveBeenCalled()+ })++ it('keeps a connection on its first route: frames after a local viewer detaches are dropped, not published', () => {+ const id = machineId(), out: unknown[] = []+ const web = attachHubClient(socket(out), id, 'web')+ const routes = createConnectionRoutes(id)+ routes.hop(hop(web.connId, 1), web.connId)+ web.detach()+ routes.hop(hop(web.connId, 2), web.connId)+ routes.targeted(json(web.connId, 3))+ expect(out).toHaveLength(1)+ expect(bus.publishTerminalUp).not.toHaveBeenCalled()+ expect(bus.publishUp).not.toHaveBeenCalled()+ })++ it('delivers a frame targeted at a local device connection', () => {+ const id = machineId(), out: string[] = []+ const device = attachHubClient(socket(out), id, 'commander')+ const routes = createConnectionRoutes(id)+ routes.targeted({ webEligible: false, commanderEligible: true, targetConnId: device.connId, frame: { type: 'e2e_welcome', payload: { sealed: 1 } } })+ expect(out).toHaveLength(1)+ expect(JSON.parse(out[0]!)).toMatchObject({ type: 'e2e_welcome', machineId: id })+ expect(bus.publishUp).not.toHaveBeenCalled()+ device.detach()+ })++ it('sends a guest only the frames addressed to it', () => {+ const id = machineId(), ownerOut: unknown[] = [], guestOut: unknown[] = []+ const owner = attachHubClient(socket(ownerOut), id, 'web')+ const guest = attachHubClient(socket(guestOut), id, 'web', { guest: true })+ const routes = createConnectionRoutes(id)+ routes.targeted(json(guest.connId, 1))+ routes.hop(hop(owner.connId, 2), owner.connId)+ expect(guestOut).toHaveLength(1)+ expect(ownerOut).toHaveLength(1)+ owner.detach(); guest.detach()+ })++ it('still delivers locally when Redis takes no publish', async () => {+ bus.publishTerminalUp.mockImplementationOnce(async () => 0)+ const id = machineId(), out: unknown[] = []+ const web = attachHubClient(socket(out), id, 'web')+ createConnectionRoutes(id).hop(hop(web.connId, 1), web.connId)+ expect(out).toHaveLength(1)+ web.detach()+ })++ it('keeps one adapter socket\'s routes when another closes', () => {+ const id = machineId(), out: unknown[] = []+ const web = attachHubClient(socket(out), id, 'web')+ const first = createConnectionRoutes(id), second = createConnectionRoutes(id)+ first.hop(hop(web.connId, 1), web.connId)+ second.hop(hop(web.connId, 2), web.connId)+ first.close()+ first.hop(hop(web.connId, 3), web.connId)+ second.hop(hop(web.connId, 4), web.connId)+ expect(out.map(f => Buffer.from(f as Uint8Array)[35])).toEqual([1, 2, 4])+ web.detach()+ })+})++it('keeps at most MAX_ROUTES routes per adapter socket, forgetting the least recently used', () => {+ const id = machineId(), out: unknown[] = []+ const web = attachHubClient(socket(out), id, 'web')+ const routes = createConnectionRoutes(id)+ routes.hop(hop(web.connId, 1), web.connId)+ // Many connections that come and go (Desktop attaches a new one per agent switch), the live one used now and then.+ for (let i = 0; i < MAX_ROUTES * 2; i++) {+ const gone = `0f0e0d0c-0b0a-4908-8706-${String(i).padStart(12, '0')}`+ routes.hop(hop(gone, 2), gone)+ if (i % 500 === 0) routes.hop(hop(web.connId, 3), web.connId)+ }+ routes.hop(hop(web.connId, 4), web.connId)+ // The live connection kept its local route throughout: every one of its frames was delivered here.+ expect(out.map(f => Buffer.from(f as Uint8Array)[35])).toEqual([1, ...Array.from({ length: (MAX_ROUTES * 2) / 500 }, () => 3), 4])+ web.detach()+})++describe('pod routes on (Phase 2)', () => {+ let stop: () => void = () => {}+ beforeEach(async () => { stop(); stop = await startPodRoutes() })++ it('records a viewer on attach and forgets it on detach', async () => {+ const id = machineId()+ const web = attachHubClient(socket([]), id, 'web')+ expect(bus.recordConnOwners).toHaveBeenCalledWith([web.connId], registry.PROCESS_ID)+ web.detach()+ expect(bus.forgetConnOwner).toHaveBeenCalledWith(web.connId)+ })++ it('publishes on the machine\'s channels while the owner is looked up, then on the owner\'s alone, in send order', async () => {+ let answer: (owner: string | null) => void = () => {}+ bus.connOwner.mockImplementation(() => new Promise(resolve => { answer = resolve }))+ const id = machineId(), conn = remoteConn(), routes = createConnectionRoutes(id)+ const packets = Array.from({ length: 5 }, (_, i) => hop(conn, i))+ routes.hop(packets[0]!, conn); routes.targeted(json(conn, 1)); routes.hop(packets[1]!, conn)+ answer('other-process')+ await flush()+ routes.hop(packets[2]!, conn); routes.targeted(json(conn, 2)); routes.hop(packets[3]!, conn)+ expect(bus.connOwner).toHaveBeenCalledTimes(1)+ expect(bus.publishTerminalUp.mock.calls.map(c => c[1])).toEqual([packets[0], packets[1]])+ expect(bus.publishUp).toHaveBeenCalledWith(id, json(conn, 1))+ expect(bus.publishPodHop.mock.calls.map(c => [c[0], c[1], c[2]])).toEqual([['other-process', id, packets[2]], ['other-process', id, packets[3]]])+ expect(bus.publishPodJson).toHaveBeenCalledWith('other-process', id, json(conn, 2))+ // Every machine-channel publish was issued before any owner-channel publish: one Redis connection keeps that order.+ const lastMachine = Math.max(...bus.publishTerminalUp.mock.invocationCallOrder, ...bus.publishUp.mock.invocationCallOrder)+ const firstOwner = Math.min(...bus.publishPodHop.mock.invocationCallOrder, ...bus.publishPodJson.mock.invocationCallOrder)+ expect(lastMachine).toBeLessThan(firstOwner)+ })++ it('stays on the machine\'s channels for a connection with no owner, even once an owner appears', async () => {+ const id = machineId(), conn = remoteConn(), routes = createConnectionRoutes(id)+ routes.hop(hop(conn, 1), conn)+ await flush()+ bus.connOwner.mockImplementation(async () => 'other-process')+ routes.hop(hop(conn, 2), conn)+ await flush()+ expect(bus.publishTerminalUp).toHaveBeenCalledTimes(2)+ expect(bus.publishPodHop).not.toHaveBeenCalled()+ })++ it('keeps the machine\'s channels when the lookup fails, and when this process is the recorded owner', async () => {+ const id = machineId(), failed = remoteConn(), mine = '1f0e0d0c-0b0a-4908-8706-050403020100', routes = createConnectionRoutes(id)+ bus.connOwner.mockImplementation(async (connId: string) => { if (connId === failed) throw new Error('down'); return registry.PROCESS_ID })+ routes.hop(hop(failed, 1), failed); routes.hop(hop(mine, 1), mine)+ await flush()+ routes.hop(hop(failed, 2), failed); routes.hop(hop(mine, 2), mine)+ expect(bus.publishTerminalUp).toHaveBeenCalledTimes(4)+ expect(bus.publishPodHop).not.toHaveBeenCalled()+ })++ it('delivers nothing from the process channels to a connection of another machine', async () => {+ const handlers = (bus.subscribePod.mock.calls.at(-1) as unknown as [string, { hop(m: string, p: Uint8Array): void; json(m: string, msg: unknown): void }])[1]+ const machineA = machineId(), machineB = machineId(), out: unknown[] = []+ const viewer = attachHubClient(socket(out), machineB, 'web')+ handlers.hop(machineA, hop(viewer.connId, 1))+ handlers.json(machineA, json(viewer.connId, 2))+ expect(out).toHaveLength(0)+ handlers.hop(machineB, hop(viewer.connId, 3))+ handlers.json(machineB, json(viewer.connId, 4))+ expect(out).toHaveLength(2)+ viewer.detach()+ })++ it('does not look up, nor record, with the switch off', async () => {+ stop(); stop = () => {}+ bus.recordConnOwners.mockClear()+ const id = machineId(), conn = remoteConn()+ const web = attachHubClient(socket([]), id, 'web')+ createConnectionRoutes(id).hop(hop(conn, 1), conn)+ expect(bus.connOwner).not.toHaveBeenCalled()+ expect(bus.recordConnOwners).not.toHaveBeenCalled()+ web.detach()+ })+})diff --git a/backend/src/lib/terminalRoute.ts b/backend/src/lib/terminalRoute.tsnew file mode 100644index 000000000..0d3006f86--- /dev/null+++ b/backend/src/lib/terminalRoute.ts@@ -0,0 +1,93 @@+/**+ * Where a frame an adapter sends for one viewer goes (spec docs/superpowers/specs/2026-10-31-terminal-relay-direct-design.md).+ * Every terminal frame, binary hop or JSON, is for exactly one viewer connection, yet it was published to the machine's+ * channel: every process holding any viewer of that machine decoded it and all but one dropped it, and the adapter's own+ * process got it back from Redis even when it held the viewer itself.+ *+ * Per connection, on this adapter socket:+ * - the viewer is on this process: delivered here, no Redis;+ * - else, with TERMINAL_POD_ROUTES on and the viewer's process recorded (hub.ts): published to that process alone, once+ * the record is read; until then, and for good without a record (a process on the previous release, a failed lookup),+ * the machine's channels, as before (see createConnectionRoutes for why the one switch keeps order).+ */+import * as registry from './registry.js'+import { deliverTerminalUpLocal, deliverUpLocal, podRoutesOn } from './hub.js'+import { connOwner, publishPodHop, publishPodJson, publishTerminalUp, publishUp } from './bus.js'+import { recordRelay } from './relayStats.js'+import type { UpBusMsg } from './tunnel.js'++/** Routes one adapter socket keeps, most recently used last. An adapter socket lives for days while Desktop attaches a new+ * connection on every agent switch; an active connection is only forgotten after this many others were used since. */+export const MAX_ROUTES = 2_000++type Route = { to: 'local' } | { to: 'machine' } | { to: 'process'; owner: string }++export interface ConnectionRoutes {+ /** A binary terminal hop (output, keyframe, sync) for `connId`. */+ hop(packet: Uint8Array, connId: string): void+ /** A JSON up frame addressed to one connection (`targetConnId`). */+ targeted(msg: UpBusMsg & { targetConnId: string }): void+ /** The adapter socket closed: forget every route. */+ close(): void+}++/**+ * A connection's route: local if its viewer is here (for good: a `connId` never moves). Otherwise the machine's channels+ * while its owner is looked up, then, if another process owns it, that process's channels. That one switch keeps order:+ * both channels are published from the same Redis connection, the owner reads both on the same subscriber connection+ * (`up:`/`uppod:`, `termup:`/`termpod:`), and it holds a viewer of the machine, so it listens to the machine's channels.+ * Nothing waits for the lookup, so the machine's untargeted frames sent after a targeted one cannot overtake it.+ */+export function createConnectionRoutes(machineId: string): ConnectionRoutes {+ const routes = new Map<string, Route>()+ const looking = new Set<string>()+ let closed = false++ const keep = (connId: string, route: Route) => {+ routes.delete(connId)+ routes.set(connId, route)+ if (routes.size > MAX_ROUTES) routes.delete(routes.keys().next().value!)+ }+ const route = (connId: string): Route => {+ const known = routes.get(connId)+ if (known) { keep(connId, known); return known }+ if (registry.clientsFor(machineId).some(c => c.connId === connId)) {+ const local: Route = { to: 'local' }+ keep(connId, local)+ return local+ }+ const machine: Route = { to: 'machine' }+ if (!podRoutesOn()) { keep(connId, machine); return machine }+ if (!looking.has(connId)) {+ looking.add(connId)+ void connOwner(connId).then(owner => {+ looking.delete(connId)+ if (closed) return+ keep(connId, owner && owner !== registry.PROCESS_ID ? { to: 'process', owner } : machine)+ }, () => { looking.delete(connId); if (!closed) keep(connId, machine) })+ }+ return machine+ }++ return {+ hop(packet, connId) {+ if (closed) return+ const r = route(connId)+ if (r.to === 'local') { deliverTerminalUpLocal(machineId, packet); recordRelay('termup', 'local'); return }+ if (r.to === 'process') { void publishPodHop(r.owner, machineId, packet).then(n => recordRelay('pod hop', 'published', n)); return }+ void publishTerminalUp(machineId, packet).then(n => recordRelay('termup', 'published', n))+ },+ targeted(msg) {+ if (closed) return+ const r = route(msg.targetConnId)+ if (r.to === 'local') { deliverUpLocal(machineId, msg); recordRelay('up targeted', 'local'); return }+ if (r.to === 'process') { void publishPodJson(r.owner, machineId, msg).then(n => recordRelay('pod json', 'published', n)); return }+ void publishUp(machineId, msg).then(n => recordRelay('up targeted', 'published', n))+ },+ close() {+ closed = true+ routes.clear()+ looking.clear()+ },+ }+}diff --git a/backend/src/lib/terminalRouteRedis.test.ts b/backend/src/lib/terminalRouteRedis.test.tsnew file mode 100644index 000000000..750b22778--- /dev/null+++ b/backend/src/lib/terminalRouteRedis.test.ts@@ -0,0 +1,103 @@+import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest'+import type { UpBusMsg } from './tunnel.js'++// A real Redis: REDIS_TEST_URL, e.g. redis://127.0.0.1:6379/15 (CI has none, so this file is skipped there). The bus+// connects to it as REDIS_URL. One process stands for several: a process's channel is just `termpod:`/`uppod:` + its id,+// so subscribing to another id here is what that other process would do.+const url = process.env.REDIS_TEST_URL+vi.hoisted(() => { if (process.env.REDIS_TEST_URL) process.env.REDIS_URL = process.env.REDIS_TEST_URL })+vi.mock('../utils/logger.js', () => ({ logger: { warn: vi.fn(), info: vi.fn(), error: vi.fn(), debug: vi.fn() } }))++describe.skipIf(!url)('terminal routes over a real Redis', () => {+ let bus: typeof import('./bus.js')+ beforeAll(async () => {+ bus = await import('./bus.js')+ // The command connection refuses commands until it is up (no offline queue), as in production at boot.+ if (bus.pub.status !== 'ready') await new Promise(resolve => bus.pub.once('ready', resolve))+ })+ afterAll(async () => { await bus?.closeBus() })++ const waitFor = async (done: () => boolean, ms = 5_000) => {+ const until = Date.now() + ms+ while (!done() && Date.now() < until) await new Promise(resolve => setTimeout(resolve, 10))+ }+ const unique = (name: string) => `${name}-${process.pid}-${Date.now()}-${Math.random().toString(36).slice(2)}`++ it('records a viewer\'s process for 60 s, reads it back, and forgets it', async () => {+ const a = unique('conn'), b = unique('conn')+ await bus.recordConnOwners([a, b], 'process-x')+ expect(await bus.connOwner(a)).toBe('process-x')+ const ttl = await bus.pub.pttl(`conn:${a}`)+ expect(ttl).toBeGreaterThan(55_000)+ expect(ttl).toBeLessThanOrEqual(60_000)+ await bus.forgetConnOwner(a)+ expect(await bus.connOwner(a)).toBeNull()+ expect(await bus.connOwner(b)).toBe('process-x')+ })++ it('hands 1,000 hops and 1,000 JSON frames to their process once and in order, and nothing to another process', async () => {+ const mine = unique('pid'), other = unique('pid')+ const hops: Array<[string, number]> = [], json: Array<[string, number]> = [], elsewhere: unknown[] = []+ const stopMine = await bus.subscribePod(mine, { hop: (m, p) => hops.push([m, p[0]!]), json: (m, msg) => json.push([m, (msg.frame.payload as { n: number }).n]) })+ const stopOther = await bus.subscribePod(other, { hop: m => elsewhere.push(m), json: m => elsewhere.push(m) })+ for (let i = 0; i < 1_000; i++) {+ void bus.publishPodHop(mine, 'machine-a', Uint8Array.of(i % 256, 1, 2))+ void bus.publishPodJson(mine, 'machine-a', { targetConnId: 'c', frame: { type: 'terminal_output', payload: { n: i } } } as UpBusMsg)+ }+ await waitFor(() => hops.length === 1_000 && json.length === 1_000)+ expect(hops.map(([, b]) => b)).toEqual(Array.from({ length: 1_000 }, (_, i) => i % 256))+ expect(json.map(([, n]) => n)).toEqual(Array.from({ length: 1_000 }, (_, i) => i))+ expect(new Set([...hops, ...json].map(([m]) => m))).toEqual(new Set(['machine-a']))+ expect(elsewhere).toEqual([])+ stopMine(); stopOther()+ })++ it('ignores a process-channel JSON message that is not for one connection', async () => {+ const mine = unique('pid'), got: unknown[] = []+ const stop = await bus.subscribePod(mine, { hop: () => {}, json: (_m, msg) => got.push(msg) })+ void bus.publishPodJson(mine, 'machine-a', { frame: { type: 'agent_synced', payload: {} } } as UpBusMsg)+ void bus.publishPodJson(mine, 'machine-a', { targetConnId: 'c', frame: { type: 'agent_synced', payload: {} } } as UpBusMsg)+ await waitFor(() => got.length === 1)+ await new Promise(resolve => setTimeout(resolve, 50))+ expect(got).toEqual([{ targetConnId: 'c', frame: { type: 'agent_synced', payload: {} } }])+ stop()+ })++ it('keeps a viewer\'s order across the switch from the machine\'s channel to its process\'s, as the owner reads both', async () => {+ const { createConnectionRoutes } = await import('./terminalRoute.js')+ const { startPodRoutes } = await import('./hub.js')+ const stopRoutes = await startPodRoutes()+ const owner = unique('pid'), conn = unique('conn'), machine = unique('machine'), got: number[] = []+ await bus.recordConnOwners([conn], owner)+ // The owner holds a viewer of the machine, so it reads the machine's channel too, on the same subscriber connection.+ const onFrame = (msg: UpBusMsg) => { if (msg.targetConnId === conn) got.push((msg.frame.payload as { n: number }).n) }+ const stopMachine = await bus.subscribeUp(machine, onFrame)+ const stopOwner = await bus.subscribePod(owner, { hop: () => {}, json: (_m, msg) => onFrame(msg) })+ const routes = createConnectionRoutes(machine)+ // The first frames go out while the owner is being looked up (the machine's channel), the rest after (the owner's).+ for (let i = 0; i < 1_000; i++) {+ routes.targeted({ targetConnId: conn, webEligible: true, frame: { type: 'terminal_output', payload: { n: i } } } as UpBusMsg & { targetConnId: string })+ if (i % 100 === 0) await new Promise(resolve => setImmediate(resolve))+ }+ await waitFor(() => got.length === 1_000)+ expect(got).toEqual(Array.from({ length: 1_000 }, (_, i) => i))+ routes.close(); stopMachine(); stopOwner(); stopRoutes()+ })++ it('routes a viewer\'s frames to the process its record names, in order, from an adapter on another process', async () => {+ const { createConnectionRoutes } = await import('./terminalRoute.js')+ const { startPodRoutes } = await import('./hub.js')+ const stopRoutes = await startPodRoutes()+ const owner = unique('pid'), conn = unique('conn'), got: number[] = []+ await bus.recordConnOwners([conn], owner)+ const stop = await bus.subscribePod(owner, { hop: () => {}, json: (_m, msg) => got.push((msg.frame.payload as { n: number }).n) })+ const routes = createConnectionRoutes('machine-b')+ // The first frame goes out on the machine's channel while the owner is looked up; once known, every frame goes to it.+ routes.targeted({ targetConnId: conn, webEligible: true, frame: { type: 'terminal_output', payload: { n: -1 } } } as UpBusMsg & { targetConnId: string })+ await new Promise(resolve => setTimeout(resolve, 50))+ for (let i = 0; i < 1_000; i++) routes.targeted({ targetConnId: conn, webEligible: true, frame: { type: 'terminal_output', payload: { n: i } } } as UpBusMsg & { targetConnId: string })+ await waitFor(() => got.length === 1_000)+ expect(got).toEqual(Array.from({ length: 1_000 }, (_, i) => i))+ routes.close(); stop(); stopRoutes()+ })+})diff --git a/backend/src/server.ts b/backend/src/server.tsindex e0c6b2bc3..17ab932a0 100644--- a/backend/src/server.ts+++ b/backend/src/server.ts@@ -34,6 +34,8 @@ import { selfProfileFromEnv } from './lib/selfProfile.js'import { startRelayUsageFlush } from './lib/productUsage.js'import { registerWorkContext, startQueryStats } from './lib/queryStats.js'import { startRetryStats } from './superTabs/retryStats.js'+import { startRelayStats } from './lib/relayStats.js'+import { startPodRoutes } from './lib/hub.js'import { registerRouteLatency, startRouteLatency } from './lib/routeLatency.js'import { drainAllSockets, openSocketCount, RELEASE_UPGRADE_SLOT, type SlotSocket } from './lib/wsServer.js'import { endpointOf, refusedUpgrade, startWsCounterLog, watchUpgrade } from './lib/wsAttribution.js'@@ -150,6 +152,13 @@ async function start(): Promise<void> {const stopRouteLatency = startRouteLatency(registerRouteLatency(app))// Every minute: who reads messages how, with the rows each page decoded and showed (superTabs/messageReadStats.ts).const stopMessageReadStats = startMessageReadStats()+ // Every minute: how each frame an adapter sends for one viewer reached it, and how many processes each publish woke.+ const stopRelayStats = startRelayStats()+ // Frames for one viewer go to the process holding it (lib/terminalRoute.ts), unless TERMINAL_POD_ROUTES=off.+ let stopPodRoutes = (): void => {}+ if (env.TERMINAL_POD_ROUTES !== 'off') {+ startPodRoutes().then(stop => { stopPodRoutes = stop }, err => logger.warn('pod routes failed to start; frames use the machine channels', { error: String(err) }))+ }app.setErrorHandler(errorHandler)@@ -231,6 +240,8 @@ async function start(): Promise<void> {stopRouteLatency()stopMachineReportLog()stopMessageReadStats()+ stopRelayStats()+ stopPodRoutes()// The database stays connected through the drain, for the requests still being served (lib/prisma.ts).await Promise.allSettled([prisma.$disconnect(), closeBus()])process.exit(exitCode)diff --git a/docs/superpowers/plans/2026-10-31-terminal-relay-direct.md b/docs/superpowers/plans/2026-10-31-terminal-relay-direct.mdnew file mode 100644index 000000000..c1b38c16f--- /dev/null+++ b/docs/superpowers/plans/2026-10-31-terminal-relay-direct.md@@ -0,0 +1,144 @@+# Terminal relay: per-connection frames go straight to their connection (Phases 1 and 2) — Implementation Plan++> **Implemented 2026-10-31** with the review changes noted in the spec (one switch instead of a queue; whole `UpBusMsg` on `uppod:`; untargeted frames local first). Task text below is the original plan.++> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.++**Goal:** An `adapter-ws` up frame for one connection is delivered directly when that connection is on this worker, and otherwise published only to the worker that holds it; a per-minute log says how many workers each published frame reaches.++**Architecture:** `terminalRoute.ts` decides, per connection and per adapter socket, local, owner's channel or today's channels (sticky from the first frame). Phase 1 (Tasks 1–3) is local delivery and the counters; Phase 2 (Tasks 4–6) adds owner records in Redis, a channel per worker, and routing to it behind `TERMINAL_POD_ROUTES`.++**Tech Stack:** TypeScript, `ws`, ioredis pub/sub (`lib/bus.ts`), vitest.++**Spec:** `docs/superpowers/specs/2026-10-31-terminal-relay-direct-design.md` (both phases, one delivery)++## Global Constraints++- Both phases ship together. Phase 2 runs unless `TERMINAL_POD_ROUTES=off`; with it off, Phase 1 alone runs.+- Owner record `PX 60 000`, renewed every `CLIENT_HEARTBEAT_MS` (25 s). A non-local connection uses the machine's channels until its owner is known, then the owner's; no owner or a failed lookup keeps the machine's channels (spec, Phase 2 "One switch, in order").+- A `pod:` message carries `machineId`; a JSON one carries the whole `UpBusMsg` and is delivered with `deliverUpLocal` (implementation call, see spec); both reach only that machine's client with that `connId`.+- Untargeted adapter frames are delivered locally first, then published with `originPid` (independent review).+- Each `pod:` channel rides the subscriber connection of the channel it replaces: binary on `terminalSub`, JSON on `sub`.+- A connection's frames arrive once, in order, byte-identical to today; send rules unchanged (`guardedSend` "must", kill-water).+- "Local" means this Node worker (`registry.PROCESS_ID`), not this pod.+- The route is sticky: decided by the connection's first frame on an adapter socket, kept until the adapter socket closes.+- Frames for every client of a machine (no `targetConnId`) are unchanged.+- No stored data, no client change, backend only. No ids in logs.+- Do not commit or push until Ken asks; each task ends with a diff checkpoint. Remove dead code.+- Commands from `backend/`: `npx vitest run <file>`; `npm run typecheck`. Real-Redis tests run with `REDIS_TEST_URL=redis://127.0.0.1:6379/15` (local docker `harness-redis`; CI skips them, as `src/superTabs/cacheRedis.test.ts` does).++## Review Focus++1. A local viewer that detaches mid-stream: later frames for its `connId` are dropped, never published to other pods — Task 2 test.+2. A JSON frame targeted at a device (`commander`) connection on this worker (E2EE pairing): delivered locally like a web one — Task 2 test.+3. A guest connection (shared machine) gets only frames addressed to it, locally as through Redis (`hub.test.ts` "a guest on a shared machine") — Task 2 test.+4. Redis down (`safePublish` returns 0): a local viewer still gets frames; stats count 0 receivers — Task 2 test.+5. Two adapter sockets for one machine (a reconnect overlap): each keeps its own sticky map; closing one does not clear the other's — Task 2 test.++---++### Task 1: Relay counters and the `terminal relay` log++**Files:**+- Create: `backend/src/lib/relayStats.ts`+- Modify: `backend/src/server.ts` (start and stop beside `startRetryStats`)+- Test: `backend/src/lib/relayStats.test.ts`++**Interfaces:**+- Produces: `type RelayKind = 'termup' | 'up targeted' | 'up broadcast'`; `recordRelay(kind: RelayKind, path: 'local' | 'published', receivers?: number): void`; `startRelayStats(options?: { everyMs?: number; log?: (message: string, context: Record<string, unknown>) => void }): () => void` logging `'terminal relay'` each `everyMs` (default 60 000) with `{ windowMs, kinds: { [kind]: { local, published, receivers } } }`, nothing in a window with no frames.++- [ ] **Step 1: Write the failing test** `'counts local and published frames and the receivers each publish reached, per kind'`: record `('termup','local')` ×2, `('termup','published',2)`, `('up targeted','published',1)`; with `everyMs: 50` expect `{ termup: { local: 2, published: 1, receivers: 2 }, 'up targeted': { local: 0, published: 1, receivers: 1 } }`, and a later empty window logs nothing.+- [ ] **Step 2: Run** `npx vitest run src/lib/relayStats.test.ts` — FAIL (module missing).+- [ ] **Step 3: Implement** the module and wire it in `server.ts`.+- [ ] **Step 4: Run** the test — PASS; `npm run typecheck` — passes.+- [ ] **Step 5: Checkpoint** `git diff --stat`.++### Task 2: Sticky per-connection routes++**Files:**+- Create: `backend/src/lib/terminalRoute.ts`+- Test: `backend/src/lib/terminalRoute.test.ts` (mock `./bus.js` as `hub.test.ts` does; attach clients with `attachHubClient`)++**Interfaces:**+- Consumes: `deliverTerminalUpLocal(machineId, packet)` and `deliverUpLocal(machineId, msg)` (`hub.ts`); `publishTerminalUp`, `publishUp` (`bus.ts`, they resolve to the receiver count); `registry.clientsFor(machineId)`; Task 1's `recordRelay`.+- Produces: `createConnectionRoutes(machineId: string): ConnectionRoutes` with+ - `hop(packet: Uint8Array, connId: string): void` — a binary terminal hop for `connId` (a `web` client);+ - `targeted(msg: UpBusMsg & { targetConnId: string }): void` — a JSON up frame for one connection (any client kind; `msg.targetKind` as today);+ - `close(): void` — forgets the sticky map (adapter socket closed).+ First frame per `connId`: local when `registry.clientsFor(machineId)` holds a client with that `connId` (and, for a hop, kind `web`), else published; the decision is kept in a `Map<connId, 'local' | 'published'>` for this object's life. Local → deliver with the hub function and `recordRelay(kind, 'local')`; published → today's publish, then `recordRelay(kind, 'published', receivers)`.+- [ ] **Step 1: Write the failing tests:**+ - `'delivers a hop and a targeted JSON frame to a local web connection without publishing'`: attach a web client, send a hop and a `terminal_output` JSON frame; the socket gets both, in order, the JSON text equal to `JSON.stringify(frame)`; `publishTerminalUp` and `publishUp` not called.+ - `'publishes for a connection that is not on this worker, as today'`: no client attached; `publishTerminalUp`/`publishUp` called with today's arguments.+ - `'keeps a connection on its first route: frames after a local viewer detaches are dropped, not published'` (Review Focus 1).+ - `'delivers a frame targeted at a local device connection'` (Review Focus 2).+ - `'sends a guest only frames addressed to it'` (Review Focus 3).+ - `'still delivers locally when Redis takes no publish'` (Review Focus 4; `publishUp` resolves 0).+ - `'keeps one adapter socket\'s routes when another closes'` (Review Focus 5).+- [ ] **Step 2: Run** `npx vitest run src/lib/terminalRoute.test.ts` — FAIL (module missing).+- [ ] **Step 3: Implement** `terminalRoute.ts`.+- [ ] **Step 4: Run** it with `src/lib/hub.test.ts` — PASS.+- [ ] **Step 5: Checkpoint** `git diff --stat`.++### Task 3: `adapter-ws` sends per-connection frames through the routes++**Files:**+- Modify: `backend/src/lib/adapterWs.ts` (create one `ConnectionRoutes` once `machineId` is known; binary hop at ~427; JSON `terminal_*` at ~443; `p2p_*` at ~458; the general publish at ~567 when `env.targetConnId` is set; `close()` in `cleanup`)+- Test: `backend/src/lib/adapterRoutes.test.ts` (copy the harness of `adapterShortTurns.test.ts`, lines 1–60, and mock `./terminalRoute.js`)++**Interfaces:**+- Consumes: Task 2's `createConnectionRoutes`; Task 1's `recordRelay` for the untargeted general publish (`'up broadcast'`, `'published'`, its receivers).+- [ ] **Step 1: Write the failing test** `'routes a hop, a terminal frame, a p2p frame and a targeted reply by their connection, and publishes the rest as before'`: through the adapter socket send a binary output hop (connId `c1`), a JSON `terminal_output` with `targetConnId: 'c1'`, a `p2p_offer` with `targetConnId: 'c1'`, an `e2e_welcome` with `targetConnId: 'c2'`, and an untargeted event; expect `hop(packet, 'c1')`, `targeted` ×3 with today's message fields (`webEligible`, `commanderEligible`, `targetKind`, `targetConnId`, `frame`), and `publishUp` once for the untargeted event; the route's `close` is called when the socket closes. Today's validation still drops a bad frame before routing (one invalid terminal frame is not routed).+- [ ] **Step 2: Run** `npx vitest run src/lib/adapterRoutes.test.ts` — FAIL.+- [ ] **Step 3: Implement** the wiring; keep every existing check (rate limits, sizes, encryption) before routing.+- [ ] **Step 4: Run** it with `src/lib/adapterShortTurns.test.ts`, `src/lib/hub.test.ts`, `src/lib/hubLiveness.test.ts`, `src/lib/webWs*.test.ts` — PASS.+- [ ] **Step 5: Checkpoint** `git diff --stat`.++### Task 4: Owner records (Phase 2)++**Files:**+- Modify: `backend/src/lib/bus.ts` (owner record functions), `backend/src/lib/hub.ts` (`attachHubClient`, the detach path, `clientHeartbeatTick`), `backend/src/config/env.ts` (`TERMINAL_POD_ROUTES`, default `'on'`)+- Test: `backend/src/lib/connOwnersRedis.test.ts` (real Redis, `describe.skipIf(!process.env.REDIS_TEST_URL)`), `backend/src/lib/hub.test.ts` (mocked bus)++**Interfaces:**+- Produces in `bus.ts`: `recordConnOwners(connIds: string[], processId: string): Promise<void>` (one pipeline, `SET conn:{connId} processId PX 60000` each); `forgetConnOwner(connId: string): Promise<void>`; `connOwner(connId: string): Promise<string | null>`. Hub calls `recordConnOwners([connId])` on attach, `forgetConnOwner` on detach, and `recordConnOwners(all local connIds)` in `clientHeartbeatTick`, all only when the switch is on.+- [ ] **Step 1: Write the failing tests:** real Redis: a recorded owner reads back, expires after 60 s (use `PTTL`), is gone after forget, and a batch of 500 is one round trip (`MONITOR`-free: count commands with a wrapped client). Mocked bus: attach records, detach forgets, the heartbeat renews every local `connId`; with `TERMINAL_POD_ROUTES=off` none of these are called.+- [ ] **Step 2: Run** `REDIS_TEST_URL=redis://127.0.0.1:6379/15 npx vitest run src/lib/connOwnersRedis.test.ts src/lib/hub.test.ts` — FAIL.+- [ ] **Step 3: Implement.**+- [ ] **Step 4: Run** them — PASS.+- [ ] **Step 5: Checkpoint** `git diff --stat`.++### Task 5: A channel per worker (Phase 2)++**Files:**+- Modify: `backend/src/lib/bus.ts`, `backend/src/lib/hub.ts` (subscribe once at startup when the switch is on)+- Test: `backend/src/lib/podChannelRedis.test.ts` (real Redis), `backend/src/lib/hub.test.ts`++**Interfaces:**+- Produces in `bus.ts`: `publishPodHop(processId: string, machineId: string, packet: Uint8Array): Promise<number>` on channel `termpod:{processId}` (payload = 2-byte length + `machineId` UTF-8 + packet); `publishPodJson(processId: string, msg: { machineId: string; connId: string; targetKind?: 'web' | 'commander'; text: string }): Promise<number>` on `uppod:{processId}`; `subscribePod(processId: string, on: { hop(machineId: string, packet: Uint8Array): void; json(msg: …): void }): Promise<() => void>` — `termpod:` on `terminalSub`, `uppod:` on the JSON `sub` connection.+- Produces in `hub.ts`: `deliverTargetedTextLocal(machineId: string, connId: string, text: string, targetKind?: 'web' | 'commander'): boolean` — sends `text` to that one client with the same checks and send rules `deliverUpLocal` applies to a `targetConnId` frame; `false` when no such client of that machine. The hop side calls `deliverTerminalUpLocal(machineId, packet)`, which already checks the client is that machine's.+- [ ] **Step 1: Write the failing tests:** real Redis: 1,000 hops and 1,000 JSON texts published to one worker arrive once and in order on their own connection; a message for `machineId` A with a `connId` that belongs to machine B is not delivered (both hop and JSON). Mocked: `deliverTargetedTextLocal` sends byte-identical text, honours `targetKind`, returns `false` for a closed connection; a guest gets only its own frames.+- [ ] **Step 2: Run** them — FAIL.+- [ ] **Step 3: Implement.**+- [ ] **Step 4: Run** them with `src/lib/hub.test.ts` and `src/lib/bus.test.ts` — PASS.+- [ ] **Step 5: Checkpoint** `git diff --stat`.++### Task 6: Routes publish to the owner (Phase 2)++**Files:**+- Modify: `backend/src/lib/terminalRoute.ts`+- Test: `backend/src/lib/terminalRoute.test.ts`, `backend/src/lib/terminalRouteRedis.test.ts` (real Redis)++**Interfaces:**+- Consumes: Task 4's `connOwner`; Task 5's `publishPodHop`, `publishPodJson`; Task 1's `recordRelay` with new kinds `'pod hop'` and `'pod json'`.+- Produces: the same `ConnectionRoutes` API. The first frame of a non-local connection starts `connOwner(connId)`; frames for that connection queue in arrival order until it answers (or 250 ms pass); then the route is fixed: an owner that is not this worker → `pod:` publishes (the JSON frame stringified once, as `text`); no owner, a failure, a timeout, or the switch off → today's channels. The queue drains in order before any newer frame.+- [ ] **Step 1: Write the failing tests:** an owner on another worker → `publishPodHop`/`publishPodJson` only, never `publishTerminalUp`/`publishUp`; no owner → today's channels for the connection's life, even when an owner appears later; a lookup that takes 300 ms → today's channels, frames sent in order; 50 frames sent while the lookup is pending go out in order after it; switch off → no lookup. Real Redis: two subscriber connections standing for two workers, an owner record for worker B, 1,000 frames from worker A for B's connection arrive at B in order and none at a third worker.+- [ ] **Step 2: Run** `npx vitest run src/lib/terminalRoute.test.ts` and the Redis test — FAIL.+- [ ] **Step 3: Implement.**+- [ ] **Step 4: Run** them with Tasks 2–5's tests — PASS.+- [ ] **Step 5: Checkpoint** `git diff --stat`.++### Task 7: Checks++- [ ] **Step 1:** `npm run typecheck`; `npx vitest run src/lib src/routes` — only failures already on clean `main`, listed in the PR.+- [ ] **Step 2:** Run the real-Redis tests locally: `REDIS_TEST_URL=redis://127.0.0.1:6379/15 npx vitest run src/lib/*Redis.test.ts`; record the result in the PR (CI skips them).+- [ ] **Step 3:** After release, from `cpu by kind` (`adapter-ws *`, `bus terminal *`, `bus main up`, `bus * termpod|uppod`) and `terminal relay`: total relay busy time before and after, the local share, and receivers per published frame. If receivers were already about 1 before, set `TERMINAL_POD_ROUTES=off` and file the removal of Phase 2.diff --git a/docs/superpowers/specs/2026-10-31-terminal-relay-direct-design.md b/docs/superpowers/specs/2026-10-31-terminal-relay-direct-design.mdnew file mode 100644index 000000000..205c9caa2--- /dev/null+++ b/docs/superpowers/specs/2026-10-31-terminal-relay-direct-design.md@@ -0,0 +1,145 @@+# Design: per-connection frames go straight to the one connection they are for (terminal relay)++Date: 2026-10-31. Code read at `origin/main` (664dd0f63). Spec 6 in the backend CPU series.++## Why++A remote terminal's output reaches its viewer like this: the machine's daemon sends it up its `adapter-ws` socket to+whichever backend pod holds that socket; that pod publishes it to Redis; every pod with a viewer of that machine receives+it and hands it to its viewer.++**Every terminal frame is for exactly one viewer connection.** A binary terminal hop carries its `connId`+(`deliverTerminalUpLocal`, `hub.ts:44`), and a JSON `terminal_*` / `p2p_*` up frame carries `targetConnId`+(`adapterWs.ts:437-470`). Yet:++- `adapterWs.ts:427` publishes every binary hop to `termup:{machineId}`, and `adapterWs.ts:443,458` every JSON one to+ `up:{machineId}`, **even when the viewer is on the same pod**: the pod receives its own frame back from Redis.+ (`managerWs.ts:255-256` already delivers locally first and skips its own echo with `originPid`; the adapter path does+ not.)+- Every pod holding any client of that machine receives the frame, decodes it, and drops it unless it owns the+ `connId`.+- A JSON up frame is encoded twice: `publishUp` stringifies the whole envelope, the receiver parses it, and+ `deliverUpLocal` (`hub.ts:59`) stringifies `msg.frame` again before sending.++**Prod (2026-10-30/31, 16 CPU profiles, 3 pods):** relay and socket work is ~13.5% of non-idle CPU (Redis bus 2.5%,+ioredis 2.5%, socket writes 6.6%, `ws` receiver 0.9%), consistent with the 17.5% of event-loop time logged as+`untagged`. #2393 names these kinds (`adapter-ws binary|text`, `bus terminal termup`, `bus main up`) from the next+release, which will say exactly how much is terminal relay.++## Goals++- A terminal frame whose viewer is on the pod that received it never goes through Redis.+- A frame whose viewer is on another pod reaches only that pod (Phase 2), behind a kill switch.+- A JSON terminal frame is not parsed and re-stringified on its way to the viewer.+- Ordering, delivery guarantees ("must" sends, kill-water) and E2EE are unchanged: the backend still forwards opaque+ sealed bytes.++## Non-goals++- Up frames that are not for one connection (status, events, RPC replies to all clients): they keep `up:{machineId}`.+- Down frames (viewer → machine): `sendDown` already goes direct when the control socket is local (`hub.ts`).+- Moving terminals to p2p, the multiplexed account socket (WebSocket roadmap Phases 4–5).++## Anti-goals++- A frame delivered twice, lost, or out of order for its connection.+- A viewer on a pod running the previous release missing frames during a rolling deploy.++## Design decided++*(Author's calls after the builder check: the phases, the sticky route, and every per-connection adapter frame.)*++### Phase 1 — local first, encode once, count the receivers++- **Which frames:** every `adapter-ws` up frame for one connection: a binary terminal hop (its `connId`) and any JSON up+ frame with a `targetConnId` (`terminal_*`, `p2p_*`, and the general per-connection publish at `adapterWs.ts:565`, such+ as `e2e_welcome` and per-client RPC replies). Keeping all of a connection's frames on one path keeps their order.+ Frames for every client of a machine (no target) are unchanged.+- **Local:** when that connection is on this process (`registry.clientsFor(machineId)` holds its `connId`; `PROCESS_ID`+ is per Node worker, so "local" means the same worker), the frame is delivered there directly with today's send rules+ (`deliverTerminalUpLocal` for a hop; for JSON the same per-connection path `deliverUpLocal` takes, from the adapter's+ already-parsed frame, stringified once) and is not published.+- **Local is for good:** a connection is local or not for its whole life (`connId` is a new `randomUUID()` per attach+ and never moves), so a local route never changes.+- **Untargeted frames go local first too** *(after the independent review)*: an adapter frame for every client of the+ machine is delivered to this process's clients first, then published with `originPid` (this process's own Redis+ subscription skips it), as `managerWs` does. Otherwise a local viewer's targeted reply would overtake an event the+ daemon sent before it. Observers and `nodeRpc` ignore `originPid` and still get the Redis copy.+- A `targetConnId` that is not a string of 1–128 characters is published as before, never used as a route or a key.+- **Receiver count:** `PUBLISH` answers how many subscriptions received a message. A per-minute log `terminal relay`+ counts, per channel kind (`termup`, `up` targeted, `up` broadcast): frames delivered locally, frames published, and the+ receivers they reached in total. That says how many pods each published frame wakes for nothing.++### Phase 2 — publish only to the owning process (built with Phase 1; Ken, 2026-10-31)++- On attach a pod records `conn:{connId} → PROCESS_ID` in Redis (`SET … PX 60 000`, renewed with the client heartbeat,+ deleted on detach) and each process subscribes to its own `pod:{PROCESS_ID}` channel(s).+- **One switch, in order** *(after the independent review; replaces a queue and a 250 ms lookup timeout)*: a+ non-local connection's frames go to the machine's channels while its owner is looked up, then, if another process+ owns it, to that process's channels; with no owner (old process, expired record, failed lookup) they stay on the+ machine's channels. The switch keeps order: both channels are published from the one `pub` connection; the owner reads+ `up:`/`uppod:` on `sub` and `termup:`/`termpod:` on `terminalSub`, one subscriber connection per kind; and the owner holds+ a viewer of the machine, so it reads the machine's channels. Nothing waits for the lookup, so untargeted frames sent+ after a targeted one never overtake it.+- The receiver delivers with the existing functions: a hop with `deliverTerminalUpLocal`, a JSON message (the whole+ `UpBusMsg`, *author's call during implementation*: a device connection's frames need the machine tag and the voice+ queue `deliverUpLocal` applies) with `deliverUpLocal`. Both only reach that process's clients **of the message's+ `machineId`** with that `connId`, so a daemon's frame never reaches another machine's connection. A `uppod:` message+ without a string `targetConnId` is ignored.+- **Kill switch:** `TERMINAL_POD_ROUTES=off` stops writing records and routing to `pod:` (on by default). With it off,+ Phase 1 still runs; records already written expire within 60 s.++### Risks (and what covers each)++| Risk | Cover |+|---|---|+| A frame lost or reordered when the path changes | A route changes at most once, machine → owner, which keeps order (one publisher connection, one subscriber connection per kind); real-Redis tests of 1,000 frames across that switch |+| A viewer on an old pod during the rolling deploy | No record from old pods → today's channels; new pods keep today's subscriptions |+| A pod that crashed leaves records for 60 s | Frames to its `pod:` channel reach nobody, as frames to its dead sockets do today; its viewers reconnect elsewhere with a new `connId`, which gets a new record |+| A daemon targeting another machine's connection | The receiver checks `connId` against its clients of the frame's `machineId` |+| More Redis writes: a record per attach, renewed per heartbeat (Desktop re-attaches on every agent switch) | One `SET` per attach and per 25 s per connection, one pipeline per heartbeat; one `GET` per connection, once; at most 2,000 routes kept per adapter socket |+| A bug that only shows across real pods | A real-Redis test (two subscriber connections standing for two processes, `REDIS_TEST_URL`), the kill switch, and the `terminal relay` log after release |+| Little gain if most machines have viewers on one pod | `terminal relay` counts receivers per publish before and after; the kill switch turns Phase 2 off if it does not pay |+++## Left to the builder++- Module layout (a small `terminalRoute.ts` beside `hub.ts` is one option), log and key names.+- Phase 2's exact channel and key names (two channels, one per subscriber connection, as decided).++## Tests++- **Phase 1:** a local viewer gets a binary hop, a JSON terminal frame and an `e2e_welcome` with no Redis publish, in the+ order sent, byte-identical to today; a remote viewer gets them through today's channels; a local connection that+ detaches mid-stream drops the rest; the `terminal relay` log counts local, published and receivers per kind.+- **Phase 2:** a remote viewer's frames go to `pod:{owner}` only and a third pod with another client of the+ machine receives nothing; an unknown owner stays on today's channels for the connection's life; 1,000 frames for one+ connection arrive once and in order, with the record appearing midway; re-attach on another pod routes the new+ `connId` to its pod.+- Existing hub, adapter, webWs and terminal E2E suites stay green.++## Measuring++After each release, from #2393's kinds and Phase 1's log: the **total** relay busy time per pod-minute across+`adapter-ws *`, `bus terminal *`, `bus main up` and (Phase 2) `bus pod *`; Redis publishes per minute; receivers per+published frame. **Done when** Phase 1 shows the local share delivered without Redis and total relay time falls, with no+rise in terminal stall reports (`TERMINAL_RENDER_STALLED`); with Phase 2, receivers per published frame fall to about 1.+If they were already about 1, Phase 2 is switched off (`TERMINAL_POD_ROUTES=off`) and removed in a later change.++## Design (AGENTS.md rules)++- **Growth:** Phase 1 stores nothing. Phase 2: one Redis key per open viewer connection, expiring 60 s after its last+ heartbeat; one channel per process.+- **Concurrency:** no locks; a connection's frames keep one sticky path, in send order.+- **Read cost:** Phase 1 none. Phase 2: one Redis `GET` per connection, once, on the sending process.+- **Scale:** at 10× viewers or pods, a frame still costs one local send or one publish to one pod; the per-machine fan-out+ no longer grows with pods. Redis publish rate is the first thing to watch.+- **Compatibility:** old pods write no record, so frames for their viewers fall back to today's channels; new pods keep+ subscribing to those channels.+- **Migration + rollback:** nothing stored beyond expiring keys; `TERMINAL_POD_ROUTES=off` turns Phase 2 off without a+ deploy; roll back by deploying the previous release.+- **Audience:** unchanged; a frame reaches only the connection it names, as today.+- **Identity:** N/A because nothing here acts as or reads the machine owner's login.+- **Failure:** Redis unreachable → local viewers still get frames; remote ones fail as today. A missing record falls+ back to broadcast, never drops.+- **Platforms:** backend only.