Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

Batch S: sidebar stops re-reading gone conversations (#2099), WebSocket Phase 1a (#2101) (#2109)

0xken157fcb051db908
diff --git a/backend/src/config/env.ts b/backend/src/config/env.ts
index b74888c2b..9dad25fd2 100644
--- a/backend/src/config/env.ts
+++ b/backend/src/config/env.ts
@@ -51,6 +51,10 @@ const envSchema = z.object({
// that lands on an instance NOT holding the agent's manager socket is forwarded to the instance that
// does (agent:{machineId}:appinst), so the data path is a direct backpressured socket, not Redis relay.
// Off by default (single-instance/dev needs nothing — it's always co-located). Enable in multi-instance.
+ // Roadmap Phase 1.3: refuse a bad token on /api/web-ws with HTTP 401 before the upgrade, instead of upgrading and closing
+ // 4401, for sockets that declare `auth=http` (the daemon's relay from 1.3.64). Off by default; turn on once that release
+ // is out. Browsers and the phone keep the 4401, since a browser cannot read the status of a failed upgrade.
+ WEB_WS_AUTH_HTTP_401: z.string().default('false').transform((v) => v === 'true'),
MESH_ENABLED: z.string().default('false').transform((v) => v === 'true'),
// Host the mesh endpoint is reachable at from OTHER instances: 127.0.0.1 for a single-host pm2 cluster,
// the VM's routable IP for multi-VM. Advertised verbatim in Redis.
diff --git a/backend/src/lib/webWs.ts b/backend/src/lib/webWs.ts
index 7dd531cc1..7bf43324b 100644
--- a/backend/src/lib/webWs.ts
+++ b/backend/src/lib/webWs.ts
@@ -28,7 +28,7 @@ import { touchDeviceKey } from './deviceKeyLog.js'
import { getAgentPresence, getAgentPresenceMany, subscribeStatus, getDevicePresence, subscribeDeviceStatus, subscribeDeviceMachineListChanged, subscribeDeviceE2eePair, subscribeMachineShareChanged } from './bus.js'
import { sharedMachineAccess } from './machineAccess.js'
import { relayWebDocumentPushes } from './webAccountPushes.js'
-import { DAEMONS } from '../config/env.js'
+import { DAEMONS, env } from '../config/env.js'
import { attachHubClient, trackSocketLiveness, type HubClient } from './hub.js'
import { authenticateAccessToken, SsoAuthError, type AuthUser } from './ssoAuth.js'
import type { Frame } from './tunnel.js'
@@ -121,8 +121,8 @@ export function handleWebUpgrade(req: IncomingMessage, socket: Duplex, head: Buf
// PER-USER mode: profile validation is async, so authenticate before completing the upgrade.
void (async () => {
+ const query = new URL(req.url ?? '/api/web-ws', 'http://backend.local').searchParams
try {
- const query = new URL(req.url ?? '/api/web-ws', 'http://backend.local').searchParams
const requestedEnvironment = parseAutonomousEnvironment(query.get('autonomousEnv'))
// Who is on the other end, as the client declares it: the mobile app and the web viewer say so;
// the daemon's relay (and any older client) says nothing and is never counted as a person.
@@ -139,6 +139,11 @@ export function handleWebUpgrade(req: IncomingMessage, socket: Duplex, head: Buf
// A Mac's team account (lib/teamSession.ts) never selects or attaches to a machine here: refused for good,
// not an expired sign-in to refresh and retry.
const team = err instanceof SsoAuthError && err.code === 'TEAM_SESSION_LIMITED'
+ if (invalid && env.WEB_WS_AUTH_HTTP_401 && query.get('auth') === 'http') {
+ try { socket.write('HTTP/1.1 401 Unauthorized\r\nConnection: close\r\nContent-Length: 0\r\n\r\n') } catch { /* ignore */ }
+ socket.destroy()
+ return
+ }
const code = invalid ? 4401 : mismatch || team ? 4403 : 1013
const reason = invalid
? 'auth expired'
@@ -168,7 +173,7 @@ function attachUserClient(ws: WebSocket, user: AuthUser, surface: 'mobile' | 'we
// Its connect and disconnect lines (lib/wsAttribution.ts): prod, 2026-10-28, 2,304 of 2,308 opens in 15 minutes
// declared no client and lived a median 0.4 s, and these lines named only the user.
const track = trackSocket('web', 'web user', ws, { ...declared, userId: user.sub })
- const refuse = (payload: { machineId: string; error: string; retryAfterMs?: number }): void => {
+ const refuse = (payload: { machineId: string; error: string; retryAfterMs?: number; reason?: 'deleted' }): void => {
track.selected(payload.error)
send({ type: 'machine_select_error', payload })
}
@@ -373,7 +378,8 @@ function attachUserClient(ws: WebSocket, user: AuthUser, surface: 'mobile' | 'we
// client's stale list; a failed share check is ours.
const reason = !binding ? 'unknown' : binding.deletedAt ? 'deleted' : shareCheckFailed ? 'share_check_failed' : 'not_shared'
logger.warn('machine_select rejected', { userId: user.sub, machineId, reason })
- refuse({ machineId: machineId, error: 'NOT_YOUR_MACHINE' })
+ // Only the person's own machine says it was deleted (a daemon then stops dialling it); anyone else's says nothing more.
+ refuse({ machineId: machineId, error: 'NOT_YOUR_MACHINE', ...(binding?.deletedAt && binding.userId === user.sub ? { reason: 'deleted' as const } : {}) })
return
}
if (storedAutonomousEnvironment(binding.autonomousEnv) !== user.autonomousEnv) {
diff --git a/backend/src/lib/webWsOffline.test.ts b/backend/src/lib/webWsOffline.test.ts
index 1f2660e50..3b019206f 100644
--- a/backend/src/lib/webWsOffline.test.ts
+++ b/backend/src/lib/webWsOffline.test.ts
@@ -29,6 +29,8 @@ vi.mock('./webAccountPushes.js', () => ({ relayWebDocumentPushes: vi.fn(async ()
vi.mock('./hub.js', () => ({ attachHubClient: m.attach, trackSocketLiveness: () => () => {} }))
import { handleWebUpgrade, OFFLINE_RETRY_AFTER_MS } from './webWs.js'
+import { SsoAuthError } from './ssoAuth.js'
+import { env } from '../config/env.js'
const owner = { sub: 'owner', email: '[email protected]', role: 'user', autonomousEnv: 'prod' }
const mac = { machineId: 'owner-mac', userId: 'owner', autonomousEnv: 'prod', billingStatus: 'not_required', authMode: 'remote', deletedAt: null }
@@ -100,9 +102,57 @@ describe('a refused machine select', () => {
if (failure) m.shared.mockRejectedValue(failure)
else m.shared.mockResolvedValue(null)
const ws = await selects({}, '&client=web')
- expect(ws.sent()).toContainEqual({ type: 'machine_select_error', payload: { machineId: 'owner-mac', error: 'NOT_YOUR_MACHINE' } })
+ const payload = { machineId: 'owner-mac', error: 'NOT_YOUR_MACHINE', ...(reason === 'deleted' ? { reason } : {}) }
+ expect(ws.sent()).toContainEqual({ type: 'machine_select_error', payload })
expect(m.warn).toHaveBeenCalledWith('machine_select rejected', { userId: 'owner', machineId: 'owner-mac', reason })
})
+
+ // Phase 1 (roadmap 2026-10-28): a daemon kept dialling a machine deleted on 2026-09-03 every few seconds, because a deleted
+ // machine, an unshared one and a share check that failed for a moment were all the same refusal. Only the person's own
+ // deleted machine says so; anyone else's says nothing more than before, so the answer tells no one that it existed.
+ it('says deleted only of the person\'s own machine', async () => {
+ m.machineFindUnique.mockResolvedValue({ ...mac, userId: 'someone-else', deletedAt: new Date('2026-10-01T00:00:00Z') })
+ const ws = await selects({}, '&client=relay')
+ expect(ws.sent()).toContainEqual({ type: 'machine_select_error', payload: { machineId: 'owner-mac', error: 'NOT_YOUR_MACHINE' } })
+ })
+})
+
+// Roadmap Phase 1.3: a bad token used to finish the upgrade and then close 4401. A socket that says it reads an HTTP refusal
+// (`auth=http`, the daemon's relay from this release) is refused before the upgrade when WEB_WS_AUTH_HTTP_401 is on. A
+// browser cannot read the status of a failed upgrade (it sees 1006), so the web build and the phone keep their 4401.
+describe('a bad token', () => {
+ const upgradeWith = async (query: string) => {
+ const ws = new Socket()
+ const socket = raw()
+ m.upgrade.mockImplementation((_r, _s, _h, done) => done(ws))
+ m.auth.mockRejectedValue(new SsoAuthError('expired', 'INVALID_TOKEN'))
+ handleWebUpgrade({ url: `/api/web-ws?autonomousEnv=prod${query}`, headers: { 'sec-websocket-protocol': 'access-token' } } as never, socket as never, Buffer.alloc(0))
+ await flush()
+ return { ws, socket }
+ }
+ const switchedOn = (on: boolean) => { (env as { WEB_WS_AUTH_HTTP_401: boolean }).WEB_WS_AUTH_HTTP_401 = on }
+
+ it('is refused with HTTP 401 before the upgrade, when switched on, for a socket that reads it', async () => {
+ switchedOn(true)
+ try {
+ const { socket } = await upgradeWith('&client=relay&v=1.3.64&auth=http')
+ expect(m.upgrade).not.toHaveBeenCalled()
+ expect(socket.write).toHaveBeenCalledWith(expect.stringMatching(/^HTTP\/1\.1 401 /))
+ expect(socket.destroy).toHaveBeenCalled()
+ } finally { switchedOn(false) }
+ })
+
+ it.each([
+ ['switched off', false, '&client=relay&auth=http'],
+ ['a socket that does not say it reads it', true, '&client=web'],
+ ])('closes 4401 after the upgrade when %s', async (_why, on, query) => {
+ switchedOn(on)
+ try {
+ const { ws, socket } = await upgradeWith(query)
+ expect(socket.write).not.toHaveBeenCalled()
+ expect(ws.close).toHaveBeenCalledWith(4401, 'auth expired')
+ } finally { switchedOn(false) }
+ })
})
// Prod, 2026-10-28: 2,304 of 2,308 opens in 15 minutes declared no client and lived a median 0.4 s, and the disconnect
diff --git a/cli/e2e/socketChurn.e2e.ts b/cli/e2e/socketChurn.e2e.ts
index 50853d291..504afc20b 100644
--- a/cli/e2e/socketChurn.e2e.ts
+++ b/cli/e2e/socketChurn.e2e.ts
@@ -25,6 +25,8 @@ import { beforeAll, describe, expect, it } from 'vitest'
import { runSocketScenario, shortfalls, type ScenarioResult } from './harness/socketScenario.js'
const minutes = Number(process.env.SOCKET_SCENARIO_MINUTES ?? 3)
+/** The scenario's daemons, A and B (harness/socketScenario.ts). */
+const DAEMONS = 2
let result: ScenarioResult
/** When each web-ws select of `machineId` was sent, in order. */
@@ -54,8 +56,11 @@ describe.skipIf(!process.env.HARNESS_E2E_SERVICES_DIR)('WebSocket phases 0 and 1
expect(below).toEqual([])
})
- it('Phase 1: a deleted machine is selected once, by the socket that finds it gone, and never again', () => {
- expect(selectTimes(result.machines.deleted).length).toBeLessThanOrEqual(1)
+ // Once per daemon, not once per account (Lead, 2026-10-29, #2101): A pinned the deleted machine, and B learns it from A's
+ // roster before A finds it gone, so each finds it gone once (remoteRelay.ts marks it gone) and neither dials it again. The
+ // meter cannot tell the two daemons apart; a daemon that kept dialling would fail this within a minute (main: 16 in 3).
+ it('Phase 1: a deleted machine is selected at most once by each daemon, by the socket that finds it gone, and never again', () => {
+ expect(selectTimes(result.machines.deleted).length).toBeLessThanOrEqual(DAEMONS)
})
it('Phase 1: an offline machine is selected again only after the backend\'s retryAfterMs (30 s)', () => {
diff --git a/cli/src/gateway/start.ts b/cli/src/gateway/start.ts
index 8a9e6ed16..ebcef5c66 100644
--- a/cli/src/gateway/start.ts
+++ b/cli/src/gateway/start.ts
@@ -32,7 +32,7 @@ import { b64d, b64e, fingerprint } from '../lib/e2ee/core.js'
import { DeviceLogStore } from '../lib/e2ee/deviceLogStore.js'
import { DeviceLogSyncer, type DeviceKeysSeen, type DeviceLogFetched, type DeviceLogTrustOutcome } from '../lib/e2ee/deviceLogSyncer.js'
import { GroupSyncer, relayRequester, SELF_STAMP } from '../lib/e2ee/groupSyncer.js'
-import { MACHINE_KINDS_READ_TIMEOUT_MS, MachineKinds } from '../lib/e2ee/machineKinds.js'
+import { MACHINE_KINDS_READ_TIMEOUT_MS, MachineKinds, machineKindsOf } from '../lib/e2ee/machineKinds.js'
import { MachinePeerStore } from '../lib/e2ee/machinePeers.js'
import { applySharedTrust, isSharedKey } from './sharedTrust.js'
import { E2eeStore } from '../lib/e2ee/store.js'
@@ -146,7 +146,12 @@ export function startGateway(host: GatewayHost): StartedGateway {
computer: { id: host.computerId, name: host.machineName, hostname: host.hostname }, environment: host.autonomousEnv,
changed: host.machines,
reachable: (ids) => { reachable = ids ? new Set(ids) : null },
- listed: (body) => { if (body) machineKinds.adopt(body); else machineKinds.clear() },
+ listed: (body) => {
+ if (!body) { machineKinds.clear(); return }
+ machineKinds.adopt(body)
+ // A machine marked gone (deleted, remoteRelay.ts) that the list names again is back: pin it again from the roster now.
+ if (relayPeers.restore(machineKindsOf(body)?.keys() ?? [])) groupSyncer?.resume()
+ },
log: (line) => console.log(`[gateway] ${line}`),
})
const gateway = new RelayGateway({
diff --git a/cli/src/lib/e2ee/groupSyncer.spec.ts b/cli/src/lib/e2ee/groupSyncer.spec.ts
index 43dfaca0f..bb10453d6 100644
--- a/cli/src/lib/e2ee/groupSyncer.spec.ts
+++ b/cli/src/lib/e2ee/groupSyncer.spec.ts
@@ -28,6 +28,10 @@ class MemPeers {
}
unlink(id: string): boolean { return this.pins.delete(id) }
list(): Array<Pin & { fingerprint: string }> { return [...this.pins.values()].map((p) => ({ ...p, fingerprint: 'fp' })) }
+ goneIds = new Set<string>()
+ gone(): Set<string> { return new Set(this.goneIds) }
+ markGone(id: string): void { this.pins.delete(id); this.goneIds.add(id) }
+ restore(listed: Iterable<string>): boolean { let any = false; for (const id of listed) any = this.goneIds.delete(id) || any; return any }
}
interface Node {
@@ -48,6 +52,8 @@ interface Node {
teamFree: boolean
/** How often the list was read again. */
reads: number
+ /** The machines this one sent a request to. */
+ asked: string[]
}
/** A small fleet whose "relay" is a direct call: a request reaches the target only if the caller pinned
@@ -78,6 +84,7 @@ function fleet(): { add: (name: string, machine?: boolean) => Node; link: (joine
kindOf: 'computer',
teamFree: false,
reads: 0,
+ asked: [],
} as unknown as Node
node.syncer = new S.GroupSyncer({
store: memStore,
@@ -93,6 +100,7 @@ function fleet(): { add: (name: string, machine?: boolean) => Node; link: (joine
teamFree: () => node.teamFree,
},
request: async (machineId, frame: Frame) => {
+ node.asked.push(machineId)
const target = nodes.find((n) => n.machineId === machineId)
if (!target?.online) return null
const pin = node.peers.get(machineId)
@@ -123,6 +131,33 @@ function fleet(): { add: (name: string, machine?: boolean) => Node; link: (joine
}
describe('GroupSyncer', () => {
+ // Roadmap Phase 1.1: a machine deleted from the account stayed in the roster, so every round pinned it again and the relay
+ // dialled it again, for good. Once the relay marks it gone (the backend said it was deleted), nothing here pins or dials it,
+ // until the account's machine list shows it again.
+ it('never pins or dials a machine marked gone, until it is listed again', async () => {
+ vi.useFakeTimers()
+ const f = fleet()
+ const [a, x, b] = [f.add('a'), f.add('c'), f.add('b')]
+ f.link(x, a); f.link(b, a)
+ await a.syncer.syncAll()
+ expect(a.peers.get(x.machineId!)).not.toBeNull()
+
+ a.peers.markGone(x.machineId!)
+ a.asked.length = 0
+ await a.syncer.syncAll()
+ // B still tells A about X; that must not put X back either.
+ await b.syncer.syncAll()
+ a.syncer.resume()
+ expect(a.peers.get(x.machineId!)).toBeNull()
+ expect(a.asked).not.toContain(x.machineId)
+ expect(a.asked).toContain(b.machineId)
+
+ a.peers.restore([x.machineId!])
+ await a.syncer.syncAll()
+ expect(a.peers.get(x.machineId!)?.pub).toBe(x.pub)
+ expect(a.asked).toContain(x.machineId)
+ })
+
it('never makes a machine shared across accounts part of the group, either way', async () => {
vi.useFakeTimers()
const f = fleet()
diff --git a/cli/src/lib/e2ee/groupSyncer.ts b/cli/src/lib/e2ee/groupSyncer.ts
index 8709ef8ba..a150c2441 100644
--- a/cli/src/lib/e2ee/groupSyncer.ts
+++ b/cli/src/lib/e2ee/groupSyncer.ts
@@ -266,6 +266,7 @@ export class GroupSyncer {
for (const m of this.deps.store.read().members) if (m.kind === 'machine' && m.machineId && !suspended.has(m.pub)) targets.add(m.machineId)
for (const p of this.deps.peers.list()) if (!suspended.has(p.pub) && !p.shared) targets.add(p.machineId)
if (self) targets.delete(self)
+ for (const machineId of this.deps.peers.gone()) targets.delete(machineId)
// A team machine, or one the account's list cannot place, is not dialed: not a member that missed this round.
for (const machineId of targets) if (!this.ordinary(machineId)) targets.delete(machineId)
// All at once: one machine that never answers (offline, or too old to know the frame) must not
@@ -400,10 +401,13 @@ export class GroupSyncer {
const selfMachine = this.deps.self().machineId
const blocked = this.deps.store.blocked()
const suspended = this.deps.suspended?.() ?? new Set<string>()
+ // A machine the backend said was deleted (remoteRelay.ts) stays in other members' rosters; pinning it again is what made
+ // the relay dial it for good. It is trusted as before: a deleted machine's identity is spent (gateway/start.ts).
+ const gone = this.deps.peers.gone()
for (const m of result.upserted) {
if (blocked.has(m.pub) || suspended.has(m.pub)) continue
try {
- if (m.kind === 'machine' && m.machineId && m.machineId !== selfMachine) {
+ if (m.kind === 'machine' && m.machineId && m.machineId !== selfMachine && !gone.has(m.machineId)) {
const pin = this.deps.peers.get(m.machineId)
if (!pin || pin.pub !== m.pub) this.deps.peers.pin(m.machineId, m.pub, m.label, m.at)
}
diff --git a/cli/src/lib/e2ee/machinePeers.spec.ts b/cli/src/lib/e2ee/machinePeers.spec.ts
new file mode 100644
index 000000000..ff0335d86
--- /dev/null
+++ b/cli/src/lib/e2ee/machinePeers.spec.ts
@@ -0,0 +1,37 @@
+import { beforeEach, describe, expect, it } from 'vitest'
+import { MachinePeerStore } from './machinePeers.js'
+
+/** A machine the backend said was deleted (remoteRelay.ts): unpinned and remembered, so the trust group does not pin it
+ * again (groupSyncer.ts), until the account's machine list shows it again. */
+describe('a machine marked gone', () => {
+ const a = 'a'.repeat(32)
+ const b = 'b'.repeat(32)
+ beforeEach(() => { const peers = new MachinePeerStore(); peers.restore(peers.gone()); for (const p of peers.list()) peers.unlink(p.machineId) })
+
+ it('loses its pin, and the mark outlives the store that made it', () => {
+ const peers = new MachinePeerStore()
+ peers.pin(a, 'pub-a', 'A'); peers.pin(b, 'pub-b', 'B')
+ peers.markGone(a)
+ expect(peers.get(a)).toBeNull()
+ expect(peers.get(b)?.pub).toBe('pub-b')
+ expect([...new MachinePeerStore().gone()]).toEqual([a])
+ })
+
+ it('is cleared once the machine list shows it again, and only then', () => {
+ const peers = new MachinePeerStore()
+ peers.markGone(a)
+ expect(peers.restore([b])).toBe(false)
+ expect(peers.gone().has(a)).toBe(true)
+ expect(peers.restore([b, a])).toBe(true)
+ expect(peers.gone().has(a)).toBe(false)
+ })
+
+ it('keeps only the most recent 256', () => {
+ const peers = new MachinePeerStore()
+ for (let i = 0; i < 300; i++) peers.markGone(i.toString(16).padStart(32, '0'))
+ const gone = peers.gone()
+ expect(gone.size).toBe(256)
+ expect(gone.has((299).toString(16).padStart(32, '0'))).toBe(true)
+ expect(gone.has((0).toString(16).padStart(32, '0'))).toBe(false)
+ })
+})
diff --git a/cli/src/lib/e2ee/machinePeers.ts b/cli/src/lib/e2ee/machinePeers.ts
index 612354f7a..783ff3406 100644
--- a/cli/src/lib/e2ee/machinePeers.ts
+++ b/cli/src/lib/e2ee/machinePeers.ts
@@ -19,6 +19,13 @@ import { b64d, fingerprint } from './core.js'
const DIR = join(env.ADAPTER_DATA_DIR, 'e2e')
const FILE = join(DIR, 'machinePeers.json')
+// Machines the backend said were deleted (remoteRelay.ts): kept out of this file until the machine list shows them again.
+// Not per account (accountTrust.ts leaves it in place): a deleted machine id is gone for every account.
+const GONE_FILE = join(DIR, 'machinePeers-gone.json')
+const GONE_MAX = 256
+// Marks the file could not take (a full or read-only disk, review of #2101), for every store in this process: they still
+// keep the trust group from pinning the machine again and the relay from dialling it, until the daemon restarts.
+const goneInMemory = new Set<string>()
export interface MachinePeer {
machineId: string
@@ -46,6 +53,11 @@ function writeAll(peers: Map<string, MachinePeer>): void {
writeFileSync(FILE, JSON.stringify([...peers.values()], null, 2), { mode: 0o600 })
}
+function writeGone(gone: Set<string>): void {
+ mkdirSync(DIR, { recursive: true, mode: 0o700 })
+ writeFileSync(GONE_FILE, JSON.stringify([...gone].slice(-GONE_MAX)), { mode: 0o600 })
+}
+
export class MachinePeerStore {
get(machineId: string): MachinePeer | null {
return readAll().get(machineId) ?? null
@@ -64,6 +76,38 @@ export class MachinePeerStore {
return true
}
+ /** The backend said [machineId] was deleted: drop its pin, and remember it so the trust group does not pin it again. */
+ /** Throws when the mark or the unpin cannot be written; the mark holds in memory for this process all the same. */
+ markGone(machineId: string): void {
+ const gone = this.gone()
+ gone.delete(machineId)
+ gone.add(machineId)
+ try { writeGone(gone) } catch (err) { goneInMemory.add(machineId); throw err }
+ this.unlink(machineId)
+ }
+
+ /** Machines marked gone. */
+ gone(): Set<string> {
+ let kept: string[] = []
+ try {
+ const raw = JSON.parse(readFileSync(GONE_FILE, 'utf-8')) as unknown
+ if (Array.isArray(raw)) kept = raw.filter((id): id is string => typeof id === 'string')
+ } catch { /* none yet, or unreadable: the marks in memory still hold */ }
+ return new Set([...kept, ...goneInMemory])
+ }
+
+ /** The account's machine list names [listed]: any of them marked gone is back (re-added, or shared again). True when one was. */
+ restore(listed: Iterable<string>): boolean {
+ const gone = this.gone()
+ let back = false
+ for (const id of listed) {
+ goneInMemory.delete(id)
+ back = gone.delete(id) || back
+ }
+ if (back) writeGone(new Set([...gone].filter((id) => !goneInMemory.has(id))))
+ return back
+ }
+
list(): Array<MachinePeer & { fingerprint: string }> {
return [...readAll().values()]
.map((p) => ({ ...p, fingerprint: fingerprint(b64d(p.pub)) }))
diff --git a/cli/src/lib/relayFrames.ts b/cli/src/lib/relayFrames.ts
index 01782a314..33d5b3d7e 100644
--- a/cli/src/lib/relayFrames.ts
+++ b/cli/src/lib/relayFrames.ts
@@ -56,8 +56,9 @@ export const GATEWAY_REQUEST_TYPES = new Set(['device_e2ee_pair', 'phone_pair',
/** A window's connection to another machine could not be made: why, and the close code to tell it. Here and
* not with the relay pool, so the local socket can tell one apart without loading the pool. */
export class RelayConnectError extends Error {
- /** `retryAfterMs`: how long the backend asked not to be asked again (its MACHINE_OFFLINE refusal of a relay). */
- constructor(message: string, readonly closeCode?: number, readonly retryAfterMs?: number) {
+ /** `retryAfterMs`: how long the backend asked not to be asked again (its MACHINE_OFFLINE refusal of a relay).
+ * `reason`: what the backend added to a refusal (`deleted`: the person's own machine was deleted). */
+ constructor(message: string, readonly closeCode?: number, readonly retryAfterMs?: number, readonly reason?: string) {
super(message)
this.name = 'RelayConnectError'
}
diff --git a/cli/src/lib/remoteRelay.ts b/cli/src/lib/remoteRelay.ts
index 3488db87d..f83d7960c 100644
--- a/cli/src/lib/remoteRelay.ts
+++ b/cli/src/lib/remoteRelay.ts
@@ -491,6 +491,19 @@ export class RemoteRelayPool {
* An unlinked machine is not held: nothing was dialled, and a link made a moment later must work at once. */
private backOff(machineId: string, error: unknown): void {
if (error instanceof RelayConnectError && error.message === 'NO_PEER_LINK') return
+ // The person's own machine was deleted (roadmap Phase 1.1: one daemon dialled a machine deleted weeks before every few
+ // seconds). Its pin goes, and stays gone until the machine list shows it again; every select is then NO_PEER_LINK,
+ // answered here without a dial. Only this reason: a share check that failed for a moment says NOT_YOUR_MACHINE too.
+ if (error instanceof RelayConnectError && error.message === 'NOT_YOUR_MACHINE' && error.reason === 'deleted') {
+ try {
+ this.peers.markGone(machineId)
+ this.refusals.delete(machineId)
+ return
+ } catch (err) {
+ // Held in memory (machinePeers.ts) and by the backoff below; never thrown out of here, where nothing would catch it.
+ console.warn(`[relay] could not keep machine=${sid(machineId)} as deleted: ${err instanceof Error ? err.message : String(err)}`)
+ }
+ }
const count = (this.refusals.get(machineId)?.count ?? 0) + 1
const backoff = Math.min(REFUSED_BACKOFF_MAX_MS, 1_000 * 2 ** (count - 1))
const asked = error instanceof RelayConnectError && typeof error.retryAfterMs === 'number' && Number.isFinite(error.retryAfterMs)
@@ -500,9 +513,11 @@ export class RemoteRelayPool {
private async dial(machineId: string, autonomousEnv: string, selectFrame: Frame, forceRefresh: boolean): Promise<Entry> {
const peer = this.peers.get(machineId)
- if (!peer) throw new RelayConnectError('NO_PEER_LINK')
+ // A machine marked gone is not dialled even when its pin could not be removed (machinePeers.ts markGone).
+ if (!peer || this.peers.gone().has(machineId)) throw new RelayConnectError('NO_PEER_LINK')
const token = await this.auth.accessToken({ force: forceRefresh })
- const url = `${this.backendWsBase}/api/web-ws?autonomousEnv=${encodeURIComponent(autonomousEnv)}&${declaredClientQuery('relay')}`
+ // `auth=http`: this relay reads a bad token refused with HTTP 401 before the upgrade (below), as a 4401.
+ const url = `${this.backendWsBase}/api/web-ws?autonomousEnv=${encodeURIComponent(autonomousEnv)}&${declaredClientQuery('relay')}&auth=http`
const ws = new WebSocket(url, [token])
// An 'error' with no listener throws out of the emitter and takes the daemon down. The handshake's
// once('error') below is consumed by its first emit, and nothing listens after the handshake at all;
@@ -599,10 +614,10 @@ export class RemoteRelayPool {
if (!settled) {
settled = true; clearTimeout(timeout)
const code = typeof payload.error === 'string' ? payload.error : 'machine_select_error'
- const retryAfterMs = (payload as { retryAfterMs?: unknown }).retryAfterMs
+ const { retryAfterMs, reason } = payload as { retryAfterMs?: unknown; reason?: unknown }
// Offline closes the window's socket as the node_status answer below does (1013, try again later).
reject(new RelayConnectError(code, code === 'MACHINE_OFFLINE' ? 1013 : undefined,
- typeof retryAfterMs === 'number' ? retryAfterMs : undefined))
+ typeof retryAfterMs === 'number' ? retryAfterMs : undefined, typeof reason === 'string' ? reason : undefined))
}
return
}
@@ -743,6 +758,15 @@ export class RemoteRelayPool {
ws.once('error', (err) => {
if (!settled) { settled = true; clearTimeout(timeout); reject(err instanceof Error ? err : new Error(String(err))) }
})
+ // The backend refused before the upgrade (WEB_WS_AUTH_HTTP_401): a 401 is the 4401 it used to close with, which
+ // connect() answers with one refresh. With this listener `ws` leaves the request open, so it is ended here.
+ ws.once('unexpected-response', (_req, res) => {
+ if (!settled) {
+ settled = true; clearTimeout(timeout)
+ reject(new RelayConnectError(`relay refused: HTTP ${res.statusCode ?? 0}`, res.statusCode === 401 ? 4401 : undefined))
+ }
+ ws.terminate()
+ })
})
try {
await handshake
diff --git a/cli/src/lib/remoteRelayRefusal.spec.ts b/cli/src/lib/remoteRelayRefusal.spec.ts
index 9a66eb66b..c87ade694 100644
--- a/cli/src/lib/remoteRelayRefusal.spec.ts
+++ b/cli/src/lib/remoteRelayRefusal.spec.ts
@@ -1,4 +1,7 @@
-import { afterEach, describe, expect, it } from 'vitest'
+import { afterEach, describe, expect, it, vi } from 'vitest'
+import { mkdirSync, rmSync } from 'node:fs'
+import { join } from 'node:path'
+import { env } from '../config/env.js'
import { WebSocketServer } from 'ws'
import type { AddressInfo } from 'node:net'
import { RemoteRelayPool } from './remoteRelay.js'
@@ -39,12 +42,14 @@ async function refusingBackend(refusal: Record<string, unknown>) {
newIdentity(), peers, { p2p: false, now: () => now },
)
cleanups.push(async () => {
+ peers.unlink(machineId); peers.restore([machineId])
pool.close()
for (const ws of wss.clients) ws.terminate()
await new Promise<void>(resolve => wss.close(() => resolve()))
})
return {
pool,
+ peers,
dials: () => dials,
advance(ms: number) { now += ms },
window: () => pool.acquire(machineId, 'prod', select, sink, () => {}),
@@ -97,4 +102,110 @@ describe('a machine the backend refuses', () => {
}
expect(expected).toEqual([1, 2, 3, 4, 5, 6, 7])
})
+
+ // Roadmap Phase 1.1: one person's daemon selected a machine deleted on 2026-09-03 every few seconds for weeks. The backend
+ // now says when the person's own machine was deleted; that pin is dropped and remembered, so nothing dials it again.
+ it('drops the pin of a machine the backend says was deleted, and never dials it again', async () => {
+ const b = await refusingBackend({ error: 'NOT_YOUR_MACHINE', reason: 'deleted' })
+ await expect(b.window()).rejects.toMatchObject({ message: 'NOT_YOUR_MACHINE' })
+ expect(b.peers.get(machineId)).toBeNull()
+ expect(b.peers.gone().has(machineId)).toBe(true)
+ b.advance(10 * 60_000)
+ await expect(b.window()).rejects.toMatchObject({ message: 'NO_PEER_LINK' })
+ await expect(b.redial()).rejects.toMatchObject({ message: 'NO_PEER_LINK' })
+ await expect(b.background()).rejects.toMatchObject({ message: 'NO_PEER_LINK' })
+ expect(b.dials()).toBe(1)
+ })
+
+ // A share check that failed for a moment, an unshared machine and an older backend all say only NOT_YOUR_MACHINE.
+ it('keeps the pin of a machine refused without the deleted reason', async () => {
+ const b = await refusingBackend({ error: 'NOT_YOUR_MACHINE' })
+ await expect(b.window()).rejects.toMatchObject({ message: 'NOT_YOUR_MACHINE' })
+ expect(b.peers.get(machineId)).not.toBeNull()
+ expect(b.peers.gone().has(machineId)).toBe(false)
+ })
+})
+
+// Roadmap Phase 1.3: with WEB_WS_AUTH_HTTP_401 on, the backend refuses a bad token with HTTP 401 before the upgrade, to the
+// sockets that say they read it. The relay says so, and reads that 401 as it reads a 4401: one refresh, then the backoff.
+describe('a bad token refused before the upgrade', () => {
+ it('is refreshed once and dialled again', async () => {
+ const urls: string[] = []
+ const good = 'fresh-token'
+ const wss = new WebSocketServer({
+ port: 0, host: '127.0.0.1',
+ verifyClient: (info, done) => {
+ urls.push(info.req.url ?? '')
+ done(info.req.headers['sec-websocket-protocol'] === good, 401)
+ },
+ })
+ await new Promise<void>(resolve => wss.once('listening', resolve))
+ wss.on('connection', ws => {
+ ws.send(JSON.stringify({ type: 'connected', payload: { userId: 'user-1' } }))
+ ws.on('message', () => ws.send(JSON.stringify({ type: 'machine_select_error', payload: { machineId, error: 'MACHINE_OFFLINE' } })))
+ })
+ const forced: boolean[] = []
+ let token = 'stale-token'
+ const peers = new MachinePeerStore(); peers.pin(machineId, b64e(newIdentity().pub), 'test remote')
+ const pool = new RemoteRelayPool(
+ { accessToken: async (o?: { force?: boolean }) => { forced.push(o?.force === true); return o?.force ? token : 'stale-token' } } as never,
+ `ws://127.0.0.1:${(wss.address() as AddressInfo).port}`, newIdentity(), peers, { p2p: false },
+ )
+ cleanups.push(async () => {
+ peers.unlink(machineId); pool.close()
+ await new Promise<void>(resolve => wss.close(() => resolve()))
+ })
+
+ token = good
+ await expect(pool.acquire(machineId, 'prod', select, sink, () => {})).rejects.toMatchObject({ message: 'MACHINE_OFFLINE' })
+ expect(forced).toEqual([false, true])
+ expect(urls.every(url => new URL(url, 'http://x').searchParams.get('auth') === 'http')).toBe(true)
+ })
+
+ it('stops after the refreshed token is refused too', async () => {
+ const wss = new WebSocketServer({ port: 0, host: '127.0.0.1', verifyClient: (_info, done) => done(false, 401) })
+ await new Promise<void>(resolve => wss.once('listening', resolve))
+ let asked = 0
+ const peers = new MachinePeerStore(); peers.pin(machineId, b64e(newIdentity().pub), 'test remote')
+ const pool = new RemoteRelayPool(
+ { accessToken: async () => { asked++; return 'refused-token' } } as never,
+ `ws://127.0.0.1:${(wss.address() as AddressInfo).port}`, newIdentity(), peers, { p2p: false },
+ )
+ cleanups.push(async () => {
+ peers.unlink(machineId); pool.close()
+ await new Promise<void>(resolve => wss.close(() => resolve()))
+ })
+ await expect(pool.acquire(machineId, 'prod', select, sink, () => {})).rejects.toMatchObject({ closeCode: 4401 })
+ expect(asked).toBe(2)
+ // Held by the backoff: the next select is answered from the refusal, without a dial or a refresh.
+ await expect(pool.acquire(machineId, 'prod', select, sink, () => {})).rejects.toMatchObject({ closeCode: 4401 })
+ expect(asked).toBe(2)
+ })
+})
+
+// Review of #2101: the deletion mark could not be written (a full or read-only disk), and the error escaped the refusal's
+// backoff as an unhandled rejection, which can take the relay down, with the pin gone and no mark kept. The machine is held
+// as gone in memory instead, said in the log, and nothing dials it again in this process.
+describe('a deleted machine whose mark cannot be written', () => {
+ it('is held as gone in memory, without an unhandled rejection', async () => {
+ const markFile = join(env.ADAPTER_DATA_DIR, 'e2e', 'machinePeers-gone.json')
+ rmSync(markFile, { recursive: true, force: true })
+ mkdirSync(markFile, { recursive: true }) // a folder where the file goes: every write of it fails
+ const unhandled: unknown[] = []
+ const onUnhandled = (reason: unknown) => { unhandled.push(reason) }
+ process.on('unhandledRejection', onUnhandled)
+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {})
+ cleanups.push(async () => { process.off('unhandledRejection', onUnhandled); warn.mockRestore(); rmSync(markFile, { recursive: true, force: true }) })
+
+ const b = await refusingBackend({ error: 'NOT_YOUR_MACHINE', reason: 'deleted' })
+ await expect(b.window()).rejects.toMatchObject({ message: 'NOT_YOUR_MACHINE' })
+ await new Promise(resolve => setImmediate(resolve))
+ expect(unhandled).toEqual([])
+ expect(warn).toHaveBeenCalledWith(expect.stringContaining('could not keep'))
+ expect(b.peers.gone().has(machineId)).toBe(true)
+ b.advance(10 * 60_000)
+ await expect(b.window()).rejects.toMatchObject({ message: 'NO_PEER_LINK' })
+ await expect(b.background()).rejects.toMatchObject({ message: 'NO_PEER_LINK' })
+ expect(b.dials()).toBe(1)
+ })
})
diff --git a/cli/src/lib/remoteRelayRestart.spec.ts b/cli/src/lib/remoteRelayRestart.spec.ts
index da47ec451..f54560624 100644
--- a/cli/src/lib/remoteRelayRestart.spec.ts
+++ b/cli/src/lib/remoteRelayRestart.spec.ts
@@ -70,10 +70,10 @@ async function fixture(opts: { offline?: boolean; lingerMs?: number } = {}) {
describe('remote daemon restart recovery', () => {
// WebSocket roadmap Phase 0: 2,304 of 2,308 web-ws opens in 15 minutes declared nothing (prod, 2026-10-28).
- it('dials as a daemon relay, saying which CLI', async () => {
+ it('dials as a daemon relay, saying which CLI and that it reads an HTTP 401', async () => {
const f = await fixture({ offline: true })
await f.pool.acquire(machineId, 'prod', f.select, { sendFrame: () => true, sendBinary: () => true }, () => {}).catch(() => {})
- expect(f.urls).toEqual([`/api/web-ws?autonomousEnv=prod&client=relay&v=${encodeURIComponent(VERSION)}`])
+ expect(f.urls).toEqual([`/api/web-ws?autonomousEnv=prod&client=relay&v=${encodeURIComponent(VERSION)}&auth=http`])
})
it('reports an offline peer immediately during the handshake and keeps its trust', async () => {
diff --git a/desktop/lib/state/app_state.dart b/desktop/lib/state/app_state.dart
index 11eb87ae3..1c5dcb3d1 100644
--- a/desktop/lib/state/app_state.dart
+++ b/desktop/lib/state/app_state.dart
@@ -9641,6 +9641,9 @@ class AppNotifier extends ChangeNotifier {
_stopOfflineRetry(entry.key);
_stopLinkRetry(entry.key);
_stopAgentSyncTimer(entry.key);
+ // The socket goes too, as on mobile: left in the pool it kept selecting a machine the
+ // backend refuses (roadmap Phase 1.1, 2026-10-28), and nothing here looked at it again.
+ unawaited(_pool?.closeMachine(entry.key));
}
}
machineStates.removeWhere((id, _) => !visible.contains(id));
@@ -12085,6 +12088,11 @@ class AppNotifier extends ChangeNotifier {
if (machineStates[machineId]?.machine.isShared == true) {
throw StateError('This harness is shared with view-only access.');
}
+ // A machine id the loaded list does not name (a deleted machine a pane or a Swarm agent still
+ // names) gets no socket: it would only select a machine the backend refuses (roadmap Phase 1.1).
+ if (_machineInventoryLoaded && !machineStates.containsKey(machineId)) {
+ throw StateError('This machine is no longer on your account.');
+ }
final testConnection = connectionForTest;
if (testConnection != null) return testConnection(machineId);
// The local-manual dev fixture exercises a locally-run backend+node stack directly (no real
@@ -14201,6 +14209,11 @@ class AppNotifier extends ChangeNotifier {
},
timeout: const Duration(seconds: 4),
);
+ } on WsRequestFailure catch (failure) {
+ // The connection throws a refusal; it is still an answer. Swallowed, a
+ // NOT_INDEXED row read as no answer and was re-read every 5 seconds:
+ // 2,139 failed reads in 3 hours from one computer (2026-10-29).
+ return {...failure.payload, 'error': failure.code};
} catch (_) {
return null;
}
diff --git a/desktop/test/machine_list_sockets_test.dart b/desktop/test/machine_list_sockets_test.dart
new file mode 100644
index 000000000..6592aa38a
--- /dev/null
+++ b/desktop/test/machine_list_sockets_test.dart
@@ -0,0 +1,143 @@
+import 'package:flutter_test/flutter_test.dart';
+import 'package:harness/api/api_client.dart';
+import 'package:harness/auth/auth_session.dart';
+import 'package:harness/core/config.dart';
+import 'package:harness/core/models.dart';
+import 'package:harness/state/app_state.dart';
+import 'package:harness/ws/local_cli_discovery.dart';
+import 'package:harness/ws/ws_conn.dart';
+
+/// Roadmap Phase 1.1 (2026-10-28): a machine that left the account kept its
+/// socket, and anything that asked for a machine id the list no longer names
+/// built one, each selecting a machine the backend refuses.
+
+class _Api extends ApiClient {
+ _Api() : super(config: AppConfig.dev, session: AuthSession());
+ List<Machine> listed = const [];
+
+ @override
+ Future<Map<String, dynamic>?> me() async => null;
+ @override
+ Future<Map<String, dynamic>?> desk() async => null;
+ @override
+ Future<Map<String, dynamic>?> daemonDevices() async => null;
+ @override
+ Future<List<Machine>> machines() async => listed;
+}
+
+class _Discovery extends LocalCliDiscovery {
+ _Discovery() : super(config: AppConfig.dev);
+ @override
+ Future<String?> computerId() async => 'aabbccddeeff0011';
+ @override
+ Future<LocalCliEndpoint?> discover({String? expectedComputerId}) async =>
+ LocalCliEndpoint(
+ computerId: 'aabbccddeeff0011',
+ wsUri: Uri.parse('ws://127.0.0.1:18473/ws'),
+ protocolVersion: localWsProtocolVersion,
+ terminalProtocolVersion: 1,
+ machineId: 'local-machine',
+ backendOnline: true,
+ );
+}
+
+class _Connection extends WsConn {
+ _Connection(String machineId)
+ : super(
+ wsBaseUrl: 'ws://fixture.invalid',
+ autonomousEnv: 'test',
+ machineId: machineId,
+ accessTokenProvider: (_, _) async => '',
+ onAuthFailure: (_) {},
+ onEvent: (_) {},
+ onStatus: (_) {},
+ );
+ bool closed = false;
+ @override
+ Future<void> close() async => closed = true;
+ @override
+ void wake(String reason) {}
+ @override
+ Future<void> waitUntilReady({required Duration timeout}) async {}
+ @override
+ Future<Map<String, dynamic>> request(
+ String type, {
+ Map<String, dynamic> payload = const {},
+ Duration timeout = const Duration(seconds: 20),
+ }) async => const {'agents': []};
+}
+
+class _App extends AppNotifier {
+ _App(_Api api, this.built)
+ : super(
+ config: AppConfig.dev,
+ authSession: AuthSession(),
+ configStore: null,
+ localCliDiscovery: _Discovery(),
+ connectionForTest: (id) {
+ built.add(id);
+ return _Connection(id);
+ },
+ ) {
+ this.api = api;
+ }
+ final List<String> built;
+ @override
+ Future<void> ensureCliDaemonReady() async {}
+}
+
+const _studio = Machine(
+ machineId: 'studio',
+ authMode: MachineAuthMode.remote,
+ name: 'Studio',
+);
+const _laptop = Machine(
+ machineId: 'laptop',
+ authMode: MachineAuthMode.remote,
+ name: 'Laptop',
+);
+
+void main() {
+ TestWidgetsFlutterBinding.ensureInitialized();
+
+ late _App app;
+ late _Api api;
+ final built = <String>[];
+ setUp(() {
+ built.clear();
+ api = _Api()..listed = const [_studio, _laptop];
+ app = _App(api, built)..status = AppStatus.authenticated;
+ });
+ tearDown(() => app.dispose());
+
+ test('a machine that leaves the list has its socket closed', () async {
+ await app.refreshMachines();
+ final socket = _Connection('laptop');
+ app.adoptPoolConnectionForTest('laptop', socket);
+
+ api.listed = const [_studio];
+ await app.refreshMachines();
+
+ expect(app.stateOf('laptop'), isNull);
+ expect(socket.closed, isTrue);
+ });
+
+ test(
+ 'no socket is built for a machine id the loaded list does not name',
+ () async {
+ api.listed = const [_studio];
+ await app.refreshMachines();
+ built.clear();
+
+ await expectLater(
+ app.superTabMachineRequest('laptop', {'action': 'renewNative'}),
+ throwsA(anything),
+ );
+ expect(built, isNot(contains('laptop')));
+
+ // A listed machine still gets one.
+ await app.superTabMachineRequest('studio', {'action': 'renewNative'});
+ expect(built, contains('studio'));
+ },
+ );
+}
diff --git a/desktop/test/session_search_rendering_test.dart b/desktop/test/session_search_rendering_test.dart
index c70ef59c2..eba2a912c 100644
--- a/desktop/test/session_search_rendering_test.dart
+++ b/desktop/test/session_search_rendering_test.dart
@@ -11,6 +11,7 @@ import 'package:harness/state/session_content_search.dart';
import 'package:harness/state/swarm_catalog.dart';
import 'package:harness/state/swarm_navigation.dart';
import 'package:harness/widgets/swarm_switcher.dart';
+import 'package:harness/ws/ws_conn.dart' show WsRequestFailure;
import 'keymap_host_test.dart' show MemoryKeymap, key;
import 'session_content_search_test.dart' show SearchConnection;
@@ -527,7 +528,19 @@ class TailConnection extends SearchConnection {
Map<String, dynamic> payload = const {},
Duration timeout = const Duration(seconds: 20),
}) async {
- if (type == 'session_tail') return tail(payload);
+ if (type == 'session_tail') {
+ final reply = tail(payload);
+ // A refusal arrives as WsConn's: thrown, its fields on the failure. A
+ // fake that answered it as a reply hid a reader that never saw one.
+ if (reply['error'] != null) {
+ throw WsRequestFailure(
+ responseType: 'session_tail_result',
+ code: '${reply['error']}',
+ payload: Map.unmodifiable(reply),
+ );
+ }
+ return reply;
+ }
if (type == 'agent_create' && create != null) {
creates.add(payload);
return create!(payload);
diff --git a/docs/plans/2026-10-28-websocket-optimization-roadmap.md b/docs/plans/2026-10-28-websocket-optimization-roadmap.md
new file mode 100644
index 000000000..49b908328
--- /dev/null
+++ b/docs/plans/2026-10-28-websocket-optimization-roadmap.md
@@ -0,0 +1,227 @@
+# WebSocket optimization roadmap
+
+Date: 2026-10-28. Code read at `origin/main` (b766b4166). Measurements from prod backend logs
+(45 min, 3 pods, v1.3.58_backend) and Atlas.
+
+**Scope decided (Ken, 2026-10-28): phases 0–3 only.** Phases 4–5 (the multiplexed account socket) are
+deferred because of their risk; they stay here for direction and are revisited after 0–3 ship and are
+measured.
+
+## Why
+
+Clients and daemons open many short-lived sockets to the backend, and the Swarm keeps the backend busy
+with HTTP polls that have no push. The question was whether to fold every data lane into one
+WebSocket. The answer is yes, eventually, but most of the cost today comes from churn bugs and
+polling, which are cheaper to fix first. Phases are ordered by priority: each one is worth doing even
+if the later ones never ship.
+
+## What we measured (prod, 45 min)
+
+- `/api/web-ws`: 8,760 connects from 74 users, about 195 per minute. 81% of sockets close without ever
+ sending `machine_select`. Bursts every 7–12 s, mostly 1–3 sockets, up to 8.
+- One user: 44 connects/min. Every ~7 s it selects 4 machines plus one deleted on 2026-09-03, which is
+ rejected with `NOT_YOUR_MACHINE`, then every socket closes.
+- `/api/adapter-ws`: 227 reconnects (5/min); 5 machines reconnected more than 5 times.
+- Hub heartbeat timeouts: 152 tracked, 148 web.
+- Backend CPU after v1.3.58: about 1.5 cores, steady. Most of the remaining busy time is polling:
+ host `/changes` 14.6%, Swarm `/changes` 10.9%, deliveries about 19%, messages 7.4%.
+- What the logs cannot tell today: which client opened a socket (daemon relay, desktop, web, mobile,
+ group sync), close codes, or how long a socket lived.
+
+## What each connect costs today
+
+- **Every web-ws connect, before any select:**
+ - Sign-in check: in-process cache, then Redis, then SSO; plus 2 Mongo `user` reads.
+ - Mongo `machine.findMany`, a presence MGET, and a `machine_nodes` read for each offline machine.
+ - Mongo `deviceBinding.findMany`, then a device re-read every 30 s.
+ - Six per-user Redis subscriptions, plus one per owned machine.
+ - References: `backend/src/lib/webWs.ts:199-324`, `ssoAuth.ts:283-317`.
+- **Every `machine_select`:**
+ - Machine and share lookups, plus two presence reads.
+ - A `machine_nodes` read even when the machine is online (`webWs.ts:341-344`).
+ - Hub attach: a Redis SUBSCRIBE, HSET/HEXPIRE and a `__clients_dirty` publish (`hub.ts:122-165`).
+- **A failed sign-in** still completes the upgrade and then closes with 4401/4403/1013
+ (`webWs.ts:133-146`). Each failure therefore counts as a connect that lives under 1 s.
+
+## Sockets in play
+
+- **Daemon** (`cli/src/lib/remoteRelay.ts`, `gateway/*`):
+ - 1 `adapter-ws`.
+ - 1 `device-ws`, only while a device is plugged in. It already multiplexes machines.
+ - 1 web-ws per remote machine for the window, plus up to 4 background web-ws per machine, each a
+ separate E2EE session.
+ - 1 `observer-ws` per watched share. These are not pooled, and each acquire re-reads
+ `/api/harness-shares`.
+ - 1 `agent-messages-ws` per shared tab.
+- **Desktop native:** one local-ws per machine to its daemon. The daemon relays every remote machine.
+- **Desktop web build and mobile:** one web-ws per machine, dialled directly
+ (`desktop/lib/ws/ws_conn.dart:388`, `mobile/lib/ws/ws_conn.dart:363`). Group sync, password link and
+ code link open one-shot web-ws sockets.
+- **The backend binds one machine per web-ws** (`webWs.ts:405-412`). `device-ws` is the existing
+ exception: N hub clients on one socket, routed by a `machineId` tag (`deviceWs.ts:400-431`).
+
+## Phase 0 — Attribute every socket (highest priority, small)
+
+Without this, phases 1–3 cannot be verified.
+
+1. Every client declares itself on connect.
+ - Daemon relay: `client=relay&v=<cli>`.
+ - Group sync / password link / code link: `client=group-sync|link`.
+ - Desktop native through the daemon: as the relay.
+ - Web and mobile already send `client=web|mobile`.
+ - The backend logs `surface`, `version`, `pod` and `connId` on connect.
+2. On disconnect, for web-ws, adapter-ws, observer-ws and agent-messages-ws, log close code, reason,
+ lifetime, whether a select happened, and the select outcome (accepted / error code).
+3. Counters per endpoint and pod: open sockets, connects/min, and sockets that lived under 2 s.
+4. Pre-upgrade refusals are counted separately from real connects.
+
+Done when ≥95% of web-ws connects can be attributed to a client kind and version, and the 81%
+"never selected" sockets are explained.
+
+## Phase 1 — Stop the churn (high priority)
+
+Bugs that open sockets for nothing. Each fix is small and independent.
+
+1. **Deleted and stale machines stay selected forever.**
+ - Daemon: on `NOT_YOUR_MACHINE` for a deleted machine, drop its pin
+ (`remoteRelay.ts:317, 560-566`), and stop `groupSyncer.applyAll` from re-pinning it
+ (`groupSyncer.ts:394`).
+ - Desktop: close the pooled socket when a machine leaves the list (`app_state.dart:9595-9605`;
+ mobile already does, :2808). Do not build sockets for unknown machine IDs (`workspace.dart:286,333`,
+ `machines.dart:660`).
+ - Mobile: the 6 s select watchdog must stop on `NOT_YOUR_MACHINE` instead of redialling
+ (`mobile/lib/ws/ws_conn.dart:667-701`, `app_state.dart:10336`).
+2. **Failed selects redial immediately.**
+ - Give the daemon's window pool a dial cooldown and honour the backend's `retryAfterMs`
+ (`remoteRelay.ts:50-54, 298, 563`).
+ - Map `MACHINE_OFFLINE` to the same local close code (1013) on both paths
+ (`localWsServer.ts:499-500`).
+ - Desktop must not redial forever on a 1011 from a machine that is gone.
+3. **Expired sign-in loops.** Clients back off on 4401 after one refresh attempt. Consider refusing a
+ bad token with HTTP 401 before the upgrade, once clients handle it.
+4. **Make an unselected socket cheap.** Start the per-user watchers only when the client asks
+ (`machines_watch` / `devices_watch`):
+ - machine list
+ - presence reads
+ - device bindings and the 30 s device re-read
+ - per-user subscriptions
+
+ Daemon relays and one-shot sockets never need them. Also drop the unconditional `machine_nodes`
+ read in the select ack.
+5. **Pool share watching.** Pool and linger `observer-ws` in the daemon like web-ws, and use the cached
+ shares list instead of `GET /api/harness-shares` on every acquire (`gateway/start.ts:215`,
+ `sharing/relay.ts:36`).
+6. **Background shelf overflow.** Raise or rethink `ISOLATED_IDLE_MAX`: a 5th concurrent background
+ reader closes its socket on detach (`remoteRelay.ts:351`).
+
+Done when:
+- web-ws connects drop from ~195/min to under 40/min;
+- the median web-ws lifetime is over 1 minute;
+- no selects of deleted machines;
+- adapter reconnects are explained by close codes.
+
+## Phase 2 — Push instead of poll for the Swarm (high priority, largest CPU win)
+
+1. **Host feed.** Today the host feed polls about every second (`cli/src/superTabs/host.ts:324`,
+ `hostFeed.ts:54-60`), even with no pools.
+ - Push `swarm_host_changed{tabId, lastChangedAt}` down `adapter-ws`, following
+ `backend/src/lib/swarmDeliveryPushes.ts`, so the host reads only when told.
+ - Keep a 60 s safety net.
+ - Do not run the loop at all on a machine with no Swarm pools.
+2. **Client Swarm reads.** These are:
+ - Swarm state: 3 s on desktop, 15 s on web.
+ - Discovery: 15 s.
+ - Invites: 15 s.
+ - Native pane renew: 20 s.
+
+ Push `swarm_changed{tabId, lastChangedAt, parts}` over the socket the client already has. Forward
+ the existing `swarm-stamp` Redis broadcast (`bus.ts:325-344`) instead of adding a new source.
+3. **Cheap "nothing changed" reads.** `/changes`, deliveries and messages answer an unchanged cursor
+ before loading any state (304 or empty).
+4. **Retire timers that already have a push**, keeping one long safety net each:
+ - CLI: Agent Messages 15 s, machine list 60 s.
+ - Desktop: experimental settings 30 s.
+ - Offline machine with a pending agent: full `/api/machines` every 5 s.
+5. **Deliver account pushes once.** A daemon with N relay sockets hears each account push N+1 times
+ (`webAccountPushes.ts:15`). Send account pushes only on adapter-ws to daemons, and only once per
+ app on web/mobile.
+
+Done when:
+- polling routes are under 20% of backend busy time (from ~60%);
+- host `/changes` requests drop at least 10×;
+- Mongo query/s and CPU fall with the same user activity.
+
+## Phase 3 — Fewer sockets per daemon within today's protocol (medium priority)
+
+1. **Background reads share one session per machine.** Demultiplex by request ID instead of keeping up
+ to 4 isolated E2EE sessions per machine (`remoteRelay.ts:330-356`). The isolation exists only so
+ pollers do not see each other's frames, and request IDs give the same guarantee.
+2. **Visibility gating everywhere.**
+ - Harness Monitor (3–15 s), the Swarm machines panel (15 s), usage (60 s) and agent sync (60 s)
+ stop when the window or tab is hidden or closed.
+ - On return, read once, then go live again.
+3. **Observer fan-out.** Each observer subscribes to the machine's whole `up:{machineId}` stream and
+ filters by `targetConnId` (`observerWs.ts:84-92`). Publish to a per-connection channel instead, and
+ count observers in `__clients`.
+
+Done when an idle daemon with K remote machines keeps at most K + 1 upstream sockets, and a hidden
+window makes no periodic backend requests.
+
+## Phase 4 — One multiplexed account socket (lower priority, largest change)
+
+One socket per client for every data lane.
+
+- **Where it runs:** the daemon first, because it carries Desktop native. Then web and mobile.
+- **Base it on `device-ws`,** which already runs N machines on one socket with one E2EE session per
+ machine (`deviceWs.ts:400-431`, `cli/src/gateway/lane.ts`).
+- **Lanes:**
+ - control and push (account events, Swarm changes)
+ - one lane per machine session (`machineId` plus a session ID, so background readers can still be
+ separate when needed)
+ - one lane per share (observer)
+ - one lane per shared tab (Agent Messages)
+- **E2EE does not change.** Each lane keeps its own end-to-end session keyed by lane connId; the
+ backend still only forwards sealed frames. `groupSeen` stays per machine and is never shared by two
+ live sessions.
+- **What must change on the backend:**
+ - **Lane errors, not socket closes.** Today 1009 (frame too big), 1013 (outbound queue over 8 MiB),
+ 4403 (sharing ended) and observer 1012 each kill the whole socket. They become `lane_closed{lane,
+ code}`.
+ - **No head-of-line blocking.** Today one ordered inbox means a slow select or wake blocks every
+ frame (`webWs.ts:570`). Each lane gets its own inbox.
+ - **Flow control and priority:** control > interactive terminal > bulk history. Output that a slow
+ lane cannot take is dropped or paused per lane.
+ - **Chunking** for frames over the per-socket limit (adapter frames reach 16 MiB, history pages
+ 5.4 MiB).
+ - **Every frame carries its lane ID,** including binary terminal frames.
+ - **Resume on reconnect:** the client re-opens its lanes in one frame, and the backend re-attaches
+ them without a select per machine.
+- **Terminals already on p2p stay on p2p.** Only signalling uses the control lane.
+- **Rollout:** dual stack behind a version check. Keep web-ws, observer-ws and agent-messages-ws until
+ the old clients age out (see Phase 0 versions).
+
+Done when each running app or daemon holds one account socket (plus adapter-ws for its own machine),
+and connects/min tracks app starts and network changes, not activity.
+
+## Phase 5 — Fold the machine side in (lowest priority)
+
+Consider carrying `adapter-ws` and `device-ws` as lanes of the same account socket, so a daemon holds
+exactly one backend socket. This is a small win after Phase 4. The adapter's billing, ownership
+claims, supersede (4000) and team-machine (4009) rules need their own lane-level equivalents.
+
+## Not in scope
+
+- Moving terminal bytes off p2p.
+- Changing the E2EE design.
+- The manager socket (`/api/manager-ws`, server to server).
+
+## Order of work and rough size
+
+| Phase | Priority | Size | Risk |
+|---|---|---|---|
+| 0 Attribute | Highest | 1–2 days | None |
+| 1 Churn bugs | High | 3–5 days, independent PRs | Low |
+| 2 Swarm push | High | 1–2 weeks | Medium: missed pushes need the safety net |
+| 3 Fewer sockets | Medium | 1 week | Low–medium |
+| 4 Account socket | Lower | 3–5 weeks | High: protocol, both sides, rollout |
+| 5 Machine side | Lowest | 1–2 weeks after 4 | Medium |