The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.
TypeScript 3d-modelingagent-frameworkagent-harnessai-agentscadclaude-codecodexcoding-agentsdeveloper-toolsesp32game-developmentmcp-servermulti-agentopen-hardwareopencodeopenharnesspcbroboticsterminaltmux
perf: measure real local and remote terminal latency
dee485e02007180
diff --git a/cli/scripts/benchmark-terminal-latency.md b/cli/scripts/benchmark-terminal-latency.mdnew file mode 100644index 000000000..20ba5dc2a--- /dev/null+++ b/cli/scripts/benchmark-terminal-latency.md@@ -0,0 +1,63 @@+# Real terminal latency++Run from `cli/` against a running, signed-in daemon. Remote machines must already+be linked and support the `terminal` engine. The script creates one named+disposable shell, runs a Python 3 probe, and deletes only that shell in `finally`.+It never launches a model or types into an existing agent.++```sh+npx tsx scripts/benchmark-terminal-latency.mts \+ --machine MACHINE_ID --label local --local \+ --samples 200 --control-samples 30 --revision SOURCE_COMMIT \+ --output /private/tmp/terminal-local-1.json+```++Omit `--local` for remote targets. Use a fresh output filename for every run.+Run machines and repetitions sequentially after builds and tests finish. Keep+all completed runs, including slow runs and failed attempts. Ten-sample runs+are calibration only. The source revision identifies the benchmark checkout;+the recorded local daemon version identifies the installed daemon actually+serving traffic. Remote daemon versions are not inferred from this checkout.++The primary boundary is **binary input send to matching PTY response received**.+Every byte crosses the real terminal input path, Python stdin/stdout, tmux, and+the local or remote transport. Sequence-specific responses check that input+arrives. This excludes OS keyboard delivery, Flutter parsing/rendering, display+presentation, and model response time. A raw-mode Python echo is reproducible;+it is not a shell editor or a model workload.++Each workload has ten warmups, followed by the requested number of observations:++- Idle terminal, one outstanding input byte at a time.+- The same terminal repainting at a requested 20 Hz, about 86 KB/s before+ compression. Actual received bytes and duration are recorded. A deterministic+ 8–42 ms think time precedes each input and is outside its measured interval.++Control requests run in a separate phase, alternating streamless machine+`terminal_capabilities` requests and requests carrying the terminal's stream ID.+These are application RPCs, **not network pings**. Requests and replies can take+different routes; the terminal's reported link mode alone does not prove that+both directions of a control RPC use that route. Do not subtract independent+percentiles to invent a server-processing or network-only number.++Creation, attach, probe readiness, and reconnect are retained as individual+observations. The reconnect closes this script's local client socket, opens a+new one, and verifies the next sequence from the same running probe. It reuses+the daemon's existing remote route: it does not simulate a network outage,+daemon restart, or cold ICE negotiation. Machine-selection acknowledgement is+local; `readyForRequests` also waits for the first remote capability response.++Results include p50/p95/p99/max using nearest-rank percentiles, every warmup and+measured observation, reported terminal link modes, and confirmed cleanup.+Results default to private permissions. Successful results omit machine IDs+and terminal content. Failed results may include a shell-output tail and an ID+needed for cleanup: inspect and redact those before publishing. A cleanup+failure is a failed run; remove only the recorded probe, never unrelated agents.++For validation:++```sh+npx tsc --noEmit --module nodenext --moduleResolution nodenext \+ --target es2022 --allowImportingTsExtensions --skipLibCheck \+ scripts/benchmark-terminal-latency.mts+```diff --git a/cli/scripts/benchmark-terminal-latency.mts b/cli/scripts/benchmark-terminal-latency.mtsnew file mode 100644index 000000000..a3f9f7a63--- /dev/null+++ b/cli/scripts/benchmark-terminal-latency.mts@@ -0,0 +1,348 @@+// Real local/remote terminal round trips through the running Harness daemon.+// Creates one named disposable terminal and deletes only that terminal in finally.+// No agent/model is launched. Results contain synthetic observations, not terminal history.+// Usage: npx tsx scripts/benchmark-terminal-latency.mts --machine ID --label local --output /tmp/result.json+import WebSocket from 'ws'+import { randomUUID } from 'node:crypto'+import { existsSync, writeFileSync } from 'node:fs'+import { platform, arch } from 'node:os'+import { deflateSync, inflateSync } from 'node:zlib'+import { performance } from 'node:perf_hooks'+import { parseArgs } from 'node:util'+import { decodeTerminalLocal, encodeTerminalLocal, TerminalBinaryKind } from '../src/lib/terminalBinary.ts'++const { values } = parseArgs({ options: {+ machine: { type: 'string' }, label: { type: 'string', default: 'target' },+ output: { type: 'string' }, samples: { type: 'string', default: '200' },+ port: { type: 'string', default: '18473' },+ local: { type: 'boolean', default: false },+ 'control-samples': { type: 'string', default: '30' },+ revision: { type: 'string' },+} })+if (!values.machine || !values.output) throw new Error('--machine and --output are required')+const sampleCount = Number(values.samples)+const controlCount = Number(values['control-samples'])+const port = Number(values.port)+if (!Number.isInteger(sampleCount) || sampleCount < 1 || sampleCount > 2000) throw new Error('samples must be 1–2000')+if (!Number.isInteger(controlCount) || controlCount < 1 || controlCount > 500) throw new Error('control-samples must be 1–500')+if (!Number.isInteger(port) || port < 1 || port > 65535) throw new Error('invalid port')+if (existsSync(values.output)) throw new Error('output already exists; use a fresh result path')+const sleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms))+const now = () => performance.now()+type Frame = { type: string; payload: Record<string, any> }+type Waiter = { test: (frame: Frame) => boolean; resolve: (frame: Frame) => void; reject: (error: Error) => void }+const nonce = randomUUID().replaceAll('-', '')+const prefix = `~HB${nonce}:`+const rows: Record<string, unknown>[] = []+const controls: Record<string, unknown>[] = []+const result: Record<string, any> = {+ schema: 2, success: false, startedAt: new Date().toISOString(), label: values.label,+ sourceRevision: values.revision ?? 'unspecified',+ boundary: 'client binary input send to matching PTY probe response received; excludes UI rendering and OS keyboard input',+ client: { platform: platform(), arch: arch(), node: process.version },+ samplesPerWorkload: sampleCount, warmupsPerWorkload: 10,+ controlSamplesPerRoute: controlCount, controlObservations: controls,+ observations: rows, stagesMs: {}, cleanup: { created: false, deleted: false },+ linkModes: [],+}++class Peer {+ readonly socket = new WebSocket(`ws://127.0.0.1:${port}/api/local-ws`, { handshakeTimeout: 10000 })+ readonly opened: Promise<void>+ readonly waiters = new Set<Waiter>()+ readonly modes: { atMs: number; mode: string }[] = []+ stream = ''+ inputSeq = -1+ mode = values.local ? 'loopback' : 'unknown'+ output = ''+ outputBytes = 0+ keyframes = 0+ fatal: Error | null = null+ outputWaiter: { marker: string; began: number; resolve: (elapsed: number) => void; reject: (error: Error) => void } | null = null+ alive: ReturnType<typeof setInterval> | null = null++ constructor() {+ this.opened = new Promise((resolve, reject) => {+ this.socket.once('open', resolve)+ this.socket.once('error', reject)+ })+ this.socket.on('error', (error) => this.fail(error))+ this.socket.on('close', (code) => this.fail(new Error(`connection closed (${code})`)))+ this.socket.on('message', (raw, binary) => {+ try {+ if (binary) {+ const frame = decodeTerminalLocal(new Uint8Array(raw as Buffer))+ if (!frame || (this.stream && frame.streamId !== this.stream)) return+ if (frame.kind === TerminalBinaryKind.keyframe) this.keyframes+++ if (frame.kind === TerminalBinaryKind.output || frame.kind === TerminalBinaryKind.keyframe) {+ const bytes = frame.compressed ? inflateSync(frame.bytes) : frame.bytes+ this.outputBytes += bytes.length+ this.output = (this.output + Buffer.from(bytes).toString('utf8')).slice(-128 * 1024)+ const pending = this.outputWaiter+ if (pending && this.output.includes(pending.marker)) {+ this.outputWaiter = null+ pending.resolve(now() - pending.began)+ }+ }+ // This is a transport consumer, not a renderer. Acknowledge consumed bytes.+ this.send('terminal_ack', { streamId: frame.streamId, lastSeq: frame.seq })+ return+ }+ const frame = JSON.parse(raw.toString()) as Frame+ if (frame.type === 'terminal_link_mode') {+ this.mode = String(frame.payload.mode)+ this.modes.push({ atMs: now(), mode: this.mode })+ }+ for (const waiter of [...this.waiters]) {+ if (waiter.test(frame)) { this.waiters.delete(waiter); waiter.resolve(frame) }+ }+ if (frame.type === 'terminal_error' && !frame.payload.requestId) {+ this.fail(new Error(`terminal error: ${String(frame.payload.error ?? frame.payload.code)}`))+ }+ } catch (error) { this.fail(error as Error) }+ })+ }++ fail(error: Error) {+ this.fatal = error+ for (const waiter of this.waiters) waiter.reject(error)+ this.waiters.clear()+ this.outputWaiter?.reject(error)+ this.outputWaiter = null+ }++ send(type: string, payload: Record<string, unknown>) {+ if (this.socket.readyState !== WebSocket.OPEN) throw new Error('socket is not open')+ this.socket.send(JSON.stringify({ type, payload }))+ }++ wait(test: Waiter['test'], ms = 30000): Promise<Frame> {+ if (this.fatal) return Promise.reject(this.fatal)+ return new Promise((resolve, reject) => {+ const finish = (error?: Error, frame?: Frame) => {+ clearTimeout(timer); this.waiters.delete(waiter)+ if (error) reject(error); else resolve(frame!)+ }+ const waiter: Waiter = { test, resolve: (frame) => finish(undefined, frame), reject: finish }+ const timer = setTimeout(() => finish(new Error('timed out waiting for a protocol reply')), ms)+ this.waiters.add(waiter)+ })+ }++ async select() {+ await this.opened+ const ready = this.wait((frame) => frame.type === 'connected' || frame.type === 'machine_select_error')+ const began = now()+ this.send('machine_select', { machineId: values.machine, localProtocolVersion: 1 })+ const frame = await ready+ if (frame.type !== 'connected') throw new Error(`machine selection: ${String(frame.payload.error)}`)+ return now() - began+ }++ async rpc(type: string, payload: Record<string, unknown>) {+ const requestId = randomUUID()+ const ready = this.wait((frame) => frame.type === `${type}_result` && frame.payload.requestId === requestId)+ this.send(type, { requestId, ...payload })+ const frame = await ready+ if (frame.payload.error) throw new Error(`${type}: ${String(frame.payload.error)}`)+ return frame.payload+ }++ async attach(agentId: string) {+ const requestId = randomUUID()+ const ready = this.wait((frame) => ['terminal_ready', 'terminal_error'].includes(frame.type) && frame.payload.requestId === requestId)+ const began = now()+ this.send('terminal_open', { requestId, agentId, cols: 120, rows: 40, protocolVersion: 3,+ compression: ['zlib'], takeover: false, client: { kind: 'benchmark', name: 'Disposable latency probe' } })+ const frame = await ready+ if (frame.type !== 'terminal_ready' || frame.payload.readOnly) throw new Error('disposable terminal could not be attached for input')+ this.stream = String(frame.payload.streamId)+ this.inputSeq = -1+ this.alive = setInterval(() => {+ if (this.socket.readyState === WebSocket.OPEN) this.send('terminal_alive', { streamId: this.stream })+ }, 5000)+ return now() - began+ }++ input(bytes: Buffer) {+ const frame = encodeTerminalLocal({ kind: TerminalBinaryKind.input, streamId: this.stream,+ seq: ++this.inputSeq, bytes, compressed: false })+ if (!frame) throw new Error('input encoding failed')+ this.socket.send(frame, { binary: true })+ }++ exchange(bytes: Buffer, marker: string, ms = 15000, paste = false): Promise<number> {+ if (this.fatal) return Promise.reject(this.fatal)+ if (this.outputWaiter) throw new Error('overlapping input probes')+ this.output = ''+ return new Promise((resolve, reject) => {+ const finish = (error?: Error, elapsed?: number) => {+ clearTimeout(timer); this.outputWaiter = null+ if (error) reject(error); else resolve(elapsed!)+ }+ const timer = setTimeout(() => finish(new Error(`PTY response timeout (${ms} ms)`)), ms)+ this.outputWaiter = { marker, began: now(), resolve: (elapsed) => finish(undefined, elapsed), reject: finish }+ if (paste) {+ const frame = encodeTerminalLocal({ kind: TerminalBinaryKind.paste, streamId: this.stream,+ seq: 0, bytes, compressed: false })+ if (!frame) throw new Error('paste encoding failed')+ this.socket.send(frame, { binary: true })+ this.input(Buffer.from('\r'))+ } else {+ this.input(bytes)+ }+ })+ }++ close() {+ if (this.alive) clearInterval(this.alive)+ this.alive = null+ result.linkModes.push(...this.modes)+ this.modes.length = 0+ this.socket.close()+ }+}++// Encoded before shell input: the shell's echo cannot satisfy READY/ECHO checks.+// Raw tty mode means every probe traverses stdin -> Python -> stdout -> tmux.+const probe = `import os,sys,termios,tty,threading,time,platform+fd=sys.stdin.fileno()+saved=termios.tcgetattr(fd)+prefix=${JSON.stringify(prefix)}+lock=threading.Lock()+busy=threading.Event()+done=threading.Event()+def emit(s):+ with lock: os.write(1,s.encode())+def output():+ while not done.wait(.05):+ if busy.is_set(): emit('\\x1b7\\x1b[2;1H'+('background output '*240)+'\\x1b8')+try:+ tty.setraw(fd)+ threading.Thread(target=output,daemon=True).start()+ emit(prefix+'READY:'+platform.system()+':'+platform.machine()+'~')+ n=0+ while True:+ b=os.read(fd,1)+ if b==b'\\x04' or not b: break+ if b==b'\\x02': busy.set()+ elif b==b'\\x01': busy.clear()+ else:+ emit(prefix+'ECHO:'+str(n)+':'+str(b[0])+'~')+ n+=1+finally:+ done.set()+ termios.tcsetattr(fd,termios.TCSADRAIN,saved)+`+let peer = new Peer()+let agentId: string | undefined+let echoSeq = 0+const interrupted = () => peer.fail(new Error('benchmark interrupted; attempting probe cleanup'))+process.once('SIGINT', interrupted)+process.once('SIGTERM', interrupted)+try {+ const status = await fetch(`http://127.0.0.1:${port}/api/status`, { signal: AbortSignal.timeout(5000) }).then((r) => r.json()) as { version?: string }+ result.client.daemonVersion = status.version ?? 'unknown'+ const connectionBegan = now()+ result.stagesMs.select = await peer.select()+ await peer.rpc('terminal_capabilities', {})+ result.stagesMs.readyForRequests = now() - connectionBegan+ const began = now()+ const created = await peer.rpc('agent_create', { engine: 'terminal', name: `Performance probe ${nonce.slice(0, 8)}`, cwd: '/tmp', bypassPermission: false })+ agentId = created.agent?.id+ if (!agentId || created.agent?.engine !== 'terminal') throw new Error('probe terminal was not created')+ result.cleanup.created = true+ result.stagesMs.createTerminal = now() - began+ result.stagesMs.attach = await peer.attach(agentId)+ const command = `python3 -u -c 'import base64,zlib;exec(zlib.decompress(base64.b64decode("${deflateSync(Buffer.from(probe)).toString('base64')}")))'`+ result.stagesMs.startProbe = await peer.exchange(Buffer.from(command), prefix + 'READY:', 15000, true)+ result.target = peer.output.match(new RegExp(`${prefix}READY:([^~]+)~`))?.[1] ?? 'unknown'+ // Permit normal direct-path negotiation; record actual modes on every observation.+ await sleep(2000)+ // Measure control separately: streamless RPCs may use the cloud while a known+ // terminal stream uses P2P. Neither is an ICMP ping or pure network latency.+ for (let sample = -3; sample < controlCount; sample++) {+ const routes = sample % 2 === 0 ? ['machine', 'terminal_stream'] : ['terminal_stream', 'machine']+ for (const route of routes) {+ const modeBefore = peer.mode+ const began = now()+ await peer.rpc('terminal_capabilities', route === 'terminal_stream' ? { streamId: peer.stream } : {})+ controls.push({ route, phase: sample < 0 ? 'warmup' : 'measured', elapsedMs: now() - began,+ terminalModeBefore: modeBefore, terminalModeAfter: peer.mode })+ }+ }+ for (const load of ['idle', 'redraw_20hz']) {+ if (load !== 'idle') { peer.input(Buffer.from([2])); await sleep(150) }+ const bytesBefore = peer.outputBytes+ const loadBegan = now()+ for (let sample = -10; sample < sampleCount; sample++) {+ // Pre-input think time is outside the measurement. Vary timing deterministically.+ await sleep(8 + ((sample + 10) * 11 % 35))+ const byte = 97 + echoSeq % 26+ const modeBefore = peer.mode+ const keyframesBefore = peer.keyframes+ const elapsed = await peer.exchange(Buffer.from([byte]), `${prefix}ECHO:${echoSeq}:${byte}~`)+ rows.push({ load, phase: sample < 0 ? 'warmup' : 'measured', sequence: echoSeq++, elapsedMs: elapsed,+ modeBefore, modeAfter: peer.mode, keyframesDuring: peer.keyframes - keyframesBefore })+ }+ result[`${load}Achieved`] = { bytes: peer.outputBytes - bytesBefore, durationMs: now() - loadBegan }+ if (load !== 'idle') peer.input(Buffer.from([1]))+ }+ const restoreBegan = now()+ peer.close()+ peer = new Peer()+ await peer.select()+ await peer.attach(agentId)+ const byte = 97 + echoSeq % 26+ const echoMs = await peer.exchange(Buffer.from([byte]), `${prefix}ECHO:${echoSeq}:${byte}~`)+ result.reattach = { totalMs: now() - restoreBegan, firstEchoMs: echoMs, sameProbeSequence: echoSeq,+ mode: peer.mode, boundary: 'new local client socket, existing daemon route and same running PTY; not a network outage' }+ result.success = true+} catch (error) {+ result.error = error instanceof Error ? error.message : String(error)+ result.failureDiagnostics = { outputBytes: peer.outputBytes, keyframes: peer.keyframes,+ mode: peer.mode, inputSeq: peer.inputSeq, outputTail: peer.output.slice(-1200) }+ process.exitCode = 1+} finally {+ if (agentId) {+ try {+ if (peer.fatal || peer.socket.readyState !== WebSocket.OPEN) {+ peer.close()+ peer = new Peer()+ await peer.select()+ }+ // The ID is only the terminal returned by this invocation's agent_create.+ const deleted = await peer.rpc('agent_delete', { agentId })+ if (deleted.deleted !== true) throw new Error('daemon did not confirm probe deletion')+ result.cleanup.deleted = true+ } catch (error) {+ result.cleanup = { deleted: false, agentId, error: String(error) }+ result.success = false; process.exitCode = 1+ }+ }+ peer.close()+ result.finishedAt = new Date().toISOString()+ result.summaries = ['idle', 'redraw_20hz'].map((load) => {+ const values = rows.filter((row) => row.load === load && row.phase === 'measured').map((row) => Number(row.elapsedMs)).sort((a,b) => a-b)+ const at = (p: number) => values.length ? values[Math.ceil(values.length * p) - 1] : null+ return { load, samples: values.length, p50Ms: at(.5), p95Ms: at(.95), p99Ms: at(.99), maxMs: values.at(-1) ?? null }+ })+ result.controlSummaries = ['machine', 'terminal_stream'].map((route) => {+ const values = controls.filter((row) => row.route === route && row.phase === 'measured').map((row) => Number(row.elapsedMs)).sort((a,b) => a-b)+ const at = (p: number) => values.length ? values[Math.ceil(values.length * p) - 1] : null+ return { route, samples: values.length, p50Ms: at(.5), p95Ms: at(.95), p99Ms: at(.99), maxMs: values.at(-1) ?? null }+ })+ result.summariesByTerminalMode = [...new Set(rows.map((row) => `${row.load}/${row.modeBefore === row.modeAfter ? row.modeAfter : 'changed_during_sample'}`))].map((group) => {+ const [load, mode] = group.split('/')+ const values = rows.filter((row) => row.phase === 'measured' && row.load === load+ && (row.modeBefore === row.modeAfter ? row.modeAfter : 'changed_during_sample') === mode)+ .map((row) => Number(row.elapsedMs)).sort((a,b) => a-b)+ const at = (p: number) => values.length ? values[Math.ceil(values.length * p) - 1] : null+ return { load, mode, samples: values.length, p50Ms: at(.5), p95Ms: at(.95), p99Ms: at(.99), maxMs: values.at(-1) ?? null }+ })+ process.removeListener('SIGINT', interrupted)+ process.removeListener('SIGTERM', interrupted)+ writeFileSync(values.output, JSON.stringify(result, null, 2) + '\n', { mode: 0o600, flag: 'wx' })+ console.log(JSON.stringify({ label: values.label, success: result.success, summaries: result.summaries, stagesMs: result.stagesMs,+ controlSummaries: result.controlSummaries, reattach: result.reattach, target: result.target, cleanup: result.cleanup, error: result.error }))+}