A local-first multi-agent app development platform, built on PenguinHarness
feat(server,web): 终端在移动网络下不再假死,断了会自己重连
lmlihengbe87f4d3f396
diff --git a/changelog/unreleased/2026-11-02-terminal-on-a-slow-link.md b/changelog/unreleased/2026-11-02-terminal-on-a-slow-link.mdnew file mode 100644index 00000000..6d8d75c8--- /dev/null+++ b/changelog/unreleased/2026-11-02-terminal-on-a-slow-link.md@@ -0,0 +1,31 @@+# The terminal survives a mobile network, and reattaches on its own++- **Date:** 2026-11-02+- **Type:** feat+- **Scope:** `server`, `web`+- **Breaking:** no++[中文版](2026-11-02-terminal-on-a-slow-link.zh.md)++A terminal opened over a phone's network used to die quietly. The socket does not close politely there — the radio hands over, a carrier NAT reaps an idle flow, the browser freezes the tab — and what the page was left holding was a connection that would never carry another byte, painted as "stream closed". The pty on the server was fine the whole time; a reload was the only way back to it.++Both ends now probe each other. Either side may send a Ping and the peer echoes it back untouched, so a socket that stopped carrying bytes is found within a bounded silence window rather than never. The server closes a viewer that stops answering; the client reattaches after one that stops answering it, with a bounded backoff, and the reattach repaints the screen from the server's own snapshot — the same repaint a fresh attach gets. A network coming back, or a tab coming back, collapses that wait: the connection retries at once, and a socket that merely *looks* open is probed before it is trusted. A tab that was suspended is not mistaken for a dead link. The one failure worth stopping on is a terminal that is genuinely gone, which a retry checks for before it reconnects. The status word for all of this is `reconnecting`, distinct from the `connecting` of a first attach.++Ported from PenguinHarness `#639` (`f52617695`), by applying its changes rather than merging branches: all sixteen of its files exist here and `git apply -3 --check` was clean. Nothing this change touches is missing from this tree (the machine-side terminal shells and `terminal-panel.tsx` are outside it), so there were no trade-offs to record.++## Details++Every step of the reattach carries its own clock, and only one attempt is ever in flight. A network or a tab that comes back while an attach is still pending joins that attempt instead of starting another: the first-attach policy runs once (twice could mean two terminals), one socket is open at a time, and an answer that arrives after its attempt was given up on opens nothing — the old socket is closed before the next one is dialled. A dial that never reaches the open state (15 seconds) and an existence probe that never answers (10 seconds) read as the same transient failure a rejection does, and go to the backoff; a pane that goes away stops even a probe that is still out. The right to be judged by silence belongs to the socket that earned it: probe answers do not carry over a reattach, so a fresh socket onto an older server is never killed by the previous one's answers.++Three numbers stopped being constants tuned on a LAN:++- **The output merge window** followed the round trip. At 200ms it merged almost nothing while producing hundreds of small frames a second; it is now a quarter of the measured trip, bounded to 5–50ms, so a fast link keeps the behaviour it had.+- **The backpressure high-water mark** was a flat megabyte, which is milliseconds of lag on a LAN and many seconds on a slow uplink — the "output arrives late, then jumps" complaint. It is now derived from what the viewer's socket actually delivers, so it bounds *time* (three quarters of a second) instead of bytes.+- **Container resizes** reached the pty one per pixel change. On a phone the address bar collapses and the soft keyboard opens constantly, and every size that lands costs a SIGWINCH and a full repaint from whatever is running. They settle for 120ms now, and only a change that moves the grid is sent at all.++The stream is also compressed on the wire (permessage-deflate, a 13-bit window and no client context takeover), and the small frames are left out of it by the sender itself, message by message. ws honours a config-level `threshold` only for a direction that negotiated away its compression context; this stream keeps its context — a shared context is what compresses repeated output best — so the 512-byte cutoff is decided at the send call, per message. The backlog is counted in the bytes the terminal *produced* (ws's own send callbacks) rather than in `bufferedAmount`: once compressed, that is wire bytes, and a screenful of repeated build output is a hundred times smaller there and would read as "not behind at all".++## Compatibility++- Both ends can be older than this change: a hot push replaces the platform under a tab that was not reloaded, and a machine runs whatever version it was last installed with. Unknown frame types were already harmless in both directions; the hazard is the conclusion each end draws from silence, so both sides share one rule with no version negotiation: **only a peer that has answered a probe is ever judged for its silence** (one `answersProbes` per socket, cleared each time a socket opens). Without it, a server would close an old page every 75 seconds (and that page has no reattach), and this page would reattach to an old server roughly every minute.+- No API or dependency change; the two new frame types live only in the server's and the Web's own frame definitions.diff --git a/changelog/unreleased/2026-11-02-terminal-on-a-slow-link.zh.md b/changelog/unreleased/2026-11-02-terminal-on-a-slow-link.zh.mdnew file mode 100644index 00000000..b7ce8bdf--- /dev/null+++ b/changelog/unreleased/2026-11-02-terminal-on-a-slow-link.zh.md@@ -0,0 +1,57 @@+# 终端在移动网络下不再假死,自己重连++- **Date:** 2026-11-02+- **Type:** feat+- **Scope:** `server`, `web`+- **Breaking:** no++[English](2026-11-02-terminal-on-a-slow-link.md)++在手机网络上打开的终端,过去是悄无声息地死掉的。那种链路上的 socket 不会礼貌地关闭——基站切换、+运营商 NAT 回收空闲连接、浏览器冻结后台标签页——页面手里剩下的是一条再也不会传输任何字节的连接,+界面上写着「stream closed」。而服务端的 pty 自始至终好好的,唯一的回去方式是刷新页面。++现在两端互相探测:任意一端都可以发 Ping,对端原样回 Pong,于是一条不再传输字节的 socket 会在有界的+沉默窗口内被发现,而不是永远发现不了。服务端关闭不再应答的观察者;客户端在对方不再应答时按有界退避+重新接入,重连后由服务端的快照重绘整屏——与首次接入拿到的是同一份重绘。网络恢复或标签页回到前台会+立即结束等待:连接马上重试,而看起来还开着的 socket 会先被探测再被信任;被浏览器挂起过的标签页不会+被误判成死链路。唯一值得停下来的失败是终端真的不在了,重连前会先确认这一点。这一整套状态在界面上叫+`reconnecting`,与首次接入的 `connecting` 区分开。++移植自 PenguinHarness `#639`(`f52617695`),照改动落、不是合分支:那一笔的 16 个文件本树全有,+`git apply -3 --check` 干净、一个字没冲突;本树没有的东西(机器侧的终端外壳、`terminal-panel.tsx` 那套+面板)不在这一笔的范围里,因此没有取舍需要记。++## 细节++重连的每一步都自带期限,同一时刻至多一次尝试在途。网络或标签页在一次接入还没结果时回来,加入的就是+这次在途的尝试,而不是再起一次:首次接入的宿主策略只跑一遍(跑两遍可能意味着两个终端),任何时刻只有+一条 socket,被放弃的尝试迟来的结果什么也不会打开——拨下一条之前,旧的 socket 已经关掉。永远到不了+open 状态的拨号(15 秒)与永远不回话的存在探测(10 秒),读作与被拒相同的暂态失败,交回退避阶梯;+面板消失时,连还在途的探测也一并停掉。被沉默判决的资格属于挣得它的那条 socket:探测的应答不跨重连+继承,重连落到一台从不应答的旧服务端上,不会被上一条的应答误杀。++三个按局域网调出来的常量不再是常量:++- **输出合并窗口**跟随往返时延。200ms 的链路上,固定 5ms 几乎合并不了任何东西,却每秒产生几百个小帧;+ 现在取实测往返的四分之一,限制在 5–50ms,快链路上行为不变。+- **背压高水位**原本是固定的 1MB,在局域网上是几毫秒的滞后,在慢上行链路上却是好几秒——正是「输出迟迟+ 不来、然后猛地跳过去」的来源。现在它由该观察者 socket 实际投递的速率推出,界定的是**时间**(0.75 秒)+ 而不是字节数。+- **容器尺寸变化**过去每次像素变动都会传到 pty。手机上地址栏收起、软键盘弹出会不断触发,而每一次尺寸+ 落地都意味着一次 SIGWINCH 与运行中程序的整屏重绘。现在它们先静置 120ms,并且只有真正改变了行列数的+ 变化才会发出去。++流本身也在链路上压缩了(permessage-deflate,限定 13 位窗口、不保留客户端上下文);小帧不压由发送方+逐条决定。ws 只对协商放弃压缩上下文的一侧应用配置级的 `threshold`,而这条流保留自己的上下文——共享+上下文正是重复输出压得最好的原因——512 字节的分界因此落在发送调用上、逐帧判定。背压的积压量也随之+改成按「终端产出了多少字节」来数(ws 的发送回调),不再看 `bufferedAmount`:压缩之后它是线上字节,+一屏重复的构建输出在线上小一百倍,会被读成「没落后」。++## 兼容性++- 两端都可能是旧的:热更新会把平台换在一个没刷新的标签页下面,一台机器跑的是它上次安装的版本。+ 两个方向的未知帧类型本来就无害,真正的风险是各自从沉默里得出的结论,因此两侧共用一条规则、不做版本+ 协商:**只有应答过探测的对端才会被沉默判决**(每侧一个 `answersProbes`,每次 socket 打开时清掉)。+ 没有这条,服务端会每 75 秒关掉一个旧页面(而它没有重连),这个页面也会约每分钟重连一次到旧服务端。+- 没有接口或依赖变化;两个新帧类型只写在服务端与 Web 各自的帧定义里。diff --git a/packages/server/src/terminal/frames.ts b/packages/server/src/terminal/frames.tsindex 9ca4a516..6c2d0dca 100644--- a/packages/server/src/terminal/frames.ts+++ b/packages/server/src/terminal/frames.ts@@ -11,6 +11,12 @@* byte 1 slot (stream index within one socket)* byte 2+ payload (raw UTF-8 terminal bytes, or JSON for the control opcodes)*+ * Ping/Pong are symmetric on purpose: each end probes for itself. A mobile link drops a+ * socket without closing it (a radio hand-off, a carrier NAT reaping an idle flow), and a+ * peer that only answers probes never finds out. The payload is the prober's own clock+ * reading, echoed verbatim, so the round trip falls out of the reply — that measurement is+ * what sizes the output merge window (link-quality.ts).+ ** `slot` is always 0 today — this server opens one socket per terminal. It is in the wire* format from the start because the alternative (adding it later) is a protocol break, and* because multiplexing every terminal of a page onto one socket is the obvious next step.@@ -27,6 +33,10 @@ export const TerminalStreamOpcode = {Restore: 0x05,/** Server -> client: the pty exited, JSON `{exitCode, signal}`. */Exit: 0x06,+ /** Either direction: liveness probe. The peer echoes the payload back as Pong. */+ Ping: 0x07,+ /** Either direction: the echo of a Ping's payload, unchanged. */+ Pong: 0x08,} as const;export type TerminalStreamOpcode = (typeof TerminalStreamOpcode)[keyof typeof TerminalStreamOpcode];diff --git a/packages/server/src/terminal/link-quality.ts b/packages/server/src/terminal/link-quality.tsnew file mode 100644index 00000000..036f3282--- /dev/null+++ b/packages/server/src/terminal/link-quality.ts@@ -0,0 +1,132 @@+/**+ * What the stream knows about the link it is writing to: how long a round trip takes, and+ * how fast the viewer's socket actually drains.+ *+ * Both numbers exist because the transport's two constants were tuned on a link with no+ * latency and no bandwidth limit, and both of them mean something different on a phone:+ *+ * - The output merge window (output-coalescer.ts) was a flat 5ms. At a 200ms round trip+ * that window merges almost nothing while producing hundreds of small frames a second —+ * pure overhead on a radio, and no faster to arrive, because the round trip dominates by+ * two orders of magnitude. Scaled to the measured trip it merges properly, and on a fast+ * link it stays where it was.+ * - The backpressure high-water mark was a flat 1MB. A byte bound is a LAG bound only if+ * you know the rate: 1MB is a few milliseconds behind on a LAN and the better part of ten+ * seconds behind on a slow uplink — which is exactly the "output arrives late, then jumps"+ * complaint. Measured delivery turns it back into the time bound it was meant to be.+ *+ * Both estimators are deliberately conservative: with no measurement yet, they answer with+ * the old constants, so a link that never reports stays on the behaviour it always had.+ */++/** Smoothing for both estimators: the new sample weighs a fifth. */+const ALPHA = 0.2;++export const MIN_COALESCE_WINDOW_MS = 5;+export const MAX_COALESCE_WINDOW_MS = 50;++/**+ * The merge window for a measured round trip: a quarter of it, bounded.+ *+ * A quarter keeps the window an order of magnitude below the delay the user is already+ * paying, so merging is invisible to them, while the ceiling keeps a satellite-grade link+ * from batching output into visible chunks. `null` (nothing measured yet) answers the+ * floor, which is what this server did before it measured anything.+ */+export function coalesceWindowFor(roundTripMs: number | null): number {+ if (roundTripMs === null || !Number.isFinite(roundTripMs)) return MIN_COALESCE_WINDOW_MS;+ const quarter = roundTripMs / 4;+ return Math.min(MAX_COALESCE_WINDOW_MS, Math.max(MIN_COALESCE_WINDOW_MS, Math.round(quarter)));+}++/** Round-trip estimate from Ping/Pong echoes, smoothed so one stalled reply cannot swing it. */+export class RoundTripEstimator {+ private estimate: number | null = null;++ /** Ignores nonsense (a clock that went backwards, an echo of something we never sent). */+ sample(roundTripMs: number): void {+ if (!Number.isFinite(roundTripMs) || roundTripMs < 0) return;+ this.estimate =+ this.estimate === null ? roundTripMs : this.estimate * (1 - ALPHA) + roundTripMs * ALPHA;+ }++ get roundTripMs(): number | null {+ return this.estimate;+ }++ windowMs(): number {+ return coalesceWindowFor(this.estimate);+ }+}++/** How far behind a viewer may fall before the stream skips it ahead with a fresh repaint. */+export const MAX_VIEWER_LAG_MS = 750;+/**+ * Floor and ceiling for the derived high-water mark. The ceiling is the byte bound this+ * server always used, so a link fast enough to drain it keeps its old behaviour; the floor+ * stays clear of the low-water mark, or a viewer would resync on every burst.+ */+export const MIN_HIGH_WATER_BYTES = 128 * 1024;+export const MAX_HIGH_WATER_BYTES = 1024 * 1024;++/** Shortest interval between two throughput samples: below this, one flush is all noise. */+const SAMPLE_INTERVAL_MS = 250;++/**+ * Throughput of one viewer's stream, measured from what has actually left.+ *+ * Every frame reports itself twice — once when it is handed to the socket, once when the+ * socket says it is gone — and the second report is the measurement: bytes delivered per+ * second. It is only taken while something WAS waiting, because a socket with an empty+ * queue delivers exactly as fast as output is produced, and an idle shell would otherwise+ * "measure" a few bytes a second and shrink the viewer's allowance to nothing.+ *+ * Both numbers are in the bytes the terminal produced, never in the bytes the socket wrote:+ * the stream is compressed on the wire, and the lag a viewer feels is measured in the+ * output it has not seen yet.+ */+export class SocketDrainMeter {+ private delivered = 0;+ private lastSampleAt: number | null = null;+ private lastDelivered = 0;+ private lastQueued = 0;+ private estimate: number | null = null;++ constructor(private readonly now: () => number = () => Date.now()) {}++ /** Reports bytes the socket has finished with, and what is still waiting behind them. */+ note(delivered: number, queued: number): void {+ this.delivered += delivered;+ const at = this.now();+ if (this.lastSampleAt === null) {+ this.reset(at, queued);+ return;+ }+ const elapsed = at - this.lastSampleAt;+ if (elapsed < SAMPLE_INTERVAL_MS) return;+ const backlogged = this.lastQueued > 0 || queued > 0;+ const drained = this.delivered - this.lastDelivered;+ this.reset(at, queued);+ if (!backlogged || drained <= 0) return;+ const rate = (drained * 1000) / elapsed;+ this.estimate = this.estimate === null ? rate : this.estimate * (1 - ALPHA) + rate * ALPHA;+ }++ private reset(at: number, queued: number): void {+ this.lastSampleAt = at;+ this.lastDelivered = this.delivered;+ this.lastQueued = queued;+ }++ /** Bytes per second, or null while the stream has never had to wait for the socket. */+ get bytesPerSecond(): number | null {+ return this.estimate;+ }++ /** The backlog that means MAX_VIEWER_LAG_MS of lag at the measured rate. */+ highWaterBytes(): number {+ if (this.estimate === null) return MAX_HIGH_WATER_BYTES;+ const allowance = (this.estimate * MAX_VIEWER_LAG_MS) / 1000;+ return Math.min(MAX_HIGH_WATER_BYTES, Math.max(MIN_HIGH_WATER_BYTES, Math.round(allowance)));+ }+}diff --git a/packages/server/src/terminal/output-coalescer.ts b/packages/server/src/terminal/output-coalescer.tsindex 76095f1c..0841cace 100644--- a/packages/server/src/terminal/output-coalescer.ts+++ b/packages/server/src/terminal/output-coalescer.ts@@ -9,6 +9,10 @@** Reverting this to trailing-only is the classic regression here: it adds a full window* (~5ms) to every single keystroke and the terminal immediately feels mushy.+ *+ * The window may be a function, which is how the stream widens it on a slow link: the+ * merge is worth more, and costs proportionally less, the longer the round trip already is+ * (link-quality.ts). It is read per burst, never cached.*/export const DEFAULT_COALESCE_WINDOW_MS = 5;@@ -20,25 +24,30 @@ export class TerminalOutputCoalescer {constructor(private readonly flushFn: (data: string) => void,- private readonly windowMs: number = DEFAULT_COALESCE_WINDOW_MS,+ private readonly window: number | (() => number) = DEFAULT_COALESCE_WINDOW_MS,private readonly now: () => number = () => Date.now(),) {}+ private windowMs(): number {+ return typeof this.window === "number" ? this.window : this.window();+ }+push(data: string): void {if (data.length === 0) return;this.pending += data;if (this.timer !== null) return; // a trailing flush is already scheduled+ const windowMs = this.windowMs();const elapsed = this.now() - this.lastFlushAt;- if (elapsed >= this.windowMs) {+ if (elapsed >= windowMs) {this.flush(); // leading edge: straight through, no added latencyreturn;}this.timer = setTimeout(() => {this.timer = null;this.flush();- }, this.windowMs - elapsed);+ }, windowMs - elapsed);}/**diff --git a/packages/server/src/terminal/stream.ts b/packages/server/src/terminal/stream.tsindex 39d99fc7..8d7cd645 100644--- a/packages/server/src/terminal/stream.ts+++ b/packages/server/src/terminal/stream.ts@@ -1,13 +1,16 @@/*** The terminal stream's PROTOCOL: frames, coalescing, the restore-first attach sequence,- * size ownership and the lagging-viewer resync.+ * size ownership, the heartbeat and the lagging-viewer resync.** This is platform code even though the socket is the runtime's. The seam cannot carry a* stream — a handler returns one whole Response — so the runtime keeps the upgrade* handshake and its authentication (terminal/ws.ts) and hands the live socket here. What* flows over it afterwards is behaviour, and behaviour a push must be able to change:- * every byte of the wire format, the ~5ms coalescing window and both backpressure- * watermarks are decided in this file.+ * every byte of the wire format, the coalescing window, the heartbeat and both+ * backpressure watermarks are decided in this file. Two of those are no longer constants:+ * the window and the high-water mark are derived per socket from what the link measures+ * (link-quality.ts), because a number tuned on a LAN means something else entirely on a+ * phone.** Attach sequence — the part that makes a browser reload look seamless:* 1. the client's geometry (?cols=&rows=) resizes the pty FIRST, so the snapshot is@@ -27,12 +30,54 @@ import {parseResizePayload,} from "./frames.js";import { TerminalOutputCoalescer } from "./output-coalescer.js";+import { RoundTripEstimator, SocketDrainMeter } from "./link-quality.js";import type { TerminalSession } from "./session.js";-const BACKPRESSURE_HIGH_WATER = 1024 * 1024;+/**+ * Per-viewer backpressure. A send queues without limit when the peer cannot drain, so a+ * viewer too far behind stops receiving live output — the bytes keep feeding the server+ * emulator, nothing of the session is lost — and once its backlog falls below LOW_WATER it+ * is repainted with one fresh Restore frame (the same self-contained repaint an attach+ * uses) and live output resumes. Skipping ahead beats both alternatives: replaying+ * megabytes the viewer could only fast-forward through, or disconnecting it.+ *+ * "Too far behind" is the one number that is no longer a constant: it is derived per socket+ * from what that link delivers (link-quality.ts), because the byte bound this used to be+ * meant milliseconds of lag on a LAN and many seconds on a phone.+ */const BACKPRESSURE_LOW_WATER = 64 * 1024;const BACKPRESSURE_POLL_MS = 250;+/**+ * Heartbeat. A phone loses a socket without closing it — a radio hand-off, a carrier NAT+ * reaping an idle flow — and both ends then hold a connection that will never carry a byte+ * again: the pty writes into a dead socket, the viewer waits for output that cannot arrive.+ * The probe is what turns that into a close, which is what the client retries on. The reply+ * doubles as the round-trip sample that sizes the merge window (link-quality.ts).+ *+ * The timeout is generous on purpose: a backgrounded tab is throttled to roughly one timer+ * a minute, and reaping a viewer that is merely asleep would cost it a full repaint on+ * every glance at the screen.+ *+ * Only a viewer that has ANSWERED a probe is ever reaped for silence. A page loaded before+ * this shipped stays open across the push that brought it — the platform is replaced under+ * a tab that is not reloaded — and that page does not know the opcode: it would go quiet,+ * be closed, and (having no reattach either) show a dead pane. Silence only means something+ * from a peer that has shown it can break it.+ */+const HEARTBEAT_INTERVAL_MS = 15_000;+const HEARTBEAT_TIMEOUT_MS = 75_000;++/**+ * Frames smaller than this go out uncompressed even when the stream negotiated+ * permessage-deflate: a keystroke's echo compresses to more bytes than it started with+ * once the deflate overhead rides along. ws's own `perMessageDeflate.threshold` cannot+ * carry this promise — it only applies to a direction that negotiated no_context_takeover,+ * and this server keeps its deflate context precisely because a shared context is what+ * compresses repeated output best — so the decision is made per message, here.+ */+const SMALL_FRAME_BYTES = 512;+export function bindTerminalStream(ws: WebSocket,session: TerminalSession,@@ -42,8 +87,28 @@ export function bindTerminalStream(const connectionId = randomUUID();ws.binaryType = "nodebuffer";+ // What this viewer's link is doing, measured from this socket alone: the round trip from+ // the heartbeat's echo, the drain rate from what the socket accepts.+ const roundTrip = new RoundTripEstimator();+ const drain = new SocketDrainMeter();++ /**+ * Bytes handed to the socket that have not left it yet — the viewer's backlog, counted in+ * the output the terminal produced rather than in what the wire carried. `bufferedAmount`+ * cannot answer this any more: the stream is compressed (terminal/ws.ts), and a screen of+ * repeated build output leaves the socket a hundred times smaller than the user is behind.+ * ws calls back once a message is flushed, so what has not called back is what is waiting.+ */+ let inFlight = 0;+const send = (bytes: Uint8Array): void => {- if (ws.readyState === ws.OPEN) ws.send(bytes);+ if (ws.readyState !== ws.OPEN) return;+ const size = bytes.byteLength;+ inFlight += size;+ ws.send(bytes, { compress: size >= SMALL_FRAME_BYTES }, () => {+ inFlight -= size;+ drain.note(size, inFlight);+ });};// Lagging-viewer state (see the watermark comment above): while desynced, this viewer's@@ -58,12 +123,15 @@ export function bindTerminalStream(const sendOutput = (data: string): void => {if (desynced) return; // dropped: the emulator keeps every byte; the resync repaintssend(encodeTerminalFrame({ opcode: TerminalStreamOpcode.Output, payload: data }));- if (ws.bufferedAmount <= BACKPRESSURE_HIGH_WATER) return;+ // A LAG bound, in bytes: how much output this viewer can still be behind and catch up+ // within MAX_VIEWER_LAG_MS at the rate its socket has been delivering (link-quality.ts),+ // which on a link fast enough to drain it is the 1MB this server always used.+ if (inFlight <= drain.highWaterBytes()) return;desynced = true;- log(`[terminal] stream ${session.id}: viewer ${ws.bufferedAmount}B behind, pausing for resync`);+ log(`[terminal] stream ${session.id}: viewer ${inFlight}B behind, pausing for resync`);resyncTimer = setInterval(() => {if (ws.readyState !== ws.OPEN) return stopResyncPoll();- if (ws.bufferedAmount > BACKPRESSURE_LOW_WATER) return;+ if (inFlight > BACKPRESSURE_LOW_WATER) return;stopResyncPoll();desynced = false; // flip first: output parsed after this snapshot flows live againtry {@@ -83,7 +151,9 @@ export function bindTerminalStream(// Buffers output until the Restore frame has been sent (see the attach sequence above).let restored = false;let preRestore = "";- const coalescer = new TerminalOutputCoalescer(sendOutput);+ // The merge window follows the measured round trip: 5ms on a LAN, up to 50ms on a link+ // where a single trip already costs twenty times that.+ const coalescer = new TerminalOutputCoalescer(sendOutput, () => roundTrip.windowMs());const unsubscribeOutput = session.onOutput((data) => {if (!restored) {@@ -131,8 +201,23 @@ export function bindTerminalStream();}+ // Anything arriving proves the socket is still a socket, heartbeat replies included.+ let lastSeenAt = Date.now();+ // Whether this viewer speaks the heartbeat at all (see HEARTBEAT_TIMEOUT_MS).+ let answersProbes = false;+ const heartbeat = setInterval(() => {+ if (ws.readyState !== ws.OPEN) return;+ if (answersProbes && Date.now() - lastSeenAt > HEARTBEAT_TIMEOUT_MS) {+ log(`[terminal] stream ${session.id}: viewer stopped answering, closing`);+ ws.terminate(); // not close(): a socket this quiet will not complete a handshake+ return;+ }+ send(encodeTerminalFrame({ opcode: TerminalStreamOpcode.Ping, payload: String(Date.now()) }));+ }, HEARTBEAT_INTERVAL_MS);+ws.on("message", (raw: Buffer, isBinary: boolean) => {if (!isBinary) return; // the data plane is binary-only+ lastSeenAt = Date.now();const frame = decodeTerminalFrame(new Uint8Array(raw.buffer, raw.byteOffset, raw.byteLength));if (!frame) return;switch (frame.opcode) {@@ -144,12 +229,24 @@ export function bindTerminalStream(if (size) session.resize({ connectionId, ...size });break;}+ // The client probes for itself (its own timers are the ones a phone suspends); the+ // payload is its clock reading and goes back untouched.+ case TerminalStreamOpcode.Ping:+ send(encodeTerminalFrame({ opcode: TerminalStreamOpcode.Pong, payload: frame.payload }));+ break;+ case TerminalStreamOpcode.Pong: {+ answersProbes = true;+ const sentAt = Number.parseInt(framePayloadText(frame), 10);+ if (Number.isFinite(sentAt)) roundTrip.sample(Date.now() - sentAt);+ break;+ }default:break; // server-only opcodes echoed back by a confused client}});const teardown = (): void => {+ clearInterval(heartbeat);unsubscribeOutput();unsubscribeExit();stopResyncPoll();diff --git a/packages/server/src/terminal/ws.ts b/packages/server/src/terminal/ws.tsindex f4e6ded6..7dd01adc 100644--- a/packages/server/src/terminal/ws.ts+++ b/packages/server/src/terminal/ws.ts@@ -21,21 +21,6 @@ import type { Auth } from "../mechanisms/identity.js";const STREAM_PATH = /^\/api\/terminals\/([^/]+)\/stream$/;-/**- * Per-viewer backpressure watermarks. `ws.send` queues without limit when the peer cannot- * drain, so a viewer more than HIGH_WATER behind stops receiving live output — the bytes- * keep feeding the server emulator, nothing of the session is lost — and once its socket- * drains below LOW_WATER it is repainted with one fresh Restore frame (the same- * self-contained repaint an attach uses) and live output resumes. Skipping ahead beats- * both alternatives: replaying megabytes the viewer could only fast-forward through, or- * disconnecting it (the client treats a closed stream as a dead pane, not a retry). The- * high watermark is therefore a LAG bound — how far behind a viewer may fall before it is- * fast-forwarded — and doubles as the per-viewer memory bound.- */-const BACKPRESSURE_HIGH_WATER = 1024 * 1024;-const BACKPRESSURE_LOW_WATER = 64 * 1024;-const BACKPRESSURE_POLL_MS = 250;-export interface TerminalWebSocketDeps {/** The booted platform owns the terminals; the runtime asks it per upgrade. */hmr: ServerHmrHost;@@ -44,7 +29,27 @@ export interface TerminalWebSocketDeps {}export function attachTerminalWebSocket(server: HttpServer, deps: TerminalWebSocketDeps): void {- const wss = new WebSocketServer({ noServer: true, perMessageDeflate: false });+ const wss = new WebSocketServer({+ noServer: true,+ /**+ * Terminal output is the most compressible traffic this server has — repeated prompts,+ * repeated SGR runs, whole repainted screens — and on a phone the bytes are the scarce+ * thing, so it is worth a little CPU here. The settings keep that little: the window is+ * capped so a per-socket deflate context costs kilobytes rather than the default's+ * hundreds, and the client's context is not kept, because a browser's uplink here is a+ * keystroke at a time and has nothing to reuse. Small frames are left uncompressed by+ * the sender itself, per message (terminal/stream.ts), not by a threshold here: ws only+ * applies a config `threshold` to a direction that negotiated no_context_takeover, and+ * this server keeps its context — a shared context is what compresses repeated output+ * best.+ */+ perMessageDeflate: {+ clientNoContextTakeover: true,+ serverMaxWindowBits: 13,+ zlibDeflateOptions: { level: 3, memLevel: 7 },+ concurrencyLimit: 10,+ },+ });server.on("upgrade", (req: IncomingMessage, socket: Duplex, head: Buffer) => {const url = new URL(req.url ?? "/", "http://localhost");diff --git a/packages/server/test/terminal-stream.test.ts b/packages/server/test/terminal-stream.test.tsindex d7c9f827..9c95f718 100644--- a/packages/server/test/terminal-stream.test.ts+++ b/packages/server/test/terminal-stream.test.ts@@ -10,6 +10,7 @@import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";import { serve } from "@hono/node-server";import type { Server as HttpServer } from "node:http";+import type { Socket as NetSocket } from "node:net";import { WebSocket } from "ws";import { createTestApp, loginAdmin, provisionUser, apiClient } from "./helpers.js";import type { TestApp } from "./helpers.js";@@ -18,6 +19,7 @@ import {TerminalStreamOpcode,decodeTerminalFrame,encodeTerminalFrame,+ framePayloadText,} from "../src/terminal/frames.js";const TEST_TIMEOUT = 20_000;@@ -111,15 +113,29 @@ async function terminalInfo(id: string): Promise<TerminalInfoJson> {return (await res.json()) as TerminalInfoJson;}+/**+ * One server→client data frame as it left the wire, before ws inflates it: the compression+ * test reads the RSV1 bit (permessage-deflate) and the frame's real length from here.+ */+interface WireFrame {+ rsv1: boolean;+ opcode: number;+ length: number;+ payload: Uint8Array;+}+interface StreamClient {ws: WebSocket;frames: Array<{ opcode: number; text: string }>;+ /** Raw first-fragment frames seen on the socket; populated only for deflate clients. */+ wire: WireFrame[];/** Concatenation of every Output payload so far. */outputText(): string;/** Payload of the first Restore frame (the attach snapshot). */restoreText(): string;sendInput(text: string): void;sendResize(cols: number, rows: number, intent: "claim" | "update"): void;+ sendPing(payload: string): void;waitFor(predicate: () => boolean, what: string, timeoutMs?: number): Promise<void>;close(): Promise<void>;}@@ -132,15 +148,27 @@ function streamUrl(id: string, cols?: number, rows?: number): string {/** Opens a stream and resolves once connected; every received frame must decode. */function attach(id: string,- options: { cookie?: string; cols?: number; rows?: number; origin?: string } = {},+ options: {+ cookie?: string;+ cols?: number;+ rows?: number;+ origin?: string;+ /** Negotiate permessage-deflate (a browser does) and record raw frames on the wire. */+ deflate?: boolean;+ } = {},): Promise<StreamClient> {const headers: Record<string, string> = { cookie: options.cookie ?? adminCookie };if (options.origin !== undefined) headers.origin = options.origin;- const ws = new WebSocket(streamUrl(id, options.cols ?? 80, options.rows ?? 24), { headers });+ const ws = new WebSocket(streamUrl(id, options.cols ?? 80, options.rows ?? 24), {+ headers,+ ...(options.deflate === true ? { perMessageDeflate: {} } : {}),+ });const frames: StreamClient["frames"] = [];+ const wire: WireFrame[] = [];// Every server-sent frame must decode; checked from waitFor (throwing inside the ws// 'message' listener would surface as an uncaught exception, not a test failure).let undecodable = 0;+ let wireError: string | null = null;ws.on("message", (raw: Buffer) => {const frame = decodeTerminalFrame(new Uint8Array(raw.buffer, raw.byteOffset, raw.byteLength));@@ -151,9 +179,60 @@ function attach(frames.push({ opcode: frame.opcode, text: new TextDecoder().decode(frame.payload) });});+ // The wire tap for the deflate client: walks the raw server→client stream and records+ // each message's first fragment — the one that carries the compression bit. Server+ // frames are never masked, so the walk is header + length + payload, nothing else.+ const watchWire = (): void => {+ const sock = (ws as unknown as { _socket?: NetSocket })._socket;+ if (sock === undefined) {+ wireError = "the deflate client has no underlying socket to watch";+ return;+ }+ let pending = Buffer.alloc(0);+ sock.on("data", (chunk: Buffer) => {+ pending = Buffer.concat([pending, chunk]);+ for (;;) {+ if (pending.length < 2) break;+ const b0 = pending[0] as number;+ const b1 = pending[1] as number;+ if ((b1 & 0x80) !== 0) {+ wireError = "server frames must arrive unmasked";+ return;+ }+ let header = 2;+ let length = b1 & 0x7f;+ if (length === 126) {+ if (pending.length < 4) break;+ length = pending.readUInt16BE(2);+ header = 4;+ } else if (length === 127) {+ if (pending.length < 10) break;+ const huge = pending.readBigUInt64BE(2);+ if (huge > 0xffff_ffffn) {+ wireError = "a frame too large to parse";+ return;+ }+ length = Number(huge);+ header = 10;+ }+ if (pending.length < header + length) break;+ if ((b0 & 0x0f) !== 0) {+ wire.push({+ rsv1: (b0 & 0x40) !== 0,+ opcode: b0 & 0x0f,+ length,+ payload: new Uint8Array(pending.subarray(header, header + length)),+ });+ }+ pending = pending.subarray(header + length);+ }+ });+ };+const client: StreamClient = {ws,frames,+ wire,outputText: () =>frames.filter((f) => f.opcode === TerminalStreamOpcode.Output)@@ -169,16 +248,20 @@ function attach(payload: JSON.stringify({ cols, rows, intent }),}),),+ sendPing: (payload) =>+ ws.send(encodeTerminalFrame({ opcode: TerminalStreamOpcode.Ping, payload })),waitFor: async (predicate, what, timeoutMs = 15_000) => {const deadline = Date.now() + timeoutMs;while (!predicate()) {if (undecodable > 0) throw new Error(`Received ${undecodable} undecodable frame(s).`);+ if (wireError !== null) throw new Error(wireError);if (Date.now() > deadline) {throw new Error(`Timed out waiting for ${what}. Frames: ${JSON.stringify(frames)}`);}await new Promise((resolve) => setTimeout(resolve, 25));}if (undecodable > 0) throw new Error(`Received ${undecodable} undecodable frame(s).`);+ if (wireError !== null) throw new Error(wireError);},close: () =>new Promise((resolve) => {@@ -189,7 +272,10 @@ function attach(};return new Promise((resolve, reject) => {- ws.once("open", () => resolve(client));+ ws.once("open", () => {+ if (options.deflate === true) watchWire();+ resolve(client);+ });ws.once("unexpected-response", (_req, res) =>reject(new Error(`handshake rejected: ${res.statusCode}`)),);@@ -317,16 +403,96 @@ describePty("terminal stream handshake", () => {});/**- * The line the backpressure flood is made of, and the marker its resync assertion reads.- * It fills every row of the screen for the whole burst, so it is there whichever instant- * the resync snapshot is taken at — unlike a marker printed once at the end of a round,- * which megabytes of scrolling carry off the screen again.+ * What the backpressure flood is made of: random bytes, not repeated ones.+ *+ * The stream is compressed on the wire (terminal/ws.ts), and repeated text leaves the+ * socket a hundred times smaller than it entered — a `yes`-style burst is delivered as fast+ * as a shell can produce it and never puts a viewer behind at all. base64 of /dev/urandom+ * is the cheapest output a shell can produce that compresses to nothing, so what the viewer+ * is behind on is real.+ */+const FLOOD_COMMAND = "head -c 4000000 /dev/urandom | base64";+/**+ * A screenful of that flood: base64 runs on two consecutive rows. The restore stream writes+ * the grid row by row, so this holds whichever instant the resync snapshot is taken at —+ * unlike a marker printed once per round, which megabytes of scrolling carry off the screen.*/-const FLOOD_LINE = "0123456789abcdef";+const FLOOD_ROWS = /[A-Za-z0-9+/]{40}\r\n[A-Za-z0-9+/]{40}/;/** Every Restore opens with this self-contained repaint (see snapshot.ts). */const RESTORE_PREAMBLE = "\x1b[0m\x1b[?1049l\x1b[H\x1b[2J\x1b[3J";+describePty("terminal stream heartbeat", () => {+ it(+ "echoes a client's probe, so a client can tell a live socket from a dead one",+ async () => {+ const terminal = await createTerminal();+ const client = await attach(terminal.id);+ try {+ client.sendPing("42");+ await client.waitFor(+ () => client.frames.some((f) => f.opcode === TerminalStreamOpcode.Pong),+ "pong",+ );++ const pong = client.frames.find((f) => f.opcode === TerminalStreamOpcode.Pong);+ // Echoed untouched: the payload is the CLIENT's clock reading, and only the client+ // can make a round trip out of it.+ expect(pong?.text).toBe("42");+ } finally {+ await client.close();+ await api.delete(`/api/terminals/${terminal.id}`);+ }+ },+ TEST_TIMEOUT,+ );+});++describePty("terminal stream compression", () => {+ it(+ "leaves small frames uncompressed and large ones compressed once permessage-deflate is negotiated",+ async () => {+ const terminal = await createTerminal();+ const client = await attach(terminal.id, { deflate: true });+ try {+ await client.waitFor(() => client.restoreText().length > 0, "attach restore");++ // A keystroke-sized burst: the command's echo, its output and the next prompt+ // together are far below the threshold, and on the wire it must arrive as-is —+ // RSV1 clear, the bytes decodable as a terminal frame with no inflate anywhere.+ const SMALL_MARKER = "zz-small-uncompressed-42";+ await runAndWait(client, `echo ${SMALL_MARKER}`, SMALL_MARKER);+ await client.waitFor(+ () =>+ client.wire.some((f) => {+ if (f.rsv1) return false;+ const frame = decodeTerminalFrame(f.payload);+ return (+ frame !== null &&+ frame.opcode === TerminalStreamOpcode.Output &&+ framePayloadText(frame).includes(SMALL_MARKER)+ );+ }),+ "the small frame on the wire uncompressed",+ );++ // A large burst of barely compressible bytes stays compressed: RSV1 set, and —+ // random base64 shrinks little — still far past the threshold on the wire.+ await runAndWait(+ client,+ "head -c 4000 /dev/urandom | base64; echo big-burst-done",+ "big-burst-done",+ );+ expect(client.wire.some((f) => f.rsv1 && f.length >= 512)).toBe(true);+ } finally {+ await client.close();+ await api.delete(`/api/terminals/${terminal.id}`);+ }+ },+ TEST_TIMEOUT,+ );+});+describePty("terminal stream backpressure", () => {it("resyncs a lagging viewer with a fresh Restore instead of disconnecting it", async () => {const terminal = await createTerminal();@@ -342,9 +508,9 @@ describePty("terminal stream backpressure", () => {client.ws.pause();// Flood until the server itself reports this viewer as lagging. A fixed burst is a// coin flip: a paused reader still lets the kernel absorb megabytes into its socket- // buffers (autotuned), so the server's own send queue — the thing the watermark- // measures — may never cross 1 MiB however much the shell wrote. This suite failed- // that way roughly one run in three, locally and on CI.+ // buffers (autotuned), so the viewer's backlog — the thing the watermark measures —+ // may never cross 1 MiB however much the shell wrote. This suite failed that way+ // roughly one run in three, locally and on CI.//// Only lines logged from here on count. `serverLogs` outlives one attempt and vitest// retries this file on macOS, so a second attempt reading the first one's line would@@ -353,29 +519,77 @@ describePty("terminal stream backpressure", () => {const loggedBefore = serverLogs.length;const isLagging = (): boolean =>serverLogs.slice(loggedBefore).some((l) => l.includes("pausing for resync"));+ let lastRound = 0;for (let round = 1; round <= 8 && !isLagging(); round += 1) {- client.sendInput(`yes ${FLOOD_LINE} | head -n 400000; echo BURST-DONE-${round}\r`);- await waitForCapture(terminal.id, `BURST-DONE-${round}`, 60_000);+ lastRound = round;+ // The pty echoes a command line the instant it is typed, so a literal marker would+ // satisfy this wait before the flood behind it had produced one byte. All eight+ // rounds would then go out back-to-back and land as one undifferentiated flood: the+ // kernel's autotuned buffers absorb far more than a single round's share, and+ // whichever way the watermark lands, the assert below races a stream that is still+ // running — the flake this suite was known for. The arithmetic is the fix: the echo+ // of "BURST-$((20+1))" matches nothing the wait accepts, so a round ends only once+ // the shell has actually run it — its burst fully landed server-side.+ client.sendInput(`${FLOOD_COMMAND}; echo BURST-$((20+${round}))\r`);+ await waitForCapture(terminal.id, `BURST-${20 + round}`, 60_000);}expect(isLagging(), "server never marked the paused viewer as lagging").toBe(true);// The burst has fully landed server-side, and the lagging viewer is still attached.expect(client.ws.readyState).toBe(WebSocket.OPEN);- // Catching up delivers a resync Restore, and the stream is live again afterwards.+ // Catching up delivers a resync Restore, and the stream is live again afterwards —+ // but which frame ends the cycle depends on the link, so the settle cannot wait for+ // a particular one. On a slow link the kernel absorbs the flood as fast as the pty+ // emits it: the resync fires mid-flood, no second high-water crossing follows, and+ // the round's end marker reaches the client as live output — no marker-bearing+ // Restore ever exists. What every timing has in common is the marker arriving at+ // the client, in whichever frame carries it: the marker is the last thing the round+ // produces, so once the client holds it, nothing of the flood is left in flight.+ const burstEnd = `BURST-${20 + lastRound}`;client.ws.resume();- const restores = (): StreamClient["frames"] =>- client.frames.filter((f) => f.opcode === TerminalStreamOpcode.Restore);- await client.waitFor(() => restores().length >= 2, "resync Restore frame", 60_000);- const resync = restores()[1]!;- // What a resync promises is the repaint, not a particular line of screen residue:- // the snapshot is taken whenever this viewer's socket drains past the low watermark,- // which the test does not get to choose and which may well be mid-flood. So assert- // the two things that hold at every such instant — it is the same self-contained- // repaint an attach sends, and it carries the flooded screen rather than the empty- // one the attach Restore captured.- expect(resync.text.slice(0, RESTORE_PREAMBLE.length)).toBe(RESTORE_PREAMBLE);- expect(resync.text).toContain(`${FLOOD_LINE}\r\n${FLOOD_LINE}`);+ await client.waitFor(+ () => client.frames.some((f) => f.text.includes(burstEnd)),+ "the flood's end to reach the client",+ // The flood's tail is the last thing through the backlog the paused socket+ // accumulated, and chewing through it is plain CPU — the slowest step of this+ // case on a starved runner, so the cap is deliberately generous.+ 90_000,+ );++ // One race is still open at that instant: a resync whose poll had not yet fired+ // repaints within one interval (250ms) of the client catching up, and a probe+ // typed before it would land in that drop window. The repaint arrives as a Restore+ // — a data frame — so a quiet window far longer than the poll, in which nothing+ // but the heartbeat arrives, proves none was pending: the pane is quiet, and the+ // live probe below is typed to an idle shell.+ let quietSince = 0;+ let quietFrames = -1;+ const streamQuiet = (): boolean => {+ const dataFrames = client.frames.filter(+ (f) => f.opcode !== TerminalStreamOpcode.Ping,+ ).length;+ if (dataFrames !== quietFrames) {+ quietFrames = dataFrames;+ quietSince = Date.now();+ }+ return Date.now() - quietSince >= 2_000;+ };+ await client.waitFor(streamQuiet, "a quiet stream after the flood", 30_000);++ // The resync this case exists to verify: the lagging gate above means the server+ // desynced this viewer, and the only way out of that state is the poll's fresh+ // Restore — so a repaint carrying the flooded pane must have arrived. It is not+ // necessarily the marker-bearing one (the mid-flood timing above legitimately+ // skips that); the flooded pane is the promise, and it identifies the repaint.+ const resync = client.frames.find(+ (f) => f.opcode === TerminalStreamOpcode.Restore && FLOOD_ROWS.test(f.text),+ );+ expect(resync, "a resync repaint of the flooded pane").toBeDefined();+ // What a resync promises is the repaint, not a particular line of screen residue —+ // it is the same self-contained repaint an attach sends, and it carries the flooded+ // screen rather than the empty one the attach Restore captured.+ expect(resync!.text.slice(0, RESTORE_PREAMBLE.length)).toBe(RESTORE_PREAMBLE);await runAndWait(client, "echo LIVE-$((40+2))", "LIVE-42");} finally {// Closed here rather than after the assertions: a socket a failed assertion left open@@ -386,7 +600,7 @@ describePty("terminal stream backpressure", () => {}// Megabytes through a pty, a paused socket and a drain: minutes of budget on a loaded// runner, where the default would report a timeout rather than a real defect.- }, 120_000);+ }, 180_000);});describePty("terminal stream attach", () => {diff --git a/packages/server/test/terminal.test.ts b/packages/server/test/terminal.test.tsindex 06df4b40..51f4b0ff 100644--- a/packages/server/test/terminal.test.ts+++ b/packages/server/test/terminal.test.ts@@ -17,6 +17,15 @@ import { expandHomePath } from "../src/terminal/session.js";import { repairSpawnHelpers } from "../src/terminal/spawn-helper.js";import { TerminalInputModeTracker } from "../src/terminal/input-mode.js";import { TerminalOutputCoalescer } from "../src/terminal/output-coalescer.js";+import {+ MAX_COALESCE_WINDOW_MS,+ MAX_HIGH_WATER_BYTES,+ MIN_COALESCE_WINDOW_MS,+ MIN_HIGH_WATER_BYTES,+ RoundTripEstimator,+ SocketDrainMeter,+ coalesceWindowFor,+} from "../src/terminal/link-quality.js";import { applyTerminalSize, releaseTerminalSize } from "../src/terminal/size-ownership.js";import { captureLines, renderRestoreAnsi } from "../src/terminal/snapshot.js";import { resolveKeys } from "../src/terminal/routes.js";@@ -253,6 +262,25 @@ describe("output coalescer", () => {expect(flushed).toEqual(["a", "bc"]);});+ it("reads its window per burst, so a link that slows down widens it", () => {+ vi.useFakeTimers();+ const flushed: string[] = [];+ let window = 5;+ const coalescer = new TerminalOutputCoalescer(+ (data) => flushed.push(data),+ () => window,+ );++ coalescer.push("a"); // leading edge+ window = 40; // the round trip grew between bursts+ coalescer.push("b");+ vi.advanceTimersByTime(5);+ expect(flushed).toEqual(["a"]); // still merging: the window is 40ms now+ vi.advanceTimersByTime(35);++ expect(flushed).toEqual(["a", "b"]);+ });+it("flushes pending output on demand so ordering is preserved", () => {vi.useFakeTimers();const flushed: string[] = [];@@ -266,6 +294,73 @@ describe("output coalescer", () => {});});+describe("link quality", () => {+ it("keeps the LAN window until a round trip is measured", () => {+ expect(coalesceWindowFor(null)).toBe(MIN_COALESCE_WINDOW_MS);+ expect(coalesceWindowFor(8)).toBe(MIN_COALESCE_WINDOW_MS);+ });++ it("widens the window with the round trip, up to the ceiling", () => {+ expect(coalesceWindowFor(80)).toBe(20);+ expect(coalesceWindowFor(200)).toBe(50);+ expect(coalesceWindowFor(2000)).toBe(MAX_COALESCE_WINDOW_MS);+ });++ it("smooths the round trip so one stalled reply cannot swing the window", () => {+ const rtt = new RoundTripEstimator();+ rtt.sample(200);+ rtt.sample(2000); // one reply behind a garbage-collecting tab++ expect(rtt.roundTripMs).toBeLessThan(700);+ expect(rtt.windowMs()).toBeLessThanOrEqual(MAX_COALESCE_WINDOW_MS);+ });++ it("ignores a round trip a clock change made negative", () => {+ const rtt = new RoundTripEstimator();+ rtt.sample(-5);++ expect(rtt.roundTripMs).toBeNull();+ });++ it("allows the full byte budget while the stream never waits on the socket", () => {+ let now = 0;+ const meter = new SocketDrainMeter(() => now);+ for (let i = 0; i < 10; i++) {+ now += 300;+ meter.note(1000, 0); // delivered, with nothing waiting behind it+ }++ expect(meter.bytesPerSecond).toBeNull();+ expect(meter.highWaterBytes()).toBe(MAX_HIGH_WATER_BYTES);+ });++ it("turns a slow delivery rate into a smaller allowance", () => {+ let now = 0;+ const meter = new SocketDrainMeter(() => now);+ // 25kB leaves every 500ms with a backlog behind it throughout: a 50kB/s link.+ for (let i = 0; i < 40; i++) {+ now += 500;+ meter.note(25_000, 400_000);+ }++ expect(meter.bytesPerSecond).toBeGreaterThan(40_000);+ expect(meter.bytesPerSecond).toBeLessThan(60_000);+ // A LAG bound now: well under the megabyte, which on this link would be twenty seconds.+ expect(meter.highWaterBytes()).toBe(MIN_HIGH_WATER_BYTES);+ });++ it("never drops the allowance below the floor the low-water mark needs", () => {+ let now = 0;+ const meter = new SocketDrainMeter(() => now);+ for (let i = 0; i < 20; i++) {+ now += 500;+ meter.note(10, 400_000); // a trickle, with the viewer far behind+ }++ expect(meter.highWaterBytes()).toBe(MIN_HIGH_WATER_BYTES);+ });+});+describe("frame codec", () => {it("round-trips an output frame", () => {const frame = decodeTerminalFrame(diff --git a/packages/web/src/features/terminal/terminal-connection.ts b/packages/web/src/features/terminal/terminal-connection.tsnew file mode 100644index 00000000..c432108d--- /dev/null+++ b/packages/web/src/features/terminal/terminal-connection.ts@@ -0,0 +1,336 @@+/**+ * The terminal socket's LIFE: the first attach, the heartbeat, and every reattach after a+ * link that dropped it. Kept out of terminal-view.tsx because none of it is about xterm,+ * and because a state machine with three timers is worth testing on its own.+ *+ * The reason it exists is a phone. On a mobile network the socket does not close politely:+ * the radio hands over, the carrier's NAT reaps an idle flow, the browser freezes the tab,+ * and what the page is left holding is a socket that will never carry another byte. Before+ * this, that state was terminal — `onclose` painted "stream closed" and the pane was dead+ * until the user reloaded the page — even though the pty on the server was fine and one+ * reattach would have repainted it from the server's own snapshot.+ *+ * So: probe for silence, retry with a bounded backoff, and treat a returning tab or a+ * returning network as a reason to try immediately rather than to wait out the timer.+ * A retry first asks whether the terminal is still there, because a gone terminal is the+ * one failure worth stopping on.+ */+import { TerminalOpcode, decodeFrame, encodeFrame, type DecodedFrame } from "./terminal-frames";++/**+ * How the client probes: one Ping per interval, and silence past the limit is a dead socket+ * — but only from a server that has ANSWERED a probe. A server from before the heartbeat+ * ignores the opcode, and an idle terminal on one is silent for perfectly good reasons; a+ * client that judged that silence would reattach every minute, snapshot and all, forever.+ */+const HEARTBEAT_INTERVAL_MS = 15_000;+const SILENCE_LIMIT_MS = 45_000;+/**+ * A tick this far late means the browser froze the tab's timers, not that the link died —+ * a backgrounded tab gets roughly one timer a minute. The silence it "measured" while+ * suspended says nothing, so that tick probes instead of dropping.+ */+const FROZEN_TICK_MS = HEARTBEAT_INTERVAL_MS * 3;+/** How long a wake-up probe waits for its echo before the socket is declared dead. */+const WAKE_PROBE_TIMEOUT_MS = 5_000;+/**+ * A dial that never reaches `onOpen` is given up on after this long. A phone's network can+ * black-hole a handshake without ever closing it — CONNECTING forever, no heartbeat started,+ * nothing to probe — so the dial carries its own clock: past it the socket is dropped like+ * any dead one and the backoff ladder tries again.+ */+export const DIAL_TIMEOUT_MS = 15_000;+/**+ * The same clock for the existence probe a retry waits on: one small request, so a link that+ * lets it hang would let it hang forever. Losing the race reads as the same transient+ * failure a rejected probe is.+ */+export const RECHECK_TIMEOUT_MS = 10_000;++/**+ * Backoff for reattaching, in milliseconds; the last entry repeats. It stays short because+ * the far end costs nothing to attach to (the pty is already running) and because the user+ * is usually staring at the pane waiting for it to come back.+ */+export const RETRY_DELAYS_MS = [500, 1_000, 2_000, 4_000, 8_000, 15_000];++export type TerminalConnectionStatus = "ready" | "reconnecting" | "error";++/** The socket, as this module needs it: three callbacks in, two verbs out. */+export interface SocketHandlers {+ onOpen(): void;+ onMessage(data: ArrayBuffer): void;+ onClose(): void;+}++export interface AttachedSocket {+ send(bytes: Uint8Array): void;+ close(): void;+}++export interface TerminalConnectionDeps {+ /**+ * The first attach: which terminal this view is for. Throwing is fatal — it is the host's+ * own policy (create a shell, honour a URL parameter) that failed, and repeating it could+ * create a second terminal.+ */+ first(): Promise<string>;+ /**+ * Before a retry: is that terminal still there? `false` stops the retries for good (the+ * shell exited and was reaped while the page was away); throwing is transient and retries.+ */+ recheck(id: string): Promise<boolean>;+ open(id: string, handlers: SocketHandlers): AttachedSocket;+ /** Everything that is not heartbeat traffic, in arrival order. */+ onFrame(frame: DecodedFrame): void;+ onStatus(status: TerminalConnectionStatus, detail: string): void;+}++export class TerminalConnection {+ private socket: AttachedSocket | null = null;+ private terminalId: string | null = null;+ private stopped = false;+ private attempt = 0;+ private retryTimer: ReturnType<typeof setTimeout> | null = null;+ private heartbeatTimer: ReturnType<typeof setInterval> | null = null;+ /** The dial's clock: armed when a socket is created, disarmed when it opens. */+ private dialTimer: ReturnType<typeof setTimeout> | null = null;+ private lastSeenAt = 0;+ private lastTickAt = 0;+ /** Whether the far end speaks the heartbeat (see the silence limit above). */+ private answersProbes = false;+ /**+ * True from the start of an attach to its socket: while `first()`/`recheck()` are pending.+ * Nothing may start a second attach meanwhile — the host policy behind `first()` creates a+ * terminal when it answers, and an answer that races a wake would create a second one.+ */+ private connecting = false;++ constructor(private readonly deps: TerminalConnectionDeps) {}++ start(): void {+ void this.connect();+ }++ /** Stops for good: the pane is gone, or the shell exited and there is nothing to reattach to. */+ stop(): void {+ this.stopped = true;+ this.clearRetry();+ this.stopHeartbeat();+ this.clearDial();+ this.socket?.close();+ this.socket = null;+ }++ /** True while a socket is open; false through every gap the user might type into. */+ get connected(): boolean {+ return this.socket !== null;+ }++ /** Sends one frame if there is a socket to send it on. Input typed into a gap is dropped. */+ send(bytes: Uint8Array): void {+ this.socket?.send(bytes);+ }++ /**+ * "The tab is back" / "the network is back". With no socket it collapses the backoff to+ * now; with one, it probes it — the socket a phone comes back to often looks open and is+ * not, and waiting for the next heartbeat to notice is fifteen seconds of dead pane.+ */+ wake(): void {+ if (this.stopped) return;+ if (this.socket === null) {+ // An attach already in flight IS the "right now" a wake asks for; letting a second+ // one start would run the host's policy twice (two terminals) or open two sockets+ // where the loser is never closed.+ if (this.connecting) return;+ this.clearRetry();+ this.attempt = 0;+ void this.connect();+ return;+ }+ const probed = this.socket;+ const askedAt = Date.now();+ this.lastSeenAt = askedAt;+ this.ping();+ if (!this.answersProbes) return; // nothing to conclude from an unanswered probe+ setTimeout(() => {+ if (this.socket !== probed || this.stopped) return;+ if (this.lastSeenAt > askedAt) return; // it answered+ this.drop(probed);+ }, WAKE_PROBE_TIMEOUT_MS);+ }++ private async connect(): Promise<void> {+ if (this.stopped || this.socket !== null || this.connecting) return;+ this.connecting = true;+ try {+ let id: string;+ try {+ if (this.terminalId === null) {+ // No clock on this one, on purpose: `first()` is the host's own policy —+ // repeating it because it is slow would create a second terminal, and failing it+ // on a timer would kill an attach a slow link is still completing.+ id = await this.deps.first();+ this.terminalId = id;+ } else {+ id = this.terminalId;+ const present = await this.within(this.deps.recheck(id), RECHECK_TIMEOUT_MS);+ if (this.stopped) return; // a stop that landed while the probe was out+ if (!present) {+ this.stopped = true;+ this.deps.onStatus("error", "terminal is gone");+ return;+ }+ }+ } catch (err) {+ if (this.stopped) return;+ const detail = err instanceof Error ? err.message : String(err);+ // A first attach is the host's policy failing, and is fatal. A recheck is just this+ // link being down again — the same thing the retry is for, and so is its timeout.+ if (this.terminalId === null) {+ this.stopped = true;+ this.deps.onStatus("error", detail);+ return;+ }+ this.deps.onStatus("reconnecting", detail);+ this.scheduleRetry();+ return;+ }+ if (this.stopped) return;++ const socket: AttachedSocket = this.deps.open(id, {+ onOpen: () => {+ if (this.socket !== socket) return;+ this.clearDial();+ this.attempt = 0;+ this.startHeartbeat();+ this.deps.onStatus("ready", "");+ },+ onMessage: (data) => {+ if (this.socket !== socket) return;+ this.lastSeenAt = Date.now();+ const frame = decodeFrame(data);+ if (!frame) return;+ // The heartbeat is this module's own traffic and never reaches the terminal.+ if (frame.opcode === TerminalOpcode.Ping) {+ socket.send(encodeFrame(TerminalOpcode.Pong, frame.text));+ return;+ }+ if (frame.opcode === TerminalOpcode.Pong) {+ this.answersProbes = true;+ return;+ }+ this.deps.onFrame(frame);+ },+ onClose: () => this.drop(socket),+ });+ this.socket = socket;+ // A new peer has answered nothing yet; what the previous socket answered says nothing+ // about this one — the reattach may have landed on an older server.+ this.answersProbes = false;+ this.dialTimer = setTimeout(() => {+ this.dialTimer = null;+ this.drop(socket);+ }, DIAL_TIMEOUT_MS);+ } finally {+ this.connecting = false;+ }+ }++ /** Gives up on one socket and starts the countdown to the next. */+ private drop(socket: AttachedSocket): void {+ if (this.socket !== socket) return;+ this.socket = null;+ this.stopHeartbeat();+ this.clearDial();+ socket.close();+ if (this.stopped) return;+ this.deps.onStatus("reconnecting", "");+ this.scheduleRetry();+ }++ private scheduleRetry(): void {+ if (this.retryTimer !== null || this.stopped) return;+ const step = Math.min(this.attempt, RETRY_DELAYS_MS.length - 1);+ const base = RETRY_DELAYS_MS[step] as number;+ this.attempt += 1;+ // Jitter, because a dock full of terminals loses its link at the same instant and+ // would otherwise reattach in one synchronised burst.+ const delay = base + Math.random() * base * 0.2;+ this.retryTimer = setTimeout(() => {+ this.retryTimer = null;+ void this.connect();+ }, delay);+ }++ private clearRetry(): void {+ if (this.retryTimer === null) return;+ clearTimeout(this.retryTimer);+ this.retryTimer = null;+ }++ private clearDial(): void {+ if (this.dialTimer === null) return;+ clearTimeout(this.dialTimer);+ this.dialTimer = null;+ }++ /**+ * Races a dependency call against a clock it cannot hang past. The losing call is left to+ * settle on its own — its late answer lands in a continuation that checks `stopped` and+ * the single-flight guard, so it changes nothing.+ */+ private within<T>(promise: Promise<T>, ms: number): Promise<T> {+ return new Promise<T>((resolve, reject) => {+ const timer = setTimeout(() => reject(new Error("the server did not answer in time")), ms);+ promise.then(+ (value) => {+ clearTimeout(timer);+ resolve(value);+ },+ (err: unknown) => {+ clearTimeout(timer);+ reject(err instanceof Error ? err : new Error(String(err)));+ },+ );+ });+ }++ private ping(): void {+ this.socket?.send(encodeFrame(TerminalOpcode.Ping, String(Date.now())));+ }++ private startHeartbeat(): void {+ this.stopHeartbeat();+ const now = Date.now();+ this.lastSeenAt = now;+ this.lastTickAt = now;+ this.heartbeatTimer = setInterval(() => this.tick(), HEARTBEAT_INTERVAL_MS);+ }++ private stopHeartbeat(): void {+ if (this.heartbeatTimer === null) return;+ clearInterval(this.heartbeatTimer);+ this.heartbeatTimer = null;+ }++ private tick(): void {+ const socket = this.socket;+ if (socket === null) return;+ const now = Date.now();+ const frozen = now - this.lastTickAt > FROZEN_TICK_MS;+ this.lastTickAt = now;+ if (frozen) {+ // Time the tab spent suspended is not silence on the wire; ask, do not judge.+ this.lastSeenAt = now;+ this.ping();+ return;+ }+ if (this.answersProbes && now - this.lastSeenAt > SILENCE_LIMIT_MS) {+ this.drop(socket);+ return;+ }+ this.ping();+ }+}diff --git a/packages/web/src/features/terminal/terminal-frames.ts b/packages/web/src/features/terminal/terminal-frames.tsindex 26727249..e33b5d2f 100644--- a/packages/web/src/features/terminal/terminal-frames.ts+++ b/packages/web/src/features/terminal/terminal-frames.ts@@ -4,6 +4,9 @@** Output is decoded straight into a string and handed to xterm — no JSON parse on the hot* path, which is what keeps a noisy build from stuttering the UI thread.+ *+ * Ping/Pong are symmetric: whoever probes sends its own clock reading and the peer echoes+ * it back untouched, so each end can decide for itself that a silent socket is dead.*/export const TerminalOpcode = {@@ -12,6 +15,10 @@ export const TerminalOpcode = {Resize: 0x03,Restore: 0x05,Exit: 0x06,+ /** Either direction: liveness probe; the peer echoes the payload back as Pong. */+ Ping: 0x07,+ /** Either direction: the echo of a Ping's payload, unchanged. */+ Pong: 0x08,} as const;export type TerminalOpcode = (typeof TerminalOpcode)[keyof typeof TerminalOpcode];diff --git a/packages/web/src/features/terminal/terminal-page.tsx b/packages/web/src/features/terminal/terminal-page.tsxindex 3eb5a2e2..732a383a 100644--- a/packages/web/src/features/terminal/terminal-page.tsx+++ b/packages/web/src/features/terminal/terminal-page.tsx@@ -208,7 +208,7 @@ export function TerminalPage() {className={`flex shrink-0 items-center gap-1 ${status === "ready"? chrome.success- : status === "connecting"+ : status === "connecting" || status === "reconnecting"? chrome.attention: chrome.danger}`}diff --git a/packages/web/src/features/terminal/terminal-view.tsx b/packages/web/src/features/terminal/terminal-view.tsxindex 49015efd..2a939856 100644--- a/packages/web/src/features/terminal/terminal-view.tsx+++ b/packages/web/src/features/terminal/terminal-view.tsx@@ -12,12 +12,16 @@* To restart with a different terminal, remount it (change the React `key`). On a touch* device the surface also carries the key bar (terminal-keybar.tsx), so both hosts get the* keys a soft keyboard lacks without either of them knowing about it.+ *+ * The socket itself is not managed here: terminal-connection.ts owns attaching, the+ * heartbeat and reattaching after a link drops, so this file stays about xterm.*/import { useEffect, useRef, useState } from "react";import "@xterm/xterm/css/xterm.css";// Types only (erased at compile time): the xterm runtime stays behind loadXterm() below.import type { ITheme, Terminal as XTerminal } from "@xterm/xterm";-import { TerminalOpcode, decodeFrame, encodeFrame, encodeResize } from "./terminal-frames";+import { TerminalOpcode, encodeFrame, encodeResize } from "./terminal-frames";+import { TerminalConnection } from "./terminal-connection";import { LinkClickTracker, openTerminalLink, positionFromPointer } from "./terminal-links";import { writeClipboard } from "../../lib/clipboard";import { TerminalKeyBar, type TerminalControl } from "./terminal-keybar";@@ -60,7 +64,18 @@ export interface TerminalInfo {title?: string | null;}-export type TerminalStatus = "connecting" | "ready" | "exited" | "error";+/**+ * How long the container has to hold still before its size is passed on. Long enough to+ * swallow a soft keyboard's animation, short enough that a deliberate drag lands promptly.+ */+const RESIZE_SETTLE_MS = 120;++/**+ * `reconnecting` is its own word on purpose: it is the state a mobile link spends most of+ * its bad minutes in, and it means something the other three do not — the pty is fine, this+ * page's socket is not, and it is being reattached (terminal-connection.ts).+ */+export type TerminalStatus = "connecting" | "ready" | "reconnecting" | "exited" | "error";/*** The screen's own palette, one per appearance. Two things matter here.@@ -311,7 +326,6 @@ export function TerminalView({async function startTerminal(container: HTMLDivElement): Promise<() => void> {const [{ Terminal }, { FitAddon }, { WebLinksAddon }, { ClipboardAddon }] = await loadXterm();let disposed = false;- let socket: WebSocket | null = null;let exited = false;const report = (status: TerminalStatus, detail = ""): void => {@@ -736,18 +750,12 @@ export function TerminalView({// another window attaches this one could never win its geometry back.container.addEventListener("focusin",- () => {- if (socket?.readyState === WebSocket.OPEN) {- socket.send(encodeResize(term.cols, term.rows, "claim"));- }- },+ () => connection.send(encodeResize(term.cols, term.rows, "claim")),{ signal },);const send = (data: string): void => {- if (socket?.readyState === WebSocket.OPEN) {- socket.send(encodeFrame(TerminalOpcode.Input, data));- }+ connection.send(encodeFrame(TerminalOpcode.Input, data));};term.onData((data) => {@@ -769,56 +777,100 @@ export function TerminalView({if (!disposed) callbacks.current.onTitle?.(title);});- void (async () => {- try {+ /**+ * One connection, however many sockets that takes (terminal-connection.ts): it owns+ * the first attach, the heartbeat and every reattach. This view only says what a+ * frame means and what a status looks like.+ */+ const connection = new TerminalConnection({+ first: async () => {const terminal = await callbacks.current.ensure(term.cols, term.rows);- if (disposed) return;callbacks.current.onInfo?.(terminal);-- socket = new WebSocket(streamUrl(terminal.id, term.cols, term.rows, userId));- socket.binaryType = "arraybuffer";-- socket.onopen = () => report("ready");- socket.onmessage = (event) => {- if (!(event.data instanceof ArrayBuffer)) return;- const frame = decodeFrame(event.data);- if (!frame) return;- switch (frame.opcode) {- // The restore stream is self-contained (reset + clear + repaint + cursor), so- // it is written like any other output; calling term.reset() here would race- // with xterm's parser instead.- case TerminalOpcode.Restore:- case TerminalOpcode.Output:- term.write(frame.text);- break;- case TerminalOpcode.Exit: {- const { exitCode } = JSON.parse(frame.text) as { exitCode: number };- exited = true;- report("exited", String(exitCode));- break;- }- default:- break;- }+ return terminal.id;+ },+ // A reattach that finds no terminal has nothing to come back to: the shell exited+ // and was reaped while this page was away.+ recheck: async (id) =>+ (await probeJson<TerminalInfo>(`/api/terminals/${encodeURIComponent(id)}`)) !== null,+ open: (id, handlers) => {+ const ws = new WebSocket(streamUrl(id, term.cols, term.rows, userId));+ ws.binaryType = "arraybuffer";+ ws.onopen = () => handlers.onOpen();+ ws.onmessage = (event) => {+ if (event.data instanceof ArrayBuffer) handlers.onMessage(event.data);};- socket.onerror = () => report("error", "stream error");- socket.onclose = () => {- if (!exited) report("error", "stream closed");+ // `error` is always followed by `close`; handling both would report twice.+ ws.onclose = () => handlers.onClose();+ return {+ send: (bytes) => {+ if (ws.readyState === WebSocket.OPEN) ws.send(bytes);+ },+ close: () => ws.close(),};- } catch (err) {- report("error", err instanceof Error ? err.message : String(err));- }- })();+ },+ onFrame: (frame) => {+ switch (frame.opcode) {+ // The restore stream is self-contained (reset + clear + repaint + cursor), so+ // it is written like any other output; calling term.reset() here would race+ // with xterm's parser instead.+ case TerminalOpcode.Restore:+ case TerminalOpcode.Output:+ term.write(frame.text);+ break;+ case TerminalOpcode.Exit: {+ const { exitCode } = JSON.parse(frame.text) as { exitCode: number };+ exited = true;+ connection.stop(); // an exited pty is not something to reattach to+ report("exited", String(exitCode));+ break;+ }+ default:+ break;+ }+ },+ onStatus: (status, detail) => {+ if (exited) return; // the exit already said what happened to this terminal+ report(status, detail);+ },+ });+ connection.start();++ // A network that comes back, or a tab that does, is a reason to reattach now instead+ // of sitting out the backoff — on a phone both happen all day. Waking also probes a+ // socket that merely LOOKS open, which is what a suspended tab usually comes back to.+ window.addEventListener("online", () => connection.wake(), { signal });+ document.addEventListener(+ "visibilitychange",+ () => {+ if (!document.hidden) connection.wake();+ },+ { signal },+ );// Geometry changes are `update`s: this connection claimed the size when it attached// (?cols/?rows on the stream URL, and again on focusin above); an update only// applies while this connection still holds ownership.+ // Settled, not per pixel. A phone resizes this container constantly — the address bar+ // collapses, the soft keyboard opens, the dock is dragged — and each size that reaches+ // the pty costs a SIGWINCH and a full repaint from whatever is running, which on a slow+ // link is the most expensive thing the terminal can do. So fit on the trailing edge,+ // and only tell the server when the GRID actually changed; most pixel changes do not+ // move it.+ let resizeTimer: ReturnType<typeof setTimeout> | null = null;+ let sentCols = term.cols;+ let sentRows = term.rows;const observer = new ResizeObserver(() => {if (disposed) return;- fit.fit();- if (socket?.readyState === WebSocket.OPEN) {- socket.send(encodeResize(term.cols, term.rows, "update"));- }+ if (resizeTimer !== null) clearTimeout(resizeTimer);+ resizeTimer = setTimeout(() => {+ resizeTimer = null;+ if (disposed) return;+ fit.fit();+ if (term.cols === sentCols && term.rows === sentRows) return;+ sentCols = term.cols;+ sentRows = term.rows;+ connection.send(encodeResize(term.cols, term.rows, "update"));+ }, RESIZE_SETTLE_MS);});observer.observe(container);term.focus();@@ -840,7 +892,8 @@ export function TerminalView({control.current = null;listenerAbort.abort();observer.disconnect();- socket?.close();+ if (resizeTimer !== null) clearTimeout(resizeTimer);+ connection.stop();term.dispose();};}diff --git a/packages/web/src/lib/strings-en.ts b/packages/web/src/lib/strings-en.tsindex 59519d1f..8221880f 100644--- a/packages/web/src/lib/strings-en.ts+++ b/packages/web/src/lib/strings-en.ts@@ -206,6 +206,8 @@ export const en: Strings = {status: {connecting: "connecting",ready: "ready",+ /** The pty is fine; this page's socket dropped and is being reattached. */+ reconnecting: "reconnecting",exited: "exited",error: "error",},diff --git a/packages/web/src/lib/strings.ts b/packages/web/src/lib/strings.tsindex fd0e18f2..96da8f13 100644--- a/packages/web/src/lib/strings.ts+++ b/packages/web/src/lib/strings.ts@@ -223,6 +223,8 @@ export const zh = {status: {connecting: "连接中",ready: "已连接",+ /** The pty is fine; this page's socket dropped and is being reattached. */+ reconnecting: "重连中",exited: "已退出",error: "连接错误",},diff --git a/packages/web/test/terminal-connection.test.ts b/packages/web/test/terminal-connection.test.tsnew file mode 100644index 00000000..5446623b--- /dev/null+++ b/packages/web/test/terminal-connection.test.ts@@ -0,0 +1,525 @@+/**+ * The terminal socket's life on a link that keeps dropping it (terminal-connection.ts).+ *+ * These are the promises the pane depends on when the network is a phone's: a dropped+ * socket comes back on its own, a suspended tab does not get mistaken for a dead link, a+ * terminal that is genuinely gone stops the retries, and the heartbeat answers whoever+ * probes it.+ */+import { afterEach, describe, expect, it, vi } from "vitest";+import {+ RETRY_DELAYS_MS,+ TerminalConnection,+ type AttachedSocket,+ type SocketHandlers,+ type TerminalConnectionStatus,+} from "../src/features/terminal/terminal-connection";+import { TerminalOpcode, decodeFrame, encodeFrame } from "../src/features/terminal/terminal-frames";++/** One fake socket: records what was sent, and lets a test play the link's part. */+class FakeSocket implements AttachedSocket {+ readonly sent: Uint8Array[] = [];+ closed = false;+ constructor(readonly handlers: SocketHandlers) {}+ send(bytes: Uint8Array): void {+ this.sent.push(bytes);+ }+ close(): void {+ this.closed = true;+ }+ /** The server accepted the upgrade. */+ open(): void {+ this.handlers.onOpen();+ }+ /** The server sent one frame. */+ deliver(opcode: TerminalOpcode, text = ""): void {+ this.handlers.onMessage(encodeFrame(opcode, text).buffer as ArrayBuffer);+ }+ /** The link went away. */+ drop(): void {+ this.handlers.onClose();+ }+ opcodes(): number[] {+ return this.sent.map((bytes) => bytes[0] as number);+ }+}++interface Harness {+ connection: TerminalConnection;+ sockets: FakeSocket[];+ statuses: { status: TerminalConnectionStatus; detail: string }[];+ frames: number[];+ /** Whether a retry finds the terminal still on the server. */+ setPresent(present: boolean): void;+}++function harness(+ options: { first?: () => Promise<string>; recheck?: (id: string) => Promise<boolean> } = {},+): Harness {+ const sockets: FakeSocket[] = [];+ const statuses: Harness["statuses"] = [];+ const frames: number[] = [];+ let present = true;+ const connection = new TerminalConnection({+ first: options.first ?? (() => Promise.resolve("t1")),+ recheck: options.recheck ?? (() => Promise.resolve(present)),+ open: (_id, handlers) => {+ const socket = new FakeSocket(handlers);+ sockets.push(socket);+ return socket;+ },+ onFrame: (frame) => frames.push(frame.opcode),+ onStatus: (status, detail) => statuses.push({ status, detail }),+ });+ return {+ connection,+ sockets,+ statuses,+ frames,+ setPresent: (next) => {+ present = next;+ },+ };+}++/** Lets the connection's own promises settle between timer steps. */+const settle = () => vi.advanceTimersByTimeAsync(0);++/** A promise the test settles by hand — the link that answers when it feels like it. */+interface Deferred<T> {+ promise: Promise<T>;+ resolve: (value: T) => void;+ reject: (err: unknown) => void;+}++function deferred<T>(): Deferred<T> {+ let resolve!: (value: T) => void;+ let reject!: (err: unknown) => void;+ const promise = new Promise<T>((res, rej) => {+ resolve = res;+ reject = rej;+ });+ return { promise, resolve, reject };+}++afterEach(() => vi.useRealTimers());++describe("terminal connection", () => {+ it("reports ready once the socket opens", async () => {+ vi.useFakeTimers();+ const h = harness();+ h.connection.start();+ await settle();+ h.sockets[0]?.open();++ expect(h.statuses).toEqual([{ status: "ready", detail: "" }]);+ });++ it("reattaches after the link drops, and says so meanwhile", async () => {+ vi.useFakeTimers();+ const h = harness();+ h.connection.start();+ await settle();+ h.sockets[0]?.open();+ h.sockets[0]?.drop();++ expect(h.statuses.at(-1)?.status).toBe("reconnecting");+ expect(h.sockets).toHaveLength(1); // the retry is a wait, not a hot loop++ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[0] as number) * 1.2);+ expect(h.sockets).toHaveLength(2);+ h.sockets[1]?.open();+ expect(h.statuses.at(-1)?.status).toBe("ready");+ });++ it("backs off further with each failed attempt", async () => {+ vi.useFakeTimers();+ const h = harness();+ h.connection.start();+ await settle();+ h.sockets[0]?.open();+ h.sockets[0]?.drop();++ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[0] as number) * 1.2);+ h.sockets[1]?.drop(); // the second attach never opened+ await vi.advanceTimersByTimeAsync(RETRY_DELAYS_MS[0] as number);+ expect(h.sockets).toHaveLength(2); // the first delay is no longer enough++ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[1] as number) * 1.2);+ expect(h.sockets).toHaveLength(3);+ });++ it("stops retrying when the terminal is gone", async () => {+ vi.useFakeTimers();+ const h = harness();+ h.connection.start();+ await settle();+ h.sockets[0]?.open();+ h.setPresent(false);+ h.sockets[0]?.drop();++ await vi.advanceTimersByTimeAsync(60_000);++ expect(h.sockets).toHaveLength(1);+ expect(h.statuses.at(-1)).toEqual({ status: "error", detail: "terminal is gone" });+ });++ it("keeps the first attach fatal: the host's policy is not retried", async () => {+ vi.useFakeTimers();+ const h = harness({ first: () => Promise.reject(new Error("no terminal API")) });+ h.connection.start();+ await settle();+ await vi.advanceTimersByTimeAsync(60_000);++ expect(h.sockets).toHaveLength(0);+ expect(h.statuses.at(-1)).toEqual({ status: "error", detail: "no terminal API" });+ });++ it("answers the server's probe and keeps it off the terminal", async () => {+ vi.useFakeTimers();+ const h = harness();+ h.connection.start();+ await settle();+ const socket = h.sockets[0] as FakeSocket;+ socket.open();+ socket.deliver(TerminalOpcode.Ping, "1234");++ const reply = socket.sent.at(-1) as Uint8Array;+ expect(reply[0]).toBe(TerminalOpcode.Pong);+ expect(decodeFrame(reply.buffer as ArrayBuffer)?.text).toBe("1234");+ expect(h.frames).toEqual([]); // heartbeat traffic never reaches xterm+ });++ it("passes output through to the terminal", async () => {+ vi.useFakeTimers();+ const h = harness();+ h.connection.start();+ await settle();+ const socket = h.sockets[0] as FakeSocket;+ socket.open();+ socket.deliver(TerminalOpcode.Output, "hello");++ expect(h.frames).toEqual([TerminalOpcode.Output]);+ });++ it("drops a socket that stopped answering, and reattaches", async () => {+ vi.useFakeTimers();+ const h = harness();+ h.connection.start();+ await settle();+ const socket = h.sockets[0] as FakeSocket;+ socket.open();+ socket.deliver(TerminalOpcode.Pong, "1"); // this server speaks the heartbeat++ // Probe after probe goes unanswered; the tick that finds the silence past the limit+ // gives up on the socket instead of probing again.+ await vi.advanceTimersByTimeAsync(65_000);++ expect(socket.opcodes()).toContain(TerminalOpcode.Ping);+ expect(socket.closed).toBe(true);+ expect(h.statuses.at(-1)?.status).toBe("reconnecting");+ });++ it("never judges the silence of a server that does not answer probes", async () => {+ vi.useFakeTimers();+ const h = harness();+ h.connection.start();+ await settle();+ const socket = h.sockets[0] as FakeSocket;+ socket.open(); // a server from before the heartbeat: it ignores the opcode++ await vi.advanceTimersByTimeAsync(300_000);+ h.connection.wake();+ await vi.advanceTimersByTimeAsync(30_000);++ // An idle terminal on such a server is quiet for good reasons, and reattaching every+ // minute would repaint the screen from a snapshot for nothing.+ expect(socket.closed).toBe(false);+ expect(h.sockets).toHaveLength(1);+ });++ it("treats a frozen tab as a wake-up, not as a dead link", async () => {+ vi.useFakeTimers();+ const h = harness();+ h.connection.start();+ await settle();+ const socket = h.sockets[0] as FakeSocket;+ socket.open();+ // This server answers probes, so its silence IS evidence — without this Pong the+ // silence rule is never armed and the frozen tick below judges nothing at all.+ socket.deliver(TerminalOpcode.Pong, "1");++ // The browser suspended this tab: one tick fires, an hour late.+ vi.setSystemTime(Date.now() + 3_600_000);+ await vi.advanceTimersByTimeAsync(15_000);++ expect(socket.closed).toBe(false);+ expect(socket.opcodes().at(-1)).toBe(TerminalOpcode.Ping); // it asks instead of judging++ // Unfrozen again, silence past the limit is still a dead socket — the frozen tick is an+ // exemption for time the tab spent suspended, not a switch-off of the silence rule.+ await vi.advanceTimersByTimeAsync(65_000);+ expect(socket.closed).toBe(true);+ expect(h.statuses.at(-1)?.status).toBe("reconnecting");+ });++ it("wakes into an immediate reattach instead of sitting out the backoff", async () => {+ vi.useFakeTimers();+ const h = harness();+ h.connection.start();+ await settle();+ h.sockets[0]?.open();+ h.sockets[0]?.drop();++ h.connection.wake();+ await settle();++ expect(h.sockets).toHaveLength(2);+ });++ it("probes a socket that only looks open when the tab comes back", async () => {+ vi.useFakeTimers();+ const h = harness();+ h.connection.start();+ await settle();+ const socket = h.sockets[0] as FakeSocket;+ socket.open();+ socket.deliver(TerminalOpcode.Pong, "1"); // this server speaks the heartbeat++ h.connection.wake();+ expect(socket.opcodes().at(-1)).toBe(TerminalOpcode.Ping);++ await vi.advanceTimersByTimeAsync(6_000); // no echo came back+ expect(socket.closed).toBe(true);+ expect(h.statuses.at(-1)?.status).toBe("reconnecting");+ });++ it("keeps a socket that answers the wake-up probe", async () => {+ vi.useFakeTimers();+ const h = harness();+ h.connection.start();+ await settle();+ const socket = h.sockets[0] as FakeSocket;+ socket.open();+ // Arm the deadline first: without a Pong the wake installs no 5-second timer, and+ // "still open" below would say nothing about the answer cancelling it.+ socket.deliver(TerminalOpcode.Pong, "1");++ h.connection.wake();+ expect(socket.opcodes().at(-1)).toBe(TerminalOpcode.Ping);++ // The echo arrives within the deadline, the drop it scheduled is cancelled…+ await vi.advanceTimersByTimeAsync(1_000);+ socket.deliver(TerminalOpcode.Pong, "1");+ // …and the socket is still here well past the deadline.+ await vi.advanceTimersByTimeAsync(6_000);++ expect(socket.closed).toBe(false);+ });++ it("drops input typed into a gap rather than replaying it later", async () => {+ vi.useFakeTimers();+ const h = harness();+ h.connection.start();+ await settle();+ h.sockets[0]?.open();+ h.sockets[0]?.drop();++ h.connection.send(encodeFrame(TerminalOpcode.Input, "rm -rf /\r"));+ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[0] as number) * 1.2);+ h.sockets[1]?.open();++ expect(h.sockets[1]?.sent).toEqual([]);+ });++ it("stops for good once the pane is gone", async () => {+ vi.useFakeTimers();+ const h = harness();+ h.connection.start();+ await settle();+ h.sockets[0]?.open();+ h.connection.stop();++ expect(h.sockets[0]?.closed).toBe(true);+ await vi.advanceTimersByTimeAsync(60_000);+ expect(h.sockets).toHaveLength(1);+ });++ it("keeps one first attach in flight, however often the network returns during it", async () => {+ vi.useFakeTimers();+ const pending = deferred<string>();+ let firstCalls = 0;+ const h = harness({+ first: () => {+ firstCalls += 1;+ return pending.promise;+ },+ });++ h.connection.start();+ await settle();+ // online and visibilitychange fire repeatedly while the first attach is still out.+ h.connection.wake();+ h.connection.wake();+ await settle();+ expect(firstCalls).toBe(1); // the host policy runs once; twice could mean two terminals++ pending.resolve("t1");+ await settle();+ expect(h.sockets).toHaveLength(1); // one socket, not one per wake+ h.sockets[0]?.open();+ expect(h.statuses.at(-1)).toEqual({ status: "ready", detail: "" });+ });++ it("keeps one recheck in flight: a wake during a pending retry opens no second socket", async () => {+ vi.useFakeTimers();+ const pending = deferred<boolean>();+ let rechecks = 0;+ const h = harness({+ recheck: () => {+ rechecks += 1;+ return pending.promise;+ },+ });+ h.connection.start();+ await settle();+ h.sockets[0]?.open();+ h.sockets[0]?.drop();++ // The first retry's recheck is out; the tab comes back twice before it answers.+ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[0] as number) * 1.2);+ expect(rechecks).toBe(1);+ h.connection.wake();+ h.connection.wake();+ await settle();+ expect(rechecks).toBe(1);++ pending.resolve(true);+ await settle();+ expect(h.sockets).toHaveLength(2); // exactly one reattach, however many wakes+ h.sockets[1]?.open();+ expect(h.statuses.at(-1)?.status).toBe("ready");+ });++ it("gives up on a dial that never opens, closes it, retries — and ignores its late open", async () => {+ vi.useFakeTimers();+ const h = harness();+ h.connection.start();+ await settle();+ // The link swallows the handshake: no onOpen, no onClose, CONNECTING forever.+ const stalled = h.sockets[0] as FakeSocket;++ // 15s: DIAL_TIMEOUT_MS in the source, written as a literal so this case also runs+ // (red) on the tree before the fix, where the export does not exist yet.+ await vi.advanceTimersByTimeAsync(15_000);++ expect(stalled.closed).toBe(true);+ expect(h.statuses.at(-1)?.status).toBe("reconnecting");++ // The backoff ladder takes it from here…+ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[0] as number) * 1.2);+ expect(h.sockets).toHaveLength(2);++ // …and a handshake that completes AFTER its dial was given up on changes nothing: the+ // late open is not this connection's socket any more, so it is ignored — and the+ // fresh dial is the one that can go ready.+ stalled.open();+ const fresh = h.sockets[1] as FakeSocket;+ expect(fresh.closed).toBe(false);+ expect(h.statuses.filter((s) => s.status === "ready")).toHaveLength(0);++ fresh.open();+ expect(h.statuses.at(-1)).toEqual({ status: "ready", detail: "" });+ });++ it("gives up on a recheck that never answers, and tries it again", async () => {+ vi.useFakeTimers();+ let rechecks = 0;+ const h = harness({+ recheck: () => {+ rechecks += 1;+ return new Promise<boolean>(() => {});+ },+ });+ h.connection.start();+ await settle();+ h.sockets[0]?.open();+ h.sockets[0]?.drop();++ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[0] as number) * 1.2); // recheck #1 is out+ const statusesBefore = h.statuses.length;+ // 10s: RECHECK_TIMEOUT_MS in the source, a literal for the same reason as above.+ await vi.advanceTimersByTimeAsync(10_000); // …and it never answers++ expect(h.statuses.length).toBe(statusesBefore + 1); // a NEW status, not the drop's old one+ expect(h.statuses.at(-1)?.status).toBe("reconnecting");++ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[1] as number) * 1.2);+ expect(rechecks).toBe(2); // the hang read as a transient failure, so the ladder went on+ });++ it("ends a pending recheck quietly when the pane is gone: nothing follows the stop", async () => {+ vi.useFakeTimers();+ const pending = deferred<boolean>();+ const h = harness({ recheck: () => pending.promise });+ h.connection.start();+ await settle();+ h.sockets[0]?.open();+ h.sockets[0]?.drop();++ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[0] as number) * 1.2); // recheck is out+ const statusesAtStop = h.statuses.length;+ h.connection.stop();+ pending.reject(new Error("the link died mid-probe"));++ await vi.advanceTimersByTimeAsync(60_000);++ expect(h.statuses.length).toBe(statusesAtStop); // no reconnecting after the stop+ expect(h.sockets).toHaveLength(1);+ expect(h.sockets[0]?.closed).toBe(true);+ });++ it("judges a fresh socket by its own answers, not the previous one's", async () => {+ vi.useFakeTimers();+ const h = harness();+ h.connection.start();+ await settle();+ const first = h.sockets[0] as FakeSocket;+ first.open();+ first.deliver(TerminalOpcode.Pong, "1"); // this server speaks the heartbeat++ first.drop();+ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[0] as number) * 1.2);+ const second = h.sockets[1] as FakeSocket;+ second.open(); // this one never answers a probe: maybe an older server came back++ await vi.advanceTimersByTimeAsync(300_000);+ h.connection.wake();+ await vi.advanceTimersByTimeAsync(30_000);++ // The previous socket's answers are not this socket's. An old server's silence is+ // legitimate, so it survives both the silence rule and the wake.+ expect(second.closed).toBe(false);+ expect(h.sockets).toHaveLength(2);+ });++ it("still judges a fresh socket that answers by its silence", async () => {+ vi.useFakeTimers();+ const h = harness();+ h.connection.start();+ await settle();+ const first = h.sockets[0] as FakeSocket;+ first.open();+ first.deliver(TerminalOpcode.Pong, "1");+ first.drop();++ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[0] as number) * 1.2);+ const second = h.sockets[1] as FakeSocket;+ second.open();+ second.deliver(TerminalOpcode.Pong, "1"); // THIS one speaks the heartbeat too++ await vi.advanceTimersByTimeAsync(300_000);+ expect(second.closed).toBe(true); // its own silence, once it has answered+ expect(h.sockets.at(-1)).not.toBe(second); // the connection moved on+ });+});