Skip to content

A local-first multi-agent app development platform, built on PenguinHarness

feat(server,web): 终端在移动网络下不再假死,断了会自己重连

lmlihengbe87f4d3f396
diff --git a/changelog/unreleased/2026-11-02-terminal-on-a-slow-link.md b/changelog/unreleased/2026-11-02-terminal-on-a-slow-link.md
new file mode 100644
index 00000000..6d8d75c8
--- /dev/null
+++ b/changelog/unreleased/2026-11-02-terminal-on-a-slow-link.md
@@ -0,0 +1,31 @@
+# The terminal survives a mobile network, and reattaches on its own
+
+- **Date:** 2026-11-02
+- **Type:** feat
+- **Scope:** `server`, `web`
+- **Breaking:** no
+
+[中文版](2026-11-02-terminal-on-a-slow-link.zh.md)
+
+A terminal opened over a phone's network used to die quietly. The socket does not close politely there — the radio hands over, a carrier NAT reaps an idle flow, the browser freezes the tab — and what the page was left holding was a connection that would never carry another byte, painted as "stream closed". The pty on the server was fine the whole time; a reload was the only way back to it.
+
+Both ends now probe each other. Either side may send a Ping and the peer echoes it back untouched, so a socket that stopped carrying bytes is found within a bounded silence window rather than never. The server closes a viewer that stops answering; the client reattaches after one that stops answering it, with a bounded backoff, and the reattach repaints the screen from the server's own snapshot — the same repaint a fresh attach gets. A network coming back, or a tab coming back, collapses that wait: the connection retries at once, and a socket that merely *looks* open is probed before it is trusted. A tab that was suspended is not mistaken for a dead link. The one failure worth stopping on is a terminal that is genuinely gone, which a retry checks for before it reconnects. The status word for all of this is `reconnecting`, distinct from the `connecting` of a first attach.
+
+Ported from PenguinHarness `#639` (`f52617695`), by applying its changes rather than merging branches: all sixteen of its files exist here and `git apply -3 --check` was clean. Nothing this change touches is missing from this tree (the machine-side terminal shells and `terminal-panel.tsx` are outside it), so there were no trade-offs to record.
+
+## Details
+
+Every step of the reattach carries its own clock, and only one attempt is ever in flight. A network or a tab that comes back while an attach is still pending joins that attempt instead of starting another: the first-attach policy runs once (twice could mean two terminals), one socket is open at a time, and an answer that arrives after its attempt was given up on opens nothing — the old socket is closed before the next one is dialled. A dial that never reaches the open state (15 seconds) and an existence probe that never answers (10 seconds) read as the same transient failure a rejection does, and go to the backoff; a pane that goes away stops even a probe that is still out. The right to be judged by silence belongs to the socket that earned it: probe answers do not carry over a reattach, so a fresh socket onto an older server is never killed by the previous one's answers.
+
+Three numbers stopped being constants tuned on a LAN:
+
+- **The output merge window** followed the round trip. At 200ms it merged almost nothing while producing hundreds of small frames a second; it is now a quarter of the measured trip, bounded to 5–50ms, so a fast link keeps the behaviour it had.
+- **The backpressure high-water mark** was a flat megabyte, which is milliseconds of lag on a LAN and many seconds on a slow uplink — the "output arrives late, then jumps" complaint. It is now derived from what the viewer's socket actually delivers, so it bounds *time* (three quarters of a second) instead of bytes.
+- **Container resizes** reached the pty one per pixel change. On a phone the address bar collapses and the soft keyboard opens constantly, and every size that lands costs a SIGWINCH and a full repaint from whatever is running. They settle for 120ms now, and only a change that moves the grid is sent at all.
+
+The stream is also compressed on the wire (permessage-deflate, a 13-bit window and no client context takeover), and the small frames are left out of it by the sender itself, message by message. ws honours a config-level `threshold` only for a direction that negotiated away its compression context; this stream keeps its context — a shared context is what compresses repeated output best — so the 512-byte cutoff is decided at the send call, per message. The backlog is counted in the bytes the terminal *produced* (ws's own send callbacks) rather than in `bufferedAmount`: once compressed, that is wire bytes, and a screenful of repeated build output is a hundred times smaller there and would read as "not behind at all".
+
+## Compatibility
+
+- Both ends can be older than this change: a hot push replaces the platform under a tab that was not reloaded, and a machine runs whatever version it was last installed with. Unknown frame types were already harmless in both directions; the hazard is the conclusion each end draws from silence, so both sides share one rule with no version negotiation: **only a peer that has answered a probe is ever judged for its silence** (one `answersProbes` per socket, cleared each time a socket opens). Without it, a server would close an old page every 75 seconds (and that page has no reattach), and this page would reattach to an old server roughly every minute.
+- No API or dependency change; the two new frame types live only in the server's and the Web's own frame definitions.
diff --git a/changelog/unreleased/2026-11-02-terminal-on-a-slow-link.zh.md b/changelog/unreleased/2026-11-02-terminal-on-a-slow-link.zh.md
new file mode 100644
index 00000000..b7ce8bdf
--- /dev/null
+++ b/changelog/unreleased/2026-11-02-terminal-on-a-slow-link.zh.md
@@ -0,0 +1,57 @@
+# 终端在移动网络下不再假死,自己重连
+
+- **Date:** 2026-11-02
+- **Type:** feat
+- **Scope:** `server`, `web`
+- **Breaking:** no
+
+[English](2026-11-02-terminal-on-a-slow-link.md)
+
+在手机网络上打开的终端,过去是悄无声息地死掉的。那种链路上的 socket 不会礼貌地关闭——基站切换、
+运营商 NAT 回收空闲连接、浏览器冻结后台标签页——页面手里剩下的是一条再也不会传输任何字节的连接,
+界面上写着「stream closed」。而服务端的 pty 自始至终好好的,唯一的回去方式是刷新页面。
+
+现在两端互相探测:任意一端都可以发 Ping,对端原样回 Pong,于是一条不再传输字节的 socket 会在有界的
+沉默窗口内被发现,而不是永远发现不了。服务端关闭不再应答的观察者;客户端在对方不再应答时按有界退避
+重新接入,重连后由服务端的快照重绘整屏——与首次接入拿到的是同一份重绘。网络恢复或标签页回到前台会
+立即结束等待:连接马上重试,而看起来还开着的 socket 会先被探测再被信任;被浏览器挂起过的标签页不会
+被误判成死链路。唯一值得停下来的失败是终端真的不在了,重连前会先确认这一点。这一整套状态在界面上叫
+`reconnecting`,与首次接入的 `connecting` 区分开。
+
+移植自 PenguinHarness `#639`(`f52617695`),照改动落、不是合分支:那一笔的 16 个文件本树全有,
+`git apply -3 --check` 干净、一个字没冲突;本树没有的东西(机器侧的终端外壳、`terminal-panel.tsx` 那套
+面板)不在这一笔的范围里,因此没有取舍需要记。
+
+## 细节
+
+重连的每一步都自带期限,同一时刻至多一次尝试在途。网络或标签页在一次接入还没结果时回来,加入的就是
+这次在途的尝试,而不是再起一次:首次接入的宿主策略只跑一遍(跑两遍可能意味着两个终端),任何时刻只有
+一条 socket,被放弃的尝试迟来的结果什么也不会打开——拨下一条之前,旧的 socket 已经关掉。永远到不了
+open 状态的拨号(15 秒)与永远不回话的存在探测(10 秒),读作与被拒相同的暂态失败,交回退避阶梯;
+面板消失时,连还在途的探测也一并停掉。被沉默判决的资格属于挣得它的那条 socket:探测的应答不跨重连
+继承,重连落到一台从不应答的旧服务端上,不会被上一条的应答误杀。
+
+三个按局域网调出来的常量不再是常量:
+
+- **输出合并窗口**跟随往返时延。200ms 的链路上,固定 5ms 几乎合并不了任何东西,却每秒产生几百个小帧;
+ 现在取实测往返的四分之一,限制在 5–50ms,快链路上行为不变。
+- **背压高水位**原本是固定的 1MB,在局域网上是几毫秒的滞后,在慢上行链路上却是好几秒——正是「输出迟迟
+ 不来、然后猛地跳过去」的来源。现在它由该观察者 socket 实际投递的速率推出,界定的是**时间**(0.75 秒)
+ 而不是字节数。
+- **容器尺寸变化**过去每次像素变动都会传到 pty。手机上地址栏收起、软键盘弹出会不断触发,而每一次尺寸
+ 落地都意味着一次 SIGWINCH 与运行中程序的整屏重绘。现在它们先静置 120ms,并且只有真正改变了行列数的
+ 变化才会发出去。
+
+流本身也在链路上压缩了(permessage-deflate,限定 13 位窗口、不保留客户端上下文);小帧不压由发送方
+逐条决定。ws 只对协商放弃压缩上下文的一侧应用配置级的 `threshold`,而这条流保留自己的上下文——共享
+上下文正是重复输出压得最好的原因——512 字节的分界因此落在发送调用上、逐帧判定。背压的积压量也随之
+改成按「终端产出了多少字节」来数(ws 的发送回调),不再看 `bufferedAmount`:压缩之后它是线上字节,
+一屏重复的构建输出在线上小一百倍,会被读成「没落后」。
+
+## 兼容性
+
+- 两端都可能是旧的:热更新会把平台换在一个没刷新的标签页下面,一台机器跑的是它上次安装的版本。
+ 两个方向的未知帧类型本来就无害,真正的风险是各自从沉默里得出的结论,因此两侧共用一条规则、不做版本
+ 协商:**只有应答过探测的对端才会被沉默判决**(每侧一个 `answersProbes`,每次 socket 打开时清掉)。
+ 没有这条,服务端会每 75 秒关掉一个旧页面(而它没有重连),这个页面也会约每分钟重连一次到旧服务端。
+- 没有接口或依赖变化;两个新帧类型只写在服务端与 Web 各自的帧定义里。
diff --git a/packages/server/src/terminal/frames.ts b/packages/server/src/terminal/frames.ts
index 9ca4a516..6c2d0dca 100644
--- a/packages/server/src/terminal/frames.ts
+++ b/packages/server/src/terminal/frames.ts
@@ -11,6 +11,12 @@
* byte 1 slot (stream index within one socket)
* byte 2+ payload (raw UTF-8 terminal bytes, or JSON for the control opcodes)
*
+ * Ping/Pong are symmetric on purpose: each end probes for itself. A mobile link drops a
+ * socket without closing it (a radio hand-off, a carrier NAT reaping an idle flow), and a
+ * peer that only answers probes never finds out. The payload is the prober's own clock
+ * reading, echoed verbatim, so the round trip falls out of the reply — that measurement is
+ * what sizes the output merge window (link-quality.ts).
+ *
* `slot` is always 0 today — this server opens one socket per terminal. It is in the wire
* format from the start because the alternative (adding it later) is a protocol break, and
* because multiplexing every terminal of a page onto one socket is the obvious next step.
@@ -27,6 +33,10 @@ export const TerminalStreamOpcode = {
Restore: 0x05,
/** Server -> client: the pty exited, JSON `{exitCode, signal}`. */
Exit: 0x06,
+ /** Either direction: liveness probe. The peer echoes the payload back as Pong. */
+ Ping: 0x07,
+ /** Either direction: the echo of a Ping's payload, unchanged. */
+ Pong: 0x08,
} as const;
export type TerminalStreamOpcode = (typeof TerminalStreamOpcode)[keyof typeof TerminalStreamOpcode];
diff --git a/packages/server/src/terminal/link-quality.ts b/packages/server/src/terminal/link-quality.ts
new file mode 100644
index 00000000..036f3282
--- /dev/null
+++ b/packages/server/src/terminal/link-quality.ts
@@ -0,0 +1,132 @@
+/**
+ * What the stream knows about the link it is writing to: how long a round trip takes, and
+ * how fast the viewer's socket actually drains.
+ *
+ * Both numbers exist because the transport's two constants were tuned on a link with no
+ * latency and no bandwidth limit, and both of them mean something different on a phone:
+ *
+ * - The output merge window (output-coalescer.ts) was a flat 5ms. At a 200ms round trip
+ * that window merges almost nothing while producing hundreds of small frames a second —
+ * pure overhead on a radio, and no faster to arrive, because the round trip dominates by
+ * two orders of magnitude. Scaled to the measured trip it merges properly, and on a fast
+ * link it stays where it was.
+ * - The backpressure high-water mark was a flat 1MB. A byte bound is a LAG bound only if
+ * you know the rate: 1MB is a few milliseconds behind on a LAN and the better part of ten
+ * seconds behind on a slow uplink — which is exactly the "output arrives late, then jumps"
+ * complaint. Measured delivery turns it back into the time bound it was meant to be.
+ *
+ * Both estimators are deliberately conservative: with no measurement yet, they answer with
+ * the old constants, so a link that never reports stays on the behaviour it always had.
+ */
+
+/** Smoothing for both estimators: the new sample weighs a fifth. */
+const ALPHA = 0.2;
+
+export const MIN_COALESCE_WINDOW_MS = 5;
+export const MAX_COALESCE_WINDOW_MS = 50;
+
+/**
+ * The merge window for a measured round trip: a quarter of it, bounded.
+ *
+ * A quarter keeps the window an order of magnitude below the delay the user is already
+ * paying, so merging is invisible to them, while the ceiling keeps a satellite-grade link
+ * from batching output into visible chunks. `null` (nothing measured yet) answers the
+ * floor, which is what this server did before it measured anything.
+ */
+export function coalesceWindowFor(roundTripMs: number | null): number {
+ if (roundTripMs === null || !Number.isFinite(roundTripMs)) return MIN_COALESCE_WINDOW_MS;
+ const quarter = roundTripMs / 4;
+ return Math.min(MAX_COALESCE_WINDOW_MS, Math.max(MIN_COALESCE_WINDOW_MS, Math.round(quarter)));
+}
+
+/** Round-trip estimate from Ping/Pong echoes, smoothed so one stalled reply cannot swing it. */
+export class RoundTripEstimator {
+ private estimate: number | null = null;
+
+ /** Ignores nonsense (a clock that went backwards, an echo of something we never sent). */
+ sample(roundTripMs: number): void {
+ if (!Number.isFinite(roundTripMs) || roundTripMs < 0) return;
+ this.estimate =
+ this.estimate === null ? roundTripMs : this.estimate * (1 - ALPHA) + roundTripMs * ALPHA;
+ }
+
+ get roundTripMs(): number | null {
+ return this.estimate;
+ }
+
+ windowMs(): number {
+ return coalesceWindowFor(this.estimate);
+ }
+}
+
+/** How far behind a viewer may fall before the stream skips it ahead with a fresh repaint. */
+export const MAX_VIEWER_LAG_MS = 750;
+/**
+ * Floor and ceiling for the derived high-water mark. The ceiling is the byte bound this
+ * server always used, so a link fast enough to drain it keeps its old behaviour; the floor
+ * stays clear of the low-water mark, or a viewer would resync on every burst.
+ */
+export const MIN_HIGH_WATER_BYTES = 128 * 1024;
+export const MAX_HIGH_WATER_BYTES = 1024 * 1024;
+
+/** Shortest interval between two throughput samples: below this, one flush is all noise. */
+const SAMPLE_INTERVAL_MS = 250;
+
+/**
+ * Throughput of one viewer's stream, measured from what has actually left.
+ *
+ * Every frame reports itself twice — once when it is handed to the socket, once when the
+ * socket says it is gone — and the second report is the measurement: bytes delivered per
+ * second. It is only taken while something WAS waiting, because a socket with an empty
+ * queue delivers exactly as fast as output is produced, and an idle shell would otherwise
+ * "measure" a few bytes a second and shrink the viewer's allowance to nothing.
+ *
+ * Both numbers are in the bytes the terminal produced, never in the bytes the socket wrote:
+ * the stream is compressed on the wire, and the lag a viewer feels is measured in the
+ * output it has not seen yet.
+ */
+export class SocketDrainMeter {
+ private delivered = 0;
+ private lastSampleAt: number | null = null;
+ private lastDelivered = 0;
+ private lastQueued = 0;
+ private estimate: number | null = null;
+
+ constructor(private readonly now: () => number = () => Date.now()) {}
+
+ /** Reports bytes the socket has finished with, and what is still waiting behind them. */
+ note(delivered: number, queued: number): void {
+ this.delivered += delivered;
+ const at = this.now();
+ if (this.lastSampleAt === null) {
+ this.reset(at, queued);
+ return;
+ }
+ const elapsed = at - this.lastSampleAt;
+ if (elapsed < SAMPLE_INTERVAL_MS) return;
+ const backlogged = this.lastQueued > 0 || queued > 0;
+ const drained = this.delivered - this.lastDelivered;
+ this.reset(at, queued);
+ if (!backlogged || drained <= 0) return;
+ const rate = (drained * 1000) / elapsed;
+ this.estimate = this.estimate === null ? rate : this.estimate * (1 - ALPHA) + rate * ALPHA;
+ }
+
+ private reset(at: number, queued: number): void {
+ this.lastSampleAt = at;
+ this.lastDelivered = this.delivered;
+ this.lastQueued = queued;
+ }
+
+ /** Bytes per second, or null while the stream has never had to wait for the socket. */
+ get bytesPerSecond(): number | null {
+ return this.estimate;
+ }
+
+ /** The backlog that means MAX_VIEWER_LAG_MS of lag at the measured rate. */
+ highWaterBytes(): number {
+ if (this.estimate === null) return MAX_HIGH_WATER_BYTES;
+ const allowance = (this.estimate * MAX_VIEWER_LAG_MS) / 1000;
+ return Math.min(MAX_HIGH_WATER_BYTES, Math.max(MIN_HIGH_WATER_BYTES, Math.round(allowance)));
+ }
+}
diff --git a/packages/server/src/terminal/output-coalescer.ts b/packages/server/src/terminal/output-coalescer.ts
index 76095f1c..0841cace 100644
--- a/packages/server/src/terminal/output-coalescer.ts
+++ b/packages/server/src/terminal/output-coalescer.ts
@@ -9,6 +9,10 @@
*
* Reverting this to trailing-only is the classic regression here: it adds a full window
* (~5ms) to every single keystroke and the terminal immediately feels mushy.
+ *
+ * The window may be a function, which is how the stream widens it on a slow link: the
+ * merge is worth more, and costs proportionally less, the longer the round trip already is
+ * (link-quality.ts). It is read per burst, never cached.
*/
export const DEFAULT_COALESCE_WINDOW_MS = 5;
@@ -20,25 +24,30 @@ export class TerminalOutputCoalescer {
constructor(
private readonly flushFn: (data: string) => void,
- private readonly windowMs: number = DEFAULT_COALESCE_WINDOW_MS,
+ private readonly window: number | (() => number) = DEFAULT_COALESCE_WINDOW_MS,
private readonly now: () => number = () => Date.now(),
) {}
+ private windowMs(): number {
+ return typeof this.window === "number" ? this.window : this.window();
+ }
+
push(data: string): void {
if (data.length === 0) return;
this.pending += data;
if (this.timer !== null) return; // a trailing flush is already scheduled
+ const windowMs = this.windowMs();
const elapsed = this.now() - this.lastFlushAt;
- if (elapsed >= this.windowMs) {
+ if (elapsed >= windowMs) {
this.flush(); // leading edge: straight through, no added latency
return;
}
this.timer = setTimeout(() => {
this.timer = null;
this.flush();
- }, this.windowMs - elapsed);
+ }, windowMs - elapsed);
}
/**
diff --git a/packages/server/src/terminal/stream.ts b/packages/server/src/terminal/stream.ts
index 39d99fc7..8d7cd645 100644
--- a/packages/server/src/terminal/stream.ts
+++ b/packages/server/src/terminal/stream.ts
@@ -1,13 +1,16 @@
/**
* The terminal stream's PROTOCOL: frames, coalescing, the restore-first attach sequence,
- * size ownership and the lagging-viewer resync.
+ * size ownership, the heartbeat and the lagging-viewer resync.
*
* This is platform code even though the socket is the runtime's. The seam cannot carry a
* stream — a handler returns one whole Response — so the runtime keeps the upgrade
* handshake and its authentication (terminal/ws.ts) and hands the live socket here. What
* flows over it afterwards is behaviour, and behaviour a push must be able to change:
- * every byte of the wire format, the ~5ms coalescing window and both backpressure
- * watermarks are decided in this file.
+ * every byte of the wire format, the coalescing window, the heartbeat and both
+ * backpressure watermarks are decided in this file. Two of those are no longer constants:
+ * the window and the high-water mark are derived per socket from what the link measures
+ * (link-quality.ts), because a number tuned on a LAN means something else entirely on a
+ * phone.
*
* Attach sequence — the part that makes a browser reload look seamless:
* 1. the client's geometry (?cols=&rows=) resizes the pty FIRST, so the snapshot is
@@ -27,12 +30,54 @@ import {
parseResizePayload,
} from "./frames.js";
import { TerminalOutputCoalescer } from "./output-coalescer.js";
+import { RoundTripEstimator, SocketDrainMeter } from "./link-quality.js";
import type { TerminalSession } from "./session.js";
-const BACKPRESSURE_HIGH_WATER = 1024 * 1024;
+/**
+ * Per-viewer backpressure. A send queues without limit when the peer cannot drain, so a
+ * viewer too far behind stops receiving live output — the bytes keep feeding the server
+ * emulator, nothing of the session is lost — and once its backlog falls below LOW_WATER it
+ * is repainted with one fresh Restore frame (the same self-contained repaint an attach
+ * uses) and live output resumes. Skipping ahead beats both alternatives: replaying
+ * megabytes the viewer could only fast-forward through, or disconnecting it.
+ *
+ * "Too far behind" is the one number that is no longer a constant: it is derived per socket
+ * from what that link delivers (link-quality.ts), because the byte bound this used to be
+ * meant milliseconds of lag on a LAN and many seconds on a phone.
+ */
const BACKPRESSURE_LOW_WATER = 64 * 1024;
const BACKPRESSURE_POLL_MS = 250;
+/**
+ * Heartbeat. A phone loses a socket without closing it — a radio hand-off, a carrier NAT
+ * reaping an idle flow — and both ends then hold a connection that will never carry a byte
+ * again: the pty writes into a dead socket, the viewer waits for output that cannot arrive.
+ * The probe is what turns that into a close, which is what the client retries on. The reply
+ * doubles as the round-trip sample that sizes the merge window (link-quality.ts).
+ *
+ * The timeout is generous on purpose: a backgrounded tab is throttled to roughly one timer
+ * a minute, and reaping a viewer that is merely asleep would cost it a full repaint on
+ * every glance at the screen.
+ *
+ * Only a viewer that has ANSWERED a probe is ever reaped for silence. A page loaded before
+ * this shipped stays open across the push that brought it — the platform is replaced under
+ * a tab that is not reloaded — and that page does not know the opcode: it would go quiet,
+ * be closed, and (having no reattach either) show a dead pane. Silence only means something
+ * from a peer that has shown it can break it.
+ */
+const HEARTBEAT_INTERVAL_MS = 15_000;
+const HEARTBEAT_TIMEOUT_MS = 75_000;
+
+/**
+ * Frames smaller than this go out uncompressed even when the stream negotiated
+ * permessage-deflate: a keystroke's echo compresses to more bytes than it started with
+ * once the deflate overhead rides along. ws's own `perMessageDeflate.threshold` cannot
+ * carry this promise — it only applies to a direction that negotiated no_context_takeover,
+ * and this server keeps its deflate context precisely because a shared context is what
+ * compresses repeated output best — so the decision is made per message, here.
+ */
+const SMALL_FRAME_BYTES = 512;
+
export function bindTerminalStream(
ws: WebSocket,
session: TerminalSession,
@@ -42,8 +87,28 @@ export function bindTerminalStream(
const connectionId = randomUUID();
ws.binaryType = "nodebuffer";
+ // What this viewer's link is doing, measured from this socket alone: the round trip from
+ // the heartbeat's echo, the drain rate from what the socket accepts.
+ const roundTrip = new RoundTripEstimator();
+ const drain = new SocketDrainMeter();
+
+ /**
+ * Bytes handed to the socket that have not left it yet — the viewer's backlog, counted in
+ * the output the terminal produced rather than in what the wire carried. `bufferedAmount`
+ * cannot answer this any more: the stream is compressed (terminal/ws.ts), and a screen of
+ * repeated build output leaves the socket a hundred times smaller than the user is behind.
+ * ws calls back once a message is flushed, so what has not called back is what is waiting.
+ */
+ let inFlight = 0;
+
const send = (bytes: Uint8Array): void => {
- if (ws.readyState === ws.OPEN) ws.send(bytes);
+ if (ws.readyState !== ws.OPEN) return;
+ const size = bytes.byteLength;
+ inFlight += size;
+ ws.send(bytes, { compress: size >= SMALL_FRAME_BYTES }, () => {
+ inFlight -= size;
+ drain.note(size, inFlight);
+ });
};
// Lagging-viewer state (see the watermark comment above): while desynced, this viewer's
@@ -58,12 +123,15 @@ export function bindTerminalStream(
const sendOutput = (data: string): void => {
if (desynced) return; // dropped: the emulator keeps every byte; the resync repaints
send(encodeTerminalFrame({ opcode: TerminalStreamOpcode.Output, payload: data }));
- if (ws.bufferedAmount <= BACKPRESSURE_HIGH_WATER) return;
+ // A LAG bound, in bytes: how much output this viewer can still be behind and catch up
+ // within MAX_VIEWER_LAG_MS at the rate its socket has been delivering (link-quality.ts),
+ // which on a link fast enough to drain it is the 1MB this server always used.
+ if (inFlight <= drain.highWaterBytes()) return;
desynced = true;
- log(`[terminal] stream ${session.id}: viewer ${ws.bufferedAmount}B behind, pausing for resync`);
+ log(`[terminal] stream ${session.id}: viewer ${inFlight}B behind, pausing for resync`);
resyncTimer = setInterval(() => {
if (ws.readyState !== ws.OPEN) return stopResyncPoll();
- if (ws.bufferedAmount > BACKPRESSURE_LOW_WATER) return;
+ if (inFlight > BACKPRESSURE_LOW_WATER) return;
stopResyncPoll();
desynced = false; // flip first: output parsed after this snapshot flows live again
try {
@@ -83,7 +151,9 @@ export function bindTerminalStream(
// Buffers output until the Restore frame has been sent (see the attach sequence above).
let restored = false;
let preRestore = "";
- const coalescer = new TerminalOutputCoalescer(sendOutput);
+ // The merge window follows the measured round trip: 5ms on a LAN, up to 50ms on a link
+ // where a single trip already costs twenty times that.
+ const coalescer = new TerminalOutputCoalescer(sendOutput, () => roundTrip.windowMs());
const unsubscribeOutput = session.onOutput((data) => {
if (!restored) {
@@ -131,8 +201,23 @@ export function bindTerminalStream(
);
}
+ // Anything arriving proves the socket is still a socket, heartbeat replies included.
+ let lastSeenAt = Date.now();
+ // Whether this viewer speaks the heartbeat at all (see HEARTBEAT_TIMEOUT_MS).
+ let answersProbes = false;
+ const heartbeat = setInterval(() => {
+ if (ws.readyState !== ws.OPEN) return;
+ if (answersProbes && Date.now() - lastSeenAt > HEARTBEAT_TIMEOUT_MS) {
+ log(`[terminal] stream ${session.id}: viewer stopped answering, closing`);
+ ws.terminate(); // not close(): a socket this quiet will not complete a handshake
+ return;
+ }
+ send(encodeTerminalFrame({ opcode: TerminalStreamOpcode.Ping, payload: String(Date.now()) }));
+ }, HEARTBEAT_INTERVAL_MS);
+
ws.on("message", (raw: Buffer, isBinary: boolean) => {
if (!isBinary) return; // the data plane is binary-only
+ lastSeenAt = Date.now();
const frame = decodeTerminalFrame(new Uint8Array(raw.buffer, raw.byteOffset, raw.byteLength));
if (!frame) return;
switch (frame.opcode) {
@@ -144,12 +229,24 @@ export function bindTerminalStream(
if (size) session.resize({ connectionId, ...size });
break;
}
+ // The client probes for itself (its own timers are the ones a phone suspends); the
+ // payload is its clock reading and goes back untouched.
+ case TerminalStreamOpcode.Ping:
+ send(encodeTerminalFrame({ opcode: TerminalStreamOpcode.Pong, payload: frame.payload }));
+ break;
+ case TerminalStreamOpcode.Pong: {
+ answersProbes = true;
+ const sentAt = Number.parseInt(framePayloadText(frame), 10);
+ if (Number.isFinite(sentAt)) roundTrip.sample(Date.now() - sentAt);
+ break;
+ }
default:
break; // server-only opcodes echoed back by a confused client
}
});
const teardown = (): void => {
+ clearInterval(heartbeat);
unsubscribeOutput();
unsubscribeExit();
stopResyncPoll();
diff --git a/packages/server/src/terminal/ws.ts b/packages/server/src/terminal/ws.ts
index f4e6ded6..7dd01adc 100644
--- a/packages/server/src/terminal/ws.ts
+++ b/packages/server/src/terminal/ws.ts
@@ -21,21 +21,6 @@ import type { Auth } from "../mechanisms/identity.js";
const STREAM_PATH = /^\/api\/terminals\/([^/]+)\/stream$/;
-/**
- * Per-viewer backpressure watermarks. `ws.send` queues without limit when the peer cannot
- * drain, so a viewer more than HIGH_WATER behind stops receiving live output — the bytes
- * keep feeding the server emulator, nothing of the session is lost — and once its socket
- * drains below LOW_WATER it is repainted with one fresh Restore frame (the same
- * self-contained repaint an attach uses) and live output resumes. Skipping ahead beats
- * both alternatives: replaying megabytes the viewer could only fast-forward through, or
- * disconnecting it (the client treats a closed stream as a dead pane, not a retry). The
- * high watermark is therefore a LAG bound — how far behind a viewer may fall before it is
- * fast-forwarded — and doubles as the per-viewer memory bound.
- */
-const BACKPRESSURE_HIGH_WATER = 1024 * 1024;
-const BACKPRESSURE_LOW_WATER = 64 * 1024;
-const BACKPRESSURE_POLL_MS = 250;
-
export interface TerminalWebSocketDeps {
/** The booted platform owns the terminals; the runtime asks it per upgrade. */
hmr: ServerHmrHost;
@@ -44,7 +29,27 @@ export interface TerminalWebSocketDeps {
}
export function attachTerminalWebSocket(server: HttpServer, deps: TerminalWebSocketDeps): void {
- const wss = new WebSocketServer({ noServer: true, perMessageDeflate: false });
+ const wss = new WebSocketServer({
+ noServer: true,
+ /**
+ * Terminal output is the most compressible traffic this server has — repeated prompts,
+ * repeated SGR runs, whole repainted screens — and on a phone the bytes are the scarce
+ * thing, so it is worth a little CPU here. The settings keep that little: the window is
+ * capped so a per-socket deflate context costs kilobytes rather than the default's
+ * hundreds, and the client's context is not kept, because a browser's uplink here is a
+ * keystroke at a time and has nothing to reuse. Small frames are left uncompressed by
+ * the sender itself, per message (terminal/stream.ts), not by a threshold here: ws only
+ * applies a config `threshold` to a direction that negotiated no_context_takeover, and
+ * this server keeps its context — a shared context is what compresses repeated output
+ * best.
+ */
+ perMessageDeflate: {
+ clientNoContextTakeover: true,
+ serverMaxWindowBits: 13,
+ zlibDeflateOptions: { level: 3, memLevel: 7 },
+ concurrencyLimit: 10,
+ },
+ });
server.on("upgrade", (req: IncomingMessage, socket: Duplex, head: Buffer) => {
const url = new URL(req.url ?? "/", "http://localhost");
diff --git a/packages/server/test/terminal-stream.test.ts b/packages/server/test/terminal-stream.test.ts
index d7c9f827..9c95f718 100644
--- a/packages/server/test/terminal-stream.test.ts
+++ b/packages/server/test/terminal-stream.test.ts
@@ -10,6 +10,7 @@
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
import { serve } from "@hono/node-server";
import type { Server as HttpServer } from "node:http";
+import type { Socket as NetSocket } from "node:net";
import { WebSocket } from "ws";
import { createTestApp, loginAdmin, provisionUser, apiClient } from "./helpers.js";
import type { TestApp } from "./helpers.js";
@@ -18,6 +19,7 @@ import {
TerminalStreamOpcode,
decodeTerminalFrame,
encodeTerminalFrame,
+ framePayloadText,
} from "../src/terminal/frames.js";
const TEST_TIMEOUT = 20_000;
@@ -111,15 +113,29 @@ async function terminalInfo(id: string): Promise<TerminalInfoJson> {
return (await res.json()) as TerminalInfoJson;
}
+/**
+ * One server→client data frame as it left the wire, before ws inflates it: the compression
+ * test reads the RSV1 bit (permessage-deflate) and the frame's real length from here.
+ */
+interface WireFrame {
+ rsv1: boolean;
+ opcode: number;
+ length: number;
+ payload: Uint8Array;
+}
+
interface StreamClient {
ws: WebSocket;
frames: Array<{ opcode: number; text: string }>;
+ /** Raw first-fragment frames seen on the socket; populated only for deflate clients. */
+ wire: WireFrame[];
/** Concatenation of every Output payload so far. */
outputText(): string;
/** Payload of the first Restore frame (the attach snapshot). */
restoreText(): string;
sendInput(text: string): void;
sendResize(cols: number, rows: number, intent: "claim" | "update"): void;
+ sendPing(payload: string): void;
waitFor(predicate: () => boolean, what: string, timeoutMs?: number): Promise<void>;
close(): Promise<void>;
}
@@ -132,15 +148,27 @@ function streamUrl(id: string, cols?: number, rows?: number): string {
/** Opens a stream and resolves once connected; every received frame must decode. */
function attach(
id: string,
- options: { cookie?: string; cols?: number; rows?: number; origin?: string } = {},
+ options: {
+ cookie?: string;
+ cols?: number;
+ rows?: number;
+ origin?: string;
+ /** Negotiate permessage-deflate (a browser does) and record raw frames on the wire. */
+ deflate?: boolean;
+ } = {},
): Promise<StreamClient> {
const headers: Record<string, string> = { cookie: options.cookie ?? adminCookie };
if (options.origin !== undefined) headers.origin = options.origin;
- const ws = new WebSocket(streamUrl(id, options.cols ?? 80, options.rows ?? 24), { headers });
+ const ws = new WebSocket(streamUrl(id, options.cols ?? 80, options.rows ?? 24), {
+ headers,
+ ...(options.deflate === true ? { perMessageDeflate: {} } : {}),
+ });
const frames: StreamClient["frames"] = [];
+ const wire: WireFrame[] = [];
// Every server-sent frame must decode; checked from waitFor (throwing inside the ws
// 'message' listener would surface as an uncaught exception, not a test failure).
let undecodable = 0;
+ let wireError: string | null = null;
ws.on("message", (raw: Buffer) => {
const frame = decodeTerminalFrame(new Uint8Array(raw.buffer, raw.byteOffset, raw.byteLength));
@@ -151,9 +179,60 @@ function attach(
frames.push({ opcode: frame.opcode, text: new TextDecoder().decode(frame.payload) });
});
+ // The wire tap for the deflate client: walks the raw server→client stream and records
+ // each message's first fragment — the one that carries the compression bit. Server
+ // frames are never masked, so the walk is header + length + payload, nothing else.
+ const watchWire = (): void => {
+ const sock = (ws as unknown as { _socket?: NetSocket })._socket;
+ if (sock === undefined) {
+ wireError = "the deflate client has no underlying socket to watch";
+ return;
+ }
+ let pending = Buffer.alloc(0);
+ sock.on("data", (chunk: Buffer) => {
+ pending = Buffer.concat([pending, chunk]);
+ for (;;) {
+ if (pending.length < 2) break;
+ const b0 = pending[0] as number;
+ const b1 = pending[1] as number;
+ if ((b1 & 0x80) !== 0) {
+ wireError = "server frames must arrive unmasked";
+ return;
+ }
+ let header = 2;
+ let length = b1 & 0x7f;
+ if (length === 126) {
+ if (pending.length < 4) break;
+ length = pending.readUInt16BE(2);
+ header = 4;
+ } else if (length === 127) {
+ if (pending.length < 10) break;
+ const huge = pending.readBigUInt64BE(2);
+ if (huge > 0xffff_ffffn) {
+ wireError = "a frame too large to parse";
+ return;
+ }
+ length = Number(huge);
+ header = 10;
+ }
+ if (pending.length < header + length) break;
+ if ((b0 & 0x0f) !== 0) {
+ wire.push({
+ rsv1: (b0 & 0x40) !== 0,
+ opcode: b0 & 0x0f,
+ length,
+ payload: new Uint8Array(pending.subarray(header, header + length)),
+ });
+ }
+ pending = pending.subarray(header + length);
+ }
+ });
+ };
+
const client: StreamClient = {
ws,
frames,
+ wire,
outputText: () =>
frames
.filter((f) => f.opcode === TerminalStreamOpcode.Output)
@@ -169,16 +248,20 @@ function attach(
payload: JSON.stringify({ cols, rows, intent }),
}),
),
+ sendPing: (payload) =>
+ ws.send(encodeTerminalFrame({ opcode: TerminalStreamOpcode.Ping, payload })),
waitFor: async (predicate, what, timeoutMs = 15_000) => {
const deadline = Date.now() + timeoutMs;
while (!predicate()) {
if (undecodable > 0) throw new Error(`Received ${undecodable} undecodable frame(s).`);
+ if (wireError !== null) throw new Error(wireError);
if (Date.now() > deadline) {
throw new Error(`Timed out waiting for ${what}. Frames: ${JSON.stringify(frames)}`);
}
await new Promise((resolve) => setTimeout(resolve, 25));
}
if (undecodable > 0) throw new Error(`Received ${undecodable} undecodable frame(s).`);
+ if (wireError !== null) throw new Error(wireError);
},
close: () =>
new Promise((resolve) => {
@@ -189,7 +272,10 @@ function attach(
};
return new Promise((resolve, reject) => {
- ws.once("open", () => resolve(client));
+ ws.once("open", () => {
+ if (options.deflate === true) watchWire();
+ resolve(client);
+ });
ws.once("unexpected-response", (_req, res) =>
reject(new Error(`handshake rejected: ${res.statusCode}`)),
);
@@ -317,16 +403,96 @@ describePty("terminal stream handshake", () => {
});
/**
- * The line the backpressure flood is made of, and the marker its resync assertion reads.
- * It fills every row of the screen for the whole burst, so it is there whichever instant
- * the resync snapshot is taken at — unlike a marker printed once at the end of a round,
- * which megabytes of scrolling carry off the screen again.
+ * What the backpressure flood is made of: random bytes, not repeated ones.
+ *
+ * The stream is compressed on the wire (terminal/ws.ts), and repeated text leaves the
+ * socket a hundred times smaller than it entered — a `yes`-style burst is delivered as fast
+ * as a shell can produce it and never puts a viewer behind at all. base64 of /dev/urandom
+ * is the cheapest output a shell can produce that compresses to nothing, so what the viewer
+ * is behind on is real.
+ */
+const FLOOD_COMMAND = "head -c 4000000 /dev/urandom | base64";
+/**
+ * A screenful of that flood: base64 runs on two consecutive rows. The restore stream writes
+ * the grid row by row, so this holds whichever instant the resync snapshot is taken at —
+ * unlike a marker printed once per round, which megabytes of scrolling carry off the screen.
*/
-const FLOOD_LINE = "0123456789abcdef";
+const FLOOD_ROWS = /[A-Za-z0-9+/]{40}\r\n[A-Za-z0-9+/]{40}/;
/** Every Restore opens with this self-contained repaint (see snapshot.ts). */
const RESTORE_PREAMBLE = "\x1b[0m\x1b[?1049l\x1b[H\x1b[2J\x1b[3J";
+describePty("terminal stream heartbeat", () => {
+ it(
+ "echoes a client's probe, so a client can tell a live socket from a dead one",
+ async () => {
+ const terminal = await createTerminal();
+ const client = await attach(terminal.id);
+ try {
+ client.sendPing("42");
+ await client.waitFor(
+ () => client.frames.some((f) => f.opcode === TerminalStreamOpcode.Pong),
+ "pong",
+ );
+
+ const pong = client.frames.find((f) => f.opcode === TerminalStreamOpcode.Pong);
+ // Echoed untouched: the payload is the CLIENT's clock reading, and only the client
+ // can make a round trip out of it.
+ expect(pong?.text).toBe("42");
+ } finally {
+ await client.close();
+ await api.delete(`/api/terminals/${terminal.id}`);
+ }
+ },
+ TEST_TIMEOUT,
+ );
+});
+
+describePty("terminal stream compression", () => {
+ it(
+ "leaves small frames uncompressed and large ones compressed once permessage-deflate is negotiated",
+ async () => {
+ const terminal = await createTerminal();
+ const client = await attach(terminal.id, { deflate: true });
+ try {
+ await client.waitFor(() => client.restoreText().length > 0, "attach restore");
+
+ // A keystroke-sized burst: the command's echo, its output and the next prompt
+ // together are far below the threshold, and on the wire it must arrive as-is —
+ // RSV1 clear, the bytes decodable as a terminal frame with no inflate anywhere.
+ const SMALL_MARKER = "zz-small-uncompressed-42";
+ await runAndWait(client, `echo ${SMALL_MARKER}`, SMALL_MARKER);
+ await client.waitFor(
+ () =>
+ client.wire.some((f) => {
+ if (f.rsv1) return false;
+ const frame = decodeTerminalFrame(f.payload);
+ return (
+ frame !== null &&
+ frame.opcode === TerminalStreamOpcode.Output &&
+ framePayloadText(frame).includes(SMALL_MARKER)
+ );
+ }),
+ "the small frame on the wire uncompressed",
+ );
+
+ // A large burst of barely compressible bytes stays compressed: RSV1 set, and —
+ // random base64 shrinks little — still far past the threshold on the wire.
+ await runAndWait(
+ client,
+ "head -c 4000 /dev/urandom | base64; echo big-burst-done",
+ "big-burst-done",
+ );
+ expect(client.wire.some((f) => f.rsv1 && f.length >= 512)).toBe(true);
+ } finally {
+ await client.close();
+ await api.delete(`/api/terminals/${terminal.id}`);
+ }
+ },
+ TEST_TIMEOUT,
+ );
+});
+
describePty("terminal stream backpressure", () => {
it("resyncs a lagging viewer with a fresh Restore instead of disconnecting it", async () => {
const terminal = await createTerminal();
@@ -342,9 +508,9 @@ describePty("terminal stream backpressure", () => {
client.ws.pause();
// Flood until the server itself reports this viewer as lagging. A fixed burst is a
// coin flip: a paused reader still lets the kernel absorb megabytes into its socket
- // buffers (autotuned), so the server's own send queue — the thing the watermark
- // measures — may never cross 1 MiB however much the shell wrote. This suite failed
- // that way roughly one run in three, locally and on CI.
+ // buffers (autotuned), so the viewer's backlog — the thing the watermark measures —
+ // may never cross 1 MiB however much the shell wrote. This suite failed that way
+ // roughly one run in three, locally and on CI.
//
// Only lines logged from here on count. `serverLogs` outlives one attempt and vitest
// retries this file on macOS, so a second attempt reading the first one's line would
@@ -353,29 +519,77 @@ describePty("terminal stream backpressure", () => {
const loggedBefore = serverLogs.length;
const isLagging = (): boolean =>
serverLogs.slice(loggedBefore).some((l) => l.includes("pausing for resync"));
+ let lastRound = 0;
for (let round = 1; round <= 8 && !isLagging(); round += 1) {
- client.sendInput(`yes ${FLOOD_LINE} | head -n 400000; echo BURST-DONE-${round}\r`);
- await waitForCapture(terminal.id, `BURST-DONE-${round}`, 60_000);
+ lastRound = round;
+ // The pty echoes a command line the instant it is typed, so a literal marker would
+ // satisfy this wait before the flood behind it had produced one byte. All eight
+ // rounds would then go out back-to-back and land as one undifferentiated flood: the
+ // kernel's autotuned buffers absorb far more than a single round's share, and
+ // whichever way the watermark lands, the assert below races a stream that is still
+ // running — the flake this suite was known for. The arithmetic is the fix: the echo
+ // of "BURST-$((20+1))" matches nothing the wait accepts, so a round ends only once
+ // the shell has actually run it — its burst fully landed server-side.
+ client.sendInput(`${FLOOD_COMMAND}; echo BURST-$((20+${round}))\r`);
+ await waitForCapture(terminal.id, `BURST-${20 + round}`, 60_000);
}
expect(isLagging(), "server never marked the paused viewer as lagging").toBe(true);
// The burst has fully landed server-side, and the lagging viewer is still attached.
expect(client.ws.readyState).toBe(WebSocket.OPEN);
- // Catching up delivers a resync Restore, and the stream is live again afterwards.
+ // Catching up delivers a resync Restore, and the stream is live again afterwards —
+ // but which frame ends the cycle depends on the link, so the settle cannot wait for
+ // a particular one. On a slow link the kernel absorbs the flood as fast as the pty
+ // emits it: the resync fires mid-flood, no second high-water crossing follows, and
+ // the round's end marker reaches the client as live output — no marker-bearing
+ // Restore ever exists. What every timing has in common is the marker arriving at
+ // the client, in whichever frame carries it: the marker is the last thing the round
+ // produces, so once the client holds it, nothing of the flood is left in flight.
+ const burstEnd = `BURST-${20 + lastRound}`;
client.ws.resume();
- const restores = (): StreamClient["frames"] =>
- client.frames.filter((f) => f.opcode === TerminalStreamOpcode.Restore);
- await client.waitFor(() => restores().length >= 2, "resync Restore frame", 60_000);
- const resync = restores()[1]!;
- // What a resync promises is the repaint, not a particular line of screen residue:
- // the snapshot is taken whenever this viewer's socket drains past the low watermark,
- // which the test does not get to choose and which may well be mid-flood. So assert
- // the two things that hold at every such instant — it is the same self-contained
- // repaint an attach sends, and it carries the flooded screen rather than the empty
- // one the attach Restore captured.
- expect(resync.text.slice(0, RESTORE_PREAMBLE.length)).toBe(RESTORE_PREAMBLE);
- expect(resync.text).toContain(`${FLOOD_LINE}\r\n${FLOOD_LINE}`);
+ await client.waitFor(
+ () => client.frames.some((f) => f.text.includes(burstEnd)),
+ "the flood's end to reach the client",
+ // The flood's tail is the last thing through the backlog the paused socket
+ // accumulated, and chewing through it is plain CPU — the slowest step of this
+ // case on a starved runner, so the cap is deliberately generous.
+ 90_000,
+ );
+
+ // One race is still open at that instant: a resync whose poll had not yet fired
+ // repaints within one interval (250ms) of the client catching up, and a probe
+ // typed before it would land in that drop window. The repaint arrives as a Restore
+ // — a data frame — so a quiet window far longer than the poll, in which nothing
+ // but the heartbeat arrives, proves none was pending: the pane is quiet, and the
+ // live probe below is typed to an idle shell.
+ let quietSince = 0;
+ let quietFrames = -1;
+ const streamQuiet = (): boolean => {
+ const dataFrames = client.frames.filter(
+ (f) => f.opcode !== TerminalStreamOpcode.Ping,
+ ).length;
+ if (dataFrames !== quietFrames) {
+ quietFrames = dataFrames;
+ quietSince = Date.now();
+ }
+ return Date.now() - quietSince >= 2_000;
+ };
+ await client.waitFor(streamQuiet, "a quiet stream after the flood", 30_000);
+
+ // The resync this case exists to verify: the lagging gate above means the server
+ // desynced this viewer, and the only way out of that state is the poll's fresh
+ // Restore — so a repaint carrying the flooded pane must have arrived. It is not
+ // necessarily the marker-bearing one (the mid-flood timing above legitimately
+ // skips that); the flooded pane is the promise, and it identifies the repaint.
+ const resync = client.frames.find(
+ (f) => f.opcode === TerminalStreamOpcode.Restore && FLOOD_ROWS.test(f.text),
+ );
+ expect(resync, "a resync repaint of the flooded pane").toBeDefined();
+ // What a resync promises is the repaint, not a particular line of screen residue —
+ // it is the same self-contained repaint an attach sends, and it carries the flooded
+ // screen rather than the empty one the attach Restore captured.
+ expect(resync!.text.slice(0, RESTORE_PREAMBLE.length)).toBe(RESTORE_PREAMBLE);
await runAndWait(client, "echo LIVE-$((40+2))", "LIVE-42");
} finally {
// Closed here rather than after the assertions: a socket a failed assertion left open
@@ -386,7 +600,7 @@ describePty("terminal stream backpressure", () => {
}
// Megabytes through a pty, a paused socket and a drain: minutes of budget on a loaded
// runner, where the default would report a timeout rather than a real defect.
- }, 120_000);
+ }, 180_000);
});
describePty("terminal stream attach", () => {
diff --git a/packages/server/test/terminal.test.ts b/packages/server/test/terminal.test.ts
index 06df4b40..51f4b0ff 100644
--- a/packages/server/test/terminal.test.ts
+++ b/packages/server/test/terminal.test.ts
@@ -17,6 +17,15 @@ import { expandHomePath } from "../src/terminal/session.js";
import { repairSpawnHelpers } from "../src/terminal/spawn-helper.js";
import { TerminalInputModeTracker } from "../src/terminal/input-mode.js";
import { TerminalOutputCoalescer } from "../src/terminal/output-coalescer.js";
+import {
+ MAX_COALESCE_WINDOW_MS,
+ MAX_HIGH_WATER_BYTES,
+ MIN_COALESCE_WINDOW_MS,
+ MIN_HIGH_WATER_BYTES,
+ RoundTripEstimator,
+ SocketDrainMeter,
+ coalesceWindowFor,
+} from "../src/terminal/link-quality.js";
import { applyTerminalSize, releaseTerminalSize } from "../src/terminal/size-ownership.js";
import { captureLines, renderRestoreAnsi } from "../src/terminal/snapshot.js";
import { resolveKeys } from "../src/terminal/routes.js";
@@ -253,6 +262,25 @@ describe("output coalescer", () => {
expect(flushed).toEqual(["a", "bc"]);
});
+ it("reads its window per burst, so a link that slows down widens it", () => {
+ vi.useFakeTimers();
+ const flushed: string[] = [];
+ let window = 5;
+ const coalescer = new TerminalOutputCoalescer(
+ (data) => flushed.push(data),
+ () => window,
+ );
+
+ coalescer.push("a"); // leading edge
+ window = 40; // the round trip grew between bursts
+ coalescer.push("b");
+ vi.advanceTimersByTime(5);
+ expect(flushed).toEqual(["a"]); // still merging: the window is 40ms now
+ vi.advanceTimersByTime(35);
+
+ expect(flushed).toEqual(["a", "b"]);
+ });
+
it("flushes pending output on demand so ordering is preserved", () => {
vi.useFakeTimers();
const flushed: string[] = [];
@@ -266,6 +294,73 @@ describe("output coalescer", () => {
});
});
+describe("link quality", () => {
+ it("keeps the LAN window until a round trip is measured", () => {
+ expect(coalesceWindowFor(null)).toBe(MIN_COALESCE_WINDOW_MS);
+ expect(coalesceWindowFor(8)).toBe(MIN_COALESCE_WINDOW_MS);
+ });
+
+ it("widens the window with the round trip, up to the ceiling", () => {
+ expect(coalesceWindowFor(80)).toBe(20);
+ expect(coalesceWindowFor(200)).toBe(50);
+ expect(coalesceWindowFor(2000)).toBe(MAX_COALESCE_WINDOW_MS);
+ });
+
+ it("smooths the round trip so one stalled reply cannot swing the window", () => {
+ const rtt = new RoundTripEstimator();
+ rtt.sample(200);
+ rtt.sample(2000); // one reply behind a garbage-collecting tab
+
+ expect(rtt.roundTripMs).toBeLessThan(700);
+ expect(rtt.windowMs()).toBeLessThanOrEqual(MAX_COALESCE_WINDOW_MS);
+ });
+
+ it("ignores a round trip a clock change made negative", () => {
+ const rtt = new RoundTripEstimator();
+ rtt.sample(-5);
+
+ expect(rtt.roundTripMs).toBeNull();
+ });
+
+ it("allows the full byte budget while the stream never waits on the socket", () => {
+ let now = 0;
+ const meter = new SocketDrainMeter(() => now);
+ for (let i = 0; i < 10; i++) {
+ now += 300;
+ meter.note(1000, 0); // delivered, with nothing waiting behind it
+ }
+
+ expect(meter.bytesPerSecond).toBeNull();
+ expect(meter.highWaterBytes()).toBe(MAX_HIGH_WATER_BYTES);
+ });
+
+ it("turns a slow delivery rate into a smaller allowance", () => {
+ let now = 0;
+ const meter = new SocketDrainMeter(() => now);
+ // 25kB leaves every 500ms with a backlog behind it throughout: a 50kB/s link.
+ for (let i = 0; i < 40; i++) {
+ now += 500;
+ meter.note(25_000, 400_000);
+ }
+
+ expect(meter.bytesPerSecond).toBeGreaterThan(40_000);
+ expect(meter.bytesPerSecond).toBeLessThan(60_000);
+ // A LAG bound now: well under the megabyte, which on this link would be twenty seconds.
+ expect(meter.highWaterBytes()).toBe(MIN_HIGH_WATER_BYTES);
+ });
+
+ it("never drops the allowance below the floor the low-water mark needs", () => {
+ let now = 0;
+ const meter = new SocketDrainMeter(() => now);
+ for (let i = 0; i < 20; i++) {
+ now += 500;
+ meter.note(10, 400_000); // a trickle, with the viewer far behind
+ }
+
+ expect(meter.highWaterBytes()).toBe(MIN_HIGH_WATER_BYTES);
+ });
+});
+
describe("frame codec", () => {
it("round-trips an output frame", () => {
const frame = decodeTerminalFrame(
diff --git a/packages/web/src/features/terminal/terminal-connection.ts b/packages/web/src/features/terminal/terminal-connection.ts
new file mode 100644
index 00000000..c432108d
--- /dev/null
+++ b/packages/web/src/features/terminal/terminal-connection.ts
@@ -0,0 +1,336 @@
+/**
+ * The terminal socket's LIFE: the first attach, the heartbeat, and every reattach after a
+ * link that dropped it. Kept out of terminal-view.tsx because none of it is about xterm,
+ * and because a state machine with three timers is worth testing on its own.
+ *
+ * The reason it exists is a phone. On a mobile network the socket does not close politely:
+ * the radio hands over, the carrier's NAT reaps an idle flow, the browser freezes the tab,
+ * and what the page is left holding is a socket that will never carry another byte. Before
+ * this, that state was terminal — `onclose` painted "stream closed" and the pane was dead
+ * until the user reloaded the page — even though the pty on the server was fine and one
+ * reattach would have repainted it from the server's own snapshot.
+ *
+ * So: probe for silence, retry with a bounded backoff, and treat a returning tab or a
+ * returning network as a reason to try immediately rather than to wait out the timer.
+ * A retry first asks whether the terminal is still there, because a gone terminal is the
+ * one failure worth stopping on.
+ */
+import { TerminalOpcode, decodeFrame, encodeFrame, type DecodedFrame } from "./terminal-frames";
+
+/**
+ * How the client probes: one Ping per interval, and silence past the limit is a dead socket
+ * — but only from a server that has ANSWERED a probe. A server from before the heartbeat
+ * ignores the opcode, and an idle terminal on one is silent for perfectly good reasons; a
+ * client that judged that silence would reattach every minute, snapshot and all, forever.
+ */
+const HEARTBEAT_INTERVAL_MS = 15_000;
+const SILENCE_LIMIT_MS = 45_000;
+/**
+ * A tick this far late means the browser froze the tab's timers, not that the link died —
+ * a backgrounded tab gets roughly one timer a minute. The silence it "measured" while
+ * suspended says nothing, so that tick probes instead of dropping.
+ */
+const FROZEN_TICK_MS = HEARTBEAT_INTERVAL_MS * 3;
+/** How long a wake-up probe waits for its echo before the socket is declared dead. */
+const WAKE_PROBE_TIMEOUT_MS = 5_000;
+/**
+ * A dial that never reaches `onOpen` is given up on after this long. A phone's network can
+ * black-hole a handshake without ever closing it — CONNECTING forever, no heartbeat started,
+ * nothing to probe — so the dial carries its own clock: past it the socket is dropped like
+ * any dead one and the backoff ladder tries again.
+ */
+export const DIAL_TIMEOUT_MS = 15_000;
+/**
+ * The same clock for the existence probe a retry waits on: one small request, so a link that
+ * lets it hang would let it hang forever. Losing the race reads as the same transient
+ * failure a rejected probe is.
+ */
+export const RECHECK_TIMEOUT_MS = 10_000;
+
+/**
+ * Backoff for reattaching, in milliseconds; the last entry repeats. It stays short because
+ * the far end costs nothing to attach to (the pty is already running) and because the user
+ * is usually staring at the pane waiting for it to come back.
+ */
+export const RETRY_DELAYS_MS = [500, 1_000, 2_000, 4_000, 8_000, 15_000];
+
+export type TerminalConnectionStatus = "ready" | "reconnecting" | "error";
+
+/** The socket, as this module needs it: three callbacks in, two verbs out. */
+export interface SocketHandlers {
+ onOpen(): void;
+ onMessage(data: ArrayBuffer): void;
+ onClose(): void;
+}
+
+export interface AttachedSocket {
+ send(bytes: Uint8Array): void;
+ close(): void;
+}
+
+export interface TerminalConnectionDeps {
+ /**
+ * The first attach: which terminal this view is for. Throwing is fatal — it is the host's
+ * own policy (create a shell, honour a URL parameter) that failed, and repeating it could
+ * create a second terminal.
+ */
+ first(): Promise<string>;
+ /**
+ * Before a retry: is that terminal still there? `false` stops the retries for good (the
+ * shell exited and was reaped while the page was away); throwing is transient and retries.
+ */
+ recheck(id: string): Promise<boolean>;
+ open(id: string, handlers: SocketHandlers): AttachedSocket;
+ /** Everything that is not heartbeat traffic, in arrival order. */
+ onFrame(frame: DecodedFrame): void;
+ onStatus(status: TerminalConnectionStatus, detail: string): void;
+}
+
+export class TerminalConnection {
+ private socket: AttachedSocket | null = null;
+ private terminalId: string | null = null;
+ private stopped = false;
+ private attempt = 0;
+ private retryTimer: ReturnType<typeof setTimeout> | null = null;
+ private heartbeatTimer: ReturnType<typeof setInterval> | null = null;
+ /** The dial's clock: armed when a socket is created, disarmed when it opens. */
+ private dialTimer: ReturnType<typeof setTimeout> | null = null;
+ private lastSeenAt = 0;
+ private lastTickAt = 0;
+ /** Whether the far end speaks the heartbeat (see the silence limit above). */
+ private answersProbes = false;
+ /**
+ * True from the start of an attach to its socket: while `first()`/`recheck()` are pending.
+ * Nothing may start a second attach meanwhile — the host policy behind `first()` creates a
+ * terminal when it answers, and an answer that races a wake would create a second one.
+ */
+ private connecting = false;
+
+ constructor(private readonly deps: TerminalConnectionDeps) {}
+
+ start(): void {
+ void this.connect();
+ }
+
+ /** Stops for good: the pane is gone, or the shell exited and there is nothing to reattach to. */
+ stop(): void {
+ this.stopped = true;
+ this.clearRetry();
+ this.stopHeartbeat();
+ this.clearDial();
+ this.socket?.close();
+ this.socket = null;
+ }
+
+ /** True while a socket is open; false through every gap the user might type into. */
+ get connected(): boolean {
+ return this.socket !== null;
+ }
+
+ /** Sends one frame if there is a socket to send it on. Input typed into a gap is dropped. */
+ send(bytes: Uint8Array): void {
+ this.socket?.send(bytes);
+ }
+
+ /**
+ * "The tab is back" / "the network is back". With no socket it collapses the backoff to
+ * now; with one, it probes it — the socket a phone comes back to often looks open and is
+ * not, and waiting for the next heartbeat to notice is fifteen seconds of dead pane.
+ */
+ wake(): void {
+ if (this.stopped) return;
+ if (this.socket === null) {
+ // An attach already in flight IS the "right now" a wake asks for; letting a second
+ // one start would run the host's policy twice (two terminals) or open two sockets
+ // where the loser is never closed.
+ if (this.connecting) return;
+ this.clearRetry();
+ this.attempt = 0;
+ void this.connect();
+ return;
+ }
+ const probed = this.socket;
+ const askedAt = Date.now();
+ this.lastSeenAt = askedAt;
+ this.ping();
+ if (!this.answersProbes) return; // nothing to conclude from an unanswered probe
+ setTimeout(() => {
+ if (this.socket !== probed || this.stopped) return;
+ if (this.lastSeenAt > askedAt) return; // it answered
+ this.drop(probed);
+ }, WAKE_PROBE_TIMEOUT_MS);
+ }
+
+ private async connect(): Promise<void> {
+ if (this.stopped || this.socket !== null || this.connecting) return;
+ this.connecting = true;
+ try {
+ let id: string;
+ try {
+ if (this.terminalId === null) {
+ // No clock on this one, on purpose: `first()` is the host's own policy —
+ // repeating it because it is slow would create a second terminal, and failing it
+ // on a timer would kill an attach a slow link is still completing.
+ id = await this.deps.first();
+ this.terminalId = id;
+ } else {
+ id = this.terminalId;
+ const present = await this.within(this.deps.recheck(id), RECHECK_TIMEOUT_MS);
+ if (this.stopped) return; // a stop that landed while the probe was out
+ if (!present) {
+ this.stopped = true;
+ this.deps.onStatus("error", "terminal is gone");
+ return;
+ }
+ }
+ } catch (err) {
+ if (this.stopped) return;
+ const detail = err instanceof Error ? err.message : String(err);
+ // A first attach is the host's policy failing, and is fatal. A recheck is just this
+ // link being down again — the same thing the retry is for, and so is its timeout.
+ if (this.terminalId === null) {
+ this.stopped = true;
+ this.deps.onStatus("error", detail);
+ return;
+ }
+ this.deps.onStatus("reconnecting", detail);
+ this.scheduleRetry();
+ return;
+ }
+ if (this.stopped) return;
+
+ const socket: AttachedSocket = this.deps.open(id, {
+ onOpen: () => {
+ if (this.socket !== socket) return;
+ this.clearDial();
+ this.attempt = 0;
+ this.startHeartbeat();
+ this.deps.onStatus("ready", "");
+ },
+ onMessage: (data) => {
+ if (this.socket !== socket) return;
+ this.lastSeenAt = Date.now();
+ const frame = decodeFrame(data);
+ if (!frame) return;
+ // The heartbeat is this module's own traffic and never reaches the terminal.
+ if (frame.opcode === TerminalOpcode.Ping) {
+ socket.send(encodeFrame(TerminalOpcode.Pong, frame.text));
+ return;
+ }
+ if (frame.opcode === TerminalOpcode.Pong) {
+ this.answersProbes = true;
+ return;
+ }
+ this.deps.onFrame(frame);
+ },
+ onClose: () => this.drop(socket),
+ });
+ this.socket = socket;
+ // A new peer has answered nothing yet; what the previous socket answered says nothing
+ // about this one — the reattach may have landed on an older server.
+ this.answersProbes = false;
+ this.dialTimer = setTimeout(() => {
+ this.dialTimer = null;
+ this.drop(socket);
+ }, DIAL_TIMEOUT_MS);
+ } finally {
+ this.connecting = false;
+ }
+ }
+
+ /** Gives up on one socket and starts the countdown to the next. */
+ private drop(socket: AttachedSocket): void {
+ if (this.socket !== socket) return;
+ this.socket = null;
+ this.stopHeartbeat();
+ this.clearDial();
+ socket.close();
+ if (this.stopped) return;
+ this.deps.onStatus("reconnecting", "");
+ this.scheduleRetry();
+ }
+
+ private scheduleRetry(): void {
+ if (this.retryTimer !== null || this.stopped) return;
+ const step = Math.min(this.attempt, RETRY_DELAYS_MS.length - 1);
+ const base = RETRY_DELAYS_MS[step] as number;
+ this.attempt += 1;
+ // Jitter, because a dock full of terminals loses its link at the same instant and
+ // would otherwise reattach in one synchronised burst.
+ const delay = base + Math.random() * base * 0.2;
+ this.retryTimer = setTimeout(() => {
+ this.retryTimer = null;
+ void this.connect();
+ }, delay);
+ }
+
+ private clearRetry(): void {
+ if (this.retryTimer === null) return;
+ clearTimeout(this.retryTimer);
+ this.retryTimer = null;
+ }
+
+ private clearDial(): void {
+ if (this.dialTimer === null) return;
+ clearTimeout(this.dialTimer);
+ this.dialTimer = null;
+ }
+
+ /**
+ * Races a dependency call against a clock it cannot hang past. The losing call is left to
+ * settle on its own — its late answer lands in a continuation that checks `stopped` and
+ * the single-flight guard, so it changes nothing.
+ */
+ private within<T>(promise: Promise<T>, ms: number): Promise<T> {
+ return new Promise<T>((resolve, reject) => {
+ const timer = setTimeout(() => reject(new Error("the server did not answer in time")), ms);
+ promise.then(
+ (value) => {
+ clearTimeout(timer);
+ resolve(value);
+ },
+ (err: unknown) => {
+ clearTimeout(timer);
+ reject(err instanceof Error ? err : new Error(String(err)));
+ },
+ );
+ });
+ }
+
+ private ping(): void {
+ this.socket?.send(encodeFrame(TerminalOpcode.Ping, String(Date.now())));
+ }
+
+ private startHeartbeat(): void {
+ this.stopHeartbeat();
+ const now = Date.now();
+ this.lastSeenAt = now;
+ this.lastTickAt = now;
+ this.heartbeatTimer = setInterval(() => this.tick(), HEARTBEAT_INTERVAL_MS);
+ }
+
+ private stopHeartbeat(): void {
+ if (this.heartbeatTimer === null) return;
+ clearInterval(this.heartbeatTimer);
+ this.heartbeatTimer = null;
+ }
+
+ private tick(): void {
+ const socket = this.socket;
+ if (socket === null) return;
+ const now = Date.now();
+ const frozen = now - this.lastTickAt > FROZEN_TICK_MS;
+ this.lastTickAt = now;
+ if (frozen) {
+ // Time the tab spent suspended is not silence on the wire; ask, do not judge.
+ this.lastSeenAt = now;
+ this.ping();
+ return;
+ }
+ if (this.answersProbes && now - this.lastSeenAt > SILENCE_LIMIT_MS) {
+ this.drop(socket);
+ return;
+ }
+ this.ping();
+ }
+}
diff --git a/packages/web/src/features/terminal/terminal-frames.ts b/packages/web/src/features/terminal/terminal-frames.ts
index 26727249..e33b5d2f 100644
--- a/packages/web/src/features/terminal/terminal-frames.ts
+++ b/packages/web/src/features/terminal/terminal-frames.ts
@@ -4,6 +4,9 @@
*
* Output is decoded straight into a string and handed to xterm — no JSON parse on the hot
* path, which is what keeps a noisy build from stuttering the UI thread.
+ *
+ * Ping/Pong are symmetric: whoever probes sends its own clock reading and the peer echoes
+ * it back untouched, so each end can decide for itself that a silent socket is dead.
*/
export const TerminalOpcode = {
@@ -12,6 +15,10 @@ export const TerminalOpcode = {
Resize: 0x03,
Restore: 0x05,
Exit: 0x06,
+ /** Either direction: liveness probe; the peer echoes the payload back as Pong. */
+ Ping: 0x07,
+ /** Either direction: the echo of a Ping's payload, unchanged. */
+ Pong: 0x08,
} as const;
export type TerminalOpcode = (typeof TerminalOpcode)[keyof typeof TerminalOpcode];
diff --git a/packages/web/src/features/terminal/terminal-page.tsx b/packages/web/src/features/terminal/terminal-page.tsx
index 3eb5a2e2..732a383a 100644
--- a/packages/web/src/features/terminal/terminal-page.tsx
+++ b/packages/web/src/features/terminal/terminal-page.tsx
@@ -208,7 +208,7 @@ export function TerminalPage() {
className={`flex shrink-0 items-center gap-1 ${
status === "ready"
? chrome.success
- : status === "connecting"
+ : status === "connecting" || status === "reconnecting"
? chrome.attention
: chrome.danger
}`}
diff --git a/packages/web/src/features/terminal/terminal-view.tsx b/packages/web/src/features/terminal/terminal-view.tsx
index 49015efd..2a939856 100644
--- a/packages/web/src/features/terminal/terminal-view.tsx
+++ b/packages/web/src/features/terminal/terminal-view.tsx
@@ -12,12 +12,16 @@
* To restart with a different terminal, remount it (change the React `key`). On a touch
* device the surface also carries the key bar (terminal-keybar.tsx), so both hosts get the
* keys a soft keyboard lacks without either of them knowing about it.
+ *
+ * The socket itself is not managed here: terminal-connection.ts owns attaching, the
+ * heartbeat and reattaching after a link drops, so this file stays about xterm.
*/
import { useEffect, useRef, useState } from "react";
import "@xterm/xterm/css/xterm.css";
// Types only (erased at compile time): the xterm runtime stays behind loadXterm() below.
import type { ITheme, Terminal as XTerminal } from "@xterm/xterm";
-import { TerminalOpcode, decodeFrame, encodeFrame, encodeResize } from "./terminal-frames";
+import { TerminalOpcode, encodeFrame, encodeResize } from "./terminal-frames";
+import { TerminalConnection } from "./terminal-connection";
import { LinkClickTracker, openTerminalLink, positionFromPointer } from "./terminal-links";
import { writeClipboard } from "../../lib/clipboard";
import { TerminalKeyBar, type TerminalControl } from "./terminal-keybar";
@@ -60,7 +64,18 @@ export interface TerminalInfo {
title?: string | null;
}
-export type TerminalStatus = "connecting" | "ready" | "exited" | "error";
+/**
+ * How long the container has to hold still before its size is passed on. Long enough to
+ * swallow a soft keyboard's animation, short enough that a deliberate drag lands promptly.
+ */
+const RESIZE_SETTLE_MS = 120;
+
+/**
+ * `reconnecting` is its own word on purpose: it is the state a mobile link spends most of
+ * its bad minutes in, and it means something the other three do not — the pty is fine, this
+ * page's socket is not, and it is being reattached (terminal-connection.ts).
+ */
+export type TerminalStatus = "connecting" | "ready" | "reconnecting" | "exited" | "error";
/**
* The screen's own palette, one per appearance. Two things matter here.
@@ -311,7 +326,6 @@ export function TerminalView({
async function startTerminal(container: HTMLDivElement): Promise<() => void> {
const [{ Terminal }, { FitAddon }, { WebLinksAddon }, { ClipboardAddon }] = await loadXterm();
let disposed = false;
- let socket: WebSocket | null = null;
let exited = false;
const report = (status: TerminalStatus, detail = ""): void => {
@@ -736,18 +750,12 @@ export function TerminalView({
// another window attaches this one could never win its geometry back.
container.addEventListener(
"focusin",
- () => {
- if (socket?.readyState === WebSocket.OPEN) {
- socket.send(encodeResize(term.cols, term.rows, "claim"));
- }
- },
+ () => connection.send(encodeResize(term.cols, term.rows, "claim")),
{ signal },
);
const send = (data: string): void => {
- if (socket?.readyState === WebSocket.OPEN) {
- socket.send(encodeFrame(TerminalOpcode.Input, data));
- }
+ connection.send(encodeFrame(TerminalOpcode.Input, data));
};
term.onData((data) => {
@@ -769,56 +777,100 @@ export function TerminalView({
if (!disposed) callbacks.current.onTitle?.(title);
});
- void (async () => {
- try {
+ /**
+ * One connection, however many sockets that takes (terminal-connection.ts): it owns
+ * the first attach, the heartbeat and every reattach. This view only says what a
+ * frame means and what a status looks like.
+ */
+ const connection = new TerminalConnection({
+ first: async () => {
const terminal = await callbacks.current.ensure(term.cols, term.rows);
- if (disposed) return;
callbacks.current.onInfo?.(terminal);
-
- socket = new WebSocket(streamUrl(terminal.id, term.cols, term.rows, userId));
- socket.binaryType = "arraybuffer";
-
- socket.onopen = () => report("ready");
- socket.onmessage = (event) => {
- if (!(event.data instanceof ArrayBuffer)) return;
- const frame = decodeFrame(event.data);
- if (!frame) return;
- switch (frame.opcode) {
- // The restore stream is self-contained (reset + clear + repaint + cursor), so
- // it is written like any other output; calling term.reset() here would race
- // with xterm's parser instead.
- case TerminalOpcode.Restore:
- case TerminalOpcode.Output:
- term.write(frame.text);
- break;
- case TerminalOpcode.Exit: {
- const { exitCode } = JSON.parse(frame.text) as { exitCode: number };
- exited = true;
- report("exited", String(exitCode));
- break;
- }
- default:
- break;
- }
+ return terminal.id;
+ },
+ // A reattach that finds no terminal has nothing to come back to: the shell exited
+ // and was reaped while this page was away.
+ recheck: async (id) =>
+ (await probeJson<TerminalInfo>(`/api/terminals/${encodeURIComponent(id)}`)) !== null,
+ open: (id, handlers) => {
+ const ws = new WebSocket(streamUrl(id, term.cols, term.rows, userId));
+ ws.binaryType = "arraybuffer";
+ ws.onopen = () => handlers.onOpen();
+ ws.onmessage = (event) => {
+ if (event.data instanceof ArrayBuffer) handlers.onMessage(event.data);
};
- socket.onerror = () => report("error", "stream error");
- socket.onclose = () => {
- if (!exited) report("error", "stream closed");
+ // `error` is always followed by `close`; handling both would report twice.
+ ws.onclose = () => handlers.onClose();
+ return {
+ send: (bytes) => {
+ if (ws.readyState === WebSocket.OPEN) ws.send(bytes);
+ },
+ close: () => ws.close(),
};
- } catch (err) {
- report("error", err instanceof Error ? err.message : String(err));
- }
- })();
+ },
+ onFrame: (frame) => {
+ switch (frame.opcode) {
+ // The restore stream is self-contained (reset + clear + repaint + cursor), so
+ // it is written like any other output; calling term.reset() here would race
+ // with xterm's parser instead.
+ case TerminalOpcode.Restore:
+ case TerminalOpcode.Output:
+ term.write(frame.text);
+ break;
+ case TerminalOpcode.Exit: {
+ const { exitCode } = JSON.parse(frame.text) as { exitCode: number };
+ exited = true;
+ connection.stop(); // an exited pty is not something to reattach to
+ report("exited", String(exitCode));
+ break;
+ }
+ default:
+ break;
+ }
+ },
+ onStatus: (status, detail) => {
+ if (exited) return; // the exit already said what happened to this terminal
+ report(status, detail);
+ },
+ });
+ connection.start();
+
+ // A network that comes back, or a tab that does, is a reason to reattach now instead
+ // of sitting out the backoff — on a phone both happen all day. Waking also probes a
+ // socket that merely LOOKS open, which is what a suspended tab usually comes back to.
+ window.addEventListener("online", () => connection.wake(), { signal });
+ document.addEventListener(
+ "visibilitychange",
+ () => {
+ if (!document.hidden) connection.wake();
+ },
+ { signal },
+ );
// Geometry changes are `update`s: this connection claimed the size when it attached
// (?cols/?rows on the stream URL, and again on focusin above); an update only
// applies while this connection still holds ownership.
+ // Settled, not per pixel. A phone resizes this container constantly — the address bar
+ // collapses, the soft keyboard opens, the dock is dragged — and each size that reaches
+ // the pty costs a SIGWINCH and a full repaint from whatever is running, which on a slow
+ // link is the most expensive thing the terminal can do. So fit on the trailing edge,
+ // and only tell the server when the GRID actually changed; most pixel changes do not
+ // move it.
+ let resizeTimer: ReturnType<typeof setTimeout> | null = null;
+ let sentCols = term.cols;
+ let sentRows = term.rows;
const observer = new ResizeObserver(() => {
if (disposed) return;
- fit.fit();
- if (socket?.readyState === WebSocket.OPEN) {
- socket.send(encodeResize(term.cols, term.rows, "update"));
- }
+ if (resizeTimer !== null) clearTimeout(resizeTimer);
+ resizeTimer = setTimeout(() => {
+ resizeTimer = null;
+ if (disposed) return;
+ fit.fit();
+ if (term.cols === sentCols && term.rows === sentRows) return;
+ sentCols = term.cols;
+ sentRows = term.rows;
+ connection.send(encodeResize(term.cols, term.rows, "update"));
+ }, RESIZE_SETTLE_MS);
});
observer.observe(container);
term.focus();
@@ -840,7 +892,8 @@ export function TerminalView({
control.current = null;
listenerAbort.abort();
observer.disconnect();
- socket?.close();
+ if (resizeTimer !== null) clearTimeout(resizeTimer);
+ connection.stop();
term.dispose();
};
}
diff --git a/packages/web/src/lib/strings-en.ts b/packages/web/src/lib/strings-en.ts
index 59519d1f..8221880f 100644
--- a/packages/web/src/lib/strings-en.ts
+++ b/packages/web/src/lib/strings-en.ts
@@ -206,6 +206,8 @@ export const en: Strings = {
status: {
connecting: "connecting",
ready: "ready",
+ /** The pty is fine; this page's socket dropped and is being reattached. */
+ reconnecting: "reconnecting",
exited: "exited",
error: "error",
},
diff --git a/packages/web/src/lib/strings.ts b/packages/web/src/lib/strings.ts
index fd0e18f2..96da8f13 100644
--- a/packages/web/src/lib/strings.ts
+++ b/packages/web/src/lib/strings.ts
@@ -223,6 +223,8 @@ export const zh = {
status: {
connecting: "连接中",
ready: "已连接",
+ /** The pty is fine; this page's socket dropped and is being reattached. */
+ reconnecting: "重连中",
exited: "已退出",
error: "连接错误",
},
diff --git a/packages/web/test/terminal-connection.test.ts b/packages/web/test/terminal-connection.test.ts
new file mode 100644
index 00000000..5446623b
--- /dev/null
+++ b/packages/web/test/terminal-connection.test.ts
@@ -0,0 +1,525 @@
+/**
+ * The terminal socket's life on a link that keeps dropping it (terminal-connection.ts).
+ *
+ * These are the promises the pane depends on when the network is a phone's: a dropped
+ * socket comes back on its own, a suspended tab does not get mistaken for a dead link, a
+ * terminal that is genuinely gone stops the retries, and the heartbeat answers whoever
+ * probes it.
+ */
+import { afterEach, describe, expect, it, vi } from "vitest";
+import {
+ RETRY_DELAYS_MS,
+ TerminalConnection,
+ type AttachedSocket,
+ type SocketHandlers,
+ type TerminalConnectionStatus,
+} from "../src/features/terminal/terminal-connection";
+import { TerminalOpcode, decodeFrame, encodeFrame } from "../src/features/terminal/terminal-frames";
+
+/** One fake socket: records what was sent, and lets a test play the link's part. */
+class FakeSocket implements AttachedSocket {
+ readonly sent: Uint8Array[] = [];
+ closed = false;
+ constructor(readonly handlers: SocketHandlers) {}
+ send(bytes: Uint8Array): void {
+ this.sent.push(bytes);
+ }
+ close(): void {
+ this.closed = true;
+ }
+ /** The server accepted the upgrade. */
+ open(): void {
+ this.handlers.onOpen();
+ }
+ /** The server sent one frame. */
+ deliver(opcode: TerminalOpcode, text = ""): void {
+ this.handlers.onMessage(encodeFrame(opcode, text).buffer as ArrayBuffer);
+ }
+ /** The link went away. */
+ drop(): void {
+ this.handlers.onClose();
+ }
+ opcodes(): number[] {
+ return this.sent.map((bytes) => bytes[0] as number);
+ }
+}
+
+interface Harness {
+ connection: TerminalConnection;
+ sockets: FakeSocket[];
+ statuses: { status: TerminalConnectionStatus; detail: string }[];
+ frames: number[];
+ /** Whether a retry finds the terminal still on the server. */
+ setPresent(present: boolean): void;
+}
+
+function harness(
+ options: { first?: () => Promise<string>; recheck?: (id: string) => Promise<boolean> } = {},
+): Harness {
+ const sockets: FakeSocket[] = [];
+ const statuses: Harness["statuses"] = [];
+ const frames: number[] = [];
+ let present = true;
+ const connection = new TerminalConnection({
+ first: options.first ?? (() => Promise.resolve("t1")),
+ recheck: options.recheck ?? (() => Promise.resolve(present)),
+ open: (_id, handlers) => {
+ const socket = new FakeSocket(handlers);
+ sockets.push(socket);
+ return socket;
+ },
+ onFrame: (frame) => frames.push(frame.opcode),
+ onStatus: (status, detail) => statuses.push({ status, detail }),
+ });
+ return {
+ connection,
+ sockets,
+ statuses,
+ frames,
+ setPresent: (next) => {
+ present = next;
+ },
+ };
+}
+
+/** Lets the connection's own promises settle between timer steps. */
+const settle = () => vi.advanceTimersByTimeAsync(0);
+
+/** A promise the test settles by hand — the link that answers when it feels like it. */
+interface Deferred<T> {
+ promise: Promise<T>;
+ resolve: (value: T) => void;
+ reject: (err: unknown) => void;
+}
+
+function deferred<T>(): Deferred<T> {
+ let resolve!: (value: T) => void;
+ let reject!: (err: unknown) => void;
+ const promise = new Promise<T>((res, rej) => {
+ resolve = res;
+ reject = rej;
+ });
+ return { promise, resolve, reject };
+}
+
+afterEach(() => vi.useRealTimers());
+
+describe("terminal connection", () => {
+ it("reports ready once the socket opens", async () => {
+ vi.useFakeTimers();
+ const h = harness();
+ h.connection.start();
+ await settle();
+ h.sockets[0]?.open();
+
+ expect(h.statuses).toEqual([{ status: "ready", detail: "" }]);
+ });
+
+ it("reattaches after the link drops, and says so meanwhile", async () => {
+ vi.useFakeTimers();
+ const h = harness();
+ h.connection.start();
+ await settle();
+ h.sockets[0]?.open();
+ h.sockets[0]?.drop();
+
+ expect(h.statuses.at(-1)?.status).toBe("reconnecting");
+ expect(h.sockets).toHaveLength(1); // the retry is a wait, not a hot loop
+
+ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[0] as number) * 1.2);
+ expect(h.sockets).toHaveLength(2);
+ h.sockets[1]?.open();
+ expect(h.statuses.at(-1)?.status).toBe("ready");
+ });
+
+ it("backs off further with each failed attempt", async () => {
+ vi.useFakeTimers();
+ const h = harness();
+ h.connection.start();
+ await settle();
+ h.sockets[0]?.open();
+ h.sockets[0]?.drop();
+
+ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[0] as number) * 1.2);
+ h.sockets[1]?.drop(); // the second attach never opened
+ await vi.advanceTimersByTimeAsync(RETRY_DELAYS_MS[0] as number);
+ expect(h.sockets).toHaveLength(2); // the first delay is no longer enough
+
+ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[1] as number) * 1.2);
+ expect(h.sockets).toHaveLength(3);
+ });
+
+ it("stops retrying when the terminal is gone", async () => {
+ vi.useFakeTimers();
+ const h = harness();
+ h.connection.start();
+ await settle();
+ h.sockets[0]?.open();
+ h.setPresent(false);
+ h.sockets[0]?.drop();
+
+ await vi.advanceTimersByTimeAsync(60_000);
+
+ expect(h.sockets).toHaveLength(1);
+ expect(h.statuses.at(-1)).toEqual({ status: "error", detail: "terminal is gone" });
+ });
+
+ it("keeps the first attach fatal: the host's policy is not retried", async () => {
+ vi.useFakeTimers();
+ const h = harness({ first: () => Promise.reject(new Error("no terminal API")) });
+ h.connection.start();
+ await settle();
+ await vi.advanceTimersByTimeAsync(60_000);
+
+ expect(h.sockets).toHaveLength(0);
+ expect(h.statuses.at(-1)).toEqual({ status: "error", detail: "no terminal API" });
+ });
+
+ it("answers the server's probe and keeps it off the terminal", async () => {
+ vi.useFakeTimers();
+ const h = harness();
+ h.connection.start();
+ await settle();
+ const socket = h.sockets[0] as FakeSocket;
+ socket.open();
+ socket.deliver(TerminalOpcode.Ping, "1234");
+
+ const reply = socket.sent.at(-1) as Uint8Array;
+ expect(reply[0]).toBe(TerminalOpcode.Pong);
+ expect(decodeFrame(reply.buffer as ArrayBuffer)?.text).toBe("1234");
+ expect(h.frames).toEqual([]); // heartbeat traffic never reaches xterm
+ });
+
+ it("passes output through to the terminal", async () => {
+ vi.useFakeTimers();
+ const h = harness();
+ h.connection.start();
+ await settle();
+ const socket = h.sockets[0] as FakeSocket;
+ socket.open();
+ socket.deliver(TerminalOpcode.Output, "hello");
+
+ expect(h.frames).toEqual([TerminalOpcode.Output]);
+ });
+
+ it("drops a socket that stopped answering, and reattaches", async () => {
+ vi.useFakeTimers();
+ const h = harness();
+ h.connection.start();
+ await settle();
+ const socket = h.sockets[0] as FakeSocket;
+ socket.open();
+ socket.deliver(TerminalOpcode.Pong, "1"); // this server speaks the heartbeat
+
+ // Probe after probe goes unanswered; the tick that finds the silence past the limit
+ // gives up on the socket instead of probing again.
+ await vi.advanceTimersByTimeAsync(65_000);
+
+ expect(socket.opcodes()).toContain(TerminalOpcode.Ping);
+ expect(socket.closed).toBe(true);
+ expect(h.statuses.at(-1)?.status).toBe("reconnecting");
+ });
+
+ it("never judges the silence of a server that does not answer probes", async () => {
+ vi.useFakeTimers();
+ const h = harness();
+ h.connection.start();
+ await settle();
+ const socket = h.sockets[0] as FakeSocket;
+ socket.open(); // a server from before the heartbeat: it ignores the opcode
+
+ await vi.advanceTimersByTimeAsync(300_000);
+ h.connection.wake();
+ await vi.advanceTimersByTimeAsync(30_000);
+
+ // An idle terminal on such a server is quiet for good reasons, and reattaching every
+ // minute would repaint the screen from a snapshot for nothing.
+ expect(socket.closed).toBe(false);
+ expect(h.sockets).toHaveLength(1);
+ });
+
+ it("treats a frozen tab as a wake-up, not as a dead link", async () => {
+ vi.useFakeTimers();
+ const h = harness();
+ h.connection.start();
+ await settle();
+ const socket = h.sockets[0] as FakeSocket;
+ socket.open();
+ // This server answers probes, so its silence IS evidence — without this Pong the
+ // silence rule is never armed and the frozen tick below judges nothing at all.
+ socket.deliver(TerminalOpcode.Pong, "1");
+
+ // The browser suspended this tab: one tick fires, an hour late.
+ vi.setSystemTime(Date.now() + 3_600_000);
+ await vi.advanceTimersByTimeAsync(15_000);
+
+ expect(socket.closed).toBe(false);
+ expect(socket.opcodes().at(-1)).toBe(TerminalOpcode.Ping); // it asks instead of judging
+
+ // Unfrozen again, silence past the limit is still a dead socket — the frozen tick is an
+ // exemption for time the tab spent suspended, not a switch-off of the silence rule.
+ await vi.advanceTimersByTimeAsync(65_000);
+ expect(socket.closed).toBe(true);
+ expect(h.statuses.at(-1)?.status).toBe("reconnecting");
+ });
+
+ it("wakes into an immediate reattach instead of sitting out the backoff", async () => {
+ vi.useFakeTimers();
+ const h = harness();
+ h.connection.start();
+ await settle();
+ h.sockets[0]?.open();
+ h.sockets[0]?.drop();
+
+ h.connection.wake();
+ await settle();
+
+ expect(h.sockets).toHaveLength(2);
+ });
+
+ it("probes a socket that only looks open when the tab comes back", async () => {
+ vi.useFakeTimers();
+ const h = harness();
+ h.connection.start();
+ await settle();
+ const socket = h.sockets[0] as FakeSocket;
+ socket.open();
+ socket.deliver(TerminalOpcode.Pong, "1"); // this server speaks the heartbeat
+
+ h.connection.wake();
+ expect(socket.opcodes().at(-1)).toBe(TerminalOpcode.Ping);
+
+ await vi.advanceTimersByTimeAsync(6_000); // no echo came back
+ expect(socket.closed).toBe(true);
+ expect(h.statuses.at(-1)?.status).toBe("reconnecting");
+ });
+
+ it("keeps a socket that answers the wake-up probe", async () => {
+ vi.useFakeTimers();
+ const h = harness();
+ h.connection.start();
+ await settle();
+ const socket = h.sockets[0] as FakeSocket;
+ socket.open();
+ // Arm the deadline first: without a Pong the wake installs no 5-second timer, and
+ // "still open" below would say nothing about the answer cancelling it.
+ socket.deliver(TerminalOpcode.Pong, "1");
+
+ h.connection.wake();
+ expect(socket.opcodes().at(-1)).toBe(TerminalOpcode.Ping);
+
+ // The echo arrives within the deadline, the drop it scheduled is cancelled…
+ await vi.advanceTimersByTimeAsync(1_000);
+ socket.deliver(TerminalOpcode.Pong, "1");
+ // …and the socket is still here well past the deadline.
+ await vi.advanceTimersByTimeAsync(6_000);
+
+ expect(socket.closed).toBe(false);
+ });
+
+ it("drops input typed into a gap rather than replaying it later", async () => {
+ vi.useFakeTimers();
+ const h = harness();
+ h.connection.start();
+ await settle();
+ h.sockets[0]?.open();
+ h.sockets[0]?.drop();
+
+ h.connection.send(encodeFrame(TerminalOpcode.Input, "rm -rf /\r"));
+ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[0] as number) * 1.2);
+ h.sockets[1]?.open();
+
+ expect(h.sockets[1]?.sent).toEqual([]);
+ });
+
+ it("stops for good once the pane is gone", async () => {
+ vi.useFakeTimers();
+ const h = harness();
+ h.connection.start();
+ await settle();
+ h.sockets[0]?.open();
+ h.connection.stop();
+
+ expect(h.sockets[0]?.closed).toBe(true);
+ await vi.advanceTimersByTimeAsync(60_000);
+ expect(h.sockets).toHaveLength(1);
+ });
+
+ it("keeps one first attach in flight, however often the network returns during it", async () => {
+ vi.useFakeTimers();
+ const pending = deferred<string>();
+ let firstCalls = 0;
+ const h = harness({
+ first: () => {
+ firstCalls += 1;
+ return pending.promise;
+ },
+ });
+
+ h.connection.start();
+ await settle();
+ // online and visibilitychange fire repeatedly while the first attach is still out.
+ h.connection.wake();
+ h.connection.wake();
+ await settle();
+ expect(firstCalls).toBe(1); // the host policy runs once; twice could mean two terminals
+
+ pending.resolve("t1");
+ await settle();
+ expect(h.sockets).toHaveLength(1); // one socket, not one per wake
+ h.sockets[0]?.open();
+ expect(h.statuses.at(-1)).toEqual({ status: "ready", detail: "" });
+ });
+
+ it("keeps one recheck in flight: a wake during a pending retry opens no second socket", async () => {
+ vi.useFakeTimers();
+ const pending = deferred<boolean>();
+ let rechecks = 0;
+ const h = harness({
+ recheck: () => {
+ rechecks += 1;
+ return pending.promise;
+ },
+ });
+ h.connection.start();
+ await settle();
+ h.sockets[0]?.open();
+ h.sockets[0]?.drop();
+
+ // The first retry's recheck is out; the tab comes back twice before it answers.
+ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[0] as number) * 1.2);
+ expect(rechecks).toBe(1);
+ h.connection.wake();
+ h.connection.wake();
+ await settle();
+ expect(rechecks).toBe(1);
+
+ pending.resolve(true);
+ await settle();
+ expect(h.sockets).toHaveLength(2); // exactly one reattach, however many wakes
+ h.sockets[1]?.open();
+ expect(h.statuses.at(-1)?.status).toBe("ready");
+ });
+
+ it("gives up on a dial that never opens, closes it, retries — and ignores its late open", async () => {
+ vi.useFakeTimers();
+ const h = harness();
+ h.connection.start();
+ await settle();
+ // The link swallows the handshake: no onOpen, no onClose, CONNECTING forever.
+ const stalled = h.sockets[0] as FakeSocket;
+
+ // 15s: DIAL_TIMEOUT_MS in the source, written as a literal so this case also runs
+ // (red) on the tree before the fix, where the export does not exist yet.
+ await vi.advanceTimersByTimeAsync(15_000);
+
+ expect(stalled.closed).toBe(true);
+ expect(h.statuses.at(-1)?.status).toBe("reconnecting");
+
+ // The backoff ladder takes it from here…
+ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[0] as number) * 1.2);
+ expect(h.sockets).toHaveLength(2);
+
+ // …and a handshake that completes AFTER its dial was given up on changes nothing: the
+ // late open is not this connection's socket any more, so it is ignored — and the
+ // fresh dial is the one that can go ready.
+ stalled.open();
+ const fresh = h.sockets[1] as FakeSocket;
+ expect(fresh.closed).toBe(false);
+ expect(h.statuses.filter((s) => s.status === "ready")).toHaveLength(0);
+
+ fresh.open();
+ expect(h.statuses.at(-1)).toEqual({ status: "ready", detail: "" });
+ });
+
+ it("gives up on a recheck that never answers, and tries it again", async () => {
+ vi.useFakeTimers();
+ let rechecks = 0;
+ const h = harness({
+ recheck: () => {
+ rechecks += 1;
+ return new Promise<boolean>(() => {});
+ },
+ });
+ h.connection.start();
+ await settle();
+ h.sockets[0]?.open();
+ h.sockets[0]?.drop();
+
+ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[0] as number) * 1.2); // recheck #1 is out
+ const statusesBefore = h.statuses.length;
+ // 10s: RECHECK_TIMEOUT_MS in the source, a literal for the same reason as above.
+ await vi.advanceTimersByTimeAsync(10_000); // …and it never answers
+
+ expect(h.statuses.length).toBe(statusesBefore + 1); // a NEW status, not the drop's old one
+ expect(h.statuses.at(-1)?.status).toBe("reconnecting");
+
+ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[1] as number) * 1.2);
+ expect(rechecks).toBe(2); // the hang read as a transient failure, so the ladder went on
+ });
+
+ it("ends a pending recheck quietly when the pane is gone: nothing follows the stop", async () => {
+ vi.useFakeTimers();
+ const pending = deferred<boolean>();
+ const h = harness({ recheck: () => pending.promise });
+ h.connection.start();
+ await settle();
+ h.sockets[0]?.open();
+ h.sockets[0]?.drop();
+
+ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[0] as number) * 1.2); // recheck is out
+ const statusesAtStop = h.statuses.length;
+ h.connection.stop();
+ pending.reject(new Error("the link died mid-probe"));
+
+ await vi.advanceTimersByTimeAsync(60_000);
+
+ expect(h.statuses.length).toBe(statusesAtStop); // no reconnecting after the stop
+ expect(h.sockets).toHaveLength(1);
+ expect(h.sockets[0]?.closed).toBe(true);
+ });
+
+ it("judges a fresh socket by its own answers, not the previous one's", async () => {
+ vi.useFakeTimers();
+ const h = harness();
+ h.connection.start();
+ await settle();
+ const first = h.sockets[0] as FakeSocket;
+ first.open();
+ first.deliver(TerminalOpcode.Pong, "1"); // this server speaks the heartbeat
+
+ first.drop();
+ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[0] as number) * 1.2);
+ const second = h.sockets[1] as FakeSocket;
+ second.open(); // this one never answers a probe: maybe an older server came back
+
+ await vi.advanceTimersByTimeAsync(300_000);
+ h.connection.wake();
+ await vi.advanceTimersByTimeAsync(30_000);
+
+ // The previous socket's answers are not this socket's. An old server's silence is
+ // legitimate, so it survives both the silence rule and the wake.
+ expect(second.closed).toBe(false);
+ expect(h.sockets).toHaveLength(2);
+ });
+
+ it("still judges a fresh socket that answers by its silence", async () => {
+ vi.useFakeTimers();
+ const h = harness();
+ h.connection.start();
+ await settle();
+ const first = h.sockets[0] as FakeSocket;
+ first.open();
+ first.deliver(TerminalOpcode.Pong, "1");
+ first.drop();
+
+ await vi.advanceTimersByTimeAsync((RETRY_DELAYS_MS[0] as number) * 1.2);
+ const second = h.sockets[1] as FakeSocket;
+ second.open();
+ second.deliver(TerminalOpcode.Pong, "1"); // THIS one speaks the heartbeat too
+
+ await vi.advanceTimersByTimeAsync(300_000);
+ expect(second.closed).toBe(true); // its own silence, once it has answered
+ expect(h.sockets.at(-1)).not.toBe(second); // the connection moved on
+ });
+});