Skip to content

7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.

TypeScript

feat(stream): opt-in SSE keep-alive pings for the silent reasoning window (#1642) (#1674)

tashdroidffbd88feca9e
diff --git a/.env.example b/.env.example
index 99e43494..3f50f297 100644
--- a/.env.example
+++ b/.env.example
@@ -33,6 +33,15 @@ PORT=3001
# to disable proxy rate limiting.
# PROXY_RATE_LIMIT_RPM=120
+# SSE heartbeat interval in ms for streaming requests (#1642). Reasoning
+# models can go minutes without a first token, and clients with an idle
+# watchdog (Claude Code) abort during that silence. Set to e.g. 15000 and
+# each waiting stream emits a content-free SSE ping every 15s so the client
+# keeps waiting. OFF by default: a ping commits the response, so a stream
+# that dies after one ends with an error frame instead of failing over to
+# the next provider invisibly. Floor 1000 ms.
+# STREAM_KEEPALIVE_MS=15000
+
# Max /api dashboard requests per minute per client IP (default: 600). This is a
# flood guard — login has its own per-email lockout and key export its own much
# tighter cap. Set to 0 to disable.
diff --git a/docs/en/env/01-variables.md b/docs/en/env/01-variables.md
index cdeabf14..6777d534 100644
--- a/docs/en/env/01-variables.md
+++ b/docs/en/env/01-variables.md
@@ -60,6 +60,7 @@ First-run note: the first dashboard account is normally created from a browser o
| `PROVIDER_STREAM_STALL_TIMEOUT_MS` | `90000` | Mid-stream inactivity timeout in milliseconds. If an SSE stream goes this long without a single byte AFTER the first byte arrived, the gateway gives up: before any output reached the client it fails over to the next candidate; after output was sent the stream ends with an error frame. Time-to-FIRST-byte is governed by `PROVIDER_TIMEOUT_<PLATFORM>` instead (whichever of the two is larger). Providers that queue heavily on free tiers can pause longer than 90s mid-stream — raise this if streamed responses stop early. `0` disables the stall watchdog entirely. |
| `PROVIDER_STREAM_STALL_TIMEOUT_<PLATFORM>` | Falls through to the global value above, then the built-in `90000` | Per-provider variant (platform name upper-cased). Precedence: per-platform > global > built-in 90s default. Example: `PROVIDER_STREAM_STALL_TIMEOUT_OLLAMA=240000` lets Ollama Cloud pause 4 minutes mid-stream without loosening the watchdog everywhere else. |
| `PROVIDER_DAILY_REQUEST_CAP_<PLATFORM>` | Built-in default per provider | Per-provider daily request cap (platform name upper-cased). Overrides the built-in default for that provider; set to `0` to disable the cap. Example: `PROVIDER_DAILY_REQUEST_CAP_OPENROUTER=50`. |
+| `STREAM_KEEPALIVE_MS` | Off (`0`) | SSE heartbeat interval in milliseconds for streaming inference (#1642). Streaming responses hold every byte until the model's first token, so a reasoning model can leave the wire silent for minutes — long enough that clients with a time-to-first-byte watchdog (Claude Code's stream-idle timeout is the reference case) abort and the user reruns the turn. When set, each in-flight stream emits a content-free frame every interval while it is still uncommitted: a bare SSE comment (`: keepalive`) on OpenAI-shaped surfaces, the protocol's own `ping` event on `/v1/messages`. Spec-compliant parsers drop both, so clients see unchanged frames — but a ping commits the response, so a stream that then dies ends with an error frame instead of failing over invisibly to the next provider. That trade is what enabling this opts into; off by default keeps today's invisible failover. Floor 1000 ms. Read per request, so it applies immediately. |
| `LLMTR_VALIDATION_MIN_GAP_MS` | `2000` | Minimum gap, in milliseconds, between the start of two LLMTR key-validation probes (issue #1369). Probes already run one at a time, and concurrent checks of the same key share a single request; this spaces the survivors so a burst of identical probes never leaves the gateway back to back. This is a floor for our own traffic, not an LLMTR-published limit — their docs state none for this endpoint — chosen because a burst of the same nonexistent-model POST from one IP is the pattern their security system reads as automation, and an account flagged that way gets suspended. Covers every caller: the 5-minute health pass, the dashboard's "check all", the post-wake re-probe, the cooldown-probe job and the 401-triggered revalidation. Raise it if your account is still being flagged; `0` disables pacing. Chat and streaming traffic is never paced. Read at startup. |
| `FALLBACK_TIME_BUDGET_MS` | `45000` | Base wall-clock budget for provider failover, in milliseconds. The initial attempt and one failover hop are always allowed. After that, exhaustion stops new attempts and can cancel an in-flight attempt still waiting for its first usable response. Streaming responses disarm this timer when the first usable response arrives. Successful TTFB history can widen the budget for a slow endpoint, as described below. `0` disables the budget entirely, including adaptive widening, and retries until the attempt cap. Also settable at runtime via the `fallback_time_budget_ms` settings key, which takes precedence. |
| `SLOW_ENDPOINT_BUFFER_MS` | `10000` | Padding added to the historical P95 successful time-to-first-byte (TTFB), in milliseconds. Samples are grouped by platform + `endpoint_scope`; the effective fallback budget is `max(base budget, P95 TTFB + buffer)`, capped at three times the base budget, and only for endpoints whose P95 is at least half the base budget. Endpoints with fewer than `TTFB_BUDGET_MIN_SAMPLES` samples, and those whose P95 + buffer fits the base budget, keep the base budget. `0` removes extra padding. The runtime `slow_endpoint_buffer_ms` settings key takes precedence over this environment variable. |
diff --git a/docs/zh-cn/env/01-variables.md b/docs/zh-cn/env/01-variables.md
index 322ceda3..cdfefea5 100644
--- a/docs/zh-cn/env/01-variables.md
+++ b/docs/zh-cn/env/01-variables.md
@@ -57,6 +57,7 @@ FreeLLMAPI 从 `.env` 读取的全部变量,按主题分组。默认值和说
| `PROVIDER_STREAM_STALL_TIMEOUT_MS` | `90000` | 流中途的停滞超时,单位毫秒。如果一条 SSE 流在首字节已到达之后持续这么久没有任何字节,网关就放弃:若尚无输出到达客户端,则故障转移到下一个候选;若已有输出,则以一个错误帧结束流。到「第一个」字节为止的时间由 `PROVIDER_TIMEOUT_<PLATFORM>` 管(两者取较大者)。在免费额度上排长队的提供方可能在流中途停顿超过 90 秒——如果流式响应总是提前中断,就调高这个值。`0` 彻底关闭停滞看门狗。 |
| `PROVIDER_STREAM_STALL_TIMEOUT_<PLATFORM>` | 先回落到上面的全局值,再到内置的 `90000` | 按提供方的变体(平台名大写)。优先级:按平台 > 全局 > 内置 90 秒默认。示例:`PROVIDER_STREAM_STALL_TIMEOUT_OLLAMA=240000` 允许 Ollama Cloud 在流中途停顿 4 分钟,而其他地方的看门狗不受影响。 |
| `PROVIDER_DAILY_REQUEST_CAP_<PLATFORM>` | 各提供方内置的默认值 | 按提供方的每日请求上限(平台名大写)。覆盖该提供方的内置默认值;设为 `0` 关闭上限。示例:`PROVIDER_DAILY_REQUEST_CAP_OPENROUTER=50`。 |
+| `STREAM_KEEPALIVE_MS` | 关闭(`0`) | 流式推理的 SSE 心跳间隔(毫秒)(#1642)。流式响应会扣住所有字节直到模型吐出第一个 token,推理模型可能让连接静默数分钟——足以让带首字节看门狗的客户端(典型是 Claude Code 的 stream-idle 超时)中断请求、用户重跑整轮。设置后,每个未提交的在途流每隔该间隔发送一个无内容帧:OpenAI 形状接口发裸 SSE 注释行(`: keepalive`),`/v1/messages` 发协议自带的 `ping` 事件。符合规范的解析器会丢弃两者,客户端看到的帧不变——但心跳一旦发出响应即已提交,之后断流将以错误帧结束,而不是静默故障转移到下一个提供方。开启即接受该权衡;默认关闭保持现有的静默故障转移。下限 1000 毫秒。每次请求读取,改完即生效。 |
| `LLMTR_VALIDATION_MIN_GAP_MS` | `2000` | 两次 LLMTR 密钥校验探测开始之间至少间隔的毫秒数(issue #1369)。探测本来就是逐个串行发出的,同一密钥的并发检查也共用一次请求;这个值为其余探测拉开间距,使一串相同的探测绝不会紧接着离开网关。这是我们自己流量的下限,不是 LLMTR 公布的限制——他们的文档对该端点没有给出任何限制——之所以这样设,是因为同一 IP 短时间内反复发出相同的「不存在的模型」POST,正是他们的安全系统识别为自动化的模式,被标记的账号会被停用。它覆盖所有调用方:5 分钟健康巡检、仪表盘的「全部检查」、唤醒后的重新探测、冷却探测任务,以及 401 触发的重新校验。如果账号仍被标记,请调高该值;`0` 关闭节流。聊天与流式流量从不节流。启动时读取。 |
| `FALLBACK_TIME_BUDGET_MS` | `45000` | 提供方故障转移的基础挂钟时间预算,单位毫秒。始终允许首次尝试及一次故障转移;之后预算耗尽时会停止发起新尝试,也可能取消仍在等待首个有效响应的进行中尝试。流式响应收到首个有效响应后会解除此计时器。成功请求的首字节耗时历史可为慢端点延长预算,详见下方。`0` 彻底关闭该预算(包括自适应延长),一直重试直到触及尝试次数上限。也可以在运行时通过 `fallback_time_budget_ms` 设置键修改,后者优先。 |
| `SLOW_ENDPOINT_BUFFER_MS` | `10000` | 在历史成功请求的首字节耗时(TTFB)P95 上增加的缓冲,单位毫秒。样本按平台 + `endpoint_scope` 分组,有效故障转移预算为 `max(基础预算, P95 TTFB + 缓冲)`,上限为基础预算的三倍,且仅适用于 P95 不低于基础预算一半的端点。样本数少于 `TTFB_BUDGET_MIN_SAMPLES` 的端点,以及 P95 加缓冲不超过基础预算的端点,仍使用基础预算。`0` 表示不额外增加缓冲。运行时的 `slow_endpoint_buffer_ms` 设置键优先于此环境变量。 |
diff --git a/server/src/__tests__/lib/module-purity.test.ts b/server/src/__tests__/lib/module-purity.test.ts
index 7592e9fd..d3d1d863 100644
--- a/server/src/__tests__/lib/module-purity.test.ts
+++ b/server/src/__tests__/lib/module-purity.test.ts
@@ -43,6 +43,8 @@ const PURE_MODULES = [
'retry-hint.ts',
// Search platform list; the provider list, keys route and search service import it.
'search-platforms.ts',
+ // SSE keep-alive helper (#1642); env is injected, the response is a callback.
+ 'stream-keepalive.ts',
'structured-output.ts',
'tool-args.ts',
'tool-call-rescue.ts',
diff --git a/server/src/__tests__/lib/stream-keepalive.test.ts b/server/src/__tests__/lib/stream-keepalive.test.ts
new file mode 100644
index 00000000..3752e956
--- /dev/null
+++ b/server/src/__tests__/lib/stream-keepalive.test.ts
@@ -0,0 +1,70 @@
+import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
+import { streamKeepaliveMs, startStreamKeepalive } from '../../lib/stream-keepalive.js';
+
+// #1642: SSE keep-alive pings for the uncommitted streaming window. The
+// helper carries the env parsing and the timer lifecycle; the route pumps
+// wire it to a real response (covered by proxy-stream-keepalive.test.ts).
+
+describe('streamKeepaliveMs (#1642)', () => {
+ it('defaults to 0 (feature off) when STREAM_KEEPALIVE_MS is unset or blank', () => {
+ expect(streamKeepaliveMs({})).toBe(0);
+ expect(streamKeepaliveMs({ STREAM_KEEPALIVE_MS: '' })).toBe(0);
+ expect(streamKeepaliveMs({ STREAM_KEEPALIVE_MS: ' ' })).toBe(0);
+ });
+
+ it('treats 0, negatives, and garbage as off', () => {
+ expect(streamKeepaliveMs({ STREAM_KEEPALIVE_MS: '0' })).toBe(0);
+ expect(streamKeepaliveMs({ STREAM_KEEPALIVE_MS: '-5' })).toBe(0);
+ expect(streamKeepaliveMs({ STREAM_KEEPALIVE_MS: 'abc' })).toBe(0);
+ expect(streamKeepaliveMs({ STREAM_KEEPALIVE_MS: 'Infinity' })).toBe(0);
+ });
+
+ it('accepts a positive interval and floors it at 1s', () => {
+ expect(streamKeepaliveMs({ STREAM_KEEPALIVE_MS: '15000' })).toBe(15000);
+ expect(streamKeepaliveMs({ STREAM_KEEPALIVE_MS: '1e9' })).toBe(1e9);
+ expect(streamKeepaliveMs({ STREAM_KEEPALIVE_MS: '5' })).toBe(1000);
+ expect(streamKeepaliveMs({ STREAM_KEEPALIVE_MS: '2500.7' })).toBe(2500);
+ });
+});
+
+describe('startStreamKeepalive (#1642)', () => {
+ beforeEach(() => vi.useFakeTimers());
+ afterEach(() => vi.useRealTimers());
+
+ it('does nothing when the interval is 0 (no timer is created)', () => {
+ const keepalive = vi.fn();
+ const stop = startStreamKeepalive(0, () => false, keepalive);
+ vi.advanceTimersByTime(10_000);
+ expect(keepalive).not.toHaveBeenCalled();
+ expect(() => stop()).not.toThrow();
+ });
+
+ it('fires on each tick until the attempt settles', () => {
+ let done = false;
+ const keepalive = vi.fn();
+ startStreamKeepalive(1000, () => done, keepalive);
+ vi.advanceTimersByTime(3500);
+ expect(keepalive).toHaveBeenCalledTimes(3);
+ done = true;
+ vi.advanceTimersByTime(5000);
+ expect(keepalive).toHaveBeenCalledTimes(3);
+ });
+
+ it('stops after a write failure (dead socket)', () => {
+ const keepalive = vi.fn(() => { throw new Error('socket gone'); });
+ startStreamKeepalive(1000, () => false, keepalive);
+ vi.advanceTimersByTime(5000);
+ expect(keepalive).toHaveBeenCalledTimes(1);
+ });
+
+ it('stop() is idempotent and prevents further ticks', () => {
+ const keepalive = vi.fn();
+ const stop = startStreamKeepalive(1000, () => false, keepalive);
+ vi.advanceTimersByTime(1500);
+ expect(keepalive).toHaveBeenCalledTimes(1);
+ stop();
+ stop();
+ vi.advanceTimersByTime(5000);
+ expect(keepalive).toHaveBeenCalledTimes(1);
+ });
+});
diff --git a/server/src/__tests__/routes/proxy-stream-keepalive.test.ts b/server/src/__tests__/routes/proxy-stream-keepalive.test.ts
new file mode 100644
index 00000000..a88d437c
--- /dev/null
+++ b/server/src/__tests__/routes/proxy-stream-keepalive.test.ts
@@ -0,0 +1,160 @@
+import { describe, it, expect, beforeAll, beforeEach, afterEach, vi } from 'vitest';
+import type { Express } from 'express';
+import { createApp } from '../../app.js';
+import { initDb, getDb, getUnifiedApiKey } from '../../db/index.js';
+import { mintDashboardToken } from '../helpers/auth.js';
+
+// #1642: SSE keep-alive pings through the real proxy route. Upstreams here
+// go quiet before the first token (the reasoning-window shape that made
+// Claude Code abort live); with STREAM_KEEPALIVE_MS set the response must
+// carry comment-line pings during that silence, and with it unset (default)
+// the byte stream must be byte-for-byte what it was before.
+
+async function streamRequest(app: Express, body: any, signal?: AbortSignal) {
+ const server = app.listen(0, '127.0.0.1');
+ if (!server.listening) await new Promise<void>(resolve => server.once('listening', () => resolve()));
+ const addr = server.address() as any;
+ const res = await fetch(`http://127.0.0.1:${addr.port}/v1/chat/completions`, {
+ method: 'POST',
+ headers: { 'Content-Type': 'application/json', Authorization: 'Bearer ' + getUnifiedApiKey() },
+ body: JSON.stringify(body),
+ signal,
+ });
+ const text = await res.text();
+ server.close();
+ return { status: res.status, headers: res.headers, text };
+}
+
+const sse = (...payloads: (object | string)[]) =>
+ payloads.map(p => `data: ${typeof p === 'string' ? p : JSON.stringify(p)}\n\n`).join('');
+
+const roleChunk = { id: 'c1', object: 'chat.completion.chunk', created: 1, model: 'm', choices: [{ index: 0, delta: { role: 'assistant', content: null }, finish_reason: null }] };
+const textChunk = (s: string, finish: string | null = null) => ({ id: 'c1', object: 'chat.completion.chunk', created: 1, model: 'm', choices: [{ index: 0, delta: { content: s }, finish_reason: finish }] });
+const finishChunk = (reason: string) => ({ id: 'c1', object: 'chat.completion.chunk', created: 1, model: 'm', choices: [{ index: 0, delta: {}, finish_reason: reason }] });
+
+/** Upstream whose first SSE bytes arrive `delayMs` after the request. */
+function mockSilentThenStream(delayMs: number, body: string) {
+ const origFetch = global.fetch;
+ let calls = 0;
+ vi.spyOn(global, 'fetch').mockImplementation(async (url, init) => {
+ const urlStr = typeof url === 'string' ? url : url.toString();
+ // Only intercept provider upstreams; the test's own localhost request
+ // goes through.
+ if (/127\.0\.0\.1|localhost/.test(urlStr)) return origFetch(url as any, init);
+ calls++;
+ const stream = new ReadableStream<Uint8Array>({
+ start(controller) {
+ setTimeout(() => {
+ controller.enqueue(new TextEncoder().encode(body));
+ controller.close();
+ }, delayMs);
+ },
+ });
+ return new Response(stream, { status: 200, headers: { 'Content-Type': 'text/event-stream' } });
+ });
+ return { calls: () => calls };
+}
+
+/** Upstream that goes quiet for `delayMs`, then dies mid-response. */
+function mockSilentThenDie(delayMs: number) {
+ const origFetch = global.fetch;
+ let calls = 0;
+ vi.spyOn(global, 'fetch').mockImplementation(async (url, init) => {
+ const urlStr = typeof url === 'string' ? url : url.toString();
+ if (/127\.0\.0\.1|localhost/.test(urlStr)) return origFetch(url as any, init);
+ calls++;
+ const stream = new ReadableStream<Uint8Array>({
+ start(controller) {
+ setTimeout(() => {
+ controller.error(new Error('upstream exploded mid-stream'));
+ }, delayMs);
+ },
+ });
+ return new Response(stream, { status: 200, headers: { 'Content-Type': 'text/event-stream' } });
+ });
+ return { calls: () => calls };
+}
+
+describe('proxy SSE keep-alive (#1642)', () => {
+ let app: Express;
+ let dashToken = '';
+
+ beforeAll(() => {
+ process.env.ENCRYPTION_KEY = '0'.repeat(64);
+ initDb(':memory:');
+ app = createApp();
+ dashToken = mintDashboardToken();
+ });
+
+ beforeEach(async () => {
+ const db = getDb();
+ db.prepare('DELETE FROM api_keys').run();
+ db.prepare('DELETE FROM requests').run();
+ db.prepare('DELETE FROM rate_limit_cooldowns').run();
+ db.prepare('DELETE FROM rate_limit_usage').run();
+ const { status } = await restKey();
+ expect(status).toBe(201);
+ });
+
+ afterEach(() => {
+ vi.restoreAllMocks();
+ delete process.env.STREAM_KEEPALIVE_MS;
+ });
+
+ async function restKey() {
+ // mint a groq key through the dashboard API (same shape as the other
+ // stream-integrity suites)
+ const server = app.listen(0, '127.0.0.1');
+ if (!server.listening) await new Promise<void>(resolve => server.once('listening', () => resolve()));
+ const addr = server.address() as any;
+ const res = await fetch(`http://127.0.0.1:${addr.port}/api/keys`, {
+ method: 'POST',
+ headers: { 'Content-Type': 'application/json', Authorization: 'Bearer ' + dashToken },
+ body: JSON.stringify({ platform: 'groq', key: 'gsk_keepalive', label: 't' }),
+ });
+ const status = res.status;
+ await res.json().catch(() => null);
+ server.close();
+ return { status };
+ }
+
+ it('emits comment-line pings during a silent reasoning window when enabled', async () => {
+ process.env.STREAM_KEEPALIVE_MS = '1000'; // 1s is the helper's floor
+ // 2.6s of silence upstream ≈ 2 scheduled pings (1000ms floor).
+ const up = mockSilentThenStream(2600, sse(roleChunk, textChunk('finally.'), finishChunk('stop'), '[DONE]'));
+ const r = await streamRequest(app, { stream: true, messages: [{ role: 'user', content: 'keepalive ping test' }] });
+ expect(r.status).toBe(200);
+ expect(up.calls()).toBe(1);
+ const pings = r.text.split('\n').filter(l => l === ': keepalive');
+ expect(pings.length).toBeGreaterThanOrEqual(2);
+ // Every ping is a bare SSE comment: a spec parser drops it, so the JSON
+ // frames the client sees are unchanged.
+ const frames = r.text.split('\n')
+ .filter(l => l.startsWith('data: ') && l.trim() !== 'data: [DONE]')
+ .map(l => JSON.parse(l.slice(6)));
+ expect(frames.some(f => f.choices?.[0]?.delta?.content === 'finally.')).toBe(true);
+ }, 15000);
+
+ it('default (unset): a silent window streams no pings at all', async () => {
+ expect(process.env.STREAM_KEEPALIVE_MS).toBeUndefined();
+ const up = mockSilentThenStream(500, sse(roleChunk, textChunk('no heartbeat here.'), finishChunk('stop'), '[DONE]'));
+ const r = await streamRequest(app, { stream: true, messages: [{ role: 'user', content: 'keepalive default-off test' }] });
+ expect(r.status).toBe(200);
+ expect(up.calls()).toBe(1);
+ expect(r.text).not.toContain(': keepalive');
+ }, 15000);
+
+ it('once a ping committed the response, a dying stream ends honestly instead of failing over', async () => {
+ // The documented trade of opting in: after bytes hit the wire there is no
+ // invisible retry. The client gets an SSE error frame, not a hang, and
+ // exactly one upstream was tried.
+ process.env.STREAM_KEEPALIVE_MS = '1000'; // 1s is the helper's floor
+ const up = mockSilentThenDie(1500);
+ const r = await streamRequest(app, { stream: true, messages: [{ role: 'user', content: 'keepalive committed trade test' }] });
+ expect(r.status).toBe(200);
+ expect(up.calls()).toBe(1);
+ expect(r.text).toContain(': keepalive');
+ expect(r.text).toContain('"stream_error"');
+ expect(r.text).toContain('data: [DONE]');
+ }, 15000);
+});
diff --git a/server/src/lib/stream-keepalive.ts b/server/src/lib/stream-keepalive.ts
new file mode 100644
index 00000000..529113c6
--- /dev/null
+++ b/server/src/lib/stream-keepalive.ts
@@ -0,0 +1,77 @@
+// SSE keep-alive pings for the uncommitted streaming window (#1642).
+//
+// The streaming routes hold every byte — headers included — until the first
+// real text token, so a reasoning model can leave the wire silent for 30-180s
+// while it thinks. Clients with a time-to-first-byte watchdog (Claude Code's
+// stream-idle timeout is the reference case) abort during that window, the
+// router logs "client disconnected mid-attempt", and the user reruns the turn.
+//
+// When STREAM_KEEPALIVE_MS is set, each streaming pump starts an interval
+// that emits a content-free keep-alive frame while the attempt is in flight.
+// On the OpenAI surfaces that is a bare SSE comment line (`: keepalive`),
+// which every spec-compliant parser ignores; on the Anthropic surface it is
+// the protocol's own `ping` event.
+//
+// The invariant this preserves: the ping carries NO content, so nothing the
+// client can act on has been sent. What it necessarily gives up when enabled
+// is invisible failover FOR THAT ATTEMPT — a ping flushes the SSE headers, so
+// a stream that then dies ends the response with an error frame instead of
+// retrying another provider behind the scenes. That trade is exactly what an
+// operator opts into; the default is OFF, keeping the current behavior for
+// everyone who has not heard of the problem.
+//
+// Pings keep firing on a committed-but-silent stream too (upstream stalled
+// mid-reasoning), which is also useful: an idle upstream no longer looks
+// different from an idle connection at the client either way, but the extra
+// bytes stop the client from killing a stream that is merely slow.
+const MIN_KEEPALIVE_MS = 1000;
+
+/**
+ * The keep-alive interval in ms, parsed from STREAM_KEEPALIVE_MS.
+ * Unset / empty / 0 / non-numeric / negative → 0 (feature off, default).
+ * Positive values are clamped to a 1s floor: a sub-second ping interval just
+ * burns bandwidth and adds SSE frames no client asked for.
+ */
+export function streamKeepaliveMs(env: NodeJS.ProcessEnv = process.env): number {
+ const raw = env.STREAM_KEEPALIVE_MS;
+ if (raw === undefined || raw.trim() === '') return 0;
+ const n = Number(raw);
+ if (!Number.isFinite(n) || n <= 0) return 0;
+ return Math.max(MIN_KEEPALIVE_MS, Math.floor(n));
+}
+
+/**
+ * Start the keep-alive interval for one stream attempt.
+ *
+ * - `intervalMs` ≤ 0 disables the feature: no timer is created and the
+ * returned stop function is a no-op.
+ * - `isDone` is polled at every tick: once the attempt has finished (response
+ * ended, failed over, or errored), the timer stops itself.
+ * - `keepalive` runs on each tick and must write a content-free frame. It is
+ * expected to commit the response first if it has not been committed yet.
+ * Throwing (dead socket) stops the timer without surfacing anything —
+ * the pump's own error paths notice the broken connection.
+ *
+ * Returns the stop function; call it when the stream ends. Idempotent.
+ */
+export function startStreamKeepalive(
+ intervalMs: number,
+ isDone: () => boolean,
+ keepalive: () => void,
+): () => void {
+ if (intervalMs <= 0) return () => {};
+ const timer = setInterval(() => {
+ if (isDone()) { stop(); return; }
+ try {
+ keepalive();
+ } catch {
+ stop();
+ }
+ }, intervalMs);
+ // Never hold the process open for a ping that outlived its request.
+ if (typeof timer.unref === 'function') timer.unref();
+ function stop(): void {
+ clearInterval(timer);
+ }
+ return stop;
+}
diff --git a/server/src/routes/anthropic.ts b/server/src/routes/anthropic.ts
index 9c3302a5..6cfc47e1 100644
--- a/server/src/routes/anthropic.ts
+++ b/server/src/routes/anthropic.ts
@@ -24,6 +24,7 @@ import { logRequest } from '../lib/request-log.js';
import { extractApiToken, timingSafeStringEqual, getStickyModel, setStickyModel } from './proxy.js';
import { runFallbackLoop, newFallbackState, fallbackRoutingTokens, recordUpstreamSuccess, type ExhaustionBody, setFallbackHeaders, setExhaustionHeaders, type AttemptRecord, type FallbackState, REASONING_HOLD_MAX_MS } from '../lib/fallback-loop.js';
import { routedViaValue } from '../lib/header-value.js';
+import { startStreamKeepalive, streamKeepaliveMs } from '../lib/stream-keepalive.js';
import { applyTokenBudget, tokenBudgetMessage } from '../lib/guardrails.js';
import { resolveAnthropicModel, claudeFamilyDiscoveryEntries } from '../services/anthropic-map.js';
import { isUnifyEnabled, getModelGroups, resolveRequestedIdForDispatch } from '../services/model-groups.js';
@@ -880,11 +881,7 @@ async function streamCompletion(
const ensureMessageStart = () => {
if (messageStarted) return;
- res.setHeader('Content-Type', 'text/event-stream');
- res.setHeader('Cache-Control', 'no-cache');
- res.setHeader('Connection', 'keep-alive');
- res.setHeader('X-Routed-Via', routedViaValue(route.platform, route.modelId));
- setFallbackHeaders(res, ctx.attempt, ctx.attemptLog);
+ sendSseHeaders();
// Committed: the answer is on its way, so the retry budget must no longer
// cancel this attempt (it could not fail over now anyway).
ctx.disarmHedge();
@@ -899,6 +896,37 @@ async function streamCompletion(
messageStarted = true;
};
+ // Send the SSE headers once. Split out of ensureMessageStart so the
+ // #1642 keep-alive can commit the response with the protocol's own content-
+ // free `ping` event while the reasoning buffer is still filling, without
+ // sending message_start early.
+ let headersSentEarly = false;
+ const sendSseHeaders = () => {
+ if (headersSentEarly) return;
+ res.setHeader('Content-Type', 'text/event-stream');
+ res.setHeader('Cache-Control', 'no-cache');
+ res.setHeader('Connection', 'keep-alive');
+ res.setHeader('X-Routed-Via', routedViaValue(route.platform, route.modelId));
+ setFallbackHeaders(res, ctx.attempt, ctx.attemptLog);
+ headersSentEarly = true;
+ };
+
+ // #1642: optional heartbeat for the silent reasoning window. The Anthropic
+ // streaming protocol carries a native ping event, so no fake frame is
+ // needed. Once it fires the attempt is committed to the wire and can no
+ // longer fail over invisibly — that is the trade STREAM_KEEPALIVE_MS opts
+ // into (off by default).
+ let streamSettled = false;
+ const keepaliveStop = startStreamKeepalive(streamKeepaliveMs(),
+ () => streamSettled,
+ () => {
+ sendSseHeaders();
+ // Bytes on the wire = the response can no longer fail over invisibly;
+ // stop the retry budget from cancelling it.
+ ctx.disarmHedge();
+ writeSse(res, 'ping', { type: 'ping' });
+ });
+
// Reasoning output (delta.reasoning_content from providers / the <think>
// extractor, delta.reasoning from Ollama-style upstreams) is buffered and
// emitted as one complete thinking block right before the first text/tool
@@ -1116,7 +1144,10 @@ async function streamCompletion(
// stops the ladder without benching or an error log row (the socket is
// gone; nothing to render).
if (isClientAbortError(err)) throw err;
- if (messageStarted) {
+ // messageStarted = real content went out. headersSentEarly also counts: a
+ // #1642 keep-alive ping flushed the SSE headers, so this attempt is on the
+ // wire and can no longer fail over invisibly.
+ if (messageStarted || headersSentEarly) {
// Real payload already reached the client — finish the SSE response
// honestly instead of leaving Claude Code hanging, and stop the retry loop.
writeSse(res, 'error', { type: 'error', error: { type: 'api_error', message: `Provider error (${route.displayName}): stream interrupted` } });
@@ -1126,6 +1157,11 @@ async function streamCompletion(
}
// Headers never sent — bubble to the outer loop for failover.
throw err;
+ } finally {
+ // #1642: this attempt is over whichever path it left by — the keep-alive
+ // interval must not outlive it.
+ streamSettled = true;
+ keepaliveStop();
}
}
diff --git a/server/src/routes/proxy.ts b/server/src/routes/proxy.ts
index b2ca76b0..adf6d6bb 100644
--- a/server/src/routes/proxy.ts
+++ b/server/src/routes/proxy.ts
@@ -41,6 +41,7 @@ import { buildModelListing, type NormalizedModel } from '../services/model-listi
import { claudeFamilyDiscoveryEntries, resolveListedClaudeAlias } from '../services/anthropic-map.js';
import { compressRequest, formatCompressionHeader } from '../services/compression/pipeline.js';
import { applyDecisionRouting, routingStrategyHeader } from '../services/routing-decision.js';
+import { startStreamKeepalive, streamKeepaliveMs } from '../lib/stream-keepalive.js';
export const proxyRouter = Router();
@@ -1558,6 +1559,24 @@ proxyRouter.post('/completions', async (req: Request, res: Response) => {
let sawText = false;
let upstreamFinish: string | null = null;
const buffered: unknown[] = [];
+ // True once this attempt is over (success, committed error, or thrown)
+ // — the keep-alive interval polls it and stops itself.
+ let streamSettled = false;
+
+ // Send the SSE headers once. Split out of flushHeaders so the
+ // #1642 keep-alive can commit the response with a content-free ping
+ // while the hold window is still open, without dumping the buffered
+ // (possibly reasoning-bearing) frames early.
+ let headersSentEarly = false;
+ const sendSseHeaders = () => {
+ if (headersSentEarly) return;
+ res.setHeader('Content-Type', 'text/event-stream');
+ res.setHeader('Cache-Control', 'no-cache');
+ res.setHeader('Connection', 'keep-alive');
+ res.setHeader('X-Routed-Via', routedViaValue(route.platform, route.modelId));
+ setFallbackHeaders(res, attempt, attemptLog);
+ headersSentEarly = true;
+ };
const flushHeaders = () => {
if (headerSent) return;
@@ -1565,11 +1584,7 @@ proxyRouter.post('/completions', async (req: Request, res: Response) => {
// reasoning) in the pump loop below; this call only backfills streams
// that reached the commit point without one.
if (ttfbMs === null) ttfbMs = Date.now() - start;
- res.setHeader('Content-Type', 'text/event-stream');
- res.setHeader('Cache-Control', 'no-cache');
- res.setHeader('Connection', 'keep-alive');
- res.setHeader('X-Routed-Via', routedViaValue(route.platform, route.modelId));
- setFallbackHeaders(res, attempt, attemptLog);
+ sendSseHeaders();
headerSent = true;
// Committed: the answer is on its way, so the retry budget must no
// longer cancel this attempt (it could not fail over now anyway).
@@ -1578,6 +1593,18 @@ proxyRouter.post('/completions', async (req: Request, res: Response) => {
buffered.length = 0;
};
+ // #1642: optional heartbeat for the silent reasoning window. A bare
+ // SSE comment resets client idle watchdogs without committing content.
+ const keepaliveStop = startStreamKeepalive(streamKeepaliveMs(),
+ () => streamSettled,
+ () => {
+ sendSseHeaders();
+ // Bytes on the wire = the response can no longer fail over
+ // invisibly; stop the retry budget from cancelling it.
+ ctx.disarmHedge();
+ res.write(': keepalive\n\n');
+ });
+
try {
const gen = route.provider.streamChatCompletion(
route.apiKey,
@@ -1658,7 +1685,10 @@ proxyRouter.post('/completions', async (req: Request, res: Response) => {
// client-abort branch stops the ladder without benching or an
// error log row (the socket is gone; nothing to render).
if (isClientAbortError(streamErr)) throw streamErr;
- if (headerSent) {
+ // headerSent = real content went out. headersSentEarly also counts:
+ // a #1642 keep-alive ping flushed the SSE headers, so the response
+ // is on the wire and this attempt can no longer fail over invisibly.
+ if (headerSent || headersSentEarly) {
console.error(`[Proxy] Mid-stream legacy completion error from ${route.displayName}:`, streamErr.message);
const payload = { error: { message: `Provider error (${route.displayName}): stream interrupted`, type: 'stream_error' } };
try { res.write(`data: ${JSON.stringify(payload)}\n\n`); } catch { /* socket gone */ }
@@ -1676,6 +1706,11 @@ proxyRouter.post('/completions', async (req: Request, res: Response) => {
return 'committed';
}
throw streamErr;
+ } finally {
+ // #1642: this attempt is over whichever path it left by — the
+ // keep-alive interval must not outlive it.
+ streamSettled = true;
+ keepaliveStop();
}
}
@@ -2579,17 +2614,28 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {
streamFrames.push(frame);
};
- const flushHeaders = () => {
- if (headerSent) return;
- // #764: backfill only — the pump loop already records ttfb on the
- // first token (content or reasoning) it sees.
- if (ttfbMs === null) ttfbMs = Date.now() - start;
+ // Send the SSE headers once. Split out of flushHeaders so the
+ // #1642 keep-alive can commit the response with a content-free ping
+ // while the hold window is still open, without dumping the held
+ // preamble frames early.
+ let headersSentEarly = false;
+ const sendSseHeaders = () => {
+ if (headersSentEarly) return;
res.setHeader('Content-Type', 'text/event-stream');
res.setHeader('Cache-Control', 'no-cache');
res.setHeader('Connection', 'keep-alive');
res.setHeader('X-Routed-Via', routedViaValue(route.platform, route.modelId));
res.setHeader('X-FreeLLM-Cache', cacheKey ? 'MISS' : 'OFF');
setFallbackHeaders(res, attempt, attemptLog);
+ headersSentEarly = true;
+ };
+
+ const flushHeaders = () => {
+ if (headerSent) return;
+ // #764: backfill only — the pump loop already records ttfb on the
+ // first token (content or reasoning) it sees.
+ if (ttfbMs === null) ttfbMs = Date.now() - start;
+ sendSseHeaders();
headerSent = true;
// Committed: the answer is on its way, so the retry budget must no
// longer cancel this attempt (it could not fail over now anyway).
@@ -2614,6 +2660,19 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {
res.write(frame);
};
+ // #1642: optional heartbeat for the silent reasoning window. A bare
+ // SSE comment resets client idle watchdogs without committing content.
+ let streamSettled = false;
+ const keepaliveStop = startStreamKeepalive(streamKeepaliveMs(),
+ () => streamSettled,
+ () => {
+ sendSseHeaders();
+ // Bytes on the wire = the response can no longer fail over
+ // invisibly; stop the retry budget from cancelling it.
+ ctx.disarmHedge();
+ res.write(': keepalive\n\n');
+ });
+
try {
const gen = route.provider.streamChatCompletion(
route.apiKey, outboundMessages, route.modelId,
@@ -2977,7 +3036,10 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {
// client-abort branch stops the ladder without benching or an
// error log row (the socket is gone; nothing to render).
if (isClientAbortError(streamErr)) throw streamErr;
- if (headerSent) {
+ // headerSent = real content went out. headersSentEarly also counts:
+ // a #1642 keep-alive ping flushed the SSE headers, so the response
+ // is on the wire and this attempt can no longer fail over invisibly.
+ if (headerSent || headersSentEarly) {
// Mid-stream error after real payload reached the client — finish
// the SSE response honestly instead of leaving the client hanging.
console.error(`[Proxy] Mid-stream error from ${route.displayName}:`, streamErr.message);
@@ -3001,6 +3063,11 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {
// error frames, abrupt EOF, stalls, empty completions, and unparseable
// dialect turns alike.
throw streamErr;
+ } finally {
+ // #1642: this attempt is over whichever path it left by — the
+ // keep-alive interval must not outlive it.
+ streamSettled = true;
+ keepaliveStop();
}
} else {
const result = await route.provider.chatCompletion(
diff --git a/server/src/routes/responses.ts b/server/src/routes/responses.ts
index 6e86ed0c..6aa66fc7 100644
--- a/server/src/routes/responses.ts
+++ b/server/src/routes/responses.ts
@@ -32,6 +32,7 @@ import {
} from './proxy.js';
import { runFallbackLoop, newFallbackState, fallbackRoutingTokens, recordUpstreamSuccess, setFallbackHeaders, exhaustionErrorPayload, setExhaustionHeaders, type AttemptRecord } from '../lib/fallback-loop.js';
import { routedViaValue, safeHeaderValue } from '../lib/header-value.js';
+import { startStreamKeepalive, streamKeepaliveMs } from '../lib/stream-keepalive.js';
import { applyTokenBudget, tokenBudgetMessage } from '../lib/guardrails.js';
import { samplingParamSchemaFields, pickSamplingParams, type ResponseFormat } from '../lib/sampling-params.js';
import { enforceJsonContent } from '../lib/structured-output.js';
@@ -1250,11 +1251,7 @@ export async function handleResponses(req: Request, res: Response): Promise<void
const commit = () => {
if (streamStarted) return;
if (ttfbMs === null) ttfbMs = Date.now() - start;
- res.setHeader('Content-Type', 'text/event-stream');
- res.setHeader('Cache-Control', 'no-cache');
- res.setHeader('Connection', 'keep-alive');
- res.setHeader('X-Routed-Via', routedViaValue(route.platform, route.modelId));
- setFallbackHeaders(res, attempt, attemptLog);
+ sendSseHeaders();
const skeleton = {
id: responseId, object: 'response', created_at: nowUnix(),
status: 'in_progress', model: route.modelId, output: [], output_text: '',
@@ -1267,6 +1264,34 @@ export async function handleResponses(req: Request, res: Response): Promise<void
ctx.disarmHedge();
};
+ // Send the SSE headers once. Split out of commit so the #1642
+ // keep-alive can commit the response with a content-free ping while
+ // the hold window is still open, without sending the event skeleton
+ // early.
+ let headersSentEarly = false;
+ const sendSseHeaders = () => {
+ if (headersSentEarly) return;
+ res.setHeader('Content-Type', 'text/event-stream');
+ res.setHeader('Cache-Control', 'no-cache');
+ res.setHeader('Connection', 'keep-alive');
+ res.setHeader('X-Routed-Via', routedViaValue(route.platform, route.modelId));
+ setFallbackHeaders(res, attempt, attemptLog);
+ headersSentEarly = true;
+ };
+
+ // #1642: optional heartbeat for the silent reasoning window. A bare
+ // SSE comment resets client idle watchdogs without committing content.
+ let streamSettled = false;
+ const keepaliveStop = startStreamKeepalive(streamKeepaliveMs(),
+ () => streamSettled,
+ () => {
+ sendSseHeaders();
+ // Bytes on the wire = the response can no longer fail over
+ // invisibly; stop the retry budget from cancelling it.
+ ctx.disarmHedge();
+ res.write(': keepalive\n\n');
+ });
+
// Open the text output item and stream `text` as its first delta.
// The text item takes the next free output index (it is NOT always 0 —
// when the model emits tool_calls first, the function_call items own
@@ -1509,7 +1534,10 @@ export async function handleResponses(req: Request, res: Response): Promise<void
// A committed stream can't fail over (bytes already sent) — surface a
// response.failed event honestly and stop. A pre-commit failure throws
// through to the shared loop for cooldown + failover.
- if (streamStarted) {
+ // streamStarted = real events went out. headersSentEarly also counts:
+ // a #1642 keep-alive ping flushed the SSE headers, so this attempt is
+ // on the wire and can no longer fail over invisibly.
+ if (streamStarted || headersSentEarly) {
const safe = sanitizeProviderErrorMessage(streamErr.message);
traceRouteEvent('Responses', {
event: 'fail',
@@ -1526,6 +1554,11 @@ export async function handleResponses(req: Request, res: Response): Promise<void
return 'committed';
}
throw streamErr;
+ } finally {
+ // #1642: this attempt is over whichever path it left by — the
+ // keep-alive interval must not outlive it.
+ streamSettled = true;
+ keepaliveStop();
}
}