7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.
TypeScript
feat(stream): opt-in SSE keep-alive pings for the silent reasoning window (#1642) (#1674)
tashdroidffbd88feca9e
diff --git a/.env.example b/.env.exampleindex 99e43494..3f50f297 100644--- a/.env.example+++ b/.env.example@@ -33,6 +33,15 @@ PORT=3001# to disable proxy rate limiting.# PROXY_RATE_LIMIT_RPM=120+# SSE heartbeat interval in ms for streaming requests (#1642). Reasoning+# models can go minutes without a first token, and clients with an idle+# watchdog (Claude Code) abort during that silence. Set to e.g. 15000 and+# each waiting stream emits a content-free SSE ping every 15s so the client+# keeps waiting. OFF by default: a ping commits the response, so a stream+# that dies after one ends with an error frame instead of failing over to+# the next provider invisibly. Floor 1000 ms.+# STREAM_KEEPALIVE_MS=15000+# Max /api dashboard requests per minute per client IP (default: 600). This is a# flood guard — login has its own per-email lockout and key export its own much# tighter cap. Set to 0 to disable.diff --git a/docs/en/env/01-variables.md b/docs/en/env/01-variables.mdindex cdeabf14..6777d534 100644--- a/docs/en/env/01-variables.md+++ b/docs/en/env/01-variables.md@@ -60,6 +60,7 @@ First-run note: the first dashboard account is normally created from a browser o| `PROVIDER_STREAM_STALL_TIMEOUT_MS` | `90000` | Mid-stream inactivity timeout in milliseconds. If an SSE stream goes this long without a single byte AFTER the first byte arrived, the gateway gives up: before any output reached the client it fails over to the next candidate; after output was sent the stream ends with an error frame. Time-to-FIRST-byte is governed by `PROVIDER_TIMEOUT_<PLATFORM>` instead (whichever of the two is larger). Providers that queue heavily on free tiers can pause longer than 90s mid-stream — raise this if streamed responses stop early. `0` disables the stall watchdog entirely. || `PROVIDER_STREAM_STALL_TIMEOUT_<PLATFORM>` | Falls through to the global value above, then the built-in `90000` | Per-provider variant (platform name upper-cased). Precedence: per-platform > global > built-in 90s default. Example: `PROVIDER_STREAM_STALL_TIMEOUT_OLLAMA=240000` lets Ollama Cloud pause 4 minutes mid-stream without loosening the watchdog everywhere else. || `PROVIDER_DAILY_REQUEST_CAP_<PLATFORM>` | Built-in default per provider | Per-provider daily request cap (platform name upper-cased). Overrides the built-in default for that provider; set to `0` to disable the cap. Example: `PROVIDER_DAILY_REQUEST_CAP_OPENROUTER=50`. |+| `STREAM_KEEPALIVE_MS` | Off (`0`) | SSE heartbeat interval in milliseconds for streaming inference (#1642). Streaming responses hold every byte until the model's first token, so a reasoning model can leave the wire silent for minutes — long enough that clients with a time-to-first-byte watchdog (Claude Code's stream-idle timeout is the reference case) abort and the user reruns the turn. When set, each in-flight stream emits a content-free frame every interval while it is still uncommitted: a bare SSE comment (`: keepalive`) on OpenAI-shaped surfaces, the protocol's own `ping` event on `/v1/messages`. Spec-compliant parsers drop both, so clients see unchanged frames — but a ping commits the response, so a stream that then dies ends with an error frame instead of failing over invisibly to the next provider. That trade is what enabling this opts into; off by default keeps today's invisible failover. Floor 1000 ms. Read per request, so it applies immediately. || `LLMTR_VALIDATION_MIN_GAP_MS` | `2000` | Minimum gap, in milliseconds, between the start of two LLMTR key-validation probes (issue #1369). Probes already run one at a time, and concurrent checks of the same key share a single request; this spaces the survivors so a burst of identical probes never leaves the gateway back to back. This is a floor for our own traffic, not an LLMTR-published limit — their docs state none for this endpoint — chosen because a burst of the same nonexistent-model POST from one IP is the pattern their security system reads as automation, and an account flagged that way gets suspended. Covers every caller: the 5-minute health pass, the dashboard's "check all", the post-wake re-probe, the cooldown-probe job and the 401-triggered revalidation. Raise it if your account is still being flagged; `0` disables pacing. Chat and streaming traffic is never paced. Read at startup. || `FALLBACK_TIME_BUDGET_MS` | `45000` | Base wall-clock budget for provider failover, in milliseconds. The initial attempt and one failover hop are always allowed. After that, exhaustion stops new attempts and can cancel an in-flight attempt still waiting for its first usable response. Streaming responses disarm this timer when the first usable response arrives. Successful TTFB history can widen the budget for a slow endpoint, as described below. `0` disables the budget entirely, including adaptive widening, and retries until the attempt cap. Also settable at runtime via the `fallback_time_budget_ms` settings key, which takes precedence. || `SLOW_ENDPOINT_BUFFER_MS` | `10000` | Padding added to the historical P95 successful time-to-first-byte (TTFB), in milliseconds. Samples are grouped by platform + `endpoint_scope`; the effective fallback budget is `max(base budget, P95 TTFB + buffer)`, capped at three times the base budget, and only for endpoints whose P95 is at least half the base budget. Endpoints with fewer than `TTFB_BUDGET_MIN_SAMPLES` samples, and those whose P95 + buffer fits the base budget, keep the base budget. `0` removes extra padding. The runtime `slow_endpoint_buffer_ms` settings key takes precedence over this environment variable. |diff --git a/docs/zh-cn/env/01-variables.md b/docs/zh-cn/env/01-variables.mdindex 322ceda3..cdfefea5 100644--- a/docs/zh-cn/env/01-variables.md+++ b/docs/zh-cn/env/01-variables.md@@ -57,6 +57,7 @@ FreeLLMAPI 从 `.env` 读取的全部变量,按主题分组。默认值和说| `PROVIDER_STREAM_STALL_TIMEOUT_MS` | `90000` | 流中途的停滞超时,单位毫秒。如果一条 SSE 流在首字节已到达之后持续这么久没有任何字节,网关就放弃:若尚无输出到达客户端,则故障转移到下一个候选;若已有输出,则以一个错误帧结束流。到「第一个」字节为止的时间由 `PROVIDER_TIMEOUT_<PLATFORM>` 管(两者取较大者)。在免费额度上排长队的提供方可能在流中途停顿超过 90 秒——如果流式响应总是提前中断,就调高这个值。`0` 彻底关闭停滞看门狗。 || `PROVIDER_STREAM_STALL_TIMEOUT_<PLATFORM>` | 先回落到上面的全局值,再到内置的 `90000` | 按提供方的变体(平台名大写)。优先级:按平台 > 全局 > 内置 90 秒默认。示例:`PROVIDER_STREAM_STALL_TIMEOUT_OLLAMA=240000` 允许 Ollama Cloud 在流中途停顿 4 分钟,而其他地方的看门狗不受影响。 || `PROVIDER_DAILY_REQUEST_CAP_<PLATFORM>` | 各提供方内置的默认值 | 按提供方的每日请求上限(平台名大写)。覆盖该提供方的内置默认值;设为 `0` 关闭上限。示例:`PROVIDER_DAILY_REQUEST_CAP_OPENROUTER=50`。 |+| `STREAM_KEEPALIVE_MS` | 关闭(`0`) | 流式推理的 SSE 心跳间隔(毫秒)(#1642)。流式响应会扣住所有字节直到模型吐出第一个 token,推理模型可能让连接静默数分钟——足以让带首字节看门狗的客户端(典型是 Claude Code 的 stream-idle 超时)中断请求、用户重跑整轮。设置后,每个未提交的在途流每隔该间隔发送一个无内容帧:OpenAI 形状接口发裸 SSE 注释行(`: keepalive`),`/v1/messages` 发协议自带的 `ping` 事件。符合规范的解析器会丢弃两者,客户端看到的帧不变——但心跳一旦发出响应即已提交,之后断流将以错误帧结束,而不是静默故障转移到下一个提供方。开启即接受该权衡;默认关闭保持现有的静默故障转移。下限 1000 毫秒。每次请求读取,改完即生效。 || `LLMTR_VALIDATION_MIN_GAP_MS` | `2000` | 两次 LLMTR 密钥校验探测开始之间至少间隔的毫秒数(issue #1369)。探测本来就是逐个串行发出的,同一密钥的并发检查也共用一次请求;这个值为其余探测拉开间距,使一串相同的探测绝不会紧接着离开网关。这是我们自己流量的下限,不是 LLMTR 公布的限制——他们的文档对该端点没有给出任何限制——之所以这样设,是因为同一 IP 短时间内反复发出相同的「不存在的模型」POST,正是他们的安全系统识别为自动化的模式,被标记的账号会被停用。它覆盖所有调用方:5 分钟健康巡检、仪表盘的「全部检查」、唤醒后的重新探测、冷却探测任务,以及 401 触发的重新校验。如果账号仍被标记,请调高该值;`0` 关闭节流。聊天与流式流量从不节流。启动时读取。 || `FALLBACK_TIME_BUDGET_MS` | `45000` | 提供方故障转移的基础挂钟时间预算,单位毫秒。始终允许首次尝试及一次故障转移;之后预算耗尽时会停止发起新尝试,也可能取消仍在等待首个有效响应的进行中尝试。流式响应收到首个有效响应后会解除此计时器。成功请求的首字节耗时历史可为慢端点延长预算,详见下方。`0` 彻底关闭该预算(包括自适应延长),一直重试直到触及尝试次数上限。也可以在运行时通过 `fallback_time_budget_ms` 设置键修改,后者优先。 || `SLOW_ENDPOINT_BUFFER_MS` | `10000` | 在历史成功请求的首字节耗时(TTFB)P95 上增加的缓冲,单位毫秒。样本按平台 + `endpoint_scope` 分组,有效故障转移预算为 `max(基础预算, P95 TTFB + 缓冲)`,上限为基础预算的三倍,且仅适用于 P95 不低于基础预算一半的端点。样本数少于 `TTFB_BUDGET_MIN_SAMPLES` 的端点,以及 P95 加缓冲不超过基础预算的端点,仍使用基础预算。`0` 表示不额外增加缓冲。运行时的 `slow_endpoint_buffer_ms` 设置键优先于此环境变量。 |diff --git a/server/src/__tests__/lib/module-purity.test.ts b/server/src/__tests__/lib/module-purity.test.tsindex 7592e9fd..d3d1d863 100644--- a/server/src/__tests__/lib/module-purity.test.ts+++ b/server/src/__tests__/lib/module-purity.test.ts@@ -43,6 +43,8 @@ const PURE_MODULES = ['retry-hint.ts',// Search platform list; the provider list, keys route and search service import it.'search-platforms.ts',+ // SSE keep-alive helper (#1642); env is injected, the response is a callback.+ 'stream-keepalive.ts','structured-output.ts','tool-args.ts','tool-call-rescue.ts',diff --git a/server/src/__tests__/lib/stream-keepalive.test.ts b/server/src/__tests__/lib/stream-keepalive.test.tsnew file mode 100644index 00000000..3752e956--- /dev/null+++ b/server/src/__tests__/lib/stream-keepalive.test.ts@@ -0,0 +1,70 @@+import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';+import { streamKeepaliveMs, startStreamKeepalive } from '../../lib/stream-keepalive.js';++// #1642: SSE keep-alive pings for the uncommitted streaming window. The+// helper carries the env parsing and the timer lifecycle; the route pumps+// wire it to a real response (covered by proxy-stream-keepalive.test.ts).++describe('streamKeepaliveMs (#1642)', () => {+ it('defaults to 0 (feature off) when STREAM_KEEPALIVE_MS is unset or blank', () => {+ expect(streamKeepaliveMs({})).toBe(0);+ expect(streamKeepaliveMs({ STREAM_KEEPALIVE_MS: '' })).toBe(0);+ expect(streamKeepaliveMs({ STREAM_KEEPALIVE_MS: ' ' })).toBe(0);+ });++ it('treats 0, negatives, and garbage as off', () => {+ expect(streamKeepaliveMs({ STREAM_KEEPALIVE_MS: '0' })).toBe(0);+ expect(streamKeepaliveMs({ STREAM_KEEPALIVE_MS: '-5' })).toBe(0);+ expect(streamKeepaliveMs({ STREAM_KEEPALIVE_MS: 'abc' })).toBe(0);+ expect(streamKeepaliveMs({ STREAM_KEEPALIVE_MS: 'Infinity' })).toBe(0);+ });++ it('accepts a positive interval and floors it at 1s', () => {+ expect(streamKeepaliveMs({ STREAM_KEEPALIVE_MS: '15000' })).toBe(15000);+ expect(streamKeepaliveMs({ STREAM_KEEPALIVE_MS: '1e9' })).toBe(1e9);+ expect(streamKeepaliveMs({ STREAM_KEEPALIVE_MS: '5' })).toBe(1000);+ expect(streamKeepaliveMs({ STREAM_KEEPALIVE_MS: '2500.7' })).toBe(2500);+ });+});++describe('startStreamKeepalive (#1642)', () => {+ beforeEach(() => vi.useFakeTimers());+ afterEach(() => vi.useRealTimers());++ it('does nothing when the interval is 0 (no timer is created)', () => {+ const keepalive = vi.fn();+ const stop = startStreamKeepalive(0, () => false, keepalive);+ vi.advanceTimersByTime(10_000);+ expect(keepalive).not.toHaveBeenCalled();+ expect(() => stop()).not.toThrow();+ });++ it('fires on each tick until the attempt settles', () => {+ let done = false;+ const keepalive = vi.fn();+ startStreamKeepalive(1000, () => done, keepalive);+ vi.advanceTimersByTime(3500);+ expect(keepalive).toHaveBeenCalledTimes(3);+ done = true;+ vi.advanceTimersByTime(5000);+ expect(keepalive).toHaveBeenCalledTimes(3);+ });++ it('stops after a write failure (dead socket)', () => {+ const keepalive = vi.fn(() => { throw new Error('socket gone'); });+ startStreamKeepalive(1000, () => false, keepalive);+ vi.advanceTimersByTime(5000);+ expect(keepalive).toHaveBeenCalledTimes(1);+ });++ it('stop() is idempotent and prevents further ticks', () => {+ const keepalive = vi.fn();+ const stop = startStreamKeepalive(1000, () => false, keepalive);+ vi.advanceTimersByTime(1500);+ expect(keepalive).toHaveBeenCalledTimes(1);+ stop();+ stop();+ vi.advanceTimersByTime(5000);+ expect(keepalive).toHaveBeenCalledTimes(1);+ });+});diff --git a/server/src/__tests__/routes/proxy-stream-keepalive.test.ts b/server/src/__tests__/routes/proxy-stream-keepalive.test.tsnew file mode 100644index 00000000..a88d437c--- /dev/null+++ b/server/src/__tests__/routes/proxy-stream-keepalive.test.ts@@ -0,0 +1,160 @@+import { describe, it, expect, beforeAll, beforeEach, afterEach, vi } from 'vitest';+import type { Express } from 'express';+import { createApp } from '../../app.js';+import { initDb, getDb, getUnifiedApiKey } from '../../db/index.js';+import { mintDashboardToken } from '../helpers/auth.js';++// #1642: SSE keep-alive pings through the real proxy route. Upstreams here+// go quiet before the first token (the reasoning-window shape that made+// Claude Code abort live); with STREAM_KEEPALIVE_MS set the response must+// carry comment-line pings during that silence, and with it unset (default)+// the byte stream must be byte-for-byte what it was before.++async function streamRequest(app: Express, body: any, signal?: AbortSignal) {+ const server = app.listen(0, '127.0.0.1');+ if (!server.listening) await new Promise<void>(resolve => server.once('listening', () => resolve()));+ const addr = server.address() as any;+ const res = await fetch(`http://127.0.0.1:${addr.port}/v1/chat/completions`, {+ method: 'POST',+ headers: { 'Content-Type': 'application/json', Authorization: 'Bearer ' + getUnifiedApiKey() },+ body: JSON.stringify(body),+ signal,+ });+ const text = await res.text();+ server.close();+ return { status: res.status, headers: res.headers, text };+}++const sse = (...payloads: (object | string)[]) =>+ payloads.map(p => `data: ${typeof p === 'string' ? p : JSON.stringify(p)}\n\n`).join('');++const roleChunk = { id: 'c1', object: 'chat.completion.chunk', created: 1, model: 'm', choices: [{ index: 0, delta: { role: 'assistant', content: null }, finish_reason: null }] };+const textChunk = (s: string, finish: string | null = null) => ({ id: 'c1', object: 'chat.completion.chunk', created: 1, model: 'm', choices: [{ index: 0, delta: { content: s }, finish_reason: finish }] });+const finishChunk = (reason: string) => ({ id: 'c1', object: 'chat.completion.chunk', created: 1, model: 'm', choices: [{ index: 0, delta: {}, finish_reason: reason }] });++/** Upstream whose first SSE bytes arrive `delayMs` after the request. */+function mockSilentThenStream(delayMs: number, body: string) {+ const origFetch = global.fetch;+ let calls = 0;+ vi.spyOn(global, 'fetch').mockImplementation(async (url, init) => {+ const urlStr = typeof url === 'string' ? url : url.toString();+ // Only intercept provider upstreams; the test's own localhost request+ // goes through.+ if (/127\.0\.0\.1|localhost/.test(urlStr)) return origFetch(url as any, init);+ calls++;+ const stream = new ReadableStream<Uint8Array>({+ start(controller) {+ setTimeout(() => {+ controller.enqueue(new TextEncoder().encode(body));+ controller.close();+ }, delayMs);+ },+ });+ return new Response(stream, { status: 200, headers: { 'Content-Type': 'text/event-stream' } });+ });+ return { calls: () => calls };+}++/** Upstream that goes quiet for `delayMs`, then dies mid-response. */+function mockSilentThenDie(delayMs: number) {+ const origFetch = global.fetch;+ let calls = 0;+ vi.spyOn(global, 'fetch').mockImplementation(async (url, init) => {+ const urlStr = typeof url === 'string' ? url : url.toString();+ if (/127\.0\.0\.1|localhost/.test(urlStr)) return origFetch(url as any, init);+ calls++;+ const stream = new ReadableStream<Uint8Array>({+ start(controller) {+ setTimeout(() => {+ controller.error(new Error('upstream exploded mid-stream'));+ }, delayMs);+ },+ });+ return new Response(stream, { status: 200, headers: { 'Content-Type': 'text/event-stream' } });+ });+ return { calls: () => calls };+}++describe('proxy SSE keep-alive (#1642)', () => {+ let app: Express;+ let dashToken = '';++ beforeAll(() => {+ process.env.ENCRYPTION_KEY = '0'.repeat(64);+ initDb(':memory:');+ app = createApp();+ dashToken = mintDashboardToken();+ });++ beforeEach(async () => {+ const db = getDb();+ db.prepare('DELETE FROM api_keys').run();+ db.prepare('DELETE FROM requests').run();+ db.prepare('DELETE FROM rate_limit_cooldowns').run();+ db.prepare('DELETE FROM rate_limit_usage').run();+ const { status } = await restKey();+ expect(status).toBe(201);+ });++ afterEach(() => {+ vi.restoreAllMocks();+ delete process.env.STREAM_KEEPALIVE_MS;+ });++ async function restKey() {+ // mint a groq key through the dashboard API (same shape as the other+ // stream-integrity suites)+ const server = app.listen(0, '127.0.0.1');+ if (!server.listening) await new Promise<void>(resolve => server.once('listening', () => resolve()));+ const addr = server.address() as any;+ const res = await fetch(`http://127.0.0.1:${addr.port}/api/keys`, {+ method: 'POST',+ headers: { 'Content-Type': 'application/json', Authorization: 'Bearer ' + dashToken },+ body: JSON.stringify({ platform: 'groq', key: 'gsk_keepalive', label: 't' }),+ });+ const status = res.status;+ await res.json().catch(() => null);+ server.close();+ return { status };+ }++ it('emits comment-line pings during a silent reasoning window when enabled', async () => {+ process.env.STREAM_KEEPALIVE_MS = '1000'; // 1s is the helper's floor+ // 2.6s of silence upstream ≈ 2 scheduled pings (1000ms floor).+ const up = mockSilentThenStream(2600, sse(roleChunk, textChunk('finally.'), finishChunk('stop'), '[DONE]'));+ const r = await streamRequest(app, { stream: true, messages: [{ role: 'user', content: 'keepalive ping test' }] });+ expect(r.status).toBe(200);+ expect(up.calls()).toBe(1);+ const pings = r.text.split('\n').filter(l => l === ': keepalive');+ expect(pings.length).toBeGreaterThanOrEqual(2);+ // Every ping is a bare SSE comment: a spec parser drops it, so the JSON+ // frames the client sees are unchanged.+ const frames = r.text.split('\n')+ .filter(l => l.startsWith('data: ') && l.trim() !== 'data: [DONE]')+ .map(l => JSON.parse(l.slice(6)));+ expect(frames.some(f => f.choices?.[0]?.delta?.content === 'finally.')).toBe(true);+ }, 15000);++ it('default (unset): a silent window streams no pings at all', async () => {+ expect(process.env.STREAM_KEEPALIVE_MS).toBeUndefined();+ const up = mockSilentThenStream(500, sse(roleChunk, textChunk('no heartbeat here.'), finishChunk('stop'), '[DONE]'));+ const r = await streamRequest(app, { stream: true, messages: [{ role: 'user', content: 'keepalive default-off test' }] });+ expect(r.status).toBe(200);+ expect(up.calls()).toBe(1);+ expect(r.text).not.toContain(': keepalive');+ }, 15000);++ it('once a ping committed the response, a dying stream ends honestly instead of failing over', async () => {+ // The documented trade of opting in: after bytes hit the wire there is no+ // invisible retry. The client gets an SSE error frame, not a hang, and+ // exactly one upstream was tried.+ process.env.STREAM_KEEPALIVE_MS = '1000'; // 1s is the helper's floor+ const up = mockSilentThenDie(1500);+ const r = await streamRequest(app, { stream: true, messages: [{ role: 'user', content: 'keepalive committed trade test' }] });+ expect(r.status).toBe(200);+ expect(up.calls()).toBe(1);+ expect(r.text).toContain(': keepalive');+ expect(r.text).toContain('"stream_error"');+ expect(r.text).toContain('data: [DONE]');+ }, 15000);+});diff --git a/server/src/lib/stream-keepalive.ts b/server/src/lib/stream-keepalive.tsnew file mode 100644index 00000000..529113c6--- /dev/null+++ b/server/src/lib/stream-keepalive.ts@@ -0,0 +1,77 @@+// SSE keep-alive pings for the uncommitted streaming window (#1642).+//+// The streaming routes hold every byte — headers included — until the first+// real text token, so a reasoning model can leave the wire silent for 30-180s+// while it thinks. Clients with a time-to-first-byte watchdog (Claude Code's+// stream-idle timeout is the reference case) abort during that window, the+// router logs "client disconnected mid-attempt", and the user reruns the turn.+//+// When STREAM_KEEPALIVE_MS is set, each streaming pump starts an interval+// that emits a content-free keep-alive frame while the attempt is in flight.+// On the OpenAI surfaces that is a bare SSE comment line (`: keepalive`),+// which every spec-compliant parser ignores; on the Anthropic surface it is+// the protocol's own `ping` event.+//+// The invariant this preserves: the ping carries NO content, so nothing the+// client can act on has been sent. What it necessarily gives up when enabled+// is invisible failover FOR THAT ATTEMPT — a ping flushes the SSE headers, so+// a stream that then dies ends the response with an error frame instead of+// retrying another provider behind the scenes. That trade is exactly what an+// operator opts into; the default is OFF, keeping the current behavior for+// everyone who has not heard of the problem.+//+// Pings keep firing on a committed-but-silent stream too (upstream stalled+// mid-reasoning), which is also useful: an idle upstream no longer looks+// different from an idle connection at the client either way, but the extra+// bytes stop the client from killing a stream that is merely slow.+const MIN_KEEPALIVE_MS = 1000;++/**+ * The keep-alive interval in ms, parsed from STREAM_KEEPALIVE_MS.+ * Unset / empty / 0 / non-numeric / negative → 0 (feature off, default).+ * Positive values are clamped to a 1s floor: a sub-second ping interval just+ * burns bandwidth and adds SSE frames no client asked for.+ */+export function streamKeepaliveMs(env: NodeJS.ProcessEnv = process.env): number {+ const raw = env.STREAM_KEEPALIVE_MS;+ if (raw === undefined || raw.trim() === '') return 0;+ const n = Number(raw);+ if (!Number.isFinite(n) || n <= 0) return 0;+ return Math.max(MIN_KEEPALIVE_MS, Math.floor(n));+}++/**+ * Start the keep-alive interval for one stream attempt.+ *+ * - `intervalMs` ≤ 0 disables the feature: no timer is created and the+ * returned stop function is a no-op.+ * - `isDone` is polled at every tick: once the attempt has finished (response+ * ended, failed over, or errored), the timer stops itself.+ * - `keepalive` runs on each tick and must write a content-free frame. It is+ * expected to commit the response first if it has not been committed yet.+ * Throwing (dead socket) stops the timer without surfacing anything —+ * the pump's own error paths notice the broken connection.+ *+ * Returns the stop function; call it when the stream ends. Idempotent.+ */+export function startStreamKeepalive(+ intervalMs: number,+ isDone: () => boolean,+ keepalive: () => void,+): () => void {+ if (intervalMs <= 0) return () => {};+ const timer = setInterval(() => {+ if (isDone()) { stop(); return; }+ try {+ keepalive();+ } catch {+ stop();+ }+ }, intervalMs);+ // Never hold the process open for a ping that outlived its request.+ if (typeof timer.unref === 'function') timer.unref();+ function stop(): void {+ clearInterval(timer);+ }+ return stop;+}diff --git a/server/src/routes/anthropic.ts b/server/src/routes/anthropic.tsindex 9c3302a5..6cfc47e1 100644--- a/server/src/routes/anthropic.ts+++ b/server/src/routes/anthropic.ts@@ -24,6 +24,7 @@ import { logRequest } from '../lib/request-log.js';import { extractApiToken, timingSafeStringEqual, getStickyModel, setStickyModel } from './proxy.js';import { runFallbackLoop, newFallbackState, fallbackRoutingTokens, recordUpstreamSuccess, type ExhaustionBody, setFallbackHeaders, setExhaustionHeaders, type AttemptRecord, type FallbackState, REASONING_HOLD_MAX_MS } from '../lib/fallback-loop.js';import { routedViaValue } from '../lib/header-value.js';+import { startStreamKeepalive, streamKeepaliveMs } from '../lib/stream-keepalive.js';import { applyTokenBudget, tokenBudgetMessage } from '../lib/guardrails.js';import { resolveAnthropicModel, claudeFamilyDiscoveryEntries } from '../services/anthropic-map.js';import { isUnifyEnabled, getModelGroups, resolveRequestedIdForDispatch } from '../services/model-groups.js';@@ -880,11 +881,7 @@ async function streamCompletion(const ensureMessageStart = () => {if (messageStarted) return;- res.setHeader('Content-Type', 'text/event-stream');- res.setHeader('Cache-Control', 'no-cache');- res.setHeader('Connection', 'keep-alive');- res.setHeader('X-Routed-Via', routedViaValue(route.platform, route.modelId));- setFallbackHeaders(res, ctx.attempt, ctx.attemptLog);+ sendSseHeaders();// Committed: the answer is on its way, so the retry budget must no longer// cancel this attempt (it could not fail over now anyway).ctx.disarmHedge();@@ -899,6 +896,37 @@ async function streamCompletion(messageStarted = true;};+ // Send the SSE headers once. Split out of ensureMessageStart so the+ // #1642 keep-alive can commit the response with the protocol's own content-+ // free `ping` event while the reasoning buffer is still filling, without+ // sending message_start early.+ let headersSentEarly = false;+ const sendSseHeaders = () => {+ if (headersSentEarly) return;+ res.setHeader('Content-Type', 'text/event-stream');+ res.setHeader('Cache-Control', 'no-cache');+ res.setHeader('Connection', 'keep-alive');+ res.setHeader('X-Routed-Via', routedViaValue(route.platform, route.modelId));+ setFallbackHeaders(res, ctx.attempt, ctx.attemptLog);+ headersSentEarly = true;+ };++ // #1642: optional heartbeat for the silent reasoning window. The Anthropic+ // streaming protocol carries a native ping event, so no fake frame is+ // needed. Once it fires the attempt is committed to the wire and can no+ // longer fail over invisibly — that is the trade STREAM_KEEPALIVE_MS opts+ // into (off by default).+ let streamSettled = false;+ const keepaliveStop = startStreamKeepalive(streamKeepaliveMs(),+ () => streamSettled,+ () => {+ sendSseHeaders();+ // Bytes on the wire = the response can no longer fail over invisibly;+ // stop the retry budget from cancelling it.+ ctx.disarmHedge();+ writeSse(res, 'ping', { type: 'ping' });+ });+// Reasoning output (delta.reasoning_content from providers / the <think>// extractor, delta.reasoning from Ollama-style upstreams) is buffered and// emitted as one complete thinking block right before the first text/tool@@ -1116,7 +1144,10 @@ async function streamCompletion(// stops the ladder without benching or an error log row (the socket is// gone; nothing to render).if (isClientAbortError(err)) throw err;- if (messageStarted) {+ // messageStarted = real content went out. headersSentEarly also counts: a+ // #1642 keep-alive ping flushed the SSE headers, so this attempt is on the+ // wire and can no longer fail over invisibly.+ if (messageStarted || headersSentEarly) {// Real payload already reached the client — finish the SSE response// honestly instead of leaving Claude Code hanging, and stop the retry loop.writeSse(res, 'error', { type: 'error', error: { type: 'api_error', message: `Provider error (${route.displayName}): stream interrupted` } });@@ -1126,6 +1157,11 @@ async function streamCompletion(}// Headers never sent — bubble to the outer loop for failover.throw err;+ } finally {+ // #1642: this attempt is over whichever path it left by — the keep-alive+ // interval must not outlive it.+ streamSettled = true;+ keepaliveStop();}}diff --git a/server/src/routes/proxy.ts b/server/src/routes/proxy.tsindex b2ca76b0..adf6d6bb 100644--- a/server/src/routes/proxy.ts+++ b/server/src/routes/proxy.ts@@ -41,6 +41,7 @@ import { buildModelListing, type NormalizedModel } from '../services/model-listiimport { claudeFamilyDiscoveryEntries, resolveListedClaudeAlias } from '../services/anthropic-map.js';import { compressRequest, formatCompressionHeader } from '../services/compression/pipeline.js';import { applyDecisionRouting, routingStrategyHeader } from '../services/routing-decision.js';+import { startStreamKeepalive, streamKeepaliveMs } from '../lib/stream-keepalive.js';export const proxyRouter = Router();@@ -1558,6 +1559,24 @@ proxyRouter.post('/completions', async (req: Request, res: Response) => {let sawText = false;let upstreamFinish: string | null = null;const buffered: unknown[] = [];+ // True once this attempt is over (success, committed error, or thrown)+ // — the keep-alive interval polls it and stops itself.+ let streamSettled = false;++ // Send the SSE headers once. Split out of flushHeaders so the+ // #1642 keep-alive can commit the response with a content-free ping+ // while the hold window is still open, without dumping the buffered+ // (possibly reasoning-bearing) frames early.+ let headersSentEarly = false;+ const sendSseHeaders = () => {+ if (headersSentEarly) return;+ res.setHeader('Content-Type', 'text/event-stream');+ res.setHeader('Cache-Control', 'no-cache');+ res.setHeader('Connection', 'keep-alive');+ res.setHeader('X-Routed-Via', routedViaValue(route.platform, route.modelId));+ setFallbackHeaders(res, attempt, attemptLog);+ headersSentEarly = true;+ };const flushHeaders = () => {if (headerSent) return;@@ -1565,11 +1584,7 @@ proxyRouter.post('/completions', async (req: Request, res: Response) => {// reasoning) in the pump loop below; this call only backfills streams// that reached the commit point without one.if (ttfbMs === null) ttfbMs = Date.now() - start;- res.setHeader('Content-Type', 'text/event-stream');- res.setHeader('Cache-Control', 'no-cache');- res.setHeader('Connection', 'keep-alive');- res.setHeader('X-Routed-Via', routedViaValue(route.platform, route.modelId));- setFallbackHeaders(res, attempt, attemptLog);+ sendSseHeaders();headerSent = true;// Committed: the answer is on its way, so the retry budget must no// longer cancel this attempt (it could not fail over now anyway).@@ -1578,6 +1593,18 @@ proxyRouter.post('/completions', async (req: Request, res: Response) => {buffered.length = 0;};+ // #1642: optional heartbeat for the silent reasoning window. A bare+ // SSE comment resets client idle watchdogs without committing content.+ const keepaliveStop = startStreamKeepalive(streamKeepaliveMs(),+ () => streamSettled,+ () => {+ sendSseHeaders();+ // Bytes on the wire = the response can no longer fail over+ // invisibly; stop the retry budget from cancelling it.+ ctx.disarmHedge();+ res.write(': keepalive\n\n');+ });+try {const gen = route.provider.streamChatCompletion(route.apiKey,@@ -1658,7 +1685,10 @@ proxyRouter.post('/completions', async (req: Request, res: Response) => {// client-abort branch stops the ladder without benching or an// error log row (the socket is gone; nothing to render).if (isClientAbortError(streamErr)) throw streamErr;- if (headerSent) {+ // headerSent = real content went out. headersSentEarly also counts:+ // a #1642 keep-alive ping flushed the SSE headers, so the response+ // is on the wire and this attempt can no longer fail over invisibly.+ if (headerSent || headersSentEarly) {console.error(`[Proxy] Mid-stream legacy completion error from ${route.displayName}:`, streamErr.message);const payload = { error: { message: `Provider error (${route.displayName}): stream interrupted`, type: 'stream_error' } };try { res.write(`data: ${JSON.stringify(payload)}\n\n`); } catch { /* socket gone */ }@@ -1676,6 +1706,11 @@ proxyRouter.post('/completions', async (req: Request, res: Response) => {return 'committed';}throw streamErr;+ } finally {+ // #1642: this attempt is over whichever path it left by — the+ // keep-alive interval must not outlive it.+ streamSettled = true;+ keepaliveStop();}}@@ -2579,17 +2614,28 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {streamFrames.push(frame);};- const flushHeaders = () => {- if (headerSent) return;- // #764: backfill only — the pump loop already records ttfb on the- // first token (content or reasoning) it sees.- if (ttfbMs === null) ttfbMs = Date.now() - start;+ // Send the SSE headers once. Split out of flushHeaders so the+ // #1642 keep-alive can commit the response with a content-free ping+ // while the hold window is still open, without dumping the held+ // preamble frames early.+ let headersSentEarly = false;+ const sendSseHeaders = () => {+ if (headersSentEarly) return;res.setHeader('Content-Type', 'text/event-stream');res.setHeader('Cache-Control', 'no-cache');res.setHeader('Connection', 'keep-alive');res.setHeader('X-Routed-Via', routedViaValue(route.platform, route.modelId));res.setHeader('X-FreeLLM-Cache', cacheKey ? 'MISS' : 'OFF');setFallbackHeaders(res, attempt, attemptLog);+ headersSentEarly = true;+ };++ const flushHeaders = () => {+ if (headerSent) return;+ // #764: backfill only — the pump loop already records ttfb on the+ // first token (content or reasoning) it sees.+ if (ttfbMs === null) ttfbMs = Date.now() - start;+ sendSseHeaders();headerSent = true;// Committed: the answer is on its way, so the retry budget must no// longer cancel this attempt (it could not fail over now anyway).@@ -2614,6 +2660,19 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {res.write(frame);};+ // #1642: optional heartbeat for the silent reasoning window. A bare+ // SSE comment resets client idle watchdogs without committing content.+ let streamSettled = false;+ const keepaliveStop = startStreamKeepalive(streamKeepaliveMs(),+ () => streamSettled,+ () => {+ sendSseHeaders();+ // Bytes on the wire = the response can no longer fail over+ // invisibly; stop the retry budget from cancelling it.+ ctx.disarmHedge();+ res.write(': keepalive\n\n');+ });+try {const gen = route.provider.streamChatCompletion(route.apiKey, outboundMessages, route.modelId,@@ -2977,7 +3036,10 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {// client-abort branch stops the ladder without benching or an// error log row (the socket is gone; nothing to render).if (isClientAbortError(streamErr)) throw streamErr;- if (headerSent) {+ // headerSent = real content went out. headersSentEarly also counts:+ // a #1642 keep-alive ping flushed the SSE headers, so the response+ // is on the wire and this attempt can no longer fail over invisibly.+ if (headerSent || headersSentEarly) {// Mid-stream error after real payload reached the client — finish// the SSE response honestly instead of leaving the client hanging.console.error(`[Proxy] Mid-stream error from ${route.displayName}:`, streamErr.message);@@ -3001,6 +3063,11 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {// error frames, abrupt EOF, stalls, empty completions, and unparseable// dialect turns alike.throw streamErr;+ } finally {+ // #1642: this attempt is over whichever path it left by — the+ // keep-alive interval must not outlive it.+ streamSettled = true;+ keepaliveStop();}} else {const result = await route.provider.chatCompletion(diff --git a/server/src/routes/responses.ts b/server/src/routes/responses.tsindex 6e86ed0c..6aa66fc7 100644--- a/server/src/routes/responses.ts+++ b/server/src/routes/responses.ts@@ -32,6 +32,7 @@ import {} from './proxy.js';import { runFallbackLoop, newFallbackState, fallbackRoutingTokens, recordUpstreamSuccess, setFallbackHeaders, exhaustionErrorPayload, setExhaustionHeaders, type AttemptRecord } from '../lib/fallback-loop.js';import { routedViaValue, safeHeaderValue } from '../lib/header-value.js';+import { startStreamKeepalive, streamKeepaliveMs } from '../lib/stream-keepalive.js';import { applyTokenBudget, tokenBudgetMessage } from '../lib/guardrails.js';import { samplingParamSchemaFields, pickSamplingParams, type ResponseFormat } from '../lib/sampling-params.js';import { enforceJsonContent } from '../lib/structured-output.js';@@ -1250,11 +1251,7 @@ export async function handleResponses(req: Request, res: Response): Promise<voidconst commit = () => {if (streamStarted) return;if (ttfbMs === null) ttfbMs = Date.now() - start;- res.setHeader('Content-Type', 'text/event-stream');- res.setHeader('Cache-Control', 'no-cache');- res.setHeader('Connection', 'keep-alive');- res.setHeader('X-Routed-Via', routedViaValue(route.platform, route.modelId));- setFallbackHeaders(res, attempt, attemptLog);+ sendSseHeaders();const skeleton = {id: responseId, object: 'response', created_at: nowUnix(),status: 'in_progress', model: route.modelId, output: [], output_text: '',@@ -1267,6 +1264,34 @@ export async function handleResponses(req: Request, res: Response): Promise<voidctx.disarmHedge();};+ // Send the SSE headers once. Split out of commit so the #1642+ // keep-alive can commit the response with a content-free ping while+ // the hold window is still open, without sending the event skeleton+ // early.+ let headersSentEarly = false;+ const sendSseHeaders = () => {+ if (headersSentEarly) return;+ res.setHeader('Content-Type', 'text/event-stream');+ res.setHeader('Cache-Control', 'no-cache');+ res.setHeader('Connection', 'keep-alive');+ res.setHeader('X-Routed-Via', routedViaValue(route.platform, route.modelId));+ setFallbackHeaders(res, attempt, attemptLog);+ headersSentEarly = true;+ };++ // #1642: optional heartbeat for the silent reasoning window. A bare+ // SSE comment resets client idle watchdogs without committing content.+ let streamSettled = false;+ const keepaliveStop = startStreamKeepalive(streamKeepaliveMs(),+ () => streamSettled,+ () => {+ sendSseHeaders();+ // Bytes on the wire = the response can no longer fail over+ // invisibly; stop the retry budget from cancelling it.+ ctx.disarmHedge();+ res.write(': keepalive\n\n');+ });+// Open the text output item and stream `text` as its first delta.// The text item takes the next free output index (it is NOT always 0 —// when the model emits tool_calls first, the function_call items own@@ -1509,7 +1534,10 @@ export async function handleResponses(req: Request, res: Response): Promise<void// A committed stream can't fail over (bytes already sent) — surface a// response.failed event honestly and stop. A pre-commit failure throws// through to the shared loop for cooldown + failover.- if (streamStarted) {+ // streamStarted = real events went out. headersSentEarly also counts:+ // a #1642 keep-alive ping flushed the SSE headers, so this attempt is+ // on the wire and can no longer fail over invisibly.+ if (streamStarted || headersSentEarly) {const safe = sanitizeProviderErrorMessage(streamErr.message);traceRouteEvent('Responses', {event: 'fail',@@ -1526,6 +1554,11 @@ export async function handleResponses(req: Request, res: Response): Promise<voidreturn 'committed';}throw streamErr;+ } finally {+ // #1642: this attempt is over whichever path it left by — the+ // keep-alive interval must not outlive it.+ streamSettled = true;+ keepaliveStop();}}