Skip to content

7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.

TypeScript
3.4 KB
Raw Blame
// SSE keep-alive pings for the uncommitted streaming window (#1642).
//
// The streaming routes hold every byte — headers included — until the first
// real text token, so a reasoning model can leave the wire silent for 30-180s
// while it thinks. Clients with a time-to-first-byte watchdog (Claude Code's
// stream-idle timeout is the reference case) abort during that window, the
// router logs "client disconnected mid-attempt", and the user reruns the turn.
//
// When STREAM_KEEPALIVE_MS is set, each streaming pump starts an interval
// that emits a content-free keep-alive frame while the attempt is in flight.
// On the OpenAI surfaces that is a bare SSE comment line (`: keepalive`),
// which every spec-compliant parser ignores; on the Anthropic surface it is
// the protocol's own `ping` event.
//
// The invariant this preserves: the ping carries NO content, so nothing the
// client can act on has been sent. What it necessarily gives up when enabled
// is invisible failover FOR THAT ATTEMPT — a ping flushes the SSE headers, so
// a stream that then dies ends the response with an error frame instead of
// retrying another provider behind the scenes. That trade is exactly what an
// operator opts into; the default is OFF, keeping the current behavior for
// everyone who has not heard of the problem.
//
// Pings keep firing on a committed-but-silent stream too (upstream stalled
// mid-reasoning), which is also useful: an idle upstream no longer looks
// different from an idle connection at the client either way, but the extra
// bytes stop the client from killing a stream that is merely slow.
const MIN_KEEPALIVE_MS = 1000;

/**
 * The keep-alive interval in ms, parsed from STREAM_KEEPALIVE_MS.
 * Unset / empty / 0 / non-numeric / negative → 0 (feature off, default).
 * Positive values are clamped to a 1s floor: a sub-second ping interval just
 * burns bandwidth and adds SSE frames no client asked for.
 */
export function streamKeepaliveMs(env: NodeJS.ProcessEnv = process.env): number {
  const raw = env.STREAM_KEEPALIVE_MS;
  if (raw === undefined || raw.trim() === '') return 0;
  const n = Number(raw);
  if (!Number.isFinite(n) || n <= 0) return 0;
  return Math.max(MIN_KEEPALIVE_MS, Math.floor(n));
}

/**
 * Start the keep-alive interval for one stream attempt.
 *
 * - `intervalMs` ≤ 0 disables the feature: no timer is created and the
 *   returned stop function is a no-op.
 * - `isDone` is polled at every tick: once the attempt has finished (response
 *   ended, failed over, or errored), the timer stops itself.
 * - `keepalive` runs on each tick and must write a content-free frame. It is
 *   expected to commit the response first if it has not been committed yet.
 *   Throwing (dead socket) stops the timer without surfacing anything —
 *   the pump's own error paths notice the broken connection.
 *
 * Returns the stop function; call it when the stream ends. Idempotent.
 */
export function startStreamKeepalive(
  intervalMs: number,
  isDone: () => boolean,
  keepalive: () => void,
): () => void {
  if (intervalMs <= 0) return () => {};
  const timer = setInterval(() => {
    if (isDone()) { stop(); return; }
    try {
      keepalive();
    } catch {
      stop();
    }
  }, intervalMs);
  // Never hold the process open for a ping that outlived its request.
  if (typeof timer.unref === 'function') timer.unref();
  function stop(): void {
    clearInterval(timer);
  }
  return stop;
}