Skip to content

Never stop coding. Free MIT AI gateway: one endpoint, 375 providers (150+ free), 1,400+ models Kimi, Claude, GPT, Gemini, GLM, DeepSeek, MiniMax. Works with Claude Code, Codex, Cursor, OpenCode, Cline & Copilot. Quota-aware auto-fallback, RTK+Caveman compression saves 15-95% tokens, MCP/A2A, Desktop/PWA. Built by 900+ contributors

16.8 KB
Raw Blame
"use strict";

/**
 * HTTP client-abort / recoverable-upstream-timeout crash guard
 * (#fix-dev-server-aborted, #12861).
 *
 * Node's http.Server turns an 'error' event on an IncomingMessage/ServerResponse
 * into an uncaughtException (and therefore a process exit) WHENEVER the emitter
 * has no listener. The single most common such error is a *client* abort: the
 * browser closes the TCP socket (navigation, Back/Forward cache, HMR reconnect,
 * cancelling a fetch) while the server is still streaming the response. Node
 * emits `Error: aborted` / `ERR_STREAM_PREMATURE_CLOSE` / `ECONNRESET` on the
 * request stream, and absent a handler it kills the whole server process.
 *
 * That surfaced as "login succeeds, then the dashboard hangs with a wall of
 * `net::ERR_CONNECTION_REFUSED`": after auth the SPA opens many polling
 * connections + a live WebSocket; stray client-side socket closes during
 * navigation/HMR were taking the dev server down.
 *
 * Two more categories were added after the 2026-09-14 agnes-cn upstream storm
 * produced two sibling escapes in production: an intentional combo hedge
 * cancellation (`AbortError: hedge-cancelled` — the sibling leg already won,
 * so the cancellation is expected, not a fault) and undici fetch failures
 * (`TypeError: fetch failed` with a socket-level code) against a flapping
 * upstream. Both are runtime/environmental conditions the request layer
 * already handles; neither is a process-fatal logic bug.
 *
 * A further, unrelated category covers #12861: `directFetchWithBoundedResponseStart`'s
 * response-start timeout (`DIRECT_RESPONSE_START_TIMEOUT`) is a *recoverable*
 * signal `proxyFetch.ts` already retries on a fresh socket — but a narrow
 * timer/promise-settlement race can still deliver its abort reason to a
 * promise nobody is awaiting anymore, which otherwise kills the whole process
 * over a single upstream stall that the retry path was built to handle.
 *
 * The same escape reached production on 2026-10-02 (exit code 7) with the
 * pooled relay path's per-attempt timeout (`RELAY_TIMEOUT`,
 * `open-sse/utils/proxyFetch.ts`): a stray copy of the 504 the combo/executor
 * layer already handles took the process down. Both timeout codes are
 * classified in `isRecoverableUpstreamTimeoutError` below.
 *
 * It escaped again on 2026-10-11 (exit code 7, #16347) with a stream-lifecycle
 * error: an empty Claude upstream (`empty_response`, stamped by
 * `open-sse/utils/stream.ts::emitClaudeEmptyStreamErrorAndAbort`) had already
 * been reported as a 502 / combo failover, but while the combo loop sat in its
 * retry wait nothing consumed the errored transform stream, so the rejection
 * reached this guard unclassified. Those errors carry the
 * `__omniroutePendingRequestCleared` marker (`PENDING_REQUEST_CLEARED_MARKER`,
 * `open-sse/utils/stream.ts`) once the stream layer has cleared the request's
 * pending entry — they are classified in `isHandledStreamLifecycleError`
 * below.
 *
 * Two layers:
 *   1. `attachRequestStreamGuards(req, res)` — per-request listeners that absorb
 *      client-abort errors so they never bubble to the process level. Call it
 *      inside every `http.createServer((req, res) => …)` request listener.
 *   2. `installProcessCrashGuard()` — a last-resort safety net on
 *      `process.on('uncaughtException' | 'unhandledRejection')` that swallows
 *      the same benign errors but otherwise preserves the existing crash
 *      semantics (so genuine bugs still surface). Idempotent.
 *
 * Kept as a `.mjs` module (no build step) so it is importable both from the
 * Node-only dev server (`scripts/dev/run-next.mjs`) and from the TypeScript
 * servers under `src/` (tsconfig `allowJs: true`).
 *
 * @module
 */

/**
 * Abort reasons raised by combo target dispatch. Mirror of
 * open-sse/services/combo/comboAbortReasons.ts — keep the two in sync.
 */
const COMBO_ABORT_REASONS = new Set(["hedge-cancelled", "combo-per-model-timeout"]);

/**
 * Raw string reasons open-sse/utils/streamHandler.ts passes to
 * handleDisconnect() / abortController.abort() when the client goes away.
 */
const CLIENT_DISCONNECT_REASONS = new Set(["request_signal_aborted", "client_closed", "cancelled"]);

/**
 * @param {unknown} err
 * @returns {boolean} true when `err` represents a client closing the
 *   connection rather than a server-side fault.
 */
export function isClientAbortError(err) {
  // open-sse/utils/streamHandler.ts aborts the stream controller with a RAW
  // STRING reason (getClientAbortReason / handleDisconnect), and undici rejects
  // with signal.reason verbatim, so a cancellation can surface at the process
  // level as a bare string rather than an Error object.
  if (typeof err === "string") {
    return COMBO_ABORT_REASONS.has(err) || CLIENT_DISCONNECT_REASONS.has(err);
  }
  if (!err || typeof err !== "object") return false;
  const e = /** @type {NodeJS.ErrnoException} */ (err);
  // Node emits `Error: aborted` (no code) from http.Server#abortIncoming.
  if (e.message === "aborted" || e.message === "Aborted") return true;
  // OmniRoute's SSE teardown aborts in-flight legs with
  // `Error [AbortError]: request_signal_aborted` on client disconnects
  // (open-sse/utils/streamHandler.ts), and fetch/DOM cancellation surfaces as
  // `AbortError` with an abort-flavoured message. Same benign class as
  // `Error: aborted` — an emitter-left 'error' event on any of these used to
  // kill the process (#fix-dev-server-aborted).
  if (e.name === "AbortError" && /abort/i.test(String(e.message))) return true;
  // Combo dispatch cancels a losing hedged target / a stalled target by
  // aborting with `new Error(reason)` for one of the reasons in
  // open-sse/services/combo/comboAbortReasons.ts (targetTimeoutRunner.ts).
  // streamHandler.ts forwards that reason to the stream controller as a plain
  // string, and a leaked abort listener in
  // open-sse/handlers/chatCore/upstreamTimeouts.ts (executeWithUpstreamStartTimeout)
  // rebuilt it via createAbortError() as an AbortError-named Error to reject a
  // promise nothing was awaiting. That unhandledRejection reached this guard,
  // which re-threw it as an uncaughtException. Production exit 2026-08-31:
  //   Error [AbortError]: hedge-cancelled
  // The listener leak is fixed at the source; this stays as the last-resort net.
  // A sibling target winning / a target stalling is never a server fault, so
  // match the exact reason text whatever `name` the thrower stamped on it.
  // Inlined rather than imported from comboAbortReasons.ts: this file runs
  // under plain node (scripts/dev/run-next.mjs, no tsx) and must not depend on
  // type-stripping being available for that .ts module.
  if (COMBO_ABORT_REASONS.has(String(e.message))) return true;
  switch (e.code) {
    case "ERR_STREAM_PREMATURE_CLOSE":
    case "ECONNRESET":
    case "EPIPE":
    case "ECONNABORTED":
    case "ETIMEDOUT":
    case "ENOTCONN":
    case "ECANCELED":
      return true;
    default:
      return false;
  }
}

/**
 * A recoverable upstream-fetch timeout whose request-layer handling already
 * exists, but a stray copy can still reach this guard as an
 * unhandledRejection/uncaughtException and must not be process-fatal:
 *
 * - `DIRECT_RESPONSE_START_TIMEOUT` (#12861): `proxyFetch.ts` retries it on a
 *   fresh socket (see `open-sse/utils/directResponseStartTimeout.ts`). A narrow
 *   timer/promise-settlement race can deliver its abort reason to a promise
 *   nobody is awaiting anymore.
 * - `RELAY_TIMEOUT` (2026-10-02 production exit 7): the pooled relay path
 *   (`open-sse/utils/proxyFetch.ts`, #9100/#9158) fails a hung attempt fast as
 *   a 504 after `RELAY_FETCH_TIMEOUT_MS` and the combo/executor layer falls
 *   back. The thrown error escaped its execute chain as a stray rejection and
 *   this guard re-threw it because the code was not classified here.
 *
 * Kept as bare string-code checks (no import of the `.ts` source of truth)
 * because this file has to stay build-free/plain-JS-loadable — see the module
 * docstring. Canonical definitions: `DIRECT_RESPONSE_START_TIMEOUT_CODE` in
 * `open-sse/utils/directResponseStartTimeout.ts`, and the `RELAY_TIMEOUT` /
 * `relay_timeout` stamps in `open-sse/utils/proxyFetch.ts` (the same pair
 * `src/lib/usage/glmResetCards.ts` matches on). Keep these literals in sync
 * with them.
 *
 * @param {unknown} err
 * @returns {boolean}
 */
export function isRecoverableUpstreamTimeoutError(err) {
  // Same reason-shape tolerance as isIntentionalComboAbort: a bare string
  // reason rejects waiters with the string itself, not an Error object.
  if (
    err === "DIRECT_RESPONSE_START_TIMEOUT" ||
    err === "RELAY_TIMEOUT" ||
    err === "relay_timeout"
  ) {
    return true;
  }
  if (!err || typeof err !== "object") return false;
  const e = /** @type {NodeJS.ErrnoException & { errorCode?: string }} */ (err);
  // open-sse's withBodyTimeout rejects with a plain Error (no .code). Restrict
  // this match to its exact name and message shape to preserve crash semantics.
  if (
    e.name === "BodyTimeoutError" &&
    typeof e.message === "string" &&
    /^Response body read timeout after [1-9]\d*ms$/.test(e.message)
  ) {
    return true;
  }
  return (
    e.code === "DIRECT_RESPONSE_START_TIMEOUT" ||
    e.code === "RELAY_TIMEOUT" ||
    e.errorCode === "relay_timeout"
  );
}

/**
 * Intentional combo-leg cancellation. When a combo dispatches hedged targets,
 * the losing legs are aborted with a distinctive reason once a sibling wins
 * (`hedge-cancelled`) or exceeds its per-model budget (`combo-per-model-timeout`)
 * — see `COMBO_HEDGE_CANCELLED_REASON` / `COMBO_PER_MODEL_TIMEOUT_REASON` in
 * `open-sse/services/combo/comboAbortReasons.ts` (bare literals duplicated here
 * because this file must stay build-free; keep in sync). On 2026-09-14 such a
 * cancellation escaped its promise chain and killed production with
 * `Error [AbortError]: hedge-cancelled` — the request it belonged to had
 * already completed 200 via the winning leg.
 *
 * Distinct from a *client* abort: only these exact reasons qualify, so an
 * AbortError from an unknown subsystem still crashes loudly.
 *
 * @param {unknown} err
 * @returns {boolean}
 */
export function isIntentionalComboAbort(err) {
  const reasons = new Set(["hedge-cancelled", "combo-per-model-timeout"]);
  // AbortSignal.reason is whatever was handed to abort(): a raw string
  // reason rejects waiters with the string itself, not an Error object.
  if (typeof err === "string") return reasons.has(err);
  if (!err || typeof err !== "object") return false;
  const e = /** @type {NodeJS.ErrnoException} */ (err);
  if (e.name !== "AbortError") return false;
  if (reasons.has(String(e.message))) return true;
  const cause = /** @type {{ cause?: unknown }} */ (err).cause;
  return typeof cause === "string" && reasons.has(cause);
}

/**
 * A network/IO failure against an upstream or its proxy — undici surfaces it
 * as `TypeError: fetch failed` (fixed message; the syscall code rides on
 * `cause`) or as an error carrying a `PROXY_UNREACHABLE` / `UND_ERR_*` code.
 * On 2026-09-14 one of these (`PROXY_UNREACHABLE` / ECONNRESET to
 * api.agnes-ai.cn) escaped as an uncaughtException and killed production.
 * The request that triggered the fetch already fails through the normal
 * error path; the stray copy delivered to nobody must not be process-fatal.
 *
 * The "fetch failed" message match is exact on purpose: it is undici's fixed
 * wrapping message, so arbitrary TypeErrors still crash loudly.
 *
 * @param {unknown} err
 * @returns {boolean}
 */
export function isUpstreamNetworkError(err) {
  if (!err || typeof err !== "object") return false;
  const e = /** @type {NodeJS.ErrnoException} */ (err);
  if (e.name === "TypeError" && e.message === "fetch failed") return true;
  switch (e.code) {
    case "PROXY_UNREACHABLE":
    case "PROXY_REQUEST_FAILED":
    case "UND_ERR_SOCKET":
    case "UND_ERR_CONNECT_TIMEOUT":
    case "UND_ERR_HEADERS_TIMEOUT":
    case "UND_ERR_BODY_TIMEOUT":
    case "ECONNREFUSED":
    case "EHOSTUNREACH":
    case "ENETUNREACH":
    case "EAI_AGAIN":
      return true;
    default:
      return false;
  }
}

/**
 * A stream-lifecycle error OmniRoute's own streaming pipeline already accounted
 * for. `open-sse/utils/stream.ts` stamps `markPendingRequestCleared()` on every
 * stream error whose pending-request entry was cleared at emit time (the empty
 * Claude flush `empty_response`, the upstream-start timeout, the empty-choices
 * flush, and the `streamFailureBoundary.ts` fallback) — the request layer has
 * by then already surfaced the failure (502 error event, combo failover,
 * single-model retry classification), so a stray copy delivered to nobody is a
 * bookkeeping leftover, not a fault. Kept as literal checks because this file
 * must stay build-free: canonical definitions are `PENDING_REQUEST_CLEARED_MARKER`
 * and the `empty_response` stamp in `open-sse/utils/stream.ts` (keep in sync).
 *
 * 2026-10-11 production exit 7 (#16347): the combo retry wait left the errored
 * transform stream unconsumed and the `empty_response` rejection escaped as an
 * uncaughtException. Same escape class as RELAY_TIMEOUT (#15339) and
 * RATE_LIMIT_QUEUE_TIMEOUT (#15706).
 *
 * @param {unknown} err
 * @returns {boolean}
 */
export function isHandledStreamLifecycleError(err) {
  if (err === "empty_response") return true;
  if (!err || typeof err !== "object") return false;
  const e = /** @type {Error & { code?: string; __omniroutePendingRequestCleared?: boolean }} */ (
    err
  );
  if (e.__omniroutePendingRequestCleared === true) return true;
  return e.code === "empty_response";
}

/**
 * Decide whether a process-level uncaughtException/unhandledRejection should be
 * swallowed (benign client-abort, or a recoverable upstream timeout the
 * request layer already handles — #12861 / RELAY_TIMEOUT) or allowed to
 * surface (genuine bug).
 *
 * Pure + exported so it can be unit-tested without poking process listeners.
 *
 * @param {unknown} err
 * @param {string | undefined} origin  Node's uncaughtException origin (e.g.
 *   "uncaughtException" / "unhandledRejection"); absent/empty for rejections.
 * @returns {boolean} true => swallow (log only), false => re-throw / let crash.
 */
export function shouldSwallowUncaught(err, origin) {
  if (
    !isClientAbortError(err) &&
    !isRecoverableUpstreamTimeoutError(err) &&
    !isIntentionalComboAbort(err) &&
    !isUpstreamNetworkError(err) &&
    !isHandledStreamLifecycleError(err)
  ) {
    return false;
  }
  // Only swallow when the origin matches what the guard installed for. If some
  // other subsystem raised it (e.g. a deliberate `throw` in a domain), keep the
  // existing crash semantics.
  return !origin || origin === "uncaughtException" || origin === "unhandledRejection";
}

/**
 * Attach `error` listeners to a request/response pair that swallow client-abort
 * errors. Idempotent per (req, res) pair via a Symbol flag.
 *
 * @param {import("node:http").IncomingMessage} req
 * @param {import("node:http").ServerResponse} res
 */
export function attachRequestStreamGuards(req, res) {
  const flag = Symbol.for("omniroute.requestAbortGuard");
  if (req[flag] || res[flag]) return;
  req[flag] = true;
  res[flag] = true;

  req.on("error", (err) => {
    if (!isClientAbortError(err)) {
      // Re-emit a genuine request error through the normal channel so it is
      // still observable in logs, but never as an uncaughtException.
      console.error("[server] request stream error:", err);
    }
  });

  res.on("error", (err) => {
    if (!isClientAbortError(err)) {
      console.error("[server] response stream error:", err);
    }
  });
}

let crashGuardInstalled = false;

/**
 * Install process-level safety nets. Idempotent. Benign client-abort errors are
 * logged once and swallowed; everything else is re-thrown on a fresh stack so
 * the process keeps its current crash semantics (genuine bugs still crash/hang
 * loudly, and a supervisor or test harness sees them).
 *
 * @param {(level: "warn" | "error", ...args: unknown[]) => void} [log]
 */
export function installProcessCrashGuard(log) {
  if (crashGuardInstalled) return;
  crashGuardInstalled = true;

  // `console` is an object, not a callable: `log ?? console` followed by
  // `logger("warn", ...)` throws TypeError and kills the process on the very
  // abort the guard exists to swallow. Default to console.warn as a function.
  const logger = typeof log === "function" ? log : console.warn.bind(console);

  process.on("uncaughtException", (err, origin) => {
    if (shouldSwallowUncaught(err, origin)) {
      // The warn line is the only evidence a swallowed error ever happened;
      // pass the full error object so the stack survives.
      logger("warn", "[server] swallowed benign uncaughtException:", err);
      return;
    }
    throw err;
  });

  process.on("unhandledRejection", (reason) => {
    if (shouldSwallowUncaught(reason, "unhandledRejection")) {
      logger("warn", "[server] swallowed benign unhandledRejection:", reason);
      return;
    }
    throw reason;
  });
}