Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

7.0 KB
Raw Blame
/**
 * One `GET {base}/models` reachability+listing probe for the local
 * OpenAI-compatible daemons (llama.cpp, Ollama, LM Studio, MTPLX).
 *
 * This existed three times in `server/` before it existed once here, and the
 * copies had already drifted: `llamaServerManager`'s passed its timeout as a
 * `timeout` key INSIDE the fetch init object, where it is not an option, so
 * that probe silently ran on the 15s default inside a 500ms startup poll loop.
 * A single implementation makes the timeout, the URL normalization, and the
 * "reachable but unlistable" distinction one decision instead of N.
 *
 * Answers three states a caller must be able to tell apart:
 *   - `reachable: false`             — nothing is serving here (with `error` naming why)
 *   - `reachable: true, models: null`— it answered, but the listing was unreadable
 *   - `reachable: true, models: []`  — it is up and genuinely serving nothing
 *
 * The listing also carries each model's SERVED context window (vLLM spells it
 * `max_model_len`, llama-server `n_ctx`, LM Studio `loaded_context_length`), and
 * `contextWindows` keeps it. Dropping it was how a prompt that provably could
 * not fit still got dispatched: nothing upstream could tell a 50K-char prompt
 * from a 15K one against a 32K endpoint, so the only backstop was the
 * provider's wall-clock timeout (#7441).
 *
 * A daemon started behind an API key (vLLM's compose stack sets `VLLM_API_KEY`)
 * answers 401/403 to an unauthenticated probe. That is a REACHABLE server whose
 * listing we cannot read — reporting it as unreachable would tell the user to
 * start a container that is already running. Pass `apiKey` to read the listing
 * too.
 */

// The toolkit's model refresh already reads a window off a listing row, and a
// second table here would let refresh-time and probe-time disagree about the
// same daemon. Reaching INTO `aiToolkit/internal/` is the established direction
// (`openAiChatStream.js`, `harnessOutput.js`, `providerGraphPreview.js` all do
// it) — the self-containment rule in its AGENTS.md forbids the toolkit
// importing OUT, not the host importing in.
import { catalogContextWindow } from './aiToolkit/internal/modelCatalog.js';
import { fetchWithTimeout } from './fetchWithTimeout.js';
import { describeFetchError } from './fetchErrorChain.js';
import { readResponseJson } from './readResponseJson.js';

/**
 * The one-token reason a probe failed, for a UI that has a line to spend on it.
 * `describeFetchError` returns the whole cause chain (`fetch failed: ECONNREFUSED:
 * connect ECONNREFUSED <host>:<port>`); the code alone is what tells the user
 * "nothing is listening" from "the host is wedged", and the two timeout
 * spellings mean the same thing to them.
 */
function shortFailureReason(err) {
  // `describeFetchError` walks `.code`/`.message` only, and an abort carries its
  // identity in `.name` (`AbortError`) — which is exactly the timeout case.
  const chain = `${err?.name || ''}: ${describeFetchError(err)}`;
  if (/AbortError|TimeoutError|ETIMEDOUT|UND_ERR_(?:CONNECT_)?TIMEOUT/.test(chain)) return 'timed out';
  const code = chain.match(/\b(E[A-Z]{3,}|UND_ERR_[A-Z_]+)\b/);
  return code ? code[1] : chain.slice(0, 120);
}

/**
 * @param {string} baseUrl - an OpenAI-compatible base (…/v1); trailing slashes tolerated
 * @param {{timeoutMs?: number, apiKey?: string, headers?: Record<string,string>}} [opts] —
 *   `headers` is for a listing that authenticates some other way than a bearer
 *   token (the Anthropic API's `x-api-key` + `anthropic-version` pair, #7563);
 *   it is merged over the bearer header when both are given.
 * @returns {Promise<{reachable:boolean, models:string[]|null, contextWindows:Record<string,number>|null, error:string|null}>}
 *   `contextWindows` carries an entry only for a model whose window the listing
 *   actually declared, so `{}` means "nothing declared one" — never "these
 *   models have no window". It is `null` exactly when `models` is. Per-row
 *   parsing (which key, and smallest-wins when a row declares several) is
 *   `catalogContextWindow`'s rule, shared with model refresh.
 */
export async function probeOpenAiModels(baseUrl, { timeoutMs = 2_000, apiKey = '', headers = {} } = {}) {
  const url = `${String(baseUrl || '').replace(/\/+$/, '')}/models`;
  const merged = { ...(apiKey ? { Authorization: `Bearer ${apiKey}` } : {}), ...headers };
  // A redirect would forward custom credential headers (undici only strips
  // `Authorization` cross-origin), so a keyed probe fails instead of following one.
  const carriesKey = Object.keys(merged).some((name) => /^(authorization|x-api-key)$/i.test(name));
  const init = {
    method: 'GET',
    ...(Object.keys(merged).length > 0 ? { headers: merged } : {}),
    ...(carriesKey ? { redirect: 'error' } : {}),
  };
  const res = await fetchWithTimeout(url, init, timeoutMs)
    // undici reports every network failure as a bare `TypeError: fetch failed`;
    // the real reason (ECONNREFUSED vs. ETIMEDOUT — "nothing is listening" vs.
    // "the host is wedged", two different fixes for the user) lives in the
    // cause chain.
    .catch((err) => ({ transportError: shortFailureReason(err) }));

  if (res.transportError) return { reachable: false, models: null, contextWindows: null, error: res.transportError };
  if (!res.ok) {
    // Undici holds the socket until an unread body is consumed; an endpoint
    // answering 404 on every poll would otherwise leak one each time.
    await res.body?.cancel().catch(() => {});
    // 401/403 is the ONE failure status that proves a daemon is up: something
    // read the request and refused it. Collapsing that into "nothing answered"
    // sends the user off to start a server that is already serving.
    if (res.status === 401 || res.status === 403) {
      return { reachable: true, models: null, contextWindows: null, error: 'authentication required' };
    }
    return { reachable: false, models: null, contextWindows: null, error: `HTTP ${res.status}` };
  }

  // The body read is its own failure path: a daemon that accepts the connection
  // and then drops it (or sends a truncated body) rejects inside `res.text()`,
  // NOT at the fetch above. Unhandled, that rejection escapes the probe and
  // fails the entire readiness request — one flaky daemon would blank the
  // checklist for every provider — so it lands on the same
  // reachable-but-unreadable sentinel as a non-JSON body.
  const body = await readResponseJson(res, { fallback: null, emptyValue: null }).catch(() => null);
  const rows = Array.isArray(body?.data) ? body.data : Array.isArray(body?.models) ? body.models : null;
  if (!rows) return { reachable: true, models: null, contextWindows: null, error: 'model listing was not readable' };
  const models = [];
  const contextWindows = {};
  for (const row of rows) {
    const id = typeof row === 'string' ? row : (row?.id || row?.name);
    if (typeof id !== 'string' || id === '') continue;
    models.push(id);
    const tokens = catalogContextWindow(row);
    if (tokens) contextWindows[id] = tokens;
  }
  return { reachable: true, models, contextWindows, error: null };
}