Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

4.0 KB
Raw Blame
/**
 * What a local daemon is CURRENTLY serving, window included — observed from
 * `GET /v1/models`, never persisted.
 *
 * A provider seeded by migration (`opencode-vllm` and friends) stores no
 * `contextWindow` and no `modelContextWindows`, so every context-budget
 * question about it answered "unknown" and every prompt was dispatched
 * regardless of size. On one install that cost ten consecutive 600s timeouts:
 * the endpoint accepted a prompt larger than its window, emitted its banner,
 * and produced 36 bytes in ten minutes (#7441). The daemon knew the answer the
 * whole time — vLLM reports `max_model_len` on the same listing row
 * `providerReadiness.js` already polls.
 *
 * Two rules make this OBSERVED RUNTIME STATE rather than configuration:
 *
 *   1. Nothing here writes `providers.json`. The window is a property of the
 *      process that is running right now — relaunch llama-server at a smaller
 *      `-c` and the stored number would be a lie — and a user editing the
 *      provider must never find a probe's value sitting in their record.
 *   2. An endpoint that is down, unreadable, or silent about its windows
 *      contributes NOTHING. `null`/`{}` flows through as "unknown", which every
 *      consumer already treats as "no constraint", so an install whose daemon
 *      is off routes exactly as it did before this module existed.
 *
 * No LLM call is involved — a model listing is free — so this is safe to ask on
 * the dispatch path under the no-cold-bootstrap policy in AGENTS.md. The listing
 * itself is TTL-cached in `lib/openAiModelsProbeCache.js`, shared with the
 * Providers page's readiness poll.
 */

import { localRuntimeForProvider } from '../lib/localProviderRuntime.js';
import { aliasServedContextWindows, mergeObservedContextWindows } from '../lib/providerContextWindows.js';
import { probeOpenAiModelsCached } from '../lib/openAiModelsProbeCache.js';

/**
 * The windows this provider's daemon is serving right now, keyed by every
 * spelling the provider could dispatch under, or `null` when there is nothing
 * observed.
 *
 * Both spellings are emitted on purpose: the listing keys by the daemon's bare
 * id, while `resolveEffectiveModel` hands downstream code whatever the provider
 * record says — which for an OpenCode wrapper is the `<kind>/`-prefixed form. A
 * map keyed on only one of the two resolves for half the providers and silently
 * answers "unknown" for the rest.
 *
 * @param {object} provider — a RAW provider record (endpoint + envVars intact)
 * @param {{probe?: Function}} [deps]
 * @returns {Promise<Record<string, number>|null>}
 */
export async function observedContextWindows(provider, deps = {}) {
  const runtime = localRuntimeForProvider(provider);
  // Not daemon-backed: a cloud endpoint is somebody else's install to probe,
  // and its catalog windows already arrive through model refresh.
  if (!runtime) return null;

  const probe = deps.probe || probeOpenAiModelsCached;
  const result = await probe(runtime.endpoint, provider?.apiKey || '').catch(() => null);
  return aliasServedContextWindows(provider, runtime, result?.contextWindows);
}

/**
 * `provider` with its daemon's live windows folded into `modelContextWindows`,
 * for a caller about to ask a context question about it.
 *
 * Returns the SAME object when there is nothing observed, so the no-daemon and
 * daemon-down paths allocate nothing and compare identically. The copy is
 * in-memory only — see this module's header for why it is never persisted.
 *
 * Where the observation RANKS is stated once, with the merge itself in
 * `lib/providerContextWindows.js` — the provider card folds the same map in off
 * the readiness payload, so the card and this gate cannot rank it differently.
 * What this wrapper adds is the probe that supplies it.
 *
 * @param {object} provider
 * @param {{probe?: Function}} [deps]
 * @returns {Promise<object>}
 */
export async function withObservedContextWindows(provider, deps = {}) {
  return mergeObservedContextWindows(provider, await observedContextWindows(provider, deps));
}