Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
// Canonical server-side image understanding (VLM completion) + the platform's
// vision provider policy. The image-side twin of api/_lib/llm.js — same
// free-first doctrine, same spend-ledger discipline, so vision never becomes a
// per-endpoint reinvention that drifts out of policy.
//
// Policy (identical to llm.js — read that file's header for the rationale):
//
// • FREE NIM VISION LANES FIRST, ALWAYS. NVIDIA NIM hosts several VLMs on the
// OpenAI-compatible chat host (integrate.api.nvidia.com) at zero marginal
// cost to the platform. They lead every chain, tried in order, and every
// consumer must survive on them alone.
//
// • Paid vision-capable backstop LAST, automatically. When OPENAI_API_KEY is
// configured, gpt-5.4-nano (vision-capable) is appended to the tail so a
// request that exhausted the free lanes still succeeds. It never leads, and
// no consumer hard-fails when it is absent.
//
// • NOTHING HARD-FAILS ON A VISION OUTAGE. describeImage throws on total
// failure, but every consumer is required to treat that as "skip the
// vision-derived enhancement", never as an error the end user sees. See
// visionConfigured() for the gate, and each consumer's degraded path.
//
// Image input — pass EITHER an http(s) URL (default; the model server fetches it
// — used for first-party R2 URLs and already-validated claim image URLs) OR a
// base64 blob + mimeType (inlined as a data URI). Both verified live against
// every NIM lane; see tasks/nvidia-nim/probes/vision.md.
import { isIP } from 'node:net';
import { env } from './env.js';
import {
AUTH_COOLDOWN_SECONDS,
clearProviderCooldown,
markProviderCooldown,
providersInCooldown,
} from './provider-health.js';
import { recordEvent } from './usage.js';
import { costMicroUsd } from './llm-pricing.js';
import { OPENROUTER_ATTRIBUTION_HEADERS } from './openrouter-attribution.js';
import { validatePublicUrl, isPrivateAddress, SsrfError } from './ssrf.js';
import { fetchSafePublicUrl } from './ssrf-guard.js';
import {
vertexGeminiAvailable,
vertexGeminiModel,
vertexGeminiChatUrl,
vertexGeminiHeaders,
} from './vertex-gemini.js';
// Free NIM vision lanes, in order.
//
// `nvidia/nemotron-nano-12b-v2-vl` used to lead here for its small image token
// footprint. NVIDIA retired it: the host answers every request for it with a
// hard 410 ("has reached its end of life on 2026-08-26T09:00:00Z"), verified
// against production on 2026-09-08. A 410 never recovers, so leaving it in the
// list only spent the first, largest slice of every vision request's deadline
// on a model that cannot answer. Adding a replacement free NIM VLM needs a live
// NVIDIA_API_KEY to probe the catalog against, which is why this is a removal
// rather than a swap.
const NVIDIA_VISION_MODELS = [
'meta/llama-3.2-11b-vision-instruct',
];
// Free OpenRouter vision lanes, in order.
//
// The text chain (llm.js providerChain) has carried an OpenRouter rung for a
// long time; vision never did, and that gap took the whole endpoint down on
// 2026-09-09: NVIDIA answered 500, the paid OpenAI backstop answered 429
// `billing_not_active`, and Vertex answered 403 `Lightning dunning decision is
// deny for project` (a project-level billing hold that denies Vertex AND the
// AI-Studio Gemini endpoint on the same key, so both Google rungs die together).
// Three rungs, one dead chain, and every consumer that judges an image with it
// (catalog seeding's quality gate, forge validation, alt text) went dark.
//
// Only the `:free` suffix is used here. llm-pricing.js prices OpenRouter by
// exactly that suffix (isOpenRouterFreeModel), so a `:free` route is metered at
// zero and the spend ledger stays truthful; OpenRouter's `openrouter/free`
// router meta-model answers fine but would be metered as paid, so it is
// deliberately not in this list.
//
// Verified against the live platform key on 2026-09-09: nemotron-omni answered
// 3/3 in a median 1.5 s with clean parseable JSON, which is what the seed judge
// needs (api/_lib/seed-quality.js parses every reply as JSON). The two Gemma
// routes were saturated (429) in the same probe and are kept behind it as spare
// capacity: a 429 falls through in ~150 ms and, unlike the 410 that retired the
// NIM model above, it recovers when the shared free pool drains.
//
// The nemotron route is a REASONING model, and OpenRouter's own model record
// says `default_enabled: true`, so left alone it thinks before it answers. In a
// bulk text run that is merely expensive; inside a vision lane's slice of a
// shared deadline it is fatal, because the thinking consumes the budget and the
// lane returns an EMPTY message.content. Probing /api/vision on production
// (2026-09-10) caught it doing exactly that: provider openrouter, model
// nemotron-omni, text ''. The same record says `mandatory: false`, which is
// OpenRouter's way of saying this model accepts being told not to, so the lane
// carries reasoning.effort:'none'.
//
// Note the parameter is host-specific and copying the wrong one ships a no-op:
// NVIDIA's own host takes chat_template_kwargs.enable_thinking (see
// scripts/i18n-translate.mjs), OpenRouter takes reasoning.effort. Check a
// model's `reasoning` object at https://openrouter.ai/api/v1/models before
// assuming either works; a model with `mandatory: true` rejects 'none' with a
// 400.
// Enumerated from https://openrouter.ai/api/v1/models on 2026-09-10 by filtering
// `:free` routes whose architecture.input_modalities includes "image": ten
// exist, and this list carries nine of them. The tenth,
// nvidia/nemotron-3.5-content-safety:free, is a safety classifier rather than a
// general VLM and is deliberately excluded; asking it to score a render is not
// what it does.
//
// ORDER IS A HEURISTIC, not a measurement, and it is ranked by how likely a
// route is to emit clean JSON rather than narrate:
// 1. Routes whose model record says reasoning default_enabled:false lead.
// They have nothing to switch off and nothing to narrate.
// 2. Then the dedicated vision (VL) route.
// 3. Then general routes with no declared reasoning default.
// 4. Reasoning-by-default routes go last, each carrying reasoning.effort:none
// (see the note above about the host-specific parameter). nemotron-omni
// sits at the back of that group because it is the one caught returning an
// empty message.content in production.
// Replace this ordering with measured latency and JSON-success rates by running
// `node scripts/probe-vision-lanes.mjs`, which sends the real rubric payload to
// every route and ranks them.
//
// Pruned 2026-09-30 against a live image probe: ling-3.0-flash-vl and both
// nex-agi routes lost their free tier (404 "unavailable for free") and both
// inkling routes are now restricted to OpenRouter's agentic surfaces (403), so
// every vision request was paying five dead round trips. qwen3.8-27b:free
// joined: the catalog lists it with image input.
const OPENROUTER_VISION_MODELS = [
{ model: 'google/gemma-4-31b-it:free' },
{ model: 'google/gemma-4-26b-a4b-it:free' },
{ model: 'qwen/qwen3.8-27b:free' },
{ model: 'dots-studio/dots-3-note-preview:free' },
{ model: 'nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free', extraBody: { reasoning: { effort: 'none' } } },
];
// Cloudflare Workers AI's vision route. Llama 4 Scout, not the 11B Llama 3.2
// vision model the NIM lane uses: on Workers AI that one sits behind a Meta
// license click-through (403 "Model Agreement") that has to be accepted on the
// account by its owner, while Scout answered a real image correctly in ~2s
// with no gate (probed 2026-09-30).
const CLOUDFLARE_VISION_MODEL = '@cf/meta/llama-4-scout-17b-16e-instruct';
// Paid last-resort tail. gpt-5.4-nano is vision-capable and already priced in
// llm-pricing.js, keeping the backstop cheap and the spend ledger truthful.
const OPENAI_VISION_MODEL = 'gpt-5.4-nano';
// ── Lane health and budget policy ───────────────────────────────────────────
//
// Two failures were costing /api/vision its 502s and 504s in production (79
// and 35 respectively in the week to 2026-08-27), and neither was a provider
// being genuinely unable to answer:
//
// 1. NO COOLDOWN. Unlike the chat chain, vision never recorded a throttling
// lane, so every incoming request re-picked the same rate-limited NIM as
// attempt zero, waited out its timeout, and only then failed over. Under
// any sustained throttle that is the whole latency budget spent on a lane
// already known to be refusing. The chat chain's remedy applies verbatim:
// skip a cooling lane, and clear the cooldown the moment one answers.
//
// 2. THE FIRST LANE COULD EAT THE WHOLE DEADLINE. The per-attempt timeout was
// min(timeoutMs, remaining), so a single hung lane consumed the entire
// 24s budget and the request 504'd having tried exactly one provider, with
// a healthy Vertex anchor sitting untried behind it. Splitting the
// remaining budget across the lanes that are still to come guarantees
// every rung a real attempt, which is the only reason a chain exists.
//
// Free NIM rungs share one host, so a transport failure or a 429 there is a
// statement about the HOST, not the model: cooling only the model that happened
// to be asked would send the very next request to a sibling on the same sick
// host. Model-specific rejections (a 404 for a retired model id, a 400) cool
// just that lane, and a 410 parks it for the long window (see below).
const VISION_LANE_COOLDOWN_SECONDS = 45;
// Below this a lane cannot complete a VLM call, so handing it a smaller slice
// only burns budget the next rung could have used.
const MIN_LANE_ATTEMPT_MS = 3_500;
// How many rungs behind the current one the budget is held open for. Two means
// a request plans for three attempts: a generous one and two fallbacks.
const MAX_RESERVED_LANES = 2;
// Share of the remaining deadline the image inline fetch may take. It runs
// BEFORE any lane, so an uncapped one starves the whole chain: at the measured
// 20s timeout against a 24s deadline it left 4s for every provider combined.
const INLINE_BUDGET_SHARE = 0.25;
const INLINE_MAX_MS = 8_000;
/** Cooldown key for one lane. Model-scoped: two lanes on one host are distinct rungs. */
function laneKey(p) {
return `vision:${p.name}:${p.model}`;
}
/** The host a lane talks to, used to cool every sibling rung when the host itself is sick. */
function laneHost(p) {
try {
return new URL(p.url).host;
} catch {
return p.url;
}
}
/**
* Cool `lane` for `seconds`, and every sibling lane sharing its host when the
* failure was about the host (transport error, rate limit, auth) rather than
* about the model. Fire-and-forget: provider-health never throws.
*/
function coolLane(chain, lane, { seconds, reason, hostWide }) {
const targets = hostWide
? chain.filter((p) => laneHost(p) === laneHost(lane))
: [lane];
for (const t of targets) void markProviderCooldown(laneKey(t), seconds, reason);
}
/**
* Per-attempt timeout for the lane about to be tried. Splits what is left of the
* deadline evenly across the lanes still to come so the first rung can never
* consume the budget of the rest, and never exceeds the caller's own timeoutMs.
* With no deadline the caller's timeout stands unchanged.
*
* @param {number} remainingMs ms left on the overall deadline (Infinity when none)
* @param {number} lanesLeft lanes still to try, including this one
* @param {number} timeoutMs caller's per-attempt ceiling
*/
export function laneAttemptTimeout(remainingMs, lanesLeft, timeoutMs) {
if (!Number.isFinite(remainingMs)) return timeoutMs;
// Reserve the floor for the rungs behind this one, then give this lane the
// rest. It used to divide the budget EQUALLY, which quietly punishes the
// chain for having depth: every lane added to the list shrinks the slice the
// FIRST and best lane gets, even though that lane is the one most likely to
// answer. Adding three OpenRouter rungs on 2026-09-09 took the free NIM
// lane's slice from about 9.7s to about 4.8s against the forge quality
// gate's 29s deadline, which is just under what it needs for the scoring
// rubric, and the gate went from 5 verdicts in 10 to 0 in 10 (measured on
// production 2026-09-10). Nothing about either lane changed; the arithmetic
// did.
//
// Reserving instead of dividing keeps the property the split existed for (a
// hung lane cannot eat the whole deadline, and every remaining rung is still
// guaranteed MIN_LANE_ATTEMPT_MS) while letting the lane in hand use the
// budget nobody else needs yet. A lane that fails fast hands its unused
// share straight to the next one, because `remainingMs` is re-read per
// attempt.
// Reserve for the next few rungs, NOT for all of them. Reserving per remaining
// lane makes the chain punish its own depth a second way: at twelve rungs the
// reserve (3.5s x 11) exceeds the whole 29s deadline, so the first and best
// lane collapses back to the floor and we are back to the starvation this
// formula was written to end. A request cannot realistically walk twelve
// providers anyway; it should spend its budget on the two or three most
// likely to answer. Capping the reserve decouples the DEPTH of the chain
// (how many providers exist, which is pure upside as cooldowns rotate
// across requests) from the SPEND of one request (bounded, and weighted
// toward the lane in hand).
const reserve = MIN_LANE_ATTEMPT_MS * Math.min(Math.max(0, lanesLeft - 1), MAX_RESERVED_LANES);
const share = Math.max(MIN_LANE_ATTEMPT_MS, remainingMs - reserve);
// The floor may exceed the share when the budget is nearly spent; capping it
// by what is actually left keeps the attempt inside the deadline either way.
//
// Math.floor is load-bearing, not cosmetic: this value is handed straight to
// AbortSignal.timeout(), which rejects a non-integer delay with
// ERR_OUT_OF_RANGE. The division above produces an integer only when the
// remaining budget happens to divide evenly by the lane count, so in
// production the fallback rung usually threw before it sent a single byte
// and was recorded as "unreachable" ("The value of \"delay\" is out of range.
// It must be an integer. Received 7924.333333333333"). That silently cost the
// chain the very rung it exists for, on every request that reached it.
return Math.floor(Math.max(1, Math.min(timeoutMs, remainingMs, Math.max(MIN_LANE_ATTEMPT_MS, share))));
}
/**
* Budget for the pre-chain image inline fetch: a slice of the remaining
* deadline, never more than INLINE_MAX_MS and never more than the caller's
* timeout. Exported for the budget regression test.
*/
export function inlineImageBudget(remainingMs, timeoutMs) {
const cap = Math.min(timeoutMs, INLINE_MAX_MS);
if (!Number.isFinite(remainingMs)) return cap;
return Math.max(1_000, Math.min(cap, Math.floor(remainingMs * INLINE_BUDGET_SHARE)));
}
// Thrown when no vision provider is available at all. Carries an HTTP status so
// a handler that *chose* to surface it can return 503 — but consumers should
// generally catch it and degrade silently instead.
export class VisionUnavailableError extends Error {
constructor(message = 'No vision provider available. Configure NVIDIA_API_KEY or OPENROUTER_API_KEY (free), GOOGLE_CLOUD_PROJECT (Vertex Gemini credits anchor), or OPENAI_API_KEY (paid backstop).') {
super(message);
this.name = 'VisionUnavailableError';
this.code = 'vision_unavailable';
this.status = 503;
}
}
// One OpenAI-compatible vision provider entry. The multimodal user message is
// the only shape difference from llm.js's text providers. `getHeaders` (async)
// replaces the static key header for keyless lanes whose auth is minted per
// request (the Vertex Gemini credits anchor).
function openaiCompatVisionProvider({ name, key, url, model, getHeaders = null, extraBody = null, extraHeaders = null }) {
return {
name,
model,
url,
...(getHeaders
? { getHeaders }
: { headers: { 'content-type': 'application/json', authorization: `Bearer ${key}`, ...(extraHeaders || {}) } }),
buildBody: (system, parts, maxTokens) => {
const messages = [];
if (system) messages.push({ role: 'system', content: system });
messages.push({ role: 'user', content: parts });
return { model, max_tokens: maxTokens, temperature: 0, messages, ...(extraBody || {}) };
},
extractText: (r) => r.choices?.[0]?.message?.content || '',
extractUsage: (r) => ({ input: r.usage?.prompt_tokens ?? 0, output: r.usage?.completion_tokens ?? 0 }),
};
}
// Build the ordered vision provider chain: free NIM lanes first, then the
// credits-funded Vertex Gemini anchor, paid OpenAI backstop appended last and
// only when its key is set. Exported for the anchor regression tests
// (tests/api/llm-vertex-anchor-surfaces).
export function visionChain() {
const chain = [];
// One lane per nvapi key: each free key carries its own rate limit.
const nvidiaKeys = [...new Set([env.NVIDIA_API_KEY, ...env.NVIDIA_FALLBACK_KEYS].filter(Boolean))];
nvidiaKeys.forEach((key, i) => {
for (const model of NVIDIA_VISION_MODELS) {
chain.push(openaiCompatVisionProvider({
name: i === 0 ? 'nvidia' : `nvidia#${i + 1}`,
key,
url: 'https://integrate.api.nvidia.com/v1/chat/completions',
model,
}));
}
});
// Free OpenRouter routes, alongside the NIM lanes and ahead of the credits
// anchor: same free tier, same zero meter, and they are the rungs that keep
// vision answering when both NVIDIA and Google are down at once. Ordering
// them here leaves the Vertex anchor exactly where it was in the chain
// relative to the paid tail; nothing is evicted.
// The routes rotate across every OpenRouter key, so one account's daily
// free-request cap never carries the whole vision lane.
const openrouterKeys = [...new Set([env.OPENROUTER_API_KEY, ...env.OPENROUTER_FALLBACK_KEYS].filter(Boolean))];
if (openrouterKeys.length) {
OPENROUTER_VISION_MODELS.forEach((spec, i) => {
chain.push(openaiCompatVisionProvider({
name: 'openrouter',
key: openrouterKeys[i % openrouterKeys.length],
url: 'https://openrouter.ai/api/v1/chat/completions',
model: spec.model,
extraBody: spec.extraBody || null,
extraHeaders: OPENROUTER_ATTRIBUTION_HEADERS,
}));
});
}
// Cloudflare Workers AI: the SAME llama-3.2-11b-vision weights the free NIM
// lane already answers with, served on a different host behind a different
// quota. That is the point of it. Until now one NVIDIA model carried the
// entire chain (the list lost nemotron-nano-12b-v2-vl to a hard 410 on
// 2026-08-26), so a single throttle took vision down platform-wide. This rung
// is a proven model, not a new bet. The vendor is already wired for text in
// llm.js and gated on the same pair of vars, so it costs no new credential.
if (env.CLOUDFLARE_ACCOUNT_ID && env.CLOUDFLARE_AI_API_TOKEN) {
chain.push(openaiCompatVisionProvider({
name: 'cloudflare',
key: env.CLOUDFLARE_AI_API_TOKEN,
url: `https://api.cloudflare.com/client/v4/accounts/${env.CLOUDFLARE_ACCOUNT_ID}/ai/v1/chat/completions`,
model: CLOUDFLARE_VISION_MODEL,
}));
}
// Vertex Gemini credits anchor: multimodal (Gemini Flash reads image_url
// parts, data URIs included, through the same OpenAI-compatible endpoint),
// keyless (OAuth token minted per request), billed to GCP credits. Sits
// after the free NIM lanes and ahead of the paid tail, exactly like the
// text anchor in llm.js's providerChain: the prod OPENAI_API_KEY is
// billing-dead, so this rung is what keeps vision answering when the NIM
// queue throttles or hangs. Nothing may evict it (see api/_lib/vertex-gemini.js).
if (vertexGeminiAvailable()) {
chain.push(openaiCompatVisionProvider({
name: 'vertex-gemini',
url: vertexGeminiChatUrl(),
model: vertexGeminiModel(),
getHeaders: vertexGeminiHeaders,
}));
}
if (env.OPENAI_API_KEY) {
chain.push(openaiCompatVisionProvider({
name: 'openai',
key: env.OPENAI_API_KEY,
url: 'https://api.openai.com/v1/chat/completions',
model: OPENAI_VISION_MODEL,
}));
}
return chain;
}
// True when at least one vision provider can serve a request. Use to gate a
// consumer's vision-derived enhancement WITHOUT making the doomed upstream call —
// this is the fail-open switch (forge validation, alt text, image evidence all
// check it first).
export function visionConfigured() {
return visionChain().length > 0;
}
// Synchronous SSRF guard for a caller-supplied image URL. Requires https (http
// only in dev) and blocks IP-literal hosts in private/loopback/link-local ranges
// plus localhost — the direct SSRF targets reachable through the provider's
// server-side image fetch. DNS-name hosts pass (we can't pin the provider's DNS
// resolution, so name→private rebinding is out of scope here). Throws a 400
// invalid_image_url so callers treat it as bad input, not a vision outage.
function assertSafeImageUrl(rawUrl) {
let url;
try {
url = validatePublicUrl(rawUrl);
} catch (e) {
if (e instanceof SsrfError) {
throw Object.assign(new Error('image URL is not a public https address'), {
status: 400,
code: 'invalid_image_url',
});
}
throw e;
}
const host = url.hostname.replace(/^\[|\]$/g, '');
const fam = isIP(host);
const blocked = fam
? isPrivateAddress(host, fam)
: host === 'localhost' || /\.(local|internal|localdomain)$/i.test(host);
if (blocked) {
throw Object.assign(new Error('image URL resolves to a non-public host'), {
status: 400,
code: 'invalid_image_url',
});
}
return url;
}
// 12 MiB — matches /api/vision's inbound cap; comfortably covers a viewer
// screenshot or photo while bounding the per-request buffer.
const MAX_INLINE_IMAGE_BYTES = 12 * 1024 * 1024;
// Fetch a caller-supplied image URL ourselves and return it as inline base64.
// The provider model servers otherwise fetch imageUrl server-side, and hosts
// with hotlink / User-Agent protection (Wikipedia thumbnails, some CDNs) reject
// that fetch — which fails EVERY URL-based lane and forces a fall-through to the
// paid backstop. Fetching here (SSRF-guarded, redirects re-validated per hop)
// makes the free NIM lanes independent of whether the provider can reach the
// host. Throws on non-2xx, oversize, or transport error so the caller can decide
// whether to fall back to URL pass-through.
async function inlineImageFromUrl(imageUrl, { timeoutMs = 8_000 } = {}) {
const res = await fetchSafePublicUrl(imageUrl, {
signal: AbortSignal.timeout(timeoutMs),
// A browser-like UA + image Accept gets past CDNs that reject empty/bot
// user-agents — the same gate that blocks the providers' own fetchers.
headers: {
'user-agent': 'Mozilla/5.0 (compatible; three.ws-vision/1.0; +https://three.ws)',
accept: 'image/avif,image/webp,image/png,image/*;q=0.8,*/*;q=0.5',
},
});
if (!res.ok) throw Object.assign(new Error(`image fetch ${res.status}`), { status: res.status });
const advertised = Number(res.headers.get('content-length') || 0);
if (advertised > MAX_INLINE_IMAGE_BYTES) {
throw Object.assign(new Error('image exceeds inline size cap'), { status: 413 });
}
const buf = Buffer.from(await res.arrayBuffer());
if (buf.length > MAX_INLINE_IMAGE_BYTES) {
throw Object.assign(new Error('image exceeds inline size cap'), { status: 413 });
}
const ct = (res.headers.get('content-type') || '').split(';')[0].trim().toLowerCase();
return { imageBase64: buf.toString('base64'), mimeType: ct.startsWith('image/') ? ct : 'image/jpeg' };
}
// Normalize a caller's image spec into one OpenAI `image_url` content part.
// Accepts { imageUrl } (pass-through) or { imageBase64, mimeType } (data URI).
function imagePart({ imageUrl, imageBase64, mimeType = 'image/jpeg' }) {
if (imageUrl) return { type: 'image_url', image_url: { url: imageUrl } };
if (imageBase64) {
const raw = imageBase64.startsWith('data:') ? imageBase64 : `data:${mimeType};base64,${imageBase64}`;
return { type: 'image_url', image_url: { url: raw } };
}
throw Object.assign(new Error('describeImage requires imageUrl or imageBase64'), {
status: 400,
code: 'no_image',
});
}
// Map a non-2xx vision response to a normalized error code, mirroring the other
// NIM provider contracts (probes/vision.md error table). Folded into lastErr so
// the final throw after the whole chain fails carries a meaningful code.
function normalizeStatus(status) {
if (status === 401 || status === 403) return 'invalid_key';
if (status === 402) return 'insufficient_credits';
if (status === 429) return 'rate_limited';
if (status >= 500) return 'provider_error';
return 'provider_error';
}
// Describe / analyze one image against a prompt, against the first available
// provider, falling over to the next on transport or non-2xx errors.
//
// { prompt, imageUrl? , imageBase64?, mimeType?, system?, maxTokens?,
// timeoutMs?, track? }
//
// `timeoutMs` bounds EACH provider attempt so a hung free lane can't stall a
// serverless function — the next lane is tried instead. `deadlineMs` bounds the
// WHOLE chain: without it, a handful of lanes each timing out at `timeoutMs`
// sequentially can blow past the function's wall-clock limit, which is exactly
// what produced the "Vercel Runtime Timeout Error: Task timed out after 30s" 504
// on /api/vision (3 free NIM models + a paid backstop × 20s each ≫ 30s). With a
// deadline we stop walking the chain and return a clean 504 before the platform
// hard-kills the invocation. Each attempt is capped at min(timeoutMs, time left).
// `track` is the same optional spend-ledger attribution as llmComplete; a
// successful call records a kind:'vision' usage event with provider/model/tokens/cost
// (free NIM prices to 0 in llm-pricing.js).
//
// `lead` (provider names) moves those rungs to the front for one caller that has
// measured them as its reliable answerers; cooldowns still apply on top, so a
// cooling lead rung drops to the back like any other.
//
// Returns { text, provider, model, usage:{input,output}, raw }.
// Throws VisionUnavailableError when nothing is configured, or the last upstream
// error (with .status = 502/504, .code = normalized) when every provider failed
// or the deadline elapsed.
/** Stable reorder: rungs whose provider name is in `lead` first, in chain order. */
export function leadWith(chain, lead) {
if (!Array.isArray(lead) || !lead.length) return chain;
const names = new Set(lead);
return [...chain.filter((p) => names.has(p.name)), ...chain.filter((p) => !names.has(p.name))];
}
export async function describeImage({
prompt,
imageUrl = null,
imageBase64 = null,
mimeType = 'image/jpeg',
system = null,
maxTokens = 512,
timeoutMs = 20_000,
deadlineMs = null,
track = null,
accept = null,
lead = null,
}) {
const chain = leadWith(visionChain(), lead);
if (!chain.length) throw new VisionUnavailableError();
const deadlineAt = deadlineMs != null ? Date.now() + deadlineMs : Infinity;
// SSRF guard: the provider's model server fetches `imageUrl` server-side, so a
// caller-supplied URL could otherwise reach internal targets (169.254.169.254,
// localhost, RFC1918) through the provider. We can't DNS-pin the provider's
// fetch, so apply a synchronous string-level guard — require https and reject
// private/loopback/link-local IP literals + localhost — before the URL leaves
// this process. Centralized here so every consumer of describeImage is covered,
// not just the forge image-validate path that already pre-validates.
if (imageUrl) assertSafeImageUrl(imageUrl);
// Prefer inlining the image ourselves over handing the URL to each provider's
// server-side fetcher. If our own fetch fails (a host the provider CAN reach
// but we can't, or a transient error), fall back to URL pass-through so we
// never regress a currently-working path. Bounded by the remaining chain
// deadline so a slow image host can't blow the whole budget before a lane runs.
if (imageUrl && !imageBase64) {
try {
const budget = inlineImageBudget(deadlineAt - Date.now(), timeoutMs);
const inlined = await inlineImageFromUrl(imageUrl, { timeoutMs: budget });
imageBase64 = inlined.imageBase64;
mimeType = inlined.mimeType;
imageUrl = null;
} catch {
// keep imageUrl set — the providers will try their own server-side fetch
}
}
const parts = [
{ type: 'text', text: prompt },
imagePart({ imageUrl, imageBase64, mimeType }),
];
// Put the lanes a recent request found throttled or key-dead at the BACK
// rather than dropping them: a chain that skips every cooling lane and finds
// nothing left must still answer, so the cooled ones remain as a last resort.
// One cache round-trip, and it is skipped entirely for a single-lane chain.
let order = chain;
if (chain.length > 1) {
const cooling = await providersInCooldown(chain.map(laneKey));
if (cooling.size) {
const hot = chain.filter((p) => !cooling.has(laneKey(p)));
const cold = chain.filter((p) => cooling.has(laneKey(p)));
if (hot.length) order = [...hot, ...cold];
}
}
let lastErr;
// Every rung's verdict, in the order they were tried. Only the LAST failure
// used to survive, which made a dead chain unreadable: the caller was told
// "openai vision 429: billing_not_active" while the actual problem was that
// the free NIM lanes and the Vertex anchor ahead of it had already failed for
// two entirely different reasons. Whoever has to fix it needs all three, so
// the whole walk is carried on the thrown error and rendered by the handler.
/** @type {{ provider: string, model: string|null, status: number|null, detail: string }[]} */
const attempts = [];
// Hosts this request has already proven sick (a throttle, a dead key, an
// unreachable socket). Skipping their remaining rungs inside THIS request is
// the difference between paying one doomed attempt and paying one per model
// the host happens to serve, and the deadline it saves is what lets the next
// healthy lane answer at all.
const sickHosts = new Set();
for (let i = 0; i < order.length; i++) {
const p = order[i];
if (sickHosts.has(laneHost(p))) continue;
// Stop walking the chain once the overall budget is spent — returning a clean
// 504 here beats letting the platform hard-kill the function mid-request.
const remaining = deadlineAt - Date.now();
if (remaining <= 0) {
lastErr = Object.assign(new Error('vision deadline exceeded before a provider answered'), {
status: 504,
code: 'deadline_exceeded',
});
break;
}
const attemptTimeout = laneAttemptTimeout(remaining, order.length - i, timeoutMs);
const startedAt = Date.now();
let upstream;
try {
// Keyless lanes (the Vertex Gemini credits anchor) mint their auth per
// attempt via getHeaders; a token-exchange failure lands in the catch
// below and fails over to the next lane like any transport error.
upstream = await fetch(p.url, {
method: 'POST',
headers: p.getHeaders ? await p.getHeaders() : p.headers,
body: JSON.stringify(p.buildBody(system, parts, maxTokens)),
signal: AbortSignal.timeout(attemptTimeout),
});
} catch (e) {
// Unreachable or timed out. Cool this lane only: a hang is as often one
// heavy model refusing to answer inside its slice as it is a dead host,
// and benching the sibling rung on that guess would discard the very
// redundancy the second rung exists to provide.
coolLane(order, p, { seconds: VISION_LANE_COOLDOWN_SECONDS, reason: 'health', hostWide: false });
attempts.push({ provider: p.name, model: p.model || null, status: null, detail: `unreachable: ${String(e.message).slice(0, 160)}` });
lastErr = Object.assign(new Error(`${p.name} vision unreachable: ${e.message}`), { status: 502, code: 'provider_unreachable' });
continue;
}
if (!upstream.ok) {
const body = await upstream.text().catch(() => '');
// 401/403/402 is a key or billing fault that will not clear on its own, so
// it parks the lane for the long window instead of being re-probed every
// request. 429 and 5xx are the host throttling or failing, which cools
// every rung sharing it. Anything else (a 404 for a retired model id, a
// 400) is specific to this model and cools this lane alone.
const st = upstream.status;
// 410 Gone is how NIM reports a retired model id. Unlike a 404 or a 500 it
// is a permanent verdict about this model, so it parks the lane for the
// long window instead of being re-probed (and re-charged a slice of the
// deadline) on every request until someone notices.
const authFault = st === 401 || st === 403 || st === 402 || st === 410;
// A 429 is the one verdict that is unambiguously about the HOST and the
// account behind it: every model served there is throttled by the same
// quota, so both the bench and the in-request skip cover every sibling
// rung. Everything else stays lane-scoped, because NIM answers a
// per-model fault with a 403 (model not enabled for this account) or a
// 500 just as readily as a host-level one, and benching a working twin
// on that guess would cost the chain its redundancy.
coolLane(order, p, {
seconds: authFault ? AUTH_COOLDOWN_SECONDS : VISION_LANE_COOLDOWN_SECONDS,
reason: authFault ? 'auth' : 'health',
hostWide: st === 429,
});
if (st === 429) sickHosts.add(laneHost(p));
attempts.push({ provider: p.name, model: p.model || null, status: st, detail: body.slice(0, 160) });
lastErr = Object.assign(
new Error(`${p.name} vision ${upstream.status}: ${body.slice(0, 200)}`),
{ status: 502, code: normalizeStatus(upstream.status) },
);
continue;
}
const data = await upstream.json();
const usage = p.extractUsage(data);
// Meter it either way: the tokens were spent whether or not the reply is
// usable, and a ledger that hides rejected replies understates the cost of
// a lane that answers fluently in the wrong shape.
recordVisionSpend(p, usage, Date.now() - startedAt, track);
const text = (p.extractText(data) || '').trim();
// A 200 is not the same as an answer. When the caller requires a shape
// (describeImageJson requires parseable JSON), validating it HERE makes the
// shape part of the chain's success test instead of a filter bolted on
// after a winner is picked. That distinction is the whole bug: with the
// check outside the loop, the first lane to emit prose ended the request
// and every healthy rung behind it went untried, which is exactly how a
// reasoning model that narrates before it answers took the forge quality
// gate from intermittent to 0 verdicts in 10 (production, 2026-09-10).
//
// A rejected lane is NOT cooled. The cooldown key is shared with callers
// that want free-form prose, and that reply is perfectly good to them;
// benching it here would degrade alt-text to punish a JSON caller.
let accepted;
if (accept) {
try {
accepted = accept(text);
} catch (e) {
attempts.push({
provider: p.name,
model: p.model || null,
status: upstream.status,
detail: `reply rejected: ${String(e?.message || e).slice(0, 120)}`,
});
lastErr = Object.assign(
new Error(`${p.name} vision reply rejected: ${e?.message || e}`),
{ status: 502, code: e?.code || 'vision_bad_reply' },
);
continue;
}
}
// A lane that just served a real request is healthy whatever an earlier
// window recorded; waiting out the rest of a disproved cooldown only keeps
// a recovered lane off the menu.
void clearProviderCooldown(laneKey(p));
return {
text,
provider: p.name,
model: p.model,
usage,
raw: data,
accepted,
};
}
throw Object.assign(lastErr || new VisionUnavailableError(), { lanes: attempts });
}
// Convenience: describeImage with "parseable JSON" as the acceptance test. VLMs
// mostly honor "reply ONLY JSON" (probes/vision.md) but may wrap it in a ```json
// fence or a trailing newline, and a reasoning model may narrate around it;
// parseJsonLoose strips what it can. A reply it cannot parse fails THAT LANE and
// the chain moves to the next rung, so one chatty model no longer costs the
// request every healthy provider behind it. Throws only when no rung produced
// parseable JSON (the caller's degraded path handles that like a vision outage).
export async function describeImageJson(opts) {
const result = await describeImage({ ...opts, accept: parseJsonLoose });
return { ...result, json: result.accepted };
}
// Strip a ```json fence / stray prose and parse the first JSON object/array in
// the text. Throws a normalized error on failure so callers treat it as a
// degraded vision result.
export function parseJsonLoose(text) {
const trimmed = String(text || '').trim();
const fenced = trimmed.replace(/^```(?:json)?\s*/i, '').replace(/\s*```$/, '');
const start = fenced.search(/[[{]/);
const candidate = start >= 0 ? fenced.slice(start) : fenced;
try {
return JSON.parse(candidate);
} catch {
// Last resort: grab the outermost {...} or [...] span.
const m = candidate.match(/[{[][\s\S]*[}\]]/);
if (m) {
try {
return JSON.parse(m[0]);
} catch {
/* fall through */
}
}
throw Object.assign(new Error('vision reply was not valid JSON'), { status: 502, code: 'vision_bad_json' });
}
}
// Fire-and-forget spend ledger write for one vision call. Free NIM prices to 0;
// the paid OpenAI backstop prices via llm-pricing.js. Never throws.
function recordVisionSpend(provider, usage, latencyMs, track) {
const input = usage?.input ?? 0;
const output = usage?.output ?? 0;
recordEvent({
kind: 'vision',
provider: provider.name,
model: provider.model,
inputTokens: input,
outputTokens: output,
costMicroUsd: costMicroUsd({ provider: provider.name, model: provider.model, input, output }),
latencyMs,
userId: track?.userId ?? null,
agentId: track?.agentId ?? null,
avatarId: track?.avatarId ?? null,
clientId: track?.clientId ?? null,
apiKeyId: track?.apiKeyId ?? null,
tool: track?.tool ?? null,
meta: track?.meta ?? undefined,
});
}