7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.
TypeScript
Issue batch: local-Ollama cooldown fix, keyless-config boot crash, served-model drift guard, failover-ladder UI, credits (#603)
Tashfeencd4d839ae88f
diff --git a/README.md b/README.mdindex e3a479f1..5e758d0a 100644--- a/README.md+++ b/README.md@@ -482,6 +482,42 @@ curl http://localhost:3001/v1/chat/completions \}'```+**Routing strategies (`auto:*`)**++Plain `auto` follows your active fallback chain. Add a suffix to steer a single request instead — no dashboard changes needed:++- `auto:smart` — favor the highest-intelligence models+- `auto:fast` — favor measured speed (throughput and time-to-first-byte)+- `auto:cheap` — budget-leaning; currently the same blend as `balanced` (everything in the pool is already free)+- `auto:reliable` — favor recent success rate+- `auto:balanced` — the default blend (reliability first, speed and intelligence split the rest)++These rank **every enabled model**, ignoring your chain order. Common synonyms resolve too (`auto:fastest`, `auto:speed`, `auto:smartest`, `auto:cheapest`, `auto:budget`, …), and the whole model string is case-insensitive.++```bash+curl http://localhost:3001/v1/chat/completions \+ -H "Authorization: Bearer freellmapi-your-unified-key" \+ -H "Content-Type: application/json" \+ -d '{+ "model": "auto:fast",+ "messages": [{"role": "user", "content": "hi"}]+ }'+```++`auto:<profile-name>` routes through a named profile's chain instead of the active one, so different tools can use different chains through the same key:++```bash+curl http://localhost:3001/v1/chat/completions \+ -H "Authorization: Bearer freellmapi-your-unified-key" \+ -H "Content-Type: application/json" \+ -d '{+ "model": "auto:coding",+ "messages": [{"role": "user", "content": "Write a binary search in Rust."}]+ }'+```++An unknown profile name returns a clear `400` rather than silently falling back. Profiles are the named fallback chains from **Model profiles** (see [Features](#features)) — create and switch them from the dashboard; whichever is active is what plain `auto` uses.+**Streaming**```python@@ -867,6 +903,11 @@ See [CONTRIBUTING.md](./CONTRIBUTING.md) for the full migration CLI and workflow<a href="https://github.com/noobix"><img src="https://images.weserv.nl/?url=github.com/noobix.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@noobix" /></a><a href="https://github.com/nandukmelath"><img src="https://images.weserv.nl/?url=github.com/nandukmelath.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@nandukmelath" /></a><a href="https://github.com/coffcoe"><img src="https://images.weserv.nl/?url=github.com/coffcoe.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@coffcoe" /></a>+<a href="https://github.com/NirvanaCh7"><img src="https://images.weserv.nl/?url=github.com/NirvanaCh7.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@NirvanaCh7" /></a>+<a href="https://github.com/Mohamed3nan"><img src="https://images.weserv.nl/?url=github.com/Mohamed3nan.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@Mohamed3nan" /></a>+<a href="https://github.com/Arman-Espiar"><img src="https://images.weserv.nl/?url=github.com/Arman-Espiar.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@Arman-Espiar" /></a>+<a href="https://github.com/MetaMysteries8"><img src="https://images.weserv.nl/?url=github.com/MetaMysteries8.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@MetaMysteries8" /></a>+<a href="https://github.com/lujun880726"><img src="https://images.weserv.nl/?url=github.com/lujun880726.png&w=60&h=60&fit=cover&mask=circle" width="60" alt="@lujun880726" /></a>## Terms of Service reviewdiff --git a/client/src/i18n/locales/en.json b/client/src/i18n/locales/en.jsonindex 5076013a..39726d03 100644--- a/client/src/i18n/locales/en.json+++ b/client/src/i18n/locales/en.json@@ -409,7 +409,20 @@"recentCalls": "Recent calls","clientIp": "Client IP","clientAgent": "Client app",- "requestedModelHint": "Requested {model}, served by fallback"+ "requestedModelHint": "Requested {model}, served by fallback",+ "attempts": "Attempts",+ "filterAll": "All",+ "providerBreakdown": "Per-provider breakdown",+ "tokensPerSec": "Tok/s",+ "allProviders": "All providers",+ "requestDetailTitle": "Request #{id}",+ "failoverLadder": "Failover ladder",+ "failoverLadderHint": "Each hop this request took through the fallback chain, in order.",+ "noAttemptTrace": "No attempt trace was recorded for this request.",+ "keyOrdinal": "key #{n}",+ "attemptTimingHint": "Start offset from request start · attempt duration",+ "requestedModel": "Requested model",+ "ttft": "TTFT"},"playground": {"title": "Playground",diff --git a/client/src/i18n/locales/es.json b/client/src/i18n/locales/es.jsonindex 6839c7bf..0eee2e01 100644--- a/client/src/i18n/locales/es.json+++ b/client/src/i18n/locales/es.json@@ -392,7 +392,20 @@"recentCalls": "Llamadas recientes","clientIp": "IP del cliente","clientAgent": "Aplicación cliente",- "requestedModelHint": "Se solicitó {model}, servido por respaldo"+ "requestedModelHint": "Se solicitó {model}, servido por respaldo",+ "attempts": "Intentos",+ "filterAll": "Todas",+ "providerBreakdown": "Desglose por proveedor",+ "tokensPerSec": "Tok/s",+ "allProviders": "Todos los proveedores",+ "requestDetailTitle": "Solicitud #{id}",+ "failoverLadder": "Escalera de failover",+ "failoverLadderHint": "Cada salto que esta solicitud dio por la cadena de respaldo, en orden.",+ "noAttemptTrace": "No se registró ninguna traza de intentos para esta solicitud.",+ "keyOrdinal": "clave #{n}",+ "attemptTimingHint": "Desfase desde el inicio de la solicitud · duración del intento",+ "requestedModel": "Modelo solicitado",+ "ttft": "TTFT"},"playground": {"title": "Entorno de pruebas",diff --git a/client/src/i18n/locales/fr.json b/client/src/i18n/locales/fr.jsonindex 49aa0fcf..aa51f18b 100644--- a/client/src/i18n/locales/fr.json+++ b/client/src/i18n/locales/fr.json@@ -392,7 +392,20 @@"recentCalls": "Appels récents","clientIp": "IP du client","clientAgent": "Application cliente",- "requestedModelHint": "{model} demandé, servi par repli"+ "requestedModelHint": "{model} demandé, servi par repli",+ "attempts": "Tentatives",+ "filterAll": "Toutes",+ "providerBreakdown": "Détail par fournisseur",+ "tokensPerSec": "Tok/s",+ "allProviders": "Tous les fournisseurs",+ "requestDetailTitle": "Requête #{id}",+ "failoverLadder": "Échelle de failover",+ "failoverLadderHint": "Chaque étape parcourue par cette requête dans la chaîne de repli, dans l'ordre.",+ "noAttemptTrace": "Aucune trace de tentative n'a été enregistrée pour cette requête.",+ "keyOrdinal": "clé #{n}",+ "attemptTimingHint": "Décalage depuis le début de la requête · durée de la tentative",+ "requestedModel": "Modèle demandé",+ "ttft": "TTFT"},"playground": {"title": "Bac à sable",diff --git a/client/src/i18n/locales/it.json b/client/src/i18n/locales/it.jsonindex 98863ea7..71b52bb2 100644--- a/client/src/i18n/locales/it.json+++ b/client/src/i18n/locales/it.json@@ -392,7 +392,20 @@"recentCalls": "Chiamate recenti","clientIp": "IP del client","clientAgent": "App client",- "requestedModelHint": "Richiesto {model}, servito dal fallback"+ "requestedModelHint": "Richiesto {model}, servito dal fallback",+ "attempts": "Tentativi",+ "filterAll": "Tutte",+ "providerBreakdown": "Dettaglio per provider",+ "tokensPerSec": "Tok/s",+ "allProviders": "Tutti i provider",+ "requestDetailTitle": "Richiesta #{id}",+ "failoverLadder": "Scala di failover",+ "failoverLadderHint": "Ogni passaggio di questa richiesta lungo la catena di fallback, in ordine.",+ "noAttemptTrace": "Nessuna traccia dei tentativi registrata per questa richiesta.",+ "keyOrdinal": "chiave #{n}",+ "attemptTimingHint": "Offset dall'inizio della richiesta · durata del tentativo",+ "requestedModel": "Modello richiesto",+ "ttft": "TTFT"},"playground": {"title": "Playground",diff --git a/client/src/i18n/locales/pt-BR.json b/client/src/i18n/locales/pt-BR.jsonindex 8d39142a..840c920e 100644--- a/client/src/i18n/locales/pt-BR.json+++ b/client/src/i18n/locales/pt-BR.json@@ -392,7 +392,20 @@"recentCalls": "Chamadas recentes","clientIp": "IP do cliente","clientAgent": "Aplicativo cliente",- "requestedModelHint": "Solicitado {model}, servido por fallback"+ "requestedModelHint": "Solicitado {model}, servido por fallback",+ "attempts": "Tentativas",+ "filterAll": "Todas",+ "providerBreakdown": "Detalhamento por provedor",+ "tokensPerSec": "Tok/s",+ "allProviders": "Todos os provedores",+ "requestDetailTitle": "Requisição #{id}",+ "failoverLadder": "Escada de failover",+ "failoverLadderHint": "Cada salto que esta requisição deu pela cadeia de fallback, em ordem.",+ "noAttemptTrace": "Nenhum rastro de tentativas foi registrado para esta requisição.",+ "keyOrdinal": "chave #{n}",+ "attemptTimingHint": "Deslocamento desde o início da requisição · duração da tentativa",+ "requestedModel": "Modelo solicitado",+ "ttft": "TTFT"},"playground": {"title": "Playground",diff --git a/client/src/i18n/locales/zh-CN.json b/client/src/i18n/locales/zh-CN.jsonindex 7bae3ea8..7861c76b 100644--- a/client/src/i18n/locales/zh-CN.json+++ b/client/src/i18n/locales/zh-CN.json@@ -392,7 +392,20 @@"recentCalls": "最近调用","clientIp": "客户端 IP","clientAgent": "客户端应用",- "requestedModelHint": "请求了 {model},由回退模型提供"+ "requestedModelHint": "请求了 {model},由回退模型提供",+ "attempts": "尝试次数",+ "filterAll": "全部",+ "providerBreakdown": "按提供商明细",+ "tokensPerSec": "Tok/s",+ "allProviders": "全部提供商",+ "requestDetailTitle": "请求 #{id}",+ "failoverLadder": "故障转移阶梯",+ "failoverLadderHint": "该请求沿回退链经过的每一跳,按顺序排列。",+ "noAttemptTrace": "此请求未记录尝试轨迹。",+ "keyOrdinal": "密钥 #{n}",+ "attemptTimingHint": "自请求开始的偏移 · 尝试耗时",+ "requestedModel": "请求的模型",+ "ttft": "TTFT"},"playground": {"title": "试玩台",diff --git a/client/src/pages/AnalyticsPage.tsx b/client/src/pages/AnalyticsPage.tsxindex eda91b84..64cb5661 100644--- a/client/src/pages/AnalyticsPage.tsx+++ b/client/src/pages/AnalyticsPage.tsx@@ -4,13 +4,18 @@ import {BarChart, Bar, XAxis, YAxis, CartesianGrid, Tooltip, ResponsiveContainer,LineChart, Line, Legend,} from 'recharts'+import { X } from 'lucide-react'import { apiFetch } from '@/lib/api'import { SegmentedControl } from '@/components/ui/segmented-control'import { Table, TableBody, TableCell, TableHead, TableHeader, TableRow } from '@/components/ui/table'+import { Badge } from '@/components/ui/badge'+import { Dialog, DialogClose, DialogPopup, DialogTitle } from '@/components/ui/dialog'+import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from '@/components/ui/select'import { PageHeader } from '@/components/page-header'import { Skeleton } from '@/components/ui/skeleton'import { Tooltip as HoverTooltip } from '@/components/tooltip'import { formatSqliteUtcToLocalTime } from '@/lib/utils'+import { platformColors } from '@/lib/routing'import { useI18n } from '@/i18n'type TimeRange = '24h' | '7d' | '30d' | '90d'@@ -109,6 +114,9 @@ interface RecentCallRow {clientIp: string | nullclientUserAgent: string | nullcreatedAt: string+ // Failover-ladder length: attempts hang off the TERMINAL row of a proxied+ // request, so mid-ladder failure rows report 0.+ attemptCount: number}interface RecentCallsResponse {@@ -116,6 +124,25 @@ interface RecentCallsResponse {rows: RecentCallRow[]}+// One hop of the failover ladder, from GET /api/analytics/requests/:id.+interface RequestAttempt {+ ordinal: number+ platform: string+ modelId: string+ keyOrdinal: number+ outcome: string+ startOffsetMs: number+ durationMs: number+ errorSummary: string | null+}++interface RequestDetail extends Omit<RecentCallRow, 'attemptCount'> {+ ttfbMs: number | null+ attempts: RequestAttempt[]+}++type StatusFilter = 'all' | 'success' | 'error'+// First product token of the UA ("python-requests/2.32.3", "curl/8.6.0", …)// is enough to tell callers apart in a narrow cell; full string on hover.function shortUserAgent(ua: string | null): string {@@ -143,17 +170,163 @@ function Stat({ label, value, hint, className }: { label: string; value: stringreturn hint ? <HoverTooltip text={hint} side="bottom" className="block">{card}</HoverTooltip> : card}-function Panel({ title, children }: { title: string; children: React.ReactNode }) {+function Panel({ title, actions, children }: { title: string; actions?: React.ReactNode; children: React.ReactNode }) {return (<div className="rounded-3xl border bg-card">- <div className="px-4 py-3 border-b">+ <div className="px-4 py-3 border-b flex flex-wrap items-center justify-between gap-2"><h3 className="text-sm font-medium">{title}</h3>+ {actions}</div><div className="p-4">{children}</div></div>)}+// Platform swatch shared by the ladder and the per-provider table; same color+// source as the token-usage legend (lib/routing.ts), same gray fallback.+function PlatformDot({ platform }: { platform: string }) {+ return (+ <span+ className="size-2 rounded-full flex-shrink-0"+ style={{ backgroundColor: platformColors[platform] ?? '#94a3b8' }}+ />+ )+}++// Compact ms rendering for ladder timings: sub-second stays in ms, longer+// spans read as seconds ("38.8 s") like the issue reports do.+function formatMs(ms: number): string {+ if (ms >= 1000) return `${(ms / 1000).toFixed(1)} s`+ return `${ms} ms`+}++// Key/value line of the request-detail summary grid.+function DetailField({ label, value, mono }: { label: string; value: React.ReactNode; mono?: boolean }) {+ return (+ <div className="min-w-0">+ <p className="text-[11px] text-muted-foreground uppercase tracking-wider">{label}</p>+ <p className={`text-sm mt-0.5 break-words ${mono ? 'tabular-nums' : ''}`}>{value}</p>+ </div>+ )+}++// Per-request drill-down: the parent row's fields plus the failover ladder —+// one entry per dispatched attempt (ordinal → provider/model → key ordinal →+// outcome → timing, with the redacted per-hop error when one was recorded).+// A dialog (the app's detail-popup idiom, cf. keys/export-keys-dialog) rather+// than a routed page so the list's range/filter context stays put behind it.+function RequestDetailDialog({ requestId, onClose }: { requestId: number | null; onClose: () => void }) {+ const { t } = useI18n()++ const { data: detail, isLoading } = useQuery({+ queryKey: ['analytics', 'request-detail', requestId],+ queryFn: () => apiFetch<RequestDetail>(`/api/analytics/requests/${requestId}`),+ enabled: requestId != null,+ })++ return (+ <Dialog open={requestId != null} onOpenChange={(open) => { if (!open) onClose() }}>+ <DialogPopup maxWidth="max-w-2xl">+ <div className="mb-4 flex items-center justify-between gap-4">+ <DialogTitle>{t('analytics.requestDetailTitle', { id: requestId ?? '' })}</DialogTitle>+ <DialogClose+ aria-label={t('common.dismiss')}+ className="-mr-1 rounded-lg p-1 text-muted-foreground/70 transition-colors outline-none hover:text-foreground focus-visible:ring-3 focus-visible:ring-ring/50"+ >+ <X className="size-4" />+ </DialogClose>+ </div>++ {isLoading || !detail ? (+ <div className="space-y-3">+ <Skeleton className="h-24 rounded-xl" />+ <Skeleton className="h-32 rounded-xl" />+ </div>+ ) : (+ <div className="space-y-5">+ <div className="grid grid-cols-2 sm:grid-cols-3 gap-x-4 gap-y-3">+ <DetailField+ label={t('analytics.time')}+ value={formatSqliteUtcToLocalTime(detail.createdAt, { month: 'short', day: 'numeric', hour: '2-digit', minute: '2-digit', second: '2-digit' })}+ mono+ />+ <DetailField+ label={t('common.status')}+ value={+ <span className={detail.status === 'success' ? '' : 'text-destructive'}>{detail.status}</span>+ }+ />+ <DetailField+ label={t('common.provider')}+ value={+ <span className="inline-flex items-center gap-1.5">+ <PlatformDot platform={detail.platform} />+ {detail.platform}+ </span>+ }+ />+ <DetailField label={t('common.model')} value={detail.modelId} />+ {detail.requestedModel && detail.requestedModel !== detail.modelId && (+ <DetailField label={t('analytics.requestedModel')} value={detail.requestedModel} />+ )}+ <DetailField+ label={`${t('analytics.inTokens')} / ${t('analytics.outTokens')}`}+ value={`${formatTokens(detail.inputTokens)} / ${formatTokens(detail.outputTokens)}`}+ mono+ />+ <DetailField label={t('analytics.latency')} value={formatMs(detail.latencyMs ?? 0)} mono />+ <DetailField label={t('analytics.ttft')} value={detail.ttfbMs != null ? formatMs(detail.ttfbMs) : '—'} mono />+ <DetailField label={t('analytics.clientIp')} value={detail.clientIp ?? '—'} mono />+ <DetailField label={t('analytics.clientAgent')} value={detail.clientUserAgent ?? '—'} />+ </div>++ {detail.error && (+ <div className="rounded-xl border border-destructive/30 bg-destructive/5 px-3 py-2">+ <p className="text-[11px] text-destructive uppercase tracking-wider">{t('analytics.message')}</p>+ <p className="text-xs text-destructive/90 mt-1 break-words">{detail.error}</p>+ </div>+ )}++ <div>+ <h4 className="text-sm font-medium">{t('analytics.failoverLadder')}</h4>+ <p className="text-xs text-muted-foreground mt-0.5">{t('analytics.failoverLadderHint')}</p>+ {detail.attempts.length === 0 ? (+ <p className="text-sm text-muted-foreground text-center py-6">{t('analytics.noAttemptTrace')}</p>+ ) : (+ <ol className="mt-3 space-y-2">+ {detail.attempts.map((a) => (+ <li key={a.ordinal} className="rounded-xl border px-3 py-2">+ <div className="flex items-center gap-2 text-xs">+ <span className="w-6 text-muted-foreground tabular-nums">#{a.ordinal + 1}</span>+ <PlatformDot platform={a.platform} />+ <span className="font-medium">{a.platform}</span>+ <span className="text-muted-foreground truncate" title={a.modelId}>{a.modelId}</span>+ <Badge variant="outline">{t('analytics.keyOrdinal', { n: a.keyOrdinal })}</Badge>+ <Badge variant={a.outcome === 'ok' || a.outcome === 'committed' ? 'secondary' : 'destructive'}>+ {a.outcome}+ </Badge>+ <span+ className="ml-auto whitespace-nowrap text-muted-foreground tabular-nums"+ title={t('analytics.attemptTimingHint')}+ >+ +{formatMs(a.startOffsetMs)} · {formatMs(a.durationMs)}+ </span>+ </div>+ {a.errorSummary && (+ <p className="mt-1 pl-8 text-xs text-destructive/90 break-words">{a.errorSummary}</p>+ )}+ </li>+ ))}+ </ol>+ )}+ </div>+ </div>+ )}+ </DialogPopup>+ </Dialog>+ )+}+const axisStyle = { fontSize: 11, fill: 'var(--muted-foreground)' } as constconst gridStyle = 'var(--border)'const primaryFill = 'var(--foreground)'@@ -214,9 +387,21 @@ export default function AnalyticsPage() {queryFn: () => apiFetch<ErrorDistribution>(`/api/analytics/error-distribution?range=${range}`),})+ // Recent-calls list filters (status/platform) + the row opened in the+ // drill-down dialog. Filters ride the query key so react-query refetches+ // (and caches) each combination on its own.+ const [statusFilter, setStatusFilter] = useState<StatusFilter>('all')+ const [platformFilter, setPlatformFilter] = useState<string>('all')+ const [detailId, setDetailId] = useState<number | null>(null)+const { data: recentCalls } = useQuery({- queryKey: ['analytics', 'requests', range],- queryFn: () => apiFetch<RecentCallsResponse>(`/api/analytics/requests?range=${range}&limit=100`),+ queryKey: ['analytics', 'requests', range, statusFilter, platformFilter],+ queryFn: () => {+ const params = new URLSearchParams({ range, limit: '100' })+ if (statusFilter !== 'all') params.set('status', statusFilter)+ if (platformFilter !== 'all') params.set('platform', platformFilter)+ return apiFetch<RecentCallsResponse>(`/api/analytics/requests?${params}`)+ },})// Savings card shows ONE stable monthly figure regardless of the selected@@ -479,9 +664,45 @@ export default function AnalyticsPage() {{/* Recent calls: one line per proxied request with the caller's IP +user agent. All local clients share the unified key, so this is- the only view that answers "who is hitting the router". */}+ the only view that answers "who is hitting the router". Rows open+ the failover-ladder drill-down; the header hosts status/provider+ filters (server-side, so total reflects the filtered set). */}<div className="lg:col-span-2">- <Panel title={t('analytics.recentCalls')}>+ <Panel+ title={t('analytics.recentCalls')}+ actions={+ <div className="flex flex-wrap items-center gap-2">+ <SegmentedControl+ value={statusFilter}+ onValueChange={setStatusFilter}+ options={[+ { value: 'all', label: t('analytics.filterAll') },+ { value: 'success', label: t('common.success') },+ { value: 'error', label: t('analytics.errors') },+ ]}+ ariaLabel={t('common.status')}+ />+ <Select value={platformFilter} onValueChange={(v) => setPlatformFilter(v ?? 'all')}>+ <SelectTrigger size="sm" aria-label={t('common.provider')}>+ <SelectValue>+ {(v: string) => (!v || v === 'all' ? t('analytics.allProviders') : v)}+ </SelectValue>+ </SelectTrigger>+ <SelectContent>+ <SelectItem value="all">{t('analytics.allProviders')}</SelectItem>+ {byPlatform.map((p) => (+ <SelectItem key={p.platform} value={p.platform}>+ <span className="flex items-center gap-2">+ <PlatformDot platform={p.platform} />+ <span>{p.platform}</span>+ </span>+ </SelectItem>+ ))}+ </SelectContent>+ </Select>+ </div>+ }+ >{!recentCalls?.rows?.length ? (<p className="text-sm text-muted-foreground text-center py-8">{t('common.noData')}</p>) : (@@ -495,6 +716,7 @@ export default function AnalyticsPage() {<TableHead>{t('common.model')}</TableHead><TableHead>{t('common.provider')}</TableHead><TableHead>{t('common.status')}</TableHead>+ <TableHead className="text-right">{t('analytics.attempts')}</TableHead><TableHead className="text-right">{t('analytics.inTokens')}</TableHead><TableHead className="text-right">{t('analytics.outTokens')}</TableHead><TableHead className="text-right pr-4">{t('analytics.latency')}</TableHead>@@ -502,7 +724,11 @@ export default function AnalyticsPage() {</TableHeader><TableBody>{recentCalls.rows.map((r) => (- <TableRow key={r.id}>+ <TableRow+ key={r.id}+ onClick={() => setDetailId(r.id)}+ className="cursor-pointer"+ ><TableCell className="pl-4 text-xs text-muted-foreground tabular-nums whitespace-nowrap">{formatSqliteUtcToLocalTime(r.createdAt, { month: 'short', day: 'numeric', hour: '2-digit', minute: '2-digit', second: '2-digit' })}</TableCell>@@ -518,6 +744,11 @@ export default function AnalyticsPage() {<TableCell className={`text-xs ${r.status === 'success' ? 'text-muted-foreground' : 'text-destructive'}`} title={r.error ?? undefined}>{r.status}</TableCell>+ {/* >1 = the request burned failover hops; that is the+ row worth drilling into, so give it weight. */}+ <TableCell className={`text-right text-xs tabular-nums ${r.attemptCount > 1 ? 'font-medium' : 'text-muted-foreground'}`}>+ {r.attemptCount > 0 ? r.attemptCount : '—'}+ </TableCell><TableCell className="text-right text-xs tabular-nums">{formatTokens(r.inputTokens)}</TableCell><TableCell className="text-right text-xs tabular-nums">{formatTokens(r.outputTokens)}</TableCell><TableCell className="text-right text-xs tabular-nums pr-4">{r.latencyMs} ms</TableCell>@@ -530,6 +761,59 @@ export default function AnalyticsPage() {</Panel></div>+ {/* Per-provider breakdown: the tabular face of the by-platform data —+ the charts above show volume/latency, this row surfaces the+ success-rate and error-count numbers (#335). */}+ <div className="lg:col-span-2">+ <Panel title={t('analytics.providerBreakdown')}>+ {byPlatform.length === 0 ? (+ <p className="text-sm text-muted-foreground text-center py-8">{t('common.noData')}</p>+ ) : (+ <div className="max-h-[360px] overflow-y-auto -mx-4">+ <Table>+ <TableHeader>+ <TableRow>+ <TableHead className="pl-4">{t('common.provider')}</TableHead>+ <TableHead className="text-right">{t('analytics.requests')}</TableHead>+ <TableHead className="text-right">{t('common.success')}</TableHead>+ <TableHead className="text-right">{t('analytics.errors')}</TableHead>+ <TableHead className="text-right">{t('analytics.avgLatency')}</TableHead>+ <TableHead className="text-right">{t('analytics.p95Latency')}</TableHead>+ <TableHead className="text-right">{t('analytics.avgTtft')}</TableHead>+ <TableHead className="text-right">{t('analytics.tokensPerSec')}</TableHead>+ <TableHead className="text-right">{t('analytics.inTokens')}</TableHead>+ <TableHead className="text-right pr-4">{t('analytics.outTokens')}</TableHead>+ </TableRow>+ </TableHeader>+ <TableBody>+ {byPlatform.map((p) => (+ <TableRow key={p.platform}>+ <TableCell className="pl-4 text-sm font-medium">+ <span className="flex items-center gap-2">+ <PlatformDot platform={p.platform} />+ {p.platform}+ </span>+ </TableCell>+ <TableCell className="text-right tabular-nums">{p.requests}</TableCell>+ <TableCell className="text-right tabular-nums">{p.successRate}%</TableCell>+ <TableCell className={`text-right tabular-nums ${p.errorCount > 0 ? 'text-destructive' : ''}`}>+ {p.errorCount > 0 ? p.errorCount : '—'}+ </TableCell>+ <TableCell className="text-right tabular-nums">{p.avgLatencyMs} ms</TableCell>+ <TableCell className="text-right tabular-nums">{p.p95LatencyMs != null ? `${p.p95LatencyMs} ms` : '—'}</TableCell>+ <TableCell className="text-right tabular-nums">{p.avgTtfbMs != null ? `${p.avgTtfbMs} ms` : '—'}</TableCell>+ <TableCell className="text-right tabular-nums">{p.avgTokensPerSecond != null ? p.avgTokensPerSecond : '—'}</TableCell>+ <TableCell className="text-right tabular-nums">{formatTokens(p.totalInputTokens)}</TableCell>+ <TableCell className="text-right tabular-nums pr-4">{formatTokens(p.totalOutputTokens)}</TableCell>+ </TableRow>+ ))}+ </TableBody>+ </Table>+ </div>+ )}+ </Panel>+ </div>+<div className="lg:col-span-2"><Panel title={t('analytics.perModelBreakdown')}>{byModel.length === 0 ? (@@ -610,6 +894,8 @@ export default function AnalyticsPage() {)}</div></div>++ <RequestDetailDialog requestId={detailId} onClose={() => setDetailId(null)} /></div>)}diff --git a/server/src/__tests__/db/migrate/roundtrip.test.ts b/server/src/__tests__/db/migrate/roundtrip.test.tsindex 990b6b15..d5ec3aaf 100644--- a/server/src/__tests__/db/migrate/roundtrip.test.ts+++ b/server/src/__tests__/db/migrate/roundtrip.test.ts@@ -17,6 +17,8 @@ const COOLDOWN_PROBE_PROVENANCE_FILENAME = '20260726_000001_cooldown_probe_proveconst REQUEST_ATTEMPTS_FILENAME = '20260726_000002_request_attempts.ts';const MODEL_SOURCE_PROVENANCE_FILENAME = '20260726_000003_model_source_provenance.ts';const MEDIA_MODEL_META_FILENAME = '20260726_000004_media_model_meta.ts';+const REQUEST_SERVED_MODEL_FILENAME = '20260726_000005_request_served_model.ts';+const ATTEMPT_ERROR_SUMMARY_FILENAME = '20260726_000006_attempt_error_summary.ts';interface SchemaRow {type: string;@@ -80,6 +82,8 @@ describe('migration round trip', () => {REQUEST_ATTEMPTS_FILENAME,MODEL_SOURCE_PROVENANCE_FILENAME,MEDIA_MODEL_META_FILENAME,+ REQUEST_SERVED_MODEL_FILENAME,+ ATTEMPT_ERROR_SUMMARY_FILENAME,]);} finally {db.close();diff --git a/server/src/__tests__/lib/error-redaction.test.ts b/server/src/__tests__/lib/error-redaction.test.tsindex 755c521c..93d4bc0a 100644--- a/server/src/__tests__/lib/error-redaction.test.ts+++ b/server/src/__tests__/lib/error-redaction.test.ts@@ -1,5 +1,5 @@import { describe, it, expect } from 'vitest';-import { sanitizeProviderErrorMessage } from '../../lib/error-redaction.js';+import { sanitizeProviderErrorMessage, summarizeAttemptError } from '../../lib/error-redaction.js';describe('sanitizeProviderErrorMessage', () => {it('redacts Bearer tokens', () => {@@ -42,3 +42,25 @@ describe('sanitizeProviderErrorMessage', () => {expect(out.length).toBeLessThanOrEqual(240);});});++// The short per-attempt error summary stored in request_attempts.error_summary:+// same redactions as sanitizeProviderErrorMessage, tighter cap (200 chars).+describe('summarizeAttemptError', () => {+ it('applies the same secret redactions', () => {+ const out = summarizeAttemptError('401 rejected: Bearer sk-abc.def-12345 for key gsk_A1b2C3d4E5f6G7h8');+ expect(out).not.toContain('sk-abc.def-12345');+ expect(out).not.toContain('gsk_A1b2C3d4E5f6G7h8');+ expect(out).toContain('Bearer [redacted]');+ });++ it('caps the summary at 200 characters with an ellipsis', () => {+ const out = summarizeAttemptError('upstream exploded because ' + 'y '.repeat(300));+ expect(out.length).toBeLessThanOrEqual(200);+ expect(out.endsWith('...')).toBe(true);+ });++ it('leaves ordinary provider prose intact', () => {+ const msg = '429 Too Many Requests: rate limit reached for llama-3.3-70b, retry in 7m12s';+ expect(summarizeAttemptError(msg)).toBe(msg);+ });+});diff --git a/server/src/__tests__/lib/request-attempts.test.ts b/server/src/__tests__/lib/request-attempts.test.tsindex b106f592..7f95ae3f 100644--- a/server/src/__tests__/lib/request-attempts.test.ts+++ b/server/src/__tests__/lib/request-attempts.test.ts@@ -54,11 +54,12 @@ interface AttemptRow {outcome: string;start_offset_ms: number;duration_ms: number;+ error_summary: string | null;}function allAttempts(): AttemptRow[] {return getDb().prepare(- 'SELECT request_id, ordinal, platform, model_id, key_ordinal, outcome, start_offset_ms, duration_ms FROM request_attempts ORDER BY ordinal',+ 'SELECT request_id, ordinal, platform, model_id, key_ordinal, outcome, start_offset_ms, duration_ms, error_summary FROM request_attempts ORDER BY ordinal',).all() as AttemptRow[];}@@ -126,6 +127,39 @@ describe('per-attempt routing traces', () => {}});+ it('persists a redacted, capped error_summary per failed attempt and null for the successful one', async () => {+ const longTail = ' then the provider rambled on'.repeat(20);+ await runFallbackLoop(hooksSkeleton({+ route: () => fakeRoute({ platform: 'groq', modelId: 'llama-x' }),+ dispatch: async (route, attempt) => {+ if (attempt === 0) {+ throw Object.assign(+ new Error(`429 rejected: Bearer sk-secret.token-12345 over quota${longTail}`),+ { status: 429 },+ );+ }+ logRequest(route.platform, route.modelId, route.keyId, 'success', 10, 5, 42, null);+ return 'done';+ },+ logFailure: (route, err) =>+ logRequest(route.platform, route.modelId, route.keyId, 'error', 10, 0, 5, err.message),+ }));++ const rows = allAttempts();+ expect(rows).toHaveLength(2);+ expect(rows.map(r => r.outcome)).toEqual(['rate_limited', 'ok']);++ // Failed hop: short, redacted, capped summary — never the raw secret.+ const summary = rows[0].error_summary;+ expect(summary).toBeTruthy();+ expect(summary!).toContain('429');+ expect(summary!).not.toContain('sk-secret.token-12345');+ expect(summary!.length).toBeLessThanOrEqual(200);++ // Successful hop: no error, no summary.+ expect(rows[1].error_summary).toBeNull();+ });+it("persists exactly one 'ok' attempt for a single-attempt success", async () => {const route = fakeRoute({ platform: 'groq', modelId: 'solo-model' });await runFallbackLoop(hooksSkeleton({diff --git a/server/src/__tests__/lib/served-model.test.ts b/server/src/__tests__/lib/served-model.test.tsnew file mode 100644index 00000000..2011a89e--- /dev/null+++ b/server/src/__tests__/lib/served-model.test.ts@@ -0,0 +1,99 @@+import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';+import {+ normalizeModelIdForDrift,+ observeServedModel,+ resetServedModelObservations,+} from '../../lib/served-model.js';++// Served-model drift guard (#534 follow-up): the proxy overwrites the+// upstream `model` field before the caller sees it, which also destroys the+// only evidence when a provider silently serves a DIFFERENT model than the+// one requested (kilo-auto/small serving Gemma, etc.). This module compares+// the captured raw upstream value against the routed id, ignoring cosmetic+// spelling differences, and reports genuine drift once per tuple.++describe('normalizeModelIdForDrift', () => {+ it('ignores case differences', () => {+ expect(normalizeModelIdForDrift('Llama-3.3-70B-Versatile'))+ .toBe(normalizeModelIdForDrift('llama-3.3-70b-versatile'));+ });++ it('ignores a provider/org prefix', () => {+ expect(normalizeModelIdForDrift('meta-llama/llama-3.3-70b-versatile'))+ .toBe(normalizeModelIdForDrift('llama-3.3-70b-versatile'));+ // Multi-segment namespaces too (@cf/meta/...).+ expect(normalizeModelIdForDrift('@cf/meta/llama-3.1-8b-instruct'))+ .toBe(normalizeModelIdForDrift('llama-3.1-8b-instruct'));+ });++ it('ignores a ":free"-style pricing-tier suffix', () => {+ expect(normalizeModelIdForDrift('gemma-3-27b-it:free'))+ .toBe(normalizeModelIdForDrift('gemma-3-27b-it'));+ });++ it('ignores separator spelling (underscore vs hyphen vs space)', () => {+ expect(normalizeModelIdForDrift('qwen3_coder 480b'))+ .toBe(normalizeModelIdForDrift('qwen3-coder-480b'));+ });++ it('keeps genuinely different ids distinct', () => {+ expect(normalizeModelIdForDrift('kilo-auto/small'))+ .not.toBe(normalizeModelIdForDrift('google/gemma-3-27b-it'));+ expect(normalizeModelIdForDrift('llama-3.3-70b-versatile'))+ .not.toBe(normalizeModelIdForDrift('llama-3.1-8b-instant'));+ });+});++describe('observeServedModel', () => {+ beforeEach(() => {+ resetServedModelObservations();+ vi.spyOn(console, 'warn').mockImplementation(() => {});+ });+ afterEach(() => {+ vi.restoreAllMocks();+ });++ it('returns the raw served id and warns on genuine drift', () => {+ const served = observeServedModel({+ platform: 'kilo', requestedModel: 'kilo-auto/small', servedModel: 'google/gemma-3-27b-it',+ });+ expect(served).toBe('google/gemma-3-27b-it');+ expect(console.warn).toHaveBeenCalledTimes(1);+ expect(vi.mocked(console.warn).mock.calls[0][0])+ .toContain('[ServedModel] platform=kilo requested=kilo-auto/small served=google/gemma-3-27b-it');+ });++ it('warns only once per (platform, requested, served) tuple', () => {+ for (let i = 0; i < 3; i++) {+ observeServedModel({ platform: 'kilo', requestedModel: 'kilo-auto/small', servedModel: 'google/gemma-3-27b-it' });+ }+ expect(console.warn).toHaveBeenCalledTimes(1);+ // A different tuple gets its own single warn.+ observeServedModel({ platform: 'kilo', requestedModel: 'kilo-auto/free', servedModel: 'google/gemma-3-27b-it' });+ expect(console.warn).toHaveBeenCalledTimes(2);+ });++ it('returns null for cosmetically-equal ids (case / prefix / tier suffix)', () => {+ expect(observeServedModel({+ platform: 'groq', requestedModel: 'llama-3.3-70b-versatile', servedModel: 'Meta-Llama/Llama-3.3-70B-Versatile:free',+ })).toBeNull();+ expect(console.warn).not.toHaveBeenCalled();+ });++ it('returns null for identical ids', () => {+ expect(observeServedModel({+ platform: 'groq', requestedModel: 'llama-3.3-70b-versatile', servedModel: 'llama-3.3-70b-versatile',+ })).toBeNull();+ expect(console.warn).not.toHaveBeenCalled();+ });++ it('returns null for missing or placeholder upstream values', () => {+ expect(observeServedModel({ platform: 'reka', requestedModel: 'reka-flash-3', servedModel: null })).toBeNull();+ expect(observeServedModel({ platform: 'reka', requestedModel: 'reka-flash-3', servedModel: undefined })).toBeNull();+ // Reka returns the literal placeholder "default" for every concrete model+ // (#568) — it names no model, so it is no evidence of drift.+ expect(observeServedModel({ platform: 'reka', requestedModel: 'reka-flash-3', servedModel: 'default' })).toBeNull();+ expect(observeServedModel({ platform: 'reka', requestedModel: 'reka-flash-3', servedModel: ' ' })).toBeNull();+ expect(console.warn).not.toHaveBeenCalled();+ });+});diff --git a/server/src/__tests__/lib/url-guard.test.ts b/server/src/__tests__/lib/url-guard.test.tsindex 05bdc46d..e8f0d9c8 100644--- a/server/src/__tests__/lib/url-guard.test.ts+++ b/server/src/__tests__/lib/url-guard.test.ts@@ -1,5 +1,5 @@import { describe, it, expect, afterEach } from 'vitest';-import { classifyIp, assessProviderUrl } from '../../lib/url-guard.js';+import { classifyIp, assessProviderUrl, isLoopbackOrPrivateUrl } from '../../lib/url-guard.js';// SSRF guard for user-supplied custom-provider base URLs (#440). Metadata and// link-local targets must never be reachable; loopback/private stay allowed by@@ -140,3 +140,34 @@ describe('assessProviderUrl', () => {expect(verdict.allowed).toBe(true);});});++describe('isLoopbackOrPrivateUrl (#592 local-endpoint cooldown exemption)', () => {+ it('true for loopback and localhost forms', () => {+ expect(isLoopbackOrPrivateUrl('http://127.0.0.1:11434/v1')).toBe(true);+ expect(isLoopbackOrPrivateUrl('http://localhost:11434/v1')).toBe(true);+ expect(isLoopbackOrPrivateUrl('http://ollama.localhost/v1')).toBe(true);+ expect(isLoopbackOrPrivateUrl('http://[::1]:8080/v1')).toBe(true);+ expect(isLoopbackOrPrivateUrl('http://0.0.0.0:8080/v1')).toBe(true);+ });++ it('true for RFC1918 / ULA literal addresses', () => {+ expect(isLoopbackOrPrivateUrl('http://192.168.1.50:8080/v1')).toBe(true);+ expect(isLoopbackOrPrivateUrl('http://10.0.0.7:11434/v1')).toBe(true);+ expect(isLoopbackOrPrivateUrl('http://172.16.4.2:5000/v1')).toBe(true);+ expect(isLoopbackOrPrivateUrl('http://[fd12:3456::1]:8080/v1')).toBe(true);+ });++ it('false for public addresses and hostnames (no DNS — pragmatically non-local)', () => {+ expect(isLoopbackOrPrivateUrl('http://203.0.113.10/v1')).toBe(false);+ expect(isLoopbackOrPrivateUrl('https://api.example.com/v1')).toBe(false);+ expect(isLoopbackOrPrivateUrl('http://my-lan-box.local:11434/v1')).toBe(false); // mDNS: undecidable without DNS+ expect(isLoopbackOrPrivateUrl('https://ollama.com/v1')).toBe(false); // Ollama CLOUD stays protected+ });++ it('false for null/empty/garbage input', () => {+ expect(isLoopbackOrPrivateUrl(null)).toBe(false);+ expect(isLoopbackOrPrivateUrl(undefined)).toBe(false);+ expect(isLoopbackOrPrivateUrl('')).toBe(false);+ expect(isLoopbackOrPrivateUrl('not a url')).toBe(false);+ });+});diff --git a/server/src/__tests__/routes/analytics-request-attempts.test.ts b/server/src/__tests__/routes/analytics-request-attempts.test.tsindex 24706bf5..7fe86125 100644--- a/server/src/__tests__/routes/analytics-request-attempts.test.ts+++ b/server/src/__tests__/routes/analytics-request-attempts.test.ts@@ -32,11 +32,11 @@ function insertRequest(status: string, createdAt: string): number {return Number(insert.lastInsertRowid);}-function insertAttempt(requestId: number, ordinal: number, platform: string, modelId: string, keyOrdinal: number, outcome: string, startOffsetMs: number, durationMs: number) {+function insertAttempt(requestId: number, ordinal: number, platform: string, modelId: string, keyOrdinal: number, outcome: string, startOffsetMs: number, durationMs: number, errorSummary: string | null = null) {getDb().prepare(`- INSERT INTO request_attempts (request_id, ordinal, platform, model_id, key_ordinal, outcome, start_offset_ms, duration_ms)- VALUES (?, ?, ?, ?, ?, ?, ?, ?)- `).run(requestId, ordinal, platform, modelId, keyOrdinal, outcome, startOffsetMs, durationMs);+ INSERT INTO request_attempts (request_id, ordinal, platform, model_id, key_ordinal, outcome, start_offset_ms, duration_ms, error_summary)+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)+ `).run(requestId, ordinal, platform, modelId, keyOrdinal, outcome, startOffsetMs, durationMs, errorSummary);}function recentSql(hour: number): string {@@ -72,10 +72,10 @@ describe('per-attempt traces in the analytics requests API', () => {expect(body.rows.map((r: any) => r.attemptCount)).toEqual([0, 3]);});- it('returns the ordered attempts ladder in the per-request detail payload', async () => {+ it('returns the ordered attempts ladder — with per-hop error summaries — in the per-request detail payload', async () => {const id = insertRequest('success', recentSql(12));- insertAttempt(id, 0, 'groq', 'llama-x', 1, 'rate_limited', 0, 210);- insertAttempt(id, 1, 'google', 'gemini-y', 2, 'timeout', 215, 10000);+ insertAttempt(id, 0, 'groq', 'llama-x', 1, 'rate_limited', 0, 210, '429 rate limit reached, retry in 7m12s');+ insertAttempt(id, 1, 'google', 'gemini-y', 2, 'timeout', 215, 10000, 'upstream timeout after 10000ms');insertAttempt(id, 2, 'cerebras', 'qwen-z', 3, 'ok', 10220, 900);const { status, body } = await request(app, `/api/analytics/requests/${id}`);@@ -88,9 +88,9 @@ describe('per-attempt traces in the analytics requests API', () => {latencyMs: 1234,});expect(body.attempts).toEqual([- { ordinal: 0, platform: 'groq', modelId: 'llama-x', keyOrdinal: 1, outcome: 'rate_limited', startOffsetMs: 0, durationMs: 210 },- { ordinal: 1, platform: 'google', modelId: 'gemini-y', keyOrdinal: 2, outcome: 'timeout', startOffsetMs: 215, durationMs: 10000 },- { ordinal: 2, platform: 'cerebras', modelId: 'qwen-z', keyOrdinal: 3, outcome: 'ok', startOffsetMs: 10220, durationMs: 900 },+ { ordinal: 0, platform: 'groq', modelId: 'llama-x', keyOrdinal: 1, outcome: 'rate_limited', startOffsetMs: 0, durationMs: 210, errorSummary: '429 rate limit reached, retry in 7m12s' },+ { ordinal: 1, platform: 'google', modelId: 'gemini-y', keyOrdinal: 2, outcome: 'timeout', startOffsetMs: 215, durationMs: 10000, errorSummary: 'upstream timeout after 10000ms' },+ { ordinal: 2, platform: 'cerebras', modelId: 'qwen-z', keyOrdinal: 3, outcome: 'ok', startOffsetMs: 10220, durationMs: 900, errorSummary: null },]);});diff --git a/server/src/__tests__/routes/analytics-requests.test.ts b/server/src/__tests__/routes/analytics-requests.test.tsindex 1cc0e97c..d47fc1cd 100644--- a/server/src/__tests__/routes/analytics-requests.test.ts+++ b/server/src/__tests__/routes/analytics-requests.test.ts@@ -21,11 +21,11 @@ async function request(app: Express, path: string) {return { status: res.status, body: data };}-function insertCall(createdAt: string, clientIp: string | null, clientUserAgent: string | null, status = 'success') {+function insertCall(createdAt: string, clientIp: string | null, clientUserAgent: string | null, status = 'success', platform = 'test') {getDb().prepare(`INSERT INTO requests (platform, model_id, status, input_tokens, output_tokens, latency_ms, error, created_at, client_ip, client_user_agent)- VALUES ('test', 'test-model', ?, 10, 5, 42, NULL, ?, ?, ?)- `).run(status, createdAt, clientIp, clientUserAgent);+ VALUES (?, 'test-model', ?, 10, 5, 42, NULL, ?, ?, ?)+ `).run(platform, status, createdAt, clientIp, clientUserAgent);}function recentUtcTimestamp(hour: number) {@@ -86,6 +86,46 @@ describe('GET /api/analytics/requests', () => {expect(clamped.body.rows).toHaveLength(5);});+ it('filters by status, keeping total consistent with the filter', async () => {+ insertCall(recentUtcTimestamp(8).sql, '10.0.0.1', 'ua', 'success');+ insertCall(recentUtcTimestamp(9).sql, '10.0.0.1', 'ua', 'error');+ insertCall(recentUtcTimestamp(10).sql, '10.0.0.1', 'ua', 'error');++ const errors = await request(app, '/api/analytics/requests?range=7d&status=error');+ expect(errors.status).toBe(200);+ expect(errors.body.total).toBe(2);+ expect(errors.body.rows.map((r: any) => r.status)).toEqual(['error', 'error']);++ const successes = await request(app, '/api/analytics/requests?range=7d&status=success');+ expect(successes.body.total).toBe(1);+ expect(successes.body.rows.map((r: any) => r.status)).toEqual(['success']);+ });++ it('filters by platform, and combines with the status filter', async () => {+ insertCall(recentUtcTimestamp(8).sql, '10.0.0.1', 'ua', 'success', 'groq');+ insertCall(recentUtcTimestamp(9).sql, '10.0.0.1', 'ua', 'error', 'groq');+ insertCall(recentUtcTimestamp(10).sql, '10.0.0.1', 'ua', 'error', 'google');++ const groq = await request(app, '/api/analytics/requests?range=7d&platform=groq');+ expect(groq.status).toBe(200);+ expect(groq.body.total).toBe(2);+ expect(groq.body.rows.map((r: any) => r.platform)).toEqual(['groq', 'groq']);++ const combined = await request(app, '/api/analytics/requests?range=7d&platform=groq&status=error');+ expect(combined.body.total).toBe(1);+ expect(combined.body.rows).toHaveLength(1);+ expect(combined.body.rows[0]).toMatchObject({ platform: 'groq', status: 'error' });+ });++ it('rejects invalid status/platform filter values with 400', async () => {+ insertCall(recentUtcTimestamp(8).sql, '10.0.0.1', 'ua');++ expect((await request(app, '/api/analytics/requests?range=7d&status=weird')).status).toBe(400);+ expect((await request(app, '/api/analytics/requests?range=7d&platform=' + encodeURIComponent('a b;--'))).status).toBe(400);+ // Absent filters keep the default behavior.+ expect((await request(app, '/api/analytics/requests?range=7d')).status).toBe(200);+ });+it('records the request-scoped caller identity through logRequest', async () => {const { clientContextMiddleware } = await import('../../lib/client-context.js');const req = { headers: { 'user-agent': 'vitest-client/1.0' }, socket: { remoteAddress: '192.168.0.99' } } as any;diff --git a/server/src/__tests__/routes/proxy-served-model.test.ts b/server/src/__tests__/routes/proxy-served-model.test.tsnew file mode 100644index 00000000..1ad78c26--- /dev/null+++ b/server/src/__tests__/routes/proxy-served-model.test.ts@@ -0,0 +1,220 @@+import { describe, it, expect, beforeAll, beforeEach, afterEach, vi } from 'vitest';+import type { Express } from 'express';+import { createApp } from '../../app.js';+import { initDb, getDb, getUnifiedApiKey } from '../../db/index.js';+import { resetServedModelObservations } from '../../lib/served-model.js';+import { mintDashboardToken, isGatedApiPath } from '../helpers/auth.js';++// Served-model drift guard (#534 follow-up): the proxy overwrites the+// upstream `model` field with the routed id before the caller sees it (the+// gateway contract — unchanged here). These tests assert the raw upstream+// value is captured BEFORE that overwrite, compared against the routed id,+// warn-logged once per tuple, and persisted to requests.served_model when it+// genuinely differs (NULL otherwise), on both the non-streaming and the+// streaming path.++let dashToken = '';++async function request(app: Express, method: string, path: string, body?: any, headers: Record<string, string> = {}) {+ const server = app.listen(0);+ const addr = server.address() as any;+ const res = await fetch(`http://127.0.0.1:${addr.port}${path}`, {+ method,+ headers: {+ ...(body ? { 'Content-Type': 'application/json' } : {}),+ ...(isGatedApiPath(path) && !('Authorization' in headers) ? { Authorization: `Bearer ${dashToken}` } : {}),+ ...headers,+ },+ body: body ? JSON.stringify(body) : undefined,+ });+ const text = await res.text();+ server.close();+ let json: any = null;+ try { json = JSON.parse(text); } catch { /* SSE body */ }+ return { status: res.status, headers: res.headers, text, body: json };+}++function authHeaders() {+ return { Authorization: `Bearer ${getUnifiedApiKey()}` };+}++/** Parse an SSE response body into JSON frames (excluding [DONE]). */+function frames(text: string): any[] {+ return text.split('\n')+ .filter(l => l.startsWith('data: ') && l.trim() !== 'data: [DONE]')+ .map(l => JSON.parse(l.slice(6)));+}++const sse = (...payloads: (object | string)[]) =>+ payloads.map(p => `data: ${typeof p === 'string' ? p : JSON.stringify(p)}\n\n`).join('');++/** Mock the groq upstream. `upstreamModel` is what the provider CLAIMS it served. */+function mockUpstream(opts: { upstreamModel: string; stream?: boolean }) {+ const origFetch = global.fetch;+ vi.spyOn(global, 'fetch').mockImplementation(async (url, init) => {+ const urlStr = typeof url === 'string' ? url : url.toString();+ if (!urlStr.includes('api.groq.com')) return origFetch(url as any, init as any);+ if (opts.stream) {+ const chunk = (delta: object, finish: string | null) => ({+ id: 'c1', object: 'chat.completion.chunk', created: 1, model: opts.upstreamModel,+ choices: [{ index: 0, delta, finish_reason: finish }],+ });+ return new Response(+ sse(chunk({ role: 'assistant' }, null), chunk({ content: 'streamed hello' }, null), chunk({}, 'stop'), '[DONE]'),+ { status: 200, headers: { 'Content-Type': 'text/event-stream' } },+ );+ }+ return new Response(JSON.stringify({+ id: 'chatcmpl-served', object: 'chat.completion', created: 1, model: opts.upstreamModel,+ choices: [{ index: 0, message: { role: 'assistant', content: 'hello' }, finish_reason: 'stop' }],+ usage: { prompt_tokens: 2, completion_tokens: 1, total_tokens: 3 },+ }), { status: 200, headers: { 'Content-Type': 'application/json' } });+ });+}++function lastRequestRow(): any {+ return getDb().prepare(+ 'SELECT id, model_id, served_model, status FROM requests ORDER BY id DESC LIMIT 1'+ ).get();+}++function servedModelWarns(): string[] {+ return vi.mocked(console.warn).mock.calls+ .map(c => String(c[0]))+ .filter(m => m.startsWith('[ServedModel]'));+}++describe('served-model drift guard (#534)', () => {+ let app: Express;+ let groqModelId: string;++ beforeAll(() => {+ process.env.ENCRYPTION_KEY = '0'.repeat(64);+ initDb(':memory:');+ app = createApp();+ dashToken = mintDashboardToken();+ groqModelId = (getDb().prepare(`+ SELECT m.model_id FROM models m+ JOIN fallback_config fc ON fc.model_db_id = m.id+ WHERE m.platform = 'groq' AND m.enabled = 1+ ORDER BY fc.priority LIMIT 1+ `).get() as { model_id: string }).model_id;+ });++ beforeEach(async () => {+ resetServedModelObservations();+ const db = getDb();+ db.prepare('DELETE FROM api_keys').run();+ db.prepare('DELETE FROM requests').run();+ db.prepare('DELETE FROM rate_limit_cooldowns').run();+ db.prepare('DELETE FROM rate_limit_usage').run();+ const addKey = await request(app, 'POST', '/api/keys',+ { platform: 'groq', key: 'gsk_served_model_guard_test', label: 'served-model' });+ expect(addKey.status).toBe(201);+ vi.spyOn(console, 'warn');+ });++ afterEach(() => {+ vi.restoreAllMocks();+ });++ it('non-streaming: captures, logs, and persists a genuinely different served model — contract unchanged', async () => {+ mockUpstream({ upstreamModel: 'gemma-3-27b-it' });+ const { status, body } = await request(app, 'POST', '/v1/chat/completions', {+ model: groqModelId, messages: [{ role: 'user', content: 'drift test' }],+ }, authHeaders());+ expect(status).toBe(200);+ // Contract unchanged: the caller still sees the ROUTED model.+ expect(body.model).toBe(groqModelId);++ const row = lastRequestRow();+ expect(row.status).toBe('success');+ expect(row.model_id).toBe(groqModelId);+ expect(row.served_model).toBe('gemma-3-27b-it');++ const warns = servedModelWarns();+ expect(warns).toHaveLength(1);+ expect(warns[0]).toContain(`platform=groq requested=${groqModelId} served=gemma-3-27b-it`);+ });++ it('non-streaming: warns once per tuple but persists every drifted row', async () => {+ mockUpstream({ upstreamModel: 'gemma-3-27b-it' });+ for (let i = 0; i < 2; i++) {+ const { status } = await request(app, 'POST', '/v1/chat/completions', {+ model: groqModelId, messages: [{ role: 'user', content: `drift repeat ${i}` }],+ }, authHeaders());+ expect(status).toBe(200);+ }+ expect(servedModelWarns()).toHaveLength(1);+ const rows = getDb().prepare(+ "SELECT served_model FROM requests WHERE status = 'success' ORDER BY id"+ ).all() as any[];+ expect(rows).toHaveLength(2);+ for (const r of rows) expect(r.served_model).toBe('gemma-3-27b-it');+ });++ it('non-streaming: cosmetically-equal id (case + org prefix + :free suffix) does not flag', async () => {+ mockUpstream({ upstreamModel: `some-org/${groqModelId}:free`.toUpperCase() });+ const { status, body } = await request(app, 'POST', '/v1/chat/completions', {+ model: groqModelId, messages: [{ role: 'user', content: 'cosmetic test' }],+ }, authHeaders());+ expect(status).toBe(200);+ expect(body.model).toBe(groqModelId);+ expect(lastRequestRow().served_model).toBeNull();+ expect(servedModelWarns()).toHaveLength(0);+ });++ it('non-streaming: identical served id stores NULL', async () => {+ mockUpstream({ upstreamModel: groqModelId });+ const { status } = await request(app, 'POST', '/v1/chat/completions', {+ model: groqModelId, messages: [{ role: 'user', content: 'same id test' }],+ }, authHeaders());+ expect(status).toBe(200);+ expect(lastRequestRow().served_model).toBeNull();+ expect(servedModelWarns()).toHaveLength(0);+ });++ it('streaming: captures, logs, and persists drift — streamed frames still show the routed model', async () => {+ mockUpstream({ upstreamModel: 'mystery-substitute-model', stream: true });+ const r = await request(app, 'POST', '/v1/chat/completions', {+ model: groqModelId, stream: true, messages: [{ role: 'user', content: 'stream drift test' }],+ }, authHeaders());+ expect(r.status).toBe(200);+ const fs = frames(r.text);+ expect(fs.length).toBeGreaterThan(0);+ // Contract unchanged: every outbound frame carries the routed model.+ for (const f of fs) expect(f.model).toBe(groqModelId);+ expect(fs.some(f => f.choices?.[0]?.delta?.content?.includes('streamed hello'))).toBe(true);++ const row = lastRequestRow();+ expect(row.status).toBe('success');+ expect(row.served_model).toBe('mystery-substitute-model');+ const warns = servedModelWarns();+ expect(warns).toHaveLength(1);+ expect(warns[0]).toContain(`platform=groq requested=${groqModelId} served=mystery-substitute-model`);+ });++ it('streaming: identical served id stores NULL and does not warn', async () => {+ mockUpstream({ upstreamModel: groqModelId, stream: true });+ const r = await request(app, 'POST', '/v1/chat/completions', {+ model: groqModelId, stream: true, messages: [{ role: 'user', content: 'stream same test' }],+ }, authHeaders());+ expect(r.status).toBe(200);+ expect(lastRequestRow().served_model).toBeNull();+ expect(servedModelWarns()).toHaveLength(0);+ });++ it('exposes servedModel in the per-request analytics detail payload', async () => {+ mockUpstream({ upstreamModel: 'gemma-3-27b-it' });+ const { status } = await request(app, 'POST', '/v1/chat/completions', {+ model: groqModelId, messages: [{ role: 'user', content: 'detail test' }],+ }, authHeaders());+ expect(status).toBe(200);+ const row = lastRequestRow();++ const detail = await request(app, 'GET', `/api/analytics/requests/${row.id}`);+ expect(detail.status).toBe(200);+ expect(detail.body.servedModel).toBe('gemma-3-27b-it');+ expect(detail.body.modelId).toBe(groqModelId);+ });+});diff --git a/server/src/__tests__/services/declarative-config.test.ts b/server/src/__tests__/services/declarative-config.test.tsindex d806c43a..d03c1620 100644--- a/server/src/__tests__/services/declarative-config.test.ts+++ b/server/src/__tests__/services/declarative-config.test.ts@@ -1,4 +1,4 @@-import { describe, it, expect, beforeAll, beforeEach, afterEach } from 'vitest';+import { describe, it, expect, beforeAll, beforeEach, afterEach, vi } from 'vitest';import { initDb, getDb } from '../../db/index.js';import { applyDeclarativeConfig, applyDeclarativeConfigFromEnv } from '../../services/declarative-config.js';import { getRoutingStrategy } from '../../services/router.js';@@ -104,4 +104,90 @@ describe('declarative config import', () => {expect(applyDeclarativeConfigFromEnv().applied).toBe(true);expect((getDb().prepare("SELECT COUNT(*) AS n FROM api_keys WHERE platform = 'groq' AND label = 'config'").get() as { n: number }).n).toBe(1);});++ // #600: pollinations lost `keyless: true` in #573, so a legacy declarative+ // config with a keyless-era pollinations entry (no `key`) used to throw at+ // boot and brick the install. Missing-key entries must now degrade to a+ // per-entry skip + warning instead of killing the apply.+ it('skips a legacy keyless entry without a key, warns, and still applies the rest', () => {+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});+ try {+ const result = applyDeclarativeConfig({+ keys: [+ { platform: 'pollinations' },+ { platform: 'groq', key: 'gsk_config_key', label: 'config' },+ ],+ });+ expect(result.applied).toBe(true);+ expect(result.keys).toBe(1);+ expect(result.warnings).toHaveLength(1);+ expect(result.warnings[0]).toContain('pollinations');+ expect(result.warnings[0]).toContain('enter.pollinations.ai');+ expect(result.warnings[0]).toContain('entry skipped');+ expect(warn).toHaveBeenCalledWith(expect.stringContaining('pollinations'));+ expect((getDb().prepare("SELECT COUNT(*) AS n FROM api_keys WHERE platform = 'pollinations'").get() as { n: number }).n).toBe(0);+ expect((getDb().prepare("SELECT COUNT(*) AS n FROM api_keys WHERE platform = 'groq'").get() as { n: number }).n).toBe(1);+ } finally {+ warn.mockRestore();+ }+ });++ it('boot apply (FromEnv) survives a legacy keyless pollinations entry', () => {+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});+ try {+ process.env.FREEAPI_CONFIG_JSON = JSON.stringify({+ keys: [+ { platform: 'pollinations', label: 'legacy' },+ { platform: 'groq', key: 'gsk_config_key', label: 'config' },+ ],+ });+ expect(() => applyDeclarativeConfigFromEnv()).not.toThrow();+ expect((getDb().prepare("SELECT COUNT(*) AS n FROM api_keys WHERE platform = 'groq'").get() as { n: number }).n).toBe(1);+ expect((getDb().prepare("SELECT COUNT(*) AS n FROM api_keys WHERE platform = 'pollinations'").get() as { n: number }).n).toBe(0);+ } finally {+ warn.mockRestore();+ }+ });++ it('still applies a pollinations entry that has a key', () => {+ const result = applyDeclarativeConfig({+ keys: [{ platform: 'pollinations', key: 'pk_live_abc', label: 'config' }],+ });+ expect(result.keys).toBe(1);+ expect(result.warnings).toHaveLength(0);+ expect((getDb().prepare("SELECT COUNT(*) AS n FROM api_keys WHERE platform = 'pollinations'").get() as { n: number }).n).toBe(1);+ });++ it('generic missing-key skip covers any non-keyless platform, with a platform-named warning', () => {+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});+ try {+ const result = applyDeclarativeConfig({+ keys: [{ platform: 'groq' }],+ });+ expect(result.keys).toBe(0);+ expect(result.warnings).toHaveLength(1);+ expect(result.warnings[0]).toContain('groq');+ expect(result.warnings[0]).toContain('entry skipped');+ } finally {+ warn.mockRestore();+ }+ });++ it('a keyless platform entry without a key still applies its sentinel row', () => {+ const result = applyDeclarativeConfig({+ keys: [{ platform: 'kilo', label: 'config' }],+ });+ expect(result.keys).toBe(1);+ expect(result.warnings).toHaveLength(0);+ expect((getDb().prepare("SELECT COUNT(*) AS n FROM api_keys WHERE platform = 'kilo'").get() as { n: number }).n).toBe(1);+ });++ it('genuinely malformed config still fails loudly', () => {+ expect(() => applyDeclarativeConfig({ keys: [{ platform: 'groq' }], nonsense: true }))+ .toThrow(/invalid declarative config/);+ expect(() => applyDeclarativeConfig({ keys: [{ platform: 'unknown-platform', key: 'x' }] }))+ .toThrow(/unknown provider platform/);+ process.env.FREEAPI_CONFIG_JSON = '{ not json';+ expect(() => applyDeclarativeConfigFromEnv()).toThrow();+ });});diff --git a/server/src/__tests__/services/ratelimit-cooldown-error-kind.test.ts b/server/src/__tests__/services/ratelimit-cooldown-error-kind.test.tsnew file mode 100644index 00000000..8a94dd05--- /dev/null+++ b/server/src/__tests__/services/ratelimit-cooldown-error-kind.test.ts@@ -0,0 +1,109 @@+import { describe, it, expect, beforeAll, beforeEach } from 'vitest';++// #592: the null-limits exhaustion heuristic (getCooldownDecisionForLimit) must+// only fire on an actual provider quota signal (a real 429 / rate-limit-classified+// error). Before the fix, ANY retryable failure — a timeout on a slow local+// generation, a 5xx blip — fed the "2+ hits in an hour = daily-exhausted"+// counter and escalated the bench 2m→10m→1h→24h, 429-ing every request on what+// is often the user's only route. Also covers the cooldownHits ladder decay on+// success: two back-to-back failures used to keep their ladder step for 24h+// even after successful requests proved the quota alive.++import { initDb } from '../../db/index.js';+import { cooldownDecisionForError } from '../../lib/fallback-loop.js';+import {+ getNextCooldownDuration,+ recordRequest,+ recentHitCount,+} from '../../services/ratelimit.js';+import type { RouteResult } from '../../services/router.js';++const MINUTE = 60_000;+const TRANSIENT = 90_000;++// Distinct keyId AND modelDbId per fake route: the cooldown/ladder maps are+// module-global, so shared ids would leak state across tests.+let keySeq = 640_000;+function nullLimitRoute(): RouteResult {+ const n = ++keySeq;+ return {+ provider: {} as any, modelId: `null-limit-model-${n}`, modelDbId: 592_000 + n,+ apiKey: 'k', keyId: n, platform: 'ollama', displayName: 'Null-Limit Model',+ rpdLimit: null, tpdLimit: null,+ };+}++const timeoutErr = () => Object.assign(+ new Error('Request timeout after 120000ms'), { name: 'TimeoutError' },+);+const serverErr = () => Object.assign(+ new Error('Ollama API error 500: Internal Server Error'), { status: 500 },+);+const real429 = () => Object.assign(+ new Error('429 Too Many Requests'), { status: 429 },+);++beforeAll(() => {+ process.env.ENCRYPTION_KEY = '0'.repeat(64);+ initDb(':memory:');+});++let route: RouteResult;+beforeEach(() => { route = nullLimitRoute(); });++describe('null-limits heuristic only fires on quota signals (#592)', () => {+ it('repeated timeouts stay on the short transient bench — no ladder', () => {+ for (let i = 0; i < 4; i++) {+ const decision = cooldownDecisionForError(route, timeoutErr());+ expect(decision.durationMs).toBe(TRANSIENT);+ expect(decision.source).toBe('heuristic');+ }+ // No hits recorded → the heuristic counter never sees a timeout.+ expect(recentHitCount(route.platform, route.modelId, route.keyId, Date.now())).toBe(0);+ });++ it('repeated 5xx stay on the short transient bench — no ladder', () => {+ for (let i = 0; i < 4; i++) {+ expect(cooldownDecisionForError(route, serverErr()).durationMs).toBe(TRANSIENT);+ }+ expect(recentHitCount(route.platform, route.modelId, route.keyId, Date.now())).toBe(0);+ });++ it('a timeout does not inherit escalation started by real 429s', () => {+ expect(cooldownDecisionForError(route, real429()).durationMs).toBe(TRANSIENT);+ // 2nd real 429 crosses the heuristic threshold → ladder (2m).+ expect(cooldownDecisionForError(route, real429()).durationMs).toBe(2 * MINUTE);+ // A timeout right after must NOT climb to the next ladder step (10m):+ // it is not a quota signal, so it neither records a hit nor escalates.+ expect(cooldownDecisionForError(route, timeoutErr()).durationMs).toBe(TRANSIENT);+ // Hit count still only reflects the two genuine 429s.+ expect(recentHitCount(route.platform, route.modelId, route.keyId, Date.now())).toBe(2);+ });++ it('real 429s still escalate — the Ollama-Cloud opaque-quota protection holds', () => {+ // 1st 429 → transient (no signal yet), then the ladder: 2m, 10m, 1h.+ expect(cooldownDecisionForError(route, real429()).durationMs).toBe(TRANSIENT);+ expect(cooldownDecisionForError(route, real429()).durationMs).toBe(2 * MINUTE);+ expect(cooldownDecisionForError(route, real429()).durationMs).toBe(10 * MINUTE);+ expect(cooldownDecisionForError(route, real429()).durationMs).toBe(60 * MINUTE);+ });+});++describe('cooldownHits ladder decays on success (#592)', () => {+ it('a successful request resets the escalation ladder', () => {+ const { platform, modelId, keyId } = nullLimitRoute();+ expect(getNextCooldownDuration(platform, modelId, keyId)).toBe(2 * MINUTE);+ expect(getNextCooldownDuration(platform, modelId, keyId)).toBe(10 * MINUTE);+ // A served request proves the quota is alive — ladder starts over.+ recordRequest(platform, modelId, keyId);+ expect(getNextCooldownDuration(platform, modelId, keyId)).toBe(2 * MINUTE);+ });++ it('without a success the ladder keeps escalating (documented contract intact)', () => {+ const { platform, modelId, keyId } = nullLimitRoute();+ expect(getNextCooldownDuration(platform, modelId, keyId)).toBe(2 * MINUTE);+ expect(getNextCooldownDuration(platform, modelId, keyId)).toBe(10 * MINUTE);+ expect(getNextCooldownDuration(platform, modelId, keyId)).toBe(60 * MINUTE);+ expect(getNextCooldownDuration(platform, modelId, keyId)).toBe(24 * 60 * MINUTE);+ });+});diff --git a/server/src/__tests__/services/ratelimit-local-endpoint.test.ts b/server/src/__tests__/services/ratelimit-local-endpoint.test.tsnew file mode 100644index 00000000..4761339e--- /dev/null+++ b/server/src/__tests__/services/ratelimit-local-endpoint.test.ts@@ -0,0 +1,96 @@+import { describe, it, expect, beforeAll } from 'vitest';++// #592 locality exemption: a key whose base_url points at loopback or an+// RFC1918/private address is a LOCAL inference server (Ollama / llama.cpp /+// LM Studio registered through the 'custom' platform). It has no provider+// quota, so neither the 90s transient bench nor the 2m→24h escalation ladder+// ever applies — the bench is capped at a few seconds, even for a literal 429+// (local servers emit those under concurrent load) and even when a Retry-After+// is attached. Non-local keys (public IPs/hostnames, built-in platforms with+// base_url NULL) keep the full pre-existing behavior.++import { initDb, getDb } from '../../db/index.js';+import { cooldownDecisionForError } from '../../lib/fallback-loop.js';+import type { RouteResult } from '../../services/router.js';++const MINUTE = 60_000;+const TRANSIENT = 90_000;+const LOCAL_CAP_MS = 5_000;++beforeAll(() => {+ process.env.ENCRYPTION_KEY = '0'.repeat(64);+ initDb(':memory:');+});++// Register a custom-platform key row carrying the endpoint, like+// routes/keys.ts POST /custom does, and return its real DB id.+function insertKey(baseUrl: string | null): number {+ const r = getDb().prepare(`+ INSERT INTO api_keys (platform, label, encrypted_key, iv, auth_tag, status, enabled, base_url)+ VALUES ('custom', 'test-endpoint', 'enc', 'iv', 'tag', 'valid', 1, ?)+ `).run(baseUrl);+ return Number(r.lastInsertRowid);+}++let modelSeq = 0;+function routeForKey(keyId: number): RouteResult {+ const n = ++modelSeq;+ return {+ provider: {} as any, modelId: `local-model-${n}`, modelDbId: 593_000 + n,+ apiKey: 'k', keyId, platform: 'custom', displayName: 'Local Model',+ rpdLimit: null, tpdLimit: null,+ };+}++const real429 = () => Object.assign(new Error('429 Too Many Requests'), { status: 429 });+const timeoutErr = () => Object.assign(new Error('Request timeout after 120000ms'), { name: 'TimeoutError' });++describe('local-endpoint cooldown cap (#592)', () => {+ it('loopback base_url: capped short bench even on repeated real 429s', () => {+ const route = routeForKey(insertKey('http://127.0.0.1:11434/v1'));+ for (let i = 0; i < 4; i++) {+ const decision = cooldownDecisionForError(route, real429());+ expect(decision.durationMs).toBe(LOCAL_CAP_MS);+ expect(decision.source).toBe('heuristic');+ }+ });++ it('localhost hostname counts as loopback without DNS', () => {+ const route = routeForKey(insertKey('http://localhost:11434/v1'));+ expect(cooldownDecisionForError(route, real429()).durationMs).toBe(LOCAL_CAP_MS);+ expect(cooldownDecisionForError(route, timeoutErr()).durationMs).toBe(LOCAL_CAP_MS);+ });++ it('RFC1918 base_url (LAN box): capped the same way', () => {+ for (const url of ['http://192.168.1.50:8080/v1', 'http://10.0.0.7:11434/v1', 'http://172.16.4.2:5000/v1']) {+ const route = routeForKey(insertKey(url));+ expect(cooldownDecisionForError(route, real429()).durationMs).toBe(LOCAL_CAP_MS);+ expect(cooldownDecisionForError(route, real429()).durationMs).toBe(LOCAL_CAP_MS);+ }+ });++ it('ignores an upstream Retry-After on a local endpoint', () => {+ const route = routeForKey(insertKey('http://127.0.0.1:8080/v1'));+ const err = Object.assign(new Error('429 Too Many Requests'), { status: 429, retryAfterMs: 5 * MINUTE });+ expect(cooldownDecisionForError(route, err).durationMs).toBe(LOCAL_CAP_MS);+ });++ it('public-IP base_url is NOT exempt: transient then ladder on real 429s', () => {+ const route = routeForKey(insertKey('http://203.0.113.10/v1'));+ expect(cooldownDecisionForError(route, real429()).durationMs).toBe(TRANSIENT);+ expect(cooldownDecisionForError(route, real429()).durationMs).toBe(2 * MINUTE);+ expect(cooldownDecisionForError(route, real429()).durationMs).toBe(10 * MINUTE);+ });++ it('non-literal hostname is treated as non-local (no DNS on the hot path)', () => {+ const route = routeForKey(insertKey('https://inference.example.com/v1'));+ expect(cooldownDecisionForError(route, real429()).durationMs).toBe(TRANSIENT);+ expect(cooldownDecisionForError(route, real429()).durationMs).toBe(2 * MINUTE);+ });++ it('a key with no base_url (built-in platform shape) keeps full behavior', () => {+ const route = routeForKey(insertKey(null));+ expect(cooldownDecisionForError(route, real429()).durationMs).toBe(TRANSIENT);+ expect(cooldownDecisionForError(route, real429()).durationMs).toBe(2 * MINUTE);+ });+});diff --git a/server/src/db/migrate/defaults.ts b/server/src/db/migrate/defaults.tsindex 633cde82..8c717092 100644--- a/server/src/db/migrate/defaults.ts+++ b/server/src/db/migrate/defaults.ts@@ -12,6 +12,8 @@ import * as cooldownProbeProvenance from '../migrations/20260726_000001_cooldownimport * as requestAttempts from '../migrations/20260726_000002_request_attempts.js';import * as modelSourceProvenance from '../migrations/20260726_000003_model_source_provenance.js';import * as mediaModelMeta from '../migrations/20260726_000004_media_model_meta.js';+import * as requestServedModel from '../migrations/20260726_000005_request_served_model.js';+import * as attemptErrorSummary from '../migrations/20260726_000006_attempt_error_summary.js';export interface MigrationModule {up(db: Db): void;@@ -36,6 +38,8 @@ export const COOLDOWN_PROBE_PROVENANCE_FILENAME = '20260726_000001_cooldown_probexport const REQUEST_ATTEMPTS_FILENAME = '20260726_000002_request_attempts.ts';export const MODEL_SOURCE_PROVENANCE_FILENAME = '20260726_000003_model_source_provenance.ts';export const MEDIA_MODEL_META_FILENAME = '20260726_000004_media_model_meta.ts';+export const REQUEST_SERVED_MODEL_FILENAME = '20260726_000005_request_served_model.ts';+export const ATTEMPT_ERROR_SUMMARY_FILENAME = '20260726_000006_attempt_error_summary.ts';export const DEFAULT_MIGRATIONS: readonly DefaultMigration[] = [{ filename: LEGACY_BASELINE_FILENAME, module: legacyBaseline },@@ -51,4 +55,6 @@ export const DEFAULT_MIGRATIONS: readonly DefaultMigration[] = [{ filename: REQUEST_ATTEMPTS_FILENAME, module: requestAttempts },{ filename: MODEL_SOURCE_PROVENANCE_FILENAME, module: modelSourceProvenance },{ filename: MEDIA_MODEL_META_FILENAME, module: mediaModelMeta },+ { filename: REQUEST_SERVED_MODEL_FILENAME, module: requestServedModel },+ { filename: ATTEMPT_ERROR_SUMMARY_FILENAME, module: attemptErrorSummary },];diff --git a/server/src/db/migrations/20260726_000005_request_served_model.ts b/server/src/db/migrations/20260726_000005_request_served_model.tsnew file mode 100644index 00000000..69113b44--- /dev/null+++ b/server/src/db/migrations/20260726_000005_request_served_model.ts@@ -0,0 +1,35 @@+// Migration: requests.served_model — upstream-reported model when it drifts+// Created: 2026-07-26+//+// DOWN: reversible+//+// The proxy overwrites the upstream `model` field with the routed id before+// the caller sees it (the gateway contract), which also destroyed the only+// evidence when a provider silently serves a DIFFERENT model than requested+// (#534 — verified live: kilo-auto/small serving Gemma). The served-model+// guard (lib/served-model.ts) now captures the raw upstream value before the+// overwrite; this column persists it — but ONLY when it genuinely differs+// from the routed model id after cosmetic normalization (case, provider+// prefix, ":free"-style tier suffixes). NULL means "matched (or upstream+// reported nothing usable)", which keeps the column empty in the healthy case+// and makes drifted rows trivially queryable:+// SELECT ... FROM requests WHERE served_model IS NOT NULL;++import type { Db } from '../types.js';++function hasColumn(db: Db, table: string, column: string): boolean {+ const columns = db.prepare(`PRAGMA table_info(${table})`).all() as { name: string }[];+ return columns.some((candidate) => candidate.name === column);+}++export function up(db: Db): void {+ if (!hasColumn(db, 'requests', 'served_model')) {+ db.prepare('ALTER TABLE requests ADD COLUMN served_model TEXT').run();+ }+}++export function down(db: Db): void {+ if (hasColumn(db, 'requests', 'served_model')) {+ db.prepare('ALTER TABLE requests DROP COLUMN served_model').run();+ }+}diff --git a/server/src/db/migrations/20260726_000006_attempt_error_summary.ts b/server/src/db/migrations/20260726_000006_attempt_error_summary.tsnew file mode 100644index 00000000..598d1792--- /dev/null+++ b/server/src/db/migrations/20260726_000006_attempt_error_summary.ts@@ -0,0 +1,33 @@+// Migration: request_attempts.error_summary — per-hop error text for the ladder+// Created: 2026-07-26+//+// DOWN: reversible+//+// The failover-ladder drill-down (issue #335) renders WHY each hop failed, but+// the `outcome` class alone loses the provider's actual words ("rate limit+// reached for llama-3.3-70b, retry in 7m12s" collapses to 'rate_limited').+// This nullable column stores a short, REDACTED summary of the per-attempt+// error — sanitized through lib/error-redaction.ts (same rules as the error+// text on the parent `requests` row: keys/tokens/URLs scrubbed) and capped at+// 200 chars at write time (lib/fallback-loop.ts). NULL for successful hops+// ('ok'/'committed') and for every pre-existing row, where the detail is+// simply unknown.++import type { Db } from '../types.js';++function hasColumn(db: Db, table: string, column: string): boolean {+ const columns = db.prepare(`PRAGMA table_info(${table})`).all() as { name: string }[];+ return columns.some((candidate) => candidate.name === column);+}++export function up(db: Db): void {+ if (!hasColumn(db, 'request_attempts', 'error_summary')) {+ db.prepare('ALTER TABLE request_attempts ADD COLUMN error_summary TEXT').run();+ }+}++export function down(db: Db): void {+ if (hasColumn(db, 'request_attempts', 'error_summary')) {+ db.prepare('ALTER TABLE request_attempts DROP COLUMN error_summary').run();+ }+}diff --git a/server/src/lib/attempt-trace.ts b/server/src/lib/attempt-trace.tsindex c84360d3..adcfcfdf 100644--- a/server/src/lib/attempt-trace.ts+++ b/server/src/lib/attempt-trace.ts@@ -40,6 +40,10 @@ export interface AttemptTraceRecord {// Milliseconds this attempt ran (for 'ok' streams: until the response// finished, i.e. including streaming time).durationMs: number;+ // Short, REDACTED summary of the error that ended this attempt (see+ // lib/error-redaction.ts summarizeAttemptError — secrets scrubbed, capped at+ // 200 chars). Null for successful hops ('ok'/'committed').+ errorSummary: string | null;}export interface RequestTrace {diff --git a/server/src/lib/error-classify.ts b/server/src/lib/error-classify.tsindex 6208ebfb..b1c02005 100644--- a/server/src/lib/error-classify.ts+++ b/server/src/lib/error-classify.ts@@ -96,6 +96,26 @@ export function isRetryableError(err: any): boolean {|| msg.includes('unparseable inline tool-call dialect');}+// A genuine provider QUOTA signal: a structured 429 or rate-limit/quota wording.+// Distinct from the much broader isRetryableError: timeouts, 5xx, transport+// failures and dead-turn classes are retryable but say nothing about quotas.+// The null-limits exhaustion heuristic in getCooldownDecisionForLimit (services/+// ratelimit.ts) keys off this: only an actual quota signal may feed its+// "effectively daily-exhausted" escalation ladder. Before this split, a slow+// local Ollama route timing out twice classified retryable → recorded as a+// null-limit "429" → escalated 2m→10m→1h→24h and 429'd every request while the+// server was merely busy generating (#592).+// Mirrors the other classifiers: trust the structured status when the adapter+// attached one (providerHttpError sets err.status everywhere); fall back to+// message markers only when there is no status.+export function isRateLimitSignal(err: any): boolean {+ const status = typeof err?.status === 'number' ? err.status : 0;+ if (status !== 0) return status === 429;+ const msg = (err?.message ?? '').toLowerCase();+ return msg.includes('429') || msg.includes('rate limit') || msg.includes('too many requests')+ || msg.includes('quota') || msg.includes('resource_exhausted');+}+// ── Client-caused aborts ─────────────────────────────────────────────────────// When the gateway's OWN client hangs up, the proxy surfaces abort the composed// fetch signal with this marked error, and undici rejects the in-flight fetch /diff --git a/server/src/lib/error-redaction.ts b/server/src/lib/error-redaction.tsindex c66a5df0..4ed3b26c 100644--- a/server/src/lib/error-redaction.ts+++ b/server/src/lib/error-redaction.ts@@ -32,3 +32,20 @@ export function sanitizeProviderErrorMessage(message: unknown): string {return sanitized;}++// Cap for the per-hop `request_attempts.error_summary` column: one short line+// per failover attempt, tighter than the parent row's error text.+const MAX_ATTEMPT_ERROR_SUMMARY_LENGTH = 200;++/**+ * The short per-attempt error summary the failover ladder stores per hop:+ * the same secret/URL redactions as sanitizeProviderErrorMessage, re-capped+ * at 200 chars so the drill-down stays one line per attempt.+ */+export function summarizeAttemptError(message: unknown): string {+ let summary = sanitizeProviderErrorMessage(message);+ if (summary.length > MAX_ATTEMPT_ERROR_SUMMARY_LENGTH) {+ summary = `${summary.slice(0, MAX_ATTEMPT_ERROR_SUMMARY_LENGTH - 3).trimEnd()}...`;+ }+ return summary;+}diff --git a/server/src/lib/fallback-loop.ts b/server/src/lib/fallback-loop.tsindex f02b90b6..861c342a 100644--- a/server/src/lib/fallback-loop.ts+++ b/server/src/lib/fallback-loop.ts@@ -30,6 +30,7 @@ import {} from '../services/ratelimit.js';import {isRetryableError,+ isRateLimitSignal,isKeyAuthError,isClientAbortError,isDailyQuotaExhaustedError,@@ -40,7 +41,7 @@ import {isProviderDegradedError,isContextTooLargeError,} from './error-classify.js';-import { sanitizeProviderErrorMessage } from './error-redaction.js';+import { sanitizeProviderErrorMessage, summarizeAttemptError } from './error-redaction.js';import { checkKeyHealth, markKeyHealthyFromRequest } from '../services/health.js';import { getSetting } from '../db/index.js';import { newBreaker, recordBreakerFailure } from './guardrails.js';@@ -142,6 +143,10 @@ export function cooldownDecisionForError(route: RouteResult, err: any): Cooldownroute.keyId,{ rpd: route.rpdLimit, tpd: route.tpdLimit },err?.retryAfterMs,+ // Only a real 429/rate-limit error may feed the null-limits exhaustion+ // heuristic; a timeout or 5xx is retryable but says nothing about quota,+ // so it stays on the short transient bench instead of the ladder (#592).+ { quotaSignal: isRateLimitSignal(err) },);}@@ -813,9 +818,12 @@ async function runFallbackLoopAttempts(hooks: FallbackHooks, trace: RequestTrace// Per-attempt trace record: pushed exactly once per dispatched attempt, on// whichever exit the attempt takes. startOffsetMs/durationMs bracket the// dispatch (for a successful stream, durationMs runs until the response- // finished — that IS the attempt).+ // finished — that IS the attempt). When the attempt ended on an error, a+ // short REDACTED summary of it rides along — the outcome class alone loses+ // the provider's actual words, which is exactly what the dashboard's+ // drill-down needs to answer "why did this hop fail".const attemptStartedAt = Date.now();- const traceAttempt = (outcome: AttemptOutcome): void => {+ const traceAttempt = (outcome: AttemptOutcome, err?: any): void => {trace.records.push({ordinal: trace.records.length,platform: route.platform,@@ -824,6 +832,7 @@ async function runFallbackLoopAttempts(hooks: FallbackHooks, trace: RequestTraceoutcome,startOffsetMs: attemptStartedAt - startedAt,durationMs: Date.now() - attemptStartedAt,+ errorSummary: err != null ? summarizeAttemptError(err?.message) : null,});};@@ -858,7 +867,7 @@ async function runFallbackLoopAttempts(hooks: FallbackHooks, trace: RequestTrace// it immediately instead of 502-ing while healthy routes sit idle.recordAuthFailure(route, hooks.state);attempts.push({ platform: route.platform, modelId: route.modelId, keyOrdinal: keyOrdinal(route), errorClass: 'auth' });- traceAttempt('auth');+ traceAttempt('auth', err);lastError = err;if (stopIfBreakerTripped()) return;continue;@@ -867,7 +876,7 @@ async function runFallbackLoopAttempts(hooks: FallbackHooks, trace: RequestTracerecordRetryableFailure(route, err, hooks.state);const errorClass = classifyAttemptError(err);attempts.push({ platform: route.platform, modelId: route.modelId, keyOrdinal: keyOrdinal(route), errorClass });- traceAttempt(errorClass);+ traceAttempt(errorClass, err);lastError = err;// skipBench failures (format ignored, hidden-reasoning truncation) are// model behavior, not provider health — recordRetryableFailure already@@ -877,7 +886,7 @@ async function runFallbackLoopAttempts(hooks: FallbackHooks, trace: RequestTraceif (err?.skipBench !== true && stopIfBreakerTripped()) return;continue;}- traceAttempt(classifyAttemptError(err));+ traceAttempt(classifyAttemptError(err), err);hooks.onFatal(route, err, attempt);return;}@@ -896,7 +905,7 @@ async function runFallbackLoopAttempts(hooks: FallbackHooks, trace: RequestTrace);console.error('[FallbackLoop]', violation.message);hooks.logFailure(route, violation, attempt);- traceAttempt('error');+ traceAttempt('error', violation);hooks.onFatal(route, violation, attempt);return;}diff --git a/server/src/lib/request-log.ts b/server/src/lib/request-log.tsindex a108e1b7..f97c7776 100644--- a/server/src/lib/request-log.ts+++ b/server/src/lib/request-log.ts@@ -55,6 +55,11 @@ export function logRequest(// analytics split pinned vs auto traffic and detect failover overrides// (requested_model set but != model_id).requestedModel: string | null = null,+ // The model the UPSTREAM claims it served, ONLY when it genuinely differs+ // from the routed model_id after cosmetic normalization (#534 — see+ // lib/served-model.ts). NULL when it matches or the provider reported+ // nothing usable, so the column stays empty in the healthy case.+ servedModel: string | null = null,) {try {const db = getDb();@@ -63,9 +68,9 @@ export function logRequest(const client = getClientContext();const tx = db.transaction(() => {const insert = db.prepare(`- INSERT INTO requests (platform, model_id, key_id, status, input_tokens, output_tokens, latency_ms, error, ttfb_ms, requested_model, client_ip, client_user_agent)- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)- `).run(platform, modelId, keyId, status, inputTokens, outputTokens, latencyMs, error, ttfbMs, requestedModel, client.ip, client.userAgent);+ INSERT INTO requests (platform, model_id, key_id, status, input_tokens, output_tokens, latency_ms, error, ttfb_ms, requested_model, served_model, client_ip, client_user_agent)+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)+ `).run(platform, modelId, keyId, status, inputTokens, outputTokens, latencyMs, error, ttfbMs, requestedModel, servedModel, client.ip, client.userAgent);// Report the row id back to the fallback loop's attempt trace (if one is// active): the LAST id noted during a loop run is the terminal row the@@ -116,12 +121,12 @@ export function persistRequestAttempts(trace: RequestTrace): void {try {const db = getDb();const insert = db.prepare(`- INSERT INTO request_attempts (request_id, ordinal, platform, model_id, key_ordinal, outcome, start_offset_ms, duration_ms)- VALUES (?, ?, ?, ?, ?, ?, ?, ?)+ INSERT INTO request_attempts (request_id, ordinal, platform, model_id, key_ordinal, outcome, start_offset_ms, duration_ms, error_summary)+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`);const tx = db.transaction(() => {for (const r of trace.records) {- insert.run(trace.lastRequestRowId, r.ordinal, r.platform, r.modelId, r.keyOrdinal, r.outcome, r.startOffsetMs, r.durationMs);+ insert.run(trace.lastRequestRowId, r.ordinal, r.platform, r.modelId, r.keyOrdinal, r.outcome, r.startOffsetMs, r.durationMs, r.errorSummary);}});tx();diff --git a/server/src/lib/served-model.ts b/server/src/lib/served-model.tsnew file mode 100644index 00000000..aaa83d6aBinary files /dev/null and b/server/src/lib/served-model.ts differdiff --git a/server/src/lib/url-guard.ts b/server/src/lib/url-guard.tsindex 6dd24d56..df41f032 100644--- a/server/src/lib/url-guard.ts+++ b/server/src/lib/url-guard.ts@@ -131,6 +131,35 @@ export function classifyIp(ip: string): AddressClass {return 'public';}+/**+ * Synchronous locality check for a stored provider base_url: true when the URL+ * points at THIS machine or the LAN (loopback, RFC1918/ULA private, 'localhost').+ * Used by the rate limiter to exempt local inference servers (Ollama/llama.cpp/+ * LM Studio via the 'custom' platform) from cloud-quota cooldown ladders (#592):+ * a local box has no quota, so long benches only strand the user's one route.+ *+ * Deliberately DNS-free so it can sit on the cooldown hot path: literal IPs and+ * 'localhost'/'*.localhost' are decidable synchronously; any other hostname+ * (LAN mDNS names included) is pragmatically treated as NON-local — the worst+ * case there is the pre-existing conservative bench, never a wrong exemption.+ */+export function isLoopbackOrPrivateUrl(rawUrl: string | null | undefined): boolean {+ if (!rawUrl) return false;+ let url: URL;+ try {+ url = new URL(rawUrl);+ } catch {+ return false;+ }+ const hostname = url.hostname.replace(/^\[|\]$/g, '').toLowerCase();+ if (hostname === 'localhost' || hostname.endsWith('.localhost')) return true;+ if (net.isIP(hostname)) {+ const cls = classifyIp(hostname);+ return cls === 'loopback' || cls === 'private';+ }+ return false;+}+export interface UrlAssessment {allowed: boolean;reason?: string;diff --git a/server/src/routes/analytics.ts b/server/src/routes/analytics.tsindex 6a0000e2..520c1001 100644--- a/server/src/routes/analytics.ts+++ b/server/src/routes/analytics.ts@@ -467,11 +467,34 @@ analyticsRouter.get('/requests', (req: Request, res: Response) => {const since = getSinceTimestamp(range);const limit = Math.min(Math.max(parseInt(req.query.limit as string, 10) || 100, 1), 500);const offset = Math.max(parseInt(req.query.offset as string, 10) || 0, 0);++ // Optional filters. Both are validated (whitelist / shape) and applied as+ // bound parameters; absent filters keep the default behavior identical.+ const status = req.query.status as string | undefined;+ if (status !== undefined && status !== 'success' && status !== 'error') {+ res.status(400).json({ error: "invalid status filter (expected 'success' or 'error')" });+ return;+ }+ // Platform ids are short slugs ('groq', 'pt-custom_1'); anything else is a+ // client bug, not a filter.+ const platform = req.query.platform as string | undefined;+ if (platform !== undefined && !/^[A-Za-z0-9_-]{1,64}$/.test(platform)) {+ res.status(400).json({ error: 'invalid platform filter' });+ return;+ }const db = getDb();+ const filterSql =+ (status !== undefined ? ' AND status = ?' : '') ++ (platform !== undefined ? ' AND platform = ?' : '');+ const filterParams = [+ ...(status !== undefined ? [status] : []),+ ...(platform !== undefined ? [platform] : []),+ ];+const total = (db.prepare(- 'SELECT COUNT(*) as c FROM requests WHERE created_at >= ?'- ).get(since) as { c: number }).c;+ `SELECT COUNT(*) as c FROM requests WHERE created_at >= ?${filterSql}`+ ).get(since, ...filterParams) as { c: number }).c;const rows = db.prepare(`SELECT id, platform, model_id, requested_model, request_type, status,@@ -480,10 +503,10 @@ analyticsRouter.get('/requests', (req: Request, res: Response) => {strftime('%Y-%m-%dT%H:%M:%SZ', created_at) as created_at_iso,(SELECT COUNT(*) FROM request_attempts a WHERE a.request_id = requests.id) as attempt_countFROM requests- WHERE created_at >= ?+ WHERE created_at >= ?${filterSql}ORDER BY created_at DESC, id DESCLIMIT ? OFFSET ?- `).all(since, limit, offset) as any[];+ `).all(since, ...filterParams, limit, offset) as any[];res.json({total,@@ -524,7 +547,7 @@ analyticsRouter.get('/requests/:id', (req: Request, res: Response) => {const db = getDb();const r = db.prepare(`- SELECT id, platform, model_id, requested_model, request_type, status,+ SELECT id, platform, model_id, requested_model, served_model, request_type, status,input_tokens, output_tokens, latency_ms, ttfb_ms, error,client_ip, client_user_agent,strftime('%Y-%m-%dT%H:%M:%SZ', created_at) as created_at_iso@@ -537,7 +560,7 @@ analyticsRouter.get('/requests/:id', (req: Request, res: Response) => {}const attempts = db.prepare(`- SELECT ordinal, platform, model_id, key_ordinal, outcome, start_offset_ms, duration_ms+ SELECT ordinal, platform, model_id, key_ordinal, outcome, start_offset_ms, duration_ms, error_summaryFROM request_attemptsWHERE request_id = ?ORDER BY ordinal ASC@@ -548,6 +571,9 @@ analyticsRouter.get('/requests/:id', (req: Request, res: Response) => {platform: r.platform,modelId: r.model_id,requestedModel: r.requested_model,+ // Upstream-reported model when it genuinely differed from the routed+ // model_id (#534 served-model drift guard); null in the healthy case.+ servedModel: r.served_model,requestType: r.request_type,status: r.status,inputTokens: r.input_tokens,@@ -566,6 +592,9 @@ analyticsRouter.get('/requests/:id', (req: Request, res: Response) => {outcome: a.outcome,startOffsetMs: a.start_offset_ms,durationMs: a.duration_ms,+ // Short, redacted per-hop error text (null for successful hops and for+ // rows written before the error_summary migration).+ errorSummary: a.error_summary ?? null,})),});});diff --git a/server/src/routes/proxy.ts b/server/src/routes/proxy.tsindex e3f21b6e..ab919fee 100644--- a/server/src/routes/proxy.ts+++ b/server/src/routes/proxy.ts@@ -17,6 +17,7 @@ import { getContextHandoffMode, recordIncomingMessages, maybeInjectContextHandofimport { isFusionModel, runFusion, fusionConfigSchema, FusionError, FUSION_MODEL_ID } from '../services/fusion.js';import { isRetryableError, isPaymentRequiredError, isModelNotFoundError, isModelAccessForbiddenError, isClientAbortError, newClientAbortError } from '../lib/error-classify.js';import { logRequest } from '../lib/request-log.js';+import { observeServedModel } from '../lib/served-model.js';import { parseCacheDirective, cacheActive, isCacheableTemperature, computeCacheKey, getCachedResponse, storeCachedResponse } from '../services/cache.js';import { runFallbackLoop, newFallbackState, recordUpstreamSuccess, exhaustedRetryError, setFallbackHeaders, exhaustionErrorPayload, setExhaustionHeaders, type AttemptRecord } from '../lib/fallback-loop.js';import { applyTokenBudget, tokenBudgetMessage } from '../lib/guardrails.js';@@ -1699,6 +1700,11 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {let upstreamFinish: string | null = null;let usageChunk: unknown = null;let lastMeta: { id?: string; model?: string; created?: number } = {};+ // Raw upstream-reported model, captured off the first frame that+ // carries one — BEFORE the per-frame overwrite below destroys it.+ // Only evidence when a provider serves a different model than routed+ // (#534); compared/persisted on success via observeServedModel.+ let upstreamModel: string | null = null;const flushHeaders = () => {if (headerSent) return;@@ -1735,6 +1741,10 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {// the literal model name "default" even when a concrete model was// requested. Normalize every streamed frame at the proxy boundary// so clients consistently see the model that was actually routed.+ const rawChunkModel = (chunk as Record<string, any>).model;+ if (upstreamModel == null && typeof rawChunkModel === 'string' && rawChunkModel.length > 0) {+ upstreamModel = rawChunkModel;+ }const anyChunk: Record<string, any> = { ...(chunk as Record<string, any>), model: route.modelId };// In-band upstream error frame (observed live: Groq emits@@ -1912,7 +1922,8 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {inputTokens: estimatedInputTokens + injectedHandoffTokens,outputTokens: totalOutputTokens,});- logRequest(route.platform, route.modelId, route.keyId, 'success', estimatedInputTokens + injectedHandoffTokens, totalOutputTokens, Date.now() - start, null, ttfbMs, pinnedModelId);+ logRequest(route.platform, route.modelId, route.keyId, 'success', estimatedInputTokens + injectedHandoffTokens, totalOutputTokens, Date.now() - start, null, ttfbMs, pinnedModelId,+ observeServedModel({ platform: route.platform, requestedModel: route.modelId, servedModel: upstreamModel }));return 'done';} catch (streamErr: any) {// Client abort mid-stream: the pump's own `if (clientGone) break`@@ -1953,6 +1964,14 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {quotaContextForRoute(route, 'chat/completions'),);+ // Raw upstream-reported model, captured BEFORE the contract overwrite+ // below destroys it — the only evidence when a provider silently+ // serves a different model than requested (#534). The OpenAI-compat,+ // cohere, and cloudflare adapters pass the upstream body through, so+ // result.model here is still the provider's own claim; google/aihorde+ // synthesize their responses with the routed id (no upstream signal).+ const upstreamModel = typeof result.model === 'string' ? result.model : null;+// Upstream `model` fields are provider-controlled and can be a generic// placeholder such as Reka's "default". The gateway contract exposes// the concrete routed model, consistently across every provider.@@ -2091,7 +2110,8 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {inputTokens: promptTokens,outputTokens: completionTokens,});- logRequest(route.platform, route.modelId, route.keyId, 'success', promptTokens, completionTokens, Date.now() - start, null, null, pinnedModelId);+ logRequest(route.platform, route.modelId, route.keyId, 'success', promptTokens, completionTokens, Date.now() - start, null, null, pinnedModelId,+ observeServedModel({ platform: route.platform, requestedModel: route.modelId, servedModel: upstreamModel }));return 'done';}},diff --git a/server/src/services/declarative-config.ts b/server/src/services/declarative-config.tsindex 2a082810..c364e732 100644--- a/server/src/services/declarative-config.ts+++ b/server/src/services/declarative-config.ts@@ -95,6 +95,8 @@ export interface DeclarativeConfigResult {models: number;fallback: number;routing: boolean;+ /** Per-entry problems that were degraded to a skip instead of failing the apply (#600). */+ warnings: string[];}interface NormalizedCustomModel {@@ -125,6 +127,25 @@ function encryptedKey(raw: string) {return { encrypted, iv, authTag };}+// Boot-time skip guard for `keys` entries (#600). When a platform stops being+// keyless (pollinations lost `keyless: true` in #573), a legacy declarative+// config still carries an entry with no `key` — applyDeclarativeConfigFromEnv()+// runs in main(), so throwing here used to brick the whole install at startup.+// Such entries degrade to a warning + skip; the rest of the config still+// applies. Platform-specific remediation hints live here.+const MISSING_KEY_HINTS: Record<string, string> = {+ pollinations: 'pollinations now requires an API key — get one at enter.pollinations.ai',+};++function missingKeyWarning(input: z.infer<typeof keySchema>): string | null {+ const platform = input.platform.trim();+ if (platform === 'custom' || input.key?.trim()) return null;+ const provider = resolveProvider(platform as never);+ if (!provider || provider.keyless) return null;+ const hint = MISSING_KEY_HINTS[platform] ?? `${platform} requires an API key — add "key" to this entry or remove it`;+ return `${hint}; entry skipped`;+}+function upsertApiKey(db: Db, input: z.infer<typeof keySchema>): number {const platform = input.platform.trim();const enabled = input.enabled === false ? 0 : 1;@@ -361,10 +382,17 @@ export function applyDeclarativeConfig(input: unknown, source = 'inline'): Declamodels: 0,fallback: 0,routing: false,+ warnings: [],};const apply = db.transaction(() => {for (const key of parsed.data.keys ?? []) {+ const warning = missingKeyWarning(key);+ if (warning) {+ result.warnings.push(warning);+ console.warn(`[config] ${warning}`);+ continue;+ }upsertApiKey(db, key);result.keys++;}@@ -391,12 +419,13 @@ export function applyDeclarativeConfig(input: unknown, source = 'inline'): Declaexport function applyDeclarativeConfigFromEnv(): DeclarativeConfigResult {const loaded = readConfigFromEnv();if (!loaded) {- return { applied: false, keys: 0, customModels: 0, models: 0, fallback: 0, routing: false };+ return { applied: false, keys: 0, customModels: 0, models: 0, fallback: 0, routing: false, warnings: [] };}const result = applyDeclarativeConfig(loaded.value, loaded.source);console.log(`[config] applied ${loaded.source}: ${result.keys} keys, ${result.customModels} custom models, ` +- `${result.models} model edits, ${result.fallback} fallback rows${result.routing ? ', routing' : ''}`,+ `${result.models} model edits, ${result.fallback} fallback rows${result.routing ? ', routing' : ''}` ++ `${result.warnings.length > 0 ? `, ${result.warnings.length} entries skipped` : ''}`,);return result;}diff --git a/server/src/services/fusion.ts b/server/src/services/fusion.tsindex 8cb2627c..d543baf8 100644--- a/server/src/services/fusion.ts+++ b/server/src/services/fusion.ts@@ -11,7 +11,7 @@ import {} from './ratelimit.js';import { logRequest } from '../lib/request-log.js';import {- isRetryableError, isPaymentRequiredError,+ isRetryableError, isRateLimitSignal, isPaymentRequiredError,isModelNotFoundError, isModelAccessForbiddenError,} from '../lib/error-classify.js';import { contentToString } from '../lib/content.js';@@ -265,7 +265,7 @@ async function runModelCall(? { durationMs: PAYMENT_REQUIRED_COOLDOWN_MS, source: 'credit' as const }: isModelAccessForbiddenError(err)? { durationMs: MODEL_FORBIDDEN_COOLDOWN_MS, source: 'tier' as const }- : getCooldownDecisionForLimit(route.platform, route.modelId, route.keyId, { rpd: route.rpdLimit, tpd: route.tpdLimit }, err.retryAfterMs);+ : getCooldownDecisionForLimit(route.platform, route.modelId, route.keyId, { rpd: route.rpdLimit, tpd: route.tpdLimit }, err.retryAfterMs, { quotaSignal: isRateLimitSignal(err) });setCooldown(route.platform, route.modelId, route.keyId, decision.durationMs, decision.source);recordRateLimitHit(route.modelDbId);continue;@@ -355,7 +355,7 @@ async function runJudgeStreaming(? { durationMs: PAYMENT_REQUIRED_COOLDOWN_MS, source: 'credit' as const }: isModelAccessForbiddenError(err)? { durationMs: MODEL_FORBIDDEN_COOLDOWN_MS, source: 'tier' as const }- : getCooldownDecisionForLimit(route.platform, route.modelId, route.keyId, { rpd: route.rpdLimit, tpd: route.tpdLimit }, err.retryAfterMs);+ : getCooldownDecisionForLimit(route.platform, route.modelId, route.keyId, { rpd: route.rpdLimit, tpd: route.tpdLimit }, err.retryAfterMs, { quotaSignal: isRateLimitSignal(err) });setCooldown(route.platform, route.modelId, route.keyId, decision.durationMs, decision.source);recordRateLimitHit(route.modelDbId);continue;diff --git a/server/src/services/ratelimit.ts b/server/src/services/ratelimit.tsindex b8fc48bc..aead6fd7 100644--- a/server/src/services/ratelimit.ts+++ b/server/src/services/ratelimit.ts@@ -1,6 +1,7 @@// Sliding window rate limit tracker with SQLite persistence.import { getDb } from '../db/index.js';+import { isLoopbackOrPrivateUrl } from '../lib/url-guard.js';interface Window {timestamps: number[];@@ -578,6 +579,7 @@ export function recordRequest(platform: string, modelId: string, keyId: number)recordUsage(platform, modelId, keyId, 'request', 0, now);clearNullLimitHits(platform, modelId, keyId);+ clearCooldownHits(platform, modelId, keyId);}export function recordTokens(@@ -621,6 +623,12 @@ export type CooldownSource = 'heuristic' | 'authoritative' | 'credit' | 'tier';// the 2-minute cooldown 20 times per request and consuming every fallback slot.// In-memory only — state resets on restart, which is fine (a clean restart// will re-escalate on the next 429 if the quota is genuinely exhausted).+// A successful request clears the counter (recordRequest → clearCooldownHits),+// same reversibility contract as nullLimitHits: a served request proves the+// quota is NOT exhausted right now, so the next failure starts the ladder over+// instead of inheriting up-to-24h steps from stale hits. While a daily quota is+// genuinely spent no successes can occur (the counters/cooldowns gate routing),+// so the intended escalation-across-the-day behavior is untouched.const cooldownHits = new Map<string, number[]>(); // key -> timestamps of recent cooldown set eventsconst HOUR = 60 * MINUTE;const COOLDOWN_DURATIONS = [@@ -640,6 +648,10 @@ export function getNextCooldownDuration(platform: string, modelId: string, keyIdreturn COOLDOWN_DURATIONS[idx]!;}+function clearCooldownHits(platform: string, modelId: string, keyId: number): void {+ cooldownHits.delete(`${platform}:${modelId}:${keyId}`);+}+// Short cooldown for a transient (per-minute) 429 — recovers within ~one window.const TRANSIENT_COOLDOWN_MS = 90 * 1000;@@ -719,14 +731,63 @@ export interface CooldownDecision {source: CooldownSource;}+// ── Local-endpoint exemption (#592) ──────────────────────────────────────────+// A key whose base_url points at this machine or the LAN (custom-platform+// Ollama / llama.cpp / LM Studio) has no provider quota to protect: every+// cooldown longer than a few seconds just strands what is usually the user's+// ONLY route to that model, turning a slow local generation (timeout) into an+// "All models exhausted" 429. Such keys are capped at this short bench and+// never enter the escalation ladder. Source stays 'heuristic' so the+// cooldown-probe recovery job may clear even that early.+export const LOCAL_ENDPOINT_COOLDOWN_MS = 5_000;++// base_url is immutable per key id (routes/keys.ts matches custom rows ON+// base_url — a new endpoint gets a new row, #212), so the verdict can be+// cached for the life of the process. Built-in platforms have base_url NULL+// → non-local. A DB that isn't ready yet is NOT cached, so a later call can+// still decide properly.+const keyLocality = new Map<number, boolean>(); // keyId -> is local endpoint++export function isLocalEndpointKey(keyId: number): boolean {+ const cached = keyLocality.get(keyId);+ if (cached !== undefined) return cached;+ const baseUrl = withDb(db => {+ const row = db.prepare('SELECT base_url FROM api_keys WHERE id = ?')+ .get(keyId) as { base_url: string | null } | undefined;+ return row?.base_url ?? '';+ });+ if (baseUrl === undefined) return false; // DB not ready — don't cache+ const local = isLoopbackOrPrivateUrl(baseUrl || null);+ keyLocality.set(keyId, local);+ return local;+}++/** Test hook: the locality verdict is cached per key id for the process+ * lifetime (see above); tests that rewrite api_keys rows need a reset. */+export function resetKeyLocalityCache(): void {+ keyLocality.clear();+}++export interface CooldownLimitOptions {+ /** Whether the triggering failure is an actual provider quota signal (a real+ * 429 / rate-limit-classified error — see isRateLimitSignal in+ * lib/error-classify.ts). Timeouts, 5xx and transport errors are retryable+ * but carry no quota information, so they must not feed the null-limits+ * exhaustion heuristic below (#592) — they get the short transient bench.+ * Defaults to true: legacy callers without error context (fusion's+ * empty-completion benches) keep their pre-#592 behavior. */+ quotaSignal?: boolean;+}+export function getCooldownDurationForLimit(platform: string,modelId: string,keyId: number,limits: { rpd: number | null; tpd: number | null },retryAfterMs?: number | null,+ opts?: CooldownLimitOptions,): number {- return getCooldownDecisionForLimit(platform, modelId, keyId, limits, retryAfterMs).durationMs;+ return getCooldownDecisionForLimit(platform, modelId, keyId, limits, retryAfterMs, opts).durationMs;}/**@@ -743,7 +804,18 @@ export function getCooldownDecisionForLimit(keyId: number,limits: { rpd: number | null; tpd: number | null },retryAfterMs?: number | null,+ opts?: CooldownLimitOptions,): CooldownDecision {+ // Local inference endpoint (loopback/RFC1918 base_url): no quota exists, so+ // neither the transient 90s bench nor the escalation ladder applies — a+ // failure there means "busy right now", and a few seconds is all the pool+ // needs before retrying the user's (usually only) local route. Applies even+ // to a literal 429 (local servers emit them under concurrent load) and+ // ignores any Retry-After: both describe momentary busyness, not a quota.+ if (isLocalEndpointKey(keyId)) {+ return { durationMs: LOCAL_ENDPOINT_COOLDOWN_MS, source: 'heuristic' };+ }+ const quotaSignal = opts?.quotaSignal ?? true;const now = Date.now();const rpdExhausted =limits.rpd !== null && requestCount(platform, modelId, keyId, DAY, now) >= limits.rpd;@@ -753,9 +825,12 @@ export function getCooldownDecisionForLimit(// last hour is treated as effectively daily-exhausted. This unsticks// providers that publish no daily cap (ollama, cloudflare, etc.) from the// 90s-cooldown-loop without requiring operator-side limit seeding.+ // Gated on quotaSignal (#592): only a REAL 429/rate-limit error is evidence+ // of quota exhaustion. Timeouts and 5xx used to feed this counter too, so+ // two slow generations on a null-limits route escalated 2m→10m→…→24h.const unknownLimits = limits.rpd === null && limits.tpd === null;let heuristicallyExhausted = false;- if (unknownLimits) {+ if (unknownLimits && quotaSignal) {// The current hit is recorded first so the threshold can be reached across// consecutive 429s, but only for providers where counters cannot decide.recordNullLimitHit(platform, modelId, keyId, now);