7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.
TypeScript
feat(proxy): opt-in X-Fallback-Detail header with per-hop failover timings (#792)
OhOkThisIsFine8cb75acda641
diff --git a/.env.example b/.env.exampleindex 4bd9b257..335e1d82 100644--- a/.env.example+++ b/.env.example@@ -103,6 +103,15 @@ PORT=3001# fallback_time_budget_ms settings key, which takes precedence.# FALLBACK_TIME_BUDGET_MS=45000+# Add X-Fallback-Detail to responses (default: off). X-Fallback-Trail already+# names every hop that failed and why; the detail header adds what each one+# COST — "groq/llama-3.3-70b key1=rate_limited t=0+39000ms msg=..." — so a+# caller can tell one 39s stall apart from four fast failures without opening+# the dashboard. Off by default because it puts hop timings and provider error+# text on the response. Can also be set at runtime via the+# expose_fallback_detail_header settings key, which takes precedence.+# FALLBACK_DETAIL_HEADER=1+# Request guardrails, both OFF (0) by default. Also runtime-tunable via the# request_max_tokens_budget / max_consecutive_upstream_fails settings keys# (dashboard API PUT /api/settings/guardrails), which take precedence.diff --git a/docs/api.md b/docs/api.mdindex 0161a14c..993fe44e 100644--- a/docs/api.md+++ b/docs/api.md@@ -249,7 +249,24 @@ Request the virtual `fusion` model and the router fans your prompt out to a pane## Response headers-Every response carries an `X-Routed-Via: <platform>/<model>` header so you can see which provider actually served each call. If a request fell over between providers, you'll also see `X-Fallback-Attempts: N`.+Every response carries an `X-Routed-Via: <platform>/<model>` header so you can see which provider actually served each call. If a request fell over between providers, you'll also see `X-Fallback-Attempts: N` and `X-Fallback-Trail`, which names each hop that failed and why:++```+X-Fallback-Attempts: 2+X-Fallback-Trail: groq/llama-3.3-70b key1=rate_limited; google/gemini-2.5-flash key2=timeout+```++`X-Fallback-Detail` adds what each of those hops **cost**, which is the part you cannot reconstruct from the trail — a request answered in 40s reads identically whether one provider stalled for 39 seconds or four failed fast:++```+X-Fallback-Detail: groq/llama-3.3-70b key1=rate_limited t=0+39000ms msg=Groq API error 429: rate limit; google/gemini-2.5-flash key2=timeout t=39000+12ms+```++`t=<start>+<duration>ms` is the offset from the start of the failover chain and how long that hop ran. `msg=` is the provider's error, redacted and truncated; it is last in each record, and any semicolon inside it becomes a comma so `; ` stays an unambiguous record separator.++The detail header is **off by default** — it puts hop timings and provider error text on the response. Turn it on with `FALLBACK_DETAIL_HEADER=1` or the `expose_fallback_detail_header` settings key, which takes precedence. Both headers list at most ten hops and then a `; +N more` marker.++Only hops that already **failed** can appear. The hop actually serving your request is recorded after its response finishes — after the JSON is sent, or after the stream closes — so its duration does not exist while the headers are still open. Use `X-Routed-Via` to see which provider that was.HTTP headers only carry printable ASCII, so a model id with characters outside that range (a Chinese name from a relay catalog, for example) is percent-encoded in the header — run the value through `decodeURIComponent` (or `urllib.parse.unquote`) to read it back.diff --git a/server/src/__tests__/routes/fallback-detail-header.test.ts b/server/src/__tests__/routes/fallback-detail-header.test.tsnew file mode 100644index 00000000..6361ebc4--- /dev/null+++ b/server/src/__tests__/routes/fallback-detail-header.test.ts@@ -0,0 +1,299 @@+import { describe, it, expect, beforeAll, beforeEach, afterEach, vi } from 'vitest';+import type { Express } from 'express';++// X-Fallback-Detail: the opt-in companion to X-Fallback-Trail.+//+// The trail already says WHICH hops burned and WHY. What a caller cannot+// reconstruct is what each hop COST: a request answered in 40s reads+// identically whether one provider stalled for 39s or four failed fast. The+// per-hop timings are already collected for request_attempts; this header+// exposes them without a dashboard round trip.++const chatCompletion = vi.fn();+const streamChatCompletion = vi.fn();+const fakeProvider = { name: 'fake', chatCompletion, streamChatCompletion } as any;++vi.mock('../../providers/index.js', async (importOriginal) => {+ const actual = await importOriginal() as any;+ return {+ ...actual,+ getProvider: () => fakeProvider,+ resolveProvider: () => fakeProvider,+ };+});++const { createApp } = await import('../../app.js');+const { initDb, getDb, getUnifiedApiKey, setSetting } = await import('../../db/index.js');+const { encrypt } = await import('../../lib/crypto.js');+const { setRoutingStrategy } = await import('../../services/router.js');+const {+ EXPOSE_FALLBACK_DETAIL_SETTING,+ formatAttemptDetail,+ isFallbackDetailHeaderEnabled,+ setFallbackHeaders,+} = await import('../../lib/fallback-loop.js');+const { newRequestTrace, runWithRequestTrace } = await import('../../lib/attempt-trace.js');++/** Remove the row entirely, so the env-var fallback is actually reachable —+ * storing '0' is a decision, not an absence. */+function clearDetailSetting(): void {+ getDb().prepare('DELETE FROM settings WHERE key = ?').run(EXPOSE_FALLBACK_DETAIL_SETTING);+}++async function post(app: Express, path: string, body: any, key: string) {+ const server = app.listen(0);+ const addr = server.address() as any;+ const res = await fetch(`http://127.0.0.1:${addr.port}${path}`, {+ method: 'POST',+ headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${key}` },+ body: JSON.stringify(body),+ });+ const raw = await res.text();+ server.close();+ let json: any = null;+ try { json = JSON.parse(raw); } catch { /* SSE */ }+ return { status: res.status, body: json, raw, headers: res.headers };+}++const GOOD_RESULT = {+ choices: [{ message: { role: 'assistant', content: 'a real answer' }, finish_reason: 'stop' }],+ usage: { prompt_tokens: 10, completion_tokens: 3, total_tokens: 13 },+};++const record = (over: Partial<Parameters<typeof formatAttemptDetail>[0][number]> = {}) => ({+ ordinal: 0,+ platform: 'groq',+ modelId: 'llama-3.3-70b',+ keyOrdinal: 1,+ outcome: 'rate_limited' as const,+ startOffsetMs: 0,+ durationMs: 11,+ errorSummary: null,+ ...over,+});++describe('formatAttemptDetail', () => {+ it('leads with the same shape as X-Fallback-Trail so the two headers line up', () => {+ expect(formatAttemptDetail([record()])).toBe('groq/llama-3.3-70b key1=rate_limited t=0+11ms');+ });++ it('carries the per-hop timing that the trail cannot express', () => {+ const value = formatAttemptDetail([+ record({ startOffsetMs: 0, durationMs: 39_000 }),+ record({ ordinal: 1, platform: 'google', modelId: 'gemini-2.5-flash', keyOrdinal: 2, outcome: 'timeout', startOffsetMs: 39_000, durationMs: 12 }),+ ]);+ // A 39s stall followed by a fast failure — indistinguishable in the trail.+ expect(value).toBe(+ 'groq/llama-3.3-70b key1=rate_limited t=0+39000ms; ' ++ 'google/gemini-2.5-flash key2=timeout t=39000+12ms',+ );+ });++ it('appends the redacted provider message when there is one', () => {+ const value = formatAttemptDetail([record({ errorSummary: 'Groq API error 429: rate limit' })]);+ expect(value).toBe('groq/llama-3.3-70b key1=rate_limited t=0+11ms msg=Groq API error 429: rate limit');+ });++ it('replaces semicolons in the message so they cannot forge a record boundary', () => {+ const value = formatAttemptDetail([record({ errorSummary: 'first; second' })]);+ expect(value).toBe('groq/llama-3.3-70b key1=rate_limited t=0+11ms msg=first, second');+ expect(value.split('; ')).toHaveLength(1);+ });++ it('truncates a long message rather than letting ten hops blow the header budget', () => {+ const value = formatAttemptDetail([record({ errorSummary: 'x'.repeat(500) })]);+ expect(value).toContain('msg=' + 'x'.repeat(120));+ expect(value).not.toContain('x'.repeat(121));+ });++ it('caps the hop count and says how many it dropped', () => {+ const many = Array.from({ length: 14 }, (_, i) => record({ ordinal: i }));+ const value = formatAttemptDetail(many);+ expect(value.split('; ')).toHaveLength(11); // 10 hops + the "+4 more" marker+ expect(value.endsWith('; +4 more')).toBe(true);+ });++ it('is empty for an empty trace', () => {+ expect(formatAttemptDetail([])).toBe('');+ });+});++describe('isFallbackDetailHeaderEnabled', () => {+ beforeAll(() => {+ process.env.ENCRYPTION_KEY = '0'.repeat(64);+ initDb(':memory:');+ });++ beforeEach(() => {+ clearDetailSetting();+ });++ afterEach(() => {+ delete process.env.FALLBACK_DETAIL_HEADER;+ clearDetailSetting();+ });++ it('is off by default — this widens what the surface reveals', () => {+ expect(isFallbackDetailHeaderEnabled()).toBe(false);+ });++ it('honours the settings-table value', () => {+ setSetting(EXPOSE_FALLBACK_DETAIL_SETTING, '1');+ expect(isFallbackDetailHeaderEnabled()).toBe(true);+ });++ it('falls back to the env var when no setting is stored', () => {+ process.env.FALLBACK_DETAIL_HEADER = 'true';+ expect(isFallbackDetailHeaderEnabled()).toBe(true);+ });++ it('lets the settings table win over the env var, like the failover budget', () => {+ setSetting(EXPOSE_FALLBACK_DETAIL_SETTING, '0');+ process.env.FALLBACK_DETAIL_HEADER = '1';+ expect(isFallbackDetailHeaderEnabled()).toBe(false);+ });+});++describe('X-Fallback-Detail on the wire', () => {+ let app: Express;+ let key: string;++ beforeAll(() => {+ process.env.ENCRYPTION_KEY = '0'.repeat(64);+ initDb(':memory:');+ app = createApp();+ key = getUnifiedApiKey();++ const db = getDb();+ setRoutingStrategy('priority');+ const { encrypted, iv, authTag } = encrypt('detail-header-test-key');+ db.prepare(`+ INSERT INTO api_keys (platform, label, encrypted_key, iv, auth_tag, status, enabled)+ VALUES ('groq', 'detail-header', ?, ?, ?, 'healthy', 1)+ `).run(encrypted, iv, authTag);+ });++ beforeEach(() => {+ chatCompletion.mockReset();+ streamChatCompletion.mockReset();+ getDb().prepare('DELETE FROM requests').run();+ getDb().prepare('DELETE FROM rate_limit_cooldowns').run();+ setSetting(EXPOSE_FALLBACK_DETAIL_SETTING, '1');+ });++ afterEach(() => {+ setSetting(EXPOSE_FALLBACK_DETAIL_SETTING, '0');+ delete process.env.FALLBACK_DETAIL_HEADER;+ });++ it('reports the failed hops with timings alongside the existing trail', async () => {+ chatCompletion+ .mockRejectedValueOnce(Object.assign(new Error('Groq API error 429: rate limit'), { status: 429 }))+ .mockResolvedValueOnce(GOOD_RESULT);++ const { status, headers } = await post(app, '/v1/chat/completions', {+ messages: [{ role: 'user', content: 'detail header test' }],+ }, key);++ expect(status).toBe(200); // failed over and served+ expect(headers.get('x-fallback-attempts')).toBe('1');+ expect(headers.get('x-fallback-trail')).toContain('key1=rate_limited');++ const detail = headers.get('x-fallback-detail');+ expect(detail).toBeTruthy();+ expect(detail).toMatch(/^groq\/\S+ key1=rate_limited t=\d+\+\d+ms msg=/);+ expect(detail).toContain('rate limit');+ });++ it('stays silent when the setting is off', async () => {+ setSetting(EXPOSE_FALLBACK_DETAIL_SETTING, '0');+ chatCompletion+ .mockRejectedValueOnce(Object.assign(new Error('Groq API error 429: rate limit'), { status: 429 }))+ .mockResolvedValueOnce(GOOD_RESULT);++ const { status, headers } = await post(app, '/v1/chat/completions', {+ messages: [{ role: 'user', content: 'detail header off test' }],+ }, key);++ expect(status).toBe(200);+ // The always-on trail is unaffected by the toggle.+ expect(headers.get('x-fallback-attempts')).toBe('1');+ expect(headers.get('x-fallback-detail')).toBeNull();+ });++ it('is absent on a request that never failed over, rather than empty', async () => {+ chatCompletion.mockResolvedValueOnce(GOOD_RESULT);++ const { status, headers } = await post(app, '/v1/chat/completions', {+ messages: [{ role: 'user', content: 'clean request' }],+ }, key);++ expect(status).toBe(200);+ expect(headers.get('x-fallback-detail')).toBeNull();+ });++ it('reaches the caller on the exhaustion path too, not just on success', async () => {+ chatCompletion.mockRejectedValue(+ Object.assign(new Error('Groq API error 429: rate limit'), { status: 429 }),+ );++ const { status, headers } = await post(app, '/v1/chat/completions', {+ messages: [{ role: 'user', content: 'exhaustion detail test' }],+ }, key);++ expect(status).toBe(429);+ const detail = headers.get('x-fallback-detail');+ expect(detail).toBeTruthy();+ expect(detail).toContain('key1=rate_limited');+ expect(detail).toMatch(/t=\d+\+\d+ms/);+ });++});++// Driving setFallbackHeaders directly, rather than steering the router into+// picking a hostile model, keeps these about the header and not about routing.+describe('X-Fallback-Detail value safety', () => {+ function headersFor(records: Parameters<typeof formatAttemptDetail>[0]): Record<string, string> {+ const trace = newRequestTrace();+ trace.records.push(...records);+ const out: Record<string, string> = {};+ setSetting(EXPOSE_FALLBACK_DETAIL_SETTING, '1');+ runWithRequestTrace(trace, () => {+ setFallbackHeaders({ setHeader: (n: string, v: string) => { out[n] = v; } }, records.length, undefined);+ });+ return out;+ }++ afterEach(() => {+ getDb().prepare('DELETE FROM settings WHERE key = ?').run(EXPOSE_FALLBACK_DETAIL_SETTING);+ });++ it('encodes a control-laden model id instead of letting it inject a header line', () => {+ // #619: a non-ASCII or control-laden id used to make Node throw+ // ERR_INVALID_CHAR after a successful upstream call, and the throw was then+ // booked as a provider failure.+ const value = headersFor([record({ modelId: 'bad\r\nX-Injected: yes' })])['X-Fallback-Detail'];++ expect(value).toMatch(/^[\x20-\x7e]*$/);+ expect(value).not.toContain('\r');+ expect(value).not.toContain('\n');+ expect(value).toContain('%0D%0A'); // encoded in place, not dropped+ });++ it('encodes a non-ASCII model id rather than rejecting the response', () => {+ const value = headersFor([record({ modelId: '我的模型' })])['X-Fallback-Detail'];++ expect(value).toMatch(/^[\x20-\x7e]*$/);+ expect(decodeURIComponent(value.slice('groq/'.length).split(' ')[0])).toBe('我的模型');+ });++ it('keeps the whole value inside the header budget even at ten long hops', () => {+ const many = Array.from({ length: 10 }, (_, i) => record({+ ordinal: i,+ modelId: 'm'.repeat(80),+ errorSummary: 'e'.repeat(200),+ }));++ expect(headersFor(many)['X-Fallback-Detail'].length).toBeLessThanOrEqual(2048);+ });+});diff --git a/server/src/lib/fallback-loop.ts b/server/src/lib/fallback-loop.tsindex e775df5c..93b8bd84 100644--- a/server/src/lib/fallback-loop.ts+++ b/server/src/lib/fallback-loop.ts@@ -48,7 +48,7 @@ import { checkKeyHealth, markKeyHealthyFromRequest } from '../services/health.jsimport { noteModelRetirementSignal } from '../services/model-retirement.js';import { getSetting } from '../db/index.js';import { newBreaker, recordBreakerFailure } from './guardrails.js';-import { getRequestTrace, newRequestTrace, runWithRequestTrace, type AttemptOutcome, type RequestTrace } from './attempt-trace.js';+import { getRequestTrace, newRequestTrace, runWithRequestTrace, type AttemptOutcome, type AttemptTraceRecord, type RequestTrace } from './attempt-trace.js';import { logRequest, persistRequestAttempts } from './request-log.js';// Every surface caps failover hops at the same number.@@ -361,6 +361,69 @@ export function classifyAttemptError(err: any): AttemptErrorClass {const TRAIL_MAX_SHOWN = 10;const TRAIL_HEADER_MAX_LENGTH = 1024;+// ── Detailed failover trace header (opt-in) ──────────────────────────────────+// X-Fallback-Trail already tells a caller WHICH hops burned and WHY, but not+// how long they cost. That is the part an agent cannot reconstruct: a request+// answered in 40s reads identically whether one provider stalled for 39s or+// four failed fast. The per-hop timings and the redacted provider message are+// already collected for the request_attempts table; this exposes them on the+// response so the caller sees them without a dashboard round trip.+//+// Off by default. It widens what an already-loopback-ish surface reveals —+// hop timings plus provider error text — so it is opt-in like the discovery+// aliases, not something a default install starts emitting. Precedence matches+// the failover budget above: settings-table value, then env var, then off.+export const EXPOSE_FALLBACK_DETAIL_SETTING = 'expose_fallback_detail_header';++// Ten hops of "platform/model keyN=class t=…+…ms msg=…" with a capped message+// each. Kept well under the 8 KB total header budget that proxies commonly+// enforce, since this rides alongside X-Fallback-Trail's own 1 KB.+const DETAIL_HEADER_MAX_LENGTH = 2048;+// summarizeAttemptError caps at 200; the header wants a tighter budget so ten+// hops still fit. The full text remains in request_attempts either way.+const DETAIL_SUMMARY_MAX_LENGTH = 120;++export function isFallbackDetailHeaderEnabled(): boolean {+ let stored: string | undefined;+ try {+ stored = getSetting(EXPOSE_FALLBACK_DETAIL_SETTING);+ } catch {+ stored = undefined; // DB not ready — never throw on the proxy hot path+ }+ for (const raw of [stored, process.env.FALLBACK_DETAIL_HEADER]) {+ if (raw === undefined || raw.trim() === '') continue;+ const value = raw.trim().toLowerCase();+ return value === '1' || value === 'true';+ }+ return false;+}++/**+ * One `platform/model keyN=outcome t=<start>+<duration>ms msg=<summary>` segment+ * per hop, `; `-joined — the same leading shape as X-Fallback-Trail so the two+ * headers line up when read together.+ *+ * The summary is the already-redacted `errorSummary`, never `err.message`. Its+ * semicolons become commas because `; ` is the record separator; nothing else+ * is escaped, which keeps the value readable, and `safeHeaderValue` handles any+ * non-ASCII on the way out.+ */+export function formatAttemptDetail(records: AttemptTraceRecord[]): string {+ const shown = records.slice(0, TRAIL_MAX_SHOWN).map(r => {+ const parts = [+ `${r.platform}/${r.modelId}`,+ `key${r.keyOrdinal}=${r.outcome}`,+ `t=${r.startOffsetMs}+${r.durationMs}ms`,+ ];+ if (r.errorSummary) {+ parts.push(`msg=${r.errorSummary.slice(0, DETAIL_SUMMARY_MAX_LENGTH).replace(/;/g, ',')}`);+ }+ return parts.join(' ');+ });+ const extra = records.length - shown.length;+ return shown.join('; ') + (extra > 0 ? `; +${extra} more` : '');+}+export function formatAttemptTrail(attempts: AttemptRecord[]): string {const shown = attempts.slice(0, TRAIL_MAX_SHOWN)@@ -378,6 +441,17 @@ export function formatAttemptTrail(attempts: AttemptRecord[]): string {* exactly the case an operator wants to notice. Values go through* safeHeaderValue so a non-ASCII or control-laden model id can neither inject* header lines nor make Node reject the response outright (#619).+ *+ * When EXPOSE_FALLBACK_DETAIL_SETTING is on, X-Fallback-Detail joins them with+ * per-hop timings and the redacted provider message. Every caller runs inside+ * the request's AsyncLocalStorage scope, so the trace is readable here without+ * threading it through all five surfaces.+ *+ * Note what the detail header can and cannot contain: at flush time the trace+ * holds exactly the hops that already FAILED, each with final timings. The hop+ * currently being served is recorded only after dispatch returns — after+ * res.json(), or after the whole stream has finished — so its duration is not+ * knowable while headers are still open, on either path.*/export function setFallbackHeaders(res: { setHeader(name: string, value: string): void },@@ -394,6 +468,13 @@ export function setFallbackHeaders(// trail gets its own budget.res.setHeader('X-Fallback-Trail', safeHeaderValue(value, TRAIL_HEADER_MAX_LENGTH));}++ // Checked before the setting so the overwhelmingly common no-failover request+ // never pays for a settings read.+ const records = getRequestTrace()?.records;+ if (records && records.length > 0 && isFallbackDetailHeaderEnabled()) {+ res.setHeader('X-Fallback-Detail', safeHeaderValue(formatAttemptDetail(records), DETAIL_HEADER_MAX_LENGTH));+ }}export interface ExhaustionBody {