Skip to content

7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.

TypeScript

feat(proxy): opt-in X-Fallback-Detail header with per-hop failover timings (#792)

OhOkThisIsFine8cb75acda641
diff --git a/.env.example b/.env.example
index 4bd9b257..335e1d82 100644
--- a/.env.example
+++ b/.env.example
@@ -103,6 +103,15 @@ PORT=3001
# fallback_time_budget_ms settings key, which takes precedence.
# FALLBACK_TIME_BUDGET_MS=45000
+# Add X-Fallback-Detail to responses (default: off). X-Fallback-Trail already
+# names every hop that failed and why; the detail header adds what each one
+# COST — "groq/llama-3.3-70b key1=rate_limited t=0+39000ms msg=..." — so a
+# caller can tell one 39s stall apart from four fast failures without opening
+# the dashboard. Off by default because it puts hop timings and provider error
+# text on the response. Can also be set at runtime via the
+# expose_fallback_detail_header settings key, which takes precedence.
+# FALLBACK_DETAIL_HEADER=1
+
# Request guardrails, both OFF (0) by default. Also runtime-tunable via the
# request_max_tokens_budget / max_consecutive_upstream_fails settings keys
# (dashboard API PUT /api/settings/guardrails), which take precedence.
diff --git a/docs/api.md b/docs/api.md
index 0161a14c..993fe44e 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -249,7 +249,24 @@ Request the virtual `fusion` model and the router fans your prompt out to a pane
## Response headers
-Every response carries an `X-Routed-Via: <platform>/<model>` header so you can see which provider actually served each call. If a request fell over between providers, you'll also see `X-Fallback-Attempts: N`.
+Every response carries an `X-Routed-Via: <platform>/<model>` header so you can see which provider actually served each call. If a request fell over between providers, you'll also see `X-Fallback-Attempts: N` and `X-Fallback-Trail`, which names each hop that failed and why:
+
+```
+X-Fallback-Attempts: 2
+X-Fallback-Trail: groq/llama-3.3-70b key1=rate_limited; google/gemini-2.5-flash key2=timeout
+```
+
+`X-Fallback-Detail` adds what each of those hops **cost**, which is the part you cannot reconstruct from the trail — a request answered in 40s reads identically whether one provider stalled for 39 seconds or four failed fast:
+
+```
+X-Fallback-Detail: groq/llama-3.3-70b key1=rate_limited t=0+39000ms msg=Groq API error 429: rate limit; google/gemini-2.5-flash key2=timeout t=39000+12ms
+```
+
+`t=<start>+<duration>ms` is the offset from the start of the failover chain and how long that hop ran. `msg=` is the provider's error, redacted and truncated; it is last in each record, and any semicolon inside it becomes a comma so `; ` stays an unambiguous record separator.
+
+The detail header is **off by default** — it puts hop timings and provider error text on the response. Turn it on with `FALLBACK_DETAIL_HEADER=1` or the `expose_fallback_detail_header` settings key, which takes precedence. Both headers list at most ten hops and then a `; +N more` marker.
+
+Only hops that already **failed** can appear. The hop actually serving your request is recorded after its response finishes — after the JSON is sent, or after the stream closes — so its duration does not exist while the headers are still open. Use `X-Routed-Via` to see which provider that was.
HTTP headers only carry printable ASCII, so a model id with characters outside that range (a Chinese name from a relay catalog, for example) is percent-encoded in the header — run the value through `decodeURIComponent` (or `urllib.parse.unquote`) to read it back.
diff --git a/server/src/__tests__/routes/fallback-detail-header.test.ts b/server/src/__tests__/routes/fallback-detail-header.test.ts
new file mode 100644
index 00000000..6361ebc4
--- /dev/null
+++ b/server/src/__tests__/routes/fallback-detail-header.test.ts
@@ -0,0 +1,299 @@
+import { describe, it, expect, beforeAll, beforeEach, afterEach, vi } from 'vitest';
+import type { Express } from 'express';
+
+// X-Fallback-Detail: the opt-in companion to X-Fallback-Trail.
+//
+// The trail already says WHICH hops burned and WHY. What a caller cannot
+// reconstruct is what each hop COST: a request answered in 40s reads
+// identically whether one provider stalled for 39s or four failed fast. The
+// per-hop timings are already collected for request_attempts; this header
+// exposes them without a dashboard round trip.
+
+const chatCompletion = vi.fn();
+const streamChatCompletion = vi.fn();
+const fakeProvider = { name: 'fake', chatCompletion, streamChatCompletion } as any;
+
+vi.mock('../../providers/index.js', async (importOriginal) => {
+ const actual = await importOriginal() as any;
+ return {
+ ...actual,
+ getProvider: () => fakeProvider,
+ resolveProvider: () => fakeProvider,
+ };
+});
+
+const { createApp } = await import('../../app.js');
+const { initDb, getDb, getUnifiedApiKey, setSetting } = await import('../../db/index.js');
+const { encrypt } = await import('../../lib/crypto.js');
+const { setRoutingStrategy } = await import('../../services/router.js');
+const {
+ EXPOSE_FALLBACK_DETAIL_SETTING,
+ formatAttemptDetail,
+ isFallbackDetailHeaderEnabled,
+ setFallbackHeaders,
+} = await import('../../lib/fallback-loop.js');
+const { newRequestTrace, runWithRequestTrace } = await import('../../lib/attempt-trace.js');
+
+/** Remove the row entirely, so the env-var fallback is actually reachable —
+ * storing '0' is a decision, not an absence. */
+function clearDetailSetting(): void {
+ getDb().prepare('DELETE FROM settings WHERE key = ?').run(EXPOSE_FALLBACK_DETAIL_SETTING);
+}
+
+async function post(app: Express, path: string, body: any, key: string) {
+ const server = app.listen(0);
+ const addr = server.address() as any;
+ const res = await fetch(`http://127.0.0.1:${addr.port}${path}`, {
+ method: 'POST',
+ headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${key}` },
+ body: JSON.stringify(body),
+ });
+ const raw = await res.text();
+ server.close();
+ let json: any = null;
+ try { json = JSON.parse(raw); } catch { /* SSE */ }
+ return { status: res.status, body: json, raw, headers: res.headers };
+}
+
+const GOOD_RESULT = {
+ choices: [{ message: { role: 'assistant', content: 'a real answer' }, finish_reason: 'stop' }],
+ usage: { prompt_tokens: 10, completion_tokens: 3, total_tokens: 13 },
+};
+
+const record = (over: Partial<Parameters<typeof formatAttemptDetail>[0][number]> = {}) => ({
+ ordinal: 0,
+ platform: 'groq',
+ modelId: 'llama-3.3-70b',
+ keyOrdinal: 1,
+ outcome: 'rate_limited' as const,
+ startOffsetMs: 0,
+ durationMs: 11,
+ errorSummary: null,
+ ...over,
+});
+
+describe('formatAttemptDetail', () => {
+ it('leads with the same shape as X-Fallback-Trail so the two headers line up', () => {
+ expect(formatAttemptDetail([record()])).toBe('groq/llama-3.3-70b key1=rate_limited t=0+11ms');
+ });
+
+ it('carries the per-hop timing that the trail cannot express', () => {
+ const value = formatAttemptDetail([
+ record({ startOffsetMs: 0, durationMs: 39_000 }),
+ record({ ordinal: 1, platform: 'google', modelId: 'gemini-2.5-flash', keyOrdinal: 2, outcome: 'timeout', startOffsetMs: 39_000, durationMs: 12 }),
+ ]);
+ // A 39s stall followed by a fast failure — indistinguishable in the trail.
+ expect(value).toBe(
+ 'groq/llama-3.3-70b key1=rate_limited t=0+39000ms; ' +
+ 'google/gemini-2.5-flash key2=timeout t=39000+12ms',
+ );
+ });
+
+ it('appends the redacted provider message when there is one', () => {
+ const value = formatAttemptDetail([record({ errorSummary: 'Groq API error 429: rate limit' })]);
+ expect(value).toBe('groq/llama-3.3-70b key1=rate_limited t=0+11ms msg=Groq API error 429: rate limit');
+ });
+
+ it('replaces semicolons in the message so they cannot forge a record boundary', () => {
+ const value = formatAttemptDetail([record({ errorSummary: 'first; second' })]);
+ expect(value).toBe('groq/llama-3.3-70b key1=rate_limited t=0+11ms msg=first, second');
+ expect(value.split('; ')).toHaveLength(1);
+ });
+
+ it('truncates a long message rather than letting ten hops blow the header budget', () => {
+ const value = formatAttemptDetail([record({ errorSummary: 'x'.repeat(500) })]);
+ expect(value).toContain('msg=' + 'x'.repeat(120));
+ expect(value).not.toContain('x'.repeat(121));
+ });
+
+ it('caps the hop count and says how many it dropped', () => {
+ const many = Array.from({ length: 14 }, (_, i) => record({ ordinal: i }));
+ const value = formatAttemptDetail(many);
+ expect(value.split('; ')).toHaveLength(11); // 10 hops + the "+4 more" marker
+ expect(value.endsWith('; +4 more')).toBe(true);
+ });
+
+ it('is empty for an empty trace', () => {
+ expect(formatAttemptDetail([])).toBe('');
+ });
+});
+
+describe('isFallbackDetailHeaderEnabled', () => {
+ beforeAll(() => {
+ process.env.ENCRYPTION_KEY = '0'.repeat(64);
+ initDb(':memory:');
+ });
+
+ beforeEach(() => {
+ clearDetailSetting();
+ });
+
+ afterEach(() => {
+ delete process.env.FALLBACK_DETAIL_HEADER;
+ clearDetailSetting();
+ });
+
+ it('is off by default — this widens what the surface reveals', () => {
+ expect(isFallbackDetailHeaderEnabled()).toBe(false);
+ });
+
+ it('honours the settings-table value', () => {
+ setSetting(EXPOSE_FALLBACK_DETAIL_SETTING, '1');
+ expect(isFallbackDetailHeaderEnabled()).toBe(true);
+ });
+
+ it('falls back to the env var when no setting is stored', () => {
+ process.env.FALLBACK_DETAIL_HEADER = 'true';
+ expect(isFallbackDetailHeaderEnabled()).toBe(true);
+ });
+
+ it('lets the settings table win over the env var, like the failover budget', () => {
+ setSetting(EXPOSE_FALLBACK_DETAIL_SETTING, '0');
+ process.env.FALLBACK_DETAIL_HEADER = '1';
+ expect(isFallbackDetailHeaderEnabled()).toBe(false);
+ });
+});
+
+describe('X-Fallback-Detail on the wire', () => {
+ let app: Express;
+ let key: string;
+
+ beforeAll(() => {
+ process.env.ENCRYPTION_KEY = '0'.repeat(64);
+ initDb(':memory:');
+ app = createApp();
+ key = getUnifiedApiKey();
+
+ const db = getDb();
+ setRoutingStrategy('priority');
+ const { encrypted, iv, authTag } = encrypt('detail-header-test-key');
+ db.prepare(`
+ INSERT INTO api_keys (platform, label, encrypted_key, iv, auth_tag, status, enabled)
+ VALUES ('groq', 'detail-header', ?, ?, ?, 'healthy', 1)
+ `).run(encrypted, iv, authTag);
+ });
+
+ beforeEach(() => {
+ chatCompletion.mockReset();
+ streamChatCompletion.mockReset();
+ getDb().prepare('DELETE FROM requests').run();
+ getDb().prepare('DELETE FROM rate_limit_cooldowns').run();
+ setSetting(EXPOSE_FALLBACK_DETAIL_SETTING, '1');
+ });
+
+ afterEach(() => {
+ setSetting(EXPOSE_FALLBACK_DETAIL_SETTING, '0');
+ delete process.env.FALLBACK_DETAIL_HEADER;
+ });
+
+ it('reports the failed hops with timings alongside the existing trail', async () => {
+ chatCompletion
+ .mockRejectedValueOnce(Object.assign(new Error('Groq API error 429: rate limit'), { status: 429 }))
+ .mockResolvedValueOnce(GOOD_RESULT);
+
+ const { status, headers } = await post(app, '/v1/chat/completions', {
+ messages: [{ role: 'user', content: 'detail header test' }],
+ }, key);
+
+ expect(status).toBe(200); // failed over and served
+ expect(headers.get('x-fallback-attempts')).toBe('1');
+ expect(headers.get('x-fallback-trail')).toContain('key1=rate_limited');
+
+ const detail = headers.get('x-fallback-detail');
+ expect(detail).toBeTruthy();
+ expect(detail).toMatch(/^groq\/\S+ key1=rate_limited t=\d+\+\d+ms msg=/);
+ expect(detail).toContain('rate limit');
+ });
+
+ it('stays silent when the setting is off', async () => {
+ setSetting(EXPOSE_FALLBACK_DETAIL_SETTING, '0');
+ chatCompletion
+ .mockRejectedValueOnce(Object.assign(new Error('Groq API error 429: rate limit'), { status: 429 }))
+ .mockResolvedValueOnce(GOOD_RESULT);
+
+ const { status, headers } = await post(app, '/v1/chat/completions', {
+ messages: [{ role: 'user', content: 'detail header off test' }],
+ }, key);
+
+ expect(status).toBe(200);
+ // The always-on trail is unaffected by the toggle.
+ expect(headers.get('x-fallback-attempts')).toBe('1');
+ expect(headers.get('x-fallback-detail')).toBeNull();
+ });
+
+ it('is absent on a request that never failed over, rather than empty', async () => {
+ chatCompletion.mockResolvedValueOnce(GOOD_RESULT);
+
+ const { status, headers } = await post(app, '/v1/chat/completions', {
+ messages: [{ role: 'user', content: 'clean request' }],
+ }, key);
+
+ expect(status).toBe(200);
+ expect(headers.get('x-fallback-detail')).toBeNull();
+ });
+
+ it('reaches the caller on the exhaustion path too, not just on success', async () => {
+ chatCompletion.mockRejectedValue(
+ Object.assign(new Error('Groq API error 429: rate limit'), { status: 429 }),
+ );
+
+ const { status, headers } = await post(app, '/v1/chat/completions', {
+ messages: [{ role: 'user', content: 'exhaustion detail test' }],
+ }, key);
+
+ expect(status).toBe(429);
+ const detail = headers.get('x-fallback-detail');
+ expect(detail).toBeTruthy();
+ expect(detail).toContain('key1=rate_limited');
+ expect(detail).toMatch(/t=\d+\+\d+ms/);
+ });
+
+});
+
+// Driving setFallbackHeaders directly, rather than steering the router into
+// picking a hostile model, keeps these about the header and not about routing.
+describe('X-Fallback-Detail value safety', () => {
+ function headersFor(records: Parameters<typeof formatAttemptDetail>[0]): Record<string, string> {
+ const trace = newRequestTrace();
+ trace.records.push(...records);
+ const out: Record<string, string> = {};
+ setSetting(EXPOSE_FALLBACK_DETAIL_SETTING, '1');
+ runWithRequestTrace(trace, () => {
+ setFallbackHeaders({ setHeader: (n: string, v: string) => { out[n] = v; } }, records.length, undefined);
+ });
+ return out;
+ }
+
+ afterEach(() => {
+ getDb().prepare('DELETE FROM settings WHERE key = ?').run(EXPOSE_FALLBACK_DETAIL_SETTING);
+ });
+
+ it('encodes a control-laden model id instead of letting it inject a header line', () => {
+ // #619: a non-ASCII or control-laden id used to make Node throw
+ // ERR_INVALID_CHAR after a successful upstream call, and the throw was then
+ // booked as a provider failure.
+ const value = headersFor([record({ modelId: 'bad\r\nX-Injected: yes' })])['X-Fallback-Detail'];
+
+ expect(value).toMatch(/^[\x20-\x7e]*$/);
+ expect(value).not.toContain('\r');
+ expect(value).not.toContain('\n');
+ expect(value).toContain('%0D%0A'); // encoded in place, not dropped
+ });
+
+ it('encodes a non-ASCII model id rather than rejecting the response', () => {
+ const value = headersFor([record({ modelId: '我的模型' })])['X-Fallback-Detail'];
+
+ expect(value).toMatch(/^[\x20-\x7e]*$/);
+ expect(decodeURIComponent(value.slice('groq/'.length).split(' ')[0])).toBe('我的模型');
+ });
+
+ it('keeps the whole value inside the header budget even at ten long hops', () => {
+ const many = Array.from({ length: 10 }, (_, i) => record({
+ ordinal: i,
+ modelId: 'm'.repeat(80),
+ errorSummary: 'e'.repeat(200),
+ }));
+
+ expect(headersFor(many)['X-Fallback-Detail'].length).toBeLessThanOrEqual(2048);
+ });
+});
diff --git a/server/src/lib/fallback-loop.ts b/server/src/lib/fallback-loop.ts
index e775df5c..93b8bd84 100644
--- a/server/src/lib/fallback-loop.ts
+++ b/server/src/lib/fallback-loop.ts
@@ -48,7 +48,7 @@ import { checkKeyHealth, markKeyHealthyFromRequest } from '../services/health.js
import { noteModelRetirementSignal } from '../services/model-retirement.js';
import { getSetting } from '../db/index.js';
import { newBreaker, recordBreakerFailure } from './guardrails.js';
-import { getRequestTrace, newRequestTrace, runWithRequestTrace, type AttemptOutcome, type RequestTrace } from './attempt-trace.js';
+import { getRequestTrace, newRequestTrace, runWithRequestTrace, type AttemptOutcome, type AttemptTraceRecord, type RequestTrace } from './attempt-trace.js';
import { logRequest, persistRequestAttempts } from './request-log.js';
// Every surface caps failover hops at the same number.
@@ -361,6 +361,69 @@ export function classifyAttemptError(err: any): AttemptErrorClass {
const TRAIL_MAX_SHOWN = 10;
const TRAIL_HEADER_MAX_LENGTH = 1024;
+// ── Detailed failover trace header (opt-in) ──────────────────────────────────
+// X-Fallback-Trail already tells a caller WHICH hops burned and WHY, but not
+// how long they cost. That is the part an agent cannot reconstruct: a request
+// answered in 40s reads identically whether one provider stalled for 39s or
+// four failed fast. The per-hop timings and the redacted provider message are
+// already collected for the request_attempts table; this exposes them on the
+// response so the caller sees them without a dashboard round trip.
+//
+// Off by default. It widens what an already-loopback-ish surface reveals —
+// hop timings plus provider error text — so it is opt-in like the discovery
+// aliases, not something a default install starts emitting. Precedence matches
+// the failover budget above: settings-table value, then env var, then off.
+export const EXPOSE_FALLBACK_DETAIL_SETTING = 'expose_fallback_detail_header';
+
+// Ten hops of "platform/model keyN=class t=…+…ms msg=…" with a capped message
+// each. Kept well under the 8 KB total header budget that proxies commonly
+// enforce, since this rides alongside X-Fallback-Trail's own 1 KB.
+const DETAIL_HEADER_MAX_LENGTH = 2048;
+// summarizeAttemptError caps at 200; the header wants a tighter budget so ten
+// hops still fit. The full text remains in request_attempts either way.
+const DETAIL_SUMMARY_MAX_LENGTH = 120;
+
+export function isFallbackDetailHeaderEnabled(): boolean {
+ let stored: string | undefined;
+ try {
+ stored = getSetting(EXPOSE_FALLBACK_DETAIL_SETTING);
+ } catch {
+ stored = undefined; // DB not ready — never throw on the proxy hot path
+ }
+ for (const raw of [stored, process.env.FALLBACK_DETAIL_HEADER]) {
+ if (raw === undefined || raw.trim() === '') continue;
+ const value = raw.trim().toLowerCase();
+ return value === '1' || value === 'true';
+ }
+ return false;
+}
+
+/**
+ * One `platform/model keyN=outcome t=<start>+<duration>ms msg=<summary>` segment
+ * per hop, `; `-joined — the same leading shape as X-Fallback-Trail so the two
+ * headers line up when read together.
+ *
+ * The summary is the already-redacted `errorSummary`, never `err.message`. Its
+ * semicolons become commas because `; ` is the record separator; nothing else
+ * is escaped, which keeps the value readable, and `safeHeaderValue` handles any
+ * non-ASCII on the way out.
+ */
+export function formatAttemptDetail(records: AttemptTraceRecord[]): string {
+ const shown = records.slice(0, TRAIL_MAX_SHOWN).map(r => {
+ const parts = [
+ `${r.platform}/${r.modelId}`,
+ `key${r.keyOrdinal}=${r.outcome}`,
+ `t=${r.startOffsetMs}+${r.durationMs}ms`,
+ ];
+ if (r.errorSummary) {
+ parts.push(`msg=${r.errorSummary.slice(0, DETAIL_SUMMARY_MAX_LENGTH).replace(/;/g, ',')}`);
+ }
+ return parts.join(' ');
+ });
+ const extra = records.length - shown.length;
+ return shown.join('; ') + (extra > 0 ? `; +${extra} more` : '');
+}
+
export function formatAttemptTrail(attempts: AttemptRecord[]): string {
const shown = attempts
.slice(0, TRAIL_MAX_SHOWN)
@@ -378,6 +441,17 @@ export function formatAttemptTrail(attempts: AttemptRecord[]): string {
* exactly the case an operator wants to notice. Values go through
* safeHeaderValue so a non-ASCII or control-laden model id can neither inject
* header lines nor make Node reject the response outright (#619).
+ *
+ * When EXPOSE_FALLBACK_DETAIL_SETTING is on, X-Fallback-Detail joins them with
+ * per-hop timings and the redacted provider message. Every caller runs inside
+ * the request's AsyncLocalStorage scope, so the trace is readable here without
+ * threading it through all five surfaces.
+ *
+ * Note what the detail header can and cannot contain: at flush time the trace
+ * holds exactly the hops that already FAILED, each with final timings. The hop
+ * currently being served is recorded only after dispatch returns — after
+ * res.json(), or after the whole stream has finished — so its duration is not
+ * knowable while headers are still open, on either path.
*/
export function setFallbackHeaders(
res: { setHeader(name: string, value: string): void },
@@ -394,6 +468,13 @@ export function setFallbackHeaders(
// trail gets its own budget.
res.setHeader('X-Fallback-Trail', safeHeaderValue(value, TRAIL_HEADER_MAX_LENGTH));
}
+
+ // Checked before the setting so the overwhelmingly common no-failover request
+ // never pays for a settings read.
+ const records = getRequestTrace()?.records;
+ if (records && records.length > 0 && isFallbackDetailHeaderEnabled()) {
+ res.setHeader('X-Fallback-Detail', safeHeaderValue(formatAttemptDetail(records), DETAIL_HEADER_MAX_LENGTH));
+ }
}
export interface ExhaustionBody {