7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.
TypeScript
fix(fallback): bench a route that ate the whole budget with no first byte (#1183)
Chong Jia Zhen2e7ac36cca22
diff --git a/server/src/__tests__/lib/fallback-loop-hedge.test.ts b/server/src/__tests__/lib/fallback-loop-hedge.test.tsindex ebd6220f..6fdc8eb1 100644--- a/server/src/__tests__/lib/fallback-loop-hedge.test.ts+++ b/server/src/__tests__/lib/fallback-loop-hedge.test.ts@@ -10,9 +10,10 @@ import type { RouteResult } from '../../services/router.js';* the loop calls abortInFlight() so a stalled attempt is abandoned instead of* only refusing to start the next retry behind it. The surface aborts its* composed fetch signal with newHedgeAbortError(); the loop must render- * timedOut exhaustion WITHOUT any provider-health bookkeeping — no cooldown,- * no skip entry, no logFailure row — because the budget is spent, not the- * provider broken.+ * timedOut exhaustion, and the provider-health bookkeeping — cooldown, skip+ * entry, logFailure row — depends on WHOSE silence spent the budget: an attempt+ * handed only the scraps a ladder left is not benched, one that stayed silent+ * for most of the whole budget is.*/let keySeq = 900;@@ -160,6 +161,48 @@ describe('fallback loop time-budget hedging', () => {expect(h.onExhausted.mock.calls[0][1].timedOut).toBe(true);});+ it('benches the route when one attempt stayed silent for the whole budget', async () => {+ // The production shape behind this branch. Same ladder position as the two+ // tests above — hedging only arms from the third attempt on (#751) — but+ // here the two failures ahead of it are INSTANT, so the stalled attempt+ // inherits essentially the whole budget and then sends no first byte for+ // all of it. That is a stalled upstream, not a scheduling accident:+ // unbenched it stays at the head of the route order and re-stalls every+ // subsequent request, burning the full budget each time while the healthy+ // routes queued behind it never run.+ const state = newFallbackState();+ const hedgeAbort = new AbortController();+ const fast429 = async () => {+ throw Object.assign(new Error('fake API error 429: rate limit'), { status: 429 });+ };+ const dispatch = vi.fn()+ .mockImplementationOnce(fast429)+ .mockImplementationOnce(fast429)+ .mockImplementation(stalledDispatch(hedgeAbort));++ const h = hooks(state, {+ maxRetries: 5,+ timeBudgetMs: 100,+ abortInFlight: () => hedgeAbort.abort(newHedgeAbortError()),+ route: () => leasedRoute().route,+ dispatch,+ });++ await runFallbackLoop(h);++ // Still a timedOut exhaustion — the render is unchanged.+ expect(h.onExhausted).toHaveBeenCalledTimes(1);+ expect(h.onExhausted.mock.calls[0][1].timedOut).toBe(true);+ // But the stalled route now carries a cooldown and a skip entry of its own,+ // on top of the two fast 429s', so the next request routes past it. This is+ // the exact assertion that inverts for the shared-budget case above, where+ // the same three attempts produce only two of each.+ const cooldowns = getDb().prepare('SELECT COUNT(*) AS n FROM rate_limit_cooldowns').get() as { n: number };+ expect(cooldowns.n).toBe(3);+ expect(state.skipKeys.size).toBe(3);+ expect(h.logFailure).toHaveBeenCalledTimes(3);+ });+it('never cancels an attempt that already committed, however long it then runs', async () => {// The whole point of the hedge is to kill a SILENT attempt. Once a stream// has flushed its headers the answer is already reaching the client and thediff --git a/server/src/lib/fallback-loop.ts b/server/src/lib/fallback-loop.tsindex ccc85c29..ece1c7e5 100644--- a/server/src/lib/fallback-loop.ts+++ b/server/src/lib/fallback-loop.ts@@ -127,6 +127,14 @@ function clearModelFailure(route: RouteResult): void {// TODO(fallback-v2): AbortController hedging so a stalled attempt can be// abandoned mid-flight instead of only refusing to start the next one.export const DEFAULT_FALLBACK_TIME_BUDGET_MS = 45_000;+// Share of the wall-clock budget an aborted attempt must have been silent for+// before the hedge abort counts as provider health rather than bad luck. An+// attempt late in a ladder inherits only the scraps of the budget its+// predecessors left, and killing it proves nothing about that route; one that+// owned most of the window and still sent no first byte is stalled. Kept a+// fraction rather than a constant because the budget IS the declared patience+// — raising FALLBACK_TIME_BUDGET_MS should raise the evidence bar with it.+export const HEDGE_BENCH_MIN_SILENT_FRACTION = 0.75;export const FALLBACK_TIME_BUDGET_SETTING = 'fallback_time_budget_ms';export function getFallbackTimeBudgetMs(): number {@@ -937,8 +945,8 @@ export interface FallbackHooks {// (remaining wall-clock budget) and calls this to ABORT the in-flight// upstream instead of just refusing to start the next retry behind a// stalled attempt. The surface aborts its composed fetch signal with- // newHedgeAbortError() — a non-provider-health signal, so the loop renders- // timedOut exhaustion without benching the model+key (see the+ // newHedgeAbortError(); the loop renders timedOut exhaustion, and benches the+ // model+key only when that attempt ate the whole budget by itself (see the// isHedgeAbortError branch below). Absent = pre-v2 behavior.abortInFlight?: () => void;// Skip state; recordRetryableFailure / recordAuthFailure (called by the loop)@@ -1179,12 +1187,33 @@ async function runFallbackLoopAttempts(hooks: FallbackHooks, trace: RequestTrace}// Time-budget hedge abort: the wall-clock retry budget expired while this// attempt was still in flight, and the surface aborted the composed fetch- // signal (see newHedgeAbortError). Not a provider-health signal — no- // cooldown, no penalty, no failure stats — and the budget is spent, so+ // signal (see newHedgeAbortError). The budget is spent either way, so we// render timedOut exhaustion exactly like the loop-top budget check does.+ //+ // Whether it also counts as a provider-health signal depends on how much+ // of the silence belongs to THIS attempt. Sharing a budget that earlier+ // attempts already mostly spent says nothing about this route — it was+ // simply last in line, and cancelling it is a scheduling accident. But an+ // attempt that stayed silent for most of the operator's whole declared+ // patience on its own produced no first byte over that entire window,+ // and that is a stalled upstream. Left unbenched it stays at the head of the route+ // order and re-stalls every subsequent request, burning the full budget+ // each time and starving the healthy routes queued behind it — observed+ // in production as one dead free-tier route 45s-ing every request for+ // hours. Benching costs a short transient cooldown (a timeout is not a+ // rate-limit signal, so cooldownDecisionForError keeps it light) plus the+ // gradual model-level sink, both of which decay on their own if the route+ // recovers.if (isHedgeAbortError(err)) {const elapsedMs = Date.now() - startedAt;- console.log(`[FallbackLoop] retry time budget expired mid-attempt on ${route.platform}/${route.modelId} after ${(elapsedMs / 1000).toFixed(1)}s — rendering timedOut exhaustion without benching`);+ const attemptElapsedMs = Date.now() - attemptStartedAt;+ const ownedWholeBudget = budgetMs > 0+ && attemptElapsedMs >= budgetMs * HEDGE_BENCH_MIN_SILENT_FRACTION;+ if (ownedWholeBudget) {+ hooks.logFailure(route, err, attempt);+ recordRetryableFailure(route, err, hooks.state);+ }+ console.log(`[FallbackLoop] retry time budget expired mid-attempt on ${route.platform}/${route.modelId} after ${(elapsedMs / 1000).toFixed(1)}s — rendering timedOut exhaustion ${ownedWholeBudget ? `and benching the route (silent for the whole ${budgetMs}ms budget)` : 'without benching'}`);hooks.onExhausted(exhaustedRetryError(lastError, maxRetries, { attempts, timedOut: true, budgetMs }),{ attempts, timedOut: true },