Skip to content

7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.

TypeScript

fix(fallback): bench a route that ate the whole budget with no first byte (#1183)

Chong Jia Zhen2e7ac36cca22
diff --git a/server/src/__tests__/lib/fallback-loop-hedge.test.ts b/server/src/__tests__/lib/fallback-loop-hedge.test.ts
index ebd6220f..6fdc8eb1 100644
--- a/server/src/__tests__/lib/fallback-loop-hedge.test.ts
+++ b/server/src/__tests__/lib/fallback-loop-hedge.test.ts
@@ -10,9 +10,10 @@ import type { RouteResult } from '../../services/router.js';
* the loop calls abortInFlight() so a stalled attempt is abandoned instead of
* only refusing to start the next retry behind it. The surface aborts its
* composed fetch signal with newHedgeAbortError(); the loop must render
- * timedOut exhaustion WITHOUT any provider-health bookkeeping — no cooldown,
- * no skip entry, no logFailure row — because the budget is spent, not the
- * provider broken.
+ * timedOut exhaustion, and the provider-health bookkeeping — cooldown, skip
+ * entry, logFailure row — depends on WHOSE silence spent the budget: an attempt
+ * handed only the scraps a ladder left is not benched, one that stayed silent
+ * for most of the whole budget is.
*/
let keySeq = 900;
@@ -160,6 +161,48 @@ describe('fallback loop time-budget hedging', () => {
expect(h.onExhausted.mock.calls[0][1].timedOut).toBe(true);
});
+ it('benches the route when one attempt stayed silent for the whole budget', async () => {
+ // The production shape behind this branch. Same ladder position as the two
+ // tests above — hedging only arms from the third attempt on (#751) — but
+ // here the two failures ahead of it are INSTANT, so the stalled attempt
+ // inherits essentially the whole budget and then sends no first byte for
+ // all of it. That is a stalled upstream, not a scheduling accident:
+ // unbenched it stays at the head of the route order and re-stalls every
+ // subsequent request, burning the full budget each time while the healthy
+ // routes queued behind it never run.
+ const state = newFallbackState();
+ const hedgeAbort = new AbortController();
+ const fast429 = async () => {
+ throw Object.assign(new Error('fake API error 429: rate limit'), { status: 429 });
+ };
+ const dispatch = vi.fn()
+ .mockImplementationOnce(fast429)
+ .mockImplementationOnce(fast429)
+ .mockImplementation(stalledDispatch(hedgeAbort));
+
+ const h = hooks(state, {
+ maxRetries: 5,
+ timeBudgetMs: 100,
+ abortInFlight: () => hedgeAbort.abort(newHedgeAbortError()),
+ route: () => leasedRoute().route,
+ dispatch,
+ });
+
+ await runFallbackLoop(h);
+
+ // Still a timedOut exhaustion — the render is unchanged.
+ expect(h.onExhausted).toHaveBeenCalledTimes(1);
+ expect(h.onExhausted.mock.calls[0][1].timedOut).toBe(true);
+ // But the stalled route now carries a cooldown and a skip entry of its own,
+ // on top of the two fast 429s', so the next request routes past it. This is
+ // the exact assertion that inverts for the shared-budget case above, where
+ // the same three attempts produce only two of each.
+ const cooldowns = getDb().prepare('SELECT COUNT(*) AS n FROM rate_limit_cooldowns').get() as { n: number };
+ expect(cooldowns.n).toBe(3);
+ expect(state.skipKeys.size).toBe(3);
+ expect(h.logFailure).toHaveBeenCalledTimes(3);
+ });
+
it('never cancels an attempt that already committed, however long it then runs', async () => {
// The whole point of the hedge is to kill a SILENT attempt. Once a stream
// has flushed its headers the answer is already reaching the client and the
diff --git a/server/src/lib/fallback-loop.ts b/server/src/lib/fallback-loop.ts
index ccc85c29..ece1c7e5 100644
--- a/server/src/lib/fallback-loop.ts
+++ b/server/src/lib/fallback-loop.ts
@@ -127,6 +127,14 @@ function clearModelFailure(route: RouteResult): void {
// TODO(fallback-v2): AbortController hedging so a stalled attempt can be
// abandoned mid-flight instead of only refusing to start the next one.
export const DEFAULT_FALLBACK_TIME_BUDGET_MS = 45_000;
+// Share of the wall-clock budget an aborted attempt must have been silent for
+// before the hedge abort counts as provider health rather than bad luck. An
+// attempt late in a ladder inherits only the scraps of the budget its
+// predecessors left, and killing it proves nothing about that route; one that
+// owned most of the window and still sent no first byte is stalled. Kept a
+// fraction rather than a constant because the budget IS the declared patience
+// — raising FALLBACK_TIME_BUDGET_MS should raise the evidence bar with it.
+export const HEDGE_BENCH_MIN_SILENT_FRACTION = 0.75;
export const FALLBACK_TIME_BUDGET_SETTING = 'fallback_time_budget_ms';
export function getFallbackTimeBudgetMs(): number {
@@ -937,8 +945,8 @@ export interface FallbackHooks {
// (remaining wall-clock budget) and calls this to ABORT the in-flight
// upstream instead of just refusing to start the next retry behind a
// stalled attempt. The surface aborts its composed fetch signal with
- // newHedgeAbortError() — a non-provider-health signal, so the loop renders
- // timedOut exhaustion without benching the model+key (see the
+ // newHedgeAbortError(); the loop renders timedOut exhaustion, and benches the
+ // model+key only when that attempt ate the whole budget by itself (see the
// isHedgeAbortError branch below). Absent = pre-v2 behavior.
abortInFlight?: () => void;
// Skip state; recordRetryableFailure / recordAuthFailure (called by the loop)
@@ -1179,12 +1187,33 @@ async function runFallbackLoopAttempts(hooks: FallbackHooks, trace: RequestTrace
}
// Time-budget hedge abort: the wall-clock retry budget expired while this
// attempt was still in flight, and the surface aborted the composed fetch
- // signal (see newHedgeAbortError). Not a provider-health signal — no
- // cooldown, no penalty, no failure stats — and the budget is spent, so
+ // signal (see newHedgeAbortError). The budget is spent either way, so we
// render timedOut exhaustion exactly like the loop-top budget check does.
+ //
+ // Whether it also counts as a provider-health signal depends on how much
+ // of the silence belongs to THIS attempt. Sharing a budget that earlier
+ // attempts already mostly spent says nothing about this route — it was
+ // simply last in line, and cancelling it is a scheduling accident. But an
+ // attempt that stayed silent for most of the operator's whole declared
+ // patience on its own produced no first byte over that entire window,
+ // and that is a stalled upstream. Left unbenched it stays at the head of the route
+ // order and re-stalls every subsequent request, burning the full budget
+ // each time and starving the healthy routes queued behind it — observed
+ // in production as one dead free-tier route 45s-ing every request for
+ // hours. Benching costs a short transient cooldown (a timeout is not a
+ // rate-limit signal, so cooldownDecisionForError keeps it light) plus the
+ // gradual model-level sink, both of which decay on their own if the route
+ // recovers.
if (isHedgeAbortError(err)) {
const elapsedMs = Date.now() - startedAt;
- console.log(`[FallbackLoop] retry time budget expired mid-attempt on ${route.platform}/${route.modelId} after ${(elapsedMs / 1000).toFixed(1)}s — rendering timedOut exhaustion without benching`);
+ const attemptElapsedMs = Date.now() - attemptStartedAt;
+ const ownedWholeBudget = budgetMs > 0
+ && attemptElapsedMs >= budgetMs * HEDGE_BENCH_MIN_SILENT_FRACTION;
+ if (ownedWholeBudget) {
+ hooks.logFailure(route, err, attempt);
+ recordRetryableFailure(route, err, hooks.state);
+ }
+ console.log(`[FallbackLoop] retry time budget expired mid-attempt on ${route.platform}/${route.modelId} after ${(elapsedMs / 1000).toFixed(1)}s — rendering timedOut exhaustion ${ownedWholeBudget ? `and benching the route (silent for the whole ${budgetMs}ms budget)` : 'without benching'}`);
hooks.onExhausted(
exhaustedRetryError(lastError, maxRetries, { attempts, timedOut: true, budgetMs }),
{ attempts, timedOut: true },