7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.
TypeScript
fix: make fallback retry budgets endpoint TTFB-aware (#1262) (#1272)
Inference_8ee41bb58233
diff --git a/.env.example b/.env.exampleindex f0a0448f..aac59a1d 100644--- a/.env.example+++ b/.env.example@@ -142,14 +142,38 @@ PORT=3001# without loosening the watchdog everywhere else:# PROVIDER_STREAM_STALL_TIMEOUT_OLLAMA=240000-# Wall-clock budget for provider failover, in milliseconds (default: 45000).-# A request that keeps failing over stops STARTING new attempts once this much-# time has passed and returns the exhaustion error instead (the first attempt-# always runs; an in-flight attempt is never cut off). Set to 0 to disable and-# retry until the attempt cap. Can also be set at runtime via the-# fallback_time_budget_ms settings key, which takes precedence.+# Base wall-clock budget for provider failover, in milliseconds (default: 45000).+# The initial attempt and one failover hop are always allowed. After that,+# exhaustion stops new attempts and can cancel an in-flight attempt still+# waiting for its first usable response. Streaming responses disarm this timer+# when the first usable response arrives. Successful TTFB history can widen+# the budget for a slow endpoint (see below). Set to 0 to disable the budget+# entirely, including adaptive widening, and retry until the attempt cap.+# Can also be set at runtime via the fallback_time_budget_ms settings key,+# which takes precedence.# FALLBACK_TIME_BUDGET_MS=45000+# Adaptive patience for slow endpoints: successful time-to-first-byte (TTFB)+# samples are grouped by platform + endpoint_scope. The effective budget is+# max(base budget, historical P95 TTFB + this buffer), in milliseconds.+# Endpoints with no samples, and those whose P95 + buffer fits the base budget,+# keep the base budget. The settings key slow_endpoint_buffer_ms takes+# precedence over this env var. Set the buffer to 0 for no extra padding.+# SLOW_ENDPOINT_BUFFER_MS=10000+# Successful TTFB sample window, in milliseconds (default: 7 days).+# The settings key ttfb_budget_window_ms takes precedence.+# TTFB_BUDGET_WINDOW_MS=604800000+# Sample-weight half-life, in milliseconds (default: 2 days). Older successes+# count less so the budget adapts as an endpoint speeds up or slows down.+# The settings key ttfb_budget_half_life_ms takes precedence.+# TTFB_BUDGET_HALF_LIFE_MS=172800000+# Successful samples an endpoint needs before its history widens the budget+# (default: 5). The widened budget is capped at three times the base budget.+# The settings key ttfb_budget_min_samples takes precedence.+# TTFB_BUDGET_MIN_SAMPLES=5+# Set to 1 to disable adaptive TTFB budgets and use only the base budget.+# TTFB_BUDGET_DISABLED=1+# Add X-Fallback-Detail to responses (default: off). X-Fallback-Trail already# names every hop that failed and why; the detail header adds what each one# COST — "groq/llama-3.3-70b key1=rate_limited t=0+39000ms msg=..." — so adiff --git a/docs/en/env/01-variables.md b/docs/en/env/01-variables.mdindex 9efff6fc..d6dbf5e9 100644--- a/docs/en/env/01-variables.md+++ b/docs/en/env/01-variables.md@@ -59,7 +59,12 @@ First-run note: the first dashboard account is normally created from a browser o| `PROVIDER_STREAM_STALL_TIMEOUT_MS` | `90000` | Mid-stream inactivity timeout in milliseconds. If an SSE stream goes this long without a single byte AFTER the first byte arrived, the gateway gives up: before any output reached the client it fails over to the next candidate; after output was sent the stream ends with an error frame. Time-to-FIRST-byte is governed by `PROVIDER_TIMEOUT_<PLATFORM>` instead (whichever of the two is larger). Providers that queue heavily on free tiers can pause longer than 90s mid-stream — raise this if streamed responses stop early. `0` disables the stall watchdog entirely. || `PROVIDER_STREAM_STALL_TIMEOUT_<PLATFORM>` | Falls through to the global value above, then the built-in `90000` | Per-provider variant (platform name upper-cased). Precedence: per-platform > global > built-in 90s default. Example: `PROVIDER_STREAM_STALL_TIMEOUT_OLLAMA=240000` lets Ollama Cloud pause 4 minutes mid-stream without loosening the watchdog everywhere else. || `PROVIDER_DAILY_REQUEST_CAP_<PLATFORM>` | Built-in default per provider | Per-provider daily request cap (platform name upper-cased). Overrides the built-in default for that provider; set to `0` to disable the cap. Example: `PROVIDER_DAILY_REQUEST_CAP_OPENROUTER=50`. |-| `FALLBACK_TIME_BUDGET_MS` | `45000` | Wall-clock budget for provider failover, in milliseconds. A request that keeps failing over stops STARTING new attempts once this much time has passed and returns the exhaustion error instead (the first attempt always runs; an in-flight attempt is never cut off). `0` disables and retries until the attempt cap. Also settable at runtime via the `fallback_time_budget_ms` settings key, which takes precedence. |+| `FALLBACK_TIME_BUDGET_MS` | `45000` | Base wall-clock budget for provider failover, in milliseconds. The initial attempt and one failover hop are always allowed. After that, exhaustion stops new attempts and can cancel an in-flight attempt still waiting for its first usable response. Streaming responses disarm this timer when the first usable response arrives. Successful TTFB history can widen the budget for a slow endpoint, as described below. `0` disables the budget entirely, including adaptive widening, and retries until the attempt cap. Also settable at runtime via the `fallback_time_budget_ms` settings key, which takes precedence. |+| `SLOW_ENDPOINT_BUFFER_MS` | `10000` | Padding added to the historical P95 successful time-to-first-byte (TTFB), in milliseconds. Samples are grouped by platform + `endpoint_scope`; the effective fallback budget is `max(base budget, P95 TTFB + buffer)`, capped at three times the base budget, and only for endpoints whose P95 is at least half the base budget. Endpoints with fewer than `TTFB_BUDGET_MIN_SAMPLES` samples, and those whose P95 + buffer fits the base budget, keep the base budget. `0` removes extra padding. The runtime `slow_endpoint_buffer_ms` settings key takes precedence over this environment variable. |+| `TTFB_BUDGET_WINDOW_MS` | `604800000` (7 days) | Rolling window of successful TTFB samples used for adaptive fallback budgets, in milliseconds. The runtime `ttfb_budget_window_ms` settings key takes precedence over this environment variable. |+| `TTFB_BUDGET_HALF_LIFE_MS` | `172800000` (2 days) | Half-life of successful TTFB sample weights, in milliseconds. Older successes count less so the P95 follows changes in endpoint speed. The runtime `ttfb_budget_half_life_ms` settings key takes precedence over this environment variable. |+| `TTFB_BUDGET_MIN_SAMPLES` | `5` | Successful TTFB samples an endpoint needs inside the window before its history can widen the budget, so a single slow success does not. The runtime `ttfb_budget_min_samples` settings key takes precedence over this environment variable. |+| `TTFB_BUDGET_DISABLED` | Off | Set to `1` to disable adaptive TTFB budgets and use only the base fallback budget. || `FALLBACK_DETAIL_HEADER` | Off | Opt-in `X-Fallback-Detail` response header (`=1` enables). `X-Fallback-Trail` already names every hop that failed and why; the detail header adds what each one COST — e.g. `groq/llama-3.3-70b key1=rate_limited t=0+39000ms msg=...` — so a caller can tell one 39s stall apart from four fast failures without opening the dashboard. Off by default because it puts hop timings and provider error text on the response. Runtime override via the `expose_fallback_detail_header` settings key takes precedence. || `VALIDATE_TOOL_ARGUMENTS` | Off (`=1` enables) | Check tool-call arguments against the schema the caller declared and fail over when they violate it. `lib/tool-args.ts` already repairs what it can prove is repairable; this catches what is left, which today reaches Anthropic clients as a `tool_use` block with `input: {}`. Off by default because a false positive costs a failover hop — turn it on, watch `X-Fallback-Trail` for `invalid_tool_arguments`, then decide. Applies wherever the turn can still fail over (non-streaming surfaces, plus `/v1/chat/completions`, `/v1/messages`, Gemini and Ollama emulations); `/v1/responses` streaming commits on the first tool-call delta and is the exception. Runtime override via the `validate_tool_arguments` settings key. || `REQUEST_MAX_TOKENS_BUDGET` | `0` (off) | Per-request token guardrail: estimated input + requested max_tokens must fit this ceiling or the request is rejected with a 413 before any provider is tried; a request that sent no max_tokens gets its output capped to the remainder instead. Runtime-tunable via the `request_max_tokens_budget` settings key (dashboard API `PUT /api/settings/guardrails`). |diff --git a/docs/zh-cn/env/01-variables.md b/docs/zh-cn/env/01-variables.mdindex 9862a365..0c70acfe 100644--- a/docs/zh-cn/env/01-variables.md+++ b/docs/zh-cn/env/01-variables.md@@ -57,7 +57,12 @@ FreeLLMAPI 从 `.env` 读取的全部变量,按主题分组。默认值和说| `PROVIDER_STREAM_STALL_TIMEOUT_MS` | `90000` | 流中途的停滞超时,单位毫秒。如果一条 SSE 流在首字节已到达之后持续这么久没有任何字节,网关就放弃:若尚无输出到达客户端,则故障转移到下一个候选;若已有输出,则以一个错误帧结束流。到「第一个」字节为止的时间由 `PROVIDER_TIMEOUT_<PLATFORM>` 管(两者取较大者)。在免费额度上排长队的提供方可能在流中途停顿超过 90 秒——如果流式响应总是提前中断,就调高这个值。`0` 彻底关闭停滞看门狗。 || `PROVIDER_STREAM_STALL_TIMEOUT_<PLATFORM>` | 先回落到上面的全局值,再到内置的 `90000` | 按提供方的变体(平台名大写)。优先级:按平台 > 全局 > 内置 90 秒默认。示例:`PROVIDER_STREAM_STALL_TIMEOUT_OLLAMA=240000` 允许 Ollama Cloud 在流中途停顿 4 分钟,而其他地方的看门狗不受影响。 || `PROVIDER_DAILY_REQUEST_CAP_<PLATFORM>` | 各提供方内置的默认值 | 按提供方的每日请求上限(平台名大写)。覆盖该提供方的内置默认值;设为 `0` 关闭上限。示例:`PROVIDER_DAILY_REQUEST_CAP_OPENROUTER=50`。 |-| `FALLBACK_TIME_BUDGET_MS` | `45000` | 提供方故障转移的挂钟时间预算,单位毫秒。一个不断失败转移的请求,一旦耗时超过这个预算就不再「发起」新的尝试,改为返回耗尽错误(第一次尝试总会执行;进行中的尝试绝不会被拦腰截断)。`0` 表示不限制,一直重试直到触及尝试次数上限。也可以在运行时通过 `fallback_time_budget_ms` 设置键修改,后者优先。 |+| `FALLBACK_TIME_BUDGET_MS` | `45000` | 提供方故障转移的基础挂钟时间预算,单位毫秒。始终允许首次尝试及一次故障转移;之后预算耗尽时会停止发起新尝试,也可能取消仍在等待首个有效响应的进行中尝试。流式响应收到首个有效响应后会解除此计时器。成功请求的首字节耗时历史可为慢端点延长预算,详见下方。`0` 彻底关闭该预算(包括自适应延长),一直重试直到触及尝试次数上限。也可以在运行时通过 `fallback_time_budget_ms` 设置键修改,后者优先。 |+| `SLOW_ENDPOINT_BUFFER_MS` | `10000` | 在历史成功请求的首字节耗时(TTFB)P95 上增加的缓冲,单位毫秒。样本按平台 + `endpoint_scope` 分组,有效故障转移预算为 `max(基础预算, P95 TTFB + 缓冲)`,上限为基础预算的三倍,且仅适用于 P95 不低于基础预算一半的端点。样本数少于 `TTFB_BUDGET_MIN_SAMPLES` 的端点,以及 P95 加缓冲不超过基础预算的端点,仍使用基础预算。`0` 表示不额外增加缓冲。运行时的 `slow_endpoint_buffer_ms` 设置键优先于此环境变量。 |+| `TTFB_BUDGET_WINDOW_MS` | `604800000`(7 天) | 计算自适应故障转移预算时使用的成功 TTFB 样本滚动时间窗口,单位毫秒。运行时的 `ttfb_budget_window_ms` 设置键优先于此环境变量。 |+| `TTFB_BUDGET_HALF_LIFE_MS` | `172800000`(2 天) | 成功 TTFB 样本权重的半衰期,单位毫秒。越旧的成功记录权重越低,使 P95 随端点速度的变化而调整。运行时的 `ttfb_budget_half_life_ms` 设置键优先于此环境变量。 |+| `TTFB_BUDGET_MIN_SAMPLES` | `5` | 端点在时间窗口内至少需要多少条成功 TTFB 样本,其历史才会放宽预算,避免单次慢速成功就放宽预算。运行时的 `ttfb_budget_min_samples` 设置键优先于此环境变量。 |+| `TTFB_BUDGET_DISABLED` | 关闭 | 设为 `1` 可关闭 TTFB 自适应预算,仅使用基础故障转移预算。 || `FALLBACK_DETAIL_HEADER` | 关闭 | 可选开启的 `X-Fallback-Detail` 响应头(`=1` 开启)。`X-Fallback-Trail` 已经列出了每一跳的失败原因;这个详情头额外给出每一跳的「代价」——例如 `groq/llama-3.3-70b key1=rate_limited t=0+39000ms msg=...`——这样调用者不用打开仪表盘就能分辨一次 39 秒的卡顿和四次快速失败。默认关闭,因为它会把各跳的耗时和提供方错误文本暴露在响应上。运行时通过 `expose_fallback_detail_header` 设置键覆盖,后者优先。 || `VALIDATE_TOOL_ARGUMENTS` | 关闭(`=1` 开启) | 对照调用方声明的 schema 校验工具调用参数,违反时触发故障转移。`lib/tool-args.ts` 已经修复它能确证可修的部分;这个开关负责剩下的漏网之鱼——它们如今到达 Anthropic 客户端时会是一个带 `input: {}` 的 `tool_use` 块。默认关闭,因为误报的代价是一跳故障转移——先打开它,盯着 `X-Fallback-Trail` 观察 `invalid_tool_arguments`,再决定去留。适用于回合仍可故障转移的所有场合(非流式端点,外加 `/v1/chat/completions`、`/v1/messages`、Gemini 与 Ollama 模拟);`/v1/responses` 的流式在第一个工具调用增量上就已提交,属于例外。运行时通过 `validate_tool_arguments` 设置键调整。 || `REQUEST_MAX_TOKENS_BUDGET` | `0`(关闭) | 按请求的词元护栏:估算输入词元加所请求的 max_tokens 必须落在这个上限之内,否则在任何提供方被尝试之前就以 413 拒绝请求;没有发送 max_tokens 的请求,其输出会被截断到剩余额度。可在运行时通过 `request_max_tokens_budget` 设置键调整(仪表盘 API `PUT /api/settings/guardrails`)。 |diff --git a/server/src/__tests__/lib/fallback-loop-ttfb-budget.test.ts b/server/src/__tests__/lib/fallback-loop-ttfb-budget.test.tsnew file mode 100644index 00000000..f57c2b23--- /dev/null+++ b/server/src/__tests__/lib/fallback-loop-ttfb-budget.test.ts@@ -0,0 +1,307 @@+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';+import { getDb, initDb, setSetting } from '../../db/index.js';+import { newClientAbortError, newHedgeAbortError } from '../../lib/error-classify.js';+import { newFallbackState, runFallbackLoop, type FallbackHooks } from '../../lib/fallback-loop.js';+import { invalidateTtfbBudgetCache } from '../../lib/ttfb-budget.js';+import { acquireLease, inFlightForKey, releaseLease, resetLeases } from '../../services/ratelimit.js';+import type { RouteResult } from '../../services/router.js';++interface Attempt {+ platform: string;+ endpointScope?: string;+ delayMs: number;+ result: 'retry' | 'done' | 'stalled' | 'stream' | 'client_abort';+}++let nextKey = 50_000;+function scenario(plan: Attempt[], overrides: Partial<FallbackHooks> = {}) {+ const controller = new AbortController();+ const routed: RouteResult[] = [];+ const hooks = {+ state: newFallbackState(),+ timeBudgetMs: 45_000,+ maxRetries: 6,+ breakerLimit: 0,+ route: vi.fn((attempt: number): RouteResult => {+ const target = plan[attempt];+ if (!target) throw new Error('no more routes');+ const keyId = ++nextKey;+ const leaseId = acquireLease(target.platform, 'test-model', keyId, 100);+ const route: RouteResult = {+ provider: {} as RouteResult['provider'],+ modelId: 'test-model',+ modelDbId: keyId,+ apiKey: 'test-key',+ keyId,+ keyLabel: null,+ platform: target.platform,+ endpointScope: target.endpointScope ?? '',+ displayName: 'Test model',+ rpdLimit: null,+ tpdLimit: null,+ release: vi.fn(() => releaseLease(leaseId)),+ };+ routed.push(route);+ return route;+ }),+ dispatch: vi.fn<FallbackHooks['dispatch']>(async (_route, attempt, ctx) => {+ const target = plan[attempt];+ if (target.result === 'stalled') {+ return await new Promise<never>((_resolve, reject) => {+ controller.signal.addEventListener('abort', () => reject(newHedgeAbortError()), { once: true });+ });+ }+ await new Promise(resolve => setTimeout(resolve, target.delayMs));+ if (target.result === 'retry') throw Object.assign(new Error('rate limit'), { status: 429 });+ if (target.result === 'client_abort') throw newClientAbortError();+ if (target.result === 'stream') {+ ctx.disarmHedge();+ await new Promise(resolve => setTimeout(resolve, 100_000));+ }+ return 'done';+ }),+ abortInFlight: vi.fn(() => controller.abort(newHedgeAbortError())),+ logFailure: vi.fn(),+ onFatal: vi.fn(),+ onExhausted: vi.fn(),+ onRoutingExhausted: vi.fn(),+ ...overrides,+ };+ return { hooks, routed };+}++function history(platform: string, ttfbMs: number, endpointScope = ''): void {+ let keyId: number | null = null;+ if (endpointScope) {+ const row = getDb().prepare(`+ INSERT INTO api_keys (platform, encrypted_key, iv, auth_tag, base_url)+ VALUES (?, 'unused', 'unused', 'unused', ?)+ `).run(platform, endpointScope);+ keyId = Number(row.lastInsertRowid);+ }+ getDb().prepare(`+ INSERT INTO requests (platform, model_id, key_id, status, ttfb_ms, created_at)+ VALUES (?, 'historical-model', ?, 'success', ?, ?)+ `).run(platform, keyId, ttfbMs, new Date(Date.now() - 1_000).toISOString());+}++function warmups(delayMs = 1_000): Attempt[] {+ return [+ { platform: 'warmup-a', delayMs, result: 'retry' },+ { platform: 'warmup-b', delayMs, result: 'retry' },+ ];+}++function expectReleased(routes: RouteResult[]): void {+ for (const route of routes) {+ expect(route.release).toHaveBeenCalledTimes(1);+ expect(inFlightForKey(route.platform, route.keyId)).toBe(0);+ }+}++describe('endpoint TTFB-aware fallback budget (#1262)', () => {+ beforeEach(() => {+ vi.spyOn(console, 'log').mockImplementation(() => {});+ vi.spyOn(console, 'warn').mockImplementation(() => {});+ vi.stubEnv('TTFB_BUDGET_DISABLED', '');+ vi.stubEnv('TTFB_BUDGET_MIN_SAMPLES', '1');+ vi.stubEnv('SLOW_ENDPOINT_BUFFER_MS', '10000');+ vi.stubEnv('TTFB_BUDGET_WINDOW_MS', '604800000');+ vi.stubEnv('TTFB_BUDGET_HALF_LIFE_MS', '172800000');+ initDb(':memory:');+ resetLeases();+ invalidateTtfbBudgetCache();+ vi.useFakeTimers();+ vi.setSystemTime(new Date('2026-09-19T12:00:00Z'));+ });++ afterEach(() => {+ vi.useRealTimers();+ vi.unstubAllEnvs();+ vi.restoreAllMocks();+ resetLeases();+ invalidateTtfbBudgetCache();+ });++ it('lets a historically slow endpoint respond after the original 45s deadline', async () => {+ history('slow', 50_000);+ const { hooks, routed } = scenario([+ ...warmups(), { platform: 'slow', delayMs: 50_000, result: 'done' },+ ]);+ const run = runFallbackLoop(hooks);+ await vi.advanceTimersByTimeAsync(65_000);+ await run;++ expect(hooks.dispatch).toHaveBeenCalledTimes(3);+ expect(hooks.abortInFlight).not.toHaveBeenCalled();+ expect(hooks.onExhausted).not.toHaveBeenCalled();+ expectReleased(routed);+ });++ it('uses the same widened deadline for the hedge and exhaustion message', async () => {+ history('slow', 50_000);+ const { hooks, routed } = scenario([+ ...warmups(), { platform: 'slow', delayMs: 0, result: 'stalled' },+ ]);+ const run = runFallbackLoop(hooks);+ await vi.advanceTimersByTimeAsync(59_999);+ expect(hooks.abortInFlight).not.toHaveBeenCalled();+ await vi.advanceTimersByTimeAsync(1);+ await run;++ expect(hooks.abortInFlight).toHaveBeenCalledTimes(1);+ expect(hooks.onExhausted).toHaveBeenCalledWith(+ expect.objectContaining({ message: expect.stringContaining('retry time budget 60s') }),+ expect.objectContaining({ timedOut: true }),+ );+ expect(hooks.logFailure).toHaveBeenCalledTimes(3);+ expectReleased(routed);+ });++ it('checks the next endpoint allowance even when earlier attempts spent the base budget', async () => {+ history('slow', 90_000);+ const { hooks } = scenario([+ ...warmups(25_000), { platform: 'slow', delayMs: 10_000, result: 'done' },+ ]);+ const run = runFallbackLoop(hooks);+ await vi.advanceTimersByTimeAsync(110_000);+ await run;++ expect(hooks.dispatch).toHaveBeenCalledTimes(3);+ expect(hooks.abortInFlight).not.toHaveBeenCalled();+ expect(hooks.onExhausted).not.toHaveBeenCalled();+ });++ it('does not carry a slow endpoint allowance into a fast endpoint and releases the rejected route', async () => {+ history('slow', 90_000);+ history('fast', 1_000);+ const { hooks, routed } = scenario([+ ...warmups(),+ { platform: 'slow', delayMs: 50_000, result: 'retry' },+ { platform: 'fast', delayMs: 0, result: 'done' },+ ]);+ const run = runFallbackLoop(hooks);+ await vi.advanceTimersByTimeAsync(60_000);+ await run;++ expect(hooks.route).toHaveBeenCalledTimes(4);+ expect(hooks.dispatch).toHaveBeenCalledTimes(3);+ expect(hooks.onExhausted).toHaveBeenCalledWith(+ expect.objectContaining({ message: expect.stringContaining('retry time budget 45s') }),+ expect.objectContaining({ timedOut: true, attempts: expect.any(Array) }),+ );+ expect(hooks.onExhausted.mock.calls[0][1].attempts).toHaveLength(3);+ expectReleased(routed);+ });++ it.each(['fast', 'unknown', 'custom'])('keeps the base deadline for %s without borrowing another relay history', async (platform) => {+ history('fast', 1_000);+ history('custom', 90_000, 'https://slow-relay.example/v1');+ const { hooks } = scenario([+ ...warmups(),+ { platform, endpointScope: platform === 'custom' ? 'https://fast-relay.example/v1' : '', delayMs: 0, result: 'stalled' },+ ]);+ const run = runFallbackLoop(hooks);+ await vi.advanceTimersByTimeAsync(45_000);+ await run;++ expect(hooks.abortInFlight).toHaveBeenCalledTimes(1);+ expect(hooks.onExhausted.mock.calls[0][0].message).toContain('retry time budget 45s');+ });++ it('uses widened patience for the hedge health threshold, not the base budget', async () => {+ history('slow', 70_000);+ const { hooks } = scenario([+ ...warmups(20_000), { platform: 'slow', delayMs: 0, result: 'stalled' },+ ]);+ const run = runFallbackLoop(hooks);+ await vi.advanceTimersByTimeAsync(80_000);+ await run;++ // The final attempt owned 40s of an 80s budget, not 40s of 45s.+ expect(hooks.logFailure).toHaveBeenCalledTimes(2);+ expect(hooks.state.skipKeys.size).toBe(2);+ expect(hooks.onExhausted.mock.calls[0][1].timedOut).toBe(true);+ });++ it('still disarms the hedge after the first usable stream response', async () => {+ history('slow', 50_000);+ const { hooks, routed } = scenario([+ ...warmups(), { platform: 'slow', delayMs: 50_000, result: 'stream' },+ ]);+ const run = runFallbackLoop(hooks);+ await vi.advanceTimersByTimeAsync(160_000);+ await run;++ expect(hooks.abortInFlight).not.toHaveBeenCalled();+ expect(hooks.onExhausted).not.toHaveBeenCalled();+ expectReleased(routed);+ });++ it('does not re-enable a disabled base budget', async () => {+ history('slow', 50_000);+ const { hooks } = scenario([+ ...warmups(), { platform: 'slow', delayMs: 150_000, result: 'done' },+ ], { timeBudgetMs: 0 });+ const run = runFallbackLoop(hooks);+ await vi.advanceTimersByTimeAsync(160_000);+ await run;++ expect(hooks.abortInFlight).not.toHaveBeenCalled();+ expect(hooks.onExhausted).not.toHaveBeenCalled();+ });++ it('restores the base deadline when adaptation is disabled', async () => {+ history('slow', 90_000);+ vi.stubEnv('TTFB_BUDGET_DISABLED', '1');+ const { hooks } = scenario([+ ...warmups(), { platform: 'slow', delayMs: 0, result: 'stalled' },+ ]);+ const run = runFallbackLoop(hooks);+ await vi.advanceTimersByTimeAsync(45_000);+ await run;++ expect(hooks.onExhausted.mock.calls[0][0].message).toContain('retry time budget 45s');+ });++ it('keeps the runtime base setting as a floor above a learned allowance', async () => {+ history('slow', 50_000);+ setSetting('fallback_time_budget_ms', '120000');+ const { hooks } = scenario([+ ...warmups(), { platform: 'slow', delayMs: 0, result: 'stalled' },+ ], { timeBudgetMs: undefined });+ const run = runFallbackLoop(hooks);+ await vi.advanceTimersByTimeAsync(119_999);+ expect(hooks.abortInFlight).not.toHaveBeenCalled();+ await vi.advanceTimersByTimeAsync(1);+ await run;++ expect(hooks.onExhausted.mock.calls[0][0].message).toContain('retry time budget 120s');+ });++ it('stops on a client abort during the extended window without benching it', async () => {+ history('slow', 90_000);+ const { hooks, routed } = scenario([+ ...warmups(), { platform: 'slow', delayMs: 50_000, result: 'client_abort' },+ ]);+ const run = runFallbackLoop(hooks);+ await vi.advanceTimersByTimeAsync(110_000);+ await run;++ expect(hooks.dispatch).toHaveBeenCalledTimes(3);+ expect(hooks.logFailure).toHaveBeenCalledTimes(2);+ expect(hooks.abortInFlight).not.toHaveBeenCalled();+ expect(hooks.onExhausted).not.toHaveBeenCalled();+ expectReleased(routed);+ });++ it('preserves timeout exhaustion when routing is also exhausted after the base deadline', async () => {+ const { hooks } = scenario(warmups(25_000));+ const run = runFallbackLoop(hooks);+ await vi.advanceTimersByTimeAsync(50_000);+ await run;++ expect(hooks.onExhausted.mock.calls[0][1].timedOut).toBe(true);+ expect(hooks.onRoutingExhausted).not.toHaveBeenCalled();+ });+});diff --git a/server/src/__tests__/lib/ttfb-budget.test.ts b/server/src/__tests__/lib/ttfb-budget.test.tsnew file mode 100644index 00000000..ed3483ac--- /dev/null+++ b/server/src/__tests__/lib/ttfb-budget.test.ts@@ -0,0 +1,200 @@+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';+import { getDb, initDb, setSetting } from '../../db/index.js';+import {+ getEndpointTimeBudgetMs,+ getEndpointTtfbStats,+ invalidateTtfbBudgetCache,+} from '../../lib/ttfb-budget.js';++const NOW = Date.parse('2026-09-19T12:00:00Z');+const DAY_MS = 24 * 60 * 60 * 1000;++function addRequest(ttfbMs: number | string | null, opts: {+ platform?: string; modelId?: string; keyId?: number; status?: string; at?: number | string;+} = {}): void {+ const at = opts.at ?? NOW;+ getDb().prepare(`+ INSERT INTO requests (platform, model_id, key_id, status, ttfb_ms, created_at)+ VALUES (?, ?, ?, ?, ?, ?)+ `).run(opts.platform ?? 'google', opts.modelId ?? 'model', opts.keyId ?? null,+ opts.status ?? 'success', ttfbMs, typeof at === 'string' ? at : new Date(at).toISOString());+}++function addCustomKey(baseUrl: string): number {+ return Number(getDb().prepare(`+ INSERT INTO api_keys (platform, encrypted_key, iv, auth_tag, base_url)+ VALUES ('custom', 'enc', 'iv', 'tag', ?)+ `).run(baseUrl).lastInsertRowid);+}++describe('endpoint TTFB budgets (#1262)', () => {+ beforeEach(() => {+ vi.stubEnv('DEV_MODE', 'true');+ vi.stubEnv('NODE_ENV', 'test');+ vi.stubEnv('TTFB_BUDGET_DISABLED', '');+ vi.stubEnv('TTFB_BUDGET_MIN_SAMPLES', '1');+ vi.stubEnv('SLOW_ENDPOINT_BUFFER_MS', '');+ vi.stubEnv('TTFB_BUDGET_WINDOW_MS', '');+ vi.stubEnv('TTFB_BUDGET_HALF_LIFE_MS', '');+ initDb(':memory:');+ invalidateTtfbBudgetCache();+ });++ afterEach(() => {+ getDb().close?.();+ vi.restoreAllMocks();+ vi.unstubAllEnvs();+ invalidateTtfbBudgetCache();+ });++ it('retains the configured base for a cold or fast endpoint, and raises it for a slow one', () => {+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(45_000);+ addRequest(1_000);+ addRequest(55_000, { platform: 'groq' });+ invalidateTtfbBudgetCache();+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(45_000);+ expect(getEndpointTimeBudgetMs(45_000, 'groq', '', NOW)).toBe(65_000);+ });++ it('calculates nearest-rank P50/P95 across models of an endpoint', () => {+ for (let i = 1; i <= 20; i++) addRequest(i * 1_000, { modelId: `model-${i}` });+ expect(getEndpointTtfbStats('google', '', NOW)).toEqual({+ p50Ms: 10_000, p95Ms: 19_000, sampleCount: 20, weightedSamples: 20,+ });+ });++ it('needs five successes by default before one slow anecdote widens the budget', () => {+ vi.stubEnv('TTFB_BUDGET_MIN_SAMPLES', '');+ for (let i = 0; i < 4; i++) addRequest(60_000);+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(45_000);+ addRequest(60_000);+ invalidateTtfbBudgetCache();+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(70_000);+ setSetting('ttfb_budget_min_samples', '6');+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(45_000);+ });++ it('caps the learned allowance at three times the base so it cannot ratchet', () => {+ addRequest(500_000);+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(135_000);+ expect(getEndpointTimeBudgetMs(6_000, 'google', '', NOW)).toBe(18_000);+ });++ it('keeps a small base budget for a fast endpoint despite the absolute buffer', () => {+ addRequest(200);+ addRequest(5_000, { platform: 'groq' });+ expect(getEndpointTimeBudgetMs(6_000, 'google', '', NOW)).toBe(6_000);+ expect(getEndpointTimeBudgetMs(6_000, 'groq', '', NOW)).toBe(15_000);+ });++ it('does not widen the budget from a zero TTFB observation alone', () => {+ addRequest(0);+ setSetting('slow_endpoint_buffer_ms', '120000');+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(45_000);+ });++ it('decays observations with a two-day half life so recent recovery wins', () => {+ addRequest(60_000, { at: NOW - 6 * DAY_MS });+ for (let i = 0; i < 3; i++) addRequest(2_000);+ expect(getEndpointTtfbStats('google', '', NOW)).toEqual({+ p50Ms: 2_000, p95Ms: 2_000, sampleCount: 4, weightedSamples: 3.125,+ });+ });++ it('isolates custom scopes, combines keys for the same relay and keeps orphan history unscoped', () => {+ const a1 = addCustomKey('https://a.example/v1/');+ const a2 = addCustomKey('https://a.example/v1');+ const b = addCustomKey('https://b.example/v1');+ addRequest(60_000, { platform: 'custom', keyId: a1 });+ addRequest(70_000, { platform: 'custom', keyId: a2 });+ addRequest(2_000, { platform: 'custom', keyId: b });+ addRequest(90_000, { platform: 'custom', keyId: 999_999 });+ expect(getEndpointTimeBudgetMs(45_000, 'custom', ' https://a.example/v1/// ', NOW)).toBe(80_000);+ expect(getEndpointTtfbStats('custom', 'https://a.example/v1', NOW)?.sampleCount).toBe(2);+ expect(getEndpointTimeBudgetMs(45_000, 'custom', 'https://b.example/v1', NOW)).toBe(45_000);+ expect(getEndpointTimeBudgetMs(45_000, 'custom', 'https://new.example/v1', NOW)).toBe(45_000);+ expect(getEndpointTimeBudgetMs(45_000, 'custom', '', NOW)).toBe(100_000);+ });++ it('ignores failed, canceled, expired, future, invalid and missing observations', () => {+ addRequest(1_000);+ addRequest(900_000, { status: 'error' });+ addRequest(900_000, { status: 'canceled' });+ addRequest(900_000, { at: NOW - 7 * DAY_MS - 1 });+ addRequest(900_000, { at: NOW + 1 });+ addRequest(900_000, { at: 'not-a-date' });+ addRequest(-1);+ addRequest(Infinity);+ addRequest('invalid');+ addRequest(null);+ expect(getEndpointTtfbStats('google', '', NOW)).toEqual({+ p50Ms: 1_000, p95Ms: 1_000, sampleCount: 1, weightedSamples: 1,+ });+ });++ it('treats SQLite times as UTC and accepts ISO offsets at the exact window boundary', () => {+ addRequest(1_000, { at: '2026-09-19 12:00:00' });+ addRequest(1_000, { at: '2026-09-19T08:00:00-04:00' });+ addRequest(1_000, { at: '2026-09-12 12:00:00' });+ addRequest(1_000, { at: '2026-09-12T00:00:00-12:00' });+ expect(getEndpointTtfbStats('google', '', NOW)?.sampleCount).toBe(4);+ });++ it('settings override env immediately and zero is a valid safety buffer', () => {+ addRequest(60_000);+ vi.stubEnv('SLOW_ENDPOINT_BUFFER_MS', '20000');+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(80_000);+ setSetting('slow_endpoint_buffer_ms', '5000');+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(65_000);+ setSetting('slow_endpoint_buffer_ms', '0');+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(60_000);+ setSetting('slow_endpoint_buffer_ms', 'NaN');+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(80_000);+ vi.stubEnv('SLOW_ENDPOINT_BUFFER_MS', '-1');+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(70_000);+ });++ it('recomputes cached samples when the configured window or half life changes', () => {+ addRequest(60_000, { at: NOW - 6 * DAY_MS });+ for (let i = 0; i < 3; i++) addRequest(2_000);+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(45_000);+ vi.stubEnv('TTFB_BUDGET_HALF_LIFE_MS', String(20 * DAY_MS));+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(70_000);+ setSetting('ttfb_budget_window_ms', String(DAY_MS));+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(45_000);+ setSetting('ttfb_budget_window_ms', '0');+ vi.stubEnv('TTFB_BUDGET_WINDOW_MS', 'Infinity');+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(70_000);+ setSetting('ttfb_budget_half_life_ms', String(2 * DAY_MS));+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(45_000);+ });++ it('caches scans for a minute but discards cache when the database changes', () => {+ addRequest(1_000);+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(45_000);+ addRequest(60_000);+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW + 59_999)).toBe(45_000);+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW + 60_000)).toBe(70_000);+ getDb().close?.();+ initDb(':memory:');+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW + 60_001)).toBe(45_000);+ });++ it('preserves an explicitly disabled budget and supports the emergency env switch', () => {+ addRequest(60_000);+ expect(getEndpointTimeBudgetMs(0, 'google', '', NOW)).toBe(0);+ vi.stubEnv('TTFB_BUDGET_DISABLED', '1');+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(45_000);+ vi.stubEnv('TTFB_BUDGET_DISABLED', '0');+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(70_000);+ });++ it('falls back safely on DB errors and nonfinite derived budgets', () => {+ addRequest(Number.MAX_VALUE);+ setSetting('slow_endpoint_buffer_ms', String(Number.MAX_VALUE));+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(45_000);+ vi.spyOn(getDb(), 'prepare').mockImplementation(() => { throw new Error('busy'); });+ expect(getEndpointTimeBudgetMs(45_000, 'google', '', NOW)).toBe(45_000);+ expect(getEndpointTtfbStats('google', '', NOW)).toBeNull();+ });+});diff --git a/server/src/lib/fallback-loop.ts b/server/src/lib/fallback-loop.tsindex ec303d31..d467578b 100644--- a/server/src/lib/fallback-loop.ts+++ b/server/src/lib/fallback-loop.ts@@ -60,6 +60,7 @@ import { newBreaker, recordBreakerFailure } from './guardrails.js';import { getRequestTrace, newRequestTrace, runWithRequestTrace, type AttemptOutcome, type AttemptTraceRecord, type RequestTrace } from './attempt-trace.js';import { logRequest, persistRequestAttempts } from './request-log.js';import { withKeyProxy } from './proxy.js';+import { getEndpointTimeBudgetMs } from './ttfb-budget.js';// Every surface caps failover hops at the same number.export const FALLBACK_MAX_RETRIES = 20;@@ -155,17 +156,16 @@ function benchKeyAcrossPlatform(// ── Wall-clock retry budget ──────────────────────────────────────────────────// Serial failover has no time bound of its own: the observed worst case was a// 38.8s TTFB over 11 attempts, and the theoretical worst is maxRetries x the-// per-attempt HTTP timeout. The budget is checked before STARTING each retry,-// so one slow attempt is never aborted mid-flight — it just becomes the last-// one. The first attempt always runs, and so does the FIRST retry: when+// per-attempt HTTP timeout. The budget is checked before STARTING each retry+// and, when abortInFlight is available, while waiting for its first byte.+// Successful endpoint TTFB history can widen the configured base budget.+// The first attempt always runs, and so does the FIRST retry: when// attempt 0 alone consumes the whole budget (a slow-failing model), refusing// attempt 1 would make failover structurally impossible for exactly the// requests that need it (#751). The budget stops attempts >= 2 only.// 0 disables the budget entirely.// Precedence mirrors the response cache: the settings-table value wins when// present (runtime-tunable), then the env var, then the default.-// TODO(fallback-v2): AbortController hedging so a stalled attempt can be-// abandoned mid-flight instead of only refusing to start the next one.export const DEFAULT_FALLBACK_TIME_BUDGET_MS = 45_000;// Share of the wall-clock budget an aborted attempt must have been silent for// before the hedge abort counts as provider health rather than bad luck. An@@ -1187,8 +1187,9 @@ export interface ExhaustionInfo {export interface FallbackHooks {// Defaults to FALLBACK_MAX_RETRIES.maxRetries?: number;- // Wall-clock retry budget override, mostly for tests. Defaults to+ // Base wall-clock retry budget override, mostly for tests. Defaults to// getFallbackTimeBudgetMs() (setting → env → 45s; 0 disables).+ // Successful endpoint TTFB history can widen this floor.timeBudgetMs?: number;// Circuit-breaker threshold override, mostly for tests. Defaults to// getMaxConsecutiveUpstreamFails() (setting → env → 0 = disabled).@@ -1299,7 +1300,7 @@ export async function runFallbackLoop(hooks: FallbackHooks): Promise<void> {async function runFallbackLoopAttempts(hooks: FallbackHooks, trace: RequestTrace): Promise<void> {const maxRetries = hooks.maxRetries ?? FALLBACK_MAX_RETRIES;- const budgetMs = hooks.timeBudgetMs ?? getFallbackTimeBudgetMs();+ const baseBudgetMs = hooks.timeBudgetMs ?? getFallbackTimeBudgetMs();const startedAt = Date.now();const attempts: AttemptRecord[] = hooks.attemptLog ?? [];const keyOrdinals = new Map<string, number>();@@ -1346,24 +1347,19 @@ async function runFallbackLoopAttempts(hooks: FallbackHooks, trace: RequestTracereturn;}- // Wall-clock budget: refuse to START another retry once spent. The first- // attempt always runs, and so does the first RETRY — when attempt 0 alone- // consumed the budget, refusing attempt 1 would make failover impossible- // for exactly the slow-failing models that need it (#751). A slow attempt- // is never aborted mid-flight (that is the TODO(fallback-v2) hedging- // work), it just becomes the last one.- if (attempt > 1 && budgetMs > 0 && Date.now() - startedAt >= budgetMs) {- hooks.onExhausted(- exhaustedRetryError(lastError, maxRetries, { attempts, timedOut: true, budgetMs }),- { attempts, timedOut: true },- );- return;- }-let route: RouteResult;try {route = hooks.route(attempt);} catch (routeErr) {+ // With no candidate to supply an endpoint-specific allowance, retain+ // the original timeout diagnosis when the base budget is already spent.+ if (attempt > 1 && baseBudgetMs > 0 && Date.now() - startedAt >= baseBudgetMs) {+ hooks.onExhausted(+ exhaustedRetryError(lastError, maxRetries, { attempts, timedOut: true, budgetMs: baseBudgetMs }),+ { attempts, timedOut: true },+ );+ return;+ }const exhaustion = lastError? exhaustedRetryError(lastError, undefined, { attempts }): routingExhaustionBody(routeErr);@@ -1375,6 +1371,25 @@ async function runFallbackLoopAttempts(hooks: FallbackHooks, trace: RequestTracereturn;}+ let hedgeTimer: NodeJS.Timeout | undefined;+ try {+ // Select the endpoint before checking the budget: a slow endpoint may+ // still have time even after the base budget has expired. Recompute from+ // the base for every candidate so its allowance cannot leak to a faster+ // endpoint later in the ladder. The clock still starts at loop entry.+ const budgetMs = attempt > 1+ ? getEndpointTimeBudgetMs(baseBudgetMs, route.platform, route.endpointScope)+ : baseBudgetMs;+ // Attempt 0 and the first retry remain exempt (#751). Routing reserves a+ // lease, so even a candidate rejected here must pass through finally.+ if (attempt > 1 && budgetMs > 0 && Date.now() - startedAt >= budgetMs) {+ hooks.onExhausted(+ exhaustedRetryError(lastError, maxRetries, { attempts, timedOut: true, budgetMs }),+ { attempts, timedOut: true },+ );+ return;+ }+// Per-attempt trace record: pushed exactly once per dispatched attempt, on// whichever exit the attempt takes. startOffsetMs/durationMs bracket the// dispatch (for a successful stream, durationMs runs until the response@@ -1413,7 +1428,6 @@ async function runFallbackLoopAttempts(hooks: FallbackHooks, trace: RequestTrace// because past that point cancelling would truncate a healthy response and// buy nothing: a committed stream can no longer fail over anyway. Slow is// not the same as stalled, and only stalled is worth killing.- let hedgeTimer: NodeJS.Timeout | undefined;const disarmHedge = () => {if (hedgeTimer) {clearTimeout(hedgeTimer);@@ -1429,7 +1443,6 @@ async function runFallbackLoopAttempts(hooks: FallbackHooks, trace: RequestTrace}, remaining);}}- try {let outcome: DispatchOutcome;try {// #590 (per-key proxy): if THIS key carries its own proxy URL, route thediff --git a/server/src/lib/ttfb-budget.ts b/server/src/lib/ttfb-budget.tsnew file mode 100644index 00000000..c21ee039--- /dev/null+++ b/server/src/lib/ttfb-budget.ts@@ -0,0 +1,185 @@+import { getDb, type Db } from '../db/index.js';+import { endpointScopeForBaseUrl } from './endpoint-scope.js';++const DEFAULT_BUFFER_MS = 10_000;+const DEFAULT_WINDOW_MS = 7 * 24 * 60 * 60 * 1000;+const DEFAULT_HALF_LIFE_MS = 2 * 24 * 60 * 60 * 1000;+const CACHE_TTL_MS = 60_000;+// One lucky 90s success must not triple an endpoint's patience: below this+// many in-window successes the history is an anecdote, not a distribution.+const DEFAULT_MIN_SAMPLES = 5;+// Logged TTFB runs from request start, so it includes the hops that failed+// before the endpoint was reached. A widened budget admits later successes,+// which log a larger TTFB, which widens the budget again — without a ceiling+// the allowance ratchets up by one buffer per refresh. Cap it at a multiple of+// the operator's base budget, which stays the declared unit of patience.+const MAX_BUDGET_MULTIPLIER = 3;+// The buffer is absolute, so against a small operator budget (say 6s) it would+// hand a 200ms endpoint 10s of patience. Only an endpoint whose P95 is itself a+// large share of the base counts as slow; with the 45s default this changes+// nothing, because a P95 under 22.5s plus the buffer already fits the base.+const SLOW_P95_BASE_FRACTION = 0.5;++export interface EndpointTtfbStats {+ p50Ms: number;+ p95Ms: number;+ sampleCount: number;+ weightedSamples: number;+}++interface Config {+ bufferMs: number;+ windowMs: number;+ halfLifeMs: number;+ minSamples: number;+}++interface Sample {+ ttfbMs: number;+ weight: number;+}++let cache: {+ db: Db;+ time: number;+ windowMs: number;+ halfLifeMs: number;+ stats: Map<string, EndpointTtfbStats>;+} | undefined;++/** Clear the derived cache; the request log remains the source of truth. */+export function invalidateTtfbBudgetCache(): void {+ cache = undefined;+}++function readConfig(db: Db): Config {+ const rows = db.prepare(`+ SELECT key, value FROM settings+ WHERE key IN ('slow_endpoint_buffer_ms', 'ttfb_budget_window_ms', 'ttfb_budget_half_life_ms', 'ttfb_budget_min_samples')+ `).all() as { key: string; value: string }[];+ const settings = new Map(rows.map(row => [row.key, row.value]));+ const duration = (key: string, fallback: number, allowZero = false): number => {+ for (const raw of [settings.get(key), process.env[key.toUpperCase()]]) {+ if (raw === undefined || raw.trim() === '') continue;+ const value = Number(raw);+ if (Number.isFinite(value) && (allowZero ? value >= 0 : value > 0)) return value;+ }+ return fallback;+ };+ return {+ bufferMs: duration('slow_endpoint_buffer_ms', DEFAULT_BUFFER_MS, true),+ windowMs: duration('ttfb_budget_window_ms', DEFAULT_WINDOW_MS),+ halfLifeMs: duration('ttfb_budget_half_life_ms', DEFAULT_HALF_LIFE_MS),+ minSamples: duration('ttfb_budget_min_samples', DEFAULT_MIN_SAMPLES),+ };+}++function endpointKey(platform: string, endpointScope: string): string {+ // Catalog platforms have one endpoint identity, while custom relays share a+ // platform and must never inherit each other's latency history.+ return JSON.stringify([platform, platform === 'custom' ? endpointScopeForBaseUrl(endpointScope) : '']);+}++function timestampMs(raw: string): number {+ // SQLite's datetime('now') has no suffix but is UTC. Date.parse would treat+ // it as local time; ISO values may already carry Z or an explicit offset.+ const iso = raw.trim().replace(' ', 'T');+ return Date.parse(/(?:Z|[+-]\d{2}:?\d{2})$/i.test(iso) ? iso : `${iso}Z`);+}++function percentile(samples: Sample[], totalWeight: number, quantile: number): number {+ const target = totalWeight * quantile;+ let cumulative = 0;+ for (const sample of samples) {+ cumulative += sample.weight;+ if (cumulative >= target) return sample.ttfbMs;+ }+ return samples[samples.length - 1].ttfbMs;+}++function statsFor(db: Db, config: Config, now: number): Map<string, EndpointTtfbStats> {+ if (cache?.db === db && cache.windowMs === config.windowMs && cache.halfLifeMs === config.halfLifeMs+ && now >= cache.time && now - cache.time < CACHE_TTL_MS) return cache.stats;++ // Keep the created_at range index usable despite SQLite/ISO timestamp+ // formats. Read one extra UTC day for offset-bearing ISO timestamps, then+ // enforce the precise sliding window after parsing all timestamps as UTC.+ const earliest = new Date(now - config.windowMs - 24 * 60 * 60 * 1000);+ const since = Number.isFinite(earliest.getTime()) ? earliest.toISOString().slice(0, 10) : '';+ const rows = db.prepare(`+ SELECT r.platform, r.ttfb_ms, r.created_at, k.base_url+ FROM requests r+ LEFT JOIN api_keys k ON k.id = r.key_id AND k.platform = 'custom'+ WHERE r.created_at >= ? AND r.status = 'success' AND r.ttfb_ms IS NOT NULL+ `).all(since) as { platform: string; ttfb_ms: number; created_at: string; base_url: string | null }[];++ const buckets = new Map<string, Sample[]>();+ for (const row of rows) {+ if (typeof row.ttfb_ms !== 'number' || !Number.isFinite(row.ttfb_ms) || row.ttfb_ms < 0+ || typeof row.created_at !== 'string') continue;+ const age = now - timestampMs(row.created_at);+ if (!Number.isFinite(age) || age < 0 || age > config.windowMs) continue;+ const weight = Math.pow(0.5, age / config.halfLifeMs);+ if (!(weight > 0)) continue;+ const key = endpointKey(row.platform, row.base_url ?? '');+ let bucket = buckets.get(key);+ if (!bucket) buckets.set(key, bucket = []);+ bucket.push({ ttfbMs: row.ttfb_ms, weight });+ }++ const stats = new Map<string, EndpointTtfbStats>();+ for (const [key, samples] of buckets) {+ samples.sort((a, b) => a.ttfbMs - b.ttfbMs);+ const weightedSamples = samples.reduce((total, sample) => total + sample.weight, 0);+ stats.set(key, {+ p50Ms: percentile(samples, weightedSamples, 0.5),+ p95Ms: percentile(samples, weightedSamples, 0.95),+ sampleCount: samples.length,+ weightedSamples,+ });+ }+ cache = { db, time: now, windowMs: config.windowMs, halfLifeMs: config.halfLifeMs, stats };+ return stats;+}++/** Decay-weighted nearest-rank percentiles of successful endpoint TTFB. */+export function getEndpointTtfbStats(+ platform: string,+ endpointScope = '',+ now = Date.now(),+): EndpointTtfbStats | null {+ if (!Number.isFinite(now)) return null;+ try {+ const db = getDb();+ const stats = statsFor(db, readConfig(db), now).get(endpointKey(platform, endpointScope));+ return stats ? { ...stats } : null;+ } catch {+ invalidateTtfbBudgetCache();+ return null;+ }+}++/** Give historically slow endpoints their P95 TTFB plus a safety buffer. */+export function getEndpointTimeBudgetMs(+ baseBudgetMs: number,+ platform: string,+ endpointScope = '',+ now = Date.now(),+): number {+ if (!Number.isFinite(baseBudgetMs) || baseBudgetMs <= 0 || !Number.isFinite(now)+ || process.env.TTFB_BUDGET_DISABLED?.trim() === '1') return baseBudgetMs;+ try {+ const db = getDb();+ const config = readConfig(db);+ const stats = statsFor(db, config, now).get(endpointKey(platform, endpointScope));+ if (!stats || stats.p95Ms <= 0 || stats.sampleCount < config.minSamples+ || stats.p95Ms < baseBudgetMs * SLOW_P95_BASE_FRACTION) return baseBudgetMs;+ const adaptiveBudget = stats.p95Ms + config.bufferMs;+ if (!Number.isFinite(adaptiveBudget)) return baseBudgetMs;+ return Math.max(baseBudgetMs, Math.min(adaptiveBudget, baseBudgetMs * MAX_BUDGET_MULTIPLIER));+ } catch {+ // Missing/locked DB or a bad log must not break the proxy hot path.+ invalidateTtfbBudgetCache();+ return baseBudgetMs;+ }+}