Skip to content

7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.

TypeScript

fix(llmtr): pace key-validation probes so a burst can't get the account banned (#1369) (#1388)

yi111316abc4f057d
diff --git a/docs/en/env/01-variables.md b/docs/en/env/01-variables.md
index 80abface..43c2b643 100644
--- a/docs/en/env/01-variables.md
+++ b/docs/en/env/01-variables.md
@@ -59,6 +59,7 @@ First-run note: the first dashboard account is normally created from a browser o
| `PROVIDER_STREAM_STALL_TIMEOUT_MS` | `90000` | Mid-stream inactivity timeout in milliseconds. If an SSE stream goes this long without a single byte AFTER the first byte arrived, the gateway gives up: before any output reached the client it fails over to the next candidate; after output was sent the stream ends with an error frame. Time-to-FIRST-byte is governed by `PROVIDER_TIMEOUT_<PLATFORM>` instead (whichever of the two is larger). Providers that queue heavily on free tiers can pause longer than 90s mid-stream — raise this if streamed responses stop early. `0` disables the stall watchdog entirely. |
| `PROVIDER_STREAM_STALL_TIMEOUT_<PLATFORM>` | Falls through to the global value above, then the built-in `90000` | Per-provider variant (platform name upper-cased). Precedence: per-platform > global > built-in 90s default. Example: `PROVIDER_STREAM_STALL_TIMEOUT_OLLAMA=240000` lets Ollama Cloud pause 4 minutes mid-stream without loosening the watchdog everywhere else. |
| `PROVIDER_DAILY_REQUEST_CAP_<PLATFORM>` | Built-in default per provider | Per-provider daily request cap (platform name upper-cased). Overrides the built-in default for that provider; set to `0` to disable the cap. Example: `PROVIDER_DAILY_REQUEST_CAP_OPENROUTER=50`. |
+| `LLMTR_VALIDATION_MIN_GAP_MS` | `2000` | Minimum gap, in milliseconds, between the start of two LLMTR key-validation probes (issue #1369). Probes already run one at a time, and concurrent checks of the same key share a single request; this spaces the survivors so a burst of identical probes never leaves the gateway back to back. This is a floor for our own traffic, not an LLMTR-published limit — their docs state none for this endpoint — chosen because a burst of the same nonexistent-model POST from one IP is the pattern their security system reads as automation, and an account flagged that way gets suspended. Covers every caller: the 5-minute health pass, the dashboard's "check all", the post-wake re-probe, the cooldown-probe job and the 401-triggered revalidation. Raise it if your account is still being flagged; `0` disables pacing. Chat and streaming traffic is never paced. Read at startup. |
| `FALLBACK_TIME_BUDGET_MS` | `45000` | Base wall-clock budget for provider failover, in milliseconds. The initial attempt and one failover hop are always allowed. After that, exhaustion stops new attempts and can cancel an in-flight attempt still waiting for its first usable response. Streaming responses disarm this timer when the first usable response arrives. Successful TTFB history can widen the budget for a slow endpoint, as described below. `0` disables the budget entirely, including adaptive widening, and retries until the attempt cap. Also settable at runtime via the `fallback_time_budget_ms` settings key, which takes precedence. |
| `SLOW_ENDPOINT_BUFFER_MS` | `10000` | Padding added to the historical P95 successful time-to-first-byte (TTFB), in milliseconds. Samples are grouped by platform + `endpoint_scope`; the effective fallback budget is `max(base budget, P95 TTFB + buffer)`, capped at three times the base budget, and only for endpoints whose P95 is at least half the base budget. Endpoints with fewer than `TTFB_BUDGET_MIN_SAMPLES` samples, and those whose P95 + buffer fits the base budget, keep the base budget. `0` removes extra padding. The runtime `slow_endpoint_buffer_ms` settings key takes precedence over this environment variable. |
| `TTFB_BUDGET_WINDOW_MS` | `604800000` (7 days) | Rolling window of successful TTFB samples used for adaptive fallback budgets, in milliseconds. The runtime `ttfb_budget_window_ms` settings key takes precedence over this environment variable. |
diff --git a/docs/providers/llmtr.md b/docs/providers/llmtr.md
index 3341ab97..db817cb6 100644
--- a/docs/providers/llmtr.md
+++ b/docs/providers/llmtr.md
@@ -41,6 +41,20 @@ a deliberately nonexistent model without generating tokens. Live valid/invalid
controls returned `404/model_not_found` and `401/auth_error`, respectively.
Other errors remain inconclusive; quota exhaustion does not mark a key invalid.
+## Key-validation pacing
+
+LLMTR publishes no numeric rate limit for this endpoint, but a burst of the same
+nonexistent-model probe from one IP is the pattern their security system reads as
+automation, and an account flagged that way gets suspended — the report on #1369
+came from an account that was already locked out and learned of it from the
+provider's own email. Probes are therefore paced: they run one at a time, and
+concurrent checks of the same key share a single request. The gap between the
+start of two probes defaults to 2 seconds and is tunable with
+`LLMTR_VALIDATION_MIN_GAP_MS` (`0` disables it).
+This covers every caller of the probe — the 5-minute health pass, the dashboard's
+"check all" button, the post-wake re-probe, the cooldown-probe job and the
+401-triggered revalidation. Chat and streaming traffic is never paced.
+
Chat and streaming use the shared OpenAI-compatible transport, preserve usage and
backoff, and reject mismatched response model IDs rather than silently accepting
substitutions. No automatic paid fallback is introduced by this adapter.
diff --git a/docs/zh-cn/env/01-variables.md b/docs/zh-cn/env/01-variables.md
index 0c70acfe..fe7c63b0 100644
--- a/docs/zh-cn/env/01-variables.md
+++ b/docs/zh-cn/env/01-variables.md
@@ -57,6 +57,7 @@ FreeLLMAPI 从 `.env` 读取的全部变量,按主题分组。默认值和说
| `PROVIDER_STREAM_STALL_TIMEOUT_MS` | `90000` | 流中途的停滞超时,单位毫秒。如果一条 SSE 流在首字节已到达之后持续这么久没有任何字节,网关就放弃:若尚无输出到达客户端,则故障转移到下一个候选;若已有输出,则以一个错误帧结束流。到「第一个」字节为止的时间由 `PROVIDER_TIMEOUT_<PLATFORM>` 管(两者取较大者)。在免费额度上排长队的提供方可能在流中途停顿超过 90 秒——如果流式响应总是提前中断,就调高这个值。`0` 彻底关闭停滞看门狗。 |
| `PROVIDER_STREAM_STALL_TIMEOUT_<PLATFORM>` | 先回落到上面的全局值,再到内置的 `90000` | 按提供方的变体(平台名大写)。优先级:按平台 > 全局 > 内置 90 秒默认。示例:`PROVIDER_STREAM_STALL_TIMEOUT_OLLAMA=240000` 允许 Ollama Cloud 在流中途停顿 4 分钟,而其他地方的看门狗不受影响。 |
| `PROVIDER_DAILY_REQUEST_CAP_<PLATFORM>` | 各提供方内置的默认值 | 按提供方的每日请求上限(平台名大写)。覆盖该提供方的内置默认值;设为 `0` 关闭上限。示例:`PROVIDER_DAILY_REQUEST_CAP_OPENROUTER=50`。 |
+| `LLMTR_VALIDATION_MIN_GAP_MS` | `2000` | 两次 LLMTR 密钥校验探测开始之间至少间隔的毫秒数(issue #1369)。探测本来就是逐个串行发出的,同一密钥的并发检查也共用一次请求;这个值为其余探测拉开间距,使一串相同的探测绝不会紧接着离开网关。这是我们自己流量的下限,不是 LLMTR 公布的限制——他们的文档对该端点没有给出任何限制——之所以这样设,是因为同一 IP 短时间内反复发出相同的「不存在的模型」POST,正是他们的安全系统识别为自动化的模式,被标记的账号会被停用。它覆盖所有调用方:5 分钟健康巡检、仪表盘的「全部检查」、唤醒后的重新探测、冷却探测任务,以及 401 触发的重新校验。如果账号仍被标记,请调高该值;`0` 关闭节流。聊天与流式流量从不节流。启动时读取。 |
| `FALLBACK_TIME_BUDGET_MS` | `45000` | 提供方故障转移的基础挂钟时间预算,单位毫秒。始终允许首次尝试及一次故障转移;之后预算耗尽时会停止发起新尝试,也可能取消仍在等待首个有效响应的进行中尝试。流式响应收到首个有效响应后会解除此计时器。成功请求的首字节耗时历史可为慢端点延长预算,详见下方。`0` 彻底关闭该预算(包括自适应延长),一直重试直到触及尝试次数上限。也可以在运行时通过 `fallback_time_budget_ms` 设置键修改,后者优先。 |
| `SLOW_ENDPOINT_BUFFER_MS` | `10000` | 在历史成功请求的首字节耗时(TTFB)P95 上增加的缓冲,单位毫秒。样本按平台 + `endpoint_scope` 分组,有效故障转移预算为 `max(基础预算, P95 TTFB + 缓冲)`,上限为基础预算的三倍,且仅适用于 P95 不低于基础预算一半的端点。样本数少于 `TTFB_BUDGET_MIN_SAMPLES` 的端点,以及 P95 加缓冲不超过基础预算的端点,仍使用基础预算。`0` 表示不额外增加缓冲。运行时的 `slow_endpoint_buffer_ms` 设置键优先于此环境变量。 |
| `TTFB_BUDGET_WINDOW_MS` | `604800000`(7 天) | 计算自适应故障转移预算时使用的成功 TTFB 样本滚动时间窗口,单位毫秒。运行时的 `ttfb_budget_window_ms` 设置键优先于此环境变量。 |
diff --git a/server/src/__tests__/providers/llmtr-validation-pacing.test.ts b/server/src/__tests__/providers/llmtr-validation-pacing.test.ts
new file mode 100644
index 00000000..3677ff93
--- /dev/null
+++ b/server/src/__tests__/providers/llmtr-validation-pacing.test.ts
@@ -0,0 +1,231 @@
+import { afterEach, describe, expect, it, vi } from 'vitest';
+import {
+ LlmtrProvider,
+ LLMTR_VALIDATION_GAP_ENV,
+ LLMTR_VALIDATION_MIN_GAP_MS,
+ type LlmtrProviderOptions,
+} from '../../providers/llmtr.js';
+
+// #1369: every key-validation probe is the same nonexistent-model POST, and a
+// burst of them from one IP is the pattern LLMTR's security system reads as
+// automation — an account flagged that way gets suspended. validateKey is the
+// only door to the network, so pacing there covers every caller: the 5-minute
+// health pass, the forced pass behind the dashboard's "check all" button, the
+// post-wake re-probe, the cooldown-probe job and the 401-triggered revalidation.
+
+const modelNotFound = () => new Response(
+ JSON.stringify({ error: { type: 'model_not_found', message: 'Model not found' } }),
+ { status: 404 },
+);
+
+/** Fake clock driven by the injected sleep, so pacing costs no real time. */
+function fakeClock() {
+ const state = { t: 10_000 };
+ return {
+ now: () => state.t,
+ sleep: async (ms: number) => { state.t += ms; },
+ at: () => state.t,
+ };
+}
+
+/** Provider wired to the fake clock, recording the start time of each probe. */
+function pacedProvider(opts: LlmtrProviderOptions = {}) {
+ const clock = fakeClock();
+ const starts: number[] = [];
+ vi.spyOn(global, 'fetch').mockImplementation(async () => {
+ starts.push(clock.at());
+ return modelNotFound();
+ });
+ const provider = new LlmtrProvider({ now: clock.now, sleep: clock.sleep, ...opts });
+ return { provider, clock, starts };
+}
+
+afterEach(() => {
+ vi.restoreAllMocks();
+ delete process.env[LLMTR_VALIDATION_GAP_ENV];
+});
+
+describe('LLMTR key-validation probe pacing', () => {
+ it('leaves the first probe unspaced and spaces the rest by the gap floor', async () => {
+ const { provider, clock, starts } = pacedProvider({ validationMinGapMs: 2_000 });
+
+ await Promise.all([
+ provider.validateKey('key-a'),
+ provider.validateKey('key-b'),
+ provider.validateKey('key-c'),
+ ]);
+
+ // Nothing probed before this provider existed, so the floor is measured
+ // from the epoch-relative 0 rather than stalling the very first probe.
+ expect(starts).toEqual([10_000, 12_000, 14_000]);
+ expect(clock.at()).toBe(14_000);
+ });
+
+ it('spaces by the shipped default when no override is given', async () => {
+ const { provider, starts } = pacedProvider();
+
+ await provider.validateKey('key-a');
+ await provider.validateKey('key-b');
+
+ expect(starts[1]! - starts[0]!).toBe(LLMTR_VALIDATION_MIN_GAP_MS);
+ });
+
+ it('runs probes one at a time, so no two overlap', async () => {
+ let live = 0;
+ let maxLive = 0;
+ vi.spyOn(global, 'fetch').mockImplementation(async () => {
+ live++;
+ maxLive = Math.max(maxLive, live);
+ await new Promise(resolve => setTimeout(resolve, 5));
+ live--;
+ return modelNotFound();
+ });
+ const clock = fakeClock();
+ const provider = new LlmtrProvider({ now: clock.now, sleep: clock.sleep, validationMinGapMs: 0 });
+
+ await Promise.all([
+ provider.validateKey('key-a'),
+ provider.validateKey('key-b'),
+ provider.validateKey('key-c'),
+ ]);
+
+ expect(maxLive).toBe(1);
+ });
+
+ it('coalesces concurrent probes of the same credential into one request', async () => {
+ const fetchSpy = vi.spyOn(global, 'fetch').mockResolvedValue(modelNotFound());
+ const clock = fakeClock();
+ const provider = new LlmtrProvider({ now: clock.now, sleep: clock.sleep });
+
+ const verdicts = await Promise.all([
+ provider.validateKey('same-key'),
+ provider.validateKey('same-key'),
+ provider.validateKey('same-key'),
+ ]);
+
+ expect(fetchSpy).toHaveBeenCalledTimes(1);
+ expect(verdicts).toEqual([true, true, true]);
+ });
+
+ it('does not serve a later probe from a settled one', async () => {
+ const fetchSpy = vi.spyOn(global, 'fetch').mockResolvedValue(modelNotFound());
+ const clock = fakeClock();
+ const provider = new LlmtrProvider({ now: clock.now, sleep: clock.sleep, validationMinGapMs: 0 });
+
+ await provider.validateKey('same-key');
+ await provider.validateKey('same-key');
+
+ // The map entry is retired on settle: a key can be revoked between checks,
+ // and a cached "valid" would keep routing to a dead credential.
+ expect(fetchSpy).toHaveBeenCalledTimes(2);
+ });
+
+ it('shares one request even across a pacing wait', async () => {
+ let resolveFetch!: (value: Response) => void;
+ const fetchSpy = vi.spyOn(global, 'fetch').mockImplementation(() => new Promise<Response>(resolve => {
+ resolveFetch = resolve;
+ }));
+ const clock = fakeClock();
+ const provider = new LlmtrProvider({ now: clock.now, sleep: clock.sleep, validationMinGapMs: 5_000 });
+
+ const first = provider.validateKey('same-key');
+ await vi.waitFor(() => expect(fetchSpy).toHaveBeenCalledOnce());
+ const second = provider.validateKey('same-key');
+ resolveFetch(modelNotFound());
+
+ expect(await first).toBe(true);
+ expect(await second).toBe(true);
+ expect(fetchSpy).toHaveBeenCalledOnce();
+ expect(clock.at()).toBe(10_000);
+ });
+
+ it('keeps the queue moving after a failed probe', async () => {
+ const fetchSpy = vi.spyOn(global, 'fetch')
+ .mockResolvedValueOnce(new Response(JSON.stringify({ error: { type: 'other' } }), { status: 500 }))
+ .mockResolvedValue(modelNotFound());
+ const clock = fakeClock();
+ const provider = new LlmtrProvider({ now: clock.now, sleep: clock.sleep, validationMinGapMs: 1_000 });
+
+ await expect(provider.validateKey('key-a')).rejects.toMatchObject({ status: 500 });
+
+ await expect(provider.validateKey('key-b')).resolves.toBe(true);
+ expect(fetchSpy).toHaveBeenCalledTimes(2);
+ // The rejected probe still consumed its slot, so the next one waited the
+ // gap rather than firing immediately behind it.
+ expect(clock.at()).toBe(11_000);
+ });
+
+ it('retires a rejected probe so a retry is a fresh request', async () => {
+ const fetchSpy = vi.spyOn(global, 'fetch')
+ .mockRejectedValueOnce(new Error('ECONNRESET'))
+ .mockResolvedValue(modelNotFound());
+ const clock = fakeClock();
+ const provider = new LlmtrProvider({ now: clock.now, sleep: clock.sleep, validationMinGapMs: 0 });
+
+ await expect(provider.validateKey('key-a')).rejects.toThrow('ECONNRESET');
+ await expect(provider.validateKey('key-a')).resolves.toBe(true);
+
+ expect(fetchSpy).toHaveBeenCalledTimes(2);
+ });
+
+ it('does not pace when the gap is 0', async () => {
+ const { provider, clock, starts } = pacedProvider({ validationMinGapMs: 0 });
+
+ await provider.validateKey('key-a');
+ await provider.validateKey('key-b');
+ await provider.validateKey('key-c');
+
+ expect(starts).toEqual([10_000, 10_000, 10_000]);
+ expect(clock.at()).toBe(10_000);
+ });
+
+ it('reads the gap floor from the environment', async () => {
+ process.env[LLMTR_VALIDATION_GAP_ENV] = '750';
+ const { provider, starts } = pacedProvider();
+
+ await provider.validateKey('key-a');
+ await provider.validateKey('key-b');
+
+ expect(starts[1]! - starts[0]!).toBe(750);
+ });
+
+ it('falls back to the default for a nonsensical gap and says so once', async () => {
+ process.env[LLMTR_VALIDATION_GAP_ENV] = 'soon';
+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
+ const { provider, starts } = pacedProvider();
+
+ await provider.validateKey('key-a');
+ await provider.validateKey('key-b');
+
+ expect(starts[1]! - starts[0]!).toBe(LLMTR_VALIDATION_MIN_GAP_MS);
+ expect(warn).toHaveBeenCalledOnce();
+ expect(warn.mock.calls[0]![0]).toContain(LLMTR_VALIDATION_GAP_ENV);
+ });
+
+ it('leaves chat traffic unpaced — real requests must not queue behind probes', async () => {
+ const clock = fakeClock();
+ const bodies: any[] = [];
+ vi.spyOn(global, 'fetch').mockImplementation(async (_url, init) => {
+ const body = JSON.parse(String(init?.body));
+ bodies.push(body);
+ // The probe names the validation model; chat names a real route.
+ return body.model === '__freellmapi_key_validation__'
+ ? modelNotFound()
+ : new Response(JSON.stringify({
+ id: 'c', object: 'chat.completion', created: 1, model: body.model,
+ choices: [{ index: 0, message: { role: 'assistant', content: 'OK' }, finish_reason: 'stop' }],
+ usage: { prompt_tokens: 1, completion_tokens: 1, total_tokens: 2 },
+ }), { status: 200 });
+ });
+ const provider = new LlmtrProvider({ now: clock.now, sleep: clock.sleep, validationMinGapMs: 2_000 });
+
+ await provider.validateKey('key-a');
+ const afterProbe = clock.at();
+ await provider.chatCompletion('key-a', [{ role: 'user', content: 'OK' }], 'qwen/qwen3.8-27b-free');
+
+ // The chat request went out without waiting out the 2s floor a second probe
+ // would have owed: only the probe is paced, never real traffic.
+ expect(clock.at()).toBe(afterProbe);
+ expect(bodies.map(b => b.model)).toEqual(['__freellmapi_key_validation__', 'qwen/qwen3.8-27b-free']);
+ });
+});
\ No newline at end of file
diff --git a/server/src/__tests__/providers/llmtr.test.ts b/server/src/__tests__/providers/llmtr.test.ts
index 9b829c46..f69760a4 100644
--- a/server/src/__tests__/providers/llmtr.test.ts
+++ b/server/src/__tests__/providers/llmtr.test.ts
@@ -15,6 +15,11 @@ const sse = (returnedModel: string) => new Response([
{ id: 's', object: 'chat.completion.chunk', created: 1, model: returnedModel, choices: [], usage: completion.usage },
].map(c => `data: ${JSON.stringify(c)}\n\n`).join('') + 'data: [DONE]\n\n', { headers: { 'Content-Type': 'text/event-stream' } });
+// These cases assert the wire shape of one probe, so they run against a
+// provider with probe pacing off rather than waiting out the real gap floor
+// between cases. Pacing itself is covered in llmtr-validation-pacing.test.ts.
+const unpaced = () => new LlmtrProvider({ validationMinGapMs: 0 });
+
describe('LLMTR provider', () => {
afterEach(() => vi.restoreAllMocks());
@@ -60,7 +65,7 @@ describe('LLMTR provider', () => {
it('validates via authenticated model resolution, not the public roster', async () => {
const fetch = vi.spyOn(global, 'fetch').mockResolvedValue(json({ error: { type: 'model_not_found', message: 'Model not found' } }, 404));
- await expect(getProvider('llmtr')!.validateKey('test-key')).resolves.toBe(true);
+ await expect(unpaced().validateKey('test-key')).resolves.toBe(true);
expect(fetch).toHaveBeenCalledTimes(1);
expect(fetch.mock.calls[0][0]).toBe('https://llmtr.com/v1/chat/completions');
expect(JSON.parse(String(fetch.mock.calls[0][1]?.body))).toMatchObject({ model: '__freellmapi_key_validation__', max_tokens: 1, stream: false });
@@ -68,12 +73,12 @@ describe('LLMTR provider', () => {
it.each([401, 403])('rejects invalid credentials (%s)', async status => {
vi.spyOn(global, 'fetch').mockResolvedValue(json({ error: { type: 'auth_error', message: 'Invalid API key' } }, status));
- await expect(getProvider('llmtr')!.validateKey('bad-key')).resolves.toMatchObject({ valid: false });
+ await expect(unpaced().validateKey('bad-key')).resolves.toMatchObject({ valid: false });
});
it.each([200, 400, 402, 404, 429, 500])('keeps inconclusive validation errors inconclusive (%s)', async status => {
vi.spyOn(global, 'fetch').mockResolvedValue(json({ error: { type: 'other' } }, status, { 'Retry-After': '17' }));
- await expect(getProvider('llmtr')!.validateKey('test-key')).rejects.toMatchObject({ status, retryAfterMs: 17_000 });
+ await expect(unpaced().validateKey('test-key')).rejects.toMatchObject({ status, retryAfterMs: 17_000 });
});
it('#1390: a 403 refusing automated validation probes is inconclusive, never invalid', async () => {
@@ -89,7 +94,7 @@ describe('LLMTR provider', () => {
it('does not accept an HTML 404', async () => {
vi.spyOn(global, 'fetch').mockResolvedValue(new Response('not JSON', { status: 404 }));
- await expect(getProvider('llmtr')!.validateKey('test-key')).rejects.toMatchObject({ status: 404 });
+ await expect(unpaced().validateKey('test-key')).rejects.toMatchObject({ status: 404 });
});
it.each([402, 429])('preserves upstream quota status and backoff without retry (%s)', async status => {
diff --git a/server/src/providers/llmtr.ts b/server/src/providers/llmtr.ts
index c0720f34..0ae09167 100644
--- a/server/src/providers/llmtr.ts
+++ b/server/src/providers/llmtr.ts
@@ -7,6 +7,43 @@ import { recordQuotaObservationsFromResponse, type QuotaObservationContext } fro
export const LLMTR_BASE_URL = 'https://llmtr.com/v1';
const VALIDATION_MODEL = '__freellmapi_key_validation__';
+/**
+ * Floor between the START of two key-validation probes (#1369).
+ *
+ * NOT a published LLMTR rate limit — their docs state none for this endpoint.
+ * It is the politeness floor for OUR probe traffic: the same nonexistent-model
+ * POST, repeated back to back from one IP, is the burst pattern their security
+ * system reads as automation, and an account flagged that way gets suspended.
+ * Every caller of validateKey reaches the network through this one method, so
+ * pacing here covers the health pass, the forced pass behind the dashboard's
+ * "check all" button, the post-wake re-probe, the cooldown-probe job and the
+ * 401-triggered revalidation alike. Operators who know their account tolerates
+ * more can lower it; 0 disables pacing.
+ */
+export const LLMTR_VALIDATION_MIN_GAP_MS = 2_000;
+export const LLMTR_VALIDATION_GAP_ENV = 'LLMTR_VALIDATION_MIN_GAP_MS';
+
+function validationMinGapMs(): number {
+ const raw = process.env[LLMTR_VALIDATION_GAP_ENV];
+ if (raw !== undefined && raw.trim() !== '') {
+ const n = Number(raw);
+ if (Number.isInteger(n) && n >= 0) return n;
+ console.warn(
+ `[config] Ignoring ${LLMTR_VALIDATION_GAP_ENV}="${raw}": expected a non-negative integer of milliseconds (0 disables pacing). Using the default ${LLMTR_VALIDATION_MIN_GAP_MS}ms.`,
+ );
+ }
+ return LLMTR_VALIDATION_MIN_GAP_MS;
+}
+
+export interface LlmtrProviderOptions {
+ /** Override the probe pacing floor. 0 disables it. */
+ validationMinGapMs?: number;
+ /** Test seams: injectable clock and sleep, so a pacing test needs no real
+ * timers and no wall-clock time. */
+ now?: () => number;
+ sleep?: (ms: number) => Promise<void>;
+}
+
function checkModel(requested: string, returned: string): void {
if (returned !== requested) {
throw Object.assign(new Error('LLMTR returned a different or missing model identity'), { status: 502 });
@@ -26,11 +63,76 @@ async function blocksAutomatedValidation(res: Response): Promise<boolean> {
* monthly cash grant. Only the signed catalog supplies model rows: the public
* roster also contains paid models, expiring promotions and BYOK-only rows. */
export class LlmtrProvider extends OpenAICompatProvider {
- constructor() {
+ private readonly validationMinGapMs: number;
+ private readonly now: () => number;
+ private readonly sleep: (ms: number) => Promise<void>;
+ /** Tail of the probe queue. Resolves when the probe ahead of a caller is
+ * done, so validation requests run one at a time, in arrival order. */
+ private validationTail: Promise<void> = Promise.resolve();
+ /** Start time of the last probe that got its turn, for the gap floor. */
+ private lastValidationStartedAtMs = 0;
+ /** Probes already in flight, keyed by credential. Two callers asking about
+ * the same key at the same time (the health pass and the cooldown-probe job
+ * overlap easily) share one request instead of sending two identical ones. */
+ private readonly inFlightValidations = new Map<string, Promise<KeyValidationResult>>();
+
+ constructor(opts: LlmtrProviderOptions = {}) {
super({ platform: 'llmtr', name: 'LLMTR', baseUrl: LLMTR_BASE_URL });
+ this.validationMinGapMs = opts.validationMinGapMs ?? validationMinGapMs();
+ this.now = opts.now ?? Date.now;
+ this.sleep = opts.sleep ?? ((ms: number) => new Promise(resolve => setTimeout(resolve, ms)));
+ }
+
+ /**
+ * validateKey, paced and coalesced. Returns the same promise to concurrent
+ * callers asking about the same credential, so the shared verdict is the one
+ * question answered once. The merged callers' quota contexts are not
+ * summed — quota observations are booked per platform+key, and a single
+ * response carries a single set of headers to book.
+ */
+ override validateKey(apiKey: string, quotaContext?: QuotaObservationContext): Promise<KeyValidationResult> {
+ const inFlight = this.inFlightValidations.get(apiKey);
+ if (inFlight) return inFlight;
+
+ const probe = this.queuedValidation(() => this.probeKey(apiKey, quotaContext));
+ this.inFlightValidations.set(apiKey, probe);
+ // Drop the entry once settled so the next question is a fresh probe, but
+ // never before: a caller that arrives while this one is still open gets
+ // the answer it would otherwise have to pay for again.
+ void probe.then(
+ () => { this.retireValidation(apiKey, probe); },
+ () => { this.retireValidation(apiKey, probe); },
+ );
+ return probe;
+ }
+
+ private retireValidation(apiKey: string, probe: Promise<KeyValidationResult>): void {
+ if (this.inFlightValidations.get(apiKey) === probe) this.inFlightValidations.delete(apiKey);
+ }
+
+ /**
+ * Take a turn in the probe queue: wait for the probe ahead, then for the gap
+ * floor to elapse, then stamp the start and run. The queue is released in a
+ * finally so a rejected probe (timeout, 5xx) cannot stall the probes behind
+ * it for the rest of the process's life.
+ */
+ private async queuedValidation<T>(probe: () => Promise<T>): Promise<T> {
+ const ahead = this.validationTail;
+ let release!: () => void;
+ this.validationTail = new Promise<void>(resolve => { release = resolve; });
+ await ahead;
+ try {
+ const wait = this.lastValidationStartedAtMs + this.validationMinGapMs - this.now();
+ if (wait > 0) await this.sleep(wait);
+ this.lastValidationStartedAtMs = this.now();
+ return await probe();
+ } finally {
+ release();
+ }
}
- override async validateKey(apiKey: string, quotaContext?: QuotaObservationContext): Promise<KeyValidationResult> {
+ /** The probe itself: one authenticated request, verdict classified below. */
+ private async probeKey(apiKey: string, quotaContext?: QuotaObservationContext): Promise<KeyValidationResult> {
// /models is public; /api/usage expects a dashboard token. This nonexistent
// model checks authentication without generating tokens: live controls on
// 2026-09-29 returned 401/auth_error for an invalid key and