7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.
TypeScript
fix(health): opt-in parking of keys after a sustained transport-error streak (#1510) (#1530)
tashdroid9a0ad4b7810e
diff --git a/docs/en/env/01-variables.md b/docs/en/env/01-variables.mdindex c8bd714c..9325f388 100644--- a/docs/en/env/01-variables.md+++ b/docs/en/env/01-variables.md@@ -70,6 +70,7 @@ First-run note: the first dashboard account is normally created from a browser o| `VALIDATE_TOOL_ARGUMENTS` | Off (`=1` enables) | Check tool-call arguments against the schema the caller declared and fail over when they violate it. `lib/tool-args.ts` already repairs what it can prove is repairable; this catches what is left, which today reaches Anthropic clients as a `tool_use` block with `input: {}`. Off by default because a false positive costs a failover hop — turn it on, watch `X-Fallback-Trail` for `invalid_tool_arguments`, then decide. Applies wherever the turn can still fail over (non-streaming surfaces, plus `/v1/chat/completions`, `/v1/messages`, Gemini and Ollama emulations); `/v1/responses` streaming commits on the first tool-call delta and is the exception. Runtime override via the `validate_tool_arguments` settings key. || `REQUEST_MAX_TOKENS_BUDGET` | `0` (off) | Per-request token guardrail: estimated input + requested max_tokens must fit this ceiling or the request is rejected with a 413 before any provider is tried; a request that sent no max_tokens gets its output capped to the remainder instead. Runtime-tunable via the `request_max_tokens_budget` settings key (dashboard API `PUT /api/settings/guardrails`). || `MAX_CONSECUTIVE_UPSTREAM_FAILS` | `0` (off) | Failover circuit breaker: after this many consecutive upstream failures in one request, stop the failover chain with a 503 instead of trying every remaining candidate of an unhealthy pool. Runtime-tunable via the `max_consecutive_upstream_fails` settings key. |+| `HEALTH_TRANSPORT_PARK_AFTER` | `0` (off) | Opt-in: take a key out of rotation (status `error`) after this many health probes in a row fail with a network error (DNS, connection refused, timeout). `6` is the suggested value, about 30 minutes at the default 5-minute cadence. A parked key is re-probed every pass and comes back on the first successful probe or served request. Unset or `0` keeps the default keep-trying behaviour, where network errors never change a key's status. || `MODELSCOPE_VALIDATE_CACHE_MS` | `86400000` (24h) | ModelScope (魔搭) validates API keys with a paid 1-token chat completion (`GET /v1/models` does not enforce auth), spending magic-grain (魔粒) quota per probe — observed as 2 魔粒 per ultra-tier request. To stop the 5-minute health pass burning ~288 paid probes per key per day, a successful validation is cached per key for this long (milliseconds). `0` probes on every pass; a revoked key is still caught by the next real request's 401. || `FREELLMAPI_CONTEXT_HANDOFF` | Off | Context handoff on model switch. When enabled (`on_model_switch`), FreeLLMAPI injects a compact system message into the outbound request whenever a session switches from one model to another (e.g. after a fallback). |diff --git a/server/src/__tests__/services/health-transport-park.test.ts b/server/src/__tests__/services/health-transport-park.test.tsnew file mode 100644index 00000000..62a8973a--- /dev/null+++ b/server/src/__tests__/services/health-transport-park.test.ts@@ -0,0 +1,151 @@+import { describe, it, expect, beforeAll, afterAll, beforeEach, vi } from 'vitest';++const validateKey = vi.fn();+vi.mock('../../providers/index.js', async (importOriginal) => {+ const actual = await importOriginal() as any;+ return {+ ...actual,+ resolveProvider: () => ({ name: 'fake', validateKey }),+ };+});++const { initDb, getDb } = await import('../../db/index.js');+const { encrypt } = await import('../../lib/crypto.js');+const {+ checkKeyHealth,+ markKeyHealthyFromRequest,+ clearTransportFailureState,+ SUGGESTED_TRANSPORT_PARK_AFTER: CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK,+} = await import('../../services/health.js');++const ORIGINAL_PARK_AFTER = process.env.HEALTH_TRANSPORT_PARK_AFTER;++/**+ * #1510: transport errors (DNS/timeout/TLS) never accumulated, so a key whose+ * provider has been unreachable for days kept status='healthy' and stayed a+ * first-class routing candidate — a real install logged 264 failures vs 8+ * successes in a day with zero keys ever showing 'error'. The fix counts+ * consecutive transport errors per key and parks the key at 'error' once the+ * streak passes the threshold. 'error' is deliberately NOT auto-disable: the+ * key is re-probed every pass and self-heals on a valid probe or a served+ * request.+ */+describe('transport-error parking (#1510)', () => {+ let nextId = 9000;++ beforeAll(() => {+ process.env.ENCRYPTION_KEY = '0'.repeat(64);+ initDb(':memory:');+ });++ afterAll(() => {+ if (ORIGINAL_PARK_AFTER === undefined) delete process.env.HEALTH_TRANSPORT_PARK_AFTER;+ else process.env.HEALTH_TRANSPORT_PARK_AFTER = ORIGINAL_PARK_AFTER;+ });++ beforeEach(() => {+ process.env.HEALTH_TRANSPORT_PARK_AFTER = String(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK);+ validateKey.mockReset();+ clearTransportFailureState();+ vi.spyOn(console, 'error').mockImplementation(() => {});+ vi.spyOn(console, 'warn').mockImplementation(() => {});+ vi.spyOn(console, 'log').mockImplementation(() => {});+ });++ function seedKey(status: string): number {+ const id = ++nextId;+ const enc = encrypt(`park-${id}`);+ getDb().prepare(`+ INSERT INTO api_keys (id, platform, label, encrypted_key, iv, auth_tag, enabled, status)+ VALUES (?, 'mistral', ?, ?, ?, ?, 1, ?)+ `).run(id, `park-${id}`, enc.encrypted, enc.iv, enc.authTag, status);+ return id;+ }++ function statusOf(id: number): string {+ return (getDb().prepare('SELECT status FROM api_keys WHERE id = ?').get(id) as { status: string }).status;+ }++ async function failTransport(times: number, id: number): Promise<void> {+ validateKey.mockRejectedValue(new Error('getaddrinfo ENOTFOUND api.mistral.ai'));+ for (let i = 0; i < times; i++) await checkKeyHealth(id);+ }++ it('parking is off by default: a long unreachable streak keeps the key in rotation', async () => {+ delete process.env.HEALTH_TRANSPORT_PARK_AFTER;+ const id = seedKey('healthy');+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK * 3, id);+ expect(statusOf(id)).toBe('healthy');+ });++ it('HEALTH_TRANSPORT_PARK_AFTER=0 also keeps parking off', async () => {+ process.env.HEALTH_TRANSPORT_PARK_AFTER = '0';+ const id = seedKey('healthy');+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK * 3, id);+ expect(statusOf(id)).toBe('healthy');+ });++ it('a short unreachable streak never parks the key', async () => {+ const id = seedKey('healthy');+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK - 1, id);+ expect(statusOf(id)).toBe('healthy');+ });++ it('a sustained unreachable streak parks the key at error', async () => {+ const id = seedKey('healthy');+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK, id);+ expect(statusOf(id)).toBe('error');+ });++ it('the parking probe itself reports error', async () => {+ const id = seedKey('healthy');+ validateKey.mockRejectedValue(new Error('connect ECONNREFUSED'));+ for (let i = 1; i < CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK; i++) {+ expect(await checkKeyHealth(id)).toBe('healthy');+ }+ expect(await checkKeyHealth(id)).toBe('error');+ });++ it('a valid probe in the middle resets the streak', async () => {+ const id = seedKey('healthy');+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK - 1, id);+ validateKey.mockResolvedValue(true);+ expect(await checkKeyHealth(id)).toBe('healthy');+ // A full fresh streak is needed from here on.+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK - 1, id);+ expect(statusOf(id)).toBe('healthy');+ });++ it('a 429 counts as reachability, not another transport failure', async () => {+ const id = seedKey('healthy');+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK - 1, id);+ const err = Object.assign(new Error('429 Too Many Requests'), { status: 429 });+ validateKey.mockRejectedValue(err);+ expect(await checkKeyHealth(id)).toBe('healthy');+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK - 1, id);+ expect(statusOf(id)).toBe('healthy');+ });++ it('a parked key heals on the next valid probe', async () => {+ const id = seedKey('healthy');+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK, id);+ expect(statusOf(id)).toBe('error');+ validateKey.mockResolvedValue(true);+ expect(await checkKeyHealth(id)).toBe('healthy');+ expect(statusOf(id)).toBe('healthy');+ });++ it('a parked key heals when it serves a live request', async () => {+ const id = seedKey('healthy');+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK, id);+ expect(statusOf(id)).toBe('error');+ markKeyHealthyFromRequest(id);+ expect(statusOf(id)).toBe('healthy');+ });++ it('parking never overwrites a confirmed-invalid verdict', async () => {+ const id = seedKey('invalid');+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK, id);+ expect(statusOf(id)).toBe('invalid');+ });+});diff --git a/server/src/services/health.ts b/server/src/services/health.tsindex 35a0c425..12a785b8 100644--- a/server/src/services/health.ts+++ b/server/src/services/health.ts@@ -63,6 +63,67 @@ function positiveIntEnv(raw: string | undefined, fallback: number): number {// Track consecutive failures per keyconst failureCount = new Map<number, number>();+// #1510: consecutive transport errors per key. A key whose provider has been+// unreachable (DNS, connection refused, TCP timeout) for this many probes in a+// row is parked at status='error'. Unlike 'invalid' this is NOT an auto-disable:+// 'error' keys are re-probed on every pass (never recency-skipped — see the+// checkAllKeys due filter) and any live request that serves restores them+// (markKeyHealthyFromRequest), so a provider outage that heals needs no+// operator action. At the 5-minute cadence the threshold is ~30 minutes of+// total unreachability — far past any single blip, and the point where a key+// still marked 'healthy' is provably worse than no key: the router keeps+// handing it traffic, every attempt dies, and failover costs the request its+// latency budget (#1510 measured 264 failures vs 8 successes in a day on a+// 39-key install with zero keys ever showing 'error').+//+// Opt-in: HEALTH_TRANSPORT_PARK_AFTER sets the streak length (6 is the+// suggested value, ~30 minutes at the default cadence). Unset or 0 keeps the+// default "keep trying" behaviour: a transport error never takes a key out of+// rotation, because a key the router stops using can only come back through a+// probe, and a provider whose validation endpoint is down while chat still+// works would otherwise lose every key at once.+export const SUGGESTED_TRANSPORT_PARK_AFTER = 6;++/** Streak length that parks a key, or 0 when parking is off (the default). */+export function getTransportParkAfter(): number {+ return positiveIntEnv(process.env.HEALTH_TRANSPORT_PARK_AFTER, 0);+}++const transportFailures = new Map<number, number>();++/** Test seam: drop transport-parking state. */+export function clearTransportFailureState(): void {+ transportFailures.clear();+}++/** Increment and apply the park rule. A 429 never reaches here — the caller+ * treats it as proof the provider is reachable. Returns true when this call+ * parked the key (status written to 'error'). */+function recordTransportFailure(keyId: number, platform: string | undefined): boolean {+ const parkAfter = getTransportParkAfter();+ if (parkAfter === 0) return false;+ const count = (transportFailures.get(keyId) ?? 0) + 1;+ transportFailures.set(keyId, count);+ if (count < parkAfter) return false;++ // Park: 'error' is out of rotation (the router only routes healthy/unknown)+ // but stays eligible for re-probing and self-heals. Never touch 'invalid' —+ // a confirmed-bad credential verdict outranks network evidence in both+ // directions, and only a successful validateKey clears it.+ const db = getDb();+ const res = db+ .prepare("UPDATE api_keys SET status = 'error' WHERE id = ? AND enabled = 1 AND status IN ('healthy', 'unknown')")+ .run(keyId);+ if (res.changes > 0) {+ providerLog(+ 'warn',+ `[Health] Key ${keyId} (${platform}) parked as 'error' after ${count} consecutive transport failures`,+ { provider: platform, event: 'key_transport_park' },+ );+ }+ return res.changes > 0;+}+// A key whose last validation was rate-limited (429) is skipped by the// scheduled pass until the skip window passes: re-asking a provider that just// told us to slow down is how validation traffic becomes a ban. LLMTR's@@ -229,6 +290,9 @@ export async function checkKeyHealth(keyId: number): Promise<KeyStatus> {if (isValid) {failureCount.delete(keyId);+ // #1510: the provider answered — whatever unreachable streak this key had+ // is over, so the transport counter starts from zero again.+ transportFailures.delete(keyId);// The provider answered again: whatever rate-limit told us to back off// is over, so the key rejoins the normal cadence.validationRateLimitedAt.delete(keyId);@@ -270,17 +334,28 @@ export async function checkKeyHealth(keyId: number): Promise<KeyStatus> {`[Health] Key ${keyId} (${row.platform}, base=${row.base_url ?? 'default'}) ` +`transport error: ${lastError} — status preserved as '${row.status}'`,);- // Do NOT write status='error'. selectKeyForModel only considers keys with- // status IN ('healthy','unknown'), so demoting here silently removes the- // key's capacity for up to a full check interval — and a transport error is- // evidence about the network, not about the key. One flaky DNS lookup, a- // laptop suspended for thirty seconds, or a provider edge outage would take- // every key on that provider out of rotation at once. Record the diagnostic- // and the timestamp; leave the verdict to a probe that actually reached the- // provider. Confirmed 401/403 (the isValid=false path above) still demotes.+ // Do NOT write status='error' on the first transport failure. selectKeyForModel+ // only considers keys with status IN ('healthy','unknown'), so demoting here+ // silently removes the key's capacity for up to a full check interval — and a+ // transport error is evidence about the network, not about the key. One flaky+ // DNS lookup, a laptop suspended for thirty seconds, or a provider edge outage+ // would take every key on that provider out of rotation at once. Record the+ // diagnostic and the timestamp; leave the verdict to a probe that actually+ // reached the provider. Confirmed 401/403 (the isValid=false path above) still+ // demotes.+ //+ // #1510 adds the accumulation this path was missing (opt-in via+ // HEALTH_TRANSPORT_PARK_AFTER): a key that has been unreachable that many+ // probes in a row is parked at 'error' — still re-probed every pass, self-healing on the first answer+ // or served request, but no longer first-class routing capacity. A 429 is+ // proof the provider is reachable, so it counts as a break in the streak,+ // not an increment.+ let parked = false;+ if (err?.status === 429) transportFailures.delete(keyId);+ else parked = recordTransportFailure(keyId, row.platform);db.prepare("UPDATE api_keys SET last_health_error = ?, last_checked_at = datetime('now') WHERE id = ?").run(lastError, keyId);- return row.status as KeyStatus;+ return (parked ? 'error' : row.status) as KeyStatus;}}@@ -342,6 +417,9 @@ export function markKeyHealthyFromRequest(keyId: number): void {.prepare("UPDATE api_keys SET status = 'healthy', last_health_error = NULL WHERE id = ? AND status IN ('error', 'unknown')").run(keyId);failureCount.delete(keyId);+ // #1510: a served request is the strongest reachability evidence there is —+ // it also ends any transport-error streak, parked or not.+ transportFailures.delete(keyId);} catch {// Never let health bookkeeping break a request that already succeeded.}