Skip to content

7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.

TypeScript

fix(health): opt-in parking of keys after a sustained transport-error streak (#1510) (#1530)

tashdroid9a0ad4b7810e
diff --git a/docs/en/env/01-variables.md b/docs/en/env/01-variables.md
index c8bd714c..9325f388 100644
--- a/docs/en/env/01-variables.md
+++ b/docs/en/env/01-variables.md
@@ -70,6 +70,7 @@ First-run note: the first dashboard account is normally created from a browser o
| `VALIDATE_TOOL_ARGUMENTS` | Off (`=1` enables) | Check tool-call arguments against the schema the caller declared and fail over when they violate it. `lib/tool-args.ts` already repairs what it can prove is repairable; this catches what is left, which today reaches Anthropic clients as a `tool_use` block with `input: {}`. Off by default because a false positive costs a failover hop — turn it on, watch `X-Fallback-Trail` for `invalid_tool_arguments`, then decide. Applies wherever the turn can still fail over (non-streaming surfaces, plus `/v1/chat/completions`, `/v1/messages`, Gemini and Ollama emulations); `/v1/responses` streaming commits on the first tool-call delta and is the exception. Runtime override via the `validate_tool_arguments` settings key. |
| `REQUEST_MAX_TOKENS_BUDGET` | `0` (off) | Per-request token guardrail: estimated input + requested max_tokens must fit this ceiling or the request is rejected with a 413 before any provider is tried; a request that sent no max_tokens gets its output capped to the remainder instead. Runtime-tunable via the `request_max_tokens_budget` settings key (dashboard API `PUT /api/settings/guardrails`). |
| `MAX_CONSECUTIVE_UPSTREAM_FAILS` | `0` (off) | Failover circuit breaker: after this many consecutive upstream failures in one request, stop the failover chain with a 503 instead of trying every remaining candidate of an unhealthy pool. Runtime-tunable via the `max_consecutive_upstream_fails` settings key. |
+| `HEALTH_TRANSPORT_PARK_AFTER` | `0` (off) | Opt-in: take a key out of rotation (status `error`) after this many health probes in a row fail with a network error (DNS, connection refused, timeout). `6` is the suggested value, about 30 minutes at the default 5-minute cadence. A parked key is re-probed every pass and comes back on the first successful probe or served request. Unset or `0` keeps the default keep-trying behaviour, where network errors never change a key's status. |
| `MODELSCOPE_VALIDATE_CACHE_MS` | `86400000` (24h) | ModelScope (魔搭) validates API keys with a paid 1-token chat completion (`GET /v1/models` does not enforce auth), spending magic-grain (魔粒) quota per probe — observed as 2 魔粒 per ultra-tier request. To stop the 5-minute health pass burning ~288 paid probes per key per day, a successful validation is cached per key for this long (milliseconds). `0` probes on every pass; a revoked key is still caught by the next real request's 401. |
| `FREELLMAPI_CONTEXT_HANDOFF` | Off | Context handoff on model switch. When enabled (`on_model_switch`), FreeLLMAPI injects a compact system message into the outbound request whenever a session switches from one model to another (e.g. after a fallback). |
diff --git a/server/src/__tests__/services/health-transport-park.test.ts b/server/src/__tests__/services/health-transport-park.test.ts
new file mode 100644
index 00000000..62a8973a
--- /dev/null
+++ b/server/src/__tests__/services/health-transport-park.test.ts
@@ -0,0 +1,151 @@
+import { describe, it, expect, beforeAll, afterAll, beforeEach, vi } from 'vitest';
+
+const validateKey = vi.fn();
+vi.mock('../../providers/index.js', async (importOriginal) => {
+ const actual = await importOriginal() as any;
+ return {
+ ...actual,
+ resolveProvider: () => ({ name: 'fake', validateKey }),
+ };
+});
+
+const { initDb, getDb } = await import('../../db/index.js');
+const { encrypt } = await import('../../lib/crypto.js');
+const {
+ checkKeyHealth,
+ markKeyHealthyFromRequest,
+ clearTransportFailureState,
+ SUGGESTED_TRANSPORT_PARK_AFTER: CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK,
+} = await import('../../services/health.js');
+
+const ORIGINAL_PARK_AFTER = process.env.HEALTH_TRANSPORT_PARK_AFTER;
+
+/**
+ * #1510: transport errors (DNS/timeout/TLS) never accumulated, so a key whose
+ * provider has been unreachable for days kept status='healthy' and stayed a
+ * first-class routing candidate — a real install logged 264 failures vs 8
+ * successes in a day with zero keys ever showing 'error'. The fix counts
+ * consecutive transport errors per key and parks the key at 'error' once the
+ * streak passes the threshold. 'error' is deliberately NOT auto-disable: the
+ * key is re-probed every pass and self-heals on a valid probe or a served
+ * request.
+ */
+describe('transport-error parking (#1510)', () => {
+ let nextId = 9000;
+
+ beforeAll(() => {
+ process.env.ENCRYPTION_KEY = '0'.repeat(64);
+ initDb(':memory:');
+ });
+
+ afterAll(() => {
+ if (ORIGINAL_PARK_AFTER === undefined) delete process.env.HEALTH_TRANSPORT_PARK_AFTER;
+ else process.env.HEALTH_TRANSPORT_PARK_AFTER = ORIGINAL_PARK_AFTER;
+ });
+
+ beforeEach(() => {
+ process.env.HEALTH_TRANSPORT_PARK_AFTER = String(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK);
+ validateKey.mockReset();
+ clearTransportFailureState();
+ vi.spyOn(console, 'error').mockImplementation(() => {});
+ vi.spyOn(console, 'warn').mockImplementation(() => {});
+ vi.spyOn(console, 'log').mockImplementation(() => {});
+ });
+
+ function seedKey(status: string): number {
+ const id = ++nextId;
+ const enc = encrypt(`park-${id}`);
+ getDb().prepare(`
+ INSERT INTO api_keys (id, platform, label, encrypted_key, iv, auth_tag, enabled, status)
+ VALUES (?, 'mistral', ?, ?, ?, ?, 1, ?)
+ `).run(id, `park-${id}`, enc.encrypted, enc.iv, enc.authTag, status);
+ return id;
+ }
+
+ function statusOf(id: number): string {
+ return (getDb().prepare('SELECT status FROM api_keys WHERE id = ?').get(id) as { status: string }).status;
+ }
+
+ async function failTransport(times: number, id: number): Promise<void> {
+ validateKey.mockRejectedValue(new Error('getaddrinfo ENOTFOUND api.mistral.ai'));
+ for (let i = 0; i < times; i++) await checkKeyHealth(id);
+ }
+
+ it('parking is off by default: a long unreachable streak keeps the key in rotation', async () => {
+ delete process.env.HEALTH_TRANSPORT_PARK_AFTER;
+ const id = seedKey('healthy');
+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK * 3, id);
+ expect(statusOf(id)).toBe('healthy');
+ });
+
+ it('HEALTH_TRANSPORT_PARK_AFTER=0 also keeps parking off', async () => {
+ process.env.HEALTH_TRANSPORT_PARK_AFTER = '0';
+ const id = seedKey('healthy');
+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK * 3, id);
+ expect(statusOf(id)).toBe('healthy');
+ });
+
+ it('a short unreachable streak never parks the key', async () => {
+ const id = seedKey('healthy');
+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK - 1, id);
+ expect(statusOf(id)).toBe('healthy');
+ });
+
+ it('a sustained unreachable streak parks the key at error', async () => {
+ const id = seedKey('healthy');
+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK, id);
+ expect(statusOf(id)).toBe('error');
+ });
+
+ it('the parking probe itself reports error', async () => {
+ const id = seedKey('healthy');
+ validateKey.mockRejectedValue(new Error('connect ECONNREFUSED'));
+ for (let i = 1; i < CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK; i++) {
+ expect(await checkKeyHealth(id)).toBe('healthy');
+ }
+ expect(await checkKeyHealth(id)).toBe('error');
+ });
+
+ it('a valid probe in the middle resets the streak', async () => {
+ const id = seedKey('healthy');
+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK - 1, id);
+ validateKey.mockResolvedValue(true);
+ expect(await checkKeyHealth(id)).toBe('healthy');
+ // A full fresh streak is needed from here on.
+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK - 1, id);
+ expect(statusOf(id)).toBe('healthy');
+ });
+
+ it('a 429 counts as reachability, not another transport failure', async () => {
+ const id = seedKey('healthy');
+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK - 1, id);
+ const err = Object.assign(new Error('429 Too Many Requests'), { status: 429 });
+ validateKey.mockRejectedValue(err);
+ expect(await checkKeyHealth(id)).toBe('healthy');
+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK - 1, id);
+ expect(statusOf(id)).toBe('healthy');
+ });
+
+ it('a parked key heals on the next valid probe', async () => {
+ const id = seedKey('healthy');
+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK, id);
+ expect(statusOf(id)).toBe('error');
+ validateKey.mockResolvedValue(true);
+ expect(await checkKeyHealth(id)).toBe('healthy');
+ expect(statusOf(id)).toBe('healthy');
+ });
+
+ it('a parked key heals when it serves a live request', async () => {
+ const id = seedKey('healthy');
+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK, id);
+ expect(statusOf(id)).toBe('error');
+ markKeyHealthyFromRequest(id);
+ expect(statusOf(id)).toBe('healthy');
+ });
+
+ it('parking never overwrites a confirmed-invalid verdict', async () => {
+ const id = seedKey('invalid');
+ await failTransport(CONSECUTIVE_TRANSPORT_ERRORS_TO_PARK, id);
+ expect(statusOf(id)).toBe('invalid');
+ });
+});
diff --git a/server/src/services/health.ts b/server/src/services/health.ts
index 35a0c425..12a785b8 100644
--- a/server/src/services/health.ts
+++ b/server/src/services/health.ts
@@ -63,6 +63,67 @@ function positiveIntEnv(raw: string | undefined, fallback: number): number {
// Track consecutive failures per key
const failureCount = new Map<number, number>();
+// #1510: consecutive transport errors per key. A key whose provider has been
+// unreachable (DNS, connection refused, TCP timeout) for this many probes in a
+// row is parked at status='error'. Unlike 'invalid' this is NOT an auto-disable:
+// 'error' keys are re-probed on every pass (never recency-skipped — see the
+// checkAllKeys due filter) and any live request that serves restores them
+// (markKeyHealthyFromRequest), so a provider outage that heals needs no
+// operator action. At the 5-minute cadence the threshold is ~30 minutes of
+// total unreachability — far past any single blip, and the point where a key
+// still marked 'healthy' is provably worse than no key: the router keeps
+// handing it traffic, every attempt dies, and failover costs the request its
+// latency budget (#1510 measured 264 failures vs 8 successes in a day on a
+// 39-key install with zero keys ever showing 'error').
+//
+// Opt-in: HEALTH_TRANSPORT_PARK_AFTER sets the streak length (6 is the
+// suggested value, ~30 minutes at the default cadence). Unset or 0 keeps the
+// default "keep trying" behaviour: a transport error never takes a key out of
+// rotation, because a key the router stops using can only come back through a
+// probe, and a provider whose validation endpoint is down while chat still
+// works would otherwise lose every key at once.
+export const SUGGESTED_TRANSPORT_PARK_AFTER = 6;
+
+/** Streak length that parks a key, or 0 when parking is off (the default). */
+export function getTransportParkAfter(): number {
+ return positiveIntEnv(process.env.HEALTH_TRANSPORT_PARK_AFTER, 0);
+}
+
+const transportFailures = new Map<number, number>();
+
+/** Test seam: drop transport-parking state. */
+export function clearTransportFailureState(): void {
+ transportFailures.clear();
+}
+
+/** Increment and apply the park rule. A 429 never reaches here — the caller
+ * treats it as proof the provider is reachable. Returns true when this call
+ * parked the key (status written to 'error'). */
+function recordTransportFailure(keyId: number, platform: string | undefined): boolean {
+ const parkAfter = getTransportParkAfter();
+ if (parkAfter === 0) return false;
+ const count = (transportFailures.get(keyId) ?? 0) + 1;
+ transportFailures.set(keyId, count);
+ if (count < parkAfter) return false;
+
+ // Park: 'error' is out of rotation (the router only routes healthy/unknown)
+ // but stays eligible for re-probing and self-heals. Never touch 'invalid' —
+ // a confirmed-bad credential verdict outranks network evidence in both
+ // directions, and only a successful validateKey clears it.
+ const db = getDb();
+ const res = db
+ .prepare("UPDATE api_keys SET status = 'error' WHERE id = ? AND enabled = 1 AND status IN ('healthy', 'unknown')")
+ .run(keyId);
+ if (res.changes > 0) {
+ providerLog(
+ 'warn',
+ `[Health] Key ${keyId} (${platform}) parked as 'error' after ${count} consecutive transport failures`,
+ { provider: platform, event: 'key_transport_park' },
+ );
+ }
+ return res.changes > 0;
+}
+
// A key whose last validation was rate-limited (429) is skipped by the
// scheduled pass until the skip window passes: re-asking a provider that just
// told us to slow down is how validation traffic becomes a ban. LLMTR's
@@ -229,6 +290,9 @@ export async function checkKeyHealth(keyId: number): Promise<KeyStatus> {
if (isValid) {
failureCount.delete(keyId);
+ // #1510: the provider answered — whatever unreachable streak this key had
+ // is over, so the transport counter starts from zero again.
+ transportFailures.delete(keyId);
// The provider answered again: whatever rate-limit told us to back off
// is over, so the key rejoins the normal cadence.
validationRateLimitedAt.delete(keyId);
@@ -270,17 +334,28 @@ export async function checkKeyHealth(keyId: number): Promise<KeyStatus> {
`[Health] Key ${keyId} (${row.platform}, base=${row.base_url ?? 'default'}) ` +
`transport error: ${lastError} — status preserved as '${row.status}'`,
);
- // Do NOT write status='error'. selectKeyForModel only considers keys with
- // status IN ('healthy','unknown'), so demoting here silently removes the
- // key's capacity for up to a full check interval — and a transport error is
- // evidence about the network, not about the key. One flaky DNS lookup, a
- // laptop suspended for thirty seconds, or a provider edge outage would take
- // every key on that provider out of rotation at once. Record the diagnostic
- // and the timestamp; leave the verdict to a probe that actually reached the
- // provider. Confirmed 401/403 (the isValid=false path above) still demotes.
+ // Do NOT write status='error' on the first transport failure. selectKeyForModel
+ // only considers keys with status IN ('healthy','unknown'), so demoting here
+ // silently removes the key's capacity for up to a full check interval — and a
+ // transport error is evidence about the network, not about the key. One flaky
+ // DNS lookup, a laptop suspended for thirty seconds, or a provider edge outage
+ // would take every key on that provider out of rotation at once. Record the
+ // diagnostic and the timestamp; leave the verdict to a probe that actually
+ // reached the provider. Confirmed 401/403 (the isValid=false path above) still
+ // demotes.
+ //
+ // #1510 adds the accumulation this path was missing (opt-in via
+ // HEALTH_TRANSPORT_PARK_AFTER): a key that has been unreachable that many
+ // probes in a row is parked at 'error' — still re-probed every pass, self-healing on the first answer
+ // or served request, but no longer first-class routing capacity. A 429 is
+ // proof the provider is reachable, so it counts as a break in the streak,
+ // not an increment.
+ let parked = false;
+ if (err?.status === 429) transportFailures.delete(keyId);
+ else parked = recordTransportFailure(keyId, row.platform);
db.prepare("UPDATE api_keys SET last_health_error = ?, last_checked_at = datetime('now') WHERE id = ?")
.run(lastError, keyId);
- return row.status as KeyStatus;
+ return (parked ? 'error' : row.status) as KeyStatus;
}
}
@@ -342,6 +417,9 @@ export function markKeyHealthyFromRequest(keyId: number): void {
.prepare("UPDATE api_keys SET status = 'healthy', last_health_error = NULL WHERE id = ? AND status IN ('error', 'unknown')")
.run(keyId);
failureCount.delete(keyId);
+ // #1510: a served request is the strongest reachability evidence there is —
+ // it also ends any transport-error streak, parked or not.
+ transportFailures.delete(keyId);
} catch {
// Never let health bookkeeping break a request that already succeeded.
}