7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.
TypeScript
feat(health): HEALTH_CHECK_INTERVAL_MS tunes the pass cadence (#1593) (#1605)
tashdroid03eb408964e0
diff --git a/docs/en/env/01-variables.md b/docs/en/env/01-variables.mdindex ee2a73b6..83ee198f 100644--- a/docs/en/env/01-variables.md+++ b/docs/en/env/01-variables.md@@ -71,6 +71,7 @@ First-run note: the first dashboard account is normally created from a browser o| `REQUEST_MAX_TOKENS_BUDGET` | `0` (off) | Per-request token guardrail: estimated input + requested max_tokens must fit this ceiling or the request is rejected with a 413 before any provider is tried; a request that sent no max_tokens gets its output capped to the remainder instead. Runtime-tunable via the `request_max_tokens_budget` settings key (dashboard API `PUT /api/settings/guardrails`). || `MAX_CONSECUTIVE_UPSTREAM_FAILS` | `0` (off) | Failover circuit breaker: after this many consecutive upstream failures in one request, stop the failover chain with a 503 instead of trying every remaining candidate of an unhealthy pool. Runtime-tunable via the `max_consecutive_upstream_fails` settings key. || `HEALTH_TRANSPORT_PARK_AFTER` | `0` (off) | Opt-in: take a key out of rotation (status `error`) after this many health probes in a row fail with a network error (DNS, connection refused, timeout). `6` is the suggested value, about 30 minutes at the default 5-minute cadence. A parked key is re-probed every pass and comes back on the first successful probe or served request. Unset or `0` keeps the default keep-trying behaviour, where network errors never change a key's status. |+| `HEALTH_CHECK_INTERVAL_MS` | `300000` (5 min) | Base cadence of the scheduled key-health pass (#1593), in milliseconds. Operators with many keys on providers that dislike frequent key checks can slow the pass down (providers whose only key check is an empty search, such as SerpSearch, Exa and Parallel, are checked once per key regardless, #1599); a fresh install being shaken out can speed it up. Each pass fires at this cadence ±20% jitter; values below 60000 (or invalid) fall back to the 5-minute default so a misconfiguration can never turn the pass into provider-visible flood. Read at startup. || `MODELSCOPE_VALIDATE_CACHE_MS` | `86400000` (24h) | ModelScope (魔搭) validates API keys with a paid 1-token chat completion (`GET /v1/models` does not enforce auth), spending magic-grain (魔粒) quota per probe — observed as 2 魔粒 per ultra-tier request. To stop the 5-minute health pass burning ~288 paid probes per key per day, a successful validation is cached per key for this long (milliseconds). `0` probes on every pass; a revoked key is still caught by the next real request's 401. || `FREELLMAPI_CONTEXT_HANDOFF` | Off | Context handoff on model switch. When enabled (`on_model_switch`), FreeLLMAPI injects a compact system message into the outbound request whenever a session switches from one model to another (e.g. after a fallback). |diff --git a/server/src/__tests__/services/health-interval-env.test.ts b/server/src/__tests__/services/health-interval-env.test.tsnew file mode 100644index 00000000..f9c8b378--- /dev/null+++ b/server/src/__tests__/services/health-interval-env.test.ts@@ -0,0 +1,53 @@+import { describe, it, expect, afterEach, vi } from 'vitest';++// #1593: the health pass cadence was a hardcoded 5 minutes, which operators on+// providers that dislike frequent key checks cannot slow down.+// HEALTH_CHECK_INTERVAL_MS now configures it, with a 60s floor so a+// typo can never turn the pass into provider-visible flood. The constants are+// read at module load, so every case re-imports the module with a fresh cache.++const DEFAULT_MS = 5 * 60 * 1000;++async function loadHealth() {+ vi.resetModules();+ return import('../../services/health.js');+}++describe('HEALTH_CHECK_INTERVAL_MS env (#1593)', () => {+ afterEach(() => {+ vi.unstubAllEnvs();+ });++ it('defaults to 5 minutes when unset', async () => {+ const m = await loadHealth();+ expect(m.HEALTH_CHECK_INTERVAL_MS).toBe(DEFAULT_MS);+ });++ it('honours a slower cadence', async () => {+ vi.stubEnv('HEALTH_CHECK_INTERVAL_MS', '1800000'); // 30 min+ const m = await loadHealth();+ expect(m.HEALTH_CHECK_INTERVAL_MS).toBe(1_800_000);+ // The jittered next-pass delay follows the configured cadence.+ expect(m.nextHealthCheckDelayMs(() => 0.5)).toBe(1_800_000);+ });++ it('honours a faster cadence down to the 60s floor', async () => {+ vi.stubEnv('HEALTH_CHECK_INTERVAL_MS', '60000');+ const m = await loadHealth();+ expect(m.HEALTH_CHECK_INTERVAL_MS).toBe(60_000);+ // The recency skip must stay under the interval or passes skip whole keys.+ expect(m.RECENT_CHECK_SKIP_MS).toBeLessThan(m.HEALTH_CHECK_INTERVAL_MS);+ });++ it('falls back to the default below the floor (flood guard)', async () => {+ vi.stubEnv('HEALTH_CHECK_INTERVAL_MS', '1000');+ const m = await loadHealth();+ expect(m.HEALTH_CHECK_INTERVAL_MS).toBe(DEFAULT_MS);+ });++ it('ignores junk values', async () => {+ vi.stubEnv('HEALTH_CHECK_INTERVAL_MS', 'not-a-number');+ const m = await loadHealth();+ expect(m.HEALTH_CHECK_INTERVAL_MS).toBe(DEFAULT_MS);+ });+});diff --git a/server/src/services/health.ts b/server/src/services/health.tsindex 7d606248..208cc07b 100644--- a/server/src/services/health.ts+++ b/server/src/services/health.ts@@ -12,11 +12,26 @@ import { sanitizeProviderErrorMessage } from '../lib/error-redaction.js';import { providerLog } from '../lib/server-logs.js';import { triggerBuiltinModelDiscovery } from './builtin-model-discovery.js';-const CHECK_INTERVAL_MS = 5 * 60 * 1000; // 5 minutes+const DEFAULT_CHECK_INTERVAL_MS = 5 * 60 * 1000; // 5 minutes+// Floor of 60s: a misconfigured tiny interval turns the health pass into a+// self-inflicted abuse-shaped flood, which is the exact traffic #553/#1599 are+// about. Anything below the floor falls back to the default.+const MIN_CHECK_INTERVAL_MS = 60 * 1000;const CONSECUTIVE_FAILURES_TO_DISABLE = 3;const DEFAULT_HEALTH_CHECK_CONCURRENCY = 8;const DEFAULT_MIN_SPACING_MS = 1000;+/** Base cadence of the scheduled pass (jittered per run, see+ * nextHealthCheckDelayMs), tunable via HEALTH_CHECK_INTERVAL_MS (#1593):+ * fleets on providers that dislike frequent key checks+ * want a slower pass; a fresh install being shaken out wants a faster one.+ * Read once at module load — the scheduler re-arms from the same constant. */+function getCheckIntervalMs(): number {+ const n = positiveIntEnv(process.env.HEALTH_CHECK_INTERVAL_MS, DEFAULT_CHECK_INTERVAL_MS);+ return n < MIN_CHECK_INTERVAL_MS ? DEFAULT_CHECK_INTERVAL_MS : n;+}+const CHECK_INTERVAL_MS = getCheckIntervalMs();+/** Base cadence of the scheduled pass (jittered per run, see* nextHealthCheckDelayMs). Exported for tests. */export const HEALTH_CHECK_INTERVAL_MS = CHECK_INTERVAL_MS;@@ -29,9 +44,10 @@ const CHECK_INTERVAL_JITTER = 0.2;// A key validated more recently than this is skipped by the scheduled pass:// re-asking a provider about a credential it just answered for is exactly the-// traffic #553 is about. Below the jittered minimum interval (4 minutes) so a-// short interval can never skip a whole pass' worth of keys.-export const RECENT_CHECK_SKIP_MS = 3.5 * 60 * 1000;+// traffic #553 is about. Capped at 70% of the configured interval so a faster+// HEALTH_CHECK_INTERVAL_MS can never skip a whole pass' worth of keys (the+// invariant nextHealthCheckDelayMs asserts).+export const RECENT_CHECK_SKIP_MS = Math.min(3.5 * 60 * 1000, CHECK_INTERVAL_MS * 0.7);// #1599: how often the scheduled pass re-asks about a key parked at 'error' on// a provider that must not be polled (supportsKeyPolling === false). Healthy