7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.
TypeScript
fix(cache): only buffer streams when cacheable, share the entry cap, count stream hits (#1197)
Tashfeen52214d573e00
diff --git a/.env.example b/.env.exampleindex 3dd79ae4..f0a0448f 100644--- a/.env.example+++ b/.env.example@@ -190,15 +190,16 @@ PORT=3001# remaining candidate of an unhealthy pool.# MAX_CONSECUTIVE_UPSTREAM_FAILS=0-# Opt-in response cache. When on, a successful non-streaming /v1/chat/completions-# answer is stored in a bounded in-memory LRU keyed by a canonical hash of the+# Opt-in response cache. When on, a successful /v1/chat/completions answer --+# streaming or not -- is stored in a bounded in-memory LRU keyed by a hash of the# request, so an IDENTICAL later request is served from memory without spending# any provider quota. Exact match only (no fuzzy matching), so a near-miss never# returns a different prompt's answer. Off by default; set to true to enable.# Can also be toggled at runtime via the dashboard (PUT /api/cache/config), and# per-request with the `X-FreeLLM-Cache: on|off` header.# RESPONSE_CACHE=false-# Persist cached answers to SQLite so they survive a restart (the daily+# Persist cached non-streaming answers to SQLite so they survive a restart+# (streamed replays are memory-only and never persisted) (the daily# quota-reset re-run pattern). Only applies when the cache above is on. Note# this writes PLAINTEXT model responses to the database file, which the# memory-only cache never did; set it to false to keep caching but hold@@ -211,7 +212,8 @@ PORT=3001# variety, so they are never cached. Default 1.0 caches everything when enabled;# lower it (e.g. 0.2) to cache only near-deterministic calls.# RESPONSE_CACHE_MAX_TEMPERATURE=1.0-# Hard cap on stored entries; the least-recently-used are evicted past this.+# Hard cap on stored entries, shared by the JSON and streaming stores; the+# least-recently-used are evicted past this.# RESPONSE_CACHE_MAX_ENTRIES=5000# Request-side prompt/context compression. The default is off. `lossless`diff --git a/docs/en/api/01-rest-api.md b/docs/en/api/01-rest-api.mdindex bda6d110..d7b6e547 100644--- a/docs/en/api/01-rest-api.md+++ b/docs/en/api/01-rest-api.md@@ -302,7 +302,7 @@ Only hops that already **failed** can appear. The hop actually serving your requHTTP headers only carry printable ASCII, so a model id with characters outside that range (a Chinese name from a relay catalog, for example) is percent-encoded in the header — run the value through `decodeURIComponent` (or `urllib.parse.unquote`) to read it back.-The opt-in response cache can be toggled per request with `X-FreeLLM-Cache: on|off` — an exact-match in-memory LRU for identical non-streaming requests (canonical SHA-256 keys over the full request, TTL and temperature gates, saved-token stats on the dashboard). Off by default; cache hits consume zero provider quota.+The opt-in response cache can be toggled per request with `X-FreeLLM-Cache: on|off` — an exact-match in-memory LRU for identical requests, streaming included (a streamed hit replays the stored SSE verbatim) (canonical SHA-256 keys over the full request, TTL and temperature gates, saved-token stats on the dashboard). Off by default; cache hits consume zero provider quota.When [prompt compression](../compression/01-compression-pipeline.md) is enabled, `X-FreeLLM-Compress: off|on|lossless|standard|aggressive` can disable or lower the configured mode for one request. It cannot raise the operator's configured mode. The response reports the effective mode and estimated savings, for example `X-FreeLLM-Compress: standard; saved~=1840`.diff --git a/docs/en/api/02-idempotency.md b/docs/en/api/02-idempotency.mdindex 48fa9cfa..3fc635b5 100644--- a/docs/en/api/02-idempotency.md+++ b/docs/en/api/02-idempotency.md@@ -24,7 +24,7 @@ Normalization (`normalizeIdempotencyKey` in `server/src/services/idempotency.ts:Only a `SHA-256 hex` of the key is stored (`hashIdempotencyKey`) — the raw key never touches SQLite, mirroring how admin/runtime tokens are handled.-Streaming requests (`stream: true`) **always bypass** idempotency (`const idemKey = !stream ? normalize... : null` in `proxy.ts:1805`). A stream cannot be replayed as a unit and the open connection is itself the retry signal — same policy as the response cache.+Streaming requests (`stream: true`) **always bypass** idempotency (`const idemKey = !stream ? normalize... : null` in `proxy.ts:1805`). A stream cannot be replayed as a unit for a *retry* and the open connection is itself the retry signal. (The response cache no longer follows this policy: it does replay streams, keyed on request content rather than a caller-supplied key.)---@@ -180,7 +180,7 @@ Tips:## 10. Relationship to response cache & degraded mode-- **Response cache** (`services/cache.ts`, `X-FreeLLM-Cache`) is orthogonal: global, temperature-gated, in-memory LRU; idempotency is durable (SQLite), per-caller, and fingerprint-bound. Both short-circuit provider calls.+- **Response cache** (`services/cache.ts`, `X-FreeLLM-Cache`) is orthogonal: global, temperature-gated, in-memory LRU covering streaming and non-streaming alike; idempotency is durable (SQLite), per-caller, fingerprint-bound, and non-streaming only. Both short-circuit provider calls.- **Degraded mode** ([`docs/en/architecture/04-degraded-mode-and-failover.md`](../architecture/04-degraded-mode-and-failover.md)) disables bandit exploration but does not affect idempotency — replays still bypass routing entirely.- **Failover**: only the final successful body is stored. If a request exhausts the fallback loop, nothing is stored for that key.diff --git a/docs/en/architecture/03-streaming-pipeline.md b/docs/en/architecture/03-streaming-pipeline.mdindex 91266baf..c9ece226 100644--- a/docs/en/architecture/03-streaming-pipeline.md+++ b/docs/en/architecture/03-streaming-pipeline.md@@ -253,7 +253,8 @@ Thin adapter over shared fallback loop:## 13. Response Cache (Opt-In)-- **Scope**: non-streaming only, cacheable temperature (`temperature ≤ 1` or unset), `X-FreeLLM-Cache: on` or `RESPONSE_CACHE=1`.+- **Scope**: streaming and non-streaming, cacheable temperature (`temperature ≤ 1` or unset), `X-FreeLLM-Cache: on` or `RESPONSE_CACHE=1`.- **Key**: SHA-256 over canonical request (model, messages, all sampling params, tools, stop, response_format, n, seed, penalties, logit_bias, logprobs, top_logprobs, reasoning_effort, compression).-- **Store**: in-memory LRU, TTL + temperature gates.-- **Hit**: returns cached body, `X-Routed-Via: cache`, `X-FreeLLM-Cache: HIT`, **zero provider quota consumed**.\ No newline at end of file+- **Store**: in-memory LRU, TTL + temperature gates. Two stores — JSON bodies (persisted to SQLite) and streaming SSE (memory-only) — sharing one `RESPONSE_CACHE_MAX_ENTRIES` budget; `entries` in the stats counts both.+- **Hit**: returns cached body, `X-Routed-Via: cache`, `X-FreeLLM-Cache: HIT`, **zero provider quota consumed**. A streaming hit replays the stored SSE byte for byte.+- **Streaming store**: frames are buffered only when the request is cacheable (nothing is retained with the cache off) and dropped past a 2 MB replay ceiling. A truncated (`finish_reason: length`), errored, or aborted stream is never stored.\ No newline at end of filediff --git a/docs/en/env/01-variables.md b/docs/en/env/01-variables.mdindex 98130714..9efff6fc 100644--- a/docs/en/env/01-variables.md+++ b/docs/en/env/01-variables.md@@ -97,11 +97,11 @@ In Docker, `127.0.0.1` is the container, not your machine — see [03-outbound-p| Variable | Default | Purpose || --- | --- | --- |-| `RESPONSE_CACHE` | `false` | Opt-in response cache. When on, a successful non-streaming `/v1/chat/completions` answer is stored in a bounded in-memory LRU keyed by a canonical hash of the request, so an IDENTICAL later request is served from memory without spending provider quota. Exact match only — a near-miss never returns a different prompt's answer. Toggled at runtime via the dashboard (`PUT /api/cache/config`) and per-request with the `X-FreeLLM-Cache: on\|off` header. |-| `RESPONSE_CACHE_PERSIST` | `true` | Persist cached answers to SQLite so they survive a restart. Only applies when `RESPONSE_CACHE` is on, so it is inert on a default install. Note this writes **plaintext model responses** to the database file, which the memory-only cache never did; set it to `false` to keep caching in memory only. Rows are deleted as soon as the entry leaves the in-memory cache (TTL expiry, LRU eviction, `DELETE /api/cache`), plus a sweep at startup. |+| `RESPONSE_CACHE` | `false` | Opt-in response cache. When on, a successful `/v1/chat/completions` answer — streaming or not — is stored in a bounded in-memory LRU keyed by a canonical hash of the request, so an IDENTICAL later request is served from memory without spending provider quota. Exact match only — a near-miss never returns a different prompt's answer. Toggled at runtime via the dashboard (`PUT /api/cache/config`) and per-request with the `X-FreeLLM-Cache: on\|off` header. |+| `RESPONSE_CACHE_PERSIST` | `true` | Persist cached **non-streaming** answers to SQLite so they survive a restart (streamed replays are memory-only and always dropped on restart). Only applies when `RESPONSE_CACHE` is on, so it is inert on a default install. Note this writes **plaintext model responses** to the database file, which the memory-only cache never did; set it to `false` to keep caching in memory only. Rows are deleted as soon as the entry leaves the in-memory cache (TTL expiry, LRU eviction, `DELETE /api/cache`), plus a sweep at startup. || `RESPONSE_CACHE_TTL_SECONDS` | `3600` (1 hour) | How long a cached answer stays fresh, in seconds. || `RESPONSE_CACHE_MAX_TEMPERATURE` | `1.0` | Only cache requests at/below this temperature; higher temperatures want fresh variety. Default caches everything when enabled; lower it (e.g. `0.2`) to cache only near-deterministic calls. |-| `RESPONSE_CACHE_MAX_ENTRIES` | `5000` | Hard cap on stored entries; least-recently-used entries are evicted past this. |+| `RESPONSE_CACHE_MAX_ENTRIES` | `5000` | Hard cap on stored entries, shared by the JSON and streaming stores (one prompt cached both ways takes two slots); least-recently-used entries are evicted past this. || `FREELLMAPI_COMPRESSION` | `off` | Request-side prompt/context compression. `lossless` applies whitespace hygiene, exact repeated-block references, and reversible tabular-JSON encoding; `standard` also filters tool output and superseded file reads; `aggressive` adds age/relevance/budget condensation. The dashboard can change this live under Settings. A request may lower the configured mode with `X-FreeLLM-Compress`, but cannot override a global `off` master switch. || `REQUEST_ANALYTICS_RETENTION_DAYS` | `90` | Request analytics retention in days. Set to `0` to disable this limit. || `REQUEST_ANALYTICS_MAX_ROWS` | `100000` | Request analytics row cap. Set to `0` to disable this limit. |diff --git a/docs/zh-cn/api/01-rest-api.md b/docs/zh-cn/api/01-rest-api.mdindex c4cebef5..69630f30 100644--- a/docs/zh-cn/api/01-rest-api.md+++ b/docs/zh-cn/api/01-rest-api.md@@ -223,7 +223,7 @@ print(resp.choices[0].message.content)HTTP 头只能携带可打印的 ASCII 字符,所以模型 id 里超出这个范围的字符(比如中转目录里的中文名称)在响应头中会被百分号编码。把这个值丢给 `decodeURIComponent`(或 `urllib.parse.unquote`)就能还原。-可选开启的响应缓存能按请求用 `X-FreeLLM-Cache: on|off` 开关。它是一个针对完全相同的非流式请求做精确匹配的内存 LRU(以整个请求的规范化 SHA-256 为键,带 TTL 和温度门槛,节省量统计显示在仪表盘上)。默认关闭;缓存命中不消耗任何提供方额度。+可选开启的响应缓存能按请求用 `X-FreeLLM-Cache: on|off` 开关。它是一个针对完全相同的请求(包括流式)做精确匹配的内存 LRU(流式命中会原样重放存储的 SSE)(以整个请求的规范化 SHA-256 为键,带 TTL 和温度门槛,节省量统计显示在仪表盘上)。默认关闭;缓存命中不消耗任何提供方额度。启用[提示词压缩](../compression/01-compression-pipeline.md)后,`X-FreeLLM-Compress: off|on|lossless|standard|aggressive` 可以对单次请求关闭或调低已配置的模式,但不能高于运维方配置的模式。响应会报告实际生效的模式和预估节省量,例如 `X-FreeLLM-Compress: standard; saved~=1840`。diff --git a/docs/zh-cn/api/02-idempotency.md b/docs/zh-cn/api/02-idempotency.mdindex 87d51108..99e9dfca 100644--- a/docs/zh-cn/api/02-idempotency.md+++ b/docs/zh-cn/api/02-idempotency.md@@ -24,7 +24,7 @@ Idempotency-Key: <opaque client token>只存储键的 `SHA-256` 十六进制摘要(`hashIdempotencyKey`)—— 原始键永远不会进入 SQLite,与管理令牌、运行时令牌的处理方式相同。-流式请求(`stream: true`)**始终绕过**幂等(`proxy.ts:1805` 中的 `const idemKey = !stream ? normalize... : null`)。一个流无法作为整体重放,而打开的连接本身就是重试信号 —— 与响应缓存的策略相同。+流式请求(`stream: true`)**始终绕过**幂等(`proxy.ts:1805` 中的 `const idemKey = !stream ? normalize... : null`)。一个流无法作为整体用于*重试*重放,而打开的连接本身就是重试信号。(响应缓存已不再沿用这一策略:它会重放流,只是以请求内容而非调用方提供的键为准。)---@@ -180,7 +180,7 @@ curl -i http://localhost:3001/v1/chat/completions \## 10. 与响应缓存及降级模式的关系-- **响应缓存**(`services/cache.ts`,`X-FreeLLM-Cache`)是正交的:全局、受温度门控、内存 LRU;幂等则是持久化的(SQLite)、按调用方、与指纹绑定。两者都会绕过提供方调用。+- **响应缓存**(`services/cache.ts`,`X-FreeLLM-Cache`)是正交的:全局、受温度门控、内存 LRU,流式与非流式一并覆盖;幂等则是持久化的(SQLite)、按调用方、与指纹绑定,且仅限非流式。两者都会绕过提供方调用。- **降级模式**(`../architecture/04-degraded-mode-and-failover.md`)会关闭老虎机探索,但不影响幂等 —— 重放仍然完全绕过路由。- **故障转移**:只存储最终成功的正文。若一个请求耗尽了回退循环,该键不会存储任何内容。diff --git a/docs/zh-cn/architecture/03-streaming-pipeline.md b/docs/zh-cn/architecture/03-streaming-pipeline.mdindex 5680dd33..99ec3c4e 100644--- a/docs/zh-cn/architecture/03-streaming-pipeline.md+++ b/docs/zh-cn/architecture/03-streaming-pipeline.md@@ -253,7 +253,8 @@ Codex CLI 和 Agents SDK 说 **Responses API**(有状态、`previous_response_## 13. 响应缓存(可选)-- **范围**:仅非流、可缓存温度(`temperature ≤ 1` 或未设)、`X-FreeLLM-Cache: on` 或 `RESPONSE_CACHE=1`。+- **范围**:流式与非流式皆可、可缓存温度(`temperature ≤ 1` 或未设)、`X-FreeLLM-Cache: on` 或 `RESPONSE_CACHE=1`。- **键**:规范化请求(模型、消息、所有采样参数、工具、stop、response_format、n、seed、惩罚、logit_bias、logprobs、top_logprobs、reasoning_effort、压缩)的 SHA-256。-- **存储**:内存 LRU、TTL + 温度闸门。-- **命中**:返回缓存体、`X-Routed-Via: cache`、`X-FreeLLM-Cache: HIT`,**零提供方额度消耗**。\ No newline at end of file+- **存储**:内存 LRU、TTL + 温度闸门。分两个存储 —— JSON 正文(持久化到 SQLite)与流式 SSE(仅内存),共用同一个 `RESPONSE_CACHE_MAX_ENTRIES` 上限;统计中的 `entries` 同时计入两者。+- **命中**:返回缓存体、`X-Routed-Via: cache`、`X-FreeLLM-Cache: HIT`,**零提供方额度消耗**。流式命中会逐字节重放存储的 SSE。+- **流式存储**:仅在请求可缓存时才缓冲帧(缓存关闭时不保留任何内容),超过 2 MB 重放上限即丢弃。被截断(`finish_reason: length`)、出错或中断的流永不存储。\ No newline at end of filediff --git a/docs/zh-cn/env/01-variables.md b/docs/zh-cn/env/01-variables.mdindex d3c1baf8..9862a365 100644--- a/docs/zh-cn/env/01-variables.md+++ b/docs/zh-cn/env/01-variables.md@@ -93,11 +93,11 @@ FreeLLMAPI 从 `.env` 读取的全部变量,按主题分组。默认值和说| 变量 | 默认值 | 用途 || --- | --- | --- |-| `RESPONSE_CACHE` | `false` | 可选开启的响应缓存。开启后,一次成功的非流式 `/v1/chat/completions` 回答会存进一个有界的内存 LRU(以整个请求的规范化哈希为键),因此完全相同的后续请求直接由内存应答,不再消耗提供方额度。只做精确匹配——近似请求绝不会返回另一个提示词的答案。可在运行时通过仪表盘切换(`PUT /api/cache/config`),也可按请求用 `X-FreeLLM-Cache: on\|off` 头控制。 |-| `RESPONSE_CACHE_PERSIST` | `true` | 将缓存的回答持久化到 SQLite,使其在重启后依然可用(对应每日额度重置后重跑相同任务的场景)。仅在 `RESPONSE_CACHE` 开启时生效,因此默认安装下完全不起作用。请注意:这会把**明文的模型响应**写入数据库文件,纯内存缓存从不这样做;设为 `false` 可继续使用缓存但只保留在内存中。条目一旦离开内存缓存(TTL 过期、LRU 逐出、`DELETE /api/cache`)对应的行会立即删除,启动时还会再清扫一次。 |+| `RESPONSE_CACHE` | `false` | 可选开启的响应缓存。开启后,一次成功的 `/v1/chat/completions` 回答(流式与非流式皆可)会存进一个有界的内存 LRU(以整个请求的规范化哈希为键),因此完全相同的后续请求直接由内存应答,不再消耗提供方额度。只做精确匹配——近似请求绝不会返回另一个提示词的答案。可在运行时通过仪表盘切换(`PUT /api/cache/config`),也可按请求用 `X-FreeLLM-Cache: on\|off` 头控制。 |+| `RESPONSE_CACHE_PERSIST` | `true` | 将缓存的**非流式**回答持久化到 SQLite,使其在重启后依然可用(流式重放仅存于内存,重启后必定丢失)(对应每日额度重置后重跑相同任务的场景)。仅在 `RESPONSE_CACHE` 开启时生效,因此默认安装下完全不起作用。请注意:这会把**明文的模型响应**写入数据库文件,纯内存缓存从不这样做;设为 `false` 可继续使用缓存但只保留在内存中。条目一旦离开内存缓存(TTL 过期、LRU 逐出、`DELETE /api/cache`)对应的行会立即删除,启动时还会再清扫一次。 || `RESPONSE_CACHE_TTL_SECONDS` | `3600`(1 小时) | 缓存的回答保持新鲜多久,单位秒。 || `RESPONSE_CACHE_MAX_TEMPERATURE` | `1.0` | 只缓存温度不高于该值的请求;更高的温度需要新鲜多样的输出。默认在开启时全部缓存;调低它(如 `0.2`)则只缓存接近确定性的调用。 |-| `RESPONSE_CACHE_MAX_ENTRIES` | `5000` | 存储条目的硬上限;超过后按最近最少使用逐出。 |+| `RESPONSE_CACHE_MAX_ENTRIES` | `5000` | 存储条目的硬上限,由 JSON 与流式两个存储共用(同一提示词两种方式各占一个名额);超过后按最近最少使用逐出。 || `FREELLMAPI_COMPRESSION` | `off` | 请求侧的提示词/上下文压缩。`lossless` 应用空白清理、完全重复块的引用以及可逆的表格化 JSON 编码;`standard` 再过滤工具输出和已被取代的文件读取;`aggressive` 追加基于年龄/相关性/预算的浓缩。仪表盘可以在设置里实时更改。单个请求可以用 `X-FreeLLM-Compress` 把配置的模式调低,但不能越过全局 `off` 总开关。 || `REQUEST_ANALYTICS_RETENTION_DAYS` | `90` | 请求分析的留存天数。设为 `0` 取消此限制。 || `REQUEST_ANALYTICS_MAX_ROWS` | `100000` | 请求分析的行数上限。设为 `0` 取消此限制。 |diff --git a/server/src/__tests__/routes/proxy-cache.test.ts b/server/src/__tests__/routes/proxy-cache.test.tsindex d943b22d..0b9dbf52 100644--- a/server/src/__tests__/routes/proxy-cache.test.ts+++ b/server/src/__tests__/routes/proxy-cache.test.ts@@ -35,7 +35,7 @@ function authHeaders() {// Mock every groq chat-completion call, counting how many actually reach the// provider (separately for streaming vs non-streaming). The whole point of the// cache is that a repeat non-streaming request does NOT reach the provider.-function mockGroq(content: string) {+function mockGroq(content: string, streamOpts: { finish?: string; failMidStream?: boolean } = {}) {const origFetch = global.fetch;const counter = { calls: 0, streamCalls: 0 };vi.spyOn(global, 'fetch').mockImplementation(async (url, init) => {@@ -47,8 +47,28 @@ function mockGroq(content: string) {const sse = [{ id: 'c1', object: 'chat.completion.chunk', created: 1, model: 'openai/gpt-oss-120b', choices: [{ index: 0, delta: { role: 'assistant', content: null }, finish_reason: null }] },{ id: 'c1', object: 'chat.completion.chunk', created: 1, model: 'openai/gpt-oss-120b', choices: [{ index: 0, delta: { content }, finish_reason: null }] },- { id: 'c1', object: 'chat.completion.chunk', created: 1, model: 'openai/gpt-oss-120b', choices: [{ index: 0, delta: {}, finish_reason: 'stop' }], usage: { prompt_tokens: 5, completion_tokens: 3, total_tokens: 8 } },+ { id: 'c1', object: 'chat.completion.chunk', created: 1, model: 'openai/gpt-oss-120b', choices: [{ index: 0, delta: {}, finish_reason: streamOpts.finish ?? 'stop' }], usage: { prompt_tokens: 5, completion_tokens: 3, total_tokens: 8 } },].map(f => `data: ${JSON.stringify(f)}\n\n`).join('') + 'data: [DONE]\n\n';+ if (streamOpts.failMidStream) {+ // Upstream dies partway through: the client keeps the frames it+ // already got, but the turn never completed, so nothing may be+ // cached. Pull-driven so the proxy really has flushed headers and+ // forwarded a content chunk before the error lands.+ const enc = new TextEncoder();+ const frame = (delta: Record<string, unknown>) =>+ enc.encode(`data: ${JSON.stringify({ id: 'c1', object: 'chat.completion.chunk', created: 1, model: 'openai/gpt-oss-120b', choices: [{ index: 0, delta, finish_reason: null }] })}\n\n`);+ let pulls = 0;+ const body = new ReadableStream<Uint8Array>({+ async pull(controller) {+ pulls++;+ if (pulls === 1) return void controller.enqueue(frame({ role: 'assistant', content: null }));+ if (pulls === 2) return void controller.enqueue(frame({ content }));+ await new Promise(resolve => setTimeout(resolve, 5));+ controller.error(new Error('upstream stream broke'));+ },+ });+ return new Response(body, { status: 200, headers: { 'Content-Type': 'text/event-stream' } }) as any;+ }return new Response(sse, { status: 200, headers: { 'Content-Type': 'text/event-stream' } }) as any;}counter.calls++;@@ -194,29 +214,81 @@ describe('Response cache (proxy integration)', () => {}});- it('a streaming request bypasses the cache: never served from it, never populates it', async () => {- const counter = mockGroq('cached four');+ it('replays a second identical streaming request from cache, byte for byte', async () => {+ const counter = mockGroq('streamed four');- // A non-streaming request populates the cache.- const first = await chat();+ const first = await chat({ stream: true });+ expect(first.status).toBe(200);expect(first.headers.get('x-freellm-cache')).toBe('MISS');- expect(counter.calls).toBe(1);+ expect(first.raw).toContain('streamed four');+ expect(first.raw.endsWith('data: [DONE]\n\n')).toBe(true);+ expect(counter.streamCalls).toBe(1);- // An identical STREAMING request must not be served from the cache, so it- // reaches the provider's streaming endpoint despite the cached entry.- const streamed = await chat({ stream: true });- expect(streamed.status).toBe(200);+ const second = await chat({ stream: true });+ expect(second.status).toBe(200);+ expect(second.headers.get('x-freellm-cache')).toBe('HIT');+ expect(second.headers.get('x-routed-via')).toBe('cache');+ expect(second.headers.get('content-type')).toContain('text/event-stream');+ // Identical SSE bytes, and no second provider stream.+ expect(second.raw).toBe(first.raw);expect(counter.streamCalls).toBe(1);+ });- // And it did not overwrite/populate a cache entry: a following identical- // non-streaming request is still a HIT served from the original entry, with- // no new non-streaming provider call.- const third = await chat();- expect(third.headers.get('x-freellm-cache')).toBe('HIT');- expect(third.body.choices[0].message.content).toBe('cached four');+ it('keeps the streaming and non-streaming entries for one prompt apart', async () => {+ const counter = mockGroq('four');+ await chat(); // JSON MISS, populates the JSON store+ const streamed = await chat({ stream: true });+ // The JSON entry is never replayed as SSE: the stream is a MISS of its own.+ expect(streamed.headers.get('x-freellm-cache')).toBe('MISS');+ expect(counter.streamCalls).toBe(1);+ // ...and storing it did not disturb the JSON entry.+ const json = await chat();+ expect(json.headers.get('x-freellm-cache')).toBe('HIT');+ expect(json.body.choices[0].message.content).toBe('four');expect(counter.calls).toBe(1);});+ it('does not cache a truncated stream (finish_reason length)', async () => {+ const counter = mockGroq('cut off mid-', { finish: 'length' });+ const first = await chat({ stream: true });+ expect(first.status).toBe(200);+ expect(counter.streamCalls).toBe(1);+ // Nothing was stored, so the repeat goes back to the provider.+ const second = await chat({ stream: true });+ expect(second.headers.get('x-freellm-cache')).toBe('MISS');+ expect(counter.streamCalls).toBe(2);+ });++ it('does not cache a stream that broke mid-flight', async () => {+ const counter = mockGroq('half an ans', { failMidStream: true });+ const first = await chat({ stream: true });+ // Headers were already flushed and content forwarded before the break.+ expect(first.headers.get('x-freellm-cache')).toBe('MISS');+ expect(first.raw).toContain('half an ans');+ expect(first.raw).toContain('stream_error'); // the break is surfaced to the client+ const before = counter.streamCalls;+ // Nothing was stored, so the repeat goes back to the provider.+ const second = await chat({ stream: true });+ expect(second.headers.get('x-freellm-cache')).toBe('MISS');+ expect(counter.streamCalls).toBeGreaterThan(before);+ });++ it('retains nothing when the cache is off: a stream buffers no frames', async () => {+ process.env.RESPONSE_CACHE = '';+ try {+ const counter = mockGroq('uncached');+ const first = await chat({ stream: true });+ expect(first.headers.get('x-freellm-cache')).toBe('OFF');+ const second = await chat({ stream: true });+ expect(second.headers.get('x-freellm-cache')).toBe('OFF');+ expect(counter.streamCalls).toBe(2);+ const stats = await request(app, 'GET', '/api/cache/stats');+ expect(stats.body.entries).toBe(0);+ } finally {+ process.env.RESPONSE_CACHE = 'on';+ }+ });+it('exposes cache stats and savings on the admin endpoint', async () => {mockGroq('four');await chat(); // MISS, populatesdiff --git a/server/src/__tests__/services/cache.test.ts b/server/src/__tests__/services/cache.test.tsindex 89dde4d8..88809756 100644--- a/server/src/__tests__/services/cache.test.ts+++ b/server/src/__tests__/services/cache.test.ts@@ -6,6 +6,7 @@ import {storeCachedResponse,getCachedStreamResponse,storeCachedStreamResponse,+ STREAM_CACHE_MAX_BYTES,getCacheStats,clearCache,loadCacheFromDb,@@ -382,7 +383,7 @@ describe('response cache', () => {});const hit = getCachedStreamResponse(key);expect(hit).not.toBeNull();- expect(hit!.frames).toEqual(frames('streamed answer'));+ expect(hit!.sse).toBe(frames('streamed answer').join(''));expect(hit!.platform).toBe('groq');expect(hit!.modelId).toBe('llama-3.3-70b');});@@ -396,7 +397,7 @@ describe('response cache', () => {});// The JSON store still serves its entry; the stream store its own.expect((getCachedResponse(key)!.body as any).choices[0].message.content).toBe('json answer');- expect(getCachedStreamResponse(key)!.frames).toEqual(frames('stream answer'));+ expect(getCachedStreamResponse(key)!.sse).toBe(frames('stream answer').join(''));});it('rejects an empty frame list (nothing to replay)', () => {@@ -422,6 +423,66 @@ describe('response cache', () => {expect(clearCache()).toBe(1);expect(getCacheStats().entries).toBe(0);});++ it('counts streaming lookups in lookupHits / lookupMisses (dashboard hit rate)', () => {+ const key = computeCacheKey({ model: 'auto', messages: [msg('user', 'rate')] });+ // Two misses: an absent key, then an expired entry.+ expect(getCachedStreamResponse(key)).toBeNull();+ storeCachedStreamResponse(key, {+ frames: frames('a'), platform: 'groq', modelId: 'm', keyId: 1, promptTokens: 1, completionTokens: 1,+ }, 1_000);+ expect(getCachedStreamResponse(key, 1_000 + 2 * 3600 * 1000)).toBeNull(); // past the 1h TTL+ // Then a hit on a fresh entry.+ storeCachedStreamResponse(key, {+ frames: frames('a'), platform: 'groq', modelId: 'm', keyId: 1, promptTokens: 1, completionTokens: 1,+ }, 2_000);+ expect(getCachedStreamResponse(key, 2_100)).not.toBeNull();++ const stats = getCacheStats();+ expect(stats.lookupMisses).toBe(2);+ expect(stats.lookupHits).toBe(1);+ expect(stats.hitRate).toBeCloseTo(1 / 3);+ });++ it('skips a stream that outgrew the replay byte ceiling', () => {+ const key = computeCacheKey({ model: 'auto', messages: [msg('user', 'huge')] });+ const huge = 'x'.repeat(STREAM_CACHE_MAX_BYTES + 1);+ storeCachedStreamResponse(key, {+ frames: [`data: ${huge}\n\n`],+ platform: 'groq', modelId: 'm', keyId: 1, promptTokens: 1, completionTokens: 1,+ });+ expect(getCachedStreamResponse(key)).toBeNull();+ expect(getCacheStats().entries).toBe(0);+ });++ it('shares one RESPONSE_CACHE_MAX_ENTRIES budget with the JSON store', () => {+ process.env.RESPONSE_CACHE_MAX_ENTRIES = '3';+ const base = 1_700_000_000_000;+ const k = (t: string) => computeCacheKey({ model: 'auto', messages: [msg('user', t)] });+ store(k('j1'), 'v1', base + 1);+ store(k('j2'), 'v2', base + 2);+ storeCachedStreamResponse(k('s1'), {+ frames: frames('s1'), platform: 'groq', modelId: 'm', keyId: 1, promptTokens: 1, completionTokens: 1,+ }, base + 3);+ expect(getCacheStats().entries).toBe(3); // full, counting both stores++ // A 4th entry (either kind) evicts the coldest across BOTH stores, so+ // the total never exceeds the documented cap.+ storeCachedStreamResponse(k('s2'), {+ frames: frames('s2'), platform: 'groq', modelId: 'm', keyId: 1, promptTokens: 1, completionTokens: 1,+ }, base + 4);+ expect(getCacheStats().entries).toBe(3);+ expect(getCachedResponse(k('j1'), base + 5)).toBeNull(); // oldest overall, evicted+ expect(getCachedResponse(k('j2'), base + 5)).not.toBeNull();+ expect(getCachedStreamResponse(k('s1'), base + 5)).not.toBeNull();+ expect(getCachedStreamResponse(k('s2'), base + 5)).not.toBeNull();++ // And a JSON insert can evict a colder STREAM entry, not just its own kind.+ store(k('j3'), 'v3', base + 6);+ store(k('j4'), 'v4', base + 7);+ expect(getCacheStats().entries).toBe(3);+ expect(getCachedStreamResponse(k('s1'), base + 8)).toBeNull();+ });});describe('settings-backed toggle (no restart)', () => {diff --git a/server/src/routes/proxy.ts b/server/src/routes/proxy.tsindex 4005f9e4..e6e2e903 100644--- a/server/src/routes/proxy.ts+++ b/server/src/routes/proxy.ts@@ -21,7 +21,7 @@ import { isFusionModel, runFusion, fusionConfigSchema, FusionError, FUSION_MODELimport { isRetryableError, isPaymentRequiredError, isModelNotFoundError, isModelAccessForbiddenError, isClientAbortError, newClientAbortError, newHedgeAbortError, isUpstreamClassificationOutput } from '../lib/error-classify.js';import { logRequest } from '../lib/request-log.js';import { observeServedModel } from '../lib/served-model.js';-import { parseCacheDirective, cacheActive, isCacheableTemperature, computeCacheKey, getCachedResponse, storeCachedResponse, getCachedStreamResponse, storeCachedStreamResponse } from '../services/cache.js';+import { parseCacheDirective, cacheActive, isCacheableTemperature, computeCacheKey, getCachedResponse, storeCachedResponse, getCachedStreamResponse, storeCachedStreamResponse, STREAM_CACHE_MAX_BYTES } from '../services/cache.js';import { normalizeIdempotencyKey, hashIdempotencyKey, computeIdempotencyFingerprint, lookupIdempotencyReplay, storeIdempotencyResult } from '../services/idempotency.js';import { runFallbackLoop, newFallbackState, recordUpstreamSuccess, exhaustedRetryError, setFallbackHeaders, exhaustionErrorPayload, setExhaustionHeaders, type AttemptRecord } from '../lib/fallback-loop.js';import { routedViaValue, safeHeaderValue } from '../lib/header-value.js';@@ -1839,7 +1839,7 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {res.setHeader('Connection', 'keep-alive');res.setHeader('X-Routed-Via', 'cache');res.setHeader('X-FreeLLM-Cache', 'HIT');- for (const frame of streamHit.frames) res.write(frame);+ res.write(streamHit.sse);res.end();return;}@@ -2165,7 +2165,24 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {let upstreamModel: string | null = null;// Every `data: ...` frame the client sees, captured for a possible// streaming cache store on success (exact SSE replay on a later hit).- const streamFrames: string[] = [];+ // Collected ONLY when this request is actually cacheable: with the+ // cache off — the default — a stream must retain nothing, so the+ // buffer stays null and every frame is written straight through.+ const streamFrames: string[] | null = cacheKey ? [] : null;+ let streamFrameBytes = 0;+ // Flipped off once the answer outgrows what is worth holding; the+ // buffer is dropped and this response is not stored.+ let streamCacheable = cacheKey !== null;+ const collectFrame = (frame: string) => {+ if (!streamCacheable || !streamFrames) return;+ streamFrameBytes += Buffer.byteLength(frame);+ if (streamFrameBytes > STREAM_CACHE_MAX_BYTES) {+ streamCacheable = false;+ streamFrames.length = 0;+ return;+ }+ streamFrames.push(frame);+ };const flushHeaders = () => {if (headerSent) return;@@ -2184,7 +2201,7 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {ctx.disarmHedge();for (const p of preamble) {const frame = `data: ${JSON.stringify(p)}\n\n`;- streamFrames.push(frame);+ collectFrame(frame);res.write(frame);}preamble.length = 0;@@ -2198,7 +2215,7 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {});const writeChunk = (c: unknown) => {const frame = `data: ${JSON.stringify(c)}\n\n`;- streamFrames.push(frame);+ collectFrame(frame);res.write(frame);};@@ -2506,7 +2523,7 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {});}const doneFrame = 'data: [DONE]\n\n';- streamFrames.push(doneFrame);+ collectFrame(doneFrame);res.write(doneFrame);res.end();@@ -2520,8 +2537,10 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {// stream request is replayed without spending another free-tier// slot. A truncated turn (finish 'length') is NOT cached, matching// the JSON cache policy — replaying a cut-off answer would be worse- // than regenerating.- if (cacheKey && finish !== 'length') {+ // than regenerating — and neither is one that outgrew the buffer+ // ceiling. A stream that errored or was aborted mid-flight never+ // reaches here at all (the catch below owns that path).+ if (cacheKey && streamFrames && streamCacheable && finish !== 'length') {storeCachedStreamResponse(cacheKey, {frames: streamFrames,platform: route.platform,diff --git a/server/src/services/cache.ts b/server/src/services/cache.tsindex 129d8adf..8868ea4c 100644--- a/server/src/services/cache.ts+++ b/server/src/services/cache.ts@@ -281,10 +281,13 @@ interface CacheEntry {const store = new Map<string, CacheEntry>();// Streaming entries live in their own LRU: a stream's replayable artifact is-// the exact SSE frame sequence, which is structurally different from a JSON-// completion body, so the two kinds never share a key across stores.+// the exact SSE byte sequence, which is structurally different from a JSON+// completion body, so the two kinds never share a key across stores. The+// frames are concatenated into one string on store — a hit replays the whole+// sequence in a single write, and one string costs far less than an array of+// hundreds of small ones.interface StreamCacheEntry {- frames: string[]; // verbatim `data: {...}\n\n` (and final `[DONE]`) frames+ sse: string; // verbatim `data: {...}\n\n` frames, `[DONE]` includedplatform: string;modelId: string;keyId: number | null;@@ -296,12 +299,35 @@ interface StreamCacheEntry {}const streamStore = new Map<string, StreamCacheEntry>();-function evictToCap<K, V>(map: Map<K, V>): void {+/** Last use of an entry, for cross-store recency comparison. */+function lastUsedAtMs(entry: { createdAtMs: number; lastHitAtMs: number | null }): number {+ return entry.lastHitAtMs ?? entry.createdAtMs;+}++/**+ * Evict least-recently-used entries until the JSON and streaming stores hold+ * RESPONSE_CACHE_MAX_ENTRIES *between them*. One shared budget, so enabling+ * streaming replay cannot silently double the documented memory ceiling.+ * Each map is itself in LRU order, so only their two front entries can be the+ * coldest; the older of the two goes first.+ */+function enforceCombinedCap(): void {const cap = cacheMaxEntries();- while (map.size > cap) {- const oldest = map.keys().next().value as K | undefined;- if (oldest === undefined) break;- map.delete(oldest);+ while (store.size + streamStore.size > cap) {+ const jsonKey = store.keys().next().value as string | undefined;+ const streamKey = streamStore.keys().next().value as string | undefined;+ const jsonAge = jsonKey === undefined ? Infinity : lastUsedAtMs(store.get(jsonKey)!);+ const streamAge = streamKey === undefined ? Infinity : lastUsedAtMs(streamStore.get(streamKey)!);+ if (streamKey !== undefined && streamAge <= jsonAge) {+ streamStore.delete(streamKey);+ } else if (jsonKey !== undefined) {+ store.delete(jsonKey);+ // Evicting in memory but not on disk would let the table grow without+ // bound and let a restart resurrect entries the LRU already gave up on.+ scheduleRowDelete(jsonKey);+ } else {+ break; // both stores empty; cap is 0+ }}}@@ -461,17 +487,10 @@ export function storeCachedResponse(cacheKey: string, input: StoreInput, now = DlastHitAtMs: null,});- // Evict least-recently-used beyond the cap. The count only drifts by one per- // insert, so at most one entry is removed per call in steady state.- const cap = cacheMaxEntries();- while (store.size > cap) {- const oldest = store.keys().next().value as string | undefined;- if (oldest === undefined) break;- store.delete(oldest);- // Evicting in memory but not on disk would let the table grow without- // bound and let a restart resurrect entries the LRU already gave up on.- scheduleRowDelete(oldest);- }+ // Evict least-recently-used beyond the cap, which is shared with the+ // streaming store. The count only drifts by one per insert, so at most one+ // entry is removed per call in steady state.+ enforceCombinedCap();// Write-through to SQLite. hit_count is stored at its current in-memory value// (0 for a fresh store, since an overwrite resets the rolling stat with the@@ -572,8 +591,18 @@ export function loadCacheFromDb(now = Date.now()): void {// ── Streaming entries ──+/**+ * Ceiling on one cached stream's replay payload. A single long answer must not+ * be able to pin megabytes of SSE text in memory, and a stream past this size+ * is cheap to regenerate relative to what it costs to hold. The caller stops+ * buffering at this point too (proxy.ts), so an oversize stream never fully+ * materializes; this is the store-side backstop.+ */+export const STREAM_CACHE_MAX_BYTES = 2 * 1024 * 1024;+export interface CachedStreamResponse {- frames: string[];+ /** The whole SSE sequence, ready to write in one go. */+ sse: string;platform: string;modelId: string;keyId: number | null;@@ -592,24 +621,31 @@ export interface StoreStreamInput {/*** Look up a cached stream. Returns null on a miss or when the entry has aged- * past the TTL. A hit bumps hit_count and moves the entry to MRU.+ * past the TTL. A hit bumps hit_count and moves the entry to MRU, and — like+ * the JSON lookup — counts into the process-wide hit/miss tallies behind the+ * dashboard's hit rate, so streaming traffic is not invisible there.*/export function getCachedStreamResponse(cacheKey: string, now = Date.now()): CachedStreamResponse | null {const entry = streamStore.get(cacheKey);- if (!entry) return null;+ if (!entry) {+ lookupMisses += 1;+ return null;+ }if (now - entry.createdAtMs > cacheTtlMs()) {streamStore.delete(cacheKey);+ lookupMisses += 1;return null;}+ lookupHits += 1;entry.hitCount += 1;entry.lastHitAtMs = now;streamStore.delete(cacheKey);streamStore.set(cacheKey, entry);return {- frames: entry.frames,+ sse: entry.sse,platform: entry.platform,modelId: entry.modelId,keyId: entry.keyId,@@ -621,15 +657,18 @@ export function getCachedStreamResponse(cacheKey: string, now = Date.now()): Cac/*** Store a completed SSE frame sequence for replay. The frames are the verbatim* `data: {...}\n\n` lines (including the final `[DONE]`) the client received,- * so a hit can reproduce the stream byte-for-byte. Best-effort like the JSON- * store.+ * concatenated, so a hit reproduces the stream byte-for-byte. Best-effort like+ * the JSON store: an empty or oversize sequence is silently skipped, and the+ * entry shares the JSON store's entry cap.*/export function storeCachedStreamResponse(cacheKey: string, input: StoreStreamInput, now = Date.now()): void {if (!Array.isArray(input.frames) || input.frames.length === 0) return;+ const sse = input.frames.join('');+ if (sse.length === 0 || Buffer.byteLength(sse) > STREAM_CACHE_MAX_BYTES) return;streamStore.delete(cacheKey);streamStore.set(cacheKey, {- frames: input.frames,+ sse,platform: input.platform,modelId: input.modelId,keyId: input.keyId,@@ -640,12 +679,18 @@ export function storeCachedStreamResponse(cacheKey: string, input: StoreStreamInlastHitAtMs: null,});- evictToCap(streamStore);+ enforceCombinedCap();}// ── Stats / admin ──export interface CacheStats {+ /**+ * Entries held across BOTH stores (JSON completions + streaming replays),+ * which is also what the shared RESPONSE_CACHE_MAX_ENTRIES cap bounds. One+ * prompt asked both streaming and non-streaming therefore counts twice —+ * they are two independent replayable artifacts, and two slots of the cap.+ */entries: number;/** Hits accumulated by the entries currently held, restored from SQLite. */totalHits: number;