Skip to content

7.4 billion tokens per month. 34 free LLM providers. 635 free model endpoints. All behind one /v1 endpoint, plus any custom OpenAI-compatible endpoint. Smart routing, automatic failover, encrypted keys. Personal experimentation only.

TypeScript

fix(cache): only buffer streams when cacheable, share the entry cap, count stream hits (#1197)

Tashfeen52214d573e00
diff --git a/.env.example b/.env.example
index 3dd79ae4..f0a0448f 100644
--- a/.env.example
+++ b/.env.example
@@ -190,15 +190,16 @@ PORT=3001
# remaining candidate of an unhealthy pool.
# MAX_CONSECUTIVE_UPSTREAM_FAILS=0
-# Opt-in response cache. When on, a successful non-streaming /v1/chat/completions
-# answer is stored in a bounded in-memory LRU keyed by a canonical hash of the
+# Opt-in response cache. When on, a successful /v1/chat/completions answer --
+# streaming or not -- is stored in a bounded in-memory LRU keyed by a hash of the
# request, so an IDENTICAL later request is served from memory without spending
# any provider quota. Exact match only (no fuzzy matching), so a near-miss never
# returns a different prompt's answer. Off by default; set to true to enable.
# Can also be toggled at runtime via the dashboard (PUT /api/cache/config), and
# per-request with the `X-FreeLLM-Cache: on|off` header.
# RESPONSE_CACHE=false
-# Persist cached answers to SQLite so they survive a restart (the daily
+# Persist cached non-streaming answers to SQLite so they survive a restart
+# (streamed replays are memory-only and never persisted) (the daily
# quota-reset re-run pattern). Only applies when the cache above is on. Note
# this writes PLAINTEXT model responses to the database file, which the
# memory-only cache never did; set it to false to keep caching but hold
@@ -211,7 +212,8 @@ PORT=3001
# variety, so they are never cached. Default 1.0 caches everything when enabled;
# lower it (e.g. 0.2) to cache only near-deterministic calls.
# RESPONSE_CACHE_MAX_TEMPERATURE=1.0
-# Hard cap on stored entries; the least-recently-used are evicted past this.
+# Hard cap on stored entries, shared by the JSON and streaming stores; the
+# least-recently-used are evicted past this.
# RESPONSE_CACHE_MAX_ENTRIES=5000
# Request-side prompt/context compression. The default is off. `lossless`
diff --git a/docs/en/api/01-rest-api.md b/docs/en/api/01-rest-api.md
index bda6d110..d7b6e547 100644
--- a/docs/en/api/01-rest-api.md
+++ b/docs/en/api/01-rest-api.md
@@ -302,7 +302,7 @@ Only hops that already **failed** can appear. The hop actually serving your requ
HTTP headers only carry printable ASCII, so a model id with characters outside that range (a Chinese name from a relay catalog, for example) is percent-encoded in the header — run the value through `decodeURIComponent` (or `urllib.parse.unquote`) to read it back.
-The opt-in response cache can be toggled per request with `X-FreeLLM-Cache: on|off` — an exact-match in-memory LRU for identical non-streaming requests (canonical SHA-256 keys over the full request, TTL and temperature gates, saved-token stats on the dashboard). Off by default; cache hits consume zero provider quota.
+The opt-in response cache can be toggled per request with `X-FreeLLM-Cache: on|off` — an exact-match in-memory LRU for identical requests, streaming included (a streamed hit replays the stored SSE verbatim) (canonical SHA-256 keys over the full request, TTL and temperature gates, saved-token stats on the dashboard). Off by default; cache hits consume zero provider quota.
When [prompt compression](../compression/01-compression-pipeline.md) is enabled, `X-FreeLLM-Compress: off|on|lossless|standard|aggressive` can disable or lower the configured mode for one request. It cannot raise the operator's configured mode. The response reports the effective mode and estimated savings, for example `X-FreeLLM-Compress: standard; saved~=1840`.
diff --git a/docs/en/api/02-idempotency.md b/docs/en/api/02-idempotency.md
index 48fa9cfa..3fc635b5 100644
--- a/docs/en/api/02-idempotency.md
+++ b/docs/en/api/02-idempotency.md
@@ -24,7 +24,7 @@ Normalization (`normalizeIdempotencyKey` in `server/src/services/idempotency.ts:
Only a `SHA-256 hex` of the key is stored (`hashIdempotencyKey`) — the raw key never touches SQLite, mirroring how admin/runtime tokens are handled.
-Streaming requests (`stream: true`) **always bypass** idempotency (`const idemKey = !stream ? normalize... : null` in `proxy.ts:1805`). A stream cannot be replayed as a unit and the open connection is itself the retry signal — same policy as the response cache.
+Streaming requests (`stream: true`) **always bypass** idempotency (`const idemKey = !stream ? normalize... : null` in `proxy.ts:1805`). A stream cannot be replayed as a unit for a *retry* and the open connection is itself the retry signal. (The response cache no longer follows this policy: it does replay streams, keyed on request content rather than a caller-supplied key.)
---
@@ -180,7 +180,7 @@ Tips:
## 10. Relationship to response cache & degraded mode
-- **Response cache** (`services/cache.ts`, `X-FreeLLM-Cache`) is orthogonal: global, temperature-gated, in-memory LRU; idempotency is durable (SQLite), per-caller, and fingerprint-bound. Both short-circuit provider calls.
+- **Response cache** (`services/cache.ts`, `X-FreeLLM-Cache`) is orthogonal: global, temperature-gated, in-memory LRU covering streaming and non-streaming alike; idempotency is durable (SQLite), per-caller, fingerprint-bound, and non-streaming only. Both short-circuit provider calls.
- **Degraded mode** ([`docs/en/architecture/04-degraded-mode-and-failover.md`](../architecture/04-degraded-mode-and-failover.md)) disables bandit exploration but does not affect idempotency — replays still bypass routing entirely.
- **Failover**: only the final successful body is stored. If a request exhausts the fallback loop, nothing is stored for that key.
diff --git a/docs/en/architecture/03-streaming-pipeline.md b/docs/en/architecture/03-streaming-pipeline.md
index 91266baf..c9ece226 100644
--- a/docs/en/architecture/03-streaming-pipeline.md
+++ b/docs/en/architecture/03-streaming-pipeline.md
@@ -253,7 +253,8 @@ Thin adapter over shared fallback loop:
## 13. Response Cache (Opt-In)
-- **Scope**: non-streaming only, cacheable temperature (`temperature ≤ 1` or unset), `X-FreeLLM-Cache: on` or `RESPONSE_CACHE=1`.
+- **Scope**: streaming and non-streaming, cacheable temperature (`temperature ≤ 1` or unset), `X-FreeLLM-Cache: on` or `RESPONSE_CACHE=1`.
- **Key**: SHA-256 over canonical request (model, messages, all sampling params, tools, stop, response_format, n, seed, penalties, logit_bias, logprobs, top_logprobs, reasoning_effort, compression).
-- **Store**: in-memory LRU, TTL + temperature gates.
-- **Hit**: returns cached body, `X-Routed-Via: cache`, `X-FreeLLM-Cache: HIT`, **zero provider quota consumed**.
\ No newline at end of file
+- **Store**: in-memory LRU, TTL + temperature gates. Two stores — JSON bodies (persisted to SQLite) and streaming SSE (memory-only) — sharing one `RESPONSE_CACHE_MAX_ENTRIES` budget; `entries` in the stats counts both.
+- **Hit**: returns cached body, `X-Routed-Via: cache`, `X-FreeLLM-Cache: HIT`, **zero provider quota consumed**. A streaming hit replays the stored SSE byte for byte.
+- **Streaming store**: frames are buffered only when the request is cacheable (nothing is retained with the cache off) and dropped past a 2 MB replay ceiling. A truncated (`finish_reason: length`), errored, or aborted stream is never stored.
\ No newline at end of file
diff --git a/docs/en/env/01-variables.md b/docs/en/env/01-variables.md
index 98130714..9efff6fc 100644
--- a/docs/en/env/01-variables.md
+++ b/docs/en/env/01-variables.md
@@ -97,11 +97,11 @@ In Docker, `127.0.0.1` is the container, not your machine — see [03-outbound-p
| Variable | Default | Purpose |
| --- | --- | --- |
-| `RESPONSE_CACHE` | `false` | Opt-in response cache. When on, a successful non-streaming `/v1/chat/completions` answer is stored in a bounded in-memory LRU keyed by a canonical hash of the request, so an IDENTICAL later request is served from memory without spending provider quota. Exact match only — a near-miss never returns a different prompt's answer. Toggled at runtime via the dashboard (`PUT /api/cache/config`) and per-request with the `X-FreeLLM-Cache: on\|off` header. |
-| `RESPONSE_CACHE_PERSIST` | `true` | Persist cached answers to SQLite so they survive a restart. Only applies when `RESPONSE_CACHE` is on, so it is inert on a default install. Note this writes **plaintext model responses** to the database file, which the memory-only cache never did; set it to `false` to keep caching in memory only. Rows are deleted as soon as the entry leaves the in-memory cache (TTL expiry, LRU eviction, `DELETE /api/cache`), plus a sweep at startup. |
+| `RESPONSE_CACHE` | `false` | Opt-in response cache. When on, a successful `/v1/chat/completions` answer — streaming or not — is stored in a bounded in-memory LRU keyed by a canonical hash of the request, so an IDENTICAL later request is served from memory without spending provider quota. Exact match only — a near-miss never returns a different prompt's answer. Toggled at runtime via the dashboard (`PUT /api/cache/config`) and per-request with the `X-FreeLLM-Cache: on\|off` header. |
+| `RESPONSE_CACHE_PERSIST` | `true` | Persist cached **non-streaming** answers to SQLite so they survive a restart (streamed replays are memory-only and always dropped on restart). Only applies when `RESPONSE_CACHE` is on, so it is inert on a default install. Note this writes **plaintext model responses** to the database file, which the memory-only cache never did; set it to `false` to keep caching in memory only. Rows are deleted as soon as the entry leaves the in-memory cache (TTL expiry, LRU eviction, `DELETE /api/cache`), plus a sweep at startup. |
| `RESPONSE_CACHE_TTL_SECONDS` | `3600` (1 hour) | How long a cached answer stays fresh, in seconds. |
| `RESPONSE_CACHE_MAX_TEMPERATURE` | `1.0` | Only cache requests at/below this temperature; higher temperatures want fresh variety. Default caches everything when enabled; lower it (e.g. `0.2`) to cache only near-deterministic calls. |
-| `RESPONSE_CACHE_MAX_ENTRIES` | `5000` | Hard cap on stored entries; least-recently-used entries are evicted past this. |
+| `RESPONSE_CACHE_MAX_ENTRIES` | `5000` | Hard cap on stored entries, shared by the JSON and streaming stores (one prompt cached both ways takes two slots); least-recently-used entries are evicted past this. |
| `FREELLMAPI_COMPRESSION` | `off` | Request-side prompt/context compression. `lossless` applies whitespace hygiene, exact repeated-block references, and reversible tabular-JSON encoding; `standard` also filters tool output and superseded file reads; `aggressive` adds age/relevance/budget condensation. The dashboard can change this live under Settings. A request may lower the configured mode with `X-FreeLLM-Compress`, but cannot override a global `off` master switch. |
| `REQUEST_ANALYTICS_RETENTION_DAYS` | `90` | Request analytics retention in days. Set to `0` to disable this limit. |
| `REQUEST_ANALYTICS_MAX_ROWS` | `100000` | Request analytics row cap. Set to `0` to disable this limit. |
diff --git a/docs/zh-cn/api/01-rest-api.md b/docs/zh-cn/api/01-rest-api.md
index c4cebef5..69630f30 100644
--- a/docs/zh-cn/api/01-rest-api.md
+++ b/docs/zh-cn/api/01-rest-api.md
@@ -223,7 +223,7 @@ print(resp.choices[0].message.content)
HTTP 头只能携带可打印的 ASCII 字符,所以模型 id 里超出这个范围的字符(比如中转目录里的中文名称)在响应头中会被百分号编码。把这个值丢给 `decodeURIComponent`(或 `urllib.parse.unquote`)就能还原。
-可选开启的响应缓存能按请求用 `X-FreeLLM-Cache: on|off` 开关。它是一个针对完全相同的非流式请求做精确匹配的内存 LRU(以整个请求的规范化 SHA-256 为键,带 TTL 和温度门槛,节省量统计显示在仪表盘上)。默认关闭;缓存命中不消耗任何提供方额度。
+可选开启的响应缓存能按请求用 `X-FreeLLM-Cache: on|off` 开关。它是一个针对完全相同的请求(包括流式)做精确匹配的内存 LRU(流式命中会原样重放存储的 SSE)(以整个请求的规范化 SHA-256 为键,带 TTL 和温度门槛,节省量统计显示在仪表盘上)。默认关闭;缓存命中不消耗任何提供方额度。
启用[提示词压缩](../compression/01-compression-pipeline.md)后,`X-FreeLLM-Compress: off|on|lossless|standard|aggressive` 可以对单次请求关闭或调低已配置的模式,但不能高于运维方配置的模式。响应会报告实际生效的模式和预估节省量,例如 `X-FreeLLM-Compress: standard; saved~=1840`。
diff --git a/docs/zh-cn/api/02-idempotency.md b/docs/zh-cn/api/02-idempotency.md
index 87d51108..99e9dfca 100644
--- a/docs/zh-cn/api/02-idempotency.md
+++ b/docs/zh-cn/api/02-idempotency.md
@@ -24,7 +24,7 @@ Idempotency-Key: <opaque client token>
只存储键的 `SHA-256` 十六进制摘要(`hashIdempotencyKey`)—— 原始键永远不会进入 SQLite,与管理令牌、运行时令牌的处理方式相同。
-流式请求(`stream: true`)**始终绕过**幂等(`proxy.ts:1805` 中的 `const idemKey = !stream ? normalize... : null`)。一个流无法作为整体重放,而打开的连接本身就是重试信号 —— 与响应缓存的策略相同。
+流式请求(`stream: true`)**始终绕过**幂等(`proxy.ts:1805` 中的 `const idemKey = !stream ? normalize... : null`)。一个流无法作为整体用于*重试*重放,而打开的连接本身就是重试信号。(响应缓存已不再沿用这一策略:它会重放流,只是以请求内容而非调用方提供的键为准。)
---
@@ -180,7 +180,7 @@ curl -i http://localhost:3001/v1/chat/completions \
## 10. 与响应缓存及降级模式的关系
-- **响应缓存**(`services/cache.ts`,`X-FreeLLM-Cache`)是正交的:全局、受温度门控、内存 LRU;幂等则是持久化的(SQLite)、按调用方、与指纹绑定。两者都会绕过提供方调用。
+- **响应缓存**(`services/cache.ts`,`X-FreeLLM-Cache`)是正交的:全局、受温度门控、内存 LRU,流式与非流式一并覆盖;幂等则是持久化的(SQLite)、按调用方、与指纹绑定,且仅限非流式。两者都会绕过提供方调用。
- **降级模式**(`../architecture/04-degraded-mode-and-failover.md`)会关闭老虎机探索,但不影响幂等 —— 重放仍然完全绕过路由。
- **故障转移**:只存储最终成功的正文。若一个请求耗尽了回退循环,该键不会存储任何内容。
diff --git a/docs/zh-cn/architecture/03-streaming-pipeline.md b/docs/zh-cn/architecture/03-streaming-pipeline.md
index 5680dd33..99ec3c4e 100644
--- a/docs/zh-cn/architecture/03-streaming-pipeline.md
+++ b/docs/zh-cn/architecture/03-streaming-pipeline.md
@@ -253,7 +253,8 @@ Codex CLI 和 Agents SDK 说 **Responses API**(有状态、`previous_response_
## 13. 响应缓存(可选)
-- **范围**:仅非流、可缓存温度(`temperature ≤ 1` 或未设)、`X-FreeLLM-Cache: on` 或 `RESPONSE_CACHE=1`。
+- **范围**:流式与非流式皆可、可缓存温度(`temperature ≤ 1` 或未设)、`X-FreeLLM-Cache: on` 或 `RESPONSE_CACHE=1`。
- **键**:规范化请求(模型、消息、所有采样参数、工具、stop、response_format、n、seed、惩罚、logit_bias、logprobs、top_logprobs、reasoning_effort、压缩)的 SHA-256。
-- **存储**:内存 LRU、TTL + 温度闸门。
-- **命中**:返回缓存体、`X-Routed-Via: cache`、`X-FreeLLM-Cache: HIT`,**零提供方额度消耗**。
\ No newline at end of file
+- **存储**:内存 LRU、TTL + 温度闸门。分两个存储 —— JSON 正文(持久化到 SQLite)与流式 SSE(仅内存),共用同一个 `RESPONSE_CACHE_MAX_ENTRIES` 上限;统计中的 `entries` 同时计入两者。
+- **命中**:返回缓存体、`X-Routed-Via: cache`、`X-FreeLLM-Cache: HIT`,**零提供方额度消耗**。流式命中会逐字节重放存储的 SSE。
+- **流式存储**:仅在请求可缓存时才缓冲帧(缓存关闭时不保留任何内容),超过 2 MB 重放上限即丢弃。被截断(`finish_reason: length`)、出错或中断的流永不存储。
\ No newline at end of file
diff --git a/docs/zh-cn/env/01-variables.md b/docs/zh-cn/env/01-variables.md
index d3c1baf8..9862a365 100644
--- a/docs/zh-cn/env/01-variables.md
+++ b/docs/zh-cn/env/01-variables.md
@@ -93,11 +93,11 @@ FreeLLMAPI 从 `.env` 读取的全部变量,按主题分组。默认值和说
| 变量 | 默认值 | 用途 |
| --- | --- | --- |
-| `RESPONSE_CACHE` | `false` | 可选开启的响应缓存。开启后,一次成功的非流式 `/v1/chat/completions` 回答会存进一个有界的内存 LRU(以整个请求的规范化哈希为键),因此完全相同的后续请求直接由内存应答,不再消耗提供方额度。只做精确匹配——近似请求绝不会返回另一个提示词的答案。可在运行时通过仪表盘切换(`PUT /api/cache/config`),也可按请求用 `X-FreeLLM-Cache: on\|off` 头控制。 |
-| `RESPONSE_CACHE_PERSIST` | `true` | 将缓存的回答持久化到 SQLite,使其在重启后依然可用(对应每日额度重置后重跑相同任务的场景)。仅在 `RESPONSE_CACHE` 开启时生效,因此默认安装下完全不起作用。请注意:这会把**明文的模型响应**写入数据库文件,纯内存缓存从不这样做;设为 `false` 可继续使用缓存但只保留在内存中。条目一旦离开内存缓存(TTL 过期、LRU 逐出、`DELETE /api/cache`)对应的行会立即删除,启动时还会再清扫一次。 |
+| `RESPONSE_CACHE` | `false` | 可选开启的响应缓存。开启后,一次成功的 `/v1/chat/completions` 回答(流式与非流式皆可)会存进一个有界的内存 LRU(以整个请求的规范化哈希为键),因此完全相同的后续请求直接由内存应答,不再消耗提供方额度。只做精确匹配——近似请求绝不会返回另一个提示词的答案。可在运行时通过仪表盘切换(`PUT /api/cache/config`),也可按请求用 `X-FreeLLM-Cache: on\|off` 头控制。 |
+| `RESPONSE_CACHE_PERSIST` | `true` | 将缓存的**非流式**回答持久化到 SQLite,使其在重启后依然可用(流式重放仅存于内存,重启后必定丢失)(对应每日额度重置后重跑相同任务的场景)。仅在 `RESPONSE_CACHE` 开启时生效,因此默认安装下完全不起作用。请注意:这会把**明文的模型响应**写入数据库文件,纯内存缓存从不这样做;设为 `false` 可继续使用缓存但只保留在内存中。条目一旦离开内存缓存(TTL 过期、LRU 逐出、`DELETE /api/cache`)对应的行会立即删除,启动时还会再清扫一次。 |
| `RESPONSE_CACHE_TTL_SECONDS` | `3600`(1 小时) | 缓存的回答保持新鲜多久,单位秒。 |
| `RESPONSE_CACHE_MAX_TEMPERATURE` | `1.0` | 只缓存温度不高于该值的请求;更高的温度需要新鲜多样的输出。默认在开启时全部缓存;调低它(如 `0.2`)则只缓存接近确定性的调用。 |
-| `RESPONSE_CACHE_MAX_ENTRIES` | `5000` | 存储条目的硬上限;超过后按最近最少使用逐出。 |
+| `RESPONSE_CACHE_MAX_ENTRIES` | `5000` | 存储条目的硬上限,由 JSON 与流式两个存储共用(同一提示词两种方式各占一个名额);超过后按最近最少使用逐出。 |
| `FREELLMAPI_COMPRESSION` | `off` | 请求侧的提示词/上下文压缩。`lossless` 应用空白清理、完全重复块的引用以及可逆的表格化 JSON 编码;`standard` 再过滤工具输出和已被取代的文件读取;`aggressive` 追加基于年龄/相关性/预算的浓缩。仪表盘可以在设置里实时更改。单个请求可以用 `X-FreeLLM-Compress` 把配置的模式调低,但不能越过全局 `off` 总开关。 |
| `REQUEST_ANALYTICS_RETENTION_DAYS` | `90` | 请求分析的留存天数。设为 `0` 取消此限制。 |
| `REQUEST_ANALYTICS_MAX_ROWS` | `100000` | 请求分析的行数上限。设为 `0` 取消此限制。 |
diff --git a/server/src/__tests__/routes/proxy-cache.test.ts b/server/src/__tests__/routes/proxy-cache.test.ts
index d943b22d..0b9dbf52 100644
--- a/server/src/__tests__/routes/proxy-cache.test.ts
+++ b/server/src/__tests__/routes/proxy-cache.test.ts
@@ -35,7 +35,7 @@ function authHeaders() {
// Mock every groq chat-completion call, counting how many actually reach the
// provider (separately for streaming vs non-streaming). The whole point of the
// cache is that a repeat non-streaming request does NOT reach the provider.
-function mockGroq(content: string) {
+function mockGroq(content: string, streamOpts: { finish?: string; failMidStream?: boolean } = {}) {
const origFetch = global.fetch;
const counter = { calls: 0, streamCalls: 0 };
vi.spyOn(global, 'fetch').mockImplementation(async (url, init) => {
@@ -47,8 +47,28 @@ function mockGroq(content: string) {
const sse = [
{ id: 'c1', object: 'chat.completion.chunk', created: 1, model: 'openai/gpt-oss-120b', choices: [{ index: 0, delta: { role: 'assistant', content: null }, finish_reason: null }] },
{ id: 'c1', object: 'chat.completion.chunk', created: 1, model: 'openai/gpt-oss-120b', choices: [{ index: 0, delta: { content }, finish_reason: null }] },
- { id: 'c1', object: 'chat.completion.chunk', created: 1, model: 'openai/gpt-oss-120b', choices: [{ index: 0, delta: {}, finish_reason: 'stop' }], usage: { prompt_tokens: 5, completion_tokens: 3, total_tokens: 8 } },
+ { id: 'c1', object: 'chat.completion.chunk', created: 1, model: 'openai/gpt-oss-120b', choices: [{ index: 0, delta: {}, finish_reason: streamOpts.finish ?? 'stop' }], usage: { prompt_tokens: 5, completion_tokens: 3, total_tokens: 8 } },
].map(f => `data: ${JSON.stringify(f)}\n\n`).join('') + 'data: [DONE]\n\n';
+ if (streamOpts.failMidStream) {
+ // Upstream dies partway through: the client keeps the frames it
+ // already got, but the turn never completed, so nothing may be
+ // cached. Pull-driven so the proxy really has flushed headers and
+ // forwarded a content chunk before the error lands.
+ const enc = new TextEncoder();
+ const frame = (delta: Record<string, unknown>) =>
+ enc.encode(`data: ${JSON.stringify({ id: 'c1', object: 'chat.completion.chunk', created: 1, model: 'openai/gpt-oss-120b', choices: [{ index: 0, delta, finish_reason: null }] })}\n\n`);
+ let pulls = 0;
+ const body = new ReadableStream<Uint8Array>({
+ async pull(controller) {
+ pulls++;
+ if (pulls === 1) return void controller.enqueue(frame({ role: 'assistant', content: null }));
+ if (pulls === 2) return void controller.enqueue(frame({ content }));
+ await new Promise(resolve => setTimeout(resolve, 5));
+ controller.error(new Error('upstream stream broke'));
+ },
+ });
+ return new Response(body, { status: 200, headers: { 'Content-Type': 'text/event-stream' } }) as any;
+ }
return new Response(sse, { status: 200, headers: { 'Content-Type': 'text/event-stream' } }) as any;
}
counter.calls++;
@@ -194,29 +214,81 @@ describe('Response cache (proxy integration)', () => {
}
});
- it('a streaming request bypasses the cache: never served from it, never populates it', async () => {
- const counter = mockGroq('cached four');
+ it('replays a second identical streaming request from cache, byte for byte', async () => {
+ const counter = mockGroq('streamed four');
- // A non-streaming request populates the cache.
- const first = await chat();
+ const first = await chat({ stream: true });
+ expect(first.status).toBe(200);
expect(first.headers.get('x-freellm-cache')).toBe('MISS');
- expect(counter.calls).toBe(1);
+ expect(first.raw).toContain('streamed four');
+ expect(first.raw.endsWith('data: [DONE]\n\n')).toBe(true);
+ expect(counter.streamCalls).toBe(1);
- // An identical STREAMING request must not be served from the cache, so it
- // reaches the provider's streaming endpoint despite the cached entry.
- const streamed = await chat({ stream: true });
- expect(streamed.status).toBe(200);
+ const second = await chat({ stream: true });
+ expect(second.status).toBe(200);
+ expect(second.headers.get('x-freellm-cache')).toBe('HIT');
+ expect(second.headers.get('x-routed-via')).toBe('cache');
+ expect(second.headers.get('content-type')).toContain('text/event-stream');
+ // Identical SSE bytes, and no second provider stream.
+ expect(second.raw).toBe(first.raw);
expect(counter.streamCalls).toBe(1);
+ });
- // And it did not overwrite/populate a cache entry: a following identical
- // non-streaming request is still a HIT served from the original entry, with
- // no new non-streaming provider call.
- const third = await chat();
- expect(third.headers.get('x-freellm-cache')).toBe('HIT');
- expect(third.body.choices[0].message.content).toBe('cached four');
+ it('keeps the streaming and non-streaming entries for one prompt apart', async () => {
+ const counter = mockGroq('four');
+ await chat(); // JSON MISS, populates the JSON store
+ const streamed = await chat({ stream: true });
+ // The JSON entry is never replayed as SSE: the stream is a MISS of its own.
+ expect(streamed.headers.get('x-freellm-cache')).toBe('MISS');
+ expect(counter.streamCalls).toBe(1);
+ // ...and storing it did not disturb the JSON entry.
+ const json = await chat();
+ expect(json.headers.get('x-freellm-cache')).toBe('HIT');
+ expect(json.body.choices[0].message.content).toBe('four');
expect(counter.calls).toBe(1);
});
+ it('does not cache a truncated stream (finish_reason length)', async () => {
+ const counter = mockGroq('cut off mid-', { finish: 'length' });
+ const first = await chat({ stream: true });
+ expect(first.status).toBe(200);
+ expect(counter.streamCalls).toBe(1);
+ // Nothing was stored, so the repeat goes back to the provider.
+ const second = await chat({ stream: true });
+ expect(second.headers.get('x-freellm-cache')).toBe('MISS');
+ expect(counter.streamCalls).toBe(2);
+ });
+
+ it('does not cache a stream that broke mid-flight', async () => {
+ const counter = mockGroq('half an ans', { failMidStream: true });
+ const first = await chat({ stream: true });
+ // Headers were already flushed and content forwarded before the break.
+ expect(first.headers.get('x-freellm-cache')).toBe('MISS');
+ expect(first.raw).toContain('half an ans');
+ expect(first.raw).toContain('stream_error'); // the break is surfaced to the client
+ const before = counter.streamCalls;
+ // Nothing was stored, so the repeat goes back to the provider.
+ const second = await chat({ stream: true });
+ expect(second.headers.get('x-freellm-cache')).toBe('MISS');
+ expect(counter.streamCalls).toBeGreaterThan(before);
+ });
+
+ it('retains nothing when the cache is off: a stream buffers no frames', async () => {
+ process.env.RESPONSE_CACHE = '';
+ try {
+ const counter = mockGroq('uncached');
+ const first = await chat({ stream: true });
+ expect(first.headers.get('x-freellm-cache')).toBe('OFF');
+ const second = await chat({ stream: true });
+ expect(second.headers.get('x-freellm-cache')).toBe('OFF');
+ expect(counter.streamCalls).toBe(2);
+ const stats = await request(app, 'GET', '/api/cache/stats');
+ expect(stats.body.entries).toBe(0);
+ } finally {
+ process.env.RESPONSE_CACHE = 'on';
+ }
+ });
+
it('exposes cache stats and savings on the admin endpoint', async () => {
mockGroq('four');
await chat(); // MISS, populates
diff --git a/server/src/__tests__/services/cache.test.ts b/server/src/__tests__/services/cache.test.ts
index 89dde4d8..88809756 100644
--- a/server/src/__tests__/services/cache.test.ts
+++ b/server/src/__tests__/services/cache.test.ts
@@ -6,6 +6,7 @@ import {
storeCachedResponse,
getCachedStreamResponse,
storeCachedStreamResponse,
+ STREAM_CACHE_MAX_BYTES,
getCacheStats,
clearCache,
loadCacheFromDb,
@@ -382,7 +383,7 @@ describe('response cache', () => {
});
const hit = getCachedStreamResponse(key);
expect(hit).not.toBeNull();
- expect(hit!.frames).toEqual(frames('streamed answer'));
+ expect(hit!.sse).toBe(frames('streamed answer').join(''));
expect(hit!.platform).toBe('groq');
expect(hit!.modelId).toBe('llama-3.3-70b');
});
@@ -396,7 +397,7 @@ describe('response cache', () => {
});
// The JSON store still serves its entry; the stream store its own.
expect((getCachedResponse(key)!.body as any).choices[0].message.content).toBe('json answer');
- expect(getCachedStreamResponse(key)!.frames).toEqual(frames('stream answer'));
+ expect(getCachedStreamResponse(key)!.sse).toBe(frames('stream answer').join(''));
});
it('rejects an empty frame list (nothing to replay)', () => {
@@ -422,6 +423,66 @@ describe('response cache', () => {
expect(clearCache()).toBe(1);
expect(getCacheStats().entries).toBe(0);
});
+
+ it('counts streaming lookups in lookupHits / lookupMisses (dashboard hit rate)', () => {
+ const key = computeCacheKey({ model: 'auto', messages: [msg('user', 'rate')] });
+ // Two misses: an absent key, then an expired entry.
+ expect(getCachedStreamResponse(key)).toBeNull();
+ storeCachedStreamResponse(key, {
+ frames: frames('a'), platform: 'groq', modelId: 'm', keyId: 1, promptTokens: 1, completionTokens: 1,
+ }, 1_000);
+ expect(getCachedStreamResponse(key, 1_000 + 2 * 3600 * 1000)).toBeNull(); // past the 1h TTL
+ // Then a hit on a fresh entry.
+ storeCachedStreamResponse(key, {
+ frames: frames('a'), platform: 'groq', modelId: 'm', keyId: 1, promptTokens: 1, completionTokens: 1,
+ }, 2_000);
+ expect(getCachedStreamResponse(key, 2_100)).not.toBeNull();
+
+ const stats = getCacheStats();
+ expect(stats.lookupMisses).toBe(2);
+ expect(stats.lookupHits).toBe(1);
+ expect(stats.hitRate).toBeCloseTo(1 / 3);
+ });
+
+ it('skips a stream that outgrew the replay byte ceiling', () => {
+ const key = computeCacheKey({ model: 'auto', messages: [msg('user', 'huge')] });
+ const huge = 'x'.repeat(STREAM_CACHE_MAX_BYTES + 1);
+ storeCachedStreamResponse(key, {
+ frames: [`data: ${huge}\n\n`],
+ platform: 'groq', modelId: 'm', keyId: 1, promptTokens: 1, completionTokens: 1,
+ });
+ expect(getCachedStreamResponse(key)).toBeNull();
+ expect(getCacheStats().entries).toBe(0);
+ });
+
+ it('shares one RESPONSE_CACHE_MAX_ENTRIES budget with the JSON store', () => {
+ process.env.RESPONSE_CACHE_MAX_ENTRIES = '3';
+ const base = 1_700_000_000_000;
+ const k = (t: string) => computeCacheKey({ model: 'auto', messages: [msg('user', t)] });
+ store(k('j1'), 'v1', base + 1);
+ store(k('j2'), 'v2', base + 2);
+ storeCachedStreamResponse(k('s1'), {
+ frames: frames('s1'), platform: 'groq', modelId: 'm', keyId: 1, promptTokens: 1, completionTokens: 1,
+ }, base + 3);
+ expect(getCacheStats().entries).toBe(3); // full, counting both stores
+
+ // A 4th entry (either kind) evicts the coldest across BOTH stores, so
+ // the total never exceeds the documented cap.
+ storeCachedStreamResponse(k('s2'), {
+ frames: frames('s2'), platform: 'groq', modelId: 'm', keyId: 1, promptTokens: 1, completionTokens: 1,
+ }, base + 4);
+ expect(getCacheStats().entries).toBe(3);
+ expect(getCachedResponse(k('j1'), base + 5)).toBeNull(); // oldest overall, evicted
+ expect(getCachedResponse(k('j2'), base + 5)).not.toBeNull();
+ expect(getCachedStreamResponse(k('s1'), base + 5)).not.toBeNull();
+ expect(getCachedStreamResponse(k('s2'), base + 5)).not.toBeNull();
+
+ // And a JSON insert can evict a colder STREAM entry, not just its own kind.
+ store(k('j3'), 'v3', base + 6);
+ store(k('j4'), 'v4', base + 7);
+ expect(getCacheStats().entries).toBe(3);
+ expect(getCachedStreamResponse(k('s1'), base + 8)).toBeNull();
+ });
});
describe('settings-backed toggle (no restart)', () => {
diff --git a/server/src/routes/proxy.ts b/server/src/routes/proxy.ts
index 4005f9e4..e6e2e903 100644
--- a/server/src/routes/proxy.ts
+++ b/server/src/routes/proxy.ts
@@ -21,7 +21,7 @@ import { isFusionModel, runFusion, fusionConfigSchema, FusionError, FUSION_MODEL
import { isRetryableError, isPaymentRequiredError, isModelNotFoundError, isModelAccessForbiddenError, isClientAbortError, newClientAbortError, newHedgeAbortError, isUpstreamClassificationOutput } from '../lib/error-classify.js';
import { logRequest } from '../lib/request-log.js';
import { observeServedModel } from '../lib/served-model.js';
-import { parseCacheDirective, cacheActive, isCacheableTemperature, computeCacheKey, getCachedResponse, storeCachedResponse, getCachedStreamResponse, storeCachedStreamResponse } from '../services/cache.js';
+import { parseCacheDirective, cacheActive, isCacheableTemperature, computeCacheKey, getCachedResponse, storeCachedResponse, getCachedStreamResponse, storeCachedStreamResponse, STREAM_CACHE_MAX_BYTES } from '../services/cache.js';
import { normalizeIdempotencyKey, hashIdempotencyKey, computeIdempotencyFingerprint, lookupIdempotencyReplay, storeIdempotencyResult } from '../services/idempotency.js';
import { runFallbackLoop, newFallbackState, recordUpstreamSuccess, exhaustedRetryError, setFallbackHeaders, exhaustionErrorPayload, setExhaustionHeaders, type AttemptRecord } from '../lib/fallback-loop.js';
import { routedViaValue, safeHeaderValue } from '../lib/header-value.js';
@@ -1839,7 +1839,7 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {
res.setHeader('Connection', 'keep-alive');
res.setHeader('X-Routed-Via', 'cache');
res.setHeader('X-FreeLLM-Cache', 'HIT');
- for (const frame of streamHit.frames) res.write(frame);
+ res.write(streamHit.sse);
res.end();
return;
}
@@ -2165,7 +2165,24 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {
let upstreamModel: string | null = null;
// Every `data: ...` frame the client sees, captured for a possible
// streaming cache store on success (exact SSE replay on a later hit).
- const streamFrames: string[] = [];
+ // Collected ONLY when this request is actually cacheable: with the
+ // cache off — the default — a stream must retain nothing, so the
+ // buffer stays null and every frame is written straight through.
+ const streamFrames: string[] | null = cacheKey ? [] : null;
+ let streamFrameBytes = 0;
+ // Flipped off once the answer outgrows what is worth holding; the
+ // buffer is dropped and this response is not stored.
+ let streamCacheable = cacheKey !== null;
+ const collectFrame = (frame: string) => {
+ if (!streamCacheable || !streamFrames) return;
+ streamFrameBytes += Buffer.byteLength(frame);
+ if (streamFrameBytes > STREAM_CACHE_MAX_BYTES) {
+ streamCacheable = false;
+ streamFrames.length = 0;
+ return;
+ }
+ streamFrames.push(frame);
+ };
const flushHeaders = () => {
if (headerSent) return;
@@ -2184,7 +2201,7 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {
ctx.disarmHedge();
for (const p of preamble) {
const frame = `data: ${JSON.stringify(p)}\n\n`;
- streamFrames.push(frame);
+ collectFrame(frame);
res.write(frame);
}
preamble.length = 0;
@@ -2198,7 +2215,7 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {
});
const writeChunk = (c: unknown) => {
const frame = `data: ${JSON.stringify(c)}\n\n`;
- streamFrames.push(frame);
+ collectFrame(frame);
res.write(frame);
};
@@ -2506,7 +2523,7 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {
});
}
const doneFrame = 'data: [DONE]\n\n';
- streamFrames.push(doneFrame);
+ collectFrame(doneFrame);
res.write(doneFrame);
res.end();
@@ -2520,8 +2537,10 @@ proxyRouter.post('/chat/completions', async (req: Request, res: Response) => {
// stream request is replayed without spending another free-tier
// slot. A truncated turn (finish 'length') is NOT cached, matching
// the JSON cache policy — replaying a cut-off answer would be worse
- // than regenerating.
- if (cacheKey && finish !== 'length') {
+ // than regenerating — and neither is one that outgrew the buffer
+ // ceiling. A stream that errored or was aborted mid-flight never
+ // reaches here at all (the catch below owns that path).
+ if (cacheKey && streamFrames && streamCacheable && finish !== 'length') {
storeCachedStreamResponse(cacheKey, {
frames: streamFrames,
platform: route.platform,
diff --git a/server/src/services/cache.ts b/server/src/services/cache.ts
index 129d8adf..8868ea4c 100644
--- a/server/src/services/cache.ts
+++ b/server/src/services/cache.ts
@@ -281,10 +281,13 @@ interface CacheEntry {
const store = new Map<string, CacheEntry>();
// Streaming entries live in their own LRU: a stream's replayable artifact is
-// the exact SSE frame sequence, which is structurally different from a JSON
-// completion body, so the two kinds never share a key across stores.
+// the exact SSE byte sequence, which is structurally different from a JSON
+// completion body, so the two kinds never share a key across stores. The
+// frames are concatenated into one string on store — a hit replays the whole
+// sequence in a single write, and one string costs far less than an array of
+// hundreds of small ones.
interface StreamCacheEntry {
- frames: string[]; // verbatim `data: {...}\n\n` (and final `[DONE]`) frames
+ sse: string; // verbatim `data: {...}\n\n` frames, `[DONE]` included
platform: string;
modelId: string;
keyId: number | null;
@@ -296,12 +299,35 @@ interface StreamCacheEntry {
}
const streamStore = new Map<string, StreamCacheEntry>();
-function evictToCap<K, V>(map: Map<K, V>): void {
+/** Last use of an entry, for cross-store recency comparison. */
+function lastUsedAtMs(entry: { createdAtMs: number; lastHitAtMs: number | null }): number {
+ return entry.lastHitAtMs ?? entry.createdAtMs;
+}
+
+/**
+ * Evict least-recently-used entries until the JSON and streaming stores hold
+ * RESPONSE_CACHE_MAX_ENTRIES *between them*. One shared budget, so enabling
+ * streaming replay cannot silently double the documented memory ceiling.
+ * Each map is itself in LRU order, so only their two front entries can be the
+ * coldest; the older of the two goes first.
+ */
+function enforceCombinedCap(): void {
const cap = cacheMaxEntries();
- while (map.size > cap) {
- const oldest = map.keys().next().value as K | undefined;
- if (oldest === undefined) break;
- map.delete(oldest);
+ while (store.size + streamStore.size > cap) {
+ const jsonKey = store.keys().next().value as string | undefined;
+ const streamKey = streamStore.keys().next().value as string | undefined;
+ const jsonAge = jsonKey === undefined ? Infinity : lastUsedAtMs(store.get(jsonKey)!);
+ const streamAge = streamKey === undefined ? Infinity : lastUsedAtMs(streamStore.get(streamKey)!);
+ if (streamKey !== undefined && streamAge <= jsonAge) {
+ streamStore.delete(streamKey);
+ } else if (jsonKey !== undefined) {
+ store.delete(jsonKey);
+ // Evicting in memory but not on disk would let the table grow without
+ // bound and let a restart resurrect entries the LRU already gave up on.
+ scheduleRowDelete(jsonKey);
+ } else {
+ break; // both stores empty; cap is 0
+ }
}
}
@@ -461,17 +487,10 @@ export function storeCachedResponse(cacheKey: string, input: StoreInput, now = D
lastHitAtMs: null,
});
- // Evict least-recently-used beyond the cap. The count only drifts by one per
- // insert, so at most one entry is removed per call in steady state.
- const cap = cacheMaxEntries();
- while (store.size > cap) {
- const oldest = store.keys().next().value as string | undefined;
- if (oldest === undefined) break;
- store.delete(oldest);
- // Evicting in memory but not on disk would let the table grow without
- // bound and let a restart resurrect entries the LRU already gave up on.
- scheduleRowDelete(oldest);
- }
+ // Evict least-recently-used beyond the cap, which is shared with the
+ // streaming store. The count only drifts by one per insert, so at most one
+ // entry is removed per call in steady state.
+ enforceCombinedCap();
// Write-through to SQLite. hit_count is stored at its current in-memory value
// (0 for a fresh store, since an overwrite resets the rolling stat with the
@@ -572,8 +591,18 @@ export function loadCacheFromDb(now = Date.now()): void {
// ── Streaming entries ──
+/**
+ * Ceiling on one cached stream's replay payload. A single long answer must not
+ * be able to pin megabytes of SSE text in memory, and a stream past this size
+ * is cheap to regenerate relative to what it costs to hold. The caller stops
+ * buffering at this point too (proxy.ts), so an oversize stream never fully
+ * materializes; this is the store-side backstop.
+ */
+export const STREAM_CACHE_MAX_BYTES = 2 * 1024 * 1024;
+
export interface CachedStreamResponse {
- frames: string[];
+ /** The whole SSE sequence, ready to write in one go. */
+ sse: string;
platform: string;
modelId: string;
keyId: number | null;
@@ -592,24 +621,31 @@ export interface StoreStreamInput {
/**
* Look up a cached stream. Returns null on a miss or when the entry has aged
- * past the TTL. A hit bumps hit_count and moves the entry to MRU.
+ * past the TTL. A hit bumps hit_count and moves the entry to MRU, and — like
+ * the JSON lookup — counts into the process-wide hit/miss tallies behind the
+ * dashboard's hit rate, so streaming traffic is not invisible there.
*/
export function getCachedStreamResponse(cacheKey: string, now = Date.now()): CachedStreamResponse | null {
const entry = streamStore.get(cacheKey);
- if (!entry) return null;
+ if (!entry) {
+ lookupMisses += 1;
+ return null;
+ }
if (now - entry.createdAtMs > cacheTtlMs()) {
streamStore.delete(cacheKey);
+ lookupMisses += 1;
return null;
}
+ lookupHits += 1;
entry.hitCount += 1;
entry.lastHitAtMs = now;
streamStore.delete(cacheKey);
streamStore.set(cacheKey, entry);
return {
- frames: entry.frames,
+ sse: entry.sse,
platform: entry.platform,
modelId: entry.modelId,
keyId: entry.keyId,
@@ -621,15 +657,18 @@ export function getCachedStreamResponse(cacheKey: string, now = Date.now()): Cac
/**
* Store a completed SSE frame sequence for replay. The frames are the verbatim
* `data: {...}\n\n` lines (including the final `[DONE]`) the client received,
- * so a hit can reproduce the stream byte-for-byte. Best-effort like the JSON
- * store.
+ * concatenated, so a hit reproduces the stream byte-for-byte. Best-effort like
+ * the JSON store: an empty or oversize sequence is silently skipped, and the
+ * entry shares the JSON store's entry cap.
*/
export function storeCachedStreamResponse(cacheKey: string, input: StoreStreamInput, now = Date.now()): void {
if (!Array.isArray(input.frames) || input.frames.length === 0) return;
+ const sse = input.frames.join('');
+ if (sse.length === 0 || Buffer.byteLength(sse) > STREAM_CACHE_MAX_BYTES) return;
streamStore.delete(cacheKey);
streamStore.set(cacheKey, {
- frames: input.frames,
+ sse,
platform: input.platform,
modelId: input.modelId,
keyId: input.keyId,
@@ -640,12 +679,18 @@ export function storeCachedStreamResponse(cacheKey: string, input: StoreStreamIn
lastHitAtMs: null,
});
- evictToCap(streamStore);
+ enforceCombinedCap();
}
// ── Stats / admin ──
export interface CacheStats {
+ /**
+ * Entries held across BOTH stores (JSON completions + streaming replays),
+ * which is also what the shared RESPONSE_CACHE_MAX_ENTRIES cap bounds. One
+ * prompt asked both streaming and non-streaming therefore counts twice —
+ * they are two independent replayable artifacts, and two slots of the cap.
+ */
entries: number;
/** Hits accumulated by the entries currently held, restored from SQLite. */
totalHits: number;