Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
fix(metrics): split rate-limited by source and label failures by provider (#3707)
Tejas Chopra5ff4ea1ef948
diff --git a/docs/content/docs/metrics.mdx b/docs/content/docs/metrics.mdxindex d7dacba53..b9ce54a65 100644--- a/docs/content/docs/metrics.mdx+++ b/docs/content/docs/metrics.mdx@@ -25,7 +25,9 @@ curl http://localhost:8787/stats"total": 42,"cached": 5,"rate_limited": 0,- "failed": 0+ "rate_limited_by_source": { "headroom": 0, "upstream": 0 },+ "failed": 0,+ "failed_by_provider": {}},"tokens": {"input": 50000,@@ -44,6 +46,14 @@ curl http://localhost:8787/stats}```+`requests.total` is the **completed**-request count, not total traffic: a request that+ended in a 4xx, 5xx or 429 is counted under `requests.failed` or `requests.rate_limited`+instead and never reaches `total`. `rate_limited_by_source` splits 429s into+`headroom` (Headroom's own rate limiter refused the request) and `upstream` (the provider+refused it) — different problems, different fixes. `failed_by_provider` attributes failures+to the upstream that produced them. The unlabelled `rate_limited` / `failed` totals are+unchanged.+Persistent savings are stored at `~/.headroom/proxy_savings.json` and survive proxy restarts. Override the path with `HEADROOM_SAVINGS_PATH`.### Historical Savings@@ -82,8 +92,27 @@ headroom_latency_ms_count 1234# HELP headroom_provider_cache_hit_requests_total Requests that read from the provider's prompt cacheheadroom_provider_cache_hit_requests_total{provider="anthropic"} 456++# HELP headroom_requests_failed_total Requests that failed upstream (4xx and 5xx, excluding 429), by provider+headroom_requests_failed_total{provider="anthropic"} 3++# HELP headroom_requests_rate_limited_total Requests rejected with 429, by who rejected them (headroom=our own limiter, upstream=the provider)+headroom_requests_rate_limited_total{source="headroom"} 0+headroom_requests_rate_limited_total{source="upstream"} 12```+<Callout type="warn" title="requests_total is completed requests, not all traffic">+`headroom_requests_total` counts requests that **completed**. A request that ended in a 4xx,+5xx or 429 lands in `headroom_requests_failed_total` / `headroom_requests_rate_limited_total`+and is not in `requests_total` — so a failure rate must add `failed` back into its own+denominator. `headroom_inbound_requests_total` is the all-traffic counter.++`headroom_requests_failed_total` gained a `provider` label and+`headroom_requests_rate_limited_total` a `source` label, so unaggregated queries on them now+return several series. `sum without (provider) (...)` and `sum without (source) (...)`+reproduce the previous single-series behaviour.+</Callout>+<Callout type="warn" title="No percentile buckets on /metrics">There are no Prometheus histogram buckets for compression ratio or latency — build percentiles or averages from `_sum` / `_count` pairs (`headroom_latency_ms_sum`, `headroom_latency_ms_count`, and so on), or use the `headroom perf` CLI for real p95/p99.</Callout>@@ -351,7 +380,8 @@ Env: `HEADROOM_BUDGET_ESTIMATED_BASIS`. `headroom doctor` reports the estimated| `headroom_persistent_savings_tokens_saved_total` | Durable lifetime tokens saved from `/stats.persistent_savings` | Higher is better || `headroom_overhead_ms_sum` / `headroom_overhead_ms_count` | Mean latency Headroom itself adds (there are no percentile buckets — see the callout above) | Low is better || `headroom_provider_cache_hit_requests_total` / `headroom_provider_cache_requests_total` | Cache effectiveness, by provider | >20% is good |-| `headroom_requests_failed_total` | Reliability (upstream 5xx errors) | 0 |+| `headroom_requests_failed_total{provider}` | Reliability — upstream **4xx and 5xx** (429s are counted separately), attributed to the provider that failed | 0 |+| `headroom_requests_rate_limited_total{source}` | Throttling — `source="headroom"` is Headroom's own rate limiter, `source="upstream"` is the provider's | 0 |## Grafana Dashboarddiff --git a/docs/metrics-technical-guide.md b/docs/metrics-technical-guide.mdindex 63dbe691e..b5ff7e152 100644--- a/docs/metrics-technical-guide.md+++ b/docs/metrics-technical-guide.md@@ -103,21 +103,55 @@ rate(headroom_cache_bust_total[5m])| Metric | What it shows ||---|---|-| `headroom_requests_total` | Requests handled. Unlabelled. |-| `headroom_requests_by_provider{provider}` | Traffic split by provider — `anthropic`, `openai`, `gemini`, `bedrock`… |-| `headroom_requests_by_model{model}` | Traffic split by model. Capped at 1024 distinct; overflow lands in `model="other"`. |-| `headroom_requests_failed_total` | Upstream 5xx errors. |-| `headroom_requests_rate_limited_total` | Requests **Headroom** rejected via its own rate limiter (not upstream 429s). |+| `headroom_requests_total` | **Completed** requests — the denominator for success-side stats, not total traffic. A request that ended in a 4xx, 5xx or 429 never reaches it. Unlabelled. |+| `headroom_inbound_requests_total` | Every inbound HTTP request the proxy accepted, whatever the outcome. This is the all-traffic counter. |+| `headroom_requests_by_provider{provider}` | Traffic split by provider — `anthropic`, `openai`, `gemini`, `bedrock`… **Completed requests only**, same denominator as `headroom_requests_total`. Use `headroom_requests_failed_total{provider}` for the failure side. |+| `headroom_requests_by_model{model}` | Traffic split by model. Capped at 1024 distinct; overflow lands in `model="other"`. Completed requests only. |+| `headroom_requests_failed_total{provider}` | Requests that failed upstream — **4xx and 5xx**, excluding 429. Excluded from `headroom_requests_total`, so add it back when you compute a rate. |+| `headroom_requests_rate_limited_total{source}` | Requests rejected with 429, by **either** Headroom's own rate limiter (`source="headroom"`) or the upstream provider (`source="upstream"`). Excluded from `headroom_requests_total`. || `headroom_compression_failed_total{reason}` | Compression failures — `timeout` or `error`. Fails open, so traffic keeps flowing but savings quietly stop. **Worth an alert.** || `headroom_compression_quarantine_total{event}` | Compression disabled after repeated timeouts — `activated`, `skipped`, `released`. || `headroom_inbound_requests_active` | In-flight requests, gauge. Counts all HTTP including `/metrics`. || `headroom_active_ws_sessions` | Live Codex WebSocket sessions, gauge. |+### `headroom_requests_total` is a denominator, not traffic++`headroom_requests_total` counts requests that **completed** — it has always been+incremented only on the success path, and since the outcome funnel started short-circuiting+at `>= 400`, every 4xx, 5xx and 429 drops out of it. So:++- `headroom_requests_total` + `headroom_requests_failed_total` + `headroom_requests_rate_limited_total`+ ≈ the requests the proxy forwarded upstream.+- `headroom_inbound_requests_total` is the honest all-traffic counter (it also counts+ `/metrics`, `/stats` and other non-proxy HTTP).++That is why the failure-rate query below adds `failed` back into its own denominator, and+why you should not "simplify" it to `… / rate(headroom_requests_total[5m])`: failures are+not in `requests_total`, so that form divides failures by successes and over-reports the+rate — badly, during exactly the incident you wrote it for (at 100% failures the denominator+goes to zero).+```promql-# Failure rate+# Failure rate. The denominator deliberately re-adds `failed`: a failed request+# is NOT in headroom_requests_total, so requests_total alone is successes-only.rate(headroom_requests_failed_total[5m])/ clamp_min(rate(headroom_requests_total[5m]) + rate(headroom_requests_failed_total[5m]), 1)+# Failure rate by provider — which upstream is actually broken.+sum by (provider) (rate(headroom_requests_failed_total[5m]))+ / clamp_min(+ sum by (provider) (rate(headroom_requests_by_provider[5m]))+ + sum by (provider) (rate(headroom_requests_failed_total[5m])),+ 1+ )++# Who is throttling you: your own limiter (raise the cap) vs the provider+# (back off, shard keys). Alert on these separately — they are different actions.+sum by (source) (rate(headroom_requests_rate_limited_total[5m]))++# Provider throttling only — the one that means "slow down".+rate(headroom_requests_rate_limited_total{source="upstream"}[5m])+# Savings silently stoppedsum by (reason) (rate(headroom_compression_failed_total[5m]))@@ -125,6 +159,22 @@ sum by (reason) (rate(headroom_compression_failed_total[5m]))sum by (provider) (rate(headroom_requests_by_provider[5m]))```+> **Migration — these two counters gained labels.** `headroom_requests_failed_total` is now+> labelled by `provider` and `headroom_requests_rate_limited_total` by `source`, so an+> unaggregated query that used to return one series now returns several, and a panel or+> alert that graphed the bare counter will fan out into one line per label value.+> `sum without (provider) (rate(headroom_requests_failed_total[5m]))` and+> `sum without (source) (rate(headroom_requests_rate_limited_total[5m]))` reproduce the old+> single-series behaviour exactly. Both `source` series are exported from process start+> (including at zero); `failed` exports one series per provider that has actually failed, so+> it has no series at all until the first failure.+>+> The same split is available outside Prometheus: `/stats` carries+> `requests.rate_limited_by_source` and `requests.failed_by_provider` alongside the+> unlabelled `requests.rate_limited` / `requests.failed` totals (which are unchanged), the+> lifetime aggregate carries the same two maps, and the OTel counters+> `headroom.proxy.requests.rate_limited` / `.failed` carry `source` / `provider` attributes.+---## Anthropic subscription paneldiff --git a/headroom/dashboard/templates/dashboard.html b/headroom/dashboard/templates/dashboard.htmlindex 15661140c..f796393b6 100644--- a/headroom/dashboard/templates/dashboard.html+++ b/headroom/dashboard/templates/dashboard.html@@ -210,7 +210,7 @@<div class="grid grid-cols-1 gap-4 mb-6 lg:grid-cols-2"><section class="bg-surface rounded-lg p-4 border border-border"><h3 class="mb-3 text-sm font-medium text-gray-300">Request Health</h3>- <div class="grid grid-cols-2 gap-3 text-sm"><div>Completed <span class="float-right tabular-nums" x-text="formatNumber(stats.requests?.total || 0)"></span></div><div>Failed <span class="float-right tabular-nums text-red-400" x-text="formatNumber(stats.requests?.failed || 0)"></span></div><div>Rate Limited <span class="float-right tabular-nums text-amber-400" x-text="formatNumber(stats.requests?.rate_limited || 0)"></span></div><div>Cached <span class="float-right tabular-nums text-cyan-400" x-text="formatNumber(stats.requests?.cached || 0)"></span></div></div>+ <div class="grid grid-cols-2 gap-3 text-sm"><div>Completed <span class="float-right tabular-nums" x-text="formatNumber(stats.requests?.total || 0)"></span></div><div title="Requests that failed upstream — 4xx and 5xx. 429s are counted under Rate Limited instead, and neither is included in Completed.">Failed <span class="float-right tabular-nums text-red-400" x-text="formatNumber(stats.requests?.failed || 0)"></span></div><div title="Requests rejected with 429 — by Headroom's own rate limiter or by the upstream provider. /stats splits them under requests.rate_limited_by_source.">Rate Limited <span class="float-right tabular-nums text-amber-400" x-text="formatNumber(stats.requests?.rate_limited || 0)"></span></div><div>Cached <span class="float-right tabular-nums text-cyan-400" x-text="formatNumber(stats.requests?.cached || 0)"></span></div></div></section><section class="bg-surface rounded-lg p-4 border border-border"><h3 class="mb-3 text-sm font-medium text-gray-300">Live Activity</h3>@@ -418,7 +418,7 @@<span class="font-mono text-sm" x-text="((stats.ttfb?.min_ms || 0) / 1000).toFixed(2) + ' - ' + ((stats.ttfb?.max_ms || 0) / 1000).toFixed(2) + 's'"></span></div><div class="flex justify-between items-center">- <span class="text-sm text-gray-400">Failed Requests</span>+ <span class="text-sm text-gray-400" title="Requests that failed upstream — 4xx and 5xx. 429s are counted under Rate Limited instead, and neither is included in Completed.">Failed Requests</span><span class="font-mono text-sm" x-text="stats.requests?.failed || 0"></span></div><!-- Per-transform timing breakdown -->@@ -1470,8 +1470,8 @@</div><div class="grid grid-cols-2 gap-4 mb-6 md:grid-cols-4"><div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500">Requests</div><div class="text-2xl tabular-nums" x-text="formatNumber(lifetimeStats.requests?.total || 0)"></div></div>- <div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500">Failed</div><div class="text-2xl tabular-nums text-red-400" x-text="formatNumber(lifetimeStats.requests?.failed || 0)"></div></div>- <div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500">Rate Limited</div><div class="text-2xl tabular-nums text-amber-400" x-text="formatNumber(lifetimeStats.requests?.rate_limited || 0)"></div></div>+ <div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500" title="Requests that failed upstream — 4xx and 5xx. 429s are counted under Rate Limited instead, and neither is included in Completed.">Failed</div><div class="text-2xl tabular-nums text-red-400" x-text="formatNumber(lifetimeStats.requests?.failed || 0)"></div></div>+ <div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500" title="Requests rejected with 429 — by Headroom's own rate limiter or by the upstream provider. /stats splits them under requests.rate_limited_by_source.">Rate Limited</div><div class="text-2xl tabular-nums text-amber-400" x-text="formatNumber(lifetimeStats.requests?.rate_limited || 0)"></div></div><div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500">Cached</div><div class="text-2xl tabular-nums text-cyan-400" x-text="formatNumber(lifetimeStats.requests?.cached || 0)"></div></div></div>diff --git a/headroom/observability/metrics.py b/headroom/observability/metrics.pyindex 93bdd93ef..430c690a3 100644--- a/headroom/observability/metrics.py+++ b/headroom/observability/metrics.py@@ -207,12 +207,15 @@ class HeadroomOtelMetrics:)self._proxy_failed_requests = self._meter.create_counter("headroom.proxy.requests.failed",- description="Proxy requests that failed.",+ description="Proxy requests that failed upstream (4xx and 5xx, excluding 429).",unit="1",)self._proxy_rate_limited_requests = self._meter.create_counter("headroom.proxy.requests.rate_limited",- description="Proxy requests rejected by rate limiting.",+ description=(+ "Proxy requests rejected with 429, by source "+ "(headroom=our own limiter, upstream=the provider)."+ ),unit="1",)self._proxy_input_tokens = self._meter.create_counter(@@ -567,8 +570,15 @@ class HeadroomOtelMetrics:*,provider: str | None = None,model: str | None = None,+ source: str = "headroom",) -> None:- self._proxy_rate_limited_requests.add(1, self._attrs(provider=provider, model=model))+ # ``source`` mirrors the headroom_requests_rate_limited_total{source}+ # Prometheus label: "headroom" = our own limiter refused the request,+ # "upstream" = the provider did. Same split in both backends so an+ # operator's query means the same thing whichever one they scrape.+ self._proxy_rate_limited_requests.add(+ 1, self._attrs(provider=provider, model=model, source=source)+ )def record_proxy_cache_bust(self, *, tokens_lost: int) -> None:self._proxy_cache_busts.add(1)diff --git a/headroom/proxy/handlers/anthropic.py b/headroom/proxy/handlers/anthropic.pyindex 66c074299..cf321e3fc 100644--- a/headroom/proxy/handlers/anthropic.py+++ b/headroom/proxy/handlers/anthropic.py@@ -1253,7 +1253,9 @@ class AnthropicHandlerMixin:rate_key = f"{api_key[:16]}:{client_ip}" if api_key else client_ipallowed, wait_seconds = await self.rate_limiter.check_request(rate_key)if not allowed:- await self.metrics.record_rate_limited(provider=provider_name)+ await self.metrics.record_rate_limited(+ provider=provider_name, source="headroom"+ )# Unit 4: release the pre-upstream semaphore before we# bail out of the handler via HTTPException — FastAPI's# exception handler will NOT run our ``finally``.diff --git a/headroom/proxy/handlers/gemini.py b/headroom/proxy/handlers/gemini.pyindex 3f82145be..0d8e1bc36 100644--- a/headroom/proxy/handlers/gemini.py+++ b/headroom/proxy/handlers/gemini.py@@ -402,7 +402,7 @@ class GeminiHandlerMixin:rate_key = headers.get("x-goog-api-key", "default")[:20]allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)if not allowed:- await self.metrics.record_rate_limited(provider=provider_name)+ await self.metrics.record_rate_limited(provider=provider_name, source="headroom")raise HTTPException(status_code=429,detail=f"Rate limited. Retry after {wait_seconds:.1f}s",diff --git a/headroom/proxy/handlers/openai.py b/headroom/proxy/handlers/openai.pyindex 41406ef5d..910be1d41 100644--- a/headroom/proxy/handlers/openai.py+++ b/headroom/proxy/handlers/openai.py@@ -3551,7 +3551,9 @@ class OpenAIHandlerMixin:rate_key = headers.get("authorization", "default")[:20]allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)if not allowed:- await self.metrics.record_rate_limited(provider=openai_chat_outcome_provider)+ await self.metrics.record_rate_limited(+ provider=openai_chat_outcome_provider, source="headroom"+ )raise HTTPException(status_code=429,detail=f"Rate limited. Retry after {wait_seconds:.1f}s",@@ -5676,7 +5678,7 @@ class OpenAIHandlerMixin:rate_key = headers.get("authorization", "default")[:20]allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)if not allowed:- await self.metrics.record_rate_limited(provider="openai")+ await self.metrics.record_rate_limited(provider="openai", source="headroom")raise HTTPException(status_code=429,detail=f"Rate limited. Retry after {wait_seconds:.1f}s",diff --git a/headroom/proxy/outcome.py b/headroom/proxy/outcome.pyindex 31c443fbd..cd85992e9 100644--- a/headroom/proxy/outcome.py+++ b/headroom/proxy/outcome.py@@ -531,9 +531,16 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:# limit is the one 4xx a user is expected to act on (back off, raise a cap),# and folding it into a generic failure count hides exactly that. Both# counters are already exported and neither feeds savings.+ #+ # source="upstream": this funnel only ever sees a 429 the PROVIDER returned.+ # Headroom's own limiter rejects before a request is ever sent and records+ # source="headroom" from the handler. The two are acted on differently —+ # raise our cap vs. back off / shard keys — so they must stay separable+ # (issue #3696). ``outcome.provider`` is the already-Copilot-relabelled+ # value, matching every other provider-labelled metric on this path.if outcome.status_code >= 400:if outcome.status_code == 429:- await handler.metrics.record_rate_limited(provider=outcome.provider)+ await handler.metrics.record_rate_limited(provider=outcome.provider, source="upstream")else:await handler.metrics.record_failed(provider=outcome.provider)returndiff --git a/headroom/proxy/persistent_metrics.py b/headroom/proxy/persistent_metrics.pyindex 9804d74a5..d7a5a6c13 100644--- a/headroom/proxy/persistent_metrics.py+++ b/headroom/proxy/persistent_metrics.py@@ -16,6 +16,9 @@ MAX_EXPOSED_MODELS = 100MAX_LABEL_LENGTH = 128KNOWN_MISS_REASONS = frozenset({"ttl_expiry", "prefix_change", "unknown"})+# Mirrors RATE_LIMIT_SOURCES in prometheus_metrics: "headroom" is our own+# limiter refusing the request, "upstream" is the provider refusing it.+RATE_LIMIT_SOURCES = frozenset({"headroom", "upstream"})# Names must match ``WasteSignals.to_dict()`` in headroom/config.py — the# parser emits ``json_bloat``; an allowlist that says ``json_noise`` silently# shoves the largest waste category into the catch-all bucket.@@ -91,6 +94,12 @@ def _empty_state() -> dict[str, Any]:"cached": 0,"failed": 0,"rate_limited": 0,+ # Who produced the failure / who refused the request. The totals+ # above stay the unlabelled figures the dashboard reads; these+ # split them the same way the Prometheus labels do, so no backend+ # reports a differently-shaped counter (issue #3696).+ "failed_by_provider": {},+ "rate_limited_by_source": {},"by_provider": {},"by_stack": {},},@@ -182,6 +191,17 @@ class PersistentMetricsState:result["requests"]["by_provider"] = self._normalize_count_map(raw_requests.get("by_provider"), MAX_PROVIDER_VALUES)+ result["requests"]["failed_by_provider"] = self._normalize_count_map(+ raw_requests.get("failed_by_provider"), MAX_PROVIDER_VALUES+ )+ # A state file written before #3696 has no split, and back-filling the+ # legacy total into one bucket would invent history we never observed —+ # so it loads empty and only new events populate it. The unlabelled+ # ``rate_limited`` total is untouched and stays comparable across the+ # upgrade.+ result["requests"]["rate_limited_by_source"] = self._normalize_enum_map(+ raw_requests.get("rate_limited_by_source"), RATE_LIMIT_SOURCES, fallback="headroom"+ )result["requests"]["by_stack"] = self._normalize_count_map(raw_requests.get("by_stack"), MAX_STACK_VALUES)@@ -474,12 +494,29 @@ class PersistentMetricsState:self._record_activity()self._state["requests"]["failed"] += 1+ self._increment_count(+ self._state["requests"]["failed_by_provider"], _label(provider), MAX_PROVIDER_VALUES+ )++ def record_rate_limited(+ self,+ *,+ provider: str | None = None,+ model: str | None = None,+ source: str = "headroom",+ ) -> None:+ """Record a rate-limited request without redefining total request semantics.- def record_rate_limited(self, *, provider: str | None = None, model: str | None = None) -> None:- """Record a rate-limited request without redefining total request semantics."""+ ``source`` is ``headroom`` (our own limiter) or ``upstream`` (the+ provider's 429); anything else is clamped to ``headroom``, which is what+ this counter meant before upstream 429s started reaching it.+ """self._record_activity()self._state["requests"]["rate_limited"] += 1+ bucket = source if source in RATE_LIMIT_SOURCES else "headroom"+ by_source = self._state["requests"]["rate_limited_by_source"]+ by_source[bucket] = by_source.get(bucket, 0) + 1def record_cache_bust(self, *, tokens_lost: Any = 0) -> None:self._record_activity()diff --git a/headroom/proxy/prometheus_metrics.py b/headroom/proxy/prometheus_metrics.pyindex b0252f008..e62759f24 100644--- a/headroom/proxy/prometheus_metrics.py+++ b/headroom/proxy/prometheus_metrics.py@@ -30,6 +30,33 @@ logger = logging.getLogger("headroom.proxy")# client-supplied model cardinality stays bounded (see record_request)._OTHER_MODEL = "other"+# Closed label set for headroom_requests_rate_limited_total{source}. Two 429s+# mean opposite things to an operator: "headroom" is OUR limiter refusing the+# request (raise the cap), "upstream" is the provider refusing it (back off or+# shard keys). Before #3615 only the first could ever increment this counter;+# the outcome funnel now routes provider 429s here too, so the split has to be+# queryable instead of silently merged. Closed set => bounded cardinality, and+# both series are exported from startup so a rate() never goes from absent to+# present mid-incident.+RATE_LIMIT_SOURCE_HEADROOM = "headroom"+RATE_LIMIT_SOURCE_UPSTREAM = "upstream"+RATE_LIMIT_SOURCES = (RATE_LIMIT_SOURCE_HEADROOM, RATE_LIMIT_SOURCE_UPSTREAM)++# Bucket for a failure with no attributed provider. Also seeded at zero so+# headroom_requests_failed_total always exports at least one sample.+_PROVIDER_UNKNOWN = "unknown"+++def _rate_limit_source(source: str | None) -> str:+ """Clamp ``source`` to the closed label set, defaulting to Headroom's limiter.++ Defaults to ``headroom`` because that is what this counter meant for its+ whole life before the outcome funnel started feeding it upstream 429s: an+ un-updated caller keeps the historical reading rather than inventing a new+ label value.+ """+ return source if source in RATE_LIMIT_SOURCES else RATE_LIMIT_SOURCE_HEADROOM+def _escape_label_value(value: str) -> str:# The /metrics body is emitted whole with .encode("utf-8") (server.py). A@@ -100,7 +127,26 @@ class PrometheusMetrics:self.requests_by_stack: dict[str, int] = defaultdict(int)self.requests_cached = 0self.requests_rate_limited = 0+ # Seeded with both sources at zero so /metrics exports the full series+ # set from the first scrape; ``requests_rate_limited`` stays the+ # unlabelled total that /stats and the session summary read.+ self.requests_rate_limited_by_source: dict[str, int] = dict.fromkeys(RATE_LIMIT_SOURCES, 0)self.requests_failed = 0+ # Per-provider failure attribution. #3615 moved 4xx out of the success+ # funnel, which also took them out of ``requests_by_provider`` (only+ # record_request touches that), leaving Prometheus with no way to tell+ # which upstream was failing. Kept alongside the unlabelled total.+ #+ # Seeded with the "unknown" bucket at zero so the metric always exports+ # at least one sample. Before this counter carried a label it was always+ # present as a bare ``headroom_requests_failed_total 0``; a labelled map+ # that starts empty would emit NO sample on a healthy proxy, which turns+ # the documented failure-rate query into an empty vector (an empty+ # numerator makes the whole expression empty, so the panel reads+ # "No data" instead of 0%) and would make ``absent()`` alerts fire on+ # healthy proxies. "unknown" is a real bucket -- record_failed uses it+ # when no provider is attributed -- so seeding it invents no provider.+ self.requests_failed_by_provider: dict[str, int] = defaultdict(int, {_PROVIDER_UNKNOWN: 0})self.inbound_requests_total = 0self.inbound_requests_completed = 0self.inbound_requests_active = 0@@ -343,7 +389,11 @@ class PrometheusMetrics:self.requests_by_stack.clear()self.requests_cached = 0self.requests_rate_limited = 0+ self.requests_rate_limited_by_source = dict.fromkeys(RATE_LIMIT_SOURCES, 0)self.requests_failed = 0+ self.requests_failed_by_provider.clear()+ # Re-seed so /metrics keeps exporting a sample after a reset.+ self.requests_failed_by_provider[_PROVIDER_UNKNOWN] = 0self.inbound_requests_total = 0self.inbound_requests_completed = 0self.inbound_requests_active = 0@@ -1175,15 +1225,38 @@ class PrometheusMetrics:if ms_val > self.ws_session_duration_max_ms[cause]:self.ws_session_duration_max_ms[cause] = ms_val- async def record_rate_limited(self, *, provider: str | None = None, model: str | None = None):+ async def record_rate_limited(+ self,+ *,+ provider: str | None = None,+ model: str | None = None,+ source: str = RATE_LIMIT_SOURCE_HEADROOM,+ ):+ """Record a 429.++ ``source`` says WHO refused the request: ``headroom`` for our own+ limiter (the handlers, which raise HTTPException and never emit an+ outcome) or ``upstream`` for a provider 429 arriving through the+ outcome funnel. Threaded to every sink — Prometheus, lifetime/persistent+ and OTel — so no backend reports a differently-shaped counter.+ """+ source = _rate_limit_source(source)async with self._lock:self.requests_rate_limited += 1- self.savings_tracker.record_lifetime_rate_limited(provider=provider, model=model)- self._get_otel_metrics().record_proxy_rate_limited(provider=provider, model=model)+ self.requests_rate_limited_by_source[source] = (+ self.requests_rate_limited_by_source.get(source, 0) + 1+ )+ self.savings_tracker.record_lifetime_rate_limited(+ provider=provider, model=model, source=source+ )+ self._get_otel_metrics().record_proxy_rate_limited(+ provider=provider, model=model, source=source+ )async def record_failed(self, *, provider: str | None = None, model: str | None = None):async with self._lock:self.requests_failed += 1+ self.requests_failed_by_provider[provider or _PROVIDER_UNKNOWN] += 1self.savings_tracker.record_lifetime_failed(provider=provider, model=model)self._get_otel_metrics().record_proxy_failed(provider=provider, model=model)@@ -1215,20 +1288,39 @@ class PrometheusMetrics:help_text="Cached request count",value=self.requests_cached,)- _append_metric(- lines,- name="headroom_requests_rate_limited_total",- metric_type="counter",- help_text="Rate limited requests",- value=self.requests_rate_limited,+ # Labelled series only — an unlabelled sample alongside these would+ # double-count under sum(). `sum without (source)` reproduces the+ # pre-label value exactly.+ lines.extend(+ [+ "# HELP headroom_requests_rate_limited_total Requests rejected with 429, "+ "by who rejected them (headroom=our own limiter, upstream=the provider)",+ "# TYPE headroom_requests_rate_limited_total counter",+ ])- _append_metric(- lines,- name="headroom_requests_failed_total",- metric_type="counter",- help_text="Failed requests",- value=self.requests_failed,+ for _source in RATE_LIMIT_SOURCES:+ _count = self.requests_rate_limited_by_source.get(_source, 0)+ lines.append(+ f'headroom_requests_rate_limited_total{{source="{_escape_label_value(_source)}"}}'+ f" {_count}"+ )+ lines.append("")++ # Failures carry the provider that produced them. `sum without+ # (provider)` reproduces the pre-label value.+ lines.extend(+ [+ "# HELP headroom_requests_failed_total Requests that failed upstream "+ "(4xx and 5xx, excluding 429), by provider",+ "# TYPE headroom_requests_failed_total counter",+ ])+ for _provider, _count in self.requests_failed_by_provider.items():+ lines.append(+ f'headroom_requests_failed_total{{provider="'+ f'{_escape_label_value(str(_provider))}"}} {_count}'+ )+ lines.append("")_append_metric(lines,name="headroom_inbound_requests_total",diff --git a/headroom/proxy/savings_tracker.py b/headroom/proxy/savings_tracker.pyindex 37627b54b..207a2b72f 100644--- a/headroom/proxy/savings_tracker.py+++ b/headroom/proxy/savings_tracker.py@@ -1210,12 +1210,22 @@ class SavingsTracker:self._maybe_save_locked()def record_lifetime_rate_limited(- self, *, provider: str | None = None, model: str | None = None+ self,+ *,+ provider: str | None = None,+ model: str | None = None,+ source: str = "headroom",) -> None:- """Record a rate-limited proxy request without changing legacy history."""+ """Record a rate-limited proxy request without changing legacy history.++ ``source`` distinguishes Headroom's own limiter from an upstream 429 —+ see ``PrometheusMetrics.record_rate_limited``.+ """with self._lock:- self._persistent_metrics.record_rate_limited(provider=provider, model=model)+ self._persistent_metrics.record_rate_limited(+ provider=provider, model=model, source=source+ )self._maybe_save_locked()def record_lifetime_cache_bust(self, *, tokens_lost: int) -> None:diff --git a/headroom/proxy/server.py b/headroom/proxy/server.pyindex 357e9fe74..a2371960a 100644--- a/headroom/proxy/server.py+++ b/headroom/proxy/server.py@@ -2263,6 +2263,12 @@ class HeadroomProxy(logger.info(f"Total requests: {m.requests_total}")logger.info(f"Cached responses: {m.requests_cached}")logger.info(f"Rate limited: {m.requests_rate_limited}")+ if m.requests_rate_limited:+ # The split an operator acts on differently: our limiter firing+ # means raise the cap, the provider's means back off / shard keys.+ by_source = m.requests_rate_limited_by_source+ logger.info(f" headroom limiter: {by_source.get('headroom', 0)}")+ logger.info(f" upstream 429s: {by_source.get('upstream', 0)}")logger.info(f"Failed: {m.requests_failed}")logger.info(f"Input tokens: {m.tokens_input_total:,}")logger.info(f"Output tokens: {m.tokens_output_total:,}")@@ -4572,7 +4578,14 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:"total": m.requests_total,"cached": m.requests_cached,"rate_limited": m.requests_rate_limited,+ # Who issued the 429: "headroom" is our own limiter, "upstream"+ # is the provider (issue #3696). The totals above stay the+ # unlabelled figures existing consumers read.+ "rate_limited_by_source": dict(m.requests_rate_limited_by_source),"failed": m.requests_failed,+ "failed_by_provider": _remap_provider_counts(+ dict(m.requests_failed_by_provider), proxy.config+ ),"by_provider": _remap_provider_counts(dict(m.requests_by_provider), proxy.config),"by_model": dict(m.requests_by_model),"by_stack": dict(m.requests_by_stack),diff --git a/tests/test_observability_metrics.py b/tests/test_observability_metrics.pyindex eb8c2c521..ffff3438d 100644--- a/tests/test_observability_metrics.py+++ b/tests/test_observability_metrics.py@@ -356,11 +356,56 @@ async def test_prometheus_metrics_reads_late_configured_otel_metrics() -> None:await metrics.record_rate_limited(provider="anthropic", model="claude-sonnet")assert spy.failed_calls == [{"provider": "openai", "model": None}]- assert spy.rate_limited_calls == [{"provider": "anthropic", "model": "claude-sonnet"}]+ # ``source`` reaches OTel too — the split must not exist in Prometheus only.+ assert spy.rate_limited_calls == [+ {"provider": "anthropic", "model": "claude-sonnet", "source": "headroom"}+ ]finally:reset_otel_metrics()+async def test_prometheus_metrics_forwards_rate_limit_source_to_otel() -> None:+ """An upstream 429 must reach OTel labelled as upstream, not as our own limiter."""++ spy = _SpyProxyMetrics()+ metrics = PrometheusMetrics(stateless=True)+ set_otel_metrics(spy) # type: ignore[arg-type]++ try:+ await metrics.record_rate_limited(provider="anthropic", source="upstream")+ await metrics.record_rate_limited(provider="anthropic", source="headroom")+ # An unrecognised value is clamped rather than exported as a new label.+ await metrics.record_rate_limited(provider="anthropic", source="nonsense")++ assert [call["source"] for call in spy.rate_limited_calls] == [+ "upstream",+ "headroom",+ "headroom",+ ]+ finally:+ reset_otel_metrics()+++def test_otel_rate_limited_counter_carries_source_attribute() -> None:+ reader = InMemoryMetricReader()+ provider = MeterProvider(metric_readers=[reader])+ otel_metrics = HeadroomOtelMetrics(meter_provider=provider)++ otel_metrics.record_proxy_rate_limited(provider="anthropic", source="upstream")+ otel_metrics.record_proxy_rate_limited(provider="openai", source="headroom")++ metrics = _collect_metrics(reader)+ upstream = _find_point(+ metrics["headroom.proxy.requests.rate_limited"], provider="anthropic", source="upstream"+ )+ headroom_side = _find_point(+ metrics["headroom.proxy.requests.rate_limited"], provider="openai", source="headroom"+ )+ assert upstream.value == 1+ assert headroom_side.value == 1++@pytest.mark.asyncioasync def test_prometheus_metrics_forwards_savings_drilldown_fields_to_otel(monkeypatch: pytest.MonkeyPatch,diff --git a/tests/test_outcome_records_5xx_as_failed.py b/tests/test_outcome_records_5xx_as_failed.pyindex 2d59b51fa..b4d68f8a6 100644--- a/tests/test_outcome_records_5xx_as_failed.py+++ b/tests/test_outcome_records_5xx_as_failed.py@@ -25,13 +25,19 @@ class _Metrics:def __init__(self):self.failed = []self.rate_limited = []+ self.rate_limited_sources = []self.requested = []async def record_failed(self, provider):self.failed.append(provider)- async def record_rate_limited(self, provider):+ # ``source`` deliberately has NO default here: the funnel only ever sees+ # 429s the provider returned, so it must pass source="upstream" explicitly.+ # If it ever stops doing so, these tests record None and fail loudly rather+ # than silently re-merging upstream throttling into Headroom's own counter.+ async def record_rate_limited(self, provider, source=None):self.rate_limited.append(provider)+ self.rate_limited_sources.append(source)async def record_request(self, **kwargs):self.requested.append(kwargs)@@ -78,6 +84,11 @@ def test_429_recorded_as_rate_limited_and_skips_success_funnel():handler = _Handler()asyncio.run(emit_request_outcome(handler, _outcome(429, tokens_saved=6380)))assert handler.metrics.rate_limited == ["anthropic"]+ # Labelled as the PROVIDER's 429, not ours. Headroom's own limiter rejects+ # before a request is ever sent and records source="headroom" from the+ # handler; merging the two would tell an operator to raise a cap that is+ # not the one being hit (issue #3696).+ assert handler.metrics.rate_limited_sources == ["upstream"]assert handler.metrics.failed == [] # a rate limit is not a generic failureassert handler.metrics.requested == [] # its 6,380 "saved" tokens are not savingsdiff --git a/tests/test_persistent_metrics.py b/tests/test_persistent_metrics.pyindex 018c71bb6..3a5d054d8 100644--- a/tests/test_persistent_metrics.py+++ b/tests/test_persistent_metrics.py@@ -50,6 +50,9 @@ def test_snapshot_accumulates_request_token_cache_cost_and_waste_metrics() -> No"cached": 1,"failed": 1,"rate_limited": 1,+ # Lifetime carries the same splits the Prometheus labels do.+ "failed_by_provider": {"anthropic": 1},+ "rate_limited_by_source": {"headroom": 1},"by_provider": {"anthropic": 1},"by_stack": {"codex": 1},}@@ -205,3 +208,56 @@ def test_miss_reasons_still_fall_back_to_unknown_on_reload() -> None:snapshot = state.snapshot(persistence={"enabled": True, "healthy": True})assert snapshot["prefix_cache"]["misses_by_reason"] == {"ttl_expiry": 2, "unknown": 1}+++def test_rate_limited_is_split_by_source_and_failures_by_provider() -> None:+ """Headroom's own limiter and an upstream 429 must stay distinguishable.++ Both land in the same ``rate_limited`` total (unchanged), but an operator+ acts on them differently: ours firing means raise the cap, the provider's+ means back off. See issue #3696.+ """+ state = _new_state()++ state.record_rate_limited(provider="anthropic", source="headroom")+ state.record_rate_limited(provider="anthropic", source="upstream")+ state.record_rate_limited(provider="openai", source="upstream")+ state.record_failed(provider="anthropic")+ state.record_failed(provider="openai")+ state.record_failed(provider="openai")++ requests = state.snapshot(persistence={"enabled": True, "healthy": True})["requests"]++ assert requests["rate_limited"] == 3+ assert requests["rate_limited_by_source"] == {"headroom": 1, "upstream": 2}+ assert requests["failed"] == 3+ assert requests["failed_by_provider"] == {"anthropic": 1, "openai": 2}+++def test_rate_limit_source_defaults_to_headroom_and_clamps_unknown_values() -> None:+ state = _new_state()++ state.record_rate_limited(provider="anthropic")+ state.record_rate_limited(provider="anthropic", source="bogus")++ requests = state.snapshot(persistence={"enabled": True, "healthy": True})["requests"]+ assert requests["rate_limited_by_source"] == {"headroom": 2}+++def test_pre_3696_state_loads_without_backfilling_invented_history() -> None:+ """A state file written before the split keeps its totals and starts the maps empty.++ Back-filling the legacy ``rate_limited`` total into one bucket would claim we+ observed a split we never recorded.+ """+ state = PersistentMetricsState(+ {"requests": {"total": 10, "failed": 4, "rate_limited": 7}},+ now=lambda: FIXED_NOW,+ )++ requests = state.snapshot(persistence={"enabled": True, "healthy": True})["requests"]++ assert requests["rate_limited"] == 7+ assert requests["failed"] == 4+ assert requests["rate_limited_by_source"] == {}+ assert requests["failed_by_provider"] == {}diff --git a/tests/test_persistent_metrics_integration.py b/tests/test_persistent_metrics_integration.pyindex fdb9f2406..3187f9143 100644--- a/tests/test_persistent_metrics_integration.py+++ b/tests/test_persistent_metrics_integration.py@@ -39,6 +39,10 @@ def test_runtime_metric_events_feed_lifetime_without_resetting_runtime_counters(assert lifetime["requests"]["cached"] == 1assert lifetime["requests"]["failed"] == 1assert lifetime["requests"]["rate_limited"] == 1+ # The Prometheus labels must not stop at Prometheus: lifetime carries the+ # same two splits (issue #3696). No explicit source here, so it is ours.+ assert lifetime["requests"]["failed_by_provider"] == {"anthropic": 1}+ assert lifetime["requests"]["rate_limited_by_source"] == {"headroom": 1}assert lifetime["requests"]["by_provider"] == {"anthropic": 1}assert lifetime["requests"]["by_stack"] == {"codex": 1}assert lifetime["tokens"]["output"] == 3@@ -47,3 +51,27 @@ def test_runtime_metric_events_feed_lifetime_without_resetting_runtime_counters(assert lifetime["prefix_cache"]["misses_by_reason"] == {"prefix_change": 1}assert lifetime["waste_signals"] == {"repetition": 4}assert metrics.requests_total == 1+++def test_upstream_rate_limit_reaches_lifetime_labelled_upstream(tmp_path) -> None:+ """An upstream 429 must not be filed under Headroom's own limiter in lifetime.++ The outcome funnel passes ``source="upstream"``; the whole chain+ (PrometheusMetrics -> SavingsTracker -> PersistentMetricsState) has to carry+ it, or the durable view re-merges what Prometheus splits (issue #3696).+ """+ tracker = SavingsTracker(path=str(tmp_path / "proxy_savings.json"), save_flush_every=25)+ metrics = PrometheusMetrics(savings_tracker=tracker)++ asyncio.run(metrics.record_rate_limited(provider="anthropic", source="upstream"))+ asyncio.run(metrics.record_rate_limited(provider="anthropic", source="headroom"))+ asyncio.run(metrics.record_failed(provider="openai"))++ lifetime = tracker.lifetime_response()++ assert lifetime["requests"]["rate_limited"] == 2+ assert lifetime["requests"]["rate_limited_by_source"] == {"upstream": 1, "headroom": 1}+ assert lifetime["requests"]["failed_by_provider"] == {"openai": 1}+ # Neither counter may touch the completed-request denominator.+ assert lifetime["requests"]["total"] == 0+ assert metrics.requests_total == 0diff --git a/tests/test_prometheus_obs_counters.py b/tests/test_prometheus_obs_counters.pyindex 41c4a7683..413f175bc 100644--- a/tests/test_prometheus_obs_counters.py+++ b/tests/test_prometheus_obs_counters.py@@ -176,3 +176,130 @@ async def test_gate_counter_is_thread_safe_under_concurrent_export() -> None:assert not errors, f"export() raced the writer: {errors[:3]}"totals = dict(metrics.kompress_size_gate_by_outcome)assert sum(totals.values()) == n_threads * per_thread+++# ---------------------------------------------------------------------------+# headroom_requests_rate_limited_total{source} / headroom_requests_failed_total{provider}+#+# Issue #3696: #3615 routed upstream 429s and 4xx into these two counters, which+# silently changed what they mean. The labels make the new meaning queryable —+# "my limiter throttled me" vs "the provider throttled me" are acted on+# differently, and a failure with no provider attached names no culprit.+# ---------------------------------------------------------------------------+++async def test_rate_limited_export_splits_headroom_from_upstream() -> None:+ metrics = PrometheusMetrics(stateless=True)++ # What the four handler sites do when Headroom's OWN limiter rejects.+ await metrics.record_rate_limited(provider="anthropic", source="headroom")+ # What the outcome funnel does with a provider 429.+ await metrics.record_rate_limited(provider="anthropic", source="upstream")+ await metrics.record_rate_limited(provider="openai", source="upstream")++ text = await metrics.export()++ assert "# TYPE headroom_requests_rate_limited_total counter" in text+ assert 'headroom_requests_rate_limited_total{source="headroom"} 1' in text+ assert 'headroom_requests_rate_limited_total{source="upstream"} 2' in text+ # `sum without (source)` must reproduce the pre-label value.+ assert metrics.requests_rate_limited == 3+ # No unlabelled sample alongside the labelled ones — that would double-count.+ assert "\nheadroom_requests_rate_limited_total 3" not in text+++async def test_rate_limited_source_series_exist_before_any_429() -> None:+ """Both series export from startup, so a rate() never appears mid-incident."""+ metrics = PrometheusMetrics(stateless=True)++ text = await metrics.export()++ assert 'headroom_requests_rate_limited_total{source="headroom"} 0' in text+ assert 'headroom_requests_rate_limited_total{source="upstream"} 0' in text+++async def test_rate_limited_source_defaults_to_headroom_and_clamps_unknown() -> None:+ metrics = PrometheusMetrics(stateless=True)++ await metrics.record_rate_limited(provider="anthropic")+ await metrics.record_rate_limited(provider="anthropic", source="bogus")++ text = await metrics.export()++ assert 'headroom_requests_rate_limited_total{source="headroom"} 2' in text+ assert 'headroom_requests_rate_limited_total{source="upstream"} 0' in text+ assert "bogus" not in text+++async def test_failed_export_is_attributed_to_the_provider_that_failed() -> None:+ metrics = PrometheusMetrics(stateless=True)++ await metrics.record_failed(provider="anthropic")+ await metrics.record_failed(provider="openai")+ await metrics.record_failed(provider="openai")++ text = await metrics.export()++ assert "# TYPE headroom_requests_failed_total counter" in text+ assert 'headroom_requests_failed_total{provider="anthropic"} 1' in text+ assert 'headroom_requests_failed_total{provider="openai"} 2' in text+ # `sum without (provider)` must reproduce the pre-label value.+ assert metrics.requests_failed == 3+ assert "\nheadroom_requests_failed_total 3" not in text+++async def test_failed_without_a_provider_lands_in_unknown() -> None:+ metrics = PrometheusMetrics(stateless=True)++ await metrics.record_failed()++ text = await metrics.export()++ assert 'headroom_requests_failed_total{provider="unknown"} 1' in text+++async def test_reset_runtime_clears_rate_limit_and_failure_splits() -> None:+ metrics = PrometheusMetrics(stateless=True)++ await metrics.record_rate_limited(provider="anthropic", source="upstream")+ await metrics.record_failed(provider="anthropic")++ await metrics.reset_runtime()++ assert metrics.requests_rate_limited == 0+ assert metrics.requests_failed == 0+ assert metrics.requests_rate_limited_by_source == {"headroom": 0, "upstream": 0}+ # Re-seeded, not emptied: the metric must keep exporting a sample after a+ # reset, same as the rate-limit source series above.+ assert dict(metrics.requests_failed_by_provider) == {"unknown": 0}+++async def test_failed_total_exports_a_sample_before_any_failure() -> None:+ """A healthy proxy must still export ``headroom_requests_failed_total``.++ Before the provider label this counter was always present as a bare+ ``headroom_requests_failed_total 0``. A labelled map that starts empty+ emits no sample at all, which makes the documented failure-rate query+ return an empty vector (an empty numerator empties the whole expression),+ so the panel reads "No data" instead of 0% while everything is fine --+ and `absent()` alerts fire on healthy proxies.+ """+ metrics = PrometheusMetrics(stateless=True)++ text = await metrics.export()++ assert 'headroom_requests_failed_total{provider="unknown"} 0' in text++ # The seed must not survive as a duplicate once a real provider fails.+ await metrics.record_failed(provider="anthropic")+ text = await metrics.export()+ assert 'headroom_requests_failed_total{provider="anthropic"} 1' in text+ assert 'headroom_requests_failed_total{provider="unknown"} 0' in text+ assert metrics.requests_failed == 1diff --git a/wiki/metrics.md b/wiki/metrics.mdindex 5a5c10e97..da1af4221 100644--- a/wiki/metrics.md+++ b/wiki/metrics.md@@ -29,7 +29,9 @@ curl http://localhost:8787/stats"total": 42,"cached": 5,"rate_limited": 0,- "failed": 0+ "rate_limited_by_source": { "headroom": 0, "upstream": 0 },+ "failed": 0,+ "failed_by_provider": {}},"tokens": {"input": 50000,@@ -48,6 +50,14 @@ curl http://localhost:8787/stats}```+`requests.total` is the **completed**-request count, not total traffic: a request that+ended in a 4xx, 5xx or 429 is counted under `requests.failed` or `requests.rate_limited`+instead and never reaches `total`. `rate_limited_by_source` splits 429s into+`headroom` (Headroom's own rate limiter refused the request) and `upstream` (the provider+refused it) — different problems, different fixes. `failed_by_provider` attributes failures+to the upstream that produced them. The unlabelled `rate_limited` / `failed` totals are+unchanged.+`/stats` keeps the existing live/session fields, including `savings_history`,for backward compatibility. The new `persistent_savings` block is durable localproxy compression history stored by default at