Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

fix(metrics): split rate-limited by source and label failures by provider (#3707)

Tejas Chopra5ff4ea1ef948
diff --git a/docs/content/docs/metrics.mdx b/docs/content/docs/metrics.mdx
index d7dacba53..b9ce54a65 100644
--- a/docs/content/docs/metrics.mdx
+++ b/docs/content/docs/metrics.mdx
@@ -25,7 +25,9 @@ curl http://localhost:8787/stats
"total": 42,
"cached": 5,
"rate_limited": 0,
- "failed": 0
+ "rate_limited_by_source": { "headroom": 0, "upstream": 0 },
+ "failed": 0,
+ "failed_by_provider": {}
},
"tokens": {
"input": 50000,
@@ -44,6 +46,14 @@ curl http://localhost:8787/stats
}
```
+`requests.total` is the **completed**-request count, not total traffic: a request that
+ended in a 4xx, 5xx or 429 is counted under `requests.failed` or `requests.rate_limited`
+instead and never reaches `total`. `rate_limited_by_source` splits 429s into
+`headroom` (Headroom's own rate limiter refused the request) and `upstream` (the provider
+refused it) — different problems, different fixes. `failed_by_provider` attributes failures
+to the upstream that produced them. The unlabelled `rate_limited` / `failed` totals are
+unchanged.
+
Persistent savings are stored at `~/.headroom/proxy_savings.json` and survive proxy restarts. Override the path with `HEADROOM_SAVINGS_PATH`.
### Historical Savings
@@ -82,8 +92,27 @@ headroom_latency_ms_count 1234
# HELP headroom_provider_cache_hit_requests_total Requests that read from the provider's prompt cache
headroom_provider_cache_hit_requests_total{provider="anthropic"} 456
+
+# HELP headroom_requests_failed_total Requests that failed upstream (4xx and 5xx, excluding 429), by provider
+headroom_requests_failed_total{provider="anthropic"} 3
+
+# HELP headroom_requests_rate_limited_total Requests rejected with 429, by who rejected them (headroom=our own limiter, upstream=the provider)
+headroom_requests_rate_limited_total{source="headroom"} 0
+headroom_requests_rate_limited_total{source="upstream"} 12
```
+<Callout type="warn" title="requests_total is completed requests, not all traffic">
+`headroom_requests_total` counts requests that **completed**. A request that ended in a 4xx,
+5xx or 429 lands in `headroom_requests_failed_total` / `headroom_requests_rate_limited_total`
+and is not in `requests_total` — so a failure rate must add `failed` back into its own
+denominator. `headroom_inbound_requests_total` is the all-traffic counter.
+
+`headroom_requests_failed_total` gained a `provider` label and
+`headroom_requests_rate_limited_total` a `source` label, so unaggregated queries on them now
+return several series. `sum without (provider) (...)` and `sum without (source) (...)`
+reproduce the previous single-series behaviour.
+</Callout>
+
<Callout type="warn" title="No percentile buckets on /metrics">
There are no Prometheus histogram buckets for compression ratio or latency — build percentiles or averages from `_sum` / `_count` pairs (`headroom_latency_ms_sum`, `headroom_latency_ms_count`, and so on), or use the `headroom perf` CLI for real p95/p99.
</Callout>
@@ -351,7 +380,8 @@ Env: `HEADROOM_BUDGET_ESTIMATED_BASIS`. `headroom doctor` reports the estimated
| `headroom_persistent_savings_tokens_saved_total` | Durable lifetime tokens saved from `/stats.persistent_savings` | Higher is better |
| `headroom_overhead_ms_sum` / `headroom_overhead_ms_count` | Mean latency Headroom itself adds (there are no percentile buckets — see the callout above) | Low is better |
| `headroom_provider_cache_hit_requests_total` / `headroom_provider_cache_requests_total` | Cache effectiveness, by provider | >20% is good |
-| `headroom_requests_failed_total` | Reliability (upstream 5xx errors) | 0 |
+| `headroom_requests_failed_total{provider}` | Reliability — upstream **4xx and 5xx** (429s are counted separately), attributed to the provider that failed | 0 |
+| `headroom_requests_rate_limited_total{source}` | Throttling — `source="headroom"` is Headroom's own rate limiter, `source="upstream"` is the provider's | 0 |
## Grafana Dashboard
diff --git a/docs/metrics-technical-guide.md b/docs/metrics-technical-guide.md
index 63dbe691e..b5ff7e152 100644
--- a/docs/metrics-technical-guide.md
+++ b/docs/metrics-technical-guide.md
@@ -103,21 +103,55 @@ rate(headroom_cache_bust_total[5m])
| Metric | What it shows |
|---|---|
-| `headroom_requests_total` | Requests handled. Unlabelled. |
-| `headroom_requests_by_provider{provider}` | Traffic split by provider — `anthropic`, `openai`, `gemini`, `bedrock`… |
-| `headroom_requests_by_model{model}` | Traffic split by model. Capped at 1024 distinct; overflow lands in `model="other"`. |
-| `headroom_requests_failed_total` | Upstream 5xx errors. |
-| `headroom_requests_rate_limited_total` | Requests **Headroom** rejected via its own rate limiter (not upstream 429s). |
+| `headroom_requests_total` | **Completed** requests — the denominator for success-side stats, not total traffic. A request that ended in a 4xx, 5xx or 429 never reaches it. Unlabelled. |
+| `headroom_inbound_requests_total` | Every inbound HTTP request the proxy accepted, whatever the outcome. This is the all-traffic counter. |
+| `headroom_requests_by_provider{provider}` | Traffic split by provider — `anthropic`, `openai`, `gemini`, `bedrock`… **Completed requests only**, same denominator as `headroom_requests_total`. Use `headroom_requests_failed_total{provider}` for the failure side. |
+| `headroom_requests_by_model{model}` | Traffic split by model. Capped at 1024 distinct; overflow lands in `model="other"`. Completed requests only. |
+| `headroom_requests_failed_total{provider}` | Requests that failed upstream — **4xx and 5xx**, excluding 429. Excluded from `headroom_requests_total`, so add it back when you compute a rate. |
+| `headroom_requests_rate_limited_total{source}` | Requests rejected with 429, by **either** Headroom's own rate limiter (`source="headroom"`) or the upstream provider (`source="upstream"`). Excluded from `headroom_requests_total`. |
| `headroom_compression_failed_total{reason}` | Compression failures — `timeout` or `error`. Fails open, so traffic keeps flowing but savings quietly stop. **Worth an alert.** |
| `headroom_compression_quarantine_total{event}` | Compression disabled after repeated timeouts — `activated`, `skipped`, `released`. |
| `headroom_inbound_requests_active` | In-flight requests, gauge. Counts all HTTP including `/metrics`. |
| `headroom_active_ws_sessions` | Live Codex WebSocket sessions, gauge. |
+### `headroom_requests_total` is a denominator, not traffic
+
+`headroom_requests_total` counts requests that **completed** — it has always been
+incremented only on the success path, and since the outcome funnel started short-circuiting
+at `>= 400`, every 4xx, 5xx and 429 drops out of it. So:
+
+- `headroom_requests_total` + `headroom_requests_failed_total` + `headroom_requests_rate_limited_total`
+ ≈ the requests the proxy forwarded upstream.
+- `headroom_inbound_requests_total` is the honest all-traffic counter (it also counts
+ `/metrics`, `/stats` and other non-proxy HTTP).
+
+That is why the failure-rate query below adds `failed` back into its own denominator, and
+why you should not "simplify" it to `… / rate(headroom_requests_total[5m])`: failures are
+not in `requests_total`, so that form divides failures by successes and over-reports the
+rate — badly, during exactly the incident you wrote it for (at 100% failures the denominator
+goes to zero).
+
```promql
-# Failure rate
+# Failure rate. The denominator deliberately re-adds `failed`: a failed request
+# is NOT in headroom_requests_total, so requests_total alone is successes-only.
rate(headroom_requests_failed_total[5m])
/ clamp_min(rate(headroom_requests_total[5m]) + rate(headroom_requests_failed_total[5m]), 1)
+# Failure rate by provider — which upstream is actually broken.
+sum by (provider) (rate(headroom_requests_failed_total[5m]))
+ / clamp_min(
+ sum by (provider) (rate(headroom_requests_by_provider[5m]))
+ + sum by (provider) (rate(headroom_requests_failed_total[5m])),
+ 1
+ )
+
+# Who is throttling you: your own limiter (raise the cap) vs the provider
+# (back off, shard keys). Alert on these separately — they are different actions.
+sum by (source) (rate(headroom_requests_rate_limited_total[5m]))
+
+# Provider throttling only — the one that means "slow down".
+rate(headroom_requests_rate_limited_total{source="upstream"}[5m])
+
# Savings silently stopped
sum by (reason) (rate(headroom_compression_failed_total[5m]))
@@ -125,6 +159,22 @@ sum by (reason) (rate(headroom_compression_failed_total[5m]))
sum by (provider) (rate(headroom_requests_by_provider[5m]))
```
+> **Migration — these two counters gained labels.** `headroom_requests_failed_total` is now
+> labelled by `provider` and `headroom_requests_rate_limited_total` by `source`, so an
+> unaggregated query that used to return one series now returns several, and a panel or
+> alert that graphed the bare counter will fan out into one line per label value.
+> `sum without (provider) (rate(headroom_requests_failed_total[5m]))` and
+> `sum without (source) (rate(headroom_requests_rate_limited_total[5m]))` reproduce the old
+> single-series behaviour exactly. Both `source` series are exported from process start
+> (including at zero); `failed` exports one series per provider that has actually failed, so
+> it has no series at all until the first failure.
+>
+> The same split is available outside Prometheus: `/stats` carries
+> `requests.rate_limited_by_source` and `requests.failed_by_provider` alongside the
+> unlabelled `requests.rate_limited` / `requests.failed` totals (which are unchanged), the
+> lifetime aggregate carries the same two maps, and the OTel counters
+> `headroom.proxy.requests.rate_limited` / `.failed` carry `source` / `provider` attributes.
+
---
## Anthropic subscription panel
diff --git a/headroom/dashboard/templates/dashboard.html b/headroom/dashboard/templates/dashboard.html
index 15661140c..f796393b6 100644
--- a/headroom/dashboard/templates/dashboard.html
+++ b/headroom/dashboard/templates/dashboard.html
@@ -210,7 +210,7 @@
<div class="grid grid-cols-1 gap-4 mb-6 lg:grid-cols-2">
<section class="bg-surface rounded-lg p-4 border border-border">
<h3 class="mb-3 text-sm font-medium text-gray-300">Request Health</h3>
- <div class="grid grid-cols-2 gap-3 text-sm"><div>Completed <span class="float-right tabular-nums" x-text="formatNumber(stats.requests?.total || 0)"></span></div><div>Failed <span class="float-right tabular-nums text-red-400" x-text="formatNumber(stats.requests?.failed || 0)"></span></div><div>Rate Limited <span class="float-right tabular-nums text-amber-400" x-text="formatNumber(stats.requests?.rate_limited || 0)"></span></div><div>Cached <span class="float-right tabular-nums text-cyan-400" x-text="formatNumber(stats.requests?.cached || 0)"></span></div></div>
+ <div class="grid grid-cols-2 gap-3 text-sm"><div>Completed <span class="float-right tabular-nums" x-text="formatNumber(stats.requests?.total || 0)"></span></div><div title="Requests that failed upstream — 4xx and 5xx. 429s are counted under Rate Limited instead, and neither is included in Completed.">Failed <span class="float-right tabular-nums text-red-400" x-text="formatNumber(stats.requests?.failed || 0)"></span></div><div title="Requests rejected with 429 — by Headroom&#39;s own rate limiter or by the upstream provider. /stats splits them under requests.rate_limited_by_source.">Rate Limited <span class="float-right tabular-nums text-amber-400" x-text="formatNumber(stats.requests?.rate_limited || 0)"></span></div><div>Cached <span class="float-right tabular-nums text-cyan-400" x-text="formatNumber(stats.requests?.cached || 0)"></span></div></div>
</section>
<section class="bg-surface rounded-lg p-4 border border-border">
<h3 class="mb-3 text-sm font-medium text-gray-300">Live Activity</h3>
@@ -418,7 +418,7 @@
<span class="font-mono text-sm" x-text="((stats.ttfb?.min_ms || 0) / 1000).toFixed(2) + ' - ' + ((stats.ttfb?.max_ms || 0) / 1000).toFixed(2) + 's'"></span>
</div>
<div class="flex justify-between items-center">
- <span class="text-sm text-gray-400">Failed Requests</span>
+ <span class="text-sm text-gray-400" title="Requests that failed upstream — 4xx and 5xx. 429s are counted under Rate Limited instead, and neither is included in Completed.">Failed Requests</span>
<span class="font-mono text-sm" x-text="stats.requests?.failed || 0"></span>
</div>
<!-- Per-transform timing breakdown -->
@@ -1470,8 +1470,8 @@
</div>
<div class="grid grid-cols-2 gap-4 mb-6 md:grid-cols-4">
<div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500">Requests</div><div class="text-2xl tabular-nums" x-text="formatNumber(lifetimeStats.requests?.total || 0)"></div></div>
- <div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500">Failed</div><div class="text-2xl tabular-nums text-red-400" x-text="formatNumber(lifetimeStats.requests?.failed || 0)"></div></div>
- <div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500">Rate Limited</div><div class="text-2xl tabular-nums text-amber-400" x-text="formatNumber(lifetimeStats.requests?.rate_limited || 0)"></div></div>
+ <div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500" title="Requests that failed upstream — 4xx and 5xx. 429s are counted under Rate Limited instead, and neither is included in Completed.">Failed</div><div class="text-2xl tabular-nums text-red-400" x-text="formatNumber(lifetimeStats.requests?.failed || 0)"></div></div>
+ <div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500" title="Requests rejected with 429 — by Headroom&#39;s own rate limiter or by the upstream provider. /stats splits them under requests.rate_limited_by_source.">Rate Limited</div><div class="text-2xl tabular-nums text-amber-400" x-text="formatNumber(lifetimeStats.requests?.rate_limited || 0)"></div></div>
<div class="bg-surface rounded-lg p-4 border border-border"><div class="text-xs text-gray-500">Cached</div><div class="text-2xl tabular-nums text-cyan-400" x-text="formatNumber(lifetimeStats.requests?.cached || 0)"></div></div>
</div>
diff --git a/headroom/observability/metrics.py b/headroom/observability/metrics.py
index 93bdd93ef..430c690a3 100644
--- a/headroom/observability/metrics.py
+++ b/headroom/observability/metrics.py
@@ -207,12 +207,15 @@ class HeadroomOtelMetrics:
)
self._proxy_failed_requests = self._meter.create_counter(
"headroom.proxy.requests.failed",
- description="Proxy requests that failed.",
+ description="Proxy requests that failed upstream (4xx and 5xx, excluding 429).",
unit="1",
)
self._proxy_rate_limited_requests = self._meter.create_counter(
"headroom.proxy.requests.rate_limited",
- description="Proxy requests rejected by rate limiting.",
+ description=(
+ "Proxy requests rejected with 429, by source "
+ "(headroom=our own limiter, upstream=the provider)."
+ ),
unit="1",
)
self._proxy_input_tokens = self._meter.create_counter(
@@ -567,8 +570,15 @@ class HeadroomOtelMetrics:
*,
provider: str | None = None,
model: str | None = None,
+ source: str = "headroom",
) -> None:
- self._proxy_rate_limited_requests.add(1, self._attrs(provider=provider, model=model))
+ # ``source`` mirrors the headroom_requests_rate_limited_total{source}
+ # Prometheus label: "headroom" = our own limiter refused the request,
+ # "upstream" = the provider did. Same split in both backends so an
+ # operator's query means the same thing whichever one they scrape.
+ self._proxy_rate_limited_requests.add(
+ 1, self._attrs(provider=provider, model=model, source=source)
+ )
def record_proxy_cache_bust(self, *, tokens_lost: int) -> None:
self._proxy_cache_busts.add(1)
diff --git a/headroom/proxy/handlers/anthropic.py b/headroom/proxy/handlers/anthropic.py
index 66c074299..cf321e3fc 100644
--- a/headroom/proxy/handlers/anthropic.py
+++ b/headroom/proxy/handlers/anthropic.py
@@ -1253,7 +1253,9 @@ class AnthropicHandlerMixin:
rate_key = f"{api_key[:16]}:{client_ip}" if api_key else client_ip
allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)
if not allowed:
- await self.metrics.record_rate_limited(provider=provider_name)
+ await self.metrics.record_rate_limited(
+ provider=provider_name, source="headroom"
+ )
# Unit 4: release the pre-upstream semaphore before we
# bail out of the handler via HTTPException — FastAPI's
# exception handler will NOT run our ``finally``.
diff --git a/headroom/proxy/handlers/gemini.py b/headroom/proxy/handlers/gemini.py
index 3f82145be..0d8e1bc36 100644
--- a/headroom/proxy/handlers/gemini.py
+++ b/headroom/proxy/handlers/gemini.py
@@ -402,7 +402,7 @@ class GeminiHandlerMixin:
rate_key = headers.get("x-goog-api-key", "default")[:20]
allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)
if not allowed:
- await self.metrics.record_rate_limited(provider=provider_name)
+ await self.metrics.record_rate_limited(provider=provider_name, source="headroom")
raise HTTPException(
status_code=429,
detail=f"Rate limited. Retry after {wait_seconds:.1f}s",
diff --git a/headroom/proxy/handlers/openai.py b/headroom/proxy/handlers/openai.py
index 41406ef5d..910be1d41 100644
--- a/headroom/proxy/handlers/openai.py
+++ b/headroom/proxy/handlers/openai.py
@@ -3551,7 +3551,9 @@ class OpenAIHandlerMixin:
rate_key = headers.get("authorization", "default")[:20]
allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)
if not allowed:
- await self.metrics.record_rate_limited(provider=openai_chat_outcome_provider)
+ await self.metrics.record_rate_limited(
+ provider=openai_chat_outcome_provider, source="headroom"
+ )
raise HTTPException(
status_code=429,
detail=f"Rate limited. Retry after {wait_seconds:.1f}s",
@@ -5676,7 +5678,7 @@ class OpenAIHandlerMixin:
rate_key = headers.get("authorization", "default")[:20]
allowed, wait_seconds = await self.rate_limiter.check_request(rate_key)
if not allowed:
- await self.metrics.record_rate_limited(provider="openai")
+ await self.metrics.record_rate_limited(provider="openai", source="headroom")
raise HTTPException(
status_code=429,
detail=f"Rate limited. Retry after {wait_seconds:.1f}s",
diff --git a/headroom/proxy/outcome.py b/headroom/proxy/outcome.py
index 31c443fbd..cd85992e9 100644
--- a/headroom/proxy/outcome.py
+++ b/headroom/proxy/outcome.py
@@ -531,9 +531,16 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
# limit is the one 4xx a user is expected to act on (back off, raise a cap),
# and folding it into a generic failure count hides exactly that. Both
# counters are already exported and neither feeds savings.
+ #
+ # source="upstream": this funnel only ever sees a 429 the PROVIDER returned.
+ # Headroom's own limiter rejects before a request is ever sent and records
+ # source="headroom" from the handler. The two are acted on differently —
+ # raise our cap vs. back off / shard keys — so they must stay separable
+ # (issue #3696). ``outcome.provider`` is the already-Copilot-relabelled
+ # value, matching every other provider-labelled metric on this path.
if outcome.status_code >= 400:
if outcome.status_code == 429:
- await handler.metrics.record_rate_limited(provider=outcome.provider)
+ await handler.metrics.record_rate_limited(provider=outcome.provider, source="upstream")
else:
await handler.metrics.record_failed(provider=outcome.provider)
return
diff --git a/headroom/proxy/persistent_metrics.py b/headroom/proxy/persistent_metrics.py
index 9804d74a5..d7a5a6c13 100644
--- a/headroom/proxy/persistent_metrics.py
+++ b/headroom/proxy/persistent_metrics.py
@@ -16,6 +16,9 @@ MAX_EXPOSED_MODELS = 100
MAX_LABEL_LENGTH = 128
KNOWN_MISS_REASONS = frozenset({"ttl_expiry", "prefix_change", "unknown"})
+# Mirrors RATE_LIMIT_SOURCES in prometheus_metrics: "headroom" is our own
+# limiter refusing the request, "upstream" is the provider refusing it.
+RATE_LIMIT_SOURCES = frozenset({"headroom", "upstream"})
# Names must match ``WasteSignals.to_dict()`` in headroom/config.py — the
# parser emits ``json_bloat``; an allowlist that says ``json_noise`` silently
# shoves the largest waste category into the catch-all bucket.
@@ -91,6 +94,12 @@ def _empty_state() -> dict[str, Any]:
"cached": 0,
"failed": 0,
"rate_limited": 0,
+ # Who produced the failure / who refused the request. The totals
+ # above stay the unlabelled figures the dashboard reads; these
+ # split them the same way the Prometheus labels do, so no backend
+ # reports a differently-shaped counter (issue #3696).
+ "failed_by_provider": {},
+ "rate_limited_by_source": {},
"by_provider": {},
"by_stack": {},
},
@@ -182,6 +191,17 @@ class PersistentMetricsState:
result["requests"]["by_provider"] = self._normalize_count_map(
raw_requests.get("by_provider"), MAX_PROVIDER_VALUES
)
+ result["requests"]["failed_by_provider"] = self._normalize_count_map(
+ raw_requests.get("failed_by_provider"), MAX_PROVIDER_VALUES
+ )
+ # A state file written before #3696 has no split, and back-filling the
+ # legacy total into one bucket would invent history we never observed —
+ # so it loads empty and only new events populate it. The unlabelled
+ # ``rate_limited`` total is untouched and stays comparable across the
+ # upgrade.
+ result["requests"]["rate_limited_by_source"] = self._normalize_enum_map(
+ raw_requests.get("rate_limited_by_source"), RATE_LIMIT_SOURCES, fallback="headroom"
+ )
result["requests"]["by_stack"] = self._normalize_count_map(
raw_requests.get("by_stack"), MAX_STACK_VALUES
)
@@ -474,12 +494,29 @@ class PersistentMetricsState:
self._record_activity()
self._state["requests"]["failed"] += 1
+ self._increment_count(
+ self._state["requests"]["failed_by_provider"], _label(provider), MAX_PROVIDER_VALUES
+ )
+
+ def record_rate_limited(
+ self,
+ *,
+ provider: str | None = None,
+ model: str | None = None,
+ source: str = "headroom",
+ ) -> None:
+ """Record a rate-limited request without redefining total request semantics.
- def record_rate_limited(self, *, provider: str | None = None, model: str | None = None) -> None:
- """Record a rate-limited request without redefining total request semantics."""
+ ``source`` is ``headroom`` (our own limiter) or ``upstream`` (the
+ provider's 429); anything else is clamped to ``headroom``, which is what
+ this counter meant before upstream 429s started reaching it.
+ """
self._record_activity()
self._state["requests"]["rate_limited"] += 1
+ bucket = source if source in RATE_LIMIT_SOURCES else "headroom"
+ by_source = self._state["requests"]["rate_limited_by_source"]
+ by_source[bucket] = by_source.get(bucket, 0) + 1
def record_cache_bust(self, *, tokens_lost: Any = 0) -> None:
self._record_activity()
diff --git a/headroom/proxy/prometheus_metrics.py b/headroom/proxy/prometheus_metrics.py
index b0252f008..e62759f24 100644
--- a/headroom/proxy/prometheus_metrics.py
+++ b/headroom/proxy/prometheus_metrics.py
@@ -30,6 +30,33 @@ logger = logging.getLogger("headroom.proxy")
# client-supplied model cardinality stays bounded (see record_request).
_OTHER_MODEL = "other"
+# Closed label set for headroom_requests_rate_limited_total{source}. Two 429s
+# mean opposite things to an operator: "headroom" is OUR limiter refusing the
+# request (raise the cap), "upstream" is the provider refusing it (back off or
+# shard keys). Before #3615 only the first could ever increment this counter;
+# the outcome funnel now routes provider 429s here too, so the split has to be
+# queryable instead of silently merged. Closed set => bounded cardinality, and
+# both series are exported from startup so a rate() never goes from absent to
+# present mid-incident.
+RATE_LIMIT_SOURCE_HEADROOM = "headroom"
+RATE_LIMIT_SOURCE_UPSTREAM = "upstream"
+RATE_LIMIT_SOURCES = (RATE_LIMIT_SOURCE_HEADROOM, RATE_LIMIT_SOURCE_UPSTREAM)
+
+# Bucket for a failure with no attributed provider. Also seeded at zero so
+# headroom_requests_failed_total always exports at least one sample.
+_PROVIDER_UNKNOWN = "unknown"
+
+
+def _rate_limit_source(source: str | None) -> str:
+ """Clamp ``source`` to the closed label set, defaulting to Headroom's limiter.
+
+ Defaults to ``headroom`` because that is what this counter meant for its
+ whole life before the outcome funnel started feeding it upstream 429s: an
+ un-updated caller keeps the historical reading rather than inventing a new
+ label value.
+ """
+ return source if source in RATE_LIMIT_SOURCES else RATE_LIMIT_SOURCE_HEADROOM
+
def _escape_label_value(value: str) -> str:
# The /metrics body is emitted whole with .encode("utf-8") (server.py). A
@@ -100,7 +127,26 @@ class PrometheusMetrics:
self.requests_by_stack: dict[str, int] = defaultdict(int)
self.requests_cached = 0
self.requests_rate_limited = 0
+ # Seeded with both sources at zero so /metrics exports the full series
+ # set from the first scrape; ``requests_rate_limited`` stays the
+ # unlabelled total that /stats and the session summary read.
+ self.requests_rate_limited_by_source: dict[str, int] = dict.fromkeys(RATE_LIMIT_SOURCES, 0)
self.requests_failed = 0
+ # Per-provider failure attribution. #3615 moved 4xx out of the success
+ # funnel, which also took them out of ``requests_by_provider`` (only
+ # record_request touches that), leaving Prometheus with no way to tell
+ # which upstream was failing. Kept alongside the unlabelled total.
+ #
+ # Seeded with the "unknown" bucket at zero so the metric always exports
+ # at least one sample. Before this counter carried a label it was always
+ # present as a bare ``headroom_requests_failed_total 0``; a labelled map
+ # that starts empty would emit NO sample on a healthy proxy, which turns
+ # the documented failure-rate query into an empty vector (an empty
+ # numerator makes the whole expression empty, so the panel reads
+ # "No data" instead of 0%) and would make ``absent()`` alerts fire on
+ # healthy proxies. "unknown" is a real bucket -- record_failed uses it
+ # when no provider is attributed -- so seeding it invents no provider.
+ self.requests_failed_by_provider: dict[str, int] = defaultdict(int, {_PROVIDER_UNKNOWN: 0})
self.inbound_requests_total = 0
self.inbound_requests_completed = 0
self.inbound_requests_active = 0
@@ -343,7 +389,11 @@ class PrometheusMetrics:
self.requests_by_stack.clear()
self.requests_cached = 0
self.requests_rate_limited = 0
+ self.requests_rate_limited_by_source = dict.fromkeys(RATE_LIMIT_SOURCES, 0)
self.requests_failed = 0
+ self.requests_failed_by_provider.clear()
+ # Re-seed so /metrics keeps exporting a sample after a reset.
+ self.requests_failed_by_provider[_PROVIDER_UNKNOWN] = 0
self.inbound_requests_total = 0
self.inbound_requests_completed = 0
self.inbound_requests_active = 0
@@ -1175,15 +1225,38 @@ class PrometheusMetrics:
if ms_val > self.ws_session_duration_max_ms[cause]:
self.ws_session_duration_max_ms[cause] = ms_val
- async def record_rate_limited(self, *, provider: str | None = None, model: str | None = None):
+ async def record_rate_limited(
+ self,
+ *,
+ provider: str | None = None,
+ model: str | None = None,
+ source: str = RATE_LIMIT_SOURCE_HEADROOM,
+ ):
+ """Record a 429.
+
+ ``source`` says WHO refused the request: ``headroom`` for our own
+ limiter (the handlers, which raise HTTPException and never emit an
+ outcome) or ``upstream`` for a provider 429 arriving through the
+ outcome funnel. Threaded to every sink — Prometheus, lifetime/persistent
+ and OTel — so no backend reports a differently-shaped counter.
+ """
+ source = _rate_limit_source(source)
async with self._lock:
self.requests_rate_limited += 1
- self.savings_tracker.record_lifetime_rate_limited(provider=provider, model=model)
- self._get_otel_metrics().record_proxy_rate_limited(provider=provider, model=model)
+ self.requests_rate_limited_by_source[source] = (
+ self.requests_rate_limited_by_source.get(source, 0) + 1
+ )
+ self.savings_tracker.record_lifetime_rate_limited(
+ provider=provider, model=model, source=source
+ )
+ self._get_otel_metrics().record_proxy_rate_limited(
+ provider=provider, model=model, source=source
+ )
async def record_failed(self, *, provider: str | None = None, model: str | None = None):
async with self._lock:
self.requests_failed += 1
+ self.requests_failed_by_provider[provider or _PROVIDER_UNKNOWN] += 1
self.savings_tracker.record_lifetime_failed(provider=provider, model=model)
self._get_otel_metrics().record_proxy_failed(provider=provider, model=model)
@@ -1215,20 +1288,39 @@ class PrometheusMetrics:
help_text="Cached request count",
value=self.requests_cached,
)
- _append_metric(
- lines,
- name="headroom_requests_rate_limited_total",
- metric_type="counter",
- help_text="Rate limited requests",
- value=self.requests_rate_limited,
+ # Labelled series only — an unlabelled sample alongside these would
+ # double-count under sum(). `sum without (source)` reproduces the
+ # pre-label value exactly.
+ lines.extend(
+ [
+ "# HELP headroom_requests_rate_limited_total Requests rejected with 429, "
+ "by who rejected them (headroom=our own limiter, upstream=the provider)",
+ "# TYPE headroom_requests_rate_limited_total counter",
+ ]
)
- _append_metric(
- lines,
- name="headroom_requests_failed_total",
- metric_type="counter",
- help_text="Failed requests",
- value=self.requests_failed,
+ for _source in RATE_LIMIT_SOURCES:
+ _count = self.requests_rate_limited_by_source.get(_source, 0)
+ lines.append(
+ f'headroom_requests_rate_limited_total{{source="{_escape_label_value(_source)}"}}'
+ f" {_count}"
+ )
+ lines.append("")
+
+ # Failures carry the provider that produced them. `sum without
+ # (provider)` reproduces the pre-label value.
+ lines.extend(
+ [
+ "# HELP headroom_requests_failed_total Requests that failed upstream "
+ "(4xx and 5xx, excluding 429), by provider",
+ "# TYPE headroom_requests_failed_total counter",
+ ]
)
+ for _provider, _count in self.requests_failed_by_provider.items():
+ lines.append(
+ f'headroom_requests_failed_total{{provider="'
+ f'{_escape_label_value(str(_provider))}"}} {_count}'
+ )
+ lines.append("")
_append_metric(
lines,
name="headroom_inbound_requests_total",
diff --git a/headroom/proxy/savings_tracker.py b/headroom/proxy/savings_tracker.py
index 37627b54b..207a2b72f 100644
--- a/headroom/proxy/savings_tracker.py
+++ b/headroom/proxy/savings_tracker.py
@@ -1210,12 +1210,22 @@ class SavingsTracker:
self._maybe_save_locked()
def record_lifetime_rate_limited(
- self, *, provider: str | None = None, model: str | None = None
+ self,
+ *,
+ provider: str | None = None,
+ model: str | None = None,
+ source: str = "headroom",
) -> None:
- """Record a rate-limited proxy request without changing legacy history."""
+ """Record a rate-limited proxy request without changing legacy history.
+
+ ``source`` distinguishes Headroom's own limiter from an upstream 429 —
+ see ``PrometheusMetrics.record_rate_limited``.
+ """
with self._lock:
- self._persistent_metrics.record_rate_limited(provider=provider, model=model)
+ self._persistent_metrics.record_rate_limited(
+ provider=provider, model=model, source=source
+ )
self._maybe_save_locked()
def record_lifetime_cache_bust(self, *, tokens_lost: int) -> None:
diff --git a/headroom/proxy/server.py b/headroom/proxy/server.py
index 357e9fe74..a2371960a 100644
--- a/headroom/proxy/server.py
+++ b/headroom/proxy/server.py
@@ -2263,6 +2263,12 @@ class HeadroomProxy(
logger.info(f"Total requests: {m.requests_total}")
logger.info(f"Cached responses: {m.requests_cached}")
logger.info(f"Rate limited: {m.requests_rate_limited}")
+ if m.requests_rate_limited:
+ # The split an operator acts on differently: our limiter firing
+ # means raise the cap, the provider's means back off / shard keys.
+ by_source = m.requests_rate_limited_by_source
+ logger.info(f" headroom limiter: {by_source.get('headroom', 0)}")
+ logger.info(f" upstream 429s: {by_source.get('upstream', 0)}")
logger.info(f"Failed: {m.requests_failed}")
logger.info(f"Input tokens: {m.tokens_input_total:,}")
logger.info(f"Output tokens: {m.tokens_output_total:,}")
@@ -4572,7 +4578,14 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
"total": m.requests_total,
"cached": m.requests_cached,
"rate_limited": m.requests_rate_limited,
+ # Who issued the 429: "headroom" is our own limiter, "upstream"
+ # is the provider (issue #3696). The totals above stay the
+ # unlabelled figures existing consumers read.
+ "rate_limited_by_source": dict(m.requests_rate_limited_by_source),
"failed": m.requests_failed,
+ "failed_by_provider": _remap_provider_counts(
+ dict(m.requests_failed_by_provider), proxy.config
+ ),
"by_provider": _remap_provider_counts(dict(m.requests_by_provider), proxy.config),
"by_model": dict(m.requests_by_model),
"by_stack": dict(m.requests_by_stack),
diff --git a/tests/test_observability_metrics.py b/tests/test_observability_metrics.py
index eb8c2c521..ffff3438d 100644
--- a/tests/test_observability_metrics.py
+++ b/tests/test_observability_metrics.py
@@ -356,11 +356,56 @@ async def test_prometheus_metrics_reads_late_configured_otel_metrics() -> None:
await metrics.record_rate_limited(provider="anthropic", model="claude-sonnet")
assert spy.failed_calls == [{"provider": "openai", "model": None}]
- assert spy.rate_limited_calls == [{"provider": "anthropic", "model": "claude-sonnet"}]
+ # ``source`` reaches OTel too — the split must not exist in Prometheus only.
+ assert spy.rate_limited_calls == [
+ {"provider": "anthropic", "model": "claude-sonnet", "source": "headroom"}
+ ]
finally:
reset_otel_metrics()
+async def test_prometheus_metrics_forwards_rate_limit_source_to_otel() -> None:
+ """An upstream 429 must reach OTel labelled as upstream, not as our own limiter."""
+
+ spy = _SpyProxyMetrics()
+ metrics = PrometheusMetrics(stateless=True)
+ set_otel_metrics(spy) # type: ignore[arg-type]
+
+ try:
+ await metrics.record_rate_limited(provider="anthropic", source="upstream")
+ await metrics.record_rate_limited(provider="anthropic", source="headroom")
+ # An unrecognised value is clamped rather than exported as a new label.
+ await metrics.record_rate_limited(provider="anthropic", source="nonsense")
+
+ assert [call["source"] for call in spy.rate_limited_calls] == [
+ "upstream",
+ "headroom",
+ "headroom",
+ ]
+ finally:
+ reset_otel_metrics()
+
+
+def test_otel_rate_limited_counter_carries_source_attribute() -> None:
+ reader = InMemoryMetricReader()
+ provider = MeterProvider(metric_readers=[reader])
+ otel_metrics = HeadroomOtelMetrics(meter_provider=provider)
+
+ otel_metrics.record_proxy_rate_limited(provider="anthropic", source="upstream")
+ otel_metrics.record_proxy_rate_limited(provider="openai", source="headroom")
+
+ metrics = _collect_metrics(reader)
+ upstream = _find_point(
+ metrics["headroom.proxy.requests.rate_limited"], provider="anthropic", source="upstream"
+ )
+ headroom_side = _find_point(
+ metrics["headroom.proxy.requests.rate_limited"], provider="openai", source="headroom"
+ )
+ assert upstream.value == 1
+ assert headroom_side.value == 1
+
+
@pytest.mark.asyncio
async def test_prometheus_metrics_forwards_savings_drilldown_fields_to_otel(
monkeypatch: pytest.MonkeyPatch,
diff --git a/tests/test_outcome_records_5xx_as_failed.py b/tests/test_outcome_records_5xx_as_failed.py
index 2d59b51fa..b4d68f8a6 100644
--- a/tests/test_outcome_records_5xx_as_failed.py
+++ b/tests/test_outcome_records_5xx_as_failed.py
@@ -25,13 +25,19 @@ class _Metrics:
def __init__(self):
self.failed = []
self.rate_limited = []
+ self.rate_limited_sources = []
self.requested = []
async def record_failed(self, provider):
self.failed.append(provider)
- async def record_rate_limited(self, provider):
+ # ``source`` deliberately has NO default here: the funnel only ever sees
+ # 429s the provider returned, so it must pass source="upstream" explicitly.
+ # If it ever stops doing so, these tests record None and fail loudly rather
+ # than silently re-merging upstream throttling into Headroom's own counter.
+ async def record_rate_limited(self, provider, source=None):
self.rate_limited.append(provider)
+ self.rate_limited_sources.append(source)
async def record_request(self, **kwargs):
self.requested.append(kwargs)
@@ -78,6 +84,11 @@ def test_429_recorded_as_rate_limited_and_skips_success_funnel():
handler = _Handler()
asyncio.run(emit_request_outcome(handler, _outcome(429, tokens_saved=6380)))
assert handler.metrics.rate_limited == ["anthropic"]
+ # Labelled as the PROVIDER's 429, not ours. Headroom's own limiter rejects
+ # before a request is ever sent and records source="headroom" from the
+ # handler; merging the two would tell an operator to raise a cap that is
+ # not the one being hit (issue #3696).
+ assert handler.metrics.rate_limited_sources == ["upstream"]
assert handler.metrics.failed == [] # a rate limit is not a generic failure
assert handler.metrics.requested == [] # its 6,380 "saved" tokens are not savings
diff --git a/tests/test_persistent_metrics.py b/tests/test_persistent_metrics.py
index 018c71bb6..3a5d054d8 100644
--- a/tests/test_persistent_metrics.py
+++ b/tests/test_persistent_metrics.py
@@ -50,6 +50,9 @@ def test_snapshot_accumulates_request_token_cache_cost_and_waste_metrics() -> No
"cached": 1,
"failed": 1,
"rate_limited": 1,
+ # Lifetime carries the same splits the Prometheus labels do.
+ "failed_by_provider": {"anthropic": 1},
+ "rate_limited_by_source": {"headroom": 1},
"by_provider": {"anthropic": 1},
"by_stack": {"codex": 1},
}
@@ -205,3 +208,56 @@ def test_miss_reasons_still_fall_back_to_unknown_on_reload() -> None:
snapshot = state.snapshot(persistence={"enabled": True, "healthy": True})
assert snapshot["prefix_cache"]["misses_by_reason"] == {"ttl_expiry": 2, "unknown": 1}
+
+
+def test_rate_limited_is_split_by_source_and_failures_by_provider() -> None:
+ """Headroom's own limiter and an upstream 429 must stay distinguishable.
+
+ Both land in the same ``rate_limited`` total (unchanged), but an operator
+ acts on them differently: ours firing means raise the cap, the provider's
+ means back off. See issue #3696.
+ """
+ state = _new_state()
+
+ state.record_rate_limited(provider="anthropic", source="headroom")
+ state.record_rate_limited(provider="anthropic", source="upstream")
+ state.record_rate_limited(provider="openai", source="upstream")
+ state.record_failed(provider="anthropic")
+ state.record_failed(provider="openai")
+ state.record_failed(provider="openai")
+
+ requests = state.snapshot(persistence={"enabled": True, "healthy": True})["requests"]
+
+ assert requests["rate_limited"] == 3
+ assert requests["rate_limited_by_source"] == {"headroom": 1, "upstream": 2}
+ assert requests["failed"] == 3
+ assert requests["failed_by_provider"] == {"anthropic": 1, "openai": 2}
+
+
+def test_rate_limit_source_defaults_to_headroom_and_clamps_unknown_values() -> None:
+ state = _new_state()
+
+ state.record_rate_limited(provider="anthropic")
+ state.record_rate_limited(provider="anthropic", source="bogus")
+
+ requests = state.snapshot(persistence={"enabled": True, "healthy": True})["requests"]
+ assert requests["rate_limited_by_source"] == {"headroom": 2}
+
+
+def test_pre_3696_state_loads_without_backfilling_invented_history() -> None:
+ """A state file written before the split keeps its totals and starts the maps empty.
+
+ Back-filling the legacy ``rate_limited`` total into one bucket would claim we
+ observed a split we never recorded.
+ """
+ state = PersistentMetricsState(
+ {"requests": {"total": 10, "failed": 4, "rate_limited": 7}},
+ now=lambda: FIXED_NOW,
+ )
+
+ requests = state.snapshot(persistence={"enabled": True, "healthy": True})["requests"]
+
+ assert requests["rate_limited"] == 7
+ assert requests["failed"] == 4
+ assert requests["rate_limited_by_source"] == {}
+ assert requests["failed_by_provider"] == {}
diff --git a/tests/test_persistent_metrics_integration.py b/tests/test_persistent_metrics_integration.py
index fdb9f2406..3187f9143 100644
--- a/tests/test_persistent_metrics_integration.py
+++ b/tests/test_persistent_metrics_integration.py
@@ -39,6 +39,10 @@ def test_runtime_metric_events_feed_lifetime_without_resetting_runtime_counters(
assert lifetime["requests"]["cached"] == 1
assert lifetime["requests"]["failed"] == 1
assert lifetime["requests"]["rate_limited"] == 1
+ # The Prometheus labels must not stop at Prometheus: lifetime carries the
+ # same two splits (issue #3696). No explicit source here, so it is ours.
+ assert lifetime["requests"]["failed_by_provider"] == {"anthropic": 1}
+ assert lifetime["requests"]["rate_limited_by_source"] == {"headroom": 1}
assert lifetime["requests"]["by_provider"] == {"anthropic": 1}
assert lifetime["requests"]["by_stack"] == {"codex": 1}
assert lifetime["tokens"]["output"] == 3
@@ -47,3 +51,27 @@ def test_runtime_metric_events_feed_lifetime_without_resetting_runtime_counters(
assert lifetime["prefix_cache"]["misses_by_reason"] == {"prefix_change": 1}
assert lifetime["waste_signals"] == {"repetition": 4}
assert metrics.requests_total == 1
+
+
+def test_upstream_rate_limit_reaches_lifetime_labelled_upstream(tmp_path) -> None:
+ """An upstream 429 must not be filed under Headroom's own limiter in lifetime.
+
+ The outcome funnel passes ``source="upstream"``; the whole chain
+ (PrometheusMetrics -> SavingsTracker -> PersistentMetricsState) has to carry
+ it, or the durable view re-merges what Prometheus splits (issue #3696).
+ """
+ tracker = SavingsTracker(path=str(tmp_path / "proxy_savings.json"), save_flush_every=25)
+ metrics = PrometheusMetrics(savings_tracker=tracker)
+
+ asyncio.run(metrics.record_rate_limited(provider="anthropic", source="upstream"))
+ asyncio.run(metrics.record_rate_limited(provider="anthropic", source="headroom"))
+ asyncio.run(metrics.record_failed(provider="openai"))
+
+ lifetime = tracker.lifetime_response()
+
+ assert lifetime["requests"]["rate_limited"] == 2
+ assert lifetime["requests"]["rate_limited_by_source"] == {"upstream": 1, "headroom": 1}
+ assert lifetime["requests"]["failed_by_provider"] == {"openai": 1}
+ # Neither counter may touch the completed-request denominator.
+ assert lifetime["requests"]["total"] == 0
+ assert metrics.requests_total == 0
diff --git a/tests/test_prometheus_obs_counters.py b/tests/test_prometheus_obs_counters.py
index 41c4a7683..413f175bc 100644
--- a/tests/test_prometheus_obs_counters.py
+++ b/tests/test_prometheus_obs_counters.py
@@ -176,3 +176,130 @@ async def test_gate_counter_is_thread_safe_under_concurrent_export() -> None:
assert not errors, f"export() raced the writer: {errors[:3]}"
totals = dict(metrics.kompress_size_gate_by_outcome)
assert sum(totals.values()) == n_threads * per_thread
+
+
+# ---------------------------------------------------------------------------
+# headroom_requests_rate_limited_total{source} / headroom_requests_failed_total{provider}
+#
+# Issue #3696: #3615 routed upstream 429s and 4xx into these two counters, which
+# silently changed what they mean. The labels make the new meaning queryable —
+# "my limiter throttled me" vs "the provider throttled me" are acted on
+# differently, and a failure with no provider attached names no culprit.
+# ---------------------------------------------------------------------------
+
+
+async def test_rate_limited_export_splits_headroom_from_upstream() -> None:
+ metrics = PrometheusMetrics(stateless=True)
+
+ # What the four handler sites do when Headroom's OWN limiter rejects.
+ await metrics.record_rate_limited(provider="anthropic", source="headroom")
+ # What the outcome funnel does with a provider 429.
+ await metrics.record_rate_limited(provider="anthropic", source="upstream")
+ await metrics.record_rate_limited(provider="openai", source="upstream")
+
+ text = await metrics.export()
+
+ assert "# TYPE headroom_requests_rate_limited_total counter" in text
+ assert 'headroom_requests_rate_limited_total{source="headroom"} 1' in text
+ assert 'headroom_requests_rate_limited_total{source="upstream"} 2' in text
+ # `sum without (source)` must reproduce the pre-label value.
+ assert metrics.requests_rate_limited == 3
+ # No unlabelled sample alongside the labelled ones — that would double-count.
+ assert "\nheadroom_requests_rate_limited_total 3" not in text
+
+
+async def test_rate_limited_source_series_exist_before_any_429() -> None:
+ """Both series export from startup, so a rate() never appears mid-incident."""
+ metrics = PrometheusMetrics(stateless=True)
+
+ text = await metrics.export()
+
+ assert 'headroom_requests_rate_limited_total{source="headroom"} 0' in text
+ assert 'headroom_requests_rate_limited_total{source="upstream"} 0' in text
+
+
+async def test_rate_limited_source_defaults_to_headroom_and_clamps_unknown() -> None:
+ metrics = PrometheusMetrics(stateless=True)
+
+ await metrics.record_rate_limited(provider="anthropic")
+ await metrics.record_rate_limited(provider="anthropic", source="bogus")
+
+ text = await metrics.export()
+
+ assert 'headroom_requests_rate_limited_total{source="headroom"} 2' in text
+ assert 'headroom_requests_rate_limited_total{source="upstream"} 0' in text
+ assert "bogus" not in text
+
+
+async def test_failed_export_is_attributed_to_the_provider_that_failed() -> None:
+ metrics = PrometheusMetrics(stateless=True)
+
+ await metrics.record_failed(provider="anthropic")
+ await metrics.record_failed(provider="openai")
+ await metrics.record_failed(provider="openai")
+
+ text = await metrics.export()
+
+ assert "# TYPE headroom_requests_failed_total counter" in text
+ assert 'headroom_requests_failed_total{provider="anthropic"} 1' in text
+ assert 'headroom_requests_failed_total{provider="openai"} 2' in text
+ # `sum without (provider)` must reproduce the pre-label value.
+ assert metrics.requests_failed == 3
+ assert "\nheadroom_requests_failed_total 3" not in text
+
+
+async def test_failed_without_a_provider_lands_in_unknown() -> None:
+ metrics = PrometheusMetrics(stateless=True)
+
+ await metrics.record_failed()
+
+ text = await metrics.export()
+
+ assert 'headroom_requests_failed_total{provider="unknown"} 1' in text
+
+
+async def test_reset_runtime_clears_rate_limit_and_failure_splits() -> None:
+ metrics = PrometheusMetrics(stateless=True)
+
+ await metrics.record_rate_limited(provider="anthropic", source="upstream")
+ await metrics.record_failed(provider="anthropic")
+
+ await metrics.reset_runtime()
+
+ assert metrics.requests_rate_limited == 0
+ assert metrics.requests_failed == 0
+ assert metrics.requests_rate_limited_by_source == {"headroom": 0, "upstream": 0}
+ # Re-seeded, not emptied: the metric must keep exporting a sample after a
+ # reset, same as the rate-limit source series above.
+ assert dict(metrics.requests_failed_by_provider) == {"unknown": 0}
+
+
+async def test_failed_total_exports_a_sample_before_any_failure() -> None:
+ """A healthy proxy must still export ``headroom_requests_failed_total``.
+
+ Before the provider label this counter was always present as a bare
+ ``headroom_requests_failed_total 0``. A labelled map that starts empty
+ emits no sample at all, which makes the documented failure-rate query
+ return an empty vector (an empty numerator empties the whole expression),
+ so the panel reads "No data" instead of 0% while everything is fine --
+ and `absent()` alerts fire on healthy proxies.
+ """
+ metrics = PrometheusMetrics(stateless=True)
+
+ text = await metrics.export()
+
+ assert 'headroom_requests_failed_total{provider="unknown"} 0' in text
+
+ # The seed must not survive as a duplicate once a real provider fails.
+ await metrics.record_failed(provider="anthropic")
+ text = await metrics.export()
+ assert 'headroom_requests_failed_total{provider="anthropic"} 1' in text
+ assert 'headroom_requests_failed_total{provider="unknown"} 0' in text
+ assert metrics.requests_failed == 1
diff --git a/wiki/metrics.md b/wiki/metrics.md
index 5a5c10e97..da1af4221 100644
--- a/wiki/metrics.md
+++ b/wiki/metrics.md
@@ -29,7 +29,9 @@ curl http://localhost:8787/stats
"total": 42,
"cached": 5,
"rate_limited": 0,
- "failed": 0
+ "rate_limited_by_source": { "headroom": 0, "upstream": 0 },
+ "failed": 0,
+ "failed_by_provider": {}
},
"tokens": {
"input": 50000,
@@ -48,6 +50,14 @@ curl http://localhost:8787/stats
}
```
+`requests.total` is the **completed**-request count, not total traffic: a request that
+ended in a 4xx, 5xx or 429 is counted under `requests.failed` or `requests.rate_limited`
+instead and never reaches `total`. `rate_limited_by_source` splits 429s into
+`headroom` (Headroom's own rate limiter refused the request) and `upstream` (the provider
+refused it) — different problems, different fixes. `failed_by_provider` attributes failures
+to the upstream that produced them. The unlabelled `rate_limited` / `failed` totals are
+unchanged.
+
`/stats` keeps the existing live/session fields, including `savings_history`,
for backward compatibility. The new `persistent_savings` block is durable local
proxy compression history stored by default at