Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

18.8 KB
Raw Blame

Headroom Metrics — Dashboard Guide

What each metric shows, so you can build panels against it.

Two endpoints. Both are on the proxy (default :8787).

Surface How to get it Use it for
Prometheus — GET /metrics Always on, no config Everything below. Start here.
OpenTelemetry — OTLP/HTTP HEADROOM_OTEL_METRICS_ENABLED=1 + pip install "headroom-ai[proxy,otel]" Same data, dotted names, plus per-tenant labels

Names differ between them: Prometheus uses headroom_tokens_saved_total (milliseconds for timings), OTel uses headroom.proxy.tokens.saved (seconds). Both are listed below.


The savings panel — start here

headroom.proxy.tokens.saved is the headline number. It already combines compression + tool-schema deferral — no need to add anything to it.

Metric What it shows
headroom.proxy.tokens.saved (OTel) Total input tokens Headroom kept out of the request. Compression + tool savings, combined. This is your hero number.
headroom.proxy.savings.usd{source} (OTel) Dollars saved, split by layer: compression, tool_schema, output_shaping, provider_cache. Sum for the total.
headroom_persistent_savings_tokens_saved_total Same tokens-saved number, but survives proxy restarts. Use for "lifetime saved" tiles.
headroom_persistent_savings_compression_savings_usd_total Lifetime dollars saved, durable across restarts.
headroom_tokens_input_total Input tokens actually sent upstream (post-compression). The denominator for a reduction %.
headroom_tokens_output_total Output tokens returned by the provider.
# Hero tile: tokens saved per second
rate(headroom_tokens_saved_total[5m])
  + sum(rate(headroom_savings_attributed_tokens_total{source="tool_search",realized="true"}[5m]))

# Context reduction %
100 * rate(headroom_tokens_saved_total[5m])
    / clamp_min(rate(headroom_tokens_input_total[5m]) + rate(headroom_tokens_saved_total[5m]), 1)

# Lifetime tiles (survive restart)
headroom_persistent_savings_tokens_saved_total
headroom_persistent_savings_compression_savings_usd_total

One catch on the Prometheus side. headroom_tokens_saved_total is compression only — it leaves out tool-schema deferral. The OTel headroom.proxy.tokens.saved includes both. That's why the query above adds the tool_search term back in. On tool-heavy workloads the gap is large.


Latency panel

All Prometheus timings are in milliseconds, exposed as _sum / _count / _min / _max. Build means with rate(sum)/rate(count).

Metric What it shows
headroom_overhead_ms_* Latency Headroom itself adds. Handler entry → end of compression. Excludes the LLM call. This is the "what does this cost us" number.
headroom_latency_ms_* Total request duration, including the provider.
headroom_ttfb_ms_* Time to first byte from upstream. Streaming requests only.
headroom_stage_timing_ms_*{path,stage} Where time went inside the handler — compression_first_stage, upstream_connect, memory_context, etc.
headroom_transform_timing_ms_*{transform} Time per compression transform. Use to find a slow transform.
# Headroom's added overhead, mean ms
rate(headroom_overhead_ms_sum[5m]) / rate(headroom_overhead_ms_count[5m])

# End-to-end, mean ms
rate(headroom_latency_ms_sum[5m]) / rate(headroom_latency_ms_count[5m])

# Slowest stages
topk(5, rate(headroom_stage_timing_ms_sum[5m]) / rate(headroom_stage_timing_ms_count[5m]))

No percentiles are available. There are no histogram buckets on /metrics, and the OTel histograms ship with default buckets that put every request into one bucket, so histogram_quantile() returns nonsense. Means work fine. For real p95/p99 today, use the headroom perf CLI.

Also: divide each _sum by its own _count. Overhead and TTFB are only sampled when > 0, so their counts are smaller than the latency count.


Cache panel

Metric What it shows
headroom_provider_cache_hit_requests_total{provider} Requests that read from the provider's prompt cache.
headroom_provider_cache_requests_total{provider} Requests with any cache activity. The correct denominator for hit rate.
headroom_cache_read_tokens_total{provider} Tokens served from cache (the discounted ones).
headroom_cache_write_tokens_total{provider} Tokens written into cache (these carry a premium).
headroom_cache_write_ttl_tokens_total{provider,ttl} Cache writes split by TTL — 5m vs 1h.
headroom_uncached_input_tokens_total{provider} Input tokens that missed cache entirely.
headroom_cache_bust_total Requests where compression broke a cached prefix. Should stay near zero.
headroom_cache_miss_attribution_total{provider,reason} Why a cached prefix missed — ttl_expiry, prefix_change, unknown.
# Cache hit rate by provider
sum by (provider) (rate(headroom_provider_cache_hit_requests_total[5m]))
  / sum by (provider) (rate(headroom_provider_cache_requests_total[5m]))

# Compression breaking cache — alert if this rises
rate(headroom_cache_bust_total[5m])

Don't use headroom_requests_cached_total as a hit rate. It mixes the provider's prompt cache with Headroom's own response cache into one boolean, so it measures neither.


Traffic & health panel

Metric What it shows
headroom_requests_total Completed requests — the denominator for success-side stats, not total traffic. A request that ended in a 4xx, 5xx or 429 never reaches it. Unlabelled.
headroom_inbound_requests_total Every inbound HTTP request the proxy accepted, whatever the outcome. This is the all-traffic counter.
headroom_requests_by_provider{provider} Traffic split by provider — anthropic, openai, gemini, bedrock… Completed requests only, same denominator as headroom_requests_total. Use headroom_requests_failed_total{provider} for the failure side.
headroom_requests_by_model{model} Traffic split by model. Capped at 1024 distinct; overflow lands in model="other". Completed requests only.
headroom_requests_failed_total{provider} Requests that failed upstream — 4xx and 5xx, excluding 429. Excluded from headroom_requests_total, so add it back when you compute a rate.
headroom_requests_rate_limited_total{source} Requests rejected with 429, by either Headroom's own rate limiter (source="headroom") or the upstream provider (source="upstream"). Excluded from headroom_requests_total.
headroom_compression_failed_total{reason} Compression failures — timeout or error. Fails open, so traffic keeps flowing but savings quietly stop. Worth an alert.
headroom_compression_quarantine_total{event} Compression disabled after repeated timeouts — activated, skipped, released.
headroom_inbound_requests_active In-flight requests, gauge. Counts all HTTP including /metrics.
headroom_active_ws_sessions Live Codex WebSocket sessions, gauge.

headroom_requests_total is a denominator, not traffic

headroom_requests_total counts requests that completed — it has always been incremented only on the success path, and since the outcome funnel started short-circuiting at >= 400, every 4xx, 5xx and 429 drops out of it. So:

  • headroom_requests_total + headroom_requests_failed_total + headroom_requests_rate_limited_total ≈ the requests the proxy forwarded upstream.
  • headroom_inbound_requests_total is the honest all-traffic counter (it also counts /metrics, /stats and other non-proxy HTTP).

That is why the failure-rate query below adds failed back into its own denominator, and why you should not "simplify" it to … / rate(headroom_requests_total[5m]): failures are not in requests_total, so that form divides failures by successes and over-reports the rate — badly, during exactly the incident you wrote it for (at 100% failures the denominator goes to zero).

# Failure rate. The denominator deliberately re-adds `failed`: a failed request
# is NOT in headroom_requests_total, so requests_total alone is successes-only.
rate(headroom_requests_failed_total[5m])
  / clamp_min(rate(headroom_requests_total[5m]) + rate(headroom_requests_failed_total[5m]), 1)

# Failure rate by provider — which upstream is actually broken.
sum by (provider) (rate(headroom_requests_failed_total[5m]))
  / clamp_min(
      sum by (provider) (rate(headroom_requests_by_provider[5m]))
        + sum by (provider) (rate(headroom_requests_failed_total[5m])),
      1
    )

# Who is throttling you: your own limiter (raise the cap) vs the provider
# (back off, shard keys). Alert on these separately — they are different actions.
sum by (source) (rate(headroom_requests_rate_limited_total[5m]))

# Provider throttling only — the one that means "slow down".
rate(headroom_requests_rate_limited_total{source="upstream"}[5m])

# Savings silently stopped
sum by (reason) (rate(headroom_compression_failed_total[5m]))

# Traffic mix
sum by (provider) (rate(headroom_requests_by_provider[5m]))

Migration — these two counters gained labels. headroom_requests_failed_total is now labelled by provider and headroom_requests_rate_limited_total by source, so an unaggregated query that used to return one series now returns several, and a panel or alert that graphed the bare counter will fan out into one line per label value. sum without (provider) (rate(headroom_requests_failed_total[5m])) and sum without (source) (rate(headroom_requests_rate_limited_total[5m])) reproduce the old single-series behaviour exactly. Both source series are exported from process start (including at zero); failed exports one series per provider that has actually failed, so it has no series at all until the first failure.

The same split is available outside Prometheus: /stats carries requests.rate_limited_by_source and requests.failed_by_provider alongside the unlabelled requests.rate_limited / requests.failed totals (which are unchanged), the lifetime aggregate carries the same two maps, and the OTel counters headroom.proxy.requests.rate_limited / .failed carry source / provider attributes.


Anthropic subscription panel

Only if you're on an Anthropic OAuth/subscription plan. OTel only, gauges, no labels.

Metric What it shows
headroom.subscription.5h_utilization_pct How much of the 5-hour rate-limit window is used (0–100).
headroom.subscription.7d_utilization_pct Same for the 7-day window.
headroom.subscription.5h_seconds_to_reset Seconds until the 5-hour window resets.
headroom.subscription.7d_seconds_to_reset Seconds until the 7-day window resets.
headroom.subscription.overage_usd Extra-usage credits consumed, in dollars.

Attribution — where savings came from

Metric What it shows
headroom_savings_attributed_tokens_total{source,realized} Tokens saved, broken out by named source. source="tool_search" is tool-schema deferral.
headroom_savings_attributed_usd_total{source,realized} Dollars saved by source. Gauge, can go negative — don't rate() it.
headroom_savings_attribution_events_total{source,realized} How often each source contributed.
headroom_waste_signal_tokens_total{signal} Wasteful patterns detected in the input — json_bloat, base64, repetition, reread… This is diagnosis, not savings.

These rows explain the headline total — they are never added to it.


Compression internals

Metric What it shows
headroom.compression.tokens.input (OTel) Tokens going into the compression pipeline.
headroom.compression.tokens.output (OTel) Tokens coming out.
headroom.compression.tokens.saved (OTel) The difference. Pipeline-level view of compression only.
headroom.compression.runs (OTel) Pipeline executions. Note: per pipeline run, not per request.
headroom.compression.pipeline.duration (OTel, seconds) How long the pipeline took.
headroom.compression.transforms{transform} (OTel) Which transforms fired. High cardinality — drop or aggregate at the collector.

Five things that will break a dashboard

  1. Only savings counters survive a restart. 55 of 60 Prometheus families reset to zero when the proxy restarts. Only headroom_persistent_savings_* is durable, and it needs HEADROOM_WORKSPACE_DIR on a persistent volume — otherwise it resets on every deploy.

  2. No percentiles anywhere. Use means. See the latency section.

  3. headroom_latency_ms measures differently for streaming. On streaming requests the timer starts after compression, so end-to-end is latency + overhead. On non-streaming it's just latency. Don't mix both in one panel.

  4. A 5xx erases its own savings. Requests that fail upstream are dropped from every savings and token counter. During a provider incident, savings rates look artificially clean while throughput falls.

  5. /metrics needs auth if you set a proxy token. With HEADROOM_PROXY_TOKEN set, any non-loopback scraper must send Authorization: Bearer <token> (or X-Headroom-Proxy-Token: <token>). Loopback is always exempt. The proxy removes the token from every request before forwarding it, so it never reaches a model provider.


Metrics the docs mention that don't exist

If panels came back empty, this is probably why. These names appear in the published docs but not in the code:

headroom_compression_ratio · headroom_latency_seconds (and _bucket) · headroom_cache_hits_total · headroom_cache_misses_total · headroom_cost_usd_total · the mode="optimize" label on headroom_requests_total

The shipped examples/grafana/headroom-dashboard.json also filters every panel on pool and hook labels that no metric emits — the dropdowns will be permanently empty. Its metric names are otherwise correct.


Setup reference

# Prometheus — nothing to do, GET /metrics is always on

# OpenTelemetry
pip install "headroom-ai[proxy,otel]"
export HEADROOM_OTEL_METRICS_ENABLED=1
export HEADROOM_OTEL_METRICS_ENDPOINT=https://otel.corp.example/v1/metrics
export HEADROOM_OTEL_METRICS_HEADERS="authorization=Bearer XXX"
export HEADROOM_OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$HOSTNAME"
Variable Default Notes
HEADROOM_OTEL_METRICS_ENABLED 0 Master switch
HEADROOM_OTEL_METRICS_EXPORTER otlp_http Or console. No gRPC exporter exists.
HEADROOM_OTEL_METRICS_ENDPOINT unset Passed verbatim — /v1/metrics is not appended
HEADROOM_OTEL_METRICS_HEADERS unset k=v,k2=v2
HEADROOM_OTEL_METRICS_EXPORT_INTERVAL_MS 10000
HEADROOM_OTEL_SERVICE_NAME headroom-proxy
HEADROOM_OTEL_RESOURCE_ATTRIBUTES unset Set service.instance.id here — Headroom doesn't, and replicas will collide

Verify with curl -s localhost:8787/stats | jq .otel.

Multi-tenant labels: register_otel_metric_attribute_provider() adds request-scoped attributes (tenant, team, cost centre) to every OTel datapoint. Max 16 attributes, 256 chars each.

Air-gapped deployments: HEADROOM_OFFLINE=1 refuses every connection Headroom itself decides to make to a destination Headroom itself chose. That is the whole list, not a sample: the anonymous usage beacon (which is on by default), the update check, the license/usage reporter, OTLP metric and Langfuse trace export, HuggingFace / Kompress / fastembed model and tokenizer downloads (Python and Rust), the remote Kompress endpoint, the headroom install release-binary and codebase-memory-mcp downloads, litellm's import-time pricing-table refresh, CrewAI's import-time install tracking, the BFCL eval-dataset fetch and the provider SDK clients the eval harness drives, GitHub Copilot device-flow auth and token exchange, the Anthropic / Codex / Copilot subscription pollers, the OpenAI embedders, the Headroom Cloud compression modes in the ASGI and LiteLLM integrations, and the TLS diagnostics (headroom doctor --network endpoint checks and the certificate-chain re-probe after an upstream TLS failure).

Four things are deliberately still allowed, and they are the complete set of exceptions:

  1. Your traffic through the proxy. Requests you send through Headroom are still forwarded to the upstream you configured. That is your traffic, not Headroom's, and an air-gapped deployment points it at an on-prem endpoint — refusing it would mean refusing to be a proxy.
  2. Operator-configured local endpoints. Today that is exactly one thing: the Ollama embedder, whose address comes entirely from your configuration and defaults to http://localhost:11434. No hard-coded internet host is permitted under this exception.
  3. Loopback. Health and readiness probes against your own proxy on 127.0.0.1 (headroom doctor, the installers, the MCP sidecar).
  4. Paths an is_offline() check already makes unreachable, where no connection is ever built in the first place.

Those four are enumerated with written reasons in _EGRESS_ALLOWLIST in tests/test_offline_egress_chokepoint.py, which fails the build if a new egress path appears that is neither guarded nor one of them. There is no category for "known violation": a path that can dial the internet with the flag set is a bug. A refusal reaches you as a message — a Click error on the CLI, a named "model unavailable" from a model loader, a logged line from a background poller — never as a traceback and never as a silent degradation. Enforcing the same policy at the network layer as well is still good practice; it is no longer the only thing standing between you and Headroom-initiated egress.

litellm's pricing-table refresh is refused by setting LITELLM_LOCAL_MODEL_COST_MAP=True before litellm is imported, so it holds only when Headroom is the first thing in the process to import litellm. A host that imports litellm before Headroom, such as a LiteLLM proxy loading the Headroom callback, has made that fetch already and must set the variable itself. Offline, cost and savings figures use the pricing table bundled with the installed litellm release, so a model released after it prices as unknown.

OTLP export is refused loudly: with HEADROOM_OFFLINE=1 set, HEADROOM_OTEL_METRICS_ENABLED=1 plus the default otlp_http exporter makes the proxy exit 78 at startup with an explanation, rather than quietly dropping metrics. There is no exemption for a collector that looks local — an in-cluster address is not reliably distinguishable from an internet one. For metrics under an air-gap, either scrape localhost:8787/metrics or set HEADROOM_OTEL_METRICS_EXPORTER=console, both of which stay on-box. HEADROOM_KOMPRESS_ENDPOINT and HEADROOM_LANGFUSE_ENABLED are refused the same way, for the same reason.