Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
Update benchmarks with real production telemetry and compression data
chopratejas4f2213b7a4ad
diff --git a/docs/benchmarks.md b/docs/benchmarks.mdindex 73da0ea27..1288544ca 100644--- a/docs/benchmarks.md+++ b/docs/benchmarks.md@@ -1,141 +1,163 @@-# Accuracy Benchmarks+# Benchmarks-Headroom's core promise: **compress context without losing accuracy**. This page shows our latest benchmark results against established open-source datasets.+Headroom's core promise: **compress context without losing accuracy**. This page shows accuracy benchmarks, compression performance, and real-world production telemetry from 250+ active proxy instances.-!!! success "Key Result"- **98.2% recall** on article extraction with **94.9% compression** — we preserve nearly all information while dramatically reducing tokens.+!!! success "Key Results"+ **98.2% recall** on article extraction with **94.9% compression**.+ **52ms median overhead** in production. **1.4 billion tokens saved** across 249 instances.----## Summary+## Compression Performance-| Benchmark | Metric | Headroom | Baseline | Status |-|-----------|--------|----------|----------|--------|-| [Scrapinghub Article Extraction](#html-extraction) | F1 Score | **0.919** | 0.958 | :white_check_mark: |-| [Scrapinghub Article Extraction](#html-extraction) | Recall | **98.2%** | — | :white_check_mark: |-| [Scrapinghub Article Extraction](#html-extraction) | Compression | **94.9%** | — | :white_check_mark: |-| [SmartCrusher (JSON)](#json-compression) | Accuracy | **100%** | — | :white_check_mark: |-| [SmartCrusher (JSON)](#json-compression) | Compression | **87.6%** | — | :white_check_mark: |+Tested on Apple M-series (CPU), headroom v0.5.18. Each test runs `compress()` on realistic tool outputs.++| Content Type | Original | Compressed | Saved | Ratio | Latency |+|---|---|---|---|---|---|+| JSON array (100 items) | 3,163 | 297 | 2,866 | **90.6%** | 1ms |+| JSON array (500 items) | 9,526 | 1,614 | 7,912 | **83.1%** | 2ms |+| Shell output (200 lines) | 3,238 | 469 | 2,769 | **85.5%** | 1ms |+| Build log (200 lines) | 2,412 | 148 | 2,264 | **93.9%** | 1ms |+| grep results (150 hits) | 2,624 | 2,624 | 0 | 0.0% | <1ms |+| Python source (~480 lines) | 2,958 | 2,958 | 0 | 0.0% | <1ms |+| **Total** | **23,921** | **8,110** | **15,811** | **66.1%** | **5ms** |++**Notes:**++- grep results and Python source show 0% compression — these are already compact structured formats. SmartCrusher only compresses JSON arrays; code passes through to preserve correctness.+- Latency is for the `compress()` SDK call, not the full proxy round-trip.++---++## Production Telemetry++Real-world data from **50,000+ proxy sessions** across 250+ unique instances (March 30 – April 2, 2026). Collected via anonymous telemetry beacon (opt-out: `HEADROOM_TELEMETRY=off`).++### Proxy Overhead++| Percentile | Latency |+|---|---|+| **Median (P50)** | **52ms** |+| P90 | 309ms |+| P99 | 4,172ms |+| Mean | 161ms |++The median 52ms overhead is negligible compared to LLM inference time (typically 2-10 seconds).++### Compression Rate++| Percentile | Compression |+|---|---|+| P25 | 4.8% |+| **Median** | **4.8%** |+| P75 | 6.9% |+| Mean | 11.3% |++Median compression is modest because many requests are short conversational turns. Heavy tool-use sessions (file reads, shell output) see 40-80% compression.++### Pipeline Step Timing (Production Median)++| Step | Median | P90 | Description |+|---|---|---|---|+| `pipeline_total` | **16.9ms** | 289ms | Full compression pipeline |+| `content_router` | 11.7ms | 259ms | Content detection + routing |+| `compressor:smart_crusher` | 50.1ms | 50ms | JSON array compression |+| `compressor:text` | 32.0ms | 576ms | Text compression (Kompress ONNX) |+| `compressor:mixed` | 316ms | 428ms | Mixed content compression |+| `compressor:code_aware` | 815ms | 886ms | Tree-sitter AST compression |+| `_initial_token_count` | 2.9ms | 16ms | Token counting (tiktoken) |+| `_deep_copy` | 0.1ms | 0.3ms | Message copy overhead |++### Fleet Summary++| Metric | Value |+|---|---|+| Clean instances | 249 |+| Total tokens saved | 1.4 billion |+| Total $ saved | ~$4,000 |+| OS distribution | Linux 57%, macOS 38%, Windows 5% |+| Top version | 0.5.17 (77%) |+| Models used | Claude Opus 4.6, Sonnet 4.6, Haiku 4.5 |----## HTML Extraction+## Accuracy Benchmarks++### HTML Extraction**Dataset**: [Scrapinghub Article Extraction Benchmark](https://huggingface.co/datasets/allenai/scrapinghub-article-extraction-benchmark)**Samples**: 181 HTML pages with ground truth article bodies**Baseline**: trafilatura (0.958 F1)-HTMLExtractor removes scripts, styles, navigation, ads, and boilerplate while preserving article content.--### Results-| Metric | Value | Description |-|--------|-------|-------------|+|---|---|---|| **F1 Score** | 0.919 | Token-level overlap with ground truth || **Precision** | 0.879 | Proportion of extracted content that's relevant || **Recall** | 0.982 | Proportion of ground truth content captured || **Compression** | 94.9% | Average size reduction |-### Why Recall Matters Most--For LLM applications, **recall is critical** — we must capture all relevant information. A 98.2% recall means:--- Nearly all article content is preserved-- LLMs can answer questions accurately from extracted content-- The slight precision drop (some extra content) doesn't hurt LLM accuracy--### Run It Yourself+For LLM applications, **recall is critical** — 98.2% means nearly all article content is preserved. The slight precision drop (some extra content) doesn't hurt LLM accuracy.```bash-# Install dependencies+# Run it yourselfpip install "headroom-ai[html]" datasets--# Run the benchmarkpytest tests/test_evals/test_html_oss_benchmarks.py::TestExtractionBenchmark -v -s```------## JSON Compression (SmartCrusher)+### JSON Compression (SmartCrusher)**Test**: 100 production log entries with critical error at position 67**Task**: Find the error, error code, resolution, and affected count-### Results-| Metric | Baseline | Headroom |-|--------|----------|----------|+|---|---|---|| Input tokens | 10,144 | 1,260 || Correct answers | 4/4 | **4/4** || Compression | — | **87.6%** |-SmartCrusher preserves:--- First N items (schema examples)-- Last N items (recency)-- All anomalies (errors, warnings, outliers)-- Statistical distribution--### Run It Yourself--```bash-python examples/needle_in_haystack_test.py-```-------## QA Accuracy Preservation--We verify that LLMs can answer questions equally well from compressed content.--**Method**:-1. Take original HTML content-2. Extract with HTMLExtractor-3. Ask LLM same question on both-4. Compare answers against ground truth--**Datasets**: SQuAD v2, HotpotQA+SmartCrusher preserves first N items (schema), last N items (recency), all anomalies (errors, warnings), and statistical distribution.-### Results+### QA Accuracy Preservation| Metric | Original HTML | Extracted | Delta |-|--------|---------------|-----------|-------|+|---|---|---|---|| F1 Score | 0.85 | 0.87 | +0.02 || Exact Match | 60% | 62% | +2% |!!! note "Extraction Can Improve Accuracy"Removing HTML noise sometimes *helps* LLMs focus on relevant content.-### Run It Yourself+----```bash-# Requires OPENAI_API_KEY-pytest tests/test_evals/test_html_oss_benchmarks.py::TestQAAccuracyPreservation -v -s-```+## Limitations----+### What Headroom Does NOT Compress-## Multi-Tool Agent Test+- **Short messages** (< 300 tokens) — overhead exceeds savings+- **Source code** — passes through unchanged to preserve correctness (unless tree-sitter AST compression is enabled)+- **grep/search results** — compact structured format, already minimal+- **Images** — counted at fixed token cost (~1,600 tokens), not compressed as text+- **System prompts** — preserved for prefix cache compatibility-**Setup**: Agno agent with 4 tools investigating a memory leak-**Total tool output**: 62,323 chars (~15,580 tokens)+### Known Overhead Sources-### Results+- **Token counting** (P90: 16ms) — runs tiktoken twice (before + after compression)+- **Tree-sitter AST parsing** (P90: 886ms) — expensive for large code files+- **Kompress ONNX** (P90: 576ms) — ML inference on CPU for text compression+- **Content detection** (Magika) — ML classification of content type-| Metric | Baseline | Headroom |-|--------|----------|----------|-| Tokens sent | 15,662 | 6,100 |-| Tool calls | 4 | 4 |-| Correct findings | All | **All** |-| Compression | — | **76.3%** |+### When Headroom Adds the Most Value-Both found: Issue #42, `cleanup_worker()` fix, OutOfMemoryError logs, relevant papers.+- **Long agent sessions** with accumulated tool outputs (40-80% compression)+- **JSON-heavy workflows** (API responses, database queries) — 83-94% compression+- **Build/test output** — 85-94% compression+- **Multi-tool agents** — 60-76% compression across tool results-### Run It Yourself+### When Headroom Adds Little Value-```bash-python examples/multi_tool_agent_test.py-```+- **Short conversational exchanges** — median 4.8% compression+- **Code-only sessions** (reading/writing files) — code passes through+- **Single-turn requests** — no accumulated context to compress---@@ -143,21 +165,12 @@ python examples/multi_tool_agent_test.py### Token-Level F1-We use the standard NLP metric for text overlap:-```Precision = |predicted ∩ ground_truth| / |predicted|Recall = |predicted ∩ ground_truth| / |ground_truth|F1 = 2 * (Precision * Recall) / (Precision + Recall)```-### QA Accuracy--For question-answering, we measure:--- **Exact Match**: Normalized answer strings match exactly-- **F1 Score**: Token overlap between predicted and ground truth answers-### Compression Ratio```@@ -166,12 +179,17 @@ Compression = 1 - (compressed_size / original_size)A 94.9% compression means the output is 5.1% of the original size.+### Production Telemetry++- Collected via anonymous beacon (no prompts, no content, no PII)+- Image-inflated instances excluded (base64 counted as text tokens — fixed in v0.5.18)+- Multi-worker beacon spam excluded (per-instance MAX, not SUM)+- Opt-out: `HEADROOM_TELEMETRY=off`+---## Reproducing Results-All benchmarks are reproducible:-```bash# Clone the repogit clone https://github.com/chopratejas/headroom.git@@ -183,23 +201,6 @@ pip install -e ".[evals,html]"# Run all benchmarkspytest tests/test_evals/ -v -s-# Run specific benchmark-pytest tests/test_evals/test_html_oss_benchmarks.py -v -s+# Run compression benchmark+python -c "from headroom import compress; print(compress([{'role':'user','content':'test'}]))"```--### CI Integration--Benchmarks run on every PR. See [.github/workflows/ci.yml](https://github.com/chopratejas/headroom/blob/main/.github/workflows/ci.yml).-------## Adding New Benchmarks--We welcome contributions! See [CONTRIBUTING.md](https://github.com/chopratejas/headroom/blob/main/CONTRIBUTING.md) for guidelines.--Benchmarks should:--1. Use established open-source datasets-2. Include reproducible evaluation code-3. Test accuracy preservation, not just compression-4. Run in CI without API keys (or skip gracefully)