Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
Rewrite README with verified before/after examples
chopratejas8766d83f6805
diff --git a/README.md b/README.mdindex 69f98d74a..4c09e0160 100644--- a/README.md+++ b/README.md@@ -4,7 +4,7 @@<strong>The Context Optimization Layer for LLM Applications</strong></p><p align="center">- Cut your LLM costs by 50-90% without losing accuracy+ Tool outputs are 70-95% redundant boilerplate. Headroom compresses that away.</p></p>@@ -24,33 +24,83 @@</p>+---++## Does It Actually Work? A Real Test++**The setup:** 100 production log entries. One critical error buried at position 67.++<details>+<summary><b>BEFORE:</b> 100 log entries (18,952 chars) - click to expand</summary>++```json+[+ {"timestamp": "2024-12-15T00:00:00Z", "level": "INFO", "service": "api-gateway", "message": "Request processed successfully - latency=50ms", "request_id": "req-000000", "status_code": 200},+ {"timestamp": "2024-12-15T01:01:00Z", "level": "INFO", "service": "user-service", "message": "Request processed successfully - latency=51ms", "request_id": "req-000001", "status_code": 200},+ {"timestamp": "2024-12-15T02:02:00Z", "level": "INFO", "service": "inventory", "message": "Request processed successfully - latency=52ms", "request_id": "req-000002", "status_code": 200},+ // ... 64 more INFO entries ...+ {"timestamp": "2024-12-15T03:47:23Z", "level": "FATAL", "service": "payment-gateway", "message": "Connection pool exhausted", "error_code": "PG-5523", "resolution": "Increase max_connections to 500 in config/database.yml", "affected_transactions": 1847},+ // ... 32 more INFO entries ...+]+```+</details>++**AFTER:** Headroom compresses to 6 entries (1,155 chars):++```json+[+ {"timestamp": "2024-12-15T00:00:00Z", "level": "INFO", "service": "api-gateway", ...},+ {"timestamp": "2024-12-15T01:01:00Z", "level": "INFO", "service": "user-service", ...},+ {"timestamp": "2024-12-15T02:02:00Z", "level": "INFO", "service": "inventory", ...},+ {"timestamp": "2024-12-15T03:47:23Z", "level": "FATAL", "service": "payment-gateway", "error_code": "PG-5523", "resolution": "Increase max_connections to 500 in config/database.yml", "affected_transactions": 1847},+ {"timestamp": "2024-12-15T02:38:00Z", "level": "INFO", "service": "inventory", ...},+ {"timestamp": "2024-12-15T03:39:00Z", "level": "INFO", "service": "auth", ...}+]+```++**What happened:** First 3 items + the FATAL error + last 2 items. The critical error at position 67 was automatically preserved.++---++**The question we asked Claude:** "What caused the outage? What's the error code? What's the fix?"++| | Baseline | Headroom |+|--|----------|----------|+| Input tokens | 10,144 | 1,260 |+| Correct answers | **4/4** | **4/4** |++Both responses: *"payment-gateway service, error PG-5523, fix: Increase max_connections to 500, 1,847 transactions affected"*++**87.6% fewer tokens. Same answer.**++Run it yourself: `python examples/needle_in_haystack_test.py`++---++## How It Works++Headroom doesn't summarize or truncate blindly. It uses **statistical analysis**:++1. **Detects redundancy** - Repeated fields like `"language": "typescript"` across 100 items+2. **Keeps what matters** - First items, last items, query-relevant matches, anomalies+3. **Preserves errors** - Never drops items containing "error", "exception", "failed"+4. **Maintains schema** - Output JSON structure stays identical++The compression is **reversible** via CCR (Compress-Cache-Retrieve). If the LLM needs more data, it can request the original.+---## Why Headroom?- **Zero code changes** - works as a transparent proxy-- **50-90% cost savings** - verified on real workloads+- **47-92% savings** - depends on your workload (tool-heavy = more savings)- **Reversible compression** - LLM retrieves original data via CCR- **Content-aware** - code, logs, JSON each handled optimally- **Provider caching** - automatic prefix optimization for cache hits-- **Persistent memory** - remember across conversations with zero-latency extraction- **Framework native** - LangChain, Agno, MCP, agents supported----## Headroom vs Alternatives--| Approach | Token Reduction | Accuracy | Reversible | Latency |-|----------|-----------------|----------|------------|---------|-| **Headroom** | 50-90% | No loss | Yes (CCR) | ~1-5ms |-| Truncation | Variable | Data loss | No | ~0ms |-| Summarization | 60-80% | Lossy | No | ~500ms+ |-| No optimization | 0% | Full | N/A | 0ms |--**Headroom wins** because it intelligently selects relevant content while keeping a retrieval path to the original data.------## 30-Second Quickstart### Option 1: Proxy (Zero Code Changes)@@ -144,16 +194,21 @@ See the full [Agno Integration Guide](docs/agno.md) for hooks, multi-provider su----## Performance+## Verified Performance++These numbers are from actual API calls, not estimates:++| Scenario | Before | After | Savings | Verified |+|----------|--------|-------|---------|----------|+| Code search (100 results) | 17,765 tokens | 1,408 tokens | 92% | Claude Sonnet |+| SRE incident debugging | 65,694 tokens | 5,118 tokens | 92% | GPT-4o |+| Codebase exploration | 78,502 tokens | 41,254 tokens | 47% | GPT-4o |+| GitHub issue triage | 54,174 tokens | 14,761 tokens | 73% | GPT-4o |-| Scenario | Before | After | Savings |-|----------|--------|-------|---------|-| Search results (1000 items) | 45,000 tokens | 4,500 tokens | 90% |-| Log analysis (500 entries) | 22,000 tokens | 3,300 tokens | 85% |-| Long conversation (50 turns) | 80,000 tokens | 32,000 tokens | 60% |-| Agent with tools (10 calls) | 100,000 tokens | 15,000 tokens | 85% |+**Overhead**: ~1-5ms compression latency-**Overhead**: ~1-5ms per request+**When savings are highest**: Tool-heavy workloads (search, logs, database queries)+**When savings are lowest**: Conversation-heavy workloads with minimal tool use---diff --git a/examples/capture_before_after_json.py b/examples/capture_before_after_json.pynew file mode 100644index 000000000..922b9e2f0--- /dev/null+++ b/examples/capture_before_after_json.py@@ -0,0 +1,80 @@+#!/usr/bin/env python3+"""Capture the actual before/after JSON for the README."""++import json++from headroom.config import SmartCrusherConfig+from headroom.transforms.smart_crusher import SmartCrusher++# The critical error (needle)+CRITICAL_ERROR = {+ "timestamp": "2024-12-15T03:47:23Z",+ "level": "FATAL",+ "service": "payment-gateway",+ "message": "Connection pool exhausted - max_connections=100 exceeded",+ "error_code": "PG-5523",+ "resolution": "Increase max_connections to 500 in config/database.yml",+ "affected_transactions": 1847,+}+++def create_log_entries(n: int = 100, needle_position: int = 67) -> list:+ """Create n log entries with the critical error at needle_position."""+ logs = []+ for i in range(n):+ if i == needle_position:+ logs.append(CRITICAL_ERROR)+ else:+ logs.append(+ {+ "timestamp": f"2024-12-15T{(i % 24):02d}:{(i % 60):02d}:00Z",+ "level": "INFO",+ "service": ["api-gateway", "user-service", "inventory", "auth"][i % 4],+ "message": f"Request processed successfully - latency={50 + (i % 100)}ms",+ "request_id": f"req-{i:06d}",+ "status_code": 200,+ }+ )+ return logs+++def main():+ logs = create_log_entries(100, needle_position=67)+ original_json = json.dumps(logs)++ # Compress with SmartCrusher+ config = SmartCrusherConfig()+ crusher = SmartCrusher(config)+ result = crusher.crush(original_json, query="error outage fatal")++ compressed_data = json.loads(result.compressed)++ print("=" * 70)+ print("BEFORE: First 3 of 100 log entries")+ print("=" * 70)+ print(json.dumps(logs[:3], indent=2))+ print(f"\n... plus 97 more entries (100 total, {len(original_json):,} chars)")++ print("\n" + "=" * 70)+ print(f"AFTER: Headroom keeps {len(compressed_data)} entries")+ print("=" * 70)+ print(json.dumps(compressed_data, indent=2))++ # Check if the needle was preserved+ print("\n" + "=" * 70)+ print("NEEDLE PRESERVED?")+ print("=" * 70)+ needle_found = any(item.get("error_code") == "PG-5523" for item in compressed_data)+ print(f"Critical error (PG-5523) in compressed output: {needle_found}")++ # Stats+ print("\n" + "=" * 70)+ print("STATS")+ print("=" * 70)+ print(f"Items: {len(logs)} → {len(compressed_data)}")+ print(f"Chars: {len(original_json):,} → {len(result.compressed):,}")+ print(f"Reduction: {(1 - len(result.compressed) / len(original_json)) * 100:.1f}%")+++if __name__ == "__main__":+ main()diff --git a/examples/needle_in_haystack_test.py b/examples/needle_in_haystack_test.pynew file mode 100644index 000000000..fed161fe8--- /dev/null+++ b/examples/needle_in_haystack_test.py@@ -0,0 +1,299 @@+#!/usr/bin/env python3+"""+Needle-in-Haystack Test: Proving Headroom Preserves Critical Information++This test embeds a SPECIFIC answer deep in a large JSON response,+then verifies both baseline and Headroom-optimized calls can find it.++The scenario: Production incident logs with ONE critical error buried in the middle.+The question: "What caused the outage and what's the fix?"+"""++import json+import os++import httpx++ANTHROPIC_API_KEY = os.environ.get("ANTHROPIC_API_KEY")+if not ANTHROPIC_API_KEY:+ raise ValueError("ANTHROPIC_API_KEY environment variable required")++# =============================================================================+# THE NEEDLE: A critical error buried at position 67 out of 100+# =============================================================================+CRITICAL_ERROR = {+ "timestamp": "2024-12-15T03:47:23Z",+ "level": "FATAL",+ "service": "payment-gateway",+ "message": "Connection pool exhausted - max_connections=100 exceeded",+ "error_code": "PG-5523",+ "stack_trace": "ConnectionPoolError at PaymentService.processTransaction()",+ "resolution": "Increase max_connections to 500 in config/database.yml",+ "affected_transactions": 1847,+ "incident_id": "INC-2024-1215-001",+}+++# =============================================================================+# THE HAYSTACK: 99 normal log entries+# =============================================================================+def create_log_entries(n: int = 100, needle_position: int = 67) -> list:+ """Create n log entries with the critical error at needle_position."""+ logs = []+ for i in range(n):+ if i == needle_position:+ # Insert the needle+ logs.append(CRITICAL_ERROR)+ else:+ # Normal log entry+ logs.append(+ {+ "timestamp": f"2024-12-15T{(i % 24):02d}:{(i % 60):02d}:00Z",+ "level": "INFO",+ "service": ["api-gateway", "user-service", "inventory", "auth"][i % 4],+ "message": f"Request processed successfully - latency={50 + (i % 100)}ms",+ "request_id": f"req-{i:06d}",+ "status_code": 200,+ "endpoint": ["/api/users", "/api/products", "/api/orders", "/health"][i % 4],+ "metadata": {+ "region": ["us-east-1", "us-west-2", "eu-west-1"][i % 3],+ "version": "2.4.1",+ },+ }+ )+ return logs+++# =============================================================================+# THE QUESTION+# =============================================================================+QUESTION = """Based on these production logs, answer these specific questions:++1. What service caused the outage?+2. What was the exact error code?+3. What is the specific fix mentioned in the logs?+4. How many transactions were affected?++Be precise - cite the exact values from the logs."""+++def create_messages(tool_content: str) -> list:+ """Create the conversation with tool output."""+ return [+ {+ "role": "user",+ "content": "Search the production logs for the root cause of last night's outage",+ },+ {+ "role": "assistant",+ "content": [+ {"type": "text", "text": "I'll search the production logs for errors."},+ {+ "type": "tool_use",+ "id": "logs_1",+ "name": "search_logs",+ "input": {"query": "error OR fatal", "limit": 100},+ },+ ],+ },+ {+ "role": "user",+ "content": [+ {+ "type": "tool_result",+ "tool_use_id": "logs_1",+ "content": tool_content,+ }+ ],+ },+ {+ "role": "user",+ "content": QUESTION,+ },+ ]+++def make_api_call(base_url: str, messages: list, label: str) -> dict:+ """Make API call and return response + usage."""+ headers = {+ "x-api-key": ANTHROPIC_API_KEY,+ "anthropic-version": "2023-06-01",+ "content-type": "application/json",+ }++ payload = {+ "model": "claude-sonnet-4-20250514",+ "max_tokens": 1000,+ "messages": messages,+ "tools": [+ {+ "name": "search_logs",+ "description": "Search production logs",+ "input_schema": {+ "type": "object",+ "properties": {+ "query": {"type": "string"},+ "limit": {"type": "integer"},+ },+ "required": ["query"],+ },+ }+ ],+ }++ print(f"\n{'=' * 70}")+ print(f"{label}")+ print(f"{'=' * 70}")++ try:+ with httpx.Client(timeout=120.0) as client:+ response = client.post(+ f"{base_url}/v1/messages",+ headers=headers,+ json=payload,+ )++ if response.status_code != 200:+ print(f"Error: {response.status_code}")+ print(response.text[:500])+ return {"error": response.text}++ data = response.json()+ usage = data.get("usage", {})++ # Extract text response+ response_text = ""+ for block in data.get("content", []):+ if block.get("type") == "text":+ response_text += block.get("text", "")++ return {+ "input_tokens": usage.get("input_tokens", 0),+ "output_tokens": usage.get("output_tokens", 0),+ "response": response_text,+ }++ except Exception as e:+ print(f"Exception: {e}")+ return {"error": str(e)}+++def verify_answer(response: str) -> dict:+ """Check if the response contains the correct answers."""+ checks = {+ "service": "payment-gateway" in response.lower(),+ "error_code": "PG-5523" in response or "pg-5523" in response.lower(),+ "fix": "max_connections" in response.lower() or "500" in response,+ "transactions": "1847" in response or "1,847" in response,+ }+ return checks+++def main():+ print("\n" + "=" * 70)+ print("NEEDLE-IN-HAYSTACK TEST")+ print("Proving Headroom preserves critical information")+ print("=" * 70)++ # Create the test data+ logs = create_log_entries(100, needle_position=67)+ log_json = json.dumps(logs)++ print("\nTest setup:")+ print(f" - Total log entries: {len(logs)}")+ print(" - Critical error at position: 67")+ print(f" - JSON size: {len(log_json):,} characters")++ print("\nThe needle (what we're looking for):")+ print(f" - Service: {CRITICAL_ERROR['service']}")+ print(f" - Error code: {CRITICAL_ERROR['error_code']}")+ print(f" - Fix: {CRITICAL_ERROR['resolution']}")+ print(f" - Affected: {CRITICAL_ERROR['affected_transactions']} transactions")++ messages = create_messages(log_json)++ # Test 1: Baseline (direct to Anthropic)+ print("\n" + "-" * 70)+ baseline = make_api_call(+ "https://api.anthropic.com",+ messages,+ "BASELINE: Direct to Anthropic API",+ )++ if "error" not in baseline:+ print(f"\nInput tokens: {baseline['input_tokens']:,}")+ print(f"\nResponse:\n{baseline['response'][:1000]}...")+ baseline_checks = verify_answer(baseline["response"])+ print(f"\nAnswer verification: {baseline_checks}")++ # Test 2: Headroom optimized+ print("\n" + "-" * 70)+ optimized = make_api_call(+ "http://localhost:8787",+ messages,+ "HEADROOM: Through optimization proxy",+ )++ if "error" not in optimized:+ print(f"\nInput tokens: {optimized['input_tokens']:,}")+ print(f"\nResponse:\n{optimized['response'][:1000]}...")+ optimized_checks = verify_answer(optimized["response"])+ print(f"\nAnswer verification: {optimized_checks}")++ # Final comparison+ print("\n" + "=" * 70)+ print("FINAL RESULTS")+ print("=" * 70)++ if "error" not in baseline and "error" not in optimized:+ baseline_input = baseline["input_tokens"]+ optimized_input = optimized["input_tokens"]+ saved = baseline_input - optimized_input+ percent = (saved / baseline_input * 100) if baseline_input > 0 else 0++ baseline_score = sum(baseline_checks.values())+ optimized_score = sum(optimized_checks.values())++ print(f"""+TOKEN COMPARISON:+ Baseline: {baseline_input:,} tokens+ Headroom: {optimized_input:,} tokens+ Saved: {saved:,} tokens ({percent:.1f}% reduction)++ANSWER ACCURACY (4 questions):+ Baseline: {baseline_score}/4 correct+ Headroom: {optimized_score}/4 correct++VERIFICATION DETAILS:+ Baseline: {baseline_checks}+ Headroom: {optimized_checks}++CONCLUSION:+ {"PASS - Headroom found the needle!" if optimized_score >= 3 else "NEEDS REVIEW"}+ {"Answer quality maintained with " + f"{percent:.0f}% fewer tokens" if optimized_score >= baseline_score else ""}+""")++ return {+ "baseline_tokens": baseline_input,+ "optimized_tokens": optimized_input,+ "tokens_saved": saved,+ "percent_saved": percent,+ "baseline_accuracy": baseline_score,+ "optimized_accuracy": optimized_score,+ "baseline_checks": baseline_checks,+ "optimized_checks": optimized_checks,+ "baseline_response": baseline["response"],+ "optimized_response": optimized["response"],+ }++ return None+++if __name__ == "__main__":+ result = main()+ if result:+ print("\n" + "=" * 70)+ print("JSON SUMMARY:")+ print("=" * 70)+ summary = {k: v for k, v in result.items() if not k.endswith("_response")}+ print(json.dumps(summary, indent=2))diff --git a/examples/real_before_after_test.py b/examples/real_before_after_test.pynew file mode 100644index 000000000..b0e14e9fb--- /dev/null+++ b/examples/real_before_after_test.py@@ -0,0 +1,212 @@+#!/usr/bin/env python3+"""+Real before/after test - NO MARKETING, JUST FACTS.++This script makes actual API calls to demonstrate Headroom compression.+"""++import json+import os++import httpx++# API Key from environment+ANTHROPIC_API_KEY = os.environ.get("ANTHROPIC_API_KEY")+if not ANTHROPIC_API_KEY:+ raise ValueError("ANTHROPIC_API_KEY environment variable required")++# Realistic tool output: 100 search results from a code search+MOCK_TOOL_OUTPUT = json.dumps(+ [+ {+ "file": f"src/components/{['Button', 'Modal', 'Form', 'Table', 'Card'][i % 5]}.tsx",+ "line": 10 + (i * 3),+ "content": f"export function {['Button', 'Modal', 'Form', 'Table', 'Card'][i % 5]}Component{i}(props: Props) {{",+ "language": "typescript",+ "repository": "frontend-app",+ "branch": "main",+ "last_modified": "2024-12-15T10:00:00Z",+ "author": f"dev{i % 10}@company.com",+ "match_score": 0.95 - (i * 0.005),+ "context": {+ "before": ["import React from 'react';", "import { useCallback } from 'react';"],+ "after": [" return <div>...</div>;", "}"],+ },+ "metadata": {+ "size_bytes": 1500 + (i * 10),+ "encoding": "utf-8",+ "mime_type": "text/typescript",+ },+ }+ for i in range(100)+ ]+)+++# The conversation we'll send+def create_messages(tool_content: str) -> list:+ return [+ {"role": "user", "content": "Find all React components that use forms"},+ {+ "role": "assistant",+ "content": [+ {"type": "text", "text": "I'll search for React form components."},+ {+ "type": "tool_use",+ "id": "search_1",+ "name": "code_search",+ "input": {"query": "React form component", "limit": 100},+ },+ ],+ },+ {+ "role": "user",+ "content": [+ {"type": "tool_result", "tool_use_id": "search_1", "content": tool_content}+ ],+ },+ ]+++def count_tokens_anthropic(text: str) -> int:+ """Rough token estimate (actual would use anthropic tokenizer)"""+ # Claude's tokenizer is roughly 4 chars per token for JSON+ return len(text) // 4+++def make_api_call(base_url: str, messages: list, label: str) -> dict:+ """Make actual API call and return usage stats."""++ headers = {+ "x-api-key": ANTHROPIC_API_KEY,+ "anthropic-version": "2023-06-01",+ "content-type": "application/json",+ }++ payload = {+ "model": "claude-sonnet-4-20250514",+ "max_tokens": 500,+ "messages": messages,+ "tools": [+ {+ "name": "code_search",+ "description": "Search for code in the repository",+ "input_schema": {+ "type": "object",+ "properties": {"query": {"type": "string"}, "limit": {"type": "integer"}},+ "required": ["query"],+ },+ }+ ],+ }++ print(f"\n{'=' * 60}")+ print(f"{label}")+ print(f"{'=' * 60}")+ print(f"Endpoint: {base_url}")++ try:+ with httpx.Client(timeout=60.0) as client:+ response = client.post(f"{base_url}/v1/messages", headers=headers, json=payload)++ if response.status_code != 200:+ print(f"Error: {response.status_code}")+ print(response.text[:500])+ return {"error": response.text}++ data = response.json()+ usage = data.get("usage", {})++ result = {+ "input_tokens": usage.get("input_tokens", 0),+ "output_tokens": usage.get("output_tokens", 0),+ "response_preview": str(data.get("content", [{}])[0].get("text", ""))[:200],+ }++ print(f"Input tokens: {result['input_tokens']:,}")+ print(f"Output tokens: {result['output_tokens']:,}")+ print(f"Response: {result['response_preview']}...")++ return result++ except Exception as e:+ print(f"Exception: {e}")+ return {"error": str(e)}+++def main():+ print("\n" + "=" * 70)+ print("HEADROOM REAL BEFORE/AFTER TEST")+ print("NO MARKETING - JUST ACTUAL API RESULTS")+ print("=" * 70)++ # Show what we're testing+ print(f"\nTest data: {len(json.loads(MOCK_TOOL_OUTPUT))} code search results")+ print(f"Raw JSON size: {len(MOCK_TOOL_OUTPUT):,} characters")+ print(f"Estimated tokens: ~{count_tokens_anthropic(MOCK_TOOL_OUTPUT):,}")++ messages = create_messages(MOCK_TOOL_OUTPUT)++ # Test 1: Direct to Anthropic API (baseline)+ baseline = make_api_call(+ "https://api.anthropic.com", messages, "BASELINE: Direct to Anthropic API"+ )++ # Test 2: Through Headroom proxy+ optimized = make_api_call(+ "http://localhost:8787", messages, "OPTIMIZED: Through Headroom Proxy"+ )++ # Results+ print("\n" + "=" * 70)+ print("RESULTS")+ print("=" * 70)++ if "error" not in baseline and "error" not in optimized:+ baseline_input = baseline["input_tokens"]+ optimized_input = optimized["input_tokens"]+ saved = baseline_input - optimized_input+ percent = (saved / baseline_input * 100) if baseline_input > 0 else 0++ # Cost calculation (Claude Sonnet: $3/1M input, $15/1M output)+ cost_baseline = (baseline_input * 3 + baseline["output_tokens"] * 15) / 1_000_000+ cost_optimized = (optimized_input * 3 + optimized["output_tokens"] * 15) / 1_000_000+ cost_saved = cost_baseline - cost_optimized++ print(f"""+Input Tokens:+ Baseline: {baseline_input:,}+ Optimized: {optimized_input:,}+ Saved: {saved:,} ({percent:.1f}%)++Cost per request (Claude Sonnet pricing):+ Baseline: ${cost_baseline:.6f}+ Optimized: ${cost_optimized:.6f}+ Saved: ${cost_saved:.6f}++At 10,000 requests/day:+ Daily savings: ${cost_saved * 10000:.2f}+ Monthly savings: ${cost_saved * 10000 * 30:.2f}+""")++ # Return data for README+ return {+ "baseline_tokens": baseline_input,+ "optimized_tokens": optimized_input,+ "tokens_saved": saved,+ "percent_saved": percent,+ "tool_output_size": len(MOCK_TOOL_OUTPUT),+ "num_items": len(json.loads(MOCK_TOOL_OUTPUT)),+ }+ else:+ print("Test failed - check errors above")+ return None+++if __name__ == "__main__":+ result = main()+ if result:+ print("\n" + "=" * 70)+ print("JSON FOR README:")+ print("=" * 70)+ print(json.dumps(result, indent=2))diff --git a/examples/show_actual_compression.py b/examples/show_actual_compression.pynew file mode 100644index 000000000..dc8bf5a76--- /dev/null+++ b/examples/show_actual_compression.py@@ -0,0 +1,92 @@+#!/usr/bin/env python3+"""+Show the actual before/after JSON - what gets sent to the LLM.+"""++import json++from headroom.config import SmartCrusherConfig+from headroom.transforms.smart_crusher import SmartCrusher+++# Same realistic data as the API test+def create_search_results(n: int = 100) -> list:+ return [+ {+ "file": f"src/components/{['Button', 'Modal', 'Form', 'Table', 'Card'][i % 5]}.tsx",+ "line": 10 + (i * 3),+ "content": f"export function {['Button', 'Modal', 'Form', 'Table', 'Card'][i % 5]}Component{i}(props: Props) {{",+ "language": "typescript",+ "repository": "frontend-app",+ "branch": "main",+ "last_modified": "2024-12-15T10:00:00Z",+ "author": f"dev{i % 10}@company.com",+ "match_score": 0.95 - (i * 0.005),+ "context": {+ "before": ["import React from 'react';", "import { useCallback } from 'react';"],+ "after": [" return <div>...</div>;", "}"],+ },+ "metadata": {+ "size_bytes": 1500 + (i * 10),+ "encoding": "utf-8",+ "mime_type": "text/typescript",+ },+ }+ for i in range(n)+ ]+++def main():+ # Create the crusher+ config = SmartCrusherConfig()+ crusher = SmartCrusher(config)++ # Original data+ original_data = create_search_results(100)+ original_json = json.dumps(original_data, indent=2)++ print("=" * 70)+ print("BEFORE: Original Tool Output (first 2 items shown)")+ print("=" * 70)+ print(json.dumps(original_data[:2], indent=2))+ print(f"\n... plus {len(original_data) - 2} more items (100 total)")+ print(f"\nTotal characters: {len(original_json):,}")++ # Compress it using the crush method+ result = crusher.crush(+ content=json.dumps(original_data), query="Find all React form components"+ )++ compressed_content = result.compressed+ compressed_data = json.loads(compressed_content)++ print("\n" + "=" * 70)+ print("AFTER: Compressed Tool Output (all items shown)")+ print("=" * 70)+ print(json.dumps(compressed_data, indent=2))++ print("\n" + "=" * 70)+ print("COMPRESSION STATS")+ print("=" * 70)+ print(f"Items before: {len(original_data)}")+ print(f"Items after: {len(compressed_data)}")+ print(f"Characters before: {len(original_json):,}")+ print(f"Characters after: {len(compressed_content):,}")+ print(f"Reduction: {(1 - len(compressed_content) / len(original_json)) * 100:.1f}%")++ # Show what was kept and why+ print("\n" + "=" * 70)+ print("WHAT WAS KEPT AND WHY")+ print("=" * 70)++ for i, item in enumerate(compressed_data):+ file_name = item.get("file", "unknown")+ score = item.get("match_score", 0)+ reason = (+ "first" if i < 3 else ("last" if i >= len(compressed_data) - 2 else "high relevance")+ )+ print(f" {i + 1}. {file_name} (score: {score:.2f}) - {reason}")+++if __name__ == "__main__":+ main()