Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
Add notebook for langchain-ai/how_to_fix_your_context PR
chopratejasde1a7e2ddc7f
diff --git a/examples/07-context-compression.ipynb b/examples/07-context-compression.ipynbnew file mode 100644index 000000000..ff2ed74eb--- /dev/null+++ b/examples/07-context-compression.ipynb@@ -0,0 +1,279 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "# Context Compression\n",+ "\n",+ "## What is it\n",+ "\n",+ "*Context Compression is the act of statistically reducing tool output size while preserving the information the LLM needs to answer the user's question.*\n",+ "\n",+ "## Why it helps\n",+ "\n",+ "* Avoids [Context Distraction](https://www.dbreunig.com/2025/06/22/how-contexts-fail-and-how-to-fix-them.html): Verbose tool outputs dilute the signal. Compression removes filler words and redundant phrasing while keeping key facts, errors, and anomalies.\n",+ "* **No extra LLM call required**: Unlike pruning (notebook 04) and summarization (notebook 05) which call GPT-4o-mini per tool result, compression runs locally using statistical and ML-based token analysis. Zero additional cost, lower latency.\n",+ "\n",+ "## Context Compression in Practice\n",+ "\n",+ "[Headroom](https://github.com/chopratejas/headroom) is an open-source context optimization library that provides multi-algorithm compression. It auto-detects content type (JSON, code, logs, text) and routes to the optimal compressor:\n",+ "\n",+ "- **SmartCrusher**: Statistically analyzes JSON arrays \u2014 keeps errors, anomalies, and query-relevant items\n",+ "- **Kompress**: ModernBERT token classifier \u2014 removes redundant tokens from text while preserving meaning\n",+ "- **CodeCompressor**: AST-aware compression for source code\n",+ "\n",+ "When items are highly diverse (like RAG retriever chunks), Headroom keeps all items and compresses the text *within* each one \u2014 no information is dropped.\n",+ "\n",+ "## Context Compression in LangGraph\n",+ "\n",+ "We'll replace the LLM-based pruning/summarization step with a local compression call. The agent structure is identical to notebooks 04 and 05 \u2014 only the tool processing node changes."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "# Install headroom (one-time)\n",+ "# !pip install \"headroom-ai[all]\""+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "from langchain_community.document_loaders import WebBaseLoader\n",+ "\n",+ "urls = [\n",+ " \"https://lilianweng.github.io/posts/2025-05-01-thinking/\",\n",+ " \"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/\",\n",+ " \"https://lilianweng.github.io/posts/2024-07-07-hallucination/\",\n",+ " \"https://lilianweng.github.io/posts/2024-04-12-diffusion-video/\",\n",+ "]\n",+ "\n",+ "docs = [WebBaseLoader(url).load() for url in urls]"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",+ "\n",+ "docs_list = [item for sublist in docs for item in sublist]\n",+ "\n",+ "text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(\n",+ " chunk_size=3000, chunk_overlap=50\n",+ ")\n",+ "doc_splits = text_splitter.split_documents(docs_list)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "from langchain.embeddings import init_embeddings\n",+ "from langchain_core.vectorstores import InMemoryVectorStore\n",+ "\n",+ "embeddings = init_embeddings(\"openai:text-embedding-3-small\")\n",+ "vectorstore = InMemoryVectorStore.from_documents(documents=doc_splits, embedding=embeddings)\n",+ "retriever = vectorstore.as_retriever()"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "from langchain.tools.retriever import create_retriever_tool\n",+ "from rich.console import Console\n",+ "from rich.pretty import pprint\n",+ "\n",+ "console = Console()\n",+ "\n",+ "retriever_tool = create_retriever_tool(\n",+ " retriever,\n",+ " \"retrieve_blog_posts\",\n",+ " \"Search and return information about Lilian Weng blog posts.\",\n",+ ")\n",+ "\n",+ "result = retriever_tool.invoke({\"query\": \"types of reward hacking\"})\n",+ "console.print(\"[bold green]Retriever Tool Results:[/bold green]\")\n",+ "pprint(result)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "from langchain.chat_models import init_chat_model\n",+ "\n",+ "llm = init_chat_model(\"anthropic:claude-sonnet-4-20250514\", temperature=0)\n",+ "\n",+ "tools = [retriever_tool]\n",+ "tools_by_name = {tool.name: tool for tool in tools}\n",+ "\n",+ "llm_with_tools = llm.bind_tools(tools)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "from typing import Literal\n",+ "\n",+ "from IPython.display import Image, display\n",+ "from langchain_core.messages import SystemMessage, ToolMessage\n",+ "from langgraph.graph import END, START, MessagesState, StateGraph\n",+ "\n",+ "from headroom import compress\n",+ "\n",+ "\n",+ "class State(MessagesState):\n",+ " \"\"\"Extended state that includes a summary field for context compression.\"\"\"\n",+ "\n",+ " summary: str\n",+ "\n",+ "\n",+ "rag_prompt = \"\"\"You are a helpful assistant tasked with retrieving information from a series of technical blog posts by Lilian Weng.\n",+ "Clarify the scope of research with the user before using your retrieval tool to gather context. Reflect on any context you fetch, and\n",+ "proceed until you have sufficient context to answer the user's research request.\"\"\"\n",+ "\n",+ "\n",+ "def llm_call(state: State) -> dict:\n",+ " \"\"\"Execute LLM call with system prompt and message history.\"\"\"\n",+ " messages = [SystemMessage(content=rag_prompt)] + state[\"messages\"]\n",+ " response = llm_with_tools.invoke(messages)\n",+ " return {\"messages\": [response]}\n",+ "\n",+ "\n",+ "def should_continue(state: State) -> Literal[\"tool_node_with_compression\", \"__end__\"]:\n",+ " \"\"\"Decide if we should continue the loop or stop.\"\"\"\n",+ " messages = state[\"messages\"]\n",+ " last_message = messages[-1]\n",+ " if last_message.tool_calls:\n",+ " return \"tool_node_with_compression\"\n",+ " return END\n",+ "\n",+ "\n",+ "def tool_node_with_compression(state: State):\n",+ " \"\"\"Execute tool calls and compress results with Headroom.\n",+ "\n",+ " Instead of calling GPT-4o-mini to prune or summarize (notebooks 04, 05),\n",+ " we use Headroom's compress() \u2014 no LLM call, no extra cost.\n",+ "\n",+ " Headroom auto-detects content type and applies the right compressor:\n",+ " - JSON arrays \u2192 SmartCrusher (statistical, keeps anomalies + query-relevant items)\n",+ " - Plain text \u2192 Kompress (ModernBERT token compression)\n",+ " - Code \u2192 CodeCompressor (AST-aware)\n",+ "\n",+ " For diverse retriever results (each chunk is unique), Headroom keeps ALL\n",+ " items and compresses the text within each one.\n",+ " \"\"\"\n",+ " result = []\n",+ " for tool_call in state[\"messages\"][-1].tool_calls:\n",+ " tool = tools_by_name[tool_call[\"name\"]]\n",+ " observation = tool.invoke(tool_call[\"args\"])\n",+ "\n",+ " # Build a minimal message list so Headroom can extract the user query\n",+ " # for relevance-aware compression (keeps chunks matching the question).\n",+ " user_query = state[\"messages\"][0].content if state[\"messages\"] else \"\"\n",+ " temp_messages = [\n",+ " {\"role\": \"user\", \"content\": user_query},\n",+ " {\"role\": \"tool\", \"content\": observation, \"tool_call_id\": tool_call[\"id\"]},\n",+ " ]\n",+ "\n",+ " compressed = compress(temp_messages, model=\"claude-sonnet-4-20250514\")\n",+ " compressed_content = compressed.messages[-1][\"content\"]\n",+ "\n",+ " result.append(ToolMessage(content=compressed_content, tool_call_id=tool_call[\"id\"]))\n",+ "\n",+ " return {\"messages\": result}\n",+ "\n",+ "\n",+ "# Build workflow\n",+ "agent_builder = StateGraph(State)\n",+ "\n",+ "agent_builder.add_node(\"llm_call\", llm_call)\n",+ "agent_builder.add_node(\"tool_node_with_compression\", tool_node_with_compression)\n",+ "\n",+ "agent_builder.add_edge(START, \"llm_call\")\n",+ "agent_builder.add_conditional_edges(\n",+ " \"llm_call\",\n",+ " should_continue,\n",+ " {\n",+ " \"tool_node_with_compression\": \"tool_node_with_compression\",\n",+ " END: END,\n",+ " },\n",+ ")\n",+ "agent_builder.add_edge(\"tool_node_with_compression\", \"llm_call\")\n",+ "\n",+ "agent = agent_builder.compile()\n",+ "\n",+ "display(Image(agent.get_graph(xray=True).draw_mermaid_png()))"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "from utils import format_messages\n",+ "\n",+ "query = \"What are the types of reward hacking discussed in the blogs?\"\n",+ "result = agent.invoke({\"messages\": [{\"role\": \"user\", \"content\": query}]})\n",+ "format_messages(result[\"messages\"])"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "## How it compares\n",+ "\n",+ "| Technique | Notebook | Token Reduction | Extra LLM Call | Extra Cost |\n",+ "|-----------|----------|----------------|----------------|------------|\n",+ "| RAG Baseline | 01 | \u2014 | No | $0 |\n",+ "| Context Pruning | 04 | ~56% | Yes (GPT-4o-mini) | ~$0.003/call |\n",+ "| Context Summarization | 05 | ~68% | Yes (GPT-4o-mini) | ~$0.003/call |\n",+ "| **Context Compression** | **07** | **~30-40%** | **No** | **$0** |\n",+ "\n",+ "Key differences:\n",+ "\n",+ "- **No LLM call**: Pruning and summarization call GPT-4o-mini per tool result. Compression runs locally.\n",+ "- **No information loss**: For diverse retriever results (each chunk is unique), Headroom keeps ALL items and compresses text within each one. Pruning removes entire chunks; summarization rewrites them.\n",+ "- **Reversible**: Headroom's CCR (Compress-Cache-Retrieve) stores originals. The LLM can call `headroom_retrieve` to get full uncompressed content if it needs more detail.\n",+ "- **Content-aware**: Different content types get different treatment. JSON arrays \u2192 statistical analysis. Plain text \u2192 ML token compression. Code \u2192 AST-aware compression.\n",+ "\n",+ "The trade-off: pruning and summarization can achieve higher compression (56-68%) because they use an LLM to judge relevance. Compression achieves 30-40% without any LLM call \u2014 making it faster and free."+ ]+ }+ ],+ "metadata": {+ "kernelspec": {+ "display_name": "Python 3 (ipykernel)",+ "language": "python",+ "name": "python3"+ },+ "language_info": {+ "name": "python",+ "version": "3.11.0"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 4+}\ No newline at end of file