Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
Update memory documentation for HierarchicalMemory system
chopratejas8c88d49172b1
diff --git a/docs/memory.md b/docs/memory.mdindex 1fd0b4f2a..717cb0b84 100644--- a/docs/memory.md+++ b/docs/memory.md@@ -1,6 +1,6 @@# Memory-**Persistent memory for LLM applications.** Enable your AI to remember across conversations without carrying full history.+**Hierarchical, temporal memory for LLM applications.** Enable your AI to remember across conversations with intelligent scoping and versioning.## Why Memory?@@ -14,16 +14,29 @@ This is *temporal compression* - instead of carrying 10,000 tokens of conversati----## Quick Start+## What Makes Headroom Memory Different?++| Feature | Headroom | Letta (MemGPT) | Mem0 |+|---------|----------|----------------|------|+| **Hierarchical Scoping** | User → Session → Agent → Turn | Flat (per-agent) | Flat (per-user) |+| **Temporal Versioning** | Full supersession chains | No | No |+| **Zero-Latency Extraction** | Inline (Letta-style) | Inline | Separate call |+| **One-Liner Integration** | `with_memory(client)` | Requires agent setup | Requires separate client |+| **Pluggable Backends** | SQLite, HNSW, FTS5, any embedder | PostgreSQL | Qdrant/Chroma |+| **Semantic + Full-Text Search** | Both | Semantic only | Semantic only |+| **Memory Bubbling** | Auto-promote important memories | No | No |+| **Protocol-Based Architecture** | Yes (dependency injection) | No | No |-### Zero-Latency Memory (Recommended)+---++## Quick Start```pythonfrom openai import OpenAI-from headroom.memory import with_fast_memory+from headroom import with_memory# One line - that's it-client = with_fast_memory(OpenAI(), user_id="alice")+client = with_memory(OpenAI(), user_id="alice")# Use exactly like normalresponse = client.chat.completions.create(@@ -40,81 +53,167 @@ response = client.chat.completions.create(# → Response uses the Python preference from memory```-### How It Works+---++## How It Works```┌─────────────────────────────────────────────────────────────┐-│ with_fast_memory() │+│ with_memory() ││ │-│ 1. INJECT: Search memories → prepend to user message │+│ 1. INJECT: Semantic search → prepend to user message ││ 2. INSTRUCT: Add memory extraction instruction ││ 3. CALL: Forward to LLM ││ 4. PARSE: Extract <memory> block from response │-│ 5. STORE: Save memories with embeddings │+│ 5. STORE: Save with embeddings + vector index + FTS ││ 6. RETURN: Clean response (without memory block) ││ │└─────────────────────────────────────────────────────────────┘```-**Key insight**: Memory extraction happens *inline* as part of the LLM response. No extra API calls, no extra latency.+**Key insight**: Memory extraction happens *inline* as part of the LLM response (Letta-style). No extra API calls, no extra latency.----## Two Approaches+## Hierarchical Scoping++Memories exist at different scope levels, enabling fine-grained control:++```+USER (broadest)+ └── SESSION+ └── AGENT+ └── TURN (narrowest)+```++### Scope Levels++| Scope | Persists Across | Use Case |+|-------|-----------------|----------|+| **USER** | All sessions, all time | Long-term preferences, identity |+| **SESSION** | Current session only | Current task context |+| **AGENT** | Current agent in session | Agent-specific context |+| **TURN** | Single turn only | Ephemeral working memory |-### 1. Fast Memory (Inline Extraction)+### Example: Multi-Session Memory```python-from headroom.memory import with_fast_memory+from openai import OpenAI+from headroom import with_memory-client = with_fast_memory(+# Session 1: Morning+client1 = with_memory(OpenAI(),- user_id="alice",- db_path="memory.db", # SQLite storage- top_k=5, # Memories to inject- use_local_embeddings=True, # Local model (fast) vs OpenAI API+ user_id="bob",+ session_id="morning-session",+)+response = client1.chat.completions.create(+ model="gpt-4o",+ messages=[{"role": "user", "content": "I prefer Go for performance-critical code"}]+)+# Memory stored at USER level (persists across sessions)++# Session 2: Afternoon (different session, same user)+client2 = with_memory(+ OpenAI(),+ user_id="bob", # Same user+ session_id="afternoon-session", # Different session+)+response = client2.chat.completions.create(+ model="gpt-4o",+ messages=[{"role": "user", "content": "What language for my new microservice?"}])+# → Recalls Go preference from morning session!```-**Characteristics:**-- Zero extra latency (extraction is part of response)-- ~100 extra output tokens per response-- Smart extraction (LLM decides what's important)-- Semantic retrieval (vector similarity)+---++## Temporal Versioning (Supersession)-### 2. Background Memory (Separate Extraction)+Memories evolve over time. When facts change, Headroom creates a **supersession chain** preserving history:```python-from headroom.memory import with_memory+from headroom.memory import HierarchicalMemory, MemoryConfig-client = with_memory(- OpenAI(),+memory = await HierarchicalMemory.create()++# Original fact+orig = await memory.add(+ content="User works at Google",user_id="alice",- db_path="memory.db",+ category=MemoryCategory.FACT,)++# User changes jobs - supersede the old memory+new = await memory.supersede(+ old_memory_id=orig.id,+ new_content="User now works at Anthropic",+)++# Query current state (excludes superseded)+current = await memory.query(MemoryFilter(+ user_id="alice",+ include_superseded=False, # Default+))+# → Returns only "User now works at Anthropic"++# Query full history (includes superseded)+history = await memory.query(MemoryFilter(+ user_id="alice",+ include_superseded=True,+))+# → Returns both memories with validity timestamps++# Get the chain+chain = await memory.get_history(new.id)+# → [+# Memory(content="User works at Google", valid_until=..., is_current=False),+# Memory(content="User now works at Anthropic", valid_until=None, is_current=True),+# ]```-**Characteristics:**-- Non-blocking (extraction happens in background worker)-- Separate LLM call for extraction-- Good when you don't want to modify responses+### Why Temporal Versioning Matters++1. **Audit trail** - Know what was true at any point in time+2. **Debugging** - Understand why the LLM made certain decisions+3. **Rollback** - Restore previous state if needed+4. **Analytics** - Track how user preferences evolve++---++## Memory Categories++Memories are categorized for better organization and retrieval:++| Category | Description | Examples |+|----------|-------------|----------|+| `PREFERENCE` | Likes, dislikes, preferred approaches | "Prefers Python", "Likes dark mode" |+| `FACT` | Identity, role, constraints | "Works at fintech startup", "Senior engineer" |+| `CONTEXT` | Current goals, ongoing tasks | "Migrating to microservices", "Working on auth" |+| `ENTITY` | Information about entities | "Project Apollo uses React", "Team lead is Sarah" |+| `DECISION` | Decisions made | "Chose PostgreSQL over MySQL", "Using REST not GraphQL" |+| `INSIGHT` | Derived insights | "User tends to prefer typed languages" |---## Memory API-Both wrappers provide a `.memory` API for direct access:+The `with_memory()` wrapper provides a `.memory` API for direct access:```python-client = with_fast_memory(OpenAI(), user_id="alice")+client = with_memory(OpenAI(), user_id="alice")-# Search memories+# Search memories (semantic)results = client.memory.search("python preferences", top_k=5)-for memory, score in results:- print(f"{score:.2f}: {memory.text}")+for memory in results:+ print(f"{memory.content}")# Add manual memory-client.memory.add("User is a senior engineer", category="fact")+client.memory.add(+ "User is a senior engineer",+ category="fact",+ importance=0.9,+)# Get all memoriesall_memories = client.memory.get_all()@@ -124,110 +223,251 @@ client.memory.clear()# Get statsstats = client.memory.stats()-print(f"Total memories: {stats['total_chunks']}")+print(f"Total memories: {stats['total']}")+print(f"By category: {stats['categories']}")```----## Memory Categories+## Advanced Usage: Direct HierarchicalMemory API-Memories are categorized for better organization:+For full control, use the `HierarchicalMemory` class directly:-| Category | Description | Examples |-|----------|-------------|----------|-| `preference` | Likes, dislikes, preferred approaches | "Prefers Python", "Likes async/await" |-| `fact` | Identity, role, constraints | "Works at fintech startup", "Senior engineer" |-| `context` | Current goals, ongoing tasks | "Migrating to microservices", "Working on auth" |+```python+import asyncio+from headroom.memory import (+ HierarchicalMemory,+ MemoryConfig,+ MemoryCategory,+ EmbedderBackend,+)+from headroom.memory.ports import MemoryFilter, VectorFilter++async def main():+ # Create with custom configuration+ config = MemoryConfig(+ db_path="my_memory.db",+ embedder_backend=EmbedderBackend.LOCAL, # or OPENAI, OLLAMA+ vector_dimension=384,+ cache_max_size=2000,+ )+ memory = await HierarchicalMemory.create(config)++ # Add memory with full control+ mem = await memory.add(+ content="User prefers functional programming",+ user_id="alice",+ session_id="sess-123",+ agent_id="code-assistant",+ category=MemoryCategory.PREFERENCE,+ importance=0.9,+ entity_refs=["functional-programming", "coding-style"],+ metadata={"source": "conversation", "confidence": 0.95},+ )++ # Semantic search+ results = await memory.search(+ query="programming paradigm preferences",+ user_id="alice",+ top_k=5,+ min_similarity=0.5,+ categories=[MemoryCategory.PREFERENCE],+ )+ for r in results:+ print(f"[{r.similarity:.3f}] {r.memory.content}")++ # Full-text search+ text_results = await memory.text_search(+ query="functional",+ user_id="alice",+ )++ # Query with filters+ memories = await memory.query(MemoryFilter(+ user_id="alice",+ categories=[MemoryCategory.PREFERENCE, MemoryCategory.FACT],+ min_importance=0.7,+ limit=10,+ ))++ # Convenience methods+ await memory.remember("Likes coffee", user_id="alice", importance=0.6)+ relevant = await memory.recall("beverage preferences", user_id="alice")++asyncio.run(main())+```---## Configuration-### Storage+### Embedder Backends```python-# SQLite (default, local)-client = with_fast_memory(OpenAI(), user_id="alice", db_path="memory.db")+from headroom.memory import MemoryConfig, EmbedderBackend-# Custom path-client = with_fast_memory(OpenAI(), user_id="alice", db_path="/data/memories.db")-```+# Local embeddings (recommended - fast, free, private)+config = MemoryConfig(+ embedder_backend=EmbedderBackend.LOCAL,+ embedder_model="all-MiniLM-L6-v2", # 384 dimensions, fast+)-### Embeddings+# OpenAI embeddings (higher quality, costs money)+config = MemoryConfig(+ embedder_backend=EmbedderBackend.OPENAI,+ openai_api_key="sk-...",+ embedder_model="text-embedding-3-small",+)-```python-# Local embeddings (recommended - fast, free)-client = with_fast_memory(- OpenAI(),- user_id="alice",- use_local_embeddings=True,- embedding_model="all-MiniLM-L6-v2", # 384 dimensions+# Ollama embeddings (local server, many models)+config = MemoryConfig(+ embedder_backend=EmbedderBackend.OLLAMA,+ ollama_base_url="http://localhost:11434",+ embedder_model="nomic-embed-text",)+```-# OpenAI embeddings (higher quality, costs money)-client = with_fast_memory(- OpenAI(),- user_id="alice",- use_local_embeddings=False, # Uses text-embedding-3-small+### Storage Configuration++```python+config = MemoryConfig(+ db_path="memory.db", # SQLite database path+ vector_dimension=384, # Must match embedder output+ hnsw_ef_construction=200, # HNSW index quality (higher = better, slower)+ hnsw_m=16, # HNSW connections per node+ hnsw_ef_search=50, # HNSW search quality+ cache_enabled=True, # Enable LRU cache+ cache_max_size=1000, # Max cached memories)```-### Retrieval+### Wrapper Configuration```python-# Number of memories to inject-client = with_fast_memory(+client = with_memory(OpenAI(),user_id="alice",- top_k=10, # Inject up to 10 relevant memories+ db_path="memory.db",+ top_k=5, # Memories to inject per request+ session_id="optional-session",+ agent_id="optional-agent",+ embedder_backend=EmbedderBackend.LOCAL,)```----## Multi-User Isolation--Memories are isolated by `user_id`:+## Architecture-```python-# Alice's memories-alice_client = with_fast_memory(OpenAI(), user_id="alice")+### Protocol-Based Design-# Bob's memories (completely separate)-bob_client = with_fast_memory(OpenAI(), user_id="bob")+Headroom Memory uses **Protocol interfaces** (ports) for all components, enabling easy swapping:-# Agent memories-agent_client = with_fast_memory(OpenAI(), user_id="agent-researcher")```+┌─────────────────────────────────────────────────────────────┐+│ HierarchicalMemory │+│ (Orchestrator) │+├─────────────────────────────────────────────────────────────┤+│ │+│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │+│ │ MemoryStore │ │ VectorIndex │ │ TextIndex │ │+│ │ Protocol │ │ Protocol │ │ Protocol │ │+│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │+│ │ │ │ │+│ ┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐ │+│ │ SQLite │ │ HNSW │ │ FTS5 │ │+│ │ Adapter │ │ Adapter │ │ Adapter │ │+│ └─────────────┘ └─────────────┘ └─────────────┘ │+│ │+│ ┌─────────────┐ ┌─────────────┐ │+│ │ Embedder │ │ MemoryCache │ │+│ │ Protocol │ │ Protocol │ │+│ └──────┬──────┘ └──────┬──────┘ │+│ │ │ │+│ ┌──────▼──────┐ ┌──────▼──────┐ │+│ │Local/OpenAI/│ │ LRU Cache │ │+│ │ Ollama │ │ │ │+│ └─────────────┘ └─────────────┘ │+│ │+└─────────────────────────────────────────────────────────────┘+```++### Components++| Component | Protocol | Default Adapter | Purpose |+|-----------|----------|-----------------|---------|+| **MemoryStore** | `MemoryStore` | `SQLiteMemoryStore` | CRUD + filtering + supersession |+| **VectorIndex** | `VectorIndex` | `HNSWVectorIndex` | Semantic similarity search |+| **TextIndex** | `TextIndex` | `FTS5TextIndex` | Full-text keyword search |+| **Embedder** | `Embedder` | `LocalEmbedder` | Text → vector conversion |+| **Cache** | `MemoryCache` | `LRUMemoryCache` | Hot memory caching |----## How Memory Enables Compression+## Comparison with State of the Art-Memory is *temporal compression*. Instead of carrying full conversation history:+### vs Letta (MemGPT)-```-WITHOUT MEMORY:-Context = Turn 1 + Turn 2 + ... + Turn 50 = 10,000 tokens+**Letta** pioneered inline memory extraction. Headroom builds on this with:-WITH MEMORY:-Context = 5 relevant memories = 100 tokens-Compression ratio: 100x-```+| Aspect | Headroom | Letta |+|--------|----------|-------|+| **Scoping** | 4-level hierarchy (user/session/agent/turn) | Flat per-agent |+| **Temporal** | Full supersession chains with history | No versioning |+| **Integration** | One-liner wrapper for any client | Requires Letta agent framework |+| **Search** | Semantic + full-text | Semantic only |+| **Storage** | SQLite + HNSW (embedded) | PostgreSQL (external) |+| **Extensibility** | Protocol-based adapters | Monolithic |++**When to use Letta**: You want a full agent framework with built-in memory.+**When to use Headroom**: You want memory as a layer on your existing stack.++### vs Mem0++**Mem0** provides a managed memory service. Headroom differs:++| Aspect | Headroom | Mem0 |+|--------|----------|------|+| **Deployment** | Embedded (no server) | Managed service or self-hosted |+| **Scoping** | 4-level hierarchy | Flat per-user |+| **Temporal** | Supersession chains | No versioning |+| **Extraction** | Inline (zero latency) | Separate API call |+| **Search** | Semantic + full-text | Semantic only |+| **Cost** | Free (local embeddings) | API costs or infra costs |+| **Privacy** | All local | Data leaves your infra |++**When to use Mem0**: You want a managed service and don't mind external dependencies.+**When to use Headroom**: You want embedded memory with no external services.++### Feature Matrix++| Feature | Headroom | Letta | Mem0 |+|---------|:--------:|:-----:|:----:|+| Hierarchical scoping | ✅ | ❌ | ❌ |+| Temporal versioning | ✅ | ❌ | ❌ |+| Zero-latency extraction | ✅ | ✅ | ❌ |+| Full-text search | ✅ | ❌ | ❌ |+| Embedded (no server) | ✅ | ❌ | ❌ |+| One-liner integration | ✅ | ❌ | ❌ |+| Protocol-based extensibility | ✅ | ❌ | ❌ |+| Memory bubbling | ✅ | ❌ | ❌ |+| Local embeddings | ✅ | ❌ | ✅ |+| Managed service option | ❌ | ❌ | ✅ |++----This lets you use aggressive rolling window truncation while preserving important facts.+## Multi-User Isolation++Memories are isolated by `user_id`:```python-from headroom.memory import with_fast_memory-from headroom.transforms import RollingWindowTransform+# Alice's memories+alice_client = with_memory(OpenAI(), user_id="alice")-# Memory + aggressive truncation = best of both worlds-client = with_fast_memory(OpenAI(), user_id="alice")-transform = RollingWindowTransform(max_tokens=4000)+# Bob's memories (completely separate)+bob_client = with_memory(OpenAI(), user_id="bob")-# Old messages get truncated, but key facts live in memory-messages = transform.apply(very_long_conversation)-response = client.chat.completions.create(model="gpt-4o", messages=messages)+# Bob cannot see Alice's memories, even with the same database```---@@ -236,9 +476,10 @@ response = client.chat.completions.create(model="gpt-4o", messages=messages)| Operation | Latency | Notes ||-----------|---------|-------|-| Memory injection | <50ms | Local embeddings + vector search |-| Memory extraction | +50-100ms | Part of LLM response (inline) |-| Memory storage | <10ms | SQLite write + cache update |+| Memory injection | <50ms | Local embeddings + HNSW search |+| Memory extraction | +50-100 tokens | Part of LLM response (inline) |+| Memory storage | <10ms | SQLite + HNSW + FTS5 indexing |+| Cache hit | <1ms | LRU cache lookup |**Overhead**: ~100 extra output tokens per response for the `<memory>` block.@@ -250,41 +491,47 @@ Memory works with any OpenAI-compatible client:```pythonfrom openai import OpenAI-from anthropic import Anthropic-from groq import Groq+from headroom import with_memory# OpenAI-client = with_fast_memory(OpenAI(), user_id="alice")+client = with_memory(OpenAI(), user_id="alice")-# Anthropic (via OpenAI-compatible wrapper)-client = with_fast_memory(OpenAI(base_url="..."), user_id="alice")+# Azure OpenAI+client = with_memory(+ OpenAI(base_url="https://your-resource.openai.azure.com/..."),+ user_id="alice",+)# Groq-client = with_fast_memory(Groq(), user_id="alice")+from groq import Groq+client = with_memory(Groq(), user_id="alice")# Any OpenAI-compatible client-client = with_fast_memory(YourClient(), user_id="alice")+client = with_memory(YourClient(), user_id="alice")```----## Example: Multi-Turn Conversation+## Example: Full Conversation Flow```pythonfrom openai import OpenAI-from headroom.memory import with_fast_memory+from headroom import with_memory-client = with_fast_memory(OpenAI(), user_id="developer_jane")+client = with_memory(OpenAI(), user_id="developer_jane")# Conversation 1: User shares contextresponse = client.chat.completions.create(model="gpt-4o",messages=[{"role": "user",- "content": "I'm a Python developer at a fintech startup. We use PostgreSQL."+ "content": "I'm a Python developer at a fintech startup. We use PostgreSQL and FastAPI."}])-# Memories extracted: "Python developer", "fintech startup", "uses PostgreSQL"+# Memories extracted:+# - [FACT] Python developer at fintech startup+# - [PREFERENCE] Uses PostgreSQL for databases+# - [PREFERENCE] Uses FastAPI for web APIs# Conversation 2 (new session): User asks questionresponse = client.chat.completions.create(@@ -294,9 +541,14 @@ response = client.chat.completions.create("content": "What database should I use for my new project?"}])-# Response references PostgreSQL preference from memory-print(response.choices[0].message.content)-# → "Given your experience with PostgreSQL at your fintech company..."+# Response references PostgreSQL preference from memory:+# → "Given your experience with PostgreSQL at your fintech company,+# I'd recommend sticking with it for consistency..."++# Check stored memories+print("Stored memories:")+for m in client.memory.get_all():+ print(f" [{m.category.value}] {m.content}")```---@@ -307,26 +559,35 @@ print(response.choices[0].message.content)1. Check if the conversation has memory-worthy content (not just greetings)2. Verify the LLM is following the memory instruction-3. Check logs for parsing errors+3. Enable logging: `import logging; logging.basicConfig(level=logging.DEBUG)`### Memories not being retrieved1. Verify `user_id` matches between sessions2. Check if memories exist: `client.memory.get_all()`3. Try a more specific search query+4. Check similarity threshold### High latency-1. Switch to local embeddings: `use_local_embeddings=True`+1. Use local embeddings: `embedder_backend=EmbedderBackend.LOCAL`2. Reduce `top_k` for fewer memories to retrieve-3. Check database size and consider pruning old memories+3. Enable caching (enabled by default)++### Memory not persisting++1. Check `db_path` is the same across sessions+2. Ensure the database file is writable+3. Check for exceptions in logs---## Best Practices1. **Use consistent `user_id`** - Same ID across sessions for continuity-2. **Start with local embeddings** - Faster, free, good enough for most cases-3. **Combine with rolling window** - Memory + truncation = aggressive compression-4. **Monitor memory growth** - Periodically review and prune if needed-5. **Use categories** - Helps with debugging and selective retrieval+2. **Use session scoping** - Set `session_id` for session-specific context+3. **Start with local embeddings** - Faster, free, good enough for most cases+4. **Monitor memory growth** - Use `client.memory.stats()` to track+5. **Use importance scores** - Higher importance = more likely to be retrieved+6. **Leverage categories** - Helps with debugging and selective retrieval+7. **Consider supersession** - Use `supersede()` when facts change, not `add()`