Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

Update memory documentation for HierarchicalMemory system

chopratejas8c88d49172b1
diff --git a/docs/memory.md b/docs/memory.md
index 1fd0b4f2a..717cb0b84 100644
--- a/docs/memory.md
+++ b/docs/memory.md
@@ -1,6 +1,6 @@
# Memory
-**Persistent memory for LLM applications.** Enable your AI to remember across conversations without carrying full history.
+**Hierarchical, temporal memory for LLM applications.** Enable your AI to remember across conversations with intelligent scoping and versioning.
## Why Memory?
@@ -14,16 +14,29 @@ This is *temporal compression* - instead of carrying 10,000 tokens of conversati
---
-## Quick Start
+## What Makes Headroom Memory Different?
+
+| Feature | Headroom | Letta (MemGPT) | Mem0 |
+|---------|----------|----------------|------|
+| **Hierarchical Scoping** | User → Session → Agent → Turn | Flat (per-agent) | Flat (per-user) |
+| **Temporal Versioning** | Full supersession chains | No | No |
+| **Zero-Latency Extraction** | Inline (Letta-style) | Inline | Separate call |
+| **One-Liner Integration** | `with_memory(client)` | Requires agent setup | Requires separate client |
+| **Pluggable Backends** | SQLite, HNSW, FTS5, any embedder | PostgreSQL | Qdrant/Chroma |
+| **Semantic + Full-Text Search** | Both | Semantic only | Semantic only |
+| **Memory Bubbling** | Auto-promote important memories | No | No |
+| **Protocol-Based Architecture** | Yes (dependency injection) | No | No |
-### Zero-Latency Memory (Recommended)
+---
+
+## Quick Start
```python
from openai import OpenAI
-from headroom.memory import with_fast_memory
+from headroom import with_memory
# One line - that's it
-client = with_fast_memory(OpenAI(), user_id="alice")
+client = with_memory(OpenAI(), user_id="alice")
# Use exactly like normal
response = client.chat.completions.create(
@@ -40,81 +53,167 @@ response = client.chat.completions.create(
# → Response uses the Python preference from memory
```
-### How It Works
+---
+
+## How It Works
```
┌─────────────────────────────────────────────────────────────┐
-│ with_fast_memory() │
+│ with_memory() │
│ │
-│ 1. INJECT: Search memories → prepend to user message │
+│ 1. INJECT: Semantic search → prepend to user message │
│ 2. INSTRUCT: Add memory extraction instruction │
│ 3. CALL: Forward to LLM │
│ 4. PARSE: Extract <memory> block from response │
-│ 5. STORE: Save memories with embeddings │
+│ 5. STORE: Save with embeddings + vector index + FTS │
│ 6. RETURN: Clean response (without memory block) │
│ │
└─────────────────────────────────────────────────────────────┘
```
-**Key insight**: Memory extraction happens *inline* as part of the LLM response. No extra API calls, no extra latency.
+**Key insight**: Memory extraction happens *inline* as part of the LLM response (Letta-style). No extra API calls, no extra latency.
---
-## Two Approaches
+## Hierarchical Scoping
+
+Memories exist at different scope levels, enabling fine-grained control:
+
+```
+USER (broadest)
+ └── SESSION
+ └── AGENT
+ └── TURN (narrowest)
+```
+
+### Scope Levels
+
+| Scope | Persists Across | Use Case |
+|-------|-----------------|----------|
+| **USER** | All sessions, all time | Long-term preferences, identity |
+| **SESSION** | Current session only | Current task context |
+| **AGENT** | Current agent in session | Agent-specific context |
+| **TURN** | Single turn only | Ephemeral working memory |
-### 1. Fast Memory (Inline Extraction)
+### Example: Multi-Session Memory
```python
-from headroom.memory import with_fast_memory
+from openai import OpenAI
+from headroom import with_memory
-client = with_fast_memory(
+# Session 1: Morning
+client1 = with_memory(
OpenAI(),
- user_id="alice",
- db_path="memory.db", # SQLite storage
- top_k=5, # Memories to inject
- use_local_embeddings=True, # Local model (fast) vs OpenAI API
+ user_id="bob",
+ session_id="morning-session",
+)
+response = client1.chat.completions.create(
+ model="gpt-4o",
+ messages=[{"role": "user", "content": "I prefer Go for performance-critical code"}]
+)
+# Memory stored at USER level (persists across sessions)
+
+# Session 2: Afternoon (different session, same user)
+client2 = with_memory(
+ OpenAI(),
+ user_id="bob", # Same user
+ session_id="afternoon-session", # Different session
+)
+response = client2.chat.completions.create(
+ model="gpt-4o",
+ messages=[{"role": "user", "content": "What language for my new microservice?"}]
)
+# → Recalls Go preference from morning session!
```
-**Characteristics:**
-- Zero extra latency (extraction is part of response)
-- ~100 extra output tokens per response
-- Smart extraction (LLM decides what's important)
-- Semantic retrieval (vector similarity)
+---
+
+## Temporal Versioning (Supersession)
-### 2. Background Memory (Separate Extraction)
+Memories evolve over time. When facts change, Headroom creates a **supersession chain** preserving history:
```python
-from headroom.memory import with_memory
+from headroom.memory import HierarchicalMemory, MemoryConfig
-client = with_memory(
- OpenAI(),
+memory = await HierarchicalMemory.create()
+
+# Original fact
+orig = await memory.add(
+ content="User works at Google",
user_id="alice",
- db_path="memory.db",
+ category=MemoryCategory.FACT,
)
+
+# User changes jobs - supersede the old memory
+new = await memory.supersede(
+ old_memory_id=orig.id,
+ new_content="User now works at Anthropic",
+)
+
+# Query current state (excludes superseded)
+current = await memory.query(MemoryFilter(
+ user_id="alice",
+ include_superseded=False, # Default
+))
+# → Returns only "User now works at Anthropic"
+
+# Query full history (includes superseded)
+history = await memory.query(MemoryFilter(
+ user_id="alice",
+ include_superseded=True,
+))
+# → Returns both memories with validity timestamps
+
+# Get the chain
+chain = await memory.get_history(new.id)
+# → [
+# Memory(content="User works at Google", valid_until=..., is_current=False),
+# Memory(content="User now works at Anthropic", valid_until=None, is_current=True),
+# ]
```
-**Characteristics:**
-- Non-blocking (extraction happens in background worker)
-- Separate LLM call for extraction
-- Good when you don't want to modify responses
+### Why Temporal Versioning Matters
+
+1. **Audit trail** - Know what was true at any point in time
+2. **Debugging** - Understand why the LLM made certain decisions
+3. **Rollback** - Restore previous state if needed
+4. **Analytics** - Track how user preferences evolve
+
+---
+
+## Memory Categories
+
+Memories are categorized for better organization and retrieval:
+
+| Category | Description | Examples |
+|----------|-------------|----------|
+| `PREFERENCE` | Likes, dislikes, preferred approaches | "Prefers Python", "Likes dark mode" |
+| `FACT` | Identity, role, constraints | "Works at fintech startup", "Senior engineer" |
+| `CONTEXT` | Current goals, ongoing tasks | "Migrating to microservices", "Working on auth" |
+| `ENTITY` | Information about entities | "Project Apollo uses React", "Team lead is Sarah" |
+| `DECISION` | Decisions made | "Chose PostgreSQL over MySQL", "Using REST not GraphQL" |
+| `INSIGHT` | Derived insights | "User tends to prefer typed languages" |
---
## Memory API
-Both wrappers provide a `.memory` API for direct access:
+The `with_memory()` wrapper provides a `.memory` API for direct access:
```python
-client = with_fast_memory(OpenAI(), user_id="alice")
+client = with_memory(OpenAI(), user_id="alice")
-# Search memories
+# Search memories (semantic)
results = client.memory.search("python preferences", top_k=5)
-for memory, score in results:
- print(f"{score:.2f}: {memory.text}")
+for memory in results:
+ print(f"{memory.content}")
# Add manual memory
-client.memory.add("User is a senior engineer", category="fact")
+client.memory.add(
+ "User is a senior engineer",
+ category="fact",
+ importance=0.9,
+)
# Get all memories
all_memories = client.memory.get_all()
@@ -124,110 +223,251 @@ client.memory.clear()
# Get stats
stats = client.memory.stats()
-print(f"Total memories: {stats['total_chunks']}")
+print(f"Total memories: {stats['total']}")
+print(f"By category: {stats['categories']}")
```
---
-## Memory Categories
+## Advanced Usage: Direct HierarchicalMemory API
-Memories are categorized for better organization:
+For full control, use the `HierarchicalMemory` class directly:
-| Category | Description | Examples |
-|----------|-------------|----------|
-| `preference` | Likes, dislikes, preferred approaches | "Prefers Python", "Likes async/await" |
-| `fact` | Identity, role, constraints | "Works at fintech startup", "Senior engineer" |
-| `context` | Current goals, ongoing tasks | "Migrating to microservices", "Working on auth" |
+```python
+import asyncio
+from headroom.memory import (
+ HierarchicalMemory,
+ MemoryConfig,
+ MemoryCategory,
+ EmbedderBackend,
+)
+from headroom.memory.ports import MemoryFilter, VectorFilter
+
+async def main():
+ # Create with custom configuration
+ config = MemoryConfig(
+ db_path="my_memory.db",
+ embedder_backend=EmbedderBackend.LOCAL, # or OPENAI, OLLAMA
+ vector_dimension=384,
+ cache_max_size=2000,
+ )
+ memory = await HierarchicalMemory.create(config)
+
+ # Add memory with full control
+ mem = await memory.add(
+ content="User prefers functional programming",
+ user_id="alice",
+ session_id="sess-123",
+ agent_id="code-assistant",
+ category=MemoryCategory.PREFERENCE,
+ importance=0.9,
+ entity_refs=["functional-programming", "coding-style"],
+ metadata={"source": "conversation", "confidence": 0.95},
+ )
+
+ # Semantic search
+ results = await memory.search(
+ query="programming paradigm preferences",
+ user_id="alice",
+ top_k=5,
+ min_similarity=0.5,
+ categories=[MemoryCategory.PREFERENCE],
+ )
+ for r in results:
+ print(f"[{r.similarity:.3f}] {r.memory.content}")
+
+ # Full-text search
+ text_results = await memory.text_search(
+ query="functional",
+ user_id="alice",
+ )
+
+ # Query with filters
+ memories = await memory.query(MemoryFilter(
+ user_id="alice",
+ categories=[MemoryCategory.PREFERENCE, MemoryCategory.FACT],
+ min_importance=0.7,
+ limit=10,
+ ))
+
+ # Convenience methods
+ await memory.remember("Likes coffee", user_id="alice", importance=0.6)
+ relevant = await memory.recall("beverage preferences", user_id="alice")
+
+asyncio.run(main())
+```
---
## Configuration
-### Storage
+### Embedder Backends
```python
-# SQLite (default, local)
-client = with_fast_memory(OpenAI(), user_id="alice", db_path="memory.db")
+from headroom.memory import MemoryConfig, EmbedderBackend
-# Custom path
-client = with_fast_memory(OpenAI(), user_id="alice", db_path="/data/memories.db")
-```
+# Local embeddings (recommended - fast, free, private)
+config = MemoryConfig(
+ embedder_backend=EmbedderBackend.LOCAL,
+ embedder_model="all-MiniLM-L6-v2", # 384 dimensions, fast
+)
-### Embeddings
+# OpenAI embeddings (higher quality, costs money)
+config = MemoryConfig(
+ embedder_backend=EmbedderBackend.OPENAI,
+ openai_api_key="sk-...",
+ embedder_model="text-embedding-3-small",
+)
-```python
-# Local embeddings (recommended - fast, free)
-client = with_fast_memory(
- OpenAI(),
- user_id="alice",
- use_local_embeddings=True,
- embedding_model="all-MiniLM-L6-v2", # 384 dimensions
+# Ollama embeddings (local server, many models)
+config = MemoryConfig(
+ embedder_backend=EmbedderBackend.OLLAMA,
+ ollama_base_url="http://localhost:11434",
+ embedder_model="nomic-embed-text",
)
+```
-# OpenAI embeddings (higher quality, costs money)
-client = with_fast_memory(
- OpenAI(),
- user_id="alice",
- use_local_embeddings=False, # Uses text-embedding-3-small
+### Storage Configuration
+
+```python
+config = MemoryConfig(
+ db_path="memory.db", # SQLite database path
+ vector_dimension=384, # Must match embedder output
+ hnsw_ef_construction=200, # HNSW index quality (higher = better, slower)
+ hnsw_m=16, # HNSW connections per node
+ hnsw_ef_search=50, # HNSW search quality
+ cache_enabled=True, # Enable LRU cache
+ cache_max_size=1000, # Max cached memories
)
```
-### Retrieval
+### Wrapper Configuration
```python
-# Number of memories to inject
-client = with_fast_memory(
+client = with_memory(
OpenAI(),
user_id="alice",
- top_k=10, # Inject up to 10 relevant memories
+ db_path="memory.db",
+ top_k=5, # Memories to inject per request
+ session_id="optional-session",
+ agent_id="optional-agent",
+ embedder_backend=EmbedderBackend.LOCAL,
)
```
---
-## Multi-User Isolation
-
-Memories are isolated by `user_id`:
+## Architecture
-```python
-# Alice's memories
-alice_client = with_fast_memory(OpenAI(), user_id="alice")
+### Protocol-Based Design
-# Bob's memories (completely separate)
-bob_client = with_fast_memory(OpenAI(), user_id="bob")
+Headroom Memory uses **Protocol interfaces** (ports) for all components, enabling easy swapping:
-# Agent memories
-agent_client = with_fast_memory(OpenAI(), user_id="agent-researcher")
```
+┌─────────────────────────────────────────────────────────────┐
+│ HierarchicalMemory │
+│ (Orchestrator) │
+├─────────────────────────────────────────────────────────────┤
+│ │
+│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
+│ │ MemoryStore │ │ VectorIndex │ │ TextIndex │ │
+│ │ Protocol │ │ Protocol │ │ Protocol │ │
+│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
+│ │ │ │ │
+│ ┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐ │
+│ │ SQLite │ │ HNSW │ │ FTS5 │ │
+│ │ Adapter │ │ Adapter │ │ Adapter │ │
+│ └─────────────┘ └─────────────┘ └─────────────┘ │
+│ │
+│ ┌─────────────┐ ┌─────────────┐ │
+│ │ Embedder │ │ MemoryCache │ │
+│ │ Protocol │ │ Protocol │ │
+│ └──────┬──────┘ └──────┬──────┘ │
+│ │ │ │
+│ ┌──────▼──────┐ ┌──────▼──────┐ │
+│ │Local/OpenAI/│ │ LRU Cache │ │
+│ │ Ollama │ │ │ │
+│ └─────────────┘ └─────────────┘ │
+│ │
+└─────────────────────────────────────────────────────────────┘
+```
+
+### Components
+
+| Component | Protocol | Default Adapter | Purpose |
+|-----------|----------|-----------------|---------|
+| **MemoryStore** | `MemoryStore` | `SQLiteMemoryStore` | CRUD + filtering + supersession |
+| **VectorIndex** | `VectorIndex` | `HNSWVectorIndex` | Semantic similarity search |
+| **TextIndex** | `TextIndex` | `FTS5TextIndex` | Full-text keyword search |
+| **Embedder** | `Embedder` | `LocalEmbedder` | Text → vector conversion |
+| **Cache** | `MemoryCache` | `LRUMemoryCache` | Hot memory caching |
---
-## How Memory Enables Compression
+## Comparison with State of the Art
-Memory is *temporal compression*. Instead of carrying full conversation history:
+### vs Letta (MemGPT)
-```
-WITHOUT MEMORY:
-Context = Turn 1 + Turn 2 + ... + Turn 50 = 10,000 tokens
+**Letta** pioneered inline memory extraction. Headroom builds on this with:
-WITH MEMORY:
-Context = 5 relevant memories = 100 tokens
-Compression ratio: 100x
-```
+| Aspect | Headroom | Letta |
+|--------|----------|-------|
+| **Scoping** | 4-level hierarchy (user/session/agent/turn) | Flat per-agent |
+| **Temporal** | Full supersession chains with history | No versioning |
+| **Integration** | One-liner wrapper for any client | Requires Letta agent framework |
+| **Search** | Semantic + full-text | Semantic only |
+| **Storage** | SQLite + HNSW (embedded) | PostgreSQL (external) |
+| **Extensibility** | Protocol-based adapters | Monolithic |
+
+**When to use Letta**: You want a full agent framework with built-in memory.
+**When to use Headroom**: You want memory as a layer on your existing stack.
+
+### vs Mem0
+
+**Mem0** provides a managed memory service. Headroom differs:
+
+| Aspect | Headroom | Mem0 |
+|--------|----------|------|
+| **Deployment** | Embedded (no server) | Managed service or self-hosted |
+| **Scoping** | 4-level hierarchy | Flat per-user |
+| **Temporal** | Supersession chains | No versioning |
+| **Extraction** | Inline (zero latency) | Separate API call |
+| **Search** | Semantic + full-text | Semantic only |
+| **Cost** | Free (local embeddings) | API costs or infra costs |
+| **Privacy** | All local | Data leaves your infra |
+
+**When to use Mem0**: You want a managed service and don't mind external dependencies.
+**When to use Headroom**: You want embedded memory with no external services.
+
+### Feature Matrix
+
+| Feature | Headroom | Letta | Mem0 |
+|---------|:--------:|:-----:|:----:|
+| Hierarchical scoping | ✅ | ❌ | ❌ |
+| Temporal versioning | ✅ | ❌ | ❌ |
+| Zero-latency extraction | ✅ | ✅ | ❌ |
+| Full-text search | ✅ | ❌ | ❌ |
+| Embedded (no server) | ✅ | ❌ | ❌ |
+| One-liner integration | ✅ | ❌ | ❌ |
+| Protocol-based extensibility | ✅ | ❌ | ❌ |
+| Memory bubbling | ✅ | ❌ | ❌ |
+| Local embeddings | ✅ | ❌ | ✅ |
+| Managed service option | ❌ | ❌ | ✅ |
+
+---
-This lets you use aggressive rolling window truncation while preserving important facts.
+## Multi-User Isolation
+
+Memories are isolated by `user_id`:
```python
-from headroom.memory import with_fast_memory
-from headroom.transforms import RollingWindowTransform
+# Alice's memories
+alice_client = with_memory(OpenAI(), user_id="alice")
-# Memory + aggressive truncation = best of both worlds
-client = with_fast_memory(OpenAI(), user_id="alice")
-transform = RollingWindowTransform(max_tokens=4000)
+# Bob's memories (completely separate)
+bob_client = with_memory(OpenAI(), user_id="bob")
-# Old messages get truncated, but key facts live in memory
-messages = transform.apply(very_long_conversation)
-response = client.chat.completions.create(model="gpt-4o", messages=messages)
+# Bob cannot see Alice's memories, even with the same database
```
---
@@ -236,9 +476,10 @@ response = client.chat.completions.create(model="gpt-4o", messages=messages)
| Operation | Latency | Notes |
|-----------|---------|-------|
-| Memory injection | <50ms | Local embeddings + vector search |
-| Memory extraction | +50-100ms | Part of LLM response (inline) |
-| Memory storage | <10ms | SQLite write + cache update |
+| Memory injection | <50ms | Local embeddings + HNSW search |
+| Memory extraction | +50-100 tokens | Part of LLM response (inline) |
+| Memory storage | <10ms | SQLite + HNSW + FTS5 indexing |
+| Cache hit | <1ms | LRU cache lookup |
**Overhead**: ~100 extra output tokens per response for the `<memory>` block.
@@ -250,41 +491,47 @@ Memory works with any OpenAI-compatible client:
```python
from openai import OpenAI
-from anthropic import Anthropic
-from groq import Groq
+from headroom import with_memory
# OpenAI
-client = with_fast_memory(OpenAI(), user_id="alice")
+client = with_memory(OpenAI(), user_id="alice")
-# Anthropic (via OpenAI-compatible wrapper)
-client = with_fast_memory(OpenAI(base_url="..."), user_id="alice")
+# Azure OpenAI
+client = with_memory(
+ OpenAI(base_url="https://your-resource.openai.azure.com/..."),
+ user_id="alice",
+)
# Groq
-client = with_fast_memory(Groq(), user_id="alice")
+from groq import Groq
+client = with_memory(Groq(), user_id="alice")
# Any OpenAI-compatible client
-client = with_fast_memory(YourClient(), user_id="alice")
+client = with_memory(YourClient(), user_id="alice")
```
---
-## Example: Multi-Turn Conversation
+## Example: Full Conversation Flow
```python
from openai import OpenAI
-from headroom.memory import with_fast_memory
+from headroom import with_memory
-client = with_fast_memory(OpenAI(), user_id="developer_jane")
+client = with_memory(OpenAI(), user_id="developer_jane")
# Conversation 1: User shares context
response = client.chat.completions.create(
model="gpt-4o",
messages=[{
"role": "user",
- "content": "I'm a Python developer at a fintech startup. We use PostgreSQL."
+ "content": "I'm a Python developer at a fintech startup. We use PostgreSQL and FastAPI."
}]
)
-# Memories extracted: "Python developer", "fintech startup", "uses PostgreSQL"
+# Memories extracted:
+# - [FACT] Python developer at fintech startup
+# - [PREFERENCE] Uses PostgreSQL for databases
+# - [PREFERENCE] Uses FastAPI for web APIs
# Conversation 2 (new session): User asks question
response = client.chat.completions.create(
@@ -294,9 +541,14 @@ response = client.chat.completions.create(
"content": "What database should I use for my new project?"
}]
)
-# Response references PostgreSQL preference from memory
-print(response.choices[0].message.content)
-# → "Given your experience with PostgreSQL at your fintech company..."
+# Response references PostgreSQL preference from memory:
+# → "Given your experience with PostgreSQL at your fintech company,
+# I'd recommend sticking with it for consistency..."
+
+# Check stored memories
+print("Stored memories:")
+for m in client.memory.get_all():
+ print(f" [{m.category.value}] {m.content}")
```
---
@@ -307,26 +559,35 @@ print(response.choices[0].message.content)
1. Check if the conversation has memory-worthy content (not just greetings)
2. Verify the LLM is following the memory instruction
-3. Check logs for parsing errors
+3. Enable logging: `import logging; logging.basicConfig(level=logging.DEBUG)`
### Memories not being retrieved
1. Verify `user_id` matches between sessions
2. Check if memories exist: `client.memory.get_all()`
3. Try a more specific search query
+4. Check similarity threshold
### High latency
-1. Switch to local embeddings: `use_local_embeddings=True`
+1. Use local embeddings: `embedder_backend=EmbedderBackend.LOCAL`
2. Reduce `top_k` for fewer memories to retrieve
-3. Check database size and consider pruning old memories
+3. Enable caching (enabled by default)
+
+### Memory not persisting
+
+1. Check `db_path` is the same across sessions
+2. Ensure the database file is writable
+3. Check for exceptions in logs
---
## Best Practices
1. **Use consistent `user_id`** - Same ID across sessions for continuity
-2. **Start with local embeddings** - Faster, free, good enough for most cases
-3. **Combine with rolling window** - Memory + truncation = aggressive compression
-4. **Monitor memory growth** - Periodically review and prune if needed
-5. **Use categories** - Helps with debugging and selective retrieval
+2. **Use session scoping** - Set `session_id` for session-specific context
+3. **Start with local embeddings** - Faster, free, good enough for most cases
+4. **Monitor memory growth** - Use `client.memory.stats()` to track
+5. **Use importance scores** - Higher importance = more likely to be retrieved
+6. **Leverage categories** - Helps with debugging and selective retrieval
+7. **Consider supersession** - Use `supersede()` when facts change, not `add()`