#1 Persistent memory for AI coding agents based on real-world benchmarks
TypeScript agentmemoryagentsaiclaudeclaudecodecodexcopilotcursorgenaiharnesshermesmemoryompopenclawopencodepi
Refresh competitor star counts and add benchmark caveat (#855)
Rohit Ghumare749c2806e0ec
diff --git a/README.md b/README.mdindex 7b4e989d..a18f2a21 100644--- a/README.md+++ b/README.md@@ -277,7 +277,7 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).</tr></table>-> Embedding model: `all-MiniLM-L6-v2` (local, free, no API key). Full reports: [`benchmark/LONGMEMEVAL.md`](benchmark/LONGMEMEVAL.md), [`benchmark/QUALITY.md`](benchmark/QUALITY.md), [`benchmark/SCALE.md`](benchmark/SCALE.md). Competitor comparison: [`benchmark/COMPARISON.md`](benchmark/COMPARISON.md) — agentmemory vs mem0, Letta, Khoj, claude-mem, Hippo.+> Embedding model: `all-MiniLM-L6-v2` (local, free, no API key). Full reports: [`benchmark/LONGMEMEVAL.md`](benchmark/LONGMEMEVAL.md), [`benchmark/QUALITY.md`](benchmark/QUALITY.md), [`benchmark/SCALE.md`](benchmark/SCALE.md). Competitor comparison: [`benchmark/COMPARISON.md`](benchmark/COMPARISON.md) covering agentmemory vs mem0, Letta, Khoj, supermemory, MemPalace, Hippo.**Reproduce locally:** [`eval/README.md`](eval/README.md) — adapter-pluggable harness for LongMemEval `_s` (public 500-Q) + `coding-agent-life-v1` (in-house 15-session corpus). Grep / vector / agentmemory adapters score side-by-side, NDJSON output, published scorecards land in [`docs/benchmarks/`](docs/benchmarks/).@@ -289,17 +289,25 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).<table><tr>-<th width="20%"></th>-<th width="20%">agentmemory</th>-<th width="20%">mem0 (53K ⭐)</th>-<th width="20%">Letta / MemGPT (22K ⭐)</th>-<th width="20%">Built-in (CLAUDE.md)</th>+<th></th>+<th>agentmemory</th>+<th>mem0 (58K ⭐)</th>+<th>Letta / MemGPT (23K ⭐)</th>+<th>Khoj (35K ⭐)</th>+<th>supermemory (26K ⭐)</th>+<th>MemPalace (54K ⭐)</th>+<th>Hippo</th>+<th>Built-in (CLAUDE.md)</th></tr><tr><td><strong>Type</strong></td><td>Memory engine + MCP server</td><td>Memory layer API</td><td>Full agent runtime</td>+<td>Personal AI</td>+<td>Memory API + app</td>+<td>Vector memory (OSS)</td>+<td>Memory system</td><td>Static file</td></tr><tr>@@ -307,6 +315,10 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).<td><strong>95.2%</strong></td><td>68.5% (LoCoMo)</td><td>83.2% (LoCoMo)</td>+<td>N/A</td>+<td>Self-reported</td>+<td>~96.6% (self-reported)</td>+<td>N/A</td><td>N/A (grep)</td></tr><tr>@@ -314,6 +326,10 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).<td>12 hooks (zero manual effort)</td><td>Manual <code>add()</code> calls</td><td>Agent self-edits</td>+<td>Manual</td>+<td>API-side extraction</td>+<td>Manual</td>+<td>Manual</td><td>Manual editing</td></tr><tr>@@ -321,6 +337,10 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).<td>BM25 + Vector + Graph (RRF fusion)</td><td>Vector + Graph</td><td>Vector (archival)</td>+<td>Semantic</td>+<td>Vector + RAG</td>+<td>Vector-only</td>+<td>Decay-weighted</td><td>Loads everything into context</td></tr><tr>@@ -328,6 +348,10 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).<td>MCP + REST + leases + signals</td><td>API (no coordination)</td><td>Within Letta runtime only</td>+<td>No</td>+<td>No</td>+<td>No</td>+<td>Multi-agent shared</td><td>Per-agent files</td></tr><tr>@@ -335,6 +359,10 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).<td>None (any MCP client)</td><td>None</td><td>High (must use Letta)</td>+<td>Standalone</td>+<td>None</td>+<td>None</td>+<td>None</td><td>Per-agent format</td></tr><tr>@@ -342,6 +370,10 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).<td>None (SQLite + iii-engine)</td><td>Qdrant / pgvector</td><td>Postgres + vector DB</td>+<td>Multiple</td>+<td>Managed cloud</td>+<td>Vector store</td>+<td>None</td><td>None</td></tr><tr>@@ -349,6 +381,10 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).<td>4-tier consolidation + decay + auto-forget</td><td>Passive extraction</td><td>Agent-managed</td>+<td>Manual</td>+<td>Auto-forget</td>+<td>None</td>+<td>Decay + consolidation</td><td>Manual pruning</td></tr><tr>@@ -356,6 +392,10 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).<td>~1,900 tokens/session ($10/yr)</td><td>Varies by integration</td><td>Core memory in context</td>+<td>Varies</td>+<td>Cloud pricing</td>+<td>No token budget</td>+<td>Varies</td><td>22K+ tokens at 240 obs</td></tr><tr>@@ -363,6 +403,10 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).<td>Yes (port 3113)</td><td>Cloud dashboard</td><td>Cloud dashboard</td>+<td>Web UI</td>+<td>Cloud dashboard</td>+<td>No</td>+<td>No</td><td>No</td></tr><tr>@@ -371,9 +415,15 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).<td>Optional</td><td>Optional</td><td>Yes</td>+<td>No (cloud-only)</td>+<td>Yes</td>+<td>Yes</td>+<td>Yes</td></tr></table>+<sub>Benchmark note: only agentmemory's R@5 is our own measured result (LongMemEval-S, reproducible from <a href="benchmark/COMPARISON.md"><code>benchmark/COMPARISON.md</code></a>). The mem0 and Letta figures are their published LoCoMo numbers (a different dataset); the MemPalace and supermemory figures are vendor self-reported claims we have not independently reproduced. Shown side by side for ballpark only, not a head-to-head on identical data. Star counts are approximate and drift over time.</sub>+---<h2 id="quick-start"><picture><source media="(prefers-color-scheme: dark)" srcset="assets/tags/light/section-quickstart.svg"><img src="assets/tags/section-quickstart.svg" alt="Quick Start" height="32" /></picture></h2>diff --git a/benchmark/COMPARISON.md b/benchmark/COMPARISON.mdindex 83f7a39d..794833a0 100644--- a/benchmark/COMPARISON.md+++ b/benchmark/COMPARISON.md@@ -14,11 +14,11 @@ All numbers here come from published benchmarks or public repositories. We link|---|---|---|---|| **agentmemory** (BM25 + Vector) | LongMemEval-S | **95.2%** | `all-MiniLM-L6-v2` embeddings, no API key || agentmemory (BM25-only) | LongMemEval-S | 86.2% | Fallback when no embedding provider available |-| MemPalace | LongMemEval-S | ~96.6% | Vector-only, bigger embedding model |+| MemPalace | LongMemEval-S | ~96.6% (self-reported) | Vendor-published number we have not independently reproduced. Vector-only with a larger embedding model and no agent-integration surface (no hooks, no MCP, no multi-agent) || Letta / MemGPT | LoCoMo | 83.2% | Different benchmark (LoCoMo, not LongMemEval) || Mem0 | LoCoMo | 68.5% | Different benchmark (LoCoMo, not LongMemEval) |-**⚠️ Apples vs oranges caveat:** agentmemory and MemPalace are measured on LongMemEval-S. Letta and Mem0 publish on [LoCoMo](https://snap-stanford.github.io/LoCoMo/), a different benchmark. We're showing both so you can see the ballpark. We'd love to run all four on the same dataset — if any maintainer wants to collaborate, open an issue.+**⚠️ Apples vs oranges caveat:** only agentmemory's 95.2% is our own measured result, reproducible from the methodology below. Every other number here is the vendor's published claim, on a different benchmark or harness, that we have not independently reproduced: MemPalace reports LongMemEval-S, while Letta and Mem0 publish on [LoCoMo](https://snap-stanford.github.io/LoCoMo/). Treat them as ballpark vendor claims, not a head-to-head on identical data. We'd love to run every system on the same dataset; if any maintainer wants to collaborate, open an issue.Full agentmemory methodology: [`LONGMEMEVAL.md`](LONGMEMEVAL.md)@@ -26,26 +26,26 @@ Full agentmemory methodology: [`LONGMEMEVAL.md`](LONGMEMEVAL.md)## Feature Matrix-| Feature | agentmemory | mem0 | Letta/MemGPT | Khoj | claude-mem | Hippo |-|---|---|---|---|---|---|---|-| **GitHub stars** | Growing | 53K+ | 22K+ | 34K+ | 46K+ | Trending |-| **Type** | Memory engine + MCP server | Memory layer API | Full agent runtime | Personal AI | MCP server | Memory system |-| **Auto-capture via hooks** | ✅ 12 lifecycle hooks | ❌ Manual `add()` | ❌ Agent self-edits | ❌ Manual | ✅ Limited | ❌ Manual |-| **Search strategy** | BM25 + Vector + Graph | Vector + Graph | Vector (archival) | Semantic | FTS5 | Decay-weighted |-| **Multi-agent coordination** | ✅ Leases + signals + mesh | ❌ | Runtime-internal only | ❌ | ❌ | Multi-agent shared |-| **Framework lock-in** | None | None | High | Standalone | Claude Code | None |-| **External deps** | None | Qdrant/pgvector | Postgres + vector | Multiple | None (SQLite) | None |-| **Self-hostable** | ✅ default | Optional | Optional | ✅ | ✅ | ✅ |-| **Knowledge graph** | ✅ Entity extraction + BFS | ✅ Mem0g variant | ❌ | Doc links | ❌ | ❌ |-| **Memory decay** | ✅ Ebbinghaus + tiered | ❌ | ❌ | ❌ | ❌ | ✅ Half-lives |-| **4-tier consolidation** | ✅ Working → episodic → semantic → procedural | ❌ | OS-inspired tiers | ❌ | ❌ | Episodic + semantic |-| **Version / supersession** | ✅ Jaccard-based | Passive | ❌ | ❌ | ❌ | ❌ |-| **Real-time viewer** | ✅ Port 3113 | Cloud dashboard | Cloud dashboard | Web UI | ❌ | ❌ |-| **Privacy filtering** | ✅ Strips secrets pre-store | ❌ | ❌ | ❌ | ❌ | ❌ |-| **Obsidian export** | ✅ Built-in | ❌ | ❌ | Native format | ❌ | ❌ |-| **Cross-agent** | ✅ MCP + REST | API calls | Within runtime | Standalone | Claude-only | Multi-agent shared |-| **Audit trail** | ✅ All mutations logged | ❌ | Limited | ❌ | ❌ | ❌ |-| **Language SDKs** | Any (REST + MCP) | Python + TS | Python only | API | Any (MCP) | Node |+| Feature | agentmemory | mem0 | Letta/MemGPT | Khoj | supermemory | MemPalace | Hippo |+|---|---|---|---|---|---|---|---|+| **GitHub stars** | Growing | 58K+ | 23K+ | 35K+ | 26K+ | 54K+ | Trending |+| **Type** | Memory engine + MCP server | Memory layer API | Full agent runtime | Personal AI | Memory API + app | Benchmark-focused OSS | Memory system |+| **Auto-capture via hooks** | ✅ 12 lifecycle hooks | ❌ Manual `add()` | ❌ Agent self-edits | ❌ Manual | ❌ API-side extraction | ❌ Manual | ❌ Manual |+| **Search strategy** | BM25 + Vector + Graph | Vector + Graph | Vector (archival) | Semantic | Vector + RAG | Vector-only (large model) | Decay-weighted |+| **Multi-agent coordination** | ✅ Leases + signals + mesh | ❌ | Runtime-internal only | ❌ | ❌ | ❌ | Multi-agent shared |+| **Framework lock-in** | None | None | High | Standalone | None (drop-in wrappers) | None | None |+| **External deps** | None | Qdrant/pgvector | Postgres + vector | Multiple | Managed cloud | Vector store | None |+| **Self-hostable** | ✅ default | Optional | Optional | ✅ | ❌ Cloud-only | ✅ | ✅ |+| **Knowledge graph** | ✅ Entity extraction + BFS | ✅ Mem0g variant | ❌ | Doc links | ❌ | ❌ | ❌ |+| **Memory decay** | ✅ Ebbinghaus + tiered | ❌ | ❌ | ❌ | ✅ Auto-forget | ❌ | ✅ Half-lives |+| **4-tier consolidation** | ✅ Working → episodic → semantic → procedural | ❌ | OS-inspired tiers | ❌ | ❌ | ❌ | Episodic + semantic |+| **Version / supersession** | ✅ Jaccard-based | Passive | ❌ | ❌ | ✅ Auto-resolve | ❌ | ❌ |+| **Real-time viewer** | ✅ Port 3113 | Cloud dashboard | Cloud dashboard | Web UI | Cloud dashboard | ❌ | ❌ |+| **Privacy filtering** | ✅ Strips secrets pre-store | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |+| **Obsidian export** | ✅ Built-in | ❌ | ❌ | Native format | ❌ | ❌ | ❌ |+| **Cross-agent** | ✅ MCP + REST | API calls | Within runtime | Standalone | MCP + API | Standalone | Multi-agent shared |+| **Audit trail** | ✅ All mutations logged | ❌ | Limited | ❌ | ❌ | ❌ | ❌ |+| **Language SDKs** | Any (REST + MCP) | Python + TS | Python only | API | Python + TS | Python | Node |---@@ -59,8 +59,8 @@ The main reason to use persistent memory at all: token cost. Here's what one yea| LLM-summarized memory (extraction-based) | ~650K | ~$500 | Lossy — summarization drops detail || **agentmemory (API embeddings)** | **~170K** | **~$10** | Token-budgeted, only relevant memories injected || **agentmemory (local embeddings)** | **~170K** | **$0** | `all-MiniLM-L6-v2` runs in-process |-| claude-mem | Reports ~10x savings | — | SQLite + FTS5 + 3-layer filter |-| Mem0 | Varies by integration | — | Extraction-based, no token budget |+| supermemory | Not published | Cloud pricing | Managed API, no local token budget |+| Mem0 | Varies by integration | Varies | Extraction-based, no token budget |**agentmemory ships with a built-in token savings calculator.** Run `npx @agentmemory/agentmemory status` after a few sessions and you'll see exactly how many tokens you've saved vs. pasting the full history.@@ -97,10 +97,17 @@ This isn't a "agentmemory wins everything" page. Different tools solve different- Obsidian/Notion/Emacs integrations- Scheduled automations and research tasks-**Choose claude-mem if you want:**-- Claude Code-specific tooling with SQLite + FTS5-- Minimal install footprint-- Token compression via LLM+**Choose supermemory if you want:**+- A managed memory API with server-side auto-extraction and automatic forgetting+- Drop-in wrappers for major AI frameworks (Vercel AI, LangChain, LangGraph)+- A hosted dashboard with no infrastructure to run yourself+- RAG plus memory served from a single query++**Choose MemPalace if you want:**+- A simple, free, open-source vector memory store+- To chase its self-reported retrieval benchmark (we have not reproduced it)+- Pure retrieval over agent workflow features+- Note: no auto-capture, no MCP, no multi-agent coordination, so you wire all integration yourself**Choose Hippo if you want:**- Biologically-inspired memory model (decay, consolidation, sleep)