Skip to content

#1 Persistent memory for AI coding agents based on real-world benchmarks

Refresh competitor star counts and add benchmark caveat (#855)

Rohit Ghumare749c2806e0ec
diff --git a/README.md b/README.md
index 7b4e989d..a18f2a21 100644
--- a/README.md
+++ b/README.md
@@ -277,7 +277,7 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).
</tr>
</table>
-> Embedding model: `all-MiniLM-L6-v2` (local, free, no API key). Full reports: [`benchmark/LONGMEMEVAL.md`](benchmark/LONGMEMEVAL.md), [`benchmark/QUALITY.md`](benchmark/QUALITY.md), [`benchmark/SCALE.md`](benchmark/SCALE.md). Competitor comparison: [`benchmark/COMPARISON.md`](benchmark/COMPARISON.md) — agentmemory vs mem0, Letta, Khoj, claude-mem, Hippo.
+> Embedding model: `all-MiniLM-L6-v2` (local, free, no API key). Full reports: [`benchmark/LONGMEMEVAL.md`](benchmark/LONGMEMEVAL.md), [`benchmark/QUALITY.md`](benchmark/QUALITY.md), [`benchmark/SCALE.md`](benchmark/SCALE.md). Competitor comparison: [`benchmark/COMPARISON.md`](benchmark/COMPARISON.md) covering agentmemory vs mem0, Letta, Khoj, supermemory, MemPalace, Hippo.
**Reproduce locally:** [`eval/README.md`](eval/README.md) — adapter-pluggable harness for LongMemEval `_s` (public 500-Q) + `coding-agent-life-v1` (in-house 15-session corpus). Grep / vector / agentmemory adapters score side-by-side, NDJSON output, published scorecards land in [`docs/benchmarks/`](docs/benchmarks/).
@@ -289,17 +289,25 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).
<table>
<tr>
-<th width="20%"></th>
-<th width="20%">agentmemory</th>
-<th width="20%">mem0 (53K ⭐)</th>
-<th width="20%">Letta / MemGPT (22K ⭐)</th>
-<th width="20%">Built-in (CLAUDE.md)</th>
+<th></th>
+<th>agentmemory</th>
+<th>mem0 (58K ⭐)</th>
+<th>Letta / MemGPT (23K ⭐)</th>
+<th>Khoj (35K ⭐)</th>
+<th>supermemory (26K ⭐)</th>
+<th>MemPalace (54K ⭐)</th>
+<th>Hippo</th>
+<th>Built-in (CLAUDE.md)</th>
</tr>
<tr>
<td><strong>Type</strong></td>
<td>Memory engine + MCP server</td>
<td>Memory layer API</td>
<td>Full agent runtime</td>
+<td>Personal AI</td>
+<td>Memory API + app</td>
+<td>Vector memory (OSS)</td>
+<td>Memory system</td>
<td>Static file</td>
</tr>
<tr>
@@ -307,6 +315,10 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).
<td><strong>95.2%</strong></td>
<td>68.5% (LoCoMo)</td>
<td>83.2% (LoCoMo)</td>
+<td>N/A</td>
+<td>Self-reported</td>
+<td>~96.6% (self-reported)</td>
+<td>N/A</td>
<td>N/A (grep)</td>
</tr>
<tr>
@@ -314,6 +326,10 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).
<td>12 hooks (zero manual effort)</td>
<td>Manual <code>add()</code> calls</td>
<td>Agent self-edits</td>
+<td>Manual</td>
+<td>API-side extraction</td>
+<td>Manual</td>
+<td>Manual</td>
<td>Manual editing</td>
</tr>
<tr>
@@ -321,6 +337,10 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).
<td>BM25 + Vector + Graph (RRF fusion)</td>
<td>Vector + Graph</td>
<td>Vector (archival)</td>
+<td>Semantic</td>
+<td>Vector + RAG</td>
+<td>Vector-only</td>
+<td>Decay-weighted</td>
<td>Loads everything into context</td>
</tr>
<tr>
@@ -328,6 +348,10 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).
<td>MCP + REST + leases + signals</td>
<td>API (no coordination)</td>
<td>Within Letta runtime only</td>
+<td>No</td>
+<td>No</td>
+<td>No</td>
+<td>Multi-agent shared</td>
<td>Per-agent files</td>
</tr>
<tr>
@@ -335,6 +359,10 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).
<td>None (any MCP client)</td>
<td>None</td>
<td>High (must use Letta)</td>
+<td>Standalone</td>
+<td>None</td>
+<td>None</td>
+<td>None</td>
<td>Per-agent format</td>
</tr>
<tr>
@@ -342,6 +370,10 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).
<td>None (SQLite + iii-engine)</td>
<td>Qdrant / pgvector</td>
<td>Postgres + vector DB</td>
+<td>Multiple</td>
+<td>Managed cloud</td>
+<td>Vector store</td>
+<td>None</td>
<td>None</td>
</tr>
<tr>
@@ -349,6 +381,10 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).
<td>4-tier consolidation + decay + auto-forget</td>
<td>Passive extraction</td>
<td>Agent-managed</td>
+<td>Manual</td>
+<td>Auto-forget</td>
+<td>None</td>
+<td>Decay + consolidation</td>
<td>Manual pruning</td>
</tr>
<tr>
@@ -356,6 +392,10 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).
<td>~1,900 tokens/session ($10/yr)</td>
<td>Varies by integration</td>
<td>Core memory in context</td>
+<td>Varies</td>
+<td>Cloud pricing</td>
+<td>No token budget</td>
+<td>Varies</td>
<td>22K+ tokens at 240 obs</td>
</tr>
<tr>
@@ -363,6 +403,10 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).
<td>Yes (port 3113)</td>
<td>Cloud dashboard</td>
<td>Cloud dashboard</td>
+<td>Web UI</td>
+<td>Cloud dashboard</td>
+<td>No</td>
+<td>No</td>
<td>No</td>
</tr>
<tr>
@@ -371,9 +415,15 @@ Latest release notes: [CHANGELOG.md](CHANGELOG.md).
<td>Optional</td>
<td>Optional</td>
<td>Yes</td>
+<td>No (cloud-only)</td>
+<td>Yes</td>
+<td>Yes</td>
+<td>Yes</td>
</tr>
</table>
+<sub>Benchmark note: only agentmemory's R@5 is our own measured result (LongMemEval-S, reproducible from <a href="benchmark/COMPARISON.md"><code>benchmark/COMPARISON.md</code></a>). The mem0 and Letta figures are their published LoCoMo numbers (a different dataset); the MemPalace and supermemory figures are vendor self-reported claims we have not independently reproduced. Shown side by side for ballpark only, not a head-to-head on identical data. Star counts are approximate and drift over time.</sub>
+
---
<h2 id="quick-start"><picture><source media="(prefers-color-scheme: dark)" srcset="assets/tags/light/section-quickstart.svg"><img src="assets/tags/section-quickstart.svg" alt="Quick Start" height="32" /></picture></h2>
diff --git a/benchmark/COMPARISON.md b/benchmark/COMPARISON.md
index 83f7a39d..794833a0 100644
--- a/benchmark/COMPARISON.md
+++ b/benchmark/COMPARISON.md
@@ -14,11 +14,11 @@ All numbers here come from published benchmarks or public repositories. We link
|---|---|---|---|
| **agentmemory** (BM25 + Vector) | LongMemEval-S | **95.2%** | `all-MiniLM-L6-v2` embeddings, no API key |
| agentmemory (BM25-only) | LongMemEval-S | 86.2% | Fallback when no embedding provider available |
-| MemPalace | LongMemEval-S | ~96.6% | Vector-only, bigger embedding model |
+| MemPalace | LongMemEval-S | ~96.6% (self-reported) | Vendor-published number we have not independently reproduced. Vector-only with a larger embedding model and no agent-integration surface (no hooks, no MCP, no multi-agent) |
| Letta / MemGPT | LoCoMo | 83.2% | Different benchmark (LoCoMo, not LongMemEval) |
| Mem0 | LoCoMo | 68.5% | Different benchmark (LoCoMo, not LongMemEval) |
-**⚠️ Apples vs oranges caveat:** agentmemory and MemPalace are measured on LongMemEval-S. Letta and Mem0 publish on [LoCoMo](https://snap-stanford.github.io/LoCoMo/), a different benchmark. We're showing both so you can see the ballpark. We'd love to run all four on the same dataset — if any maintainer wants to collaborate, open an issue.
+**⚠️ Apples vs oranges caveat:** only agentmemory's 95.2% is our own measured result, reproducible from the methodology below. Every other number here is the vendor's published claim, on a different benchmark or harness, that we have not independently reproduced: MemPalace reports LongMemEval-S, while Letta and Mem0 publish on [LoCoMo](https://snap-stanford.github.io/LoCoMo/). Treat them as ballpark vendor claims, not a head-to-head on identical data. We'd love to run every system on the same dataset; if any maintainer wants to collaborate, open an issue.
Full agentmemory methodology: [`LONGMEMEVAL.md`](LONGMEMEVAL.md)
@@ -26,26 +26,26 @@ Full agentmemory methodology: [`LONGMEMEVAL.md`](LONGMEMEVAL.md)
## Feature Matrix
-| Feature | agentmemory | mem0 | Letta/MemGPT | Khoj | claude-mem | Hippo |
-|---|---|---|---|---|---|---|
-| **GitHub stars** | Growing | 53K+ | 22K+ | 34K+ | 46K+ | Trending |
-| **Type** | Memory engine + MCP server | Memory layer API | Full agent runtime | Personal AI | MCP server | Memory system |
-| **Auto-capture via hooks** | ✅ 12 lifecycle hooks | ❌ Manual `add()` | ❌ Agent self-edits | ❌ Manual | ✅ Limited | ❌ Manual |
-| **Search strategy** | BM25 + Vector + Graph | Vector + Graph | Vector (archival) | Semantic | FTS5 | Decay-weighted |
-| **Multi-agent coordination** | ✅ Leases + signals + mesh | ❌ | Runtime-internal only | ❌ | ❌ | Multi-agent shared |
-| **Framework lock-in** | None | None | High | Standalone | Claude Code | None |
-| **External deps** | None | Qdrant/pgvector | Postgres + vector | Multiple | None (SQLite) | None |
-| **Self-hostable** | ✅ default | Optional | Optional | ✅ | ✅ | ✅ |
-| **Knowledge graph** | ✅ Entity extraction + BFS | ✅ Mem0g variant | ❌ | Doc links | ❌ | ❌ |
-| **Memory decay** | ✅ Ebbinghaus + tiered | ❌ | ❌ | ❌ | ❌ | ✅ Half-lives |
-| **4-tier consolidation** | ✅ Working → episodic → semantic → procedural | ❌ | OS-inspired tiers | ❌ | ❌ | Episodic + semantic |
-| **Version / supersession** | ✅ Jaccard-based | Passive | ❌ | ❌ | ❌ | ❌ |
-| **Real-time viewer** | ✅ Port 3113 | Cloud dashboard | Cloud dashboard | Web UI | ❌ | ❌ |
-| **Privacy filtering** | ✅ Strips secrets pre-store | ❌ | ❌ | ❌ | ❌ | ❌ |
-| **Obsidian export** | ✅ Built-in | ❌ | ❌ | Native format | ❌ | ❌ |
-| **Cross-agent** | ✅ MCP + REST | API calls | Within runtime | Standalone | Claude-only | Multi-agent shared |
-| **Audit trail** | ✅ All mutations logged | ❌ | Limited | ❌ | ❌ | ❌ |
-| **Language SDKs** | Any (REST + MCP) | Python + TS | Python only | API | Any (MCP) | Node |
+| Feature | agentmemory | mem0 | Letta/MemGPT | Khoj | supermemory | MemPalace | Hippo |
+|---|---|---|---|---|---|---|---|
+| **GitHub stars** | Growing | 58K+ | 23K+ | 35K+ | 26K+ | 54K+ | Trending |
+| **Type** | Memory engine + MCP server | Memory layer API | Full agent runtime | Personal AI | Memory API + app | Benchmark-focused OSS | Memory system |
+| **Auto-capture via hooks** | ✅ 12 lifecycle hooks | ❌ Manual `add()` | ❌ Agent self-edits | ❌ Manual | ❌ API-side extraction | ❌ Manual | ❌ Manual |
+| **Search strategy** | BM25 + Vector + Graph | Vector + Graph | Vector (archival) | Semantic | Vector + RAG | Vector-only (large model) | Decay-weighted |
+| **Multi-agent coordination** | ✅ Leases + signals + mesh | ❌ | Runtime-internal only | ❌ | ❌ | ❌ | Multi-agent shared |
+| **Framework lock-in** | None | None | High | Standalone | None (drop-in wrappers) | None | None |
+| **External deps** | None | Qdrant/pgvector | Postgres + vector | Multiple | Managed cloud | Vector store | None |
+| **Self-hostable** | ✅ default | Optional | Optional | ✅ | ❌ Cloud-only | ✅ | ✅ |
+| **Knowledge graph** | ✅ Entity extraction + BFS | ✅ Mem0g variant | ❌ | Doc links | ❌ | ❌ | ❌ |
+| **Memory decay** | ✅ Ebbinghaus + tiered | ❌ | ❌ | ❌ | ✅ Auto-forget | ❌ | ✅ Half-lives |
+| **4-tier consolidation** | ✅ Working → episodic → semantic → procedural | ❌ | OS-inspired tiers | ❌ | ❌ | ❌ | Episodic + semantic |
+| **Version / supersession** | ✅ Jaccard-based | Passive | ❌ | ❌ | ✅ Auto-resolve | ❌ | ❌ |
+| **Real-time viewer** | ✅ Port 3113 | Cloud dashboard | Cloud dashboard | Web UI | Cloud dashboard | ❌ | ❌ |
+| **Privacy filtering** | ✅ Strips secrets pre-store | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
+| **Obsidian export** | ✅ Built-in | ❌ | ❌ | Native format | ❌ | ❌ | ❌ |
+| **Cross-agent** | ✅ MCP + REST | API calls | Within runtime | Standalone | MCP + API | Standalone | Multi-agent shared |
+| **Audit trail** | ✅ All mutations logged | ❌ | Limited | ❌ | ❌ | ❌ | ❌ |
+| **Language SDKs** | Any (REST + MCP) | Python + TS | Python only | API | Python + TS | Python | Node |
---
@@ -59,8 +59,8 @@ The main reason to use persistent memory at all: token cost. Here's what one yea
| LLM-summarized memory (extraction-based) | ~650K | ~$500 | Lossy — summarization drops detail |
| **agentmemory (API embeddings)** | **~170K** | **~$10** | Token-budgeted, only relevant memories injected |
| **agentmemory (local embeddings)** | **~170K** | **$0** | `all-MiniLM-L6-v2` runs in-process |
-| claude-mem | Reports ~10x savings | — | SQLite + FTS5 + 3-layer filter |
-| Mem0 | Varies by integration | — | Extraction-based, no token budget |
+| supermemory | Not published | Cloud pricing | Managed API, no local token budget |
+| Mem0 | Varies by integration | Varies | Extraction-based, no token budget |
**agentmemory ships with a built-in token savings calculator.** Run `npx @agentmemory/agentmemory status` after a few sessions and you'll see exactly how many tokens you've saved vs. pasting the full history.
@@ -97,10 +97,17 @@ This isn't a "agentmemory wins everything" page. Different tools solve different
- Obsidian/Notion/Emacs integrations
- Scheduled automations and research tasks
-**Choose claude-mem if you want:**
-- Claude Code-specific tooling with SQLite + FTS5
-- Minimal install footprint
-- Token compression via LLM
+**Choose supermemory if you want:**
+- A managed memory API with server-side auto-extraction and automatic forgetting
+- Drop-in wrappers for major AI frameworks (Vercel AI, LangChain, LangGraph)
+- A hosted dashboard with no infrastructure to run yourself
+- RAG plus memory served from a single query
+
+**Choose MemPalace if you want:**
+- A simple, free, open-source vector memory store
+- To chase its self-reported retrieval benchmark (we have not reproduced it)
+- Pure retrieval over agent workflow features
+- Note: no auto-capture, no MCP, no multi-agent coordination, so you wire all integration yourself
**Choose Hippo if you want:**
- Biologically-inspired memory model (decay, consolidation, sleep)