Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
docs: MemPalace issue + architecture synthesis
AkitaOnRailse95d17a8241f
diff --git a/docs/issues-mempalace.md b/docs/issues-mempalace.mdnew file mode 100644index 00000000..d2e4147d--- /dev/null+++ b/docs/issues-mempalace.md@@ -0,0 +1,266 @@+# MemPalace — Issue & Architecture Synthesis++> Source: GitHub `MemPalace/mempalace` + local checkout at `~/Projects/mempalace`.+> Captured 2026-05-24. Repo: 52.7k stars, MIT, Python, created 2026-04-05,+> v3.3.5, 241 open issues, ~1600 PRs in ~7 weeks. Extremely active, unusually+> transparent maintainers (see `docs/HISTORY.md` — they publicly retract+> overclaims). Dominant themes: **ChromaDB/HNSW/FTS5 corruption under+> concurrent writes**, **silent persistence failures**, **repair tooling that+> destroys data**, **embedding-model drift**.++## What MemPalace is (one paragraph)++Local-first AI memory with the *opposite* philosophy to ai-memory: **store+verbatim, never compile**. Conversation history and files are chunked raw into+"drawers" (verbatim text), organised into "wings" (person/project) and "rooms"+(topic), indexed in ChromaDB (SQLite + HNSW vector index, 384-dim+all-MiniLM-L6-v2, zero API calls). Retrieval is hybrid vector + BM25 with a+"closet" keyword-boost layer. Capture is via explicit `mempalace mine`, a+transcript "convo miner", and Claude Code Stop/PreCompact hooks that spawn+detached `mempalace mine` subprocesses. Claims 96.6% R@5 on LongMemEval (raw,+no LLM). It is a genuinely strong retrieval system whose pain is almost entirely+**operational/storage**, not retrieval.++## Top recurring pain points (ranked)++### 1. Concurrent-write corruption of ChromaDB / HNSW / FTS5 — THE dominant theme++This single cluster is the majority of open bugs. Root cause: **multiple+uncoordinated writer paths to one ChromaDB store** — the Stop hook fires an+async `mempalace mine` on every session end, the PreCompact hook fires another,+the MCP server writes on tool calls, and manual `mempalace mine` can run at the+same time. ChromaDB's HNSW index and FTS5 shadow tables are not safe under+concurrent writers, and a writer killed mid-write (SIGTERM during compaction /+permission prompts / window close) leaves a malformed index.++- **#1596** — *"The Claude Code Stop hook fires `mempalace mine` asynchronously+ on every session end. When multiple Claude sessions close simultaneously …+ multiple mine processes start with no mutual-exclusion guard. These+ overlapping processes write to FTS5 shadow tables concurrently, and when one+ is killed mid-write … it leaves `embedding_fulltext_search` in a malformed+ state."* `PRAGMA integrity_check` returns `ok` (rows intact) but+ `quick_check` fails (FTS5 index malformed) → `repair` refuses to run.+- **#1599** — recurring FTS5 corruption under live writes, *after* the+ transient-vector-failure mitigation that was supposed to fix it.+- **#1581** — HNSW corruption from concurrent ChromaDB clients (MCP server ++ hook subprocesses + auto-ingest mine) **within a single Claude Code session**.+- **#1253** — PreCompact hook double-ingest race corrupts HNSW: *"no+ palace-wide write lock."*+- **#1564** — even the single-writer sweep can trigger stale HNSW quarantine.+- **#1514** (community patch, in Chinese) — dual-client concurrent read/write:+ user shipped a `flock + threading.Lock` double-lock patch themselves.+- **#1343 / #974 / #965** (referenced) — the original "wrap every ChromaDB+ write in `mine_palace_lock`" retrofit; corruption persists in edge cases.+- **#1497** (the tell) — a user asks: *"should MemPalace recommend a+ single-writer/gateway pattern for multi-agent setups? … our reading is:+ `mcp-proxy` helps avoid each client spawning its own server, but it does not+ by itself guarantee that all palace mutations are serialized."* They sketch a+ gateway → write-queue → **single writer**. **They are proposing, in an+ issue, the exact architecture ai-memory has by construction.**++### 2. Silent persistence failure — "filed N drawers" but nothing persisted++The write path returns success *before* the data is durably in ChromaDB. Below+`hnsw:sync_threshold`, the HNSW segment + `index_metadata.pickle` never flush,+so small mines vanish while the CLI cheerfully reports success.++- **#1597** — *"`mempalace mine` reports `Drawers filed: 40` … but the chromadb+ does not persist them. `mempalace status` keeps showing the pre-upgrade+ total."* Confirmed not a cache issue (`mempalace_reconnect` agrees with the+ stale count). The v3.3.5 notes had specifically claimed *"integrity,+ recovery, and cross-process correctness."*+- **#1579** — mines smaller than `hnsw:sync_threshold` never persist+ `index_metadata.pickle`.+- **#1526** — silent empty index below batch threshold (one of two HNSW failure+ modes).+- **#1537** — `mempalace mine` reports `Done ✓` with full counts when the mine+ produced an FTS5-corrupt palace.+- **#1398** — MCP search returns `Error finding id` after bulk-add until WAL+ flushes; CLI works (read-after-write inconsistency).+- **#1489** — request to expose `hnsw:sync_threshold` because low-volume MCP+ workloads "never flush metadata."++### 3. Repair/migrate tooling that cannot recover — or actively destroys data++When corruption happens (theme 1) the recovery path is itself unsafe.++- **#1394** — *"`mempalace repair --mode legacy` destroys 99% of SQLite+ embedding rows on 3.3.4."*+- **#1545** — `repair --mode from-sqlite` "structurally broken at scale:+ auto-quarantine destroys rebuild progress mid-run."+- **#1586** — repair cannot recover combined HNSW-dimensionality + FTS5+ corruption (313k drawers).+- **#1589** — repair *rebuilds* an HNSW segment that ChromaDB 1.5.9 then cannot+ load; the corrupt segment SIGSEGVs in `chromadb_rust_bindings`.+- **#1595** — `mempalace-health.py` parses `header.bin` as uint32 →+ trillion-element HNSW corruption silently passes all checks.+- **#1266** — HNSW pickle corruption after long-running mine: "unsupported+ opcode" on next read; repair/migrate cannot recover.+- **#1492 / #1493** — `rebuild_index` deterministically writes+ `dimensionality=None` into ChromaDB's `index_metadata`.++### 4. Data loss by design — miners and swaps destroy verbatim history++- **#1593** — *filed by the founder*: "stop delete-on-remine — verbatim history+ should never be destroyed by MemPalace miners." Re-mining a source deletes the+ old drawers first; for a system whose whole pitch is "verbatim, never lossy,"+ this is an existential bug.+- **#1533** — MCP-server processes destroy newly-swapped palace files on first+ read after an atomic swap (no quarantine re-arm).+- **#1341** — no `SessionEnd` hook: short sessions lose memories on clean exit+ (only Stop/PreCompact exist, and they don't always fire).+- **#1535** — `mempalace mine --limit N` truncates files *before* the+ already-mined skip, so a run can finish with 0 new drawers.+- **#1329** — Stop hook → 1.9 **TB** palace bloat + ChromaDB Rust-bindings+ segfault (verbatim-everything with no decay → unbounded growth).++### 5. Embedding-model drift & pluggability gaps++- **#1561** — "persist and verify embedding model metadata in palace+ collections." There is currently **no stored `{provider, model, dim}` to+ refuse on mismatch** — exactly the failure ai-memory's invariant #8 prevents.+- **#1559** — support external embedding APIs (LM Studio, Ollama,+ OpenAI-compatible).+- **#1563 / #1261** — embedding model is hardcoded to all-MiniLM-L6-v2; users+ want configuration-driven switching without source edits.+- **#1380** — hash-embedding fallback mode needs a real BM25 lexical path.++### 6. Native-binding / platform instability (the heavy-dependency tax)++ChromaDB drags in `onnxruntime`, `grpcio`, `numpy`, and a Rust bindings layer.++- **#1355** — `chromadb_rust_bindings` segfaults on macOS 26.4 ARM64 — **all+ `>=1.5.4,<2` versions affected**. Hard floor on a core dependency.+- **#1247** — Windows ONNX `bad_alloc` + convo mine crash.+- **#1488** — MCP server fails on Cyrillic/non-ASCII on Windows (cp1252 stdin).+- **#1570** — openai-compat provider probe blocked (403) on Cloudflare-fronted+ endpoints because of the default `Python-urllib` User-Agent.++### 7. MCP correctness & scale++- **#1574** — MCP returns generic `-32000` instead of `-32602 (Invalid params)`+ on missing required params.+- **#1580** — `_expand_with_neighbors` stitches *unrelated* chunks across MCP+ drawers that share an empty `source_file` (silent cross-contamination of+ results).+- **#1379** — overview tools time out on large palaces.+- **#1471** — "HNSW capacity divergence" check runs per-request in the MCP+ server (perf).++### 8. Capture noise (markdown/transcript parsing)++- **#1333** — `<local-command-*>` tags and ANSI escapes survive+ `strip_noise()`, polluting drawers (same class of problem as basic-memory+ #738 and our own Claude-Code-tag stripping).++## Design choices that caused the most issues++| Design choice | Cited in | Symptom |+|---|---|---|+| ChromaDB (SQLite + binary HNSW segments + FTS5) as both store **and** index | #1596, #1599, #1581, #1589, #1355 | Index corruption + native segfaults; integrity_check ok but quick_check fails |+| Multiple uncoordinated writer paths (Stop hook + PreCompact hook + MCP + manual mine), lock retrofitted later | #1596, #1253, #1497, #1514, #1343 | Concurrent-write corruption; community ships own dual-lock |+| Persistence deferred until `hnsw:sync_threshold` batch; write returns before durable | #1597, #1579, #1526, #1489, #1398 | "Filed N drawers" with nothing persisted; read-after-write misses |+| Repair/migrate without graveyard-before-destroy | #1394, #1545, #1586, #1266 | Recovery tooling destroys the data it was meant to save |+| Verbatim-everything with no decay/forget | #1329, #1593 | 1.9 TB palace bloat; delete-on-remine to manage size → history loss |+| Hardcoded local embedding model, no `{model, dim}` stamped on the collection | #1561, #1563, #1261 | No refuse-on-mismatch; silent drift on model change |+| Heavy native ML stack (onnxruntime + chromadb_rust_bindings) | #1355, #1247, #1488 | Platform-specific crashes, hard version floors |+| Hooks spawn detached `mempalace mine` subprocesses | #1596, #1253, #1329 | N concurrent writers per machine; killed-mid-write corruption |++## How ai-memory compares (invariant-by-invariant)++The headline: **MemPalace's biggest pain cluster is the set of failures+ai-memory's cross-cutting invariants exist to prevent.** This is strong+external validation of the architecture, captured by someone else's 241 open+issues rather than our own.++| MemPalace pain | ai-memory invariant that prevents it |+|---|---|+| #1497/#1514/#1343 — debating/retrofitting a single-writer gateway; concurrent writers corrupt HNSW | **#2 Single-writer SQLite actor.** All writes through one mpsc channel, by construction, since M1. No retrofit. |+| #1597/#1579/#1537 — "filed N" but not persisted; write returns before durable | **#3 Indexes commit in the same transaction as the data.** No background-task-indexing-after-return; tool responses block until durable. |+| #1561/#1563 — no stored embedding model identity; drift on model change | **#8 `{provider, model, dim}` stored next to every embedding; refuse on mismatch.** Schema column since M1, enforced at startup. |+| #1394/#1545/#1586 — repair tooling destroys data it should save | **#17 Wiki migrations: no destructive deletes without a graveyard step** (`_graveyard/<migration>/…`) + **#10 atomic tmp+rename+fsync**. |+| #1593/#1533 — miners/swaps destroy verbatim history | **#10 Atomic file writes** + git-versioned wiki (M5): every page change is a commit, recoverable via `git`. |+| #1355/#1247/#1488 — native ONNX + Rust-bindings segfaults, platform floors | **Bundled `rusqlite` (static), pure-Rust, zero-LLM default path (#13).** Vectors (sqlite-vec) deferred to v0.2 precisely to avoid this class of dependency risk. |+| #1596/#1253 — hooks spawn N concurrent writer subprocesses | **#5 Hooks are fire-and-forget POST to one server** with one writer actor — never a process per event. |+| #1329 — verbatim-everything → 1.9 TB bloat | **Karpathy compile-don't-hoard** + **M8 decay/forget-sweep** + log rotation. We compress, we don't accumulate raw forever. |+| #1574 — wrong JSON-RPC error codes | rmcp typed `McpError` variants. (Worth a spot-check that we map param errors correctly — see "gotchas".) |++## What ai-memory could learn / steal from MemPalace++These are the genuinely useful ideas worth a brainstorm — NOT yet adopted.++1. **A verbatim recall tier as a safety net under the compiled wiki.** This is+ the deepest idea. MemPalace proves that *raw text + local semantic search*+ scores 96.6% R@5 on LongMemEval with zero LLM. ai-memory compiles+ observations into wiki pages — which is higher-signal but **lossy**:+ consolidation can drop a detail the user later needs. We already keep `raw/`+ immutable session logs. When vectors land (v0.2), we could expose semantic+ search over the raw observation log as a *fallback recall path* when the+ compiled wiki misses — "compiled-first, verbatim-fallback." Best of both+ philosophies.+2. **LongMemEval as a first-class, honest eval (our M10).** MemPalace has the+ harness wired and *publishes held-out R@5 with retractions* in+ `docs/HISTORY.md`. Their `tests/test_readme_claims.py` — tests that fail the+ build if a public claim isn't backed by code — is a clever, stealable+ pattern. Adopt the methodology (held-out split, no headline-chasing) when we+ build M10.+3. **Deterministic IDs for idempotent, resume-safe ingest.** MemPalace's sweeper+ uses `session_id + message_uuid` deterministic drawer IDs + a cursor so a+ crashed mine resumes without dupes. Our hook ingestion could adopt the same+ for replay-safety.+4. **Lightweight temporal knowledge-graph triples** (subject/predicate/object ++ `valid_from`/`valid_to` with explicit invalidation). We have a `links`+ table; MemPalace's KG (`knowledge_graph.py`) and their #1416 "promote drawer+ claims into KG triples at end of day" rhymes with our consolidation. Could be+ a future tier for "facts that change over time" (who owns X now vs. before).+5. **Index-health monitoring + a repair path — done safely.** When sqlite-vec+ lands in v0.2, monitor index health (MemPalace's link-to-data ratio guard is+ smart) AND make repair graveyard-first. Their #1394/#1595 are the cautionary+ tales: a repair that destroys rows, and a health check that parses a field+ wrong and passes trillion-element corruption.+6. **Transparent benchmark/claims culture.** A `HISTORY.md` that records+ corrections builds more trust than a clean marketing page. Worth emulating.++## Gotchas to AVOID that bit MemPalace (cautions for our v0.2 vector work)++- **Never report "indexed/saved" before durable commit** — even for vectors.+ MemPalace #1597 is the whole movie. Invariant #3 already covers this; keep it+ ironclad when sqlite-vec writes land.+- **Stamp `{provider, model, dim}` on the collection and refuse on mismatch.**+ MemPalace #1561 shows the cost of not doing it. We have it — don't regress.+- **Any re-embed / rebuild-index path must graveyard before delete.** MemPalace+ #1394 destroyed 99% of rows in a "repair." Invariant #17 already mandates+ this; apply it to vector rebuilds too.+- **Keep hooks one-writer fire-and-forget.** Do not, under load, spawn a worker+ per event (MemPalace #1596). Our design is safe; keep it.+- **Watch raw-log growth.** Verbatim is great for recall but #1329 hit 1.9 TB.+ Our compile + decay + log rotation is the antidote — but if we add a verbatim+ recall tier (idea #1), give it a retention budget from the start.+- **Spot-check MCP param-error codes.** MemPalace #1574 returns the wrong+ JSON-RPC code on missing params. Confirm rmcp maps our missing/invalid params+ to `-32602`, not a generic internal error. (Verify; do not assume.)++## Non-technical gotcha: supply-chain / impostor domains++`docs/HISTORY.md` and the README's top banner document a real incident: once+MemPalace got popular, **impostor domains (`mempalace.tech`, etc.) appeared+distributing malware** under the project name. Their mitigation was a loud+"ONLY official sources are GitHub + PyPI + mempalaceofficial.com" warning.+Lesson for ai-memory: when/if it gains traction, expect typosquatting of the+name, the install one-liner (`curl … | sh`), and the Docker image. Cheap+pre-emptive moves: claim the obvious domains, document the *only* official+install sources (GitHub repo, the `akitaonrails/ai-memory` Docker Hub image,+the raw wrapper URL on `main`), and pin/sign releases so users can verify. This+is a "popular-project" problem we don't have yet — but it's free to pre-empt.++## Bottom line++MemPalace is a strong, popular, well-tested retrieval engine with a smart+verbatim philosophy and admirably honest maintainers — and it is **drowning in+exactly the storage/concurrency/durability failures ai-memory designed around+from the first milestone**. The single most valuable thing to take from it is+not a fix but a feature idea: a *verbatim semantic-recall fallback* beneath the+compiled wiki, so we get MemPalace's lossless recall without inheriting its+ChromaDB corruption surface. The single most valuable thing to take as+reassurance: our invariants #2, #3, #8, #10, #17 are not over-engineering — they+are precisely the 241-open-issues someone else is living through.