[MLsys2026 Best Paper]: https://arxiv.org/abs/2506.08276. RAG on Everything with LEANN. Enjoy 97% storage savings while running a fast, accurate, and 100% private RAG application on your personal device.
Python aifaissgpt-osslangchainllama-indexllmlocalstorageoffline-firstollamaprivacypythonragretrieval-augmented-generationvector-databasevector-searchvectors
fix: incremental build chunking inconsistency and shared metadata dict (#278)
Yichuan Wangf22bcdd68fd2
diff --git a/CLAUDE.md b/CLAUDE.mdindex da1c396..774a3f9 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -8,21 +8,28 @@ LEANN is a lightweight vector database and RAG (Retrieval-Augmented Generation)## Build & Development Commands+### Quick install (pip)++```bash+pip install leann+```++### Development setup (from source)+```bash# Install uv first (required package manager)curl -LsSf https://astral.sh/uv/install.sh | sh-# Development setup (from source)git submodule update --init --recursive# macOSbrew install libomp boost protobuf zeromq pkgconf-uv sync --extra diskann+uv sync# Ubuntu/Debiansudo apt-get install libomp-dev libboost-all-dev protobuf-compiler \libabsl-dev libmkl-full-dev libaio-dev libzmq3-dev-uv sync --extra diskann+uv sync# Install lint toolsuv sync --group lint@@ -31,27 +38,6 @@ uv sync --group lintuv sync --group test```-## Testing--```bash-# Run all tests-uv run pytest--# Run specific test file-uv run pytest tests/test_basic.py--# Run with coverage-uv run pytest --cov=leann--# Skip slow tests-uv run pytest -m "not slow"--# Skip tests requiring OpenAI API-uv run pytest -m "not openai"-```--Test markers: `slow`, `openai`, `integration`-## Code Quality```bash@@ -82,6 +68,7 @@ uv run pre-commit run --all-files### Backend Layer (`packages/`)- `leann-backend-hnsw/`: Default backend using FAISS HNSW for fast in-memory search+- `leann-backend-ivf/`: IVF backend (FAISS IndexIVFFlat + DirectMap.Hashtable) supporting in-place add/remove without rebuild- `leann-backend-diskann/`: DiskANN backend for larger-than-memory datasets- `leann-mcp/`: MCP server for Claude Code integration@@ -99,11 +86,17 @@ Example applications demonstrating RAG on various data sources:## Key Design Patterns-### Threading Environment Variables+### Incremental Update (IVF backend)-LEANN sets critical threading environment variables in `__init__.py` to prevent FAISS/ZMQ hangs:-- macOS: `OMP_NUM_THREADS=1`, `MKL_NUM_THREADS=1`, `KMP_DUPLICATE_LIB_OK=TRUE`-- Linux: `OMP_NUM_THREADS=1`, `FAISS_NUM_THREADS=1`, `OMP_WAIT_POLICY=PASSIVE`+The IVF backend supports in-place updates and deletes without rebuilding the entire index:+- `add_vectors(index_path, embeddings, passage_ids)`: Append new vectors to an existing index.+- `remove_ids(index_path, passage_ids)`: Remove vectors by passage ID using FAISS DirectMap.Hashtable.+- `LeannBuilder.update_index()`: High-level API that orchestrates remove-then-add for changed files, compacts `passages.jsonl`, and updates the offset map.++`leann build` is idempotent — re-running it on an existing index automatically performs an incremental update instead of a full rebuild. It detects new, modified, and removed files and applies the minimal set of changes:+- **IVF**: Supports add, remove, and modify incrementally (remove old chunks then re-insert).+- **HNSW** (non-compact): Supports add-only incremental updates; modified/removed files trigger a full rebuild.+- Use `--force` / `-f` to force a full rebuild regardless.### Index Structure@@ -150,3 +143,66 @@ python -m apps.code_rag --repo-dir ./src --query "How does search work?"## Python VersionRequires Python 3.10+ (uses PEP 604 union syntax `X | Y`).+++++# Agent Coding Guidelines++## General+- Voice input may contain typos — interpret intent, not literal text.+- When you encounter a problem, fix it immediately and keep going until there are no more problems.+- Do not ask about ordering or sequencing — figure it out. If something is unclear, note it and skip it; only escalate when all paths are blocked.+- Obvious bugs: fix silently without reporting.+- No fallbacks or compatibility shims. One correct implementation per feature — no redundancy.++## Roadmap+- Public roadmap: `docs/roadmap.md` — tracks P0/P1 priorities, completed milestones, and timeline.+- Long-term vision: `docs/ultimate_goal.md` — the north star for where LEANN is headed.+- Keep in sync with [GitHub issue #237](https://github.com/yichuan-w/LEANN/issues/237).+- Welcome everyone to add more, and the craziest feature you want to put here! If people want some feature, all put there.++## Changelog (for contributors)+- Maintain `docs/CHANGELOG.md` — append-only log of major changes (new features, breaking changes, important fixes).+- Format: `## YYYY-MM-DD: <short summary>` followed by bullet points.+- Update the changelog when merging significant PRs or completing notable work.+- See `docs/CONTRIBUTING.md` for full contributor workflow (conventional commits, PR process, CI).++## Personal Dev Notes (gitignored)+- `docs/dev/` is gitignored for personal development notes (TODO, progress, experiments).+- Use `docs/dev/TODO.md` for in-progress tasks, `docs/dev/PROGRESS.md` for completed work.+- These are private scratch space — but must follow the Self-Contained Principle below.++## Documentation — Self-Contained Principle++All dev docs (`PROGRESS.md`, `STATES.md`, `EXPERIMENTS.md`, `TODO.md`) must be fully understandable from the document alone, with no reliance on conversation context or implied knowledge.++Requirements:+1. **Every technique/approach must be explained on first use.** Not "switched to IVF backend" — write "switched to IVF backend (FAISS IndexIVFFlat + DirectMap.Hashtable, supports in-place add/remove without full index rebuild)."+2. **Never assume the reader knows any abbreviation.** On first use: full name + one-sentence explanation. E.g., "HNSW (Hierarchical Navigable Small World — a graph-based ANN index used as LEANN's default backend)."+3. **Benchmark results must include full context.** Not "recall improved to 0.95" — write "recall@10 improved from 0.91 to 0.95 after switching from flat chunking (512 tokens, no overlap) to AST-aware chunking (function-level splits with 64-token overlap)."+4. **Numbers must have reference points.** Not "build time: 12s" — write "build time: 12s (vs. 45s before incremental update support, on 10k-document corpus)."+5. **Include the causal chain — not just conclusions.** Not "duplicate chunks appeared after incremental build" — write "Duplicate chunks appeared after incremental build because `passages.jsonl` was appended without first removing stale entries for modified files. The IVF index had correct vectors (remove-then-add), but the passage store was append-only, causing the same text to appear at multiple offsets."+6. **`docs/dev/STATES.md` top section maintains a glossary** of all key terms (backends, index files, chunking strategies, embedding models). Other docs reference it at the top.++Bad examples (forbidden):+- "Fixed the chunking bug" → Which bug? What was the symptom? What was the root cause?+- "Improved search quality" → By what metric? From what baseline? What change caused it?+- "Used nprobe=32" → What is nprobe? Why 32? What was it before and what effect did the change have?++## Doc Maintenance+- Maintain `docs/dev/PROGRESS.md` — completed work only (with key script/log/config paths). No plans.+- Maintain `docs/dev/TODO.md` — incomplete/in-progress/next-steps only (aim for one-command reproducibility). When done: remove from TODO, write result to PROGRESS, update STATES/EXPERIMENTS if needed.+- Both files: **append-only, chronological order** (oldest first). Use `tail -n 80 docs/dev/PROGRESS.md` to read recent entries; increase range or grep by date/keyword if needed.+- Keep TODO clean — either do items or remove them. Ask the user when unsure how to handle a TODO item.+- Maintain `docs/dev/STATES.md` — tracks all currently useful state (index configs, backend choices, known limitations); does NOT grow indefinitely (delete stale entries).+- Maintain `docs/dev/EXPERIMENTS.md` — benchmarks, A/B comparisons, parameter sweeps (recall@k, latency, storage size). Experimental content goes here, not in STATES.md.++## Commits+Commit when: (1) a complete feature is finished and tested, or (2) a destructive change is unavoidable.+```bash+git add <specific files>+git commit -m “feat: ...” # follow conventional commits+```+- When correcting errors: fix directly with no trace of the error.+- If you write a correct new version of a file, delete the wrong version. No duplicate implementations.diff --git a/docs/roadmap.md b/docs/roadmap.mdindex fa04b5c..2f87114 100644--- a/docs/roadmap.md+++ b/docs/roadmap.md@@ -1,21 +1,67 @@-# 📈 Roadmap+# LEANN Roadmap-## 🎯 Q2 2025+LEANN aims to be a **personal knowledge layer** — not just a storage-efficient vector database, but a unified, always-up-to-date knowledge base that runs entirely on your own machine. It connects your code, images, and personal data (documents, emails, browser history, chats) into a single multimodal search interface.-- [X] HNSW backend integration-- [X] DiskANN backend with MIPS/L2/Cosine support-- [X] Real-time embedding pipeline-- [X] Memory-efficient graph pruning+Contributions and feedback are welcome. Join our [Slack](https://join.slack.com/t/leann-e2u9779/shared_invite/zt-3ol2ww9ic-Eg_kB8omwe6xmYVd0epr4Q) to discuss.-## 🚀 Q3 2025+----- [ ] Advanced caching strategies-- [ ] Add contextual-retrieval https://www.anthropic.com/news/contextual-retrieval-- [ ] Add sleep-time-compute and summarize agent! to summarilze the file on computer!-- [ ] Add OpenAI recompute API+## Completed-## 🌟 Q4 2025+- [x] HNSW backend integration+- [x] DiskANN backend with MIPS/L2/Cosine support+- [x] Real-time embedding pipeline+- [x] Memory-efficient graph pruning+- [x] IVF backend with incremental add/remove (#231, #89, #141)+- [x] Merkle tree file-change detection — `leann watch` (#41)-- [ ] Integration with LangChain/LlamaIndex-- [ ] Visual similarity search-- [ ] Query rewrtiting, rerank and expansion+---++## P0 — Core (Q1 2026)++### LEANN MCP — The Best Code Retrieval MCP++The primary near-term goal: make LEANN the go-to MCP server for code-aware AI assistants. This means **dynamic updates** (your index stays current as you edit code), **rich code context** (AST-aware chunking that understands functions, classes, and modules — not just raw text), and a **dead-simple interface** (one command to build, automatic incremental updates, zero configuration for common setups).++- [x] IVF backend — incremental add/remove without full rebuild (#231, #89, #141)+- [x] Merkle tree file-change detection — `leann watch` for automatic re-indexing on file changes (#41)+- [ ] Cold start optimization — faster first-build experience for new users (#166, #177)+- [ ] Live index updates — push index changes as files are saved, not just on rebuild+- [ ] Smarter code context — cross-file symbol resolution, call graph awareness, import tracking++### Search Quality++- [ ] Hybrid search — combine dense vector retrieval with sparse keyword matching (BM25) for better recall on exact identifiers and variable names (#233, #90)++### Documentation++- [ ] ReadTheDocs — hosted documentation site (#234)+- [ ] Benchmarks — recall@k, latency, and storage comparisons across backends++---++## P1 (Q2 2026)++### Multimodal++- [ ] Video retrieval (#160)+- [ ] CLIP support — image-text cross-modal search (#94)+- [ ] OCR — extract text from images/scanned documents (#158)++### Platform & Distribution++- [ ] Windows support (#14)+- [ ] Web UI (#229)++### Applications & Integrations++- [ ] Agent + Deep research (#104)+- [ ] Local Cursor — local model + local retrieval for code assistance (#47)+- [ ] LlamaIndex integration (#217)+- [ ] Obsidian support (#96)++---++## Contributing++If you're interested in working on any of the items above, please reach out to [@yichuan-w](https://github.com/yichuan-w) or [@andylizf](https://github.com/andylizf). See [CONTRIBUTING.md](CONTRIBUTING.md) for the full contributor workflow.diff --git a/docs/ultimate_goal.md b/docs/ultimate_goal.mdnew file mode 100644index 0000000..8d2d7bd--- /dev/null+++ b/docs/ultimate_goal.md@@ -0,0 +1,41 @@+# LEANN — Long-Term Vision++## The Best Personal Data Management Platform++LEANN's ultimate goal is to be the **unified personal knowledge layer** that lives on your machine. Not a cloud service. Not a SaaS product. A local-first system that understands everything you've ever worked on — code, documents, emails, chats, browser history, images — and makes it all instantly searchable and usable.++### 1. Continuous Learning from Your Context++LEANN should get smarter over time by continuously ingesting and indexing your data as you produce it:++- **Always-on indexing**: Watch your filesystem, email, browser, and chat sources. Incrementally update indexes as new data arrives — no manual rebuilds.+- **Cross-source connections**: Surface relationships across data sources. A Slack conversation about a bug should link to the relevant code change, the related email thread, and the document that describes the feature.+- **Temporal awareness**: Understand when things happened. "What was I working on last Tuesday?" should be a searchable query.+- **Personalized ranking**: Learn from your search patterns and usage to rank results by relevance to *you*, not just semantic similarity.++### 2. The Best MCP for Code Retrieval++AI coding assistants are only as good as the context they receive. LEANN aims to be the best context provider:++- **AST-aware chunking**: Understand code structure — functions, classes, modules — not just raw text blocks. Retrieve semantically meaningful units.+- **Dynamic index updates**: The index stays current as you edit. No stale results. No manual rebuilds. Save a file, and the index reflects the change within seconds.+- **Cross-file understanding**: Resolve symbols across files. When you search for a function, also surface its callers, its tests, and the types it depends on.+- **Dead-simple interface**: One command to build (`leann build`), automatic incremental updates, MCP server that any AI assistant can plug into with zero configuration.+- **Repository-scale search**: Handle monorepos and large codebases without breaking a sweat. The storage efficiency (97% reduction vs. traditional vector DBs) makes this feasible on a laptop.++### 3. Multimodal Knowledge Base++Text is just the beginning:++- **Image search**: CLIP-based retrieval for screenshots, diagrams, photos. "Find the architecture diagram from last month."+- **Video retrieval**: Index video content — lectures, meetings, screen recordings — and search by what was said or shown.+- **OCR pipeline**: Extract and index text from scanned documents, handwritten notes, whiteboard photos.+- **Unified search**: One query searches across all modalities. The answer might be in a PDF, a screenshot, a code comment, or a Slack message.++### 4. Platform for Personal AI++Looking further ahead, LEANN becomes the memory and retrieval layer for personal AI agents:++- **Agent memory**: AI agents that remember your preferences, past decisions, and project context across sessions.+- **Deep research**: Agents that can search your entire personal knowledge base to answer complex questions, synthesize information, and generate insights.+- **Proactive suggestions**: Surface relevant context before you ask for it — "You discussed this exact problem with a colleague 3 months ago, here's what you decided."diff --git a/packages/leann-core/src/leann/cli.py b/packages/leann-core/src/leann/cli.pyindex 4afb0c3..5a178f9 100644--- a/packages/leann-core/src/leann/cli.py+++ b/packages/leann-core/src/leann/cli.py@@ -1303,6 +1303,9 @@ Examples:# exclude_hidden only affects directory scans; input_files are explicitfilename_as_id=True,).load_data()+ for doc in file_docs:+ if not doc.metadata.get("source"):+ doc.metadata["source"] = doc.metadata.get("file_path", "")all_documents.extend(file_docs)print(f" ✅ Loaded {len(file_docs)} document{'s' if len(file_docs) > 1 else ''}"@@ -1584,7 +1587,9 @@ Examples:# Check if this is a code file based on source pathsource_path = doc.metadata.get("source", "")file_path = doc.metadata.get("file_path", "")- is_code_file = any(source_path.endswith(ext) for ext in code_file_exts)+ is_code_file = any(+ (source_path or file_path).endswith(ext) for ext in code_file_exts+ )# For code files, prepend line numbers so chunks carry themif is_code_file:@@ -1621,7 +1626,7 @@ Examples:first_nl = text.find("\n")if first_nl != -1:text = text[first_nl + 1 :]- all_texts.append({"text": text, "metadata": chunk_metadata})+ all_texts.append({"text": text, "metadata": chunk_metadata.copy()})print(f"Loaded {len(documents)} documents, {len(all_texts)} chunks")return all_texts