Skip to content

armgit / LEANN

Public Imported from GitHub
0

[MLsys2026 Best Paper]: https://arxiv.org/abs/2506.08276. RAG on Everything with LEANN. Enjoy 97% storage savings while running a fast, accurate, and 100% private RAG application on your personal device.

fix: incremental build chunking inconsistency and shared metadata dict (#278)

Yichuan Wangf22bcdd68fd2
diff --git a/CLAUDE.md b/CLAUDE.md
index da1c396..774a3f9 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -8,21 +8,28 @@ LEANN is a lightweight vector database and RAG (Retrieval-Augmented Generation)
## Build & Development Commands
+### Quick install (pip)
+
+```bash
+pip install leann
+```
+
+### Development setup (from source)
+
```bash
# Install uv first (required package manager)
curl -LsSf https://astral.sh/uv/install.sh | sh
-# Development setup (from source)
git submodule update --init --recursive
# macOS
brew install libomp boost protobuf zeromq pkgconf
-uv sync --extra diskann
+uv sync
# Ubuntu/Debian
sudo apt-get install libomp-dev libboost-all-dev protobuf-compiler \
libabsl-dev libmkl-full-dev libaio-dev libzmq3-dev
-uv sync --extra diskann
+uv sync
# Install lint tools
uv sync --group lint
@@ -31,27 +38,6 @@ uv sync --group lint
uv sync --group test
```
-## Testing
-
-```bash
-# Run all tests
-uv run pytest
-
-# Run specific test file
-uv run pytest tests/test_basic.py
-
-# Run with coverage
-uv run pytest --cov=leann
-
-# Skip slow tests
-uv run pytest -m "not slow"
-
-# Skip tests requiring OpenAI API
-uv run pytest -m "not openai"
-```
-
-Test markers: `slow`, `openai`, `integration`
-
## Code Quality
```bash
@@ -82,6 +68,7 @@ uv run pre-commit run --all-files
### Backend Layer (`packages/`)
- `leann-backend-hnsw/`: Default backend using FAISS HNSW for fast in-memory search
+- `leann-backend-ivf/`: IVF backend (FAISS IndexIVFFlat + DirectMap.Hashtable) supporting in-place add/remove without rebuild
- `leann-backend-diskann/`: DiskANN backend for larger-than-memory datasets
- `leann-mcp/`: MCP server for Claude Code integration
@@ -99,11 +86,17 @@ Example applications demonstrating RAG on various data sources:
## Key Design Patterns
-### Threading Environment Variables
+### Incremental Update (IVF backend)
-LEANN sets critical threading environment variables in `__init__.py` to prevent FAISS/ZMQ hangs:
-- macOS: `OMP_NUM_THREADS=1`, `MKL_NUM_THREADS=1`, `KMP_DUPLICATE_LIB_OK=TRUE`
-- Linux: `OMP_NUM_THREADS=1`, `FAISS_NUM_THREADS=1`, `OMP_WAIT_POLICY=PASSIVE`
+The IVF backend supports in-place updates and deletes without rebuilding the entire index:
+- `add_vectors(index_path, embeddings, passage_ids)`: Append new vectors to an existing index.
+- `remove_ids(index_path, passage_ids)`: Remove vectors by passage ID using FAISS DirectMap.Hashtable.
+- `LeannBuilder.update_index()`: High-level API that orchestrates remove-then-add for changed files, compacts `passages.jsonl`, and updates the offset map.
+
+`leann build` is idempotent — re-running it on an existing index automatically performs an incremental update instead of a full rebuild. It detects new, modified, and removed files and applies the minimal set of changes:
+- **IVF**: Supports add, remove, and modify incrementally (remove old chunks then re-insert).
+- **HNSW** (non-compact): Supports add-only incremental updates; modified/removed files trigger a full rebuild.
+- Use `--force` / `-f` to force a full rebuild regardless.
### Index Structure
@@ -150,3 +143,66 @@ python -m apps.code_rag --repo-dir ./src --query "How does search work?"
## Python Version
Requires Python 3.10+ (uses PEP 604 union syntax `X | Y`).
+
+
+
+
+# Agent Coding Guidelines
+
+## General
+- Voice input may contain typos — interpret intent, not literal text.
+- When you encounter a problem, fix it immediately and keep going until there are no more problems.
+- Do not ask about ordering or sequencing — figure it out. If something is unclear, note it and skip it; only escalate when all paths are blocked.
+- Obvious bugs: fix silently without reporting.
+- No fallbacks or compatibility shims. One correct implementation per feature — no redundancy.
+
+## Roadmap
+- Public roadmap: `docs/roadmap.md` — tracks P0/P1 priorities, completed milestones, and timeline.
+- Long-term vision: `docs/ultimate_goal.md` — the north star for where LEANN is headed.
+- Keep in sync with [GitHub issue #237](https://github.com/yichuan-w/LEANN/issues/237).
+- Welcome everyone to add more, and the craziest feature you want to put here! If people want some feature, all put there.
+
+## Changelog (for contributors)
+- Maintain `docs/CHANGELOG.md` — append-only log of major changes (new features, breaking changes, important fixes).
+- Format: `## YYYY-MM-DD: <short summary>` followed by bullet points.
+- Update the changelog when merging significant PRs or completing notable work.
+- See `docs/CONTRIBUTING.md` for full contributor workflow (conventional commits, PR process, CI).
+
+## Personal Dev Notes (gitignored)
+- `docs/dev/` is gitignored for personal development notes (TODO, progress, experiments).
+- Use `docs/dev/TODO.md` for in-progress tasks, `docs/dev/PROGRESS.md` for completed work.
+- These are private scratch space — but must follow the Self-Contained Principle below.
+
+## Documentation — Self-Contained Principle
+
+All dev docs (`PROGRESS.md`, `STATES.md`, `EXPERIMENTS.md`, `TODO.md`) must be fully understandable from the document alone, with no reliance on conversation context or implied knowledge.
+
+Requirements:
+1. **Every technique/approach must be explained on first use.** Not "switched to IVF backend" — write "switched to IVF backend (FAISS IndexIVFFlat + DirectMap.Hashtable, supports in-place add/remove without full index rebuild)."
+2. **Never assume the reader knows any abbreviation.** On first use: full name + one-sentence explanation. E.g., "HNSW (Hierarchical Navigable Small World — a graph-based ANN index used as LEANN's default backend)."
+3. **Benchmark results must include full context.** Not "recall improved to 0.95" — write "recall@10 improved from 0.91 to 0.95 after switching from flat chunking (512 tokens, no overlap) to AST-aware chunking (function-level splits with 64-token overlap)."
+4. **Numbers must have reference points.** Not "build time: 12s" — write "build time: 12s (vs. 45s before incremental update support, on 10k-document corpus)."
+5. **Include the causal chain — not just conclusions.** Not "duplicate chunks appeared after incremental build" — write "Duplicate chunks appeared after incremental build because `passages.jsonl` was appended without first removing stale entries for modified files. The IVF index had correct vectors (remove-then-add), but the passage store was append-only, causing the same text to appear at multiple offsets."
+6. **`docs/dev/STATES.md` top section maintains a glossary** of all key terms (backends, index files, chunking strategies, embedding models). Other docs reference it at the top.
+
+Bad examples (forbidden):
+- "Fixed the chunking bug" → Which bug? What was the symptom? What was the root cause?
+- "Improved search quality" → By what metric? From what baseline? What change caused it?
+- "Used nprobe=32" → What is nprobe? Why 32? What was it before and what effect did the change have?
+
+## Doc Maintenance
+- Maintain `docs/dev/PROGRESS.md` — completed work only (with key script/log/config paths). No plans.
+- Maintain `docs/dev/TODO.md` — incomplete/in-progress/next-steps only (aim for one-command reproducibility). When done: remove from TODO, write result to PROGRESS, update STATES/EXPERIMENTS if needed.
+- Both files: **append-only, chronological order** (oldest first). Use `tail -n 80 docs/dev/PROGRESS.md` to read recent entries; increase range or grep by date/keyword if needed.
+- Keep TODO clean — either do items or remove them. Ask the user when unsure how to handle a TODO item.
+- Maintain `docs/dev/STATES.md` — tracks all currently useful state (index configs, backend choices, known limitations); does NOT grow indefinitely (delete stale entries).
+- Maintain `docs/dev/EXPERIMENTS.md` — benchmarks, A/B comparisons, parameter sweeps (recall@k, latency, storage size). Experimental content goes here, not in STATES.md.
+
+## Commits
+Commit when: (1) a complete feature is finished and tested, or (2) a destructive change is unavoidable.
+```bash
+git add <specific files>
+git commit -m “feat: ...” # follow conventional commits
+```
+- When correcting errors: fix directly with no trace of the error.
+- If you write a correct new version of a file, delete the wrong version. No duplicate implementations.
diff --git a/docs/roadmap.md b/docs/roadmap.md
index fa04b5c..2f87114 100644
--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@@ -1,21 +1,67 @@
-# 📈 Roadmap
+# LEANN Roadmap
-## 🎯 Q2 2025
+LEANN aims to be a **personal knowledge layer** — not just a storage-efficient vector database, but a unified, always-up-to-date knowledge base that runs entirely on your own machine. It connects your code, images, and personal data (documents, emails, browser history, chats) into a single multimodal search interface.
-- [X] HNSW backend integration
-- [X] DiskANN backend with MIPS/L2/Cosine support
-- [X] Real-time embedding pipeline
-- [X] Memory-efficient graph pruning
+Contributions and feedback are welcome. Join our [Slack](https://join.slack.com/t/leann-e2u9779/shared_invite/zt-3ol2ww9ic-Eg_kB8omwe6xmYVd0epr4Q) to discuss.
-## 🚀 Q3 2025
+---
-- [ ] Advanced caching strategies
-- [ ] Add contextual-retrieval https://www.anthropic.com/news/contextual-retrieval
-- [ ] Add sleep-time-compute and summarize agent! to summarilze the file on computer!
-- [ ] Add OpenAI recompute API
+## Completed
-## 🌟 Q4 2025
+- [x] HNSW backend integration
+- [x] DiskANN backend with MIPS/L2/Cosine support
+- [x] Real-time embedding pipeline
+- [x] Memory-efficient graph pruning
+- [x] IVF backend with incremental add/remove (#231, #89, #141)
+- [x] Merkle tree file-change detection — `leann watch` (#41)
-- [ ] Integration with LangChain/LlamaIndex
-- [ ] Visual similarity search
-- [ ] Query rewrtiting, rerank and expansion
+---
+
+## P0 — Core (Q1 2026)
+
+### LEANN MCP — The Best Code Retrieval MCP
+
+The primary near-term goal: make LEANN the go-to MCP server for code-aware AI assistants. This means **dynamic updates** (your index stays current as you edit code), **rich code context** (AST-aware chunking that understands functions, classes, and modules — not just raw text), and a **dead-simple interface** (one command to build, automatic incremental updates, zero configuration for common setups).
+
+- [x] IVF backend — incremental add/remove without full rebuild (#231, #89, #141)
+- [x] Merkle tree file-change detection — `leann watch` for automatic re-indexing on file changes (#41)
+- [ ] Cold start optimization — faster first-build experience for new users (#166, #177)
+- [ ] Live index updates — push index changes as files are saved, not just on rebuild
+- [ ] Smarter code context — cross-file symbol resolution, call graph awareness, import tracking
+
+### Search Quality
+
+- [ ] Hybrid search — combine dense vector retrieval with sparse keyword matching (BM25) for better recall on exact identifiers and variable names (#233, #90)
+
+### Documentation
+
+- [ ] ReadTheDocs — hosted documentation site (#234)
+- [ ] Benchmarks — recall@k, latency, and storage comparisons across backends
+
+---
+
+## P1 (Q2 2026)
+
+### Multimodal
+
+- [ ] Video retrieval (#160)
+- [ ] CLIP support — image-text cross-modal search (#94)
+- [ ] OCR — extract text from images/scanned documents (#158)
+
+### Platform & Distribution
+
+- [ ] Windows support (#14)
+- [ ] Web UI (#229)
+
+### Applications & Integrations
+
+- [ ] Agent + Deep research (#104)
+- [ ] Local Cursor — local model + local retrieval for code assistance (#47)
+- [ ] LlamaIndex integration (#217)
+- [ ] Obsidian support (#96)
+
+---
+
+## Contributing
+
+If you're interested in working on any of the items above, please reach out to [@yichuan-w](https://github.com/yichuan-w) or [@andylizf](https://github.com/andylizf). See [CONTRIBUTING.md](CONTRIBUTING.md) for the full contributor workflow.
diff --git a/docs/ultimate_goal.md b/docs/ultimate_goal.md
new file mode 100644
index 0000000..8d2d7bd
--- /dev/null
+++ b/docs/ultimate_goal.md
@@ -0,0 +1,41 @@
+# LEANN — Long-Term Vision
+
+## The Best Personal Data Management Platform
+
+LEANN's ultimate goal is to be the **unified personal knowledge layer** that lives on your machine. Not a cloud service. Not a SaaS product. A local-first system that understands everything you've ever worked on — code, documents, emails, chats, browser history, images — and makes it all instantly searchable and usable.
+
+### 1. Continuous Learning from Your Context
+
+LEANN should get smarter over time by continuously ingesting and indexing your data as you produce it:
+
+- **Always-on indexing**: Watch your filesystem, email, browser, and chat sources. Incrementally update indexes as new data arrives — no manual rebuilds.
+- **Cross-source connections**: Surface relationships across data sources. A Slack conversation about a bug should link to the relevant code change, the related email thread, and the document that describes the feature.
+- **Temporal awareness**: Understand when things happened. "What was I working on last Tuesday?" should be a searchable query.
+- **Personalized ranking**: Learn from your search patterns and usage to rank results by relevance to *you*, not just semantic similarity.
+
+### 2. The Best MCP for Code Retrieval
+
+AI coding assistants are only as good as the context they receive. LEANN aims to be the best context provider:
+
+- **AST-aware chunking**: Understand code structure — functions, classes, modules — not just raw text blocks. Retrieve semantically meaningful units.
+- **Dynamic index updates**: The index stays current as you edit. No stale results. No manual rebuilds. Save a file, and the index reflects the change within seconds.
+- **Cross-file understanding**: Resolve symbols across files. When you search for a function, also surface its callers, its tests, and the types it depends on.
+- **Dead-simple interface**: One command to build (`leann build`), automatic incremental updates, MCP server that any AI assistant can plug into with zero configuration.
+- **Repository-scale search**: Handle monorepos and large codebases without breaking a sweat. The storage efficiency (97% reduction vs. traditional vector DBs) makes this feasible on a laptop.
+
+### 3. Multimodal Knowledge Base
+
+Text is just the beginning:
+
+- **Image search**: CLIP-based retrieval for screenshots, diagrams, photos. "Find the architecture diagram from last month."
+- **Video retrieval**: Index video content — lectures, meetings, screen recordings — and search by what was said or shown.
+- **OCR pipeline**: Extract and index text from scanned documents, handwritten notes, whiteboard photos.
+- **Unified search**: One query searches across all modalities. The answer might be in a PDF, a screenshot, a code comment, or a Slack message.
+
+### 4. Platform for Personal AI
+
+Looking further ahead, LEANN becomes the memory and retrieval layer for personal AI agents:
+
+- **Agent memory**: AI agents that remember your preferences, past decisions, and project context across sessions.
+- **Deep research**: Agents that can search your entire personal knowledge base to answer complex questions, synthesize information, and generate insights.
+- **Proactive suggestions**: Surface relevant context before you ask for it — "You discussed this exact problem with a colleague 3 months ago, here's what you decided."
diff --git a/packages/leann-core/src/leann/cli.py b/packages/leann-core/src/leann/cli.py
index 4afb0c3..5a178f9 100644
--- a/packages/leann-core/src/leann/cli.py
+++ b/packages/leann-core/src/leann/cli.py
@@ -1303,6 +1303,9 @@ Examples:
# exclude_hidden only affects directory scans; input_files are explicit
filename_as_id=True,
).load_data()
+ for doc in file_docs:
+ if not doc.metadata.get("source"):
+ doc.metadata["source"] = doc.metadata.get("file_path", "")
all_documents.extend(file_docs)
print(
f" ✅ Loaded {len(file_docs)} document{'s' if len(file_docs) > 1 else ''}"
@@ -1584,7 +1587,9 @@ Examples:
# Check if this is a code file based on source path
source_path = doc.metadata.get("source", "")
file_path = doc.metadata.get("file_path", "")
- is_code_file = any(source_path.endswith(ext) for ext in code_file_exts)
+ is_code_file = any(
+ (source_path or file_path).endswith(ext) for ext in code_file_exts
+ )
# For code files, prepend line numbers so chunks carry them
if is_code_file:
@@ -1621,7 +1626,7 @@ Examples:
first_nl = text.find("\n")
if first_nl != -1:
text = text[first_nl + 1 :]
- all_texts.append({"text": text, "metadata": chunk_metadata})
+ all_texts.append({"text": text, "metadata": chunk_metadata.copy()})
print(f"Loaded {len(documents)} documents, {len(all_texts)} chunks")
return all_texts