Skip to content

armgit / LEANN

Public Imported from GitHub
0

[MLsys2026 Best Paper]: https://arxiv.org/abs/2506.08276. RAG on Everything with LEANN. Enjoy 97% storage savings while running a fast, accurate, and 100% private RAG application on your personal device.

docs: restructure all documentation into Diataxis methodology

Tolga Karatas2bbf738aee19
diff --git a/README.md b/README.md
index 33ddea8..51a12d6 100755
--- a/README.md
+++ b/README.md
@@ -36,39 +36,49 @@ LEANN is an innovative vector database that democratizes personal AI. Transform
LEANN achieves this through *graph-based selective recomputation* with *high-degree preserving pruning*, computing embeddings on-demand instead of storing them all. [Illustration Fig →](#️-architecture--how-it-works) | [Paper →](https://arxiv.org/abs/2506.08276)
-**Ready to RAG Everything?** Transform your laptop into a personal AI assistant that can semantic search your **[file system](#-personal-data-manager-process-any-documents-pdf-txt-md)**, **[emails](#-your-personal-email-secretary-rag-on-apple-mail)**, **[browser history](#-time-machine-for-the-web-rag-your-entire-browser-history)**, **[chat history](#-wechat-detective-unlock-your-golden-memories)** ([WeChat](https://www.wechat.com/), [iMessage](https://support.apple.com/messages)), **[agent memory](#-chatgpt-chat-history-your-personal-ai-conversation-archive)** ([ChatGPT](https://chat.openai.com/), [Claude](https://claude.ai/)), **[live data](#mcp-integration-rag-on-live-data-from-any-platform)** ([Slack](https://slack.com/), [Twitter/X](https://x.com/)), **[codebase](#-claude-code-integration-transform-your-development-workflow)**\* , or external knowledge bases (i.e., 60M documents) - all on your laptop, with zero cloud costs and complete privacy.
+**Ready to RAG Everything?** Transform your laptop into a personal AI assistant that can semantic search your:
+- 📂 **[File System](#-personal-data-manager)**: Any folder with PDFs, Word, Excel, PowerPoint, and Mindmaps.
+- 📧 **[Emails](#-your-personal-email-secretary-rag-on-apple-mail)**: Direct indexing of [Apple Mail](https://support.apple.com/mail).
+- 📅 **[Calendar](#-your-personal-calendar-assistant)**: Search events in [Apple Calendar](https://support.apple.com/calendar).
+- 🌐 **[Browser History](#-time-machine-for-the-web-rag-your-entire-browser-history)**: Your journey through [Chrome](https://www.google.com/chrome/) or [Brave](https://brave.com/).
+- 💬 **[Chat History](#-wechat-detective-unlock-your-golden-memories)**: Memories from [WeChat](https://www.wechat.com/) or [iMessage](https://support.apple.com/messages).
+- 🤖 **[Agent Memory](#-chatgpt-chat-history-your-personal-ai-conversation-archive)**: Archive of [ChatGPT](https://chatgpt.com/) and [Claude](https://claude.ai/) chats.
+- 📡 **[Live Data](#mcp-integration-rag-on-live-data-from-any-platform)**: Real-time data from [Slack](https://slack.com/) and [Twitter/X](https://x.com/) via [MCP](https://modelcontextprotocol.io/).
-* Claude Code only supports basic `grep`-style keyword search. **LEANN** is a drop-in **semantic search MCP service fully compatible with Claude Code**, unlocking intelligent retrieval without changing your workflow. 🔥 Check out [the easy setup →](packages/leann-mcp/README.md)
+---
-## Why LEANN?
+## 📚 Documentation
-<p align="center">
- <img src="assets/effects.png" alt="LEANN vs Traditional Vector DB Storage Comparison" width="70%">
-</p>
-
-> **The numbers speak for themselves:** Index 60 million text chunks in just 6GB instead of 201GB. From emails to browser history, everything fits on your laptop. [See detailed benchmarks for different applications below ↓](#-storage-comparison)
-
-
-🔒 **Privacy:** Your data never leaves your laptop. No OpenAI, no cloud, no "terms of service".
+LEANN uses a structured documentation approach to get you up to speed quickly:
-🪶 **Lightweight:** Graph-based recomputation eliminates heavy embedding storage, while smart graph pruning and CSR format minimize graph storage overhead. Always less storage, less memory usage!
+- 🚀 **[Getting Started](docs/getting-started.md)**: Install LEANN and build your first index in minutes.
+- 📧 **[Ingestion Guides](docs/ingestion-guides.md)**: How to index emails, calendar, codebases, and large-scale archives.
+- 🏗️ **[Architecture & Concepts](docs/architecture-concepts.md)**: Deep dive into selective recomputation, AST-aware chunking, and backends.
+- 💻 **[CLI Reference](docs/cli-reference.md)**: Complete list of commands, flags, and environment variables.
-📦 **Portable:** Transfer your entire knowledge base between devices (even with others) with minimal cost - your personal AI memory travels with you.
+## 🚀 Key Features
-📈 **Scalability:** Handle messy personal data that would crash traditional vector DBs, easily managing your growing personalized data and agent generated memory!
+- **📑 Universal Ingestion**: Native support for `.pdf`, `.docx`, `.xlsx`, `.pptx`, `.mm` (Mindmaps), and code files.
+- **🚀 Advanced PDF Processing**: Intelligent fallback chain featuring [PyMuPDF](https://github.com/pymupdf/PyMuPDF), [pypdf](https://github.com/py-pdf/pypdf), and [**Docling OCR**](https://github.com/docling-project/docling).
+- **🧠 AST-Aware Chunking**: Semantic code parsing that understands function and class boundaries.
+- **⚡ High Performance**: Optimized index discovery and terabyte-scale support via **DiskANN**.
+- **📱 Built-in Connectors**: Index your Mail, Calendar, iMessage, and Browser history with zero config.
+- **🔒 Private & Local**: Your data stays on your machine. Compatible with [Ollama](https://ollama.ai).
-✨ **No Accuracy Loss:** Maintain the same search quality as heavyweight solutions while using 97% less storage.
+## 🛠️ Installation
-## Installation
-
-### 📦 Prerequisites: Install uv
-
-[Install uv](https://docs.astral.sh/uv/getting-started/installation/#installation-methods) first if you don't have it. Typically, you can install it with:
+Quickest way to install globally via `uv`:
```bash
-curl -LsSf https://astral.sh/uv/install.sh | sh
+uv tool install leann \
+ --with leann-backend-hnsw \
+ --with leann-backend-diskann \
+ --with astchunk-leann
```
+For source builds or low-resource setups, see the **[Getting Started Guide](docs/getting-started.md)**.
+
+
### 🚀 Quick Install
Clone the repository to access all examples and try amazing applications,
diff --git a/docs/architecture-concepts.md b/docs/architecture-concepts.md
new file mode 100644
index 0000000..e0c1997
--- /dev/null
+++ b/docs/architecture-concepts.md
@@ -0,0 +1,27 @@
+# Architecture & Concepts
+
+Understanding the technology that makes LEANN the smallest vector index in the world.
+
+## Selective Recomputation
+
+Traditional vector databases store every embedding (heavy floating-point vectors) in RAM or on disk. For 60 million chunks, this can take over 200GB.
+
+LEANN achieves **97% storage savings** by storing only the raw text and a lightweight relational graph. During a search, it recomputes only the necessary embeddings on-the-fly. This shifts the burden from storage to compute, which is ideal for modern laptops.
+
+## AST-Aware Chunking
+
+Standard RAG systems break code into chunks based on fixed line counts. This often splits a function in the middle, destroying the context for the LLM.
+
+LEANN's **Abstract Syntax Tree (AST)** parser understands code structure. It identifies function and class boundaries, ensuring that the retriever always returns a complete, logical block of code.
+
+## Backend Selection: HNSW vs. DiskANN
+
+- **HNSW**: Best for small to medium indexes (< 500k chunks). It is extremely fast but keeps the index structure in RAM.
+- **DiskANN**: Optimized for V-Scale data. It uses memory-mapped files to handle millions of chunks without crashing your laptop's memory.
+
+## The Docling Pipeline
+
+For PDF processing, LEANN uses a multi-layer fallback system to ensure 100% reliability:
+1. **PyMuPDF**: Fast, high-quality layout extraction.
+2. **pypdf/pdfplumber**: Secondary fallbacks for specialized structures.
+3. **Docling (IBM Research)**: The ultimate fallback using OCR and multimodal models for scanned or highly complex documents.
diff --git a/docs/cli-reference.md b/docs/cli-reference.md
new file mode 100644
index 0000000..b13fcb5
--- /dev/null
+++ b/docs/cli-reference.md
@@ -0,0 +1,50 @@
+# CLI Reference
+
+Comprehensive list of LEANN commands and options.
+
+## Global Environment Variables
+
+| Variable | Description | Default |
+| :--- | :--- | :--- |
+| `LEANN_HOME` | Directory where all indexes are stored | `~/.leann` |
+| `OLLAMA_HOST`| Host address for the Ollama API | `http://localhost:11434` |
+
+## Primary Commands
+
+### `build`
+Create a new index from local files.
+- `--docs`: Path to directory or file.
+- `--backend-name`: `hnsw` or `diskann`.
+- `--use-ast-chunking`: Enable code-aware parsing.
+- `--doc-chunk-size`: Token limit per chunk (default: 256).
+
+### `search`
+Fast semantic retrieval.
+- `--top-k`: Number of results to return.
+- `--no-recompute`: Faster search using cached embeddings (if available).
+- `--complexity`: Search depth (higher = more accurate, slower).
+
+### `ask`
+Interactive RAG session.
+- `--model`: LLM model name (e.g., `gpt-oss:20b`).
+- `--interactive`: Start a persistent chat session.
+
+## Source-Specific Commands
+
+| Command | Target Source |
+| :--- | :--- |
+| `index-email` | Apple Mail (local SQLite) |
+| `index-calendar`| Apple Calendar events |
+| `index-imessage`| iMessage history |
+| `index-browser` | Chrome/Brave history |
+| `index-chatgpt` | ChatGPT JSON/ZIP exports |
+| `index-claude` | Claude JSON/ZIP exports |
+
+## Management Commands
+
+### `list`
+Instantly list all indexes across all registered projects. Highly optimized for large storage systems.
+
+### `remove`
+Safely delete an index.
+- `-f`, `--force`: Bypass confirmation (only if no naming ambiguity exists).
diff --git a/docs/getting-started.md b/docs/getting-started.md
new file mode 100644
index 0000000..882ee91
--- /dev/null
+++ b/docs/getting-started.md
@@ -0,0 +1,55 @@
+# Getting Started with LEANN
+
+Welcome! This tutorial will guide you through installing LEANN and performing your first semantic search in under 5 minutes.
+
+## 1. Prerequisites
+
+LEANN requires **Python 3.10+** and [uv](https://docs.astral.sh/uv/) for the best installation experience.
+
+**For macOS Users:**
+Install system dependencies for the high-performance backends:
+```bash
+brew install libomp boost protobuf zeromq pkgconf
+```
+
+## 2. Global Installation
+
+Install LEANN as a global tool with all supported backends (HNSW, DiskANN, and AST-aware chunking):
+
+```bash
+uv tool install leann \
+ --with leann-backend-hnsw \
+ --with leann-backend-diskann \
+ --with astchunk-leann
+```
+
+Verify the installation:
+```bash
+leann --help
+```
+
+## 3. Your First Index
+
+Pick a folder containing some text files, PDFs, or Markdown notes and create an index named `my-notes`:
+
+```bash
+leann build my-notes --docs ~/Documents/Notes
+```
+
+## 4. Semantic Search
+
+Find information in your notes using natural language instead of keywords:
+
+```bash
+leann search my-notes "What were the main points of the last meeting?"
+```
+
+## 5. Interactive RAG
+
+Chat with your data using a local LLM via Ollama (ensure Ollama is running):
+
+```bash
+leann ask my-notes "Summarize the project goals mentioned in my notes" --interactive
+```
+
+Next: Check out the [Ingestion Guides](ingestion-guides.md) to learn how to index your emails, calendar, and code.
diff --git a/docs/ingestion-guides.md b/docs/ingestion-guides.md
new file mode 100644
index 0000000..59ffb1c
--- /dev/null
+++ b/docs/ingestion-guides.md
@@ -0,0 +1,58 @@
+# Ingestion Guides
+
+This guide provides practical recipes for indexing various data sources and optimizing LEANN for different scales.
+
+## Personal Data (macOS)
+
+LEANN features built-in connectors for your local communication data.
+
+### Apple Mail
+Index your local email database for semantic retrieval:
+```bash
+leann index-email --max-items 5000
+```
+*Note: Requires "Full Disk Access" for your Terminal app in macOS System Settings.*
+
+### Apple Calendar
+Make your schedule searchable:
+```bash
+leann index-calendar
+```
+
+### iMessage History
+Index your chats grouped by conversation:
+```bash
+leann index-imessage --max-items 2000
+```
+
+## Code Intelligence
+
+### AST-Aware Chunking
+For deep analysis of source code (Python, JS, TS, etc.), use the AST-aware parser. It ensures logical blocks like functions and classes stay together.
+
+**For complex logic (e.g., Trading Strategies):**
+Use a larger chunk size to keep entire functions in one context:
+```bash
+leann build my-code --docs ~/src --use-ast-chunking --ast-chunk-size 1000
+```
+
+## Large Scale Archives
+
+### Using DiskANN
+For vaults exceeding 10GB or containing millions of chunks, the **DiskANN** backend is recommended to save RAM:
+
+```bash
+leann build big-vault --docs /path/to/archive --backend-name diskann
+```
+
+### Deep Directory Optimization
+If your folder structure is very deep (e.g., Nextcloud), increase the metadata headroom:
+
+```bash
+leann build my-vault --docs ~/Nextcloud \
+ --doc-chunk-size 1024 \
+ --doc-chunk-overlap 200
+```
+
+## Automating Sync
+LEANN provides utility scripts in `scripts/sync_utilities/` to keep your vaults updated. You can run these via cron or manual triggers to refresh your data.