[MLsys2026 Best Paper]: https://arxiv.org/abs/2506.08276. RAG on Everything with LEANN. Enjoy 97% storage savings while running a fast, accurate, and 100% private RAG application on your personal device.
Python aifaissgpt-osslangchainllama-indexllmlocalstorageoffline-firstollamaprivacypythonragretrieval-augmented-generationvector-databasevector-searchvectors
docs: comprehensive update and perf: optimized list command with AST-aware recommendations
Tolga Karatas3014ed41b423
diff --git a/README.md b/README.mdindex 6878666..d0fc6bb 100755--- a/README.md+++ b/README.md@@ -1086,8 +1086,9 @@ leann remove my-docs**Key CLI features:**- Auto-detects document formats (PDF, TXT, MD, DOCX, PPTX, XLSX, MM + code files)-- **🚀 Advanced PDF Processing**: Multi-layer fallback chain (PyMuPDF → pypdf → pdfplumber → Docling OCR) for maximum reliability-- **🧠 AST-aware chunking** for Python, Java, C#, TypeScript files+- **🚀 Advanced PDF Processing**: Multi-layer fallback chain ([PyMuPDF](https://github.com/pymupdf/PyMuPDF) → [pypdf](https://github.com/py-pdf/pypdf) → [pdfplumber](https://github.com/jsvine/pdfplumber) → [Docling OCR](https://github.com/docling-project/docling)) for maximum reliability+- **⚡ High Performance**: Optimized index discovery and caching for instant CLI response even with TB-scale vaults+- **🧠 AST-aware chunking** for Python, Java, C#, TypeScript files. Use larger chunks (`--ast-chunk-size 1000`) for complex logic analysis!- **📱 Integrated Source Indexing**: Dedicated commands for Mail, Calendar, iMessage, and more- Smart text chunking with overlap for all other content- Multiple LLM providers (Ollama, OpenAI, HuggingFace)diff --git a/docs/features.md b/docs/features.mdindex b9b1d73..a7182bb 100644--- a/docs/features.md+++ b/docs/features.md@@ -14,13 +14,13 @@- **🚀 Advanced PDF Pipeline**: Intelligent fallback chain featuring [PyMuPDF](https://github.com/pymupdf/PyMuPDF), [pypdf](https://github.com/py-pdf/pypdf), [pdfplumber](https://github.com/jsvine/pdfplumber), and [**Docling OCR**](https://github.com/docling-project/docling) (IBM Research) for high-fidelity document parsing.- **💼 Office Document Extractors**: Structural awareness for Word ([python-docx](https://github.com/python-openxml/python-docx)), Excel ([openpyxl](https://github.com/ericgazoni/openpyxl)), and PowerPoint ([python-pptx](https://github.com/python-openxml/python-pptx)) preserving tables, slides, and sheets.- **🧠 Mindmap Parsing**: Hierarchical node extraction for [FreeMind](http://freemind.sourceforge.net/) and [Freeplane](https://www.freeplane.org/) (`.mm`) preserving semantic relationships in zettelkasten-like structures.-- **📱 Integrated Source Connectors**: Dedicated CLI commands for:- - **Apple Mail**: Direct indexing of the macOS Mail SQLite database (`Envelope Index`).- - **Apple Calendar**: Indexing of local events and descriptions from the macOS Calendar cache.- - **iMessage**: Grouping and indexing of local chat history from the `chat.db` database.- - **WeChat**: Parsing and indexing of [WeChat](https://www.wechat.com/) chat history from JSON exports.- - **MCP Integration**: Real-time retrieval from [Slack](https://slack.com/), [Twitter/X](https://x.com/), and more through the [Model Context Protocol](https://modelcontextprotocol.io/).- - **AI Conversation Archives**: Direct indexing of [ChatGPT](https://chat.openai.com/) and [Claude](https://claude.ai/) conversation data exports.+- **📱 Integrated Source Connectors**: Dedicated CLI commands for Apple Mail, Calendar, iMessage, WeChat, and more.++## ⚡ Performance & Scalability++- **🚀 Instant CLI Discovery**: Optimized `leann list` and `leann remove` commands use intelligent scanning and size caching to respond in milliseconds, regardless of the number of registered projects.+- **📈 Huge Vault Support**: Native **DiskANN** backend integration allows indexing terabytes of data (millions of chunks) without exceeding laptop RAM limits.+- **🧠 Advanced Code Analysis**: AST-aware chunking supports complex logic retrieval. For large functions (e.g., algorithmic trading strategies), using `--ast-chunk-size 1000` ensures that entire logical blocks stay in the same context for the LLM.## 🛠️ Technical Highlights- **🔄 Recompute Mode** - Highest accuracy scenarios while eliminating vector storage overheaddiff --git a/packages/leann-core/src/leann/cli.py b/packages/leann-core/src/leann/cli.pyindex 341ef44..a7b9a6a 100644--- a/packages/leann-core/src/leann/cli.py+++ b/packages/leann-core/src/leann/cli.py@@ -960,11 +960,19 @@ Examples:try:local_indexes_dir = project_path / ".leann" / "indexes"if local_indexes_dir.exists():- # Avoid duplicate scan if local matches central+ # Avoid duplicate scan if local matches central or is inside itlocal_resolved = local_indexes_dir.resolve()central_resolved = self.indexes_dir.resolve()- if local_resolved != central_resolved:- dirs_to_check.append(local_indexes_dir)+ try:+ if local_resolved != central_resolved and not local_resolved.is_relative_to(+ central_resolved+ ):+ dirs_to_check.append(local_indexes_dir)+ except AttributeError: # Python < 3.9 fallback+ if str(local_resolved) != str(central_resolved) and str(+ central_resolved+ ) not in str(local_resolved):+ dirs_to_check.append(local_indexes_dir)except (OSError, PermissionError):pass@@ -1114,10 +1122,10 @@ Examples:"""Find all indexes with the given name across all projects"""matches = []- # Get all registered projects+ # Optimization: use the same discovery logic as list_indexes but filter by name+ # First, find all projectsglobal_registry = Path.home() / ".leann" / "projects.json"all_projects = []-if global_registry.exists():try:import json@@ -1127,87 +1135,42 @@ Examples:except Exception:pass- # Always include current projectcurrent_path = Path.cwd()- if str(current_path) not in all_projects:- all_projects.append(str(current_path))-- # Search across all projects- for project_dir in all_projects:- project_path = Path(project_dir)- if not project_path.exists():- continue-- # 1) CLI-format index under .leann/indexes/<name>- index_dir = project_path / ".leann" / "indexes" / index_name- if index_dir.exists():- is_current = project_path == current_path- matches.append(- {- "project_path": project_path,- "index_dir": index_dir,- "is_current": is_current,- "kind": "cli",- }- )-- # 2) App-format indexes- # We support two ways of addressing apps:- # a) by the file base (e.g., `pdf_documents`)- # b) by the parent directory name (e.g., `new_txt`)- seen_app_meta = set()-- # 2a) by file base- for meta_file in project_path.rglob(f"{index_name}.leann.meta.json"):- if meta_file.is_file():- # Skip CLI-built indexes' meta under .leann/indexes- try:- cli_indexes_dir = project_path / ".leann" / "indexes"- if cli_indexes_dir.exists() and cli_indexes_dir in meta_file.parents:- continue- except Exception:- pass- is_current = project_path == current_path- key = (str(project_path), str(meta_file))- if key in seen_app_meta:+ project_paths = [Path(p) for p in all_projects if Path(p).exists()]+ if current_path not in project_paths:+ project_paths.append(current_path)++ # Track seen paths to avoid duplicates+ seen_dirs = set()++ for project_path in project_paths:+ # Discover indexes in this project using the optimized method+ # We exclude central from others to avoid massive redundancy+ is_current = project_path == current_path+ discovered = self._discover_indexes_in_project(project_path, include_central=is_current)++ for idx in discovered:+ if idx["name"] == index_name:+ # Resolve path for deduplication+ path_resolved = idx["path"].resolve()+ if path_resolved in seen_dirs:continue- seen_app_meta.add(key)- matches.append(- {- "project_path": project_path,- "files_dir": meta_file.parent,- "meta_file": meta_file,- "is_current": is_current,- "kind": "app",- "display_name": meta_file.parent.name,- "file_base": meta_file.name.replace(".leann.meta.json", ""),- }- )+ seen_dirs.add(path_resolved)- # 2b) by parent directory name- for meta_file in project_path.rglob("*.leann.meta.json"):- if meta_file.is_file() and meta_file.parent.name == index_name:- # Skip CLI-built indexes' meta under .leann/indexes- try:- cli_indexes_dir = project_path / ".leann" / "indexes"- if cli_indexes_dir.exists() and cli_indexes_dir in meta_file.parents:- continue- except Exception:- pass- is_current = project_path == current_path- key = (str(project_path), str(meta_file))- if key in seen_app_meta:- continue- seen_app_meta.add(key)matches.append({"project_path": project_path,- "files_dir": meta_file.parent,- "meta_file": meta_file,+ "index_dir": idx["path"]+ if idx["type"] == "cli"+ else idx["path"].parent,+ "meta_file": idx["path"]+ if idx["type"] == "app"+ else (idx["path"] / "documents.leann.meta.json"),"is_current": is_current,- "kind": "app",- "display_name": meta_file.parent.name,- "file_base": meta_file.name.replace(".leann.meta.json", ""),+ "kind": idx["type"],+ "display_name": idx["name"],+ "file_base": idx["path"].name if idx["type"] == "app" else "documents",+ "path": idx["path"],})@@ -1231,7 +1194,7 @@ Examples:print(f"✅ Found 1 index named '{index_name}':")print(f" {emoji} Location: {location_info}")if kind == "cli":- print(f" 📍 Path: {project_path / '.leann' / 'indexes' / index_name}")+ print(f" 📍 Path: {match['path']}")else:print(f" 📍 Meta: {match['meta_file']}")@@ -1254,7 +1217,7 @@ Examples:)else:return self._delete_index_directory(- match["files_dir"],+ match["index_dir"],match.get("display_name", index_name),project_path if not is_current else None,is_app=True,@@ -1280,7 +1243,7 @@ Examples:# Show path detailsif kind == "cli":- print(f" 📍 {project_path / '.leann' / 'indexes' / index_name}")+ print(f" 📍 {match['path']}")else:print(f" 📍 {match['meta_file']}")@@ -1296,7 +1259,7 @@ Examples:if file_base:size_mb = sum(f.stat().st_size- for f in match["files_dir"].glob(f"{file_base}.leann*")+ for f in match["index_dir"].glob(f"{file_base}.leann*")if f.is_file()) / (1024 * 1024)print(f" 📦 Size: {size_mb:.1f} MB")@@ -1306,8 +1269,12 @@ Examples:print(" " + "─" * 50)if force:- print(" ❌ Multiple matches found, but --force specified.")- print(" Please run without --force to choose which one to remove.")+ # If all matches point to the same physical location (should be handled by deduplication, but just in case)+ # Or if the user really wants to force delete everything with this name.+ # But for safety, we only allow force delete if there's no ambiguity.+ # Since we added deduplication, multiple matches now means DIFFERENT physical locations.+ print(" ❌ Multiple matches found at DIFFERENT locations, but --force specified.")+ print(" Please run without --force to choose which one to remove safely.")return Falsetry:@@ -1345,7 +1312,7 @@ Examples:)else:return self._delete_index_directory(- selected_match["files_dir"],+ selected_match["index_dir"],selected_match.get("display_name", index_name),project_path if not is_current else None,is_app=True,