~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Research Library & RAG Guide
This guide covers the Research Library for document management and the RAG (Retrieval-Augmented Generation) system for semantic search.
Table of Contents
Overview
The Research Library allows you to:
- Upload documents (PDFs, text files, markdown)
- Organize into collections for different projects or topics
- Index for semantic search using RAG (vector embeddings)
- Search your documents using natural language queries
Access the library at: http://localhost:5000/library
Managing Documents
Supported File Types
| Format | Extension | Notes |
|---|---|---|
.pdf |
Text extracted automatically | |
| Plain Text | .txt |
Direct text storage |
| Markdown | .md, .markdown |
Rendered as text |
| HTML | .html, .htm |
Tags stripped, text extracted |
| Word | .docx (and .doc*) |
Text extracted via unstructured |
| OpenDocument Text | .odt |
Converted to HTML with pandoc, then text extracted |
| PowerPoint | .pptx (and .ppt*) |
Slide text extracted |
| Excel | .xlsx, .xls |
Cell text extracted |
| Rich Text | .rtf |
Text extracted |
| EPUB | .epub |
Text extracted |
.eml |
Body text extracted | |
| Data | .csv, .tsv, .json, .yaml, .yml, .xml, .toml |
Parsed to text |
| Notebooks | .ipynb |
Cell sources and outputs |
| Web archives | .mhtml, .mht |
Saved web pages |
The upload dialog's file picker is populated from the live list of formats the
server can actually parse (GET /library/api/config/supported-formats), so it
only offers formats whose parser dependencies are installed.
For PDF text extracted from bytes, including RAG uploads and Zotero sync, the server extracts text from at most 500 pages and keeps at most 10,000,000 characters. It stops starting new pages after 600 seconds of extraction-thread CPU time. Documents that reach a limit retain the extracted prefix; the current RAG and Zotero flows do not display a partial-extraction marker.
* The legacy binary formats .doc and .ppt are offered only when
LibreOffice (soffice) is installed: Local Deep Research converts each one to
.docx/.pptx with soffice itself (with a 120-second timeout) and then
extracts the converted file like any other upload. Only a genuine legacy file
is converted: one that does not start with the OLE compound-file signature
(for example a .docx renamed to .doc) is refused. Image formats (.png,
.jpg, …) are offered only when the optional OCR extras (pytesseract plus the tesseract
system binary) are installed. The default Docker image ships neither, so those
formats are not offered there.
Office, OpenDocument and EPUB uploads are size- and structure-checked before
they are parsed, and long or complex documents are refused instead of indexed,
not only hostile ones. In particular, a .docx of more than roughly 5,000 to
11,000 paragraphs (about 5,250 for a Word-like document of paragraphs of 5-10
formatted runs, about 11,300 for plain one-run paragraphs), with more
than about 580 sections of one paragraph each (as a mail merge to individual
documents writes; never more than 1,000 sections) or more than 20,000
rendered page breaks, with a table column
merged down more than about 545 rows, a .pptx of more than about 2,600
title-and-content slides (fewer under templates with many shapes per layout),
or an .xlsx of more than 2,000 sheets (or with more cell formats, or a
longer number format code, than Excel itself writes, with more than
8,192 fonts, fills or borders or 8,192 custom number formats, or with more than roughly 17,000 to
30,000 named styles, fewer with many custom number formats, or with a sheet
name over 255 characters, two sheets of the same name, a defined name over
8,192 characters, or a part openpyxl reads that is encoded in anything but
UTF-8 or nests elements more than 256 deep) is refused, as is a .docx or
.pptx with a part the checks read that is not UTF-8 (Office, LibreOffice and
the Python libraries write UTF-8; UTF-16, which OPC also allows, is refused).
The upload then reports "Could not extract text from docx file"
(or pptx, and so on) for that file, and the server log records "Refused
document: it exceeds the parsing limits for its format". Splitting can help
with size and complexity limits; for a package with UTF-16 XML parts, resave
the document in Office or LibreOffice to produce UTF-8 parts. The release
notes list every limit.
The final Office/spreadsheet, HTML, MHTML (.mht/.mhtml), Markdown,
email, XML and Evernote parser runs in a separate child process with a 90-second deadline, a 10-million-character
text limit and a 10,000-document limit, and its result is capped at 40 MB
including metadata. This includes files converted from legacy Office,
OpenDocument, EPUB and pandoc-backed formats. On Linux, each parser child
also has a soft 3 GiB data-memory limit, of which the parser libraries
themselves use several hundred MB. It is soft in two ways: it is the soft
RLIMIT_DATA limit, which code in the child could raise again up to the hard
limit, and it counts the data segment and private writable mappings, not the
child's whole memory use. Allocation failures reported by the parser refuse
the parse; a library may handle a failed allocation internally. This limit
does not guarantee a whole-process memory ceiling and is unavailable on other operating
systems. If the server process crashes or is killed during a parse, nothing
enforces the deadline any more, so the child also stops itself: on Linux and
macOS it is ended after 120 seconds of CPU time, and on Linux it is killed as
soon as the server process exits. Windows has neither safeguard. Every such
file starts a fresh child that imports the parser
libraries again, which adds about 2.5 to 5.5 seconds per file for a small
document on a development machine, and more on slower hosts. The upload page
sends at most 15 files per request and the server parses them one after
another, so a request of 15 such files spends about 40 to 80 seconds in
parser children on that machine before it can finish. Behind a reverse
proxy, allow for this in its read timeout: nginx's default
proxy_read_timeout is 60 seconds, and the
reverse-proxy guide's example uses 300. At
most two of these parsers run at once per server process; an upload arriving
while both are busy reports "Document parser busy with other uploads; upload
this file again" for that file instead of an extraction failure, and the same
file can be uploaded again once the other uploads finish. Plain-text, CSV,
TSV, JSON, YAML, TOML and notebook uploads retain their existing parsing
path. PDFs retain their separate page, character and CPU-time limits.
Image/OCR support depends on optional packages and system tools and is
outside this child-parser change.
Markdown and email HTML bodies use the same child boundary. Email
attachments remain excluded from parsing. Evernote notes retain their
metadata, including dates, optional values and tags. A very large Evernote
export (roughly 9,000 notes or more in one .enex file) exceeds the
per-file metadata and document limits and is not imported; export the
notebooks as several smaller files instead.
The parser child never downloads NLTK data. Its Word, PowerPoint, .xlsx,
HTML, Markdown, email (.eml) and XML parsers (unstructured) classify text
with NLTK's punkt_tab and averaged_perceptron_tagger_eng data, and so do
the formats converted to them (.doc, .ppt, .odt, .epub, .rtf, .rst, .org).
Inside this child, NLTK text processing can use the existing document
timeout, including for long paragraphs. The parent deadline and the child's
CPU, memory and output limits still bound the whole parse. Other uses of
NLTK keep their existing library defaults.
- Docker image: the image ships this data in
/usr/local/share/nltk_data, which is on NLTK's default search path for every user, the unprivilegedldruserthe container runs as included. Nothing needs installing. The build fetches each archive from a pinned commit ofnltk/nltk_dataand checks its SHA-256 (scripts/install_nltk_data.py). - pip and source installs: the pip package does not include the data.
Until usable data is available, an upload that needs it reports a server
NLTK data setup error for that file, with
the command
python -m nltk.downloader punkt_tab averaged_perceptron_tagger_eng. Run that command once with the server's Python environment, as the user that runs the server: by default it installs the data into that user's~/nltk_data(%APPDATA%\nltk_dataon Windows). To keep it elsewhere, add-d <directory>to the command and setNLTK_DATA=<directory>in the server's environment. With current NLTK releases, existing data directories and the files in them must also be owned by the server user or administrator and must not be group- or world-writable (for example, fix withchmod -R go-won the data directory). If installed data is reported as missing, check those permissions before downloading it again.
The child starts with Python's -P option and without empty PYTHONPATH
entries, which Python reads as the working directory. Set PYTHONPATH
explicitly when using a source tree; installed packages remain available.
For source-tree runs, install LDR in the server's environment (for example,
with the repository's development setup) or set PYTHONPATH to the absolute
src directory. Starting the server from inside src/ alone does not make
the package available to a child started with -P.
Importing LDR sets SCARF_NO_ANALYTICS=true in its process environment,
overwriting any existing value; parser children receive the same setting.
This disables unstructured's import-time usage request. DO_NOT_TRACK is
left unchanged.
.rtf, .epub, .rst, .org and .odt uploads are converted with pandoc
run with --sandbox, which needs pandoc 2.15 or newer: the bundled
pypandoc-binary ships a pandoc 3 build, but if an older pandoc is found
instead, every such conversion fails (the file is refused, never converted
without the sandbox).
Uploading Documents
- Navigate to Library in the sidebar
- Click Upload or drag files into the upload area
- Select a collection (or use the default "Library")
- Documents are processed and text is extracted
Storage Modes
| Mode | Description | Use Case |
|---|---|---|
| Database | PDFs stored encrypted in SQLCipher | Default, most secure |
| Text-only | Only extracted text stored | Save space |
Document Actions
- View - Open document details and extracted text
- Download PDF - Get original file (if stored)
- Download Text - Export extracted text
- Delete - Remove from library
Collections
Collections organize your documents into groups.
Creating a Collection
- Go to Library → Collections
- Click Create Collection
- Enter a name and optional description
- Click Create
Managing Collections
- Add documents - Upload directly to collection or move existing docs
- Remove documents - Documents can exist in multiple collections
- Delete collection - Choose to keep or delete orphaned documents
- Index collection - Build RAG index for semantic search
Default Collection
The "Library" collection is created automatically and serves as the default destination for uploads.
RAG Indexing
RAG (Retrieval-Augmented Generation) enables semantic search over your documents.
How It Works
Document → Split into Chunks → Generate Embeddings → Store in Vector Index
- Chunking - Documents split into overlapping segments
- Embedding - Each chunk converted to a vector using AI model
- Indexing - Vectors stored in FAISS for fast similarity search
For high CPU usage, smaller models, and Ollama GPU options, see Choosing embeddings for your hardware.
Indexing a Collection
- Go to Library → Collections
- Select a collection
- Click Index for Search (or Rebuild Index)
- Wait for indexing to complete (progress shown)
Re-Index All Documents builds a complete replacement while searches use the published index. Keep the rebuild running until it completes. Cancelling before publication discards the replacement and keeps the previous index and embedding settings, even if that index is old. Once publication begins, cancellation no longer interrupts its transaction. If publication succeeds, a late cancellation then reports that the new index is in use.
An incomplete rebuild leaves a warning on the collection, including the published index's last update time when available. This timestamp can also change during incremental indexing; it is not the date of the last full rebuild. The warning survives reloads and later incremental jobs, until a full rebuild succeeds or the index is explicitly cleared.
Automatic indexing, when enabled, can add missing or unindexed documents using the collection's existing settings. It does not resume or retry a cancelled full rebuild, and it does not apply the replacement embedding settings. Run Re-Index All Documents again to finish that change.
Index Status
| Status | Meaning |
|---|---|
| Not Indexed | Documents not searchable |
| Indexing | Currently processing |
| Indexed | Ready for semantic search |
| Needs Reindex | New documents added since last index |
Semantic Search
Once indexed, search your documents using natural language.
Using Collection Search
- Select a collection with indexed documents
- Enter a natural language query
- Results ranked by semantic similarity
Using in Research
When conducting research, you can:
- Set search tool to your collection name
- LDR will search your documents instead of the web
- Combine with web search via the default langgraph-agent strategy, which can query your collections and web engines in the same run
Example with Python API:
from local_deep_research.api import quick_summary
result = quick_summary(
query="What does the documentation say about authentication?",
search_tool="my_collection", # Use your collection name
programmatic_mode=True
)
Embedding Models
Choose the embedding model based on your needs.
Available Providers
Sentence Transformers (Local - Default)
Runs locally with no API key. Curated Sentence Transformers models, including the recommended default, require a one-time policy-authorized download and are then reused from the local cache.
| Model | Dimensions | Best For |
|---|---|---|
Alibaba-NLP/gte-modernbert-base |
768 | Recommended default; higher-quality English retrieval and longer inputs |
all-MiniLM-L6-v2 |
384 | General use (fast) |
all-mpnet-base-v2 |
768 | Higher quality |
multi-qa-MiniLM-L6-cos-v1 |
384 | Q&A tasks |
paraphrase-multilingual-MiniLM-L12-v2 |
384 | Multi-language |
On Library → Embedding Settings, temporarily use Public only, or Adaptive with a public primary and no private collection selected. Turn off Require local embeddings if you enabled it, select the model, and use Test Embedding Model. Once the test succeeds, return to your preferred scope; subsequent inference uses the local cache. Collection content is not used for this test.
New Hugging Face downloads are restricted to the vetted entries above. Existing cached models outside that list and confined local models can be reused offline when compatible with safetensors weights and disabled remote code. Binary-only weights and models requiring remote Python code to be enabled are not supported by this provider. For an existing model, supply a trusted compatible safetensors export under the application's models directory, or use Ollama. Local model paths must resolve inside that directory and load without network access. A model directory there that resolves inside the models directory takes precedence in every scope, including when its name matches a catalog entry, and the substitution is written to the policy audit log. A symlink there that points outside the models directory is ignored, and the curated Hub model of that name is used instead. Arbitrary server paths are refused.
If a download was interrupted, repeat the test under an authorized public
scope to repair missing files. Existing cached legacy models are repaired at
their cached revision, without advancing to newer model bytes. Repair requires
a known immutable revision; models outside the catalog remain cache-only.
If configuration and module metadata are both missing, the local Hub reference
still pins repair to the cached revision. Existing cache state with no single
trustworthy revision is refused instead of silently advancing the model behind
an index. To recover, either delete that model's cache directory (its
models--... folder under the Hugging Face cache) and download it again under
a public-capable scope — only when no collection was indexed with it — or place
a trusted copy under the application's models directory and select that,
or select a different model.
Use Ollama for a broader choice of locally managed embedding models.
Replacing model files under a live session. If you replace or repair the
files of a model while a session (or the service) has it loaded, the next
embedding call that must reload it raises ValueError: Local embedding model files changed; create a new embedding client. This is deliberate, fail-closed
vector safety: the check refuses to silently mix embeddings produced by
different weights. Clients keep working while the old model stays loaded, so
retrying the failed operation — each one builds its own embedding client —
loads the replacement. Programmatic callers that keep one manager alive
should create a new embedding client. See
Troubleshooting - Embedding Issues.
Ollama (Local)
Uses your local Ollama installation.
- Default model:
nomic-embed-text - Requires Ollama running locally
- Configure URL in Settings → LLM → Ollama
- Recommended when you want a wider choice of local embedding models
OpenAI (Cloud)
Uses OpenAI's embedding API.
- Default model:
text-embedding-3-small - Requires OpenAI API key
- Higher quality, requires internet
Changing Embedding Model
- Go to Library → Embedding Settings
- Select provider and model
- Use Test Embedding Model before indexing; changes save automatically
Note: Changing models requires reindexing existing collections.
Configuration
Chunking Settings
| Setting | Default | Description |
|---|---|---|
| Chunk Size | 1000 | Characters per chunk |
| Chunk Overlap | 200 | Overlap between chunks |
| Splitter Type | recursive | How text is split |
Splitter Types:
recursive- Split by paragraphs, then sentences (recommended)token- Split by token countsentence- Split by sentencessemantic- Split by semantic similarity
Index Settings
| Setting | Default | Description |
|---|---|---|
| Distance Metric | cosine | Similarity calculation |
| Index Type | flat | Exact search (most accurate) |
Distance Metrics:
cosine- Angle-based similarity (recommended)l2- Euclidean distancedot_product- Dot product similarity
File Locations
| Data | Location |
|---|---|
| Document database | ~/.local-deep-research/ |
| FAISS indices | ~/.cache/local_deep_research/rag_indices/ |
API Reference
Collection Endpoints
| Endpoint | Method | Description |
|---|---|---|
/library/api/collections |
GET | List all collections |
/library/api/collections |
POST | Create collection |
/library/api/collections/<id> |
PUT | Update collection |
/library/api/collections/<id> |
DELETE | Delete collection |
Document Endpoints
| Endpoint | Method | Description |
|---|---|---|
/library/api/documents |
GET | List documents |
/library/api/document/<id> |
GET | Get document details |
/library/api/document/<id> |
DELETE | Delete document |
/library/api/document/<id>/text |
GET | Get extracted text |
/library/api/document/<id>/pdf |
GET | Download PDF |
RAG Endpoints
| Endpoint | Method | Description |
|---|---|---|
/library/api/rag/settings |
GET | Get RAG configuration |
/library/api/rag/configure |
POST | Update RAG settings |
/library/api/rag/info |
GET | Get index statistics |
/library/api/collections/<id>/index |
POST | Start indexing (SSE) |
Troubleshooting
Documents Not Appearing
- Check file format is supported
- Verify upload completed successfully
- Refresh the library page
Search Not Working
- Ensure collection is indexed (check status)
- Try rebuilding the index
- Check embedding model is configured
Slow Indexing
- Large documents take longer
- Consider using smaller chunk sizes
- Local embedding models are slower than cloud
Memory Issues
FAISS indexes use segments of at most 4,096 vectors. Appends rewrite the partially filled segment and write new segments; they do not rewrite the whole collection. Searches visit segments sequentially and merge their best matches. Each segment file is verified (hashed) against the authenticated manifest the first time the server process reads it, and again whenever the file changes: a later read compares the file's device, inode, size, modification time and change time with those seen at verification and re-hashes on any difference. Since those fields move only with the filesystem's timestamp resolution, a segment whose stored times are whole seconds (coarse-timestamp filesystems such as NFSv2/3, CIFS/SMB, VFAT, HFS+, some FUSE and overlay mounts) is hashed on every read instead of cached. Full chunk records are loaded only for the requested results after their IDs are selected. This bounds the vector working set, at the cost of reading every segment for each search of a large collection. A deletion that leaves a segment less than half full merges it with a neighbouring segment when both fit. Embedding models, PDF extraction, individual document preparation, and reconciliation metadata still contribute to memory use.
Existing native .faiss files remain readable and convert on their next write
without re-embedding. Conversion reads the old payload through read-only memory
mapping, so mapped pages can contribute to RSS while remaining reclaimable.
Conversion temporarily needs disk space for both formats. The new root .faiss
file is an authenticated manifest; its companion .faiss.segments/ directory
contains vectors, numeric IDs, and no document text. Back up the complete index
directory together with the encrypted database. Older application versions
cannot read the segmented format and require rebuilding their indexes from the
encrypted documents after a downgrade.
Chunk ownership includes the full physical index configuration. Incremental indexing replaces only that configuration's chunks, and document deletion purges every retained configuration. A successful full collection rebuild replaces the previous published configurations. Writes and reconciliation remove unowned vectors; surviving chunks from partially indexed documents are retained and reported separately. Reconciliation always checks every vector. A write checks every vector at the latest every 65th write — 64 writes raise the scan counter to its threshold and the next write scans — and sooner on the first write after an upgrade (whose index has no scan record) or when the index holds vector IDs above the highest chunk row, the only IDs a new chunk could reuse; a write that converts a legacy index checks every vector as part of the conversion, and a fresh empty index has nothing to check on its first write. A rollback checkpoint preserves the previous vector generation until the database commits. Crash recovery restores a checkpoint only when it matches the checksum in the encrypted database.
Search refuses corrupt vector components. An indexing write can quarantine the damaged generation and retry the document once; other documents are marked for re-indexing. Quarantine retention removes vector segments together with their manifests, including payloads left by older cleanup attempts.
Index coordination retains the application's single-process scope. Segmentation does not add coordination between independent server processes.
Use a smaller embedding dimension or reduce concurrent document preparation if the embedding model or a single document exceeds available memory. Reducing chunk size can increase the total vector count and index size.
Synthetic PDF scale test
Generate and index a small offline corpus from the checkout:
pdm run python scripts/ci/rag_index_workload.py --report /tmp/rag-scale.json
PDFs are generated in memory using repeated words and unique page markers. The test uses deterministic fake embeddings, real PDF extraction and chunking, real SQLCipher storage, and real FAISS indexing, search, configuration changes, rollback recovery, bulk deletion, and staged collection rebuilding. No embedding server, API credentials, model download, or external documents are needed.
The required RAG Index Scale release workflow runs separate flat and HNSW shards, each with 128 PDFs of 16 pages and 1,024-dimensional embeddings:
OMP_NUM_THREADS=2 OPENBLAS_NUM_THREADS=2 pdm run python scripts/ci/rag_index_workload.py \
--documents 128 --pages 16 --dimension 1024 --index-type hnsw \
--max-rss-mib 768 --report /tmp/rag-scale-hnsw.json
The report includes vector/segment counts, peak process RSS, elapsed time, and bytes written. The workload fails on oversized segment writes, excessive append write volume, orphan vectors, incorrect search/deletion results, plaintext document persistence, or an exceeded RSS budget. It tests storage behavior, not embedding quality or the memory requirements of a real embedding model. The large shards run on releases or manual workflow requests; focused lifecycle regressions and the small PDF generator test run in the regular pytest suite.
Large-collection latency benchmark
The synthetic PDF test bounds memory but not time. A separate benchmark builds a large collection of synthetic vectors directly in the store, with real SQLCipher storage and no PDFs or embedding model, then times per-document writes and searches:
OMP_NUM_THREADS=2 OPENBLAS_NUM_THREADS=2 pdm run python scripts/ci/rag_vector_scale_benchmark.py \
--vectors 300000 --dimension 1024 --index-type flat \
--report /tmp/rag-vector-scale.json
The report includes the first write after the bulk build (which runs the
periodic full orphan scan), median and maximum steady write time, the cold
first search, median and maximum warm search time, and peak RSS. Optional
--max-write-seconds, --max-search-seconds and --max-rss-mib arguments
turn these into limits. The default of 657 chunks per document matches the
1,024-dimension library measured in #7628. The manual RAG Vector Scale
workflow runs it with 1,000,000 vectors for flat and HNSW indexes.
See Also
- Architecture Overview - System architecture
- Extension Guide - Adding custom retrievers
- Full Configuration Reference - All settings and environment variables
- API Quickstart - Using the API