Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

1.6 KB
Raw Blame
"""
Document loaders module.

Provides centralized document loading functionality for both:
- Collection uploads (bytes from HTTP requests)
- Local search engine (file paths on disk)

Supported formats (35+ formats):

Documents:
- PDF (.pdf)
- Text (.txt)
- Markdown (.md, .markdown)
- Word (.doc, .docx)
- RTF (.rtf) - Rich Text Format
- RST (.rst) - reStructuredText documentation

Presentations:
- PowerPoint (.ppt, .pptx)

Spreadsheets:
- Excel (.xls, .xlsx)
- CSV (.csv), TSV (.tsv)
- ODT (.odt) - OpenDocument text

Data formats:
- JSON (.json)
- YAML (.yaml, .yml)
- XML (.xml) - important for USPTO patent data
- TOML (.toml) - config files

Web content:
- HTML (.html, .htm)
- MHTML (.mhtml, .mht) - saved web pages

Images (OCR):
- PNG, JPG, JPEG, TIFF, BMP, HEIC

Research/Notes:
- Jupyter Notebooks (.ipynb)
- Evernote exports (.enex)
- EPUB (.epub) - ebooks (requires pandoc)
- Org (.org) - Emacs org-mode files
- Email (.eml) - email messages
"""

from .bytes_loader import extract_text_from_bytes, load_from_bytes
from .json_loader import SimpleJSONLoader
from .loader_registry import (
    get_loader_class_for_extension,
    get_loader_for_path,
    get_supported_extensions,
    is_extension_supported,
)
from .yaml_loader import YAMLLoader

__all__ = [
    # Bytes loading (for uploads)
    "load_from_bytes",
    "extract_text_from_bytes",
    # Path loading (for local files)
    "get_loader_for_path",
    # Registry functions
    "get_supported_extensions",
    "is_extension_supported",
    "get_loader_class_for_extension",
    # Custom loaders
    "YAMLLoader",
    "SimpleJSONLoader",
]