A fast, helpful, and open-source document parser
readme updates
Logan Markewich0d0176caa38a
diff --git a/AGENTS.md b/AGENTS.mdindex e97f2f8..e751c75 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -1,136 +1,174 @@# LiteParse - Agent Documentation-> This file provides comprehensive context for AI coding agents working on this codebase. Each subdirectory contains its own README with file-specific documentation.+> This file provides comprehensive context for AI coding agents working on this codebase.## Project Overview-**LiteParse** is an open-source PDF parsing library focused on fast, lightweight document processing with spatial text extraction. It runs entirely locally with zero cloud dependencies by default.+**LiteParse** is an open-source PDF parsing library written in **Rust**, focused on fast, lightweight document processing with spatial text extraction. It runs entirely locally with zero cloud dependencies by default.++Language bindings are provided for **Node.js/TypeScript** (via napi-rs), **Python** (via PyO3), and **WebAssembly** (via wasm-bindgen).### Key Capabilities- **Spatial text extraction** with precise bounding boxes-- **Flexible OCR** (built-in Tesseract.js or pluggable HTTP servers)+- **Flexible OCR** (built-in Tesseract or pluggable HTTP servers)- **Multi-format support** (PDFs, DOCX, XLSX, PPTX, images via conversion)-- **TypeScript/Node.js** with both library and CLI interfaces+- **Multi-language bindings**: Rust, Node.js/TypeScript, Python, Browser (WASM)+- **CLI** available from all installation methods (`cargo`, `npm`, `pip`)## Directory Structure```liteparse/-├── src/-│ ├── core/ # Configuration, types, main orchestrator-│ ├── engines/ # Pluggable PDF and OCR engines-│ │ ├── pdf/ # PDF parsing engines (PDF.js, PDFium)-│ │ └── ocr/ # OCR engines (Tesseract, HTTP)-│ ├── processing/ # Text extraction and spatial analysis-│ ├── output/ # Output formatters (JSON, text)-│ ├── conversion/ # Multi-format conversion to PDF-│ ├── vendor/ # Bundled dependencies (PDF.js)-│ ├── index.ts # CLI entry point-│ └── lib.ts # Library public API-├── cli/ # CLI implementation-├── ocr/ # Example OCR server implementations-│ ├── easyocr/ # EasyOCR wrapper server-│ └── paddleocr/ # PaddleOCR wrapper server-└── dist/ # Compiled JavaScript output+├── crates/+│ ├── liteparse/ # Core Rust library + CLI binary+│ │ └── src/+│ │ ├── main.rs # CLI entry point (clap)+│ │ ├── lib.rs # Library root+│ │ ├── parser.rs # LiteParse orchestrator+│ │ ├── config.rs # Configuration types and defaults+│ │ ├── types.rs # Core data types (ParseResult, TextItem, etc.)+│ │ ├── projection.rs # Spatial grid projection (layout reconstruction)+│ │ ├── extract.rs # Raw text extraction from PDFium+│ │ ├── render.rs # Page rendering / screenshots+│ │ ├── conversion.rs # Non-PDF format conversion (LibreOffice, ImageMagick)+│ │ ├── ocr_merge.rs # Merging OCR results with native text+│ │ ├── error.rs # Error types+│ │ ├── ocr/ # OCR engine implementations+│ │ │ ├── mod.rs # OcrEngine trait+│ │ │ ├── tesseract.rs # Built-in Tesseract OCR+│ │ │ └── http_simple.rs # HTTP OCR server client+│ │ └── output/ # Output formatters+│ │ ├── mod.rs+│ │ ├── json.rs+│ │ └── text.rs+│ ├── liteparse-napi/ # Node.js bindings (napi-rs)+│ ├── liteparse-python/ # Python bindings (PyO3 / maturin)+│ ├── liteparse-wasm/ # WASM bindings (wasm-bindgen)+│ ├── pdfium/ # Rust wrapper around PDFium C API+│ └── pdfium-sys/ # PDFium FFI (C → Rust) bindings+├── packages/+│ ├── node/ # npm package: TS wrapper + CLI around native binary+│ │ └── src/+│ │ ├── lib.ts # Public LiteParse class for Node.js+│ │ ├── cli.ts # CLI entry point (commander)+│ │ └── native.ts # Native binary loader+│ ├── python/ # PyPI package: Python wrapper around native binary+│ │ └── liteparse/+│ │ ├── __init__.py+│ │ ├── parser.py # Public LiteParse class for Python+│ │ ├── types.py # Python dataclass types+│ │ └── cli.py # CLI entry point+│ └── wasm/ # WASM npm package+├── ocr/ # Example OCR server implementations+│ ├── easyocr/ # EasyOCR wrapper server+│ └── paddleocr/ # PaddleOCR wrapper server+└── Cargo.toml # Workspace root```## Data Flow-1. **Input**: File path received (any supported format)-2. **Conversion** (if dependencies installed): Non-PDF formats converted to PDF via LibreOffice/ImageMagick-3. **PDF Loading**: PDF.js extracts text items, images, metadata-4. **OCR** (if enabled): Images rendered and OCR'd for text-sparse areas+1. **Input**: File path or raw bytes received (any supported format)+2. **Conversion** (if needed): Non-PDF formats converted to PDF via LibreOffice/ImageMagick+3. **PDF Loading**: PDFium extracts text items, images, metadata+4. **OCR** (if enabled): Pages rendered and OCR'd for text-sparse areas5. **Grid Projection**: Spatial reconstruction of text layout using anchor system6. **Post-processing**: Bounding boxes, text cleanup7. **Output**: Formatted as JSON or plain text## Key Design Decisions-### 1. Engine Abstraction Pattern-Both PDF and OCR functionality use interface-based abstraction (`PdfEngine`, `OcrEngine`). This allows:-- Swapping implementations without changing core logic-- Auto-detection: HTTP OCR if URL provided, otherwise Tesseract.js-- Future extensibility for new engines-- Future possibility of custom conversion engines for non-PDF formats+### 1. Rust Core with Language Bindings+The core parsing logic is written in Rust for performance and safety. Language-specific crates expose the same API surface:+- `liteparse-napi` → Node.js via napi-rs+- `liteparse-python` → Python via PyO3/maturin+- `liteparse-wasm` → Browser via wasm-bindgen++Each binding crate is thin — it wraps the core `liteparse` crate's types and async API.++### 2. OCR Engine Trait+OCR functionality uses a trait-based abstraction (`OcrEngine`). This allows:+- Built-in Tesseract (default, compiled in via `tesseract-rs`)+- HTTP OCR server client for remote engines+- Custom JS-side OCR in the WASM build via a callback interface-### 2. Spatial Grid Projection-The most complex (and important!) part of the codebase (`src/processing/gridProjection.ts`, ~1650 lines). Uses:+### 3. Spatial Grid Projection+The most complex (and important!) part of the codebase (`crates/liteparse/src/projection.rs`). Uses:- **Anchor-based layout**: Tracks text alignment (left, right, center, floating)- **Forward anchors**: Carry alignment information between lines- **Column detection**: Identifies multi-column layouts- **Rotation handling**: Transforms 90°, 180°, 270° rotated text to correct reading order- **OCR merging**: Combines native PDF text with OCR results, preserving confidence scores and source flags in output-### 3. Selective OCR+### 4. Selective OCROCR only runs on embedded images where text extraction failed, not the entire document. This balances accuracy with performance.-### 4. Configuration Merging-Uses a default-first approach where users only override what they need. Configuration flows: defaults → file config → CLI options.+### 5. Configuration+Uses a default-first approach where users only override what they need. See `crates/liteparse/src/config.rs` for defaults.-### 5. Format Conversion via External Tools-Rather than implementing format parsers, LiteParse converts non-PDF formats using system tools (LibreOffice, ImageMagick) into a single format (PDF). This provides broad format support with minimal code.+### 6. Format Conversion via External Tools+Rather than implementing format parsers, LiteParse converts non-PDF formats using system tools (LibreOffice, ImageMagick) into PDF. This provides broad format support with minimal code.## Common Tasks### Adding a New Output Format-1. Create new file in `src/output/` implementing the formatter-2. Add format option to `cli/parse.ts`-3. Update `src/core/parser.ts` to use new formatter+1. Create new file in `crates/liteparse/src/output/`+2. Add variant to `OutputFormat` enum in `config.rs`+3. Wire it up in `main.rs` and binding crates### Adding a New OCR Engine-1. Implement `OcrEngine` interface in `src/engines/ocr/`-2. Add initialization logic in `src/core/parser.ts`-3. Add configuration options in `src/core/types.ts`+1. Implement `OcrEngine` trait in `crates/liteparse/src/ocr/`+2. Add initialization logic in `parser.rs`+3. Add configuration options in `config.rs`### Modifying Text Extraction Logic-The processing pipeline is in `src/processing/`. Key files:-- `gridProjection.ts` - Layout reconstruction (most complex)-- `bbox.ts` - Bounding box calculation-- `cleanText.ts` - Text cleanup+Key files in `crates/liteparse/src/`:+- `projection.rs` — Layout reconstruction (most complex)+- `extract.rs` — Raw text item extraction from PDFium+- `ocr_merge.rs` — Merging OCR and native text### Adding CLI Options-1. Update `cli/parse.ts` with new Commander.js option-2. Add corresponding config field in `src/core/types.ts`-3. Update `src/core/config.ts` with default value-4. Use the option in `src/core/parser.ts`+1. Add field to `LiteParseConfig` in `config.rs`+2. Add clap arg in `main.rs`+3. Wire through `parser.rs`+4. Expose in binding crates (`liteparse-napi`, `liteparse-python`, `liteparse-wasm`)-## Testing Approach+### Adding / Modifying Node.js Wrapper+- Edit `packages/node/src/lib.ts` for library API changes+- Edit `packages/node/src/cli.ts` for CLI changes+- The native binary interface is defined in `packages/node/src/native.ts`-Currently tested via manual verification with sample documents. The project would benefit from:-- Unit tests for processing utilities-- Integration tests with known PDFs-- Snapshot tests for output formats+### Adding / Modifying Python Wrapper+- Edit `packages/python/liteparse/parser.py` for library API changes+- Types are in `packages/python/liteparse/types.py`+- CLI entry point is `packages/python/liteparse/cli.py`## Key Dependencies| Dependency | Purpose ||------------|---------|-| `pdfjs-dist` | PDF parsing and text extraction |-| `@hyzyla/pdfium` | High-quality PDF rendering for screenshots |-| `tesseract.js` | In-process OCR (zero setup) |-| `sharp` | Image processing |-| `commander` | CLI framework |-| `zod` | Schema validation |+| `pdfium` (C library) | PDF text extraction and rendering |+| `tesseract-rs` | Built-in OCR engine (optional, via `tesseract` feature) |+| `clap` | CLI framework |+| `serde` / `serde_json` | Serialization |+| `tokio` | Async runtime |+| `reqwest` | HTTP client (for OCR server) |+| `image` | Image processing (PNG encoding) |+| `napi-rs` | Node.js native bindings |+| `pyo3` / `maturin` | Python native bindings |+| `wasm-bindgen` | WASM bindings |## Entry Points-- **CLI**: `src/index.ts` → `cli/parse.ts`-- **Library**: `src/lib.ts` exports `LiteParse` class and types-- **Main Class**: `src/core/parser.ts` contains `LiteParse` orchestrator+- **Rust CLI**: `crates/liteparse/src/main.rs`+- **Rust Library**: `crates/liteparse/src/lib.rs` → `parser.rs` contains `LiteParse` struct+- **Node.js**: `packages/node/src/lib.ts` exports `LiteParse` class+- **Python**: `packages/python/liteparse/parser.py` exports `LiteParse` class+- **WASM**: `crates/liteparse-wasm/` exposes `LiteParse` via wasm-bindgen## Related Documentation-These files are key to understanding the codebase and should be referenced for specific implementation details.--If changes to the codebase are being made, please update the relevant documentation files to reflect those changes and keep them up to date.-- [User-facing documentation](README.md)-- [src/conversion/README.md](src/conversion/README.md) - Format conversion details-- [src/core/README.md](src/core/README.md) - Core architecture and configuration-- [src/engines/README.md](src/engines/README.md) - Engine abstraction and implementations- - [src/engines/pdf/README.md](src/engines/pdf/README.md) - PDF engines (PDF.js, PDFium)- - [src/engines/ocr/README.md](src/engines/ocr/README.md) - OCR engines (Tesseract, HTTP)-- [src/output/README.md](src/output/README.md) - Output formatters-- [src/processing/README.md](src/processing/README.md) - Text extraction and spatial processing-- [ocr/README.md](ocr/README.md) - OCR server implementations (EasyOCR, PaddleOCR)-- [cli/README.md](cli/README.md) - CLI usage and options+- [OCR API Specification](OCR_API_SPEC.md)+- [WASM package README](packages/wasm/README.md)+- [Python package README](packages/python/README.md)+- [OCR server examples](ocr/README.md)diff --git a/README.md b/README.mdindex bd57aee..a5838a5 100644--- a/README.md+++ b/README.md@@ -2,8 +2,14 @@[](https://github.com/run-llama/liteparse/actions/workflows/ci.yml)|+[](https://crates.io/crates/liteparse)+|[](https://www.npmjs.com/package/@llamaindex/liteparse)|+[](https://www.npmjs.com/package/@llamaindex/liteparse-wasm)+|+[](https://pypi.org/project/liteparse/)+|[](https://opensource.org/licenses/Apache-2.0)|[Docs](https://developers.llamaindex.ai/liteparse/)@@ -22,48 +28,91 @@ hard stuff so your models see clean, structured data and markdown.## Overview-- **Fast Text Parsing**: Spatial text parsing using PDF.js+- **Fast Text Parsing**: Spatial text parsing using PDFium- **Flexible OCR System**:- - **Built-in**: Tesseract.js (zero setup, works out of the box!)+ - **Built-in**: Tesseract (zero setup, bundled with the library)- **HTTP Servers**: Plug in any OCR server (EasyOCR, PaddleOCR, custom)- **Standard API**: Simple, well-defined OCR API specification- **Screenshot Generation**: Generate high-quality page screenshots for LLM agents- **Multiple Output Formats**: JSON and Text- **Bounding Boxes**: Precise text positioning information-- **Standalone Binary**: No cloud dependencies, runs entirely locally+- **Multi-language**: Use from Rust, Node.js/TypeScript, Python, or the browser (WASM)- **Multi-platform**: Linux, macOS (Intel/ARM), Windows## Installation-### CLI Tool+All versions (except WASM) ship with the same CLI and library API. Install the one that fits your environment:++<details>+ <summary>Node.js / TypeScript</summary>++Install via npm to use the `lit` CLI or the library API:++```bash+npm i -g @llamaindex/liteparse # CLI (global)+npm i @llamaindex/liteparse # Library (project dependency)+```++Parse your first document right away:++```bash+lit parse document.pdf+```++Or use the library API in your Node.js or TypeScript project:++```typescript+import { LiteParse } from '@llamaindex/liteparse';++const parser = new LiteParse({ ocrEnabled: true });++const result = await parser.parse('document.pdf');+console.log(result.text);+```+</details>-#### Option 1: Global Install (Recommended)+<details>+ <summary>Python</summary>-Install globally via npm to use the `lit` command anywhere:+Install via pip to use the `lit` CLI or the library API:```bash-npm i -g @llamaindex/liteparse+pip install liteparse```-Then use it:+Parse your first document right away:```bashlit parse document.pdf-lit screenshot document.pdf```-#### Option 2: Install from Source+Or use the library API in your Python project:-You can clone the repo and install the CLI globally from source:+```python+from liteparse import LiteParse+parser = LiteParse(ocr_enabled=True)+result = parser.parse('document.pdf')+print(result.text)```-git clone https://github.com/run-llama/liteparse.git-cd liteparse-npm run build-npm pack-npm install -g ./llamaindex-liteparse-*.tgz++</details>++<details>+ <summary>Browser (WASM)</summary>++You can install a trimmed-down version of LiteParse that runs entirely in the browser, with no server or cloud dependencies.++```bash+npm install @llamaindex/liteparse-wasm```+It supports PDF parsing and custom OCR engines implemented in JavaScript.++See the [WASM package README](packages/wasm/README.md) for usage details.++</details>+### Agent SkillYou can use `liteparse` as an agent skill, downloading it with the `skills` CLI tool:@@ -74,7 +123,9 @@ npx skills add run-llama/llamaparse-agent-skills --skill liteparseOr copy-pasting the [`SKILL.md`](https://github.com/run-llama/llamaparse-agent-skills/blob/main/skills/liteparse/SKILL.md) file to your own skills setup.-## Usage+## CLI Usage++The CLI is the same across all installations (`npm`, `pip`, or the Rust binary).### Parse Files@@ -83,7 +134,7 @@ Or copy-pasting the [`SKILL.md`](https://github.com/run-llama/llamaparse-agent-slit parse document.pdf# Parse with specific format-lit parse document.pdf --format json -o output.md+lit parse document.pdf --format json -o output.json# Parse specific pageslit parse document.pdf --target-pages "1-5,10,15-20"@@ -97,7 +148,7 @@ curl -sL https://example.com/report.pdf | lit parse -### Batch Parsing-You can also parse an entire directory of documents:+Parse an entire directory of documents:```bashlit batch-parse ./input-directory ./output-directory@@ -116,36 +167,77 @@ lit screenshot document.pdf --target-pages "1,3,5" -o ./screenshots# Custom DPIlit screenshot document.pdf --dpi 300 -o ./screenshots+```++### CLI Reference++#### Parse Command-# Screenshot page range-lit screenshot document.pdf --target-pages "1-10" -o ./screenshots```+lit parse [OPTIONS] <file>-### Library Usage+Options:+ -o, --output <file> Output file path+ --format <format> Output format: json|text [default: text]+ --no-ocr Disable OCR+ --ocr-language <lang> OCR language, Tesseract format [default: eng]+ --ocr-server-url <url> HTTP OCR server URL (uses Tesseract if not provided)+ --tessdata-path <path> Path to tessdata directory+ --max-pages <n> Max pages to parse [default: 1000]+ --target-pages <pages> Pages to parse (e.g., "1-5,10,15-20")+ --dpi <dpi> Rendering DPI [default: 150]+ --preserve-small-text Keep very small text+ --password <password> Password for encrypted documents+ --num-workers <n> Concurrent OCR workers [default: CPU cores - 1]+ -q, --quiet Suppress progress output+ -h, --help Print help+```-Install as a dependency in your project:+#### Batch Parse Command-```bash-npm install @llamaindex/liteparse-# or-pnpm add @llamaindex/liteparse```+lit batch-parse [OPTIONS] <input-dir> <output-dir>-```typescript-import { LiteParse } from '@llamaindex/liteparse';+Options:+ --format <format> Output format: json|text [default: text]+ --no-ocr Disable OCR+ --ocr-language <lang> OCR language [default: eng]+ --ocr-server-url <url> HTTP OCR server URL+ --tessdata-path <path> Path to tessdata directory+ --max-pages <n> Max pages per file [default: 1000]+ --dpi <dpi> Rendering DPI [default: 150]+ --recursive Recursively search input directory+ --extension <ext> Only process files with this extension (e.g., ".pdf")+ --password <password> Password for encrypted documents+ --num-workers <n> Concurrent OCR workers+ -q, --quiet Suppress progress output+ -h, --help Print help+```-const parser = new LiteParse({ ocrEnabled: true });-const result = await parser.parse('document.pdf');-console.log(result.text);+#### Screenshot Command++```+lit screenshot [OPTIONS] <file>++Options:+ -o, --output-dir <dir> Output directory [default: ./screenshots]+ --target-pages <pages> Pages to screenshot (e.g., "1,3,5" or "1-5")+ --dpi <dpi> Rendering DPI [default: 150]+ --password <password> Password for encrypted documents+ -q, --quiet Suppress progress output+ -h, --help Print help```-#### Buffer / Uint8Array Input+## Library Usage-You can pass raw bytes directly instead of a file path, which is useful for remote files:+### Buffer / Uint8Array Input++All APIs that accept file paths also accept raw bytes, so you can parse documents from any source (e.g. HTTP responses, in-memory buffers) without writing to disk first.++The WASM package only accepts `Uint8Array` input, while the Node.js and Python versions accept both file paths and bytes.```typescriptimport { LiteParse } from '@llamaindex/liteparse';-import { readFile } from 'fs/promises';const parser = new LiteParse();@@ -159,186 +251,39 @@ const buffer = Buffer.from(await response.arrayBuffer());const result2 = await parser.parse(buffer);```-Non-PDF buffers (images, Office documents) are written to a temp directory for format conversion. Screenshots also work with buffer input:--```typescript-const screenshots = await parser.screenshot(pdfBytes, [1, 2, 3]);-```--### Browser Usage--LiteParse's core parsing engine (PDF.js text extraction, grid projection, OCR via Tesseract.js) can run in the browser. Since the library has Node-only dependencies (sharp, fs, child_process), you'll need a bundler like Vite to swap those out with browser stubs.--#### Vite Configuration--The key is a Vite plugin that redirects Node-only source files to browser-safe replacements, plus `resolve.alias` entries that stub out Node built-in modules:+#### Screenshots```typescript-// vite.config.ts-import { defineConfig, type Plugin } from "vite";-import { resolve, dirname } from "node:path";--// Node-only files → browser stubs (you write these)-const FILE_REDIRECTS = [- { match: /\/engines\/pdf\/pdfium-renderer(\.js|\.ts)?$/, target: "stubs/pdfium-renderer.ts" },- { match: /\/engines\/pdf\/pdfjsImporter(\.js|\.ts)?$/, target: "stubs/pdfjsImporter.ts" },- { match: /\/engines\/ocr\/http-simple(\.js|\.ts)?$/, target: "stubs/http-simple.ts" },- { match: /\/conversion\/convertToPdf(\.js|\.ts)?$/, target: "stubs/convertToPdf.ts" },- { match: /\/processing\/gridDebugLogger(\.js|\.ts)?$/, target: "stubs/gridDebugLogger.ts" },- { match: /\/processing\/gridVisualizer(\.js|\.ts)?$/, target: "stubs/gridVisualizer.ts" },-];--function liteparseNodeRedirects(): Plugin {- return {- name: "liteparse-node-redirects",- enforce: "pre",- async resolveId(source, importer) {- if (!importer) return null;- const abs = source.startsWith(".") ? resolve(dirname(importer), source) : source;- for (const { match, target } of FILE_REDIRECTS) {- if (match.test(abs) || match.test(source)) return resolve(target);- }- return null;- },- };+const screenshots = await parser.screenshot('document.pdf', [1, 2, 3]);+for (const s of screenshots) {+ console.log(`Page ${s.pageNum}: ${s.width}x${s.height}`);+ // s.imageBuffer contains PNG bytes}--export default defineConfig({- plugins: [liteparseNodeRedirects()],- optimizeDeps: { include: ["tesseract.js"] },- resolve: {- alias: [- { find: "node:fs/promises", replacement: "stubs/empty.ts" },- { find: "node:fs", replacement: "stubs/empty.ts" },- { find: "node:url", replacement: "stubs/empty.ts" },- { find: "node:path", replacement: "stubs/empty.ts" },- { find: "node:os", replacement: "stubs/empty.ts" },- { find: "node:child_process", replacement: "stubs/empty.ts" },- { find: /^fs$/, replacement: "stubs/empty.ts" },- { find: /^path$/, replacement: "stubs/empty.ts" },- { find: /^os$/, replacement: "stubs/empty.ts" },- { find: /^child_process$/, replacement: "stubs/empty.ts" },- { find: "form-data", replacement: "stubs/empty.ts" },- { find: "axios", replacement: "stubs/empty.ts" },- { find: "file-type", replacement: "stubs/file-type.ts" },- ],- },-});-```--See [`scripts/browser-compat/`](scripts/browser-compat/) for a complete working example with all the stub files.--#### What works in the browser--- PDF parsing from `Uint8Array` input (use `file.arrayBuffer()` to get bytes from a `<input type="file">`)-- OCR via Tesseract.js (runs in Web Workers, fetches language data from CDN on first use)-- Text and JSON output formats--#### What doesn't work--- File path input (pass `Uint8Array` instead)-- DOCX/XLSX/PPTX/image conversion (requires LibreOffice/ImageMagick)-- HTTP OCR server backend-- Screenshots (these use PDFium + sharp, which are native Node addons)--### CLI Options--#### Parse Command--```-$ lit parse --help-Usage: lit parse [options] <file>--Parse a document file (PDF, DOCX, XLSX, PPTX, images, etc.)--Options:- -o, --output <file> Output file path- --format <format> Output format: json|text (default: "text")- --ocr-server-url <url> HTTP OCR server URL (uses Tesseract if not provided)- --no-ocr Disable OCR- --ocr-language <lang> OCR language(s) (default: "en")- --num-workers <n> Number of pages to OCR in parallel (default: CPU cores - 1)- --max-pages <n> Max pages to parse (default: "10000")- --target-pages <pages> Target pages (e.g., "1-5,10,15-20")- --dpi <dpi> DPI for rendering (default: "150")- --no-precise-bbox Disable precise bounding boxes- --preserve-small-text Preserve very small text- --password <password> Password for encrypted/protected documents- --config <file> Config file (JSON)- -q, --quiet Suppress progress output- -h, --help display help for command-```--#### Batch Parse Command--```-$ lit batch-parse --help-Usage: lit batch-parse [options] <input-dir> <output-dir>--Parse multiple documents in batch mode (reuses PDF engine for efficiency)--Options:- --format <format> Output format: json|text (default: "text")- --ocr-server-url <url> HTTP OCR server URL (uses Tesseract if not provided)- --no-ocr Disable OCR- --ocr-language <lang> OCR language(s) (default: "en")- --num-workers <n> Number of pages to OCR in parallel (default: CPU cores - 1)- --max-pages <n> Max pages to parse per file (default: "10000")- --dpi <dpi> DPI for rendering (default: "150")- --no-precise-bbox Disable precise bounding boxes- --recursive Recursively search input directory- --extension <ext> Only process files with this extension (e.g., ".pdf")- --password <password> Password for encrypted/protected documents (applied to all files)- --config <file> Config file (JSON)- -q, --quiet Suppress progress output- -h, --help display help for command-```--#### Screenshot Command--```-$ lit screenshot --help-Usage: lit screenshot [options] <file>--Generate screenshots of PDF pages--Options:- -o, --output-dir <dir> Output directory for screenshots (default: "./screenshots")- --target-pages <pages> Page numbers to screenshot (e.g., "1,3,5" or "1-5")- --dpi <dpi> DPI for rendering (default: "150")- --format <format> Image format: png|jpg (default: "png")- --password <password> Password for encrypted/protected documents- --config <file> Config file (JSON)- -q, --quiet Suppress progress output- -h, --help display help for command```## OCR Setup-### Default: Tesseract.js--```bash-# Tesseract is enabled by default-lit parse document.pdf+### Default: Tesseract-# Specify language-lit parse document.pdf --ocr-language fra+Tesseract is bundled and works out of the box:-# Disable OCR-lit parse document.pdf --no-ocr+```bash+lit parse document.pdf # OCR enabled by default+lit parse document.pdf --ocr-language fra # Specify language+lit parse document.pdf --no-ocr # Disable OCR```-By default, Tesseract.js downloads language data from the internet on first use. For offline or air-gapped environments, set the `TESSDATA_PREFIX` environment variable to a directory containing pre-downloaded `.traineddata` files:+For offline or air-gapped environments, set `TESSDATA_PREFIX` to a directory containing pre-downloaded `.traineddata` files:```bashexport TESSDATA_PREFIX=/path/to/tessdatalit parse document.pdf --ocr-language eng```-You can also pass `tessdataPath` in the library config:+Or pass the path directly:-```typescript-const parser = new LiteParse({ tessdataPath: '/path/to/tessdata' });+```bash+lit parse document.pdf --tessdata-path /path/to/tessdata```### Optional: HTTP OCR Servers@@ -355,22 +300,18 @@ The API requires:- Accepts `file` and `language` parameters- Returns JSON: `{ results: [{ text, bbox: [x1,y1,x2,y2], confidence }] }`-See the example servers in `ocr/easyocr/` and `ocr/paddleocr/` as templates.--For the complete OCR API specification, see [`OCR_API_SPEC.md`](OCR_API_SPEC.md).-## Multi-Format Input Support-LiteParse supports **automatic conversion** of various document formats to PDF before parsing. This makes it unique compared to other PDF-only parsing tools!+LiteParse supports **automatic conversion** of various document formats to PDF before parsing.### Supported Input Formats#### Office Documents (via LibreOffice)-- **Word**: `.doc`, `.docx`, `.docm`, `.odt`, `.rtf`-- **PowerPoint**: `.ppt`, `.pptx`, `.pptm`, `.odp`-- **Spreadsheets**: `.xls`, `.xlsx`, `.xlsm`, `.ods`, `.csv`, `.tsv`+- **Word**: `.doc`, `.docx`, `.docm`, `.odt`, `.rtf`, `.pages`+- **PowerPoint**: `.ppt`, `.pptx`, `.pptm`, `.odp`, `.key`+- **Spreadsheets**: `.xls`, `.xlsx`, `.xlsm`, `.ods`, `.csv`, `.tsv`, `.numbers`-Just install the dependency and LiteParse will automatically convert these formats to PDF for parsing:+Install LibreOffice for automatic conversion:```bash# macOS@@ -380,15 +321,15 @@ brew install --cask libreofficeapt-get install libreoffice# Windows-choco install libreoffice-fresh # might require admin permissions+choco install libreoffice-fresh```-> _For Windows, you might need to add the path to the directory containing LibreOffice CLI executable (generally `C:\Program Files\LibreOffice\program`) to the environment variables and re-start the machine._+> _On Windows, you may need to add LibreOffice's program directory (usually `C:\Program Files\LibreOffice\program`) to your PATH._#### Images (via ImageMagick)- **Formats**: `.jpg`, `.jpeg`, `.png`, `.gif`, `.bmp`, `.tiff`, `.webp`, `.svg`-Just install ImageMagick and LiteParse will convert images to PDF for parsing (with OCR):+Install ImageMagick for image-to-PDF conversion:```bash# macOS@@ -398,74 +339,51 @@ brew install imagemagickapt-get install imagemagick# Windows-choco install imagemagick.app # might require admin permissions+choco install imagemagick.app```## Environment Variables| Variable | Description ||----------|-------------|-| `TESSDATA_PREFIX` | Path to a directory containing Tesseract `.traineddata` files. Used for offline/air-gapped environments where Tesseract.js cannot download language data from the internet. |-| `LITEPARSE_TMPDIR` | Override the temp directory used for format conversion and intermediate files. Defaults to the OS temp directory (`os.tmpdir()`). Useful in containerized or read-only filesystem environments. |--## Configuration--You can configure parsing options via CLI flags or a JSON config file. The config file allows you to set sensible defaults and override as needed.--### Config File Example+| `TESSDATA_PREFIX` | Path to a directory containing Tesseract `.traineddata` files. Used for offline/air-gapped environments. |-Create a `liteparse.config.json` file:--```json-{- "ocrLanguage": "en",- "ocrEnabled": true,- "maxPages": 1000,- "dpi": 150,- "outputFormat": "json",- "preciseBoundingBox": true,- "preserveVerySmallText": false,- "password": "optional_password"-}-```+## Development-For HTTP OCR servers, just add `ocrServerUrl`:+The project is a Rust workspace with the core library and language-specific binding crates.-```json-{- "ocrServerUrl": "http://localhost:8828/ocr",- "ocrLanguage": "en",- "outputFormat": "json"-}```--Use with:--```bash-lit parse document.pdf --config liteparse.config.json+crates/+├── liteparse/ # Core library + CLI binary+├── liteparse-napi/ # Node.js bindings (napi-rs)+├── liteparse-python/ # Python bindings (PyO3)+├── liteparse-wasm/ # WASM bindings (wasm-bindgen)+├── pdfium/ # PDFium Rust wrapper+└── pdfium-sys/ # PDFium FFI bindings+packages/+├── node/ # npm package (TS wrapper + native binary)+├── python/ # PyPI package (Python wrapper + native binary)+└── wasm/ # WASM npm package```-## Development--We provide a fairly rich `AGENTS.md`/`CLAUDE.md` that we recommend using to help with development + coding agents.+### Building```bash-# Install dependencies-npm install+# Build the CLI+cargo build --release -p liteparse-# Build TypeScript (Linux/macOs)-npm run build+# Build Node.js bindings+cd packages/node && npm run build-# Build Typescript (Windows)-npm run build:windows+# Build Python bindings+cd packages/python && maturin develop --release-# Watch mode-npm run dev--# Test parsing-npm test+# Build WASM+cd packages/wasm && npm run build```+We provide a fairly rich `AGENTS.md`/`CLAUDE.md` that we recommend using to help with development + coding agents.+## LicenseApache 2.0@@ -474,8 +392,10 @@ Apache 2.0Built on top of:-- [PDF.js](https://github.com/mozilla/pdf.js) - PDF parsing engine-- [Tesseract.js](https://github.com/naptha/tesseract.js) - In-process OCR engine+- [PDFium](https://pdfium.googlesource.com/pdfium/) - PDF rendering and text extraction+- [Tesseract](https://github.com/tesseract-ocr/tesseract) - OCR engine (via tesseract-rs)- [EasyOCR](https://github.com/JaidedAI/EasyOCR) - HTTP OCR server (optional)- [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR) - HTTP OCR server (optional)-- [Sharp](https://github.com/lovell/sharp) - Image processing+- [napi-rs](https://napi.rs/) - Node.js native bindings+- [PyO3](https://pyo3.rs/) - Python native bindings+- [wasm-bindgen](https://github.com/wasm-bindgen/wasm-bindgen) - WebAssembly bindingsdiff --git a/packages/node/README.md b/packages/node/README.mdnew file mode 100644index 0000000..ed6f9d4--- /dev/null+++ b/packages/node/README.md@@ -0,0 +1,89 @@+# LiteParse Node.js++Node.js/TypeScript bindings for [LiteParse](https://github.com/run-llama/liteparse) — fast, lightweight PDF and document parsing with spatial text extraction.++## Installation++```bash+npm i @llamaindex/liteparse+```++This also installs the `lit` CLI command (use `npm i -g` for global access).++## Quick Start++```typescript+import { LiteParse } from '@llamaindex/liteparse';++const parser = new LiteParse();+const result = await parser.parse('document.pdf');+console.log(result.text);++// Access structured data+for (const page of result.pages) {+ console.log(`Page ${page.pageNum}: ${page.textItems.length} text items`);+}+```++## Configuration++All options are passed to the constructor:++```typescript+const parser = new LiteParse({+ ocrEnabled: true, // Enable OCR (default: true)+ ocrLanguage: 'eng', // Tesseract language code+ ocrServerUrl: undefined, // HTTP OCR server URL (optional)+ tessdataPath: undefined, // Path to tessdata directory (optional)+ maxPages: 1000, // Max pages to parse+ targetPages: '1-5,10', // Specific pages (optional)+ dpi: 150, // Rendering DPI+ preserveVerySmallText: false, // Keep tiny text+ password: undefined, // Password for protected documents+ quiet: false, // Suppress progress output+ numWorkers: 4, // Concurrent OCR workers+});+```++## Parsing from Bytes++Pass a `Buffer` or `Uint8Array` directly — useful for HTTP responses or in-memory data:++```typescript+import { readFile } from 'fs/promises';++const pdfBytes = await readFile('document.pdf');+const result = await parser.parse(pdfBytes);+console.log(result.text);+```++## Screenshots++Generate PNG screenshots of document pages:++```typescript+const screenshots = parser.screenshot('document.pdf', [1, 2, 3]);+for (const s of screenshots) {+ console.log(`Page ${s.pageNum}: ${s.width}x${s.height}`);+ // s.imageBuffer contains PNG bytes+}+```++## Supported Formats++- PDF (`.pdf`)+- Microsoft Office (`.docx`, `.xlsx`, `.pptx`, etc.) — requires LibreOffice+- OpenDocument (`.odt`, `.ods`, `.odp`) — requires LibreOffice+- Images (`.png`, `.jpg`, `.tiff`, etc.) — requires ImageMagick+- And more!++## CLI++The npm package includes the `lit` CLI:++```bash+lit parse document.pdf+lit parse document.pdf --format json -o output.json+lit screenshot document.pdf -o ./screenshots+lit batch-parse ./input ./output+```diff --git a/packages/python/README.md b/packages/python/README.mdindex 5871876..c155039 100644--- a/packages/python/README.md+++ b/packages/python/README.md@@ -1,122 +1,86 @@# LiteParse Python-Python wrapper for [LiteParse](https://github.com/run-llama/liteparse) - fast, lightweight document parsing with optional OCR.--> **Important:** This package is a Python wrapper around the LiteParse Node.js CLI.-> You must have **Node.js** (>= 18) installed on your system. The CLI will be auto-installed-> via npm on first use if not already present, or you can install it manually beforehand.+Python bindings for [LiteParse](https://github.com/run-llama/liteparse) — fast, lightweight PDF and document parsing with spatial text extraction.## Installation-### Step 1: Install Node.js--LiteParse requires Node.js (>= 18). Install it from [nodejs.org](https://nodejs.org/) or via your package manager.--### Step 2: Install the LiteParse CLI--```bash-npm install -g @llamaindex/liteparse-```--### Step 3: Install the Python package-```bashpip install liteparse```-> **Note:** If you skip Step 2, the Python package will attempt to auto-install the CLI-> via `npm install -g @llamaindex/liteparse` on first use (requires npm in your PATH).+This also installs the `lit` CLI command.## Quick Start```pythonfrom liteparse import LiteParse-# Create parserparser = LiteParse()--# Parse a documentresult = parser.parse("document.pdf")print(result.text)# Access structured datafor page in result.pages:- print(f"Page {page.pageNum}: {len(page.textItems)} text items")+ print(f"Page {page.page_num}: {len(page.text_items)} text items")```## Configuration-All parsing options are passed per-call to `parse()`:+All options are passed to the constructor:```python-from liteparse import LiteParse--parser = LiteParse()--result = parser.parse(- "document.pdf",- ocr_enabled=False,- max_pages=10,- dpi=150,- preserve_very_small_text=True,+parser = LiteParse(+ ocr_enabled=True, # Enable OCR (default: True)+ ocr_language="eng", # Tesseract language code+ ocr_server_url=None, # HTTP OCR server URL (optional)+ tessdata_path=None, # Path to tessdata directory (optional)+ max_pages=1000, # Max pages to parse+ target_pages="1-5,10", # Specific pages (optional)+ dpi=150, # Rendering DPI+ preserve_very_small_text=False, # Keep tiny text+ password=None, # Password for protected documents+ quiet=False, # Suppress progress output+ num_workers=4, # Concurrent OCR workers)-print(result.text)```-## Parsing from bytes+## Parsing from Bytes-If you already have file contents in memory (e.g. from a web upload), pass them directly:+Pass raw PDF bytes directly — useful for web uploads or downloaded files:```pythonwith open("document.pdf", "rb") as f:- pdf_bytes = f.read()--result = parser.parse(pdf_bytes)+ result = parser.parse(f.read())print(result.text)```-## Batch Processing+## Screenshots-For parsing multiple files, batch mode is significantly faster as it reuses the PDF engine:+Generate PNG screenshots of document pages:```python-from liteparse import LiteParse--parser = LiteParse()--# Parse all documents in a directory-result = parser.batch_parse(- input_dir="./documents",- output_dir="./output",- ocr_enabled=False,- recursive=True, # Include subdirectories- extension_filter=".pdf", # Only PDF files-)--print(f"Output written to: {result.output_dir}")+screenshots = parser.screenshot("document.pdf", page_numbers=[1, 2, 3])+for s in screenshots:+ print(f"Page {s.page_num}: {s.width}x{s.height}")+ with open(f"page_{s.page_num}.png", "wb") as f:+ f.write(s.image_bytes)```## Supported Formats- PDF (`.pdf`)-- Microsoft Office (`.docx`, `.xlsx`, `.pptx`, etc.) - requires LibreOffice-- OpenDocument (`.odt`, `.ods`, `.odp`) - requires LibreOffice-- Images (`.png`, `.jpg`, `.tiff`, etc.) - requires ImageMagick+- Microsoft Office (`.docx`, `.xlsx`, `.pptx`, etc.) — requires LibreOffice+- OpenDocument (`.odt`, `.ods`, `.odp`) — requires LibreOffice+- Images (`.png`, `.jpg`, `.tiff`, etc.) — requires ImageMagick- And more!-## Performance Tips+## CLI-1. **Disable OCR** if your documents have selectable text:- ```python- result = parser.parse("doc.pdf", ocr_enabled=False)- ```+The Python package includes the `lit` CLI:-2. **Use batch mode** for multiple files to avoid cold-start overhead:- ```python- parser.batch_parse("./input", "./output")- ```--3. **Limit pages** if you only need specific pages:- ```python- result = parser.parse("doc.pdf", target_pages="1-5")- ```+```bash+lit parse document.pdf+lit parse document.pdf --format json -o output.json+lit screenshot document.pdf -o ./screenshots+lit batch-parse ./input ./output+```