Skip to content

A fast, helpful, and open-source document parser

readme updates

Logan Markewich0d0176caa38a
diff --git a/AGENTS.md b/AGENTS.md
index e97f2f8..e751c75 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -1,136 +1,174 @@
# LiteParse - Agent Documentation
-> This file provides comprehensive context for AI coding agents working on this codebase. Each subdirectory contains its own README with file-specific documentation.
+> This file provides comprehensive context for AI coding agents working on this codebase.
## Project Overview
-**LiteParse** is an open-source PDF parsing library focused on fast, lightweight document processing with spatial text extraction. It runs entirely locally with zero cloud dependencies by default.
+**LiteParse** is an open-source PDF parsing library written in **Rust**, focused on fast, lightweight document processing with spatial text extraction. It runs entirely locally with zero cloud dependencies by default.
+
+Language bindings are provided for **Node.js/TypeScript** (via napi-rs), **Python** (via PyO3), and **WebAssembly** (via wasm-bindgen).
### Key Capabilities
- **Spatial text extraction** with precise bounding boxes
-- **Flexible OCR** (built-in Tesseract.js or pluggable HTTP servers)
+- **Flexible OCR** (built-in Tesseract or pluggable HTTP servers)
- **Multi-format support** (PDFs, DOCX, XLSX, PPTX, images via conversion)
-- **TypeScript/Node.js** with both library and CLI interfaces
+- **Multi-language bindings**: Rust, Node.js/TypeScript, Python, Browser (WASM)
+- **CLI** available from all installation methods (`cargo`, `npm`, `pip`)
## Directory Structure
```
liteparse/
-├── src/
-│ ├── core/ # Configuration, types, main orchestrator
-│ ├── engines/ # Pluggable PDF and OCR engines
-│ │ ├── pdf/ # PDF parsing engines (PDF.js, PDFium)
-│ │ └── ocr/ # OCR engines (Tesseract, HTTP)
-│ ├── processing/ # Text extraction and spatial analysis
-│ ├── output/ # Output formatters (JSON, text)
-│ ├── conversion/ # Multi-format conversion to PDF
-│ ├── vendor/ # Bundled dependencies (PDF.js)
-│ ├── index.ts # CLI entry point
-│ └── lib.ts # Library public API
-├── cli/ # CLI implementation
-├── ocr/ # Example OCR server implementations
-│ ├── easyocr/ # EasyOCR wrapper server
-│ └── paddleocr/ # PaddleOCR wrapper server
-└── dist/ # Compiled JavaScript output
+├── crates/
+│ ├── liteparse/ # Core Rust library + CLI binary
+│ │ └── src/
+│ │ ├── main.rs # CLI entry point (clap)
+│ │ ├── lib.rs # Library root
+│ │ ├── parser.rs # LiteParse orchestrator
+│ │ ├── config.rs # Configuration types and defaults
+│ │ ├── types.rs # Core data types (ParseResult, TextItem, etc.)
+│ │ ├── projection.rs # Spatial grid projection (layout reconstruction)
+│ │ ├── extract.rs # Raw text extraction from PDFium
+│ │ ├── render.rs # Page rendering / screenshots
+│ │ ├── conversion.rs # Non-PDF format conversion (LibreOffice, ImageMagick)
+│ │ ├── ocr_merge.rs # Merging OCR results with native text
+│ │ ├── error.rs # Error types
+│ │ ├── ocr/ # OCR engine implementations
+│ │ │ ├── mod.rs # OcrEngine trait
+│ │ │ ├── tesseract.rs # Built-in Tesseract OCR
+│ │ │ └── http_simple.rs # HTTP OCR server client
+│ │ └── output/ # Output formatters
+│ │ ├── mod.rs
+│ │ ├── json.rs
+│ │ └── text.rs
+│ ├── liteparse-napi/ # Node.js bindings (napi-rs)
+│ ├── liteparse-python/ # Python bindings (PyO3 / maturin)
+│ ├── liteparse-wasm/ # WASM bindings (wasm-bindgen)
+│ ├── pdfium/ # Rust wrapper around PDFium C API
+│ └── pdfium-sys/ # PDFium FFI (C → Rust) bindings
+├── packages/
+│ ├── node/ # npm package: TS wrapper + CLI around native binary
+│ │ └── src/
+│ │ ├── lib.ts # Public LiteParse class for Node.js
+│ │ ├── cli.ts # CLI entry point (commander)
+│ │ └── native.ts # Native binary loader
+│ ├── python/ # PyPI package: Python wrapper around native binary
+│ │ └── liteparse/
+│ │ ├── __init__.py
+│ │ ├── parser.py # Public LiteParse class for Python
+│ │ ├── types.py # Python dataclass types
+│ │ └── cli.py # CLI entry point
+│ └── wasm/ # WASM npm package
+├── ocr/ # Example OCR server implementations
+│ ├── easyocr/ # EasyOCR wrapper server
+│ └── paddleocr/ # PaddleOCR wrapper server
+└── Cargo.toml # Workspace root
```
## Data Flow
-1. **Input**: File path received (any supported format)
-2. **Conversion** (if dependencies installed): Non-PDF formats converted to PDF via LibreOffice/ImageMagick
-3. **PDF Loading**: PDF.js extracts text items, images, metadata
-4. **OCR** (if enabled): Images rendered and OCR'd for text-sparse areas
+1. **Input**: File path or raw bytes received (any supported format)
+2. **Conversion** (if needed): Non-PDF formats converted to PDF via LibreOffice/ImageMagick
+3. **PDF Loading**: PDFium extracts text items, images, metadata
+4. **OCR** (if enabled): Pages rendered and OCR'd for text-sparse areas
5. **Grid Projection**: Spatial reconstruction of text layout using anchor system
6. **Post-processing**: Bounding boxes, text cleanup
7. **Output**: Formatted as JSON or plain text
## Key Design Decisions
-### 1. Engine Abstraction Pattern
-Both PDF and OCR functionality use interface-based abstraction (`PdfEngine`, `OcrEngine`). This allows:
-- Swapping implementations without changing core logic
-- Auto-detection: HTTP OCR if URL provided, otherwise Tesseract.js
-- Future extensibility for new engines
-- Future possibility of custom conversion engines for non-PDF formats
+### 1. Rust Core with Language Bindings
+The core parsing logic is written in Rust for performance and safety. Language-specific crates expose the same API surface:
+- `liteparse-napi` → Node.js via napi-rs
+- `liteparse-python` → Python via PyO3/maturin
+- `liteparse-wasm` → Browser via wasm-bindgen
+
+Each binding crate is thin — it wraps the core `liteparse` crate's types and async API.
+
+### 2. OCR Engine Trait
+OCR functionality uses a trait-based abstraction (`OcrEngine`). This allows:
+- Built-in Tesseract (default, compiled in via `tesseract-rs`)
+- HTTP OCR server client for remote engines
+- Custom JS-side OCR in the WASM build via a callback interface
-### 2. Spatial Grid Projection
-The most complex (and important!) part of the codebase (`src/processing/gridProjection.ts`, ~1650 lines). Uses:
+### 3. Spatial Grid Projection
+The most complex (and important!) part of the codebase (`crates/liteparse/src/projection.rs`). Uses:
- **Anchor-based layout**: Tracks text alignment (left, right, center, floating)
- **Forward anchors**: Carry alignment information between lines
- **Column detection**: Identifies multi-column layouts
- **Rotation handling**: Transforms 90°, 180°, 270° rotated text to correct reading order
- **OCR merging**: Combines native PDF text with OCR results, preserving confidence scores and source flags in output
-### 3. Selective OCR
+### 4. Selective OCR
OCR only runs on embedded images where text extraction failed, not the entire document. This balances accuracy with performance.
-### 4. Configuration Merging
-Uses a default-first approach where users only override what they need. Configuration flows: defaults → file config → CLI options.
+### 5. Configuration
+Uses a default-first approach where users only override what they need. See `crates/liteparse/src/config.rs` for defaults.
-### 5. Format Conversion via External Tools
-Rather than implementing format parsers, LiteParse converts non-PDF formats using system tools (LibreOffice, ImageMagick) into a single format (PDF). This provides broad format support with minimal code.
+### 6. Format Conversion via External Tools
+Rather than implementing format parsers, LiteParse converts non-PDF formats using system tools (LibreOffice, ImageMagick) into PDF. This provides broad format support with minimal code.
## Common Tasks
### Adding a New Output Format
-1. Create new file in `src/output/` implementing the formatter
-2. Add format option to `cli/parse.ts`
-3. Update `src/core/parser.ts` to use new formatter
+1. Create new file in `crates/liteparse/src/output/`
+2. Add variant to `OutputFormat` enum in `config.rs`
+3. Wire it up in `main.rs` and binding crates
### Adding a New OCR Engine
-1. Implement `OcrEngine` interface in `src/engines/ocr/`
-2. Add initialization logic in `src/core/parser.ts`
-3. Add configuration options in `src/core/types.ts`
+1. Implement `OcrEngine` trait in `crates/liteparse/src/ocr/`
+2. Add initialization logic in `parser.rs`
+3. Add configuration options in `config.rs`
### Modifying Text Extraction Logic
-The processing pipeline is in `src/processing/`. Key files:
-- `gridProjection.ts` - Layout reconstruction (most complex)
-- `bbox.ts` - Bounding box calculation
-- `cleanText.ts` - Text cleanup
+Key files in `crates/liteparse/src/`:
+- `projection.rs` — Layout reconstruction (most complex)
+- `extract.rs` — Raw text item extraction from PDFium
+- `ocr_merge.rs` — Merging OCR and native text
### Adding CLI Options
-1. Update `cli/parse.ts` with new Commander.js option
-2. Add corresponding config field in `src/core/types.ts`
-3. Update `src/core/config.ts` with default value
-4. Use the option in `src/core/parser.ts`
+1. Add field to `LiteParseConfig` in `config.rs`
+2. Add clap arg in `main.rs`
+3. Wire through `parser.rs`
+4. Expose in binding crates (`liteparse-napi`, `liteparse-python`, `liteparse-wasm`)
-## Testing Approach
+### Adding / Modifying Node.js Wrapper
+- Edit `packages/node/src/lib.ts` for library API changes
+- Edit `packages/node/src/cli.ts` for CLI changes
+- The native binary interface is defined in `packages/node/src/native.ts`
-Currently tested via manual verification with sample documents. The project would benefit from:
-- Unit tests for processing utilities
-- Integration tests with known PDFs
-- Snapshot tests for output formats
+### Adding / Modifying Python Wrapper
+- Edit `packages/python/liteparse/parser.py` for library API changes
+- Types are in `packages/python/liteparse/types.py`
+- CLI entry point is `packages/python/liteparse/cli.py`
## Key Dependencies
| Dependency | Purpose |
|------------|---------|
-| `pdfjs-dist` | PDF parsing and text extraction |
-| `@hyzyla/pdfium` | High-quality PDF rendering for screenshots |
-| `tesseract.js` | In-process OCR (zero setup) |
-| `sharp` | Image processing |
-| `commander` | CLI framework |
-| `zod` | Schema validation |
+| `pdfium` (C library) | PDF text extraction and rendering |
+| `tesseract-rs` | Built-in OCR engine (optional, via `tesseract` feature) |
+| `clap` | CLI framework |
+| `serde` / `serde_json` | Serialization |
+| `tokio` | Async runtime |
+| `reqwest` | HTTP client (for OCR server) |
+| `image` | Image processing (PNG encoding) |
+| `napi-rs` | Node.js native bindings |
+| `pyo3` / `maturin` | Python native bindings |
+| `wasm-bindgen` | WASM bindings |
## Entry Points
-- **CLI**: `src/index.ts` → `cli/parse.ts`
-- **Library**: `src/lib.ts` exports `LiteParse` class and types
-- **Main Class**: `src/core/parser.ts` contains `LiteParse` orchestrator
+- **Rust CLI**: `crates/liteparse/src/main.rs`
+- **Rust Library**: `crates/liteparse/src/lib.rs` → `parser.rs` contains `LiteParse` struct
+- **Node.js**: `packages/node/src/lib.ts` exports `LiteParse` class
+- **Python**: `packages/python/liteparse/parser.py` exports `LiteParse` class
+- **WASM**: `crates/liteparse-wasm/` exposes `LiteParse` via wasm-bindgen
## Related Documentation
-These files are key to understanding the codebase and should be referenced for specific implementation details.
-
-If changes to the codebase are being made, please update the relevant documentation files to reflect those changes and keep them up to date.
-
- [User-facing documentation](README.md)
-- [src/conversion/README.md](src/conversion/README.md) - Format conversion details
-- [src/core/README.md](src/core/README.md) - Core architecture and configuration
-- [src/engines/README.md](src/engines/README.md) - Engine abstraction and implementations
- - [src/engines/pdf/README.md](src/engines/pdf/README.md) - PDF engines (PDF.js, PDFium)
- - [src/engines/ocr/README.md](src/engines/ocr/README.md) - OCR engines (Tesseract, HTTP)
-- [src/output/README.md](src/output/README.md) - Output formatters
-- [src/processing/README.md](src/processing/README.md) - Text extraction and spatial processing
-- [ocr/README.md](ocr/README.md) - OCR server implementations (EasyOCR, PaddleOCR)
-- [cli/README.md](cli/README.md) - CLI usage and options
+- [OCR API Specification](OCR_API_SPEC.md)
+- [WASM package README](packages/wasm/README.md)
+- [Python package README](packages/python/README.md)
+- [OCR server examples](ocr/README.md)
diff --git a/README.md b/README.md
index bd57aee..a5838a5 100644
--- a/README.md
+++ b/README.md
@@ -2,8 +2,14 @@
[![CI](https://github.com/run-llama/liteparse/actions/workflows/ci.yml/badge.svg)](https://github.com/run-llama/liteparse/actions/workflows/ci.yml)
|
+[![Crates.io version](https://img.shields.io/crates/v/liteparse.svg)](https://crates.io/crates/liteparse)
+|
[![npm version](https://img.shields.io/npm/v/@llamaindex/liteparse.svg)](https://www.npmjs.com/package/@llamaindex/liteparse)
|
+[![wasm version](https://img.shields.io/npm/v/@llamaindex/liteparse-wasm.svg)](https://www.npmjs.com/package/@llamaindex/liteparse-wasm)
+|
+[![PyPI version](https://img.shields.io/pypi/v/liteparse.svg)](https://pypi.org/project/liteparse/)
+|
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
|
[Docs](https://developers.llamaindex.ai/liteparse/)
@@ -22,48 +28,91 @@ hard stuff so your models see clean, structured data and markdown.
## Overview
-- **Fast Text Parsing**: Spatial text parsing using PDF.js
+- **Fast Text Parsing**: Spatial text parsing using PDFium
- **Flexible OCR System**:
- - **Built-in**: Tesseract.js (zero setup, works out of the box!)
+ - **Built-in**: Tesseract (zero setup, bundled with the library)
- **HTTP Servers**: Plug in any OCR server (EasyOCR, PaddleOCR, custom)
- **Standard API**: Simple, well-defined OCR API specification
- **Screenshot Generation**: Generate high-quality page screenshots for LLM agents
- **Multiple Output Formats**: JSON and Text
- **Bounding Boxes**: Precise text positioning information
-- **Standalone Binary**: No cloud dependencies, runs entirely locally
+- **Multi-language**: Use from Rust, Node.js/TypeScript, Python, or the browser (WASM)
- **Multi-platform**: Linux, macOS (Intel/ARM), Windows
## Installation
-### CLI Tool
+All versions (except WASM) ship with the same CLI and library API. Install the one that fits your environment:
+
+<details>
+ <summary>Node.js / TypeScript</summary>
+
+Install via npm to use the `lit` CLI or the library API:
+
+```bash
+npm i -g @llamaindex/liteparse # CLI (global)
+npm i @llamaindex/liteparse # Library (project dependency)
+```
+
+Parse your first document right away:
+
+```bash
+lit parse document.pdf
+```
+
+Or use the library API in your Node.js or TypeScript project:
+
+```typescript
+import { LiteParse } from '@llamaindex/liteparse';
+
+const parser = new LiteParse({ ocrEnabled: true });
+
+const result = await parser.parse('document.pdf');
+console.log(result.text);
+```
+</details>
-#### Option 1: Global Install (Recommended)
+<details>
+ <summary>Python</summary>
-Install globally via npm to use the `lit` command anywhere:
+Install via pip to use the `lit` CLI or the library API:
```bash
-npm i -g @llamaindex/liteparse
+pip install liteparse
```
-Then use it:
+Parse your first document right away:
```bash
lit parse document.pdf
-lit screenshot document.pdf
```
-#### Option 2: Install from Source
+Or use the library API in your Python project:
-You can clone the repo and install the CLI globally from source:
+```python
+from liteparse import LiteParse
+parser = LiteParse(ocr_enabled=True)
+result = parser.parse('document.pdf')
+print(result.text)
```
-git clone https://github.com/run-llama/liteparse.git
-cd liteparse
-npm run build
-npm pack
-npm install -g ./llamaindex-liteparse-*.tgz
+
+</details>
+
+<details>
+ <summary>Browser (WASM)</summary>
+
+You can install a trimmed-down version of LiteParse that runs entirely in the browser, with no server or cloud dependencies.
+
+```bash
+npm install @llamaindex/liteparse-wasm
```
+It supports PDF parsing and custom OCR engines implemented in JavaScript.
+
+See the [WASM package README](packages/wasm/README.md) for usage details.
+
+</details>
+
### Agent Skill
You can use `liteparse` as an agent skill, downloading it with the `skills` CLI tool:
@@ -74,7 +123,9 @@ npx skills add run-llama/llamaparse-agent-skills --skill liteparse
Or copy-pasting the [`SKILL.md`](https://github.com/run-llama/llamaparse-agent-skills/blob/main/skills/liteparse/SKILL.md) file to your own skills setup.
-## Usage
+## CLI Usage
+
+The CLI is the same across all installations (`npm`, `pip`, or the Rust binary).
### Parse Files
@@ -83,7 +134,7 @@ Or copy-pasting the [`SKILL.md`](https://github.com/run-llama/llamaparse-agent-s
lit parse document.pdf
# Parse with specific format
-lit parse document.pdf --format json -o output.md
+lit parse document.pdf --format json -o output.json
# Parse specific pages
lit parse document.pdf --target-pages "1-5,10,15-20"
@@ -97,7 +148,7 @@ curl -sL https://example.com/report.pdf | lit parse -
### Batch Parsing
-You can also parse an entire directory of documents:
+Parse an entire directory of documents:
```bash
lit batch-parse ./input-directory ./output-directory
@@ -116,36 +167,77 @@ lit screenshot document.pdf --target-pages "1,3,5" -o ./screenshots
# Custom DPI
lit screenshot document.pdf --dpi 300 -o ./screenshots
+```
+
+### CLI Reference
+
+#### Parse Command
-# Screenshot page range
-lit screenshot document.pdf --target-pages "1-10" -o ./screenshots
```
+lit parse [OPTIONS] <file>
-### Library Usage
+Options:
+ -o, --output <file> Output file path
+ --format <format> Output format: json|text [default: text]
+ --no-ocr Disable OCR
+ --ocr-language <lang> OCR language, Tesseract format [default: eng]
+ --ocr-server-url <url> HTTP OCR server URL (uses Tesseract if not provided)
+ --tessdata-path <path> Path to tessdata directory
+ --max-pages <n> Max pages to parse [default: 1000]
+ --target-pages <pages> Pages to parse (e.g., "1-5,10,15-20")
+ --dpi <dpi> Rendering DPI [default: 150]
+ --preserve-small-text Keep very small text
+ --password <password> Password for encrypted documents
+ --num-workers <n> Concurrent OCR workers [default: CPU cores - 1]
+ -q, --quiet Suppress progress output
+ -h, --help Print help
+```
-Install as a dependency in your project:
+#### Batch Parse Command
-```bash
-npm install @llamaindex/liteparse
-# or
-pnpm add @llamaindex/liteparse
```
+lit batch-parse [OPTIONS] <input-dir> <output-dir>
-```typescript
-import { LiteParse } from '@llamaindex/liteparse';
+Options:
+ --format <format> Output format: json|text [default: text]
+ --no-ocr Disable OCR
+ --ocr-language <lang> OCR language [default: eng]
+ --ocr-server-url <url> HTTP OCR server URL
+ --tessdata-path <path> Path to tessdata directory
+ --max-pages <n> Max pages per file [default: 1000]
+ --dpi <dpi> Rendering DPI [default: 150]
+ --recursive Recursively search input directory
+ --extension <ext> Only process files with this extension (e.g., ".pdf")
+ --password <password> Password for encrypted documents
+ --num-workers <n> Concurrent OCR workers
+ -q, --quiet Suppress progress output
+ -h, --help Print help
+```
-const parser = new LiteParse({ ocrEnabled: true });
-const result = await parser.parse('document.pdf');
-console.log(result.text);
+#### Screenshot Command
+
+```
+lit screenshot [OPTIONS] <file>
+
+Options:
+ -o, --output-dir <dir> Output directory [default: ./screenshots]
+ --target-pages <pages> Pages to screenshot (e.g., "1,3,5" or "1-5")
+ --dpi <dpi> Rendering DPI [default: 150]
+ --password <password> Password for encrypted documents
+ -q, --quiet Suppress progress output
+ -h, --help Print help
```
-#### Buffer / Uint8Array Input
+## Library Usage
-You can pass raw bytes directly instead of a file path, which is useful for remote files:
+### Buffer / Uint8Array Input
+
+All APIs that accept file paths also accept raw bytes, so you can parse documents from any source (e.g. HTTP responses, in-memory buffers) without writing to disk first.
+
+The WASM package only accepts `Uint8Array` input, while the Node.js and Python versions accept both file paths and bytes.
```typescript
import { LiteParse } from '@llamaindex/liteparse';
-import { readFile } from 'fs/promises';
const parser = new LiteParse();
@@ -159,186 +251,39 @@ const buffer = Buffer.from(await response.arrayBuffer());
const result2 = await parser.parse(buffer);
```
-Non-PDF buffers (images, Office documents) are written to a temp directory for format conversion. Screenshots also work with buffer input:
-
-```typescript
-const screenshots = await parser.screenshot(pdfBytes, [1, 2, 3]);
-```
-
-### Browser Usage
-
-LiteParse's core parsing engine (PDF.js text extraction, grid projection, OCR via Tesseract.js) can run in the browser. Since the library has Node-only dependencies (sharp, fs, child_process), you'll need a bundler like Vite to swap those out with browser stubs.
-
-#### Vite Configuration
-
-The key is a Vite plugin that redirects Node-only source files to browser-safe replacements, plus `resolve.alias` entries that stub out Node built-in modules:
+#### Screenshots
```typescript
-// vite.config.ts
-import { defineConfig, type Plugin } from "vite";
-import { resolve, dirname } from "node:path";
-
-// Node-only files → browser stubs (you write these)
-const FILE_REDIRECTS = [
- { match: /\/engines\/pdf\/pdfium-renderer(\.js|\.ts)?$/, target: "stubs/pdfium-renderer.ts" },
- { match: /\/engines\/pdf\/pdfjsImporter(\.js|\.ts)?$/, target: "stubs/pdfjsImporter.ts" },
- { match: /\/engines\/ocr\/http-simple(\.js|\.ts)?$/, target: "stubs/http-simple.ts" },
- { match: /\/conversion\/convertToPdf(\.js|\.ts)?$/, target: "stubs/convertToPdf.ts" },
- { match: /\/processing\/gridDebugLogger(\.js|\.ts)?$/, target: "stubs/gridDebugLogger.ts" },
- { match: /\/processing\/gridVisualizer(\.js|\.ts)?$/, target: "stubs/gridVisualizer.ts" },
-];
-
-function liteparseNodeRedirects(): Plugin {
- return {
- name: "liteparse-node-redirects",
- enforce: "pre",
- async resolveId(source, importer) {
- if (!importer) return null;
- const abs = source.startsWith(".") ? resolve(dirname(importer), source) : source;
- for (const { match, target } of FILE_REDIRECTS) {
- if (match.test(abs) || match.test(source)) return resolve(target);
- }
- return null;
- },
- };
+const screenshots = await parser.screenshot('document.pdf', [1, 2, 3]);
+for (const s of screenshots) {
+ console.log(`Page ${s.pageNum}: ${s.width}x${s.height}`);
+ // s.imageBuffer contains PNG bytes
}
-
-export default defineConfig({
- plugins: [liteparseNodeRedirects()],
- optimizeDeps: { include: ["tesseract.js"] },
- resolve: {
- alias: [
- { find: "node:fs/promises", replacement: "stubs/empty.ts" },
- { find: "node:fs", replacement: "stubs/empty.ts" },
- { find: "node:url", replacement: "stubs/empty.ts" },
- { find: "node:path", replacement: "stubs/empty.ts" },
- { find: "node:os", replacement: "stubs/empty.ts" },
- { find: "node:child_process", replacement: "stubs/empty.ts" },
- { find: /^fs$/, replacement: "stubs/empty.ts" },
- { find: /^path$/, replacement: "stubs/empty.ts" },
- { find: /^os$/, replacement: "stubs/empty.ts" },
- { find: /^child_process$/, replacement: "stubs/empty.ts" },
- { find: "form-data", replacement: "stubs/empty.ts" },
- { find: "axios", replacement: "stubs/empty.ts" },
- { find: "file-type", replacement: "stubs/file-type.ts" },
- ],
- },
-});
-```
-
-See [`scripts/browser-compat/`](scripts/browser-compat/) for a complete working example with all the stub files.
-
-#### What works in the browser
-
-- PDF parsing from `Uint8Array` input (use `file.arrayBuffer()` to get bytes from a `<input type="file">`)
-- OCR via Tesseract.js (runs in Web Workers, fetches language data from CDN on first use)
-- Text and JSON output formats
-
-#### What doesn't work
-
-- File path input (pass `Uint8Array` instead)
-- DOCX/XLSX/PPTX/image conversion (requires LibreOffice/ImageMagick)
-- HTTP OCR server backend
-- Screenshots (these use PDFium + sharp, which are native Node addons)
-
-### CLI Options
-
-#### Parse Command
-
-```
-$ lit parse --help
-Usage: lit parse [options] <file>
-
-Parse a document file (PDF, DOCX, XLSX, PPTX, images, etc.)
-
-Options:
- -o, --output <file> Output file path
- --format <format> Output format: json|text (default: "text")
- --ocr-server-url <url> HTTP OCR server URL (uses Tesseract if not provided)
- --no-ocr Disable OCR
- --ocr-language <lang> OCR language(s) (default: "en")
- --num-workers <n> Number of pages to OCR in parallel (default: CPU cores - 1)
- --max-pages <n> Max pages to parse (default: "10000")
- --target-pages <pages> Target pages (e.g., "1-5,10,15-20")
- --dpi <dpi> DPI for rendering (default: "150")
- --no-precise-bbox Disable precise bounding boxes
- --preserve-small-text Preserve very small text
- --password <password> Password for encrypted/protected documents
- --config <file> Config file (JSON)
- -q, --quiet Suppress progress output
- -h, --help display help for command
-```
-
-#### Batch Parse Command
-
-```
-$ lit batch-parse --help
-Usage: lit batch-parse [options] <input-dir> <output-dir>
-
-Parse multiple documents in batch mode (reuses PDF engine for efficiency)
-
-Options:
- --format <format> Output format: json|text (default: "text")
- --ocr-server-url <url> HTTP OCR server URL (uses Tesseract if not provided)
- --no-ocr Disable OCR
- --ocr-language <lang> OCR language(s) (default: "en")
- --num-workers <n> Number of pages to OCR in parallel (default: CPU cores - 1)
- --max-pages <n> Max pages to parse per file (default: "10000")
- --dpi <dpi> DPI for rendering (default: "150")
- --no-precise-bbox Disable precise bounding boxes
- --recursive Recursively search input directory
- --extension <ext> Only process files with this extension (e.g., ".pdf")
- --password <password> Password for encrypted/protected documents (applied to all files)
- --config <file> Config file (JSON)
- -q, --quiet Suppress progress output
- -h, --help display help for command
-```
-
-#### Screenshot Command
-
-```
-$ lit screenshot --help
-Usage: lit screenshot [options] <file>
-
-Generate screenshots of PDF pages
-
-Options:
- -o, --output-dir <dir> Output directory for screenshots (default: "./screenshots")
- --target-pages <pages> Page numbers to screenshot (e.g., "1,3,5" or "1-5")
- --dpi <dpi> DPI for rendering (default: "150")
- --format <format> Image format: png|jpg (default: "png")
- --password <password> Password for encrypted/protected documents
- --config <file> Config file (JSON)
- -q, --quiet Suppress progress output
- -h, --help display help for command
```
## OCR Setup
-### Default: Tesseract.js
-
-```bash
-# Tesseract is enabled by default
-lit parse document.pdf
+### Default: Tesseract
-# Specify language
-lit parse document.pdf --ocr-language fra
+Tesseract is bundled and works out of the box:
-# Disable OCR
-lit parse document.pdf --no-ocr
+```bash
+lit parse document.pdf # OCR enabled by default
+lit parse document.pdf --ocr-language fra # Specify language
+lit parse document.pdf --no-ocr # Disable OCR
```
-By default, Tesseract.js downloads language data from the internet on first use. For offline or air-gapped environments, set the `TESSDATA_PREFIX` environment variable to a directory containing pre-downloaded `.traineddata` files:
+For offline or air-gapped environments, set `TESSDATA_PREFIX` to a directory containing pre-downloaded `.traineddata` files:
```bash
export TESSDATA_PREFIX=/path/to/tessdata
lit parse document.pdf --ocr-language eng
```
-You can also pass `tessdataPath` in the library config:
+Or pass the path directly:
-```typescript
-const parser = new LiteParse({ tessdataPath: '/path/to/tessdata' });
+```bash
+lit parse document.pdf --tessdata-path /path/to/tessdata
```
### Optional: HTTP OCR Servers
@@ -355,22 +300,18 @@ The API requires:
- Accepts `file` and `language` parameters
- Returns JSON: `{ results: [{ text, bbox: [x1,y1,x2,y2], confidence }] }`
-See the example servers in `ocr/easyocr/` and `ocr/paddleocr/` as templates.
-
-For the complete OCR API specification, see [`OCR_API_SPEC.md`](OCR_API_SPEC.md).
-
## Multi-Format Input Support
-LiteParse supports **automatic conversion** of various document formats to PDF before parsing. This makes it unique compared to other PDF-only parsing tools!
+LiteParse supports **automatic conversion** of various document formats to PDF before parsing.
### Supported Input Formats
#### Office Documents (via LibreOffice)
-- **Word**: `.doc`, `.docx`, `.docm`, `.odt`, `.rtf`
-- **PowerPoint**: `.ppt`, `.pptx`, `.pptm`, `.odp`
-- **Spreadsheets**: `.xls`, `.xlsx`, `.xlsm`, `.ods`, `.csv`, `.tsv`
+- **Word**: `.doc`, `.docx`, `.docm`, `.odt`, `.rtf`, `.pages`
+- **PowerPoint**: `.ppt`, `.pptx`, `.pptm`, `.odp`, `.key`
+- **Spreadsheets**: `.xls`, `.xlsx`, `.xlsm`, `.ods`, `.csv`, `.tsv`, `.numbers`
-Just install the dependency and LiteParse will automatically convert these formats to PDF for parsing:
+Install LibreOffice for automatic conversion:
```bash
# macOS
@@ -380,15 +321,15 @@ brew install --cask libreoffice
apt-get install libreoffice
# Windows
-choco install libreoffice-fresh # might require admin permissions
+choco install libreoffice-fresh
```
-> _For Windows, you might need to add the path to the directory containing LibreOffice CLI executable (generally `C:\Program Files\LibreOffice\program`) to the environment variables and re-start the machine._
+> _On Windows, you may need to add LibreOffice's program directory (usually `C:\Program Files\LibreOffice\program`) to your PATH._
#### Images (via ImageMagick)
- **Formats**: `.jpg`, `.jpeg`, `.png`, `.gif`, `.bmp`, `.tiff`, `.webp`, `.svg`
-Just install ImageMagick and LiteParse will convert images to PDF for parsing (with OCR):
+Install ImageMagick for image-to-PDF conversion:
```bash
# macOS
@@ -398,74 +339,51 @@ brew install imagemagick
apt-get install imagemagick
# Windows
-choco install imagemagick.app # might require admin permissions
+choco install imagemagick.app
```
## Environment Variables
| Variable | Description |
|----------|-------------|
-| `TESSDATA_PREFIX` | Path to a directory containing Tesseract `.traineddata` files. Used for offline/air-gapped environments where Tesseract.js cannot download language data from the internet. |
-| `LITEPARSE_TMPDIR` | Override the temp directory used for format conversion and intermediate files. Defaults to the OS temp directory (`os.tmpdir()`). Useful in containerized or read-only filesystem environments. |
-
-## Configuration
-
-You can configure parsing options via CLI flags or a JSON config file. The config file allows you to set sensible defaults and override as needed.
-
-### Config File Example
+| `TESSDATA_PREFIX` | Path to a directory containing Tesseract `.traineddata` files. Used for offline/air-gapped environments. |
-Create a `liteparse.config.json` file:
-
-```json
-{
- "ocrLanguage": "en",
- "ocrEnabled": true,
- "maxPages": 1000,
- "dpi": 150,
- "outputFormat": "json",
- "preciseBoundingBox": true,
- "preserveVerySmallText": false,
- "password": "optional_password"
-}
-```
+## Development
-For HTTP OCR servers, just add `ocrServerUrl`:
+The project is a Rust workspace with the core library and language-specific binding crates.
-```json
-{
- "ocrServerUrl": "http://localhost:8828/ocr",
- "ocrLanguage": "en",
- "outputFormat": "json"
-}
```
-
-Use with:
-
-```bash
-lit parse document.pdf --config liteparse.config.json
+crates/
+├── liteparse/ # Core library + CLI binary
+├── liteparse-napi/ # Node.js bindings (napi-rs)
+├── liteparse-python/ # Python bindings (PyO3)
+├── liteparse-wasm/ # WASM bindings (wasm-bindgen)
+├── pdfium/ # PDFium Rust wrapper
+└── pdfium-sys/ # PDFium FFI bindings
+packages/
+├── node/ # npm package (TS wrapper + native binary)
+├── python/ # PyPI package (Python wrapper + native binary)
+└── wasm/ # WASM npm package
```
-## Development
-
-We provide a fairly rich `AGENTS.md`/`CLAUDE.md` that we recommend using to help with development + coding agents.
+### Building
```bash
-# Install dependencies
-npm install
+# Build the CLI
+cargo build --release -p liteparse
-# Build TypeScript (Linux/macOs)
-npm run build
+# Build Node.js bindings
+cd packages/node && npm run build
-# Build Typescript (Windows)
-npm run build:windows
+# Build Python bindings
+cd packages/python && maturin develop --release
-# Watch mode
-npm run dev
-
-# Test parsing
-npm test
+# Build WASM
+cd packages/wasm && npm run build
```
+We provide a fairly rich `AGENTS.md`/`CLAUDE.md` that we recommend using to help with development + coding agents.
+
## License
Apache 2.0
@@ -474,8 +392,10 @@ Apache 2.0
Built on top of:
-- [PDF.js](https://github.com/mozilla/pdf.js) - PDF parsing engine
-- [Tesseract.js](https://github.com/naptha/tesseract.js) - In-process OCR engine
+- [PDFium](https://pdfium.googlesource.com/pdfium/) - PDF rendering and text extraction
+- [Tesseract](https://github.com/tesseract-ocr/tesseract) - OCR engine (via tesseract-rs)
- [EasyOCR](https://github.com/JaidedAI/EasyOCR) - HTTP OCR server (optional)
- [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR) - HTTP OCR server (optional)
-- [Sharp](https://github.com/lovell/sharp) - Image processing
+- [napi-rs](https://napi.rs/) - Node.js native bindings
+- [PyO3](https://pyo3.rs/) - Python native bindings
+- [wasm-bindgen](https://github.com/wasm-bindgen/wasm-bindgen) - WebAssembly bindings
diff --git a/packages/node/README.md b/packages/node/README.md
new file mode 100644
index 0000000..ed6f9d4
--- /dev/null
+++ b/packages/node/README.md
@@ -0,0 +1,89 @@
+# LiteParse Node.js
+
+Node.js/TypeScript bindings for [LiteParse](https://github.com/run-llama/liteparse) — fast, lightweight PDF and document parsing with spatial text extraction.
+
+## Installation
+
+```bash
+npm i @llamaindex/liteparse
+```
+
+This also installs the `lit` CLI command (use `npm i -g` for global access).
+
+## Quick Start
+
+```typescript
+import { LiteParse } from '@llamaindex/liteparse';
+
+const parser = new LiteParse();
+const result = await parser.parse('document.pdf');
+console.log(result.text);
+
+// Access structured data
+for (const page of result.pages) {
+ console.log(`Page ${page.pageNum}: ${page.textItems.length} text items`);
+}
+```
+
+## Configuration
+
+All options are passed to the constructor:
+
+```typescript
+const parser = new LiteParse({
+ ocrEnabled: true, // Enable OCR (default: true)
+ ocrLanguage: 'eng', // Tesseract language code
+ ocrServerUrl: undefined, // HTTP OCR server URL (optional)
+ tessdataPath: undefined, // Path to tessdata directory (optional)
+ maxPages: 1000, // Max pages to parse
+ targetPages: '1-5,10', // Specific pages (optional)
+ dpi: 150, // Rendering DPI
+ preserveVerySmallText: false, // Keep tiny text
+ password: undefined, // Password for protected documents
+ quiet: false, // Suppress progress output
+ numWorkers: 4, // Concurrent OCR workers
+});
+```
+
+## Parsing from Bytes
+
+Pass a `Buffer` or `Uint8Array` directly — useful for HTTP responses or in-memory data:
+
+```typescript
+import { readFile } from 'fs/promises';
+
+const pdfBytes = await readFile('document.pdf');
+const result = await parser.parse(pdfBytes);
+console.log(result.text);
+```
+
+## Screenshots
+
+Generate PNG screenshots of document pages:
+
+```typescript
+const screenshots = parser.screenshot('document.pdf', [1, 2, 3]);
+for (const s of screenshots) {
+ console.log(`Page ${s.pageNum}: ${s.width}x${s.height}`);
+ // s.imageBuffer contains PNG bytes
+}
+```
+
+## Supported Formats
+
+- PDF (`.pdf`)
+- Microsoft Office (`.docx`, `.xlsx`, `.pptx`, etc.) — requires LibreOffice
+- OpenDocument (`.odt`, `.ods`, `.odp`) — requires LibreOffice
+- Images (`.png`, `.jpg`, `.tiff`, etc.) — requires ImageMagick
+- And more!
+
+## CLI
+
+The npm package includes the `lit` CLI:
+
+```bash
+lit parse document.pdf
+lit parse document.pdf --format json -o output.json
+lit screenshot document.pdf -o ./screenshots
+lit batch-parse ./input ./output
+```
diff --git a/packages/python/README.md b/packages/python/README.md
index 5871876..c155039 100644
--- a/packages/python/README.md
+++ b/packages/python/README.md
@@ -1,122 +1,86 @@
# LiteParse Python
-Python wrapper for [LiteParse](https://github.com/run-llama/liteparse) - fast, lightweight document parsing with optional OCR.
-
-> **Important:** This package is a Python wrapper around the LiteParse Node.js CLI.
-> You must have **Node.js** (>= 18) installed on your system. The CLI will be auto-installed
-> via npm on first use if not already present, or you can install it manually beforehand.
+Python bindings for [LiteParse](https://github.com/run-llama/liteparse) — fast, lightweight PDF and document parsing with spatial text extraction.
## Installation
-### Step 1: Install Node.js
-
-LiteParse requires Node.js (>= 18). Install it from [nodejs.org](https://nodejs.org/) or via your package manager.
-
-### Step 2: Install the LiteParse CLI
-
-```bash
-npm install -g @llamaindex/liteparse
-```
-
-### Step 3: Install the Python package
-
```bash
pip install liteparse
```
-> **Note:** If you skip Step 2, the Python package will attempt to auto-install the CLI
-> via `npm install -g @llamaindex/liteparse` on first use (requires npm in your PATH).
+This also installs the `lit` CLI command.
## Quick Start
```python
from liteparse import LiteParse
-# Create parser
parser = LiteParse()
-
-# Parse a document
result = parser.parse("document.pdf")
print(result.text)
# Access structured data
for page in result.pages:
- print(f"Page {page.pageNum}: {len(page.textItems)} text items")
+ print(f"Page {page.page_num}: {len(page.text_items)} text items")
```
## Configuration
-All parsing options are passed per-call to `parse()`:
+All options are passed to the constructor:
```python
-from liteparse import LiteParse
-
-parser = LiteParse()
-
-result = parser.parse(
- "document.pdf",
- ocr_enabled=False,
- max_pages=10,
- dpi=150,
- preserve_very_small_text=True,
+parser = LiteParse(
+ ocr_enabled=True, # Enable OCR (default: True)
+ ocr_language="eng", # Tesseract language code
+ ocr_server_url=None, # HTTP OCR server URL (optional)
+ tessdata_path=None, # Path to tessdata directory (optional)
+ max_pages=1000, # Max pages to parse
+ target_pages="1-5,10", # Specific pages (optional)
+ dpi=150, # Rendering DPI
+ preserve_very_small_text=False, # Keep tiny text
+ password=None, # Password for protected documents
+ quiet=False, # Suppress progress output
+ num_workers=4, # Concurrent OCR workers
)
-print(result.text)
```
-## Parsing from bytes
+## Parsing from Bytes
-If you already have file contents in memory (e.g. from a web upload), pass them directly:
+Pass raw PDF bytes directly — useful for web uploads or downloaded files:
```python
with open("document.pdf", "rb") as f:
- pdf_bytes = f.read()
-
-result = parser.parse(pdf_bytes)
+ result = parser.parse(f.read())
print(result.text)
```
-## Batch Processing
+## Screenshots
-For parsing multiple files, batch mode is significantly faster as it reuses the PDF engine:
+Generate PNG screenshots of document pages:
```python
-from liteparse import LiteParse
-
-parser = LiteParse()
-
-# Parse all documents in a directory
-result = parser.batch_parse(
- input_dir="./documents",
- output_dir="./output",
- ocr_enabled=False,
- recursive=True, # Include subdirectories
- extension_filter=".pdf", # Only PDF files
-)
-
-print(f"Output written to: {result.output_dir}")
+screenshots = parser.screenshot("document.pdf", page_numbers=[1, 2, 3])
+for s in screenshots:
+ print(f"Page {s.page_num}: {s.width}x{s.height}")
+ with open(f"page_{s.page_num}.png", "wb") as f:
+ f.write(s.image_bytes)
```
## Supported Formats
- PDF (`.pdf`)
-- Microsoft Office (`.docx`, `.xlsx`, `.pptx`, etc.) - requires LibreOffice
-- OpenDocument (`.odt`, `.ods`, `.odp`) - requires LibreOffice
-- Images (`.png`, `.jpg`, `.tiff`, etc.) - requires ImageMagick
+- Microsoft Office (`.docx`, `.xlsx`, `.pptx`, etc.) — requires LibreOffice
+- OpenDocument (`.odt`, `.ods`, `.odp`) — requires LibreOffice
+- Images (`.png`, `.jpg`, `.tiff`, etc.) — requires ImageMagick
- And more!
-## Performance Tips
+## CLI
-1. **Disable OCR** if your documents have selectable text:
- ```python
- result = parser.parse("doc.pdf", ocr_enabled=False)
- ```
+The Python package includes the `lit` CLI:
-2. **Use batch mode** for multiple files to avoid cold-start overhead:
- ```python
- parser.batch_parse("./input", "./output")
- ```
-
-3. **Limit pages** if you only need specific pages:
- ```python
- result = parser.parse("doc.pdf", target_pages="1-5")
- ```
+```bash
+lit parse document.pdf
+lit parse document.pdf --format json -o output.json
+lit screenshot document.pdf -o ./screenshots
+lit batch-parse ./input ./output
+```