A fast, helpful, and open-source document parser
add readme architecture
Logan Markewichf6a2b83ebbba
diff --git a/AGENTS.md b/AGENTS.mdnew file mode 100644index 0000000..f8dd5fc--- /dev/null+++ b/AGENTS.md@@ -0,0 +1,133 @@+# LiteParse - Agent Documentation++> This file provides comprehensive context for AI coding agents working on this codebase. Each subdirectory contains its own README with file-specific documentation.++## Project Overview++**LiteParse** is an open-source PDF parsing library focused on fast, lightweight document processing with spatial text extraction. It runs entirely locally with zero cloud dependencies by default.++### Key Capabilities+- **Spatial text extraction** with precise bounding boxes+- **Flexible OCR** (built-in Tesseract.js or pluggable HTTP servers)+- **Multi-format support** (PDFs, DOCX, XLSX, PPTX, images via conversion)+- **Table detection** via heuristic line/path analysis+- **TypeScript/Node.js** with both library and CLI interfaces++## Directory Structure++```+liteparse/+├── src/+│ ├── core/ # Configuration, types, main orchestrator+│ ├── engines/ # Pluggable PDF and OCR engines+│ │ ├── pdf/ # PDF parsing engines (PDF.js, PDFium)+│ │ └── ocr/ # OCR engines (Tesseract, HTTP)+│ ├── processing/ # Text extraction and spatial analysis+│ ├── output/ # Output formatters (JSON, text)+│ ├── conversion/ # Multi-format conversion to PDF+│ ├── vendor/ # Bundled dependencies (PDF.js)+│ ├── index.ts # CLI entry point+│ └── lib.ts # Library public API+├── cli/ # CLI implementation+├── ocr/ # Example OCR server implementations+│ ├── easyocr/ # EasyOCR wrapper server+│ └── paddleocr/ # PaddleOCR wrapper server+└── dist/ # Compiled JavaScript output+```++## Data Flow++1. **Input**: File path received (any supported format)+2. **Conversion** (if dependencies installed): Non-PDF formats converted to PDF via LibreOffice/ImageMagick+3. **PDF Loading**: PDF.js extracts text items, paths, images, metadata+4. **OCR** (if enabled): Images rendered and OCR'd for text-sparse areas+5. **Grid Projection**: Spatial reconstruction of text layout using anchor system+6. **Post-processing**: Bounding boxes, table detection, text cleanup+7. **Output**: Formatted as JSON or plain text++## Key Design Decisions++### 1. Engine Abstraction Pattern+Both PDF and OCR functionality use interface-based abstraction (`PdfEngine`, `OcrEngine`). This allows:+- Swapping implementations without changing core logic+- Auto-detection: HTTP OCR if URL provided, otherwise Tesseract.js+- Future extensibility for new engines+- Future possibility of custom conversion engines for non-PDF formats++### 2. Spatial Grid Projection+The most complex (and important!) part of the codebase (`src/processing/gridProjection.ts`, ~1500 lines). Uses:+- **Anchor-based layout**: Tracks text alignment (left, right, center, floating)+- **Forward anchors**: Carry alignment information between lines+- **Column detection**: Identifies multi-column layouts+- **OCR merging**: Combines native PDF text with OCR results++### 3. Selective OCR+OCR only runs on embedded images where text extraction failed, not the entire document. This balances accuracy with performance.++### 4. Configuration Merging+Uses a default-first approach where users only override what they need. Configuration flows: defaults → file config → CLI options.++### 5. Format Conversion via External Tools+Rather than implementing format parsers, LiteParse converts non-PDF formats using system tools (LibreOffice, ImageMagick) into a single format (PDF). This provides broad format support with minimal code.++## Common Tasks++### Adding a New Output Format+1. Create new file in `src/output/` implementing the formatter+2. Add format option to `cli/parse.ts`+3. Update `src/core/parser.ts` to use new formatter++### Adding a New OCR Engine+1. Implement `OcrEngine` interface in `src/engines/ocr/`+2. Add initialization logic in `src/core/parser.ts`+3. Add configuration options in `src/core/types.ts`++### Modifying Text Extraction Logic+The processing pipeline is in `src/processing/`. Key files:+- `gridProjection.ts` - Layout reconstruction (most complex)+- `bbox.ts` - Bounding box calculation+- `tables.ts` - Table detection+- `cleanText.ts` - Text cleanup++### Adding CLI Options+1. Update `cli/parse.ts` with new Commander.js option+2. Add corresponding config field in `src/core/types.ts`+3. Update `src/core/config.ts` with default value+4. Use the option in `src/core/parser.ts`++## Testing Approach++Currently tested via manual verification with sample documents. The project would benefit from:+- Unit tests for processing utilities+- Integration tests with known PDFs+- Snapshot tests for output formats++## Key Dependencies++| Dependency | Purpose |+|------------|---------|+| `pdfjs-dist` | PDF parsing and text extraction |+| `@hyzyla/pdfium` | High-quality PDF rendering for screenshots |+| `tesseract.js` | In-process OCR (zero setup) |+| `sharp` | Image processing |+| `commander` | CLI framework |+| `zod` | Schema validation |++## Entry Points++- **CLI**: `src/index.ts` → `cli/parse.ts`+- **Library**: `src/lib.ts` exports `LiteParse` class and types+- **Main Class**: `src/core/parser.ts` contains `LiteParse` orchestrator++## Related Documentation++- [User-facing documentation](README.md)+- [src/conversion/README.md](src/conversion/README.md) - Format conversion details+- [src/core/README.md](src/core/README.md) - Core architecture and configuration+- [src/engines/README.md](src/engines/README.md) - Engine abstraction and implementations+ - [src/engines/pdf/README.md](src/engines/pdf/README.md) - PDF engines (PDF.js, PDFium)+ - [src/engines/ocr/README.md](src/engines/ocr/README.md) - OCR engines (Tesseract, HTTP)+- [src/output/README.md](src/output/README.md) - Output formatters+- [src/processing/README.md](src/processing/README.md) - Text extraction and spatial processing+- [ocr/README.md](ocr/README.md) - OCR server implementations (EasyOCR, PaddleOCR)+- [cli/README.md](cli/README.md) - CLI usage and optionsdiff --git a/CLAUDE.md b/CLAUDE.mdnew file mode 100644index 0000000..eef4bd2--- /dev/null+++ b/CLAUDE.md@@ -0,0 +1 @@\ No newline at end of filediff --git a/cli/README.md b/cli/README.mdnew file mode 100644index 0000000..4b373e9--- /dev/null+++ b/cli/README.md@@ -0,0 +1,126 @@+# cli/++Command-line interface for LiteParse using Commander.js.++## Files++### parse.ts+**CLI entry point with two main commands: `parse` and `screenshot`.**++---++## Commands++### `liteparse parse <file>`+Parse documents and extract text.++**Options:**+| Flag | Description | Default |+|------|-------------|---------|+| `-o, --output <file>` | Save output to file | stdout |+| `--format <format>` | Output format: `json` or `text` | `text` |+| `--ocr-server-url <url>` | HTTP OCR server URL | (uses Tesseract) |+| `--no-ocr` | Disable OCR entirely | enabled |+| `--ocr-language <lang>` | OCR language code | `en` |+| `--max-pages <n>` | Maximum pages to parse | `1000` |+| `--target-pages <pages>` | Specific pages (e.g., "1-5,10") | all |+| `--dpi <dpi>` | Render resolution | `150` |+| `--no-tables` | Disable table detection | enabled |+| `--no-precise-bbox` | Disable bounding boxes | enabled |+| `--skip-diagonal-text` | Skip diagonal/rotated text | false |+| `--preserve-small-text` | Keep very small text | false |+| `--config <file>` | Load config from JSON file | - |+| `-q, --quiet` | Suppress progress output | false |++**Examples:**+```bash+# Basic text extraction+liteparse parse document.pdf++# JSON output with OCR disabled+liteparse parse document.pdf --format json --no-ocr++# Parse specific pages+liteparse parse document.pdf --target-pages "1-5,10,15-20"++# Use external OCR server+liteparse parse document.pdf --ocr-server-url http://localhost:5000/ocr++# Save to file+liteparse parse document.pdf -o output.txt+```++---++### `liteparse screenshot <file>`+Generate page screenshots.++**Options:**+| Flag | Description | Default |+|------|-------------|---------|+| `-o, --output-dir <dir>` | Output directory | `./screenshots` |+| `--pages <pages>` | Page numbers (e.g., "1,3,5" or "1-5") | all |+| `--dpi <dpi>` | Render resolution | `150` |+| `--format <format>` | Image format: `png` or `jpg` | `png` |+| `--config <file>` | Load config from JSON file | - |+| `-q, --quiet` | Suppress progress output | false |++**Examples:**+```bash+# All pages as PNG+liteparse screenshot document.pdf++# Specific pages as JPG at 300 DPI+liteparse screenshot document.pdf --pages "1,3,5" --format jpg --dpi 300++# Custom output directory+liteparse screenshot document.pdf -o ./my-screenshots+```++---++## Configuration File++Both commands accept `--config <file>` to load settings from JSON:++```json+{+ "ocrEnabled": true,+ "ocrLanguage": "en",+ "ocrServerUrl": "http://localhost:5000/ocr",+ "maxPages": 100,+ "dpi": 200,+ "outputFormat": "json",+ "tableDetection": true,+ "preciseBoundingBox": true+}+```++CLI options override config file values.++---++## Output Behavior++- **Progress messages** go to stderr (visible in terminal)+- **Parsed content** goes to stdout (can be piped)+- Use `-o` to save to file instead of stdout+- Use `-q` to suppress all progress messages++**Piping example:**+```bash+# Extract and search+liteparse parse document.pdf -q | grep "keyword"++# Chain with other tools+liteparse parse document.pdf --format json -q | jq '.pages[0].text'+```++---++## Adding CLI Options++1. Add `.option()` call in the command definition+2. Read option in action handler from `options` object+3. Add to config object that's passed to `LiteParse`+4. If new config field, update `src/core/types.ts` and `src/core/config.ts`diff --git a/ocr/README.md b/ocr/README.mdnew file mode 100644index 0000000..5987c13--- /dev/null+++ b/ocr/README.md@@ -0,0 +1,119 @@+# ocr/++Example OCR server implementations that conform to the LiteParse OCR API specification.++These servers allow you to use alternative OCR engines instead of the built-in Tesseract.js.++## Why Use an External OCR Server?++| Feature | Tesseract.js (built-in) | EasyOCR | PaddleOCR |+|---------|-------------------------|---------|-----------|+| Setup | Zero (included) | Docker | Docker |+| Speed | Moderate | Moderate | Fast (2-3x) |+| Accuracy (Latin) | Good | Good | Good |+| Accuracy (CJK) | Fair | Good | Excellent |+| Languages | 100+ | 80+ | 80+ |+| Memory | In-process | Separate | Separate |++**Recommendations:**+- **Quick start**: Use built-in Tesseract (no setup)+- **Asian languages**: Use PaddleOCR (best CJK support)+- **General use**: EasyOCR (good balance)++## Available Servers++### [easyocr/](./easyocr/)+Flask server wrapping EasyOCR library.+- Port: **8828**+- Good general-purpose OCR+- 80+ languages++### [paddleocr/](./paddleocr/)+Flask server wrapping PaddleOCR library.+- Port: **8829**+- Excellent for Chinese, Japanese, Korean+- 2-3x faster than EasyOCR++## Quick Start++```bash+# Start EasyOCR server+cd ocr/easyocr+docker build -t liteparse-easyocr .+docker run -p 8828:8828 liteparse-easyocr++# OR start PaddleOCR server+cd ocr/paddleocr+docker build -t liteparse-paddleocr .+docker run -p 8829:8829 liteparse-paddleocr+```++Then use with LiteParse:++```bash+# CLI+liteparse parse document.pdf --ocr-server-url http://localhost:8828/ocr++# Code+const parser = new LiteParse({+ ocrServerUrl: 'http://localhost:8828/ocr',+ ocrLanguage: 'en',+});+```++## API Specification++All servers implement the same API (defined in `OCR_API_SPEC.md`):++**Endpoint:** `POST /ocr`++**Request:**+- Content-Type: `multipart/form-data`+- Fields:+ - `file` - Image file+ - `language` - Language code (e.g., 'en', 'zh', 'ja')++**Response:**+```json+{+ "results": [+ {+ "text": "recognized text",+ "bbox": [x1, y1, x2, y2],+ "confidence": 0.95+ }+ ]+}+```++## Creating a Custom OCR Server++To implement your own OCR server:++1. Create a Flask/FastAPI/Express server+2. Accept `POST /ocr` with multipart form data+3. Return JSON with `results` array containing:+ - `text` - Recognized text string+ - `bbox` - Bounding box as `[x1, y1, x2, y2]`+ - `confidence` - Confidence score (0-1)++4. (Optional) Implement `GET /health` endpoint++See the existing servers as reference implementations.++## Language Codes++Most servers accept ISO 639-1 codes (e.g., 'en', 'zh', 'ja') and map them internally:++| ISO Code | Language | Notes |+|----------|----------|-------|+| en | English | |+| zh | Chinese (Simplified) | |+| zh-tw | Chinese (Traditional) | |+| ja | Japanese | |+| ko | Korean | |+| fr | French | |+| de | German | |+| es | Spanish | |+| ar | Arabic | |+| hi | Hindi | |diff --git a/src/conversion/README.md b/src/conversion/README.mdnew file mode 100644index 0000000..514db2f--- /dev/null+++ b/src/conversion/README.md@@ -0,0 +1,61 @@+# src/conversion/++Multi-format document conversion to PDF using external tools.++## Files++### convertToPdf.ts+**Converts non-PDF documents to PDF for parsing.**++LiteParse's core parsing works on PDFs. This module extends support to 50+ formats by converting them to PDF first using system tools.++**Supported Formats:**++| Category | Extensions |+|----------|------------|+| Office | .doc, .docx, .docm, .dot, .dotm, .dotx, .odt, .ott, .rtf, .pages |+| Presentations | .ppt, .pptx, .pptm, .pot, .potm, .potx, .odp, .otp, .key |+| Spreadsheets | .xls, .xlsx, .xlsm, .xlsb, .ods, .ots, .csv, .tsv, .numbers |+| Images | .jpg, .jpeg, .png, .gif, .bmp, .tiff, .tif, .webp, .svg |+| Web | .htm, .html, .xhtml (not yet implemented) |++**External Dependencies:**+- **LibreOffice** - For office documents and spreadsheets+ - macOS: `brew install --cask libreoffice`+ - Ubuntu: `apt-get install libreoffice`+- **ImageMagick** - For images+ - macOS: `brew install imagemagick`+ - Ubuntu: `apt-get install imagemagick`++**Key Functions:**++`convertToPdf(filePath)` - Main entry point+- Returns `ConversionResult` with `pdfPath` and `originalExtension`+- Returns `ConversionPassthrough` with `content` for text-based formats (e.g. .txt, .csv)+- Returns `ConversionError` with `message` and `code` on failure+- If already PDF, returns path unchanged++`cleanupConversionFiles(pdfPath)` - Removes temp files after parsing+- Only deletes files in system temp directory+- Called by `LiteParse.parse()` after processing++`guessFileExtension(filePath)` - Detects format from extension or magic bytes+- Checks file extension first+- Falls back to magic byte detection (PDF, PNG, JPEG, ZIP-based)++**Design Decisions:**++1. **External tools over native parsers**: Using LibreOffice and ImageMagick provides broad format support with minimal code. Trade-off is external dependency requirement.++2. **Subprocess-based**: Conversion runs as separate process to isolate crashes and memory issues.++3. **Temporary file management**: Converted PDFs go to system temp directory, cleaned up after parsing.++4. **Timeout handling**: 2 minutes for LibreOffice, 1 minute for ImageMagick.++5. **Magic byte fallback**: Handles files without extensions or with wrong extensions.++**Error Codes:**+- `FILE_NOT_FOUND` - Input file doesn't exist+- `UNSUPPORTED_FORMAT` - Format not supported+- `CONVERSION_ERROR` - Conversion process faileddiff --git a/src/core/README.md b/src/core/README.mdnew file mode 100644index 0000000..6d1323a--- /dev/null+++ b/src/core/README.md@@ -0,0 +1,116 @@+# src/core/++The core module contains the main orchestrator class, configuration management, and TypeScript type definitions that form the foundation of LiteParse.++## Files++### parser.ts+**The main LiteParse class** - orchestrates the entire parsing pipeline.++**Key Responsibilities:**+- Initializes PDF and OCR engines based on configuration+- Coordinates the document processing pipeline+- Manages document lifecycle (load, process, cleanup)+- Handles format conversion for non-PDF inputs++**Public Methods:**+- `parse(filePath, quiet?)` - Main entry point for document parsing+- `screenshot(filePath, pageNumbers?, quiet?)` - Generate page screenshots+- `getConfig()` - Returns current configuration++**Pipeline Flow (in `parse()`):**+1. Convert to PDF if needed (via `convertToPdf`)+2. Load PDF document with PDF engine+3. Extract pages+4. Run OCR on text-sparse pages/embedded images+5. Project pages to grid (spatial text reconstruction)+6. Build bounding boxes (if enabled)+7. Detect tables (if enabled)+8. Format output (JSON or text)+9. Cleanup resources++**Design Decisions:**+- **Engine auto-detection**: If `ocrServerUrl` is provided, uses HTTP OCR; otherwise defaults to Tesseract.js for zero-setup experience+- **Selective OCR**: Only runs OCR on pages with <100 characters of text OR pages with embedded images+- **Progress logging to stderr**: Keeps stdout clean for piped output+- **Graceful cleanup**: Always cleans up temp files and terminates OCR workers++---++### types.ts+**TypeScript type definitions** for the entire library.++**Configuration Types:**+- `LiteParseConfig` - All configuration options+- `OutputFormat` - 'json' | 'text'++**Data Types:**+- `TextItem` - Individual text element with position, font, rotation+- `ProjectionTextBox` - Extended text item with projection metadata (snap, anchors, etc.)+- `BoundingBox` - Coordinates (x1, y1, x2, y2)+- `ParsedPage` - Complete page data (text, textItems, tables, images)+- `ParseResult` - Final parse output+- `ScreenshotResult` - Screenshot output with image buffer++**Key Type Details:**++`TextItem` contains:+- Position: `x`, `y`, `width`, `height` (also `w`, `h` aliases)+- Font info: `fontName`, `fontSize`+- Rotation: `r` (degrees), `rx`/`ry` (rotated coordinates)+- Markup: `markup` (highlight, underline, strikeout)+- Layout hints: `vgap`, `isPlaceholder`++`ProjectionTextBox` adds projection metadata:+- `snap` - Alignment type ('left', 'right', 'center', or undefined for floating)+- `leftAnchor`, `rightAnchor`, `centerAnchor` - Column anchor strings+- `isDup` - Marks duplicate text items+- `forceUnsnapped` - Override snap detection+- `fromOCR` - Text came from OCR++---++### config.ts+**Default configuration and merging logic.**++**DEFAULT_CONFIG Values:**+```typescript+{+ ocrLanguage: 'en',+ ocrEnabled: true,+ ocrServerUrl: undefined, // Uses Tesseract if not set+ maxPages: 1000,+ dpi: 150,+ outputFormat: 'json',+ includeImages: true,+ includeCharts: true,+ tableDetection: true,+ preciseBoundingBox: true,+ skipDiagonalText: false,+ preserveVerySmallText: false,+ preserveLayoutAlignmentAcrossPages: false,+}+```++**mergeConfig(userConfig)**+Simple spread-based merge: `{ ...DEFAULT_CONFIG, ...userConfig }`++**Design Decision:**+The merge is intentionally shallow. Users provide only the options they want to change. This keeps the API simple while ensuring all required fields exist.++---++## Common Modifications++### Adding a new configuration option+1. Add field to `LiteParseConfig` interface in `types.ts`+2. Add default value in `DEFAULT_CONFIG` in `config.ts`+3. Use the option in `parser.ts` or pass to relevant module++### Adding a new output format+1. Add to `OutputFormat` type in `types.ts`+2. Create formatter in `src/output/`+3. Add case to switch in `parser.ts` `parse()` method++### Adding lifecycle hooks+The `parse()` method in `parser.ts` has clear pipeline stages. Add hooks between stages as needed (e.g., before/after OCR, before/after grid projection).diff --git a/src/engines/README.md b/src/engines/README.mdnew file mode 100644index 0000000..7be732d--- /dev/null+++ b/src/engines/README.md@@ -0,0 +1,40 @@+# src/engines/++The engines module provides pluggable abstractions for PDF parsing and OCR functionality. Each engine type has an interface and one or more implementations.++## Directory Structure++```+engines/+├── pdf/ # PDF parsing engines+│ ├── interface.ts # PdfEngine interface and data types+│ ├── pdfjs.ts # PDF.js implementation (default)+│ └── pdfium-renderer.ts # PDFium for high-quality screenshots+└── ocr/ # OCR engines+ ├── interface.ts # OcrEngine interface+ ├── tesseract.ts # Tesseract.js (in-process, zero-setup)+ └── http-simple.ts # HTTP client for external OCR servers+```++## Design Philosophy++**Interface-based abstraction** allows swapping implementations without changing core logic:++1. **PdfEngine** - Defines how PDFs are loaded, pages extracted, and images rendered+2. **OcrEngine** - Defines how images are OCR'd to extract text++The `LiteParse` class in `src/core/parser.ts` uses these interfaces, allowing future engines to be added without modifying the orchestrator.++## Adding a New Engine++### New PDF Engine+1. Implement `PdfEngine` interface in `src/engines/pdf/`+2. Add initialization logic in `src/core/parser.ts` constructor+3. Optionally add config option to select the engine++### New OCR Engine+1. Implement `OcrEngine` interface in `src/engines/ocr/`+2. Add initialization logic in `src/core/parser.ts` constructor+3. Update engine auto-detection logic if needed++See subdirectory READMEs for file-specific documentation.diff --git a/src/engines/ocr/README.md b/src/engines/ocr/README.mdnew file mode 100644index 0000000..fa55bde--- /dev/null+++ b/src/engines/ocr/README.md@@ -0,0 +1,84 @@+# src/engines/ocr/++OCR engines for extracting text from images.++## Files++### interface.ts+**Defines the OcrEngine contract.**++```typescript+interface OcrEngine {+ name: string;+ recognize(imagePath: string, options: OcrOptions): Promise<OcrResult[]>;+ recognizeBatch(imagePaths: string[], options: OcrOptions): Promise<OcrResult[][]>;+}++interface OcrOptions {+ language: string | string[];+ correctRotation?: boolean;+}++interface OcrResult {+ text: string;+ bbox: [number, number, number, number]; // [x1, y1, x2, y2]+ confidence: number; // 0-1 scale+}+```++---++### tesseract.ts+**In-process OCR using Tesseract.js - zero setup required.**++This is the default OCR engine when no `ocrServerUrl` is configured.++**Key Features:**+- Worker-based processing (runs in background thread)+- Language caching (reuses worker for same language)+- Automatic language code normalization (e.g., `en` → `eng`)+- Low-confidence filtering (removes results < 30%)++**Language Normalization:**+Maps common ISO 639-1 codes to Tesseract's 3-letter codes:+- `en` → `eng`, `fr` → `fra`, `de` → `deu`, `es` → `spa`+- `zh` → `chi_sim`, `zh-tw` → `chi_tra`, `ja` → `jpn`, `ko` → `kor`+- Unknown codes passed through as-is++**Lifecycle:**+- `initialize(language)` - Create/recreate worker for language+- `recognize(imagePath, options)` - OCR single image+- `recognizeBatch(imagePaths, options)` - OCR multiple images sequentially+- `terminate()` - Clean up worker (called by LiteParse after parsing)++**Design Decisions:**+- **30% confidence threshold**: Filters noisy OCR results (tesseract.ts:59)+- **Worker caching**: Avoids expensive reinitialization when language unchanged+- **Sequential batch processing**: Tesseract.js workers aren't parallelizable++---++### http-simple.ts+**HTTP client for external OCR servers.**++Used when `ocrServerUrl` is configured. Allows using more powerful OCR backends (EasyOCR, PaddleOCR, etc.) running as separate services.++**API Specification:**+Servers must conform to `OCR_API_SPEC.md`:+- **Endpoint**: POST to configured URL+- **Request**: `multipart/form-data` with `file` (image) and `language` fields+- **Response**: `{ results: [{ text, bbox: [x1,y1,x2,y2], confidence }] }`++**Features:**+- 60-second timeout per request+- Graceful error handling (returns empty array on failure)+- Detailed error logging for debugging++**Design Decision:**+Simple sequential processing rather than batching because:+1. External servers may have their own batching/queuing+2. Keeps the API simple and predictable+3. Allows per-image error handling++**Example Servers:**+See `ocr/easyocr/` and `ocr/paddleocr/` for reference implementations.diff --git a/src/engines/pdf/README.md b/src/engines/pdf/README.mdnew file mode 100644index 0000000..1070b79--- /dev/null+++ b/src/engines/pdf/README.md@@ -0,0 +1,80 @@+# src/engines/pdf/++PDF parsing engines for loading documents and extracting content.++## Files++### interface.ts+**Defines the PdfEngine contract and data types.**++**PdfEngine Interface:**+```typescript+interface PdfEngine {+ name: string;+ loadDocument(filePath: string): Promise<PdfDocument>;+ extractPage(doc: PdfDocument, pageNum: number): Promise<PageData>;+ extractAllPages(doc, maxPages?, targetPages?): Promise<PageData[]>;+ renderPageImage(doc, pageNum, dpi): Promise<Buffer>;+ close(doc: PdfDocument): Promise<void>;+}+```++**Key Data Types:**+- `PdfDocument` - Loaded document with `numPages`, `data` (Uint8Array), `metadata`+- `PageData` - Extracted page: `pageNum`, `width`, `height`, `textItems`, `paths`, `images`, `annotations`+- `Path` - Vector path: `type` ('rectangle'|'line'|'curve'), `points`, `color`, `width`+- `Image` - Embedded image with position, dimensions, and optional OCR data+- `Annotation` - PDF annotation (links, etc.)++---++### pdfjs.ts+**Default PDF engine using Mozilla's PDF.js library.**++**Key Responsibilities:**+- Load PDF documents with CMap and font support+- Extract text items with precise coordinates and font information+- Handle coordinate transformations (PDF space → viewport space)+- Parse `targetPages` syntax (e.g., "1-5,10,15-20")+- Delegate screenshot rendering to PDFium for quality++**Coordinate Transformation:**+PDF uses a bottom-left origin with Y pointing up. PDF.js provides transformation matrices that this code processes using:+- `multiplyMatrices()` - Combine viewport and item transforms+- `applyTransformation()` - Apply matrix to points+- `singularValueDecompose2dScale()` - Extract scale factors via SVD+- `getRotation()` - Extract rotation angle from matrix++**Text Decoding:**+Handles special PDF.js markers for problematic fonts:+- Decodes `:->|>_<charCode>_<fontChar>_<|<-:` format+- Handles pipe-separated characters: `|a| |b| |c|` → `abc`++**Design Decisions:**+- **Stores PDF path**: Needed for PDFium rendering (pdfjs.ts:95)+- **Filters off-page items**: Removes text with negative coords or beyond page bounds (pdfjs.ts:191-198)+- **Zero-size filtering**: Skips text items with 0 width/height (pdfjs.ts:125)++---++### pdfium-renderer.ts+**High-quality screenshot renderer using native PDFium library.**++Used for generating page images for OCR and the `screenshot` command. Provides better quality than PDF.js canvas rendering, especially for documents with inline images.++**Key Features:**+- Native C++ PDFium via `@hyzyla/pdfium` WASM binding+- Sharp for image processing (PNG output with configurable compression)+- DPI-based scaling (72 DPI baseline)+- Lazy initialization (only loads when first needed)++**Methods:**+- `init()` - Initialize PDFium library (called automatically)+- `renderPageToBuffer(pdfPath, pageNumber, dpi)` - Render page to PNG buffer+- `close()` - Cleanup PDFium resources++**Design Decision:**+Separate from PdfJsEngine because:+1. PDFium renders better quality images (important for OCR accuracy)+2. PDF.js is better for text extraction (mature parsing)+3. Separation allows independent optimization of each concerndiff --git a/src/output/README.md b/src/output/README.mdnew file mode 100644index 0000000..b9de3b4--- /dev/null+++ b/src/output/README.md@@ -0,0 +1,81 @@+# src/output/++Output formatters that convert parsed document data into different formats.++## Files++### json.ts+**Structured JSON output formatter.**++**Functions:**++`buildJSON(pages)` - Builds structured JSON object:+```javascript+{+ pages: [{+ page: number,+ width: number,+ height: number,+ text: string,+ textItems: [{+ text: string,+ x, y, width, height: number,+ fontName: string,+ fontSize: number+ }],+ boundingBoxes: BoundingBox[],+ tables: DetectedTable[],+ images: Image[]+ }]+}+```++`formatJSON(result)` - Returns JSON string with 2-space indentation.++**Use Case:**+- When you need structured data for further processing+- Integration with other tools or pipelines+- Debugging text extraction issues++---++### text.ts+**Plain text output formatter.**++**Functions:**++`formatText(result)` - Formats all pages with headers:+```+--- Page 1 ---+[page 1 text]++--- Page 2 ---+[page 2 text]+```++`formatPageText(page)` - Returns single page text (no header).++**Use Case:**+- Human-readable output+- Piping to other tools (grep, awk, etc.)+- Quick document inspection++---++## Adding a New Output Format++1. Create new file in this directory (e.g., `markdown.ts`)+2. Export formatter function(s)+3. Add format to `OutputFormat` type in `src/core/types.ts`+4. Add case to switch in `src/core/parser.ts` `parse()` method+5. Add CLI option in `cli/parse.ts`++**Example: Adding Markdown format**+```typescript+// src/output/markdown.ts+export function formatMarkdown(result: ParseResult): string {+ return result.pages.map(page => {+ return `## Page ${page.pageNum}\n\n${page.text}`;+ }).join('\n\n---\n\n');+}+```diff --git a/src/processing/README.md b/src/processing/README.mdnew file mode 100644index 0000000..2a1a046--- /dev/null+++ b/src/processing/README.md@@ -0,0 +1,243 @@+# src/processing/++The processing module is the **heart of LiteParse** - responsible for transforming raw PDF content into structured, spatially-aware text. This is where text extraction, layout reconstruction, table detection, and OCR integration happen.++## Files++### grid.ts+**Thin wrapper around gridProjection.ts.**++Simply merges user config with defaults and calls `projectPagesToGridComplete`. Exists to provide a cleaner import path and handle configuration.++---++### gridProjection.ts (~1500 lines)+**The most complex module - spatial text layout reconstruction.**++This is the core algorithm that converts raw PDF text items into readable, properly-ordered text that preserves document layout.++**Key Concepts:**++1. **Anchors**: Track where text aligns on the page+ - `anchorLeft` - Text left-edges (column starts)+ - `anchorRight` - Text right-edges (column ends)+ - `anchorCenter` - Text centers (centered content)++2. **Snap Types**: How text aligns to columns+ - `left` - Text snaps to left edge of a column+ - `right` - Text snaps to right edge+ - `center` - Text is centered+ - `floating` - Unaligned/justified text++3. **Forward Anchors**: Carry alignment information between lines+ - Enables consistent column detection across the page+ - Prevents duplicate text detection (`isDup` flag)++**Algorithm Flow:**+1. Build bounding boxes from text items and OCR data+2. Sort text into lines by Y-coordinate+3. Extract anchor points from all text items+4. Detect text snapping (left, right, center, or floating)+5. Project lines onto character grid with proper spacing+6. Apply markup tags (highlight, underline, strikeout)+7. Clean up sparse blocks and margins++**Constants:**+- `FLOATING_SPACES = 2` - Minimum spaces between floating text+- `COLUMN_SPACES = 4` - Minimum spaces between columns+- `SMALL_FONT_SIZE_THRESHOLD = 2` - Filter very small text (2pt)++**Design Decisions:**+- **Anchor rounding**: Groups anchor x-coords by nearest 1/4 unit to handle slight variations+- **Sparse block compression**: Reduces excessive whitespace in sparse layouts (>70% whitespace)+- **Vertical line anchoring**: Uses PDF path lines to establish column boundaries+- **Small text filtering**: Lines with >50% small text can be filtered (configurable)++---++### bbox.ts+**Bounding box construction and OCR integration.**++**Key Functions:**++`buildBbox(pageData, config)` - Main function that:+1. Converts `TextItem[]` to `ProjectionTextBox[]` with additional metadata+2. Processes embedded images for OCR if enabled+3. Filters images by size, position, and type+4. Filters OCR results that overlap with existing text (50% threshold)+5. Returns combined text boxes for grid projection++`buildBoundingBoxes(textItems)` - Simple conversion of text items to `BoundingBox[]` format (x1, y1, x2, y2).++**OCR Filtering Constants:**+- `OCR_CONFIDENCE_THRESHOLD = 0.1` - Minimum OCR confidence+- `OCR_OVERLAP_THRESHOLD = 0.5` - Reject OCR if >50% overlaps existing text+- `MAX_IMAGES_PER_PAGE = 10` - Limit images processed per page+- `MIN_IMAGE_DIMENSION = 12` - Skip tiny images+- `MIN_IMAGE_AREA = 200` - Skip small-area images++**Design Decision:**+OCR overlap filtering prevents duplicate text when both PDF text extraction and OCR detect the same content. Native PDF text is preferred over OCR.++---++### tables.ts+**Heuristic table detection via line intersection analysis.**++Detects tables with explicit borders (outlined tables). Does not detect borderless tables.++**Algorithm:**+1. Extract horizontal and vertical lines from PDF paths+2. Cluster nearby lines to reduce noise (5px threshold)+3. Find line intersections (tolerance: 3px)+4. Create grid cells from intersection points+5. Assign text items to cells based on center point+6. Validate table has minimum rows/cols (2x2, 4 cells min)++**Key Functions:**+- `detectTables(paths, textItems)` - Main entry point+- `extractLines(paths)` - Gets lines from paths and rectangles+- `findIntersections(horizontal, vertical)` - Finds grid points+- `createCellsFromIntersections(intersections)` - Builds cell grid+- `assignTextToCells(cells, textItems)` - Maps text to cells++**Output:**+```typescript+interface DetectedTable {+ x1, y1, x2, y2: number; // Table bounds+ rows, cols: number; // Grid dimensions+ cells: TableCell[]; // Cell data with text+}+```++**Limitation:** Only works with outlined/bordered tables. Borderless tables require layout analysis not yet implemented.++---++### pathUtils.ts+**Extract lines from PDF vector paths.**++Provides input for table detection and layout analysis.++`getVerticalAndHorizontalLinesFromPagePaths(config, paths)`:+- Extracts vertical lines (x1 ≈ x2) from line paths+- Extracts horizontal lines (y1 ≈ y2) from line paths+- Extracts edges from rectangle paths+- Returns `{ verticalLines, horizontalLines }`++Used by `gridProjection.ts` to establish column anchors from visible lines.++---++### ocrUtils.ts+**OCR result parsing and coordinate conversion.**++**Key Functions:**++`parseImageOcrBlocks(image)`:+- Converts OCR bounding boxes from image space to page space+- Handles scale factors and coordinate ratios+- Returns `OcrBlock[]` with both page-space and raw coordinates++`easyOcrResultLinesToList(stdOutResult)`:+- Parses EasyOCR stdout format into structured data+- Format: `([[x1,y1], [x2,y2], [x3,y3], [x4,y4]], 'text', confidence)`++**Coordinate Systems:**+- **Image space**: Coordinates relative to the OCR'd image+- **Page space**: Coordinates relative to the PDF page viewport+- Conversion uses `xRatio = image.width / coords.w`++---++### textUtils.ts+**Unicode subscript and superscript conversion.**++Used for scientific notation and mathematical text.++- `strToSubscriptString(str)` - Converts "H2O" → "H₂O"+- `strToPostScript(str)` - Converts "x2" → "x²"++Supports: digits 0-9, +/-, common letters (a-z, A-Z subset)++If any character lacks a Unicode equivalent, returns original string unchanged.++---++### cleanText.ts+**Post-processing text cleanup.**++`cleanRawText(pages, config)`:+1. **Margin removal** (per page):+ - Detects consistent left margin (leading whitespace)+ - Removes top margin (empty lines at start)+ - Removes bottom margin (empty lines at end)+ - Trims right margin (trailing whitespace)+2. **Null character removal**: Replaces `\u0000` with spaces++---++### markupUtils.ts+**Apply inline markup tags from PDF annotations.**++`applyMarkupTags(markup, text)` - Wraps text with markup:+- Strikeout → `~~text~~`+- Underline → `__text__`+- Squiggly → `__text__` (same as underline)+- Highlight → `==text==`++These tags can be converted to markdown or other formats downstream.++---++## Data Flow++```+PageData (from PDF engine)+ │+ ▼+buildBbox() ─────────────────────────────┐+ │ │+ │ Converts TextItems to │ Processes embedded+ │ ProjectionTextBox[] │ images with OCR data+ │ │+ └────────────┬───────────────────────┘+ │+ ▼+ ProjectionTextBox[] (unified text boxes)+ │+ ▼+ projectPagesToGrid()+ │+ ┌────────────┼────────────┐+ │ │ │+ ▼ ▼ ▼+Extract Detect Project+Anchors Snapping to Grid+ │ │ │+ └────────────┴────────────┘+ │+ ▼+ ParsedPage[] with reconstructed text+ │+ ▼+ cleanRawText() - Remove margins, cleanup+ │+ ▼+ Final text output+```++## Common Modifications++### Adjusting column detection sensitivity+Modify `roundAnchor()` in `gridProjection.ts` - currently rounds to nearest 1/4 unit.++### Changing OCR overlap threshold+Modify `OCR_OVERLAP_THRESHOLD` in `bbox.ts` (default: 0.5 = 50%).++### Adding new markup types+1. Add field to `MarkupData` in `src/core/types.ts`+2. Add case in `applyMarkupTags()` in `markupUtils.ts`++### Improving table detection+The current approach only handles bordered tables. To support borderless tables, you'd need to add layout analysis based on text alignment patterns in `tables.ts`.