A fast, helpful, and open-source document parser
complexity docs
Logan Markewicha97abfcf81bc
diff --git a/README.md b/README.mdindex 03f86bb..a6a721a 100644--- a/README.md+++ b/README.md@@ -37,6 +37,7 @@ hard stuff so your models see clean, structured data and markdown.- **Built-in**: Tesseract (zero setup, bundled with the library)- **HTTP Servers**: Plug in any OCR server (EasyOCR, PaddleOCR, custom)- **Standard API**: Simple, well-defined OCR API specification+- **Complexity Detection**: Cheaply check whether a document needs OCR or heavier parsing — route, reject, or estimate cost before a full parse- **Screenshot Generation**: Generate high-quality page screenshots for LLM agents- **Multiple Output Formats**: Markdown, JSON, and Text- **Markdown Output**: Structured Markdown with headings, tables, lists, images, and links — great for feeding LLMs and RAG pipelines@@ -196,6 +197,28 @@ Image handling is controlled by `--image-mode`:> [LlamaParse](https://developers.llamaindex.ai/python/cloud/llamaparse/?utm_source=github&utm_medium=liteparse)> remains the most accurate option.+### Check Complexity++Before committing to a full parse, check whether a document actually needs OCR or+heavier processing. This is a cheap, text-layer-only pass — useful for routing+documents to different pipelines, rejecting ones you can't handle, or estimating cost.++```bash+# Print the complexity verdict and per-page JSON+lit is-complex document.pdf++# Use as a shell predicate — only parse with --no-ocr when the document is simple+lit is-complex document.pdf --quiet && lit parse document.pdf --no-ocr++# List the pages that need OCR+lit is-complex document.pdf --compact | jq '[.[] | select(.needs_ocr) | .page_number]'+```++It always prints per-page JSON to **stdout**, a human-readable verdict to **stderr**, and+exits **non-zero when any page needs OCR**. Each page carries a `needs_ocr` verdict and a+list of `reasons` (`scanned`, `no-text`, `sparse-text`, `embedded-images`, `garbled`,+`vector-text`).+### Batch ParsingParse an entire directory of documents:@@ -281,6 +304,23 @@ Options:-h, --help Print help```+#### Is-Complex Command++```+lit is-complex [OPTIONS] <file>++Options:+ --compact Emit dense, whitespace-free JSON instead of pretty-printed+ --max-pages <n> Max pages to check [default: 1000]+ --target-pages <pages> Pages to check (e.g., "1-5,10,15-20")+ --password <password> Password for encrypted documents+ -q, --quiet Suppress the stderr verdict+ -h, --help Print help+```++Prints per-page JSON to stdout and a `COMPLEX`/`SIMPLE` verdict to stderr; exits non-zero+when any page needs OCR, so it composes as a shell predicate.+## OCR Setup### Default: Tesseractdiff --git a/crates/liteparse/README.md b/crates/liteparse/README.mdindex 342fa71..0cfd892 100644--- a/crates/liteparse/README.md+++ b/crates/liteparse/README.md@@ -102,6 +102,32 @@ let result = parser.parse_input(PdfInput::Bytes(pdf_bytes)).await?;println!("{}", result.text);```+## Document Complexity++Before committing to a full parse, check whether a document needs OCR or heavier+processing. `is_complex` is a cheap, text-layer-only pass that returns a+`PageComplexityStats` per page with a `needs_ocr` verdict and the signals behind it —+useful for routing documents to different pipelines, rejecting ones you can't handle, or+estimating cost.++```rust+use liteparse::types::PdfInput;++let parser = LiteParse::new(LiteParseConfig::default());+let pages = parser.is_complex(PdfInput::Path("document.pdf".into())).await?;++if pages.iter().any(|p| p.needs_ocr) {+ // Route to the OCR-enabled pipeline, inspect `p.reasons`, etc.+ for page in pages.iter().filter(|p| p.needs_ocr) {+ println!("Page {} needs OCR: {:?}", page.page_number, page.reasons);+ }+}+```++`reasons` is a `Vec<ComplexityReason>` (`Scanned`, `NoText`, `SparseText`,+`EmbeddedImages`, `Garbled`, `VectorText`); new variants may be added over time, so match+leniently.+## Custom OCR EngineImplement the `OcrEngine` trait to plug in your own OCR backend:@@ -135,6 +161,7 @@ lit parse document.pdf --format json -o output.jsonlit parse document.pdf --format markdown -o output.mdlit screenshot document.pdf -o ./screenshotslit batch-parse ./input ./output+lit is-complex document.pdf```See `lit --help` for all options.diff --git a/docs/src/content/docs/liteparse/cli-reference.md b/docs/src/content/docs/liteparse/cli-reference.mdindex fc9182f..6649207 100644--- a/docs/src/content/docs/liteparse/cli-reference.md+++ b/docs/src/content/docs/liteparse/cli-reference.md@@ -5,7 +5,7 @@ sidebar:order: 5----LiteParse provides the `lit` CLI with three commands: `parse`, `batch-parse`, and `screenshot`. The CLI is the same whether installed via `npm`, `pip`, or built from Rust source.+LiteParse provides the `lit` CLI with four commands: `parse`, `batch-parse`, `screenshot`, and `is-complex`. The CLI is the same whether installed via `npm`, `pip`, or built from Rust source.## `lit parse`@@ -164,6 +164,47 @@ lit screenshot document.pdf --target-pages "1,5,10" -o ./pages---+## `lit is-complex`++Check whether a document needs OCR or heavier parsing — a cheap pre-parse pass over the text layer only (no rasterization, no OCR). See the [Document Complexity guide](/liteparse/guides/complexity/) for details.++```+lit is-complex [options] <file>+```++The command prints per-page JSON to **stdout**, a human-readable verdict to **stderr**, and exits **non-zero when any page needs OCR** — so it works as a shell predicate or a `jq` source.++### Arguments++| Argument | Description |+|----------|-------------|+| `file` | Path to the document file |++### Options++| Option | Description | Default |+|--------|-------------|---------|+| `--compact` | Emit dense, whitespace-free JSON instead of pretty-printed | — |+| `--max-pages <n>` | Maximum pages to check | `1000` |+| `--target-pages <pages>` | Pages to check (e.g., `"1-5,10,15-20"`) | — (all pages) |+| `--password <password>` | Password for encrypted/protected documents | — |+| `-q, --quiet` | Suppress the stderr verdict | — |++### Examples++```bash+# Print the complexity verdict and per-page JSON+lit is-complex document.pdf++# Use as a shell predicate: only parse with --no-ocr when simple+lit is-complex document.pdf --quiet && lit parse document.pdf --no-ocr++# List the page numbers that need OCR+lit is-complex document.pdf --compact | jq '[.[] | select(.needs_ocr) | .page_number]'+```++---+## Global optionsThese options are available on all commands:diff --git a/docs/src/content/docs/liteparse/getting_started.mdx b/docs/src/content/docs/liteparse/getting_started.mdxindex ba9ea5b..74c5ec1 100644--- a/docs/src/content/docs/liteparse/getting_started.mdx+++ b/docs/src/content/docs/liteparse/getting_started.mdx@@ -87,6 +87,7 @@ lit screenshot document.pdf -o ./screenshots- [Library usage](/liteparse/guides/library-usage/): Use LiteParse programmatically from TypeScript or Python.- [OCR configuration](/liteparse/guides/ocr/): Configure Tesseract, use an external OCR server, or bring your own.- [Multi-format support](/liteparse/guides/multi-format/): Parse DOCX, XLSX, PPTX, images, and more.+- [Document complexity](/liteparse/guides/complexity/): Check whether a document needs OCR / heavy processing before parsing, and route accordingly.- [Browser usage (WASM)](/liteparse/guides/browser-usage/): Run LiteParse entirely in the browser.- [Agent skill](/liteparse/guides/agent-skill/): Add LiteParse as a skill for coding agents.- [CLI reference](/liteparse/cli-reference/): Complete command and option reference.diff --git a/docs/src/content/docs/liteparse/guides/complexity.mdx b/docs/src/content/docs/liteparse/guides/complexity.mdxnew file mode 100644index 0000000..1e6d345--- /dev/null+++ b/docs/src/content/docs/liteparse/guides/complexity.mdx@@ -0,0 +1,181 @@+---+title: Document Complexity+description: Use is_complex to decide whether a document needs OCR or heavier parsing before you commit to it.+sidebar:+ order: 9+---++The `is_complex` command in LiteParse checks whether a document is "complex". This means looking for images, broken/garbled text, large amounts of vector graphics, and sparse text.++Use it to:++- **Route documents** to cheaper or more expensive parsing/OCR backends.+- **Reject or flag** documents you can't handle — e.g. when running with `--no-ocr`, find the pages that would come back empty.+- **Estimate cost** up front by counting how many pages actually need OCR.++## How it works++Complexity is computed **per page**. Each page gets a `needs_ocr` verdict and a list of `reasons` explaining why it was flagged. A document is "complex" if any of its pages needs OCR.++The reasons are derived from a few signals:++| Reason | Meaning |+|--------|---------|+| `scanned` | A single raster covers ~the whole page with little or no text behind it — a scanned/photographed page. |+| `no-text` | Almost no extractable text and no full-page image — a blank page, cover, or divider. |+| `sparse-text` | Some real text, but it covers very little of the page — typically a figure with a thin caption. |+| `embedded-images` | Substantial embedded raster figures sit alongside the native text. |+| `garbled` | The native text decodes to garbage and text is likely unreadable. |+| `vector-text` | Text is painted as filled vector outlines outside the text layer, so no native text items represent it. |++> The set of reasons may grow over time as the router learns to recommend heavier pipelines. Treat `reasons` as an open-ended list and route on the values you care about rather than assuming it's exhaustive.++## CLI++```bash+lit is-complex document.pdf+```++The command always prints per-page JSON to **stdout**, a human-readable verdict to **stderr**, and sets its **exit code** to reflect the result — so you can consume it however fits your workflow.++```jsonc+// stdout+[+ {+ "page_number": 1,+ "text_length": 0,+ "text_coverage": 0.0,+ "has_substantial_images": false,+ "image_block_count": 0,+ "image_coverage": 0.0,+ "largest_image_coverage": 0.0,+ "full_page_image": true,+ "uncovered_vector_area": null,+ "is_garbled": false,+ "page_area": 482400.0,+ "needs_ocr": true,+ "reasons": ["scanned"]+ }+]+```++```+# stderr+COMPLEX — 1/1 page(s) need OCR+```++The exit code is **non-zero when any page needs OCR**, so the command works as a shell predicate:++```bash+# Only parse with --no-ocr when the document is simple+lit is-complex document.pdf --quiet && lit parse document.pdf --no-ocr+```++Pipe the JSON into `jq` to act on individual pages. The example below will list the page numbers that need OCR:++```bash+lit is-complex document.pdf --compact | jq '[.[] | select(.needs_ocr) | .page_number]'+```++### Options++| Flag | Description |+|------|-------------|+| `--compact` | Emit dense, whitespace-free JSON instead of pretty-printed. |+| `--max-pages <N>` | Maximum number of pages to check (default: 1000). |+| `--target-pages <SPEC>` | Check only specific pages, e.g. `"1-5,10,15-20"`. |+| `--password <PASS>` | Password for encrypted/protected documents. |+| `-q`, `--quiet` | Suppress the stderr logging output. |++## Library++The same check is available programmatically. It returns one entry per page.++<Tabs>+<TabItem label="TypeScript" value="typescript">++```typescript+import { LiteParse } from "@llamaindex/liteparse";++const parser = new LiteParse({ ocrEnabled: false });+const pages = await parser.isComplex("document.pdf");++const complex = pages.some((p) => p.needsOcr);+if (complex) {+ // Route to a heavweight pipeline, like LlamaParse+ ...+} else {+ // Cheap path — skip OCR entirely+ const result = await parser.parse("document.pdf");+}++// Inspect why specific pages were flagged+for (const page of pages.filter((p) => p.needsOcr)) {+ console.log(`Page ${page.pageNumber}: ${page.reasons.join(", ")}`);+}+```++</TabItem>+<TabItem label="Python" value="python">++```python+from liteparse import LiteParse++parser = LiteParse(ocr_enabled=False)+pages = parser.is_complex("document.pdf")++if any(p.needs_ocr for p in pages):+ # Route to a heavweight pipeline, like LlamaParse+ ...+else:+ # Cheap path — skip OCR entirely+ result = parser.parse("document.pdf")++# Inspect why specific pages were flagged+for page in pages:+ if page.needs_ocr:+ print(f"Page {page.page_number}: {', '.join(page.reasons)}")+```++</TabItem>+<TabItem label="Browser (WASM)" value="wasm">++```javascript+import { LiteParse } from "@llamaindex/liteparse-wasm";++const parser = new LiteParse();+const pages = await parser.isComplex(pdfBytes); // Uint8Array++const complex = pages.some((p) => p.needsOcr);+```++</TabItem>+</Tabs>++Both the library and CLI accept raw bytes as well as file paths, so you can run the check on documents you've already loaded into memory.++## Per-page fields++Every entry includes the raw signals behind the verdict, so you can apply your own thresholds instead of relying solely on `needs_ocr`:++| Field | Description |+|-------|-------------|+| `page_number` | 1-indexed page number. |+| `needs_ocr` | Verdict: the page needs more than the cheap text-only path. Equivalent to `reasons` being non-empty. |+| `reasons` | Every reason the page was flagged (see the table above). Empty exactly when `needs_ocr` is false. |+| `text_length` | Length of usable native text (garbled/unmappable text excluded). |+| `text_coverage` | Fraction of the page area covered by native text (0–1). |+| `has_substantial_images` | Whether any counted inline raster figures are present. |+| `image_block_count` | Number of counted raster image objects (full-page backgrounds excluded). |+| `image_coverage` | Summed image-bbox area over page area, clamped to 1. |+| `largest_image_coverage` | Largest single counted image's area over page area, clamped to 1. |+| `full_page_image` | A single raster covers ≥90% of the page — the signal that tells a scan apart from a blank page. |+| `uncovered_vector_area` | Filled vector-outline area not covered by native text (pt²). `null`/`undefined` when a cheaper signal already decided the page. |+| `is_garbled` | Whether the native text decodes to garbage. |+| `page_area` | Page area in pt². |++## Next steps++- [OCR configuration](/liteparse/guides/ocr/): Set up the OCR backend you route complex documents to.+- [Library usage](/liteparse/guides/library-usage/): Full programmatic API for TypeScript and Python.+- [CLI reference](/liteparse/cli-reference/): Complete command and option reference.diff --git a/packages/node/README.md b/packages/node/README.mdindex 271e9c6..c30a02f 100644--- a/packages/node/README.md+++ b/packages/node/README.md@@ -90,6 +90,34 @@ for (const s of screenshots) {}```+## Document Complexity++Before committing to a full parse, check whether a document needs OCR or heavier+processing. `isComplex` is a cheap, text-layer-only pass that returns one entry per page+with a `needsOcr` verdict and the signals behind it — useful for routing documents to+different pipelines, rejecting ones you can't handle, or estimating cost.++```typescript+const parser = new LiteParse();+const pages = await parser.isComplex('document.pdf');++if (pages.some((p) => p.needsOcr)) {+ // Route to the OCR-enabled pipeline+ const result = await parser.parse('document.pdf');+} else {+ // Cheap path — skip OCR entirely+ const result = await new LiteParse({ ocrEnabled: false }).parse('document.pdf');+}++// Inspect why specific pages were flagged+for (const page of pages.filter((p) => p.needsOcr)) {+ console.log(`Page ${page.pageNumber}: ${page.reasons.join(', ')}`);+}+```++`reasons` is one of `"scanned"`, `"no-text"`, `"sparse-text"`, `"embedded-images"`,+`"garbled"`, or `"vector-text"`. Raw bytes work here too.+## Supported Formats- PDF (`.pdf`)@@ -107,4 +135,5 @@ lit parse document.pdflit parse document.pdf --format json -o output.jsonlit screenshot document.pdf -o ./screenshotslit batch-parse ./input ./output+lit is-complex document.pdf```diff --git a/packages/python/README.md b/packages/python/README.mdindex c938cee..a624501 100644--- a/packages/python/README.md+++ b/packages/python/README.md@@ -87,6 +87,33 @@ for s in screenshots:f.write(s.image_bytes)```+## Document Complexity++Before committing to a full parse, check whether a document needs OCR or heavier+processing. `is_complex` is a cheap, text-layer-only pass that returns one entry per page+with a `needs_ocr` verdict and the signals behind it — useful for routing documents to+different pipelines, rejecting ones you can't handle, or estimating cost.++```python+parser = LiteParse()+pages = parser.is_complex("document.pdf")++if any(p.needs_ocr for p in pages):+ # Route to the OCR-enabled pipeline+ result = parser.parse("document.pdf")+else:+ # Cheap path — skip OCR entirely+ result = LiteParse(ocr_enabled=False).parse("document.pdf")++# Inspect why specific pages were flagged+for page in pages:+ if page.needs_ocr:+ print(f"Page {page.page_number}: {', '.join(page.reasons)}")+```++`reasons` is one of `"scanned"`, `"no-text"`, `"sparse-text"`, `"embedded-images"`,+`"garbled"`, or `"vector-text"`. Raw bytes work here too.+## Supported Formats- PDF (`.pdf`)@@ -104,4 +131,5 @@ lit parse document.pdflit parse document.pdf --format json -o output.jsonlit screenshot document.pdf -o ./screenshotslit batch-parse ./input ./output+lit is-complex document.pdf```diff --git a/packages/wasm/README.md b/packages/wasm/README.mdindex 3ed4221..14db051 100644--- a/packages/wasm/README.md+++ b/packages/wasm/README.md@@ -31,6 +31,29 @@ console.log(result.text); // full document textconsole.log(result.pages[0]); // per-page items with bboxes```+## Document complexity++Before committing to a full parse, check whether a document needs OCR or heavier+processing. `isComplex` is a cheap, text-layer-only pass that returns one entry per page+with a `needsOcr` verdict and the signals behind it — useful for routing documents or+deciding whether the JS-side OCR engine is worth wiring up.++```ts+const parser = new LiteParse({ ocrEnabled: false });+const bytes = new Uint8Array(await file.arrayBuffer());+const pages = await parser.isComplex(bytes);++if (pages.some((p) => p.needsOcr)) {+ // This document would benefit from OCR — see "OCR in the browser" below+ for (const page of pages.filter((p) => p.needsOcr)) {+ console.log(`Page ${page.pageNumber}: ${page.reasons.join(", ")}`);+ }+}+```++`reasons` is one of `"scanned"`, `"no-text"`, `"sparse-text"`, `"embedded-images"`,+`"garbled"`, or `"vector-text"`.+## Config optionsAll optional, camelCase: