Skip to content

A fast, helpful, and open-source document parser

complexity docs

Logan Markewicha97abfcf81bc
diff --git a/README.md b/README.md
index 03f86bb..a6a721a 100644
--- a/README.md
+++ b/README.md
@@ -37,6 +37,7 @@ hard stuff so your models see clean, structured data and markdown.
- **Built-in**: Tesseract (zero setup, bundled with the library)
- **HTTP Servers**: Plug in any OCR server (EasyOCR, PaddleOCR, custom)
- **Standard API**: Simple, well-defined OCR API specification
+- **Complexity Detection**: Cheaply check whether a document needs OCR or heavier parsing — route, reject, or estimate cost before a full parse
- **Screenshot Generation**: Generate high-quality page screenshots for LLM agents
- **Multiple Output Formats**: Markdown, JSON, and Text
- **Markdown Output**: Structured Markdown with headings, tables, lists, images, and links — great for feeding LLMs and RAG pipelines
@@ -196,6 +197,28 @@ Image handling is controlled by `--image-mode`:
> [LlamaParse](https://developers.llamaindex.ai/python/cloud/llamaparse/?utm_source=github&utm_medium=liteparse)
> remains the most accurate option.
+### Check Complexity
+
+Before committing to a full parse, check whether a document actually needs OCR or
+heavier processing. This is a cheap, text-layer-only pass — useful for routing
+documents to different pipelines, rejecting ones you can't handle, or estimating cost.
+
+```bash
+# Print the complexity verdict and per-page JSON
+lit is-complex document.pdf
+
+# Use as a shell predicate — only parse with --no-ocr when the document is simple
+lit is-complex document.pdf --quiet && lit parse document.pdf --no-ocr
+
+# List the pages that need OCR
+lit is-complex document.pdf --compact | jq '[.[] | select(.needs_ocr) | .page_number]'
+```
+
+It always prints per-page JSON to **stdout**, a human-readable verdict to **stderr**, and
+exits **non-zero when any page needs OCR**. Each page carries a `needs_ocr` verdict and a
+list of `reasons` (`scanned`, `no-text`, `sparse-text`, `embedded-images`, `garbled`,
+`vector-text`).
+
### Batch Parsing
Parse an entire directory of documents:
@@ -281,6 +304,23 @@ Options:
-h, --help Print help
```
+#### Is-Complex Command
+
+```
+lit is-complex [OPTIONS] <file>
+
+Options:
+ --compact Emit dense, whitespace-free JSON instead of pretty-printed
+ --max-pages <n> Max pages to check [default: 1000]
+ --target-pages <pages> Pages to check (e.g., "1-5,10,15-20")
+ --password <password> Password for encrypted documents
+ -q, --quiet Suppress the stderr verdict
+ -h, --help Print help
+```
+
+Prints per-page JSON to stdout and a `COMPLEX`/`SIMPLE` verdict to stderr; exits non-zero
+when any page needs OCR, so it composes as a shell predicate.
+
## OCR Setup
### Default: Tesseract
diff --git a/crates/liteparse/README.md b/crates/liteparse/README.md
index 342fa71..0cfd892 100644
--- a/crates/liteparse/README.md
+++ b/crates/liteparse/README.md
@@ -102,6 +102,32 @@ let result = parser.parse_input(PdfInput::Bytes(pdf_bytes)).await?;
println!("{}", result.text);
```
+## Document Complexity
+
+Before committing to a full parse, check whether a document needs OCR or heavier
+processing. `is_complex` is a cheap, text-layer-only pass that returns a
+`PageComplexityStats` per page with a `needs_ocr` verdict and the signals behind it —
+useful for routing documents to different pipelines, rejecting ones you can't handle, or
+estimating cost.
+
+```rust
+use liteparse::types::PdfInput;
+
+let parser = LiteParse::new(LiteParseConfig::default());
+let pages = parser.is_complex(PdfInput::Path("document.pdf".into())).await?;
+
+if pages.iter().any(|p| p.needs_ocr) {
+ // Route to the OCR-enabled pipeline, inspect `p.reasons`, etc.
+ for page in pages.iter().filter(|p| p.needs_ocr) {
+ println!("Page {} needs OCR: {:?}", page.page_number, page.reasons);
+ }
+}
+```
+
+`reasons` is a `Vec<ComplexityReason>` (`Scanned`, `NoText`, `SparseText`,
+`EmbeddedImages`, `Garbled`, `VectorText`); new variants may be added over time, so match
+leniently.
+
## Custom OCR Engine
Implement the `OcrEngine` trait to plug in your own OCR backend:
@@ -135,6 +161,7 @@ lit parse document.pdf --format json -o output.json
lit parse document.pdf --format markdown -o output.md
lit screenshot document.pdf -o ./screenshots
lit batch-parse ./input ./output
+lit is-complex document.pdf
```
See `lit --help` for all options.
diff --git a/docs/src/content/docs/liteparse/cli-reference.md b/docs/src/content/docs/liteparse/cli-reference.md
index fc9182f..6649207 100644
--- a/docs/src/content/docs/liteparse/cli-reference.md
+++ b/docs/src/content/docs/liteparse/cli-reference.md
@@ -5,7 +5,7 @@ sidebar:
order: 5
---
-LiteParse provides the `lit` CLI with three commands: `parse`, `batch-parse`, and `screenshot`. The CLI is the same whether installed via `npm`, `pip`, or built from Rust source.
+LiteParse provides the `lit` CLI with four commands: `parse`, `batch-parse`, `screenshot`, and `is-complex`. The CLI is the same whether installed via `npm`, `pip`, or built from Rust source.
## `lit parse`
@@ -164,6 +164,47 @@ lit screenshot document.pdf --target-pages "1,5,10" -o ./pages
---
+## `lit is-complex`
+
+Check whether a document needs OCR or heavier parsing — a cheap pre-parse pass over the text layer only (no rasterization, no OCR). See the [Document Complexity guide](/liteparse/guides/complexity/) for details.
+
+```
+lit is-complex [options] <file>
+```
+
+The command prints per-page JSON to **stdout**, a human-readable verdict to **stderr**, and exits **non-zero when any page needs OCR** — so it works as a shell predicate or a `jq` source.
+
+### Arguments
+
+| Argument | Description |
+|----------|-------------|
+| `file` | Path to the document file |
+
+### Options
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `--compact` | Emit dense, whitespace-free JSON instead of pretty-printed | — |
+| `--max-pages <n>` | Maximum pages to check | `1000` |
+| `--target-pages <pages>` | Pages to check (e.g., `"1-5,10,15-20"`) | — (all pages) |
+| `--password <password>` | Password for encrypted/protected documents | — |
+| `-q, --quiet` | Suppress the stderr verdict | — |
+
+### Examples
+
+```bash
+# Print the complexity verdict and per-page JSON
+lit is-complex document.pdf
+
+# Use as a shell predicate: only parse with --no-ocr when simple
+lit is-complex document.pdf --quiet && lit parse document.pdf --no-ocr
+
+# List the page numbers that need OCR
+lit is-complex document.pdf --compact | jq '[.[] | select(.needs_ocr) | .page_number]'
+```
+
+---
+
## Global options
These options are available on all commands:
diff --git a/docs/src/content/docs/liteparse/getting_started.mdx b/docs/src/content/docs/liteparse/getting_started.mdx
index ba9ea5b..74c5ec1 100644
--- a/docs/src/content/docs/liteparse/getting_started.mdx
+++ b/docs/src/content/docs/liteparse/getting_started.mdx
@@ -87,6 +87,7 @@ lit screenshot document.pdf -o ./screenshots
- [Library usage](/liteparse/guides/library-usage/): Use LiteParse programmatically from TypeScript or Python.
- [OCR configuration](/liteparse/guides/ocr/): Configure Tesseract, use an external OCR server, or bring your own.
- [Multi-format support](/liteparse/guides/multi-format/): Parse DOCX, XLSX, PPTX, images, and more.
+- [Document complexity](/liteparse/guides/complexity/): Check whether a document needs OCR / heavy processing before parsing, and route accordingly.
- [Browser usage (WASM)](/liteparse/guides/browser-usage/): Run LiteParse entirely in the browser.
- [Agent skill](/liteparse/guides/agent-skill/): Add LiteParse as a skill for coding agents.
- [CLI reference](/liteparse/cli-reference/): Complete command and option reference.
diff --git a/docs/src/content/docs/liteparse/guides/complexity.mdx b/docs/src/content/docs/liteparse/guides/complexity.mdx
new file mode 100644
index 0000000..1e6d345
--- /dev/null
+++ b/docs/src/content/docs/liteparse/guides/complexity.mdx
@@ -0,0 +1,181 @@
+---
+title: Document Complexity
+description: Use is_complex to decide whether a document needs OCR or heavier parsing before you commit to it.
+sidebar:
+ order: 9
+---
+
+The `is_complex` command in LiteParse checks whether a document is "complex". This means looking for images, broken/garbled text, large amounts of vector graphics, and sparse text.
+
+Use it to:
+
+- **Route documents** to cheaper or more expensive parsing/OCR backends.
+- **Reject or flag** documents you can't handle — e.g. when running with `--no-ocr`, find the pages that would come back empty.
+- **Estimate cost** up front by counting how many pages actually need OCR.
+
+## How it works
+
+Complexity is computed **per page**. Each page gets a `needs_ocr` verdict and a list of `reasons` explaining why it was flagged. A document is "complex" if any of its pages needs OCR.
+
+The reasons are derived from a few signals:
+
+| Reason | Meaning |
+|--------|---------|
+| `scanned` | A single raster covers ~the whole page with little or no text behind it — a scanned/photographed page. |
+| `no-text` | Almost no extractable text and no full-page image — a blank page, cover, or divider. |
+| `sparse-text` | Some real text, but it covers very little of the page — typically a figure with a thin caption. |
+| `embedded-images` | Substantial embedded raster figures sit alongside the native text. |
+| `garbled` | The native text decodes to garbage and text is likely unreadable. |
+| `vector-text` | Text is painted as filled vector outlines outside the text layer, so no native text items represent it. |
+
+> The set of reasons may grow over time as the router learns to recommend heavier pipelines. Treat `reasons` as an open-ended list and route on the values you care about rather than assuming it's exhaustive.
+
+## CLI
+
+```bash
+lit is-complex document.pdf
+```
+
+The command always prints per-page JSON to **stdout**, a human-readable verdict to **stderr**, and sets its **exit code** to reflect the result — so you can consume it however fits your workflow.
+
+```jsonc
+// stdout
+[
+ {
+ "page_number": 1,
+ "text_length": 0,
+ "text_coverage": 0.0,
+ "has_substantial_images": false,
+ "image_block_count": 0,
+ "image_coverage": 0.0,
+ "largest_image_coverage": 0.0,
+ "full_page_image": true,
+ "uncovered_vector_area": null,
+ "is_garbled": false,
+ "page_area": 482400.0,
+ "needs_ocr": true,
+ "reasons": ["scanned"]
+ }
+]
+```
+
+```
+# stderr
+COMPLEX — 1/1 page(s) need OCR
+```
+
+The exit code is **non-zero when any page needs OCR**, so the command works as a shell predicate:
+
+```bash
+# Only parse with --no-ocr when the document is simple
+lit is-complex document.pdf --quiet && lit parse document.pdf --no-ocr
+```
+
+Pipe the JSON into `jq` to act on individual pages. The example below will list the page numbers that need OCR:
+
+```bash
+lit is-complex document.pdf --compact | jq '[.[] | select(.needs_ocr) | .page_number]'
+```
+
+### Options
+
+| Flag | Description |
+|------|-------------|
+| `--compact` | Emit dense, whitespace-free JSON instead of pretty-printed. |
+| `--max-pages <N>` | Maximum number of pages to check (default: 1000). |
+| `--target-pages <SPEC>` | Check only specific pages, e.g. `"1-5,10,15-20"`. |
+| `--password <PASS>` | Password for encrypted/protected documents. |
+| `-q`, `--quiet` | Suppress the stderr logging output. |
+
+## Library
+
+The same check is available programmatically. It returns one entry per page.
+
+<Tabs>
+<TabItem label="TypeScript" value="typescript">
+
+```typescript
+import { LiteParse } from "@llamaindex/liteparse";
+
+const parser = new LiteParse({ ocrEnabled: false });
+const pages = await parser.isComplex("document.pdf");
+
+const complex = pages.some((p) => p.needsOcr);
+if (complex) {
+ // Route to a heavweight pipeline, like LlamaParse
+ ...
+} else {
+ // Cheap path — skip OCR entirely
+ const result = await parser.parse("document.pdf");
+}
+
+// Inspect why specific pages were flagged
+for (const page of pages.filter((p) => p.needsOcr)) {
+ console.log(`Page ${page.pageNumber}: ${page.reasons.join(", ")}`);
+}
+```
+
+</TabItem>
+<TabItem label="Python" value="python">
+
+```python
+from liteparse import LiteParse
+
+parser = LiteParse(ocr_enabled=False)
+pages = parser.is_complex("document.pdf")
+
+if any(p.needs_ocr for p in pages):
+ # Route to a heavweight pipeline, like LlamaParse
+ ...
+else:
+ # Cheap path — skip OCR entirely
+ result = parser.parse("document.pdf")
+
+# Inspect why specific pages were flagged
+for page in pages:
+ if page.needs_ocr:
+ print(f"Page {page.page_number}: {', '.join(page.reasons)}")
+```
+
+</TabItem>
+<TabItem label="Browser (WASM)" value="wasm">
+
+```javascript
+import { LiteParse } from "@llamaindex/liteparse-wasm";
+
+const parser = new LiteParse();
+const pages = await parser.isComplex(pdfBytes); // Uint8Array
+
+const complex = pages.some((p) => p.needsOcr);
+```
+
+</TabItem>
+</Tabs>
+
+Both the library and CLI accept raw bytes as well as file paths, so you can run the check on documents you've already loaded into memory.
+
+## Per-page fields
+
+Every entry includes the raw signals behind the verdict, so you can apply your own thresholds instead of relying solely on `needs_ocr`:
+
+| Field | Description |
+|-------|-------------|
+| `page_number` | 1-indexed page number. |
+| `needs_ocr` | Verdict: the page needs more than the cheap text-only path. Equivalent to `reasons` being non-empty. |
+| `reasons` | Every reason the page was flagged (see the table above). Empty exactly when `needs_ocr` is false. |
+| `text_length` | Length of usable native text (garbled/unmappable text excluded). |
+| `text_coverage` | Fraction of the page area covered by native text (0–1). |
+| `has_substantial_images` | Whether any counted inline raster figures are present. |
+| `image_block_count` | Number of counted raster image objects (full-page backgrounds excluded). |
+| `image_coverage` | Summed image-bbox area over page area, clamped to 1. |
+| `largest_image_coverage` | Largest single counted image's area over page area, clamped to 1. |
+| `full_page_image` | A single raster covers ≥90% of the page — the signal that tells a scan apart from a blank page. |
+| `uncovered_vector_area` | Filled vector-outline area not covered by native text (pt²). `null`/`undefined` when a cheaper signal already decided the page. |
+| `is_garbled` | Whether the native text decodes to garbage. |
+| `page_area` | Page area in pt². |
+
+## Next steps
+
+- [OCR configuration](/liteparse/guides/ocr/): Set up the OCR backend you route complex documents to.
+- [Library usage](/liteparse/guides/library-usage/): Full programmatic API for TypeScript and Python.
+- [CLI reference](/liteparse/cli-reference/): Complete command and option reference.
diff --git a/packages/node/README.md b/packages/node/README.md
index 271e9c6..c30a02f 100644
--- a/packages/node/README.md
+++ b/packages/node/README.md
@@ -90,6 +90,34 @@ for (const s of screenshots) {
}
```
+## Document Complexity
+
+Before committing to a full parse, check whether a document needs OCR or heavier
+processing. `isComplex` is a cheap, text-layer-only pass that returns one entry per page
+with a `needsOcr` verdict and the signals behind it — useful for routing documents to
+different pipelines, rejecting ones you can't handle, or estimating cost.
+
+```typescript
+const parser = new LiteParse();
+const pages = await parser.isComplex('document.pdf');
+
+if (pages.some((p) => p.needsOcr)) {
+ // Route to the OCR-enabled pipeline
+ const result = await parser.parse('document.pdf');
+} else {
+ // Cheap path — skip OCR entirely
+ const result = await new LiteParse({ ocrEnabled: false }).parse('document.pdf');
+}
+
+// Inspect why specific pages were flagged
+for (const page of pages.filter((p) => p.needsOcr)) {
+ console.log(`Page ${page.pageNumber}: ${page.reasons.join(', ')}`);
+}
+```
+
+`reasons` is one of `"scanned"`, `"no-text"`, `"sparse-text"`, `"embedded-images"`,
+`"garbled"`, or `"vector-text"`. Raw bytes work here too.
+
## Supported Formats
- PDF (`.pdf`)
@@ -107,4 +135,5 @@ lit parse document.pdf
lit parse document.pdf --format json -o output.json
lit screenshot document.pdf -o ./screenshots
lit batch-parse ./input ./output
+lit is-complex document.pdf
```
diff --git a/packages/python/README.md b/packages/python/README.md
index c938cee..a624501 100644
--- a/packages/python/README.md
+++ b/packages/python/README.md
@@ -87,6 +87,33 @@ for s in screenshots:
f.write(s.image_bytes)
```
+## Document Complexity
+
+Before committing to a full parse, check whether a document needs OCR or heavier
+processing. `is_complex` is a cheap, text-layer-only pass that returns one entry per page
+with a `needs_ocr` verdict and the signals behind it — useful for routing documents to
+different pipelines, rejecting ones you can't handle, or estimating cost.
+
+```python
+parser = LiteParse()
+pages = parser.is_complex("document.pdf")
+
+if any(p.needs_ocr for p in pages):
+ # Route to the OCR-enabled pipeline
+ result = parser.parse("document.pdf")
+else:
+ # Cheap path — skip OCR entirely
+ result = LiteParse(ocr_enabled=False).parse("document.pdf")
+
+# Inspect why specific pages were flagged
+for page in pages:
+ if page.needs_ocr:
+ print(f"Page {page.page_number}: {', '.join(page.reasons)}")
+```
+
+`reasons` is one of `"scanned"`, `"no-text"`, `"sparse-text"`, `"embedded-images"`,
+`"garbled"`, or `"vector-text"`. Raw bytes work here too.
+
## Supported Formats
- PDF (`.pdf`)
@@ -104,4 +131,5 @@ lit parse document.pdf
lit parse document.pdf --format json -o output.json
lit screenshot document.pdf -o ./screenshots
lit batch-parse ./input ./output
+lit is-complex document.pdf
```
diff --git a/packages/wasm/README.md b/packages/wasm/README.md
index 3ed4221..14db051 100644
--- a/packages/wasm/README.md
+++ b/packages/wasm/README.md
@@ -31,6 +31,29 @@ console.log(result.text); // full document text
console.log(result.pages[0]); // per-page items with bboxes
```
+## Document complexity
+
+Before committing to a full parse, check whether a document needs OCR or heavier
+processing. `isComplex` is a cheap, text-layer-only pass that returns one entry per page
+with a `needsOcr` verdict and the signals behind it — useful for routing documents or
+deciding whether the JS-side OCR engine is worth wiring up.
+
+```ts
+const parser = new LiteParse({ ocrEnabled: false });
+const bytes = new Uint8Array(await file.arrayBuffer());
+const pages = await parser.isComplex(bytes);
+
+if (pages.some((p) => p.needsOcr)) {
+ // This document would benefit from OCR — see "OCR in the browser" below
+ for (const page of pages.filter((p) => p.needsOcr)) {
+ console.log(`Page ${page.pageNumber}: ${page.reasons.join(", ")}`);
+ }
+}
+```
+
+`reasons` is one of `"scanned"`, `"no-text"`, `"sparse-text"`, `"embedded-images"`,
+`"garbled"`, or `"vector-text"`.
+
## Config options
All optional, camelCase: