Skip to content

A fast, helpful, and open-source document parser

update to latest versions

Logan Markewich1ce0653cb4ef
diff --git a/README.md b/README.md
index aaaf091..8dd21fe 100644
--- a/README.md
+++ b/README.md
@@ -32,7 +32,7 @@ hard stuff so your models see clean, structured data and markdown.
## Overview
-- **Fast Text Parsing**: Spatial text parsing using PDFium
+- **Fast Text Parsing**: Spatial text parsing using PDFium, ~2-5ms per page
- **Flexible OCR System**:
- **Built-in**: Tesseract (zero setup, bundled with the library)
- **HTTP Servers**: Plug in any OCR server (EasyOCR, PaddleOCR, custom)
@@ -117,6 +117,142 @@ flowchart LR
style CLI fill:#FFBFF8,color:#000000,stroke:#FF8DF2,stroke-width:1px
```
+## Benchmarks
+
+LiteParse is measured on multiple public Doc→Markdown benchmarks. All numbers below were produced on
+this machine from one command (see [Reproducing](#reproducing-the-benchmarks)), with every tool at
+its latest release as of 2026-09-09. LiteParse and the "model-free" competitors use **no ML model at
+all** (no LLM, no layout model, no GPU). pymupdf4llm ≥ 1.28 bundles a layout-detection model
+(`pymupdf-layout`, ONNX, CPU) that is active by default, so it gets two rows: as shipped, and with
+that model switched off.
+
+| Benchmark | Metric | LiteParse | + Tesseract OCR | + PaddleOCR | Best other model-free tool | pymupdf4llm + layout model | Commercial reference |
+|---|---|---|---:|---:|---:|---:|---:|
+| [ParseBench](https://github.com/run-llama/parse-bench) (2,049 docs) | Overall (mean of 5 categories) | 0.364 | 0.380 | **0.389** | pymupdf4llm (model off) 0.309 | 0.541 | — |
+| [opendataloader-bench](https://github.com/opendataloader-project/opendataloader-bench) (200 docs) | Overall (NID + TEDS + MHS) | 0.886 | 0.896 | **0.901** | opendataloader 0.842 | n/a | nutrient 0.885 |
+| [olmOCR-bench](https://github.com/allenai/olmocr/tree/main/olmocr/bench) (1,403 pages) | % tests passed | 39.6 | 41.1 | **42.2** | pdf-inspector 33.7 | 37.3 | — |
+
+LiteParse leads every model-free tool on all three benchmarks. With its layout model on, pymupdf4llm
+leads ParseBench (tables and bounding boxes are what a layout model is for) and trails LiteParse on
+olmOCR-bench; it has no runnable entry in opendataloader-bench.
+
+<details>
+<summary><b>ParseBench</b> — tables, charts, content faithfulness, formatting, visual grounding</summary>
+
+Rule-based scoring, no LLM judge. Each column is [ParseBench's](https://parsebench.ai) canonical per-category metric
+(Tables = GriTS/TRM composite; the others are rule pass-rates). Overall is the mean of the five,
+as on the ParseBench leaderboard.
+
+| Pipeline | Overall | Tables | Charts | Content Faithfulness | Semantic Formatting | Visual Grounding |
+|---|---:|---:|---:|---:|---:|---:|
+| pymupdf4llm 1.28 *(layout model + RapidOCR)* | **0.541** | **0.722** | 0.016 | **0.796** | **0.552** | **0.618** |
+| **LiteParse + PaddleOCR** | 0.389 | 0.430 | 0.013 | 0.787 | 0.402 | 0.314 |
+| **LiteParse + Tesseract** | 0.380 | 0.428 | 0.012 | 0.751 | 0.399 | 0.307 |
+| **LiteParse (no OCR)** | 0.364 | 0.424 | 0.013 | 0.700 | 0.385 | 0.297 |
+| pymupdf4llm 1.28 *(layout model off, no OCR)* | 0.309 | 0.403 | 0.009 | 0.616 | 0.422 | 0.096 |
+| pdf-inspector 1.19 | 0.283 | 0.277 | 0.017 | 0.598 | 0.426 | 0.099 |
+| opendataloader 2.5.7 | 0.277 | 0.349 | 0.006 | 0.663 | 0.258 | 0.110 |
+| markitdown 0.1.7 | 0.185 | 0.158 | **0.020** | 0.652 | 0.001 | 0.110 |
+
+Notes:
+- **Visual Grounding** scores layout blocks with bounding boxes (`lit parse --extract-blocks`).
+ pymupdf4llm's layout model emits them too; the other tools emit no layout data, so their score
+ is only what the reading-order rules award. A page for which a tool emits no layout data counts
+ as 0, so rows with different coverage are comparable.
+- **Charts** is near zero for every tool here — none reconstruct chart data.
+- ParseBench's own pymupdf4llm pipeline turns on its bundled RapidOCR (PaddleOCR models on ONNX
+ Runtime — the same models as the LiteParse + PaddleOCR row), so the two OCR rows are like for like.
+
+</details>
+
+<details>
+<summary><b>opendataloader-bench</b> — reading order, table structure, heading hierarchy</summary>
+
+NID = reading-order similarity, TEDS = table structure, MHS = heading hierarchy. Overall is the
+harness's own mean.
+
+| Engine | Overall | NID | TEDS | MHS |
+|---|---:|---:|---:|---:|
+| **LiteParse + PaddleOCR** | **0.901** | **0.932** | **0.832** | **0.840** |
+| **LiteParse + Tesseract** | 0.896 | 0.928 | 0.829 | 0.828 |
+| **LiteParse (no OCR)** | 0.886 | 0.917 | 0.818 | 0.821 |
+| nutrient *(commercial)* | 0.885 | 0.925 | 0.708 | 0.819 |
+| opendataloader 2.5.7 | 0.842 | 0.912 | 0.483 | 0.757 |
+| pymupdf4llm | 0.732 | 0.885 | 0.401 | 0.412 |
+| markitdown 0.1.7 | 0.589 | 0.844 | 0.273 | 0.000 |
+
+Notes:
+- nutrient and pymupdf4llm have no runnable parser in this harness (commercial / AGPL); their
+ rows are the harness's preserved predictions on the same corpus and ground truth, so the
+ pymupdf4llm row predates its layout model.
+- The corpus is native-text PDFs, so OCR gains come from embedded figures and a handful of
+ scanned pages.
+
+</details>
+
+<details>
+<summary><b>olmOCR-bench</b> — 1,403 single pages, 8,413 unit tests</summary>
+
+Score = average of per-category pass rates. The two math categories require LaTeX output and
+are 0% for every tool here; they still count in the average.
+
+| Engine | Overall | baseline | headers_footers | multi_column | table_tests | long_tiny_text | old_scans | arxiv_math | old_scans_math |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| **LiteParse + PaddleOCR** | **42.2** | **99.9** | 48.7 | **69.1** | 54.0 | **46.4** | **19.4** | 0.0 | 0.0 |
+| **LiteParse + Tesseract** | 41.1 | **99.9** | 52.1 | 66.2 | 54.1 | 42.5 | 13.9 | 0.0 | 0.0 |
+| **LiteParse (no OCR)** | 39.6 | **99.9** | 55.8 | 66.3 | 52.5 | 29.2 | 13.3 | 0.0 | 0.0 |
+| pymupdf4llm 1.28 *(layout model)* | 37.3 | 87.2 | 37.6 | 66.6 | **60.3** | 33.7 | 13.3 | 0.0 | 0.0 |
+| pdf-inspector 1.19 | 33.7 | 82.9 | **62.1** | 49.7 | 43.6 | 17.6 | 13.3 | 0.0 | 0.0 |
+| opendataloader 2.5.7 | 32.5 | 86.9 | 36.6 | 63.7 | 24.9 | 34.8 | 13.3 | 0.0 | 0.0 |
+| pymupdf4llm 1.28 *(layout model off)* | 32.2 | 85.4 | 41.3 | 59.3 | 22.6 | 36.0 | 13.3 | 0.0 | 0.0 |
+| markitdown 0.1.7 | 28.7 | 86.8 | 38.8 | 39.3 | 19.9 | 31.2 | 13.3 | 0.0 | 0.0 |
+
+Notes:
+- **headers_footers** expects letterhead and footer text to be *absent*. OCR recovers that text
+ from logo and address images on single-page documents, where the repeated-header filter cannot
+ fire, so the OCR rows score lower there by design. We chose not to drop text by page position.
+ pdf-inspector 1.19 refuses scanned and image-based pages outright (192 of the 1,403), which is
+ why it tops this category while scoring nothing on those pages elsewhere.
+- **old_scans** is largely cursive handwriting; Tesseract cannot read it, PaddleOCR partially can.
+
+</details>
+
+<details>
+<summary><b>How the runs are configured</b></summary>
+
+- **Same scorer for every tool.** Each benchmark's own evaluator, unmodified. The ground truth in
+ all three corpora is plain text, so every tool including LiteParse runs with hyperlink syntax
+ off (`--no-links`); everything else is the default `lit parse --format markdown`.
+- **OCR modes.** *No OCR* is `--no-ocr`. *Tesseract* is the built-in engine with no setup.
+ *PaddleOCR* is the PP-OCRv5 mobile detector and English recognizer served over the
+ [OCR HTTP API](OCR_API_SPEC.md) by [`ocr/rapidocr`](ocr/rapidocr), which runs the PaddleOCR
+ models on ONNX Runtime — the same models as [`ocr/paddleocr`](ocr/paddleocr), just 20–40× faster
+ per page on CPU-only machines.
+- **Competitors** are the free converters we could run locally, each at its latest release on
+ 2026-09-09: pymupdf4llm 1.28.2, markitdown 0.1.7, opendataloader-pdf 2.5.7 (its non-hybrid mode),
+ pdf-inspector 1.19.0. Each benchmark's own runner for the tool is used as-is. Numbers from public
+ leaderboards or LLM-assisted modes are not mixed in.
+- **pymupdf4llm and its layout model.** Since 1.28, `pip install pymupdf4llm` pulls in
+ `pymupdf-layout`, a ~50 MB ONNX layout-detection model (AGPL) that is activated on import. The
+ "layout model off" rows call `pymupdf4llm.use_layout(False)` and are the model-free comparison;
+ the default rows are what a user gets out of the box.
+- **Machine:** Apple M2 Max, 12 cores, 32 GB. LiteParse rows: v2.14.4 (2026-09-09).
+
+</details>
+
+### Reproducing the benchmarks
+
+```sh
+./run_benchmarks.sh --liteparse-only # all three benches, no OCR
+./run_benchmarks.sh --liteparse-only --ocr=tesseract
+( cd ocr/rapidocr && uv run server.py ) & # then:
+./run_benchmarks.sh --liteparse-only --ocr=paddle
+./run_benchmarks.sh --competitors-only # re-run the free competitors only
+./run_benchmarks.sh # everything
+```
+
+Results and a `SUMMARY.md` land in `bench_results/latest/`.
+
## Installation
Install via your preferred package manager. All versions (except WASM) ship with the same `lit` CLI.