Skip to content

A fast, helpful, and open-source document parser

3.8 KB
Raw Blame

LiteParse Eval Utils

Utilities for generating and evaluating datasets for PDF parsing performance. Compares text extraction quality across multiple PDF parsers using LLM-based QA evaluation.

Setup

Requires Python 3.12+.

# Install the package (from the dataset_eval_utils directory)
pip install -e .

You'll need an ANTHROPIC_API_KEY environment variable set for the LLM-based evaluation and dataset processing tools.

Dataset

An existing dataset that was generated and evaluated using this framework can be found on huggingface.

You can download the dataset using the Hugging Face CLI:

hf download run-llama/liteparse-eval-dataset --repo-type dataset --local-dir ./liteparse-eval-dataset

CLI Tools

lp-process — Generate Ground Truth Datasets

Processes PDF and image files using Claude's vision capabilities to generate structured QA ground truth data.

lp-process /path/to/documents --output-dir ./ground_truth

Options:

  • --output-dir — Directory to save output JSON files (default: ./output)
  • --model — Claude model to use (default: claude-sonnet-4-5-20250929)
  • --api-key — Anthropic API key (or set ANTHROPIC_API_KEY env var)

Each output JSON file contains document metadata and QA pairs extracted from the document pages.

lp-evaluate — Run QA Evaluation

Evaluates parser text extraction quality by having an LLM answer questions from extracted text and judging correctness against ground truth answers.

lp-evaluate \
  --data-dir ./documents \
  --ground-truth-dir ./ground_truth \
  --parse-provider liteparse \
  --output ./results/run1

Options:

  • --data-dir — Directory containing source PDF documents (required)
  • --ground-truth-dir — Directory containing ground truth JSON files (required)
  • --output — Path to save results (JSON + HTML report)
  • --parse-provider — Parser to evaluate: liteparse, pymupdf, pypdf, markitdown (default: liteparse)
  • --llm-provider — LLM for answering questions: anthropic (default: anthropic)

Outputs:

  • <output>.json — Aggregate results with pass rates
  • <output>_detailed.json — Per-document results with extracted text and individual QA results
  • <output>_report.html — Interactive HTML report with PDF previews and QA breakdowns

lp-benchmark — Performance Benchmarking

Measures parse latency and memory usage across providers.

lp-benchmark ./documents --providers pymupdf liteparse --warmup-runs 0

Takes a directory of PDFs (searched recursively), not a single file.

Options:

  • --providers — Providers to benchmark (default: all local providers)
  • --warmup-runs — Full passes over the corpus before timing starts (default: 5). Each warmup run parses every document, so on a large corpus this dominates wall time; use 0 for a quick measurement.
  • --output — Path to save JSON results

Reports per-document latency plus TOTAL / AVG-per-doc / MS-per-page rows.

Parser Providers

Provider Library Notes
liteparse liteparse Spatial text extraction with OCR support
pymupdf PyMuPDF Fast, mature PDF library
pypdf pypdf Pure-Python PDF library
markitdown MarkItDown Microsoft's document-to-markdown converter

Evaluation Pipeline

  1. Extract text from PDF using the selected parser provider
  2. Answer questions — LLM reads the extracted text and answers ground truth questions
  3. Judge answers — A separate LLM judge evaluates whether predicted answers are semantically equivalent to expected answers
  4. Aggregate — Pass rates are computed per-document and overall