Skip to content

A fast, helpful, and open-source document parser

wip

Logan Markewich4dc2a9464263
diff --git a/MARKDOWN_PROGRESS.md b/MARKDOWN_PROGRESS.md
index 844fdd8..bc821d1 100644
--- a/MARKDOWN_PROGRESS.md
+++ b/MARKDOWN_PROGRESS.md
@@ -2,6 +2,44 @@
Tracks build-order progress against [MARKDOWN_PLAN.md](MARKDOWN_PLAN.md). Update as steps complete.
+> **2026-06-11 launch-readiness quality audit (research only, no code changes).** Competitor-independent audit of absolute output quality, validated on real current output via a 5-way parallel investigation (word fusion, reading order, dropped/duplicated content, math, structural/inline formatting). Data sources: opendataloader 200 paired prediction/ground-truth markdown (`opendataloader-bench/{prediction/liteparse,ground-truth}/markdown/<id>.md`), olmOCR category outputs + predicate JSONLs, ParseBench formatting CSVs. Reusable triage script written: `dataset_eval_utils/order_defect_triage.py`.
+>
+> **Headline reframe — reading order is less broken than the metric implies.** The ~52.8% olmOCR order-pass rate overstates it. Of 320 failing multi_column order tests: ~84 are metric noise (both strings present *in correct order*, fail only on strict char-match from a dropped space — free wins via tolerance, not defects), ~61 are extraction/OCR drops (orthogonal), only **~8 are true column inversions**, and **~150 are the real defect: caption/sidebar text spliced into body prose** (floating captions / narrow side-blocks not isolated before column ordering — e.g. `multi_column/0b045ab8…page_18` prints caption #2 → body → caption #1 interleaved; `02943986…page_7`, `0904c7071…page_4`, `0925342e…page_4`, `0145bc47…page_11`). Fix target is sharp: caption/sidebar isolation in the XY-cut/anchor pass, NOT a reading-order rewrite.
+>
+> **Validated defect map, ranked by (impact × fixability):**
+>
+> *Tier 1 — fix before launch:*
+> 1. **Caption/sidebar splicing** (~150 order tests). The real reading-order defect; most damaging for ingestion, well-scoped.
+> 2. **Word fusion in dense/small-font text.** Near-zero in normal prose (opendataloader distinctive-token recall 0.98, ~1 confirmed fusion in 55k words), but **22.6% of olmOCR `long_tiny_text` files affected** — and lowercase→lowercase fusion (`theynexttakeonthe`, `againstManchester`, `Cityand`, `forSpurs` in `long_tiny_text/20_pg58…`) is far worse than the camelCase tip a regex catches. Confirmed `ofTurkey`→"of Turkey" (opendataloader 01030000000012, GT-verified). Concentrated, not pervasive; matches the `project_missing_spaces` adaptive-gap-threshold need. Trickiest fix (must stay table/column-safe).
+> 3. **Table label-column drop** — the *only* real content-loss defect (all other losses are legitimately OCR-gated, expected under `--no-ocr`; median word-count ratio 0.96, no duplication). Numeric columns render but the leading label column drops/garbles: docs `01030000000082` (state names Gujarat/Punjab/…), `01030000000046` (party names Funcinpec/Candlelight/…), `121`/`122` (in-table prose → `. a . a tip` fragments). Root cause: table cell text reconstruction in the projection layer. Small but real correctness miss.
+>
+> *Tier 2 — cheap wins:*
+> 4. **Code-block language tag.** Code blocks actually *work* (real Python correctly fenced on `text_misc__justcode.pdf`) but score 0.0 on ParseBench because we emit bare ` ``` ` with no language tag (`rules_formatting.py:583` requires `lang=="python"`), and snippets fragment across multiple fences. Near-trivial unblock. (NB: the ~273 fenced output files are mostly GridFallback ` ```text `, not code.)
+> 5. **Links entirely absent** — no `GetURI`/annotation extraction exists in `extract.rs` or the pdfium wrapper at all. PDFium exposes it; genuinely missing feature, moderate effort.
+>
+> *Tier 3 — document as known v1 limitations (large effort, not launch blockers):*
+> 6. **Math notation structurally absent.** Glyphs/Greek/operators (α 841×, ≤ 1016×, ∈ 2239×) decode fine, but sub/superscript positioning is flattened (**0/522 arxiv outputs** contain any super/sub char or `\frac`/`\begin`), display equations dumped inline with no `$$` delimiters, degree symbol dropped (`95°C`→`958C`). Affects *all* scientific/technical docs, not just arxiv. The #1 sub-gap is sub/superscript positioning, then equation delimiting — symbol decoding is mostly fine. Scope explicitly as out-of-scope for v1 rather than half-do.
+> 7. **Bold/italic whole-line-only** (ParseBench bold 0.53 / italic 0.38 — emitted but rarely fine-grained spans; occasionally over-applied/malformed) and **heading-level accuracy weak** (title_hierarchy 0.37 — wrong levels, not mere presence). Lists work but flat (nesting untested — no nested list in either corpus). Fidelity tier, lower ingestion impact.
+>
+> **Recommended order when resuming:** #1 caption/sidebar splicing first (biggest "looks scrambled" win, well-scoped — start from `dataset_eval_utils/order_defect_triage.py` and the ~150 SHREDDED_INTERLEAVE docs to find the geometric signal separating a floating caption from body text); bundle #3 + #4 as small high-confidence correctness fixes; #2 dense-text word fusion is highest-value-invisible but budget for iteration; math (#6) document as a known limitation.
+>
+> **2026-06-11 global ruled-table pass — opendataloader Overall 0.8694 → 0.8710 (TEDS 0.6678 → 0.6877 (+0.0199), NID 0.9115 → 0.9120, MHS 0.8032 → 0.8036); ParseBench table grits_con 0.5507 → 0.5647, TRM 0.2123 → 0.2213, composite 0.3922 → 0.4040; olmOCR table_tests 46.7 → 47.3% (+6), Overall + multi_column flat at 38.4% / 63.8%.** Clean win, zero regressions. Crosses two competitor lines: **ParseBench TRM 0.2213 now beats pymupdf4llm 0.2205** (the last remaining table deficit), and the opendataloader TEDS gap to nutrient halves (−0.040 → −0.020, 0.6877 vs 0.708).
+>
+> **Root cause.** `detect_ruled_tables` runs per xy_cut leaf. A ruled table whose rows scatter across ≥2 leaves (a V-cut sliced it, or a banner/caption peeled a level) never has its full text in any single leaf, so per-region detection rejects it as mostly-empty and the grid is lost entirely. (Distinct from the cross-region pass, which re-fuses *borderless* slices via aligned baselines — this is the *ruled*-grid analog, keyed off the stroke geometry directly.)
+>
+> **New `detect_ruled_tables_global`** (`tables.rs`), run once over all filtered lines in `classify_page_with_filters` *before* cross-region merge. `build_ruled_table` gained a `global: bool` and now returns the consumed line indices so the caller can pull them out of the region pipeline and emit each table as a y-positioned `Interruption::Table`. Global-only refinements inside `build_ruled_table`:
+> 1. **`RULED_HLINE_MIN_COVERAGE = 0.5`** — drop horizontal rules spanning <50% of the column extent before deriving row boundaries. A full-height left/right page border unions a stray box (logo, sidebar) into the grid component through the shared vertical; that box's short rules would otherwise become phantom top rows that vacuum surrounding prose. Falls back to all rules if the filter leaves <3.
+> 2. **y-sorted line + span iteration** — the global pass receives lines in xy_cut leaf order (column-by-column), which scrambles multi-line-cell text on array-order concatenation; sorting by `bbox.y` (and spans by `x`) restores reading order within each cell. No-op for the already-y-ordered per-region path.
+>
+> **Acceptance gates in `classify.rs`** (each tuned against a measured misfire):
+> - **≥2-leaf scatter required.** Only rescue tables the per-region path *can't* see whole — those whose consumed lines span ≥2 distinct `region_path`s. A table living in one leaf is already handled (often better) there.
+> - **`already_handled` skip.** If any single leaf's share of the consumed lines already forms a table on its own (`detect_ruled_tables` or `detect_tables` non-empty), don't override — doc 127's decorative frame spans the data table *plus* surrounding prose; the data region tables cleanly by itself, the frame would fuse prose into garbage.
+> - **1-column gate** (the load-bearing false-positive guard). A decorative banner/frame around a single prose column (left+right border, no interior verticals) unions into a 1-col "grid" whose lone cell is the whole paragraph block. A real data table always has ≥2 columns. Caught docs **028649** (2-col prose + "THE PLAN" banner fused into one 250-word cell) and **06ba2a90** (journal page + "Table II" caption box). **Word-count gating was tried first and reverted** — it killed *all* the opendataloader wins too: the genuine scattered tables this pass rescues (competence/legal description grids) have cells just as long as the fused-prose false positives, so cell length can't separate them; column count can. With the 1-col gate, opendataloader/ParseBench wins are byte-identical to the ungated run and olmOCR multi_column fully recovers (63.3 ungated → 63.8). Two changed multi_column docs remain (081875 3-col, 0d7383 2-col) — the documented Mode-5 ambiguous slide-feature-table family; they emit as tables but cost no order tests.
+>
+> Kill-switch `LITEPARSE_DISABLE_GLOBAL_RULED=1`. 190 lib tests pass.
+>
+> **Triage note:** A/B the multi_column dip by converting the disabled baseline to a named candidate (`liteparse:name=liteparse_noglobal`) and `diff`ing the per-doc `.md` — isolated exactly 4 changed docs out of 884, which is what surfaced the 1-col-vs-≥2-col split. Opendataloader A/B via the kill-switch on the same binary (no rebuild needed) confirmed the col-gate is byte-identical to the ungated win.
+>
> **2026-06-10 (pm) tight-gutter column rescue + XY depth/fill-gate session — opendataloader Overall 0.8633 → 0.8694 (NID 0.9071 → 0.9115, MHS 0.7865 → 0.8032, TEDS 0.6716 → 0.6678); olmOCR Overall 37.7 → 38.4% (multi_column 60.0 → 63.8%, table_tests 45.5 → 46.7% — now ahead of pymupdf4llm's 46.2); ParseBench table grits_con 0.5481 → 0.5507, TRM 0.2083 → 0.2123, composite 0.3886 → 0.3922.** Targeted the docs 38/189 "bold heading fused inline across an unsplit 2-col layout" family — the head of the nutrient MHS/NID gap (38/189 alone were MHS −0.97/−0.94 vs nutrient).
>
> Five changes, all `projection.rs::xy_find_column_cut` + recursion, each motivated by a measured geometry:
diff --git a/crates/liteparse/src/markdown_layout/classify.rs b/crates/liteparse/src/markdown_layout/classify.rs
index fe3857e..fc97ff4 100644
--- a/crates/liteparse/src/markdown_layout/classify.rs
+++ b/crates/liteparse/src/markdown_layout/classify.rs
@@ -26,6 +26,10 @@ use super::tables::{detect_ruled_tables, detect_tables, merge_table_runs};
enum Interruption {
Hr,
Figure(crate::types::ImageRef),
+ /// A ruled table detected by the global pass, already fully built. Emitted
+ /// at its top-y position like other interruptions; its lines were pulled
+ /// out of the region pipeline so it won't be re-detected per-region.
+ Table(super::blocks::Block),
}
/// Returns true if any span on the line is rotated more than ~5° off
@@ -145,6 +149,99 @@ pub fn classify_page_with_filters(
&filtered_owned
};
+ // Global ruled-table pass. `detect_ruled_tables` is otherwise called per
+ // xy_cut leaf, but a ruled table whose rows scatter across several leaves
+ // never has its full text in any single leaf and gets rejected as
+ // mostly-empty. Run it once over all lines, pull the consumed lines out of
+ // the region pipeline, and emit each table as a y-positioned interruption.
+ // Runs before cross-region merge so that pass (and the region indices its
+ // runs carry) operate on the already-filtered line list. Disable with
+ // LITEPARSE_DISABLE_GLOBAL_RULED=1.
+ let mut global_ruled_tables: Vec<(f32, Block)> = Vec::new();
+ let mut global_ruled_consumed: std::collections::HashSet<usize> =
+ std::collections::HashSet::new();
+ if std::env::var("LITEPARSE_DISABLE_GLOBAL_RULED").is_err() {
+ for (run, consumed) in super::tables::detect_ruled_tables_global(
+ lines,
+ &page.graphics,
+ page.page_width,
+ page.page_height,
+ ) {
+ // Only rescue tables the per-region path can't see whole: those
+ // whose rows scatter across ≥2 xy_cut leaves. A table living in a
+ // single region is already handled (often better) by per-region
+ // detection — replacing it here only risks regressions.
+ let mut groups: std::collections::HashMap<&Vec<u16>, Vec<usize>> =
+ std::collections::HashMap::new();
+ for &i in &consumed {
+ groups.entry(&lines[i].region_path).or_default().push(i);
+ }
+ if groups.len() < 2 {
+ continue;
+ }
+ // If any single leaf's share of these lines already forms a table
+ // on its own, the per-region path handles this content — don't
+ // override (e.g. doc 127: a decorative frame spans the data table
+ // plus surrounding prose; the data region tables cleanly by
+ // itself, the frame would fuse prose into garbage cells).
+ let already_handled = groups.values().any(|idxs| {
+ if idxs.len() < 2 {
+ return false;
+ }
+ let sub: Vec<ProjectedLine> = idxs.iter().map(|&i| lines[i].clone()).collect();
+ !super::tables::detect_ruled_tables(
+ &sub,
+ &page.graphics,
+ page.page_width,
+ page.page_height,
+ )
+ .is_empty()
+ || !super::tables::detect_tables(&sub).is_empty()
+ });
+ if already_handled {
+ continue;
+ }
+ // Single-column gate. A decorative banner/frame around a prose
+ // column (just a left+right border, no interior verticals) unions
+ // into a 1-column "grid" whose lone cell is the whole paragraph
+ // block. A real data table always has ≥2 columns; a 1-col ruled
+ // region is a framed text box, never tabular. (Catches docs 028649
+ // and 06ba2a90 — both emitted a 1-col table of paragraph blobs.
+ // Word-count gating can't be used here: the genuine scattered
+ // tables this pass rescues on opendataloader — competence/legal
+ // description grids — have equally long cells.)
+ if let Block::Table { header, rows } = &run.block {
+ let cols = header
+ .as_ref()
+ .map(|h| h.len())
+ .or_else(|| rows.first().map(Vec::len))
+ .unwrap_or(0);
+ if cols < 2 {
+ continue;
+ }
+ }
+ let top_y = consumed
+ .iter()
+ .map(|&i| lines[i].bbox.y)
+ .fold(f32::INFINITY, f32::min);
+ global_ruled_tables.push((top_y, run.block));
+ global_ruled_consumed.extend(consumed);
+ }
+ }
+ let global_ruled_owned: Option<Vec<ProjectedLine>> = if global_ruled_consumed.is_empty() {
+ None
+ } else {
+ Some(
+ lines
+ .iter()
+ .enumerate()
+ .filter(|(i, _)| !global_ruled_consumed.contains(i))
+ .map(|(_, l)| l.clone())
+ .collect(),
+ )
+ };
+ let lines: &[ProjectedLine] = global_ruled_owned.as_deref().unwrap_or(lines);
+
// Cross-region table re-merge: when a V-cut sliced a table into sibling
// leaves, fuse the slices back into single rows (one synthetic region)
// before grouping. Each successful merge carries its own validated table
@@ -210,6 +307,9 @@ pub fn classify_page_with_filters(
all_interruptions.push((r.bbox.y, Interruption::Figure(r.clone())));
}
}
+ for (y, block) in global_ruled_tables {
+ all_interruptions.push((y, Interruption::Table(block)));
+ }
all_interruptions.sort_by(|a, b| a.0.total_cmp(&b.0));
// region_boundary_idx[k] = index into `blocks` where region k's first
@@ -228,6 +328,7 @@ pub fn classify_page_with_filters(
id: r.id,
bbox: r.bbox,
},
+ Interruption::Table(b) => b,
});
};
@@ -405,6 +506,7 @@ fn classify_region(
id: r.id,
bbox: r.bbox,
},
+ Interruption::Table(b) => b,
});
}
};
diff --git a/crates/liteparse/src/markdown_layout/tables.rs b/crates/liteparse/src/markdown_layout/tables.rs
index 5f4d184..025e495 100644
--- a/crates/liteparse/src/markdown_layout/tables.rs
+++ b/crates/liteparse/src/markdown_layout/tables.rs
@@ -2361,6 +2361,14 @@ const TABLE_MAX_EMPTY_CELL_FRACTION_WITH_SPINE: f32 = 0.75;
/// a page border, not a real table.
const TABLE_MAX_PAGE_COVERAGE: f32 = 0.95;
+/// Global-pass only: minimum fraction of the column extent a horizontal rule
+/// must span to count as a row boundary. A full-height left/right border can
+/// union a stray box (logo, sidebar) into a grid component through the shared
+/// vertical line; that box's short rules would otherwise become phantom top
+/// rows that vacuum surrounding prose into the table. Real row boundaries span
+/// most of the table width.
+const RULED_HLINE_MIN_COVERAGE: f32 = 0.5;
+
/// Extract horizontal and vertical line segments from a page's graphics. Each
/// `Stroke` becomes one HSeg or VSeg depending on orientation; each stroked
/// `Rect` contributes its four edges (cell-border rects, table frames).
@@ -2549,19 +2557,47 @@ fn build_ruled_table(
lines: &[ProjectedLine],
page_width: f32,
page_height: f32,
-) -> Option<TableRun> {
+ global: bool,
+) -> Option<(TableRun, Vec<usize>)> {
let dbg = std::env::var("LITEPARSE_DEBUG_RULED").is_ok();
- // Distinct row y-coords (cluster again — multiple H lines may share a y).
- let mut ys: Vec<f32> = h_indices.iter().map(|&i| hs[i].y).collect();
- ys.sort_by(|a, b| a.total_cmp(b));
- dedup_close(&mut ys, TABLE_GRID_CLUSTER_PT);
-
let mut xs: Vec<f32> = v_indices.iter().map(|&i| vs[i].x).collect();
xs.sort_by(|a, b| a.total_cmp(b));
// Coarser, mean-centered clustering for column boundaries: cell-border
// rects contribute paired edges 4-6pt apart that would otherwise become
// phantom 5pt "columns" the span splitter then shreds text into.
cluster_boundaries(&mut xs, TABLE_COL_BOUNDARY_CLUSTER_PT);
+
+ // Distinct row y-coords (cluster again — multiple H lines may share a y).
+ // In the global pass, first drop horizontal rules that span only a small
+ // fraction of the column extent (see `RULED_HLINE_MIN_COVERAGE`).
+ let raw_ys = |idxs: &[usize]| {
+ let mut v: Vec<f32> = idxs.iter().map(|&i| hs[i].y).collect();
+ v.sort_by(|a, b| a.total_cmp(b));
+ dedup_close(&mut v, TABLE_GRID_CLUSTER_PT);
+ v
+ };
+ let ys: Vec<f32> = if global && xs.len() >= 2 {
+ let col_lo = xs[0];
+ let col_hi = xs[xs.len() - 1];
+ let extent = (col_hi - col_lo).max(1.0);
+ let kept: Vec<usize> = h_indices
+ .iter()
+ .copied()
+ .filter(|&i| {
+ let h = &hs[i];
+ let ov = (h.x_max.min(col_hi) - h.x_min.max(col_lo)).max(0.0);
+ ov / extent >= RULED_HLINE_MIN_COVERAGE
+ })
+ .collect();
+ let filtered = raw_ys(&kept);
+ if filtered.len() >= 3 {
+ filtered
+ } else {
+ raw_ys(h_indices)
+ }
+ } else {
+ raw_ys(h_indices)
+ };
if dbg {
eprintln!(
"[ruled] component: ys={:?} xs={:?} ({} lines in scope)",
@@ -2651,7 +2687,15 @@ fn build_ruled_table(
cell_has_text[row][col] = true;
};
- for (idx, line) in lines.iter().enumerate() {
+ // Iterate lines top-to-bottom. The global pass receives lines in xy_cut
+ // leaf order (column-by-column), which scrambles a multi-line cell's text
+ // when concatenated in array order; y-sorted iteration restores reading
+ // order within each cell. Per-region lines are already y-ordered, so this
+ // is a no-op there.
+ let mut line_order: Vec<usize> = (0..lines.len()).collect();
+ line_order.sort_by(|&a, &b| lines[a].bbox.y.total_cmp(&lines[b].bbox.y));
+ for idx in line_order {
+ let line = &lines[idx];
let cy = line.bbox.y + line.bbox.height * 0.5;
if cy < ys[0] || cy > ys[n_rows] {
continue;
@@ -2685,11 +2729,12 @@ fn build_ruled_table(
continue;
}
- let text_spans: Vec<&TextItem> = line
+ let mut text_spans: Vec<&TextItem> = line
.spans
.iter()
.filter(|s| !s.text.trim().is_empty())
.collect();
+ text_spans.sort_by(|a, b| a.x.total_cmp(&b.x));
let push_repl = |cells_repl: &mut Vec<Vec<String>>, row: usize, col: usize, txt: &str| {
let txt = txt.trim();
if txt.is_empty() {
@@ -3167,15 +3212,18 @@ fn build_ruled_table(
let start = *consumed_indices.iter().min().unwrap();
let end = *consumed_indices.iter().max().unwrap() + 1;
- Some(TableRun {
- start,
- end,
- body_start: start,
- block: Block::Table {
- header,
- rows: body_rows,
+ Some((
+ TableRun {
+ start,
+ end,
+ body_start: start,
+ block: Block::Table {
+ header,
+ rows: body_rows,
+ },
},
- })
+ consumed_indices,
+ ))
}
/// Single-link cluster a sorted Vec of boundary coordinates, replacing each
@@ -3295,12 +3343,13 @@ pub fn detect_table_rects(
/// Detect ruled-grid tables on a page from its vector graphics. Returns runs
/// in document order (sorted by `start`).
-pub(super) fn detect_ruled_tables(
+fn detect_ruled_tables_impl(
lines: &[ProjectedLine],
graphics: &[GraphicPrimitive],
page_width: f32,
page_height: f32,
-) -> Vec<TableRun> {
+ global: bool,
+) -> Vec<(TableRun, Vec<usize>)> {
let (hs, vs) = extract_h_v_segments(graphics);
let hs = cluster_h_segments(hs);
let vs = cluster_v_segments(vs);
@@ -3311,15 +3360,43 @@ pub(super) fn detect_ruled_tables(
let mut out = Vec::new();
for (h_idx, v_idx) in components {
if let Some(run) =
- build_ruled_table(&hs, &vs, &h_idx, &v_idx, lines, page_width, page_height)
+ build_ruled_table(&hs, &vs, &h_idx, &v_idx, lines, page_width, page_height, global)
{
out.push(run);
}
}
- out.sort_by_key(|r| r.start);
+ out.sort_by_key(|(r, _)| r.start);
out
}
+/// Per-region ruled-table detection. Drops the consumed-line bookkeeping the
+/// global pass needs.
+pub(super) fn detect_ruled_tables(
+ lines: &[ProjectedLine],
+ graphics: &[GraphicPrimitive],
+ page_width: f32,
+ page_height: f32,
+) -> Vec<TableRun> {
+ detect_ruled_tables_impl(lines, graphics, page_width, page_height, false)
+ .into_iter()
+ .map(|(r, _)| r)
+ .collect()
+}
+
+/// Page-level ruled-table detection over *all* lines. A table whose rows
+/// scatter across several xy_cut leaves never has its full text in any single
+/// leaf, so per-region detection rejects it as mostly-empty; running once
+/// globally reassembles it. Returns each run with the set of line indices it
+/// consumed so the caller can pull them out of the region pipeline.
+pub(super) fn detect_ruled_tables_global(
+ lines: &[ProjectedLine],
+ graphics: &[GraphicPrimitive],
+ page_width: f32,
+ page_height: f32,
+) -> Vec<(TableRun, Vec<usize>)> {
+ detect_ruled_tables_impl(lines, graphics, page_width, page_height, true)
+}
+
/// Count filled (non-empty) cells in a TableRun. GridFallback returns 0 so
/// it never beats a real Table in density comparisons.
fn run_filled_cells(run: &TableRun) -> usize {