A fast, helpful, and open-source document parser
wip
Logan Markewich4dc2a9464263
diff --git a/MARKDOWN_PROGRESS.md b/MARKDOWN_PROGRESS.mdindex 844fdd8..bc821d1 100644--- a/MARKDOWN_PROGRESS.md+++ b/MARKDOWN_PROGRESS.md@@ -2,6 +2,44 @@Tracks build-order progress against [MARKDOWN_PLAN.md](MARKDOWN_PLAN.md). Update as steps complete.+> **2026-06-11 launch-readiness quality audit (research only, no code changes).** Competitor-independent audit of absolute output quality, validated on real current output via a 5-way parallel investigation (word fusion, reading order, dropped/duplicated content, math, structural/inline formatting). Data sources: opendataloader 200 paired prediction/ground-truth markdown (`opendataloader-bench/{prediction/liteparse,ground-truth}/markdown/<id>.md`), olmOCR category outputs + predicate JSONLs, ParseBench formatting CSVs. Reusable triage script written: `dataset_eval_utils/order_defect_triage.py`.+>+> **Headline reframe — reading order is less broken than the metric implies.** The ~52.8% olmOCR order-pass rate overstates it. Of 320 failing multi_column order tests: ~84 are metric noise (both strings present *in correct order*, fail only on strict char-match from a dropped space — free wins via tolerance, not defects), ~61 are extraction/OCR drops (orthogonal), only **~8 are true column inversions**, and **~150 are the real defect: caption/sidebar text spliced into body prose** (floating captions / narrow side-blocks not isolated before column ordering — e.g. `multi_column/0b045ab8…page_18` prints caption #2 → body → caption #1 interleaved; `02943986…page_7`, `0904c7071…page_4`, `0925342e…page_4`, `0145bc47…page_11`). Fix target is sharp: caption/sidebar isolation in the XY-cut/anchor pass, NOT a reading-order rewrite.+>+> **Validated defect map, ranked by (impact × fixability):**+>+> *Tier 1 — fix before launch:*+> 1. **Caption/sidebar splicing** (~150 order tests). The real reading-order defect; most damaging for ingestion, well-scoped.+> 2. **Word fusion in dense/small-font text.** Near-zero in normal prose (opendataloader distinctive-token recall 0.98, ~1 confirmed fusion in 55k words), but **22.6% of olmOCR `long_tiny_text` files affected** — and lowercase→lowercase fusion (`theynexttakeonthe`, `againstManchester`, `Cityand`, `forSpurs` in `long_tiny_text/20_pg58…`) is far worse than the camelCase tip a regex catches. Confirmed `ofTurkey`→"of Turkey" (opendataloader 01030000000012, GT-verified). Concentrated, not pervasive; matches the `project_missing_spaces` adaptive-gap-threshold need. Trickiest fix (must stay table/column-safe).+> 3. **Table label-column drop** — the *only* real content-loss defect (all other losses are legitimately OCR-gated, expected under `--no-ocr`; median word-count ratio 0.96, no duplication). Numeric columns render but the leading label column drops/garbles: docs `01030000000082` (state names Gujarat/Punjab/…), `01030000000046` (party names Funcinpec/Candlelight/…), `121`/`122` (in-table prose → `. a . a tip` fragments). Root cause: table cell text reconstruction in the projection layer. Small but real correctness miss.+>+> *Tier 2 — cheap wins:*+> 4. **Code-block language tag.** Code blocks actually *work* (real Python correctly fenced on `text_misc__justcode.pdf`) but score 0.0 on ParseBench because we emit bare ` ``` ` with no language tag (`rules_formatting.py:583` requires `lang=="python"`), and snippets fragment across multiple fences. Near-trivial unblock. (NB: the ~273 fenced output files are mostly GridFallback ` ```text `, not code.)+> 5. **Links entirely absent** — no `GetURI`/annotation extraction exists in `extract.rs` or the pdfium wrapper at all. PDFium exposes it; genuinely missing feature, moderate effort.+>+> *Tier 3 — document as known v1 limitations (large effort, not launch blockers):*+> 6. **Math notation structurally absent.** Glyphs/Greek/operators (α 841×, ≤ 1016×, ∈ 2239×) decode fine, but sub/superscript positioning is flattened (**0/522 arxiv outputs** contain any super/sub char or `\frac`/`\begin`), display equations dumped inline with no `$$` delimiters, degree symbol dropped (`95°C`→`958C`). Affects *all* scientific/technical docs, not just arxiv. The #1 sub-gap is sub/superscript positioning, then equation delimiting — symbol decoding is mostly fine. Scope explicitly as out-of-scope for v1 rather than half-do.+> 7. **Bold/italic whole-line-only** (ParseBench bold 0.53 / italic 0.38 — emitted but rarely fine-grained spans; occasionally over-applied/malformed) and **heading-level accuracy weak** (title_hierarchy 0.37 — wrong levels, not mere presence). Lists work but flat (nesting untested — no nested list in either corpus). Fidelity tier, lower ingestion impact.+>+> **Recommended order when resuming:** #1 caption/sidebar splicing first (biggest "looks scrambled" win, well-scoped — start from `dataset_eval_utils/order_defect_triage.py` and the ~150 SHREDDED_INTERLEAVE docs to find the geometric signal separating a floating caption from body text); bundle #3 + #4 as small high-confidence correctness fixes; #2 dense-text word fusion is highest-value-invisible but budget for iteration; math (#6) document as a known limitation.+>+> **2026-06-11 global ruled-table pass — opendataloader Overall 0.8694 → 0.8710 (TEDS 0.6678 → 0.6877 (+0.0199), NID 0.9115 → 0.9120, MHS 0.8032 → 0.8036); ParseBench table grits_con 0.5507 → 0.5647, TRM 0.2123 → 0.2213, composite 0.3922 → 0.4040; olmOCR table_tests 46.7 → 47.3% (+6), Overall + multi_column flat at 38.4% / 63.8%.** Clean win, zero regressions. Crosses two competitor lines: **ParseBench TRM 0.2213 now beats pymupdf4llm 0.2205** (the last remaining table deficit), and the opendataloader TEDS gap to nutrient halves (−0.040 → −0.020, 0.6877 vs 0.708).+>+> **Root cause.** `detect_ruled_tables` runs per xy_cut leaf. A ruled table whose rows scatter across ≥2 leaves (a V-cut sliced it, or a banner/caption peeled a level) never has its full text in any single leaf, so per-region detection rejects it as mostly-empty and the grid is lost entirely. (Distinct from the cross-region pass, which re-fuses *borderless* slices via aligned baselines — this is the *ruled*-grid analog, keyed off the stroke geometry directly.)+>+> **New `detect_ruled_tables_global`** (`tables.rs`), run once over all filtered lines in `classify_page_with_filters` *before* cross-region merge. `build_ruled_table` gained a `global: bool` and now returns the consumed line indices so the caller can pull them out of the region pipeline and emit each table as a y-positioned `Interruption::Table`. Global-only refinements inside `build_ruled_table`:+> 1. **`RULED_HLINE_MIN_COVERAGE = 0.5`** — drop horizontal rules spanning <50% of the column extent before deriving row boundaries. A full-height left/right page border unions a stray box (logo, sidebar) into the grid component through the shared vertical; that box's short rules would otherwise become phantom top rows that vacuum surrounding prose. Falls back to all rules if the filter leaves <3.+> 2. **y-sorted line + span iteration** — the global pass receives lines in xy_cut leaf order (column-by-column), which scrambles multi-line-cell text on array-order concatenation; sorting by `bbox.y` (and spans by `x`) restores reading order within each cell. No-op for the already-y-ordered per-region path.+>+> **Acceptance gates in `classify.rs`** (each tuned against a measured misfire):+> - **≥2-leaf scatter required.** Only rescue tables the per-region path *can't* see whole — those whose consumed lines span ≥2 distinct `region_path`s. A table living in one leaf is already handled (often better) there.+> - **`already_handled` skip.** If any single leaf's share of the consumed lines already forms a table on its own (`detect_ruled_tables` or `detect_tables` non-empty), don't override — doc 127's decorative frame spans the data table *plus* surrounding prose; the data region tables cleanly by itself, the frame would fuse prose into garbage.+> - **1-column gate** (the load-bearing false-positive guard). A decorative banner/frame around a single prose column (left+right border, no interior verticals) unions into a 1-col "grid" whose lone cell is the whole paragraph block. A real data table always has ≥2 columns. Caught docs **028649** (2-col prose + "THE PLAN" banner fused into one 250-word cell) and **06ba2a90** (journal page + "Table II" caption box). **Word-count gating was tried first and reverted** — it killed *all* the opendataloader wins too: the genuine scattered tables this pass rescues (competence/legal description grids) have cells just as long as the fused-prose false positives, so cell length can't separate them; column count can. With the 1-col gate, opendataloader/ParseBench wins are byte-identical to the ungated run and olmOCR multi_column fully recovers (63.3 ungated → 63.8). Two changed multi_column docs remain (081875 3-col, 0d7383 2-col) — the documented Mode-5 ambiguous slide-feature-table family; they emit as tables but cost no order tests.+>+> Kill-switch `LITEPARSE_DISABLE_GLOBAL_RULED=1`. 190 lib tests pass.+>+> **Triage note:** A/B the multi_column dip by converting the disabled baseline to a named candidate (`liteparse:name=liteparse_noglobal`) and `diff`ing the per-doc `.md` — isolated exactly 4 changed docs out of 884, which is what surfaced the 1-col-vs-≥2-col split. Opendataloader A/B via the kill-switch on the same binary (no rebuild needed) confirmed the col-gate is byte-identical to the ungated win.+>> **2026-06-10 (pm) tight-gutter column rescue + XY depth/fill-gate session — opendataloader Overall 0.8633 → 0.8694 (NID 0.9071 → 0.9115, MHS 0.7865 → 0.8032, TEDS 0.6716 → 0.6678); olmOCR Overall 37.7 → 38.4% (multi_column 60.0 → 63.8%, table_tests 45.5 → 46.7% — now ahead of pymupdf4llm's 46.2); ParseBench table grits_con 0.5481 → 0.5507, TRM 0.2083 → 0.2123, composite 0.3886 → 0.3922.** Targeted the docs 38/189 "bold heading fused inline across an unsplit 2-col layout" family — the head of the nutrient MHS/NID gap (38/189 alone were MHS −0.97/−0.94 vs nutrient).>> Five changes, all `projection.rs::xy_find_column_cut` + recursion, each motivated by a measured geometry:diff --git a/crates/liteparse/src/markdown_layout/classify.rs b/crates/liteparse/src/markdown_layout/classify.rsindex fe3857e..fc97ff4 100644--- a/crates/liteparse/src/markdown_layout/classify.rs+++ b/crates/liteparse/src/markdown_layout/classify.rs@@ -26,6 +26,10 @@ use super::tables::{detect_ruled_tables, detect_tables, merge_table_runs};enum Interruption {Hr,Figure(crate::types::ImageRef),+ /// A ruled table detected by the global pass, already fully built. Emitted+ /// at its top-y position like other interruptions; its lines were pulled+ /// out of the region pipeline so it won't be re-detected per-region.+ Table(super::blocks::Block),}/// Returns true if any span on the line is rotated more than ~5° off@@ -145,6 +149,99 @@ pub fn classify_page_with_filters(&filtered_owned};+ // Global ruled-table pass. `detect_ruled_tables` is otherwise called per+ // xy_cut leaf, but a ruled table whose rows scatter across several leaves+ // never has its full text in any single leaf and gets rejected as+ // mostly-empty. Run it once over all lines, pull the consumed lines out of+ // the region pipeline, and emit each table as a y-positioned interruption.+ // Runs before cross-region merge so that pass (and the region indices its+ // runs carry) operate on the already-filtered line list. Disable with+ // LITEPARSE_DISABLE_GLOBAL_RULED=1.+ let mut global_ruled_tables: Vec<(f32, Block)> = Vec::new();+ let mut global_ruled_consumed: std::collections::HashSet<usize> =+ std::collections::HashSet::new();+ if std::env::var("LITEPARSE_DISABLE_GLOBAL_RULED").is_err() {+ for (run, consumed) in super::tables::detect_ruled_tables_global(+ lines,+ &page.graphics,+ page.page_width,+ page.page_height,+ ) {+ // Only rescue tables the per-region path can't see whole: those+ // whose rows scatter across ≥2 xy_cut leaves. A table living in a+ // single region is already handled (often better) by per-region+ // detection — replacing it here only risks regressions.+ let mut groups: std::collections::HashMap<&Vec<u16>, Vec<usize>> =+ std::collections::HashMap::new();+ for &i in &consumed {+ groups.entry(&lines[i].region_path).or_default().push(i);+ }+ if groups.len() < 2 {+ continue;+ }+ // If any single leaf's share of these lines already forms a table+ // on its own, the per-region path handles this content — don't+ // override (e.g. doc 127: a decorative frame spans the data table+ // plus surrounding prose; the data region tables cleanly by+ // itself, the frame would fuse prose into garbage cells).+ let already_handled = groups.values().any(|idxs| {+ if idxs.len() < 2 {+ return false;+ }+ let sub: Vec<ProjectedLine> = idxs.iter().map(|&i| lines[i].clone()).collect();+ !super::tables::detect_ruled_tables(+ &sub,+ &page.graphics,+ page.page_width,+ page.page_height,+ )+ .is_empty()+ || !super::tables::detect_tables(&sub).is_empty()+ });+ if already_handled {+ continue;+ }+ // Single-column gate. A decorative banner/frame around a prose+ // column (just a left+right border, no interior verticals) unions+ // into a 1-column "grid" whose lone cell is the whole paragraph+ // block. A real data table always has ≥2 columns; a 1-col ruled+ // region is a framed text box, never tabular. (Catches docs 028649+ // and 06ba2a90 — both emitted a 1-col table of paragraph blobs.+ // Word-count gating can't be used here: the genuine scattered+ // tables this pass rescues on opendataloader — competence/legal+ // description grids — have equally long cells.)+ if let Block::Table { header, rows } = &run.block {+ let cols = header+ .as_ref()+ .map(|h| h.len())+ .or_else(|| rows.first().map(Vec::len))+ .unwrap_or(0);+ if cols < 2 {+ continue;+ }+ }+ let top_y = consumed+ .iter()+ .map(|&i| lines[i].bbox.y)+ .fold(f32::INFINITY, f32::min);+ global_ruled_tables.push((top_y, run.block));+ global_ruled_consumed.extend(consumed);+ }+ }+ let global_ruled_owned: Option<Vec<ProjectedLine>> = if global_ruled_consumed.is_empty() {+ None+ } else {+ Some(+ lines+ .iter()+ .enumerate()+ .filter(|(i, _)| !global_ruled_consumed.contains(i))+ .map(|(_, l)| l.clone())+ .collect(),+ )+ };+ let lines: &[ProjectedLine] = global_ruled_owned.as_deref().unwrap_or(lines);+// Cross-region table re-merge: when a V-cut sliced a table into sibling// leaves, fuse the slices back into single rows (one synthetic region)// before grouping. Each successful merge carries its own validated table@@ -210,6 +307,9 @@ pub fn classify_page_with_filters(all_interruptions.push((r.bbox.y, Interruption::Figure(r.clone())));}}+ for (y, block) in global_ruled_tables {+ all_interruptions.push((y, Interruption::Table(block)));+ }all_interruptions.sort_by(|a, b| a.0.total_cmp(&b.0));// region_boundary_idx[k] = index into `blocks` where region k's first@@ -228,6 +328,7 @@ pub fn classify_page_with_filters(id: r.id,bbox: r.bbox,},+ Interruption::Table(b) => b,});};@@ -405,6 +506,7 @@ fn classify_region(id: r.id,bbox: r.bbox,},+ Interruption::Table(b) => b,});}};diff --git a/crates/liteparse/src/markdown_layout/tables.rs b/crates/liteparse/src/markdown_layout/tables.rsindex 5f4d184..025e495 100644--- a/crates/liteparse/src/markdown_layout/tables.rs+++ b/crates/liteparse/src/markdown_layout/tables.rs@@ -2361,6 +2361,14 @@ const TABLE_MAX_EMPTY_CELL_FRACTION_WITH_SPINE: f32 = 0.75;/// a page border, not a real table.const TABLE_MAX_PAGE_COVERAGE: f32 = 0.95;+/// Global-pass only: minimum fraction of the column extent a horizontal rule+/// must span to count as a row boundary. A full-height left/right border can+/// union a stray box (logo, sidebar) into a grid component through the shared+/// vertical line; that box's short rules would otherwise become phantom top+/// rows that vacuum surrounding prose into the table. Real row boundaries span+/// most of the table width.+const RULED_HLINE_MIN_COVERAGE: f32 = 0.5;+/// Extract horizontal and vertical line segments from a page's graphics. Each/// `Stroke` becomes one HSeg or VSeg depending on orientation; each stroked/// `Rect` contributes its four edges (cell-border rects, table frames).@@ -2549,19 +2557,47 @@ fn build_ruled_table(lines: &[ProjectedLine],page_width: f32,page_height: f32,-) -> Option<TableRun> {+ global: bool,+) -> Option<(TableRun, Vec<usize>)> {let dbg = std::env::var("LITEPARSE_DEBUG_RULED").is_ok();- // Distinct row y-coords (cluster again — multiple H lines may share a y).- let mut ys: Vec<f32> = h_indices.iter().map(|&i| hs[i].y).collect();- ys.sort_by(|a, b| a.total_cmp(b));- dedup_close(&mut ys, TABLE_GRID_CLUSTER_PT);-let mut xs: Vec<f32> = v_indices.iter().map(|&i| vs[i].x).collect();xs.sort_by(|a, b| a.total_cmp(b));// Coarser, mean-centered clustering for column boundaries: cell-border// rects contribute paired edges 4-6pt apart that would otherwise become// phantom 5pt "columns" the span splitter then shreds text into.cluster_boundaries(&mut xs, TABLE_COL_BOUNDARY_CLUSTER_PT);++ // Distinct row y-coords (cluster again — multiple H lines may share a y).+ // In the global pass, first drop horizontal rules that span only a small+ // fraction of the column extent (see `RULED_HLINE_MIN_COVERAGE`).+ let raw_ys = |idxs: &[usize]| {+ let mut v: Vec<f32> = idxs.iter().map(|&i| hs[i].y).collect();+ v.sort_by(|a, b| a.total_cmp(b));+ dedup_close(&mut v, TABLE_GRID_CLUSTER_PT);+ v+ };+ let ys: Vec<f32> = if global && xs.len() >= 2 {+ let col_lo = xs[0];+ let col_hi = xs[xs.len() - 1];+ let extent = (col_hi - col_lo).max(1.0);+ let kept: Vec<usize> = h_indices+ .iter()+ .copied()+ .filter(|&i| {+ let h = &hs[i];+ let ov = (h.x_max.min(col_hi) - h.x_min.max(col_lo)).max(0.0);+ ov / extent >= RULED_HLINE_MIN_COVERAGE+ })+ .collect();+ let filtered = raw_ys(&kept);+ if filtered.len() >= 3 {+ filtered+ } else {+ raw_ys(h_indices)+ }+ } else {+ raw_ys(h_indices)+ };if dbg {eprintln!("[ruled] component: ys={:?} xs={:?} ({} lines in scope)",@@ -2651,7 +2687,15 @@ fn build_ruled_table(cell_has_text[row][col] = true;};- for (idx, line) in lines.iter().enumerate() {+ // Iterate lines top-to-bottom. The global pass receives lines in xy_cut+ // leaf order (column-by-column), which scrambles a multi-line cell's text+ // when concatenated in array order; y-sorted iteration restores reading+ // order within each cell. Per-region lines are already y-ordered, so this+ // is a no-op there.+ let mut line_order: Vec<usize> = (0..lines.len()).collect();+ line_order.sort_by(|&a, &b| lines[a].bbox.y.total_cmp(&lines[b].bbox.y));+ for idx in line_order {+ let line = &lines[idx];let cy = line.bbox.y + line.bbox.height * 0.5;if cy < ys[0] || cy > ys[n_rows] {continue;@@ -2685,11 +2729,12 @@ fn build_ruled_table(continue;}- let text_spans: Vec<&TextItem> = line+ let mut text_spans: Vec<&TextItem> = line.spans.iter().filter(|s| !s.text.trim().is_empty()).collect();+ text_spans.sort_by(|a, b| a.x.total_cmp(&b.x));let push_repl = |cells_repl: &mut Vec<Vec<String>>, row: usize, col: usize, txt: &str| {let txt = txt.trim();if txt.is_empty() {@@ -3167,15 +3212,18 @@ fn build_ruled_table(let start = *consumed_indices.iter().min().unwrap();let end = *consumed_indices.iter().max().unwrap() + 1;- Some(TableRun {- start,- end,- body_start: start,- block: Block::Table {- header,- rows: body_rows,+ Some((+ TableRun {+ start,+ end,+ body_start: start,+ block: Block::Table {+ header,+ rows: body_rows,+ },},- })+ consumed_indices,+ ))}/// Single-link cluster a sorted Vec of boundary coordinates, replacing each@@ -3295,12 +3343,13 @@ pub fn detect_table_rects(/// Detect ruled-grid tables on a page from its vector graphics. Returns runs/// in document order (sorted by `start`).-pub(super) fn detect_ruled_tables(+fn detect_ruled_tables_impl(lines: &[ProjectedLine],graphics: &[GraphicPrimitive],page_width: f32,page_height: f32,-) -> Vec<TableRun> {+ global: bool,+) -> Vec<(TableRun, Vec<usize>)> {let (hs, vs) = extract_h_v_segments(graphics);let hs = cluster_h_segments(hs);let vs = cluster_v_segments(vs);@@ -3311,15 +3360,43 @@ pub(super) fn detect_ruled_tables(let mut out = Vec::new();for (h_idx, v_idx) in components {if let Some(run) =- build_ruled_table(&hs, &vs, &h_idx, &v_idx, lines, page_width, page_height)+ build_ruled_table(&hs, &vs, &h_idx, &v_idx, lines, page_width, page_height, global){out.push(run);}}- out.sort_by_key(|r| r.start);+ out.sort_by_key(|(r, _)| r.start);out}+/// Per-region ruled-table detection. Drops the consumed-line bookkeeping the+/// global pass needs.+pub(super) fn detect_ruled_tables(+ lines: &[ProjectedLine],+ graphics: &[GraphicPrimitive],+ page_width: f32,+ page_height: f32,+) -> Vec<TableRun> {+ detect_ruled_tables_impl(lines, graphics, page_width, page_height, false)+ .into_iter()+ .map(|(r, _)| r)+ .collect()+}++/// Page-level ruled-table detection over *all* lines. A table whose rows+/// scatter across several xy_cut leaves never has its full text in any single+/// leaf, so per-region detection rejects it as mostly-empty; running once+/// globally reassembles it. Returns each run with the set of line indices it+/// consumed so the caller can pull them out of the region pipeline.+pub(super) fn detect_ruled_tables_global(+ lines: &[ProjectedLine],+ graphics: &[GraphicPrimitive],+ page_width: f32,+ page_height: f32,+) -> Vec<(TableRun, Vec<usize>)> {+ detect_ruled_tables_impl(lines, graphics, page_width, page_height, true)+}+/// Count filled (non-empty) cells in a TableRun. GridFallback returns 0 so/// it never beats a real Table in density comparisons.fn run_filled_cells(run: &TableRun) -> usize {