A fast, helpful, and open-source document parser
wip
Logan Markewich0b8e5d8d385b
diff --git a/MARKDOWN_PROGRESS.md b/MARKDOWN_PROGRESS.mdindex ef35987..c23031c 100644--- a/MARKDOWN_PROGRESS.md+++ b/MARKDOWN_PROGRESS.md@@ -2,6 +2,38 @@Tracks build-order progress against [MARKDOWN_PLAN.md](MARKDOWN_PLAN.md). Update as steps complete.+> **2026-06-09 (pm) MHS/heading session — opendataloader Overall 0.8275 → 0.8602 (MHS 0.698 → 0.781, NID 0.890 → 0.907, TEDS flat 0.651); olmOCR table_tests 43.0 → 44.2, multi_column 59.5 → 60.0.** Five changes, descending impact:+>+> 1. **Dropped the `<!-- page N -->` markers from markdown output** (`output/markdown.rs`; `-----` separator kept). The opendataloader MHS evaluator builds a flat heading/content tree (**heading levels are ignored entirely** — `evaluator_heading_level.py` tags nodes only `heading`/`content`); on small trees (GT = 1 heading ≈ 4 nodes) the comment created a pre-heading content node costing ~0.25/doc across ~30 docs. Simulated +0.058 MHS / +0.004 NID before touching code, confirmed by bench. **olmOCR side effect:** headers_footers drops 58.9 → 57.4 — verified by re-prepending the comment to converted outputs (59.3) that the comment was *displacing un-stripped chrome out of the windowed `absent` checks*, i.e. an accidental benchmark hack we no longer benefit from. The honest recovery is better chrome stripping (the known Tel/email/letterhead catalog gaps).+> 2. **Off-page text clip** (`extract.rs`, end of `extract_page_text_items`): drop items entirely outside the page view box (print-spread/imposed PDFs carry the neighbouring page's text at x > page width in the same content stream; viewers never show it). Fixes the doc 28/29/31 family (sections from the *next* page emitted + heading-promoted) and strips prepress slug chrome on olmOCR hf docs. Partially-visible items kept; kill-switch `LITEPARSE_DISABLE_OFFPAGE_CLIP=1`.+> 3. **Ruled-table straddle guard** (`tables.rs`): reject a ruled-grid candidate when >45% of its spans (≥6) cross interior column boundaries by ≥3pt on both sides — decorative slide/layout boxes over flowing text (doc 198, a TOC slide scrambled into a 4-col table via the spine bypass) slice through prose runs; real ruled tables keep text inside cells. Zero hits across all 503 ParseBench table pages; doc 198 MHS 0.0 → clean `# Contents` + list. Kill-switch `LITEPARSE_DISABLE_STRADDLE_GUARD=1`.+> 4. **Two heading run-in guards** (`classify.rs` / `headings.rs`): (a) `": "` joins `". "` in the bold-heading run-in rejection (`Parameter Model: We have released the SO-`); (b) size-promotion suppressed when the *previous* line ends in a mid-word hyphen wrap regardless of capitalization (the existing guard required starts-lowercase). Doc 195: 6 → 4 headings, exact GT match. Kill-switch `LITEPARSE_DISABLE_HEADING_GUARDS=1`.+> 5. **Bare-HR suppression** (`output/markdown.rs`): a page whose only surviving blocks are horizontal rules (all text stripped as chrome) renders empty instead of stacked `---`.+>+> **Regression checks:** olmOCR Overall flat 37.7% (gated changes alone are +1.4 table / +0.5 multi_column; the hf −1.5 is the artifact above, baseline 99.9% is the same artifact on one chrome-only page). Full ParseBench post-session: text_content 0.6492 (+0.001), layout 0.8374 (flat), text_formatting 0.3755 (+0.005), chart 0.0335 (flat). ParseBench table grits_con measured 0.5188 vs the 0.5268 recorded this morning — **proven not caused by today's changes**: A/B with all kill-switches over all 503 table pages = byte-identical markdown, and re-injecting the page comment into stored results + `--skip_inference` reproduces 0.5188 exactly. The morning 0.5268 is another instance of the recorded-score/binary drift already flagged in the sweep note. 185 lib tests pass.+>+> **MHS gap vs nutrient after this session: −16.2 → −8.2 doc-points (0.781 vs 0.819).** Remaining head: docs 38/189 (bold heading fused inline across an unsplit 2-col layout — col-fill-gate territory in projection.rs), 103/36/184 (infographic line fusion), 146 (cross-region table: V-cut splits the Area/Competence table into two leaves so no detector sees the rows whole). opendataloader overall 0.8602 vs nutrient 0.8851 — remaining gap is NID 0.907 vs 0.925, MHS 0.781 vs 0.819, TEDS 0.651 vs 0.708.+> **2026-06-09 full benchmark sweep — baseline snapshot + competitor comparison (no code changes).** Re-ran all three benches from fresh artifacts (`lit` + root-venv Python binding rebuilt), including the first *full* ParseBench run (all 5 groups — runs fine without `ANTHROPIC_API_KEY`; rule-based eval only). Competitor candidates rescored on olmOCR from their existing converted markdown.+>+> **Current state — liteparse:**+>+> | Bench | Metric | Value |+> |---|---|---|+> | olmOCR | Overall / multi_column / table_tests / headers_footers / long_tiny_text | 37.7% / 59.5% / 43.0% / 58.9% / 26.9% |+> | opendataloader | Overall / NID / TEDS / MHS | 0.8275 / 0.890 / 0.651 / 0.698 |+> | ParseBench table | GriTS_con / TRM / composite | 0.5268 / 0.1985 / 0.3737 |+> | ParseBench (new groups) | text_content / layout / text_formatting / chart | 0.648 / 0.837 / 0.371 / 0.032 |+>+> (ParseBench GriTS_con reads 0.5268 vs the 0.4389 recorded in the cluster-merge entry below; the pre-existing top-level report in `output/liteparse_markdown/` already showed 0.5266 before this sweep, so the jump came from work landed between that entry's bench run and today's binary, not from this re-run.)+>+> **vs model-free competitors:**+> - **olmOCR Overall: #1.** liteparse 37.7 > pymupdf4llm 32.9 > opendataloader 32.7 > pdf_inspector 30.5 > markitdown 28.7. Per-category gaps remaining: multi_column 59.5 vs pymupdf4llm 66.7 (−7.2pp), table_tests 43.0 vs 46.2 (−3.2pp), long_tiny_text 26.9 vs opendataloader 35.1 (−8.2pp). Only candidate at 100% baseline.+> - **opendataloader-bench: #3 model-free**, effectively tied with opendataloader (0.8275 vs 0.8312; we lead TEDS 0.651 vs 0.489, trail NID −0.013 and MHS −0.042). **nutrient 0.885 is the one model-free engine still clearly ahead** — gap is NID (0.925 vs 0.890) and MHS (0.819 vs 0.698), i.e. reading order + heading structure, matching the known 2-col-heading / XY-cut failure modes. Others: pymupdf4llm 0.732, markitdown 0.589.+> - **ParseBench table:** GriTS_con now beats pymupdf4llm (0.527 vs 0.499); record-match still behind (0.199 vs 0.221); composite tied (0.374 vs 0.373). text_content ties opendataloader (0.648 vs 0.648), beats pdf_inspector (0.580). No competitor runs exist yet for layout/formatting/chart.+> - **text_formatting weak spots** (first measurement): code blocks 0%, sup/sub/underline/strike/mark ~0%, title pass 0.44, title hierarchy 0.37, bold 0.51, italic 0.37. chart 0.032 is expected (no chart-data extraction).+>+> **Highest-leverage next targets:** MHS/NID vs nutrient (2-col heading isolation, reading order), olmOCR multi_column, ParseBench record-match + title hierarchy.+> **2026-06-09 union-track cluster merge — ParseBench grits_con 0.4308 → 0.4389, TRM composite 0.2887 → 0.2947, unmatched predicted tables −26%; olmOCR & opendataloader regression-clean.** Triage showed 394 of ParseBench's 425 unmatched predicted tables were fragments of real tables (runs broken at column-count changes, stranded PDFium-merged rows, 56% of emitted tables headerless). New `merge_fragmented_table_clusters` post-pass in `markdown_layout/tables.rs` re-derives union column tracks across adjacent table runs and re-extracts every line against them; `union_header_from_above` bins stacked/colspan headers per-span with replication. Track-compatibility gate (≥75% of narrower run's tracks must map into wider run's) prevents fusing genuinely separate stacked tables. Env kill-switch `LITEPARSE_DISABLE_CLUSTER_MERGE=1`. Full details + tried-and-reverted notes (gap-based wrap merge has no signal; opendataloader TEDS baseline correction 0.5917 → 0.568) in FAILURE_MODES.md fix log. Next table levers: multi-line-cell row merging (needs ruled/label-anchor signal, not gaps) and cross-region fragments (V-cut sliced tables).> **2026-06-04 inferred-tracks table detector — olmOCR-bench table_tests 38.8% → 39.5% (+0.7pp, +7 tests), Overall 35.1% → 35.2%; opendataloader & ParseBench unchanged.** Targeted the dominant `wrong_neighbors` failure on financial/earnings tables (e.g. Apple/Wiley `a769f9a3…pg2`, `fa18a15c…`, hundreds of similar narrow-numeric tables): the existing `try_detect_table` derives column tracks from the FIRST row's `split_cells` output, which uses `TABLE_CELL_GAP_FONT_MULTIPLIER × font_size` (= 14pt at 14pt body) as the inter-span gap threshold. The Apple Revenue Q3 row has `$448 → $427` and `$427 → 7%` gaps of 13.9pt — *just under* the threshold — so its 5 distinct PDFium items collapse to 3 cells (`$448 $427 7%` lumped). Every row in the Revenue table fails identically, so the table establishes as 3-col, and the cross-run merge can't recover (`merge_consecutive_table_runs` only handles `|A|+1 == |B|`).diff --git a/crates/liteparse/src/extract.rs b/crates/liteparse/src/extract.rsindex 5d6e002..27937b9 100644--- a/crates/liteparse/src/extract.rs+++ b/crates/liteparse/src/extract.rs@@ -660,6 +660,28 @@ fn extract_page_text_items(seg.flush(&mut items);+ // Drop items entirely outside the page view box. Print-spread / imposed+ // PDFs carry the neighbouring page's text at x beyond the page edge in+ // the same content stream; viewers never show it. Partially-visible+ // items are kept.+ let vb_w = (view_box.right - view_box.left).abs();+ let vb_h = (view_box.top - view_box.bottom).abs();+ let pre_clip_count = items.len();+ if std::env::var("LITEPARSE_DISABLE_OFFPAGE_CLIP").is_err() {+ items.retain(|it| {+ it.x < vb_w+ && it.x + it.width.max(0.1) > 0.0+ && it.y < vb_h+ && it.y + it.height.max(0.1) > 0.0+ });+ }+ if debug && items.len() < pre_clip_count {+ eprintln!(+ "[extract-debug] off-page clip removed {} items",+ pre_clip_count - items.len()+ );+ }+if debug {eprintln!("[extract-debug] items before dedup: {}", items.len());}diff --git a/crates/liteparse/src/markdown_layout/classify.rs b/crates/liteparse/src/markdown_layout/classify.rsindex b0f5ee2..272a96f 100644--- a/crates/liteparse/src/markdown_layout/classify.rs+++ b/crates/liteparse/src/markdown_layout/classify.rs@@ -490,7 +490,16 @@ fn classify_region(.as_ref().map(|p| &p.last).or(last_list_line.map(|i| &lines[i]));- !(starts_lower && prev.is_some_and(|p| continues_paragraph(p, line)))+ // A previous line ending in a mid-word hyphen wrap means this+ // line is its continuation regardless of capitalization+ // ("…SOLAR 10.7 Billion-" / "Parameter Model: We have…").+ let prev_hyphen_wrap = std::env::var("LITEPARSE_DISABLE_HEADING_GUARDS").is_err()+ && prev.is_some_and(|p| {+ let t = p.text.trim_end();+ t.ends_with('-') && t.chars().rev().nth(1).is_some_and(|c| c.is_alphanumeric())+ });+ !((starts_lower || prev_hyphen_wrap)+ && prev.is_some_and(|p| continues_paragraph(p, line)))});// Long-text guard: a real heading is a label, not a sentence with// multiple clauses. Footnotes / citations / captions that happen to@@ -512,7 +521,7 @@ fn classify_region(// heading. Both signals must fire — a heading occasionally has either// on its own (e.g. abbreviation periods, intentional hyphen).let size_level = size_level.filter(|_| {- let has_mid_sentence = text.contains(". ");+ let has_mid_sentence = text.contains(". ") || text.contains(": ");let mid_word_wrap = text.trim_end().ends_with('-');!(has_mid_sentence && mid_word_wrap)});diff --git a/crates/liteparse/src/markdown_layout/headings.rs b/crates/liteparse/src/markdown_layout/headings.rsindex 68b08b3..266e4b7 100644--- a/crates/liteparse/src/markdown_layout/headings.rs+++ b/crates/liteparse/src/markdown_layout/headings.rs@@ -383,7 +383,12 @@ pub(super) fn looks_like_bold_heading(// - ≥3 space-separated words after the break// - line ends with mid-word "-" (wrap continuation) OR is >50 chars// - first char after the break is uppercase ASCII letter- if let Some(pos) = text.find(". ") {+ let colon_ok = std::env::var("LITEPARSE_DISABLE_HEADING_GUARDS").is_err();+ let run_in_break = text+ .find(". ")+ .map(|p| (p, 2))+ .or_else(|| text.find(": ").map(|p| (p, 2)).filter(|_| colon_ok));+ if let Some((pos, sep_len)) = run_in_break {let before = &text[..pos];// Section-number prefix exemption: when the segment before ". " is a// numbered section identifier (e.g. "1", "1.5", "A.2", "Sec. 2",@@ -391,7 +396,7 @@ pub(super) fn looks_like_bold_heading(// "1.5. Migrant Workers..." — the period is part of the section// number, not a sentence terminator. Skip the run-in rejection.let is_section_number = is_section_number_prefix(before);- let after = text[pos + 2..].trim();+ let after = text[pos + sep_len..].trim();let starts_upper = after.chars().next().is_some_and(|c| c.is_ascii_uppercase());let word_count = after.split_whitespace().count();let ends_hyphen = text.trim_end().ends_with('-');diff --git a/crates/liteparse/src/markdown_layout/tables.rs b/crates/liteparse/src/markdown_layout/tables.rsindex 05ee051..5f4d184 100644--- a/crates/liteparse/src/markdown_layout/tables.rs+++ b/crates/liteparse/src/markdown_layout/tables.rs@@ -2627,6 +2627,13 @@ fn build_ruled_table(let mut row_alpha_spanner: Vec<bool> = vec![false; n_rows];let mut consumed_indices: Vec<usize> = Vec::new();const GRID_X_SLACK_PT: f32 = 6.0;+ // Straddle census: spans that cross an interior column boundary by a+ // clear margin on both sides. A real ruled table keeps text inside cells+ // (the occasional PDFium merged run aside); decorative slide/layout boxes+ // over flowing prose slice through most runs (doc 198, a TOC slide).+ const STRADDLE_MARGIN_PT: f32 = 3.0;+ let mut span_total = 0usize;+ let mut span_straddle = 0usize;let mut push_cell = |cells: &mut Vec<Vec<String>>,cell_has_text: &mut Vec<Vec<bool>>,@@ -2718,6 +2725,14 @@ fn build_ruled_table(let sx1 = (span.x + span.width).clamp(xs[0], xs[n_cols]);let c_lo = find_bucket(&xs, sx0).unwrap_or(0);let c_hi = find_bucket(&xs, sx1).unwrap_or(n_cols - 1);+ span_total += 1;+ {+ let m0 = (span.x + STRADDLE_MARGIN_PT).clamp(xs[0], xs[n_cols]);+ let m1 = (span.x + span.width - STRADDLE_MARGIN_PT).clamp(xs[0], xs[n_cols]);+ if m1 > m0 && find_bucket(&xs, m0) != find_bucket(&xs, m1) {+ span_straddle += 1;+ }+ }if c_lo == c_hi {push_cell(&mut cells, &mut cell_has_text, row, c_lo, &span.text);push_repl(&mut cells_repl, row, c_lo, &span.text);@@ -2768,6 +2783,24 @@ fn build_ruled_table(return None;}+ let straddle_frac = if span_total > 0 {+ span_straddle as f32 / span_total as f32+ } else {+ 0.0+ };+ if dbg {+ eprintln!("[ruled] straddle {span_straddle}/{span_total} = {straddle_frac:.2}");+ }+ if span_total >= 6+ && straddle_frac > 0.45+ && std::env::var("LITEPARSE_DISABLE_STRADDLE_GUARD").is_err()+ {+ if dbg {+ eprintln!("[ruled] REJECT straddle-frac {straddle_frac:.2}");+ }+ return None;+ }+// Collapse "phantom rows" produced by stacked thin border-strip rects// (doc 149 draws each visual table row as: top border strip ~1pt, body// rect ~22pt, bottom border strip ~5pt — each contributes y-coords thatdiff --git a/crates/liteparse/src/output/markdown.rs b/crates/liteparse/src/output/markdown.rsindex 2090bf3..c4a8d48 100644--- a/crates/liteparse/src/output/markdown.rs+++ b/crates/liteparse/src/output/markdown.rs@@ -13,8 +13,8 @@ use crate::types::{OutlineTarget, ParsedPage};/// cover headings, paragraphs (with de-hyphenation and inline emphasis), lists,/// code blocks, ruled and borderless tables, horizontal rules, and figures.///-/// Pages are emitted in order, separated by `\n\n-----\n\n` with a-/// `<!-- page N -->` marker. Pages that contain no projected lines (e.g. blank+/// Pages are emitted in order, separated by `\n\n-----\n\n`.+/// Pages that contain no projected lines (e.g. blank/// or fully-OCR pages without font-size info) fall back to the projected text/// wrapped in a fenced block so we never silently drop content.pub fn format_markdown(@@ -35,8 +35,6 @@ pub fn format_markdown(if i > 0 {out.push_str("\n\n-----\n\n");}- out.push_str(&format!("<!-- page {} -->\n\n", page.page_number));-if page.projected_lines.is_empty() {// No structural metadata for this page — fall back to the// projection text inside a fence so nothing is dropped.@@ -58,7 +56,7 @@ pub fn format_markdown(.cloned().collect();let chrome_indices = detect_single_page_chrome(page, body_size);- let blocks = classify_page_with_filters(+ let mut blocks = classify_page_with_filters(page,&heading_map,&header_footer,@@ -66,6 +64,19 @@ pub fn format_markdown(image_mode,&chrome_indices,);+ // A page whose only surviving blocks are horizontal rules (all its+ // text was stripped as chrome) should render empty, not as a stack+ // of bare `---` separators.+ let has_content = blocks.iter().any(|b| {+ !matches!(+ b,+ crate::markdown_layout::Block::HorizontalRule+ | crate::markdown_layout::Block::Figure { .. }+ )+ });+ if !has_content {+ blocks.retain(|b| !matches!(b, crate::markdown_layout::Block::HorizontalRule));+ }out.push_str(&render_blocks(&blocks));}out@@ -139,7 +150,6 @@ mod tests {let out = format_markdown(&[p], &[], ImageMode::Placeholder);assert!(out.contains("```text"));assert!(out.contains("hello"));- assert!(out.contains("<!-- page 1 -->"));}#[test]