A fast, helpful, and open-source document parser
wip
Logan Markewichb2459acab58f
diff --git a/MARKDOWN_PROGRESS.md b/MARKDOWN_PROGRESS.mdindex c23031c..056a698 100644--- a/MARKDOWN_PROGRESS.md+++ b/MARKDOWN_PROGRESS.md@@ -2,6 +2,21 @@Tracks build-order progress against [MARKDOWN_PLAN.md](MARKDOWN_PLAN.md). Update as steps complete.+> **2026-06-10 cross-region table re-merge — opendataloader Overall 0.8602 → 0.8633 (TEDS 0.6514 → 0.6716, MHS 0.7807 → 0.7865, NID +0.0006); ParseBench table grits_con 0.5188 → 0.5481 (+0.029 A/B-attributed), TRM 0.1978 → 0.2083, composite 0.3682 → 0.3886; olmOCR table_tests 44.2 → 45.5% (+13 tests).** Targeted the "V-cut slices a table into sibling leaves" failure (docs 146/64 on opendataloader, the deferred cross-region lever from the 06-09 cluster-merge entry).+>+> **New pass: `markdown_layout/cross_region.rs`**, run in `classify_page_with_filters` *before* region grouping (up to 3 merges/page). Pipeline:+> 1. **Candidate pair**: two leaves either x-disjoint (gap ≥ 8pt) or direct region-tree siblings with x-overlap < 40% of the narrower side (straddle-widened V-cut slices, e.g. Federal Register), overlapping ≥ 50% in y. Extended to all leaves predominantly on one side of the cut inside the pair's **union** y-band (a side may itself be H-cut — 146's Area column is two leaves). Set must tile a contiguous line range.+> 2. **Gates** (each killed a real misfire): TOC pages skipped (doc 155 — split number/title/page-number leaves fuse into a convincing fake table); **either**-side-prose reject where prose = ≥3 lines, median-width/side-width ≥ 0.7, vertical fill ≥ 0.55, <30% multi-cell lines (doc 156 body+sidebar, doc 183 infographic; the multi-cell clause keeps sliced *wide* tables whose row-halves also fill their side); ≥3 baselines aligned across the cut and ≥ 40% of the sparser side (doc 43 infographic %-labels).+> 3. **Fuse**: same-baseline lines concatenate spans (cell boundaries fall out of span gaps in `split_cells`), zone gets the cut-parent `region_path`.+> 4. **Validate or drop**: (A) re-run `detect_tables` on the fused lines — accept if covering runs ≥ 60% of rows AND <30% of cells are ≥5-word "cells" (the long-cell gate killed the newspaper/references-column misfires: olmOCR 20_pg35, 0208fbb5 family — fused prose aligns into convincing grids but its cells are sentence fragments); else (B) direct 2-col path for label/description shapes whose list-marker labels (`1. Embodying…`) `is_label_like` rightly rejects in normal flow — V-cut geometry + aligned rows substitute for the label gate; rows anchored on left-side lines, wrap-joins for labels and descriptions, header promoted when bold/short. Validated runs are threaded into `classify_region` as precomputed borderless runs so emission is guaranteed; a failed candidate leaves the page untouched.+>+> Kill-switch `LITEPARSE_DISABLE_CROSS_REGION_TABLES=1`, debug `LITEPARSE_DEBUG_CROSS_REGION=1`. 190 lib tests pass (5 new).+>+> **Per-doc:** opendataloader 64 TEDS 0.511 → **1.000** (PORT SHIPCALLS 3-col), 146 TEDS 0.0 → 0.358 + MHS 0.317 → 0.942 + NID +0.055 (Area/Competence emitted exactly); per-doc diff vs 06-09 history: **+2/−0 on every metric** across all 200 docs. olmOCR same-binary A/B: table +13, multi_column −2 net / long_tiny_text flat (the residual "regressions" are pages where a genuine sliced table now emits — e.g. 0208fbb5 Spoil-Attribute ± SE table — trading an order test for table wins), Overall flat 37.6-37.7%.+>+> **vs competitors after this session:** ParseBench table composite **0.3886 vs pymupdf4llm 0.3726 — now ahead**; grits_con 0.548 vs 0.499; TRM 0.208 vs 0.221 (last remaining table deficit). opendataloader Overall 0.8633 vs nutrient 0.8851 — gap now NID 0.907 vs 0.925 + MHS 0.787 vs 0.819; TEDS 0.672 vs 0.708.+>+> **Residuals / next:** (a) doc 147-style 4-col tables with long wrapped paragraph cells — candidate forms, validation correctly refuses; needs the multi-line-cell row-merge signal (Mode-5). (b) Federal Register `1653739079_page34` — sibling pair now forms but the leaves mix column prose *above* the table band, diluting validation; needs sub-leaf y-banding (merge only the table band, not whole leaves). (c) doc 116 — slices fully overlap in x (item-level histogram partition), outside this geometry. (d) doc 43-family infographics stay rejected (good).> **2026-06-09 (pm) MHS/heading session — opendataloader Overall 0.8275 → 0.8602 (MHS 0.698 → 0.781, NID 0.890 → 0.907, TEDS flat 0.651); olmOCR table_tests 43.0 → 44.2, multi_column 59.5 → 60.0.** Five changes, descending impact:>> 1. **Dropped the `<!-- page N -->` markers from markdown output** (`output/markdown.rs`; `-----` separator kept). The opendataloader MHS evaluator builds a flat heading/content tree (**heading levels are ignored entirely** — `evaluator_heading_level.py` tags nodes only `heading`/`content`); on small trees (GT = 1 heading ≈ 4 nodes) the comment created a pre-heading content node costing ~0.25/doc across ~30 docs. Simulated +0.058 MHS / +0.004 NID before touching code, confirmed by bench. **olmOCR side effect:** headers_footers drops 58.9 → 57.4 — verified by re-prepending the comment to converted outputs (59.3) that the comment was *displacing un-stripped chrome out of the windowed `absent` checks*, i.e. an accidental benchmark hack we no longer benefit from. The honest recovery is better chrome stripping (the known Tel/email/letterhead catalog gaps).diff --git a/crates/liteparse/src/markdown_layout/classify.rs b/crates/liteparse/src/markdown_layout/classify.rsindex 272a96f..fe3857e 100644--- a/crates/liteparse/src/markdown_layout/classify.rs+++ b/crates/liteparse/src/markdown_layout/classify.rs@@ -145,6 +145,33 @@ pub fn classify_page_with_filters(&filtered_owned};+ // Cross-region table re-merge: when a V-cut sliced a table into sibling+ // leaves, fuse the slices back into single rows (one synthetic region)+ // before grouping. Each successful merge carries its own validated table+ // runs, which are handed to `classify_region` so emission is guaranteed+ // even for shapes the standard detectors would reject in normal flow.+ // TOC pages are skipped: their split number/title/page-number leaves+ // baseline-align perfectly and fuse into a convincing-but-wrong table.+ let mut cross_merged_owned: Option<Vec<ProjectedLine>> = None;+ let mut cross_region_runs: Vec<(Vec<u16>, Vec<super::tables::TableRun>)> = Vec::new();+ for _ in 0..3 {+ if toc_page {+ break;+ }+ let cur: &[ProjectedLine] = cross_merged_owned.as_deref().unwrap_or(lines);+ let Some(m) = super::cross_region::find_cross_region_table_merge(cur) else {+ break;+ };+ let mut next: Vec<ProjectedLine> = Vec::with_capacity(cur.len());+ next.extend_from_slice(&cur[..m.start]);+ let merged_path = m.merged[0].region_path.clone();+ next.extend(m.merged);+ next.extend_from_slice(&cur[m.end..]);+ cross_region_runs.push((merged_path, m.runs));+ cross_merged_owned = Some(next);+ }+ let lines: &[ProjectedLine] = cross_merged_owned.as_deref().unwrap_or(lines);+// Region-grouped pipeline: split the filtered line list into contiguous// runs sharing a `region_path` (one xy_cut leaf each) and classify each// leaf as its own mini-page. Paragraph / list / code / heading state is@@ -239,6 +266,10 @@ pub fn classify_page_with_filters(}let region_start = blocks.len();+ let precomputed_tables = cross_region_runs+ .iter()+ .find(|(path, _)| *path == lines[rstart].region_path)+ .map(|(_, runs)| runs.clone());let region_blocks = classify_region(region_lines,region_interruptions,@@ -248,6 +279,7 @@ pub fn classify_page_with_filters(image_mode,toc_page,debug,+ precomputed_tables,);if !region_blocks.is_empty() {region_boundaries.push(region_start);@@ -271,6 +303,7 @@ pub fn classify_page_with_filters(/// `stitch_regions` post-pass. Page-level signals (`heading_map`,/// `struct_nodes`, `outline`, header/footer y-bands) are still consulted/// because they're computed once per document.+#[allow(clippy::too_many_arguments)]fn classify_region(lines: &[ProjectedLine],interruptions: Vec<(f32, Interruption)>,@@ -280,6 +313,7 @@ fn classify_region(image_mode: crate::config::ImageMode,toc_page: bool,debug: bool,+ precomputed_tables: Option<Vec<super::tables::TableRun>>,) -> Vec<Block> {let _ = image_mode; // interruption emission already gated upstreamlet mut blocks: Vec<Block> = Vec::new();@@ -315,7 +349,7 @@ fn classify_region(// objects are page-coordinate; the detector intersects them against the// sub-list's line bboxes anyway.let ruled_runs = detect_ruled_tables(lines, &page.graphics, page.page_width, page.page_height);- let borderless_runs = detect_tables(lines);+ let borderless_runs = precomputed_tables.unwrap_or_else(|| detect_tables(lines));let table_runs = merge_table_runs(ruled_runs, borderless_runs);const TABLE_HR_SUPPRESS_HEADROOM_ROWS: f32 = 4.0;diff --git a/crates/liteparse/src/markdown_layout/mod.rs b/crates/liteparse/src/markdown_layout/mod.rsindex 6064479..ec5bf09 100644--- a/crates/liteparse/src/markdown_layout/mod.rs+++ b/crates/liteparse/src/markdown_layout/mod.rs@@ -8,6 +8,7 @@mod blocks;mod classify;+mod cross_region;mod headings;mod hr;mod inline;