Skip to content

A fast, helpful, and open-source document parser

wip

Logan Markewichb2459acab58f
diff --git a/MARKDOWN_PROGRESS.md b/MARKDOWN_PROGRESS.md
index c23031c..056a698 100644
--- a/MARKDOWN_PROGRESS.md
+++ b/MARKDOWN_PROGRESS.md
@@ -2,6 +2,21 @@
Tracks build-order progress against [MARKDOWN_PLAN.md](MARKDOWN_PLAN.md). Update as steps complete.
+> **2026-06-10 cross-region table re-merge — opendataloader Overall 0.8602 → 0.8633 (TEDS 0.6514 → 0.6716, MHS 0.7807 → 0.7865, NID +0.0006); ParseBench table grits_con 0.5188 → 0.5481 (+0.029 A/B-attributed), TRM 0.1978 → 0.2083, composite 0.3682 → 0.3886; olmOCR table_tests 44.2 → 45.5% (+13 tests).** Targeted the "V-cut slices a table into sibling leaves" failure (docs 146/64 on opendataloader, the deferred cross-region lever from the 06-09 cluster-merge entry).
+>
+> **New pass: `markdown_layout/cross_region.rs`**, run in `classify_page_with_filters` *before* region grouping (up to 3 merges/page). Pipeline:
+> 1. **Candidate pair**: two leaves either x-disjoint (gap ≥ 8pt) or direct region-tree siblings with x-overlap < 40% of the narrower side (straddle-widened V-cut slices, e.g. Federal Register), overlapping ≥ 50% in y. Extended to all leaves predominantly on one side of the cut inside the pair's **union** y-band (a side may itself be H-cut — 146's Area column is two leaves). Set must tile a contiguous line range.
+> 2. **Gates** (each killed a real misfire): TOC pages skipped (doc 155 — split number/title/page-number leaves fuse into a convincing fake table); **either**-side-prose reject where prose = ≥3 lines, median-width/side-width ≥ 0.7, vertical fill ≥ 0.55, <30% multi-cell lines (doc 156 body+sidebar, doc 183 infographic; the multi-cell clause keeps sliced *wide* tables whose row-halves also fill their side); ≥3 baselines aligned across the cut and ≥ 40% of the sparser side (doc 43 infographic %-labels).
+> 3. **Fuse**: same-baseline lines concatenate spans (cell boundaries fall out of span gaps in `split_cells`), zone gets the cut-parent `region_path`.
+> 4. **Validate or drop**: (A) re-run `detect_tables` on the fused lines — accept if covering runs ≥ 60% of rows AND <30% of cells are ≥5-word "cells" (the long-cell gate killed the newspaper/references-column misfires: olmOCR 20_pg35, 0208fbb5 family — fused prose aligns into convincing grids but its cells are sentence fragments); else (B) direct 2-col path for label/description shapes whose list-marker labels (`1. Embodying…`) `is_label_like` rightly rejects in normal flow — V-cut geometry + aligned rows substitute for the label gate; rows anchored on left-side lines, wrap-joins for labels and descriptions, header promoted when bold/short. Validated runs are threaded into `classify_region` as precomputed borderless runs so emission is guaranteed; a failed candidate leaves the page untouched.
+>
+> Kill-switch `LITEPARSE_DISABLE_CROSS_REGION_TABLES=1`, debug `LITEPARSE_DEBUG_CROSS_REGION=1`. 190 lib tests pass (5 new).
+>
+> **Per-doc:** opendataloader 64 TEDS 0.511 → **1.000** (PORT SHIPCALLS 3-col), 146 TEDS 0.0 → 0.358 + MHS 0.317 → 0.942 + NID +0.055 (Area/Competence emitted exactly); per-doc diff vs 06-09 history: **+2/−0 on every metric** across all 200 docs. olmOCR same-binary A/B: table +13, multi_column −2 net / long_tiny_text flat (the residual "regressions" are pages where a genuine sliced table now emits — e.g. 0208fbb5 Spoil-Attribute ± SE table — trading an order test for table wins), Overall flat 37.6-37.7%.
+>
+> **vs competitors after this session:** ParseBench table composite **0.3886 vs pymupdf4llm 0.3726 — now ahead**; grits_con 0.548 vs 0.499; TRM 0.208 vs 0.221 (last remaining table deficit). opendataloader Overall 0.8633 vs nutrient 0.8851 — gap now NID 0.907 vs 0.925 + MHS 0.787 vs 0.819; TEDS 0.672 vs 0.708.
+>
+> **Residuals / next:** (a) doc 147-style 4-col tables with long wrapped paragraph cells — candidate forms, validation correctly refuses; needs the multi-line-cell row-merge signal (Mode-5). (b) Federal Register `1653739079_page34` — sibling pair now forms but the leaves mix column prose *above* the table band, diluting validation; needs sub-leaf y-banding (merge only the table band, not whole leaves). (c) doc 116 — slices fully overlap in x (item-level histogram partition), outside this geometry. (d) doc 43-family infographics stay rejected (good).
> **2026-06-09 (pm) MHS/heading session — opendataloader Overall 0.8275 → 0.8602 (MHS 0.698 → 0.781, NID 0.890 → 0.907, TEDS flat 0.651); olmOCR table_tests 43.0 → 44.2, multi_column 59.5 → 60.0.** Five changes, descending impact:
>
> 1. **Dropped the `<!-- page N -->` markers from markdown output** (`output/markdown.rs`; `-----` separator kept). The opendataloader MHS evaluator builds a flat heading/content tree (**heading levels are ignored entirely** — `evaluator_heading_level.py` tags nodes only `heading`/`content`); on small trees (GT = 1 heading ≈ 4 nodes) the comment created a pre-heading content node costing ~0.25/doc across ~30 docs. Simulated +0.058 MHS / +0.004 NID before touching code, confirmed by bench. **olmOCR side effect:** headers_footers drops 58.9 → 57.4 — verified by re-prepending the comment to converted outputs (59.3) that the comment was *displacing un-stripped chrome out of the windowed `absent` checks*, i.e. an accidental benchmark hack we no longer benefit from. The honest recovery is better chrome stripping (the known Tel/email/letterhead catalog gaps).
diff --git a/crates/liteparse/src/markdown_layout/classify.rs b/crates/liteparse/src/markdown_layout/classify.rs
index 272a96f..fe3857e 100644
--- a/crates/liteparse/src/markdown_layout/classify.rs
+++ b/crates/liteparse/src/markdown_layout/classify.rs
@@ -145,6 +145,33 @@ pub fn classify_page_with_filters(
&filtered_owned
};
+ // Cross-region table re-merge: when a V-cut sliced a table into sibling
+ // leaves, fuse the slices back into single rows (one synthetic region)
+ // before grouping. Each successful merge carries its own validated table
+ // runs, which are handed to `classify_region` so emission is guaranteed
+ // even for shapes the standard detectors would reject in normal flow.
+ // TOC pages are skipped: their split number/title/page-number leaves
+ // baseline-align perfectly and fuse into a convincing-but-wrong table.
+ let mut cross_merged_owned: Option<Vec<ProjectedLine>> = None;
+ let mut cross_region_runs: Vec<(Vec<u16>, Vec<super::tables::TableRun>)> = Vec::new();
+ for _ in 0..3 {
+ if toc_page {
+ break;
+ }
+ let cur: &[ProjectedLine] = cross_merged_owned.as_deref().unwrap_or(lines);
+ let Some(m) = super::cross_region::find_cross_region_table_merge(cur) else {
+ break;
+ };
+ let mut next: Vec<ProjectedLine> = Vec::with_capacity(cur.len());
+ next.extend_from_slice(&cur[..m.start]);
+ let merged_path = m.merged[0].region_path.clone();
+ next.extend(m.merged);
+ next.extend_from_slice(&cur[m.end..]);
+ cross_region_runs.push((merged_path, m.runs));
+ cross_merged_owned = Some(next);
+ }
+ let lines: &[ProjectedLine] = cross_merged_owned.as_deref().unwrap_or(lines);
+
// Region-grouped pipeline: split the filtered line list into contiguous
// runs sharing a `region_path` (one xy_cut leaf each) and classify each
// leaf as its own mini-page. Paragraph / list / code / heading state is
@@ -239,6 +266,10 @@ pub fn classify_page_with_filters(
}
let region_start = blocks.len();
+ let precomputed_tables = cross_region_runs
+ .iter()
+ .find(|(path, _)| *path == lines[rstart].region_path)
+ .map(|(_, runs)| runs.clone());
let region_blocks = classify_region(
region_lines,
region_interruptions,
@@ -248,6 +279,7 @@ pub fn classify_page_with_filters(
image_mode,
toc_page,
debug,
+ precomputed_tables,
);
if !region_blocks.is_empty() {
region_boundaries.push(region_start);
@@ -271,6 +303,7 @@ pub fn classify_page_with_filters(
/// `stitch_regions` post-pass. Page-level signals (`heading_map`,
/// `struct_nodes`, `outline`, header/footer y-bands) are still consulted
/// because they're computed once per document.
+#[allow(clippy::too_many_arguments)]
fn classify_region(
lines: &[ProjectedLine],
interruptions: Vec<(f32, Interruption)>,
@@ -280,6 +313,7 @@ fn classify_region(
image_mode: crate::config::ImageMode,
toc_page: bool,
debug: bool,
+ precomputed_tables: Option<Vec<super::tables::TableRun>>,
) -> Vec<Block> {
let _ = image_mode; // interruption emission already gated upstream
let mut blocks: Vec<Block> = Vec::new();
@@ -315,7 +349,7 @@ fn classify_region(
// objects are page-coordinate; the detector intersects them against the
// sub-list's line bboxes anyway.
let ruled_runs = detect_ruled_tables(lines, &page.graphics, page.page_width, page.page_height);
- let borderless_runs = detect_tables(lines);
+ let borderless_runs = precomputed_tables.unwrap_or_else(|| detect_tables(lines));
let table_runs = merge_table_runs(ruled_runs, borderless_runs);
const TABLE_HR_SUPPRESS_HEADROOM_ROWS: f32 = 4.0;
diff --git a/crates/liteparse/src/markdown_layout/mod.rs b/crates/liteparse/src/markdown_layout/mod.rs
index 6064479..ec5bf09 100644
--- a/crates/liteparse/src/markdown_layout/mod.rs
+++ b/crates/liteparse/src/markdown_layout/mod.rs
@@ -8,6 +8,7 @@
mod blocks;
mod classify;
+mod cross_region;
mod headings;
mod hr;
mod inline;