Skip to content

A fast, helpful, and open-source document parser

wip

Logan Markewichbbe6427f829d
diff --git a/MARKDOWN_PROGRESS.md b/MARKDOWN_PROGRESS.md
index 056a698..844fdd8 100644
--- a/MARKDOWN_PROGRESS.md
+++ b/MARKDOWN_PROGRESS.md
@@ -2,6 +2,22 @@
Tracks build-order progress against [MARKDOWN_PLAN.md](MARKDOWN_PLAN.md). Update as steps complete.
+> **2026-06-10 (pm) tight-gutter column rescue + XY depth/fill-gate session — opendataloader Overall 0.8633 → 0.8694 (NID 0.9071 → 0.9115, MHS 0.7865 → 0.8032, TEDS 0.6716 → 0.6678); olmOCR Overall 37.7 → 38.4% (multi_column 60.0 → 63.8%, table_tests 45.5 → 46.7% — now ahead of pymupdf4llm's 46.2); ParseBench table grits_con 0.5481 → 0.5507, TRM 0.2083 → 0.2123, composite 0.3886 → 0.3922.** Targeted the docs 38/189 "bold heading fused inline across an unsplit 2-col layout" family — the head of the nutrient MHS/NID gap (38/189 alone were MHS −0.97/−0.94 vs nutrient).
+>
+> Five changes, all `projection.rs::xy_find_column_cut` + recursion, each motivated by a measured geometry:
+>
+> 1. **Near-gap line-start rescue.** Doc 38's gutter is 8.7–9.4pt vs the strict `gutter_gap = median_h.max(4.0) = 10pt` threshold (word gaps 4.0–4.8pt), so 6 of 8 right-column lines never registered as line-starts and the right peak died on `min_per_peak`. Starts with gap in `[0.6×, 1.0×] gutter_gap` now count toward the histogram **iff exactly aligned (±1.5pt) with ≥2 strict starts** — a column edge aligns sub-point across bands; justified-text "rivers" jitter. The first cut used a loose ±1-bucket match and manufactured a fake second column inside doc 80's large-font narrow justified prose (NID 0.986 → 0.626); the exact-alignment requirement restored 80 byte-identically while keeping 38. Kill-switch `LITEPARSE_DISABLE_NEARSTART_RESCUE=1`.
+> 2. **Histogram V-cut placed at the actual gutter, not midway between peak centers.** Peaks mark column *left* edges, so the center midpoint (e.g. x=212 between peaks 97/327) lands inside the left column's text and `partition_by_item` misroutes mid-line word clusters (doc 38's "October 2020, 17% said" jumped columns). Boundary = right column's leftmost *counted start* — NOT the bucket-quantized peak center: doc 64's right-aligned numerals start at 178.5 with peak center ~203, and a center-anchored cut at 185.7 routed "2454" left, made the leaves x-overlap, and killed the cross-region candidate (64 TEDS 1.000 → 0.533). With the min-start boundary 64 is byte-identical to baseline. Kill-switch `LITEPARSE_DISABLE_GUTTER_CUT=1`.
+> 3. **`XY_MAX_DEPTH` 6 → 12.** Doc 189 (SOLAR paper): five stacked table/caption bands each burn a banner-peel level, so the 2-col body leafed out *un-split at the cap* (trace just stops at d=5 COMMIT). The cap is a backstop, not a gate — every extra level still passes all cut gates.
+> 4. **Fill gate extended to 2-peak cuts** (was >2-peaks only). With deeper recursion, 2-col label/value tables started reaching the histogram path and getting sliced (doc 166's "Mineral or colloid type | CEC" table → two leaves, TEDS 0.849 → 0). The min-fill hard floor stays **n>2 only**: it exists for ≥3-col dilution, and at n=2 a sparse second column of two drags the weighted average itself — keeping the floor at n=2 false-positived on doc 157's short second prose column (NID 0.951 → 0.429) because the tabular flag also suppresses the density V.
+> 5. **Fill-binning bucket slack.** Items bin to slot c+1 at `x ≥ peaks[c+1].0 − bucket_pt` (was `≥ center`): peak centers are bucket-quantized, so doc 166's right column at x=167.3 vs center 169 binned entirely into slot 0 (`band_counts=[7,0]`) and the gate never saw the sparse column.
+>
+> **Per-doc:** opendataloader 38 MHS 0 → 0.887 + NID +0.152 (the `# 6.2. Expectations for Re-Hiring Employees` page now reads as clean 2-col prose with the heading isolated), 189 MHS 0 → 0.958 + NID +0.192, 15 NID +0.190, 37/36/185 MHS +0.10–0.12; 166/64/80/157/39 all restored byte-identical-or-better after the refinements above. **Accepted residuals (net −0.5 vs +4.8 wins):** docs 200/147/182 — slide-style feature tables with long wrapped paragraph cells where the baseline's *scrambled* interleaved table happened to earn TEDS partial credit and the new clean column prose doesn't (147 NID +0.085 while TEDS −0.165). That family is genuinely ambiguous with N-col prose at the projection layer; it's the known Mode-5 multi-line-cell lever, not a gating bug.
+>
+> **vs nutrient: Overall gap −0.0218 → −0.0157 (0.8694 vs 0.8851); NID 0.9115 vs 0.925, MHS 0.8032 vs 0.819, TEDS 0.6678 vs 0.708.** olmOCR multi_column gap to pymupdf4llm −7.2 → −2.9pp; olmOCR table_tests 46.7% now ahead of pymupdf4llm 46.2%. ParseBench TRM gap to pymupdf4llm −0.013 → −0.008 (0.2123 vs 0.2205). 190 lib tests pass.
+>
+> **Triage workflow note:** the opendataloader `history/<date>` archive is keyed by date — a same-day re-run **clobbers the morning's archive**, so the cross-region baseline had to be reconstructed from a stash-built binary (`git stash push -- projection.rs` → build `lit-base` → pop). A/B per-doc against a baseline *binary*, not the history archive, when iterating same-day.
+
> **2026-06-10 cross-region table re-merge — opendataloader Overall 0.8602 → 0.8633 (TEDS 0.6514 → 0.6716, MHS 0.7807 → 0.7865, NID +0.0006); ParseBench table grits_con 0.5188 → 0.5481 (+0.029 A/B-attributed), TRM 0.1978 → 0.2083, composite 0.3682 → 0.3886; olmOCR table_tests 44.2 → 45.5% (+13 tests).** Targeted the "V-cut slices a table into sibling leaves" failure (docs 146/64 on opendataloader, the deferred cross-region lever from the 06-09 cluster-merge entry).
>
> **New pass: `markdown_layout/cross_region.rs`**, run in `classify_page_with_filters` *before* region grouping (up to 3 merges/page). Pipeline:
diff --git a/crates/liteparse/src/projection.rs b/crates/liteparse/src/projection.rs
index 9d7c507..8f54601 100644
--- a/crates/liteparse/src/projection.rs
+++ b/crates/liteparse/src/projection.rs
@@ -2929,7 +2929,11 @@ const XY_MIN_V_VALLEY_PT: f32 = 8.0;
/// the median line height.
const XY_MIN_H_VALLEY_FACTOR: f32 = 1.4;
/// Hard recursion cap.
-const XY_MAX_DEPTH: u32 = 6;
+// Backstop only — every cut still passes the banner/density/histogram gates.
+// 6 starved pages with several stacked floats (tables + captions above a
+// 2-col body): each banner peel burns a level, and the body band leafs out
+// un-split at the cap.
+const XY_MAX_DEPTH: u32 = 12;
/// Below this item count we stop trying to cut.
const XY_MIN_ITEMS_TO_CUT: usize = 4;
/// A horizontal cut must leave at least this many distinct text lines on each
@@ -3343,6 +3347,14 @@ fn xy_find_column_cut(
});
let mut line_starts: Vec<f32> = Vec::with_capacity(idxs.len() / 2);
+ // Items whose gap to the previous item falls just under `gutter_gap` but
+ // clearly above word-gap range. A tight column gutter (e.g. ~9pt at
+ // median_h=10) fails the strict test on justified lines whose left column
+ // runs right up to the gutter; these get a second chance below if their x
+ // aligns with a column peak that has strict evidence.
+ let mut near_starts: Vec<f32> = Vec::new();
+ let near_gap_floor = gutter_gap * 0.6;
+ let disable_rescue = std::env::var("LITEPARSE_DISABLE_NEARSTART_RESCUE").is_ok();
let mut k = 0;
while k < sorted.len() {
let band_y = items[sorted[k]].item.y;
@@ -3355,8 +3367,13 @@ fn xy_find_column_cut(
let mut prev_right = f32::NEG_INFINITY;
for &idx in &band_items {
let it = &items[idx].item;
- if it.x.is_finite() && it.x - prev_right > gutter_gap {
- line_starts.push(it.x);
+ if it.x.is_finite() {
+ let gap = it.x - prev_right;
+ if gap > gutter_gap {
+ line_starts.push(it.x);
+ } else if gap > near_gap_floor {
+ near_starts.push(it.x);
+ }
}
let right = it.x + it.width.max(0.0);
if right > prev_right {
@@ -3401,6 +3418,39 @@ fn xy_find_column_cut(
}
}
+ // Peak-aligned rescue for tight gutters: a near-gap start only counts
+ // when it lands at (nearly) the same x as existing strict-evidence
+ // starts. A column left edge aligns to sub-point precision across
+ // bands; word-gap "rivers" in justified text jitter by several points —
+ // a looser bucket-width match here let rivers stack onto stray strict
+ // starts and manufacture a fake second column inside narrow justified
+ // prose. Require near-exact agreement with at least 2 strict starts.
+ let mut rescued = 0usize;
+ let mut counted_starts: Vec<f32> = line_starts.clone();
+ if !near_starts.is_empty() && !disable_rescue {
+ const RESCUE_ALIGN_TOL_PT: f32 = 1.5;
+ for &x in &near_starts {
+ let support = line_starts
+ .iter()
+ .filter(|&&s| (s - x).abs() <= RESCUE_ALIGN_TOL_PT)
+ .count();
+ if support >= 2 {
+ let b = (((x - min_x) / bucket_pt).floor()).max(0.0) as usize;
+ if b < n_buckets {
+ hist[b] += 1;
+ rescued += 1;
+ counted_starts.push(x);
+ }
+ }
+ }
+ if dbg && rescued > 0 {
+ eprintln!(
+ "[xy col-rescue] {rescued} near-gap starts (gap in [{near_gap_floor:.1},{gutter_gap:.1}]) exactly aligned to >=2 strict starts"
+ );
+ }
+ }
+ let n_starts = line_starts.len() + rescued;
+
// Cluster adjacent occupied buckets into peaks. A peak is a contiguous
// run of buckets with ≥1 count, merged through gaps of ≤2 zero buckets
// (~10pt) to absorb slight intra-column drift.
@@ -3446,7 +3496,7 @@ fn xy_find_column_cut(
// out the tiny indent peaks, but a 2-column region with ~5 lines/peak now
// qualifies. Capped by the original `XY_COLUMN_MIN_LINES_PER_PEAK` to
// preserve behavior on large regions.
- let adaptive_min = (line_starts.len() / 6).clamp(4, XY_COLUMN_MIN_LINES_PER_PEAK);
+ let adaptive_min = (n_starts / 6).clamp(4, XY_COLUMN_MIN_LINES_PER_PEAK);
peaks.retain(|(_, c)| *c >= adaptive_min);
if peaks.len() < 2 {
if dbg {
@@ -3502,113 +3552,6 @@ fn xy_find_column_cut(
if peaks.len() < 2 {
reject!("peaks>2 but <2 balanced (largest={largest})");
}
- // Fill-density gate: prose columns fill each inter-peak slot
- // nearly edge-to-edge; table cells leave wide whitespace. Per
- // band, measure each column's covered span / slot width; if the
- // least-filled column is too sparse, treat as a table and bail.
- let n = peaks.len();
- let mut covered_sum = vec![0.0f32; n];
- let mut band_count = vec![0usize; n];
- let mut kk = 0;
- while kk < sorted.len() {
- let by = items[sorted[kk]].item.y;
- let mut jj = kk;
- while jj < sorted.len() && (items[sorted[jj]].item.y - by).abs() <= band_tol {
- jj += 1;
- }
- let mut lo_x = vec![f32::INFINITY; n];
- let mut hi_x = vec![f32::NEG_INFINITY; n];
- for &idx in &sorted[kk..jj] {
- let it = &items[idx].item;
- if !it.x.is_finite() {
- continue;
- }
- let mut c = 0;
- while c + 1 < n && it.x >= peaks[c + 1].0 {
- c += 1;
- }
- lo_x[c] = lo_x[c].min(it.x);
- hi_x[c] = hi_x[c].max(it.x + it.width.max(0.0));
- }
- for c in 0..n {
- if hi_x[c] > lo_x[c] {
- let col_w = if c + 1 < n {
- peaks[c + 1].0 - peaks[c].0
- } else {
- max_x - peaks[c].0
- };
- if col_w > 1.0 {
- covered_sum[c] += ((hi_x[c] - lo_x[c]) / col_w).min(1.0);
- band_count[c] += 1;
- }
- }
- }
- kk = jj;
- }
- // Two-part gate that distinguishes prose columns from tables:
- //
- // 1. min_fill floor (MIN_FILL_HARD_FLOOR = 0.38) — rejects
- // pages where any column has narrow text in many bands
- // (e.g. table with numeric column). Real tables have
- // at least one column with fill ≤ 0.36 (verified on
- // Coca Cola 10-k page 1, Apple 10-k page 1, BRWS, SERFF,
- // Goldman Sachs, etc.).
- // 2. Band-count-weighted avg_fill ≥ XY_COLUMN_MIN_FILL —
- // catches table layouts where ALL columns are uniformly
- // narrow (no column dominates the weighted average).
- //
- // Both must pass. The min-only gate was too brittle: a real
- // 3-column prose page with one page-bottom column having ~4
- // bands of short text dragged the min below 0.55 even though
- // every meaningful column was clearly prose. The avg-only
- // gate let through tables where the narrow column had enough
- // bands to be informative but its weight was diluted by
- // wider neighboring columns.
- let mut min_fill = f32::INFINITY;
- let mut weighted_sum = 0.0f32;
- let mut weighted_n = 0.0f32;
- let mut per_col_fill: Vec<f32> = Vec::with_capacity(n);
- for c in 0..n {
- if band_count[c] > 0 {
- let f = covered_sum[c] / band_count[c] as f32;
- per_col_fill.push(f);
- min_fill = min_fill.min(f);
- weighted_sum += f * band_count[c] as f32;
- weighted_n += band_count[c] as f32;
- }
- }
- let avg_fill = if weighted_n > 0.0 {
- weighted_sum / weighted_n
- } else {
- f32::INFINITY
- };
- if dbg {
- eprintln!(
- "[xy col-fill] avg_fill={avg_fill:.2} min_fill={min_fill:.2} (gate={XY_COLUMN_MIN_FILL}) per_col=[{}] band_counts=[{}]",
- per_col_fill
- .iter()
- .map(|f| format!("{f:.2}"))
- .collect::<Vec<_>>()
- .join(","),
- band_count
- .iter()
- .map(|c| c.to_string())
- .collect::<Vec<_>>()
- .join(",")
- );
- }
- const MIN_FILL_HARD_FLOOR: f32 = 0.38;
- if min_fill.is_finite() && min_fill < MIN_FILL_HARD_FLOOR {
- *tabular = true;
- reject!("N-column min fill {min_fill:.2} < {MIN_FILL_HARD_FLOOR} (tabular column)");
- }
- if avg_fill.is_finite() && avg_fill < XY_COLUMN_MIN_FILL {
- *tabular = true;
- reject!(
- "N-column avg fill {avg_fill:.2} < {} (tabular)",
- XY_COLUMN_MIN_FILL
- );
- }
if dbg {
eprintln!(
"[xy col-keep-N] kept {} of {} balanced peaks (N-column); xs=[{}]",
@@ -3639,17 +3582,138 @@ fn xy_find_column_cut(
peaks = top2;
}
}
+ // Fill-density gate: prose columns fill each inter-peak slot nearly
+ // edge-to-edge; table cells leave wide whitespace. Applies to ALL peak
+ // counts including 2 — a 2-col label/value table (description list)
+ // produces the same two left-edge peaks as 2-col prose, and slicing it
+ // strands each half in its own leaf where no table detector can see
+ // the rows whole. Per band, measure each column's covered span / slot
+ // width; if too sparse, treat as a table and bail.
+ {
+ let n = peaks.len();
+ let mut covered_sum = vec![0.0f32; n];
+ let mut band_count = vec![0usize; n];
+ let mut kk = 0;
+ while kk < sorted.len() {
+ let by = items[sorted[kk]].item.y;
+ let mut jj = kk;
+ while jj < sorted.len() && (items[sorted[jj]].item.y - by).abs() <= band_tol {
+ jj += 1;
+ }
+ let mut lo_x = vec![f32::INFINITY; n];
+ let mut hi_x = vec![f32::NEG_INFINITY; n];
+ for &idx in &sorted[kk..jj] {
+ let it = &items[idx].item;
+ if !it.x.is_finite() {
+ continue;
+ }
+ // Peak centers are bucket-quantized, so a column's items can
+ // start up to a bucket left of its peak center; without the
+ // slack they all bin into the previous slot and that column
+ // reads as empty (its sparse cells never get fill-checked).
+ let mut c = 0;
+ while c + 1 < n && it.x >= peaks[c + 1].0 - bucket_pt {
+ c += 1;
+ }
+ lo_x[c] = lo_x[c].min(it.x);
+ hi_x[c] = hi_x[c].max(it.x + it.width.max(0.0));
+ }
+ for c in 0..n {
+ if hi_x[c] > lo_x[c] {
+ let col_w = if c + 1 < n {
+ peaks[c + 1].0 - peaks[c].0
+ } else {
+ max_x - peaks[c].0
+ };
+ if col_w > 1.0 {
+ covered_sum[c] += ((hi_x[c] - lo_x[c]) / col_w).min(1.0);
+ band_count[c] += 1;
+ }
+ }
+ }
+ kk = jj;
+ }
+ // Two-part gate that distinguishes prose columns from tables:
+ //
+ // 1. min_fill floor (MIN_FILL_HARD_FLOOR = 0.38) — rejects
+ // pages where any column has narrow text in many bands
+ // (e.g. table with numeric column). Real tables have
+ // at least one column with fill ≤ 0.36 (verified on
+ // Coca Cola 10-k page 1, Apple 10-k page 1, BRWS, SERFF,
+ // Goldman Sachs, etc.).
+ // 2. Band-count-weighted avg_fill ≥ XY_COLUMN_MIN_FILL —
+ // catches table layouts where ALL columns are uniformly
+ // narrow (no column dominates the weighted average).
+ //
+ // Both must pass. The min-only gate was too brittle: a real
+ // 3-column prose page with one page-bottom column having ~4
+ // bands of short text dragged the min below 0.55 even though
+ // every meaningful column was clearly prose. The avg-only
+ // gate let through tables where the narrow column had enough
+ // bands to be informative but its weight was diluted by
+ // wider neighboring columns.
+ let mut min_fill = f32::INFINITY;
+ let mut weighted_sum = 0.0f32;
+ let mut weighted_n = 0.0f32;
+ let mut per_col_fill: Vec<f32> = Vec::with_capacity(n);
+ for c in 0..n {
+ if band_count[c] > 0 {
+ let f = covered_sum[c] / band_count[c] as f32;
+ per_col_fill.push(f);
+ min_fill = min_fill.min(f);
+ weighted_sum += f * band_count[c] as f32;
+ weighted_n += band_count[c] as f32;
+ }
+ }
+ let avg_fill = if weighted_n > 0.0 {
+ weighted_sum / weighted_n
+ } else {
+ f32::INFINITY
+ };
+ if dbg {
+ eprintln!(
+ "[xy col-fill] avg_fill={avg_fill:.2} min_fill={min_fill:.2} (gate={XY_COLUMN_MIN_FILL}) per_col=[{}] band_counts=[{}]",
+ per_col_fill
+ .iter()
+ .map(|f| format!("{f:.2}"))
+ .collect::<Vec<_>>()
+ .join(","),
+ band_count
+ .iter()
+ .map(|c| c.to_string())
+ .collect::<Vec<_>>()
+ .join(",")
+ );
+ }
+ const MIN_FILL_HARD_FLOOR: f32 = 0.38;
+ // The min-floor catches ≥3-col tables whose narrow numeric column is
+ // diluted out of the weighted average by wide neighbors. With only 2
+ // columns no dilution is possible — a sparse column of two drags the
+ // average itself — and a short second prose column (a page-bottom
+ // column with a few bands) would false-positive here, suppressing a
+ // needed column split AND the density V via the tabular flag.
+ if n > 2 && min_fill.is_finite() && min_fill < MIN_FILL_HARD_FLOOR {
+ *tabular = true;
+ reject!("N-column min fill {min_fill:.2} < {MIN_FILL_HARD_FLOOR} (tabular column)");
+ }
+ if avg_fill.is_finite() && avg_fill < XY_COLUMN_MIN_FILL {
+ *tabular = true;
+ reject!(
+ "N-column avg fill {avg_fill:.2} < {} (tabular)",
+ XY_COLUMN_MIN_FILL
+ );
+ }
+ }
// Require the two peaks to together dominate the histogram — a real
// 2-column layout has the great majority of items starting at one of
// the two column-left edges. Tables, lists, and prose with scattered
// indents distribute their item-starts across many x's and won't pass.
let peak_sum: usize = peaks.iter().map(|(_, c)| *c).sum();
- let dominance = (peak_sum as f32) / (line_starts.len() as f32);
+ let dominance = (peak_sum as f32) / (n_starts as f32);
if dominance < XY_COLUMN_PEAK_DOMINANCE {
reject!(
- "dominance {dominance:.2} < {} (peak_sum={peak_sum} line_starts={})",
- XY_COLUMN_PEAK_DOMINANCE,
- line_starts.len()
+ "dominance {dominance:.2} < {} (peak_sum={peak_sum} line_starts={n_starts})",
+ XY_COLUMN_PEAK_DOMINANCE
);
}
// Require the peaks to be roughly balanced in size. A real multi-column
@@ -3699,7 +3763,43 @@ fn xy_find_column_cut(
peaks[1].0 - peaks[0].0
),
};
- let cut_x = (peaks[li].0 + peaks[li + 1].0) * 0.5;
+ // Place the cut at the actual gutter, not midway between peak centers.
+ // Peaks mark column LEFT edges, so the center midpoint lands inside the
+ // left column's text span and misroutes its mid-line items under
+ // partition_by_item. The gutter spans from the rightmost left-side item
+ // extent to the right column's leftmost counted start — NOT the peak
+ // center: a right-aligned numeric column scatters its starts left of
+ // the bucket-quantized center, and a cut placed at the center would
+ // route those values into the left column.
+ let mid_centers = (peaks[li].0 + peaks[li + 1].0) * 0.5;
+ let right_min_start = counted_starts
+ .iter()
+ .copied()
+ .filter(|&x| x > mid_centers)
+ .fold(f32::INFINITY, f32::min);
+ let boundary = if right_min_start.is_finite() {
+ right_min_start
+ } else {
+ peaks[li + 1].0
+ };
+ let mut gutter_left = f32::NEG_INFINITY;
+ for &i in idxs {
+ let it = &items[i].item;
+ if it.x.is_finite() {
+ let r = it.x + it.width.max(0.0);
+ if r <= boundary - 1.0 && r > gutter_left {
+ gutter_left = r;
+ }
+ }
+ }
+ let cut_x = if gutter_left.is_finite()
+ && gutter_left > peaks[li].0
+ && std::env::var("LITEPARSE_DISABLE_GUTTER_CUT").is_err()
+ {
+ (gutter_left + boundary) * 0.5
+ } else {
+ mid_centers
+ };
// Validate cut lands inside the bbox; if bbox-clamping moved the bbox
// off the items (some PDFs have content with negative x), fall back to
// an item-relative midpoint check.