Skip to content

A fast, helpful, and open-source document parser

wip

Logan Markewich6c6dc731c75a
diff --git a/MARKDOWN_PROGRESS.md b/MARKDOWN_PROGRESS.md
index 1d88c6e..105055e 100644
--- a/MARKDOWN_PROGRESS.md
+++ b/MARKDOWN_PROGRESS.md
@@ -2,6 +2,22 @@
Tracks build-order progress against [MARKDOWN_PLAN.md](MARKDOWN_PLAN.md). Update as steps complete.
+> **2026-06-12 (pm) per-font empirical space-width recovery (Mode 14 word fusion, no-metric fonts) — olmOCR long_tiny_text 25.3 → 25.8% (+2 tests, clean, zero regressions); multi_column / table_tests / ParseBench BYTE-IDENTICAL; opendataloader flat (NID/TEDS/MHS unchanged); 204 lib tests pass.** Targeted the 2026-06-11 audit's Tier-1 #2 dense/small-font word fusion (22.6% of `long_tiny_text` files), reproduced on `long_tiny_text/20_pg58` (`theynexttakeonthe`, `againstManchester`, `transferbusiness`, `forSpurs` — body rendered as word-soup).
+>
+> **Root cause (Layer 1, `extract.rs`).** The missing-space recovery in the no-pending-space branch uses `thresh = 0.7 * space_w` when the font exposes an ASCII-space advance (`ch.font_space_width()`), else a fixed `0.35 * em_vp` fallback. `20_pg58`'s body font exposes **no space metric** (`sw=-1` in `LITEPARSE_DEBUG_GAPS`), so every boundary hit the fallback — which computes ~3pt for an 8.5pt font while the real inter-word gap is ~1pt, so nothing ever fired. Histogramming `loose_gap` by the PDFium `ps` flag showed the gaps are cleanly **bimodal**: intra-word kerning loose ≤ 0.4pt (434 events), genuine spaces 0.8–1.4pt (34 events), near-empty valley between — and the boundaries where PDFium *did* emit a space (`ps=1`) land in that exact 0.8–1.4 band, giving a free per-font calibration signal.
+>
+> **Fix.** Recover the missing `space_w` empirically: in the pending-space commit branch, sample `loose_gap / em_vp` at alpha-alpha genuine-space boundaries into a `HashMap<font_name, Vec<f32>>`; in the no-metric `ps=0` branch, when the font has ≥ `MIN_SPACE_CAL_SAMPLES` (6) samples, set `thresh = 0.7 * median(ratios) * em_vp` — i.e. feed the recovered width through the **same 0.7 factor the metric path uses**. Falls back to `0.35 * em_vp` otherwise. Single-pass running accumulator (dense docs accumulate 6 samples within the first line or two). New `median_f32` helper. Kill-switch `LITEPARSE_NO_SPACE_CAL=1`.
+>
+> **The load-bearing safety property.** Calibration activates **only on no-space-metric fonts**, so the metric-bearing fonts that dominate multi_column / tables — the exact set that produced the documented `+179`-mirage catastrophe (multi_column −308 / tables −187 from a global gap constant) — are byte-untouched. Verified: olmOCR multi_column / table_tests **0/231 and 0/188 docs changed**; ParseBench 0/125 sampled docs changed; opendataloader flat. The old regression came from a *global* constant; a *per-font* threshold confined to no-metric fonts is safe.
+>
+> **A/B method:** snapshotted the pre-change binary (`target/release/lit-base-wf`), byte-diffed all olmOCR categories to bound blast radius (changes confined to long_tiny_text 10/62 + headers_footers 1/266), then scored a same-era stored baseline candidate (still contains the fusions, multi_column/table_tests byte-match) against the new `liteparse_spacecal` candidate. long_tiny_text FAIL-set diff = clean **+2 / −0** (newly-passing `20_pg48`, `20_pg58`). The +2 understates the win: olmOCR's char-exact anchors undercount readability because these dense docs fail on compounding reasons (column interleave, glyph soup) too — the qualitative output went from word-soup to clean prose.
+>
+> **Two follow-up levers investigated and empirically KILLED (so the next session doesn't re-chase them):**
+> 1. **Pre-scan to calibrate early-page boundaries** — probed by lowering `MIN_SPACE_CAL_SAMPLES` 6→3 (faster warm-up). olmOCR long_tiny_text **unchanged** (114/442 both), and min=3 began bleeding into 3 metric-corpus docs (strictly worse). Warm-up is not the bottleneck → a pre-scan gains ~nothing. Kept 6.
+> 2. **Layer-2 projection silent-merge** (the suspected `ofTurkey` cause) — instrumented the step-3 silent merge in `build_projected_lines`: **0 alpha-alpha events across all 200 opendataloader docs**. The 2026-06-08 alpha-alpha guard already closed it. `build_one_line` (~4636) *always* inserts a space between separate items, so any fused token is a SINGLE extract item → **`ofTurkey` is Layer-1 compressed-metric**, not Layer-2: a metric-bearing font whose typeset space is squeezed below `0.7×sw`, where intra-word kerning *overlaps* the genuine-space band (no clean valley). That class is the high-risk −308/−187 danger zone and is deliberately left untouched.
+>
+> **Takeaway:** word fusion is now exhausted in the safe (no-metric) regime; the only sizable remaining territory is high-risk compressed-metric fonts (no clean geometric discriminator). Next step is **semantic formatting** (per-span bold/italic + heading-level accuracy → ParseBench text_formatting, currently bold 0.53 / italic 0.38 / title_hierarchy 0.37), the Tier-3 fidelity lever from the 2026-06-11 audit.
+>
> **2026-06-12 decimal-fragment merge + body-seeded inferred tables — opendataloader Overall 0.8710 → 0.8719 (TEDS 0.6877 → 0.6929 (+0.0052), NID 0.9120 → 0.9125, MHS flat 0.8036); ParseBench table grits_con 0.5647 → 0.5737 (+0.0090), composite 0.4040 → 0.4079, TRM 0.2213 → 0.2196 (−0.0016); olmOCR table_tests 47.3 → 48.7% (+1.4pp), multi_column 63.6% (noise).** Targeted the 2026-06-11 audit's Tier-1 #3 "table label-column drop" content-loss defect, reproduced on doc `01030000000082` Table 29 (state names Gujarat/Punjab/… silently dropped, table emitted 3-col instead of 4-col). Two compounding root causes, both in the markdown/projection layers (the names extract fine — verified in the `--format text` grid):
>
> 1. **Decimal fragmentation (`projection.rs` word-merge pass).** PDFium emits `15.6` / `200.4` as two x-contiguous items `"15"`+`".6"`; `looks_like_table_number` flags both, so the `both_are_numbers` guard kept them apart, leaving `15 .6` / `21 .3%` in rendered cells *and* injecting spurious close-spaced column tracks. **Fix:** a decimal-continuation carve-out — when the next run begins with `.`+digit and butts directly against a digit-terminated run (`delta_x ∈ [−2, 1]`, y-compatible), merge silently. A real pair of adjacent table numbers always shows a visible inter-cell gap, never a digit against a leading-decimal run, so this is safe. Clean win on its own (fixes the `X .Y` artifact across many numeric/financial docs; NID +0.0005, zero regressions).
diff --git a/crates/liteparse/src/extract.rs b/crates/liteparse/src/extract.rs
index 070a493..2ee37e6 100644
--- a/crates/liteparse/src/extract.rs
+++ b/crates/liteparse/src/extract.rs
@@ -409,6 +409,13 @@ fn extract_page_text_items(
let debug = std::env::var("LITEPARSE_DEBUG").is_ok();
let dbg_gaps = std::env::var("LITEPARSE_DEBUG_GAPS").is_ok();
let space_fix = std::env::var("LITEPARSE_NO_SPACE_FIX").is_err();
+ // Empirical per-font space calibration: for fonts that expose no
+ // space-glyph metric, recover the genuine inter-word gap from the spaces
+ // PDFium *does* emit for that font (normalized by rendered em height) and
+ // feed it through the same threshold rule the metric path uses.
+ let space_cal = std::env::var("LITEPARSE_NO_SPACE_CAL").is_err();
+ let mut font_space_cal: std::collections::HashMap<String, Vec<f32>> =
+ std::collections::HashMap::new();
// Pre-scan: check if ALL text on this page is invisible (render mode 3).
// Some scanned PDFs have an invisible OCR text layer as the only text.
@@ -629,6 +636,28 @@ fn extract_page_text_items(
seg.start(c, &vp_loose, &vp_strict, &ch, page_rotation);
seg.append_ligature_tail(ligature_tail);
} else {
+ // Genuine inline space PDFium emitted: sample its size
+ // (loose gap / em height) per font, alpha-alpha only, to
+ // calibrate the no-space-metric recovery below.
+ if space_cal {
+ if let Some(fk) = seg.font_name.as_ref() {
+ let prev_alnum = seg
+ .text
+ .chars()
+ .last()
+ .is_some_and(|p| p.is_ascii_alphanumeric());
+ if prev_alnum && c.is_ascii_alphanumeric() {
+ let em_vp = (vp_loose.bottom - vp_loose.top).abs();
+ let loose_gap = vp_strict.left - seg.last_char_loose_right;
+ if em_vp > 0.0 && loose_gap > 0.0 {
+ let s = font_space_cal.entry(fk.clone()).or_default();
+ if s.len() < 512 {
+ s.push(loose_gap / em_vp);
+ }
+ }
+ }
+ }
+ }
seg.commit_pending_space();
seg.push_char(c, &vp_loose, &vp_strict, &ch);
seg.append_ligature_tail(ligature_tail);
@@ -655,7 +684,22 @@ fn extract_page_text_items(
let thresh = if space_w > 0.0 {
0.7 * space_w
} else {
- 0.35 * em_vp
+ // No space-glyph metric. Prefer an empirically-recovered
+ // space width (median genuine-space ratio for this font ×
+ // em height) run through the same 0.7 factor as the metric
+ // path; fall back to a fixed em fraction when we lack
+ // enough samples for the font.
+ let calibrated = if space_cal {
+ seg.font_name
+ .as_ref()
+ .and_then(|fk| font_space_cal.get(fk))
+ .filter(|s| s.len() >= MIN_SPACE_CAL_SAMPLES)
+ .and_then(|s| median_f32(s))
+ .map(|ratio| 0.7 * ratio * em_vp)
+ } else {
+ None
+ };
+ calibrated.unwrap_or(0.35 * em_vp)
};
if space_fix && both_alnum && thresh > 0.0 && loose_gap > thresh {
seg.text.push(' ');
@@ -875,6 +919,24 @@ fn decompose_scale(m: &pdfium::Matrix) -> (f32, f32) {
(sx as f32, sy as f32)
}
+/// Minimum genuine-space samples required before trusting per-font calibration.
+const MIN_SPACE_CAL_SAMPLES: usize = 6;
+
+/// Median of a slice of finite, non-negative f32 values. Returns None if empty.
+fn median_f32(values: &[f32]) -> Option<f32> {
+ if values.is_empty() {
+ return None;
+ }
+ let mut v: Vec<f32> = values.to_vec();
+ v.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));
+ let mid = v.len() / 2;
+ if v.len() % 2 == 0 {
+ Some((v[mid - 1] + v[mid]) / 2.0)
+ } else {
+ Some(v[mid])
+ }
+}
+
/// Check if a font is "buggy" based on its name and type.
/// Mirrors ParseFont_isBuggyFont from the platform.
fn is_buggy_font(font_name: &str, font_type: FontType) -> bool {