A fast, helpful, and open-source document parser
wip
Logan Markewich9352dfbeb4ad
diff --git a/MARKDOWN_PLAN.md b/MARKDOWN_PLAN.mddeleted file mode 100644index a7a3677..0000000--- a/MARKDOWN_PLAN.md+++ /dev/null@@ -1,305 +0,0 @@-# Markdown Output for LiteParse — Design Doc--Status: draft-Owner: @loganmarkewich-Last updated: 2026-05-29--## Goal--Add `OutputFormat::Markdown` to LiteParse — a 100% heuristic, dependency-light markdown emitter that produces output suitable for LLM ingestion (RAG, summarization, chunking).--Two non-negotiables:--1. **No new runtime deps.** Pure Rust, runs in Node / Python / WASM with the same code path. Our forked PDFium already ships in all three.-2. **Honest fallback.** When heuristics aren't confident (especially tables), emit the grid-projection text inside a fenced block. The structure stays readable even when we can't classify it.--We will lean on:--- The existing grid projection (anchor system, column detection, rotation, OCR merge).-- Signals our PDFium fork can expose that upstream cannot (structure tree, marked content, paths/rects with stroke+fill, image bounds, outline).--## Prior art: pymupdf4llm--Annotated read of `src/helpers/pymupdf_rag.py` in `pymupdf/RAG`. Worth knowing what they do — most of it is sound — and where they fall short.--### What they do well--| Feature | Approach |-|---|---|-| Heading detection | `IdentifyHeaders`: char-weighted font-size histogram. Body = most frequent size. Larger sizes mapped to `#`..`######` in descending order. |-| TOC-based headings | `TocHeaders`: alt path that uses doc outline; matches spans by string prefix to a TOC entry to get an exact level. |-| Inline styling | Per-span PDF font flags (`flags & 2/8/16`, `char_flags & 1/8`) for italic / mono / bold / strikeout. Per-line shortcut if whole line shares the style. |-| Code blocks | Consecutive all-mono lines → fenced block, with synthetic indent `(x0 - clip.x0) / (fontsize * 0.5)`. |-| Lists | `startswith_bullet()` regex on first char; leading-space indent from x-offset. |-| Tables | `page.find_tables(strategy=...)` (PyMuPDF's vector-graphics-based finder). Drop tables with <2 rows or <2 cols. Emit `Table.to_markdown()`. Subtract table rects from text rects so content doesn't double up. |-| Figures | `get_image_info()` + clustered vector graphics. `is_significant()` filter drops H/V-line-only clusters (underlines, dividers) so they don't become bogus figures. |-| Background-color awareness | Detects page bg color; vector paths filled with bg are ignored. |-| Multi-column | `column_boxes()` — geometry-based column boxes, processed top→bottom, left→right. |-| Annotations | Uses `page.get_links()` to wrap matching spans with markdown link syntax. |-| Header/footer | Fixed user-supplied top/bottom margin in points. |--### Where we can beat them--| Weakness | What we'll do |-|---|---|-| Actively **deletes** `StructTreeRoot` to avoid a perf bug. | Use the structure tree as the highest-priority signal. Our fork can expose it cheaply. |-| Pure Python, slow on large docs. | Native Rust, single-pass over already-projected lines. |-| Borderless tables fall back to nothing useful. | Use grid projection's column detection as a borderless-table detector. When uncertain → fenced projection text. |-| Header/footer = static margin. | Cross-page repetition detection on top/bottom bands. |-| No de-hyphenation. | Conservative join: `word-\nbreak` → `wordbreak` only when next line starts lowercase. |-| Paragraph break = big y-gap only. | Combine y-gap + font change + indent change + anchor change. |-| Reading order from generic column boxes. | We already have anchor-driven projection; reuse it directly. |-| Bold from font flag only (misses synthetic bold). | Combine flag + `font_weight` + font-name substring (`Bold`, `Black`, `Heavy`). |-| Drops invisible text by default. | Keep it gated by config; preserve `confidence` and OCR-vs-native source. |--## Architecture--```-┌─────────────────────────────────────────────────────────────┐-│ PDFium (forked) │-│ text items │ images │ paths │ struct tree │ outline │-└──────────┬──────────────────────────────────────────────────┘- │- extract.rs (existing) ────────────────┐- │ │- projection.rs (existing) ─────────────┤- │ │- ▼ ▼- ┌──────────────┐ ┌────────────────────┐- │ ProjectedLine│ │ PageGeometry │- │ with style │ │ paths, images, │- │ metadata │ │ struct nodes, │- │ (NEW api) │ │ outline targets │- └──────┬───────┘ └─────────┬──────────┘- └───────────┬─────────────────────┘- ▼- ┌──────────────────────┐- │ markdown_layout.rs │ (NEW)- │ block classifier │- └─────────┬────────────┘- ▼- Vec<Block>- ▼- ┌──────────────────────┐- │ output/markdown.rs │ (NEW)- │ block → markdown │- └─────────┬────────────┘- ▼- String-```--### New types--In `types.rs`:--```rust-pub enum Block {- Heading { level: u8, text: String, inline: Vec<Span> },- Paragraph(Vec<Span>),- ListItem { level: u8, ordered: bool, marker: String, inline: Vec<Span> },- Table(TableBlock), // confident table- GridFallback(String), // projection text in ```/<pre>- CodeBlock { lang: Option<String>, body: String },- Figure { image_id: Option<String>, alt: Option<String>, ocr_text: Option<String> },- HorizontalRule,- Raw(String), // escape hatch-}--pub struct Span {- pub text: String,- pub bold: bool,- pub italic: bool,- pub mono: bool,- pub strike: bool,- pub link: Option<String>,-}--pub struct TableBlock {- pub header: Option<Vec<String>>,- pub rows: Vec<Vec<String>>,-}-```--### `projection.rs` — exposing line metadata--Today projection produces a joined string. Add an internal API:--```rust-pub struct ProjectedLine {- pub text: String,- pub bbox: Rect,- pub anchor: Anchor, // Left | Right | Center | Floating- pub indent_x: f32, // from column left edge- pub dominant_font_size: f32,- pub dominant_font_name: Option<String>,- pub all_bold: bool,- pub all_italic: bool,- pub all_mono: bool,- pub all_strike: bool,- pub spans: Vec<TextItem>, // raw items, preserved- pub column_id: usize, // for multi-column reading order- pub mcid: Option<i32>, // marked content id-}-```--The text emitter keeps working as-is by joining these lines; the markdown emitter consumes them structurally. **No behavior change to existing outputs.**--## Block classification (markdown_layout.rs)--Order matters — each pass either consumes lines or hands them to the next.--### Pass 0 — Whole-document signals (once per doc)--- Collect font-size → char-count histogram → `body_size`. For OCR, use height as a proxy.-- Build heading-level map: top N sizes > body_size → H1..HN (N capped at 6).-- Load outline / bookmarks → `Vec<OutlineTarget { page, y, level, title }>`. Not available for OCR docs.-- Walk structure tree → `Vec<StructNode { mcid, role, bbox }>` where role ∈ {H1..H6, P, L, LI, Table, TR, TH, TD, Figure, Caption}. Not available for OCR docs.-- Detect repeating top/bottom-band text across pages → header/footer set.--### Pass 1 — Page-level rect classification--Before touching lines, classify rects:--- **Image rects** — from PDFium image objects (already exposed).-- **Table rects** —- - **Strong**: cluster ruled paths into a grid; verify ≥2 rows × ≥2 cols of cells. Use stroke geometry from the fork.- - **Medium**: column-projection alignment — group of ≥3 vertically-aligned text "columns" repeated across ≥2 horizontal "rows" inside one column box. This works for docs that required OCR.- - **Weak (fallback flag)**: lots of short text snippets in a clearly tabular layout but heuristic 1 & 2 disagree → mark as `GridFallback` candidate.-- **Code-block rects** — fill rect with a near-grey/near-white-non-bg color enclosing mono text (PyMuPDF-style is to skip this; we can do better).-- **HR rects** — long thin horizontal path, width > 50% column width, height ≤ 2pt.-- **Figure rects** — significant vector clusters + image rects, merged (`refine_boxes` analog).--Lines whose bbox falls inside a classified rect get tagged and skipped by Pass 2.--### Pass 2 — Line classification (in reading order)--For each `ProjectedLine` not consumed by a rect:--1. If line bbox is inside structure-tree `Figure`/`Table` node → emit accordingly.-2. If `mcid` maps to a structure node with role `H1..H6` → use that level. **Highest-priority heading signal.**-3. Else if bbox y matches an outline target (within tolerance) and text prefix-matches outline title → use outline level.-4. Else if `dominant_font_size > body_size` → use histogram heading map.-5. Else if `startswith_bullet(text)` or `^\d+[.)]\s` → list item; level from `indent_x` bucketing.-6. Else if `all_mono` → start/extend code block.-7. Else → paragraph line. Group consecutive paragraph lines with: same column, no major y-gap (> 1.5× line height), no font-size change, no anchor change.--De-hyphenation runs during paragraph join: if previous line ends `-` and next line starts lowercase ASCII, drop the hyphen and the newline.--### Pass 3 — Inline styling--Per span inside each emitted block:--- bold = font-flag-bold OR `font_weight ≥ 600` OR font-name contains `Bold|Black|Heavy|Semibold`.-- italic = font-flag-italic OR font-name contains `Italic|Oblique`.-- mono = font-flag-mono OR font-name in known mono list (`Courier`, `Mono`, `Consolas`, `Menlo`, `Source Code`, ...).-- link = bbox overlaps an annotation URI link rect.--Per-line shortcut: if all spans share a style, apply once around the whole line text (matches pymupdf4llm; avoids `**foo** **bar** **baz**` noise).--## Tables in detail--This is the hardest part and where we differentiate. Three modes, configurable via `MarkdownConfig::tables`:--- `Off` — never emit tables; tabular regions become `GridFallback`.-- `Auto` (default) — try strong → medium → fallback in order.-- `RuledOnly` — only emit when path-based grid detection succeeds; otherwise fallback.--### Strong: ruled grid--1. Collect horizontal segments (`y0 ≈ y1`, length > 1pt) and vertical segments inside the page clip.-2. Build a set of grid intersection x's and y's by clustering segment endpoints.-3. Walk cells; assign text items whose centroid falls in each cell.-4. Reject if rows < 2 or cols < 2, or if >30% of cells are empty.-5. First row → header iff its cells are bold or have a fill background.--### Medium: borderless via projection columns--The grid projection already tells us column boundaries (anchors). Inside a column box:--1. Look for ≥3 vertical "tracks" of text where x0 clusters within ±2pt.-2. Look for ≥2 horizontal "rows" of those tracks at distinct y bands.-3. Require row-spacing consistency (CV of gaps < 0.5).-4. If matched, emit as markdown table. Otherwise → `GridFallback`.--### Fallback: fenced projection--```text-\`\`\`-<projection.rs output for this rect, untouched>-\`\`\`-```--This preserves visual structure for the LLM and is **strictly better** than mangled markdown.--## Config--```rust-pub struct MarkdownConfig {- pub headings: HeadingMode, // Auto | StructTree | Outline | FontSize | Off- pub tables: TableMode, // Auto | RuledOnly | Off- pub images: ImageMode, // Embed { dir } | Placeholder | Off- pub code_blocks: bool, // default true- pub strip_headers_footers: bool, // default true- pub dehyphenate: bool, // default true- pub keep_invisible_text: bool, // default false- pub max_heading_levels: u8, // default 6- pub body_size_floor: Option<f32>, // optional manual override-}-```--Sensible defaults so users can just pass `-f markdown` and get good output. Debatable if we should even expose any options at all in v1. We can always add them later if users want more control.--## Surface area--- `crates/liteparse/src/output/markdown.rs` — new.-- `crates/liteparse/src/markdown_layout/` — new module directory: `mod.rs` (public re-exports), `blocks.rs`, `inline.rs`, `tables.rs`, `headings.rs`, `lists.rs`, `paragraphs.rs`, `repetition.rs`, `hr.rs`, `classify.rs`.-- `crates/liteparse/src/types.rs` — add `Block`, `Span`, `TableBlock`.-- `crates/liteparse/src/config.rs` — add `OutputFormat::Markdown`, `MarkdownConfig`.-- `crates/liteparse/src/projection.rs` — expose `ProjectedLine` (additive).-- `crates/liteparse/src/main.rs` — `-f markdown`, plus subflags for the config above.-- `crates/liteparse-napi`, `liteparse-python`, `liteparse-wasm` — pass through the new format + config struct.-- `packages/node/src/lib.ts`, `packages/python/liteparse/parser.py` — wrapper API.--## PDFium fork additions needed (incremental)--Order roughly matches build order below.--1. **Image bbox enumeration** — likely already exposed; confirm.-2. **Path objects** with stroke + fill, in viewport coordinates. Needed for ruled-table detection, HR detection, code-block backgrounds.-3. **Structure tree walk** with role + mcid → bbox map. Highest leverage signal.-4. **Outline / bookmarks** with destination page+y per entry. Cheap.-5. **Page background color** — derivable from full-page fill paths once (2) lands.--Each of these is additive in the FFI; we can ship markdown v1 without (3)/(4) and add them later as quality bumps.--The PDFium fork is already on disk in `~/pdfium-binaries` -- we can directly modify here or the bindings in @crates/pdfium-sys and @crates/pdfium.--## Build order--1. **Wiring**: `OutputFormat::Markdown` end-to-end, returning current text output wrapped in `<pre>` fence. Proves CLI, napi, py, wasm plumbing.-2. **`ProjectedLine` API** in `projection.rs` (additive; text/json output unchanged).-3. **Font-size histogram + heading detection.** Paragraph grouping. De-hyphenation.-4. **Lists** (bullet + ordered + nesting from indent).-5. **Code blocks** (all-mono lines).-6. **Inline styling** (bold/italic/mono/strike/link) with per-line shortcut.-7. **Tables — medium (column-alignment) + fallback.** Ship without fork changes.-8. **Header/footer stripping** (cross-page repetition).-9. **Fork: paths exposed** → tables — strong (ruled grid), HR detection, figure-cluster refinement.-10. **Fork: structure tree + outline** → highest-priority heading + table + figure source. Replaces step 3 when available, falls back when not.-11. **Image extraction modes** (placeholder vs embed).--Steps 1–8 are a usable v1. Steps 9–11 are quality bumps that mostly improve specific document classes (tagged PDFs, ruled tables, image-heavy docs).--## Open questions--- **Image embed format in WASM.** No filesystem — base64 data URI? Return as separate blob array alongside the markdown string? Probably the latter; the markdown references `image_0.png` and the API returns a `images: { name, bytes }[]` sidecar.-- **Per-page vs whole-doc emission.** pymupdf4llm emits per page with `\n\n-----\n\n` separators. Cleaner for chunking but loses paragraphs that span page breaks. Recommend per-page by default with a `joinPages: true` option.-- **Should we expose `Block[]` as a structured output?** It's a natural intermediate. JSON consumers might prefer `Block[]` over a markdown string. Cheap to add (`OutputFormat::Blocks`).-- **Should `GridFallback` use `~~~text` or `<pre>`?** Code fence is friendlier to most LLMs; `<pre>` is friendlier to renderers. Default to fenced.-- **Confidence/OCR provenance in markdown.** Drop it? Encode as HTML comments? Most users won't want it; gate behind a config flag.--## Success criteria--- End-to-end on the eval set in `dataset_eval_utils/` produces markdown that scores ≥ pymupdf4llm on whatever similarity metric we settle on (the existing `*_report.html` runs suggest the harness is already there).-- Single-page parse time stays within 2× of current text output for typical documents.-- No new runtime deps; WASM bundle size growth < 100KB.diff --git a/MARKDOWN_PROGRESS.md b/MARKDOWN_PROGRESS.mddeleted file mode 100644index 2964aae..0000000--- a/MARKDOWN_PROGRESS.md+++ /dev/null@@ -1,788 +0,0 @@-# Markdown Output — Implementation Progress--Tracks build-order progress against [MARKDOWN_PLAN.md](MARKDOWN_PLAN.md). Update as steps complete.--> **2026-06-15 launch benchmark sweep + reproduce script + strikethrough feature (clean) + semantic_formatting gap analysis. 213 lib tests pass (5 new). No regression on any tracked metric.** Three pieces this session: (1) a fresh full model-free sweep across all three benches with the core-4 competitors → `LAUNCH_BENCHMARKS.md`; (2) `run_benchmarks.sh` to reproduce it (with `--liteparse-only` to skip competitors), documented in `BENCHMARKS.md`; (3) strikethrough support, shipped in its clean form after establishing it's a dead benchmark lever.->-> **Launch sweep (fresh, model-free, 2026-06-15; full numbers in `LAUNCH_BENCHMARKS.md`).** liteparse leads the headline metric of all three benches among free tools: olmOCR Overall 39.1% (next-best pymupdf4llm 32.9%), opendataloader #1 open/free on all four (overall 0.875; only the *commercial* nutrient edges it at 0.885), ParseBench #1 Overall + Tables + Content-Faithfulness. Speed 144 ms/doc, 17–27× faster than pymupdf4llm/markitdown/opendataloader. Where we DON'T lead: nutrient (commercial) on opendataloader; olmOCR multi_column (−0.9 tie) and long_tiny_text; ParseBench `semantic_formatting` (pymupdf4llm 0.446 vs 0.409). Re-run gotchas captured in `BENCHMARKS.md` + [[project_competitor_baselines]]: opendataloader needs openjdk@11 (system Java 8 fails) and errors on raw `.png`; ParseBench writes per-group reports to `output/<pipe>/<group>/_evaluation_report.json` (metrics under `aggregate_metrics`); top-level `_evaluation_report.json` is a stale aggregate; `layout`/Visual-Grounding scored over only 64/500 docs (unusable).->-> **Strikethrough (`~~…~~`) — SHIPPED clean; real output-quality win, ZERO benchmark movement (by design).** New: geometric detection in `extract.rs::assign_strikethrough` (a thin horizontal stroke/thin-rect crossing a text item's vertical *middle* band, 0.20–0.65 of height, ≥60% width coverage — cleanly separated from underlines ≈0.93 and overlines <0; calibrated on `text_simple__{strikeout,edits,redlined,strikeUnderline}.pdf`). Plumbed via new `TextItem.strike` → `is_strike_item` → `SpanStyle.strike` → `apply_style` (`~~` wraps outermost, composes with bold/italic/mono). Required a paragraph-path fix: `classify.rs`/`paragraphs.rs` route struck paragraphs through the per-line `inline` renderer instead of the uniform raw-text fast path (which silently dropped the markers — the `uniform` tuple only carries bold/italic). Kill: none (always on, gated on detection).->-> **Why strikethrough is a dead benchmark lever (don't re-chase).** ParseBench `is_strikeout` stayed exactly **0.0** despite correct `~~` output. Root cause: the rule matches `~~query~~` with markers *tight* against the queried text; each struck line in these PDFs is extracted as a **single text item**, so we emit one GFM-correct `~~whole run~~` and the query sits mid-run → no match. The ONLY way to score is splitting runs into per-word `~~word~~ ~~word~~` (the rule's markup-tolerance bridges the gaps; this is what pymupdf4llm effectively does because PyMuPDF emits short spans) — but that's noisy output and metric-gaming, so per the project's anti-gaming stance ([[project_formatting_triage]]) we kept the clean merged version. Regression-checked: opendataloader 0.8750/0.9131/0.6929/0.8165 (−0.0001 noise); ParseBench text_content + table **byte-flat**; `semantic_formatting` unchanged at 0.4092 (so `LAUNCH_BENCHMARKS.md` needed no edit). Files: `types.rs`, `extract.rs`, `projection.rs`, `markdown_layout/{inline,classify,paragraphs}.rs`.->-> **`semantic_formatting` gap decomposition (the one ParseBench category we trail).** Formula = weighted mean of text_styling(×1.0) + title_accuracy(×1.0) + latex(×0.2) + code_block(×0.2). Per-tool: title_accuracy 0.398 vs 0.442 (**biggest drag**, full weight, = heading detection: is_title + title_hierarchy); text_styling 0.436 vs 0.456 (driven almost entirely by **is_italic** 0.367 vs 0.492 — **is_bold is TIED** 0.519 vs 0.532); latex + code_block we WIN (0.255 vs 0.214, 0.200 vs 0.000). text_styling is a *per-doc* F₀.₅ blend of the style rules present on that doc. Counterfactual: forcing `is_strikeout` to perfect 1.0 lifts semantic_formatting only **+0.0094** (only 13/476 docs carry a strikeout rule) — so strikeout is ≤15% of the gap at its ceiling, not the bottleneck. **The real levers are heading detection (= same capability as opendataloader MHS, which we already lead) and is_italic — and is_italic is ~⅔ `text_absent` (extraction/reading-order) + ~¾ `no_signal` (italic absent from font flags AND name → unrecoverable from metadata); the buildable slice is table-cell/heading emphasis, the same lever that regressed ODL TEDS for bold.** Net: pushing `semantic_formatting` is really a heading-detection + extraction-fidelity project (both shared with categories we already win), not formatting-markup work. Pure bold/italic detection is at its metadata ceiling.--> **2026-06-14 (pm) launch-readiness cosmetic-noise + prose-chrome cleanup — audit followups #2 and #3 from the 2026-06-14 audit entry. Three contained markdown-emission fixes SHIPPED; 208 lib tests pass (2 new). Net: opendataloader clean win, ParseBench table + olmOCR flat.** All three are emission-/classification-layer changes that the three benches structurally barely see (the defects live in prose ornaments + form-table cosmetics that the corpora lack), so they were measured primarily for *non-regression*, with the real payoff being human-perceived output quality. Audit scratch under `/tmp/lp_audit/` (base/ = pre-change binary `lit-base`, new/ = shipping binary).->-> **Fix A — decorative divider / flourish lines (`markdown_layout/classify.rs`, audit #2).** A line whose trimmed text has **no alphanumeric content** and consists only of divider symbols (`* * * *`, `———`, `____`) was flowing into the paragraph accumulator and gluing the ornament onto the *start* of the next paragraph (8× in the Berkshire 2023 letter: `\* \* \* \* … Writers find it useful…`), and a lone decorative em-dash under the title was getting size-promoted to a stray `# -` heading. New `decorative_divider_kind(text)` run *before* heading/paragraph classification (after the code-block flush): ≥3 symbols → `Block::HorizontalRule` (a real thematic break); 1–2 symbols → dropped (too small to be a meaningful rule). Conservative symbol set (`* - _ – — • · ● ▪ ■ ◦ ★ ☆`) — excludes `.`/`=`/`~`/`+`/`#` to avoid dot-leaders, ellipses, setext rules. `is_alphanumeric()` gate means any letter/digit (incl. CJK) → not decorative, so single-char numeric/letter headings are untouched. Verified on Berkshire: `# -` gone, the 8 dividers now emit as standalone `---` with the previously-glued prose intact as its own paragraph.->-> **Fix B — horizontal-rule dedupe (`output/markdown.rs`, audit #3 part 1).** `dedupe_rules()` runs per page before `render_blocks`: drops leading/trailing `Block::HorizontalRule` (which abut the `\n\n-----\n\n` page separator) and collapses consecutive HR runs to one. HRs come from two sources (vector-graphics detection + Fix A's decorative dividers); doubling reads as sloppy and carries no structure. Verified: irs_1040_instr `---` count 338 → 173, zero doubled `---` runs remain across all audit docs.->-> **Fix C — empty-header table rows (`markdown_layout/blocks.rs`, audit #3 part 2).** Header-less tables (`header == None`) were rendering a visible blank `| | | |` header (17× in irs_1040, pervasive in the IRS forms). GFM requires a header row before the separator, so instead of synthesizing a blank one we now **promote the first body row** to the header. Verified: irs_1040 empty-header rows 17 → 0.->-> **A/B (lit-base = pre-change build incl. the SERFF fix, same session):**-> - **opendataloader** (Fix A+B): clean win — overall 0.874144 → **0.875060** (+0.00092), NID 0.912719 → **0.913230** (+0.0005), **MHS 0.813624 → 0.816560 (+0.00294)**, TEDS flat 0.692940. Blast radius 17/200 docs. Fix C on top: **byte-flat** (overall/NID/TEDS/MHS all identical — header-less tables here either don't change TEDS structure or TEDS normalizes the thead/tbody tag).-> - **ParseBench table**: **flat across all three fixes** — grits_con 0.57309 / TRM 0.21960 / composite 0.40756 (matches the documented post-SERFF baseline to 5 digits; decorative dividers/HRs are absent from the single-page table crops, and promoting row0→header doesn't change which cells exist).-> - **olmOCR** (regression-check only per [[project_bench_focus]]): Overall **39.1%**, headers_footers **56.1%**, multi_column 65.8%, table_tests 48.2%, long_tiny_text 29.2% — matches the post-SERFF baseline (39.1% / 56.1%) within noise. The changes only *remove* noise text (ornaments, `# -`, doubled rules) so they can't help-or-hurt the content-keyed `absent`/`order`/`table` predicates beyond noise.->-> **Decision: ship all three** (cosmetic quality wins, net-positive/flat on the two prioritized benches, no olmOCR regression). Remaining open audit items are **#2 chrome's sibling** (none — fully addressed) and the two intentionally-deferred classes: prose run-on in forms and **dot-leader TOCs / noisy form tables** (audit #4/#5, documented as known-hard, not chased). Files: `crates/liteparse/src/markdown_layout/{classify.rs,blocks.rs}`, `crates/liteparse/src/output/markdown.rs`.->-> **2026-06-14 (pm) SERFF row-drop followup — root cause was CHROME STRIPPING, not the table/region-merge path the prior note guessed. Fix SHIPPED (owner: ship-as-is); 207 lib tests pass (1 new).** Reproduced on `FBLB-134215544_page6`: the entire top key-value band `SERFF Tracking #: FBLB-134215544 State Tracking #: Company Tracking #: MS-2024-07` was absent from markdown while the `--format text` grid had it. Traced through xy_cut (the line IS a clean 1-line leaf, `d=1`), region grouping, and table detection — none dropped it. The actual culprit: `markdown_layout/repetition.rs::detect_single_page_chrome` → `matches_chrome_pattern`'s journal-citation branch (`has_year && has_digit_range && len ≤ 120`) **misfired on the embedded date** `MS-2024-07`, read as year `2024` + page-range `2024-07`. Line is top-band + isolated (gap 19pt ≥ 10pt) → stripped as chrome.->-> **Fix (`has_digit_range`).** A `YYYY-MM` / `YYYY-MM-DD` token is a date, not a citation page range: when the first digit group is a 4-digit `19xx`/`20xx` year and the second group is ≤2 digits, skip that match and keep scanning for a genuine range elsewhere on the line. Real journal cites carry a *distinct* page range (`1–9`, `436-444`) that is unaffected; the existing `detects_bottom_journal_citation_chrome` test still passes. Semantically correct — the date-as-range read was always wrong; it just happened to delete some real chrome by accident (see olmOCR cost below). Provably *un-strip only*: the change can only make `has_digit_range` return false, so it adds content back, never strips more.->-> **A/B (true same-session baselines — stash-built `lit` binary + reverted-and-rebuilt Python binding):**-> - **opendataloader** (clean win): overall 0.874082 → **0.874144** (+0.00006), NID 0.912594 → **0.912719** (+0.000125), TEDS/MHS flat. Byte-diff blast radius **1/200** — doc `01030000000149` recovers `Project No: : 2021-2-FR02-KA220-YOU-000048126` (was chrome-stripped via the same date-as-range misfire), verbatim in the GT.-> - **ParseBench table**: grits_con 0.573091 / TRM 0.219599 / composite 0.407560 — **byte-flat**. The recovered line is loose content above the table; GriTS scores cells only, so no structure change. **NB: the prior note's claim that this fix recovers the −0.0007 table dip was a MISDIAGNOSIS** — that dip is the V-cut min-lines guard's table-structure effect, unrelated to this content drop. The dip is still open (separate followup).-> - **olmOCR** (regresses, accepted): overall 39.2% → 39.1%, headers_footers 56.8% → **56.1% (−6 tests)**. Blast radius 4 docs: 2 wins (`8dd368bd…` metadata table `CM-FECL3-E | issued: 2019-04-17 …` recovered from chrome; `tables/7b9b…` cell `1972-21-143 | Geriadu` recovered), and 2–3 short date-only chrome lines now survive and fail `absent` predicates (`Date: 2019-03-15`, `2020-06-11 1/4`, `2018-04-17 10:41 - Tyler Nguyen`). Those lines were only ever deleted *via the wrong citation heuristic*; the principled recovery would be an explicit short-date-chrome rule, but every length/structure discriminator tried overlaps the recovered content (metadata-table rows are also short, date-bearing), so it was deferred rather than risk re-introducing content drops.->-> **Decision: ship as-is** (owner). The dropped-row class is the audit's #1 user-facing defect (silent loss of tracking/project IDs + metadata tables for RAG); net-positive on the two prioritized benches ([[project_bench_focus]]), and the olmOCR cost is short date chrome that was passing on an accident. Files: `crates/liteparse/src/markdown_layout/repetition.rs`. Memory: [[project_chrome_date_falsepos]].->-> **2026-06-14 launch-readiness sanity audit on out-of-distribution internet PDFs — two correctness fixes SHIPPED, net-positive A/B, 218 lib tests pass.** Ran the markdown emitter (default user-facing settings) on five PDFs deliberately *not* in any benchmark corpus (Berkshire Hathaway 2023 shareholder letter, IRS Form 1040 + i1040 instructions, a CDC MMWR multi-column bulletin, an ACL Anthology 2-column paper) to check for benchmark overfitting and real legibility. The layout engine generalized well (clean prose, correct multi-column reading order); the defects clustered in territory the three benches structurally don't cover. Audit scratch outputs were under `/tmp/lp_audit/`.->-> **Fix #1 — ligature doubling on no-`/ToUnicode` LaTeX/Type1 fonts (`extract.rs::GlyphDecoder`).** On the ACL paper every `fi`/`fl` word was doubled: `find`→`fifind`, `beneficial`→`benefificial`, `coefficients`→`coeffificients`, `fluctuations`→`flfluctuations` (21 occurrences; **0** in the other four docs → font-family-specific, the pdfTeX/Nimbus family that dominates academic PDFs — the #1 RAG input). Root cause: with no `/ToUnicode`, PDFium derives per-char unicode from the glyph names itself and *already* expands a single ligature glyph (e.g. the `fi` glyph at char_code `0x02`) into separate `'f'` and `'i'` TextChar entries that **share that one char_code**. Because the font is flagged `untrusted` (no ToUnicode + "Custom" encoding), the glyph-name recovery re-resolved the multi-char name `"fi"` once per entry → `"fi"+"fi"`. This was NOT the documented Mode 16 ligature *substitution*; it is *duplication*, and was independent of the (non-existent — see below) `DISABLE_GLYPH_NAMES` switch. **Fix:** when a glyph-name resolution is multi-char but PDFium's per-char unicode is non-suspicious *and* a member of the resolved string, trust PDFium's own expansion (`return None`). Suspicious-char recoveries (control-code ligatures `0x1A–0x1F`, glyph soup) fire on control/PUA/map-error codepoints and are untouched — verified: `text_simple__spd` still recovers "Fraktion … Bezirksvertretung Dortmund Mengede", `paper_cn_trad` still recovers CJK 消息/出席/幹部. Bench impact not separately measured this session (pure extraction fidelity → expected to move olmOCR char-exact predicates; both arms of the #2 A/B below already include this fix). Memory: [[project_glyph_decode]].->-> **Fix #2 — vertical-cut min-lines guard (`projection.rs::xy_cut_rec`).** On a single-column page the markdown emitter rendered correctly-ordered prose out of order: Berkshire p7 emitted "…pay dividends, and its **discretionary. Annual debt maturities are never material.** / share repurchases are 100%" (materially wrong financial statement) while the `--format text` grid was correct. Root cause: `xy_find_best_cut` found a spurious vertical "gutter" in a 3-line region where one line is full-width with a wide justified inter-word gap and the line below is short — and the min-lines-per-side guard in `xy_cut_rec` was gated on `cut.axis == Horizontal`, so **vertical cuts had no such guard**. The V-cut isolated the 1-line fragment "share repurchases are 100%" into a sibling region, which the region-ordered emitter then placed after the rest. **Fix:** apply the same guard to vertical cuts — reject when either side has `< XY_MIN_LINES_PER_H_SIDE` (2) distinct lines. By construction this can only block `lc<2 || rc<2` cuts; a real multi-row table/column split always has ≥2 lines per side, so it never damages genuine column or table structure. Kill-switch `LITEPARSE_DISABLE_VCUT_MINLINES`. Memory: [[project_column_layout]].->-> **Scored A/B (guard ON = shipped vs OFF = baseline; both binaries include Fix #1):**-> - opendataloader (clean win): overall 0.873345 → **0.874082** (+0.00074), MHS 0.810963 → **0.813624** (+0.00266), NID 0.912543 → 0.912594 (+0.00005), TEDS 0.692941 **flat**. Byte-diff blast radius 5/200 docs, all wins/neutral: docs 124/126 a section heading now correctly follows its preceding figure caption (was hoisted above it), 144 an equation number `(3.16)` rejoins its equation line, 031 a math caption kept whole, 069 a page-number token kept inline.-> - ParseBench table (tiny dip, within noise): grits_con 0.57309 → 0.57239 (−0.00070), composite 0.40756 → 0.40721 (−0.00035), TRM 0.21960 → 0.21961 (+0.00001). Byte-diff blast radius 169/2579. The dip is **not lost table structure** (the guard cannot block multi-row cuts) — it is the SERFF row-drop followup below.->-> Net: opendataloader + (esp. MHS), ParseBench table − a hair (both within the ±0.001–0.002 band the logs treat as noise), plus a real qualitative correctness win (no more scrambled prose / mis-ordered captions). Shipped as-is per owner. **On-disk bench outputs currently reflect the baseline (guard-OFF) arm of the A/B — re-run a fresh sweep on the shipping binary before trusting `output/`/`prediction/`.**->-> **Doc-hygiene note:** several kill-switches referenced in this file / FAILURE_MODES are **not actually implemented** in the code and are silently ignored (e.g. `LITEPARSE_DISABLE_GLYPH_NAMES`, and the `LITEPARSE_DISABLE_GUTTER_CUT` / `DISABLE_NEARSTART_RESCUE` / `DISABLE_GLOBAL_RULED` referenced in earlier entries). Verify a switch exists before relying on it for A/B.->-> **Followups (deferred to next session — tracked here, not yet started):**-> 1. **SERFF row-drop (cause of the ParseBench table −0.0007 dip).** On key-value header bands like `FBLB-134215544_page6` ("SERFF Tracking #: FBLB-… | State Tracking #:"), the blocked 1-line V-cut (`lc=1 rc=1`, correctly rejected) causes that whole row to be **dropped** downstream — it survives in neither the table nor as a loose line. So the row was being kept only because the (wrong) V-cut pre-split it. The real fix is in the table/region-merge path (a 1-line key-value band that isn't pre-split must still emit), not the guard.-> 2. **Audit #3 — prose "chrome" never handled.** Decorative section dividers `* * * * * * …` get glued onto the *start* of the next paragraph (8× in the Berkshire letter), and a decorative em-dash was promoted to a stray `# -` heading. The bench corpora (gov/academic/table docs) contain none of these, so they were never tuned.-> 3. **Audit #4 — pervasive cosmetic noise (cheap, high perceived-quality impact).** Stray/doubled `---` horizontal rules and empty-header `| | | |` table rows appear across most docs (worst in the IRS forms). Bench scorers tokenize/normalize these away (zero metric cost) so they were never penalized, but they look sloppy to a human reader. Suppressing empty-header rows and collapsing adjacent/empty HRs would visibly lift output quality everywhere.-> 4. **Audit #5 — forms & dot-leader TOCs (intrinsically hard).** IRS 1040 form + i1040 instructions TOC render as noisy empty tables and run-on dot-leader entries. Document as a known-hard class rather than chase; even pymupdf4llm struggles here.->-> **2026-06-13 (pm) trailing-period loss fix (Mode 16, `extract.rs` dot-leader threshold) — olmOCR Overall 38.6 → 39.1% (+0.5pp); order 53.0 → 54.4% (+14), multi_column 63.6 → 65.3% (+15 tests), long_tiny_text 25.8 → 29.2% (+15 tests), present 15.8 → 17.9%; headers_footers −2 / table_tests −5 (CI noise). opendataloader BYTE-FLAT (NID 0.9126 / TEDS 0.6929 / MHS 0.8036), ParseBench text_content FLAT (0.65097 both, forced re-infer A/B) and table FLAT (grits_con 0.5731 / composite 0.4076 / TRM 0.2196). 207 lib tests pass. One-line change, shipped.** Resolves the 2026-06-08 FAILURE_MODES Mode 16 "trailing-period loss (~25 tests, probably localized)" item — the cleanest remaining lever in the `text_absent` class flagged by the 2026-06-13 formatting-triage pivot, and explicitly NOT the compressed-metric-font danger zone.->-> **Root cause (single misfiring threshold).** The no-pending-space dot-leader splitter at `extract.rs:644` broke a `.` off preceding content whenever `gap > avg_char_width * 0.4`. A loosely-kerned abbreviation/sentence period sits at **~1.0× avg char width** (measured `g/cw ≈ 1.0` on `Sci.`/`Chem.`/`Environ.`/`Phys.`), so it tripped the splitter; the split-off `.` segment then merged forward into the next token and vanished. Tightly-kerned periods (`Mater.` at `g/cw=0.14`) stayed attached, which is why the loss was *positional* (only before column/tab gaps in reference lists) and looked per-word. Genuine no-space dot leaders run `g/cw ≈ 2–23` (verified a real one at 23.2), and spaced leaders take the *other* (`pending_space`) branch entirely — so raising the cutoff `0.4 → 2.0` opens a wide safe valley.->-> **Why olmOCR-only.** Pure extraction-fidelity fix → only moves the benchmark whose char-exact `order`/anchor predicates a dropped period actually breaks (a 1-char Levenshtein miss fails the whole anchor). opendataloader (NID/TEDS/MHS) and ParseBench (word-match rules normalize trailing punctuation) are structurally insensitive, hence byte-flat/flat — confirmed, not assumed. Qualitatively large for RAG ingestion regardless: reference lists go from `Cravatt, B.F (2004) … Angew Chem Int Ed` to fully-periodized `Cravatt, B.F. (2004). … Angew. Chem. Int. Ed.`.->-> **A/B method (the load-bearing rigor).** First text_content run reused **stale cached inference** (`parse-bench run` defaults `--force=False`) and reported a false-flat to 15 digits — re-ran *both* binaries with `--force` for a true same-harness delta. For olmOCR (in-process Python binding, not the CLI binary) a same-session base required reverting the one line, rebuilding the binding, and converting a clean `liteparse_m16base` candidate — historical numbers carry documented binary drift and can't be trusted as the baseline. Byte-diff bound the blast radius first: multi_column 57/231, headers_footers 47/266, long_tiny_text 12/62 changed — all sampled diffs were period restorations (plus one re-merged wrongly-fragmented paragraph), one trivial `, and`→`,and` collateral against dozens of restorations. Scratch candidates `liteparse_m16` / `liteparse_m16base` left in `bench_data/`. Memory: [[project_missing_spaces]].->-> **2026-06-13 semantic-formatting triage + table-cell emphasis A/B — lever empirically KILLED, REVERTED (byte-identical baseline, 207 lib tests pass). No code shipped; the finding is the deliverable.** Set out to lift ParseBench `text_formatting` (the 2026-06-11 audit's Tier-3 #7 fidelity lever: bold 0.53 / italic 0.37 / title_hierarchy 0.37). Measured first instead of editing, and the measurement says **stop**: bold/italic detection has ~no benchmark headroom.->-> **Fresh baseline confirmed (no drift):** is_bold 0.5273, is_italic 0.3696, is_title 0.4413, title_hierarchy 0.3716, underline/sup/mark 0.0, code_block 0.2, overall avg_rule_pass_rate 0.3736.->-> **Triage reframe (`dataset_eval_utils/triage_formatting.py` — runs each rule through ParseBench's own `create_test_rule`, buckets each failure by whether the GT text is even in our output).** The inline emphasis pipeline ALREADY exists (`inline.rs::render_line_inline` does per-span `**`/`*`/`` ` ``/links) — the audit's "whole-line-only" framing was stale. The gap is NOT missing capability:-> - is_bold: **76% of failures are `text_absent`** (word not in output at all → extraction/reading-order, ORTHOGONAL), 24% `present_plain` (216 rules, detector-addressable).-> - is_italic: 47% text_absent, 53% `present_plain` (208 rules — most addressable relative to size).-> - is_title: 63% text_absent, 37% present_plain (422 rules).-> - title_hierarchy: 97% of unsatisfied constraints are missing_title/missing_edge (PRESENCE), only 2% level + 0% order → **NOT a leveling problem**; it's heading detection.->-> **Drill-down (`dataset_eval_utils/drill_formatting_misses.py` — temporarily exposed font_flags/font_weight in JSON, reverted).** Of the introspectable `present_plain` misses: ~25% `flag_set` (item carries the italic/bold flag → detection SHOULD fire, miss is downstream), ~75% `no_signal` (italic/bold absent from font flags AND name — synthetic skew / outline-only weight, **unrecoverable from font metadata**). ~50% were `no_match` (JSON `text_items` empty for those PDFs — tooling gap, not a product fact). The `flag_set` misses cluster in table-heavy financial/SERFF/IPO docs → root cause: **table cells (`escape_table_cell`) and headings (`# text`) render PLAIN, never through the inline pipeline.** (`name_contains_any` in projection.rs is also case-sensitive.)->-> **Built + A/B'd the one buildable lever: per-cell BOLD emphasis** at the two shared builders `finalize_table_run` + `build_ruled_table` (reusing the already-tracked `TableCell.bold` / ruled `cell_is_bold`; `wrap_emphasis` made `pub(super)` — no struct churn, late-wrapping so cell-splitting logic is untouched). Result:-> - ParseBench is_bold **+0.0000** (per-doc avg over 476 docs — the few flipped cells round to zero; bold/italic GT targets aren't in table cells, and in-table bold detection hits the same no_signal wall).-> - ParseBench table GriTS_con / TRM **0.0000** (safe — `normalize_cell_text` strips `**`).-> - opendataloader **TEDS −0.0167** (0.6929→0.6762), overall −0.0019 — ODL's TEDS does NOT strip markdown from cells, so `**Foo**`≠`Foo`.-> Net loss, zero benefit → reverted. **Do not re-attempt table/heading emphasis for benchmark gain.**->-> **Conclusion + pivot.** Bold/italic semantic formatting is a dead benchmark lever (gap = ⅔ text_absent + ¾ no_signal; buildable pool tiny and not in tables). The real opportunity is the **`text_absent` (missing text)** class — the documented Tier-1 word-fusion / column-interleave / glyph-soup extraction work, which lifts is_bold/is_title/text_content/order/NID together. Next step (not yet done): a fresh scoped triage of the current text_absent set (cross-ref olmOCR `order`) to find whether any sub-cause is still a clean lever before committing to the risky compressed-metric-font work (the −308/−187 danger zone). Memory: [[project_formatting_triage]]. Scripts: `triage_formatting.py` (SIGALRM 3s guard — ParseBench bold/italic tempered-greedy `.DOTALL` regexes backtrack catastrophically on large non-matching docs; reliable for cause buckets, NOT precise pass-rate deltas — use the official eval for those), `drill_formatting_misses.py`, CSV `formatting_triage.csv`.->-> **2026-06-13 hyperlink extraction → `[text](url)` (new capability, default ON; ALL benchmarks zero-regression because harnesses disable it) — opendataloader BYTE-IDENTICAL (Overall 0.8719 / NID 0.9125 / TEDS 0.6929 / MHS 0.8036), ParseBench table BYTE-IDENTICAL (grits_con 0.5737 / TRM 0.2196 / composite 0.4079), olmOCR matched (Overall 38.6% / multi_column 63.6% / table_tests 48.7% / long_tiny_text 25.8%); 207 lib tests pass (3 new).** Implements the 2026-06-11 audit's Tier-2 #5 "links entirely absent" — the one genuine missing *capability* (vs a metric gap): no `GetURI`/annotation extraction existed anywhere in `extract.rs` or the pdfium wrapper. Targets RAG ingestion, not a benchmark lever.->-> **Annotation-based only** (`FPDFLink_Enumerate` → `FPDFLink_GetAction` → `FPDFAction_GetURIPath`), not auto-detected web-links — captures real clickable links whose anchor text differs from the URL (`[click here](https://…)`), which web-link scanning can't. No PDFium fork rebuild needed: all symbols were already in the static `bindings.rs`; only the libloading table in `pdfium-sys/src/dynamic.rs` needed the four fns added (plus `FPDFLink_CountQuadPoints`/`GetQuadPoints`).->-> **Data path.** New `Page::links(view_box) -> Vec<PdfLink { rect (viewport), uri }>` in `pdfium/src/page.rs` (mirrors `path_objects`; rect mapped via the existing `bounds_to_viewport`). New `TextItem.link: Option<String>` (rides through projection's `..proj.item` for free). `assign_links` in `extract.rs` attaches a URI to every text item whose bbox center falls in the link rect. Rendered in `markdown_layout/inline.rs`: group spans by `(style, link)`, wrap each group `[…](url)` *outside* the emphasis wrap (`[*cite*](url)`), angle-bracket form for URLs with spaces/parens. `line_uniform_style` returns `None` for any link-bearing line so the paragraph/list uniform fast-paths route through the per-span renderer — every new branch is gated on link presence, so link-free output is byte-identical.->-> **Per-line quad points preferred over the union rect.** A link wrapping across lines has one tight quad per line; the single annotation `/Rect` is their *union* (a tall box that swallows the unlinked words between the lines). When quads exist we emit one `PdfLink` per quad; otherwise fall back to the union rect. For the no-quad-point multi-line case the true anchor is unrecoverable, so `assign_links` **drops** the link when `rect height > 1.8× median covered-item height` rather than wrap a whole sentence (verified on opendataloader `…086`, a `perma.cc/Z89Y-` / `SEVS.` wrapped citation). Single-line links (`…095` `[*(Niederle and Vesterlund 2007)*](…)`, `…185`) render correctly.->-> **Why default-ON but every harness disables it.** The GT corpora **never use `[](url)` syntax** — 0/200 opendataloader GT files; URLs appear as plain text. Measured: links-ON regresses opendataloader **NID 0.9125 → 0.9102, MHS 0.8036 → 0.8021** (TEDS flat). So real users get richer markdown by default, but the three bench providers opt out for parity: opendataloader + ParseBench pass CLI `--no-links`; the olmOCR in-process Python runner passes `extract_links=False`. Config `extract_links` (default true) is wired through CLI (`--no-links`), the pyo3 binding + `packages/python` wrapper, and napi. The parser only extracts links when `output_format == Markdown` (text/JSON paths skip the per-page `FPDFLink_Enumerate` call entirely).->-> **ParseBench nuance (flagged, not yet measured).** There is **no `is_link` rule type** in ParseBench (formatting kinds are bold/italic/underline/strikeout/mark/sup/sub/title/latex/code_block/title_hierarchy), so links can't win `text_formatting` points. The tracked `table` group is byte-identical with/without links (tables carry no link annotations). **But `text_content.jsonl` GT contains `[text](url)` syntax in ~7 lines / ~4 docs** (examgenius.in, epa.gov, bonnieplants, artsandculture), scored by `missing_specific_word`/`order`/`unexpected_word`. So blanket `--no-links` on ParseBench is safe for the tracked table metric but is an *unmeasured assumption* for `text_content` — links-ON could match GT better on those ~4 docs while adding unexpected `](url)` tokens on the plain-text majority. Net `text_content` effect unmeasured (only ran `--group table` this session). Memory: [[project_markdown_links]].->-> Kill-switch: `--no-links` (CLI) / `extract_links=False` (Python) / `extractLinks: false` (Node). Files: `pdfium-sys/src/dynamic.rs`, `pdfium/src/{page,lib}.rs`, `liteparse/src/{types,extract,parser,config,main}.rs`, `liteparse/src/markdown_layout/inline.rs`, `liteparse-python/src/lib.rs`, `liteparse-napi/src/types.rs`, `packages/python/liteparse/parser.py`, + 3 bench provider scripts.->-> **2026-06-12 (pm) per-font empirical space-width recovery (Mode 14 word fusion, no-metric fonts) — olmOCR long_tiny_text 25.3 → 25.8% (+2 tests, clean, zero regressions); multi_column / table_tests / ParseBench BYTE-IDENTICAL; opendataloader flat (NID/TEDS/MHS unchanged); 204 lib tests pass.** Targeted the 2026-06-11 audit's Tier-1 #2 dense/small-font word fusion (22.6% of `long_tiny_text` files), reproduced on `long_tiny_text/20_pg58` (`theynexttakeonthe`, `againstManchester`, `transferbusiness`, `forSpurs` — body rendered as word-soup).->-> **Root cause (Layer 1, `extract.rs`).** The missing-space recovery in the no-pending-space branch uses `thresh = 0.7 * space_w` when the font exposes an ASCII-space advance (`ch.font_space_width()`), else a fixed `0.35 * em_vp` fallback. `20_pg58`'s body font exposes **no space metric** (`sw=-1` in `LITEPARSE_DEBUG_GAPS`), so every boundary hit the fallback — which computes ~3pt for an 8.5pt font while the real inter-word gap is ~1pt, so nothing ever fired. Histogramming `loose_gap` by the PDFium `ps` flag showed the gaps are cleanly **bimodal**: intra-word kerning loose ≤ 0.4pt (434 events), genuine spaces 0.8–1.4pt (34 events), near-empty valley between — and the boundaries where PDFium *did* emit a space (`ps=1`) land in that exact 0.8–1.4 band, giving a free per-font calibration signal.->-> **Fix.** Recover the missing `space_w` empirically: in the pending-space commit branch, sample `loose_gap / em_vp` at alpha-alpha genuine-space boundaries into a `HashMap<font_name, Vec<f32>>`; in the no-metric `ps=0` branch, when the font has ≥ `MIN_SPACE_CAL_SAMPLES` (6) samples, set `thresh = 0.7 * median(ratios) * em_vp` — i.e. feed the recovered width through the **same 0.7 factor the metric path uses**. Falls back to `0.35 * em_vp` otherwise. Single-pass running accumulator (dense docs accumulate 6 samples within the first line or two). New `median_f32` helper. Kill-switch `LITEPARSE_NO_SPACE_CAL=1`.->-> **The load-bearing safety property.** Calibration activates **only on no-space-metric fonts**, so the metric-bearing fonts that dominate multi_column / tables — the exact set that produced the documented `+179`-mirage catastrophe (multi_column −308 / tables −187 from a global gap constant) — are byte-untouched. Verified: olmOCR multi_column / table_tests **0/231 and 0/188 docs changed**; ParseBench 0/125 sampled docs changed; opendataloader flat. The old regression came from a *global* constant; a *per-font* threshold confined to no-metric fonts is safe.->-> **A/B method:** snapshotted the pre-change binary (`target/release/lit-base-wf`), byte-diffed all olmOCR categories to bound blast radius (changes confined to long_tiny_text 10/62 + headers_footers 1/266), then scored a same-era stored baseline candidate (still contains the fusions, multi_column/table_tests byte-match) against the new `liteparse_spacecal` candidate. long_tiny_text FAIL-set diff = clean **+2 / −0** (newly-passing `20_pg48`, `20_pg58`). The +2 understates the win: olmOCR's char-exact anchors undercount readability because these dense docs fail on compounding reasons (column interleave, glyph soup) too — the qualitative output went from word-soup to clean prose.->-> **Two follow-up levers investigated and empirically KILLED (so the next session doesn't re-chase them):**-> 1. **Pre-scan to calibrate early-page boundaries** — probed by lowering `MIN_SPACE_CAL_SAMPLES` 6→3 (faster warm-up). olmOCR long_tiny_text **unchanged** (114/442 both), and min=3 began bleeding into 3 metric-corpus docs (strictly worse). Warm-up is not the bottleneck → a pre-scan gains ~nothing. Kept 6.-> 2. **Layer-2 projection silent-merge** (the suspected `ofTurkey` cause) — instrumented the step-3 silent merge in `build_projected_lines`: **0 alpha-alpha events across all 200 opendataloader docs**. The 2026-06-08 alpha-alpha guard already closed it. `build_one_line` (~4636) *always* inserts a space between separate items, so any fused token is a SINGLE extract item → **`ofTurkey` is Layer-1 compressed-metric**, not Layer-2: a metric-bearing font whose typeset space is squeezed below `0.7×sw`, where intra-word kerning *overlaps* the genuine-space band (no clean valley). That class is the high-risk −308/−187 danger zone and is deliberately left untouched.->-> **Takeaway:** word fusion is now exhausted in the safe (no-metric) regime; the only sizable remaining territory is high-risk compressed-metric fonts (no clean geometric discriminator). Next step is **semantic formatting** (per-span bold/italic + heading-level accuracy → ParseBench text_formatting, currently bold 0.53 / italic 0.38 / title_hierarchy 0.37), the Tier-3 fidelity lever from the 2026-06-11 audit.->-> **2026-06-12 decimal-fragment merge + body-seeded inferred tables — opendataloader Overall 0.8710 → 0.8719 (TEDS 0.6877 → 0.6929 (+0.0052), NID 0.9120 → 0.9125, MHS flat 0.8036); ParseBench table grits_con 0.5647 → 0.5737 (+0.0090), composite 0.4040 → 0.4079, TRM 0.2213 → 0.2196 (−0.0016); olmOCR table_tests 47.3 → 48.7% (+1.4pp), multi_column 63.6% (noise).** Targeted the 2026-06-11 audit's Tier-1 #3 "table label-column drop" content-loss defect, reproduced on doc `01030000000082` Table 29 (state names Gujarat/Punjab/… silently dropped, table emitted 3-col instead of 4-col). Two compounding root causes, both in the markdown/projection layers (the names extract fine — verified in the `--format text` grid):->-> 1. **Decimal fragmentation (`projection.rs` word-merge pass).** PDFium emits `15.6` / `200.4` as two x-contiguous items `"15"`+`".6"`; `looks_like_table_number` flags both, so the `both_are_numbers` guard kept them apart, leaving `15 .6` / `21 .3%` in rendered cells *and* injecting spurious close-spaced column tracks. **Fix:** a decimal-continuation carve-out — when the next run begins with `.`+digit and butts directly against a digit-terminated run (`delta_x ∈ [−2, 1]`, y-compatible), merge silently. A real pair of adjacent table numbers always shows a visible inter-cell gap, never a digit against a leading-decimal run, so this is safe. Clean win on its own (fixes the `X .Y` artifact across many numeric/financial docs; NID +0.0005, zero regressions).->-> 2. **Header-seeded tracks drop the left column (`tables.rs::try_detect_table_inferred`).** The inferred-track path captured the State column (`x≈104`) but **bailed** (`min_gap < TABLE_MIN_TRACK_GAP`) because the stacked header (`"Number of"`@271.8 / `"clauses"`@280.8) and the decimal fragments injected close phantom tracks; it then fell to the header-seeded `try_detect_table`, which seeds tracks from the first line (the 3-cell header `"Number of | GSDP | GSDP"`) and **silently discards** every data row's `Gujarat`@103 in the `cells.len() > column_count` keep-filter. **Fix, two parts:** (a) **support-filter the inferred tracks** in `infer_tracks_from_raw_items` — each cluster carries its item-support; when a clear multi-row body exists (some cluster supported by ≥3 items) prune the single-item phantom clusters (header cells offset from their data column). For Table 29 this leaves exactly the 4 data columns (104/289/369/444, min_gap 75 ≫ threshold). (b) **seed the body at the first strong row, not `start_idx`** — a strong row single-covers every track with a distinct span; the *seeding* line may be a stacked/partial header, so scan forward for the first strong row, seed there, and let `finalize_table_run`/`absorb_header_lines` absorb the header lines above.->-> **The load-bearing narrowing (caught by A/B byte-diff).** The first cut of (2b) fired the inferred path *eagerly* on short header bands — a strong row found inside a 2-line header anchored a pseudo-table that consumed the lead lines and starved the header-seeded path of the real table below (docs **117/187** lost their tables entirely → loose text; opendataloader TEDS regressed −0.0058). Guard: **only advance `body_start` past `start_idx` when the resulting body has ≥3 rows** (`body_start > start_idx && rows.len() < 3 → bail`). Already-strong seeds (`body_start == start_idx`) keep the standard `TABLE_MIN_ROWS` threshold and are byte-unaffected. This flipped opendataloader TEDS from −0.0058 to **+0.0052** while keeping the doc-082/083/084 column recoveries; 117/187 restored byte-identical to baseline. Blast radius 10/200 opendataloader docs, all wins/neutral.->-> **A/B method:** snapshotted the pre-change binary as `target/release/lit-base`, byte-diffed all 200 opendataloader docs to bound blast radius, then ran same-harness evals for both binaries (opendataloader via `LITEPARSE_BIN` override; ParseBench by swapping `target/release/lit`). olmOCR rebuilt the root-venv Python binding and ran as regression-check only.->-> **Residuals / known limitations:** (a) **recovered tables get an empty header row** (ParseBench TRM −0.0016) — a *centered* header label (`"State"`@166) can't bind to a *left-aligned* data column (names@103) under point-tracks, so `absorb_header_lines` breaks and the header text spills into a loose line above; the body content is recovered but the header is imperfect. Proper fix is nearest-track header binding (separate, riskier). (b) The audit's other content-loss docs **046/121/122** are untouched — different root cause (party-name / in-table-prose extraction in the projection layer), separate follow-up. 204 lib tests pass.->-> **2026-06-12 code-block language tag — ParseBench `is_code_block` 0.0 → 0.2 (per-doc avg; 0/10 → 5/10 individual tests, all five python cases).** Code blocks were already detected and fenced correctly with intact content, but emitted as bare ` ``` ` with no info-string, so `rules_formatting.py`'s `is_code_block` (which requires a fence whose lang matches the expected language, then checks the snippet is in that block's body) scored a hard 0.0. Added `lang: Option<String>` to `Block::CodeBlock` and a conservative `detect_code_language` (`markdown_layout/classify.rs`) called from `flush_code`: JSON (brace/bracket opener + quoted keys), C/C++ (≥2 of `#include`/`std::`/`int main`/`nullptr`/`->`/`::`/`template<`), Python (≥2 keyword/f-string/`self.`/comprehension signals, or ≥1 plus a colon-terminated control line). Returns `None` (bare fence) when unsure, so docs with other-language tests can't be mis-tagged. Emitter writes the info-string after the fence (` ```python `).->-> **Win is python-only.** `text_misc__justcode` now emits four ` ```python ` blocks whose bodies contain the rule snippets (whitespace-normalized match tolerates lost indentation) → 5/5 on that doc. The other 5 tests are **detection-layer gaps, out of scope for the tag**: `text_misc__json` / `text_simple__paper` (cpp) never emit code blocks at all because their content isn't flagged `all_mono` upstream; `text_ocr__code` (fortran) is OCR-gated; `text_simple__hex` has a junk language string (`mki@mki:~/...>`). The overall `text_formatting` `avg_rule_pass_rate` read 0.3683 vs the 0.371 reference — within documented recorded-score/binary drift, **not a regression**: the change is strictly additive (only appends a language word to ` ``` ` fences in code-block docs), so the thousands of `is_title`/`is_bold`/`is_italic` tests on non-code docs are byte-identical. 204 lib tests pass (1 new: `detect_code_language_classifies_common_langs`). No olmOCR/opendataloader impact (code blocks ~absent there).->-> **Reading-order (figure-caption) detour — investigated, nothing shipped.** Before this, dug into the 2026-06-11 audit's "#1 caption/sidebar splicing (~150 order tests)" lever. Re-triage (`order_defect_triage.py`): 319 failing olmOCR multi_column order tests = 147 SHREDDED_INTERLEAVE (target) + 85 NOISE_NEAR_PASS (tolerance, free) + 61 TEXT_DROPPED (extraction, orthogonal) + 17 PARTIAL + 9 inversions. The 147 span 84 docs, of which only **17 are figure-caption** (image + caption marker), 10 image-only, **57 no-image** (running-header/author-chrome splices, reference-list interleave, some glyph-garble confounded). Two contained attempts, both empirically dead: (a) **co-baseline caption guard** in `build_projected_lines` (refuse same-y-band merge across a column gutter when italic style differs) — de-splices the *output* cleanly (verified on `0b045ab8…page_18`) but **0 wins / 3 losses** on the metric, because the scored failures are cross-column *reading-order* (page_18 wants the left-column caption before the right-column one), and splitting a co-baseline line separates without reordering; the italic-XOR also misfires on emphasized terms at column edges. (b) **column-cut histogram exclusion** — abandoned before coding: per-doc `LITEPARSE_DEBUG_XY` traces show the fig-caption docs' columns **already split correctly** (`0083fb21` V@395+V@179, `02e31046` V@276); the `peaks` rejects are benign single-column-leaf decisions, not split failures, so excluding caption items from the line-start histogram would be a no-op. The genuine residual wins need true cross-column **reordering** (semantic) or a **banner false-positive fix** (~3 docs, `027880a8…page_6` / page_18 style, where spurious H-cuts fragment a column into a sub-8-line region that can't V-cut) — neither a clean lever. **Accepted as a known limitation for now.** The audit's "well-scoped ~150-test win" was over-optimistic: the real target is ~17 heterogeneous docs on the riskiest file.--> **2026-06-11 launch-readiness quality audit (research only, no code changes).** Competitor-independent audit of absolute output quality, validated on real current output via a 5-way parallel investigation (word fusion, reading order, dropped/duplicated content, math, structural/inline formatting). Data sources: opendataloader 200 paired prediction/ground-truth markdown (`opendataloader-bench/{prediction/liteparse,ground-truth}/markdown/<id>.md`), olmOCR category outputs + predicate JSONLs, ParseBench formatting CSVs. Reusable triage script written: `dataset_eval_utils/order_defect_triage.py`.->-> **Headline reframe — reading order is less broken than the metric implies.** The ~52.8% olmOCR order-pass rate overstates it. Of 320 failing multi_column order tests: ~84 are metric noise (both strings present *in correct order*, fail only on strict char-match from a dropped space — free wins via tolerance, not defects), ~61 are extraction/OCR drops (orthogonal), only **~8 are true column inversions**, and **~150 are the real defect: caption/sidebar text spliced into body prose** (floating captions / narrow side-blocks not isolated before column ordering — e.g. `multi_column/0b045ab8…page_18` prints caption #2 → body → caption #1 interleaved; `02943986…page_7`, `0904c7071…page_4`, `0925342e…page_4`, `0145bc47…page_11`). Fix target is sharp: caption/sidebar isolation in the XY-cut/anchor pass, NOT a reading-order rewrite.->-> **Validated defect map, ranked by (impact × fixability):**->-> *Tier 1 — fix before launch:*-> 1. **Caption/sidebar splicing** (~150 order tests). The real reading-order defect; most damaging for ingestion, well-scoped.-> 2. **Word fusion in dense/small-font text.** Near-zero in normal prose (opendataloader distinctive-token recall 0.98, ~1 confirmed fusion in 55k words), but **22.6% of olmOCR `long_tiny_text` files affected** — and lowercase→lowercase fusion (`theynexttakeonthe`, `againstManchester`, `Cityand`, `forSpurs` in `long_tiny_text/20_pg58…`) is far worse than the camelCase tip a regex catches. Confirmed `ofTurkey`→"of Turkey" (opendataloader 01030000000012, GT-verified). Concentrated, not pervasive; matches the `project_missing_spaces` adaptive-gap-threshold need. Trickiest fix (must stay table/column-safe).-> 3. **Table label-column drop** — the *only* real content-loss defect (all other losses are legitimately OCR-gated, expected under `--no-ocr`; median word-count ratio 0.96, no duplication). Numeric columns render but the leading label column drops/garbles: docs `01030000000082` (state names Gujarat/Punjab/…), `01030000000046` (party names Funcinpec/Candlelight/…), `121`/`122` (in-table prose → `. a . a tip` fragments). Root cause: table cell text reconstruction in the projection layer. Small but real correctness miss.->-> *Tier 2 — cheap wins:*-> 4. **Code-block language tag.** Code blocks actually *work* (real Python correctly fenced on `text_misc__justcode.pdf`) but score 0.0 on ParseBench because we emit bare ` ``` ` with no language tag (`rules_formatting.py:583` requires `lang=="python"`), and snippets fragment across multiple fences. Near-trivial unblock. (NB: the ~273 fenced output files are mostly GridFallback ` ```text `, not code.)-> 5. **Links entirely absent** — no `GetURI`/annotation extraction exists in `extract.rs` or the pdfium wrapper at all. PDFium exposes it; genuinely missing feature, moderate effort.->-> *Tier 3 — document as known v1 limitations (large effort, not launch blockers):*-> 6. **Math notation structurally absent.** Glyphs/Greek/operators (α 841×, ≤ 1016×, ∈ 2239×) decode fine, but sub/superscript positioning is flattened (**0/522 arxiv outputs** contain any super/sub char or `\frac`/`\begin`), display equations dumped inline with no `$$` delimiters, degree symbol dropped (`95°C`→`958C`). Affects *all* scientific/technical docs, not just arxiv. The #1 sub-gap is sub/superscript positioning, then equation delimiting — symbol decoding is mostly fine. Scope explicitly as out-of-scope for v1 rather than half-do.-> 7. **Bold/italic whole-line-only** (ParseBench bold 0.53 / italic 0.38 — emitted but rarely fine-grained spans; occasionally over-applied/malformed) and **heading-level accuracy weak** (title_hierarchy 0.37 — wrong levels, not mere presence). Lists work but flat (nesting untested — no nested list in either corpus). Fidelity tier, lower ingestion impact.->-> **Recommended order when resuming:** #1 caption/sidebar splicing first (biggest "looks scrambled" win, well-scoped — start from `dataset_eval_utils/order_defect_triage.py` and the ~150 SHREDDED_INTERLEAVE docs to find the geometric signal separating a floating caption from body text); bundle #3 + #4 as small high-confidence correctness fixes; #2 dense-text word fusion is highest-value-invisible but budget for iteration; math (#6) document as a known limitation.->-> **2026-06-11 global ruled-table pass — opendataloader Overall 0.8694 → 0.8710 (TEDS 0.6678 → 0.6877 (+0.0199), NID 0.9115 → 0.9120, MHS 0.8032 → 0.8036); ParseBench table grits_con 0.5507 → 0.5647, TRM 0.2123 → 0.2213, composite 0.3922 → 0.4040; olmOCR table_tests 46.7 → 47.3% (+6), Overall + multi_column flat at 38.4% / 63.8%.** Clean win, zero regressions. Crosses two competitor lines: **ParseBench TRM 0.2213 now beats pymupdf4llm 0.2205** (the last remaining table deficit), and the opendataloader TEDS gap to nutrient halves (−0.040 → −0.020, 0.6877 vs 0.708).->-> **Root cause.** `detect_ruled_tables` runs per xy_cut leaf. A ruled table whose rows scatter across ≥2 leaves (a V-cut sliced it, or a banner/caption peeled a level) never has its full text in any single leaf, so per-region detection rejects it as mostly-empty and the grid is lost entirely. (Distinct from the cross-region pass, which re-fuses *borderless* slices via aligned baselines — this is the *ruled*-grid analog, keyed off the stroke geometry directly.)->-> **New `detect_ruled_tables_global`** (`tables.rs`), run once over all filtered lines in `classify_page_with_filters` *before* cross-region merge. `build_ruled_table` gained a `global: bool` and now returns the consumed line indices so the caller can pull them out of the region pipeline and emit each table as a y-positioned `Interruption::Table`. Global-only refinements inside `build_ruled_table`:-> 1. **`RULED_HLINE_MIN_COVERAGE = 0.5`** — drop horizontal rules spanning <50% of the column extent before deriving row boundaries. A full-height left/right page border unions a stray box (logo, sidebar) into the grid component through the shared vertical; that box's short rules would otherwise become phantom top rows that vacuum surrounding prose. Falls back to all rules if the filter leaves <3.-> 2. **y-sorted line + span iteration** — the global pass receives lines in xy_cut leaf order (column-by-column), which scrambles multi-line-cell text on array-order concatenation; sorting by `bbox.y` (and spans by `x`) restores reading order within each cell. No-op for the already-y-ordered per-region path.->-> **Acceptance gates in `classify.rs`** (each tuned against a measured misfire):-> - **≥2-leaf scatter required.** Only rescue tables the per-region path *can't* see whole — those whose consumed lines span ≥2 distinct `region_path`s. A table living in one leaf is already handled (often better) there.-> - **`already_handled` skip.** If any single leaf's share of the consumed lines already forms a table on its own (`detect_ruled_tables` or `detect_tables` non-empty), don't override — doc 127's decorative frame spans the data table *plus* surrounding prose; the data region tables cleanly by itself, the frame would fuse prose into garbage.-> - **1-column gate** (the load-bearing false-positive guard). A decorative banner/frame around a single prose column (left+right border, no interior verticals) unions into a 1-col "grid" whose lone cell is the whole paragraph block. A real data table always has ≥2 columns. Caught docs **028649** (2-col prose + "THE PLAN" banner fused into one 250-word cell) and **06ba2a90** (journal page + "Table II" caption box). **Word-count gating was tried first and reverted** — it killed *all* the opendataloader wins too: the genuine scattered tables this pass rescues (competence/legal description grids) have cells just as long as the fused-prose false positives, so cell length can't separate them; column count can. With the 1-col gate, opendataloader/ParseBench wins are byte-identical to the ungated run and olmOCR multi_column fully recovers (63.3 ungated → 63.8). Two changed multi_column docs remain (081875 3-col, 0d7383 2-col) — the documented Mode-5 ambiguous slide-feature-table family; they emit as tables but cost no order tests.->-> Kill-switch `LITEPARSE_DISABLE_GLOBAL_RULED=1`. 190 lib tests pass.->-> **Triage note:** A/B the multi_column dip by converting the disabled baseline to a named candidate (`liteparse:name=liteparse_noglobal`) and `diff`ing the per-doc `.md` — isolated exactly 4 changed docs out of 884, which is what surfaced the 1-col-vs-≥2-col split. Opendataloader A/B via the kill-switch on the same binary (no rebuild needed) confirmed the col-gate is byte-identical to the ungated win.->-> **2026-06-10 (pm) tight-gutter column rescue + XY depth/fill-gate session — opendataloader Overall 0.8633 → 0.8694 (NID 0.9071 → 0.9115, MHS 0.7865 → 0.8032, TEDS 0.6716 → 0.6678); olmOCR Overall 37.7 → 38.4% (multi_column 60.0 → 63.8%, table_tests 45.5 → 46.7% — now ahead of pymupdf4llm's 46.2); ParseBench table grits_con 0.5481 → 0.5507, TRM 0.2083 → 0.2123, composite 0.3886 → 0.3922.** Targeted the docs 38/189 "bold heading fused inline across an unsplit 2-col layout" family — the head of the nutrient MHS/NID gap (38/189 alone were MHS −0.97/−0.94 vs nutrient).->-> Five changes, all `projection.rs::xy_find_column_cut` + recursion, each motivated by a measured geometry:->-> 1. **Near-gap line-start rescue.** Doc 38's gutter is 8.7–9.4pt vs the strict `gutter_gap = median_h.max(4.0) = 10pt` threshold (word gaps 4.0–4.8pt), so 6 of 8 right-column lines never registered as line-starts and the right peak died on `min_per_peak`. Starts with gap in `[0.6×, 1.0×] gutter_gap` now count toward the histogram **iff exactly aligned (±1.5pt) with ≥2 strict starts** — a column edge aligns sub-point across bands; justified-text "rivers" jitter. The first cut used a loose ±1-bucket match and manufactured a fake second column inside doc 80's large-font narrow justified prose (NID 0.986 → 0.626); the exact-alignment requirement restored 80 byte-identically while keeping 38. Kill-switch `LITEPARSE_DISABLE_NEARSTART_RESCUE=1`.-> 2. **Histogram V-cut placed at the actual gutter, not midway between peak centers.** Peaks mark column *left* edges, so the center midpoint (e.g. x=212 between peaks 97/327) lands inside the left column's text and `partition_by_item` misroutes mid-line word clusters (doc 38's "October 2020, 17% said" jumped columns). Boundary = right column's leftmost *counted start* — NOT the bucket-quantized peak center: doc 64's right-aligned numerals start at 178.5 with peak center ~203, and a center-anchored cut at 185.7 routed "2454" left, made the leaves x-overlap, and killed the cross-region candidate (64 TEDS 1.000 → 0.533). With the min-start boundary 64 is byte-identical to baseline. Kill-switch `LITEPARSE_DISABLE_GUTTER_CUT=1`.-> 3. **`XY_MAX_DEPTH` 6 → 12.** Doc 189 (SOLAR paper): five stacked table/caption bands each burn a banner-peel level, so the 2-col body leafed out *un-split at the cap* (trace just stops at d=5 COMMIT). The cap is a backstop, not a gate — every extra level still passes all cut gates.-> 4. **Fill gate extended to 2-peak cuts** (was >2-peaks only). With deeper recursion, 2-col label/value tables started reaching the histogram path and getting sliced (doc 166's "Mineral or colloid type | CEC" table → two leaves, TEDS 0.849 → 0). The min-fill hard floor stays **n>2 only**: it exists for ≥3-col dilution, and at n=2 a sparse second column of two drags the weighted average itself — keeping the floor at n=2 false-positived on doc 157's short second prose column (NID 0.951 → 0.429) because the tabular flag also suppresses the density V.-> 5. **Fill-binning bucket slack.** Items bin to slot c+1 at `x ≥ peaks[c+1].0 − bucket_pt` (was `≥ center`): peak centers are bucket-quantized, so doc 166's right column at x=167.3 vs center 169 binned entirely into slot 0 (`band_counts=[7,0]`) and the gate never saw the sparse column.->-> **Per-doc:** opendataloader 38 MHS 0 → 0.887 + NID +0.152 (the `# 6.2. Expectations for Re-Hiring Employees` page now reads as clean 2-col prose with the heading isolated), 189 MHS 0 → 0.958 + NID +0.192, 15 NID +0.190, 37/36/185 MHS +0.10–0.12; 166/64/80/157/39 all restored byte-identical-or-better after the refinements above. **Accepted residuals (net −0.5 vs +4.8 wins):** docs 200/147/182 — slide-style feature tables with long wrapped paragraph cells where the baseline's *scrambled* interleaved table happened to earn TEDS partial credit and the new clean column prose doesn't (147 NID +0.085 while TEDS −0.165). That family is genuinely ambiguous with N-col prose at the projection layer; it's the known Mode-5 multi-line-cell lever, not a gating bug.->-> **vs nutrient: Overall gap −0.0218 → −0.0157 (0.8694 vs 0.8851); NID 0.9115 vs 0.925, MHS 0.8032 vs 0.819, TEDS 0.6678 vs 0.708.** olmOCR multi_column gap to pymupdf4llm −7.2 → −2.9pp; olmOCR table_tests 46.7% now ahead of pymupdf4llm 46.2%. ParseBench TRM gap to pymupdf4llm −0.013 → −0.008 (0.2123 vs 0.2205). 190 lib tests pass.->-> **Triage workflow note:** the opendataloader `history/<date>` archive is keyed by date — a same-day re-run **clobbers the morning's archive**, so the cross-region baseline had to be reconstructed from a stash-built binary (`git stash push -- projection.rs` → build `lit-base` → pop). A/B per-doc against a baseline *binary*, not the history archive, when iterating same-day.--> **2026-06-10 cross-region table re-merge — opendataloader Overall 0.8602 → 0.8633 (TEDS 0.6514 → 0.6716, MHS 0.7807 → 0.7865, NID +0.0006); ParseBench table grits_con 0.5188 → 0.5481 (+0.029 A/B-attributed), TRM 0.1978 → 0.2083, composite 0.3682 → 0.3886; olmOCR table_tests 44.2 → 45.5% (+13 tests).** Targeted the "V-cut slices a table into sibling leaves" failure (docs 146/64 on opendataloader, the deferred cross-region lever from the 06-09 cluster-merge entry).->-> **New pass: `markdown_layout/cross_region.rs`**, run in `classify_page_with_filters` *before* region grouping (up to 3 merges/page). Pipeline:-> 1. **Candidate pair**: two leaves either x-disjoint (gap ≥ 8pt) or direct region-tree siblings with x-overlap < 40% of the narrower side (straddle-widened V-cut slices, e.g. Federal Register), overlapping ≥ 50% in y. Extended to all leaves predominantly on one side of the cut inside the pair's **union** y-band (a side may itself be H-cut — 146's Area column is two leaves). Set must tile a contiguous line range.-> 2. **Gates** (each killed a real misfire): TOC pages skipped (doc 155 — split number/title/page-number leaves fuse into a convincing fake table); **either**-side-prose reject where prose = ≥3 lines, median-width/side-width ≥ 0.7, vertical fill ≥ 0.55, <30% multi-cell lines (doc 156 body+sidebar, doc 183 infographic; the multi-cell clause keeps sliced *wide* tables whose row-halves also fill their side); ≥3 baselines aligned across the cut and ≥ 40% of the sparser side (doc 43 infographic %-labels).-> 3. **Fuse**: same-baseline lines concatenate spans (cell boundaries fall out of span gaps in `split_cells`), zone gets the cut-parent `region_path`.-> 4. **Validate or drop**: (A) re-run `detect_tables` on the fused lines — accept if covering runs ≥ 60% of rows AND <30% of cells are ≥5-word "cells" (the long-cell gate killed the newspaper/references-column misfires: olmOCR 20_pg35, 0208fbb5 family — fused prose aligns into convincing grids but its cells are sentence fragments); else (B) direct 2-col path for label/description shapes whose list-marker labels (`1. Embodying…`) `is_label_like` rightly rejects in normal flow — V-cut geometry + aligned rows substitute for the label gate; rows anchored on left-side lines, wrap-joins for labels and descriptions, header promoted when bold/short. Validated runs are threaded into `classify_region` as precomputed borderless runs so emission is guaranteed; a failed candidate leaves the page untouched.->-> Kill-switch `LITEPARSE_DISABLE_CROSS_REGION_TABLES=1`, debug `LITEPARSE_DEBUG_CROSS_REGION=1`. 190 lib tests pass (5 new).->-> **Per-doc:** opendataloader 64 TEDS 0.511 → **1.000** (PORT SHIPCALLS 3-col), 146 TEDS 0.0 → 0.358 + MHS 0.317 → 0.942 + NID +0.055 (Area/Competence emitted exactly); per-doc diff vs 06-09 history: **+2/−0 on every metric** across all 200 docs. olmOCR same-binary A/B: table +13, multi_column −2 net / long_tiny_text flat (the residual "regressions" are pages where a genuine sliced table now emits — e.g. 0208fbb5 Spoil-Attribute ± SE table — trading an order test for table wins), Overall flat 37.6-37.7%.->-> **vs competitors after this session:** ParseBench table composite **0.3886 vs pymupdf4llm 0.3726 — now ahead**; grits_con 0.548 vs 0.499; TRM 0.208 vs 0.221 (last remaining table deficit). opendataloader Overall 0.8633 vs nutrient 0.8851 — gap now NID 0.907 vs 0.925 + MHS 0.787 vs 0.819; TEDS 0.672 vs 0.708.->-> **Residuals / next:** (a) doc 147-style 4-col tables with long wrapped paragraph cells — candidate forms, validation correctly refuses; needs the multi-line-cell row-merge signal (Mode-5). (b) Federal Register `1653739079_page34` — sibling pair now forms but the leaves mix column prose *above* the table band, diluting validation; needs sub-leaf y-banding (merge only the table band, not whole leaves). (c) doc 116 — slices fully overlap in x (item-level histogram partition), outside this geometry. (d) doc 43-family infographics stay rejected (good).-> **2026-06-09 (pm) MHS/heading session — opendataloader Overall 0.8275 → 0.8602 (MHS 0.698 → 0.781, NID 0.890 → 0.907, TEDS flat 0.651); olmOCR table_tests 43.0 → 44.2, multi_column 59.5 → 60.0.** Five changes, descending impact:->-> 1. **Dropped the `<!-- page N -->` markers from markdown output** (`output/markdown.rs`; `-----` separator kept). The opendataloader MHS evaluator builds a flat heading/content tree (**heading levels are ignored entirely** — `evaluator_heading_level.py` tags nodes only `heading`/`content`); on small trees (GT = 1 heading ≈ 4 nodes) the comment created a pre-heading content node costing ~0.25/doc across ~30 docs. Simulated +0.058 MHS / +0.004 NID before touching code, confirmed by bench. **olmOCR side effect:** headers_footers drops 58.9 → 57.4 — verified by re-prepending the comment to converted outputs (59.3) that the comment was *displacing un-stripped chrome out of the windowed `absent` checks*, i.e. an accidental benchmark hack we no longer benefit from. The honest recovery is better chrome stripping (the known Tel/email/letterhead catalog gaps).-> 2. **Off-page text clip** (`extract.rs`, end of `extract_page_text_items`): drop items entirely outside the page view box (print-spread/imposed PDFs carry the neighbouring page's text at x > page width in the same content stream; viewers never show it). Fixes the doc 28/29/31 family (sections from the *next* page emitted + heading-promoted) and strips prepress slug chrome on olmOCR hf docs. Partially-visible items kept; kill-switch `LITEPARSE_DISABLE_OFFPAGE_CLIP=1`.-> 3. **Ruled-table straddle guard** (`tables.rs`): reject a ruled-grid candidate when >45% of its spans (≥6) cross interior column boundaries by ≥3pt on both sides — decorative slide/layout boxes over flowing text (doc 198, a TOC slide scrambled into a 4-col table via the spine bypass) slice through prose runs; real ruled tables keep text inside cells. Zero hits across all 503 ParseBench table pages; doc 198 MHS 0.0 → clean `# Contents` + list. Kill-switch `LITEPARSE_DISABLE_STRADDLE_GUARD=1`.-> 4. **Two heading run-in guards** (`classify.rs` / `headings.rs`): (a) `": "` joins `". "` in the bold-heading run-in rejection (`Parameter Model: We have released the SO-`); (b) size-promotion suppressed when the *previous* line ends in a mid-word hyphen wrap regardless of capitalization (the existing guard required starts-lowercase). Doc 195: 6 → 4 headings, exact GT match. Kill-switch `LITEPARSE_DISABLE_HEADING_GUARDS=1`.-> 5. **Bare-HR suppression** (`output/markdown.rs`): a page whose only surviving blocks are horizontal rules (all text stripped as chrome) renders empty instead of stacked `---`.->-> **Regression checks:** olmOCR Overall flat 37.7% (gated changes alone are +1.4 table / +0.5 multi_column; the hf −1.5 is the artifact above, baseline 99.9% is the same artifact on one chrome-only page). Full ParseBench post-session: text_content 0.6492 (+0.001), layout 0.8374 (flat), text_formatting 0.3755 (+0.005), chart 0.0335 (flat). ParseBench table grits_con measured 0.5188 vs the 0.5268 recorded this morning — **proven not caused by today's changes**: A/B with all kill-switches over all 503 table pages = byte-identical markdown, and re-injecting the page comment into stored results + `--skip_inference` reproduces 0.5188 exactly. The morning 0.5268 is another instance of the recorded-score/binary drift already flagged in the sweep note. 185 lib tests pass.->-> **MHS gap vs nutrient after this session: −16.2 → −8.2 doc-points (0.781 vs 0.819).** Remaining head: docs 38/189 (bold heading fused inline across an unsplit 2-col layout — col-fill-gate territory in projection.rs), 103/36/184 (infographic line fusion), 146 (cross-region table: V-cut splits the Area/Competence table into two leaves so no detector sees the rows whole). opendataloader overall 0.8602 vs nutrient 0.8851 — remaining gap is NID 0.907 vs 0.925, MHS 0.781 vs 0.819, TEDS 0.651 vs 0.708.-> **2026-06-09 full benchmark sweep — baseline snapshot + competitor comparison (no code changes).** Re-ran all three benches from fresh artifacts (`lit` + root-venv Python binding rebuilt), including the first *full* ParseBench run (all 5 groups — runs fine without `ANTHROPIC_API_KEY`; rule-based eval only). Competitor candidates rescored on olmOCR from their existing converted markdown.->-> **Current state — liteparse:**->-> | Bench | Metric | Value |-> |---|---|---|-> | olmOCR | Overall / multi_column / table_tests / headers_footers / long_tiny_text | 37.7% / 59.5% / 43.0% / 58.9% / 26.9% |-> | opendataloader | Overall / NID / TEDS / MHS | 0.8275 / 0.890 / 0.651 / 0.698 |-> | ParseBench table | GriTS_con / TRM / composite | 0.5268 / 0.1985 / 0.3737 |-> | ParseBench (new groups) | text_content / layout / text_formatting / chart | 0.648 / 0.837 / 0.371 / 0.032 |->-> (ParseBench GriTS_con reads 0.5268 vs the 0.4389 recorded in the cluster-merge entry below; the pre-existing top-level report in `output/liteparse_markdown/` already showed 0.5266 before this sweep, so the jump came from work landed between that entry's bench run and today's binary, not from this re-run.)->-> **vs model-free competitors:**-> - **olmOCR Overall: #1.** liteparse 37.7 > pymupdf4llm 32.9 > opendataloader 32.7 > pdf_inspector 30.5 > markitdown 28.7. Per-category gaps remaining: multi_column 59.5 vs pymupdf4llm 66.7 (−7.2pp), table_tests 43.0 vs 46.2 (−3.2pp), long_tiny_text 26.9 vs opendataloader 35.1 (−8.2pp). Only candidate at 100% baseline.-> - **opendataloader-bench: #3 model-free**, effectively tied with opendataloader (0.8275 vs 0.8312; we lead TEDS 0.651 vs 0.489, trail NID −0.013 and MHS −0.042). **nutrient 0.885 is the one model-free engine still clearly ahead** — gap is NID (0.925 vs 0.890) and MHS (0.819 vs 0.698), i.e. reading order + heading structure, matching the known 2-col-heading / XY-cut failure modes. Others: pymupdf4llm 0.732, markitdown 0.589.-> - **ParseBench table:** GriTS_con now beats pymupdf4llm (0.527 vs 0.499); record-match still behind (0.199 vs 0.221); composite tied (0.374 vs 0.373). text_content ties opendataloader (0.648 vs 0.648), beats pdf_inspector (0.580). No competitor runs exist yet for layout/formatting/chart.-> - **text_formatting weak spots** (first measurement): code blocks 0%, sup/sub/underline/strike/mark ~0%, title pass 0.44, title hierarchy 0.37, bold 0.51, italic 0.37. chart 0.032 is expected (no chart-data extraction).->-> **Highest-leverage next targets:** MHS/NID vs nutrient (2-col heading isolation, reading order), olmOCR multi_column, ParseBench record-match + title hierarchy.--> **2026-06-09 union-track cluster merge — ParseBench grits_con 0.4308 → 0.4389, TRM composite 0.2887 → 0.2947, unmatched predicted tables −26%; olmOCR & opendataloader regression-clean.** Triage showed 394 of ParseBench's 425 unmatched predicted tables were fragments of real tables (runs broken at column-count changes, stranded PDFium-merged rows, 56% of emitted tables headerless). New `merge_fragmented_table_clusters` post-pass in `markdown_layout/tables.rs` re-derives union column tracks across adjacent table runs and re-extracts every line against them; `union_header_from_above` bins stacked/colspan headers per-span with replication. Track-compatibility gate (≥75% of narrower run's tracks must map into wider run's) prevents fusing genuinely separate stacked tables. Env kill-switch `LITEPARSE_DISABLE_CLUSTER_MERGE=1`. Full details + tried-and-reverted notes (gap-based wrap merge has no signal; opendataloader TEDS baseline correction 0.5917 → 0.568) in FAILURE_MODES.md fix log. Next table levers: multi-line-cell row merging (needs ruled/label-anchor signal, not gaps) and cross-region fragments (V-cut sliced tables).--> **2026-06-04 inferred-tracks table detector — olmOCR-bench table_tests 38.8% → 39.5% (+0.7pp, +7 tests), Overall 35.1% → 35.2%; opendataloader & ParseBench unchanged.** Targeted the dominant `wrong_neighbors` failure on financial/earnings tables (e.g. Apple/Wiley `a769f9a3…pg2`, `fa18a15c…`, hundreds of similar narrow-numeric tables): the existing `try_detect_table` derives column tracks from the FIRST row's `split_cells` output, which uses `TABLE_CELL_GAP_FONT_MULTIPLIER × font_size` (= 14pt at 14pt body) as the inter-span gap threshold. The Apple Revenue Q3 row has `$448 → $427` and `$427 → 7%` gaps of 13.9pt — *just under* the threshold — so its 5 distinct PDFium items collapse to 3 cells (`$448 $427 7%` lumped). Every row in the Revenue table fails identically, so the table establishes as 3-col, and the cross-run merge can't recover (`merge_consecutive_table_runs` only handles `|A|+1 == |B|`).->-> **New `try_detect_table_inferred` (`markdown_layout/tables.rs`).** Runs in `detect_tables` *before* `try_detect_table`. Steps:-> 1. **`infer_tracks_from_raw_items`**: walks forward from `start_idx` while `table_rows_adjacent` holds (cap `TABLE_TRACK_INFERENCE_MAX_ROWS=12`), collects `span.x` for every non-whitespace span on each row with ≥2 spans, then single-link clusters at `TABLE_TRACK_TOLERANCE_PT` (6pt). The Apple Q3 row alone exposes all 5 column anchors (49.5/119/152.6/186.2/253.3); even with PDFium-level merge on 9 Months row, the union across rows recovers the full geometry.-> 2. **Track-soundness guards** (all required): `tracks.len() ≥ TABLE_MIN_COLUMNS`, `tracks.len() > baseline_cells.len()` (only re-seed if we'd actually unlock more columns), and `min(inter-track gap) ≥ max(1.5 × seed_font_size, 12pt)` — rejects inferred "tracks" that are really inter-word positions in prose (word gaps are typically <1× font size).-> 3. **Strict seed-row check**: the seed (`lines[start_idx]`) must have `≥ tracks.len()` raw PDFium spans AND each span must single-cover (no track ranges where one span x-extent overlaps ≥2 anchors). This is the load-bearing false-positive guard — see "Seed strictness" below.-> 4. **`cells_from_raw_items_with_tracks`** per row: walk raw items, assign each to its nearest track within tol; for items whose `[x, x+width]` covers multiple tracks AND whose `x` anchors at the leftmost covered track (within tol), generalize `split_merged_at_anchor` to N anchors — pick whitespace positions whose linearly-interpolated x is closest to each subsequent track. Rejects rows with <2 PDFium spans (single-span lines spanning the track region are prose, not merged-numeric rows). Returns `Some(cells)` of length `tracks.len()` (some cells may be empty for sparse rows).-> 5. **Cell-text dedup via existing absorb path**: re-uses `absorb_header_lines` and the bold-first-row promotion already in `try_detect_table`. CV check via `row_spacing_cv` rejects runs with irregular cadence and falls back to the existing path.->-> **Seed strictness was load-bearing.** The first cut required only `seed_spans + 1 ≥ tracks.len()` (one multi-cover split allowed in seed). That preserved +16 table tests but regressed multi_column by -5 — the failure was 2-column prose like `027c39…page_2` where PDFium emits "in the" and "study. The subjects including 343 males and 293" as two spans on one baseline. The first span at `x=56.7` width-extent reaches past `x=84.2` (the second span's anchor), so my detector saw "in the" as covering tracks 0 AND 1, split it at whitespace, and shredded the line into `| in | the study. The subjects... | males and 293 |` — a fake table. Tightening to `seed_spans ≥ tracks.len()` AND no multi-cover spans in the seed eliminated the regression (multi_column held exactly at 49.4%, no `order` tests lost) while still catching Apple's Q3 row (5 spans, all single-cover at 5 tracks). The trade: 9 of the 16 initially-won table tests were lost (cases where the table's first row had merged spans), but the +7 retained tests are net positive with zero collateral.->-> **Per-PDF verification on Apple Wiley `a769f9a3…pg2`**: before, `| Q3 | $448 $427 7% | 5% |` and `| 9 Months | $1,298 $1,263 5% | 3% |` (3-col, merged); after, `| Q3 | $448 | $427 | 7% | 5% |` and `| 9 Months | $1,298 | $1,263 | 5% | 3% |` (5-col, split). The 9 Months row's PDFium-merged `$1,298 $1,263 5%` single span gets split at whitespace positions nearest each downstream track anchor. Adjusted EPS and Reported EPS sub-tables retain their existing correct 5-col emission.->-> **Bench-by-bench:**->-> | Bench | Metric | Baseline | After | Δ |-> |---|---|---|---|---|-> | olmOCR | Overall | 35.1% | **35.2%** | +0.1pp |-> | | `table_tests` | 38.8% (397) | **39.5% (404)** | **+0.7pp, +7 tests** |-> | | multi_column / order / headers_footers / baseline / long_tiny_text | — | — | flat |-> | opendataloader | NID / TEDS / MHS / Overall | 0.881 / 0.603 / 0.631 / 0.800 | (same) | flat |-> | ParseBench | table (GriTS / TRM / composite) | 0.458 / 0.130 / 0.304 | (same) | flat |-> | | chart / layout / text_content | — | (same) | flat |-> | | text_formatting | 0.3212 | 0.3209 | -0.03pp (noise, <0.5%) |->-> **Why ParseBench didn't move** (same as description-list pass): GriTS + Table Record Match score against full multi-level header chains. The fix correctly emits one more column on the targeted tables but doesn't reconstruct multi-row header layers (Aug 21 / 2011 / (percent) on `fa18a15c`), which is what the bench's `top_heading` lookups require. Need the upward header absorption fix to move that needle.->-> **Why opendataloader didn't move**: its `try_detect_description_list` path already covers the 2-col borderless shape that's the dominant failure mode on that corpus. The strict-seed inferred path doesn't fire on description-list layouts (col 0 is single span, col 1 is single span — only 2 PDFium spans for 3+ tracks would never happen because there's no 3rd column).->-> **Why olmOCR only +7 instead of the +40-80 originally projected**: most of `wrong_neighbors` failures aren't the Apple shape. The B2 vertical-baseline-merge case (where PDFium fuses 3 stacked baselines into one span) needs a different fix entirely — vertical detection of intra-span baseline boundaries, not lateral anchor splits. The strict-seed guard also rejects tables where *every* row has at least one merged-span column (rare in practice but exists). The `+7` we got are tables where at least one row has the Apple shape — every column a distinct PDFium item.->-> **Files**: `crates/liteparse/src/markdown_layout/tables.rs` only — added `TABLE_TRACK_INFERENCE_MAX_ROWS`, `infer_tracks_from_raw_items`, `cells_from_raw_items_with_tracks`, `split_span_at_anchors`, `try_detect_table_inferred`, wired into `detect_tables` ahead of `try_detect_table`. No projection-layer changes. CLI builds clean; lib-test compile is broken project-wide on this branch by pre-existing local changes in `ocr_merge.rs` / `projection.rs` (unrelated). No new tests added yet — would follow up with unit coverage of the seed-strictness guard, the multi-anchor split path, and the 027c39-style false-positive guard.->-> **Deferred — next leverage**:-> 1. **B2 vertical wrapped-row split** — the *other* half of the original "+40-80" projection. Needs a "this single PDFium span contains multiple stacked baselines" detector — symmetric to the lateral fix but different signal (internal baseline-gap signature). Targets the financial earnings tables where PDFium fuses 3 stacked numbers into one span.-> 2. **Multi-level header upward extension** in `absorb_header_lines` — moves ParseBench. The strict-seed guard here doesn't help the header-row case; needs the `cells.len() > column_count` slack with whitespace-split recovery for header lines.-> 3. **Relax seed strictness in the inferred path** if a separate prose-line discriminator can be found — would recover the lost 9 table tests. The CV-of-cell-widths guard (CV ≥ 0.7) was tried and rejected (cost 6 table tests without fixing the multi_column regression). The line-numbered academic paper case (e.g. `05eac72b…page_6`) is a *pre-existing* failure of the existing `try_detect_table`, not introduced by this fix — would need a separate "table inside vs across page columns" geometry check.--> **2026-06-04 description-list 2-col table detector — opendataloader TEDS 0.5527 → 0.6028 (+0.050), Overall 0.7953 → 0.8000 (+0.005); olmOCR-bench table_tests 35.8% → 38.8% (+3.0pp, +31 tests), Overall 34.7% → 35.2%; ParseBench unchanged.** Targeted the 9 opendataloader docs at TEDS=0 — borderless 2-column "label | description" tables that the existing `try_detect_table` rejected because `TABLE_MIN_COLUMNS = 3`. Five of the nine zeros (132 species, 146 framework, 150 competence, 166 CEC, 197 task names) have this shape; 110 is OCR-gated, 121/122 are upstream PDFium item-order garble, 165 is a sparse fill-in worksheet with one-cell body rows.->-> **New `try_detect_description_list` (`markdown_layout/tables.rs`).** Runs after `try_detect_table` in the `detect_tables` loop. Conditions:-> 1. **Layout shape**: exactly 2 cells, stable col-0 / col-1 anchors (±`DESC_LIST_TRACK_TOL_PT = 8pt`), inter-column gap ≥ `DESC_LIST_MIN_COL_GAP_PT = 12pt`.-> 2. **Col-0 label discriminator (`is_label_like`)**: 1-4 words, ≤40 chars, no internal `". X"` sentence boundary, not a pure bullet glyph (`•`, `●`, `*`, `-`, `‣`, `⮚`, …), not a list marker (`1.`, `(1)`, `i.`, `IV.`). Filters bulleted lists (doc 056 biomass categories) and prose-mid-sentence false positives.-> 3. **≥2 real 2-cell rows** (`real_two_cell_rows`). A run extended *only* via the merged-span split path almost always slices body prose where a heading like `7 Variants of sj Observer Models` happens to have a tab-stopped section number — that anchored a fake table on docs 001-014 in the first cut. Real description-list tables have ≥2 baselines where PDFium emitted two distinct spans.-> 4. **Cadence OR asymmetry**: ≥3 rows OR one row's col 1 is ≥ 2× col 0. A single 2-row pair of symmetric short cells is too close to projection-merged adjacent-column text.-> 5. **At least one alpha+alpha row**: filters footnote lists where col 0 is always a digit (doc 008's `25 | Wiliam Beckford, …`).-> 6. **Not all-page-ref**: `is_page_ref` rejects runs where every col 1 is a digit or roman numeral (doc 171/016 TOC entries). Required because TOC items like `Acknowledgments | vii` pass the alpha+alpha check (`vii` is alphabetic).-> 7. **Caption guard**: `looks_like_caption` breaks the run before a line beginning `Figure N` / `Table N` (doc 132's `Table 6.1: Four fish species…` was being whitespace-split into a bogus row in v1). Combined with a **cadence guard** (next y-gap > 1.6× the run's max prior gap) so the visually-detached caption never gets absorbed.->-> **PDFium tight-kerning recovery (`split_merged_at_anchor`).** Doc 132's middle rows (`La Palma Pupfish Cyprinodon longidorsalis` / `Butterfly Splitfin Ameca splendens`) emit as a single PDFium text item spanning both columns — kerning between col-0 end and col-1 start is tighter than the gap threshold. When a 1-cell line sits at col-0's anchor and extends past col-1's anchor (`end_x > col1_x + tol`), split on the whitespace position whose linear x-estimate is closest to col-1's anchor. Recovered cells must themselves be label-like to be accepted (prevents lossy splits of prose body lines that happen to straddle).->-> **Per-doc deltas on opendataloader-bench**: doc 132 0.000 → 0.867 (+0.867), doc 166 0.000 → 0.849 (+0.849), doc 150 0.000 → 0.387 (+0.387). Zero regressions ≤ -0.03 across all 200 docs. Aggregate: NID 0.8807 → 0.8808, TEDS 0.5527 → **0.6028**, MHS 0.6275 → **0.6307**, Overall 0.7953 → **0.8000**.->-> **olmOCR-bench delta**: table_tests 35.8% (366/1022) → 38.8% (397/1022). Multi-column / headers_footers / long_tiny_text / baseline all held. The +31 tests are mostly borderless 2-col definition/glossary shapes in `table_tests.jsonl` that the column-alignment detector previously skipped.->-> **Why ParseBench was byte-identical**: its `table` evaluator scores GriTS + Table Record Match on multi-column enterprise tables (financial statements, insurance forms, government reports). Description-list shape doesn't match the eval's expected output structure, so the detector firing or not firing on those PDFs doesn't change the metric.->-> **Remaining TEDS=0 docs (4 of original 9)**: 110 (OCR-gated — table is a raster image, no text extraction possible), 121/122 (PDFium item-order garbled — `extract.rs` / projection issue, not classifier), 165 (sparse fill-in worksheet: every body row has only the col-0 label cell populated, right cells blank — the `cells.len() == 2` guard rejects all body rows; would need a "label-column spine" bypass like ruled-table sparse handling but for borderless). Plus 146/197 still TEDS=0 — 146's "Area | Competence" table has col-1 anchor that drifts between rows because of bullet indenting; 197 is a 1-column list rendered as a 1-col table in GT, different shape entirely.->-> 168 lib tests still pass; no new tests added yet (will follow up with unit tests covering the 132 merged-span case, the 150 paragraph-cell wrap, the 001 false-positive guard, and the 171 TOC filter).--> **2026-06-03 heading/caption shredding — olmOCR-bench multi_column 48.4% → 49.5% (+1.1, ~+10 tests), order pass-rate 40.3% → 41.3%, overall 34.6% → 34.8%.** Targeted the `heading/caption shredding 72` sub-bucket left after the spanning-line fix. Two distinct sub-causes, two fixes:->-> **(a) body-size mis-detection (`compute_body_size`, `headings.rs`).** `compute_body_size` char-weights font sizes and picks the single mode. On dense pages where a small-print block (references / acknowledgments) out-weighs the narrative by raw char count, the mode lands on the *small* size, so all real body prose measures *above* body and every line is promoted to `##`. Example: `multi_column/02e31…page_5` has ~60 lines at 7pt (refs: "We are grateful to Trevor Ireland…", "Armstrong, J.T. (1988)…") vs ~58 lines at 9pt body; char-weighting picked 7pt. **Fix:** added `BODY_CODOMINANT_FRACTION = 0.5` — instead of the single mode, take the *largest* size whose char-weight is ≥ 50% of the top weight. The 9pt narrative now wins (or ties) and stays body. Verified `test_heading_and_paragraph` still holds (body 10pt, title 18pt: 18pt weight far under 50%, doesn't qualify).->-> **(b) caption shredding (`classify.rs`).** Photo captions set one size above body (e.g. 15pt over 11pt body in `multi_column/0b045…page_18`: "ABOVE: Maj Gen Michael Ramantswana, Chief / Military Policy, Strategy and Planning, laid a / wreath on behalf of…") are genuinely heading-sized, so each *wrapped* line became its own `###` block — a 4-line caption shredded into 4 headings. **Fix:** a heading-emit branch now merges a wrapped continuation into the heading directly above when (1) same level, (2) the heading is still `blocks.last()` (adjacency — never reaches across an intervening block), and (3) `continues_paragraph(prev, line)`. A real section heading is followed by body text, which breaks the run; only a genuinely multi-line heading/caption merges. Joined via the existing `append_inline_continuation` (de-hyphenates). New `heading_run: Option<(u8, ProjectedLine)>` state. Bonus: legitimate 2-line wrapped *titles* now also collapse to one heading instead of two. The caption now emits as one `### ABOVE: Maj Gen … laid a wreath on behalf of the Chief of the SANDF, Gen Solly Shoke.`->-> **Bench (1403 PDFs, no OCR): multi_column 428→438/884, order 40.3%→41.3%, overall 34.6%→34.8%, no regressions** (baseline 100%, table_tests 35.8%, headers_footers 58.6%, long_tiny_text 21.0%, old_scans 13.3% all held). 168 lib tests pass. **Remaining multi_column both_missing levers:** the OCR-gated 43 (image-only PDFs, need OCR pass) and garbled-extraction 18.->-> **2026-06-03 spanning-line H-cut rescue — olmOCR-bench multi_column 44.3% → 48.4% (+4.1, +36 tests), overall 34.1% → 34.6%.** Targeted the dominant multi_column failure: `both_missing` order tests (269 of 492 failures). Triage of that bucket: **interleaving 136** (XY-cut never split the columns, reading order zigzags col1+col2 line-by-line), **heading/caption shredding 72**, image-only/OCR-gated 43, garbled-extraction 18. This entry fixes the interleaving 136.->-> **Root cause.** On 2-column papers a *centered full-width line* (authors / affiliation) sits above the body inside one region leaf. Its x-extent straddles the gutter, so the density V-cut sees no valley and can't split the columns; `build_projected_lines` then merges col1+col2 on shared baselines (e.g. "Abstract—This paper considers the problem of designing **surface in the presence of bounded** …"). The H-cut that would *peel* the spanning line exists, but `xy_cut_rec`'s `XY_MIN_LINES_PER_H_SIDE` guard rejects it because the spanning line is a single line, and the depth-0 banner bypass doesn't apply mid-recursion. Critically, the spanning line is often *just under* the 60% banner-width threshold (the IEEE authors line measured 290pt vs a 303pt threshold), so it's not even classified as a banner — the peeling H-cut has a **finite** density score.->-> **Fix (`projection.rs`).** Added a `spanning_rescue` branch to the min-lines guard in `xy_cut_rec`: when one side of an H-cut is a single line (`thin`) and the other side (`wide`) genuinely splits into columns (`column_cut_x` — density V-cut OR `xy_find_column_cut` histogram returns a gutter x), AND the thin line's x-extent *crosses* that gutter, allow the cut despite the min-lines guard. The "thin line crosses the gutter" test is the discriminator that distinguishes a column-straddling banner (peel it) from a single-column caption in running prose (leave it) — a caption sitting inside one column never crosses the revealed gutter. Not gated on infinite banner score, so it catches the sub-banner-width case. New helpers `column_cut_x` (returns the gutter position) and `x_extent`. 2 new unit tests (`spanning_line_above_columns_is_peeled_to_reveal_gutter`, `single_column_caption_is_not_peeled_into_columns`); 168 lib tests pass.->-> **Verified on `multi_column/0c6c8f76…page_1` (IEEE 2-col paper):** before, the entire body was one region leaf `[1,1]` with every line reading col1+col2 interleaved; after, the recursion peels the authors line at depth 2 then V-cuts the body into two column leaves at depth 3, and the Introduction reads as contiguous prose ("In continuous time, a sliding mode is generated by means of discontinuities in the control signals about a surface in the state space [27]…" — the bench's `before` string). **Bench (1403 PDFs, no OCR): multi_column 392→428/884, order pass-rate 36.9%→40.3%, no regressions** (table_tests 364→366, headers_footers 445, long_tiny_text 92, old_scans 70, baseline 100% all held). **Remaining multi_column both_missing levers:** heading/caption shredding (72, classify layer — captions like "ABOVE: …" promoted to `###` and block-split), then the OCR-gated 43.->-> **2026-06-03 estimated-font-size heading guard — olmOCR-bench multi_column 43.4% → 44.3% (+0.9, +8 tests), overall 34.0% → 34.1%.** Targeted the multi_column "text missing" cluster (both_missing 274, before_missing 117, after_missing 106). Root cause was `dominant_font_size` bbox-height jitter: when PDFium bakes the font size into the text matrix (Tf operator reports ~1.0), `build_projected_lines` falls back to char-weighted `bbox.height`, which jitters a point or two line-to-line on the *same* body paragraph (9.0 vs 10.0). That jitter manufactured a bogus heading level in `build_heading_map`, which shredded body paragraphs into per-line `####` headings and blocked de-hyphenation, so the bench's contiguous `before`/`after` strings never matched.->-> **Two-part fix.** (a) **`font_size_is_estimated` flag** (`types.rs` + `projection.rs`): the height-fallback path now tags the line so downstream code knows the size is approximate. `build_heading_map` (`headings.rs`) uses a wider margin over body for estimated sizes (`ESTIMATED_HEADING_SIZE_MARGIN = 1.5pt`) vs the tight `HEADING_SIZE_EPSILON` for real Tf-reported sizes — a 1pt height wobble no longer clears the bar. (b) **continuation+lowercase guard** (`classify.rs`): font-size heading promotion is suppressed when the line starts lowercase AND `continues_paragraph(prev, line)` holds, so a wrapped body line that happens to measure a hair taller can't become a heading mid-paragraph. Real headings (larger by a clear margin, or not mid-paragraph) survive both guards.->-> **Verified on `multi_column/09f90a8fad0997f7cf454cbcbe79cab3bc0f_page_1_pg1.pdf`:** paragraphs now read as coherent prose ("Textual Question Answering (Q/A) aims at identifying the answer of a question in large collections…" — the bench's exact `before` string) with de-hyphenation restored and `####` shredding gone, while real headings (`# Experiments…`, `### 1 Background`, `### 2 The NLP Techniques`) are preserved. **Bench (1403 PDFs, no OCR): multi_column 384→392/884, no regressions elsewhere** (table_tests 364, headers_footers 445, long_tiny_text 92, baseline 100% all held). Overall moves only +0.1 because multi_column is one of 8 per-JSONL categories and the big residual buckets (math 0% — no LaTeX emission; old_scans/long_tiny_text — OCR-gated image-only PDFs) are untouched by this layout fix. 166 lib tests pass. Also dropped an unused `mut` warning in `tables.rs:591` that surfaced during the rebuild.->-> **2026-06-03 absorb_header_lines relaxation attempt — reverted (small regression).** Tried two extensions of `absorb_header_lines` to chase the `fa18a15c` NASS-style multi-level header that Step 1's merge couldn't fully reach. (a) When a candidate header row has `cells.len() > column_count` (e.g. an embedded `|` token splits off as an extra cell), filter to the subset whose centers map to distinct body tracks and accept if `>= 2` survive. Direct unit tests on `absorb_header_lines` pass — the filter correctly drops the off-track `|` and absorbs the remaining 3 cells. **Bench regressed by 2 tests** (table_tests 364 → 362). The real-world failure mode: some PDFs have a "wider" line above the body whose 4-5 cells happen to align to 3 body tracks but is actually prose / a different table fragment / a sub-title, and folding it as a header changes downstream cell relationships. Reverted. (b) Allow single-cell header absorption when the cell maps to a track (the "Item" col-0 header case). Broke the pre-existing `does_not_absorb_single_cell_title_above_body` test — page titles like `"Results"` above a body where the body's track 0 sits at the same x as the title would get folded as a header. Reverted. The fundamental problem: the lines above a body table look statistically very similar to legitimate single-cell column headers vs page titles / prose / unrelated content — no robust heuristic. **Conclusion: header-absorption alone can't bridge the multi-level-header case** because the date row above the `(percent)`/`2011` chain in `fa18a15c` is in a *different XY-cut leaf* (the body's region and the title-area region were split early in projection), so absorption can't even reach it sequentially.->-> **Next leverage candidates** (changed from prior pass):-> 1. **B2 vertical wrapped-row split** — `recover_merged_cell`'s vertical analog for the Apple-earnings `Q3 | $448 $427 7% | 5%` case (3 cells where 5 expected, because PDFium merged 3 stacked baselines into one span). Detection: cells whose text contains internal baseline-gap signatures, OR cells whose width / token-count ratio is much higher than peers in the same column.-> 2. **Description-list → 2-col table** — `af3fd8e2…pg15` (`Option Description` header + `**Term**` body + paragraph) shape. Detection: consecutive lines where col 0 is short + bold + indent-stable and col 1 is a paragraph at a stable x0. Lower architectural risk; targets the no_table_cell_in_text bucket (154 tests).-> 3. **Region-aware header absorption** — let `absorb_header_lines` walk past a layout-boundary line if the line above it (in projection y-order, ignoring region) aligns to body tracks. Higher risk but the only way to bridge the date-row case in `fa18a15c`.->-> **2026-06-03 cross-run table merging — bench-neutral but structural cleanup.** Added `merge_consecutive_table_runs` post-pass to `detect_tables` in `markdown_layout/tables.rs`. Targets the dominant table_test failure bucket (~252 wrong_neighbors out of 589 non-image table failures): when `try_detect_table` breaks the run at a column-count change, the adjacent runs are merged in two cases. **Case Same**: identical column count + per-column track centers within `TABLE_TRACK_TOLERANCE_PT` → concat rows, keep A's header. **Case Subset**: `|A| + 1 == |B|` and A's tracks are a column-subset of B's tracks under a looser `TABLE_SUBSET_TRACK_TOLERANCE_PT = 12pt` (header cells often have different content widths than body cells) → fold A as B's header layers, joined per-column top-to-bottom with adjacent-dedup, padded with empty at the missing B-column index. Tries every possible "skip one B column" assignment and picks the lowest-total-error mapping; uses min(start_diff, end_diff, interior-half center-in-range) as the per-column score so a narrow header sitting at the edge of a wide row-label cell doesn't spuriously match. Allows up to 1 interstitial "absorbable" line between A.end and B.start (single-cell label, ≤60 chars, not ending in sentence punctuation) — preserved as a body row in col 0 of the merged table so no content is lost. Track derivation in `run_body_tracks` aggregates min/max across all body rows so tight numeric per-row cells still produce track ranges wide enough to match wider headers. Guarded by `TABLE_HEADER_MAX_ABSORB_ROWS = 3` (don't fold a real 5-row table as a header), `TABLE_MERGE_MAX_Y_GAP_LINES = 2.0` (visually distinct tables don't merge), and "don't merge two complete-looking tables across a gap" for Case Same. 7 new tests cover the matrix (same-cols, subset, distant, large-prior, 2-col-diff, GridFallback-untouched, multi-cell interstitial reject, absorbable interstitial as body row). 166 lib + 10 integration tests pass.->-> **Smoke result on `fa18a15c1dbbfcb71b1f1ea1b8f116e24b8a_pg2_pg1.pdf` (NASS crop weather):** confirmed the merge fires — 4-col `2011×4` + `(percent)×4` header table previously emitted standalone is now folded into the 5-col body table as a composite header `| | 2011 (percent) | 2011 (percent) | 2010 (percent) | Average (percent) |` with `Topsoil` preserved as the next body row.->-> **Bench result on olmOCR-bench (1403 PDFs, no OCR): table_tests unchanged at 35.6% (364/1022).** Compared identical builds with merge enabled vs disabled (LITEPARSE_DISABLE_MERGE env flag, since removed). Per-category: every category byte-identical. The merge is correct and fires on the targeted layouts but doesn't flip any test predicates because the bench's `top_heading`/`left_heading`/`right`/`up`/`down` lookups need a *full* multi-level header. On `fa18a15c`, the column-header chain in the source is *3 lines* (`Aug 21,`/`Aug 14,`/`Aug 21,`/`2006-2010` → `2011`/`2011`/`2010`/`Average` → `(percent)×4`), my merge folds 2 of them; the topmost date row sits above the first detected run and is rejected by `absorb_header_lines` (it splits to 5 cells because of an embedded `|` token, exceeding the 4-col body's column count). The bench expects `top_heading='Aug 21, 2011'` but gets `top_heading='2011 (percent)'` — failure reasons changed but pass/fail did not. Same pattern across other wrong_neighbors PDFs sampled. Kept the merge: zero regressions, fires correctly on targeted layouts, the rendered tables ARE structurally cleaner (one table instead of fragmented two), and the infrastructure (`subset_mapping`, `run_body_tracks`, `is_absorbable_interstitial`) is reusable when we tackle multi-level header absorption next.->-> **What would actually move table_tests** (deferred):-> 1. **Upward header extension past `absorb_header_lines`'s break.** Today the function bails on the first line that has `cells.len() > column_count` or doesn't align all-tracks. The date row case has 5 cells (one extra from an embedded `|` token) so it bails. Need either (a) tolerance for "all cells except 1 align" — same `misaligned <= 1` slack the body-row loop already allows, OR (b) a `recover_merged_cell`-style split-on-whitespace for header rows that have more cells than expected. Plausibly +30–60 wrong_neighbors.-> 2. **B2 wrapped-row split (vertical analog of `recover_merged_cell`)** — `a769f9a3…pg2` (Apple earnings) has `Q3 | $448 $427 7% | 5%` because PDFium merged 3 baselines into one span. Symmetric to lateral-merge recovery but harder to detect: need a "this cell has visible internal baseline boundaries" signal. Plausibly +40–80 wrong_neighbors.-> 3. **Description-list / definition-list as 2-col table** — `af3fd8e2…pg15` shape: `**Term**` (bold prefix at col-0 anchor) + paragraph at col-1 anchor → 2-col table. ~30–50 no_table_cell_in_text. Cheapest of the three.->-> **2026-06-03 single-page chrome detector — olmOCR-bench headers_footers 38.9% → 58.6% (+19.7), overall 31.2% → 34.0% (+2.8).** The existing `compute_header_footer_set` is structurally inadequate on the bench's single-page inputs — cross-page repetition can't fire when there's only one page. Added a complementary single-page detector in `markdown_layout/repetition.rs::detect_single_page_chrome(page, body_size) -> HashSet<line_idx>` that runs *additively* alongside the cross-page set. Wired through `format_markdown` → `classify_page_with_filters` as a 6th param so the filter step strips both cross-page-matched lines AND per-page chrome before table detection / heading promotion runs.->-> **Detection signal:** position band (top 15% OR bottom 15% of page) + isolation gap (≥ 1.0 × body_size to the nearest non-chrome-pattern neighbor in y-order) + chrome pattern signature. Patterns covered in `matches_chrome_pattern`: URL/DOI prefixes (`http://`, `https://`, `www.`, `doi:`, `doi.org/`, `dx.doi.org`), journal banner phrases (`please cite this article`, `contents lists available at`, `available online at`, `downloaded from`), copyright (`©`, `copyright `, `all rights reserved`), `Page N`/`Page N of M`/lone-digit page numbers (≤4 digits), `Vol.`/`Vol`/`Vol,` + digit (ASCII byte-scan, not regex), and journal-cite-style (4-digit year `19xx`/`20xx` + numeric `N-M` / `N–M` / `N—M` page range + ≤120 chars). 200-char ceiling on candidate strip so a body paragraph that happens to start with a URL isn't nuked. No regex dep — manual char/byte scans (one bug found and fixed during dev: byte-indexing into `&lower[i..]` panics on non-ASCII `–` in `Cell Chemical Biology 24, 1–9, …` — replaced with explicit ASCII-byte check).->-> **Crucial isolation refinement:** the first attempt filtered isolation-neighbor candidates by "must be outside the position band," which made tightly-stacked top text always-isolated (no out-of-band neighbor exists). Real body chrome would then strip even when followed immediately by body content. Fix: filter by "must not match a chrome pattern itself," so multi-line chrome groups (e.g. `Please cite this article in press as: ... \n Cell Chemical Biology (2017), http://dx.doi.org/...`) are treated as one unit (both lines pattern-match), and isolation is measured against the first non-chrome body line. False-positive guard: required the pattern hint as a hard gate — initially also accepted a font-size delta from body, but dropped that because real headings/titles routinely satisfy it, and the bench has too many "preserve the title" cases to risk it.->-> **Bench result (1403 PDFs, no OCR):**->-> | Category | Pre | Post | Δ |-> |---|---|---|---|-> | **Overall** | 31.2% ± 0.9% | **34.0% ± 0.9%** | **+2.8** |-> | headers_footers | 38.9% | **58.6%** | **+19.7** |-> | table_tests | 32.3% | 35.6% | +3.3 |-> | multi_column | 43.2% | 43.4% | +0.2 |-> | long_tiny_text | 21.9% | 20.8% | -1.1 (noise on 442-test category) |-> | baseline | 100.0% | 100.0% | 0 |->-> The +3.3 bonus on `table_tests` is the indirect win — running header/footer chrome was previously being absorbed as fake H3 headings above tables, fragmenting them or polluting cell context. Stripping it leaves cleaner table boundaries.->-> **Why the unlock is smaller than the failure-triage projection (~98% of `headers_footers`).** The triage observed "98% of failures are emitted_verbatim chrome" — meaning the chrome was present in output, not that the chrome was *pattern-matchable*. Post-fix `emitted_verbatim` still owns 97.8% of the residual 315 failures (308 cases). Two families fall through the pattern catalog:-> 1. **Letterhead / institutional disclaimers as prose** — e.g. `Este documento será custodiado por la Agencia de Tecnología y Certificación Electrónica - ISTEC Pista de Ademuz S/N. 46100 Burjassot (Valencia). Tel. 902 482 481 Correo-e: [email protected]`. Plain Spanish prose with embedded phone/email. No URL / DOI / Page-N / journal signature. Pattern catalog miss.-> 2. **Bold short-line company chrome** — e.g. `MIDWIFERY xxx (xxxx) xxx–xxx` style placeholders, journal letterheads. Some are caught by the journal-cite regex; others aren't.->-> **Next leverage on this category** (deferred — would push 58.6% → ~75-80% but with non-trivial false-positive risk on bench `present` tests):-> - Tel/email pattern (`\bTel[. :]\s*\d` + `[a-z]+@[a-z]+\.`).-> - "Bottom-of-page short line + font-size delta from body" — gated on being the very last line in the page band, since real body conclusions usually have follow-on text.-> - Loosen position band slightly (12% → 18%) to catch banner-style chrome that sits 13-15% from the top.->-> **Tests added** (5 unit tests in `repetition.rs`): `chrome_pattern_recognizes_common_signatures` (URL/`Please cite`/Page N/©/journal-cite all positive; prose / `Acme Annual Report 2023` / `Introduction` all negative), `detects_top_url_chrome_on_single_page`, `detects_bottom_journal_citation_chrome`, `preserves_title_at_top_without_chrome_pattern` (no pattern → no strip, the load-bearing false-positive guard), `chrome_with_no_isolation_gap_is_not_stripped` (URL at top followed by body within 1× body height → no strip). 158 lib + 10 integration tests pass.->-> **2026-06-03 olmOCR-bench baseline + failure triage.** Wired liteparse into `olmocr/bench` via a new `runners/run_liteparse.py` (single-page `target_pages=str(page_num)`, threaded `_pdfium_lock` since PDFium isn't thread-safe under the harness's `ThreadPoolExecutor`). Registered as `liteparse` in `convert.py`. Build/run gotcha: build the wheel from `packages/python/` (`maturin develop --release`), not the raw `crates/liteparse-python` crate — the latter ships only the `_liteparse` extension and the bench's `from liteparse import LiteParse` import fails. Bench harness Python: 3.13 (3.14's argparse rejects `%` in `benchmark.py` help strings). Convert runs at ~158 it/s on 1403 PDFs with `--parallel 1`; higher `--parallel` triggers "PDF error: invalid PDF format" from concurrent PDFium use even with the lock (harness re-instantiates parsers per task).->-> **Headline result (no OCR, baseline markdown emitter):** Overall **20.3% ± 0.8%** (average of per-JSONL).->-> | Category | Pass | Tests |-> |---|---|---|-> | baseline | 87.3% | 1394 |-> | headers_footers | 34.5% | 760 |-> | long_tiny_text | 16.7% | 442 |-> | multi_column | 10.2% | 884 |-> | old_scans | 13.3% | 526 (OCR-gated; ignore for now) |-> | arxiv_math | 0.0% | 2927 (LaTeX not emitted; ignore) |-> | old_scans_math | 0.0% | 458 (OCR + LaTeX; ignore) |-> | table_tests | 0.1% | 1022 |->-> **Failure triage script:** `dataset_eval_utils/olmocr_failure_analysis.py` (replays each test's predicate against the existing `liteparse/<subdir>/<basename>_pg<N>_repeat1.md` outputs, buckets failures with snippets). Skipping math + OCR-only categories, four dominant failure modes account for nearly every miss:->-> 1. **Column interleaving** — touches multi_column (~85% of failures), long_tiny_text (33% — newspaper-style ≥3-col layouts row-merged), and leaks into headers_footers `first_n` window misses. Reproducers: `multi_column/09f90a8fad…_page_1_pg1.md` (2-col paper; "Surprising quality is achieved when several lightweight knowledge-based NLP techniques complement…" — the second half is spliced from col 2 mid-sentence), `long_tiny_text/20_pg42_pg1.md` (Guardian books page, 4 cols interleaved row-wise). Estimated unlock: ~550 tests. Same root cause as the long-tracked `paper.pdf` page-1 banner case and as the opendataloader-bench column-interleaving residual on docs 029/038/039/077/189. **Picking this up first.**-> 2. **No `\|` markdown table emitted on borderless layouts** — 80% of table_tests failures have the cell text present as space-aligned columns but no pipes/HTML. Estimated unlock: ~800 tests. Highest absolute count but lower architectural value than (1).-> 3. **75 PDFs produce 0-byte `.md`** — concentrated in long_tiny_text (40.7% of its tests, 180 cases). All `Producer: pypdf`, not encrypted, normal MediaBox. Reproducer: `bench_data/pdfs/long_tiny_text/14a_pg1.pdf`. Likely a content-stream / subset-font edge case in our pdfium-rs path; needs a focused dump. Estimated unlock: ~180 tests.-> 4. **No header/footer suppression at all** — 98% of headers_footers failures emit the header verbatim at char 0. A repeating-y-band detector (or top/bottom 5–10% strip on the single-page subset) would push headers_footers from 34.5% → ~97% in <100 LOC. Nearly free; deferred only because (1) is more valuable.->-> Baseline category (87.3%) and order tests inside multi_column where columns happen to be single-column-shaped are the only places we're already strong. Math (0%) and old_scans (OCR-gated) deliberately ignored.->-> **2026-06-03 runner bug — was returning projection grid, not markdown (+10.9 overall).** The runner returned `result.pages[0].text`, which is the per-page projection grid (ASCII column layout), not the markdown emitter's output. `result.text` carries the rendered markdown. One-line fix in `runners/run_liteparse.py`: `return result.text`. Re-running convert+benchmark on the same wheel produced the *actual* baseline — and obsoleted the failure analysis above, which had been reading interleaved projection output and concluding "no columns / no tables / 0-byte mds" when the real markdown output had columns serialized, tables emitted with `|` pipes, and content present.->-> | Category | Pre (projection) | Post (markdown) | Δ |-> |---|---|---|---|-> | Overall | 20.3% ± 0.8% | **31.2% ± 0.9%** | **+10.9** |-> | baseline | 87.3% | **100.0%** | +12.7 |-> | headers_footers | 34.5% | 38.9% | +4.4 |-> | long_tiny_text | 16.7% | 21.9% | +5.2 |-> | multi_column | 10.2% | **43.2%** | +33.0 |-> | table_tests | 0.1% | **32.3%** | +32.2 |->-> By test type: `absent` 39.6→43.6, `order` 8.5→**36.0**, `present` 10.3→13.5, `table` 0.0→**32.2**. Baseline at 100% (1394/1394) confirms basic single-column reading works end-to-end. Tables jumping from 0 → 32% confirms the borderless + ruled-grid detectors do fire on this corpus; the remaining 68% is cell-match precision / multi-line headers / sparse forms — the same residuals we've been tracking against opendataloader-bench. Multi-column 43% means XY-cut serializes most 2-column layouts correctly; the remaining 57% is where banner-title / figure-straddle / 3+ column / extreme density cases still trip the recursion.->-> **Real priorities going forward** (post-correction):-> 1. **headers_footers absent tests** — still only 39%. The repeating-y-band detector (`compute_header_footer_set`) is firing but isn't catching most of these. On single-page bench inputs there's no cross-page repetition to detect, so the existing approach is structurally inadequate here. Needs a single-page heuristic: top/bottom 5–10% band + font-size delta + isolation gap, or a tagged-PDF artifact-role check when struct tree is present.-> 2. **multi_column 43% → push toward 70%+** — remaining failures cluster on layouts where banner-title detection or figure-rect-as-obstacle don't fully isolate the columns (same family as `paper.pdf` page 1).-> 3. **table_tests 32% → push toward 60%+** — current detector lands the *shape*; cell matching fails on multi-line headers, merged cells, and sparse forms. Same Pattern B/D residuals as opendataloader-bench.-> 4. **long_tiny_text 22%** — dense ≥3-column newspaper/dictionary pages. XY-cut conservative thresholds bail on these because density valleys are noisy. Probably needs a separate "tiny-font dense layout" mode or column-histogram fallback tuning.-> 5. **0-byte MDs are not a liteparse bug — they're image-only scanned PDFs (OCR-gated).** Verified 2026-06-03 — original hypothesis ("content-stream / subset-font edge case in pdfium-rs") was wrong. All 23 zero-byte outputs in `long_tiny_text` (10a/b/c, 14a-e, 15a-e, 16a-c, 17_pg{2,4,17,24,33,34,86}) are single embedded JPEGs with no text objects. `pdftotext` returns 1 byte on every one; `pdfimages -list` shows 1 raster image per page. Same shape across categories: ~13/18 headers_footers, 11/11 multi_column, 12/14 tables, and 98/98 old_scans empties. Deferred until paddleOCR (or another higher-quality engine) is wired up — Tesseract output would tank the `present` tests and isn't worth the integration time now. **The real long_tiny_text residual** beyond these is dense ≥3-column newspaper/dictionary layouts (`11_pg*`, `20_pg42`, etc.) where extraction works but XY-cut bails — same family as the multi_column residual, not a new failure mode. Genuine non-image extraction misses (~5 in headers_footers, 2 in tables) have small but real `pdftotext` output; their content is chrome-shaped (journal banners, "Notes", page numbers) — likely chrome-detector over-strip on pages whose entire content is chrome. Worth a follow-up to confirm.--> **2026-06-02 sparse-new-row path in borderless detector.** Targeted the "first data row gets eaten by the header" failure mode (doc 180, 045, 088/089/090, 127, 083, 084 …). Old logic in `try_detect_table` (`markdown_layout/tables.rs`) only had one branch for partial-cell lines — a wrap-merge that fired any time `centroid_dy ≤ 1.5 × line_height AND all_align_track`. That ate any data row whose cell count was less than the header's into the header itself, fragmenting tables with sparse columns.-> - **New branch** runs *before* the wrap path: when (a) all cells map to distinct column tracks, (b) `cells.len() >= 2`, and (c) `bottom_gap >= 0.5 × line_height` (clear inter-row spacing, not a stacked continuation), promote the partial line to a new row with empty `TableCell`s at the missing track positions.-> - **`cells.len() >= 2` is the load-bearing guard.** Without it, single-cell continuations like `"2022"` 5pt below `"1.0 April 30,"` get promoted to standalone rows (gap signal is noisy on a single cell). With it, 1-cell partial lines always fall through to the existing wrap-merge path (which is what we want — those are real continuations). The first attempt without this guard regressed TEDS by 0.04 across docs 130/051/082/090/147; adding it converted the net into +0.049.-> - **`bottom_gap >= 0.5 × line_height` discriminates "clear new row" from "wrap"** — a wrap typically overlaps the prior baseline (negative gap) or sits within ~0.3 × line_height of it. The 0.5 floor leaves a dead-zone (0.3, 0.5) that falls through to the existing wrap path, which is correct for the in-between cases.-> - **Wrap path unchanged.** First attempt tightened the wrap path's gap condition simultaneously, which regressed previously-clean tables (doc 130 0.84 → 0.10) because the existing wrap-merge was load-bearing for tightly-stacked multi-line cells. Reverted and reordered — sparse-new-row runs first, wrap path runs second with original conditions.->-> **Bench result on opendataloader-bench (200 docs):**->-> | Metric | Pre-sparse | Post-sparse | Δ |-> |---|---|---|---|-> | NID | 0.8848 | **0.8851** | +0.0003 |-> | TEDS | 0.4905 | **0.5394** | **+0.0489** |-> | MHS | 0.5911 | 0.5927 | +0.0016 |-> | Overall | 0.7829 | **0.7874** | +0.0045 |->-> **Per-doc wins (no regressions on the watch list):**-> - doc 180: 0.000 → 1.000-> - doc 127: 0.692 → 0.831-> - doc 083: 0.555 → 0.757-> - doc 088: 0.278 → 0.437-> - doc 090: 0.357 → 0.438-> - doc 089: 0.115 → 0.204-> - doc 045: 0.784 → 0.892-> - doc 084: 0.694 → 0.775-> - doc 128: 0.521 → 0.622->-> **Preserved:** docs 130 / 051 / 052 / 082 stay at their post-Pattern-A values (0.84 / 0.87 / 0.86 / 0.81).->-> **Cumulative session totals on TEDS:** 0.4144 → 0.5394 (+0.125 absolute, +30% relative). NID 0.8820 → 0.8851. Overall 0.7737 → 0.7874.->-> **Hard-won lesson (added to `project_table_detector` memory):** When relaxing a partial-line-row decision, the cell *count* matters more than the cell *position*. Single-cell partial lines are almost always wraps or noise; multi-cell partial lines with distinct-track mapping and clear gap are almost always real sparse rows. Don't tighten an existing working path — *add* a sibling branch that runs first.--> **2026-06-02 hard-reject XY-cuts through obstacles (defensive).** Added `cut_line_passes_through_obstacle` check in `xy_find_best_cut` (`projection.rs`) to complement the existing density-stamping. The hypothesis (and the user's ask): density-stamping makes the recursion *prefer* cuts around obstacles but doesn't *forbid* cuts through them, so a sparsely-filled table's inter-cell gutter could still score best despite obstacle stamping. The check refuses any candidate cut whose midpoint position lands inside an obstacle rect (with 1pt edge tolerance so a cut at an obstacle's edge stays legal — obstacle edges are themselves natural cut lines).->-> **Bench result on opendataloader-bench (200 docs):** identical to the pre-hard-reject run. NID 0.8848 / TEDS 0.4905 / MHS 0.5911 / Overall 0.7829. Zero change.->-> **Why it didn't move the needle on docs 120 / 180 / 150** — these turned out *not* to be XY-cut failures. Dumped projection for both: the table region is already a single leaf with clean row-major lines (doc 120 `region=[1,1]`: `"Normal hemoglobin dissolves in Disk-shaped red blood cells can"`, etc. — one baseline holds two columns of text). The Pattern A obstacle pass was already preventing the column-slice. The real blocker is the **borderless detector**:-> - Doc 120: ≥3-cell-per-row requirement fails because column 1 (`Normal hemoglobin…`/`2 copies…`/etc.) appears on every other baseline while the wider right column wraps to 2-3 baselines per cell. Most rows show 2 cells.-> - Doc 180: same shape — `"1.0 April 30, Original"` then `"2022"` on the next baseline. Each logical row spans 2 baselines, and the second baseline has 1-2 cells. The detector closes the run after one row.->-> Kept the hard-reject change anyway as defensive (no regression, principled "never slice through a detected table" guarantee, may matter on docs we haven't seen). The follow-up that *would* move 120/180/150 is **borderless detector multi-baseline row merging** — when a "row" has fewer cells than the established track count but every present cell aligns to a track AND the y-gap to the previous row is ≤ ~1.5× line height, *append* it to the previous row instead of treating it as a row of its own. The 2026-06-01 "wrap merge" change already does something like this for borderless tables but only when the run is *already* a table; it'd need to relax the establishment condition so a run can start from a row that has fewer cells than the final track count.--> **2026-06-02 fill-in-form spine bypass for sparse ruled grids.** Targeted docs 119/120/180/150 where `TABLE_MAX_EMPTY_CELL_FRACTION=0.30` rejects legitimate fill-in-blank forms (comparison charts with a row-label column + header row but mostly empty body cells). The plan-doc note already flagged this; raising the empty-cell ceiling unconditionally backfires (prior attempt regressed TEDS by ~0.09). The fix needed a signature that distinguishes real fill-in forms from decorative wrappers / fragmented borderless tables that the ruled detector would otherwise grab on the relaxation.-> - **Signature** (`build_ruled_table` in `markdown_layout/tables.rs`): when `empty_frac > 0.30`, accept iff (a) **col 0** is well-populated (`col0_fill ≥ TABLE_SPINE_FILL_FRACTION=0.7`) AND (b) col-0 cells are *short* (`col0_max_chars ≤ TABLE_SPINE_MAX_CELL_CHARS=60`). Real labels are 1-5 words; multi-sentence prose stuffed into a column is the false-positive signature. Hard ceiling at `TABLE_MAX_EMPTY_CELL_FRACTION_WITH_SPINE=0.75` even on bypass — past that, even a strong spine isn't enough.-> - **Why col 0 only, not row 0?** Tried both. Row-0 bypass passes doc 051's multi-line header (every cell populated by happenstance of header text spanning columns), then the body fragments badly because the cells are filled per x-band — net regression on doc 051 from 0.874 → 0.303. Col-0 label-column signature is far rarer in body prose and far more universal in fill-in forms (every form I checked has col 0 = labels).-> - **Why area-cap and max-fill-anywhere fell over (intermediate attempts):**-> - `max_col_fill ≥ 0.7` (any column): fires on doc 130 where the rightmost column has a long prose chunk = 80% col-fill. Dropped 130: 0.840 → 0.103.-> - + area cap (≤ 0.5 page): trims 130 but the borderless detector still produced a fragmented output that beat the cleaner borderless result via `merge_table_runs`. Net: 130 stayed bad.-> - col-0 + short-cells together separate the two cases cleanly without needing area or per-doc tuning.->-> **Bench result on opendataloader-bench (200 docs, image_mode=off):**->-> | Metric | Pre-spine | Post-spine | Δ vs pre-spine | Δ vs session start (pre Pattern A) |-> |---|---|---|---|---|-> | NID | 0.8845 | **0.8848** | +0.0003 | +0.0028 |-> | TEDS | 0.4737 | **0.4905** | +0.0168 | **+0.0761** |-> | MHS | 0.5911 | 0.5911 | 0 | -0.0011 |-> | Overall | 0.7810 | **0.7829** | +0.0019 | +0.0092 |->-> **Per-doc:** doc 119 0.000 → 0.708 (+0.708). All previously-good docs preserved (130: 0.84, 045: 0.78, 051: 0.87, 052: 0.86, 082: 0.81, 084: 0.69, 047: 0.75, 083: 0.56, 127: 0.69). Net session gain of +0.076 absolute TEDS (~18% relative).->-> **Still TEDS=0 — root cause is upstream of the table detector now:**-> - **docs 120 / 180 / 150** — same comparison-chart layout as 119. `detect_table_rects` *does* return a candidate rect (cov=0.84×0.49 on doc 120), the obstacle *is* passed to XY-cut, but XY-cut **still slices the table column-wise** and the output shows column-major reading order through the table region. The obstacle is being overridden — likely the inter-column gaps in the table are so dominant in the density projection that even with the obstacle's perpendicular-extent weight stamped in, the gutter still scores best. Worth instrumenting `xy_find_best_cut` on doc 120 specifically and seeing what the density profile looks like with and without the obstacle. Possible fixes: (a) bump figure-obstacle stamp weight when the obstacle is a table-rect specifically, (b) hard-reject any V-cut whose position falls inside an obstacle rect (instead of just biasing density), (c) detect "table-like column stacks" and reject V-cuts that would slice through them.-> - **doc 121** — different problem entirely. PDFium reports items in a garbled order (item-extraction quirk, not layout). Not a table-detection issue.-> - **docs 132 / 146 / 166 / 197** — no significant graphics, would need borderless-detector improvements (Pattern B / Pattern D from the earlier audit).-> - **docs 088 / 089 / 090 / 128 / 182** — multi-line wrapped header collapse (Pattern B). Header absorption pass is the next big lever here (~0.04 TEDS potential).->-> **Hard-won lesson (saved to memory):** Just dropping `TABLE_MAX_EMPTY_CELL_FRACTION` regresses net — the relaxation creates false-positive ruled tables that beat legitimate borderless tables via `merge_table_runs`, fragmenting them. Any bypass needs a *positive* fill-in-form signature, not just a relaxed threshold.--> **2026-06-02 ruled-table-rects as XY-cut obstacles.** Targeted the dominant TEDS=0 failure mode on the bench: XY-cut treating inter-cell gaps inside a ruled table as page-level column gutters, slicing the table into per-column leaves and producing column-major reading order (docs 083, 127, 130, 047, 082, 147 …). Same machinery `figure_cluster::detect_figure_rects` already uses — figures stamped as density obstacles in `xy_find_best_cut` — extended to ruled-grid bboxes.-> - **New `markdown_layout::detect_table_rects(graphics, page_w, page_h) -> Vec<Rect>`** (`markdown_layout/tables.rs`) — graphics-only sibling of `detect_ruled_tables`. Reuses `extract_h_v_segments` → `cluster_h_segments` → `cluster_v_segments` → `find_grid_components`, but skips the lines-assignment + empty-cell-fraction + header-bold-detection steps and returns the *bbox* of each H/V grid component. Empty-cell filter deliberately omitted: we want the bbox even of sparse forms (doc 119) for obstacle purposes. Only filters: ≥5pt on both axes, and `w/page_w ≥ TABLE_MAX_PAGE_COVERAGE` ∧ `h/page_h ≥ TABLE_MAX_PAGE_COVERAGE` (the page-border guard, same constant as the post-projection detector). Re-exported as `pub(crate) use tables::detect_table_rects;` from `markdown_layout/mod.rs`.-> - **`projection.rs::to_parsed_pages`** computes `table_rects` alongside the existing `figures`, concatenates into a single `obstacles: Vec<Rect>`, and passes that to `build_projected_lines` (parameter still named `figures` but the call site is the only public interface — the obstacle pipeline doesn't distinguish, all rects stamp density into perpendicular buckets identically).-> - **No new tests added** — `detect_table_rects` is mechanically a subset of `detect_ruled_tables` which already has 5 tests covering the segment/cluster/component pipeline. 153 lib tests still pass.->-> **Bench result on opendataloader-bench (200 docs, image_mode=off):**->-> | Metric | Pre-fix | Post-fix | Δ |-> |---|---|---|---|-> | NID | 0.8820 | **0.8845** | +0.0025 |-> | TEDS | 0.4144 | **0.4737** | **+0.0593** |-> | MHS | 0.5922 | 0.5911 | -0.0011 (noise) |-> | Overall | 0.7737 | **0.7810** | +0.0073 |->-> **Per-doc TEDS wins** (from the prior-pass watch list):-> - doc 130: 0.000 → 0.840 (+0.840) — the carry-forward "column-major reading order" case-> - doc 047: 0.086 → 0.750 (+0.664)-> - doc 127: 0.194 → 0.692 (+0.498)-> - doc 083: 0.253 → 0.555 (+0.302)-> - doc 147: 0.000 → 0.124 (+0.124, partial — multi-line header still merges)-> - doc 082: 0.749 → 0.811 (+0.062)->-> No watch-list regressions.->-> **Still TEDS=0 after this pass** — these don't have ruled grids the segment extractor sees, OR have grids but other failure modes dominate:-> - docs 119/120/121/180 — pages where every cell is a stroked Rect rather than shared H/V lines; the dump shows 90+ stroke_rects each, so `extract_h_v_segments` *should* be turning each rect into 4 edges and clustering them. Worth instrumenting `detect_table_rects` on doc 119 to see whether the grid component is being built but the bbox not surfacing, or whether the rects are getting filtered by the page-coverage guard.-> - docs 132/146/150/166/197 — small graphics counts; mostly Pattern B (multi-line wrapped header collapses) territory, not Pattern A.-> - docs 088/089/090/128/182 — the cluster of multi-line wrapped headers; needs header absorption (walk back from detected body table to grab unaligned header baselines above).->-> **Next leverage candidates** (rough priority):-> - **Pattern B: multi-line header absorption** — would lift 088/089/090/128/182/166 from ~0.1–0.4 to ~0.7+. The body table now lands cleanly thanks to this pass; the merge logic needs to walk N lines back from the detected table top and absorb any line whose cells fit the established column tracks (or fragments thereof).-> - **Why don't 119/120/121/180 benefit?** Either (a) per-cell stroked rects aren't producing H/V clusters dense enough to form a component, or (b) the rects exist but they form one ≥95% page-coverage component that gets filtered. Worth a 10-minute debug on doc 119.-> - **Pattern D: 2-column tables + sparse grids** — once 119 is unblocked by the above, lowering `TABLE_MIN_COLUMNS` to 2 (with a stricter row-count gate) and raising the empty-cell threshold for ruled-grid detection only would pick up docs 132, 146, 150.--> **2026-06-01 module split — `markdown_layout.rs` → `markdown_layout/`.** The single 3819-line file got broken up into a directory of focused submodules; pure move-only refactor, no logic changes. New layout: `mod.rs` (24 lines, re-exports the 5 external API symbols + `Block`), `blocks.rs` (347 — `Block`, `render_blocks`, `wrap_emphasis`, `paragraph_from_accum`), `classify.rs` (783 — `classify_page_with_filters`, `classify_page`, `is_rotated_line`, orchestration), `headings.rs` (526 — body-size histogram, heading map, struct/outline level resolution, bold-heading heuristics, caption rejection), `tables.rs` (1118 — borderless + ruled-grid table detection, `escape_table_cell`, `merge_table_runs`), `inline.rs` (319 — `SpanStyle`, emphasis rendering, list-item text), `paragraphs.rs` (214 — `ParaAccum`, `continues_paragraph`, `collapse_whitespace`), `repetition.rs` (170 — running header/footer detection), `hr.rs` (132 — `detect_horizontal_rules`), `lists.rs` (96 — `parse_list_marker`), `tests_common.rs` (202, `#[cfg(test)]` only — shared `line`/`page`/`styled_line` fixtures so per-module test modules don't duplicate them). Visibility: only `build_heading_map`, `classify_page_with_filters`, `compute_body_size`, `compute_header_footer_set`, `render_blocks`, and `Block` are `pub` (re-exported from `mod.rs`); everything else is `pub(super)` within the submodule tree. External consumer (`output/markdown.rs`) and `lib.rs` unchanged. 147 lib tests still pass; `cargo check -p liteparse` clean. Companion plan to also split `projection.rs` (4050 lines) deferred — too load-bearing while heading/layout work is still iterating.->-> **2026-05-30 rebuild note.** Branch had been cleaned during a branch switch — `markdown_layout.rs` / `output/markdown.rs` were sitting untracked, the supporting types (`ProjectedLine`, `Rect`, `Anchor::Floating`, `ParsedPage.projected_lines`) didn't exist, `OutputFormat::Markdown` wasn't on the enum, and nothing was wired into `lib.rs` / `output/mod.rs`. Restored from `stash@{0}` (which had the plumbing diff) + reimplemented step 2 (`build_projected_lines` in `projection.rs` derives per-line metadata from the projected items). Tests pass; smoke on `paper.pdf` emits headings + bold + page separators end-to-end via `cargo run -p liteparse -- parse <pdf> --format markdown`. Steps 3–6 in the table below are status-accurate again as of this note.--> **2026-05-31 XY-cut layout decomposition.** Replaced the flat x-density histogram column detector with a recursive modified XY-cut. Motivation: the old detector failed on layouts where the gutter is interrupted by a wide element (figures straddling both columns, full-width headings/banners) — every page-1-style case in the eval set, and the open carry-forward "column interleaving on page 1" from 2026-05-30.->-> - **Types** (`types.rs`): added `Region` / `RegionKind { Leaf, Split }` / `CutAxis`. `ProjectedLine.column_id: usize` → `region_path: Vec<u16>` (index trail from root; equality = same leaf, prefix = containment). `ParsedPage.regions: Region` added behind `#[serde(skip)]` so JSON/text outputs stay byte-identical.-> - **Algorithm** (`projection.rs::xy_cut`): char-weighted 1D projection onto each axis at 2pt buckets. A "valley" is a contiguous run of buckets where density < `T_DENS × median(non-zero buckets)`. Conservative defaults — `T_DENS=0.10`, min vertical-cut valley `8pt`, min horizontal-cut valley `1.4 × median line height`. Each recursion picks the best valley across both axes, tie-breaking 1.1× in favor of horizontal so pages slice into bands before columns (natural reading order). Hard recursion depth cap of 6; H-cuts require ≥2 distinct y-rows per side so we don't slice prose into slivers. Pre-order leaf walk produces reading order for free, including nested layouts.-> - **`build_projected_lines`** now takes `page_height` (additive) and returns `(Vec<ProjectedLine>, Region)`. Lines inside each leaf are y-banded and built with the leaf's `region_path`. `detect_columns` deleted entirely.-> - **`markdown_layout.rs`** paragraph-continuation and table-row-adjacency checks switched from `column_id` equality to `region_path` equality. No other consumers needed changes — `column_id` was internal.-> - **Tests**: 119 lib + 10 integration tests still pass. Added 3 unit tests on synthetic items: 2-column layout → single V-split, tight single-column prose → leaf root, pre-order walk of a 2-column layout → left-column lines all precede right-column lines (`region_path` prefix `[0]` vs `[1]`).-> - **Smoke on `paper.pdf`**:-> - **Pages 2+** (the multi-column case that interleaved before): now read coherently column-by-column. Resolves the 2026-05-30 column-interleaving carry-forward.-> - **Page 1** (title banner + abstract + figure straddle + intro): reading order is still wrong but shaped differently — title `# A-MEM…` now appears *after* the abstract block rather than at the top. Root cause: with the figure straddling both columns and the title+authors banner above them, no single H-cut or V-cut at the page root produces the right partition. Needs either (a) higher-level "wide-banner detector" that seeds an H-cut around full-page-width centered text, or (b) figure-rect-aware seeding so the recursion treats the figure as an obstacle and H-cuts above/below it first. Deferred — tracked as next layout-detection pass; not blocking v1.-> - **Threshold tuning** is the most likely future knob. Bumping `T_DENS` lets noisier gutters cut (e.g. page numbers in the gap); dropping `XY_V_PREFERENCE_MARGIN` toward 1.0 makes V-cuts more aggressive on pages where the page header has weak inter-line gaps but a strong central gutter.->-> **2026-05-31 heading-quality pass.** Smoke on `paper.pdf` exposed two bad failure modes: (a) the size-based heading map was getting eaten by figure-label numbers (size 36/24pt), the rotated arXiv sidebar (size 20pt), and graph-axis values, pushing the real title to H6 and authors out of the map entirely; (b) academic section heads ("Abstract", "1 Introduction", "5 Conclusion") are body-sized + bold so the size-only detector missed them. Fixed in `markdown_layout.rs`:-> - **Rotation filter** (`is_rotated_line`): per-line check using `TextItem.rotation`. Excludes rotated text (arXiv banners, vertical legends, watermarks) from both `compute_body_size` and `build_heading_map`, and also from the per-line classifier pass so they don't surface as paragraphs. This is what finally drops the 20pt arXiv sidebar from H-slot competition.-> - **`build_heading_map` quality filters**, applied unconditionally (not gated on candidate count): sizes must clear `MIN_HEADING_TOTAL_CHARS=25` total non-whitespace chars *and* `avg_chars_per_line ≤ 200` *and* `alpha_ratio ≥ 0.5`. The total-char floor (instead of a line-count floor) is the key change — it lets a one-off paper title at 14.35pt survive (31 chars on a single line) while still excluding chart-legend tokens like "A-mem"×2 + "Base"×2 = 18–20 chars across 4 occurrences. Alpha-ratio kills scoreboard/equation/graph-axis sizes (36pt = 0% alpha, 18pt = 0% alpha, 24pt = 22% alpha on `paper.pdf`).-> - **`looks_like_bold_heading`** rule: a uniformly-bold body-size line ≤ 80 chars with paragraph-break gap above + below and alpha-ratio ≥ 0.5 becomes a heading at `heading_map.len() + 1`. Runs *after* list-marker detection so bold bullets stay as list items. Drives "Abstract" / "2 Related Work" / "5 Conclusion" detection on `paper.pdf`.-> - **Final `paper.pdf` headings**: `# A-MEM: Agentic Memory for LLM Agents` (title at H1) → `## Abstract` / `## 2 Related Work` / `## 3 Methodolodgy` / `## 4 Experiment` / `## 5 Conclusion` (section heads at H2) → `### 2.1 Memory for LLM Agents` (subsection at H3). Reads like the document.-> - **Known residuals on `paper.pdf`**:-> - Results-table header rows ("Category Average", "F1 BLEU-1 F1 BLEU-1 ...", "Qwen2.5 GPT Llama 3.2 ...") still classify as H3 because they're bold body-size short lines and the table detector doesn't fire on them. Needs better table detection (step 7 follow-up) — not a heading-detection problem.-> - "Abstract Write" — the figure label "Write" (at fs=1.0, height ~9pt) lands on the same projected line as the "Abstract" heading because column detection treats page 1's figure-bearing right half as part of the abstract column. Column-detection problem, not heading.-> - Authors / affiliation block at H2 — they share font size with section headings (11.96pt). Could plausibly be filtered as "page-1 top centered block before any body text" once title detection sees them, but not in this pass.-> - The dominant remaining quality issue on `paper.pdf` is column interleaving on page 1 (figure straddles both columns → projection mixes left/right column lines). This is layout-detection territory, not heading detection. Tracked as a separate carry-forward.--> **2026-06-01 step 9 partial — paths + HR + ruled tables.** No actual PDFium fork edits needed; all 10 missing path getters were already declared in `bindings.rs`, just absent from the runtime-loaded `PdfiumBindings` struct in `crates/pdfium-sys/src/dynamic.rs`. Wired them through; added `PathObject`/`PathSegment` in `crates/pdfium/src/page.rs::path_objects`; distilled to `GraphicPrimitive::{Stroke, Rect}` in `liteparse/src/types.rs` + `extract.rs::extract_page_graphics`; threaded `graphics` onto `Page` → `ParsedPage`. Added `Block::HorizontalRule` rendered as `---`, `detect_horizontal_rules` with underline-band filter, and `detect_ruled_tables` (union-find on crossing H/V grid lines, ≥2×2 grid, ≤30% empty cells, ≤95% page coverage). `merge_table_runs` runs ruled-grid before the existing borderless detector and drops overlapping borderless runs. 136 lib + 10 integration tests pass (11 new for the graphics pipeline). Figure clustering (9c-3) deferred — see carry-forwards.->-> **Worth smoke-testing on real docs.** Ruled-table candidates: `Anonymized rate card.pdf` (the same doc step 8 was tested on), any tagged invoice / table-heavy spreadsheet export. HR candidates: anything with `<hr>`-style section dividers (letters, contracts, multi-section reports). `paper.pdf` has known residuals from step 7 that the borderless detector caught — confirm the new ruled-grid path doesn't regress them; if it does, `merge_table_runs` keeps the borderless run when ranges don't overlap, but a partial-overlap could still drop one. Command: `cargo run -p liteparse -- parse <pdf> --format markdown`.--> **2026-06-01 paper.pdf quality pass — table fragmentation, cross-region continuation, HR cleanup, figure misclassification.**->-> - **Borderless-table merged-cell recovery** (`markdown_layout.rs::recover_merged_cell`). PDFium occasionally merges adjacent words into one text run when inter-word kerning is tight (e.g. "MEMORYBANK 5.00 4.77" emitted as one span on `paper.pdf` page 6's results table). That dropped the row's cell count below the established column count, so the table-row matcher closed the run — fragmenting the table into 2-row chunks around every shifted-indent row. Fix: when a candidate row has fewer cells than the established `tracks`, iteratively find the cell that straddles the most tracks (≥2), split its text on whitespace at the boundary positions linearly interpolated to track x-coords, replace with the split pieces. Handles off-by-N (loop), returns `None` cleanly when no whitespace boundary is available (e.g. hyphenated tokens). Added `end_x` to `TableCell`. 3 new unit tests (`recover_merged_cell_splits_off_by_one`, `recover_merged_cell_splits_off_by_two`, `recover_merged_cell_bails_without_enough_whitespace`). **Result on `paper.pdf` page 6:** the 15-row result-table chunks that were previously fragmented into 7+ 2-row pieces are now two coherent 15-row pipe tables (only split by a mid-table "Qwen2.5 GPT Llama 3.2 …" group-label row that's still classified as H3 — header-row absorption remains future work).-> - **Cross-region paragraph continuation** (`markdown_layout.rs::continues_paragraph`). The hard `prev.region_path == cur.region_path` gate broke the common 2-column-paper case where a sentence wraps from the bottom of column 1 into the top of column 2 (e.g. the `- *We present A-MEM …* tem … This system equips LLM agents with` bullet ending mid-thought, and resuming with `long-term interaction capabilities …` in column 2). Relaxed: when `region_path` differs, bridge regions iff the previous line ends without sentence-terminating punctuation (`. ! ? : ; ” " ) ] 。 』 」`) AND the next line starts with a lowercase character. Both conditions are strict signals — together they catch column-wraps cleanly while rejecting unrelated paragraphs that happen to land in adjacent leaves. Smoke on `paper.pdf` collapses ~50 erroneous mid-paragraph breaks (522→479 lines in the rendered markdown).-> - **HR suppression inside / near tables** (`markdown_layout.rs::classify_page_with_filters`). Table internal horizontal rules (row dividers) and the rules drawn above a table's column headers were being emitted as document-level `---`. Build a `table_y_extents` list once after `merge_table_runs`; filter `detect_horizontal_rules` output to drop any HR whose y falls inside `[top - 4 × row_height, bot + 2pt]`. The upward expansion catches header-band rules above the table data that we don't (yet) absorb into the table block. Eliminates the `---` chains that previously surrounded every table on `paper.pdf`.-> - **Figure misclassification of tables** (`figure_cluster.rs::text_coverage`). The old per-glyph area coverage was too tight — left huge gaps between numeric cells unaccounted for, so the page-6 results table (~32% glyph coverage) sneaked under the 55% threshold and got classified as a figure. The figure rect then got obstacle-seeded into XY-cut, which actually helped the rows stay together but the misclassification was waiting to break downstream consumers. Rewrote `text_coverage` to y-band overlapping text items into lines (median-height-based band tolerance) and sum the union bbox of each band — gives a far better proxy for "this region is dominated by text". Result on `paper.pdf`: page 6's table no longer matches as a figure (`figures = []`). No vector figures on `paper.pdf` to regression-test against (page 1's figure is a raster image with 0 vector primitives, so figure_cluster never saw it either way).->-> **Test count:** 147 lib + 10 integration tests pass.->-> **Known residuals on `paper.pdf`:**-> - **Page 1 banner-title layout** — unchanged, still tracked separately.-> - **Mid-table group-label rows** misclassified as H3 (e.g. `### Qwen2.5 GPT Llama 3.2 …`). Needs either header-row absorption from above/below a detected table, OR a "line is between two table runs sharing the same tracks → group label" detector.-> - **Single-column section-header rows** above tables (`### Category Average`, `### F1 BLEU-1 …`) still emit as H3. Same header-absorption story.->-> **2026-06-01 step 9c-3 — figure clustering + XY-cut obstacle seeding.** New `crates/liteparse/src/figure_cluster.rs::detect_figure_rects(graphics, text_items, page_w, page_h) -> Vec<Rect>`. Pre-filters HR-like strokes (long thin horizontal, length ≥ 30% page width, ≤ 2pt thick) and full-page rects, then union-finds the remaining primitives by bbox proximity (within 10pt on both axes). Each cluster's union bbox is kept only when: min extent ≥ 30pt both axes, area ≥ 1500pt², not ≥95% page in both dims, aspect ratio ≤ 15, has ≥2 primitives OR a single primitive ≥ 5000pt², and text coverage inside the bbox ≤ 55% (rejects text-highlight backgrounds and banner fills). **Wired through XY-cut as obstacles:** `xy_cut`, `xy_cut_rec`, `xy_find_best_cut`, and `build_projected_lines` now take `figures: &[Rect]`. In `xy_find_best_cut`, each figure intersecting the current bbox is clipped and stamped into the density buckets along the axis being scanned, with `weight = perpendicular extent`. Effect: figures behave like very wide/tall pseudo-items, so the valley search naturally prefers cuts *adjacent* to figures over slicing through them. `figures` is also persisted on `ParsedPage` (#[serde(skip)]) so downstream passes can reuse it. `to_parsed_pages` computes figures from `page.graphics` + `page.text_items` before invoking `build_projected_lines`. **Test count:** 144 lib + 10 integration tests pass (7 new in `figure_cluster::tests` covering empty input, HR rejection, dense-stroke clustering, large-solo-rect acceptance, full-page-background rejection, text-heavy rejection, and distant-strokes-don't-merge; 1 new in `projection::tests::figure_obstacle_forces_h_cut_around_straddling_figure` confirming a wide figure between two text bands forces an H-cut at the root). **Smoke on `paper.pdf`:** pages 2+ unchanged (still clean), abstract column on page 1 still reads cleanly as before. Page 1's title-still-mid-document carry-forward is *unchanged* — the issue isn't the straddling figure (which the obstacle seeding does isolate), it's the arXiv stamp + affiliation lines at the top that satisfy the banner heuristic and disrupt the H-cut position above the title. Banner-detection refinement remains the next layout knob. **Carry-forward update:** the "feeding figure rects into XY-cut as obstacles" item is now resolved.--## Status legend--- ✅ done-- 🚧 in progress-- ⬜ not started--## Build order--| # | Step | Status | Notes |-|---|---|---|---|-| 1 | Wire `OutputFormat::Markdown` end-to-end | ✅ | Stub returns per-page text wrapped in ` ```text ` fence with `<!-- page N -->` comments and `\n\n-----\n\n` separators. `md` alias accepted. |-| 2 | `ProjectedLine` API in `projection.rs` (additive) | ✅ | `Rect`, `ProjectedLine`, `Anchor::Floating` added. `ParsedPage.projected_lines` is `#[serde(skip)]` — JSON/text output byte-identical. `column_id` + `indent_x` are placeholders. |-| 3 | Font-size histogram, headings, paragraph grouping, de-hyphenation | ✅ | New `markdown_layout.rs` builds a global char-weighted size histogram, maps sizes > body to H1..H6, groups consecutive lines into paragraphs (same anchor/column/font/indent + gap ≤ 1.5× line height), and de-hyphenates `word-\nlowercase` joins. `output/markdown.rs` now renders blocks instead of stubbing a fence (fence retained as fallback for pages without `projected_lines`). |-| 4 | Lists (bullet + ordered + nesting from indent) | ✅ | `parse_list_marker` recognizes Unicode bullets (`•·◦▪▸▶●○■□`) and decimal markers (`1.` / `12)`). Nesting derived from `indent_x` bucketed by 12pt steps relative to the first item of the run. Wrapped continuation lines reuse the paragraph-continuation rule so footnote-style hanging indents merge correctly. Original marker preserved (`138.` stays `138.`) so footnote/section numbering survives. ASCII `-`/`*`/`+` deliberately excluded to avoid false matches in prose. |-| 5 | Code blocks (all-mono lines) | ✅ | `Block::CodeBlock { lines }` accumulates consecutive `all_mono` lines and emits a triple-backtick fence (falls back to `~~~` if body contains `` ``` ``). Detection runs before heading/list/paragraph passes so mono content always wins. |-| 6 | Inline styling (bold/italic/mono/strike/link) with per-line shortcut | ✅ | Per-span (mid-line) emphasis now lands. `render_line_inline` walks `ProjectedLine.spans` in x-order, groups adjacent same-style spans (bold/italic/mono via the shared `projection::is_*_item` detectors, now `pub(crate)`), escapes `*`/`_`/`\\`, and emits `**…**` / `*…*` / `***…***` / `` `…` `` runs. Three-tier rendering: (1) per-line shortcut when all spans on a line agree; (2) paragraph-uniform shortcut preserved — when **every** line of a paragraph is uniformly the same non-plain style, we still wrap the whole paragraph once with `wrap_emphasis(raw, …)` to avoid `**foo** **bar**` per-line noise; (3) per-line inline strings otherwise. `Block::Paragraph`/`ListItem` carry both `raw` (plain) and the rendered text — the classifier picks one at flush time. List items strip the marker before rendering (uniform-style path peels it from `rest`; mixed-style path peels emphasis-wrapped marker prefixes via `strip_leading_marker_from_inline`). Smoke on `paper.pdf`: bold author block + mid-line `**Abstract**` + italic `*We present A-MEM…*` runs + backtick-fenced URLs + escaped `\*` corresponding-author marker all render correctly. **Strike + links still deferred** (no PDFium span attribute for strike; links need annotation rects from the fork, gated with step 10). |-| 7 | Tables — medium (column-alignment) + fallback | ✅ | `Block::Table { header, rows }` and `Block::GridFallback { lines }` added. `detect_tables` runs as a pre-pass over a page's `ProjectedLine`s before per-line classification. `split_cells` slices each line's `spans` into cells when adjacent-span gaps exceed `1.0 × dominant_font_size`. A run qualifies as a table when ≥3 cells per row repeat across ≥2 consecutive rows with column-track alignment (≤1 column allowed to drift outside ±6pt — handles indented sub-group row labels); if row-spacing CV > 0.5 the run is downgraded to `GridFallback`. First row promotes to header only when every cell is bold (synthesizes blank header otherwise). `table_rows_adjacent` deliberately doesn't gate on `region_path` because XY-cut sometimes puts an indented row in a different leaf than the rest of the table — column-alignment + y-gap are strong enough on their own. `looks_like_bold_heading` now rejects lines that look tabular (≥3 spans by `split_cells` OR ≥3 multi-space-separated tokens by text fallback) so multi-line table headers like "Model Method Single Hop Multi Hop ..." stop getting promoted to H3. Smoke on `paper.pdf` (results tables): 8 well-formed pipe tables emitted, "Model Method..." header line now lands as a bold paragraph instead of H3. **Known residuals**: (a) `MEMORYBANK` rows in academic results tables are at x=102.5 vs neighbor rows at x=70 — ALL column tracks shift by ~30pt so the 1-column-tolerance rule can't bridge them; tables still fragment into 2-4 row chunks around those rows. Fix needs ruled-grid detection from PDFium fork (step 9) — won't be solvable from text-only signals. (b) Short header rows like `Category Average` (2 cells) and the full-width `F1 BLEU-1 F1 BLEU-1...` (1 collapsed projection span, single-spaced text) still classify as H3 — text-only heuristics can't distinguish them from real one-line section headings without table-rect context. (c) Header absorption (walking back from a detected table to grab multi-line headers above) deliberately deferred — would need careful per-column concatenation against unaligned multi-line headers; better to wait for fork-driven cell-rect data. |-| 8 | Header/footer stripping (cross-page repetition) | ✅ | `compute_header_footer_set` collects normalized line text (lowercased, digit-runs collapsed to `#` so `Page 1 of 6`/`Page 2 of 6` match) appearing in the top 12% / bottom 12% page-height band on ≥ 50% of pages (floor of 2). Header and footer bands tracked separately so a heading that doubles as both running header on some pages and body title on others isn't stripped from the body. `format_markdown` builds the set once and passes it to `classify_page_with_filters`, which removes matching lines before table detection and per-line classification. Smoke-tested on `Anonymized rate card.pdf` — repeating `Page N of 6` chrome is gone from output. |-| 9 | Fork: paths exposed → strong (ruled grid) tables, HR detection, figure clustering | ✅ | **Paths exposed + HR + ruled-grid tables landed 2026-06-01; figure clustering deferred.** No actual PDFium fork modifications were required — every needed FFI symbol was already in `bindings.rs`; only the runtime-loaded `PdfiumBindings` struct in `crates/pdfium-sys/src/dynamic.rs` was missing the 10 path-related getters (`FPDFPageObj_GetMatrix`, `FPDFPath_CountSegments`, `FPDFPathSegment_*`, color/width getters, `FPDFPath_GetDrawMode`). Added all. **9a — pdfium wrapper:** new `PathObject` / `PathSegment` / `SegmentKind` in `crates/pdfium/src/page.rs`; `Page::path_objects(view_box)` iterates `FPDF_PAGEOBJ_PATH` objects, applies the object matrix → page → viewport so all segment points land in viewport coords for direct comparison against `ProjectedLine.bbox`. **9b — liteparse `GraphicPrimitive`:** added `GraphicPrimitive::{Stroke { x1,y1,x2,y2,color,width }, Rect { bbox, fill, stroke }}` to `types.rs` + `ParsedPage.graphics` (#[serde(skip)]). `extract.rs::extract_page_graphics` decomposes paths: filled paths → one `Rect` for the bbox (covers cell backgrounds / banner fills regardless of internal complexity); stroked paths → one `Stroke` per `MoveTo→LineTo` / `LineTo→LineTo` pair (plus the implicit closing stroke when a subpath's close flag is set). `BezierTo` segments advance the current point but don't emit a stroke (curves aren't useful for table/HR detection). **9c-1 — HR:** `detect_horizontal_rules(page)` filters strokes to `|y1-y2| ≤ 2pt`, length ≥ `0.3 × page_width`, stroke-width ≤ 2pt, with an underline-band filter that drops strokes within ±3pt of a text-line baseline whose x-extent matches the line. `Block::HorizontalRule` interleaves with text lines in `classify_page_with_filters` by y-coord via a peekable `hr_iter`; renders as `---`. 6 unit tests. **9c-2 — ruled-grid tables:** `detect_ruled_tables` extracts H/V segments from strokes + stroked Rect edges (so cell-rect-per-cell tables work too), clusters them into grid lines (`TABLE_GRID_CLUSTER_PT=2pt`), finds connected components via union-find on crossing pairs (`TABLE_CROSS_TOLERANCE_PT=3pt`), then per component: clusters distinct row/col coords, requires ≥2×2 cells, assigns text-line centroids to cells, rejects when empty-cell fraction > 30% or coverage > 95% of page (kills page-border rects), promotes the first row to header when all non-empty cells are `all_bold`. Runs *before* the existing borderless detector; `merge_table_runs` keeps both outputs but drops borderless runs whose range overlaps a ruled run (path geometry strictly beats text alignment). 5 unit tests. **9c-3 — figure clustering:** deferred to a later pass; limited value without image extraction (step 11) and the highest-leverage piece of figure detection (feeding figure rects into the XY-cut layout pass to fix the paper.pdf page-1 carry-forward) is now tracked as its own follow-up. **Test count:** 136 lib + 10 integration tests pass. **Smoke-test commands:** `cargo run -p liteparse -- parse <pdf> --format markdown`. Good test docs: `Anonymized rate card.pdf` (rate-card with ruled tables + HR-style dividers; should produce `---` and `|...|` table rows), any tagged invoice / financial statement / spreadsheet export. The `paper.pdf` results tables still have known residuals (MEMORYBANK row indent + short header rows) that the borderless detector handled; need to confirm the ruled detector doesn't regress those — if paper.pdf renders ruled tables incorrectly, the borderless fallback should still catch them via `merge_table_runs`. |-| 10 | Fork: structure tree + outline → highest-priority headings | ✅ | **Landed 2026-06-01.** As with step 9, no actual PDFium fork edits were needed — all FFI symbols (`FPDFBookmark_*`, `FPDFAction_*`, `FPDFDest_*`, `FPDF_StructTree_*`, `FPDF_StructElement_*`) were already in `bindings.rs`; wired the 21 missing function pointers into `PdfiumBindings` in `crates/pdfium-sys/src/dynamic.rs`. **10a — outline wrapper:** new `OutlineEntry` + `Document::outline()` in `crates/pdfium/src/document.rs` walks the bookmark tree depth-first, resolves each entry's destination (direct `FPDFBookmark_GetDest` falling back through `FPDFBookmark_GetAction` → `FPDFAction_GetDest`), then pulls the destination's page index + y (PDF user space) via `FPDFDest_GetDestPageIndex` + `FPDFDest_GetLocationInPage`. Titles decoded from UTF-16LE. **10b — struct-tree wrapper:** new `crates/pdfium/src/struct_tree.rs::Page::struct_tree(view_box)` returns `Vec<StructNode { role, mcids, bbox, alt_text }>` in pre-order. Implementation pre-scans page objects once into an `mcid → union(bbox)` map (in viewport coords via the page's `ViewportTransform`), then walks the tree calling both `FPDF_StructElement_GetMarkedContentIdCount`/`AtIndex` (modern multi-mcid form) and the legacy `GetMarkedContentID` + `GetChildMarkedContentID` getters so older tagged PDFs aren't missed. Each node's bbox is the union of bboxes for its tagged mcids. **10c — liteparse plumbing:** `OutlineTarget` + `StructNode` types added to `types.rs` (`#[serde(skip)]` so JSON/text output stays byte-identical). `LitePage.struct_nodes` populated in `extract::extract_page_struct_nodes`; threaded through `to_parsed_pages` onto `ParsedPage.struct_nodes`. New `extract::extract_outline(doc)` returns `Vec<OutlineTarget>`. `ParseResult` gains a `pub outline: Vec<OutlineTarget>` field. `parse_input` now opens the document once, pulls outline + pages + OCR-rendered pages inside a scope so `Document` is dropped before any `.await` (PDFium handle isn't `Send`). **10d — markdown classifier integration:** added `struct_heading_level(line, struct_nodes)` and `outline_heading_level(line, page_height, outline, text)` in `markdown_layout.rs`. Heading-promotion priority chain at the per-line classifier: struct-tree H1–H6 (matched via line `mcid` → node role) → outline (y_view = `page_height - y_pdf` within `1.5 × line_height`, normalized lowercased prefix-match either direction) → existing font-size histogram. `classify_page_with_filters` gains a 4th `outline: &[OutlineTarget]` param; `format_markdown` gains a 2nd `outline: &[OutlineTarget]` param and filters per-page before dispatching. **Test count:** 147 lib + 10 integration tests pass; `cargo check -p liteparse-python -p liteparse-wasm` clean. **Smoke on `paper.pdf`:** unchanged — arXiv preprints aren't tagged and don't ship an outline, so the new code is dormant (this is the expected non-regression on untagged docs). **Needs a tagged PDF for real validation:** good candidates are most government/legal forms, accessible academic publisher exports (ACM/IEEE/Springer), modern Word/InDesign exports with "Tagged PDF" enabled, and most invoices generated by enterprise tools. Smoke-test command: `cargo run -p liteparse -- parse <tagged.pdf> --format markdown`. Expected wins: section headings hit the right level even when the doc has weird font-size choices; bookmarks-driven headings pick up the right level on docs that have a TOC but no font-size hierarchy. |-| 11 | Image extraction modes (placeholder vs embed) | ✅ | **Stages 11a + 11b landed 2026-06-01.** **Stage 11a (placeholder mode):** Added `ImageMode` enum (`Off | Placeholder | Embed`, default `Placeholder`) to `config.rs` as a flat `LiteParseConfig.image_mode` field (skipped `MarkdownConfig` wrapper — easier for the language bindings, can add later if we accumulate more knobs). New `ImageRef { id, bbox, obj_index }` in `types.rs`, threaded `LitePage.image_refs` → `ParsedPage.image_refs` (#[serde(skip)] both). `extract::extract_page_image_refs(page, page_number)` reuses `Page::image_bounds(min=25pt, max_coverage=0.9)` from the existing FFI surface — no PDFium changes needed. IDs are page-scoped (`p{n}_{i}`) so they're stable across runs. New `Block::Figure { id, bbox }` rendered as ``. Classifier interleaves figures with text lines by y position (parallel to the existing `hr_iter` machinery via a new `figure_iter` + `emit_figures_before` helper, suppressed inside table y-extents). `classify_page_with_filters` gains a 5th `image_mode` param; `format_markdown` gains a 3rd. CLI: `--image-mode off|placeholder|embed` on `parse`. **Stage 11b (embed mode + bytes sidecar):** added `ExtractedImage { id, page, bbox, format, bytes }` to `types.rs`; `ParseResult.images: Vec<ExtractedImage>` (populated only when `ImageMode::Embed`, empty otherwise — placeholder mode pays no render cost). New `extract::render_page_images` reuses `Page::render_image_object` + the existing `render::encode_png` (made `pub(crate)`) to encode each image as RGBA PNG. `extract_pages_and_images` is the new core entry point; `extract_pages_from_document` is now a one-line wrapper passing `render_images=false` for callers that don't need bytes. CLI gains `--image-output-dir <dir>` which, when set with `--image-mode embed`, writes `image_{id}.png` next to the markdown output. **Bindings:** all three threaded through — pyo3 (`PyExtractedImage` with `bytes` property exposing `PyBytes`, `images` on `PyParseResult`), napi (`JsExtractedImage` with napi `Buffer` for bytes, `images` on `JsParseResult`, `image_mode` added to `JsLiteParseConfig`), wasm (`JsExtractedImage` in the result; bytes serialize as `number[]` for now, callers can wrap with `new Uint8Array(image.bytes)`). Python wrapper `packages/python/liteparse/` updated with matching `ExtractedImage` dataclass + `images` field on `ParseResult` + `image_mode` constructor arg. **Test count:** 147 lib + 10 integration tests still pass. **Smoke on `SMC2486.pdf` page 2** (the only of our local PDFs that surfaces an image object — `paper.pdf`'s figure is a vector cluster, not a raster object): `` lands above the `Status Accepted` heading in reading order; `--image-mode off` drops it cleanly. **Stage 11b smoke-tested:** `cargo run -p liteparse -- parse SMC2486.pdf --format markdown --target-pages 2 --image-mode embed --image-output-dir /tmp/lp-img --quiet` emits `` in markdown and writes the 22.4kB PNG to disk. Python: `LiteParse(image_mode="embed").parse(...).images` returns `[ExtractedImage(id='p2_0', page=2, format='png', bytes=22420)]` end-to-end. **Open items:** `--image-mode` flag on `BatchParseCommand` not added; should be a trivial follow-up. WASM bytes are a `number[]` array rather than a proper `Uint8Array` — works but inefficient; can upgrade by hand-rolling `JsParseResult` construction via `Reflect` + `js_sys::Uint8Array` later. |--Steps 1–8 = usable v1. Steps 9–11 = quality bumps for specific doc classes.--> **2026-06-01 step 10 validation pass — police.pdf (Broomfield budget) + expected-dataset/data/who_indicators.pdf.** Wrote `crates/liteparse/examples/inspect_tagged.rs` to summarize outline + struct-tree presence and cross-check outline-title coverage against rendered markdown headings without having to read the whole doc (`cargo run -q -p liteparse --example inspect_tagged -- <pdf>`). First run on `police.pdf` exposed two issues:-> - **Y-coord-out-of-page bookmarks.** Every entry came back as `y_pdf=866` on letter-size 792pt pages → `y_view = -74` → never within tolerance of any line. Either the creator's "top of page" dest type encodes y outside the MediaBox, or pages have a non-standard MediaBox; either way it's common enough to design around. Fix in `outline_heading_level`: when `y_view < 0` OR `y_view > page_height`, drop the spatial check and accept any line on the page that prefix-matches the title. Coverage 0/78 → 77/78 on `police.pdf`.-> - **Short titles latching onto long paragraphs.** Once spatial-check fallback was in, "Nutrition" (a 9-char outline title) prefix-matched a 144-char paragraph in WHO that happened to start with "Nutrition,". Fix: require the matched line to be heading-shaped — line char-count ≤ `max(title*3, 120)` AND fewer than 2 sentence-breaks (`. `). Coverage 143/156 on WHO with no observed false-positive headings.-> - **`struct_tree`: 0/457 pages tagged on `police.pdf`, 0/164 on `who_indicators.pdf`.** Neither doc actually ships a structure tree despite being well-formed PDFs with bookmarks. Validates that the outline source is doing the work; struct-tree source is dormant on these. Need a deliberately-tagged PDF (Word/InDesign export with "Tagged PDF" enabled) to exercise the H1–H6 mcid path end-to-end.-> - **Residual misses on WHO are bookmarks with no body heading.** The 13 remaining unmatched WHO entries ("Morbidity", "Mortality by cause", "Environmental risk factors", "Noncommunicable diseases", "Injuries/harmful traditional practices") are *category labels* that appear in the TOC but not as a rendered line in the body — their bookmarks point to the first sub-indicator's page (which we do match). Not a heuristic failure.-> - **Net behavior on `paper.pdf`:** unchanged (no outline / no struct tree) — verifies the change is dormant when neither source is present.->-> **2026-06-01 step 10 — outline + structure tree.** Two new high-priority heading sources, additive to the existing font-size histogram so untagged PDFs without an outline (like `paper.pdf`) get bit-identical output. **Outline source**: walks the bookmark tree via the new `Document::outline()` wrapper; per-page entries are matched to projected lines by `|y_view - line.y| ≤ 1.5 × line_height` with a lowercased+collapsed prefix-match either direction (handles "Section 1: Foo" outline vs "Foo" rendered line and vice versa). **Struct-tree source**: per-page `Page::struct_tree(view_box)` returns flattened nodes carrying their role, mcid list, and viewport-space union bbox (derived from a one-pass scan of page-object mcids). Lines whose `mcid` lands in a struct node with role `H1..H6` are promoted at that level — outranks both outline and font-size map. Figure/Table struct nodes are surfaced but not yet consumed (existing detectors handle those passes already). One subtle gotcha to track: `parse_input` now opens the PDFium document once outside the OCR await scope so the `!Send` handle is dropped before any future-suspension point.--> **2026-06-01 opendataloader-bench accuracy pass.** Switched the bench adapter `opendataloader-bench/src/pdf_parser_liteparse.py` from text mode → `output_format="markdown", image_mode="off", ocr_enabled=False` (added matching `image_mode` parameter to `packages/python/liteparse/parser.py` and the `liteparse-python` pyo3 binding so it threads through). Two follow-on fixes after smoke-reading the worst 20 docs by overall score:->-> - **Bold-style paragraph break in `continues_paragraph`** (`markdown_layout.rs`). PDFs that put section headings in a bold font variant of the body font (Brill-Bold over Brill-Roman, etc.) were losing the heading because no font-size delta separated it from the body. The classifier bold-wrapped the line then merged it into the next paragraph. Fix: when both prev and cur have a uniform style (`line_uniform_style` returns Some), a `bold` mismatch is a paragraph break. Six lines. Bumped MHS 0.320 → 0.487 on the bench corpus.-> - **Multi-line-cell wrap merge in `try_detect_table`** (`markdown_layout.rs`). Borderless tables where one column wraps onto a second baseline (e.g. "Physical or / digital" while neighbour columns stay single-line) were getting fragmented because the second baseline came back as a "partial row" with too few cells. Fix: when `cells.len() < column_count`, if every cell aligns with a known track within `TABLE_TRACK_TOLERANCE_PT` AND vertical centroid distance ≤ 1.5 × line_height, append each wrap cell's text into the matching track of the previous row rather than closing the table run. Roughly 30 lines, including comment. Bumped TEDS 0.323 → 0.384.->-> **Cumulative bench result vs original (text-mode, public 1.2.1):**->-> | Metric | Before | After 11a + fixes | Δ |-> |---|---|---|---|-> | NID | 0.866 | 0.882 | +0.016 |-> | TEDS | 0.000 | **0.384** | +0.384 |-> | MHS | 0.000 | **0.488** | +0.488 |->-> Two metrics went from 0 → mid-pack with ~80 lines of code. liteparse jumps from last place (0.576 overall) to roughly the docling/marker neighbourhood on the OpenDataLoader leaderboard.->-> **Surprise: Tesseract OCR _hurts_ on this corpus.** Re-ran with `ocr_enabled=True` and TEDS dropped to 0.369, MHS to 0.482, NID barely moved (+0.003), at 160× the per-page latency (0.81s vs 0.005s). Strongly suggests a real bug in `ocr_merge` — OCR-text likely duplicating native tokens or unbalancing the spatial merge — worth a focused investigation later. For now `ocr_enabled=False` is the right bench default and arguably the right production default for clean PDFs.->-> **2026-06-01 numbered bold-section headings + caption rejection.** Two targeted heuristic fixes in `markdown_layout.rs` after a second bench pass identified MHS=0 patterns:->-> - **`is_caption_line`** detects `Figure N`, `Fig. N`, `Table N`, `Tab. N`, `Equation N`, `Scheme`, `Chart`, `Plate`, `Photo`, `Algorithm`, `Listing` prefixes followed by a digit / roman numeral / `(N`. Applied in three places: `build_heading_map` (don't let caption sizes inflate the heading map), the font-size promotion site (don't promote captions even when their font matches a heading size), and `looks_like_bold_heading` (don't promote bold body-sized captions). Fixes doc 01030000000027 which previously emitted `# Figure 7. / # Figure 8. / # Figure 9. / # Figures 7-9 show…` as H1 headings.-> - **`looks_like_numbered_bold_heading`** runs inside the list-item branch right after `parse_list_marker` matches an ordered marker. Conditions: post-marker body ≤ 80 chars, no double-sentence punctuation, **every non-marker span in `line.spans` must pass `projection::is_bold_item` and fail `projection::is_mono_item`**, alpha-ratio ≥ 0.5, and a paragraph-break gap above (gap_below deliberately *not* required — the line is often followed by another bold body line such as a bold sub-heading or multi-line title continuation). On match, the line is promoted to a heading at `heading_map.len()+1` (typically H1 when no font-size hierarchy exists, capped at H6) instead of emitting as an ordered list item. Fixes doc 01030000000062 (`1. Shipping as a vector for marine IAS` → `# 1. …`) and doc 01030000000027's `6. Conclusion`.-> - **Span-walking trick**: marker characters often arrive inside the same span as the body text (e.g. one span `"1. Shipping as a vector for marine IAS"`), not as a separate marker span. The check classifies a span as "marker" only when *all* its non-whitespace chars are in `[0-9.()]` — so when the marker is embedded, the whole span goes through the bold check, which correctly returns true. This is why `line.all_bold == true` is the typical hit case but the per-span loop is what handles the rest.->-> **Bench result on opendataloader-bench/ (200 docs):**->-> | Metric | Pre-fix (post 11a) | Post-fix | Δ |-> |---|---|---|---|-> | NID | 0.8820 | 0.8819 | -0.0001 (noise) |-> | TEDS | 0.3842 | 0.3842 | 0.0 |-> | MHS | 0.4877 | 0.5093 | **+0.0216** |-> | Overall | 0.7451 | 0.7508 | **+0.0057** |->-> MHS=0 doc count: 17 → 17 (some new misses appeared because docs that previously had non-zero MHS lost some captions-as-headings hits — net positive though). Doc 01030000000062 went 0 → 0.495 MHS.->-> **Remaining MHS=0 docs are now dominated by column-interleaving on 2-column layouts** — e.g. doc 01030000000038's heading `6.2. Expectations for Re-Hiring Employees` projects onto the same line as body text from the adjacent column (`they had no plans to re-hire and another 36% said`), so neither the bold-heading nor the numbered-bold-heading rule sees a clean candidate. Same root cause as the long-tracked `paper.pdf` page 1 banner/figure-straddle case. Also surfaces in docs 029, 038, 039, 077, 189 — all decimal section numbers (`5.`, `6.2.`, `9.5.`, `1.5.`, `4.3.2`) on 2-column academic/policy/business reports. Fixing these needs an XY-cut / column-detection pass, not a heading-detection knob.->-> **Build orientation note:** the bench's `liteparse` Python package is installed from a built wheel pointing at `packages/python/liteparse/_liteparse.cpython-314-darwin.so` via a `.pth` file — `maturin develop` against `crates/liteparse-python/Cargo.toml` builds a *different* `_liteparse` package (under site-packages/`_liteparse/`) that the wrapper doesn't import. To get a rebuilt `.so` to take effect, run `maturin develop --release` from `packages/python/` directly. This wasted ~30 minutes the first iteration when the bench kept reporting unchanged scores against stale code.->-> **2026-06-01 column-start histogram fallback in XY-cut.** Targets the column-interleaving failure mode behind most remaining MHS=0 docs (decimal section heads on 2-column policy/academic/business reports — docs 029, 038, 039, 077, 189, …). The density-based V-cut search inside `xy_find_best_cut` fails on these layouts because a column-spanning element (wide heading, figure, table) fills the gutter at one y-band, inflating the density at the gutter's x and erasing the valley. Added `xy_find_column_cut` as a *fallback* consulted only when **both** `xy_find_best_cut` calls (H and V) return `None`.->-> **Algorithm.** Histogram every item's left-edge x with 5pt buckets across the full item span (`min_x..max_x`, not bbox-relative — see the bbox aside below). Cluster contiguous occupied buckets (allowing ≤2-bucket gaps for intra-column drift) into peaks; each peak carries its summed item count. Then apply four conservative gates *in this order*:-> - `XY_COLUMN_MIN_LINES_PER_PEAK = 10` — drop peaks with fewer than 10 items. Indented paragraph-starts and footnote markers produce 3–6 item peaks; real column-left edges have 15+ items in a typical academic page.-> - **Exactly 2 strong peaks remaining.** 3+ peaks means tabular structure (one peak per cell column); we bail rather than slice a table.-> - `XY_COLUMN_PEAK_DOMINANCE = 0.55` — the two peaks together must hold ≥55% of all items. Rules out scattered-x prose layouts (lists, lab manuals, multi-indent docs) where the two tallest peaks are tall only relative to a flat tail.-> - `XY_COLUMN_PEAK_BALANCE_RATIO = 0.4` — smaller-peak / larger-peak count ratio ≥ 0.4. Filters out "single column with one off-x bullet stack" cases (90/10 split → false-positive column cut).->-> Plus the existing geometric constraint: gap between the two peaks must be ≥ `XY_COLUMN_MIN_GAP_FRACTION = 0.25` × total item-span width. Cut position is the midpoint between the two peaks; returned `CutCandidate` carries `score = 1e9` so it beats every density-derived cut but still loses to `xy_find_banner_cut`'s `f32::INFINITY`.->-> **Why histogram items, not lines.** The natural first cut was: y-band items into lines, take each line's `x_min`, histogram those. That fails on the *exact* layouts we're trying to fix — items from the left and right columns at the same baseline fold into one "line" whose `x_min` is the left-column edge and whose `x_max` reaches the right-column tail. The right column's left edge is hidden. Switching to per-item left edges lets each column's items contribute to their own peak directly.->-> **xy_root_bbox aside.** Doc 029 (and a handful of book-spread PDFs) places content at negative user-space x (items at x=-435 with the page reported as 510pt wide). `xy_root_bbox` clamps `min_x.max(0.0)` and caps width to `page_width`, so on those pages half the items live *outside* the returned bbox and `xy_split_bbox` produces a zero-width left child after any V-cut into the negative half. Tried removing the clamp; that fixed doc 029 visibly (clean column reading order, both `# 5. The dynamics` and `# 6. Modeling the dynamics` headings emit) but regressed the bench across the board (NID 0.882→0.857, TEDS 0.384→0.296, MHS roughly flat — net -0.025 overall). Reverted. The column-start histogram still works on doc 029 *despite* the clamped bbox because the histogram operates in item-space and the cut position is in item-coords; downstream item partitioning by centroid still routes items correctly even when the bbox doesn't include the cut. **Status: clamped bbox stays. Future work would need a per-page "items extend past MediaBox" detector that conditionally widens the bbox without changing it for normal docs.**->-> **Gating evolution (what didn't work).**-> - First version fired whenever V-cut returned `None`. Hijacked tables and spreadsheets where density H-cut would have correctly emitted a row split. Regressed TEDS heavily (0.384→0.266).-> - First peak filter was `min_lines_per_peak=4`, which kept tabular peaks alive. Bumped to 10.-> - First version allowed any number of peaks, picking the widest gap. Tables with 5+ peaks (one per cell column) routinely picked a column-gap that sliced rows in half. Restricted to exactly 2 strong peaks.-> - First version had no dominance or balance gates. Many single-column docs with one footnote stack or one block-quote indent passed peak filtering. Added the 55% dominance + 0.4 balance gates.->-> **Bench result on opendataloader-bench/ (200 docs):**->-> | Metric | Pre-fix (post caption + numbered-bold) | Post-fix | Δ |-> |---|---|---|---|-> | NID | 0.8820 | **0.8822** | +0.0002 |-> | TEDS | 0.3842 | **0.3862** | +0.0020 |-> | MHS | 0.5093 | **0.5146** | +0.0053 |-> | Overall | 0.7508 | **0.7516** | +0.0008 |->-> **Doc-level wins:** doc 01030000000029 (MHS 0 → 0.278) is the most visible — `# 4. Entropy`, `# 5. The dynamics`, `# 6. Modeling the dynamics`, `## Definition 1`, `## Principle 1` all now emit correctly because the body is properly column-split before line-banding.->-> **Doc-level losses:** doc 01030000000122 (lab manual with sparse bullets in two columns) NID 0.525 → 0.471. Doc 122's histogram does show two roughly-balanced peaks at x=229 and x=457 with 92/17 items — technically passes the gates because the 92 includes a long chunk of straight-through-the-page paragraphs that all happen to start at the same x — but the doc is structurally a single column of mixed bullet/figure/table content where slicing into a "right column" creates phantom reading-order jumps. Not worth tighter gates that would also lose doc 029.->-> **Carry-forwards still open after this pass:**-> - **Approach #2 (mask spanning items from density projection).** ❌ TRIED AND REVERTED 2026-06-02 — see the 2026-06-02 entry below. Masking items ≥70% region width from the V-cut made density cuts slice tables/wide layouts column-wise and regressed all three metrics. The legitimate column case is already handled by the gated `xy_find_column_cut` fallback. Do not retry without a guard that prevents the revealed valley from cutting through tabular content.-> - **Approach #3 (split-at-big-gap inside `build_projected_lines`).** Even if XY-cut fails, the per-leaf y-banding step could split a band whose items have a horizontal gap ≥ ~6–8 × median glyph advance AND ≥ ~30pt absolute. This is what `pdftotext` does. Riskier — justified prose has wide inter-word gaps, especially near inline figures — so should be tightly gated and only as a last resort if #2 doesn't close the remainder.-> - **`xy_root_bbox` negative-x handling.** Book-spread layouts where items have x<0 break bbox-relative reasoning everywhere downstream. A conditional "items extend past MediaBox" detector that widens the bbox only for those pages would unlock the rest of doc 029 and likely a handful of other book-interior PDFs.->-> **Still-open levers identified, ranked by leverage:**-> - Multi-row table headers (16 docs, TEDS=0): patterns like `Source / Year / Description` stacked across 3 baselines where the wrap-merge fix only bridges 2.-> - Same-font headings with no font/size signal (20 docs, MHS<0.1): no detectable text-only signal short of y-clearance heuristics — risky for false positives.-> - OCR regression in `ocr_merge` — not a markdown problem per se but blocks any real OCR-on benchmark.->-> **Bench plumbing landed:** `opendataloader-bench/src/pdf_parser_liteparse_placeholder.py` (image_mode=placeholder variant) and `pdf_parser_liteparse_ocr.py` (ocr_enabled=True variant), both registered in `engine_registry.py`. `liteparse-placeholder` confirms image placeholders cost ~0.003 NID — negligible vs the structural gains, so `Placeholder` is the right default.--> **2026-06-01 TOC suppression pass.** Bench analysis vs. nutrient (0.885 overall, lp 0.745) identified two MHS failure clusters: (a) **TOC entry over-promotion** — docs 16, 18, 113, 171 each emitted 4-20 false-positive `# Chapter X 41`-style headings when the doc was just a TOC where GT wants 1 `# Contents` heading; (b) docs that are missing a single section heading on 2-column layouts (38, 77, 189) — layout-related, deferred. Fix in `markdown_layout/headings.rs` + `classify.rs`:-> - **`toc_entry_arabic_number(text)`**: returns the trailing arabic page number when the line looks like a TOC entry — ≥8 alpha chars in the body, separator that includes whitespace (rejecting decimals like "94.2" or chart values "OCR-Recall 7 94.2"), 1-4 digit tail, body doesn't end in a digit (rejecting "vol 5 12" multi-number tails). The ≥8-alpha threshold protects real one-word headings like "Chapter 7" / "Section 4".-> - **`looks_like_toc_entry`**: wraps arabic + a separate Roman-numeral check (matches "Author's Note ... ix"). Used only for the page-level row floor count.-> - **`is_toc_title`**: matches normalized "contents" / "table of contents" / "index" / "list of figures" etc. Force-promoted to H1 on TOC pages even when the heading-map size filters would have dropped it (single-occurrence, short, etc.).-> - **`page_is_toc`**: page qualifies when ≥4 TOC-like lines AND ≥3 arabic-trailing entries whose page numbers form a ≥70% non-decreasing sequence. The monotonicity check is what separates real TOCs from chart/graph pages with random axis-value tails (doc 199 was the calibration target — was getting false-positive'd, now correctly rejected).-> - **Classifier wiring** (`classify.rs::classify_page_with_filters`): per-page `toc_page = page_is_toc(page)`. Per-line `toc_suppress = toc_page && !is_toc_title(text)` guards the size-based, numbered-bold-heading, and bold-heading promotion sites. A separate `toc_title_level = Some(1)` path force-promotes the TOC's own title even when size_level was None.-> - **Tests**: 4 new unit tests (`toc_entry_arabic_extracts_trailing_page_number`, `toc_entry_arabic_rejects_decimals_and_axis_labels`, `is_toc_title_matches_common_variants`, `page_is_toc_requires_monotonic_page_numbers`). 151 lib + 10 integration tests pass.->-> **Bench result on opendataloader-bench (200 docs):**->-> | Metric | Pre-fix | Post-fix | Δ |-> |---|---|---|---|-> | NID | 0.8822 | 0.8822 | ~0 |-> | TEDS | 0.3862 | 0.3862 | 0 (no table changes) |-> | MHS | 0.5146 | **0.5358** | **+0.0212** |-> | Overall | 0.7451 | **0.7573** | **+0.0122** |->-> **Per-doc wins** (MHS): doc 16: 0.074→0.750, doc 18: 0.025→0.651, doc 113: ?→0.892, doc 171: 0.133→0.744, doc 51: regression-then-recovered 0.94, doc 153: 0.83.->-> **Still wide-open after this pass** (rough TODO priority):-> - **TEDS** still 0.39 vs nutrient 0.71. Biggest gaps are docs 89, 47, 128, 182, 88, 90 — multi-line wrapped header rows where each column's header text spans multiple baselines at staggered positions; the borderless detector closes the run after 1 row because the first "row" comes back as 1 merged cell. Other failure mode: complex spreadsheet-like grids (docs 119, 120, 121, 130, 149, 180, 121) where ground truth uses `<table>` HTML with multi-line cells and our column detector slices the layout into 2-3 tiny tables. Both need either ML-grade cell detection or a much smarter borderless table merger that walks back to absorb headers above a detected body.-> - **MHS** — remaining MHS=0 docs are dominated by 2-column section headings being eaten by adjacent-column body text (docs 7, 13, 38, 77, 118, 148, 177, 178, 189, 198). Same root cause as the long-tracked `paper.pdf` page-1 banner issue: layout decomposition can't isolate single-line bold section heads that sit at the same y-band as content in the other column. Needs the wide-banner / figure-rect-aware XY-cut refinements.-> - **TEDS multi-line header absorption** is the highest-leverage table fix — would lift docs 47, 89, 182, 88, 90 from 0.1–0.4 to ~0.7+. Approach: when ruled-grid detection or borderless detection finds a body table, walk back N lines (where N matches the detected column count's track positions) and check if those upper lines have cells aligning to the same column tracks → absorb as header rows.--> **2026-06-02 projection layer investigation — height cap, table-blocker categorization.** A diagnostic example `crates/liteparse/examples/dump_projection.rs` was added to compare raw `TextItem` y/x/height/font_size against the projected lines. Used it to characterize where the TEDS=0 docs are actually blocked. The investigation reshaped my earlier intuition that "the projection layer is the bottleneck for tables":->-> - **Doc 132** (2-col simple table): projection gives 5 clean separate lines for the 5 table rows at y=60.4/76.4/92.8/109.1/125.4 — not projection-blocked. Blocker is `TABLE_MIN_COLUMNS=3` rejecting 2-cell rows, then `continues_paragraph` merging the lines.-> - **Doc 119** (3-col fill-in-blank): projection gives 6 clean lines for the 6 row labels at y=164/211/243/269/296/323 — not projection-blocked. Blocker is `TABLE_MAX_EMPTY_CELL_FRACTION=0.30` rejecting the sparse ruled grid (≈67% empty body).-> - **Doc 130** (3-col financial): genuinely projection-blocked. XY-cut splits the table into three per-column leaves (`region_path = [1,0,0]`, `[1,0,1,0]`, `[1,0,1,1]`), producing column-major reading order. No table heuristic can fix this without re-assembling rows across leaves.-> - **Doc 121, 122** (lab worksheets): partly projection-blocked. PDFium reports anomalously large heights (h=50–80pt for ~13pt-tall glyphs — the font's bbox/lineheight got baked into the text-matrix scale). The y-band rule `(y - current_y).abs() < current_h.max(h) * 0.5` then has 25–40pt tolerance, merging distinct baselines.->-> Single-line targeted fix in `projection.rs::build_projected_lines`: clamp item height at 24pt only for the y-banding tolerance computation. The stored `bbox.height` on `ProjectedLine` is unchanged. Bench result on 200 docs: **essentially neutral** (NID 0.8819 → 0.8820, TEDS 0.4144 → 0.4144, MHS 0.5928 → 0.5922, Overall 0.7738 → 0.7737). The cap separates a few previously-merged baselines on docs 121/122 (visible in dumps) but those docs have additional reading-order issues that prevent the recovered baselines from being usable for table detection.->-> **The actual takeaway** for the user's original question: of the 14 TEDS=0 docs, **projection is a real blocker for ~3 of them (130, 121, 122)** and a contributing factor for ~2 more (146 ruled-grid component merging is upstream of build_projected_lines but in `find_grid_components`). The other 9 are markdown_layout heuristic blockers — specifically `TABLE_MIN_COLUMNS` rejecting 2-col tables (132, 165, 166, 150, 121, 197) and `TABLE_MAX_EMPTY_CELL_FRACTION` rejecting sparse ruled grids (119, 146, 180). Two attempts to relax those thresholds in this session (TABLE_MIN_COLUMNS to 2 with guards; TABLE_MAX_EMPTY_CELL_FRACTION to 85% for high-evidence grids) both went net-negative because the relaxation hijacks previously-correct borderless detections — needs an independent cell-content-quality signal (e.g. text x-position matches grid boundaries tightly) before retrying. See the next sub-note for the specific path-dependence.->-> **2026-06-02 ruled-grid single-column tables (`tables.rs`).** Targeted the largest TEDS=0 cluster (15 docs with GT tables we emitted 0 columns for). Single-col GT table was 1 doc (149), but the same code path also unblocked thin-border-strip docs whose true content column was wedged between two phantom rule-strip columns. Three coordinated changes in `build_ruled_table`:->-> - **Threshold relaxation:** `xs.len() >= 3` → `xs.len() >= 2`. Allows the detector to produce 1-column tables when the geometry only gives a single content column between borders.-> - **Phantom-row collapse:** after cell assignment, drop rows whose height < 0.8 × median non-empty row height AND that have no text. Doc 149 draws each visual row as three stacked rects (top-border 1pt, body 22pt, bottom-strip 5pt) which all survive 2pt y-clustering as separate grid rows — without collapse, the empty-cell fraction kills the table. Threshold preserves real fill-in-the-blank forms (empty body rows match median non-empty height).-> - **Phantom-column collapse:** mirror of the row rule for columns. Drop empty columns whose width < 0.3 × median non-empty column width. Doc 149 has 5.5pt left/right border-strip columns next to a 305pt content column — both drop cleanly.->-> One stricter rule in `merge_table_runs`: a 1-column ruled run **yields to** a multi-column borderless run that overlaps the same line range. Without this, doc 078's data table (drawn with top/bottom/header rules but no internal V-rules) gets detected as a degenerate 1-col grid that overrides the correct 7-col borderless result. Path geometry strictly beats text alignment for *multi*-col ruled runs, but a 1-col ruled run is ambiguous — could be a real single-column table or a multi-col table missing its V-rules — so we defer to the multi-col borderless evidence when present.->-> **Bench result (200 docs):**->-> | Metric | Pre | Post | Δ |-> |---|---|---|---|-> | NID | 0.8824 | 0.8819 | −0.0005 (noise) |-> | TEDS | 0.3906 | **0.4144** | **+0.0238** |-> | MHS | 0.5913 | 0.5928 | +0.0015 |-> | Overall | 0.7713 | **0.7738** | **+0.0025** |->-> **Per-doc:** doc 149 0 → 1.0 TEDS (clean recovery — 8-row single-col ECO Circle competence stack). Several mid-TEDS docs (128, 170, 188) also improved a few hundredths each — likely because the `0.8 × median` row collapse now bridges thin-rule rows in their existing detections. No TEDS regressions ≥ 0.01.->-> **Still wide open after this pass** — the bulk of the TEDS=0 cluster is **2-column tables** (7 docs: 121, 132, 146, 150, 165, 166, 197). The borderless detector explicitly requires `TABLE_MIN_COLUMNS = 3`, which kills every 2-col borderless table on the bench. Lowering it to 2 is high-leverage but risky: key:value bullet lists and 2-column body layouts will look the same. Needs guards before attempting. The ruled-grid path I just relaxed can pick up 2-col ruled tables (doc 146 has heavy graphics), but they don't appear to fire — likely because the connected-component step is splitting the grid, or because `find_grid_components` requires ≥2 H AND ≥2 V components to cross within 3pt and doc 146 has many tiny rect segments that don't cleanly cross. Worth dumping next pass.->-> **2026-06-02 heading char-floor, TOC-title, header absorption; XY-cut spanning-mask tried and reverted.** Four targeted heuristic changes plus one reverted layout experiment. Net result on opendataloader-bench (200 docs):->-> | Metric | Pre (260601) | Post | Δ |-> |---|---|---|---|-> | NID | 0.8822 | 0.8824 | +0.0002 |-> | TEDS | 0.3862 | **0.3906** | **+0.0044** |-> | MHS | 0.5358 | **0.5913** | **+0.0555** |-> | Overall | 0.7573 | **0.7713** | **+0.0140** |->-> - **Item 1 — heading char-floor (`headings.rs`).** `MIN_HEADING_TOTAL_CHARS` 25→10 (now just a noise guard); added `MIN_HEADING_AVG_LINE_CHARS = 8.0` as the real legend-token discriminator. `build_heading_map` now keeps a size-outlier cluster iff `avg_line_chars` is in `[8, MAX]`. Rescues clean single-line headings like "Aligning with Your Identity" (24 chars/line) that the 25-char floor dropped, while still rejecting paper-legend tokens ("A-mem"/"Base" ≈ 4-5 chars/line). MHS contributor.-> - **Item 3 — TOC title without `page_is_toc` (`classify.rs`).** `toc_title_level` now fires on `is_toc_title(text)` alone (was gated on `toc_page`). `is_toc_title` matches the whole trimmed line against an exact whitelist (contents / table of contents / index / …) so it can't fire mid-prose. MHS contributor.-> - **Item 4 — multi-line header absorption (`tables.rs`).** After the borderless detector builds a body run, `absorb_header_lines` walks backward (bounded by `floor` = prior run's end) pulling in lines whose cells (2 ≤ n ≤ column_count) all align to the established column tracks and sit tight above the row below. Multiple wrapped header lines collapse into one markdown header row (joined per-track top-to-bottom). `try_detect_table` gained a `floor` param; `detect_tables` threads the last run end through it. Single-cell titles are explicitly NOT absorbed (≥2 aligned cells required). 2 new unit tests. **TEDS +0.0044, NID neutral** — a clean win. Per-doc: 128 +0.069, 170 +0.062, 188 +0.059.-> - **Item 5 — XY-cut spanning-item mask (REVERTED).** Implemented Approach #2 (mask items ≥70% region width from the V-cut density projection). Bench showed it **hurt all three metrics**: TEDS 0.3906→0.3614, NID 0.8824→0.8780, MHS 0.5913→0.5821. Root cause: masking wide items makes density V-cuts fire *inside* tables and wide layouts, slicing them column-wise (docs 082/187/189 lost both TEDS and NID together — the signature of a split table). The existing `xy_find_column_cut` fallback (gated to fire only when both density cuts return None) already covers the legitimate column case without the collateral damage. **Conclusion: the density V-cut must stay conservative; spanning-mask is the wrong lever. Do not retry without a tabular-content guard that prevents the revealed valley from cutting through a table.** Fully reverted (constant + mask loop removed from `xy_find_best_cut`).->-> **Item 2 (sparse-form ruled-table empty-cell relaxation) remains deferred** — `TABLE_MAX_EMPTY_CELL_FRACTION` stays 0.30. Real sparse table (doc 180) and spurious decorative grid (doc 198 TOC) are both ~75% empty, so empty-fraction is not a valid discriminator; loosening it regressed TEDS −0.089.->-> **Struct-tag question answered:** 0/200 bench docs carry PDF struct tags (verified via `survey_tags` example), so the struct-tree heading path is moot for this benchmark — all gains must come from text/geometry heuristics.--## Carried-forward items--Things deferred from earlier steps that need to be revisited:--- **Figure clustering (step 9c-3)** — ✅ resolved 2026-06-01. `figure_cluster::detect_figure_rects` is wired through XY-cut as density-projection obstacles; `ParsedPage.figures` is persisted for downstream consumers (no `Block::Figure` emission yet — that lands with step 11 image extraction so figures can carry a real reference). On `paper.pdf` the abstract-column-next-to-straddling-figure case is now isolated correctly; the *separate* banner-title issue at the top of page 1 (arXiv stamp + affiliation triggering banner heuristic) is not improved by this work and remains tracked under "Page 1 banner-title layout".--- **`ProjectedLine.column_id`** — ✅ resolved 2026-05-30, then ♻️ superseded 2026-05-31 by `region_path` from the new XY-cut tree (see top note). Pages 2+ of `paper.pdf` now read coherently column-by-column. Page 1's banner-title / figure-straddle layout is still wrong but in a new shape — bumped to its own follow-up (banner detection / figure-rect-aware seeding).-- **Page 1 banner-title layout** — ⚠️ partially resolved 2026-05-31 by a text-only `xy_find_banner_cut` pre-pass in `projection.rs`. Detects y-bands (or runs of consecutive close y-bands) whose x-extent covers ≥ `XY_BANNER_WIDTH_FRACTION = 0.6` of the parent region width AND that have ≥ `XY_BANNER_CLEARANCE_FACTOR × median_h` of vertical clearance on both sides; emits a horizontal `CutCandidate` with `score = ∞` that wins unconditionally over density-based valleys. `XY_MIN_LINES_PER_H_SIDE` is bypassed for banner cuts **only at depth=0** — a 1-line title can be isolated at the page root, but aggressive single-line cuts mid-recursion (which fragment paragraphs around centered captions) are still blocked. Unit-tested: a wide centered title above a 2-column body now lands in its own root-leaf, separate from both columns; a plain 2-column layout with no banner still V-cuts at the gutter as before. Net real-doc impact: pages 2+ of `paper.pdf` gain a correct `### 4.4 Empirical Results` heading (previously absorbed into the prior paragraph). `paper.pdf` page 1 specifically (arXiv stamp + title + authors + a figure that straddles both columns + abstract) is still not fully untangled — the abstract column reads cleanly now (a win) but the title lands mid-document instead of at the top, because the arXiv timestamp and Rutgers affiliation lines satisfy the banner heuristic and disrupt the cut position. Fully solving page 1 needs ruled-figure-rect seeding from the PDFium fork (step 9) so the recursion treats the figure as an obstacle to V-cut around.-- **`ProjectedLine.indent_x`** — ✅ resolved 2026-05-31 (second pass). After `build_projected_lines` produces all lines, each leaf's min `indent_x` is subtracted from every line in that leaf, so the value is now an offset from the column's left edge. Fixes list-nesting buckets on multi-column pages — a column-2 bullet at page-x=300 previously looked indented +230pt vs column-1 prose, now reads as `indent_x ≈ 0`.-- **`ProjectedLine.all_strike`** — still placeholder `false`. PDFium doesn't expose strikethrough as a span attribute; would need either path-cluster detection (a horizontal line at mid-glyph) or marked-content role hints. Not blocking v1.-- **Mid-line (per-span) emphasis** — ✅ resolved 2026-05-30. `render_line_inline` consumes `ProjectedLine.spans` directly. Known minor: when a styled span ends in a soft hyphen at end of line, dehyphenation skips it (so `*sys-*` + `tem` stays as `*sys-* tem` rather than merging into `*system*`). Acceptable trade-off vs. surgically stripping markers around emphasis wraps.-- **Inline links** — needs PDF annotation link rects exposed via the fork; deferred with the rest of the structure-tree-tier work (step 10).-- **`MarkdownConfig`** — not yet introduced. Holding off until at least one heuristic has a knob worth exposing. Default-only v1 is fine per the plan.--## Open decisions still pending (from plan)--- Per-page vs whole-doc emission (default per-page with `joinPages` opt-in?).-- `Block[]` as a structured output format (`OutputFormat::Blocks`)?-- `GridFallback` fence style: ``` ```text ``` vs `<pre>`. Currently using ` ```text `.-- Confidence/OCR provenance encoding in markdown.--## Notes / surprises--- The Node CLI formats output client-side from `ParseResult`. ✅ resolved 2026-05-31: `result.text` from the napi binding already carries the Rust-formatted markdown (parser's `parse_input` calls `markdown::format_markdown` when `output_format == Markdown` and stuffs the result into `ParseResult.text`). No separate TS-side stub turned out to be in the tree — the note pre-dated the wiring. `node dist/cli.js parse file.pdf --format markdown` and the Python CLI both produce real markdown end-to-end.-- **Step 5/6 sanity check on `paper.pdf`:** headings still detect (`##` / `####` / `#####`), no false code blocks fired on prose (font-name heuristics correctly skip serif/sans body fonts), and no spurious bold paragraphs. No mono content on this paper to exercise the fence path — covered by unit tests instead.-- Smoke test on `paper.pdf` (a two-column arXiv paper): headings (`#`/`##`/`###`/`####`) detect correctly and de-hyphenation joins `prede-/fined → predefined`. Two columns interleave on the same lines — expected until real `column_id` lands (carry-forward from step 2).-- **Tuning that came out of single-column smoke tests (`hard_5.pdf`, `llama2_4pages.pdf`):**- - PDFium can return `font_size: 1.0` for every glyph when the font size is baked into the text matrix instead of the Tf operator. `build_projected_lines` now falls back to char-weighted `bbox.height` when the histogram size is ≤ 1.5pt — necessary for `llama2_4pages.pdf` to even detect headings.- - `ProjectedLine.text` carries column-alignment spaces from the grid projection. The paragraph emitter runs `collapse_whitespace` on every line before joining so prose doesn't look like `for instance`.- - Justified body text alternates dominant anchors (Left / Right / Floating) line-by-line. Treating any anchor change as a paragraph break shreds normal paragraphs. The break rule now only fires on a *centered* vs *non-centered* flip.- - Two font-size tolerances: tight (`0.6pt`) for matching the heading map, loose (`2.5pt`) for paragraph continuation, since the height fallback varies a few points based on descenders.-- **Known limitation:** de-hyphenation joins `closed-/source → closedsource`. We can't tell line-break hyphens apart from genuine compound hyphens without a dictionary. pymupdf4llm has the same issue; not blocking.-- **Footnote follow-up:** academic/legal docs often have footnote numbers like `138.` mid-paragraph (`...settlement outcomes.138`) that are real superscripts. They render attached to the preceding word, but the dedicated footnote block at the page bottom is correctly captured as `138. <text>`. The inline marker is informational; not worth splitting in v1.-- `cargo build --workspace` fails on macOS because `liteparse-python` needs to link against Python (build via maturin instead). Use `cargo build -p liteparse -p liteparse-napi` + `cargo check -p liteparse-python -p liteparse-wasm` for local verification.