A fast, helpful, and open-source document parser
wip
Logan Markewich8dd0b3553cbf
diff --git a/MARKDOWN_PROGRESS.md b/MARKDOWN_PROGRESS.mdindex 541289b..5e98f9a 100644--- a/MARKDOWN_PROGRESS.md+++ b/MARKDOWN_PROGRESS.md@@ -2,6 +2,52 @@Tracks build-order progress against [MARKDOWN_PLAN.md](MARKDOWN_PLAN.md). Update as steps complete.+> **2026-06-14 (pm) launch-readiness cosmetic-noise + prose-chrome cleanup — audit followups #2 and #3 from the 2026-06-14 audit entry. Three contained markdown-emission fixes SHIPPED; 208 lib tests pass (2 new). Net: opendataloader clean win, ParseBench table + olmOCR flat.** All three are emission-/classification-layer changes that the three benches structurally barely see (the defects live in prose ornaments + form-table cosmetics that the corpora lack), so they were measured primarily for *non-regression*, with the real payoff being human-perceived output quality. Audit scratch under `/tmp/lp_audit/` (base/ = pre-change binary `lit-base`, new/ = shipping binary).+>+> **Fix A — decorative divider / flourish lines (`markdown_layout/classify.rs`, audit #2).** A line whose trimmed text has **no alphanumeric content** and consists only of divider symbols (`* * * *`, `———`, `____`) was flowing into the paragraph accumulator and gluing the ornament onto the *start* of the next paragraph (8× in the Berkshire 2023 letter: `\* \* \* \* … Writers find it useful…`), and a lone decorative em-dash under the title was getting size-promoted to a stray `# -` heading. New `decorative_divider_kind(text)` run *before* heading/paragraph classification (after the code-block flush): ≥3 symbols → `Block::HorizontalRule` (a real thematic break); 1–2 symbols → dropped (too small to be a meaningful rule). Conservative symbol set (`* - _ – — • · ● ▪ ■ ◦ ★ ☆`) — excludes `.`/`=`/`~`/`+`/`#` to avoid dot-leaders, ellipses, setext rules. `is_alphanumeric()` gate means any letter/digit (incl. CJK) → not decorative, so single-char numeric/letter headings are untouched. Verified on Berkshire: `# -` gone, the 8 dividers now emit as standalone `---` with the previously-glued prose intact as its own paragraph.+>+> **Fix B — horizontal-rule dedupe (`output/markdown.rs`, audit #3 part 1).** `dedupe_rules()` runs per page before `render_blocks`: drops leading/trailing `Block::HorizontalRule` (which abut the `\n\n-----\n\n` page separator) and collapses consecutive HR runs to one. HRs come from two sources (vector-graphics detection + Fix A's decorative dividers); doubling reads as sloppy and carries no structure. Verified: irs_1040_instr `---` count 338 → 173, zero doubled `---` runs remain across all audit docs.+>+> **Fix C — empty-header table rows (`markdown_layout/blocks.rs`, audit #3 part 2).** Header-less tables (`header == None`) were rendering a visible blank `| | | |` header (17× in irs_1040, pervasive in the IRS forms). GFM requires a header row before the separator, so instead of synthesizing a blank one we now **promote the first body row** to the header. Verified: irs_1040 empty-header rows 17 → 0.+>+> **A/B (lit-base = pre-change build incl. the SERFF fix, same session):**+> - **opendataloader** (Fix A+B): clean win — overall 0.874144 → **0.875060** (+0.00092), NID 0.912719 → **0.913230** (+0.0005), **MHS 0.813624 → 0.816560 (+0.00294)**, TEDS flat 0.692940. Blast radius 17/200 docs. Fix C on top: **byte-flat** (overall/NID/TEDS/MHS all identical — header-less tables here either don't change TEDS structure or TEDS normalizes the thead/tbody tag).+> - **ParseBench table**: **flat across all three fixes** — grits_con 0.57309 / TRM 0.21960 / composite 0.40756 (matches the documented post-SERFF baseline to 5 digits; decorative dividers/HRs are absent from the single-page table crops, and promoting row0→header doesn't change which cells exist).+> - **olmOCR** (regression-check only per [[project_bench_focus]]): Overall **39.1%**, headers_footers **56.1%**, multi_column 65.8%, table_tests 48.2%, long_tiny_text 29.2% — matches the post-SERFF baseline (39.1% / 56.1%) within noise. The changes only *remove* noise text (ornaments, `# -`, doubled rules) so they can't help-or-hurt the content-keyed `absent`/`order`/`table` predicates beyond noise.+>+> **Decision: ship all three** (cosmetic quality wins, net-positive/flat on the two prioritized benches, no olmOCR regression). Remaining open audit items are **#2 chrome's sibling** (none — fully addressed) and the two intentionally-deferred classes: prose run-on in forms and **dot-leader TOCs / noisy form tables** (audit #4/#5, documented as known-hard, not chased). Files: `crates/liteparse/src/markdown_layout/{classify.rs,blocks.rs}`, `crates/liteparse/src/output/markdown.rs`.+>+> **2026-06-14 (pm) SERFF row-drop followup — root cause was CHROME STRIPPING, not the table/region-merge path the prior note guessed. Fix SHIPPED (owner: ship-as-is); 207 lib tests pass (1 new).** Reproduced on `FBLB-134215544_page6`: the entire top key-value band `SERFF Tracking #: FBLB-134215544 State Tracking #: Company Tracking #: MS-2024-07` was absent from markdown while the `--format text` grid had it. Traced through xy_cut (the line IS a clean 1-line leaf, `d=1`), region grouping, and table detection — none dropped it. The actual culprit: `markdown_layout/repetition.rs::detect_single_page_chrome` → `matches_chrome_pattern`'s journal-citation branch (`has_year && has_digit_range && len ≤ 120`) **misfired on the embedded date** `MS-2024-07`, read as year `2024` + page-range `2024-07`. Line is top-band + isolated (gap 19pt ≥ 10pt) → stripped as chrome.+>+> **Fix (`has_digit_range`).** A `YYYY-MM` / `YYYY-MM-DD` token is a date, not a citation page range: when the first digit group is a 4-digit `19xx`/`20xx` year and the second group is ≤2 digits, skip that match and keep scanning for a genuine range elsewhere on the line. Real journal cites carry a *distinct* page range (`1–9`, `436-444`) that is unaffected; the existing `detects_bottom_journal_citation_chrome` test still passes. Semantically correct — the date-as-range read was always wrong; it just happened to delete some real chrome by accident (see olmOCR cost below). Provably *un-strip only*: the change can only make `has_digit_range` return false, so it adds content back, never strips more.+>+> **A/B (true same-session baselines — stash-built `lit` binary + reverted-and-rebuilt Python binding):**+> - **opendataloader** (clean win): overall 0.874082 → **0.874144** (+0.00006), NID 0.912594 → **0.912719** (+0.000125), TEDS/MHS flat. Byte-diff blast radius **1/200** — doc `01030000000149` recovers `Project No: : 2021-2-FR02-KA220-YOU-000048126` (was chrome-stripped via the same date-as-range misfire), verbatim in the GT.+> - **ParseBench table**: grits_con 0.573091 / TRM 0.219599 / composite 0.407560 — **byte-flat**. The recovered line is loose content above the table; GriTS scores cells only, so no structure change. **NB: the prior note's claim that this fix recovers the −0.0007 table dip was a MISDIAGNOSIS** — that dip is the V-cut min-lines guard's table-structure effect, unrelated to this content drop. The dip is still open (separate followup).+> - **olmOCR** (regresses, accepted): overall 39.2% → 39.1%, headers_footers 56.8% → **56.1% (−6 tests)**. Blast radius 4 docs: 2 wins (`8dd368bd…` metadata table `CM-FECL3-E | issued: 2019-04-17 …` recovered from chrome; `tables/7b9b…` cell `1972-21-143 | Geriadu` recovered), and 2–3 short date-only chrome lines now survive and fail `absent` predicates (`Date: 2019-03-15`, `2020-06-11 1/4`, `2018-04-17 10:41 - Tyler Nguyen`). Those lines were only ever deleted *via the wrong citation heuristic*; the principled recovery would be an explicit short-date-chrome rule, but every length/structure discriminator tried overlaps the recovered content (metadata-table rows are also short, date-bearing), so it was deferred rather than risk re-introducing content drops.+>+> **Decision: ship as-is** (owner). The dropped-row class is the audit's #1 user-facing defect (silent loss of tracking/project IDs + metadata tables for RAG); net-positive on the two prioritized benches ([[project_bench_focus]]), and the olmOCR cost is short date chrome that was passing on an accident. Files: `crates/liteparse/src/markdown_layout/repetition.rs`. Memory: [[project_chrome_date_falsepos]].+>+> **2026-06-14 launch-readiness sanity audit on out-of-distribution internet PDFs — two correctness fixes SHIPPED, net-positive A/B, 218 lib tests pass.** Ran the markdown emitter (default user-facing settings) on five PDFs deliberately *not* in any benchmark corpus (Berkshire Hathaway 2023 shareholder letter, IRS Form 1040 + i1040 instructions, a CDC MMWR multi-column bulletin, an ACL Anthology 2-column paper) to check for benchmark overfitting and real legibility. The layout engine generalized well (clean prose, correct multi-column reading order); the defects clustered in territory the three benches structurally don't cover. Audit scratch outputs were under `/tmp/lp_audit/`.+>+> **Fix #1 — ligature doubling on no-`/ToUnicode` LaTeX/Type1 fonts (`extract.rs::GlyphDecoder`).** On the ACL paper every `fi`/`fl` word was doubled: `find`→`fifind`, `beneficial`→`benefificial`, `coefficients`→`coeffificients`, `fluctuations`→`flfluctuations` (21 occurrences; **0** in the other four docs → font-family-specific, the pdfTeX/Nimbus family that dominates academic PDFs — the #1 RAG input). Root cause: with no `/ToUnicode`, PDFium derives per-char unicode from the glyph names itself and *already* expands a single ligature glyph (e.g. the `fi` glyph at char_code `0x02`) into separate `'f'` and `'i'` TextChar entries that **share that one char_code**. Because the font is flagged `untrusted` (no ToUnicode + "Custom" encoding), the glyph-name recovery re-resolved the multi-char name `"fi"` once per entry → `"fi"+"fi"`. This was NOT the documented Mode 16 ligature *substitution*; it is *duplication*, and was independent of the (non-existent — see below) `DISABLE_GLYPH_NAMES` switch. **Fix:** when a glyph-name resolution is multi-char but PDFium's per-char unicode is non-suspicious *and* a member of the resolved string, trust PDFium's own expansion (`return None`). Suspicious-char recoveries (control-code ligatures `0x1A–0x1F`, glyph soup) fire on control/PUA/map-error codepoints and are untouched — verified: `text_simple__spd` still recovers "Fraktion … Bezirksvertretung Dortmund Mengede", `paper_cn_trad` still recovers CJK 消息/出席/幹部. Bench impact not separately measured this session (pure extraction fidelity → expected to move olmOCR char-exact predicates; both arms of the #2 A/B below already include this fix). Memory: [[project_glyph_decode]].+>+> **Fix #2 — vertical-cut min-lines guard (`projection.rs::xy_cut_rec`).** On a single-column page the markdown emitter rendered correctly-ordered prose out of order: Berkshire p7 emitted "…pay dividends, and its **discretionary. Annual debt maturities are never material.** / share repurchases are 100%" (materially wrong financial statement) while the `--format text` grid was correct. Root cause: `xy_find_best_cut` found a spurious vertical "gutter" in a 3-line region where one line is full-width with a wide justified inter-word gap and the line below is short — and the min-lines-per-side guard in `xy_cut_rec` was gated on `cut.axis == Horizontal`, so **vertical cuts had no such guard**. The V-cut isolated the 1-line fragment "share repurchases are 100%" into a sibling region, which the region-ordered emitter then placed after the rest. **Fix:** apply the same guard to vertical cuts — reject when either side has `< XY_MIN_LINES_PER_H_SIDE` (2) distinct lines. By construction this can only block `lc<2 || rc<2` cuts; a real multi-row table/column split always has ≥2 lines per side, so it never damages genuine column or table structure. Kill-switch `LITEPARSE_DISABLE_VCUT_MINLINES`. Memory: [[project_column_layout]].+>+> **Scored A/B (guard ON = shipped vs OFF = baseline; both binaries include Fix #1):**+> - opendataloader (clean win): overall 0.873345 → **0.874082** (+0.00074), MHS 0.810963 → **0.813624** (+0.00266), NID 0.912543 → 0.912594 (+0.00005), TEDS 0.692941 **flat**. Byte-diff blast radius 5/200 docs, all wins/neutral: docs 124/126 a section heading now correctly follows its preceding figure caption (was hoisted above it), 144 an equation number `(3.16)` rejoins its equation line, 031 a math caption kept whole, 069 a page-number token kept inline.+> - ParseBench table (tiny dip, within noise): grits_con 0.57309 → 0.57239 (−0.00070), composite 0.40756 → 0.40721 (−0.00035), TRM 0.21960 → 0.21961 (+0.00001). Byte-diff blast radius 169/2579. The dip is **not lost table structure** (the guard cannot block multi-row cuts) — it is the SERFF row-drop followup below.+>+> Net: opendataloader + (esp. MHS), ParseBench table − a hair (both within the ±0.001–0.002 band the logs treat as noise), plus a real qualitative correctness win (no more scrambled prose / mis-ordered captions). Shipped as-is per owner. **On-disk bench outputs currently reflect the baseline (guard-OFF) arm of the A/B — re-run a fresh sweep on the shipping binary before trusting `output/`/`prediction/`.**+>+> **Doc-hygiene note:** several kill-switches referenced in this file / FAILURE_MODES are **not actually implemented** in the code and are silently ignored (e.g. `LITEPARSE_DISABLE_GLYPH_NAMES`, and the `LITEPARSE_DISABLE_GUTTER_CUT` / `DISABLE_NEARSTART_RESCUE` / `DISABLE_GLOBAL_RULED` referenced in earlier entries). Verify a switch exists before relying on it for A/B.+>+> **Followups (deferred to next session — tracked here, not yet started):**+> 1. **SERFF row-drop (cause of the ParseBench table −0.0007 dip).** On key-value header bands like `FBLB-134215544_page6` ("SERFF Tracking #: FBLB-… | State Tracking #:"), the blocked 1-line V-cut (`lc=1 rc=1`, correctly rejected) causes that whole row to be **dropped** downstream — it survives in neither the table nor as a loose line. So the row was being kept only because the (wrong) V-cut pre-split it. The real fix is in the table/region-merge path (a 1-line key-value band that isn't pre-split must still emit), not the guard.+> 2. **Audit #3 — prose "chrome" never handled.** Decorative section dividers `* * * * * * …` get glued onto the *start* of the next paragraph (8× in the Berkshire letter), and a decorative em-dash was promoted to a stray `# -` heading. The bench corpora (gov/academic/table docs) contain none of these, so they were never tuned.+> 3. **Audit #4 — pervasive cosmetic noise (cheap, high perceived-quality impact).** Stray/doubled `---` horizontal rules and empty-header `| | | |` table rows appear across most docs (worst in the IRS forms). Bench scorers tokenize/normalize these away (zero metric cost) so they were never penalized, but they look sloppy to a human reader. Suppressing empty-header rows and collapsing adjacent/empty HRs would visibly lift output quality everywhere.+> 4. **Audit #5 — forms & dot-leader TOCs (intrinsically hard).** IRS 1040 form + i1040 instructions TOC render as noisy empty tables and run-on dot-leader entries. Document as a known-hard class rather than chase; even pymupdf4llm struggles here.+>> **2026-06-13 (pm) trailing-period loss fix (Mode 16, `extract.rs` dot-leader threshold) — olmOCR Overall 38.6 → 39.1% (+0.5pp); order 53.0 → 54.4% (+14), multi_column 63.6 → 65.3% (+15 tests), long_tiny_text 25.8 → 29.2% (+15 tests), present 15.8 → 17.9%; headers_footers −2 / table_tests −5 (CI noise). opendataloader BYTE-FLAT (NID 0.9126 / TEDS 0.6929 / MHS 0.8036), ParseBench text_content FLAT (0.65097 both, forced re-infer A/B) and table FLAT (grits_con 0.5731 / composite 0.4076 / TRM 0.2196). 207 lib tests pass. One-line change, shipped.** Resolves the 2026-06-08 FAILURE_MODES Mode 16 "trailing-period loss (~25 tests, probably localized)" item — the cleanest remaining lever in the `text_absent` class flagged by the 2026-06-13 formatting-triage pivot, and explicitly NOT the compressed-metric-font danger zone.>> **Root cause (single misfiring threshold).** The no-pending-space dot-leader splitter at `extract.rs:644` broke a `.` off preceding content whenever `gap > avg_char_width * 0.4`. A loosely-kerned abbreviation/sentence period sits at **~1.0× avg char width** (measured `g/cw ≈ 1.0` on `Sci.`/`Chem.`/`Environ.`/`Phys.`), so it tripped the splitter; the split-off `.` segment then merged forward into the next token and vanished. Tightly-kerned periods (`Mater.` at `g/cw=0.14`) stayed attached, which is why the loss was *positional* (only before column/tab gaps in reference lists) and looked per-word. Genuine no-space dot leaders run `g/cw ≈ 2–23` (verified a real one at 23.2), and spaced leaders take the *other* (`pending_space`) branch entirely — so raising the cutoff `0.4 → 2.0` opens a wide safe valley.diff --git a/crates/liteparse/src/extract.rs b/crates/liteparse/src/extract.rsindex a528d56..2f4fd72 100644--- a/crates/liteparse/src/extract.rs+++ b/crates/liteparse/src/extract.rs@@ -1110,7 +1110,7 @@ impl GlyphDecoder {reverse_cmap,..} = info;- cache+ let resolved = cache.entry(char_code).or_insert_with(|| {let name = font.char_glyph_name(char_code);@@ -1159,8 +1159,25 @@ impl GlyphDecoder {);}resolved- })- .as_deref()+ });+ // Don't double-expand a ligature PDFium already split. With no+ // /ToUnicode, PDFium derives per-char unicodes from the glyph names+ // itself, expanding a single ligature glyph (e.g. the "fi" glyph at+ // char_code 0x02) into separate 'f' and 'i' TextChar entries that all+ // share that one char_code. Resolving the multi-char glyph name ("fi")+ // once per entry would emit "fi"+"fi" → "fifind". When PDFium already+ // gave a clean (non-suspicious) char that is part of the resolved+ // string, it has done the expansion — keep its char. Suspicious-char+ // recoveries (control-code ligatures, glyph soup) still expand.+ if let Some(r) = resolved.as_deref()+ && r.chars().count() > 1+ && !cheap_suspicious+ && let Some(u) = char::from_u32(unicode)+ && r.contains(u)+ {+ return None;+ }+ resolved.as_deref()}}diff --git a/crates/liteparse/src/markdown_layout/blocks.rs b/crates/liteparse/src/markdown_layout/blocks.rsindex 2a97dfe..b5d25bb 100644--- a/crates/liteparse/src/markdown_layout/blocks.rs+++ b/crates/liteparse/src/markdown_layout/blocks.rs@@ -169,38 +169,34 @@ pub fn render_blocks(blocks: &[Block]) -> String {out.push_str(&wrap_emphasis(text, *bold, *italic));}Block::Table { header, rows } => {- let column_count = header- .as_ref()- .map(|h| h.len())- .or_else(|| rows.first().map(|r| r.len()))- .unwrap_or(0);+ // GFM requires a header row before the separator. When the+ // detector found no header, promote the first body row instead+ // of synthesizing a blank `| | |` header — a visible empty+ // row reads as sloppy output and carries no information.+ let (head, body): (Option<&[String]>, &[Vec<String>]) = match header {+ Some(h) => (Some(h.as_slice()), rows.as_slice()),+ None => match rows.split_first() {+ Some((first, rest)) => (Some(first.as_slice()), rest),+ None => (None, rows.as_slice()),+ },+ };+ let column_count = head.map(|h| h.len()).unwrap_or(0);if column_count == 0 {continue;}- if let Some(h) = header {- out.push_str("| ");- for (i, cell) in h.iter().enumerate() {- if i > 0 {- out.push_str(" | ");- }- out.push_str(&escape_table_cell(cell));- }- out.push_str(" |\n");- } else {- // CommonMark/GFM requires a header row before the- // separator; synthesize a blank header so renderers that- // refuse header-less tables still display the body.- out.push('|');- for _ in 0..column_count {- out.push_str(" |");+ out.push_str("| ");+ for (i, cell) in head.unwrap().iter().enumerate() {+ if i > 0 {+ out.push_str(" | ");}- out.push('\n');+ out.push_str(&escape_table_cell(cell));}+ out.push_str(" |\n");out.push('|');for _ in 0..column_count {out.push_str("---|");}- for row in rows {+ for row in body {out.push_str("\n| ");for (i, cell) in row.iter().enumerate() {if i > 0 {@@ -380,15 +376,13 @@ mod tests {}#[test]- fn render_table_without_header_synthesizes_blank_header() {+ fn render_table_without_header_promotes_first_row() {let blocks = vec![Block::Table {header: None,- rows: vec![vec!["1".into(), "2".into()]],+ rows: vec![vec!["h1".into(), "h2".into()], vec!["1".into(), "2".into()]],}];let s = render_blocks(&blocks);- // GFM/CommonMark needs a header row before the separator; we emit a- // blank one so renderers don't choke.- assert!(s.contains("|---|---|"));- assert!(s.ends_with("| 1 | 2 |"));+ // No blank `| | |` header: the first row becomes the header.+ assert_eq!(s, "| h1 | h2 |\n|---|---|\n| 1 | 2 |");}}diff --git a/crates/liteparse/src/markdown_layout/classify.rs b/crates/liteparse/src/markdown_layout/classify.rsindex 7453618..0ffcd75 100644--- a/crates/liteparse/src/markdown_layout/classify.rs+++ b/crates/liteparse/src/markdown_layout/classify.rs@@ -577,6 +577,26 @@ fn classify_region(// Any non-mono line ends the current code block (if any).state.flush_code(&mut blocks);+ // Decorative divider / flourish lines (`* * * *`, a lone em-dash).+ // Handled before heading/paragraph classification so the ornament+ // neither glues onto the next paragraph nor gets promoted to a heading.+ if let Some(is_rule) = decorative_divider_kind(text) {+ state.flush_paragraph(&mut blocks);+ state.reset_list();+ heading_run = None;+ if is_rule {+ blocks.push(Block::HorizontalRule);+ }+ if debug {+ eprintln!(+ "[md decorative] {} '{}'",+ if is_rule { "rule" } else { "drop" },+ text.chars().take(40).collect::<String>(),+ );+ }+ continue;+ }+// Priority chain: tagged-PDF struct tree → outline → font-size map.let tagged_level = struct_heading_level(line, &page.struct_nodes);// Caption lines ("Figure 7", "Table 3.") are routinely set in a@@ -903,6 +923,43 @@ fn classify_region(blocks}+/// Classify a line that is *purely decorative* — no alphanumeric content, made+/// up only of divider symbols (`* * * *`, `———`, `____`). Returns:+/// - `Some(true)` → a section divider (≥3 symbols): emit a thematic break.+/// - `Some(false)` → a lone 1–2 char flourish (a stray em-dash under a+/// title): drop it (too small to be a meaningful rule).+/// - `None` → not decorative; classify normally.+///+/// Without this, a `* * * *` divider line flows into the paragraph accumulator+/// and glues the ornament onto the start of the following paragraph, and a lone+/// decorative dash gets size-promoted to a `# -` heading.+fn decorative_divider_kind(text: &str) -> Option<bool> {+ let mut symbols = 0usize;+ for c in text.chars() {+ if c.is_whitespace() {+ continue;+ }+ if c.is_alphanumeric() || !is_divider_symbol(c) {+ return None;+ }+ symbols += 1;+ }+ if symbols == 0 {+ return None;+ }+ Some(symbols >= 3)+}++/// Characters treated as decorative divider/ornament glyphs. Deliberately+/// conservative — excludes `.`, `=`, `~`, `+`, `#` which carry meaning elsewhere+/// (dot leaders, ellipses, setext-ish rules, headers).+fn is_divider_symbol(c: char) -> bool {+ matches!(+ c,+ '*' | '-' | '_' | '–' | '—' | '•' | '·' | '●' | '▪' | '■' | '◦' | '★' | '☆'+ )+}+/// Best-effort language hint for a fenced code block, used as the fence/// info-string. Conservative: returns `Some` only when the body carries/// strong, language-specific signals; otherwise `None` (bare fence). Thisdiff --git a/crates/liteparse/src/markdown_layout/repetition.rs b/crates/liteparse/src/markdown_layout/repetition.rsindex 273ba64..fc20e14 100644--- a/crates/liteparse/src/markdown_layout/repetition.rs+++ b/crates/liteparse/src/markdown_layout/repetition.rs@@ -220,6 +220,7 @@ fn has_digit_range(t: &str) -> bool {while j < chars.len() && chars[j].is_ascii_digit() {j += 1;}+ let g1_len = j - i;// Skip optional spaces.let mut k = j;while k < chars.len() && chars[k] == ' ' {@@ -231,7 +232,25 @@ fn has_digit_range(t: &str) -> bool {m += 1;}if m < chars.len() && chars[m].is_ascii_digit() {- return true;+ let mut n = m + 1;+ while n < chars.len() && chars[n].is_ascii_digit() {+ n += 1;+ }+ let g2_len = n - m;+ // A `YYYY-MM` / `YYYY-MM-DD` date is not a page range. A real+ // citation page range never starts with a 4-digit calendar+ // year whose second component is a 1-2 digit month — that+ // pattern is an administrative date (e.g. a "MS-2024-07"+ // tracking number), so skip it and keep scanning for a+ // genuine range elsewhere on the line.+ let g1_is_year = g1_len == 4+ && ((chars[i] == '1' && chars[i + 1] == '9')+ || (chars[i] == '2' && chars[i + 1] == '0'));+ if !(g1_is_year && g2_len <= 2) {+ return true;+ }+ i = n;+ continue;}}i = j;@@ -445,6 +464,16 @@ mod tests {assert!(!matches_chrome_pattern("Introduction"));// Title with year but no range — should not be stripped as chrome.assert!(!matches_chrome_pattern("Acme Annual Report 2023"));+ // Administrative key-value band with a YYYY-MM tracking date — the+ // date must not be read as a journal page range.+ assert!(!matches_chrome_pattern(+ "SERFF Tracking #: FBLB-134215544 State Tracking #: Company Tracking #: MS-2024-07"+ ));+ // A genuine YYYY-MM / YYYY-MM-DD date is not a page range.+ assert!(!has_digit_range("Filed 2024-07"));+ assert!(!has_digit_range("Generated 2025-01-23"));+ // A real page range still registers, even with a year on the line.+ assert!(has_digit_range("Vol 24, 1-9, 2017"));}#[test]diff --git a/crates/liteparse/src/output/markdown.rs b/crates/liteparse/src/output/markdown.rsindex 1ebd8d2..ee3cb1d 100644--- a/crates/liteparse/src/output/markdown.rs+++ b/crates/liteparse/src/output/markdown.rs@@ -77,11 +77,29 @@ pub fn format_markdown(if !has_content {blocks.retain(|b| !matches!(b, crate::markdown_layout::Block::HorizontalRule));}+ dedupe_rules(&mut blocks);out.push_str(&render_blocks(&blocks));}out}+/// Collapse cosmetic horizontal-rule noise on a single page's block stream:+/// drop leading/trailing rules (which would otherwise abut the `-----` page+/// separator) and collapse runs of consecutive rules to one. Rules come from+/// two sources — vector-graphics detection and decorative divider text — and+/// doubling up reads as sloppy output to a human, while carrying no extra+/// structure for an LLM.+fn dedupe_rules(blocks: &mut Vec<crate::markdown_layout::Block>) {+ use crate::markdown_layout::Block::HorizontalRule;+ while matches!(blocks.first(), Some(HorizontalRule)) {+ blocks.remove(0);+ }+ while matches!(blocks.last(), Some(HorizontalRule)) {+ blocks.pop();+ }+ blocks.dedup_by(|a, b| matches!((a, b), (HorizontalRule, HorizontalRule)));+}+#[cfg(test)]mod tests {use super::*;@@ -133,6 +151,31 @@ mod tests {assert_eq!(format_markdown(&[], &[], ImageMode::Placeholder), "");}+ #[test]+ fn dedupe_rules_drops_edges_and_collapses_runs() {+ use crate::markdown_layout::Block::{self, HorizontalRule, Paragraph};+ let p = |t: &str| Paragraph {+ text: t.into(),+ bold: false,+ italic: false,+ };+ let mut blocks = vec![+ HorizontalRule,+ p("a"),+ HorizontalRule,+ HorizontalRule,+ p("b"),+ HorizontalRule,+ ];+ dedupe_rules(&mut blocks);+ let kinds: Vec<bool> = blocks+ .iter()+ .map(|b| matches!(b, Block::HorizontalRule))+ .collect();+ // Leading + trailing rules gone; the doubled interior run collapsed to one.+ assert_eq!(kinds, vec![false, true, false]);+ }+#[test]fn test_fallback_when_no_projected_lines() {let p = ParsedPage {diff --git a/crates/liteparse/src/projection.rs b/crates/liteparse/src/projection.rsindex c525918..5008c46 100644--- a/crates/liteparse/src/projection.rs+++ b/crates/liteparse/src/projection.rs@@ -4046,6 +4046,7 @@ fn xy_cut_rec(None};let debug_xy = std::env::var("LITEPARSE_DEBUG_XY").is_ok();+ let disable_vcut_minlines = std::env::var("LITEPARSE_DISABLE_VCUT_MINLINES").is_ok();if debug_xy && depth == 0 && std::env::var("LITEPARSE_DEBUG_ITEMS").is_ok() {for &i in &idxs {let it = &items[i].item;@@ -4376,6 +4377,29 @@ fn xy_cut_rec(continue;}}+ // Vertical (column) min-lines guard. A real column gutter is sustained+ // across multiple stacked lines on both sides. A vertical cut that+ // isolates a single-line "column" is a false gutter: in single-column+ // justified prose, a short final line plus a wide inter-word gap in the+ // full-width line above it align an empty vertical band that is not a+ // column boundary. Without this, that band splits the physical line+ // into two sibling regions and the markdown emitter renders them out of+ // order (e.g. "…and its share repurchases are 100%" → the tail is+ // sliced off into its own region). The initial true 2-column split has+ // many lines per side, so this never blocks real columns.+ if cut.axis == CutAxis::Vertical && !disable_vcut_minlines {+ let lc = xy_distinct_lines(items, &left, median_h);+ let rc = xy_distinct_lines(items, &right, median_h);+ if lc < XY_MIN_LINES_PER_H_SIDE || rc < XY_MIN_LINES_PER_H_SIDE {+ if debug_xy {+ let pad = " ".repeat(depth as usize);+ eprintln!(+ "[xy d={depth}]{pad} -> v-cut min-lines fail (lc={lc} rc={rc}), try next"+ );+ }+ continue;+ }+ }if debug_xy {let pad = " ".repeat(depth as usize);