Skip to content

A fast, helpful, and open-source document parser

wip

Logan Markewich8dd0b3553cbf
diff --git a/MARKDOWN_PROGRESS.md b/MARKDOWN_PROGRESS.md
index 541289b..5e98f9a 100644
--- a/MARKDOWN_PROGRESS.md
+++ b/MARKDOWN_PROGRESS.md
@@ -2,6 +2,52 @@
Tracks build-order progress against [MARKDOWN_PLAN.md](MARKDOWN_PLAN.md). Update as steps complete.
+> **2026-06-14 (pm) launch-readiness cosmetic-noise + prose-chrome cleanup — audit followups #2 and #3 from the 2026-06-14 audit entry. Three contained markdown-emission fixes SHIPPED; 208 lib tests pass (2 new). Net: opendataloader clean win, ParseBench table + olmOCR flat.** All three are emission-/classification-layer changes that the three benches structurally barely see (the defects live in prose ornaments + form-table cosmetics that the corpora lack), so they were measured primarily for *non-regression*, with the real payoff being human-perceived output quality. Audit scratch under `/tmp/lp_audit/` (base/ = pre-change binary `lit-base`, new/ = shipping binary).
+>
+> **Fix A — decorative divider / flourish lines (`markdown_layout/classify.rs`, audit #2).** A line whose trimmed text has **no alphanumeric content** and consists only of divider symbols (`* * * *`, `———`, `____`) was flowing into the paragraph accumulator and gluing the ornament onto the *start* of the next paragraph (8× in the Berkshire 2023 letter: `\* \* \* \* … Writers find it useful…`), and a lone decorative em-dash under the title was getting size-promoted to a stray `# -` heading. New `decorative_divider_kind(text)` run *before* heading/paragraph classification (after the code-block flush): ≥3 symbols → `Block::HorizontalRule` (a real thematic break); 1–2 symbols → dropped (too small to be a meaningful rule). Conservative symbol set (`* - _ – — • · ● ▪ ■ ◦ ★ ☆`) — excludes `.`/`=`/`~`/`+`/`#` to avoid dot-leaders, ellipses, setext rules. `is_alphanumeric()` gate means any letter/digit (incl. CJK) → not decorative, so single-char numeric/letter headings are untouched. Verified on Berkshire: `# -` gone, the 8 dividers now emit as standalone `---` with the previously-glued prose intact as its own paragraph.
+>
+> **Fix B — horizontal-rule dedupe (`output/markdown.rs`, audit #3 part 1).** `dedupe_rules()` runs per page before `render_blocks`: drops leading/trailing `Block::HorizontalRule` (which abut the `\n\n-----\n\n` page separator) and collapses consecutive HR runs to one. HRs come from two sources (vector-graphics detection + Fix A's decorative dividers); doubling reads as sloppy and carries no structure. Verified: irs_1040_instr `---` count 338 → 173, zero doubled `---` runs remain across all audit docs.
+>
+> **Fix C — empty-header table rows (`markdown_layout/blocks.rs`, audit #3 part 2).** Header-less tables (`header == None`) were rendering a visible blank `| | | |` header (17× in irs_1040, pervasive in the IRS forms). GFM requires a header row before the separator, so instead of synthesizing a blank one we now **promote the first body row** to the header. Verified: irs_1040 empty-header rows 17 → 0.
+>
+> **A/B (lit-base = pre-change build incl. the SERFF fix, same session):**
+> - **opendataloader** (Fix A+B): clean win — overall 0.874144 → **0.875060** (+0.00092), NID 0.912719 → **0.913230** (+0.0005), **MHS 0.813624 → 0.816560 (+0.00294)**, TEDS flat 0.692940. Blast radius 17/200 docs. Fix C on top: **byte-flat** (overall/NID/TEDS/MHS all identical — header-less tables here either don't change TEDS structure or TEDS normalizes the thead/tbody tag).
+> - **ParseBench table**: **flat across all three fixes** — grits_con 0.57309 / TRM 0.21960 / composite 0.40756 (matches the documented post-SERFF baseline to 5 digits; decorative dividers/HRs are absent from the single-page table crops, and promoting row0→header doesn't change which cells exist).
+> - **olmOCR** (regression-check only per [[project_bench_focus]]): Overall **39.1%**, headers_footers **56.1%**, multi_column 65.8%, table_tests 48.2%, long_tiny_text 29.2% — matches the post-SERFF baseline (39.1% / 56.1%) within noise. The changes only *remove* noise text (ornaments, `# -`, doubled rules) so they can't help-or-hurt the content-keyed `absent`/`order`/`table` predicates beyond noise.
+>
+> **Decision: ship all three** (cosmetic quality wins, net-positive/flat on the two prioritized benches, no olmOCR regression). Remaining open audit items are **#2 chrome's sibling** (none — fully addressed) and the two intentionally-deferred classes: prose run-on in forms and **dot-leader TOCs / noisy form tables** (audit #4/#5, documented as known-hard, not chased). Files: `crates/liteparse/src/markdown_layout/{classify.rs,blocks.rs}`, `crates/liteparse/src/output/markdown.rs`.
+>
+> **2026-06-14 (pm) SERFF row-drop followup — root cause was CHROME STRIPPING, not the table/region-merge path the prior note guessed. Fix SHIPPED (owner: ship-as-is); 207 lib tests pass (1 new).** Reproduced on `FBLB-134215544_page6`: the entire top key-value band `SERFF Tracking #: FBLB-134215544 State Tracking #: Company Tracking #: MS-2024-07` was absent from markdown while the `--format text` grid had it. Traced through xy_cut (the line IS a clean 1-line leaf, `d=1`), region grouping, and table detection — none dropped it. The actual culprit: `markdown_layout/repetition.rs::detect_single_page_chrome` → `matches_chrome_pattern`'s journal-citation branch (`has_year && has_digit_range && len ≤ 120`) **misfired on the embedded date** `MS-2024-07`, read as year `2024` + page-range `2024-07`. Line is top-band + isolated (gap 19pt ≥ 10pt) → stripped as chrome.
+>
+> **Fix (`has_digit_range`).** A `YYYY-MM` / `YYYY-MM-DD` token is a date, not a citation page range: when the first digit group is a 4-digit `19xx`/`20xx` year and the second group is ≤2 digits, skip that match and keep scanning for a genuine range elsewhere on the line. Real journal cites carry a *distinct* page range (`1–9`, `436-444`) that is unaffected; the existing `detects_bottom_journal_citation_chrome` test still passes. Semantically correct — the date-as-range read was always wrong; it just happened to delete some real chrome by accident (see olmOCR cost below). Provably *un-strip only*: the change can only make `has_digit_range` return false, so it adds content back, never strips more.
+>
+> **A/B (true same-session baselines — stash-built `lit` binary + reverted-and-rebuilt Python binding):**
+> - **opendataloader** (clean win): overall 0.874082 → **0.874144** (+0.00006), NID 0.912594 → **0.912719** (+0.000125), TEDS/MHS flat. Byte-diff blast radius **1/200** — doc `01030000000149` recovers `Project No: : 2021-2-FR02-KA220-YOU-000048126` (was chrome-stripped via the same date-as-range misfire), verbatim in the GT.
+> - **ParseBench table**: grits_con 0.573091 / TRM 0.219599 / composite 0.407560 — **byte-flat**. The recovered line is loose content above the table; GriTS scores cells only, so no structure change. **NB: the prior note's claim that this fix recovers the −0.0007 table dip was a MISDIAGNOSIS** — that dip is the V-cut min-lines guard's table-structure effect, unrelated to this content drop. The dip is still open (separate followup).
+> - **olmOCR** (regresses, accepted): overall 39.2% → 39.1%, headers_footers 56.8% → **56.1% (−6 tests)**. Blast radius 4 docs: 2 wins (`8dd368bd…` metadata table `CM-FECL3-E | issued: 2019-04-17 …` recovered from chrome; `tables/7b9b…` cell `1972-21-143 | Geriadu` recovered), and 2–3 short date-only chrome lines now survive and fail `absent` predicates (`Date: 2019-03-15`, `2020-06-11 1/4`, `2018-04-17 10:41 - Tyler Nguyen`). Those lines were only ever deleted *via the wrong citation heuristic*; the principled recovery would be an explicit short-date-chrome rule, but every length/structure discriminator tried overlaps the recovered content (metadata-table rows are also short, date-bearing), so it was deferred rather than risk re-introducing content drops.
+>
+> **Decision: ship as-is** (owner). The dropped-row class is the audit's #1 user-facing defect (silent loss of tracking/project IDs + metadata tables for RAG); net-positive on the two prioritized benches ([[project_bench_focus]]), and the olmOCR cost is short date chrome that was passing on an accident. Files: `crates/liteparse/src/markdown_layout/repetition.rs`. Memory: [[project_chrome_date_falsepos]].
+>
+> **2026-06-14 launch-readiness sanity audit on out-of-distribution internet PDFs — two correctness fixes SHIPPED, net-positive A/B, 218 lib tests pass.** Ran the markdown emitter (default user-facing settings) on five PDFs deliberately *not* in any benchmark corpus (Berkshire Hathaway 2023 shareholder letter, IRS Form 1040 + i1040 instructions, a CDC MMWR multi-column bulletin, an ACL Anthology 2-column paper) to check for benchmark overfitting and real legibility. The layout engine generalized well (clean prose, correct multi-column reading order); the defects clustered in territory the three benches structurally don't cover. Audit scratch outputs were under `/tmp/lp_audit/`.
+>
+> **Fix #1 — ligature doubling on no-`/ToUnicode` LaTeX/Type1 fonts (`extract.rs::GlyphDecoder`).** On the ACL paper every `fi`/`fl` word was doubled: `find`→`fifind`, `beneficial`→`benefificial`, `coefficients`→`coeffificients`, `fluctuations`→`flfluctuations` (21 occurrences; **0** in the other four docs → font-family-specific, the pdfTeX/Nimbus family that dominates academic PDFs — the #1 RAG input). Root cause: with no `/ToUnicode`, PDFium derives per-char unicode from the glyph names itself and *already* expands a single ligature glyph (e.g. the `fi` glyph at char_code `0x02`) into separate `'f'` and `'i'` TextChar entries that **share that one char_code**. Because the font is flagged `untrusted` (no ToUnicode + "Custom" encoding), the glyph-name recovery re-resolved the multi-char name `"fi"` once per entry → `"fi"+"fi"`. This was NOT the documented Mode 16 ligature *substitution*; it is *duplication*, and was independent of the (non-existent — see below) `DISABLE_GLYPH_NAMES` switch. **Fix:** when a glyph-name resolution is multi-char but PDFium's per-char unicode is non-suspicious *and* a member of the resolved string, trust PDFium's own expansion (`return None`). Suspicious-char recoveries (control-code ligatures `0x1A–0x1F`, glyph soup) fire on control/PUA/map-error codepoints and are untouched — verified: `text_simple__spd` still recovers "Fraktion … Bezirksvertretung Dortmund Mengede", `paper_cn_trad` still recovers CJK 消息/出席/幹部. Bench impact not separately measured this session (pure extraction fidelity → expected to move olmOCR char-exact predicates; both arms of the #2 A/B below already include this fix). Memory: [[project_glyph_decode]].
+>
+> **Fix #2 — vertical-cut min-lines guard (`projection.rs::xy_cut_rec`).** On a single-column page the markdown emitter rendered correctly-ordered prose out of order: Berkshire p7 emitted "…pay dividends, and its **discretionary. Annual debt maturities are never material.** / share repurchases are 100%" (materially wrong financial statement) while the `--format text` grid was correct. Root cause: `xy_find_best_cut` found a spurious vertical "gutter" in a 3-line region where one line is full-width with a wide justified inter-word gap and the line below is short — and the min-lines-per-side guard in `xy_cut_rec` was gated on `cut.axis == Horizontal`, so **vertical cuts had no such guard**. The V-cut isolated the 1-line fragment "share repurchases are 100%" into a sibling region, which the region-ordered emitter then placed after the rest. **Fix:** apply the same guard to vertical cuts — reject when either side has `< XY_MIN_LINES_PER_H_SIDE` (2) distinct lines. By construction this can only block `lc<2 || rc<2` cuts; a real multi-row table/column split always has ≥2 lines per side, so it never damages genuine column or table structure. Kill-switch `LITEPARSE_DISABLE_VCUT_MINLINES`. Memory: [[project_column_layout]].
+>
+> **Scored A/B (guard ON = shipped vs OFF = baseline; both binaries include Fix #1):**
+> - opendataloader (clean win): overall 0.873345 → **0.874082** (+0.00074), MHS 0.810963 → **0.813624** (+0.00266), NID 0.912543 → 0.912594 (+0.00005), TEDS 0.692941 **flat**. Byte-diff blast radius 5/200 docs, all wins/neutral: docs 124/126 a section heading now correctly follows its preceding figure caption (was hoisted above it), 144 an equation number `(3.16)` rejoins its equation line, 031 a math caption kept whole, 069 a page-number token kept inline.
+> - ParseBench table (tiny dip, within noise): grits_con 0.57309 → 0.57239 (−0.00070), composite 0.40756 → 0.40721 (−0.00035), TRM 0.21960 → 0.21961 (+0.00001). Byte-diff blast radius 169/2579. The dip is **not lost table structure** (the guard cannot block multi-row cuts) — it is the SERFF row-drop followup below.
+>
+> Net: opendataloader + (esp. MHS), ParseBench table − a hair (both within the ±0.001–0.002 band the logs treat as noise), plus a real qualitative correctness win (no more scrambled prose / mis-ordered captions). Shipped as-is per owner. **On-disk bench outputs currently reflect the baseline (guard-OFF) arm of the A/B — re-run a fresh sweep on the shipping binary before trusting `output/`/`prediction/`.**
+>
+> **Doc-hygiene note:** several kill-switches referenced in this file / FAILURE_MODES are **not actually implemented** in the code and are silently ignored (e.g. `LITEPARSE_DISABLE_GLYPH_NAMES`, and the `LITEPARSE_DISABLE_GUTTER_CUT` / `DISABLE_NEARSTART_RESCUE` / `DISABLE_GLOBAL_RULED` referenced in earlier entries). Verify a switch exists before relying on it for A/B.
+>
+> **Followups (deferred to next session — tracked here, not yet started):**
+> 1. **SERFF row-drop (cause of the ParseBench table −0.0007 dip).** On key-value header bands like `FBLB-134215544_page6` ("SERFF Tracking #: FBLB-… | State Tracking #:"), the blocked 1-line V-cut (`lc=1 rc=1`, correctly rejected) causes that whole row to be **dropped** downstream — it survives in neither the table nor as a loose line. So the row was being kept only because the (wrong) V-cut pre-split it. The real fix is in the table/region-merge path (a 1-line key-value band that isn't pre-split must still emit), not the guard.
+> 2. **Audit #3 — prose "chrome" never handled.** Decorative section dividers `* * * * * * …` get glued onto the *start* of the next paragraph (8× in the Berkshire letter), and a decorative em-dash was promoted to a stray `# -` heading. The bench corpora (gov/academic/table docs) contain none of these, so they were never tuned.
+> 3. **Audit #4 — pervasive cosmetic noise (cheap, high perceived-quality impact).** Stray/doubled `---` horizontal rules and empty-header `| | | |` table rows appear across most docs (worst in the IRS forms). Bench scorers tokenize/normalize these away (zero metric cost) so they were never penalized, but they look sloppy to a human reader. Suppressing empty-header rows and collapsing adjacent/empty HRs would visibly lift output quality everywhere.
+> 4. **Audit #5 — forms & dot-leader TOCs (intrinsically hard).** IRS 1040 form + i1040 instructions TOC render as noisy empty tables and run-on dot-leader entries. Document as a known-hard class rather than chase; even pymupdf4llm struggles here.
+>
> **2026-06-13 (pm) trailing-period loss fix (Mode 16, `extract.rs` dot-leader threshold) — olmOCR Overall 38.6 → 39.1% (+0.5pp); order 53.0 → 54.4% (+14), multi_column 63.6 → 65.3% (+15 tests), long_tiny_text 25.8 → 29.2% (+15 tests), present 15.8 → 17.9%; headers_footers −2 / table_tests −5 (CI noise). opendataloader BYTE-FLAT (NID 0.9126 / TEDS 0.6929 / MHS 0.8036), ParseBench text_content FLAT (0.65097 both, forced re-infer A/B) and table FLAT (grits_con 0.5731 / composite 0.4076 / TRM 0.2196). 207 lib tests pass. One-line change, shipped.** Resolves the 2026-06-08 FAILURE_MODES Mode 16 "trailing-period loss (~25 tests, probably localized)" item — the cleanest remaining lever in the `text_absent` class flagged by the 2026-06-13 formatting-triage pivot, and explicitly NOT the compressed-metric-font danger zone.
>
> **Root cause (single misfiring threshold).** The no-pending-space dot-leader splitter at `extract.rs:644` broke a `.` off preceding content whenever `gap > avg_char_width * 0.4`. A loosely-kerned abbreviation/sentence period sits at **~1.0× avg char width** (measured `g/cw ≈ 1.0` on `Sci.`/`Chem.`/`Environ.`/`Phys.`), so it tripped the splitter; the split-off `.` segment then merged forward into the next token and vanished. Tightly-kerned periods (`Mater.` at `g/cw=0.14`) stayed attached, which is why the loss was *positional* (only before column/tab gaps in reference lists) and looked per-word. Genuine no-space dot leaders run `g/cw ≈ 2–23` (verified a real one at 23.2), and spaced leaders take the *other* (`pending_space`) branch entirely — so raising the cutoff `0.4 → 2.0` opens a wide safe valley.
diff --git a/crates/liteparse/src/extract.rs b/crates/liteparse/src/extract.rs
index a528d56..2f4fd72 100644
--- a/crates/liteparse/src/extract.rs
+++ b/crates/liteparse/src/extract.rs
@@ -1110,7 +1110,7 @@ impl GlyphDecoder {
reverse_cmap,
..
} = info;
- cache
+ let resolved = cache
.entry(char_code)
.or_insert_with(|| {
let name = font.char_glyph_name(char_code);
@@ -1159,8 +1159,25 @@ impl GlyphDecoder {
);
}
resolved
- })
- .as_deref()
+ });
+ // Don't double-expand a ligature PDFium already split. With no
+ // /ToUnicode, PDFium derives per-char unicodes from the glyph names
+ // itself, expanding a single ligature glyph (e.g. the "fi" glyph at
+ // char_code 0x02) into separate 'f' and 'i' TextChar entries that all
+ // share that one char_code. Resolving the multi-char glyph name ("fi")
+ // once per entry would emit "fi"+"fi" → "fifind". When PDFium already
+ // gave a clean (non-suspicious) char that is part of the resolved
+ // string, it has done the expansion — keep its char. Suspicious-char
+ // recoveries (control-code ligatures, glyph soup) still expand.
+ if let Some(r) = resolved.as_deref()
+ && r.chars().count() > 1
+ && !cheap_suspicious
+ && let Some(u) = char::from_u32(unicode)
+ && r.contains(u)
+ {
+ return None;
+ }
+ resolved.as_deref()
}
}
diff --git a/crates/liteparse/src/markdown_layout/blocks.rs b/crates/liteparse/src/markdown_layout/blocks.rs
index 2a97dfe..b5d25bb 100644
--- a/crates/liteparse/src/markdown_layout/blocks.rs
+++ b/crates/liteparse/src/markdown_layout/blocks.rs
@@ -169,38 +169,34 @@ pub fn render_blocks(blocks: &[Block]) -> String {
out.push_str(&wrap_emphasis(text, *bold, *italic));
}
Block::Table { header, rows } => {
- let column_count = header
- .as_ref()
- .map(|h| h.len())
- .or_else(|| rows.first().map(|r| r.len()))
- .unwrap_or(0);
+ // GFM requires a header row before the separator. When the
+ // detector found no header, promote the first body row instead
+ // of synthesizing a blank `| | |` header — a visible empty
+ // row reads as sloppy output and carries no information.
+ let (head, body): (Option<&[String]>, &[Vec<String>]) = match header {
+ Some(h) => (Some(h.as_slice()), rows.as_slice()),
+ None => match rows.split_first() {
+ Some((first, rest)) => (Some(first.as_slice()), rest),
+ None => (None, rows.as_slice()),
+ },
+ };
+ let column_count = head.map(|h| h.len()).unwrap_or(0);
if column_count == 0 {
continue;
}
- if let Some(h) = header {
- out.push_str("| ");
- for (i, cell) in h.iter().enumerate() {
- if i > 0 {
- out.push_str(" | ");
- }
- out.push_str(&escape_table_cell(cell));
- }
- out.push_str(" |\n");
- } else {
- // CommonMark/GFM requires a header row before the
- // separator; synthesize a blank header so renderers that
- // refuse header-less tables still display the body.
- out.push('|');
- for _ in 0..column_count {
- out.push_str(" |");
+ out.push_str("| ");
+ for (i, cell) in head.unwrap().iter().enumerate() {
+ if i > 0 {
+ out.push_str(" | ");
}
- out.push('\n');
+ out.push_str(&escape_table_cell(cell));
}
+ out.push_str(" |\n");
out.push('|');
for _ in 0..column_count {
out.push_str("---|");
}
- for row in rows {
+ for row in body {
out.push_str("\n| ");
for (i, cell) in row.iter().enumerate() {
if i > 0 {
@@ -380,15 +376,13 @@ mod tests {
}
#[test]
- fn render_table_without_header_synthesizes_blank_header() {
+ fn render_table_without_header_promotes_first_row() {
let blocks = vec![Block::Table {
header: None,
- rows: vec![vec!["1".into(), "2".into()]],
+ rows: vec![vec!["h1".into(), "h2".into()], vec!["1".into(), "2".into()]],
}];
let s = render_blocks(&blocks);
- // GFM/CommonMark needs a header row before the separator; we emit a
- // blank one so renderers don't choke.
- assert!(s.contains("|---|---|"));
- assert!(s.ends_with("| 1 | 2 |"));
+ // No blank `| | |` header: the first row becomes the header.
+ assert_eq!(s, "| h1 | h2 |\n|---|---|\n| 1 | 2 |");
}
}
diff --git a/crates/liteparse/src/markdown_layout/classify.rs b/crates/liteparse/src/markdown_layout/classify.rs
index 7453618..0ffcd75 100644
--- a/crates/liteparse/src/markdown_layout/classify.rs
+++ b/crates/liteparse/src/markdown_layout/classify.rs
@@ -577,6 +577,26 @@ fn classify_region(
// Any non-mono line ends the current code block (if any).
state.flush_code(&mut blocks);
+ // Decorative divider / flourish lines (`* * * *`, a lone em-dash).
+ // Handled before heading/paragraph classification so the ornament
+ // neither glues onto the next paragraph nor gets promoted to a heading.
+ if let Some(is_rule) = decorative_divider_kind(text) {
+ state.flush_paragraph(&mut blocks);
+ state.reset_list();
+ heading_run = None;
+ if is_rule {
+ blocks.push(Block::HorizontalRule);
+ }
+ if debug {
+ eprintln!(
+ "[md decorative] {} '{}'",
+ if is_rule { "rule" } else { "drop" },
+ text.chars().take(40).collect::<String>(),
+ );
+ }
+ continue;
+ }
+
// Priority chain: tagged-PDF struct tree → outline → font-size map.
let tagged_level = struct_heading_level(line, &page.struct_nodes);
// Caption lines ("Figure 7", "Table 3.") are routinely set in a
@@ -903,6 +923,43 @@ fn classify_region(
blocks
}
+/// Classify a line that is *purely decorative* — no alphanumeric content, made
+/// up only of divider symbols (`* * * *`, `———`, `____`). Returns:
+/// - `Some(true)` → a section divider (≥3 symbols): emit a thematic break.
+/// - `Some(false)` → a lone 1–2 char flourish (a stray em-dash under a
+/// title): drop it (too small to be a meaningful rule).
+/// - `None` → not decorative; classify normally.
+///
+/// Without this, a `* * * *` divider line flows into the paragraph accumulator
+/// and glues the ornament onto the start of the following paragraph, and a lone
+/// decorative dash gets size-promoted to a `# -` heading.
+fn decorative_divider_kind(text: &str) -> Option<bool> {
+ let mut symbols = 0usize;
+ for c in text.chars() {
+ if c.is_whitespace() {
+ continue;
+ }
+ if c.is_alphanumeric() || !is_divider_symbol(c) {
+ return None;
+ }
+ symbols += 1;
+ }
+ if symbols == 0 {
+ return None;
+ }
+ Some(symbols >= 3)
+}
+
+/// Characters treated as decorative divider/ornament glyphs. Deliberately
+/// conservative — excludes `.`, `=`, `~`, `+`, `#` which carry meaning elsewhere
+/// (dot leaders, ellipses, setext-ish rules, headers).
+fn is_divider_symbol(c: char) -> bool {
+ matches!(
+ c,
+ '*' | '-' | '_' | '–' | '—' | '•' | '·' | '●' | '▪' | '■' | '◦' | '★' | '☆'
+ )
+}
+
/// Best-effort language hint for a fenced code block, used as the fence
/// info-string. Conservative: returns `Some` only when the body carries
/// strong, language-specific signals; otherwise `None` (bare fence). This
diff --git a/crates/liteparse/src/markdown_layout/repetition.rs b/crates/liteparse/src/markdown_layout/repetition.rs
index 273ba64..fc20e14 100644
--- a/crates/liteparse/src/markdown_layout/repetition.rs
+++ b/crates/liteparse/src/markdown_layout/repetition.rs
@@ -220,6 +220,7 @@ fn has_digit_range(t: &str) -> bool {
while j < chars.len() && chars[j].is_ascii_digit() {
j += 1;
}
+ let g1_len = j - i;
// Skip optional spaces.
let mut k = j;
while k < chars.len() && chars[k] == ' ' {
@@ -231,7 +232,25 @@ fn has_digit_range(t: &str) -> bool {
m += 1;
}
if m < chars.len() && chars[m].is_ascii_digit() {
- return true;
+ let mut n = m + 1;
+ while n < chars.len() && chars[n].is_ascii_digit() {
+ n += 1;
+ }
+ let g2_len = n - m;
+ // A `YYYY-MM` / `YYYY-MM-DD` date is not a page range. A real
+ // citation page range never starts with a 4-digit calendar
+ // year whose second component is a 1-2 digit month — that
+ // pattern is an administrative date (e.g. a "MS-2024-07"
+ // tracking number), so skip it and keep scanning for a
+ // genuine range elsewhere on the line.
+ let g1_is_year = g1_len == 4
+ && ((chars[i] == '1' && chars[i + 1] == '9')
+ || (chars[i] == '2' && chars[i + 1] == '0'));
+ if !(g1_is_year && g2_len <= 2) {
+ return true;
+ }
+ i = n;
+ continue;
}
}
i = j;
@@ -445,6 +464,16 @@ mod tests {
assert!(!matches_chrome_pattern("Introduction"));
// Title with year but no range — should not be stripped as chrome.
assert!(!matches_chrome_pattern("Acme Annual Report 2023"));
+ // Administrative key-value band with a YYYY-MM tracking date — the
+ // date must not be read as a journal page range.
+ assert!(!matches_chrome_pattern(
+ "SERFF Tracking #: FBLB-134215544 State Tracking #: Company Tracking #: MS-2024-07"
+ ));
+ // A genuine YYYY-MM / YYYY-MM-DD date is not a page range.
+ assert!(!has_digit_range("Filed 2024-07"));
+ assert!(!has_digit_range("Generated 2025-01-23"));
+ // A real page range still registers, even with a year on the line.
+ assert!(has_digit_range("Vol 24, 1-9, 2017"));
}
#[test]
diff --git a/crates/liteparse/src/output/markdown.rs b/crates/liteparse/src/output/markdown.rs
index 1ebd8d2..ee3cb1d 100644
--- a/crates/liteparse/src/output/markdown.rs
+++ b/crates/liteparse/src/output/markdown.rs
@@ -77,11 +77,29 @@ pub fn format_markdown(
if !has_content {
blocks.retain(|b| !matches!(b, crate::markdown_layout::Block::HorizontalRule));
}
+ dedupe_rules(&mut blocks);
out.push_str(&render_blocks(&blocks));
}
out
}
+/// Collapse cosmetic horizontal-rule noise on a single page's block stream:
+/// drop leading/trailing rules (which would otherwise abut the `-----` page
+/// separator) and collapse runs of consecutive rules to one. Rules come from
+/// two sources — vector-graphics detection and decorative divider text — and
+/// doubling up reads as sloppy output to a human, while carrying no extra
+/// structure for an LLM.
+fn dedupe_rules(blocks: &mut Vec<crate::markdown_layout::Block>) {
+ use crate::markdown_layout::Block::HorizontalRule;
+ while matches!(blocks.first(), Some(HorizontalRule)) {
+ blocks.remove(0);
+ }
+ while matches!(blocks.last(), Some(HorizontalRule)) {
+ blocks.pop();
+ }
+ blocks.dedup_by(|a, b| matches!((a, b), (HorizontalRule, HorizontalRule)));
+}
+
#[cfg(test)]
mod tests {
use super::*;
@@ -133,6 +151,31 @@ mod tests {
assert_eq!(format_markdown(&[], &[], ImageMode::Placeholder), "");
}
+ #[test]
+ fn dedupe_rules_drops_edges_and_collapses_runs() {
+ use crate::markdown_layout::Block::{self, HorizontalRule, Paragraph};
+ let p = |t: &str| Paragraph {
+ text: t.into(),
+ bold: false,
+ italic: false,
+ };
+ let mut blocks = vec![
+ HorizontalRule,
+ p("a"),
+ HorizontalRule,
+ HorizontalRule,
+ p("b"),
+ HorizontalRule,
+ ];
+ dedupe_rules(&mut blocks);
+ let kinds: Vec<bool> = blocks
+ .iter()
+ .map(|b| matches!(b, Block::HorizontalRule))
+ .collect();
+ // Leading + trailing rules gone; the doubled interior run collapsed to one.
+ assert_eq!(kinds, vec![false, true, false]);
+ }
+
#[test]
fn test_fallback_when_no_projected_lines() {
let p = ParsedPage {
diff --git a/crates/liteparse/src/projection.rs b/crates/liteparse/src/projection.rs
index c525918..5008c46 100644
--- a/crates/liteparse/src/projection.rs
+++ b/crates/liteparse/src/projection.rs
@@ -4046,6 +4046,7 @@ fn xy_cut_rec(
None
};
let debug_xy = std::env::var("LITEPARSE_DEBUG_XY").is_ok();
+ let disable_vcut_minlines = std::env::var("LITEPARSE_DISABLE_VCUT_MINLINES").is_ok();
if debug_xy && depth == 0 && std::env::var("LITEPARSE_DEBUG_ITEMS").is_ok() {
for &i in &idxs {
let it = &items[i].item;
@@ -4376,6 +4377,29 @@ fn xy_cut_rec(
continue;
}
}
+ // Vertical (column) min-lines guard. A real column gutter is sustained
+ // across multiple stacked lines on both sides. A vertical cut that
+ // isolates a single-line "column" is a false gutter: in single-column
+ // justified prose, a short final line plus a wide inter-word gap in the
+ // full-width line above it align an empty vertical band that is not a
+ // column boundary. Without this, that band splits the physical line
+ // into two sibling regions and the markdown emitter renders them out of
+ // order (e.g. "…and its share repurchases are 100%" → the tail is
+ // sliced off into its own region). The initial true 2-column split has
+ // many lines per side, so this never blocks real columns.
+ if cut.axis == CutAxis::Vertical && !disable_vcut_minlines {
+ let lc = xy_distinct_lines(items, &left, median_h);
+ let rc = xy_distinct_lines(items, &right, median_h);
+ if lc < XY_MIN_LINES_PER_H_SIDE || rc < XY_MIN_LINES_PER_H_SIDE {
+ if debug_xy {
+ let pad = " ".repeat(depth as usize);
+ eprintln!(
+ "[xy d={depth}]{pad} -> v-cut min-lines fail (lc={lc} rc={rc}), try next"
+ );
+ }
+ continue;
+ }
+ }
if debug_xy {
let pad = " ".repeat(depth as usize);