A fast, helpful, and open-source document parser
wip
Logan Markewichfe03a4f3159b
diff --git a/MARKDOWN_PROGRESS.md b/MARKDOWN_PROGRESS.mdindex 73ea9d4..1488435 100644--- a/MARKDOWN_PROGRESS.md+++ b/MARKDOWN_PROGRESS.md@@ -154,7 +154,7 @@ Steps 1–8 = usable v1. Steps 9–11 = quality bumps for specific doc classes.> **Doc-level losses:** doc 01030000000122 (lab manual with sparse bullets in two columns) NID 0.525 → 0.471. Doc 122's histogram does show two roughly-balanced peaks at x=229 and x=457 with 92/17 items — technically passes the gates because the 92 includes a long chunk of straight-through-the-page paragraphs that all happen to start at the same x — but the doc is structurally a single column of mixed bullet/figure/table content where slicing into a "right column" creates phantom reading-order jumps. Not worth tighter gates that would also lose doc 029.>> **Carry-forwards still open after this pass:**-> - **Approach #2 (mask spanning items from density projection).** Before computing the V-cut density valley, drop items whose width ≥ ~70% of the region width — those are headings/figures/tables that span columns and are the reason the density valley search fails in the first place. Symmetric to the figure-obstacle code added 2026-06-01 (which masks figures along *H-cuts*). Should catch the 14 still-MHS=0 docs whose layout has a clean gutter once spanning elements are excluded.+> - **Approach #2 (mask spanning items from density projection).** ❌ TRIED AND REVERTED 2026-06-02 — see the 2026-06-02 entry below. Masking items ≥70% region width from the V-cut made density cuts slice tables/wide layouts column-wise and regressed all three metrics. The legitimate column case is already handled by the gated `xy_find_column_cut` fallback. Do not retry without a guard that prevents the revealed valley from cutting through tabular content.> - **Approach #3 (split-at-big-gap inside `build_projected_lines`).** Even if XY-cut fails, the per-leaf y-banding step could split a band whose items have a horizontal gap ≥ ~6–8 × median glyph advance AND ≥ ~30pt absolute. This is what `pdftotext` does. Riskier — justified prose has wide inter-word gaps, especially near inline figures — so should be tightly gated and only as a last resort if #2 doesn't close the remainder.> - **`xy_root_bbox` negative-x handling.** Book-spread layouts where items have x<0 break bbox-relative reasoning everywhere downstream. A conditional "items extend past MediaBox" detector that widens the bbox only for those pages would unlock the rest of doc 029 and likely a handful of other book-interior PDFs.>@@ -165,6 +165,69 @@ Steps 1–8 = usable v1. Steps 9–11 = quality bumps for specific doc classes.>> **Bench plumbing landed:** `opendataloader-bench/src/pdf_parser_liteparse_placeholder.py` (image_mode=placeholder variant) and `pdf_parser_liteparse_ocr.py` (ocr_enabled=True variant), both registered in `engine_registry.py`. `liteparse-placeholder` confirms image placeholders cost ~0.003 NID — negligible vs the structural gains, so `Placeholder` is the right default.+> **2026-06-01 TOC suppression pass.** Bench analysis vs. nutrient (0.885 overall, lp 0.745) identified two MHS failure clusters: (a) **TOC entry over-promotion** — docs 16, 18, 113, 171 each emitted 4-20 false-positive `# Chapter X 41`-style headings when the doc was just a TOC where GT wants 1 `# Contents` heading; (b) docs that are missing a single section heading on 2-column layouts (38, 77, 189) — layout-related, deferred. Fix in `markdown_layout/headings.rs` + `classify.rs`:+> - **`toc_entry_arabic_number(text)`**: returns the trailing arabic page number when the line looks like a TOC entry — ≥8 alpha chars in the body, separator that includes whitespace (rejecting decimals like "94.2" or chart values "OCR-Recall 7 94.2"), 1-4 digit tail, body doesn't end in a digit (rejecting "vol 5 12" multi-number tails). The ≥8-alpha threshold protects real one-word headings like "Chapter 7" / "Section 4".+> - **`looks_like_toc_entry`**: wraps arabic + a separate Roman-numeral check (matches "Author's Note ... ix"). Used only for the page-level row floor count.+> - **`is_toc_title`**: matches normalized "contents" / "table of contents" / "index" / "list of figures" etc. Force-promoted to H1 on TOC pages even when the heading-map size filters would have dropped it (single-occurrence, short, etc.).+> - **`page_is_toc`**: page qualifies when ≥4 TOC-like lines AND ≥3 arabic-trailing entries whose page numbers form a ≥70% non-decreasing sequence. The monotonicity check is what separates real TOCs from chart/graph pages with random axis-value tails (doc 199 was the calibration target — was getting false-positive'd, now correctly rejected).+> - **Classifier wiring** (`classify.rs::classify_page_with_filters`): per-page `toc_page = page_is_toc(page)`. Per-line `toc_suppress = toc_page && !is_toc_title(text)` guards the size-based, numbered-bold-heading, and bold-heading promotion sites. A separate `toc_title_level = Some(1)` path force-promotes the TOC's own title even when size_level was None.+> - **Tests**: 4 new unit tests (`toc_entry_arabic_extracts_trailing_page_number`, `toc_entry_arabic_rejects_decimals_and_axis_labels`, `is_toc_title_matches_common_variants`, `page_is_toc_requires_monotonic_page_numbers`). 151 lib + 10 integration tests pass.+>+> **Bench result on opendataloader-bench (200 docs):**+>+> | Metric | Pre-fix | Post-fix | Δ |+> |---|---|---|---|+> | NID | 0.8822 | 0.8822 | ~0 |+> | TEDS | 0.3862 | 0.3862 | 0 (no table changes) |+> | MHS | 0.5146 | **0.5358** | **+0.0212** |+> | Overall | 0.7451 | **0.7573** | **+0.0122** |+>+> **Per-doc wins** (MHS): doc 16: 0.074→0.750, doc 18: 0.025→0.651, doc 113: ?→0.892, doc 171: 0.133→0.744, doc 51: regression-then-recovered 0.94, doc 153: 0.83.+>+> **Still wide-open after this pass** (rough TODO priority):+> - **TEDS** still 0.39 vs nutrient 0.71. Biggest gaps are docs 89, 47, 128, 182, 88, 90 — multi-line wrapped header rows where each column's header text spans multiple baselines at staggered positions; the borderless detector closes the run after 1 row because the first "row" comes back as 1 merged cell. Other failure mode: complex spreadsheet-like grids (docs 119, 120, 121, 130, 149, 180, 121) where ground truth uses `<table>` HTML with multi-line cells and our column detector slices the layout into 2-3 tiny tables. Both need either ML-grade cell detection or a much smarter borderless table merger that walks back to absorb headers above a detected body.+> - **MHS** — remaining MHS=0 docs are dominated by 2-column section headings being eaten by adjacent-column body text (docs 7, 13, 38, 77, 118, 148, 177, 178, 189, 198). Same root cause as the long-tracked `paper.pdf` page-1 banner issue: layout decomposition can't isolate single-line bold section heads that sit at the same y-band as content in the other column. Needs the wide-banner / figure-rect-aware XY-cut refinements.+> - **TEDS multi-line header absorption** is the highest-leverage table fix — would lift docs 47, 89, 182, 88, 90 from 0.1–0.4 to ~0.7+. Approach: when ruled-grid detection or borderless detection finds a body table, walk back N lines (where N matches the detected column count's track positions) and check if those upper lines have cells aligning to the same column tracks → absorb as header rows.++> **2026-06-02 ruled-grid single-column tables (`tables.rs`).** Targeted the largest TEDS=0 cluster (15 docs with GT tables we emitted 0 columns for). Single-col GT table was 1 doc (149), but the same code path also unblocked thin-border-strip docs whose true content column was wedged between two phantom rule-strip columns. Three coordinated changes in `build_ruled_table`:+>+> - **Threshold relaxation:** `xs.len() >= 3` → `xs.len() >= 2`. Allows the detector to produce 1-column tables when the geometry only gives a single content column between borders.+> - **Phantom-row collapse:** after cell assignment, drop rows whose height < 0.8 × median non-empty row height AND that have no text. Doc 149 draws each visual row as three stacked rects (top-border 1pt, body 22pt, bottom-strip 5pt) which all survive 2pt y-clustering as separate grid rows — without collapse, the empty-cell fraction kills the table. Threshold preserves real fill-in-the-blank forms (empty body rows match median non-empty height).+> - **Phantom-column collapse:** mirror of the row rule for columns. Drop empty columns whose width < 0.3 × median non-empty column width. Doc 149 has 5.5pt left/right border-strip columns next to a 305pt content column — both drop cleanly.+>+> One stricter rule in `merge_table_runs`: a 1-column ruled run **yields to** a multi-column borderless run that overlaps the same line range. Without this, doc 078's data table (drawn with top/bottom/header rules but no internal V-rules) gets detected as a degenerate 1-col grid that overrides the correct 7-col borderless result. Path geometry strictly beats text alignment for *multi*-col ruled runs, but a 1-col ruled run is ambiguous — could be a real single-column table or a multi-col table missing its V-rules — so we defer to the multi-col borderless evidence when present.+>+> **Bench result (200 docs):**+>+> | Metric | Pre | Post | Δ |+> |---|---|---|---|+> | NID | 0.8824 | 0.8819 | −0.0005 (noise) |+> | TEDS | 0.3906 | **0.4144** | **+0.0238** |+> | MHS | 0.5913 | 0.5928 | +0.0015 |+> | Overall | 0.7713 | **0.7738** | **+0.0025** |+>+> **Per-doc:** doc 149 0 → 1.0 TEDS (clean recovery — 8-row single-col ECO Circle competence stack). Several mid-TEDS docs (128, 170, 188) also improved a few hundredths each — likely because the `0.8 × median` row collapse now bridges thin-rule rows in their existing detections. No TEDS regressions ≥ 0.01.+>+> **Still wide open after this pass** — the bulk of the TEDS=0 cluster is **2-column tables** (7 docs: 121, 132, 146, 150, 165, 166, 197). The borderless detector explicitly requires `TABLE_MIN_COLUMNS = 3`, which kills every 2-col borderless table on the bench. Lowering it to 2 is high-leverage but risky: key:value bullet lists and 2-column body layouts will look the same. Needs guards before attempting. The ruled-grid path I just relaxed can pick up 2-col ruled tables (doc 146 has heavy graphics), but they don't appear to fire — likely because the connected-component step is splitting the grid, or because `find_grid_components` requires ≥2 H AND ≥2 V components to cross within 3pt and doc 146 has many tiny rect segments that don't cleanly cross. Worth dumping next pass.+>+> **2026-06-02 heading char-floor, TOC-title, header absorption; XY-cut spanning-mask tried and reverted.** Four targeted heuristic changes plus one reverted layout experiment. Net result on opendataloader-bench (200 docs):+>+> | Metric | Pre (260601) | Post | Δ |+> |---|---|---|---|+> | NID | 0.8822 | 0.8824 | +0.0002 |+> | TEDS | 0.3862 | **0.3906** | **+0.0044** |+> | MHS | 0.5358 | **0.5913** | **+0.0555** |+> | Overall | 0.7573 | **0.7713** | **+0.0140** |+>+> - **Item 1 — heading char-floor (`headings.rs`).** `MIN_HEADING_TOTAL_CHARS` 25→10 (now just a noise guard); added `MIN_HEADING_AVG_LINE_CHARS = 8.0` as the real legend-token discriminator. `build_heading_map` now keeps a size-outlier cluster iff `avg_line_chars` is in `[8, MAX]`. Rescues clean single-line headings like "Aligning with Your Identity" (24 chars/line) that the 25-char floor dropped, while still rejecting paper-legend tokens ("A-mem"/"Base" ≈ 4-5 chars/line). MHS contributor.+> - **Item 3 — TOC title without `page_is_toc` (`classify.rs`).** `toc_title_level` now fires on `is_toc_title(text)` alone (was gated on `toc_page`). `is_toc_title` matches the whole trimmed line against an exact whitelist (contents / table of contents / index / …) so it can't fire mid-prose. MHS contributor.+> - **Item 4 — multi-line header absorption (`tables.rs`).** After the borderless detector builds a body run, `absorb_header_lines` walks backward (bounded by `floor` = prior run's end) pulling in lines whose cells (2 ≤ n ≤ column_count) all align to the established column tracks and sit tight above the row below. Multiple wrapped header lines collapse into one markdown header row (joined per-track top-to-bottom). `try_detect_table` gained a `floor` param; `detect_tables` threads the last run end through it. Single-cell titles are explicitly NOT absorbed (≥2 aligned cells required). 2 new unit tests. **TEDS +0.0044, NID neutral** — a clean win. Per-doc: 128 +0.069, 170 +0.062, 188 +0.059.+> - **Item 5 — XY-cut spanning-item mask (REVERTED).** Implemented Approach #2 (mask items ≥70% region width from the V-cut density projection). Bench showed it **hurt all three metrics**: TEDS 0.3906→0.3614, NID 0.8824→0.8780, MHS 0.5913→0.5821. Root cause: masking wide items makes density V-cuts fire *inside* tables and wide layouts, slicing them column-wise (docs 082/187/189 lost both TEDS and NID together — the signature of a split table). The existing `xy_find_column_cut` fallback (gated to fire only when both density cuts return None) already covers the legitimate column case without the collateral damage. **Conclusion: the density V-cut must stay conservative; spanning-mask is the wrong lever. Do not retry without a tabular-content guard that prevents the revealed valley from cutting through a table.** Fully reverted (constant + mask loop removed from `xy_find_best_cut`).+>+> **Item 2 (sparse-form ruled-table empty-cell relaxation) remains deferred** — `TABLE_MAX_EMPTY_CELL_FRACTION` stays 0.30. Real sparse table (doc 180) and spurious decorative grid (doc 198 TOC) are both ~75% empty, so empty-fraction is not a valid discriminator; loosening it regressed TEDS −0.089.+>+> **Struct-tag question answered:** 0/200 bench docs carry PDF struct tags (verified via `survey_tags` example), so the struct-tree heading path is moot for this benchmark — all gains must come from text/geometry heuristics.+## Carried-forward itemsThings deferred from earlier steps that need to be revisited:diff --git a/crates/liteparse/src/markdown_layout/classify.rs b/crates/liteparse/src/markdown_layout/classify.rsindex 2de9c2d..fe0ad7a 100644--- a/crates/liteparse/src/markdown_layout/classify.rs+++ b/crates/liteparse/src/markdown_layout/classify.rs@@ -3,7 +3,8 @@ use crate::types::{OutlineTarget, ParsedPage, ProjectedLine};use super::blocks::{Block, paragraph_from_accum};use super::headings::{MAX_HEADING_LEVELS, heading_level_for, is_caption_line, looks_like_bold_heading,- looks_like_numbered_bold_heading, outline_heading_level, struct_heading_level,+ is_toc_title, looks_like_numbered_bold_heading, outline_heading_level, page_is_toc,+ struct_heading_level,};use super::hr::detect_horizontal_rules;use super::inline::{@@ -86,6 +87,14 @@ pub fn classify_page_with_filters(let debug = std::env::var("LITEPARSE_DEBUG_MD").is_ok();+ // TOC suppression: when ≥3 lines on this page look like TOC entries+ // (alpha body + trailing page-number), demote heading promotion so each+ // entry stays a paragraph instead of becoming a fake H1/H2. The TOC's+ // own title ("Contents", "Table of Contents") is shorter without a+ // trailing number — it falls through the size/bold heuristics with no+ // special handling needed.+ let toc_page = page_is_toc(page);+// Strip running header/footer lines up-front so they don't leak into// table detection (a repeating two-column footer would otherwise look// like a 2-row table) or paragraph grouping.@@ -338,13 +347,25 @@ pub fn classify_page_with_filters(// distinct (and slightly larger) font that lands them in the// font-size heading map. Suppress font-size promotion for them;// outline / struct-tree signals still win since those are explicit.- let size_level = if is_caption_line(text) {+ let toc_suppress = toc_page && !is_toc_title(text);+ let size_level = if is_caption_line(text) || toc_suppress {None} else {heading_level_for(line.dominant_font_size, heading_map)};+ // A standalone "Contents" / "Table of Contents" / "Index" line is+ // almost always a real H1, even when `page_is_toc` is false — many+ // TOCs list entries without inline trailing page numbers, so the+ // page-level detector misses them. `is_toc_title` matches the *entire*+ // trimmed line against an exact whitelist, so it can't fire mid-prose.+ let toc_title_level = if is_toc_title(text) {+ Some(1u8)+ } else {+ None+ };let level = outline_level.or(size_level)+ .or(toc_title_level).map(|l| l.clamp(1, MAX_HEADING_LEVELS as u8));if let Some(level) = level {flush_paragraph(&mut blocks, paragraph.take());@@ -368,6 +389,7 @@ pub fn classify_page_with_filters(// headings in technical/legal/scientific PDFs silently emit as// ordered list items and lose all heading structure.if ordered+ && !toc_suppress&& looks_like_numbered_bold_heading(line,rest,@@ -438,7 +460,7 @@ pub fn classify_page_with_filters(.map(|p| &p.last).or(last_list_line.as_ref());let next_for_gap = lines.get(idx);- if looks_like_bold_heading(line, prev_for_gap, next_for_gap) {+ if !toc_suppress && looks_like_bold_heading(line, prev_for_gap, next_for_gap) {flush_paragraph(&mut blocks, paragraph.take());list_base_indent = None;last_list_item_idx = None;diff --git a/crates/liteparse/src/markdown_layout/headings.rs b/crates/liteparse/src/markdown_layout/headings.rsindex 3421d5a..003bf25 100644--- a/crates/liteparse/src/markdown_layout/headings.rs+++ b/crates/liteparse/src/markdown_layout/headings.rs@@ -57,6 +57,168 @@ pub(super) fn is_caption_line(text: &str) -> bool {false}+/// Trailing page-number extractor for TOC-entry shaped lines. Returns the+/// numeric value of the trailing page number when the line looks like a TOC+/// entry: alphabetic body, separator that *includes whitespace* (to reject+/// decimals like "94.2"), then a trailing arabic 1–4 digit number. Roman+/// numerals are accepted in `looks_like_toc_entry` but not returned for the+/// monotonic-sequence check.+pub(super) fn toc_entry_arabic_number(text: &str) -> Option<i32> {+ let s = text.trim();+ if s.is_empty() {+ return None;+ }+ let chars: Vec<char> = s.chars().collect();+ let n = chars.len();+ let mut tail_start = n;+ while tail_start > 0 && chars[tail_start - 1].is_ascii_digit() {+ tail_start -= 1;+ }+ let tail_len = n - tail_start;+ if tail_len == 0 || tail_len > 4 {+ return None;+ }+ // Separator: ≥1 whitespace + optional '.' leaders. The mandatory+ // whitespace rules out decimals ("94.2") and inline number suffixes+ // ("Recall3 7 94.2") that aren't TOC entries.+ let mut sep_end = tail_start;+ let mut saw_ws = false;+ while sep_end > 0 {+ let c = chars[sep_end - 1];+ if c.is_whitespace() {+ sep_end -= 1;+ saw_ws = true;+ } else if c == '.' {+ sep_end -= 1;+ } else {+ break;+ }+ }+ if !saw_ws {+ return None;+ }+ // Body must (a) carry meaningful alpha content and (b) NOT end with a+ // digit — that would mean we sliced a multi-part number ("vol 5 12").+ let body = &chars[..sep_end];+ let alpha = body.iter().filter(|c| c.is_alphabetic()).count();+ // Require ≥8 alpha chars in the body. This keeps real one-word headings+ // like "Chapter 7" / "Section 4" out of the TOC bucket while still+ // accepting typical TOC entries ("Introduction 7", "Conclusion 127", ...).+ if alpha < 8 {+ return None;+ }+ if body+ .iter()+ .rev()+ .find(|c| !c.is_whitespace())+ .is_some_and(|c| c.is_ascii_digit())+ {+ return None;+ }+ let tail: String = chars[tail_start..].iter().collect();+ tail.parse::<i32>().ok()+}++/// Roman-or-arabic TOC-entry detector — used by tests and the TOC-page+/// detector to count TOC-shaped lines that the strict arabic extractor+/// above misses (e.g. "Author's Note ... ix").+pub(super) fn looks_like_toc_entry(text: &str) -> bool {+ if toc_entry_arabic_number(text).is_some() {+ return true;+ }+ // Try Roman numerals at the tail.+ let s = text.trim();+ let chars: Vec<char> = s.chars().collect();+ let n = chars.len();+ let mut tail_start = n;+ while tail_start > 0+ && matches!(+ chars[tail_start - 1],+ 'i' | 'v' | 'x' | 'l' | 'c' | 'd' | 'm' | 'I' | 'V' | 'X' | 'L' | 'C' | 'D' | 'M'+ )+ {+ tail_start -= 1;+ }+ let tail_len = n - tail_start;+ if tail_len < 2 || tail_len > 6 {+ return false;+ }+ let mut sep_end = tail_start;+ let mut saw_ws = false;+ while sep_end > 0 {+ let c = chars[sep_end - 1];+ if c.is_whitespace() {+ sep_end -= 1;+ saw_ws = true;+ } else if c == '.' {+ sep_end -= 1;+ } else {+ break;+ }+ }+ if !saw_ws {+ return false;+ }+ let alpha = chars[..sep_end]+ .iter()+ .filter(|c| c.is_alphabetic())+ .count();+ alpha >= 5+}++/// Returns true if `text` is the canonical title of a TOC page: "Contents",+/// "Table of Contents", "Index", etc. We use this to let the TOC's own title+/// through the heading promotion that's otherwise suppressed on TOC pages.+pub(super) fn is_toc_title(text: &str) -> bool {+ let t = text.trim().trim_end_matches(':').to_ascii_lowercase();+ matches!(+ t.as_str(),+ "contents"+ | "table of contents"+ | "table of content"+ | "index"+ | "list of figures"+ | "list of tables"+ | "table of figures"+ | "toc"+ )+}++/// TOC-page detection. A page is a TOC iff it carries ≥4 arabic-trailing+/// TOC-entry lines whose page numbers form a *mostly non-decreasing*+/// sequence (≥70% of adjacent pairs satisfy `next >= prev`). The+/// monotonicity check is what separates real TOCs from chart/graph pages+/// with random axis-value tails. Lines whose tail isn't arabic but matches+/// the looser `looks_like_toc_entry` (e.g. roman numerals "ix", "xi") still+/// count toward the row floor — they just don't participate in the+/// monotonicity check.+pub(super) fn page_is_toc(page: &ParsedPage) -> bool {+ let mut nums: Vec<i32> = Vec::new();+ let mut total_toc_like = 0usize;+ for line in &page.projected_lines {+ if is_rotated_line(line) {+ continue;+ }+ if let Some(n) = toc_entry_arabic_number(&line.text) {+ nums.push(n);+ total_toc_like += 1;+ } else if looks_like_toc_entry(&line.text) {+ total_toc_like += 1;+ }+ }+ if total_toc_like < 4 || nums.len() < 3 {+ return false;+ }+ let mut nondec = 0usize;+ for w in nums.windows(2) {+ if w[1] >= w[0] {+ nondec += 1;+ }+ }+ let frac = nondec as f32 / (nums.len() - 1) as f32;+ frac >= 0.7+}+/// Returns true if `line` looks like a section heading rendered in body-size/// bold text (a very common style for academic / technical PDFs where every/// "real" heading uses the same font size as body, distinguished only by@@ -255,14 +417,20 @@ pub fn compute_body_size(pages: &[ParsedPage]) -> f32 {}/// Minimum total non-whitespace characters across all occurrences at a font-/// size for it to qualify as a heading level. Calibrated against `paper.pdf`:-/// the 30pt chart-legend tokens ("A-mem"×2 + "Base"×2 = 18-20 chars) need to-/// fail this filter, while the 14.35pt title ("A-MEM: Agentic Memory for LLM-/// Agents" = 31 chars on a single line) needs to pass. 25 is the gap. Smaller-/// single-word headings like a lone "Summary" on a short doc still survive-/// because they share their font size with other (larger) headings in the-/// histogram entry.-const MIN_HEADING_TOTAL_CHARS: usize = 25;+/// size for it to qualify as a heading level. Low floor — just a noise guard+/// against 1-2 char artifacts. The real legend-token discriminator is+/// `MIN_HEADING_AVG_LINE_CHARS` below.+const MIN_HEADING_TOTAL_CHARS: usize = 10;++/// Minimum average non-whitespace characters *per line* at a font size for it+/// to qualify as a heading. This is what separates a real heading (a coherent+/// line like "Annual Events" = 12 chars/line, or "A-MEM: Agentic Memory for+/// LLM Agents" = 31) from scattered chart-legend tokens at a one-off display+/// size (paper.pdf's "A-mem"/"Base" ~ 4-5 chars/line). A per-line average is+/// robust where a total-char floor was not: a single short-but-real heading+/// (14pt "Aligning with Your Identity" = 24 non-ws chars, over 10.5pt body)+/// clears it, while many tiny repeated tokens do not no matter how many.+const MIN_HEADING_AVG_LINE_CHARS: f32 = 8.0;/// Maximum average characters per line for a size to qualify as a heading./// A "size larger than body" with very long lines is almost always a@@ -325,8 +493,10 @@ pub fn build_heading_map(pages: &[ParsedPage], body_size: f32) -> Vec<(f32, u8)>} else {(*alpha as f32) / (*chars as f32)};+ let avg_line_chars = *chars as f32 / (*lines).max(1) as f32;*chars >= MIN_HEADING_TOTAL_CHARS- && (*chars as f32 / (*lines).max(1) as f32) <= MAX_HEADING_AVG_LINE_CHARS+ && avg_line_chars >= MIN_HEADING_AVG_LINE_CHARS+ && avg_line_chars <= MAX_HEADING_AVG_LINE_CHARS&& alpha_ratio >= MIN_HEADING_ALPHA_RATIO}).map(|(s, _, _, _)| *s)@@ -466,6 +636,61 @@ mod tests {use super::super::test_helpers::{line, page};use super::*;+ #[test]+ fn toc_entry_arabic_extracts_trailing_page_number() {+ assert_eq!(toc_entry_arabic_number("Introduction 7"), Some(7));+ assert_eq!(+ toc_entry_arabic_number("1. A Fountain in the Square 1"),+ Some(1)+ );+ assert_eq!(+ toc_entry_arabic_number("6. For the Love of Iran . . . 41"),+ Some(41)+ );+ }++ #[test]+ fn toc_entry_arabic_rejects_decimals_and_axis_labels() {+ // "94.2" → decimal, not a TOC entry+ assert_eq!(toc_entry_arabic_number("OCR-Recall3 7 94.2"), None);+ // Chapter 7 — too short an alpha body to be a TOC entry+ assert_eq!(toc_entry_arabic_number("Chapter 7"), None);+ // No separator+ assert_eq!(toc_entry_arabic_number("Section1"), None);+ }++ #[test]+ fn is_toc_title_matches_common_variants() {+ assert!(is_toc_title("Contents"));+ assert!(is_toc_title("Table of Contents"));+ assert!(is_toc_title("table of contents"));+ assert!(is_toc_title("Index"));+ assert!(!is_toc_title("Introduction"));+ }++ #[test]+ fn page_is_toc_requires_monotonic_page_numbers() {+ // Real TOC: monotonically increasing page numbers.+ let pages_toc = page(vec![+ line("Table of contents", 50.0, 30.0, 18.0, 18.0),+ line("Introduction 7", 50.0, 60.0, 12.0, 12.0),+ line("Part I: New Children 21", 50.0, 72.0, 12.0, 12.0),+ line("Part II: From Solitary 45", 50.0, 84.0, 12.0, 12.0),+ line("Part III: Commercial 71", 50.0, 96.0, 12.0, 12.0),+ line("Conclusion 127", 50.0, 108.0, 12.0, 12.0),+ ]);+ assert!(page_is_toc(&pages_toc));++ // Chart page: trailing numbers but NOT monotonic and many fail the+ // separator/alpha rules anyway.+ let pages_chart = page(vec![+ line("OCR-Recall is 94.2", 50.0, 60.0, 9.0, 9.0),+ line("Precision rate 89.0", 50.0, 72.0, 9.0, 9.0),+ line("F1 score 80.4", 50.0, 84.0, 9.0, 9.0),+ ]);+ assert!(!page_is_toc(&pages_chart));+ }+#[test]fn body_size_picks_most_common() {let pages = vec![page(vec![diff --git a/crates/liteparse/src/markdown_layout/tables.rs b/crates/liteparse/src/markdown_layout/tables.rsindex 102270b..8506974 100644--- a/crates/liteparse/src/markdown_layout/tables.rs+++ b/crates/liteparse/src/markdown_layout/tables.rs@@ -281,7 +281,7 @@ fn row_spacing_cv(rows: &[(usize, &ProjectedLine, Vec<TableCell>)]) -> f32 {/// a `TableRun` with `Block::Table` or `Block::GridFallback`; on failure/// returns `None` (and the caller should fall through to per-line/// classification).-fn try_detect_table(lines: &[ProjectedLine], start_idx: usize) -> Option<TableRun> {+fn try_detect_table(lines: &[ProjectedLine], start_idx: usize, floor: usize) -> Option<TableRun> {let first_cells = split_cells(&lines[start_idx]);if first_cells.len() < TABLE_MIN_COLUMNS {return None;@@ -383,16 +383,30 @@ fn try_detect_table(lines: &[ProjectedLine], start_idx: usize) -> Option<TableRu});}- // Promote the first row to header iff every cell in it is bold (matches- // pymupdf4llm's "bold-or-filled" heuristic; fills require fork data).+ // Walk back above the detected body and absorb header lines that align to+ // the same column tracks but weren't includable as body rows (merged /+ // partial header cells). Multiple wrapped header lines collapse into one+ // markdown header row, joined per-column top-to-bottom.+ let absorbed = absorb_header_lines(lines, start_idx, &tracks, column_count, floor);++ // Promote the first body row to header iff every cell in it is bold+ // (matches pymupdf4llm's "bold-or-filled" heuristic; fills require fork+ // data). Skipped when we already absorbed an explicit header above.let first_row = &rows[0].2;- let header_qualifies = first_row.iter().all(|c| c.bold);- let header = if header_qualifies {- Some(first_row.iter().map(|c| c.text.clone()).collect())- } else {- None+ let bold_header_qualifies = absorbed.is_none() && first_row.iter().all(|c| c.bold);++ // `row_start` is the index of the first body row within `rows`. When the+ // header came from absorbed lines above, every detected row is body data;+ // only the bold-first-row promotion consumes rows[0].+ let (run_start, header, row_start) = match absorbed {+ Some((hstart, header_texts)) => (hstart, Some(header_texts), 0),+ None if bold_header_qualifies => (+ start_idx,+ Some(first_row.iter().map(|c| c.text.clone()).collect()),+ 1,+ ),+ None => (start_idx, None, 0),};- let row_start = if header.is_some() { 1 } else { 0 };let body_rows: Vec<Vec<String>> = rows[row_start..].iter().map(|(_, _, cells)| cells.iter().map(|c| c.text.clone()).collect())@@ -402,7 +416,7 @@ fn try_detect_table(lines: &[ProjectedLine], start_idx: usize) -> Option<TableRu}Some(TableRun {- start: start_idx,+ start: run_start,end,block: Block::Table {header,@@ -411,14 +425,81 @@ fn try_detect_table(lines: &[ProjectedLine], start_idx: usize) -> Option<TableRu})}+/// Walk backward from `start_idx` (not below `floor`), pulling in lines whose+/// cells all align to the table's `tracks` as header rows. Returns the new+/// start index and a single merged header row (`column_count` columns) with+/// each absorbed line's text appended into its nearest column track.+fn absorb_header_lines(+ lines: &[ProjectedLine],+ start_idx: usize,+ tracks: &[f32],+ column_count: usize,+ floor: usize,+) -> Option<(usize, Vec<String>)> {+ let mut absorbed: Vec<Vec<TableCell>> = Vec::new();+ let mut j = start_idx;+ while j > floor {+ let cand = j - 1;+ let cells = split_cells(&lines[cand]);+ // A header line must carry at least two track-aligned cells (a single+ // cell is a title/caption, not a header), no more than column_count,+ // sit tight above the row below it, and have every cell land on a+ // known column track.+ if cells.len() < 2 || cells.len() > column_count {+ break;+ }+ if !table_rows_adjacent(&lines[cand], &lines[j]) {+ break;+ }+ let all_align = cells.iter().all(|c| {+ tracks+ .iter()+ .any(|t| (c.start_x - *t).abs() <= TABLE_TRACK_TOLERANCE_PT)+ });+ if !all_align {+ break;+ }+ absorbed.push(cells);+ j = cand;+ }+ if absorbed.is_empty() {+ return None;+ }+ // Collected bottom-up; reverse so text reads top-to-bottom per column.+ absorbed.reverse();+ let mut header = vec![String::new(); column_count];+ for cells in &absorbed {+ for c in cells {+ let Some(idx) = tracks+ .iter()+ .enumerate()+ .filter(|(_, t)| (c.start_x - **t).abs() <= TABLE_TRACK_TOLERANCE_PT)+ .min_by(|(_, a), (_, b)| {+ (c.start_x - **a).abs().total_cmp(&(c.start_x - **b).abs())+ })+ .map(|(i, _)| i)+ else {+ continue;+ };+ if !header[idx].is_empty() && !c.text.is_empty() {+ header[idx].push(' ');+ }+ header[idx].push_str(&c.text);+ }+ }+ Some((j, header))+}+/// Scan `lines` once and return all detected tabular regions (sorted by/// `start`). Caller uses these as cut-points so the per-line classifier never/// sees lines inside a table.pub(super) fn detect_tables(lines: &[ProjectedLine]) -> Vec<TableRun> {let mut out = Vec::new();let mut i = 0;+ let mut floor = 0;while i < lines.len() {- if let Some(run) = try_detect_table(lines, i) {+ if let Some(run) = try_detect_table(lines, i, floor) {+ floor = run.end;i = run.end;out.push(run);} else {@@ -474,6 +555,11 @@ const TABLE_GRID_CLUSTER_PT: f32 = 2.0;const TABLE_CROSS_TOLERANCE_PT: f32 = 3.0;/// Reject ruled-table candidates whose empty-cell fraction exceeds this.+/// NOTE: this can't be loosened to recover blank worksheets/forms — a real+/// sparse table (doc 180, a 4-col Version History, ~75% empty) and a spurious+/// grid from decorative layout boxes (doc 198, a TOC, also ~75% empty) are+/// indistinguishable on empty-fraction, and relaxing it net-regressed TEDS by+/// ~0.09 on the bench (more false tables than real forms recovered).const TABLE_MAX_EMPTY_CELL_FRACTION: f32 = 0.30;/// Reject candidates whose grid covers nearly the whole page — almost always@@ -672,8 +758,12 @@ fn build_ruled_table(xs.sort_by(|a, b| a.total_cmp(b));dedup_close(&mut xs, TABLE_GRID_CLUSTER_PT);- // Need ≥2 cells per axis → ≥3 boundary lines.- if ys.len() < 3 || xs.len() < 3 {+ // Need ≥2 row boundaries (1 row) and ≥2 column boundaries (1 col); but+ // a 1×1 grid is just a callout box, so also require ≥1 inner divider+ // (i.e. ys.len() ≥ 3 for ≥2 rows). Single-column tables (`xs.len() == 2`)+ // are accepted when row evidence is strong enough — extra guards apply+ // below after the empty-row collapse.+ if ys.len() < 3 || xs.len() < 2 {return None;}@@ -733,6 +823,122 @@ fn build_ruled_table(return None;}+ // Collapse "phantom rows" produced by stacked thin border-strip rects+ // (doc 149 draws each visual table row as: top border strip ~1pt, body+ // rect ~22pt, bottom border strip ~5pt — each contributes y-coords that+ // survive the 2pt clustering as separate grid rows). Rule: drop a row+ // iff (a) it has no text in any cell AND (b) its height is < 50% of the+ // median non-empty row height. The height gate preserves real+ // fill-in-the-blank forms where empty body rows are full-height.+ let row_heights: Vec<f32> = (0..n_rows).map(|r| ys[r + 1] - ys[r]).collect();+ let nonempty_heights: Vec<f32> = (0..n_rows)+ .filter(|r| cell_has_text[*r].iter().any(|t| *t))+ .map(|r| row_heights[r])+ .collect();+ let median_h = if !nonempty_heights.is_empty() {+ let mut s = nonempty_heights.clone();+ s.sort_by(|a, b| a.total_cmp(b));+ s[s.len() / 2]+ } else {+ let mut s = row_heights.clone();+ s.sort_by(|a, b| a.total_cmp(b));+ s[s.len() / 2]+ };+ let keep: Vec<bool> = (0..n_rows)+ .map(|r| {+ let has_text = cell_has_text[r].iter().any(|t| *t);+ has_text || row_heights[r] >= median_h * 0.8+ })+ .collect();+ let cells: Vec<Vec<String>> = (0..n_rows)+ .filter(|r| keep[*r])+ .map(|r| cells[r].clone())+ .collect();+ let cell_has_text: Vec<Vec<bool>> = (0..n_rows)+ .filter(|r| keep[*r])+ .map(|r| cell_has_text[r].clone())+ .collect();+ let cell_is_bold: Vec<Vec<bool>> = (0..n_rows)+ .filter(|r| keep[*r])+ .map(|r| cell_is_bold[r].clone())+ .collect();+ let n_rows = cells.len();+ if n_rows < 2 {+ return None;+ }++ // Mirror the row-collapse rule on columns. Some ruled tables (doc 149)+ // draw their left/right borders as thin strip rects 5pt wide — those+ // become phantom columns with no text. Drop columns that are both empty+ // AND noticeably narrower than the median text-bearing column.+ let col_widths: Vec<f32> = (0..n_cols).map(|c| xs[c + 1] - xs[c]).collect();+ let nonempty_col_widths: Vec<f32> = (0..n_cols)+ .filter(|c| (0..n_rows).any(|r| cell_has_text[r][*c]))+ .map(|c| col_widths[c])+ .collect();+ let median_w = if !nonempty_col_widths.is_empty() {+ let mut s = nonempty_col_widths.clone();+ s.sort_by(|a, b| a.total_cmp(b));+ s[s.len() / 2]+ } else {+ let mut s = col_widths.clone();+ s.sort_by(|a, b| a.total_cmp(b));+ s[s.len() / 2]+ };+ let keep_col: Vec<bool> = (0..n_cols)+ .map(|c| {+ let has_text = (0..n_rows).any(|r| cell_has_text[r][c]);+ has_text || col_widths[c] >= median_w * 0.3+ })+ .collect();+ // Cap how aggressively we drop columns. A real table with one phantom+ // border-strip column (doc 149: 5pt-wide left border) drops exactly one.+ // Anything more than that is almost certainly a chart whose vertical+ // grid-lines got merged with text data (doc 078: a chart's 18 V-lines+ // straddle a table to its left/right and collapse to 1 col), so the+ // "table" is bogus — bail and let the borderless detector handle it.+ // Note: keep_col already only drops columns where both (a) no text AND+ // (b) width < 30% of median text-bearing column.+ let cells: Vec<Vec<String>> = cells+ .into_iter()+ .map(|row| {+ row.into_iter()+ .enumerate()+ .filter(|(c, _)| keep_col[*c])+ .map(|(_, v)| v)+ .collect()+ })+ .collect();+ let cell_has_text: Vec<Vec<bool>> = cell_has_text+ .into_iter()+ .map(|row| {+ row.into_iter()+ .enumerate()+ .filter(|(c, _)| keep_col[*c])+ .map(|(_, v)| v)+ .collect()+ })+ .collect();+ let cell_is_bold: Vec<Vec<bool>> = cell_is_bold+ .into_iter()+ .map(|row| {+ row.into_iter()+ .enumerate()+ .filter(|(c, _)| keep_col[*c])+ .map(|(_, v)| v)+ .collect()+ })+ .collect();+ let n_cols = cells.first().map(|r| r.len()).unwrap_or(0);+ if n_cols == 0 {+ return None;+ }+ // Single-column tables are ambiguous (could be a captioned card) — require+ // ≥3 rows of geometric + textual evidence.+ if n_cols == 1 && n_rows < 3 {+ return None;+ }+let total = n_rows * n_cols;let empty_count = cell_has_text.iter()@@ -837,16 +1043,39 @@ pub(super) fn detect_ruled_tables(/// ranges overlap the ruled run wins (path-based geometry is strictly stronger/// than text-alignment heuristics) — overlapping borderless runs are dropped.pub(super) fn merge_table_runs(mut ruled: Vec<TableRun>, borderless: Vec<TableRun>) -> Vec<TableRun> {+ // A ruled run normally beats an overlapping borderless run (path geometry+ // is a stronger signal than text alignment). But a 1-column ruled run is+ // ambiguous — it can come from a real single-column table (doc 149's+ // stacked cards) OR from a multi-column table whose vertical separators+ // are implicit and only the top/bottom rules were drawn (doc 078). Yield+ // to a multi-column borderless run that covers the same range.+ let mut kept: Vec<TableRun> = Vec::with_capacity(ruled.len());+ for r in ruled.drain(..) {+ let is_one_col = matches!(&r.block, Block::Table { rows, .. } if rows.first().map(|row| row.len()).unwrap_or(0) <= 1);+ if is_one_col {+ let beaten = borderless.iter().any(|b| {+ let overlaps = !(b.end <= r.start || b.start >= r.end);+ if !overlaps {+ return false;+ }+ matches!(&b.block, Block::Table { rows, .. } if rows.first().map(|row| row.len()).unwrap_or(0) >= 2)+ });+ if beaten {+ continue;+ }+ }+ kept.push(r);+ }for b in borderless {- let overlaps = ruled+ let overlaps = kept.iter().any(|r| !(b.end <= r.start || b.start >= r.end));if !overlaps {- ruled.push(b);+ kept.push(b);}}- ruled.sort_by_key(|r| r.start);- ruled+ kept.sort_by_key(|r| r.start);+ kept}/// Escape `|` and `\n` inside a markdown table cell so the pipe-table grammar@@ -951,6 +1180,49 @@ mod tests {assert_eq!(cells[0].text, "Hello world");}+ #[test]+ fn absorbs_partial_header_line_above_body() {+ // A header line with only two track-aligned cells sits above a clean+ // 3-column body. It can't start the table on its own (fewer than+ // TABLE_MIN_COLUMNS cells) but should be walked back in as the header.+ let lines = vec![+ line_with_spans(&[("Name", 50.0), ("Scores", 150.0)], 100.0, 10.0),+ line_with_spans(&[("A", 50.0), ("1", 150.0), ("2", 250.0)], 115.0, 10.0),+ line_with_spans(&[("B", 50.0), ("3", 150.0), ("4", 250.0)], 130.0, 10.0),+ line_with_spans(&[("C", 50.0), ("5", 150.0), ("6", 250.0)], 145.0, 10.0),+ ];+ let runs = detect_tables(&lines);+ assert_eq!(runs.len(), 1);+ let run = &runs[0];+ assert_eq!(run.start, 0, "header line should be absorbed into the run");+ assert_eq!(run.end, 4);+ match &run.block {+ Block::Table { header, rows } => {+ let header = header.as_ref().expect("header should be present");+ assert_eq!(header, &vec!["Name".to_string(), "Scores".to_string(), String::new()]);+ // All three body rows survive — the header came from above, so+ // rows[0] is not consumed as a header.+ assert_eq!(rows.len(), 3);+ }+ other => panic!("expected Block::Table, got {other:?}"),+ }+ }++ #[test]+ fn does_not_absorb_single_cell_title_above_body() {+ // A one-cell title/caption above a table is NOT a header row and must+ // not be absorbed.+ let lines = vec![+ line_with_spans(&[("Results", 50.0)], 100.0, 10.0),+ line_with_spans(&[("A", 50.0), ("1", 150.0), ("2", 250.0)], 115.0, 10.0),+ line_with_spans(&[("B", 50.0), ("3", 150.0), ("4", 250.0)], 130.0, 10.0),+ line_with_spans(&[("C", 50.0), ("5", 150.0), ("6", 250.0)], 145.0, 10.0),+ ];+ let runs = detect_tables(&lines);+ assert_eq!(runs.len(), 1);+ assert_eq!(runs[0].start, 1, "single-cell title must stay out of the run");+ }+#[test]fn rejects_table_when_row_count_too_low() {let lines = vec![line_with_spans(diff --git a/crates/pdfium/src/struct_tree.rs b/crates/pdfium/src/struct_tree.rsnew file mode 100644index 0000000..ce5f1fb--- /dev/null+++ b/crates/pdfium/src/struct_tree.rs@@ -0,0 +1,224 @@+//! Walk the PDF structure tree (tagged-PDF tree) for a page.+//!+//! Each struct element has:+//! - a role (string: "H1", "Figure", "Table", ...)+//! - zero or more marked content ids (mcids) that tie the element to+//! page-content stream blocks+//! - zero or more child elements+//!+//! For LiteParse's purposes we want a flat list of nodes with:+//! - the role string+//! - viewport-space bbox derived from joining page objects whose+//! `FPDFPageObj_GetMarkedContentID` matches one of this node's mcids+//! - the mcid list itself (so the layout pass can map text runs to nodes)++use crate::ffi;+use crate::page::{Page, ViewportTransform};+use crate::types::RectF;++/// One node in the structure tree, flattened for downstream use.+#[derive(Debug, Clone)]+pub struct StructNode {+ /// Role string from `FPDF_StructElement_GetType` (e.g. "H1", "P", "Figure").+ pub role: String,+ /// Marked content ids attached to this element (and its non-struct child markers).+ pub mcids: Vec<i32>,+ /// Union bbox of page objects tagged with any of `mcids`, in viewport+ /// coordinates (top-left origin, 72 DPI). `None` when none of the mcids+ /// resolved to a bbox on the page.+ pub bbox: Option<RectF>,+ /// Optional alt text (set for Figure / Formula elements when present).+ pub alt_text: Option<String>,+}++impl Page<'_> {+ /// Walk this page's structure tree (tagged-PDF tree). Returns an empty+ /// vec when the page is untagged or the document has no struct tree.+ /// Nodes are returned in pre-order (parent before children).+ pub fn struct_tree(&self, view_box: &RectF) -> Vec<StructNode> {+ let tree = unsafe { ffi!(FPDF_StructTree_GetForPage(self.handle)) };+ if tree.is_null() {+ return Vec::new();+ }++ let mcid_bboxes = collect_mcid_bboxes(self, view_box);+ let mut out = Vec::new();++ let count = unsafe { ffi!(FPDF_StructTree_CountChildren(tree)) };+ for i in 0..count {+ let elem = unsafe { ffi!(FPDF_StructTree_GetChildAtIndex(tree, i)) };+ if !elem.is_null() {+ walk_element(elem, &mcid_bboxes, &mut out);+ }+ }++ unsafe { ffi!(FPDF_StructTree_Close(tree)) };++ out+ }+}++/// Pre-scan all page objects on the page, building `mcid → union(bbox)` in+/// viewport space. Each struct node then unions the bboxes for its own mcids.+fn collect_mcid_bboxes(+ page: &Page<'_>,+ view_box: &RectF,+) -> std::collections::HashMap<i32, RectF> {+ let vp = page.viewport_transform(view_box);+ let obj_count = unsafe { ffi!(FPDFPage_CountObjects(page.handle)) };+ let mut map: std::collections::HashMap<i32, RectF> = std::collections::HashMap::new();++ for i in 0..obj_count {+ let obj = unsafe { ffi!(FPDFPage_GetObject(page.handle, i)) };+ if obj.is_null() {+ continue;+ }+ let mcid = unsafe { ffi!(FPDFPageObj_GetMarkedContentID(obj)) };+ if mcid < 0 {+ continue;+ }+ let bbox = page_object_bbox(obj, &vp);+ if let Some(b) = bbox {+ map.entry(mcid)+ .and_modify(|cur| *cur = union_rect(cur, &b))+ .or_insert(b);+ }+ }++ map+}++fn page_object_bbox(obj: pdfium_sys::FPDF_PAGEOBJECT, vp: &ViewportTransform) -> Option<RectF> {+ let mut left = 0.0f32;+ let mut bottom = 0.0f32;+ let mut right = 0.0f32;+ let mut top = 0.0f32;+ let ok = unsafe {+ ffi!(FPDFPageObj_GetBounds(+ obj,+ &mut left,+ &mut bottom,+ &mut right,+ &mut top+ ))+ };+ if ok == 0 {+ return None;+ }+ Some(vp.transform_bounds(&RectF {+ left,+ top,+ right,+ bottom,+ }))+}++fn union_rect(a: &RectF, b: &RectF) -> RectF {+ RectF {+ left: a.left.min(b.left),+ top: a.top.min(b.top),+ right: a.right.max(b.right),+ bottom: a.bottom.max(b.bottom),+ }+}++fn walk_element(+ elem: pdfium_sys::FPDF_STRUCTELEMENT,+ mcid_bboxes: &std::collections::HashMap<i32, RectF>,+ out: &mut Vec<StructNode>,+) {+ let role = read_element_type(elem);+ let alt_text = read_alt_text(elem);++ // Collect mcids: the multi-mcid getters + per-child marked-content-only children.+ let mut mcids: Vec<i32> = Vec::new();+ let n_mcids = unsafe { ffi!(FPDF_StructElement_GetMarkedContentIdCount(elem)) };+ for i in 0..n_mcids {+ let m = unsafe { ffi!(FPDF_StructElement_GetMarkedContentIdAtIndex(elem, i)) };+ if m >= 0 {+ mcids.push(m);+ }+ }+ // Also legacy: single direct mcid getter (older tag-trees expose it here).+ let single = unsafe { ffi!(FPDF_StructElement_GetMarkedContentID(elem)) };+ if single >= 0 && !mcids.contains(&single) {+ mcids.push(single);+ }++ let n_children = unsafe { ffi!(FPDF_StructElement_CountChildren(elem)) };+ for i in 0..n_children {+ // Non-struct children expose their mcid via GetChildMarkedContentID+ // (returns -1 when the child is itself a struct element).+ let child_mcid = unsafe { ffi!(FPDF_StructElement_GetChildMarkedContentID(elem, i)) };+ if child_mcid >= 0 && !mcids.contains(&child_mcid) {+ mcids.push(child_mcid);+ }+ }++ let bbox = union_mcid_bboxes(&mcids, mcid_bboxes);+ out.push(StructNode {+ role,+ mcids,+ bbox,+ alt_text,+ });++ for i in 0..n_children {+ let child_elem = unsafe { ffi!(FPDF_StructElement_GetChildAtIndex(elem, i)) };+ if !child_elem.is_null() {+ walk_element(child_elem, mcid_bboxes, out);+ }+ }+}++fn union_mcid_bboxes(+ mcids: &[i32],+ mcid_bboxes: &std::collections::HashMap<i32, RectF>,+) -> Option<RectF> {+ let mut acc: Option<RectF> = None;+ for m in mcids {+ if let Some(b) = mcid_bboxes.get(m) {+ acc = Some(match acc {+ Some(a) => union_rect(&a, b),+ None => *b,+ });+ }+ }+ acc+}++fn read_element_type(elem: pdfium_sys::FPDF_STRUCTELEMENT) -> String {+ read_widestring(|buf, len| unsafe { ffi!(FPDF_StructElement_GetType(elem, buf, len)) })+}++fn read_alt_text(elem: pdfium_sys::FPDF_STRUCTELEMENT) -> Option<String> {+ let s = read_widestring(|buf, len| unsafe { ffi!(FPDF_StructElement_GetAltText(elem, buf, len)) });+ if s.is_empty() { None } else { Some(s) }+}++/// Read a PDFium UTF-16LE widestring out-param via the "call once for size,+/// allocate, call again" pattern. `getter` is `(buf, buflen) -> bytes_written`.+fn read_widestring<F>(getter: F) -> String+where+ F: Fn(*mut std::os::raw::c_void, std::os::raw::c_ulong) -> std::os::raw::c_ulong,+{+ let needed = getter(std::ptr::null_mut(), 0) as usize;+ if needed < 2 {+ return String::new();+ }+ let mut buf: Vec<u16> = vec![0; needed / 2];+ let written = getter(+ buf.as_mut_ptr() as *mut std::os::raw::c_void,+ needed as std::os::raw::c_ulong,+ ) as usize;+ if written < 2 {+ return String::new();+ }+ let chars = written / 2;+ let end = if buf.get(chars - 1) == Some(&0) {+ chars - 1+ } else {+ chars+ };+ String::from_utf16_lossy(&buf[..end])+}