Skip to content

A fast, helpful, and open-source document parser

wip

Logan Markewichfe03a4f3159b
diff --git a/MARKDOWN_PROGRESS.md b/MARKDOWN_PROGRESS.md
index 73ea9d4..1488435 100644
--- a/MARKDOWN_PROGRESS.md
+++ b/MARKDOWN_PROGRESS.md
@@ -154,7 +154,7 @@ Steps 1–8 = usable v1. Steps 9–11 = quality bumps for specific doc classes.
> **Doc-level losses:** doc 01030000000122 (lab manual with sparse bullets in two columns) NID 0.525 → 0.471. Doc 122's histogram does show two roughly-balanced peaks at x=229 and x=457 with 92/17 items — technically passes the gates because the 92 includes a long chunk of straight-through-the-page paragraphs that all happen to start at the same x — but the doc is structurally a single column of mixed bullet/figure/table content where slicing into a "right column" creates phantom reading-order jumps. Not worth tighter gates that would also lose doc 029.
>
> **Carry-forwards still open after this pass:**
-> - **Approach #2 (mask spanning items from density projection).** Before computing the V-cut density valley, drop items whose width ≥ ~70% of the region width — those are headings/figures/tables that span columns and are the reason the density valley search fails in the first place. Symmetric to the figure-obstacle code added 2026-06-01 (which masks figures along *H-cuts*). Should catch the 14 still-MHS=0 docs whose layout has a clean gutter once spanning elements are excluded.
+> - **Approach #2 (mask spanning items from density projection).** ❌ TRIED AND REVERTED 2026-06-02 — see the 2026-06-02 entry below. Masking items ≥70% region width from the V-cut made density cuts slice tables/wide layouts column-wise and regressed all three metrics. The legitimate column case is already handled by the gated `xy_find_column_cut` fallback. Do not retry without a guard that prevents the revealed valley from cutting through tabular content.
> - **Approach #3 (split-at-big-gap inside `build_projected_lines`).** Even if XY-cut fails, the per-leaf y-banding step could split a band whose items have a horizontal gap ≥ ~6–8 × median glyph advance AND ≥ ~30pt absolute. This is what `pdftotext` does. Riskier — justified prose has wide inter-word gaps, especially near inline figures — so should be tightly gated and only as a last resort if #2 doesn't close the remainder.
> - **`xy_root_bbox` negative-x handling.** Book-spread layouts where items have x<0 break bbox-relative reasoning everywhere downstream. A conditional "items extend past MediaBox" detector that widens the bbox only for those pages would unlock the rest of doc 029 and likely a handful of other book-interior PDFs.
>
@@ -165,6 +165,69 @@ Steps 1–8 = usable v1. Steps 9–11 = quality bumps for specific doc classes.
>
> **Bench plumbing landed:** `opendataloader-bench/src/pdf_parser_liteparse_placeholder.py` (image_mode=placeholder variant) and `pdf_parser_liteparse_ocr.py` (ocr_enabled=True variant), both registered in `engine_registry.py`. `liteparse-placeholder` confirms image placeholders cost ~0.003 NID — negligible vs the structural gains, so `Placeholder` is the right default.
+> **2026-06-01 TOC suppression pass.** Bench analysis vs. nutrient (0.885 overall, lp 0.745) identified two MHS failure clusters: (a) **TOC entry over-promotion** — docs 16, 18, 113, 171 each emitted 4-20 false-positive `# Chapter X 41`-style headings when the doc was just a TOC where GT wants 1 `# Contents` heading; (b) docs that are missing a single section heading on 2-column layouts (38, 77, 189) — layout-related, deferred. Fix in `markdown_layout/headings.rs` + `classify.rs`:
+> - **`toc_entry_arabic_number(text)`**: returns the trailing arabic page number when the line looks like a TOC entry — ≥8 alpha chars in the body, separator that includes whitespace (rejecting decimals like "94.2" or chart values "OCR-Recall 7 94.2"), 1-4 digit tail, body doesn't end in a digit (rejecting "vol 5 12" multi-number tails). The ≥8-alpha threshold protects real one-word headings like "Chapter 7" / "Section 4".
+> - **`looks_like_toc_entry`**: wraps arabic + a separate Roman-numeral check (matches "Author's Note ... ix"). Used only for the page-level row floor count.
+> - **`is_toc_title`**: matches normalized "contents" / "table of contents" / "index" / "list of figures" etc. Force-promoted to H1 on TOC pages even when the heading-map size filters would have dropped it (single-occurrence, short, etc.).
+> - **`page_is_toc`**: page qualifies when ≥4 TOC-like lines AND ≥3 arabic-trailing entries whose page numbers form a ≥70% non-decreasing sequence. The monotonicity check is what separates real TOCs from chart/graph pages with random axis-value tails (doc 199 was the calibration target — was getting false-positive'd, now correctly rejected).
+> - **Classifier wiring** (`classify.rs::classify_page_with_filters`): per-page `toc_page = page_is_toc(page)`. Per-line `toc_suppress = toc_page && !is_toc_title(text)` guards the size-based, numbered-bold-heading, and bold-heading promotion sites. A separate `toc_title_level = Some(1)` path force-promotes the TOC's own title even when size_level was None.
+> - **Tests**: 4 new unit tests (`toc_entry_arabic_extracts_trailing_page_number`, `toc_entry_arabic_rejects_decimals_and_axis_labels`, `is_toc_title_matches_common_variants`, `page_is_toc_requires_monotonic_page_numbers`). 151 lib + 10 integration tests pass.
+>
+> **Bench result on opendataloader-bench (200 docs):**
+>
+> | Metric | Pre-fix | Post-fix | Δ |
+> |---|---|---|---|
+> | NID | 0.8822 | 0.8822 | ~0 |
+> | TEDS | 0.3862 | 0.3862 | 0 (no table changes) |
+> | MHS | 0.5146 | **0.5358** | **+0.0212** |
+> | Overall | 0.7451 | **0.7573** | **+0.0122** |
+>
+> **Per-doc wins** (MHS): doc 16: 0.074→0.750, doc 18: 0.025→0.651, doc 113: ?→0.892, doc 171: 0.133→0.744, doc 51: regression-then-recovered 0.94, doc 153: 0.83.
+>
+> **Still wide-open after this pass** (rough TODO priority):
+> - **TEDS** still 0.39 vs nutrient 0.71. Biggest gaps are docs 89, 47, 128, 182, 88, 90 — multi-line wrapped header rows where each column's header text spans multiple baselines at staggered positions; the borderless detector closes the run after 1 row because the first "row" comes back as 1 merged cell. Other failure mode: complex spreadsheet-like grids (docs 119, 120, 121, 130, 149, 180, 121) where ground truth uses `<table>` HTML with multi-line cells and our column detector slices the layout into 2-3 tiny tables. Both need either ML-grade cell detection or a much smarter borderless table merger that walks back to absorb headers above a detected body.
+> - **MHS** — remaining MHS=0 docs are dominated by 2-column section headings being eaten by adjacent-column body text (docs 7, 13, 38, 77, 118, 148, 177, 178, 189, 198). Same root cause as the long-tracked `paper.pdf` page-1 banner issue: layout decomposition can't isolate single-line bold section heads that sit at the same y-band as content in the other column. Needs the wide-banner / figure-rect-aware XY-cut refinements.
+> - **TEDS multi-line header absorption** is the highest-leverage table fix — would lift docs 47, 89, 182, 88, 90 from 0.1–0.4 to ~0.7+. Approach: when ruled-grid detection or borderless detection finds a body table, walk back N lines (where N matches the detected column count's track positions) and check if those upper lines have cells aligning to the same column tracks → absorb as header rows.
+
+> **2026-06-02 ruled-grid single-column tables (`tables.rs`).** Targeted the largest TEDS=0 cluster (15 docs with GT tables we emitted 0 columns for). Single-col GT table was 1 doc (149), but the same code path also unblocked thin-border-strip docs whose true content column was wedged between two phantom rule-strip columns. Three coordinated changes in `build_ruled_table`:
+>
+> - **Threshold relaxation:** `xs.len() >= 3` → `xs.len() >= 2`. Allows the detector to produce 1-column tables when the geometry only gives a single content column between borders.
+> - **Phantom-row collapse:** after cell assignment, drop rows whose height < 0.8 × median non-empty row height AND that have no text. Doc 149 draws each visual row as three stacked rects (top-border 1pt, body 22pt, bottom-strip 5pt) which all survive 2pt y-clustering as separate grid rows — without collapse, the empty-cell fraction kills the table. Threshold preserves real fill-in-the-blank forms (empty body rows match median non-empty height).
+> - **Phantom-column collapse:** mirror of the row rule for columns. Drop empty columns whose width < 0.3 × median non-empty column width. Doc 149 has 5.5pt left/right border-strip columns next to a 305pt content column — both drop cleanly.
+>
+> One stricter rule in `merge_table_runs`: a 1-column ruled run **yields to** a multi-column borderless run that overlaps the same line range. Without this, doc 078's data table (drawn with top/bottom/header rules but no internal V-rules) gets detected as a degenerate 1-col grid that overrides the correct 7-col borderless result. Path geometry strictly beats text alignment for *multi*-col ruled runs, but a 1-col ruled run is ambiguous — could be a real single-column table or a multi-col table missing its V-rules — so we defer to the multi-col borderless evidence when present.
+>
+> **Bench result (200 docs):**
+>
+> | Metric | Pre | Post | Δ |
+> |---|---|---|---|
+> | NID | 0.8824 | 0.8819 | −0.0005 (noise) |
+> | TEDS | 0.3906 | **0.4144** | **+0.0238** |
+> | MHS | 0.5913 | 0.5928 | +0.0015 |
+> | Overall | 0.7713 | **0.7738** | **+0.0025** |
+>
+> **Per-doc:** doc 149 0 → 1.0 TEDS (clean recovery — 8-row single-col ECO Circle competence stack). Several mid-TEDS docs (128, 170, 188) also improved a few hundredths each — likely because the `0.8 × median` row collapse now bridges thin-rule rows in their existing detections. No TEDS regressions ≥ 0.01.
+>
+> **Still wide open after this pass** — the bulk of the TEDS=0 cluster is **2-column tables** (7 docs: 121, 132, 146, 150, 165, 166, 197). The borderless detector explicitly requires `TABLE_MIN_COLUMNS = 3`, which kills every 2-col borderless table on the bench. Lowering it to 2 is high-leverage but risky: key:value bullet lists and 2-column body layouts will look the same. Needs guards before attempting. The ruled-grid path I just relaxed can pick up 2-col ruled tables (doc 146 has heavy graphics), but they don't appear to fire — likely because the connected-component step is splitting the grid, or because `find_grid_components` requires ≥2 H AND ≥2 V components to cross within 3pt and doc 146 has many tiny rect segments that don't cleanly cross. Worth dumping next pass.
+>
+> **2026-06-02 heading char-floor, TOC-title, header absorption; XY-cut spanning-mask tried and reverted.** Four targeted heuristic changes plus one reverted layout experiment. Net result on opendataloader-bench (200 docs):
+>
+> | Metric | Pre (260601) | Post | Δ |
+> |---|---|---|---|
+> | NID | 0.8822 | 0.8824 | +0.0002 |
+> | TEDS | 0.3862 | **0.3906** | **+0.0044** |
+> | MHS | 0.5358 | **0.5913** | **+0.0555** |
+> | Overall | 0.7573 | **0.7713** | **+0.0140** |
+>
+> - **Item 1 — heading char-floor (`headings.rs`).** `MIN_HEADING_TOTAL_CHARS` 25→10 (now just a noise guard); added `MIN_HEADING_AVG_LINE_CHARS = 8.0` as the real legend-token discriminator. `build_heading_map` now keeps a size-outlier cluster iff `avg_line_chars` is in `[8, MAX]`. Rescues clean single-line headings like "Aligning with Your Identity" (24 chars/line) that the 25-char floor dropped, while still rejecting paper-legend tokens ("A-mem"/"Base" ≈ 4-5 chars/line). MHS contributor.
+> - **Item 3 — TOC title without `page_is_toc` (`classify.rs`).** `toc_title_level` now fires on `is_toc_title(text)` alone (was gated on `toc_page`). `is_toc_title` matches the whole trimmed line against an exact whitelist (contents / table of contents / index / …) so it can't fire mid-prose. MHS contributor.
+> - **Item 4 — multi-line header absorption (`tables.rs`).** After the borderless detector builds a body run, `absorb_header_lines` walks backward (bounded by `floor` = prior run's end) pulling in lines whose cells (2 ≤ n ≤ column_count) all align to the established column tracks and sit tight above the row below. Multiple wrapped header lines collapse into one markdown header row (joined per-track top-to-bottom). `try_detect_table` gained a `floor` param; `detect_tables` threads the last run end through it. Single-cell titles are explicitly NOT absorbed (≥2 aligned cells required). 2 new unit tests. **TEDS +0.0044, NID neutral** — a clean win. Per-doc: 128 +0.069, 170 +0.062, 188 +0.059.
+> - **Item 5 — XY-cut spanning-item mask (REVERTED).** Implemented Approach #2 (mask items ≥70% region width from the V-cut density projection). Bench showed it **hurt all three metrics**: TEDS 0.3906→0.3614, NID 0.8824→0.8780, MHS 0.5913→0.5821. Root cause: masking wide items makes density V-cuts fire *inside* tables and wide layouts, slicing them column-wise (docs 082/187/189 lost both TEDS and NID together — the signature of a split table). The existing `xy_find_column_cut` fallback (gated to fire only when both density cuts return None) already covers the legitimate column case without the collateral damage. **Conclusion: the density V-cut must stay conservative; spanning-mask is the wrong lever. Do not retry without a tabular-content guard that prevents the revealed valley from cutting through a table.** Fully reverted (constant + mask loop removed from `xy_find_best_cut`).
+>
+> **Item 2 (sparse-form ruled-table empty-cell relaxation) remains deferred** — `TABLE_MAX_EMPTY_CELL_FRACTION` stays 0.30. Real sparse table (doc 180) and spurious decorative grid (doc 198 TOC) are both ~75% empty, so empty-fraction is not a valid discriminator; loosening it regressed TEDS −0.089.
+>
+> **Struct-tag question answered:** 0/200 bench docs carry PDF struct tags (verified via `survey_tags` example), so the struct-tree heading path is moot for this benchmark — all gains must come from text/geometry heuristics.
+
## Carried-forward items
Things deferred from earlier steps that need to be revisited:
diff --git a/crates/liteparse/src/markdown_layout/classify.rs b/crates/liteparse/src/markdown_layout/classify.rs
index 2de9c2d..fe0ad7a 100644
--- a/crates/liteparse/src/markdown_layout/classify.rs
+++ b/crates/liteparse/src/markdown_layout/classify.rs
@@ -3,7 +3,8 @@ use crate::types::{OutlineTarget, ParsedPage, ProjectedLine};
use super::blocks::{Block, paragraph_from_accum};
use super::headings::{
MAX_HEADING_LEVELS, heading_level_for, is_caption_line, looks_like_bold_heading,
- looks_like_numbered_bold_heading, outline_heading_level, struct_heading_level,
+ is_toc_title, looks_like_numbered_bold_heading, outline_heading_level, page_is_toc,
+ struct_heading_level,
};
use super::hr::detect_horizontal_rules;
use super::inline::{
@@ -86,6 +87,14 @@ pub fn classify_page_with_filters(
let debug = std::env::var("LITEPARSE_DEBUG_MD").is_ok();
+ // TOC suppression: when ≥3 lines on this page look like TOC entries
+ // (alpha body + trailing page-number), demote heading promotion so each
+ // entry stays a paragraph instead of becoming a fake H1/H2. The TOC's
+ // own title ("Contents", "Table of Contents") is shorter without a
+ // trailing number — it falls through the size/bold heuristics with no
+ // special handling needed.
+ let toc_page = page_is_toc(page);
+
// Strip running header/footer lines up-front so they don't leak into
// table detection (a repeating two-column footer would otherwise look
// like a 2-row table) or paragraph grouping.
@@ -338,13 +347,25 @@ pub fn classify_page_with_filters(
// distinct (and slightly larger) font that lands them in the
// font-size heading map. Suppress font-size promotion for them;
// outline / struct-tree signals still win since those are explicit.
- let size_level = if is_caption_line(text) {
+ let toc_suppress = toc_page && !is_toc_title(text);
+ let size_level = if is_caption_line(text) || toc_suppress {
None
} else {
heading_level_for(line.dominant_font_size, heading_map)
};
+ // A standalone "Contents" / "Table of Contents" / "Index" line is
+ // almost always a real H1, even when `page_is_toc` is false — many
+ // TOCs list entries without inline trailing page numbers, so the
+ // page-level detector misses them. `is_toc_title` matches the *entire*
+ // trimmed line against an exact whitelist, so it can't fire mid-prose.
+ let toc_title_level = if is_toc_title(text) {
+ Some(1u8)
+ } else {
+ None
+ };
let level = outline_level
.or(size_level)
+ .or(toc_title_level)
.map(|l| l.clamp(1, MAX_HEADING_LEVELS as u8));
if let Some(level) = level {
flush_paragraph(&mut blocks, paragraph.take());
@@ -368,6 +389,7 @@ pub fn classify_page_with_filters(
// headings in technical/legal/scientific PDFs silently emit as
// ordered list items and lose all heading structure.
if ordered
+ && !toc_suppress
&& looks_like_numbered_bold_heading(
line,
rest,
@@ -438,7 +460,7 @@ pub fn classify_page_with_filters(
.map(|p| &p.last)
.or(last_list_line.as_ref());
let next_for_gap = lines.get(idx);
- if looks_like_bold_heading(line, prev_for_gap, next_for_gap) {
+ if !toc_suppress && looks_like_bold_heading(line, prev_for_gap, next_for_gap) {
flush_paragraph(&mut blocks, paragraph.take());
list_base_indent = None;
last_list_item_idx = None;
diff --git a/crates/liteparse/src/markdown_layout/headings.rs b/crates/liteparse/src/markdown_layout/headings.rs
index 3421d5a..003bf25 100644
--- a/crates/liteparse/src/markdown_layout/headings.rs
+++ b/crates/liteparse/src/markdown_layout/headings.rs
@@ -57,6 +57,168 @@ pub(super) fn is_caption_line(text: &str) -> bool {
false
}
+/// Trailing page-number extractor for TOC-entry shaped lines. Returns the
+/// numeric value of the trailing page number when the line looks like a TOC
+/// entry: alphabetic body, separator that *includes whitespace* (to reject
+/// decimals like "94.2"), then a trailing arabic 1–4 digit number. Roman
+/// numerals are accepted in `looks_like_toc_entry` but not returned for the
+/// monotonic-sequence check.
+pub(super) fn toc_entry_arabic_number(text: &str) -> Option<i32> {
+ let s = text.trim();
+ if s.is_empty() {
+ return None;
+ }
+ let chars: Vec<char> = s.chars().collect();
+ let n = chars.len();
+ let mut tail_start = n;
+ while tail_start > 0 && chars[tail_start - 1].is_ascii_digit() {
+ tail_start -= 1;
+ }
+ let tail_len = n - tail_start;
+ if tail_len == 0 || tail_len > 4 {
+ return None;
+ }
+ // Separator: ≥1 whitespace + optional '.' leaders. The mandatory
+ // whitespace rules out decimals ("94.2") and inline number suffixes
+ // ("Recall3 7 94.2") that aren't TOC entries.
+ let mut sep_end = tail_start;
+ let mut saw_ws = false;
+ while sep_end > 0 {
+ let c = chars[sep_end - 1];
+ if c.is_whitespace() {
+ sep_end -= 1;
+ saw_ws = true;
+ } else if c == '.' {
+ sep_end -= 1;
+ } else {
+ break;
+ }
+ }
+ if !saw_ws {
+ return None;
+ }
+ // Body must (a) carry meaningful alpha content and (b) NOT end with a
+ // digit — that would mean we sliced a multi-part number ("vol 5 12").
+ let body = &chars[..sep_end];
+ let alpha = body.iter().filter(|c| c.is_alphabetic()).count();
+ // Require ≥8 alpha chars in the body. This keeps real one-word headings
+ // like "Chapter 7" / "Section 4" out of the TOC bucket while still
+ // accepting typical TOC entries ("Introduction 7", "Conclusion 127", ...).
+ if alpha < 8 {
+ return None;
+ }
+ if body
+ .iter()
+ .rev()
+ .find(|c| !c.is_whitespace())
+ .is_some_and(|c| c.is_ascii_digit())
+ {
+ return None;
+ }
+ let tail: String = chars[tail_start..].iter().collect();
+ tail.parse::<i32>().ok()
+}
+
+/// Roman-or-arabic TOC-entry detector — used by tests and the TOC-page
+/// detector to count TOC-shaped lines that the strict arabic extractor
+/// above misses (e.g. "Author's Note ... ix").
+pub(super) fn looks_like_toc_entry(text: &str) -> bool {
+ if toc_entry_arabic_number(text).is_some() {
+ return true;
+ }
+ // Try Roman numerals at the tail.
+ let s = text.trim();
+ let chars: Vec<char> = s.chars().collect();
+ let n = chars.len();
+ let mut tail_start = n;
+ while tail_start > 0
+ && matches!(
+ chars[tail_start - 1],
+ 'i' | 'v' | 'x' | 'l' | 'c' | 'd' | 'm' | 'I' | 'V' | 'X' | 'L' | 'C' | 'D' | 'M'
+ )
+ {
+ tail_start -= 1;
+ }
+ let tail_len = n - tail_start;
+ if tail_len < 2 || tail_len > 6 {
+ return false;
+ }
+ let mut sep_end = tail_start;
+ let mut saw_ws = false;
+ while sep_end > 0 {
+ let c = chars[sep_end - 1];
+ if c.is_whitespace() {
+ sep_end -= 1;
+ saw_ws = true;
+ } else if c == '.' {
+ sep_end -= 1;
+ } else {
+ break;
+ }
+ }
+ if !saw_ws {
+ return false;
+ }
+ let alpha = chars[..sep_end]
+ .iter()
+ .filter(|c| c.is_alphabetic())
+ .count();
+ alpha >= 5
+}
+
+/// Returns true if `text` is the canonical title of a TOC page: "Contents",
+/// "Table of Contents", "Index", etc. We use this to let the TOC's own title
+/// through the heading promotion that's otherwise suppressed on TOC pages.
+pub(super) fn is_toc_title(text: &str) -> bool {
+ let t = text.trim().trim_end_matches(':').to_ascii_lowercase();
+ matches!(
+ t.as_str(),
+ "contents"
+ | "table of contents"
+ | "table of content"
+ | "index"
+ | "list of figures"
+ | "list of tables"
+ | "table of figures"
+ | "toc"
+ )
+}
+
+/// TOC-page detection. A page is a TOC iff it carries ≥4 arabic-trailing
+/// TOC-entry lines whose page numbers form a *mostly non-decreasing*
+/// sequence (≥70% of adjacent pairs satisfy `next >= prev`). The
+/// monotonicity check is what separates real TOCs from chart/graph pages
+/// with random axis-value tails. Lines whose tail isn't arabic but matches
+/// the looser `looks_like_toc_entry` (e.g. roman numerals "ix", "xi") still
+/// count toward the row floor — they just don't participate in the
+/// monotonicity check.
+pub(super) fn page_is_toc(page: &ParsedPage) -> bool {
+ let mut nums: Vec<i32> = Vec::new();
+ let mut total_toc_like = 0usize;
+ for line in &page.projected_lines {
+ if is_rotated_line(line) {
+ continue;
+ }
+ if let Some(n) = toc_entry_arabic_number(&line.text) {
+ nums.push(n);
+ total_toc_like += 1;
+ } else if looks_like_toc_entry(&line.text) {
+ total_toc_like += 1;
+ }
+ }
+ if total_toc_like < 4 || nums.len() < 3 {
+ return false;
+ }
+ let mut nondec = 0usize;
+ for w in nums.windows(2) {
+ if w[1] >= w[0] {
+ nondec += 1;
+ }
+ }
+ let frac = nondec as f32 / (nums.len() - 1) as f32;
+ frac >= 0.7
+}
+
/// Returns true if `line` looks like a section heading rendered in body-size
/// bold text (a very common style for academic / technical PDFs where every
/// "real" heading uses the same font size as body, distinguished only by
@@ -255,14 +417,20 @@ pub fn compute_body_size(pages: &[ParsedPage]) -> f32 {
}
/// Minimum total non-whitespace characters across all occurrences at a font
-/// size for it to qualify as a heading level. Calibrated against `paper.pdf`:
-/// the 30pt chart-legend tokens ("A-mem"×2 + "Base"×2 = 18-20 chars) need to
-/// fail this filter, while the 14.35pt title ("A-MEM: Agentic Memory for LLM
-/// Agents" = 31 chars on a single line) needs to pass. 25 is the gap. Smaller
-/// single-word headings like a lone "Summary" on a short doc still survive
-/// because they share their font size with other (larger) headings in the
-/// histogram entry.
-const MIN_HEADING_TOTAL_CHARS: usize = 25;
+/// size for it to qualify as a heading level. Low floor — just a noise guard
+/// against 1-2 char artifacts. The real legend-token discriminator is
+/// `MIN_HEADING_AVG_LINE_CHARS` below.
+const MIN_HEADING_TOTAL_CHARS: usize = 10;
+
+/// Minimum average non-whitespace characters *per line* at a font size for it
+/// to qualify as a heading. This is what separates a real heading (a coherent
+/// line like "Annual Events" = 12 chars/line, or "A-MEM: Agentic Memory for
+/// LLM Agents" = 31) from scattered chart-legend tokens at a one-off display
+/// size (paper.pdf's "A-mem"/"Base" ~ 4-5 chars/line). A per-line average is
+/// robust where a total-char floor was not: a single short-but-real heading
+/// (14pt "Aligning with Your Identity" = 24 non-ws chars, over 10.5pt body)
+/// clears it, while many tiny repeated tokens do not no matter how many.
+const MIN_HEADING_AVG_LINE_CHARS: f32 = 8.0;
/// Maximum average characters per line for a size to qualify as a heading.
/// A "size larger than body" with very long lines is almost always a
@@ -325,8 +493,10 @@ pub fn build_heading_map(pages: &[ParsedPage], body_size: f32) -> Vec<(f32, u8)>
} else {
(*alpha as f32) / (*chars as f32)
};
+ let avg_line_chars = *chars as f32 / (*lines).max(1) as f32;
*chars >= MIN_HEADING_TOTAL_CHARS
- && (*chars as f32 / (*lines).max(1) as f32) <= MAX_HEADING_AVG_LINE_CHARS
+ && avg_line_chars >= MIN_HEADING_AVG_LINE_CHARS
+ && avg_line_chars <= MAX_HEADING_AVG_LINE_CHARS
&& alpha_ratio >= MIN_HEADING_ALPHA_RATIO
})
.map(|(s, _, _, _)| *s)
@@ -466,6 +636,61 @@ mod tests {
use super::super::test_helpers::{line, page};
use super::*;
+ #[test]
+ fn toc_entry_arabic_extracts_trailing_page_number() {
+ assert_eq!(toc_entry_arabic_number("Introduction 7"), Some(7));
+ assert_eq!(
+ toc_entry_arabic_number("1. A Fountain in the Square 1"),
+ Some(1)
+ );
+ assert_eq!(
+ toc_entry_arabic_number("6. For the Love of Iran . . . 41"),
+ Some(41)
+ );
+ }
+
+ #[test]
+ fn toc_entry_arabic_rejects_decimals_and_axis_labels() {
+ // "94.2" → decimal, not a TOC entry
+ assert_eq!(toc_entry_arabic_number("OCR-Recall3 7 94.2"), None);
+ // Chapter 7 — too short an alpha body to be a TOC entry
+ assert_eq!(toc_entry_arabic_number("Chapter 7"), None);
+ // No separator
+ assert_eq!(toc_entry_arabic_number("Section1"), None);
+ }
+
+ #[test]
+ fn is_toc_title_matches_common_variants() {
+ assert!(is_toc_title("Contents"));
+ assert!(is_toc_title("Table of Contents"));
+ assert!(is_toc_title("table of contents"));
+ assert!(is_toc_title("Index"));
+ assert!(!is_toc_title("Introduction"));
+ }
+
+ #[test]
+ fn page_is_toc_requires_monotonic_page_numbers() {
+ // Real TOC: monotonically increasing page numbers.
+ let pages_toc = page(vec![
+ line("Table of contents", 50.0, 30.0, 18.0, 18.0),
+ line("Introduction 7", 50.0, 60.0, 12.0, 12.0),
+ line("Part I: New Children 21", 50.0, 72.0, 12.0, 12.0),
+ line("Part II: From Solitary 45", 50.0, 84.0, 12.0, 12.0),
+ line("Part III: Commercial 71", 50.0, 96.0, 12.0, 12.0),
+ line("Conclusion 127", 50.0, 108.0, 12.0, 12.0),
+ ]);
+ assert!(page_is_toc(&pages_toc));
+
+ // Chart page: trailing numbers but NOT monotonic and many fail the
+ // separator/alpha rules anyway.
+ let pages_chart = page(vec![
+ line("OCR-Recall is 94.2", 50.0, 60.0, 9.0, 9.0),
+ line("Precision rate 89.0", 50.0, 72.0, 9.0, 9.0),
+ line("F1 score 80.4", 50.0, 84.0, 9.0, 9.0),
+ ]);
+ assert!(!page_is_toc(&pages_chart));
+ }
+
#[test]
fn body_size_picks_most_common() {
let pages = vec![page(vec![
diff --git a/crates/liteparse/src/markdown_layout/tables.rs b/crates/liteparse/src/markdown_layout/tables.rs
index 102270b..8506974 100644
--- a/crates/liteparse/src/markdown_layout/tables.rs
+++ b/crates/liteparse/src/markdown_layout/tables.rs
@@ -281,7 +281,7 @@ fn row_spacing_cv(rows: &[(usize, &ProjectedLine, Vec<TableCell>)]) -> f32 {
/// a `TableRun` with `Block::Table` or `Block::GridFallback`; on failure
/// returns `None` (and the caller should fall through to per-line
/// classification).
-fn try_detect_table(lines: &[ProjectedLine], start_idx: usize) -> Option<TableRun> {
+fn try_detect_table(lines: &[ProjectedLine], start_idx: usize, floor: usize) -> Option<TableRun> {
let first_cells = split_cells(&lines[start_idx]);
if first_cells.len() < TABLE_MIN_COLUMNS {
return None;
@@ -383,16 +383,30 @@ fn try_detect_table(lines: &[ProjectedLine], start_idx: usize) -> Option<TableRu
});
}
- // Promote the first row to header iff every cell in it is bold (matches
- // pymupdf4llm's "bold-or-filled" heuristic; fills require fork data).
+ // Walk back above the detected body and absorb header lines that align to
+ // the same column tracks but weren't includable as body rows (merged /
+ // partial header cells). Multiple wrapped header lines collapse into one
+ // markdown header row, joined per-column top-to-bottom.
+ let absorbed = absorb_header_lines(lines, start_idx, &tracks, column_count, floor);
+
+ // Promote the first body row to header iff every cell in it is bold
+ // (matches pymupdf4llm's "bold-or-filled" heuristic; fills require fork
+ // data). Skipped when we already absorbed an explicit header above.
let first_row = &rows[0].2;
- let header_qualifies = first_row.iter().all(|c| c.bold);
- let header = if header_qualifies {
- Some(first_row.iter().map(|c| c.text.clone()).collect())
- } else {
- None
+ let bold_header_qualifies = absorbed.is_none() && first_row.iter().all(|c| c.bold);
+
+ // `row_start` is the index of the first body row within `rows`. When the
+ // header came from absorbed lines above, every detected row is body data;
+ // only the bold-first-row promotion consumes rows[0].
+ let (run_start, header, row_start) = match absorbed {
+ Some((hstart, header_texts)) => (hstart, Some(header_texts), 0),
+ None if bold_header_qualifies => (
+ start_idx,
+ Some(first_row.iter().map(|c| c.text.clone()).collect()),
+ 1,
+ ),
+ None => (start_idx, None, 0),
};
- let row_start = if header.is_some() { 1 } else { 0 };
let body_rows: Vec<Vec<String>> = rows[row_start..]
.iter()
.map(|(_, _, cells)| cells.iter().map(|c| c.text.clone()).collect())
@@ -402,7 +416,7 @@ fn try_detect_table(lines: &[ProjectedLine], start_idx: usize) -> Option<TableRu
}
Some(TableRun {
- start: start_idx,
+ start: run_start,
end,
block: Block::Table {
header,
@@ -411,14 +425,81 @@ fn try_detect_table(lines: &[ProjectedLine], start_idx: usize) -> Option<TableRu
})
}
+/// Walk backward from `start_idx` (not below `floor`), pulling in lines whose
+/// cells all align to the table's `tracks` as header rows. Returns the new
+/// start index and a single merged header row (`column_count` columns) with
+/// each absorbed line's text appended into its nearest column track.
+fn absorb_header_lines(
+ lines: &[ProjectedLine],
+ start_idx: usize,
+ tracks: &[f32],
+ column_count: usize,
+ floor: usize,
+) -> Option<(usize, Vec<String>)> {
+ let mut absorbed: Vec<Vec<TableCell>> = Vec::new();
+ let mut j = start_idx;
+ while j > floor {
+ let cand = j - 1;
+ let cells = split_cells(&lines[cand]);
+ // A header line must carry at least two track-aligned cells (a single
+ // cell is a title/caption, not a header), no more than column_count,
+ // sit tight above the row below it, and have every cell land on a
+ // known column track.
+ if cells.len() < 2 || cells.len() > column_count {
+ break;
+ }
+ if !table_rows_adjacent(&lines[cand], &lines[j]) {
+ break;
+ }
+ let all_align = cells.iter().all(|c| {
+ tracks
+ .iter()
+ .any(|t| (c.start_x - *t).abs() <= TABLE_TRACK_TOLERANCE_PT)
+ });
+ if !all_align {
+ break;
+ }
+ absorbed.push(cells);
+ j = cand;
+ }
+ if absorbed.is_empty() {
+ return None;
+ }
+ // Collected bottom-up; reverse so text reads top-to-bottom per column.
+ absorbed.reverse();
+ let mut header = vec![String::new(); column_count];
+ for cells in &absorbed {
+ for c in cells {
+ let Some(idx) = tracks
+ .iter()
+ .enumerate()
+ .filter(|(_, t)| (c.start_x - **t).abs() <= TABLE_TRACK_TOLERANCE_PT)
+ .min_by(|(_, a), (_, b)| {
+ (c.start_x - **a).abs().total_cmp(&(c.start_x - **b).abs())
+ })
+ .map(|(i, _)| i)
+ else {
+ continue;
+ };
+ if !header[idx].is_empty() && !c.text.is_empty() {
+ header[idx].push(' ');
+ }
+ header[idx].push_str(&c.text);
+ }
+ }
+ Some((j, header))
+}
+
/// Scan `lines` once and return all detected tabular regions (sorted by
/// `start`). Caller uses these as cut-points so the per-line classifier never
/// sees lines inside a table.
pub(super) fn detect_tables(lines: &[ProjectedLine]) -> Vec<TableRun> {
let mut out = Vec::new();
let mut i = 0;
+ let mut floor = 0;
while i < lines.len() {
- if let Some(run) = try_detect_table(lines, i) {
+ if let Some(run) = try_detect_table(lines, i, floor) {
+ floor = run.end;
i = run.end;
out.push(run);
} else {
@@ -474,6 +555,11 @@ const TABLE_GRID_CLUSTER_PT: f32 = 2.0;
const TABLE_CROSS_TOLERANCE_PT: f32 = 3.0;
/// Reject ruled-table candidates whose empty-cell fraction exceeds this.
+/// NOTE: this can't be loosened to recover blank worksheets/forms — a real
+/// sparse table (doc 180, a 4-col Version History, ~75% empty) and a spurious
+/// grid from decorative layout boxes (doc 198, a TOC, also ~75% empty) are
+/// indistinguishable on empty-fraction, and relaxing it net-regressed TEDS by
+/// ~0.09 on the bench (more false tables than real forms recovered).
const TABLE_MAX_EMPTY_CELL_FRACTION: f32 = 0.30;
/// Reject candidates whose grid covers nearly the whole page — almost always
@@ -672,8 +758,12 @@ fn build_ruled_table(
xs.sort_by(|a, b| a.total_cmp(b));
dedup_close(&mut xs, TABLE_GRID_CLUSTER_PT);
- // Need ≥2 cells per axis → ≥3 boundary lines.
- if ys.len() < 3 || xs.len() < 3 {
+ // Need ≥2 row boundaries (1 row) and ≥2 column boundaries (1 col); but
+ // a 1×1 grid is just a callout box, so also require ≥1 inner divider
+ // (i.e. ys.len() ≥ 3 for ≥2 rows). Single-column tables (`xs.len() == 2`)
+ // are accepted when row evidence is strong enough — extra guards apply
+ // below after the empty-row collapse.
+ if ys.len() < 3 || xs.len() < 2 {
return None;
}
@@ -733,6 +823,122 @@ fn build_ruled_table(
return None;
}
+ // Collapse "phantom rows" produced by stacked thin border-strip rects
+ // (doc 149 draws each visual table row as: top border strip ~1pt, body
+ // rect ~22pt, bottom border strip ~5pt — each contributes y-coords that
+ // survive the 2pt clustering as separate grid rows). Rule: drop a row
+ // iff (a) it has no text in any cell AND (b) its height is < 50% of the
+ // median non-empty row height. The height gate preserves real
+ // fill-in-the-blank forms where empty body rows are full-height.
+ let row_heights: Vec<f32> = (0..n_rows).map(|r| ys[r + 1] - ys[r]).collect();
+ let nonempty_heights: Vec<f32> = (0..n_rows)
+ .filter(|r| cell_has_text[*r].iter().any(|t| *t))
+ .map(|r| row_heights[r])
+ .collect();
+ let median_h = if !nonempty_heights.is_empty() {
+ let mut s = nonempty_heights.clone();
+ s.sort_by(|a, b| a.total_cmp(b));
+ s[s.len() / 2]
+ } else {
+ let mut s = row_heights.clone();
+ s.sort_by(|a, b| a.total_cmp(b));
+ s[s.len() / 2]
+ };
+ let keep: Vec<bool> = (0..n_rows)
+ .map(|r| {
+ let has_text = cell_has_text[r].iter().any(|t| *t);
+ has_text || row_heights[r] >= median_h * 0.8
+ })
+ .collect();
+ let cells: Vec<Vec<String>> = (0..n_rows)
+ .filter(|r| keep[*r])
+ .map(|r| cells[r].clone())
+ .collect();
+ let cell_has_text: Vec<Vec<bool>> = (0..n_rows)
+ .filter(|r| keep[*r])
+ .map(|r| cell_has_text[r].clone())
+ .collect();
+ let cell_is_bold: Vec<Vec<bool>> = (0..n_rows)
+ .filter(|r| keep[*r])
+ .map(|r| cell_is_bold[r].clone())
+ .collect();
+ let n_rows = cells.len();
+ if n_rows < 2 {
+ return None;
+ }
+
+ // Mirror the row-collapse rule on columns. Some ruled tables (doc 149)
+ // draw their left/right borders as thin strip rects 5pt wide — those
+ // become phantom columns with no text. Drop columns that are both empty
+ // AND noticeably narrower than the median text-bearing column.
+ let col_widths: Vec<f32> = (0..n_cols).map(|c| xs[c + 1] - xs[c]).collect();
+ let nonempty_col_widths: Vec<f32> = (0..n_cols)
+ .filter(|c| (0..n_rows).any(|r| cell_has_text[r][*c]))
+ .map(|c| col_widths[c])
+ .collect();
+ let median_w = if !nonempty_col_widths.is_empty() {
+ let mut s = nonempty_col_widths.clone();
+ s.sort_by(|a, b| a.total_cmp(b));
+ s[s.len() / 2]
+ } else {
+ let mut s = col_widths.clone();
+ s.sort_by(|a, b| a.total_cmp(b));
+ s[s.len() / 2]
+ };
+ let keep_col: Vec<bool> = (0..n_cols)
+ .map(|c| {
+ let has_text = (0..n_rows).any(|r| cell_has_text[r][c]);
+ has_text || col_widths[c] >= median_w * 0.3
+ })
+ .collect();
+ // Cap how aggressively we drop columns. A real table with one phantom
+ // border-strip column (doc 149: 5pt-wide left border) drops exactly one.
+ // Anything more than that is almost certainly a chart whose vertical
+ // grid-lines got merged with text data (doc 078: a chart's 18 V-lines
+ // straddle a table to its left/right and collapse to 1 col), so the
+ // "table" is bogus — bail and let the borderless detector handle it.
+ // Note: keep_col already only drops columns where both (a) no text AND
+ // (b) width < 30% of median text-bearing column.
+ let cells: Vec<Vec<String>> = cells
+ .into_iter()
+ .map(|row| {
+ row.into_iter()
+ .enumerate()
+ .filter(|(c, _)| keep_col[*c])
+ .map(|(_, v)| v)
+ .collect()
+ })
+ .collect();
+ let cell_has_text: Vec<Vec<bool>> = cell_has_text
+ .into_iter()
+ .map(|row| {
+ row.into_iter()
+ .enumerate()
+ .filter(|(c, _)| keep_col[*c])
+ .map(|(_, v)| v)
+ .collect()
+ })
+ .collect();
+ let cell_is_bold: Vec<Vec<bool>> = cell_is_bold
+ .into_iter()
+ .map(|row| {
+ row.into_iter()
+ .enumerate()
+ .filter(|(c, _)| keep_col[*c])
+ .map(|(_, v)| v)
+ .collect()
+ })
+ .collect();
+ let n_cols = cells.first().map(|r| r.len()).unwrap_or(0);
+ if n_cols == 0 {
+ return None;
+ }
+ // Single-column tables are ambiguous (could be a captioned card) — require
+ // ≥3 rows of geometric + textual evidence.
+ if n_cols == 1 && n_rows < 3 {
+ return None;
+ }
+
let total = n_rows * n_cols;
let empty_count = cell_has_text
.iter()
@@ -837,16 +1043,39 @@ pub(super) fn detect_ruled_tables(
/// ranges overlap the ruled run wins (path-based geometry is strictly stronger
/// than text-alignment heuristics) — overlapping borderless runs are dropped.
pub(super) fn merge_table_runs(mut ruled: Vec<TableRun>, borderless: Vec<TableRun>) -> Vec<TableRun> {
+ // A ruled run normally beats an overlapping borderless run (path geometry
+ // is a stronger signal than text alignment). But a 1-column ruled run is
+ // ambiguous — it can come from a real single-column table (doc 149's
+ // stacked cards) OR from a multi-column table whose vertical separators
+ // are implicit and only the top/bottom rules were drawn (doc 078). Yield
+ // to a multi-column borderless run that covers the same range.
+ let mut kept: Vec<TableRun> = Vec::with_capacity(ruled.len());
+ for r in ruled.drain(..) {
+ let is_one_col = matches!(&r.block, Block::Table { rows, .. } if rows.first().map(|row| row.len()).unwrap_or(0) <= 1);
+ if is_one_col {
+ let beaten = borderless.iter().any(|b| {
+ let overlaps = !(b.end <= r.start || b.start >= r.end);
+ if !overlaps {
+ return false;
+ }
+ matches!(&b.block, Block::Table { rows, .. } if rows.first().map(|row| row.len()).unwrap_or(0) >= 2)
+ });
+ if beaten {
+ continue;
+ }
+ }
+ kept.push(r);
+ }
for b in borderless {
- let overlaps = ruled
+ let overlaps = kept
.iter()
.any(|r| !(b.end <= r.start || b.start >= r.end));
if !overlaps {
- ruled.push(b);
+ kept.push(b);
}
}
- ruled.sort_by_key(|r| r.start);
- ruled
+ kept.sort_by_key(|r| r.start);
+ kept
}
/// Escape `|` and `\n` inside a markdown table cell so the pipe-table grammar
@@ -951,6 +1180,49 @@ mod tests {
assert_eq!(cells[0].text, "Hello world");
}
+ #[test]
+ fn absorbs_partial_header_line_above_body() {
+ // A header line with only two track-aligned cells sits above a clean
+ // 3-column body. It can't start the table on its own (fewer than
+ // TABLE_MIN_COLUMNS cells) but should be walked back in as the header.
+ let lines = vec![
+ line_with_spans(&[("Name", 50.0), ("Scores", 150.0)], 100.0, 10.0),
+ line_with_spans(&[("A", 50.0), ("1", 150.0), ("2", 250.0)], 115.0, 10.0),
+ line_with_spans(&[("B", 50.0), ("3", 150.0), ("4", 250.0)], 130.0, 10.0),
+ line_with_spans(&[("C", 50.0), ("5", 150.0), ("6", 250.0)], 145.0, 10.0),
+ ];
+ let runs = detect_tables(&lines);
+ assert_eq!(runs.len(), 1);
+ let run = &runs[0];
+ assert_eq!(run.start, 0, "header line should be absorbed into the run");
+ assert_eq!(run.end, 4);
+ match &run.block {
+ Block::Table { header, rows } => {
+ let header = header.as_ref().expect("header should be present");
+ assert_eq!(header, &vec!["Name".to_string(), "Scores".to_string(), String::new()]);
+ // All three body rows survive — the header came from above, so
+ // rows[0] is not consumed as a header.
+ assert_eq!(rows.len(), 3);
+ }
+ other => panic!("expected Block::Table, got {other:?}"),
+ }
+ }
+
+ #[test]
+ fn does_not_absorb_single_cell_title_above_body() {
+ // A one-cell title/caption above a table is NOT a header row and must
+ // not be absorbed.
+ let lines = vec![
+ line_with_spans(&[("Results", 50.0)], 100.0, 10.0),
+ line_with_spans(&[("A", 50.0), ("1", 150.0), ("2", 250.0)], 115.0, 10.0),
+ line_with_spans(&[("B", 50.0), ("3", 150.0), ("4", 250.0)], 130.0, 10.0),
+ line_with_spans(&[("C", 50.0), ("5", 150.0), ("6", 250.0)], 145.0, 10.0),
+ ];
+ let runs = detect_tables(&lines);
+ assert_eq!(runs.len(), 1);
+ assert_eq!(runs[0].start, 1, "single-cell title must stay out of the run");
+ }
+
#[test]
fn rejects_table_when_row_count_too_low() {
let lines = vec![line_with_spans(
diff --git a/crates/pdfium/src/struct_tree.rs b/crates/pdfium/src/struct_tree.rs
new file mode 100644
index 0000000..ce5f1fb
--- /dev/null
+++ b/crates/pdfium/src/struct_tree.rs
@@ -0,0 +1,224 @@
+//! Walk the PDF structure tree (tagged-PDF tree) for a page.
+//!
+//! Each struct element has:
+//! - a role (string: "H1", "Figure", "Table", ...)
+//! - zero or more marked content ids (mcids) that tie the element to
+//! page-content stream blocks
+//! - zero or more child elements
+//!
+//! For LiteParse's purposes we want a flat list of nodes with:
+//! - the role string
+//! - viewport-space bbox derived from joining page objects whose
+//! `FPDFPageObj_GetMarkedContentID` matches one of this node's mcids
+//! - the mcid list itself (so the layout pass can map text runs to nodes)
+
+use crate::ffi;
+use crate::page::{Page, ViewportTransform};
+use crate::types::RectF;
+
+/// One node in the structure tree, flattened for downstream use.
+#[derive(Debug, Clone)]
+pub struct StructNode {
+ /// Role string from `FPDF_StructElement_GetType` (e.g. "H1", "P", "Figure").
+ pub role: String,
+ /// Marked content ids attached to this element (and its non-struct child markers).
+ pub mcids: Vec<i32>,
+ /// Union bbox of page objects tagged with any of `mcids`, in viewport
+ /// coordinates (top-left origin, 72 DPI). `None` when none of the mcids
+ /// resolved to a bbox on the page.
+ pub bbox: Option<RectF>,
+ /// Optional alt text (set for Figure / Formula elements when present).
+ pub alt_text: Option<String>,
+}
+
+impl Page<'_> {
+ /// Walk this page's structure tree (tagged-PDF tree). Returns an empty
+ /// vec when the page is untagged or the document has no struct tree.
+ /// Nodes are returned in pre-order (parent before children).
+ pub fn struct_tree(&self, view_box: &RectF) -> Vec<StructNode> {
+ let tree = unsafe { ffi!(FPDF_StructTree_GetForPage(self.handle)) };
+ if tree.is_null() {
+ return Vec::new();
+ }
+
+ let mcid_bboxes = collect_mcid_bboxes(self, view_box);
+ let mut out = Vec::new();
+
+ let count = unsafe { ffi!(FPDF_StructTree_CountChildren(tree)) };
+ for i in 0..count {
+ let elem = unsafe { ffi!(FPDF_StructTree_GetChildAtIndex(tree, i)) };
+ if !elem.is_null() {
+ walk_element(elem, &mcid_bboxes, &mut out);
+ }
+ }
+
+ unsafe { ffi!(FPDF_StructTree_Close(tree)) };
+
+ out
+ }
+}
+
+/// Pre-scan all page objects on the page, building `mcid → union(bbox)` in
+/// viewport space. Each struct node then unions the bboxes for its own mcids.
+fn collect_mcid_bboxes(
+ page: &Page<'_>,
+ view_box: &RectF,
+) -> std::collections::HashMap<i32, RectF> {
+ let vp = page.viewport_transform(view_box);
+ let obj_count = unsafe { ffi!(FPDFPage_CountObjects(page.handle)) };
+ let mut map: std::collections::HashMap<i32, RectF> = std::collections::HashMap::new();
+
+ for i in 0..obj_count {
+ let obj = unsafe { ffi!(FPDFPage_GetObject(page.handle, i)) };
+ if obj.is_null() {
+ continue;
+ }
+ let mcid = unsafe { ffi!(FPDFPageObj_GetMarkedContentID(obj)) };
+ if mcid < 0 {
+ continue;
+ }
+ let bbox = page_object_bbox(obj, &vp);
+ if let Some(b) = bbox {
+ map.entry(mcid)
+ .and_modify(|cur| *cur = union_rect(cur, &b))
+ .or_insert(b);
+ }
+ }
+
+ map
+}
+
+fn page_object_bbox(obj: pdfium_sys::FPDF_PAGEOBJECT, vp: &ViewportTransform) -> Option<RectF> {
+ let mut left = 0.0f32;
+ let mut bottom = 0.0f32;
+ let mut right = 0.0f32;
+ let mut top = 0.0f32;
+ let ok = unsafe {
+ ffi!(FPDFPageObj_GetBounds(
+ obj,
+ &mut left,
+ &mut bottom,
+ &mut right,
+ &mut top
+ ))
+ };
+ if ok == 0 {
+ return None;
+ }
+ Some(vp.transform_bounds(&RectF {
+ left,
+ top,
+ right,
+ bottom,
+ }))
+}
+
+fn union_rect(a: &RectF, b: &RectF) -> RectF {
+ RectF {
+ left: a.left.min(b.left),
+ top: a.top.min(b.top),
+ right: a.right.max(b.right),
+ bottom: a.bottom.max(b.bottom),
+ }
+}
+
+fn walk_element(
+ elem: pdfium_sys::FPDF_STRUCTELEMENT,
+ mcid_bboxes: &std::collections::HashMap<i32, RectF>,
+ out: &mut Vec<StructNode>,
+) {
+ let role = read_element_type(elem);
+ let alt_text = read_alt_text(elem);
+
+ // Collect mcids: the multi-mcid getters + per-child marked-content-only children.
+ let mut mcids: Vec<i32> = Vec::new();
+ let n_mcids = unsafe { ffi!(FPDF_StructElement_GetMarkedContentIdCount(elem)) };
+ for i in 0..n_mcids {
+ let m = unsafe { ffi!(FPDF_StructElement_GetMarkedContentIdAtIndex(elem, i)) };
+ if m >= 0 {
+ mcids.push(m);
+ }
+ }
+ // Also legacy: single direct mcid getter (older tag-trees expose it here).
+ let single = unsafe { ffi!(FPDF_StructElement_GetMarkedContentID(elem)) };
+ if single >= 0 && !mcids.contains(&single) {
+ mcids.push(single);
+ }
+
+ let n_children = unsafe { ffi!(FPDF_StructElement_CountChildren(elem)) };
+ for i in 0..n_children {
+ // Non-struct children expose their mcid via GetChildMarkedContentID
+ // (returns -1 when the child is itself a struct element).
+ let child_mcid = unsafe { ffi!(FPDF_StructElement_GetChildMarkedContentID(elem, i)) };
+ if child_mcid >= 0 && !mcids.contains(&child_mcid) {
+ mcids.push(child_mcid);
+ }
+ }
+
+ let bbox = union_mcid_bboxes(&mcids, mcid_bboxes);
+ out.push(StructNode {
+ role,
+ mcids,
+ bbox,
+ alt_text,
+ });
+
+ for i in 0..n_children {
+ let child_elem = unsafe { ffi!(FPDF_StructElement_GetChildAtIndex(elem, i)) };
+ if !child_elem.is_null() {
+ walk_element(child_elem, mcid_bboxes, out);
+ }
+ }
+}
+
+fn union_mcid_bboxes(
+ mcids: &[i32],
+ mcid_bboxes: &std::collections::HashMap<i32, RectF>,
+) -> Option<RectF> {
+ let mut acc: Option<RectF> = None;
+ for m in mcids {
+ if let Some(b) = mcid_bboxes.get(m) {
+ acc = Some(match acc {
+ Some(a) => union_rect(&a, b),
+ None => *b,
+ });
+ }
+ }
+ acc
+}
+
+fn read_element_type(elem: pdfium_sys::FPDF_STRUCTELEMENT) -> String {
+ read_widestring(|buf, len| unsafe { ffi!(FPDF_StructElement_GetType(elem, buf, len)) })
+}
+
+fn read_alt_text(elem: pdfium_sys::FPDF_STRUCTELEMENT) -> Option<String> {
+ let s = read_widestring(|buf, len| unsafe { ffi!(FPDF_StructElement_GetAltText(elem, buf, len)) });
+ if s.is_empty() { None } else { Some(s) }
+}
+
+/// Read a PDFium UTF-16LE widestring out-param via the "call once for size,
+/// allocate, call again" pattern. `getter` is `(buf, buflen) -> bytes_written`.
+fn read_widestring<F>(getter: F) -> String
+where
+ F: Fn(*mut std::os::raw::c_void, std::os::raw::c_ulong) -> std::os::raw::c_ulong,
+{
+ let needed = getter(std::ptr::null_mut(), 0) as usize;
+ if needed < 2 {
+ return String::new();
+ }
+ let mut buf: Vec<u16> = vec![0; needed / 2];
+ let written = getter(
+ buf.as_mut_ptr() as *mut std::os::raw::c_void,
+ needed as std::os::raw::c_ulong,
+ ) as usize;
+ if written < 2 {
+ return String::new();
+ }
+ let chars = written / 2;
+ let end = if buf.get(chars - 1) == Some(&0) {
+ chars - 1
+ } else {
+ chars
+ };
+ String::from_utf16_lossy(&buf[..end])
+}