Skip to content

A fast, helpful, and open-source document parser

wip: split logic

Logan Markewichac3291a6ecc5
diff --git a/MARKDOWN_PLAN.md b/MARKDOWN_PLAN.md
index 5daa78e..a7a3677 100644
--- a/MARKDOWN_PLAN.md
+++ b/MARKDOWN_PLAN.md
@@ -252,7 +252,7 @@ Sensible defaults so users can just pass `-f markdown` and get good output. Deba
## Surface area
- `crates/liteparse/src/output/markdown.rs` — new.
-- `crates/liteparse/src/markdown_layout.rs` — new.
+- `crates/liteparse/src/markdown_layout/` — new module directory: `mod.rs` (public re-exports), `blocks.rs`, `inline.rs`, `tables.rs`, `headings.rs`, `lists.rs`, `paragraphs.rs`, `repetition.rs`, `hr.rs`, `classify.rs`.
- `crates/liteparse/src/types.rs` — add `Block`, `Span`, `TableBlock`.
- `crates/liteparse/src/config.rs` — add `OutputFormat::Markdown`, `MarkdownConfig`.
- `crates/liteparse/src/projection.rs` — expose `ProjectedLine` (additive).
diff --git a/MARKDOWN_PROGRESS.md b/MARKDOWN_PROGRESS.md
index cfd5932..73ea9d4 100644
--- a/MARKDOWN_PROGRESS.md
+++ b/MARKDOWN_PROGRESS.md
@@ -2,6 +2,8 @@
Tracks build-order progress against [MARKDOWN_PLAN.md](MARKDOWN_PLAN.md). Update as steps complete.
+> **2026-06-01 module split — `markdown_layout.rs` → `markdown_layout/`.** The single 3819-line file got broken up into a directory of focused submodules; pure move-only refactor, no logic changes. New layout: `mod.rs` (24 lines, re-exports the 5 external API symbols + `Block`), `blocks.rs` (347 — `Block`, `render_blocks`, `wrap_emphasis`, `paragraph_from_accum`), `classify.rs` (783 — `classify_page_with_filters`, `classify_page`, `is_rotated_line`, orchestration), `headings.rs` (526 — body-size histogram, heading map, struct/outline level resolution, bold-heading heuristics, caption rejection), `tables.rs` (1118 — borderless + ruled-grid table detection, `escape_table_cell`, `merge_table_runs`), `inline.rs` (319 — `SpanStyle`, emphasis rendering, list-item text), `paragraphs.rs` (214 — `ParaAccum`, `continues_paragraph`, `collapse_whitespace`), `repetition.rs` (170 — running header/footer detection), `hr.rs` (132 — `detect_horizontal_rules`), `lists.rs` (96 — `parse_list_marker`), `tests_common.rs` (202, `#[cfg(test)]` only — shared `line`/`page`/`styled_line` fixtures so per-module test modules don't duplicate them). Visibility: only `build_heading_map`, `classify_page_with_filters`, `compute_body_size`, `compute_header_footer_set`, `render_blocks`, and `Block` are `pub` (re-exported from `mod.rs`); everything else is `pub(super)` within the submodule tree. External consumer (`output/markdown.rs`) and `lib.rs` unchanged. 147 lib tests still pass; `cargo check -p liteparse` clean. Companion plan to also split `projection.rs` (4050 lines) deferred — too load-bearing while heading/layout work is still iterating.
+>
> **2026-05-30 rebuild note.** Branch had been cleaned during a branch switch — `markdown_layout.rs` / `output/markdown.rs` were sitting untracked, the supporting types (`ProjectedLine`, `Rect`, `Anchor::Floating`, `ParsedPage.projected_lines`) didn't exist, `OutputFormat::Markdown` wasn't on the enum, and nothing was wired into `lib.rs` / `output/mod.rs`. Restored from `stash@{0}` (which had the plumbing diff) + reimplemented step 2 (`build_projected_lines` in `projection.rs` derives per-line metadata from the projected items). Tests pass; smoke on `paper.pdf` emits headings + bold + page separators end-to-end via `cargo run -p liteparse -- parse <pdf> --format markdown`. Steps 3–6 in the table below are status-accurate again as of this note.
> **2026-05-31 XY-cut layout decomposition.** Replaced the flat x-density histogram column detector with a recursive modified XY-cut. Motivation: the old detector failed on layouts where the gutter is interrupted by a wide element (figures straddling both columns, full-width headings/banners) — every page-1-style case in the eval set, and the open carry-forward "column interleaving on page 1" from 2026-05-30.
@@ -66,11 +68,103 @@ Tracks build-order progress against [MARKDOWN_PLAN.md](MARKDOWN_PLAN.md). Update
| 7 | Tables — medium (column-alignment) + fallback | ✅ | `Block::Table { header, rows }` and `Block::GridFallback { lines }` added. `detect_tables` runs as a pre-pass over a page's `ProjectedLine`s before per-line classification. `split_cells` slices each line's `spans` into cells when adjacent-span gaps exceed `1.0 × dominant_font_size`. A run qualifies as a table when ≥3 cells per row repeat across ≥2 consecutive rows with column-track alignment (≤1 column allowed to drift outside ±6pt — handles indented sub-group row labels); if row-spacing CV > 0.5 the run is downgraded to `GridFallback`. First row promotes to header only when every cell is bold (synthesizes blank header otherwise). `table_rows_adjacent` deliberately doesn't gate on `region_path` because XY-cut sometimes puts an indented row in a different leaf than the rest of the table — column-alignment + y-gap are strong enough on their own. `looks_like_bold_heading` now rejects lines that look tabular (≥3 spans by `split_cells` OR ≥3 multi-space-separated tokens by text fallback) so multi-line table headers like "Model Method Single Hop Multi Hop ..." stop getting promoted to H3. Smoke on `paper.pdf` (results tables): 8 well-formed pipe tables emitted, "Model Method..." header line now lands as a bold paragraph instead of H3. **Known residuals**: (a) `MEMORYBANK` rows in academic results tables are at x=102.5 vs neighbor rows at x=70 — ALL column tracks shift by ~30pt so the 1-column-tolerance rule can't bridge them; tables still fragment into 2-4 row chunks around those rows. Fix needs ruled-grid detection from PDFium fork (step 9) — won't be solvable from text-only signals. (b) Short header rows like `Category Average` (2 cells) and the full-width `F1 BLEU-1 F1 BLEU-1...` (1 collapsed projection span, single-spaced text) still classify as H3 — text-only heuristics can't distinguish them from real one-line section headings without table-rect context. (c) Header absorption (walking back from a detected table to grab multi-line headers above) deliberately deferred — would need careful per-column concatenation against unaligned multi-line headers; better to wait for fork-driven cell-rect data. |
| 8 | Header/footer stripping (cross-page repetition) | ✅ | `compute_header_footer_set` collects normalized line text (lowercased, digit-runs collapsed to `#` so `Page 1 of 6`/`Page 2 of 6` match) appearing in the top 12% / bottom 12% page-height band on ≥ 50% of pages (floor of 2). Header and footer bands tracked separately so a heading that doubles as both running header on some pages and body title on others isn't stripped from the body. `format_markdown` builds the set once and passes it to `classify_page_with_filters`, which removes matching lines before table detection and per-line classification. Smoke-tested on `Anonymized rate card.pdf` — repeating `Page N of 6` chrome is gone from output. |
| 9 | Fork: paths exposed → strong (ruled grid) tables, HR detection, figure clustering | ✅ | **Paths exposed + HR + ruled-grid tables landed 2026-06-01; figure clustering deferred.** No actual PDFium fork modifications were required — every needed FFI symbol was already in `bindings.rs`; only the runtime-loaded `PdfiumBindings` struct in `crates/pdfium-sys/src/dynamic.rs` was missing the 10 path-related getters (`FPDFPageObj_GetMatrix`, `FPDFPath_CountSegments`, `FPDFPathSegment_*`, color/width getters, `FPDFPath_GetDrawMode`). Added all. **9a — pdfium wrapper:** new `PathObject` / `PathSegment` / `SegmentKind` in `crates/pdfium/src/page.rs`; `Page::path_objects(view_box)` iterates `FPDF_PAGEOBJ_PATH` objects, applies the object matrix → page → viewport so all segment points land in viewport coords for direct comparison against `ProjectedLine.bbox`. **9b — liteparse `GraphicPrimitive`:** added `GraphicPrimitive::{Stroke { x1,y1,x2,y2,color,width }, Rect { bbox, fill, stroke }}` to `types.rs` + `ParsedPage.graphics` (#[serde(skip)]). `extract.rs::extract_page_graphics` decomposes paths: filled paths → one `Rect` for the bbox (covers cell backgrounds / banner fills regardless of internal complexity); stroked paths → one `Stroke` per `MoveTo→LineTo` / `LineTo→LineTo` pair (plus the implicit closing stroke when a subpath's close flag is set). `BezierTo` segments advance the current point but don't emit a stroke (curves aren't useful for table/HR detection). **9c-1 — HR:** `detect_horizontal_rules(page)` filters strokes to `|y1-y2| ≤ 2pt`, length ≥ `0.3 × page_width`, stroke-width ≤ 2pt, with an underline-band filter that drops strokes within ±3pt of a text-line baseline whose x-extent matches the line. `Block::HorizontalRule` interleaves with text lines in `classify_page_with_filters` by y-coord via a peekable `hr_iter`; renders as `---`. 6 unit tests. **9c-2 — ruled-grid tables:** `detect_ruled_tables` extracts H/V segments from strokes + stroked Rect edges (so cell-rect-per-cell tables work too), clusters them into grid lines (`TABLE_GRID_CLUSTER_PT=2pt`), finds connected components via union-find on crossing pairs (`TABLE_CROSS_TOLERANCE_PT=3pt`), then per component: clusters distinct row/col coords, requires ≥2×2 cells, assigns text-line centroids to cells, rejects when empty-cell fraction > 30% or coverage > 95% of page (kills page-border rects), promotes the first row to header when all non-empty cells are `all_bold`. Runs *before* the existing borderless detector; `merge_table_runs` keeps both outputs but drops borderless runs whose range overlaps a ruled run (path geometry strictly beats text alignment). 5 unit tests. **9c-3 — figure clustering:** deferred to a later pass; limited value without image extraction (step 11) and the highest-leverage piece of figure detection (feeding figure rects into the XY-cut layout pass to fix the paper.pdf page-1 carry-forward) is now tracked as its own follow-up. **Test count:** 136 lib + 10 integration tests pass. **Smoke-test commands:** `cargo run -p liteparse -- parse <pdf> --format markdown`. Good test docs: `Anonymized rate card.pdf` (rate-card with ruled tables + HR-style dividers; should produce `---` and `|...|` table rows), any tagged invoice / financial statement / spreadsheet export. The `paper.pdf` results tables still have known residuals (MEMORYBANK row indent + short header rows) that the borderless detector handled; need to confirm the ruled detector doesn't regress those — if paper.pdf renders ruled tables incorrectly, the borderless fallback should still catch them via `merge_table_runs`. |
-| 10 | Fork: structure tree + outline → highest-priority headings | ⬜ | Requires PDFium fork changes. |
-| 11 | Image extraction modes (placeholder vs embed) | ⬜ | |
+| 10 | Fork: structure tree + outline → highest-priority headings | ✅ | **Landed 2026-06-01.** As with step 9, no actual PDFium fork edits were needed — all FFI symbols (`FPDFBookmark_*`, `FPDFAction_*`, `FPDFDest_*`, `FPDF_StructTree_*`, `FPDF_StructElement_*`) were already in `bindings.rs`; wired the 21 missing function pointers into `PdfiumBindings` in `crates/pdfium-sys/src/dynamic.rs`. **10a — outline wrapper:** new `OutlineEntry` + `Document::outline()` in `crates/pdfium/src/document.rs` walks the bookmark tree depth-first, resolves each entry's destination (direct `FPDFBookmark_GetDest` falling back through `FPDFBookmark_GetAction` → `FPDFAction_GetDest`), then pulls the destination's page index + y (PDF user space) via `FPDFDest_GetDestPageIndex` + `FPDFDest_GetLocationInPage`. Titles decoded from UTF-16LE. **10b — struct-tree wrapper:** new `crates/pdfium/src/struct_tree.rs::Page::struct_tree(view_box)` returns `Vec<StructNode { role, mcids, bbox, alt_text }>` in pre-order. Implementation pre-scans page objects once into an `mcid → union(bbox)` map (in viewport coords via the page's `ViewportTransform`), then walks the tree calling both `FPDF_StructElement_GetMarkedContentIdCount`/`AtIndex` (modern multi-mcid form) and the legacy `GetMarkedContentID` + `GetChildMarkedContentID` getters so older tagged PDFs aren't missed. Each node's bbox is the union of bboxes for its tagged mcids. **10c — liteparse plumbing:** `OutlineTarget` + `StructNode` types added to `types.rs` (`#[serde(skip)]` so JSON/text output stays byte-identical). `LitePage.struct_nodes` populated in `extract::extract_page_struct_nodes`; threaded through `to_parsed_pages` onto `ParsedPage.struct_nodes`. New `extract::extract_outline(doc)` returns `Vec<OutlineTarget>`. `ParseResult` gains a `pub outline: Vec<OutlineTarget>` field. `parse_input` now opens the document once, pulls outline + pages + OCR-rendered pages inside a scope so `Document` is dropped before any `.await` (PDFium handle isn't `Send`). **10d — markdown classifier integration:** added `struct_heading_level(line, struct_nodes)` and `outline_heading_level(line, page_height, outline, text)` in `markdown_layout.rs`. Heading-promotion priority chain at the per-line classifier: struct-tree H1–H6 (matched via line `mcid` → node role) → outline (y_view = `page_height - y_pdf` within `1.5 × line_height`, normalized lowercased prefix-match either direction) → existing font-size histogram. `classify_page_with_filters` gains a 4th `outline: &[OutlineTarget]` param; `format_markdown` gains a 2nd `outline: &[OutlineTarget]` param and filters per-page before dispatching. **Test count:** 147 lib + 10 integration tests pass; `cargo check -p liteparse-python -p liteparse-wasm` clean. **Smoke on `paper.pdf`:** unchanged — arXiv preprints aren't tagged and don't ship an outline, so the new code is dormant (this is the expected non-regression on untagged docs). **Needs a tagged PDF for real validation:** good candidates are most government/legal forms, accessible academic publisher exports (ACM/IEEE/Springer), modern Word/InDesign exports with "Tagged PDF" enabled, and most invoices generated by enterprise tools. Smoke-test command: `cargo run -p liteparse -- parse <tagged.pdf> --format markdown`. Expected wins: section headings hit the right level even when the doc has weird font-size choices; bookmarks-driven headings pick up the right level on docs that have a TOC but no font-size hierarchy. |
+| 11 | Image extraction modes (placeholder vs embed) | ✅ | **Stages 11a + 11b landed 2026-06-01.** **Stage 11a (placeholder mode):** Added `ImageMode` enum (`Off | Placeholder | Embed`, default `Placeholder`) to `config.rs` as a flat `LiteParseConfig.image_mode` field (skipped `MarkdownConfig` wrapper — easier for the language bindings, can add later if we accumulate more knobs). New `ImageRef { id, bbox, obj_index }` in `types.rs`, threaded `LitePage.image_refs` → `ParsedPage.image_refs` (#[serde(skip)] both). `extract::extract_page_image_refs(page, page_number)` reuses `Page::image_bounds(min=25pt, max_coverage=0.9)` from the existing FFI surface — no PDFium changes needed. IDs are page-scoped (`p{n}_{i}`) so they're stable across runs. New `Block::Figure { id, bbox }` rendered as `![](image_{id}.png)`. Classifier interleaves figures with text lines by y position (parallel to the existing `hr_iter` machinery via a new `figure_iter` + `emit_figures_before` helper, suppressed inside table y-extents). `classify_page_with_filters` gains a 5th `image_mode` param; `format_markdown` gains a 3rd. CLI: `--image-mode off|placeholder|embed` on `parse`. **Stage 11b (embed mode + bytes sidecar):** added `ExtractedImage { id, page, bbox, format, bytes }` to `types.rs`; `ParseResult.images: Vec<ExtractedImage>` (populated only when `ImageMode::Embed`, empty otherwise — placeholder mode pays no render cost). New `extract::render_page_images` reuses `Page::render_image_object` + the existing `render::encode_png` (made `pub(crate)`) to encode each image as RGBA PNG. `extract_pages_and_images` is the new core entry point; `extract_pages_from_document` is now a one-line wrapper passing `render_images=false` for callers that don't need bytes. CLI gains `--image-output-dir <dir>` which, when set with `--image-mode embed`, writes `image_{id}.png` next to the markdown output. **Bindings:** all three threaded through — pyo3 (`PyExtractedImage` with `bytes` property exposing `PyBytes`, `images` on `PyParseResult`), napi (`JsExtractedImage` with napi `Buffer` for bytes, `images` on `JsParseResult`, `image_mode` added to `JsLiteParseConfig`), wasm (`JsExtractedImage` in the result; bytes serialize as `number[]` for now, callers can wrap with `new Uint8Array(image.bytes)`). Python wrapper `packages/python/liteparse/` updated with matching `ExtractedImage` dataclass + `images` field on `ParseResult` + `image_mode` constructor arg. **Test count:** 147 lib + 10 integration tests still pass. **Smoke on `SMC2486.pdf` page 2** (the only of our local PDFs that surfaces an image object — `paper.pdf`'s figure is a vector cluster, not a raster object): `![](image_p2_0.png)` lands above the `Status Accepted` heading in reading order; `--image-mode off` drops it cleanly. **Stage 11b smoke-tested:** `cargo run -p liteparse -- parse SMC2486.pdf --format markdown --target-pages 2 --image-mode embed --image-output-dir /tmp/lp-img --quiet` emits `![](image_p2_0.png)` in markdown and writes the 22.4kB PNG to disk. Python: `LiteParse(image_mode="embed").parse(...).images` returns `[ExtractedImage(id='p2_0', page=2, format='png', bytes=22420)]` end-to-end. **Open items:** `--image-mode` flag on `BatchParseCommand` not added; should be a trivial follow-up. WASM bytes are a `number[]` array rather than a proper `Uint8Array` — works but inefficient; can upgrade by hand-rolling `JsParseResult` construction via `Reflect` + `js_sys::Uint8Array` later. |
Steps 1–8 = usable v1. Steps 9–11 = quality bumps for specific doc classes.
+> **2026-06-01 step 10 validation pass — police.pdf (Broomfield budget) + expected-dataset/data/who_indicators.pdf.** Wrote `crates/liteparse/examples/inspect_tagged.rs` to summarize outline + struct-tree presence and cross-check outline-title coverage against rendered markdown headings without having to read the whole doc (`cargo run -q -p liteparse --example inspect_tagged -- <pdf>`). First run on `police.pdf` exposed two issues:
+> - **Y-coord-out-of-page bookmarks.** Every entry came back as `y_pdf=866` on letter-size 792pt pages → `y_view = -74` → never within tolerance of any line. Either the creator's "top of page" dest type encodes y outside the MediaBox, or pages have a non-standard MediaBox; either way it's common enough to design around. Fix in `outline_heading_level`: when `y_view < 0` OR `y_view > page_height`, drop the spatial check and accept any line on the page that prefix-matches the title. Coverage 0/78 → 77/78 on `police.pdf`.
+> - **Short titles latching onto long paragraphs.** Once spatial-check fallback was in, "Nutrition" (a 9-char outline title) prefix-matched a 144-char paragraph in WHO that happened to start with "Nutrition,". Fix: require the matched line to be heading-shaped — line char-count ≤ `max(title*3, 120)` AND fewer than 2 sentence-breaks (`. `). Coverage 143/156 on WHO with no observed false-positive headings.
+> - **`struct_tree`: 0/457 pages tagged on `police.pdf`, 0/164 on `who_indicators.pdf`.** Neither doc actually ships a structure tree despite being well-formed PDFs with bookmarks. Validates that the outline source is doing the work; struct-tree source is dormant on these. Need a deliberately-tagged PDF (Word/InDesign export with "Tagged PDF" enabled) to exercise the H1–H6 mcid path end-to-end.
+> - **Residual misses on WHO are bookmarks with no body heading.** The 13 remaining unmatched WHO entries ("Morbidity", "Mortality by cause", "Environmental risk factors", "Noncommunicable diseases", "Injuries/harmful traditional practices") are *category labels* that appear in the TOC but not as a rendered line in the body — their bookmarks point to the first sub-indicator's page (which we do match). Not a heuristic failure.
+> - **Net behavior on `paper.pdf`:** unchanged (no outline / no struct tree) — verifies the change is dormant when neither source is present.
+>
+> **2026-06-01 step 10 — outline + structure tree.** Two new high-priority heading sources, additive to the existing font-size histogram so untagged PDFs without an outline (like `paper.pdf`) get bit-identical output. **Outline source**: walks the bookmark tree via the new `Document::outline()` wrapper; per-page entries are matched to projected lines by `|y_view - line.y| ≤ 1.5 × line_height` with a lowercased+collapsed prefix-match either direction (handles "Section 1: Foo" outline vs "Foo" rendered line and vice versa). **Struct-tree source**: per-page `Page::struct_tree(view_box)` returns flattened nodes carrying their role, mcid list, and viewport-space union bbox (derived from a one-pass scan of page-object mcids). Lines whose `mcid` lands in a struct node with role `H1..H6` are promoted at that level — outranks both outline and font-size map. Figure/Table struct nodes are surfaced but not yet consumed (existing detectors handle those passes already). One subtle gotcha to track: `parse_input` now opens the PDFium document once outside the OCR await scope so the `!Send` handle is dropped before any future-suspension point.
+
+> **2026-06-01 opendataloader-bench accuracy pass.** Switched the bench adapter `opendataloader-bench/src/pdf_parser_liteparse.py` from text mode → `output_format="markdown", image_mode="off", ocr_enabled=False` (added matching `image_mode` parameter to `packages/python/liteparse/parser.py` and the `liteparse-python` pyo3 binding so it threads through). Two follow-on fixes after smoke-reading the worst 20 docs by overall score:
+>
+> - **Bold-style paragraph break in `continues_paragraph`** (`markdown_layout.rs`). PDFs that put section headings in a bold font variant of the body font (Brill-Bold over Brill-Roman, etc.) were losing the heading because no font-size delta separated it from the body. The classifier bold-wrapped the line then merged it into the next paragraph. Fix: when both prev and cur have a uniform style (`line_uniform_style` returns Some), a `bold` mismatch is a paragraph break. Six lines. Bumped MHS 0.320 → 0.487 on the bench corpus.
+> - **Multi-line-cell wrap merge in `try_detect_table`** (`markdown_layout.rs`). Borderless tables where one column wraps onto a second baseline (e.g. "Physical or / digital" while neighbour columns stay single-line) were getting fragmented because the second baseline came back as a "partial row" with too few cells. Fix: when `cells.len() < column_count`, if every cell aligns with a known track within `TABLE_TRACK_TOLERANCE_PT` AND vertical centroid distance ≤ 1.5 × line_height, append each wrap cell's text into the matching track of the previous row rather than closing the table run. Roughly 30 lines, including comment. Bumped TEDS 0.323 → 0.384.
+>
+> **Cumulative bench result vs original (text-mode, public 1.2.1):**
+>
+> | Metric | Before | After 11a + fixes | Δ |
+> |---|---|---|---|
+> | NID | 0.866 | 0.882 | +0.016 |
+> | TEDS | 0.000 | **0.384** | +0.384 |
+> | MHS | 0.000 | **0.488** | +0.488 |
+>
+> Two metrics went from 0 → mid-pack with ~80 lines of code. liteparse jumps from last place (0.576 overall) to roughly the docling/marker neighbourhood on the OpenDataLoader leaderboard.
+>
+> **Surprise: Tesseract OCR _hurts_ on this corpus.** Re-ran with `ocr_enabled=True` and TEDS dropped to 0.369, MHS to 0.482, NID barely moved (+0.003), at 160× the per-page latency (0.81s vs 0.005s). Strongly suggests a real bug in `ocr_merge` — OCR-text likely duplicating native tokens or unbalancing the spatial merge — worth a focused investigation later. For now `ocr_enabled=False` is the right bench default and arguably the right production default for clean PDFs.
+>
+> **2026-06-01 numbered bold-section headings + caption rejection.** Two targeted heuristic fixes in `markdown_layout.rs` after a second bench pass identified MHS=0 patterns:
+>
+> - **`is_caption_line`** detects `Figure N`, `Fig. N`, `Table N`, `Tab. N`, `Equation N`, `Scheme`, `Chart`, `Plate`, `Photo`, `Algorithm`, `Listing` prefixes followed by a digit / roman numeral / `(N`. Applied in three places: `build_heading_map` (don't let caption sizes inflate the heading map), the font-size promotion site (don't promote captions even when their font matches a heading size), and `looks_like_bold_heading` (don't promote bold body-sized captions). Fixes doc 01030000000027 which previously emitted `# Figure 7. / # Figure 8. / # Figure 9. / # Figures 7-9 show…` as H1 headings.
+> - **`looks_like_numbered_bold_heading`** runs inside the list-item branch right after `parse_list_marker` matches an ordered marker. Conditions: post-marker body ≤ 80 chars, no double-sentence punctuation, **every non-marker span in `line.spans` must pass `projection::is_bold_item` and fail `projection::is_mono_item`**, alpha-ratio ≥ 0.5, and a paragraph-break gap above (gap_below deliberately *not* required — the line is often followed by another bold body line such as a bold sub-heading or multi-line title continuation). On match, the line is promoted to a heading at `heading_map.len()+1` (typically H1 when no font-size hierarchy exists, capped at H6) instead of emitting as an ordered list item. Fixes doc 01030000000062 (`1. Shipping as a vector for marine IAS` → `# 1. …`) and doc 01030000000027's `6. Conclusion`.
+> - **Span-walking trick**: marker characters often arrive inside the same span as the body text (e.g. one span `"1. Shipping as a vector for marine IAS"`), not as a separate marker span. The check classifies a span as "marker" only when *all* its non-whitespace chars are in `[0-9.()]` — so when the marker is embedded, the whole span goes through the bold check, which correctly returns true. This is why `line.all_bold == true` is the typical hit case but the per-span loop is what handles the rest.
+>
+> **Bench result on opendataloader-bench/ (200 docs):**
+>
+> | Metric | Pre-fix (post 11a) | Post-fix | Δ |
+> |---|---|---|---|
+> | NID | 0.8820 | 0.8819 | -0.0001 (noise) |
+> | TEDS | 0.3842 | 0.3842 | 0.0 |
+> | MHS | 0.4877 | 0.5093 | **+0.0216** |
+> | Overall | 0.7451 | 0.7508 | **+0.0057** |
+>
+> MHS=0 doc count: 17 → 17 (some new misses appeared because docs that previously had non-zero MHS lost some captions-as-headings hits — net positive though). Doc 01030000000062 went 0 → 0.495 MHS.
+>
+> **Remaining MHS=0 docs are now dominated by column-interleaving on 2-column layouts** — e.g. doc 01030000000038's heading `6.2. Expectations for Re-Hiring Employees` projects onto the same line as body text from the adjacent column (`they had no plans to re-hire and another 36% said`), so neither the bold-heading nor the numbered-bold-heading rule sees a clean candidate. Same root cause as the long-tracked `paper.pdf` page 1 banner/figure-straddle case. Also surfaces in docs 029, 038, 039, 077, 189 — all decimal section numbers (`5.`, `6.2.`, `9.5.`, `1.5.`, `4.3.2`) on 2-column academic/policy/business reports. Fixing these needs an XY-cut / column-detection pass, not a heading-detection knob.
+>
+> **Build orientation note:** the bench's `liteparse` Python package is installed from a built wheel pointing at `packages/python/liteparse/_liteparse.cpython-314-darwin.so` via a `.pth` file — `maturin develop` against `crates/liteparse-python/Cargo.toml` builds a *different* `_liteparse` package (under site-packages/`_liteparse/`) that the wrapper doesn't import. To get a rebuilt `.so` to take effect, run `maturin develop --release` from `packages/python/` directly. This wasted ~30 minutes the first iteration when the bench kept reporting unchanged scores against stale code.
+>
+> **2026-06-01 column-start histogram fallback in XY-cut.** Targets the column-interleaving failure mode behind most remaining MHS=0 docs (decimal section heads on 2-column policy/academic/business reports — docs 029, 038, 039, 077, 189, …). The density-based V-cut search inside `xy_find_best_cut` fails on these layouts because a column-spanning element (wide heading, figure, table) fills the gutter at one y-band, inflating the density at the gutter's x and erasing the valley. Added `xy_find_column_cut` as a *fallback* consulted only when **both** `xy_find_best_cut` calls (H and V) return `None`.
+>
+> **Algorithm.** Histogram every item's left-edge x with 5pt buckets across the full item span (`min_x..max_x`, not bbox-relative — see the bbox aside below). Cluster contiguous occupied buckets (allowing ≤2-bucket gaps for intra-column drift) into peaks; each peak carries its summed item count. Then apply four conservative gates *in this order*:
+> - `XY_COLUMN_MIN_LINES_PER_PEAK = 10` — drop peaks with fewer than 10 items. Indented paragraph-starts and footnote markers produce 3–6 item peaks; real column-left edges have 15+ items in a typical academic page.
+> - **Exactly 2 strong peaks remaining.** 3+ peaks means tabular structure (one peak per cell column); we bail rather than slice a table.
+> - `XY_COLUMN_PEAK_DOMINANCE = 0.55` — the two peaks together must hold ≥55% of all items. Rules out scattered-x prose layouts (lists, lab manuals, multi-indent docs) where the two tallest peaks are tall only relative to a flat tail.
+> - `XY_COLUMN_PEAK_BALANCE_RATIO = 0.4` — smaller-peak / larger-peak count ratio ≥ 0.4. Filters out "single column with one off-x bullet stack" cases (90/10 split → false-positive column cut).
+>
+> Plus the existing geometric constraint: gap between the two peaks must be ≥ `XY_COLUMN_MIN_GAP_FRACTION = 0.25` × total item-span width. Cut position is the midpoint between the two peaks; returned `CutCandidate` carries `score = 1e9` so it beats every density-derived cut but still loses to `xy_find_banner_cut`'s `f32::INFINITY`.
+>
+> **Why histogram items, not lines.** The natural first cut was: y-band items into lines, take each line's `x_min`, histogram those. That fails on the *exact* layouts we're trying to fix — items from the left and right columns at the same baseline fold into one "line" whose `x_min` is the left-column edge and whose `x_max` reaches the right-column tail. The right column's left edge is hidden. Switching to per-item left edges lets each column's items contribute to their own peak directly.
+>
+> **xy_root_bbox aside.** Doc 029 (and a handful of book-spread PDFs) places content at negative user-space x (items at x=-435 with the page reported as 510pt wide). `xy_root_bbox` clamps `min_x.max(0.0)` and caps width to `page_width`, so on those pages half the items live *outside* the returned bbox and `xy_split_bbox` produces a zero-width left child after any V-cut into the negative half. Tried removing the clamp; that fixed doc 029 visibly (clean column reading order, both `# 5. The dynamics` and `# 6. Modeling the dynamics` headings emit) but regressed the bench across the board (NID 0.882→0.857, TEDS 0.384→0.296, MHS roughly flat — net -0.025 overall). Reverted. The column-start histogram still works on doc 029 *despite* the clamped bbox because the histogram operates in item-space and the cut position is in item-coords; downstream item partitioning by centroid still routes items correctly even when the bbox doesn't include the cut. **Status: clamped bbox stays. Future work would need a per-page "items extend past MediaBox" detector that conditionally widens the bbox without changing it for normal docs.**
+>
+> **Gating evolution (what didn't work).**
+> - First version fired whenever V-cut returned `None`. Hijacked tables and spreadsheets where density H-cut would have correctly emitted a row split. Regressed TEDS heavily (0.384→0.266).
+> - First peak filter was `min_lines_per_peak=4`, which kept tabular peaks alive. Bumped to 10.
+> - First version allowed any number of peaks, picking the widest gap. Tables with 5+ peaks (one per cell column) routinely picked a column-gap that sliced rows in half. Restricted to exactly 2 strong peaks.
+> - First version had no dominance or balance gates. Many single-column docs with one footnote stack or one block-quote indent passed peak filtering. Added the 55% dominance + 0.4 balance gates.
+>
+> **Bench result on opendataloader-bench/ (200 docs):**
+>
+> | Metric | Pre-fix (post caption + numbered-bold) | Post-fix | Δ |
+> |---|---|---|---|
+> | NID | 0.8820 | **0.8822** | +0.0002 |
+> | TEDS | 0.3842 | **0.3862** | +0.0020 |
+> | MHS | 0.5093 | **0.5146** | +0.0053 |
+> | Overall | 0.7508 | **0.7516** | +0.0008 |
+>
+> **Doc-level wins:** doc 01030000000029 (MHS 0 → 0.278) is the most visible — `# 4. Entropy`, `# 5. The dynamics`, `# 6. Modeling the dynamics`, `## Definition 1`, `## Principle 1` all now emit correctly because the body is properly column-split before line-banding.
+>
+> **Doc-level losses:** doc 01030000000122 (lab manual with sparse bullets in two columns) NID 0.525 → 0.471. Doc 122's histogram does show two roughly-balanced peaks at x=229 and x=457 with 92/17 items — technically passes the gates because the 92 includes a long chunk of straight-through-the-page paragraphs that all happen to start at the same x — but the doc is structurally a single column of mixed bullet/figure/table content where slicing into a "right column" creates phantom reading-order jumps. Not worth tighter gates that would also lose doc 029.
+>
+> **Carry-forwards still open after this pass:**
+> - **Approach #2 (mask spanning items from density projection).** Before computing the V-cut density valley, drop items whose width ≥ ~70% of the region width — those are headings/figures/tables that span columns and are the reason the density valley search fails in the first place. Symmetric to the figure-obstacle code added 2026-06-01 (which masks figures along *H-cuts*). Should catch the 14 still-MHS=0 docs whose layout has a clean gutter once spanning elements are excluded.
+> - **Approach #3 (split-at-big-gap inside `build_projected_lines`).** Even if XY-cut fails, the per-leaf y-banding step could split a band whose items have a horizontal gap ≥ ~6–8 × median glyph advance AND ≥ ~30pt absolute. This is what `pdftotext` does. Riskier — justified prose has wide inter-word gaps, especially near inline figures — so should be tightly gated and only as a last resort if #2 doesn't close the remainder.
+> - **`xy_root_bbox` negative-x handling.** Book-spread layouts where items have x<0 break bbox-relative reasoning everywhere downstream. A conditional "items extend past MediaBox" detector that widens the bbox only for those pages would unlock the rest of doc 029 and likely a handful of other book-interior PDFs.
+>
+> **Still-open levers identified, ranked by leverage:**
+> - Multi-row table headers (16 docs, TEDS=0): patterns like `Source / Year / Description` stacked across 3 baselines where the wrap-merge fix only bridges 2.
+> - Same-font headings with no font/size signal (20 docs, MHS<0.1): no detectable text-only signal short of y-clearance heuristics — risky for false positives.
+> - OCR regression in `ocr_merge` — not a markdown problem per se but blocks any real OCR-on benchmark.
+>
+> **Bench plumbing landed:** `opendataloader-bench/src/pdf_parser_liteparse_placeholder.py` (image_mode=placeholder variant) and `pdf_parser_liteparse_ocr.py` (ocr_enabled=True variant), both registered in `engine_registry.py`. `liteparse-placeholder` confirms image placeholders cost ~0.003 NID — negligible vs the structural gains, so `Placeholder` is the right default.
+
## Carried-forward items
Things deferred from earlier steps that need to be revisited:
diff --git a/crates/liteparse-napi/src/types.rs b/crates/liteparse-napi/src/types.rs
index 130b633..d8844c7 100644
--- a/crates/liteparse-napi/src/types.rs
+++ b/crates/liteparse-napi/src/types.rs
@@ -1,6 +1,6 @@
use napi_derive::napi;
-use liteparse::config::{LiteParseConfig, OutputFormat};
+use liteparse::config::{ImageMode, LiteParseConfig, OutputFormat};
use liteparse::parser::ParseResult;
use liteparse::types::{ParsedPage, TextItem};
@@ -35,6 +35,10 @@ pub struct JsLiteParseConfig {
pub quiet: Option<bool>,
/// Number of concurrent OCR workers (default: CPU cores - 1).
pub num_workers: Option<u32>,
+ /// How to surface raster images in markdown output: "off", "placeholder"
+ /// (default — emits `![](image_pN_K.png)` references with no bytes), or
+ /// "embed" (also returns each image's PNG bytes on `images`).
+ pub image_mode: Option<String>,
}
impl JsLiteParseConfig {
@@ -80,6 +84,13 @@ impl JsLiteParseConfig {
if let Some(v) = self.num_workers {
cfg.num_workers = v as usize;
}
+ if let Some(v) = self.image_mode {
+ cfg.image_mode = match v.as_str() {
+ "off" | "none" => ImageMode::Off,
+ "embed" => ImageMode::Embed,
+ _ => ImageMode::Placeholder,
+ };
+ }
cfg
}
@@ -101,6 +112,11 @@ impl JsLiteParseConfig {
password: cfg.password.clone(),
quiet: Some(cfg.quiet),
num_workers: Some(cfg.num_workers as u32),
+ image_mode: Some(match cfg.image_mode {
+ ImageMode::Off => "off".to_string(),
+ ImageMode::Placeholder => "placeholder".to_string(),
+ ImageMode::Embed => "embed".to_string(),
+ }),
}
}
}
@@ -186,6 +202,16 @@ impl JsParsedPage {
pub struct JsParseResult {
pub pages: Vec<JsParsedPage>,
pub text: String,
+ pub images: Vec<JsExtractedImage>,
+}
+
+#[napi(object)]
+#[derive(Clone)]
+pub struct JsExtractedImage {
+ pub id: String,
+ pub page: u32,
+ pub format: String,
+ pub bytes: napi::bindgen_prelude::Buffer,
}
// ---------------------------------------------------------------------------
@@ -206,6 +232,16 @@ impl JsParseResult {
Self {
pages: result.pages.iter().map(JsParsedPage::from_rust).collect(),
text: result.text.clone(),
+ images: result
+ .images
+ .iter()
+ .map(|img| JsExtractedImage {
+ id: img.id.clone(),
+ page: img.page,
+ format: img.format.clone(),
+ bytes: img.bytes.clone().into(),
+ })
+ .collect(),
}
}
}
diff --git a/crates/liteparse-python/src/lib.rs b/crates/liteparse-python/src/lib.rs
index 3f131ca..246b490 100644
--- a/crates/liteparse-python/src/lib.rs
+++ b/crates/liteparse-python/src/lib.rs
@@ -1,7 +1,7 @@
use pyo3::prelude::*;
use pyo3::types::PyBytes;
-use liteparse::config::{LiteParseConfig, OutputFormat};
+use liteparse::config::{ImageMode, LiteParseConfig, OutputFormat};
use liteparse::types::PdfInput;
mod cli;
@@ -121,6 +121,8 @@ struct PyParseResult {
pages: Vec<PyParsedPage>,
#[pyo3(get)]
text: String,
+ #[pyo3(get)]
+ images: Vec<PyExtractedImage>,
}
#[pymethods]
@@ -136,9 +138,10 @@ impl PyParseResult {
fn __repr__(&self) -> String {
format!(
- "ParseResult(pages={}, text_len={})",
+ "ParseResult(pages={}, text_len={}, images={})",
self.pages.len(),
- self.text.len()
+ self.text.len(),
+ self.images.len()
)
}
}
@@ -152,6 +155,52 @@ impl PyParseResult {
.map(PyParsedPage::from_rust)
.collect(),
text: result.text,
+ images: result
+ .images
+ .into_iter()
+ .map(PyExtractedImage::from_rust)
+ .collect(),
+ }
+ }
+}
+
+#[pyclass(frozen, from_py_object)]
+#[derive(Clone)]
+struct PyExtractedImage {
+ #[pyo3(get)]
+ id: String,
+ #[pyo3(get)]
+ page: u32,
+ #[pyo3(get)]
+ format: String,
+ bytes_buffer: Vec<u8>,
+}
+
+#[pymethods]
+impl PyExtractedImage {
+ #[getter]
+ fn bytes<'py>(&self, py: Python<'py>) -> Bound<'py, PyBytes> {
+ PyBytes::new(py, &self.bytes_buffer)
+ }
+
+ fn __repr__(&self) -> String {
+ format!(
+ "ExtractedImage(id='{}', page={}, format='{}', bytes_len={})",
+ self.id,
+ self.page,
+ self.format,
+ self.bytes_buffer.len()
+ )
+ }
+}
+
+impl PyExtractedImage {
+ fn from_rust(img: liteparse::types::ExtractedImage) -> Self {
+ Self {
+ id: img.id,
+ page: img.page,
+ format: img.format,
+ bytes_buffer: img.bytes,
}
}
}
@@ -277,6 +326,7 @@ impl LiteParse {
password = None,
quiet = None,
num_workers = None,
+ image_mode = None,
))]
fn new(
ocr_language: Option<String>,
@@ -291,6 +341,7 @@ impl LiteParse {
password: Option<String>,
quiet: Option<bool>,
num_workers: Option<usize>,
+ image_mode: Option<String>,
) -> PyResult<Self> {
let mut cfg = LiteParseConfig::default();
if let Some(v) = ocr_language {
@@ -333,6 +384,13 @@ impl LiteParse {
if let Some(v) = num_workers {
cfg.num_workers = v;
}
+ if let Some(v) = image_mode {
+ cfg.image_mode = match v.as_str() {
+ "off" | "none" => ImageMode::Off,
+ "embed" => ImageMode::Embed,
+ _ => ImageMode::Placeholder,
+ };
+ }
let inner = liteparse::parser::LiteParse::new(cfg.clone());
let runtime = tokio::runtime::Runtime::new()
@@ -436,6 +494,7 @@ fn _liteparse(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_class::<LiteParse>()?;
m.add_class::<PyLiteParseConfig>()?;
m.add_class::<PyParseResult>()?;
+ m.add_class::<PyExtractedImage>()?;
m.add_class::<PyParsedPage>()?;
m.add_class::<PyTextItem>()?;
m.add_class::<PyScreenshotResult>()?;
diff --git a/crates/liteparse-wasm/src/lib.rs b/crates/liteparse-wasm/src/lib.rs
index dbb1812..9381e4d 100644
--- a/crates/liteparse-wasm/src/lib.rs
+++ b/crates/liteparse-wasm/src/lib.rs
@@ -156,6 +156,19 @@ struct JsParsedPage<'a> {
struct JsParseResult<'a> {
pages: Vec<JsParsedPage<'a>>,
text: &'a str,
+ images: Vec<JsExtractedImage<'a>>,
+}
+
+#[derive(Serialize)]
+#[serde(rename_all = "camelCase")]
+struct JsExtractedImage<'a> {
+ id: &'a str,
+ page: u32,
+ format: &'a str,
+ /// Serialized as a JS `number[]`. Callers that want a Uint8Array can
+ /// wrap with `new Uint8Array(image.bytes)`. (Could be upgraded to a real
+ /// Uint8Array later by switching to a hand-rolled to_value path.)
+ bytes: &'a [u8],
}
// ---------------------------------------------------------------------------
@@ -332,9 +345,20 @@ impl LiteParse {
})
.collect();
+ let js_images: Vec<JsExtractedImage> = result
+ .images
+ .iter()
+ .map(|img| JsExtractedImage {
+ id: &img.id,
+ page: img.page,
+ format: &img.format,
+ bytes: &img.bytes,
+ })
+ .collect();
let js_result = JsParseResult {
pages: js_pages,
text: &result.text,
+ images: js_images,
};
serde_wasm_bindgen::to_value(&js_result)
diff --git a/crates/liteparse/src/config.rs b/crates/liteparse/src/config.rs
index ca1ce0a..21d0a07 100644
--- a/crates/liteparse/src/config.rs
+++ b/crates/liteparse/src/config.rs
@@ -27,6 +27,28 @@ pub struct LiteParseConfig {
pub quiet: bool,
/// Number of concurrent OCR workers. Defaults to (number of CPU cores - 1), minimum 1.
pub num_workers: usize,
+ /// Controls how raster images are surfaced in markdown output. Has no
+ /// effect on JSON / text outputs.
+ pub image_mode: ImageMode,
+}
+
+/// Image handling for the markdown emitter.
+///
+/// * `Off` — strip image references entirely.
+/// * `Placeholder` (default) — emit `![](image_pN_K.png)` references in
+/// reading order at each image's y position, but do **not** extract or
+/// return pixel bytes. Keeps response size small while letting the LLM see
+/// where figures live in the document.
+/// * `Embed` — same references, plus bytes returned via `ParseResult.images`.
+/// Opt-in because pixel bytes can dwarf the text payload on image-heavy
+/// PDFs. (Bytes plumbing lands in stage 11b — current variant is parsed but
+/// behaves like `Placeholder` until then.)
+#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
+#[serde(rename_all = "lowercase")]
+pub enum ImageMode {
+ Off,
+ Placeholder,
+ Embed,
}
/// Supported output formats.
@@ -53,6 +75,7 @@ impl Default for LiteParseConfig {
password: None,
quiet: false,
num_workers: default_num_workers(),
+ image_mode: ImageMode::Placeholder,
}
}
}
diff --git a/crates/liteparse/src/extract.rs b/crates/liteparse/src/extract.rs
index 402f97b..45fbd4d 100644
--- a/crates/liteparse/src/extract.rs
+++ b/crates/liteparse/src/extract.rs
@@ -1,5 +1,9 @@
use crate::error::LiteParseError;
-use crate::types::{GraphicPrimitive, Page as LitePage, PdfInput, Rect, TextItem};
+use crate::render::encode_png;
+use crate::types::{
+ ExtractedImage, GraphicPrimitive, ImageRef, OutlineTarget, Page as LitePage, PdfInput, Rect,
+ StructNode, TextItem,
+};
use pdfium::{Document, Font, FontType, Library, Page, PathObject, RectF, SegmentKind, TextPage};
/// Open a PDF from path or bytes with an optional password.
@@ -31,8 +35,23 @@ pub(crate) fn extract_pages_from_document(
target_pages: Option<&[u32]>,
max_pages: usize,
) -> Result<Vec<LitePage>, LiteParseError> {
+ Ok(extract_pages_and_images(document, target_pages, max_pages, false)?.0)
+}
+
+/// Same as `extract_pages_from_document` but optionally also renders every
+/// raster image object to PNG bytes (when `render_images = true`). Returned
+/// `ExtractedImage`s carry the same ids the markdown emitter will reference,
+/// so callers can match them up by id. When `render_images = false` the
+/// returned image vec is always empty.
+pub(crate) fn extract_pages_and_images(
+ document: &Document,
+ target_pages: Option<&[u32]>,
+ max_pages: usize,
+ render_images: bool,
+) -> Result<(Vec<LitePage>, Vec<ExtractedImage>), LiteParseError> {
let page_count = document.page_count();
let mut pages = Vec::new();
+ let mut images: Vec<ExtractedImage> = Vec::new();
for page_index in 0..page_count {
let page_number = page_index as u32 + 1;
@@ -57,6 +76,12 @@ pub(crate) fn extract_pages_from_document(
});
let text_items = extract_page_text_items(&page, &text_page, &view_box)?;
let graphics = extract_page_graphics(&page, &view_box);
+ let struct_nodes = extract_page_struct_nodes(&page, &view_box);
+ let image_refs = extract_page_image_refs(&page, page_number);
+
+ if render_images && !image_refs.is_empty() {
+ images.extend(render_page_images(&page, page_number, &image_refs));
+ }
pages.push(LitePage {
page_number: page_number as usize,
@@ -64,10 +89,48 @@ pub(crate) fn extract_pages_from_document(
page_height: page.height(),
text_items,
graphics,
+ struct_nodes,
+ image_refs,
});
}
- Ok(pages)
+ Ok((pages, images))
+}
+
+/// Walk the document outline (bookmarks). Returns entries in pre-order.
+/// Empty when the PDF has no outline.
+pub(crate) fn extract_outline(document: &Document) -> Vec<OutlineTarget> {
+ document
+ .outline()
+ .into_iter()
+ .filter_map(|e| {
+ Some(OutlineTarget {
+ level: e.level,
+ title: e.title,
+ page_index: e.page_index?,
+ y_pdf: e.y,
+ })
+ })
+ .collect()
+}
+
+/// Walk the page's structure tree (tagged PDFs). Returns nodes in pre-order;
+/// empty when the page is untagged.
+fn extract_page_struct_nodes(page: &Page, view_box: &RectF) -> Vec<StructNode> {
+ page.struct_tree(view_box)
+ .into_iter()
+ .map(|n| StructNode {
+ role: n.role,
+ mcids: n.mcids,
+ bbox: n.bbox.map(|b| Rect {
+ x: b.left,
+ y: b.top,
+ width: b.right - b.left,
+ height: b.bottom - b.top,
+ }),
+ alt_text: n.alt_text,
+ })
+ .collect()
}
/// Extract raw text items and print each page as a JSON-line object to stdout.
@@ -137,6 +200,76 @@ fn should_skip_invisible(text_page: &TextPage, char_count: i32) -> bool {
invisible_ratio < 0.3
}
+/// Minimum image extent (in PDF points) below which we ignore the image
+/// object. Filters out hairline rasterized rules, icons embedded in glyphs,
+/// and other sub-25pt fragments that would otherwise pollute the figure
+/// stream. Matches the threshold used by `ocr_merge::has_images`.
+const IMAGE_MIN_SIZE_PT: f32 = 25.0;
+
+/// Max fraction of the page each axis can cover. Drops full-page background
+/// images (scanned pages, watermarks).
+const IMAGE_MAX_COVERAGE: f32 = 0.9;
+
+/// Render every image referenced in `refs` to PNG bytes using
+/// `Page::render_image_object`. Returns one `ExtractedImage` per ref. Used by
+/// the parser only when `ImageMode::Embed` is configured — otherwise the
+/// extraction loop skips this entirely. Failures for individual images are
+/// silently dropped (a malformed embedded image shouldn't fail the whole
+/// parse).
+pub(crate) fn render_page_images(
+ page: &Page,
+ page_number: u32,
+ refs: &[ImageRef],
+) -> Vec<ExtractedImage> {
+ let mut out = Vec::with_capacity(refs.len());
+ for r in refs {
+ let bmp = match page.render_image_object(r.obj_index) {
+ Ok(b) => b,
+ Err(_) => continue,
+ };
+ let w = bmp.width().max(0) as u32;
+ let h = bmp.height().max(0) as u32;
+ if w == 0 || h == 0 {
+ continue;
+ }
+ let rgba = bmp.to_rgba();
+ let png = match encode_png(&rgba, w, h) {
+ Ok(p) => p,
+ Err(_) => continue,
+ };
+ out.push(ExtractedImage {
+ id: r.id.clone(),
+ page: page_number,
+ bbox: r.bbox.clone(),
+ format: "png".into(),
+ bytes: png,
+ });
+ }
+ out
+}
+
+/// Walk image objects on a page and return a stable per-page `ImageRef` for
+/// each one. `obj_index` is the index among image-typed page objects (not all
+/// page objects), so a later embed pass can pull pixel bytes via
+/// `Page::render_image_object`. IDs are scoped to the page number so they
+/// remain stable across runs.
+fn extract_page_image_refs(page: &Page, page_number: u32) -> Vec<ImageRef> {
+ page.image_bounds(IMAGE_MIN_SIZE_PT, IMAGE_MAX_COVERAGE)
+ .into_iter()
+ .enumerate()
+ .map(|(i, b)| ImageRef {
+ id: format!("p{}_{}", page_number, i),
+ bbox: Rect {
+ x: b.x,
+ y: b.y,
+ width: b.width,
+ height: b.height,
+ },
+ obj_index: i,
+ })
+ .collect()
+}
+
/// Extract simplified vector graphics from a page. We keep only what the
/// markdown layout pass cares about:
/// - filled paths → a single bounding `Rect` (covers cell backgrounds /
diff --git a/crates/liteparse/src/main.rs b/crates/liteparse/src/main.rs
index 5c9e8d0..136c54f 100644
--- a/crates/liteparse/src/main.rs
+++ b/crates/liteparse/src/main.rs
@@ -89,6 +89,21 @@ struct ParseCommand {
/// Number of concurrent OCR workers (default: CPU cores - 1)
#[arg(long)]
num_workers: Option<usize>,
+
+ /// How to surface raster images in markdown output:
+ /// `off` strips them, `placeholder` (default) emits `![](image_pN_K.png)`
+ /// references in reading order, `embed` extracts each image's PNG bytes
+ /// and writes them next to the markdown output when `--image-output-dir`
+ /// is set.
+ #[arg(long, default_value = "placeholder")]
+ image_mode: String,
+
+ /// Directory to write embedded images to when `--image-mode embed` is
+ /// set. Each image is written as `image_{id}.png` to match the
+ /// references in the markdown output. Has no effect for other image
+ /// modes. Created if missing.
+ #[arg(long)]
+ image_output_dir: Option<String>,
}
#[derive(Args, Debug)]
@@ -197,6 +212,19 @@ fn parse_output_format(s: &str) -> Result<OutputFormat, String> {
}
}
+fn parse_image_mode(s: &str) -> Result<liteparse::config::ImageMode, String> {
+ use liteparse::config::ImageMode;
+ match s.to_lowercase().as_str() {
+ "off" | "none" => Ok(ImageMode::Off),
+ "placeholder" => Ok(ImageMode::Placeholder),
+ "embed" => Ok(ImageMode::Embed),
+ _ => Err(format!(
+ "unknown image-mode '{}', expected 'off', 'placeholder', or 'embed'",
+ s
+ )),
+ }
+}
+
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let cli = Cli::parse();
@@ -204,6 +232,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
match cli.command {
Commands::Parse(cmd) => {
let format = parse_output_format(&cmd.format)?;
+ let image_mode = parse_image_mode(&cmd.image_mode)?;
let mut config = LiteParseConfig {
ocr_language: cmd.ocr_language,
@@ -217,6 +246,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
password: cmd.password,
quiet: cmd.quiet,
ocr_server_url: cmd.ocr_server_url,
+ image_mode,
..Default::default()
};
if let Some(n) = cmd.num_workers {
@@ -230,6 +260,22 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
OutputFormat::Text => text::format_text(&result.pages),
OutputFormat::Markdown => result.text.clone(),
};
+ if let Some(dir) = cmd.image_output_dir.as_deref()
+ && !result.images.is_empty()
+ {
+ std::fs::create_dir_all(dir)?;
+ for img in &result.images {
+ let path = format!("{}/image_{}.{}", dir, img.id, img.format);
+ std::fs::write(&path, &img.bytes)?;
+ }
+ if !cmd.quiet {
+ eprintln!(
+ "[liteparse] wrote {} image(s) to {}",
+ result.images.len(),
+ dir
+ );
+ }
+ }
match cmd.output {
Some(path) => {
diff --git a/crates/liteparse/src/markdown_layout.rs b/crates/liteparse/src/markdown_layout.rs
deleted file mode 100644
index c7d255a..0000000
--- a/crates/liteparse/src/markdown_layout.rs
+++ /dev/null
@@ -1,3369 +0,0 @@
-//! Block classification for the markdown emitter.
-//!
-//! Consumes `ProjectedLine` entries from each `ParsedPage` and groups them into
-//! a sequence of `Block`s: headings, paragraphs, and (for now) raw lines that
-//! don't fit a recognized shape. Tables, lists, and code blocks land in later
-//! build-order steps.
-
-use crate::projection::{is_bold_item, is_italic_item, is_mono_item};
-use crate::types::{Anchor, GraphicPrimitive, ParsedPage, ProjectedLine, TextItem};
-
-/// Maximum stroke thickness (or |y1-y2|) for a stroke to count as a candidate HR.
-/// Thicker shapes are filled rects, not rules.
-const HR_MAX_THICKNESS_PT: f32 = 2.0;
-
-/// Minimum fraction of page width a horizontal stroke must span to count as an HR.
-/// Shorter strokes are typically table borders, list bullets, or inline marks.
-const HR_MIN_WIDTH_FRACTION: f32 = 0.3;
-
-/// Vertical tolerance (points) for treating a stroke as "underlining" the
-/// nearest text line. Strokes within this band of a text line's baseline are
-/// dropped — they're underlines, not rules.
-const HR_UNDERLINE_PROXIMITY_PT: f32 = 3.0;
-
-/// Detect horizontal rules from a page's vector graphics.
-///
-/// Returns the y-coordinates (viewport space) of accepted HRs, sorted ascending.
-/// An HR is a roughly horizontal stroke that spans at least
-/// `HR_MIN_WIDTH_FRACTION` of the page width, is thinner than
-/// `HR_MAX_THICKNESS_PT`, and does not sit on the baseline of any text line
-/// (which would make it an underline).
-pub fn detect_horizontal_rules(page: &ParsedPage) -> Vec<f32> {
- if page.graphics.is_empty() || page.page_width <= 0.0 {
- return Vec::new();
- }
- let min_width = page.page_width * HR_MIN_WIDTH_FRACTION;
- let mut ys: Vec<f32> = Vec::new();
-
- for g in &page.graphics {
- let GraphicPrimitive::Stroke {
- x1,
- y1,
- x2,
- y2,
- width,
- ..
- } = g
- else {
- continue;
- };
- let (x1, y1, x2, y2, width) = (*x1, *y1, *x2, *y2, *width);
- let dy = (y1 - y2).abs();
- let dx = (x1 - x2).abs();
- if dy > HR_MAX_THICKNESS_PT || width > HR_MAX_THICKNESS_PT {
- continue;
- }
- if dx < min_width {
- continue;
- }
- let y = (y1 + y2) * 0.5;
- let xmin = x1.min(x2);
- let xmax = x1.max(x2);
-
- // Drop if this stroke sits on a text-line baseline — it's an underline,
- // not a divider.
- let is_underline = page.projected_lines.iter().any(|line| {
- let bottom = line.bbox.y + line.bbox.height;
- (y - bottom).abs() < HR_UNDERLINE_PROXIMITY_PT
- && xmin >= line.bbox.x - 2.0
- && xmax <= line.bbox.x + line.bbox.width + 2.0
- });
- if is_underline {
- continue;
- }
- ys.push(y);
- }
-
- // Sort + dedup near-duplicates (some PDFs draw the same rule twice).
- ys.sort_by(|a, b| a.total_cmp(b));
- ys.dedup_by(|a, b| (*a - *b).abs() < 1.0);
- ys
-}
-
-/// Coarse block representation. Intentionally minimal — extended as later
-/// build-order steps land (tables, figures).
-#[derive(Debug, Clone)]
-pub enum Block {
- Heading {
- level: u8,
- text: String,
- },
- Paragraph {
- text: String,
- bold: bool,
- italic: bool,
- },
- ListItem {
- ordered: bool,
- marker: String,
- level: u8,
- text: String,
- bold: bool,
- italic: bool,
- },
- /// Fenced code block — content rendered between triple-backtick fences.
- /// Each entry in `lines` is one source line; preserved as-is (only trailing
- /// whitespace stripped) so indentation survives.
- CodeBlock {
- lines: Vec<String>,
- },
- /// Confident table emitted as a markdown pipe table. `header` is `None`
- /// when the first row didn't qualify (e.g. wasn't bold and the table mode
- /// can't otherwise distinguish it).
- Table {
- header: Option<Vec<String>>,
- rows: Vec<Vec<String>>,
- },
- /// Tabular-looking region we couldn't classify confidently — rendered
- /// verbatim inside a fenced block to preserve visual structure for the
- /// downstream LLM. Strictly better than emitting a mangled pipe table.
- GridFallback {
- lines: Vec<String>,
- },
- /// A horizontal rule detected from a long thin horizontal stroke in the
- /// page's vector graphics (e.g. divider line between sections).
- HorizontalRule,
- Blank,
-}
-
-/// Minimum cells per row for a region to qualify as a table.
-const TABLE_MIN_COLUMNS: usize = 3;
-
-/// Minimum consecutive rows for a region to qualify as a table.
-const TABLE_MIN_ROWS: usize = 2;
-
-/// Gap between adjacent spans (in multiples of dominant font size) above which
-/// we treat the gap as a cell boundary.
-const TABLE_CELL_GAP_FONT_MULTIPLIER: f32 = 1.0;
-
-/// Tolerance (points) for matching a cell's start-x to an existing column
-/// track when extending a candidate table run.
-const TABLE_TRACK_TOLERANCE_PT: f32 = 6.0;
-
-/// Maximum vertical gap between consecutive table rows, expressed in multiples
-/// of the line height. Looser than the paragraph rule because table rows often
-/// have more vertical padding than prose lines.
-const TABLE_ROW_GAP_MULTIPLIER: f32 = 2.5;
-
-/// Maximum coefficient-of-variation for row spacing within a confident table
-/// (rejecting irregular spacing that's more likely prose or a footer block).
-const TABLE_ROW_SPACING_MAX_CV: f32 = 0.5;
-
-/// One cell within a tabular row: contributing spans aggregated to text and
-/// its leftmost x position, used to align cells across rows into column
-/// "tracks".
-#[derive(Debug, Clone)]
-struct TableCell {
- start_x: f32,
- /// Right edge of the cell (x of the last span's right). Used by
- /// `recover_merged_cell` to detect cells that straddle two column tracks
- /// when the projection merged two adjacent words into one span.
- end_x: f32,
- text: String,
- bold: bool,
-}
-
-/// A contiguous tabular run: line indices `[start, end)` plus the detected
-/// rows. Used so the line-classifier can skip the consumed range and so
-/// fallback rendering can reach back for the original projected text.
-#[derive(Debug, Clone)]
-struct TableRun {
- start: usize,
- end: usize,
- block: Block,
-}
-
-/// Split a `ProjectedLine`'s spans into cells. A gap larger than
-/// `TABLE_CELL_GAP_FONT_MULTIPLIER × font_size` between adjacent spans starts
-/// a new cell; otherwise spans join into the same cell with a single space.
-fn split_cells(line: &ProjectedLine) -> Vec<TableCell> {
- // Skip whitespace-only spans before computing gaps — leading/trailing
- // empty items would otherwise add spurious cell boundaries.
- let mut spans: Vec<&TextItem> = line
- .spans
- .iter()
- .filter(|s| !s.text.trim().is_empty())
- .collect();
- spans.sort_by(|a, b| a.x.total_cmp(&b.x));
- if spans.is_empty() {
- return Vec::new();
- }
- let font_size = if line.dominant_font_size > 0.0 {
- line.dominant_font_size
- } else {
- line.bbox.height.max(1.0)
- };
- let gap_threshold = font_size * TABLE_CELL_GAP_FONT_MULTIPLIER;
-
- let mut cells: Vec<TableCell> = Vec::new();
- let mut current_text = String::new();
- let mut current_start = spans[0].x;
- let mut current_bold_chars: usize = 0;
- let mut current_total_chars: usize = 0;
- let mut prev_right = spans[0].x;
-
- for (i, span) in spans.iter().enumerate() {
- let gap = span.x - prev_right;
- let break_cell = i > 0 && gap > gap_threshold;
- if break_cell {
- let bold = current_total_chars > 0 && current_bold_chars * 2 > current_total_chars;
- cells.push(TableCell {
- start_x: current_start,
- end_x: prev_right,
- text: collapse_whitespace(current_text.trim()),
- bold,
- });
- current_text.clear();
- current_start = span.x;
- current_bold_chars = 0;
- current_total_chars = 0;
- }
- if !current_text.is_empty() && !current_text.ends_with(' ') {
- current_text.push(' ');
- }
- current_text.push_str(&span.text);
- let n = span.text.chars().count();
- current_total_chars += n;
- if is_bold_span(span) {
- current_bold_chars += n;
- }
- prev_right = span.x + span.width.max(0.0);
- }
- if !current_text.trim().is_empty() {
- let bold = current_total_chars > 0 && current_bold_chars * 2 > current_total_chars;
- cells.push(TableCell {
- start_x: current_start,
- end_x: prev_right,
- text: collapse_whitespace(current_text.trim()),
- bold,
- });
- }
- cells
-}
-
-/// When a candidate row has fewer cells than the established column count,
-/// look for cells whose x-range straddles multiple column tracks (likely two
-/// or more adjacent words that PDFium merged into a single text run) and
-/// split each on internal whitespace at the boundaries nearest to the
-/// straddled tracks.
-///
-/// Returns the patched cells if every short cell could be cleanly split to
-/// recover `tracks.len()` cells total; otherwise `None`.
-fn recover_merged_cell(mut cells: Vec<TableCell>, tracks: &[f32]) -> Option<Vec<TableCell>> {
- let target = tracks.len();
- if cells.len() >= target {
- return None;
- }
- // Repeatedly find the cell that straddles the most tracks (≥2) and split
- // it. Each iteration strictly grows `cells.len()`, so termination is
- // guaranteed; if no cell straddles ≥2 tracks before we hit the target,
- // recovery fails.
- while cells.len() < target {
- let mut best_i: Option<usize> = None;
- let mut best_count: usize = 1;
- let mut best_contained: Vec<f32> = Vec::new();
- for (i, cell) in cells.iter().enumerate() {
- let contained: Vec<f32> = tracks
- .iter()
- .copied()
- .filter(|t| {
- *t >= cell.start_x - TABLE_TRACK_TOLERANCE_PT
- && *t <= cell.end_x + TABLE_TRACK_TOLERANCE_PT
- })
- .collect();
- if contained.len() > best_count {
- best_count = contained.len();
- best_i = Some(i);
- best_contained = contained;
- }
- }
- let Some(i) = best_i else {
- return None;
- };
- let cell = cells[i].clone();
- let chars: Vec<char> = cell.text.trim().chars().collect();
- let n = chars.len();
- if n == 0 || best_contained.len() < 2 {
- return None;
- }
- let text_width = (cell.end_x - cell.start_x).max(1.0);
- // For each track after the first, pick the whitespace boundary in
- // `chars` whose linearly-interpolated x is closest to the track.
- let mut split_indices: Vec<usize> = Vec::new();
- for t in best_contained.iter().skip(1) {
- let mut best: Option<(usize, f32)> = None;
- for (k, ch) in chars.iter().enumerate() {
- if !ch.is_whitespace() {
- continue;
- }
- if split_indices.contains(&k) {
- continue;
- }
- let frac = k as f32 / n as f32;
- let x = cell.start_x + frac * text_width;
- let d = (x - t).abs();
- if best.as_ref().is_none_or(|b| d < b.1) {
- best = Some((k, d));
- }
- }
- let (k, _) = best?;
- split_indices.push(k);
- }
- split_indices.sort();
- // Build the split pieces.
- let mut pieces: Vec<String> = Vec::new();
- let mut prev = 0usize;
- for k in &split_indices {
- let piece: String = chars[prev..*k]
- .iter()
- .collect::<String>()
- .trim()
- .to_string();
- if piece.is_empty() {
- return None;
- }
- pieces.push(piece);
- prev = *k;
- }
- let last: String = chars[prev..].iter().collect::<String>().trim().to_string();
- if last.is_empty() {
- return None;
- }
- pieces.push(last);
- if pieces.len() != best_contained.len() {
- return None;
- }
- // Synthesize new TableCells aligned with each track.
- let mut new_cells: Vec<TableCell> = Vec::with_capacity(pieces.len());
- for (p, piece) in pieces.iter().enumerate() {
- let start_x = if p == 0 {
- cell.start_x
- } else {
- best_contained[p]
- };
- let end_x = if p + 1 < best_contained.len() {
- (best_contained[p + 1] - 1.0).max(start_x)
- } else {
- cell.end_x
- };
- new_cells.push(TableCell {
- start_x,
- end_x,
- text: piece.clone(),
- bold: cell.bold,
- });
- }
- cells.remove(i);
- for (offset, c) in new_cells.into_iter().enumerate() {
- cells.insert(i + offset, c);
- }
- }
- if cells.len() == target {
- Some(cells)
- } else {
- None
- }
-}
-
-/// Lightweight bold detection on a raw `TextItem`. Delegates to
-/// `projection::is_bold_item` so per-span and per-line logic stay in sync.
-fn is_bold_span(s: &TextItem) -> bool {
- is_bold_item(s)
-}
-
-/// Per-span style flags used by the inline-emphasis renderer.
-#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
-struct SpanStyle {
- bold: bool,
- italic: bool,
- mono: bool,
-}
-
-impl SpanStyle {
- fn from_item(item: &TextItem) -> Self {
- SpanStyle {
- bold: is_bold_item(item),
- italic: is_italic_item(item),
- mono: is_mono_item(item),
- }
- }
-
- fn is_plain(self) -> bool {
- !self.bold && !self.italic && !self.mono
- }
-}
-
-/// Escape characters that would otherwise be interpreted as markdown emphasis.
-/// Deliberately narrow: only `*`, `_`, and backslash. Aggressive escaping
-/// (`#`, `[`, backticks, etc.) breaks more output than it saves in practice —
-/// pymupdf4llm takes the same conservative stance.
-fn escape_inline(s: &str) -> String {
- let mut out = String::with_capacity(s.len());
- for c in s.chars() {
- match c {
- '\\' | '*' | '_' => {
- out.push('\\');
- out.push(c);
- }
- _ => out.push(c),
- }
- }
- out
-}
-
-/// Wrap `inner` with the markdown markers for `style`. Mono wins over bold/italic:
-/// inline code (`` `…` ``) doesn't compose with emphasis in CommonMark, so when
-/// a span is mono we drop the `**/*` wrap. Bold + italic → `***…***`.
-fn apply_style(inner: &str, style: SpanStyle) -> String {
- if style.mono {
- // Use backticks; if inner already contains backticks, switch to a
- // longer fence (pair of backticks plus a space buffer) per CommonMark.
- if inner.contains('`') {
- return format!("`` {} ``", inner);
- }
- return format!("`{}`", inner);
- }
- match (style.bold, style.italic) {
- (true, true) => format!("***{}***", inner),
- (true, false) => format!("**{}**", inner),
- (false, true) => format!("*{}*", inner),
- (false, false) => inner.to_string(),
- }
-}
-
-/// Render a `ProjectedLine` to markdown with per-span emphasis. Adjacent
-/// same-style spans are merged into a single emphasis run; whitespace between
-/// spans is preserved as one space (the underlying projection output already
-/// has the right inter-word spacing baked into span text).
-///
-/// Per-line shortcut: when every non-whitespace span shares the same style,
-/// emit one outer wrap around the collapsed line text instead of run-by-run
-/// (matches pymupdf4llm; avoids `**foo** **bar** **baz**` noise on uniformly
-/// styled lines).
-///
-/// Falls back to `collapse_whitespace(line.text)` when the line has no usable
-/// spans (e.g. OCR-only lines where TextItem styling isn't populated).
-fn render_line_inline(line: &ProjectedLine) -> String {
- let spans: Vec<&TextItem> = line
- .spans
- .iter()
- .filter(|s| !s.text.trim().is_empty())
- .collect();
- if spans.is_empty() {
- return collapse_whitespace(&line.text);
- }
-
- // Sort spans by x so we render in visual reading order regardless of
- // extraction order. Stable so equal-x spans keep their original sequence.
- let mut spans = spans;
- spans.sort_by(|a, b| a.x.total_cmp(&b.x));
-
- let styles: Vec<SpanStyle> = spans.iter().map(|s| SpanStyle::from_item(s)).collect();
-
- // Per-line shortcut.
- let uniform = styles.iter().all(|s| *s == styles[0]);
- if uniform {
- let joined = collapse_whitespace(&line.text);
- if joined.is_empty() {
- return joined;
- }
- let escaped = escape_inline(&joined);
- if styles[0].is_plain() {
- return escaped;
- }
- return apply_style(&escaped, styles[0]);
- }
-
- // Group consecutive spans by style. Within a group, span texts join with
- // a single space (we lose intra-group spacing precision; acceptable).
- let mut out = String::new();
- let mut i = 0;
- while i < spans.len() {
- let style = styles[i];
- let mut j = i + 1;
- while j < spans.len() && styles[j] == style {
- j += 1;
- }
- let mut group_text = String::new();
- for k in i..j {
- if !group_text.is_empty() && !group_text.ends_with(' ') {
- group_text.push(' ');
- }
- group_text.push_str(spans[k].text.trim());
- }
- let group_text = collapse_whitespace(&group_text);
- let escaped = escape_inline(&group_text);
- let rendered = if style.is_plain() {
- escaped
- } else {
- apply_style(&escaped, style)
- };
- if !out.is_empty() && !out.ends_with(' ') {
- out.push(' ');
- }
- out.push_str(&rendered);
- i = j;
- }
- out
-}
-
-/// Render the text portion of a list item with per-span emphasis. The marker
-/// itself isn't included in the output (the renderer handles it separately).
-///
-/// When the line is uniformly styled we wrap the marker-stripped `rest` with
-/// the line's style — this avoids the awkward emphasis-marker mismatch we'd
-/// hit if we naively stripped a leading bullet out of an already-wrapped
-/// rendered line (`**• item**` → `** item**`).
-///
-/// When the line is mixed-style we render the full line via the inline pipeline
-/// and then best-effort-strip the marker prefix (with optional emphasis wrap
-/// around it). On any failure we fall back to plain escaped `rest`.
-fn render_list_item_text(line: &ProjectedLine, marker: &str, rest: &str) -> String {
- if let Some(style) = line_uniform_style(line) {
- let plain = collapse_whitespace(rest);
- let escaped = escape_inline(&plain);
- return if style.is_plain() {
- escaped
- } else {
- apply_style(&escaped, style)
- };
- }
- let full = render_line_inline(line);
- if let Some(stripped) = strip_leading_marker_from_inline(&full, marker) {
- return stripped;
- }
- escape_inline(&collapse_whitespace(rest))
-}
-
-/// Try to strip a leading list marker (optionally wrapped in emphasis markers)
-/// off `s`. Recognizes `***MARK*** `, `**MARK** `, `*MARK* `, `` `MARK` ``,
-/// and bare `MARK `. Returns the suffix on a match.
-fn strip_leading_marker_from_inline(s: &str, marker: &str) -> Option<String> {
- for wrap in ["***", "**", "*", "`"] {
- let prefix = format!("{wrap}{marker}{wrap} ");
- if let Some(rest) = s.strip_prefix(&prefix) {
- return Some(rest.to_string());
- }
- }
- let prefix = format!("{marker} ");
- s.strip_prefix(&prefix).map(|r| r.to_string())
-}
-
-/// Append an inline-rendered continuation line to an existing list-item body.
-/// De-hyphenates against the raw text boundary (mirrors the paragraph rule)
-/// and falls back to a space join otherwise.
-fn append_inline_continuation(prev_text: &mut String, next_raw: &str, next_inline: &str) {
- let next_raw = collapse_whitespace(next_raw);
- if next_raw.is_empty() {
- return;
- }
- if prev_text.is_empty() {
- prev_text.push_str(next_inline);
- return;
- }
- let dehyphenate = prev_text.ends_with('-')
- && next_raw
- .chars()
- .next()
- .is_some_and(|c| c.is_ascii_lowercase());
- if dehyphenate {
- prev_text.pop();
- prev_text.push_str(next_inline);
- } else {
- prev_text.push(' ');
- prev_text.push_str(next_inline);
- }
-}
-
-/// Vertical-gap check for table rows. Looser than paragraph continuation
-/// because table rows often have extra padding between them.
-fn table_rows_adjacent(prev: &ProjectedLine, cur: &ProjectedLine) -> bool {
- // Intentionally don't require region_path equality. Indented sub-group
- // rows (e.g. an indented "MEMORYBANK" row in a grouped academic results
- // table) sometimes land in a different XY-cut leaf than the rest of the
- // table — but the column-track alignment and y-gap checks below are
- // strong enough signals on their own to keep us from spuriously
- // bridging unrelated regions.
- let prev_bottom = prev.bbox.y + prev.bbox.height;
- let gap = cur.bbox.y - prev_bottom;
- let line_height = prev.bbox.height.max(cur.bbox.height).max(1.0);
- gap >= -line_height && gap <= line_height * TABLE_ROW_GAP_MULTIPLIER
-}
-
-/// Coefficient of variation (std-dev / mean) of inter-row vertical gaps.
-/// Returns 0.0 for runs with <2 gaps (nothing to compare). Used to reject
-/// runs whose row spacing is too irregular to be a real table.
-fn row_spacing_cv(rows: &[(usize, &ProjectedLine, Vec<TableCell>)]) -> f32 {
- if rows.len() < 3 {
- return 0.0;
- }
- let gaps: Vec<f32> = rows
- .windows(2)
- .map(|w| (w[1].1.bbox.y - w[0].1.bbox.y).abs())
- .collect();
- let mean = gaps.iter().sum::<f32>() / gaps.len() as f32;
- if mean <= 0.0 {
- return f32::INFINITY;
- }
- let var = gaps.iter().map(|g| (g - mean).powi(2)).sum::<f32>() / gaps.len() as f32;
- var.sqrt() / mean
-}
-
-/// Try to extend a candidate table starting at `start_idx`. On success returns
-/// a `TableRun` with `Block::Table` or `Block::GridFallback`; on failure
-/// returns `None` (and the caller should fall through to per-line
-/// classification).
-fn try_detect_table(lines: &[ProjectedLine], start_idx: usize) -> Option<TableRun> {
- let first_cells = split_cells(&lines[start_idx]);
- if first_cells.len() < TABLE_MIN_COLUMNS {
- return None;
- }
-
- let mut rows: Vec<(usize, &ProjectedLine, Vec<TableCell>)> =
- vec![(start_idx, &lines[start_idx], first_cells.clone())];
- let column_count = first_cells.len();
- let tracks: Vec<f32> = first_cells.iter().map(|c| c.start_x).collect();
-
- let mut j = start_idx + 1;
- while j < lines.len() {
- if !table_rows_adjacent(rows.last().unwrap().1, &lines[j]) {
- break;
- }
- let mut cells = split_cells(&lines[j]);
- if cells.len() < column_count && cells.len() >= TABLE_MIN_COLUMNS {
- // PDFium occasionally merges two (or more) adjacent words into one
- // text run when inter-word kerning is tighter than the gap
- // threshold — common in tightly-set numeric tables (e.g. the
- // "MEMORYBANK 5.00 4.77" case on page 6 of the AMEM paper).
- // Recover by splitting straddling cells on internal whitespace.
- if let Some(patched) = recover_merged_cell(cells.clone(), &tracks) {
- cells = patched;
- }
- }
- if cells.len() != column_count {
- break;
- }
- // Allow at most one column track to drift out of tolerance, which lets
- // grouped row-labels in academic tables (e.g. an indented "MEMORYBANK"
- // row whose label column shifts right by ~30pt while the numeric
- // columns stay aligned) stay in the same run. Without this slack a
- // single indented label fragments a 6-row table into three 2-row chunks.
- let misaligned = cells
- .iter()
- .zip(tracks.iter())
- .filter(|(c, t)| (c.start_x - **t).abs() > TABLE_TRACK_TOLERANCE_PT)
- .count();
- if misaligned > 1 {
- break;
- }
- rows.push((j, &lines[j], cells));
- j += 1;
- }
-
- if rows.len() < TABLE_MIN_ROWS {
- return None;
- }
-
- let cv = row_spacing_cv(&rows);
- let end = j;
-
- if cv > TABLE_ROW_SPACING_MAX_CV {
- // Suggestive layout but the row cadence is too irregular to trust as a
- // clean table — surface as a fenced fallback so the structure is at
- // least preserved.
- let raw: Vec<String> = rows
- .iter()
- .map(|(_, line, _)| line.text.trim_end().to_string())
- .collect();
- return Some(TableRun {
- start: start_idx,
- end,
- block: Block::GridFallback { lines: raw },
- });
- }
-
- // Promote the first row to header iff every cell in it is bold (matches
- // pymupdf4llm's "bold-or-filled" heuristic; fills require fork data).
- let first_row = &rows[0].2;
- let header_qualifies = first_row.iter().all(|c| c.bold);
- let header = if header_qualifies {
- Some(first_row.iter().map(|c| c.text.clone()).collect())
- } else {
- None
- };
- let row_start = if header.is_some() { 1 } else { 0 };
- let body_rows: Vec<Vec<String>> = rows[row_start..]
- .iter()
- .map(|(_, _, cells)| cells.iter().map(|c| c.text.clone()).collect())
- .collect();
- if header.is_none() && body_rows.len() < TABLE_MIN_ROWS {
- return None;
- }
-
- Some(TableRun {
- start: start_idx,
- end,
- block: Block::Table {
- header,
- rows: body_rows,
- },
- })
-}
-
-/// Scan `lines` once and return all detected tabular regions (sorted by
-/// `start`). Caller uses these as cut-points so the per-line classifier never
-/// sees lines inside a table.
-fn detect_tables(lines: &[ProjectedLine]) -> Vec<TableRun> {
- let mut out = Vec::new();
- let mut i = 0;
- while i < lines.len() {
- if let Some(run) = try_detect_table(lines, i) {
- i = run.end;
- out.push(run);
- } else {
- i += 1;
- }
- }
- out
-}
-
-// ── Ruled-grid table detection ─────────────────────────────────────────────
-//
-// Detect tables drawn with explicit horizontal + vertical rules (the "Strong"
-// mode in MARKDOWN_PLAN.md). Strokes are clustered into H/V grid lines, then
-// union-find groups crossing lines into table regions. For each region the
-// distinct row/column boundaries form a cell grid; text lines are assigned to
-// cells by centroid containment.
-//
-// Ruled tables are detected before the borderless `detect_tables`. The caller
-// merges the two outputs; overlapping ranges defer to the ruled run because
-// path-based geometry is a strictly stronger signal than text alignment alone.
-
-/// Horizontal segment in viewport coords (top-left origin). `y` is the rule's
-/// y-position; `x_min..x_max` is its horizontal span. Endpoints of multiple
-/// short segments sharing a y get unioned into one wider segment during
-/// clustering.
-#[derive(Debug, Clone, Copy)]
-struct HSeg {
- x_min: f32,
- x_max: f32,
- y: f32,
-}
-
-#[derive(Debug, Clone, Copy)]
-struct VSeg {
- y_min: f32,
- y_max: f32,
- x: f32,
-}
-
-/// Strokes are considered "axis-aligned" when the perpendicular delta is at
-/// most this many points. Generous to absorb antialiased near-pixel strokes.
-const TABLE_AXIS_TOLERANCE_PT: f32 = 1.0;
-
-/// Two H lines (or two V lines) are merged into one grid line when their
-/// perpendicular coords are within this many points. Slightly looser than the
-/// axis tolerance because rules drawn at the same row can have ±1pt jitter
-/// from different stroke widths.
-const TABLE_GRID_CLUSTER_PT: f32 = 2.0;
-
-/// Slack added when checking whether a V line "crosses" an H line. Helps
-/// when rules don't quite reach the corner because the PDF drew them as
-/// individual segments with small gaps.
-const TABLE_CROSS_TOLERANCE_PT: f32 = 3.0;
-
-/// Reject ruled-table candidates whose empty-cell fraction exceeds this.
-const TABLE_MAX_EMPTY_CELL_FRACTION: f32 = 0.30;
-
-/// Reject candidates whose grid covers nearly the whole page — almost always
-/// a page border, not a real table.
-const TABLE_MAX_PAGE_COVERAGE: f32 = 0.95;
-
-/// Extract horizontal and vertical line segments from a page's graphics. Each
-/// `Stroke` becomes one HSeg or VSeg depending on orientation; each stroked
-/// `Rect` contributes its four edges (cell-border rects, table frames).
-fn extract_h_v_segments(graphics: &[GraphicPrimitive]) -> (Vec<HSeg>, Vec<VSeg>) {
- let mut hs = Vec::new();
- let mut vs = Vec::new();
- for g in graphics {
- match g {
- GraphicPrimitive::Stroke { x1, y1, x2, y2, .. } => {
- let (x1, y1, x2, y2) = (*x1, *y1, *x2, *y2);
- let dy = (y1 - y2).abs();
- let dx = (x1 - x2).abs();
- if dy <= TABLE_AXIS_TOLERANCE_PT && dx > 1.0 {
- hs.push(HSeg {
- x_min: x1.min(x2),
- x_max: x1.max(x2),
- y: (y1 + y2) * 0.5,
- });
- } else if dx <= TABLE_AXIS_TOLERANCE_PT && dy > 1.0 {
- vs.push(VSeg {
- y_min: y1.min(y2),
- y_max: y1.max(y2),
- x: (x1 + x2) * 0.5,
- });
- }
- }
- GraphicPrimitive::Rect { bbox, stroke, .. } => {
- if stroke.is_none() {
- continue;
- }
- let top = bbox.y;
- let bottom = bbox.y + bbox.height;
- let left = bbox.x;
- let right = bbox.x + bbox.width;
- if bbox.width > 1.0 {
- hs.push(HSeg {
- x_min: left,
- x_max: right,
- y: top,
- });
- hs.push(HSeg {
- x_min: left,
- x_max: right,
- y: bottom,
- });
- }
- if bbox.height > 1.0 {
- vs.push(VSeg {
- y_min: top,
- y_max: bottom,
- x: left,
- });
- vs.push(VSeg {
- y_min: top,
- y_max: bottom,
- x: right,
- });
- }
- }
- }
- }
- (hs, vs)
-}
-
-/// Cluster H segments sharing a y-coordinate (within `TABLE_GRID_CLUSTER_PT`)
-/// into a single wider grid line whose x-extent is the union of the inputs.
-fn cluster_h_segments(mut segs: Vec<HSeg>) -> Vec<HSeg> {
- if segs.is_empty() {
- return segs;
- }
- segs.sort_by(|a, b| a.y.total_cmp(&b.y));
- let mut out: Vec<HSeg> = Vec::with_capacity(segs.len());
- for seg in segs {
- if let Some(last) = out.last_mut()
- && (last.y - seg.y).abs() <= TABLE_GRID_CLUSTER_PT
- {
- last.x_min = last.x_min.min(seg.x_min);
- last.x_max = last.x_max.max(seg.x_max);
- continue;
- }
- out.push(seg);
- }
- out
-}
-
-fn cluster_v_segments(mut segs: Vec<VSeg>) -> Vec<VSeg> {
- if segs.is_empty() {
- return segs;
- }
- segs.sort_by(|a, b| a.x.total_cmp(&b.x));
- let mut out: Vec<VSeg> = Vec::with_capacity(segs.len());
- for seg in segs {
- if let Some(last) = out.last_mut()
- && (last.x - seg.x).abs() <= TABLE_GRID_CLUSTER_PT
- {
- last.y_min = last.y_min.min(seg.y_min);
- last.y_max = last.y_max.max(seg.y_max);
- continue;
- }
- out.push(seg);
- }
- out
-}
-
-/// Union-find root with path compression.
-fn uf_find(parent: &mut [usize], mut x: usize) -> usize {
- while parent[x] != x {
- parent[x] = parent[parent[x]];
- x = parent[x];
- }
- x
-}
-
-fn uf_union(parent: &mut [usize], a: usize, b: usize) {
- let ra = uf_find(parent, a);
- let rb = uf_find(parent, b);
- if ra != rb {
- parent[ra] = rb;
- }
-}
-
-/// Group H/V grid lines that cross each other into connected components.
-/// Each component is a candidate ruled table — typically one component per
-/// distinct table on the page. Returns `(h_indices, v_indices)` per component,
-/// dropping components without ≥2 H and ≥2 V lines (a single L-shape doesn't
-/// make a table).
-fn find_grid_components(hs: &[HSeg], vs: &[VSeg]) -> Vec<(Vec<usize>, Vec<usize>)> {
- let n_h = hs.len();
- let n_v = vs.len();
- if n_h < 2 || n_v < 2 {
- return Vec::new();
- }
- let n = n_h + n_v;
- let mut parent: Vec<usize> = (0..n).collect();
- let mut connected = vec![false; n];
-
- let tol = TABLE_CROSS_TOLERANCE_PT;
- for (i, h) in hs.iter().enumerate() {
- for (j, v) in vs.iter().enumerate() {
- let v_crosses_h_x = v.x >= h.x_min - tol && v.x <= h.x_max + tol;
- let h_crosses_v_y = h.y >= v.y_min - tol && h.y <= v.y_max + tol;
- if v_crosses_h_x && h_crosses_v_y {
- uf_union(&mut parent, i, n_h + j);
- connected[i] = true;
- connected[n_h + j] = true;
- }
- }
- }
-
- use std::collections::HashMap;
- let mut groups: HashMap<usize, (Vec<usize>, Vec<usize>)> = HashMap::new();
- for i in 0..n_h {
- if !connected[i] {
- continue;
- }
- let r = uf_find(&mut parent, i);
- groups.entry(r).or_default().0.push(i);
- }
- for j in 0..n_v {
- if !connected[n_h + j] {
- continue;
- }
- let r = uf_find(&mut parent, n_h + j);
- groups.entry(r).or_default().1.push(j);
- }
- groups
- .into_values()
- .filter(|(h_idx, v_idx)| h_idx.len() >= 2 && v_idx.len() >= 2)
- .collect()
-}
-
-/// Build a `TableRun` for one ruled-grid component. Returns `None` if the
-/// resulting grid is too small (< 2 cols or < 2 rows), covers nearly the
-/// whole page (likely the page border), or is mostly empty cells.
-fn build_ruled_table(
- hs: &[HSeg],
- vs: &[VSeg],
- h_indices: &[usize],
- v_indices: &[usize],
- lines: &[ProjectedLine],
- page_width: f32,
- page_height: f32,
-) -> Option<TableRun> {
- // Distinct row y-coords (cluster again — multiple H lines may share a y).
- let mut ys: Vec<f32> = h_indices.iter().map(|&i| hs[i].y).collect();
- ys.sort_by(|a, b| a.total_cmp(b));
- dedup_close(&mut ys, TABLE_GRID_CLUSTER_PT);
-
- let mut xs: Vec<f32> = v_indices.iter().map(|&i| vs[i].x).collect();
- xs.sort_by(|a, b| a.total_cmp(b));
- dedup_close(&mut xs, TABLE_GRID_CLUSTER_PT);
-
- // Need ≥2 cells per axis → ≥3 boundary lines.
- if ys.len() < 3 || xs.len() < 3 {
- return None;
- }
-
- let n_rows = ys.len() - 1;
- let n_cols = xs.len() - 1;
- let bbox = crate::types::Rect {
- x: xs[0],
- y: ys[0],
- width: xs[n_cols] - xs[0],
- height: ys[n_rows] - ys[0],
- };
-
- // Reject page-border-as-table.
- if page_width > 0.0 && page_height > 0.0 {
- let coverage = (bbox.width / page_width) * (bbox.height / page_height);
- if coverage > TABLE_MAX_PAGE_COVERAGE {
- return None;
- }
- }
-
- // Assign each text line to its cell by centroid.
- let mut cells: Vec<Vec<String>> = vec![vec![String::new(); n_cols]; n_rows];
- let mut cell_is_bold: Vec<Vec<bool>> = vec![vec![true; n_cols]; n_rows];
- let mut cell_has_text: Vec<Vec<bool>> = vec![vec![false; n_cols]; n_rows];
- let mut consumed_indices: Vec<usize> = Vec::new();
-
- for (idx, line) in lines.iter().enumerate() {
- let cx = line.bbox.x + line.bbox.width * 0.5;
- let cy = line.bbox.y + line.bbox.height * 0.5;
- if cy < ys[0] || cy > ys[n_rows] || cx < xs[0] || cx > xs[n_cols] {
- continue;
- }
- let row = match find_bucket(&ys, cy) {
- Some(r) => r,
- None => continue,
- };
- let col = match find_bucket(&xs, cx) {
- Some(c) => c,
- None => continue,
- };
- let txt = line.text.trim();
- if txt.is_empty() {
- continue;
- }
- if !cells[row][col].is_empty() {
- cells[row][col].push(' ');
- }
- cells[row][col].push_str(txt);
- cell_has_text[row][col] = true;
- if !line.all_bold {
- cell_is_bold[row][col] = false;
- }
- consumed_indices.push(idx);
- }
-
- if consumed_indices.is_empty() {
- return None;
- }
-
- let total = n_rows * n_cols;
- let empty_count = cell_has_text
- .iter()
- .flatten()
- .filter(|filled| !**filled)
- .count();
- if (empty_count as f32) / (total as f32) > TABLE_MAX_EMPTY_CELL_FRACTION {
- return None;
- }
-
- // Header = first row iff every non-empty cell in it is bold.
- let header_qualifies = cell_has_text[0]
- .iter()
- .zip(cell_is_bold[0].iter())
- .all(|(has, bold)| !has || *bold)
- && cell_has_text[0].iter().any(|has| *has);
- let header = if header_qualifies {
- Some(cells[0].clone())
- } else {
- None
- };
- let body_start = if header.is_some() { 1 } else { 0 };
- let body_rows: Vec<Vec<String>> = cells[body_start..].to_vec();
- if body_rows.is_empty() {
- return None;
- }
-
- // Line index span this table covers.
- let start = *consumed_indices.iter().min().unwrap();
- let end = *consumed_indices.iter().max().unwrap() + 1;
-
- Some(TableRun {
- start,
- end,
- block: Block::Table {
- header,
- rows: body_rows,
- },
- })
-}
-
-/// In-place dedup of a sorted Vec, collapsing entries within `tol` to the
-/// first of each cluster.
-fn dedup_close(v: &mut Vec<f32>, tol: f32) {
- if v.len() < 2 {
- return;
- }
- let mut out: Vec<f32> = Vec::with_capacity(v.len());
- for x in v.iter().copied() {
- if let Some(&last) = out.last()
- && (x - last).abs() <= tol
- {
- continue;
- }
- out.push(x);
- }
- *v = out;
-}
-
-/// Find the bucket index `i` such that `boundaries[i] <= val < boundaries[i+1]`.
-/// Returns `None` if `val` is outside the boundaries.
-fn find_bucket(boundaries: &[f32], val: f32) -> Option<usize> {
- if boundaries.len() < 2 || val < boundaries[0] || val > *boundaries.last().unwrap() {
- return None;
- }
- for (i, w) in boundaries.windows(2).enumerate() {
- if val >= w[0] && val <= w[1] {
- return Some(i);
- }
- }
- None
-}
-
-/// Detect ruled-grid tables on a page from its vector graphics. Returns runs
-/// in document order (sorted by `start`).
-fn detect_ruled_tables(
- lines: &[ProjectedLine],
- graphics: &[GraphicPrimitive],
- page_width: f32,
- page_height: f32,
-) -> Vec<TableRun> {
- let (hs, vs) = extract_h_v_segments(graphics);
- let hs = cluster_h_segments(hs);
- let vs = cluster_v_segments(vs);
- if hs.len() < 2 || vs.len() < 2 {
- return Vec::new();
- }
- let components = find_grid_components(&hs, &vs);
- let mut out = Vec::new();
- for (h_idx, v_idx) in components {
- if let Some(run) =
- build_ruled_table(&hs, &vs, &h_idx, &v_idx, lines, page_width, page_height)
- {
- out.push(run);
- }
- }
- out.sort_by_key(|r| r.start);
- out
-}
-
-/// Merge ruled-grid runs with borderless runs into a single sorted list. When
-/// ranges overlap the ruled run wins (path-based geometry is strictly stronger
-/// than text-alignment heuristics) — overlapping borderless runs are dropped.
-fn merge_table_runs(mut ruled: Vec<TableRun>, borderless: Vec<TableRun>) -> Vec<TableRun> {
- for b in borderless {
- let overlaps = ruled
- .iter()
- .any(|r| !(b.end <= r.start || b.start >= r.end));
- if !overlaps {
- ruled.push(b);
- }
- }
- ruled.sort_by_key(|r| r.start);
- ruled
-}
-
-/// Escape `|` and `\n` inside a markdown table cell so the pipe-table grammar
-/// stays valid. Newlines should be impossible inside a single cell (we built
-/// cells from spans on the same projected line) but guard anyway.
-fn escape_table_cell(s: &str) -> String {
- s.replace('\\', "\\\\")
- .replace('|', "\\|")
- .replace('\n', " ")
-}
-
-/// Tolerance in points for "is this size larger than the body size".
-const HEADING_SIZE_EPSILON: f32 = 0.5;
-
-/// Cap on heading levels (matches plan: H1..H6).
-const MAX_HEADING_LEVELS: usize = 6;
-
-/// Multiplier on line height used as the paragraph-break threshold.
-const PARAGRAPH_GAP_MULTIPLIER: f32 = 1.5;
-
-/// Tolerance for treating two font sizes as "the same" when grouping
-/// paragraph lines. Generous because we sometimes derive the "size" from
-/// `bbox.height`, which varies a few points line-to-line based on whether
-/// the glyphs include descenders (`y`, `g`, `p`).
-const FONT_SIZE_PARAGRAPH_TOLERANCE: f32 = 2.5;
-
-/// Tighter tolerance for matching against the heading-size map. Keeps the
-/// heading detector strict so descender-induced height jitter doesn't promote
-/// regular body lines to headings.
-const FONT_SIZE_HEADING_TOLERANCE: f32 = 0.6;
-
-/// Tolerance in points for treating two indent positions as "the same column".
-const INDENT_TOLERANCE: f32 = 6.0;
-
-/// Fraction of page height treated as the "top band" for header detection.
-/// Most running headers sit within the top 8–12% of a page; 12% gives some
-/// slack for two-line headers without sweeping in body text.
-const HEADER_BAND_FRACTION: f32 = 0.12;
-
-/// Fraction of page height treated as the "bottom band" for footer detection.
-const FOOTER_BAND_FRACTION: f32 = 0.12;
-
-/// Fraction of pages on which a normalized line must appear (in the same
-/// band) to be classified as a running header/footer.
-const HEADER_FOOTER_MIN_FRACTION: f32 = 0.5;
-
-/// Absolute floor on header/footer matches — single-page docs can't have a
-/// "running" header by definition, and a single match on a 2-page doc is too
-/// weak to act on.
-const HEADER_FOOTER_MIN_PAGES: usize = 2;
-
-/// Roughly one indent step in PDF points. Used to bucket list items into
-/// nesting levels relative to the first item of the list.
-const LIST_INDENT_STEP_PT: f32 = 12.0;
-
-/// Maximum characters in a "bold body-size heading" candidate. Section
-/// headings like "Abstract", "1 Introduction", "2.1 Related Work" are short;
-/// a bold body-size line longer than this is almost always a bold *sentence*
-/// inside a paragraph, not a heading.
-const BOLD_HEADING_MAX_CHARS: usize = 80;
-
-/// Returns true if `line` looks like a section heading rendered in body-size
-/// bold text (a very common style for academic / technical PDFs where every
-/// "real" heading uses the same font size as body, distinguished only by
-/// weight). Requires:
-/// - uniformly bold across all spans
-/// - short (≤ `BOLD_HEADING_MAX_CHARS`)
-/// - paragraph-break gap above (or first line on the page)
-/// - paragraph-break gap below (or last line on the page)
-fn looks_like_bold_heading(
- line: &ProjectedLine,
- prev: Option<&ProjectedLine>,
- next: Option<&ProjectedLine>,
-) -> bool {
- let text = line.text.trim();
- if text.is_empty() || text.chars().count() > BOLD_HEADING_MAX_CHARS {
- return false;
- }
- let style = match line_uniform_style(line) {
- Some(s) => s,
- None => return false,
- };
- if !style.bold || style.mono {
- return false;
- }
- // Reject bold-uniform lines dominated by digits / punctuation — these are
- // almost always cells inside a tabular layout the table detector didn't
- // pick up (results tables, scoreboards, math display). A real section
- // heading is mostly letters: "1 Introduction" passes (~92% alpha across
- // non-whitespace chars), "47.5 14" doesn't (0%), "BLEU-1 25.87" doesn't.
- let mut alpha = 0usize;
- let mut total = 0usize;
- for c in text.chars() {
- if c.is_whitespace() {
- continue;
- }
- total += 1;
- if c.is_alphabetic() {
- alpha += 1;
- }
- }
- if total == 0 || (alpha as f32) / (total as f32) < 0.5 {
- return false;
- }
- // Reject when the line itself looks tabular: ≥3 cells separated by font-size
- // gaps. A bold body-size line with that many cell tracks is almost always a
- // table header row, not a section heading. Without this guard, multi-line
- // table headers ("Model Method ... | F1 BLEU-1 F1 BLEU-1 ...") get promoted
- // to H3 instead of being absorbed by the table detector.
- // Tabular shape rejection. Two passes because the projection sometimes
- // collapses a wide multi-column line into a single span with column-
- // padding spaces — span-based detection misses those.
- if split_cells(line).len() >= TABLE_MIN_COLUMNS {
- return false;
- }
- // Text-based fallback: ≥3 tokens separated by runs of 2+ spaces (the
- // projection inserts alignment padding between cells) → table header row
- // collapsed into one span, not a section heading.
- let multi_space_tokens = text
- .split(" ")
- .map(|s| s.trim())
- .filter(|s| !s.is_empty())
- .count();
- if multi_space_tokens >= TABLE_MIN_COLUMNS {
- return false;
- }
- let gap_above_ok = match prev {
- None => true,
- Some(p) => !continues_paragraph(p, line),
- };
- if !gap_above_ok {
- return false;
- }
- let gap_below_ok = match next {
- None => true,
- Some(n) => !continues_paragraph(line, n),
- };
- gap_below_ok
-}
-
-/// Characters recognized as bullet markers when followed by whitespace.
-/// Limited to glyphs that are unlikely to appear at line-start in normal prose.
-const BULLET_CHARS: &[char] = &['•', '·', '◦', '▪', '▸', '▶', '●', '○', '■', '□'];
-
-/// Detect a list marker at the start of `text`. Returns `(ordered, marker_str,
-/// remainder)` when matched; otherwise `None`.
-///
-/// Recognizes:
-/// - Unicode bullet characters (`BULLET_CHARS`) followed by whitespace.
-/// - Decimal-prefix markers like `1.` / `1)` / `12.` / `12)` followed by
-/// whitespace — kept strict (digits only) so things like footnote callers
-/// (`1` alone) and section refs (`A.1`) don't match.
-fn parse_list_marker(text: &str) -> Option<(bool, String, &str)> {
- let trimmed = text.trim_start();
- if trimmed.is_empty() {
- return None;
- }
-
- let mut chars = trimmed.chars();
- let first = chars.next()?;
-
- // Unicode bullet
- if BULLET_CHARS.contains(&first) {
- let rest = chars.as_str();
- if let Some(rest_trim) = rest.strip_prefix(|c: char| c.is_whitespace()) {
- return Some((false, first.to_string(), rest_trim.trim_start()));
- }
- }
-
- // Decimal: 1. / 1) / 12. / 12)
- if first.is_ascii_digit() {
- let mut digit_end = 1;
- for c in trimmed[1..].chars() {
- if c.is_ascii_digit() {
- digit_end += c.len_utf8();
- } else {
- break;
- }
- }
- // Cap to keep us from matching page-number-like prefixes
- if digit_end <= 3 {
- let after_digits = &trimmed[digit_end..];
- let mut after_iter = after_digits.chars();
- if let Some(punct) = after_iter.next()
- && (punct == '.' || punct == ')')
- {
- let after_punct = after_iter.as_str();
- if let Some(rest_trim) = after_punct.strip_prefix(|c: char| c.is_whitespace()) {
- let marker = format!("{}{}", &trimmed[..digit_end], punct);
- return Some((true, marker, rest_trim.trim_start()));
- }
- }
- }
- }
-
- None
-}
-
-/// Returns true if any span on the line is rotated more than ~5° off
-/// horizontal — used to exclude sidebar / margin-stamp text (arXiv banners,
-/// watermarks, vertical legends) from the body-size and heading-size
-/// histograms so it doesn't compete with normal-flow text for heading slots.
-fn is_rotated_line(line: &ProjectedLine) -> bool {
- line.spans.iter().any(|s| {
- let r = s.rotation.abs() % 360.0;
- // Anything more than ~5° off the horizontal axes is "rotated" for
- // our purposes. 0° and 180° are both horizontal text.
- !(r < 5.0 || (175.0..=185.0).contains(&r) || (355.0..=360.0).contains(&r))
- })
-}
-
-/// Compute the body font size as the char-weighted mode across all lines in
-/// all pages. Rotated lines are excluded so a long rotated sidebar can't
-/// pull the body estimate. Falls back to `0.0` when no font-size info is
-/// available.
-pub fn compute_body_size(pages: &[ParsedPage]) -> f32 {
- use std::collections::HashMap;
- let mut weights: HashMap<u32, (f32, usize)> = HashMap::new();
- for page in pages {
- for line in &page.projected_lines {
- if is_rotated_line(line) {
- continue;
- }
- let size = line.dominant_font_size;
- if size <= 0.0 {
- continue;
- }
- let chars = line.text.chars().count().max(1);
- let key = (size * 100.0).round() as u32;
- let entry = weights.entry(key).or_insert((size, 0));
- entry.1 += chars;
- }
- }
- weights
- .values()
- .max_by_key(|(_, n)| *n)
- .map(|(s, _)| *s)
- .unwrap_or(0.0)
-}
-
-/// Minimum total non-whitespace characters across all occurrences at a font
-/// size for it to qualify as a heading level. Calibrated against `paper.pdf`:
-/// the 30pt chart-legend tokens ("A-mem"×2 + "Base"×2 = 18-20 chars) need to
-/// fail this filter, while the 14.35pt title ("A-MEM: Agentic Memory for LLM
-/// Agents" = 31 chars on a single line) needs to pass. 25 is the gap. Smaller
-/// single-word headings like a lone "Summary" on a short doc still survive
-/// because they share their font size with other (larger) headings in the
-/// histogram entry.
-const MIN_HEADING_TOTAL_CHARS: usize = 25;
-
-/// Maximum average characters per line for a size to qualify as a heading.
-/// A "size larger than body" with very long lines is almost always a
-/// large-print body block (callouts, footnotes-as-display, intro paragraph),
-/// not a real heading.
-const MAX_HEADING_AVG_LINE_CHARS: f32 = 200.0;
-
-/// Minimum fraction of non-whitespace chars at a size that must be alphabetic
-/// for it to qualify as a heading. Drops sizes dominated by digits (graph
-/// axes, results tables, math display) which otherwise pollute the top
-/// heading slots.
-const MIN_HEADING_ALPHA_RATIO: f32 = 0.5;
-
-/// Build a heading-size → level map: sizes strictly larger than `body_size`,
-/// filtered to those with at least `MIN_HEADING_LINES` distinct occurrences
-/// (drops one-off equation/figure-label artifacts), sorted descending, mapped
-/// to levels 1..=MAX_HEADING_LEVELS.
-pub fn build_heading_map(pages: &[ParsedPage], body_size: f32) -> Vec<(f32, u8)> {
- use std::collections::HashMap;
- // (size_key → (size, line_count, total_chars, alpha_chars))
- let mut sizes: HashMap<u32, (f32, usize, usize, usize)> = HashMap::new();
- for page in pages {
- for line in &page.projected_lines {
- if is_rotated_line(line) {
- continue;
- }
- let size = line.dominant_font_size;
- if size > body_size + HEADING_SIZE_EPSILON {
- let key = (size * 100.0).round() as u32;
- let entry = sizes.entry(key).or_insert((size, 0, 0, 0));
- entry.1 += 1;
- for c in line.text.chars() {
- if c.is_whitespace() {
- continue;
- }
- entry.2 += 1;
- if c.is_alphabetic() {
- entry.3 += 1;
- }
- }
- }
- }
- }
- let all: Vec<(f32, usize, usize, usize)> = sizes.into_values().collect();
- // Always apply quality filters: total-char floor, average-line cap, and
- // alpha-ratio floor. The total-char floor (rather than a line-count one)
- // lets one-off titles survive — a 31-char title on a single line passes —
- // while still rejecting chart-legend tokens like "A-mem" + "Base" that
- // total fewer chars across 4 occurrences than a single heading line does.
- let mut kept: Vec<f32> = all
- .iter()
- .filter(|(_, lines, chars, alpha)| {
- let alpha_ratio = if *chars == 0 {
- 0.0
- } else {
- (*alpha as f32) / (*chars as f32)
- };
- *chars >= MIN_HEADING_TOTAL_CHARS
- && (*chars as f32 / (*lines).max(1) as f32) <= MAX_HEADING_AVG_LINE_CHARS
- && alpha_ratio >= MIN_HEADING_ALPHA_RATIO
- })
- .map(|(s, _, _, _)| *s)
- .collect();
- kept.sort_by(|a, b| b.total_cmp(a));
- kept.truncate(MAX_HEADING_LEVELS);
- kept.into_iter()
- .enumerate()
- .map(|(i, s)| (s, (i + 1) as u8))
- .collect()
-}
-
-fn heading_level_for(size: f32, heading_map: &[(f32, u8)]) -> Option<u8> {
- for (s, level) in heading_map {
- if (size - *s).abs() < FONT_SIZE_HEADING_TOLERANCE {
- return Some(*level);
- }
- }
- None
-}
-
-/// Collapse runs of whitespace into single spaces. The projected text from
-/// `projection.rs` pads with column-alignment spaces (e.g. `for instance`)
-/// which look fine as a layout grid but are wrong for prose.
-fn collapse_whitespace(s: &str) -> String {
- let mut out = String::with_capacity(s.len());
- let mut prev_space = false;
- for c in s.chars() {
- if c.is_whitespace() {
- if !prev_space && !out.is_empty() {
- out.push(' ');
- }
- prev_space = true;
- } else {
- out.push(c);
- prev_space = false;
- }
- }
- if out.ends_with(' ') {
- out.pop();
- }
- out
-}
-
-/// Append `next` to `accum` as the continuation of a paragraph. De-hyphenates
-/// when `accum` ends with `-` and `next` starts with an ASCII lowercase letter;
-/// otherwise joins with a single space.
-#[cfg(test)]
-fn append_paragraph_line(accum: &mut String, next: &str) {
- let next = collapse_whitespace(next);
- if next.is_empty() {
- return;
- }
- if accum.is_empty() {
- accum.push_str(&next);
- return;
- }
- let dehyphenate =
- accum.ends_with('-') && next.chars().next().is_some_and(|c| c.is_ascii_lowercase());
- if dehyphenate {
- accum.pop(); // drop the '-'
- accum.push_str(&next);
- } else {
- accum.push(' ');
- accum.push_str(&next);
- }
-}
-
-/// Decide whether `cur` continues the paragraph started by `prev`.
-fn continues_paragraph(prev: &ProjectedLine, cur: &ProjectedLine) -> bool {
- // Anchor only signals a paragraph break when one of the lines is clearly
- // centered while the other isn't — justified prose routinely alternates
- // between Left / Right / Floating dominant anchors as line widths flex,
- // and treating those as paragraph breaks shreds normal text.
- let centered_mismatch = (prev.anchor == Anchor::Center) ^ (cur.anchor == Anchor::Center);
- if centered_mismatch {
- return false;
- }
- if (prev.dominant_font_size - cur.dominant_font_size).abs() > FONT_SIZE_PARAGRAPH_TOLERANCE {
- return false;
- }
- if prev.region_path != cur.region_path {
- // Cross-region continuation: the same paragraph can wrap from the
- // bottom of one column into the top of the next. Only bridge regions
- // when the previous line clearly breaks mid-sentence (no terminal
- // punctuation) AND the next line starts with a lowercase letter — a
- // strict signal that catches the column-wrap case while rejecting
- // unrelated paragraphs that happen to sit in adjacent leaves.
- let prev_trim = prev.text.trim_end();
- let ends_open = !prev_trim.ends_with(|c: char| {
- matches!(
- c,
- '.' | '!' | '?' | ':' | ';' | '”' | '"' | ')' | ']' | '。' | '』' | '」'
- )
- });
- let starts_lower = cur
- .text
- .trim_start()
- .chars()
- .next()
- .is_some_and(|c| c.is_lowercase());
- return ends_open && starts_lower;
- }
- if (prev.indent_x - cur.indent_x).abs() > INDENT_TOLERANCE && cur.anchor == Anchor::Left {
- // Indent change on a left-aligned block usually means a new paragraph
- // (block-quote, list, indented passage, etc.). Allow first-line indent
- // by checking only when the *next* line shifts right relative to prev.
- if cur.indent_x > prev.indent_x + INDENT_TOLERANCE {
- return false;
- }
- }
- // Vertical gap check.
- let prev_bottom = prev.bbox.y + prev.bbox.height;
- let gap = cur.bbox.y - prev_bottom;
- let line_height = prev.bbox.height.max(cur.bbox.height).max(1.0);
- gap <= line_height * PARAGRAPH_GAP_MULTIPLIER
-}
-
-/// Paragraph accumulator state. We track two parallel representations of the
-/// running paragraph text:
-///
-/// - `raw` — plain text (no emphasis markers). Used for the paragraph-uniform
-/// shortcut: if every contributing line had the same uniform style, we wrap
-/// the whole paragraph once with `wrap_emphasis(raw, …)` to avoid the
-/// `**foo** **bar** **baz**` per-line noise pymupdf4llm warns about.
-/// - `inline` — per-line markdown with emphasis baked in via
-/// `render_line_inline`. Used when the paragraph contains mid-line emphasis
-/// shifts or lines with differing uniform styles.
-///
-/// `uniform` is `Some((bold, italic))` while every line so far has been a
-/// uniformly-styled line sharing the same (bold, italic) flags, and `None` as
-/// soon as that invariant breaks.
-struct ParaAccum {
- raw: String,
- inline: String,
- last: ProjectedLine,
- uniform: Option<(bool, bool)>,
-}
-
-/// Returns the shared `SpanStyle` of `line` when every non-whitespace span on
-/// the line has the same style; `None` when spans disagree. Mono is folded
-/// into the style for the purpose of "uniform" — a fully-mono line is
-/// uniform-mono. Used by the paragraph-level optimization to decide whether
-/// to wrap once around the whole paragraph or fall back to per-line inline.
-fn line_uniform_style(line: &ProjectedLine) -> Option<SpanStyle> {
- let mut iter = line
- .spans
- .iter()
- .filter(|s| !s.text.trim().is_empty())
- .map(SpanStyle::from_item);
- let first = iter.next()?;
- for s in iter {
- if s != first {
- return None;
- }
- }
- Some(first)
-}
-
-/// Append `next_line` to a paragraph accumulator. Maintains both the `raw` and
-/// `inline` text representations and updates the running `uniform` flag.
-/// De-hyphenation runs on the `raw` boundary; the `inline` boundary mirrors it
-/// when the trailing char is still a literal `-` (i.e. the hyphen sits outside
-/// any emphasis wrap — the common case).
-fn append_to_paragraph(accum: &mut ParaAccum, next_line: &ProjectedLine) {
- let next_raw = collapse_whitespace(next_line.text.trim());
- if next_raw.is_empty() {
- return;
- }
- let next_inline = render_line_inline(next_line);
- let next_uniform = line_uniform_style(next_line);
-
- if accum.raw.is_empty() {
- accum.raw.push_str(&next_raw);
- accum.inline.push_str(&next_inline);
- accum.uniform = next_uniform.map(|s| (s.bold, s.italic));
- accum.last = next_line.clone();
- return;
- }
-
- let dehyphenate = accum.raw.ends_with('-')
- && next_raw
- .chars()
- .next()
- .is_some_and(|c| c.is_ascii_lowercase());
- if dehyphenate {
- accum.raw.pop();
- accum.raw.push_str(&next_raw);
- if accum.inline.ends_with('-') {
- accum.inline.pop();
- accum.inline.push_str(&next_inline);
- } else {
- // Hyphen sits inside an emphasis wrap — give up on stripping it
- // from `inline` cleanly. Join with a space; raw is still correct
- // for the uniform-paragraph case.
- accum.inline.push(' ');
- accum.inline.push_str(&next_inline);
- }
- } else {
- accum.raw.push(' ');
- accum.raw.push_str(&next_raw);
- accum.inline.push(' ');
- accum.inline.push_str(&next_inline);
- }
-
- // Maintain the running uniform-style flag.
- accum.uniform = match (accum.uniform, next_uniform) {
- (Some(cur), Some(s)) if cur == (s.bold, s.italic) => Some(cur),
- _ => None,
- };
- accum.last = next_line.clone();
-}
-
-/// Resolve a `ParaAccum` to a `Block::Paragraph`. When the paragraph was
-/// uniformly styled across all lines, return the raw text with block-level
-/// `bold`/`italic` flags set so the renderer wraps it once. Otherwise return
-/// the per-line inline-styled text with the flags cleared.
-fn paragraph_from_accum(accum: ParaAccum) -> Block {
- match accum.uniform {
- Some((bold, italic)) if bold || italic => Block::Paragraph {
- text: escape_inline(&accum.raw),
- bold,
- italic,
- },
- Some(_) => Block::Paragraph {
- // Uniformly plain — no emphasis markers anywhere, so the raw text
- // (with markdown specials escaped) is the right rendering.
- text: escape_inline(&accum.raw),
- bold: false,
- italic: false,
- },
- None => Block::Paragraph {
- text: accum.inline,
- bold: false,
- italic: false,
- },
- }
-}
-
-/// Normalize a line for cross-page header/footer matching. Lowercases,
-/// collapses whitespace, and replaces every run of ASCII digits with `#` so
-/// `Page 1 of 6` and `Page 2 of 6` collapse to the same key.
-fn normalize_for_repetition(s: &str) -> String {
- let collapsed = collapse_whitespace(s).to_lowercase();
- let mut out = String::with_capacity(collapsed.len());
- let mut in_digits = false;
- for c in collapsed.chars() {
- if c.is_ascii_digit() {
- if !in_digits {
- out.push('#');
- in_digits = true;
- }
- } else {
- out.push(c);
- in_digits = false;
- }
- }
- out
-}
-
-/// Cross-page repetition detector. Returns the set of normalized strings that
-/// appear in the top or bottom band of ≥ `HEADER_FOOTER_MIN_FRACTION` of
-/// pages (capped below by `HEADER_FOOTER_MIN_PAGES`). The caller uses this to
-/// filter `ProjectedLine`s before classification.
-///
-/// "Same band" means a line whose top is within `HEADER_BAND_FRACTION` of the
-/// page top (header) or whose bottom is within `FOOTER_BAND_FRACTION` of the
-/// page bottom (footer). Header and footer bands are tracked separately so a
-/// company name that appears as both a header and a body-section title on
-/// different pages isn't stripped from the body.
-pub fn compute_header_footer_set(pages: &[ParsedPage]) -> std::collections::HashSet<String> {
- use std::collections::{HashMap, HashSet};
- let mut set: HashSet<String> = HashSet::new();
- if pages.len() < HEADER_FOOTER_MIN_PAGES {
- return set;
- }
- // Two counters keyed by `(band, normalized_text)` — band is `'h'` or `'f'`.
- let mut counts: HashMap<(char, String), HashSet<usize>> = HashMap::new();
- for page in pages {
- let header_cutoff = page.page_height * HEADER_BAND_FRACTION;
- let footer_cutoff = page.page_height * (1.0 - FOOTER_BAND_FRACTION);
- for line in &page.projected_lines {
- let text = line.text.trim();
- if text.is_empty() {
- continue;
- }
- let norm = normalize_for_repetition(text);
- if norm.is_empty() {
- continue;
- }
- // Header band: top of line within the top band.
- if line.bbox.y <= header_cutoff {
- counts
- .entry(('h', norm.clone()))
- .or_default()
- .insert(page.page_number);
- }
- // Footer band: bottom of line at or below the footer cutoff.
- let line_bottom = line.bbox.y + line.bbox.height;
- if line_bottom >= footer_cutoff {
- counts
- .entry(('f', norm))
- .or_default()
- .insert(page.page_number);
- }
- }
- }
- let threshold = (pages.len() as f32 * HEADER_FOOTER_MIN_FRACTION)
- .ceil()
- .max(HEADER_FOOTER_MIN_PAGES as f32) as usize;
- for ((_, norm), pages_seen) in counts {
- if pages_seen.len() >= threshold {
- set.insert(norm);
- }
- }
- set
-}
-
-/// Returns true if `line` (located on `page`) matches the running
-/// header/footer set: the line sits in the top or bottom band AND its
-/// normalized text is in `header_footer`.
-fn is_header_or_footer(
- line: &ProjectedLine,
- page: &ParsedPage,
- header_footer: &std::collections::HashSet<String>,
-) -> bool {
- if header_footer.is_empty() {
- return false;
- }
- let header_cutoff = page.page_height * HEADER_BAND_FRACTION;
- let footer_cutoff = page.page_height * (1.0 - FOOTER_BAND_FRACTION);
- let in_band = line.bbox.y <= header_cutoff || line.bbox.y + line.bbox.height >= footer_cutoff;
- if !in_band {
- return false;
- }
- let norm = normalize_for_repetition(line.text.trim());
- header_footer.contains(&norm)
-}
-
-/// Classify a single page's `ProjectedLine`s into blocks.
-pub fn classify_page(page: &ParsedPage, heading_map: &[(f32, u8)]) -> Vec<Block> {
- classify_page_with_filters(page, heading_map, &std::collections::HashSet::new())
-}
-
-/// Same as `classify_page` but also strips lines matching a precomputed
-/// running header/footer set. Use this when emitting a whole document so
-/// repeating chrome (titles, page numbers) doesn't show up in every page.
-pub fn classify_page_with_filters(
- page: &ParsedPage,
- heading_map: &[(f32, u8)],
- header_footer: &std::collections::HashSet<String>,
-) -> Vec<Block> {
- let mut blocks: Vec<Block> = Vec::new();
- let mut paragraph: Option<ParaAccum> = None;
- // Active fenced code block being accumulated (consecutive `all_mono` lines).
- let mut code: Option<Vec<String>> = None;
- // Tracks the "level 0" indent of the current contiguous list run so we can
- // bucket deeper items into nesting levels. Reset whenever a non-list block
- // breaks the run.
- let mut list_base_indent: Option<f32> = None;
- // Index into `blocks` of the most recent ListItem in the current run — used
- // to merge wrapped continuation lines into the same item.
- let mut last_list_item_idx: Option<usize> = None;
- // Most recent ProjectedLine appended to the active list item, for
- // gap/font-size checks on continuation lines.
- let mut last_list_line: Option<ProjectedLine> = None;
-
- let flush_paragraph = |blocks: &mut Vec<Block>, p: Option<ParaAccum>| {
- if let Some(acc) = p
- && !acc.raw.trim().is_empty()
- {
- blocks.push(paragraph_from_accum(acc));
- }
- };
- let flush_code = |blocks: &mut Vec<Block>, c: Option<Vec<String>>| {
- if let Some(lines) = c
- && !lines.is_empty()
- {
- blocks.push(Block::CodeBlock { lines });
- }
- };
-
- let debug = std::env::var("LITEPARSE_DEBUG_MD").is_ok();
-
- // Strip running header/footer lines up-front so they don't leak into
- // table detection (a repeating two-column footer would otherwise look
- // like a 2-row table) or paragraph grouping.
- let filtered_owned: Vec<ProjectedLine> = if header_footer.is_empty() {
- Vec::new()
- } else {
- page.projected_lines
- .iter()
- .filter(|l| !is_header_or_footer(l, page, header_footer))
- .cloned()
- .collect()
- };
- let lines: &[ProjectedLine] = if header_footer.is_empty() {
- &page.projected_lines
- } else {
- &filtered_owned
- };
-
- // Pre-pass: detect tabular regions so the per-line classifier below can
- // skip over them. Tables take priority over heading / paragraph / list
- // classification because a row of bold short cells would otherwise be
- // misread as a heading or list item.
- //
- // Two detectors run in sequence: ruled-grid (path-based, strongest signal)
- // and the borderless column-alignment fallback. Where ranges overlap, the
- // ruled output wins.
- let ruled_runs = detect_ruled_tables(lines, &page.graphics, page.page_width, page.page_height);
- let borderless_runs = detect_tables(lines);
- let table_runs = merge_table_runs(ruled_runs, borderless_runs);
-
- // Suppress HRs that fall inside a detected table's y-range — they're the
- // table's own row dividers, not document-level section breaks. Build the
- // y-extents once before we move table_runs into the iterator.
- // Extend each table's HR-suppression band upward to cover any
- // header/sub-header rows we didn't absorb into the table. The expansion
- // is a few row heights — large enough to catch a 2–3 line bold/italic
- // header sitting just above the table, small enough not to swallow a
- // real section divider belonging to a different block. The downward
- // edge gets a small slack to catch HRs drawn flush with the last row.
- const TABLE_HR_SUPPRESS_HEADROOM_ROWS: f32 = 4.0;
- let table_y_extents: Vec<(f32, f32)> = table_runs
- .iter()
- .map(|run| {
- let top_line = &lines[run.start];
- let row_h = top_line.bbox.height.max(8.0);
- let top = top_line.bbox.y - row_h * TABLE_HR_SUPPRESS_HEADROOM_ROWS;
- let last = &lines[run.end.saturating_sub(1).max(run.start)];
- let bot = last.bbox.y + last.bbox.height;
- (top, bot)
- })
- .collect();
-
- let mut table_iter = table_runs.into_iter().peekable();
-
- // Pre-pass: detect horizontal rules from vector graphics so they can be
- // emitted in document order between surrounding text lines.
- let hr_ys: Vec<f32> = detect_horizontal_rules(page)
- .into_iter()
- .filter(|y| {
- !table_y_extents
- .iter()
- .any(|(top, bot)| *y >= *top - 2.0 && *y <= *bot + 2.0)
- })
- .collect();
- let mut hr_iter = hr_ys.into_iter().peekable();
-
- // Emit any HRs whose y is at or above `before_y`. Flushes the active
- // paragraph/code/list state first so the rule lands as its own block.
- let emit_hrs_before = |blocks: &mut Vec<Block>,
- paragraph: &mut Option<ParaAccum>,
- code: &mut Option<Vec<String>>,
- list_base: &mut Option<f32>,
- last_item: &mut Option<usize>,
- last_line: &mut Option<ProjectedLine>,
- hr_iter: &mut std::iter::Peekable<std::vec::IntoIter<f32>>,
- before_y: f32| {
- while let Some(&hy) = hr_iter.peek() {
- if hy > before_y {
- break;
- }
- hr_iter.next();
- flush_paragraph(blocks, paragraph.take());
- flush_code(blocks, code.take());
- *list_base = None;
- *last_item = None;
- *last_line = None;
- blocks.push(Block::HorizontalRule);
- }
- };
-
- let mut idx = 0;
- while idx < lines.len() {
- if let Some(run) = table_iter.peek()
- && run.start == idx
- {
- // Flush any HRs above this table's top edge first.
- let table_top = lines[run.start].bbox.y;
- emit_hrs_before(
- &mut blocks,
- &mut paragraph,
- &mut code,
- &mut list_base_indent,
- &mut last_list_item_idx,
- &mut last_list_line,
- &mut hr_iter,
- table_top,
- );
- flush_paragraph(&mut blocks, paragraph.take());
- flush_code(&mut blocks, code.take());
- list_base_indent = None;
- last_list_item_idx = None;
- last_list_line = None;
- let run = table_iter.next().unwrap();
- blocks.push(run.block);
- idx = run.end;
- continue;
- }
- let line = &lines[idx];
- // Emit any HRs that fall above this line.
- emit_hrs_before(
- &mut blocks,
- &mut paragraph,
- &mut code,
- &mut list_base_indent,
- &mut last_list_item_idx,
- &mut last_list_line,
- &mut hr_iter,
- line.bbox.y,
- );
- idx += 1;
- let text = line.text.trim();
- if text.is_empty() {
- continue;
- }
- // Skip rotated text (vertical sidebars, arXiv banners, watermarks).
- // Including it would either inject a paragraph of disconnected
- // characters or be misclassified as a heading.
- if is_rotated_line(line) {
- continue;
- }
- if debug {
- eprintln!(
- "[md] y={:.1} h={:.1} size={:.2} anchor={:?} indent={:.1} text={:?}",
- line.bbox.y,
- line.bbox.height,
- line.dominant_font_size,
- line.anchor,
- line.indent_x,
- text
- );
- }
-
- // Code block detection runs first so a mono heading-shaped line
- // (rare but plausible — e.g., a code identifier in a large mono font)
- // still becomes code. Mono content also wouldn't make a useful
- // heading.
- if line.all_mono {
- flush_paragraph(&mut blocks, paragraph.take());
- list_base_indent = None;
- last_list_item_idx = None;
- last_list_line = None;
- code.get_or_insert_with(Vec::new)
- .push(line.text.trim_end().to_string());
- continue;
- }
- // Any non-mono line ends the current code block (if any).
- flush_code(&mut blocks, code.take());
-
- if let Some(level) = heading_level_for(line.dominant_font_size, heading_map) {
- flush_paragraph(&mut blocks, paragraph.take());
- list_base_indent = None;
- last_list_item_idx = None;
- last_list_line = None;
- blocks.push(Block::Heading {
- level,
- text: collapse_whitespace(text),
- });
- continue;
- }
-
- // List item?
- if let Some((ordered, marker, rest)) = parse_list_marker(text) {
- flush_paragraph(&mut blocks, paragraph.take());
- let base = *list_base_indent.get_or_insert(line.indent_x);
- let level = (((line.indent_x - base) / LIST_INDENT_STEP_PT)
- .round()
- .max(0.0)) as u8;
- last_list_item_idx = Some(blocks.len());
- last_list_line = Some(line.clone());
- // Render the list-item text via the inline pipeline so per-span
- // emphasis surfaces. We strip the marker from `rest` (already
- // done by `parse_list_marker`), but emphasis lives on `line.spans`,
- // which still contain the marker span — render the line and then
- // peel the marker off the front of the rendered string.
- let item_text = render_list_item_text(line, &marker, rest);
- blocks.push(Block::ListItem {
- ordered,
- marker,
- level,
- text: item_text,
- bold: false,
- italic: false,
- });
- continue;
- }
-
- // Continuation of a list item: same gap/font rules as paragraphs.
- // Footnote-style continuations often left-flush below the marker's
- // hanging indent, so we don't require indent ≥ marker indent.
- if let Some(idx) = last_list_item_idx
- && let Some(prev_line) = last_list_line.as_ref()
- && continues_paragraph(prev_line, line)
- && let Some(Block::ListItem {
- text: prev_text, ..
- }) = blocks.get_mut(idx)
- {
- // De-hyphenate against the prior rendered text, then append the
- // inline-styled continuation.
- let cont_inline = render_line_inline(line);
- append_inline_continuation(prev_text, text, &cont_inline);
- last_list_line = Some(line.clone());
- continue;
- }
-
- // Bold body-size heading. Section headings in academic / technical
- // PDFs are routinely body-sized + bold (e.g. "Abstract",
- // "1 Introduction"); without this rule they look just like a bold
- // first sentence of a paragraph. Runs after list-marker detection so
- // bold bullet items stay as list items.
- let prev_for_gap = paragraph
- .as_ref()
- .map(|p| &p.last)
- .or(last_list_line.as_ref());
- let next_for_gap = lines.get(idx);
- if looks_like_bold_heading(line, prev_for_gap, next_for_gap) {
- flush_paragraph(&mut blocks, paragraph.take());
- list_base_indent = None;
- last_list_item_idx = None;
- last_list_line = None;
- // Level: one deeper than the deepest size-based level we already
- // have. With an empty heading_map this lands on H1; with a full
- // 6-level map it caps at H6.
- let level = (heading_map.len() as u8 + 1).clamp(1, MAX_HEADING_LEVELS as u8);
- blocks.push(Block::Heading {
- level,
- text: collapse_whitespace(text),
- });
- continue;
- }
-
- match paragraph.as_mut() {
- Some(acc) if continues_paragraph(&acc.last, line) => {
- append_to_paragraph(acc, line);
- }
- _ => {
- flush_paragraph(&mut blocks, paragraph.take());
- list_base_indent = None;
- last_list_item_idx = None;
- last_list_line = None;
- let inline = render_line_inline(line);
- let raw = collapse_whitespace(text);
- let uniform = line_uniform_style(line).map(|s| (s.bold, s.italic));
- paragraph = Some(ParaAccum {
- raw,
- inline,
- last: line.clone(),
- uniform,
- });
- }
- }
- }
-
- flush_paragraph(&mut blocks, paragraph.take());
- flush_code(&mut blocks, code.take());
- // Flush any trailing HRs that sat below the last text line.
- emit_hrs_before(
- &mut blocks,
- &mut paragraph,
- &mut code,
- &mut list_base_indent,
- &mut last_list_item_idx,
- &mut last_list_line,
- &mut hr_iter,
- f32::INFINITY,
- );
- blocks
-}
-
-/// Wrap `text` in markdown emphasis markers based on `bold`/`italic`. Both →
-/// `***text***`. Headings deliberately skip this (the `#` is the emphasis).
-fn wrap_emphasis(text: &str, bold: bool, italic: bool) -> String {
- if text.trim().is_empty() {
- return text.to_string();
- }
- match (bold, italic) {
- (true, true) => format!("***{text}***"),
- (true, false) => format!("**{text}**"),
- (false, true) => format!("*{text}*"),
- (false, false) => text.to_string(),
- }
-}
-
-/// Render a list of blocks to a markdown string.
-pub fn render_blocks(blocks: &[Block]) -> String {
- let mut out = String::new();
- for (i, block) in blocks.iter().enumerate() {
- if i > 0 {
- // Consecutive list items render as a tight list (single newline).
- // Everything else gets a blank line between blocks.
- let tight = matches!(block, Block::ListItem { .. })
- && matches!(blocks[i - 1], Block::ListItem { .. });
- if tight {
- out.push('\n');
- } else {
- out.push_str("\n\n");
- }
- }
- match block {
- Block::Heading { level, text } => {
- let level = (*level).clamp(1, 6) as usize;
- out.push_str(&"#".repeat(level));
- out.push(' ');
- out.push_str(text);
- }
- Block::Paragraph { text, bold, italic } => {
- out.push_str(&wrap_emphasis(text, *bold, *italic));
- }
- Block::ListItem {
- ordered,
- marker,
- level,
- text,
- bold,
- italic,
- } => {
- let indent = " ".repeat((*level).min(6) as usize);
- out.push_str(&indent);
- if *ordered {
- // Preserve the original marker (e.g. `138.` for footnotes
- // or `iii)` for roman numerals) so semantic numbering
- // survives the round-trip. CommonMark requires `1.` /
- // `1)` style but most LLM consumers tolerate alt forms,
- // and the alternative — renumbering as `1.` — drops info.
- out.push_str(marker);
- out.push(' ');
- } else {
- out.push_str("- ");
- }
- out.push_str(&wrap_emphasis(text, *bold, *italic));
- }
- Block::Table { header, rows } => {
- let column_count = header
- .as_ref()
- .map(|h| h.len())
- .or_else(|| rows.first().map(|r| r.len()))
- .unwrap_or(0);
- if column_count == 0 {
- continue;
- }
- if let Some(h) = header {
- out.push_str("| ");
- for (i, cell) in h.iter().enumerate() {
- if i > 0 {
- out.push_str(" | ");
- }
- out.push_str(&escape_table_cell(cell));
- }
- out.push_str(" |\n");
- } else {
- // CommonMark/GFM requires a header row before the
- // separator; synthesize a blank header so renderers that
- // refuse header-less tables still display the body.
- out.push_str("|");
- for _ in 0..column_count {
- out.push_str(" |");
- }
- out.push('\n');
- }
- out.push('|');
- for _ in 0..column_count {
- out.push_str("---|");
- }
- for row in rows {
- out.push_str("\n| ");
- for (i, cell) in row.iter().enumerate() {
- if i > 0 {
- out.push_str(" | ");
- }
- out.push_str(&escape_table_cell(cell));
- }
- out.push_str(" |");
- }
- }
- Block::GridFallback { lines } => {
- out.push_str("```text\n");
- for line in lines {
- out.push_str(line);
- out.push('\n');
- }
- out.push_str("```");
- }
- Block::CodeBlock { lines } => {
- // Pick a fence that doesn't appear inside the body. Standard
- // triple-backtick handles ~all real-world code; bump to ~~~ if
- // the body itself contains ``` (rare).
- let fence = if lines.iter().any(|l| l.contains("```")) {
- "~~~"
- } else {
- "```"
- };
- out.push_str(fence);
- out.push('\n');
- for line in lines {
- out.push_str(line);
- out.push('\n');
- }
- out.push_str(fence);
- }
- Block::HorizontalRule => {
- out.push_str("---");
- }
- Block::Blank => {}
- }
- }
- out
-}
-
-#[cfg(test)]
-mod tests {
- use super::*;
- use crate::types::{Rect, TextItem};
-
- fn line(text: &str, x: f32, y: f32, h: f32, size: f32) -> ProjectedLine {
- ProjectedLine {
- text: text.into(),
- bbox: Rect {
- x,
- y,
- width: text.chars().count() as f32 * (size * 0.5),
- height: h,
- },
- anchor: Anchor::Left,
- indent_x: x,
- dominant_font_size: size,
- dominant_font_name: Some("Arial".into()),
- all_bold: false,
- all_italic: false,
- all_mono: false,
- all_strike: false,
- spans: vec![TextItem::default()],
- region_path: Vec::new(),
- mcid: None,
- }
- }
-
- fn page(lines: Vec<ProjectedLine>) -> ParsedPage {
- ParsedPage {
- page_number: 1,
- page_width: 612.0,
- page_height: 792.0,
- text: String::new(),
- text_items: vec![],
- projected_lines: lines,
- regions: crate::types::Region::default(),
- graphics: vec![],
- figures: vec![],
- }
- }
-
- #[test]
- fn body_size_picks_most_common() {
- let pages = vec![page(vec![
- line("Title", 50.0, 50.0, 18.0, 18.0),
- line("body line one", 50.0, 80.0, 10.0, 10.0),
- line("body line two", 50.0, 92.0, 10.0, 10.0),
- line("body line three", 50.0, 104.0, 10.0, 10.0),
- ])];
- let body = compute_body_size(&pages);
- assert!((body - 10.0).abs() < 0.01, "body size = {body}");
- }
-
- #[test]
- fn heading_map_descending_levels() {
- // Heading text needs to clear `MIN_HEADING_TOTAL_CHARS` (25) so the
- // size qualifies as a real heading rather than chart-legend noise.
- let pages = vec![page(vec![
- line("The largest heading on the page", 50.0, 50.0, 24.0, 24.0),
- line("A smaller heading right below it", 50.0, 80.0, 18.0, 18.0),
- // Several lines of body so it beats the heading text in the
- // char-weighted body-size mode.
- line(
- "body text line one with plenty of content",
- 50.0,
- 110.0,
- 10.0,
- 10.0,
- ),
- line(
- "body text line two with plenty of content",
- 50.0,
- 122.0,
- 10.0,
- 10.0,
- ),
- line(
- "body text line three with even more content",
- 50.0,
- 134.0,
- 10.0,
- 10.0,
- ),
- line(
- "body text line four with even more content",
- 50.0,
- 146.0,
- 10.0,
- 10.0,
- ),
- ])];
- let body = compute_body_size(&pages);
- let map = build_heading_map(&pages, body);
- assert_eq!(map.len(), 2);
- assert_eq!(map[0].1, 1);
- assert_eq!(map[1].1, 2);
- assert!(map[0].0 > map[1].0);
- }
-
- #[test]
- fn classify_emits_heading_and_paragraph() {
- let p = page(vec![
- line("Title of the document goes here", 50.0, 50.0, 18.0, 18.0),
- line("First sentence of the para-", 50.0, 80.0, 10.0, 10.0),
- line("graph continues here.", 50.0, 92.0, 10.0, 10.0),
- line("Another paragraph.", 50.0, 130.0, 10.0, 10.0),
- ]);
- let pages = vec![p];
- let body = compute_body_size(&pages);
- let map = build_heading_map(&pages, body);
- let blocks = classify_page(&pages[0], &map);
- assert_eq!(blocks.len(), 3);
- match &blocks[0] {
- Block::Heading { level, text } => {
- assert_eq!(*level, 1);
- assert_eq!(text, "Title of the document goes here");
- }
- other => panic!("expected heading, got {other:?}"),
- }
- match &blocks[1] {
- Block::Paragraph { text: t, .. } => {
- assert!(t.contains("paragraph continues"), "got: {t}");
- assert!(!t.contains("para-"), "de-hyphenation failed: {t}");
- }
- other => panic!("expected paragraph, got {other:?}"),
- }
- match &blocks[2] {
- Block::Paragraph { text: t, .. } => assert_eq!(t, "Another paragraph."),
- other => panic!("expected paragraph, got {other:?}"),
- }
- }
-
- #[test]
- fn paragraph_break_on_big_gap() {
- let p = page(vec![
- line("Line A.", 50.0, 80.0, 10.0, 10.0),
- line("Line B.", 50.0, 200.0, 10.0, 10.0),
- ]);
- let pages = vec![p];
- let body = compute_body_size(&pages);
- let map = build_heading_map(&pages, body);
- let blocks = classify_page(&pages[0], &map);
- assert_eq!(blocks.len(), 2);
- }
-
- #[test]
- fn render_blocks_formats_markdown() {
- let blocks = vec![
- Block::Heading {
- level: 1,
- text: "Title".into(),
- },
- Block::Paragraph {
- text: "A paragraph.".into(),
- bold: false,
- italic: false,
- },
- Block::Heading {
- level: 2,
- text: "Sub".into(),
- },
- ];
- let s = render_blocks(&blocks);
- assert_eq!(s, "# Title\n\nA paragraph.\n\n## Sub");
- }
-
- #[test]
- fn parse_list_marker_bullets() {
- let (ordered, marker, rest) = parse_list_marker("• item one").unwrap();
- assert!(!ordered);
- assert_eq!(marker, "•");
- assert_eq!(rest, "item one");
- }
-
- #[test]
- fn parse_list_marker_decimal() {
- let (ordered, marker, rest) = parse_list_marker("1. first").unwrap();
- assert!(ordered);
- assert_eq!(marker, "1.");
- assert_eq!(rest, "first");
-
- let (ordered, marker, rest) = parse_list_marker("12) twelfth").unwrap();
- assert!(ordered);
- assert_eq!(marker, "12)");
- assert_eq!(rest, "twelfth");
- }
-
- #[test]
- fn parse_list_marker_rejects_prose() {
- assert!(parse_list_marker("This sentence.").is_none());
- // Bare digit with no terminator → not a list
- assert!(parse_list_marker("2023 was a year").is_none());
- // Footnote caller / page number style — no whitespace after
- assert!(parse_list_marker("1.5x growth").is_none());
- }
-
- #[test]
- fn classify_emits_list_items() {
- let p = page(vec![
- line("Intro line.", 50.0, 50.0, 10.0, 10.0),
- line("• first bullet", 60.0, 80.0, 10.0, 10.0),
- line("• second bullet", 60.0, 92.0, 10.0, 10.0),
- line("◦ nested item", 72.0, 104.0, 10.0, 10.0),
- line("• back to top", 60.0, 116.0, 10.0, 10.0),
- ]);
- let pages = vec![p];
- let body = compute_body_size(&pages);
- let map = build_heading_map(&pages, body);
- let blocks = classify_page(&pages[0], &map);
- let list_items: Vec<&Block> = blocks
- .iter()
- .filter(|b| matches!(b, Block::ListItem { .. }))
- .collect();
- assert_eq!(list_items.len(), 4);
- if let Block::ListItem { level, text, .. } = list_items[0] {
- assert_eq!(*level, 0);
- assert_eq!(text, "first bullet");
- } else {
- panic!();
- }
- // The "- nested item" line is indented +12pt from the base bullet.
- if let Block::ListItem { level, .. } = list_items[2] {
- assert_eq!(*level, 1);
- } else {
- panic!();
- }
- }
-
- #[test]
- fn render_lists_are_tight() {
- let blocks = vec![
- Block::Paragraph {
- text: "Intro.".into(),
- bold: false,
- italic: false,
- },
- Block::ListItem {
- ordered: false,
- marker: "•".into(),
- level: 0,
- text: "a".into(),
- bold: false,
- italic: false,
- },
- Block::ListItem {
- ordered: false,
- marker: "•".into(),
- level: 0,
- text: "b".into(),
- bold: false,
- italic: false,
- },
- Block::Paragraph {
- text: "Outro.".into(),
- bold: false,
- italic: false,
- },
- ];
- let s = render_blocks(&blocks);
- assert_eq!(s, "Intro.\n\n- a\n- b\n\nOutro.");
-
- // Ordered: original marker preserved
- let s = render_blocks(&[
- Block::ListItem {
- ordered: true,
- marker: "138.".into(),
- level: 0,
- text: "footnote".into(),
- bold: false,
- italic: false,
- },
- Block::ListItem {
- ordered: true,
- marker: "139.".into(),
- level: 0,
- text: "next footnote".into(),
- bold: false,
- italic: false,
- },
- ]);
- assert_eq!(s, "138. footnote\n139. next footnote");
- }
-
- fn mono_line(text: &str, y: f32) -> ProjectedLine {
- let mut l = line(text, 50.0, y, 10.0, 10.0);
- l.all_mono = true;
- l
- }
-
- #[test]
- fn classify_emits_code_block() {
- let p = page(vec![
- line("Intro line.", 50.0, 50.0, 10.0, 10.0),
- mono_line(" let x = 1;", 80.0),
- mono_line(" let y = x + 2;", 92.0),
- line("After the code.", 50.0, 120.0, 10.0, 10.0),
- ]);
- let pages = vec![p];
- let body = compute_body_size(&pages);
- let map = build_heading_map(&pages, body);
- let blocks = classify_page(&pages[0], &map);
- // Expect: Paragraph("Intro line."), CodeBlock(2 lines), Paragraph("After...")
- assert_eq!(blocks.len(), 3);
- match &blocks[1] {
- Block::CodeBlock { lines } => {
- assert_eq!(lines.len(), 2);
- assert!(lines[0].contains("let x = 1;"));
- assert!(lines[1].contains("let y = x + 2;"));
- }
- other => panic!("expected code block, got {other:?}"),
- }
- let s = render_blocks(&blocks);
- assert!(s.contains("```\n let x = 1;"));
- assert!(s.ends_with("After the code."));
- }
-
- #[test]
- fn classify_marks_paragraph_bold_when_all_lines_bold() {
- let mut a = line("Bold line one.", 50.0, 50.0, 10.0, 10.0);
- let mut b = line("bold continuation.", 50.0, 62.0, 10.0, 10.0);
- // Mark the underlying spans as bold so per-span style detection sees
- // it — the new inline pipeline reads from `spans`, not the per-line
- // `all_bold` shortcut flag.
- let bold_span = TextItem {
- text: "x".into(),
- font_name: Some("Arial-Bold".into()),
- ..Default::default()
- };
- a.spans = vec![bold_span.clone()];
- b.spans = vec![bold_span];
- a.all_bold = true;
- b.all_bold = true;
- let p = page(vec![a, b]);
- let pages = vec![p];
- let body = compute_body_size(&pages);
- let map = build_heading_map(&pages, body);
- let blocks = classify_page(&pages[0], &map);
- assert_eq!(blocks.len(), 1);
- match &blocks[0] {
- Block::Paragraph { bold, italic, .. } => {
- assert!(*bold);
- assert!(!*italic);
- }
- other => panic!("expected paragraph, got {other:?}"),
- }
- let s = render_blocks(&blocks);
- assert!(s.starts_with("**") && s.ends_with("**"), "got: {s}");
- }
-
- #[test]
- fn render_emphasis_combinations() {
- assert_eq!(wrap_emphasis("hi", false, false), "hi");
- assert_eq!(wrap_emphasis("hi", true, false), "**hi**");
- assert_eq!(wrap_emphasis("hi", false, true), "*hi*");
- assert_eq!(wrap_emphasis("hi", true, true), "***hi***");
- }
-
- #[test]
- fn code_block_escapes_internal_fence() {
- let blocks = vec![Block::CodeBlock {
- lines: vec!["body containing ``` backticks".into()],
- }];
- let s = render_blocks(&blocks);
- assert!(s.starts_with("~~~\n"));
- assert!(s.ends_with("~~~"));
- }
-
- /// Build a line whose spans are placed at explicit x positions — used to
- /// drive the table detector, which relies on per-span x for cell splitting.
- fn line_with_spans(cells: &[(&str, f32)], y: f32, size: f32) -> ProjectedLine {
- let spans: Vec<TextItem> = cells
- .iter()
- .map(|(t, x)| TextItem {
- text: (*t).into(),
- x: *x,
- y,
- width: t.chars().count() as f32 * size * 0.5,
- height: size,
- font_size: Some(size),
- font_name: Some("Arial".into()),
- ..Default::default()
- })
- .collect();
- let min_x = spans.iter().map(|s| s.x).fold(f32::INFINITY, f32::min);
- let max_x = spans
- .iter()
- .map(|s| s.x + s.width)
- .fold(f32::NEG_INFINITY, f32::max);
- ProjectedLine {
- text: cells
- .iter()
- .map(|(t, _)| *t)
- .collect::<Vec<_>>()
- .join(" "),
- bbox: Rect {
- x: min_x,
- y,
- width: (max_x - min_x).max(0.0),
- height: size,
- },
- anchor: Anchor::Left,
- indent_x: min_x,
- dominant_font_size: size,
- dominant_font_name: Some("Arial".into()),
- all_bold: false,
- all_italic: false,
- all_mono: false,
- all_strike: false,
- spans,
- region_path: Vec::new(),
- mcid: None,
- }
- }
-
- #[test]
- fn split_cells_splits_on_wide_gaps() {
- let l = line_with_spans(&[("A", 50.0), ("B", 150.0), ("C", 250.0)], 100.0, 10.0);
- let cells = split_cells(&l);
- assert_eq!(cells.len(), 3);
- assert_eq!(cells[0].text, "A");
- assert_eq!(cells[1].text, "B");
- assert_eq!(cells[2].text, "C");
- }
-
- #[test]
- fn recover_merged_cell_splits_off_by_one() {
- // Mimics the page-6 case: row 0 establishes 3 tracks at 50/150/250.
- // Row 1's projection merges "MEMORYBANK" + "5.00" into one span at
- // x=50 width=110, so split_cells yields 2 cells while the table
- // expects 3. Recovery must split on whitespace at the missing track.
- let row = vec![
- TableCell {
- start_x: 50.0,
- end_x: 160.0,
- text: "MEMORYBANK 5.00".into(),
- bold: false,
- },
- TableCell {
- start_x: 250.0,
- end_x: 280.0,
- text: "4.77".into(),
- bold: false,
- },
- ];
- let tracks = vec![50.0, 150.0, 250.0];
- let out = recover_merged_cell(row, &tracks).expect("recovery should succeed");
- assert_eq!(out.len(), 3);
- assert_eq!(out[0].text, "MEMORYBANK");
- assert_eq!(out[1].text, "5.00");
- assert_eq!(out[2].text, "4.77");
- }
-
- #[test]
- fn recover_merged_cell_splits_off_by_two() {
- // Three merged tokens in one cell: "MEMORYBANK 13.18 10.03" straddles
- // tracks at 50/150/250 and the row has only 2 cells, off by 2.
- let row = vec![
- TableCell {
- start_x: 50.0,
- end_x: 260.0,
- text: "MEMORYBANK 13.18 10.03".into(),
- bold: false,
- },
- TableCell {
- start_x: 350.0,
- end_x: 380.0,
- text: "7.61".into(),
- bold: false,
- },
- ];
- let tracks = vec![50.0, 150.0, 250.0, 350.0];
- let out = recover_merged_cell(row, &tracks).expect("recovery should succeed");
- assert_eq!(out.len(), 4);
- assert_eq!(out[0].text, "MEMORYBANK");
- assert_eq!(out[1].text, "13.18");
- assert_eq!(out[2].text, "10.03");
- assert_eq!(out[3].text, "7.61");
- }
-
- #[test]
- fn recover_merged_cell_bails_without_enough_whitespace() {
- // A cell that straddles two tracks but has no internal whitespace
- // (e.g. a hyphenated token) can't be safely split — return None.
- let row = vec![TableCell {
- start_x: 50.0,
- end_x: 200.0,
- text: "ABC-DEF-GHI".into(),
- bold: false,
- }];
- let tracks = vec![50.0, 150.0];
- assert!(recover_merged_cell(row, &tracks).is_none());
- }
-
- #[test]
- fn split_cells_keeps_close_spans_together() {
- // Two spans 2pt apart at 10pt font (gap < font_size) → same cell.
- let l = line_with_spans(&[("Hello", 50.0), ("world", 80.0)], 100.0, 10.0);
- let cells = split_cells(&l);
- assert_eq!(cells.len(), 1);
- assert_eq!(cells[0].text, "Hello world");
- }
-
- #[test]
- fn detects_simple_borderless_table() {
- let lines = vec![
- line_with_spans(
- &[("Name", 50.0), ("Age", 150.0), ("City", 250.0)],
- 100.0,
- 10.0,
- ),
- line_with_spans(
- &[("Alice", 50.0), ("30", 150.0), ("NYC", 250.0)],
- 115.0,
- 10.0,
- ),
- line_with_spans(&[("Bob", 50.0), ("25", 150.0), ("LA", 250.0)], 130.0, 10.0),
- ];
- let p = page(lines);
- let pages = vec![p];
- let body = compute_body_size(&pages);
- let map = build_heading_map(&pages, body);
- let blocks = classify_page(&pages[0], &map);
- assert_eq!(blocks.len(), 1, "got: {blocks:?}");
- match &blocks[0] {
- Block::Table { header, rows } => {
- // Header isn't bold so no header row promoted.
- assert!(header.is_none());
- assert_eq!(rows.len(), 3);
- assert_eq!(rows[0][0], "Name");
- assert_eq!(rows[1][2], "NYC");
- }
- other => panic!("expected table, got {other:?}"),
- }
- }
-
- #[test]
- fn rejects_table_when_row_count_too_low() {
- let lines = vec![line_with_spans(
- &[("A", 50.0), ("B", 150.0), ("C", 250.0)],
- 100.0,
- 10.0,
- )];
- let runs = detect_tables(&lines);
- assert!(runs.is_empty());
- }
-
- #[test]
- fn rejects_table_when_column_count_too_low() {
- let lines = vec![
- line_with_spans(&[("A", 50.0), ("B", 200.0)], 100.0, 10.0),
- line_with_spans(&[("C", 50.0), ("D", 200.0)], 115.0, 10.0),
- ];
- let runs = detect_tables(&lines);
- assert!(runs.is_empty());
- }
-
- #[test]
- fn renders_table_to_pipe_format() {
- let blocks = vec![Block::Table {
- header: Some(vec!["a".into(), "b".into()]),
- rows: vec![vec!["1".into(), "2".into()], vec!["3".into(), "4".into()]],
- }];
- let s = render_blocks(&blocks);
- assert_eq!(s, "| a | b |\n|---|---|\n| 1 | 2 |\n| 3 | 4 |");
- }
-
- #[test]
- fn render_table_without_header_synthesizes_blank_header() {
- let blocks = vec![Block::Table {
- header: None,
- rows: vec![vec!["1".into(), "2".into()]],
- }];
- let s = render_blocks(&blocks);
- // GFM/CommonMark needs a header row before the separator; we emit a
- // blank one so renderers don't choke.
- assert!(s.contains("|---|---|"));
- assert!(s.ends_with("| 1 | 2 |"));
- }
-
- #[test]
- fn normalize_collapses_digits_and_case() {
- assert_eq!(normalize_for_repetition("Page 1 of 6"), "page # of #");
- assert_eq!(normalize_for_repetition("PAGE 12 OF 6"), "page # of #");
- assert_eq!(normalize_for_repetition("Confidential"), "confidential");
- }
-
- fn header_footer_page(n: usize, header: &str, footer: &str, body: &str) -> ParsedPage {
- // Page height 100 → header band ≤12pt, footer band ≥88pt.
- let mut lines = vec![
- line(header, 50.0, 5.0, 8.0, 8.0),
- line(body, 50.0, 50.0, 10.0, 10.0),
- line(footer, 50.0, 92.0, 6.0, 6.0),
- ];
- for l in &mut lines {
- l.region_path = Vec::new();
- }
- ParsedPage {
- page_number: n,
- page_width: 612.0,
- page_height: 100.0,
- text: String::new(),
- text_items: vec![],
- projected_lines: lines,
- regions: crate::types::Region::default(),
- graphics: vec![],
- figures: vec![],
- }
- }
-
- #[test]
- fn detects_repeating_header_and_footer() {
- let pages = vec![
- header_footer_page(1, "Acme Confidential", "Page 1 of 3", "Body one."),
- header_footer_page(2, "Acme Confidential", "Page 2 of 3", "Body two."),
- header_footer_page(3, "Acme Confidential", "Page 3 of 3", "Body three."),
- ];
- let set = compute_header_footer_set(&pages);
- assert!(set.contains("acme confidential"));
- assert!(set.contains("page # of #"));
- }
-
- #[test]
- fn skips_repetition_check_on_single_page() {
- let pages = vec![header_footer_page(1, "Solo", "footer", "body")];
- let set = compute_header_footer_set(&pages);
- assert!(set.is_empty());
- }
-
- #[test]
- fn body_text_not_classified_as_header() {
- // Same text in the body of every page should NOT be stripped — only
- // text within the top/bottom band qualifies.
- let mut pages = Vec::new();
- for n in 1..=3 {
- let mut p = header_footer_page(n, "unique header", "unique footer", "shared body text");
- // Move "shared body text" out of header/footer bands (already at y=50 in mid-page).
- // No-op — just illustrating intent.
- p.projected_lines[0].text = format!("unique header {n}");
- p.projected_lines[2].text = format!("unique footer {n}");
- pages.push(p);
- }
- let set = compute_header_footer_set(&pages);
- // "shared body text" sits mid-page — never matched.
- assert!(!set.contains("shared body text"));
- }
-
- #[test]
- fn full_format_strips_header_footer() {
- let pages = vec![
- header_footer_page(1, "Acme Confidential", "Page 1 of 2", "First page body."),
- header_footer_page(2, "Acme Confidential", "Page 2 of 2", "Second page body."),
- ];
- let body = compute_body_size(&pages);
- let map = build_heading_map(&pages, body);
- let set = compute_header_footer_set(&pages);
- let blocks = classify_page_with_filters(&pages[0], &map, &set);
- let s = render_blocks(&blocks);
- assert!(!s.contains("Acme Confidential"), "got: {s}");
- assert!(!s.contains("Page 1 of 2"), "got: {s}");
- assert!(s.contains("First page body."));
- }
-
- #[test]
- fn escapes_pipe_inside_cell() {
- assert_eq!(escape_table_cell("a|b"), "a\\|b");
- }
-
- #[test]
- fn dehyphenation_only_for_lowercase_followups() {
- let mut s = String::from("co-");
- append_paragraph_line(&mut s, "operate");
- assert_eq!(s, "cooperate");
-
- let mut s = String::from("Vitamin-");
- append_paragraph_line(&mut s, "A");
- assert_eq!(s, "Vitamin- A");
- }
-
- /// Build a line whose spans carry explicit per-span font metadata. Lets us
- /// exercise the mid-line emphasis pipeline without needing real PDF input.
- fn styled_line(spans: &[(&str, f32, Option<&str>)], y: f32, size: f32) -> ProjectedLine {
- let items: Vec<TextItem> = spans
- .iter()
- .map(|(t, x, font)| TextItem {
- text: (*t).into(),
- x: *x,
- y,
- width: t.chars().count() as f32 * size * 0.5,
- height: size,
- font_size: Some(size),
- font_name: font.map(String::from),
- ..Default::default()
- })
- .collect();
- let joined: String = spans
- .iter()
- .map(|(t, _, _)| *t)
- .collect::<Vec<_>>()
- .join(" ");
- let min_x = items.iter().map(|s| s.x).fold(f32::INFINITY, f32::min);
- let max_x = items
- .iter()
- .map(|s| s.x + s.width)
- .fold(f32::NEG_INFINITY, f32::max);
- ProjectedLine {
- text: joined,
- bbox: Rect {
- x: min_x,
- y,
- width: (max_x - min_x).max(0.0),
- height: size,
- },
- anchor: Anchor::Left,
- indent_x: min_x,
- dominant_font_size: size,
- dominant_font_name: Some("Arial".into()),
- all_bold: false,
- all_italic: false,
- all_mono: false,
- all_strike: false,
- spans: items,
- region_path: Vec::new(),
- mcid: None,
- }
- }
-
- #[test]
- fn render_line_inline_mid_line_bold() {
- // Plain span, then bold span: should produce a mid-line `**bold**` run.
- let l = styled_line(
- &[
- ("regular text with", 50.0, Some("Arial")),
- ("bold word", 200.0, Some("Arial-Bold")),
- ],
- 100.0,
- 10.0,
- );
- let out = render_line_inline(&l);
- assert!(out.contains("regular text with"), "got: {out}");
- assert!(out.contains("**bold word**"), "got: {out}");
- assert!(
- !out.starts_with("**"),
- "mid-line shouldn't open with bold: {out}"
- );
- }
-
- #[test]
- fn render_line_inline_uniform_bold_uses_shortcut() {
- // All spans bold → single outer wrap, no per-span noise.
- let l = styled_line(
- &[
- ("first", 50.0, Some("Arial-Bold")),
- ("second", 100.0, Some("Arial-Bold")),
- ],
- 100.0,
- 10.0,
- );
- let out = render_line_inline(&l);
- assert!(out.starts_with("**") && out.ends_with("**"), "got: {out}");
- // Only one bold run, not two — the shortcut should kick in.
- assert_eq!(out.matches("**").count(), 2, "got: {out}");
- }
-
- #[test]
- fn render_line_inline_escapes_emphasis_chars() {
- let l = styled_line(&[("5*4=20", 50.0, Some("Arial"))], 100.0, 10.0);
- let out = render_line_inline(&l);
- assert_eq!(out, "5\\*4=20");
- }
-
- #[test]
- fn render_line_inline_italic_then_bold() {
- let l = styled_line(
- &[
- ("italic", 50.0, Some("Arial-Italic")),
- ("plain", 100.0, Some("Arial")),
- ("bold", 150.0, Some("Arial-Bold")),
- ],
- 100.0,
- 10.0,
- );
- let out = render_line_inline(&l);
- assert!(out.contains("*italic*"), "got: {out}");
- assert!(out.contains("plain"), "got: {out}");
- assert!(out.contains("**bold**"), "got: {out}");
- }
-
- #[test]
- fn render_line_inline_mono_span() {
- let l = styled_line(
- &[
- ("call", 50.0, Some("Arial")),
- ("foo()", 100.0, Some("Courier")),
- ("on it", 150.0, Some("Arial")),
- ],
- 100.0,
- 10.0,
- );
- let out = render_line_inline(&l);
- assert!(out.contains("`foo()`"), "got: {out}");
- // Plain spans stay unwrapped.
- assert!(out.contains("call"));
- assert!(out.contains("on it"));
- }
-
- #[test]
- fn classify_paragraph_with_mid_line_bold() {
- // First line has a bold word mid-line → not uniformly styled; paragraph
- // should emit baked-in `**bold**` inside the text and `bold=false` at
- // the block level.
- let a = styled_line(
- &[
- ("a sentence with a", 50.0, Some("Arial")),
- ("bold", 180.0, Some("Arial-Bold")),
- ("word in it.", 230.0, Some("Arial")),
- ],
- 50.0,
- 10.0,
- );
- let p = page(vec![a]);
- let pages = vec![p];
- let body = compute_body_size(&pages);
- let map = build_heading_map(&pages, body);
- let blocks = classify_page(&pages[0], &map);
- assert_eq!(blocks.len(), 1, "got: {blocks:?}");
- match &blocks[0] {
- Block::Paragraph { text, bold, italic } => {
- assert!(!*bold, "mixed-style paragraph shouldn't set block bold");
- assert!(!*italic);
- assert!(text.contains("**bold**"), "got: {text}");
- }
- other => panic!("expected paragraph, got {other:?}"),
- }
- }
-
- #[test]
- fn classify_list_item_strips_marker_under_emphasis() {
- // Whole bullet line is bold (marker + text). Rendered text should be
- // wrapped, with the marker dropped (the renderer prints it).
- let l = styled_line(
- &[
- ("•", 60.0, Some("Arial-Bold")),
- ("important item", 80.0, Some("Arial-Bold")),
- ],
- 50.0,
- 10.0,
- );
- let p = page(vec![l]);
- let pages = vec![p];
- let body = compute_body_size(&pages);
- let map = build_heading_map(&pages, body);
- let blocks = classify_page(&pages[0], &map);
- assert_eq!(blocks.len(), 1);
- match &blocks[0] {
- Block::ListItem { text, .. } => {
- assert_eq!(text, "**important item**");
- }
- other => panic!("expected list item, got {other:?}"),
- }
- }
-
- fn stroke(x1: f32, y1: f32, x2: f32, y2: f32, width: f32) -> GraphicPrimitive {
- GraphicPrimitive::Stroke {
- x1,
- y1,
- x2,
- y2,
- width,
- color: None,
- }
- }
-
- fn page_with_graphics(
- lines: Vec<ProjectedLine>,
- graphics: Vec<GraphicPrimitive>,
- ) -> ParsedPage {
- let mut p = page(lines);
- p.graphics = graphics;
- p
- }
-
- #[test]
- fn hr_long_thin_horizontal_stroke_detected() {
- // 400pt wide stroke on a 612pt page → comfortably above 30% threshold.
- let p = page_with_graphics(vec![], vec![stroke(50.0, 200.0, 450.0, 200.5, 0.5)]);
- let ys = detect_horizontal_rules(&p);
- assert_eq!(ys, vec![200.25]);
- }
-
- #[test]
- fn hr_short_stroke_rejected() {
- // 50pt wide — table border or list bullet, not an HR.
- let p = page_with_graphics(vec![], vec![stroke(50.0, 200.0, 100.0, 200.0, 0.5)]);
- assert!(detect_horizontal_rules(&p).is_empty());
- }
-
- #[test]
- fn hr_vertical_stroke_rejected() {
- let p = page_with_graphics(vec![], vec![stroke(50.0, 50.0, 50.0, 500.0, 0.5)]);
- assert!(detect_horizontal_rules(&p).is_empty());
- }
-
- #[test]
- fn hr_thick_stroke_rejected() {
- // 4pt-thick stroke → a filled bar, not a rule.
- let p = page_with_graphics(vec![], vec![stroke(50.0, 200.0, 450.0, 200.0, 4.0)]);
- assert!(detect_horizontal_rules(&p).is_empty());
- }
-
- #[test]
- fn hr_underline_at_text_baseline_dropped() {
- // Text line at y=100 height=10 → bottom at y=110. Stroke at y=111 within
- // the line's horizontal extent → underline, not an HR.
- let text_line = line(
- "Some underlined heading text on the page",
- 50.0,
- 100.0,
- 10.0,
- 10.0,
- );
- let bottom = text_line.bbox.y + text_line.bbox.height;
- let p = page_with_graphics(
- vec![text_line.clone()],
- vec![stroke(
- 50.0,
- bottom + 1.0,
- 50.0 + text_line.bbox.width,
- bottom + 1.0,
- 0.5,
- )],
- );
- assert!(detect_horizontal_rules(&p).is_empty());
- }
-
- /// Helper: build the four borders of a rectangle as four strokes.
- fn rect_borders(x: f32, y: f32, w: f32, h: f32) -> Vec<GraphicPrimitive> {
- vec![
- stroke(x, y, x + w, y, 0.5), // top
- stroke(x, y + h, x + w, y + h, 0.5), // bottom
- stroke(x, y, x, y + h, 0.5), // left
- stroke(x + w, y, x + w, y + h, 0.5), // right
- ]
- }
-
- #[test]
- fn ruled_table_2x2_detected() {
- // 2 rows × 2 cols grid: 3 H lines (y=100,140,180), 3 V lines (x=50,150,250)
- // Cell text dropped in the centroid of each cell.
- let mut graphics = Vec::new();
- for y in [100.0_f32, 140.0, 180.0] {
- graphics.push(stroke(50.0, y, 250.0, y, 0.5));
- }
- for x in [50.0_f32, 150.0, 250.0] {
- graphics.push(stroke(x, 100.0, x, 180.0, 0.5));
- }
-
- // Text lines: one per cell, centered.
- let lines = vec![
- line("a", 90.0, 115.0, 10.0, 10.0), // row 0, col 0
- line("b", 190.0, 115.0, 10.0, 10.0), // row 0, col 1
- line("c", 90.0, 155.0, 10.0, 10.0), // row 1, col 0
- line("d", 190.0, 155.0, 10.0, 10.0), // row 1, col 1
- ];
-
- let runs = detect_ruled_tables(&lines, &graphics, 612.0, 792.0);
- assert_eq!(runs.len(), 1, "expected 1 ruled table, got {runs:?}");
- match &runs[0].block {
- Block::Table { header, rows } => {
- assert!(header.is_none(), "no bold first row → no header");
- assert_eq!(rows.len(), 2);
- assert_eq!(rows[0], vec!["a", "b"]);
- assert_eq!(rows[1], vec!["c", "d"]);
- }
- other => panic!("expected Block::Table, got {other:?}"),
- }
- }
-
- #[test]
- fn ruled_table_rect_borders_detected() {
- // Same 2×2 table but drawn as 4 individual cell rects (each cell is a
- // stroked rectangle). Each rect contributes 4 strokes via
- // extract_h_v_segments.
- let mut graphics = Vec::new();
- graphics.extend(rect_borders(50.0, 100.0, 100.0, 40.0)); // r0 c0
- graphics.extend(rect_borders(150.0, 100.0, 100.0, 40.0)); // r0 c1
- graphics.extend(rect_borders(50.0, 140.0, 100.0, 40.0)); // r1 c0
- graphics.extend(rect_borders(150.0, 140.0, 100.0, 40.0)); // r1 c1
-
- let lines = vec![
- line("a", 90.0, 115.0, 10.0, 10.0),
- line("b", 190.0, 115.0, 10.0, 10.0),
- line("c", 90.0, 155.0, 10.0, 10.0),
- line("d", 190.0, 155.0, 10.0, 10.0),
- ];
- let runs = detect_ruled_tables(&lines, &graphics, 612.0, 792.0);
- assert_eq!(runs.len(), 1);
- }
-
- #[test]
- fn ruled_table_page_border_rejected() {
- // Single big rect covering ~the whole page → should NOT be treated as a
- // table even though it has H+V lines on all four sides.
- let graphics = rect_borders(10.0, 10.0, 590.0, 770.0);
- let lines = vec![line("body text", 50.0, 400.0, 10.0, 10.0)];
- let runs = detect_ruled_tables(&lines, &graphics, 612.0, 792.0);
- assert!(
- runs.is_empty(),
- "page-border rect should not become a table, got {runs:?}"
- );
- }
-
- #[test]
- fn ruled_table_mostly_empty_rejected() {
- // 3×3 grid with text in only one cell — empty fraction 8/9 ≈ 89% >> 30%.
- let mut graphics = Vec::new();
- for y in [100.0_f32, 130.0, 160.0, 190.0] {
- graphics.push(stroke(50.0, y, 350.0, y, 0.5));
- }
- for x in [50.0_f32, 150.0, 250.0, 350.0] {
- graphics.push(stroke(x, 100.0, x, 190.0, 0.5));
- }
- let lines = vec![line("only", 90.0, 115.0, 10.0, 10.0)];
- let runs = detect_ruled_tables(&lines, &graphics, 612.0, 792.0);
- assert!(runs.is_empty());
- }
-
- #[test]
- fn ruled_table_first_row_bold_becomes_header() {
- // 2×2 with first row text marked all_bold → header promotion.
- let mut graphics = Vec::new();
- for y in [100.0_f32, 140.0, 180.0] {
- graphics.push(stroke(50.0, y, 250.0, y, 0.5));
- }
- for x in [50.0_f32, 150.0, 250.0] {
- graphics.push(stroke(x, 100.0, x, 180.0, 0.5));
- }
- let mut a = line("Name", 90.0, 115.0, 10.0, 10.0);
- let mut b = line("Score", 190.0, 115.0, 10.0, 10.0);
- a.all_bold = true;
- b.all_bold = true;
- let lines = vec![
- a,
- b,
- line("alice", 90.0, 155.0, 10.0, 10.0),
- line("99", 190.0, 155.0, 10.0, 10.0),
- ];
- let runs = detect_ruled_tables(&lines, &graphics, 612.0, 792.0);
- assert_eq!(runs.len(), 1);
- match &runs[0].block {
- Block::Table { header, rows } => {
- assert_eq!(
- header.as_deref(),
- Some(&["Name".into(), "Score".into()][..])
- );
- assert_eq!(rows.len(), 1);
- assert_eq!(rows[0], vec!["alice", "99"]);
- }
- other => panic!("expected Block::Table, got {other:?}"),
- }
- }
-
- #[test]
- fn merge_prefers_ruled_when_overlapping() {
- let ruled = vec![TableRun {
- start: 5,
- end: 10,
- block: Block::Table {
- header: None,
- rows: vec![vec!["ruled".into()]],
- },
- }];
- let borderless = vec![TableRun {
- start: 6,
- end: 11,
- block: Block::GridFallback {
- lines: vec!["bl".into()],
- },
- }];
- let merged = merge_table_runs(ruled, borderless);
- assert_eq!(merged.len(), 1);
- assert!(matches!(&merged[0].block, Block::Table { .. }));
- }
-
- #[test]
- fn hr_emitted_between_lines_by_y_order() {
- let lines = vec![
- line("before the rule", 50.0, 100.0, 10.0, 10.0),
- line("after the rule", 50.0, 300.0, 10.0, 10.0),
- ];
- // Stroke between the two lines, far from either's baseline.
- let p = page_with_graphics(lines, vec![stroke(50.0, 200.0, 450.0, 200.0, 0.5)]);
- let blocks = classify_page(&p, &[]);
- let has_hr = blocks
- .iter()
- .position(|b| matches!(b, Block::HorizontalRule));
- assert!(has_hr.is_some(), "expected an HR block, got {blocks:?}");
- // HR must land between the two paragraphs, not before/after both.
- let pos = has_hr.unwrap();
- assert!(pos > 0 && pos < blocks.len() - 1);
- }
-}
diff --git a/crates/liteparse/src/markdown_layout/blocks.rs b/crates/liteparse/src/markdown_layout/blocks.rs
new file mode 100644
index 0000000..a737a37
--- /dev/null
+++ b/crates/liteparse/src/markdown_layout/blocks.rs
@@ -0,0 +1,347 @@
+use super::inline::escape_inline;
+use super::paragraphs::ParaAccum;
+use super::tables::escape_table_cell;
+
+/// Coarse block representation. Intentionally minimal — extended as later
+/// build-order steps land (tables, figures).
+#[derive(Debug, Clone)]
+pub enum Block {
+ Heading {
+ level: u8,
+ text: String,
+ },
+ Paragraph {
+ text: String,
+ bold: bool,
+ italic: bool,
+ },
+ ListItem {
+ ordered: bool,
+ marker: String,
+ level: u8,
+ text: String,
+ bold: bool,
+ italic: bool,
+ },
+ /// Fenced code block — content rendered between triple-backtick fences.
+ /// Each entry in `lines` is one source line; preserved as-is (only trailing
+ /// whitespace stripped) so indentation survives.
+ CodeBlock {
+ lines: Vec<String>,
+ },
+ /// Confident table emitted as a markdown pipe table. `header` is `None`
+ /// when the first row didn't qualify (e.g. wasn't bold and the table mode
+ /// can't otherwise distinguish it).
+ Table {
+ header: Option<Vec<String>>,
+ rows: Vec<Vec<String>>,
+ },
+ /// Tabular-looking region we couldn't classify confidently — rendered
+ /// verbatim inside a fenced block to preserve visual structure for the
+ /// downstream LLM. Strictly better than emitting a mangled pipe table.
+ GridFallback {
+ lines: Vec<String>,
+ },
+ /// A horizontal rule detected from a long thin horizontal stroke in the
+ /// page's vector graphics (e.g. divider line between sections).
+ HorizontalRule,
+ /// Reference to a raster image on the page. Rendered as
+ /// `![](image_{id}.png)`. Suppressed entirely when `ImageMode::Off`.
+ Figure {
+ id: String,
+ bbox: crate::types::Rect,
+ },
+ Blank,
+}
+
+/// Resolve a `ParaAccum` to a `Block::Paragraph`. When the paragraph was
+/// uniformly styled across all lines, return the raw text with block-level
+/// `bold`/`italic` flags set so the renderer wraps it once. Otherwise return
+/// the per-line inline-styled text with the flags cleared.
+pub(super) fn paragraph_from_accum(accum: ParaAccum) -> Block {
+ match accum.uniform {
+ Some((bold, italic)) if bold || italic => Block::Paragraph {
+ text: escape_inline(&accum.raw),
+ bold,
+ italic,
+ },
+ Some(_) => Block::Paragraph {
+ // Uniformly plain — no emphasis markers anywhere, so the raw text
+ // (with markdown specials escaped) is the right rendering.
+ text: escape_inline(&accum.raw),
+ bold: false,
+ italic: false,
+ },
+ None => Block::Paragraph {
+ text: accum.inline,
+ bold: false,
+ italic: false,
+ },
+ }
+}
+
+/// Wrap `text` in markdown emphasis markers based on `bold`/`italic`. Both →
+/// `***text***`. Headings deliberately skip this (the `#` is the emphasis).
+fn wrap_emphasis(text: &str, bold: bool, italic: bool) -> String {
+ if text.trim().is_empty() {
+ return text.to_string();
+ }
+ match (bold, italic) {
+ (true, true) => format!("***{text}***"),
+ (true, false) => format!("**{text}**"),
+ (false, true) => format!("*{text}*"),
+ (false, false) => text.to_string(),
+ }
+}
+
+/// Render a list of blocks to a markdown string.
+pub fn render_blocks(blocks: &[Block]) -> String {
+ let mut out = String::new();
+ for (i, block) in blocks.iter().enumerate() {
+ if i > 0 {
+ // Consecutive list items render as a tight list (single newline).
+ // Everything else gets a blank line between blocks.
+ let tight = matches!(block, Block::ListItem { .. })
+ && matches!(blocks[i - 1], Block::ListItem { .. });
+ if tight {
+ out.push('\n');
+ } else {
+ out.push_str("\n\n");
+ }
+ }
+ match block {
+ Block::Heading { level, text } => {
+ let level = (*level).clamp(1, 6) as usize;
+ out.push_str(&"#".repeat(level));
+ out.push(' ');
+ out.push_str(text);
+ }
+ Block::Paragraph { text, bold, italic } => {
+ out.push_str(&wrap_emphasis(text, *bold, *italic));
+ }
+ Block::ListItem {
+ ordered,
+ marker,
+ level,
+ text,
+ bold,
+ italic,
+ } => {
+ let indent = " ".repeat((*level).min(6) as usize);
+ out.push_str(&indent);
+ if *ordered {
+ // Preserve the original marker (e.g. `138.` for footnotes
+ // or `iii)` for roman numerals) so semantic numbering
+ // survives the round-trip. CommonMark requires `1.` /
+ // `1)` style but most LLM consumers tolerate alt forms,
+ // and the alternative — renumbering as `1.` — drops info.
+ out.push_str(marker);
+ out.push(' ');
+ } else {
+ out.push_str("- ");
+ }
+ out.push_str(&wrap_emphasis(text, *bold, *italic));
+ }
+ Block::Table { header, rows } => {
+ let column_count = header
+ .as_ref()
+ .map(|h| h.len())
+ .or_else(|| rows.first().map(|r| r.len()))
+ .unwrap_or(0);
+ if column_count == 0 {
+ continue;
+ }
+ if let Some(h) = header {
+ out.push_str("| ");
+ for (i, cell) in h.iter().enumerate() {
+ if i > 0 {
+ out.push_str(" | ");
+ }
+ out.push_str(&escape_table_cell(cell));
+ }
+ out.push_str(" |\n");
+ } else {
+ // CommonMark/GFM requires a header row before the
+ // separator; synthesize a blank header so renderers that
+ // refuse header-less tables still display the body.
+ out.push_str("|");
+ for _ in 0..column_count {
+ out.push_str(" |");
+ }
+ out.push('\n');
+ }
+ out.push('|');
+ for _ in 0..column_count {
+ out.push_str("---|");
+ }
+ for row in rows {
+ out.push_str("\n| ");
+ for (i, cell) in row.iter().enumerate() {
+ if i > 0 {
+ out.push_str(" | ");
+ }
+ out.push_str(&escape_table_cell(cell));
+ }
+ out.push_str(" |");
+ }
+ }
+ Block::GridFallback { lines } => {
+ out.push_str("```text\n");
+ for line in lines {
+ out.push_str(line);
+ out.push('\n');
+ }
+ out.push_str("```");
+ }
+ Block::CodeBlock { lines } => {
+ // Pick a fence that doesn't appear inside the body. Standard
+ // triple-backtick handles ~all real-world code; bump to ~~~ if
+ // the body itself contains ``` (rare).
+ let fence = if lines.iter().any(|l| l.contains("```")) {
+ "~~~"
+ } else {
+ "```"
+ };
+ out.push_str(fence);
+ out.push('\n');
+ for line in lines {
+ out.push_str(line);
+ out.push('\n');
+ }
+ out.push_str(fence);
+ }
+ Block::HorizontalRule => {
+ out.push_str("---");
+ }
+ Block::Figure { id, .. } => {
+ out.push_str("![](image_");
+ out.push_str(id);
+ out.push_str(".png)");
+ }
+ Block::Blank => {}
+ }
+ }
+ out
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn render_blocks_formats_markdown() {
+ let blocks = vec![
+ Block::Heading {
+ level: 1,
+ text: "Title".into(),
+ },
+ Block::Paragraph {
+ text: "A paragraph.".into(),
+ bold: false,
+ italic: false,
+ },
+ Block::Heading {
+ level: 2,
+ text: "Sub".into(),
+ },
+ ];
+ let s = render_blocks(&blocks);
+ assert_eq!(s, "# Title\n\nA paragraph.\n\n## Sub");
+ }
+
+ #[test]
+ fn render_lists_are_tight() {
+ let blocks = vec![
+ Block::Paragraph {
+ text: "Intro.".into(),
+ bold: false,
+ italic: false,
+ },
+ Block::ListItem {
+ ordered: false,
+ marker: "•".into(),
+ level: 0,
+ text: "a".into(),
+ bold: false,
+ italic: false,
+ },
+ Block::ListItem {
+ ordered: false,
+ marker: "•".into(),
+ level: 0,
+ text: "b".into(),
+ bold: false,
+ italic: false,
+ },
+ Block::Paragraph {
+ text: "Outro.".into(),
+ bold: false,
+ italic: false,
+ },
+ ];
+ let s = render_blocks(&blocks);
+ assert_eq!(s, "Intro.\n\n- a\n- b\n\nOutro.");
+
+ // Ordered: original marker preserved
+ let s = render_blocks(&[
+ Block::ListItem {
+ ordered: true,
+ marker: "138.".into(),
+ level: 0,
+ text: "footnote".into(),
+ bold: false,
+ italic: false,
+ },
+ Block::ListItem {
+ ordered: true,
+ marker: "139.".into(),
+ level: 0,
+ text: "next footnote".into(),
+ bold: false,
+ italic: false,
+ },
+ ]);
+ assert_eq!(s, "138. footnote\n139. next footnote");
+ }
+
+ #[test]
+ fn render_emphasis_combinations() {
+ assert_eq!(wrap_emphasis("hi", false, false), "hi");
+ assert_eq!(wrap_emphasis("hi", true, false), "**hi**");
+ assert_eq!(wrap_emphasis("hi", false, true), "*hi*");
+ assert_eq!(wrap_emphasis("hi", true, true), "***hi***");
+ }
+
+ #[test]
+ fn code_block_escapes_internal_fence() {
+ let blocks = vec![Block::CodeBlock {
+ lines: vec!["body containing ``` backticks".into()],
+ }];
+ let s = render_blocks(&blocks);
+ assert!(s.starts_with("~~~\n"));
+ assert!(s.ends_with("~~~"));
+ }
+
+ #[test]
+ fn renders_table_to_pipe_format() {
+ let blocks = vec![Block::Table {
+ header: Some(vec!["a".into(), "b".into()]),
+ rows: vec![vec!["1".into(), "2".into()], vec!["3".into(), "4".into()]],
+ }];
+ let s = render_blocks(&blocks);
+ assert_eq!(s, "| a | b |\n|---|---|\n| 1 | 2 |\n| 3 | 4 |");
+ }
+
+ #[test]
+ fn render_table_without_header_synthesizes_blank_header() {
+ let blocks = vec![Block::Table {
+ header: None,
+ rows: vec![vec!["1".into(), "2".into()]],
+ }];
+ let s = render_blocks(&blocks);
+ // GFM/CommonMark needs a header row before the separator; we emit a
+ // blank one so renderers don't choke.
+ assert!(s.contains("|---|---|"));
+ assert!(s.ends_with("| 1 | 2 |"));
+ }
+}
diff --git a/crates/liteparse/src/markdown_layout/classify.rs b/crates/liteparse/src/markdown_layout/classify.rs
new file mode 100644
index 0000000..2de9c2d
--- /dev/null
+++ b/crates/liteparse/src/markdown_layout/classify.rs
@@ -0,0 +1,783 @@
+use crate::types::{OutlineTarget, ParsedPage, ProjectedLine};
+
+use super::blocks::{Block, paragraph_from_accum};
+use super::headings::{
+ MAX_HEADING_LEVELS, heading_level_for, is_caption_line, looks_like_bold_heading,
+ looks_like_numbered_bold_heading, outline_heading_level, struct_heading_level,
+};
+use super::hr::detect_horizontal_rules;
+use super::inline::{
+ append_inline_continuation, line_uniform_style, render_line_inline, render_list_item_text,
+};
+use super::lists::{LIST_INDENT_STEP_PT, parse_list_marker};
+use super::paragraphs::{ParaAccum, append_to_paragraph, collapse_whitespace, continues_paragraph};
+use super::repetition::is_header_or_footer;
+use super::tables::{detect_ruled_tables, detect_tables, merge_table_runs};
+
+/// Returns true if any span on the line is rotated more than ~5° off
+/// horizontal — used to exclude sidebar / margin-stamp text (arXiv banners,
+/// watermarks, vertical legends) from the body-size and heading-size
+/// histograms so it doesn't compete with normal-flow text for heading slots.
+pub(super) fn is_rotated_line(line: &ProjectedLine) -> bool {
+ line.spans.iter().any(|s| {
+ let r = s.rotation.abs() % 360.0;
+ // Anything more than ~5° off the horizontal axes is "rotated" for
+ // our purposes. 0° and 180° are both horizontal text.
+ !(r < 5.0 || (175.0..=185.0).contains(&r) || (355.0..=360.0).contains(&r))
+ })
+}
+
+/// Classify a single page's `ProjectedLine`s into blocks.
+#[cfg(test)]
+pub(super) fn classify_page(page: &ParsedPage, heading_map: &[(f32, u8)]) -> Vec<Block> {
+ classify_page_with_filters(
+ page,
+ heading_map,
+ &std::collections::HashSet::new(),
+ &[],
+ crate::config::ImageMode::Placeholder,
+ )
+}
+
+/// Same as `classify_page` but also strips lines matching a precomputed
+/// running header/footer set. Use this when emitting a whole document so
+/// repeating chrome (titles, page numbers) doesn't show up in every page.
+///
+/// `outline` is the document outline filtered to entries whose `page_index`
+/// matches this page (or the full outline — out-of-page entries are
+/// ignored). Heading promotion from struct tree + outline outranks the
+/// font-size heading map.
+pub fn classify_page_with_filters(
+ page: &ParsedPage,
+ heading_map: &[(f32, u8)],
+ header_footer: &std::collections::HashSet<String>,
+ outline: &[OutlineTarget],
+ image_mode: crate::config::ImageMode,
+) -> Vec<Block> {
+ let mut blocks: Vec<Block> = Vec::new();
+ let mut paragraph: Option<ParaAccum> = None;
+ // Active fenced code block being accumulated (consecutive `all_mono` lines).
+ let mut code: Option<Vec<String>> = None;
+ // Tracks the "level 0" indent of the current contiguous list run so we can
+ // bucket deeper items into nesting levels. Reset whenever a non-list block
+ // breaks the run.
+ let mut list_base_indent: Option<f32> = None;
+ // Index into `blocks` of the most recent ListItem in the current run — used
+ // to merge wrapped continuation lines into the same item.
+ let mut last_list_item_idx: Option<usize> = None;
+ // Most recent ProjectedLine appended to the active list item, for
+ // gap/font-size checks on continuation lines.
+ let mut last_list_line: Option<ProjectedLine> = None;
+
+ let flush_paragraph = |blocks: &mut Vec<Block>, p: Option<ParaAccum>| {
+ if let Some(acc) = p
+ && !acc.raw.trim().is_empty()
+ {
+ blocks.push(paragraph_from_accum(acc));
+ }
+ };
+ let flush_code = |blocks: &mut Vec<Block>, c: Option<Vec<String>>| {
+ if let Some(lines) = c
+ && !lines.is_empty()
+ {
+ blocks.push(Block::CodeBlock { lines });
+ }
+ };
+
+ let debug = std::env::var("LITEPARSE_DEBUG_MD").is_ok();
+
+ // Strip running header/footer lines up-front so they don't leak into
+ // table detection (a repeating two-column footer would otherwise look
+ // like a 2-row table) or paragraph grouping.
+ let filtered_owned: Vec<ProjectedLine> = if header_footer.is_empty() {
+ Vec::new()
+ } else {
+ page.projected_lines
+ .iter()
+ .filter(|l| !is_header_or_footer(l, page, header_footer))
+ .cloned()
+ .collect()
+ };
+ let lines: &[ProjectedLine] = if header_footer.is_empty() {
+ &page.projected_lines
+ } else {
+ &filtered_owned
+ };
+
+ // Pre-pass: detect tabular regions so the per-line classifier below can
+ // skip over them. Tables take priority over heading / paragraph / list
+ // classification because a row of bold short cells would otherwise be
+ // misread as a heading or list item.
+ //
+ // Two detectors run in sequence: ruled-grid (path-based, strongest signal)
+ // and the borderless column-alignment fallback. Where ranges overlap, the
+ // ruled output wins.
+ let ruled_runs = detect_ruled_tables(lines, &page.graphics, page.page_width, page.page_height);
+ let borderless_runs = detect_tables(lines);
+ let table_runs = merge_table_runs(ruled_runs, borderless_runs);
+
+ // Suppress HRs that fall inside a detected table's y-range — they're the
+ // table's own row dividers, not document-level section breaks. Build the
+ // y-extents once before we move table_runs into the iterator.
+ // Extend each table's HR-suppression band upward to cover any
+ // header/sub-header rows we didn't absorb into the table. The expansion
+ // is a few row heights — large enough to catch a 2–3 line bold/italic
+ // header sitting just above the table, small enough not to swallow a
+ // real section divider belonging to a different block. The downward
+ // edge gets a small slack to catch HRs drawn flush with the last row.
+ const TABLE_HR_SUPPRESS_HEADROOM_ROWS: f32 = 4.0;
+ let table_y_extents: Vec<(f32, f32)> = table_runs
+ .iter()
+ .map(|run| {
+ let top_line = &lines[run.start];
+ let row_h = top_line.bbox.height.max(8.0);
+ let top = top_line.bbox.y - row_h * TABLE_HR_SUPPRESS_HEADROOM_ROWS;
+ let last = &lines[run.end.saturating_sub(1).max(run.start)];
+ let bot = last.bbox.y + last.bbox.height;
+ (top, bot)
+ })
+ .collect();
+
+ let mut table_iter = table_runs.into_iter().peekable();
+
+ // Pre-pass: detect horizontal rules from vector graphics so they can be
+ // emitted in document order between surrounding text lines.
+ let hr_ys: Vec<f32> = detect_horizontal_rules(page)
+ .into_iter()
+ .filter(|y| {
+ !table_y_extents
+ .iter()
+ .any(|(top, bot)| *y >= *top - 2.0 && *y <= *bot + 2.0)
+ })
+ .collect();
+ let mut hr_iter = hr_ys.into_iter().peekable();
+
+ // Figure injection: when image mode is on, walk the page's image refs in
+ // y-order and inject `Block::Figure` between text lines whose y has
+ // already passed. Suppressed inside table y-extents to keep cell-spanning
+ // raster cell backgrounds (rare but possible) from spawning a figure
+ // mid-table.
+ let figure_entries: Vec<(f32, crate::types::ImageRef)> = if matches!(
+ image_mode,
+ crate::config::ImageMode::Off
+ ) {
+ Vec::new()
+ } else {
+ let mut v: Vec<(f32, crate::types::ImageRef)> = page
+ .image_refs
+ .iter()
+ .filter(|r| {
+ !table_y_extents
+ .iter()
+ .any(|(top, bot)| r.bbox.y >= *top - 2.0 && r.bbox.y <= *bot + 2.0)
+ })
+ .map(|r| (r.bbox.y, r.clone()))
+ .collect();
+ v.sort_by(|a, b| a.0.total_cmp(&b.0));
+ v
+ };
+ let mut figure_iter = figure_entries.into_iter().peekable();
+
+ // Emit any HRs whose y is at or above `before_y`. Flushes the active
+ // paragraph/code/list state first so the rule lands as its own block.
+ let emit_hrs_before = |blocks: &mut Vec<Block>,
+ paragraph: &mut Option<ParaAccum>,
+ code: &mut Option<Vec<String>>,
+ list_base: &mut Option<f32>,
+ last_item: &mut Option<usize>,
+ last_line: &mut Option<ProjectedLine>,
+ hr_iter: &mut std::iter::Peekable<std::vec::IntoIter<f32>>,
+ before_y: f32| {
+ while let Some(&hy) = hr_iter.peek() {
+ if hy > before_y {
+ break;
+ }
+ hr_iter.next();
+ flush_paragraph(blocks, paragraph.take());
+ flush_code(blocks, code.take());
+ *list_base = None;
+ *last_item = None;
+ *last_line = None;
+ blocks.push(Block::HorizontalRule);
+ }
+ };
+
+ // Mirror of `emit_hrs_before` for figure entries. Same flush semantics so
+ // a figure sitting between two paragraphs lands as its own block in the
+ // correct y order.
+ let emit_figures_before = |blocks: &mut Vec<Block>,
+ paragraph: &mut Option<ParaAccum>,
+ code: &mut Option<Vec<String>>,
+ list_base: &mut Option<f32>,
+ last_item: &mut Option<usize>,
+ last_line: &mut Option<ProjectedLine>,
+ figure_iter: &mut std::iter::Peekable<
+ std::vec::IntoIter<(f32, crate::types::ImageRef)>,
+ >,
+ before_y: f32| {
+ while let Some((fy, _)) = figure_iter.peek() {
+ if *fy > before_y {
+ break;
+ }
+ let (_, r) = figure_iter.next().unwrap();
+ flush_paragraph(blocks, paragraph.take());
+ flush_code(blocks, code.take());
+ *list_base = None;
+ *last_item = None;
+ *last_line = None;
+ blocks.push(Block::Figure {
+ id: r.id,
+ bbox: r.bbox,
+ });
+ }
+ };
+
+ let mut idx = 0;
+ while idx < lines.len() {
+ if let Some(run) = table_iter.peek()
+ && run.start == idx
+ {
+ // Flush any HRs above this table's top edge first.
+ let table_top = lines[run.start].bbox.y;
+ emit_hrs_before(
+ &mut blocks,
+ &mut paragraph,
+ &mut code,
+ &mut list_base_indent,
+ &mut last_list_item_idx,
+ &mut last_list_line,
+ &mut hr_iter,
+ table_top,
+ );
+ emit_figures_before(
+ &mut blocks,
+ &mut paragraph,
+ &mut code,
+ &mut list_base_indent,
+ &mut last_list_item_idx,
+ &mut last_list_line,
+ &mut figure_iter,
+ table_top,
+ );
+ flush_paragraph(&mut blocks, paragraph.take());
+ flush_code(&mut blocks, code.take());
+ list_base_indent = None;
+ last_list_item_idx = None;
+ last_list_line = None;
+ let run = table_iter.next().unwrap();
+ blocks.push(run.block);
+ idx = run.end;
+ continue;
+ }
+ let line = &lines[idx];
+ // Emit any HRs that fall above this line.
+ emit_hrs_before(
+ &mut blocks,
+ &mut paragraph,
+ &mut code,
+ &mut list_base_indent,
+ &mut last_list_item_idx,
+ &mut last_list_line,
+ &mut hr_iter,
+ line.bbox.y,
+ );
+ emit_figures_before(
+ &mut blocks,
+ &mut paragraph,
+ &mut code,
+ &mut list_base_indent,
+ &mut last_list_item_idx,
+ &mut last_list_line,
+ &mut figure_iter,
+ line.bbox.y,
+ );
+ idx += 1;
+ let text = line.text.trim();
+ if text.is_empty() {
+ continue;
+ }
+ // Skip rotated text (vertical sidebars, arXiv banners, watermarks).
+ // Including it would either inject a paragraph of disconnected
+ // characters or be misclassified as a heading.
+ if is_rotated_line(line) {
+ continue;
+ }
+ if debug {
+ eprintln!(
+ "[md] y={:.1} h={:.1} size={:.2} anchor={:?} indent={:.1} text={:?}",
+ line.bbox.y,
+ line.bbox.height,
+ line.dominant_font_size,
+ line.anchor,
+ line.indent_x,
+ text
+ );
+ }
+
+ // Code block detection runs first so a mono heading-shaped line
+ // (rare but plausible — e.g., a code identifier in a large mono font)
+ // still becomes code. Mono content also wouldn't make a useful
+ // heading.
+ if line.all_mono {
+ flush_paragraph(&mut blocks, paragraph.take());
+ list_base_indent = None;
+ last_list_item_idx = None;
+ last_list_line = None;
+ code.get_or_insert_with(Vec::new)
+ .push(line.text.trim_end().to_string());
+ continue;
+ }
+ // Any non-mono line ends the current code block (if any).
+ flush_code(&mut blocks, code.take());
+
+ // Priority chain: tagged-PDF struct tree → outline → font-size map.
+ let tagged_level = struct_heading_level(line, &page.struct_nodes);
+ let outline_level =
+ tagged_level.or_else(|| outline_heading_level(line, page.page_height, outline, text));
+ // Caption lines ("Figure 7", "Table 3.") are routinely set in a
+ // distinct (and slightly larger) font that lands them in the
+ // font-size heading map. Suppress font-size promotion for them;
+ // outline / struct-tree signals still win since those are explicit.
+ let size_level = if is_caption_line(text) {
+ None
+ } else {
+ heading_level_for(line.dominant_font_size, heading_map)
+ };
+ let level = outline_level
+ .or(size_level)
+ .map(|l| l.clamp(1, MAX_HEADING_LEVELS as u8));
+ if let Some(level) = level {
+ flush_paragraph(&mut blocks, paragraph.take());
+ list_base_indent = None;
+ last_list_item_idx = None;
+ last_list_line = None;
+ blocks.push(Block::Heading {
+ level,
+ text: collapse_whitespace(text),
+ });
+ continue;
+ }
+
+ // List item?
+ if let Some((ordered, marker, rest)) = parse_list_marker(text) {
+ // Numbered bold-section heading: "1. **Foo**" / "5. **The dynamics**".
+ // When the post-marker body is uniformly bold + body-sized,
+ // standalone (paragraph-break gap above and below), short, and
+ // mostly alpha, treat it as a heading rather than the first item
+ // of an ordered list. Without this, decimal-numbered section
+ // headings in technical/legal/scientific PDFs silently emit as
+ // ordered list items and lose all heading structure.
+ if ordered
+ && looks_like_numbered_bold_heading(
+ line,
+ rest,
+ paragraph.as_ref().map(|p| &p.last).or(last_list_line.as_ref()),
+ lines.get(idx),
+ )
+ {
+ flush_paragraph(&mut blocks, paragraph.take());
+ list_base_indent = None;
+ last_list_item_idx = None;
+ last_list_line = None;
+ let level = (heading_map.len() as u8 + 1).clamp(1, MAX_HEADING_LEVELS as u8);
+ blocks.push(Block::Heading {
+ level,
+ text: collapse_whitespace(text),
+ });
+ continue;
+ }
+ flush_paragraph(&mut blocks, paragraph.take());
+ let base = *list_base_indent.get_or_insert(line.indent_x);
+ let level = (((line.indent_x - base) / LIST_INDENT_STEP_PT)
+ .round()
+ .max(0.0)) as u8;
+ last_list_item_idx = Some(blocks.len());
+ last_list_line = Some(line.clone());
+ // Render the list-item text via the inline pipeline so per-span
+ // emphasis surfaces. We strip the marker from `rest` (already
+ // done by `parse_list_marker`), but emphasis lives on `line.spans`,
+ // which still contain the marker span — render the line and then
+ // peel the marker off the front of the rendered string.
+ let item_text = render_list_item_text(line, &marker, rest);
+ blocks.push(Block::ListItem {
+ ordered,
+ marker,
+ level,
+ text: item_text,
+ bold: false,
+ italic: false,
+ });
+ continue;
+ }
+
+ // Continuation of a list item: same gap/font rules as paragraphs.
+ // Footnote-style continuations often left-flush below the marker's
+ // hanging indent, so we don't require indent ≥ marker indent.
+ if let Some(idx) = last_list_item_idx
+ && let Some(prev_line) = last_list_line.as_ref()
+ && continues_paragraph(prev_line, line)
+ && let Some(Block::ListItem {
+ text: prev_text, ..
+ }) = blocks.get_mut(idx)
+ {
+ // De-hyphenate against the prior rendered text, then append the
+ // inline-styled continuation.
+ let cont_inline = render_line_inline(line);
+ append_inline_continuation(prev_text, text, &cont_inline);
+ last_list_line = Some(line.clone());
+ continue;
+ }
+
+ // Bold body-size heading. Section headings in academic / technical
+ // PDFs are routinely body-sized + bold (e.g. "Abstract",
+ // "1 Introduction"); without this rule they look just like a bold
+ // first sentence of a paragraph. Runs after list-marker detection so
+ // bold bullet items stay as list items.
+ let prev_for_gap = paragraph
+ .as_ref()
+ .map(|p| &p.last)
+ .or(last_list_line.as_ref());
+ let next_for_gap = lines.get(idx);
+ if looks_like_bold_heading(line, prev_for_gap, next_for_gap) {
+ flush_paragraph(&mut blocks, paragraph.take());
+ list_base_indent = None;
+ last_list_item_idx = None;
+ last_list_line = None;
+ // Level: one deeper than the deepest size-based level we already
+ // have. With an empty heading_map this lands on H1; with a full
+ // 6-level map it caps at H6.
+ let level = (heading_map.len() as u8 + 1).clamp(1, MAX_HEADING_LEVELS as u8);
+ blocks.push(Block::Heading {
+ level,
+ text: collapse_whitespace(text),
+ });
+ continue;
+ }
+
+ match paragraph.as_mut() {
+ Some(acc) if continues_paragraph(&acc.last, line) => {
+ append_to_paragraph(acc, line);
+ }
+ _ => {
+ flush_paragraph(&mut blocks, paragraph.take());
+ list_base_indent = None;
+ last_list_item_idx = None;
+ last_list_line = None;
+ let inline = render_line_inline(line);
+ let raw = collapse_whitespace(text);
+ let uniform = line_uniform_style(line).map(|s| (s.bold, s.italic));
+ paragraph = Some(ParaAccum {
+ raw,
+ inline,
+ last: line.clone(),
+ uniform,
+ });
+ }
+ }
+ }
+
+ flush_paragraph(&mut blocks, paragraph.take());
+ flush_code(&mut blocks, code.take());
+ // Flush any trailing HRs / figures that sat below the last text line.
+ emit_hrs_before(
+ &mut blocks,
+ &mut paragraph,
+ &mut code,
+ &mut list_base_indent,
+ &mut last_list_item_idx,
+ &mut last_list_line,
+ &mut hr_iter,
+ f32::INFINITY,
+ );
+ emit_figures_before(
+ &mut blocks,
+ &mut paragraph,
+ &mut code,
+ &mut list_base_indent,
+ &mut last_list_item_idx,
+ &mut last_list_line,
+ &mut figure_iter,
+ f32::INFINITY,
+ );
+ blocks
+}
+
+#[cfg(test)]
+mod tests {
+ use super::super::headings::{build_heading_map, compute_body_size};
+ use super::super::repetition::compute_header_footer_set;
+ use super::super::test_helpers::{
+ header_footer_page, line, mono_line, page, page_with_graphics, styled_line, stroke,
+ };
+ use super::*;
+ use crate::types::TextItem;
+ use super::super::blocks::{Block, render_blocks};
+
+ #[test]
+ fn classify_emits_heading_and_paragraph() {
+ let p = page(vec![
+ line("Title of the document goes here", 50.0, 50.0, 18.0, 18.0),
+ line("First sentence of the para-", 50.0, 80.0, 10.0, 10.0),
+ line("graph continues here.", 50.0, 92.0, 10.0, 10.0),
+ line("Another paragraph.", 50.0, 130.0, 10.0, 10.0),
+ ]);
+ let pages = vec![p];
+ let body = compute_body_size(&pages);
+ let map = build_heading_map(&pages, body);
+ let blocks = classify_page(&pages[0], &map);
+ assert_eq!(blocks.len(), 3);
+ match &blocks[0] {
+ Block::Heading { level, text } => {
+ assert_eq!(*level, 1);
+ assert_eq!(text, "Title of the document goes here");
+ }
+ other => panic!("expected heading, got {other:?}"),
+ }
+ match &blocks[1] {
+ Block::Paragraph { text: t, .. } => {
+ assert!(t.contains("paragraph continues"), "got: {t}");
+ assert!(!t.contains("para-"), "de-hyphenation failed: {t}");
+ }
+ other => panic!("expected paragraph, got {other:?}"),
+ }
+ match &blocks[2] {
+ Block::Paragraph { text: t, .. } => assert_eq!(t, "Another paragraph."),
+ other => panic!("expected paragraph, got {other:?}"),
+ }
+ }
+
+ #[test]
+ fn paragraph_break_on_big_gap() {
+ let p = page(vec![
+ line("Line A.", 50.0, 80.0, 10.0, 10.0),
+ line("Line B.", 50.0, 200.0, 10.0, 10.0),
+ ]);
+ let pages = vec![p];
+ let body = compute_body_size(&pages);
+ let map = build_heading_map(&pages, body);
+ let blocks = classify_page(&pages[0], &map);
+ assert_eq!(blocks.len(), 2);
+ }
+
+ #[test]
+ fn classify_emits_list_items() {
+ let p = page(vec![
+ line("Intro line.", 50.0, 50.0, 10.0, 10.0),
+ line("• first bullet", 60.0, 80.0, 10.0, 10.0),
+ line("• second bullet", 60.0, 92.0, 10.0, 10.0),
+ line("◦ nested item", 72.0, 104.0, 10.0, 10.0),
+ line("• back to top", 60.0, 116.0, 10.0, 10.0),
+ ]);
+ let pages = vec![p];
+ let body = compute_body_size(&pages);
+ let map = build_heading_map(&pages, body);
+ let blocks = classify_page(&pages[0], &map);
+ let list_items: Vec<&Block> = blocks
+ .iter()
+ .filter(|b| matches!(b, Block::ListItem { .. }))
+ .collect();
+ assert_eq!(list_items.len(), 4);
+ if let Block::ListItem { level, text, .. } = list_items[0] {
+ assert_eq!(*level, 0);
+ assert_eq!(text, "first bullet");
+ } else {
+ panic!();
+ }
+ // The "- nested item" line is indented +12pt from the base bullet.
+ if let Block::ListItem { level, .. } = list_items[2] {
+ assert_eq!(*level, 1);
+ } else {
+ panic!();
+ }
+ }
+
+ #[test]
+ fn classify_emits_code_block() {
+ let p = page(vec![
+ line("Intro line.", 50.0, 50.0, 10.0, 10.0),
+ mono_line(" let x = 1;", 80.0),
+ mono_line(" let y = x + 2;", 92.0),
+ line("After the code.", 50.0, 120.0, 10.0, 10.0),
+ ]);
+ let pages = vec![p];
+ let body = compute_body_size(&pages);
+ let map = build_heading_map(&pages, body);
+ let blocks = classify_page(&pages[0], &map);
+ // Expect: Paragraph("Intro line."), CodeBlock(2 lines), Paragraph("After...")
+ assert_eq!(blocks.len(), 3);
+ match &blocks[1] {
+ Block::CodeBlock { lines } => {
+ assert_eq!(lines.len(), 2);
+ assert!(lines[0].contains("let x = 1;"));
+ assert!(lines[1].contains("let y = x + 2;"));
+ }
+ other => panic!("expected code block, got {other:?}"),
+ }
+ let s = render_blocks(&blocks);
+ assert!(s.contains("```\n let x = 1;"));
+ assert!(s.ends_with("After the code."));
+ }
+
+ #[test]
+ fn classify_marks_paragraph_bold_when_all_lines_bold() {
+ let mut a = line("Bold line one.", 50.0, 50.0, 10.0, 10.0);
+ let mut b = line("bold continuation.", 50.0, 62.0, 10.0, 10.0);
+ // Mark the underlying spans as bold so per-span style detection sees
+ // it — the new inline pipeline reads from `spans`, not the per-line
+ // `all_bold` shortcut flag.
+ let bold_span = TextItem {
+ text: "x".into(),
+ font_name: Some("Arial-Bold".into()),
+ ..Default::default()
+ };
+ a.spans = vec![bold_span.clone()];
+ b.spans = vec![bold_span];
+ a.all_bold = true;
+ b.all_bold = true;
+ let p = page(vec![a, b]);
+ let pages = vec![p];
+ let body = compute_body_size(&pages);
+ let map = build_heading_map(&pages, body);
+ let blocks = classify_page(&pages[0], &map);
+ assert_eq!(blocks.len(), 1);
+ match &blocks[0] {
+ Block::Paragraph { bold, italic, .. } => {
+ assert!(*bold);
+ assert!(!*italic);
+ }
+ other => panic!("expected paragraph, got {other:?}"),
+ }
+ let s = render_blocks(&blocks);
+ assert!(s.starts_with("**") && s.ends_with("**"), "got: {s}");
+ }
+
+ #[test]
+ fn detects_simple_borderless_table() {
+ use super::super::test_helpers::line_with_spans;
+ let lines = vec![
+ line_with_spans(
+ &[("Name", 50.0), ("Age", 150.0), ("City", 250.0)],
+ 100.0,
+ 10.0,
+ ),
+ line_with_spans(
+ &[("Alice", 50.0), ("30", 150.0), ("NYC", 250.0)],
+ 115.0,
+ 10.0,
+ ),
+ line_with_spans(&[("Bob", 50.0), ("25", 150.0), ("LA", 250.0)], 130.0, 10.0),
+ ];
+ let p = page(lines);
+ let pages = vec![p];
+ let body = compute_body_size(&pages);
+ let map = build_heading_map(&pages, body);
+ let blocks = classify_page(&pages[0], &map);
+ assert_eq!(blocks.len(), 1, "got: {blocks:?}");
+ match &blocks[0] {
+ Block::Table { header, rows } => {
+ // Header isn't bold so no header row promoted.
+ assert!(header.is_none());
+ assert_eq!(rows.len(), 3);
+ assert_eq!(rows[0][0], "Name");
+ assert_eq!(rows[1][2], "NYC");
+ }
+ other => panic!("expected table, got {other:?}"),
+ }
+ }
+
+ #[test]
+ fn full_format_strips_header_footer() {
+ let pages = vec![
+ header_footer_page(1, "Acme Confidential", "Page 1 of 2", "First page body."),
+ header_footer_page(2, "Acme Confidential", "Page 2 of 2", "Second page body."),
+ ];
+ let body = compute_body_size(&pages);
+ let map = build_heading_map(&pages, body);
+ let set = compute_header_footer_set(&pages);
+ let blocks = classify_page_with_filters(
+ &pages[0],
+ &map,
+ &set,
+ &[],
+ crate::config::ImageMode::Placeholder,
+ );
+ let s = render_blocks(&blocks);
+ assert!(!s.contains("Acme Confidential"), "got: {s}");
+ assert!(!s.contains("Page 1 of 2"), "got: {s}");
+ assert!(s.contains("First page body."));
+ }
+
+ #[test]
+ fn classify_paragraph_with_mid_line_bold() {
+ // First line has a bold word mid-line → not uniformly styled; paragraph
+ // should emit baked-in `**bold**` inside the text and `bold=false` at
+ // the block level.
+ let a = styled_line(
+ &[
+ ("a sentence with a", 50.0, Some("Arial")),
+ ("bold", 180.0, Some("Arial-Bold")),
+ ("word in it.", 230.0, Some("Arial")),
+ ],
+ 50.0,
+ 10.0,
+ );
+ let p = page(vec![a]);
+ let pages = vec![p];
+ let body = compute_body_size(&pages);
+ let map = build_heading_map(&pages, body);
+ let blocks = classify_page(&pages[0], &map);
+ assert_eq!(blocks.len(), 1, "got: {blocks:?}");
+ match &blocks[0] {
+ Block::Paragraph { text, bold, italic } => {
+ assert!(!*bold, "mixed-style paragraph shouldn't set block bold");
+ assert!(!*italic);
+ assert!(text.contains("**bold**"), "got: {text}");
+ }
+ other => panic!("expected paragraph, got {other:?}"),
+ }
+ }
+
+ #[test]
+ fn classify_list_item_strips_marker_under_emphasis() {
+ // Whole bullet line is bold (marker + text). Rendered text should be
+ // wrapped, with the marker dropped (the renderer prints it).
+ let l = styled_line(
+ &[
+ ("•", 60.0, Some("Arial-Bold")),
+ ("important item", 80.0, Some("Arial-Bold")),
+ ],
+ 50.0,
+ 10.0,
+ );
+ let p = page(vec![l]);
+ let pages = vec![p];
+ let body = compute_body_size(&pages);
+ let map = build_heading_map(&pages, body);
+ let blocks = classify_page(&pages[0], &map);
+ assert_eq!(blocks.len(), 1);
+ match &blocks[0] {
+ Block::ListItem { text, .. } => {
+ assert_eq!(text, "**important item**");
+ }
+ other => panic!("expected list item, got {other:?}"),
+ }
+ }
+
+ #[test]
+ fn hr_emitted_between_lines_by_y_order() {
+ let lines = vec![
+ line("before the rule", 50.0, 100.0, 10.0, 10.0),
+ line("after the rule", 50.0, 300.0, 10.0, 10.0),
+ ];
+ // Stroke between the two lines, far from either's baseline.
+ let p = page_with_graphics(lines, vec![stroke(50.0, 200.0, 450.0, 200.0, 0.5)]);
+ let blocks = classify_page(&p, &[]);
+ let has_hr = blocks
+ .iter()
+ .position(|b| matches!(b, Block::HorizontalRule));
+ assert!(has_hr.is_some(), "expected an HR block, got {blocks:?}");
+ // HR must land between the two paragraphs, not before/after both.
+ let pos = has_hr.unwrap();
+ assert!(pos > 0 && pos < blocks.len() - 1);
+ }
+}
diff --git a/crates/liteparse/src/markdown_layout/headings.rs b/crates/liteparse/src/markdown_layout/headings.rs
new file mode 100644
index 0000000..3421d5a
--- /dev/null
+++ b/crates/liteparse/src/markdown_layout/headings.rs
@@ -0,0 +1,526 @@
+use crate::types::{OutlineTarget, ParsedPage, ProjectedLine, StructNode};
+
+use super::classify::is_rotated_line;
+use super::inline::line_uniform_style;
+use super::paragraphs::continues_paragraph;
+use super::tables::{TABLE_MIN_COLUMNS, split_cells};
+
+/// Tolerance in points for "is this size larger than the body size".
+pub(super) const HEADING_SIZE_EPSILON: f32 = 0.5;
+
+/// Cap on heading levels (matches plan: H1..H6).
+pub(super) const MAX_HEADING_LEVELS: usize = 6;
+
+/// Tighter tolerance for matching against the heading-size map. Keeps the
+/// heading detector strict so descender-induced height jitter doesn't promote
+/// regular body lines to headings.
+pub(super) const FONT_SIZE_HEADING_TOLERANCE: f32 = 0.6;
+
+/// Maximum characters in a "bold body-size heading" candidate. Section
+/// headings like "Abstract", "1 Introduction", "2.1 Related Work" are short;
+/// a bold body-size line longer than this is almost always a bold *sentence*
+/// inside a paragraph, not a heading.
+pub(super) const BOLD_HEADING_MAX_CHARS: usize = 80;
+
+/// Recognize caption-prefix lines like "Figure 7", "Fig. 12.", "Table 3:",
+/// "Tab. 5", "Equation (4)" — these routinely render in a slightly distinct
+/// font/size that lands them in the heading_map and gets them promoted to a
+/// document-level heading. We want to keep them as plain paragraphs.
+pub(super) fn is_caption_line(text: &str) -> bool {
+ let t = text.trim_start();
+ // Try each known prefix: must be followed by a number (optionally with
+ // separators) within the first ~20 chars.
+ const PREFIXES: &[&str] = &[
+ "Figure", "Figures", "Fig.", "Fig ", "Table", "Tables", "Tab.", "Tab ",
+ "Equation", "Eq.", "Eq ", "Scheme", "Chart", "Plate", "Photo", "Algorithm", "Listing",
+ ];
+ let lower_t_first_word: String = t
+ .chars()
+ .take_while(|c| c.is_alphabetic() || *c == '.')
+ .collect();
+ for p in PREFIXES {
+ let p_trim = p.trim_end();
+ if lower_t_first_word.eq_ignore_ascii_case(p_trim) {
+ // Look at what follows the prefix word.
+ let rest = t[lower_t_first_word.len()..].trim_start();
+ // Allow a leading "(" then digit, or directly a digit / roman numeral.
+ let mut chars = rest.chars();
+ if let Some(c0) = chars.next()
+ && (c0.is_ascii_digit()
+ || (c0 == '(' && chars.next().is_some_and(|c| c.is_ascii_digit()))
+ || matches!(c0, 'I' | 'V' | 'X' | 'L' | 'C'))
+ {
+ return true;
+ }
+ }
+ }
+ false
+}
+
+/// Returns true if `line` looks like a section heading rendered in body-size
+/// bold text (a very common style for academic / technical PDFs where every
+/// "real" heading uses the same font size as body, distinguished only by
+/// weight). Requires:
+/// - uniformly bold across all spans
+/// - short (≤ `BOLD_HEADING_MAX_CHARS`)
+/// - paragraph-break gap above (or first line on the page)
+/// - paragraph-break gap below (or last line on the page)
+pub(super) fn looks_like_bold_heading(
+ line: &ProjectedLine,
+ prev: Option<&ProjectedLine>,
+ next: Option<&ProjectedLine>,
+) -> bool {
+ let text = line.text.trim();
+ if text.is_empty() || text.chars().count() > BOLD_HEADING_MAX_CHARS {
+ return false;
+ }
+ // Captions ("Figure 7", "Table 3.") are commonly bold body-sized lines
+ // that would otherwise satisfy every other rule here. Keep them as
+ // paragraphs so they don't appear in the heading hierarchy.
+ if is_caption_line(text) {
+ return false;
+ }
+ let style = match line_uniform_style(line) {
+ Some(s) => s,
+ None => return false,
+ };
+ if !style.bold || style.mono {
+ return false;
+ }
+ // Reject bold-uniform lines dominated by digits / punctuation — these are
+ // almost always cells inside a tabular layout the table detector didn't
+ // pick up (results tables, scoreboards, math display). A real section
+ // heading is mostly letters: "1 Introduction" passes (~92% alpha across
+ // non-whitespace chars), "47.5 14" doesn't (0%), "BLEU-1 25.87" doesn't.
+ let mut alpha = 0usize;
+ let mut total = 0usize;
+ for c in text.chars() {
+ if c.is_whitespace() {
+ continue;
+ }
+ total += 1;
+ if c.is_alphabetic() {
+ alpha += 1;
+ }
+ }
+ if total == 0 || (alpha as f32) / (total as f32) < 0.5 {
+ return false;
+ }
+ // Reject when the line itself looks tabular: ≥3 cells separated by font-size
+ // gaps. A bold body-size line with that many cell tracks is almost always a
+ // table header row, not a section heading. Without this guard, multi-line
+ // table headers ("Model Method ... | F1 BLEU-1 F1 BLEU-1 ...") get promoted
+ // to H3 instead of being absorbed by the table detector.
+ // Tabular shape rejection. Two passes because the projection sometimes
+ // collapses a wide multi-column line into a single span with column-
+ // padding spaces — span-based detection misses those.
+ if split_cells(line).len() >= TABLE_MIN_COLUMNS {
+ return false;
+ }
+ // Text-based fallback: ≥3 tokens separated by runs of 2+ spaces (the
+ // projection inserts alignment padding between cells) → table header row
+ // collapsed into one span, not a section heading.
+ let multi_space_tokens = text
+ .split(" ")
+ .map(|s| s.trim())
+ .filter(|s| !s.is_empty())
+ .count();
+ if multi_space_tokens >= TABLE_MIN_COLUMNS {
+ return false;
+ }
+ let gap_above_ok = match prev {
+ None => true,
+ Some(p) => !continues_paragraph(p, line),
+ };
+ if !gap_above_ok {
+ return false;
+ }
+ let gap_below_ok = match next {
+ None => true,
+ Some(n) => !continues_paragraph(line, n),
+ };
+ gap_below_ok
+}
+
+/// Returns true if `line` is a numbered section heading like "1. **Foo**" —
+/// `parse_list_marker` already matched the "N." / "N)" prefix; this checks
+/// that the body after the marker is uniformly bold body-size text and that
+/// the line stands alone (paragraph break above and below). When true the
+/// caller should emit a Heading at `heading_map.len()+1` rather than a
+/// ListItem. Mirrors `looks_like_bold_heading`'s gating modulo the marker.
+pub(super) fn looks_like_numbered_bold_heading(
+ line: &ProjectedLine,
+ rest: &str,
+ prev: Option<&ProjectedLine>,
+ _next: Option<&ProjectedLine>,
+) -> bool {
+ let rest_trim = rest.trim();
+ if rest_trim.is_empty() || rest_trim.chars().count() > BOLD_HEADING_MAX_CHARS {
+ return false;
+ }
+ if is_caption_line(&line.text) {
+ return false;
+ }
+ if rest_trim.ends_with('.')
+ && rest_trim
+ .chars()
+ .filter(|c| *c == '.' || *c == '?' || *c == '!')
+ .count()
+ >= 2
+ {
+ // "1. Sentence one. Sentence two." → not a heading.
+ return false;
+ }
+ // The spans after the marker must all be bold and non-mono. Marker
+ // characters are typically `'0'..='9'`, `'.'`, `')'`, plus whitespace —
+ // identify and skip them at the front of the span list.
+ let mut saw_bold_body = false;
+ let mut saw_non_bold_body = false;
+ for span in &line.spans {
+ let text = span.text.trim();
+ if text.is_empty() {
+ continue;
+ }
+ let is_marker = text
+ .chars()
+ .all(|c| c.is_ascii_digit() || c == '.' || c == ')' || c == '(');
+ if is_marker {
+ continue;
+ }
+ if crate::projection::is_mono_item(span) {
+ return false;
+ }
+ if crate::projection::is_bold_item(span) {
+ saw_bold_body = true;
+ } else {
+ saw_non_bold_body = true;
+ }
+ }
+ if !saw_bold_body || saw_non_bold_body {
+ return false;
+ }
+ // Mostly alphabetic — same intuition as `looks_like_bold_heading`'s
+ // alpha-ratio filter: rejects tabular bold rows of digits.
+ let (mut alpha, mut total) = (0usize, 0usize);
+ for c in rest_trim.chars() {
+ if c.is_whitespace() {
+ continue;
+ }
+ total += 1;
+ if c.is_alphabetic() {
+ alpha += 1;
+ }
+ }
+ if total == 0 || (alpha as f32) / (total as f32) < 0.5 {
+ return false;
+ }
+ // Paragraph-break gap above. We deliberately don't require gap_below:
+ // a numbered section heading is often followed by another bold body
+ // line (a sub-heading or a multi-line title continuation) which would
+ // satisfy `continues_paragraph`. The numbered+bold combination is
+ // distinctive enough that the false-positive risk is small.
+ match prev {
+ None => true,
+ Some(p) => !continues_paragraph(p, line),
+ }
+}
+
+/// Compute the body font size as the char-weighted mode across all lines in
+/// all pages. Rotated lines are excluded so a long rotated sidebar can't
+/// pull the body estimate. Falls back to `0.0` when no font-size info is
+/// available.
+pub fn compute_body_size(pages: &[ParsedPage]) -> f32 {
+ use std::collections::HashMap;
+ let mut weights: HashMap<u32, (f32, usize)> = HashMap::new();
+ for page in pages {
+ for line in &page.projected_lines {
+ if is_rotated_line(line) {
+ continue;
+ }
+ let size = line.dominant_font_size;
+ if size <= 0.0 {
+ continue;
+ }
+ let chars = line.text.chars().count().max(1);
+ let key = (size * 100.0).round() as u32;
+ let entry = weights.entry(key).or_insert((size, 0));
+ entry.1 += chars;
+ }
+ }
+ weights
+ .values()
+ .max_by_key(|(_, n)| *n)
+ .map(|(s, _)| *s)
+ .unwrap_or(0.0)
+}
+
+/// Minimum total non-whitespace characters across all occurrences at a font
+/// size for it to qualify as a heading level. Calibrated against `paper.pdf`:
+/// the 30pt chart-legend tokens ("A-mem"×2 + "Base"×2 = 18-20 chars) need to
+/// fail this filter, while the 14.35pt title ("A-MEM: Agentic Memory for LLM
+/// Agents" = 31 chars on a single line) needs to pass. 25 is the gap. Smaller
+/// single-word headings like a lone "Summary" on a short doc still survive
+/// because they share their font size with other (larger) headings in the
+/// histogram entry.
+const MIN_HEADING_TOTAL_CHARS: usize = 25;
+
+/// Maximum average characters per line for a size to qualify as a heading.
+/// A "size larger than body" with very long lines is almost always a
+/// large-print body block (callouts, footnotes-as-display, intro paragraph),
+/// not a real heading.
+const MAX_HEADING_AVG_LINE_CHARS: f32 = 200.0;
+
+/// Minimum fraction of non-whitespace chars at a size that must be alphabetic
+/// for it to qualify as a heading. Drops sizes dominated by digits (graph
+/// axes, results tables, math display) which otherwise pollute the top
+/// heading slots.
+const MIN_HEADING_ALPHA_RATIO: f32 = 0.5;
+
+/// Build a heading-size → level map: sizes strictly larger than `body_size`,
+/// filtered to those with at least `MIN_HEADING_LINES` distinct occurrences
+/// (drops one-off equation/figure-label artifacts), sorted descending, mapped
+/// to levels 1..=MAX_HEADING_LEVELS.
+pub fn build_heading_map(pages: &[ParsedPage], body_size: f32) -> Vec<(f32, u8)> {
+ use std::collections::HashMap;
+ // (size_key → (size, line_count, total_chars, alpha_chars))
+ let mut sizes: HashMap<u32, (f32, usize, usize, usize)> = HashMap::new();
+ for page in pages {
+ for line in &page.projected_lines {
+ if is_rotated_line(line) {
+ continue;
+ }
+ // Captions ("Figure 7", "Table 3.") often render slightly larger
+ // than body and would otherwise inflate / hijack the heading map.
+ if is_caption_line(&line.text) {
+ continue;
+ }
+ let size = line.dominant_font_size;
+ if size > body_size + HEADING_SIZE_EPSILON {
+ let key = (size * 100.0).round() as u32;
+ let entry = sizes.entry(key).or_insert((size, 0, 0, 0));
+ entry.1 += 1;
+ for c in line.text.chars() {
+ if c.is_whitespace() {
+ continue;
+ }
+ entry.2 += 1;
+ if c.is_alphabetic() {
+ entry.3 += 1;
+ }
+ }
+ }
+ }
+ }
+ let all: Vec<(f32, usize, usize, usize)> = sizes.into_values().collect();
+ // Always apply quality filters: total-char floor, average-line cap, and
+ // alpha-ratio floor. The total-char floor (rather than a line-count one)
+ // lets one-off titles survive — a 31-char title on a single line passes —
+ // while still rejecting chart-legend tokens like "A-mem" + "Base" that
+ // total fewer chars across 4 occurrences than a single heading line does.
+ let mut kept: Vec<f32> = all
+ .iter()
+ .filter(|(_, lines, chars, alpha)| {
+ let alpha_ratio = if *chars == 0 {
+ 0.0
+ } else {
+ (*alpha as f32) / (*chars as f32)
+ };
+ *chars >= MIN_HEADING_TOTAL_CHARS
+ && (*chars as f32 / (*lines).max(1) as f32) <= MAX_HEADING_AVG_LINE_CHARS
+ && alpha_ratio >= MIN_HEADING_ALPHA_RATIO
+ })
+ .map(|(s, _, _, _)| *s)
+ .collect();
+ kept.sort_by(|a, b| b.total_cmp(a));
+ kept.truncate(MAX_HEADING_LEVELS);
+ kept.into_iter()
+ .enumerate()
+ .map(|(i, s)| (s, (i + 1) as u8))
+ .collect()
+}
+
+pub(super) fn heading_level_for(size: f32, heading_map: &[(f32, u8)]) -> Option<u8> {
+ for (s, level) in heading_map {
+ if (size - *s).abs() < FONT_SIZE_HEADING_TOLERANCE {
+ return Some(*level);
+ }
+ }
+ None
+}
+
+/// Highest-priority heading source: a struct-tree node `H1`..`H6` directly
+/// tagging this line via its `mcid`. Available only for tagged PDFs.
+pub(super) fn struct_heading_level(line: &ProjectedLine, struct_nodes: &[StructNode]) -> Option<u8> {
+ let mcid = line.mcid?;
+ for node in struct_nodes {
+ if !node.mcids.contains(&mcid) {
+ continue;
+ }
+ if let Some(level) = parse_heading_role(&node.role) {
+ return Some(level);
+ }
+ }
+ None
+}
+
+/// Parse a struct-tree role string like "H1" or "H3" into a heading level.
+/// Returns None for non-heading roles (P, L, Figure, Table, ...).
+fn parse_heading_role(role: &str) -> Option<u8> {
+ let trimmed = role.trim();
+ if !trimmed.starts_with('H') && !trimmed.starts_with('h') {
+ return None;
+ }
+ let digits = &trimmed[1..];
+ let n: u8 = digits.parse().ok()?;
+ if (1..=6).contains(&n) { Some(n) } else { None }
+}
+
+/// Second-priority heading source: a document outline (bookmark) entry that
+/// points at this page near this line's y coordinate, with a title that
+/// prefix-matches the line text. Used on untagged PDFs that ship a TOC.
+pub(super) fn outline_heading_level(
+ line: &ProjectedLine,
+ page_height: f32,
+ outline: &[OutlineTarget],
+ line_text: &str,
+) -> Option<u8> {
+ if outline.is_empty() {
+ return None;
+ }
+ let normalized_line = normalize_outline_text(line_text);
+ if normalized_line.is_empty() {
+ return None;
+ }
+ let row_h = line.bbox.height.max(8.0);
+ let y_tolerance = row_h * 1.5;
+ for entry in outline {
+ let normalized_title = normalize_outline_text(&entry.title);
+ if normalized_title.is_empty() {
+ continue;
+ }
+ // Spatial check is *strict* only when the entry actually carries a
+ // usable y (in-page). Many bookmarks point at "top of page" with a
+ // Y outside the MediaBox or no Y at all — in that case we still
+ // accept any line on the page that prefix-matches the title.
+ let y_ok = match entry.y_pdf {
+ Some(y) => {
+ let y_view = page_height - y;
+ if y_view < 0.0 || y_view > page_height {
+ true
+ } else {
+ (y_view - line.bbox.y).abs() <= y_tolerance
+ }
+ }
+ None => true,
+ };
+ if !y_ok {
+ continue;
+ }
+ // Short outline titles ("Nutrition") would otherwise false-match any
+ // paragraph that happens to start with them. Require the matched line
+ // to be heading-shaped: not much longer than the title itself.
+ let line_len = normalized_line.chars().count();
+ let title_len = normalized_title.chars().count();
+ let max_line_len = (title_len * 3).max(120);
+ // Multiple sentences → almost certainly prose, not a heading line.
+ let sentence_breaks = normalized_line.matches(". ").count();
+ if line_len > max_line_len || sentence_breaks >= 2 {
+ continue;
+ }
+ if normalized_line.starts_with(&normalized_title)
+ || normalized_title.starts_with(&normalized_line)
+ {
+ return Some(entry.level.min(MAX_HEADING_LEVELS as u8));
+ }
+ }
+ None
+}
+
+/// Lowercase + collapse whitespace, for forgiving outline-title vs line-text
+/// comparison. Outline titles often have trailing numbering or punctuation
+/// not present on the rendered line (and vice versa).
+fn normalize_outline_text(s: &str) -> String {
+ let mut out = String::with_capacity(s.len());
+ let mut prev_space = true;
+ for c in s.chars() {
+ if c.is_whitespace() {
+ if !prev_space {
+ out.push(' ');
+ prev_space = true;
+ }
+ } else {
+ for lc in c.to_lowercase() {
+ out.push(lc);
+ }
+ prev_space = false;
+ }
+ }
+ if out.ends_with(' ') {
+ out.pop();
+ }
+ out
+}
+
+#[cfg(test)]
+mod tests {
+ use super::super::test_helpers::{line, page};
+ use super::*;
+
+ #[test]
+ fn body_size_picks_most_common() {
+ let pages = vec![page(vec![
+ line("Title", 50.0, 50.0, 18.0, 18.0),
+ line("body line one", 50.0, 80.0, 10.0, 10.0),
+ line("body line two", 50.0, 92.0, 10.0, 10.0),
+ line("body line three", 50.0, 104.0, 10.0, 10.0),
+ ])];
+ let body = compute_body_size(&pages);
+ assert!((body - 10.0).abs() < 0.01, "body size = {body}");
+ }
+
+ #[test]
+ fn heading_map_descending_levels() {
+ // Heading text needs to clear `MIN_HEADING_TOTAL_CHARS` (25) so the
+ // size qualifies as a real heading rather than chart-legend noise.
+ let pages = vec![page(vec![
+ line("The largest heading on the page", 50.0, 50.0, 24.0, 24.0),
+ line("A smaller heading right below it", 50.0, 80.0, 18.0, 18.0),
+ // Several lines of body so it beats the heading text in the
+ // char-weighted body-size mode.
+ line(
+ "body text line one with plenty of content",
+ 50.0,
+ 110.0,
+ 10.0,
+ 10.0,
+ ),
+ line(
+ "body text line two with plenty of content",
+ 50.0,
+ 122.0,
+ 10.0,
+ 10.0,
+ ),
+ line(
+ "body text line three with even more content",
+ 50.0,
+ 134.0,
+ 10.0,
+ 10.0,
+ ),
+ line(
+ "body text line four with even more content",
+ 50.0,
+ 146.0,
+ 10.0,
+ 10.0,
+ ),
+ ])];
+ let body = compute_body_size(&pages);
+ let map = build_heading_map(&pages, body);
+ assert_eq!(map.len(), 2);
+ assert_eq!(map[0].1, 1);
+ assert_eq!(map[1].1, 2);
+ assert!(map[0].0 > map[1].0);
+ }
+}
diff --git a/crates/liteparse/src/markdown_layout/hr.rs b/crates/liteparse/src/markdown_layout/hr.rs
new file mode 100644
index 0000000..7e8f4ad
--- /dev/null
+++ b/crates/liteparse/src/markdown_layout/hr.rs
@@ -0,0 +1,132 @@
+use crate::types::{GraphicPrimitive, ParsedPage};
+
+/// Maximum stroke thickness (or |y1-y2|) for a stroke to count as a candidate HR.
+/// Thicker shapes are filled rects, not rules.
+const HR_MAX_THICKNESS_PT: f32 = 2.0;
+
+/// Minimum fraction of page width a horizontal stroke must span to count as an HR.
+/// Shorter strokes are typically table borders, list bullets, or inline marks.
+const HR_MIN_WIDTH_FRACTION: f32 = 0.3;
+
+/// Vertical tolerance (points) for treating a stroke as "underlining" the
+/// nearest text line. Strokes within this band of a text line's baseline are
+/// dropped — they're underlines, not rules.
+const HR_UNDERLINE_PROXIMITY_PT: f32 = 3.0;
+
+/// Detect horizontal rules from a page's vector graphics.
+///
+/// Returns the y-coordinates (viewport space) of accepted HRs, sorted ascending.
+/// An HR is a roughly horizontal stroke that spans at least
+/// `HR_MIN_WIDTH_FRACTION` of the page width, is thinner than
+/// `HR_MAX_THICKNESS_PT`, and does not sit on the baseline of any text line
+/// (which would make it an underline).
+pub(super) fn detect_horizontal_rules(page: &ParsedPage) -> Vec<f32> {
+ if page.graphics.is_empty() || page.page_width <= 0.0 {
+ return Vec::new();
+ }
+ let min_width = page.page_width * HR_MIN_WIDTH_FRACTION;
+ let mut ys: Vec<f32> = Vec::new();
+
+ for g in &page.graphics {
+ let GraphicPrimitive::Stroke {
+ x1,
+ y1,
+ x2,
+ y2,
+ width,
+ ..
+ } = g
+ else {
+ continue;
+ };
+ let (x1, y1, x2, y2, width) = (*x1, *y1, *x2, *y2, *width);
+ let dy = (y1 - y2).abs();
+ let dx = (x1 - x2).abs();
+ if dy > HR_MAX_THICKNESS_PT || width > HR_MAX_THICKNESS_PT {
+ continue;
+ }
+ if dx < min_width {
+ continue;
+ }
+ let y = (y1 + y2) * 0.5;
+ let xmin = x1.min(x2);
+ let xmax = x1.max(x2);
+
+ // Drop if this stroke sits on a text-line baseline — it's an underline,
+ // not a divider.
+ let is_underline = page.projected_lines.iter().any(|line| {
+ let bottom = line.bbox.y + line.bbox.height;
+ (y - bottom).abs() < HR_UNDERLINE_PROXIMITY_PT
+ && xmin >= line.bbox.x - 2.0
+ && xmax <= line.bbox.x + line.bbox.width + 2.0
+ });
+ if is_underline {
+ continue;
+ }
+ ys.push(y);
+ }
+
+ // Sort + dedup near-duplicates (some PDFs draw the same rule twice).
+ ys.sort_by(|a, b| a.total_cmp(b));
+ ys.dedup_by(|a, b| (*a - *b).abs() < 1.0);
+ ys
+}
+
+#[cfg(test)]
+mod tests {
+ use super::super::test_helpers::{line, page_with_graphics, stroke};
+ use super::*;
+
+ #[test]
+ fn hr_long_thin_horizontal_stroke_detected() {
+ // 400pt wide stroke on a 612pt page → comfortably above 30% threshold.
+ let p = page_with_graphics(vec![], vec![stroke(50.0, 200.0, 450.0, 200.5, 0.5)]);
+ let ys = detect_horizontal_rules(&p);
+ assert_eq!(ys, vec![200.25]);
+ }
+
+ #[test]
+ fn hr_short_stroke_rejected() {
+ // 50pt wide — table border or list bullet, not an HR.
+ let p = page_with_graphics(vec![], vec![stroke(50.0, 200.0, 100.0, 200.0, 0.5)]);
+ assert!(detect_horizontal_rules(&p).is_empty());
+ }
+
+ #[test]
+ fn hr_vertical_stroke_rejected() {
+ let p = page_with_graphics(vec![], vec![stroke(50.0, 50.0, 50.0, 500.0, 0.5)]);
+ assert!(detect_horizontal_rules(&p).is_empty());
+ }
+
+ #[test]
+ fn hr_thick_stroke_rejected() {
+ // 4pt-thick stroke → a filled bar, not a rule.
+ let p = page_with_graphics(vec![], vec![stroke(50.0, 200.0, 450.0, 200.0, 4.0)]);
+ assert!(detect_horizontal_rules(&p).is_empty());
+ }
+
+ #[test]
+ fn hr_underline_at_text_baseline_dropped() {
+ // Text line at y=100 height=10 → bottom at y=110. Stroke at y=111 within
+ // the line's horizontal extent → underline, not an HR.
+ let text_line = line(
+ "Some underlined heading text on the page",
+ 50.0,
+ 100.0,
+ 10.0,
+ 10.0,
+ );
+ let bottom = text_line.bbox.y + text_line.bbox.height;
+ let p = page_with_graphics(
+ vec![text_line.clone()],
+ vec![stroke(
+ 50.0,
+ bottom + 1.0,
+ 50.0 + text_line.bbox.width,
+ bottom + 1.0,
+ 0.5,
+ )],
+ );
+ assert!(detect_horizontal_rules(&p).is_empty());
+ }
+}
diff --git a/crates/liteparse/src/markdown_layout/inline.rs b/crates/liteparse/src/markdown_layout/inline.rs
new file mode 100644
index 0000000..bed7f80
--- /dev/null
+++ b/crates/liteparse/src/markdown_layout/inline.rs
@@ -0,0 +1,319 @@
+use crate::projection::{is_bold_item, is_italic_item, is_mono_item};
+use crate::types::{ProjectedLine, TextItem};
+
+use super::paragraphs::collapse_whitespace;
+
+/// Lightweight bold detection on a raw `TextItem`. Delegates to
+/// `projection::is_bold_item` so per-span and per-line logic stay in sync.
+pub(super) fn is_bold_span(s: &TextItem) -> bool {
+ is_bold_item(s)
+}
+
+/// Per-span style flags used by the inline-emphasis renderer.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub(super) struct SpanStyle {
+ pub(super) bold: bool,
+ pub(super) italic: bool,
+ pub(super) mono: bool,
+}
+
+impl SpanStyle {
+ pub(super) fn from_item(item: &TextItem) -> Self {
+ SpanStyle {
+ bold: is_bold_item(item),
+ italic: is_italic_item(item),
+ mono: is_mono_item(item),
+ }
+ }
+
+ pub(super) fn is_plain(self) -> bool {
+ !self.bold && !self.italic && !self.mono
+ }
+}
+
+/// Escape characters that would otherwise be interpreted as markdown emphasis.
+/// Deliberately narrow: only `*`, `_`, and backslash. Aggressive escaping
+/// (`#`, `[`, backticks, etc.) breaks more output than it saves in practice —
+/// pymupdf4llm takes the same conservative stance.
+pub(super) fn escape_inline(s: &str) -> String {
+ let mut out = String::with_capacity(s.len());
+ for c in s.chars() {
+ match c {
+ '\\' | '*' | '_' => {
+ out.push('\\');
+ out.push(c);
+ }
+ _ => out.push(c),
+ }
+ }
+ out
+}
+
+/// Wrap `inner` with the markdown markers for `style`. Mono wins over bold/italic:
+/// inline code (`` `…` ``) doesn't compose with emphasis in CommonMark, so when
+/// a span is mono we drop the `**/*` wrap. Bold + italic → `***…***`.
+pub(super) fn apply_style(inner: &str, style: SpanStyle) -> String {
+ if style.mono {
+ // Use backticks; if inner already contains backticks, switch to a
+ // longer fence (pair of backticks plus a space buffer) per CommonMark.
+ if inner.contains('`') {
+ return format!("`` {} ``", inner);
+ }
+ return format!("`{}`", inner);
+ }
+ match (style.bold, style.italic) {
+ (true, true) => format!("***{}***", inner),
+ (true, false) => format!("**{}**", inner),
+ (false, true) => format!("*{}*", inner),
+ (false, false) => inner.to_string(),
+ }
+}
+
+/// Render a `ProjectedLine` to markdown with per-span emphasis. Adjacent
+/// same-style spans are merged into a single emphasis run; whitespace between
+/// spans is preserved as one space (the underlying projection output already
+/// has the right inter-word spacing baked into span text).
+///
+/// Per-line shortcut: when every non-whitespace span shares the same style,
+/// emit one outer wrap around the collapsed line text instead of run-by-run
+/// (matches pymupdf4llm; avoids `**foo** **bar** **baz**` noise on uniformly
+/// styled lines).
+///
+/// Falls back to `collapse_whitespace(line.text)` when the line has no usable
+/// spans (e.g. OCR-only lines where TextItem styling isn't populated).
+pub(super) fn render_line_inline(line: &ProjectedLine) -> String {
+ let spans: Vec<&TextItem> = line
+ .spans
+ .iter()
+ .filter(|s| !s.text.trim().is_empty())
+ .collect();
+ if spans.is_empty() {
+ return collapse_whitespace(&line.text);
+ }
+
+ // Sort spans by x so we render in visual reading order regardless of
+ // extraction order. Stable so equal-x spans keep their original sequence.
+ let mut spans = spans;
+ spans.sort_by(|a, b| a.x.total_cmp(&b.x));
+
+ let styles: Vec<SpanStyle> = spans.iter().map(|s| SpanStyle::from_item(s)).collect();
+
+ // Per-line shortcut.
+ let uniform = styles.iter().all(|s| *s == styles[0]);
+ if uniform {
+ let joined = collapse_whitespace(&line.text);
+ if joined.is_empty() {
+ return joined;
+ }
+ let escaped = escape_inline(&joined);
+ if styles[0].is_plain() {
+ return escaped;
+ }
+ return apply_style(&escaped, styles[0]);
+ }
+
+ // Group consecutive spans by style. Within a group, span texts join with
+ // a single space (we lose intra-group spacing precision; acceptable).
+ let mut out = String::new();
+ let mut i = 0;
+ while i < spans.len() {
+ let style = styles[i];
+ let mut j = i + 1;
+ while j < spans.len() && styles[j] == style {
+ j += 1;
+ }
+ let mut group_text = String::new();
+ for k in i..j {
+ if !group_text.is_empty() && !group_text.ends_with(' ') {
+ group_text.push(' ');
+ }
+ group_text.push_str(spans[k].text.trim());
+ }
+ let group_text = collapse_whitespace(&group_text);
+ let escaped = escape_inline(&group_text);
+ let rendered = if style.is_plain() {
+ escaped
+ } else {
+ apply_style(&escaped, style)
+ };
+ if !out.is_empty() && !out.ends_with(' ') {
+ out.push(' ');
+ }
+ out.push_str(&rendered);
+ i = j;
+ }
+ out
+}
+
+/// Render the text portion of a list item with per-span emphasis. The marker
+/// itself isn't included in the output (the renderer handles it separately).
+///
+/// When the line is uniformly styled we wrap the marker-stripped `rest` with
+/// the line's style — this avoids the awkward emphasis-marker mismatch we'd
+/// hit if we naively stripped a leading bullet out of an already-wrapped
+/// rendered line (`**• item**` → `** item**`).
+///
+/// When the line is mixed-style we render the full line via the inline pipeline
+/// and then best-effort-strip the marker prefix (with optional emphasis wrap
+/// around it). On any failure we fall back to plain escaped `rest`.
+pub(super) fn render_list_item_text(line: &ProjectedLine, marker: &str, rest: &str) -> String {
+ if let Some(style) = line_uniform_style(line) {
+ let plain = collapse_whitespace(rest);
+ let escaped = escape_inline(&plain);
+ return if style.is_plain() {
+ escaped
+ } else {
+ apply_style(&escaped, style)
+ };
+ }
+ let full = render_line_inline(line);
+ if let Some(stripped) = strip_leading_marker_from_inline(&full, marker) {
+ return stripped;
+ }
+ escape_inline(&collapse_whitespace(rest))
+}
+
+/// Try to strip a leading list marker (optionally wrapped in emphasis markers)
+/// off `s`. Recognizes `***MARK*** `, `**MARK** `, `*MARK* `, `` `MARK` ``,
+/// and bare `MARK `. Returns the suffix on a match.
+pub(super) fn strip_leading_marker_from_inline(s: &str, marker: &str) -> Option<String> {
+ for wrap in ["***", "**", "*", "`"] {
+ let prefix = format!("{wrap}{marker}{wrap} ");
+ if let Some(rest) = s.strip_prefix(&prefix) {
+ return Some(rest.to_string());
+ }
+ }
+ let prefix = format!("{marker} ");
+ s.strip_prefix(&prefix).map(|r| r.to_string())
+}
+
+/// Append an inline-rendered continuation line to an existing list-item body.
+/// De-hyphenates against the raw text boundary (mirrors the paragraph rule)
+/// and falls back to a space join otherwise.
+pub(super) fn append_inline_continuation(prev_text: &mut String, next_raw: &str, next_inline: &str) {
+ let next_raw = collapse_whitespace(next_raw);
+ if next_raw.is_empty() {
+ return;
+ }
+ if prev_text.is_empty() {
+ prev_text.push_str(next_inline);
+ return;
+ }
+ let dehyphenate = prev_text.ends_with('-')
+ && next_raw
+ .chars()
+ .next()
+ .is_some_and(|c| c.is_ascii_lowercase());
+ if dehyphenate {
+ prev_text.pop();
+ prev_text.push_str(next_inline);
+ } else {
+ prev_text.push(' ');
+ prev_text.push_str(next_inline);
+ }
+}
+
+/// Returns the shared `SpanStyle` of `line` when every non-whitespace span on
+/// the line has the same style; `None` when spans disagree. Mono is folded
+/// into the style for the purpose of "uniform" — a fully-mono line is
+/// uniform-mono. Used by the paragraph-level optimization to decide whether
+/// to wrap once around the whole paragraph or fall back to per-line inline.
+pub(super) fn line_uniform_style(line: &ProjectedLine) -> Option<SpanStyle> {
+ let mut iter = line
+ .spans
+ .iter()
+ .filter(|s| !s.text.trim().is_empty())
+ .map(SpanStyle::from_item);
+ let first = iter.next()?;
+ for s in iter {
+ if s != first {
+ return None;
+ }
+ }
+ Some(first)
+}
+
+#[cfg(test)]
+mod tests {
+ use super::super::test_helpers::styled_line;
+ use super::*;
+
+ #[test]
+ fn render_line_inline_mid_line_bold() {
+ // Plain span, then bold span: should produce a mid-line `**bold**` run.
+ let l = styled_line(
+ &[
+ ("regular text with", 50.0, Some("Arial")),
+ ("bold word", 200.0, Some("Arial-Bold")),
+ ],
+ 100.0,
+ 10.0,
+ );
+ let out = render_line_inline(&l);
+ assert!(out.contains("regular text with"), "got: {out}");
+ assert!(out.contains("**bold word**"), "got: {out}");
+ assert!(
+ !out.starts_with("**"),
+ "mid-line shouldn't open with bold: {out}"
+ );
+ }
+
+ #[test]
+ fn render_line_inline_uniform_bold_uses_shortcut() {
+ // All spans bold → single outer wrap, no per-span noise.
+ let l = styled_line(
+ &[
+ ("first", 50.0, Some("Arial-Bold")),
+ ("second", 100.0, Some("Arial-Bold")),
+ ],
+ 100.0,
+ 10.0,
+ );
+ let out = render_line_inline(&l);
+ assert!(out.starts_with("**") && out.ends_with("**"), "got: {out}");
+ // Only one bold run, not two — the shortcut should kick in.
+ assert_eq!(out.matches("**").count(), 2, "got: {out}");
+ }
+
+ #[test]
+ fn render_line_inline_escapes_emphasis_chars() {
+ let l = styled_line(&[("5*4=20", 50.0, Some("Arial"))], 100.0, 10.0);
+ let out = render_line_inline(&l);
+ assert_eq!(out, "5\\*4=20");
+ }
+
+ #[test]
+ fn render_line_inline_italic_then_bold() {
+ let l = styled_line(
+ &[
+ ("italic", 50.0, Some("Arial-Italic")),
+ ("plain", 100.0, Some("Arial")),
+ ("bold", 150.0, Some("Arial-Bold")),
+ ],
+ 100.0,
+ 10.0,
+ );
+ let out = render_line_inline(&l);
+ assert!(out.contains("*italic*"), "got: {out}");
+ assert!(out.contains("plain"), "got: {out}");
+ assert!(out.contains("**bold**"), "got: {out}");
+ }
+
+ #[test]
+ fn render_line_inline_mono_span() {
+ let l = styled_line(
+ &[
+ ("call", 50.0, Some("Arial")),
+ ("foo()", 100.0, Some("Courier")),
+ ("on it", 150.0, Some("Arial")),
+ ],
+ 100.0,
+ 10.0,
+ );
+ let out = render_line_inline(&l);
+ assert!(out.contains("`foo()`"), "got: {out}");
+ // Plain spans stay unwrapped.
+ assert!(out.contains("call"));
+ assert!(out.contains("on it"));
+ }
+}
diff --git a/crates/liteparse/src/markdown_layout/lists.rs b/crates/liteparse/src/markdown_layout/lists.rs
new file mode 100644
index 0000000..aaaa22d
--- /dev/null
+++ b/crates/liteparse/src/markdown_layout/lists.rs
@@ -0,0 +1,96 @@
+/// Roughly one indent step in PDF points. Used to bucket list items into
+/// nesting levels relative to the first item of the list.
+pub(super) const LIST_INDENT_STEP_PT: f32 = 12.0;
+
+/// Characters recognized as bullet markers when followed by whitespace.
+/// Limited to glyphs that are unlikely to appear at line-start in normal prose.
+const BULLET_CHARS: &[char] = &['•', '·', '◦', '▪', '▸', '▶', '●', '○', '■', '□'];
+
+/// Detect a list marker at the start of `text`. Returns `(ordered, marker_str,
+/// remainder)` when matched; otherwise `None`.
+///
+/// Recognizes:
+/// - Unicode bullet characters (`BULLET_CHARS`) followed by whitespace.
+/// - Decimal-prefix markers like `1.` / `1)` / `12.` / `12)` followed by
+/// whitespace — kept strict (digits only) so things like footnote callers
+/// (`1` alone) and section refs (`A.1`) don't match.
+pub(super) fn parse_list_marker(text: &str) -> Option<(bool, String, &str)> {
+ let trimmed = text.trim_start();
+ if trimmed.is_empty() {
+ return None;
+ }
+
+ let mut chars = trimmed.chars();
+ let first = chars.next()?;
+
+ // Unicode bullet
+ if BULLET_CHARS.contains(&first) {
+ let rest = chars.as_str();
+ if let Some(rest_trim) = rest.strip_prefix(|c: char| c.is_whitespace()) {
+ return Some((false, first.to_string(), rest_trim.trim_start()));
+ }
+ }
+
+ // Decimal: 1. / 1) / 12. / 12)
+ if first.is_ascii_digit() {
+ let mut digit_end = 1;
+ for c in trimmed[1..].chars() {
+ if c.is_ascii_digit() {
+ digit_end += c.len_utf8();
+ } else {
+ break;
+ }
+ }
+ // Cap to keep us from matching page-number-like prefixes
+ if digit_end <= 3 {
+ let after_digits = &trimmed[digit_end..];
+ let mut after_iter = after_digits.chars();
+ if let Some(punct) = after_iter.next()
+ && (punct == '.' || punct == ')')
+ {
+ let after_punct = after_iter.as_str();
+ if let Some(rest_trim) = after_punct.strip_prefix(|c: char| c.is_whitespace()) {
+ let marker = format!("{}{}", &trimmed[..digit_end], punct);
+ return Some((true, marker, rest_trim.trim_start()));
+ }
+ }
+ }
+ }
+
+ None
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn parse_list_marker_bullets() {
+ let (ordered, marker, rest) = parse_list_marker("• item one").unwrap();
+ assert!(!ordered);
+ assert_eq!(marker, "•");
+ assert_eq!(rest, "item one");
+ }
+
+ #[test]
+ fn parse_list_marker_decimal() {
+ let (ordered, marker, rest) = parse_list_marker("1. first").unwrap();
+ assert!(ordered);
+ assert_eq!(marker, "1.");
+ assert_eq!(rest, "first");
+
+ let (ordered, marker, rest) = parse_list_marker("12) twelfth").unwrap();
+ assert!(ordered);
+ assert_eq!(marker, "12)");
+ assert_eq!(rest, "twelfth");
+ }
+
+ #[test]
+ fn parse_list_marker_rejects_prose() {
+ assert!(parse_list_marker("This sentence.").is_none());
+ // Bare digit with no terminator → not a list
+ assert!(parse_list_marker("2023 was a year").is_none());
+ // Footnote caller / page number style — no whitespace after
+ assert!(parse_list_marker("1.5x growth").is_none());
+ }
+}
diff --git a/crates/liteparse/src/markdown_layout/mod.rs b/crates/liteparse/src/markdown_layout/mod.rs
new file mode 100644
index 0000000..9ccf865
--- /dev/null
+++ b/crates/liteparse/src/markdown_layout/mod.rs
@@ -0,0 +1,24 @@
+//! Block classification for the markdown emitter.
+//!
+//! Consumes `ProjectedLine` entries from each `ParsedPage` and groups them into
+//! a sequence of `Block`s: headings, paragraphs, and (for now) raw lines that
+//! don't fit a recognized shape. Tables, lists, and code blocks land in later
+//! build-order steps.
+
+mod blocks;
+mod classify;
+mod headings;
+mod hr;
+mod inline;
+mod lists;
+mod paragraphs;
+mod repetition;
+mod tables;
+
+pub use blocks::{Block, render_blocks};
+pub use classify::classify_page_with_filters;
+pub use headings::{build_heading_map, compute_body_size};
+pub use repetition::compute_header_footer_set;
+
+#[cfg(test)]
+pub(crate) mod test_helpers;
diff --git a/crates/liteparse/src/markdown_layout/paragraphs.rs b/crates/liteparse/src/markdown_layout/paragraphs.rs
new file mode 100644
index 0000000..ba866ec
--- /dev/null
+++ b/crates/liteparse/src/markdown_layout/paragraphs.rs
@@ -0,0 +1,214 @@
+use crate::types::{Anchor, ProjectedLine};
+
+use super::inline::{SpanStyle, line_uniform_style, render_line_inline};
+
+/// Multiplier on line height used as the paragraph-break threshold.
+pub(super) const PARAGRAPH_GAP_MULTIPLIER: f32 = 1.5;
+
+/// Tolerance for treating two font sizes as "the same" when grouping
+/// paragraph lines. Generous because we sometimes derive the "size" from
+/// `bbox.height`, which varies a few points line-to-line based on whether
+/// the glyphs include descenders (`y`, `g`, `p`).
+pub(super) const FONT_SIZE_PARAGRAPH_TOLERANCE: f32 = 2.5;
+
+/// Tolerance in points for treating two indent positions as "the same column".
+pub(super) const INDENT_TOLERANCE: f32 = 6.0;
+
+/// Collapse runs of whitespace into single spaces. The projected text from
+/// `projection.rs` pads with column-alignment spaces (e.g. `for instance`)
+/// which look fine as a layout grid but are wrong for prose.
+pub(super) fn collapse_whitespace(s: &str) -> String {
+ let mut out = String::with_capacity(s.len());
+ let mut prev_space = false;
+ for c in s.chars() {
+ if c.is_whitespace() {
+ if !prev_space && !out.is_empty() {
+ out.push(' ');
+ }
+ prev_space = true;
+ } else {
+ out.push(c);
+ prev_space = false;
+ }
+ }
+ if out.ends_with(' ') {
+ out.pop();
+ }
+ out
+}
+
+/// Append `next` to `accum` as the continuation of a paragraph. De-hyphenates
+/// when `accum` ends with `-` and `next` starts with an ASCII lowercase letter;
+/// otherwise joins with a single space.
+#[cfg(test)]
+pub(super) fn append_paragraph_line(accum: &mut String, next: &str) {
+ let next = collapse_whitespace(next);
+ if next.is_empty() {
+ return;
+ }
+ if accum.is_empty() {
+ accum.push_str(&next);
+ return;
+ }
+ let dehyphenate =
+ accum.ends_with('-') && next.chars().next().is_some_and(|c| c.is_ascii_lowercase());
+ if dehyphenate {
+ accum.pop(); // drop the '-'
+ accum.push_str(&next);
+ } else {
+ accum.push(' ');
+ accum.push_str(&next);
+ }
+}
+
+/// Decide whether `cur` continues the paragraph started by `prev`.
+pub(super) fn continues_paragraph(prev: &ProjectedLine, cur: &ProjectedLine) -> bool {
+ // Anchor only signals a paragraph break when one of the lines is clearly
+ // centered while the other isn't — justified prose routinely alternates
+ // between Left / Right / Floating dominant anchors as line widths flex,
+ // and treating those as paragraph breaks shreds normal text.
+ let centered_mismatch = (prev.anchor == Anchor::Center) ^ (cur.anchor == Anchor::Center);
+ if centered_mismatch {
+ return false;
+ }
+ if (prev.dominant_font_size - cur.dominant_font_size).abs() > FONT_SIZE_PARAGRAPH_TOLERANCE {
+ return false;
+ }
+ // Uniform-bold ↔ non-bold transition is a paragraph break. Catches
+ // body-size headings that share font size with the surrounding prose but
+ // are emitted in a bold variant (e.g. Brill-Bold over Brill-Roman). Both
+ // sides must have a uniform style for this to fire; mid-line emphasis
+ // (None style) falls through to the gap/indent checks so prose with
+ // mid-paragraph bold spans doesn't get fragmented.
+ if let (Some(p), Some(c)) = (line_uniform_style(prev), line_uniform_style(cur))
+ && p.bold != c.bold
+ {
+ return false;
+ }
+ if prev.region_path != cur.region_path {
+ // Cross-region continuation: the same paragraph can wrap from the
+ // bottom of one column into the top of the next. Only bridge regions
+ // when the previous line clearly breaks mid-sentence (no terminal
+ // punctuation) AND the next line starts with a lowercase letter — a
+ // strict signal that catches the column-wrap case while rejecting
+ // unrelated paragraphs that happen to sit in adjacent leaves.
+ let prev_trim = prev.text.trim_end();
+ let ends_open = !prev_trim.ends_with(|c: char| {
+ matches!(
+ c,
+ '.' | '!' | '?' | ':' | ';' | '”' | '"' | ')' | ']' | '。' | '』' | '」'
+ )
+ });
+ let starts_lower = cur
+ .text
+ .trim_start()
+ .chars()
+ .next()
+ .is_some_and(|c| c.is_lowercase());
+ return ends_open && starts_lower;
+ }
+ if (prev.indent_x - cur.indent_x).abs() > INDENT_TOLERANCE && cur.anchor == Anchor::Left {
+ // Indent change on a left-aligned block usually means a new paragraph
+ // (block-quote, list, indented passage, etc.). Allow first-line indent
+ // by checking only when the *next* line shifts right relative to prev.
+ if cur.indent_x > prev.indent_x + INDENT_TOLERANCE {
+ return false;
+ }
+ }
+ // Vertical gap check.
+ let prev_bottom = prev.bbox.y + prev.bbox.height;
+ let gap = cur.bbox.y - prev_bottom;
+ let line_height = prev.bbox.height.max(cur.bbox.height).max(1.0);
+ gap <= line_height * PARAGRAPH_GAP_MULTIPLIER
+}
+
+/// Paragraph accumulator state. We track two parallel representations of the
+/// running paragraph text:
+///
+/// - `raw` — plain text (no emphasis markers). Used for the paragraph-uniform
+/// shortcut: if every contributing line had the same uniform style, we wrap
+/// the whole paragraph once with `wrap_emphasis(raw, …)` to avoid the
+/// `**foo** **bar** **baz**` per-line noise pymupdf4llm warns about.
+/// - `inline` — per-line markdown with emphasis baked in via
+/// `render_line_inline`. Used when the paragraph contains mid-line emphasis
+/// shifts or lines with differing uniform styles.
+///
+/// `uniform` is `Some((bold, italic))` while every line so far has been a
+/// uniformly-styled line sharing the same (bold, italic) flags, and `None` as
+/// soon as that invariant breaks.
+pub(super) struct ParaAccum {
+ pub(super) raw: String,
+ pub(super) inline: String,
+ pub(super) last: ProjectedLine,
+ pub(super) uniform: Option<(bool, bool)>,
+}
+
+/// Append `next_line` to a paragraph accumulator. Maintains both the `raw` and
+/// `inline` text representations and updates the running `uniform` flag.
+/// De-hyphenation runs on the `raw` boundary; the `inline` boundary mirrors it
+/// when the trailing char is still a literal `-` (i.e. the hyphen sits outside
+/// any emphasis wrap — the common case).
+pub(super) fn append_to_paragraph(accum: &mut ParaAccum, next_line: &ProjectedLine) {
+ let next_raw = collapse_whitespace(next_line.text.trim());
+ if next_raw.is_empty() {
+ return;
+ }
+ let next_inline = render_line_inline(next_line);
+ let next_uniform: Option<SpanStyle> = line_uniform_style(next_line);
+
+ if accum.raw.is_empty() {
+ accum.raw.push_str(&next_raw);
+ accum.inline.push_str(&next_inline);
+ accum.uniform = next_uniform.map(|s| (s.bold, s.italic));
+ accum.last = next_line.clone();
+ return;
+ }
+
+ let dehyphenate = accum.raw.ends_with('-')
+ && next_raw
+ .chars()
+ .next()
+ .is_some_and(|c| c.is_ascii_lowercase());
+ if dehyphenate {
+ accum.raw.pop();
+ accum.raw.push_str(&next_raw);
+ if accum.inline.ends_with('-') {
+ accum.inline.pop();
+ accum.inline.push_str(&next_inline);
+ } else {
+ // Hyphen sits inside an emphasis wrap — give up on stripping it
+ // from `inline` cleanly. Join with a space; raw is still correct
+ // for the uniform-paragraph case.
+ accum.inline.push(' ');
+ accum.inline.push_str(&next_inline);
+ }
+ } else {
+ accum.raw.push(' ');
+ accum.raw.push_str(&next_raw);
+ accum.inline.push(' ');
+ accum.inline.push_str(&next_inline);
+ }
+
+ // Maintain the running uniform-style flag.
+ accum.uniform = match (accum.uniform, next_uniform) {
+ (Some(cur), Some(s)) if cur == (s.bold, s.italic) => Some(cur),
+ _ => None,
+ };
+ accum.last = next_line.clone();
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn dehyphenation_only_for_lowercase_followups() {
+ let mut s = String::from("co-");
+ append_paragraph_line(&mut s, "operate");
+ assert_eq!(s, "cooperate");
+
+ let mut s = String::from("Vitamin-");
+ append_paragraph_line(&mut s, "A");
+ assert_eq!(s, "Vitamin- A");
+ }
+}
diff --git a/crates/liteparse/src/markdown_layout/repetition.rs b/crates/liteparse/src/markdown_layout/repetition.rs
new file mode 100644
index 0000000..1b8a678
--- /dev/null
+++ b/crates/liteparse/src/markdown_layout/repetition.rs
@@ -0,0 +1,170 @@
+use crate::types::{ParsedPage, ProjectedLine};
+
+use super::paragraphs::collapse_whitespace;
+
+/// Fraction of page height treated as the "top band" for header detection.
+/// Most running headers sit within the top 8–12% of a page; 12% gives some
+/// slack for two-line headers without sweeping in body text.
+pub(super) const HEADER_BAND_FRACTION: f32 = 0.12;
+
+/// Fraction of page height treated as the "bottom band" for footer detection.
+pub(super) const FOOTER_BAND_FRACTION: f32 = 0.12;
+
+/// Fraction of pages on which a normalized line must appear (in the same
+/// band) to be classified as a running header/footer.
+const HEADER_FOOTER_MIN_FRACTION: f32 = 0.5;
+
+/// Absolute floor on header/footer matches — single-page docs can't have a
+/// "running" header by definition, and a single match on a 2-page doc is too
+/// weak to act on.
+const HEADER_FOOTER_MIN_PAGES: usize = 2;
+
+/// Normalize a line for cross-page header/footer matching. Lowercases,
+/// collapses whitespace, and replaces every run of ASCII digits with `#` so
+/// `Page 1 of 6` and `Page 2 of 6` collapse to the same key.
+pub(super) fn normalize_for_repetition(s: &str) -> String {
+ let collapsed = collapse_whitespace(s).to_lowercase();
+ let mut out = String::with_capacity(collapsed.len());
+ let mut in_digits = false;
+ for c in collapsed.chars() {
+ if c.is_ascii_digit() {
+ if !in_digits {
+ out.push('#');
+ in_digits = true;
+ }
+ } else {
+ out.push(c);
+ in_digits = false;
+ }
+ }
+ out
+}
+
+/// Cross-page repetition detector. Returns the set of normalized strings that
+/// appear in the top or bottom band of ≥ `HEADER_FOOTER_MIN_FRACTION` of
+/// pages (capped below by `HEADER_FOOTER_MIN_PAGES`). The caller uses this to
+/// filter `ProjectedLine`s before classification.
+///
+/// "Same band" means a line whose top is within `HEADER_BAND_FRACTION` of the
+/// page top (header) or whose bottom is within `FOOTER_BAND_FRACTION` of the
+/// page bottom (footer). Header and footer bands are tracked separately so a
+/// company name that appears as both a header and a body-section title on
+/// different pages isn't stripped from the body.
+pub fn compute_header_footer_set(pages: &[ParsedPage]) -> std::collections::HashSet<String> {
+ use std::collections::{HashMap, HashSet};
+ let mut set: HashSet<String> = HashSet::new();
+ if pages.len() < HEADER_FOOTER_MIN_PAGES {
+ return set;
+ }
+ // Two counters keyed by `(band, normalized_text)` — band is `'h'` or `'f'`.
+ let mut counts: HashMap<(char, String), HashSet<usize>> = HashMap::new();
+ for page in pages {
+ let header_cutoff = page.page_height * HEADER_BAND_FRACTION;
+ let footer_cutoff = page.page_height * (1.0 - FOOTER_BAND_FRACTION);
+ for line in &page.projected_lines {
+ let text = line.text.trim();
+ if text.is_empty() {
+ continue;
+ }
+ let norm = normalize_for_repetition(text);
+ if norm.is_empty() {
+ continue;
+ }
+ // Header band: top of line within the top band.
+ if line.bbox.y <= header_cutoff {
+ counts
+ .entry(('h', norm.clone()))
+ .or_default()
+ .insert(page.page_number);
+ }
+ // Footer band: bottom of line at or below the footer cutoff.
+ let line_bottom = line.bbox.y + line.bbox.height;
+ if line_bottom >= footer_cutoff {
+ counts
+ .entry(('f', norm))
+ .or_default()
+ .insert(page.page_number);
+ }
+ }
+ }
+ let threshold = (pages.len() as f32 * HEADER_FOOTER_MIN_FRACTION)
+ .ceil()
+ .max(HEADER_FOOTER_MIN_PAGES as f32) as usize;
+ for ((_, norm), pages_seen) in counts {
+ if pages_seen.len() >= threshold {
+ set.insert(norm);
+ }
+ }
+ set
+}
+
+/// Returns true if `line` (located on `page`) matches the running
+/// header/footer set: the line sits in the top or bottom band AND its
+/// normalized text is in `header_footer`.
+pub(super) fn is_header_or_footer(
+ line: &ProjectedLine,
+ page: &ParsedPage,
+ header_footer: &std::collections::HashSet<String>,
+) -> bool {
+ if header_footer.is_empty() {
+ return false;
+ }
+ let header_cutoff = page.page_height * HEADER_BAND_FRACTION;
+ let footer_cutoff = page.page_height * (1.0 - FOOTER_BAND_FRACTION);
+ let in_band = line.bbox.y <= header_cutoff || line.bbox.y + line.bbox.height >= footer_cutoff;
+ if !in_band {
+ return false;
+ }
+ let norm = normalize_for_repetition(line.text.trim());
+ header_footer.contains(&norm)
+}
+
+#[cfg(test)]
+mod tests {
+ use super::super::test_helpers::header_footer_page;
+ use super::*;
+
+ #[test]
+ fn normalize_collapses_digits_and_case() {
+ assert_eq!(normalize_for_repetition("Page 1 of 6"), "page # of #");
+ assert_eq!(normalize_for_repetition("PAGE 12 OF 6"), "page # of #");
+ assert_eq!(normalize_for_repetition("Confidential"), "confidential");
+ }
+
+ #[test]
+ fn detects_repeating_header_and_footer() {
+ let pages = vec![
+ header_footer_page(1, "Acme Confidential", "Page 1 of 3", "Body one."),
+ header_footer_page(2, "Acme Confidential", "Page 2 of 3", "Body two."),
+ header_footer_page(3, "Acme Confidential", "Page 3 of 3", "Body three."),
+ ];
+ let set = compute_header_footer_set(&pages);
+ assert!(set.contains("acme confidential"));
+ assert!(set.contains("page # of #"));
+ }
+
+ #[test]
+ fn skips_repetition_check_on_single_page() {
+ let pages = vec![header_footer_page(1, "Solo", "footer", "body")];
+ let set = compute_header_footer_set(&pages);
+ assert!(set.is_empty());
+ }
+
+ #[test]
+ fn body_text_not_classified_as_header() {
+ // Same text in the body of every page should NOT be stripped — only
+ // text within the top/bottom band qualifies.
+ let mut pages = Vec::new();
+ for n in 1..=3 {
+ let mut p = header_footer_page(n, "unique header", "unique footer", "shared body text");
+ // Move "shared body text" out of header/footer bands (already at y=50 in mid-page).
+ // No-op — just illustrating intent.
+ p.projected_lines[0].text = format!("unique header {n}");
+ p.projected_lines[2].text = format!("unique footer {n}");
+ pages.push(p);
+ }
+ let set = compute_header_footer_set(&pages);
+ // "shared body text" sits mid-page — never matched.
+ assert!(!set.contains("shared body text"));
+ }
+}
diff --git a/crates/liteparse/src/markdown_layout/tables.rs b/crates/liteparse/src/markdown_layout/tables.rs
new file mode 100644
index 0000000..102270b
--- /dev/null
+++ b/crates/liteparse/src/markdown_layout/tables.rs
@@ -0,0 +1,1118 @@
+use crate::types::{GraphicPrimitive, ProjectedLine, TextItem};
+
+use super::blocks::Block;
+use super::inline::is_bold_span;
+use super::paragraphs::collapse_whitespace;
+
+/// Minimum cells per row for a region to qualify as a table.
+pub(super) const TABLE_MIN_COLUMNS: usize = 3;
+
+/// Minimum consecutive rows for a region to qualify as a table.
+const TABLE_MIN_ROWS: usize = 2;
+
+/// Gap between adjacent spans (in multiples of dominant font size) above which
+/// we treat the gap as a cell boundary.
+const TABLE_CELL_GAP_FONT_MULTIPLIER: f32 = 1.0;
+
+/// Tolerance (points) for matching a cell's start-x to an existing column
+/// track when extending a candidate table run.
+const TABLE_TRACK_TOLERANCE_PT: f32 = 6.0;
+
+/// Maximum vertical gap between consecutive table rows, expressed in multiples
+/// of the line height. Looser than the paragraph rule because table rows often
+/// have more vertical padding than prose lines.
+const TABLE_ROW_GAP_MULTIPLIER: f32 = 2.5;
+
+/// Maximum coefficient-of-variation for row spacing within a confident table
+/// (rejecting irregular spacing that's more likely prose or a footer block).
+const TABLE_ROW_SPACING_MAX_CV: f32 = 0.5;
+
+/// One cell within a tabular row: contributing spans aggregated to text and
+/// its leftmost x position, used to align cells across rows into column
+/// "tracks".
+#[derive(Debug, Clone)]
+pub(super) struct TableCell {
+ pub(super) start_x: f32,
+ /// Right edge of the cell (x of the last span's right). Used by
+ /// `recover_merged_cell` to detect cells that straddle two column tracks
+ /// when the projection merged two adjacent words into one span.
+ pub(super) end_x: f32,
+ pub(super) text: String,
+ pub(super) bold: bool,
+}
+
+/// A contiguous tabular run: line indices `[start, end)` plus the detected
+/// rows. Used so the line-classifier can skip the consumed range and so
+/// fallback rendering can reach back for the original projected text.
+#[derive(Debug, Clone)]
+pub(super) struct TableRun {
+ pub(super) start: usize,
+ pub(super) end: usize,
+ pub(super) block: Block,
+}
+
+/// Split a `ProjectedLine`'s spans into cells. A gap larger than
+/// `TABLE_CELL_GAP_FONT_MULTIPLIER × font_size` between adjacent spans starts
+/// a new cell; otherwise spans join into the same cell with a single space.
+pub(super) fn split_cells(line: &ProjectedLine) -> Vec<TableCell> {
+ // Skip whitespace-only spans before computing gaps — leading/trailing
+ // empty items would otherwise add spurious cell boundaries.
+ let mut spans: Vec<&TextItem> = line
+ .spans
+ .iter()
+ .filter(|s| !s.text.trim().is_empty())
+ .collect();
+ spans.sort_by(|a, b| a.x.total_cmp(&b.x));
+ if spans.is_empty() {
+ return Vec::new();
+ }
+ let font_size = if line.dominant_font_size > 0.0 {
+ line.dominant_font_size
+ } else {
+ line.bbox.height.max(1.0)
+ };
+ let gap_threshold = font_size * TABLE_CELL_GAP_FONT_MULTIPLIER;
+
+ let mut cells: Vec<TableCell> = Vec::new();
+ let mut current_text = String::new();
+ let mut current_start = spans[0].x;
+ let mut current_bold_chars: usize = 0;
+ let mut current_total_chars: usize = 0;
+ let mut prev_right = spans[0].x;
+
+ for (i, span) in spans.iter().enumerate() {
+ let gap = span.x - prev_right;
+ let break_cell = i > 0 && gap > gap_threshold;
+ if break_cell {
+ let bold = current_total_chars > 0 && current_bold_chars * 2 > current_total_chars;
+ cells.push(TableCell {
+ start_x: current_start,
+ end_x: prev_right,
+ text: collapse_whitespace(current_text.trim()),
+ bold,
+ });
+ current_text.clear();
+ current_start = span.x;
+ current_bold_chars = 0;
+ current_total_chars = 0;
+ }
+ if !current_text.is_empty() && !current_text.ends_with(' ') {
+ current_text.push(' ');
+ }
+ current_text.push_str(&span.text);
+ let n = span.text.chars().count();
+ current_total_chars += n;
+ if is_bold_span(span) {
+ current_bold_chars += n;
+ }
+ prev_right = span.x + span.width.max(0.0);
+ }
+ if !current_text.trim().is_empty() {
+ let bold = current_total_chars > 0 && current_bold_chars * 2 > current_total_chars;
+ cells.push(TableCell {
+ start_x: current_start,
+ end_x: prev_right,
+ text: collapse_whitespace(current_text.trim()),
+ bold,
+ });
+ }
+ cells
+}
+
+/// When a candidate row has fewer cells than the established column count,
+/// look for cells whose x-range straddles multiple column tracks (likely two
+/// or more adjacent words that PDFium merged into a single text run) and
+/// split each on internal whitespace at the boundaries nearest to the
+/// straddled tracks.
+///
+/// Returns the patched cells if every short cell could be cleanly split to
+/// recover `tracks.len()` cells total; otherwise `None`.
+pub(super) fn recover_merged_cell(mut cells: Vec<TableCell>, tracks: &[f32]) -> Option<Vec<TableCell>> {
+ let target = tracks.len();
+ if cells.len() >= target {
+ return None;
+ }
+ // Repeatedly find the cell that straddles the most tracks (≥2) and split
+ // it. Each iteration strictly grows `cells.len()`, so termination is
+ // guaranteed; if no cell straddles ≥2 tracks before we hit the target,
+ // recovery fails.
+ while cells.len() < target {
+ let mut best_i: Option<usize> = None;
+ let mut best_count: usize = 1;
+ let mut best_contained: Vec<f32> = Vec::new();
+ for (i, cell) in cells.iter().enumerate() {
+ let contained: Vec<f32> = tracks
+ .iter()
+ .copied()
+ .filter(|t| {
+ *t >= cell.start_x - TABLE_TRACK_TOLERANCE_PT
+ && *t <= cell.end_x + TABLE_TRACK_TOLERANCE_PT
+ })
+ .collect();
+ if contained.len() > best_count {
+ best_count = contained.len();
+ best_i = Some(i);
+ best_contained = contained;
+ }
+ }
+ let Some(i) = best_i else {
+ return None;
+ };
+ let cell = cells[i].clone();
+ let chars: Vec<char> = cell.text.trim().chars().collect();
+ let n = chars.len();
+ if n == 0 || best_contained.len() < 2 {
+ return None;
+ }
+ let text_width = (cell.end_x - cell.start_x).max(1.0);
+ // For each track after the first, pick the whitespace boundary in
+ // `chars` whose linearly-interpolated x is closest to the track.
+ let mut split_indices: Vec<usize> = Vec::new();
+ for t in best_contained.iter().skip(1) {
+ let mut best: Option<(usize, f32)> = None;
+ for (k, ch) in chars.iter().enumerate() {
+ if !ch.is_whitespace() {
+ continue;
+ }
+ if split_indices.contains(&k) {
+ continue;
+ }
+ let frac = k as f32 / n as f32;
+ let x = cell.start_x + frac * text_width;
+ let d = (x - t).abs();
+ if best.as_ref().is_none_or(|b| d < b.1) {
+ best = Some((k, d));
+ }
+ }
+ let (k, _) = best?;
+ split_indices.push(k);
+ }
+ split_indices.sort();
+ // Build the split pieces.
+ let mut pieces: Vec<String> = Vec::new();
+ let mut prev = 0usize;
+ for k in &split_indices {
+ let piece: String = chars[prev..*k]
+ .iter()
+ .collect::<String>()
+ .trim()
+ .to_string();
+ if piece.is_empty() {
+ return None;
+ }
+ pieces.push(piece);
+ prev = *k;
+ }
+ let last: String = chars[prev..].iter().collect::<String>().trim().to_string();
+ if last.is_empty() {
+ return None;
+ }
+ pieces.push(last);
+ if pieces.len() != best_contained.len() {
+ return None;
+ }
+ // Synthesize new TableCells aligned with each track.
+ let mut new_cells: Vec<TableCell> = Vec::with_capacity(pieces.len());
+ for (p, piece) in pieces.iter().enumerate() {
+ let start_x = if p == 0 {
+ cell.start_x
+ } else {
+ best_contained[p]
+ };
+ let end_x = if p + 1 < best_contained.len() {
+ (best_contained[p + 1] - 1.0).max(start_x)
+ } else {
+ cell.end_x
+ };
+ new_cells.push(TableCell {
+ start_x,
+ end_x,
+ text: piece.clone(),
+ bold: cell.bold,
+ });
+ }
+ cells.remove(i);
+ for (offset, c) in new_cells.into_iter().enumerate() {
+ cells.insert(i + offset, c);
+ }
+ }
+ if cells.len() == target {
+ Some(cells)
+ } else {
+ None
+ }
+}
+
+/// Vertical-gap check for table rows. Looser than paragraph continuation
+/// because table rows often have extra padding between them.
+fn table_rows_adjacent(prev: &ProjectedLine, cur: &ProjectedLine) -> bool {
+ // Intentionally don't require region_path equality. Indented sub-group
+ // rows (e.g. an indented "MEMORYBANK" row in a grouped academic results
+ // table) sometimes land in a different XY-cut leaf than the rest of the
+ // table — but the column-track alignment and y-gap checks below are
+ // strong enough signals on their own to keep us from spuriously
+ // bridging unrelated regions.
+ let prev_bottom = prev.bbox.y + prev.bbox.height;
+ let gap = cur.bbox.y - prev_bottom;
+ let line_height = prev.bbox.height.max(cur.bbox.height).max(1.0);
+ gap >= -line_height && gap <= line_height * TABLE_ROW_GAP_MULTIPLIER
+}
+
+/// Coefficient of variation (std-dev / mean) of inter-row vertical gaps.
+/// Returns 0.0 for runs with <2 gaps (nothing to compare). Used to reject
+/// runs whose row spacing is too irregular to be a real table.
+fn row_spacing_cv(rows: &[(usize, &ProjectedLine, Vec<TableCell>)]) -> f32 {
+ if rows.len() < 3 {
+ return 0.0;
+ }
+ let gaps: Vec<f32> = rows
+ .windows(2)
+ .map(|w| (w[1].1.bbox.y - w[0].1.bbox.y).abs())
+ .collect();
+ let mean = gaps.iter().sum::<f32>() / gaps.len() as f32;
+ if mean <= 0.0 {
+ return f32::INFINITY;
+ }
+ let var = gaps.iter().map(|g| (g - mean).powi(2)).sum::<f32>() / gaps.len() as f32;
+ var.sqrt() / mean
+}
+
+/// Try to extend a candidate table starting at `start_idx`. On success returns
+/// a `TableRun` with `Block::Table` or `Block::GridFallback`; on failure
+/// returns `None` (and the caller should fall through to per-line
+/// classification).
+fn try_detect_table(lines: &[ProjectedLine], start_idx: usize) -> Option<TableRun> {
+ let first_cells = split_cells(&lines[start_idx]);
+ if first_cells.len() < TABLE_MIN_COLUMNS {
+ return None;
+ }
+
+ let mut rows: Vec<(usize, &ProjectedLine, Vec<TableCell>)> =
+ vec![(start_idx, &lines[start_idx], first_cells.clone())];
+ let column_count = first_cells.len();
+ let tracks: Vec<f32> = first_cells.iter().map(|c| c.start_x).collect();
+
+ let mut j = start_idx + 1;
+ while j < lines.len() {
+ if !table_rows_adjacent(rows.last().unwrap().1, &lines[j]) {
+ break;
+ }
+ let mut cells = split_cells(&lines[j]);
+ if cells.len() < column_count && cells.len() >= TABLE_MIN_COLUMNS {
+ // PDFium occasionally merges two (or more) adjacent words into one
+ // text run when inter-word kerning is tighter than the gap
+ // threshold — common in tightly-set numeric tables (e.g. the
+ // "MEMORYBANK 5.00 4.77" case on page 6 of the AMEM paper).
+ // Recover by splitting straddling cells on internal whitespace.
+ if let Some(patched) = recover_merged_cell(cells.clone(), &tracks) {
+ cells = patched;
+ }
+ }
+ // Wrapped-continuation merge: a partial-cell line that sits tight
+ // beneath (or overlaps) the previous row AND whose cells all align
+ // with existing column tracks is a *wrap* of the prior row, not a
+ // new row. Common in borderless tables where one column has a
+ // multi-line cell while neighbouring columns stay on one line. Merge
+ // each wrap cell into its matching track's text rather than
+ // breaking the run.
+ if cells.len() < column_count && !cells.is_empty() {
+ let line_height = lines[j].bbox.height.max(1.0);
+ let prev_y_top = rows.last().unwrap().1.bbox.y;
+ let centroid_dy = lines[j].bbox.y - prev_y_top;
+ let all_align_track = cells.iter().all(|c| {
+ tracks
+ .iter()
+ .any(|t| (c.start_x - *t).abs() <= TABLE_TRACK_TOLERANCE_PT)
+ });
+ if centroid_dy <= line_height * 1.5 && all_align_track {
+ let prev_cells = &mut rows.last_mut().unwrap().2;
+ for c in &cells {
+ if let Some(idx) = tracks
+ .iter()
+ .position(|t| (c.start_x - *t).abs() <= TABLE_TRACK_TOLERANCE_PT)
+ {
+ if !prev_cells[idx].text.is_empty() && !c.text.is_empty() {
+ prev_cells[idx].text.push(' ');
+ }
+ prev_cells[idx].text.push_str(&c.text);
+ }
+ }
+ j += 1;
+ continue;
+ }
+ }
+ if cells.len() != column_count {
+ break;
+ }
+ // Allow at most one column track to drift out of tolerance, which lets
+ // grouped row-labels in academic tables (e.g. an indented "MEMORYBANK"
+ // row whose label column shifts right by ~30pt while the numeric
+ // columns stay aligned) stay in the same run. Without this slack a
+ // single indented label fragments a 6-row table into three 2-row chunks.
+ let misaligned = cells
+ .iter()
+ .zip(tracks.iter())
+ .filter(|(c, t)| (c.start_x - **t).abs() > TABLE_TRACK_TOLERANCE_PT)
+ .count();
+ if misaligned > 1 {
+ break;
+ }
+ rows.push((j, &lines[j], cells));
+ j += 1;
+ }
+
+ if rows.len() < TABLE_MIN_ROWS {
+ return None;
+ }
+
+ let cv = row_spacing_cv(&rows);
+ let end = j;
+
+ if cv > TABLE_ROW_SPACING_MAX_CV {
+ // Suggestive layout but the row cadence is too irregular to trust as a
+ // clean table — surface as a fenced fallback so the structure is at
+ // least preserved.
+ let raw: Vec<String> = rows
+ .iter()
+ .map(|(_, line, _)| line.text.trim_end().to_string())
+ .collect();
+ return Some(TableRun {
+ start: start_idx,
+ end,
+ block: Block::GridFallback { lines: raw },
+ });
+ }
+
+ // Promote the first row to header iff every cell in it is bold (matches
+ // pymupdf4llm's "bold-or-filled" heuristic; fills require fork data).
+ let first_row = &rows[0].2;
+ let header_qualifies = first_row.iter().all(|c| c.bold);
+ let header = if header_qualifies {
+ Some(first_row.iter().map(|c| c.text.clone()).collect())
+ } else {
+ None
+ };
+ let row_start = if header.is_some() { 1 } else { 0 };
+ let body_rows: Vec<Vec<String>> = rows[row_start..]
+ .iter()
+ .map(|(_, _, cells)| cells.iter().map(|c| c.text.clone()).collect())
+ .collect();
+ if header.is_none() && body_rows.len() < TABLE_MIN_ROWS {
+ return None;
+ }
+
+ Some(TableRun {
+ start: start_idx,
+ end,
+ block: Block::Table {
+ header,
+ rows: body_rows,
+ },
+ })
+}
+
+/// Scan `lines` once and return all detected tabular regions (sorted by
+/// `start`). Caller uses these as cut-points so the per-line classifier never
+/// sees lines inside a table.
+pub(super) fn detect_tables(lines: &[ProjectedLine]) -> Vec<TableRun> {
+ let mut out = Vec::new();
+ let mut i = 0;
+ while i < lines.len() {
+ if let Some(run) = try_detect_table(lines, i) {
+ i = run.end;
+ out.push(run);
+ } else {
+ i += 1;
+ }
+ }
+ out
+}
+
+// ── Ruled-grid table detection ─────────────────────────────────────────────
+//
+// Detect tables drawn with explicit horizontal + vertical rules (the "Strong"
+// mode in MARKDOWN_PLAN.md). Strokes are clustered into H/V grid lines, then
+// union-find groups crossing lines into table regions. For each region the
+// distinct row/column boundaries form a cell grid; text lines are assigned to
+// cells by centroid containment.
+//
+// Ruled tables are detected before the borderless `detect_tables`. The caller
+// merges the two outputs; overlapping ranges defer to the ruled run because
+// path-based geometry is a strictly stronger signal than text alignment alone.
+
+/// Horizontal segment in viewport coords (top-left origin). `y` is the rule's
+/// y-position; `x_min..x_max` is its horizontal span. Endpoints of multiple
+/// short segments sharing a y get unioned into one wider segment during
+/// clustering.
+#[derive(Debug, Clone, Copy)]
+struct HSeg {
+ x_min: f32,
+ x_max: f32,
+ y: f32,
+}
+
+#[derive(Debug, Clone, Copy)]
+struct VSeg {
+ y_min: f32,
+ y_max: f32,
+ x: f32,
+}
+
+/// Strokes are considered "axis-aligned" when the perpendicular delta is at
+/// most this many points. Generous to absorb antialiased near-pixel strokes.
+const TABLE_AXIS_TOLERANCE_PT: f32 = 1.0;
+
+/// Two H lines (or two V lines) are merged into one grid line when their
+/// perpendicular coords are within this many points. Slightly looser than the
+/// axis tolerance because rules drawn at the same row can have ±1pt jitter
+/// from different stroke widths.
+const TABLE_GRID_CLUSTER_PT: f32 = 2.0;
+
+/// Slack added when checking whether a V line "crosses" an H line. Helps
+/// when rules don't quite reach the corner because the PDF drew them as
+/// individual segments with small gaps.
+const TABLE_CROSS_TOLERANCE_PT: f32 = 3.0;
+
+/// Reject ruled-table candidates whose empty-cell fraction exceeds this.
+const TABLE_MAX_EMPTY_CELL_FRACTION: f32 = 0.30;
+
+/// Reject candidates whose grid covers nearly the whole page — almost always
+/// a page border, not a real table.
+const TABLE_MAX_PAGE_COVERAGE: f32 = 0.95;
+
+/// Extract horizontal and vertical line segments from a page's graphics. Each
+/// `Stroke` becomes one HSeg or VSeg depending on orientation; each stroked
+/// `Rect` contributes its four edges (cell-border rects, table frames).
+fn extract_h_v_segments(graphics: &[GraphicPrimitive]) -> (Vec<HSeg>, Vec<VSeg>) {
+ let mut hs = Vec::new();
+ let mut vs = Vec::new();
+ for g in graphics {
+ match g {
+ GraphicPrimitive::Stroke { x1, y1, x2, y2, .. } => {
+ let (x1, y1, x2, y2) = (*x1, *y1, *x2, *y2);
+ let dy = (y1 - y2).abs();
+ let dx = (x1 - x2).abs();
+ if dy <= TABLE_AXIS_TOLERANCE_PT && dx > 1.0 {
+ hs.push(HSeg {
+ x_min: x1.min(x2),
+ x_max: x1.max(x2),
+ y: (y1 + y2) * 0.5,
+ });
+ } else if dx <= TABLE_AXIS_TOLERANCE_PT && dy > 1.0 {
+ vs.push(VSeg {
+ y_min: y1.min(y2),
+ y_max: y1.max(y2),
+ x: (x1 + x2) * 0.5,
+ });
+ }
+ }
+ GraphicPrimitive::Rect { bbox, stroke, .. } => {
+ if stroke.is_none() {
+ continue;
+ }
+ let top = bbox.y;
+ let bottom = bbox.y + bbox.height;
+ let left = bbox.x;
+ let right = bbox.x + bbox.width;
+ if bbox.width > 1.0 {
+ hs.push(HSeg {
+ x_min: left,
+ x_max: right,
+ y: top,
+ });
+ hs.push(HSeg {
+ x_min: left,
+ x_max: right,
+ y: bottom,
+ });
+ }
+ if bbox.height > 1.0 {
+ vs.push(VSeg {
+ y_min: top,
+ y_max: bottom,
+ x: left,
+ });
+ vs.push(VSeg {
+ y_min: top,
+ y_max: bottom,
+ x: right,
+ });
+ }
+ }
+ }
+ }
+ (hs, vs)
+}
+
+/// Cluster H segments sharing a y-coordinate (within `TABLE_GRID_CLUSTER_PT`)
+/// into a single wider grid line whose x-extent is the union of the inputs.
+fn cluster_h_segments(mut segs: Vec<HSeg>) -> Vec<HSeg> {
+ if segs.is_empty() {
+ return segs;
+ }
+ segs.sort_by(|a, b| a.y.total_cmp(&b.y));
+ let mut out: Vec<HSeg> = Vec::with_capacity(segs.len());
+ for seg in segs {
+ if let Some(last) = out.last_mut()
+ && (last.y - seg.y).abs() <= TABLE_GRID_CLUSTER_PT
+ {
+ last.x_min = last.x_min.min(seg.x_min);
+ last.x_max = last.x_max.max(seg.x_max);
+ continue;
+ }
+ out.push(seg);
+ }
+ out
+}
+
+fn cluster_v_segments(mut segs: Vec<VSeg>) -> Vec<VSeg> {
+ if segs.is_empty() {
+ return segs;
+ }
+ segs.sort_by(|a, b| a.x.total_cmp(&b.x));
+ let mut out: Vec<VSeg> = Vec::with_capacity(segs.len());
+ for seg in segs {
+ if let Some(last) = out.last_mut()
+ && (last.x - seg.x).abs() <= TABLE_GRID_CLUSTER_PT
+ {
+ last.y_min = last.y_min.min(seg.y_min);
+ last.y_max = last.y_max.max(seg.y_max);
+ continue;
+ }
+ out.push(seg);
+ }
+ out
+}
+
+/// Union-find root with path compression.
+fn uf_find(parent: &mut [usize], mut x: usize) -> usize {
+ while parent[x] != x {
+ parent[x] = parent[parent[x]];
+ x = parent[x];
+ }
+ x
+}
+
+fn uf_union(parent: &mut [usize], a: usize, b: usize) {
+ let ra = uf_find(parent, a);
+ let rb = uf_find(parent, b);
+ if ra != rb {
+ parent[ra] = rb;
+ }
+}
+
+/// Group H/V grid lines that cross each other into connected components.
+/// Each component is a candidate ruled table — typically one component per
+/// distinct table on the page. Returns `(h_indices, v_indices)` per component,
+/// dropping components without ≥2 H and ≥2 V lines (a single L-shape doesn't
+/// make a table).
+fn find_grid_components(hs: &[HSeg], vs: &[VSeg]) -> Vec<(Vec<usize>, Vec<usize>)> {
+ let n_h = hs.len();
+ let n_v = vs.len();
+ if n_h < 2 || n_v < 2 {
+ return Vec::new();
+ }
+ let n = n_h + n_v;
+ let mut parent: Vec<usize> = (0..n).collect();
+ let mut connected = vec![false; n];
+
+ let tol = TABLE_CROSS_TOLERANCE_PT;
+ for (i, h) in hs.iter().enumerate() {
+ for (j, v) in vs.iter().enumerate() {
+ let v_crosses_h_x = v.x >= h.x_min - tol && v.x <= h.x_max + tol;
+ let h_crosses_v_y = h.y >= v.y_min - tol && h.y <= v.y_max + tol;
+ if v_crosses_h_x && h_crosses_v_y {
+ uf_union(&mut parent, i, n_h + j);
+ connected[i] = true;
+ connected[n_h + j] = true;
+ }
+ }
+ }
+
+ use std::collections::HashMap;
+ let mut groups: HashMap<usize, (Vec<usize>, Vec<usize>)> = HashMap::new();
+ for i in 0..n_h {
+ if !connected[i] {
+ continue;
+ }
+ let r = uf_find(&mut parent, i);
+ groups.entry(r).or_default().0.push(i);
+ }
+ for j in 0..n_v {
+ if !connected[n_h + j] {
+ continue;
+ }
+ let r = uf_find(&mut parent, n_h + j);
+ groups.entry(r).or_default().1.push(j);
+ }
+ groups
+ .into_values()
+ .filter(|(h_idx, v_idx)| h_idx.len() >= 2 && v_idx.len() >= 2)
+ .collect()
+}
+
+/// Build a `TableRun` for one ruled-grid component. Returns `None` if the
+/// resulting grid is too small (< 2 cols or < 2 rows), covers nearly the
+/// whole page (likely the page border), or is mostly empty cells.
+fn build_ruled_table(
+ hs: &[HSeg],
+ vs: &[VSeg],
+ h_indices: &[usize],
+ v_indices: &[usize],
+ lines: &[ProjectedLine],
+ page_width: f32,
+ page_height: f32,
+) -> Option<TableRun> {
+ // Distinct row y-coords (cluster again — multiple H lines may share a y).
+ let mut ys: Vec<f32> = h_indices.iter().map(|&i| hs[i].y).collect();
+ ys.sort_by(|a, b| a.total_cmp(b));
+ dedup_close(&mut ys, TABLE_GRID_CLUSTER_PT);
+
+ let mut xs: Vec<f32> = v_indices.iter().map(|&i| vs[i].x).collect();
+ xs.sort_by(|a, b| a.total_cmp(b));
+ dedup_close(&mut xs, TABLE_GRID_CLUSTER_PT);
+
+ // Need ≥2 cells per axis → ≥3 boundary lines.
+ if ys.len() < 3 || xs.len() < 3 {
+ return None;
+ }
+
+ let n_rows = ys.len() - 1;
+ let n_cols = xs.len() - 1;
+ let bbox = crate::types::Rect {
+ x: xs[0],
+ y: ys[0],
+ width: xs[n_cols] - xs[0],
+ height: ys[n_rows] - ys[0],
+ };
+
+ // Reject page-border-as-table.
+ if page_width > 0.0 && page_height > 0.0 {
+ let coverage = (bbox.width / page_width) * (bbox.height / page_height);
+ if coverage > TABLE_MAX_PAGE_COVERAGE {
+ return None;
+ }
+ }
+
+ // Assign each text line to its cell by centroid.
+ let mut cells: Vec<Vec<String>> = vec![vec![String::new(); n_cols]; n_rows];
+ let mut cell_is_bold: Vec<Vec<bool>> = vec![vec![true; n_cols]; n_rows];
+ let mut cell_has_text: Vec<Vec<bool>> = vec![vec![false; n_cols]; n_rows];
+ let mut consumed_indices: Vec<usize> = Vec::new();
+
+ for (idx, line) in lines.iter().enumerate() {
+ let cx = line.bbox.x + line.bbox.width * 0.5;
+ let cy = line.bbox.y + line.bbox.height * 0.5;
+ if cy < ys[0] || cy > ys[n_rows] || cx < xs[0] || cx > xs[n_cols] {
+ continue;
+ }
+ let row = match find_bucket(&ys, cy) {
+ Some(r) => r,
+ None => continue,
+ };
+ let col = match find_bucket(&xs, cx) {
+ Some(c) => c,
+ None => continue,
+ };
+ let txt = line.text.trim();
+ if txt.is_empty() {
+ continue;
+ }
+ if !cells[row][col].is_empty() {
+ cells[row][col].push(' ');
+ }
+ cells[row][col].push_str(txt);
+ cell_has_text[row][col] = true;
+ if !line.all_bold {
+ cell_is_bold[row][col] = false;
+ }
+ consumed_indices.push(idx);
+ }
+
+ if consumed_indices.is_empty() {
+ return None;
+ }
+
+ let total = n_rows * n_cols;
+ let empty_count = cell_has_text
+ .iter()
+ .flatten()
+ .filter(|filled| !**filled)
+ .count();
+ if (empty_count as f32) / (total as f32) > TABLE_MAX_EMPTY_CELL_FRACTION {
+ return None;
+ }
+
+ // Header = first row iff every non-empty cell in it is bold.
+ let header_qualifies = cell_has_text[0]
+ .iter()
+ .zip(cell_is_bold[0].iter())
+ .all(|(has, bold)| !has || *bold)
+ && cell_has_text[0].iter().any(|has| *has);
+ let header = if header_qualifies {
+ Some(cells[0].clone())
+ } else {
+ None
+ };
+ let body_start = if header.is_some() { 1 } else { 0 };
+ let body_rows: Vec<Vec<String>> = cells[body_start..].to_vec();
+ if body_rows.is_empty() {
+ return None;
+ }
+
+ // Line index span this table covers.
+ let start = *consumed_indices.iter().min().unwrap();
+ let end = *consumed_indices.iter().max().unwrap() + 1;
+
+ Some(TableRun {
+ start,
+ end,
+ block: Block::Table {
+ header,
+ rows: body_rows,
+ },
+ })
+}
+
+/// In-place dedup of a sorted Vec, collapsing entries within `tol` to the
+/// first of each cluster.
+fn dedup_close(v: &mut Vec<f32>, tol: f32) {
+ if v.len() < 2 {
+ return;
+ }
+ let mut out: Vec<f32> = Vec::with_capacity(v.len());
+ for x in v.iter().copied() {
+ if let Some(&last) = out.last()
+ && (x - last).abs() <= tol
+ {
+ continue;
+ }
+ out.push(x);
+ }
+ *v = out;
+}
+
+/// Find the bucket index `i` such that `boundaries[i] <= val < boundaries[i+1]`.
+/// Returns `None` if `val` is outside the boundaries.
+fn find_bucket(boundaries: &[f32], val: f32) -> Option<usize> {
+ if boundaries.len() < 2 || val < boundaries[0] || val > *boundaries.last().unwrap() {
+ return None;
+ }
+ for (i, w) in boundaries.windows(2).enumerate() {
+ if val >= w[0] && val <= w[1] {
+ return Some(i);
+ }
+ }
+ None
+}
+
+/// Detect ruled-grid tables on a page from its vector graphics. Returns runs
+/// in document order (sorted by `start`).
+pub(super) fn detect_ruled_tables(
+ lines: &[ProjectedLine],
+ graphics: &[GraphicPrimitive],
+ page_width: f32,
+ page_height: f32,
+) -> Vec<TableRun> {
+ let (hs, vs) = extract_h_v_segments(graphics);
+ let hs = cluster_h_segments(hs);
+ let vs = cluster_v_segments(vs);
+ if hs.len() < 2 || vs.len() < 2 {
+ return Vec::new();
+ }
+ let components = find_grid_components(&hs, &vs);
+ let mut out = Vec::new();
+ for (h_idx, v_idx) in components {
+ if let Some(run) =
+ build_ruled_table(&hs, &vs, &h_idx, &v_idx, lines, page_width, page_height)
+ {
+ out.push(run);
+ }
+ }
+ out.sort_by_key(|r| r.start);
+ out
+}
+
+/// Merge ruled-grid runs with borderless runs into a single sorted list. When
+/// ranges overlap the ruled run wins (path-based geometry is strictly stronger
+/// than text-alignment heuristics) — overlapping borderless runs are dropped.
+pub(super) fn merge_table_runs(mut ruled: Vec<TableRun>, borderless: Vec<TableRun>) -> Vec<TableRun> {
+ for b in borderless {
+ let overlaps = ruled
+ .iter()
+ .any(|r| !(b.end <= r.start || b.start >= r.end));
+ if !overlaps {
+ ruled.push(b);
+ }
+ }
+ ruled.sort_by_key(|r| r.start);
+ ruled
+}
+
+/// Escape `|` and `\n` inside a markdown table cell so the pipe-table grammar
+/// stays valid. Newlines should be impossible inside a single cell (we built
+/// cells from spans on the same projected line) but guard anyway.
+pub(super) fn escape_table_cell(s: &str) -> String {
+ s.replace('\\', "\\\\")
+ .replace('|', "\\|")
+ .replace('\n', " ")
+}
+
+#[cfg(test)]
+mod tests {
+ use super::super::test_helpers::{line, line_with_spans, rect_borders, stroke};
+ use super::*;
+
+ #[test]
+ fn split_cells_splits_on_wide_gaps() {
+ let l = line_with_spans(&[("A", 50.0), ("B", 150.0), ("C", 250.0)], 100.0, 10.0);
+ let cells = split_cells(&l);
+ assert_eq!(cells.len(), 3);
+ assert_eq!(cells[0].text, "A");
+ assert_eq!(cells[1].text, "B");
+ assert_eq!(cells[2].text, "C");
+ }
+
+ #[test]
+ fn recover_merged_cell_splits_off_by_one() {
+ // Mimics the page-6 case: row 0 establishes 3 tracks at 50/150/250.
+ // Row 1's projection merges "MEMORYBANK" + "5.00" into one span at
+ // x=50 width=110, so split_cells yields 2 cells while the table
+ // expects 3. Recovery must split on whitespace at the missing track.
+ let row = vec![
+ TableCell {
+ start_x: 50.0,
+ end_x: 160.0,
+ text: "MEMORYBANK 5.00".into(),
+ bold: false,
+ },
+ TableCell {
+ start_x: 250.0,
+ end_x: 280.0,
+ text: "4.77".into(),
+ bold: false,
+ },
+ ];
+ let tracks = vec![50.0, 150.0, 250.0];
+ let out = recover_merged_cell(row, &tracks).expect("recovery should succeed");
+ assert_eq!(out.len(), 3);
+ assert_eq!(out[0].text, "MEMORYBANK");
+ assert_eq!(out[1].text, "5.00");
+ assert_eq!(out[2].text, "4.77");
+ }
+
+ #[test]
+ fn recover_merged_cell_splits_off_by_two() {
+ // Three merged tokens in one cell: "MEMORYBANK 13.18 10.03" straddles
+ // tracks at 50/150/250 and the row has only 2 cells, off by 2.
+ let row = vec![
+ TableCell {
+ start_x: 50.0,
+ end_x: 260.0,
+ text: "MEMORYBANK 13.18 10.03".into(),
+ bold: false,
+ },
+ TableCell {
+ start_x: 350.0,
+ end_x: 380.0,
+ text: "7.61".into(),
+ bold: false,
+ },
+ ];
+ let tracks = vec![50.0, 150.0, 250.0, 350.0];
+ let out = recover_merged_cell(row, &tracks).expect("recovery should succeed");
+ assert_eq!(out.len(), 4);
+ assert_eq!(out[0].text, "MEMORYBANK");
+ assert_eq!(out[1].text, "13.18");
+ assert_eq!(out[2].text, "10.03");
+ assert_eq!(out[3].text, "7.61");
+ }
+
+ #[test]
+ fn recover_merged_cell_bails_without_enough_whitespace() {
+ // A cell that straddles two tracks but has no internal whitespace
+ // (e.g. a hyphenated token) can't be safely split — return None.
+ let row = vec![TableCell {
+ start_x: 50.0,
+ end_x: 200.0,
+ text: "ABC-DEF-GHI".into(),
+ bold: false,
+ }];
+ let tracks = vec![50.0, 150.0];
+ assert!(recover_merged_cell(row, &tracks).is_none());
+ }
+
+ #[test]
+ fn split_cells_keeps_close_spans_together() {
+ // Two spans 2pt apart at 10pt font (gap < font_size) → same cell.
+ let l = line_with_spans(&[("Hello", 50.0), ("world", 80.0)], 100.0, 10.0);
+ let cells = split_cells(&l);
+ assert_eq!(cells.len(), 1);
+ assert_eq!(cells[0].text, "Hello world");
+ }
+
+ #[test]
+ fn rejects_table_when_row_count_too_low() {
+ let lines = vec![line_with_spans(
+ &[("A", 50.0), ("B", 150.0), ("C", 250.0)],
+ 100.0,
+ 10.0,
+ )];
+ let runs = detect_tables(&lines);
+ assert!(runs.is_empty());
+ }
+
+ #[test]
+ fn rejects_table_when_column_count_too_low() {
+ let lines = vec![
+ line_with_spans(&[("A", 50.0), ("B", 200.0)], 100.0, 10.0),
+ line_with_spans(&[("C", 50.0), ("D", 200.0)], 115.0, 10.0),
+ ];
+ let runs = detect_tables(&lines);
+ assert!(runs.is_empty());
+ }
+
+ #[test]
+ fn escapes_pipe_inside_cell() {
+ assert_eq!(escape_table_cell("a|b"), "a\\|b");
+ }
+
+ #[test]
+ fn ruled_table_2x2_detected() {
+ // 2 rows × 2 cols grid: 3 H lines (y=100,140,180), 3 V lines (x=50,150,250)
+ // Cell text dropped in the centroid of each cell.
+ let mut graphics = Vec::new();
+ for y in [100.0_f32, 140.0, 180.0] {
+ graphics.push(stroke(50.0, y, 250.0, y, 0.5));
+ }
+ for x in [50.0_f32, 150.0, 250.0] {
+ graphics.push(stroke(x, 100.0, x, 180.0, 0.5));
+ }
+
+ // Text lines: one per cell, centered.
+ let lines = vec![
+ line("a", 90.0, 115.0, 10.0, 10.0), // row 0, col 0
+ line("b", 190.0, 115.0, 10.0, 10.0), // row 0, col 1
+ line("c", 90.0, 155.0, 10.0, 10.0), // row 1, col 0
+ line("d", 190.0, 155.0, 10.0, 10.0), // row 1, col 1
+ ];
+
+ let runs = detect_ruled_tables(&lines, &graphics, 612.0, 792.0);
+ assert_eq!(runs.len(), 1, "expected 1 ruled table, got {runs:?}");
+ match &runs[0].block {
+ Block::Table { header, rows } => {
+ assert!(header.is_none(), "no bold first row → no header");
+ assert_eq!(rows.len(), 2);
+ assert_eq!(rows[0], vec!["a", "b"]);
+ assert_eq!(rows[1], vec!["c", "d"]);
+ }
+ other => panic!("expected Block::Table, got {other:?}"),
+ }
+ }
+
+ #[test]
+ fn ruled_table_rect_borders_detected() {
+ // Same 2×2 table but drawn as 4 individual cell rects (each cell is a
+ // stroked rectangle). Each rect contributes 4 strokes via
+ // extract_h_v_segments.
+ let mut graphics = Vec::new();
+ graphics.extend(rect_borders(50.0, 100.0, 100.0, 40.0)); // r0 c0
+ graphics.extend(rect_borders(150.0, 100.0, 100.0, 40.0)); // r0 c1
+ graphics.extend(rect_borders(50.0, 140.0, 100.0, 40.0)); // r1 c0
+ graphics.extend(rect_borders(150.0, 140.0, 100.0, 40.0)); // r1 c1
+
+ let lines = vec![
+ line("a", 90.0, 115.0, 10.0, 10.0),
+ line("b", 190.0, 115.0, 10.0, 10.0),
+ line("c", 90.0, 155.0, 10.0, 10.0),
+ line("d", 190.0, 155.0, 10.0, 10.0),
+ ];
+ let runs = detect_ruled_tables(&lines, &graphics, 612.0, 792.0);
+ assert_eq!(runs.len(), 1);
+ }
+
+ #[test]
+ fn ruled_table_page_border_rejected() {
+ // Single big rect covering ~the whole page → should NOT be treated as a
+ // table even though it has H+V lines on all four sides.
+ let graphics = rect_borders(10.0, 10.0, 590.0, 770.0);
+ let lines = vec![line("body text", 50.0, 400.0, 10.0, 10.0)];
+ let runs = detect_ruled_tables(&lines, &graphics, 612.0, 792.0);
+ assert!(
+ runs.is_empty(),
+ "page-border rect should not become a table, got {runs:?}"
+ );
+ }
+
+ #[test]
+ fn ruled_table_mostly_empty_rejected() {
+ // 3×3 grid with text in only one cell — empty fraction 8/9 ≈ 89% >> 30%.
+ let mut graphics = Vec::new();
+ for y in [100.0_f32, 130.0, 160.0, 190.0] {
+ graphics.push(stroke(50.0, y, 350.0, y, 0.5));
+ }
+ for x in [50.0_f32, 150.0, 250.0, 350.0] {
+ graphics.push(stroke(x, 100.0, x, 190.0, 0.5));
+ }
+ let lines = vec![line("only", 90.0, 115.0, 10.0, 10.0)];
+ let runs = detect_ruled_tables(&lines, &graphics, 612.0, 792.0);
+ assert!(runs.is_empty());
+ }
+
+ #[test]
+ fn ruled_table_first_row_bold_becomes_header() {
+ // 2×2 with first row text marked all_bold → header promotion.
+ let mut graphics = Vec::new();
+ for y in [100.0_f32, 140.0, 180.0] {
+ graphics.push(stroke(50.0, y, 250.0, y, 0.5));
+ }
+ for x in [50.0_f32, 150.0, 250.0] {
+ graphics.push(stroke(x, 100.0, x, 180.0, 0.5));
+ }
+ let mut a = line("Name", 90.0, 115.0, 10.0, 10.0);
+ let mut b = line("Score", 190.0, 115.0, 10.0, 10.0);
+ a.all_bold = true;
+ b.all_bold = true;
+ let lines = vec![
+ a,
+ b,
+ line("alice", 90.0, 155.0, 10.0, 10.0),
+ line("99", 190.0, 155.0, 10.0, 10.0),
+ ];
+ let runs = detect_ruled_tables(&lines, &graphics, 612.0, 792.0);
+ assert_eq!(runs.len(), 1);
+ match &runs[0].block {
+ Block::Table { header, rows } => {
+ assert_eq!(
+ header.as_deref(),
+ Some(&["Name".into(), "Score".into()][..])
+ );
+ assert_eq!(rows.len(), 1);
+ assert_eq!(rows[0], vec!["alice", "99"]);
+ }
+ other => panic!("expected Block::Table, got {other:?}"),
+ }
+ }
+
+ #[test]
+ fn merge_prefers_ruled_when_overlapping() {
+ let ruled = vec![TableRun {
+ start: 5,
+ end: 10,
+ block: Block::Table {
+ header: None,
+ rows: vec![vec!["ruled".into()]],
+ },
+ }];
+ let borderless = vec![TableRun {
+ start: 6,
+ end: 11,
+ block: Block::GridFallback {
+ lines: vec!["bl".into()],
+ },
+ }];
+ let merged = merge_table_runs(ruled, borderless);
+ assert_eq!(merged.len(), 1);
+ assert!(matches!(&merged[0].block, Block::Table { .. }));
+ }
+}
diff --git a/crates/liteparse/src/markdown_layout/test_helpers.rs b/crates/liteparse/src/markdown_layout/test_helpers.rs
new file mode 100644
index 0000000..4a91968
--- /dev/null
+++ b/crates/liteparse/src/markdown_layout/test_helpers.rs
@@ -0,0 +1,202 @@
+//! Shared test fixtures for `markdown_layout` submodule tests.
+
+use crate::types::{Anchor, GraphicPrimitive, ParsedPage, ProjectedLine, Rect, TextItem};
+
+pub(crate) fn line(text: &str, x: f32, y: f32, h: f32, size: f32) -> ProjectedLine {
+ ProjectedLine {
+ text: text.into(),
+ bbox: Rect {
+ x,
+ y,
+ width: text.chars().count() as f32 * (size * 0.5),
+ height: h,
+ },
+ anchor: Anchor::Left,
+ indent_x: x,
+ dominant_font_size: size,
+ dominant_font_name: Some("Arial".into()),
+ all_bold: false,
+ all_italic: false,
+ all_mono: false,
+ all_strike: false,
+ spans: vec![TextItem::default()],
+ region_path: Vec::new(),
+ mcid: None,
+ }
+}
+
+pub(crate) fn page(lines: Vec<ProjectedLine>) -> ParsedPage {
+ ParsedPage {
+ page_number: 1,
+ page_width: 612.0,
+ page_height: 792.0,
+ text: String::new(),
+ text_items: vec![],
+ projected_lines: lines,
+ regions: crate::types::Region::default(),
+ graphics: vec![],
+ figures: vec![],
+ struct_nodes: vec![],
+ image_refs: vec![],
+ }
+}
+
+pub(crate) fn mono_line(text: &str, y: f32) -> ProjectedLine {
+ let mut l = line(text, 50.0, y, 10.0, 10.0);
+ l.all_mono = true;
+ l
+}
+
+/// Build a line whose spans are placed at explicit x positions — used to
+/// drive the table detector, which relies on per-span x for cell splitting.
+pub(crate) fn line_with_spans(cells: &[(&str, f32)], y: f32, size: f32) -> ProjectedLine {
+ let spans: Vec<TextItem> = cells
+ .iter()
+ .map(|(t, x)| TextItem {
+ text: (*t).into(),
+ x: *x,
+ y,
+ width: t.chars().count() as f32 * size * 0.5,
+ height: size,
+ font_size: Some(size),
+ font_name: Some("Arial".into()),
+ ..Default::default()
+ })
+ .collect();
+ let min_x = spans.iter().map(|s| s.x).fold(f32::INFINITY, f32::min);
+ let max_x = spans
+ .iter()
+ .map(|s| s.x + s.width)
+ .fold(f32::NEG_INFINITY, f32::max);
+ ProjectedLine {
+ text: cells
+ .iter()
+ .map(|(t, _)| *t)
+ .collect::<Vec<_>>()
+ .join(" "),
+ bbox: Rect {
+ x: min_x,
+ y,
+ width: (max_x - min_x).max(0.0),
+ height: size,
+ },
+ anchor: Anchor::Left,
+ indent_x: min_x,
+ dominant_font_size: size,
+ dominant_font_name: Some("Arial".into()),
+ all_bold: false,
+ all_italic: false,
+ all_mono: false,
+ all_strike: false,
+ spans,
+ region_path: Vec::new(),
+ mcid: None,
+ }
+}
+
+/// Build a line whose spans carry explicit per-span font metadata. Lets us
+/// exercise the mid-line emphasis pipeline without needing real PDF input.
+pub(crate) fn styled_line(
+ spans: &[(&str, f32, Option<&str>)],
+ y: f32,
+ size: f32,
+) -> ProjectedLine {
+ let items: Vec<TextItem> = spans
+ .iter()
+ .map(|(t, x, font)| TextItem {
+ text: (*t).into(),
+ x: *x,
+ y,
+ width: t.chars().count() as f32 * size * 0.5,
+ height: size,
+ font_size: Some(size),
+ font_name: font.map(String::from),
+ ..Default::default()
+ })
+ .collect();
+ let joined: String = spans
+ .iter()
+ .map(|(t, _, _)| *t)
+ .collect::<Vec<_>>()
+ .join(" ");
+ let min_x = items.iter().map(|s| s.x).fold(f32::INFINITY, f32::min);
+ let max_x = items
+ .iter()
+ .map(|s| s.x + s.width)
+ .fold(f32::NEG_INFINITY, f32::max);
+ ProjectedLine {
+ text: joined,
+ bbox: Rect {
+ x: min_x,
+ y,
+ width: (max_x - min_x).max(0.0),
+ height: size,
+ },
+ anchor: Anchor::Left,
+ indent_x: min_x,
+ dominant_font_size: size,
+ dominant_font_name: Some("Arial".into()),
+ all_bold: false,
+ all_italic: false,
+ all_mono: false,
+ all_strike: false,
+ spans: items,
+ region_path: Vec::new(),
+ mcid: None,
+ }
+}
+
+pub(crate) fn stroke(x1: f32, y1: f32, x2: f32, y2: f32, width: f32) -> GraphicPrimitive {
+ GraphicPrimitive::Stroke {
+ x1,
+ y1,
+ x2,
+ y2,
+ width,
+ color: None,
+ }
+}
+
+pub(crate) fn page_with_graphics(
+ lines: Vec<ProjectedLine>,
+ graphics: Vec<GraphicPrimitive>,
+) -> ParsedPage {
+ let mut p = page(lines);
+ p.graphics = graphics;
+ p
+}
+
+pub(crate) fn header_footer_page(n: usize, header: &str, footer: &str, body: &str) -> ParsedPage {
+ // Page height 100 → header band ≤12pt, footer band ≥88pt.
+ let mut lines = vec![
+ line(header, 50.0, 5.0, 8.0, 8.0),
+ line(body, 50.0, 50.0, 10.0, 10.0),
+ line(footer, 50.0, 92.0, 6.0, 6.0),
+ ];
+ for l in &mut lines {
+ l.region_path = Vec::new();
+ }
+ ParsedPage {
+ page_number: n,
+ page_width: 612.0,
+ page_height: 100.0,
+ text: String::new(),
+ text_items: vec![],
+ projected_lines: lines,
+ regions: crate::types::Region::default(),
+ graphics: vec![],
+ figures: vec![],
+ struct_nodes: vec![],
+ image_refs: vec![],
+ }
+}
+
+/// Helper: build the four borders of a rectangle as four strokes.
+pub(crate) fn rect_borders(x: f32, y: f32, w: f32, h: f32) -> Vec<GraphicPrimitive> {
+ vec![
+ stroke(x, y, x + w, y, 0.5), // top
+ stroke(x, y + h, x + w, y + h, 0.5), // bottom
+ stroke(x, y, x, y + h, 0.5), // left
+ stroke(x + w, y, x + w, y + h, 0.5), // right
+ ]
+}
diff --git a/crates/liteparse/src/output/json.rs b/crates/liteparse/src/output/json.rs
index b6db995..1ac0482 100644
--- a/crates/liteparse/src/output/json.rs
+++ b/crates/liteparse/src/output/json.rs
@@ -95,6 +95,8 @@ mod tests {
regions: crate::types::Region::default(),
graphics: vec![],
figures: vec![],
+ struct_nodes: vec![],
+ image_refs: vec![],
}
}
diff --git a/crates/liteparse/src/output/markdown.rs b/crates/liteparse/src/output/markdown.rs
index 69a8658..e6aebcb 100644
--- a/crates/liteparse/src/output/markdown.rs
+++ b/crates/liteparse/src/output/markdown.rs
@@ -1,8 +1,9 @@
+use crate::config::ImageMode;
use crate::markdown_layout::{
build_heading_map, classify_page_with_filters, compute_body_size, compute_header_footer_set,
render_blocks,
};
-use crate::types::ParsedPage;
+use crate::types::{OutlineTarget, ParsedPage};
/// Format parsed pages as markdown.
///
@@ -15,7 +16,11 @@ use crate::types::ParsedPage;
/// `<!-- page N -->` marker. Pages that contain no projected lines (e.g. blank
/// or fully-OCR pages without font-size info) fall back to the projected text
/// wrapped in a fenced block so we never silently drop content.
-pub fn format_markdown(pages: &[ParsedPage]) -> String {
+pub fn format_markdown(
+ pages: &[ParsedPage],
+ outline: &[OutlineTarget],
+ image_mode: ImageMode,
+) -> String {
if pages.is_empty() {
return String::new();
}
@@ -43,7 +48,21 @@ pub fn format_markdown(pages: &[ParsedPage]) -> String {
continue;
}
- let blocks = classify_page_with_filters(page, &heading_map, &header_footer);
+ // Filter outline entries to this page so the classifier's y/title
+ // match is a O(entries_on_page) scan per line, not O(whole doc).
+ let target_index = (page.page_number as i32).saturating_sub(1);
+ let page_outline: Vec<OutlineTarget> = outline
+ .iter()
+ .filter(|e| e.page_index == target_index)
+ .cloned()
+ .collect();
+ let blocks = classify_page_with_filters(
+ page,
+ &heading_map,
+ &header_footer,
+ &page_outline,
+ image_mode,
+ );
out.push_str(&render_blocks(&blocks));
}
out
@@ -88,12 +107,14 @@ mod tests {
regions: crate::types::Region::default(),
graphics: vec![],
figures: vec![],
+ struct_nodes: vec![],
+ image_refs: vec![],
}
}
#[test]
fn test_empty() {
- assert_eq!(format_markdown(&[]), "");
+ assert_eq!(format_markdown(&[], &[], ImageMode::Placeholder), "");
}
#[test]
@@ -108,8 +129,10 @@ mod tests {
regions: crate::types::Region::default(),
graphics: vec![],
figures: vec![],
+ struct_nodes: vec![],
+ image_refs: vec![],
};
- let out = format_markdown(&[p]);
+ let out = format_markdown(&[p], &[], ImageMode::Placeholder);
assert!(out.contains("```text"));
assert!(out.contains("hello"));
assert!(out.contains("<!-- page 1 -->"));
@@ -140,7 +163,7 @@ mod tests {
),
],
);
- let out = format_markdown(&[p]);
+ let out = format_markdown(&[p], &[], ImageMode::Placeholder);
assert!(out.contains("# My Title For This Test Document"));
assert!(out.contains("First sentence of body prose here."));
}
@@ -149,7 +172,7 @@ mod tests {
fn test_multi_page_separator() {
let a = page_with(1, vec![line("A page.", 50.0, 80.0, 10.0, 10.0)]);
let b = page_with(2, vec![line("B page.", 50.0, 80.0, 10.0, 10.0)]);
- let out = format_markdown(&[a, b]);
+ let out = format_markdown(&[a, b], &[], ImageMode::Placeholder);
assert!(out.contains("-----"));
assert!(out.find("A page.").unwrap() < out.find("B page.").unwrap());
}
diff --git a/crates/liteparse/src/output/text.rs b/crates/liteparse/src/output/text.rs
index 82bf1a9..196ce7b 100644
--- a/crates/liteparse/src/output/text.rs
+++ b/crates/liteparse/src/output/text.rs
@@ -24,6 +24,8 @@ mod tests {
regions: crate::types::Region::default(),
graphics: vec![],
figures: vec![],
+ struct_nodes: vec![],
+ image_refs: vec![],
}
}
diff --git a/crates/liteparse/src/parser.rs b/crates/liteparse/src/parser.rs
index 4fa3512..6f53ceb 100644
--- a/crates/liteparse/src/parser.rs
+++ b/crates/liteparse/src/parser.rs
@@ -13,7 +13,7 @@ use crate::output::markdown;
use crate::projection;
#[cfg(not(target_arch = "wasm32"))]
use crate::render;
-use crate::types::{ParsedPage, PdfInput};
+use crate::types::{ExtractedImage, OutlineTarget, ParsedPage, PdfInput};
/// Result of parsing a document.
pub struct ParseResult {
@@ -21,6 +21,14 @@ pub struct ParseResult {
pub pages: Vec<ParsedPage>,
/// Full document text, concatenated from all pages.
pub text: String,
+ /// Document outline (bookmarks) when present. Used by the markdown
+ /// emitter as a high-priority heading source on untagged PDFs.
+ pub outline: Vec<OutlineTarget>,
+ /// Raster images extracted from the document. Empty unless the parser
+ /// was configured with `ImageMode::Embed`. Each entry carries the same
+ /// `id` the markdown emitter referenced in `![](image_{id}.png)`, so the
+ /// caller can match them up without parsing markdown.
+ pub images: Vec<ExtractedImage>,
}
/// Result of rendering a single page screenshot.
@@ -99,13 +107,21 @@ impl LiteParse {
.map_err(|e| format!("invalid --target-pages: {}", e))?;
// Extract text (and pre-render OCR pages in one PDF load when OCR is on).
+ // The Document handle isn't Send, so we open it, pull every bit of
+ // document-level state we need, and drop it before any `.await`.
let password = self.config.password.as_deref();
- let (mut pages, ocr_rendered) = if self.config.ocr_enabled {
+ let render_images = matches!(
+ self.config.image_mode,
+ crate::config::ImageMode::Embed
+ );
+ let (pages, ocr_rendered, outline, images) = {
let document = extract::load_document_from_input(&validated_input, password)?;
- let pages = extract::extract_pages_from_document(
+ let outline = extract::extract_outline(&document);
+ let (pages, images) = extract::extract_pages_and_images(
&document,
target_pages.as_deref(),
self.config.max_pages,
+ render_images,
)?;
let t_extract = web_time::Instant::now();
log(&format!(
@@ -113,30 +129,23 @@ impl LiteParse {
t_extract.duration_since(t0).as_secs_f64() * 1000.0,
pages.len()
));
- let rendered = ocr_merge::render_pages_for_ocr(&document, &pages, self.config.dpi)?;
- log(&format!(
- "[liteparse] ocr render: {:.1}ms ({} pages)",
- web_time::Instant::now()
- .duration_since(t_extract)
- .as_secs_f64()
- * 1000.0,
- rendered.len()
- ));
- (pages, rendered)
- } else {
- let pages = extract::extract_pages_from_input(
- &validated_input,
- target_pages.as_deref(),
- self.config.max_pages,
- password,
- )?;
- log(&format!(
- "[liteparse] extract: {:.1}ms ({} pages)",
- web_time::Instant::now().duration_since(t0).as_secs_f64() * 1000.0,
- pages.len()
- ));
- (pages, Vec::new())
+ let rendered = if self.config.ocr_enabled {
+ let r = ocr_merge::render_pages_for_ocr(&document, &pages, self.config.dpi)?;
+ log(&format!(
+ "[liteparse] ocr render: {:.1}ms ({} pages)",
+ web_time::Instant::now()
+ .duration_since(t_extract)
+ .as_secs_f64()
+ * 1000.0,
+ r.len()
+ ));
+ r
+ } else {
+ Vec::new()
+ };
+ (pages, rendered, outline, images)
};
+ let mut pages = pages;
let t1 = web_time::Instant::now();
// OCR pass
@@ -195,7 +204,7 @@ impl LiteParse {
));
let full_text = if self.config.output_format == crate::config::OutputFormat::Markdown {
- let md = markdown::format_markdown(&parsed_pages);
+ let md = markdown::format_markdown(&parsed_pages, &outline, self.config.image_mode);
let t3 = web_time::Instant::now();
log(&format!(
"[liteparse] markdown: {:.1}ms",
@@ -216,6 +225,8 @@ impl LiteParse {
Ok(ParseResult {
pages: parsed_pages,
text: full_text,
+ outline,
+ images,
})
}
diff --git a/crates/liteparse/src/projection.rs b/crates/liteparse/src/projection.rs
index 3eeffa8..b592308 100644
--- a/crates/liteparse/src/projection.rs
+++ b/crates/liteparse/src/projection.rs
@@ -2680,6 +2680,8 @@ pub fn project_pages_to_grid(pages: Vec<Page>) -> Vec<ParsedPage> {
regions,
graphics: page.graphics,
figures,
+ struct_nodes: page.struct_nodes,
+ image_refs: page.image_refs,
}
})
.collect()
@@ -2803,6 +2805,30 @@ const XY_MIN_LINES_PER_H_SIDE: usize = 2;
/// before columns — matches natural reading order.
const XY_V_PREFERENCE_MARGIN: f32 = 1.1;
+/// Bucket width used by the column-start histogram fallback. 5pt is finer
+/// than typical inter-column gutters (8–20pt) and coarser than baseline
+/// noise / kerning drift, so column edges land in distinct buckets.
+const XY_COLUMN_BUCKET_PT: f32 = 5.0;
+/// Minimum lines stacked at the same left-edge cluster for the cluster to
+/// count as a column peak. A real column has many lines at one x (15+);
+/// indented paragraph-starts and footnote markers produce a handful (3–6).
+/// We set this above the typical "indented paragraph" count so those drop
+/// out and only true column-left edges remain.
+const XY_COLUMN_MIN_LINES_PER_PEAK: usize = 10;
+/// Minimum horizontal distance between two column peaks, as a fraction of
+/// `bbox.width`. Sub-paragraph indents and list nesting produce
+/// closely-spaced peaks; a real column gutter is much further out.
+const XY_COLUMN_MIN_GAP_FRACTION: f32 = 0.25;
+/// Minimum fraction of items that must start at one of the two detected
+/// column peaks. Real 2-column layouts cluster nearly every line at one of
+/// two left edges; tables, lists, and scattered prose don't.
+const XY_COLUMN_PEAK_DOMINANCE: f32 = 0.55;
+/// Minimum smaller/larger ratio between the two column peaks' line counts.
+/// Balanced 2-column layouts have ~equal line counts in each column; a
+/// 90/10 split is almost certainly a single column with one off-x bullet
+/// stack, not a real two-column structure.
+const XY_COLUMN_PEAK_BALANCE_RATIO: f32 = 0.4;
+
#[derive(Debug)]
struct CutCandidate {
axis: CutAxis,
@@ -3081,6 +3107,154 @@ const XY_BANNER_CLEARANCE_FACTOR: f32 = 1.5;
/// Also helps mid-document section headings that span the full content width:
/// a wide centered "5 Conclusion" line above a 2-column body would otherwise
/// get pulled into one column.
+/// Fallback column-gutter detector for cases where the density-based V-cut
+/// search finds no valley — typically because a column-spanning element
+/// (wide heading, figure, table) fills the gutter at one y-band and inflates
+/// the projection at the gutter's x. Detects 2+ column structure by clustering
+/// the left edges of "narrow" lines (lines that don't span the full region
+/// width). When two clusters with ≥`XY_COLUMN_MIN_LINES_PER_PEAK` lines each
+/// are separated by ≥`XY_COLUMN_MIN_GAP_FRACTION × bbox.width`, returns a
+/// V-cut at the midpoint between them with a high score so it beats any
+/// density-based cut except banner.
+fn xy_find_column_cut(
+ items: &[ProjectedTextItem],
+ idxs: &[usize],
+ bbox: &Rect,
+ _median_h: f32,
+) -> Option<CutCandidate> {
+ if bbox.width <= 1.0 || idxs.len() < 8 {
+ return None;
+ }
+
+ // Histogram of individual item left edges. We deliberately avoid
+ // first-line-then-leftmost banding because items in the same y-band can
+ // belong to *different* columns (left-column and right-column items at
+ // the same baseline), and folding them into "one line's x_min" hides
+ // the right column entirely. Instead, every item contributes its own x
+ // to the histogram; a real column edge shows up as a tall spike (many
+ // line-starts at one x) while mid-line continuation fragments spread
+ // their contribution across many buckets and don't form spikes.
+ let mut min_x = f32::INFINITY;
+ let mut max_x = f32::NEG_INFINITY;
+ for &i in idxs {
+ let it = &items[i].item;
+ let x = it.x;
+ if x.is_finite() {
+ min_x = min_x.min(x);
+ max_x = max_x.max(x + it.width.max(0.0));
+ }
+ }
+ if !min_x.is_finite() || max_x <= min_x + 1.0 {
+ return None;
+ }
+ let total_w = max_x - min_x;
+ let bucket_pt = XY_COLUMN_BUCKET_PT;
+ let n_buckets = ((total_w / bucket_pt).ceil() as usize).max(1);
+ let mut hist = vec![0usize; n_buckets];
+ for &i in idxs {
+ let it = &items[i].item;
+ let b_f = ((it.x - min_x) / bucket_pt).floor();
+ let b = b_f.max(0.0) as usize;
+ if b < n_buckets {
+ hist[b] += 1;
+ }
+ }
+
+ // Cluster adjacent occupied buckets into peaks. A peak is a contiguous
+ // run of buckets with ≥1 count, merged through gaps of ≤2 zero buckets
+ // (~10pt) to absorb slight intra-column drift.
+ let mut peaks: Vec<(f32, usize)> = Vec::new(); // (x_center, item_count)
+ let mut i = 0;
+ while i < n_buckets {
+ if hist[i] == 0 {
+ i += 1;
+ continue;
+ }
+ let s = i;
+ let mut total = 0usize;
+ let mut last_nonzero = i;
+ let mut j = i;
+ while j < n_buckets {
+ if hist[j] > 0 {
+ total += hist[j];
+ last_nonzero = j;
+ j += 1;
+ } else if j - last_nonzero <= 2 {
+ j += 1;
+ } else {
+ break;
+ }
+ }
+ let x_center = min_x + bucket_pt * (s as f32 + last_nonzero as f32 + 1.0) * 0.5;
+ peaks.push((x_center, total));
+ i = j;
+ }
+
+ // Keep only "strong" peaks. A real column edge is sat-stacked by many
+ // lines at the same left x; spurious peaks (a stray indented quote, a
+ // numbered marker, etc.) carry only a few lines.
+ peaks.retain(|(_, c)| *c >= XY_COLUMN_MIN_LINES_PER_PEAK);
+ if peaks.len() < 2 {
+ return None;
+ }
+ peaks.sort_by(|a, b| a.0.total_cmp(&b.0));
+
+ // Tabular layouts produce 3+ strong peaks (one per cell column). Real
+ // page columns have exactly 2 dominant peaks (left edge of col 1, left
+ // edge of col 2) — even multi-paragraph indents are sub-peaks weak
+ // enough to drop out under the min_per_peak filter. Bail when we see
+ // more than 2 strong peaks so we don't slice tables.
+ if peaks.len() > 2 {
+ return None;
+ }
+ // Require the two peaks to together dominate the histogram — a real
+ // 2-column layout has the great majority of items starting at one of
+ // the two column-left edges. Tables, lists, and prose with scattered
+ // indents distribute their item-starts across many x's and won't pass.
+ let peak_sum: usize = peaks.iter().map(|(_, c)| *c).sum();
+ let total_items = idxs.len();
+ if (peak_sum as f32) / (total_items as f32) < XY_COLUMN_PEAK_DOMINANCE {
+ return None;
+ }
+ // Require the two peaks to be roughly balanced in size. A real 2-column
+ // layout has comparable line counts on each side; a long-single-column
+ // page that happens to have one bullet stack at a different x produces
+ // a tall + tiny peak pair that should not trigger a column split.
+ let (a, b) = (peaks[0].1, peaks[1].1);
+ let (smaller, larger) = (a.min(b) as f32, a.max(b) as f32);
+ if smaller / larger < XY_COLUMN_PEAK_BALANCE_RATIO {
+ return None;
+ }
+
+ // Find the widest gap between adjacent strong peaks. Must clear
+ // `XY_COLUMN_MIN_GAP_FRACTION × total_w` — list-item indents and
+ // block-quote indents produce closely spaced peaks; a real column
+ // gutter is much further apart.
+ let min_gap = total_w * XY_COLUMN_MIN_GAP_FRACTION;
+ let mut best: Option<(f32, usize)> = None;
+ for w_idx in 0..peaks.len() - 1 {
+ let gap = peaks[w_idx + 1].0 - peaks[w_idx].0;
+ if gap >= min_gap && best.is_none_or(|(g, _)| gap > g) {
+ best = Some((gap, w_idx));
+ }
+ }
+ let (_, li) = best?;
+ let cut_x = (peaks[li].0 + peaks[li + 1].0) * 0.5;
+ // Validate cut lands inside the bbox; if bbox-clamping moved the bbox
+ // off the items (some PDFs have content with negative x), fall back to
+ // an item-relative midpoint check.
+ if cut_x <= min_x + 1.0 || cut_x >= max_x - 1.0 {
+ return None;
+ }
+ // High finite score so this beats every density cut but stays below
+ // banner's infinity.
+ Some(CutCandidate {
+ axis: CutAxis::Vertical,
+ position: cut_x,
+ score: 1.0e9,
+ })
+}
+
fn xy_find_banner_cut(
items: &[ProjectedTextItem],
idxs: &[usize],
@@ -3207,18 +3381,29 @@ fn xy_cut_rec(
let banner = xy_find_banner_cut(items, &idxs, &bbox, median_h);
let h = xy_find_best_cut(items, &idxs, &bbox, CutAxis::Horizontal, median_h, figures);
let v = xy_find_best_cut(items, &idxs, &bbox, CutAxis::Vertical, median_h, figures);
- let cut = match (banner, h, v) {
- (Some(bc), _, _) => Some(bc),
- (None, Some(hc), Some(vc)) => {
+ // Fallback column-start histogram. Only consulted when BOTH density
+ // cuts returned None — those are the cases (wide-heading filled gutter,
+ // figure-straddle layouts) where the column structure is real but the
+ // density projection can't see it. If either density cut found a
+ // valley, trust it instead of firing the histogram fallback.
+ let column = if v.is_none() && h.is_none() {
+ xy_find_column_cut(items, &idxs, &bbox, median_h)
+ } else {
+ None
+ };
+ let cut = match (banner, column, h, v) {
+ (Some(bc), _, _, _) => Some(bc),
+ (None, Some(cc), _, _) => Some(cc),
+ (None, None, Some(hc), Some(vc)) => {
if vc.score > hc.score * XY_V_PREFERENCE_MARGIN {
Some(vc)
} else {
Some(hc)
}
}
- (None, Some(hc), None) => Some(hc),
- (None, None, Some(vc)) => Some(vc),
- (None, None, None) => None,
+ (None, None, Some(hc), None) => Some(hc),
+ (None, None, None, Some(vc)) => Some(vc),
+ (None, None, None, None) => None,
};
let Some(cut) = cut else {
return Region {
@@ -3626,6 +3811,8 @@ mod tests {
page_height: 792.0,
graphics: Vec::new(),
text_items: Vec::new(),
+ struct_nodes: Vec::new(),
+ image_refs: Vec::new(),
};
let projection_boxes = vec![
projected_item("", 10.0, 0.0, 10.0),
@@ -3850,6 +4037,8 @@ mod tests {
page_height: 792.0,
text_items: Vec::new(),
graphics: Vec::new(),
+ struct_nodes: Vec::new(),
+ image_refs: Vec::new(),
}];
let parsed = project_pages_to_grid(pages);
diff --git a/crates/liteparse/src/render.rs b/crates/liteparse/src/render.rs
index 53eb140..3ac068e 100644
--- a/crates/liteparse/src/render.rs
+++ b/crates/liteparse/src/render.rs
@@ -61,7 +61,7 @@ fn render_document_pages(
Ok(results)
}
-fn encode_png(rgba: &[u8], width: u32, height: u32) -> Result<Vec<u8>, LiteParseError> {
+pub(crate) fn encode_png(rgba: &[u8], width: u32, height: u32) -> Result<Vec<u8>, LiteParseError> {
let mut png_buf = Vec::new();
let encoder = image::codecs::png::PngEncoder::new(&mut png_buf);
encoder.write_image(rgba, width, height, image::ColorType::Rgba8.into())?;
diff --git a/crates/liteparse/src/types.rs b/crates/liteparse/src/types.rs
index 977b731..b48c8c0 100644
--- a/crates/liteparse/src/types.rs
+++ b/crates/liteparse/src/types.rs
@@ -66,6 +66,44 @@ pub struct Page {
/// Not emitted in JSON/text outputs — consumed by the markdown layout pass.
#[serde(skip)]
pub graphics: Vec<GraphicPrimitive>,
+ /// Structure-tree nodes for this page when the PDF is tagged. Each node
+ /// carries its role, marked-content ids, and the union bbox of its tagged
+ /// content. Empty for untagged PDFs.
+ #[serde(skip)]
+ pub struct_nodes: Vec<StructNode>,
+ /// Raster image objects detected on the page. Empty when the page has no
+ /// images. Threaded through to `ParsedPage.image_refs`.
+ #[serde(skip)]
+ pub image_refs: Vec<ImageRef>,
+}
+
+/// One entry in the document outline (bookmarks). Coordinates are in PDF
+/// user space (origin bottom-left) — convert to viewport with
+/// `page_height - y` once you know the page.
+#[doc(hidden)]
+#[derive(Debug, Clone)]
+pub struct OutlineTarget {
+ /// Hierarchy depth, 1-based.
+ pub level: u8,
+ pub title: String,
+ /// Zero-based page index of the destination.
+ pub page_index: i32,
+ /// Y in PDF user space, top of the target location. `None` when the
+ /// destination doesn't specify a Y.
+ pub y_pdf: Option<f32>,
+}
+
+/// One node from the structure tree of a page. Pre-flattened in pre-order
+/// (parent before children).
+#[doc(hidden)]
+#[derive(Debug, Clone)]
+pub struct StructNode {
+ pub role: String,
+ pub mcids: Vec<i32>,
+ /// Union bbox of page objects tagged with `mcids`, in viewport coords.
+ /// `None` when none of the mcids resolved to a bbox.
+ pub bbox: Option<Rect>,
+ pub alt_text: Option<String>,
}
/// Represents a fully parsed page with projected text layout.
@@ -94,6 +132,45 @@ pub struct ParsedPage {
/// obstacles, and reused downstream for figure classification.
#[serde(skip)]
pub figures: Vec<Rect>,
+ /// Structure-tree nodes for this page (tagged PDFs only). Pre-flattened in
+ /// pre-order. Consumed by the markdown classifier for highest-priority
+ /// heading / figure / table detection.
+ #[serde(skip)]
+ pub struct_nodes: Vec<StructNode>,
+ /// Raster image objects on the page. Bbox in viewport coords. Populated
+ /// during extraction; consumed by the markdown emitter to interleave
+ /// `Block::Figure` references at the right y position. Empty when the
+ /// page has no embedded images. Not part of JSON/text output.
+ #[serde(skip)]
+ pub image_refs: Vec<ImageRef>,
+}
+
+/// One embedded raster image on a page. `id` is a stable, page-scoped slug
+/// used as the markdown link target (e.g. `image_p1_0.png`). `obj_index` is
+/// the image's position among image page-objects, so a later embed pass can
+/// re-open the document and pull pixel bytes with `render_image_object`.
+#[doc(hidden)]
+#[derive(Debug, Clone)]
+pub struct ImageRef {
+ pub id: String,
+ pub bbox: Rect,
+ pub obj_index: usize,
+}
+
+/// A raster image extracted from a page along with its pixel bytes. Surfaced
+/// on `ParseResult.images` only when `ImageMode::Embed` is configured —
+/// otherwise the extraction step skips the render and only `ImageRef`s are
+/// produced. `format` is currently always `"png"` (encoded from the
+/// FPDFImageObj_GetRenderedBitmap output via the same path used for page
+/// screenshots).
+#[derive(Debug, Clone, Serialize)]
+pub struct ExtractedImage {
+ pub id: String,
+ pub page: u32,
+ pub bbox: Rect,
+ pub format: String,
+ #[serde(skip)]
+ pub bytes: Vec<u8>,
}
#[doc(hidden)]
@@ -290,6 +367,8 @@ mod tests {
page_height: 200.0,
text_items: vec![sample_item()],
graphics: vec![],
+ struct_nodes: vec![],
+ image_refs: vec![],
};
let s = serde_json::to_string(&p).unwrap();
assert!(s.contains("\"page_number\":1"));
diff --git a/crates/pdfium-sys/src/dynamic.rs b/crates/pdfium-sys/src/dynamic.rs
index 660545d..e4a033a 100644
--- a/crates/pdfium-sys/src/dynamic.rs
+++ b/crates/pdfium-sys/src/dynamic.rs
@@ -230,6 +230,66 @@ pub struct PdfiumBindings {
pub FPDFFont_GetGlyphWidth: unsafe extern "C" fn(FPDF_FONT, u32, f32, *mut f32) -> FPDF_BOOL,
pub FPDFFont_GetGlyphWidthFromCharCode:
unsafe extern "C" fn(FPDF_FONT, u32, f32, *mut f32) -> FPDF_BOOL,
+
+ // -- Outline (bookmarks) --
+ pub FPDFBookmark_GetFirstChild:
+ unsafe extern "C" fn(FPDF_DOCUMENT, FPDF_BOOKMARK) -> FPDF_BOOKMARK,
+ pub FPDFBookmark_GetNextSibling:
+ unsafe extern "C" fn(FPDF_DOCUMENT, FPDF_BOOKMARK) -> FPDF_BOOKMARK,
+ pub FPDFBookmark_GetTitle: unsafe extern "C" fn(
+ FPDF_BOOKMARK,
+ *mut std::os::raw::c_void,
+ std::os::raw::c_ulong,
+ ) -> std::os::raw::c_ulong,
+ pub FPDFBookmark_GetDest: unsafe extern "C" fn(FPDF_DOCUMENT, FPDF_BOOKMARK) -> FPDF_DEST,
+ pub FPDFBookmark_GetAction: unsafe extern "C" fn(FPDF_BOOKMARK) -> FPDF_ACTION,
+ pub FPDFAction_GetDest: unsafe extern "C" fn(FPDF_DOCUMENT, FPDF_ACTION) -> FPDF_DEST,
+ pub FPDFDest_GetDestPageIndex:
+ unsafe extern "C" fn(FPDF_DOCUMENT, FPDF_DEST) -> std::os::raw::c_int,
+ pub FPDFDest_GetLocationInPage: unsafe extern "C" fn(
+ FPDF_DEST,
+ *mut FPDF_BOOL,
+ *mut FPDF_BOOL,
+ *mut FPDF_BOOL,
+ *mut FS_FLOAT,
+ *mut FS_FLOAT,
+ *mut FS_FLOAT,
+ ) -> FPDF_BOOL,
+
+ // -- Structure tree --
+ pub FPDF_StructTree_GetForPage: unsafe extern "C" fn(FPDF_PAGE) -> FPDF_STRUCTTREE,
+ pub FPDF_StructTree_Close: unsafe extern "C" fn(FPDF_STRUCTTREE),
+ pub FPDF_StructTree_CountChildren:
+ unsafe extern "C" fn(FPDF_STRUCTTREE) -> std::os::raw::c_int,
+ pub FPDF_StructTree_GetChildAtIndex:
+ unsafe extern "C" fn(FPDF_STRUCTTREE, std::os::raw::c_int) -> FPDF_STRUCTELEMENT,
+ pub FPDF_StructElement_GetType: unsafe extern "C" fn(
+ FPDF_STRUCTELEMENT,
+ *mut std::os::raw::c_void,
+ std::os::raw::c_ulong,
+ ) -> std::os::raw::c_ulong,
+ pub FPDF_StructElement_GetAltText: unsafe extern "C" fn(
+ FPDF_STRUCTELEMENT,
+ *mut std::os::raw::c_void,
+ std::os::raw::c_ulong,
+ ) -> std::os::raw::c_ulong,
+ pub FPDF_StructElement_GetTitle: unsafe extern "C" fn(
+ FPDF_STRUCTELEMENT,
+ *mut std::os::raw::c_void,
+ std::os::raw::c_ulong,
+ ) -> std::os::raw::c_ulong,
+ pub FPDF_StructElement_CountChildren:
+ unsafe extern "C" fn(FPDF_STRUCTELEMENT) -> std::os::raw::c_int,
+ pub FPDF_StructElement_GetChildAtIndex:
+ unsafe extern "C" fn(FPDF_STRUCTELEMENT, std::os::raw::c_int) -> FPDF_STRUCTELEMENT,
+ pub FPDF_StructElement_GetChildMarkedContentID:
+ unsafe extern "C" fn(FPDF_STRUCTELEMENT, std::os::raw::c_int) -> std::os::raw::c_int,
+ pub FPDF_StructElement_GetMarkedContentID:
+ unsafe extern "C" fn(FPDF_STRUCTELEMENT) -> std::os::raw::c_int,
+ pub FPDF_StructElement_GetMarkedContentIdCount:
+ unsafe extern "C" fn(FPDF_STRUCTELEMENT) -> std::os::raw::c_int,
+ pub FPDF_StructElement_GetMarkedContentIdAtIndex:
+ unsafe extern "C" fn(FPDF_STRUCTELEMENT, std::os::raw::c_int) -> std::os::raw::c_int,
}
// SAFETY: PdfiumBindings contains only function pointers and a Library handle.
@@ -308,6 +368,42 @@ impl PdfiumBindings {
FPDFFont_GetDescent: load_fn!(lib, "FPDFFont_GetDescent"),
FPDFFont_GetGlyphWidth: load_fn!(lib, "FPDFFont_GetGlyphWidth"),
FPDFFont_GetGlyphWidthFromCharCode: load_fn!(lib, "FPDFFont_GetGlyphWidthFromCharCode"),
+
+ FPDFBookmark_GetFirstChild: load_fn!(lib, "FPDFBookmark_GetFirstChild"),
+ FPDFBookmark_GetNextSibling: load_fn!(lib, "FPDFBookmark_GetNextSibling"),
+ FPDFBookmark_GetTitle: load_fn!(lib, "FPDFBookmark_GetTitle"),
+ FPDFBookmark_GetDest: load_fn!(lib, "FPDFBookmark_GetDest"),
+ FPDFBookmark_GetAction: load_fn!(lib, "FPDFBookmark_GetAction"),
+ FPDFAction_GetDest: load_fn!(lib, "FPDFAction_GetDest"),
+ FPDFDest_GetDestPageIndex: load_fn!(lib, "FPDFDest_GetDestPageIndex"),
+ FPDFDest_GetLocationInPage: load_fn!(lib, "FPDFDest_GetLocationInPage"),
+
+ FPDF_StructTree_GetForPage: load_fn!(lib, "FPDF_StructTree_GetForPage"),
+ FPDF_StructTree_Close: load_fn!(lib, "FPDF_StructTree_Close"),
+ FPDF_StructTree_CountChildren: load_fn!(lib, "FPDF_StructTree_CountChildren"),
+ FPDF_StructTree_GetChildAtIndex: load_fn!(lib, "FPDF_StructTree_GetChildAtIndex"),
+ FPDF_StructElement_GetType: load_fn!(lib, "FPDF_StructElement_GetType"),
+ FPDF_StructElement_GetAltText: load_fn!(lib, "FPDF_StructElement_GetAltText"),
+ FPDF_StructElement_GetTitle: load_fn!(lib, "FPDF_StructElement_GetTitle"),
+ FPDF_StructElement_CountChildren: load_fn!(lib, "FPDF_StructElement_CountChildren"),
+ FPDF_StructElement_GetChildAtIndex: load_fn!(lib, "FPDF_StructElement_GetChildAtIndex"),
+ FPDF_StructElement_GetChildMarkedContentID: load_fn!(
+ lib,
+ "FPDF_StructElement_GetChildMarkedContentID"
+ ),
+ FPDF_StructElement_GetMarkedContentID: load_fn!(
+ lib,
+ "FPDF_StructElement_GetMarkedContentID"
+ ),
+ FPDF_StructElement_GetMarkedContentIdCount: load_fn!(
+ lib,
+ "FPDF_StructElement_GetMarkedContentIdCount"
+ ),
+ FPDF_StructElement_GetMarkedContentIdAtIndex: load_fn!(
+ lib,
+ "FPDF_StructElement_GetMarkedContentIdAtIndex"
+ ),
+
_lib: lib,
})
}
diff --git a/crates/pdfium/src/document.rs b/crates/pdfium/src/document.rs
index 221809f..9ef97b8 100644
--- a/crates/pdfium/src/document.rs
+++ b/crates/pdfium/src/document.rs
@@ -6,6 +6,23 @@ pub struct Document {
pub(crate) handle: pdfium_sys::FPDF_DOCUMENT,
}
+/// One entry in the document's outline (bookmarks tree).
+#[derive(Debug, Clone)]
+pub struct OutlineEntry {
+ /// Hierarchy depth, 1-based (top-level entries are level 1).
+ pub level: u8,
+ /// Bookmark title.
+ pub title: String,
+ /// Zero-based page index of the destination, or `None` if the destination
+ /// isn't a page in this document (external link, missing dest, etc).
+ pub page_index: Option<i32>,
+ /// Y coordinate of the destination on the page in PDF user space (origin
+ /// bottom-left), or `None` if the destination doesn't specify one. To
+ /// compare against viewport-space line bboxes (origin top-left) use
+ /// `page_height - y`.
+ pub y: Option<f32>,
+}
+
impl Document {
pub fn page_count(&self) -> i32 {
unsafe { ffi!(FPDF_GetPageCount(self.handle)) }
@@ -22,6 +39,103 @@ impl Document {
_doc: std::marker::PhantomData,
})
}
+
+ /// Walk the document outline (bookmarks). Returns entries in pre-order
+ /// (depth-first), so parents precede their children. Empty when the
+ /// document has no outline.
+ pub fn outline(&self) -> Vec<OutlineEntry> {
+ let mut out = Vec::new();
+ let root = unsafe { ffi!(FPDFBookmark_GetFirstChild(self.handle, std::ptr::null_mut())) };
+ if !root.is_null() {
+ self.walk_bookmark(root, 1, &mut out);
+ }
+ out
+ }
+
+ fn walk_bookmark(
+ &self,
+ bookmark: pdfium_sys::FPDF_BOOKMARK,
+ level: u8,
+ out: &mut Vec<OutlineEntry>,
+ ) {
+ let mut cur = bookmark;
+ while !cur.is_null() {
+ let title = read_bookmark_title(cur);
+ let (page_index, y) = resolve_dest(self.handle, cur);
+ out.push(OutlineEntry {
+ level,
+ title,
+ page_index,
+ y,
+ });
+
+ let child = unsafe { ffi!(FPDFBookmark_GetFirstChild(self.handle, cur)) };
+ if !child.is_null() {
+ self.walk_bookmark(child, level.saturating_add(1), out);
+ }
+
+ cur = unsafe { ffi!(FPDFBookmark_GetNextSibling(self.handle, cur)) };
+ }
+ }
+}
+
+fn read_bookmark_title(bookmark: pdfium_sys::FPDF_BOOKMARK) -> String {
+ let needed =
+ unsafe { ffi!(FPDFBookmark_GetTitle(bookmark, std::ptr::null_mut(), 0)) } as usize;
+ if needed < 2 {
+ return String::new();
+ }
+ // `needed` is byte length including a trailing UTF-16 NUL terminator.
+ let mut buf: Vec<u16> = vec![0; needed / 2];
+ let written = unsafe {
+ ffi!(FPDFBookmark_GetTitle(
+ bookmark,
+ buf.as_mut_ptr() as *mut std::os::raw::c_void,
+ needed as std::os::raw::c_ulong,
+ ))
+ } as usize;
+ if written < 2 {
+ return String::new();
+ }
+ let chars = written / 2;
+ let end = if buf.get(chars - 1) == Some(&0) {
+ chars - 1
+ } else {
+ chars
+ };
+ String::from_utf16_lossy(&buf[..end])
+}
+
+fn resolve_dest(
+ doc: pdfium_sys::FPDF_DOCUMENT,
+ bookmark: pdfium_sys::FPDF_BOOKMARK,
+) -> (Option<i32>, Option<f32>) {
+ let mut dest = unsafe { ffi!(FPDFBookmark_GetDest(doc, bookmark)) };
+ if dest.is_null() {
+ let action = unsafe { ffi!(FPDFBookmark_GetAction(bookmark)) };
+ if !action.is_null() {
+ dest = unsafe { ffi!(FPDFAction_GetDest(doc, action)) };
+ }
+ }
+ if dest.is_null() {
+ return (None, None);
+ }
+ let page_index = unsafe { ffi!(FPDFDest_GetDestPageIndex(doc, dest)) };
+ let page_index = if page_index >= 0 { Some(page_index) } else { None };
+
+ let mut has_x: pdfium_sys::FPDF_BOOL = 0;
+ let mut has_y: pdfium_sys::FPDF_BOOL = 0;
+ let mut has_z: pdfium_sys::FPDF_BOOL = 0;
+ let mut x: f32 = 0.0;
+ let mut y: f32 = 0.0;
+ let mut z: f32 = 0.0;
+ let ok = unsafe {
+ ffi!(FPDFDest_GetLocationInPage(
+ dest, &mut has_x, &mut has_y, &mut has_z, &mut x, &mut y, &mut z
+ ))
+ };
+ let y_out = if ok != 0 && has_y != 0 { Some(y) } else { None };
+ (page_index, y_out)
}
impl Drop for Document {
diff --git a/crates/pdfium/src/lib.rs b/crates/pdfium/src/lib.rs
index 389e9f1..59ec49e 100644
--- a/crates/pdfium/src/lib.rs
+++ b/crates/pdfium/src/lib.rs
@@ -4,15 +4,17 @@ mod error;
mod font;
mod library;
mod page;
+mod struct_tree;
mod text_page;
mod types;
pub use bitmap::Bitmap;
-pub use document::Document;
+pub use document::{Document, OutlineEntry};
pub use error::PdfiumError;
pub use font::{Font, FontType};
pub use library::Library;
pub use page::{ImageBounds, Page, PathObject, PathSegment, SegmentKind, ViewportTransform};
+pub use struct_tree::StructNode;
pub use text_page::{TextChar, TextCharIter, TextPage};
pub use types::*;
diff --git a/packages/python/liteparse/__init__.py b/packages/python/liteparse/__init__.py
index f96f5d0..5b9a6ef 100644
--- a/packages/python/liteparse/__init__.py
+++ b/packages/python/liteparse/__init__.py
@@ -1,5 +1,6 @@
from .parser import LiteParse, search_items
from .types import (
+ ExtractedImage,
LiteParseConfig,
ParseResult,
ParsedPage,
@@ -16,6 +17,7 @@ __all__ = [
"ParsedPage",
"TextItem",
"ScreenshotResult",
+ "ExtractedImage",
"ParseError",
"search_items",
]
diff --git a/packages/python/liteparse/parser.py b/packages/python/liteparse/parser.py
index 22dd98d..9fe4317 100644
--- a/packages/python/liteparse/parser.py
+++ b/packages/python/liteparse/parser.py
@@ -7,6 +7,7 @@ from liteparse._liteparse import LiteParse as _NativeLiteParse
from liteparse._liteparse import search_items as _native_search_items
from .types import (
+ ExtractedImage,
LiteParseConfig,
ParsedPage,
ParseError,
@@ -42,9 +43,19 @@ def _convert_native_result(native_result: Any) -> ParseResult:
text_items=text_items,
)
)
+ images = [
+ ExtractedImage(
+ id=img.id,
+ page=img.page,
+ format=img.format,
+ bytes=img.bytes,
+ )
+ for img in getattr(native_result, "images", [])
+ ]
return ParseResult(
pages=pages,
text=native_result.text,
+ images=images,
)
@@ -76,6 +87,7 @@ class LiteParse:
password: Optional[str] = None,
quiet: Optional[bool] = None,
num_workers: Optional[int] = None,
+ image_mode: Optional[str] = None,
):
"""
Initialize LiteParse parser.
@@ -119,6 +131,8 @@ class LiteParse:
kwargs["quiet"] = quiet
if num_workers is not None:
kwargs["num_workers"] = num_workers
+ if image_mode is not None:
+ kwargs["image_mode"] = image_mode
self._native = _NativeLiteParse(**kwargs)
diff --git a/packages/python/liteparse/types.py b/packages/python/liteparse/types.py
index 40f4fbf..1bbea13 100644
--- a/packages/python/liteparse/types.py
+++ b/packages/python/liteparse/types.py
@@ -29,11 +29,26 @@ class ParsedPage:
text_items: List[TextItem] = field(default_factory=list)
+@dataclass
+class ExtractedImage:
+ """An embedded raster image extracted from a page.
+
+ Populated only when the parser was configured with ``image_mode="embed"``.
+ The ``id`` matches the reference used in the markdown output
+ (e.g. ``![](image_p1_0.png)`` → ``id="p1_0"``).
+ """
+ id: str
+ page: int
+ format: str
+ bytes: bytes
+
+
@dataclass
class ParseResult:
"""Result of parsing a document."""
pages: List[ParsedPage]
text: str
+ images: List[ExtractedImage] = field(default_factory=list)
@property
def num_pages(self) -> int:
diff --git a/packages/python/uv.lock b/packages/python/uv.lock
index 4b1a41e..b1e10f3 100644
--- a/packages/python/uv.lock
+++ b/packages/python/uv.lock
@@ -128,7 +128,7 @@ wheels = [
[[package]]
name = "liteparse"
-version = "2.0.0"
+version = "2.0.3"
source = { editable = "." }
[package.optional-dependencies]