A fast, helpful, and open-source document parser
wip: split logic
Logan Markewichac3291a6ecc5
diff --git a/MARKDOWN_PLAN.md b/MARKDOWN_PLAN.mdindex 5daa78e..a7a3677 100644--- a/MARKDOWN_PLAN.md+++ b/MARKDOWN_PLAN.md@@ -252,7 +252,7 @@ Sensible defaults so users can just pass `-f markdown` and get good output. Deba## Surface area- `crates/liteparse/src/output/markdown.rs` — new.-- `crates/liteparse/src/markdown_layout.rs` — new.+- `crates/liteparse/src/markdown_layout/` — new module directory: `mod.rs` (public re-exports), `blocks.rs`, `inline.rs`, `tables.rs`, `headings.rs`, `lists.rs`, `paragraphs.rs`, `repetition.rs`, `hr.rs`, `classify.rs`.- `crates/liteparse/src/types.rs` — add `Block`, `Span`, `TableBlock`.- `crates/liteparse/src/config.rs` — add `OutputFormat::Markdown`, `MarkdownConfig`.- `crates/liteparse/src/projection.rs` — expose `ProjectedLine` (additive).diff --git a/MARKDOWN_PROGRESS.md b/MARKDOWN_PROGRESS.mdindex cfd5932..73ea9d4 100644--- a/MARKDOWN_PROGRESS.md+++ b/MARKDOWN_PROGRESS.md@@ -2,6 +2,8 @@Tracks build-order progress against [MARKDOWN_PLAN.md](MARKDOWN_PLAN.md). Update as steps complete.+> **2026-06-01 module split — `markdown_layout.rs` → `markdown_layout/`.** The single 3819-line file got broken up into a directory of focused submodules; pure move-only refactor, no logic changes. New layout: `mod.rs` (24 lines, re-exports the 5 external API symbols + `Block`), `blocks.rs` (347 — `Block`, `render_blocks`, `wrap_emphasis`, `paragraph_from_accum`), `classify.rs` (783 — `classify_page_with_filters`, `classify_page`, `is_rotated_line`, orchestration), `headings.rs` (526 — body-size histogram, heading map, struct/outline level resolution, bold-heading heuristics, caption rejection), `tables.rs` (1118 — borderless + ruled-grid table detection, `escape_table_cell`, `merge_table_runs`), `inline.rs` (319 — `SpanStyle`, emphasis rendering, list-item text), `paragraphs.rs` (214 — `ParaAccum`, `continues_paragraph`, `collapse_whitespace`), `repetition.rs` (170 — running header/footer detection), `hr.rs` (132 — `detect_horizontal_rules`), `lists.rs` (96 — `parse_list_marker`), `tests_common.rs` (202, `#[cfg(test)]` only — shared `line`/`page`/`styled_line` fixtures so per-module test modules don't duplicate them). Visibility: only `build_heading_map`, `classify_page_with_filters`, `compute_body_size`, `compute_header_footer_set`, `render_blocks`, and `Block` are `pub` (re-exported from `mod.rs`); everything else is `pub(super)` within the submodule tree. External consumer (`output/markdown.rs`) and `lib.rs` unchanged. 147 lib tests still pass; `cargo check -p liteparse` clean. Companion plan to also split `projection.rs` (4050 lines) deferred — too load-bearing while heading/layout work is still iterating.+>> **2026-05-30 rebuild note.** Branch had been cleaned during a branch switch — `markdown_layout.rs` / `output/markdown.rs` were sitting untracked, the supporting types (`ProjectedLine`, `Rect`, `Anchor::Floating`, `ParsedPage.projected_lines`) didn't exist, `OutputFormat::Markdown` wasn't on the enum, and nothing was wired into `lib.rs` / `output/mod.rs`. Restored from `stash@{0}` (which had the plumbing diff) + reimplemented step 2 (`build_projected_lines` in `projection.rs` derives per-line metadata from the projected items). Tests pass; smoke on `paper.pdf` emits headings + bold + page separators end-to-end via `cargo run -p liteparse -- parse <pdf> --format markdown`. Steps 3–6 in the table below are status-accurate again as of this note.> **2026-05-31 XY-cut layout decomposition.** Replaced the flat x-density histogram column detector with a recursive modified XY-cut. Motivation: the old detector failed on layouts where the gutter is interrupted by a wide element (figures straddling both columns, full-width headings/banners) — every page-1-style case in the eval set, and the open carry-forward "column interleaving on page 1" from 2026-05-30.@@ -66,11 +68,103 @@ Tracks build-order progress against [MARKDOWN_PLAN.md](MARKDOWN_PLAN.md). Update| 7 | Tables — medium (column-alignment) + fallback | ✅ | `Block::Table { header, rows }` and `Block::GridFallback { lines }` added. `detect_tables` runs as a pre-pass over a page's `ProjectedLine`s before per-line classification. `split_cells` slices each line's `spans` into cells when adjacent-span gaps exceed `1.0 × dominant_font_size`. A run qualifies as a table when ≥3 cells per row repeat across ≥2 consecutive rows with column-track alignment (≤1 column allowed to drift outside ±6pt — handles indented sub-group row labels); if row-spacing CV > 0.5 the run is downgraded to `GridFallback`. First row promotes to header only when every cell is bold (synthesizes blank header otherwise). `table_rows_adjacent` deliberately doesn't gate on `region_path` because XY-cut sometimes puts an indented row in a different leaf than the rest of the table — column-alignment + y-gap are strong enough on their own. `looks_like_bold_heading` now rejects lines that look tabular (≥3 spans by `split_cells` OR ≥3 multi-space-separated tokens by text fallback) so multi-line table headers like "Model Method Single Hop Multi Hop ..." stop getting promoted to H3. Smoke on `paper.pdf` (results tables): 8 well-formed pipe tables emitted, "Model Method..." header line now lands as a bold paragraph instead of H3. **Known residuals**: (a) `MEMORYBANK` rows in academic results tables are at x=102.5 vs neighbor rows at x=70 — ALL column tracks shift by ~30pt so the 1-column-tolerance rule can't bridge them; tables still fragment into 2-4 row chunks around those rows. Fix needs ruled-grid detection from PDFium fork (step 9) — won't be solvable from text-only signals. (b) Short header rows like `Category Average` (2 cells) and the full-width `F1 BLEU-1 F1 BLEU-1...` (1 collapsed projection span, single-spaced text) still classify as H3 — text-only heuristics can't distinguish them from real one-line section headings without table-rect context. (c) Header absorption (walking back from a detected table to grab multi-line headers above) deliberately deferred — would need careful per-column concatenation against unaligned multi-line headers; better to wait for fork-driven cell-rect data. || 8 | Header/footer stripping (cross-page repetition) | ✅ | `compute_header_footer_set` collects normalized line text (lowercased, digit-runs collapsed to `#` so `Page 1 of 6`/`Page 2 of 6` match) appearing in the top 12% / bottom 12% page-height band on ≥ 50% of pages (floor of 2). Header and footer bands tracked separately so a heading that doubles as both running header on some pages and body title on others isn't stripped from the body. `format_markdown` builds the set once and passes it to `classify_page_with_filters`, which removes matching lines before table detection and per-line classification. Smoke-tested on `Anonymized rate card.pdf` — repeating `Page N of 6` chrome is gone from output. || 9 | Fork: paths exposed → strong (ruled grid) tables, HR detection, figure clustering | ✅ | **Paths exposed + HR + ruled-grid tables landed 2026-06-01; figure clustering deferred.** No actual PDFium fork modifications were required — every needed FFI symbol was already in `bindings.rs`; only the runtime-loaded `PdfiumBindings` struct in `crates/pdfium-sys/src/dynamic.rs` was missing the 10 path-related getters (`FPDFPageObj_GetMatrix`, `FPDFPath_CountSegments`, `FPDFPathSegment_*`, color/width getters, `FPDFPath_GetDrawMode`). Added all. **9a — pdfium wrapper:** new `PathObject` / `PathSegment` / `SegmentKind` in `crates/pdfium/src/page.rs`; `Page::path_objects(view_box)` iterates `FPDF_PAGEOBJ_PATH` objects, applies the object matrix → page → viewport so all segment points land in viewport coords for direct comparison against `ProjectedLine.bbox`. **9b — liteparse `GraphicPrimitive`:** added `GraphicPrimitive::{Stroke { x1,y1,x2,y2,color,width }, Rect { bbox, fill, stroke }}` to `types.rs` + `ParsedPage.graphics` (#[serde(skip)]). `extract.rs::extract_page_graphics` decomposes paths: filled paths → one `Rect` for the bbox (covers cell backgrounds / banner fills regardless of internal complexity); stroked paths → one `Stroke` per `MoveTo→LineTo` / `LineTo→LineTo` pair (plus the implicit closing stroke when a subpath's close flag is set). `BezierTo` segments advance the current point but don't emit a stroke (curves aren't useful for table/HR detection). **9c-1 — HR:** `detect_horizontal_rules(page)` filters strokes to `|y1-y2| ≤ 2pt`, length ≥ `0.3 × page_width`, stroke-width ≤ 2pt, with an underline-band filter that drops strokes within ±3pt of a text-line baseline whose x-extent matches the line. `Block::HorizontalRule` interleaves with text lines in `classify_page_with_filters` by y-coord via a peekable `hr_iter`; renders as `---`. 6 unit tests. **9c-2 — ruled-grid tables:** `detect_ruled_tables` extracts H/V segments from strokes + stroked Rect edges (so cell-rect-per-cell tables work too), clusters them into grid lines (`TABLE_GRID_CLUSTER_PT=2pt`), finds connected components via union-find on crossing pairs (`TABLE_CROSS_TOLERANCE_PT=3pt`), then per component: clusters distinct row/col coords, requires ≥2×2 cells, assigns text-line centroids to cells, rejects when empty-cell fraction > 30% or coverage > 95% of page (kills page-border rects), promotes the first row to header when all non-empty cells are `all_bold`. Runs *before* the existing borderless detector; `merge_table_runs` keeps both outputs but drops borderless runs whose range overlaps a ruled run (path geometry strictly beats text alignment). 5 unit tests. **9c-3 — figure clustering:** deferred to a later pass; limited value without image extraction (step 11) and the highest-leverage piece of figure detection (feeding figure rects into the XY-cut layout pass to fix the paper.pdf page-1 carry-forward) is now tracked as its own follow-up. **Test count:** 136 lib + 10 integration tests pass. **Smoke-test commands:** `cargo run -p liteparse -- parse <pdf> --format markdown`. Good test docs: `Anonymized rate card.pdf` (rate-card with ruled tables + HR-style dividers; should produce `---` and `|...|` table rows), any tagged invoice / financial statement / spreadsheet export. The `paper.pdf` results tables still have known residuals (MEMORYBANK row indent + short header rows) that the borderless detector handled; need to confirm the ruled detector doesn't regress those — if paper.pdf renders ruled tables incorrectly, the borderless fallback should still catch them via `merge_table_runs`. |-| 10 | Fork: structure tree + outline → highest-priority headings | ⬜ | Requires PDFium fork changes. |-| 11 | Image extraction modes (placeholder vs embed) | ⬜ | |+| 10 | Fork: structure tree + outline → highest-priority headings | ✅ | **Landed 2026-06-01.** As with step 9, no actual PDFium fork edits were needed — all FFI symbols (`FPDFBookmark_*`, `FPDFAction_*`, `FPDFDest_*`, `FPDF_StructTree_*`, `FPDF_StructElement_*`) were already in `bindings.rs`; wired the 21 missing function pointers into `PdfiumBindings` in `crates/pdfium-sys/src/dynamic.rs`. **10a — outline wrapper:** new `OutlineEntry` + `Document::outline()` in `crates/pdfium/src/document.rs` walks the bookmark tree depth-first, resolves each entry's destination (direct `FPDFBookmark_GetDest` falling back through `FPDFBookmark_GetAction` → `FPDFAction_GetDest`), then pulls the destination's page index + y (PDF user space) via `FPDFDest_GetDestPageIndex` + `FPDFDest_GetLocationInPage`. Titles decoded from UTF-16LE. **10b — struct-tree wrapper:** new `crates/pdfium/src/struct_tree.rs::Page::struct_tree(view_box)` returns `Vec<StructNode { role, mcids, bbox, alt_text }>` in pre-order. Implementation pre-scans page objects once into an `mcid → union(bbox)` map (in viewport coords via the page's `ViewportTransform`), then walks the tree calling both `FPDF_StructElement_GetMarkedContentIdCount`/`AtIndex` (modern multi-mcid form) and the legacy `GetMarkedContentID` + `GetChildMarkedContentID` getters so older tagged PDFs aren't missed. Each node's bbox is the union of bboxes for its tagged mcids. **10c — liteparse plumbing:** `OutlineTarget` + `StructNode` types added to `types.rs` (`#[serde(skip)]` so JSON/text output stays byte-identical). `LitePage.struct_nodes` populated in `extract::extract_page_struct_nodes`; threaded through `to_parsed_pages` onto `ParsedPage.struct_nodes`. New `extract::extract_outline(doc)` returns `Vec<OutlineTarget>`. `ParseResult` gains a `pub outline: Vec<OutlineTarget>` field. `parse_input` now opens the document once, pulls outline + pages + OCR-rendered pages inside a scope so `Document` is dropped before any `.await` (PDFium handle isn't `Send`). **10d — markdown classifier integration:** added `struct_heading_level(line, struct_nodes)` and `outline_heading_level(line, page_height, outline, text)` in `markdown_layout.rs`. Heading-promotion priority chain at the per-line classifier: struct-tree H1–H6 (matched via line `mcid` → node role) → outline (y_view = `page_height - y_pdf` within `1.5 × line_height`, normalized lowercased prefix-match either direction) → existing font-size histogram. `classify_page_with_filters` gains a 4th `outline: &[OutlineTarget]` param; `format_markdown` gains a 2nd `outline: &[OutlineTarget]` param and filters per-page before dispatching. **Test count:** 147 lib + 10 integration tests pass; `cargo check -p liteparse-python -p liteparse-wasm` clean. **Smoke on `paper.pdf`:** unchanged — arXiv preprints aren't tagged and don't ship an outline, so the new code is dormant (this is the expected non-regression on untagged docs). **Needs a tagged PDF for real validation:** good candidates are most government/legal forms, accessible academic publisher exports (ACM/IEEE/Springer), modern Word/InDesign exports with "Tagged PDF" enabled, and most invoices generated by enterprise tools. Smoke-test command: `cargo run -p liteparse -- parse <tagged.pdf> --format markdown`. Expected wins: section headings hit the right level even when the doc has weird font-size choices; bookmarks-driven headings pick up the right level on docs that have a TOC but no font-size hierarchy. |+| 11 | Image extraction modes (placeholder vs embed) | ✅ | **Stages 11a + 11b landed 2026-06-01.** **Stage 11a (placeholder mode):** Added `ImageMode` enum (`Off | Placeholder | Embed`, default `Placeholder`) to `config.rs` as a flat `LiteParseConfig.image_mode` field (skipped `MarkdownConfig` wrapper — easier for the language bindings, can add later if we accumulate more knobs). New `ImageRef { id, bbox, obj_index }` in `types.rs`, threaded `LitePage.image_refs` → `ParsedPage.image_refs` (#[serde(skip)] both). `extract::extract_page_image_refs(page, page_number)` reuses `Page::image_bounds(min=25pt, max_coverage=0.9)` from the existing FFI surface — no PDFium changes needed. IDs are page-scoped (`p{n}_{i}`) so they're stable across runs. New `Block::Figure { id, bbox }` rendered as ``. Classifier interleaves figures with text lines by y position (parallel to the existing `hr_iter` machinery via a new `figure_iter` + `emit_figures_before` helper, suppressed inside table y-extents). `classify_page_with_filters` gains a 5th `image_mode` param; `format_markdown` gains a 3rd. CLI: `--image-mode off|placeholder|embed` on `parse`. **Stage 11b (embed mode + bytes sidecar):** added `ExtractedImage { id, page, bbox, format, bytes }` to `types.rs`; `ParseResult.images: Vec<ExtractedImage>` (populated only when `ImageMode::Embed`, empty otherwise — placeholder mode pays no render cost). New `extract::render_page_images` reuses `Page::render_image_object` + the existing `render::encode_png` (made `pub(crate)`) to encode each image as RGBA PNG. `extract_pages_and_images` is the new core entry point; `extract_pages_from_document` is now a one-line wrapper passing `render_images=false` for callers that don't need bytes. CLI gains `--image-output-dir <dir>` which, when set with `--image-mode embed`, writes `image_{id}.png` next to the markdown output. **Bindings:** all three threaded through — pyo3 (`PyExtractedImage` with `bytes` property exposing `PyBytes`, `images` on `PyParseResult`), napi (`JsExtractedImage` with napi `Buffer` for bytes, `images` on `JsParseResult`, `image_mode` added to `JsLiteParseConfig`), wasm (`JsExtractedImage` in the result; bytes serialize as `number[]` for now, callers can wrap with `new Uint8Array(image.bytes)`). Python wrapper `packages/python/liteparse/` updated with matching `ExtractedImage` dataclass + `images` field on `ParseResult` + `image_mode` constructor arg. **Test count:** 147 lib + 10 integration tests still pass. **Smoke on `SMC2486.pdf` page 2** (the only of our local PDFs that surfaces an image object — `paper.pdf`'s figure is a vector cluster, not a raster object): `` lands above the `Status Accepted` heading in reading order; `--image-mode off` drops it cleanly. **Stage 11b smoke-tested:** `cargo run -p liteparse -- parse SMC2486.pdf --format markdown --target-pages 2 --image-mode embed --image-output-dir /tmp/lp-img --quiet` emits `` in markdown and writes the 22.4kB PNG to disk. Python: `LiteParse(image_mode="embed").parse(...).images` returns `[ExtractedImage(id='p2_0', page=2, format='png', bytes=22420)]` end-to-end. **Open items:** `--image-mode` flag on `BatchParseCommand` not added; should be a trivial follow-up. WASM bytes are a `number[]` array rather than a proper `Uint8Array` — works but inefficient; can upgrade by hand-rolling `JsParseResult` construction via `Reflect` + `js_sys::Uint8Array` later. |Steps 1–8 = usable v1. Steps 9–11 = quality bumps for specific doc classes.+> **2026-06-01 step 10 validation pass — police.pdf (Broomfield budget) + expected-dataset/data/who_indicators.pdf.** Wrote `crates/liteparse/examples/inspect_tagged.rs` to summarize outline + struct-tree presence and cross-check outline-title coverage against rendered markdown headings without having to read the whole doc (`cargo run -q -p liteparse --example inspect_tagged -- <pdf>`). First run on `police.pdf` exposed two issues:+> - **Y-coord-out-of-page bookmarks.** Every entry came back as `y_pdf=866` on letter-size 792pt pages → `y_view = -74` → never within tolerance of any line. Either the creator's "top of page" dest type encodes y outside the MediaBox, or pages have a non-standard MediaBox; either way it's common enough to design around. Fix in `outline_heading_level`: when `y_view < 0` OR `y_view > page_height`, drop the spatial check and accept any line on the page that prefix-matches the title. Coverage 0/78 → 77/78 on `police.pdf`.+> - **Short titles latching onto long paragraphs.** Once spatial-check fallback was in, "Nutrition" (a 9-char outline title) prefix-matched a 144-char paragraph in WHO that happened to start with "Nutrition,". Fix: require the matched line to be heading-shaped — line char-count ≤ `max(title*3, 120)` AND fewer than 2 sentence-breaks (`. `). Coverage 143/156 on WHO with no observed false-positive headings.+> - **`struct_tree`: 0/457 pages tagged on `police.pdf`, 0/164 on `who_indicators.pdf`.** Neither doc actually ships a structure tree despite being well-formed PDFs with bookmarks. Validates that the outline source is doing the work; struct-tree source is dormant on these. Need a deliberately-tagged PDF (Word/InDesign export with "Tagged PDF" enabled) to exercise the H1–H6 mcid path end-to-end.+> - **Residual misses on WHO are bookmarks with no body heading.** The 13 remaining unmatched WHO entries ("Morbidity", "Mortality by cause", "Environmental risk factors", "Noncommunicable diseases", "Injuries/harmful traditional practices") are *category labels* that appear in the TOC but not as a rendered line in the body — their bookmarks point to the first sub-indicator's page (which we do match). Not a heuristic failure.+> - **Net behavior on `paper.pdf`:** unchanged (no outline / no struct tree) — verifies the change is dormant when neither source is present.+>+> **2026-06-01 step 10 — outline + structure tree.** Two new high-priority heading sources, additive to the existing font-size histogram so untagged PDFs without an outline (like `paper.pdf`) get bit-identical output. **Outline source**: walks the bookmark tree via the new `Document::outline()` wrapper; per-page entries are matched to projected lines by `|y_view - line.y| ≤ 1.5 × line_height` with a lowercased+collapsed prefix-match either direction (handles "Section 1: Foo" outline vs "Foo" rendered line and vice versa). **Struct-tree source**: per-page `Page::struct_tree(view_box)` returns flattened nodes carrying their role, mcid list, and viewport-space union bbox (derived from a one-pass scan of page-object mcids). Lines whose `mcid` lands in a struct node with role `H1..H6` are promoted at that level — outranks both outline and font-size map. Figure/Table struct nodes are surfaced but not yet consumed (existing detectors handle those passes already). One subtle gotcha to track: `parse_input` now opens the PDFium document once outside the OCR await scope so the `!Send` handle is dropped before any future-suspension point.++> **2026-06-01 opendataloader-bench accuracy pass.** Switched the bench adapter `opendataloader-bench/src/pdf_parser_liteparse.py` from text mode → `output_format="markdown", image_mode="off", ocr_enabled=False` (added matching `image_mode` parameter to `packages/python/liteparse/parser.py` and the `liteparse-python` pyo3 binding so it threads through). Two follow-on fixes after smoke-reading the worst 20 docs by overall score:+>+> - **Bold-style paragraph break in `continues_paragraph`** (`markdown_layout.rs`). PDFs that put section headings in a bold font variant of the body font (Brill-Bold over Brill-Roman, etc.) were losing the heading because no font-size delta separated it from the body. The classifier bold-wrapped the line then merged it into the next paragraph. Fix: when both prev and cur have a uniform style (`line_uniform_style` returns Some), a `bold` mismatch is a paragraph break. Six lines. Bumped MHS 0.320 → 0.487 on the bench corpus.+> - **Multi-line-cell wrap merge in `try_detect_table`** (`markdown_layout.rs`). Borderless tables where one column wraps onto a second baseline (e.g. "Physical or / digital" while neighbour columns stay single-line) were getting fragmented because the second baseline came back as a "partial row" with too few cells. Fix: when `cells.len() < column_count`, if every cell aligns with a known track within `TABLE_TRACK_TOLERANCE_PT` AND vertical centroid distance ≤ 1.5 × line_height, append each wrap cell's text into the matching track of the previous row rather than closing the table run. Roughly 30 lines, including comment. Bumped TEDS 0.323 → 0.384.+>+> **Cumulative bench result vs original (text-mode, public 1.2.1):**+>+> | Metric | Before | After 11a + fixes | Δ |+> |---|---|---|---|+> | NID | 0.866 | 0.882 | +0.016 |+> | TEDS | 0.000 | **0.384** | +0.384 |+> | MHS | 0.000 | **0.488** | +0.488 |+>+> Two metrics went from 0 → mid-pack with ~80 lines of code. liteparse jumps from last place (0.576 overall) to roughly the docling/marker neighbourhood on the OpenDataLoader leaderboard.+>+> **Surprise: Tesseract OCR _hurts_ on this corpus.** Re-ran with `ocr_enabled=True` and TEDS dropped to 0.369, MHS to 0.482, NID barely moved (+0.003), at 160× the per-page latency (0.81s vs 0.005s). Strongly suggests a real bug in `ocr_merge` — OCR-text likely duplicating native tokens or unbalancing the spatial merge — worth a focused investigation later. For now `ocr_enabled=False` is the right bench default and arguably the right production default for clean PDFs.+>+> **2026-06-01 numbered bold-section headings + caption rejection.** Two targeted heuristic fixes in `markdown_layout.rs` after a second bench pass identified MHS=0 patterns:+>+> - **`is_caption_line`** detects `Figure N`, `Fig. N`, `Table N`, `Tab. N`, `Equation N`, `Scheme`, `Chart`, `Plate`, `Photo`, `Algorithm`, `Listing` prefixes followed by a digit / roman numeral / `(N`. Applied in three places: `build_heading_map` (don't let caption sizes inflate the heading map), the font-size promotion site (don't promote captions even when their font matches a heading size), and `looks_like_bold_heading` (don't promote bold body-sized captions). Fixes doc 01030000000027 which previously emitted `# Figure 7. / # Figure 8. / # Figure 9. / # Figures 7-9 show…` as H1 headings.+> - **`looks_like_numbered_bold_heading`** runs inside the list-item branch right after `parse_list_marker` matches an ordered marker. Conditions: post-marker body ≤ 80 chars, no double-sentence punctuation, **every non-marker span in `line.spans` must pass `projection::is_bold_item` and fail `projection::is_mono_item`**, alpha-ratio ≥ 0.5, and a paragraph-break gap above (gap_below deliberately *not* required — the line is often followed by another bold body line such as a bold sub-heading or multi-line title continuation). On match, the line is promoted to a heading at `heading_map.len()+1` (typically H1 when no font-size hierarchy exists, capped at H6) instead of emitting as an ordered list item. Fixes doc 01030000000062 (`1. Shipping as a vector for marine IAS` → `# 1. …`) and doc 01030000000027's `6. Conclusion`.+> - **Span-walking trick**: marker characters often arrive inside the same span as the body text (e.g. one span `"1. Shipping as a vector for marine IAS"`), not as a separate marker span. The check classifies a span as "marker" only when *all* its non-whitespace chars are in `[0-9.()]` — so when the marker is embedded, the whole span goes through the bold check, which correctly returns true. This is why `line.all_bold == true` is the typical hit case but the per-span loop is what handles the rest.+>+> **Bench result on opendataloader-bench/ (200 docs):**+>+> | Metric | Pre-fix (post 11a) | Post-fix | Δ |+> |---|---|---|---|+> | NID | 0.8820 | 0.8819 | -0.0001 (noise) |+> | TEDS | 0.3842 | 0.3842 | 0.0 |+> | MHS | 0.4877 | 0.5093 | **+0.0216** |+> | Overall | 0.7451 | 0.7508 | **+0.0057** |+>+> MHS=0 doc count: 17 → 17 (some new misses appeared because docs that previously had non-zero MHS lost some captions-as-headings hits — net positive though). Doc 01030000000062 went 0 → 0.495 MHS.+>+> **Remaining MHS=0 docs are now dominated by column-interleaving on 2-column layouts** — e.g. doc 01030000000038's heading `6.2. Expectations for Re-Hiring Employees` projects onto the same line as body text from the adjacent column (`they had no plans to re-hire and another 36% said`), so neither the bold-heading nor the numbered-bold-heading rule sees a clean candidate. Same root cause as the long-tracked `paper.pdf` page 1 banner/figure-straddle case. Also surfaces in docs 029, 038, 039, 077, 189 — all decimal section numbers (`5.`, `6.2.`, `9.5.`, `1.5.`, `4.3.2`) on 2-column academic/policy/business reports. Fixing these needs an XY-cut / column-detection pass, not a heading-detection knob.+>+> **Build orientation note:** the bench's `liteparse` Python package is installed from a built wheel pointing at `packages/python/liteparse/_liteparse.cpython-314-darwin.so` via a `.pth` file — `maturin develop` against `crates/liteparse-python/Cargo.toml` builds a *different* `_liteparse` package (under site-packages/`_liteparse/`) that the wrapper doesn't import. To get a rebuilt `.so` to take effect, run `maturin develop --release` from `packages/python/` directly. This wasted ~30 minutes the first iteration when the bench kept reporting unchanged scores against stale code.+>+> **2026-06-01 column-start histogram fallback in XY-cut.** Targets the column-interleaving failure mode behind most remaining MHS=0 docs (decimal section heads on 2-column policy/academic/business reports — docs 029, 038, 039, 077, 189, …). The density-based V-cut search inside `xy_find_best_cut` fails on these layouts because a column-spanning element (wide heading, figure, table) fills the gutter at one y-band, inflating the density at the gutter's x and erasing the valley. Added `xy_find_column_cut` as a *fallback* consulted only when **both** `xy_find_best_cut` calls (H and V) return `None`.+>+> **Algorithm.** Histogram every item's left-edge x with 5pt buckets across the full item span (`min_x..max_x`, not bbox-relative — see the bbox aside below). Cluster contiguous occupied buckets (allowing ≤2-bucket gaps for intra-column drift) into peaks; each peak carries its summed item count. Then apply four conservative gates *in this order*:+> - `XY_COLUMN_MIN_LINES_PER_PEAK = 10` — drop peaks with fewer than 10 items. Indented paragraph-starts and footnote markers produce 3–6 item peaks; real column-left edges have 15+ items in a typical academic page.+> - **Exactly 2 strong peaks remaining.** 3+ peaks means tabular structure (one peak per cell column); we bail rather than slice a table.+> - `XY_COLUMN_PEAK_DOMINANCE = 0.55` — the two peaks together must hold ≥55% of all items. Rules out scattered-x prose layouts (lists, lab manuals, multi-indent docs) where the two tallest peaks are tall only relative to a flat tail.+> - `XY_COLUMN_PEAK_BALANCE_RATIO = 0.4` — smaller-peak / larger-peak count ratio ≥ 0.4. Filters out "single column with one off-x bullet stack" cases (90/10 split → false-positive column cut).+>+> Plus the existing geometric constraint: gap between the two peaks must be ≥ `XY_COLUMN_MIN_GAP_FRACTION = 0.25` × total item-span width. Cut position is the midpoint between the two peaks; returned `CutCandidate` carries `score = 1e9` so it beats every density-derived cut but still loses to `xy_find_banner_cut`'s `f32::INFINITY`.+>+> **Why histogram items, not lines.** The natural first cut was: y-band items into lines, take each line's `x_min`, histogram those. That fails on the *exact* layouts we're trying to fix — items from the left and right columns at the same baseline fold into one "line" whose `x_min` is the left-column edge and whose `x_max` reaches the right-column tail. The right column's left edge is hidden. Switching to per-item left edges lets each column's items contribute to their own peak directly.+>+> **xy_root_bbox aside.** Doc 029 (and a handful of book-spread PDFs) places content at negative user-space x (items at x=-435 with the page reported as 510pt wide). `xy_root_bbox` clamps `min_x.max(0.0)` and caps width to `page_width`, so on those pages half the items live *outside* the returned bbox and `xy_split_bbox` produces a zero-width left child after any V-cut into the negative half. Tried removing the clamp; that fixed doc 029 visibly (clean column reading order, both `# 5. The dynamics` and `# 6. Modeling the dynamics` headings emit) but regressed the bench across the board (NID 0.882→0.857, TEDS 0.384→0.296, MHS roughly flat — net -0.025 overall). Reverted. The column-start histogram still works on doc 029 *despite* the clamped bbox because the histogram operates in item-space and the cut position is in item-coords; downstream item partitioning by centroid still routes items correctly even when the bbox doesn't include the cut. **Status: clamped bbox stays. Future work would need a per-page "items extend past MediaBox" detector that conditionally widens the bbox without changing it for normal docs.**+>+> **Gating evolution (what didn't work).**+> - First version fired whenever V-cut returned `None`. Hijacked tables and spreadsheets where density H-cut would have correctly emitted a row split. Regressed TEDS heavily (0.384→0.266).+> - First peak filter was `min_lines_per_peak=4`, which kept tabular peaks alive. Bumped to 10.+> - First version allowed any number of peaks, picking the widest gap. Tables with 5+ peaks (one per cell column) routinely picked a column-gap that sliced rows in half. Restricted to exactly 2 strong peaks.+> - First version had no dominance or balance gates. Many single-column docs with one footnote stack or one block-quote indent passed peak filtering. Added the 55% dominance + 0.4 balance gates.+>+> **Bench result on opendataloader-bench/ (200 docs):**+>+> | Metric | Pre-fix (post caption + numbered-bold) | Post-fix | Δ |+> |---|---|---|---|+> | NID | 0.8820 | **0.8822** | +0.0002 |+> | TEDS | 0.3842 | **0.3862** | +0.0020 |+> | MHS | 0.5093 | **0.5146** | +0.0053 |+> | Overall | 0.7508 | **0.7516** | +0.0008 |+>+> **Doc-level wins:** doc 01030000000029 (MHS 0 → 0.278) is the most visible — `# 4. Entropy`, `# 5. The dynamics`, `# 6. Modeling the dynamics`, `## Definition 1`, `## Principle 1` all now emit correctly because the body is properly column-split before line-banding.+>+> **Doc-level losses:** doc 01030000000122 (lab manual with sparse bullets in two columns) NID 0.525 → 0.471. Doc 122's histogram does show two roughly-balanced peaks at x=229 and x=457 with 92/17 items — technically passes the gates because the 92 includes a long chunk of straight-through-the-page paragraphs that all happen to start at the same x — but the doc is structurally a single column of mixed bullet/figure/table content where slicing into a "right column" creates phantom reading-order jumps. Not worth tighter gates that would also lose doc 029.+>+> **Carry-forwards still open after this pass:**+> - **Approach #2 (mask spanning items from density projection).** Before computing the V-cut density valley, drop items whose width ≥ ~70% of the region width — those are headings/figures/tables that span columns and are the reason the density valley search fails in the first place. Symmetric to the figure-obstacle code added 2026-06-01 (which masks figures along *H-cuts*). Should catch the 14 still-MHS=0 docs whose layout has a clean gutter once spanning elements are excluded.+> - **Approach #3 (split-at-big-gap inside `build_projected_lines`).** Even if XY-cut fails, the per-leaf y-banding step could split a band whose items have a horizontal gap ≥ ~6–8 × median glyph advance AND ≥ ~30pt absolute. This is what `pdftotext` does. Riskier — justified prose has wide inter-word gaps, especially near inline figures — so should be tightly gated and only as a last resort if #2 doesn't close the remainder.+> - **`xy_root_bbox` negative-x handling.** Book-spread layouts where items have x<0 break bbox-relative reasoning everywhere downstream. A conditional "items extend past MediaBox" detector that widens the bbox only for those pages would unlock the rest of doc 029 and likely a handful of other book-interior PDFs.+>+> **Still-open levers identified, ranked by leverage:**+> - Multi-row table headers (16 docs, TEDS=0): patterns like `Source / Year / Description` stacked across 3 baselines where the wrap-merge fix only bridges 2.+> - Same-font headings with no font/size signal (20 docs, MHS<0.1): no detectable text-only signal short of y-clearance heuristics — risky for false positives.+> - OCR regression in `ocr_merge` — not a markdown problem per se but blocks any real OCR-on benchmark.+>+> **Bench plumbing landed:** `opendataloader-bench/src/pdf_parser_liteparse_placeholder.py` (image_mode=placeholder variant) and `pdf_parser_liteparse_ocr.py` (ocr_enabled=True variant), both registered in `engine_registry.py`. `liteparse-placeholder` confirms image placeholders cost ~0.003 NID — negligible vs the structural gains, so `Placeholder` is the right default.+## Carried-forward itemsThings deferred from earlier steps that need to be revisited:diff --git a/crates/liteparse-napi/src/types.rs b/crates/liteparse-napi/src/types.rsindex 130b633..d8844c7 100644--- a/crates/liteparse-napi/src/types.rs+++ b/crates/liteparse-napi/src/types.rs@@ -1,6 +1,6 @@use napi_derive::napi;-use liteparse::config::{LiteParseConfig, OutputFormat};+use liteparse::config::{ImageMode, LiteParseConfig, OutputFormat};use liteparse::parser::ParseResult;use liteparse::types::{ParsedPage, TextItem};@@ -35,6 +35,10 @@ pub struct JsLiteParseConfig {pub quiet: Option<bool>,/// Number of concurrent OCR workers (default: CPU cores - 1).pub num_workers: Option<u32>,+ /// How to surface raster images in markdown output: "off", "placeholder"+ /// (default — emits `` references with no bytes), or+ /// "embed" (also returns each image's PNG bytes on `images`).+ pub image_mode: Option<String>,}impl JsLiteParseConfig {@@ -80,6 +84,13 @@ impl JsLiteParseConfig {if let Some(v) = self.num_workers {cfg.num_workers = v as usize;}+ if let Some(v) = self.image_mode {+ cfg.image_mode = match v.as_str() {+ "off" | "none" => ImageMode::Off,+ "embed" => ImageMode::Embed,+ _ => ImageMode::Placeholder,+ };+ }cfg}@@ -101,6 +112,11 @@ impl JsLiteParseConfig {password: cfg.password.clone(),quiet: Some(cfg.quiet),num_workers: Some(cfg.num_workers as u32),+ image_mode: Some(match cfg.image_mode {+ ImageMode::Off => "off".to_string(),+ ImageMode::Placeholder => "placeholder".to_string(),+ ImageMode::Embed => "embed".to_string(),+ }),}}}@@ -186,6 +202,16 @@ impl JsParsedPage {pub struct JsParseResult {pub pages: Vec<JsParsedPage>,pub text: String,+ pub images: Vec<JsExtractedImage>,+}++#[napi(object)]+#[derive(Clone)]+pub struct JsExtractedImage {+ pub id: String,+ pub page: u32,+ pub format: String,+ pub bytes: napi::bindgen_prelude::Buffer,}// ---------------------------------------------------------------------------@@ -206,6 +232,16 @@ impl JsParseResult {Self {pages: result.pages.iter().map(JsParsedPage::from_rust).collect(),text: result.text.clone(),+ images: result+ .images+ .iter()+ .map(|img| JsExtractedImage {+ id: img.id.clone(),+ page: img.page,+ format: img.format.clone(),+ bytes: img.bytes.clone().into(),+ })+ .collect(),}}}diff --git a/crates/liteparse-python/src/lib.rs b/crates/liteparse-python/src/lib.rsindex 3f131ca..246b490 100644--- a/crates/liteparse-python/src/lib.rs+++ b/crates/liteparse-python/src/lib.rs@@ -1,7 +1,7 @@use pyo3::prelude::*;use pyo3::types::PyBytes;-use liteparse::config::{LiteParseConfig, OutputFormat};+use liteparse::config::{ImageMode, LiteParseConfig, OutputFormat};use liteparse::types::PdfInput;mod cli;@@ -121,6 +121,8 @@ struct PyParseResult {pages: Vec<PyParsedPage>,#[pyo3(get)]text: String,+ #[pyo3(get)]+ images: Vec<PyExtractedImage>,}#[pymethods]@@ -136,9 +138,10 @@ impl PyParseResult {fn __repr__(&self) -> String {format!(- "ParseResult(pages={}, text_len={})",+ "ParseResult(pages={}, text_len={}, images={})",self.pages.len(),- self.text.len()+ self.text.len(),+ self.images.len())}}@@ -152,6 +155,52 @@ impl PyParseResult {.map(PyParsedPage::from_rust).collect(),text: result.text,+ images: result+ .images+ .into_iter()+ .map(PyExtractedImage::from_rust)+ .collect(),+ }+ }+}++#[pyclass(frozen, from_py_object)]+#[derive(Clone)]+struct PyExtractedImage {+ #[pyo3(get)]+ id: String,+ #[pyo3(get)]+ page: u32,+ #[pyo3(get)]+ format: String,+ bytes_buffer: Vec<u8>,+}++#[pymethods]+impl PyExtractedImage {+ #[getter]+ fn bytes<'py>(&self, py: Python<'py>) -> Bound<'py, PyBytes> {+ PyBytes::new(py, &self.bytes_buffer)+ }++ fn __repr__(&self) -> String {+ format!(+ "ExtractedImage(id='{}', page={}, format='{}', bytes_len={})",+ self.id,+ self.page,+ self.format,+ self.bytes_buffer.len()+ )+ }+}++impl PyExtractedImage {+ fn from_rust(img: liteparse::types::ExtractedImage) -> Self {+ Self {+ id: img.id,+ page: img.page,+ format: img.format,+ bytes_buffer: img.bytes,}}}@@ -277,6 +326,7 @@ impl LiteParse {password = None,quiet = None,num_workers = None,+ image_mode = None,))]fn new(ocr_language: Option<String>,@@ -291,6 +341,7 @@ impl LiteParse {password: Option<String>,quiet: Option<bool>,num_workers: Option<usize>,+ image_mode: Option<String>,) -> PyResult<Self> {let mut cfg = LiteParseConfig::default();if let Some(v) = ocr_language {@@ -333,6 +384,13 @@ impl LiteParse {if let Some(v) = num_workers {cfg.num_workers = v;}+ if let Some(v) = image_mode {+ cfg.image_mode = match v.as_str() {+ "off" | "none" => ImageMode::Off,+ "embed" => ImageMode::Embed,+ _ => ImageMode::Placeholder,+ };+ }let inner = liteparse::parser::LiteParse::new(cfg.clone());let runtime = tokio::runtime::Runtime::new()@@ -436,6 +494,7 @@ fn _liteparse(m: &Bound<'_, PyModule>) -> PyResult<()> {m.add_class::<LiteParse>()?;m.add_class::<PyLiteParseConfig>()?;m.add_class::<PyParseResult>()?;+ m.add_class::<PyExtractedImage>()?;m.add_class::<PyParsedPage>()?;m.add_class::<PyTextItem>()?;m.add_class::<PyScreenshotResult>()?;diff --git a/crates/liteparse-wasm/src/lib.rs b/crates/liteparse-wasm/src/lib.rsindex dbb1812..9381e4d 100644--- a/crates/liteparse-wasm/src/lib.rs+++ b/crates/liteparse-wasm/src/lib.rs@@ -156,6 +156,19 @@ struct JsParsedPage<'a> {struct JsParseResult<'a> {pages: Vec<JsParsedPage<'a>>,text: &'a str,+ images: Vec<JsExtractedImage<'a>>,+}++#[derive(Serialize)]+#[serde(rename_all = "camelCase")]+struct JsExtractedImage<'a> {+ id: &'a str,+ page: u32,+ format: &'a str,+ /// Serialized as a JS `number[]`. Callers that want a Uint8Array can+ /// wrap with `new Uint8Array(image.bytes)`. (Could be upgraded to a real+ /// Uint8Array later by switching to a hand-rolled to_value path.)+ bytes: &'a [u8],}// ---------------------------------------------------------------------------@@ -332,9 +345,20 @@ impl LiteParse {}).collect();+ let js_images: Vec<JsExtractedImage> = result+ .images+ .iter()+ .map(|img| JsExtractedImage {+ id: &img.id,+ page: img.page,+ format: &img.format,+ bytes: &img.bytes,+ })+ .collect();let js_result = JsParseResult {pages: js_pages,text: &result.text,+ images: js_images,};serde_wasm_bindgen::to_value(&js_result)diff --git a/crates/liteparse/src/config.rs b/crates/liteparse/src/config.rsindex ca1ce0a..21d0a07 100644--- a/crates/liteparse/src/config.rs+++ b/crates/liteparse/src/config.rs@@ -27,6 +27,28 @@ pub struct LiteParseConfig {pub quiet: bool,/// Number of concurrent OCR workers. Defaults to (number of CPU cores - 1), minimum 1.pub num_workers: usize,+ /// Controls how raster images are surfaced in markdown output. Has no+ /// effect on JSON / text outputs.+ pub image_mode: ImageMode,+}++/// Image handling for the markdown emitter.+///+/// * `Off` — strip image references entirely.+/// * `Placeholder` (default) — emit `` references in+/// reading order at each image's y position, but do **not** extract or+/// return pixel bytes. Keeps response size small while letting the LLM see+/// where figures live in the document.+/// * `Embed` — same references, plus bytes returned via `ParseResult.images`.+/// Opt-in because pixel bytes can dwarf the text payload on image-heavy+/// PDFs. (Bytes plumbing lands in stage 11b — current variant is parsed but+/// behaves like `Placeholder` until then.)+#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]+#[serde(rename_all = "lowercase")]+pub enum ImageMode {+ Off,+ Placeholder,+ Embed,}/// Supported output formats.@@ -53,6 +75,7 @@ impl Default for LiteParseConfig {password: None,quiet: false,num_workers: default_num_workers(),+ image_mode: ImageMode::Placeholder,}}}diff --git a/crates/liteparse/src/extract.rs b/crates/liteparse/src/extract.rsindex 402f97b..45fbd4d 100644--- a/crates/liteparse/src/extract.rs+++ b/crates/liteparse/src/extract.rs@@ -1,5 +1,9 @@use crate::error::LiteParseError;-use crate::types::{GraphicPrimitive, Page as LitePage, PdfInput, Rect, TextItem};+use crate::render::encode_png;+use crate::types::{+ ExtractedImage, GraphicPrimitive, ImageRef, OutlineTarget, Page as LitePage, PdfInput, Rect,+ StructNode, TextItem,+};use pdfium::{Document, Font, FontType, Library, Page, PathObject, RectF, SegmentKind, TextPage};/// Open a PDF from path or bytes with an optional password.@@ -31,8 +35,23 @@ pub(crate) fn extract_pages_from_document(target_pages: Option<&[u32]>,max_pages: usize,) -> Result<Vec<LitePage>, LiteParseError> {+ Ok(extract_pages_and_images(document, target_pages, max_pages, false)?.0)+}++/// Same as `extract_pages_from_document` but optionally also renders every+/// raster image object to PNG bytes (when `render_images = true`). Returned+/// `ExtractedImage`s carry the same ids the markdown emitter will reference,+/// so callers can match them up by id. When `render_images = false` the+/// returned image vec is always empty.+pub(crate) fn extract_pages_and_images(+ document: &Document,+ target_pages: Option<&[u32]>,+ max_pages: usize,+ render_images: bool,+) -> Result<(Vec<LitePage>, Vec<ExtractedImage>), LiteParseError> {let page_count = document.page_count();let mut pages = Vec::new();+ let mut images: Vec<ExtractedImage> = Vec::new();for page_index in 0..page_count {let page_number = page_index as u32 + 1;@@ -57,6 +76,12 @@ pub(crate) fn extract_pages_from_document(});let text_items = extract_page_text_items(&page, &text_page, &view_box)?;let graphics = extract_page_graphics(&page, &view_box);+ let struct_nodes = extract_page_struct_nodes(&page, &view_box);+ let image_refs = extract_page_image_refs(&page, page_number);++ if render_images && !image_refs.is_empty() {+ images.extend(render_page_images(&page, page_number, &image_refs));+ }pages.push(LitePage {page_number: page_number as usize,@@ -64,10 +89,48 @@ pub(crate) fn extract_pages_from_document(page_height: page.height(),text_items,graphics,+ struct_nodes,+ image_refs,});}- Ok(pages)+ Ok((pages, images))+}++/// Walk the document outline (bookmarks). Returns entries in pre-order.+/// Empty when the PDF has no outline.+pub(crate) fn extract_outline(document: &Document) -> Vec<OutlineTarget> {+ document+ .outline()+ .into_iter()+ .filter_map(|e| {+ Some(OutlineTarget {+ level: e.level,+ title: e.title,+ page_index: e.page_index?,+ y_pdf: e.y,+ })+ })+ .collect()+}++/// Walk the page's structure tree (tagged PDFs). Returns nodes in pre-order;+/// empty when the page is untagged.+fn extract_page_struct_nodes(page: &Page, view_box: &RectF) -> Vec<StructNode> {+ page.struct_tree(view_box)+ .into_iter()+ .map(|n| StructNode {+ role: n.role,+ mcids: n.mcids,+ bbox: n.bbox.map(|b| Rect {+ x: b.left,+ y: b.top,+ width: b.right - b.left,+ height: b.bottom - b.top,+ }),+ alt_text: n.alt_text,+ })+ .collect()}/// Extract raw text items and print each page as a JSON-line object to stdout.@@ -137,6 +200,76 @@ fn should_skip_invisible(text_page: &TextPage, char_count: i32) -> bool {invisible_ratio < 0.3}+/// Minimum image extent (in PDF points) below which we ignore the image+/// object. Filters out hairline rasterized rules, icons embedded in glyphs,+/// and other sub-25pt fragments that would otherwise pollute the figure+/// stream. Matches the threshold used by `ocr_merge::has_images`.+const IMAGE_MIN_SIZE_PT: f32 = 25.0;++/// Max fraction of the page each axis can cover. Drops full-page background+/// images (scanned pages, watermarks).+const IMAGE_MAX_COVERAGE: f32 = 0.9;++/// Render every image referenced in `refs` to PNG bytes using+/// `Page::render_image_object`. Returns one `ExtractedImage` per ref. Used by+/// the parser only when `ImageMode::Embed` is configured — otherwise the+/// extraction loop skips this entirely. Failures for individual images are+/// silently dropped (a malformed embedded image shouldn't fail the whole+/// parse).+pub(crate) fn render_page_images(+ page: &Page,+ page_number: u32,+ refs: &[ImageRef],+) -> Vec<ExtractedImage> {+ let mut out = Vec::with_capacity(refs.len());+ for r in refs {+ let bmp = match page.render_image_object(r.obj_index) {+ Ok(b) => b,+ Err(_) => continue,+ };+ let w = bmp.width().max(0) as u32;+ let h = bmp.height().max(0) as u32;+ if w == 0 || h == 0 {+ continue;+ }+ let rgba = bmp.to_rgba();+ let png = match encode_png(&rgba, w, h) {+ Ok(p) => p,+ Err(_) => continue,+ };+ out.push(ExtractedImage {+ id: r.id.clone(),+ page: page_number,+ bbox: r.bbox.clone(),+ format: "png".into(),+ bytes: png,+ });+ }+ out+}++/// Walk image objects on a page and return a stable per-page `ImageRef` for+/// each one. `obj_index` is the index among image-typed page objects (not all+/// page objects), so a later embed pass can pull pixel bytes via+/// `Page::render_image_object`. IDs are scoped to the page number so they+/// remain stable across runs.+fn extract_page_image_refs(page: &Page, page_number: u32) -> Vec<ImageRef> {+ page.image_bounds(IMAGE_MIN_SIZE_PT, IMAGE_MAX_COVERAGE)+ .into_iter()+ .enumerate()+ .map(|(i, b)| ImageRef {+ id: format!("p{}_{}", page_number, i),+ bbox: Rect {+ x: b.x,+ y: b.y,+ width: b.width,+ height: b.height,+ },+ obj_index: i,+ })+ .collect()+}+/// Extract simplified vector graphics from a page. We keep only what the/// markdown layout pass cares about:/// - filled paths → a single bounding `Rect` (covers cell backgrounds /diff --git a/crates/liteparse/src/main.rs b/crates/liteparse/src/main.rsindex 5c9e8d0..136c54f 100644--- a/crates/liteparse/src/main.rs+++ b/crates/liteparse/src/main.rs@@ -89,6 +89,21 @@ struct ParseCommand {/// Number of concurrent OCR workers (default: CPU cores - 1)#[arg(long)]num_workers: Option<usize>,++ /// How to surface raster images in markdown output:+ /// `off` strips them, `placeholder` (default) emits ``+ /// references in reading order, `embed` extracts each image's PNG bytes+ /// and writes them next to the markdown output when `--image-output-dir`+ /// is set.+ #[arg(long, default_value = "placeholder")]+ image_mode: String,++ /// Directory to write embedded images to when `--image-mode embed` is+ /// set. Each image is written as `image_{id}.png` to match the+ /// references in the markdown output. Has no effect for other image+ /// modes. Created if missing.+ #[arg(long)]+ image_output_dir: Option<String>,}#[derive(Args, Debug)]@@ -197,6 +212,19 @@ fn parse_output_format(s: &str) -> Result<OutputFormat, String> {}}+fn parse_image_mode(s: &str) -> Result<liteparse::config::ImageMode, String> {+ use liteparse::config::ImageMode;+ match s.to_lowercase().as_str() {+ "off" | "none" => Ok(ImageMode::Off),+ "placeholder" => Ok(ImageMode::Placeholder),+ "embed" => Ok(ImageMode::Embed),+ _ => Err(format!(+ "unknown image-mode '{}', expected 'off', 'placeholder', or 'embed'",+ s+ )),+ }+}+#[tokio::main]async fn main() -> Result<(), Box<dyn std::error::Error>> {let cli = Cli::parse();@@ -204,6 +232,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {match cli.command {Commands::Parse(cmd) => {let format = parse_output_format(&cmd.format)?;+ let image_mode = parse_image_mode(&cmd.image_mode)?;let mut config = LiteParseConfig {ocr_language: cmd.ocr_language,@@ -217,6 +246,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {password: cmd.password,quiet: cmd.quiet,ocr_server_url: cmd.ocr_server_url,+ image_mode,..Default::default()};if let Some(n) = cmd.num_workers {@@ -230,6 +260,22 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {OutputFormat::Text => text::format_text(&result.pages),OutputFormat::Markdown => result.text.clone(),};+ if let Some(dir) = cmd.image_output_dir.as_deref()+ && !result.images.is_empty()+ {+ std::fs::create_dir_all(dir)?;+ for img in &result.images {+ let path = format!("{}/image_{}.{}", dir, img.id, img.format);+ std::fs::write(&path, &img.bytes)?;+ }+ if !cmd.quiet {+ eprintln!(+ "[liteparse] wrote {} image(s) to {}",+ result.images.len(),+ dir+ );+ }+ }match cmd.output {Some(path) => {diff --git a/crates/liteparse/src/markdown_layout.rs b/crates/liteparse/src/markdown_layout.rsdeleted file mode 100644index c7d255a..0000000--- a/crates/liteparse/src/markdown_layout.rs+++ /dev/null@@ -1,3369 +0,0 @@-//! Block classification for the markdown emitter.-//!-//! Consumes `ProjectedLine` entries from each `ParsedPage` and groups them into-//! a sequence of `Block`s: headings, paragraphs, and (for now) raw lines that-//! don't fit a recognized shape. Tables, lists, and code blocks land in later-//! build-order steps.--use crate::projection::{is_bold_item, is_italic_item, is_mono_item};-use crate::types::{Anchor, GraphicPrimitive, ParsedPage, ProjectedLine, TextItem};--/// Maximum stroke thickness (or |y1-y2|) for a stroke to count as a candidate HR.-/// Thicker shapes are filled rects, not rules.-const HR_MAX_THICKNESS_PT: f32 = 2.0;--/// Minimum fraction of page width a horizontal stroke must span to count as an HR.-/// Shorter strokes are typically table borders, list bullets, or inline marks.-const HR_MIN_WIDTH_FRACTION: f32 = 0.3;--/// Vertical tolerance (points) for treating a stroke as "underlining" the-/// nearest text line. Strokes within this band of a text line's baseline are-/// dropped — they're underlines, not rules.-const HR_UNDERLINE_PROXIMITY_PT: f32 = 3.0;--/// Detect horizontal rules from a page's vector graphics.-///-/// Returns the y-coordinates (viewport space) of accepted HRs, sorted ascending.-/// An HR is a roughly horizontal stroke that spans at least-/// `HR_MIN_WIDTH_FRACTION` of the page width, is thinner than-/// `HR_MAX_THICKNESS_PT`, and does not sit on the baseline of any text line-/// (which would make it an underline).-pub fn detect_horizontal_rules(page: &ParsedPage) -> Vec<f32> {- if page.graphics.is_empty() || page.page_width <= 0.0 {- return Vec::new();- }- let min_width = page.page_width * HR_MIN_WIDTH_FRACTION;- let mut ys: Vec<f32> = Vec::new();-- for g in &page.graphics {- let GraphicPrimitive::Stroke {- x1,- y1,- x2,- y2,- width,- ..- } = g- else {- continue;- };- let (x1, y1, x2, y2, width) = (*x1, *y1, *x2, *y2, *width);- let dy = (y1 - y2).abs();- let dx = (x1 - x2).abs();- if dy > HR_MAX_THICKNESS_PT || width > HR_MAX_THICKNESS_PT {- continue;- }- if dx < min_width {- continue;- }- let y = (y1 + y2) * 0.5;- let xmin = x1.min(x2);- let xmax = x1.max(x2);-- // Drop if this stroke sits on a text-line baseline — it's an underline,- // not a divider.- let is_underline = page.projected_lines.iter().any(|line| {- let bottom = line.bbox.y + line.bbox.height;- (y - bottom).abs() < HR_UNDERLINE_PROXIMITY_PT- && xmin >= line.bbox.x - 2.0- && xmax <= line.bbox.x + line.bbox.width + 2.0- });- if is_underline {- continue;- }- ys.push(y);- }-- // Sort + dedup near-duplicates (some PDFs draw the same rule twice).- ys.sort_by(|a, b| a.total_cmp(b));- ys.dedup_by(|a, b| (*a - *b).abs() < 1.0);- ys-}--/// Coarse block representation. Intentionally minimal — extended as later-/// build-order steps land (tables, figures).-#[derive(Debug, Clone)]-pub enum Block {- Heading {- level: u8,- text: String,- },- Paragraph {- text: String,- bold: bool,- italic: bool,- },- ListItem {- ordered: bool,- marker: String,- level: u8,- text: String,- bold: bool,- italic: bool,- },- /// Fenced code block — content rendered between triple-backtick fences.- /// Each entry in `lines` is one source line; preserved as-is (only trailing- /// whitespace stripped) so indentation survives.- CodeBlock {- lines: Vec<String>,- },- /// Confident table emitted as a markdown pipe table. `header` is `None`- /// when the first row didn't qualify (e.g. wasn't bold and the table mode- /// can't otherwise distinguish it).- Table {- header: Option<Vec<String>>,- rows: Vec<Vec<String>>,- },- /// Tabular-looking region we couldn't classify confidently — rendered- /// verbatim inside a fenced block to preserve visual structure for the- /// downstream LLM. Strictly better than emitting a mangled pipe table.- GridFallback {- lines: Vec<String>,- },- /// A horizontal rule detected from a long thin horizontal stroke in the- /// page's vector graphics (e.g. divider line between sections).- HorizontalRule,- Blank,-}--/// Minimum cells per row for a region to qualify as a table.-const TABLE_MIN_COLUMNS: usize = 3;--/// Minimum consecutive rows for a region to qualify as a table.-const TABLE_MIN_ROWS: usize = 2;--/// Gap between adjacent spans (in multiples of dominant font size) above which-/// we treat the gap as a cell boundary.-const TABLE_CELL_GAP_FONT_MULTIPLIER: f32 = 1.0;--/// Tolerance (points) for matching a cell's start-x to an existing column-/// track when extending a candidate table run.-const TABLE_TRACK_TOLERANCE_PT: f32 = 6.0;--/// Maximum vertical gap between consecutive table rows, expressed in multiples-/// of the line height. Looser than the paragraph rule because table rows often-/// have more vertical padding than prose lines.-const TABLE_ROW_GAP_MULTIPLIER: f32 = 2.5;--/// Maximum coefficient-of-variation for row spacing within a confident table-/// (rejecting irregular spacing that's more likely prose or a footer block).-const TABLE_ROW_SPACING_MAX_CV: f32 = 0.5;--/// One cell within a tabular row: contributing spans aggregated to text and-/// its leftmost x position, used to align cells across rows into column-/// "tracks".-#[derive(Debug, Clone)]-struct TableCell {- start_x: f32,- /// Right edge of the cell (x of the last span's right). Used by- /// `recover_merged_cell` to detect cells that straddle two column tracks- /// when the projection merged two adjacent words into one span.- end_x: f32,- text: String,- bold: bool,-}--/// A contiguous tabular run: line indices `[start, end)` plus the detected-/// rows. Used so the line-classifier can skip the consumed range and so-/// fallback rendering can reach back for the original projected text.-#[derive(Debug, Clone)]-struct TableRun {- start: usize,- end: usize,- block: Block,-}--/// Split a `ProjectedLine`'s spans into cells. A gap larger than-/// `TABLE_CELL_GAP_FONT_MULTIPLIER × font_size` between adjacent spans starts-/// a new cell; otherwise spans join into the same cell with a single space.-fn split_cells(line: &ProjectedLine) -> Vec<TableCell> {- // Skip whitespace-only spans before computing gaps — leading/trailing- // empty items would otherwise add spurious cell boundaries.- let mut spans: Vec<&TextItem> = line- .spans- .iter()- .filter(|s| !s.text.trim().is_empty())- .collect();- spans.sort_by(|a, b| a.x.total_cmp(&b.x));- if spans.is_empty() {- return Vec::new();- }- let font_size = if line.dominant_font_size > 0.0 {- line.dominant_font_size- } else {- line.bbox.height.max(1.0)- };- let gap_threshold = font_size * TABLE_CELL_GAP_FONT_MULTIPLIER;-- let mut cells: Vec<TableCell> = Vec::new();- let mut current_text = String::new();- let mut current_start = spans[0].x;- let mut current_bold_chars: usize = 0;- let mut current_total_chars: usize = 0;- let mut prev_right = spans[0].x;-- for (i, span) in spans.iter().enumerate() {- let gap = span.x - prev_right;- let break_cell = i > 0 && gap > gap_threshold;- if break_cell {- let bold = current_total_chars > 0 && current_bold_chars * 2 > current_total_chars;- cells.push(TableCell {- start_x: current_start,- end_x: prev_right,- text: collapse_whitespace(current_text.trim()),- bold,- });- current_text.clear();- current_start = span.x;- current_bold_chars = 0;- current_total_chars = 0;- }- if !current_text.is_empty() && !current_text.ends_with(' ') {- current_text.push(' ');- }- current_text.push_str(&span.text);- let n = span.text.chars().count();- current_total_chars += n;- if is_bold_span(span) {- current_bold_chars += n;- }- prev_right = span.x + span.width.max(0.0);- }- if !current_text.trim().is_empty() {- let bold = current_total_chars > 0 && current_bold_chars * 2 > current_total_chars;- cells.push(TableCell {- start_x: current_start,- end_x: prev_right,- text: collapse_whitespace(current_text.trim()),- bold,- });- }- cells-}--/// When a candidate row has fewer cells than the established column count,-/// look for cells whose x-range straddles multiple column tracks (likely two-/// or more adjacent words that PDFium merged into a single text run) and-/// split each on internal whitespace at the boundaries nearest to the-/// straddled tracks.-///-/// Returns the patched cells if every short cell could be cleanly split to-/// recover `tracks.len()` cells total; otherwise `None`.-fn recover_merged_cell(mut cells: Vec<TableCell>, tracks: &[f32]) -> Option<Vec<TableCell>> {- let target = tracks.len();- if cells.len() >= target {- return None;- }- // Repeatedly find the cell that straddles the most tracks (≥2) and split- // it. Each iteration strictly grows `cells.len()`, so termination is- // guaranteed; if no cell straddles ≥2 tracks before we hit the target,- // recovery fails.- while cells.len() < target {- let mut best_i: Option<usize> = None;- let mut best_count: usize = 1;- let mut best_contained: Vec<f32> = Vec::new();- for (i, cell) in cells.iter().enumerate() {- let contained: Vec<f32> = tracks- .iter()- .copied()- .filter(|t| {- *t >= cell.start_x - TABLE_TRACK_TOLERANCE_PT- && *t <= cell.end_x + TABLE_TRACK_TOLERANCE_PT- })- .collect();- if contained.len() > best_count {- best_count = contained.len();- best_i = Some(i);- best_contained = contained;- }- }- let Some(i) = best_i else {- return None;- };- let cell = cells[i].clone();- let chars: Vec<char> = cell.text.trim().chars().collect();- let n = chars.len();- if n == 0 || best_contained.len() < 2 {- return None;- }- let text_width = (cell.end_x - cell.start_x).max(1.0);- // For each track after the first, pick the whitespace boundary in- // `chars` whose linearly-interpolated x is closest to the track.- let mut split_indices: Vec<usize> = Vec::new();- for t in best_contained.iter().skip(1) {- let mut best: Option<(usize, f32)> = None;- for (k, ch) in chars.iter().enumerate() {- if !ch.is_whitespace() {- continue;- }- if split_indices.contains(&k) {- continue;- }- let frac = k as f32 / n as f32;- let x = cell.start_x + frac * text_width;- let d = (x - t).abs();- if best.as_ref().is_none_or(|b| d < b.1) {- best = Some((k, d));- }- }- let (k, _) = best?;- split_indices.push(k);- }- split_indices.sort();- // Build the split pieces.- let mut pieces: Vec<String> = Vec::new();- let mut prev = 0usize;- for k in &split_indices {- let piece: String = chars[prev..*k]- .iter()- .collect::<String>()- .trim()- .to_string();- if piece.is_empty() {- return None;- }- pieces.push(piece);- prev = *k;- }- let last: String = chars[prev..].iter().collect::<String>().trim().to_string();- if last.is_empty() {- return None;- }- pieces.push(last);- if pieces.len() != best_contained.len() {- return None;- }- // Synthesize new TableCells aligned with each track.- let mut new_cells: Vec<TableCell> = Vec::with_capacity(pieces.len());- for (p, piece) in pieces.iter().enumerate() {- let start_x = if p == 0 {- cell.start_x- } else {- best_contained[p]- };- let end_x = if p + 1 < best_contained.len() {- (best_contained[p + 1] - 1.0).max(start_x)- } else {- cell.end_x- };- new_cells.push(TableCell {- start_x,- end_x,- text: piece.clone(),- bold: cell.bold,- });- }- cells.remove(i);- for (offset, c) in new_cells.into_iter().enumerate() {- cells.insert(i + offset, c);- }- }- if cells.len() == target {- Some(cells)- } else {- None- }-}--/// Lightweight bold detection on a raw `TextItem`. Delegates to-/// `projection::is_bold_item` so per-span and per-line logic stay in sync.-fn is_bold_span(s: &TextItem) -> bool {- is_bold_item(s)-}--/// Per-span style flags used by the inline-emphasis renderer.-#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]-struct SpanStyle {- bold: bool,- italic: bool,- mono: bool,-}--impl SpanStyle {- fn from_item(item: &TextItem) -> Self {- SpanStyle {- bold: is_bold_item(item),- italic: is_italic_item(item),- mono: is_mono_item(item),- }- }-- fn is_plain(self) -> bool {- !self.bold && !self.italic && !self.mono- }-}--/// Escape characters that would otherwise be interpreted as markdown emphasis.-/// Deliberately narrow: only `*`, `_`, and backslash. Aggressive escaping-/// (`#`, `[`, backticks, etc.) breaks more output than it saves in practice —-/// pymupdf4llm takes the same conservative stance.-fn escape_inline(s: &str) -> String {- let mut out = String::with_capacity(s.len());- for c in s.chars() {- match c {- '\\' | '*' | '_' => {- out.push('\\');- out.push(c);- }- _ => out.push(c),- }- }- out-}--/// Wrap `inner` with the markdown markers for `style`. Mono wins over bold/italic:-/// inline code (`` `…` ``) doesn't compose with emphasis in CommonMark, so when-/// a span is mono we drop the `**/*` wrap. Bold + italic → `***…***`.-fn apply_style(inner: &str, style: SpanStyle) -> String {- if style.mono {- // Use backticks; if inner already contains backticks, switch to a- // longer fence (pair of backticks plus a space buffer) per CommonMark.- if inner.contains('`') {- return format!("`` {} ``", inner);- }- return format!("`{}`", inner);- }- match (style.bold, style.italic) {- (true, true) => format!("***{}***", inner),- (true, false) => format!("**{}**", inner),- (false, true) => format!("*{}*", inner),- (false, false) => inner.to_string(),- }-}--/// Render a `ProjectedLine` to markdown with per-span emphasis. Adjacent-/// same-style spans are merged into a single emphasis run; whitespace between-/// spans is preserved as one space (the underlying projection output already-/// has the right inter-word spacing baked into span text).-///-/// Per-line shortcut: when every non-whitespace span shares the same style,-/// emit one outer wrap around the collapsed line text instead of run-by-run-/// (matches pymupdf4llm; avoids `**foo** **bar** **baz**` noise on uniformly-/// styled lines).-///-/// Falls back to `collapse_whitespace(line.text)` when the line has no usable-/// spans (e.g. OCR-only lines where TextItem styling isn't populated).-fn render_line_inline(line: &ProjectedLine) -> String {- let spans: Vec<&TextItem> = line- .spans- .iter()- .filter(|s| !s.text.trim().is_empty())- .collect();- if spans.is_empty() {- return collapse_whitespace(&line.text);- }-- // Sort spans by x so we render in visual reading order regardless of- // extraction order. Stable so equal-x spans keep their original sequence.- let mut spans = spans;- spans.sort_by(|a, b| a.x.total_cmp(&b.x));-- let styles: Vec<SpanStyle> = spans.iter().map(|s| SpanStyle::from_item(s)).collect();-- // Per-line shortcut.- let uniform = styles.iter().all(|s| *s == styles[0]);- if uniform {- let joined = collapse_whitespace(&line.text);- if joined.is_empty() {- return joined;- }- let escaped = escape_inline(&joined);- if styles[0].is_plain() {- return escaped;- }- return apply_style(&escaped, styles[0]);- }-- // Group consecutive spans by style. Within a group, span texts join with- // a single space (we lose intra-group spacing precision; acceptable).- let mut out = String::new();- let mut i = 0;- while i < spans.len() {- let style = styles[i];- let mut j = i + 1;- while j < spans.len() && styles[j] == style {- j += 1;- }- let mut group_text = String::new();- for k in i..j {- if !group_text.is_empty() && !group_text.ends_with(' ') {- group_text.push(' ');- }- group_text.push_str(spans[k].text.trim());- }- let group_text = collapse_whitespace(&group_text);- let escaped = escape_inline(&group_text);- let rendered = if style.is_plain() {- escaped- } else {- apply_style(&escaped, style)- };- if !out.is_empty() && !out.ends_with(' ') {- out.push(' ');- }- out.push_str(&rendered);- i = j;- }- out-}--/// Render the text portion of a list item with per-span emphasis. The marker-/// itself isn't included in the output (the renderer handles it separately).-///-/// When the line is uniformly styled we wrap the marker-stripped `rest` with-/// the line's style — this avoids the awkward emphasis-marker mismatch we'd-/// hit if we naively stripped a leading bullet out of an already-wrapped-/// rendered line (`**• item**` → `** item**`).-///-/// When the line is mixed-style we render the full line via the inline pipeline-/// and then best-effort-strip the marker prefix (with optional emphasis wrap-/// around it). On any failure we fall back to plain escaped `rest`.-fn render_list_item_text(line: &ProjectedLine, marker: &str, rest: &str) -> String {- if let Some(style) = line_uniform_style(line) {- let plain = collapse_whitespace(rest);- let escaped = escape_inline(&plain);- return if style.is_plain() {- escaped- } else {- apply_style(&escaped, style)- };- }- let full = render_line_inline(line);- if let Some(stripped) = strip_leading_marker_from_inline(&full, marker) {- return stripped;- }- escape_inline(&collapse_whitespace(rest))-}--/// Try to strip a leading list marker (optionally wrapped in emphasis markers)-/// off `s`. Recognizes `***MARK*** `, `**MARK** `, `*MARK* `, `` `MARK` ``,-/// and bare `MARK `. Returns the suffix on a match.-fn strip_leading_marker_from_inline(s: &str, marker: &str) -> Option<String> {- for wrap in ["***", "**", "*", "`"] {- let prefix = format!("{wrap}{marker}{wrap} ");- if let Some(rest) = s.strip_prefix(&prefix) {- return Some(rest.to_string());- }- }- let prefix = format!("{marker} ");- s.strip_prefix(&prefix).map(|r| r.to_string())-}--/// Append an inline-rendered continuation line to an existing list-item body.-/// De-hyphenates against the raw text boundary (mirrors the paragraph rule)-/// and falls back to a space join otherwise.-fn append_inline_continuation(prev_text: &mut String, next_raw: &str, next_inline: &str) {- let next_raw = collapse_whitespace(next_raw);- if next_raw.is_empty() {- return;- }- if prev_text.is_empty() {- prev_text.push_str(next_inline);- return;- }- let dehyphenate = prev_text.ends_with('-')- && next_raw- .chars()- .next()- .is_some_and(|c| c.is_ascii_lowercase());- if dehyphenate {- prev_text.pop();- prev_text.push_str(next_inline);- } else {- prev_text.push(' ');- prev_text.push_str(next_inline);- }-}--/// Vertical-gap check for table rows. Looser than paragraph continuation-/// because table rows often have extra padding between them.-fn table_rows_adjacent(prev: &ProjectedLine, cur: &ProjectedLine) -> bool {- // Intentionally don't require region_path equality. Indented sub-group- // rows (e.g. an indented "MEMORYBANK" row in a grouped academic results- // table) sometimes land in a different XY-cut leaf than the rest of the- // table — but the column-track alignment and y-gap checks below are- // strong enough signals on their own to keep us from spuriously- // bridging unrelated regions.- let prev_bottom = prev.bbox.y + prev.bbox.height;- let gap = cur.bbox.y - prev_bottom;- let line_height = prev.bbox.height.max(cur.bbox.height).max(1.0);- gap >= -line_height && gap <= line_height * TABLE_ROW_GAP_MULTIPLIER-}--/// Coefficient of variation (std-dev / mean) of inter-row vertical gaps.-/// Returns 0.0 for runs with <2 gaps (nothing to compare). Used to reject-/// runs whose row spacing is too irregular to be a real table.-fn row_spacing_cv(rows: &[(usize, &ProjectedLine, Vec<TableCell>)]) -> f32 {- if rows.len() < 3 {- return 0.0;- }- let gaps: Vec<f32> = rows- .windows(2)- .map(|w| (w[1].1.bbox.y - w[0].1.bbox.y).abs())- .collect();- let mean = gaps.iter().sum::<f32>() / gaps.len() as f32;- if mean <= 0.0 {- return f32::INFINITY;- }- let var = gaps.iter().map(|g| (g - mean).powi(2)).sum::<f32>() / gaps.len() as f32;- var.sqrt() / mean-}--/// Try to extend a candidate table starting at `start_idx`. On success returns-/// a `TableRun` with `Block::Table` or `Block::GridFallback`; on failure-/// returns `None` (and the caller should fall through to per-line-/// classification).-fn try_detect_table(lines: &[ProjectedLine], start_idx: usize) -> Option<TableRun> {- let first_cells = split_cells(&lines[start_idx]);- if first_cells.len() < TABLE_MIN_COLUMNS {- return None;- }-- let mut rows: Vec<(usize, &ProjectedLine, Vec<TableCell>)> =- vec![(start_idx, &lines[start_idx], first_cells.clone())];- let column_count = first_cells.len();- let tracks: Vec<f32> = first_cells.iter().map(|c| c.start_x).collect();-- let mut j = start_idx + 1;- while j < lines.len() {- if !table_rows_adjacent(rows.last().unwrap().1, &lines[j]) {- break;- }- let mut cells = split_cells(&lines[j]);- if cells.len() < column_count && cells.len() >= TABLE_MIN_COLUMNS {- // PDFium occasionally merges two (or more) adjacent words into one- // text run when inter-word kerning is tighter than the gap- // threshold — common in tightly-set numeric tables (e.g. the- // "MEMORYBANK 5.00 4.77" case on page 6 of the AMEM paper).- // Recover by splitting straddling cells on internal whitespace.- if let Some(patched) = recover_merged_cell(cells.clone(), &tracks) {- cells = patched;- }- }- if cells.len() != column_count {- break;- }- // Allow at most one column track to drift out of tolerance, which lets- // grouped row-labels in academic tables (e.g. an indented "MEMORYBANK"- // row whose label column shifts right by ~30pt while the numeric- // columns stay aligned) stay in the same run. Without this slack a- // single indented label fragments a 6-row table into three 2-row chunks.- let misaligned = cells- .iter()- .zip(tracks.iter())- .filter(|(c, t)| (c.start_x - **t).abs() > TABLE_TRACK_TOLERANCE_PT)- .count();- if misaligned > 1 {- break;- }- rows.push((j, &lines[j], cells));- j += 1;- }-- if rows.len() < TABLE_MIN_ROWS {- return None;- }-- let cv = row_spacing_cv(&rows);- let end = j;-- if cv > TABLE_ROW_SPACING_MAX_CV {- // Suggestive layout but the row cadence is too irregular to trust as a- // clean table — surface as a fenced fallback so the structure is at- // least preserved.- let raw: Vec<String> = rows- .iter()- .map(|(_, line, _)| line.text.trim_end().to_string())- .collect();- return Some(TableRun {- start: start_idx,- end,- block: Block::GridFallback { lines: raw },- });- }-- // Promote the first row to header iff every cell in it is bold (matches- // pymupdf4llm's "bold-or-filled" heuristic; fills require fork data).- let first_row = &rows[0].2;- let header_qualifies = first_row.iter().all(|c| c.bold);- let header = if header_qualifies {- Some(first_row.iter().map(|c| c.text.clone()).collect())- } else {- None- };- let row_start = if header.is_some() { 1 } else { 0 };- let body_rows: Vec<Vec<String>> = rows[row_start..]- .iter()- .map(|(_, _, cells)| cells.iter().map(|c| c.text.clone()).collect())- .collect();- if header.is_none() && body_rows.len() < TABLE_MIN_ROWS {- return None;- }-- Some(TableRun {- start: start_idx,- end,- block: Block::Table {- header,- rows: body_rows,- },- })-}--/// Scan `lines` once and return all detected tabular regions (sorted by-/// `start`). Caller uses these as cut-points so the per-line classifier never-/// sees lines inside a table.-fn detect_tables(lines: &[ProjectedLine]) -> Vec<TableRun> {- let mut out = Vec::new();- let mut i = 0;- while i < lines.len() {- if let Some(run) = try_detect_table(lines, i) {- i = run.end;- out.push(run);- } else {- i += 1;- }- }- out-}--// ── Ruled-grid table detection ─────────────────────────────────────────────-//-// Detect tables drawn with explicit horizontal + vertical rules (the "Strong"-// mode in MARKDOWN_PLAN.md). Strokes are clustered into H/V grid lines, then-// union-find groups crossing lines into table regions. For each region the-// distinct row/column boundaries form a cell grid; text lines are assigned to-// cells by centroid containment.-//-// Ruled tables are detected before the borderless `detect_tables`. The caller-// merges the two outputs; overlapping ranges defer to the ruled run because-// path-based geometry is a strictly stronger signal than text alignment alone.--/// Horizontal segment in viewport coords (top-left origin). `y` is the rule's-/// y-position; `x_min..x_max` is its horizontal span. Endpoints of multiple-/// short segments sharing a y get unioned into one wider segment during-/// clustering.-#[derive(Debug, Clone, Copy)]-struct HSeg {- x_min: f32,- x_max: f32,- y: f32,-}--#[derive(Debug, Clone, Copy)]-struct VSeg {- y_min: f32,- y_max: f32,- x: f32,-}--/// Strokes are considered "axis-aligned" when the perpendicular delta is at-/// most this many points. Generous to absorb antialiased near-pixel strokes.-const TABLE_AXIS_TOLERANCE_PT: f32 = 1.0;--/// Two H lines (or two V lines) are merged into one grid line when their-/// perpendicular coords are within this many points. Slightly looser than the-/// axis tolerance because rules drawn at the same row can have ±1pt jitter-/// from different stroke widths.-const TABLE_GRID_CLUSTER_PT: f32 = 2.0;--/// Slack added when checking whether a V line "crosses" an H line. Helps-/// when rules don't quite reach the corner because the PDF drew them as-/// individual segments with small gaps.-const TABLE_CROSS_TOLERANCE_PT: f32 = 3.0;--/// Reject ruled-table candidates whose empty-cell fraction exceeds this.-const TABLE_MAX_EMPTY_CELL_FRACTION: f32 = 0.30;--/// Reject candidates whose grid covers nearly the whole page — almost always-/// a page border, not a real table.-const TABLE_MAX_PAGE_COVERAGE: f32 = 0.95;--/// Extract horizontal and vertical line segments from a page's graphics. Each-/// `Stroke` becomes one HSeg or VSeg depending on orientation; each stroked-/// `Rect` contributes its four edges (cell-border rects, table frames).-fn extract_h_v_segments(graphics: &[GraphicPrimitive]) -> (Vec<HSeg>, Vec<VSeg>) {- let mut hs = Vec::new();- let mut vs = Vec::new();- for g in graphics {- match g {- GraphicPrimitive::Stroke { x1, y1, x2, y2, .. } => {- let (x1, y1, x2, y2) = (*x1, *y1, *x2, *y2);- let dy = (y1 - y2).abs();- let dx = (x1 - x2).abs();- if dy <= TABLE_AXIS_TOLERANCE_PT && dx > 1.0 {- hs.push(HSeg {- x_min: x1.min(x2),- x_max: x1.max(x2),- y: (y1 + y2) * 0.5,- });- } else if dx <= TABLE_AXIS_TOLERANCE_PT && dy > 1.0 {- vs.push(VSeg {- y_min: y1.min(y2),- y_max: y1.max(y2),- x: (x1 + x2) * 0.5,- });- }- }- GraphicPrimitive::Rect { bbox, stroke, .. } => {- if stroke.is_none() {- continue;- }- let top = bbox.y;- let bottom = bbox.y + bbox.height;- let left = bbox.x;- let right = bbox.x + bbox.width;- if bbox.width > 1.0 {- hs.push(HSeg {- x_min: left,- x_max: right,- y: top,- });- hs.push(HSeg {- x_min: left,- x_max: right,- y: bottom,- });- }- if bbox.height > 1.0 {- vs.push(VSeg {- y_min: top,- y_max: bottom,- x: left,- });- vs.push(VSeg {- y_min: top,- y_max: bottom,- x: right,- });- }- }- }- }- (hs, vs)-}--/// Cluster H segments sharing a y-coordinate (within `TABLE_GRID_CLUSTER_PT`)-/// into a single wider grid line whose x-extent is the union of the inputs.-fn cluster_h_segments(mut segs: Vec<HSeg>) -> Vec<HSeg> {- if segs.is_empty() {- return segs;- }- segs.sort_by(|a, b| a.y.total_cmp(&b.y));- let mut out: Vec<HSeg> = Vec::with_capacity(segs.len());- for seg in segs {- if let Some(last) = out.last_mut()- && (last.y - seg.y).abs() <= TABLE_GRID_CLUSTER_PT- {- last.x_min = last.x_min.min(seg.x_min);- last.x_max = last.x_max.max(seg.x_max);- continue;- }- out.push(seg);- }- out-}--fn cluster_v_segments(mut segs: Vec<VSeg>) -> Vec<VSeg> {- if segs.is_empty() {- return segs;- }- segs.sort_by(|a, b| a.x.total_cmp(&b.x));- let mut out: Vec<VSeg> = Vec::with_capacity(segs.len());- for seg in segs {- if let Some(last) = out.last_mut()- && (last.x - seg.x).abs() <= TABLE_GRID_CLUSTER_PT- {- last.y_min = last.y_min.min(seg.y_min);- last.y_max = last.y_max.max(seg.y_max);- continue;- }- out.push(seg);- }- out-}--/// Union-find root with path compression.-fn uf_find(parent: &mut [usize], mut x: usize) -> usize {- while parent[x] != x {- parent[x] = parent[parent[x]];- x = parent[x];- }- x-}--fn uf_union(parent: &mut [usize], a: usize, b: usize) {- let ra = uf_find(parent, a);- let rb = uf_find(parent, b);- if ra != rb {- parent[ra] = rb;- }-}--/// Group H/V grid lines that cross each other into connected components.-/// Each component is a candidate ruled table — typically one component per-/// distinct table on the page. Returns `(h_indices, v_indices)` per component,-/// dropping components without ≥2 H and ≥2 V lines (a single L-shape doesn't-/// make a table).-fn find_grid_components(hs: &[HSeg], vs: &[VSeg]) -> Vec<(Vec<usize>, Vec<usize>)> {- let n_h = hs.len();- let n_v = vs.len();- if n_h < 2 || n_v < 2 {- return Vec::new();- }- let n = n_h + n_v;- let mut parent: Vec<usize> = (0..n).collect();- let mut connected = vec![false; n];-- let tol = TABLE_CROSS_TOLERANCE_PT;- for (i, h) in hs.iter().enumerate() {- for (j, v) in vs.iter().enumerate() {- let v_crosses_h_x = v.x >= h.x_min - tol && v.x <= h.x_max + tol;- let h_crosses_v_y = h.y >= v.y_min - tol && h.y <= v.y_max + tol;- if v_crosses_h_x && h_crosses_v_y {- uf_union(&mut parent, i, n_h + j);- connected[i] = true;- connected[n_h + j] = true;- }- }- }-- use std::collections::HashMap;- let mut groups: HashMap<usize, (Vec<usize>, Vec<usize>)> = HashMap::new();- for i in 0..n_h {- if !connected[i] {- continue;- }- let r = uf_find(&mut parent, i);- groups.entry(r).or_default().0.push(i);- }- for j in 0..n_v {- if !connected[n_h + j] {- continue;- }- let r = uf_find(&mut parent, n_h + j);- groups.entry(r).or_default().1.push(j);- }- groups- .into_values()- .filter(|(h_idx, v_idx)| h_idx.len() >= 2 && v_idx.len() >= 2)- .collect()-}--/// Build a `TableRun` for one ruled-grid component. Returns `None` if the-/// resulting grid is too small (< 2 cols or < 2 rows), covers nearly the-/// whole page (likely the page border), or is mostly empty cells.-fn build_ruled_table(- hs: &[HSeg],- vs: &[VSeg],- h_indices: &[usize],- v_indices: &[usize],- lines: &[ProjectedLine],- page_width: f32,- page_height: f32,-) -> Option<TableRun> {- // Distinct row y-coords (cluster again — multiple H lines may share a y).- let mut ys: Vec<f32> = h_indices.iter().map(|&i| hs[i].y).collect();- ys.sort_by(|a, b| a.total_cmp(b));- dedup_close(&mut ys, TABLE_GRID_CLUSTER_PT);-- let mut xs: Vec<f32> = v_indices.iter().map(|&i| vs[i].x).collect();- xs.sort_by(|a, b| a.total_cmp(b));- dedup_close(&mut xs, TABLE_GRID_CLUSTER_PT);-- // Need ≥2 cells per axis → ≥3 boundary lines.- if ys.len() < 3 || xs.len() < 3 {- return None;- }-- let n_rows = ys.len() - 1;- let n_cols = xs.len() - 1;- let bbox = crate::types::Rect {- x: xs[0],- y: ys[0],- width: xs[n_cols] - xs[0],- height: ys[n_rows] - ys[0],- };-- // Reject page-border-as-table.- if page_width > 0.0 && page_height > 0.0 {- let coverage = (bbox.width / page_width) * (bbox.height / page_height);- if coverage > TABLE_MAX_PAGE_COVERAGE {- return None;- }- }-- // Assign each text line to its cell by centroid.- let mut cells: Vec<Vec<String>> = vec![vec![String::new(); n_cols]; n_rows];- let mut cell_is_bold: Vec<Vec<bool>> = vec![vec![true; n_cols]; n_rows];- let mut cell_has_text: Vec<Vec<bool>> = vec![vec![false; n_cols]; n_rows];- let mut consumed_indices: Vec<usize> = Vec::new();-- for (idx, line) in lines.iter().enumerate() {- let cx = line.bbox.x + line.bbox.width * 0.5;- let cy = line.bbox.y + line.bbox.height * 0.5;- if cy < ys[0] || cy > ys[n_rows] || cx < xs[0] || cx > xs[n_cols] {- continue;- }- let row = match find_bucket(&ys, cy) {- Some(r) => r,- None => continue,- };- let col = match find_bucket(&xs, cx) {- Some(c) => c,- None => continue,- };- let txt = line.text.trim();- if txt.is_empty() {- continue;- }- if !cells[row][col].is_empty() {- cells[row][col].push(' ');- }- cells[row][col].push_str(txt);- cell_has_text[row][col] = true;- if !line.all_bold {- cell_is_bold[row][col] = false;- }- consumed_indices.push(idx);- }-- if consumed_indices.is_empty() {- return None;- }-- let total = n_rows * n_cols;- let empty_count = cell_has_text- .iter()- .flatten()- .filter(|filled| !**filled)- .count();- if (empty_count as f32) / (total as f32) > TABLE_MAX_EMPTY_CELL_FRACTION {- return None;- }-- // Header = first row iff every non-empty cell in it is bold.- let header_qualifies = cell_has_text[0]- .iter()- .zip(cell_is_bold[0].iter())- .all(|(has, bold)| !has || *bold)- && cell_has_text[0].iter().any(|has| *has);- let header = if header_qualifies {- Some(cells[0].clone())- } else {- None- };- let body_start = if header.is_some() { 1 } else { 0 };- let body_rows: Vec<Vec<String>> = cells[body_start..].to_vec();- if body_rows.is_empty() {- return None;- }-- // Line index span this table covers.- let start = *consumed_indices.iter().min().unwrap();- let end = *consumed_indices.iter().max().unwrap() + 1;-- Some(TableRun {- start,- end,- block: Block::Table {- header,- rows: body_rows,- },- })-}--/// In-place dedup of a sorted Vec, collapsing entries within `tol` to the-/// first of each cluster.-fn dedup_close(v: &mut Vec<f32>, tol: f32) {- if v.len() < 2 {- return;- }- let mut out: Vec<f32> = Vec::with_capacity(v.len());- for x in v.iter().copied() {- if let Some(&last) = out.last()- && (x - last).abs() <= tol- {- continue;- }- out.push(x);- }- *v = out;-}--/// Find the bucket index `i` such that `boundaries[i] <= val < boundaries[i+1]`.-/// Returns `None` if `val` is outside the boundaries.-fn find_bucket(boundaries: &[f32], val: f32) -> Option<usize> {- if boundaries.len() < 2 || val < boundaries[0] || val > *boundaries.last().unwrap() {- return None;- }- for (i, w) in boundaries.windows(2).enumerate() {- if val >= w[0] && val <= w[1] {- return Some(i);- }- }- None-}--/// Detect ruled-grid tables on a page from its vector graphics. Returns runs-/// in document order (sorted by `start`).-fn detect_ruled_tables(- lines: &[ProjectedLine],- graphics: &[GraphicPrimitive],- page_width: f32,- page_height: f32,-) -> Vec<TableRun> {- let (hs, vs) = extract_h_v_segments(graphics);- let hs = cluster_h_segments(hs);- let vs = cluster_v_segments(vs);- if hs.len() < 2 || vs.len() < 2 {- return Vec::new();- }- let components = find_grid_components(&hs, &vs);- let mut out = Vec::new();- for (h_idx, v_idx) in components {- if let Some(run) =- build_ruled_table(&hs, &vs, &h_idx, &v_idx, lines, page_width, page_height)- {- out.push(run);- }- }- out.sort_by_key(|r| r.start);- out-}--/// Merge ruled-grid runs with borderless runs into a single sorted list. When-/// ranges overlap the ruled run wins (path-based geometry is strictly stronger-/// than text-alignment heuristics) — overlapping borderless runs are dropped.-fn merge_table_runs(mut ruled: Vec<TableRun>, borderless: Vec<TableRun>) -> Vec<TableRun> {- for b in borderless {- let overlaps = ruled- .iter()- .any(|r| !(b.end <= r.start || b.start >= r.end));- if !overlaps {- ruled.push(b);- }- }- ruled.sort_by_key(|r| r.start);- ruled-}--/// Escape `|` and `\n` inside a markdown table cell so the pipe-table grammar-/// stays valid. Newlines should be impossible inside a single cell (we built-/// cells from spans on the same projected line) but guard anyway.-fn escape_table_cell(s: &str) -> String {- s.replace('\\', "\\\\")- .replace('|', "\\|")- .replace('\n', " ")-}--/// Tolerance in points for "is this size larger than the body size".-const HEADING_SIZE_EPSILON: f32 = 0.5;--/// Cap on heading levels (matches plan: H1..H6).-const MAX_HEADING_LEVELS: usize = 6;--/// Multiplier on line height used as the paragraph-break threshold.-const PARAGRAPH_GAP_MULTIPLIER: f32 = 1.5;--/// Tolerance for treating two font sizes as "the same" when grouping-/// paragraph lines. Generous because we sometimes derive the "size" from-/// `bbox.height`, which varies a few points line-to-line based on whether-/// the glyphs include descenders (`y`, `g`, `p`).-const FONT_SIZE_PARAGRAPH_TOLERANCE: f32 = 2.5;--/// Tighter tolerance for matching against the heading-size map. Keeps the-/// heading detector strict so descender-induced height jitter doesn't promote-/// regular body lines to headings.-const FONT_SIZE_HEADING_TOLERANCE: f32 = 0.6;--/// Tolerance in points for treating two indent positions as "the same column".-const INDENT_TOLERANCE: f32 = 6.0;--/// Fraction of page height treated as the "top band" for header detection.-/// Most running headers sit within the top 8–12% of a page; 12% gives some-/// slack for two-line headers without sweeping in body text.-const HEADER_BAND_FRACTION: f32 = 0.12;--/// Fraction of page height treated as the "bottom band" for footer detection.-const FOOTER_BAND_FRACTION: f32 = 0.12;--/// Fraction of pages on which a normalized line must appear (in the same-/// band) to be classified as a running header/footer.-const HEADER_FOOTER_MIN_FRACTION: f32 = 0.5;--/// Absolute floor on header/footer matches — single-page docs can't have a-/// "running" header by definition, and a single match on a 2-page doc is too-/// weak to act on.-const HEADER_FOOTER_MIN_PAGES: usize = 2;--/// Roughly one indent step in PDF points. Used to bucket list items into-/// nesting levels relative to the first item of the list.-const LIST_INDENT_STEP_PT: f32 = 12.0;--/// Maximum characters in a "bold body-size heading" candidate. Section-/// headings like "Abstract", "1 Introduction", "2.1 Related Work" are short;-/// a bold body-size line longer than this is almost always a bold *sentence*-/// inside a paragraph, not a heading.-const BOLD_HEADING_MAX_CHARS: usize = 80;--/// Returns true if `line` looks like a section heading rendered in body-size-/// bold text (a very common style for academic / technical PDFs where every-/// "real" heading uses the same font size as body, distinguished only by-/// weight). Requires:-/// - uniformly bold across all spans-/// - short (≤ `BOLD_HEADING_MAX_CHARS`)-/// - paragraph-break gap above (or first line on the page)-/// - paragraph-break gap below (or last line on the page)-fn looks_like_bold_heading(- line: &ProjectedLine,- prev: Option<&ProjectedLine>,- next: Option<&ProjectedLine>,-) -> bool {- let text = line.text.trim();- if text.is_empty() || text.chars().count() > BOLD_HEADING_MAX_CHARS {- return false;- }- let style = match line_uniform_style(line) {- Some(s) => s,- None => return false,- };- if !style.bold || style.mono {- return false;- }- // Reject bold-uniform lines dominated by digits / punctuation — these are- // almost always cells inside a tabular layout the table detector didn't- // pick up (results tables, scoreboards, math display). A real section- // heading is mostly letters: "1 Introduction" passes (~92% alpha across- // non-whitespace chars), "47.5 14" doesn't (0%), "BLEU-1 25.87" doesn't.- let mut alpha = 0usize;- let mut total = 0usize;- for c in text.chars() {- if c.is_whitespace() {- continue;- }- total += 1;- if c.is_alphabetic() {- alpha += 1;- }- }- if total == 0 || (alpha as f32) / (total as f32) < 0.5 {- return false;- }- // Reject when the line itself looks tabular: ≥3 cells separated by font-size- // gaps. A bold body-size line with that many cell tracks is almost always a- // table header row, not a section heading. Without this guard, multi-line- // table headers ("Model Method ... | F1 BLEU-1 F1 BLEU-1 ...") get promoted- // to H3 instead of being absorbed by the table detector.- // Tabular shape rejection. Two passes because the projection sometimes- // collapses a wide multi-column line into a single span with column-- // padding spaces — span-based detection misses those.- if split_cells(line).len() >= TABLE_MIN_COLUMNS {- return false;- }- // Text-based fallback: ≥3 tokens separated by runs of 2+ spaces (the- // projection inserts alignment padding between cells) → table header row- // collapsed into one span, not a section heading.- let multi_space_tokens = text- .split(" ")- .map(|s| s.trim())- .filter(|s| !s.is_empty())- .count();- if multi_space_tokens >= TABLE_MIN_COLUMNS {- return false;- }- let gap_above_ok = match prev {- None => true,- Some(p) => !continues_paragraph(p, line),- };- if !gap_above_ok {- return false;- }- let gap_below_ok = match next {- None => true,- Some(n) => !continues_paragraph(line, n),- };- gap_below_ok-}--/// Characters recognized as bullet markers when followed by whitespace.-/// Limited to glyphs that are unlikely to appear at line-start in normal prose.-const BULLET_CHARS: &[char] = &['•', '·', '◦', '▪', '▸', '▶', '●', '○', '■', '□'];--/// Detect a list marker at the start of `text`. Returns `(ordered, marker_str,-/// remainder)` when matched; otherwise `None`.-///-/// Recognizes:-/// - Unicode bullet characters (`BULLET_CHARS`) followed by whitespace.-/// - Decimal-prefix markers like `1.` / `1)` / `12.` / `12)` followed by-/// whitespace — kept strict (digits only) so things like footnote callers-/// (`1` alone) and section refs (`A.1`) don't match.-fn parse_list_marker(text: &str) -> Option<(bool, String, &str)> {- let trimmed = text.trim_start();- if trimmed.is_empty() {- return None;- }-- let mut chars = trimmed.chars();- let first = chars.next()?;-- // Unicode bullet- if BULLET_CHARS.contains(&first) {- let rest = chars.as_str();- if let Some(rest_trim) = rest.strip_prefix(|c: char| c.is_whitespace()) {- return Some((false, first.to_string(), rest_trim.trim_start()));- }- }-- // Decimal: 1. / 1) / 12. / 12)- if first.is_ascii_digit() {- let mut digit_end = 1;- for c in trimmed[1..].chars() {- if c.is_ascii_digit() {- digit_end += c.len_utf8();- } else {- break;- }- }- // Cap to keep us from matching page-number-like prefixes- if digit_end <= 3 {- let after_digits = &trimmed[digit_end..];- let mut after_iter = after_digits.chars();- if let Some(punct) = after_iter.next()- && (punct == '.' || punct == ')')- {- let after_punct = after_iter.as_str();- if let Some(rest_trim) = after_punct.strip_prefix(|c: char| c.is_whitespace()) {- let marker = format!("{}{}", &trimmed[..digit_end], punct);- return Some((true, marker, rest_trim.trim_start()));- }- }- }- }-- None-}--/// Returns true if any span on the line is rotated more than ~5° off-/// horizontal — used to exclude sidebar / margin-stamp text (arXiv banners,-/// watermarks, vertical legends) from the body-size and heading-size-/// histograms so it doesn't compete with normal-flow text for heading slots.-fn is_rotated_line(line: &ProjectedLine) -> bool {- line.spans.iter().any(|s| {- let r = s.rotation.abs() % 360.0;- // Anything more than ~5° off the horizontal axes is "rotated" for- // our purposes. 0° and 180° are both horizontal text.- !(r < 5.0 || (175.0..=185.0).contains(&r) || (355.0..=360.0).contains(&r))- })-}--/// Compute the body font size as the char-weighted mode across all lines in-/// all pages. Rotated lines are excluded so a long rotated sidebar can't-/// pull the body estimate. Falls back to `0.0` when no font-size info is-/// available.-pub fn compute_body_size(pages: &[ParsedPage]) -> f32 {- use std::collections::HashMap;- let mut weights: HashMap<u32, (f32, usize)> = HashMap::new();- for page in pages {- for line in &page.projected_lines {- if is_rotated_line(line) {- continue;- }- let size = line.dominant_font_size;- if size <= 0.0 {- continue;- }- let chars = line.text.chars().count().max(1);- let key = (size * 100.0).round() as u32;- let entry = weights.entry(key).or_insert((size, 0));- entry.1 += chars;- }- }- weights- .values()- .max_by_key(|(_, n)| *n)- .map(|(s, _)| *s)- .unwrap_or(0.0)-}--/// Minimum total non-whitespace characters across all occurrences at a font-/// size for it to qualify as a heading level. Calibrated against `paper.pdf`:-/// the 30pt chart-legend tokens ("A-mem"×2 + "Base"×2 = 18-20 chars) need to-/// fail this filter, while the 14.35pt title ("A-MEM: Agentic Memory for LLM-/// Agents" = 31 chars on a single line) needs to pass. 25 is the gap. Smaller-/// single-word headings like a lone "Summary" on a short doc still survive-/// because they share their font size with other (larger) headings in the-/// histogram entry.-const MIN_HEADING_TOTAL_CHARS: usize = 25;--/// Maximum average characters per line for a size to qualify as a heading.-/// A "size larger than body" with very long lines is almost always a-/// large-print body block (callouts, footnotes-as-display, intro paragraph),-/// not a real heading.-const MAX_HEADING_AVG_LINE_CHARS: f32 = 200.0;--/// Minimum fraction of non-whitespace chars at a size that must be alphabetic-/// for it to qualify as a heading. Drops sizes dominated by digits (graph-/// axes, results tables, math display) which otherwise pollute the top-/// heading slots.-const MIN_HEADING_ALPHA_RATIO: f32 = 0.5;--/// Build a heading-size → level map: sizes strictly larger than `body_size`,-/// filtered to those with at least `MIN_HEADING_LINES` distinct occurrences-/// (drops one-off equation/figure-label artifacts), sorted descending, mapped-/// to levels 1..=MAX_HEADING_LEVELS.-pub fn build_heading_map(pages: &[ParsedPage], body_size: f32) -> Vec<(f32, u8)> {- use std::collections::HashMap;- // (size_key → (size, line_count, total_chars, alpha_chars))- let mut sizes: HashMap<u32, (f32, usize, usize, usize)> = HashMap::new();- for page in pages {- for line in &page.projected_lines {- if is_rotated_line(line) {- continue;- }- let size = line.dominant_font_size;- if size > body_size + HEADING_SIZE_EPSILON {- let key = (size * 100.0).round() as u32;- let entry = sizes.entry(key).or_insert((size, 0, 0, 0));- entry.1 += 1;- for c in line.text.chars() {- if c.is_whitespace() {- continue;- }- entry.2 += 1;- if c.is_alphabetic() {- entry.3 += 1;- }- }- }- }- }- let all: Vec<(f32, usize, usize, usize)> = sizes.into_values().collect();- // Always apply quality filters: total-char floor, average-line cap, and- // alpha-ratio floor. The total-char floor (rather than a line-count one)- // lets one-off titles survive — a 31-char title on a single line passes —- // while still rejecting chart-legend tokens like "A-mem" + "Base" that- // total fewer chars across 4 occurrences than a single heading line does.- let mut kept: Vec<f32> = all- .iter()- .filter(|(_, lines, chars, alpha)| {- let alpha_ratio = if *chars == 0 {- 0.0- } else {- (*alpha as f32) / (*chars as f32)- };- *chars >= MIN_HEADING_TOTAL_CHARS- && (*chars as f32 / (*lines).max(1) as f32) <= MAX_HEADING_AVG_LINE_CHARS- && alpha_ratio >= MIN_HEADING_ALPHA_RATIO- })- .map(|(s, _, _, _)| *s)- .collect();- kept.sort_by(|a, b| b.total_cmp(a));- kept.truncate(MAX_HEADING_LEVELS);- kept.into_iter()- .enumerate()- .map(|(i, s)| (s, (i + 1) as u8))- .collect()-}--fn heading_level_for(size: f32, heading_map: &[(f32, u8)]) -> Option<u8> {- for (s, level) in heading_map {- if (size - *s).abs() < FONT_SIZE_HEADING_TOLERANCE {- return Some(*level);- }- }- None-}--/// Collapse runs of whitespace into single spaces. The projected text from-/// `projection.rs` pads with column-alignment spaces (e.g. `for instance`)-/// which look fine as a layout grid but are wrong for prose.-fn collapse_whitespace(s: &str) -> String {- let mut out = String::with_capacity(s.len());- let mut prev_space = false;- for c in s.chars() {- if c.is_whitespace() {- if !prev_space && !out.is_empty() {- out.push(' ');- }- prev_space = true;- } else {- out.push(c);- prev_space = false;- }- }- if out.ends_with(' ') {- out.pop();- }- out-}--/// Append `next` to `accum` as the continuation of a paragraph. De-hyphenates-/// when `accum` ends with `-` and `next` starts with an ASCII lowercase letter;-/// otherwise joins with a single space.-#[cfg(test)]-fn append_paragraph_line(accum: &mut String, next: &str) {- let next = collapse_whitespace(next);- if next.is_empty() {- return;- }- if accum.is_empty() {- accum.push_str(&next);- return;- }- let dehyphenate =- accum.ends_with('-') && next.chars().next().is_some_and(|c| c.is_ascii_lowercase());- if dehyphenate {- accum.pop(); // drop the '-'- accum.push_str(&next);- } else {- accum.push(' ');- accum.push_str(&next);- }-}--/// Decide whether `cur` continues the paragraph started by `prev`.-fn continues_paragraph(prev: &ProjectedLine, cur: &ProjectedLine) -> bool {- // Anchor only signals a paragraph break when one of the lines is clearly- // centered while the other isn't — justified prose routinely alternates- // between Left / Right / Floating dominant anchors as line widths flex,- // and treating those as paragraph breaks shreds normal text.- let centered_mismatch = (prev.anchor == Anchor::Center) ^ (cur.anchor == Anchor::Center);- if centered_mismatch {- return false;- }- if (prev.dominant_font_size - cur.dominant_font_size).abs() > FONT_SIZE_PARAGRAPH_TOLERANCE {- return false;- }- if prev.region_path != cur.region_path {- // Cross-region continuation: the same paragraph can wrap from the- // bottom of one column into the top of the next. Only bridge regions- // when the previous line clearly breaks mid-sentence (no terminal- // punctuation) AND the next line starts with a lowercase letter — a- // strict signal that catches the column-wrap case while rejecting- // unrelated paragraphs that happen to sit in adjacent leaves.- let prev_trim = prev.text.trim_end();- let ends_open = !prev_trim.ends_with(|c: char| {- matches!(- c,- '.' | '!' | '?' | ':' | ';' | '”' | '"' | ')' | ']' | '。' | '』' | '」'- )- });- let starts_lower = cur- .text- .trim_start()- .chars()- .next()- .is_some_and(|c| c.is_lowercase());- return ends_open && starts_lower;- }- if (prev.indent_x - cur.indent_x).abs() > INDENT_TOLERANCE && cur.anchor == Anchor::Left {- // Indent change on a left-aligned block usually means a new paragraph- // (block-quote, list, indented passage, etc.). Allow first-line indent- // by checking only when the *next* line shifts right relative to prev.- if cur.indent_x > prev.indent_x + INDENT_TOLERANCE {- return false;- }- }- // Vertical gap check.- let prev_bottom = prev.bbox.y + prev.bbox.height;- let gap = cur.bbox.y - prev_bottom;- let line_height = prev.bbox.height.max(cur.bbox.height).max(1.0);- gap <= line_height * PARAGRAPH_GAP_MULTIPLIER-}--/// Paragraph accumulator state. We track two parallel representations of the-/// running paragraph text:-///-/// - `raw` — plain text (no emphasis markers). Used for the paragraph-uniform-/// shortcut: if every contributing line had the same uniform style, we wrap-/// the whole paragraph once with `wrap_emphasis(raw, …)` to avoid the-/// `**foo** **bar** **baz**` per-line noise pymupdf4llm warns about.-/// - `inline` — per-line markdown with emphasis baked in via-/// `render_line_inline`. Used when the paragraph contains mid-line emphasis-/// shifts or lines with differing uniform styles.-///-/// `uniform` is `Some((bold, italic))` while every line so far has been a-/// uniformly-styled line sharing the same (bold, italic) flags, and `None` as-/// soon as that invariant breaks.-struct ParaAccum {- raw: String,- inline: String,- last: ProjectedLine,- uniform: Option<(bool, bool)>,-}--/// Returns the shared `SpanStyle` of `line` when every non-whitespace span on-/// the line has the same style; `None` when spans disagree. Mono is folded-/// into the style for the purpose of "uniform" — a fully-mono line is-/// uniform-mono. Used by the paragraph-level optimization to decide whether-/// to wrap once around the whole paragraph or fall back to per-line inline.-fn line_uniform_style(line: &ProjectedLine) -> Option<SpanStyle> {- let mut iter = line- .spans- .iter()- .filter(|s| !s.text.trim().is_empty())- .map(SpanStyle::from_item);- let first = iter.next()?;- for s in iter {- if s != first {- return None;- }- }- Some(first)-}--/// Append `next_line` to a paragraph accumulator. Maintains both the `raw` and-/// `inline` text representations and updates the running `uniform` flag.-/// De-hyphenation runs on the `raw` boundary; the `inline` boundary mirrors it-/// when the trailing char is still a literal `-` (i.e. the hyphen sits outside-/// any emphasis wrap — the common case).-fn append_to_paragraph(accum: &mut ParaAccum, next_line: &ProjectedLine) {- let next_raw = collapse_whitespace(next_line.text.trim());- if next_raw.is_empty() {- return;- }- let next_inline = render_line_inline(next_line);- let next_uniform = line_uniform_style(next_line);-- if accum.raw.is_empty() {- accum.raw.push_str(&next_raw);- accum.inline.push_str(&next_inline);- accum.uniform = next_uniform.map(|s| (s.bold, s.italic));- accum.last = next_line.clone();- return;- }-- let dehyphenate = accum.raw.ends_with('-')- && next_raw- .chars()- .next()- .is_some_and(|c| c.is_ascii_lowercase());- if dehyphenate {- accum.raw.pop();- accum.raw.push_str(&next_raw);- if accum.inline.ends_with('-') {- accum.inline.pop();- accum.inline.push_str(&next_inline);- } else {- // Hyphen sits inside an emphasis wrap — give up on stripping it- // from `inline` cleanly. Join with a space; raw is still correct- // for the uniform-paragraph case.- accum.inline.push(' ');- accum.inline.push_str(&next_inline);- }- } else {- accum.raw.push(' ');- accum.raw.push_str(&next_raw);- accum.inline.push(' ');- accum.inline.push_str(&next_inline);- }-- // Maintain the running uniform-style flag.- accum.uniform = match (accum.uniform, next_uniform) {- (Some(cur), Some(s)) if cur == (s.bold, s.italic) => Some(cur),- _ => None,- };- accum.last = next_line.clone();-}--/// Resolve a `ParaAccum` to a `Block::Paragraph`. When the paragraph was-/// uniformly styled across all lines, return the raw text with block-level-/// `bold`/`italic` flags set so the renderer wraps it once. Otherwise return-/// the per-line inline-styled text with the flags cleared.-fn paragraph_from_accum(accum: ParaAccum) -> Block {- match accum.uniform {- Some((bold, italic)) if bold || italic => Block::Paragraph {- text: escape_inline(&accum.raw),- bold,- italic,- },- Some(_) => Block::Paragraph {- // Uniformly plain — no emphasis markers anywhere, so the raw text- // (with markdown specials escaped) is the right rendering.- text: escape_inline(&accum.raw),- bold: false,- italic: false,- },- None => Block::Paragraph {- text: accum.inline,- bold: false,- italic: false,- },- }-}--/// Normalize a line for cross-page header/footer matching. Lowercases,-/// collapses whitespace, and replaces every run of ASCII digits with `#` so-/// `Page 1 of 6` and `Page 2 of 6` collapse to the same key.-fn normalize_for_repetition(s: &str) -> String {- let collapsed = collapse_whitespace(s).to_lowercase();- let mut out = String::with_capacity(collapsed.len());- let mut in_digits = false;- for c in collapsed.chars() {- if c.is_ascii_digit() {- if !in_digits {- out.push('#');- in_digits = true;- }- } else {- out.push(c);- in_digits = false;- }- }- out-}--/// Cross-page repetition detector. Returns the set of normalized strings that-/// appear in the top or bottom band of ≥ `HEADER_FOOTER_MIN_FRACTION` of-/// pages (capped below by `HEADER_FOOTER_MIN_PAGES`). The caller uses this to-/// filter `ProjectedLine`s before classification.-///-/// "Same band" means a line whose top is within `HEADER_BAND_FRACTION` of the-/// page top (header) or whose bottom is within `FOOTER_BAND_FRACTION` of the-/// page bottom (footer). Header and footer bands are tracked separately so a-/// company name that appears as both a header and a body-section title on-/// different pages isn't stripped from the body.-pub fn compute_header_footer_set(pages: &[ParsedPage]) -> std::collections::HashSet<String> {- use std::collections::{HashMap, HashSet};- let mut set: HashSet<String> = HashSet::new();- if pages.len() < HEADER_FOOTER_MIN_PAGES {- return set;- }- // Two counters keyed by `(band, normalized_text)` — band is `'h'` or `'f'`.- let mut counts: HashMap<(char, String), HashSet<usize>> = HashMap::new();- for page in pages {- let header_cutoff = page.page_height * HEADER_BAND_FRACTION;- let footer_cutoff = page.page_height * (1.0 - FOOTER_BAND_FRACTION);- for line in &page.projected_lines {- let text = line.text.trim();- if text.is_empty() {- continue;- }- let norm = normalize_for_repetition(text);- if norm.is_empty() {- continue;- }- // Header band: top of line within the top band.- if line.bbox.y <= header_cutoff {- counts- .entry(('h', norm.clone()))- .or_default()- .insert(page.page_number);- }- // Footer band: bottom of line at or below the footer cutoff.- let line_bottom = line.bbox.y + line.bbox.height;- if line_bottom >= footer_cutoff {- counts- .entry(('f', norm))- .or_default()- .insert(page.page_number);- }- }- }- let threshold = (pages.len() as f32 * HEADER_FOOTER_MIN_FRACTION)- .ceil()- .max(HEADER_FOOTER_MIN_PAGES as f32) as usize;- for ((_, norm), pages_seen) in counts {- if pages_seen.len() >= threshold {- set.insert(norm);- }- }- set-}--/// Returns true if `line` (located on `page`) matches the running-/// header/footer set: the line sits in the top or bottom band AND its-/// normalized text is in `header_footer`.-fn is_header_or_footer(- line: &ProjectedLine,- page: &ParsedPage,- header_footer: &std::collections::HashSet<String>,-) -> bool {- if header_footer.is_empty() {- return false;- }- let header_cutoff = page.page_height * HEADER_BAND_FRACTION;- let footer_cutoff = page.page_height * (1.0 - FOOTER_BAND_FRACTION);- let in_band = line.bbox.y <= header_cutoff || line.bbox.y + line.bbox.height >= footer_cutoff;- if !in_band {- return false;- }- let norm = normalize_for_repetition(line.text.trim());- header_footer.contains(&norm)-}--/// Classify a single page's `ProjectedLine`s into blocks.-pub fn classify_page(page: &ParsedPage, heading_map: &[(f32, u8)]) -> Vec<Block> {- classify_page_with_filters(page, heading_map, &std::collections::HashSet::new())-}--/// Same as `classify_page` but also strips lines matching a precomputed-/// running header/footer set. Use this when emitting a whole document so-/// repeating chrome (titles, page numbers) doesn't show up in every page.-pub fn classify_page_with_filters(- page: &ParsedPage,- heading_map: &[(f32, u8)],- header_footer: &std::collections::HashSet<String>,-) -> Vec<Block> {- let mut blocks: Vec<Block> = Vec::new();- let mut paragraph: Option<ParaAccum> = None;- // Active fenced code block being accumulated (consecutive `all_mono` lines).- let mut code: Option<Vec<String>> = None;- // Tracks the "level 0" indent of the current contiguous list run so we can- // bucket deeper items into nesting levels. Reset whenever a non-list block- // breaks the run.- let mut list_base_indent: Option<f32> = None;- // Index into `blocks` of the most recent ListItem in the current run — used- // to merge wrapped continuation lines into the same item.- let mut last_list_item_idx: Option<usize> = None;- // Most recent ProjectedLine appended to the active list item, for- // gap/font-size checks on continuation lines.- let mut last_list_line: Option<ProjectedLine> = None;-- let flush_paragraph = |blocks: &mut Vec<Block>, p: Option<ParaAccum>| {- if let Some(acc) = p- && !acc.raw.trim().is_empty()- {- blocks.push(paragraph_from_accum(acc));- }- };- let flush_code = |blocks: &mut Vec<Block>, c: Option<Vec<String>>| {- if let Some(lines) = c- && !lines.is_empty()- {- blocks.push(Block::CodeBlock { lines });- }- };-- let debug = std::env::var("LITEPARSE_DEBUG_MD").is_ok();-- // Strip running header/footer lines up-front so they don't leak into- // table detection (a repeating two-column footer would otherwise look- // like a 2-row table) or paragraph grouping.- let filtered_owned: Vec<ProjectedLine> = if header_footer.is_empty() {- Vec::new()- } else {- page.projected_lines- .iter()- .filter(|l| !is_header_or_footer(l, page, header_footer))- .cloned()- .collect()- };- let lines: &[ProjectedLine] = if header_footer.is_empty() {- &page.projected_lines- } else {- &filtered_owned- };-- // Pre-pass: detect tabular regions so the per-line classifier below can- // skip over them. Tables take priority over heading / paragraph / list- // classification because a row of bold short cells would otherwise be- // misread as a heading or list item.- //- // Two detectors run in sequence: ruled-grid (path-based, strongest signal)- // and the borderless column-alignment fallback. Where ranges overlap, the- // ruled output wins.- let ruled_runs = detect_ruled_tables(lines, &page.graphics, page.page_width, page.page_height);- let borderless_runs = detect_tables(lines);- let table_runs = merge_table_runs(ruled_runs, borderless_runs);-- // Suppress HRs that fall inside a detected table's y-range — they're the- // table's own row dividers, not document-level section breaks. Build the- // y-extents once before we move table_runs into the iterator.- // Extend each table's HR-suppression band upward to cover any- // header/sub-header rows we didn't absorb into the table. The expansion- // is a few row heights — large enough to catch a 2–3 line bold/italic- // header sitting just above the table, small enough not to swallow a- // real section divider belonging to a different block. The downward- // edge gets a small slack to catch HRs drawn flush with the last row.- const TABLE_HR_SUPPRESS_HEADROOM_ROWS: f32 = 4.0;- let table_y_extents: Vec<(f32, f32)> = table_runs- .iter()- .map(|run| {- let top_line = &lines[run.start];- let row_h = top_line.bbox.height.max(8.0);- let top = top_line.bbox.y - row_h * TABLE_HR_SUPPRESS_HEADROOM_ROWS;- let last = &lines[run.end.saturating_sub(1).max(run.start)];- let bot = last.bbox.y + last.bbox.height;- (top, bot)- })- .collect();-- let mut table_iter = table_runs.into_iter().peekable();-- // Pre-pass: detect horizontal rules from vector graphics so they can be- // emitted in document order between surrounding text lines.- let hr_ys: Vec<f32> = detect_horizontal_rules(page)- .into_iter()- .filter(|y| {- !table_y_extents- .iter()- .any(|(top, bot)| *y >= *top - 2.0 && *y <= *bot + 2.0)- })- .collect();- let mut hr_iter = hr_ys.into_iter().peekable();-- // Emit any HRs whose y is at or above `before_y`. Flushes the active- // paragraph/code/list state first so the rule lands as its own block.- let emit_hrs_before = |blocks: &mut Vec<Block>,- paragraph: &mut Option<ParaAccum>,- code: &mut Option<Vec<String>>,- list_base: &mut Option<f32>,- last_item: &mut Option<usize>,- last_line: &mut Option<ProjectedLine>,- hr_iter: &mut std::iter::Peekable<std::vec::IntoIter<f32>>,- before_y: f32| {- while let Some(&hy) = hr_iter.peek() {- if hy > before_y {- break;- }- hr_iter.next();- flush_paragraph(blocks, paragraph.take());- flush_code(blocks, code.take());- *list_base = None;- *last_item = None;- *last_line = None;- blocks.push(Block::HorizontalRule);- }- };-- let mut idx = 0;- while idx < lines.len() {- if let Some(run) = table_iter.peek()- && run.start == idx- {- // Flush any HRs above this table's top edge first.- let table_top = lines[run.start].bbox.y;- emit_hrs_before(- &mut blocks,- &mut paragraph,- &mut code,- &mut list_base_indent,- &mut last_list_item_idx,- &mut last_list_line,- &mut hr_iter,- table_top,- );- flush_paragraph(&mut blocks, paragraph.take());- flush_code(&mut blocks, code.take());- list_base_indent = None;- last_list_item_idx = None;- last_list_line = None;- let run = table_iter.next().unwrap();- blocks.push(run.block);- idx = run.end;- continue;- }- let line = &lines[idx];- // Emit any HRs that fall above this line.- emit_hrs_before(- &mut blocks,- &mut paragraph,- &mut code,- &mut list_base_indent,- &mut last_list_item_idx,- &mut last_list_line,- &mut hr_iter,- line.bbox.y,- );- idx += 1;- let text = line.text.trim();- if text.is_empty() {- continue;- }- // Skip rotated text (vertical sidebars, arXiv banners, watermarks).- // Including it would either inject a paragraph of disconnected- // characters or be misclassified as a heading.- if is_rotated_line(line) {- continue;- }- if debug {- eprintln!(- "[md] y={:.1} h={:.1} size={:.2} anchor={:?} indent={:.1} text={:?}",- line.bbox.y,- line.bbox.height,- line.dominant_font_size,- line.anchor,- line.indent_x,- text- );- }-- // Code block detection runs first so a mono heading-shaped line- // (rare but plausible — e.g., a code identifier in a large mono font)- // still becomes code. Mono content also wouldn't make a useful- // heading.- if line.all_mono {- flush_paragraph(&mut blocks, paragraph.take());- list_base_indent = None;- last_list_item_idx = None;- last_list_line = None;- code.get_or_insert_with(Vec::new)- .push(line.text.trim_end().to_string());- continue;- }- // Any non-mono line ends the current code block (if any).- flush_code(&mut blocks, code.take());-- if let Some(level) = heading_level_for(line.dominant_font_size, heading_map) {- flush_paragraph(&mut blocks, paragraph.take());- list_base_indent = None;- last_list_item_idx = None;- last_list_line = None;- blocks.push(Block::Heading {- level,- text: collapse_whitespace(text),- });- continue;- }-- // List item?- if let Some((ordered, marker, rest)) = parse_list_marker(text) {- flush_paragraph(&mut blocks, paragraph.take());- let base = *list_base_indent.get_or_insert(line.indent_x);- let level = (((line.indent_x - base) / LIST_INDENT_STEP_PT)- .round()- .max(0.0)) as u8;- last_list_item_idx = Some(blocks.len());- last_list_line = Some(line.clone());- // Render the list-item text via the inline pipeline so per-span- // emphasis surfaces. We strip the marker from `rest` (already- // done by `parse_list_marker`), but emphasis lives on `line.spans`,- // which still contain the marker span — render the line and then- // peel the marker off the front of the rendered string.- let item_text = render_list_item_text(line, &marker, rest);- blocks.push(Block::ListItem {- ordered,- marker,- level,- text: item_text,- bold: false,- italic: false,- });- continue;- }-- // Continuation of a list item: same gap/font rules as paragraphs.- // Footnote-style continuations often left-flush below the marker's- // hanging indent, so we don't require indent ≥ marker indent.- if let Some(idx) = last_list_item_idx- && let Some(prev_line) = last_list_line.as_ref()- && continues_paragraph(prev_line, line)- && let Some(Block::ListItem {- text: prev_text, ..- }) = blocks.get_mut(idx)- {- // De-hyphenate against the prior rendered text, then append the- // inline-styled continuation.- let cont_inline = render_line_inline(line);- append_inline_continuation(prev_text, text, &cont_inline);- last_list_line = Some(line.clone());- continue;- }-- // Bold body-size heading. Section headings in academic / technical- // PDFs are routinely body-sized + bold (e.g. "Abstract",- // "1 Introduction"); without this rule they look just like a bold- // first sentence of a paragraph. Runs after list-marker detection so- // bold bullet items stay as list items.- let prev_for_gap = paragraph- .as_ref()- .map(|p| &p.last)- .or(last_list_line.as_ref());- let next_for_gap = lines.get(idx);- if looks_like_bold_heading(line, prev_for_gap, next_for_gap) {- flush_paragraph(&mut blocks, paragraph.take());- list_base_indent = None;- last_list_item_idx = None;- last_list_line = None;- // Level: one deeper than the deepest size-based level we already- // have. With an empty heading_map this lands on H1; with a full- // 6-level map it caps at H6.- let level = (heading_map.len() as u8 + 1).clamp(1, MAX_HEADING_LEVELS as u8);- blocks.push(Block::Heading {- level,- text: collapse_whitespace(text),- });- continue;- }-- match paragraph.as_mut() {- Some(acc) if continues_paragraph(&acc.last, line) => {- append_to_paragraph(acc, line);- }- _ => {- flush_paragraph(&mut blocks, paragraph.take());- list_base_indent = None;- last_list_item_idx = None;- last_list_line = None;- let inline = render_line_inline(line);- let raw = collapse_whitespace(text);- let uniform = line_uniform_style(line).map(|s| (s.bold, s.italic));- paragraph = Some(ParaAccum {- raw,- inline,- last: line.clone(),- uniform,- });- }- }- }-- flush_paragraph(&mut blocks, paragraph.take());- flush_code(&mut blocks, code.take());- // Flush any trailing HRs that sat below the last text line.- emit_hrs_before(- &mut blocks,- &mut paragraph,- &mut code,- &mut list_base_indent,- &mut last_list_item_idx,- &mut last_list_line,- &mut hr_iter,- f32::INFINITY,- );- blocks-}--/// Wrap `text` in markdown emphasis markers based on `bold`/`italic`. Both →-/// `***text***`. Headings deliberately skip this (the `#` is the emphasis).-fn wrap_emphasis(text: &str, bold: bool, italic: bool) -> String {- if text.trim().is_empty() {- return text.to_string();- }- match (bold, italic) {- (true, true) => format!("***{text}***"),- (true, false) => format!("**{text}**"),- (false, true) => format!("*{text}*"),- (false, false) => text.to_string(),- }-}--/// Render a list of blocks to a markdown string.-pub fn render_blocks(blocks: &[Block]) -> String {- let mut out = String::new();- for (i, block) in blocks.iter().enumerate() {- if i > 0 {- // Consecutive list items render as a tight list (single newline).- // Everything else gets a blank line between blocks.- let tight = matches!(block, Block::ListItem { .. })- && matches!(blocks[i - 1], Block::ListItem { .. });- if tight {- out.push('\n');- } else {- out.push_str("\n\n");- }- }- match block {- Block::Heading { level, text } => {- let level = (*level).clamp(1, 6) as usize;- out.push_str(&"#".repeat(level));- out.push(' ');- out.push_str(text);- }- Block::Paragraph { text, bold, italic } => {- out.push_str(&wrap_emphasis(text, *bold, *italic));- }- Block::ListItem {- ordered,- marker,- level,- text,- bold,- italic,- } => {- let indent = " ".repeat((*level).min(6) as usize);- out.push_str(&indent);- if *ordered {- // Preserve the original marker (e.g. `138.` for footnotes- // or `iii)` for roman numerals) so semantic numbering- // survives the round-trip. CommonMark requires `1.` /- // `1)` style but most LLM consumers tolerate alt forms,- // and the alternative — renumbering as `1.` — drops info.- out.push_str(marker);- out.push(' ');- } else {- out.push_str("- ");- }- out.push_str(&wrap_emphasis(text, *bold, *italic));- }- Block::Table { header, rows } => {- let column_count = header- .as_ref()- .map(|h| h.len())- .or_else(|| rows.first().map(|r| r.len()))- .unwrap_or(0);- if column_count == 0 {- continue;- }- if let Some(h) = header {- out.push_str("| ");- for (i, cell) in h.iter().enumerate() {- if i > 0 {- out.push_str(" | ");- }- out.push_str(&escape_table_cell(cell));- }- out.push_str(" |\n");- } else {- // CommonMark/GFM requires a header row before the- // separator; synthesize a blank header so renderers that- // refuse header-less tables still display the body.- out.push_str("|");- for _ in 0..column_count {- out.push_str(" |");- }- out.push('\n');- }- out.push('|');- for _ in 0..column_count {- out.push_str("---|");- }- for row in rows {- out.push_str("\n| ");- for (i, cell) in row.iter().enumerate() {- if i > 0 {- out.push_str(" | ");- }- out.push_str(&escape_table_cell(cell));- }- out.push_str(" |");- }- }- Block::GridFallback { lines } => {- out.push_str("```text\n");- for line in lines {- out.push_str(line);- out.push('\n');- }- out.push_str("```");- }- Block::CodeBlock { lines } => {- // Pick a fence that doesn't appear inside the body. Standard- // triple-backtick handles ~all real-world code; bump to ~~~ if- // the body itself contains ``` (rare).- let fence = if lines.iter().any(|l| l.contains("```")) {- "~~~"- } else {- "```"- };- out.push_str(fence);- out.push('\n');- for line in lines {- out.push_str(line);- out.push('\n');- }- out.push_str(fence);- }- Block::HorizontalRule => {- out.push_str("---");- }- Block::Blank => {}- }- }- out-}--#[cfg(test)]-mod tests {- use super::*;- use crate::types::{Rect, TextItem};-- fn line(text: &str, x: f32, y: f32, h: f32, size: f32) -> ProjectedLine {- ProjectedLine {- text: text.into(),- bbox: Rect {- x,- y,- width: text.chars().count() as f32 * (size * 0.5),- height: h,- },- anchor: Anchor::Left,- indent_x: x,- dominant_font_size: size,- dominant_font_name: Some("Arial".into()),- all_bold: false,- all_italic: false,- all_mono: false,- all_strike: false,- spans: vec![TextItem::default()],- region_path: Vec::new(),- mcid: None,- }- }-- fn page(lines: Vec<ProjectedLine>) -> ParsedPage {- ParsedPage {- page_number: 1,- page_width: 612.0,- page_height: 792.0,- text: String::new(),- text_items: vec![],- projected_lines: lines,- regions: crate::types::Region::default(),- graphics: vec![],- figures: vec![],- }- }-- #[test]- fn body_size_picks_most_common() {- let pages = vec![page(vec![- line("Title", 50.0, 50.0, 18.0, 18.0),- line("body line one", 50.0, 80.0, 10.0, 10.0),- line("body line two", 50.0, 92.0, 10.0, 10.0),- line("body line three", 50.0, 104.0, 10.0, 10.0),- ])];- let body = compute_body_size(&pages);- assert!((body - 10.0).abs() < 0.01, "body size = {body}");- }-- #[test]- fn heading_map_descending_levels() {- // Heading text needs to clear `MIN_HEADING_TOTAL_CHARS` (25) so the- // size qualifies as a real heading rather than chart-legend noise.- let pages = vec![page(vec![- line("The largest heading on the page", 50.0, 50.0, 24.0, 24.0),- line("A smaller heading right below it", 50.0, 80.0, 18.0, 18.0),- // Several lines of body so it beats the heading text in the- // char-weighted body-size mode.- line(- "body text line one with plenty of content",- 50.0,- 110.0,- 10.0,- 10.0,- ),- line(- "body text line two with plenty of content",- 50.0,- 122.0,- 10.0,- 10.0,- ),- line(- "body text line three with even more content",- 50.0,- 134.0,- 10.0,- 10.0,- ),- line(- "body text line four with even more content",- 50.0,- 146.0,- 10.0,- 10.0,- ),- ])];- let body = compute_body_size(&pages);- let map = build_heading_map(&pages, body);- assert_eq!(map.len(), 2);- assert_eq!(map[0].1, 1);- assert_eq!(map[1].1, 2);- assert!(map[0].0 > map[1].0);- }-- #[test]- fn classify_emits_heading_and_paragraph() {- let p = page(vec![- line("Title of the document goes here", 50.0, 50.0, 18.0, 18.0),- line("First sentence of the para-", 50.0, 80.0, 10.0, 10.0),- line("graph continues here.", 50.0, 92.0, 10.0, 10.0),- line("Another paragraph.", 50.0, 130.0, 10.0, 10.0),- ]);- let pages = vec![p];- let body = compute_body_size(&pages);- let map = build_heading_map(&pages, body);- let blocks = classify_page(&pages[0], &map);- assert_eq!(blocks.len(), 3);- match &blocks[0] {- Block::Heading { level, text } => {- assert_eq!(*level, 1);- assert_eq!(text, "Title of the document goes here");- }- other => panic!("expected heading, got {other:?}"),- }- match &blocks[1] {- Block::Paragraph { text: t, .. } => {- assert!(t.contains("paragraph continues"), "got: {t}");- assert!(!t.contains("para-"), "de-hyphenation failed: {t}");- }- other => panic!("expected paragraph, got {other:?}"),- }- match &blocks[2] {- Block::Paragraph { text: t, .. } => assert_eq!(t, "Another paragraph."),- other => panic!("expected paragraph, got {other:?}"),- }- }-- #[test]- fn paragraph_break_on_big_gap() {- let p = page(vec![- line("Line A.", 50.0, 80.0, 10.0, 10.0),- line("Line B.", 50.0, 200.0, 10.0, 10.0),- ]);- let pages = vec![p];- let body = compute_body_size(&pages);- let map = build_heading_map(&pages, body);- let blocks = classify_page(&pages[0], &map);- assert_eq!(blocks.len(), 2);- }-- #[test]- fn render_blocks_formats_markdown() {- let blocks = vec![- Block::Heading {- level: 1,- text: "Title".into(),- },- Block::Paragraph {- text: "A paragraph.".into(),- bold: false,- italic: false,- },- Block::Heading {- level: 2,- text: "Sub".into(),- },- ];- let s = render_blocks(&blocks);- assert_eq!(s, "# Title\n\nA paragraph.\n\n## Sub");- }-- #[test]- fn parse_list_marker_bullets() {- let (ordered, marker, rest) = parse_list_marker("• item one").unwrap();- assert!(!ordered);- assert_eq!(marker, "•");- assert_eq!(rest, "item one");- }-- #[test]- fn parse_list_marker_decimal() {- let (ordered, marker, rest) = parse_list_marker("1. first").unwrap();- assert!(ordered);- assert_eq!(marker, "1.");- assert_eq!(rest, "first");-- let (ordered, marker, rest) = parse_list_marker("12) twelfth").unwrap();- assert!(ordered);- assert_eq!(marker, "12)");- assert_eq!(rest, "twelfth");- }-- #[test]- fn parse_list_marker_rejects_prose() {- assert!(parse_list_marker("This sentence.").is_none());- // Bare digit with no terminator → not a list- assert!(parse_list_marker("2023 was a year").is_none());- // Footnote caller / page number style — no whitespace after- assert!(parse_list_marker("1.5x growth").is_none());- }-- #[test]- fn classify_emits_list_items() {- let p = page(vec![- line("Intro line.", 50.0, 50.0, 10.0, 10.0),- line("• first bullet", 60.0, 80.0, 10.0, 10.0),- line("• second bullet", 60.0, 92.0, 10.0, 10.0),- line("◦ nested item", 72.0, 104.0, 10.0, 10.0),- line("• back to top", 60.0, 116.0, 10.0, 10.0),- ]);- let pages = vec![p];- let body = compute_body_size(&pages);- let map = build_heading_map(&pages, body);- let blocks = classify_page(&pages[0], &map);- let list_items: Vec<&Block> = blocks- .iter()- .filter(|b| matches!(b, Block::ListItem { .. }))- .collect();- assert_eq!(list_items.len(), 4);- if let Block::ListItem { level, text, .. } = list_items[0] {- assert_eq!(*level, 0);- assert_eq!(text, "first bullet");- } else {- panic!();- }- // The "- nested item" line is indented +12pt from the base bullet.- if let Block::ListItem { level, .. } = list_items[2] {- assert_eq!(*level, 1);- } else {- panic!();- }- }-- #[test]- fn render_lists_are_tight() {- let blocks = vec![- Block::Paragraph {- text: "Intro.".into(),- bold: false,- italic: false,- },- Block::ListItem {- ordered: false,- marker: "•".into(),- level: 0,- text: "a".into(),- bold: false,- italic: false,- },- Block::ListItem {- ordered: false,- marker: "•".into(),- level: 0,- text: "b".into(),- bold: false,- italic: false,- },- Block::Paragraph {- text: "Outro.".into(),- bold: false,- italic: false,- },- ];- let s = render_blocks(&blocks);- assert_eq!(s, "Intro.\n\n- a\n- b\n\nOutro.");-- // Ordered: original marker preserved- let s = render_blocks(&[- Block::ListItem {- ordered: true,- marker: "138.".into(),- level: 0,- text: "footnote".into(),- bold: false,- italic: false,- },- Block::ListItem {- ordered: true,- marker: "139.".into(),- level: 0,- text: "next footnote".into(),- bold: false,- italic: false,- },- ]);- assert_eq!(s, "138. footnote\n139. next footnote");- }-- fn mono_line(text: &str, y: f32) -> ProjectedLine {- let mut l = line(text, 50.0, y, 10.0, 10.0);- l.all_mono = true;- l- }-- #[test]- fn classify_emits_code_block() {- let p = page(vec![- line("Intro line.", 50.0, 50.0, 10.0, 10.0),- mono_line(" let x = 1;", 80.0),- mono_line(" let y = x + 2;", 92.0),- line("After the code.", 50.0, 120.0, 10.0, 10.0),- ]);- let pages = vec![p];- let body = compute_body_size(&pages);- let map = build_heading_map(&pages, body);- let blocks = classify_page(&pages[0], &map);- // Expect: Paragraph("Intro line."), CodeBlock(2 lines), Paragraph("After...")- assert_eq!(blocks.len(), 3);- match &blocks[1] {- Block::CodeBlock { lines } => {- assert_eq!(lines.len(), 2);- assert!(lines[0].contains("let x = 1;"));- assert!(lines[1].contains("let y = x + 2;"));- }- other => panic!("expected code block, got {other:?}"),- }- let s = render_blocks(&blocks);- assert!(s.contains("```\n let x = 1;"));- assert!(s.ends_with("After the code."));- }-- #[test]- fn classify_marks_paragraph_bold_when_all_lines_bold() {- let mut a = line("Bold line one.", 50.0, 50.0, 10.0, 10.0);- let mut b = line("bold continuation.", 50.0, 62.0, 10.0, 10.0);- // Mark the underlying spans as bold so per-span style detection sees- // it — the new inline pipeline reads from `spans`, not the per-line- // `all_bold` shortcut flag.- let bold_span = TextItem {- text: "x".into(),- font_name: Some("Arial-Bold".into()),- ..Default::default()- };- a.spans = vec![bold_span.clone()];- b.spans = vec![bold_span];- a.all_bold = true;- b.all_bold = true;- let p = page(vec![a, b]);- let pages = vec![p];- let body = compute_body_size(&pages);- let map = build_heading_map(&pages, body);- let blocks = classify_page(&pages[0], &map);- assert_eq!(blocks.len(), 1);- match &blocks[0] {- Block::Paragraph { bold, italic, .. } => {- assert!(*bold);- assert!(!*italic);- }- other => panic!("expected paragraph, got {other:?}"),- }- let s = render_blocks(&blocks);- assert!(s.starts_with("**") && s.ends_with("**"), "got: {s}");- }-- #[test]- fn render_emphasis_combinations() {- assert_eq!(wrap_emphasis("hi", false, false), "hi");- assert_eq!(wrap_emphasis("hi", true, false), "**hi**");- assert_eq!(wrap_emphasis("hi", false, true), "*hi*");- assert_eq!(wrap_emphasis("hi", true, true), "***hi***");- }-- #[test]- fn code_block_escapes_internal_fence() {- let blocks = vec![Block::CodeBlock {- lines: vec!["body containing ``` backticks".into()],- }];- let s = render_blocks(&blocks);- assert!(s.starts_with("~~~\n"));- assert!(s.ends_with("~~~"));- }-- /// Build a line whose spans are placed at explicit x positions — used to- /// drive the table detector, which relies on per-span x for cell splitting.- fn line_with_spans(cells: &[(&str, f32)], y: f32, size: f32) -> ProjectedLine {- let spans: Vec<TextItem> = cells- .iter()- .map(|(t, x)| TextItem {- text: (*t).into(),- x: *x,- y,- width: t.chars().count() as f32 * size * 0.5,- height: size,- font_size: Some(size),- font_name: Some("Arial".into()),- ..Default::default()- })- .collect();- let min_x = spans.iter().map(|s| s.x).fold(f32::INFINITY, f32::min);- let max_x = spans- .iter()- .map(|s| s.x + s.width)- .fold(f32::NEG_INFINITY, f32::max);- ProjectedLine {- text: cells- .iter()- .map(|(t, _)| *t)- .collect::<Vec<_>>()- .join(" "),- bbox: Rect {- x: min_x,- y,- width: (max_x - min_x).max(0.0),- height: size,- },- anchor: Anchor::Left,- indent_x: min_x,- dominant_font_size: size,- dominant_font_name: Some("Arial".into()),- all_bold: false,- all_italic: false,- all_mono: false,- all_strike: false,- spans,- region_path: Vec::new(),- mcid: None,- }- }-- #[test]- fn split_cells_splits_on_wide_gaps() {- let l = line_with_spans(&[("A", 50.0), ("B", 150.0), ("C", 250.0)], 100.0, 10.0);- let cells = split_cells(&l);- assert_eq!(cells.len(), 3);- assert_eq!(cells[0].text, "A");- assert_eq!(cells[1].text, "B");- assert_eq!(cells[2].text, "C");- }-- #[test]- fn recover_merged_cell_splits_off_by_one() {- // Mimics the page-6 case: row 0 establishes 3 tracks at 50/150/250.- // Row 1's projection merges "MEMORYBANK" + "5.00" into one span at- // x=50 width=110, so split_cells yields 2 cells while the table- // expects 3. Recovery must split on whitespace at the missing track.- let row = vec![- TableCell {- start_x: 50.0,- end_x: 160.0,- text: "MEMORYBANK 5.00".into(),- bold: false,- },- TableCell {- start_x: 250.0,- end_x: 280.0,- text: "4.77".into(),- bold: false,- },- ];- let tracks = vec![50.0, 150.0, 250.0];- let out = recover_merged_cell(row, &tracks).expect("recovery should succeed");- assert_eq!(out.len(), 3);- assert_eq!(out[0].text, "MEMORYBANK");- assert_eq!(out[1].text, "5.00");- assert_eq!(out[2].text, "4.77");- }-- #[test]- fn recover_merged_cell_splits_off_by_two() {- // Three merged tokens in one cell: "MEMORYBANK 13.18 10.03" straddles- // tracks at 50/150/250 and the row has only 2 cells, off by 2.- let row = vec![- TableCell {- start_x: 50.0,- end_x: 260.0,- text: "MEMORYBANK 13.18 10.03".into(),- bold: false,- },- TableCell {- start_x: 350.0,- end_x: 380.0,- text: "7.61".into(),- bold: false,- },- ];- let tracks = vec![50.0, 150.0, 250.0, 350.0];- let out = recover_merged_cell(row, &tracks).expect("recovery should succeed");- assert_eq!(out.len(), 4);- assert_eq!(out[0].text, "MEMORYBANK");- assert_eq!(out[1].text, "13.18");- assert_eq!(out[2].text, "10.03");- assert_eq!(out[3].text, "7.61");- }-- #[test]- fn recover_merged_cell_bails_without_enough_whitespace() {- // A cell that straddles two tracks but has no internal whitespace- // (e.g. a hyphenated token) can't be safely split — return None.- let row = vec![TableCell {- start_x: 50.0,- end_x: 200.0,- text: "ABC-DEF-GHI".into(),- bold: false,- }];- let tracks = vec![50.0, 150.0];- assert!(recover_merged_cell(row, &tracks).is_none());- }-- #[test]- fn split_cells_keeps_close_spans_together() {- // Two spans 2pt apart at 10pt font (gap < font_size) → same cell.- let l = line_with_spans(&[("Hello", 50.0), ("world", 80.0)], 100.0, 10.0);- let cells = split_cells(&l);- assert_eq!(cells.len(), 1);- assert_eq!(cells[0].text, "Hello world");- }-- #[test]- fn detects_simple_borderless_table() {- let lines = vec![- line_with_spans(- &[("Name", 50.0), ("Age", 150.0), ("City", 250.0)],- 100.0,- 10.0,- ),- line_with_spans(- &[("Alice", 50.0), ("30", 150.0), ("NYC", 250.0)],- 115.0,- 10.0,- ),- line_with_spans(&[("Bob", 50.0), ("25", 150.0), ("LA", 250.0)], 130.0, 10.0),- ];- let p = page(lines);- let pages = vec![p];- let body = compute_body_size(&pages);- let map = build_heading_map(&pages, body);- let blocks = classify_page(&pages[0], &map);- assert_eq!(blocks.len(), 1, "got: {blocks:?}");- match &blocks[0] {- Block::Table { header, rows } => {- // Header isn't bold so no header row promoted.- assert!(header.is_none());- assert_eq!(rows.len(), 3);- assert_eq!(rows[0][0], "Name");- assert_eq!(rows[1][2], "NYC");- }- other => panic!("expected table, got {other:?}"),- }- }-- #[test]- fn rejects_table_when_row_count_too_low() {- let lines = vec![line_with_spans(- &[("A", 50.0), ("B", 150.0), ("C", 250.0)],- 100.0,- 10.0,- )];- let runs = detect_tables(&lines);- assert!(runs.is_empty());- }-- #[test]- fn rejects_table_when_column_count_too_low() {- let lines = vec![- line_with_spans(&[("A", 50.0), ("B", 200.0)], 100.0, 10.0),- line_with_spans(&[("C", 50.0), ("D", 200.0)], 115.0, 10.0),- ];- let runs = detect_tables(&lines);- assert!(runs.is_empty());- }-- #[test]- fn renders_table_to_pipe_format() {- let blocks = vec![Block::Table {- header: Some(vec!["a".into(), "b".into()]),- rows: vec![vec!["1".into(), "2".into()], vec!["3".into(), "4".into()]],- }];- let s = render_blocks(&blocks);- assert_eq!(s, "| a | b |\n|---|---|\n| 1 | 2 |\n| 3 | 4 |");- }-- #[test]- fn render_table_without_header_synthesizes_blank_header() {- let blocks = vec![Block::Table {- header: None,- rows: vec![vec!["1".into(), "2".into()]],- }];- let s = render_blocks(&blocks);- // GFM/CommonMark needs a header row before the separator; we emit a- // blank one so renderers don't choke.- assert!(s.contains("|---|---|"));- assert!(s.ends_with("| 1 | 2 |"));- }-- #[test]- fn normalize_collapses_digits_and_case() {- assert_eq!(normalize_for_repetition("Page 1 of 6"), "page # of #");- assert_eq!(normalize_for_repetition("PAGE 12 OF 6"), "page # of #");- assert_eq!(normalize_for_repetition("Confidential"), "confidential");- }-- fn header_footer_page(n: usize, header: &str, footer: &str, body: &str) -> ParsedPage {- // Page height 100 → header band ≤12pt, footer band ≥88pt.- let mut lines = vec![- line(header, 50.0, 5.0, 8.0, 8.0),- line(body, 50.0, 50.0, 10.0, 10.0),- line(footer, 50.0, 92.0, 6.0, 6.0),- ];- for l in &mut lines {- l.region_path = Vec::new();- }- ParsedPage {- page_number: n,- page_width: 612.0,- page_height: 100.0,- text: String::new(),- text_items: vec![],- projected_lines: lines,- regions: crate::types::Region::default(),- graphics: vec![],- figures: vec![],- }- }-- #[test]- fn detects_repeating_header_and_footer() {- let pages = vec![- header_footer_page(1, "Acme Confidential", "Page 1 of 3", "Body one."),- header_footer_page(2, "Acme Confidential", "Page 2 of 3", "Body two."),- header_footer_page(3, "Acme Confidential", "Page 3 of 3", "Body three."),- ];- let set = compute_header_footer_set(&pages);- assert!(set.contains("acme confidential"));- assert!(set.contains("page # of #"));- }-- #[test]- fn skips_repetition_check_on_single_page() {- let pages = vec![header_footer_page(1, "Solo", "footer", "body")];- let set = compute_header_footer_set(&pages);- assert!(set.is_empty());- }-- #[test]- fn body_text_not_classified_as_header() {- // Same text in the body of every page should NOT be stripped — only- // text within the top/bottom band qualifies.- let mut pages = Vec::new();- for n in 1..=3 {- let mut p = header_footer_page(n, "unique header", "unique footer", "shared body text");- // Move "shared body text" out of header/footer bands (already at y=50 in mid-page).- // No-op — just illustrating intent.- p.projected_lines[0].text = format!("unique header {n}");- p.projected_lines[2].text = format!("unique footer {n}");- pages.push(p);- }- let set = compute_header_footer_set(&pages);- // "shared body text" sits mid-page — never matched.- assert!(!set.contains("shared body text"));- }-- #[test]- fn full_format_strips_header_footer() {- let pages = vec![- header_footer_page(1, "Acme Confidential", "Page 1 of 2", "First page body."),- header_footer_page(2, "Acme Confidential", "Page 2 of 2", "Second page body."),- ];- let body = compute_body_size(&pages);- let map = build_heading_map(&pages, body);- let set = compute_header_footer_set(&pages);- let blocks = classify_page_with_filters(&pages[0], &map, &set);- let s = render_blocks(&blocks);- assert!(!s.contains("Acme Confidential"), "got: {s}");- assert!(!s.contains("Page 1 of 2"), "got: {s}");- assert!(s.contains("First page body."));- }-- #[test]- fn escapes_pipe_inside_cell() {- assert_eq!(escape_table_cell("a|b"), "a\\|b");- }-- #[test]- fn dehyphenation_only_for_lowercase_followups() {- let mut s = String::from("co-");- append_paragraph_line(&mut s, "operate");- assert_eq!(s, "cooperate");-- let mut s = String::from("Vitamin-");- append_paragraph_line(&mut s, "A");- assert_eq!(s, "Vitamin- A");- }-- /// Build a line whose spans carry explicit per-span font metadata. Lets us- /// exercise the mid-line emphasis pipeline without needing real PDF input.- fn styled_line(spans: &[(&str, f32, Option<&str>)], y: f32, size: f32) -> ProjectedLine {- let items: Vec<TextItem> = spans- .iter()- .map(|(t, x, font)| TextItem {- text: (*t).into(),- x: *x,- y,- width: t.chars().count() as f32 * size * 0.5,- height: size,- font_size: Some(size),- font_name: font.map(String::from),- ..Default::default()- })- .collect();- let joined: String = spans- .iter()- .map(|(t, _, _)| *t)- .collect::<Vec<_>>()- .join(" ");- let min_x = items.iter().map(|s| s.x).fold(f32::INFINITY, f32::min);- let max_x = items- .iter()- .map(|s| s.x + s.width)- .fold(f32::NEG_INFINITY, f32::max);- ProjectedLine {- text: joined,- bbox: Rect {- x: min_x,- y,- width: (max_x - min_x).max(0.0),- height: size,- },- anchor: Anchor::Left,- indent_x: min_x,- dominant_font_size: size,- dominant_font_name: Some("Arial".into()),- all_bold: false,- all_italic: false,- all_mono: false,- all_strike: false,- spans: items,- region_path: Vec::new(),- mcid: None,- }- }-- #[test]- fn render_line_inline_mid_line_bold() {- // Plain span, then bold span: should produce a mid-line `**bold**` run.- let l = styled_line(- &[- ("regular text with", 50.0, Some("Arial")),- ("bold word", 200.0, Some("Arial-Bold")),- ],- 100.0,- 10.0,- );- let out = render_line_inline(&l);- assert!(out.contains("regular text with"), "got: {out}");- assert!(out.contains("**bold word**"), "got: {out}");- assert!(- !out.starts_with("**"),- "mid-line shouldn't open with bold: {out}"- );- }-- #[test]- fn render_line_inline_uniform_bold_uses_shortcut() {- // All spans bold → single outer wrap, no per-span noise.- let l = styled_line(- &[- ("first", 50.0, Some("Arial-Bold")),- ("second", 100.0, Some("Arial-Bold")),- ],- 100.0,- 10.0,- );- let out = render_line_inline(&l);- assert!(out.starts_with("**") && out.ends_with("**"), "got: {out}");- // Only one bold run, not two — the shortcut should kick in.- assert_eq!(out.matches("**").count(), 2, "got: {out}");- }-- #[test]- fn render_line_inline_escapes_emphasis_chars() {- let l = styled_line(&[("5*4=20", 50.0, Some("Arial"))], 100.0, 10.0);- let out = render_line_inline(&l);- assert_eq!(out, "5\\*4=20");- }-- #[test]- fn render_line_inline_italic_then_bold() {- let l = styled_line(- &[- ("italic", 50.0, Some("Arial-Italic")),- ("plain", 100.0, Some("Arial")),- ("bold", 150.0, Some("Arial-Bold")),- ],- 100.0,- 10.0,- );- let out = render_line_inline(&l);- assert!(out.contains("*italic*"), "got: {out}");- assert!(out.contains("plain"), "got: {out}");- assert!(out.contains("**bold**"), "got: {out}");- }-- #[test]- fn render_line_inline_mono_span() {- let l = styled_line(- &[- ("call", 50.0, Some("Arial")),- ("foo()", 100.0, Some("Courier")),- ("on it", 150.0, Some("Arial")),- ],- 100.0,- 10.0,- );- let out = render_line_inline(&l);- assert!(out.contains("`foo()`"), "got: {out}");- // Plain spans stay unwrapped.- assert!(out.contains("call"));- assert!(out.contains("on it"));- }-- #[test]- fn classify_paragraph_with_mid_line_bold() {- // First line has a bold word mid-line → not uniformly styled; paragraph- // should emit baked-in `**bold**` inside the text and `bold=false` at- // the block level.- let a = styled_line(- &[- ("a sentence with a", 50.0, Some("Arial")),- ("bold", 180.0, Some("Arial-Bold")),- ("word in it.", 230.0, Some("Arial")),- ],- 50.0,- 10.0,- );- let p = page(vec![a]);- let pages = vec![p];- let body = compute_body_size(&pages);- let map = build_heading_map(&pages, body);- let blocks = classify_page(&pages[0], &map);- assert_eq!(blocks.len(), 1, "got: {blocks:?}");- match &blocks[0] {- Block::Paragraph { text, bold, italic } => {- assert!(!*bold, "mixed-style paragraph shouldn't set block bold");- assert!(!*italic);- assert!(text.contains("**bold**"), "got: {text}");- }- other => panic!("expected paragraph, got {other:?}"),- }- }-- #[test]- fn classify_list_item_strips_marker_under_emphasis() {- // Whole bullet line is bold (marker + text). Rendered text should be- // wrapped, with the marker dropped (the renderer prints it).- let l = styled_line(- &[- ("•", 60.0, Some("Arial-Bold")),- ("important item", 80.0, Some("Arial-Bold")),- ],- 50.0,- 10.0,- );- let p = page(vec![l]);- let pages = vec![p];- let body = compute_body_size(&pages);- let map = build_heading_map(&pages, body);- let blocks = classify_page(&pages[0], &map);- assert_eq!(blocks.len(), 1);- match &blocks[0] {- Block::ListItem { text, .. } => {- assert_eq!(text, "**important item**");- }- other => panic!("expected list item, got {other:?}"),- }- }-- fn stroke(x1: f32, y1: f32, x2: f32, y2: f32, width: f32) -> GraphicPrimitive {- GraphicPrimitive::Stroke {- x1,- y1,- x2,- y2,- width,- color: None,- }- }-- fn page_with_graphics(- lines: Vec<ProjectedLine>,- graphics: Vec<GraphicPrimitive>,- ) -> ParsedPage {- let mut p = page(lines);- p.graphics = graphics;- p- }-- #[test]- fn hr_long_thin_horizontal_stroke_detected() {- // 400pt wide stroke on a 612pt page → comfortably above 30% threshold.- let p = page_with_graphics(vec![], vec![stroke(50.0, 200.0, 450.0, 200.5, 0.5)]);- let ys = detect_horizontal_rules(&p);- assert_eq!(ys, vec![200.25]);- }-- #[test]- fn hr_short_stroke_rejected() {- // 50pt wide — table border or list bullet, not an HR.- let p = page_with_graphics(vec![], vec![stroke(50.0, 200.0, 100.0, 200.0, 0.5)]);- assert!(detect_horizontal_rules(&p).is_empty());- }-- #[test]- fn hr_vertical_stroke_rejected() {- let p = page_with_graphics(vec![], vec![stroke(50.0, 50.0, 50.0, 500.0, 0.5)]);- assert!(detect_horizontal_rules(&p).is_empty());- }-- #[test]- fn hr_thick_stroke_rejected() {- // 4pt-thick stroke → a filled bar, not a rule.- let p = page_with_graphics(vec![], vec![stroke(50.0, 200.0, 450.0, 200.0, 4.0)]);- assert!(detect_horizontal_rules(&p).is_empty());- }-- #[test]- fn hr_underline_at_text_baseline_dropped() {- // Text line at y=100 height=10 → bottom at y=110. Stroke at y=111 within- // the line's horizontal extent → underline, not an HR.- let text_line = line(- "Some underlined heading text on the page",- 50.0,- 100.0,- 10.0,- 10.0,- );- let bottom = text_line.bbox.y + text_line.bbox.height;- let p = page_with_graphics(- vec![text_line.clone()],- vec![stroke(- 50.0,- bottom + 1.0,- 50.0 + text_line.bbox.width,- bottom + 1.0,- 0.5,- )],- );- assert!(detect_horizontal_rules(&p).is_empty());- }-- /// Helper: build the four borders of a rectangle as four strokes.- fn rect_borders(x: f32, y: f32, w: f32, h: f32) -> Vec<GraphicPrimitive> {- vec![- stroke(x, y, x + w, y, 0.5), // top- stroke(x, y + h, x + w, y + h, 0.5), // bottom- stroke(x, y, x, y + h, 0.5), // left- stroke(x + w, y, x + w, y + h, 0.5), // right- ]- }-- #[test]- fn ruled_table_2x2_detected() {- // 2 rows × 2 cols grid: 3 H lines (y=100,140,180), 3 V lines (x=50,150,250)- // Cell text dropped in the centroid of each cell.- let mut graphics = Vec::new();- for y in [100.0_f32, 140.0, 180.0] {- graphics.push(stroke(50.0, y, 250.0, y, 0.5));- }- for x in [50.0_f32, 150.0, 250.0] {- graphics.push(stroke(x, 100.0, x, 180.0, 0.5));- }-- // Text lines: one per cell, centered.- let lines = vec![- line("a", 90.0, 115.0, 10.0, 10.0), // row 0, col 0- line("b", 190.0, 115.0, 10.0, 10.0), // row 0, col 1- line("c", 90.0, 155.0, 10.0, 10.0), // row 1, col 0- line("d", 190.0, 155.0, 10.0, 10.0), // row 1, col 1- ];-- let runs = detect_ruled_tables(&lines, &graphics, 612.0, 792.0);- assert_eq!(runs.len(), 1, "expected 1 ruled table, got {runs:?}");- match &runs[0].block {- Block::Table { header, rows } => {- assert!(header.is_none(), "no bold first row → no header");- assert_eq!(rows.len(), 2);- assert_eq!(rows[0], vec!["a", "b"]);- assert_eq!(rows[1], vec!["c", "d"]);- }- other => panic!("expected Block::Table, got {other:?}"),- }- }-- #[test]- fn ruled_table_rect_borders_detected() {- // Same 2×2 table but drawn as 4 individual cell rects (each cell is a- // stroked rectangle). Each rect contributes 4 strokes via- // extract_h_v_segments.- let mut graphics = Vec::new();- graphics.extend(rect_borders(50.0, 100.0, 100.0, 40.0)); // r0 c0- graphics.extend(rect_borders(150.0, 100.0, 100.0, 40.0)); // r0 c1- graphics.extend(rect_borders(50.0, 140.0, 100.0, 40.0)); // r1 c0- graphics.extend(rect_borders(150.0, 140.0, 100.0, 40.0)); // r1 c1-- let lines = vec![- line("a", 90.0, 115.0, 10.0, 10.0),- line("b", 190.0, 115.0, 10.0, 10.0),- line("c", 90.0, 155.0, 10.0, 10.0),- line("d", 190.0, 155.0, 10.0, 10.0),- ];- let runs = detect_ruled_tables(&lines, &graphics, 612.0, 792.0);- assert_eq!(runs.len(), 1);- }-- #[test]- fn ruled_table_page_border_rejected() {- // Single big rect covering ~the whole page → should NOT be treated as a- // table even though it has H+V lines on all four sides.- let graphics = rect_borders(10.0, 10.0, 590.0, 770.0);- let lines = vec![line("body text", 50.0, 400.0, 10.0, 10.0)];- let runs = detect_ruled_tables(&lines, &graphics, 612.0, 792.0);- assert!(- runs.is_empty(),- "page-border rect should not become a table, got {runs:?}"- );- }-- #[test]- fn ruled_table_mostly_empty_rejected() {- // 3×3 grid with text in only one cell — empty fraction 8/9 ≈ 89% >> 30%.- let mut graphics = Vec::new();- for y in [100.0_f32, 130.0, 160.0, 190.0] {- graphics.push(stroke(50.0, y, 350.0, y, 0.5));- }- for x in [50.0_f32, 150.0, 250.0, 350.0] {- graphics.push(stroke(x, 100.0, x, 190.0, 0.5));- }- let lines = vec![line("only", 90.0, 115.0, 10.0, 10.0)];- let runs = detect_ruled_tables(&lines, &graphics, 612.0, 792.0);- assert!(runs.is_empty());- }-- #[test]- fn ruled_table_first_row_bold_becomes_header() {- // 2×2 with first row text marked all_bold → header promotion.- let mut graphics = Vec::new();- for y in [100.0_f32, 140.0, 180.0] {- graphics.push(stroke(50.0, y, 250.0, y, 0.5));- }- for x in [50.0_f32, 150.0, 250.0] {- graphics.push(stroke(x, 100.0, x, 180.0, 0.5));- }- let mut a = line("Name", 90.0, 115.0, 10.0, 10.0);- let mut b = line("Score", 190.0, 115.0, 10.0, 10.0);- a.all_bold = true;- b.all_bold = true;- let lines = vec![- a,- b,- line("alice", 90.0, 155.0, 10.0, 10.0),- line("99", 190.0, 155.0, 10.0, 10.0),- ];- let runs = detect_ruled_tables(&lines, &graphics, 612.0, 792.0);- assert_eq!(runs.len(), 1);- match &runs[0].block {- Block::Table { header, rows } => {- assert_eq!(- header.as_deref(),- Some(&["Name".into(), "Score".into()][..])- );- assert_eq!(rows.len(), 1);- assert_eq!(rows[0], vec!["alice", "99"]);- }- other => panic!("expected Block::Table, got {other:?}"),- }- }-- #[test]- fn merge_prefers_ruled_when_overlapping() {- let ruled = vec![TableRun {- start: 5,- end: 10,- block: Block::Table {- header: None,- rows: vec![vec!["ruled".into()]],- },- }];- let borderless = vec![TableRun {- start: 6,- end: 11,- block: Block::GridFallback {- lines: vec!["bl".into()],- },- }];- let merged = merge_table_runs(ruled, borderless);- assert_eq!(merged.len(), 1);- assert!(matches!(&merged[0].block, Block::Table { .. }));- }-- #[test]- fn hr_emitted_between_lines_by_y_order() {- let lines = vec![- line("before the rule", 50.0, 100.0, 10.0, 10.0),- line("after the rule", 50.0, 300.0, 10.0, 10.0),- ];- // Stroke between the two lines, far from either's baseline.- let p = page_with_graphics(lines, vec![stroke(50.0, 200.0, 450.0, 200.0, 0.5)]);- let blocks = classify_page(&p, &[]);- let has_hr = blocks- .iter()- .position(|b| matches!(b, Block::HorizontalRule));- assert!(has_hr.is_some(), "expected an HR block, got {blocks:?}");- // HR must land between the two paragraphs, not before/after both.- let pos = has_hr.unwrap();- assert!(pos > 0 && pos < blocks.len() - 1);- }-}diff --git a/crates/liteparse/src/markdown_layout/blocks.rs b/crates/liteparse/src/markdown_layout/blocks.rsnew file mode 100644index 0000000..a737a37--- /dev/null+++ b/crates/liteparse/src/markdown_layout/blocks.rs@@ -0,0 +1,347 @@+use super::inline::escape_inline;+use super::paragraphs::ParaAccum;+use super::tables::escape_table_cell;++/// Coarse block representation. Intentionally minimal — extended as later+/// build-order steps land (tables, figures).+#[derive(Debug, Clone)]+pub enum Block {+ Heading {+ level: u8,+ text: String,+ },+ Paragraph {+ text: String,+ bold: bool,+ italic: bool,+ },+ ListItem {+ ordered: bool,+ marker: String,+ level: u8,+ text: String,+ bold: bool,+ italic: bool,+ },+ /// Fenced code block — content rendered between triple-backtick fences.+ /// Each entry in `lines` is one source line; preserved as-is (only trailing+ /// whitespace stripped) so indentation survives.+ CodeBlock {+ lines: Vec<String>,+ },+ /// Confident table emitted as a markdown pipe table. `header` is `None`+ /// when the first row didn't qualify (e.g. wasn't bold and the table mode+ /// can't otherwise distinguish it).+ Table {+ header: Option<Vec<String>>,+ rows: Vec<Vec<String>>,+ },+ /// Tabular-looking region we couldn't classify confidently — rendered+ /// verbatim inside a fenced block to preserve visual structure for the+ /// downstream LLM. Strictly better than emitting a mangled pipe table.+ GridFallback {+ lines: Vec<String>,+ },+ /// A horizontal rule detected from a long thin horizontal stroke in the+ /// page's vector graphics (e.g. divider line between sections).+ HorizontalRule,+ /// Reference to a raster image on the page. Rendered as+ /// ``. Suppressed entirely when `ImageMode::Off`.+ Figure {+ id: String,+ bbox: crate::types::Rect,+ },+ Blank,+}++/// Resolve a `ParaAccum` to a `Block::Paragraph`. When the paragraph was+/// uniformly styled across all lines, return the raw text with block-level+/// `bold`/`italic` flags set so the renderer wraps it once. Otherwise return+/// the per-line inline-styled text with the flags cleared.+pub(super) fn paragraph_from_accum(accum: ParaAccum) -> Block {+ match accum.uniform {+ Some((bold, italic)) if bold || italic => Block::Paragraph {+ text: escape_inline(&accum.raw),+ bold,+ italic,+ },+ Some(_) => Block::Paragraph {+ // Uniformly plain — no emphasis markers anywhere, so the raw text+ // (with markdown specials escaped) is the right rendering.+ text: escape_inline(&accum.raw),+ bold: false,+ italic: false,+ },+ None => Block::Paragraph {+ text: accum.inline,+ bold: false,+ italic: false,+ },+ }+}++/// Wrap `text` in markdown emphasis markers based on `bold`/`italic`. Both →+/// `***text***`. Headings deliberately skip this (the `#` is the emphasis).+fn wrap_emphasis(text: &str, bold: bool, italic: bool) -> String {+ if text.trim().is_empty() {+ return text.to_string();+ }+ match (bold, italic) {+ (true, true) => format!("***{text}***"),+ (true, false) => format!("**{text}**"),+ (false, true) => format!("*{text}*"),+ (false, false) => text.to_string(),+ }+}++/// Render a list of blocks to a markdown string.+pub fn render_blocks(blocks: &[Block]) -> String {+ let mut out = String::new();+ for (i, block) in blocks.iter().enumerate() {+ if i > 0 {+ // Consecutive list items render as a tight list (single newline).+ // Everything else gets a blank line between blocks.+ let tight = matches!(block, Block::ListItem { .. })+ && matches!(blocks[i - 1], Block::ListItem { .. });+ if tight {+ out.push('\n');+ } else {+ out.push_str("\n\n");+ }+ }+ match block {+ Block::Heading { level, text } => {+ let level = (*level).clamp(1, 6) as usize;+ out.push_str(&"#".repeat(level));+ out.push(' ');+ out.push_str(text);+ }+ Block::Paragraph { text, bold, italic } => {+ out.push_str(&wrap_emphasis(text, *bold, *italic));+ }+ Block::ListItem {+ ordered,+ marker,+ level,+ text,+ bold,+ italic,+ } => {+ let indent = " ".repeat((*level).min(6) as usize);+ out.push_str(&indent);+ if *ordered {+ // Preserve the original marker (e.g. `138.` for footnotes+ // or `iii)` for roman numerals) so semantic numbering+ // survives the round-trip. CommonMark requires `1.` /+ // `1)` style but most LLM consumers tolerate alt forms,+ // and the alternative — renumbering as `1.` — drops info.+ out.push_str(marker);+ out.push(' ');+ } else {+ out.push_str("- ");+ }+ out.push_str(&wrap_emphasis(text, *bold, *italic));+ }+ Block::Table { header, rows } => {+ let column_count = header+ .as_ref()+ .map(|h| h.len())+ .or_else(|| rows.first().map(|r| r.len()))+ .unwrap_or(0);+ if column_count == 0 {+ continue;+ }+ if let Some(h) = header {+ out.push_str("| ");+ for (i, cell) in h.iter().enumerate() {+ if i > 0 {+ out.push_str(" | ");+ }+ out.push_str(&escape_table_cell(cell));+ }+ out.push_str(" |\n");+ } else {+ // CommonMark/GFM requires a header row before the+ // separator; synthesize a blank header so renderers that+ // refuse header-less tables still display the body.+ out.push_str("|");+ for _ in 0..column_count {+ out.push_str(" |");+ }+ out.push('\n');+ }+ out.push('|');+ for _ in 0..column_count {+ out.push_str("---|");+ }+ for row in rows {+ out.push_str("\n| ");+ for (i, cell) in row.iter().enumerate() {+ if i > 0 {+ out.push_str(" | ");+ }+ out.push_str(&escape_table_cell(cell));+ }+ out.push_str(" |");+ }+ }+ Block::GridFallback { lines } => {+ out.push_str("```text\n");+ for line in lines {+ out.push_str(line);+ out.push('\n');+ }+ out.push_str("```");+ }+ Block::CodeBlock { lines } => {+ // Pick a fence that doesn't appear inside the body. Standard+ // triple-backtick handles ~all real-world code; bump to ~~~ if+ // the body itself contains ``` (rare).+ let fence = if lines.iter().any(|l| l.contains("```")) {+ "~~~"+ } else {+ "```"+ };+ out.push_str(fence);+ out.push('\n');+ for line in lines {+ out.push_str(line);+ out.push('\n');+ }+ out.push_str(fence);+ }+ Block::HorizontalRule => {+ out.push_str("---");+ }+ Block::Figure { id, .. } => {+ out.push_str(";+ out.push_str(id);+ out.push_str(".png)");+ }+ Block::Blank => {}+ }+ }+ out+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn render_blocks_formats_markdown() {+ let blocks = vec![+ Block::Heading {+ level: 1,+ text: "Title".into(),+ },+ Block::Paragraph {+ text: "A paragraph.".into(),+ bold: false,+ italic: false,+ },+ Block::Heading {+ level: 2,+ text: "Sub".into(),+ },+ ];+ let s = render_blocks(&blocks);+ assert_eq!(s, "# Title\n\nA paragraph.\n\n## Sub");+ }++ #[test]+ fn render_lists_are_tight() {+ let blocks = vec![+ Block::Paragraph {+ text: "Intro.".into(),+ bold: false,+ italic: false,+ },+ Block::ListItem {+ ordered: false,+ marker: "•".into(),+ level: 0,+ text: "a".into(),+ bold: false,+ italic: false,+ },+ Block::ListItem {+ ordered: false,+ marker: "•".into(),+ level: 0,+ text: "b".into(),+ bold: false,+ italic: false,+ },+ Block::Paragraph {+ text: "Outro.".into(),+ bold: false,+ italic: false,+ },+ ];+ let s = render_blocks(&blocks);+ assert_eq!(s, "Intro.\n\n- a\n- b\n\nOutro.");++ // Ordered: original marker preserved+ let s = render_blocks(&[+ Block::ListItem {+ ordered: true,+ marker: "138.".into(),+ level: 0,+ text: "footnote".into(),+ bold: false,+ italic: false,+ },+ Block::ListItem {+ ordered: true,+ marker: "139.".into(),+ level: 0,+ text: "next footnote".into(),+ bold: false,+ italic: false,+ },+ ]);+ assert_eq!(s, "138. footnote\n139. next footnote");+ }++ #[test]+ fn render_emphasis_combinations() {+ assert_eq!(wrap_emphasis("hi", false, false), "hi");+ assert_eq!(wrap_emphasis("hi", true, false), "**hi**");+ assert_eq!(wrap_emphasis("hi", false, true), "*hi*");+ assert_eq!(wrap_emphasis("hi", true, true), "***hi***");+ }++ #[test]+ fn code_block_escapes_internal_fence() {+ let blocks = vec![Block::CodeBlock {+ lines: vec!["body containing ``` backticks".into()],+ }];+ let s = render_blocks(&blocks);+ assert!(s.starts_with("~~~\n"));+ assert!(s.ends_with("~~~"));+ }++ #[test]+ fn renders_table_to_pipe_format() {+ let blocks = vec![Block::Table {+ header: Some(vec!["a".into(), "b".into()]),+ rows: vec![vec!["1".into(), "2".into()], vec!["3".into(), "4".into()]],+ }];+ let s = render_blocks(&blocks);+ assert_eq!(s, "| a | b |\n|---|---|\n| 1 | 2 |\n| 3 | 4 |");+ }++ #[test]+ fn render_table_without_header_synthesizes_blank_header() {+ let blocks = vec![Block::Table {+ header: None,+ rows: vec![vec!["1".into(), "2".into()]],+ }];+ let s = render_blocks(&blocks);+ // GFM/CommonMark needs a header row before the separator; we emit a+ // blank one so renderers don't choke.+ assert!(s.contains("|---|---|"));+ assert!(s.ends_with("| 1 | 2 |"));+ }+}diff --git a/crates/liteparse/src/markdown_layout/classify.rs b/crates/liteparse/src/markdown_layout/classify.rsnew file mode 100644index 0000000..2de9c2d--- /dev/null+++ b/crates/liteparse/src/markdown_layout/classify.rs@@ -0,0 +1,783 @@+use crate::types::{OutlineTarget, ParsedPage, ProjectedLine};++use super::blocks::{Block, paragraph_from_accum};+use super::headings::{+ MAX_HEADING_LEVELS, heading_level_for, is_caption_line, looks_like_bold_heading,+ looks_like_numbered_bold_heading, outline_heading_level, struct_heading_level,+};+use super::hr::detect_horizontal_rules;+use super::inline::{+ append_inline_continuation, line_uniform_style, render_line_inline, render_list_item_text,+};+use super::lists::{LIST_INDENT_STEP_PT, parse_list_marker};+use super::paragraphs::{ParaAccum, append_to_paragraph, collapse_whitespace, continues_paragraph};+use super::repetition::is_header_or_footer;+use super::tables::{detect_ruled_tables, detect_tables, merge_table_runs};++/// Returns true if any span on the line is rotated more than ~5° off+/// horizontal — used to exclude sidebar / margin-stamp text (arXiv banners,+/// watermarks, vertical legends) from the body-size and heading-size+/// histograms so it doesn't compete with normal-flow text for heading slots.+pub(super) fn is_rotated_line(line: &ProjectedLine) -> bool {+ line.spans.iter().any(|s| {+ let r = s.rotation.abs() % 360.0;+ // Anything more than ~5° off the horizontal axes is "rotated" for+ // our purposes. 0° and 180° are both horizontal text.+ !(r < 5.0 || (175.0..=185.0).contains(&r) || (355.0..=360.0).contains(&r))+ })+}++/// Classify a single page's `ProjectedLine`s into blocks.+#[cfg(test)]+pub(super) fn classify_page(page: &ParsedPage, heading_map: &[(f32, u8)]) -> Vec<Block> {+ classify_page_with_filters(+ page,+ heading_map,+ &std::collections::HashSet::new(),+ &[],+ crate::config::ImageMode::Placeholder,+ )+}++/// Same as `classify_page` but also strips lines matching a precomputed+/// running header/footer set. Use this when emitting a whole document so+/// repeating chrome (titles, page numbers) doesn't show up in every page.+///+/// `outline` is the document outline filtered to entries whose `page_index`+/// matches this page (or the full outline — out-of-page entries are+/// ignored). Heading promotion from struct tree + outline outranks the+/// font-size heading map.+pub fn classify_page_with_filters(+ page: &ParsedPage,+ heading_map: &[(f32, u8)],+ header_footer: &std::collections::HashSet<String>,+ outline: &[OutlineTarget],+ image_mode: crate::config::ImageMode,+) -> Vec<Block> {+ let mut blocks: Vec<Block> = Vec::new();+ let mut paragraph: Option<ParaAccum> = None;+ // Active fenced code block being accumulated (consecutive `all_mono` lines).+ let mut code: Option<Vec<String>> = None;+ // Tracks the "level 0" indent of the current contiguous list run so we can+ // bucket deeper items into nesting levels. Reset whenever a non-list block+ // breaks the run.+ let mut list_base_indent: Option<f32> = None;+ // Index into `blocks` of the most recent ListItem in the current run — used+ // to merge wrapped continuation lines into the same item.+ let mut last_list_item_idx: Option<usize> = None;+ // Most recent ProjectedLine appended to the active list item, for+ // gap/font-size checks on continuation lines.+ let mut last_list_line: Option<ProjectedLine> = None;++ let flush_paragraph = |blocks: &mut Vec<Block>, p: Option<ParaAccum>| {+ if let Some(acc) = p+ && !acc.raw.trim().is_empty()+ {+ blocks.push(paragraph_from_accum(acc));+ }+ };+ let flush_code = |blocks: &mut Vec<Block>, c: Option<Vec<String>>| {+ if let Some(lines) = c+ && !lines.is_empty()+ {+ blocks.push(Block::CodeBlock { lines });+ }+ };++ let debug = std::env::var("LITEPARSE_DEBUG_MD").is_ok();++ // Strip running header/footer lines up-front so they don't leak into+ // table detection (a repeating two-column footer would otherwise look+ // like a 2-row table) or paragraph grouping.+ let filtered_owned: Vec<ProjectedLine> = if header_footer.is_empty() {+ Vec::new()+ } else {+ page.projected_lines+ .iter()+ .filter(|l| !is_header_or_footer(l, page, header_footer))+ .cloned()+ .collect()+ };+ let lines: &[ProjectedLine] = if header_footer.is_empty() {+ &page.projected_lines+ } else {+ &filtered_owned+ };++ // Pre-pass: detect tabular regions so the per-line classifier below can+ // skip over them. Tables take priority over heading / paragraph / list+ // classification because a row of bold short cells would otherwise be+ // misread as a heading or list item.+ //+ // Two detectors run in sequence: ruled-grid (path-based, strongest signal)+ // and the borderless column-alignment fallback. Where ranges overlap, the+ // ruled output wins.+ let ruled_runs = detect_ruled_tables(lines, &page.graphics, page.page_width, page.page_height);+ let borderless_runs = detect_tables(lines);+ let table_runs = merge_table_runs(ruled_runs, borderless_runs);++ // Suppress HRs that fall inside a detected table's y-range — they're the+ // table's own row dividers, not document-level section breaks. Build the+ // y-extents once before we move table_runs into the iterator.+ // Extend each table's HR-suppression band upward to cover any+ // header/sub-header rows we didn't absorb into the table. The expansion+ // is a few row heights — large enough to catch a 2–3 line bold/italic+ // header sitting just above the table, small enough not to swallow a+ // real section divider belonging to a different block. The downward+ // edge gets a small slack to catch HRs drawn flush with the last row.+ const TABLE_HR_SUPPRESS_HEADROOM_ROWS: f32 = 4.0;+ let table_y_extents: Vec<(f32, f32)> = table_runs+ .iter()+ .map(|run| {+ let top_line = &lines[run.start];+ let row_h = top_line.bbox.height.max(8.0);+ let top = top_line.bbox.y - row_h * TABLE_HR_SUPPRESS_HEADROOM_ROWS;+ let last = &lines[run.end.saturating_sub(1).max(run.start)];+ let bot = last.bbox.y + last.bbox.height;+ (top, bot)+ })+ .collect();++ let mut table_iter = table_runs.into_iter().peekable();++ // Pre-pass: detect horizontal rules from vector graphics so they can be+ // emitted in document order between surrounding text lines.+ let hr_ys: Vec<f32> = detect_horizontal_rules(page)+ .into_iter()+ .filter(|y| {+ !table_y_extents+ .iter()+ .any(|(top, bot)| *y >= *top - 2.0 && *y <= *bot + 2.0)+ })+ .collect();+ let mut hr_iter = hr_ys.into_iter().peekable();++ // Figure injection: when image mode is on, walk the page's image refs in+ // y-order and inject `Block::Figure` between text lines whose y has+ // already passed. Suppressed inside table y-extents to keep cell-spanning+ // raster cell backgrounds (rare but possible) from spawning a figure+ // mid-table.+ let figure_entries: Vec<(f32, crate::types::ImageRef)> = if matches!(+ image_mode,+ crate::config::ImageMode::Off+ ) {+ Vec::new()+ } else {+ let mut v: Vec<(f32, crate::types::ImageRef)> = page+ .image_refs+ .iter()+ .filter(|r| {+ !table_y_extents+ .iter()+ .any(|(top, bot)| r.bbox.y >= *top - 2.0 && r.bbox.y <= *bot + 2.0)+ })+ .map(|r| (r.bbox.y, r.clone()))+ .collect();+ v.sort_by(|a, b| a.0.total_cmp(&b.0));+ v+ };+ let mut figure_iter = figure_entries.into_iter().peekable();++ // Emit any HRs whose y is at or above `before_y`. Flushes the active+ // paragraph/code/list state first so the rule lands as its own block.+ let emit_hrs_before = |blocks: &mut Vec<Block>,+ paragraph: &mut Option<ParaAccum>,+ code: &mut Option<Vec<String>>,+ list_base: &mut Option<f32>,+ last_item: &mut Option<usize>,+ last_line: &mut Option<ProjectedLine>,+ hr_iter: &mut std::iter::Peekable<std::vec::IntoIter<f32>>,+ before_y: f32| {+ while let Some(&hy) = hr_iter.peek() {+ if hy > before_y {+ break;+ }+ hr_iter.next();+ flush_paragraph(blocks, paragraph.take());+ flush_code(blocks, code.take());+ *list_base = None;+ *last_item = None;+ *last_line = None;+ blocks.push(Block::HorizontalRule);+ }+ };++ // Mirror of `emit_hrs_before` for figure entries. Same flush semantics so+ // a figure sitting between two paragraphs lands as its own block in the+ // correct y order.+ let emit_figures_before = |blocks: &mut Vec<Block>,+ paragraph: &mut Option<ParaAccum>,+ code: &mut Option<Vec<String>>,+ list_base: &mut Option<f32>,+ last_item: &mut Option<usize>,+ last_line: &mut Option<ProjectedLine>,+ figure_iter: &mut std::iter::Peekable<+ std::vec::IntoIter<(f32, crate::types::ImageRef)>,+ >,+ before_y: f32| {+ while let Some((fy, _)) = figure_iter.peek() {+ if *fy > before_y {+ break;+ }+ let (_, r) = figure_iter.next().unwrap();+ flush_paragraph(blocks, paragraph.take());+ flush_code(blocks, code.take());+ *list_base = None;+ *last_item = None;+ *last_line = None;+ blocks.push(Block::Figure {+ id: r.id,+ bbox: r.bbox,+ });+ }+ };++ let mut idx = 0;+ while idx < lines.len() {+ if let Some(run) = table_iter.peek()+ && run.start == idx+ {+ // Flush any HRs above this table's top edge first.+ let table_top = lines[run.start].bbox.y;+ emit_hrs_before(+ &mut blocks,+ &mut paragraph,+ &mut code,+ &mut list_base_indent,+ &mut last_list_item_idx,+ &mut last_list_line,+ &mut hr_iter,+ table_top,+ );+ emit_figures_before(+ &mut blocks,+ &mut paragraph,+ &mut code,+ &mut list_base_indent,+ &mut last_list_item_idx,+ &mut last_list_line,+ &mut figure_iter,+ table_top,+ );+ flush_paragraph(&mut blocks, paragraph.take());+ flush_code(&mut blocks, code.take());+ list_base_indent = None;+ last_list_item_idx = None;+ last_list_line = None;+ let run = table_iter.next().unwrap();+ blocks.push(run.block);+ idx = run.end;+ continue;+ }+ let line = &lines[idx];+ // Emit any HRs that fall above this line.+ emit_hrs_before(+ &mut blocks,+ &mut paragraph,+ &mut code,+ &mut list_base_indent,+ &mut last_list_item_idx,+ &mut last_list_line,+ &mut hr_iter,+ line.bbox.y,+ );+ emit_figures_before(+ &mut blocks,+ &mut paragraph,+ &mut code,+ &mut list_base_indent,+ &mut last_list_item_idx,+ &mut last_list_line,+ &mut figure_iter,+ line.bbox.y,+ );+ idx += 1;+ let text = line.text.trim();+ if text.is_empty() {+ continue;+ }+ // Skip rotated text (vertical sidebars, arXiv banners, watermarks).+ // Including it would either inject a paragraph of disconnected+ // characters or be misclassified as a heading.+ if is_rotated_line(line) {+ continue;+ }+ if debug {+ eprintln!(+ "[md] y={:.1} h={:.1} size={:.2} anchor={:?} indent={:.1} text={:?}",+ line.bbox.y,+ line.bbox.height,+ line.dominant_font_size,+ line.anchor,+ line.indent_x,+ text+ );+ }++ // Code block detection runs first so a mono heading-shaped line+ // (rare but plausible — e.g., a code identifier in a large mono font)+ // still becomes code. Mono content also wouldn't make a useful+ // heading.+ if line.all_mono {+ flush_paragraph(&mut blocks, paragraph.take());+ list_base_indent = None;+ last_list_item_idx = None;+ last_list_line = None;+ code.get_or_insert_with(Vec::new)+ .push(line.text.trim_end().to_string());+ continue;+ }+ // Any non-mono line ends the current code block (if any).+ flush_code(&mut blocks, code.take());++ // Priority chain: tagged-PDF struct tree → outline → font-size map.+ let tagged_level = struct_heading_level(line, &page.struct_nodes);+ let outline_level =+ tagged_level.or_else(|| outline_heading_level(line, page.page_height, outline, text));+ // Caption lines ("Figure 7", "Table 3.") are routinely set in a+ // distinct (and slightly larger) font that lands them in the+ // font-size heading map. Suppress font-size promotion for them;+ // outline / struct-tree signals still win since those are explicit.+ let size_level = if is_caption_line(text) {+ None+ } else {+ heading_level_for(line.dominant_font_size, heading_map)+ };+ let level = outline_level+ .or(size_level)+ .map(|l| l.clamp(1, MAX_HEADING_LEVELS as u8));+ if let Some(level) = level {+ flush_paragraph(&mut blocks, paragraph.take());+ list_base_indent = None;+ last_list_item_idx = None;+ last_list_line = None;+ blocks.push(Block::Heading {+ level,+ text: collapse_whitespace(text),+ });+ continue;+ }++ // List item?+ if let Some((ordered, marker, rest)) = parse_list_marker(text) {+ // Numbered bold-section heading: "1. **Foo**" / "5. **The dynamics**".+ // When the post-marker body is uniformly bold + body-sized,+ // standalone (paragraph-break gap above and below), short, and+ // mostly alpha, treat it as a heading rather than the first item+ // of an ordered list. Without this, decimal-numbered section+ // headings in technical/legal/scientific PDFs silently emit as+ // ordered list items and lose all heading structure.+ if ordered+ && looks_like_numbered_bold_heading(+ line,+ rest,+ paragraph.as_ref().map(|p| &p.last).or(last_list_line.as_ref()),+ lines.get(idx),+ )+ {+ flush_paragraph(&mut blocks, paragraph.take());+ list_base_indent = None;+ last_list_item_idx = None;+ last_list_line = None;+ let level = (heading_map.len() as u8 + 1).clamp(1, MAX_HEADING_LEVELS as u8);+ blocks.push(Block::Heading {+ level,+ text: collapse_whitespace(text),+ });+ continue;+ }+ flush_paragraph(&mut blocks, paragraph.take());+ let base = *list_base_indent.get_or_insert(line.indent_x);+ let level = (((line.indent_x - base) / LIST_INDENT_STEP_PT)+ .round()+ .max(0.0)) as u8;+ last_list_item_idx = Some(blocks.len());+ last_list_line = Some(line.clone());+ // Render the list-item text via the inline pipeline so per-span+ // emphasis surfaces. We strip the marker from `rest` (already+ // done by `parse_list_marker`), but emphasis lives on `line.spans`,+ // which still contain the marker span — render the line and then+ // peel the marker off the front of the rendered string.+ let item_text = render_list_item_text(line, &marker, rest);+ blocks.push(Block::ListItem {+ ordered,+ marker,+ level,+ text: item_text,+ bold: false,+ italic: false,+ });+ continue;+ }++ // Continuation of a list item: same gap/font rules as paragraphs.+ // Footnote-style continuations often left-flush below the marker's+ // hanging indent, so we don't require indent ≥ marker indent.+ if let Some(idx) = last_list_item_idx+ && let Some(prev_line) = last_list_line.as_ref()+ && continues_paragraph(prev_line, line)+ && let Some(Block::ListItem {+ text: prev_text, ..+ }) = blocks.get_mut(idx)+ {+ // De-hyphenate against the prior rendered text, then append the+ // inline-styled continuation.+ let cont_inline = render_line_inline(line);+ append_inline_continuation(prev_text, text, &cont_inline);+ last_list_line = Some(line.clone());+ continue;+ }++ // Bold body-size heading. Section headings in academic / technical+ // PDFs are routinely body-sized + bold (e.g. "Abstract",+ // "1 Introduction"); without this rule they look just like a bold+ // first sentence of a paragraph. Runs after list-marker detection so+ // bold bullet items stay as list items.+ let prev_for_gap = paragraph+ .as_ref()+ .map(|p| &p.last)+ .or(last_list_line.as_ref());+ let next_for_gap = lines.get(idx);+ if looks_like_bold_heading(line, prev_for_gap, next_for_gap) {+ flush_paragraph(&mut blocks, paragraph.take());+ list_base_indent = None;+ last_list_item_idx = None;+ last_list_line = None;+ // Level: one deeper than the deepest size-based level we already+ // have. With an empty heading_map this lands on H1; with a full+ // 6-level map it caps at H6.+ let level = (heading_map.len() as u8 + 1).clamp(1, MAX_HEADING_LEVELS as u8);+ blocks.push(Block::Heading {+ level,+ text: collapse_whitespace(text),+ });+ continue;+ }++ match paragraph.as_mut() {+ Some(acc) if continues_paragraph(&acc.last, line) => {+ append_to_paragraph(acc, line);+ }+ _ => {+ flush_paragraph(&mut blocks, paragraph.take());+ list_base_indent = None;+ last_list_item_idx = None;+ last_list_line = None;+ let inline = render_line_inline(line);+ let raw = collapse_whitespace(text);+ let uniform = line_uniform_style(line).map(|s| (s.bold, s.italic));+ paragraph = Some(ParaAccum {+ raw,+ inline,+ last: line.clone(),+ uniform,+ });+ }+ }+ }++ flush_paragraph(&mut blocks, paragraph.take());+ flush_code(&mut blocks, code.take());+ // Flush any trailing HRs / figures that sat below the last text line.+ emit_hrs_before(+ &mut blocks,+ &mut paragraph,+ &mut code,+ &mut list_base_indent,+ &mut last_list_item_idx,+ &mut last_list_line,+ &mut hr_iter,+ f32::INFINITY,+ );+ emit_figures_before(+ &mut blocks,+ &mut paragraph,+ &mut code,+ &mut list_base_indent,+ &mut last_list_item_idx,+ &mut last_list_line,+ &mut figure_iter,+ f32::INFINITY,+ );+ blocks+}++#[cfg(test)]+mod tests {+ use super::super::headings::{build_heading_map, compute_body_size};+ use super::super::repetition::compute_header_footer_set;+ use super::super::test_helpers::{+ header_footer_page, line, mono_line, page, page_with_graphics, styled_line, stroke,+ };+ use super::*;+ use crate::types::TextItem;+ use super::super::blocks::{Block, render_blocks};++ #[test]+ fn classify_emits_heading_and_paragraph() {+ let p = page(vec![+ line("Title of the document goes here", 50.0, 50.0, 18.0, 18.0),+ line("First sentence of the para-", 50.0, 80.0, 10.0, 10.0),+ line("graph continues here.", 50.0, 92.0, 10.0, 10.0),+ line("Another paragraph.", 50.0, 130.0, 10.0, 10.0),+ ]);+ let pages = vec![p];+ let body = compute_body_size(&pages);+ let map = build_heading_map(&pages, body);+ let blocks = classify_page(&pages[0], &map);+ assert_eq!(blocks.len(), 3);+ match &blocks[0] {+ Block::Heading { level, text } => {+ assert_eq!(*level, 1);+ assert_eq!(text, "Title of the document goes here");+ }+ other => panic!("expected heading, got {other:?}"),+ }+ match &blocks[1] {+ Block::Paragraph { text: t, .. } => {+ assert!(t.contains("paragraph continues"), "got: {t}");+ assert!(!t.contains("para-"), "de-hyphenation failed: {t}");+ }+ other => panic!("expected paragraph, got {other:?}"),+ }+ match &blocks[2] {+ Block::Paragraph { text: t, .. } => assert_eq!(t, "Another paragraph."),+ other => panic!("expected paragraph, got {other:?}"),+ }+ }++ #[test]+ fn paragraph_break_on_big_gap() {+ let p = page(vec![+ line("Line A.", 50.0, 80.0, 10.0, 10.0),+ line("Line B.", 50.0, 200.0, 10.0, 10.0),+ ]);+ let pages = vec![p];+ let body = compute_body_size(&pages);+ let map = build_heading_map(&pages, body);+ let blocks = classify_page(&pages[0], &map);+ assert_eq!(blocks.len(), 2);+ }++ #[test]+ fn classify_emits_list_items() {+ let p = page(vec![+ line("Intro line.", 50.0, 50.0, 10.0, 10.0),+ line("• first bullet", 60.0, 80.0, 10.0, 10.0),+ line("• second bullet", 60.0, 92.0, 10.0, 10.0),+ line("◦ nested item", 72.0, 104.0, 10.0, 10.0),+ line("• back to top", 60.0, 116.0, 10.0, 10.0),+ ]);+ let pages = vec![p];+ let body = compute_body_size(&pages);+ let map = build_heading_map(&pages, body);+ let blocks = classify_page(&pages[0], &map);+ let list_items: Vec<&Block> = blocks+ .iter()+ .filter(|b| matches!(b, Block::ListItem { .. }))+ .collect();+ assert_eq!(list_items.len(), 4);+ if let Block::ListItem { level, text, .. } = list_items[0] {+ assert_eq!(*level, 0);+ assert_eq!(text, "first bullet");+ } else {+ panic!();+ }+ // The "- nested item" line is indented +12pt from the base bullet.+ if let Block::ListItem { level, .. } = list_items[2] {+ assert_eq!(*level, 1);+ } else {+ panic!();+ }+ }++ #[test]+ fn classify_emits_code_block() {+ let p = page(vec![+ line("Intro line.", 50.0, 50.0, 10.0, 10.0),+ mono_line(" let x = 1;", 80.0),+ mono_line(" let y = x + 2;", 92.0),+ line("After the code.", 50.0, 120.0, 10.0, 10.0),+ ]);+ let pages = vec![p];+ let body = compute_body_size(&pages);+ let map = build_heading_map(&pages, body);+ let blocks = classify_page(&pages[0], &map);+ // Expect: Paragraph("Intro line."), CodeBlock(2 lines), Paragraph("After...")+ assert_eq!(blocks.len(), 3);+ match &blocks[1] {+ Block::CodeBlock { lines } => {+ assert_eq!(lines.len(), 2);+ assert!(lines[0].contains("let x = 1;"));+ assert!(lines[1].contains("let y = x + 2;"));+ }+ other => panic!("expected code block, got {other:?}"),+ }+ let s = render_blocks(&blocks);+ assert!(s.contains("```\n let x = 1;"));+ assert!(s.ends_with("After the code."));+ }++ #[test]+ fn classify_marks_paragraph_bold_when_all_lines_bold() {+ let mut a = line("Bold line one.", 50.0, 50.0, 10.0, 10.0);+ let mut b = line("bold continuation.", 50.0, 62.0, 10.0, 10.0);+ // Mark the underlying spans as bold so per-span style detection sees+ // it — the new inline pipeline reads from `spans`, not the per-line+ // `all_bold` shortcut flag.+ let bold_span = TextItem {+ text: "x".into(),+ font_name: Some("Arial-Bold".into()),+ ..Default::default()+ };+ a.spans = vec![bold_span.clone()];+ b.spans = vec![bold_span];+ a.all_bold = true;+ b.all_bold = true;+ let p = page(vec![a, b]);+ let pages = vec![p];+ let body = compute_body_size(&pages);+ let map = build_heading_map(&pages, body);+ let blocks = classify_page(&pages[0], &map);+ assert_eq!(blocks.len(), 1);+ match &blocks[0] {+ Block::Paragraph { bold, italic, .. } => {+ assert!(*bold);+ assert!(!*italic);+ }+ other => panic!("expected paragraph, got {other:?}"),+ }+ let s = render_blocks(&blocks);+ assert!(s.starts_with("**") && s.ends_with("**"), "got: {s}");+ }++ #[test]+ fn detects_simple_borderless_table() {+ use super::super::test_helpers::line_with_spans;+ let lines = vec![+ line_with_spans(+ &[("Name", 50.0), ("Age", 150.0), ("City", 250.0)],+ 100.0,+ 10.0,+ ),+ line_with_spans(+ &[("Alice", 50.0), ("30", 150.0), ("NYC", 250.0)],+ 115.0,+ 10.0,+ ),+ line_with_spans(&[("Bob", 50.0), ("25", 150.0), ("LA", 250.0)], 130.0, 10.0),+ ];+ let p = page(lines);+ let pages = vec![p];+ let body = compute_body_size(&pages);+ let map = build_heading_map(&pages, body);+ let blocks = classify_page(&pages[0], &map);+ assert_eq!(blocks.len(), 1, "got: {blocks:?}");+ match &blocks[0] {+ Block::Table { header, rows } => {+ // Header isn't bold so no header row promoted.+ assert!(header.is_none());+ assert_eq!(rows.len(), 3);+ assert_eq!(rows[0][0], "Name");+ assert_eq!(rows[1][2], "NYC");+ }+ other => panic!("expected table, got {other:?}"),+ }+ }++ #[test]+ fn full_format_strips_header_footer() {+ let pages = vec![+ header_footer_page(1, "Acme Confidential", "Page 1 of 2", "First page body."),+ header_footer_page(2, "Acme Confidential", "Page 2 of 2", "Second page body."),+ ];+ let body = compute_body_size(&pages);+ let map = build_heading_map(&pages, body);+ let set = compute_header_footer_set(&pages);+ let blocks = classify_page_with_filters(+ &pages[0],+ &map,+ &set,+ &[],+ crate::config::ImageMode::Placeholder,+ );+ let s = render_blocks(&blocks);+ assert!(!s.contains("Acme Confidential"), "got: {s}");+ assert!(!s.contains("Page 1 of 2"), "got: {s}");+ assert!(s.contains("First page body."));+ }++ #[test]+ fn classify_paragraph_with_mid_line_bold() {+ // First line has a bold word mid-line → not uniformly styled; paragraph+ // should emit baked-in `**bold**` inside the text and `bold=false` at+ // the block level.+ let a = styled_line(+ &[+ ("a sentence with a", 50.0, Some("Arial")),+ ("bold", 180.0, Some("Arial-Bold")),+ ("word in it.", 230.0, Some("Arial")),+ ],+ 50.0,+ 10.0,+ );+ let p = page(vec![a]);+ let pages = vec![p];+ let body = compute_body_size(&pages);+ let map = build_heading_map(&pages, body);+ let blocks = classify_page(&pages[0], &map);+ assert_eq!(blocks.len(), 1, "got: {blocks:?}");+ match &blocks[0] {+ Block::Paragraph { text, bold, italic } => {+ assert!(!*bold, "mixed-style paragraph shouldn't set block bold");+ assert!(!*italic);+ assert!(text.contains("**bold**"), "got: {text}");+ }+ other => panic!("expected paragraph, got {other:?}"),+ }+ }++ #[test]+ fn classify_list_item_strips_marker_under_emphasis() {+ // Whole bullet line is bold (marker + text). Rendered text should be+ // wrapped, with the marker dropped (the renderer prints it).+ let l = styled_line(+ &[+ ("•", 60.0, Some("Arial-Bold")),+ ("important item", 80.0, Some("Arial-Bold")),+ ],+ 50.0,+ 10.0,+ );+ let p = page(vec![l]);+ let pages = vec![p];+ let body = compute_body_size(&pages);+ let map = build_heading_map(&pages, body);+ let blocks = classify_page(&pages[0], &map);+ assert_eq!(blocks.len(), 1);+ match &blocks[0] {+ Block::ListItem { text, .. } => {+ assert_eq!(text, "**important item**");+ }+ other => panic!("expected list item, got {other:?}"),+ }+ }++ #[test]+ fn hr_emitted_between_lines_by_y_order() {+ let lines = vec![+ line("before the rule", 50.0, 100.0, 10.0, 10.0),+ line("after the rule", 50.0, 300.0, 10.0, 10.0),+ ];+ // Stroke between the two lines, far from either's baseline.+ let p = page_with_graphics(lines, vec![stroke(50.0, 200.0, 450.0, 200.0, 0.5)]);+ let blocks = classify_page(&p, &[]);+ let has_hr = blocks+ .iter()+ .position(|b| matches!(b, Block::HorizontalRule));+ assert!(has_hr.is_some(), "expected an HR block, got {blocks:?}");+ // HR must land between the two paragraphs, not before/after both.+ let pos = has_hr.unwrap();+ assert!(pos > 0 && pos < blocks.len() - 1);+ }+}diff --git a/crates/liteparse/src/markdown_layout/headings.rs b/crates/liteparse/src/markdown_layout/headings.rsnew file mode 100644index 0000000..3421d5a--- /dev/null+++ b/crates/liteparse/src/markdown_layout/headings.rs@@ -0,0 +1,526 @@+use crate::types::{OutlineTarget, ParsedPage, ProjectedLine, StructNode};++use super::classify::is_rotated_line;+use super::inline::line_uniform_style;+use super::paragraphs::continues_paragraph;+use super::tables::{TABLE_MIN_COLUMNS, split_cells};++/// Tolerance in points for "is this size larger than the body size".+pub(super) const HEADING_SIZE_EPSILON: f32 = 0.5;++/// Cap on heading levels (matches plan: H1..H6).+pub(super) const MAX_HEADING_LEVELS: usize = 6;++/// Tighter tolerance for matching against the heading-size map. Keeps the+/// heading detector strict so descender-induced height jitter doesn't promote+/// regular body lines to headings.+pub(super) const FONT_SIZE_HEADING_TOLERANCE: f32 = 0.6;++/// Maximum characters in a "bold body-size heading" candidate. Section+/// headings like "Abstract", "1 Introduction", "2.1 Related Work" are short;+/// a bold body-size line longer than this is almost always a bold *sentence*+/// inside a paragraph, not a heading.+pub(super) const BOLD_HEADING_MAX_CHARS: usize = 80;++/// Recognize caption-prefix lines like "Figure 7", "Fig. 12.", "Table 3:",+/// "Tab. 5", "Equation (4)" — these routinely render in a slightly distinct+/// font/size that lands them in the heading_map and gets them promoted to a+/// document-level heading. We want to keep them as plain paragraphs.+pub(super) fn is_caption_line(text: &str) -> bool {+ let t = text.trim_start();+ // Try each known prefix: must be followed by a number (optionally with+ // separators) within the first ~20 chars.+ const PREFIXES: &[&str] = &[+ "Figure", "Figures", "Fig.", "Fig ", "Table", "Tables", "Tab.", "Tab ",+ "Equation", "Eq.", "Eq ", "Scheme", "Chart", "Plate", "Photo", "Algorithm", "Listing",+ ];+ let lower_t_first_word: String = t+ .chars()+ .take_while(|c| c.is_alphabetic() || *c == '.')+ .collect();+ for p in PREFIXES {+ let p_trim = p.trim_end();+ if lower_t_first_word.eq_ignore_ascii_case(p_trim) {+ // Look at what follows the prefix word.+ let rest = t[lower_t_first_word.len()..].trim_start();+ // Allow a leading "(" then digit, or directly a digit / roman numeral.+ let mut chars = rest.chars();+ if let Some(c0) = chars.next()+ && (c0.is_ascii_digit()+ || (c0 == '(' && chars.next().is_some_and(|c| c.is_ascii_digit()))+ || matches!(c0, 'I' | 'V' | 'X' | 'L' | 'C'))+ {+ return true;+ }+ }+ }+ false+}++/// Returns true if `line` looks like a section heading rendered in body-size+/// bold text (a very common style for academic / technical PDFs where every+/// "real" heading uses the same font size as body, distinguished only by+/// weight). Requires:+/// - uniformly bold across all spans+/// - short (≤ `BOLD_HEADING_MAX_CHARS`)+/// - paragraph-break gap above (or first line on the page)+/// - paragraph-break gap below (or last line on the page)+pub(super) fn looks_like_bold_heading(+ line: &ProjectedLine,+ prev: Option<&ProjectedLine>,+ next: Option<&ProjectedLine>,+) -> bool {+ let text = line.text.trim();+ if text.is_empty() || text.chars().count() > BOLD_HEADING_MAX_CHARS {+ return false;+ }+ // Captions ("Figure 7", "Table 3.") are commonly bold body-sized lines+ // that would otherwise satisfy every other rule here. Keep them as+ // paragraphs so they don't appear in the heading hierarchy.+ if is_caption_line(text) {+ return false;+ }+ let style = match line_uniform_style(line) {+ Some(s) => s,+ None => return false,+ };+ if !style.bold || style.mono {+ return false;+ }+ // Reject bold-uniform lines dominated by digits / punctuation — these are+ // almost always cells inside a tabular layout the table detector didn't+ // pick up (results tables, scoreboards, math display). A real section+ // heading is mostly letters: "1 Introduction" passes (~92% alpha across+ // non-whitespace chars), "47.5 14" doesn't (0%), "BLEU-1 25.87" doesn't.+ let mut alpha = 0usize;+ let mut total = 0usize;+ for c in text.chars() {+ if c.is_whitespace() {+ continue;+ }+ total += 1;+ if c.is_alphabetic() {+ alpha += 1;+ }+ }+ if total == 0 || (alpha as f32) / (total as f32) < 0.5 {+ return false;+ }+ // Reject when the line itself looks tabular: ≥3 cells separated by font-size+ // gaps. A bold body-size line with that many cell tracks is almost always a+ // table header row, not a section heading. Without this guard, multi-line+ // table headers ("Model Method ... | F1 BLEU-1 F1 BLEU-1 ...") get promoted+ // to H3 instead of being absorbed by the table detector.+ // Tabular shape rejection. Two passes because the projection sometimes+ // collapses a wide multi-column line into a single span with column-+ // padding spaces — span-based detection misses those.+ if split_cells(line).len() >= TABLE_MIN_COLUMNS {+ return false;+ }+ // Text-based fallback: ≥3 tokens separated by runs of 2+ spaces (the+ // projection inserts alignment padding between cells) → table header row+ // collapsed into one span, not a section heading.+ let multi_space_tokens = text+ .split(" ")+ .map(|s| s.trim())+ .filter(|s| !s.is_empty())+ .count();+ if multi_space_tokens >= TABLE_MIN_COLUMNS {+ return false;+ }+ let gap_above_ok = match prev {+ None => true,+ Some(p) => !continues_paragraph(p, line),+ };+ if !gap_above_ok {+ return false;+ }+ let gap_below_ok = match next {+ None => true,+ Some(n) => !continues_paragraph(line, n),+ };+ gap_below_ok+}++/// Returns true if `line` is a numbered section heading like "1. **Foo**" —+/// `parse_list_marker` already matched the "N." / "N)" prefix; this checks+/// that the body after the marker is uniformly bold body-size text and that+/// the line stands alone (paragraph break above and below). When true the+/// caller should emit a Heading at `heading_map.len()+1` rather than a+/// ListItem. Mirrors `looks_like_bold_heading`'s gating modulo the marker.+pub(super) fn looks_like_numbered_bold_heading(+ line: &ProjectedLine,+ rest: &str,+ prev: Option<&ProjectedLine>,+ _next: Option<&ProjectedLine>,+) -> bool {+ let rest_trim = rest.trim();+ if rest_trim.is_empty() || rest_trim.chars().count() > BOLD_HEADING_MAX_CHARS {+ return false;+ }+ if is_caption_line(&line.text) {+ return false;+ }+ if rest_trim.ends_with('.')+ && rest_trim+ .chars()+ .filter(|c| *c == '.' || *c == '?' || *c == '!')+ .count()+ >= 2+ {+ // "1. Sentence one. Sentence two." → not a heading.+ return false;+ }+ // The spans after the marker must all be bold and non-mono. Marker+ // characters are typically `'0'..='9'`, `'.'`, `')'`, plus whitespace —+ // identify and skip them at the front of the span list.+ let mut saw_bold_body = false;+ let mut saw_non_bold_body = false;+ for span in &line.spans {+ let text = span.text.trim();+ if text.is_empty() {+ continue;+ }+ let is_marker = text+ .chars()+ .all(|c| c.is_ascii_digit() || c == '.' || c == ')' || c == '(');+ if is_marker {+ continue;+ }+ if crate::projection::is_mono_item(span) {+ return false;+ }+ if crate::projection::is_bold_item(span) {+ saw_bold_body = true;+ } else {+ saw_non_bold_body = true;+ }+ }+ if !saw_bold_body || saw_non_bold_body {+ return false;+ }+ // Mostly alphabetic — same intuition as `looks_like_bold_heading`'s+ // alpha-ratio filter: rejects tabular bold rows of digits.+ let (mut alpha, mut total) = (0usize, 0usize);+ for c in rest_trim.chars() {+ if c.is_whitespace() {+ continue;+ }+ total += 1;+ if c.is_alphabetic() {+ alpha += 1;+ }+ }+ if total == 0 || (alpha as f32) / (total as f32) < 0.5 {+ return false;+ }+ // Paragraph-break gap above. We deliberately don't require gap_below:+ // a numbered section heading is often followed by another bold body+ // line (a sub-heading or a multi-line title continuation) which would+ // satisfy `continues_paragraph`. The numbered+bold combination is+ // distinctive enough that the false-positive risk is small.+ match prev {+ None => true,+ Some(p) => !continues_paragraph(p, line),+ }+}++/// Compute the body font size as the char-weighted mode across all lines in+/// all pages. Rotated lines are excluded so a long rotated sidebar can't+/// pull the body estimate. Falls back to `0.0` when no font-size info is+/// available.+pub fn compute_body_size(pages: &[ParsedPage]) -> f32 {+ use std::collections::HashMap;+ let mut weights: HashMap<u32, (f32, usize)> = HashMap::new();+ for page in pages {+ for line in &page.projected_lines {+ if is_rotated_line(line) {+ continue;+ }+ let size = line.dominant_font_size;+ if size <= 0.0 {+ continue;+ }+ let chars = line.text.chars().count().max(1);+ let key = (size * 100.0).round() as u32;+ let entry = weights.entry(key).or_insert((size, 0));+ entry.1 += chars;+ }+ }+ weights+ .values()+ .max_by_key(|(_, n)| *n)+ .map(|(s, _)| *s)+ .unwrap_or(0.0)+}++/// Minimum total non-whitespace characters across all occurrences at a font+/// size for it to qualify as a heading level. Calibrated against `paper.pdf`:+/// the 30pt chart-legend tokens ("A-mem"×2 + "Base"×2 = 18-20 chars) need to+/// fail this filter, while the 14.35pt title ("A-MEM: Agentic Memory for LLM+/// Agents" = 31 chars on a single line) needs to pass. 25 is the gap. Smaller+/// single-word headings like a lone "Summary" on a short doc still survive+/// because they share their font size with other (larger) headings in the+/// histogram entry.+const MIN_HEADING_TOTAL_CHARS: usize = 25;++/// Maximum average characters per line for a size to qualify as a heading.+/// A "size larger than body" with very long lines is almost always a+/// large-print body block (callouts, footnotes-as-display, intro paragraph),+/// not a real heading.+const MAX_HEADING_AVG_LINE_CHARS: f32 = 200.0;++/// Minimum fraction of non-whitespace chars at a size that must be alphabetic+/// for it to qualify as a heading. Drops sizes dominated by digits (graph+/// axes, results tables, math display) which otherwise pollute the top+/// heading slots.+const MIN_HEADING_ALPHA_RATIO: f32 = 0.5;++/// Build a heading-size → level map: sizes strictly larger than `body_size`,+/// filtered to those with at least `MIN_HEADING_LINES` distinct occurrences+/// (drops one-off equation/figure-label artifacts), sorted descending, mapped+/// to levels 1..=MAX_HEADING_LEVELS.+pub fn build_heading_map(pages: &[ParsedPage], body_size: f32) -> Vec<(f32, u8)> {+ use std::collections::HashMap;+ // (size_key → (size, line_count, total_chars, alpha_chars))+ let mut sizes: HashMap<u32, (f32, usize, usize, usize)> = HashMap::new();+ for page in pages {+ for line in &page.projected_lines {+ if is_rotated_line(line) {+ continue;+ }+ // Captions ("Figure 7", "Table 3.") often render slightly larger+ // than body and would otherwise inflate / hijack the heading map.+ if is_caption_line(&line.text) {+ continue;+ }+ let size = line.dominant_font_size;+ if size > body_size + HEADING_SIZE_EPSILON {+ let key = (size * 100.0).round() as u32;+ let entry = sizes.entry(key).or_insert((size, 0, 0, 0));+ entry.1 += 1;+ for c in line.text.chars() {+ if c.is_whitespace() {+ continue;+ }+ entry.2 += 1;+ if c.is_alphabetic() {+ entry.3 += 1;+ }+ }+ }+ }+ }+ let all: Vec<(f32, usize, usize, usize)> = sizes.into_values().collect();+ // Always apply quality filters: total-char floor, average-line cap, and+ // alpha-ratio floor. The total-char floor (rather than a line-count one)+ // lets one-off titles survive — a 31-char title on a single line passes —+ // while still rejecting chart-legend tokens like "A-mem" + "Base" that+ // total fewer chars across 4 occurrences than a single heading line does.+ let mut kept: Vec<f32> = all+ .iter()+ .filter(|(_, lines, chars, alpha)| {+ let alpha_ratio = if *chars == 0 {+ 0.0+ } else {+ (*alpha as f32) / (*chars as f32)+ };+ *chars >= MIN_HEADING_TOTAL_CHARS+ && (*chars as f32 / (*lines).max(1) as f32) <= MAX_HEADING_AVG_LINE_CHARS+ && alpha_ratio >= MIN_HEADING_ALPHA_RATIO+ })+ .map(|(s, _, _, _)| *s)+ .collect();+ kept.sort_by(|a, b| b.total_cmp(a));+ kept.truncate(MAX_HEADING_LEVELS);+ kept.into_iter()+ .enumerate()+ .map(|(i, s)| (s, (i + 1) as u8))+ .collect()+}++pub(super) fn heading_level_for(size: f32, heading_map: &[(f32, u8)]) -> Option<u8> {+ for (s, level) in heading_map {+ if (size - *s).abs() < FONT_SIZE_HEADING_TOLERANCE {+ return Some(*level);+ }+ }+ None+}++/// Highest-priority heading source: a struct-tree node `H1`..`H6` directly+/// tagging this line via its `mcid`. Available only for tagged PDFs.+pub(super) fn struct_heading_level(line: &ProjectedLine, struct_nodes: &[StructNode]) -> Option<u8> {+ let mcid = line.mcid?;+ for node in struct_nodes {+ if !node.mcids.contains(&mcid) {+ continue;+ }+ if let Some(level) = parse_heading_role(&node.role) {+ return Some(level);+ }+ }+ None+}++/// Parse a struct-tree role string like "H1" or "H3" into a heading level.+/// Returns None for non-heading roles (P, L, Figure, Table, ...).+fn parse_heading_role(role: &str) -> Option<u8> {+ let trimmed = role.trim();+ if !trimmed.starts_with('H') && !trimmed.starts_with('h') {+ return None;+ }+ let digits = &trimmed[1..];+ let n: u8 = digits.parse().ok()?;+ if (1..=6).contains(&n) { Some(n) } else { None }+}++/// Second-priority heading source: a document outline (bookmark) entry that+/// points at this page near this line's y coordinate, with a title that+/// prefix-matches the line text. Used on untagged PDFs that ship a TOC.+pub(super) fn outline_heading_level(+ line: &ProjectedLine,+ page_height: f32,+ outline: &[OutlineTarget],+ line_text: &str,+) -> Option<u8> {+ if outline.is_empty() {+ return None;+ }+ let normalized_line = normalize_outline_text(line_text);+ if normalized_line.is_empty() {+ return None;+ }+ let row_h = line.bbox.height.max(8.0);+ let y_tolerance = row_h * 1.5;+ for entry in outline {+ let normalized_title = normalize_outline_text(&entry.title);+ if normalized_title.is_empty() {+ continue;+ }+ // Spatial check is *strict* only when the entry actually carries a+ // usable y (in-page). Many bookmarks point at "top of page" with a+ // Y outside the MediaBox or no Y at all — in that case we still+ // accept any line on the page that prefix-matches the title.+ let y_ok = match entry.y_pdf {+ Some(y) => {+ let y_view = page_height - y;+ if y_view < 0.0 || y_view > page_height {+ true+ } else {+ (y_view - line.bbox.y).abs() <= y_tolerance+ }+ }+ None => true,+ };+ if !y_ok {+ continue;+ }+ // Short outline titles ("Nutrition") would otherwise false-match any+ // paragraph that happens to start with them. Require the matched line+ // to be heading-shaped: not much longer than the title itself.+ let line_len = normalized_line.chars().count();+ let title_len = normalized_title.chars().count();+ let max_line_len = (title_len * 3).max(120);+ // Multiple sentences → almost certainly prose, not a heading line.+ let sentence_breaks = normalized_line.matches(". ").count();+ if line_len > max_line_len || sentence_breaks >= 2 {+ continue;+ }+ if normalized_line.starts_with(&normalized_title)+ || normalized_title.starts_with(&normalized_line)+ {+ return Some(entry.level.min(MAX_HEADING_LEVELS as u8));+ }+ }+ None+}++/// Lowercase + collapse whitespace, for forgiving outline-title vs line-text+/// comparison. Outline titles often have trailing numbering or punctuation+/// not present on the rendered line (and vice versa).+fn normalize_outline_text(s: &str) -> String {+ let mut out = String::with_capacity(s.len());+ let mut prev_space = true;+ for c in s.chars() {+ if c.is_whitespace() {+ if !prev_space {+ out.push(' ');+ prev_space = true;+ }+ } else {+ for lc in c.to_lowercase() {+ out.push(lc);+ }+ prev_space = false;+ }+ }+ if out.ends_with(' ') {+ out.pop();+ }+ out+}++#[cfg(test)]+mod tests {+ use super::super::test_helpers::{line, page};+ use super::*;++ #[test]+ fn body_size_picks_most_common() {+ let pages = vec![page(vec![+ line("Title", 50.0, 50.0, 18.0, 18.0),+ line("body line one", 50.0, 80.0, 10.0, 10.0),+ line("body line two", 50.0, 92.0, 10.0, 10.0),+ line("body line three", 50.0, 104.0, 10.0, 10.0),+ ])];+ let body = compute_body_size(&pages);+ assert!((body - 10.0).abs() < 0.01, "body size = {body}");+ }++ #[test]+ fn heading_map_descending_levels() {+ // Heading text needs to clear `MIN_HEADING_TOTAL_CHARS` (25) so the+ // size qualifies as a real heading rather than chart-legend noise.+ let pages = vec![page(vec![+ line("The largest heading on the page", 50.0, 50.0, 24.0, 24.0),+ line("A smaller heading right below it", 50.0, 80.0, 18.0, 18.0),+ // Several lines of body so it beats the heading text in the+ // char-weighted body-size mode.+ line(+ "body text line one with plenty of content",+ 50.0,+ 110.0,+ 10.0,+ 10.0,+ ),+ line(+ "body text line two with plenty of content",+ 50.0,+ 122.0,+ 10.0,+ 10.0,+ ),+ line(+ "body text line three with even more content",+ 50.0,+ 134.0,+ 10.0,+ 10.0,+ ),+ line(+ "body text line four with even more content",+ 50.0,+ 146.0,+ 10.0,+ 10.0,+ ),+ ])];+ let body = compute_body_size(&pages);+ let map = build_heading_map(&pages, body);+ assert_eq!(map.len(), 2);+ assert_eq!(map[0].1, 1);+ assert_eq!(map[1].1, 2);+ assert!(map[0].0 > map[1].0);+ }+}diff --git a/crates/liteparse/src/markdown_layout/hr.rs b/crates/liteparse/src/markdown_layout/hr.rsnew file mode 100644index 0000000..7e8f4ad--- /dev/null+++ b/crates/liteparse/src/markdown_layout/hr.rs@@ -0,0 +1,132 @@+use crate::types::{GraphicPrimitive, ParsedPage};++/// Maximum stroke thickness (or |y1-y2|) for a stroke to count as a candidate HR.+/// Thicker shapes are filled rects, not rules.+const HR_MAX_THICKNESS_PT: f32 = 2.0;++/// Minimum fraction of page width a horizontal stroke must span to count as an HR.+/// Shorter strokes are typically table borders, list bullets, or inline marks.+const HR_MIN_WIDTH_FRACTION: f32 = 0.3;++/// Vertical tolerance (points) for treating a stroke as "underlining" the+/// nearest text line. Strokes within this band of a text line's baseline are+/// dropped — they're underlines, not rules.+const HR_UNDERLINE_PROXIMITY_PT: f32 = 3.0;++/// Detect horizontal rules from a page's vector graphics.+///+/// Returns the y-coordinates (viewport space) of accepted HRs, sorted ascending.+/// An HR is a roughly horizontal stroke that spans at least+/// `HR_MIN_WIDTH_FRACTION` of the page width, is thinner than+/// `HR_MAX_THICKNESS_PT`, and does not sit on the baseline of any text line+/// (which would make it an underline).+pub(super) fn detect_horizontal_rules(page: &ParsedPage) -> Vec<f32> {+ if page.graphics.is_empty() || page.page_width <= 0.0 {+ return Vec::new();+ }+ let min_width = page.page_width * HR_MIN_WIDTH_FRACTION;+ let mut ys: Vec<f32> = Vec::new();++ for g in &page.graphics {+ let GraphicPrimitive::Stroke {+ x1,+ y1,+ x2,+ y2,+ width,+ ..+ } = g+ else {+ continue;+ };+ let (x1, y1, x2, y2, width) = (*x1, *y1, *x2, *y2, *width);+ let dy = (y1 - y2).abs();+ let dx = (x1 - x2).abs();+ if dy > HR_MAX_THICKNESS_PT || width > HR_MAX_THICKNESS_PT {+ continue;+ }+ if dx < min_width {+ continue;+ }+ let y = (y1 + y2) * 0.5;+ let xmin = x1.min(x2);+ let xmax = x1.max(x2);++ // Drop if this stroke sits on a text-line baseline — it's an underline,+ // not a divider.+ let is_underline = page.projected_lines.iter().any(|line| {+ let bottom = line.bbox.y + line.bbox.height;+ (y - bottom).abs() < HR_UNDERLINE_PROXIMITY_PT+ && xmin >= line.bbox.x - 2.0+ && xmax <= line.bbox.x + line.bbox.width + 2.0+ });+ if is_underline {+ continue;+ }+ ys.push(y);+ }++ // Sort + dedup near-duplicates (some PDFs draw the same rule twice).+ ys.sort_by(|a, b| a.total_cmp(b));+ ys.dedup_by(|a, b| (*a - *b).abs() < 1.0);+ ys+}++#[cfg(test)]+mod tests {+ use super::super::test_helpers::{line, page_with_graphics, stroke};+ use super::*;++ #[test]+ fn hr_long_thin_horizontal_stroke_detected() {+ // 400pt wide stroke on a 612pt page → comfortably above 30% threshold.+ let p = page_with_graphics(vec![], vec![stroke(50.0, 200.0, 450.0, 200.5, 0.5)]);+ let ys = detect_horizontal_rules(&p);+ assert_eq!(ys, vec![200.25]);+ }++ #[test]+ fn hr_short_stroke_rejected() {+ // 50pt wide — table border or list bullet, not an HR.+ let p = page_with_graphics(vec![], vec![stroke(50.0, 200.0, 100.0, 200.0, 0.5)]);+ assert!(detect_horizontal_rules(&p).is_empty());+ }++ #[test]+ fn hr_vertical_stroke_rejected() {+ let p = page_with_graphics(vec![], vec![stroke(50.0, 50.0, 50.0, 500.0, 0.5)]);+ assert!(detect_horizontal_rules(&p).is_empty());+ }++ #[test]+ fn hr_thick_stroke_rejected() {+ // 4pt-thick stroke → a filled bar, not a rule.+ let p = page_with_graphics(vec![], vec![stroke(50.0, 200.0, 450.0, 200.0, 4.0)]);+ assert!(detect_horizontal_rules(&p).is_empty());+ }++ #[test]+ fn hr_underline_at_text_baseline_dropped() {+ // Text line at y=100 height=10 → bottom at y=110. Stroke at y=111 within+ // the line's horizontal extent → underline, not an HR.+ let text_line = line(+ "Some underlined heading text on the page",+ 50.0,+ 100.0,+ 10.0,+ 10.0,+ );+ let bottom = text_line.bbox.y + text_line.bbox.height;+ let p = page_with_graphics(+ vec![text_line.clone()],+ vec![stroke(+ 50.0,+ bottom + 1.0,+ 50.0 + text_line.bbox.width,+ bottom + 1.0,+ 0.5,+ )],+ );+ assert!(detect_horizontal_rules(&p).is_empty());+ }+}diff --git a/crates/liteparse/src/markdown_layout/inline.rs b/crates/liteparse/src/markdown_layout/inline.rsnew file mode 100644index 0000000..bed7f80--- /dev/null+++ b/crates/liteparse/src/markdown_layout/inline.rs@@ -0,0 +1,319 @@+use crate::projection::{is_bold_item, is_italic_item, is_mono_item};+use crate::types::{ProjectedLine, TextItem};++use super::paragraphs::collapse_whitespace;++/// Lightweight bold detection on a raw `TextItem`. Delegates to+/// `projection::is_bold_item` so per-span and per-line logic stay in sync.+pub(super) fn is_bold_span(s: &TextItem) -> bool {+ is_bold_item(s)+}++/// Per-span style flags used by the inline-emphasis renderer.+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]+pub(super) struct SpanStyle {+ pub(super) bold: bool,+ pub(super) italic: bool,+ pub(super) mono: bool,+}++impl SpanStyle {+ pub(super) fn from_item(item: &TextItem) -> Self {+ SpanStyle {+ bold: is_bold_item(item),+ italic: is_italic_item(item),+ mono: is_mono_item(item),+ }+ }++ pub(super) fn is_plain(self) -> bool {+ !self.bold && !self.italic && !self.mono+ }+}++/// Escape characters that would otherwise be interpreted as markdown emphasis.+/// Deliberately narrow: only `*`, `_`, and backslash. Aggressive escaping+/// (`#`, `[`, backticks, etc.) breaks more output than it saves in practice —+/// pymupdf4llm takes the same conservative stance.+pub(super) fn escape_inline(s: &str) -> String {+ let mut out = String::with_capacity(s.len());+ for c in s.chars() {+ match c {+ '\\' | '*' | '_' => {+ out.push('\\');+ out.push(c);+ }+ _ => out.push(c),+ }+ }+ out+}++/// Wrap `inner` with the markdown markers for `style`. Mono wins over bold/italic:+/// inline code (`` `…` ``) doesn't compose with emphasis in CommonMark, so when+/// a span is mono we drop the `**/*` wrap. Bold + italic → `***…***`.+pub(super) fn apply_style(inner: &str, style: SpanStyle) -> String {+ if style.mono {+ // Use backticks; if inner already contains backticks, switch to a+ // longer fence (pair of backticks plus a space buffer) per CommonMark.+ if inner.contains('`') {+ return format!("`` {} ``", inner);+ }+ return format!("`{}`", inner);+ }+ match (style.bold, style.italic) {+ (true, true) => format!("***{}***", inner),+ (true, false) => format!("**{}**", inner),+ (false, true) => format!("*{}*", inner),+ (false, false) => inner.to_string(),+ }+}++/// Render a `ProjectedLine` to markdown with per-span emphasis. Adjacent+/// same-style spans are merged into a single emphasis run; whitespace between+/// spans is preserved as one space (the underlying projection output already+/// has the right inter-word spacing baked into span text).+///+/// Per-line shortcut: when every non-whitespace span shares the same style,+/// emit one outer wrap around the collapsed line text instead of run-by-run+/// (matches pymupdf4llm; avoids `**foo** **bar** **baz**` noise on uniformly+/// styled lines).+///+/// Falls back to `collapse_whitespace(line.text)` when the line has no usable+/// spans (e.g. OCR-only lines where TextItem styling isn't populated).+pub(super) fn render_line_inline(line: &ProjectedLine) -> String {+ let spans: Vec<&TextItem> = line+ .spans+ .iter()+ .filter(|s| !s.text.trim().is_empty())+ .collect();+ if spans.is_empty() {+ return collapse_whitespace(&line.text);+ }++ // Sort spans by x so we render in visual reading order regardless of+ // extraction order. Stable so equal-x spans keep their original sequence.+ let mut spans = spans;+ spans.sort_by(|a, b| a.x.total_cmp(&b.x));++ let styles: Vec<SpanStyle> = spans.iter().map(|s| SpanStyle::from_item(s)).collect();++ // Per-line shortcut.+ let uniform = styles.iter().all(|s| *s == styles[0]);+ if uniform {+ let joined = collapse_whitespace(&line.text);+ if joined.is_empty() {+ return joined;+ }+ let escaped = escape_inline(&joined);+ if styles[0].is_plain() {+ return escaped;+ }+ return apply_style(&escaped, styles[0]);+ }++ // Group consecutive spans by style. Within a group, span texts join with+ // a single space (we lose intra-group spacing precision; acceptable).+ let mut out = String::new();+ let mut i = 0;+ while i < spans.len() {+ let style = styles[i];+ let mut j = i + 1;+ while j < spans.len() && styles[j] == style {+ j += 1;+ }+ let mut group_text = String::new();+ for k in i..j {+ if !group_text.is_empty() && !group_text.ends_with(' ') {+ group_text.push(' ');+ }+ group_text.push_str(spans[k].text.trim());+ }+ let group_text = collapse_whitespace(&group_text);+ let escaped = escape_inline(&group_text);+ let rendered = if style.is_plain() {+ escaped+ } else {+ apply_style(&escaped, style)+ };+ if !out.is_empty() && !out.ends_with(' ') {+ out.push(' ');+ }+ out.push_str(&rendered);+ i = j;+ }+ out+}++/// Render the text portion of a list item with per-span emphasis. The marker+/// itself isn't included in the output (the renderer handles it separately).+///+/// When the line is uniformly styled we wrap the marker-stripped `rest` with+/// the line's style — this avoids the awkward emphasis-marker mismatch we'd+/// hit if we naively stripped a leading bullet out of an already-wrapped+/// rendered line (`**• item**` → `** item**`).+///+/// When the line is mixed-style we render the full line via the inline pipeline+/// and then best-effort-strip the marker prefix (with optional emphasis wrap+/// around it). On any failure we fall back to plain escaped `rest`.+pub(super) fn render_list_item_text(line: &ProjectedLine, marker: &str, rest: &str) -> String {+ if let Some(style) = line_uniform_style(line) {+ let plain = collapse_whitespace(rest);+ let escaped = escape_inline(&plain);+ return if style.is_plain() {+ escaped+ } else {+ apply_style(&escaped, style)+ };+ }+ let full = render_line_inline(line);+ if let Some(stripped) = strip_leading_marker_from_inline(&full, marker) {+ return stripped;+ }+ escape_inline(&collapse_whitespace(rest))+}++/// Try to strip a leading list marker (optionally wrapped in emphasis markers)+/// off `s`. Recognizes `***MARK*** `, `**MARK** `, `*MARK* `, `` `MARK` ``,+/// and bare `MARK `. Returns the suffix on a match.+pub(super) fn strip_leading_marker_from_inline(s: &str, marker: &str) -> Option<String> {+ for wrap in ["***", "**", "*", "`"] {+ let prefix = format!("{wrap}{marker}{wrap} ");+ if let Some(rest) = s.strip_prefix(&prefix) {+ return Some(rest.to_string());+ }+ }+ let prefix = format!("{marker} ");+ s.strip_prefix(&prefix).map(|r| r.to_string())+}++/// Append an inline-rendered continuation line to an existing list-item body.+/// De-hyphenates against the raw text boundary (mirrors the paragraph rule)+/// and falls back to a space join otherwise.+pub(super) fn append_inline_continuation(prev_text: &mut String, next_raw: &str, next_inline: &str) {+ let next_raw = collapse_whitespace(next_raw);+ if next_raw.is_empty() {+ return;+ }+ if prev_text.is_empty() {+ prev_text.push_str(next_inline);+ return;+ }+ let dehyphenate = prev_text.ends_with('-')+ && next_raw+ .chars()+ .next()+ .is_some_and(|c| c.is_ascii_lowercase());+ if dehyphenate {+ prev_text.pop();+ prev_text.push_str(next_inline);+ } else {+ prev_text.push(' ');+ prev_text.push_str(next_inline);+ }+}++/// Returns the shared `SpanStyle` of `line` when every non-whitespace span on+/// the line has the same style; `None` when spans disagree. Mono is folded+/// into the style for the purpose of "uniform" — a fully-mono line is+/// uniform-mono. Used by the paragraph-level optimization to decide whether+/// to wrap once around the whole paragraph or fall back to per-line inline.+pub(super) fn line_uniform_style(line: &ProjectedLine) -> Option<SpanStyle> {+ let mut iter = line+ .spans+ .iter()+ .filter(|s| !s.text.trim().is_empty())+ .map(SpanStyle::from_item);+ let first = iter.next()?;+ for s in iter {+ if s != first {+ return None;+ }+ }+ Some(first)+}++#[cfg(test)]+mod tests {+ use super::super::test_helpers::styled_line;+ use super::*;++ #[test]+ fn render_line_inline_mid_line_bold() {+ // Plain span, then bold span: should produce a mid-line `**bold**` run.+ let l = styled_line(+ &[+ ("regular text with", 50.0, Some("Arial")),+ ("bold word", 200.0, Some("Arial-Bold")),+ ],+ 100.0,+ 10.0,+ );+ let out = render_line_inline(&l);+ assert!(out.contains("regular text with"), "got: {out}");+ assert!(out.contains("**bold word**"), "got: {out}");+ assert!(+ !out.starts_with("**"),+ "mid-line shouldn't open with bold: {out}"+ );+ }++ #[test]+ fn render_line_inline_uniform_bold_uses_shortcut() {+ // All spans bold → single outer wrap, no per-span noise.+ let l = styled_line(+ &[+ ("first", 50.0, Some("Arial-Bold")),+ ("second", 100.0, Some("Arial-Bold")),+ ],+ 100.0,+ 10.0,+ );+ let out = render_line_inline(&l);+ assert!(out.starts_with("**") && out.ends_with("**"), "got: {out}");+ // Only one bold run, not two — the shortcut should kick in.+ assert_eq!(out.matches("**").count(), 2, "got: {out}");+ }++ #[test]+ fn render_line_inline_escapes_emphasis_chars() {+ let l = styled_line(&[("5*4=20", 50.0, Some("Arial"))], 100.0, 10.0);+ let out = render_line_inline(&l);+ assert_eq!(out, "5\\*4=20");+ }++ #[test]+ fn render_line_inline_italic_then_bold() {+ let l = styled_line(+ &[+ ("italic", 50.0, Some("Arial-Italic")),+ ("plain", 100.0, Some("Arial")),+ ("bold", 150.0, Some("Arial-Bold")),+ ],+ 100.0,+ 10.0,+ );+ let out = render_line_inline(&l);+ assert!(out.contains("*italic*"), "got: {out}");+ assert!(out.contains("plain"), "got: {out}");+ assert!(out.contains("**bold**"), "got: {out}");+ }++ #[test]+ fn render_line_inline_mono_span() {+ let l = styled_line(+ &[+ ("call", 50.0, Some("Arial")),+ ("foo()", 100.0, Some("Courier")),+ ("on it", 150.0, Some("Arial")),+ ],+ 100.0,+ 10.0,+ );+ let out = render_line_inline(&l);+ assert!(out.contains("`foo()`"), "got: {out}");+ // Plain spans stay unwrapped.+ assert!(out.contains("call"));+ assert!(out.contains("on it"));+ }+}diff --git a/crates/liteparse/src/markdown_layout/lists.rs b/crates/liteparse/src/markdown_layout/lists.rsnew file mode 100644index 0000000..aaaa22d--- /dev/null+++ b/crates/liteparse/src/markdown_layout/lists.rs@@ -0,0 +1,96 @@+/// Roughly one indent step in PDF points. Used to bucket list items into+/// nesting levels relative to the first item of the list.+pub(super) const LIST_INDENT_STEP_PT: f32 = 12.0;++/// Characters recognized as bullet markers when followed by whitespace.+/// Limited to glyphs that are unlikely to appear at line-start in normal prose.+const BULLET_CHARS: &[char] = &['•', '·', '◦', '▪', '▸', '▶', '●', '○', '■', '□'];++/// Detect a list marker at the start of `text`. Returns `(ordered, marker_str,+/// remainder)` when matched; otherwise `None`.+///+/// Recognizes:+/// - Unicode bullet characters (`BULLET_CHARS`) followed by whitespace.+/// - Decimal-prefix markers like `1.` / `1)` / `12.` / `12)` followed by+/// whitespace — kept strict (digits only) so things like footnote callers+/// (`1` alone) and section refs (`A.1`) don't match.+pub(super) fn parse_list_marker(text: &str) -> Option<(bool, String, &str)> {+ let trimmed = text.trim_start();+ if trimmed.is_empty() {+ return None;+ }++ let mut chars = trimmed.chars();+ let first = chars.next()?;++ // Unicode bullet+ if BULLET_CHARS.contains(&first) {+ let rest = chars.as_str();+ if let Some(rest_trim) = rest.strip_prefix(|c: char| c.is_whitespace()) {+ return Some((false, first.to_string(), rest_trim.trim_start()));+ }+ }++ // Decimal: 1. / 1) / 12. / 12)+ if first.is_ascii_digit() {+ let mut digit_end = 1;+ for c in trimmed[1..].chars() {+ if c.is_ascii_digit() {+ digit_end += c.len_utf8();+ } else {+ break;+ }+ }+ // Cap to keep us from matching page-number-like prefixes+ if digit_end <= 3 {+ let after_digits = &trimmed[digit_end..];+ let mut after_iter = after_digits.chars();+ if let Some(punct) = after_iter.next()+ && (punct == '.' || punct == ')')+ {+ let after_punct = after_iter.as_str();+ if let Some(rest_trim) = after_punct.strip_prefix(|c: char| c.is_whitespace()) {+ let marker = format!("{}{}", &trimmed[..digit_end], punct);+ return Some((true, marker, rest_trim.trim_start()));+ }+ }+ }+ }++ None+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn parse_list_marker_bullets() {+ let (ordered, marker, rest) = parse_list_marker("• item one").unwrap();+ assert!(!ordered);+ assert_eq!(marker, "•");+ assert_eq!(rest, "item one");+ }++ #[test]+ fn parse_list_marker_decimal() {+ let (ordered, marker, rest) = parse_list_marker("1. first").unwrap();+ assert!(ordered);+ assert_eq!(marker, "1.");+ assert_eq!(rest, "first");++ let (ordered, marker, rest) = parse_list_marker("12) twelfth").unwrap();+ assert!(ordered);+ assert_eq!(marker, "12)");+ assert_eq!(rest, "twelfth");+ }++ #[test]+ fn parse_list_marker_rejects_prose() {+ assert!(parse_list_marker("This sentence.").is_none());+ // Bare digit with no terminator → not a list+ assert!(parse_list_marker("2023 was a year").is_none());+ // Footnote caller / page number style — no whitespace after+ assert!(parse_list_marker("1.5x growth").is_none());+ }+}diff --git a/crates/liteparse/src/markdown_layout/mod.rs b/crates/liteparse/src/markdown_layout/mod.rsnew file mode 100644index 0000000..9ccf865--- /dev/null+++ b/crates/liteparse/src/markdown_layout/mod.rs@@ -0,0 +1,24 @@+//! Block classification for the markdown emitter.+//!+//! Consumes `ProjectedLine` entries from each `ParsedPage` and groups them into+//! a sequence of `Block`s: headings, paragraphs, and (for now) raw lines that+//! don't fit a recognized shape. Tables, lists, and code blocks land in later+//! build-order steps.++mod blocks;+mod classify;+mod headings;+mod hr;+mod inline;+mod lists;+mod paragraphs;+mod repetition;+mod tables;++pub use blocks::{Block, render_blocks};+pub use classify::classify_page_with_filters;+pub use headings::{build_heading_map, compute_body_size};+pub use repetition::compute_header_footer_set;++#[cfg(test)]+pub(crate) mod test_helpers;diff --git a/crates/liteparse/src/markdown_layout/paragraphs.rs b/crates/liteparse/src/markdown_layout/paragraphs.rsnew file mode 100644index 0000000..ba866ec--- /dev/null+++ b/crates/liteparse/src/markdown_layout/paragraphs.rs@@ -0,0 +1,214 @@+use crate::types::{Anchor, ProjectedLine};++use super::inline::{SpanStyle, line_uniform_style, render_line_inline};++/// Multiplier on line height used as the paragraph-break threshold.+pub(super) const PARAGRAPH_GAP_MULTIPLIER: f32 = 1.5;++/// Tolerance for treating two font sizes as "the same" when grouping+/// paragraph lines. Generous because we sometimes derive the "size" from+/// `bbox.height`, which varies a few points line-to-line based on whether+/// the glyphs include descenders (`y`, `g`, `p`).+pub(super) const FONT_SIZE_PARAGRAPH_TOLERANCE: f32 = 2.5;++/// Tolerance in points for treating two indent positions as "the same column".+pub(super) const INDENT_TOLERANCE: f32 = 6.0;++/// Collapse runs of whitespace into single spaces. The projected text from+/// `projection.rs` pads with column-alignment spaces (e.g. `for instance`)+/// which look fine as a layout grid but are wrong for prose.+pub(super) fn collapse_whitespace(s: &str) -> String {+ let mut out = String::with_capacity(s.len());+ let mut prev_space = false;+ for c in s.chars() {+ if c.is_whitespace() {+ if !prev_space && !out.is_empty() {+ out.push(' ');+ }+ prev_space = true;+ } else {+ out.push(c);+ prev_space = false;+ }+ }+ if out.ends_with(' ') {+ out.pop();+ }+ out+}++/// Append `next` to `accum` as the continuation of a paragraph. De-hyphenates+/// when `accum` ends with `-` and `next` starts with an ASCII lowercase letter;+/// otherwise joins with a single space.+#[cfg(test)]+pub(super) fn append_paragraph_line(accum: &mut String, next: &str) {+ let next = collapse_whitespace(next);+ if next.is_empty() {+ return;+ }+ if accum.is_empty() {+ accum.push_str(&next);+ return;+ }+ let dehyphenate =+ accum.ends_with('-') && next.chars().next().is_some_and(|c| c.is_ascii_lowercase());+ if dehyphenate {+ accum.pop(); // drop the '-'+ accum.push_str(&next);+ } else {+ accum.push(' ');+ accum.push_str(&next);+ }+}++/// Decide whether `cur` continues the paragraph started by `prev`.+pub(super) fn continues_paragraph(prev: &ProjectedLine, cur: &ProjectedLine) -> bool {+ // Anchor only signals a paragraph break when one of the lines is clearly+ // centered while the other isn't — justified prose routinely alternates+ // between Left / Right / Floating dominant anchors as line widths flex,+ // and treating those as paragraph breaks shreds normal text.+ let centered_mismatch = (prev.anchor == Anchor::Center) ^ (cur.anchor == Anchor::Center);+ if centered_mismatch {+ return false;+ }+ if (prev.dominant_font_size - cur.dominant_font_size).abs() > FONT_SIZE_PARAGRAPH_TOLERANCE {+ return false;+ }+ // Uniform-bold ↔ non-bold transition is a paragraph break. Catches+ // body-size headings that share font size with the surrounding prose but+ // are emitted in a bold variant (e.g. Brill-Bold over Brill-Roman). Both+ // sides must have a uniform style for this to fire; mid-line emphasis+ // (None style) falls through to the gap/indent checks so prose with+ // mid-paragraph bold spans doesn't get fragmented.+ if let (Some(p), Some(c)) = (line_uniform_style(prev), line_uniform_style(cur))+ && p.bold != c.bold+ {+ return false;+ }+ if prev.region_path != cur.region_path {+ // Cross-region continuation: the same paragraph can wrap from the+ // bottom of one column into the top of the next. Only bridge regions+ // when the previous line clearly breaks mid-sentence (no terminal+ // punctuation) AND the next line starts with a lowercase letter — a+ // strict signal that catches the column-wrap case while rejecting+ // unrelated paragraphs that happen to sit in adjacent leaves.+ let prev_trim = prev.text.trim_end();+ let ends_open = !prev_trim.ends_with(|c: char| {+ matches!(+ c,+ '.' | '!' | '?' | ':' | ';' | '”' | '"' | ')' | ']' | '。' | '』' | '」'+ )+ });+ let starts_lower = cur+ .text+ .trim_start()+ .chars()+ .next()+ .is_some_and(|c| c.is_lowercase());+ return ends_open && starts_lower;+ }+ if (prev.indent_x - cur.indent_x).abs() > INDENT_TOLERANCE && cur.anchor == Anchor::Left {+ // Indent change on a left-aligned block usually means a new paragraph+ // (block-quote, list, indented passage, etc.). Allow first-line indent+ // by checking only when the *next* line shifts right relative to prev.+ if cur.indent_x > prev.indent_x + INDENT_TOLERANCE {+ return false;+ }+ }+ // Vertical gap check.+ let prev_bottom = prev.bbox.y + prev.bbox.height;+ let gap = cur.bbox.y - prev_bottom;+ let line_height = prev.bbox.height.max(cur.bbox.height).max(1.0);+ gap <= line_height * PARAGRAPH_GAP_MULTIPLIER+}++/// Paragraph accumulator state. We track two parallel representations of the+/// running paragraph text:+///+/// - `raw` — plain text (no emphasis markers). Used for the paragraph-uniform+/// shortcut: if every contributing line had the same uniform style, we wrap+/// the whole paragraph once with `wrap_emphasis(raw, …)` to avoid the+/// `**foo** **bar** **baz**` per-line noise pymupdf4llm warns about.+/// - `inline` — per-line markdown with emphasis baked in via+/// `render_line_inline`. Used when the paragraph contains mid-line emphasis+/// shifts or lines with differing uniform styles.+///+/// `uniform` is `Some((bold, italic))` while every line so far has been a+/// uniformly-styled line sharing the same (bold, italic) flags, and `None` as+/// soon as that invariant breaks.+pub(super) struct ParaAccum {+ pub(super) raw: String,+ pub(super) inline: String,+ pub(super) last: ProjectedLine,+ pub(super) uniform: Option<(bool, bool)>,+}++/// Append `next_line` to a paragraph accumulator. Maintains both the `raw` and+/// `inline` text representations and updates the running `uniform` flag.+/// De-hyphenation runs on the `raw` boundary; the `inline` boundary mirrors it+/// when the trailing char is still a literal `-` (i.e. the hyphen sits outside+/// any emphasis wrap — the common case).+pub(super) fn append_to_paragraph(accum: &mut ParaAccum, next_line: &ProjectedLine) {+ let next_raw = collapse_whitespace(next_line.text.trim());+ if next_raw.is_empty() {+ return;+ }+ let next_inline = render_line_inline(next_line);+ let next_uniform: Option<SpanStyle> = line_uniform_style(next_line);++ if accum.raw.is_empty() {+ accum.raw.push_str(&next_raw);+ accum.inline.push_str(&next_inline);+ accum.uniform = next_uniform.map(|s| (s.bold, s.italic));+ accum.last = next_line.clone();+ return;+ }++ let dehyphenate = accum.raw.ends_with('-')+ && next_raw+ .chars()+ .next()+ .is_some_and(|c| c.is_ascii_lowercase());+ if dehyphenate {+ accum.raw.pop();+ accum.raw.push_str(&next_raw);+ if accum.inline.ends_with('-') {+ accum.inline.pop();+ accum.inline.push_str(&next_inline);+ } else {+ // Hyphen sits inside an emphasis wrap — give up on stripping it+ // from `inline` cleanly. Join with a space; raw is still correct+ // for the uniform-paragraph case.+ accum.inline.push(' ');+ accum.inline.push_str(&next_inline);+ }+ } else {+ accum.raw.push(' ');+ accum.raw.push_str(&next_raw);+ accum.inline.push(' ');+ accum.inline.push_str(&next_inline);+ }++ // Maintain the running uniform-style flag.+ accum.uniform = match (accum.uniform, next_uniform) {+ (Some(cur), Some(s)) if cur == (s.bold, s.italic) => Some(cur),+ _ => None,+ };+ accum.last = next_line.clone();+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn dehyphenation_only_for_lowercase_followups() {+ let mut s = String::from("co-");+ append_paragraph_line(&mut s, "operate");+ assert_eq!(s, "cooperate");++ let mut s = String::from("Vitamin-");+ append_paragraph_line(&mut s, "A");+ assert_eq!(s, "Vitamin- A");+ }+}diff --git a/crates/liteparse/src/markdown_layout/repetition.rs b/crates/liteparse/src/markdown_layout/repetition.rsnew file mode 100644index 0000000..1b8a678--- /dev/null+++ b/crates/liteparse/src/markdown_layout/repetition.rs@@ -0,0 +1,170 @@+use crate::types::{ParsedPage, ProjectedLine};++use super::paragraphs::collapse_whitespace;++/// Fraction of page height treated as the "top band" for header detection.+/// Most running headers sit within the top 8–12% of a page; 12% gives some+/// slack for two-line headers without sweeping in body text.+pub(super) const HEADER_BAND_FRACTION: f32 = 0.12;++/// Fraction of page height treated as the "bottom band" for footer detection.+pub(super) const FOOTER_BAND_FRACTION: f32 = 0.12;++/// Fraction of pages on which a normalized line must appear (in the same+/// band) to be classified as a running header/footer.+const HEADER_FOOTER_MIN_FRACTION: f32 = 0.5;++/// Absolute floor on header/footer matches — single-page docs can't have a+/// "running" header by definition, and a single match on a 2-page doc is too+/// weak to act on.+const HEADER_FOOTER_MIN_PAGES: usize = 2;++/// Normalize a line for cross-page header/footer matching. Lowercases,+/// collapses whitespace, and replaces every run of ASCII digits with `#` so+/// `Page 1 of 6` and `Page 2 of 6` collapse to the same key.+pub(super) fn normalize_for_repetition(s: &str) -> String {+ let collapsed = collapse_whitespace(s).to_lowercase();+ let mut out = String::with_capacity(collapsed.len());+ let mut in_digits = false;+ for c in collapsed.chars() {+ if c.is_ascii_digit() {+ if !in_digits {+ out.push('#');+ in_digits = true;+ }+ } else {+ out.push(c);+ in_digits = false;+ }+ }+ out+}++/// Cross-page repetition detector. Returns the set of normalized strings that+/// appear in the top or bottom band of ≥ `HEADER_FOOTER_MIN_FRACTION` of+/// pages (capped below by `HEADER_FOOTER_MIN_PAGES`). The caller uses this to+/// filter `ProjectedLine`s before classification.+///+/// "Same band" means a line whose top is within `HEADER_BAND_FRACTION` of the+/// page top (header) or whose bottom is within `FOOTER_BAND_FRACTION` of the+/// page bottom (footer). Header and footer bands are tracked separately so a+/// company name that appears as both a header and a body-section title on+/// different pages isn't stripped from the body.+pub fn compute_header_footer_set(pages: &[ParsedPage]) -> std::collections::HashSet<String> {+ use std::collections::{HashMap, HashSet};+ let mut set: HashSet<String> = HashSet::new();+ if pages.len() < HEADER_FOOTER_MIN_PAGES {+ return set;+ }+ // Two counters keyed by `(band, normalized_text)` — band is `'h'` or `'f'`.+ let mut counts: HashMap<(char, String), HashSet<usize>> = HashMap::new();+ for page in pages {+ let header_cutoff = page.page_height * HEADER_BAND_FRACTION;+ let footer_cutoff = page.page_height * (1.0 - FOOTER_BAND_FRACTION);+ for line in &page.projected_lines {+ let text = line.text.trim();+ if text.is_empty() {+ continue;+ }+ let norm = normalize_for_repetition(text);+ if norm.is_empty() {+ continue;+ }+ // Header band: top of line within the top band.+ if line.bbox.y <= header_cutoff {+ counts+ .entry(('h', norm.clone()))+ .or_default()+ .insert(page.page_number);+ }+ // Footer band: bottom of line at or below the footer cutoff.+ let line_bottom = line.bbox.y + line.bbox.height;+ if line_bottom >= footer_cutoff {+ counts+ .entry(('f', norm))+ .or_default()+ .insert(page.page_number);+ }+ }+ }+ let threshold = (pages.len() as f32 * HEADER_FOOTER_MIN_FRACTION)+ .ceil()+ .max(HEADER_FOOTER_MIN_PAGES as f32) as usize;+ for ((_, norm), pages_seen) in counts {+ if pages_seen.len() >= threshold {+ set.insert(norm);+ }+ }+ set+}++/// Returns true if `line` (located on `page`) matches the running+/// header/footer set: the line sits in the top or bottom band AND its+/// normalized text is in `header_footer`.+pub(super) fn is_header_or_footer(+ line: &ProjectedLine,+ page: &ParsedPage,+ header_footer: &std::collections::HashSet<String>,+) -> bool {+ if header_footer.is_empty() {+ return false;+ }+ let header_cutoff = page.page_height * HEADER_BAND_FRACTION;+ let footer_cutoff = page.page_height * (1.0 - FOOTER_BAND_FRACTION);+ let in_band = line.bbox.y <= header_cutoff || line.bbox.y + line.bbox.height >= footer_cutoff;+ if !in_band {+ return false;+ }+ let norm = normalize_for_repetition(line.text.trim());+ header_footer.contains(&norm)+}++#[cfg(test)]+mod tests {+ use super::super::test_helpers::header_footer_page;+ use super::*;++ #[test]+ fn normalize_collapses_digits_and_case() {+ assert_eq!(normalize_for_repetition("Page 1 of 6"), "page # of #");+ assert_eq!(normalize_for_repetition("PAGE 12 OF 6"), "page # of #");+ assert_eq!(normalize_for_repetition("Confidential"), "confidential");+ }++ #[test]+ fn detects_repeating_header_and_footer() {+ let pages = vec![+ header_footer_page(1, "Acme Confidential", "Page 1 of 3", "Body one."),+ header_footer_page(2, "Acme Confidential", "Page 2 of 3", "Body two."),+ header_footer_page(3, "Acme Confidential", "Page 3 of 3", "Body three."),+ ];+ let set = compute_header_footer_set(&pages);+ assert!(set.contains("acme confidential"));+ assert!(set.contains("page # of #"));+ }++ #[test]+ fn skips_repetition_check_on_single_page() {+ let pages = vec![header_footer_page(1, "Solo", "footer", "body")];+ let set = compute_header_footer_set(&pages);+ assert!(set.is_empty());+ }++ #[test]+ fn body_text_not_classified_as_header() {+ // Same text in the body of every page should NOT be stripped — only+ // text within the top/bottom band qualifies.+ let mut pages = Vec::new();+ for n in 1..=3 {+ let mut p = header_footer_page(n, "unique header", "unique footer", "shared body text");+ // Move "shared body text" out of header/footer bands (already at y=50 in mid-page).+ // No-op — just illustrating intent.+ p.projected_lines[0].text = format!("unique header {n}");+ p.projected_lines[2].text = format!("unique footer {n}");+ pages.push(p);+ }+ let set = compute_header_footer_set(&pages);+ // "shared body text" sits mid-page — never matched.+ assert!(!set.contains("shared body text"));+ }+}diff --git a/crates/liteparse/src/markdown_layout/tables.rs b/crates/liteparse/src/markdown_layout/tables.rsnew file mode 100644index 0000000..102270b--- /dev/null+++ b/crates/liteparse/src/markdown_layout/tables.rs@@ -0,0 +1,1118 @@+use crate::types::{GraphicPrimitive, ProjectedLine, TextItem};++use super::blocks::Block;+use super::inline::is_bold_span;+use super::paragraphs::collapse_whitespace;++/// Minimum cells per row for a region to qualify as a table.+pub(super) const TABLE_MIN_COLUMNS: usize = 3;++/// Minimum consecutive rows for a region to qualify as a table.+const TABLE_MIN_ROWS: usize = 2;++/// Gap between adjacent spans (in multiples of dominant font size) above which+/// we treat the gap as a cell boundary.+const TABLE_CELL_GAP_FONT_MULTIPLIER: f32 = 1.0;++/// Tolerance (points) for matching a cell's start-x to an existing column+/// track when extending a candidate table run.+const TABLE_TRACK_TOLERANCE_PT: f32 = 6.0;++/// Maximum vertical gap between consecutive table rows, expressed in multiples+/// of the line height. Looser than the paragraph rule because table rows often+/// have more vertical padding than prose lines.+const TABLE_ROW_GAP_MULTIPLIER: f32 = 2.5;++/// Maximum coefficient-of-variation for row spacing within a confident table+/// (rejecting irregular spacing that's more likely prose or a footer block).+const TABLE_ROW_SPACING_MAX_CV: f32 = 0.5;++/// One cell within a tabular row: contributing spans aggregated to text and+/// its leftmost x position, used to align cells across rows into column+/// "tracks".+#[derive(Debug, Clone)]+pub(super) struct TableCell {+ pub(super) start_x: f32,+ /// Right edge of the cell (x of the last span's right). Used by+ /// `recover_merged_cell` to detect cells that straddle two column tracks+ /// when the projection merged two adjacent words into one span.+ pub(super) end_x: f32,+ pub(super) text: String,+ pub(super) bold: bool,+}++/// A contiguous tabular run: line indices `[start, end)` plus the detected+/// rows. Used so the line-classifier can skip the consumed range and so+/// fallback rendering can reach back for the original projected text.+#[derive(Debug, Clone)]+pub(super) struct TableRun {+ pub(super) start: usize,+ pub(super) end: usize,+ pub(super) block: Block,+}++/// Split a `ProjectedLine`'s spans into cells. A gap larger than+/// `TABLE_CELL_GAP_FONT_MULTIPLIER × font_size` between adjacent spans starts+/// a new cell; otherwise spans join into the same cell with a single space.+pub(super) fn split_cells(line: &ProjectedLine) -> Vec<TableCell> {+ // Skip whitespace-only spans before computing gaps — leading/trailing+ // empty items would otherwise add spurious cell boundaries.+ let mut spans: Vec<&TextItem> = line+ .spans+ .iter()+ .filter(|s| !s.text.trim().is_empty())+ .collect();+ spans.sort_by(|a, b| a.x.total_cmp(&b.x));+ if spans.is_empty() {+ return Vec::new();+ }+ let font_size = if line.dominant_font_size > 0.0 {+ line.dominant_font_size+ } else {+ line.bbox.height.max(1.0)+ };+ let gap_threshold = font_size * TABLE_CELL_GAP_FONT_MULTIPLIER;++ let mut cells: Vec<TableCell> = Vec::new();+ let mut current_text = String::new();+ let mut current_start = spans[0].x;+ let mut current_bold_chars: usize = 0;+ let mut current_total_chars: usize = 0;+ let mut prev_right = spans[0].x;++ for (i, span) in spans.iter().enumerate() {+ let gap = span.x - prev_right;+ let break_cell = i > 0 && gap > gap_threshold;+ if break_cell {+ let bold = current_total_chars > 0 && current_bold_chars * 2 > current_total_chars;+ cells.push(TableCell {+ start_x: current_start,+ end_x: prev_right,+ text: collapse_whitespace(current_text.trim()),+ bold,+ });+ current_text.clear();+ current_start = span.x;+ current_bold_chars = 0;+ current_total_chars = 0;+ }+ if !current_text.is_empty() && !current_text.ends_with(' ') {+ current_text.push(' ');+ }+ current_text.push_str(&span.text);+ let n = span.text.chars().count();+ current_total_chars += n;+ if is_bold_span(span) {+ current_bold_chars += n;+ }+ prev_right = span.x + span.width.max(0.0);+ }+ if !current_text.trim().is_empty() {+ let bold = current_total_chars > 0 && current_bold_chars * 2 > current_total_chars;+ cells.push(TableCell {+ start_x: current_start,+ end_x: prev_right,+ text: collapse_whitespace(current_text.trim()),+ bold,+ });+ }+ cells+}++/// When a candidate row has fewer cells than the established column count,+/// look for cells whose x-range straddles multiple column tracks (likely two+/// or more adjacent words that PDFium merged into a single text run) and+/// split each on internal whitespace at the boundaries nearest to the+/// straddled tracks.+///+/// Returns the patched cells if every short cell could be cleanly split to+/// recover `tracks.len()` cells total; otherwise `None`.+pub(super) fn recover_merged_cell(mut cells: Vec<TableCell>, tracks: &[f32]) -> Option<Vec<TableCell>> {+ let target = tracks.len();+ if cells.len() >= target {+ return None;+ }+ // Repeatedly find the cell that straddles the most tracks (≥2) and split+ // it. Each iteration strictly grows `cells.len()`, so termination is+ // guaranteed; if no cell straddles ≥2 tracks before we hit the target,+ // recovery fails.+ while cells.len() < target {+ let mut best_i: Option<usize> = None;+ let mut best_count: usize = 1;+ let mut best_contained: Vec<f32> = Vec::new();+ for (i, cell) in cells.iter().enumerate() {+ let contained: Vec<f32> = tracks+ .iter()+ .copied()+ .filter(|t| {+ *t >= cell.start_x - TABLE_TRACK_TOLERANCE_PT+ && *t <= cell.end_x + TABLE_TRACK_TOLERANCE_PT+ })+ .collect();+ if contained.len() > best_count {+ best_count = contained.len();+ best_i = Some(i);+ best_contained = contained;+ }+ }+ let Some(i) = best_i else {+ return None;+ };+ let cell = cells[i].clone();+ let chars: Vec<char> = cell.text.trim().chars().collect();+ let n = chars.len();+ if n == 0 || best_contained.len() < 2 {+ return None;+ }+ let text_width = (cell.end_x - cell.start_x).max(1.0);+ // For each track after the first, pick the whitespace boundary in+ // `chars` whose linearly-interpolated x is closest to the track.+ let mut split_indices: Vec<usize> = Vec::new();+ for t in best_contained.iter().skip(1) {+ let mut best: Option<(usize, f32)> = None;+ for (k, ch) in chars.iter().enumerate() {+ if !ch.is_whitespace() {+ continue;+ }+ if split_indices.contains(&k) {+ continue;+ }+ let frac = k as f32 / n as f32;+ let x = cell.start_x + frac * text_width;+ let d = (x - t).abs();+ if best.as_ref().is_none_or(|b| d < b.1) {+ best = Some((k, d));+ }+ }+ let (k, _) = best?;+ split_indices.push(k);+ }+ split_indices.sort();+ // Build the split pieces.+ let mut pieces: Vec<String> = Vec::new();+ let mut prev = 0usize;+ for k in &split_indices {+ let piece: String = chars[prev..*k]+ .iter()+ .collect::<String>()+ .trim()+ .to_string();+ if piece.is_empty() {+ return None;+ }+ pieces.push(piece);+ prev = *k;+ }+ let last: String = chars[prev..].iter().collect::<String>().trim().to_string();+ if last.is_empty() {+ return None;+ }+ pieces.push(last);+ if pieces.len() != best_contained.len() {+ return None;+ }+ // Synthesize new TableCells aligned with each track.+ let mut new_cells: Vec<TableCell> = Vec::with_capacity(pieces.len());+ for (p, piece) in pieces.iter().enumerate() {+ let start_x = if p == 0 {+ cell.start_x+ } else {+ best_contained[p]+ };+ let end_x = if p + 1 < best_contained.len() {+ (best_contained[p + 1] - 1.0).max(start_x)+ } else {+ cell.end_x+ };+ new_cells.push(TableCell {+ start_x,+ end_x,+ text: piece.clone(),+ bold: cell.bold,+ });+ }+ cells.remove(i);+ for (offset, c) in new_cells.into_iter().enumerate() {+ cells.insert(i + offset, c);+ }+ }+ if cells.len() == target {+ Some(cells)+ } else {+ None+ }+}++/// Vertical-gap check for table rows. Looser than paragraph continuation+/// because table rows often have extra padding between them.+fn table_rows_adjacent(prev: &ProjectedLine, cur: &ProjectedLine) -> bool {+ // Intentionally don't require region_path equality. Indented sub-group+ // rows (e.g. an indented "MEMORYBANK" row in a grouped academic results+ // table) sometimes land in a different XY-cut leaf than the rest of the+ // table — but the column-track alignment and y-gap checks below are+ // strong enough signals on their own to keep us from spuriously+ // bridging unrelated regions.+ let prev_bottom = prev.bbox.y + prev.bbox.height;+ let gap = cur.bbox.y - prev_bottom;+ let line_height = prev.bbox.height.max(cur.bbox.height).max(1.0);+ gap >= -line_height && gap <= line_height * TABLE_ROW_GAP_MULTIPLIER+}++/// Coefficient of variation (std-dev / mean) of inter-row vertical gaps.+/// Returns 0.0 for runs with <2 gaps (nothing to compare). Used to reject+/// runs whose row spacing is too irregular to be a real table.+fn row_spacing_cv(rows: &[(usize, &ProjectedLine, Vec<TableCell>)]) -> f32 {+ if rows.len() < 3 {+ return 0.0;+ }+ let gaps: Vec<f32> = rows+ .windows(2)+ .map(|w| (w[1].1.bbox.y - w[0].1.bbox.y).abs())+ .collect();+ let mean = gaps.iter().sum::<f32>() / gaps.len() as f32;+ if mean <= 0.0 {+ return f32::INFINITY;+ }+ let var = gaps.iter().map(|g| (g - mean).powi(2)).sum::<f32>() / gaps.len() as f32;+ var.sqrt() / mean+}++/// Try to extend a candidate table starting at `start_idx`. On success returns+/// a `TableRun` with `Block::Table` or `Block::GridFallback`; on failure+/// returns `None` (and the caller should fall through to per-line+/// classification).+fn try_detect_table(lines: &[ProjectedLine], start_idx: usize) -> Option<TableRun> {+ let first_cells = split_cells(&lines[start_idx]);+ if first_cells.len() < TABLE_MIN_COLUMNS {+ return None;+ }++ let mut rows: Vec<(usize, &ProjectedLine, Vec<TableCell>)> =+ vec![(start_idx, &lines[start_idx], first_cells.clone())];+ let column_count = first_cells.len();+ let tracks: Vec<f32> = first_cells.iter().map(|c| c.start_x).collect();++ let mut j = start_idx + 1;+ while j < lines.len() {+ if !table_rows_adjacent(rows.last().unwrap().1, &lines[j]) {+ break;+ }+ let mut cells = split_cells(&lines[j]);+ if cells.len() < column_count && cells.len() >= TABLE_MIN_COLUMNS {+ // PDFium occasionally merges two (or more) adjacent words into one+ // text run when inter-word kerning is tighter than the gap+ // threshold — common in tightly-set numeric tables (e.g. the+ // "MEMORYBANK 5.00 4.77" case on page 6 of the AMEM paper).+ // Recover by splitting straddling cells on internal whitespace.+ if let Some(patched) = recover_merged_cell(cells.clone(), &tracks) {+ cells = patched;+ }+ }+ // Wrapped-continuation merge: a partial-cell line that sits tight+ // beneath (or overlaps) the previous row AND whose cells all align+ // with existing column tracks is a *wrap* of the prior row, not a+ // new row. Common in borderless tables where one column has a+ // multi-line cell while neighbouring columns stay on one line. Merge+ // each wrap cell into its matching track's text rather than+ // breaking the run.+ if cells.len() < column_count && !cells.is_empty() {+ let line_height = lines[j].bbox.height.max(1.0);+ let prev_y_top = rows.last().unwrap().1.bbox.y;+ let centroid_dy = lines[j].bbox.y - prev_y_top;+ let all_align_track = cells.iter().all(|c| {+ tracks+ .iter()+ .any(|t| (c.start_x - *t).abs() <= TABLE_TRACK_TOLERANCE_PT)+ });+ if centroid_dy <= line_height * 1.5 && all_align_track {+ let prev_cells = &mut rows.last_mut().unwrap().2;+ for c in &cells {+ if let Some(idx) = tracks+ .iter()+ .position(|t| (c.start_x - *t).abs() <= TABLE_TRACK_TOLERANCE_PT)+ {+ if !prev_cells[idx].text.is_empty() && !c.text.is_empty() {+ prev_cells[idx].text.push(' ');+ }+ prev_cells[idx].text.push_str(&c.text);+ }+ }+ j += 1;+ continue;+ }+ }+ if cells.len() != column_count {+ break;+ }+ // Allow at most one column track to drift out of tolerance, which lets+ // grouped row-labels in academic tables (e.g. an indented "MEMORYBANK"+ // row whose label column shifts right by ~30pt while the numeric+ // columns stay aligned) stay in the same run. Without this slack a+ // single indented label fragments a 6-row table into three 2-row chunks.+ let misaligned = cells+ .iter()+ .zip(tracks.iter())+ .filter(|(c, t)| (c.start_x - **t).abs() > TABLE_TRACK_TOLERANCE_PT)+ .count();+ if misaligned > 1 {+ break;+ }+ rows.push((j, &lines[j], cells));+ j += 1;+ }++ if rows.len() < TABLE_MIN_ROWS {+ return None;+ }++ let cv = row_spacing_cv(&rows);+ let end = j;++ if cv > TABLE_ROW_SPACING_MAX_CV {+ // Suggestive layout but the row cadence is too irregular to trust as a+ // clean table — surface as a fenced fallback so the structure is at+ // least preserved.+ let raw: Vec<String> = rows+ .iter()+ .map(|(_, line, _)| line.text.trim_end().to_string())+ .collect();+ return Some(TableRun {+ start: start_idx,+ end,+ block: Block::GridFallback { lines: raw },+ });+ }++ // Promote the first row to header iff every cell in it is bold (matches+ // pymupdf4llm's "bold-or-filled" heuristic; fills require fork data).+ let first_row = &rows[0].2;+ let header_qualifies = first_row.iter().all(|c| c.bold);+ let header = if header_qualifies {+ Some(first_row.iter().map(|c| c.text.clone()).collect())+ } else {+ None+ };+ let row_start = if header.is_some() { 1 } else { 0 };+ let body_rows: Vec<Vec<String>> = rows[row_start..]+ .iter()+ .map(|(_, _, cells)| cells.iter().map(|c| c.text.clone()).collect())+ .collect();+ if header.is_none() && body_rows.len() < TABLE_MIN_ROWS {+ return None;+ }++ Some(TableRun {+ start: start_idx,+ end,+ block: Block::Table {+ header,+ rows: body_rows,+ },+ })+}++/// Scan `lines` once and return all detected tabular regions (sorted by+/// `start`). Caller uses these as cut-points so the per-line classifier never+/// sees lines inside a table.+pub(super) fn detect_tables(lines: &[ProjectedLine]) -> Vec<TableRun> {+ let mut out = Vec::new();+ let mut i = 0;+ while i < lines.len() {+ if let Some(run) = try_detect_table(lines, i) {+ i = run.end;+ out.push(run);+ } else {+ i += 1;+ }+ }+ out+}++// ── Ruled-grid table detection ─────────────────────────────────────────────+//+// Detect tables drawn with explicit horizontal + vertical rules (the "Strong"+// mode in MARKDOWN_PLAN.md). Strokes are clustered into H/V grid lines, then+// union-find groups crossing lines into table regions. For each region the+// distinct row/column boundaries form a cell grid; text lines are assigned to+// cells by centroid containment.+//+// Ruled tables are detected before the borderless `detect_tables`. The caller+// merges the two outputs; overlapping ranges defer to the ruled run because+// path-based geometry is a strictly stronger signal than text alignment alone.++/// Horizontal segment in viewport coords (top-left origin). `y` is the rule's+/// y-position; `x_min..x_max` is its horizontal span. Endpoints of multiple+/// short segments sharing a y get unioned into one wider segment during+/// clustering.+#[derive(Debug, Clone, Copy)]+struct HSeg {+ x_min: f32,+ x_max: f32,+ y: f32,+}++#[derive(Debug, Clone, Copy)]+struct VSeg {+ y_min: f32,+ y_max: f32,+ x: f32,+}++/// Strokes are considered "axis-aligned" when the perpendicular delta is at+/// most this many points. Generous to absorb antialiased near-pixel strokes.+const TABLE_AXIS_TOLERANCE_PT: f32 = 1.0;++/// Two H lines (or two V lines) are merged into one grid line when their+/// perpendicular coords are within this many points. Slightly looser than the+/// axis tolerance because rules drawn at the same row can have ±1pt jitter+/// from different stroke widths.+const TABLE_GRID_CLUSTER_PT: f32 = 2.0;++/// Slack added when checking whether a V line "crosses" an H line. Helps+/// when rules don't quite reach the corner because the PDF drew them as+/// individual segments with small gaps.+const TABLE_CROSS_TOLERANCE_PT: f32 = 3.0;++/// Reject ruled-table candidates whose empty-cell fraction exceeds this.+const TABLE_MAX_EMPTY_CELL_FRACTION: f32 = 0.30;++/// Reject candidates whose grid covers nearly the whole page — almost always+/// a page border, not a real table.+const TABLE_MAX_PAGE_COVERAGE: f32 = 0.95;++/// Extract horizontal and vertical line segments from a page's graphics. Each+/// `Stroke` becomes one HSeg or VSeg depending on orientation; each stroked+/// `Rect` contributes its four edges (cell-border rects, table frames).+fn extract_h_v_segments(graphics: &[GraphicPrimitive]) -> (Vec<HSeg>, Vec<VSeg>) {+ let mut hs = Vec::new();+ let mut vs = Vec::new();+ for g in graphics {+ match g {+ GraphicPrimitive::Stroke { x1, y1, x2, y2, .. } => {+ let (x1, y1, x2, y2) = (*x1, *y1, *x2, *y2);+ let dy = (y1 - y2).abs();+ let dx = (x1 - x2).abs();+ if dy <= TABLE_AXIS_TOLERANCE_PT && dx > 1.0 {+ hs.push(HSeg {+ x_min: x1.min(x2),+ x_max: x1.max(x2),+ y: (y1 + y2) * 0.5,+ });+ } else if dx <= TABLE_AXIS_TOLERANCE_PT && dy > 1.0 {+ vs.push(VSeg {+ y_min: y1.min(y2),+ y_max: y1.max(y2),+ x: (x1 + x2) * 0.5,+ });+ }+ }+ GraphicPrimitive::Rect { bbox, stroke, .. } => {+ if stroke.is_none() {+ continue;+ }+ let top = bbox.y;+ let bottom = bbox.y + bbox.height;+ let left = bbox.x;+ let right = bbox.x + bbox.width;+ if bbox.width > 1.0 {+ hs.push(HSeg {+ x_min: left,+ x_max: right,+ y: top,+ });+ hs.push(HSeg {+ x_min: left,+ x_max: right,+ y: bottom,+ });+ }+ if bbox.height > 1.0 {+ vs.push(VSeg {+ y_min: top,+ y_max: bottom,+ x: left,+ });+ vs.push(VSeg {+ y_min: top,+ y_max: bottom,+ x: right,+ });+ }+ }+ }+ }+ (hs, vs)+}++/// Cluster H segments sharing a y-coordinate (within `TABLE_GRID_CLUSTER_PT`)+/// into a single wider grid line whose x-extent is the union of the inputs.+fn cluster_h_segments(mut segs: Vec<HSeg>) -> Vec<HSeg> {+ if segs.is_empty() {+ return segs;+ }+ segs.sort_by(|a, b| a.y.total_cmp(&b.y));+ let mut out: Vec<HSeg> = Vec::with_capacity(segs.len());+ for seg in segs {+ if let Some(last) = out.last_mut()+ && (last.y - seg.y).abs() <= TABLE_GRID_CLUSTER_PT+ {+ last.x_min = last.x_min.min(seg.x_min);+ last.x_max = last.x_max.max(seg.x_max);+ continue;+ }+ out.push(seg);+ }+ out+}++fn cluster_v_segments(mut segs: Vec<VSeg>) -> Vec<VSeg> {+ if segs.is_empty() {+ return segs;+ }+ segs.sort_by(|a, b| a.x.total_cmp(&b.x));+ let mut out: Vec<VSeg> = Vec::with_capacity(segs.len());+ for seg in segs {+ if let Some(last) = out.last_mut()+ && (last.x - seg.x).abs() <= TABLE_GRID_CLUSTER_PT+ {+ last.y_min = last.y_min.min(seg.y_min);+ last.y_max = last.y_max.max(seg.y_max);+ continue;+ }+ out.push(seg);+ }+ out+}++/// Union-find root with path compression.+fn uf_find(parent: &mut [usize], mut x: usize) -> usize {+ while parent[x] != x {+ parent[x] = parent[parent[x]];+ x = parent[x];+ }+ x+}++fn uf_union(parent: &mut [usize], a: usize, b: usize) {+ let ra = uf_find(parent, a);+ let rb = uf_find(parent, b);+ if ra != rb {+ parent[ra] = rb;+ }+}++/// Group H/V grid lines that cross each other into connected components.+/// Each component is a candidate ruled table — typically one component per+/// distinct table on the page. Returns `(h_indices, v_indices)` per component,+/// dropping components without ≥2 H and ≥2 V lines (a single L-shape doesn't+/// make a table).+fn find_grid_components(hs: &[HSeg], vs: &[VSeg]) -> Vec<(Vec<usize>, Vec<usize>)> {+ let n_h = hs.len();+ let n_v = vs.len();+ if n_h < 2 || n_v < 2 {+ return Vec::new();+ }+ let n = n_h + n_v;+ let mut parent: Vec<usize> = (0..n).collect();+ let mut connected = vec![false; n];++ let tol = TABLE_CROSS_TOLERANCE_PT;+ for (i, h) in hs.iter().enumerate() {+ for (j, v) in vs.iter().enumerate() {+ let v_crosses_h_x = v.x >= h.x_min - tol && v.x <= h.x_max + tol;+ let h_crosses_v_y = h.y >= v.y_min - tol && h.y <= v.y_max + tol;+ if v_crosses_h_x && h_crosses_v_y {+ uf_union(&mut parent, i, n_h + j);+ connected[i] = true;+ connected[n_h + j] = true;+ }+ }+ }++ use std::collections::HashMap;+ let mut groups: HashMap<usize, (Vec<usize>, Vec<usize>)> = HashMap::new();+ for i in 0..n_h {+ if !connected[i] {+ continue;+ }+ let r = uf_find(&mut parent, i);+ groups.entry(r).or_default().0.push(i);+ }+ for j in 0..n_v {+ if !connected[n_h + j] {+ continue;+ }+ let r = uf_find(&mut parent, n_h + j);+ groups.entry(r).or_default().1.push(j);+ }+ groups+ .into_values()+ .filter(|(h_idx, v_idx)| h_idx.len() >= 2 && v_idx.len() >= 2)+ .collect()+}++/// Build a `TableRun` for one ruled-grid component. Returns `None` if the+/// resulting grid is too small (< 2 cols or < 2 rows), covers nearly the+/// whole page (likely the page border), or is mostly empty cells.+fn build_ruled_table(+ hs: &[HSeg],+ vs: &[VSeg],+ h_indices: &[usize],+ v_indices: &[usize],+ lines: &[ProjectedLine],+ page_width: f32,+ page_height: f32,+) -> Option<TableRun> {+ // Distinct row y-coords (cluster again — multiple H lines may share a y).+ let mut ys: Vec<f32> = h_indices.iter().map(|&i| hs[i].y).collect();+ ys.sort_by(|a, b| a.total_cmp(b));+ dedup_close(&mut ys, TABLE_GRID_CLUSTER_PT);++ let mut xs: Vec<f32> = v_indices.iter().map(|&i| vs[i].x).collect();+ xs.sort_by(|a, b| a.total_cmp(b));+ dedup_close(&mut xs, TABLE_GRID_CLUSTER_PT);++ // Need ≥2 cells per axis → ≥3 boundary lines.+ if ys.len() < 3 || xs.len() < 3 {+ return None;+ }++ let n_rows = ys.len() - 1;+ let n_cols = xs.len() - 1;+ let bbox = crate::types::Rect {+ x: xs[0],+ y: ys[0],+ width: xs[n_cols] - xs[0],+ height: ys[n_rows] - ys[0],+ };++ // Reject page-border-as-table.+ if page_width > 0.0 && page_height > 0.0 {+ let coverage = (bbox.width / page_width) * (bbox.height / page_height);+ if coverage > TABLE_MAX_PAGE_COVERAGE {+ return None;+ }+ }++ // Assign each text line to its cell by centroid.+ let mut cells: Vec<Vec<String>> = vec![vec![String::new(); n_cols]; n_rows];+ let mut cell_is_bold: Vec<Vec<bool>> = vec![vec![true; n_cols]; n_rows];+ let mut cell_has_text: Vec<Vec<bool>> = vec![vec![false; n_cols]; n_rows];+ let mut consumed_indices: Vec<usize> = Vec::new();++ for (idx, line) in lines.iter().enumerate() {+ let cx = line.bbox.x + line.bbox.width * 0.5;+ let cy = line.bbox.y + line.bbox.height * 0.5;+ if cy < ys[0] || cy > ys[n_rows] || cx < xs[0] || cx > xs[n_cols] {+ continue;+ }+ let row = match find_bucket(&ys, cy) {+ Some(r) => r,+ None => continue,+ };+ let col = match find_bucket(&xs, cx) {+ Some(c) => c,+ None => continue,+ };+ let txt = line.text.trim();+ if txt.is_empty() {+ continue;+ }+ if !cells[row][col].is_empty() {+ cells[row][col].push(' ');+ }+ cells[row][col].push_str(txt);+ cell_has_text[row][col] = true;+ if !line.all_bold {+ cell_is_bold[row][col] = false;+ }+ consumed_indices.push(idx);+ }++ if consumed_indices.is_empty() {+ return None;+ }++ let total = n_rows * n_cols;+ let empty_count = cell_has_text+ .iter()+ .flatten()+ .filter(|filled| !**filled)+ .count();+ if (empty_count as f32) / (total as f32) > TABLE_MAX_EMPTY_CELL_FRACTION {+ return None;+ }++ // Header = first row iff every non-empty cell in it is bold.+ let header_qualifies = cell_has_text[0]+ .iter()+ .zip(cell_is_bold[0].iter())+ .all(|(has, bold)| !has || *bold)+ && cell_has_text[0].iter().any(|has| *has);+ let header = if header_qualifies {+ Some(cells[0].clone())+ } else {+ None+ };+ let body_start = if header.is_some() { 1 } else { 0 };+ let body_rows: Vec<Vec<String>> = cells[body_start..].to_vec();+ if body_rows.is_empty() {+ return None;+ }++ // Line index span this table covers.+ let start = *consumed_indices.iter().min().unwrap();+ let end = *consumed_indices.iter().max().unwrap() + 1;++ Some(TableRun {+ start,+ end,+ block: Block::Table {+ header,+ rows: body_rows,+ },+ })+}++/// In-place dedup of a sorted Vec, collapsing entries within `tol` to the+/// first of each cluster.+fn dedup_close(v: &mut Vec<f32>, tol: f32) {+ if v.len() < 2 {+ return;+ }+ let mut out: Vec<f32> = Vec::with_capacity(v.len());+ for x in v.iter().copied() {+ if let Some(&last) = out.last()+ && (x - last).abs() <= tol+ {+ continue;+ }+ out.push(x);+ }+ *v = out;+}++/// Find the bucket index `i` such that `boundaries[i] <= val < boundaries[i+1]`.+/// Returns `None` if `val` is outside the boundaries.+fn find_bucket(boundaries: &[f32], val: f32) -> Option<usize> {+ if boundaries.len() < 2 || val < boundaries[0] || val > *boundaries.last().unwrap() {+ return None;+ }+ for (i, w) in boundaries.windows(2).enumerate() {+ if val >= w[0] && val <= w[1] {+ return Some(i);+ }+ }+ None+}++/// Detect ruled-grid tables on a page from its vector graphics. Returns runs+/// in document order (sorted by `start`).+pub(super) fn detect_ruled_tables(+ lines: &[ProjectedLine],+ graphics: &[GraphicPrimitive],+ page_width: f32,+ page_height: f32,+) -> Vec<TableRun> {+ let (hs, vs) = extract_h_v_segments(graphics);+ let hs = cluster_h_segments(hs);+ let vs = cluster_v_segments(vs);+ if hs.len() < 2 || vs.len() < 2 {+ return Vec::new();+ }+ let components = find_grid_components(&hs, &vs);+ let mut out = Vec::new();+ for (h_idx, v_idx) in components {+ if let Some(run) =+ build_ruled_table(&hs, &vs, &h_idx, &v_idx, lines, page_width, page_height)+ {+ out.push(run);+ }+ }+ out.sort_by_key(|r| r.start);+ out+}++/// Merge ruled-grid runs with borderless runs into a single sorted list. When+/// ranges overlap the ruled run wins (path-based geometry is strictly stronger+/// than text-alignment heuristics) — overlapping borderless runs are dropped.+pub(super) fn merge_table_runs(mut ruled: Vec<TableRun>, borderless: Vec<TableRun>) -> Vec<TableRun> {+ for b in borderless {+ let overlaps = ruled+ .iter()+ .any(|r| !(b.end <= r.start || b.start >= r.end));+ if !overlaps {+ ruled.push(b);+ }+ }+ ruled.sort_by_key(|r| r.start);+ ruled+}++/// Escape `|` and `\n` inside a markdown table cell so the pipe-table grammar+/// stays valid. Newlines should be impossible inside a single cell (we built+/// cells from spans on the same projected line) but guard anyway.+pub(super) fn escape_table_cell(s: &str) -> String {+ s.replace('\\', "\\\\")+ .replace('|', "\\|")+ .replace('\n', " ")+}++#[cfg(test)]+mod tests {+ use super::super::test_helpers::{line, line_with_spans, rect_borders, stroke};+ use super::*;++ #[test]+ fn split_cells_splits_on_wide_gaps() {+ let l = line_with_spans(&[("A", 50.0), ("B", 150.0), ("C", 250.0)], 100.0, 10.0);+ let cells = split_cells(&l);+ assert_eq!(cells.len(), 3);+ assert_eq!(cells[0].text, "A");+ assert_eq!(cells[1].text, "B");+ assert_eq!(cells[2].text, "C");+ }++ #[test]+ fn recover_merged_cell_splits_off_by_one() {+ // Mimics the page-6 case: row 0 establishes 3 tracks at 50/150/250.+ // Row 1's projection merges "MEMORYBANK" + "5.00" into one span at+ // x=50 width=110, so split_cells yields 2 cells while the table+ // expects 3. Recovery must split on whitespace at the missing track.+ let row = vec![+ TableCell {+ start_x: 50.0,+ end_x: 160.0,+ text: "MEMORYBANK 5.00".into(),+ bold: false,+ },+ TableCell {+ start_x: 250.0,+ end_x: 280.0,+ text: "4.77".into(),+ bold: false,+ },+ ];+ let tracks = vec![50.0, 150.0, 250.0];+ let out = recover_merged_cell(row, &tracks).expect("recovery should succeed");+ assert_eq!(out.len(), 3);+ assert_eq!(out[0].text, "MEMORYBANK");+ assert_eq!(out[1].text, "5.00");+ assert_eq!(out[2].text, "4.77");+ }++ #[test]+ fn recover_merged_cell_splits_off_by_two() {+ // Three merged tokens in one cell: "MEMORYBANK 13.18 10.03" straddles+ // tracks at 50/150/250 and the row has only 2 cells, off by 2.+ let row = vec![+ TableCell {+ start_x: 50.0,+ end_x: 260.0,+ text: "MEMORYBANK 13.18 10.03".into(),+ bold: false,+ },+ TableCell {+ start_x: 350.0,+ end_x: 380.0,+ text: "7.61".into(),+ bold: false,+ },+ ];+ let tracks = vec![50.0, 150.0, 250.0, 350.0];+ let out = recover_merged_cell(row, &tracks).expect("recovery should succeed");+ assert_eq!(out.len(), 4);+ assert_eq!(out[0].text, "MEMORYBANK");+ assert_eq!(out[1].text, "13.18");+ assert_eq!(out[2].text, "10.03");+ assert_eq!(out[3].text, "7.61");+ }++ #[test]+ fn recover_merged_cell_bails_without_enough_whitespace() {+ // A cell that straddles two tracks but has no internal whitespace+ // (e.g. a hyphenated token) can't be safely split — return None.+ let row = vec![TableCell {+ start_x: 50.0,+ end_x: 200.0,+ text: "ABC-DEF-GHI".into(),+ bold: false,+ }];+ let tracks = vec![50.0, 150.0];+ assert!(recover_merged_cell(row, &tracks).is_none());+ }++ #[test]+ fn split_cells_keeps_close_spans_together() {+ // Two spans 2pt apart at 10pt font (gap < font_size) → same cell.+ let l = line_with_spans(&[("Hello", 50.0), ("world", 80.0)], 100.0, 10.0);+ let cells = split_cells(&l);+ assert_eq!(cells.len(), 1);+ assert_eq!(cells[0].text, "Hello world");+ }++ #[test]+ fn rejects_table_when_row_count_too_low() {+ let lines = vec![line_with_spans(+ &[("A", 50.0), ("B", 150.0), ("C", 250.0)],+ 100.0,+ 10.0,+ )];+ let runs = detect_tables(&lines);+ assert!(runs.is_empty());+ }++ #[test]+ fn rejects_table_when_column_count_too_low() {+ let lines = vec![+ line_with_spans(&[("A", 50.0), ("B", 200.0)], 100.0, 10.0),+ line_with_spans(&[("C", 50.0), ("D", 200.0)], 115.0, 10.0),+ ];+ let runs = detect_tables(&lines);+ assert!(runs.is_empty());+ }++ #[test]+ fn escapes_pipe_inside_cell() {+ assert_eq!(escape_table_cell("a|b"), "a\\|b");+ }++ #[test]+ fn ruled_table_2x2_detected() {+ // 2 rows × 2 cols grid: 3 H lines (y=100,140,180), 3 V lines (x=50,150,250)+ // Cell text dropped in the centroid of each cell.+ let mut graphics = Vec::new();+ for y in [100.0_f32, 140.0, 180.0] {+ graphics.push(stroke(50.0, y, 250.0, y, 0.5));+ }+ for x in [50.0_f32, 150.0, 250.0] {+ graphics.push(stroke(x, 100.0, x, 180.0, 0.5));+ }++ // Text lines: one per cell, centered.+ let lines = vec![+ line("a", 90.0, 115.0, 10.0, 10.0), // row 0, col 0+ line("b", 190.0, 115.0, 10.0, 10.0), // row 0, col 1+ line("c", 90.0, 155.0, 10.0, 10.0), // row 1, col 0+ line("d", 190.0, 155.0, 10.0, 10.0), // row 1, col 1+ ];++ let runs = detect_ruled_tables(&lines, &graphics, 612.0, 792.0);+ assert_eq!(runs.len(), 1, "expected 1 ruled table, got {runs:?}");+ match &runs[0].block {+ Block::Table { header, rows } => {+ assert!(header.is_none(), "no bold first row → no header");+ assert_eq!(rows.len(), 2);+ assert_eq!(rows[0], vec!["a", "b"]);+ assert_eq!(rows[1], vec!["c", "d"]);+ }+ other => panic!("expected Block::Table, got {other:?}"),+ }+ }++ #[test]+ fn ruled_table_rect_borders_detected() {+ // Same 2×2 table but drawn as 4 individual cell rects (each cell is a+ // stroked rectangle). Each rect contributes 4 strokes via+ // extract_h_v_segments.+ let mut graphics = Vec::new();+ graphics.extend(rect_borders(50.0, 100.0, 100.0, 40.0)); // r0 c0+ graphics.extend(rect_borders(150.0, 100.0, 100.0, 40.0)); // r0 c1+ graphics.extend(rect_borders(50.0, 140.0, 100.0, 40.0)); // r1 c0+ graphics.extend(rect_borders(150.0, 140.0, 100.0, 40.0)); // r1 c1++ let lines = vec![+ line("a", 90.0, 115.0, 10.0, 10.0),+ line("b", 190.0, 115.0, 10.0, 10.0),+ line("c", 90.0, 155.0, 10.0, 10.0),+ line("d", 190.0, 155.0, 10.0, 10.0),+ ];+ let runs = detect_ruled_tables(&lines, &graphics, 612.0, 792.0);+ assert_eq!(runs.len(), 1);+ }++ #[test]+ fn ruled_table_page_border_rejected() {+ // Single big rect covering ~the whole page → should NOT be treated as a+ // table even though it has H+V lines on all four sides.+ let graphics = rect_borders(10.0, 10.0, 590.0, 770.0);+ let lines = vec![line("body text", 50.0, 400.0, 10.0, 10.0)];+ let runs = detect_ruled_tables(&lines, &graphics, 612.0, 792.0);+ assert!(+ runs.is_empty(),+ "page-border rect should not become a table, got {runs:?}"+ );+ }++ #[test]+ fn ruled_table_mostly_empty_rejected() {+ // 3×3 grid with text in only one cell — empty fraction 8/9 ≈ 89% >> 30%.+ let mut graphics = Vec::new();+ for y in [100.0_f32, 130.0, 160.0, 190.0] {+ graphics.push(stroke(50.0, y, 350.0, y, 0.5));+ }+ for x in [50.0_f32, 150.0, 250.0, 350.0] {+ graphics.push(stroke(x, 100.0, x, 190.0, 0.5));+ }+ let lines = vec![line("only", 90.0, 115.0, 10.0, 10.0)];+ let runs = detect_ruled_tables(&lines, &graphics, 612.0, 792.0);+ assert!(runs.is_empty());+ }++ #[test]+ fn ruled_table_first_row_bold_becomes_header() {+ // 2×2 with first row text marked all_bold → header promotion.+ let mut graphics = Vec::new();+ for y in [100.0_f32, 140.0, 180.0] {+ graphics.push(stroke(50.0, y, 250.0, y, 0.5));+ }+ for x in [50.0_f32, 150.0, 250.0] {+ graphics.push(stroke(x, 100.0, x, 180.0, 0.5));+ }+ let mut a = line("Name", 90.0, 115.0, 10.0, 10.0);+ let mut b = line("Score", 190.0, 115.0, 10.0, 10.0);+ a.all_bold = true;+ b.all_bold = true;+ let lines = vec![+ a,+ b,+ line("alice", 90.0, 155.0, 10.0, 10.0),+ line("99", 190.0, 155.0, 10.0, 10.0),+ ];+ let runs = detect_ruled_tables(&lines, &graphics, 612.0, 792.0);+ assert_eq!(runs.len(), 1);+ match &runs[0].block {+ Block::Table { header, rows } => {+ assert_eq!(+ header.as_deref(),+ Some(&["Name".into(), "Score".into()][..])+ );+ assert_eq!(rows.len(), 1);+ assert_eq!(rows[0], vec!["alice", "99"]);+ }+ other => panic!("expected Block::Table, got {other:?}"),+ }+ }++ #[test]+ fn merge_prefers_ruled_when_overlapping() {+ let ruled = vec![TableRun {+ start: 5,+ end: 10,+ block: Block::Table {+ header: None,+ rows: vec![vec!["ruled".into()]],+ },+ }];+ let borderless = vec![TableRun {+ start: 6,+ end: 11,+ block: Block::GridFallback {+ lines: vec!["bl".into()],+ },+ }];+ let merged = merge_table_runs(ruled, borderless);+ assert_eq!(merged.len(), 1);+ assert!(matches!(&merged[0].block, Block::Table { .. }));+ }+}diff --git a/crates/liteparse/src/markdown_layout/test_helpers.rs b/crates/liteparse/src/markdown_layout/test_helpers.rsnew file mode 100644index 0000000..4a91968--- /dev/null+++ b/crates/liteparse/src/markdown_layout/test_helpers.rs@@ -0,0 +1,202 @@+//! Shared test fixtures for `markdown_layout` submodule tests.++use crate::types::{Anchor, GraphicPrimitive, ParsedPage, ProjectedLine, Rect, TextItem};++pub(crate) fn line(text: &str, x: f32, y: f32, h: f32, size: f32) -> ProjectedLine {+ ProjectedLine {+ text: text.into(),+ bbox: Rect {+ x,+ y,+ width: text.chars().count() as f32 * (size * 0.5),+ height: h,+ },+ anchor: Anchor::Left,+ indent_x: x,+ dominant_font_size: size,+ dominant_font_name: Some("Arial".into()),+ all_bold: false,+ all_italic: false,+ all_mono: false,+ all_strike: false,+ spans: vec![TextItem::default()],+ region_path: Vec::new(),+ mcid: None,+ }+}++pub(crate) fn page(lines: Vec<ProjectedLine>) -> ParsedPage {+ ParsedPage {+ page_number: 1,+ page_width: 612.0,+ page_height: 792.0,+ text: String::new(),+ text_items: vec![],+ projected_lines: lines,+ regions: crate::types::Region::default(),+ graphics: vec![],+ figures: vec![],+ struct_nodes: vec![],+ image_refs: vec![],+ }+}++pub(crate) fn mono_line(text: &str, y: f32) -> ProjectedLine {+ let mut l = line(text, 50.0, y, 10.0, 10.0);+ l.all_mono = true;+ l+}++/// Build a line whose spans are placed at explicit x positions — used to+/// drive the table detector, which relies on per-span x for cell splitting.+pub(crate) fn line_with_spans(cells: &[(&str, f32)], y: f32, size: f32) -> ProjectedLine {+ let spans: Vec<TextItem> = cells+ .iter()+ .map(|(t, x)| TextItem {+ text: (*t).into(),+ x: *x,+ y,+ width: t.chars().count() as f32 * size * 0.5,+ height: size,+ font_size: Some(size),+ font_name: Some("Arial".into()),+ ..Default::default()+ })+ .collect();+ let min_x = spans.iter().map(|s| s.x).fold(f32::INFINITY, f32::min);+ let max_x = spans+ .iter()+ .map(|s| s.x + s.width)+ .fold(f32::NEG_INFINITY, f32::max);+ ProjectedLine {+ text: cells+ .iter()+ .map(|(t, _)| *t)+ .collect::<Vec<_>>()+ .join(" "),+ bbox: Rect {+ x: min_x,+ y,+ width: (max_x - min_x).max(0.0),+ height: size,+ },+ anchor: Anchor::Left,+ indent_x: min_x,+ dominant_font_size: size,+ dominant_font_name: Some("Arial".into()),+ all_bold: false,+ all_italic: false,+ all_mono: false,+ all_strike: false,+ spans,+ region_path: Vec::new(),+ mcid: None,+ }+}++/// Build a line whose spans carry explicit per-span font metadata. Lets us+/// exercise the mid-line emphasis pipeline without needing real PDF input.+pub(crate) fn styled_line(+ spans: &[(&str, f32, Option<&str>)],+ y: f32,+ size: f32,+) -> ProjectedLine {+ let items: Vec<TextItem> = spans+ .iter()+ .map(|(t, x, font)| TextItem {+ text: (*t).into(),+ x: *x,+ y,+ width: t.chars().count() as f32 * size * 0.5,+ height: size,+ font_size: Some(size),+ font_name: font.map(String::from),+ ..Default::default()+ })+ .collect();+ let joined: String = spans+ .iter()+ .map(|(t, _, _)| *t)+ .collect::<Vec<_>>()+ .join(" ");+ let min_x = items.iter().map(|s| s.x).fold(f32::INFINITY, f32::min);+ let max_x = items+ .iter()+ .map(|s| s.x + s.width)+ .fold(f32::NEG_INFINITY, f32::max);+ ProjectedLine {+ text: joined,+ bbox: Rect {+ x: min_x,+ y,+ width: (max_x - min_x).max(0.0),+ height: size,+ },+ anchor: Anchor::Left,+ indent_x: min_x,+ dominant_font_size: size,+ dominant_font_name: Some("Arial".into()),+ all_bold: false,+ all_italic: false,+ all_mono: false,+ all_strike: false,+ spans: items,+ region_path: Vec::new(),+ mcid: None,+ }+}++pub(crate) fn stroke(x1: f32, y1: f32, x2: f32, y2: f32, width: f32) -> GraphicPrimitive {+ GraphicPrimitive::Stroke {+ x1,+ y1,+ x2,+ y2,+ width,+ color: None,+ }+}++pub(crate) fn page_with_graphics(+ lines: Vec<ProjectedLine>,+ graphics: Vec<GraphicPrimitive>,+) -> ParsedPage {+ let mut p = page(lines);+ p.graphics = graphics;+ p+}++pub(crate) fn header_footer_page(n: usize, header: &str, footer: &str, body: &str) -> ParsedPage {+ // Page height 100 → header band ≤12pt, footer band ≥88pt.+ let mut lines = vec![+ line(header, 50.0, 5.0, 8.0, 8.0),+ line(body, 50.0, 50.0, 10.0, 10.0),+ line(footer, 50.0, 92.0, 6.0, 6.0),+ ];+ for l in &mut lines {+ l.region_path = Vec::new();+ }+ ParsedPage {+ page_number: n,+ page_width: 612.0,+ page_height: 100.0,+ text: String::new(),+ text_items: vec![],+ projected_lines: lines,+ regions: crate::types::Region::default(),+ graphics: vec![],+ figures: vec![],+ struct_nodes: vec![],+ image_refs: vec![],+ }+}++/// Helper: build the four borders of a rectangle as four strokes.+pub(crate) fn rect_borders(x: f32, y: f32, w: f32, h: f32) -> Vec<GraphicPrimitive> {+ vec![+ stroke(x, y, x + w, y, 0.5), // top+ stroke(x, y + h, x + w, y + h, 0.5), // bottom+ stroke(x, y, x, y + h, 0.5), // left+ stroke(x + w, y, x + w, y + h, 0.5), // right+ ]+}diff --git a/crates/liteparse/src/output/json.rs b/crates/liteparse/src/output/json.rsindex b6db995..1ac0482 100644--- a/crates/liteparse/src/output/json.rs+++ b/crates/liteparse/src/output/json.rs@@ -95,6 +95,8 @@ mod tests {regions: crate::types::Region::default(),graphics: vec![],figures: vec![],+ struct_nodes: vec![],+ image_refs: vec![],}}diff --git a/crates/liteparse/src/output/markdown.rs b/crates/liteparse/src/output/markdown.rsindex 69a8658..e6aebcb 100644--- a/crates/liteparse/src/output/markdown.rs+++ b/crates/liteparse/src/output/markdown.rs@@ -1,8 +1,9 @@+use crate::config::ImageMode;use crate::markdown_layout::{build_heading_map, classify_page_with_filters, compute_body_size, compute_header_footer_set,render_blocks,};-use crate::types::ParsedPage;+use crate::types::{OutlineTarget, ParsedPage};/// Format parsed pages as markdown.///@@ -15,7 +16,11 @@ use crate::types::ParsedPage;/// `<!-- page N -->` marker. Pages that contain no projected lines (e.g. blank/// or fully-OCR pages without font-size info) fall back to the projected text/// wrapped in a fenced block so we never silently drop content.-pub fn format_markdown(pages: &[ParsedPage]) -> String {+pub fn format_markdown(+ pages: &[ParsedPage],+ outline: &[OutlineTarget],+ image_mode: ImageMode,+) -> String {if pages.is_empty() {return String::new();}@@ -43,7 +48,21 @@ pub fn format_markdown(pages: &[ParsedPage]) -> String {continue;}- let blocks = classify_page_with_filters(page, &heading_map, &header_footer);+ // Filter outline entries to this page so the classifier's y/title+ // match is a O(entries_on_page) scan per line, not O(whole doc).+ let target_index = (page.page_number as i32).saturating_sub(1);+ let page_outline: Vec<OutlineTarget> = outline+ .iter()+ .filter(|e| e.page_index == target_index)+ .cloned()+ .collect();+ let blocks = classify_page_with_filters(+ page,+ &heading_map,+ &header_footer,+ &page_outline,+ image_mode,+ );out.push_str(&render_blocks(&blocks));}out@@ -88,12 +107,14 @@ mod tests {regions: crate::types::Region::default(),graphics: vec![],figures: vec![],+ struct_nodes: vec![],+ image_refs: vec![],}}#[test]fn test_empty() {- assert_eq!(format_markdown(&[]), "");+ assert_eq!(format_markdown(&[], &[], ImageMode::Placeholder), "");}#[test]@@ -108,8 +129,10 @@ mod tests {regions: crate::types::Region::default(),graphics: vec![],figures: vec![],+ struct_nodes: vec![],+ image_refs: vec![],};- let out = format_markdown(&[p]);+ let out = format_markdown(&[p], &[], ImageMode::Placeholder);assert!(out.contains("```text"));assert!(out.contains("hello"));assert!(out.contains("<!-- page 1 -->"));@@ -140,7 +163,7 @@ mod tests {),],);- let out = format_markdown(&[p]);+ let out = format_markdown(&[p], &[], ImageMode::Placeholder);assert!(out.contains("# My Title For This Test Document"));assert!(out.contains("First sentence of body prose here."));}@@ -149,7 +172,7 @@ mod tests {fn test_multi_page_separator() {let a = page_with(1, vec![line("A page.", 50.0, 80.0, 10.0, 10.0)]);let b = page_with(2, vec![line("B page.", 50.0, 80.0, 10.0, 10.0)]);- let out = format_markdown(&[a, b]);+ let out = format_markdown(&[a, b], &[], ImageMode::Placeholder);assert!(out.contains("-----"));assert!(out.find("A page.").unwrap() < out.find("B page.").unwrap());}diff --git a/crates/liteparse/src/output/text.rs b/crates/liteparse/src/output/text.rsindex 82bf1a9..196ce7b 100644--- a/crates/liteparse/src/output/text.rs+++ b/crates/liteparse/src/output/text.rs@@ -24,6 +24,8 @@ mod tests {regions: crate::types::Region::default(),graphics: vec![],figures: vec![],+ struct_nodes: vec![],+ image_refs: vec![],}}diff --git a/crates/liteparse/src/parser.rs b/crates/liteparse/src/parser.rsindex 4fa3512..6f53ceb 100644--- a/crates/liteparse/src/parser.rs+++ b/crates/liteparse/src/parser.rs@@ -13,7 +13,7 @@ use crate::output::markdown;use crate::projection;#[cfg(not(target_arch = "wasm32"))]use crate::render;-use crate::types::{ParsedPage, PdfInput};+use crate::types::{ExtractedImage, OutlineTarget, ParsedPage, PdfInput};/// Result of parsing a document.pub struct ParseResult {@@ -21,6 +21,14 @@ pub struct ParseResult {pub pages: Vec<ParsedPage>,/// Full document text, concatenated from all pages.pub text: String,+ /// Document outline (bookmarks) when present. Used by the markdown+ /// emitter as a high-priority heading source on untagged PDFs.+ pub outline: Vec<OutlineTarget>,+ /// Raster images extracted from the document. Empty unless the parser+ /// was configured with `ImageMode::Embed`. Each entry carries the same+ /// `id` the markdown emitter referenced in ``, so the+ /// caller can match them up without parsing markdown.+ pub images: Vec<ExtractedImage>,}/// Result of rendering a single page screenshot.@@ -99,13 +107,21 @@ impl LiteParse {.map_err(|e| format!("invalid --target-pages: {}", e))?;// Extract text (and pre-render OCR pages in one PDF load when OCR is on).+ // The Document handle isn't Send, so we open it, pull every bit of+ // document-level state we need, and drop it before any `.await`.let password = self.config.password.as_deref();- let (mut pages, ocr_rendered) = if self.config.ocr_enabled {+ let render_images = matches!(+ self.config.image_mode,+ crate::config::ImageMode::Embed+ );+ let (pages, ocr_rendered, outline, images) = {let document = extract::load_document_from_input(&validated_input, password)?;- let pages = extract::extract_pages_from_document(+ let outline = extract::extract_outline(&document);+ let (pages, images) = extract::extract_pages_and_images(&document,target_pages.as_deref(),self.config.max_pages,+ render_images,)?;let t_extract = web_time::Instant::now();log(&format!(@@ -113,30 +129,23 @@ impl LiteParse {t_extract.duration_since(t0).as_secs_f64() * 1000.0,pages.len()));- let rendered = ocr_merge::render_pages_for_ocr(&document, &pages, self.config.dpi)?;- log(&format!(- "[liteparse] ocr render: {:.1}ms ({} pages)",- web_time::Instant::now()- .duration_since(t_extract)- .as_secs_f64()- * 1000.0,- rendered.len()- ));- (pages, rendered)- } else {- let pages = extract::extract_pages_from_input(- &validated_input,- target_pages.as_deref(),- self.config.max_pages,- password,- )?;- log(&format!(- "[liteparse] extract: {:.1}ms ({} pages)",- web_time::Instant::now().duration_since(t0).as_secs_f64() * 1000.0,- pages.len()- ));- (pages, Vec::new())+ let rendered = if self.config.ocr_enabled {+ let r = ocr_merge::render_pages_for_ocr(&document, &pages, self.config.dpi)?;+ log(&format!(+ "[liteparse] ocr render: {:.1}ms ({} pages)",+ web_time::Instant::now()+ .duration_since(t_extract)+ .as_secs_f64()+ * 1000.0,+ r.len()+ ));+ r+ } else {+ Vec::new()+ };+ (pages, rendered, outline, images)};+ let mut pages = pages;let t1 = web_time::Instant::now();// OCR pass@@ -195,7 +204,7 @@ impl LiteParse {));let full_text = if self.config.output_format == crate::config::OutputFormat::Markdown {- let md = markdown::format_markdown(&parsed_pages);+ let md = markdown::format_markdown(&parsed_pages, &outline, self.config.image_mode);let t3 = web_time::Instant::now();log(&format!("[liteparse] markdown: {:.1}ms",@@ -216,6 +225,8 @@ impl LiteParse {Ok(ParseResult {pages: parsed_pages,text: full_text,+ outline,+ images,})}diff --git a/crates/liteparse/src/projection.rs b/crates/liteparse/src/projection.rsindex 3eeffa8..b592308 100644--- a/crates/liteparse/src/projection.rs+++ b/crates/liteparse/src/projection.rs@@ -2680,6 +2680,8 @@ pub fn project_pages_to_grid(pages: Vec<Page>) -> Vec<ParsedPage> {regions,graphics: page.graphics,figures,+ struct_nodes: page.struct_nodes,+ image_refs: page.image_refs,}}).collect()@@ -2803,6 +2805,30 @@ const XY_MIN_LINES_PER_H_SIDE: usize = 2;/// before columns — matches natural reading order.const XY_V_PREFERENCE_MARGIN: f32 = 1.1;+/// Bucket width used by the column-start histogram fallback. 5pt is finer+/// than typical inter-column gutters (8–20pt) and coarser than baseline+/// noise / kerning drift, so column edges land in distinct buckets.+const XY_COLUMN_BUCKET_PT: f32 = 5.0;+/// Minimum lines stacked at the same left-edge cluster for the cluster to+/// count as a column peak. A real column has many lines at one x (15+);+/// indented paragraph-starts and footnote markers produce a handful (3–6).+/// We set this above the typical "indented paragraph" count so those drop+/// out and only true column-left edges remain.+const XY_COLUMN_MIN_LINES_PER_PEAK: usize = 10;+/// Minimum horizontal distance between two column peaks, as a fraction of+/// `bbox.width`. Sub-paragraph indents and list nesting produce+/// closely-spaced peaks; a real column gutter is much further out.+const XY_COLUMN_MIN_GAP_FRACTION: f32 = 0.25;+/// Minimum fraction of items that must start at one of the two detected+/// column peaks. Real 2-column layouts cluster nearly every line at one of+/// two left edges; tables, lists, and scattered prose don't.+const XY_COLUMN_PEAK_DOMINANCE: f32 = 0.55;+/// Minimum smaller/larger ratio between the two column peaks' line counts.+/// Balanced 2-column layouts have ~equal line counts in each column; a+/// 90/10 split is almost certainly a single column with one off-x bullet+/// stack, not a real two-column structure.+const XY_COLUMN_PEAK_BALANCE_RATIO: f32 = 0.4;+#[derive(Debug)]struct CutCandidate {axis: CutAxis,@@ -3081,6 +3107,154 @@ const XY_BANNER_CLEARANCE_FACTOR: f32 = 1.5;/// Also helps mid-document section headings that span the full content width:/// a wide centered "5 Conclusion" line above a 2-column body would otherwise/// get pulled into one column.+/// Fallback column-gutter detector for cases where the density-based V-cut+/// search finds no valley — typically because a column-spanning element+/// (wide heading, figure, table) fills the gutter at one y-band and inflates+/// the projection at the gutter's x. Detects 2+ column structure by clustering+/// the left edges of "narrow" lines (lines that don't span the full region+/// width). When two clusters with ≥`XY_COLUMN_MIN_LINES_PER_PEAK` lines each+/// are separated by ≥`XY_COLUMN_MIN_GAP_FRACTION × bbox.width`, returns a+/// V-cut at the midpoint between them with a high score so it beats any+/// density-based cut except banner.+fn xy_find_column_cut(+ items: &[ProjectedTextItem],+ idxs: &[usize],+ bbox: &Rect,+ _median_h: f32,+) -> Option<CutCandidate> {+ if bbox.width <= 1.0 || idxs.len() < 8 {+ return None;+ }++ // Histogram of individual item left edges. We deliberately avoid+ // first-line-then-leftmost banding because items in the same y-band can+ // belong to *different* columns (left-column and right-column items at+ // the same baseline), and folding them into "one line's x_min" hides+ // the right column entirely. Instead, every item contributes its own x+ // to the histogram; a real column edge shows up as a tall spike (many+ // line-starts at one x) while mid-line continuation fragments spread+ // their contribution across many buckets and don't form spikes.+ let mut min_x = f32::INFINITY;+ let mut max_x = f32::NEG_INFINITY;+ for &i in idxs {+ let it = &items[i].item;+ let x = it.x;+ if x.is_finite() {+ min_x = min_x.min(x);+ max_x = max_x.max(x + it.width.max(0.0));+ }+ }+ if !min_x.is_finite() || max_x <= min_x + 1.0 {+ return None;+ }+ let total_w = max_x - min_x;+ let bucket_pt = XY_COLUMN_BUCKET_PT;+ let n_buckets = ((total_w / bucket_pt).ceil() as usize).max(1);+ let mut hist = vec![0usize; n_buckets];+ for &i in idxs {+ let it = &items[i].item;+ let b_f = ((it.x - min_x) / bucket_pt).floor();+ let b = b_f.max(0.0) as usize;+ if b < n_buckets {+ hist[b] += 1;+ }+ }++ // Cluster adjacent occupied buckets into peaks. A peak is a contiguous+ // run of buckets with ≥1 count, merged through gaps of ≤2 zero buckets+ // (~10pt) to absorb slight intra-column drift.+ let mut peaks: Vec<(f32, usize)> = Vec::new(); // (x_center, item_count)+ let mut i = 0;+ while i < n_buckets {+ if hist[i] == 0 {+ i += 1;+ continue;+ }+ let s = i;+ let mut total = 0usize;+ let mut last_nonzero = i;+ let mut j = i;+ while j < n_buckets {+ if hist[j] > 0 {+ total += hist[j];+ last_nonzero = j;+ j += 1;+ } else if j - last_nonzero <= 2 {+ j += 1;+ } else {+ break;+ }+ }+ let x_center = min_x + bucket_pt * (s as f32 + last_nonzero as f32 + 1.0) * 0.5;+ peaks.push((x_center, total));+ i = j;+ }++ // Keep only "strong" peaks. A real column edge is sat-stacked by many+ // lines at the same left x; spurious peaks (a stray indented quote, a+ // numbered marker, etc.) carry only a few lines.+ peaks.retain(|(_, c)| *c >= XY_COLUMN_MIN_LINES_PER_PEAK);+ if peaks.len() < 2 {+ return None;+ }+ peaks.sort_by(|a, b| a.0.total_cmp(&b.0));++ // Tabular layouts produce 3+ strong peaks (one per cell column). Real+ // page columns have exactly 2 dominant peaks (left edge of col 1, left+ // edge of col 2) — even multi-paragraph indents are sub-peaks weak+ // enough to drop out under the min_per_peak filter. Bail when we see+ // more than 2 strong peaks so we don't slice tables.+ if peaks.len() > 2 {+ return None;+ }+ // Require the two peaks to together dominate the histogram — a real+ // 2-column layout has the great majority of items starting at one of+ // the two column-left edges. Tables, lists, and prose with scattered+ // indents distribute their item-starts across many x's and won't pass.+ let peak_sum: usize = peaks.iter().map(|(_, c)| *c).sum();+ let total_items = idxs.len();+ if (peak_sum as f32) / (total_items as f32) < XY_COLUMN_PEAK_DOMINANCE {+ return None;+ }+ // Require the two peaks to be roughly balanced in size. A real 2-column+ // layout has comparable line counts on each side; a long-single-column+ // page that happens to have one bullet stack at a different x produces+ // a tall + tiny peak pair that should not trigger a column split.+ let (a, b) = (peaks[0].1, peaks[1].1);+ let (smaller, larger) = (a.min(b) as f32, a.max(b) as f32);+ if smaller / larger < XY_COLUMN_PEAK_BALANCE_RATIO {+ return None;+ }++ // Find the widest gap between adjacent strong peaks. Must clear+ // `XY_COLUMN_MIN_GAP_FRACTION × total_w` — list-item indents and+ // block-quote indents produce closely spaced peaks; a real column+ // gutter is much further apart.+ let min_gap = total_w * XY_COLUMN_MIN_GAP_FRACTION;+ let mut best: Option<(f32, usize)> = None;+ for w_idx in 0..peaks.len() - 1 {+ let gap = peaks[w_idx + 1].0 - peaks[w_idx].0;+ if gap >= min_gap && best.is_none_or(|(g, _)| gap > g) {+ best = Some((gap, w_idx));+ }+ }+ let (_, li) = best?;+ let cut_x = (peaks[li].0 + peaks[li + 1].0) * 0.5;+ // Validate cut lands inside the bbox; if bbox-clamping moved the bbox+ // off the items (some PDFs have content with negative x), fall back to+ // an item-relative midpoint check.+ if cut_x <= min_x + 1.0 || cut_x >= max_x - 1.0 {+ return None;+ }+ // High finite score so this beats every density cut but stays below+ // banner's infinity.+ Some(CutCandidate {+ axis: CutAxis::Vertical,+ position: cut_x,+ score: 1.0e9,+ })+}+fn xy_find_banner_cut(items: &[ProjectedTextItem],idxs: &[usize],@@ -3207,18 +3381,29 @@ fn xy_cut_rec(let banner = xy_find_banner_cut(items, &idxs, &bbox, median_h);let h = xy_find_best_cut(items, &idxs, &bbox, CutAxis::Horizontal, median_h, figures);let v = xy_find_best_cut(items, &idxs, &bbox, CutAxis::Vertical, median_h, figures);- let cut = match (banner, h, v) {- (Some(bc), _, _) => Some(bc),- (None, Some(hc), Some(vc)) => {+ // Fallback column-start histogram. Only consulted when BOTH density+ // cuts returned None — those are the cases (wide-heading filled gutter,+ // figure-straddle layouts) where the column structure is real but the+ // density projection can't see it. If either density cut found a+ // valley, trust it instead of firing the histogram fallback.+ let column = if v.is_none() && h.is_none() {+ xy_find_column_cut(items, &idxs, &bbox, median_h)+ } else {+ None+ };+ let cut = match (banner, column, h, v) {+ (Some(bc), _, _, _) => Some(bc),+ (None, Some(cc), _, _) => Some(cc),+ (None, None, Some(hc), Some(vc)) => {if vc.score > hc.score * XY_V_PREFERENCE_MARGIN {Some(vc)} else {Some(hc)}}- (None, Some(hc), None) => Some(hc),- (None, None, Some(vc)) => Some(vc),- (None, None, None) => None,+ (None, None, Some(hc), None) => Some(hc),+ (None, None, None, Some(vc)) => Some(vc),+ (None, None, None, None) => None,};let Some(cut) = cut else {return Region {@@ -3626,6 +3811,8 @@ mod tests {page_height: 792.0,graphics: Vec::new(),text_items: Vec::new(),+ struct_nodes: Vec::new(),+ image_refs: Vec::new(),};let projection_boxes = vec![projected_item("", 10.0, 0.0, 10.0),@@ -3850,6 +4037,8 @@ mod tests {page_height: 792.0,text_items: Vec::new(),graphics: Vec::new(),+ struct_nodes: Vec::new(),+ image_refs: Vec::new(),}];let parsed = project_pages_to_grid(pages);diff --git a/crates/liteparse/src/render.rs b/crates/liteparse/src/render.rsindex 53eb140..3ac068e 100644--- a/crates/liteparse/src/render.rs+++ b/crates/liteparse/src/render.rs@@ -61,7 +61,7 @@ fn render_document_pages(Ok(results)}-fn encode_png(rgba: &[u8], width: u32, height: u32) -> Result<Vec<u8>, LiteParseError> {+pub(crate) fn encode_png(rgba: &[u8], width: u32, height: u32) -> Result<Vec<u8>, LiteParseError> {let mut png_buf = Vec::new();let encoder = image::codecs::png::PngEncoder::new(&mut png_buf);encoder.write_image(rgba, width, height, image::ColorType::Rgba8.into())?;diff --git a/crates/liteparse/src/types.rs b/crates/liteparse/src/types.rsindex 977b731..b48c8c0 100644--- a/crates/liteparse/src/types.rs+++ b/crates/liteparse/src/types.rs@@ -66,6 +66,44 @@ pub struct Page {/// Not emitted in JSON/text outputs — consumed by the markdown layout pass.#[serde(skip)]pub graphics: Vec<GraphicPrimitive>,+ /// Structure-tree nodes for this page when the PDF is tagged. Each node+ /// carries its role, marked-content ids, and the union bbox of its tagged+ /// content. Empty for untagged PDFs.+ #[serde(skip)]+ pub struct_nodes: Vec<StructNode>,+ /// Raster image objects detected on the page. Empty when the page has no+ /// images. Threaded through to `ParsedPage.image_refs`.+ #[serde(skip)]+ pub image_refs: Vec<ImageRef>,+}++/// One entry in the document outline (bookmarks). Coordinates are in PDF+/// user space (origin bottom-left) — convert to viewport with+/// `page_height - y` once you know the page.+#[doc(hidden)]+#[derive(Debug, Clone)]+pub struct OutlineTarget {+ /// Hierarchy depth, 1-based.+ pub level: u8,+ pub title: String,+ /// Zero-based page index of the destination.+ pub page_index: i32,+ /// Y in PDF user space, top of the target location. `None` when the+ /// destination doesn't specify a Y.+ pub y_pdf: Option<f32>,+}++/// One node from the structure tree of a page. Pre-flattened in pre-order+/// (parent before children).+#[doc(hidden)]+#[derive(Debug, Clone)]+pub struct StructNode {+ pub role: String,+ pub mcids: Vec<i32>,+ /// Union bbox of page objects tagged with `mcids`, in viewport coords.+ /// `None` when none of the mcids resolved to a bbox.+ pub bbox: Option<Rect>,+ pub alt_text: Option<String>,}/// Represents a fully parsed page with projected text layout.@@ -94,6 +132,45 @@ pub struct ParsedPage {/// obstacles, and reused downstream for figure classification.#[serde(skip)]pub figures: Vec<Rect>,+ /// Structure-tree nodes for this page (tagged PDFs only). Pre-flattened in+ /// pre-order. Consumed by the markdown classifier for highest-priority+ /// heading / figure / table detection.+ #[serde(skip)]+ pub struct_nodes: Vec<StructNode>,+ /// Raster image objects on the page. Bbox in viewport coords. Populated+ /// during extraction; consumed by the markdown emitter to interleave+ /// `Block::Figure` references at the right y position. Empty when the+ /// page has no embedded images. Not part of JSON/text output.+ #[serde(skip)]+ pub image_refs: Vec<ImageRef>,+}++/// One embedded raster image on a page. `id` is a stable, page-scoped slug+/// used as the markdown link target (e.g. `image_p1_0.png`). `obj_index` is+/// the image's position among image page-objects, so a later embed pass can+/// re-open the document and pull pixel bytes with `render_image_object`.+#[doc(hidden)]+#[derive(Debug, Clone)]+pub struct ImageRef {+ pub id: String,+ pub bbox: Rect,+ pub obj_index: usize,+}++/// A raster image extracted from a page along with its pixel bytes. Surfaced+/// on `ParseResult.images` only when `ImageMode::Embed` is configured —+/// otherwise the extraction step skips the render and only `ImageRef`s are+/// produced. `format` is currently always `"png"` (encoded from the+/// FPDFImageObj_GetRenderedBitmap output via the same path used for page+/// screenshots).+#[derive(Debug, Clone, Serialize)]+pub struct ExtractedImage {+ pub id: String,+ pub page: u32,+ pub bbox: Rect,+ pub format: String,+ #[serde(skip)]+ pub bytes: Vec<u8>,}#[doc(hidden)]@@ -290,6 +367,8 @@ mod tests {page_height: 200.0,text_items: vec![sample_item()],graphics: vec![],+ struct_nodes: vec![],+ image_refs: vec![],};let s = serde_json::to_string(&p).unwrap();assert!(s.contains("\"page_number\":1"));diff --git a/crates/pdfium-sys/src/dynamic.rs b/crates/pdfium-sys/src/dynamic.rsindex 660545d..e4a033a 100644--- a/crates/pdfium-sys/src/dynamic.rs+++ b/crates/pdfium-sys/src/dynamic.rs@@ -230,6 +230,66 @@ pub struct PdfiumBindings {pub FPDFFont_GetGlyphWidth: unsafe extern "C" fn(FPDF_FONT, u32, f32, *mut f32) -> FPDF_BOOL,pub FPDFFont_GetGlyphWidthFromCharCode:unsafe extern "C" fn(FPDF_FONT, u32, f32, *mut f32) -> FPDF_BOOL,++ // -- Outline (bookmarks) --+ pub FPDFBookmark_GetFirstChild:+ unsafe extern "C" fn(FPDF_DOCUMENT, FPDF_BOOKMARK) -> FPDF_BOOKMARK,+ pub FPDFBookmark_GetNextSibling:+ unsafe extern "C" fn(FPDF_DOCUMENT, FPDF_BOOKMARK) -> FPDF_BOOKMARK,+ pub FPDFBookmark_GetTitle: unsafe extern "C" fn(+ FPDF_BOOKMARK,+ *mut std::os::raw::c_void,+ std::os::raw::c_ulong,+ ) -> std::os::raw::c_ulong,+ pub FPDFBookmark_GetDest: unsafe extern "C" fn(FPDF_DOCUMENT, FPDF_BOOKMARK) -> FPDF_DEST,+ pub FPDFBookmark_GetAction: unsafe extern "C" fn(FPDF_BOOKMARK) -> FPDF_ACTION,+ pub FPDFAction_GetDest: unsafe extern "C" fn(FPDF_DOCUMENT, FPDF_ACTION) -> FPDF_DEST,+ pub FPDFDest_GetDestPageIndex:+ unsafe extern "C" fn(FPDF_DOCUMENT, FPDF_DEST) -> std::os::raw::c_int,+ pub FPDFDest_GetLocationInPage: unsafe extern "C" fn(+ FPDF_DEST,+ *mut FPDF_BOOL,+ *mut FPDF_BOOL,+ *mut FPDF_BOOL,+ *mut FS_FLOAT,+ *mut FS_FLOAT,+ *mut FS_FLOAT,+ ) -> FPDF_BOOL,++ // -- Structure tree --+ pub FPDF_StructTree_GetForPage: unsafe extern "C" fn(FPDF_PAGE) -> FPDF_STRUCTTREE,+ pub FPDF_StructTree_Close: unsafe extern "C" fn(FPDF_STRUCTTREE),+ pub FPDF_StructTree_CountChildren:+ unsafe extern "C" fn(FPDF_STRUCTTREE) -> std::os::raw::c_int,+ pub FPDF_StructTree_GetChildAtIndex:+ unsafe extern "C" fn(FPDF_STRUCTTREE, std::os::raw::c_int) -> FPDF_STRUCTELEMENT,+ pub FPDF_StructElement_GetType: unsafe extern "C" fn(+ FPDF_STRUCTELEMENT,+ *mut std::os::raw::c_void,+ std::os::raw::c_ulong,+ ) -> std::os::raw::c_ulong,+ pub FPDF_StructElement_GetAltText: unsafe extern "C" fn(+ FPDF_STRUCTELEMENT,+ *mut std::os::raw::c_void,+ std::os::raw::c_ulong,+ ) -> std::os::raw::c_ulong,+ pub FPDF_StructElement_GetTitle: unsafe extern "C" fn(+ FPDF_STRUCTELEMENT,+ *mut std::os::raw::c_void,+ std::os::raw::c_ulong,+ ) -> std::os::raw::c_ulong,+ pub FPDF_StructElement_CountChildren:+ unsafe extern "C" fn(FPDF_STRUCTELEMENT) -> std::os::raw::c_int,+ pub FPDF_StructElement_GetChildAtIndex:+ unsafe extern "C" fn(FPDF_STRUCTELEMENT, std::os::raw::c_int) -> FPDF_STRUCTELEMENT,+ pub FPDF_StructElement_GetChildMarkedContentID:+ unsafe extern "C" fn(FPDF_STRUCTELEMENT, std::os::raw::c_int) -> std::os::raw::c_int,+ pub FPDF_StructElement_GetMarkedContentID:+ unsafe extern "C" fn(FPDF_STRUCTELEMENT) -> std::os::raw::c_int,+ pub FPDF_StructElement_GetMarkedContentIdCount:+ unsafe extern "C" fn(FPDF_STRUCTELEMENT) -> std::os::raw::c_int,+ pub FPDF_StructElement_GetMarkedContentIdAtIndex:+ unsafe extern "C" fn(FPDF_STRUCTELEMENT, std::os::raw::c_int) -> std::os::raw::c_int,}// SAFETY: PdfiumBindings contains only function pointers and a Library handle.@@ -308,6 +368,42 @@ impl PdfiumBindings {FPDFFont_GetDescent: load_fn!(lib, "FPDFFont_GetDescent"),FPDFFont_GetGlyphWidth: load_fn!(lib, "FPDFFont_GetGlyphWidth"),FPDFFont_GetGlyphWidthFromCharCode: load_fn!(lib, "FPDFFont_GetGlyphWidthFromCharCode"),++ FPDFBookmark_GetFirstChild: load_fn!(lib, "FPDFBookmark_GetFirstChild"),+ FPDFBookmark_GetNextSibling: load_fn!(lib, "FPDFBookmark_GetNextSibling"),+ FPDFBookmark_GetTitle: load_fn!(lib, "FPDFBookmark_GetTitle"),+ FPDFBookmark_GetDest: load_fn!(lib, "FPDFBookmark_GetDest"),+ FPDFBookmark_GetAction: load_fn!(lib, "FPDFBookmark_GetAction"),+ FPDFAction_GetDest: load_fn!(lib, "FPDFAction_GetDest"),+ FPDFDest_GetDestPageIndex: load_fn!(lib, "FPDFDest_GetDestPageIndex"),+ FPDFDest_GetLocationInPage: load_fn!(lib, "FPDFDest_GetLocationInPage"),++ FPDF_StructTree_GetForPage: load_fn!(lib, "FPDF_StructTree_GetForPage"),+ FPDF_StructTree_Close: load_fn!(lib, "FPDF_StructTree_Close"),+ FPDF_StructTree_CountChildren: load_fn!(lib, "FPDF_StructTree_CountChildren"),+ FPDF_StructTree_GetChildAtIndex: load_fn!(lib, "FPDF_StructTree_GetChildAtIndex"),+ FPDF_StructElement_GetType: load_fn!(lib, "FPDF_StructElement_GetType"),+ FPDF_StructElement_GetAltText: load_fn!(lib, "FPDF_StructElement_GetAltText"),+ FPDF_StructElement_GetTitle: load_fn!(lib, "FPDF_StructElement_GetTitle"),+ FPDF_StructElement_CountChildren: load_fn!(lib, "FPDF_StructElement_CountChildren"),+ FPDF_StructElement_GetChildAtIndex: load_fn!(lib, "FPDF_StructElement_GetChildAtIndex"),+ FPDF_StructElement_GetChildMarkedContentID: load_fn!(+ lib,+ "FPDF_StructElement_GetChildMarkedContentID"+ ),+ FPDF_StructElement_GetMarkedContentID: load_fn!(+ lib,+ "FPDF_StructElement_GetMarkedContentID"+ ),+ FPDF_StructElement_GetMarkedContentIdCount: load_fn!(+ lib,+ "FPDF_StructElement_GetMarkedContentIdCount"+ ),+ FPDF_StructElement_GetMarkedContentIdAtIndex: load_fn!(+ lib,+ "FPDF_StructElement_GetMarkedContentIdAtIndex"+ ),+_lib: lib,})}diff --git a/crates/pdfium/src/document.rs b/crates/pdfium/src/document.rsindex 221809f..9ef97b8 100644--- a/crates/pdfium/src/document.rs+++ b/crates/pdfium/src/document.rs@@ -6,6 +6,23 @@ pub struct Document {pub(crate) handle: pdfium_sys::FPDF_DOCUMENT,}+/// One entry in the document's outline (bookmarks tree).+#[derive(Debug, Clone)]+pub struct OutlineEntry {+ /// Hierarchy depth, 1-based (top-level entries are level 1).+ pub level: u8,+ /// Bookmark title.+ pub title: String,+ /// Zero-based page index of the destination, or `None` if the destination+ /// isn't a page in this document (external link, missing dest, etc).+ pub page_index: Option<i32>,+ /// Y coordinate of the destination on the page in PDF user space (origin+ /// bottom-left), or `None` if the destination doesn't specify one. To+ /// compare against viewport-space line bboxes (origin top-left) use+ /// `page_height - y`.+ pub y: Option<f32>,+}+impl Document {pub fn page_count(&self) -> i32 {unsafe { ffi!(FPDF_GetPageCount(self.handle)) }@@ -22,6 +39,103 @@ impl Document {_doc: std::marker::PhantomData,})}++ /// Walk the document outline (bookmarks). Returns entries in pre-order+ /// (depth-first), so parents precede their children. Empty when the+ /// document has no outline.+ pub fn outline(&self) -> Vec<OutlineEntry> {+ let mut out = Vec::new();+ let root = unsafe { ffi!(FPDFBookmark_GetFirstChild(self.handle, std::ptr::null_mut())) };+ if !root.is_null() {+ self.walk_bookmark(root, 1, &mut out);+ }+ out+ }++ fn walk_bookmark(+ &self,+ bookmark: pdfium_sys::FPDF_BOOKMARK,+ level: u8,+ out: &mut Vec<OutlineEntry>,+ ) {+ let mut cur = bookmark;+ while !cur.is_null() {+ let title = read_bookmark_title(cur);+ let (page_index, y) = resolve_dest(self.handle, cur);+ out.push(OutlineEntry {+ level,+ title,+ page_index,+ y,+ });++ let child = unsafe { ffi!(FPDFBookmark_GetFirstChild(self.handle, cur)) };+ if !child.is_null() {+ self.walk_bookmark(child, level.saturating_add(1), out);+ }++ cur = unsafe { ffi!(FPDFBookmark_GetNextSibling(self.handle, cur)) };+ }+ }+}++fn read_bookmark_title(bookmark: pdfium_sys::FPDF_BOOKMARK) -> String {+ let needed =+ unsafe { ffi!(FPDFBookmark_GetTitle(bookmark, std::ptr::null_mut(), 0)) } as usize;+ if needed < 2 {+ return String::new();+ }+ // `needed` is byte length including a trailing UTF-16 NUL terminator.+ let mut buf: Vec<u16> = vec![0; needed / 2];+ let written = unsafe {+ ffi!(FPDFBookmark_GetTitle(+ bookmark,+ buf.as_mut_ptr() as *mut std::os::raw::c_void,+ needed as std::os::raw::c_ulong,+ ))+ } as usize;+ if written < 2 {+ return String::new();+ }+ let chars = written / 2;+ let end = if buf.get(chars - 1) == Some(&0) {+ chars - 1+ } else {+ chars+ };+ String::from_utf16_lossy(&buf[..end])+}++fn resolve_dest(+ doc: pdfium_sys::FPDF_DOCUMENT,+ bookmark: pdfium_sys::FPDF_BOOKMARK,+) -> (Option<i32>, Option<f32>) {+ let mut dest = unsafe { ffi!(FPDFBookmark_GetDest(doc, bookmark)) };+ if dest.is_null() {+ let action = unsafe { ffi!(FPDFBookmark_GetAction(bookmark)) };+ if !action.is_null() {+ dest = unsafe { ffi!(FPDFAction_GetDest(doc, action)) };+ }+ }+ if dest.is_null() {+ return (None, None);+ }+ let page_index = unsafe { ffi!(FPDFDest_GetDestPageIndex(doc, dest)) };+ let page_index = if page_index >= 0 { Some(page_index) } else { None };++ let mut has_x: pdfium_sys::FPDF_BOOL = 0;+ let mut has_y: pdfium_sys::FPDF_BOOL = 0;+ let mut has_z: pdfium_sys::FPDF_BOOL = 0;+ let mut x: f32 = 0.0;+ let mut y: f32 = 0.0;+ let mut z: f32 = 0.0;+ let ok = unsafe {+ ffi!(FPDFDest_GetLocationInPage(+ dest, &mut has_x, &mut has_y, &mut has_z, &mut x, &mut y, &mut z+ ))+ };+ let y_out = if ok != 0 && has_y != 0 { Some(y) } else { None };+ (page_index, y_out)}impl Drop for Document {diff --git a/crates/pdfium/src/lib.rs b/crates/pdfium/src/lib.rsindex 389e9f1..59ec49e 100644--- a/crates/pdfium/src/lib.rs+++ b/crates/pdfium/src/lib.rs@@ -4,15 +4,17 @@ mod error;mod font;mod library;mod page;+mod struct_tree;mod text_page;mod types;pub use bitmap::Bitmap;-pub use document::Document;+pub use document::{Document, OutlineEntry};pub use error::PdfiumError;pub use font::{Font, FontType};pub use library::Library;pub use page::{ImageBounds, Page, PathObject, PathSegment, SegmentKind, ViewportTransform};+pub use struct_tree::StructNode;pub use text_page::{TextChar, TextCharIter, TextPage};pub use types::*;diff --git a/packages/python/liteparse/__init__.py b/packages/python/liteparse/__init__.pyindex f96f5d0..5b9a6ef 100644--- a/packages/python/liteparse/__init__.py+++ b/packages/python/liteparse/__init__.py@@ -1,5 +1,6 @@from .parser import LiteParse, search_itemsfrom .types import (+ ExtractedImage,LiteParseConfig,ParseResult,ParsedPage,@@ -16,6 +17,7 @@ __all__ = ["ParsedPage","TextItem","ScreenshotResult",+ "ExtractedImage","ParseError","search_items",]diff --git a/packages/python/liteparse/parser.py b/packages/python/liteparse/parser.pyindex 22dd98d..9fe4317 100644--- a/packages/python/liteparse/parser.py+++ b/packages/python/liteparse/parser.py@@ -7,6 +7,7 @@ from liteparse._liteparse import LiteParse as _NativeLiteParsefrom liteparse._liteparse import search_items as _native_search_itemsfrom .types import (+ ExtractedImage,LiteParseConfig,ParsedPage,ParseError,@@ -42,9 +43,19 @@ def _convert_native_result(native_result: Any) -> ParseResult:text_items=text_items,))+ images = [+ ExtractedImage(+ id=img.id,+ page=img.page,+ format=img.format,+ bytes=img.bytes,+ )+ for img in getattr(native_result, "images", [])+ ]return ParseResult(pages=pages,text=native_result.text,+ images=images,)@@ -76,6 +87,7 @@ class LiteParse:password: Optional[str] = None,quiet: Optional[bool] = None,num_workers: Optional[int] = None,+ image_mode: Optional[str] = None,):"""Initialize LiteParse parser.@@ -119,6 +131,8 @@ class LiteParse:kwargs["quiet"] = quietif num_workers is not None:kwargs["num_workers"] = num_workers+ if image_mode is not None:+ kwargs["image_mode"] = image_modeself._native = _NativeLiteParse(**kwargs)diff --git a/packages/python/liteparse/types.py b/packages/python/liteparse/types.pyindex 40f4fbf..1bbea13 100644--- a/packages/python/liteparse/types.py+++ b/packages/python/liteparse/types.py@@ -29,11 +29,26 @@ class ParsedPage:text_items: List[TextItem] = field(default_factory=list)+@dataclass+class ExtractedImage:+ """An embedded raster image extracted from a page.++ Populated only when the parser was configured with ``image_mode="embed"``.+ The ``id`` matches the reference used in the markdown output+ (e.g. ```` → ``id="p1_0"``).+ """+ id: str+ page: int+ format: str+ bytes: bytes++@dataclassclass ParseResult:"""Result of parsing a document."""pages: List[ParsedPage]text: str+ images: List[ExtractedImage] = field(default_factory=list)@propertydef num_pages(self) -> int:diff --git a/packages/python/uv.lock b/packages/python/uv.lockindex 4b1a41e..b1e10f3 100644--- a/packages/python/uv.lock+++ b/packages/python/uv.lock@@ -128,7 +128,7 @@ wheels = [[[package]]name = "liteparse"-version = "2.0.0"+version = "2.0.3"source = { editable = "." }[package.optional-dependencies]