A fast, helpful, and open-source document parser
wip
Logan Markewichff3d1ae22073
diff --git a/MARKDOWN_PROGRESS.md b/MARKDOWN_PROGRESS.mdindex 105055e..541289b 100644--- a/MARKDOWN_PROGRESS.md+++ b/MARKDOWN_PROGRESS.md@@ -2,6 +2,48 @@Tracks build-order progress against [MARKDOWN_PLAN.md](MARKDOWN_PLAN.md). Update as steps complete.+> **2026-06-13 (pm) trailing-period loss fix (Mode 16, `extract.rs` dot-leader threshold) — olmOCR Overall 38.6 → 39.1% (+0.5pp); order 53.0 → 54.4% (+14), multi_column 63.6 → 65.3% (+15 tests), long_tiny_text 25.8 → 29.2% (+15 tests), present 15.8 → 17.9%; headers_footers −2 / table_tests −5 (CI noise). opendataloader BYTE-FLAT (NID 0.9126 / TEDS 0.6929 / MHS 0.8036), ParseBench text_content FLAT (0.65097 both, forced re-infer A/B) and table FLAT (grits_con 0.5731 / composite 0.4076 / TRM 0.2196). 207 lib tests pass. One-line change, shipped.** Resolves the 2026-06-08 FAILURE_MODES Mode 16 "trailing-period loss (~25 tests, probably localized)" item — the cleanest remaining lever in the `text_absent` class flagged by the 2026-06-13 formatting-triage pivot, and explicitly NOT the compressed-metric-font danger zone.+>+> **Root cause (single misfiring threshold).** The no-pending-space dot-leader splitter at `extract.rs:644` broke a `.` off preceding content whenever `gap > avg_char_width * 0.4`. A loosely-kerned abbreviation/sentence period sits at **~1.0× avg char width** (measured `g/cw ≈ 1.0` on `Sci.`/`Chem.`/`Environ.`/`Phys.`), so it tripped the splitter; the split-off `.` segment then merged forward into the next token and vanished. Tightly-kerned periods (`Mater.` at `g/cw=0.14`) stayed attached, which is why the loss was *positional* (only before column/tab gaps in reference lists) and looked per-word. Genuine no-space dot leaders run `g/cw ≈ 2–23` (verified a real one at 23.2), and spaced leaders take the *other* (`pending_space`) branch entirely — so raising the cutoff `0.4 → 2.0` opens a wide safe valley.+>+> **Why olmOCR-only.** Pure extraction-fidelity fix → only moves the benchmark whose char-exact `order`/anchor predicates a dropped period actually breaks (a 1-char Levenshtein miss fails the whole anchor). opendataloader (NID/TEDS/MHS) and ParseBench (word-match rules normalize trailing punctuation) are structurally insensitive, hence byte-flat/flat — confirmed, not assumed. Qualitatively large for RAG ingestion regardless: reference lists go from `Cravatt, B.F (2004) … Angew Chem Int Ed` to fully-periodized `Cravatt, B.F. (2004). … Angew. Chem. Int. Ed.`.+>+> **A/B method (the load-bearing rigor).** First text_content run reused **stale cached inference** (`parse-bench run` defaults `--force=False`) and reported a false-flat to 15 digits — re-ran *both* binaries with `--force` for a true same-harness delta. For olmOCR (in-process Python binding, not the CLI binary) a same-session base required reverting the one line, rebuilding the binding, and converting a clean `liteparse_m16base` candidate — historical numbers carry documented binary drift and can't be trusted as the baseline. Byte-diff bound the blast radius first: multi_column 57/231, headers_footers 47/266, long_tiny_text 12/62 changed — all sampled diffs were period restorations (plus one re-merged wrongly-fragmented paragraph), one trivial `, and`→`,and` collateral against dozens of restorations. Scratch candidates `liteparse_m16` / `liteparse_m16base` left in `bench_data/`. Memory: [[project_missing_spaces]].+>+> **2026-06-13 semantic-formatting triage + table-cell emphasis A/B — lever empirically KILLED, REVERTED (byte-identical baseline, 207 lib tests pass). No code shipped; the finding is the deliverable.** Set out to lift ParseBench `text_formatting` (the 2026-06-11 audit's Tier-3 #7 fidelity lever: bold 0.53 / italic 0.37 / title_hierarchy 0.37). Measured first instead of editing, and the measurement says **stop**: bold/italic detection has ~no benchmark headroom.+>+> **Fresh baseline confirmed (no drift):** is_bold 0.5273, is_italic 0.3696, is_title 0.4413, title_hierarchy 0.3716, underline/sup/mark 0.0, code_block 0.2, overall avg_rule_pass_rate 0.3736.+>+> **Triage reframe (`dataset_eval_utils/triage_formatting.py` — runs each rule through ParseBench's own `create_test_rule`, buckets each failure by whether the GT text is even in our output).** The inline emphasis pipeline ALREADY exists (`inline.rs::render_line_inline` does per-span `**`/`*`/`` ` ``/links) — the audit's "whole-line-only" framing was stale. The gap is NOT missing capability:+> - is_bold: **76% of failures are `text_absent`** (word not in output at all → extraction/reading-order, ORTHOGONAL), 24% `present_plain` (216 rules, detector-addressable).+> - is_italic: 47% text_absent, 53% `present_plain` (208 rules — most addressable relative to size).+> - is_title: 63% text_absent, 37% present_plain (422 rules).+> - title_hierarchy: 97% of unsatisfied constraints are missing_title/missing_edge (PRESENCE), only 2% level + 0% order → **NOT a leveling problem**; it's heading detection.+>+> **Drill-down (`dataset_eval_utils/drill_formatting_misses.py` — temporarily exposed font_flags/font_weight in JSON, reverted).** Of the introspectable `present_plain` misses: ~25% `flag_set` (item carries the italic/bold flag → detection SHOULD fire, miss is downstream), ~75% `no_signal` (italic/bold absent from font flags AND name — synthetic skew / outline-only weight, **unrecoverable from font metadata**). ~50% were `no_match` (JSON `text_items` empty for those PDFs — tooling gap, not a product fact). The `flag_set` misses cluster in table-heavy financial/SERFF/IPO docs → root cause: **table cells (`escape_table_cell`) and headings (`# text`) render PLAIN, never through the inline pipeline.** (`name_contains_any` in projection.rs is also case-sensitive.)+>+> **Built + A/B'd the one buildable lever: per-cell BOLD emphasis** at the two shared builders `finalize_table_run` + `build_ruled_table` (reusing the already-tracked `TableCell.bold` / ruled `cell_is_bold`; `wrap_emphasis` made `pub(super)` — no struct churn, late-wrapping so cell-splitting logic is untouched). Result:+> - ParseBench is_bold **+0.0000** (per-doc avg over 476 docs — the few flipped cells round to zero; bold/italic GT targets aren't in table cells, and in-table bold detection hits the same no_signal wall).+> - ParseBench table GriTS_con / TRM **0.0000** (safe — `normalize_cell_text` strips `**`).+> - opendataloader **TEDS −0.0167** (0.6929→0.6762), overall −0.0019 — ODL's TEDS does NOT strip markdown from cells, so `**Foo**`≠`Foo`.+> Net loss, zero benefit → reverted. **Do not re-attempt table/heading emphasis for benchmark gain.**+>+> **Conclusion + pivot.** Bold/italic semantic formatting is a dead benchmark lever (gap = ⅔ text_absent + ¾ no_signal; buildable pool tiny and not in tables). The real opportunity is the **`text_absent` (missing text)** class — the documented Tier-1 word-fusion / column-interleave / glyph-soup extraction work, which lifts is_bold/is_title/text_content/order/NID together. Next step (not yet done): a fresh scoped triage of the current text_absent set (cross-ref olmOCR `order`) to find whether any sub-cause is still a clean lever before committing to the risky compressed-metric-font work (the −308/−187 danger zone). Memory: [[project_formatting_triage]]. Scripts: `triage_formatting.py` (SIGALRM 3s guard — ParseBench bold/italic tempered-greedy `.DOTALL` regexes backtrack catastrophically on large non-matching docs; reliable for cause buckets, NOT precise pass-rate deltas — use the official eval for those), `drill_formatting_misses.py`, CSV `formatting_triage.csv`.+>+> **2026-06-13 hyperlink extraction → `[text](url)` (new capability, default ON; ALL benchmarks zero-regression because harnesses disable it) — opendataloader BYTE-IDENTICAL (Overall 0.8719 / NID 0.9125 / TEDS 0.6929 / MHS 0.8036), ParseBench table BYTE-IDENTICAL (grits_con 0.5737 / TRM 0.2196 / composite 0.4079), olmOCR matched (Overall 38.6% / multi_column 63.6% / table_tests 48.7% / long_tiny_text 25.8%); 207 lib tests pass (3 new).** Implements the 2026-06-11 audit's Tier-2 #5 "links entirely absent" — the one genuine missing *capability* (vs a metric gap): no `GetURI`/annotation extraction existed anywhere in `extract.rs` or the pdfium wrapper. Targets RAG ingestion, not a benchmark lever.+>+> **Annotation-based only** (`FPDFLink_Enumerate` → `FPDFLink_GetAction` → `FPDFAction_GetURIPath`), not auto-detected web-links — captures real clickable links whose anchor text differs from the URL (`[click here](https://…)`), which web-link scanning can't. No PDFium fork rebuild needed: all symbols were already in the static `bindings.rs`; only the libloading table in `pdfium-sys/src/dynamic.rs` needed the four fns added (plus `FPDFLink_CountQuadPoints`/`GetQuadPoints`).+>+> **Data path.** New `Page::links(view_box) -> Vec<PdfLink { rect (viewport), uri }>` in `pdfium/src/page.rs` (mirrors `path_objects`; rect mapped via the existing `bounds_to_viewport`). New `TextItem.link: Option<String>` (rides through projection's `..proj.item` for free). `assign_links` in `extract.rs` attaches a URI to every text item whose bbox center falls in the link rect. Rendered in `markdown_layout/inline.rs`: group spans by `(style, link)`, wrap each group `[…](url)` *outside* the emphasis wrap (`[*cite*](url)`), angle-bracket form for URLs with spaces/parens. `line_uniform_style` returns `None` for any link-bearing line so the paragraph/list uniform fast-paths route through the per-span renderer — every new branch is gated on link presence, so link-free output is byte-identical.+>+> **Per-line quad points preferred over the union rect.** A link wrapping across lines has one tight quad per line; the single annotation `/Rect` is their *union* (a tall box that swallows the unlinked words between the lines). When quads exist we emit one `PdfLink` per quad; otherwise fall back to the union rect. For the no-quad-point multi-line case the true anchor is unrecoverable, so `assign_links` **drops** the link when `rect height > 1.8× median covered-item height` rather than wrap a whole sentence (verified on opendataloader `…086`, a `perma.cc/Z89Y-` / `SEVS.` wrapped citation). Single-line links (`…095` `[*(Niederle and Vesterlund 2007)*](…)`, `…185`) render correctly.+>+> **Why default-ON but every harness disables it.** The GT corpora **never use `[](url)` syntax** — 0/200 opendataloader GT files; URLs appear as plain text. Measured: links-ON regresses opendataloader **NID 0.9125 → 0.9102, MHS 0.8036 → 0.8021** (TEDS flat). So real users get richer markdown by default, but the three bench providers opt out for parity: opendataloader + ParseBench pass CLI `--no-links`; the olmOCR in-process Python runner passes `extract_links=False`. Config `extract_links` (default true) is wired through CLI (`--no-links`), the pyo3 binding + `packages/python` wrapper, and napi. The parser only extracts links when `output_format == Markdown` (text/JSON paths skip the per-page `FPDFLink_Enumerate` call entirely).+>+> **ParseBench nuance (flagged, not yet measured).** There is **no `is_link` rule type** in ParseBench (formatting kinds are bold/italic/underline/strikeout/mark/sup/sub/title/latex/code_block/title_hierarchy), so links can't win `text_formatting` points. The tracked `table` group is byte-identical with/without links (tables carry no link annotations). **But `text_content.jsonl` GT contains `[text](url)` syntax in ~7 lines / ~4 docs** (examgenius.in, epa.gov, bonnieplants, artsandculture), scored by `missing_specific_word`/`order`/`unexpected_word`. So blanket `--no-links` on ParseBench is safe for the tracked table metric but is an *unmeasured assumption* for `text_content` — links-ON could match GT better on those ~4 docs while adding unexpected `](url)` tokens on the plain-text majority. Net `text_content` effect unmeasured (only ran `--group table` this session). Memory: [[project_markdown_links]].+>+> Kill-switch: `--no-links` (CLI) / `extract_links=False` (Python) / `extractLinks: false` (Node). Files: `pdfium-sys/src/dynamic.rs`, `pdfium/src/{page,lib}.rs`, `liteparse/src/{types,extract,parser,config,main}.rs`, `liteparse/src/markdown_layout/inline.rs`, `liteparse-python/src/lib.rs`, `liteparse-napi/src/types.rs`, `packages/python/liteparse/parser.py`, + 3 bench provider scripts.+>> **2026-06-12 (pm) per-font empirical space-width recovery (Mode 14 word fusion, no-metric fonts) — olmOCR long_tiny_text 25.3 → 25.8% (+2 tests, clean, zero regressions); multi_column / table_tests / ParseBench BYTE-IDENTICAL; opendataloader flat (NID/TEDS/MHS unchanged); 204 lib tests pass.** Targeted the 2026-06-11 audit's Tier-1 #2 dense/small-font word fusion (22.6% of `long_tiny_text` files), reproduced on `long_tiny_text/20_pg58` (`theynexttakeonthe`, `againstManchester`, `transferbusiness`, `forSpurs` — body rendered as word-soup).>> **Root cause (Layer 1, `extract.rs`).** The missing-space recovery in the no-pending-space branch uses `thresh = 0.7 * space_w` when the font exposes an ASCII-space advance (`ch.font_space_width()`), else a fixed `0.35 * em_vp` fallback. `20_pg58`'s body font exposes **no space metric** (`sw=-1` in `LITEPARSE_DEBUG_GAPS`), so every boundary hit the fallback — which computes ~3pt for an 8.5pt font while the real inter-word gap is ~1pt, so nothing ever fired. Histogramming `loose_gap` by the PDFium `ps` flag showed the gaps are cleanly **bimodal**: intra-word kerning loose ≤ 0.4pt (434 events), genuine spaces 0.8–1.4pt (34 events), near-empty valley between — and the boundaries where PDFium *did* emit a space (`ps=1`) land in that exact 0.8–1.4 band, giving a free per-font calibration signal.diff --git a/crates/liteparse-napi/src/types.rs b/crates/liteparse-napi/src/types.rsindex d8844c7..0c483c4 100644--- a/crates/liteparse-napi/src/types.rs+++ b/crates/liteparse-napi/src/types.rs@@ -39,6 +39,9 @@ pub struct JsLiteParseConfig {/// (default — emits `` references with no bytes), or/// "embed" (also returns each image's PNG bytes on `images`).pub image_mode: Option<String>,+ /// Render hyperlink annotations as `[text](url)` in markdown output+ /// (default true). Set false for plain anchor text.+ pub extract_links: Option<bool>,}impl JsLiteParseConfig {@@ -91,6 +94,9 @@ impl JsLiteParseConfig {_ => ImageMode::Placeholder,};}+ if let Some(v) = self.extract_links {+ cfg.extract_links = v;+ }cfg}@@ -117,6 +123,7 @@ impl JsLiteParseConfig {ImageMode::Placeholder => "placeholder".to_string(),ImageMode::Embed => "embed".to_string(),}),+ extract_links: Some(cfg.extract_links),}}}diff --git a/crates/liteparse-python/src/lib.rs b/crates/liteparse-python/src/lib.rsindex 246b490..a3b4886 100644--- a/crates/liteparse-python/src/lib.rs+++ b/crates/liteparse-python/src/lib.rs@@ -327,6 +327,7 @@ impl LiteParse {quiet = None,num_workers = None,image_mode = None,+ extract_links = None,))]fn new(ocr_language: Option<String>,@@ -342,6 +343,7 @@ impl LiteParse {quiet: Option<bool>,num_workers: Option<usize>,image_mode: Option<String>,+ extract_links: Option<bool>,) -> PyResult<Self> {let mut cfg = LiteParseConfig::default();if let Some(v) = ocr_language {@@ -391,6 +393,9 @@ impl LiteParse {_ => ImageMode::Placeholder,};}+ if let Some(v) = extract_links {+ cfg.extract_links = v;+ }let inner = liteparse::parser::LiteParse::new(cfg.clone());let runtime = tokio::runtime::Runtime::new()diff --git a/crates/liteparse/src/config.rs b/crates/liteparse/src/config.rsindex 21d0a07..f3ff319 100644--- a/crates/liteparse/src/config.rs+++ b/crates/liteparse/src/config.rs@@ -30,6 +30,10 @@ pub struct LiteParseConfig {/// Controls how raster images are surfaced in markdown output. Has no/// effect on JSON / text outputs.pub image_mode: ImageMode,+ /// Extract hyperlink annotations and render them as `[text](url)` in+ /// markdown output. Default on. Disable for benchmark parity with+ /// plain-text ground truth (the GT corpora never use link syntax).+ pub extract_links: bool,}/// Image handling for the markdown emitter.@@ -76,6 +80,7 @@ impl Default for LiteParseConfig {quiet: false,num_workers: default_num_workers(),image_mode: ImageMode::Placeholder,+ extract_links: true,}}}diff --git a/crates/liteparse/src/extract.rs b/crates/liteparse/src/extract.rsindex 2ee37e6..bb467dd 100644--- a/crates/liteparse/src/extract.rs+++ b/crates/liteparse/src/extract.rs@@ -5,7 +5,9 @@ use crate::types::{ExtractedImage, GraphicPrimitive, ImageRef, OutlineTarget, Page as LitePage, PdfInput, Rect,StructNode, TextItem,};-use pdfium::{Document, Font, FontType, Library, Page, PathObject, RectF, SegmentKind, TextPage};+use pdfium::{+ Document, Font, FontType, Library, Page, PathObject, PdfLink, RectF, SegmentKind, TextPage,+};/// Open a PDF from path or bytes with an optional password.///@@ -47,7 +49,7 @@ pub(crate) fn extract_pages_from_document(target_pages: Option<&[u32]>,max_pages: usize,) -> Result<Vec<LitePage>, LiteParseError> {- Ok(extract_pages_and_images(document, target_pages, max_pages, false)?.0)+ Ok(extract_pages_and_images(document, target_pages, max_pages, false, false)?.0)}/// Same as `extract_pages_from_document` but optionally also renders every@@ -60,6 +62,7 @@ pub(crate) fn extract_pages_and_images(target_pages: Option<&[u32]>,max_pages: usize,render_images: bool,+ extract_links: bool,) -> Result<(Vec<LitePage>, Vec<ExtractedImage>), LiteParseError> {let page_count = document.page_count();let mut pages = Vec::new();@@ -86,7 +89,10 @@ pub(crate) fn extract_pages_and_images(right: page.width(),bottom: 0.0,});- let text_items = extract_page_text_items(&page, &text_page, &view_box)?;+ let mut text_items = extract_page_text_items(&page, &text_page, &view_box)?;+ if extract_links {+ assign_links(&mut text_items, &page.links(&view_box));+ }let graphics = extract_page_graphics(&page, &view_box);let struct_nodes = extract_page_struct_nodes(&page, &view_box);let image_refs = extract_page_image_refs(&page, page_number);@@ -109,6 +115,50 @@ pub(crate) fn extract_pages_and_images(Ok((pages, images))}+/// Assign hyperlink URIs to text items whose bbox center falls inside a link+/// annotation's rectangle. Both the item bbox and the link rect are in+/// viewport space. First matching link wins.+///+/// A link rect taller than `MULTILINE_DROP_FACTOR`× the height of the text it+/// covers is a multi-line annotation given to us as a single *union* box (no+/// per-line quad points). Its true anchor — which words on the intervening+/// lines are actually linked — is unrecoverable, so we drop it rather than+/// wrap a whole sentence in a misleading link. Well-formed multi-line links+/// expose quad points and arrive here as one single-line rect per line.+fn assign_links(items: &mut [TextItem], links: &[PdfLink]) {+ if links.is_empty() {+ return;+ }+ const MULTILINE_DROP_FACTOR: f32 = 1.8;+ for link in links {+ let r = &link.rect;+ let covered: Vec<usize> = items+ .iter()+ .enumerate()+ .filter(|(_, it)| {+ let cx = it.x + it.width / 2.0;+ let cy = it.y + it.height / 2.0;+ cx >= r.left && cx <= r.right && cy >= r.top && cy <= r.bottom+ })+ .map(|(i, _)| i)+ .collect();+ if covered.is_empty() {+ continue;+ }+ let mut heights: Vec<f32> = covered.iter().map(|&i| items[i].height).collect();+ heights.sort_by(f32::total_cmp);+ let median_h = heights[heights.len() / 2];+ if median_h > 0.0 && (r.bottom - r.top) > MULTILINE_DROP_FACTOR * median_h {+ continue;+ }+ for &i in &covered {+ if items[i].link.is_none() {+ items[i].link = Some(link.uri.clone());+ }+ }+ }+}+/// Walk the document outline (bookmarks). Returns entries in pre-order./// Empty when the PDF has no outline.pub(crate) fn extract_outline(document: &Document) -> Vec<OutlineTarget> {@@ -590,8 +640,13 @@ fn extract_page_text_items(} else {// Without a pending space: break when a dot follows non-dot content// with a gap larger than typical intra-word spacing (dot leader dots- // are spaced apart, unlike periods in abbreviations like "U.S.")- c == '.' && seg.has_non_dot_content() && gap > seg.avg_char_width() * 0.4+ // are spaced apart, unlike periods in abbreviations like "U.S.").+ // A loosely-kerned abbreviation/sentence period sits at ~1x the+ // average char width; genuine no-space dot leaders run far wider+ // (2x+). The earlier 0.4x cutoff sheared the trailing period off+ // abbreviations like "Sci."/"Chem." whenever the font kerned the+ // period a hair loose, dropping it entirely downstream.+ c == '.' && seg.has_non_dot_content() && gap > seg.avg_char_width() * 2.0};if dbg_gaps && y_overlap && !line_changed && gap > 0.0 {@@ -1479,6 +1534,7 @@ impl SegmentBuilder {fill_color: self.fill_color.clone(),stroke_color: self.stroke_color.clone(),confidence: None,+ link: None,});}diff --git a/crates/liteparse/src/main.rs b/crates/liteparse/src/main.rsindex 136c54f..4ce228b 100644--- a/crates/liteparse/src/main.rs+++ b/crates/liteparse/src/main.rs@@ -104,6 +104,13 @@ struct ParseCommand {/// modes. Created if missing.#[arg(long)]image_output_dir: Option<String>,++ /// Disable hyperlink extraction. By default, URI link annotations are+ /// rendered as `[text](url)` in markdown output. Pass this to emit the+ /// anchor text as plain text instead (e.g. for plain-text benchmark+ /// parity, where ground truth uses no link syntax).+ #[arg(long)]+ no_links: bool,}#[derive(Args, Debug)]@@ -247,6 +254,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {quiet: cmd.quiet,ocr_server_url: cmd.ocr_server_url,image_mode,+ extract_links: !cmd.no_links,..Default::default()};if let Some(n) = cmd.num_workers {diff --git a/crates/liteparse/src/markdown_layout/inline.rs b/crates/liteparse/src/markdown_layout/inline.rsindex 32a9813..ab90f07 100644--- a/crates/liteparse/src/markdown_layout/inline.rs+++ b/crates/liteparse/src/markdown_layout/inline.rs@@ -43,6 +43,17 @@ pub(super) fn escape_inline(s: &str) -> String {out}+/// Wrap `inner` in a markdown inline link to `url`. Uses the angle-bracket+/// destination form when the URL contains characters that would otherwise+/// terminate or break the `(url)` form (whitespace or parentheses).+fn apply_link(inner: &str, url: &str) -> String {+ if url.contains([' ', '\t', '(', ')']) {+ format!("[{}](<{}>)", inner, url)+ } else {+ format!("[{}]({})", inner, url)+ }+}+/// Wrap `inner` with the markdown markers for `style`. Mono wins over bold/italic:/// inline code (`` `…` ``) doesn't compose with emphasis in CommonMark, so when/// a span is mono we drop the `**/*` wrap. Bold + italic → `***…***`.@@ -91,9 +102,12 @@ pub(super) fn render_line_inline(line: &ProjectedLine) -> String {spans.sort_by(|a, b| a.x.total_cmp(&b.x));let styles: Vec<SpanStyle> = spans.iter().map(|s| SpanStyle::from_item(s)).collect();+ let links: Vec<Option<&str>> = spans.iter().map(|s| s.link.as_deref()).collect();- // Per-line shortcut.- let uniform = styles.iter().all(|s| *s == styles[0]);+ // Per-line shortcut — only when the whole line shares one style AND carries+ // no hyperlinks (a link must wrap only the spans it covers, so link lines+ // always take the per-group path below).+ let uniform = styles.iter().all(|s| *s == styles[0]) && links.iter().all(|l| l.is_none());if uniform {let joined = collapse_whitespace(&line.text);if joined.is_empty() {@@ -112,8 +126,9 @@ pub(super) fn render_line_inline(line: &ProjectedLine) -> String {let mut i = 0;while i < spans.len() {let style = styles[i];+ let link = links[i];let mut j = i + 1;- while j < spans.len() && styles[j] == style {+ while j < spans.len() && styles[j] == style && links[j] == link {j += 1;}let mut group_text = String::new();@@ -125,11 +140,14 @@ pub(super) fn render_line_inline(line: &ProjectedLine) -> String {}let group_text = collapse_whitespace(&group_text);let escaped = escape_inline(&group_text);- let rendered = if style.is_plain() {+ let mut rendered = if style.is_plain() {escaped} else {apply_style(&escaped, style)};+ if let Some(url) = link {+ rendered = apply_link(&rendered, url);+ }if !out.is_empty() && !out.ends_with(' ') {out.push(' ');}@@ -199,6 +217,16 @@ pub(super) fn append_inline_continuation(/// uniform-mono. Used by the paragraph-level optimization to decide whether/// to wrap once around the whole paragraph or fall back to per-line inline.pub(super) fn line_uniform_style(line: &ProjectedLine) -> Option<SpanStyle> {+ // A line carrying any hyperlink can't use the uniform fast path: the link+ // wraps only its own spans, so such lines must render via the per-group+ // path in `render_line_inline`.+ if line+ .spans+ .iter()+ .any(|s| !s.text.trim().is_empty() && s.link.is_some())+ {+ return None;+ }let mut iter = line.spans.iter()@@ -298,6 +326,43 @@ mod tests {assert!(out.contains("**bold**"), "got: {out}");}+ #[test]+ fn render_line_inline_wraps_link_span() {+ // Plain span followed by a linked span → `[anchor](url)` mid-line.+ let mut l = styled_line(+ &[+ ("see", 50.0, Some("Arial")),+ ("the docs", 150.0, Some("Arial")),+ ],+ 100.0,+ 10.0,+ );+ l.spans[1].link = Some("https://example.com/docs".to_string());+ let out = render_line_inline(&l);+ assert!(out.contains("see"), "got: {out}");+ assert!(+ out.contains("[the docs](https://example.com/docs)"),+ "got: {out}"+ );+ }++ #[test]+ fn render_line_inline_link_wraps_outside_emphasis() {+ // An italic linked span → `[*anchor*](url)` (link outside emphasis).+ let mut l = styled_line(&[("cite", 50.0, Some("Arial-Italic"))], 100.0, 10.0);+ l.spans[0].link = Some("https://example.com/p.pdf".to_string());+ let out = render_line_inline(&l);+ assert_eq!(out, "[*cite*](https://example.com/p.pdf)");+ }++ #[test]+ fn render_line_inline_link_url_with_space_uses_angle_brackets() {+ let mut l = styled_line(&[("link", 50.0, Some("Arial"))], 100.0, 10.0);+ l.spans[0].link = Some("https://example.com/a b".to_string());+ let out = render_line_inline(&l);+ assert_eq!(out, "[link](<https://example.com/a b>)");+ }+#[test]fn render_line_inline_mono_span() {let l = styled_line(diff --git a/crates/liteparse/src/parser.rs b/crates/liteparse/src/parser.rsindex 76afe02..f64c9fa 100644--- a/crates/liteparse/src/parser.rs+++ b/crates/liteparse/src/parser.rs@@ -137,6 +137,8 @@ impl LiteParse {target_pages.as_deref(),self.config.max_pages,render_images,+ self.config.extract_links+ && self.config.output_format == crate::config::OutputFormat::Markdown,)?;let t_extract = web_time::Instant::now();log(&format!(diff --git a/crates/liteparse/src/types.rs b/crates/liteparse/src/types.rsindex 616ccfc..9fe4a97 100644--- a/crates/liteparse/src/types.rs+++ b/crates/liteparse/src/types.rs@@ -58,6 +58,10 @@ pub struct TextItem {/// OCR confidence score (0.0–1.0). None for native PDF text.#[serde(skip_serializing_if = "Option::is_none")]pub confidence: Option<f32>,+ /// Target URI when this item falls inside a hyperlink annotation's+ /// rectangle. Populated in `extract.rs`; consumed by the markdown emitter.+ #[serde(skip_serializing_if = "Option::is_none")]+ pub link: Option<String>,}#[doc(hidden)]diff --git a/crates/pdfium-sys/src/dynamic.rs b/crates/pdfium-sys/src/dynamic.rsindex c5dbb15..cb87bda 100644--- a/crates/pdfium-sys/src/dynamic.rs+++ b/crates/pdfium-sys/src/dynamic.rs@@ -262,6 +262,19 @@ pub struct PdfiumBindings {pub FPDFBookmark_GetDest: unsafe extern "C" fn(FPDF_DOCUMENT, FPDF_BOOKMARK) -> FPDF_DEST,pub FPDFBookmark_GetAction: unsafe extern "C" fn(FPDF_BOOKMARK) -> FPDF_ACTION,pub FPDFAction_GetDest: unsafe extern "C" fn(FPDF_DOCUMENT, FPDF_ACTION) -> FPDF_DEST,+ pub FPDFAction_GetURIPath: unsafe extern "C" fn(+ FPDF_DOCUMENT,+ FPDF_ACTION,+ *mut std::os::raw::c_void,+ std::os::raw::c_ulong,+ ) -> std::os::raw::c_ulong,+ pub FPDFLink_Enumerate:+ unsafe extern "C" fn(FPDF_PAGE, *mut std::os::raw::c_int, *mut FPDF_LINK) -> FPDF_BOOL,+ pub FPDFLink_GetAction: unsafe extern "C" fn(FPDF_LINK) -> FPDF_ACTION,+ pub FPDFLink_GetAnnotRect: unsafe extern "C" fn(FPDF_LINK, *mut FS_RECTF) -> FPDF_BOOL,+ pub FPDFLink_CountQuadPoints: unsafe extern "C" fn(FPDF_LINK) -> std::os::raw::c_int,+ pub FPDFLink_GetQuadPoints:+ unsafe extern "C" fn(FPDF_LINK, std::os::raw::c_int, *mut FS_QUADPOINTSF) -> FPDF_BOOL,pub FPDFDest_GetDestPageIndex:unsafe extern "C" fn(FPDF_DOCUMENT, FPDF_DEST) -> std::os::raw::c_int,pub FPDFDest_GetLocationInPage: unsafe extern "C" fn(@@ -399,6 +412,12 @@ impl PdfiumBindings {FPDFBookmark_GetDest: load_fn!(lib, "FPDFBookmark_GetDest"),FPDFBookmark_GetAction: load_fn!(lib, "FPDFBookmark_GetAction"),FPDFAction_GetDest: load_fn!(lib, "FPDFAction_GetDest"),+ FPDFAction_GetURIPath: load_fn!(lib, "FPDFAction_GetURIPath"),+ FPDFLink_Enumerate: load_fn!(lib, "FPDFLink_Enumerate"),+ FPDFLink_GetAction: load_fn!(lib, "FPDFLink_GetAction"),+ FPDFLink_GetAnnotRect: load_fn!(lib, "FPDFLink_GetAnnotRect"),+ FPDFLink_CountQuadPoints: load_fn!(lib, "FPDFLink_CountQuadPoints"),+ FPDFLink_GetQuadPoints: load_fn!(lib, "FPDFLink_GetQuadPoints"),FPDFDest_GetDestPageIndex: load_fn!(lib, "FPDFDest_GetDestPageIndex"),FPDFDest_GetLocationInPage: load_fn!(lib, "FPDFDest_GetLocationInPage"),diff --git a/crates/pdfium/src/lib.rs b/crates/pdfium/src/lib.rsindex 59ec49e..f357b3c 100644--- a/crates/pdfium/src/lib.rs+++ b/crates/pdfium/src/lib.rs@@ -13,7 +13,9 @@ pub use document::{Document, OutlineEntry};pub use error::PdfiumError;pub use font::{Font, FontType};pub use library::Library;-pub use page::{ImageBounds, Page, PathObject, PathSegment, SegmentKind, ViewportTransform};+pub use page::{+ ImageBounds, Page, PathObject, PathSegment, PdfLink, SegmentKind, ViewportTransform,+};pub use struct_tree::StructNode;pub use text_page::{TextChar, TextCharIter, TextPage};pub use types::*;diff --git a/crates/pdfium/src/page.rs b/crates/pdfium/src/page.rsindex 1d8d915..f6db71b 100644--- a/crates/pdfium/src/page.rs+++ b/crates/pdfium/src/page.rs@@ -51,6 +51,16 @@ pub struct PathObject {pub segments: Vec<PathSegment>,}+/// A URI hyperlink annotation on a page. `rect` is in viewport space+/// (top-left origin, 72 DPI), matching `TextItem` coordinates so the URI can+/// be assigned to overlapping text. Only external URI links are represented;+/// internal GoTo/named destinations are excluded.+#[derive(Debug, Clone)]+pub struct PdfLink {+ pub rect: RectF,+ pub uri: String,+}+/// A loaded page within a [`Document`].////// The `'doc` lifetime ties the page to its owning document; `'lib` carries@@ -346,6 +356,121 @@ impl<'doc, 'lib: 'doc> Page<'doc, 'lib> {}out}++ /// Enumerate URI hyperlink annotations on this page. Each link's clickable+ /// rectangle is mapped into viewport space (matching `TextItem`); the URI+ /// is read from the link's URI action. Annotations whose action is not a+ /// URI (internal GoTo / named destinations) are skipped.+ pub fn links(&self, view_box: &RectF) -> Vec<PdfLink> {+ let mut out = Vec::new();+ let mut start_pos: std::os::raw::c_int = 0;+ let mut link_annot: pdfium_sys::FPDF_LINK = std::ptr::null_mut();+ loop {+ let ok = unsafe {+ ffi!(FPDFLink_Enumerate(+ self.handle,+ &mut start_pos,+ &mut link_annot+ ))+ };+ if ok == 0 {+ break;+ }+ if link_annot.is_null() {+ continue;+ }+ let action = unsafe { ffi!(FPDFLink_GetAction(link_annot)) };+ if action.is_null() {+ continue;+ }+ let Some(uri) = read_uri_path(self.doc_handle, action) else {+ continue;+ };++ // Prefer per-line quad points: a link that wraps across lines has+ // one quad per line, each tight around the anchor text. The single+ // annotation rect is their *union* — a tall box that would swallow+ // the unlinked words sitting between the lines. Fall back to the+ // annot rect only when no quads are present.+ let quad_count = unsafe { ffi!(FPDFLink_CountQuadPoints(link_annot)) };+ let mut emitted = false;+ for q in 0..quad_count {+ let mut quad = pdfium_sys::FS_QUADPOINTSF::default();+ let ok = unsafe { ffi!(FPDFLink_GetQuadPoints(link_annot, q, &mut quad)) };+ if ok == 0 {+ continue;+ }+ let page_bounds = RectF {+ left: quad.x1.min(quad.x2).min(quad.x3).min(quad.x4),+ bottom: quad.y1.min(quad.y2).min(quad.y3).min(quad.y4),+ right: quad.x1.max(quad.x2).max(quad.x3).max(quad.x4),+ top: quad.y1.max(quad.y2).max(quad.y3).max(quad.y4),+ };+ out.push(PdfLink {+ rect: self.bounds_to_viewport(view_box, &page_bounds),+ uri: uri.clone(),+ });+ emitted = true;+ }+ if emitted {+ continue;+ }++ let mut rect = pdfium_sys::FS_RECTF {+ left: 0.0,+ top: 0.0,+ right: 0.0,+ bottom: 0.0,+ };+ let got = unsafe { ffi!(FPDFLink_GetAnnotRect(link_annot, &mut rect)) };+ if got == 0 {+ continue;+ }+ let page_bounds = RectF {+ left: rect.left,+ top: rect.top,+ right: rect.right,+ bottom: rect.bottom,+ };+ out.push(PdfLink {+ rect: self.bounds_to_viewport(view_box, &page_bounds),+ uri,+ });+ }+ out+ }+}++/// Read a link action's URI path. PDFium returns the URI as a NUL-terminated+/// 7-bit-ASCII byte string; the two-call protocol queries the length first.+/// Returns `None` for non-URI actions (length 0) or empty URIs.+fn read_uri_path(+ doc: pdfium_sys::FPDF_DOCUMENT,+ action: pdfium_sys::FPDF_ACTION,+) -> Option<String> {+ let needed =+ unsafe { ffi!(FPDFAction_GetURIPath(doc, action, std::ptr::null_mut(), 0)) } as usize;+ if needed < 2 {+ return None;+ }+ let mut buf: Vec<u8> = vec![0; needed];+ let written = unsafe {+ ffi!(FPDFAction_GetURIPath(+ doc,+ action,+ buf.as_mut_ptr() as *mut std::os::raw::c_void,+ needed as std::os::raw::c_ulong,+ ))+ } as usize;+ if written < 2 {+ return None;+ }+ // `written` includes the trailing NUL.+ let end = written.saturating_sub(1).min(buf.len());+ let uri = String::from_utf8_lossy(&buf[..end])+ .trim_matches(char::from(0))+ .to_string();+ if uri.is_empty() { None } else { Some(uri) }}const FS_IDENTITY: pdfium_sys::FS_MATRIX = pdfium_sys::FS_MATRIX {diff --git a/packages/python/liteparse/parser.py b/packages/python/liteparse/parser.pyindex 9fe4317..895030e 100644--- a/packages/python/liteparse/parser.py+++ b/packages/python/liteparse/parser.py@@ -88,6 +88,7 @@ class LiteParse:quiet: Optional[bool] = None,num_workers: Optional[int] = None,image_mode: Optional[str] = None,+ extract_links: Optional[bool] = None,):"""Initialize LiteParse parser.@@ -105,6 +106,8 @@ class LiteParse:password: Password for encrypted/protected documentsquiet: Suppress progress outputnum_workers: Number of concurrent OCR workers (default: CPU cores - 1)+ extract_links: Render hyperlink annotations as ``[text](url)`` in+ markdown output (default: True). Set False for plain anchor text."""kwargs = {}if ocr_enabled is not None:@@ -133,6 +136,8 @@ class LiteParse:kwargs["num_workers"] = num_workersif image_mode is not None:kwargs["image_mode"] = image_mode+ if extract_links is not None:+ kwargs["extract_links"] = extract_linksself._native = _NativeLiteParse(**kwargs)