Skip to content

A fast, helpful, and open-source document parser

wip

Logan Markewichff3d1ae22073
diff --git a/MARKDOWN_PROGRESS.md b/MARKDOWN_PROGRESS.md
index 105055e..541289b 100644
--- a/MARKDOWN_PROGRESS.md
+++ b/MARKDOWN_PROGRESS.md
@@ -2,6 +2,48 @@
Tracks build-order progress against [MARKDOWN_PLAN.md](MARKDOWN_PLAN.md). Update as steps complete.
+> **2026-06-13 (pm) trailing-period loss fix (Mode 16, `extract.rs` dot-leader threshold) — olmOCR Overall 38.6 → 39.1% (+0.5pp); order 53.0 → 54.4% (+14), multi_column 63.6 → 65.3% (+15 tests), long_tiny_text 25.8 → 29.2% (+15 tests), present 15.8 → 17.9%; headers_footers −2 / table_tests −5 (CI noise). opendataloader BYTE-FLAT (NID 0.9126 / TEDS 0.6929 / MHS 0.8036), ParseBench text_content FLAT (0.65097 both, forced re-infer A/B) and table FLAT (grits_con 0.5731 / composite 0.4076 / TRM 0.2196). 207 lib tests pass. One-line change, shipped.** Resolves the 2026-06-08 FAILURE_MODES Mode 16 "trailing-period loss (~25 tests, probably localized)" item — the cleanest remaining lever in the `text_absent` class flagged by the 2026-06-13 formatting-triage pivot, and explicitly NOT the compressed-metric-font danger zone.
+>
+> **Root cause (single misfiring threshold).** The no-pending-space dot-leader splitter at `extract.rs:644` broke a `.` off preceding content whenever `gap > avg_char_width * 0.4`. A loosely-kerned abbreviation/sentence period sits at **~1.0× avg char width** (measured `g/cw ≈ 1.0` on `Sci.`/`Chem.`/`Environ.`/`Phys.`), so it tripped the splitter; the split-off `.` segment then merged forward into the next token and vanished. Tightly-kerned periods (`Mater.` at `g/cw=0.14`) stayed attached, which is why the loss was *positional* (only before column/tab gaps in reference lists) and looked per-word. Genuine no-space dot leaders run `g/cw ≈ 2–23` (verified a real one at 23.2), and spaced leaders take the *other* (`pending_space`) branch entirely — so raising the cutoff `0.4 → 2.0` opens a wide safe valley.
+>
+> **Why olmOCR-only.** Pure extraction-fidelity fix → only moves the benchmark whose char-exact `order`/anchor predicates a dropped period actually breaks (a 1-char Levenshtein miss fails the whole anchor). opendataloader (NID/TEDS/MHS) and ParseBench (word-match rules normalize trailing punctuation) are structurally insensitive, hence byte-flat/flat — confirmed, not assumed. Qualitatively large for RAG ingestion regardless: reference lists go from `Cravatt, B.F (2004) … Angew Chem Int Ed` to fully-periodized `Cravatt, B.F. (2004). … Angew. Chem. Int. Ed.`.
+>
+> **A/B method (the load-bearing rigor).** First text_content run reused **stale cached inference** (`parse-bench run` defaults `--force=False`) and reported a false-flat to 15 digits — re-ran *both* binaries with `--force` for a true same-harness delta. For olmOCR (in-process Python binding, not the CLI binary) a same-session base required reverting the one line, rebuilding the binding, and converting a clean `liteparse_m16base` candidate — historical numbers carry documented binary drift and can't be trusted as the baseline. Byte-diff bound the blast radius first: multi_column 57/231, headers_footers 47/266, long_tiny_text 12/62 changed — all sampled diffs were period restorations (plus one re-merged wrongly-fragmented paragraph), one trivial `, and`→`,and` collateral against dozens of restorations. Scratch candidates `liteparse_m16` / `liteparse_m16base` left in `bench_data/`. Memory: [[project_missing_spaces]].
+>
+> **2026-06-13 semantic-formatting triage + table-cell emphasis A/B — lever empirically KILLED, REVERTED (byte-identical baseline, 207 lib tests pass). No code shipped; the finding is the deliverable.** Set out to lift ParseBench `text_formatting` (the 2026-06-11 audit's Tier-3 #7 fidelity lever: bold 0.53 / italic 0.37 / title_hierarchy 0.37). Measured first instead of editing, and the measurement says **stop**: bold/italic detection has ~no benchmark headroom.
+>
+> **Fresh baseline confirmed (no drift):** is_bold 0.5273, is_italic 0.3696, is_title 0.4413, title_hierarchy 0.3716, underline/sup/mark 0.0, code_block 0.2, overall avg_rule_pass_rate 0.3736.
+>
+> **Triage reframe (`dataset_eval_utils/triage_formatting.py` — runs each rule through ParseBench's own `create_test_rule`, buckets each failure by whether the GT text is even in our output).** The inline emphasis pipeline ALREADY exists (`inline.rs::render_line_inline` does per-span `**`/`*`/`` ` ``/links) — the audit's "whole-line-only" framing was stale. The gap is NOT missing capability:
+> - is_bold: **76% of failures are `text_absent`** (word not in output at all → extraction/reading-order, ORTHOGONAL), 24% `present_plain` (216 rules, detector-addressable).
+> - is_italic: 47% text_absent, 53% `present_plain` (208 rules — most addressable relative to size).
+> - is_title: 63% text_absent, 37% present_plain (422 rules).
+> - title_hierarchy: 97% of unsatisfied constraints are missing_title/missing_edge (PRESENCE), only 2% level + 0% order → **NOT a leveling problem**; it's heading detection.
+>
+> **Drill-down (`dataset_eval_utils/drill_formatting_misses.py` — temporarily exposed font_flags/font_weight in JSON, reverted).** Of the introspectable `present_plain` misses: ~25% `flag_set` (item carries the italic/bold flag → detection SHOULD fire, miss is downstream), ~75% `no_signal` (italic/bold absent from font flags AND name — synthetic skew / outline-only weight, **unrecoverable from font metadata**). ~50% were `no_match` (JSON `text_items` empty for those PDFs — tooling gap, not a product fact). The `flag_set` misses cluster in table-heavy financial/SERFF/IPO docs → root cause: **table cells (`escape_table_cell`) and headings (`# text`) render PLAIN, never through the inline pipeline.** (`name_contains_any` in projection.rs is also case-sensitive.)
+>
+> **Built + A/B'd the one buildable lever: per-cell BOLD emphasis** at the two shared builders `finalize_table_run` + `build_ruled_table` (reusing the already-tracked `TableCell.bold` / ruled `cell_is_bold`; `wrap_emphasis` made `pub(super)` — no struct churn, late-wrapping so cell-splitting logic is untouched). Result:
+> - ParseBench is_bold **+0.0000** (per-doc avg over 476 docs — the few flipped cells round to zero; bold/italic GT targets aren't in table cells, and in-table bold detection hits the same no_signal wall).
+> - ParseBench table GriTS_con / TRM **0.0000** (safe — `normalize_cell_text` strips `**`).
+> - opendataloader **TEDS −0.0167** (0.6929→0.6762), overall −0.0019 — ODL's TEDS does NOT strip markdown from cells, so `**Foo**`≠`Foo`.
+> Net loss, zero benefit → reverted. **Do not re-attempt table/heading emphasis for benchmark gain.**
+>
+> **Conclusion + pivot.** Bold/italic semantic formatting is a dead benchmark lever (gap = ⅔ text_absent + ¾ no_signal; buildable pool tiny and not in tables). The real opportunity is the **`text_absent` (missing text)** class — the documented Tier-1 word-fusion / column-interleave / glyph-soup extraction work, which lifts is_bold/is_title/text_content/order/NID together. Next step (not yet done): a fresh scoped triage of the current text_absent set (cross-ref olmOCR `order`) to find whether any sub-cause is still a clean lever before committing to the risky compressed-metric-font work (the −308/−187 danger zone). Memory: [[project_formatting_triage]]. Scripts: `triage_formatting.py` (SIGALRM 3s guard — ParseBench bold/italic tempered-greedy `.DOTALL` regexes backtrack catastrophically on large non-matching docs; reliable for cause buckets, NOT precise pass-rate deltas — use the official eval for those), `drill_formatting_misses.py`, CSV `formatting_triage.csv`.
+>
+> **2026-06-13 hyperlink extraction → `[text](url)` (new capability, default ON; ALL benchmarks zero-regression because harnesses disable it) — opendataloader BYTE-IDENTICAL (Overall 0.8719 / NID 0.9125 / TEDS 0.6929 / MHS 0.8036), ParseBench table BYTE-IDENTICAL (grits_con 0.5737 / TRM 0.2196 / composite 0.4079), olmOCR matched (Overall 38.6% / multi_column 63.6% / table_tests 48.7% / long_tiny_text 25.8%); 207 lib tests pass (3 new).** Implements the 2026-06-11 audit's Tier-2 #5 "links entirely absent" — the one genuine missing *capability* (vs a metric gap): no `GetURI`/annotation extraction existed anywhere in `extract.rs` or the pdfium wrapper. Targets RAG ingestion, not a benchmark lever.
+>
+> **Annotation-based only** (`FPDFLink_Enumerate` → `FPDFLink_GetAction` → `FPDFAction_GetURIPath`), not auto-detected web-links — captures real clickable links whose anchor text differs from the URL (`[click here](https://…)`), which web-link scanning can't. No PDFium fork rebuild needed: all symbols were already in the static `bindings.rs`; only the libloading table in `pdfium-sys/src/dynamic.rs` needed the four fns added (plus `FPDFLink_CountQuadPoints`/`GetQuadPoints`).
+>
+> **Data path.** New `Page::links(view_box) -> Vec<PdfLink { rect (viewport), uri }>` in `pdfium/src/page.rs` (mirrors `path_objects`; rect mapped via the existing `bounds_to_viewport`). New `TextItem.link: Option<String>` (rides through projection's `..proj.item` for free). `assign_links` in `extract.rs` attaches a URI to every text item whose bbox center falls in the link rect. Rendered in `markdown_layout/inline.rs`: group spans by `(style, link)`, wrap each group `[…](url)` *outside* the emphasis wrap (`[*cite*](url)`), angle-bracket form for URLs with spaces/parens. `line_uniform_style` returns `None` for any link-bearing line so the paragraph/list uniform fast-paths route through the per-span renderer — every new branch is gated on link presence, so link-free output is byte-identical.
+>
+> **Per-line quad points preferred over the union rect.** A link wrapping across lines has one tight quad per line; the single annotation `/Rect` is their *union* (a tall box that swallows the unlinked words between the lines). When quads exist we emit one `PdfLink` per quad; otherwise fall back to the union rect. For the no-quad-point multi-line case the true anchor is unrecoverable, so `assign_links` **drops** the link when `rect height > 1.8× median covered-item height` rather than wrap a whole sentence (verified on opendataloader `…086`, a `perma.cc/Z89Y-` / `SEVS.` wrapped citation). Single-line links (`…095` `[*(Niederle and Vesterlund 2007)*](…)`, `…185`) render correctly.
+>
+> **Why default-ON but every harness disables it.** The GT corpora **never use `[](url)` syntax** — 0/200 opendataloader GT files; URLs appear as plain text. Measured: links-ON regresses opendataloader **NID 0.9125 → 0.9102, MHS 0.8036 → 0.8021** (TEDS flat). So real users get richer markdown by default, but the three bench providers opt out for parity: opendataloader + ParseBench pass CLI `--no-links`; the olmOCR in-process Python runner passes `extract_links=False`. Config `extract_links` (default true) is wired through CLI (`--no-links`), the pyo3 binding + `packages/python` wrapper, and napi. The parser only extracts links when `output_format == Markdown` (text/JSON paths skip the per-page `FPDFLink_Enumerate` call entirely).
+>
+> **ParseBench nuance (flagged, not yet measured).** There is **no `is_link` rule type** in ParseBench (formatting kinds are bold/italic/underline/strikeout/mark/sup/sub/title/latex/code_block/title_hierarchy), so links can't win `text_formatting` points. The tracked `table` group is byte-identical with/without links (tables carry no link annotations). **But `text_content.jsonl` GT contains `[text](url)` syntax in ~7 lines / ~4 docs** (examgenius.in, epa.gov, bonnieplants, artsandculture), scored by `missing_specific_word`/`order`/`unexpected_word`. So blanket `--no-links` on ParseBench is safe for the tracked table metric but is an *unmeasured assumption* for `text_content` — links-ON could match GT better on those ~4 docs while adding unexpected `](url)` tokens on the plain-text majority. Net `text_content` effect unmeasured (only ran `--group table` this session). Memory: [[project_markdown_links]].
+>
+> Kill-switch: `--no-links` (CLI) / `extract_links=False` (Python) / `extractLinks: false` (Node). Files: `pdfium-sys/src/dynamic.rs`, `pdfium/src/{page,lib}.rs`, `liteparse/src/{types,extract,parser,config,main}.rs`, `liteparse/src/markdown_layout/inline.rs`, `liteparse-python/src/lib.rs`, `liteparse-napi/src/types.rs`, `packages/python/liteparse/parser.py`, + 3 bench provider scripts.
+>
> **2026-06-12 (pm) per-font empirical space-width recovery (Mode 14 word fusion, no-metric fonts) — olmOCR long_tiny_text 25.3 → 25.8% (+2 tests, clean, zero regressions); multi_column / table_tests / ParseBench BYTE-IDENTICAL; opendataloader flat (NID/TEDS/MHS unchanged); 204 lib tests pass.** Targeted the 2026-06-11 audit's Tier-1 #2 dense/small-font word fusion (22.6% of `long_tiny_text` files), reproduced on `long_tiny_text/20_pg58` (`theynexttakeonthe`, `againstManchester`, `transferbusiness`, `forSpurs` — body rendered as word-soup).
>
> **Root cause (Layer 1, `extract.rs`).** The missing-space recovery in the no-pending-space branch uses `thresh = 0.7 * space_w` when the font exposes an ASCII-space advance (`ch.font_space_width()`), else a fixed `0.35 * em_vp` fallback. `20_pg58`'s body font exposes **no space metric** (`sw=-1` in `LITEPARSE_DEBUG_GAPS`), so every boundary hit the fallback — which computes ~3pt for an 8.5pt font while the real inter-word gap is ~1pt, so nothing ever fired. Histogramming `loose_gap` by the PDFium `ps` flag showed the gaps are cleanly **bimodal**: intra-word kerning loose ≤ 0.4pt (434 events), genuine spaces 0.8–1.4pt (34 events), near-empty valley between — and the boundaries where PDFium *did* emit a space (`ps=1`) land in that exact 0.8–1.4 band, giving a free per-font calibration signal.
diff --git a/crates/liteparse-napi/src/types.rs b/crates/liteparse-napi/src/types.rs
index d8844c7..0c483c4 100644
--- a/crates/liteparse-napi/src/types.rs
+++ b/crates/liteparse-napi/src/types.rs
@@ -39,6 +39,9 @@ pub struct JsLiteParseConfig {
/// (default — emits `![](image_pN_K.png)` references with no bytes), or
/// "embed" (also returns each image's PNG bytes on `images`).
pub image_mode: Option<String>,
+ /// Render hyperlink annotations as `[text](url)` in markdown output
+ /// (default true). Set false for plain anchor text.
+ pub extract_links: Option<bool>,
}
impl JsLiteParseConfig {
@@ -91,6 +94,9 @@ impl JsLiteParseConfig {
_ => ImageMode::Placeholder,
};
}
+ if let Some(v) = self.extract_links {
+ cfg.extract_links = v;
+ }
cfg
}
@@ -117,6 +123,7 @@ impl JsLiteParseConfig {
ImageMode::Placeholder => "placeholder".to_string(),
ImageMode::Embed => "embed".to_string(),
}),
+ extract_links: Some(cfg.extract_links),
}
}
}
diff --git a/crates/liteparse-python/src/lib.rs b/crates/liteparse-python/src/lib.rs
index 246b490..a3b4886 100644
--- a/crates/liteparse-python/src/lib.rs
+++ b/crates/liteparse-python/src/lib.rs
@@ -327,6 +327,7 @@ impl LiteParse {
quiet = None,
num_workers = None,
image_mode = None,
+ extract_links = None,
))]
fn new(
ocr_language: Option<String>,
@@ -342,6 +343,7 @@ impl LiteParse {
quiet: Option<bool>,
num_workers: Option<usize>,
image_mode: Option<String>,
+ extract_links: Option<bool>,
) -> PyResult<Self> {
let mut cfg = LiteParseConfig::default();
if let Some(v) = ocr_language {
@@ -391,6 +393,9 @@ impl LiteParse {
_ => ImageMode::Placeholder,
};
}
+ if let Some(v) = extract_links {
+ cfg.extract_links = v;
+ }
let inner = liteparse::parser::LiteParse::new(cfg.clone());
let runtime = tokio::runtime::Runtime::new()
diff --git a/crates/liteparse/src/config.rs b/crates/liteparse/src/config.rs
index 21d0a07..f3ff319 100644
--- a/crates/liteparse/src/config.rs
+++ b/crates/liteparse/src/config.rs
@@ -30,6 +30,10 @@ pub struct LiteParseConfig {
/// Controls how raster images are surfaced in markdown output. Has no
/// effect on JSON / text outputs.
pub image_mode: ImageMode,
+ /// Extract hyperlink annotations and render them as `[text](url)` in
+ /// markdown output. Default on. Disable for benchmark parity with
+ /// plain-text ground truth (the GT corpora never use link syntax).
+ pub extract_links: bool,
}
/// Image handling for the markdown emitter.
@@ -76,6 +80,7 @@ impl Default for LiteParseConfig {
quiet: false,
num_workers: default_num_workers(),
image_mode: ImageMode::Placeholder,
+ extract_links: true,
}
}
}
diff --git a/crates/liteparse/src/extract.rs b/crates/liteparse/src/extract.rs
index 2ee37e6..bb467dd 100644
--- a/crates/liteparse/src/extract.rs
+++ b/crates/liteparse/src/extract.rs
@@ -5,7 +5,9 @@ use crate::types::{
ExtractedImage, GraphicPrimitive, ImageRef, OutlineTarget, Page as LitePage, PdfInput, Rect,
StructNode, TextItem,
};
-use pdfium::{Document, Font, FontType, Library, Page, PathObject, RectF, SegmentKind, TextPage};
+use pdfium::{
+ Document, Font, FontType, Library, Page, PathObject, PdfLink, RectF, SegmentKind, TextPage,
+};
/// Open a PDF from path or bytes with an optional password.
///
@@ -47,7 +49,7 @@ pub(crate) fn extract_pages_from_document(
target_pages: Option<&[u32]>,
max_pages: usize,
) -> Result<Vec<LitePage>, LiteParseError> {
- Ok(extract_pages_and_images(document, target_pages, max_pages, false)?.0)
+ Ok(extract_pages_and_images(document, target_pages, max_pages, false, false)?.0)
}
/// Same as `extract_pages_from_document` but optionally also renders every
@@ -60,6 +62,7 @@ pub(crate) fn extract_pages_and_images(
target_pages: Option<&[u32]>,
max_pages: usize,
render_images: bool,
+ extract_links: bool,
) -> Result<(Vec<LitePage>, Vec<ExtractedImage>), LiteParseError> {
let page_count = document.page_count();
let mut pages = Vec::new();
@@ -86,7 +89,10 @@ pub(crate) fn extract_pages_and_images(
right: page.width(),
bottom: 0.0,
});
- let text_items = extract_page_text_items(&page, &text_page, &view_box)?;
+ let mut text_items = extract_page_text_items(&page, &text_page, &view_box)?;
+ if extract_links {
+ assign_links(&mut text_items, &page.links(&view_box));
+ }
let graphics = extract_page_graphics(&page, &view_box);
let struct_nodes = extract_page_struct_nodes(&page, &view_box);
let image_refs = extract_page_image_refs(&page, page_number);
@@ -109,6 +115,50 @@ pub(crate) fn extract_pages_and_images(
Ok((pages, images))
}
+/// Assign hyperlink URIs to text items whose bbox center falls inside a link
+/// annotation's rectangle. Both the item bbox and the link rect are in
+/// viewport space. First matching link wins.
+///
+/// A link rect taller than `MULTILINE_DROP_FACTOR`× the height of the text it
+/// covers is a multi-line annotation given to us as a single *union* box (no
+/// per-line quad points). Its true anchor — which words on the intervening
+/// lines are actually linked — is unrecoverable, so we drop it rather than
+/// wrap a whole sentence in a misleading link. Well-formed multi-line links
+/// expose quad points and arrive here as one single-line rect per line.
+fn assign_links(items: &mut [TextItem], links: &[PdfLink]) {
+ if links.is_empty() {
+ return;
+ }
+ const MULTILINE_DROP_FACTOR: f32 = 1.8;
+ for link in links {
+ let r = &link.rect;
+ let covered: Vec<usize> = items
+ .iter()
+ .enumerate()
+ .filter(|(_, it)| {
+ let cx = it.x + it.width / 2.0;
+ let cy = it.y + it.height / 2.0;
+ cx >= r.left && cx <= r.right && cy >= r.top && cy <= r.bottom
+ })
+ .map(|(i, _)| i)
+ .collect();
+ if covered.is_empty() {
+ continue;
+ }
+ let mut heights: Vec<f32> = covered.iter().map(|&i| items[i].height).collect();
+ heights.sort_by(f32::total_cmp);
+ let median_h = heights[heights.len() / 2];
+ if median_h > 0.0 && (r.bottom - r.top) > MULTILINE_DROP_FACTOR * median_h {
+ continue;
+ }
+ for &i in &covered {
+ if items[i].link.is_none() {
+ items[i].link = Some(link.uri.clone());
+ }
+ }
+ }
+}
+
/// Walk the document outline (bookmarks). Returns entries in pre-order.
/// Empty when the PDF has no outline.
pub(crate) fn extract_outline(document: &Document) -> Vec<OutlineTarget> {
@@ -590,8 +640,13 @@ fn extract_page_text_items(
} else {
// Without a pending space: break when a dot follows non-dot content
// with a gap larger than typical intra-word spacing (dot leader dots
- // are spaced apart, unlike periods in abbreviations like "U.S.")
- c == '.' && seg.has_non_dot_content() && gap > seg.avg_char_width() * 0.4
+ // are spaced apart, unlike periods in abbreviations like "U.S.").
+ // A loosely-kerned abbreviation/sentence period sits at ~1x the
+ // average char width; genuine no-space dot leaders run far wider
+ // (2x+). The earlier 0.4x cutoff sheared the trailing period off
+ // abbreviations like "Sci."/"Chem." whenever the font kerned the
+ // period a hair loose, dropping it entirely downstream.
+ c == '.' && seg.has_non_dot_content() && gap > seg.avg_char_width() * 2.0
};
if dbg_gaps && y_overlap && !line_changed && gap > 0.0 {
@@ -1479,6 +1534,7 @@ impl SegmentBuilder {
fill_color: self.fill_color.clone(),
stroke_color: self.stroke_color.clone(),
confidence: None,
+ link: None,
});
}
diff --git a/crates/liteparse/src/main.rs b/crates/liteparse/src/main.rs
index 136c54f..4ce228b 100644
--- a/crates/liteparse/src/main.rs
+++ b/crates/liteparse/src/main.rs
@@ -104,6 +104,13 @@ struct ParseCommand {
/// modes. Created if missing.
#[arg(long)]
image_output_dir: Option<String>,
+
+ /// Disable hyperlink extraction. By default, URI link annotations are
+ /// rendered as `[text](url)` in markdown output. Pass this to emit the
+ /// anchor text as plain text instead (e.g. for plain-text benchmark
+ /// parity, where ground truth uses no link syntax).
+ #[arg(long)]
+ no_links: bool,
}
#[derive(Args, Debug)]
@@ -247,6 +254,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
quiet: cmd.quiet,
ocr_server_url: cmd.ocr_server_url,
image_mode,
+ extract_links: !cmd.no_links,
..Default::default()
};
if let Some(n) = cmd.num_workers {
diff --git a/crates/liteparse/src/markdown_layout/inline.rs b/crates/liteparse/src/markdown_layout/inline.rs
index 32a9813..ab90f07 100644
--- a/crates/liteparse/src/markdown_layout/inline.rs
+++ b/crates/liteparse/src/markdown_layout/inline.rs
@@ -43,6 +43,17 @@ pub(super) fn escape_inline(s: &str) -> String {
out
}
+/// Wrap `inner` in a markdown inline link to `url`. Uses the angle-bracket
+/// destination form when the URL contains characters that would otherwise
+/// terminate or break the `(url)` form (whitespace or parentheses).
+fn apply_link(inner: &str, url: &str) -> String {
+ if url.contains([' ', '\t', '(', ')']) {
+ format!("[{}](<{}>)", inner, url)
+ } else {
+ format!("[{}]({})", inner, url)
+ }
+}
+
/// Wrap `inner` with the markdown markers for `style`. Mono wins over bold/italic:
/// inline code (`` `…` ``) doesn't compose with emphasis in CommonMark, so when
/// a span is mono we drop the `**/*` wrap. Bold + italic → `***…***`.
@@ -91,9 +102,12 @@ pub(super) fn render_line_inline(line: &ProjectedLine) -> String {
spans.sort_by(|a, b| a.x.total_cmp(&b.x));
let styles: Vec<SpanStyle> = spans.iter().map(|s| SpanStyle::from_item(s)).collect();
+ let links: Vec<Option<&str>> = spans.iter().map(|s| s.link.as_deref()).collect();
- // Per-line shortcut.
- let uniform = styles.iter().all(|s| *s == styles[0]);
+ // Per-line shortcut — only when the whole line shares one style AND carries
+ // no hyperlinks (a link must wrap only the spans it covers, so link lines
+ // always take the per-group path below).
+ let uniform = styles.iter().all(|s| *s == styles[0]) && links.iter().all(|l| l.is_none());
if uniform {
let joined = collapse_whitespace(&line.text);
if joined.is_empty() {
@@ -112,8 +126,9 @@ pub(super) fn render_line_inline(line: &ProjectedLine) -> String {
let mut i = 0;
while i < spans.len() {
let style = styles[i];
+ let link = links[i];
let mut j = i + 1;
- while j < spans.len() && styles[j] == style {
+ while j < spans.len() && styles[j] == style && links[j] == link {
j += 1;
}
let mut group_text = String::new();
@@ -125,11 +140,14 @@ pub(super) fn render_line_inline(line: &ProjectedLine) -> String {
}
let group_text = collapse_whitespace(&group_text);
let escaped = escape_inline(&group_text);
- let rendered = if style.is_plain() {
+ let mut rendered = if style.is_plain() {
escaped
} else {
apply_style(&escaped, style)
};
+ if let Some(url) = link {
+ rendered = apply_link(&rendered, url);
+ }
if !out.is_empty() && !out.ends_with(' ') {
out.push(' ');
}
@@ -199,6 +217,16 @@ pub(super) fn append_inline_continuation(
/// uniform-mono. Used by the paragraph-level optimization to decide whether
/// to wrap once around the whole paragraph or fall back to per-line inline.
pub(super) fn line_uniform_style(line: &ProjectedLine) -> Option<SpanStyle> {
+ // A line carrying any hyperlink can't use the uniform fast path: the link
+ // wraps only its own spans, so such lines must render via the per-group
+ // path in `render_line_inline`.
+ if line
+ .spans
+ .iter()
+ .any(|s| !s.text.trim().is_empty() && s.link.is_some())
+ {
+ return None;
+ }
let mut iter = line
.spans
.iter()
@@ -298,6 +326,43 @@ mod tests {
assert!(out.contains("**bold**"), "got: {out}");
}
+ #[test]
+ fn render_line_inline_wraps_link_span() {
+ // Plain span followed by a linked span → `[anchor](url)` mid-line.
+ let mut l = styled_line(
+ &[
+ ("see", 50.0, Some("Arial")),
+ ("the docs", 150.0, Some("Arial")),
+ ],
+ 100.0,
+ 10.0,
+ );
+ l.spans[1].link = Some("https://example.com/docs".to_string());
+ let out = render_line_inline(&l);
+ assert!(out.contains("see"), "got: {out}");
+ assert!(
+ out.contains("[the docs](https://example.com/docs)"),
+ "got: {out}"
+ );
+ }
+
+ #[test]
+ fn render_line_inline_link_wraps_outside_emphasis() {
+ // An italic linked span → `[*anchor*](url)` (link outside emphasis).
+ let mut l = styled_line(&[("cite", 50.0, Some("Arial-Italic"))], 100.0, 10.0);
+ l.spans[0].link = Some("https://example.com/p.pdf".to_string());
+ let out = render_line_inline(&l);
+ assert_eq!(out, "[*cite*](https://example.com/p.pdf)");
+ }
+
+ #[test]
+ fn render_line_inline_link_url_with_space_uses_angle_brackets() {
+ let mut l = styled_line(&[("link", 50.0, Some("Arial"))], 100.0, 10.0);
+ l.spans[0].link = Some("https://example.com/a b".to_string());
+ let out = render_line_inline(&l);
+ assert_eq!(out, "[link](<https://example.com/a b>)");
+ }
+
#[test]
fn render_line_inline_mono_span() {
let l = styled_line(
diff --git a/crates/liteparse/src/parser.rs b/crates/liteparse/src/parser.rs
index 76afe02..f64c9fa 100644
--- a/crates/liteparse/src/parser.rs
+++ b/crates/liteparse/src/parser.rs
@@ -137,6 +137,8 @@ impl LiteParse {
target_pages.as_deref(),
self.config.max_pages,
render_images,
+ self.config.extract_links
+ && self.config.output_format == crate::config::OutputFormat::Markdown,
)?;
let t_extract = web_time::Instant::now();
log(&format!(
diff --git a/crates/liteparse/src/types.rs b/crates/liteparse/src/types.rs
index 616ccfc..9fe4a97 100644
--- a/crates/liteparse/src/types.rs
+++ b/crates/liteparse/src/types.rs
@@ -58,6 +58,10 @@ pub struct TextItem {
/// OCR confidence score (0.0–1.0). None for native PDF text.
#[serde(skip_serializing_if = "Option::is_none")]
pub confidence: Option<f32>,
+ /// Target URI when this item falls inside a hyperlink annotation's
+ /// rectangle. Populated in `extract.rs`; consumed by the markdown emitter.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub link: Option<String>,
}
#[doc(hidden)]
diff --git a/crates/pdfium-sys/src/dynamic.rs b/crates/pdfium-sys/src/dynamic.rs
index c5dbb15..cb87bda 100644
--- a/crates/pdfium-sys/src/dynamic.rs
+++ b/crates/pdfium-sys/src/dynamic.rs
@@ -262,6 +262,19 @@ pub struct PdfiumBindings {
pub FPDFBookmark_GetDest: unsafe extern "C" fn(FPDF_DOCUMENT, FPDF_BOOKMARK) -> FPDF_DEST,
pub FPDFBookmark_GetAction: unsafe extern "C" fn(FPDF_BOOKMARK) -> FPDF_ACTION,
pub FPDFAction_GetDest: unsafe extern "C" fn(FPDF_DOCUMENT, FPDF_ACTION) -> FPDF_DEST,
+ pub FPDFAction_GetURIPath: unsafe extern "C" fn(
+ FPDF_DOCUMENT,
+ FPDF_ACTION,
+ *mut std::os::raw::c_void,
+ std::os::raw::c_ulong,
+ ) -> std::os::raw::c_ulong,
+ pub FPDFLink_Enumerate:
+ unsafe extern "C" fn(FPDF_PAGE, *mut std::os::raw::c_int, *mut FPDF_LINK) -> FPDF_BOOL,
+ pub FPDFLink_GetAction: unsafe extern "C" fn(FPDF_LINK) -> FPDF_ACTION,
+ pub FPDFLink_GetAnnotRect: unsafe extern "C" fn(FPDF_LINK, *mut FS_RECTF) -> FPDF_BOOL,
+ pub FPDFLink_CountQuadPoints: unsafe extern "C" fn(FPDF_LINK) -> std::os::raw::c_int,
+ pub FPDFLink_GetQuadPoints:
+ unsafe extern "C" fn(FPDF_LINK, std::os::raw::c_int, *mut FS_QUADPOINTSF) -> FPDF_BOOL,
pub FPDFDest_GetDestPageIndex:
unsafe extern "C" fn(FPDF_DOCUMENT, FPDF_DEST) -> std::os::raw::c_int,
pub FPDFDest_GetLocationInPage: unsafe extern "C" fn(
@@ -399,6 +412,12 @@ impl PdfiumBindings {
FPDFBookmark_GetDest: load_fn!(lib, "FPDFBookmark_GetDest"),
FPDFBookmark_GetAction: load_fn!(lib, "FPDFBookmark_GetAction"),
FPDFAction_GetDest: load_fn!(lib, "FPDFAction_GetDest"),
+ FPDFAction_GetURIPath: load_fn!(lib, "FPDFAction_GetURIPath"),
+ FPDFLink_Enumerate: load_fn!(lib, "FPDFLink_Enumerate"),
+ FPDFLink_GetAction: load_fn!(lib, "FPDFLink_GetAction"),
+ FPDFLink_GetAnnotRect: load_fn!(lib, "FPDFLink_GetAnnotRect"),
+ FPDFLink_CountQuadPoints: load_fn!(lib, "FPDFLink_CountQuadPoints"),
+ FPDFLink_GetQuadPoints: load_fn!(lib, "FPDFLink_GetQuadPoints"),
FPDFDest_GetDestPageIndex: load_fn!(lib, "FPDFDest_GetDestPageIndex"),
FPDFDest_GetLocationInPage: load_fn!(lib, "FPDFDest_GetLocationInPage"),
diff --git a/crates/pdfium/src/lib.rs b/crates/pdfium/src/lib.rs
index 59ec49e..f357b3c 100644
--- a/crates/pdfium/src/lib.rs
+++ b/crates/pdfium/src/lib.rs
@@ -13,7 +13,9 @@ pub use document::{Document, OutlineEntry};
pub use error::PdfiumError;
pub use font::{Font, FontType};
pub use library::Library;
-pub use page::{ImageBounds, Page, PathObject, PathSegment, SegmentKind, ViewportTransform};
+pub use page::{
+ ImageBounds, Page, PathObject, PathSegment, PdfLink, SegmentKind, ViewportTransform,
+};
pub use struct_tree::StructNode;
pub use text_page::{TextChar, TextCharIter, TextPage};
pub use types::*;
diff --git a/crates/pdfium/src/page.rs b/crates/pdfium/src/page.rs
index 1d8d915..f6db71b 100644
--- a/crates/pdfium/src/page.rs
+++ b/crates/pdfium/src/page.rs
@@ -51,6 +51,16 @@ pub struct PathObject {
pub segments: Vec<PathSegment>,
}
+/// A URI hyperlink annotation on a page. `rect` is in viewport space
+/// (top-left origin, 72 DPI), matching `TextItem` coordinates so the URI can
+/// be assigned to overlapping text. Only external URI links are represented;
+/// internal GoTo/named destinations are excluded.
+#[derive(Debug, Clone)]
+pub struct PdfLink {
+ pub rect: RectF,
+ pub uri: String,
+}
+
/// A loaded page within a [`Document`].
///
/// The `'doc` lifetime ties the page to its owning document; `'lib` carries
@@ -346,6 +356,121 @@ impl<'doc, 'lib: 'doc> Page<'doc, 'lib> {
}
out
}
+
+ /// Enumerate URI hyperlink annotations on this page. Each link's clickable
+ /// rectangle is mapped into viewport space (matching `TextItem`); the URI
+ /// is read from the link's URI action. Annotations whose action is not a
+ /// URI (internal GoTo / named destinations) are skipped.
+ pub fn links(&self, view_box: &RectF) -> Vec<PdfLink> {
+ let mut out = Vec::new();
+ let mut start_pos: std::os::raw::c_int = 0;
+ let mut link_annot: pdfium_sys::FPDF_LINK = std::ptr::null_mut();
+ loop {
+ let ok = unsafe {
+ ffi!(FPDFLink_Enumerate(
+ self.handle,
+ &mut start_pos,
+ &mut link_annot
+ ))
+ };
+ if ok == 0 {
+ break;
+ }
+ if link_annot.is_null() {
+ continue;
+ }
+ let action = unsafe { ffi!(FPDFLink_GetAction(link_annot)) };
+ if action.is_null() {
+ continue;
+ }
+ let Some(uri) = read_uri_path(self.doc_handle, action) else {
+ continue;
+ };
+
+ // Prefer per-line quad points: a link that wraps across lines has
+ // one quad per line, each tight around the anchor text. The single
+ // annotation rect is their *union* — a tall box that would swallow
+ // the unlinked words sitting between the lines. Fall back to the
+ // annot rect only when no quads are present.
+ let quad_count = unsafe { ffi!(FPDFLink_CountQuadPoints(link_annot)) };
+ let mut emitted = false;
+ for q in 0..quad_count {
+ let mut quad = pdfium_sys::FS_QUADPOINTSF::default();
+ let ok = unsafe { ffi!(FPDFLink_GetQuadPoints(link_annot, q, &mut quad)) };
+ if ok == 0 {
+ continue;
+ }
+ let page_bounds = RectF {
+ left: quad.x1.min(quad.x2).min(quad.x3).min(quad.x4),
+ bottom: quad.y1.min(quad.y2).min(quad.y3).min(quad.y4),
+ right: quad.x1.max(quad.x2).max(quad.x3).max(quad.x4),
+ top: quad.y1.max(quad.y2).max(quad.y3).max(quad.y4),
+ };
+ out.push(PdfLink {
+ rect: self.bounds_to_viewport(view_box, &page_bounds),
+ uri: uri.clone(),
+ });
+ emitted = true;
+ }
+ if emitted {
+ continue;
+ }
+
+ let mut rect = pdfium_sys::FS_RECTF {
+ left: 0.0,
+ top: 0.0,
+ right: 0.0,
+ bottom: 0.0,
+ };
+ let got = unsafe { ffi!(FPDFLink_GetAnnotRect(link_annot, &mut rect)) };
+ if got == 0 {
+ continue;
+ }
+ let page_bounds = RectF {
+ left: rect.left,
+ top: rect.top,
+ right: rect.right,
+ bottom: rect.bottom,
+ };
+ out.push(PdfLink {
+ rect: self.bounds_to_viewport(view_box, &page_bounds),
+ uri,
+ });
+ }
+ out
+ }
+}
+
+/// Read a link action's URI path. PDFium returns the URI as a NUL-terminated
+/// 7-bit-ASCII byte string; the two-call protocol queries the length first.
+/// Returns `None` for non-URI actions (length 0) or empty URIs.
+fn read_uri_path(
+ doc: pdfium_sys::FPDF_DOCUMENT,
+ action: pdfium_sys::FPDF_ACTION,
+) -> Option<String> {
+ let needed =
+ unsafe { ffi!(FPDFAction_GetURIPath(doc, action, std::ptr::null_mut(), 0)) } as usize;
+ if needed < 2 {
+ return None;
+ }
+ let mut buf: Vec<u8> = vec![0; needed];
+ let written = unsafe {
+ ffi!(FPDFAction_GetURIPath(
+ doc,
+ action,
+ buf.as_mut_ptr() as *mut std::os::raw::c_void,
+ needed as std::os::raw::c_ulong,
+ ))
+ } as usize;
+ if written < 2 {
+ return None;
+ }
+ // `written` includes the trailing NUL.
+ let end = written.saturating_sub(1).min(buf.len());
+ let uri = String::from_utf8_lossy(&buf[..end])
+ .trim_matches(char::from(0))
+ .to_string();
+ if uri.is_empty() { None } else { Some(uri) }
}
const FS_IDENTITY: pdfium_sys::FS_MATRIX = pdfium_sys::FS_MATRIX {
diff --git a/packages/python/liteparse/parser.py b/packages/python/liteparse/parser.py
index 9fe4317..895030e 100644
--- a/packages/python/liteparse/parser.py
+++ b/packages/python/liteparse/parser.py
@@ -88,6 +88,7 @@ class LiteParse:
quiet: Optional[bool] = None,
num_workers: Optional[int] = None,
image_mode: Optional[str] = None,
+ extract_links: Optional[bool] = None,
):
"""
Initialize LiteParse parser.
@@ -105,6 +106,8 @@ class LiteParse:
password: Password for encrypted/protected documents
quiet: Suppress progress output
num_workers: Number of concurrent OCR workers (default: CPU cores - 1)
+ extract_links: Render hyperlink annotations as ``[text](url)`` in
+ markdown output (default: True). Set False for plain anchor text.
"""
kwargs = {}
if ocr_enabled is not None:
@@ -133,6 +136,8 @@ class LiteParse:
kwargs["num_workers"] = num_workers
if image_mode is not None:
kwargs["image_mode"] = image_mode
+ if extract_links is not None:
+ kwargs["extract_links"] = extract_links
self._native = _NativeLiteParse(**kwargs)