A fast, helpful, and open-source document parser
wip
Logan Markewich2efc2ce4963e
diff --git a/crates/liteparse-docx/ATTRIBUTION.md b/crates/liteparse-docx/ATTRIBUTION.mdnew file mode 100644index 0000000..396762f--- /dev/null+++ b/crates/liteparse-docx/ATTRIBUTION.md@@ -0,0 +1,135 @@+# Attribution++This crate is a vendored subset of **[dxpdf](https://github.com/nerdy-pro/dxpdf)+v0.4.0** by nerdy.pro, used under the MIT License (see `LICENSE`).++Upstream is a DOCX→PDF engine (`parse → resolve → layout → subset → paint`)+built on Skia. We vendor only the first two stages, which contain no Skia+references, so the result is pure Rust and builds on musl (rust-skia ships no+musl prebuilts, and `skia-safe` on musl falls back to a ~1hr from-source build).++## What was copied verbatim++```+src/docx/ src/model/+src/field/ src/render/resolve/+src/error.rs+src/render/{dimension,geometry,error}.rs+src/render/layout/draw_command.rs (referenced by resolve::shape_visuals)+src/render/emoji/cluster.rs (referenced by draw_command)+```++## What was changed++1. `src/render/emf.rs` — **dropped**. The subset's only genuine `skia_safe`+ import; converts Windows metafiles for the painter, unreachable from the+ structure path.+2. `src/render/fonts.rs` — **replaced with a shim.** Upstream is a Skia+ `FontRegistry`; only `TypefaceEntry` is named here (one field of+ `DrawCommand::Text`). This is the seam for a future harfrust+skrifa port.+3. `src/lib.rs`, `src/render/mod.rs`, `src/render/layout/mod.rs`,+ `src/render/emoji/mod.rs` — **rewritten** to declare only the copied modules.+4. Edition-2024 pattern fixes (`ref mut` in an implicit borrow) in+ `render/resolve/properties.rs` and `docx/parse/rel_rewrite.rs` — upstream+ pins an older toolchain.+5. Unknown-element tolerance — see below.++## Fail-open parsing (the main deliberate divergence)++**Upstream fails closed at every level. We fail open.** A single malformed+value anywhere in a file must never cost the whole document — that is a+standing requirement of this copy, not a one-off fix. Three distinct classes,+all of which upstream treats as fatal:++| class | upstream | here |+|---|---|---|+| **unknown elements** (`commentReference`, …) | aborts document | `#[serde(other)]` catch-all, element skipped |+| **unknown attribute *values*** (`w:jc val="bogus"`) | aborts document | `lenient::` deserializers → unspecified |+| **malformed scalars** (colours, measurements, ids) | aborts document | dropped / spec default |++The mechanism lives in `docx/parse/primitives/lenient.rs`:++- `opt_attr` — `Option<T>` attribute → `None`. ECMA-376 §17.17 says an invalid+ value is treated as absent, and absent means *inherit from the style chain*,+ so this is both the spec-correct and the least-destructive degradation.+- `opt_val_attr` — same for `<w:x w:val="…"/>` elements; collapses the+ `ValAttr<T>` wrapper so the field is plain `Option<T>`.+- `or_default` — required attributes whose type has a *spec* default.+- `nonneg_or_default` — required non-negative measurements → zero, keeping the+ non-negativity guard.++Two rules that are easy to get wrong and were both hit during the port:++1. **Never invent a value to keep an infallible `From`.** Where the model has+ no "unspecified" variant (colours, adjust handles, gradient stops) the+ conversion was made fallible and callers drop the item. Substituting black+ or zero would be silent corruption.+2. **`AttrValueDeserializer` must coerce, not just visit strings.** serde's own+ string deserializer refuses to produce an integer, so routing numeric+ attributes through it would silently turn every *valid* `gridSpan`, `numId`,+ `ilvl` and `outlineLvl` into `None` — corrupting documents rather than+ merely being strict. Its unit tests guard exactly this.++Upstream tests that asserted the strict behaviour were rewritten to assert the+new contract *plus* the guarantee they originally protected (e.g. `"1.0"` is+dropped as a numbering id, but must still never be coerced to `1`).++### Verifying++```+cargo test -p liteparse-docx+cargo build -p liteparse-docx --example parse_probe+python3 bench/docx_native_spike/fuzz_attribute_values.py --mode all+```++The fuzz harness corrupts every attribute value in the `docx_files` corpus+(623k values) and requires all 48 documents to still parse.++## How much of this copy is actually used (measured 2026-08-05)++The headline "37k LOC" overstates it, and the answer changes once layout lands,+so this is a snapshot with a reproducible recipe — not a standing fact.++| | |+|---|---|+| total | 38,098 |+| dxpdf's own `#[cfg(test)]` blocks | 13,643 |+| **production LOC** | **24,455** |++**Statically reachable** from the two entry points this crate exists to provide+(`docx::parse`, `render::resolve::resolve`): 156 items are not, and they form+two islands — `field/` (2,340 L), which nothing outside `field/` references at+all, and the `layout/draw_command.rs` cluster (+ `resolve::shape_geometry`,+`shape_visuals`, `drawing_color`, `emoji::cluster`; 3,960 L), which is reachable+only from itself. Note `dead_code` cannot see the rest: a serde schema struct+counts as "used" merely by being deserialized into, whether or not any field is+ever read.++**Actually executed** over the 48-doc `docx_files` corpus — 47.3% of+instrumented lines. The core parse path is genuinely exercised+(`docx/parse/*` 86.6%, `properties` 88.6%); `render/resolve` is 24.7%,+`docx/parse/drawing` 36.4%, `field/` 16.3%, and these are flat zero:+`emoji`, `render/{geometry,dimension}`, `layout/draw_command`,+`resolve/{shape_geometry,shape_visuals,drawing_color,conditional,locale,images,header_footer,color}`.++Three buckets, and only the first is a trim candidate:++1. **Painting-only, dead for us permanently (~4-6k)** — preset shape geometry,+ shape fills/strokes/effects, DrawingML colour transforms, emoji clusters,+ VML path commands/formulas.+2. **Dead today, live under C′ (~6k)** — `draw_command` *is* the geometry tap+ (`Text{position}` / `LinkAnnotation` / `Image`), `render/{dimension,geometry}`+ are its Pt types, and `field/` is how `PAGE`/`REF` become text, which matters+ as soon as headers/footers paginate.+3. **Low coverage but load-bearing** — `parse/drawing` and `parse/vml` schemas+ are what make the parser fail *open* on drawings, and are the raw material+ for the two known content gaps (textbox content, image rects).++**Decision: do not trim.** The trimmable set is ~4-6k of 24k, and deleting it+forfeits the clean re-sync this file exists to protect. Re-measure after the+layout vendor, when bucket 2 is a measured fact rather than a projection.++Recipe: `bench/docx_vendor_coverage.sh`.++Keeping this list current matters: it is what makes a future re-sync against a+newer dxpdf tractable.diff --git a/crates/liteparse-ooxml/src/render/layout/build/convert.rs b/crates/liteparse-ooxml/src/render/layout/build/convert.rsindex 6b751fa..b95542a 100644--- a/crates/liteparse-ooxml/src/render/layout/build/convert.rs+++ b/crates/liteparse-ooxml/src/render/layout/build/convert.rs@@ -257,9 +257,18 @@ pub(super) fn paragraph_style_from_props(}).unwrap_or(Pt::ZERO);- // §17.3.1.33: when autoSpacing is true, use 14pt instead of explicit value.- let space_before = if props.spacing.and_then(|s| s.before_auto_spacing) == Some(true) {+ // §17.3.1.33: when autoSpacing is true, the explicit value is ignored and+ // spacing is consumer-defined. Word uses 14pt for ordinary paragraphs but+ // 0 for paragraphs carrying list numbering (`w:numPr` with a live numId) —+ // observable in HTML-paste documents where consecutive `NormalWeb` list+ // items sit at plain line pitch. LibreOffice emulates the same rule.+ let auto_space = if props.numbering.as_ref().is_some_and(|n| n.num_id != 0) {+ Pt::ZERO+ } else {Pt::new(14.0)+ };+ let space_before = if props.spacing.and_then(|s| s.before_auto_spacing) == Some(true) {+ auto_space} else {props.spacing@@ -268,7 +277,7 @@ pub(super) fn paragraph_style_from_props(.unwrap_or(Pt::ZERO)};let space_after = if props.spacing.and_then(|s| s.after_auto_spacing) == Some(true) {- Pt::new(14.0)+ auto_space} else {props.spacingdiff --git a/crates/liteparse-ooxml/src/render/layout/draw_command.rs b/crates/liteparse-ooxml/src/render/layout/draw_command.rsindex 8cac5e2..ed32004 100644--- a/crates/liteparse-ooxml/src/render/layout/draw_command.rs+++ b/crates/liteparse-ooxml/src/render/layout/draw_command.rs@@ -495,6 +495,16 @@ pub struct LayoutedPage {/// `Continuous`-section continuation for free. Blocks that emit no content/// (empty paragraphs) appear on no page; consumers fill forward.pub block_starts: Vec<usize>,+ /// liteparse instrumentation (no upstream equivalent): the header blocks+ /// the §17.10.5/§17.10.6 slot selection picked for this page — the same+ /// choice the paint pass makes in `render_headers_footers`. Populated in+ /// the deferred header/footer phase, after pagination is final, so it+ /// needs no checkpoint/replay handling. Empty when the page's section has+ /// no applicable slot (e.g. a `titlePg` first page without a `first`+ /// header).+ pub header_blocks: Vec<crate::model::Block>,+ /// Footer counterpart of [`LayoutedPage::header_blocks`].+ pub footer_blocks: Vec<crate::model::Block>,}impl LayoutedPage {@@ -503,6 +513,8 @@ impl LayoutedPage {commands: Vec::new(),page_size,block_starts: Vec::new(),+ header_blocks: Vec::new(),+ footer_blocks: Vec::new(),}}}diff --git a/crates/liteparse-ooxml/src/render/layout/header_footer.rs b/crates/liteparse-ooxml/src/render/layout/header_footer.rsindex 959ee1a..3608bdb 100644--- a/crates/liteparse-ooxml/src/render/layout/header_footer.rs+++ b/crates/liteparse-ooxml/src/render/layout/header_footer.rs@@ -245,6 +245,9 @@ pub fn render_headers_footers();if let Some(blocks) = header_blocks {+ // Side-channel for consumers that emit markdown/text: record the+ // selected slot's blocks on the page (see `LayoutedPage::header_blocks`).+ page.header_blocks = blocks.clone();// Set per-page field context for PAGE/NUMPAGES evaluation.state.field_ctx = crate::render::layout::fragment::FieldContext {page_number: Some(logical_page_number),@@ -263,6 +266,8 @@ pub fn render_headers_footers(}if let Some(blocks) = footer_blocks {+ // Side-channel counterpart of `header_blocks` above.+ page.footer_blocks = blocks.clone();state.field_ctx = crate::render::layout::fragment::FieldContext {page_number: Some(logical_page_number),num_pages: Some(page_range.total_pages),diff --git a/crates/liteparse-ooxml/src/render/layout/section/layout.rs b/crates/liteparse-ooxml/src/render/layout/section/layout.rsindex 90c2a45..a7ecd49 100644--- a/crates/liteparse-ooxml/src/render/layout/section/layout.rs+++ b/crates/liteparse-ooxml/src/render/layout/section/layout.rs@@ -1054,9 +1054,13 @@ fn emit_split_paragraph<'doc>(// §17.6.4: a fresh column offers full height, like a fresh page.let at_page_top = state.cursor_y <= state.column_top;let avail = (state.bottom - state.cursor_y).max(Pt::ZERO);- // §17.3.1.24/§17.3.1.33: space_after and the bottom border space are- // only spent once, on the segment carrying the paragraph's last line.- let trailing_extra = cont_style.space_after + placed.bottom_border_space();+ // §17.3.1.24/§17.3.1.33: the bottom border space is only spent once,+ // on the segment carrying the paragraph's last line. `space_after` is+ // deliberately NOT charged to the fit test: Word discards trailing+ // blank space at the page bottom instead of pushing the last line to+ // the next page. The cursor still advances by the full height, so a+ // following paragraph that no longer fits breaks the page as before.+ let trailing_extra = placed.bottom_border_space();// Count how many of this placement's lines fit, charging space_before// (first segment only, §17.3.1.33) and the trailing spacing on the last.@@ -1738,7 +1742,11 @@ pub(crate) fn layout_section_with_clearance(} else {let mut para = placed.emit_full();// Column/page overflow: advance column, then page.- if state.cursor_y + para.size.height > state.bottom+ // §17.3.1.33: `size.height` includes trailing+ // `space_after`, but Word discards that blank space+ // at the page bottom — only the ink has to fit.+ let fit_height = para.size.height - effective_style.space_after;+ if state.cursor_y + fit_height > state.bottom&& state.cursor_y > state.column_top{// `placed` (which borrows `effective_style`) is nodiff --git a/crates/liteparse-ooxml/src/render/raster.rs b/crates/liteparse-ooxml/src/render/raster.rsindex 9bfd0e7..09dfb18 100644--- a/crates/liteparse-ooxml/src/render/raster.rs+++ b/crates/liteparse-ooxml/src/render/raster.rs@@ -970,6 +970,8 @@ mod tests {],page_size: PtSize::new(Pt::new(100.0), Pt::new(100.0)),block_starts: Vec::new(),+ header_blocks: Vec::new(),+ footer_blocks: Vec::new(),}}@@ -1094,6 +1096,8 @@ mod tests {],page_size: PtSize::new(Pt::new(100.0), Pt::new(100.0)),block_starts: Vec::new(),+ header_blocks: Vec::new(),+ footer_blocks: Vec::new(),}}@@ -1157,6 +1161,8 @@ mod tests {commands: Vec::new(),page_size: PtSize::new(Pt::new(w), Pt::new(h)),block_starts: Vec::new(),+ header_blocks: Vec::new(),+ footer_blocks: Vec::new(),};let r = rasterize_page(&page, &FontRegistry::new(), 150.0 / 72.0).expect("raster");assert!(r.scale < 150.0 / 72.0, "this page must have been clamped");diff --git a/crates/liteparse/src/office/docx.rs b/crates/liteparse/src/office/docx.rsindex 85da6c1..de0ce2a 100644--- a/crates/liteparse/src/office/docx.rs+++ b/crates/liteparse/src/office/docx.rs@@ -72,6 +72,30 @@ pub fn emit(doc: &ResolvedDocument, extract_links: bool) -> Vec<Block> {.collect()}+/// Emit an out-of-body block list — a page's header or footer content, as+/// selected per page by the layout (`LayoutedPage::header_blocks` /+/// `footer_blocks`) — as markdown blocks.+///+/// A fresh `Emitter` per call: header/footer content must not continue body+/// list numbering, and vice versa. Structure-only (no figures): a header+/// image ref would collide with the layout's page-scoped figure ids.+pub fn emit_header_footer(+ doc: &ResolvedDocument,+ blocks: &[liteparse_ooxml::model::Block],+ extract_links: bool,+) -> Vec<Block> {+ let mut em = Emitter::new(+ doc,+ EmitOptions {+ links: extract_links,+ figures: None,+ },+ );+ let mut out = Vec::new();+ em.blocks(blocks, &mut out);+ out+}+/// Where an emitted [`Block`] came from, for the layout block→page join.#[derive(Clone, Copy, Debug, PartialEq, Eq)]pub enum BlockSource {@@ -648,6 +672,18 @@ impl<'a> Emitter<'a> {// A figure inside a cell can't live in single-line cell text;// requeue it so the enclosing `table` emits it afterwards.Block::Figure { id, format } => self.pending_figures.push((id, format)),+ // A nested table can't render as markup inside a single-line+ // cell; flatten its cells in reading order so the content+ // survives rather than being dropped.+ Block::MergedTable { rows, .. } => {+ for row in rows {+ for cell in row {+ if !cell.text.is_empty() {+ parts.push(cell.text);+ }+ }+ }+ }_ => {}}}diff --git a/crates/liteparse/src/office/docx_layout.rs b/crates/liteparse/src/office/docx_layout.rsindex 7d26f6b..465c12a 100644--- a/crates/liteparse/src/office/docx_layout.rs+++ b/crates/liteparse/src/office/docx_layout.rs@@ -682,6 +682,8 @@ mod tests {commands,page_size: PtSize::new(Pt::new(612.0), Pt::new(792.0)),block_starts: Vec::new(),+ header_blocks: Vec::new(),+ footer_blocks: Vec::new(),}}diff --git a/crates/liteparse/src/office/pptx_layout.rs b/crates/liteparse/src/office/pptx_layout.rsindex f902049..afe3304 100644--- a/crates/liteparse/src/office/pptx_layout.rs+++ b/crates/liteparse/src/office/pptx_layout.rs@@ -484,9 +484,12 @@ fn layout_deck(pkg: &PresentationPackage, registry: &FontRegistry) -> SlideGeomeout.layouts.push(LayoutedPage {commands,page_size,- // A DOCX-only side-channel: it indexes the flattened body blocks a- // page starts, and a slide is one page with no such flattening.+ // DOCX-only side-channels: `block_starts` indexes the flattened+ // body blocks a page starts (a slide is one page with no such+ // flattening), and slides have no OOXML headers/footers.block_starts: Vec::new(),+ header_blocks: Vec::new(),+ footer_blocks: Vec::new(),});let content_bounds = union_bounds(&items);out.pages.push(Page {@@ -1156,6 +1159,8 @@ fn commands_to_items(commands,page_size: extent,block_starts: Vec::new(),+ header_blocks: Vec::new(),+ footer_blocks: Vec::new(),};let converted = docx_layout::layout_to_pages(&[page], ctx.registry, false, false);converteddiff --git a/crates/liteparse/src/parser.rs b/crates/liteparse/src/parser.rsindex d95c012..0fd6279 100644--- a/crates/liteparse/src/parser.rs+++ b/crates/liteparse/src/parser.rs@@ -1228,10 +1228,37 @@ impl LiteParse {},);let n_pages = native.pages.len();- let page_blocks = split_blocks_by_page(&tagged, &native.block_pages, n_pages);+ let mut page_blocks = split_blocks_by_page(&tagged, &native.block_pages, n_pages);let all_blocks: Vec<crate::markdown_layout::Block> =tagged.into_iter().map(|(b, _)| b).collect();+ // Headers/footers: the layout records each page's selected slot+ // (`LayoutedPage::header_blocks`/`footer_blocks`; the raster and+ // `text_items` always carry them). Markdown mirrors the PDF path's+ // default of suppressing running chrome, so the blocks join the+ // markdown stream only under `keep_headers_footers` — header at the+ // page top, footer at the page bottom. The doc-level text then has to+ // be assembled per page (a whole-document block walk has no page+ // boundaries to repeat them at), which `all_blocks: None` selects.+ let keep_hf = self.config.keep_headers_footers;+ if keep_hf {+ for (i, blocks) in page_blocks.iter_mut().enumerate() {+ let Some(lp) = layouted.get(i) else { break };+ let header = docx::emit_header_footer(+ &resolved,+ &lp.header_blocks,+ self.config.extract_links,+ );+ let footer = docx::emit_header_footer(+ &resolved,+ &lp.footer_blocks,+ self.config.extract_links,+ );+ blocks.splice(0..0, header);+ blocks.extend(footer);+ }+ }+// Per-page complexity from native facts: placed-media rects, the// page's table blocks (exact, not detected), and the section-declared// column count — see `calculate_native_page_complexity` for the@@ -1278,7 +1305,7 @@ impl LiteParse {let mut result = self.parse_from_native_blocks(pages,page_blocks,- (!page_filtered).then_some(all_blocks),+ (!page_filtered && !keep_hf).then_some(all_blocks),native.outline,images,page_stats,