Skip to content

A fast, helpful, and open-source document parser

wip

Logan Markewich2efc2ce4963e
diff --git a/crates/liteparse-docx/ATTRIBUTION.md b/crates/liteparse-docx/ATTRIBUTION.md
new file mode 100644
index 0000000..396762f
--- /dev/null
+++ b/crates/liteparse-docx/ATTRIBUTION.md
@@ -0,0 +1,135 @@
+# Attribution
+
+This crate is a vendored subset of **[dxpdf](https://github.com/nerdy-pro/dxpdf)
+v0.4.0** by nerdy.pro, used under the MIT License (see `LICENSE`).
+
+Upstream is a DOCX→PDF engine (`parse → resolve → layout → subset → paint`)
+built on Skia. We vendor only the first two stages, which contain no Skia
+references, so the result is pure Rust and builds on musl (rust-skia ships no
+musl prebuilts, and `skia-safe` on musl falls back to a ~1hr from-source build).
+
+## What was copied verbatim
+
+```
+src/docx/ src/model/
+src/field/ src/render/resolve/
+src/error.rs
+src/render/{dimension,geometry,error}.rs
+src/render/layout/draw_command.rs (referenced by resolve::shape_visuals)
+src/render/emoji/cluster.rs (referenced by draw_command)
+```
+
+## What was changed
+
+1. `src/render/emf.rs` — **dropped**. The subset's only genuine `skia_safe`
+ import; converts Windows metafiles for the painter, unreachable from the
+ structure path.
+2. `src/render/fonts.rs` — **replaced with a shim.** Upstream is a Skia
+ `FontRegistry`; only `TypefaceEntry` is named here (one field of
+ `DrawCommand::Text`). This is the seam for a future harfrust+skrifa port.
+3. `src/lib.rs`, `src/render/mod.rs`, `src/render/layout/mod.rs`,
+ `src/render/emoji/mod.rs` — **rewritten** to declare only the copied modules.
+4. Edition-2024 pattern fixes (`ref mut` in an implicit borrow) in
+ `render/resolve/properties.rs` and `docx/parse/rel_rewrite.rs` — upstream
+ pins an older toolchain.
+5. Unknown-element tolerance — see below.
+
+## Fail-open parsing (the main deliberate divergence)
+
+**Upstream fails closed at every level. We fail open.** A single malformed
+value anywhere in a file must never cost the whole document — that is a
+standing requirement of this copy, not a one-off fix. Three distinct classes,
+all of which upstream treats as fatal:
+
+| class | upstream | here |
+|---|---|---|
+| **unknown elements** (`commentReference`, …) | aborts document | `#[serde(other)]` catch-all, element skipped |
+| **unknown attribute *values*** (`w:jc val="bogus"`) | aborts document | `lenient::` deserializers → unspecified |
+| **malformed scalars** (colours, measurements, ids) | aborts document | dropped / spec default |
+
+The mechanism lives in `docx/parse/primitives/lenient.rs`:
+
+- `opt_attr` — `Option<T>` attribute → `None`. ECMA-376 §17.17 says an invalid
+ value is treated as absent, and absent means *inherit from the style chain*,
+ so this is both the spec-correct and the least-destructive degradation.
+- `opt_val_attr` — same for `<w:x w:val="…"/>` elements; collapses the
+ `ValAttr<T>` wrapper so the field is plain `Option<T>`.
+- `or_default` — required attributes whose type has a *spec* default.
+- `nonneg_or_default` — required non-negative measurements → zero, keeping the
+ non-negativity guard.
+
+Two rules that are easy to get wrong and were both hit during the port:
+
+1. **Never invent a value to keep an infallible `From`.** Where the model has
+ no "unspecified" variant (colours, adjust handles, gradient stops) the
+ conversion was made fallible and callers drop the item. Substituting black
+ or zero would be silent corruption.
+2. **`AttrValueDeserializer` must coerce, not just visit strings.** serde's own
+ string deserializer refuses to produce an integer, so routing numeric
+ attributes through it would silently turn every *valid* `gridSpan`, `numId`,
+ `ilvl` and `outlineLvl` into `None` — corrupting documents rather than
+ merely being strict. Its unit tests guard exactly this.
+
+Upstream tests that asserted the strict behaviour were rewritten to assert the
+new contract *plus* the guarantee they originally protected (e.g. `"1.0"` is
+dropped as a numbering id, but must still never be coerced to `1`).
+
+### Verifying
+
+```
+cargo test -p liteparse-docx
+cargo build -p liteparse-docx --example parse_probe
+python3 bench/docx_native_spike/fuzz_attribute_values.py --mode all
+```
+
+The fuzz harness corrupts every attribute value in the `docx_files` corpus
+(623k values) and requires all 48 documents to still parse.
+
+## How much of this copy is actually used (measured 2026-08-05)
+
+The headline "37k LOC" overstates it, and the answer changes once layout lands,
+so this is a snapshot with a reproducible recipe — not a standing fact.
+
+| | |
+|---|---|
+| total | 38,098 |
+| dxpdf's own `#[cfg(test)]` blocks | 13,643 |
+| **production LOC** | **24,455** |
+
+**Statically reachable** from the two entry points this crate exists to provide
+(`docx::parse`, `render::resolve::resolve`): 156 items are not, and they form
+two islands — `field/` (2,340 L), which nothing outside `field/` references at
+all, and the `layout/draw_command.rs` cluster (+ `resolve::shape_geometry`,
+`shape_visuals`, `drawing_color`, `emoji::cluster`; 3,960 L), which is reachable
+only from itself. Note `dead_code` cannot see the rest: a serde schema struct
+counts as "used" merely by being deserialized into, whether or not any field is
+ever read.
+
+**Actually executed** over the 48-doc `docx_files` corpus — 47.3% of
+instrumented lines. The core parse path is genuinely exercised
+(`docx/parse/*` 86.6%, `properties` 88.6%); `render/resolve` is 24.7%,
+`docx/parse/drawing` 36.4%, `field/` 16.3%, and these are flat zero:
+`emoji`, `render/{geometry,dimension}`, `layout/draw_command`,
+`resolve/{shape_geometry,shape_visuals,drawing_color,conditional,locale,images,header_footer,color}`.
+
+Three buckets, and only the first is a trim candidate:
+
+1. **Painting-only, dead for us permanently (~4-6k)** — preset shape geometry,
+ shape fills/strokes/effects, DrawingML colour transforms, emoji clusters,
+ VML path commands/formulas.
+2. **Dead today, live under C′ (~6k)** — `draw_command` *is* the geometry tap
+ (`Text{position}` / `LinkAnnotation` / `Image`), `render/{dimension,geometry}`
+ are its Pt types, and `field/` is how `PAGE`/`REF` become text, which matters
+ as soon as headers/footers paginate.
+3. **Low coverage but load-bearing** — `parse/drawing` and `parse/vml` schemas
+ are what make the parser fail *open* on drawings, and are the raw material
+ for the two known content gaps (textbox content, image rects).
+
+**Decision: do not trim.** The trimmable set is ~4-6k of 24k, and deleting it
+forfeits the clean re-sync this file exists to protect. Re-measure after the
+layout vendor, when bucket 2 is a measured fact rather than a projection.
+
+Recipe: `bench/docx_vendor_coverage.sh`.
+
+Keeping this list current matters: it is what makes a future re-sync against a
+newer dxpdf tractable.
diff --git a/crates/liteparse-ooxml/src/render/layout/build/convert.rs b/crates/liteparse-ooxml/src/render/layout/build/convert.rs
index 6b751fa..b95542a 100644
--- a/crates/liteparse-ooxml/src/render/layout/build/convert.rs
+++ b/crates/liteparse-ooxml/src/render/layout/build/convert.rs
@@ -257,9 +257,18 @@ pub(super) fn paragraph_style_from_props(
})
.unwrap_or(Pt::ZERO);
- // §17.3.1.33: when autoSpacing is true, use 14pt instead of explicit value.
- let space_before = if props.spacing.and_then(|s| s.before_auto_spacing) == Some(true) {
+ // §17.3.1.33: when autoSpacing is true, the explicit value is ignored and
+ // spacing is consumer-defined. Word uses 14pt for ordinary paragraphs but
+ // 0 for paragraphs carrying list numbering (`w:numPr` with a live numId) —
+ // observable in HTML-paste documents where consecutive `NormalWeb` list
+ // items sit at plain line pitch. LibreOffice emulates the same rule.
+ let auto_space = if props.numbering.as_ref().is_some_and(|n| n.num_id != 0) {
+ Pt::ZERO
+ } else {
Pt::new(14.0)
+ };
+ let space_before = if props.spacing.and_then(|s| s.before_auto_spacing) == Some(true) {
+ auto_space
} else {
props
.spacing
@@ -268,7 +277,7 @@ pub(super) fn paragraph_style_from_props(
.unwrap_or(Pt::ZERO)
};
let space_after = if props.spacing.and_then(|s| s.after_auto_spacing) == Some(true) {
- Pt::new(14.0)
+ auto_space
} else {
props
.spacing
diff --git a/crates/liteparse-ooxml/src/render/layout/draw_command.rs b/crates/liteparse-ooxml/src/render/layout/draw_command.rs
index 8cac5e2..ed32004 100644
--- a/crates/liteparse-ooxml/src/render/layout/draw_command.rs
+++ b/crates/liteparse-ooxml/src/render/layout/draw_command.rs
@@ -495,6 +495,16 @@ pub struct LayoutedPage {
/// `Continuous`-section continuation for free. Blocks that emit no content
/// (empty paragraphs) appear on no page; consumers fill forward.
pub block_starts: Vec<usize>,
+ /// liteparse instrumentation (no upstream equivalent): the header blocks
+ /// the §17.10.5/§17.10.6 slot selection picked for this page — the same
+ /// choice the paint pass makes in `render_headers_footers`. Populated in
+ /// the deferred header/footer phase, after pagination is final, so it
+ /// needs no checkpoint/replay handling. Empty when the page's section has
+ /// no applicable slot (e.g. a `titlePg` first page without a `first`
+ /// header).
+ pub header_blocks: Vec<crate::model::Block>,
+ /// Footer counterpart of [`LayoutedPage::header_blocks`].
+ pub footer_blocks: Vec<crate::model::Block>,
}
impl LayoutedPage {
@@ -503,6 +513,8 @@ impl LayoutedPage {
commands: Vec::new(),
page_size,
block_starts: Vec::new(),
+ header_blocks: Vec::new(),
+ footer_blocks: Vec::new(),
}
}
}
diff --git a/crates/liteparse-ooxml/src/render/layout/header_footer.rs b/crates/liteparse-ooxml/src/render/layout/header_footer.rs
index 959ee1a..3608bdb 100644
--- a/crates/liteparse-ooxml/src/render/layout/header_footer.rs
+++ b/crates/liteparse-ooxml/src/render/layout/header_footer.rs
@@ -245,6 +245,9 @@ pub fn render_headers_footers(
);
if let Some(blocks) = header_blocks {
+ // Side-channel for consumers that emit markdown/text: record the
+ // selected slot's blocks on the page (see `LayoutedPage::header_blocks`).
+ page.header_blocks = blocks.clone();
// Set per-page field context for PAGE/NUMPAGES evaluation.
state.field_ctx = crate::render::layout::fragment::FieldContext {
page_number: Some(logical_page_number),
@@ -263,6 +266,8 @@ pub fn render_headers_footers(
}
if let Some(blocks) = footer_blocks {
+ // Side-channel counterpart of `header_blocks` above.
+ page.footer_blocks = blocks.clone();
state.field_ctx = crate::render::layout::fragment::FieldContext {
page_number: Some(logical_page_number),
num_pages: Some(page_range.total_pages),
diff --git a/crates/liteparse-ooxml/src/render/layout/section/layout.rs b/crates/liteparse-ooxml/src/render/layout/section/layout.rs
index 90c2a45..a7ecd49 100644
--- a/crates/liteparse-ooxml/src/render/layout/section/layout.rs
+++ b/crates/liteparse-ooxml/src/render/layout/section/layout.rs
@@ -1054,9 +1054,13 @@ fn emit_split_paragraph<'doc>(
// §17.6.4: a fresh column offers full height, like a fresh page.
let at_page_top = state.cursor_y <= state.column_top;
let avail = (state.bottom - state.cursor_y).max(Pt::ZERO);
- // §17.3.1.24/§17.3.1.33: space_after and the bottom border space are
- // only spent once, on the segment carrying the paragraph's last line.
- let trailing_extra = cont_style.space_after + placed.bottom_border_space();
+ // §17.3.1.24/§17.3.1.33: the bottom border space is only spent once,
+ // on the segment carrying the paragraph's last line. `space_after` is
+ // deliberately NOT charged to the fit test: Word discards trailing
+ // blank space at the page bottom instead of pushing the last line to
+ // the next page. The cursor still advances by the full height, so a
+ // following paragraph that no longer fits breaks the page as before.
+ let trailing_extra = placed.bottom_border_space();
// Count how many of this placement's lines fit, charging space_before
// (first segment only, §17.3.1.33) and the trailing spacing on the last.
@@ -1738,7 +1742,11 @@ pub(crate) fn layout_section_with_clearance(
} else {
let mut para = placed.emit_full();
// Column/page overflow: advance column, then page.
- if state.cursor_y + para.size.height > state.bottom
+ // §17.3.1.33: `size.height` includes trailing
+ // `space_after`, but Word discards that blank space
+ // at the page bottom — only the ink has to fit.
+ let fit_height = para.size.height - effective_style.space_after;
+ if state.cursor_y + fit_height > state.bottom
&& state.cursor_y > state.column_top
{
// `placed` (which borrows `effective_style`) is no
diff --git a/crates/liteparse-ooxml/src/render/raster.rs b/crates/liteparse-ooxml/src/render/raster.rs
index 9bfd0e7..09dfb18 100644
--- a/crates/liteparse-ooxml/src/render/raster.rs
+++ b/crates/liteparse-ooxml/src/render/raster.rs
@@ -970,6 +970,8 @@ mod tests {
],
page_size: PtSize::new(Pt::new(100.0), Pt::new(100.0)),
block_starts: Vec::new(),
+ header_blocks: Vec::new(),
+ footer_blocks: Vec::new(),
}
}
@@ -1094,6 +1096,8 @@ mod tests {
],
page_size: PtSize::new(Pt::new(100.0), Pt::new(100.0)),
block_starts: Vec::new(),
+ header_blocks: Vec::new(),
+ footer_blocks: Vec::new(),
}
}
@@ -1157,6 +1161,8 @@ mod tests {
commands: Vec::new(),
page_size: PtSize::new(Pt::new(w), Pt::new(h)),
block_starts: Vec::new(),
+ header_blocks: Vec::new(),
+ footer_blocks: Vec::new(),
};
let r = rasterize_page(&page, &FontRegistry::new(), 150.0 / 72.0).expect("raster");
assert!(r.scale < 150.0 / 72.0, "this page must have been clamped");
diff --git a/crates/liteparse/src/office/docx.rs b/crates/liteparse/src/office/docx.rs
index 85da6c1..de0ce2a 100644
--- a/crates/liteparse/src/office/docx.rs
+++ b/crates/liteparse/src/office/docx.rs
@@ -72,6 +72,30 @@ pub fn emit(doc: &ResolvedDocument, extract_links: bool) -> Vec<Block> {
.collect()
}
+/// Emit an out-of-body block list — a page's header or footer content, as
+/// selected per page by the layout (`LayoutedPage::header_blocks` /
+/// `footer_blocks`) — as markdown blocks.
+///
+/// A fresh `Emitter` per call: header/footer content must not continue body
+/// list numbering, and vice versa. Structure-only (no figures): a header
+/// image ref would collide with the layout's page-scoped figure ids.
+pub fn emit_header_footer(
+ doc: &ResolvedDocument,
+ blocks: &[liteparse_ooxml::model::Block],
+ extract_links: bool,
+) -> Vec<Block> {
+ let mut em = Emitter::new(
+ doc,
+ EmitOptions {
+ links: extract_links,
+ figures: None,
+ },
+ );
+ let mut out = Vec::new();
+ em.blocks(blocks, &mut out);
+ out
+}
+
/// Where an emitted [`Block`] came from, for the layout block→page join.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum BlockSource {
@@ -648,6 +672,18 @@ impl<'a> Emitter<'a> {
// A figure inside a cell can't live in single-line cell text;
// requeue it so the enclosing `table` emits it afterwards.
Block::Figure { id, format } => self.pending_figures.push((id, format)),
+ // A nested table can't render as markup inside a single-line
+ // cell; flatten its cells in reading order so the content
+ // survives rather than being dropped.
+ Block::MergedTable { rows, .. } => {
+ for row in rows {
+ for cell in row {
+ if !cell.text.is_empty() {
+ parts.push(cell.text);
+ }
+ }
+ }
+ }
_ => {}
}
}
diff --git a/crates/liteparse/src/office/docx_layout.rs b/crates/liteparse/src/office/docx_layout.rs
index 7d26f6b..465c12a 100644
--- a/crates/liteparse/src/office/docx_layout.rs
+++ b/crates/liteparse/src/office/docx_layout.rs
@@ -682,6 +682,8 @@ mod tests {
commands,
page_size: PtSize::new(Pt::new(612.0), Pt::new(792.0)),
block_starts: Vec::new(),
+ header_blocks: Vec::new(),
+ footer_blocks: Vec::new(),
}
}
diff --git a/crates/liteparse/src/office/pptx_layout.rs b/crates/liteparse/src/office/pptx_layout.rs
index f902049..afe3304 100644
--- a/crates/liteparse/src/office/pptx_layout.rs
+++ b/crates/liteparse/src/office/pptx_layout.rs
@@ -484,9 +484,12 @@ fn layout_deck(pkg: &PresentationPackage, registry: &FontRegistry) -> SlideGeome
out.layouts.push(LayoutedPage {
commands,
page_size,
- // A DOCX-only side-channel: it indexes the flattened body blocks a
- // page starts, and a slide is one page with no such flattening.
+ // DOCX-only side-channels: `block_starts` indexes the flattened
+ // body blocks a page starts (a slide is one page with no such
+ // flattening), and slides have no OOXML headers/footers.
block_starts: Vec::new(),
+ header_blocks: Vec::new(),
+ footer_blocks: Vec::new(),
});
let content_bounds = union_bounds(&items);
out.pages.push(Page {
@@ -1156,6 +1159,8 @@ fn commands_to_items(
commands,
page_size: extent,
block_starts: Vec::new(),
+ header_blocks: Vec::new(),
+ footer_blocks: Vec::new(),
};
let converted = docx_layout::layout_to_pages(&[page], ctx.registry, false, false);
converted
diff --git a/crates/liteparse/src/parser.rs b/crates/liteparse/src/parser.rs
index d95c012..0fd6279 100644
--- a/crates/liteparse/src/parser.rs
+++ b/crates/liteparse/src/parser.rs
@@ -1228,10 +1228,37 @@ impl LiteParse {
},
);
let n_pages = native.pages.len();
- let page_blocks = split_blocks_by_page(&tagged, &native.block_pages, n_pages);
+ let mut page_blocks = split_blocks_by_page(&tagged, &native.block_pages, n_pages);
let all_blocks: Vec<crate::markdown_layout::Block> =
tagged.into_iter().map(|(b, _)| b).collect();
+ // Headers/footers: the layout records each page's selected slot
+ // (`LayoutedPage::header_blocks`/`footer_blocks`; the raster and
+ // `text_items` always carry them). Markdown mirrors the PDF path's
+ // default of suppressing running chrome, so the blocks join the
+ // markdown stream only under `keep_headers_footers` — header at the
+ // page top, footer at the page bottom. The doc-level text then has to
+ // be assembled per page (a whole-document block walk has no page
+ // boundaries to repeat them at), which `all_blocks: None` selects.
+ let keep_hf = self.config.keep_headers_footers;
+ if keep_hf {
+ for (i, blocks) in page_blocks.iter_mut().enumerate() {
+ let Some(lp) = layouted.get(i) else { break };
+ let header = docx::emit_header_footer(
+ &resolved,
+ &lp.header_blocks,
+ self.config.extract_links,
+ );
+ let footer = docx::emit_header_footer(
+ &resolved,
+ &lp.footer_blocks,
+ self.config.extract_links,
+ );
+ blocks.splice(0..0, header);
+ blocks.extend(footer);
+ }
+ }
+
// Per-page complexity from native facts: placed-media rects, the
// page's table blocks (exact, not detected), and the section-declared
// column count — see `calculate_native_page_complexity` for the
@@ -1278,7 +1305,7 @@ impl LiteParse {
let mut result = self.parse_from_native_blocks(
pages,
page_blocks,
- (!page_filtered).then_some(all_blocks),
+ (!page_filtered && !keep_hf).then_some(all_blocks),
native.outline,
images,
page_stats,