A fast, helpful, and open-source document parser
wip
Logan Markewichcfd22a688a63
diff --git a/MARKDOWN_PROGRESS.md b/MARKDOWN_PROGRESS.mdindex bc821d1..38bfca2 100644--- a/MARKDOWN_PROGRESS.md+++ b/MARKDOWN_PROGRESS.md@@ -2,6 +2,12 @@Tracks build-order progress against [MARKDOWN_PLAN.md](MARKDOWN_PLAN.md). Update as steps complete.+> **2026-06-12 code-block language tag — ParseBench `is_code_block` 0.0 → 0.2 (per-doc avg; 0/10 → 5/10 individual tests, all five python cases).** Code blocks were already detected and fenced correctly with intact content, but emitted as bare ` ``` ` with no info-string, so `rules_formatting.py`'s `is_code_block` (which requires a fence whose lang matches the expected language, then checks the snippet is in that block's body) scored a hard 0.0. Added `lang: Option<String>` to `Block::CodeBlock` and a conservative `detect_code_language` (`markdown_layout/classify.rs`) called from `flush_code`: JSON (brace/bracket opener + quoted keys), C/C++ (≥2 of `#include`/`std::`/`int main`/`nullptr`/`->`/`::`/`template<`), Python (≥2 keyword/f-string/`self.`/comprehension signals, or ≥1 plus a colon-terminated control line). Returns `None` (bare fence) when unsure, so docs with other-language tests can't be mis-tagged. Emitter writes the info-string after the fence (` ```python `).+>+> **Win is python-only.** `text_misc__justcode` now emits four ` ```python ` blocks whose bodies contain the rule snippets (whitespace-normalized match tolerates lost indentation) → 5/5 on that doc. The other 5 tests are **detection-layer gaps, out of scope for the tag**: `text_misc__json` / `text_simple__paper` (cpp) never emit code blocks at all because their content isn't flagged `all_mono` upstream; `text_ocr__code` (fortran) is OCR-gated; `text_simple__hex` has a junk language string (`mki@mki:~/...>`). The overall `text_formatting` `avg_rule_pass_rate` read 0.3683 vs the 0.371 reference — within documented recorded-score/binary drift, **not a regression**: the change is strictly additive (only appends a language word to ` ``` ` fences in code-block docs), so the thousands of `is_title`/`is_bold`/`is_italic` tests on non-code docs are byte-identical. 204 lib tests pass (1 new: `detect_code_language_classifies_common_langs`). No olmOCR/opendataloader impact (code blocks ~absent there).+>+> **Reading-order (figure-caption) detour — investigated, nothing shipped.** Before this, dug into the 2026-06-11 audit's "#1 caption/sidebar splicing (~150 order tests)" lever. Re-triage (`order_defect_triage.py`): 319 failing olmOCR multi_column order tests = 147 SHREDDED_INTERLEAVE (target) + 85 NOISE_NEAR_PASS (tolerance, free) + 61 TEXT_DROPPED (extraction, orthogonal) + 17 PARTIAL + 9 inversions. The 147 span 84 docs, of which only **17 are figure-caption** (image + caption marker), 10 image-only, **57 no-image** (running-header/author-chrome splices, reference-list interleave, some glyph-garble confounded). Two contained attempts, both empirically dead: (a) **co-baseline caption guard** in `build_projected_lines` (refuse same-y-band merge across a column gutter when italic style differs) — de-splices the *output* cleanly (verified on `0b045ab8…page_18`) but **0 wins / 3 losses** on the metric, because the scored failures are cross-column *reading-order* (page_18 wants the left-column caption before the right-column one), and splitting a co-baseline line separates without reordering; the italic-XOR also misfires on emphasized terms at column edges. (b) **column-cut histogram exclusion** — abandoned before coding: per-doc `LITEPARSE_DEBUG_XY` traces show the fig-caption docs' columns **already split correctly** (`0083fb21` V@395+V@179, `02e31046` V@276); the `peaks` rejects are benign single-column-leaf decisions, not split failures, so excluding caption items from the line-start histogram would be a no-op. The genuine residual wins need true cross-column **reordering** (semantic) or a **banner false-positive fix** (~3 docs, `027880a8…page_6` / page_18 style, where spurious H-cuts fragment a column into a sub-8-line region that can't V-cut) — neither a clean lever. **Accepted as a known limitation for now.** The audit's "well-scoped ~150-test win" was over-optimistic: the real target is ~17 heterogeneous docs on the riskiest file.+> **2026-06-11 launch-readiness quality audit (research only, no code changes).** Competitor-independent audit of absolute output quality, validated on real current output via a 5-way parallel investigation (word fusion, reading order, dropped/duplicated content, math, structural/inline formatting). Data sources: opendataloader 200 paired prediction/ground-truth markdown (`opendataloader-bench/{prediction/liteparse,ground-truth}/markdown/<id>.md`), olmOCR category outputs + predicate JSONLs, ParseBench formatting CSVs. Reusable triage script written: `dataset_eval_utils/order_defect_triage.py`.>> **Headline reframe — reading order is less broken than the metric implies.** The ~52.8% olmOCR order-pass rate overstates it. Of 320 failing multi_column order tests: ~84 are metric noise (both strings present *in correct order*, fail only on strict char-match from a dropped space — free wins via tolerance, not defects), ~61 are extraction/OCR drops (orthogonal), only **~8 are true column inversions**, and **~150 are the real defect: caption/sidebar text spliced into body prose** (floating captions / narrow side-blocks not isolated before column ordering — e.g. `multi_column/0b045ab8…page_18` prints caption #2 → body → caption #1 interleaved; `02943986…page_7`, `0904c7071…page_4`, `0925342e…page_4`, `0145bc47…page_11`). Fix target is sharp: caption/sidebar isolation in the XY-cut/anchor pass, NOT a reading-order rewrite.diff --git a/crates/liteparse/src/markdown_layout/blocks.rs b/crates/liteparse/src/markdown_layout/blocks.rsindex 3cc4931..2a97dfe 100644--- a/crates/liteparse/src/markdown_layout/blocks.rs+++ b/crates/liteparse/src/markdown_layout/blocks.rs@@ -25,9 +25,12 @@ pub enum Block {},/// Fenced code block — content rendered between triple-backtick fences./// Each entry in `lines` is one source line; preserved as-is (only trailing- /// whitespace stripped) so indentation survives.+ /// whitespace stripped) so indentation survives. `lang` is a best-effort+ /// language hint emitted as the fence info-string (e.g. ```` ```python ````)+ /// when the body matches a known language; `None` emits a bare fence.CodeBlock {lines: Vec<String>,+ lang: Option<String>,},/// Confident table emitted as a markdown pipe table. `header` is `None`/// when the first row didn't qualify (e.g. wasn't bold and the table mode@@ -216,7 +219,7 @@ pub fn render_blocks(blocks: &[Block]) -> String {}out.push_str("```");}- Block::CodeBlock { lines } => {+ Block::CodeBlock { lines, lang } => {// Pick a fence that doesn't appear inside the body. Standard// triple-backtick handles ~all real-world code; bump to ~~~ if// the body itself contains ``` (rare).@@ -226,6 +229,9 @@ pub fn render_blocks(blocks: &[Block]) -> String {"```"};out.push_str(fence);+ if let Some(lang) = lang {+ out.push_str(lang);+ }out.push('\n');for line in lines {out.push_str(line);@@ -338,6 +344,7 @@ mod tests {fn code_block_escapes_internal_fence() {let blocks = vec![Block::CodeBlock {lines: vec!["body containing ``` backticks".into()],+ lang: None,}];let s = render_blocks(&blocks);assert!(s.starts_with("~~~\n"));diff --git a/crates/liteparse/src/markdown_layout/classify.rs b/crates/liteparse/src/markdown_layout/classify.rsindex 9f400c7..7648426 100644--- a/crates/liteparse/src/markdown_layout/classify.rs+++ b/crates/liteparse/src/markdown_layout/classify.rs@@ -433,7 +433,8 @@ impl FlowState {if let Some(lines) = self.code.take()&& !lines.is_empty(){- blocks.push(Block::CodeBlock { lines });+ let lang = detect_code_language(&lines);+ blocks.push(Block::CodeBlock { lines, lang });}}@@ -908,6 +909,67 @@ fn classify_region(blocks}+/// Best-effort language hint for a fenced code block, used as the fence+/// info-string. Conservative: returns `Some` only when the body carries+/// strong, language-specific signals; otherwise `None` (bare fence). This+/// feeds markdown consumers that key off the language tag (e.g. syntax+/// highlighting, ParseBench's `is_code_block` rule).+fn detect_code_language(lines: &[String]) -> Option<String> {+ let body = lines.join("\n");+ let trimmed = body.trim_start();++ // JSON: starts with a brace/bracket and reads as an object/array with+ // quoted keys. Checked first because `{`/`[` are unambiguous openers.+ if let Some(first) = trimmed.chars().find(|c| !c.is_whitespace())+ && (first == '{' || first == '[')+ && body.contains("\":")+ && body.matches('{').count() + body.matches('[').count() >= 1+ {+ return Some("json".to_string());+ }++ // C / C++: preprocessor includes, namespace qualifiers, stream ops.+ let cpp_hits = [+ "#include",+ "std::",+ "int main",+ "nullptr",+ "->",+ "::",+ "template<",+ ]+ .iter()+ .filter(|s| body.contains(**s))+ .count();+ if cpp_hits >= 2 {+ return Some("cpp".to_string());+ }++ // Python: keyword-led lines, f-strings, comprehensions, dunder/`self.`.+ let py_signals = [+ "self.", "import ", "from ", "def ", "class ", "print(", "lambda ", "elif ", "f'", "f\"",+ "__", " for ", "len(", "sorted(", "range(",+ ];+ let py_hits = py_signals.iter().filter(|s| body.contains(**s)).count();+ // A colon-terminated control line (`if ...:`, `for ...:`, `def ...:`) is a+ // strong Python tell that other curly-brace languages lack.+ let py_colon_block = lines.iter().any(|l| {+ let t = l.trim_end();+ t.ends_with(':')+ && [+ "if ", "for ", "while ", "def ", "class ", "elif ", "else", "try", "except",+ "with ",+ ]+ .iter()+ .any(|kw| t.trim_start().starts_with(kw))+ });+ if py_hits >= 2 || (py_hits >= 1 && py_colon_block) {+ return Some("python".to_string());+ }++ None+}+/// Cross-region merging. Walks the concatenated block stream and, at each/// region boundary, tries to fuse the last block of region A with the first/// block of region B when they represent one logical unit split across leaves.@@ -1091,7 +1153,7 @@ mod tests {// Expect: Paragraph("Intro line."), CodeBlock(2 lines), Paragraph("After...")assert_eq!(blocks.len(), 3);match &blocks[1] {- Block::CodeBlock { lines } => {+ Block::CodeBlock { lines, .. } => {assert_eq!(lines.len(), 2);assert!(lines[0].contains("let x = 1;"));assert!(lines[1].contains("let y = x + 2;"));@@ -1103,6 +1165,38 @@ mod tests {assert!(s.ends_with("After the code."));}+ #[test]+ fn detect_code_language_classifies_common_langs() {+ let py = vec![+ "self.mm_list = sorted([x for x in self.files_list])".to_string(),+ "self.mm_total = len(self.mm_list)".to_string(),+ ];+ assert_eq!(detect_code_language(&py).as_deref(), Some("python"));++ let py_block = vec![+ "if item.total > 0:".to_string(),+ " print('many')".to_string(),+ ];+ assert_eq!(detect_code_language(&py_block).as_deref(), Some("python"));++ let json = vec![+ "{".to_string(),+ " \"formatVersion\": \"1.0\",".to_string(),+ "}".to_string(),+ ];+ assert_eq!(detect_code_language(&json).as_deref(), Some("json"));++ let cpp = vec![+ "#include <vector>".to_string(),+ "std::vector<int> v;".to_string(),+ ];+ assert_eq!(detect_code_language(&cpp).as_deref(), Some("cpp"));++ // Ambiguous / unknown content stays untagged (bare fence).+ let unknown = vec!["let x = 1;".to_string(), "let y = x + 2;".to_string()];+ assert_eq!(detect_code_language(&unknown), None);+ }+#[test]fn classify_marks_paragraph_bold_when_all_lines_bold() {let mut a = line("Bold line one.", 50.0, 50.0, 10.0, 10.0);