Skip to content

A fast, helpful, and open-source document parser

wip

Logan Markewichcfd22a688a63
diff --git a/MARKDOWN_PROGRESS.md b/MARKDOWN_PROGRESS.md
index bc821d1..38bfca2 100644
--- a/MARKDOWN_PROGRESS.md
+++ b/MARKDOWN_PROGRESS.md
@@ -2,6 +2,12 @@
Tracks build-order progress against [MARKDOWN_PLAN.md](MARKDOWN_PLAN.md). Update as steps complete.
+> **2026-06-12 code-block language tag — ParseBench `is_code_block` 0.0 → 0.2 (per-doc avg; 0/10 → 5/10 individual tests, all five python cases).** Code blocks were already detected and fenced correctly with intact content, but emitted as bare ` ``` ` with no info-string, so `rules_formatting.py`'s `is_code_block` (which requires a fence whose lang matches the expected language, then checks the snippet is in that block's body) scored a hard 0.0. Added `lang: Option<String>` to `Block::CodeBlock` and a conservative `detect_code_language` (`markdown_layout/classify.rs`) called from `flush_code`: JSON (brace/bracket opener + quoted keys), C/C++ (≥2 of `#include`/`std::`/`int main`/`nullptr`/`->`/`::`/`template<`), Python (≥2 keyword/f-string/`self.`/comprehension signals, or ≥1 plus a colon-terminated control line). Returns `None` (bare fence) when unsure, so docs with other-language tests can't be mis-tagged. Emitter writes the info-string after the fence (` ```python `).
+>
+> **Win is python-only.** `text_misc__justcode` now emits four ` ```python ` blocks whose bodies contain the rule snippets (whitespace-normalized match tolerates lost indentation) → 5/5 on that doc. The other 5 tests are **detection-layer gaps, out of scope for the tag**: `text_misc__json` / `text_simple__paper` (cpp) never emit code blocks at all because their content isn't flagged `all_mono` upstream; `text_ocr__code` (fortran) is OCR-gated; `text_simple__hex` has a junk language string (`mki@mki:~/...>`). The overall `text_formatting` `avg_rule_pass_rate` read 0.3683 vs the 0.371 reference — within documented recorded-score/binary drift, **not a regression**: the change is strictly additive (only appends a language word to ` ``` ` fences in code-block docs), so the thousands of `is_title`/`is_bold`/`is_italic` tests on non-code docs are byte-identical. 204 lib tests pass (1 new: `detect_code_language_classifies_common_langs`). No olmOCR/opendataloader impact (code blocks ~absent there).
+>
+> **Reading-order (figure-caption) detour — investigated, nothing shipped.** Before this, dug into the 2026-06-11 audit's "#1 caption/sidebar splicing (~150 order tests)" lever. Re-triage (`order_defect_triage.py`): 319 failing olmOCR multi_column order tests = 147 SHREDDED_INTERLEAVE (target) + 85 NOISE_NEAR_PASS (tolerance, free) + 61 TEXT_DROPPED (extraction, orthogonal) + 17 PARTIAL + 9 inversions. The 147 span 84 docs, of which only **17 are figure-caption** (image + caption marker), 10 image-only, **57 no-image** (running-header/author-chrome splices, reference-list interleave, some glyph-garble confounded). Two contained attempts, both empirically dead: (a) **co-baseline caption guard** in `build_projected_lines` (refuse same-y-band merge across a column gutter when italic style differs) — de-splices the *output* cleanly (verified on `0b045ab8…page_18`) but **0 wins / 3 losses** on the metric, because the scored failures are cross-column *reading-order* (page_18 wants the left-column caption before the right-column one), and splitting a co-baseline line separates without reordering; the italic-XOR also misfires on emphasized terms at column edges. (b) **column-cut histogram exclusion** — abandoned before coding: per-doc `LITEPARSE_DEBUG_XY` traces show the fig-caption docs' columns **already split correctly** (`0083fb21` V@395+V@179, `02e31046` V@276); the `peaks` rejects are benign single-column-leaf decisions, not split failures, so excluding caption items from the line-start histogram would be a no-op. The genuine residual wins need true cross-column **reordering** (semantic) or a **banner false-positive fix** (~3 docs, `027880a8…page_6` / page_18 style, where spurious H-cuts fragment a column into a sub-8-line region that can't V-cut) — neither a clean lever. **Accepted as a known limitation for now.** The audit's "well-scoped ~150-test win" was over-optimistic: the real target is ~17 heterogeneous docs on the riskiest file.
+
> **2026-06-11 launch-readiness quality audit (research only, no code changes).** Competitor-independent audit of absolute output quality, validated on real current output via a 5-way parallel investigation (word fusion, reading order, dropped/duplicated content, math, structural/inline formatting). Data sources: opendataloader 200 paired prediction/ground-truth markdown (`opendataloader-bench/{prediction/liteparse,ground-truth}/markdown/<id>.md`), olmOCR category outputs + predicate JSONLs, ParseBench formatting CSVs. Reusable triage script written: `dataset_eval_utils/order_defect_triage.py`.
>
> **Headline reframe — reading order is less broken than the metric implies.** The ~52.8% olmOCR order-pass rate overstates it. Of 320 failing multi_column order tests: ~84 are metric noise (both strings present *in correct order*, fail only on strict char-match from a dropped space — free wins via tolerance, not defects), ~61 are extraction/OCR drops (orthogonal), only **~8 are true column inversions**, and **~150 are the real defect: caption/sidebar text spliced into body prose** (floating captions / narrow side-blocks not isolated before column ordering — e.g. `multi_column/0b045ab8…page_18` prints caption #2 → body → caption #1 interleaved; `02943986…page_7`, `0904c7071…page_4`, `0925342e…page_4`, `0145bc47…page_11`). Fix target is sharp: caption/sidebar isolation in the XY-cut/anchor pass, NOT a reading-order rewrite.
diff --git a/crates/liteparse/src/markdown_layout/blocks.rs b/crates/liteparse/src/markdown_layout/blocks.rs
index 3cc4931..2a97dfe 100644
--- a/crates/liteparse/src/markdown_layout/blocks.rs
+++ b/crates/liteparse/src/markdown_layout/blocks.rs
@@ -25,9 +25,12 @@ pub enum Block {
},
/// Fenced code block — content rendered between triple-backtick fences.
/// Each entry in `lines` is one source line; preserved as-is (only trailing
- /// whitespace stripped) so indentation survives.
+ /// whitespace stripped) so indentation survives. `lang` is a best-effort
+ /// language hint emitted as the fence info-string (e.g. ```` ```python ````)
+ /// when the body matches a known language; `None` emits a bare fence.
CodeBlock {
lines: Vec<String>,
+ lang: Option<String>,
},
/// Confident table emitted as a markdown pipe table. `header` is `None`
/// when the first row didn't qualify (e.g. wasn't bold and the table mode
@@ -216,7 +219,7 @@ pub fn render_blocks(blocks: &[Block]) -> String {
}
out.push_str("```");
}
- Block::CodeBlock { lines } => {
+ Block::CodeBlock { lines, lang } => {
// Pick a fence that doesn't appear inside the body. Standard
// triple-backtick handles ~all real-world code; bump to ~~~ if
// the body itself contains ``` (rare).
@@ -226,6 +229,9 @@ pub fn render_blocks(blocks: &[Block]) -> String {
"```"
};
out.push_str(fence);
+ if let Some(lang) = lang {
+ out.push_str(lang);
+ }
out.push('\n');
for line in lines {
out.push_str(line);
@@ -338,6 +344,7 @@ mod tests {
fn code_block_escapes_internal_fence() {
let blocks = vec![Block::CodeBlock {
lines: vec!["body containing ``` backticks".into()],
+ lang: None,
}];
let s = render_blocks(&blocks);
assert!(s.starts_with("~~~\n"));
diff --git a/crates/liteparse/src/markdown_layout/classify.rs b/crates/liteparse/src/markdown_layout/classify.rs
index 9f400c7..7648426 100644
--- a/crates/liteparse/src/markdown_layout/classify.rs
+++ b/crates/liteparse/src/markdown_layout/classify.rs
@@ -433,7 +433,8 @@ impl FlowState {
if let Some(lines) = self.code.take()
&& !lines.is_empty()
{
- blocks.push(Block::CodeBlock { lines });
+ let lang = detect_code_language(&lines);
+ blocks.push(Block::CodeBlock { lines, lang });
}
}
@@ -908,6 +909,67 @@ fn classify_region(
blocks
}
+/// Best-effort language hint for a fenced code block, used as the fence
+/// info-string. Conservative: returns `Some` only when the body carries
+/// strong, language-specific signals; otherwise `None` (bare fence). This
+/// feeds markdown consumers that key off the language tag (e.g. syntax
+/// highlighting, ParseBench's `is_code_block` rule).
+fn detect_code_language(lines: &[String]) -> Option<String> {
+ let body = lines.join("\n");
+ let trimmed = body.trim_start();
+
+ // JSON: starts with a brace/bracket and reads as an object/array with
+ // quoted keys. Checked first because `{`/`[` are unambiguous openers.
+ if let Some(first) = trimmed.chars().find(|c| !c.is_whitespace())
+ && (first == '{' || first == '[')
+ && body.contains("\":")
+ && body.matches('{').count() + body.matches('[').count() >= 1
+ {
+ return Some("json".to_string());
+ }
+
+ // C / C++: preprocessor includes, namespace qualifiers, stream ops.
+ let cpp_hits = [
+ "#include",
+ "std::",
+ "int main",
+ "nullptr",
+ "->",
+ "::",
+ "template<",
+ ]
+ .iter()
+ .filter(|s| body.contains(**s))
+ .count();
+ if cpp_hits >= 2 {
+ return Some("cpp".to_string());
+ }
+
+ // Python: keyword-led lines, f-strings, comprehensions, dunder/`self.`.
+ let py_signals = [
+ "self.", "import ", "from ", "def ", "class ", "print(", "lambda ", "elif ", "f'", "f\"",
+ "__", " for ", "len(", "sorted(", "range(",
+ ];
+ let py_hits = py_signals.iter().filter(|s| body.contains(**s)).count();
+ // A colon-terminated control line (`if ...:`, `for ...:`, `def ...:`) is a
+ // strong Python tell that other curly-brace languages lack.
+ let py_colon_block = lines.iter().any(|l| {
+ let t = l.trim_end();
+ t.ends_with(':')
+ && [
+ "if ", "for ", "while ", "def ", "class ", "elif ", "else", "try", "except",
+ "with ",
+ ]
+ .iter()
+ .any(|kw| t.trim_start().starts_with(kw))
+ });
+ if py_hits >= 2 || (py_hits >= 1 && py_colon_block) {
+ return Some("python".to_string());
+ }
+
+ None
+}
+
/// Cross-region merging. Walks the concatenated block stream and, at each
/// region boundary, tries to fuse the last block of region A with the first
/// block of region B when they represent one logical unit split across leaves.
@@ -1091,7 +1153,7 @@ mod tests {
// Expect: Paragraph("Intro line."), CodeBlock(2 lines), Paragraph("After...")
assert_eq!(blocks.len(), 3);
match &blocks[1] {
- Block::CodeBlock { lines } => {
+ Block::CodeBlock { lines, .. } => {
assert_eq!(lines.len(), 2);
assert!(lines[0].contains("let x = 1;"));
assert!(lines[1].contains("let y = x + 2;"));
@@ -1103,6 +1165,38 @@ mod tests {
assert!(s.ends_with("After the code."));
}
+ #[test]
+ fn detect_code_language_classifies_common_langs() {
+ let py = vec![
+ "self.mm_list = sorted([x for x in self.files_list])".to_string(),
+ "self.mm_total = len(self.mm_list)".to_string(),
+ ];
+ assert_eq!(detect_code_language(&py).as_deref(), Some("python"));
+
+ let py_block = vec![
+ "if item.total > 0:".to_string(),
+ " print('many')".to_string(),
+ ];
+ assert_eq!(detect_code_language(&py_block).as_deref(), Some("python"));
+
+ let json = vec![
+ "{".to_string(),
+ " \"formatVersion\": \"1.0\",".to_string(),
+ "}".to_string(),
+ ];
+ assert_eq!(detect_code_language(&json).as_deref(), Some("json"));
+
+ let cpp = vec![
+ "#include <vector>".to_string(),
+ "std::vector<int> v;".to_string(),
+ ];
+ assert_eq!(detect_code_language(&cpp).as_deref(), Some("cpp"));
+
+ // Ambiguous / unknown content stays untagged (bare fence).
+ let unknown = vec!["let x = 1;".to_string(), "let y = x + 2;".to_string()];
+ assert_eq!(detect_code_language(&unknown), None);
+ }
+
#[test]
fn classify_marks_paragraph_bold_when_all_lines_bold() {
let mut a = line("Bold line one.", 50.0, 50.0, 10.0, 10.0);