A fast, helpful, and open-source document parser
remove tracking file
Logan Markewich48dc7f4628c5
diff --git a/TABLE_TEDS_PROGRESS.md b/TABLE_TEDS_PROGRESS.mddeleted file mode 100644index c160f75..0000000--- a/TABLE_TEDS_PROGRESS.md+++ /dev/null@@ -1,537 +0,0 @@-# Table / TEDS gap vs pdf-inspector — progress & followups--Goal: close the table-fidelity (TEDS) gap against **pdf-inspector** (firecrawl) on-`opendataloader-bench` (200 PDFs). Started 2026-08-01.--## Standing--| metric | baseline (2.10.1) | r1 (`093ddba`) | r2 (`520413d`) | r3 (`08225ac`) | r4 (`8aad56b`) | r5 (`09c9080`) | Δ total | pdf-inspector |-|---|---|---|---|---|---|---|---|---|-| **overall** | 0.8732 | 0.8757 | 0.8783 | 0.8786 | 0.8816 | **0.8855** | +0.0123 | 0.8753 |-| TEDS | 0.6929 | 0.7128 | 0.7441 | 0.7471 | 0.7817 | **0.8134** | **+0.1205** | 0.8141 |-| MHS | 0.8114 | 0.8175 | 0.8179 | 0.8179 | 0.8203 | **0.8210** | +0.0096 | 0.7879 |-| NID | 0.9127 | 0.9129 | 0.9136 | 0.9136 | 0.9143 | **0.9169** | +0.0042 | 0.9147 |--ParseBench table composite (the mandatory cross-check): 0.4034 → 0.4063 (r3) →-0.4074 (r4) → **0.4171** (r5), so every round so far is positive-or-flat on-*both* benchmarks.--**After round 5 we lead on every metric.** The TEDS gap is **−0.0007** (was-0.032, and 0.101 before r4) — parity within noise. **TEDS is scored on only 42-of the 200 docs**, so one doc = 0.024 of the mean.--Round 5 is three commits (`bd96364`, `6e1557b`, `09c9080`), ~230 lines in-`tables.rs`, no other file touched. 314 lib tests pass.--## What shipped--1. **`collapse_gutter_columns`** — called from `build_ruled_table` right after- `cluster_boundaries`. Padded cell borders contribute *paired* vertical edges- (15–25pt apart), leaving a sliver column between every real column — doc 200 had 13- columns for a 4-column table. `split_span_at_anchors` shreds text into the slivers, so- `collapse_phantom_cols` can't see they're empty and the grid dies on the empty-cell- gate. Fuse a column only when **no raw span center lands inside it** AND it is- < 0.5× the median *content-bearing* column width.- *Both halves are load-bearing*: a width-only/bimodal variant broke docs 178+120 from- 1.000 → 0.65, and the width guard is what preserves blank worksheet columns.--2. **Merged-run recovery for absorbed headers** — in `absorb_header_lines`. PDFium emits- a header row's words as one run, so an 8-col table got a 2-cell header whose blob- bound to a single column (doc 190). Reuses the existing `recover_merged_cell`.- Needs **three** gates or it shreds prose into fake headers:- `absorbed.is_empty()` (first candidate line only), every recovered piece ≤ 30 chars,- and `cells.len() < column_count`. Dropping any one of these regressed NID.--## What shipped — round 2 (followups 1 + 2)--3. **Two-pass last-resort for header + single-data-row tables** (+0.0188 TEDS,- +0.0002 NID; doc 197 alone 0.000 → 0.789, *one* doc changed, no collateral).- `try_detect_table_inferred` takes `allow_two_row`; `two_row_second_pass` retries- only in the index gaps where the normal pass found no table.- *The gap restriction alone is NOT enough* — that was the plan and it failed:- doc 147 still broke (−0.82) and 14 docs lost NID. The real counterfeit is- **fully-justified prose**, whose stretched inter-word spaces infer as clean- tracks, so any two consecutive lines shred into- `| when travelling | to | conflict | zones, | more |`. What actually works is- `two_row_run_plausible`: **isolation** (neighbours not table-adjacent — a prose- pair is mid-paragraph) plus **header shape** (cells start uppercase/digit, no- trailing comma). Gates apply only when the relaxation actually fired, so they- can never reject something the normal pass accepted.--4. **Multi-line cell / row grouping** (+0.0125 TEDS, 3 docs up, none down;- doc 150 0.446 → 0.861, now *beating* pdf-inspector's 0.847).- `merge_continuation_rows`, applied at the end of `merge_table_runs` — the one- funnel both ruled and borderless runs pass through. Post-hoc and text-only, as- predicted; the geometry really is unusable. A row folds into its predecessor- when it has an empty first cell, its filled columns are a **subset** of the- predecessor's, and at least one extended cell reads as a soft wrap.- Two vetoes are load-bearing, and **each was found by a different benchmark**:- - **Any value-like cell → not a continuation** (odl-bench). Numbers don't- soft-wrap, and a `Total` row has a blank label column precisely because the- label isn't its own. Vetoing on `all` value-like instead of `any` cost docs- 45 (−0.13) and 47 (−0.09); `any` turned both positive.- - **First column must be filled in ≥ half the rows** (ParseBench only —- see the cross-benchmark note below). "Empty first cell" only means "wrapped- line" if the first column is a *label* column; plenty of tables just have a- sparse one.--## What shipped — round 3 (word-geometry column splitting)--5. **Split PDFium's merged multi-cell runs on real word geometry**- (+0.0030 odl TEDS, +0.0029 ParseBench table, +0.0009 NID; docs 189 and 190- up, **nothing down on either benchmark**).-- This is the root of the "track chicken-and-egg" recorded as the table- quality ceiling: the detector needs column tracks to split a merged run, but- the merged runs are exactly where the track evidence lives. Word boxes break- the cycle — the gutter is visible in the geometry *before* any table- hypothesis exists. Measured on doc 190's header: in-cell word gaps 1.76pt,- column gutters 7.7–12.2pt.-- - `split_span_at_gutters(span)` splits one run at its internal gutters;- `line_pieces(line)` is the tabular view of a row. A run is now what PDFium- emitted; a *piece* is what the page calls a cell.- - Wired into all three places that previously reasoned over raw spans:- `infer_tracks_from_raw_items` (a single merged run now witnesses its own- columns), `is_strong_row`, and `cells_from_raw_items_with_tracks`.- - `parser.rs` forces `emit_word_boxes` on for markdown output. Cost measured- at ~3% user CPU over 40 docs.-- **The threshold must be relative, never absolute.** The same 4pt gap is a- gutter in 6pt type and an ordinary space in 12pt type — this is the same- trap recorded in the missing-spaces work. Instead, sort the run's own gaps- and look for a bimodal jump: real gutters run 3.4–4.4× the in-cell gap,- while fully-justified prose (the one reliable counterfeit) tops out ~1.3×.- Requires ≥3 words, since two words give one gap with nothing to compare it- to. Unit tests cover both the firing case and the justified-prose case.--## What shipped — round 4 (rowspans, the veto share, booktabs bands)--6. **Rowspan-aware density gate** (`87d54ef`; +0.0055 odl TEDS, +0.0011- ParseBench; doc 146 0.358 → 0.588, nothing down).-- The ruled-grid density gate counted a rowspan continuation cell as a *failed- text assignment*, so a table with a merged label column ("1. Embodying- sustainability values" beside three competence rows) died as mostly-empty and- its text spilled into prose. This is the general form of the "empty-frac is a- symptom, not a threshold" note below — the empties were real, they just had a- geometric explanation.-- `rowspan_mask` reads the merge straight off the rules: a cell whose top- boundary carries **no horizontal stroke over its own centre**, at a boundary- **some vertical rule runs through**, is merged with the cell above. The mask- rides on `CellGrid` so it survives `collapse_phantom_rows`/`_cols` (and is- realigned across `merge_stacked_header`) — an early scalar version forgave- *pre*-collapse spanned cells against a *post*-collapse empty count and let a- junk grid through on the scale mismatch alone.-- Both guards are load-bearing, each found by a document it broke:- - **vertical must continue through the boundary** — otherwise the grid has- merely *ended* there, and a page-frame component shredded body prose into a- 2-column table.- - **the merged head must hold text** (walk up the run of spanned cells; the- first unspanned cell must be filled) — otherwise a chart's vertical- gridlines made every cell "spanned" and produced a 27-column table,- −0.0072 NID over 4 docs.-- Test the column *centre*, never containment of the whole column: `xs` comes- from clustered + gutter-collapsed boundaries and the outer ones can sit tens- of points off the drawn border (doc 146: `xs[0]=42.9` vs a border at 79.7).--7. **`already_handled` needs a material share** (`46e95b4`; +0.0053 odl TEDS,- ParseBench flat; doc 200 0.054 → 0.276, NID +0.097, MHS −0.143).-- The global ruled pass builds doc 200's table whole, then `classify.rs` throws- it away because *some* xy_cut leaf tables on its own. That leaf was the- landscape slide's title band — **3 of the run's 58 lines**, "tabling" as a row- of gutter-split title fragments. Require the vetoing leaf to hold ≥¼ of the- run's lines. This is the mechanism behind the two dead `flatten_header_band`- guards recorded below: that component was rejected here, downstream of the- flatten, which is why both guards measured exactly zero.--8. **Booktabs rule-band 2-column pass** (`8aad56b`; +0.0238 odl TEDS,- ParseBench flat; doc 165 0.000 → **1.000**, matching pdf-inspector, NID- +0.046, *one* doc changed).-- Item 4's "bordered worksheet" diagnosis was wrong. Doc 165's table is two- hairlines with **no vertical rules anywhere on the page**, so `extract_h_v_segments`- yields 0 VSegs (they're gated on `height > 1.0`) and `find_grid_components`- never forms a component — `build_ruled_table` is never called, and no density- gate is ever reached. The real killer is `TABLE_MIN_COLUMNS = 3` on a- genuinely 2-column table. Round 3's splitter already recovers both tracks- (`ratio=3.42`, xs=[60,118]) from the merged header run; they were discarded- one line later.-- `rule_bands` pairs two horizontal rules with ≥60% x-overlap, 10pt < gap <- ½ page, and **no vertical crossing between them** — a band with verticals is- a real grid and the ruled detector owns it. Inside a band, in gaps the normal- pass left empty, retry with `allow_two_col`: column minimum 2, and- single-piece body rows allowed (a blank second column makes *every* body row- one piece — `cells_from_raw_items_with_tracks` rejects those by default, and- that guard must stay default-off).-- Detection runs **against the band's lines alone** (sub-slice, indices shifted- back), so a run can't reach past the rules that justify the relaxation. Doing- it in-place instead failed on `row spacing cv 0.53` because the run swallowed- the paragraph below the band. Further gates: band lines contiguous, ≥3 of- them, seed line reads as exactly 2 pieces.--## What shipped — round 5 (the three things nobody was reading off the page)--Every round-5 change has the same shape: the geometry already answered the-question and no code was looking at it.--9. **`RULED_VLINE_MIN_COVERAGE` — vertical rules too short to be columns**- (`bd96364`; +0.0040 odl TEDS, ParseBench flat; doc 200 0.276 → 0.445).-- A slide deck draws a table as one stroked rect *per cell*, so a 21.6×9.4pt- highlight box behind a phrase **inside** a cell contributes a pair of- vertical edges that split that column for the whole table — doc 200's 210pt- `Explanation` column became three, taking a 4-column grid to 6.- `collapse_gutter_columns` cannot fuse the sliver back, because text centres- genuinely do land in it. The mirror of `RULED_HLINE_MIN_COVERAGE`: a vertical- must span 20% of the component's row extent.-- **Applying the filter directly is what does not work, and this is the- general lesson: dropping evidence can flip a grid from rejected to- accepted.** Three documents found that one at a time — a bar chart lost its- last column and released its gridlines as thematic breaks (doc 71, MHS- 0.98 → 0.62); a pie chart's callout rules were dropped and a grid the density- gate had rejected became a junk 2-column table (doc 70, NID −0.10); and- ParseBench `FBLB-134215544_page88` turned a rejected component into an- 11-column table that swallowed the page (0.66 → 0.03) after dropping **one- 4%-coverage stub out of 20**. Raising the fallback threshold patched each in- turn and kept finding new ones. What kills the class is structural: the- unfiltered build runs first and gatekeeps, so the filtered grid can only- ever *replace* a table that would have been produced anyway.-- It must also *earn* the replacement. `CellGrid` now carries `straddle_frac`,- and the refined grid is taken only if the straddle census drops by ≥0.05 —- doc 200 goes 0.33 → 0.03, whereas a stub sitting on a real column edge (very- common in dense financial tables) leaves it flat and merely costs a column- (ParseBench `corp-q1-2025_page10`, 0.73 → 0.33 without this).--10. **Unruled outer rows and columns, recovered from the perpendicular rules**- (`6e1557b`; +0.0123 odl TEDS, **+0.0070 ParseBench** — the largest- single-change ParseBench gain so far; doc 182 0.247 → 0.762).-- A table with interior dividers but no outer frame gives `xs` that stop at- the first and last vertical rule, so its outermost columns are *missing*.- Doc 182's 4×4 table came out as the 2×2 grid its rules literally describe- and all 31 lines tripped the overhang guard before the component collapsed.- The horizontals already know how wide the table is and the verticals how- tall; nothing read them. Extend each axis to the **median** extent of the- perpendicular rules.-- Three guards, each load-bearing:- - **median, not min/max** — one overshooting stroke must not widen the grid;- - **the new band must hold a span centre**, the `collapse_gutter_columns`- idiom, or a pen-cap overshoot manufactures an empty outer column;- - **the band must be at least as tall as the shortest existing row.**- Verticals routinely overrun the last horizontal by a few points; doc 45- turned a 15.5pt tail on 26pt rows into a phantom row, which was enough for- the ruled grid to outrank a better borderless table and merge five rows- into one (−0.50). The *column* axis takes no such floor — an unruled label- column is common and legitimately much narrower than its data columns- (doc 182's is 94pt against 259pt).-- On ParseBench four pages of one insurance filing go from 0.09–0.42 to ~0.99.--11. **Split a grid component at row bands no vertical rule spans**- (`09c9080`; +0.0154 odl TEDS, +0.0027 ParseBench; docs 82/84 → 1.000,- 83 → 0.995, 81 → 0.935).-- This closes followup 8, and the mechanism was **not** the one recorded- there. `cluster_v_segments` merges same-x verticals by taking the *union* of- their y-ranges **with no gap check**. Two tables stacked in one column, each- drawing its own short strokes at the same left edge, therefore fuse into one- component across 113pt of blank page. Each is then scored with the other's- rows empty and dies on the empty-cell fraction — and where the two layouts- differ their column sets are unioned too. Doc 81's `Number | of clauses`- split across two columns, filed in the negative results as a- detector-priority problem, is that union and needs no priority plumbing.-- The cut signal needs no emptiness threshold: a band between consecutive- horizontals that **no raw, pre-cluster `VSeg` spans**. That is strictly- stronger than "a big gap", and unlike a gap rule it cannot fire on a- rowspan, where the vertical does continue through the boundary. Each band's- verticals are then **clipped to the band**, or the shared spine carries the- neighbour's height straight back in through items 9 and 10.-- Guards: median row pitch ≥ 8pt (vector-drawn maths glyphs make components- with a 3–4pt pitch), gap ≥ max(2.5 × pitch, 30pt), and every band must keep- ≥2 rows or the split is abandoned whole. Docs 127/130/165/188 — the ones the- round-4 trim experiment broke — are untouched, because a post-hoc row trim- keeps the unioned *columns* and splitting the component does not.-- **Requiring 3 rows per band instead of 2 was measured and is worse**- (−0.0004 ParseBench: it loses `FBLB-134215544_page6` and recovers neither of- the two docs it was aimed at). Left at 2.--## Cross-benchmark check is mandatory for table work--**opendataloader-bench alone is not enough.** Round 2 looked perfect on it — four-docs up, none down — while silently costing **−0.0048** on the ParseBench table-dimension. Attribution (one run per variant):--| variant | odl TEDS | ParseBench table composite |-|---|---|---|-| HEAD `093ddba` | 0.7128 | 0.4032 |-| + two-row pass | 0.7316 | 0.4032 (free) |-| + continuation merge | 0.7441 | **0.3984** ← regression |-| + first-column fill guard | 0.7441 | **0.4034** |--The continuation merge owned the whole regression, concentrated in a few docs-(`1 timetable_page6` 0.859 → **0.006**, `sizingchart`, `myco hierarchical table-header`). All were tables with a legitimately sparse first column, where every-row reads as a continuation and the table collapses to one row. The fill guard-fixes them at zero cost to odl-bench.--Note `is_value_like` does **not** match bare times (`8:24`) or bare integers-(`15`) — it needs `d.d` / `d,d` / `$` / `%`. Don't widen it to patch a table bug;-it gates several unrelated paths. Prefer a structural guard.--Run: `cd ParseBench && uv run parse-bench run liteparse_markdown --group table`-(~12 min, 503 examples). Read `avg_grits_trm_composite` from-`output/liteparse_markdown/_evaluation_report.json` — this is the leaderboard's-"Tables" column (0.4032 = the 40.3 in `leaderboard.csv`). **The run overwrites-that report in place**, so copy it aside before running a variant.--## Tooling--- **Per-doc deltas**: `opendataloader-bench/perdoc.py <label-b> [label-a] [metric]`- — prints every doc whose metric moved between two prediction labels, worst- first. Doc numbers match the ones used in this file. This is what caught that- the round-2 mean hid a −0.82 doc in the first two-row attempt.-- **A/B harness**: `opendataloader-bench/ab.sh <label>` — parses all 200 PDFs with the- tuned bench config and prints the delta vs `prediction/liteparse` (the frozen 2.10.1- baseline). Verified byte-identical to the recorded baseline on all 200 docs.- Takes ~1–2 min. **Always A/B; per-doc deltas matter more than the mean.**-- **Geometry dumps** (`crates/liteparse/examples/`, run with- `cargo run -q --release -p liteparse --example <name> -- <pdf>`):- `dump_rects_raw` prints every graphic primitive plus each text item with its- word boxes and inter-word gaps — this is what made the gutter signal visible;- `dump_chars_raw <pdf> <page> <ymin> <ymax>` drops to per-character boxes.- `dump_table_rects` gives the h/v stroke counts per page, which is how you tell- a booktabs table (`h_strokes=6 v_strokes=0`) from a fully ruled one.- Beware: vector-drawn math glyphs show up as ~170 tiny `stroked_rects` and can- produce a phantom table rect (doc 165).-- **Debug envs**: `LITEPARSE_DEBUG_TABLE=1 LITEPARSE_DEBUG_RULED=1` on a single- `lit parse ... --format markdown --image-mode off --no-ocr --no-links -q`.- These print every gate rejection with its numbers. **Use these first** — twice I- inferred a mechanism from the emitted markdown and was wrong.- `LITEPARSE_DEBUG_GUTTER=1` prints every in-run column split with its ratio.-- **Never rebuild the binary while a bench run is in flight** — both harnesses- shell out to `target/release/lit`, so a mid-run `cargo build` silently mixes- two variants into one score.-- **Attribution runs are cheap and mandatory.** Two changes bundled into one- ParseBench run cost a day of ambiguity in round 4; one run per variant- (~12 min) is always worth it. `ab.sh` on odl is 1-2 min, so bundle there and- split on ParseBench.-- **pdf-inspector repro**: `git clone --depth 1 https://github.com/firecrawl/pdf-inspector`,- `cargo build --release --bin pdf2md` (~28s), run `pdf2md <pdf> --raw`. Its outputs and- scores are already committed at `opendataloader-bench/prediction/pdf-inspector/`.- Re-evaluate with `uv run src/evaluator.py --engine pdf-inspector`.--## Where the remaining −0.0007 sits--We are at parity on the mean, so from here the question is no longer "close the-gap" but "which docs are still individually behind". Top per-doc gaps after-round 5 (perdoc index, ours vs pdf-inspector):-199 −0.346, 118 −0.276, 196 −0.211, 120 −0.173, 45 −0.156, 189 −0.127,-145 −0.126, 169 −0.122, 115 −0.108, 165 −0.107, 187 −0.083, 80 −0.065.--Note the index shift: these are `perdoc.py` indices (doc id = index + 1). The-old "doc 200 / 83 / 81" labels in the sections above are ids; 199/82/80 are the-same documents.--Docs we *lead* on offset all of this, which is why the mean is level — but-every entry above is still a real defect.--## Where the 0.101 sat before round 4--Measured by GT-cell-text retention (liteparse **0.747** vs pdf-inspector **0.870**):--| class | TEDS cost | meaning | docs |-|---|---|---|---|-| **MISPLACED** | −0.073 | text is in the doc but never lands in a table | 165, 197, 200, 182, 116 |-| **STRUCTURE-only** | −0.058 | text *is* in the table, structure wrong | 150, 146, 119, 190, 81, 84, 166, 188, 82, 90, 89 |-| LOST-upstream | −0.008 | genuine extraction failure | 121, 46 |--So the dominant problem is **misfiling**, not dropping, text.--## Followups, ranked--*(followups 1 and 2 are done — see "round 2" above.)*--2b. **More of the continuation class is still available.** `merge_continuation_rows`- only fires on rows with an **empty first cell**. pdf-inspector's veto set is richer- (`looks_like_data_row`, `is_short_subheader`, `looks_like_hierarchical_subrow`,- `looks_like_spanning_first_column_row`), which lets them merge wraps where the first- column *is* populated. Worth trying next, but the empty-first-cell rule is what makes- the current version regression-free — relax it only behind an A/B.--3. **Doc 200 / index 199 — the columns are fixed, the rows are not**- (0.445 vs 0.791, ~+0.008 available). Item 9 fixed the column- over-segmentation this slot used to describe; `xs` is now GT's 4. Two things- remain, and they are one mechanism:- - The page is **not a ruled grid at all — it is 44 independent per-cell- rects** (4 distinct x/width pairs × 11 rows; see `dump_rects_raw`). The- columns' rects are not y-aligned (each cell's rect hugs its own text), so- unioning every top and bottom yields 24 row boundaries for 11 rows. Tall- cells then get cut by a neighbouring column's boundaries and their text- splits across rows (`2. Data labeling and` / `fine-tuning`).- - `flatten_header_band` still folds the two full-width *title* banners into- the header row.- The right model for this class is to cluster the cell rects directly —- columns by near-identical (x, width), rows by **y-interval overlap** (rows- overlap ~100%, adjacent rows 0%, which is unambiguous where no y threshold- is). That is a new detector, not a tweak, and it needs priority plumbing- against the ruled path.- A cheaper partial: `rowspan_mask` already marks these cells correctly (it is- how the density gate survives), but `spanned` is only *counted*, never used- to merge cell text. Making a spanned cell fold into the cell above is- contained and reuses round-4 machinery.- Do **not** relax the `flatten_header_band` anchor — measured at −0.0043 on- ParseBench, see negative results.--4. ~~**Doc 165 worksheet**~~ — **DONE in round 4** (item 8), 0.000 → 1.000. The- "bordered worksheet / `TABLE_MAX_EMPTY_CELL_FRACTION`" diagnosis in this slot- was wrong; see item 8 for what it actually was.--5. ~~**Tagged `/Table` structure tree**~~ — **DEAD. Do not build this.** Measured- before writing any code: **0 of the 200 opendataloader-bench PDFs carry a- structure tree at all**, and 0 of the first 120 ParseBench docs do either.- A tagged-`/Table` detector would be pure dead code against every benchmark we- optimize. The extractor is fine — verified against a LibreOffice-exported- tagged PDF, which returns the full `Document > Table > TR > TH/TD` tree — the- corpora are simply untagged. pdf-inspector's lead comes from somewhere else.- Probe: `lit parse <pdf> --format json --extract-structure-tree` and count- nodes under `pages[].structure_tree.roots`. Still worth having for *real- user* documents someday, but it can never show up in these numbers.--6. ~~**Sub-word tracks**~~ — **built and measured inert.** `split_words_at_x_anchors`- (real word x's) and `bucket_words_into_columns` (per-word ruled-column- assignment) now front both anchor-split paths, and the 200-doc output is- **byte-identical**. The borderless multi-track branch fires only **12 times in- the whole corpus** and word/char agree every time; the ruled branch fires 434- times and buckets 219 of them differently, but every one of those components- is rejected later anyway. Round 3's gutter split already consumes this class- upstream. Kept (geometry beats interpolation where it does fire), but don't- expect a number from it — doc 83's `10 .1%` comes from somewhere else.--7. **Doc 190's last row (`Merge v4 | SLERP`, −0.126).** The header row is fixed;- the final body row still falls out because its second cell (`SLERP`, x=182.9)- sits 14pt off the track inferred from the rows above (`Average`, x=168) —- past the 6pt `TABLE_TRACK_TOLERANCE_PT`. Needs track *ranges* rather than- points; touch with care, that tolerance gates several paths.--8. ~~**Docs 81/83/84 (same source family)**~~ — **DONE in round 5** (item 11).- The diagnosis in this slot was right about the symptom and wrong about the- cause: the component does span two stacked tables, but because- `cluster_v_segments` unions y-ranges across a 113pt gap, not because of- anything about the rows. Splitting the component (not trimming rows) fixes- all four and leaves the trim experiment's casualties alone.--9. **Doc 119 / index 118 (0.724 vs 1.000, ~+0.007).** Diagnosed, not built. Two- independent defects:- - **9 phantom rows.** The PDF draws a background fill rect per *text line*- inside each cell, and `RULED_HLINE_MIN_COVERAGE` is **Global-pass only** —- the per-region pass (which is what emits this table) keeps every- horizontal. Coverage over the column extent is perfectly bimodal: 7- boundaries at ≥0.998 (exactly GT's 6 rows) against 9 phantoms at 0.661 and- 0.336. Note the existing 0.5 threshold would *not* fix it even if promoted;- it needs a strict tier (~0.9) gated on the full-width frame being real.- **Blast radius is measured and large**: over the 200-doc corpus, 24 docs- lose boundaries under a 0.9 filter, including 165, 150, 199, 200 and 141.- This needs its own attribution run and a hand check of 165.- - **A wrong column split.** `bucket_words_into_columns` returns `None` when- every word centre lands in one column — the doc comment says the caller's- "whole-span fallbacks" are then the honest answer, but the actual next- fallback is `split_span_at_anchors`, which *interpolates* a split from a- character index and cuts `(begins` off into the previous column. Word- geometry had the right answer and it was discarded precisely because it was- unanimous. Small, independent, and worth doing first.--10. **Doc 197 / index 196 (0.789 vs 1.000) — do not chase this to 1.0.**- Measured by patching the markdown and re-scoring: the *faithful* structural- reading of the missing table (1 column × 11 rows) scores **0.447**, i.e.- −0.34. GT's 1.000 requires reproducing an idiosyncrasy (a header cell and a- merged body cell side by side in one row). A 1-column header + single merged- body cell gets 0.850 (+0.061 on one doc = +0.0015 mean) and would require a- new 1-column-table producer in `two_col_band_pass`. Low priority, and the- ceiling is an artefact rather than a defect.--## Negative results — do not retry blind--- **Relaxing `TABLE_MAX_EMPTY_CELL_FRACTION`** (swept 0.45/0.60/0.75): 0.45 neutral,- 0.60 −0.013 TEDS, 0.75 −0.040. `empty-frac` in the gate logs is a *symptom* of text- not being assigned, not a tight threshold.-- **Relaxing the straddle-frac gate (0.45)**: zero TEDS gain, costs NID and MHS. High- straddle was a symptom of column over-segmentation — it fell 0.48 → 0.32 by itself- once gutters collapsed.-- **Full-width-spanner exclusion in `flatten_header_band`**: exactly zero delta, guard- never fires (see item 3).-- **Directly allowing 2-row header-led tables**: fixes doc 197 (+0.789) but breaks doc- 147 (−0.822) plus ~8 NID regressions. The existing code comment is right — a 2-row run- forms early and consumes the real table's header.-- **Restricting the 2-row relaxation to table-free gaps, with no content gate**:- doc 147 still −0.822, 14 docs lose NID (−0.0030). The gap restriction addresses the- wrong mechanism. Justified prose is the counterfeit, and it lives *in* the gaps.-- **Trimming fully-empty edge rows off a ruled grid** (aimed at docs 81/83/84,- where one component covers two stacked tables so each sees the other's rows as- empty). Tried four increasingly-gated variants; all net-negative or a wash:- | variant | odl TEDS vs round 3 |- |---|---|- | trim unconditionally | −0.0122 (docs 130, 127, 81 fall from ~1.0 to ~0.75) |- | trim only if the untrimmed grid fails the density gate | identical — the pre-check is not predictive, `flatten_header_band` / `merge_stacked_header` rescue those grids *later* |- | retry trimmed only after the untrimmed build returns `None` | identical again |- | + strict density on the retained block, + require ≥3 trimmed rows and ≥4 kept | −0.0006 (83 +0.084, 84 +0.066, 81 −0.119, 82 −0.055) |-- The mechanism the gating missed: the trimmed run doesn't fill a hole, it- **outranks a better table another detector already produced**. Docs 81/127- regress because the ruled run wins with one column too many- (`Number | of clauses` split across two columns) over a correct borderless- table. Fixing this class needs either detector-priority plumbing or splitting- the *component* — not a post-hoc row trim. Reverted; not in the shipped diff.- **Round 5 settled this**: splitting the component (item 11) was the answer,- and the extra column was the two tables' x-sets being unioned, so no- detector-priority plumbing was needed after all.-- **Filtering short vertical rules before the grid is built** (round 5, item 9).- Three separate documents turned a *rejected* grid into an accepted junk table- because the filter removed the evidence the gates were rejecting it on.- Raising the fallback threshold (2 → 3 → 4 surviving rules) patched them one at- a time and kept finding new ones. Never let a cleanup pass run before the- gatekeeper.-- **Requiring 3 rows per band in the component split** (round 5, item 11):- −0.0004 ParseBench, loses `FBLB-134215544_page6`, recovers neither doc it was- aimed at. Left at 2.-- **Tagged-`/Table` and sub-word tracks** — see items 5 and 6 above; both were- measured inert *before* being built out, which is the cheapest thing in this- whole document. Do that first.-- **Relaxing the `flatten_header_band` anchor** to accept a half-filled row of- short (≤40 char), column-distinct labels when column over-segmentation pushes- the real header below `TABLE_ROW_MIN_FILL`. Doc 200 +0.009 TEDS / +0.020 NID- with no odl collateral — and **−0.0043 on ParseBench**, attributed by a- one-variable run. Reverted. This is the second time odl-bench alone would have- shipped a table regression.-- **A second full-width-spanner guard in `flatten_header_band`** (reject the flatten- when a produced header cell exceeds 100 chars, aimed at doc 200): **exactly zero- delta** — this is the second guard to die here. The `[ruled] colspan header flatten`- line you see in doc 200's debug output belongs to a component that is *rejected- later anyway*; the emitted 5-column table comes from a different component. Before- touching `flatten_header_band` again, confirm the component you're looking at is the- one that actually reaches the output.--## Methodology gotchas (both bit me)--- **Gate-reason histograms over the corpus are mostly noise.** `tracks < MIN_COLUMNS`- fires on every prose line; `grid-too-small ys=2 xs=2` is the page-background rect;- `no-lines-consumed` is every non-overlapping region/component pair. Rank by per-doc- loss instead, then read that doc's debug output.-- **`difflib.SequenceMatcher` needs `autojunk=False`.** The default silently corrupts- comparisons on strings over 200 chars and produced impossible numbers (a doc's table- text matching better than the whole document containing it).-- Table detection runs on region lines **before** interruptions (HRs, figures) are- interleaved (`classify.rs:499`), so thematic breaks do *not* fragment table rows —- an early theory of mine that the debug output disproved.