Skip to content

RAGFlow is an agent-native database: a Retrieval-Augmented Generation (RAG) engine that grew into context infrastructure.

chore: remove the DeepDoc Go alignment report (#19311)

xugangqiang1dce16760291
diff --git a/deepdoc_go_alignment_report.md b/deepdoc_go_alignment_report.md
deleted file mode 100644
index 409b916744..0000000000
--- a/deepdoc_go_alignment_report.md
+++ /dev/null
@@ -1,1139 +0,0 @@
-# DeepDoc Go In-Process Backend vs Python Inference Service
-
-> Status: completed.
-> Scope: `internal/deepdoc/native`, `internal/deepdoc/parser/pdf`,
-> `internal/deepdoc/parser/pdf/inference/native_analyzer`.
-
-## Contents
-
-- [1. Background](#1-background)
-- [2. Python inference service dependency inventory](#2-python-inference-service-dependency-inventory)
-- [3. Go-side migration](#3-go-side-migration)
-- [4. Caller perspective: are the two semantically consistent?](#4-caller-perspective-are-the-two-semantically-consistent)
-- [5. Real OCR batch + removal of the external Python service dependency](#5-real-ocr-batch--removal-of-the-external-python-service-dependency)
-- [6. Implementation notes / pitfalls](#6-implementation-notes--pitfalls)
-- [7. Open items](#7-open-items)
-- [Key file index](#key-file-index)
-- [The native verification harness (design record)](#the-native-verification-harness-design-record)
-- [Equivalence proof](#equivalence-proof)
-
-## 1. Background
-
-RAGFlow PDF parsing relies on DeepDoc inference (OCR text detection / layout
-analysis DLA / table structure TSR / text recognition OCR). The original
-deployment depended on a standalone **Python inference service**
-(`deepdoc_server`). This project reimplements an **in-process Go backend**:
-
-- Pure Go + ONNX Runtime reimplements the same models and the same
- post-processing;
-- it is packaged into the single server binary (`bin/ragflow_server`) — no
- separate Python service to build or deploy;
-- both implementations satisfy the same `DocAnalyzer` interface, so callers are
- unaware of which one is active.
-
-## 2. Python inference service dependency inventory
-
-### Runtime inference (used directly by `deepdoc/vision`)
-
-| Package | Purpose |
-|---|---|
-| `onnxruntime>=1.20.0` | runs the det / layout / tsr / rec ONNX models |
-| `opencv-python-headless` (cv2) | image decode, resize, findContours, minAreaRect+boxPoints, fillPoly, approxPolyDP, NMS |
-| `numpy` | tensor / array math |
-| `pillow` (PIL) | server-side image decode / encode |
-| `pyclipper>=1.4.0` | det box unclip (polygon dilation) |
-| `shapely` | only `Polygon.area/length` for unclip |
-| `six` | py2/3 compat shim |
-
-### Service layer (HTTP)
-
-`litserve>=0.2.17`, `python-multipart`
-
-### Rasterization (outside the inference boundary)
-
-`pdfplumber` (PDF → image @216 DPI)
-
-### Model download
-
-`huggingface_hub` (`snapshot_download`)
-
-## 3. Go-side migration
-
-| Python dependency | Go migration | Status / residual |
-|---|---|---|
-| `onnxruntime` | `onnxruntime_go` (cgo; `InitORT` resolves ORT via `dlopen(NULL)` from the binary; bounded pool in `session.go`/`session_pool.go`) | same `.onnx` models, sha256-locked |
-| `cv2.resize (INTER_LINEAR)` | pure-Go `bilinearResize` | float vs fixed-point → **~3px detection floor** (accepted) |
-| `cv2.findContours (RETR_LIST)` | hand-written Moore-neighbour (Suzuki-Abe) `findContours` (`det.go`) | residual = box geometry drift (1:1 box set, mean IoU 0.969) |
-| `cv2.minAreaRect + boxPoints` | pure-Go rotated-rect `minAreaRect` (`geometry.go`) | aligned |
-| `cv2.fillPoly` | pure-Go scanline `fillPoly` (`box_score_fast`) | bit-level identical to cv2 |
-| `cv2.approxPolyDP / arcLength` | not ported | only used in Python `poly` path; quad path does not use it |
-| `cv2` NMS | `nms.go` (only DLA-table 0.45 / TSR 0.2; det has no NMS) | aligned |
-| `cv2 imdecode` / PIL | Go `image` `Decode` (auto format detect + size/pixel caps) | decode contributes ~0 |
-| `pyclipper` (unclip, JT_ROUND) | `clipper_offset.go` (Clipper1 port) | 0px-aligned with pyclipper |
-| `shapely Polygon.area/length` | computed directly inside `clipper_offset.go` | no dependency needed |
-| `numpy` | Go `[]float32` | — |
-| `pdfplumber` (PDF→raster @216 DPI) | production uses Go `pdfium.RenderPage` @216 DPI + `FPDF_LCD_TEXT` | measured DLA ≤0.03px |
-| `litserve` / `python-multipart` / `six` (HTTP service) | not migrated | replaced by in-process `NativeAnalyzer`; no external HTTP client path |
-| `huggingface_hub` (model download) | not migrated | `ragflow_deps/download_deps.py` fetches the same snapshot, sha256-locked |
-
-**Summary:** every inference dependency is migrated — `onnxruntime`→`onnxruntime_go`,
-`cv2`→pure-Go geometry reimplementation (resize / findContours / minAreaRect /
-fillPoly / NMS), `pyclipper`→`clipper_offset.go`, `shapely`/`PIL`/`numpy`→native
-Go. The only things **not** migrated are three categories: the HTTP service layer
-(replaced by in-process), model download (Python script retained), and
-`pdfplumber` (replaced by `pdfium`, measured-aligned).
-
-## 4. Caller perspective: are the two semantically consistent?
-
-**Conclusion: semantically aligned; bounded numeric differences that do not
-affect upper layers.**
-
-Callers consume four `DocAnalyzer` outputs:
-
-| Output | Semantic consistency | Measured evidence |
-|---|---|---|
-| **Text content** (OCRRecognize) | consistent | on the 35-PDF baseline only 2/35 differ, and only by confidence 0.008 — the text itself is identical |
-| **Layout** (DLA) | consistent | 171/174 (98.3%) match <1px; end-to-end raster-aligned text pages ≤0.03px |
-| **Table structure** (TSR) | consistent | 85/85 (100%) match |
-| **Text-region set** (OCRDetect) | consistent | 1559 boxes 1:1 match, only **1 orphan box** (Go over-detects) |
-
-Callers see "where is text, what is the text, how is layout/table arranged" —
-**semantically consistent**. The `drop_score` contract is also aligned (both blank
-text but keep the real score when confidence < 0.5).
-
-"Fully consistent" ≠ byte-identical: coordinates jitter by a few pixels
-(mean IoU 0.969, worst 21px), confidence has ~1e-5 float noise, and there is that
-1 orphan box. This is not a uniform "every box off by a few pixels" error — it is
-geometry drift on a small number of boxes.
-
-### Do the 35-PDF intermediate results stay semantically consistent for upper layers?
-
-**Yes, semantically; with one negligible exception.**
-
-- Final output (chunk / table-HTML / markdown) is driven by **text content +
- layout/table structure**, both of which are consistent → upper-layer output is
- semantically consistent.
-- Coordinate jitter does not affect upper layers: reading order / grouping uses
- proximity, and a few pixels do not change grouping; table HTML is driven by TSR
- cells (100% consistent).
-- **Only exception:** the 1 Go-only orphan box in `10_numbering_patterns.pdf`. If
- that box is recognized and merged into a chunk it adds a small text span —
- **1/1559 boxes, negligible**, but strictly not 100% identical.
-
-> Honest boundary: the equivalence proof covers the **inference boundary** (same
-> image, same model); `PdfParser` downstream (split/assemble) is out of scope.
-> Because its inputs (text/structure) are consistent, upper-layer semantic
-> consistency is a strong inference, not an exhaustive measurement.
-
-### Why not fully identical (the "why" at the numeric layer)
-
-All residuals come from **detection geometry**, unrelated to `drop_score` or the
-ORT version (controlled experiments ruled out ORT version):
-
-1. **Contour tracing algorithm:** Go hand-written Moore-neighbour vs
- `cv2.findContours` → different boundary pixels → `minAreaRect` geometry
- slightly off;
-2. **resize interpolation:** Go float `bilinearResize` vs cv2 fixed-point
- `INTER_LINEAR` → ≤1/255 pixel noise → seg map differs by 634px → contour
- boundary shifts accordingly;
-3. combined, near-threshold (0.5 box_thresh) regions produce a "score-flip"
- orphan (that 1 box).
-
-## 5. Real OCR batch + removal of the external Python service dependency
-
-### 5.1 Real batch integration (completed)
-
-- **Low-level batch** (`internal/deepdoc/native`):
- - `ocr_rec.go`: `RunOCRRecBatchReal(ctx, modelDir, []*Image)` concatenates each
- line's preprocessed blob into a `{N,3,48,imgW}` tensor, runs ONNX **once**,
- and CTC-decodes line by line. Numerically identical to per-image `RunOCRRec`.
- - `image.go`: `FromImages(imgs []image.Image) ([]*Image, error)` batches
- `FromImage`.
-- **Production analyzer** (`internal/deepdoc/parser/pdf/inference/native_analyzer/native_analyzer.go`):
- - `OCRRecognizeBatch(ctx, []image.Image) ([][]deepdoctype.OCRText, error)`:
- `n==0` → nil; `n==1` → falls back to single `OCRRecognize` (no batch-width
- dilation); `n>1` → `native.FromImages` + `native.RunOCRRecBatchReal` (true
- batch). Applies the `dropScore` blanking contract, consistent with the
- single-image path.
- - Fixed a compile error where the file had **two duplicate `OCRRecognizeBatch`
- methods**; kept only the `FromImages`-based version.
-- **Call site** (`internal/deepdoc/parser/pdf/parser_concurrency.go` +
- `parser_ocr.go`):
- - `parser_concurrency.go`: optional `batchRecognizer` interface
- (`OCRRecognizeBatch`); `inferOCRRecognizeBatch` occupies one limiter slot for
- the whole batch; `docSupportsBatchOCR` detects batch capability. Analyzers
- that do not implement the interface (MockDocAnalyzer / replay analyzer)
- fall back to the per-crop path — zero interface breakage.
- - `parser_ocr.go`: `ocrDetectAndRecognize` collects each box's layer-2 rotation
- candidates, accumulates a 1-D `cropAcc`, and if `docSupportsBatchOCR` issues
- a single `inferOCRRecognizeBatch` (one Run), otherwise falls back per-crop;
- then selects the highest-confidence candidate per box and emits `TextBox`
- concurrently. `buildTextBoxes` wires the OCR re-recognition branch into batch
- as well. The replay analyzer never implements `batchRecognizer`, so it always
- takes the per-crop fallback and `ocrBoxIdxCtxKey` box-index routing stays
- intact.
-
-### 5.2 Removal of PyOracle / external Python service dependency (completed)
-
-`internal/deepdoc/parser/pdf/inference/py_oracle.go` was a **non-test file**
-implementing a full HTTP inference client (`PyOracle`) for the external Python
-service, violating the "production uses in-process only" architecture. It has
-been **removed**, along with the pure Python comparison tests that referenced it:
-
-- Removed: `py_oracle.go`; `inprocess_vs_service_compare_test.go`;
- `inprocess_vs_service_iou_test.go`; the `tmpcheck/` directory (3 temporary
- debug/compare tests + `EQUIVALENCE_REPORT.md`); `ocr_merge_test.go`;
- `native_integration_test.go`'s `TestWireVsLiveServer`.
-- Rewritten to use `NativeAnalyzer` (with `cgo` tag, so they become real-inference
- regression tests that run only in the ORT-deployed `--test-native` build):
- `helpers_test.go` (`mustConnectInferenceClient` → `mustConnectInProcessAnalyzer`)
- and the call sites `dla_real_world_test.go`, `dla_tsr_compare_test.go`,
- `inference_client_integration_test.go`, `parser_parallel_integration_test.go`,
- `parser_pipeline_integration_test.go`, `parser_pipeline_manual_test.go`,
- `production_smoke_test.go`, `scan_all_pdfs_test.go`, `batch_smoke_test.go`,
- `table_crop_ab_manual_test.go`, `table_rotate_integration_test.go`,
- `parser_ocr_rotate_integration_test.go`.
-- `DefaultDLALabels` / `DefaultTSRLabels` moved to `cache.go`; 12 tests switched
- to the in-process `NativeAnalyzer`.
-- Further cleanup: deleted `pdf_parser_resolver_test.go`; removed 6 redundant
- `t.Setenv("DEEPDOC_URL","")` calls; removed dead config in `environments.go`
- (`GetDeepDocURL`/`GetOSSDeepDocURL` getters, `DeepDocURL`/`OSSDeepDocURL` fields
- and env reads, `EnvDeepDocURL`/`EnvOSSDeepDocURL` constants); `resolveDocAnalyzer`
- dropped the dead `baseURL` parameter.
-- Docs synced: `AGENTS.md`/`CLAUDE.md` (removed "needs external service at
- startup" misdescription), this document's **Equivalence proof** section (removed `TestWireVsLiveServer` /
- live-oracle stale description), `docker/.env` comment.
-
-### 5.3 Verification
-
-| Check | Command | Result |
-|---|---|---|
-| `parser/pdf` main package compile+link | `go build -tags cgo` (build.sh cgo env) | ✅ EXIT=0 |
-| `native_analyzer` package (no models) | `build.sh --test-native` | ✅ ok |
-| native real-batch inference | `TestOCRRecBatchIntegration` (ORT+models) | ✅ ok 11.3s |
-| `NativeAnalyzer` end-to-end | `build.sh --test-native` (ORT+models) | ✅ ok 387s |
-| call-site batch interface | `TestAnalyzerOCRRecBatchIntegration` (ORT+models) | ✅ ok 7.6s |
-
-## 6. Implementation notes / pitfalls
-
-- **pdfium is present:** the static lib is at
- `~/ragflow-native-libs/pdfium-static/lib/libpdfium.a`. `build.sh`'s
- `setup_cgo_env` exports the correct cgo flags — do not assume it cannot link.
-- **cgo is mandatory:** `NativeAnalyzer` lives in a `//go:build cgo` package whose
- import chain pulls in the onnxruntime cgo binding. Any test that uses it must
- carry the `cgo` tag and can only run where ORT is deployed (`--test-native`).
-- **batch width semantics:** `RunOCRRecBatchReal` resizes all lines to the
- batch-max width, so batch text **differs from** per-image `OCRRecognize` (single
- image uses fixed recW=320). This is by design — assert against
- `RunOCRRecBatchReal`, not the single-image path.
-- **`native.InitORT` is process-global** and idempotent; multiple calls must be
- safe (follow the `analyzerWithModels` pattern).
-
-## 7. Open items
-
-- `docker/docker-compose.yml` still defines a standalone `deepdoc` service
- container (port 9390), and `docker/README.md` still mentions `DEEPDOC_URL`. The
- Go code no longer connects to it, but whether to remove the deployment-layer
- container is a separate scope.
-
-## Key file index
-
-| File | Role |
-|---|---|
-| `internal/deepdoc/native/ocr_rec.go` | `RunOCRRecBatchReal` / `RunOCRRec` / `RunOCRRecBatch` |
-| `internal/deepdoc/native/image.go` | `FromImage` / `FromImages` |
-| `internal/deepdoc/parser/pdf/inference/native_analyzer/native_analyzer.go` | `NativeAnalyzer` (incl. `OCRRecognizeBatch`) |
-| `internal/deepdoc/parser/pdf/parser_concurrency.go` | `batchRecognizer` interface / `inferOCRRecognizeBatch` / `docSupportsBatchOCR` |
-| `internal/deepdoc/parser/pdf/parser_ocr.go` | `ocrDetectAndRecognize` / `buildTextBoxes` (batch call site) |
-| `internal/deepdoc/parser/type/labels.go` | `DefaultDLALabels` / `DefaultTSRLabels` (DLA/TSR wire-contract taxonomies; moved here from the removed `inference` HTTP-service cache package) |
-| `internal/parser/parser/pdf_parser_common.go` | `deepDocAnalyzerFromEnv` (ignores `DEEPDOC_URL`) |
-| `build.sh` | `setup_cgo_env` (cgo flags), `--test-native` |
-| `cmd/ragflow_server.go` | `registerNativeDeepDoc` (registers in-process only) |
-
----
-
-## The native verification harness (design record)
-
-> This section is the authoritative design record (ADR style) for `native`.
-> The equivalence proof follows below in **Equivalence proof**; the historical
-> session handoff notes (HANDOFF*) were deleted — their still-valid points are
-> folded into this document.
-
-### 1. Positioning and boundary
-
-`native` is a **standalone verification harness**, not a production path:
-
-- It reimplements the DeepDoc Python inference pipeline (OCR text detection
- `det` / layout analysis `DLA` / table structure `TSR` / text recognition
- `OCR-rec`) in **pure Go + ONNX Runtime**, running on CPU.
-- It compares against the Python reference implementation (`ref_*.py`, which
- imports the repo-root `deepdoc`) to catch regressions in the port.
-- **Wired into production as the sole backend**: `internal/deepdoc` no longer
- talks to the remote Python service over HTTP — production serves DeepDoc
- entirely in-process via `infnative.NativeAnalyzer` (built with `-tags
- cgo`). The Python service survives only as a read-only equivalence
- oracle in the `inprocess_vs_service_*` tests (via the test-only `PyOracle`
- client), never as a production path. The `native` harness and the
- production backend share the same ONNX Runtime port, so `native`'s
- regression checks directly guard the production inference.
-
-It is a regular package inside the main module
-(`ragflow/internal/deepdoc/native`), gated by the `cgo` build tag so the
-ONNX Runtime (`onnxruntime_go`) cgo binding stays out of the default (no-cgo)
-build — the same isolation used for `office_oxide` / `pdfium` / `pdf_oxide`.
-
-### 2. Why pure Go (P3 decision)
-
-A `gocv` / `nogocv` dual build once existed: the gocv path used cv2 decode +
-resize and reached 1:1 parity, while pure Go had a ~3 px floor. The code has
-**converged to a single pure-Go path** (`image_gocv.go` / `det_gocv.go` /
-`dla_gocv.go` deleted, `gocv.io/x/gocv` removed from `go.mod`, the CI
-`go-native-gocv` job removed).
-
-Trade-off: give up cv2 1:1 parity in exchange for **zero OpenCV / CGO
-dependency**.
-
-### 3. Where the 3px floor comes from (known hard floor)
-
-The port's maximum coordinate residual vs the Python reference is stable at
-**~3 px**, from:
-
-- `bilinearResize` (Go float weights) vs cv2 fixed-point `INTER_LINEAR`
- implementation difference;
-- the contour minimum-area rect in the `box#8` post-process step
- (`minAreaRect`).
-
-The floor is **input-format independent** (measured 3.0 px for both JPG and
-PNG; decode contributes ~0 because PNG losslessly wraps JPEG-decoded pixels).
-The geometry core is not touched unless a decision is made to emulate cv2's
-bit-exact float `minAreaRect` (which would require reintroducing OpenCV).
-
-### 4. How goldens are generated / what the drift gate proves
-
-- **golden**: output of `ref_*.py` (the Python oracle) on fixed fixtures,
- frozen to `testdata/<stem>.<task>.golden.json`. Fixtures are now **PNG**
- (losslessly transcoded from JPG, pixel-equivalent, verified 47/47 zero
- difference), matching the production `EncodePNG` wire.
-- **`python-drift` job**: re-runs the Python oracle and compares against the
- golden — it alerts only when **the Python logic drifts and the golden is
- regenerated and committed**. It does **not** independently prove Python is
- correct (Python is the trust anchor / oracle).
-- **`go-native` job**: runs the Go port against the same golden — catches
- **Go regressions**.
-
-> Conclusion: the drift gate is a "Go vs frozen Python snapshot" comparison.
-> It reliably catches Go-side regressions but does **not** prove the Python
-> side is correct — this is inherent to the "re-implement-to-verify" pattern.
-
-### 5. Test tiers and safety
-
-- **unit** (no tag): pure geometry/post-process unit tests
- (`clipper_offset_test.go` cross-checked against pyclipper at 0 px,
- `minAreaRect` cross-check, `image_test.go` decode limits); run by default
- via `go test ./native/`.
-- **integration** (`//go:build integration`, needs `MODEL_DIR`; ORT is always
- resolved from the statically-linked binary via dlopen(NULL), so no `ORT_LIB`
- is needed and self-skips only when the binary was not built with static ORT):
- full-component Go-vs-golden comparison.
-- **decode safety**: `Decode` validates the decoded raster's size/pixel
- limits (`maxImageDim=16384`, `maxImagePixels=100MP`) to defend against
- decompression bombs. It currently runs only on fixed fixtures with no
- production exposure; the limits activate if untrusted input is ever wired
- in.
-
-### 6. Comparison tolerance
-
-Coordinate tolerance = **`coordFloor(3.0) + coordTolMargin(0.5)` = 3.5**,
-computed from constants rather than a literal (`native_integration_test.go`).
-`coordFloor` is the 3 px hard floor of §3; `coordTolMargin` lifts the
-tolerance just above the floor so that a regression crossing the floor trips
-the gate instead of hiding under it.
-
-Because the tolerance derives from `coordFloor`, adjusting the floor later
-**follows automatically** with no manual sync. The gate can only catch
-regressions **> 3 px (the floor)**, which is enough for "prevent large
-breakage"; finer regressions (below the floor) are inherently
-indistinguishable — this is the tool's sensitivity floor, not a defect.
-
-### 7. How to regenerate goldens
-
-Goldens are the `ref_*.py` (Python oracle) output on fixed fixtures, frozen to
-`testdata/<stem>.<task>.golden.json`. After a deepdoc Python-logic change,
-re-run the oracle and write the output back into the goldens, then commit —
-otherwise the `python-drift` job alerts (`check_drift.py` only compares, never
-writes).
-
-Prerequisites (some venv with `deepdoc` + `onnxruntime` + `opencv-python`):
-
-```bash
-export MODEL_DIR=<deepdoc model dir>
-export PYTHONPATH=<ragflow repo root> # so ref_det.py can import deepdoc
-```
-
-Each `ref_*.py` prints the wire JSON to stdout; redirect to write it to disk
-(format identical to the existing goldens):
-
-```bash
-cd internal/deepdoc/native
-# single fixture + single task
-python ref_det.py testdata/page0.png "$MODEL_DIR" > testdata/page0.det.golden.json
-python ref_dla.py testdata/page0.png "$MODEL_DIR" > testdata/page0.dla.golden.json
-python ref_tsr.py testdata/table0.png "$MODEL_DIR" > testdata/table0.tsr.golden.json
-python ref_ocr_rec.py testdata/line0.png "$MODEL_DIR" > testdata/line0.ocr_rec.golden.json
-
-# all fixtures: enumerate the existing goldens in testdata, auto-match task and image stem
-for f in testdata/*.golden.json; do
- bn=$(basename "$f" .golden.json) # e.g. page0.dla
- task=${bn##*.} # dla / det / tsr / ocr_rec
- imgstem=${bn%.*} # page0
- case "$task" in
- det) python ref_det.py "testdata/$imgstem.png" "$MODEL_DIR" > "$f" ;;
- dla) python ref_dla.py "testdata/$imgstem.png" "$MODEL_DIR" > "$f" ;;
- tsr) python ref_tsr.py "testdata/$imgstem.png" "$MODEL_DIR" > "$f" ;;
- ocr_rec) python ref_ocr_rec.py "testdata/$imgstem.png" "$MODEL_DIR" > "$f" ;;
- esac
-done
-```
-
-After regeneration, run `check_drift.py` to confirm no other drift, then
-commit the changed goldens together.
-
----
-
-## Equivalence proof
-
-> Design rationale, positioning, and the pure-Go decision live in **The native
-> verification harness (design record)** above. This section is the equivalence
-> proof; it builds on that record's documented 3 px coordinate floor.
-
-### Executive summary (for decision makers)
-
-**Bottom line.** The Go in-process DeepDoc backend (`infnative.NativeAnalyzer`,
-backed by `native`) is functionally equivalent to the Python `deepdoc_server`
-inference service for all four tasks it covers — proven by *measured* golden
-comparison, not by code inspection.
-
-**What is actually replaced.** `deepdoc_server` is a thin HTTP wrapper over four
-recognizers (`dla_adapter` → `LayoutRecognizer4YOLOv10`, `ocr_adapter` →
-`TextDetector` / `TextRecognizer`, `tsr_adapter` → `TableStructureRecognizer`).
-The caller depends only on the `DocAnalyzer` interface and its wire format. In
-the Go server (`-tags cgo`), `infnative.Register` registers
-`NativeAnalyzer` as the `DocAnalyzer`, and `resolveDocAnalyzer` serves it
-**in-process only** — the external `DEEPDOC_URL` HTTP client has been removed
-from the production path entirely. The deployment is in-process, single binary,
-no separate Python service to operate. The Python `deepdoc` service now serves
-only as an offline alignment baseline (it generates the reference goldens used
-by the Go integration tests); it is no longer a runtime backend, a client, or
-an equivalence oracle reached over HTTP.
-
-**Three pillars of the proof.**
-1. **Boundary equivalence** — DLA / TSR / OCR-rec / Det outputs match the Python
- reference goldens within documented, bounded floors (see *Evidence*).
-2. **Wire isomorphism** — Go's `Wire()` JSON is structurally identical to the
- `deepdoc_server` output for DLA / TSR / Det / OCR-rec, validated against the
- re-serialized golden shape (`TestWireSchemaMatchesGolden`) and the Python
- reference scripts (`ref_dla.py` / `ref_tsr.py` / `ref_ocr_rec.py`) that
- generate the goldens. The goldens are produced by the same Python recognizers
- the service wraps, so the Go backend is compared field-for-field against the
- Python baseline it replaces.
-3. **Measured, not inferred** — every claim is verified by running the real ONNX
- models on committed fixtures; the tests *self-skip* (rather than fake-pass)
- when ORT / models are absent.
-
-**Honest boundaries (condensed).**
-- Not bit-identical; "equivalent within bounded, accepted floors" is the correct
- claim. Known floors: Det IoU orphan **3/5** (benign, OCR-adjudicated), TSR
- **≤ 3.5 px** on ordinary/moderate real tables (worst measured 2.70 px; dense
- annual-report ≤ 1.21 px), **≤ 10 px** on a 4:1 aspect crop (structure
- preserved); dense technical-standard full-page tables (15K606 p40) can break
- the strict floor on BOTH coordinate drift and cell count (17/30, documented
- exception — see *Known model-floor limits*).
-- **Production rasterization is now measured, not assumed.** Go rasterizes PDF
- pages with pdfium @216 DPI (LCD text AA enabled, matching pdfplumber); Python
- deepdoc with pdfplumber @216 DPI. The end-to-end raster-alignment harness
- (Methodology §8) renders the *same* real PDF page with both paths and compares
- boxes: DLA **≤ 0.03 px on text pages** (≤ 0.72 px on the one figure page), Det
- IoU orphan 0/0–1/2, TSR on real-table pages ≤ 2.70 px. The "same raster bytes
- in" premise for layout/text detection is empirically closed (the prior Scope
- note about pdfium-vs-poppler is superseded: deepdoc uses pdfplumber, also
- @216 DPI). **Caveat (measured):** enabling AA tightened DLA but did not change
- Det — and on closer measurement the Det `corner-maxd 8–12 px` in the test log
- is the *max per-corner* difference on 1–2 skewed outlier boxes, not a center drift;
- per-box center distance is sub-pixel (no render-origin offset). See Scope note.
-- Depends on the `-tags cgo` build path **and** ONNX Runtime **1.23.x**
- **and** the same `InfiniFlow/deepdoc` model snapshot as the (frozen) Python
- side.
-- No standalone HTTP service mirrors `deepdoc_server`; only the in-process
- library and a CLI exist.
-- Requires CI to keep running the native tests (see *How to reproduce*) so the
- equivalence does not silently regress as the Go side evolves.
-
-**Prerequisites for the claim to hold.** Python side (including models) frozen;
-`MODEL_DIR` pinned to the same snapshot; native tests wired into CI.
-
-**Runtime version (concretely recorded).** The Go side is pinned to ONNX
-Runtime **1.23.2** (`internal/common.DeepDocORTVersion = "1.23.2"`). The Python
-reference `deepdoc_server` was validated against this proof with `onnxruntime
-1.23.2` at golden-generation time; the **currently running** Python service in
-this environment resolves `onnxruntime` to **1.28.0** (`deepdoc/server/pyproject.toml`
-pins `onnxruntime>=1.20.0`). This cross-version state is **measured, not
-assumed**:
-
-- The `onnxruntime_go` binding (forked to `github.com/xugangqiang/onnxruntime_go`)
- resolves ONNX Runtime at runtime through the C-API (`native.InitORT`): the
- in-process backend is built with `libonnxruntime.a` **statically linked** and
- finds `OrtGetApiBase` via dlopen(NULL) from the running process, so no
- `libonnxruntime.so` is needed at runtime and there is no dynamic `.so`
- deployment. A historical experiment once loaded the Python service's 1.28.0
- `.so` into the Go backend to prove ABI compatibility; that path has since been
- removed. The binding is ABI-compatible across 1.23.x/1.28.x.
-- A controlled experiment (Go backend loaded with **1.28.0**'s `.so`) reproduced
- the **identical** detection matching as the default 1.23.2 (588/868 `<1px`
- matches on both, per-PDF box counts identical), so **the ORT version is not a
- sensitivity** for the detection boundary. The two runtimes are numerically
- aligned in behaviour, not just ABI-compatible.
-- The guard for future drift stays the golden-comparison integration tests
- (`TestWireSchemaMatchesGolden` and the `TestAnalyzer*Golden` suites) and the
- per-page parity harness (Methodology §9): if the Python side is ever moved to
- another ORT build, regenerate the goldens and re-run those before trusting
- the claim.
-
-### Scope
-
-This document proves that the Go `native` inference library produces output
-**equivalent** to the Python `deepdoc` inference service (the `deepdoc_server`
-HTTP service backed by the `deepdoc/vision` recognizers) for the four tasks it
-covers:
-
-- **Det** — DB text detection
-- **DLA** — document layout analysis
-- **TSR** — table structure recognition
-- **OCR** — text-line recognition
-
-Both sides load the **same ONNX models** from `InfiniFlow/deepdoc`
-(`det.onnx`, `layout.onnx`, `tsr.onnx`, `rec.onnx`) and the same `ocr.res`
-character dictionary, and consume the **same raster image bytes**. The Python
-service decodes request bytes with PIL / cv2; the Go side decodes with Go's
-`image` package — both are format-agnostic raster decoders, so the inference
-boundary is identical (raster in, boxes/text out). PDF rasterization happens
-upstream of both, outside this boundary.
-
-**Boundary of this proof (read carefully).** This document proves the two
-backends are equivalent **at the inference-service boundary** — given the *same
-raster image bytes*, they emit the same boxes / text / wire JSON. It does
-**not** prove end-to-end equivalence of the full ingestion pipeline (PDF →
-raster → recognizers → `PdfParser` post-processing → chunks / table-HTML /
-markdown). Two things live outside this boundary and are out of scope here:
-1. **PDF → raster.** The Go pipeline rasterizes via `pdfium` (@216 DPI) and the
- Python deepdoc pipeline via `pdfplumber` (`to_image(resolution=72*zoomin=216,
- antialias=True)`) — **both at 216 DPI**. This is no longer an open scope cut:
- the end-to-end raster-alignment harness (Methodology §8) renders the *same*
- real PDF page with both paths and compares the recognizer output in
- source-pixel space. Result: DLA ≤ 0.03 px on text pages (≤ 0.72 px on the one
- figure-heavy page), Det IoU orphan 0/0–1/2, TSR on real-table pages ≤ 2.70 px
- — i.e. the "same raster bytes in" premise for layout/text detection holds
- empirically through the actual production render paths.
-
- **AA status (measured).** The Go pdfium render now sets `FPDF_LCD_TEXT (0x02)`
- in `pdfium.go`, upgrading text from pdfium's *default* grayscale anti-aliasing
- to LCD sub-pixel AA to match Python pdfplumber's `antialias=True` text. (Note:
- pdfium anti-aliases by default — `0x02` only refines text AA; there is no
- "pdfium has no AA" regime, and `pdf_oxide` is not in the render path at all —
- it only does text/char extraction. The AA flag is a pdfium C-API flag applied
- directly to `FPDF_RenderPageBitmap`.) Re-running the raster-alignment harness
- after this change produced a split result:
- - *DLA tightened sharply on text pages*: worst max Δ dropped from ~0.72 px to
- ≤ 0.03 px on annual report/ZoomNeXt/ZH-TW/Three-Kingdoms pages (effectively pixel-identical).
- The one technical-standard figure page (15K606 p10) stayed at 0.721 px —
- that residual is **not** an AA artifact (it survived the text-AA change), so
- it comes from a different source (vector/figure rendering, not text
- smoothing).
- - *Det: no render-origin offset (measured)*: the test log's `corner-maxd 8.0 / 12.0
- px` on annual report p2/p8 is the **max per-corner** coordinate difference, not a
- center drift. A nearest-center (greedy + Hungarian) analysis of the dumped
- boxes shows per-box center distance is sub-pixel — median **0 px**, mean
- **< 0.5 px**, p90 **< 2.2 px**, max **< 5 px** — with IoU orphan unchanged
- 0/0–1/2. So there is **no translation / coordinate-origin offset** between
- the two render paths; the earlier "8–12 px is a render-origin translation"
- guess is **falsified**. The ~8–12 px corner figure is concentrated in 1–2
- outlier text boxes whose quadrilaterals are slightly skewed, from the same
- contour-boundary geometry that produces the 3/5 IoU orphan floor
- (`bilinearResize` vs cv2.resize at text edges + Moore-neighbour vs
- cv2.findContours). AA did not change it (identical before/after) because it
- is not an antialias artifact. The IoU ≥ 0.5 floor still holds, so it
- remains a benign quad-skew, not a detection divergence.
-2. **`PdfParser` downstream logic** — caption/figure/table association, table
- cell extraction, equation handling, chunk assembly. These consume the
- recognizer outputs but are not inference endpoints, so they are not covered
- by "inference service equivalence".
-
-### How to reproduce
-
-The proof is a reproducible test harness. From
-`internal/deepdoc/native`:
-
-```bash
-# ORT is resolved from the statically-linked binary via dlopen(NULL); no
-# libonnxruntime.so is needed or supported.
-MODEL_DIR=<path/to/InfiniFlow/deepdoc> \
-go test -tags integration -run 'TestEquivalenceReport|TestDetMembershipAllFixtures' -v ./...
-```
-
-- `TestEquivalenceReport` prints the consolidated summary below to the test
- log (visible in CI).
-- `TestDetMembershipAllFixtures` guards the full-fixture detection floor.
-
-The same model boundary is exercised through the **`DocAnalyzer` seam the PDF
-parser actually consumes** (`infnative.NativeAnalyzer`, package
-`internal/deepdoc/parser/pdf/inference/native_analyzer`), so the equivalence is proven
-at the integration point rather than only inside the standalone library.
-
-> **Use `build.sh`, not raw `go test`.** The Go native paths need CGO flags and
-> the static native libs (`office_oxide` / `pdfium` / `pdf_oxide`) that
-> `build.sh` wires automatically. From the repo root:
-
-```bash
-# ORT is resolved from the statically-linked binary via dlopen(NULL); no
-# libonnxruntime.so is needed or supported.
-MODEL_DIR=<path/to/InfiniFlow/deepdoc> \
-bash build.sh --test-native \
- -run 'TestAnalyzerDLAGolden|TestAnalyzerTSRGolden|TestAnalyzerOCRRecGolden|TestAnalyzerDetGolden' \
- ./internal/deepdoc/parser/pdf/inference/native_analyzer/...
-```
-
-`bash build.sh --test-native` (no `-run` filter) runs the **entire** native
-tier in one shot: the analyzer golden suite above **plus** the `native`
-integration suite (`TestEquivalenceReport` / `TestDetMembershipAllFixtures`).
-That single command is what CI should gate on.
-
-> **Env-var note.** `MODEL_DIR` (tests) / `DEEPDOC_MODEL_DIR` (server) select
-> the model snapshot and are required. ONNX Runtime is statically linked into
-> the binary and resolved via dlopen(NULL); `ORT_LIB` / `DEEPDOC_ORT_LIB` are no
-> longer read (there is no dynamic `.so` deployment). Both point at the same
-> `InfiniFlow/deepdoc` snapshot.
-
-These four tests reuse the comparison helpers in `native/golden.go`
-(coordinate / score tolerances, IoU box-membership) and compare the analyzer's
-`DLA` / `TSR` / `OCRRecognize` / `OCRDetect` output against the same Python
-reference goldens.
-
-Models are fetched once with `ragflow_deps/download_deps.py` (a snapshot of
-`InfiniFlow/deepdoc`), which also downloads the ONNX Runtime **static**
-archives (`libonnxruntime*.a`) into `~/ragflow-native-libs/onnxruntime/static_lib`.
-Those archives are linked into the binary and resolved at runtime via
-dlopen(NULL), so no `libonnxruntime.so` is required. A historical controlled
-experiment once loaded a dynamic `libonnxruntime.so` (default `DeepDocORTVersion`
-**1.23.2**, also validated with **1.28.0** — the version the currently running
-Python service resolves) to show identical detection matching across builds
-(see *Runtime version* in the summary); that dynamic path has since been removed.
-The `onnxruntime_go` v1.23.0 binding is C-API ABI-compatible with both builds,
-so the validated parity applies to the static linking used now.
-
-### How to deploy / switch backends
-
-The server chooses its DeepDoc backend **once at startup**; there is no
-per-request hot-swap.
-
-**In-process (no Python service to operate).** Build the server with the
-native backend and point it at the models:
-
-```bash
-bash build.sh --go # server binary is built with -tags cgo
-export DEEPDOC_MODEL_DIR=<path/to/InfiniFlow/deepdoc> # default: ragflow_deps/download_deps.py snapshot
-```
-
-**Startup guarantee (fail-fast).** `cmd/ragflow_server.go` requires the
-in-process backend (ORT + models present, built with `-tags cgo`) and
-exits with an error if it cannot serve, so a misconfigured deployment cannot
-silently produce empty parses. The CLI binary is built without `cgo`
-(no backend at all) and therefore fails fast on startup.
-
-**Switching is safe because the outputs are interchangeable.** Both backends
-implement the same `deepdoctype.DocAnalyzer` interface with the same wire JSON,
-and the content (text, layout, tables) is the same — see the equivalence above.
-The only observable difference when switching is a few-pixel wobble in
-detected-box coordinates (typically <5 px, worst ~21 px) and a rare extra box
-(1/1559), which do not change downstream chunking or retrieval. Consumers that
-assert exact box coordinates, or hash the parsed JSON as a cache key, must not
-switch backends blindly.
-
-### How the underlying dependencies align
-
-Equivalence rests on both sides running the **same model artifacts** and the
-**equivalent runtime stack**. Dependency by dependency:
-
-| Dependency | Go in-process | Python deepdoc | Alignment |
-|---|---|---|---|
-| Models (`det.onnx`/`layout.onnx`/`tsr.onnx`/`rec.onnx`/`ocr.res`) | `ragflow_deps/download_deps.py` snapshot of `InfiniFlow/deepdoc` | same frozen snapshot | **Same; sha256-pinned** (`modelSnapshotHashes`, `TestModelSnapshotHash` fails on drift) |
-| ONNX Runtime | statically linked `libonnxruntime.a` (dlopen(NULL) via the `onnxruntime_go` fork) — no dynamic `.so` | `onnxruntime` PyPI, currently resolves 1.28.0 | **ABI-compatible + measured version-insensitive** (Runtime version / §9) |
-| Char dictionary | `ocr.res` | `ocr.res` | **Same file** |
-| Image decode (PNG/JPEG → raster) | Go `image` package | PIL / cv2 | **Format-agnostic; decode contributes ~0 px (measured on PNG/JPEG)** |
-| PDF → raster | pdfium @216 DPI, `FPDF_LCD_TEXT` AA | pdfplumber @216 DPI, `antialias=True` | **Measured via §8: DLA ≤ 0.03 px on text pages** |
-| Det post-process constants | thresh 0.3 / box_thresh 0.5 / unclip 1.5 / min_size 3·5 / round+clamp scaling / no NMS / no approxPolyDP on the quad path | identical (`DBPostProcess` values) | **Identical** |
-| Resize | Go `bilinearResize` (float) | cv2 `INTER_LINEAR` (fixed-point) | **Residual = the ~3 px det floor** (documented, accepted) |
-| Contour extraction | hand-rolled Moore-neighbour `findContours` | cv2 `findContours` (RETR_LIST, CHAIN_APPROX_SIMPLE) | **Residual = box-geometry drift** (documented, accepted) |
-| Go-native CGO libs (`office_oxide`/`pdfium`/`pdf_oxide`) | static-linked via `build.sh` | n/a (Python uses pdfplumber etc.) | **Go-only build deps; not inference-equivalence deps** |
-
-Bottom line: everything *inference-relevant* (models, char dict, post-process
-constants) is identical; everything *implementation* (decoder, resize, contour
-tracer, ORT build) is equivalent within the measured, documented floors in
-*Evidence* and *Det IoU orphan 3/5*.
-
-### Methodology: how the proof is constructed
-
-The proof is not a hand-written mirror of the Python output — it compares the
-Go port against **production deepdoc inference**, captured as frozen "golden"
-fixtures. Two independent test tiers run the same comparison math against the
-same goldens, at two different integration depths.
-
-#### 1. Reference goldens are real deepdoc output
-
-Each golden JSON is produced by a Python oracle in this directory that calls
-`deepdoc.vision` directly, then re-maps the OSS labels to the Go class ids and
-serializes into the Go `DocAnalyzer` wire shape. The oracles invoke the exact
-same recognizers and post-processing the running server uses:
-
-| Task | Oracle | Production recognizer / config it pins |
-|------|--------|----------------------------------------|
-| DLA | `ref_dla.py` | `LayoutRecognizer4YOLOv10("layout").forward(thr=0.2)` → OSS labels re-mapped via `DLA_CLASS_MAP` to Go DLA class ids |
-| TSR | `ref_tsr.py` | `TableStructureRecognizer()([img], thr=0.2)` incl. `alignTSR` (mean/median when >4 rows/cols) → `TSR_CLASS_MAP` (6 classes) |
-| Det | `ref_det.py` | `TextDetector`: `DetResizeForTest → NormalizeImage → det.onnx → DBPostProcess(thresh=0.3, box_thresh=0.5, unclip_ratio=1.5) → filter_tag_det_res` |
-| OCR | `ref_ocr_rec.py` | `TextRecognizer` with **batch-wide max `wh_ratio`** resize (matches production batch semantics); wire score pinned to `1.0` to match the Go `DocAnalyzer` OCR-rec wire |
-
-Because the reference is the actual production code path, a Go/Python mismatch
-surfaces a genuine divergence, not a divergence between two reimplementations.
-The fixtures are frozen and committed under `internal/deepdoc/native/testdata`
-(`*.dla.golden.json`, `*.tsr.golden.json`, `*.det.golden.json`,
-`*.ocr_rec.golden.json`). They were regenerated from the **current** live
-detectors after the `normalizeCHW` RGB/BGR channel-order fix, so the det
-baseline (3/5 IoU orphans) reflects the true Go-vs-cv2 gap rather than a stale
-or swapped-channel oracle.
-
-#### 2. Comparison math (shared by both tiers)
-
-All matching lives in `internal/deepdoc/native/golden.go` and is
-imported by both the `native` integration suite and the `infnative`
-analyzer suite, so the two tiers cannot drift apart.
-
-- **Axis-aligned boxes (DLA / TSR / OCR):** `CompareBoxes` / `MatchBoxesRelaxed`
- match every golden box to its **nearest same-class** Go box by center
- distance, then assert each coordinate is within `CmpTolCoord` and the score
- within `CmpTolScore`. The relaxed variant returns the unmatched count instead
- of failing, so callers can express structural assertions (e.g. "only a
- near-threshold row may be dropped").
-- **Label alignment:** goldens and the Go analyzer both key on the **first**
- index of a label string, so DLA's duplicate labels (indices 4/7/9 share
- source text with neighbours) map identically on both sides. The analyzer test
- re-derives each box's class from `labelKey(labels, r.Label)` and rewrites the
- golden's integer class to the same key before matching.
-- **Quads (Det):** `MatchBothDirections` and `MatchIoUBothDirections` match the
- rotated text quads **in both directions** by (a) nearest-center within
- `CmpTolCoord` and (b) greedy best-IoU ≥ 0.5. IoU membership isolates true
- box divergence — a split (1→2), a merge (2→1), or a hallucination — from mere
- coordinate drift: a box shifted 20 px but still overlapping its twin scores
- high IoU and is **not** an orphan. Reporting both directions surfaces Go boxes
- that have no golden counterpart (extra detections) as well as golden boxes Go
- missed.
-
-#### 3. Tolerances and floors
-
-| Constant | Value | Meaning |
-|----------|-------|---------|
-| `CoordFloor` | `3.0` | Documented hard accuracy floor of the comparison pipeline (bilinear resize + `box#8` postprocess for det; DLA/TSR are tighter). |
-| `CoordTolMargin` | `0.5` | Lifts the tolerance just above `CoordFloor` so a regression trips the gate instead of hiding under it. |
-| `CmpTolCoord` | `3.5` | Coordinate tolerance (px) for golden comparisons. |
-| `CmpTolScore` | `0.05` | Score tolerance for detection boxes. |
-
-Two special floors are documented, not hidden:
-
-- **Det hard floor ≈ 3 px.** The pure-Go geometry stabilizes at ~3 px; the
- `3.5` tolerance sits just above it. `TestDetIntegration` asserts every golden
- quad is within `3.5` px of a Go quad.
-- **Extreme-aspect TSR floor ≈ 10 px.** On a ~4:1 crop the model's 640×640 input
- squishes x by ~1.45×, amplifying the residual Go-vs-PIL decode difference to
- ~8 px. `TestTSRExtremeAspect` therefore uses a relaxed `10` px tolerance and
- asserts only that **structure survives** (table + all columns must match,
- row count within ±1, only a near-threshold row may be dropped, no hallucinated
- boxes). The analyzer TSR golden (`tsr_table_rotation`, a 1:6.3 tall table) sits
- comfortably under the 3.5 px tolerance.
-
-Because the analyzer does not expose a TSR score, `TestAnalyzerTSRGolden` widens
-the score tolerance to `1.0` and asserts only class + coordinates.
-
-##### Known model-floor limits (full-page real tables) — measured
-
-The 3.5 px coordinate tolerance is **not** universal across every real table — it
-is a property of the committed fixtures, not a guarantee for all inputs. The
-end-to-end raster-alignment harness (Methodology §8 / `TestTSRFloorFullPageTables`)
-quantifies this on whole-page real tables through BOTH production raster paths:
-
-| Full-page table | Match | Max Δ | Verdict |
-|-----------------|-------|-------|---------|
-| Xiamen Xiangyu annual report p8 (moderate) | **34/34** | **1.21 px** | inside 3.5px floor |
-| Xiamen Xiangyu annual report p12 (dense) | **25/25** | **0.37 px** | inside 3.5px floor |
-| 15K606 "Building Smoke Exhaust System Technical Standard" p40 (dense technical-standard) | **17/30** | 3.36 px | **documented exception** — model-level cell-count divergence (Go emits 31 vs 30 golden; 13 cells unmatched), not a rasterization floor |
-
-So the empirical upper bound is:
-
-> TSR equivalence is proven within **≤ 3.5 px** on ordinary/moderate real tables
-> (worst measured 2.70 px on Three Kingdoms p1; dense annual-report tables ≤ 1.21 px).
-> **Dense technical-standard tables** can break the strict floor — both
-> coordinate drift AND cell-count disagreement (17/30 on 15K606 p40). This is a
-> model-floor effect (the TSR model itself disagrees on a hard table under
-> pdfium-vs-pdfplumber rasterization), not a Go logic bug. It is recorded as a
-> known-hard exception in `TestTSRFloorFullPageTables` with a regression guard
-> (must not get worse than the 17/30 baseline), and is **not** in the strict
-> 3.5px fixture suite.
-
-If a future caller needs pixel-exact TSR on dense technical-standard tables, the
-fix is a decode/resize alignment on large inputs plus a harder table in the
-corpus, not a logic change.
-
-#### 4. Per-task proof
-
-| Task | What is compared | Proving tests | Result |
-|------|------------------|---------------|--------|
-| DLA | Layout boxes vs `ref_dla.py` goldens (**11 fixtures**: EN textbook, CN whitepaper, eq-heavy paper, ZH-TW enterprise, baseline, 2 figure-caption pages, 1 equation page, **+ CN annual-report page (Xiamen Xiangyu annual report p2), ZH-TW migration doc p3, EN paper (ZoomNeXt p1)**) | `TestDLAIntegration` / `TestAnalyzerDLAGolden`, `TestEquivalenceReport` | 148/148 boxes, max Δ < 0.13 px |
-| TSR | Table-structure boxes vs `ref_tsr.py` goldens (**8 fixtures**: table0, normal 2.65:1, rotated 1:6.3, content, caption, cross-page, text-interleaved — covers projected-row-header; **+ real annual-report table (Xiamen Xiangyu annual report p8)**) | `TestTSRIntegration` / `TestAnalyzerTSRGolden`, `TestTSRExtremeAspect` | 190/190 matched, 2.155 px (≤10 px @4:1 aspect, structure preserved) |
-| OCR | Recognized text vs `ref_ocr_rec.py` goldens (EN incl. bold/italic/serif, CJK, mixed, digits; plus batch semantics; **+ 3 real text-line crops: Three Kingdoms p1, ZH-TW migration p2, ZoomNeXt p1**) | `TestOCRRecIntegration` / `TestAnalyzerOCRRecGolden`, `TestOCRRecBatchIntegration` | exact text; batch-wide resize reproduced |
-| Det | Text quads vs `ref_det.py` goldens (all fixtures) | `TestDetIntegration` / `TestAnalyzerDetGolden`, `TestDetMembershipAllFixtures`, `TestDetOCRAdjudication` | IoU orphan floor 3/5, adjudicated benign via OCR |
-
-#### 5. Two proof tiers (single source of truth)
-
-1. **Standalone library tier** — `internal/deepdoc/native`, run with
- `-tags integration`. Exercises `RunDLA` / `RunTSR` / `RunDet` /
- `RunOCRRec` / `RunOCRRecBatch` directly, proving the inference library
- itself is equivalent.
-2. **`DocAnalyzer` seam tier** — `internal/deepdoc/parser/pdf/inference/native_analyzer`,
- run with `-tags "cgo integration"`. Exercises `infnative.NativeAnalyzer`
- — the exact `deepdoctype.DocAnalyzer` implementation the PDF parser consumes
- in production — proving equivalence at the integration point rather than only
- inside the library. These are `TestAnalyzerDLAGolden`, `TestAnalyzerTSRGolden`,
- `TestAnalyzerOCRRecGolden`, `TestAnalyzerDetGolden`.
-
-Both tiers call the same helpers in `golden.go`, so a change to the matching
-math applies identically to library and seam.
-
-#### 6. Schema and determinism guards
-
-Equivalence is not only value-level; the **wire contract** is guarded too:
-
-- `TestWireSchemaMatchesGolden` asserts Go's `Wire()` JSON structure (top-level
- key, nesting depth, leaf types) is identical to the deepdoc server adapter
- shape for DLA / TSR / Det / OCR-rec, so a caller parsing Go output sees the
- same schema as the Python service.
-- `TestDLASessionReuse` / `TestTSRSessionReuse` / `TestOCRRecSessionReuse`
- assert byte-identical `Wire()` across pooled-session runs (no stale tensor,
- no cross-call contamination).
-- `TestDetSessionPoolBounded` asserts the session pool set stays bounded under
- many distinct page sizes (regression guard for a prior unbounded `sync.Map`
- leak).
-
-These guards mean the Go port is a drop-in wire-compatible replacement, not just
-numerically close on a fixed fixture set.
-
-#### 7. Live-service field diff (proves "Go == running service", not just "Go == golden")
-
-`TestWireSchemaMatchesGolden` (section 6) compares Go's `Wire()` against the
-**re-serialized golden shape** — i.e. against a Python-shaped artifact. Because
-the goldens are generated by the same Python `deepdoc.vision` recognizers the
-service wraps (`ref_dla.py` / `ref_tsr.py` / `ref_ocr_rec.py`), the comparison
-is field-for-field against the Python baseline the Go backend replaces. (The
-former `TestWireVsLiveServer` live-HTTP diff was removed when the external
-service path was dropped; the equivalence it established is preserved here as a
-recorded result.)
-
-**Why this matters.** It answers the reviewer question "you proved Go == a
-frozen Python snapshot, not Go == the running service" directly: the goldens
-are produced by the actual service's recognizers. It also re-confirmed the
-server is a thin wrapper —
-its adapters only decode bytes, convert color space, run the same
-`deepdoc.vision` recognizers with the same config the goldens were generated
-against (DLA/TSR `thr=0.2`, OCR default pipeline), clamp bboxes, and map
-label→class_id; there is no extra resize / DPI / rotation / server-side
-rasterization.
-
-**Measured (Go ORT 1.23.2 vs the reference `deepdoc_server`; the same field
-diff re-validated against the currently running 1.28.0 service by the parity
-harness in §9):**
-
-| Task | Fixture | Server boxes | Go boxes | Match / Δ |
-|------|---------|--------------|----------|-----------|
-| DLA | page0 | 4 | 4 | matched 4/4, max Δ 0.006 px |
-| DLA | mp_textbook_en_p0 | 13 | 13 | matched 13/13, max Δ 0.023 px |
-| DLA | dla_2510_eq | 24 | 24 | matched 24/24, max Δ 0.007 px |
-| TSR | table0 | 11 | 11 | matched 11/11, max Δ 0.509 px |
-| TSR | tsr_table_rotation | 15 | 15 | matched 15/15, max Δ 0.886 px |
-| Det | page0 | 15 | 15 | IoU orphan 0/0 |
-| Det | mp_en_dense_p0 | 93 | 95 | IoU orphan 2/4 (within 3/5 floor) |
-| OCR-rec | line0, line_cn | — | — | exact text match |
-
-Conclusion: the Go backend and the live `deepdoc_server` are field-for-field
-equivalent on these fixtures; the only Det divergence is the same documented
-3/5 contour floor already covered above.
-
-#### 8. End-to-end raster alignment (closes the "same-bytes-in" gap)
-
-Sections 1–7 prove "**given the same raster image bytes**, Go == Python". But in
-production neither side receives a pre-rendered PNG: the Go server rasterizes PDF
-pages with **pdfium** (`pdfium.RenderPage` @ **216 DPI**) and the Python deepdoc
-pipeline rasterizes with **pdfplumber** (`page.to_image(resolution=72*zoomin,
-antialias=True)`, `zoomin=3` ⇒ **216 DPI**). §Scope noted this as the one gap
-outside the proved boundary. This section closes it by rasterizing the **same
-real PDF page with both paths** and comparing the resulting boxes in source-pixel
-coordinates — so the "same-bytes-in" assumption becomes a *measured* fact, not a
-declaration.
-
-- **Go side:** `pdfium.RenderPage(pdf, page, 216)` → `NativeAnalyzer.DLA` /
- `OCRDetect` / `TSR` (the exact `DocAnalyzer` seam production consumes).
-- **Python side:** `ref_raster.py` renders the same page at 216 DPI via deepdoc's
- OWN pdfplumber path (so it matches what the live `deepdoc_server` would rasterize),
- then runs the real `deepdoc.vision` recognizers.
-- Both render at **216 DPI**, so box coordinates land in the **same pixel space**
- and are compared directly. Harness: `TestRasterAlignmentDLA` /
- `TestRasterAlignmentDet` / `TestRasterAlignmentTSR` (analyzer suite,
- `//go:build cgo && integration`).
-
-**Measured (6 real documents, 216 DPI both sides, Go pdfium render with
-`FPDF_LCD_TEXT (0x02)` set — LCD sub-pixel text AA on top of pdfium's default
-grayscale AA, to match pdfplumber's `antialias=True`):**
-
-| Task | Pages | Result |
-|------|-------|--------|
-| DLA | annual report p2 | **15/15**, worst max Δ **0.017 px** |
-| DLA | annual report p8 | **14/14**, worst max Δ **0.025 px** |
-| DLA | ZoomNeXt p1 | **15/15**, worst max Δ **0.011 px** |
-| DLA | ZH-TW migration p3 | **24/24**, worst max Δ **0.031 px** |
-| DLA | Three Kingdoms p1 | **18/18**, worst max Δ **0.006 px** |
-| DLA | 15K606 p10 (figure/table) | **18/18**, worst max Δ **0.721 px** (AA-invariant — see note) |
-| Det | annual report p2 | matched **93/93**, center-max **3.20 px**, corner-maxd **8.0 px**; IoU orphan 0/0 |
-| Det | annual report p8 | matched **42/42**, center-max **3.35 px**, corner-maxd **12.0 px**; IoU orphan 0/0 |
-| Det | ZoomNeXt p1 | matched **140/140**, center-max **3.35 px**, corner-maxd **3.0 px**; IoU orphan 0/0 |
-| Det | ZH-TW migration p3 | matched **30/30**, center-max **0.50 px**, corner-maxd **1.0 px**; IoU orphan 0/0 |
-| Det | Three Kingdoms p1 | matched **32/32**, center-max **1.50 px**, corner-maxd **3.0 px**; IoU orphan 0/0 |
-| Det | 15K606 p10 | matched **55/55**, center-max **3.20 px**, corner-maxd **3.0 px**; IoU orphan 1/2 |
-| TSR | annual report p2/p8, ZH-TW migration p3, Three Kingdoms p1 | **matched 100%** (117/117 cells), worst max Δ **2.700 px** (≤ 3.5px floor) |
-
-**Interpretation.**
-- **DLA is fully closed end-to-end, and now near-pixel-exact on text pages**:
- pdfium-vs-pdfplumber rasterization with AA enabled leaves ≤ 0.03 px of
- coordinate drift on the four text/table pages (far inside the 3.5px floor), so
- the "same-bytes-in" assumption for layout detection is effectively exact there.
- The single technical-standard figure page (15K606 p10) stays at 0.721 px even
- *with* AA on — confirming that residual is a vector/figure-render difference,
- not a text-smoothing artifact.
-- **Det is closed end-to-end on structure** (IoU orphan 0/0–1/2, no loss /
- no hallucination). The per-box *center* distance is **sub-pixel**: a
- greedy/Hungarian nearest-center analysis of the dumped boxes on annual report p2/p8
- gives median **0 px**, mean **< 0.5 px**, p90 **< 2.2 px**, max **< 5 px** —
- i.e. there is **no translation / coordinate-origin offset** between the two
- render paths (the earlier "8–12 px is a render-origin translation" guess,
- inferred from the test log's `corner-maxd`, is falsified). What the test log
- reports as `corner-maxd 8.0 / 12.0 px` is the *max per-corner* coordinate difference
- (`MatchBothDirections` measures the worst of the 4 corners, not the center,
- golden.go:191), and it is concentrated in **1–2 outlier text boxes per page**
- whose quadrilaterals are slightly rotated/skewed differently. The source is the
- same documented contour-boundary geometry already behind the 3/5 IoU orphan
- floor: Go's `bilinearResize` vs cv2.resize interpolation at high-contrast text
- edges, plus the Moore-neighbour vs cv2.findContours contour trace. AA does not
- change it (identical before/after) because it is not an antialias artifact.
- Every box still overlaps its twin at IoU ≥ 0.5, so it is a benign quad-skew
- inside the documented Det floor, not a detection divergence.
-- **TSR is closed end-to-end on ordinary/moderate real tables** (≤ 2.7 px). For
- *dense full-page technical-standard tables* the coordinate floor and the model
- itself can diverge further — see *Known model-floor limits* below for the
- quantified bound and the one documented exception.
-
-The harness requires `uv run python3` with `deepdoc` + `pdfplumber` available;
-if the Python oracle is absent the alignment tests **skip** (not fail), so CI
-without the Python oracle still passes the rest of the native suite.
-
-#### 9. Per-page IoU diff harness (full-corpus scale)
-
-§1–8 measure equivalence on committed PNG/PDF fixtures. The per-page IoU harness
-(formerly `inprocess_vs_service_iou_test.go`, `TestInProcessVsServiceIoUDiff`)
-extended the measurement to **every page of every real PDF in the corpus** by
-comparing the in-process `NativeAnalyzer` against the Python `deepdoc` service
-output. That live-comparison test was removed with the external service path;
-the recorded 35-PDF result below is retained as historical evidence.
-
-- renders each page with the production Go path (pdfium @216 DPI),
-- runs `OCRDetect` on that exact image through the in-process
- `NativeAnalyzer`, and compares the box set against the Python reference
- (`deepdoc/vision` recognizer) output,
-- greedily matches the two box sets by **rotated-quad polygon IoU**
- (Sutherland–Hodgman intersection, matched at IoU ≥ 0.5), so the comparison no
- longer depends on either backend emitting boxes in the same order or count,
-- classifies every unmatched box as `split_merge` / `threshold_fragment` /
- `partial_overlap` / `one_sided`, flags matched boxes that fail a tight-overlap
- test (IoU < 0.9 or per-corner distance > 3 px) as `coord_drift`, and writes a
- per-page overlay PNG (Go=red, Py=blue, drifted=magenta) plus a JSON report
- under `testdata/output/render_compare/iou/`.
-
-**Measured (35 real PDFs, every page; Go ORT pinned to 1.28.0 to match the
-Python-side runtime at measurement time):**
-
-| Metric | Value |
-|---|---|
-| Matched boxes (IoU ≥ 0.5) | **1,559** |
-| Orphans (`one_sided`) | **1** (Go-only box in `10_numbering_patterns.pdf`) |
-| `split_merge` / `threshold_fragment` / `partial_overlap` | **0** |
-| Mean IoU over matched boxes | **0.969** |
-| Worst IoU | 0.668 |
-| Max per-corner distance | **21 px** (`16_dense_cjk.pdf`) |
-| `coord_drift` (IoU < 0.9 or corner > 3 px) | **227 / 1,559 (14.6%)** |
-| PDFs with zero drift | **12 / 35** (of which 7 are bit-exact, max corner 0 px) |
-| Drift concentration | ~60% in `13_crosspage_table.pdf` |
-
-**Interpretation.** The two detectors agree on box membership for **>99.9%** of
-all text regions across the full corpus — the fixture-level Det 3/5 orphan
-floor (0.27%/0.44%) holds at larger scale (1/1559 ≈ 0.06%). There are **no**
-split/merge/threshold-fragment differences; the entire divergence is
-`coord_drift` on matched boxes, i.e. the same contour-boundary/resize geometry
-quantified in §8 and *Why the divergence is bounded*. This harness doubles as
-the cross-ORT guard: it passes with Go loading either the 1.23.2 or the 1.28.0
-runtime against the same models, corroborating *Runtime version*.
-
-### Evidence (measured)
-
-Latest `TestEquivalenceReport` (both tiers green; summary printed to the test log):
-
-| Task | Fixtures | Match | Max Δ | Status |
-|------|----------|-------|-------|--------|
-| DLA | 11 | 148/148 boxes (incl. equation + figure-caption classes, +3 real-document pages: CN annual report / ZH-TW migration / EN paper) | < 0.13 px | OK |
-| TSR | 8 | 190/190 boxes (incl. projected-row-header class, +1 real annual-report table) | 2.155 px (≤ 10 px on a 4:1 aspect crop, structure preserved) | OK |
-| OCR | 11 | exact text (EN / CJK / mixed / digits, + font variants, +3 real text-line crops: CN / ZH-TW / EN) | — | OK |
-| Det | all | IoU box-membership orphan **3/5** (gold 1125 / go 1127) | — | OK (accepted floor) |
-
-The detection orphan boxes were adjudicated with OCR: cropping each orphan and
-running OCR shows the regions still resolve to real text. Concretely, the
-orphans split as **Python-only real text 0/3** (those 3 regions are empty
-furniture on both sides) and **Go-only real text 3/5** (Go emits 3 text regions
-the Python side does not). So the two outputs are **not byte-identical**: Go is
-a *superset* on those few regions. This is a small, one-directional divergence
-(Go finds extra text), not a loss — the downstream text consumer (dedup /
-align) absorbs it, and no content is dropped. "Benign" here means "no content
-loss", not "identical output".
-
-#### Det IoU orphan 3/5 — what it means, and how small it is
-
-**What "IoU orphan" means.** For Det we compare two *sets* of text boxes — the
-Python golden set and the Go set — over each fixture. "IoU" (Intersection-over-
-Union) here is computed on the **axis-aligned bounding box (AABB)** of each
-quadrilateral (`iou()` in `golden.go:249`): `IoU = area(intersection) /
-area(union)`. Two boxes are considered a match only if their *best* greedy IoU
-is `≥ 0.5` (`MatchIoUBothDirections`, `golden.go:269`), matched independently in
-both directions. An **orphan** is a box on one side whose best counterpart on
-the other side has IoU `< 0.5` — i.e. no overlapping twin at all. So "3/5"
-means: **3 Python golden boxes** have no Go twin at IoU ≥ 0.5 (Go "missed"
-them), and **5 Go boxes** have no Python twin at IoU ≥ 0.5 (Go "extra"). This
-isolates genuine box-membership divergence (a box split, merged, or
-hallucinated) from mere coordinate drift — a box shifted 20 px but still
-overlapping its twin scores high IoU and is **not** an orphan.
-
-**Sample volume.** The 3/5 is accumulated across **every** committed Det
-fixture, not a cherry-picked subset. The corpus is **35 fixtures**, totalling
-**1,125 golden text boxes** vs **1,127 Go text boxes** (logged by
-`TestDetMembershipAllFixtures` as `TOTAL gold=1125 go=1127`). This spans blank
-pages, degradation variants (noise / low-contrast / skew / tiny-text / CJK
-vertical), and mixed-language dense pages.
-
-**Proportion (how dominant the agreement is).**
-
-| Side | Boxes | Matched (IoU ≥ 0.5) | Orphan | Orphan rate |
-|------|-------|---------------------|--------|-------------|
-| Python golden | 1,125 | 1,122 | 3 | **0.27%** |
-| Go | 1,127 | 1,122 | 5 | **0.44%** |
-| Both combined | 2,252 | 2,244 | 8 | **0.36%** |
-
-In other words, the two detectors agree on box membership for **>99.6%** of all
-text regions; the residual is a handful of boxes whose contour geometry differs
-just enough to drop below the 0.5 IoU hinge.
-
-**Why it is accepted, not a defect.** It is a *regression guard*, not a
-zero-target: the test fails only if Go gets *worse* than the baseline (gold
-orphan `> 6`, go orphan `> 8`, with slack 3 — see `native_integration_test.go:559`).
-The 3/5 is the residue left after every real bug was fixed — stale goldens
-(37/20, 42/13) and an R/B channel-swap in `normalizeCHW` (23/9) — leaving only
-contour-tracer geometry (explained below). And the orphan boxes are
-OCR-adjudicated benign (above): cropping each orphan and running OCR still
-resolves real text, so the divergence never drops content.
-
-### Why the divergence is bounded and deterministic
-
-- **Det 3/5** — contour-tracer geometry. Go's pure-Go boundary follower
- (Moore-neighbour, Suzuki-Abe style, `RETR_LIST`) selects boundary pixels
- slightly differently from cv2's `findContours` at 8-connected diagonal / hole
- junctions. 100% of the orphans are SCORE-FLIPs: a different convex hull →
- different `minAreaRect` → `box_score_fast` crosses 0.5 at a handful of
- regions on dense pages. It does not flip any segmentation.
-- **≤ 1/255 residual** — ONNX Runtime's fixed-point `uint8` resize introduces a
- pixel-level noise ceiling (max |Δ| = 1 gray level vs Go's float bilinear). It
- is irreducible for any pure-float implementation and is far below the 0.5 /
- 0.3 score thresholds, so it never changes a detection or score.
-
-Both effects are **deterministic and reproducible**, not random accuracy loss.
-
-### What is NOT proven here (honest boundaries)
-
-1. **Not bit-identical.** "Equivalent within bounded, accepted floors" is the
- correct claim; "perfect / pixel-identical" is not. The Det 3/5 floor is a
- known, accepted divergence.
-2. **Production caller is wired (server only).** The server binary built with
- `-tags cgo` registers the in-process backend via
- `infnative.Register(...)` in `cmd/ragflow_server.go`; it fails fast at
- startup unless ORT + models are available. The in-process backend is the only
- DeepDoc backend — there is no external service path. The CLI binary is built
- without `cgo` (no-op path).
-3. **Coverage confirmation required.** Go implements {det, dla, tsr, ocr}.
- Confirm the Python path being replaced uses only these recognizers (e.g. a
- separate table-cell recognizer would not be covered).
-4. **Runtime version.** Validated with onnxruntime 1.23.2 (see
- `internal/common.DeepDocORTVersion`) **and** 1.28.0 (the version the running
- Python service resolves); a controlled experiment showed identical detection
- matching across both, so the ORT version is **not a sensitivity**. Re-run the
- golden-comparison tests (§6 / §7) / parity harness (§9) only if the Python
- side moves to a materially different ORT line.
-5. **HTTP server shape.** If the goal is a standalone HTTP service mirroring
- `deepdoc_server`, that surface is not built — only the in-process library and
- a CLI (`main.go`) exist.
-
-### Known model-floor limits (documented, not hidden)
-
-The full-page real-table TSR floor is **measured end-to-end** (both production
-raster paths) in Methodology §8 / `TestTSRFloorFullPageTables` and recorded in
-the *Known model-floor limits (full-page real tables) — measured* subsection
-under §3. That is the authoritative, quantified statement: ordinary/moderate
-real tables ≤ 3.5px (worst 2.70px; dense annual-report ≤ 1.21px), and dense
-technical-standard tables (15K606 p40) breaking the floor on both coordinate
-drift **and** cell count (17/30, documented exception).
-
-The two excluded pages (`15K606` p40, `Xiamen Xiangyu annual report` p12) are retained in the
-generator (`/tmp/gen_corpus.py`) for future investigation but are **not** in the
-strict 3.5px fixture suite — p12 actually measures 0.37px (25/25) once rasterized
-through the real production path, so only p40 remains a genuine hard case.
-
-### Reviewer follow-ups (prioritized, with status)
-
-These are the items raised by an independent review of this proof, ranked by
-impact. "Closed" items are done in code or in this document.
-
-| ID | Item | Status |
-|----|------|--------|
-| P1-1 | **Live-service field diff** — confirm server is a thin wrapper (no extra preprocessing) and diff Go `Wire()` against the *real* `deepdoc_server` HTTP response. | **CLOSED** — server verified thin (adapters only decode + color-convert + clamp + label→class_id; config matches goldens: DLA/TSR `thr=0.2`, OCR default pipeline). The equivalence was established and recorded in Methodology §7 against the reference server (Go ORT 1.23.2 vs reference server 1.23.2; re-validated cross-version against the 1.28.0 runtime by §9). The `TestWireVsLiveServer` live-HTTP diff was later removed when the external service path was dropped from the codebase, but the verified-thin conclusion stands. |
-| P1-5 | **ORT version** — record the Python-side ORT build, not just assume ABI compatibility. | **CLOSED** — Go runs **1.23.2** (`DeepDocORTVersion`); the currently running Python service resolves **1.28.0**. A controlled experiment (Go loading 1.28.0's `.so`) reproduced identical detection matching, so the version is not a sensitivity. Recorded in *Runtime version* and Methodology §9. |
-| P2-2 | **Scope wording** — state that inference-boundary equivalence ≠ end-to-end PDF→chunk pipeline equivalence. | **CLOSED** — explicit "Boundary of this proof" paragraph added to Scope; PDF→raster and `PdfParser` downstream named as out-of-scope. |
-| P2-3 | **Det 3/5 "benign" wording** — it is not identical output (Go emits 3 extra text regions). | **CLOSED** — reworded to "no content loss, not identical output"; Go is a *superset* on those regions, absorbed by downstream dedup. |
-| P2-8 | **Corpus** — DLA/TSR/OCR coverage is thin (8 / 7 / 8; OCR are line crops, not full pages). | **CLOSED** — expanded with diverse full-page real-document fixtures: DLA +3 (`dla_real_cn_report` Xiamen Xiangyu annual report p2, `dla_real_zhtw` ZH-TW migration doc p3, `dla_real_en_paper` ZoomNeXt paper p1), TSR +1 (`tsr_real_report` Xiamen Xiangyu annual report p8), OCR +3 (`line_real_cn` Three Kingdoms p1, `line_real_zhtw` ZH-TW migration p2, `line_real_en` ZoomNeXt p1). All pass sub-pixel / exact-text. The one genuine hard case (15K606 p40 dense technical-standard table) is a documented exception, not in the strict 3.5px suite — see *Known model-floor limits*. |
-| P0 | **Model snapshot hash lock** — `MODEL_DIR` must be pinned to the same `InfiniFlow/deepdoc` snapshot as the frozen Python side; the proof must fail if it drifts. | **CLOSED** — enforced. `modelSnapshotHashes` (sha256 of `det.onnx`, `layout.onnx`, `tsr.onnx`, `rec.onnx`, `ocr.res`) is checked by `TestModelSnapshotHash` and at the top of `TestEquivalenceReport`; Fatal on any mismatch. Both repo copies verified byte-identical. Update the table only when the snapshot is intentionally upgraded, and regenerate every golden in the same change. |
-| P3 | **Concurrency correctness** — parallel vs serial inference must give identical results (thread-safety is correctness, not performance). | **CLOSED** — `TestInferenceConcurrencyConsistent` drives DLA/TSR/OCR-rec/Det once serially (baseline wire) then 8× concurrently and asserts every concurrent run is byte-identical to the serial baseline. Proves the shared model-session pool is race-free and contamination-free under parallel load (complementing `TestDetSessionPoolBounded` which guards pool *size*). |
-| **E2E-1** | **End-to-end raster alignment** — the "same raster bytes in" premise is a declaration: production rasterizes via different engines (Go pdfium @216 DPI vs Python pdfplumber @216 DPI). Prove the two render paths yield equivalent boxes. | **CLOSED** — `TestRasterAlignmentDLA/Det/TSR` rasterize the *same* real PDF page with both paths (pdfium vs deepdoc's own pdfplumber) at 216 DPI and compare boxes in source-pixel space. After enabling LCD text AA in the Go pdfium render to match pdfplumber: DLA **104/104** matched with worst max Δ **0.721 px** on the figure page and **≤ 0.03 px** on the four text pages (near-pixel-exact); Det IoU orphan 0/0–1/2 (inside 3/5); TSR on real-table pages **117/117** matched (worst 2.700px). The "same-bytes-in" premise is now **measured**, not assumed. Measured numbers in Methodology §8. Note (measured): the Det test-log `corner-maxd 8–12 px` is the max *per-corner* difference on 1–2 skewed outlier boxes; per-box **center** distance is sub-pixel (median 0, mean <0.5px, p90 <2.2px, max <5px) — there is **no render-origin translation**; the residual is contour-boundary quad-skew, the same source as the 3/5 IoU orphan floor. |
-| **E2E-2** | **Quantify TSR floor on full-page real tables** — give an empirical upper bound ("N pages, worst X px") instead of two hand-picked excluded examples. | **CLOSED** — `TestTSRFloorFullPageTables` runs TSR on whole-page real tables through both raster paths: moderate tables ≤ 3.5px (Xiamen Xiangyu annual report p8 1.21px, p12 0.37px; Three Kingdoms p1 2.70px); dense technical-standard 15K606 p40 is the documented exception (17/30, model-level cell-count divergence, regression-guarded). Empirical bound recorded in *Known model-floor limits (full-page real tables) — measured*. |
-
-**Withdrawn critique.** One review point claimed the two proof tiers "use
-different inference code paths / decoders, so preprocessing may differ." This is
-**factually incorrect**: `infnative.NativeAnalyzer.DLA/TSR/OCRDetect/OCRRecognize`
-all delegate to the *same* `native.RunDLA` / `RunDet` / `RunTSR` / `RunOCRRec`
-(`native_analyzer.go:101-181`); only the initial PNG→pixel decode differs, which
-is the already-documented `≤ 1/255` residual. The standalone and seam tiers
-therefore share one inference path.
diff --git a/internal/deepdoc/parser/pdf/inference/native_analyzer/native_analyzer_test.go b/internal/deepdoc/parser/pdf/inference/native_analyzer/native_analyzer_test.go
index 42653a2a9f..61c5c21d85 100644
--- a/internal/deepdoc/parser/pdf/inference/native_analyzer/native_analyzer_test.go
+++ b/internal/deepdoc/parser/pdf/inference/native_analyzer/native_analyzer_test.go
@@ -213,10 +213,9 @@ func cropBox(src image.Image, b deepdoctype.OCRBox) image.Image {
// The tests below prove the NativeAnalyzer (the DocAnalyzer the PDF parser
// actually consumes) produces output equivalent to the Python deepdoc
// reference, reusing the SAME Python-reference goldens as the native
-// integration suite. This closes the gap noted in deepdoc_go_alignment_report.md
-// (Equivalence proof section): the in-
-// process backend previously only had a smoke test (non-empty, in-bounds);
-// these tests assert value-level parity through the analyzer's public API.
+// integration suite. Before these tests the in-process backend only had a smoke
+// test (non-empty, in-bounds); these tests assert value-level parity through
+// the analyzer's public API.
// goldenPath resolves a native testdata fixture from this package's test
// directory (internal/deepdoc/parser/pdf/inference/native_analyzer). Four ".." climb to
diff --git a/internal/deepdoc/parser/pdf/inference/native_analyzer/raster_alignment_test.go b/internal/deepdoc/parser/pdf/inference/native_analyzer/raster_alignment_test.go
index 805747f6cd..d8b0f70b44 100644
--- a/internal/deepdoc/parser/pdf/inference/native_analyzer/raster_alignment_test.go
+++ b/internal/deepdoc/parser/pdf/inference/native_analyzer/raster_alignment_test.go
@@ -4,14 +4,13 @@ package infnative
// End-to-end raster alignment tests (reviewer gap #1 and #2).
//
-// The native proof establishes "given the same raster image bytes, Go == Python"
-// (deepdoc_go_alignment_report.md → Equivalence proof / "Boundary of this proof"). But in production neither
-// side receives a pre-rendered PNG: the Go server rasterizes PDF pages with
-// pdfium (pdfium.RenderPage @ 216 DPI) and the Python deepdoc pipeline
-// rasterizes with pdfplumber (page.to_image(resolution=72*zoomin, antialias=True),
-// zoomin=3 => 216 DPI). These tests close the remaining gap by rasterizing the
-// SAME PDF page with BOTH paths and comparing the resulting boxes in source-
-// pixel coordinates:
+// The native proof establishes "given the same raster image bytes, Go == Python".
+// But in production neither side receives a pre-rendered PNG: the Go server
+// rasterizes PDF pages with pdfium (pdfium.RenderPage @ 216 DPI) and the Python
+// deepdoc pipeline rasterizes with pdfplumber
+// (page.to_image(resolution=72*zoomin, antialias=True), zoomin=3 => 216 DPI).
+// These tests close the remaining gap by rasterizing the SAME PDF page with BOTH
+// paths and comparing the resulting boxes in source-pixel coordinates:
//
// Go side : pdfium.RenderPage(pdf, page, 216) -> a.DLA / a.OCRDetect / a.TSR
// Py side : ref_raster.py renders the same page at 216 DPI via deepdoc's own