Skip to content

Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.

Refactor imports to align with DocVortex 0.4 structure and update dependency versions

myhlolida3b3d04d67c
diff --git a/.github/workflows/docvortex-boundary.yml b/.github/workflows/docvortex-boundary.yml
new file mode 100644
index 00000000..f8659e34
--- /dev/null
+++ b/.github/workflows/docvortex-boundary.yml
@@ -0,0 +1,58 @@
+name: DocVortex boundary integration
+
+on:
+ workflow_dispatch:
+ inputs:
+ docvortex-wheel-url:
+ description: DocVortex 0.4.x wheel URL for joint validation before publication
+ required: true
+ type: string
+ utils-wheel-url:
+ description: Optional mineru-vl-utils 2.0.1+ wheel URL for unpublished dependency validation
+ required: false
+ type: string
+
+permissions:
+ contents: read
+
+jobs:
+ integration:
+ strategy:
+ fail-fast: false
+ matrix:
+ os: [ubuntu-latest, windows-latest, macos-latest]
+ python: ['3.10', '3.13', '3.14']
+ runs-on: ${{ matrix.os }}
+ defaults:
+ run:
+ shell: bash
+ steps:
+ - uses: actions/checkout@v6
+ - uses: actions/setup-python@v6
+ with:
+ python-version: ${{ matrix.python }}
+ - uses: astral-sh/setup-uv@v7
+ - name: Install matching engine wheel and host
+ env:
+ DOCVORTEX_WHEEL_URL: ${{ inputs.docvortex-wheel-url }}
+ UTILS_WHEEL_URL: ${{ inputs.utils-wheel-url }}
+ run: |
+ uv venv
+ uv pip install "$DOCVORTEX_WHEEL_URL"
+ if [[ -n "$UTILS_WHEEL_URL" ]]; then uv pip install "$UTILS_WHEEL_URL"; fi
+ uv pip install '.[test]'
+ uv pip check
+ - name: Verify public contracts and deterministic inference replay
+ run: |
+ if [[ -f .venv/Scripts/activate ]]; then source .venv/Scripts/activate; else source .venv/bin/activate; fi
+ python -I -c "import docvortex; print(docvortex.__file__); from docvortex.version import __version__; assert __version__.startswith('0.4.')"
+ python -m pytest -q -o addopts='' \
+ tests/unittest/test_docvortex_import_boundary.py \
+ tests/unittest/test_backend_architecture.py \
+ tests/unittest/test_docvortex_protocol.py \
+ tests/unittest/test_docvortex_routing.py \
+ tests/unittest/test_pdf_text_styles.py \
+ tests/unittest/test_pdf_analyze_block_content.py \
+ tests/unittest/test_native_pdf_table_pipeline.py \
+ tests/unittest/test_pdf_analyze_visual_blocks.py \
+ tests/unittest/test_pdf_model_inputs.py
diff --git a/.github/workflows/transformers5.yml b/.github/workflows/transformers5.yml
index 822a505e..c2163b2c 100644
--- a/.github/workflows/transformers5.yml
+++ b/.github/workflows/transformers5.yml
@@ -8,7 +8,7 @@ on:
workflow_dispatch:
inputs:
docvortex-wheel-url:
- description: Optional DocVortex 0.3.x wheel URL for joint pre-release validation
+ description: Optional DocVortex 0.4.x wheel URL for joint pre-release validation
required: false
type: string
utils-wheel-url:
@@ -104,7 +104,7 @@ jobs:
python -m build --wheel
mkdir -p dependency-wheels
python -m pip download --no-deps --dest dependency-wheels "${UTILS_WHEEL_URL:-mineru-vl-utils>=2,<3}"
- python -m pip download --no-deps --dest dependency-wheels "${DOCVORTEX_WHEEL_URL:-docvortex>=0.3.0,<1}"
+ python -m pip download --no-deps --dest dependency-wheels "${DOCVORTEX_WHEEL_URL:-docvortex>=0.4.0,<0.5.0}"
- name: Resolve base, torch, full and all on Python 3.10 through 3.14
run: |
python scripts/check_transformers5_dependencies.py \
diff --git a/docs/docvortex-0.4-migration.md b/docs/docvortex-0.4-migration.md
new file mode 100644
index 00000000..2e30c0ce
--- /dev/null
+++ b/docs/docvortex-0.4-migration.md
@@ -0,0 +1,39 @@
+# DocVortex 0.4 配套迁移
+
+MinerU 使用 `docvortex>=0.4.0,<0.5.0`。本轮调整 Python 模块边界,保持文档协议、解析路由、模型回退与渲染语义。
+
+## 公开入口
+
+| 原入口或职责 | 新入口 |
+| --- | --- |
+| foundation 几何及 PDF 通用坐标原语 | `docvortex.geometry` |
+| foundation 文本与链接规则 | `docvortex.content.text`、`docvortex.content.links` |
+| 图像统计、素材编码与路径校验 | `docvortex.assets` |
+| PDFDocument、PDFPage、字符和几何契约 | `docvortex.document.pdf` |
+| HTML 来源上下文 | `docvortex.document.contracts.HtmlSourceContext` |
+| 原生文字证据、区域表格及 OCR 文本投影 | `docvortex.analyzers.pdf` |
+| 模型文本、坐标自动判断与虚拟 OCR token | MinerU PDF 分析层 |
+| xhigh layout 容器补全 | MinerU PDF 分析层 |
+| 推理平台支持判断 | MinerU model/runtime |
+
+`docvortex.public_api.PUBLIC_API` 是静态模块及符号清单。新增跨库依赖必须同时更新清单、文档和契约测试;不能通过基础实现、私有模块或动态别名绕过边界。
+
+## 行为与资源所有权
+
+`prepare_text_evidence` 返回具名的 `PDFTextEvidence`,`apply_text_evidence` 统一完成链接、样式、脚本与 InlineSpan 物化。MinerU 仍决定调用时机、公式排除区域、超大字符页和 OCR 回退。
+
+`prepare_table_page` 仅在存在候选表格时调用;同页复用其物化数据。`recover_table_region` 返回最终 HTML、来源、置信度和诊断,`None` 表示没有可接受结果。结构恢复异常以 `PDFTableRecoveryError` 保留原始原因,交给宿主回退;HTML 物化异常继续传播,避免改变旧异常边界。页面几何可以复用,证据不持有 PDFium 裸句柄。
+
+通用 `convert_bbox` 显式声明 `unit`、`pixel` 或 `point`;`page_size` 使用 point,`render_scale` 是每 point 的像素数。模型 bbox 的自动解释与三位小数规则保留在 MinerU。共享 PDFium 锁、进程池和资源关闭职责仍归 DocVortex。
+
+## 升级与回退
+
+本轮直接删除被替代的旧门面及重导出,不提供过渡版本或旧 pickle 路径兼容。JSON、Bundle、HTTP API 和输出格式保持既有契约。
+
+配套升级两个 wheel;旧版 MinerU 必须显式约束 `docvortex<0.4.0`。旧版依赖上限 `<1` 无法阻止解析器选择 0.4,因此不要只升级 DocVortex。回退时同时恢复旧版 MinerU 与其兼容的 DocVortex。
+
+## 验证
+
+`tools/validate_docvortex_boundary.py` 可对指定两仓库源码或当前安装包捕获真实 PDF 的中间协议与全部渲染。记录实际导入路径、样本 SHA256 和页数;比较时只允许忽略 DocVortex producer 版本、DOCX/EPUB 容器时间字段,以及经独立复算确认由 producer 版本派生的 EPUB 标识。
+
+MinerU 的 `DocVortex boundary integration` 手动工作流接受未发布的 0.4 wheel URL,以及可选的 mineru-vl-utils 配套 wheel URL,在三种操作系统及 Python 3.10、3.13、3.14 上验证边界、协议和固定推理结果。DocVortex 自身 CI 验证独立安装与基础算法。配置工作流不代表已经通过远端 CI。
diff --git a/docs/docvortex-upgrade-validation.md b/docs/docvortex-upgrade-validation.md
index 97237e41..39d14976 100644
--- a/docs/docvortex-upgrade-validation.md
+++ b/docs/docvortex-upgrade-validation.md
@@ -1,5 +1,7 @@
# DocVortex 接入与标点规则验证
+> 当前配套版本为 DocVortex 0.4;升级边界与旧入口迁移见 [0.4 迁移说明](docvortex-0.4-migration.md)。以下保留历史协议验收记录。
+
独立引擎已更名为 DocVortex,MinerU 依赖为 `docvortex>=0.2.1,<0.3.0`。
导入及集成模块同步迁移,公开文档类型继续由引擎唯一维护。MinerU 的 schema 2.0、
产品元数据、已有旧结果适配、OCR/VLM/Hybrid 路由与 UI 名称保持不变。
diff --git a/docs/next/middle-json/migration.md b/docs/next/middle-json/migration.md
index f894e3a3..ea94aa32 100644
--- a/docs/next/middle-json/migration.md
+++ b/docs/next/middle-json/migration.md
@@ -1,5 +1,7 @@
# JSON 2.0 直接迁移
+> 当前配套版本为 DocVortex 0.4;升级边界与旧入口迁移见 [0.4 迁移说明](../../docvortex-0.4-migration.md)。以下保留历史协议验收记录。
+
采用[统一外层协议](envelope.md),通用解析接口保持严格;仅 Doclib 持久化读取提供[历史转换](doclib-compatibility.md)。
1. 升级 DocVortex 至 0.2.x,MinerU 要求 `docvortex>=0.2.1,<0.3.0`。
diff --git a/mineru/backend/analysis/html.py b/mineru/backend/analysis/html.py
index 76448789..bfb658df 100644
--- a/mineru/backend/analysis/html.py
+++ b/mineru/backend/analysis/html.py
@@ -7,7 +7,8 @@ import time
from io import BytesIO
from docvortex.analyzers.native import HtmlModel
-from docvortex.analyzers.native.html import HtmlSourceContext
+from docvortex.document.contracts import HtmlSourceContext
+
from .contracts import AnalysisResult
diff --git a/mineru/backend/analysis/pdf/formulas.py b/mineru/backend/analysis/pdf/formulas.py
index a0729a9a..4f939751 100644
--- a/mineru/backend/analysis/pdf/formulas.py
+++ b/mineru/backend/analysis/pdf/formulas.py
@@ -8,20 +8,16 @@ from typing import Any
import cv2
import numpy as np
+from docvortex.assets import image_size as _normalize_page_size
+from docvortex.content.text import build_tagged_formula_content
+from docvortex.geometry import bbox_to_quad as _medium_bbox_to_quad
+from docvortex.geometry import normalize_to_int_bbox
from PIL import Image
-from ....model.runtime.hybrid import HybridLocalModelContext
-from docvortex.foundation.geometry import normalize_to_int_bbox
from ....model.ocr.image import get_rotate_crop_image_for_text_rec
-from docvortex.foundation.text import build_tagged_formula_content
-
+from ....model.runtime.hybrid import HybridLocalModelContext
from ....types import RAW_FORMULA_NUMBER, BlockType
-
-from docvortex.document.pdf.geometry import bbox_to_pixel_bbox as _bbox_to_pixel_bbox
-from docvortex.document.pdf.geometry import medium_bbox_to_quad as _medium_bbox_to_quad
-from docvortex.document.pdf.geometry import normalize_layout_bbox_to_unit as _normalize_layout_bbox_to_unit
-from docvortex.document.pdf.geometry import normalize_medium_content as _normalize_medium_content
-from docvortex.document.pdf.geometry import normalize_page_size as _normalize_page_size
+from .model_inputs import _bbox_to_pixel_bbox, _normalize_layout_bbox_to_unit, _normalize_medium_content
def _is_hybrid_equation_block(block: dict[str, Any]) -> bool:
diff --git a/mineru/backend/analysis/pdf/layout.py b/mineru/backend/analysis/pdf/layout.py
index b26c110e..5a273a52 100644
--- a/mineru/backend/analysis/pdf/layout.py
+++ b/mineru/backend/analysis/pdf/layout.py
@@ -8,19 +8,18 @@ from copy import deepcopy
from typing import Any
import numpy as np
+from docvortex.assets import image_size as _normalize_page_size
+from docvortex.geometry import normalize_to_int_bbox
from PIL import Image
from ....types import BBox, BlockType
-from docvortex.foundation.geometry import normalize_to_int_bbox
-from .images import get_crop_np_img
-
-from docvortex.document.pdf.geometry import normalize_layout_bbox_to_unit as _normalize_layout_bbox_to_unit
-from docvortex.document.pdf.geometry import normalize_page_size as _normalize_page_size
from .constants import (
VLM_LAYOUT_LABEL_MAP,
VLM_MODEL_LIST_FIELDS,
VLM_VISUAL_ANNOTATION_TYPE_MAP,
)
+from .images import get_crop_np_img
+from .model_inputs import _normalize_layout_bbox_to_unit
def _layout_item_to_content_block(layout_item: dict[str, Any], page_size: tuple[int, int]) -> dict | None:
diff --git a/mineru/backend/analysis/pdf/model_inputs.py b/mineru/backend/analysis/pdf/model_inputs.py
new file mode 100644
index 00000000..3a1a8b6e
--- /dev/null
+++ b/mineru/backend/analysis/pdf/model_inputs.py
@@ -0,0 +1,89 @@
+# Copyright (c) Opendatalab. All rights reserved.
+"""模型输入坐标、文本字段及表格 OCR token 的适配规则。"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from docvortex.geometry import bbox_center as _table_bbox_center
+from docvortex.geometry import convert_bbox
+
+from ....types import BBox
+
+
+def _bbox_to_pixel_bbox(bbox: BBox | None, page_size: tuple[int, int]) -> BBox | None:
+ """在模型输入边界解释归一化或像素框,再调用共享的显式坐标转换。"""
+ if bbox is None or len(bbox) != 4:
+ return None
+ try:
+ coordinates = tuple(float(value) for value in bbox)
+ except (TypeError, ValueError):
+ return None
+ space = "unit" if all(0.0 <= value <= 1.0 for value in coordinates) else "pixel"
+ return convert_bbox(coordinates, source_space=space, target_space="pixel", page_size=page_size)
+
+
+def _normalize_layout_bbox_to_unit(bbox: BBox | None, page_size: tuple[int, int]) -> list[float] | None:
+ """将 layout 像素 bbox 归一化为 VLM ContentBlock 需要的 0-1 坐标。"""
+ pixel_bbox = _bbox_to_pixel_bbox(bbox, page_size)
+ if pixel_bbox is None:
+ return None
+
+ page_width, page_height = page_size
+ if page_width <= 0 or page_height <= 0:
+ return None
+
+ x0, y0, x1, y1 = pixel_bbox
+ unit_bbox = [
+ round(max(0.0, min(1.0, float(x0) / page_width)), 3),
+ round(max(0.0, min(1.0, float(y0) / page_height)), 3),
+ round(max(0.0, min(1.0, float(x1) / page_width)), 3),
+ round(max(0.0, min(1.0, float(y1) / page_height)), 3),
+ ]
+ if unit_bbox[2] <= unit_bbox[0] or unit_bbox[3] <= unit_bbox[1]:
+ return None
+ return unit_bbox
+
+
+def _normalize_medium_content(value: Any) -> str:
+ """将 medium 本地模型输出的文本字段规范成 Hybrid block 可消费的字符串。"""
+ if isinstance(value, list):
+ return "\n".join(str(item) for item in value if str(item).strip())
+ if isinstance(value, str):
+ return value.strip()
+ return ""
+
+
+def _get_medium_table_virtual_image_bbox(
+ bbox: BBox,
+ image_size: tuple[int, int],
+ box_size: float = 10.0,
+) -> BBox:
+ """在图片中心生成小 OCR token 框,避免图片大框干扰单元格匹配。"""
+ image_width, image_height = image_size
+ center_x, center_y = _table_bbox_center(bbox)
+ half_size = box_size / 2.0
+ return (
+ max(0.0, center_x - half_size),
+ max(0.0, center_y - half_size),
+ min(float(image_width), center_x + half_size),
+ min(float(image_height), center_y + half_size),
+ )
+
+
+def _sidecar_bbox_to_page_bbox(
+ bbox: BBox | None,
+ page_size: tuple[float, float],
+ render_scale: float,
+) -> BBox | None:
+ """在 OCR/公式 sidecar 边界解释输入单位,并显式转换到裁剪后的页面 point。"""
+ if bbox is None or len(bbox) != 4:
+ return None
+ try:
+ coordinates = tuple(float(value) for value in bbox)
+ except (TypeError, ValueError):
+ return None
+ space = "unit" if all(0.0 <= value <= 1.0 for value in coordinates) else "pixel"
+ return convert_bbox(
+ coordinates, source_space=space, target_space="point", page_size=page_size, render_scale=render_scale, clip=True
+ )
diff --git a/mineru/backend/analysis/pdf/normalization.py b/mineru/backend/analysis/pdf/normalization.py
index 92d9f9b9..0635bf87 100644
--- a/mineru/backend/analysis/pdf/normalization.py
+++ b/mineru/backend/analysis/pdf/normalization.py
@@ -7,21 +7,18 @@ import math
import re
from typing import Any
-from docvortex.content.spans import append_equation_span
-from docvortex.content.spans import append_text_span
-from docvortex.content.spans import inline_span_plain_text
-from docvortex.content.spans import strip_span_dicts
-from ....types import BBox, BlockType, RAW_ALGORITHM, RAW_PHONETIC
-from docvortex.foundation.geometry import calculate_overlap_area_2_minbox_area_ratio
from docvortex.content import normalize_pdf_model_text
+from docvortex.content.spans import append_equation_span, append_text_span, inline_span_plain_text, strip_span_dicts
+from docvortex.geometry import calculate_overlap_area_2_minbox_area_ratio
+from ....types import RAW_ALGORITHM, RAW_PHONETIC, BBox, BlockType
from .constants import (
+ _VLM_UNCLASSIFIED_TITLE_TYPE,
LAYOUT_TITLE_SPLIT_OVERLAP_THRESHOLD,
LINE_METADATA_BLOCK_TYPES,
NATURAL_LANGUAGE_CONTENT_BLOCK_TYPES,
- _VLM_UNCLASSIFIED_TITLE_TYPE,
)
-from docvortex.document.pdf.geometry import bbox_to_pixel_bbox as _bbox_to_pixel_bbox
+from .model_inputs import _bbox_to_pixel_bbox
def _collect_layout_doc_title_bboxes(layout_res: list[dict[str, Any]], page_size: tuple[int, int]) -> list[BBox]:
diff --git a/mineru/backend/analysis/pdf/ocr.py b/mineru/backend/analysis/pdf/ocr.py
index baf52487..f2aa6791 100644
--- a/mineru/backend/analysis/pdf/ocr.py
+++ b/mineru/backend/analysis/pdf/ocr.py
@@ -8,16 +8,14 @@ from typing import Any, Literal
import cv2
import numpy as np
+from docvortex.geometry import normalize_to_int_bbox
from tqdm import tqdm
-from ....model.runtime.hybrid import HybridLocalModelContext
-from ....types import BlockType
-from docvortex.foundation.geometry import normalize_to_int_bbox
-from .images import crop_img
from ....model.ocr.geometry import merge_det_boxes, sorted_boxes, update_det_boxes
from ....model.ocr.image import mask_formula_regions_for_ocr_det
from ....model.ocr.results import OcrConfidence, get_adjusted_mfdetrec_res, get_ocr_result_list
-
+from ....model.runtime.hybrid import HybridLocalModelContext
+from ....types import BlockType
from .constants import (
BATCH_RATIO,
OCR_DET_BASE_BATCH_SIZE,
@@ -25,8 +23,8 @@ from .constants import (
VLM_OCR_DET_TYPE,
VLM_TXT_DET_TYPE,
)
-from docvortex.document.pdf.geometry import bbox_to_pixel_bbox as _bbox_to_pixel_bbox
-from docvortex.document.pdf.geometry import normalize_medium_content as _normalize_medium_content
+from .images import crop_img
+from .model_inputs import _bbox_to_pixel_bbox, _normalize_medium_content
@dataclass
diff --git a/mineru/backend/analysis/pdf/pipeline.py b/mineru/backend/analysis/pdf/pipeline.py
index 4eae83f9..17897ac0 100644
--- a/mineru/backend/analysis/pdf/pipeline.py
+++ b/mineru/backend/analysis/pdf/pipeline.py
@@ -6,13 +6,13 @@ from __future__ import annotations
import time
from typing import Any, cast
-from ..contracts import AnalysisResult, AnalyzeEffort, ParseMode, ResolvedParseMode
+from docvortex.document.pdf import PDFDocument
+
from ....config import VlmConfig
from ....model.runtime.hybrid import HybridLocalModelContextSingleton
from ....model.runtime.memory import clean_memory, trim_process_heap
from ....model.vlm.client import get_vlm_predictor
-from docvortex.document.pdf.document import PDFDocument
-
+from ..contracts import AnalysisResult, AnalyzeEffort, ParseMode, ResolvedParseMode
from .normalization import _normalize_pdf_model_list
from .window import process_pdf_windows
diff --git a/mineru/backend/analysis/pdf/tables.py b/mineru/backend/analysis/pdf/tables.py
index f8bf83ec..8c725618 100644
--- a/mineru/backend/analysis/pdf/tables.py
+++ b/mineru/backend/analysis/pdf/tables.py
@@ -11,24 +11,23 @@ from typing import Any, Literal
import cv2
import numpy as np
+from docvortex.analyzers.pdf import PDFTableRecoveryError, prepare_table_page, recover_table_region
+from docvortex.analyzers.pdf import project_table_text as project_ocr_table_text
+from docvortex.assets import encode_crop_as_jpeg_data_uri as _encode_page_crop_as_jpeg_data_uri
+from docvortex.assets import image_size as _normalize_page_size
+from docvortex.assets import rotate_image_to_upright as _rotate_visual_block_image_to_upright
+from docvortex.document.pdf import PDFPage, PDFPageTextGeometry, get_lines_from_chars
+from docvortex.geometry import bbox_center as _table_bbox_center
+from docvortex.geometry import bbox_to_quad as _medium_bbox_to_quad
+from docvortex.geometry import calculate_overlap_area_in_bbox1_area_ratio, normalize_to_int_bbox
+from docvortex.geometry import normalize_quarter_turn_angle as _normalize_visual_block_angle
+from docvortex.geometry import rotate_bbox as _rotate_medium_table_bbox
from loguru import logger
-from ....model.runtime.hybrid import HybridLocalModelContext, run_ocr_inference
+from ....model.ocr.image import mask_formula_regions_for_ocr_det
from ....model.runtime.contracts import AtomicModelName
+from ....model.runtime.hybrid import HybridLocalModelContext, run_ocr_inference
from ....types import RAW_ALGORITHM, RAW_FORMULA_NUMBER, RAW_PHONETIC, BBox, BlockType
-from docvortex.foundation.geometry import calculate_overlap_area_in_bbox1_area_ratio
-from docvortex.foundation.geometry import normalize_to_int_bbox
-from docvortex.analyzers.native.pdf.table_recovery import NativeTableInput
-from docvortex.analyzers.native.pdf.table_recovery import coerce_native_table_rectangles
-from docvortex.analyzers.native.pdf.table_recovery import coerce_native_table_rules
-from docvortex.analyzers.native.pdf.table_recovery import recover_native_pdf_table
-from docvortex.analyzers.native.pdf.table_text_styles import render_native_table_html_with_scripts
-from ....model.ocr.image import mask_formula_regions_for_ocr_det
-from docvortex.document.pdf.document import PDFPage
-from docvortex.document.pdf.document import PDFPageTextGeometry
-from docvortex.document.pdf.document import get_lines_from_chars
-from docvortex.analyzers.native.pdf.shared import project_table_text as project_ocr_table_text
-
from .constants import (
BATCH_RATIO,
OCR_DET_BASE_BATCH_SIZE,
@@ -37,20 +36,14 @@ from .constants import (
TABLE_TEXT_ORIENTATION_MIN_DOMINANCE_RATIO,
TABLE_TEXT_ORIENTATION_MIN_VALID_LINES,
)
-from docvortex.document.pdf.geometry import bbox_to_pixel_bbox as _bbox_to_pixel_bbox
-from docvortex.document.pdf.geometry import encode_page_crop_as_jpeg_data_uri as _encode_page_crop_as_jpeg_data_uri
-from docvortex.document.pdf.geometry import get_medium_table_virtual_image_bbox as _get_medium_table_virtual_image_bbox
-from docvortex.document.pdf.geometry import medium_bbox_to_quad as _medium_bbox_to_quad
-from docvortex.document.pdf.geometry import normalize_medium_content as _normalize_medium_content
-from docvortex.document.pdf.geometry import normalize_page_size as _normalize_page_size
-from docvortex.document.pdf.geometry import normalize_visual_block_angle as _normalize_visual_block_angle
-from docvortex.document.pdf.geometry import rotate_medium_table_bbox as _rotate_medium_table_bbox
-from docvortex.document.pdf.geometry import rotate_visual_block_image_to_upright as _rotate_visual_block_image_to_upright
-from docvortex.document.pdf.geometry import sidecar_bbox_to_page_bbox as _sidecar_bbox_to_page_bbox
-from docvortex.document.pdf.geometry import table_bbox_center as _table_bbox_center
+from .model_inputs import (
+ _bbox_to_pixel_bbox,
+ _get_medium_table_virtual_image_bbox,
+ _normalize_medium_content,
+ _sidecar_bbox_to_page_bbox,
+)
from .text.native import _is_supported_rotation
-
_NATIVE_TABLE_ALWAYS_COMPLEX_BLOCK_TYPES = {
BlockType.IMAGE,
BlockType.CHART,
@@ -268,14 +261,11 @@ def _apply_native_txt_table_priority(
try:
page_text_geometry = page_text_geometries[page_idx] if page_text_geometries is not None else None
- if page_text_geometry is None:
- page_text_geometry = pdf_page.get_chars_with_geometry()
- if page_text_geometries is not None:
- page_text_geometries[page_idx] = page_text_geometry
- native_chars = tuple(page_text_geometry.chars)
- native_rules = coerce_native_table_rules(pdf_page.get_drawing_lines())
- native_rectangles = coerce_native_table_rectangles(pdf_page.get_path_infos())
- native_page_size = tuple(float(value) for value in pdf_page.size)
+ table_page = prepare_table_page(pdf_page, geometry=page_text_geometry)
+ page_text_geometry = table_page.geometry
+ if page_text_geometries is not None:
+ page_text_geometries[page_idx] = page_text_geometry
+ native_page_size = table_page.page_size
render_scale = float(image_dict.get("scale", 1.0) or 1.0)
except Exception as exc:
errors += len(eligible_entries)
@@ -297,16 +287,10 @@ def _apply_native_txt_table_priority(
rejected += 1
continue
try:
- table_input = NativeTableInput(
- table_bbox=table_bbox,
- page_size=native_page_size,
- angle=_normalize_visual_block_angle(table_block.get("angle", 0)),
- chars=native_chars,
- drawing_lines=native_rules,
- rectangles=native_rectangles,
+ result = recover_table_region(
+ table_page, table_bbox, angle=_normalize_visual_block_angle(table_block.get("angle", 0))
)
- result = recover_native_pdf_table(table_input)
- except Exception as exc:
+ except PDFTableRecoveryError as exc:
errors += 1
logger.warning(
"Hybrid native table recovery failed and kept model fallback: "
@@ -319,12 +303,7 @@ def _apply_native_txt_table_priority(
f"Hybrid native table rejected and kept model fallback: page_idx={page_idx}, bbox={table_block.get('bbox')}"
)
continue
- table_block["content"] = render_native_table_html_with_scripts(
- result,
- table_input,
- page_text_geometry.tight_bboxes,
- page_text_geometry.origins,
- )
+ table_block["content"] = result.html
accepted_entries.append(table_entry)
accepted += 1
logger.debug(
diff --git a/mineru/backend/analysis/pdf/text/content.py b/mineru/backend/analysis/pdf/text/content.py
index 0db0e5f0..e3a6e605 100644
--- a/mineru/backend/analysis/pdf/text/content.py
+++ b/mineru/backend/analysis/pdf/text/content.py
@@ -6,31 +6,21 @@ from __future__ import annotations
from typing import Any, Literal
import numpy as np
+from docvortex.analyzers.pdf import PDF_NATIVE_SCRIPT_MARKUP_KEY, PDFTextEvidence, apply_text_evidence, prepare_text_evidence
+from docvortex.content.text import merge_text_line_contents
+from docvortex.document.pdf import PDFPage, PDFPageTextGeometry, get_lines_from_chars
from PIL import Image
-from .....model.runtime.hybrid import HybridLocalModelContext, run_ocr_inference
-from .....types import BBox, BlockType, ContentType
+
from .....model.ocr.image import rotate_vertical_crop_if_needed
from .....model.ocr.results import OcrConfidence
-from docvortex.document.pdf.document import PDFPage
-from docvortex.document.pdf.document import PDFPageTextGeometry
-from docvortex.document.pdf.document import get_lines_from_chars
-from docvortex.analyzers.native.pdf.text_styles import PDF_NATIVE_SCRIPT_MARKUP_KEY
-from docvortex.analyzers.native.pdf.text_styles import PDFTextLinkLine
-from docvortex.analyzers.native.pdf.text_styles import PDFTextScriptLine
-from docvortex.analyzers.native.pdf.text_styles import PDFTextStyleLine
-from docvortex.analyzers.native.pdf.text_styles import apply_pdf_text_links
-from docvortex.analyzers.native.pdf.text_styles import apply_pdf_text_scripts
-from docvortex.analyzers.native.pdf.text_styles import apply_pdf_text_styles
-from docvortex.analyzers.native.pdf.text_styles import materialize_pdf_inline_spans
-from docvortex.foundation.text import merge_text_line_contents
-
+from .....model.runtime.hybrid import HybridLocalModelContext, run_ocr_inference
+from .....types import BBox, BlockType, ContentType
from ..constants import (
CODE_CONTENT_BLOCK_TYPES,
LINE_METADATA_BLOCK_TYPES,
TITLE_BLOCK_TYPES,
)
-from docvortex.document.pdf.geometry import bbox_to_pixel_bbox as _bbox_to_pixel_bbox
-from docvortex.document.pdf.geometry import sidecar_bbox_to_page_bbox as _sidecar_bbox_to_page_bbox
+from ..model_inputs import _bbox_to_pixel_bbox, _sidecar_bbox_to_page_bbox
from .lines import group_spans_to_lines
from .models import _AnalyzeLine, _AnalyzeSpan
from .native import (
@@ -43,7 +33,6 @@ from .native import (
_restore_post_ocr_fallback,
txt_spans_extract,
)
-from .styles import build_pdf_native_visual_lines_and_styles
def _validate_text_formula_window_inputs(
@@ -389,9 +378,7 @@ def _fill_window_block_content_and_lines(
list[dict[str, Any]],
dict[int, list[_AnalyzeLine]],
tuple[float, float],
- list[PDFTextStyleLine],
- list[PDFTextLinkLine],
- list[PDFTextScriptLine],
+ PDFTextEvidence,
]
] = []
for page_idx, (image_dict, pdf_page, page_model_list, page_inline_formula_list, page_ocr_res_list) in enumerate(
@@ -422,9 +409,7 @@ def _fill_window_block_content_and_lines(
page_model_list,
page_size,
)
- style_lines: list[PDFTextStyleLine] = []
- link_lines: list[PDFTextLinkLine] = []
- script_lines: list[PDFTextScriptLine] = []
+ evidence = PDFTextEvidence(page_size)
if parse_mode == "txt":
page_text_geometry = page_text_geometries[page_idx] if page_text_geometries is not None else None
try:
@@ -436,18 +421,13 @@ def _fill_window_block_content_and_lines(
or isinstance(page_char_count, bool)
or page_char_count <= MAX_NATIVE_TEXT_CHARS_PER_PAGE
):
- (
- page_text_geometry,
- _line_items,
- style_lines,
- link_lines,
- script_lines,
- ) = build_pdf_native_visual_lines_and_styles(
+ evidence = prepare_text_evidence(
pdf_page,
- page_text_geometry=page_text_geometry,
- inline_math_regions=inline_math_regions,
+ geometry=page_text_geometry,
+ excluded_script_regions=inline_math_regions,
table_regions=table_regions,
)
+ page_text_geometry = evidence.geometry
if page_text_geometries is not None:
page_text_geometries[page_idx] = page_text_geometry
page_spans = _fill_native_pdf_text_spans(
@@ -471,47 +451,28 @@ def _fill_window_block_content_and_lines(
page_model_list,
block_lines,
page_size,
- style_lines,
- link_lines,
- script_lines,
+ evidence,
)
)
if parse_mode == "txt":
_apply_window_post_ocr(
local_model_context,
- [block_lines for _, block_lines, _, _style_lines, _link_lines, _script_lines in page_block_line_results],
+ [block_lines for _, block_lines, _, _evidence in page_block_line_results],
)
for (
page_model_list,
block_lines,
page_size,
- style_lines,
- link_lines,
- script_lines,
+ evidence,
) in page_block_line_results:
_apply_block_content_and_line_metadata(
page_model_list,
block_lines,
page_size,
)
- apply_pdf_text_links(
- page_model_list,
- link_lines,
- page_size,
- )
- apply_pdf_text_styles(
- page_model_list,
- style_lines,
- page_size,
- )
- apply_pdf_text_scripts(
- page_model_list,
- script_lines,
- page_size,
- )
- materialize_pdf_inline_spans(page_model_list)
+ apply_text_evidence(page_model_list, evidence)
return model_list
diff --git a/mineru/backend/analysis/pdf/text/native.py b/mineru/backend/analysis/pdf/text/native.py
index d46591ae..fd7d356d 100644
--- a/mineru/backend/analysis/pdf/text/native.py
+++ b/mineru/backend/analysis/pdf/text/native.py
@@ -12,17 +12,13 @@ from typing import Any, Callable, cast
import cv2
import numpy as np
+from docvortex.analyzers.pdf import PDF_NATIVE_SCRIPT_MARKUP_KEY, ScriptRole, classify_char_script_roles
+from docvortex.assets import calculate_contrast
+from docvortex.document.pdf import Char, PDFPage, get_lines_from_chars
+from docvortex.geometry import calculate_overlap_area_in_bbox1_area_ratio
from loguru import logger
-from docvortex.document.pdf.text.contracts import Char
from .....types import BBox, BlockType, ContentType
-from docvortex.document.pdf.document import PDFPage
-from docvortex.document.pdf.document import get_lines_from_chars
-from docvortex.analyzers.native.pdf.script_geometry import ScriptRole
-from docvortex.analyzers.native.pdf.script_geometry import classify_char_script_roles
-from docvortex.analyzers.native.pdf.text_styles import PDF_NATIVE_SCRIPT_MARKUP_KEY
-from docvortex.foundation.geometry import calculate_overlap_area_in_bbox1_area_ratio
-from docvortex.foundation.image import calculate_contrast
from ..images import get_crop_img
from .models import _AnalyzeSpan
diff --git a/mineru/backend/analysis/pdf/text/styles.py b/mineru/backend/analysis/pdf/text/styles.py
deleted file mode 100644
index a24cf238..00000000
--- a/mineru/backend/analysis/pdf/text/styles.py
+++ /dev/null
@@ -1,127 +0,0 @@
-# Copyright (c) Opendatalab. All rights reserved.
-"""Hybrid TXT 路径复用的 PDF 原生文本富化准备入口。"""
-
-from __future__ import annotations
-
-from typing import Any, Sequence
-
-from docvortex.document.pdf.document import PDFPage
-from docvortex.document.pdf.document import PDFPageTextGeometry
-from docvortex.document.pdf.document import get_lines_from_chars
-from docvortex.analyzers.native.pdf.text_styles import PDFTextLinkLine
-from docvortex.analyzers.native.pdf.text_styles import PDFTextScriptLine
-from docvortex.analyzers.native.pdf.text_styles import PDFTextStyleLine
-from docvortex.analyzers.native.pdf.text_styles import detect_pdf_text_link_lines
-from docvortex.analyzers.native.pdf.text_styles import detect_pdf_text_script_lines
-from docvortex.analyzers.native.pdf.text_styles import detect_pdf_text_style_lines
-from .....types import BBox
-
-
-def _script_range_center_in_regions(
- bbox: BBox,
- regions: Sequence[BBox],
-) -> bool:
- """判断脚本证据中心是否落入由 layout 确认的行内公式区域。"""
-
- center_x = (bbox[0] + bbox[2]) / 2.0
- center_y = (bbox[1] + bbox[3]) / 2.0
- return any(region[0] <= center_x <= region[2] and region[1] <= center_y <= region[3] for region in regions)
-
-
-def _exclude_layout_formula_script_ranges(
- lines: list[PDFTextScriptLine],
- regions: Sequence[BBox],
-) -> list[PDFTextScriptLine]:
- """从 Hybrid TXT 脚本 sidecar 中剔除已由 layout/MFR 拥有的公式字符。"""
-
- if not regions:
- return lines
- return [
- PDFTextScriptLine(
- bbox=line.bbox,
- text=line.text,
- script_ranges=tuple(
- script_range
- for script_range in line.script_ranges
- if not _script_range_center_in_regions(script_range.bbox, regions)
- ),
- source_index=line.source_index,
- angle=line.angle,
- )
- for line in lines
- ]
-
-
-def build_pdf_native_visual_lines_and_styles(
- pdf_page: PDFPage,
- *,
- page_text_geometry: PDFPageTextGeometry | None = None,
- supported_angles: Sequence[float] = (0.0,),
- inline_math_regions: Sequence[BBox] = (),
- table_regions: Sequence[BBox] = (),
-) -> tuple[
- PDFPageTextGeometry,
- list[Any],
- list[PDFTextStyleLine],
- list[PDFTextLinkLine],
- list[PDFTextScriptLine],
-]:
- """一次读取当前页字符,构造视觉 run、普通样式、链接和脚本证据。"""
-
- # 延迟导入避免 Hybrid 模块初始化时提前加载完整 Flash PDF 流水线。
- from docvortex.analyzers.native.pdf.shared import build_native_line_items as _build_native_line_items
- from docvortex.analyzers.native.pdf.shared import (
- merge_overlapping_inline_text_clusters as _merge_overlapping_inline_text_clusters,
- )
- from docvortex.analyzers.native.pdf.shared import merge_same_baseline_text_lines as _merge_same_baseline_text_lines
-
- geometry = page_text_geometry if page_text_geometry is not None else pdf_page.get_chars_with_geometry()
- chars = geometry.chars
- line_items = _build_native_line_items(
- get_lines_from_chars(chars),
- tuple(float(value) for value in pdf_page.size),
- page_rotation=pdf_page.rotation,
- supported_angles=supported_angles,
- )
- resolved_inline_math_regions = [tuple(float(value) for value in region) for region in inline_math_regions]
- resolved_table_regions = [tuple(float(value) for value in region) for region in table_regions]
- drawing_lines = pdf_page.get_drawing_lines()
- style_lines = detect_pdf_text_style_lines(
- line_items,
- drawing_lines,
- )
- link_lines = detect_pdf_text_link_lines(
- line_items,
- pdf_page.get_link_annotations(),
- )
- script_line_items = _merge_same_baseline_text_lines(
- list(line_items),
- tuple(float(value) for value in pdf_page.size),
- resolved_table_regions,
- )
- script_line_items = _merge_overlapping_inline_text_clusters(
- script_line_items,
- tuple(float(value) for value in pdf_page.size),
- resolved_table_regions,
- )
- script_line_items = _merge_same_baseline_text_lines(
- script_line_items,
- tuple(float(value) for value in pdf_page.size),
- resolved_table_regions,
- )
- script_lines = detect_pdf_text_script_lines(
- script_line_items,
- tuple(float(value) for value in pdf_page.size),
- geometry.tight_bboxes,
- geometry.origins,
- all_chars=chars,
- drawing_lines=drawing_lines,
- )
- script_lines = _exclude_layout_formula_script_ranges(
- script_lines,
- resolved_inline_math_regions,
- )
- return geometry, line_items, style_lines, link_lines, script_lines
-
-
-__all__ = ["build_pdf_native_visual_lines_and_styles"]
diff --git a/mineru/backend/analysis/pdf/visual_containers.py b/mineru/backend/analysis/pdf/visual_containers.py
new file mode 100644
index 00000000..fc7dcf21
--- /dev/null
+++ b/mineru/backend/analysis/pdf/visual_containers.py
@@ -0,0 +1,140 @@
+# Copyright (c) Opendatalab. All rights reserved.
+"""依据模型 layout 证据补全 xhigh 图片容器。"""
+
+from __future__ import annotations
+
+import math
+from typing import Any
+
+from docvortex.geometry import calculate_overlap_area_2_minbox_area_ratio, calculate_overlap_area_in_bbox1_area_ratio
+from docvortex.geometry import normalize_bbox as _normalize_model_bbox_for_containment
+
+from ....types import BBox, BlockType
+from .constants import (
+ IMAGE_BLOCK_CONTAINMENT_THRESHOLD,
+ IMAGE_BLOCK_LAYOUT_COVERAGE_THRESHOLD,
+ IMAGE_BLOCK_LAYOUT_MIN_VISUAL_COUNT,
+ LOCAL_LAYOUT_IMAGE_BLOCK_AREA_TYPES,
+ LOCAL_LAYOUT_IMAGE_BLOCK_BODY_TYPES,
+)
+
+
+def _supplement_missing_image_block_containers(
+ model_list: list[list[dict[str, Any]]],
+ layout_blocks_list: list[list[dict[str, Any]]],
+ containment_threshold: float = IMAGE_BLOCK_CONTAINMENT_THRESHOLD,
+ coverage_threshold: float = IMAGE_BLOCK_LAYOUT_COVERAGE_THRESHOLD,
+ min_visual_count: int = IMAGE_BLOCK_LAYOUT_MIN_VISUAL_COUNT,
+) -> None:
+ """用本地 layout 整图框为 xhigh 结果补充缺失的 image_block 容器。"""
+ if len(model_list) != len(layout_blocks_list):
+ raise ValueError(
+ "Hybrid image-block fallback page count mismatch: "
+ f"model_list={len(model_list)}, layout_blocks={len(layout_blocks_list)}"
+ )
+
+ for page_model_list, page_layout_blocks in zip(model_list, layout_blocks_list):
+ existing_image_block_bboxes = [
+ bbox
+ for block in page_model_list
+ if block.get("type") == "image_block"
+ if (bbox := _normalize_model_bbox_for_containment(block.get("bbox"))) is not None
+ ]
+
+ existing_claimed_block_ids: set[int] = set()
+ if existing_image_block_bboxes:
+ for block in page_model_list:
+ if block.get("type") == "image_block":
+ continue
+ block_bbox = _normalize_model_bbox_for_containment(block.get("bbox"))
+ if block_bbox is not None and any(
+ calculate_overlap_area_in_bbox1_area_ratio(block_bbox, image_block_bbox) >= containment_threshold
+ for image_block_bbox in existing_image_block_bboxes
+ ):
+ existing_claimed_block_ids.add(id(block))
+
+ candidates: list[tuple[int, float, int, int, dict[str, Any], set[int]]] = []
+ for layout_order, layout_block in enumerate(page_layout_blocks):
+ if layout_block.get("type") != BlockType.IMAGE or layout_block.get("sub_type") == "seal":
+ continue
+
+ layout_bbox = _normalize_model_bbox_for_containment(layout_block.get("bbox"))
+ if layout_bbox is None:
+ continue
+ if any(
+ calculate_overlap_area_2_minbox_area_ratio(layout_bbox, image_block_bbox) >= containment_threshold
+ for image_block_bbox in existing_image_block_bboxes
+ ):
+ continue
+
+ contained_blocks: list[tuple[int, dict[str, Any], BBox]] = []
+ for block_index, block in enumerate(page_model_list):
+ if block.get("type") == "image_block" or id(block) in existing_claimed_block_ids:
+ continue
+ block_bbox = _normalize_model_bbox_for_containment(block.get("bbox"))
+ if block_bbox is None:
+ continue
+ if calculate_overlap_area_in_bbox1_area_ratio(block_bbox, layout_bbox) >= containment_threshold:
+ contained_blocks.append((block_index, block, block_bbox))
+
+ contained_visuals = [
+ (block_index, block)
+ for block_index, block, _ in contained_blocks
+ if block.get("type") in LOCAL_LAYOUT_IMAGE_BLOCK_BODY_TYPES
+ ]
+ if len(contained_visuals) < min_visual_count:
+ continue
+
+ layout_area = (layout_bbox[2] - layout_bbox[0]) * (layout_bbox[3] - layout_bbox[1])
+ contained_area = sum(
+ (block_bbox[2] - block_bbox[0]) * (block_bbox[3] - block_bbox[1])
+ for _, block, block_bbox in contained_blocks
+ if block.get("type") in LOCAL_LAYOUT_IMAGE_BLOCK_AREA_TYPES
+ )
+ coverage_ratio = contained_area / layout_area
+ if coverage_ratio < coverage_threshold and not math.isclose(
+ coverage_ratio,
+ coverage_threshold,
+ rel_tol=0.0,
+ abs_tol=1e-12,
+ ):
+ continue
+
+ contained_block_ids = {id(block) for _, block, _ in contained_blocks}
+ first_block_index = min(block_index for block_index, _, _ in contained_blocks)
+ candidates.append(
+ (
+ -len(contained_visuals),
+ layout_area,
+ layout_order,
+ first_block_index,
+ layout_block,
+ contained_block_ids,
+ )
+ )
+
+ claimed_block_ids: set[int] = set()
+ selected_containers: list[tuple[int, dict[str, Any]]] = []
+ for _, _, _, first_block_index, layout_block, block_ids in sorted(candidates):
+ if claimed_block_ids.intersection(block_ids):
+ continue
+ claimed_block_ids.update(block_ids)
+ selected_containers.append(
+ (
+ first_block_index,
+ {
+ "type": "image_block",
+ "bbox": list(layout_block["bbox"]),
+ "angle": layout_block.get("angle", 0),
+ "content": None,
+ },
+ )
+ )
+
+ for insert_index, image_block in sorted(selected_containers, reverse=True):
+ page_model_list.insert(insert_index, image_block)
+
+
+supplement_missing_image_block_containers = _supplement_missing_image_block_containers
+
+__all__ = ["supplement_missing_image_block_containers"]
diff --git a/mineru/backend/analysis/pdf/window.py b/mineru/backend/analysis/pdf/window.py
index 4b7c58d1..b7189a67 100644
--- a/mineru/backend/analysis/pdf/window.py
+++ b/mineru/backend/analysis/pdf/window.py
@@ -7,15 +7,14 @@ from dataclasses import dataclass
from typing import Any, Literal
import numpy as np
+from docvortex.assets import image_size as _normalize_page_size
+from docvortex.document.pdf import PDFDocument, PDFPage, PDFPageTextGeometry
+from docvortex.document.pdf.visuals import attach_visual_block_images as _attach_visual_block_images
+from docvortex.document.pdf.visuals import attach_visual_block_images_from_pdf
from loguru import logger
from ....model.runtime.hybrid import HybridLocalModelContext
from ....model.runtime.memory import trim_process_heap
-from docvortex.document.pdf.document import PDFDocument
-from docvortex.document.pdf.document import PDFPage
-from docvortex.document.pdf.document import PDFPageTextGeometry
-from .images import get_load_images_threads, get_load_images_timeout, load_images_from_pdf_bytes_range
-
from ..contracts import AnalyzeEffort
from .constants import (
BATCH_RATIO,
@@ -24,14 +23,6 @@ from .constants import (
NOT_EXTRACT_TYPES,
PIPELINE_DET_TYPE,
)
-from docvortex.document.pdf.geometry import normalize_page_size as _normalize_page_size
-from .layout import (
- _build_vl_style_layout_blocks,
- _collect_table_items,
- _convert_vlm_results_to_model_list,
- _normalize_xhigh_vlm_blocks,
-)
-from .normalization import _apply_layout_title_split
from .formulas import (
_apply_medium_display_formula_results,
_apply_medium_formula_number_ocr,
@@ -39,11 +30,19 @@ from .formulas import (
_split_formula_results,
optimize_hybrid_formula_number_blocks,
)
+from .images import get_load_images_threads, get_load_images_timeout, load_images_from_pdf_bytes_range
+from .layout import (
+ _build_vl_style_layout_blocks,
+ _collect_table_items,
+ _convert_vlm_results_to_model_list,
+ _normalize_xhigh_vlm_blocks,
+)
+from .normalization import _apply_layout_title_split
from .ocr import (
_apply_ocr_rec_results,
+ _apply_seal_ocr,
_build_ocr_det_type_and_mfr_enable,
_ocr_det,
- _apply_seal_ocr,
)
from .tables import (
_apply_medium_table_recognition,
@@ -53,15 +52,11 @@ from .tables import (
_restore_native_high_table_blocks,
_split_native_high_table_blocks,
)
-from docvortex.document.pdf.visuals import attach_visual_block_images as _attach_visual_block_images
-from docvortex.document.pdf.visuals import attach_visual_block_images_from_pdf
-from docvortex.document.pdf.visuals import (
- supplement_missing_image_block_containers as _supplement_missing_image_block_containers,
-)
from .text.content import (
_fill_window_block_content_and_lines,
_validate_text_formula_window_inputs,
)
+from .visual_containers import supplement_missing_image_block_containers as _supplement_missing_image_block_containers
def _configured_window_size(default: int = 64) -> int:
diff --git a/mineru/backend/analyze.py b/mineru/backend/analyze.py
index e02e5446..f71f40f1 100644
--- a/mineru/backend/analyze.py
+++ b/mineru/backend/analyze.py
@@ -2,19 +2,19 @@
"""统一 PDF、EPUB、HTML、OFD、CSV 与 Office/RTF 文档分析的稳定公共门面。"""
from __future__ import annotations
-from docvortex.schema import DocumentMetadata, DocumentProperties, Producer
-from ..integrations.docvortex import build_metadata, read_source_properties
import asyncio
from typing import cast
+from docvortex.document.contracts import HtmlSourceContext
+from docvortex.schema import DocumentMetadata, DocumentProperties, Producer
from loguru import logger
-from .analysis.contracts import AnalyzeEffort, OfficeSuffix, ParseMode
from ..config import VlmConfig, config
-from docvortex.analyzers.native.html import HtmlSourceContext
+from ..integrations.docvortex import build_metadata, read_source_properties
from ..types import FILE_SUFFIXES, FileSuffix, MiddleJson, ModelJson
from ..version import __version__ as mineru_version
+from .analysis.contracts import AnalyzeEffort, OfficeSuffix, ParseMode
_SUPPORTED_ANALYZE_EFFORTS = {"flash", "medium", "high", "xhigh"}
diff --git a/mineru/backend/postprocess/legacy_schema_adapter.py b/mineru/backend/postprocess/legacy_schema_adapter.py
index d6150f05..2052d39e 100644
--- a/mineru/backend/postprocess/legacy_schema_adapter.py
+++ b/mineru/backend/postprocess/legacy_schema_adapter.py
@@ -14,9 +14,9 @@ import math
import re
from typing import Any
-from docvortex.schema import INLINE_STYLE_ORDER, InlineSpan, parse_inline_spans
-from docvortex.foundation.hyperlink import OFFICE_EXTERNAL_HYPERLINK_SCHEMES, sanitize_hyperlink_target
from docvortex.content.inline import join_inline_spans
+from docvortex.content.links import OFFICE_EXTERNAL_HYPERLINK_SCHEMES, sanitize_hyperlink_target
+from docvortex.schema import INLINE_STYLE_ORDER, InlineSpan, parse_inline_spans
# 这些父块在 3.4.5 中通过 ``blocks`` 保存 body/caption/footnote;当前 raw
# ModelJson 则要求先展平成同级 block,再由统一后处理重新分组。
diff --git a/mineru/backend/postprocess/table_merge/llm_cell_merge.py b/mineru/backend/postprocess/table_merge/llm_cell_merge.py
index ca04fa0d..a5fb09df 100644
--- a/mineru/backend/postprocess/table_merge/llm_cell_merge.py
+++ b/mineru/backend/postprocess/table_merge/llm_cell_merge.py
@@ -8,12 +8,10 @@ import json
from dataclasses import dataclass
from typing import Literal, cast
-from ....types import PageInfo, TableBlock, TableBodyBlock
-
-from docvortex.content.table.html import build_row_rendered_cell_segments
-from docvortex.content.table.html import build_table_state_from_html
+from docvortex.content.table import build_row_rendered_cell_segments, build_table_state_from_html, detect_table_headers
from docvortex.content.table import expand_header_count_by_rowspan as _expand_header_count_by_rowspan
-from docvortex.content.table.structure import detect_table_headers
+
+from ....types import PageInfo, TableBlock, TableBodyBlock
from ..llm_client import LLMAidedClient
CellMergeFlag = Literal[0, 1]
diff --git a/mineru/kit/commands/vlm_server.py b/mineru/kit/commands/vlm_server.py
index 370d0e6d..69dd9289 100644
--- a/mineru/kit/commands/vlm_server.py
+++ b/mineru/kit/commands/vlm_server.py
@@ -1,8 +1,8 @@
from __future__ import annotations
import importlib
-import importlib.util
import importlib.metadata
+import importlib.util
import platform
import sys
from collections.abc import Callable
@@ -11,7 +11,7 @@ from typing import Literal
import typer
from loguru import logger
-from docvortex.foundation.platform import is_mac_os_version_supported
+from ...model.runtime.platform import is_mac_os_version_supported
from ...utils.stdio import configure_standard_streams
from ..errors import exit_with_message
diff --git a/mineru/kit/gradio/artifacts.py b/mineru/kit/gradio/artifacts.py
index 63859a81..4147fd1b 100644
--- a/mineru/kit/gradio/artifacts.py
+++ b/mineru/kit/gradio/artifacts.py
@@ -15,11 +15,12 @@ from typing import Any, Callable, Literal, cast
from urllib.parse import quote, unquote, urlsplit
from bs4 import BeautifulSoup
+from docvortex.assets import parse_image_data_uri_strict, validate_image_sidecar_path
+from docvortex.document.pdf import PDFDocument
+from docvortex.document.pdf.pdfium import safe_rewrite_pdf_bytes_with_pdfium_result
from loguru import logger
from ...filetypes import IMAGE_EXTENSIONS, PDF_EXTENSIONS
-from docvortex.document.pdf.document import PDFDocument
-from docvortex.document.pdf.pdfium import safe_rewrite_pdf_bytes_with_pdfium_result
from ...parser.base import ParseResult
from ...parser.page_range import parse_page_range
from ...render import (
@@ -47,8 +48,6 @@ from ...types import (
TableBlock,
TableBodyBlock,
)
-from docvortex.foundation.image_payload import parse_image_data_uri_strict
-from docvortex.foundation.image_payload import validate_image_sidecar_path
DownloadFormat = Literal["markdown", "json", "html", "docx", "latex", "epub", "pdf"]
diff --git a/mineru/model/layout/pp_doclayout_v2_base.py b/mineru/model/layout/pp_doclayout_v2_base.py
index 9ffdf493..58d1bcef 100644
--- a/mineru/model/layout/pp_doclayout_v2_base.py
+++ b/mineru/model/layout/pp_doclayout_v2_base.py
@@ -16,10 +16,9 @@ import os
from typing import Dict, List, Optional, Sequence, Tuple, Union
import numpy as np
+from docvortex.geometry import normalize_to_int_bbox
from PIL import Image, ImageDraw, ImageFont
-from docvortex.foundation.geometry import normalize_to_int_bbox
-
DEFAULT_IMAGE_SIZE = (800, 800)
DEFAULT_RESCALE_FACTOR = 1.0 / 255.0
diff --git a/mineru/model/ocr/image.py b/mineru/model/ocr/image.py
index daa2ed7e..74330204 100644
--- a/mineru/model/ocr/image.py
+++ b/mineru/model/ocr/image.py
@@ -7,8 +7,8 @@ from typing import Any
import cv2
import numpy as np
+from docvortex.geometry import normalize_to_int_bbox
-from docvortex.foundation.geometry import normalize_to_int_bbox
from .geometry import is_bbox_aligned_rect
TEXT_REC_ROTATE_RATIO = 1.5
diff --git a/mineru/model/ocr/results.py b/mineru/model/ocr/results.py
index ddcc66f9..aed1b7e4 100644
--- a/mineru/model/ocr/results.py
+++ b/mineru/model/ocr/results.py
@@ -7,8 +7,8 @@ import copy
from typing import Any
import numpy as np
+from docvortex.geometry import normalize_to_int_bbox
-from docvortex.foundation.geometry import normalize_to_int_bbox
from .geometry import calculate_is_angle
from .image import get_rotate_crop_image_for_text_rec
diff --git a/mineru/model/runtime/platform.py b/mineru/model/runtime/platform.py
new file mode 100644
index 00000000..5dd6a26d
--- /dev/null
+++ b/mineru/model/runtime/platform.py
@@ -0,0 +1,46 @@
+# Copyright (c) Opendatalab. All rights reserved.
+"""推理运行环境的平台事实与版本门槛。"""
+
+import platform
+
+from packaging import version
+
+
+def is_windows_environment() -> bool:
+ """判断当前操作系统是否为 Windows。"""
+ return platform.system() == "Windows"
+
+
+def is_mac_environment() -> bool:
+ """判断当前操作系统是否为 macOS。"""
+ return platform.system() == "Darwin"
+
+
+def is_linux_environment() -> bool:
+ """判断当前操作系统是否为 Linux。"""
+ return platform.system() == "Linux"
+
+
+def is_apple_silicon_cpu() -> bool:
+ """判断 CPU 是否使用 Apple Silicon 兼容架构。"""
+ return platform.machine() in ["arm64", "aarch64"]
+
+
+def is_mac_os_version_supported(min_version: str = "13.5") -> bool:
+ """按调用方要求检查 macOS 版本与 CPU 架构。"""
+ if not is_mac_environment() or not is_apple_silicon_cpu():
+ return False
+ mac_version = platform.mac_ver()[0]
+ if not mac_version:
+ return False
+ # print("Mac OS Version:", mac_version)
+ return version.parse(mac_version) >= version.parse(min_version)
+
+
+__all__ = [
+ "is_windows_environment",
+ "is_mac_environment",
+ "is_linux_environment",
+ "is_apple_silicon_cpu",
+ "is_mac_os_version_supported",
+]
diff --git a/mineru/model/table/rec/unet_table/main.py b/mineru/model/table/rec/unet_table/main.py
index 25bd247d..bf23f3c3 100644
--- a/mineru/model/table/rec/unet_table/main.py
+++ b/mineru/model/table/rec/unet_table/main.py
@@ -9,12 +9,11 @@ from typing import Any, Dict, List, Optional, Union
import cv2
import numpy as np
from bs4 import BeautifulSoup
+from docvortex.assets import calculate_contrast
from loguru import logger
from PIL import Image
-from docvortex.foundation.image import calculate_contrast
from ....registry import PDF_EXTRACT_KIT
-
from .table_recover import TableRecover
from .table_structure_unet import TSRUnet
from .utils import InputType, LoadImage
diff --git a/mineru/model/vlm/engine_utils.py b/mineru/model/vlm/engine_utils.py
index fdec0e7e..c9dc2fd0 100644
--- a/mineru/model/vlm/engine_utils.py
+++ b/mineru/model/vlm/engine_utils.py
@@ -6,10 +6,9 @@ from typing import Any, Literal
from loguru import logger
from packaging import version
-from docvortex.foundation.platform import is_linux_environment
-from docvortex.foundation.platform import is_windows_environment
from ..runtime.device import get_device
from ..runtime.memory import get_vram
+from ..runtime.platform import is_linux_environment, is_windows_environment
def enable_custom_logits_processors() -> bool:
diff --git a/mineru/model/vlm/runtime.py b/mineru/model/vlm/runtime.py
index c56c646a..f2fdb99b 100644
--- a/mineru/model/vlm/runtime.py
+++ b/mineru/model/vlm/runtime.py
@@ -15,9 +15,9 @@ from typing import Any, AsyncIterator, Generator, Iterator, Literal
from loguru import logger
from mineru_vl_utils import MinerUClient
-from docvortex.foundation.platform import is_mac_os_version_supported
-from ..runtime.device import get_device
from ..registry import MINERU_2_5_PRO_2605_1_2B, MINERU_2_5_PRO_2605_1_2B_GGUF
+from ..runtime.device import get_device
+from ..runtime.platform import is_mac_os_version_supported
from .engine_utils import (
enable_custom_logits_processors,
mod_kwargs_by_device_type,
diff --git a/mineru/model/vlm/selector.py b/mineru/model/vlm/selector.py
index f1fdf2b1..1567a573 100644
--- a/mineru/model/vlm/selector.py
+++ b/mineru/model/vlm/selector.py
@@ -3,10 +3,7 @@ from typing import Literal, TypeAlias
from loguru import logger
-from docvortex.foundation.platform import is_linux_environment
-from docvortex.foundation.platform import is_mac_environment
-from docvortex.foundation.platform import is_mac_os_version_supported
-from docvortex.foundation.platform import is_windows_environment
+from ..runtime.platform import is_linux_environment, is_mac_environment, is_mac_os_version_supported, is_windows_environment
VlmEngine: TypeAlias = Literal[
"llama-cpp-engine",
diff --git a/mineru/parser/__init__.py b/mineru/parser/__init__.py
index b165245d..a608566a 100644
--- a/mineru/parser/__init__.py
+++ b/mineru/parser/__init__.py
@@ -4,8 +4,9 @@ from __future__ import annotations
from pathlib import Path
from typing import Literal
+from docvortex.document.contracts import HtmlSourceContext
+
from ..config import VlmConfig
-from docvortex.analyzers.native.html import HtmlSourceContext
from ..types import Tier
from .api_client import ApiJobStatus, MinerUApiParser
from .base import MIDDLE_JSON_SCHEMA_VERSION, DocumentParser, ParseResult
diff --git a/mineru/parser/api_client.py b/mineru/parser/api_client.py
index 45a27152..9da616e5 100644
--- a/mineru/parser/api_client.py
+++ b/mineru/parser/api_client.py
@@ -25,10 +25,10 @@ from typing import Any, Literal, cast
from urllib.parse import urlparse
import httpx
+from docvortex.assets import validate_image_sidecar_path
from ..filetypes import mime_type_for_extension
from ..types import PageInfo, Tier, validate_tier
-from docvortex.foundation.image_payload import validate_image_sidecar_path
from .base import DocumentParser, ParseResult
from .page_range import normalize_page_range_input
diff --git a/mineru/parser/api_server.py b/mineru/parser/api_server.py
index 21f5f660..ab1e1a81 100644
--- a/mineru/parser/api_server.py
+++ b/mineru/parser/api_server.py
@@ -32,6 +32,7 @@ from urllib.parse import urlparse
import click
import httpx
import uvicorn
+from docvortex.assets import validate_image_sidecar_path
from fastapi import APIRouter, Body, Depends, FastAPI, HTTPException, Path, Query, Request, Response, status
from fastapi.exceptions import RequestValidationError
from fastapi.middleware.gzip import GZipMiddleware
@@ -39,7 +40,8 @@ from fastapi.responses import JSONResponse
from pydantic import BaseModel, ConfigDict, Field
from starlette.datastructures import State
-from ..config import VlmConfig, config as mineru_config
+from ..config import VlmConfig
+from ..config import config as mineru_config
from ..errors import MineruError
from ..filetypes import (
PARSEABLE_EXTENSIONS,
@@ -47,28 +49,31 @@ from ..filetypes import (
file_type_for_extension,
is_flash_only_parse_extension,
)
-from .writer import DataWriter
-from ..types import SERVER_TIERS, TIERS_BY_SERVER_TIER, DeploymentTier, PageInfo, ServerTier, Tier, select_default_quality_tier
-from .tier import effort_for_tier
-from docvortex.foundation.image_payload import validate_image_sidecar_path
-from .process_control import ManagedProcessControlWatcher
from ..model.ocr.language import PUBLIC_OCR_LANGUAGES, validate_public_ocr_lang
+from ..types import SERVER_TIERS, TIERS_BY_SERVER_TIER, DeploymentTier, PageInfo, ServerTier, Tier, select_default_quality_tier
from ..utils.stdio import configure_standard_streams
from ..version import __version__
from . import parse_async
from .base import ParseResult
from .page_range import (
PAGE_RANGE_DESCRIPTION,
+ normalize_page_range_input,
+)
+from .page_range import (
count_pages_in_range as _count_pages_in_range,
+)
+from .page_range import (
format_page_range as _compact_page_numbers,
- normalize_page_range_input,
)
+from .process_control import ManagedProcessControlWatcher
from .tier import (
ParserRuntimeOptions,
TierDependencyError,
+ effort_for_tier,
ensure_tier_runtime_dependencies,
runtime_options_for_tier,
)
+from .writer import DataWriter
_DEFAULT_API_SERVER_TIER: ServerTier = "standard"
_API_SERVER_LANGUAGES = PUBLIC_OCR_LANGUAGES
@@ -1375,7 +1380,7 @@ async def _run_job(
source_context = None
if stype == "html":
- from docvortex.analyzers.native.html import HtmlSourceContext
+ from docvortex.document.contracts import HtmlSourceContext
source_context = HtmlSourceContext(
source_uri=extracted.source_uri,
diff --git a/mineru/parser/mineru_parser.py b/mineru/parser/mineru_parser.py
index 3c5ba79a..ba44ba37 100644
--- a/mineru/parser/mineru_parser.py
+++ b/mineru/parser/mineru_parser.py
@@ -10,16 +10,17 @@ from dataclasses import dataclass
from pathlib import Path
from typing import Any, Literal, cast
-from ..backend.analyze import aio_doc_analyze, doc_analyze
-from ..integrations.docvortex import read_source_properties
+from docvortex.document.contracts import HtmlSourceContext
from docvortex.schema import DocumentProperties
+
+from ..backend.analyze import aio_doc_analyze, doc_analyze
from ..config import VlmConfig, config
from ..errors import InvalidRequestError
from ..filetypes import IMAGE_EXTENSIONS, PAGE_RANGE_PARSE_EXTENSIONS
-from docvortex.analyzers.native.html import HtmlSourceContext
+from ..integrations.docvortex import read_source_properties
from ..types import FILE_SUFFIXES, FileSuffix, MiddleJson, ModelJson, PageInfo, Tier
-from .tier import effort_for_tier
from .base import DocumentParser, ParseResult
+from .tier import effort_for_tier
logger = logging.getLogger(__name__)
@@ -152,7 +153,7 @@ class MinerUParser(DocumentParser):
source_suffix = suffix
if suffix in IMAGE_EXTENSIONS:
- from docvortex.document.pdf.document import PDFDocument
+ from docvortex.document.pdf import PDFDocument
conversion_started_at = time.perf_counter()
input_size = len(file_bytes)
@@ -183,7 +184,7 @@ class MinerUParser(DocumentParser):
)
resolved_source_context = source_context
if suffix == "html" and resolved_source_context is None:
- from docvortex.analyzers.native.html import HtmlSourceContext
+ from docvortex.document.contracts import HtmlSourceContext
resolved_path = path.resolve()
resolved_source_context = HtmlSourceContext(
@@ -220,7 +221,8 @@ class MinerUParser(DocumentParser):
if suffix != "pdf":
return file_bytes, None, None
- from docvortex.document.pdf.document import PDFDocument
+ from docvortex.document.pdf import PDFDocument
+
from .page_range import parse_page_range
with PDFDocument(file_bytes) as doc:
diff --git a/pyproject.toml b/pyproject.toml
index 32eeaaee..12757a23 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -39,7 +39,7 @@ classifiers = [
"Programming Language :: Python :: 3.14",
]
dependencies = [
- "docvortex>=0.3.0,<1",
+ "docvortex>=0.4.0,<1",
"mineru-vl-utils>=2.0.1,<3",
"mineru-llama-cpp>=0.1.1,<1",
"click>=8.1.7",
@@ -148,6 +148,7 @@ include-package-data = true
zip-safe = false
[tool.pytest.ini_options]
+testpaths = ["tests"]
addopts = "-s --cov=mineru --cov-report html"
markers = [
"e2e: MinerU CLI 端到端用例(脚本化 docs/next/cli/mineru-e2e-test-cases.md)",
diff --git a/tests/fixtures/capture_hybrid_native_script_inputs.py b/tests/fixtures/capture_hybrid_native_script_inputs.py
index 60798cc2..aedc5904 100644
--- a/tests/fixtures/capture_hybrid_native_script_inputs.py
+++ b/tests/fixtures/capture_hybrid_native_script_inputs.py
@@ -9,8 +9,7 @@ from importlib.metadata import version
from pathlib import Path
from typing import Any
-from docvortex.document.pdf import PDFDocument
-from docvortex.document.pdf.document import get_lines_from_chars
+from docvortex.document.pdf import PDFDocument, get_lines_from_chars
_CONTROL_CHARS = {"\r", "\n", "\x02", "\ufffe", "\uffff"}
diff --git a/tests/unittest/test_docvortex_import_boundary.py b/tests/unittest/test_docvortex_import_boundary.py
new file mode 100644
index 00000000..81b52eb4
--- /dev/null
+++ b/tests/unittest/test_docvortex_import_boundary.py
@@ -0,0 +1,104 @@
+"""守卫宿主只通过声明的公开接口消费 DocVortex。"""
+
+from __future__ import annotations
+
+import ast
+from pathlib import Path
+
+import pytest
+from docvortex.public_api import PUBLIC_API
+
+
+def _dotted_name(node: ast.AST) -> str:
+ """读取静态属性链,函数调用和下标表达式不冒充模块路径。"""
+ if isinstance(node, ast.Name):
+ return node.id
+ if isinstance(node, ast.Attribute):
+ base = _dotted_name(node.value)
+ return f"{base}.{node.attr}" if base else ""
+ return ""
+
+
+def _check_source(source: str, *, model_layer: bool = False) -> list[str]:
+ """检查包括惰性和类型导入的全部语法节点,并追踪模块别名。"""
+ tree = ast.parse(source)
+ aliases: dict[str, str] = {}
+ errors: list[str] = []
+ model_modules = {"docvortex.schema", "docvortex.options", "docvortex.geometry", "docvortex.assets"}
+
+ def check_module(module: str, line: int) -> None:
+ """按公开清单和宿主层级校验模块依赖。"""
+ if module not in PUBLIC_API:
+ errors.append(f"{line}: undeclared module {module}")
+ if model_layer and module not in model_modules:
+ errors.append(f"{line}: model layer cannot import {module}")
+
+ for node in ast.walk(tree):
+ if isinstance(node, ast.ImportFrom) and not node.level and (node.module or "").startswith("docvortex"):
+ module = node.module or ""
+ check_module(module, node.lineno)
+ for alias in node.names:
+ if alias.name not in PUBLIC_API.get(module, ()):
+ errors.append(f"{node.lineno}: undeclared symbol {module}.{alias.name}")
+ target = f"{module}.{alias.name}"
+ if target in PUBLIC_API:
+ aliases[alias.asname or alias.name] = target
+ elif isinstance(node, ast.Import):
+ for alias in node.names:
+ if alias.name == "docvortex" or alias.name.startswith("docvortex."):
+ check_module(alias.name, node.lineno)
+ aliases[alias.asname or "docvortex"] = alias.name if alias.asname else "docvortex"
+ elif isinstance(node, ast.Call) and _dotted_name(node.func) in {"importlib.import_module", "__import__"}:
+ if node.args and isinstance(node.args[0], ast.Constant) and isinstance(node.args[0].value, str):
+ module = node.args[0].value
+ if module == "docvortex" or module.startswith("docvortex."):
+ check_module(module, node.lineno)
+
+ for node in ast.walk(tree):
+ if not isinstance(node, ast.Attribute):
+ continue
+ dotted = _dotted_name(node)
+ root, _, suffix = dotted.partition(".")
+ if root not in aliases:
+ continue
+ qualified = f"{aliases[root]}.{suffix}"
+ if qualified in PUBLIC_API:
+ continue
+ module, _, symbol = qualified.rpartition(".")
+ if module in PUBLIC_API and symbol not in PUBLIC_API[module]:
+ errors.append(f"{node.lineno}: undeclared member {qualified}")
+ return errors
+
+
+def test_mineru_uses_declared_docvortex_api() -> None:
+ """扫描活动生产代码,不让新增依赖穿透底层实现。"""
+ root = Path(__file__).parents[2] / "mineru"
+ errors = []
+ for path in root.rglob("*.py"):
+ errors.extend(
+ f"{path.relative_to(root)}:{error}"
+ for error in _check_source(path.read_text(), model_layer=path.is_relative_to(root / "model"))
+ )
+ assert not errors, "\n".join(errors)
+
+
+ "source",
+ [
+ "from docvortex.foundation._geometry import normalize_to_int_bbox",
+ "def load():\n from docvortex.analyzers.native.pdf import native_text",
+ "if TYPE_CHECKING:\n from docvortex.geometry import unknown",
+ "import docvortex.geometry as geometry\ngeometry.unknown()",
+ "from docvortex.document import page_range as ranges\nranges.unknown()",
+ "import docvortex\ndocvortex.foundation._geometry.normalize_to_int_bbox([])",
+ "import importlib\nimportlib.import_module('docvortex.foundation._geometry')",
+ ],
+)
+def test_guard_rejects_unpublished_imports(source: str) -> None:
+ """覆盖不同导入形式和模块成员访问,避免只检查 import 行前缀。"""
+ assert _check_source(source)
+
+
+def test_model_layer_cannot_import_analyzers() -> None:
+ """即使符号公开,模型层也不能反向依赖分析器。"""
+ assert _check_source("from docvortex.analyzers.pdf import prepare_text_evidence", model_layer=True)
diff --git a/tests/unittest/test_kit_gradio.py b/tests/unittest/test_kit_gradio.py
index 47ed0a67..30314782 100644
--- a/tests/unittest/test_kit_gradio.py
+++ b/tests/unittest/test_kit_gradio.py
@@ -1,6 +1,4 @@
from __future__ import annotations
-from docvortex.schema import Producer
-from mineru.integrations.docvortex import build_metadata
import asyncio
import io
@@ -16,21 +14,25 @@ from unittest.mock import AsyncMock, Mock
import httpx
import pytest
from click import unstyle
+from docvortex.schema import Producer
from fastapi.testclient import TestClient
from PIL import Image, ImageStat
from pypdf import PdfReader, PdfWriter
from reportlab.pdfgen import canvas
+from typer.main import get_command
+from typer.testing import CliRunner
from mineru.filetypes import FLASH_ONLY_PARSE_EXTENSIONS, IMAGE_EXTENSIONS, PARSEABLE_EXTENSIONS
+from mineru.integrations.docvortex import build_metadata
from mineru.kit.commands import gradio as gradio_command
from mineru.kit.gradio import app as gradio_app
from mineru.kit.gradio import client as gradio_client
from mineru.kit.gradio.app import build_gradio_app
from mineru.kit.gradio.artifacts import create_run_artifacts, persist_parse_result, render_download
from mineru.kit.gradio.client import (
+ STATUS_DOWNLOADING_RESULT,
GradioArtifactClient,
ManagedLocalApiServer,
- STATUS_DOWNLOADING_RESULT,
V1ArtifactClient,
V1ArtifactError,
V1ServerCapabilities,
@@ -42,8 +44,6 @@ from mineru.parser import api_server as parser_api_server
from mineru.parser.base import ParseResult
from mineru.types import BlockType, ImageBlock, ImageBodyBlock, MiddleJson, ModelJson, PageInfo, TextBlock, TextSpan
from mineru.version import __version__
-from typer.main import get_command
-from typer.testing import CliRunner
runner = CliRunner()
@@ -1436,8 +1436,9 @@ def test_gradio_output_failure_stops_timer_and_allows_next_conversion(tmp_path:
@pytest.mark.parametrize("page_indices", [(), (2, 0, 1)])
def test_layout_preview_matches_shared_renderer_and_pdf_document(tmp_path: Path, page_indices: tuple[int, ...]) -> None:
"""三个公开入口使用相同页映射、边框和页面属性,缺失页不回退到其他结果。"""
- from docvortex.document.pdf.document import PDFDocument
+ from docvortex.document.pdf import PDFDocument
from docvortex.visualization import render_layout_pdf
+
from mineru.kit.gradio.preview import draw_layout_overlay
source = tmp_path / "source.pdf"
diff --git a/tests/unittest/test_native_pdf_table_pipeline.py b/tests/unittest/test_native_pdf_table_pipeline.py
index e2d9eb00..223c4614 100644
--- a/tests/unittest/test_native_pdf_table_pipeline.py
+++ b/tests/unittest/test_native_pdf_table_pipeline.py
@@ -10,7 +10,7 @@ from unittest.mock import MagicMock
import numpy as np
import pytest
-from docvortex.document.pdf.document import PDFDocument, PDFPageTextGeometry
+from docvortex.document.pdf import PDFDocument, PDFPageTextGeometry
from PIL import Image
from mineru.backend.analysis.pdf import formulas as pdf_formulas
@@ -68,7 +68,7 @@ def test_medium_native_table_priority_accepts_html_and_removes_internal_text_and
html = "<table><tbody><tr><td>A</td></tr></tbody></table>"
recover = MagicMock(return_value=SimpleNamespace(html=html, source="vector_grid", confidence=1.0))
- monkeypatch.setattr(pdf_tables, "recover_native_pdf_table", recover)
+ monkeypatch.setattr(pdf_tables, "recover_table_region", recover)
table_block = {
"type": BlockType.TABLE,
"bbox": [0.1, 0.2, 0.9, 0.8],
@@ -141,10 +141,10 @@ def test_medium_native_table_priority_accepts_html_and_removes_internal_text_and
removed_formula_blocks=2,
removed_formula_layout_items=3,
)
- table_input = recover.call_args.args[0]
- assert table_input.table_bbox == pytest.approx((10.0, 40.0, 90.0, 160.0))
- assert table_input.page_size == (100.0, 200.0)
- assert table_input.angle == 0
+ table_page = recover.call_args.args[0]
+ assert recover.call_args.args[1] == pytest.approx((10.0, 40.0, 90.0, 160.0))
+ assert table_page.page_size == (100.0, 200.0)
+ assert recover.call_args.kwargs["angle"] == 0
page.get_chars_with_geometry.assert_called_once_with()
page.get_drawing_lines.assert_called_once_with()
page.get_path_infos.assert_called_once_with()
@@ -165,7 +165,7 @@ def test_hybrid_native_table_priority_falls_back_for_complex_content(
"""验证 Medium 图片和代码复杂内容继续保留现有模型回落。"""
recover = MagicMock()
- monkeypatch.setattr(pdf_tables, "recover_native_pdf_table", recover)
+ monkeypatch.setattr(pdf_tables, "recover_table_region", recover)
table_block = {
"type": BlockType.TABLE,
"bbox": [0.1, 0.1, 0.9, 0.9],
@@ -198,7 +198,7 @@ def test_high_native_table_priority_ignores_layout_formula_and_removes_duplicate
html = "<table><tbody><tr><td>T</td></tr></tbody></table>"
recover = MagicMock(return_value=SimpleNamespace(html=html, source="vector_grid", confidence=1.0))
- monkeypatch.setattr(pdf_tables, "recover_native_pdf_table", recover)
+ monkeypatch.setattr(pdf_tables, "recover_table_region", recover)
table_block = {"type": BlockType.TABLE, "bbox": [0.1, 0.1, 0.9, 0.9], "angle": 0}
equation_block = {"type": BlockType.EQUATION, "bbox": [0.2, 0.2, 0.4, 0.4]}
layout_res = [{"label": "inline_formula", "bbox": [20, 20, 40, 40]}]
@@ -241,7 +241,7 @@ def test_hybrid_native_table_priority_attempts_rotated_table(
html = "<table><tbody><tr><td>rotated</td></tr></tbody></table>"
recover = MagicMock(return_value=SimpleNamespace(html=html, source="vector_grid", confidence=1.0))
- monkeypatch.setattr(pdf_tables, "recover_native_pdf_table", recover)
+ monkeypatch.setattr(pdf_tables, "recover_table_region", recover)
table_block = {
"type": BlockType.TABLE,
"bbox": [0.1, 0.1, 0.9, 0.9],
@@ -265,7 +265,7 @@ def test_hybrid_native_table_priority_attempts_rotated_table(
accepted=1,
)
recover.assert_called_once()
- assert recover.call_args.args[0].angle == angle
+ assert recover.call_args.kwargs["angle"] == angle
@pytest.mark.parametrize("effort", ["medium", "high"])
@@ -306,8 +306,8 @@ def test_hybrid_native_table_priority_keeps_none_and_exception_fallbacks(
) -> None:
"""验证规则拒绝或异常时逐表保留现有模型路径。"""
- recover = MagicMock(side_effect=[None, RuntimeError("broken")])
- monkeypatch.setattr(pdf_tables, "recover_native_pdf_table", recover)
+ recover = MagicMock(side_effect=[None, pdf_tables.PDFTableRecoveryError("broken")])
+ monkeypatch.setattr(pdf_tables, "recover_table_region", recover)
tables = [
{"type": BlockType.TABLE, "bbox": [0.05, 0.1, 0.45, 0.9], "angle": 270},
{"type": BlockType.TABLE, "bbox": [0.55, 0.1, 0.95, 0.9], "angle": 90},
@@ -337,7 +337,7 @@ def test_hybrid_native_table_priority_keeps_none_and_exception_fallbacks(
errors=1,
)
assert recover.call_count == 2
- assert [call.args[0].angle for call in recover.call_args_list] == [270, 90]
+ assert [call.kwargs["angle"] for call in recover.call_args_list] == [270, 90]
page.get_chars_with_geometry.assert_called_once_with()
@@ -537,3 +537,17 @@ def test_high_txt_window_excludes_native_table_from_vlm(
vlm_predictor.batch_extract_with_layout.assert_called_once()
with pytest.raises(ValueError, match="closed image"):
page_image.getpixel((0, 0))
+
+
+def test_table_materialization_failure_is_not_silently_downgraded(monkeypatch: pytest.MonkeyPatch) -> None:
+ """保留原有边界:结构恢复失败可回退,HTML 物化异常仍向调用方传播。"""
+ monkeypatch.setattr(pdf_tables, "recover_table_region", MagicMock(side_effect=ValueError("materialization")))
+ with Image.new("RGB", (100, 100), "white") as image:
+ with pytest.raises(ValueError, match="materialization"):
+ pdf_tables._apply_native_txt_table_priority(
+ [[{"type": BlockType.TABLE, "bbox": [0.1, 0.1, 0.9, 0.9]}]],
+ [[]],
+ [_build_native_pdf_page()],
+ [{"img_pil": image, "scale": 1.0}],
+ effort="medium",
+ )
diff --git a/tests/unittest/test_pdf_analyze_block_content.py b/tests/unittest/test_pdf_analyze_block_content.py
index 5b72b364..d9fa9553 100644
--- a/tests/unittest/test_pdf_analyze_block_content.py
+++ b/tests/unittest/test_pdf_analyze_block_content.py
@@ -1,31 +1,28 @@
from __future__ import annotations
-from docvortex.schema import Producer
-from mineru.integrations.docvortex import build_metadata
from unittest.mock import MagicMock
import numpy as np
import pytest
-from PIL import Image
+from _span_test_utils import inline, inline_text
+from docvortex.analyzers.pdf import PDF_NATIVE_SCRIPT_MARKUP_KEY, PDFTextEvidence, apply_text_evidence
+from docvortex.document.pdf import PDFPageTextGeometry
+from docvortex.schema import Producer
from mineru_vl_utils.structs import ContentBlock as VlmContentBlock
from mineru_vl_utils.structs import ExtractResult
+from PIL import Image
from mineru.backend import analyze
from mineru.backend.analysis.pdf import constants, layout, normalization, ocr, pipeline, window
from mineru.backend.analysis.pdf.text import content as text_content
from mineru.backend.analysis.pdf.text.models import _AnalyzeLine, _AnalyzeSpan
-from mineru.backend.postprocess import document as postprocess_document
-from docvortex.document.pdf.document import PDFPageTextGeometry
-from mineru.types import RAW_ALGORITHM, RAW_CAPTION, RAW_FOOTNOTE
from mineru.backend.analysis.pdf.text.native import (
POST_OCR_FALLBACK_CONTENT_KEY,
POST_OCR_FALLBACK_SCORE_KEY,
)
-from docvortex.analyzers.native.pdf.text_styles import PDF_NATIVE_SCRIPT_MARKUP_KEY
-from docvortex.analyzers.native.pdf.text_styles import materialize_pdf_inline_spans
-from mineru.types import BlockType, ContentType, MiddleJson, ModelJson
-
-from _span_test_utils import inline, inline_text
+from mineru.backend.postprocess import document as postprocess_document
+from mineru.integrations.docvortex import build_metadata
+from mineru.types import RAW_ALGORITHM, RAW_CAPTION, RAW_FOOTNOTE, BlockType, ContentType, MiddleJson, ModelJson
def _build_text_lines(*contents: str) -> list[_AnalyzeLine]:
@@ -448,8 +445,8 @@ def test_fill_window_batches_txt_post_ocr_before_page_content(monkeypatch: pytes
monkeypatch.setattr(text_content, "_build_page_text_formula_spans", lambda *_args: [])
monkeypatch.setattr(
text_content,
- "build_pdf_native_visual_lines_and_styles",
- lambda *_args, **_kwargs: (PDFPageTextGeometry([], {}, {}), [], [], [], []),
+ "prepare_text_evidence",
+ lambda *_args, **_kwargs: PDFTextEvidence((100.0, 100.0), PDFPageTextGeometry([], {}, {})),
)
monkeypatch.setattr(
text_content,
@@ -683,7 +680,7 @@ def test_native_script_ownership_propagates_only_when_content_is_filled() -> Non
assert filled_block[PDF_NATIVE_SCRIPT_MARKUP_KEY] is True
assert PDF_NATIVE_SCRIPT_MARKUP_KEY not in existing_block
- materialize_pdf_inline_spans([filled_block, existing_block])
+ apply_text_evidence([filled_block, existing_block], PDFTextEvidence((100.0, 100.0)))
assert inline_text(filled_block["content"]) == "A2"
assert filled_block["content"][1]["styles"] == ["superscript"]
diff --git a/tests/unittest/test_pdf_analyze_visual_blocks.py b/tests/unittest/test_pdf_analyze_visual_blocks.py
index 28b983e5..22166bdd 100644
--- a/tests/unittest/test_pdf_analyze_visual_blocks.py
+++ b/tests/unittest/test_pdf_analyze_visual_blocks.py
@@ -1,10 +1,9 @@
from __future__ import annotations
-import json
-from docvortex.codecs.json import load_middle, load_model
import asyncio
import base64
import inspect
+import json
import threading
from io import BytesIO
from pathlib import Path
@@ -13,24 +12,28 @@ from unittest.mock import MagicMock
import numpy as np
import pytest
+from _span_test_utils import inline, inline_text
+from docvortex.codecs.json import load_middle, load_model
+from docvortex.document.pdf import visuals
from PIL import Image, ImageDraw
from mineru.backend import analyze
from mineru.backend.analysis import office
-from mineru.backend.analysis.pdf import constants
-from mineru.backend.analysis.pdf import formulas
-from mineru.backend.analysis.pdf import normalization
-from mineru.backend.analysis.pdf import pipeline
-from mineru.backend.analysis.pdf import tables
-from docvortex.document.pdf import visuals
-from mineru.backend.analysis.pdf import window
-from mineru.types import RAW_ALGORITHM, RAW_CAPTION, RAW_FOOTNOTE, RAW_FORMULA_NUMBER, RAW_PHONETIC
-from mineru.types import FILE_SUFFIXES, BlockType, FileSuffix, MiddleJson, ModelJson
+from mineru.backend.analysis.pdf import constants, formulas, normalization, pipeline, tables, visual_containers, window
+from mineru.types import (
+ FILE_SUFFIXES,
+ RAW_ALGORITHM,
+ RAW_CAPTION,
+ RAW_FOOTNOTE,
+ RAW_FORMULA_NUMBER,
+ RAW_PHONETIC,
+ BlockType,
+ FileSuffix,
+ MiddleJson,
+ ModelJson,
+)
from mineru.version import __version__ as mineru_version
-from _span_test_utils import inline, inline_text
-
-
JPEG_DATA_URI_PREFIX = "data:image/jpeg;base64,"
RED = (255, 0, 0)
GREEN = (0, 255, 0)
@@ -371,7 +374,7 @@ def test_xhigh_layout_image_supplements_missing_container_before_crop() -> None:
]
]
- visuals._supplement_missing_image_block_containers(
+ visual_containers.supplement_missing_image_block_containers(
[page_model_list],
layout_blocks_list,
)
@@ -448,7 +451,7 @@ def test_xhigh_layout_image_fallback_requires_visual_count_and_coverage(
page_model_list = [{"type": block_type, "bbox": bbox} for block_type, bbox in zip(block_types, block_bboxes)]
layout_blocks_list = [[{"type": layout_type, "bbox": [0.0, 0.0, 1.0, 1.0], "angle": 0}]]
- visuals._supplement_missing_image_block_containers(
+ visual_containers.supplement_missing_image_block_containers(
[page_model_list],
layout_blocks_list,
)
@@ -474,7 +477,7 @@ def test_xhigh_layout_image_fallback_uses_whitelist_for_area_but_absorbs_all_typ
{"type": BlockType.CHART_CAPTION, "bbox": [0.9, 0.0, 1.0, 1.0]},
]
- visuals._supplement_missing_image_block_containers(
+ visual_containers.supplement_missing_image_block_containers(
[insufficient_page],
layout_blocks_list,
)
@@ -492,7 +495,7 @@ def test_xhigh_layout_image_fallback_uses_whitelist_for_area_but_absorbs_all_typ
external_text,
]
- visuals._supplement_missing_image_block_containers(
+ visual_containers.supplement_missing_image_block_containers(
[qualifying_page],
layout_blocks_list,
)
@@ -525,7 +528,7 @@ def test_xhigh_layout_image_fallback_does_not_duplicate_existing_or_overlapping_
]
]
- visuals._supplement_missing_image_block_containers(
+ visual_containers.supplement_missing_image_block_containers(
[page_model_list],
layout_blocks_list,
)
@@ -1141,6 +1144,7 @@ def test_aio_doc_analyze_runs_sync_entrypoint_in_thread_and_forwards_arguments(
"file_suffix": "pptx",
"source_context": None,
"vlm_config": None,
+ "source_properties": None,
}
assert observed["thread_id"] != caller_thread_id
diff --git a/tests/unittest/test_pdf_model_inputs.py b/tests/unittest/test_pdf_model_inputs.py
new file mode 100644
index 00000000..1c560395
--- /dev/null
+++ b/tests/unittest/test_pdf_model_inputs.py
@@ -0,0 +1,85 @@
+"""验证工具归位后模型输入与推理平台策略保持原行为。"""
+
+from __future__ import annotations
+
+import numpy as np
+import pytest
+from docvortex.assets import image_size
+
+from mineru.backend.analysis.pdf.model_inputs import (
+ _bbox_to_pixel_bbox,
+ _get_medium_table_virtual_image_bbox,
+ _normalize_layout_bbox_to_unit,
+ _normalize_medium_content,
+ _sidecar_bbox_to_page_bbox,
+)
+from mineru.model.runtime import platform
+
+
+ ("bbox", "pixels", "points"),
+ [
+ ((0.1, 0.2, 0.9, 0.8), (10, 40, 90, 160), (10, 40, 90, 160)),
+ ((10, 40, 90, 160), (10, 40, 90, 160), (5, 20, 45, 80)),
+ ((90, 160, 10, 40), (10, 40, 90, 160), (5, 20, 45, 80)),
+ ((-10, -20, 300, 500), (-10, -20, 300, 500), (0, 0, 100, 200)),
+ ],
+)
+def test_model_coordinate_interpretation(bbox: tuple, pixels: tuple, points: tuple) -> None:
+ """归一化判断留在模型边界,像素和 sidecar 的解释不被混为一谈。"""
+ assert _bbox_to_pixel_bbox(bbox, (100, 200)) == pytest.approx(pixels)
+ assert _sidecar_bbox_to_page_bbox(bbox, (100, 200), 2) == pytest.approx(points)
+
+
+def test_model_precision_and_virtual_token() -> None:
+ """保持模型三位小数精度、token 尺寸与图像边界裁剪。"""
+ assert _normalize_layout_bbox_to_unit((10, 20, 90, 100), (300, 600)) == [0.033, 0.033, 0.3, 0.167]
+ assert _get_medium_table_virtual_image_bbox((0, 0, 2, 2), (100, 200)) == (0, 0, 6, 6)
+ assert _get_medium_table_virtual_image_bbox((20, 30, 40, 50), (100, 200)) == (25, 35, 35, 45)
+ assert _bbox_to_pixel_bbox((1, 1, 1, 2), (100, 200)) is None
+ assert _sidecar_bbox_to_page_bbox((1, 1, 2, 2), (100, 200), 0) is None
+
+
+ ("value", "expected"),
+ [
+ (" x ", "x"),
+ ([" a ", "", "b"], " a \nb"),
+ (None, ""),
+ (42, ""),
+ ],
+)
+def test_medium_model_text(value: object, expected: str) -> None:
+ """保留字符串与列表输出的不同清洗约定。"""
+ assert _normalize_medium_content(value) == expected
+
+
+def test_numpy_image_dimensions() -> None:
+ """图像尺寸读取返回宽高,不把 NumPy size 标量作为图像宽高。"""
+ assert image_size(np.zeros((20, 30, 3), dtype=np.uint8)) == (30, 20)
+
+
+ ("system", "machine", "version", "supported"),
+ [
+ ("Darwin", "arm64", "14.0", True),
+ ("Darwin", "arm64", "13.5", False),
+ ("Darwin", "x86_64", "14.0", False),
+ ("Linux", "aarch64", "14.0", False),
+ ("Windows", "AMD64", "", False),
+ ],
+)
+def test_inference_platform_threshold(
+ monkeypatch: pytest.MonkeyPatch,
+ system: str,
+ machine: str,
+ version: str,
+ supported: bool,
+) -> None:
+ """平台门槛由宿主显式给定,保留操作系统与 CPU 双重约束。"""
+ monkeypatch.setattr(platform.platform, "system", lambda: system)
+ monkeypatch.setattr(platform.platform, "machine", lambda: machine)
+ monkeypatch.setattr(platform.platform, "mac_ver", lambda: (version, (), ""))
+ assert platform.is_mac_os_version_supported("14.0") is supported
+ assert platform.is_windows_environment() is (system == "Windows")
+ assert platform.is_linux_environment() is (system == "Linux")
diff --git a/tests/unittest/test_pdf_native_script_detection.py b/tests/unittest/test_pdf_native_script_detection.py
index 9678333d..06397be3 100644
--- a/tests/unittest/test_pdf_native_script_detection.py
+++ b/tests/unittest/test_pdf_native_script_detection.py
@@ -7,9 +7,8 @@ from dataclasses import dataclass
from pathlib import Path
import pytest
-from docvortex.analyzers.native.pdf.text_styles import PDF_NATIVE_SCRIPT_MARKUP_KEY
-from docvortex.document.pdf.document import PDFDocument, get_lines_from_chars
-from docvortex.document.pdf.text.contracts import Bbox, Char
+from docvortex.analyzers.pdf import PDF_NATIVE_SCRIPT_MARKUP_KEY
+from docvortex.document.pdf import Bbox, Char, PDFDocument, get_lines_from_chars
from mineru.backend.analysis.pdf.text import native
from mineru.backend.analysis.pdf.text.models import _AnalyzeSpan
diff --git a/tests/unittest/test_pdf_text_styles.py b/tests/unittest/test_pdf_text_styles.py
index 3b99bba7..7c5ae8a1 100644
--- a/tests/unittest/test_pdf_text_styles.py
+++ b/tests/unittest/test_pdf_text_styles.py
@@ -6,7 +6,8 @@ from unittest.mock import MagicMock
import pytest
from _span_test_utils import inline_text
-from docvortex.analyzers.native.pdf.text_styles import (
+from docvortex.analyzers.pdf import (
+ PDFTextEvidence,
PDFTextLinkLine,
PDFTextLinkRange,
PDFTextScriptLine,
@@ -14,10 +15,9 @@ from docvortex.analyzers.native.pdf.text_styles import (
PDFTextStyleLine,
PDFTextStyleRange,
)
-from docvortex.document.pdf.document import PDFPageTextGeometry
+from docvortex.document.pdf import PDFPageTextGeometry
from mineru.backend.analysis.pdf.text import content as text_content
-from mineru.backend.analysis.pdf.text import styles as text_style_enrichment
from mineru.backend.analysis.pdf.text.models import _AnalyzeLine, _AnalyzeSpan
from mineru.backend.analysis.pdf.text.native import txt_spans_extract
from mineru.types import (
@@ -88,7 +88,9 @@ def test_hybrid_txt_reuses_loaded_chars_and_applies_styles(
0,
)
]
- build_styles = MagicMock(return_value=(page_text_geometry, [], style_lines, link_lines, []))
+ build_styles = MagicMock(
+ return_value=PDFTextEvidence((100.0, 100.0), page_text_geometry, tuple(style_lines), tuple(link_lines))
+ )
observed_geometry: list[object] = []
def fake_fill_native(
@@ -116,7 +118,7 @@ def test_hybrid_txt_reuses_loaded_chars_and_applies_styles(
monkeypatch.setattr(
text_content,
- "build_pdf_native_visual_lines_and_styles",
+ "prepare_text_evidence",
build_styles,
)
monkeypatch.setattr(text_content, "_fill_native_pdf_text_spans", fake_fill_native)
@@ -190,8 +192,10 @@ def test_hybrid_txt_efforts_apply_dehyphenated_links(
]
monkeypatch.setattr(
text_content,
- "build_pdf_native_visual_lines_and_styles",
- lambda *_args, **_kwargs: (PDFPageTextGeometry([], {}, {}), [], [], link_lines, []),
+ "prepare_text_evidence",
+ lambda *_args, **_kwargs: PDFTextEvidence(
+ (100.0, 100.0), PDFPageTextGeometry([], {}, {}), tuple([]), tuple(link_lines), tuple([])
+ ),
)
monkeypatch.setattr(
text_content,
@@ -276,8 +280,8 @@ def test_hybrid_txt_efforts_apply_shared_script_sidecar(
source_index=0,
angle=0,
)
- build_evidence = MagicMock(return_value=(page_geometry, [], [], [], [script_line]))
- monkeypatch.setattr(text_content, "build_pdf_native_visual_lines_and_styles", build_evidence)
+ build_evidence = MagicMock(return_value=PDFTextEvidence((100.0, 100.0), page_geometry, scripts=(script_line,)))
+ monkeypatch.setattr(text_content, "prepare_text_evidence", build_evidence)
monkeypatch.setattr(text_content, "_build_page_text_formula_spans", lambda *_args: [])
def fake_fill_native(*_args: object, detect_scripts: bool, **_kwargs: object) -> list[_AnalyzeSpan]:
@@ -322,28 +326,6 @@ def test_hybrid_txt_efforts_apply_shared_script_sidecar(
]
-def test_hybrid_layout_formula_regions_hard_exclude_script_ranges() -> None:
- """验证 Hybrid TXT 只保留 layout 行内公式 bbox 外的脚本证据。"""
-
- line = PDFTextScriptLine(
- bbox=(0.0, 0.0, 40.0, 20.0),
- text="x2y3",
- script_ranges=(
- PDFTextScriptRange(1, 2, "superscript", (10.0, 2.0, 15.0, 8.0), 2, True),
- PDFTextScriptRange(3, 4, "subscript", (30.0, 12.0, 35.0, 18.0), 2, False),
- ),
- source_index=0,
- angle=0,
- )
-
- filtered = text_style_enrichment._exclude_layout_formula_script_ranges(
- [line],
- [(8.0, 0.0, 18.0, 10.0)],
- )
-
- assert [(item.start, item.end, item.style) for item in filtered[0].script_ranges] == [(3, 4, "subscript")]
-
-
def test_ocr_path_does_not_collect_or_apply_pdf_styles(
monkeypatch: pytest.MonkeyPatch,
) -> None:
@@ -352,7 +334,7 @@ def test_ocr_path_does_not_collect_or_apply_pdf_styles(
build_styles = MagicMock()
monkeypatch.setattr(
text_content,
- "build_pdf_native_visual_lines_and_styles",
+ "prepare_text_evidence",
build_styles,
)
monkeypatch.setattr(
@@ -412,7 +394,7 @@ def test_high_char_count_txt_path_skips_pdf_text_enrichment(
build_enrichment = MagicMock()
monkeypatch.setattr(
text_content,
- "build_pdf_native_visual_lines_and_styles",
+ "prepare_text_evidence",
build_enrichment,
)
plain_span = _AnalyzeSpan(
diff --git a/tools/compare_docvortex_boundary.py b/tools/compare_docvortex_boundary.py
new file mode 100644
index 00000000..df9a365e
--- /dev/null
+++ b/tools/compare_docvortex_boundary.py
@@ -0,0 +1,121 @@
+"""比较冻结基线和新版本产物,仅忽略明确声明的版本与容器时间字段。"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import re
+from pathlib import Path
+from uuid import NAMESPACE_URL, uuid5
+from xml.etree import ElementTree
+from zipfile import ZipFile
+
+
+def normalized_json(path: Path) -> object:
+ """仅归一化根文档的引擎 producer 版本,保留全部语义和来源属性。"""
+ value = json.loads(path.read_text(encoding="utf-8"))
+ if isinstance(value, dict):
+ producer = value.get("metadata", {}).get("producer")
+ if isinstance(producer, dict) and producer.get("name") == "docvortex":
+ producer["version"] = "<engine-version>"
+ return value
+
+
+def archive_members(path: Path) -> dict[str, bytes]:
+ """比较容器内容,校验并归一化由 producer 版本派生的 EPUB 标识。"""
+ with ZipFile(path) as archive:
+ members = {name: archive.read(name) for name in archive.namelist()}
+ if path.suffix == ".epub":
+ for name in members:
+ if name.endswith(".opf"):
+ document = ElementTree.fromstring(members[name])
+ namespaces = {"opf": "http://www.idpf.org/2007/opf", "dc": "http://purl.org/dc/elements/1.1/"}
+ metadata = document.find("opf:metadata", namespaces)
+ identifier = metadata.find("dc:identifier", namespaces).text
+ seed = {
+ "title": metadata.find("dc:title", namespaces).text,
+ "authors": tuple(item.text for item in metadata.findall("dc:creator", namespaces)),
+ "language": metadata.find("dc:language", namespaces).text,
+ "middle_json": json.loads((path.parent / "middle.json").read_text()),
+ }
+ expected = stable_identifier(seed)
+ if identifier != expected:
+ raise ValueError(f"EPUB identifier is not derived from captured document: {path}")
+ seed["middle_json"] = normalized_json(path.parent / "middle.json")
+ members[name] = members[name].replace(identifier.encode(), stable_identifier(seed).encode())
+ members[name] = re.sub(
+ rb'(<meta property="dcterms:modified">)[^<]*(</meta>)', rb"\1<timestamp>\2", members[name]
+ )
+ return members
+
+
+def stable_identifier(seed: dict[str, object]) -> str:
+ """独立复算现有 EPUB 的确定性标识,避免盲目忽略内容标识差异。"""
+ payload = json.dumps(seed, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode()
+ return f"urn:uuid:{uuid5(NAMESPACE_URL, hashlib.sha256(payload).hexdigest())}"
+
+
+def compare(before: Path, after: Path) -> dict[str, object]:
+ """逐文档核对样本、协议、全部渲染和素材,输出机器可读差分证据。"""
+ before_manifest = json.loads((before / "manifest.json").read_text())
+ after_manifest = json.loads((after / "manifest.json").read_text())
+ checks = []
+ errors = []
+ if len(before_manifest["documents"]) != len(after_manifest["documents"]):
+ errors.append("document count differs")
+ for old, new in zip(before_manifest["documents"], after_manifest["documents"]):
+ for key in ("name", "sha256", "pages"):
+ if old.get(key) != new.get(key):
+ errors.append(f"{old['name']}: {key} differs")
+ if "error" in old or "error" in new:
+ errors.append(f"{old['name']}: capture failed")
+ for engine in ("mineru", "docvortex"):
+ directory = Path(Path(old["name"]).stem) / engine
+ original_files = {path.relative_to(before) for path in (before / directory).rglob("*") if path.is_file()}
+ current_files = {path.relative_to(after) for path in (after / directory).rglob("*") if path.is_file()}
+ if original_files != current_files:
+ errors.append(f"{directory}: file inventory differs")
+ for relative in sorted(original_files & current_files):
+ a, b = before / relative, after / relative
+ identical = a.read_bytes() == b.read_bytes()
+ if identical:
+ mode, equal = "bytes", True
+ elif a.suffix == ".json":
+ mode, equal = "producer-version-only", normalized_json(a) == normalized_json(b)
+ elif a.suffix in {".docx", ".epub"}:
+ mode, equal = "archive-timestamps-and-verified-producer-id", archive_members(a) == archive_members(b)
+ else:
+ mode, equal = "bytes", False
+ checks.append(
+ {
+ "path": str(relative),
+ "equal": equal,
+ "comparison": mode,
+ "sha256": hashlib.sha256(b.read_bytes()).hexdigest(),
+ }
+ )
+ if not equal:
+ errors.append(str(relative))
+ for outputs in (old["outputs"], new["outputs"]):
+ for name, value in outputs.items():
+ if isinstance(value, dict):
+ errors.append(f"{old['name']}/{name}: {value}")
+ return {"documents": len(before_manifest["documents"]), "checks": checks, "errors": errors, "passed": not errors}
+
+
+def main() -> None:
+ """将差分报告保存到指定路径,发生未声明的变化时返回非零状态。"""
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("before", type=Path)
+ parser.add_argument("after", type=Path)
+ parser.add_argument("--output", type=Path, required=True)
+ args = parser.parse_args()
+ report = compare(args.before, args.after)
+ args.output.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
+ print(json.dumps({key: value for key, value in report.items() if key != "checks"}, ensure_ascii=False))
+ raise SystemExit(0 if report["passed"] else 1)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tools/validate_docvortex_boundary.py b/tools/validate_docvortex_boundary.py
new file mode 100644
index 00000000..df305d1c
--- /dev/null
+++ b/tools/validate_docvortex_boundary.py
@@ -0,0 +1,95 @@
+"""捕获两仓库的真实解析与九种输出,支持冻结源码和新 wheel 的差分验收。"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import sys
+from pathlib import Path
+
+
+def capture(args: argparse.Namespace) -> None:
+ """使用显式源码或当前安装包保存完整协议、素材和可视化产物。"""
+ if args.mineru_source:
+ sys.path.insert(0, str(args.mineru_source.resolve()))
+ if args.docvortex_source:
+ sys.path.insert(0, str(args.docvortex_source.resolve()))
+ import docvortex
+ from docvortex.api import parse
+ from docvortex.api import render as render_document
+
+ import mineru
+ from mineru.backend.analyze import doc_analyze
+ from mineru.config import config
+ from mineru.render import RenderFormat, render
+
+ config.llm_aided.features.title_leveling = False
+ config.llm_aided.features.cross_page_table_cell_merge = False
+ args.output.mkdir(parents=True, exist_ok=True)
+ manifest = {"imports": {"mineru": mineru.__file__, "docvortex": docvortex.__file__}, "documents": []}
+ for name in args.documents:
+ source = args.samples / name
+ payload = source.read_bytes()
+ entry = {"name": name, "sha256": hashlib.sha256(payload).hexdigest(), "outputs": {}}
+ try:
+ for engine in ("mineru", "docvortex"):
+ directory = args.output / source.stem / engine
+ directory.mkdir(parents=True, exist_ok=True)
+ if engine == "mineru":
+ middle, model = doc_analyze(payload, effort="flash", parse_mode="txt")
+ assets = None
+ else:
+ result = parse(payload, file_suffix="pdf", keep_model_json=True)
+ middle, model, assets = result.middle_json, result.model_json, result.assets
+ entry.setdefault("pages", {})[engine] = len(middle.pages)
+ for label, value in (("model", model), ("middle", middle)):
+ (directory / f"{label}.json").write_text(value.to_json(skip_defaults=False), encoding="utf-8")
+ for target in RenderFormat:
+ if engine == "docvortex" and target.value.startswith("content_list"):
+ continue
+ try:
+ if engine == "mineru":
+ content = render(middle, target)
+ data = (
+ content
+ if isinstance(content, bytes)
+ else (
+ content
+ if isinstance(content, str)
+ else json.dumps(content, ensure_ascii=False, sort_keys=True)
+ ).encode()
+ )
+ else:
+ artifact = render_document(middle, target.value, assets=assets)
+ data = artifact.content
+ for asset_name, asset_bytes in artifact.assets.items():
+ asset_path = directory / asset_name
+ asset_path.parent.mkdir(parents=True, exist_ok=True)
+ asset_path.write_bytes(asset_bytes)
+ suffix = {"markdown": "md", "structured_content": "json"}.get(target.value, target.value)
+ output = directory / f"render.{suffix}"
+ output.write_bytes(data)
+ entry["outputs"][f"{engine}/{target.value}"] = hashlib.sha256(data).hexdigest()
+ except Exception as error:
+ entry["outputs"][f"{engine}/{target.value}"] = {"error": f"{type(error).__name__}: {error}"}
+ except Exception as error:
+ entry["error"] = f"{type(error).__name__}: {error}"
+ manifest["documents"].append(entry)
+ (args.output / "manifest.json").write_text(json.dumps(manifest, ensure_ascii=False, indent=2), encoding="utf-8")
+ print(json.dumps(entry, ensure_ascii=False), flush=True)
+
+
+def main() -> None:
+ """解析可重放验收所需的源码路径、样本清单和输出目录。"""
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--mineru-source", type=Path)
+ parser.add_argument("--docvortex-source", type=Path)
+ parser.add_argument("--samples", type=Path, required=True)
+ parser.add_argument("--output", type=Path, required=True)
+ parser.add_argument("documents", nargs="+", help="样本目录内的 PDF 文件名")
+ capture(parser.parse_args())
+
+
+if __name__ == "__main__":
+ main()