Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.
Python ai4sciencedocument-analysisdocxextract-datalayout-analysisocrparserpdfpdf-converterpdf-extractor-llmpdf-extractor-pretrainpdf-extractor-ragpdf-parserpptxpythonxlsx
Update DocVortex dependency version and enhance Middle JSON compatibility handling
myhloli332bb866cf51
diff --git a/docs/doclib-middle-json-compat-validation.md b/docs/doclib-middle-json-compat-validation.mdnew file mode 100644index 00000000..8ca94c4e--- /dev/null+++ b/docs/doclib-middle-json-compat-validation.md@@ -0,0 +1,52 @@+# Doclib 历史 Middle JSON 兼容验证++日期:2026-09-07。实现基于本地 `48fdd2b66` 工作区,仅在 Doclib 持久化读取边界恢复兼容。+使用约定见 [Doclib 历史兼容](next/middle-json/doclib-compatibility.md)。++## 实现与边界++- 新增 `read_cached_middle_json()`,在内存中读取当前格式、3.4.5/旧 1.0 包装及旧 MinerU Schema 2.0。+- 页面适配器与 `dc0944984` 中的原文件逐字节一致;新构造器及元数据处理放在 Doclib 读取层。+- 旧 Schema 2.0 只调整外层,测试明确禁止调用共享语义后处理或 LLM。+- 历史生产版本缺失时记为 unknown,运行参数不可靠时不补造产品记录。+- 缓存命中、覆盖、默认读取档位、FTS 重建和压缩使用同一读取入口。+- 普通读取不改写历史 JSON 或数据库记录;兼容且元数据一致的批次可在正常压缩时写成新格式。+- 通用 ParseResult、DocVortex codec、HTTP/ZIP 和 Gradio 没有增加旧格式读取能力。+- 本次没有调整包版本、协议版本、解析算法或兼容有效期,也未新增兼容开关。++## 验证结果++使用本地 Magic-PDF `.venv1`:++| 检查 | 结果 |+| --- | --- |+| 适配器、Doclib、通用读取、HTTP/ZIP、Gradio 专项 | 560 passed |+| MinerU 完整测试 | 1870 passed、3 skipped、3 个既有失败 |+| 本次修改的 Python 文件 Ruff | 通过 |+| 本次修改的补丁空白检查 | 通过 |++专项覆盖原 3.4.5 文字、公式、链接、图表、列表、目录和坐标转换;旧 2.0 页面保持;+版本缺失、元数据冲突、未知扩展保留、输入不变和转换幂等性;以及显式错误 schema 不兜底。+真实 SQLite/FTS5 用例覆盖两种历史协议族的缓存命中、内容读取、范围覆盖、索引重建、+默认档位、混合协议压缩、重复页取新及冲突/失败时保留源数据。++完整测试仍存在三项此前已确认的失败,均不涉及本次兼容代码:++- `test_basic_extra_includes_preflight_runtime_dependencies`:既有 basic extra 缺失。+- `test_standard_is_the_highest_model_runtime_extra`:既有 test extra 不含 mineru[standard]。+- `test_pp_formulanet_fix_latex_uses_shared_mathring_repair`:既有 UniMERNetDecode 缺少 fix_latex。++执行期间工作区还存在另外进行的 PDF 渲染、依赖、版本和测试调整,本次未修改这些文件。+初轮完整测试额外出现 PDF 补图用例失败;后续并行更新后,最终完整测试已不再出现该失败。+以上完整测试数字对应最终运行时的整个共享工作区,不将其他修改计入本次兼容实现。++## 复跑与证据++```bash+python -m pytest -q tests/unittest/test_doclib_middle_json.py tests/unittest/test_doclib_legacy_schema_adapter.py tests/unittest/test_doclib_legacy_page_ranges.py tests/unittest/test_doclib_cache_semantics.py tests/unittest/test_legacy_schema_adapter.py tests/unittest/test_parse_result_contract.py tests/unittest/test_parser_api_contract.py tests/unittest/test_kit_gradio.py+python -m pytest -q+```++本地日志保存在 `/tmp/doclib-middle-compat/verified-focused.log` 和+`/tmp/doclib-middle-compat/full-final.log`。先前 JSON 2.0 直接迁移的验证报告不改写;+此记录描述后续收敛为 Doclib 专属兼容边界的行为。diff --git a/docs/docvortex-upgrade-validation.md b/docs/docvortex-upgrade-validation.mdindex 23954dd1..97237e41 100644--- a/docs/docvortex-upgrade-validation.md+++ b/docs/docvortex-upgrade-validation.md@@ -1,6 +1,6 @@# DocVortex 接入与标点规则验证-独立引擎已更名为 DocVortex,MinerU 依赖为 `docvortex>=0.1.0,<1.0.0`。+独立引擎已更名为 DocVortex,MinerU 依赖为 `docvortex>=0.2.1,<0.3.0`。导入及集成模块同步迁移,公开文档类型继续由引擎唯一维护。MinerU 的 schema 2.0、产品元数据、已有旧结果适配、OCR/VLM/Hybrid 路由与 UI 名称保持不变。diff --git a/docs/en/usage/cli_tools.md b/docs/en/usage/cli_tools.mdindex 7edf6467..8183f271 100644--- a/docs/en/usage/cli_tools.md+++ b/docs/en/usage/cli_tools.md@@ -69,7 +69,7 @@ mineru-kit vlm-server --engine auto --port 30000- `MINERU_LOCAL_API_STARTUP_TIMEOUT_SECONDS`: startup timeout for the Gradio-managed local V1 server; default `300` seconds.- `MINERU_API_ENABLE_FASTAPI_DOCS`: enable `/docs`, `/openapi.json`, and `/redoc` on the V1 API server; default `true`.- `MINERU_PDF_RENDER_TIMEOUT` / `MINERU_PDF_RENDER_THREADS`: PDF rendering timeout and worker count.-- `MINERU_PROCESSING_WINDOW_SIZE`: processing window size used for large documents.+- `MINERU_PROCESSING_WINDOW_SIZE`: processing window size used for large documents. Flash native text parsing renders only pages with visual blocks, preserving this window boundary, rendering timeout, and worker settings. A 32 MiB pixel budget can split batches further. OCR and other tiers still render entire windows.- `MINERU_INTRA_OP_NUM_THREADS` / `MINERU_INTER_OP_NUM_THREADS`: ONNX operator thread settings.Prefer each command's `--help` output and [model source documentation](./model_source.md) for current defaults.diff --git a/docs/next/code-map.md b/docs/next/code-map.mdindex e063046f..53bc44c2 100644--- a/docs/next/code-map.md+++ b/docs/next/code-map.md@@ -26,7 +26,7 @@| `mineru/doclib/` | 本地文档库常驻服务:入库、SHA256 去重、解析调度、缓存、搜索、配置。FastAPI + SQLite(WAL) + FTS5,路由→`services/`→`core/` 三层 | `doclib/server.py`(2439 行);`doclib/services/{parse,search,scan,config,cleanup}_svc.py`;`doclib/core/{db,fts,file_io}.py` || `mineru/doclib/background/` | 后台 asyncio 任务:ingest(入库)、parse_worker(解析执行)、scan_worker / watch(目录监听)、parse_server_health(parse-server 托管与健康探测)、device_monitor、compaction(批次合并)、telemetry_flush | 各文件顶部 docstring;`background/parse_worker.py:15` || `mineru/parser/` | parse-server 与解析门面:`api_server.py`(2495 行,独立进程)、`api_client.py`(HTTP 客户端)、`tier.py`(tier/backend 解析)、`mineru_parser.py`(`MinerUParser` 统一解析入口)、`base.py`(Middle JSON schema) | `parser/mineru_parser.py`、`parser/tier.py` |-| `mineru/backend/` | 解析引擎:`analyze.py`(`doc_analyze` 统一门面)→ `analysis/`(按格式分派:`pdf/`、`csv.py`、`epub.py`、`html.py`、`ofd.py`、`office.py`)→ `postprocess/`(Middle JSON 组装、段落/标题/列表/表格合并、`llm_aided.py`) | `backend/analyze.py`、`backend/analysis/pdf/pipeline.py` |+| `mineru/backend/` | 解析引擎:`analyze.py`(`doc_analyze` 统一门面)→ `analysis/`(按格式分派:`pdf/`、`csv.py`、`epub.py`、`html.py`、`ofd.py`、`office.py`)→ `postprocess/`(Middle JSON 组装、段落/标题/列表/表格合并、`llm_aided.py`、仅用于 Doclib 的 `legacy_schema_adapter.py`) | `backend/analyze.py`、`backend/analysis/pdf/pipeline.py` || `mineru/model/` | 模型层:`flash/`(纯 CPU 本地结构化解析器族:pdf / office / epub / html / ofd / csv);`runtime/`(`device.py` 设备与 light/full 栈、`hybrid.py` 原子模型分派、`onnx.py`);`layout/ mfr/ ocr/ table/ vlm/`(ML 模型实现);`_internal/pytorchocr` | `model/runtime/device.py:48`、`model/runtime/hybrid.py` || `mineru/render/` | 输出渲染:markdown / content_list(v2) / structured_content / docx / epub / html / pdf | `render/api.py`(唯一渲染入口)、`render/contracts.py` || `mineru/kit/` | 服务端与 Web 工具 CLI:`parse` / `gradio` / `api-server` / `vlm-server` / `router` / `models`,面向无状态批处理、自部署和多卡部署 | `kit/main.py`、`kit/commands/*.py`、`kit/gradio/` |diff --git a/docs/next/invariants.md b/docs/next/invariants.mdindex 268e0abc..b46d6c87 100644--- a/docs/next/invariants.md+++ b/docs/next/invariants.md@@ -32,7 +32,7 @@ Middle JSON 的对外 schema 固定为 `2.0`(`mineru/parser/base.py:20` `MIDDL红线:-- 不接受旧 `_backend` / `_version_name` / `pdf_info` 等历史文档字段;加载统一验证 schema 身份、版本和 metadata,不执行旧协议迁移。+- 不接受旧 `_backend` / `_version_name` / `pdf_info` 等历史文档字段;通用加载统一验证 schema 身份、版本和 metadata。仅 Doclib 的持久化读取边界可转换受支持的历史 MinerU Middle JSON。- 改动输出结构即破坏性变更:必须走 ADR 流程,并同步 `docs/next/api/changes.md`、`docs/next/middle-json.md` 与 compaction(`doclib/background/compaction.py:27`)。## 4. 依赖方向:基础层不得反向引用上层diff --git a/docs/next/middle-json.md b/docs/next/middle-json.mdindex 817a0b84..529b9f8e 100644--- a/docs/next/middle-json.md+++ b/docs/next/middle-json.md@@ -27,7 +27,7 @@ Middle JSON 已经收敛为 schema 2.0 的严格 Pydantic 模型:`docvortex.sc- 以当前代码中的 Pydantic 严格模型为起点(schema 2.0)。- 区分“已经解决”、“部分解决”、“仍需工作”。- Agent 引用与稳定 page/block locator 是 P0 目标。-- 统一使用带 `schema` 身份的 2.0 协议及 `metadata`;所有历史文档协议均明确拒绝,旧缓存须重新解析。+- 统一使用带 `schema` 身份的 2.0 协议及 `metadata`;通用接口拒绝历史协议;仅 Doclib 持久化读取兼容旧 MinerU Middle JSON,见 [缓存兼容说明](middle-json/doclib-compatibility.md)。- render 统一不能只看入口 facade,还要收敛格式-specific 分支。## 与其他文档的关系diff --git a/docs/next/middle-json/README.md b/docs/next/middle-json/README.mdindex 5a77ccba..fa0b02ad 100644--- a/docs/next/middle-json/README.md+++ b/docs/next/middle-json/README.md@@ -34,7 +34,7 @@ InlineContentBlockMiddle JSON 下一版要达到以下目标:-1. 统一使用带协议身份和 metadata 的 JSON 2.0;所有历史 JSON 必须重新解析,见 [envelope.md](envelope.md)。+1. 统一使用带协议身份和 metadata 的 JSON 2.0;通用接口严格读取,Doclib 历史数据经独立缓存边界转换,见 [envelope.md](envelope.md)。2. 明确 `PageInfo` / `Block` / `InlineSpan` 字段契约。3. 为每个 backend 提供 normalization 任务清单。4. 定义 Agent 可引用的稳定 locator 规则。diff --git a/docs/next/middle-json/current-medium.md b/docs/next/middle-json/current-medium.mdindex fb049e3a..e012e9ad 100644--- a/docs/next/middle-json/current-medium.md+++ b/docs/next/middle-json/current-medium.md@@ -20,7 +20,7 @@`schema_version="2.0"`。读取必须匹配 `schema` 与 `schema_version`,详见[统一外层协议](envelope.md)。-所有历史文档协议和缺少身份的 JSON 均拒绝,不提供自动迁移。+通用接口拒绝历史文档协议和缺少身份的 JSON;Doclib 通过独立缓存读取器兼容旧 MinerU 数据。## ModelJson@@ -150,5 +150,5 @@ Markdown、HTML、DOCX 与 Structured Content renderer 按 Span discriminator## 缓存与协议-只接受当前完整文档协议;旧结果不命中缓存、不参与压缩,并提示从源文件重建。+通用接口只接受当前完整文档协议。Doclib 兼容转换 3.4.5 和旧 MinerU 2.0 缓存,转换成功可命中和参与压缩;未知或损坏结果仍要求重新解析。压缩不得合并不同来源、扩展或整本标识的数据。Block/Span 类型和内部语义保持不变。diff --git a/docs/next/middle-json/doclib-compatibility.md b/docs/next/middle-json/doclib-compatibility.mdnew file mode 100644index 00000000..59f1a65b--- /dev/null+++ b/docs/next/middle-json/doclib-compatibility.md@@ -0,0 +1,49 @@+# Doclib 历史 Middle JSON 兼容++兼容仅用于 Doclib 已持久化的 Middle JSON。新写出的文档仍采用+`docvortex.middle` 及当前共享协议,不恢复旧字段或旧格式导出。+通用 ParseResult、DocVortex codec、HTTP/ZIP 与 Gradio 继续严格读取新协议。++## 唯一读取边界++`mineru.doclib.core.middle_json.read_cached_middle_json(payload)` 返回当前+`MiddleJson`。缓存读取、缓存命中、覆盖范围、默认读取档位、FTS 重建与压缩+统一使用此入口。转换只发生在内存,不改写原文件、数据库记录或历史文件名。++读取优先级如下:++1. 只要存在 `schema` 键,就使用当前严格读取器;校验失败不尝试旧分支。+2. 无 `schema`、无有效版本标识的 `pdf_info` 进入 3.4.5 分支。+3. 无 `schema` 的 `schema_version="1.0"` pages 包装,仅在页面包含旧字段时进入同一分支。+4. 无 `schema/metadata` 的旧 `schema_version="2.0"`,只转换文档外层。+5. 混合页面结构、未知身份或版本、字段冲突和损坏数据明确失败。++3.4.5 的发行标签内部实际报告 3.4.4,不能用版本字符串是否等于 3.4.5 识别格式。+对应页面适配器恢复自 `dc0944984`,保持原有文本、公式、链接、图表、列表、目录和坐标转换算法。+转换后的 raw ModelJson 只经过共享确定性后处理,不重新读取源文件或调用 OCR/VLM/LLM。++旧 Schema 2.0 已有当前页面树,直接保留 pages 与必需的 is_full_document,+不重新执行段落、表格或标题后处理。未知根字段不能静默丢弃。++## 来源和产品记录++- 保留历史版本;缺失或不可辨认时记录 `mineru / unknown`,不填当前运行版本。+- 缺失 file_suffix 使用历史 PDF 默认值;显式不支持的格式报错。+- 仅当档位和模式都可可靠识别时写入 extensions.mineru;旧 _ocr_enable 布尔值可恢复 txt/ocr。+- 缺失或未知的解析记录不以 basic/txt 代填,也不借数据库存储档位推断实际运行档位。+- 应用扩展原样保留;已存在的产品扩展必须合法并与可识别的历史记录一致。++## 缓存与压缩++成功转换且覆盖记录页范围的旧批次可命中缓存,避免重复解析。读取旧半角波浪号范围的+记录时仍按原文件名定位,公开返回的页范围继续使用当前语法。++压缩先转换各批次,再比较 metadata、extensions 和 is_full_document。+一致的不同协议批次可以合并,重复页保留最新结果;新文件只输出当前协议。+冲突、损坏或无法识别时跳过压缩并保留源数据。普通读取不会触发回写。++## 后续移除++两个历史分支及专属测试集中维护,尚未确定移除版本,不提供配置开关或自动过期时间。+未来统一移除时,删除 Doclib 读取器中的两个旧分支和专用适配器即可,+无需改变通用解析 API。已有验证报告保持历史事实,不追溯修改其测试结论。diff --git a/docs/next/middle-json/envelope.md b/docs/next/middle-json/envelope.mdindex 4e771aa2..d6ef8e81 100644--- a/docs/next/middle-json/envelope.md+++ b/docs/next/middle-json/envelope.md@@ -46,13 +46,12 @@ Structured Content 迁移 metadata/extensions,但不带 Model/Middle 协议标Content List V1/V2 内容契约不变。HTTP 请求、输出文件名和 ZIP 布局保持不变;ZIP 内文档必须使用完整新版协议。-## 直接迁移与缓存+## 严格协议与 Doclib 历史缓存-读取联合检查 schema 和 schema_version;不接受旧 DocVortex 1.0、缺少 schema 的-MinerU 2.0、旧 1.0 pages 或 pdf_info。不猜测、不自动转换,也不提供旧格式写出。-需要从源文件重新解析。Bundle 使用 docvortex.bundle 2.0 和原有素材摘要清单。+通用 ParseResult、DocVortex codec、HTTP/ZIP 和 Gradio 仍联合校验 schema 与版本,+不接受历史协议,不提供旧格式写出。Bundle 继续使用 docvortex.bundle 2.0。-Doclib 仅将协议有效且覆盖记录页范围的批次视为可用缓存。旧缓存不计入缓存命中、-可用页范围和默认读取档位。请求解析时重新生成缺失结果;直接读取旧页返回重新解析提示。-压缩仅合并 metadata、extensions、is_full_document 一致的新批次,重复页仍取最新值;-遇到旧格式、损坏数据或冲突时跳过压缩并保留源数据。+Doclib 的持久化读取边界单独兼容 3.4.5(含旧 1.0 pages 包装)和旧 MinerU Schema 2.0,+在内存中转换为当前 MiddleJson。兼容成功的批次可用于读取、缓存命中、覆盖范围和 FTS。+压缩先转换并比较来源、扩展及整本标识,一致才合并并写出新协议;损坏或冲突时保留源数据。+详见 [Doclib 历史兼容](doclib-compatibility.md)。普通读取不修改历史文件和数据库记录。diff --git a/docs/next/middle-json/migration.md b/docs/next/middle-json/migration.mdindex 50acb075..f894e3a3 100644--- a/docs/next/middle-json/migration.md+++ b/docs/next/middle-json/migration.md@@ -1,13 +1,14 @@# JSON 2.0 直接迁移-采用[统一外层协议](envelope.md),不保留旧格式转换入口。+采用[统一外层协议](envelope.md),通用解析接口保持严格;仅 Doclib 持久化读取提供[历史转换](doclib-compatibility.md)。-1. 升级 DocVortex 至 0.2.x,MinerU 要求 `docvortex>=0.2.0,<0.3.0`。+1. 升级 DocVortex 至 0.2.x,MinerU 要求 `docvortex>=0.2.1,<0.3.0`。2. 将 Model/Middle 构造器的 file_suffix、producer 放入 DocumentMetadata;Python 访问同步改为 metadata.file_suffix、metadata.producer。3. JSON 使用完整 schema/schema_version/metadata/extensions;两边统一 from_dict/from_json。4. 产品记录只输出实际 tier 与 txt/ocr;删除旧 effort、顶层 mineru_version 和协议适配调用。-5. 历史 JSON、结果包及旧缓存需从源文件重建;加载不会改写旧文件。+5. Doclib 中可识别的历史 Middle JSON 在读取时自动转换,无需重新解析或改写旧文件。+ DocVortex 旧原生协议、旧结果包和通用接口收到的历史 JSON 仍需重新生成。6. 同步升级自建 HTTP 服务与客户端。旧远端协议会明确拒绝,不尝试迁移。验证协议 Schema、跨入口往返、真实生产者与扩展保留、旧协议拒绝、ZIP 和素材,diff --git a/docs/next/pitfalls.md b/docs/next/pitfalls.mdindex eca6ad68..3cb09e7c 100644--- a/docs/next/pitfalls.md+++ b/docs/next/pitfalls.md@@ -19,8 +19,8 @@### 1.2 共享文档协议与缓存必须同步改外层结构时同步 DocVortex schema/readers、MinerU ParseResult 和 Doclib 缓存。-当前使用 schema 身份 + 版本 2.0 + metadata,旧协议全部拒绝;不能仅按版本号识别。-缓存命中、覆盖与默认读取档位均需检查新版协议;压缩只合并元数据一致的新批次。+当前使用 schema 身份 + 版本 2.0 + metadata,通用接口拒绝旧协议;不能仅按版本号识别。+Doclib 缓存命中、覆盖、默认档位与压缩均使用专属 read_cached_middle_json,兼容转换旧 3.4.5 和 MinerU 2.0;压缩只合并转换后元数据一致的批次。详见 [envelope.md](middle-json/envelope.md)。## 2. Alpha 高频迭代,公开 API 未稳定diff --git a/docs/zh/usage/cli_tools.md b/docs/zh/usage/cli_tools.mdindex 9e141259..2edb68a7 100644--- a/docs/zh/usage/cli_tools.md+++ b/docs/zh/usage/cli_tools.md@@ -69,7 +69,7 @@ mineru-kit vlm-server --engine auto --port 30000- `MINERU_LOCAL_API_STARTUP_TIMEOUT_SECONDS`:Gradio 托管本地 V1 服务的启动超时,默认 `300` 秒。- `MINERU_API_ENABLE_FASTAPI_DOCS`:是否为 V1 API Server 启用 `/docs`、`/openapi.json` 和 `/redoc`,默认 `true`。- `MINERU_PDF_RENDER_TIMEOUT` / `MINERU_PDF_RENDER_THREADS`:PDF 渲染超时和 worker 数量。-- `MINERU_PROCESSING_WINDOW_SIZE`:大文档处理窗口大小。+- `MINERU_PROCESSING_WINDOW_SIZE`:大文档处理窗口大小。Flash 原生文本路径仅渲染有视觉块的页面,仍遵守该窗口边界、渲染超时和 worker 配置,并按 32 MiB 像素预算进一步分批。OCR 和其他档位保持整窗口渲染。- `MINERU_INTRA_OP_NUM_THREADS` / `MINERU_INTER_OP_NUM_THREADS`:ONNX 算子线程配置。当前默认值以各命令的 `--help` 和[模型源说明](./model_source.md)为准。diff --git a/mineru/backend/analysis/pdf/window.py b/mineru/backend/analysis/pdf/window.pyindex 561cf1b8..5923ac43 100644--- a/mineru/backend/analysis/pdf/window.py+++ b/mineru/backend/analysis/pdf/window.py@@ -13,7 +13,7 @@ from ....model.runtime.hybrid import HybridLocalModelContextfrom docvortex.document.pdf.document import PDFDocumentfrom docvortex.document.pdf.document import PDFPagefrom docvortex.document.pdf.document import PDFPageTextGeometry-from .images import load_images_from_pdf_bytes_range+from .images import get_load_images_threads, get_load_images_timeout, load_images_from_pdf_bytes_rangefrom ..contracts import AnalyzeEffortfrom .constants import (@@ -53,7 +53,10 @@ from .tables import (_split_native_high_table_blocks,)from docvortex.document.pdf.visuals import attach_visual_block_images as _attach_visual_block_images-from docvortex.document.pdf.visuals import supplement_missing_image_block_containers as _supplement_missing_image_block_containers+from docvortex.document.pdf.visuals import attach_visual_block_images_from_pdf+from docvortex.document.pdf.visuals import (+ supplement_missing_image_block_containers as _supplement_missing_image_block_containers,+)from .text.content import (_fill_window_block_content_and_lines,_validate_text_formula_window_inputs,@@ -294,10 +297,18 @@ def process_pdf_windows(page_count = document.page_countmodel_list: list[list[dict[str, Any]]] = []if flash_txt_mode:- # Flash 先对整份 PDF 生成完整 model_list,不依赖页面渲染和处理窗口。+ # Flash 原生结果只按视觉块需求补图,不再进入供推理使用的全页渲染窗口。from docvortex.analyzers.native import PdfModelmodel_list = PdfModel().predict(document)+ attach_visual_block_images_from_pdf(+ document,+ model_list,+ window_size=_configured_window_size(default=64),+ timeout=get_load_images_timeout(),+ threads=get_load_images_threads(),+ )+ return model_listconfigured_window_size = _configured_window_size(default=64)windows = _build_processing_windows(page_count, configured_window_size)@@ -318,159 +329,154 @@ def process_pdf_windows(_log_processing_window(window, page_count, len(images_pil_list))page_text_geometries: list[PDFPageTextGeometry | None] | None = ([None] * len(window_pages)- if not flash_txt_mode and parse_mode == "txt" and effort in {"medium", "high", "xhigh"}+ if parse_mode == "txt" and effort in {"medium", "high", "xhigh"}else None)- if flash_txt_mode:- # Flash 仅切割当前窗口的外层列表,用于页图释放前原地补充视觉块裁图。- window_model_list = model_list[window.start : window.end + 1]- else:- local_model_context = hybrid_model- if local_model_context is None:- raise ValueError("Hybrid local model context is required outside Flash TXT mode")- np_images = [np.asarray(pil_image).copy() for pil_image in images_pil_list]- images_layout_res = local_model_context.layout_model.batch_predict(- images_pil_list, batch_size=min(8, BATCH_RATIO * LAYOUT_BASE_BATCH_SIZE)- )+ local_model_context = hybrid_model+ if local_model_context is None:+ raise ValueError("Hybrid local model context is required outside Flash TXT mode")+ np_images = [np.asarray(pil_image).copy() for pil_image in images_pil_list]+ images_layout_res = local_model_context.layout_model.batch_predict(+ images_pil_list, batch_size=min(8, BATCH_RATIO * LAYOUT_BASE_BATCH_SIZE)+ )- # 使用小模型layout时对layout的表格做旋转检测- if effort in ["flash", "medium", "high"]:- table_items = _collect_table_items(images_layout_res, np_images)- if table_items:- _apply_table_orientations(- table_items,- parse_mode,- window_pages,- images_list,- local_model_context,- )-- vl_style_layout_blocks = _build_vl_style_layout_blocks(images_layout_res, images_pil_list)-- if parse_mode == "txt" and effort in {"medium", "high"}:- native_table_summary = _apply_native_txt_table_priority(- vl_style_layout_blocks,- images_layout_res,+ # 使用小模型layout时对layout的表格做旋转检测+ if effort in ["flash", "medium", "high"]:+ table_items = _collect_table_items(images_layout_res, np_images)+ if table_items:+ _apply_table_orientations(+ table_items,+ parse_mode,window_pages,images_list,- effort=effort,- page_text_geometries=page_text_geometries,+ local_model_context,)- if native_table_summary.total:- native_table_stats = {- "effort": effort,- "total": native_table_summary.total,- "accepted": native_table_summary.accepted,- "complex_fallbacks": native_table_summary.complex_fallbacks,- "rejected": native_table_summary.rejected,- "errors": native_table_summary.errors,- "removed_internal_text": native_table_summary.removed_internal_text,- "removed_formula_blocks": native_table_summary.removed_formula_blocks,- "removed_formula_layout_items": native_table_summary.removed_formula_layout_items,- }- logger.bind(native_table_priority=native_table_stats).info(- "Hybrid native table priority. "- f"effort={native_table_stats['effort']}, total={native_table_stats['total']}, "- f"accepted={native_table_stats['accepted']}, "- f"complex_fallbacks={native_table_stats['complex_fallbacks']}, "- f"rejected={native_table_stats['rejected']}, "- f"errors={native_table_stats['errors']}, "- f"removed_internal_text={native_table_stats['removed_internal_text']}, "- f"removed_formula_blocks={native_table_stats['removed_formula_blocks']}, "- f"removed_formula_layout_items={native_table_stats['removed_formula_layout_items']}"- )-- if parse_mode == "txt":- if effort == "medium":- window_model_list = vl_style_layout_blocks- elif effort == "high":- high_vlm_blocks, accepted_native_tables = _split_native_high_table_blocks(vl_style_layout_blocks)- high_vlm_results = vlm_predictor.batch_extract_with_layout(- images=images_pil_list,- blocks_list=high_vlm_blocks,- not_extract_list=NOT_EXTRACT_TYPES,- image_analysis=False,- )- window_model_list = _restore_native_high_table_blocks(- high_vlm_results,- accepted_native_tables,- )- elif effort == "xhigh":- window_model_list = vlm_predictor.batch_two_step_extract(- images=images_pil_list,- not_extract_list=NOT_EXTRACT_TYPES,- image_analysis=image_analysis,- )- else:- raise ValueError(f"Unsupported analyze effort: {effort}")- elif parse_mode == "ocr":- if effort in ["flash", "medium"]:- window_model_list = vl_style_layout_blocks- elif effort == "high":- window_model_list = vlm_predictor.batch_extract_with_layout(- images=images_pil_list,- blocks_list=vl_style_layout_blocks,- image_analysis=False,- )- elif effort == "xhigh":- window_model_list = vlm_predictor.batch_two_step_extract(- images=images_pil_list,- image_analysis=image_analysis,- )- else:- raise ValueError(f"Unsupported analyze effort: {effort}")- else:- raise ValueError(f"Unsupported parse mode: {parse_mode}")-- if effort in {"high", "xhigh"}:- window_model_list = _convert_vlm_results_to_model_list(window_model_list)- if effort == "xhigh":- _normalize_xhigh_vlm_blocks(window_model_list)- _apply_layout_title_split(- window_model_list,- images_layout_res,- [_normalize_page_size(image) for image in images_pil_list],++ vl_style_layout_blocks = _build_vl_style_layout_blocks(images_layout_res, images_pil_list)++ if parse_mode == "txt" and effort in {"medium", "high"}:+ native_table_summary = _apply_native_txt_table_priority(+ vl_style_layout_blocks,+ images_layout_res,+ window_pages,+ images_list,+ effort=effort,+ page_text_geometries=page_text_geometries,+ )+ if native_table_summary.total:+ native_table_stats = {+ "effort": effort,+ "total": native_table_summary.total,+ "accepted": native_table_summary.accepted,+ "complex_fallbacks": native_table_summary.complex_fallbacks,+ "rejected": native_table_summary.rejected,+ "errors": native_table_summary.errors,+ "removed_internal_text": native_table_summary.removed_internal_text,+ "removed_formula_blocks": native_table_summary.removed_formula_blocks,+ "removed_formula_layout_items": native_table_summary.removed_formula_layout_items,+ }+ logger.bind(native_table_priority=native_table_stats).info(+ "Hybrid native table priority. "+ f"effort={native_table_stats['effort']}, total={native_table_stats['total']}, "+ f"accepted={native_table_stats['accepted']}, "+ f"complex_fallbacks={native_table_stats['complex_fallbacks']}, "+ f"rejected={native_table_stats['rejected']}, "+ f"errors={native_table_stats['errors']}, "+ f"removed_internal_text={native_table_stats['removed_internal_text']}, "+ f"removed_formula_blocks={native_table_stats['removed_formula_blocks']}, "+ f"removed_formula_layout_items={native_table_stats['removed_formula_layout_items']}")- if effort == "flash":- window_model_list = _process_flash_ocr(- images_list,- window_pages,- window_model_list,- local_model_context,- images_layout_res,+ if parse_mode == "txt":+ if effort == "medium":+ window_model_list = vl_style_layout_blocks+ elif effort == "high":+ high_vlm_blocks, accepted_native_tables = _split_native_high_table_blocks(vl_style_layout_blocks)+ high_vlm_results = vlm_predictor.batch_extract_with_layout(+ images=images_pil_list,+ blocks_list=high_vlm_blocks,+ not_extract_list=NOT_EXTRACT_TYPES,+ image_analysis=False,+ )+ window_model_list = _restore_native_high_table_blocks(+ high_vlm_results,+ accepted_native_tables,+ )+ elif effort == "xhigh":+ window_model_list = vlm_predictor.batch_two_step_extract(+ images=images_pil_list,+ not_extract_list=NOT_EXTRACT_TYPES,+ image_analysis=image_analysis,)- # Flash OCR 在表内对象清理后复用统一公式编号合并,确保视觉裁图包含编号区域。- for page_model_list in window_model_list:- page_model_list[:] = optimize_hybrid_formula_number_blocks(page_model_list)else:- window_model_list = _process_text_and_formulas(- images_list,- window_pages,- window_model_list,- parse_mode,- effort,- local_model_context,- images_layout_res,- page_text_geometries,+ raise ValueError(f"Unsupported analyze effort: {effort}")+ elif parse_mode == "ocr":+ if effort in ["flash", "medium"]:+ window_model_list = vl_style_layout_blocks+ elif effort == "high":+ window_model_list = vlm_predictor.batch_extract_with_layout(+ images=images_pil_list,+ blocks_list=vl_style_layout_blocks,+ image_analysis=False,)-- if effort in {"medium", "high"}:- _apply_seal_ocr(local_model_context, window_model_list, np_images)elif effort == "xhigh":- _supplement_missing_image_block_containers(- window_model_list,- vl_style_layout_blocks,+ window_model_list = vlm_predictor.batch_two_step_extract(+ images=images_pil_list,+ image_analysis=image_analysis,)+ else:+ raise ValueError(f"Unsupported analyze effort: {effort}")+ else:+ raise ValueError(f"Unsupported parse mode: {parse_mode}")++ if effort in {"high", "xhigh"}:+ window_model_list = _convert_vlm_results_to_model_list(window_model_list)+ if effort == "xhigh":+ _normalize_xhigh_vlm_blocks(window_model_list)+ _apply_layout_title_split(+ window_model_list,+ images_layout_res,+ [_normalize_page_size(image) for image in images_pil_list],+ )++ if effort == "flash":+ window_model_list = _process_flash_ocr(+ images_list,+ window_pages,+ window_model_list,+ local_model_context,+ images_layout_res,+ )+ # Flash OCR 在表内对象清理后复用统一公式编号合并,确保视觉裁图包含编号区域。+ for page_model_list in window_model_list:+ page_model_list[:] = optimize_hybrid_formula_number_blocks(page_model_list)+ else:+ window_model_list = _process_text_and_formulas(+ images_list,+ window_pages,+ window_model_list,+ parse_mode,+ effort,+ local_model_context,+ images_layout_res,+ page_text_geometries,+ )++ if effort in {"medium", "high"}:+ _apply_seal_ocr(local_model_context, window_model_list, np_images)+ elif effort == "xhigh":+ _supplement_missing_image_block_containers(+ window_model_list,+ vl_style_layout_blocks,+ )_attach_visual_block_images(window_model_list,images_list,page_start_index=window.start,)- if not flash_txt_mode:- model_list.extend(window_model_list)+ model_list.extend(window_model_list)finally:_close_images(images_list)diff --git a/mineru/backend/postprocess/legacy_schema_adapter.py b/mineru/backend/postprocess/legacy_schema_adapter.pynew file mode 100644index 00000000..d6150f05--- /dev/null+++ b/mineru/backend/postprocess/legacy_schema_adapter.py@@ -0,0 +1,727 @@+# Copyright (c) Opendatalab. All rights reserved.+"""MinerU 3.4.5 Middle JSON page 到当前 raw ModelJson 的适配器。++3.4.5 的页面仍使用 ``preproc_blocks``/``para_blocks``、``lines`` 和+``spans``,自然语言行内语义则通过旧字符串标签或旧 span 字段表达。本模块只负责+把单页回推成当前 ``model_json_to_pages`` 能消费的 raw block 列表;文档 envelope+及当前 MiddleJson 的元数据由调用方负责。+"""++from __future__ import annotations++import html+import math+import re+from typing import Any++from docvortex.schema import INLINE_STYLE_ORDER, InlineSpan, parse_inline_spans+from docvortex.foundation.hyperlink import OFFICE_EXTERNAL_HYPERLINK_SCHEMES, sanitize_hyperlink_target+from docvortex.content.inline import join_inline_spans++# 这些父块在 3.4.5 中通过 ``blocks`` 保存 body/caption/footnote;当前 raw+# ModelJson 则要求先展平成同级 block,再由统一后处理重新分组。+_VISUAL_PARENT_TYPES: frozenset[str] = frozenset(+ {+ "image",+ "table",+ "chart",+ "code",+ }+)++_INLINE_CONTENT_BLOCK_TYPES: frozenset[str] = frozenset(+ {+ "text",+ "ref_text",+ "doc_title",+ "paragraph_title",+ "header",+ "footer",+ "page_number",+ "aside_text",+ "page_footnote",+ "image_caption",+ "image_footnote",+ "table_caption",+ "table_footnote",+ "chart_caption",+ "chart_footnote",+ "code_caption",+ "code_footnote",+ "caption",+ "footnote",+ "phonetic",+ }+)++_STRING_CONTENT_BLOCK_TYPES: frozenset[str] = frozenset(+ {+ "equation",+ "image_body",+ "table_body",+ "chart_body",+ "code_body",+ }+)++_BLOCK_TYPE_ALIASES: dict[str, str] = {+ "abstract": "text",+ "phonetic": "text",+ "interline_equation": "equation",+}++_SUPPORTED_BLOCK_TYPES: frozenset[str] = frozenset(+ _VISUAL_PARENT_TYPES | _INLINE_CONTENT_BLOCK_TYPES | _STRING_CONTENT_BLOCK_TYPES | {"list", "index"}+)++_INLINE_START_RE = re.compile(+ r"<(?P<tag>eq|code|text|hyperlink|sup|sub|strong|b|em|i|s|u)(?P<attrs>\s[^<>]*?)?>",+ re.IGNORECASE,+)+_STYLE_ATTR_RE = re.compile(r"\bstyle\s*=\s*([\"'])(?P<style>.*?)\1", re.IGNORECASE | re.DOTALL)+_URL_RE = re.compile(r"<url>(?P<url>.*?)</url>", re.IGNORECASE | re.DOTALL)+_DIRECT_TAG_STYLES: dict[str, str] = {+ "strong": "bold",+ "b": "bold",+ "em": "italic",+ "i": "italic",+ "s": "strikethrough",+ "u": "underline",+ "sup": "superscript",+ "sub": "subscript",+}++# 3.4.5 PDF 的异常零面积框需要替换后才能通过当前严格 bbox 校验。+_PLACEHOLDER_BBOX: tuple[float, float, float, float] = (0.0, 0.0, 0.001, 0.001)+++def legacy_page_to_model_list(page: dict[str, Any]) -> list[dict[str, Any]]:+ """把 MinerU 3.4.5 的单页 Middle JSON 转换为当前 raw model-list。"""+ if not isinstance(page, dict):+ raise ValueError("legacy Middle JSON page must be a dict")++ width, height = _parse_page_size(page.get("page_size"))+ source_blocks = page.get("preproc_blocks")+ if source_blocks is None:+ source_blocks = page.get("para_blocks", [])+ if not isinstance(source_blocks, list):+ raise ValueError("legacy Middle JSON page blocks must be a list")++ discarded_blocks = page.get("discarded_blocks", [])+ if not isinstance(discarded_blocks, list):+ raise ValueError("legacy Middle JSON discarded_blocks must be a list")++ legacy_blocks = [*source_blocks, *discarded_blocks]+ model_list: list[dict[str, Any]] = []+ for group_index, block in enumerate(legacy_blocks):+ if not isinstance(block, dict):+ raise ValueError("legacy Middle JSON block entries must be dicts")+ group_start = len(model_list)+ _collect_blocks(block, width, height, model_list, inherited=None)+ if width <= 0 or height <= 0:+ _assign_synthetic_group_bbox(model_list[group_start:], group_index, len(legacy_blocks))+ return model_list+++def _parse_page_size(value: Any) -> tuple[float, float]:+ """读取合法 page_size;缺失、非数值或非正值统一返回零值。"""+ if not isinstance(value, (list, tuple)) or len(value) < 2:+ return 0.0, 0.0+ width, height = value[:2]+ if any(isinstance(item, bool) or not isinstance(item, (int, float)) for item in (width, height)):+ return 0.0, 0.0+ parsed_width, parsed_height = float(width), float(height)+ if not all(math.isfinite(item) and item > 0 for item in (parsed_width, parsed_height)):+ return 0.0, 0.0+ return parsed_width, parsed_height+++def _assign_synthetic_group_bbox(blocks: list[dict[str, Any]], group_index: int, group_count: int) -> None:+ """为无 page_size 文档分配互不重叠的占位框,并保持原顶层块的子项同组。"""+ safe_count = max(group_count, 1)+ y0 = (group_index + 0.2) / safe_count+ y1 = (group_index + 0.8) / safe_count+ bbox = (0.05, y0, 0.95, y1)+ for block in blocks:+ block["bbox"] = bbox+ lines = block.get("lines")+ if isinstance(lines, list):+ for line in lines:+ if isinstance(line, dict):+ line["bbox"] = list(bbox)+++def _collect_blocks(+ block: dict[str, Any],+ width: float,+ height: float,+ output: list[dict[str, Any]],+ *,+ inherited: dict[str, Any] | None,+) -> None:+ """递归展平视觉父块,并把普通叶子转换为当前 raw block。"""+ block_type = _normalize_block_type(block)+ nested = block.get("blocks")+ if block_type in _VISUAL_PARENT_TYPES and isinstance(nested, list) and nested:+ child_context = _visual_child_context(block, inherited)+ for child in nested:+ if not isinstance(child, dict):+ raise ValueError("legacy visual block children must be dicts")+ _collect_blocks(child, width, height, output, inherited=child_context)+ return++ if block_type == "list" and isinstance(nested, list) and nested:+ output.append(_convert_leaf_block(block, block_type, width, height, inherited))+ output.extend(_flatten_nested_list_items(nested, width, height))+ return+ if block_type == "index" and isinstance(nested, list) and nested:+ output.append(_flatten_nested_index(block, nested, width, height))+ return+ if block_type == "list" and isinstance(block.get("lines"), list) and block["lines"]:+ output.append(_convert_leaf_block(block, block_type, width, height, inherited))+ output.extend(_convert_pdf_list_items(block, width, height))+ return++ raw = _convert_leaf_block(block, block_type, width, height, inherited)+ output.append(raw)+++def _visual_child_context(+ block: dict[str, Any],+ inherited: dict[str, Any] | None,+) -> dict[str, Any]:+ """收集视觉父块上需要下沉到 body 的元数据。"""+ context = dict(inherited or {})+ context["bbox"] = block.get("bbox", context.get("bbox"))+ sub_type = block.get("sub_type")+ if not isinstance(sub_type, str) or not sub_type.strip():+ sub_type = None+ if sub_type:+ context["sub_type"] = sub_type+ if isinstance(block.get("guess_lang"), str) and block["guess_lang"].strip():+ context["guess_lang"] = block["guess_lang"].strip()+ if "cell_merge" in block:+ context["cell_merge"] = block.get("cell_merge")+ return context+++def _flatten_nested_list_items(nested: list[Any], width: float, height: float) -> list[dict[str, Any]]:+ """把 3.4.5 Office 嵌套列表叶子展平为当前 PDF list 可关联的文本块。"""+ items: list[dict[str, Any]] = []+ for child in nested:+ if not isinstance(child, dict):+ raise ValueError("legacy list children must be dicts")+ child_type = _normalize_block_type(child)+ child_nested = child.get("blocks")+ if child_type == "list" and isinstance(child_nested, list):+ items.extend(_flatten_nested_list_items(child_nested, width, height))+ continue+ if child_type not in {"text", "ref_text"}:+ child_type = "text"+ items.append(_convert_leaf_block(child, child_type, width, height, inherited=None))+ return items+++def _flatten_nested_index(+ block: dict[str, Any],+ nested: list[Any],+ width: float,+ height: float,+) -> dict[str, Any]:+ """把 3.4.5 Office 嵌套目录折叠为当前 PDF index 使用的换行 Span。"""+ contents: list[list[dict[str, Any]]] = []++ def collect(children: list[Any]) -> None:+ """按深度优先顺序收集目录叶子的结构化行内内容。"""+ for child in children:+ if not isinstance(child, dict):+ raise ValueError("legacy index children must be dicts")+ child_nested = child.get("blocks")+ if _normalize_block_type(child) == "index" and isinstance(child_nested, list):+ collect(child_nested)+ continue+ converted = _convert_leaf_block(child, "text", width, height, inherited=None)+ content = converted.get("content")+ if isinstance(content, list) and content:+ contents.append([span for span in content if isinstance(span, dict)])++ collect(nested)+ joined: list[dict[str, Any]] = []+ for content_index, content in enumerate(contents):+ if content_index:+ _append_text_span(joined, "\n", ())+ _extend_span_dicts(joined, content)+ return {+ "type": "index",+ "bbox": _normalize_bbox(block.get("bbox"), width, height),+ "content": [span.model_dump(mode="json") for span in parse_inline_spans(joined)],+ }+++def _convert_pdf_list_items(block: dict[str, Any], width: float, height: float) -> list[dict[str, Any]]:+ """把 3.4.5 PDF list 的物理行恢复为当前 bbox 关联需要的文本项。"""+ raw_lines = [line for line in block.get("lines", []) if isinstance(line, dict)]+ if not raw_lines:+ return []+ has_item_flags = any(line.get("is_list_start_line") or line.get("is_list_end_line") for line in raw_lines)+ line_groups: list[list[dict[str, Any]]] = []+ current_group: list[dict[str, Any]] = []+ for line in raw_lines:+ if current_group and (line.get("is_list_start_line") or not has_item_flags):+ line_groups.append(current_group)+ current_group = []+ current_group.append(line)+ if line.get("is_list_end_line"):+ line_groups.append(current_group)+ current_group = []+ if current_group:+ line_groups.append(current_group)++ item_type = "ref_text" if block.get("sub_type") == "ref_text" else "text"+ items: list[dict[str, Any]] = []+ for lines in line_groups:+ synthetic_block = {+ "type": item_type,+ "bbox": _union_line_bboxes(lines, block.get("bbox")),+ "lines": lines,+ }+ items.append(_convert_leaf_block(synthetic_block, item_type, width, height, inherited=None))+ return items+++def _union_line_bboxes(lines: list[dict[str, Any]], fallback: Any) -> Any:+ """合并一个旧列表项的物理行框,缺失时回退列表父框。"""+ bboxes = [line.get("bbox") for line in lines]+ valid = [+ bbox+ for bbox in bboxes+ if isinstance(bbox, (list, tuple))+ and len(bbox) == 4+ and all(isinstance(item, (int, float)) and not isinstance(item, bool) for item in bbox)+ ]+ if not valid:+ return fallback+ return [+ min(float(bbox[0]) for bbox in valid),+ min(float(bbox[1]) for bbox in valid),+ max(float(bbox[2]) for bbox in valid),+ max(float(bbox[3]) for bbox in valid),+ ]+++def _convert_leaf_block(+ block: dict[str, Any],+ block_type: str,+ width: float,+ height: float,+ inherited: dict[str, Any] | None,+) -> dict[str, Any]:+ """把一个 3.4.5 叶子块转换为当前 raw block 字典。"""+ inherited = inherited or {}+ raw_bbox = block.get("bbox", inherited.get("bbox"))+ bbox = _normalize_bbox(raw_bbox, width, height)+ lines = block.get("lines")+ legacy_lines = lines if isinstance(lines, list) else []+ effective_sub_type = _effective_sub_type(block, block_type, inherited)++ raw: dict[str, Any] = {"type": block_type, "bbox": bbox}+ if _uses_inline_content(block_type, effective_sub_type):+ raw["content"] = _extract_inline_content(block, legacy_lines, preserve_line_breaks=block_type == "index")+ elif block_type in _STRING_CONTENT_BLOCK_TYPES:+ raw["content"] = _extract_string_content(block, legacy_lines, block_type)++ image_fields = _extract_image_fields(block, legacy_lines)+ raw.update(image_fields)++ normalized_lines = _normalize_lines(legacy_lines, width, height)+ if normalized_lines:+ raw["lines"] = normalized_lines++ _copy_leaf_metadata(raw, block, block_type, effective_sub_type, inherited)+ return raw+++def _effective_sub_type(block: dict[str, Any], block_type: str, inherited: dict[str, Any]) -> str | None:+ """解析 code/image/chart body 的有效 sub_type。"""+ sub_type = block.get("sub_type", inherited.get("sub_type"))+ if isinstance(sub_type, str) and sub_type.strip():+ return sub_type.strip()+ if block_type == "code_body":+ return "code"+ return None+++def _uses_inline_content(block_type: str, sub_type: str | None) -> bool:+ """判断当前 raw block 是否必须使用结构化 InlineSpan 列表。"""+ return block_type in _INLINE_CONTENT_BLOCK_TYPES or (block_type == "code_body" and sub_type == "algorithm")+++def _extract_inline_content(+ block: dict[str, Any],+ lines: list[Any],+ *,+ preserve_line_breaks: bool,+) -> list[dict[str, Any]]:+ """从旧 lines/spans 或直接 content 构造严格 InlineSpan 列表。"""+ line_contents: list[list[InlineSpan]] = []+ for line in lines:+ if not isinstance(line, dict):+ continue+ spans = line.get("spans")+ if not isinstance(spans, list):+ continue+ converted: list[dict[str, Any]] = []+ for span in spans:+ if not isinstance(span, dict):+ continue+ _append_legacy_span(converted, span)+ parsed = parse_inline_spans(converted)+ if parsed:+ line_contents.append(parsed)++ if not line_contents:+ direct_content = block.get("content")+ converted = _convert_direct_inline_content(direct_content)+ return [span.model_dump(mode="json") for span in parse_inline_spans(converted)]++ if preserve_line_breaks:+ joined: list[dict[str, Any]] = []+ for line_index, content in enumerate(line_contents):+ if line_index:+ _append_text_span(joined, "\n", ())+ _extend_span_dicts(joined, [span.model_dump(mode="json") for span in content])+ return [span.model_dump(mode="json") for span in parse_inline_spans(joined)]++ joined_spans = join_inline_spans(line_contents)+ return [span.model_dump(mode="json") for span in joined_spans]+++def _convert_direct_inline_content(content: Any) -> list[dict[str, Any]]:+ """转换没有 lines 包装的旧字符串或 span 列表。"""+ if isinstance(content, str):+ return _parse_legacy_markup(content)+ if not isinstance(content, list):+ return []+ output: list[dict[str, Any]] = []+ for item in content:+ if isinstance(item, dict):+ _append_legacy_span(output, item)+ elif isinstance(item, str):+ _extend_span_dicts(output, _parse_legacy_markup(item))+ return output+++def _append_legacy_span(output: list[dict[str, Any]], span: dict[str, Any]) -> None:+ """把一个 3.4.5 span 追加为当前 InlineSpan 字典。"""+ span_type = str(span.get("type", "text") or "text")+ content = span.get("content")+ if span_type in {"inline_equation", "equation_inline", "interline_equation", "equation"}:+ normalized = str(content or "").strip()+ if normalized:+ output.append({"type": "equation_inline", "content": normalized})+ return+ if span_type in {"code", "code_inline"}:+ if isinstance(content, str) and content:+ output.append({"type": "code_inline", "content": content})+ return+ if span_type == "hyperlink":+ _append_legacy_hyperlink(output, span)+ return+ if not isinstance(content, str) or not content:+ return++ styles = _normalize_styles(span.get("styles", span.get("style")))+ if styles:+ _append_text_span(output, content, styles)+ else:+ _extend_span_dicts(output, _parse_legacy_markup(content))+++def _append_legacy_hyperlink(output: list[dict[str, Any]], span: dict[str, Any]) -> None:+ """转换结构化旧 hyperlink;不安全目标降级为可见子内容。"""+ children: list[dict[str, Any]] = []+ raw_children = span.get("children")+ if isinstance(raw_children, list):+ for child in raw_children:+ if isinstance(child, dict):+ _append_legacy_span(children, child)+ elif isinstance(span.get("content"), str):+ styles = _normalize_styles(span.get("styles", span.get("style")))+ if styles:+ _append_text_span(children, span["content"], styles)+ else:+ _extend_span_dicts(children, _parse_legacy_markup(span["content"]))++ safe_children = [child for child in children if child.get("type") != "hyperlink"]+ safe_url = sanitize_hyperlink_target(+ html.unescape(str(span.get("url", ""))),+ allowed_schemes=OFFICE_EXTERNAL_HYPERLINK_SCHEMES,+ allow_relative=True,+ allow_fragment=True,+ )+ if safe_url and safe_children:+ output.append({"type": "hyperlink", "url": safe_url, "content": safe_children})+ else:+ _extend_span_dicts(output, safe_children)+++def _parse_legacy_markup(content: str) -> list[dict[str, Any]]:+ """按 3.4.5 的私有字符串标签协议解析行内内容。"""+ output: list[dict[str, Any]] = []+ cursor = 0+ while match := _INLINE_START_RE.search(content, cursor):+ _append_text_span(output, content[cursor : match.start()], ())+ tag = match.group("tag").lower()+ closing = _find_matching_close(content, tag, match.end())+ if closing is None:+ _append_text_span(output, match.group(0), ())+ cursor = match.end()+ continue++ inner = content[match.end() : closing[0]]+ original = content[match.start() : closing[1]]+ parsed = _parse_legacy_element(tag, match.group("attrs") or "", inner)+ if parsed is None:+ _append_text_span(output, original, ())+ else:+ _extend_span_dicts(output, parsed)+ cursor = closing[1]+ _append_text_span(output, content[cursor:], ())+ return output+++def _find_matching_close(content: str, tag: str, start: int) -> tuple[int, int] | None:+ """查找旧标签的配对结束位置,并兼容同名嵌套。"""+ token_re = re.compile(rf"<(?P<close>/)?{re.escape(tag)}(?:\s[^<>]*?)?>", re.IGNORECASE)+ depth = 1+ for match in token_re.finditer(content, start):+ depth += -1 if match.group("close") else 1+ if depth == 0:+ return match.start(), match.end()+ return None+++def _parse_legacy_element(tag: str, attrs: str, inner: str) -> list[dict[str, Any]] | None:+ """把一个完整旧标签转换为当前 InlineSpan 字典。"""+ if tag == "eq":+ latex = html.unescape(inner).strip()+ return [{"type": "equation_inline", "content": latex}] if latex else []+ if tag == "code":+ code = html.unescape(inner)+ return [{"type": "code_inline", "content": code}] if code else []+ if tag == "hyperlink":+ return _parse_markup_hyperlink(inner)++ children = _parse_legacy_markup(inner)+ styles = _parse_style_attribute(attrs) if tag == "text" else [_DIRECT_TAG_STYLES[tag]]+ return _apply_styles(children, styles)+++def _parse_markup_hyperlink(inner: str) -> list[dict[str, Any]] | None:+ """解析旧 hyperlink 标签;危险 URL 只移除链接语义。"""+ match = _URL_RE.search(inner)+ if match is None:+ return None+ label_content = f"{inner[: match.start()]}{inner[match.end() :]}"+ children = [span for span in _parse_legacy_markup(label_content) if span.get("type") != "hyperlink"]+ if not children:+ return []+ safe_url = sanitize_hyperlink_target(+ html.unescape(match.group("url")),+ allowed_schemes=OFFICE_EXTERNAL_HYPERLINK_SCHEMES,+ allow_relative=True,+ allow_fragment=True,+ )+ if safe_url is None:+ return children+ return [{"type": "hyperlink", "url": safe_url, "content": children}]+++def _parse_style_attribute(attrs: str) -> list[str]:+ """读取旧 ``text style`` 属性并过滤为当前白名单样式。"""+ match = _STYLE_ATTR_RE.search(attrs)+ return _normalize_styles(match.group("style") if match else None)+++def _normalize_styles(value: Any) -> list[str]:+ """按当前固定顺序去重旧样式,并消解同时上下标的非法组合。"""+ if isinstance(value, str):+ candidates = [item.strip().lower() for item in value.split(",")]+ elif isinstance(value, (list, tuple, set)):+ candidates = [str(item).strip().lower() for item in value]+ else:+ candidates = []+ style_set = {item for item in candidates if item in INLINE_STYLE_ORDER}+ if "superscript" in style_set and "subscript" in style_set:+ style_set.remove("subscript")+ return [style for style in INLINE_STYLE_ORDER if style in style_set]+++def _apply_styles(spans: list[dict[str, Any]], styles: list[str]) -> list[dict[str, Any]]:+ """把旧包装标签的样式递归应用到文字及链接文字。"""+ if not styles:+ return spans+ output: list[dict[str, Any]] = []+ for span in spans:+ span_type = span.get("type")+ if span_type == "text":+ merged_styles = _normalize_styles([*_normalize_styles(span.get("styles")), *styles])+ _append_text_span(output, str(span.get("content", "")), merged_styles)+ elif span_type == "hyperlink" and isinstance(span.get("content"), list):+ children = _apply_styles([child for child in span["content"] if isinstance(child, dict)], styles)+ output.append({**span, "content": children})+ else:+ output.append(dict(span))+ return output+++def _append_text_span(output: list[dict[str, Any]], content: str, styles: Any) -> None:+ """追加非空文字,并合并相邻同样式 TextSpan。"""+ if not content:+ return+ normalized_styles = _normalize_styles(styles)+ if output and output[-1].get("type") == "text" and output[-1].get("styles", []) == normalized_styles:+ output[-1]["content"] = f"{output[-1].get('content', '')}{content}"+ return+ span: dict[str, Any] = {"type": "text", "content": content}+ if normalized_styles:+ span["styles"] = normalized_styles+ output.append(span)+++def _extend_span_dicts(output: list[dict[str, Any]], spans: list[dict[str, Any]]) -> None:+ """追加一组 Span,并规范化序列边界上的相邻文字。"""+ for span in spans:+ if span.get("type") == "text" and isinstance(span.get("content"), str):+ _append_text_span(output, span["content"], span.get("styles", []))+ else:+ output.append(dict(span))+++def _extract_string_content(block: dict[str, Any], lines: list[Any], block_type: str) -> str:+ """从旧 spans 提取公式、视觉主体、表格 HTML 或代码字符串。"""+ parts: list[str] = []+ for line in lines:+ if not isinstance(line, dict):+ continue+ line_parts: list[str] = []+ spans = line.get("spans")+ if not isinstance(spans, list):+ continue+ for span in spans:+ if not isinstance(span, dict):+ continue+ value = span.get("html") if block_type == "table_body" and span.get("html") is not None else span.get("content")+ normalized = _stringify_visual_content(value)+ if normalized:+ line_parts.append(normalized)+ if line_parts:+ parts.append("".join(line_parts))++ if not parts:+ direct = block.get("html") if block_type == "table_body" and block.get("html") is not None else block.get("content")+ return _stringify_visual_content(direct)+ return "\n".join(parts) if block_type == "code_body" else "".join(parts)+++def _stringify_visual_content(value: Any) -> str:+ """把旧视觉识别列表稳定折叠为当前字符串载荷。"""+ if isinstance(value, str):+ return value+ if isinstance(value, list):+ return "\n".join(str(item) for item in value if str(item).strip())+ return "" if value is None else str(value)+++def _extract_image_fields(block: dict[str, Any], lines: list[Any]) -> dict[str, str]:+ """从 block 或第一个视觉 span 提升当前支持的图片载荷字段。"""+ fields: dict[str, str] = {}+ for field_name in ("image_path", "image_base64", "image_url"):+ value = block.get(field_name)+ if isinstance(value, str) and value:+ fields[field_name] = value+ if fields:+ return fields+ for line in lines:+ if not isinstance(line, dict):+ continue+ spans = line.get("spans")+ if not isinstance(spans, list):+ continue+ for span in spans:+ if not isinstance(span, dict):+ continue+ for field_name in ("image_path", "image_base64", "image_url"):+ value = span.get(field_name)+ if isinstance(value, str) and value and field_name not in fields:+ fields[field_name] = value+ if fields:+ return fields+ return fields+++def _copy_leaf_metadata(+ raw: dict[str, Any],+ block: dict[str, Any],+ block_type: str,+ sub_type: str | None,+ inherited: dict[str, Any],+) -> None:+ """复制当前后处理仍会读取、且不会越过严格模型边界的字段。"""+ if block.get("level") is not None:+ raw["level"] = block["level"]+ if sub_type:+ raw["sub_type"] = sub_type+ guess_lang = block.get("guess_lang", inherited.get("guess_lang"))+ if block_type == "code_body" and isinstance(guess_lang, str) and guess_lang.strip():+ raw["guess_lang"] = guess_lang.strip()+ if block_type == "table_body" and "cell_merge" in inherited:+ raw["cell_merge"] = inherited["cell_merge"]+ if block_type in {"doc_title", "paragraph_title", "page_footnote"}:+ anchor = block.get("anchor")+ if isinstance(anchor, str) and anchor.strip():+ raw["anchor"] = anchor.strip()+++def _normalize_block_type(block: dict[str, Any]) -> str:+ """把最终 3.4.5 discriminator 映射到当前 raw 类型并拒绝内部标签。"""+ block_type = str(block.get("type", "") or "")+ if block_type == "title":+ return "doc_title" if block.get("level") == 1 else "paragraph_title"+ normalized = _BLOCK_TYPE_ALIASES.get(block_type, block_type)+ if normalized not in _SUPPORTED_BLOCK_TYPES:+ raise ValueError(f"unsupported MinerU 3.4.5 block type {block_type!r}; source reparse required")+ return normalized+++def _normalize_bbox(value: Any, width: float, height: float) -> tuple[float, float, float, float]:+ """把旧 bbox 规范为当前严格的 0-1 非零面积坐标。"""+ if not isinstance(value, (list, tuple)) or len(value) != 4:+ return _PLACEHOLDER_BBOX+ if any(isinstance(item, bool) or not isinstance(item, (int, float)) for item in value):+ return _PLACEHOLDER_BBOX+ coords = tuple(float(item) for item in value)+ if not all(math.isfinite(item) for item in coords):+ return _PLACEHOLDER_BBOX+ if width > 0 and height > 0:+ coords = (coords[0] / width, coords[1] / height, coords[2] / width, coords[3] / height)+ elif not all(0.0 <= item <= 1.0 for item in coords):+ return _PLACEHOLDER_BBOX+ clamped = tuple(min(max(item, 0.0), 1.0) for item in coords)+ if clamped[2] <= clamped[0] or clamped[3] <= clamped[1]:+ return _PLACEHOLDER_BBOX+ return clamped # type: ignore[return-value]+++def _normalize_lines(lines: list[Any], width: float, height: float) -> list[dict[str, list[float]]]:+ """保留跨页文本合并需要的行框,并转换到当前归一化坐标。"""+ result: list[dict[str, list[float]]] = []+ for line in lines:+ if not isinstance(line, dict):+ continue+ bbox = _normalize_bbox(line.get("bbox"), width, height)+ result.append({"bbox": list(bbox)})+ return result+++__all__ = ["legacy_page_to_model_list"]diff --git a/mineru/doclib/background/compaction.py b/mineru/doclib/background/compaction.pyindex 5e272414..29628072 100644--- a/mineru/doclib/background/compaction.py+++ b/mineru/doclib/background/compaction.py@@ -14,6 +14,7 @@ from typing import Any, castfrom ...parser.page_range import format_page_rangefrom ...types import Tierfrom ..core.db import DatabaseManager+from ..core.middle_json import read_cached_middle_jsonfrom ..rows import ParseBatchRow, ParseGroupRow, ParseRowfrom ..services.parse_svc import parse_batch_json_path, parse_page_range_setfrom ..types import PARSE_STATUS_DONE, PARSE_STATUS_SUPERSEDED@@ -22,8 +23,8 @@ logger = logging.getLogger("mineru.compaction")def _normalize_batch_pages(batch_payload: dict[str, Any]) -> list[dict[str, Any]]:- """仅校验并读取新版批次,拒绝猜测或改写旧协议数据。"""- return [page.to_dict() for page in ParseResult.from_dict(batch_payload).pages]+ """读取当前和受支持的历史批次,仅在内存中统一页面结构。"""+ return [page.to_dict() for page in read_cached_middle_json(batch_payload).pages]class Compaction:@@ -149,7 +150,8 @@ class Compaction:try:with open(fpath, encoding="utf-8") as f:batch_payload = json.load(f)- batch_pages = _normalize_batch_pages(batch_payload)+ middle_json = read_cached_middle_json(batch_payload)+ batch_pages = [page.to_dict() for page in middle_json.pages]except Exception:return Noneif not batch_pages:@@ -159,9 +161,7 @@ class Compaction:if type(page_idx) is not int or page_idx < 0:return Nonepages_by_page_idx[page_idx] = page- current_envelope = {- key: value for key, value in ParseResult.from_dict(batch_payload).to_dict().items() if key != "pages"- }+ current_envelope = {key: value for key, value in middle_json.to_dict().items() if key != "pages"}if not envelope:envelope = current_envelopeelif json.dumps(envelope, sort_keys=True) != json.dumps(current_envelope, sort_keys=True):diff --git a/mineru/doclib/core/middle_json.py b/mineru/doclib/core/middle_json.pynew file mode 100644index 00000000..c0fdc723--- /dev/null+++ b/mineru/doclib/core/middle_json.py@@ -0,0 +1,130 @@+"""仅供 Doclib 持久化结果使用的 Middle JSON 读取与历史转换边界。++两个历史分支集中在此,未来可一并移除;通用 ParseResult 与 DocVortex codec+不调用本模块。读取只构造内存对象,不重写文件、不重新推理。+"""++from __future__ import annotations++from copy import deepcopy+from typing import Any++from docvortex.schema import DocumentMetadata, MiddleJson, ModelJson, Producer++from ...integrations.docvortex import MinerUMetadata, build_metadata, validate_mineru_metadata++_LEGACY_PAGE_FIELDS = frozenset({"page_size", "preproc_blocks", "para_blocks", "discarded_blocks"})+_LEGACY_V2_FIELDS = frozenset(+ {"schema_version", "pages", "is_full_document", "file_suffix", "effort", "parse_mode", "mineru_version", "extensions"}+)+++def _legacy_metadata(payload: dict[str, Any], *, version_keys: tuple[str, ...]) -> DocumentMetadata:+ """保留可辨识的历史生产版本,缺失时标记未知而不使用当前程序版本。"""+ versions = [payload[key].strip() for key in version_keys if isinstance(payload.get(key), str) and payload[key].strip()]+ if len(set(versions)) > 1:+ raise ValueError("Conflicting legacy producer versions")+ version = versions[0] if versions else "unknown"+ return DocumentMetadata(+ file_suffix=payload.get("file_suffix", "pdf"),+ producer=Producer(name="mineru", version=version),+ )+++def _legacy_extensions(payload: dict[str, Any]) -> dict[str, Any]:+ """只迁移可靠的档位及模式,保留应用扩展并拒绝保留字段冲突。"""+ extensions = deepcopy(payload.get("extensions", {}))+ if not isinstance(extensions, dict):+ raise ValueError("legacy extensions must be a JSON object")+ efforts = {+ payload[key]+ for key in ("_effort", "effort")+ if isinstance(payload.get(key), str) and payload[key] in {"flash", "medium", "high", "xhigh"}+ }+ if len(efforts) > 1:+ raise ValueError("Conflicting legacy effort records")+ effort = next(iter(efforts), None)+ mode = payload.get("parse_mode")+ ocr_enabled = payload.get("_ocr_enable")+ if mode in ("txt", "ocr") and type(ocr_enabled) is bool and (mode == "ocr") != ocr_enabled:+ raise ValueError("Conflicting legacy parse_mode and _ocr_enable")+ if mode not in ("txt", "ocr"):+ mode = ("ocr" if ocr_enabled else "txt") if type(ocr_enabled) is bool else None++ if "mineru" in extensions:+ existing = MinerUMetadata.model_validate(extensions["mineru"])+ if mode is not None and existing.parse_mode != mode:+ raise ValueError("Conflicting legacy parse_mode and extensions.mineru")+ if effort is not None and existing.tier != build_metadata(effort=effort, parse_mode="txt")["mineru"]["tier"]:+ raise ValueError("Conflicting legacy effort and extensions.mineru")+ elif effort is not None and mode is not None:+ extensions.update(build_metadata(effort=effort, parse_mode=mode))+ return extensions+++def _read_legacy_345(payload: dict[str, Any], *, page_field: str) -> MiddleJson:+ """把可识别的 3.4.5 页面转为当前 raw ModelJson,再做确定性后处理。"""+ from ...backend.postprocess.legacy_schema_adapter import legacy_page_to_model_list+ from docvortex.postprocess.document import model_json_to_middle_json++ pages = payload[page_field]+ if not isinstance(pages, list) or any(not isinstance(page, dict) for page in pages):+ raise ValueError("legacy pages must be a list of page objects")+ for page in pages:+ if "blocks" in page or (page_field == "pages" and not _LEGACY_PAGE_FIELDS.intersection(page)):+ raise ValueError("Mixed or unrecognized legacy page structure")+ indices = [page.get("page_idx", index) for index, page in enumerate(pages)]+ if any(type(index) is not int or index < 0 for index in indices):+ raise ValueError("legacy page indices must be non-negative integers")+ model = ModelJson(+ pages=[legacy_page_to_model_list(page) for page in pages],+ page_index_map=[] if indices == list(range(len(pages))) else indices,+ metadata=_legacy_metadata(payload, version_keys=("_version_name", "mineru_version")),+ extensions=_legacy_extensions(payload),+ )+ if "is_full_document" in payload:+ full_document = payload["is_full_document"]+ if type(full_document) is not bool or full_document != model.is_full_document:+ raise ValueError("Conflicting legacy is_full_document and page indices")+ return model_json_to_middle_json(model)+++def _read_legacy_v2(payload: dict[str, Any]) -> MiddleJson:+ """只移动旧 Schema 2.0 外层字段,页面树直接经过当前类型校验。"""+ unexpected = payload.keys() - _LEGACY_V2_FIELDS+ if unexpected:+ raise ValueError(f"Conflicting or unsupported legacy Middle JSON fields: {sorted(unexpected)}")+ return MiddleJson(+ pages=payload.get("pages"),+ is_full_document=payload.get("is_full_document"),+ metadata=_legacy_metadata(payload, version_keys=("mineru_version",)),+ extensions=_legacy_extensions(payload),+ )+++def read_cached_middle_json(payload: dict[str, Any]) -> MiddleJson:+ """先识别显式协议,再转换两个历史缓存族;失败时绝不跨协议兜底。"""+ if not isinstance(payload, dict):+ raise ValueError("Cached Middle JSON must be an object; source reparse required")+ if "schema" in payload:+ document = MiddleJson.from_dict(payload)+ else:+ if {"metadata", "producer", "page_index_map"}.intersection(payload):+ raise ValueError("Mixed cached document envelope; source reparse required")+ version = payload.get("schema_version")+ source = deepcopy(payload)+ if "pdf_info" in source:+ if version is not None or "pages" in source:+ raise ValueError("Mixed legacy pdf_info envelope; source reparse required")+ document = _read_legacy_345(source, page_field="pdf_info")+ elif version == "1.0" and "pages" in source:+ document = _read_legacy_345(source, page_field="pages")+ elif version == "2.0" and "pages" in source:+ document = _read_legacy_v2(source)+ else:+ raise ValueError("Unsupported cached Middle JSON format; source reparse required")+ validate_mineru_metadata(document)+ return document+++__all__ = ["read_cached_middle_json"]diff --git a/mineru/doclib/server.py b/mineru/doclib/server.pyindex f9b0aee8..828b809d 100644--- a/mineru/doclib/server.py+++ b/mineru/doclib/server.py@@ -75,7 +75,7 @@ from .rows import (WatchTargetRow,)from .services.parse_svc import (- _json_batch_is_current,+ _json_batch_is_readable,accessible_file_for_sha256,filter_pages_by_user_range,load_pages_from_done_batches,@@ -423,7 +423,7 @@ class DoclibServer(AsyncDoclibInterface):rowfor row in rowsif row["status"] != PARSE_STATUS_DONE- or _json_batch_is_current(_effective_data_dir(self.state), row["sha256"], row["tier"], row)+ or _json_batch_is_readable(_effective_data_dir(self.state), row["sha256"], row["tier"], row)]coverage = _parse_coverage(page_range, coverage_rows) if resolved_sha256 and tier and page_range else Nonereturn ListParsesResponse(@@ -882,7 +882,7 @@ class DoclibServer(AsyncDoclibInterface):_tier_parse_info(row)for row in parse_rowsif row["status"] == PARSE_STATUS_DONE- and _json_batch_is_current(_effective_data_dir(self.state), row["sha256"], row["tier"], row)+ and _json_batch_is_readable(_effective_data_dir(self.state), row["sha256"], row["tier"], row)]active_parses = [_parse_info(row) for row in parse_rows if row["status"] in {PARSE_STATUS_PENDING, PARSE_STATUS_PARSING}@@ -1448,7 +1448,7 @@ class DoclibServer(AsyncDoclibInterface):for row in rowsif row["tier"] in TIERSand row["tier"] != "flash"- and _json_batch_is_current(_effective_data_dir(self.state), sha256, row["tier"], row)+ and _json_batch_is_readable(_effective_data_dir(self.state), sha256, row["tier"], row)}if not tiers:return Nonediff --git a/mineru/doclib/services/parse_svc.py b/mineru/doclib/services/parse_svc.pyindex 3a351a81..acec445f 100644--- a/mineru/doclib/services/parse_svc.py+++ b/mineru/doclib/services/parse_svc.py@@ -28,6 +28,7 @@ from ...filetypes import ()from ...parser.api_client import _APITransportError, _V1APIErrorfrom ...parser.base import ParseResult+from ..core.middle_json import read_cached_middle_jsonfrom ...parser.page_range import (expand_page_range,format_page_range as _page_numbers_to_range_str,@@ -745,7 +746,7 @@ class ParseService:),)valid_done_batches = [- batch for batch in done_batches if _json_batch_is_current(self.data_dir, sha256, requested_tier, batch)+ batch for batch in done_batches if _json_batch_is_readable(self.data_dir, sha256, requested_tier, batch)]if supports_page_range:for batch in valid_done_batches:@@ -1405,7 +1406,7 @@ class ParseService:tiers = sorted(tier_set, key=lambda item: TIER_ORDER.get(item, -1), reverse=True)for tier in tiers:tier_rows = [- row for row in rows if row["tier"] == tier and _json_batch_is_current(self.data_dir, sha256, tier, row)+ row for row in rows if row["tier"] == tier and _json_batch_is_readable(self.data_dir, sha256, tier, row)]pages = load_pages_from_done_batches(self.data_dir, sha256, tier, tier_rows)if not pages:@@ -1497,12 +1498,12 @@ def _probe_error_message(prefix: str, error_msg: str | None) -> str:return prefix-def _json_batch_is_current(data_dir: str, sha256: str, tier: Tier, batch: ParseRow) -> bool:- """仅把协议有效且覆盖记录页范围的新批次视为可命中的缓存。"""+def _json_batch_is_readable(data_dir: str, sha256: str, tier: Tier, batch: ParseRow) -> bool:+ """将可转换为当前协议且覆盖记录页范围的持久化批次视为有效缓存。"""json_path = parse_batch_json_path(data_dir, sha256, tier, batch["page_range"], batch["done_at"])try:with open(json_path, encoding="utf-8") as stream:- result = ParseResult.from_dict(json.load(stream))+ result = read_cached_middle_json(json.load(stream))available = {page.page_idx + 1 for page in result.pages}return bool(available) and parse_page_range_set(batch["page_range"]) <= availableexcept (OSError, ValueError, TypeError):@@ -1523,7 +1524,7 @@ def load_pages_from_done_batches(*,requested_page_numbers: set[int] | None = None,) -> list[PageInfo]:- """读取新协议批次并保留最新重复页,所请求的旧缓存需明确重新解析。"""+ """读取当前或受支持的历史批次,保留最新重复页;损坏或未知缓存须重新解析。"""pages_by_page_idx: dict[int, PageInfo] = {}stale_page_numbers: set[int] = set()for row in reversed(done_rows):@@ -1533,7 +1534,7 @@ def load_pages_from_done_batches(try:with open(fpath, encoding="utf-8") as f:data = json.load(f)- parse_result = ParseResult.from_dict(data)+ parse_result = read_cached_middle_json(data)for page in parse_result.pages:pages_by_page_idx[page.page_idx] = pageexcept (OSError, ValueError, TypeError):diff --git a/pyproject.toml b/pyproject.tomlindex b3fa82f4..4836922b 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -38,7 +38,7 @@ classifiers = ["Programming Language :: Python :: 3.13",]dependencies = [- "docvortex>=0.2.0,<0.3.0",+ "docvortex>=0.2.1,<0.3.0","mineru-vl-utils>=1.2.1,<2","metafile-render>=0.3.0,<1.0.0","mineru-llama-cpp>=0.1.0,<1",diff --git a/tests/unittest/test_doclib_legacy_page_ranges.py b/tests/unittest/test_doclib_legacy_page_ranges.pyindex 023a9afb..2d738772 100644--- a/tests/unittest/test_doclib_legacy_page_ranges.py+++ b/tests/unittest/test_doclib_legacy_page_ranges.py@@ -28,6 +28,108 @@ from mineru.errors import MineruError_SHA256 = "a" * 64+def _convert_fixture_family(path: Path, family: str) -> None:+ """仅在测试建库阶段生成不同历史格式,正文及来源记录保持可对照。"""+ if family == "current":+ return+ source = json.loads(path.read_text())+ metadata = source["metadata"]+ product = source["extensions"]["mineru"]+ effort = {"flash": "flash", "basic": "medium", "standard": "high", "advanced": "xhigh"}[product["tier"]]+ if family == "v2":+ output = {+ "schema_version": "2.0",+ "pages": source["pages"],+ "is_full_document": source["is_full_document"],+ "file_suffix": metadata["file_suffix"],+ "mineru_version": metadata["producer"]["version"],+ "effort": effort,+ "parse_mode": product["parse_mode"],+ }+ else:+ pages = []+ for page in source["pages"]:+ blocks = []+ for block in page["blocks"]:+ assert block["type"] == "text"+ bbox = [coordinate * 100 for coordinate in block["bbox"]]+ text = "".join(span["content"] for span in block["content"])+ blocks.append(+ {"type": "text", "bbox": bbox, "lines": [{"bbox": bbox, "spans": [{"type": "text", "content": text}]}]}+ )+ pages.append(+ {"page_idx": page["page_idx"], "page_size": [100, 100], "preproc_blocks": blocks, "discarded_blocks": []}+ )+ output = {+ "_version_name": metadata["producer"]["version"],+ "_effort": effort,+ "_ocr_enable": product["parse_mode"] == "ocr",+ }+ if family == "345":+ output["pdf_info"] = pages+ else:+ assert family == "v1"+ output.update(schema_version="1.0", pages=pages)+ path.write_text(json.dumps(output))++[email protected]("family", ["345", "v1", "v2"])+def test_historical_cache_rebuilds_fts_and_default_tier(tmp_path: Path, family: str) -> None:+ """历史缓存参与全文索引重建和默认档位选择,读取不修改源批次。"""++ async def verify() -> None:+ """用真实 FTS5 和数据库查询核对所有历史格式的消费链。"""+ async with _legacy_store(tmp_path, family=family) as (db, service, server):+ path = _cache_path(tmp_path, "1~5", 1000)+ standard_path = Path(parse_batch_json_path(str(tmp_path), _SHA256, "standard", "1~5", 1000))+ standard_path.parent.mkdir(parents=True, exist_ok=True)+ standard_path.write_bytes(path.read_bytes())+ await db.execute("UPDATE parses SET tier='standard'")+ before = standard_path.read_bytes()+ assert await server._default_read_tier(_SHA256) == "standard"+ await service._rebuild_fts_after_invalidate(_SHA256)+ assert await service.fts.search("old")+ assert await service.fts.get_tier(_SHA256) == "standard"+ assert standard_path.read_bytes() == before++ asyncio.run(verify())++[email protected]("legacy_family", ["345", "v1"])+def test_all_supported_formats_compact_to_current_protocol(tmp_path: Path, legacy_family: str) -> None:+ """混合三种协议的兼容批次合并后只输出新协议,重复页保留最新正文。"""++ async def verify() -> None:+ """使用非零起始页模拟旧抽页结果,统一后保持同一份来源及整本语义。"""+ async with _legacy_store(tmp_path) as (db, _service, server):+ await db.execute("DELETE FROM parses")+ _cache_path(tmp_path, "1~5", 1000).unlink()+ for family, page_range, done_at, numbers, label in [+ (legacy_family, "3-4", 1000, [3, 4], "old"),+ ("v2", "4-5", 2000, [4, 5], "intermediate"),+ ("current", "5-6", 3000, [5, 6], "latest"),+ ]:+ await _add_result(db, tmp_path, page_range, done_at, numbers, label)+ _convert_fixture_family(_cache_path(tmp_path, page_range, done_at), family)+ compaction = Compaction(db=db, interval_sec=600, data_dir=str(tmp_path))+ assert await compaction._compact_doc_tier(_SHA256, "flash") == 2+ path = _cache_path(tmp_path, "3-6", 3000)+ payload = json.loads(path.read_text())+ assert payload["schema"] == "docvortex.middle"+ assert payload["schema_version"] == "2.0"+ assert "file_suffix" not in payload and "effort" not in payload+ assert payload["metadata"]["producer"]["version"] == __version__+ assert [page["page_idx"] for page in payload["pages"]] == [2, 3, 4, 5]+ content = await server._render_doc_content(+ _SHA256, tier="flash", page_range="3-6", format="markdown", no_marker=True+ )+ assert "old page 3" in content and "intermediate page 4" in content and "latest page 5" in content+ assert "old page 4" not in content and "intermediate page 5" not in content+ assert [item.name for item in path.parent.glob("*.json")] == [path.name]++ asyncio.run(verify())++@pytest.mark.parametrize("old_version", [None, "1.0"])def test_old_protocol_cache_is_not_available_or_compacted(tmp_path: Path, old_version: str | None) -> None:"""旧协议不命中、不参与覆盖和压缩,读取旧页明确要求重新解析。"""@@ -142,7 +244,9 @@ async def _add_result(@asynccontextmanager-async def _legacy_store(root: Path) -> AsyncIterator[tuple[DatabaseManager, ParseService, DoclibServer]]:+async def _legacy_store(+ root: Path, *, family: str = "current"+) -> AsyncIterator[tuple[DatabaseManager, ParseService, DoclibServer]]:"""建立已入库的源文件元数据、旧名称缓存及真实 SQLite 服务,退出时关闭连接。"""source = root / "document.pdf"source.write_bytes(b"%PDF-1.7\n")@@ -161,6 +265,7 @@ async def _legacy_store(root: Path) -> AsyncIterator[tuple[DatabaseManager, Pars(str(source), source.name, "pdf", stat.st_size, int(stat.st_mtime * 1000), _SHA256, "active", 1000, 1000),)await _add_result(db, root, "1~5", 1000, range(1, 6), "old")+ _convert_fixture_family(_cache_path(root, "1~5", 1000), family)service = ParseService(db=db, fts=FTSManager(db), config_svc=ConfigService(db), data_dir=str(root), parse_lock_timeout_sec=1800)@@ -170,12 +275,13 @@ async def _legacy_store(root: Path) -> AsyncIterator[tuple[DatabaseManager, Parsawait db.close()-def test_old_cache_hits_exports_and_serializes_without_mutating_storage(tmp_path: Path) -> None:[email protected]("family", ["current", "345", "v1", "v2"])+def test_old_cache_hits_exports_and_serializes_without_mutating_storage(tmp_path: Path, family: str) -> None:"""旧结果正常命中、导出和查询,返回新格式而文件与数据库始终保留旧名称。"""async def verify() -> None:"""读取各个结果出口,确认规范化不破坏原始缓存定位。"""- async with _legacy_store(tmp_path) as (db, service, server):+ async with _legacy_store(tmp_path, family=family) as (db, service, server):old_path = _cache_path(tmp_path, "1~5", 1000)original_bytes = old_path.read_bytes()row = await db.fetchone("SELECT * FROM parses WHERE sha256=?", (_SHA256,))@@ -211,12 +317,13 @@ def test_old_cache_hits_exports_and_serializes_without_mutating_storage(tmp_pathasyncio.run(verify())-def test_old_cache_only_schedules_uncovered_pages(tmp_path: Path) -> None:[email protected]("family", ["current", "345", "v1", "v2"])+def test_old_cache_only_schedules_uncovered_pages(tmp_path: Path, family: str) -> None:"""旧 1~5 批次覆盖前五页,新请求只创建 6-10,重复请求复用新任务。"""async def verify() -> None:"""检查真实任务入队和原记录内容。"""- async with _legacy_store(tmp_path) as (db, service, _server):+ async with _legacy_store(tmp_path, family=family) as (db, service, _server):result = await service.request_parse(str(tmp_path / "document.pdf"), tier="flash", page_range="1-10")assert not result.cache_hit and len(result.created_parse_ids) == 1rows = await db.fetchall("SELECT page_range, status FROM parses ORDER BY id")@@ -229,12 +336,13 @@ def test_old_cache_only_schedules_uncovered_pages(tmp_path: Path) -> None:asyncio.run(verify())-def test_mixed_cache_compaction_keeps_latest_pages_and_writes_new_ranges(tmp_path: Path) -> None:[email protected]("family", ["current", "v2"])+def test_mixed_cache_compaction_keeps_latest_pages_and_writes_new_ranges(tmp_path: Path, family: str) -> None:"""新旧缓存合并时重复页取最新正文,并生成连字符记录与文件名。"""async def verify() -> None:"""压缩前后均核对页序与重复页正文。"""- async with _legacy_store(tmp_path) as (db, _service, server):+ async with _legacy_store(tmp_path, family=family) as (db, _service, server):await _add_result(db, tmp_path, "4-7", 2000, range(4, 8), "new")rows = await db.fetchall("SELECT * FROM parses ORDER BY done_at DESC")pages = load_pages_from_done_batches(str(tmp_path), _SHA256, "flash", rows)diff --git a/tests/unittest/test_doclib_legacy_schema_adapter.py b/tests/unittest/test_doclib_legacy_schema_adapter.pynew file mode 100644index 00000000..0d594510--- /dev/null+++ b/tests/unittest/test_doclib_legacy_schema_adapter.py@@ -0,0 +1,391 @@+from docvortex.schema import Producer+from mineru.integrations.docvortex import build_metadata+from copy import deepcopy++import pytest++from mineru.backend.postprocess.legacy_schema_adapter import legacy_page_to_model_list+from docvortex.postprocess.pages import model_json_to_pages+from mineru.doclib.background.compaction import _normalize_batch_pages+from mineru.types import BlockType, MiddleJson, ModelJson+++def _legacy_line(content: str, *, bbox: list[int] | None = None) -> dict:+ """构造 3.4.5 普通文字行。"""+ return {+ "bbox": bbox or [10, 10, 90, 20],+ "spans": [{"type": "text", "content": content}],+ }+++def _current_pages(raw_pages: list[list[dict]]) -> list:+ """把适配后的 raw pages 送入当前严格后处理。"""+ model_json = ModelJson(+ pages=raw_pages,+ page_index_map=[],+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version="3.4.5")},+ extensions=build_metadata(effort="medium", parse_mode="txt"),+ )+ return model_json_to_pages(model_json)+++def test_legacy_adapter_converts_inline_spans_without_mutating_input() -> None:+ """验证文字标签、公式、链接及危险链接降级,并保证输入不变。"""+ page = {+ "page_idx": 0,+ "page_size": [100, 100],+ "preproc_blocks": [+ {+ "type": "text",+ "bbox": [10, 10, 90, 30],+ "lines": [+ {+ "bbox": [10, 10, 90, 30],+ "spans": [+ {+ "type": "text",+ "content": (+ 'plain <text style="bold,italic">styled</text> '+ "<eq>x^2</eq> "+ "<hyperlink><text>safe</text><url>https://example.com</url></hyperlink> "+ "<hyperlink><text>unsafe</text><url>javascript:alert(1)</url></hyperlink>"+ ),+ }+ ],+ }+ ],+ }+ ],+ "discarded_blocks": [],+ }+ original = deepcopy(page)++ raw_page = legacy_page_to_model_list(page)+ content = raw_page[0]["content"]++ assert page == original+ assert content == [+ {"type": "text", "content": "plain ", "styles": []},+ {"type": "text", "content": "styled", "styles": ["bold", "italic"]},+ {"type": "text", "content": " ", "styles": []},+ {"type": "equation_inline", "content": "x^2"},+ {"type": "text", "content": " ", "styles": []},+ {+ "type": "hyperlink",+ "url": "https://example.com",+ "content": [{"type": "text", "content": "safe", "styles": []}],+ },+ {"type": "text", "content": " unsafe", "styles": []},+ ]+ pages = _current_pages([raw_page])+ assert pages[0].blocks[0].type == BlockType.TEXT+++def test_legacy_adapter_preserves_visual_metadata_and_algorithm_spans() -> None:+ """验证视觉父块展平后能由当前后处理恢复表格与算法块。"""+ page = {+ "page_idx": 0,+ "page_size": [100, 100],+ "para_blocks": [+ {+ "type": "table",+ "bbox": [10, 10, 90, 60],+ "cell_merge": [0, 1],+ "blocks": [+ {+ "type": "table_caption",+ "bbox": [10, 10, 90, 20],+ "lines": [_legacy_line("表 1")],+ },+ {+ "type": "table_body",+ "bbox": [10, 20, 90, 60],+ "lines": [+ {+ "bbox": [10, 20, 90, 60],+ "spans": [+ {+ "type": "table",+ "html": "<table><tr><td>x</td></tr></table>",+ "image_path": "images/table.png",+ }+ ],+ }+ ],+ },+ ],+ },+ {+ "type": "code",+ "sub_type": "algorithm",+ "bbox": [10, 70, 90, 90],+ "blocks": [+ {+ "type": "code_body",+ "bbox": [10, 70, 90, 90],+ "lines": [+ {+ "bbox": [10, 70, 90, 90],+ "spans": [+ {"type": "text", "content": "a="},+ {"type": "inline_equation", "content": "x"},+ ],+ }+ ],+ }+ ],+ },+ ],+ "discarded_blocks": [{"type": "header", "bbox": [10, 1, 90, 5], "lines": [_legacy_line("Header")]}],+ }++ pages = _current_pages([legacy_page_to_model_list(page)])+ table, code, header = pages[0].blocks++ assert table.type == BlockType.TABLE+ assert table.cell_merge == [0, 1]+ assert table.content[1].image_path == "images/table.png"+ assert code.type == BlockType.CODE+ assert code.sub_type == "algorithm"+ assert code.content[0].type == BlockType.ALGORITHM_BODY+ assert [span.type for span in code.content[0].content] == ["text", "equation_inline"]+ assert header.type == BlockType.HEADER+++def test_legacy_adapter_normalizes_bbox_and_block_aliases() -> None:+ """验证像素坐标、越界坐标及旧 discriminator 均转换为当前合法值。"""+ page = {+ "page_size": [200, 100],+ "preproc_blocks": [+ {+ "type": "title",+ "level": 1,+ "bbox": [-10, 0, 220, 20],+ "lines": [_legacy_line("Document")],+ },+ {+ "type": "interline_equation",+ "bbox": [0, 30, 100, 50],+ "lines": [+ {+ "bbox": [0, 30, 100, 50],+ "spans": [{"type": "interline_equation", "content": "y=1"}],+ }+ ],+ },+ ],+ "discarded_blocks": [],+ }++ raw_page = legacy_page_to_model_list(page)++ assert raw_page[0]["type"] == "doc_title"+ assert raw_page[0]["bbox"] == (0.0, 0.0, 1.0, 0.2)+ assert raw_page[1]["type"] == "equation"+ assert raw_page[1]["content"] == "y=1"+ _current_pages([raw_page])+++def test_legacy_adapter_keeps_only_final_middle_json_aliases() -> None:+ """验证最终 3.4.5 仍会序列化的三个旧类型继续转换。"""+ page = {+ "page_size": [100, 100],+ "preproc_blocks": [+ {"type": "abstract", "bbox": [0, 0, 100, 20], "lines": [_legacy_line("Abstract")]},+ {"type": "phonetic", "bbox": [0, 30, 100, 50], "lines": [_legacy_line("phonetic")]},+ {+ "type": "interline_equation",+ "bbox": [0, 60, 100, 80],+ "lines": [+ {+ "bbox": [0, 60, 100, 80],+ "spans": [{"type": "interline_equation", "content": "x=1"}],+ }+ ],+ },+ ],+ "discarded_blocks": [],+ }++ raw_page = legacy_page_to_model_list(page)++ assert [block["type"] for block in raw_page] == ["text", "text", "equation"]+ assert raw_page[0]["content"][0]["content"] == "Abstract"+ assert raw_page[1]["content"][0]["content"] == "phonetic"+ assert raw_page[2]["content"] == "x=1"+ _current_pages([raw_page])+++ "block_type",+ [+ "vertical_text",+ "formula_number",+ "discarded",+ "image_block",+ "algorithm",+ "algorithm_body",+ "algorithm_caption",+ "algorithm_footnote",+ "header_image",+ "footer_image",+ ],+)+def test_legacy_adapter_rejects_pre_middle_json_block_types(block_type: str) -> None:+ """验证 finalize 前布局、raw model 与 Content List 类型要求源文件重解析。"""+ block = {+ "type": block_type,+ "bbox": [0, 0, 100, 20],+ "lines": [_legacy_line("internal")],+ }+ if block_type in {"image_block", "algorithm"}:+ block["blocks"] = [{"type": "text", "bbox": [0, 0, 100, 20], "lines": [_legacy_line("nested")]}]+ page = {+ "page_size": [100, 100],+ "preproc_blocks": [block],+ "discarded_blocks": [],+ }++ with pytest.raises(ValueError, match=rf"{block_type!s}.*source reparse"):+ legacy_page_to_model_list(page)+++def test_legacy_adapter_preserves_canonical_image_and_auxiliary_blocks() -> None:+ """验证规范 image 视觉块及 header/footer 辅助块不会提升为正文图片。"""+ page = {+ "page_size": [100, 100],+ "preproc_blocks": [+ {+ "type": "image",+ "bbox": [10, 20, 90, 80],+ "blocks": [+ {+ "type": "image_body",+ "bbox": [10, 20, 90, 80],+ "lines": [+ {+ "bbox": [10, 20, 90, 80],+ "spans": [{"type": "image", "image_path": "images/body.png"}],+ }+ ],+ }+ ],+ }+ ],+ "discarded_blocks": [+ {"type": "header", "bbox": [0, 0, 100, 10], "lines": [_legacy_line("Header")]},+ {"type": "footer", "bbox": [0, 90, 100, 100], "lines": [_legacy_line("Footer")]},+ ],+ }++ pages = _current_pages([legacy_page_to_model_list(page)])++ assert [block.type for block in pages[0].blocks] == [BlockType.IMAGE, BlockType.HEADER, BlockType.FOOTER]+ image = pages[0].blocks[0]+ assert image.content[0].image_path == "images/body.png"+++def test_legacy_adapter_recovers_pdf_list_items_from_line_flags() -> None:+ """验证旧 PDF list 的物理行重新生成当前列表文本子块。"""+ page = {+ "page_size": [100, 100],+ "preproc_blocks": [+ {+ "type": "list",+ "bbox": [10, 10, 90, 40],+ "lines": [+ {+ "bbox": [10, 10, 90, 20],+ "is_list_start_line": True,+ "spans": [{"type": "text", "content": "one"}],+ },+ {+ "bbox": [10, 25, 90, 35],+ "is_list_start_line": True,+ "spans": [{"type": "text", "content": "two"}],+ },+ ],+ }+ ],+ "discarded_blocks": [],+ }++ pages = _current_pages([legacy_page_to_model_list(page)])+ list_block = pages[0].blocks[0]++ assert list_block.type == BlockType.LIST+ assert [child.content[0].content for child in list_block.content] == ["one", "two"]+++def test_legacy_adapter_separates_office_blocks_with_synthetic_bbox_groups() -> None:+ """验证无 page_size 时列表不会吸收相邻 Office 正文块。"""+ page = {+ "para_blocks": [+ {"type": "text", "lines": [{"spans": [{"type": "text", "content": "before"}]}]},+ {+ "type": "list",+ "blocks": [+ {"type": "text", "lines": [{"spans": [{"type": "text", "content": "item"}]}]},+ ],+ },+ {"type": "text", "lines": [{"spans": [{"type": "text", "content": "after"}]}]},+ ],+ "discarded_blocks": [],+ }++ pages = _current_pages([legacy_page_to_model_list(page)])++ assert [block.type for block in pages[0].blocks] == [BlockType.TEXT, BlockType.LIST, BlockType.TEXT]+ assert pages[0].blocks[0].content[0].content == "before"+ assert pages[0].blocks[1].content[0].content[0].content == "item"+ assert pages[0].blocks[2].content[0].content == "after"+++def test_doclib_compaction_converts_schema_v1_pages_with_current_adapter() -> None:+ """验证 Doclib 历史调用点重新通过宿主适配器生成 2.0 page 字典。"""+ payload = {+ "schema_version": "1.0",+ "pages": [+ {+ "page_idx": 3,+ "page_size": [100, 100],+ "preproc_blocks": [+ {+ "type": "text",+ "bbox": [10, 10, 90, 20],+ "lines": [_legacy_line("cached")],+ }+ ],+ "discarded_blocks": [],+ }+ ],+ }++ pages = _normalize_batch_pages(payload)++ assert pages[0]["page_idx"] == 3+ assert pages[0]["blocks"][0]["content"] == [{"type": "text", "content": "cached"}]+++def test_doclib_compaction_rejects_unknown_legacy_schema() -> None:+ """验证未知 schema 不会被误当成 3.4.5 页面吞掉。"""+ with pytest.raises(ValueError, match="source reparse"):+ _normalize_batch_pages({"schema_version": "99.0", "pages": [{"page_idx": 0, "blocks": []}]})+++def test_doclib_compaction_accepts_current_schema_without_legacy_conversion(monkeypatch: pytest.MonkeyPatch) -> None:+ """验证当前 2.0 缓存直接读取页面,不进入旧版转换。"""+ from mineru.backend.postprocess import legacy_schema_adapter++ def unexpected_conversion(_page: dict) -> list[dict]:+ """当前缓存触发旧版转换时立即报告路由错误。"""+ raise AssertionError("current schema must not use legacy conversion")++ monkeypatch.setattr(legacy_schema_adapter, "legacy_page_to_model_list", unexpected_conversion)+ pages = [{"page_idx": 0, "blocks": []}]+ document = MiddleJson(+ pages=pages,+ is_full_document=True,+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version="test")},+ )+ assert _normalize_batch_pages(document.to_dict()) == document.to_dict()["pages"]diff --git a/tests/unittest/test_doclib_middle_json.py b/tests/unittest/test_doclib_middle_json.pynew file mode 100644index 00000000..367d9598--- /dev/null+++ b/tests/unittest/test_doclib_middle_json.py@@ -0,0 +1,256 @@+"""Doclib 专属历史文档读取的结构、来源及通用边界回归。"""++from copy import deepcopy+from collections.abc import Callable+import json++import pytest+from docvortex.schema import MiddleJson++from mineru.doclib.core.middle_json import read_cached_middle_json+from mineru.parser import ParseResult+++def _legacy_v2() -> dict:+ """构造带续接、标题、表格素材和应用扩展的旧 Schema 2.0 文档。"""+ return {+ "schema_version": "2.0",+ "file_suffix": "pdf",+ "mineru_version": "3.4.4",+ "effort": "high",+ "parse_mode": "ocr",+ "is_full_document": False,+ "extensions": {"application": {"items": [None, True, 1, {"type": "image", "image_base64": "keep"}]}},+ "pages": [+ {+ "page_idx": 3,+ "blocks": [+ {+ "type": "paragraph_title",+ "index": 0,+ "bbox": [0.1, 0.1, 0.9, 0.2],+ "level": 4,+ "anchor": "section",+ "content": [{"type": "text", "content": "标题"}],+ },+ {+ "type": "text",+ "index": 1,+ "bbox": [0.1, 0.3, 0.9, 0.4],+ "continues_prev": True,+ "content": [+ {+ "type": "hyperlink",+ "url": "https://example.com",+ "content": [{"type": "text", "content": "保留正文"}],+ }+ ],+ },+ {+ "type": "table",+ "index": 2,+ "bbox": [0.1, 0.5, 0.9, 0.8],+ "cell_merge": [0, 1],+ "content": [+ {+ "type": "table_body",+ "index": 2,+ "bbox": [0.1, 0.5, 0.9, 0.8],+ "content": "<table><tr><td>保留表格</td></tr></table>",+ "image_path": "images/table.png",+ }+ ],+ },+ ],+ },+ {"page_idx": 5},+ ],+ }+++def _legacy_345(*, wrapped: bool = False) -> dict:+ """构造真实旧字段布局及非连续页号,版本保留发行标签中的 3.4.4。"""+ pages = [+ {+ "page_idx": index,+ "page_size": [100, 200],+ "preproc_blocks": [+ {+ "type": "text",+ "bbox": [10, 20, 90, 40],+ "lines": [{"bbox": [10, 20, 90, 40], "spans": [{"type": "text", "content": "正文 <eq>x^2</eq>"}]}],+ },+ ],+ "discarded_blocks": [],+ }+ for index in [3, 5]+ ]+ payload = {"_version_name": "3.4.4", "_effort": "high", "_ocr_enable": True}+ if wrapped:+ payload.update(schema_version="1.0", pages=pages)+ else:+ payload["pdf_info"] = pages+ return payload++[email protected]("wrapped", [False, True])+def test_legacy_345_roundtrip_preserves_origin_and_pages(wrapped: bool) -> None:+ """两个旧页面封装均可在缓存边界转换,输入和来源不受污染。"""+ payload = _legacy_345(wrapped=wrapped)+ before = deepcopy(payload)+ result = read_cached_middle_json(payload)+ assert payload == before+ assert [page.page_idx for page in result.pages] == [3, 5]+ assert result.is_full_document is False+ assert result.metadata.producer.version == "3.4.4"+ assert result.extensions["mineru"] == {"tier": "standard", "parse_mode": "ocr"}+ assert result.pages[0].blocks[0].content[0].content == "正文 "+ assert result.pages[0].blocks[0].content[1].content == "x^2"+ assert read_cached_middle_json(result.to_dict()) == result+ assert MiddleJson.from_json(result.to_json()) == result+++def test_v2_changes_only_envelope_without_postprocess(monkeypatch: pytest.MonkeyPatch) -> None:+ """旧 2.0 的页面保持不变,转换不触发段落表格后处理或 LLM。"""+ from mineru.backend.postprocess import document as host_document++ def unexpected(*_args: object, **_kwargs: object) -> None:+ """捕获任何越过外层迁移范围的后处理调用。"""+ raise AssertionError("Envelope migration must not postprocess pages")++ monkeypatch.setattr("docvortex.postprocess.document.model_json_to_middle_json", unexpected)+ monkeypatch.setattr("docvortex.postprocess.pages.model_json_to_pages", unexpected)+ monkeypatch.setattr(host_document, "apply_llm_aided_postprocess", unexpected)+ monkeypatch.setattr(host_document, "build_middle_json", unexpected)+ payload = _legacy_v2()+ before = deepcopy(payload)+ result = read_cached_middle_json(payload)+ assert payload == before+ assert result.to_dict()["pages"] == payload["pages"]+ assert result.metadata.file_suffix == "pdf"+ assert result.metadata.producer.version == "3.4.4"+ assert result.extensions["application"] == payload["extensions"]["application"]+ assert result.extensions["mineru"] == {"tier": "standard", "parse_mode": "ocr"}+ assert read_cached_middle_json(result.to_dict()).to_dict() == result.to_dict()+ result.extensions["application"]["items"].append("changed")+ assert payload == before+++ "updates,expected",+ [+ ({}, None),+ ({"_effort": "high"}, None),+ ({"_ocr_enable": True}, None),+ ({"_effort": "high", "_ocr_enable": False}, {"tier": "standard", "parse_mode": "txt"}),+ ({"_effort": "xhigh", "parse_mode": "ocr"}, {"tier": "advanced", "parse_mode": "ocr"}),+ ({"_effort": "unrecognized", "parse_mode": "txt"}, None),+ ],+)+def test_missing_metadata_stays_unknown(updates: dict, expected: dict | None) -> None:+ """缺失或不可识别的记录保持未知,不制造当前版本或默认解析档位。"""+ payload = {"pdf_info": [], **updates}+ result = read_cached_middle_json(payload)+ assert result.metadata.producer.model_dump() == {"name": "mineru", "version": "unknown"}+ assert result.metadata.file_suffix == "pdf"+ assert result.extensions.get("mineru") == expected+++def test_v2_missing_product_records_preserve_application_extensions() -> None:+ """旧 2.0 缺少版本及运行记录时也保持未知,应用扩展不受影响。"""+ payload = _legacy_v2()+ for key in ("mineru_version", "effort", "parse_mode"):+ del payload[key]+ document = read_cached_middle_json(payload)+ assert document.metadata.producer.version == "unknown"+ assert document.extensions == payload["extensions"]+++def test_consistent_existing_product_extension_is_preserved() -> None:+ """已有可靠产品扩展与旧字段一致时直接保留,不产生重复记录。"""+ payload = _legacy_v2()+ payload["extensions"]["mineru"] = {"tier": "standard", "parse_mode": "ocr"}+ assert read_cached_middle_json(payload).extensions == payload["extensions"]+++def test_345_conflicting_full_document_flag_fails() -> None:+ """旧根字段与实际抽页映射矛盾时不得静默覆盖整本语义。"""+ payload = _legacy_345()+ payload["is_full_document"] = True+ with pytest.raises(ValueError, match="Conflicting legacy is_full_document"):+ read_cached_middle_json(payload)+++ "updates",+ [+ {"mineru_version": "other-version"},+ {"effort": "medium"},+ {"parse_mode": "txt"},+ ],+)+def test_conflicting_legacy_metadata_aliases_fail(updates: dict) -> None:+ """同一来源或运行参数的旧别名互相矛盾时明确报错,不静默选取一方。"""+ payload = _legacy_345()+ payload.update(updates)+ with pytest.raises(ValueError, match="Conflicting legacy"):+ read_cached_middle_json(payload)+++ "mutate",+ [+ lambda p: p.update(schema="docvortex.middle", schema_version="1.0"),+ lambda p: p.update(schema=None),+ lambda p: p.update(schema="other.middle"),+ lambda p: p.update(metadata={}),+ lambda p: p.update(schema_version="99.0"),+ lambda p: p.update(file_suffix="unsupported"),+ lambda p: p.pop("is_full_document"),+ lambda p: p["extensions"].update(mineru={"tier": "basic", "parse_mode": "ocr"}),+ ],+)+def test_ambiguous_or_conflicting_v2_is_rejected(mutate: Callable[[dict], object]) -> None:+ """新协议损坏、未知版本及保留字段冲突不能被历史兼容掩盖。"""+ payload = _legacy_v2()+ mutate(payload)+ with pytest.raises(ValueError):+ read_cached_middle_json(payload)+++ "payload",+ [+ {"schema_version": "1.0", "pages": [{"page_idx": 0, "blocks": []}]},+ {"pdf_info": [{"page_idx": 0, "blocks": []}]},+ {"pdf_info": [], "pages": []},+ {"pdf_info": [], "schema_version": "99.0"},+ {"pages": []},+ ],+)+def test_mixed_pages_are_not_silently_emptied(payload: dict) -> None:+ """拒绝把当前 blocks 误当作旧页面后悄悄丢弃正文。"""+ with pytest.raises(ValueError):+ read_cached_middle_json(payload)++[email protected]("indices", [[3, 3], [5, 3], [-1, 3], [True, 3]])+def test_invalid_legacy_page_indices_fail(indices: list) -> None:+ """旧页号仍受唯一、顺序、非负和整数约束。"""+ payload = _legacy_345()+ for page, index in zip(payload["pdf_info"], indices):+ page["page_idx"] = index+ with pytest.raises(ValueError):+ read_cached_middle_json(payload)++[email protected]("family", ["345", "v1", "v2"])+def test_compatibility_is_not_exposed_by_generic_readers(family: str) -> None:+ """Doclib 兼容成功不改变通用 ParseResult 或 DocVortex 的严格边界。"""+ payload = _legacy_v2() if family == "v2" else _legacy_345(wrapped=family == "v1")+ assert read_cached_middle_json(payload).pages+ for reader in [ParseResult.from_dict, MiddleJson.from_dict]:+ with pytest.raises(ValueError):+ reader(payload)+ with pytest.raises(ValueError):+ ParseResult.from_json(json.dumps(payload))diff --git a/tests/unittest/test_flash_pdf_render_scheduling.py b/tests/unittest/test_flash_pdf_render_scheduling.pynew file mode 100644index 00000000..4c1dceee--- /dev/null+++ b/tests/unittest/test_flash_pdf_render_scheduling.py@@ -0,0 +1,166 @@+"""验证 Flash 原生按需截图与 MinerU 配置、选页和后处理边界。"""++from __future__ import annotations++import asyncio+from pathlib import Path+from types import SimpleNamespace+from typing import Any+from unittest.mock import AsyncMock, MagicMock++import pytest+from PIL import Image+from reportlab.pdfgen.canvas import Canvas++from docvortex.analyzers.native.models import PdfModel+from docvortex.document.pdf import PDFDocument, images+from mineru.backend.analysis.pdf import window+from mineru.backend.postprocess import llm_aided+from mineru.config import LLMAidedConfig, LLMAidedFeaturesConfig, config+from mineru.parser import parse, parse_async+++def _source(path: Path) -> Path:+ """构造五页 PDF,让实际输入准备负责连续和非连续选页。"""+ canvas = Canvas(str(path))+ for index in range(5):+ canvas.drawString(60, 720, f"Page {index + 1}")+ canvas.showPage()+ canvas.save()+ return path++[email protected]("asynchronous", [False, True])[email protected]("page_range,indices", [("all", [0, 1, 2, 3, 4]), ("2-4", [1, 2, 3]), ("1,3,5", [0, 2, 4])])+def test_flash_sparse_render_keeps_page_mapping_and_title_gate(+ monkeypatch: pytest.MonkeyPatch,+ tmp_path: Path,+ asynchronous: bool,+ page_range: str,+ indices: list[int],+) -> None:+ """真实同步和异步入口只渲染有图页,页号和整本标题开关仍由 MinerU 管理。"""+ document_sizes: list[int] = []+ rendered: list[int] = []+ created: list[Image.Image] = []++ def predict(_model: PdfModel, document: PDFDocument) -> list[list[dict[str, Any]]]:+ """替换原生语义分析,每个物理页含标题,奇数物理页额外含图片。"""+ document_sizes.append(document.page_count)+ pages: list[list[dict[str, Any]]] = []+ for index in range(document.page_count):+ blocks = [+ {+ "type": "paragraph_title",+ "bbox": [0.1, 0.1, 0.9, 0.2],+ "level": 2,+ "lines": [{"bbox": [0.1, 0.1, 0.9, 0.2]}],+ "content": [{"type": "text", "content": f"Section {index}"}],+ }+ ]+ if index % 2:+ blocks.append({"type": "image", "bbox": [0.1, 0.3, 0.9, 0.8]})+ pages.append(blocks)+ return pages++ def raster(data: bytes, **options: Any) -> list[dict[str, Any]]:+ """以真实物理页索引生成颜色图片,同时检查宿主设置优先于引擎默认值。"""+ assert data.startswith(b"%PDF")+ assert options["timeout"] == 19 and options["threads"] == 2+ batch = list(range(options["start_page_id"], options["end_page_id"] + 1))+ rendered.extend(batch)+ result = []+ for index in batch:+ image = Image.new("RGB", (40, 20), (index * 30, 60, 120))+ created.append(image)+ result.append({"img_pil": image})+ return result++ monkeypatch.setenv("MINERU_PROCESSING_WINDOW_SIZE", "2")+ monkeypatch.setenv("MINERU_PDF_RENDER_TIMEOUT", "19")+ monkeypatch.setenv("MINERU_PDF_RENDER_THREADS", "2")+ monkeypatch.setenv("DOCVORTEX_PDF_RENDER_TIMEOUT", "31")+ monkeypatch.setattr(PdfModel, "predict", predict)+ monkeypatch.setattr(images, "load_images_from_pdf_bytes_range", raster)+ forbidden = MagicMock(side_effect=AssertionError("Flash TXT must skip inference windows"))+ monkeypatch.setattr(window, "_get_window_pdf_pages", forbidden)+ title = AsyncMock()+ client = SimpleNamespace(close=AsyncMock())+ monkeypatch.setattr(llm_aided, "LLMAidedClient", MagicMock(return_value=client))+ monkeypatch.setattr(llm_aided, "apply_llm_title_leveling", title)+ monkeypatch.setattr(+ config,+ "llm_aided",+ LLMAidedConfig(+ api_key="isolated-test",+ features=LLMAidedFeaturesConfig(title_leveling=True),+ ),+ )+ source = _source(tmp_path / "pages.pdf")+ options = {"tier": "flash", "ocr_mode": "txt", "image_analysis": False, "page_range": page_range}+ result = asyncio.run(parse_async(source, **options)) if asynchronous else parse(source, **options)++ full = page_range == "all"+ assert document_sizes == [len(indices)]+ assert rendered == list(range(1, len(indices), 2))+ assert [page.page_idx for page in result.pages] == indices+ assert result._model_output.page_index_map == ([] if full else indices)+ assert result.middle_json.is_full_document is full+ assert result._model_output.metadata.producer.name == "mineru"+ assert result.middle_json.extensions == result._model_output.extensions+ assert title.await_count == int(full)+ if full:+ assert title.await_args.args[0] is result.pages+ for index in rendered:+ assert result._model_output.pages[index][1]["image_base64"].startswith("data:image/jpeg;base64,")+ for image in created:+ with pytest.raises(ValueError):+ image.getpixel((0, 0))+ forbidden.assert_not_called()++[email protected]("parse_mode", ["txt", "auto"])+def test_flash_text_only_does_not_render(monkeypatch: pytest.MonkeyPatch, tmp_path: Path, parse_mode: str) -> None:+ """真实原生解析的纯文本 PDF 不创建页图,auto 仍只在入口分类一次。"""+ classify = MagicMock(return_value="txt")+ monkeypatch.setattr(PDFDocument, "classify", classify)+ forbidden = MagicMock(side_effect=AssertionError("Text-only PDF must not rasterize"))+ monkeypatch.setattr(images, "load_images_from_pdf_bytes_range", forbidden)+ result = parse(_source(tmp_path / "text.pdf"), tier="flash", ocr_mode=parse_mode, image_analysis=False)+ assert len(result.pages) == 5+ assert classify.call_count == int(parse_mode == "auto")+ forbidden.assert_not_called()+++ "values,expected",+ [(None, (64, 300, 3)), (("bad", "0", "-1"), (64, 300, 3)), (("0", "17", "2"), (1, 17, 2))],+)+def test_flash_preserves_render_config_fallbacks(+ monkeypatch: pytest.MonkeyPatch,+ values: tuple[str, str, str] | None,+ expected: tuple[int, int, int],+) -> None:+ """沿用宿主现有环境变量缺省、非法值和窗口下限规则。"""+ names = ("MINERU_PROCESSING_WINDOW_SIZE", "MINERU_PDF_RENDER_TIMEOUT", "MINERU_PDF_RENDER_THREADS")+ for index, name in enumerate(names):+ monkeypatch.delenv(name, raising=False)+ if values is not None:+ monkeypatch.setenv(name, values[index])+ pages: list[list[dict[str, Any]]] = [[]]+ monkeypatch.setattr(PdfModel, "predict", MagicMock(return_value=pages))+ attach = MagicMock()+ monkeypatch.setattr(window, "attach_visual_block_images_from_pdf", attach)+ document = MagicMock(page_count=1)+ actual = window.process_pdf_windows(+ b"pdf",+ document,+ effort="flash",+ parse_mode="txt",+ image_analysis=False,+ flash_txt_mode=True,+ hybrid_model=None,+ vlm_predictor=None,+ )+ assert actual is pages+ attach.assert_called_once_with(document, pages, window_size=expected[0], timeout=expected[1], threads=expected[2])diff --git a/tests/unittest/test_pdf_analyze_visual_blocks.py b/tests/unittest/test_pdf_analyze_visual_blocks.pyindex 92114f18..28b983e5 100644--- a/tests/unittest/test_pdf_analyze_visual_blocks.py+++ b/tests/unittest/test_pdf_analyze_visual_blocks.py@@ -1668,7 +1668,7 @@ def test_doc_analyze_flash_demo1_uses_canonical_equation_type() -> None:def test_doc_analyze_flash_returns_complete_model_json_and_typed_middle_json(monkeypatch: pytest.MonkeyPatch) -> None:- """验证 Flash 多窗口补充完整 raw pages,并返回严格 ModelJson 与 MiddleJson。"""+ """验证 Flash 按需窗口补充完整 raw pages,并返回严格 ModelJson 与 MiddleJson。"""from docvortex.analyzers import native as flash_modelevents: list[str] = []@@ -1699,6 +1699,8 @@ def test_doc_analyze_flash_returns_complete_model_json_and_typed_middle_json(mon]fake_pdf_doc = MagicMock()fake_pdf_doc.page_count = len(source_model_list)+ fake_pdf_doc.bytes = b"fake-pdf"+ fake_pdf_doc.page_size.return_value = (40, 20)fake_pdf_doc.__getitem__.side_effect = lambda page_idx: MagicMock(page_idx=page_idx)def fake_document_close() -> None:@@ -1718,11 +1720,13 @@ def test_doc_analyze_flash_returns_complete_model_json_and_typed_middle_json(monrequested_ranges: list[tuple[int, int]] = []def fake_load_images_for_window(- *,pdf_bytes: bytes,+ *,start_page_id: int,end_page_id: int,image_type: str,+ timeout: int | None = None,+ threads: int | None = None,) -> list[dict[str, Image.Image]]:"""按请求范围生成测试页图,并记录窗口以验证分段与释放行为。"""assert pdf_bytes == b"fake-pdf"@@ -1735,7 +1739,7 @@ def test_doc_analyze_flash_returns_complete_model_json_and_typed_middle_json(monrendered_images.extend(window_images)return [{"img_pil": image} for image in window_images]- original_attach_visual_block_images = visuals._attach_visual_block_images+ original_attach_visual_block_images = visuals._attach_prepared_visual_block_imagesdef tracked_attach_visual_block_images(*args: object, **kwargs: object) -> None:"""记录视觉块补图顺序并调用真实实现。"""@@ -1759,8 +1763,8 @@ def test_doc_analyze_flash_returns_complete_model_json_and_typed_middle_json(monmonkeypatch.setattr(pipeline, "PDFDocument", lambda _: fake_pdf_doc)monkeypatch.setattr(window, "_configured_window_size", lambda default: 2)- monkeypatch.setattr(window, "load_images_from_pdf_bytes_range", fake_load_images_for_window)- monkeypatch.setattr(window, "_attach_visual_block_images", tracked_attach_visual_block_images)+ monkeypatch.setattr("docvortex.document.pdf.images.load_images_from_pdf_bytes_range", fake_load_images_for_window)+ monkeypatch.setattr(visuals, "_attach_prepared_visual_block_images", tracked_attach_visual_block_images)monkeypatch.setattr(pipeline, "_normalize_pdf_model_list", tracked_normalize_model_list)monkeypatch.setattr(pipeline.time, "perf_counter", fake_perf_counter)monkeypatch.setattr(flash_model, "PdfModel", MagicMock(return_value=fake_pdf_model))@@ -1779,7 +1783,7 @@ def test_doc_analyze_flash_returns_complete_model_json_and_typed_middle_json(monassert model_json.pages == source_model_listassert model_json.page_index_map == [7, 8, 9]assert model_json.is_full_document is False- assert requested_ranges == [(0, 1), (2, 2)]+ assert requested_ranges == [(1, 1), (2, 2)]assert inline_text(model_json.pages[0][0]["content"]) == "第一页 x+y"assert [span["content"] for span in model_json.pages[0][0]["content"] if span.get("type") == "equation_inline"] == ["x+y"]assert model_json.pages[2][0]["content"] == "z"