Skip to content

Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.

Update DocVortex dependency version and enhance Middle JSON compatibility handling

myhloli332bb866cf51
diff --git a/docs/doclib-middle-json-compat-validation.md b/docs/doclib-middle-json-compat-validation.md
new file mode 100644
index 00000000..8ca94c4e
--- /dev/null
+++ b/docs/doclib-middle-json-compat-validation.md
@@ -0,0 +1,52 @@
+# Doclib 历史 Middle JSON 兼容验证
+
+日期:2026-09-07。实现基于本地 `48fdd2b66` 工作区,仅在 Doclib 持久化读取边界恢复兼容。
+使用约定见 [Doclib 历史兼容](next/middle-json/doclib-compatibility.md)。
+
+## 实现与边界
+
+- 新增 `read_cached_middle_json()`,在内存中读取当前格式、3.4.5/旧 1.0 包装及旧 MinerU Schema 2.0。
+- 页面适配器与 `dc0944984` 中的原文件逐字节一致;新构造器及元数据处理放在 Doclib 读取层。
+- 旧 Schema 2.0 只调整外层,测试明确禁止调用共享语义后处理或 LLM。
+- 历史生产版本缺失时记为 unknown,运行参数不可靠时不补造产品记录。
+- 缓存命中、覆盖、默认读取档位、FTS 重建和压缩使用同一读取入口。
+- 普通读取不改写历史 JSON 或数据库记录;兼容且元数据一致的批次可在正常压缩时写成新格式。
+- 通用 ParseResult、DocVortex codec、HTTP/ZIP 和 Gradio 没有增加旧格式读取能力。
+- 本次没有调整包版本、协议版本、解析算法或兼容有效期,也未新增兼容开关。
+
+## 验证结果
+
+使用本地 Magic-PDF `.venv1`:
+
+| 检查 | 结果 |
+| --- | --- |
+| 适配器、Doclib、通用读取、HTTP/ZIP、Gradio 专项 | 560 passed |
+| MinerU 完整测试 | 1870 passed、3 skipped、3 个既有失败 |
+| 本次修改的 Python 文件 Ruff | 通过 |
+| 本次修改的补丁空白检查 | 通过 |
+
+专项覆盖原 3.4.5 文字、公式、链接、图表、列表、目录和坐标转换;旧 2.0 页面保持;
+版本缺失、元数据冲突、未知扩展保留、输入不变和转换幂等性;以及显式错误 schema 不兜底。
+真实 SQLite/FTS5 用例覆盖两种历史协议族的缓存命中、内容读取、范围覆盖、索引重建、
+默认档位、混合协议压缩、重复页取新及冲突/失败时保留源数据。
+
+完整测试仍存在三项此前已确认的失败,均不涉及本次兼容代码:
+
+- `test_basic_extra_includes_preflight_runtime_dependencies`:既有 basic extra 缺失。
+- `test_standard_is_the_highest_model_runtime_extra`:既有 test extra 不含 mineru[standard]。
+- `test_pp_formulanet_fix_latex_uses_shared_mathring_repair`:既有 UniMERNetDecode 缺少 fix_latex。
+
+执行期间工作区还存在另外进行的 PDF 渲染、依赖、版本和测试调整,本次未修改这些文件。
+初轮完整测试额外出现 PDF 补图用例失败;后续并行更新后,最终完整测试已不再出现该失败。
+以上完整测试数字对应最终运行时的整个共享工作区,不将其他修改计入本次兼容实现。
+
+## 复跑与证据
+
+```bash
+python -m pytest -q tests/unittest/test_doclib_middle_json.py tests/unittest/test_doclib_legacy_schema_adapter.py tests/unittest/test_doclib_legacy_page_ranges.py tests/unittest/test_doclib_cache_semantics.py tests/unittest/test_legacy_schema_adapter.py tests/unittest/test_parse_result_contract.py tests/unittest/test_parser_api_contract.py tests/unittest/test_kit_gradio.py
+python -m pytest -q
+```
+
+本地日志保存在 `/tmp/doclib-middle-compat/verified-focused.log` 和
+`/tmp/doclib-middle-compat/full-final.log`。先前 JSON 2.0 直接迁移的验证报告不改写;
+此记录描述后续收敛为 Doclib 专属兼容边界的行为。
diff --git a/docs/docvortex-upgrade-validation.md b/docs/docvortex-upgrade-validation.md
index 23954dd1..97237e41 100644
--- a/docs/docvortex-upgrade-validation.md
+++ b/docs/docvortex-upgrade-validation.md
@@ -1,6 +1,6 @@
# DocVortex 接入与标点规则验证
-独立引擎已更名为 DocVortex,MinerU 依赖为 `docvortex>=0.1.0,<1.0.0`。
+独立引擎已更名为 DocVortex,MinerU 依赖为 `docvortex>=0.2.1,<0.3.0`。
导入及集成模块同步迁移,公开文档类型继续由引擎唯一维护。MinerU 的 schema 2.0、
产品元数据、已有旧结果适配、OCR/VLM/Hybrid 路由与 UI 名称保持不变。
diff --git a/docs/en/usage/cli_tools.md b/docs/en/usage/cli_tools.md
index 7edf6467..8183f271 100644
--- a/docs/en/usage/cli_tools.md
+++ b/docs/en/usage/cli_tools.md
@@ -69,7 +69,7 @@ mineru-kit vlm-server --engine auto --port 30000
- `MINERU_LOCAL_API_STARTUP_TIMEOUT_SECONDS`: startup timeout for the Gradio-managed local V1 server; default `300` seconds.
- `MINERU_API_ENABLE_FASTAPI_DOCS`: enable `/docs`, `/openapi.json`, and `/redoc` on the V1 API server; default `true`.
- `MINERU_PDF_RENDER_TIMEOUT` / `MINERU_PDF_RENDER_THREADS`: PDF rendering timeout and worker count.
-- `MINERU_PROCESSING_WINDOW_SIZE`: processing window size used for large documents.
+- `MINERU_PROCESSING_WINDOW_SIZE`: processing window size used for large documents. Flash native text parsing renders only pages with visual blocks, preserving this window boundary, rendering timeout, and worker settings. A 32 MiB pixel budget can split batches further. OCR and other tiers still render entire windows.
- `MINERU_INTRA_OP_NUM_THREADS` / `MINERU_INTER_OP_NUM_THREADS`: ONNX operator thread settings.
Prefer each command's `--help` output and [model source documentation](./model_source.md) for current defaults.
diff --git a/docs/next/code-map.md b/docs/next/code-map.md
index e063046f..53bc44c2 100644
--- a/docs/next/code-map.md
+++ b/docs/next/code-map.md
@@ -26,7 +26,7 @@
| `mineru/doclib/` | 本地文档库常驻服务:入库、SHA256 去重、解析调度、缓存、搜索、配置。FastAPI + SQLite(WAL) + FTS5,路由→`services/`→`core/` 三层 | `doclib/server.py`(2439 行);`doclib/services/{parse,search,scan,config,cleanup}_svc.py`;`doclib/core/{db,fts,file_io}.py` |
| `mineru/doclib/background/` | 后台 asyncio 任务:ingest(入库)、parse_worker(解析执行)、scan_worker / watch(目录监听)、parse_server_health(parse-server 托管与健康探测)、device_monitor、compaction(批次合并)、telemetry_flush | 各文件顶部 docstring;`background/parse_worker.py:15` |
| `mineru/parser/` | parse-server 与解析门面:`api_server.py`(2495 行,独立进程)、`api_client.py`(HTTP 客户端)、`tier.py`(tier/backend 解析)、`mineru_parser.py`(`MinerUParser` 统一解析入口)、`base.py`(Middle JSON schema) | `parser/mineru_parser.py`、`parser/tier.py` |
-| `mineru/backend/` | 解析引擎:`analyze.py`(`doc_analyze` 统一门面)→ `analysis/`(按格式分派:`pdf/`、`csv.py`、`epub.py`、`html.py`、`ofd.py`、`office.py`)→ `postprocess/`(Middle JSON 组装、段落/标题/列表/表格合并、`llm_aided.py`) | `backend/analyze.py`、`backend/analysis/pdf/pipeline.py` |
+| `mineru/backend/` | 解析引擎:`analyze.py`(`doc_analyze` 统一门面)→ `analysis/`(按格式分派:`pdf/`、`csv.py`、`epub.py`、`html.py`、`ofd.py`、`office.py`)→ `postprocess/`(Middle JSON 组装、段落/标题/列表/表格合并、`llm_aided.py`、仅用于 Doclib 的 `legacy_schema_adapter.py`) | `backend/analyze.py`、`backend/analysis/pdf/pipeline.py` |
| `mineru/model/` | 模型层:`flash/`(纯 CPU 本地结构化解析器族:pdf / office / epub / html / ofd / csv);`runtime/`(`device.py` 设备与 light/full 栈、`hybrid.py` 原子模型分派、`onnx.py`);`layout/ mfr/ ocr/ table/ vlm/`(ML 模型实现);`_internal/pytorchocr` | `model/runtime/device.py:48`、`model/runtime/hybrid.py` |
| `mineru/render/` | 输出渲染:markdown / content_list(v2) / structured_content / docx / epub / html / pdf | `render/api.py`(唯一渲染入口)、`render/contracts.py` |
| `mineru/kit/` | 服务端与 Web 工具 CLI:`parse` / `gradio` / `api-server` / `vlm-server` / `router` / `models`,面向无状态批处理、自部署和多卡部署 | `kit/main.py`、`kit/commands/*.py`、`kit/gradio/` |
diff --git a/docs/next/invariants.md b/docs/next/invariants.md
index 268e0abc..b46d6c87 100644
--- a/docs/next/invariants.md
+++ b/docs/next/invariants.md
@@ -32,7 +32,7 @@ Middle JSON 的对外 schema 固定为 `2.0`(`mineru/parser/base.py:20` `MIDDL
红线:
-- 不接受旧 `_backend` / `_version_name` / `pdf_info` 等历史文档字段;加载统一验证 schema 身份、版本和 metadata,不执行旧协议迁移。
+- 不接受旧 `_backend` / `_version_name` / `pdf_info` 等历史文档字段;通用加载统一验证 schema 身份、版本和 metadata。仅 Doclib 的持久化读取边界可转换受支持的历史 MinerU Middle JSON。
- 改动输出结构即破坏性变更:必须走 ADR 流程,并同步 `docs/next/api/changes.md`、`docs/next/middle-json.md` 与 compaction(`doclib/background/compaction.py:27`)。
## 4. 依赖方向:基础层不得反向引用上层
diff --git a/docs/next/middle-json.md b/docs/next/middle-json.md
index 817a0b84..529b9f8e 100644
--- a/docs/next/middle-json.md
+++ b/docs/next/middle-json.md
@@ -27,7 +27,7 @@ Middle JSON 已经收敛为 schema 2.0 的严格 Pydantic 模型:`docvortex.sc
- 以当前代码中的 Pydantic 严格模型为起点(schema 2.0)。
- 区分“已经解决”、“部分解决”、“仍需工作”。
- Agent 引用与稳定 page/block locator 是 P0 目标。
-- 统一使用带 `schema` 身份的 2.0 协议及 `metadata`;所有历史文档协议均明确拒绝,旧缓存须重新解析。
+- 统一使用带 `schema` 身份的 2.0 协议及 `metadata`;通用接口拒绝历史协议;仅 Doclib 持久化读取兼容旧 MinerU Middle JSON,见 [缓存兼容说明](middle-json/doclib-compatibility.md)。
- render 统一不能只看入口 facade,还要收敛格式-specific 分支。
## 与其他文档的关系
diff --git a/docs/next/middle-json/README.md b/docs/next/middle-json/README.md
index 5a77ccba..fa0b02ad 100644
--- a/docs/next/middle-json/README.md
+++ b/docs/next/middle-json/README.md
@@ -34,7 +34,7 @@ InlineContentBlock
Middle JSON 下一版要达到以下目标:
-1. 统一使用带协议身份和 metadata 的 JSON 2.0;所有历史 JSON 必须重新解析,见 [envelope.md](envelope.md)。
+1. 统一使用带协议身份和 metadata 的 JSON 2.0;通用接口严格读取,Doclib 历史数据经独立缓存边界转换,见 [envelope.md](envelope.md)。
2. 明确 `PageInfo` / `Block` / `InlineSpan` 字段契约。
3. 为每个 backend 提供 normalization 任务清单。
4. 定义 Agent 可引用的稳定 locator 规则。
diff --git a/docs/next/middle-json/current-medium.md b/docs/next/middle-json/current-medium.md
index fb049e3a..e012e9ad 100644
--- a/docs/next/middle-json/current-medium.md
+++ b/docs/next/middle-json/current-medium.md
@@ -20,7 +20,7 @@
`schema_version="2.0"`。
读取必须匹配 `schema` 与 `schema_version`,详见[统一外层协议](envelope.md)。
-所有历史文档协议和缺少身份的 JSON 均拒绝,不提供自动迁移。
+通用接口拒绝历史文档协议和缺少身份的 JSON;Doclib 通过独立缓存读取器兼容旧 MinerU 数据。
## ModelJson
@@ -150,5 +150,5 @@ Markdown、HTML、DOCX 与 Structured Content renderer 按 Span discriminator
## 缓存与协议
-只接受当前完整文档协议;旧结果不命中缓存、不参与压缩,并提示从源文件重建。
+通用接口只接受当前完整文档协议。Doclib 兼容转换 3.4.5 和旧 MinerU 2.0 缓存,转换成功可命中和参与压缩;未知或损坏结果仍要求重新解析。
压缩不得合并不同来源、扩展或整本标识的数据。Block/Span 类型和内部语义保持不变。
diff --git a/docs/next/middle-json/doclib-compatibility.md b/docs/next/middle-json/doclib-compatibility.md
new file mode 100644
index 00000000..59f1a65b
--- /dev/null
+++ b/docs/next/middle-json/doclib-compatibility.md
@@ -0,0 +1,49 @@
+# Doclib 历史 Middle JSON 兼容
+
+兼容仅用于 Doclib 已持久化的 Middle JSON。新写出的文档仍采用
+`docvortex.middle` 及当前共享协议,不恢复旧字段或旧格式导出。
+通用 ParseResult、DocVortex codec、HTTP/ZIP 与 Gradio 继续严格读取新协议。
+
+## 唯一读取边界
+
+`mineru.doclib.core.middle_json.read_cached_middle_json(payload)` 返回当前
+`MiddleJson`。缓存读取、缓存命中、覆盖范围、默认读取档位、FTS 重建与压缩
+统一使用此入口。转换只发生在内存,不改写原文件、数据库记录或历史文件名。
+
+读取优先级如下:
+
+1. 只要存在 `schema` 键,就使用当前严格读取器;校验失败不尝试旧分支。
+2. 无 `schema`、无有效版本标识的 `pdf_info` 进入 3.4.5 分支。
+3. 无 `schema` 的 `schema_version="1.0"` pages 包装,仅在页面包含旧字段时进入同一分支。
+4. 无 `schema/metadata` 的旧 `schema_version="2.0"`,只转换文档外层。
+5. 混合页面结构、未知身份或版本、字段冲突和损坏数据明确失败。
+
+3.4.5 的发行标签内部实际报告 3.4.4,不能用版本字符串是否等于 3.4.5 识别格式。
+对应页面适配器恢复自 `dc0944984`,保持原有文本、公式、链接、图表、列表、目录和坐标转换算法。
+转换后的 raw ModelJson 只经过共享确定性后处理,不重新读取源文件或调用 OCR/VLM/LLM。
+
+旧 Schema 2.0 已有当前页面树,直接保留 pages 与必需的 is_full_document,
+不重新执行段落、表格或标题后处理。未知根字段不能静默丢弃。
+
+## 来源和产品记录
+
+- 保留历史版本;缺失或不可辨认时记录 `mineru / unknown`,不填当前运行版本。
+- 缺失 file_suffix 使用历史 PDF 默认值;显式不支持的格式报错。
+- 仅当档位和模式都可可靠识别时写入 extensions.mineru;旧 _ocr_enable 布尔值可恢复 txt/ocr。
+- 缺失或未知的解析记录不以 basic/txt 代填,也不借数据库存储档位推断实际运行档位。
+- 应用扩展原样保留;已存在的产品扩展必须合法并与可识别的历史记录一致。
+
+## 缓存与压缩
+
+成功转换且覆盖记录页范围的旧批次可命中缓存,避免重复解析。读取旧半角波浪号范围的
+记录时仍按原文件名定位,公开返回的页范围继续使用当前语法。
+
+压缩先转换各批次,再比较 metadata、extensions 和 is_full_document。
+一致的不同协议批次可以合并,重复页保留最新结果;新文件只输出当前协议。
+冲突、损坏或无法识别时跳过压缩并保留源数据。普通读取不会触发回写。
+
+## 后续移除
+
+两个历史分支及专属测试集中维护,尚未确定移除版本,不提供配置开关或自动过期时间。
+未来统一移除时,删除 Doclib 读取器中的两个旧分支和专用适配器即可,
+无需改变通用解析 API。已有验证报告保持历史事实,不追溯修改其测试结论。
diff --git a/docs/next/middle-json/envelope.md b/docs/next/middle-json/envelope.md
index 4e771aa2..d6ef8e81 100644
--- a/docs/next/middle-json/envelope.md
+++ b/docs/next/middle-json/envelope.md
@@ -46,13 +46,12 @@ Structured Content 迁移 metadata/extensions,但不带 Model/Middle 协议标
Content List V1/V2 内容契约不变。
HTTP 请求、输出文件名和 ZIP 布局保持不变;ZIP 内文档必须使用完整新版协议。
-## 直接迁移与缓存
+## 严格协议与 Doclib 历史缓存
-读取联合检查 schema 和 schema_version;不接受旧 DocVortex 1.0、缺少 schema 的
-MinerU 2.0、旧 1.0 pages 或 pdf_info。不猜测、不自动转换,也不提供旧格式写出。
-需要从源文件重新解析。Bundle 使用 docvortex.bundle 2.0 和原有素材摘要清单。
+通用 ParseResult、DocVortex codec、HTTP/ZIP 和 Gradio 仍联合校验 schema 与版本,
+不接受历史协议,不提供旧格式写出。Bundle 继续使用 docvortex.bundle 2.0。
-Doclib 仅将协议有效且覆盖记录页范围的批次视为可用缓存。旧缓存不计入缓存命中、
-可用页范围和默认读取档位。请求解析时重新生成缺失结果;直接读取旧页返回重新解析提示。
-压缩仅合并 metadata、extensions、is_full_document 一致的新批次,重复页仍取最新值;
-遇到旧格式、损坏数据或冲突时跳过压缩并保留源数据。
+Doclib 的持久化读取边界单独兼容 3.4.5(含旧 1.0 pages 包装)和旧 MinerU Schema 2.0,
+在内存中转换为当前 MiddleJson。兼容成功的批次可用于读取、缓存命中、覆盖范围和 FTS。
+压缩先转换并比较来源、扩展及整本标识,一致才合并并写出新协议;损坏或冲突时保留源数据。
+详见 [Doclib 历史兼容](doclib-compatibility.md)。普通读取不修改历史文件和数据库记录。
diff --git a/docs/next/middle-json/migration.md b/docs/next/middle-json/migration.md
index 50acb075..f894e3a3 100644
--- a/docs/next/middle-json/migration.md
+++ b/docs/next/middle-json/migration.md
@@ -1,13 +1,14 @@
# JSON 2.0 直接迁移
-采用[统一外层协议](envelope.md),不保留旧格式转换入口。
+采用[统一外层协议](envelope.md),通用解析接口保持严格;仅 Doclib 持久化读取提供[历史转换](doclib-compatibility.md)。
-1. 升级 DocVortex 至 0.2.x,MinerU 要求 `docvortex>=0.2.0,<0.3.0`。
+1. 升级 DocVortex 至 0.2.x,MinerU 要求 `docvortex>=0.2.1,<0.3.0`。
2. 将 Model/Middle 构造器的 file_suffix、producer 放入 DocumentMetadata;
Python 访问同步改为 metadata.file_suffix、metadata.producer。
3. JSON 使用完整 schema/schema_version/metadata/extensions;两边统一 from_dict/from_json。
4. 产品记录只输出实际 tier 与 txt/ocr;删除旧 effort、顶层 mineru_version 和协议适配调用。
-5. 历史 JSON、结果包及旧缓存需从源文件重建;加载不会改写旧文件。
+5. Doclib 中可识别的历史 Middle JSON 在读取时自动转换,无需重新解析或改写旧文件。
+ DocVortex 旧原生协议、旧结果包和通用接口收到的历史 JSON 仍需重新生成。
6. 同步升级自建 HTTP 服务与客户端。旧远端协议会明确拒绝,不尝试迁移。
验证协议 Schema、跨入口往返、真实生产者与扩展保留、旧协议拒绝、ZIP 和素材,
diff --git a/docs/next/pitfalls.md b/docs/next/pitfalls.md
index eca6ad68..3cb09e7c 100644
--- a/docs/next/pitfalls.md
+++ b/docs/next/pitfalls.md
@@ -19,8 +19,8 @@
### 1.2 共享文档协议与缓存必须同步
改外层结构时同步 DocVortex schema/readers、MinerU ParseResult 和 Doclib 缓存。
-当前使用 schema 身份 + 版本 2.0 + metadata,旧协议全部拒绝;不能仅按版本号识别。
-缓存命中、覆盖与默认读取档位均需检查新版协议;压缩只合并元数据一致的新批次。
+当前使用 schema 身份 + 版本 2.0 + metadata,通用接口拒绝旧协议;不能仅按版本号识别。
+Doclib 缓存命中、覆盖、默认档位与压缩均使用专属 read_cached_middle_json,兼容转换旧 3.4.5 和 MinerU 2.0;压缩只合并转换后元数据一致的批次。
详见 [envelope.md](middle-json/envelope.md)。
## 2. Alpha 高频迭代,公开 API 未稳定
diff --git a/docs/zh/usage/cli_tools.md b/docs/zh/usage/cli_tools.md
index 9e141259..2edb68a7 100644
--- a/docs/zh/usage/cli_tools.md
+++ b/docs/zh/usage/cli_tools.md
@@ -69,7 +69,7 @@ mineru-kit vlm-server --engine auto --port 30000
- `MINERU_LOCAL_API_STARTUP_TIMEOUT_SECONDS`:Gradio 托管本地 V1 服务的启动超时,默认 `300` 秒。
- `MINERU_API_ENABLE_FASTAPI_DOCS`:是否为 V1 API Server 启用 `/docs`、`/openapi.json` 和 `/redoc`,默认 `true`。
- `MINERU_PDF_RENDER_TIMEOUT` / `MINERU_PDF_RENDER_THREADS`:PDF 渲染超时和 worker 数量。
-- `MINERU_PROCESSING_WINDOW_SIZE`:大文档处理窗口大小。
+- `MINERU_PROCESSING_WINDOW_SIZE`:大文档处理窗口大小。Flash 原生文本路径仅渲染有视觉块的页面,仍遵守该窗口边界、渲染超时和 worker 配置,并按 32 MiB 像素预算进一步分批。OCR 和其他档位保持整窗口渲染。
- `MINERU_INTRA_OP_NUM_THREADS` / `MINERU_INTER_OP_NUM_THREADS`:ONNX 算子线程配置。
当前默认值以各命令的 `--help` 和[模型源说明](./model_source.md)为准。
diff --git a/mineru/backend/analysis/pdf/window.py b/mineru/backend/analysis/pdf/window.py
index 561cf1b8..5923ac43 100644
--- a/mineru/backend/analysis/pdf/window.py
+++ b/mineru/backend/analysis/pdf/window.py
@@ -13,7 +13,7 @@ from ....model.runtime.hybrid import HybridLocalModelContext
from docvortex.document.pdf.document import PDFDocument
from docvortex.document.pdf.document import PDFPage
from docvortex.document.pdf.document import PDFPageTextGeometry
-from .images import load_images_from_pdf_bytes_range
+from .images import get_load_images_threads, get_load_images_timeout, load_images_from_pdf_bytes_range
from ..contracts import AnalyzeEffort
from .constants import (
@@ -53,7 +53,10 @@ from .tables import (
_split_native_high_table_blocks,
)
from docvortex.document.pdf.visuals import attach_visual_block_images as _attach_visual_block_images
-from docvortex.document.pdf.visuals import supplement_missing_image_block_containers as _supplement_missing_image_block_containers
+from docvortex.document.pdf.visuals import attach_visual_block_images_from_pdf
+from docvortex.document.pdf.visuals import (
+ supplement_missing_image_block_containers as _supplement_missing_image_block_containers,
+)
from .text.content import (
_fill_window_block_content_and_lines,
_validate_text_formula_window_inputs,
@@ -294,10 +297,18 @@ def process_pdf_windows(
page_count = document.page_count
model_list: list[list[dict[str, Any]]] = []
if flash_txt_mode:
- # Flash 先对整份 PDF 生成完整 model_list,不依赖页面渲染和处理窗口。
+ # Flash 原生结果只按视觉块需求补图,不再进入供推理使用的全页渲染窗口。
from docvortex.analyzers.native import PdfModel
model_list = PdfModel().predict(document)
+ attach_visual_block_images_from_pdf(
+ document,
+ model_list,
+ window_size=_configured_window_size(default=64),
+ timeout=get_load_images_timeout(),
+ threads=get_load_images_threads(),
+ )
+ return model_list
configured_window_size = _configured_window_size(default=64)
windows = _build_processing_windows(page_count, configured_window_size)
@@ -318,159 +329,154 @@ def process_pdf_windows(
_log_processing_window(window, page_count, len(images_pil_list))
page_text_geometries: list[PDFPageTextGeometry | None] | None = (
[None] * len(window_pages)
- if not flash_txt_mode and parse_mode == "txt" and effort in {"medium", "high", "xhigh"}
+ if parse_mode == "txt" and effort in {"medium", "high", "xhigh"}
else None
)
- if flash_txt_mode:
- # Flash 仅切割当前窗口的外层列表,用于页图释放前原地补充视觉块裁图。
- window_model_list = model_list[window.start : window.end + 1]
- else:
- local_model_context = hybrid_model
- if local_model_context is None:
- raise ValueError("Hybrid local model context is required outside Flash TXT mode")
- np_images = [np.asarray(pil_image).copy() for pil_image in images_pil_list]
- images_layout_res = local_model_context.layout_model.batch_predict(
- images_pil_list, batch_size=min(8, BATCH_RATIO * LAYOUT_BASE_BATCH_SIZE)
- )
+ local_model_context = hybrid_model
+ if local_model_context is None:
+ raise ValueError("Hybrid local model context is required outside Flash TXT mode")
+ np_images = [np.asarray(pil_image).copy() for pil_image in images_pil_list]
+ images_layout_res = local_model_context.layout_model.batch_predict(
+ images_pil_list, batch_size=min(8, BATCH_RATIO * LAYOUT_BASE_BATCH_SIZE)
+ )
- # 使用小模型layout时对layout的表格做旋转检测
- if effort in ["flash", "medium", "high"]:
- table_items = _collect_table_items(images_layout_res, np_images)
- if table_items:
- _apply_table_orientations(
- table_items,
- parse_mode,
- window_pages,
- images_list,
- local_model_context,
- )
-
- vl_style_layout_blocks = _build_vl_style_layout_blocks(images_layout_res, images_pil_list)
-
- if parse_mode == "txt" and effort in {"medium", "high"}:
- native_table_summary = _apply_native_txt_table_priority(
- vl_style_layout_blocks,
- images_layout_res,
+ # 使用小模型layout时对layout的表格做旋转检测
+ if effort in ["flash", "medium", "high"]:
+ table_items = _collect_table_items(images_layout_res, np_images)
+ if table_items:
+ _apply_table_orientations(
+ table_items,
+ parse_mode,
window_pages,
images_list,
- effort=effort,
- page_text_geometries=page_text_geometries,
+ local_model_context,
)
- if native_table_summary.total:
- native_table_stats = {
- "effort": effort,
- "total": native_table_summary.total,
- "accepted": native_table_summary.accepted,
- "complex_fallbacks": native_table_summary.complex_fallbacks,
- "rejected": native_table_summary.rejected,
- "errors": native_table_summary.errors,
- "removed_internal_text": native_table_summary.removed_internal_text,
- "removed_formula_blocks": native_table_summary.removed_formula_blocks,
- "removed_formula_layout_items": native_table_summary.removed_formula_layout_items,
- }
- logger.bind(native_table_priority=native_table_stats).info(
- "Hybrid native table priority. "
- f"effort={native_table_stats['effort']}, total={native_table_stats['total']}, "
- f"accepted={native_table_stats['accepted']}, "
- f"complex_fallbacks={native_table_stats['complex_fallbacks']}, "
- f"rejected={native_table_stats['rejected']}, "
- f"errors={native_table_stats['errors']}, "
- f"removed_internal_text={native_table_stats['removed_internal_text']}, "
- f"removed_formula_blocks={native_table_stats['removed_formula_blocks']}, "
- f"removed_formula_layout_items={native_table_stats['removed_formula_layout_items']}"
- )
-
- if parse_mode == "txt":
- if effort == "medium":
- window_model_list = vl_style_layout_blocks
- elif effort == "high":
- high_vlm_blocks, accepted_native_tables = _split_native_high_table_blocks(vl_style_layout_blocks)
- high_vlm_results = vlm_predictor.batch_extract_with_layout(
- images=images_pil_list,
- blocks_list=high_vlm_blocks,
- not_extract_list=NOT_EXTRACT_TYPES,
- image_analysis=False,
- )
- window_model_list = _restore_native_high_table_blocks(
- high_vlm_results,
- accepted_native_tables,
- )
- elif effort == "xhigh":
- window_model_list = vlm_predictor.batch_two_step_extract(
- images=images_pil_list,
- not_extract_list=NOT_EXTRACT_TYPES,
- image_analysis=image_analysis,
- )
- else:
- raise ValueError(f"Unsupported analyze effort: {effort}")
- elif parse_mode == "ocr":
- if effort in ["flash", "medium"]:
- window_model_list = vl_style_layout_blocks
- elif effort == "high":
- window_model_list = vlm_predictor.batch_extract_with_layout(
- images=images_pil_list,
- blocks_list=vl_style_layout_blocks,
- image_analysis=False,
- )
- elif effort == "xhigh":
- window_model_list = vlm_predictor.batch_two_step_extract(
- images=images_pil_list,
- image_analysis=image_analysis,
- )
- else:
- raise ValueError(f"Unsupported analyze effort: {effort}")
- else:
- raise ValueError(f"Unsupported parse mode: {parse_mode}")
-
- if effort in {"high", "xhigh"}:
- window_model_list = _convert_vlm_results_to_model_list(window_model_list)
- if effort == "xhigh":
- _normalize_xhigh_vlm_blocks(window_model_list)
- _apply_layout_title_split(
- window_model_list,
- images_layout_res,
- [_normalize_page_size(image) for image in images_pil_list],
+
+ vl_style_layout_blocks = _build_vl_style_layout_blocks(images_layout_res, images_pil_list)
+
+ if parse_mode == "txt" and effort in {"medium", "high"}:
+ native_table_summary = _apply_native_txt_table_priority(
+ vl_style_layout_blocks,
+ images_layout_res,
+ window_pages,
+ images_list,
+ effort=effort,
+ page_text_geometries=page_text_geometries,
+ )
+ if native_table_summary.total:
+ native_table_stats = {
+ "effort": effort,
+ "total": native_table_summary.total,
+ "accepted": native_table_summary.accepted,
+ "complex_fallbacks": native_table_summary.complex_fallbacks,
+ "rejected": native_table_summary.rejected,
+ "errors": native_table_summary.errors,
+ "removed_internal_text": native_table_summary.removed_internal_text,
+ "removed_formula_blocks": native_table_summary.removed_formula_blocks,
+ "removed_formula_layout_items": native_table_summary.removed_formula_layout_items,
+ }
+ logger.bind(native_table_priority=native_table_stats).info(
+ "Hybrid native table priority. "
+ f"effort={native_table_stats['effort']}, total={native_table_stats['total']}, "
+ f"accepted={native_table_stats['accepted']}, "
+ f"complex_fallbacks={native_table_stats['complex_fallbacks']}, "
+ f"rejected={native_table_stats['rejected']}, "
+ f"errors={native_table_stats['errors']}, "
+ f"removed_internal_text={native_table_stats['removed_internal_text']}, "
+ f"removed_formula_blocks={native_table_stats['removed_formula_blocks']}, "
+ f"removed_formula_layout_items={native_table_stats['removed_formula_layout_items']}"
)
- if effort == "flash":
- window_model_list = _process_flash_ocr(
- images_list,
- window_pages,
- window_model_list,
- local_model_context,
- images_layout_res,
+ if parse_mode == "txt":
+ if effort == "medium":
+ window_model_list = vl_style_layout_blocks
+ elif effort == "high":
+ high_vlm_blocks, accepted_native_tables = _split_native_high_table_blocks(vl_style_layout_blocks)
+ high_vlm_results = vlm_predictor.batch_extract_with_layout(
+ images=images_pil_list,
+ blocks_list=high_vlm_blocks,
+ not_extract_list=NOT_EXTRACT_TYPES,
+ image_analysis=False,
+ )
+ window_model_list = _restore_native_high_table_blocks(
+ high_vlm_results,
+ accepted_native_tables,
+ )
+ elif effort == "xhigh":
+ window_model_list = vlm_predictor.batch_two_step_extract(
+ images=images_pil_list,
+ not_extract_list=NOT_EXTRACT_TYPES,
+ image_analysis=image_analysis,
)
- # Flash OCR 在表内对象清理后复用统一公式编号合并,确保视觉裁图包含编号区域。
- for page_model_list in window_model_list:
- page_model_list[:] = optimize_hybrid_formula_number_blocks(page_model_list)
else:
- window_model_list = _process_text_and_formulas(
- images_list,
- window_pages,
- window_model_list,
- parse_mode,
- effort,
- local_model_context,
- images_layout_res,
- page_text_geometries,
+ raise ValueError(f"Unsupported analyze effort: {effort}")
+ elif parse_mode == "ocr":
+ if effort in ["flash", "medium"]:
+ window_model_list = vl_style_layout_blocks
+ elif effort == "high":
+ window_model_list = vlm_predictor.batch_extract_with_layout(
+ images=images_pil_list,
+ blocks_list=vl_style_layout_blocks,
+ image_analysis=False,
)
-
- if effort in {"medium", "high"}:
- _apply_seal_ocr(local_model_context, window_model_list, np_images)
elif effort == "xhigh":
- _supplement_missing_image_block_containers(
- window_model_list,
- vl_style_layout_blocks,
+ window_model_list = vlm_predictor.batch_two_step_extract(
+ images=images_pil_list,
+ image_analysis=image_analysis,
)
+ else:
+ raise ValueError(f"Unsupported analyze effort: {effort}")
+ else:
+ raise ValueError(f"Unsupported parse mode: {parse_mode}")
+
+ if effort in {"high", "xhigh"}:
+ window_model_list = _convert_vlm_results_to_model_list(window_model_list)
+ if effort == "xhigh":
+ _normalize_xhigh_vlm_blocks(window_model_list)
+ _apply_layout_title_split(
+ window_model_list,
+ images_layout_res,
+ [_normalize_page_size(image) for image in images_pil_list],
+ )
+
+ if effort == "flash":
+ window_model_list = _process_flash_ocr(
+ images_list,
+ window_pages,
+ window_model_list,
+ local_model_context,
+ images_layout_res,
+ )
+ # Flash OCR 在表内对象清理后复用统一公式编号合并,确保视觉裁图包含编号区域。
+ for page_model_list in window_model_list:
+ page_model_list[:] = optimize_hybrid_formula_number_blocks(page_model_list)
+ else:
+ window_model_list = _process_text_and_formulas(
+ images_list,
+ window_pages,
+ window_model_list,
+ parse_mode,
+ effort,
+ local_model_context,
+ images_layout_res,
+ page_text_geometries,
+ )
+
+ if effort in {"medium", "high"}:
+ _apply_seal_ocr(local_model_context, window_model_list, np_images)
+ elif effort == "xhigh":
+ _supplement_missing_image_block_containers(
+ window_model_list,
+ vl_style_layout_blocks,
+ )
_attach_visual_block_images(
window_model_list,
images_list,
page_start_index=window.start,
)
- if not flash_txt_mode:
- model_list.extend(window_model_list)
+ model_list.extend(window_model_list)
finally:
_close_images(images_list)
diff --git a/mineru/backend/postprocess/legacy_schema_adapter.py b/mineru/backend/postprocess/legacy_schema_adapter.py
new file mode 100644
index 00000000..d6150f05
--- /dev/null
+++ b/mineru/backend/postprocess/legacy_schema_adapter.py
@@ -0,0 +1,727 @@
+# Copyright (c) Opendatalab. All rights reserved.
+"""MinerU 3.4.5 Middle JSON page 到当前 raw ModelJson 的适配器。
+
+3.4.5 的页面仍使用 ``preproc_blocks``/``para_blocks``、``lines`` 和
+``spans``,自然语言行内语义则通过旧字符串标签或旧 span 字段表达。本模块只负责
+把单页回推成当前 ``model_json_to_pages`` 能消费的 raw block 列表;文档 envelope
+及当前 MiddleJson 的元数据由调用方负责。
+"""
+
+from __future__ import annotations
+
+import html
+import math
+import re
+from typing import Any
+
+from docvortex.schema import INLINE_STYLE_ORDER, InlineSpan, parse_inline_spans
+from docvortex.foundation.hyperlink import OFFICE_EXTERNAL_HYPERLINK_SCHEMES, sanitize_hyperlink_target
+from docvortex.content.inline import join_inline_spans
+
+# 这些父块在 3.4.5 中通过 ``blocks`` 保存 body/caption/footnote;当前 raw
+# ModelJson 则要求先展平成同级 block,再由统一后处理重新分组。
+_VISUAL_PARENT_TYPES: frozenset[str] = frozenset(
+ {
+ "image",
+ "table",
+ "chart",
+ "code",
+ }
+)
+
+_INLINE_CONTENT_BLOCK_TYPES: frozenset[str] = frozenset(
+ {
+ "text",
+ "ref_text",
+ "doc_title",
+ "paragraph_title",
+ "header",
+ "footer",
+ "page_number",
+ "aside_text",
+ "page_footnote",
+ "image_caption",
+ "image_footnote",
+ "table_caption",
+ "table_footnote",
+ "chart_caption",
+ "chart_footnote",
+ "code_caption",
+ "code_footnote",
+ "caption",
+ "footnote",
+ "phonetic",
+ }
+)
+
+_STRING_CONTENT_BLOCK_TYPES: frozenset[str] = frozenset(
+ {
+ "equation",
+ "image_body",
+ "table_body",
+ "chart_body",
+ "code_body",
+ }
+)
+
+_BLOCK_TYPE_ALIASES: dict[str, str] = {
+ "abstract": "text",
+ "phonetic": "text",
+ "interline_equation": "equation",
+}
+
+_SUPPORTED_BLOCK_TYPES: frozenset[str] = frozenset(
+ _VISUAL_PARENT_TYPES | _INLINE_CONTENT_BLOCK_TYPES | _STRING_CONTENT_BLOCK_TYPES | {"list", "index"}
+)
+
+_INLINE_START_RE = re.compile(
+ r"<(?P<tag>eq|code|text|hyperlink|sup|sub|strong|b|em|i|s|u)(?P<attrs>\s[^<>]*?)?>",
+ re.IGNORECASE,
+)
+_STYLE_ATTR_RE = re.compile(r"\bstyle\s*=\s*([\"'])(?P<style>.*?)\1", re.IGNORECASE | re.DOTALL)
+_URL_RE = re.compile(r"<url>(?P<url>.*?)</url>", re.IGNORECASE | re.DOTALL)
+_DIRECT_TAG_STYLES: dict[str, str] = {
+ "strong": "bold",
+ "b": "bold",
+ "em": "italic",
+ "i": "italic",
+ "s": "strikethrough",
+ "u": "underline",
+ "sup": "superscript",
+ "sub": "subscript",
+}
+
+# 3.4.5 PDF 的异常零面积框需要替换后才能通过当前严格 bbox 校验。
+_PLACEHOLDER_BBOX: tuple[float, float, float, float] = (0.0, 0.0, 0.001, 0.001)
+
+
+def legacy_page_to_model_list(page: dict[str, Any]) -> list[dict[str, Any]]:
+ """把 MinerU 3.4.5 的单页 Middle JSON 转换为当前 raw model-list。"""
+ if not isinstance(page, dict):
+ raise ValueError("legacy Middle JSON page must be a dict")
+
+ width, height = _parse_page_size(page.get("page_size"))
+ source_blocks = page.get("preproc_blocks")
+ if source_blocks is None:
+ source_blocks = page.get("para_blocks", [])
+ if not isinstance(source_blocks, list):
+ raise ValueError("legacy Middle JSON page blocks must be a list")
+
+ discarded_blocks = page.get("discarded_blocks", [])
+ if not isinstance(discarded_blocks, list):
+ raise ValueError("legacy Middle JSON discarded_blocks must be a list")
+
+ legacy_blocks = [*source_blocks, *discarded_blocks]
+ model_list: list[dict[str, Any]] = []
+ for group_index, block in enumerate(legacy_blocks):
+ if not isinstance(block, dict):
+ raise ValueError("legacy Middle JSON block entries must be dicts")
+ group_start = len(model_list)
+ _collect_blocks(block, width, height, model_list, inherited=None)
+ if width <= 0 or height <= 0:
+ _assign_synthetic_group_bbox(model_list[group_start:], group_index, len(legacy_blocks))
+ return model_list
+
+
+def _parse_page_size(value: Any) -> tuple[float, float]:
+ """读取合法 page_size;缺失、非数值或非正值统一返回零值。"""
+ if not isinstance(value, (list, tuple)) or len(value) < 2:
+ return 0.0, 0.0
+ width, height = value[:2]
+ if any(isinstance(item, bool) or not isinstance(item, (int, float)) for item in (width, height)):
+ return 0.0, 0.0
+ parsed_width, parsed_height = float(width), float(height)
+ if not all(math.isfinite(item) and item > 0 for item in (parsed_width, parsed_height)):
+ return 0.0, 0.0
+ return parsed_width, parsed_height
+
+
+def _assign_synthetic_group_bbox(blocks: list[dict[str, Any]], group_index: int, group_count: int) -> None:
+ """为无 page_size 文档分配互不重叠的占位框,并保持原顶层块的子项同组。"""
+ safe_count = max(group_count, 1)
+ y0 = (group_index + 0.2) / safe_count
+ y1 = (group_index + 0.8) / safe_count
+ bbox = (0.05, y0, 0.95, y1)
+ for block in blocks:
+ block["bbox"] = bbox
+ lines = block.get("lines")
+ if isinstance(lines, list):
+ for line in lines:
+ if isinstance(line, dict):
+ line["bbox"] = list(bbox)
+
+
+def _collect_blocks(
+ block: dict[str, Any],
+ width: float,
+ height: float,
+ output: list[dict[str, Any]],
+ *,
+ inherited: dict[str, Any] | None,
+) -> None:
+ """递归展平视觉父块,并把普通叶子转换为当前 raw block。"""
+ block_type = _normalize_block_type(block)
+ nested = block.get("blocks")
+ if block_type in _VISUAL_PARENT_TYPES and isinstance(nested, list) and nested:
+ child_context = _visual_child_context(block, inherited)
+ for child in nested:
+ if not isinstance(child, dict):
+ raise ValueError("legacy visual block children must be dicts")
+ _collect_blocks(child, width, height, output, inherited=child_context)
+ return
+
+ if block_type == "list" and isinstance(nested, list) and nested:
+ output.append(_convert_leaf_block(block, block_type, width, height, inherited))
+ output.extend(_flatten_nested_list_items(nested, width, height))
+ return
+ if block_type == "index" and isinstance(nested, list) and nested:
+ output.append(_flatten_nested_index(block, nested, width, height))
+ return
+ if block_type == "list" and isinstance(block.get("lines"), list) and block["lines"]:
+ output.append(_convert_leaf_block(block, block_type, width, height, inherited))
+ output.extend(_convert_pdf_list_items(block, width, height))
+ return
+
+ raw = _convert_leaf_block(block, block_type, width, height, inherited)
+ output.append(raw)
+
+
+def _visual_child_context(
+ block: dict[str, Any],
+ inherited: dict[str, Any] | None,
+) -> dict[str, Any]:
+ """收集视觉父块上需要下沉到 body 的元数据。"""
+ context = dict(inherited or {})
+ context["bbox"] = block.get("bbox", context.get("bbox"))
+ sub_type = block.get("sub_type")
+ if not isinstance(sub_type, str) or not sub_type.strip():
+ sub_type = None
+ if sub_type:
+ context["sub_type"] = sub_type
+ if isinstance(block.get("guess_lang"), str) and block["guess_lang"].strip():
+ context["guess_lang"] = block["guess_lang"].strip()
+ if "cell_merge" in block:
+ context["cell_merge"] = block.get("cell_merge")
+ return context
+
+
+def _flatten_nested_list_items(nested: list[Any], width: float, height: float) -> list[dict[str, Any]]:
+ """把 3.4.5 Office 嵌套列表叶子展平为当前 PDF list 可关联的文本块。"""
+ items: list[dict[str, Any]] = []
+ for child in nested:
+ if not isinstance(child, dict):
+ raise ValueError("legacy list children must be dicts")
+ child_type = _normalize_block_type(child)
+ child_nested = child.get("blocks")
+ if child_type == "list" and isinstance(child_nested, list):
+ items.extend(_flatten_nested_list_items(child_nested, width, height))
+ continue
+ if child_type not in {"text", "ref_text"}:
+ child_type = "text"
+ items.append(_convert_leaf_block(child, child_type, width, height, inherited=None))
+ return items
+
+
+def _flatten_nested_index(
+ block: dict[str, Any],
+ nested: list[Any],
+ width: float,
+ height: float,
+) -> dict[str, Any]:
+ """把 3.4.5 Office 嵌套目录折叠为当前 PDF index 使用的换行 Span。"""
+ contents: list[list[dict[str, Any]]] = []
+
+ def collect(children: list[Any]) -> None:
+ """按深度优先顺序收集目录叶子的结构化行内内容。"""
+ for child in children:
+ if not isinstance(child, dict):
+ raise ValueError("legacy index children must be dicts")
+ child_nested = child.get("blocks")
+ if _normalize_block_type(child) == "index" and isinstance(child_nested, list):
+ collect(child_nested)
+ continue
+ converted = _convert_leaf_block(child, "text", width, height, inherited=None)
+ content = converted.get("content")
+ if isinstance(content, list) and content:
+ contents.append([span for span in content if isinstance(span, dict)])
+
+ collect(nested)
+ joined: list[dict[str, Any]] = []
+ for content_index, content in enumerate(contents):
+ if content_index:
+ _append_text_span(joined, "\n", ())
+ _extend_span_dicts(joined, content)
+ return {
+ "type": "index",
+ "bbox": _normalize_bbox(block.get("bbox"), width, height),
+ "content": [span.model_dump(mode="json") for span in parse_inline_spans(joined)],
+ }
+
+
+def _convert_pdf_list_items(block: dict[str, Any], width: float, height: float) -> list[dict[str, Any]]:
+ """把 3.4.5 PDF list 的物理行恢复为当前 bbox 关联需要的文本项。"""
+ raw_lines = [line for line in block.get("lines", []) if isinstance(line, dict)]
+ if not raw_lines:
+ return []
+ has_item_flags = any(line.get("is_list_start_line") or line.get("is_list_end_line") for line in raw_lines)
+ line_groups: list[list[dict[str, Any]]] = []
+ current_group: list[dict[str, Any]] = []
+ for line in raw_lines:
+ if current_group and (line.get("is_list_start_line") or not has_item_flags):
+ line_groups.append(current_group)
+ current_group = []
+ current_group.append(line)
+ if line.get("is_list_end_line"):
+ line_groups.append(current_group)
+ current_group = []
+ if current_group:
+ line_groups.append(current_group)
+
+ item_type = "ref_text" if block.get("sub_type") == "ref_text" else "text"
+ items: list[dict[str, Any]] = []
+ for lines in line_groups:
+ synthetic_block = {
+ "type": item_type,
+ "bbox": _union_line_bboxes(lines, block.get("bbox")),
+ "lines": lines,
+ }
+ items.append(_convert_leaf_block(synthetic_block, item_type, width, height, inherited=None))
+ return items
+
+
+def _union_line_bboxes(lines: list[dict[str, Any]], fallback: Any) -> Any:
+ """合并一个旧列表项的物理行框,缺失时回退列表父框。"""
+ bboxes = [line.get("bbox") for line in lines]
+ valid = [
+ bbox
+ for bbox in bboxes
+ if isinstance(bbox, (list, tuple))
+ and len(bbox) == 4
+ and all(isinstance(item, (int, float)) and not isinstance(item, bool) for item in bbox)
+ ]
+ if not valid:
+ return fallback
+ return [
+ min(float(bbox[0]) for bbox in valid),
+ min(float(bbox[1]) for bbox in valid),
+ max(float(bbox[2]) for bbox in valid),
+ max(float(bbox[3]) for bbox in valid),
+ ]
+
+
+def _convert_leaf_block(
+ block: dict[str, Any],
+ block_type: str,
+ width: float,
+ height: float,
+ inherited: dict[str, Any] | None,
+) -> dict[str, Any]:
+ """把一个 3.4.5 叶子块转换为当前 raw block 字典。"""
+ inherited = inherited or {}
+ raw_bbox = block.get("bbox", inherited.get("bbox"))
+ bbox = _normalize_bbox(raw_bbox, width, height)
+ lines = block.get("lines")
+ legacy_lines = lines if isinstance(lines, list) else []
+ effective_sub_type = _effective_sub_type(block, block_type, inherited)
+
+ raw: dict[str, Any] = {"type": block_type, "bbox": bbox}
+ if _uses_inline_content(block_type, effective_sub_type):
+ raw["content"] = _extract_inline_content(block, legacy_lines, preserve_line_breaks=block_type == "index")
+ elif block_type in _STRING_CONTENT_BLOCK_TYPES:
+ raw["content"] = _extract_string_content(block, legacy_lines, block_type)
+
+ image_fields = _extract_image_fields(block, legacy_lines)
+ raw.update(image_fields)
+
+ normalized_lines = _normalize_lines(legacy_lines, width, height)
+ if normalized_lines:
+ raw["lines"] = normalized_lines
+
+ _copy_leaf_metadata(raw, block, block_type, effective_sub_type, inherited)
+ return raw
+
+
+def _effective_sub_type(block: dict[str, Any], block_type: str, inherited: dict[str, Any]) -> str | None:
+ """解析 code/image/chart body 的有效 sub_type。"""
+ sub_type = block.get("sub_type", inherited.get("sub_type"))
+ if isinstance(sub_type, str) and sub_type.strip():
+ return sub_type.strip()
+ if block_type == "code_body":
+ return "code"
+ return None
+
+
+def _uses_inline_content(block_type: str, sub_type: str | None) -> bool:
+ """判断当前 raw block 是否必须使用结构化 InlineSpan 列表。"""
+ return block_type in _INLINE_CONTENT_BLOCK_TYPES or (block_type == "code_body" and sub_type == "algorithm")
+
+
+def _extract_inline_content(
+ block: dict[str, Any],
+ lines: list[Any],
+ *,
+ preserve_line_breaks: bool,
+) -> list[dict[str, Any]]:
+ """从旧 lines/spans 或直接 content 构造严格 InlineSpan 列表。"""
+ line_contents: list[list[InlineSpan]] = []
+ for line in lines:
+ if not isinstance(line, dict):
+ continue
+ spans = line.get("spans")
+ if not isinstance(spans, list):
+ continue
+ converted: list[dict[str, Any]] = []
+ for span in spans:
+ if not isinstance(span, dict):
+ continue
+ _append_legacy_span(converted, span)
+ parsed = parse_inline_spans(converted)
+ if parsed:
+ line_contents.append(parsed)
+
+ if not line_contents:
+ direct_content = block.get("content")
+ converted = _convert_direct_inline_content(direct_content)
+ return [span.model_dump(mode="json") for span in parse_inline_spans(converted)]
+
+ if preserve_line_breaks:
+ joined: list[dict[str, Any]] = []
+ for line_index, content in enumerate(line_contents):
+ if line_index:
+ _append_text_span(joined, "\n", ())
+ _extend_span_dicts(joined, [span.model_dump(mode="json") for span in content])
+ return [span.model_dump(mode="json") for span in parse_inline_spans(joined)]
+
+ joined_spans = join_inline_spans(line_contents)
+ return [span.model_dump(mode="json") for span in joined_spans]
+
+
+def _convert_direct_inline_content(content: Any) -> list[dict[str, Any]]:
+ """转换没有 lines 包装的旧字符串或 span 列表。"""
+ if isinstance(content, str):
+ return _parse_legacy_markup(content)
+ if not isinstance(content, list):
+ return []
+ output: list[dict[str, Any]] = []
+ for item in content:
+ if isinstance(item, dict):
+ _append_legacy_span(output, item)
+ elif isinstance(item, str):
+ _extend_span_dicts(output, _parse_legacy_markup(item))
+ return output
+
+
+def _append_legacy_span(output: list[dict[str, Any]], span: dict[str, Any]) -> None:
+ """把一个 3.4.5 span 追加为当前 InlineSpan 字典。"""
+ span_type = str(span.get("type", "text") or "text")
+ content = span.get("content")
+ if span_type in {"inline_equation", "equation_inline", "interline_equation", "equation"}:
+ normalized = str(content or "").strip()
+ if normalized:
+ output.append({"type": "equation_inline", "content": normalized})
+ return
+ if span_type in {"code", "code_inline"}:
+ if isinstance(content, str) and content:
+ output.append({"type": "code_inline", "content": content})
+ return
+ if span_type == "hyperlink":
+ _append_legacy_hyperlink(output, span)
+ return
+ if not isinstance(content, str) or not content:
+ return
+
+ styles = _normalize_styles(span.get("styles", span.get("style")))
+ if styles:
+ _append_text_span(output, content, styles)
+ else:
+ _extend_span_dicts(output, _parse_legacy_markup(content))
+
+
+def _append_legacy_hyperlink(output: list[dict[str, Any]], span: dict[str, Any]) -> None:
+ """转换结构化旧 hyperlink;不安全目标降级为可见子内容。"""
+ children: list[dict[str, Any]] = []
+ raw_children = span.get("children")
+ if isinstance(raw_children, list):
+ for child in raw_children:
+ if isinstance(child, dict):
+ _append_legacy_span(children, child)
+ elif isinstance(span.get("content"), str):
+ styles = _normalize_styles(span.get("styles", span.get("style")))
+ if styles:
+ _append_text_span(children, span["content"], styles)
+ else:
+ _extend_span_dicts(children, _parse_legacy_markup(span["content"]))
+
+ safe_children = [child for child in children if child.get("type") != "hyperlink"]
+ safe_url = sanitize_hyperlink_target(
+ html.unescape(str(span.get("url", ""))),
+ allowed_schemes=OFFICE_EXTERNAL_HYPERLINK_SCHEMES,
+ allow_relative=True,
+ allow_fragment=True,
+ )
+ if safe_url and safe_children:
+ output.append({"type": "hyperlink", "url": safe_url, "content": safe_children})
+ else:
+ _extend_span_dicts(output, safe_children)
+
+
+def _parse_legacy_markup(content: str) -> list[dict[str, Any]]:
+ """按 3.4.5 的私有字符串标签协议解析行内内容。"""
+ output: list[dict[str, Any]] = []
+ cursor = 0
+ while match := _INLINE_START_RE.search(content, cursor):
+ _append_text_span(output, content[cursor : match.start()], ())
+ tag = match.group("tag").lower()
+ closing = _find_matching_close(content, tag, match.end())
+ if closing is None:
+ _append_text_span(output, match.group(0), ())
+ cursor = match.end()
+ continue
+
+ inner = content[match.end() : closing[0]]
+ original = content[match.start() : closing[1]]
+ parsed = _parse_legacy_element(tag, match.group("attrs") or "", inner)
+ if parsed is None:
+ _append_text_span(output, original, ())
+ else:
+ _extend_span_dicts(output, parsed)
+ cursor = closing[1]
+ _append_text_span(output, content[cursor:], ())
+ return output
+
+
+def _find_matching_close(content: str, tag: str, start: int) -> tuple[int, int] | None:
+ """查找旧标签的配对结束位置,并兼容同名嵌套。"""
+ token_re = re.compile(rf"<(?P<close>/)?{re.escape(tag)}(?:\s[^<>]*?)?>", re.IGNORECASE)
+ depth = 1
+ for match in token_re.finditer(content, start):
+ depth += -1 if match.group("close") else 1
+ if depth == 0:
+ return match.start(), match.end()
+ return None
+
+
+def _parse_legacy_element(tag: str, attrs: str, inner: str) -> list[dict[str, Any]] | None:
+ """把一个完整旧标签转换为当前 InlineSpan 字典。"""
+ if tag == "eq":
+ latex = html.unescape(inner).strip()
+ return [{"type": "equation_inline", "content": latex}] if latex else []
+ if tag == "code":
+ code = html.unescape(inner)
+ return [{"type": "code_inline", "content": code}] if code else []
+ if tag == "hyperlink":
+ return _parse_markup_hyperlink(inner)
+
+ children = _parse_legacy_markup(inner)
+ styles = _parse_style_attribute(attrs) if tag == "text" else [_DIRECT_TAG_STYLES[tag]]
+ return _apply_styles(children, styles)
+
+
+def _parse_markup_hyperlink(inner: str) -> list[dict[str, Any]] | None:
+ """解析旧 hyperlink 标签;危险 URL 只移除链接语义。"""
+ match = _URL_RE.search(inner)
+ if match is None:
+ return None
+ label_content = f"{inner[: match.start()]}{inner[match.end() :]}"
+ children = [span for span in _parse_legacy_markup(label_content) if span.get("type") != "hyperlink"]
+ if not children:
+ return []
+ safe_url = sanitize_hyperlink_target(
+ html.unescape(match.group("url")),
+ allowed_schemes=OFFICE_EXTERNAL_HYPERLINK_SCHEMES,
+ allow_relative=True,
+ allow_fragment=True,
+ )
+ if safe_url is None:
+ return children
+ return [{"type": "hyperlink", "url": safe_url, "content": children}]
+
+
+def _parse_style_attribute(attrs: str) -> list[str]:
+ """读取旧 ``text style`` 属性并过滤为当前白名单样式。"""
+ match = _STYLE_ATTR_RE.search(attrs)
+ return _normalize_styles(match.group("style") if match else None)
+
+
+def _normalize_styles(value: Any) -> list[str]:
+ """按当前固定顺序去重旧样式,并消解同时上下标的非法组合。"""
+ if isinstance(value, str):
+ candidates = [item.strip().lower() for item in value.split(",")]
+ elif isinstance(value, (list, tuple, set)):
+ candidates = [str(item).strip().lower() for item in value]
+ else:
+ candidates = []
+ style_set = {item for item in candidates if item in INLINE_STYLE_ORDER}
+ if "superscript" in style_set and "subscript" in style_set:
+ style_set.remove("subscript")
+ return [style for style in INLINE_STYLE_ORDER if style in style_set]
+
+
+def _apply_styles(spans: list[dict[str, Any]], styles: list[str]) -> list[dict[str, Any]]:
+ """把旧包装标签的样式递归应用到文字及链接文字。"""
+ if not styles:
+ return spans
+ output: list[dict[str, Any]] = []
+ for span in spans:
+ span_type = span.get("type")
+ if span_type == "text":
+ merged_styles = _normalize_styles([*_normalize_styles(span.get("styles")), *styles])
+ _append_text_span(output, str(span.get("content", "")), merged_styles)
+ elif span_type == "hyperlink" and isinstance(span.get("content"), list):
+ children = _apply_styles([child for child in span["content"] if isinstance(child, dict)], styles)
+ output.append({**span, "content": children})
+ else:
+ output.append(dict(span))
+ return output
+
+
+def _append_text_span(output: list[dict[str, Any]], content: str, styles: Any) -> None:
+ """追加非空文字,并合并相邻同样式 TextSpan。"""
+ if not content:
+ return
+ normalized_styles = _normalize_styles(styles)
+ if output and output[-1].get("type") == "text" and output[-1].get("styles", []) == normalized_styles:
+ output[-1]["content"] = f"{output[-1].get('content', '')}{content}"
+ return
+ span: dict[str, Any] = {"type": "text", "content": content}
+ if normalized_styles:
+ span["styles"] = normalized_styles
+ output.append(span)
+
+
+def _extend_span_dicts(output: list[dict[str, Any]], spans: list[dict[str, Any]]) -> None:
+ """追加一组 Span,并规范化序列边界上的相邻文字。"""
+ for span in spans:
+ if span.get("type") == "text" and isinstance(span.get("content"), str):
+ _append_text_span(output, span["content"], span.get("styles", []))
+ else:
+ output.append(dict(span))
+
+
+def _extract_string_content(block: dict[str, Any], lines: list[Any], block_type: str) -> str:
+ """从旧 spans 提取公式、视觉主体、表格 HTML 或代码字符串。"""
+ parts: list[str] = []
+ for line in lines:
+ if not isinstance(line, dict):
+ continue
+ line_parts: list[str] = []
+ spans = line.get("spans")
+ if not isinstance(spans, list):
+ continue
+ for span in spans:
+ if not isinstance(span, dict):
+ continue
+ value = span.get("html") if block_type == "table_body" and span.get("html") is not None else span.get("content")
+ normalized = _stringify_visual_content(value)
+ if normalized:
+ line_parts.append(normalized)
+ if line_parts:
+ parts.append("".join(line_parts))
+
+ if not parts:
+ direct = block.get("html") if block_type == "table_body" and block.get("html") is not None else block.get("content")
+ return _stringify_visual_content(direct)
+ return "\n".join(parts) if block_type == "code_body" else "".join(parts)
+
+
+def _stringify_visual_content(value: Any) -> str:
+ """把旧视觉识别列表稳定折叠为当前字符串载荷。"""
+ if isinstance(value, str):
+ return value
+ if isinstance(value, list):
+ return "\n".join(str(item) for item in value if str(item).strip())
+ return "" if value is None else str(value)
+
+
+def _extract_image_fields(block: dict[str, Any], lines: list[Any]) -> dict[str, str]:
+ """从 block 或第一个视觉 span 提升当前支持的图片载荷字段。"""
+ fields: dict[str, str] = {}
+ for field_name in ("image_path", "image_base64", "image_url"):
+ value = block.get(field_name)
+ if isinstance(value, str) and value:
+ fields[field_name] = value
+ if fields:
+ return fields
+ for line in lines:
+ if not isinstance(line, dict):
+ continue
+ spans = line.get("spans")
+ if not isinstance(spans, list):
+ continue
+ for span in spans:
+ if not isinstance(span, dict):
+ continue
+ for field_name in ("image_path", "image_base64", "image_url"):
+ value = span.get(field_name)
+ if isinstance(value, str) and value and field_name not in fields:
+ fields[field_name] = value
+ if fields:
+ return fields
+ return fields
+
+
+def _copy_leaf_metadata(
+ raw: dict[str, Any],
+ block: dict[str, Any],
+ block_type: str,
+ sub_type: str | None,
+ inherited: dict[str, Any],
+) -> None:
+ """复制当前后处理仍会读取、且不会越过严格模型边界的字段。"""
+ if block.get("level") is not None:
+ raw["level"] = block["level"]
+ if sub_type:
+ raw["sub_type"] = sub_type
+ guess_lang = block.get("guess_lang", inherited.get("guess_lang"))
+ if block_type == "code_body" and isinstance(guess_lang, str) and guess_lang.strip():
+ raw["guess_lang"] = guess_lang.strip()
+ if block_type == "table_body" and "cell_merge" in inherited:
+ raw["cell_merge"] = inherited["cell_merge"]
+ if block_type in {"doc_title", "paragraph_title", "page_footnote"}:
+ anchor = block.get("anchor")
+ if isinstance(anchor, str) and anchor.strip():
+ raw["anchor"] = anchor.strip()
+
+
+def _normalize_block_type(block: dict[str, Any]) -> str:
+ """把最终 3.4.5 discriminator 映射到当前 raw 类型并拒绝内部标签。"""
+ block_type = str(block.get("type", "") or "")
+ if block_type == "title":
+ return "doc_title" if block.get("level") == 1 else "paragraph_title"
+ normalized = _BLOCK_TYPE_ALIASES.get(block_type, block_type)
+ if normalized not in _SUPPORTED_BLOCK_TYPES:
+ raise ValueError(f"unsupported MinerU 3.4.5 block type {block_type!r}; source reparse required")
+ return normalized
+
+
+def _normalize_bbox(value: Any, width: float, height: float) -> tuple[float, float, float, float]:
+ """把旧 bbox 规范为当前严格的 0-1 非零面积坐标。"""
+ if not isinstance(value, (list, tuple)) or len(value) != 4:
+ return _PLACEHOLDER_BBOX
+ if any(isinstance(item, bool) or not isinstance(item, (int, float)) for item in value):
+ return _PLACEHOLDER_BBOX
+ coords = tuple(float(item) for item in value)
+ if not all(math.isfinite(item) for item in coords):
+ return _PLACEHOLDER_BBOX
+ if width > 0 and height > 0:
+ coords = (coords[0] / width, coords[1] / height, coords[2] / width, coords[3] / height)
+ elif not all(0.0 <= item <= 1.0 for item in coords):
+ return _PLACEHOLDER_BBOX
+ clamped = tuple(min(max(item, 0.0), 1.0) for item in coords)
+ if clamped[2] <= clamped[0] or clamped[3] <= clamped[1]:
+ return _PLACEHOLDER_BBOX
+ return clamped # type: ignore[return-value]
+
+
+def _normalize_lines(lines: list[Any], width: float, height: float) -> list[dict[str, list[float]]]:
+ """保留跨页文本合并需要的行框,并转换到当前归一化坐标。"""
+ result: list[dict[str, list[float]]] = []
+ for line in lines:
+ if not isinstance(line, dict):
+ continue
+ bbox = _normalize_bbox(line.get("bbox"), width, height)
+ result.append({"bbox": list(bbox)})
+ return result
+
+
+__all__ = ["legacy_page_to_model_list"]
diff --git a/mineru/doclib/background/compaction.py b/mineru/doclib/background/compaction.py
index 5e272414..29628072 100644
--- a/mineru/doclib/background/compaction.py
+++ b/mineru/doclib/background/compaction.py
@@ -14,6 +14,7 @@ from typing import Any, cast
from ...parser.page_range import format_page_range
from ...types import Tier
from ..core.db import DatabaseManager
+from ..core.middle_json import read_cached_middle_json
from ..rows import ParseBatchRow, ParseGroupRow, ParseRow
from ..services.parse_svc import parse_batch_json_path, parse_page_range_set
from ..types import PARSE_STATUS_DONE, PARSE_STATUS_SUPERSEDED
@@ -22,8 +23,8 @@ logger = logging.getLogger("mineru.compaction")
def _normalize_batch_pages(batch_payload: dict[str, Any]) -> list[dict[str, Any]]:
- """仅校验并读取新版批次,拒绝猜测或改写旧协议数据。"""
- return [page.to_dict() for page in ParseResult.from_dict(batch_payload).pages]
+ """读取当前和受支持的历史批次,仅在内存中统一页面结构。"""
+ return [page.to_dict() for page in read_cached_middle_json(batch_payload).pages]
class Compaction:
@@ -149,7 +150,8 @@ class Compaction:
try:
with open(fpath, encoding="utf-8") as f:
batch_payload = json.load(f)
- batch_pages = _normalize_batch_pages(batch_payload)
+ middle_json = read_cached_middle_json(batch_payload)
+ batch_pages = [page.to_dict() for page in middle_json.pages]
except Exception:
return None
if not batch_pages:
@@ -159,9 +161,7 @@ class Compaction:
if type(page_idx) is not int or page_idx < 0:
return None
pages_by_page_idx[page_idx] = page
- current_envelope = {
- key: value for key, value in ParseResult.from_dict(batch_payload).to_dict().items() if key != "pages"
- }
+ current_envelope = {key: value for key, value in middle_json.to_dict().items() if key != "pages"}
if not envelope:
envelope = current_envelope
elif json.dumps(envelope, sort_keys=True) != json.dumps(current_envelope, sort_keys=True):
diff --git a/mineru/doclib/core/middle_json.py b/mineru/doclib/core/middle_json.py
new file mode 100644
index 00000000..c0fdc723
--- /dev/null
+++ b/mineru/doclib/core/middle_json.py
@@ -0,0 +1,130 @@
+"""仅供 Doclib 持久化结果使用的 Middle JSON 读取与历史转换边界。
+
+两个历史分支集中在此,未来可一并移除;通用 ParseResult 与 DocVortex codec
+不调用本模块。读取只构造内存对象,不重写文件、不重新推理。
+"""
+
+from __future__ import annotations
+
+from copy import deepcopy
+from typing import Any
+
+from docvortex.schema import DocumentMetadata, MiddleJson, ModelJson, Producer
+
+from ...integrations.docvortex import MinerUMetadata, build_metadata, validate_mineru_metadata
+
+_LEGACY_PAGE_FIELDS = frozenset({"page_size", "preproc_blocks", "para_blocks", "discarded_blocks"})
+_LEGACY_V2_FIELDS = frozenset(
+ {"schema_version", "pages", "is_full_document", "file_suffix", "effort", "parse_mode", "mineru_version", "extensions"}
+)
+
+
+def _legacy_metadata(payload: dict[str, Any], *, version_keys: tuple[str, ...]) -> DocumentMetadata:
+ """保留可辨识的历史生产版本,缺失时标记未知而不使用当前程序版本。"""
+ versions = [payload[key].strip() for key in version_keys if isinstance(payload.get(key), str) and payload[key].strip()]
+ if len(set(versions)) > 1:
+ raise ValueError("Conflicting legacy producer versions")
+ version = versions[0] if versions else "unknown"
+ return DocumentMetadata(
+ file_suffix=payload.get("file_suffix", "pdf"),
+ producer=Producer(name="mineru", version=version),
+ )
+
+
+def _legacy_extensions(payload: dict[str, Any]) -> dict[str, Any]:
+ """只迁移可靠的档位及模式,保留应用扩展并拒绝保留字段冲突。"""
+ extensions = deepcopy(payload.get("extensions", {}))
+ if not isinstance(extensions, dict):
+ raise ValueError("legacy extensions must be a JSON object")
+ efforts = {
+ payload[key]
+ for key in ("_effort", "effort")
+ if isinstance(payload.get(key), str) and payload[key] in {"flash", "medium", "high", "xhigh"}
+ }
+ if len(efforts) > 1:
+ raise ValueError("Conflicting legacy effort records")
+ effort = next(iter(efforts), None)
+ mode = payload.get("parse_mode")
+ ocr_enabled = payload.get("_ocr_enable")
+ if mode in ("txt", "ocr") and type(ocr_enabled) is bool and (mode == "ocr") != ocr_enabled:
+ raise ValueError("Conflicting legacy parse_mode and _ocr_enable")
+ if mode not in ("txt", "ocr"):
+ mode = ("ocr" if ocr_enabled else "txt") if type(ocr_enabled) is bool else None
+
+ if "mineru" in extensions:
+ existing = MinerUMetadata.model_validate(extensions["mineru"])
+ if mode is not None and existing.parse_mode != mode:
+ raise ValueError("Conflicting legacy parse_mode and extensions.mineru")
+ if effort is not None and existing.tier != build_metadata(effort=effort, parse_mode="txt")["mineru"]["tier"]:
+ raise ValueError("Conflicting legacy effort and extensions.mineru")
+ elif effort is not None and mode is not None:
+ extensions.update(build_metadata(effort=effort, parse_mode=mode))
+ return extensions
+
+
+def _read_legacy_345(payload: dict[str, Any], *, page_field: str) -> MiddleJson:
+ """把可识别的 3.4.5 页面转为当前 raw ModelJson,再做确定性后处理。"""
+ from ...backend.postprocess.legacy_schema_adapter import legacy_page_to_model_list
+ from docvortex.postprocess.document import model_json_to_middle_json
+
+ pages = payload[page_field]
+ if not isinstance(pages, list) or any(not isinstance(page, dict) for page in pages):
+ raise ValueError("legacy pages must be a list of page objects")
+ for page in pages:
+ if "blocks" in page or (page_field == "pages" and not _LEGACY_PAGE_FIELDS.intersection(page)):
+ raise ValueError("Mixed or unrecognized legacy page structure")
+ indices = [page.get("page_idx", index) for index, page in enumerate(pages)]
+ if any(type(index) is not int or index < 0 for index in indices):
+ raise ValueError("legacy page indices must be non-negative integers")
+ model = ModelJson(
+ pages=[legacy_page_to_model_list(page) for page in pages],
+ page_index_map=[] if indices == list(range(len(pages))) else indices,
+ metadata=_legacy_metadata(payload, version_keys=("_version_name", "mineru_version")),
+ extensions=_legacy_extensions(payload),
+ )
+ if "is_full_document" in payload:
+ full_document = payload["is_full_document"]
+ if type(full_document) is not bool or full_document != model.is_full_document:
+ raise ValueError("Conflicting legacy is_full_document and page indices")
+ return model_json_to_middle_json(model)
+
+
+def _read_legacy_v2(payload: dict[str, Any]) -> MiddleJson:
+ """只移动旧 Schema 2.0 外层字段,页面树直接经过当前类型校验。"""
+ unexpected = payload.keys() - _LEGACY_V2_FIELDS
+ if unexpected:
+ raise ValueError(f"Conflicting or unsupported legacy Middle JSON fields: {sorted(unexpected)}")
+ return MiddleJson(
+ pages=payload.get("pages"),
+ is_full_document=payload.get("is_full_document"),
+ metadata=_legacy_metadata(payload, version_keys=("mineru_version",)),
+ extensions=_legacy_extensions(payload),
+ )
+
+
+def read_cached_middle_json(payload: dict[str, Any]) -> MiddleJson:
+ """先识别显式协议,再转换两个历史缓存族;失败时绝不跨协议兜底。"""
+ if not isinstance(payload, dict):
+ raise ValueError("Cached Middle JSON must be an object; source reparse required")
+ if "schema" in payload:
+ document = MiddleJson.from_dict(payload)
+ else:
+ if {"metadata", "producer", "page_index_map"}.intersection(payload):
+ raise ValueError("Mixed cached document envelope; source reparse required")
+ version = payload.get("schema_version")
+ source = deepcopy(payload)
+ if "pdf_info" in source:
+ if version is not None or "pages" in source:
+ raise ValueError("Mixed legacy pdf_info envelope; source reparse required")
+ document = _read_legacy_345(source, page_field="pdf_info")
+ elif version == "1.0" and "pages" in source:
+ document = _read_legacy_345(source, page_field="pages")
+ elif version == "2.0" and "pages" in source:
+ document = _read_legacy_v2(source)
+ else:
+ raise ValueError("Unsupported cached Middle JSON format; source reparse required")
+ validate_mineru_metadata(document)
+ return document
+
+
+__all__ = ["read_cached_middle_json"]
diff --git a/mineru/doclib/server.py b/mineru/doclib/server.py
index f9b0aee8..828b809d 100644
--- a/mineru/doclib/server.py
+++ b/mineru/doclib/server.py
@@ -75,7 +75,7 @@ from .rows import (
WatchTargetRow,
)
from .services.parse_svc import (
- _json_batch_is_current,
+ _json_batch_is_readable,
accessible_file_for_sha256,
filter_pages_by_user_range,
load_pages_from_done_batches,
@@ -423,7 +423,7 @@ class DoclibServer(AsyncDoclibInterface):
row
for row in rows
if row["status"] != PARSE_STATUS_DONE
- or _json_batch_is_current(_effective_data_dir(self.state), row["sha256"], row["tier"], row)
+ or _json_batch_is_readable(_effective_data_dir(self.state), row["sha256"], row["tier"], row)
]
coverage = _parse_coverage(page_range, coverage_rows) if resolved_sha256 and tier and page_range else None
return ListParsesResponse(
@@ -882,7 +882,7 @@ class DoclibServer(AsyncDoclibInterface):
_tier_parse_info(row)
for row in parse_rows
if row["status"] == PARSE_STATUS_DONE
- and _json_batch_is_current(_effective_data_dir(self.state), row["sha256"], row["tier"], row)
+ and _json_batch_is_readable(_effective_data_dir(self.state), row["sha256"], row["tier"], row)
]
active_parses = [
_parse_info(row) for row in parse_rows if row["status"] in {PARSE_STATUS_PENDING, PARSE_STATUS_PARSING}
@@ -1448,7 +1448,7 @@ class DoclibServer(AsyncDoclibInterface):
for row in rows
if row["tier"] in TIERS
and row["tier"] != "flash"
- and _json_batch_is_current(_effective_data_dir(self.state), sha256, row["tier"], row)
+ and _json_batch_is_readable(_effective_data_dir(self.state), sha256, row["tier"], row)
}
if not tiers:
return None
diff --git a/mineru/doclib/services/parse_svc.py b/mineru/doclib/services/parse_svc.py
index 3a351a81..acec445f 100644
--- a/mineru/doclib/services/parse_svc.py
+++ b/mineru/doclib/services/parse_svc.py
@@ -28,6 +28,7 @@ from ...filetypes import (
)
from ...parser.api_client import _APITransportError, _V1APIError
from ...parser.base import ParseResult
+from ..core.middle_json import read_cached_middle_json
from ...parser.page_range import (
expand_page_range,
format_page_range as _page_numbers_to_range_str,
@@ -745,7 +746,7 @@ class ParseService:
),
)
valid_done_batches = [
- batch for batch in done_batches if _json_batch_is_current(self.data_dir, sha256, requested_tier, batch)
+ batch for batch in done_batches if _json_batch_is_readable(self.data_dir, sha256, requested_tier, batch)
]
if supports_page_range:
for batch in valid_done_batches:
@@ -1405,7 +1406,7 @@ class ParseService:
tiers = sorted(tier_set, key=lambda item: TIER_ORDER.get(item, -1), reverse=True)
for tier in tiers:
tier_rows = [
- row for row in rows if row["tier"] == tier and _json_batch_is_current(self.data_dir, sha256, tier, row)
+ row for row in rows if row["tier"] == tier and _json_batch_is_readable(self.data_dir, sha256, tier, row)
]
pages = load_pages_from_done_batches(self.data_dir, sha256, tier, tier_rows)
if not pages:
@@ -1497,12 +1498,12 @@ def _probe_error_message(prefix: str, error_msg: str | None) -> str:
return prefix
-def _json_batch_is_current(data_dir: str, sha256: str, tier: Tier, batch: ParseRow) -> bool:
- """仅把协议有效且覆盖记录页范围的新批次视为可命中的缓存。"""
+def _json_batch_is_readable(data_dir: str, sha256: str, tier: Tier, batch: ParseRow) -> bool:
+ """将可转换为当前协议且覆盖记录页范围的持久化批次视为有效缓存。"""
json_path = parse_batch_json_path(data_dir, sha256, tier, batch["page_range"], batch["done_at"])
try:
with open(json_path, encoding="utf-8") as stream:
- result = ParseResult.from_dict(json.load(stream))
+ result = read_cached_middle_json(json.load(stream))
available = {page.page_idx + 1 for page in result.pages}
return bool(available) and parse_page_range_set(batch["page_range"]) <= available
except (OSError, ValueError, TypeError):
@@ -1523,7 +1524,7 @@ def load_pages_from_done_batches(
*,
requested_page_numbers: set[int] | None = None,
) -> list[PageInfo]:
- """读取新协议批次并保留最新重复页,所请求的旧缓存需明确重新解析。"""
+ """读取当前或受支持的历史批次,保留最新重复页;损坏或未知缓存须重新解析。"""
pages_by_page_idx: dict[int, PageInfo] = {}
stale_page_numbers: set[int] = set()
for row in reversed(done_rows):
@@ -1533,7 +1534,7 @@ def load_pages_from_done_batches(
try:
with open(fpath, encoding="utf-8") as f:
data = json.load(f)
- parse_result = ParseResult.from_dict(data)
+ parse_result = read_cached_middle_json(data)
for page in parse_result.pages:
pages_by_page_idx[page.page_idx] = page
except (OSError, ValueError, TypeError):
diff --git a/pyproject.toml b/pyproject.toml
index b3fa82f4..4836922b 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -38,7 +38,7 @@ classifiers = [
"Programming Language :: Python :: 3.13",
]
dependencies = [
- "docvortex>=0.2.0,<0.3.0",
+ "docvortex>=0.2.1,<0.3.0",
"mineru-vl-utils>=1.2.1,<2",
"metafile-render>=0.3.0,<1.0.0",
"mineru-llama-cpp>=0.1.0,<1",
diff --git a/tests/unittest/test_doclib_legacy_page_ranges.py b/tests/unittest/test_doclib_legacy_page_ranges.py
index 023a9afb..2d738772 100644
--- a/tests/unittest/test_doclib_legacy_page_ranges.py
+++ b/tests/unittest/test_doclib_legacy_page_ranges.py
@@ -28,6 +28,108 @@ from mineru.errors import MineruError
_SHA256 = "a" * 64
+def _convert_fixture_family(path: Path, family: str) -> None:
+ """仅在测试建库阶段生成不同历史格式,正文及来源记录保持可对照。"""
+ if family == "current":
+ return
+ source = json.loads(path.read_text())
+ metadata = source["metadata"]
+ product = source["extensions"]["mineru"]
+ effort = {"flash": "flash", "basic": "medium", "standard": "high", "advanced": "xhigh"}[product["tier"]]
+ if family == "v2":
+ output = {
+ "schema_version": "2.0",
+ "pages": source["pages"],
+ "is_full_document": source["is_full_document"],
+ "file_suffix": metadata["file_suffix"],
+ "mineru_version": metadata["producer"]["version"],
+ "effort": effort,
+ "parse_mode": product["parse_mode"],
+ }
+ else:
+ pages = []
+ for page in source["pages"]:
+ blocks = []
+ for block in page["blocks"]:
+ assert block["type"] == "text"
+ bbox = [coordinate * 100 for coordinate in block["bbox"]]
+ text = "".join(span["content"] for span in block["content"])
+ blocks.append(
+ {"type": "text", "bbox": bbox, "lines": [{"bbox": bbox, "spans": [{"type": "text", "content": text}]}]}
+ )
+ pages.append(
+ {"page_idx": page["page_idx"], "page_size": [100, 100], "preproc_blocks": blocks, "discarded_blocks": []}
+ )
+ output = {
+ "_version_name": metadata["producer"]["version"],
+ "_effort": effort,
+ "_ocr_enable": product["parse_mode"] == "ocr",
+ }
+ if family == "345":
+ output["pdf_info"] = pages
+ else:
+ assert family == "v1"
+ output.update(schema_version="1.0", pages=pages)
+ path.write_text(json.dumps(output))
+
+
[email protected]("family", ["345", "v1", "v2"])
+def test_historical_cache_rebuilds_fts_and_default_tier(tmp_path: Path, family: str) -> None:
+ """历史缓存参与全文索引重建和默认档位选择,读取不修改源批次。"""
+
+ async def verify() -> None:
+ """用真实 FTS5 和数据库查询核对所有历史格式的消费链。"""
+ async with _legacy_store(tmp_path, family=family) as (db, service, server):
+ path = _cache_path(tmp_path, "1~5", 1000)
+ standard_path = Path(parse_batch_json_path(str(tmp_path), _SHA256, "standard", "1~5", 1000))
+ standard_path.parent.mkdir(parents=True, exist_ok=True)
+ standard_path.write_bytes(path.read_bytes())
+ await db.execute("UPDATE parses SET tier='standard'")
+ before = standard_path.read_bytes()
+ assert await server._default_read_tier(_SHA256) == "standard"
+ await service._rebuild_fts_after_invalidate(_SHA256)
+ assert await service.fts.search("old")
+ assert await service.fts.get_tier(_SHA256) == "standard"
+ assert standard_path.read_bytes() == before
+
+ asyncio.run(verify())
+
+
[email protected]("legacy_family", ["345", "v1"])
+def test_all_supported_formats_compact_to_current_protocol(tmp_path: Path, legacy_family: str) -> None:
+ """混合三种协议的兼容批次合并后只输出新协议,重复页保留最新正文。"""
+
+ async def verify() -> None:
+ """使用非零起始页模拟旧抽页结果,统一后保持同一份来源及整本语义。"""
+ async with _legacy_store(tmp_path) as (db, _service, server):
+ await db.execute("DELETE FROM parses")
+ _cache_path(tmp_path, "1~5", 1000).unlink()
+ for family, page_range, done_at, numbers, label in [
+ (legacy_family, "3-4", 1000, [3, 4], "old"),
+ ("v2", "4-5", 2000, [4, 5], "intermediate"),
+ ("current", "5-6", 3000, [5, 6], "latest"),
+ ]:
+ await _add_result(db, tmp_path, page_range, done_at, numbers, label)
+ _convert_fixture_family(_cache_path(tmp_path, page_range, done_at), family)
+ compaction = Compaction(db=db, interval_sec=600, data_dir=str(tmp_path))
+ assert await compaction._compact_doc_tier(_SHA256, "flash") == 2
+ path = _cache_path(tmp_path, "3-6", 3000)
+ payload = json.loads(path.read_text())
+ assert payload["schema"] == "docvortex.middle"
+ assert payload["schema_version"] == "2.0"
+ assert "file_suffix" not in payload and "effort" not in payload
+ assert payload["metadata"]["producer"]["version"] == __version__
+ assert [page["page_idx"] for page in payload["pages"]] == [2, 3, 4, 5]
+ content = await server._render_doc_content(
+ _SHA256, tier="flash", page_range="3-6", format="markdown", no_marker=True
+ )
+ assert "old page 3" in content and "intermediate page 4" in content and "latest page 5" in content
+ assert "old page 4" not in content and "intermediate page 5" not in content
+ assert [item.name for item in path.parent.glob("*.json")] == [path.name]
+
+ asyncio.run(verify())
+
+
@pytest.mark.parametrize("old_version", [None, "1.0"])
def test_old_protocol_cache_is_not_available_or_compacted(tmp_path: Path, old_version: str | None) -> None:
"""旧协议不命中、不参与覆盖和压缩,读取旧页明确要求重新解析。"""
@@ -142,7 +244,9 @@ async def _add_result(
@asynccontextmanager
-async def _legacy_store(root: Path) -> AsyncIterator[tuple[DatabaseManager, ParseService, DoclibServer]]:
+async def _legacy_store(
+ root: Path, *, family: str = "current"
+) -> AsyncIterator[tuple[DatabaseManager, ParseService, DoclibServer]]:
"""建立已入库的源文件元数据、旧名称缓存及真实 SQLite 服务,退出时关闭连接。"""
source = root / "document.pdf"
source.write_bytes(b"%PDF-1.7\n")
@@ -161,6 +265,7 @@ async def _legacy_store(root: Path) -> AsyncIterator[tuple[DatabaseManager, Pars
(str(source), source.name, "pdf", stat.st_size, int(stat.st_mtime * 1000), _SHA256, "active", 1000, 1000),
)
await _add_result(db, root, "1~5", 1000, range(1, 6), "old")
+ _convert_fixture_family(_cache_path(root, "1~5", 1000), family)
service = ParseService(
db=db, fts=FTSManager(db), config_svc=ConfigService(db), data_dir=str(root), parse_lock_timeout_sec=1800
)
@@ -170,12 +275,13 @@ async def _legacy_store(root: Path) -> AsyncIterator[tuple[DatabaseManager, Pars
await db.close()
-def test_old_cache_hits_exports_and_serializes_without_mutating_storage(tmp_path: Path) -> None:
[email protected]("family", ["current", "345", "v1", "v2"])
+def test_old_cache_hits_exports_and_serializes_without_mutating_storage(tmp_path: Path, family: str) -> None:
"""旧结果正常命中、导出和查询,返回新格式而文件与数据库始终保留旧名称。"""
async def verify() -> None:
"""读取各个结果出口,确认规范化不破坏原始缓存定位。"""
- async with _legacy_store(tmp_path) as (db, service, server):
+ async with _legacy_store(tmp_path, family=family) as (db, service, server):
old_path = _cache_path(tmp_path, "1~5", 1000)
original_bytes = old_path.read_bytes()
row = await db.fetchone("SELECT * FROM parses WHERE sha256=?", (_SHA256,))
@@ -211,12 +317,13 @@ def test_old_cache_hits_exports_and_serializes_without_mutating_storage(tmp_path
asyncio.run(verify())
-def test_old_cache_only_schedules_uncovered_pages(tmp_path: Path) -> None:
[email protected]("family", ["current", "345", "v1", "v2"])
+def test_old_cache_only_schedules_uncovered_pages(tmp_path: Path, family: str) -> None:
"""旧 1~5 批次覆盖前五页,新请求只创建 6-10,重复请求复用新任务。"""
async def verify() -> None:
"""检查真实任务入队和原记录内容。"""
- async with _legacy_store(tmp_path) as (db, service, _server):
+ async with _legacy_store(tmp_path, family=family) as (db, service, _server):
result = await service.request_parse(str(tmp_path / "document.pdf"), tier="flash", page_range="1-10")
assert not result.cache_hit and len(result.created_parse_ids) == 1
rows = await db.fetchall("SELECT page_range, status FROM parses ORDER BY id")
@@ -229,12 +336,13 @@ def test_old_cache_only_schedules_uncovered_pages(tmp_path: Path) -> None:
asyncio.run(verify())
-def test_mixed_cache_compaction_keeps_latest_pages_and_writes_new_ranges(tmp_path: Path) -> None:
[email protected]("family", ["current", "v2"])
+def test_mixed_cache_compaction_keeps_latest_pages_and_writes_new_ranges(tmp_path: Path, family: str) -> None:
"""新旧缓存合并时重复页取最新正文,并生成连字符记录与文件名。"""
async def verify() -> None:
"""压缩前后均核对页序与重复页正文。"""
- async with _legacy_store(tmp_path) as (db, _service, server):
+ async with _legacy_store(tmp_path, family=family) as (db, _service, server):
await _add_result(db, tmp_path, "4-7", 2000, range(4, 8), "new")
rows = await db.fetchall("SELECT * FROM parses ORDER BY done_at DESC")
pages = load_pages_from_done_batches(str(tmp_path), _SHA256, "flash", rows)
diff --git a/tests/unittest/test_doclib_legacy_schema_adapter.py b/tests/unittest/test_doclib_legacy_schema_adapter.py
new file mode 100644
index 00000000..0d594510
--- /dev/null
+++ b/tests/unittest/test_doclib_legacy_schema_adapter.py
@@ -0,0 +1,391 @@
+from docvortex.schema import Producer
+from mineru.integrations.docvortex import build_metadata
+from copy import deepcopy
+
+import pytest
+
+from mineru.backend.postprocess.legacy_schema_adapter import legacy_page_to_model_list
+from docvortex.postprocess.pages import model_json_to_pages
+from mineru.doclib.background.compaction import _normalize_batch_pages
+from mineru.types import BlockType, MiddleJson, ModelJson
+
+
+def _legacy_line(content: str, *, bbox: list[int] | None = None) -> dict:
+ """构造 3.4.5 普通文字行。"""
+ return {
+ "bbox": bbox or [10, 10, 90, 20],
+ "spans": [{"type": "text", "content": content}],
+ }
+
+
+def _current_pages(raw_pages: list[list[dict]]) -> list:
+ """把适配后的 raw pages 送入当前严格后处理。"""
+ model_json = ModelJson(
+ pages=raw_pages,
+ page_index_map=[],
+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version="3.4.5")},
+ extensions=build_metadata(effort="medium", parse_mode="txt"),
+ )
+ return model_json_to_pages(model_json)
+
+
+def test_legacy_adapter_converts_inline_spans_without_mutating_input() -> None:
+ """验证文字标签、公式、链接及危险链接降级,并保证输入不变。"""
+ page = {
+ "page_idx": 0,
+ "page_size": [100, 100],
+ "preproc_blocks": [
+ {
+ "type": "text",
+ "bbox": [10, 10, 90, 30],
+ "lines": [
+ {
+ "bbox": [10, 10, 90, 30],
+ "spans": [
+ {
+ "type": "text",
+ "content": (
+ 'plain <text style="bold,italic">styled</text> '
+ "<eq>x^2</eq> "
+ "<hyperlink><text>safe</text><url>https://example.com</url></hyperlink> "
+ "<hyperlink><text>unsafe</text><url>javascript:alert(1)</url></hyperlink>"
+ ),
+ }
+ ],
+ }
+ ],
+ }
+ ],
+ "discarded_blocks": [],
+ }
+ original = deepcopy(page)
+
+ raw_page = legacy_page_to_model_list(page)
+ content = raw_page[0]["content"]
+
+ assert page == original
+ assert content == [
+ {"type": "text", "content": "plain ", "styles": []},
+ {"type": "text", "content": "styled", "styles": ["bold", "italic"]},
+ {"type": "text", "content": " ", "styles": []},
+ {"type": "equation_inline", "content": "x^2"},
+ {"type": "text", "content": " ", "styles": []},
+ {
+ "type": "hyperlink",
+ "url": "https://example.com",
+ "content": [{"type": "text", "content": "safe", "styles": []}],
+ },
+ {"type": "text", "content": " unsafe", "styles": []},
+ ]
+ pages = _current_pages([raw_page])
+ assert pages[0].blocks[0].type == BlockType.TEXT
+
+
+def test_legacy_adapter_preserves_visual_metadata_and_algorithm_spans() -> None:
+ """验证视觉父块展平后能由当前后处理恢复表格与算法块。"""
+ page = {
+ "page_idx": 0,
+ "page_size": [100, 100],
+ "para_blocks": [
+ {
+ "type": "table",
+ "bbox": [10, 10, 90, 60],
+ "cell_merge": [0, 1],
+ "blocks": [
+ {
+ "type": "table_caption",
+ "bbox": [10, 10, 90, 20],
+ "lines": [_legacy_line("表 1")],
+ },
+ {
+ "type": "table_body",
+ "bbox": [10, 20, 90, 60],
+ "lines": [
+ {
+ "bbox": [10, 20, 90, 60],
+ "spans": [
+ {
+ "type": "table",
+ "html": "<table><tr><td>x</td></tr></table>",
+ "image_path": "images/table.png",
+ }
+ ],
+ }
+ ],
+ },
+ ],
+ },
+ {
+ "type": "code",
+ "sub_type": "algorithm",
+ "bbox": [10, 70, 90, 90],
+ "blocks": [
+ {
+ "type": "code_body",
+ "bbox": [10, 70, 90, 90],
+ "lines": [
+ {
+ "bbox": [10, 70, 90, 90],
+ "spans": [
+ {"type": "text", "content": "a="},
+ {"type": "inline_equation", "content": "x"},
+ ],
+ }
+ ],
+ }
+ ],
+ },
+ ],
+ "discarded_blocks": [{"type": "header", "bbox": [10, 1, 90, 5], "lines": [_legacy_line("Header")]}],
+ }
+
+ pages = _current_pages([legacy_page_to_model_list(page)])
+ table, code, header = pages[0].blocks
+
+ assert table.type == BlockType.TABLE
+ assert table.cell_merge == [0, 1]
+ assert table.content[1].image_path == "images/table.png"
+ assert code.type == BlockType.CODE
+ assert code.sub_type == "algorithm"
+ assert code.content[0].type == BlockType.ALGORITHM_BODY
+ assert [span.type for span in code.content[0].content] == ["text", "equation_inline"]
+ assert header.type == BlockType.HEADER
+
+
+def test_legacy_adapter_normalizes_bbox_and_block_aliases() -> None:
+ """验证像素坐标、越界坐标及旧 discriminator 均转换为当前合法值。"""
+ page = {
+ "page_size": [200, 100],
+ "preproc_blocks": [
+ {
+ "type": "title",
+ "level": 1,
+ "bbox": [-10, 0, 220, 20],
+ "lines": [_legacy_line("Document")],
+ },
+ {
+ "type": "interline_equation",
+ "bbox": [0, 30, 100, 50],
+ "lines": [
+ {
+ "bbox": [0, 30, 100, 50],
+ "spans": [{"type": "interline_equation", "content": "y=1"}],
+ }
+ ],
+ },
+ ],
+ "discarded_blocks": [],
+ }
+
+ raw_page = legacy_page_to_model_list(page)
+
+ assert raw_page[0]["type"] == "doc_title"
+ assert raw_page[0]["bbox"] == (0.0, 0.0, 1.0, 0.2)
+ assert raw_page[1]["type"] == "equation"
+ assert raw_page[1]["content"] == "y=1"
+ _current_pages([raw_page])
+
+
+def test_legacy_adapter_keeps_only_final_middle_json_aliases() -> None:
+ """验证最终 3.4.5 仍会序列化的三个旧类型继续转换。"""
+ page = {
+ "page_size": [100, 100],
+ "preproc_blocks": [
+ {"type": "abstract", "bbox": [0, 0, 100, 20], "lines": [_legacy_line("Abstract")]},
+ {"type": "phonetic", "bbox": [0, 30, 100, 50], "lines": [_legacy_line("phonetic")]},
+ {
+ "type": "interline_equation",
+ "bbox": [0, 60, 100, 80],
+ "lines": [
+ {
+ "bbox": [0, 60, 100, 80],
+ "spans": [{"type": "interline_equation", "content": "x=1"}],
+ }
+ ],
+ },
+ ],
+ "discarded_blocks": [],
+ }
+
+ raw_page = legacy_page_to_model_list(page)
+
+ assert [block["type"] for block in raw_page] == ["text", "text", "equation"]
+ assert raw_page[0]["content"][0]["content"] == "Abstract"
+ assert raw_page[1]["content"][0]["content"] == "phonetic"
+ assert raw_page[2]["content"] == "x=1"
+ _current_pages([raw_page])
+
+
+ "block_type",
+ [
+ "vertical_text",
+ "formula_number",
+ "discarded",
+ "image_block",
+ "algorithm",
+ "algorithm_body",
+ "algorithm_caption",
+ "algorithm_footnote",
+ "header_image",
+ "footer_image",
+ ],
+)
+def test_legacy_adapter_rejects_pre_middle_json_block_types(block_type: str) -> None:
+ """验证 finalize 前布局、raw model 与 Content List 类型要求源文件重解析。"""
+ block = {
+ "type": block_type,
+ "bbox": [0, 0, 100, 20],
+ "lines": [_legacy_line("internal")],
+ }
+ if block_type in {"image_block", "algorithm"}:
+ block["blocks"] = [{"type": "text", "bbox": [0, 0, 100, 20], "lines": [_legacy_line("nested")]}]
+ page = {
+ "page_size": [100, 100],
+ "preproc_blocks": [block],
+ "discarded_blocks": [],
+ }
+
+ with pytest.raises(ValueError, match=rf"{block_type!s}.*source reparse"):
+ legacy_page_to_model_list(page)
+
+
+def test_legacy_adapter_preserves_canonical_image_and_auxiliary_blocks() -> None:
+ """验证规范 image 视觉块及 header/footer 辅助块不会提升为正文图片。"""
+ page = {
+ "page_size": [100, 100],
+ "preproc_blocks": [
+ {
+ "type": "image",
+ "bbox": [10, 20, 90, 80],
+ "blocks": [
+ {
+ "type": "image_body",
+ "bbox": [10, 20, 90, 80],
+ "lines": [
+ {
+ "bbox": [10, 20, 90, 80],
+ "spans": [{"type": "image", "image_path": "images/body.png"}],
+ }
+ ],
+ }
+ ],
+ }
+ ],
+ "discarded_blocks": [
+ {"type": "header", "bbox": [0, 0, 100, 10], "lines": [_legacy_line("Header")]},
+ {"type": "footer", "bbox": [0, 90, 100, 100], "lines": [_legacy_line("Footer")]},
+ ],
+ }
+
+ pages = _current_pages([legacy_page_to_model_list(page)])
+
+ assert [block.type for block in pages[0].blocks] == [BlockType.IMAGE, BlockType.HEADER, BlockType.FOOTER]
+ image = pages[0].blocks[0]
+ assert image.content[0].image_path == "images/body.png"
+
+
+def test_legacy_adapter_recovers_pdf_list_items_from_line_flags() -> None:
+ """验证旧 PDF list 的物理行重新生成当前列表文本子块。"""
+ page = {
+ "page_size": [100, 100],
+ "preproc_blocks": [
+ {
+ "type": "list",
+ "bbox": [10, 10, 90, 40],
+ "lines": [
+ {
+ "bbox": [10, 10, 90, 20],
+ "is_list_start_line": True,
+ "spans": [{"type": "text", "content": "one"}],
+ },
+ {
+ "bbox": [10, 25, 90, 35],
+ "is_list_start_line": True,
+ "spans": [{"type": "text", "content": "two"}],
+ },
+ ],
+ }
+ ],
+ "discarded_blocks": [],
+ }
+
+ pages = _current_pages([legacy_page_to_model_list(page)])
+ list_block = pages[0].blocks[0]
+
+ assert list_block.type == BlockType.LIST
+ assert [child.content[0].content for child in list_block.content] == ["one", "two"]
+
+
+def test_legacy_adapter_separates_office_blocks_with_synthetic_bbox_groups() -> None:
+ """验证无 page_size 时列表不会吸收相邻 Office 正文块。"""
+ page = {
+ "para_blocks": [
+ {"type": "text", "lines": [{"spans": [{"type": "text", "content": "before"}]}]},
+ {
+ "type": "list",
+ "blocks": [
+ {"type": "text", "lines": [{"spans": [{"type": "text", "content": "item"}]}]},
+ ],
+ },
+ {"type": "text", "lines": [{"spans": [{"type": "text", "content": "after"}]}]},
+ ],
+ "discarded_blocks": [],
+ }
+
+ pages = _current_pages([legacy_page_to_model_list(page)])
+
+ assert [block.type for block in pages[0].blocks] == [BlockType.TEXT, BlockType.LIST, BlockType.TEXT]
+ assert pages[0].blocks[0].content[0].content == "before"
+ assert pages[0].blocks[1].content[0].content[0].content == "item"
+ assert pages[0].blocks[2].content[0].content == "after"
+
+
+def test_doclib_compaction_converts_schema_v1_pages_with_current_adapter() -> None:
+ """验证 Doclib 历史调用点重新通过宿主适配器生成 2.0 page 字典。"""
+ payload = {
+ "schema_version": "1.0",
+ "pages": [
+ {
+ "page_idx": 3,
+ "page_size": [100, 100],
+ "preproc_blocks": [
+ {
+ "type": "text",
+ "bbox": [10, 10, 90, 20],
+ "lines": [_legacy_line("cached")],
+ }
+ ],
+ "discarded_blocks": [],
+ }
+ ],
+ }
+
+ pages = _normalize_batch_pages(payload)
+
+ assert pages[0]["page_idx"] == 3
+ assert pages[0]["blocks"][0]["content"] == [{"type": "text", "content": "cached"}]
+
+
+def test_doclib_compaction_rejects_unknown_legacy_schema() -> None:
+ """验证未知 schema 不会被误当成 3.4.5 页面吞掉。"""
+ with pytest.raises(ValueError, match="source reparse"):
+ _normalize_batch_pages({"schema_version": "99.0", "pages": [{"page_idx": 0, "blocks": []}]})
+
+
+def test_doclib_compaction_accepts_current_schema_without_legacy_conversion(monkeypatch: pytest.MonkeyPatch) -> None:
+ """验证当前 2.0 缓存直接读取页面,不进入旧版转换。"""
+ from mineru.backend.postprocess import legacy_schema_adapter
+
+ def unexpected_conversion(_page: dict) -> list[dict]:
+ """当前缓存触发旧版转换时立即报告路由错误。"""
+ raise AssertionError("current schema must not use legacy conversion")
+
+ monkeypatch.setattr(legacy_schema_adapter, "legacy_page_to_model_list", unexpected_conversion)
+ pages = [{"page_idx": 0, "blocks": []}]
+ document = MiddleJson(
+ pages=pages,
+ is_full_document=True,
+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version="test")},
+ )
+ assert _normalize_batch_pages(document.to_dict()) == document.to_dict()["pages"]
diff --git a/tests/unittest/test_doclib_middle_json.py b/tests/unittest/test_doclib_middle_json.py
new file mode 100644
index 00000000..367d9598
--- /dev/null
+++ b/tests/unittest/test_doclib_middle_json.py
@@ -0,0 +1,256 @@
+"""Doclib 专属历史文档读取的结构、来源及通用边界回归。"""
+
+from copy import deepcopy
+from collections.abc import Callable
+import json
+
+import pytest
+from docvortex.schema import MiddleJson
+
+from mineru.doclib.core.middle_json import read_cached_middle_json
+from mineru.parser import ParseResult
+
+
+def _legacy_v2() -> dict:
+ """构造带续接、标题、表格素材和应用扩展的旧 Schema 2.0 文档。"""
+ return {
+ "schema_version": "2.0",
+ "file_suffix": "pdf",
+ "mineru_version": "3.4.4",
+ "effort": "high",
+ "parse_mode": "ocr",
+ "is_full_document": False,
+ "extensions": {"application": {"items": [None, True, 1, {"type": "image", "image_base64": "keep"}]}},
+ "pages": [
+ {
+ "page_idx": 3,
+ "blocks": [
+ {
+ "type": "paragraph_title",
+ "index": 0,
+ "bbox": [0.1, 0.1, 0.9, 0.2],
+ "level": 4,
+ "anchor": "section",
+ "content": [{"type": "text", "content": "标题"}],
+ },
+ {
+ "type": "text",
+ "index": 1,
+ "bbox": [0.1, 0.3, 0.9, 0.4],
+ "continues_prev": True,
+ "content": [
+ {
+ "type": "hyperlink",
+ "url": "https://example.com",
+ "content": [{"type": "text", "content": "保留正文"}],
+ }
+ ],
+ },
+ {
+ "type": "table",
+ "index": 2,
+ "bbox": [0.1, 0.5, 0.9, 0.8],
+ "cell_merge": [0, 1],
+ "content": [
+ {
+ "type": "table_body",
+ "index": 2,
+ "bbox": [0.1, 0.5, 0.9, 0.8],
+ "content": "<table><tr><td>保留表格</td></tr></table>",
+ "image_path": "images/table.png",
+ }
+ ],
+ },
+ ],
+ },
+ {"page_idx": 5},
+ ],
+ }
+
+
+def _legacy_345(*, wrapped: bool = False) -> dict:
+ """构造真实旧字段布局及非连续页号,版本保留发行标签中的 3.4.4。"""
+ pages = [
+ {
+ "page_idx": index,
+ "page_size": [100, 200],
+ "preproc_blocks": [
+ {
+ "type": "text",
+ "bbox": [10, 20, 90, 40],
+ "lines": [{"bbox": [10, 20, 90, 40], "spans": [{"type": "text", "content": "正文 <eq>x^2</eq>"}]}],
+ },
+ ],
+ "discarded_blocks": [],
+ }
+ for index in [3, 5]
+ ]
+ payload = {"_version_name": "3.4.4", "_effort": "high", "_ocr_enable": True}
+ if wrapped:
+ payload.update(schema_version="1.0", pages=pages)
+ else:
+ payload["pdf_info"] = pages
+ return payload
+
+
[email protected]("wrapped", [False, True])
+def test_legacy_345_roundtrip_preserves_origin_and_pages(wrapped: bool) -> None:
+ """两个旧页面封装均可在缓存边界转换,输入和来源不受污染。"""
+ payload = _legacy_345(wrapped=wrapped)
+ before = deepcopy(payload)
+ result = read_cached_middle_json(payload)
+ assert payload == before
+ assert [page.page_idx for page in result.pages] == [3, 5]
+ assert result.is_full_document is False
+ assert result.metadata.producer.version == "3.4.4"
+ assert result.extensions["mineru"] == {"tier": "standard", "parse_mode": "ocr"}
+ assert result.pages[0].blocks[0].content[0].content == "正文 "
+ assert result.pages[0].blocks[0].content[1].content == "x^2"
+ assert read_cached_middle_json(result.to_dict()) == result
+ assert MiddleJson.from_json(result.to_json()) == result
+
+
+def test_v2_changes_only_envelope_without_postprocess(monkeypatch: pytest.MonkeyPatch) -> None:
+ """旧 2.0 的页面保持不变,转换不触发段落表格后处理或 LLM。"""
+ from mineru.backend.postprocess import document as host_document
+
+ def unexpected(*_args: object, **_kwargs: object) -> None:
+ """捕获任何越过外层迁移范围的后处理调用。"""
+ raise AssertionError("Envelope migration must not postprocess pages")
+
+ monkeypatch.setattr("docvortex.postprocess.document.model_json_to_middle_json", unexpected)
+ monkeypatch.setattr("docvortex.postprocess.pages.model_json_to_pages", unexpected)
+ monkeypatch.setattr(host_document, "apply_llm_aided_postprocess", unexpected)
+ monkeypatch.setattr(host_document, "build_middle_json", unexpected)
+ payload = _legacy_v2()
+ before = deepcopy(payload)
+ result = read_cached_middle_json(payload)
+ assert payload == before
+ assert result.to_dict()["pages"] == payload["pages"]
+ assert result.metadata.file_suffix == "pdf"
+ assert result.metadata.producer.version == "3.4.4"
+ assert result.extensions["application"] == payload["extensions"]["application"]
+ assert result.extensions["mineru"] == {"tier": "standard", "parse_mode": "ocr"}
+ assert read_cached_middle_json(result.to_dict()).to_dict() == result.to_dict()
+ result.extensions["application"]["items"].append("changed")
+ assert payload == before
+
+
+ "updates,expected",
+ [
+ ({}, None),
+ ({"_effort": "high"}, None),
+ ({"_ocr_enable": True}, None),
+ ({"_effort": "high", "_ocr_enable": False}, {"tier": "standard", "parse_mode": "txt"}),
+ ({"_effort": "xhigh", "parse_mode": "ocr"}, {"tier": "advanced", "parse_mode": "ocr"}),
+ ({"_effort": "unrecognized", "parse_mode": "txt"}, None),
+ ],
+)
+def test_missing_metadata_stays_unknown(updates: dict, expected: dict | None) -> None:
+ """缺失或不可识别的记录保持未知,不制造当前版本或默认解析档位。"""
+ payload = {"pdf_info": [], **updates}
+ result = read_cached_middle_json(payload)
+ assert result.metadata.producer.model_dump() == {"name": "mineru", "version": "unknown"}
+ assert result.metadata.file_suffix == "pdf"
+ assert result.extensions.get("mineru") == expected
+
+
+def test_v2_missing_product_records_preserve_application_extensions() -> None:
+ """旧 2.0 缺少版本及运行记录时也保持未知,应用扩展不受影响。"""
+ payload = _legacy_v2()
+ for key in ("mineru_version", "effort", "parse_mode"):
+ del payload[key]
+ document = read_cached_middle_json(payload)
+ assert document.metadata.producer.version == "unknown"
+ assert document.extensions == payload["extensions"]
+
+
+def test_consistent_existing_product_extension_is_preserved() -> None:
+ """已有可靠产品扩展与旧字段一致时直接保留,不产生重复记录。"""
+ payload = _legacy_v2()
+ payload["extensions"]["mineru"] = {"tier": "standard", "parse_mode": "ocr"}
+ assert read_cached_middle_json(payload).extensions == payload["extensions"]
+
+
+def test_345_conflicting_full_document_flag_fails() -> None:
+ """旧根字段与实际抽页映射矛盾时不得静默覆盖整本语义。"""
+ payload = _legacy_345()
+ payload["is_full_document"] = True
+ with pytest.raises(ValueError, match="Conflicting legacy is_full_document"):
+ read_cached_middle_json(payload)
+
+
+ "updates",
+ [
+ {"mineru_version": "other-version"},
+ {"effort": "medium"},
+ {"parse_mode": "txt"},
+ ],
+)
+def test_conflicting_legacy_metadata_aliases_fail(updates: dict) -> None:
+ """同一来源或运行参数的旧别名互相矛盾时明确报错,不静默选取一方。"""
+ payload = _legacy_345()
+ payload.update(updates)
+ with pytest.raises(ValueError, match="Conflicting legacy"):
+ read_cached_middle_json(payload)
+
+
+ "mutate",
+ [
+ lambda p: p.update(schema="docvortex.middle", schema_version="1.0"),
+ lambda p: p.update(schema=None),
+ lambda p: p.update(schema="other.middle"),
+ lambda p: p.update(metadata={}),
+ lambda p: p.update(schema_version="99.0"),
+ lambda p: p.update(file_suffix="unsupported"),
+ lambda p: p.pop("is_full_document"),
+ lambda p: p["extensions"].update(mineru={"tier": "basic", "parse_mode": "ocr"}),
+ ],
+)
+def test_ambiguous_or_conflicting_v2_is_rejected(mutate: Callable[[dict], object]) -> None:
+ """新协议损坏、未知版本及保留字段冲突不能被历史兼容掩盖。"""
+ payload = _legacy_v2()
+ mutate(payload)
+ with pytest.raises(ValueError):
+ read_cached_middle_json(payload)
+
+
+ "payload",
+ [
+ {"schema_version": "1.0", "pages": [{"page_idx": 0, "blocks": []}]},
+ {"pdf_info": [{"page_idx": 0, "blocks": []}]},
+ {"pdf_info": [], "pages": []},
+ {"pdf_info": [], "schema_version": "99.0"},
+ {"pages": []},
+ ],
+)
+def test_mixed_pages_are_not_silently_emptied(payload: dict) -> None:
+ """拒绝把当前 blocks 误当作旧页面后悄悄丢弃正文。"""
+ with pytest.raises(ValueError):
+ read_cached_middle_json(payload)
+
+
[email protected]("indices", [[3, 3], [5, 3], [-1, 3], [True, 3]])
+def test_invalid_legacy_page_indices_fail(indices: list) -> None:
+ """旧页号仍受唯一、顺序、非负和整数约束。"""
+ payload = _legacy_345()
+ for page, index in zip(payload["pdf_info"], indices):
+ page["page_idx"] = index
+ with pytest.raises(ValueError):
+ read_cached_middle_json(payload)
+
+
[email protected]("family", ["345", "v1", "v2"])
+def test_compatibility_is_not_exposed_by_generic_readers(family: str) -> None:
+ """Doclib 兼容成功不改变通用 ParseResult 或 DocVortex 的严格边界。"""
+ payload = _legacy_v2() if family == "v2" else _legacy_345(wrapped=family == "v1")
+ assert read_cached_middle_json(payload).pages
+ for reader in [ParseResult.from_dict, MiddleJson.from_dict]:
+ with pytest.raises(ValueError):
+ reader(payload)
+ with pytest.raises(ValueError):
+ ParseResult.from_json(json.dumps(payload))
diff --git a/tests/unittest/test_flash_pdf_render_scheduling.py b/tests/unittest/test_flash_pdf_render_scheduling.py
new file mode 100644
index 00000000..4c1dceee
--- /dev/null
+++ b/tests/unittest/test_flash_pdf_render_scheduling.py
@@ -0,0 +1,166 @@
+"""验证 Flash 原生按需截图与 MinerU 配置、选页和后处理边界。"""
+
+from __future__ import annotations
+
+import asyncio
+from pathlib import Path
+from types import SimpleNamespace
+from typing import Any
+from unittest.mock import AsyncMock, MagicMock
+
+import pytest
+from PIL import Image
+from reportlab.pdfgen.canvas import Canvas
+
+from docvortex.analyzers.native.models import PdfModel
+from docvortex.document.pdf import PDFDocument, images
+from mineru.backend.analysis.pdf import window
+from mineru.backend.postprocess import llm_aided
+from mineru.config import LLMAidedConfig, LLMAidedFeaturesConfig, config
+from mineru.parser import parse, parse_async
+
+
+def _source(path: Path) -> Path:
+ """构造五页 PDF,让实际输入准备负责连续和非连续选页。"""
+ canvas = Canvas(str(path))
+ for index in range(5):
+ canvas.drawString(60, 720, f"Page {index + 1}")
+ canvas.showPage()
+ canvas.save()
+ return path
+
+
[email protected]("asynchronous", [False, True])
[email protected]("page_range,indices", [("all", [0, 1, 2, 3, 4]), ("2-4", [1, 2, 3]), ("1,3,5", [0, 2, 4])])
+def test_flash_sparse_render_keeps_page_mapping_and_title_gate(
+ monkeypatch: pytest.MonkeyPatch,
+ tmp_path: Path,
+ asynchronous: bool,
+ page_range: str,
+ indices: list[int],
+) -> None:
+ """真实同步和异步入口只渲染有图页,页号和整本标题开关仍由 MinerU 管理。"""
+ document_sizes: list[int] = []
+ rendered: list[int] = []
+ created: list[Image.Image] = []
+
+ def predict(_model: PdfModel, document: PDFDocument) -> list[list[dict[str, Any]]]:
+ """替换原生语义分析,每个物理页含标题,奇数物理页额外含图片。"""
+ document_sizes.append(document.page_count)
+ pages: list[list[dict[str, Any]]] = []
+ for index in range(document.page_count):
+ blocks = [
+ {
+ "type": "paragraph_title",
+ "bbox": [0.1, 0.1, 0.9, 0.2],
+ "level": 2,
+ "lines": [{"bbox": [0.1, 0.1, 0.9, 0.2]}],
+ "content": [{"type": "text", "content": f"Section {index}"}],
+ }
+ ]
+ if index % 2:
+ blocks.append({"type": "image", "bbox": [0.1, 0.3, 0.9, 0.8]})
+ pages.append(blocks)
+ return pages
+
+ def raster(data: bytes, **options: Any) -> list[dict[str, Any]]:
+ """以真实物理页索引生成颜色图片,同时检查宿主设置优先于引擎默认值。"""
+ assert data.startswith(b"%PDF")
+ assert options["timeout"] == 19 and options["threads"] == 2
+ batch = list(range(options["start_page_id"], options["end_page_id"] + 1))
+ rendered.extend(batch)
+ result = []
+ for index in batch:
+ image = Image.new("RGB", (40, 20), (index * 30, 60, 120))
+ created.append(image)
+ result.append({"img_pil": image})
+ return result
+
+ monkeypatch.setenv("MINERU_PROCESSING_WINDOW_SIZE", "2")
+ monkeypatch.setenv("MINERU_PDF_RENDER_TIMEOUT", "19")
+ monkeypatch.setenv("MINERU_PDF_RENDER_THREADS", "2")
+ monkeypatch.setenv("DOCVORTEX_PDF_RENDER_TIMEOUT", "31")
+ monkeypatch.setattr(PdfModel, "predict", predict)
+ monkeypatch.setattr(images, "load_images_from_pdf_bytes_range", raster)
+ forbidden = MagicMock(side_effect=AssertionError("Flash TXT must skip inference windows"))
+ monkeypatch.setattr(window, "_get_window_pdf_pages", forbidden)
+ title = AsyncMock()
+ client = SimpleNamespace(close=AsyncMock())
+ monkeypatch.setattr(llm_aided, "LLMAidedClient", MagicMock(return_value=client))
+ monkeypatch.setattr(llm_aided, "apply_llm_title_leveling", title)
+ monkeypatch.setattr(
+ config,
+ "llm_aided",
+ LLMAidedConfig(
+ api_key="isolated-test",
+ features=LLMAidedFeaturesConfig(title_leveling=True),
+ ),
+ )
+ source = _source(tmp_path / "pages.pdf")
+ options = {"tier": "flash", "ocr_mode": "txt", "image_analysis": False, "page_range": page_range}
+ result = asyncio.run(parse_async(source, **options)) if asynchronous else parse(source, **options)
+
+ full = page_range == "all"
+ assert document_sizes == [len(indices)]
+ assert rendered == list(range(1, len(indices), 2))
+ assert [page.page_idx for page in result.pages] == indices
+ assert result._model_output.page_index_map == ([] if full else indices)
+ assert result.middle_json.is_full_document is full
+ assert result._model_output.metadata.producer.name == "mineru"
+ assert result.middle_json.extensions == result._model_output.extensions
+ assert title.await_count == int(full)
+ if full:
+ assert title.await_args.args[0] is result.pages
+ for index in rendered:
+ assert result._model_output.pages[index][1]["image_base64"].startswith("data:image/jpeg;base64,")
+ for image in created:
+ with pytest.raises(ValueError):
+ image.getpixel((0, 0))
+ forbidden.assert_not_called()
+
+
[email protected]("parse_mode", ["txt", "auto"])
+def test_flash_text_only_does_not_render(monkeypatch: pytest.MonkeyPatch, tmp_path: Path, parse_mode: str) -> None:
+ """真实原生解析的纯文本 PDF 不创建页图,auto 仍只在入口分类一次。"""
+ classify = MagicMock(return_value="txt")
+ monkeypatch.setattr(PDFDocument, "classify", classify)
+ forbidden = MagicMock(side_effect=AssertionError("Text-only PDF must not rasterize"))
+ monkeypatch.setattr(images, "load_images_from_pdf_bytes_range", forbidden)
+ result = parse(_source(tmp_path / "text.pdf"), tier="flash", ocr_mode=parse_mode, image_analysis=False)
+ assert len(result.pages) == 5
+ assert classify.call_count == int(parse_mode == "auto")
+ forbidden.assert_not_called()
+
+
+ "values,expected",
+ [(None, (64, 300, 3)), (("bad", "0", "-1"), (64, 300, 3)), (("0", "17", "2"), (1, 17, 2))],
+)
+def test_flash_preserves_render_config_fallbacks(
+ monkeypatch: pytest.MonkeyPatch,
+ values: tuple[str, str, str] | None,
+ expected: tuple[int, int, int],
+) -> None:
+ """沿用宿主现有环境变量缺省、非法值和窗口下限规则。"""
+ names = ("MINERU_PROCESSING_WINDOW_SIZE", "MINERU_PDF_RENDER_TIMEOUT", "MINERU_PDF_RENDER_THREADS")
+ for index, name in enumerate(names):
+ monkeypatch.delenv(name, raising=False)
+ if values is not None:
+ monkeypatch.setenv(name, values[index])
+ pages: list[list[dict[str, Any]]] = [[]]
+ monkeypatch.setattr(PdfModel, "predict", MagicMock(return_value=pages))
+ attach = MagicMock()
+ monkeypatch.setattr(window, "attach_visual_block_images_from_pdf", attach)
+ document = MagicMock(page_count=1)
+ actual = window.process_pdf_windows(
+ b"pdf",
+ document,
+ effort="flash",
+ parse_mode="txt",
+ image_analysis=False,
+ flash_txt_mode=True,
+ hybrid_model=None,
+ vlm_predictor=None,
+ )
+ assert actual is pages
+ attach.assert_called_once_with(document, pages, window_size=expected[0], timeout=expected[1], threads=expected[2])
diff --git a/tests/unittest/test_pdf_analyze_visual_blocks.py b/tests/unittest/test_pdf_analyze_visual_blocks.py
index 92114f18..28b983e5 100644
--- a/tests/unittest/test_pdf_analyze_visual_blocks.py
+++ b/tests/unittest/test_pdf_analyze_visual_blocks.py
@@ -1668,7 +1668,7 @@ def test_doc_analyze_flash_demo1_uses_canonical_equation_type() -> None:
def test_doc_analyze_flash_returns_complete_model_json_and_typed_middle_json(monkeypatch: pytest.MonkeyPatch) -> None:
- """验证 Flash 多窗口补充完整 raw pages,并返回严格 ModelJson 与 MiddleJson。"""
+ """验证 Flash 按需窗口补充完整 raw pages,并返回严格 ModelJson 与 MiddleJson。"""
from docvortex.analyzers import native as flash_model
events: list[str] = []
@@ -1699,6 +1699,8 @@ def test_doc_analyze_flash_returns_complete_model_json_and_typed_middle_json(mon
]
fake_pdf_doc = MagicMock()
fake_pdf_doc.page_count = len(source_model_list)
+ fake_pdf_doc.bytes = b"fake-pdf"
+ fake_pdf_doc.page_size.return_value = (40, 20)
fake_pdf_doc.__getitem__.side_effect = lambda page_idx: MagicMock(page_idx=page_idx)
def fake_document_close() -> None:
@@ -1718,11 +1720,13 @@ def test_doc_analyze_flash_returns_complete_model_json_and_typed_middle_json(mon
requested_ranges: list[tuple[int, int]] = []
def fake_load_images_for_window(
- *,
pdf_bytes: bytes,
+ *,
start_page_id: int,
end_page_id: int,
image_type: str,
+ timeout: int | None = None,
+ threads: int | None = None,
) -> list[dict[str, Image.Image]]:
"""按请求范围生成测试页图,并记录窗口以验证分段与释放行为。"""
assert pdf_bytes == b"fake-pdf"
@@ -1735,7 +1739,7 @@ def test_doc_analyze_flash_returns_complete_model_json_and_typed_middle_json(mon
rendered_images.extend(window_images)
return [{"img_pil": image} for image in window_images]
- original_attach_visual_block_images = visuals._attach_visual_block_images
+ original_attach_visual_block_images = visuals._attach_prepared_visual_block_images
def tracked_attach_visual_block_images(*args: object, **kwargs: object) -> None:
"""记录视觉块补图顺序并调用真实实现。"""
@@ -1759,8 +1763,8 @@ def test_doc_analyze_flash_returns_complete_model_json_and_typed_middle_json(mon
monkeypatch.setattr(pipeline, "PDFDocument", lambda _: fake_pdf_doc)
monkeypatch.setattr(window, "_configured_window_size", lambda default: 2)
- monkeypatch.setattr(window, "load_images_from_pdf_bytes_range", fake_load_images_for_window)
- monkeypatch.setattr(window, "_attach_visual_block_images", tracked_attach_visual_block_images)
+ monkeypatch.setattr("docvortex.document.pdf.images.load_images_from_pdf_bytes_range", fake_load_images_for_window)
+ monkeypatch.setattr(visuals, "_attach_prepared_visual_block_images", tracked_attach_visual_block_images)
monkeypatch.setattr(pipeline, "_normalize_pdf_model_list", tracked_normalize_model_list)
monkeypatch.setattr(pipeline.time, "perf_counter", fake_perf_counter)
monkeypatch.setattr(flash_model, "PdfModel", MagicMock(return_value=fake_pdf_model))
@@ -1779,7 +1783,7 @@ def test_doc_analyze_flash_returns_complete_model_json_and_typed_middle_json(mon
assert model_json.pages == source_model_list
assert model_json.page_index_map == [7, 8, 9]
assert model_json.is_full_document is False
- assert requested_ranges == [(0, 1), (2, 2)]
+ assert requested_ranges == [(1, 1), (2, 2)]
assert inline_text(model_json.pages[0][0]["content"]) == "第一页 x+y"
assert [span["content"] for span in model_json.pages[0][0]["content"] if span.get("type") == "equation_inline"] == ["x+y"]
assert model_json.pages[2][0]["content"] == "z"