Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.
Python ai4sciencedocument-analysisdocxextract-datalayout-analysisocrparserpdfpdf-converterpdf-extractor-llmpdf-extractor-pretrainpdf-extractor-ragpdf-parserpptxpythonxlsx
Update Middle JSON to schema 2.0 with unified metadata handling and remove legacy support
myhlolica15b938262f
diff --git a/docs/next/api/changes.md b/docs/next/api/changes.mdindex 4e5f0432..2de23a5b 100644--- a/docs/next/api/changes.md+++ b/docs/next/api/changes.md@@ -1,3 +1,9 @@+# 共享文档 JSON 2.0++Model/Middle 输出切换为统一 schema 身份及 metadata 分组,产品信息为实际 tier/parse_mode。+历史 JSON 不再读取,自建 HTTP 服务与客户端需同步升级;请求参数和输出文件名不变。+详见 [协议](../middle-json/envelope.md) 和 [迁移](../middle-json/migration.md)。+# API 变更记录状态: Implementeddiff --git a/docs/next/code-map.md b/docs/next/code-map.mdindex ec283800..e063046f 100644--- a/docs/next/code-map.md+++ b/docs/next/code-map.md@@ -26,7 +26,7 @@| `mineru/doclib/` | 本地文档库常驻服务:入库、SHA256 去重、解析调度、缓存、搜索、配置。FastAPI + SQLite(WAL) + FTS5,路由→`services/`→`core/` 三层 | `doclib/server.py`(2439 行);`doclib/services/{parse,search,scan,config,cleanup}_svc.py`;`doclib/core/{db,fts,file_io}.py` || `mineru/doclib/background/` | 后台 asyncio 任务:ingest(入库)、parse_worker(解析执行)、scan_worker / watch(目录监听)、parse_server_health(parse-server 托管与健康探测)、device_monitor、compaction(批次合并)、telemetry_flush | 各文件顶部 docstring;`background/parse_worker.py:15` || `mineru/parser/` | parse-server 与解析门面:`api_server.py`(2495 行,独立进程)、`api_client.py`(HTTP 客户端)、`tier.py`(tier/backend 解析)、`mineru_parser.py`(`MinerUParser` 统一解析入口)、`base.py`(Middle JSON schema) | `parser/mineru_parser.py`、`parser/tier.py` |-| `mineru/backend/` | 解析引擎:`analyze.py`(`doc_analyze` 统一门面)→ `analysis/`(按格式分派:`pdf/`、`csv.py`、`epub.py`、`html.py`、`ofd.py`、`office.py`)→ `postprocess/`(Middle JSON 组装、段落/标题/列表/表格合并、`llm_aided.py`、`legacy_schema_adapter.py`) | `backend/analyze.py`、`backend/analysis/pdf/pipeline.py` |+| `mineru/backend/` | 解析引擎:`analyze.py`(`doc_analyze` 统一门面)→ `analysis/`(按格式分派:`pdf/`、`csv.py`、`epub.py`、`html.py`、`ofd.py`、`office.py`)→ `postprocess/`(Middle JSON 组装、段落/标题/列表/表格合并、`llm_aided.py`) | `backend/analyze.py`、`backend/analysis/pdf/pipeline.py` || `mineru/model/` | 模型层:`flash/`(纯 CPU 本地结构化解析器族:pdf / office / epub / html / ofd / csv);`runtime/`(`device.py` 设备与 light/full 栈、`hybrid.py` 原子模型分派、`onnx.py`);`layout/ mfr/ ocr/ table/ vlm/`(ML 模型实现);`_internal/pytorchocr` | `model/runtime/device.py:48`、`model/runtime/hybrid.py` || `mineru/render/` | 输出渲染:markdown / content_list(v2) / structured_content / docx / epub / html / pdf | `render/api.py`(唯一渲染入口)、`render/contracts.py` || `mineru/kit/` | 服务端与 Web 工具 CLI:`parse` / `gradio` / `api-server` / `vlm-server` / `router` / `models`,面向无状态批处理、自部署和多卡部署 | `kit/main.py`、`kit/commands/*.py`、`kit/gradio/` |diff --git a/docs/next/invariants.md b/docs/next/invariants.mdindex 79b01ce8..268e0abc 100644--- a/docs/next/invariants.md+++ b/docs/next/invariants.md@@ -32,7 +32,7 @@ Middle JSON 的对外 schema 固定为 `2.0`(`mineru/parser/base.py:20` `MIDDL红线:-- 不输出 `_backend` / `_version_name` / `pdf_info` / `_ocr_enable` / `_vlm_ocr_enable` 等旧字段;旧字段只能作为受支持旧 payload 的迁移输入出现(legacy 读取路径见 `parser/base.py:20-47` 与 `backend/postprocess/legacy_schema_adapter.py`)。+- 不接受旧 `_backend` / `_version_name` / `pdf_info` 等历史文档字段;加载统一验证 schema 身份、版本和 metadata,不执行旧协议迁移。- 改动输出结构即破坏性变更:必须走 ADR 流程,并同步 `docs/next/api/changes.md`、`docs/next/middle-json.md` 与 compaction(`doclib/background/compaction.py:27`)。## 4. 依赖方向:基础层不得反向引用上层diff --git a/docs/next/middle-json.md b/docs/next/middle-json.mdindex 05c77490..817a0b84 100644--- a/docs/next/middle-json.md+++ b/docs/next/middle-json.md@@ -8,7 +8,7 @@## 当前定位-Middle JSON 已经收敛为 schema 2.0 的严格 Pydantic 模型:`mineru/types.py` 中的 `MiddleJson`、`PageInfo`、Block 联合与 InlineSpan 联合。自然语言使用结构化 Span,不再使用字符串标签协议。+Middle JSON 已经收敛为 schema 2.0 的严格 Pydantic 模型:`docvortex.schema` 定义、`mineru/types.py` 重导出的 `MiddleJson`、`PageInfo`、Block 联合与 InlineSpan 联合。自然语言使用结构化 Span,不再使用字符串标签协议。## 目录@@ -27,8 +27,7 @@ Middle JSON 已经收敛为 schema 2.0 的严格 Pydantic 模型:`mineru/types- 以当前代码中的 Pydantic 严格模型为起点(schema 2.0)。- 区分“已经解决”、“部分解决”、“仍需工作”。- Agent 引用与稳定 page/block locator 是 P0 目标。-- MinerU 3.4.5 页面及对应 1.0 pages 包装经 `legacy_schema_adapter` 转换;其它旧 payload- 和无法识别的缓存视为 stale,必须从源文件重新解析。+- 统一使用带 `schema` 身份的 2.0 协议及 `metadata`;所有历史文档协议均明确拒绝,旧缓存须重新解析。- render 统一不能只看入口 facade,还要收敛格式-specific 分支。## 与其他文档的关系diff --git a/docs/next/middle-json/README.md b/docs/next/middle-json/README.mdindex 44d10ff3..5a77ccba 100644--- a/docs/next/middle-json/README.md+++ b/docs/next/middle-json/README.md@@ -34,7 +34,7 @@ InlineContentBlockMiddle JSON 下一版要达到以下目标:-1. 统一顶层 envelope,运行时只接受 schema 2.0 `pages` 结构;旧 payload(3.4.5 `pdf_info`、schema 1.0 `pages` 包装)仅通过 `from_dict()` 的 legacy 分支单向迁移,无版本号的裸 `{"pages": []}` 直接拒绝并要求重新解析。+1. 统一使用带协议身份和 metadata 的 JSON 2.0;所有历史 JSON 必须重新解析,见 [envelope.md](envelope.md)。2. 明确 `PageInfo` / `Block` / `InlineSpan` 字段契约。3. 为每个 backend 提供 normalization 任务清单。4. 定义 Agent 可引用的稳定 locator 规则。diff --git a/docs/next/middle-json/current-medium.md b/docs/next/middle-json/current-medium.mdindex 8fa2ff16..fb049e3a 100644--- a/docs/next/middle-json/current-medium.md+++ b/docs/next/middle-json/current-medium.md@@ -8,7 +8,7 @@## 标准来源-事实标准由 `mineru/types.py` 中的严格 Pydantic v2 模型定义:+事实标准由 `docvortex.schema` 中的严格 Pydantic v2 模型定义,`mineru/types.py` 重导出:- `ModelJson`、`MiddleJson`、`PageInfo`;- `Block` / `PageBlock` discriminated union;@@ -19,9 +19,8 @@`tuple[MiddleJson, ModelJson]`,`ParseResult.to_dict()` 输出`schema_version="2.0"`。-Schema 2.0 是严格的运行时与公开输出契约(3.0 从未发布)。`ParseResult.from_dict()` 直接读取 2.0,-并仅将可识别的 MinerU 3.4.5 `pdf_info` 或对应 schema 1.0 `pages` 包装经运行时 legacy 分支单向迁移为 2.0;-其它缺失版本号的 envelope 与未知旧 payload 必须从源文件重新解析。+读取必须匹配 `schema` 与 `schema_version`,详见[统一外层协议](envelope.md)。+所有历史文档协议和缺少身份的 JSON 均拒绝,不提供自动迁移。## ModelJson@@ -29,10 +28,9 @@ Schema 2.0 是严格的运行时与公开输出契约(3.0 从未发布)。`P|------|------|------|| `pages` | `list[list[dict]]` | Analyze 产生的 raw blocks || `page_index_map` | `list[int]` | 空列表表示整本;非空时与 pages 等长、唯一且严格递增 |-| `file_suffix` | `pdf/doc/docx/ppt/pptx/xls/xlsx/rtf/csv/epub/html/ofd/odt/ods/odp` | 必填 |-| `effort` | `flash/medium/high/xhigh` | 必填 |-| `parse_mode` | `txt/ocr` | 必填 |-| `mineru_version` | 非空字符串 | 必填 |+| `schema` / `schema_version` | 协议标识及版本 | Model/Middle 各自身份,版本 2.0 |+| `metadata` | DocumentMetadata | 必填 file_suffix 与 producer |+| `extensions` | JSON 字典 | 可选产品记录,默认输出空对象 |文本型 raw block 必须直接携带 InlineSpan 列表。校验失败会报告`pages[页号][块号]`。PDF 的扁平 `list/index` layout block 在对象化前也可以携带@@ -44,10 +42,9 @@ Span;形成递归容器后,其 `content` 改为子 block 列表。|------|------|------|| `pages` | `list[PageInfo]` | `page_idx` 唯一且严格递增 || `is_full_document` | `bool` | 必填,不提供默认值 |-| `file_suffix` | 与 ModelJson 相同 | 必填 |-| `effort` | `flash/medium/high/xhigh` | 必填 |-| `parse_mode` | `txt/ocr` | 必填 |-| `mineru_version` | 非空字符串 | 必填 |+| `schema` / `schema_version` | 协议标识及版本 | Model/Middle 各自身份,版本 2.0 |+| `metadata` | DocumentMetadata | 必填 file_suffix 与 producer |+| `extensions` | JSON 字典 | 可选产品记录,默认输出空对象 |`PageInfo` 只包含 `page_idx` 与 `blocks`。顶层 block 必须具有唯一且严格递增的`index`。PDF 与 OFD 属于固定版式输入,其顶层 block 还必须具有有效的 `[0, 1]`@@ -151,10 +148,7 @@ Markdown、HTML、DOCX 与 Structured Content renderer 按 Span discriminator分派。HTML versioned wire 使用明确 DOM metadata 往返恢复类型、样式、公式和链接,不从渲染后的标签字符串猜测语义。-## 缓存与兼容+## 缓存与协议-- `ParseResult.from_dict()` 直接接受 `schema_version="2.0"`;-- MinerU 3.4.5 `pdf_info` 与对应 1.0 pages 包装经 `legacy_schema_adapter` 在运行时(`from_dict()` 与 doclib compaction)回推;-- Doclib compaction 使用同一适配器迁移可识别旧 batch,无法识别时仍按 stale 处理;-- 无版本号的裸 `{"pages": []}` 与其它未知旧版本返回“重新解析源文件”的明确错误;-- 不提供 `str | list[InlineSpan]` 联合,也不恢复旧版带 bbox/图片职责的 Line/Span。+只接受当前完整文档协议;旧结果不命中缓存、不参与压缩,并提示从源文件重建。+压缩不得合并不同来源、扩展或整本标识的数据。Block/Span 类型和内部语义保持不变。diff --git a/docs/next/middle-json/envelope.md b/docs/next/middle-json/envelope.mdindex 6df574b3..4e771aa2 100644--- a/docs/next/middle-json/envelope.md+++ b/docs/next/middle-json/envelope.md@@ -1,221 +1,58 @@-# Canonical Envelope+# Model / Middle JSON 统一外层协议-状态: Draft-读者: SDK 开发者、API 开发者、doclib 开发者-范围: Middle JSON 顶层结构、metadata、版本和兼容输入-来源: 由根目录旧 Middle JSON 底稿迁移整理而来+状态:Implemented,DocVortex 0.2.0 起直接切换。-## 目标结构--当前 P0 写出的 `pages` 结构先包含顶层 `schema_version` 和 `pages`,暂不写 `_meta`:+两边共用 `docvortex.schema` 定义的类型和 JSON 2.0,MinerU 不再维护独立的文档封装。```json{+ "schema": "docvortex.middle","schema_version": "2.0",+ "metadata": {+ "file_suffix": "pdf",+ "producer": {"name": "mineru", "version": "4.0.0b1"}+ },+ "extensions": {"mineru": {"tier": "standard", "parse_mode": "txt"}},+ "is_full_document": true,"pages": []}```-后续 canonical envelope 目标结构:--```json-{- "schema_version": "2.0",- "pages": [],- "_meta": {- "mineru_version": "2.x",- "backend": "hybrid",- "tier": "basic",- "file": {- "sha256": "...",- "page_count": 12,- "filename": null- },- "features": {- "ocr_enabled": true,- "vlm_ocr_enabled": false- },- "models": {}- }-}-```--设计选择:--- 新结构使用 `pages`,对应 `ParseResult.pages`。-- 运行时直接读取 schema 2.0 `pages` 结构;MinerU 3.4.5 `pdf_info` 与 schema 1.0 `pages` 包装由 `ParseResult.from_dict()` 与 doclib compaction 的运行时 legacy 分支单向迁移,不是离线工具专属;无版本号的裸 `{"pages": []}` 被拒绝并要求重新解析。-- 当前 `ParseResult.to_dict()` 只输出 `schema_version` 与 MiddleJson 顶层字段,不写顶层 `_backend`/`_meta`。-- 若后续引入 canonical `_meta`,可由 `_meta.backend` 表达 backend 维度;当前 schema 2.0 没有 `_backend` 字段(是否引入 `_meta` 见 [open-questions.md](../open-questions.md))。-- `schema_version` 放在顶层,便于快速判断 migration。-- 代码常量定义为 `mineru.parser.MIDDLE_JSON_SCHEMA_VERSION`,由 normalize、validate、writer 和 exporter 统一引用。-- 当前 P0 写出路径只增加 `schema_version`,不新增 `_meta`;`_meta` 由后续 canonical envelope migration / writer 引入。--## 字段--| 字段 | 类型 | 必带 | 说明 |-|------|------|:--:|------|-| `schema_version` | string | 是 | 当前 `"2.0"`;代码常量为 `MIDDLE_JSON_SCHEMA_VERSION`。 |-| `pages` | list[PageInfo] | 是 | typed pages 的 JSON 表达。 |-| `is_full_document` | bool | 是 | 是否整本文档解析(空 `page_index_map` 时为 `true`)。 |-| `file_suffix` | string | 是 | 输入文件类型(`pdf`、`docx`、`pptx`、`epub`、`html`、`ofd` 等)。 |-| `effort` | string | 是 | 分析强度:`flash`、`medium`、`high`、`xhigh`。 |-| `parse_mode` | string | 是 | `txt` 或 `ocr`。 |-| `mineru_version` | string | 是 | 生成该结果的 MinerU 版本。 |-| `_meta` | object | 后续 | 元数据;当前 P0 写出路径暂不增加。 |--## `_meta`--| 字段 | 类型 | 必带 | 说明 |-|------|------|:--:|------|-| `mineru_version` | string | 是 | 生成该结果的 MinerU 版本。 |-| `backend` | string | 是 | `hybrid`、`office`、`html`、`flash`。 |-| `tier` | string 或 null | 是 | `flash`、`basic`、`standard`、`advanced` 的解析结果语义;未经过 tier 解析的工具层结果可为 `null`。 |-| `file` | object | 是 | 文件级信息。 |-| `features` | object | 是 | 本次解析启用的能力。 |-| `models` | object | 是 | 实际模型信息。 |-| `parsed_at` | string 或 null | 否 | 解析时间。默认不参与 locator。 |--## `_meta.file`--| 字段 | 类型 | 必带 | 说明 |-|------|------|:--:|------|-| `sha256` | string | 是 | 原文件 SHA-256。Agent locator 严格校验时需要。 |-| `page_count` | integer | 是 | `len(pages)`。 |-| `filename` | string 或 null | 否 | 原文件名。存在隐私争议,默认可为 null。 |--如果缺少 `sha256`,可以生成局部 locator,但不能生成可严格校验的跨文档 block reference。migration 可以允许临时为空,但 Agent citation 功能必须明确降级或报错。--## `_meta.features`--建议字段:--| 字段 | 类型 | 说明 |-|------|------|------|-| `ocr_enabled` | bool | 是否启用 OCR。 |-| `vlm_ocr_enabled` | bool | Hybrid 中 VLM OCR 是否启用。 |-| `image_analysis` | bool | 是否启用图片分析。 |--features 是开放字典。新增字段不破坏 schema。--## `_meta.models`--建议字段:--| 字段 | 类型 | 说明 |-|------|------|------|-| `layout` | string | layout 模型。 |-| `ocr` | string | OCR 模型。 |-| `formula` | string | 公式模型。 |-| `table` | string | 表格模型。 |-| `vlm` | string | VLM 模型。 |--models 也是开放字典。字段粒度可以随 backend 增加。--## 兼容输入--`ParseResult.from_dict()` 在运行时接受 schema 2.0 envelope 与两类可识别 legacy 输入。目标读入函数可支持以下输入:--### 1. Canonical envelope--```json-{- "schema_version": "2.0",- "pages": [],- "_meta": {}-}-```--直接读取。--### 2. 历史 1.0 `pages` 包装--```json-{- "schema_version": "1.0",- "pages": []-}-```--`schema_version` 为 `"1.0"` 且 `pages` 为 page dict 列表时,经 `legacy_schema_adapter` 单向回推为 raw model-list 后重走统一后处理,生成 2.0 结果。--### 3. 历史旧 CLI middle_json(MinerU 3.4.5 `pdf_info`)--```json-{- "pdf_info": [],- "_backend": "hybrid",- "_version_name": "2.x"-}-```--`pdf_info` 是 `ParseResult.from_dict()` 的运行时兼容分支(按 envelope 识别,不要求 `_version_name` 作为唯一判据):经 `legacy_schema_adapter` 单向迁移为 2.0。旧产物中的 `_backend: "pipeline"` 等字段不参与迁移语义,产物版本以文件内版本信息为准。--### 4. 被拒绝的输入--无 `schema_version` 且无 `pdf_info` 的裸 `{"pages": []}`、`schema_version: "3.0"` 与其它未知旧版本都会抛出“重新解析源文件”的明确错误,不做静默迁移。--## Migration 函数--目标 API:--```python-def normalize_middle_json(- payload: dict | list,- *,- sha256: str | None = None,- backend: str | None = None,- tier: str | None = None,- filename: str | None = None,-) -> dict: ...-```--规则:--1. 运行时 payload 必须是 dict,且 `pages` 必须是 list。-2. 历史 `pdf_info` 与 schema 1.0 `pages` 包装由 `ParseResult.from_dict()` / doclib compaction 的运行时 legacy 分支单向迁移;不可识别的旧 payload 标记为 stale 并要求重新解析,不做静默转换。-3. 如果没有 sha256,则保留 null,但禁用需要严格校验 source identity 的 citation 能力。-4. 输出必须是 canonical envelope。--## Validator--当前生产代码不提供独立的 envelope validator API;结构合法性由 strict Pydantic 模型(`MiddleJson` 等,`extra="forbid"` + `strict`,`mineru/types.py:1092`)在解析时强制,页面树语义校验仅保留在单测中作为 test-local helper。--```python-from mineru.parser import MIDDLE_JSON_SCHEMA_VERSION-```--目标 envelope API:--```python-def validate_middle_json(payload: dict) -> list[ValidationIssue]: ...-```--P0 校验:+Model 使用 `docvortex.model`、`page_index_map` 及原来的逐页 raw block 列表;+Middle 保留 `is_full_document`、PageInfo、Block 和 Span。默认写出保留协议头、+完整 metadata 和空 extensions。JSON Schema 由同一组声明字段生成。-- 有 `schema_version`。-- 有 `pages` list。-- 每个 page 有 `page_idx`。-- 每个 block 有 `index`、`type`;固定版式(`pdf`/`ofd`)顶层 block 必须有 `bbox`。-- 自然语言 block 的 `content` 是合法的 `InlineSpan` 列表(`TextSpan` / `EquationInlineSpan` / `CodeInlineSpan` / `HyperlinkSpan`),span 只表达行内语义,不携带 bbox。-- `page_count == len(pages)`。+Python 使用 `document.metadata.file_suffix` 和 `document.metadata.producer`,+旧顶层字段和转发属性全部移除。构造器要求明确提供格式和生产者;解析入口参数不变。+`ModelJson.from_dict/from_json`、`MiddleJson.from_dict/from_json` 是统一读取入口,+`ParseResult.from_dict/from_json` 直接委托并校验可选的 MinerU 扩展。-P1 校验:+## 来源与产品扩展-- block index 页内唯一。-- locator 可生成。-- bbox 为 schema 约定的 0-1 归一化坐标或 `null`(unknown)。-- 内部字段不出现在 public output。+`producer` 只表达真实生成来源。加载、渲染、保存不改写,Model 到 Middle 深拷贝+来源和扩展。没有 MinerU 扩展的 DocVortex 文档可以直接在 MinerU 渲染。-## 与 `ParseResult`+`extensions.mineru` 仅含实际 `tier` 和最终 `parse_mode`;+`flash/medium/high/xhigh` 映射为 `flash/basic/standard/advanced`。+仅接受 `txt/ocr`,不存请求值或 `auto`,不重复存 `effort/mineru_version`。+MinerU 校验已存在的自身扩展;DocVortex 不依赖产品枚举,完整保留其他应用扩展。-`ParseResult.to_dict()` 当前输出 `{"schema_version": "2.0", "pages": ...}` 等 MiddleJson 顶层字段,不包含顶层 `_backend` 或 PDF 页映射内部 metadata。`ParseResult.from_dict()` 当前要求 payload 为 dict,且只接受 schema 2.0 与可识别 legacy 输入(见上文“兼容输入”)。+## 序列化和素材-建议:+DocumentResult 与 ParseResult 继续存在。MinerU 的 PDF ParseResult 序列化仍省略+递归 image_base64,非 PDF 保持原有图片表示;源对象不被修改。因此协议统一不代表+所有结果包装的图片字段完全相同。DocVortex 继续使用 AssetStore 和显式素材外置。+Structured Content 迁移 metadata/extensions,但不带 Model/Middle 协议标识;+Content List V1/V2 内容契约不变。+HTTP 请求、输出文件名和 ZIP 布局保持不变;ZIP 内文档必须使用完整新版协议。-- `ParseResult.to_dict()` 继续输出 `schema_version + pages`。-- 新增 `ParseResult.to_envelope(meta=...)` 输出 canonical envelope。-- `ParseResult.from_dict()` 调用 `normalize_middle_json()`。+## 直接迁移与缓存-## 未决问题+读取联合检查 schema 和 schema_version;不接受旧 DocVortex 1.0、缺少 schema 的+MinerU 2.0、旧 1.0 pages 或 pdf_info。不猜测、不自动转换,也不提供旧格式写出。+需要从源文件重新解析。Bundle 使用 docvortex.bundle 2.0 和原有素材摘要清单。-`filename` 是否默认写入、`parsed_at` 是否默认写入,集中维护在 [开放问题清单](../open-questions.md)。+Doclib 仅将协议有效且覆盖记录页范围的批次视为可用缓存。旧缓存不计入缓存命中、+可用页范围和默认读取档位。请求解析时重新生成缺失结果;直接读取旧页返回重新解析提示。+压缩仅合并 metadata、extensions、is_full_document 一致的新批次,重复页仍取最新值;+遇到旧格式、损坏数据或冲突时跳过压缩并保留源数据。diff --git a/docs/next/middle-json/migration.md b/docs/next/middle-json/migration.mdindex ea76eebb..50acb075 100644--- a/docs/next/middle-json/migration.md+++ b/docs/next/middle-json/migration.md@@ -1,149 +1,15 @@-# 迁移计划+# JSON 2.0 直接迁移-状态: Draft-读者: 核心开发者、backend 开发者、SDK 开发者-范围: Middle JSON 统一工作的阶段、任务、验收和风险-来源: 由根目录旧 Middle JSON 底稿迁移整理而来+采用[统一外层协议](envelope.md),不保留旧格式转换入口。-## 目标+1. 升级 DocVortex 至 0.2.x,MinerU 要求 `docvortex>=0.2.0,<0.3.0`。+2. 将 Model/Middle 构造器的 file_suffix、producer 放入 DocumentMetadata;+ Python 访问同步改为 metadata.file_suffix、metadata.producer。+3. JSON 使用完整 schema/schema_version/metadata/extensions;两边统一 from_dict/from_json。+4. 产品记录只输出实际 tier 与 txt/ocr;删除旧 effort、顶层 mineru_version 和协议适配调用。+5. 历史 JSON、结果包及旧缓存需从源文件重建;加载不会改写旧文件。+6. 同步升级自建 HTTP 服务与客户端。旧远端协议会明确拒绝,不尝试迁移。-把当前 typed document model 推进为可验证、可迁移、可供 Agent 引用的 Middle JSON 标准。--## Phase 0: 确认事实标准--任务:--1. 明确 `mineru/types.py` 是当前 Middle JSON 基础类型来源。-2. 将 `PageInfo` / 严格 Pydantic block tree / `InlineSpan` 字段写入文档。-3. 标注内部字段和 public 字段。--验收:--- 新增文档能回答“当前标准是什么”。-- backend 开发者知道新增字段应该加在哪里。--## Phase 1: Envelope 与恢复--任务:--1. 定义 canonical envelope。-2. 已实现 `ParseResult.from_dict()`。-3. 已实现 `ParseResult.from_json()`。-4. 已实现 `ParseResult.to_dict()` 输出 `schema_version + pages`。-5. 待实现 `normalize_middle_json()`。-6. 当前运行时兼容的输入(`ParseResult.from_dict()`):- - `{"schema_version": "2.0", "pages": [...], ...}` 直接读取。- - `{"schema_version": "1.0", "pages": [...]}` 经 legacy 分支单向迁移。- - `{"pdf_info": [...], "_backend": ..., ...}`(MinerU 3.4.5 产物)按 envelope 识别后经同一 legacy 分支单向迁移。-7. 无版本号的裸 `{"pages": [...]}` 与其它未知旧 payload 被明确拒绝,要求重新解析,不做静默迁移。--验收:--- API `middle_json` output 可以恢复为 `ParseResult`。-- doclib 当前缓存 JSON 可以恢复为 `ParseResult`。-- 历史旧 CLI `pdf_info` 产物经运行时 legacy 分支单向恢复;无法识别的旧 payload 按 stale 处理并要求重新解析。--## Phase 2: Validator--任务:--1. 生产代码暂不提供独立 validator API;结构合法性由 strict Pydantic 模型(`extra="forbid"` + `strict`)在解析时强制。-2. 单测中保留 `validate_pages()` 与 `ValidationIssue` 作为 test-local helper。-3. 单测已覆盖 P0 页面树校验:- - page_idx- - block index/type/bbox(固定版式顶层 block 强制 bbox)- - 自然语言 block 的行内 `content` 为合法 InlineSpan 列表-4. 待补 envelope-level 校验:- - schema_version- - pages list- - page_count-5. 增加 fixtures。--验收:--- Pipeline/VLM/Hybrid/Office 至少各有一个 fixture。-- validator 能区分 error / warning。-- 当前已知 Office unknown bbox 以 warning 处理,不阻塞。--## Phase 3: Agent Locator--任务:--1. (已完成)定义 locator(`mineru/doclib/locators.py`)。-2. (已完成)实现 block locator(`locator_for_block()` / `block_ref()` / `block_char_ref()`,doclib server 与 CLI 已消费)。-3. 待实现 citation record helper。-4. doclib 已持久化 `short_id` 与 `sha256`;`parses` 表暂无 schema_version 列,仍待补。--验收:--- 同一文件重复解析 locator 稳定。-- 不同 tier 的 block reference 可区分。-- unknown bbox 输出 `bbox_known=false`。-- Agent 可以从 block reference 回查 page/block。--## Phase 4: Backend Normalization--任务:--1. Pipeline: 检查 index 全页稳定性。-2. VLM: 确认 bbox 为 schema 约定的归一化坐标(0-1)且不含内部 metadata。-3. Hybrid: 将 features/models 进入 `_meta`。-4. Office: 明确 unknown bbox(`bbox: null`)语义。-5. HTML: 修正必填字段和 DOM order index。-6. 所有 backend 输出通过 validator。--验收:--- 每个 backend 生成 canonical envelope。-- 每个 backend 可生成 locator。-- `PageInfo` 不携带 `_backend` 等 backend metadata。--## Phase 5: Rendering 收敛--任务:--1. render facade 接受 envelope 或 pages + meta。-2. 为 markdown 增加可选 Agent marker。-3. 明确 Structured Content schema。-4. 把 Office-specific structured_content 逐步收敛到 type-specific helper。--验收:--- Markdown / Content List / Structured Content 均可从 canonical envelope 生成。-- renderer 不依赖 `pdf_info`。-- backend-specific dispatch 数量下降。--## Phase 6: 历史数据迁移--任务:--1. 为 doclib 缓存结果提供 lazy migration。-2. 评估历史 `pdf_info` CLI 输出是否仍需离线批量迁移命令(单文档恢复已由运行时 legacy 分支覆盖),或仅保留重新生成说明。-3. 对缺少 sha256 的历史数据给出明确错误或要求调用方提供文件 hash。--验收:--- 老 `*_middle.json` 能被新版 SDK 读取。-- 无 sha256 的数据不能生成可严格校验的跨文档引用,但仍可 render。-- 迁移不会改变用户可见 markdown 输出。--## 风险--| 风险 | 缓解 |-|------|------|-| 强行要求 Office bbox 导致大量无效框 | 使用 `bbox_known=false`,先承认 unknown。 |-| 改 envelope 破坏旧 CLI | 运行时 legacy 分支单向迁移可识别旧 payload;不可识别的按 stale 处理并重新解析,不静默降级。 |-| locator 因 index 不稳定而漂移 | 先做 normalization,再生成 locator。 |-| render 收敛过大 | 分阶段,先 facade,后 type-specific helper。 |-| filename 泄露隐私 | `_meta.file.filename` 默认可为空。 |--## 首批可执行任务--1. 实现 `normalize_middle_json()`。-2. 增加 envelope-level validator。-3. 修正 HTML 解析必填字段。-4. 定义 `locator_for_block()`。-5. 给 Pipeline/VLM/Office 各加一个 fixture。-6. 复核历史 `pdf_info` 迁移路径(运行时 legacy 分支已覆盖单文档恢复,确认离线批量迁移是否仍有必要)。--完成这些任务后,Middle JSON 就可以支撑 API/SDK 的 `middle_json` output、doclib 缓存恢复和 Agent citation 的第一版闭环。+验证协议 Schema、跨入口往返、真实生产者与扩展保留、旧协议拒绝、ZIP 和素材,+以及 Doclib 新旧缓存混存、有效页覆盖、重复页与压缩冲突。冻结迁移前页面、正文、+几何及渲染语义,不把外层迁移变成解析算法调整。diff --git a/docs/next/pitfalls.md b/docs/next/pitfalls.mdindex c33a4e4e..eca6ad68 100644--- a/docs/next/pitfalls.md+++ b/docs/next/pitfalls.md@@ -16,15 +16,12 @@- 新代码只走 `tier`;`backend_for_tier`(`parser/tier.py:183`)已保证 basic/standard/advanced 共用同一个 hybrid-engine,仅 effort 不同(`HYBRID_EFFORT_BY_TIER` `parser/tier.py:25`)。- 不要新增依赖 `backend` 入参的分支;`pipeline`/`vlm-*` 仅作为 legacy 别名存在。-### 1.2 Middle JSON legacy 读取三件套+### 1.2 共享文档协议与缓存必须同步-改 Middle JSON 结构时必须同步检查三处,否则旧缓存文档读取会断:--1. legacy 读取:`mineru/parser/base.py:20-47` —— 识别 3.4.5 `pdf_info` 与 schema 1.0 `pages` 包装(`_legacy_raw_pages`),`MIDDLE_JSON_SCHEMA_VERSION = "2.0"`;-2. legacy 适配:`mineru/backend/postprocess/legacy_schema_adapter.py` —— 旧 payload 单向回推为 raw model-list;-3. 批次合并:`mineru/doclib/background/compaction.py:23` `_normalize_batch_pages` 会把 schema 2.0、1.0、3.4.5 批次归一化后**跨 schema 合并**(legacy 批次经 `legacy_schema_adapter` 转换参与),并非只合并 2.0 批次;schema 无法识别时抛 stale-cache 错误并放弃本轮压缩(`compaction.py:37`)。--新增或调整 Middle JSON 字段时,先确认 ADR-0020(schema stability boundary)是否覆盖该字段,再同步上述三处与 `docs/next/middle-json.md`。+改外层结构时同步 DocVortex schema/readers、MinerU ParseResult 和 Doclib 缓存。+当前使用 schema 身份 + 版本 2.0 + metadata,旧协议全部拒绝;不能仅按版本号识别。+缓存命中、覆盖与默认读取档位均需检查新版协议;压缩只合并元数据一致的新批次。+详见 [envelope.md](middle-json/envelope.md)。## 2. Alpha 高频迭代,公开 API 未稳定diff --git a/mineru/backend/analyze.py b/mineru/backend/analyze.pyindex 8d3cf74e..d90eb962 100644--- a/mineru/backend/analyze.py+++ b/mineru/backend/analyze.py@@ -2,7 +2,7 @@"""统一 PDF、EPUB、HTML、OFD、CSV 与 Office/RTF 文档分析的稳定公共门面。"""from __future__ import annotations-from docvortex.schema import Producer+from docvortex.schema import DocumentMetadata, Producerfrom ..integrations.docvortex import build_metadataimport asyncio@@ -81,9 +81,8 @@ def doc_analyze(model_json = ModelJson(pages=result.model_list,page_index_map=page_index_map or [],- file_suffix=file_suffix,- producer=Producer(name="mineru", version=mineru_version),- extensions=build_metadata(effort=result.effort, parse_mode=result.parse_mode, mineru_version=mineru_version),+ metadata=DocumentMetadata(file_suffix=file_suffix, producer=Producer(name="mineru", version=mineru_version)),+ extensions=build_metadata(effort=result.effort, parse_mode=result.parse_mode),)from .postprocess.document import model_json_to_middle_jsondiff --git a/mineru/backend/postprocess/document.py b/mineru/backend/postprocess/document.pyindex ebef0e3d..d3ddd14e 100644--- a/mineru/backend/postprocess/document.py+++ b/mineru/backend/postprocess/document.py@@ -11,7 +11,7 @@ from .llm_aided import apply_llm_aided_postprocessdef model_json_to_middle_json(model_json: ModelJson, *, llm_aided_config: LLMAidedConfig) -> MiddleJson:"""只由宿主显式启用 LLM,DocVortex 的默认后处理完整且独立。"""middle_json = build_middle_json(model_json)- if model_json.file_suffix == "pdf":+ if model_json.metadata.file_suffix == "pdf":apply_llm_aided_postprocess(middle_json, llm_aided_config)return middle_jsondiff --git a/mineru/backend/postprocess/legacy_schema_adapter.py b/mineru/backend/postprocess/legacy_schema_adapter.pydeleted file mode 100644index d6150f05..00000000--- a/mineru/backend/postprocess/legacy_schema_adapter.py+++ /dev/null@@ -1,727 +0,0 @@-# Copyright (c) Opendatalab. All rights reserved.-"""MinerU 3.4.5 Middle JSON page 到当前 raw ModelJson 的适配器。--3.4.5 的页面仍使用 ``preproc_blocks``/``para_blocks``、``lines`` 和-``spans``,自然语言行内语义则通过旧字符串标签或旧 span 字段表达。本模块只负责-把单页回推成当前 ``model_json_to_pages`` 能消费的 raw block 列表;文档 envelope-及当前 MiddleJson 的元数据由调用方负责。-"""--from __future__ import annotations--import html-import math-import re-from typing import Any--from docvortex.schema import INLINE_STYLE_ORDER, InlineSpan, parse_inline_spans-from docvortex.foundation.hyperlink import OFFICE_EXTERNAL_HYPERLINK_SCHEMES, sanitize_hyperlink_target-from docvortex.content.inline import join_inline_spans--# 这些父块在 3.4.5 中通过 ``blocks`` 保存 body/caption/footnote;当前 raw-# ModelJson 则要求先展平成同级 block,再由统一后处理重新分组。-_VISUAL_PARENT_TYPES: frozenset[str] = frozenset(- {- "image",- "table",- "chart",- "code",- }-)--_INLINE_CONTENT_BLOCK_TYPES: frozenset[str] = frozenset(- {- "text",- "ref_text",- "doc_title",- "paragraph_title",- "header",- "footer",- "page_number",- "aside_text",- "page_footnote",- "image_caption",- "image_footnote",- "table_caption",- "table_footnote",- "chart_caption",- "chart_footnote",- "code_caption",- "code_footnote",- "caption",- "footnote",- "phonetic",- }-)--_STRING_CONTENT_BLOCK_TYPES: frozenset[str] = frozenset(- {- "equation",- "image_body",- "table_body",- "chart_body",- "code_body",- }-)--_BLOCK_TYPE_ALIASES: dict[str, str] = {- "abstract": "text",- "phonetic": "text",- "interline_equation": "equation",-}--_SUPPORTED_BLOCK_TYPES: frozenset[str] = frozenset(- _VISUAL_PARENT_TYPES | _INLINE_CONTENT_BLOCK_TYPES | _STRING_CONTENT_BLOCK_TYPES | {"list", "index"}-)--_INLINE_START_RE = re.compile(- r"<(?P<tag>eq|code|text|hyperlink|sup|sub|strong|b|em|i|s|u)(?P<attrs>\s[^<>]*?)?>",- re.IGNORECASE,-)-_STYLE_ATTR_RE = re.compile(r"\bstyle\s*=\s*([\"'])(?P<style>.*?)\1", re.IGNORECASE | re.DOTALL)-_URL_RE = re.compile(r"<url>(?P<url>.*?)</url>", re.IGNORECASE | re.DOTALL)-_DIRECT_TAG_STYLES: dict[str, str] = {- "strong": "bold",- "b": "bold",- "em": "italic",- "i": "italic",- "s": "strikethrough",- "u": "underline",- "sup": "superscript",- "sub": "subscript",-}--# 3.4.5 PDF 的异常零面积框需要替换后才能通过当前严格 bbox 校验。-_PLACEHOLDER_BBOX: tuple[float, float, float, float] = (0.0, 0.0, 0.001, 0.001)---def legacy_page_to_model_list(page: dict[str, Any]) -> list[dict[str, Any]]:- """把 MinerU 3.4.5 的单页 Middle JSON 转换为当前 raw model-list。"""- if not isinstance(page, dict):- raise ValueError("legacy Middle JSON page must be a dict")-- width, height = _parse_page_size(page.get("page_size"))- source_blocks = page.get("preproc_blocks")- if source_blocks is None:- source_blocks = page.get("para_blocks", [])- if not isinstance(source_blocks, list):- raise ValueError("legacy Middle JSON page blocks must be a list")-- discarded_blocks = page.get("discarded_blocks", [])- if not isinstance(discarded_blocks, list):- raise ValueError("legacy Middle JSON discarded_blocks must be a list")-- legacy_blocks = [*source_blocks, *discarded_blocks]- model_list: list[dict[str, Any]] = []- for group_index, block in enumerate(legacy_blocks):- if not isinstance(block, dict):- raise ValueError("legacy Middle JSON block entries must be dicts")- group_start = len(model_list)- _collect_blocks(block, width, height, model_list, inherited=None)- if width <= 0 or height <= 0:- _assign_synthetic_group_bbox(model_list[group_start:], group_index, len(legacy_blocks))- return model_list---def _parse_page_size(value: Any) -> tuple[float, float]:- """读取合法 page_size;缺失、非数值或非正值统一返回零值。"""- if not isinstance(value, (list, tuple)) or len(value) < 2:- return 0.0, 0.0- width, height = value[:2]- if any(isinstance(item, bool) or not isinstance(item, (int, float)) for item in (width, height)):- return 0.0, 0.0- parsed_width, parsed_height = float(width), float(height)- if not all(math.isfinite(item) and item > 0 for item in (parsed_width, parsed_height)):- return 0.0, 0.0- return parsed_width, parsed_height---def _assign_synthetic_group_bbox(blocks: list[dict[str, Any]], group_index: int, group_count: int) -> None:- """为无 page_size 文档分配互不重叠的占位框,并保持原顶层块的子项同组。"""- safe_count = max(group_count, 1)- y0 = (group_index + 0.2) / safe_count- y1 = (group_index + 0.8) / safe_count- bbox = (0.05, y0, 0.95, y1)- for block in blocks:- block["bbox"] = bbox- lines = block.get("lines")- if isinstance(lines, list):- for line in lines:- if isinstance(line, dict):- line["bbox"] = list(bbox)---def _collect_blocks(- block: dict[str, Any],- width: float,- height: float,- output: list[dict[str, Any]],- *,- inherited: dict[str, Any] | None,-) -> None:- """递归展平视觉父块,并把普通叶子转换为当前 raw block。"""- block_type = _normalize_block_type(block)- nested = block.get("blocks")- if block_type in _VISUAL_PARENT_TYPES and isinstance(nested, list) and nested:- child_context = _visual_child_context(block, inherited)- for child in nested:- if not isinstance(child, dict):- raise ValueError("legacy visual block children must be dicts")- _collect_blocks(child, width, height, output, inherited=child_context)- return-- if block_type == "list" and isinstance(nested, list) and nested:- output.append(_convert_leaf_block(block, block_type, width, height, inherited))- output.extend(_flatten_nested_list_items(nested, width, height))- return- if block_type == "index" and isinstance(nested, list) and nested:- output.append(_flatten_nested_index(block, nested, width, height))- return- if block_type == "list" and isinstance(block.get("lines"), list) and block["lines"]:- output.append(_convert_leaf_block(block, block_type, width, height, inherited))- output.extend(_convert_pdf_list_items(block, width, height))- return-- raw = _convert_leaf_block(block, block_type, width, height, inherited)- output.append(raw)---def _visual_child_context(- block: dict[str, Any],- inherited: dict[str, Any] | None,-) -> dict[str, Any]:- """收集视觉父块上需要下沉到 body 的元数据。"""- context = dict(inherited or {})- context["bbox"] = block.get("bbox", context.get("bbox"))- sub_type = block.get("sub_type")- if not isinstance(sub_type, str) or not sub_type.strip():- sub_type = None- if sub_type:- context["sub_type"] = sub_type- if isinstance(block.get("guess_lang"), str) and block["guess_lang"].strip():- context["guess_lang"] = block["guess_lang"].strip()- if "cell_merge" in block:- context["cell_merge"] = block.get("cell_merge")- return context---def _flatten_nested_list_items(nested: list[Any], width: float, height: float) -> list[dict[str, Any]]:- """把 3.4.5 Office 嵌套列表叶子展平为当前 PDF list 可关联的文本块。"""- items: list[dict[str, Any]] = []- for child in nested:- if not isinstance(child, dict):- raise ValueError("legacy list children must be dicts")- child_type = _normalize_block_type(child)- child_nested = child.get("blocks")- if child_type == "list" and isinstance(child_nested, list):- items.extend(_flatten_nested_list_items(child_nested, width, height))- continue- if child_type not in {"text", "ref_text"}:- child_type = "text"- items.append(_convert_leaf_block(child, child_type, width, height, inherited=None))- return items---def _flatten_nested_index(- block: dict[str, Any],- nested: list[Any],- width: float,- height: float,-) -> dict[str, Any]:- """把 3.4.5 Office 嵌套目录折叠为当前 PDF index 使用的换行 Span。"""- contents: list[list[dict[str, Any]]] = []-- def collect(children: list[Any]) -> None:- """按深度优先顺序收集目录叶子的结构化行内内容。"""- for child in children:- if not isinstance(child, dict):- raise ValueError("legacy index children must be dicts")- child_nested = child.get("blocks")- if _normalize_block_type(child) == "index" and isinstance(child_nested, list):- collect(child_nested)- continue- converted = _convert_leaf_block(child, "text", width, height, inherited=None)- content = converted.get("content")- if isinstance(content, list) and content:- contents.append([span for span in content if isinstance(span, dict)])-- collect(nested)- joined: list[dict[str, Any]] = []- for content_index, content in enumerate(contents):- if content_index:- _append_text_span(joined, "\n", ())- _extend_span_dicts(joined, content)- return {- "type": "index",- "bbox": _normalize_bbox(block.get("bbox"), width, height),- "content": [span.model_dump(mode="json") for span in parse_inline_spans(joined)],- }---def _convert_pdf_list_items(block: dict[str, Any], width: float, height: float) -> list[dict[str, Any]]:- """把 3.4.5 PDF list 的物理行恢复为当前 bbox 关联需要的文本项。"""- raw_lines = [line for line in block.get("lines", []) if isinstance(line, dict)]- if not raw_lines:- return []- has_item_flags = any(line.get("is_list_start_line") or line.get("is_list_end_line") for line in raw_lines)- line_groups: list[list[dict[str, Any]]] = []- current_group: list[dict[str, Any]] = []- for line in raw_lines:- if current_group and (line.get("is_list_start_line") or not has_item_flags):- line_groups.append(current_group)- current_group = []- current_group.append(line)- if line.get("is_list_end_line"):- line_groups.append(current_group)- current_group = []- if current_group:- line_groups.append(current_group)-- item_type = "ref_text" if block.get("sub_type") == "ref_text" else "text"- items: list[dict[str, Any]] = []- for lines in line_groups:- synthetic_block = {- "type": item_type,- "bbox": _union_line_bboxes(lines, block.get("bbox")),- "lines": lines,- }- items.append(_convert_leaf_block(synthetic_block, item_type, width, height, inherited=None))- return items---def _union_line_bboxes(lines: list[dict[str, Any]], fallback: Any) -> Any:- """合并一个旧列表项的物理行框,缺失时回退列表父框。"""- bboxes = [line.get("bbox") for line in lines]- valid = [- bbox- for bbox in bboxes- if isinstance(bbox, (list, tuple))- and len(bbox) == 4- and all(isinstance(item, (int, float)) and not isinstance(item, bool) for item in bbox)- ]- if not valid:- return fallback- return [- min(float(bbox[0]) for bbox in valid),- min(float(bbox[1]) for bbox in valid),- max(float(bbox[2]) for bbox in valid),- max(float(bbox[3]) for bbox in valid),- ]---def _convert_leaf_block(- block: dict[str, Any],- block_type: str,- width: float,- height: float,- inherited: dict[str, Any] | None,-) -> dict[str, Any]:- """把一个 3.4.5 叶子块转换为当前 raw block 字典。"""- inherited = inherited or {}- raw_bbox = block.get("bbox", inherited.get("bbox"))- bbox = _normalize_bbox(raw_bbox, width, height)- lines = block.get("lines")- legacy_lines = lines if isinstance(lines, list) else []- effective_sub_type = _effective_sub_type(block, block_type, inherited)-- raw: dict[str, Any] = {"type": block_type, "bbox": bbox}- if _uses_inline_content(block_type, effective_sub_type):- raw["content"] = _extract_inline_content(block, legacy_lines, preserve_line_breaks=block_type == "index")- elif block_type in _STRING_CONTENT_BLOCK_TYPES:- raw["content"] = _extract_string_content(block, legacy_lines, block_type)-- image_fields = _extract_image_fields(block, legacy_lines)- raw.update(image_fields)-- normalized_lines = _normalize_lines(legacy_lines, width, height)- if normalized_lines:- raw["lines"] = normalized_lines-- _copy_leaf_metadata(raw, block, block_type, effective_sub_type, inherited)- return raw---def _effective_sub_type(block: dict[str, Any], block_type: str, inherited: dict[str, Any]) -> str | None:- """解析 code/image/chart body 的有效 sub_type。"""- sub_type = block.get("sub_type", inherited.get("sub_type"))- if isinstance(sub_type, str) and sub_type.strip():- return sub_type.strip()- if block_type == "code_body":- return "code"- return None---def _uses_inline_content(block_type: str, sub_type: str | None) -> bool:- """判断当前 raw block 是否必须使用结构化 InlineSpan 列表。"""- return block_type in _INLINE_CONTENT_BLOCK_TYPES or (block_type == "code_body" and sub_type == "algorithm")---def _extract_inline_content(- block: dict[str, Any],- lines: list[Any],- *,- preserve_line_breaks: bool,-) -> list[dict[str, Any]]:- """从旧 lines/spans 或直接 content 构造严格 InlineSpan 列表。"""- line_contents: list[list[InlineSpan]] = []- for line in lines:- if not isinstance(line, dict):- continue- spans = line.get("spans")- if not isinstance(spans, list):- continue- converted: list[dict[str, Any]] = []- for span in spans:- if not isinstance(span, dict):- continue- _append_legacy_span(converted, span)- parsed = parse_inline_spans(converted)- if parsed:- line_contents.append(parsed)-- if not line_contents:- direct_content = block.get("content")- converted = _convert_direct_inline_content(direct_content)- return [span.model_dump(mode="json") for span in parse_inline_spans(converted)]-- if preserve_line_breaks:- joined: list[dict[str, Any]] = []- for line_index, content in enumerate(line_contents):- if line_index:- _append_text_span(joined, "\n", ())- _extend_span_dicts(joined, [span.model_dump(mode="json") for span in content])- return [span.model_dump(mode="json") for span in parse_inline_spans(joined)]-- joined_spans = join_inline_spans(line_contents)- return [span.model_dump(mode="json") for span in joined_spans]---def _convert_direct_inline_content(content: Any) -> list[dict[str, Any]]:- """转换没有 lines 包装的旧字符串或 span 列表。"""- if isinstance(content, str):- return _parse_legacy_markup(content)- if not isinstance(content, list):- return []- output: list[dict[str, Any]] = []- for item in content:- if isinstance(item, dict):- _append_legacy_span(output, item)- elif isinstance(item, str):- _extend_span_dicts(output, _parse_legacy_markup(item))- return output---def _append_legacy_span(output: list[dict[str, Any]], span: dict[str, Any]) -> None:- """把一个 3.4.5 span 追加为当前 InlineSpan 字典。"""- span_type = str(span.get("type", "text") or "text")- content = span.get("content")- if span_type in {"inline_equation", "equation_inline", "interline_equation", "equation"}:- normalized = str(content or "").strip()- if normalized:- output.append({"type": "equation_inline", "content": normalized})- return- if span_type in {"code", "code_inline"}:- if isinstance(content, str) and content:- output.append({"type": "code_inline", "content": content})- return- if span_type == "hyperlink":- _append_legacy_hyperlink(output, span)- return- if not isinstance(content, str) or not content:- return-- styles = _normalize_styles(span.get("styles", span.get("style")))- if styles:- _append_text_span(output, content, styles)- else:- _extend_span_dicts(output, _parse_legacy_markup(content))---def _append_legacy_hyperlink(output: list[dict[str, Any]], span: dict[str, Any]) -> None:- """转换结构化旧 hyperlink;不安全目标降级为可见子内容。"""- children: list[dict[str, Any]] = []- raw_children = span.get("children")- if isinstance(raw_children, list):- for child in raw_children:- if isinstance(child, dict):- _append_legacy_span(children, child)- elif isinstance(span.get("content"), str):- styles = _normalize_styles(span.get("styles", span.get("style")))- if styles:- _append_text_span(children, span["content"], styles)- else:- _extend_span_dicts(children, _parse_legacy_markup(span["content"]))-- safe_children = [child for child in children if child.get("type") != "hyperlink"]- safe_url = sanitize_hyperlink_target(- html.unescape(str(span.get("url", ""))),- allowed_schemes=OFFICE_EXTERNAL_HYPERLINK_SCHEMES,- allow_relative=True,- allow_fragment=True,- )- if safe_url and safe_children:- output.append({"type": "hyperlink", "url": safe_url, "content": safe_children})- else:- _extend_span_dicts(output, safe_children)---def _parse_legacy_markup(content: str) -> list[dict[str, Any]]:- """按 3.4.5 的私有字符串标签协议解析行内内容。"""- output: list[dict[str, Any]] = []- cursor = 0- while match := _INLINE_START_RE.search(content, cursor):- _append_text_span(output, content[cursor : match.start()], ())- tag = match.group("tag").lower()- closing = _find_matching_close(content, tag, match.end())- if closing is None:- _append_text_span(output, match.group(0), ())- cursor = match.end()- continue-- inner = content[match.end() : closing[0]]- original = content[match.start() : closing[1]]- parsed = _parse_legacy_element(tag, match.group("attrs") or "", inner)- if parsed is None:- _append_text_span(output, original, ())- else:- _extend_span_dicts(output, parsed)- cursor = closing[1]- _append_text_span(output, content[cursor:], ())- return output---def _find_matching_close(content: str, tag: str, start: int) -> tuple[int, int] | None:- """查找旧标签的配对结束位置,并兼容同名嵌套。"""- token_re = re.compile(rf"<(?P<close>/)?{re.escape(tag)}(?:\s[^<>]*?)?>", re.IGNORECASE)- depth = 1- for match in token_re.finditer(content, start):- depth += -1 if match.group("close") else 1- if depth == 0:- return match.start(), match.end()- return None---def _parse_legacy_element(tag: str, attrs: str, inner: str) -> list[dict[str, Any]] | None:- """把一个完整旧标签转换为当前 InlineSpan 字典。"""- if tag == "eq":- latex = html.unescape(inner).strip()- return [{"type": "equation_inline", "content": latex}] if latex else []- if tag == "code":- code = html.unescape(inner)- return [{"type": "code_inline", "content": code}] if code else []- if tag == "hyperlink":- return _parse_markup_hyperlink(inner)-- children = _parse_legacy_markup(inner)- styles = _parse_style_attribute(attrs) if tag == "text" else [_DIRECT_TAG_STYLES[tag]]- return _apply_styles(children, styles)---def _parse_markup_hyperlink(inner: str) -> list[dict[str, Any]] | None:- """解析旧 hyperlink 标签;危险 URL 只移除链接语义。"""- match = _URL_RE.search(inner)- if match is None:- return None- label_content = f"{inner[: match.start()]}{inner[match.end() :]}"- children = [span for span in _parse_legacy_markup(label_content) if span.get("type") != "hyperlink"]- if not children:- return []- safe_url = sanitize_hyperlink_target(- html.unescape(match.group("url")),- allowed_schemes=OFFICE_EXTERNAL_HYPERLINK_SCHEMES,- allow_relative=True,- allow_fragment=True,- )- if safe_url is None:- return children- return [{"type": "hyperlink", "url": safe_url, "content": children}]---def _parse_style_attribute(attrs: str) -> list[str]:- """读取旧 ``text style`` 属性并过滤为当前白名单样式。"""- match = _STYLE_ATTR_RE.search(attrs)- return _normalize_styles(match.group("style") if match else None)---def _normalize_styles(value: Any) -> list[str]:- """按当前固定顺序去重旧样式,并消解同时上下标的非法组合。"""- if isinstance(value, str):- candidates = [item.strip().lower() for item in value.split(",")]- elif isinstance(value, (list, tuple, set)):- candidates = [str(item).strip().lower() for item in value]- else:- candidates = []- style_set = {item for item in candidates if item in INLINE_STYLE_ORDER}- if "superscript" in style_set and "subscript" in style_set:- style_set.remove("subscript")- return [style for style in INLINE_STYLE_ORDER if style in style_set]---def _apply_styles(spans: list[dict[str, Any]], styles: list[str]) -> list[dict[str, Any]]:- """把旧包装标签的样式递归应用到文字及链接文字。"""- if not styles:- return spans- output: list[dict[str, Any]] = []- for span in spans:- span_type = span.get("type")- if span_type == "text":- merged_styles = _normalize_styles([*_normalize_styles(span.get("styles")), *styles])- _append_text_span(output, str(span.get("content", "")), merged_styles)- elif span_type == "hyperlink" and isinstance(span.get("content"), list):- children = _apply_styles([child for child in span["content"] if isinstance(child, dict)], styles)- output.append({**span, "content": children})- else:- output.append(dict(span))- return output---def _append_text_span(output: list[dict[str, Any]], content: str, styles: Any) -> None:- """追加非空文字,并合并相邻同样式 TextSpan。"""- if not content:- return- normalized_styles = _normalize_styles(styles)- if output and output[-1].get("type") == "text" and output[-1].get("styles", []) == normalized_styles:- output[-1]["content"] = f"{output[-1].get('content', '')}{content}"- return- span: dict[str, Any] = {"type": "text", "content": content}- if normalized_styles:- span["styles"] = normalized_styles- output.append(span)---def _extend_span_dicts(output: list[dict[str, Any]], spans: list[dict[str, Any]]) -> None:- """追加一组 Span,并规范化序列边界上的相邻文字。"""- for span in spans:- if span.get("type") == "text" and isinstance(span.get("content"), str):- _append_text_span(output, span["content"], span.get("styles", []))- else:- output.append(dict(span))---def _extract_string_content(block: dict[str, Any], lines: list[Any], block_type: str) -> str:- """从旧 spans 提取公式、视觉主体、表格 HTML 或代码字符串。"""- parts: list[str] = []- for line in lines:- if not isinstance(line, dict):- continue- line_parts: list[str] = []- spans = line.get("spans")- if not isinstance(spans, list):- continue- for span in spans:- if not isinstance(span, dict):- continue- value = span.get("html") if block_type == "table_body" and span.get("html") is not None else span.get("content")- normalized = _stringify_visual_content(value)- if normalized:- line_parts.append(normalized)- if line_parts:- parts.append("".join(line_parts))-- if not parts:- direct = block.get("html") if block_type == "table_body" and block.get("html") is not None else block.get("content")- return _stringify_visual_content(direct)- return "\n".join(parts) if block_type == "code_body" else "".join(parts)---def _stringify_visual_content(value: Any) -> str:- """把旧视觉识别列表稳定折叠为当前字符串载荷。"""- if isinstance(value, str):- return value- if isinstance(value, list):- return "\n".join(str(item) for item in value if str(item).strip())- return "" if value is None else str(value)---def _extract_image_fields(block: dict[str, Any], lines: list[Any]) -> dict[str, str]:- """从 block 或第一个视觉 span 提升当前支持的图片载荷字段。"""- fields: dict[str, str] = {}- for field_name in ("image_path", "image_base64", "image_url"):- value = block.get(field_name)- if isinstance(value, str) and value:- fields[field_name] = value- if fields:- return fields- for line in lines:- if not isinstance(line, dict):- continue- spans = line.get("spans")- if not isinstance(spans, list):- continue- for span in spans:- if not isinstance(span, dict):- continue- for field_name in ("image_path", "image_base64", "image_url"):- value = span.get(field_name)- if isinstance(value, str) and value and field_name not in fields:- fields[field_name] = value- if fields:- return fields- return fields---def _copy_leaf_metadata(- raw: dict[str, Any],- block: dict[str, Any],- block_type: str,- sub_type: str | None,- inherited: dict[str, Any],-) -> None:- """复制当前后处理仍会读取、且不会越过严格模型边界的字段。"""- if block.get("level") is not None:- raw["level"] = block["level"]- if sub_type:- raw["sub_type"] = sub_type- guess_lang = block.get("guess_lang", inherited.get("guess_lang"))- if block_type == "code_body" and isinstance(guess_lang, str) and guess_lang.strip():- raw["guess_lang"] = guess_lang.strip()- if block_type == "table_body" and "cell_merge" in inherited:- raw["cell_merge"] = inherited["cell_merge"]- if block_type in {"doc_title", "paragraph_title", "page_footnote"}:- anchor = block.get("anchor")- if isinstance(anchor, str) and anchor.strip():- raw["anchor"] = anchor.strip()---def _normalize_block_type(block: dict[str, Any]) -> str:- """把最终 3.4.5 discriminator 映射到当前 raw 类型并拒绝内部标签。"""- block_type = str(block.get("type", "") or "")- if block_type == "title":- return "doc_title" if block.get("level") == 1 else "paragraph_title"- normalized = _BLOCK_TYPE_ALIASES.get(block_type, block_type)- if normalized not in _SUPPORTED_BLOCK_TYPES:- raise ValueError(f"unsupported MinerU 3.4.5 block type {block_type!r}; source reparse required")- return normalized---def _normalize_bbox(value: Any, width: float, height: float) -> tuple[float, float, float, float]:- """把旧 bbox 规范为当前严格的 0-1 非零面积坐标。"""- if not isinstance(value, (list, tuple)) or len(value) != 4:- return _PLACEHOLDER_BBOX- if any(isinstance(item, bool) or not isinstance(item, (int, float)) for item in value):- return _PLACEHOLDER_BBOX- coords = tuple(float(item) for item in value)- if not all(math.isfinite(item) for item in coords):- return _PLACEHOLDER_BBOX- if width > 0 and height > 0:- coords = (coords[0] / width, coords[1] / height, coords[2] / width, coords[3] / height)- elif not all(0.0 <= item <= 1.0 for item in coords):- return _PLACEHOLDER_BBOX- clamped = tuple(min(max(item, 0.0), 1.0) for item in coords)- if clamped[2] <= clamped[0] or clamped[3] <= clamped[1]:- return _PLACEHOLDER_BBOX- return clamped # type: ignore[return-value]---def _normalize_lines(lines: list[Any], width: float, height: float) -> list[dict[str, list[float]]]:- """保留跨页文本合并需要的行框,并转换到当前归一化坐标。"""- result: list[dict[str, list[float]]] = []- for line in lines:- if not isinstance(line, dict):- continue- bbox = _normalize_bbox(line.get("bbox"), width, height)- result.append({"bbox": list(bbox)})- return result---__all__ = ["legacy_page_to_model_list"]diff --git a/mineru/doclib/background/compaction.py b/mineru/doclib/background/compaction.pyindex c5563ae9..5e272414 100644--- a/mineru/doclib/background/compaction.py+++ b/mineru/doclib/background/compaction.py@@ -1,8 +1,7 @@"""Compaction — merges overlapping / adjacent done parse batches to keep the parses table lean."""from __future__ import annotations-from docvortex.schema import Producer-from ...integrations.docvortex import build_metadata+from ...parser.base import ParseResultimport asyncioimport json@@ -12,7 +11,6 @@ import timefrom collections.abc import Sequencefrom typing import Any, cast-from ...parser.base import MIDDLE_JSON_SCHEMA_VERSIONfrom ...parser.page_range import format_page_rangefrom ...types import Tierfrom ..core.db import DatabaseManager@@ -24,46 +22,8 @@ logger = logging.getLogger("mineru.compaction")def _normalize_batch_pages(batch_payload: dict[str, Any]) -> list[dict[str, Any]]:- """统一读取 2.0 batch,或把 3.4.5/1.0 页面转换为当前 page 字典。"""- schema_version = batch_payload.get("schema_version")- raw_pages = batch_payload.get("pages")- if schema_version == MIDDLE_JSON_SCHEMA_VERSION:- return raw_pages if isinstance(raw_pages, list) else []-- if schema_version == "1.0":- pass- elif schema_version is None:- raw_pages = batch_payload.get("pdf_info")- else:- raw_pages = None- if not isinstance(raw_pages, list) or any(not isinstance(page, dict) for page in raw_pages):- raise ValueError("stale Middle JSON cache requires source reparse")-- from ...backend.postprocess.legacy_schema_adapter import legacy_page_to_model_list- from docvortex.postprocess.pages import model_json_to_pages- from ...parser.base import (- _legacy_effort,- _legacy_file_suffix,- _legacy_page_index_map,- _legacy_parse_mode,- )- from ...types import ModelJson- from ...version import __version__ as current_mineru_version-- source_version = batch_payload.get("_version_name", batch_payload.get("mineru_version"))- mineru_version = (- source_version.strip() if isinstance(source_version, str) and source_version.strip() else current_mineru_version- )- model_json = ModelJson(- pages=[legacy_page_to_model_list(page) for page in raw_pages],- page_index_map=_legacy_page_index_map(raw_pages),- file_suffix=_legacy_file_suffix(batch_payload),- producer=Producer(name="mineru", version=mineru_version),- extensions=build_metadata(- effort=_legacy_effort(batch_payload), parse_mode=_legacy_parse_mode(batch_payload), mineru_version=mineru_version- ),- )- return [page.to_dict() for page in model_json_to_pages(model_json)]+ """仅校验并读取新版批次,拒绝猜测或改写旧协议数据。"""+ return [page.to_dict() for page in ParseResult.from_dict(batch_payload).pages]class Compaction:@@ -199,12 +159,13 @@ class Compaction:if type(page_idx) is not int or page_idx < 0:return Nonepages_by_page_idx[page_idx] = page+ current_envelope = {+ key: value for key, value in ParseResult.from_dict(batch_payload).to_dict().items() if key != "pages"+ }if not envelope:- envelope = {- key: batch_payload[key]- for key in ("is_full_document", "file_suffix", "effort", "parse_mode", "mineru_version")- if key in batch_payload- }+ envelope = current_envelope+ elif json.dumps(envelope, sort_keys=True) != json.dumps(current_envelope, sort_keys=True):+ return Nonereturn (pages_by_page_idx, envelope) if pages_by_page_idx else Nonedef _write_compacted_json_files(@@ -228,8 +189,7 @@ class Compaction:raise ValueError(f"Compacted page range has no source pages: {page_range}")final_path = parse_batch_json_path(self.data_dir, sha256, tier, page_range, max_done_at)temp_path = f"{final_path}.tmp-{time.time_ns()}"- payload: dict[str, Any] = {"schema_version": MIDDLE_JSON_SCHEMA_VERSION, "pages": json_pages}- payload.update(envelope)+ payload = ParseResult.from_dict({**envelope, "pages": json_pages}).to_dict()with open(temp_path, "w", encoding="utf-8") as f:json.dump(payload, f, ensure_ascii=False, indent=2)prepared.append((temp_path, final_path))diff --git a/mineru/doclib/server.py b/mineru/doclib/server.pyindex 706eb860..f9b0aee8 100644--- a/mineru/doclib/server.py+++ b/mineru/doclib/server.py@@ -75,6 +75,7 @@ from .rows import (WatchTargetRow,)from .services.parse_svc import (+ _json_batch_is_current,accessible_file_for_sha256,filter_pages_by_user_range,load_pages_from_done_batches,@@ -418,7 +419,13 @@ class DoclibServer(AsyncDoclibInterface):if doc is None:raise NotFoundError("doc_not_found", f"Document {resolved_sha256} not found.", "doc_ref")page_range = _expand_page_range(page_range, doc["page_count"] or 1)- coverage = _parse_coverage(page_range, rows) if resolved_sha256 and tier and page_range else None+ coverage_rows = [+ row+ for row in rows+ if row["status"] != PARSE_STATUS_DONE+ or _json_batch_is_current(_effective_data_dir(self.state), row["sha256"], row["tier"], row)+ ]+ coverage = _parse_coverage(page_range, coverage_rows) if resolved_sha256 and tier and page_range else Nonereturn ListParsesResponse(parses=[_parse_info(row, coverage=coverage) for row in rows],coverage=coverage,@@ -717,7 +724,11 @@ class DoclibServer(AsyncDoclibInterface):tier = cursor.tier or await self._default_read_tier(doc["sha256"])if tier is None:- raise NotFoundError("tier_not_cached", f"No parsed tier is cached for document {cursor.short_id}.", "locator")+ raise NotFoundError(+ "tier_not_cached",+ f"No current parsed tier is cached for document {cursor.short_id}; reparse the source document.",+ "locator",+ )if format == "image" and context:raise InvalidRequestError("context_not_applicable", "context is not supported for image reads.", "context")if cursor.page_no is None and context:@@ -867,7 +878,12 @@ class DoclibServer(AsyncDoclibInterface):if sha256else [])- parsed_tiers = [_tier_parse_info(row) for row in parse_rows if row["status"] == PARSE_STATUS_DONE]+ parsed_tiers = [+ _tier_parse_info(row)+ for row in parse_rows+ if row["status"] == PARSE_STATUS_DONE+ and _json_batch_is_current(_effective_data_dir(self.state), row["sha256"], row["tier"], row)+ ]active_parses = [_parse_info(row) for row in parse_rows if row["status"] in {PARSE_STATUS_PENDING, PARSE_STATUS_PARSING}]@@ -1221,16 +1237,22 @@ class DoclibServer(AsyncDoclibInterface):"SELECT page_range, done_at FROM parses WHERE sha256=? AND tier=? AND status=? ORDER BY done_at DESC",(sha256, tier, PARSE_STATUS_DONE),)- loaded_pages = load_pages_from_done_batches(data_dir, sha256, tier, cast(list[ParseBatchRow], rows))doc = cast(DocRow | None, await self.state.db.fetchone("SELECT * FROM docs WHERE sha256=?", (sha256,)))if doc is None:raise NotFoundError("doc_not_found", f"Document {sha256} not found.", "doc_ref")page_range = normalize_page_range_input(page_range)if page_range and doc["page_count"] is not None:- loaded_pages = filter_pages_by_user_range(loaded_pages, _expand_page_range(page_range, doc["page_count"]))- elif page_range and page_range != "all":- # 总页数未知时只允许绝对页码,不能用已缓存页数推断文档末页。- loaded_pages = filter_pages_by_user_range(loaded_pages, page_range)+ page_range = _expand_page_range(page_range, doc["page_count"])+ requested = parse_page_range_set(page_range) if page_range and page_range != "all" else None+ loaded_pages = load_pages_from_done_batches(+ data_dir,+ sha256,+ tier,+ cast(list[ParseBatchRow], rows),+ requested_page_numbers=requested,+ )+ if requested is not None:+ loaded_pages = [page for page in loaded_pages if page.page_idx + 1 in requested]if not loaded_pages:raise NotFoundError("not_cached", "Requested parsed content is not cached.", "page_range")output: list[str] = []@@ -1254,7 +1276,13 @@ class DoclibServer(AsyncDoclibInterface):"SELECT page_range, done_at FROM parses WHERE sha256=? AND tier=? AND status=? ORDER BY done_at DESC",(plan.sha256, plan.tier, PARSE_STATUS_DONE),)- loaded_pages = load_pages_from_done_batches(data_dir, plan.sha256, plan.tier, cast(list[ParseBatchRow], rows))+ loaded_pages = load_pages_from_done_batches(+ data_dir,+ plan.sha256,+ plan.tier,+ cast(list[ParseBatchRow], rows),+ requested_page_numbers=parse_page_range_set(plan.page_range) if plan.page_range else None,+ )if plan.page_range:loaded_pages = filter_pages_by_user_range(loaded_pages, plan.page_range)if not loaded_pages:@@ -1411,11 +1439,17 @@ class DoclibServer(AsyncDoclibInterface):rows = cast(list[ParseRow],await self.state.db.fetchall(- "SELECT tier FROM parses WHERE sha256=? AND status=? GROUP BY tier",+ "SELECT * FROM parses WHERE sha256=? AND status=? ORDER BY done_at DESC",(sha256, PARSE_STATUS_DONE),),)- tiers: set[Tier] = {row["tier"] for row in rows if row["tier"] in TIERS and row["tier"] != "flash"}+ tiers: set[Tier] = {+ row["tier"]+ for row in rows+ if row["tier"] in TIERS+ and row["tier"] != "flash"+ and _json_batch_is_current(_effective_data_dir(self.state), sha256, row["tier"], row)+ }if not tiers:return Nonereturn select_highest_cached_tier(tiers)diff --git a/mineru/doclib/services/parse_svc.py b/mineru/doclib/services/parse_svc.pyindex 39f3caa8..3a351a81 100644--- a/mineru/doclib/services/parse_svc.py+++ b/mineru/doclib/services/parse_svc.py@@ -745,7 +745,7 @@ class ParseService:),)valid_done_batches = [- batch for batch in done_batches if _json_file_exists_by_batch(self.data_dir, sha256, requested_tier, batch)+ batch for batch in done_batches if _json_batch_is_current(self.data_dir, sha256, requested_tier, batch)]if supports_page_range:for batch in valid_done_batches:@@ -1404,7 +1404,9 @@ class ParseService:tier_set: set[Tier] = {row["tier"] for row in rows}tiers = sorted(tier_set, key=lambda item: TIER_ORDER.get(item, -1), reverse=True)for tier in tiers:- tier_rows = [row for row in rows if row["tier"] == tier]+ tier_rows = [+ row for row in rows if row["tier"] == tier and _json_batch_is_current(self.data_dir, sha256, tier, row)+ ]pages = load_pages_from_done_batches(self.data_dir, sha256, tier, tier_rows)if not pages:continue@@ -1495,10 +1497,16 @@ def _probe_error_message(prefix: str, error_msg: str | None) -> str:return prefix-def _json_file_exists_by_batch(data_dir: str, sha256: str, tier: Tier, batch: ParseRow) -> bool:- """Check that the JSON result file for a parses batch row actually exists on disk."""+def _json_batch_is_current(data_dir: str, sha256: str, tier: Tier, batch: ParseRow) -> bool:+ """仅把协议有效且覆盖记录页范围的新批次视为可命中的缓存。"""json_path = parse_batch_json_path(data_dir, sha256, tier, batch["page_range"], batch["done_at"])- return os.path.isfile(json_path)+ try:+ with open(json_path, encoding="utf-8") as stream:+ result = ParseResult.from_dict(json.load(stream))+ available = {page.page_idx + 1 for page in result.pages}+ return bool(available) and parse_page_range_set(batch["page_range"]) <= available+ except (OSError, ValueError, TypeError):+ return Falsedef parse_batch_json_path(data_dir: str, sha256: str, tier: Tier, page_range: str, done_at: int | None = 0) -> str:@@ -1507,9 +1515,17 @@ def parse_batch_json_path(data_dir: str, sha256: str, tier: Tier, page_range: streturn os.path.join(os.path.expanduser(data_dir), "parsed", sha256[:2], sha256, tier, filename)-def load_pages_from_done_batches(data_dir: str, sha256: str, tier: Tier, done_rows: Sequence[ParseBatchRow]) -> list[PageInfo]:- """Load valid done JSON batches and keep the newest page for duplicate page_idx values."""+def load_pages_from_done_batches(+ data_dir: str,+ sha256: str,+ tier: Tier,+ done_rows: Sequence[ParseBatchRow],+ *,+ requested_page_numbers: set[int] | None = None,+) -> list[PageInfo]:+ """读取新协议批次并保留最新重复页,所请求的旧缓存需明确重新解析。"""pages_by_page_idx: dict[int, PageInfo] = {}+ stale_page_numbers: set[int] = set()for row in reversed(done_rows):fpath = parse_batch_json_path(data_dir, sha256, tier, row["page_range"], row["done_at"])if not os.path.isfile(fpath):@@ -1520,8 +1536,15 @@ def load_pages_from_done_batches(data_dir: str, sha256: str, tier: Tier, done_roparse_result = ParseResult.from_dict(data)for page in parse_result.pages:pages_by_page_idx[page.page_idx] = page- except Exception:- pass+ except (OSError, ValueError, TypeError):+ stale_page_numbers.update(parse_page_range_set(row["page_range"]))+ uncovered_stale = stale_page_numbers - {index + 1 for index in pages_by_page_idx}+ if requested_page_numbers is not None:+ uncovered_stale &= requested_page_numbers+ if uncovered_stale:+ raise MineruError(+ "stale_cache", "Cached document protocol is outdated or invalid; reparse the source document.", "page_range"+ )return [pages_by_page_idx[page_idx] for page_idx in sorted(pages_by_page_idx)]diff --git a/mineru/integrations/docvortex.py b/mineru/integrations/docvortex.pyindex 8971b65c..0f1a35bd 100644--- a/mineru/integrations/docvortex.py+++ b/mineru/integrations/docvortex.py@@ -1,102 +1,38 @@-"""在 DocVortex 共享文档对象与 MinerU 产品协议之间进行显式适配。"""+"""校验 MinerU 产品扩展,共享文档协议由 DocVortex 统一维护。"""from __future__ import annotations-from typing import Any, Literal--from pydantic import BaseModel, ConfigDict, Field, JsonValue--from docvortex.schema import MiddleJson, ModelJson, Producer+from typing import Literal+from pydantic import BaseModel, ConfigDict, JsonValue+from docvortex.schema import MiddleJson, ModelJsonclass MinerUMetadata(BaseModel):- """校验旧协议的产品元数据,禁止将未知档位静默映射。"""+ """仅记录实际执行的产品档位和最终文字提取模式。"""model_config = ConfigDict(extra="forbid", strict=True)- effort: Literal["flash", "medium", "high", "xhigh"]+ tier: Literal["flash", "basic", "standard", "advanced"]parse_mode: Literal["txt", "ocr"]- mineru_version: str = Field(min_length=1)-def build_metadata(*, effort: str, parse_mode: str, mineru_version: str) -> dict[str, JsonValue]:- """校验产品参数并生成命名空间扩展,不给公共语义类型添加宿主字段。"""- metadata = MinerUMetadata.model_validate({"effort": effort, "parse_mode": parse_mode, "mineru_version": mineru_version})+def build_metadata(*, effort: str, parse_mode: str) -> dict[str, JsonValue]:+ """将分析结果的实际 effort 映射为产品档位,不记录请求值。"""+ tiers = {"flash": "flash", "medium": "basic", "high": "standard", "xhigh": "advanced"}+ if effort not in tiers:+ raise ValueError(f"Unsupported actual analyze effort: {effort}")+ metadata = MinerUMetadata(tier=tiers[effort], parse_mode=parse_mode)return {"mineru": metadata.model_dump(mode="json")}def with_mineru_metadata(document: ModelJson | MiddleJson, metadata: MinerUMetadata) -> None:- """由宿主显式附加产品信息,文档本身不猜测 MinerU 版本。"""+ """附加产品扩展并保留其他应用信息,不修改文档生产者。"""document.extensions = {**document.extensions, "mineru": metadata.model_dump(mode="json")}-def _from_payload(payload: dict[str, Any], *, middle: bool) -> ModelJson | MiddleJson:- """把旧版顶层字段移入扩展,并复用唯一一套语义类型验证。"""- data = dict(payload)- data.pop("schema_version", None)- if "pages" not in data:- raise ValueError("Missing required pages in MinerU document")- metadata = MinerUMetadata.model_validate({key: data.pop(key, None) for key in MinerUMetadata.model_fields})- data["producer"] = Producer(name="mineru", version=metadata.mineru_version)- data["extensions"] = {"mineru": metadata.model_dump(mode="json")}- return MiddleJson.model_validate(data) if middle else ModelJson.model_validate(data)---def from_mineru_model(payload: dict[str, Any]) -> ModelJson:- """将当前 MinerU Model JSON 转换成中性分析对象。"""- result = _from_payload(payload, middle=False)- assert isinstance(result, ModelJson)- return result---def from_mineru_middle(payload: dict[str, Any]) -> MiddleJson:- """将当前 MinerU Middle JSON 转换成中性语义对象。"""- result = _from_payload(payload, middle=True)- assert isinstance(result, MiddleJson)- return result---def _to_payload(- document: ModelJson | MiddleJson, *, skip_defaults: bool, exclude_block_fields: set[str] | None = None-) -> dict[str, Any]:- """恢复旧版顶层字段,仅接受调用方真实提供的产品元数据。"""- metadata = MinerUMetadata.model_validate(document.extensions.get("mineru"))- payload = document.to_dict(skip_defaults=skip_defaults, exclude_block_fields=exclude_block_fields)- for key in ("schema", "schema_version", "producer", "extensions"):- payload.pop(key, None)- return {**payload, **metadata.model_dump(mode="json")}---def to_mineru_model(document: ModelJson, *, skip_defaults: bool = False) -> dict[str, Any]:- """写出与现有 MinerU 相同的 Model JSON 封装。"""- return _to_payload(document, skip_defaults=skip_defaults)---def to_mineru_middle(- document: MiddleJson,- *,- skip_defaults: bool = True,- include_schema_version: bool = True,- exclude_block_fields: set[str] | None = None,-) -> dict[str, Any]:- """写出 MinerU schema 2.0,允许调用方明确选择素材省略策略。"""- payload = _to_payload(document, skip_defaults=skip_defaults, exclude_block_fields=exclude_block_fields)- return {"schema_version": "2.0", **payload} if include_schema_version else payload---def to_mineru_structured_content(value: dict[str, Any]) -> dict[str, Any]:- """将共享 renderer 的中性文档头还原为现有 Structured Content 头。"""- metadata = MinerUMetadata.model_validate(value.get("extensions", {}).get("mineru"))- payload = {key: item for key, item in value.items() if key not in {"schema", "schema_version", "producer", "extensions"}}- return {**payload, **metadata.model_dump(mode="json")}+def validate_mineru_metadata(document: ModelJson | MiddleJson) -> None:+ """只校验已存在的 MinerU 扩展,原生及其他生产者文档可直接消费。"""+ if "mineru" in document.extensions:+ MinerUMetadata.model_validate(document.extensions["mineru"])-__all__ = [- "build_metadata",- "MinerUMetadata",- "from_mineru_model",- "from_mineru_middle",- "to_mineru_model",- "to_mineru_middle",- "to_mineru_structured_content",- "with_mineru_metadata",-]+__all__ = ["MinerUMetadata", "build_metadata", "with_mineru_metadata", "validate_mineru_metadata"]diff --git a/mineru/kit/gradio/artifacts.py b/mineru/kit/gradio/artifacts.pyindex e9c82c35..e7386060 100644--- a/mineru/kit/gradio/artifacts.py+++ b/mineru/kit/gradio/artifacts.py@@ -225,11 +225,7 @@ def persist_parse_result(shutil.copyfile(source_path, artifacts.source_path)artifacts.middle_json_path.write_text(result.to_json(), encoding="utf-8")if result._model_output is not None:- model_output = (- result._model_output.to_dict(skip_defaults=False)- if hasattr(result._model_output, "to_dict")- else result._model_output- )+ model_output = result._model_output.to_dict(skip_defaults=False)(artifacts.root / "model_output.json").write_text(json.dumps(model_output, ensure_ascii=False, indent=2),encoding="utf-8",diff --git a/mineru/parser/api_client.py b/mineru/parser/api_client.pyindex a2857e48..45a27152 100644--- a/mineru/parser/api_client.py+++ b/mineru/parser/api_client.py@@ -818,11 +818,17 @@ def _extract_model_output_from_archive(archive: zipfile.ZipFile) -> Any | None:raise _V1APIError("invalid_model_output", f"ZIP output contained multiple model outputs: {available}")model_output_name = legacy_names[0]+ from ..types import ModelJson+raw = archive.read(model_output_name)try:- return json.loads(raw)- except json.JSONDecodeError as exc:- raise _V1APIError("invalid_model_output", f"model output JSON is not valid JSON: {exc}") from exc+ model = ModelJson.from_json(raw)+ from ..integrations.docvortex import validate_mineru_metadata++ validate_mineru_metadata(model)+ return model+ except (ValueError, TypeError) as exc:+ raise _V1APIError("invalid_model_output", f"Invalid Model JSON document: {exc}") from excdef _download_zip_output(parser: MinerUApiParser, outputs: dict[str, Any]) -> bytes:@@ -1113,14 +1119,9 @@ def _pages_from_middle_json(mid_json: dict[str, Any] | None) -> list[PageInfo]:def _parse_result_from_middle_json(mid_json: dict[str, Any]) -> ParseResult:- """把远端 middle_json 恢复为 ParseResult;legacy pdf_info 交给 ParseResult.from_dict 迁移。"""+ """把远端 middle_json 恢复为 ParseResult;只接受当前共享文档协议。"""if not isinstance(mid_json, dict):raise _V1APIError("invalid_middle_json_output", "middle_json output must be a JSON object")- if mid_json.get("pages") is None and mid_json.get("pdf_info") is None:- raise _V1APIError(- "invalid_middle_json_output",- "middle_json output must contain a list field named pages or pdf_info",- )try:return ParseResult.from_dict(mid_json)except ValueError as exc:diff --git a/mineru/parser/base.py b/mineru/parser/base.pyindex 38a5f34a..265ad72e 100644--- a/mineru/parser/base.py+++ b/mineru/parser/base.py@@ -1,83 +1,20 @@# Copyright (c) Opendatalab. All rights reserved.from __future__ import annotations-from ..integrations.docvortex import from_mineru_middle, to_mineru_middle, to_mineru_model-from docvortex.schema import Producer-from ..integrations.docvortex import build_metadata+from ..integrations.docvortex import validate_mineru_metadataimport jsonfrom abc import ABC, abstractmethodfrom copy import deepcopyfrom dataclasses import dataclassfrom pathlib import Path-from typing import Any, Literal, cast+from typing import Anyfrom ..render import render_markdown, render_structured_contentfrom ..render.contracts import ImageRenderer, RenderMode-from ..types import FILE_SUFFIXES, FileSuffix, MiddleJson, ModelJson, PageInfo+from ..types import MiddleJson, ModelJson, PageInfofrom .writer import DataWriter-MIDDLE_JSON_SCHEMA_VERSION: str = "2.0"-_LEGACY_SCHEMA_VERSION: str = "1.0"-_LEGACY_DEFAULT_FILE_SUFFIX: FileSuffix = "pdf"-_LEGACY_DEFAULT_EFFORT: Literal["flash", "medium", "high", "xhigh"] = "medium"-_LEGACY_DEFAULT_PARSE_MODE: Literal["txt", "ocr"] = "txt"-_TO_DICT_EXCLUDED_KEYS: frozenset[str] = frozenset({"schema_version"})---def _legacy_raw_pages(payload: dict[str, Any]) -> list[dict[str, Any]] | None:- """按 envelope 识别 3.4.5 原始 pdf_info 或后续 1.0 pages 包装。-- 3.4.5 release tag 内部仍报告 3.4.4,因此不能把 ``_version_name`` 作为- 唯一判据。- """- schema_version = payload.get("schema_version")- raw_pages = payload.get("pdf_info") if schema_version is None else None- if raw_pages is None and schema_version == _LEGACY_SCHEMA_VERSION:- raw_pages = payload.get("pages")- if not isinstance(raw_pages, list) or any(not isinstance(page, dict) for page in raw_pages):- return None- return raw_pages---def _legacy_page_index_map(raw_pages: list[dict[str, Any]]) -> list[int]:- """从旧 page_idx 恢复抽页映射;连续零起始页面仍表示整本文档。"""- page_indices: list[int] = []- for fallback_index, page in enumerate(raw_pages):- page_idx = page.get("page_idx", fallback_index)- if isinstance(page_idx, bool) or not isinstance(page_idx, int) or page_idx < 0:- raise ValueError("legacy Middle JSON page_idx values must be non-negative integers")- page_indices.append(page_idx)- if len(page_indices) != len(set(page_indices)) or any(- current <= previous for previous, current in zip(page_indices, page_indices[1:])- ):- raise ValueError("legacy Middle JSON page_idx values must be unique and strictly increasing")- return [] if page_indices == list(range(len(raw_pages))) else page_indices---def _legacy_file_suffix(payload: dict[str, Any]) -> FileSuffix:- """读取旧 payload 可选后缀;缺失时沿用历史 PDF 默认值。"""- value = payload.get("file_suffix")- if isinstance(value, str) and value in FILE_SUFFIXES:- return cast(FileSuffix, value)- return _LEGACY_DEFAULT_FILE_SUFFIX---def _legacy_effort(payload: dict[str, Any]) -> Literal["flash", "medium", "high", "xhigh"]:- """把 3.4.5 的分析强度映射到当前严格枚举。"""- value = payload.get("_effort", payload.get("effort"))- if value in {"flash", "medium", "high", "xhigh"}:- return cast(Literal["flash", "medium", "high", "xhigh"], value)- return _LEGACY_DEFAULT_EFFORT---def _legacy_parse_mode(payload: dict[str, Any]) -> Literal["txt", "ocr"]:- """优先读取显式 parse_mode,否则按 3.4.5 的 _ocr_enable 推断。"""- value = payload.get("parse_mode")- if value in {"txt", "ocr"}:- return cast(Literal["txt", "ocr"], value)- if payload.get("_ocr_enable") is True:- return "ocr"- return _LEGACY_DEFAULT_PARSE_MODE+MIDDLE_JSON_SCHEMA_VERSION: str = MiddleJson.model_fields["schema_version"].default@dataclass@@ -89,7 +26,14 @@ class ParseResult:"""middle_json: MiddleJson- _model_output: Any = None+ _model_output: ModelJson | None = None++ def __post_init__(self) -> None:+ """要求可选分析结果为共享类型,禁止把任意旧列表重新导出为 Model JSON。"""+ if self._model_output is not None:+ if not isinstance(self._model_output, ModelJson):+ raise TypeError("model output must be a ModelJson document")+ validate_mineru_metadata(self._model_output)@propertydef pages(self) -> list[PageInfo]:@@ -98,58 +42,17 @@ class ParseResult:@staticmethoddef from_dict(d: dict[str, Any]) -> ParseResult:- if not isinstance(d, dict):- raise ValueError("ParseResult.from_dict expects a dict.")-- schema_version = d.get("schema_version")-- if schema_version == MIDDLE_JSON_SCHEMA_VERSION:- middle_json = ParseResult._build_middle_json_from_current(d)- elif (raw_pages := _legacy_raw_pages(d)) is not None:- middle_json = ParseResult._build_middle_json_from_legacy(d, raw_pages)- else:- raise ValueError(- f"Unsupported Middle JSON schema_version={schema_version!r}; "- f"expected {MIDDLE_JSON_SCHEMA_VERSION!r}. Reparse the source document."- )-+ """只读取统一新版协议,保留真实生产者并校验可选产品扩展。"""+ middle_json = MiddleJson.from_dict(d)+ validate_mineru_metadata(middle_json)return ParseResult(middle_json=middle_json)- @staticmethod- def _build_middle_json_from_current(d: dict[str, Any]) -> MiddleJson:- """从当前版本 schema 直接构造 Span 化 MiddleJson。"""- payload = {k: v for k, v in d.items() if k not in _TO_DICT_EXCLUDED_KEYS}- return from_mineru_middle(payload)-- @staticmethod- def _build_middle_json_from_legacy(d: dict[str, Any], raw_pages: list[dict[str, Any]]) -> MiddleJson:- """把 3.4.5 页面回推为 raw ModelJson,再走当前统一后处理生成 2.0。"""- from ..backend.postprocess.legacy_schema_adapter import legacy_page_to_model_list- from ..version import __version__ as current_mineru_version-- source_version = d.get("_version_name", d.get("mineru_version"))- mineru_version = (- source_version.strip() if isinstance(source_version, str) and source_version.strip() else current_mineru_version- )- model_json = ModelJson(- pages=[legacy_page_to_model_list(page) for page in raw_pages],- page_index_map=_legacy_page_index_map(raw_pages),- file_suffix=_legacy_file_suffix(d),- producer=Producer(name="mineru", version=mineru_version),- extensions=build_metadata(- effort=_legacy_effort(d), parse_mode=_legacy_parse_mode(d), mineru_version=mineru_version- ),- )- from docvortex.postprocess.document import model_json_to_middle_json-- return model_json_to_middle_json(model_json)-def to_dict(self, *, skip_defaults: bool = True) -> dict[str, Any]:- """保留 schema 2.0 输出及 PDF 图片省略约定,由独立 codec 还原封装。"""- return to_mineru_middle(- self.middle_json,+ """输出共享协议,并保留 PDF 图片省略约定且不修改源对象。"""+ validate_mineru_metadata(self.middle_json)+ return self.middle_json.to_dict(skip_defaults=skip_defaults,- exclude_block_fields={"image_base64"} if self.middle_json.file_suffix == "pdf" else None,+ exclude_block_fields={"image_base64"} if self.middle_json.metadata.file_suffix == "pdf" else None,)@staticmethod@@ -192,11 +95,8 @@ class ParseResult:)if self._model_output is not None:- model_output = (- to_mineru_model(self._model_output, skip_defaults=False)- if isinstance(self._model_output, ModelJson)- else self._model_output- )+ validate_mineru_metadata(self._model_output)+ model_output = self._model_output.to_dict(skip_defaults=False)writer.write_string("model_output.json",json.dumps(model_output, ensure_ascii=False, indent=2),diff --git a/mineru/render/structured_content.py b/mineru/render/structured_content.pyindex f639af70..95d937de 100644--- a/mineru/render/structured_content.py+++ b/mineru/render/structured_content.py@@ -3,7 +3,7 @@from __future__ import annotations-from ..integrations.docvortex import to_mineru_structured_content+from ..integrations.docvortex import validate_mineru_metadatafrom typing import Anyfrom ..types import MiddleJson@@ -18,8 +18,9 @@ def render_structured_content(from ..config import configfrom docvortex.render.structured_content import render_structured_content as _render_structured_content- return to_mineru_structured_content(- _render_structured_content(middle_json, asset_base_url=asset_base_url, latex_delimiters=config.render.latex_delimiters)+ validate_mineru_metadata(middle_json)+ return _render_structured_content(+ middle_json, asset_base_url=asset_base_url, latex_delimiters=config.render.latex_delimiters)diff --git a/mineru/types.py b/mineru/types.pyindex 0652bd26..90a12066 100644--- a/mineru/types.py+++ b/mineru/types.py@@ -78,6 +78,7 @@ from docvortex.schema import (parse_block,Producer,DocumentModel,+ DocumentMetadata,ModelJson,PageInfo,MiddleJson,@@ -299,6 +300,7 @@ __all__ = ["parse_block","Producer","DocumentModel",+ "DocumentMetadata","ModelJson","PageInfo","MiddleJson",diff --git a/pyproject.toml b/pyproject.tomlindex e6095e6b..b3fa82f4 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -38,7 +38,7 @@ classifiers = ["Programming Language :: Python :: 3.13",]dependencies = [- "docvortex>=0.1.0,<1.0.0",+ "docvortex>=0.2.0,<0.3.0","mineru-vl-utils>=1.2.1,<2","metafile-render>=0.3.0,<1.0.0","mineru-llama-cpp>=0.1.0,<1",diff --git a/tests/unittest/test_content_list_ownership.py b/tests/unittest/test_content_list_ownership.pyindex 4406713c..bbca7726 100644--- a/tests/unittest/test_content_list_ownership.py+++ b/tests/unittest/test_content_list_ownership.py@@ -51,7 +51,7 @@ def test_migrated_inline_delimiter_cases(version: int, monkeypatch: pytest.Monke],)],- file_suffix="html",+ metadata={"file_suffix": "html", "producer": {"name": "docvortex", "version": "0.2.0"}},is_full_document=True,)before = middle.to_dict(skip_defaults=False)diff --git a/tests/unittest/test_content_list_render.py b/tests/unittest/test_content_list_render.pyindex 02f35755..24fb167e 100644--- a/tests/unittest/test_content_list_render.py+++ b/tests/unittest/test_content_list_render.py@@ -35,9 +35,11 @@ def _middle(*pages: PageInfo) -> MiddleJson:return MiddleJson(pages=list(pages),is_full_document=True,- file_suffix="docx",- producer=Producer(name="mineru", version="test"),- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="test"),+ metadata={"file_suffix": "docx", "producer": Producer(name="mineru", version="test")},+ extensions=build_metadata(+ effort="flash",+ parse_mode="txt",+ ),)diff --git a/tests/unittest/test_content_list_v2_render.py b/tests/unittest/test_content_list_v2_render.pyindex a4f02271..81c2793e 100644--- a/tests/unittest/test_content_list_v2_render.py+++ b/tests/unittest/test_content_list_v2_render.py@@ -35,9 +35,11 @@ def _middle(*pages: PageInfo) -> MiddleJson:return MiddleJson(pages=list(pages),is_full_document=True,- file_suffix="docx",- producer=Producer(name="mineru", version="test"),- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="test"),+ metadata={"file_suffix": "docx", "producer": Producer(name="mineru", version="test")},+ extensions=build_metadata(+ effort="flash",+ parse_mode="txt",+ ),)diff --git a/tests/unittest/test_doclib_cache_semantics.py b/tests/unittest/test_doclib_cache_semantics.pyindex 68278ab1..25984c9c 100644--- a/tests/unittest/test_doclib_cache_semantics.py+++ b/tests/unittest/test_doclib_cache_semantics.py@@ -289,10 +289,9 @@ def _write_batch("schema_version": MIDDLE_JSON_SCHEMA_VERSION,"pages": json_pages,"is_full_document": is_full_document,- "file_suffix": file_suffix,- "effort": "medium",- "parse_mode": "txt",- "mineru_version": __version__,+ "metadata": {"file_suffix": file_suffix, "producer": {"name": "mineru", "version": __version__}},+ "schema": "docvortex.middle",+ "extensions": {"mineru": {"tier": "basic", "parse_mode": "txt"}},}path.write_text(json.dumps(payload, ensure_ascii=False, indent=4), encoding="utf-8")@@ -476,16 +475,18 @@ def test_parsing_rule_default_tier_allows_flash_fallback(monkeypatch: pytest.Monassert _resolve_parsing_rule_default_tier() == "flash"-def test_read_without_tier_uses_highest_cached_tier() -> None:- class _DB:- async def fetchall(self, sql: str, params: tuple[Any, ...]) -> list[dict[str, str]]:- assert sql == "SELECT tier FROM parses WHERE sha256=? AND status=? GROUP BY tier"- assert params == ("a" * 64, "done")- return [{"tier": "flash"}, {"tier": "basic"}, {"tier": "standard"}, {"tier": "advanced"}]-- server = DoclibServer(SimpleNamespace(db=_DB()))-- assert asyncio.run(server._default_read_tier("a" * 64)) == "advanced"+def test_read_without_tier_uses_highest_cached_tier(tmp_path: Path) -> None:+ """默认档位仅从有效协议缓存选择,忽略更高档位的历史结果。"""+ sha256 = "a" * 64+ rows = []+ for tier in ["flash", "basic", "standard", "advanced"]:+ _write_batch(tmp_path, sha256, tier, "1", 1000, [_text_page(tier).to_dict()])+ rows.append({"sha256": sha256, "tier": tier, "page_range": "1", "done_at": 1000, "status": "done"})+ server = DoclibServer(SimpleNamespace(db=_FakeDB(parses=rows, file_row=None), data_dir=str(tmp_path)))+ assert asyncio.run(server._default_read_tier(sha256)) == "advanced"+ stale = Path(parse_batch_json_path(str(tmp_path), sha256, "advanced", "1", 1000))+ stale.write_text('{"schema_version":"1.0","pages":[]}')+ assert asyncio.run(server._default_read_tier(sha256)) == "standard"def test_managed_local_parse_server_url_uses_health_managed_url() -> None:@@ -1223,9 +1224,15 @@ def test_remote_parse_server_default_url_is_declared_once() -> None:def test_compaction_uses_configured_data_dir(tmp_path: Path) -> None:sha256 = "b" * 64tier = "standard"- older_page = {"page_idx": 0, "content": "old"}- older_duplicate = {"page_idx": 1, "content": "old"}- newer_duplicate = {"page_idx": 1, "content": "new"}+ older_page = {+ "page_idx": 0,+ "blocks": [{"type": "text", "index": 0, "bbox": [0.0, 0.0, 0.1, 0.1], "content": _inline("old")}],+ }+ older_duplicate = {"page_idx": 1, "blocks": older_page["blocks"]}+ newer_duplicate = {+ "page_idx": 1,+ "blocks": [{"type": "text", "index": 0, "bbox": [0.0, 0.0, 0.1, 0.1], "content": _inline("new")}],+ }_write_batch(tmp_path, sha256, tier, "1-2", 1000, [older_page, older_duplicate])_write_batch(tmp_path, sha256, tier, "2", 2000, [newer_duplicate])@@ -1352,8 +1359,8 @@ def test_invalidate_deletes_fts_when_no_done_batches_remain(tmp_path: Path) -> Ndef test_invalidate_rebuilds_fts_from_highest_remaining_done_tier(tmp_path: Path) -> None:sha256 = "d" * 64- _write_batch(tmp_path, sha256, "flash", "1", 1000, [{"page_idx": 1, "blocks": []}])- _write_batch(tmp_path, sha256, "standard", "1", 2000, [{"page_idx": 1, "blocks": []}])+ _write_batch(tmp_path, sha256, "flash", "1", 1000, [{"page_idx": 0, "blocks": []}])+ _write_batch(tmp_path, sha256, "standard", "1", 2000, [{"page_idx": 0, "blocks": []}])parses = [{"sha256": sha256, "tier": "flash", "page_range": "1", "status": "done", "done_at": 1000},{"sha256": sha256, "tier": "standard", "page_range": "1", "status": "done", "done_at": 2000},@@ -1803,8 +1810,8 @@ def test_csv_doclib_parse_creates_flash_cache_and_rendered_fts(cached_files = list((tmp_path / "data" / "parsed").rglob("*.json"))assert len(cached_files) == 1cached_payload = json.loads(cached_files[0].read_text(encoding="utf-8"))- assert cached_payload["file_suffix"] == "csv"- assert cached_payload["effort"] == "flash"+ assert cached_payload["metadata"]["file_suffix"] == "csv"+ assert cached_payload["extensions"]["mineru"]["tier"] == "flash"finally:await db.close()@@ -2168,9 +2175,11 @@ def test_remap_api_result_pages_to_non_contiguous_page_range() -> None:middle_json=MiddleJson(pages=[PageInfo(page_idx=0), PageInfo(page_idx=1), PageInfo(page_idx=2)],is_full_document=False,- file_suffix="pdf",- producer=Producer(name="mineru", version=__version__),- extensions=build_metadata(effort="medium", parse_mode="txt", mineru_version=__version__),+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version=__version__)},+ extensions=build_metadata(+ effort="medium",+ parse_mode="txt",+ ),))@@ -2187,10 +2196,9 @@ def test_remap_api_result_pages_remaps_page_indices() -> None:"schema_version": MIDDLE_JSON_SCHEMA_VERSION,"pages": [{"page_idx": 0, "blocks": []}],"is_full_document": False,- "file_suffix": "pdf",- "effort": "medium",- "parse_mode": "txt",- "mineru_version": __version__,+ "metadata": {"file_suffix": "pdf", "producer": {"name": "mineru", "version": __version__}},+ "schema": "docvortex.middle",+ "extensions": {"mineru": {"tier": "basic", "parse_mode": "txt"}},})@@ -2206,9 +2214,11 @@ def test_remap_api_result_pages_rejects_count_mismatch() -> None:middle_json=MiddleJson(pages=[PageInfo(page_idx=0), PageInfo(page_idx=1)],is_full_document=False,- file_suffix="pdf",- producer=Producer(name="mineru", version=__version__),- extensions=build_metadata(effort="medium", parse_mode="txt", mineru_version=__version__),+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version=__version__)},+ extensions=build_metadata(+ effort="medium",+ parse_mode="txt",+ ),))@@ -4421,9 +4431,11 @@ def test_process_doc_marks_empty_page_result_failed(tmp_path: Path) -> None:middle_json=MiddleJson(pages=[],is_full_document=False,- file_suffix="pdf",- producer=Producer(name="mineru", version=__version__),- extensions=build_metadata(effort="medium", parse_mode="txt", mineru_version=__version__),+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version=__version__)},+ extensions=build_metadata(+ effort="medium",+ parse_mode="txt",+ ),))@@ -4585,9 +4597,11 @@ def test_parse_via_api_requests_image_cache_only_for_office(middle_json=MiddleJson(pages=[],is_full_document=True,- file_suffix="pdf",- producer=Producer(name="mineru", version=__version__),- extensions=build_metadata(effort="medium", parse_mode="txt", mineru_version=__version__),+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version=__version__)},+ extensions=build_metadata(+ effort="medium",+ parse_mode="txt",+ ),))@@ -4653,9 +4667,11 @@ def test_process_doc_fails_when_batch_json_cannot_be_written(tmp_path: Path) ->middle_json=MiddleJson(pages=[PageInfo(page_idx=0)],is_full_document=False,- file_suffix="pdf",- producer=Producer(name="mineru", version=__version__),- extensions=build_metadata(effort="medium", parse_mode="txt", mineru_version=__version__),+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version=__version__)},+ extensions=build_metadata(+ effort="medium",+ parse_mode="txt",+ ),))@@ -4716,9 +4732,11 @@ def test_process_doc_normalizes_full_document_range_from_actual_pages(tmp_path:middle_json=MiddleJson(pages=[PageInfo(page_idx=page_idx) for page_idx in range(12)],is_full_document=True,- file_suffix="epub",- producer=Producer(name="mineru", version=__version__),- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version=__version__),+ metadata={"file_suffix": "epub", "producer": Producer(name="mineru", version=__version__)},+ extensions=build_metadata(+ effort="flash",+ parse_mode="txt",+ ),))diff --git a/tests/unittest/test_doclib_legacy_page_ranges.py b/tests/unittest/test_doclib_legacy_page_ranges.pyindex f1f24f9a..023a9afb 100644--- a/tests/unittest/test_doclib_legacy_page_ranges.py+++ b/tests/unittest/test_doclib_legacy_page_ranges.py@@ -23,10 +23,84 @@ from mineru.doclib.services.parse_svc import ()from mineru.parser.base import MIDDLE_JSON_SCHEMA_VERSIONfrom mineru.version import __version__+from mineru.errors import MineruError_SHA256 = "a" * 64[email protected]("old_version", [None, "1.0"])+def test_old_protocol_cache_is_not_available_or_compacted(tmp_path: Path, old_version: str | None) -> None:+ """旧协议不命中、不参与覆盖和压缩,读取旧页明确要求重新解析。"""++ async def verify() -> None:+ """在真实数据库中混存新旧批次,验证读写和缓存调度边界。"""+ async with _legacy_store(tmp_path) as (db, service, server):+ old_path = _cache_path(tmp_path, "1~5", 1000)+ payload = json.loads(old_path.read_text())+ if old_version is None:+ payload.pop("schema")+ else:+ payload["schema_version"] = old_version+ old_path.write_text(json.dumps(payload))+ await _add_result(db, tmp_path, "4-7", 2000, range(4, 8), "new")+ rows = await db.fetchall("SELECT * FROM parses ORDER BY done_at DESC")+ before_files = {path.name: path.read_bytes() for path in old_path.parent.iterdir()}+ compaction = Compaction(db=db, interval_sec=600, data_dir=str(tmp_path))+ assert await compaction._compact_doc_tier(_SHA256, "flash") == 0+ assert {path.name: path.read_bytes() for path in old_path.parent.iterdir()} == before_files+ with pytest.raises(MineruError, match="reparse"):+ load_pages_from_done_batches(str(tmp_path), _SHA256, "flash", rows)+ current = await server._render_doc_content(+ _SHA256, tier="flash", page_range="4-7", format="markdown", no_marker=True+ )+ assert "new page 4" in current and "old page" not in current+ with pytest.raises(MineruError, match="reparse"):+ await server._render_doc_content(_SHA256, tier="flash", page_range="1-3", format="markdown", no_marker=True)+ listing = await server.list_parses(doc_ref="aaaaaaa", tier="flash", page_range="1-7")+ assert listing.coverage.done_page_range == "4-7"+ assert listing.coverage.missing_page_range == "1-3"+ result = await service.request_parse(str(tmp_path / "document.pdf"), tier="flash", page_range="1-7")+ assert not result.cache_hit and len(result.created_parse_ids) == 1+ queued = await db.fetchone("SELECT page_range FROM parses WHERE id=?", (result.created_parse_ids[0],))+ assert queued["page_range"] == "1-3"+ assert old_path.read_bytes() == before_files[old_path.name]++ asyncio.run(verify())++[email protected]("field", ["metadata", "extensions", "is_full_document", "extension_value_type"])+def test_compaction_preserves_batches_with_conflicting_envelopes(tmp_path: Path, field: str) -> None:+ """不同来源、扩展或整本语义的批次不得被合并为虚假来源的文档。"""++ async def verify() -> None:+ """逐项注入头部冲突并确认源文件和数据库完全保留。"""+ async with _legacy_store(tmp_path) as (db, _service, _server):+ await _add_result(db, tmp_path, "4-7", 2000, range(4, 8), "new")+ path = _cache_path(tmp_path, "4-7", 2000)+ payload = json.loads(path.read_text())+ if field == "metadata":+ payload[field]["producer"]["version"] = "different"+ elif field == "extensions":+ payload[field]["application"] = {"source": "different"}+ elif field == "extension_value_type":+ old_path = _cache_path(tmp_path, "1~5", 1000)+ old_payload = json.loads(old_path.read_text())+ old_payload["extensions"]["application"] = {"value": True}+ old_path.write_text(json.dumps(old_payload))+ payload["extensions"]["application"] = {"value": 1}+ else:+ payload[field] = True+ path.write_text(json.dumps(payload))+ rows = await db.fetchall("SELECT * FROM parses ORDER BY id")+ files = {item.name: item.read_bytes() for item in path.parent.iterdir()}+ compaction = Compaction(db=db, interval_sec=600, data_dir=str(tmp_path))+ assert await compaction._compact_doc_tier(_SHA256, "flash") == 0+ assert await db.fetchall("SELECT * FROM parses ORDER BY id") == rows+ assert {item.name: item.read_bytes() for item in path.parent.iterdir()} == files++ asyncio.run(verify())++def _cache_path(root: Path, page_range: str, done_at: int) -> Path:"""按数据库原始范围定位批次文件,保留旧半角波浪号文件名。"""return Path(parse_batch_json_path(str(root), _SHA256, "flash", page_range, done_at))@@ -41,10 +115,9 @@ async def _add_result(payload = {"schema_version": MIDDLE_JSON_SCHEMA_VERSION,"is_full_document": False,- "file_suffix": "pdf",- "effort": "flash",- "parse_mode": "txt",- "mineru_version": __version__,+ "metadata": {"file_suffix": "pdf", "producer": {"name": "mineru", "version": __version__}},+ "schema": "docvortex.middle",+ "extensions": {"mineru": {"tier": "flash", "parse_mode": "txt"}},"pages": [{"page_idx": page_no - 1,diff --git a/tests/unittest/test_docvortex_protocol.py b/tests/unittest/test_docvortex_protocol.pyindex 716c8516..45ead2d3 100644--- a/tests/unittest/test_docvortex_protocol.py+++ b/tests/unittest/test_docvortex_protocol.py@@ -1,116 +1,95 @@-"""验证 MinerU 独立维护产品封装,DocVortex 只提供共享语义对象。"""+"""验证宿主与原生文档共享协议,同时保留产品扩展及素材约定。"""-from collections.abc import Callablefrom copy import deepcopy-from typing import Literal-import pytestfrom pydantic import ValidationError--from docvortex.schema import EquationBlock, ImageBlock, ImageBodyBlock, MiddleJson, PageInfo, TextBlock, TextSpan-from mineru.integrations.docvortex import (- MinerUMetadata,- build_metadata,- from_mineru_middle,- to_mineru_middle,- with_mineru_metadata,-)-from mineru.render.structured_content import render_structured_content+from docvortex.schema import DocumentMetadata, Producer, MiddleJson, PageInfo, TextBlock, TextSpan, ImageBodyBlock, ImageBlock+from mineru.integrations.docvortex import MinerUMetadata, build_metadata, validate_mineru_metadata, with_mineru_metadatafrom mineru.parser import ParseResult+from mineru.render.structured_content import render_structured_contentdef _document() -> MiddleJson:- """建立不携带宿主信息的最小语义文档,覆盖文字和公式。"""+ """建立有真实正文且无 MinerU 运行记录的原生文档。"""return MiddleJson(pages=[- PageInfo(- page_idx=0,- blocks=[- TextBlock(type="text", index=0, content=[TextSpan(type="text", content="Hello 文档")]),- EquationBlock(type="equation", index=1, content="x^2"),- ],- )+ PageInfo(page_idx=0, blocks=[TextBlock(type="text", index=0, content=[TextSpan(type="text", content="保留正文")])])],is_full_document=True,- file_suffix="html",+ metadata=DocumentMetadata(file_suffix="html", producer=Producer(name="docvortex", version="0.2.0")),)-def test_mineru_envelope_roundtrip() -> None:- """从引擎迁入的往返契约保证产品字段和页面语义不变。"""- payload = _document().to_dict(skip_defaults=False)- for key in ("schema", "schema_version", "producer", "extensions"):- payload.pop(key, None)- payload.update(schema_version="2.0", effort="flash", parse_mode="txt", mineru_version="3.4.5")- before = deepcopy(payload)-- assert to_mineru_middle(from_mineru_middle(payload), skip_defaults=False) == payload- assert payload == before+def test_native_middle_does_not_lose_text_or_rewrite_producer() -> None:+ """回归原生 1.0 被误认作宿主历史格式后正文丢失的问题。"""+ document = _document()+ before = document.to_dict()+ result = ParseResult.from_dict(before)+ assert result.to_dict() == before+ assert result.pages[0].blocks[0].content[0].content == "保留正文"+ assert result.middle_json.metadata.producer.name == "docvortex"+ structured = render_structured_content(document)+ assert structured["metadata"] == before["metadata"]+ assert structured["extensions"] == {}+ assert not {"schema", "schema_version", "schema_id"} & structured.keys()+ assert structured["pages"][0]["blocks"][0]["content"] == "保留正文"+ assert document.to_dict() == before-def test_host_metadata_preserves_other_extensions_and_rendering_does_not_mutate_document() -> None:- """宿主附加产品信息后仍保留其它扩展,公共渲染只加工输出封装。"""- document = _document()- document.extensions = {"application": {"source": "fixture"}}- metadata = MinerUMetadata(effort="high", parse_mode="ocr", mineru_version="3.4.5")- with_mineru_metadata(document, metadata)- before = document.to_dict(skip_defaults=False)[email protected]("effort,tier", [("flash", "flash"), ("medium", "basic"), ("high", "standard"), ("xhigh", "advanced")])[email protected]("mode", ["txt", "ocr"])+def test_actual_effort_maps_to_product_tier(effort: str, tier: str, mode: str) -> None:+ """所有分析强度均映射为实际公开档位,扩展不重复存放版本。"""+ assert build_metadata(effort=effort, parse_mode=mode) == {"mineru": {"tier": tier, "parse_mode": mode}}- payload = to_mineru_middle(document)- structured = render_structured_content(document)- assert document.extensions["application"] == {"source": "fixture"}- assert document.extensions["mineru"] == metadata.model_dump(mode="json")- assert build_metadata(**metadata.model_dump()) == {"mineru": document.extensions["mineru"]}- assert payload["schema_version"] == "2.0"- assert structured["pages"][0]["blocks"][0]["content"] == "Hello 文档"- for output in (payload, structured):- assert {key: output[key] for key in MinerUMetadata.model_fields} == metadata.model_dump(mode="json")- assert not {"schema", "producer", "extensions"} & output.keys()- assert "schema_version" not in structured- assert document.to_dict(skip_defaults=False) == before+def test_extensions_survive_host_rendering_and_roundtrip() -> None:+ """保留未知应用扩展,产品登记和渲染均不改写真实来源。"""+ document = _document()+ document.extensions = {"application": {"items": [None, True, 3, {"text": "中文"}]}}+ with_mineru_metadata(document, MinerUMetadata(tier="advanced", parse_mode="ocr"))+ before = deepcopy(document.to_dict())+ assert ParseResult.from_dict(before).to_dict() == before+ assert render_structured_content(document)["extensions"] == before["extensions"]+ assert document.to_dict() == before[email protected]("operation", [to_mineru_middle, render_structured_content])-def test_host_output_requires_explicit_mineru_metadata(operation: Callable[[MiddleJson], object]) -> None:- """没有产品元数据的原生返回值不能被静默伪装为 MinerU 输出。"""+ "metadata",+ [+ {"tier": "low", "parse_mode": "txt"},+ {"tier": "basic", "parse_mode": "auto"},+ {"tier": "flash", "parse_mode": "txt", "effort": "flash"},+ None,+ ],+)+def test_mineru_validates_only_present_product_extension(metadata: object) -> None:+ """原生扩展可缺省,已存在的产品记录则必须满足严格枚举。"""+ document = _document()+ validate_mineru_metadata(document)+ document.extensions = {"mineru": metadata}with pytest.raises(ValidationError):- operation(_document())+ ParseResult.from_dict(document.to_dict())@pytest.mark.parametrize("suffix", ["pdf", "docx"])-def test_parse_result_preserves_format_specific_image_omission(suffix: Literal["pdf", "docx"]) -> None:- """宿主结果写出时仅省略 PDF 内嵌图片,保留图片路径且不修改语义对象。"""+def test_format_specific_image_omission_is_preserved(suffix: str) -> None:+ """PDF 包装继续省略内嵌图片,其他格式保留,且源对象不变。"""body = ImageBodyBlock(type="image_body",index=0,content="figure",bbox=(0.1, 0.1, 0.9, 0.9),- image_path="images/figure.png",+ image_path="images/a.png",image_base64="data:image/png;base64,aW1hZ2U=",)document = MiddleJson(pages=[PageInfo(page_idx=0, blocks=[ImageBlock(type="image", index=0, bbox=body.bbox, content=[body])])],is_full_document=True,- file_suffix=suffix,- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="3.4.5"),+ metadata=DocumentMetadata(file_suffix=suffix, producer=Producer(name="external", version="original")),)before = document.to_dict(skip_defaults=False)-payload = ParseResult(document).to_dict()-- exported_body = payload["pages"][0]["blocks"][0]["content"][0]- assert ("image_base64" in exported_body) is (suffix != "pdf")- assert exported_body["image_path"] == "images/figure.png"+ assert ("image_base64" in payload["pages"][0]["blocks"][0]["content"][0]) is (suffix != "pdf")+ assert payload["pages"][0]["blocks"][0]["content"][0]["image_path"] == "images/a.png"assert document.to_dict(skip_defaults=False) == before--- ("field", "value"),- [("effort", "low"), ("parse_mode", "auto"), ("mineru_version", ""), ("mineru_version", 345)],-)-def test_host_metadata_validation_stays_strict(field: str, value: object) -> None:- """产品枚举、版本非空约束及类型检查由 MinerU 继续严格执行。"""- metadata = {"effort": "flash", "parse_mode": "txt", "mineru_version": "3.4.5", field: value}- with pytest.raises(ValidationError):- build_metadata(**metadata)diff --git a/tests/unittest/test_flash_csv.py b/tests/unittest/test_flash_csv.pyindex 57ed8e41..b8045700 100644--- a/tests/unittest/test_flash_csv.py+++ b/tests/unittest/test_flash_csv.py@@ -21,8 +21,8 @@ def test_csv_doc_analyze_sync_async_and_render_contracts_match() -> None:middle, model = doc_analyze(payload, file_suffix="csv")async_middle, async_model = asyncio.run(aio_doc_analyze(payload, file_suffix="csv"))- assert model.file_suffix == async_model.file_suffix == "csv"- assert model.extensions["mineru"]["effort"] == async_model.extensions["mineru"]["effort"] == "flash"+ assert model.metadata.file_suffix == async_model.metadata.file_suffix == "csv"+ assert model.extensions["mineru"]["tier"] == async_model.extensions["mineru"]["tier"] == "flash"assert model.extensions["mineru"]["parse_mode"] == async_model.extensions["mineru"]["parse_mode"] == "txt"assert middle.model_dump() == async_middle.model_dump()assert len(middle.pages) == 1@@ -50,7 +50,7 @@ def test_csv_path_parsing_and_signatureless_detection(tmp_path: Path) -> None:result = parse(csv_path)async_result = asyncio.run(parse_async(csv_path))- assert result.middle_json.file_suffix == async_result.middle_json.file_suffix == "csv"+ assert result.middle_json.metadata.file_suffix == async_result.middle_json.metadata.file_suffix == "csv"assert result.markdown() == async_result.markdown()with pytest.raises(ValueError, match="Unsupported file type: txt"):parse(text_path)@@ -95,12 +95,12 @@ def test_csv_parse_server_job_emits_structured_outputs_with_flash_metadata(tmp_pmiddle_record = file_store.get_file(parsed_file.output_files.middle_json.file_id)assert middle_record.sha256sum is not Nonemiddle_payload = json.loads(file_store.read_blob(middle_record.sha256sum))- assert middle_payload["file_suffix"] == "csv"- assert middle_payload["effort"] == "flash"- assert middle_payload["parse_mode"] == "txt"+ assert middle_payload["metadata"]["file_suffix"] == "csv"+ assert middle_payload["extensions"]["mineru"]["tier"] == "flash"+ assert middle_payload["extensions"]["mineru"]["parse_mode"] == "txt"structured_record = file_store.get_file(parsed_file.output_files.structured_content.file_id)assert structured_record.sha256sum is not Nonestructured_payload = json.loads(file_store.read_blob(structured_record.sha256sum))- assert structured_payload["file_suffix"] == "csv"+ assert structured_payload["metadata"]["file_suffix"] == "csv"assert structured_payload["pages"][0]["blocks"][0]["type"] == "table"diff --git a/tests/unittest/test_flash_docx_equationxml.py b/tests/unittest/test_flash_docx_equationxml.pyindex ed44b3f0..110a2ca6 100644--- a/tests/unittest/test_flash_docx_equationxml.py+++ b/tests/unittest/test_flash_docx_equationxml.py@@ -32,7 +32,7 @@ def test_docx_equationxml_analyze_lifecycle_and_strict_contracts() -> None:assert _equation_contents(pages) == [r"\frac{a}{b}"]assert isinstance(model, ModelJson)assert isinstance(middle, MiddleJson)- assert model.file_suffix == middle.file_suffix == "docx"+ assert model.metadata.file_suffix == middle.metadata.file_suffix == "docx"assert middle.pages[0].blocks[0].type == BlockType.EQUATIONassert middle.pages[0].blocks[0].content == r"\frac{a}{b}"assert async_model == modeldiff --git a/tests/unittest/test_flash_epub.py b/tests/unittest/test_flash_epub.pyindex 019e2cbd..40b47a0b 100644--- a/tests/unittest/test_flash_epub.py+++ b/tests/unittest/test_flash_epub.py@@ -41,8 +41,8 @@ def test_epub_model_analyze_and_renderers_preserve_structured_content() -> None:assert explicit_full_model.pages == model_pagesassert explicit_full_middle.is_full_document is Trueassert middle.model_dump() == async_middle.model_dump()- assert model.file_suffix == middle.file_suffix == "epub"- assert model.extensions["mineru"]["effort"] == middle.extensions["mineru"]["effort"] == "flash"+ assert model.metadata.file_suffix == middle.metadata.file_suffix == "epub"+ assert model.extensions["mineru"]["tier"] == middle.extensions["mineru"]["tier"] == "flash"assert model.extensions["mineru"]["parse_mode"] == middle.extensions["mineru"]["parse_mode"] == "txt"assert [page.page_idx for page in middle.pages] == [0, 1, 2]assert middle.pages[0].blocks[0].type == BlockType.DOC_TITLE@@ -75,7 +75,7 @@ def test_epub_model_analyze_and_renderers_preserve_structured_content() -> None:assert "hidden secret" not in markdownassert "Data table" in markdownassert "<table" in html_output- assert structured["file_suffix"] == "epub"+ assert structured["metadata"]["file_suffix"] == "epub"assert docx.startswith(b"PK")@@ -180,9 +180,9 @@ def test_epub_local_parse_job_emits_spine_aligned_flash_outputs(tmp_path: Path)middle_record = file_store.get_file(parsed_file.output_files.middle_json.file_id) # type: ignore[union-attr]assert middle_record.sha256sum is not Nonepayload = json.loads(file_store.read_blob(middle_record.sha256sum))- assert payload["file_suffix"] == "epub"- assert payload["effort"] == "flash"- assert payload["parse_mode"] == "txt"+ assert payload["metadata"]["file_suffix"] == "epub"+ assert payload["extensions"]["mineru"]["tier"] == "flash"+ assert payload["extensions"]["mineru"]["parse_mode"] == "txt"assert [page["page_idx"] for page in payload["pages"]] == [0, 1, 2]assert payload["pages"][0]["blocks"][0]["type"] == "doc_title"@@ -306,5 +306,5 @@ def test_epub_middle_json_roundtrip_remains_schema_2() -> None:"""验证 EPUB 只扩展 file_suffix,不引入新的 schema 或 Block 字段。"""middle, _ = doc_analyze(build_epub_fixture(), file_suffix="epub")payload = middle.to_dict(skip_defaults=False)- assert payload["file_suffix"] == "epub"- assert json.loads(middle.to_json(skip_defaults=False))["file_suffix"] == "epub"+ assert payload["metadata"]["file_suffix"] == "epub"+ assert json.loads(middle.to_json(skip_defaults=False))["metadata"]["file_suffix"] == "epub"diff --git a/tests/unittest/test_flash_html.py b/tests/unittest/test_flash_html.pyindex 1f6d6d79..2fc49658 100644--- a/tests/unittest/test_flash_html.py+++ b/tests/unittest/test_flash_html.py@@ -63,8 +63,8 @@ def test_html_doc_analyze_projects_static_semantics_and_renderers() -> None:assert middle.model_dump() == async_middle.model_dump()assert model.pages == async_model.pages- assert middle.file_suffix == model.file_suffix == "html"- assert middle.extensions["mineru"]["effort"] == model.extensions["mineru"]["effort"] == "flash"+ assert middle.metadata.file_suffix == model.metadata.file_suffix == "html"+ assert middle.extensions["mineru"]["tier"] == model.extensions["mineru"]["tier"] == "flash"assert middle.extensions["mineru"]["parse_mode"] == model.extensions["mineru"]["parse_mode"] == "txt"assert middle.is_full_document is Trueassert [page.page_idx for page in middle.pages] == [0]@@ -97,7 +97,7 @@ def test_html_doc_analyze_projects_static_semantics_and_renderers() -> None:assert "```python" in markdown and "x^2" in markdownassert "https://cdn.example.com/a.png" in markdownassert "<table" in render_html(middle)- assert render_structured_content(middle)["file_suffix"] == "html"+ assert render_structured_content(middle)["metadata"]["file_suffix"] == "html"docx = render_docx(middle)assert docx.startswith(b"PK")with ZipFile(BytesIO(docx)) as archive:@@ -285,7 +285,7 @@ def test_html_local_base_images_styles_and_escape_are_bounded(tmp_path: Path) ->async_result = asyncio.run(parse_async(source))assert result.middle_json.model_dump() == async_result.middle_json.model_dump()- assert result.middle_json.file_suffix == "html"+ assert result.middle_json.metadata.file_suffix == "html"assert _image_body(result.middle_json).image_base64.startswith("data:image/png;base64,")markdown = result.markdown()assert "hidden css" not in markdown@@ -329,8 +329,8 @@ def test_html_parse_server_local_source_keeps_relative_assets(tmp_path: Path) ->middle_record = file_store.get_file(parsed_file.output_files.middle_json.file_id)assert middle_record.sha256sum is not Nonemiddle_payload = json.loads(file_store.read_blob(middle_record.sha256sum))- assert middle_payload["file_suffix"] == "html"- assert middle_payload["effort"] == "flash"+ assert middle_payload["metadata"]["file_suffix"] == "html"+ assert middle_payload["extensions"]["mineru"]["tier"] == "flash"image_body = middle_payload["pages"][0]["blocks"][1]["content"][0]assert image_body["image_base64"].startswith("data:image/png;base64,")@@ -349,8 +349,8 @@ def test_html_doclib_local_bridge_uses_flash_parser(tmp_path: Path) -> None:))- assert result.middle_json.file_suffix == "html"- assert result.middle_json.extensions["mineru"]["effort"] == "flash"+ assert result.middle_json.metadata.file_suffix == "html"+ assert result.middle_json.extensions["mineru"]["tier"] == "flash"assert "Doclib HTML" in result.markdown()diff --git a/tests/unittest/test_flash_legacy_doc.py b/tests/unittest/test_flash_legacy_doc.pyindex 00bf53ca..8cbb4dd0 100644--- a/tests/unittest/test_flash_legacy_doc.py+++ b/tests/unittest/test_flash_legacy_doc.py@@ -20,8 +20,8 @@ def test_doc_analyze_sync_and_async_return_strict_doc_contract() -> None:assert isinstance(model, ModelJson)assert isinstance(middle, MiddleJson)- assert model.file_suffix == middle.file_suffix == "doc"- assert model.extensions["mineru"]["effort"] == middle.extensions["mineru"]["effort"] == "flash"+ assert model.metadata.file_suffix == middle.metadata.file_suffix == "doc"+ assert model.extensions["mineru"]["tier"] == middle.extensions["mineru"]["tier"] == "flash"assert model.extensions["mineru"]["parse_mode"] == middle.extensions["mineru"]["parse_mode"] == "txt"assert async_model == modelassert async_middle == middle@@ -35,5 +35,5 @@ def test_doc_is_supported_by_public_parser(tmp_path: Path) -> None:result = parse(path, tier="flash")- assert result.middle_json.file_suffix == "doc"+ assert result.middle_json.metadata.file_suffix == "doc"assert inline_text(result.pages[0].blocks[0].content) == "Hello"diff --git a/tests/unittest/test_flash_legacy_ppt.py b/tests/unittest/test_flash_legacy_ppt.pyindex ecadb2c5..2d7660b9 100644--- a/tests/unittest/test_flash_legacy_ppt.py+++ b/tests/unittest/test_flash_legacy_ppt.py@@ -19,9 +19,9 @@ def test_backend_analyze_accepts_ppt_and_async_contract() -> None:assert isinstance(model_json, ModelJson)assert isinstance(middle_json, MiddleJson)- assert model_json.file_suffix == "ppt"- assert middle_json.file_suffix == "ppt"- assert model_json.extensions["mineru"]["effort"] == middle_json.extensions["mineru"]["effort"] == "flash"+ assert model_json.metadata.file_suffix == "ppt"+ assert middle_json.metadata.file_suffix == "ppt"+ assert model_json.extensions["mineru"]["tier"] == middle_json.extensions["mineru"]["tier"] == "flash"assert model_json.extensions["mineru"]["parse_mode"] == middle_json.extensions["mineru"]["parse_mode"] == "txt"assert model_json.is_full_document is middle_json.is_full_document is Trueassert [page.page_idx for page in middle_json.pages] == [0, 1]@@ -37,5 +37,5 @@ def test_ppt_is_supported_by_public_parser(tmp_path: Path) -> None:result = parse(path, tier="flash")- assert result.middle_json.file_suffix == "ppt"+ assert result.middle_json.metadata.file_suffix == "ppt"assert len(result.pages) == 2diff --git a/tests/unittest/test_flash_legacy_xls.py b/tests/unittest/test_flash_legacy_xls.pyindex f61ad11b..ae92c48f 100644--- a/tests/unittest/test_flash_legacy_xls.py+++ b/tests/unittest/test_flash_legacy_xls.py@@ -24,8 +24,8 @@ def test_backend_analyze_accepts_xls_and_async_contract() -> None:assert isinstance(model_json, ModelJson)assert isinstance(middle_json, MiddleJson)- assert model_json.file_suffix == middle_json.file_suffix == "xls"- assert model_json.extensions["mineru"]["effort"] == middle_json.extensions["mineru"]["effort"] == "flash"+ assert model_json.metadata.file_suffix == middle_json.metadata.file_suffix == "xls"+ assert model_json.extensions["mineru"]["tier"] == middle_json.extensions["mineru"]["tier"] == "flash"assert model_json.extensions["mineru"]["parse_mode"] == middle_json.extensions["mineru"]["parse_mode"] == "txt"assert model_json.is_full_document is middle_json.is_full_document is Trueassert async_model == model_json@@ -40,5 +40,5 @@ def test_xls_is_supported_by_public_parser(tmp_path: Path) -> None:result = parse(path, tier="flash")- assert result.middle_json.file_suffix == "xls"+ assert result.middle_json.metadata.file_suffix == "xls"assert inline_text(result.pages[0].blocks[0].content) == "value"diff --git a/tests/unittest/test_flash_legacy_xls_ppt_mtef.py b/tests/unittest/test_flash_legacy_xls_ppt_mtef.pyindex ea44ef9f..b03ad084 100644--- a/tests/unittest/test_flash_legacy_xls_ppt_mtef.py+++ b/tests/unittest/test_flash_legacy_xls_ppt_mtef.py@@ -28,7 +28,7 @@ def test_backend_analyze_preserves_native_equations_sync_and_async(assert isinstance(model, ModelJson)assert isinstance(middle, MiddleJson)- assert model.file_suffix == middle.file_suffix == file_suffix+ assert model.metadata.file_suffix == middle.metadata.file_suffix == file_suffixassert middle.pages[0].blocks[0].type == BlockType.EQUATIONassert middle.pages[0].blocks[0].content == expectedassert async_model == modeldiff --git a/tests/unittest/test_flash_odf.py b/tests/unittest/test_flash_odf.pyindex 59102180..42263dd0 100644--- a/tests/unittest/test_flash_odf.py+++ b/tests/unittest/test_flash_odf.py@@ -49,8 +49,8 @@ def test_odf_models_and_analyze_keep_flash_contract()assert model.pages == async_model.pages == model_pagesassert middle.model_dump() == async_middle.model_dump()- assert model.file_suffix == middle.file_suffix == suffix- assert model.extensions["mineru"]["effort"] == middle.extensions["mineru"]["effort"] == "flash"+ assert model.metadata.file_suffix == middle.metadata.file_suffix == suffix+ assert model.extensions["mineru"]["tier"] == middle.extensions["mineru"]["tier"] == "flash"assert model.extensions["mineru"]["parse_mode"] == middle.extensions["mineru"]["parse_mode"] == "txt"@@ -119,7 +119,7 @@ def test_public_parser_handles_odf_sync_and_async(source.write_bytes(payload)result = parse(source)async_result = asyncio.run(parse_async(source))- assert result.middle_json.file_suffix == async_result.middle_json.file_suffix == suffix+ assert result.middle_json.metadata.file_suffix == async_result.middle_json.metadata.file_suffix == suffixassert result.middle_json.model_dump() == async_result.middle_json.model_dump()@@ -151,9 +151,9 @@ def test_odf_parse_server_job_emits_flash_outputs(tmp_path: Path) -> None:middle_record = file_store.get_file(parsed_file.output_files.middle_json.file_id) # type: ignore[union-attr]assert middle_record.sha256sum is not Nonepayload = json.loads(file_store.read_blob(middle_record.sha256sum))- assert payload["file_suffix"] == "odt"- assert payload["effort"] == "flash"- assert payload["parse_mode"] == "txt"+ assert payload["metadata"]["file_suffix"] == "odt"+ assert payload["extensions"]["mineru"]["tier"] == "flash"+ assert payload["extensions"]["mineru"]["parse_mode"] == "txt"@pytest.mark.parametrize(diff --git a/tests/unittest/test_flash_ofd.py b/tests/unittest/test_flash_ofd.pyindex dde45289..c48f438e 100644--- a/tests/unittest/test_flash_ofd.py+++ b/tests/unittest/test_flash_ofd.py@@ -48,15 +48,15 @@ def test_ofd_model_analyze_detection_and_renderers(tmp_path: Path) -> None:assert model.pages == async_model.pages == model_pagesassert middle.model_dump() == async_middle.model_dump()- assert model.file_suffix == middle.file_suffix == "ofd"- assert model.extensions["mineru"]["effort"] == middle.extensions["mineru"]["effort"] == "flash"+ assert model.metadata.file_suffix == middle.metadata.file_suffix == "ofd"+ assert model.extensions["mineru"]["tier"] == middle.extensions["mineru"]["tier"] == "flash"assert model.extensions["mineru"]["parse_mode"] == middle.extensions["mineru"]["parse_mode"] == "txt"assert middle.is_full_document is Trueassert middle.pages[0].blocks[0].type == BlockType.TEXTassert middle.pages[0].blocks[0].bbox is not Noneassert "你好,OFD" in render_markdown(middle)assert "你好,OFD" in render_html(middle)- assert render_structured_content(middle)["file_suffix"] == "ofd"+ assert render_structured_content(middle)["metadata"]["file_suffix"] == "ofd"assert render_docx(middle).startswith(b"PK")@@ -104,9 +104,9 @@ def test_ofd_parse_server_job_emits_flash_outputs(tmp_path: Path) -> None:middle_record = file_store.get_file(parsed_file.output_files.middle_json.file_id) # type: ignore[union-attr]assert middle_record.sha256sum is not Nonepayload = json.loads(file_store.read_blob(middle_record.sha256sum))- assert payload["file_suffix"] == "ofd"- assert payload["effort"] == "flash"- assert payload["parse_mode"] == "txt"+ assert payload["metadata"]["file_suffix"] == "ofd"+ assert payload["extensions"]["mineru"]["tier"] == "flash"+ assert payload["extensions"]["mineru"]["parse_mode"] == "txt"assert payload["pages"][0]["blocks"][0]["bbox"]diff --git a/tests/unittest/test_flash_office_mtef_v5.py b/tests/unittest/test_flash_office_mtef_v5.pyindex f9782a5b..b70221b8 100644--- a/tests/unittest/test_flash_office_mtef_v5.py+++ b/tests/unittest/test_flash_office_mtef_v5.py@@ -51,7 +51,7 @@ def test_mtef_v5_runs_through_sync_async_analyze(assert isinstance(model, ModelJson)assert isinstance(middle, MiddleJson)- assert model.file_suffix == middle.file_suffix == file_suffix+ assert model.metadata.file_suffix == middle.metadata.file_suffix == file_suffixassert middle.pages[0].blocks[0].type == BlockType.EQUATIONassert middle.pages[0].blocks[0].content == expectedassert async_model == modeldiff --git a/tests/unittest/test_flash_ooxml_mtef.py b/tests/unittest/test_flash_ooxml_mtef.pyindex f6a592a2..09033d8b 100644--- a/tests/unittest/test_flash_ooxml_mtef.py+++ b/tests/unittest/test_flash_ooxml_mtef.py@@ -32,7 +32,7 @@ def test_ooxml_mtef_backend_analyze_sync_async_contract(file_suffix: str) -> Nonassert isinstance(model, ModelJson)assert isinstance(middle, MiddleJson)- assert model.file_suffix == middle.file_suffix == file_suffix+ assert model.metadata.file_suffix == middle.metadata.file_suffix == file_suffixassert middle.pages[0].blocks[0].type == BlockType.EQUATIONassert middle.pages[0].blocks[0].content == expectedassert async_model == modeldiff --git a/tests/unittest/test_flash_rtf.py b/tests/unittest/test_flash_rtf.pyindex c08c3e1e..0981e133 100644--- a/tests/unittest/test_flash_rtf.py+++ b/tests/unittest/test_flash_rtf.py@@ -35,8 +35,8 @@ def test_rtf_doc_analyze_and_renderers_share_strict_metadata() -> None:aio_doc_analyze(_complex_rtf(), effort="medium", parse_mode="auto", file_suffix="rtf"))- assert middle.file_suffix == model.file_suffix == "rtf"- assert middle.extensions["mineru"]["effort"] == model.extensions["mineru"]["effort"] == "flash"+ assert middle.metadata.file_suffix == model.metadata.file_suffix == "rtf"+ assert middle.extensions["mineru"]["tier"] == model.extensions["mineru"]["tier"] == "flash"assert middle.extensions["mineru"]["parse_mode"] == model.extensions["mineru"]["parse_mode"] == "txt"assert middle.is_full_document is model.is_full_document is Trueassert len(middle.pages) == len(model.pages) == 1@@ -51,7 +51,7 @@ def test_rtf_doc_analyze_and_renderers_share_strict_metadata() -> None:assert "Foot body" in markdownassert "<table" in htmlassert docx.startswith(b"PK\x03\x04")- assert structured["file_suffix"] == "rtf"+ assert structured["metadata"]["file_suffix"] == "rtf"def test_public_parser_detects_rtf_content_before_extension(tmp_path: Path) -> None:@@ -65,7 +65,7 @@ def test_public_parser_detects_rtf_content_before_extension(tmp_path: Path) -> Nparse(source, tier="flash", page_range="99")assert exc_info.value.code == "page_range_invalid"- assert result.middle_json.file_suffix == "rtf"+ assert result.middle_json.metadata.file_suffix == "rtf"assert result.middle_json.is_full_document is Trueassert len(result.pages) == 1exported = export_middle_json(result.middle_json, tmp_path / "export")diff --git a/tests/unittest/test_kit_gradio.py b/tests/unittest/test_kit_gradio.pyindex 1eb08a62..8dafbbc8 100644--- a/tests/unittest/test_kit_gradio.py+++ b/tests/unittest/test_kit_gradio.py@@ -106,12 +106,19 @@ def _middle_json(return MiddleJson(pages=pages,is_full_document=page_indices == tuple(range(len(page_indices))),- file_suffix=file_suffix,- producer=Producer(name="mineru", version=__version__),- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version=__version__),+ metadata={"file_suffix": file_suffix, "producer": Producer(name="mineru", version=__version__)},+ extensions=build_metadata(+ effort="flash",+ parse_mode="txt",+ ),)+def _model_json() -> ModelJson:+ """为真实 API/ZIP 测试提供完整分析协议而不是旧任意字典。"""+ return ModelJson(pages=[[{"raw": "model"}]], page_index_map=[], metadata=_middle_json().metadata.model_copy(deep=True))++def _httpx_proxy_for_test_client(test_client: TestClient,request_log: list[tuple[str, str, str | None]],@@ -433,7 +440,7 @@ def test_v1_client_full_asgi_upload_job_poll_and_zip(async def fake_parse_async(path: str, **kwargs: Any) -> ParseResult:"""替代模型推理,同时保留 server 的真实 job 和打包流程。"""parse_calls.append({"path": path, **kwargs})- return ParseResult(middle_json=_middle_json(with_image=False), _model_output={"raw": "model"})+ return ParseResult(middle_json=_middle_json(with_image=False), _model_output=_model_json())monkeypatch.setattr(parser_api_server, "parse_async", fake_parse_async)api = parser_api_server.create_app(@@ -458,7 +465,7 @@ def test_v1_client_full_asgi_upload_job_poll_and_zip(assert any(path.startswith("/v1/files/file-") and path.endswith("/content") for path in paths)protected_requests = [item for item in request_log if item[1] not in {"/v1/health", "/v1/tiers"}]assert protected_requests and all(authorization == "Bearer secret" for _method, _path, authorization in protected_requests)- assert result._model_output == {"raw": "model"}+ assert result._model_output.pages == [[{"raw": "model"}]]assert parse_calls[0]["page_range"] == "1"@@ -471,7 +478,7 @@ def test_gradio_routes_real_v1_jobs_and_isolates_remote_auth(monkeypatch: pytestasync def fake_parse_async(path: str, **kwargs: Any) -> ParseResult:"""只替换模型推理,保留两套服务各自的上传、任务和 ZIP 打包流程。"""- return ParseResult(middle_json=_middle_json(with_image=False), _model_output={"raw": "model"})+ return ParseResult(middle_json=_middle_json(with_image=False), _model_output=_model_json())monkeypatch.setattr(parser_api_server, "parse_async", fake_parse_async)remote_api = parser_api_server.create_app(@@ -493,7 +500,11 @@ def test_gradio_routes_real_v1_jobs_and_isolates_remote_auth(monkeypatch: pytestheaders=dict(request.headers),content=request.content,)- return httpx.Response(response.status_code, headers=response.headers, content=response.content)+ # TestClient 已解压响应,转交 MockTransport 时同步移除原压缩头。+ headers = {+ key: value for key, value in response.headers.items() if key not in {"content-encoding", "content-length"}+ }+ return httpx.Response(response.status_code, headers=headers, content=response.content)def async_client(**kwargs: Any) -> httpx.AsyncClient:"""为能力发现和 API parser 注入同一双服务传输层。"""@@ -515,13 +526,13 @@ def test_gradio_routes_real_v1_jobs_and_isolates_remote_auth(monkeypatch: pytestrequests.clear()flash_result = asyncio.run(routed.parse_file(source, tier="flash", page_range=""))- assert flash_result._model_output == {"raw": "model"}+ assert flash_result._model_output.pages == [[{"raw": "model"}]]assert requests and all(host == "local.test" and auth is None for host, _method, _path, auth in requests)assert any(method == "POST" and path == "/v1/parse/jobs" for _host, method, path, _auth in requests)requests.clear()remote_result = asyncio.run(routed.parse_file(source, tier="standard", page_range="1"))- assert remote_result._model_output == {"raw": "model"}+ assert remote_result._model_output.pages == [[{"raw": "model"}]]assert requests and all(host == "remote.test" and auth == "Bearer remote-secret" for host, _method, _path, auth in requests)@@ -602,10 +613,10 @@ def test_gradio_ocr_reaches_analysis_through_real_v1_jobs(monkeypatch: pytest.Momodel = ModelJson(pages=[[]],page_index_map=[],- file_suffix="pdf",- producer=Producer(name="mineru", version=__version__),+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version=__version__)},extensions=build_metadata(- effort="flash", parse_mode=middle.extensions["mineru"]["parse_mode"], mineru_version=__version__+ effort="flash",+ parse_mode=middle.extensions["mineru"]["parse_mode"],),)return middle, model@@ -629,7 +640,7 @@ def test_gradio_ocr_reaches_analysis_through_real_v1_jobs(monkeypatch: pytest.Mostate = updates[-1][8]assert state is not None, updates[-1][0]payload = json.loads(Path(state["middle_json_path"]).read_text())- assert payload["parse_mode"] == ("ocr" if enabled else "txt")+ assert payload["extensions"]["mineru"]["parse_mode"] == ("ocr" if enabled else "txt")default_parser = parser_api_client.MinerUApiParser(api_url="http://testserver", tier="flash")result = await default_parser.parse_async(source)assert result.middle_json.extensions["mineru"]["parse_mode"] == "txt"@@ -834,7 +845,7 @@ def test_persist_and_render_all_gradio_download_formats(tmp_path: Path) -> None:"""验证严格 Middle JSON、布局 PDF 和六种下载产物。"""source = tmp_path / "报告.pdf"source.write_bytes(_pdf_bytes())- result = ParseResult(middle_json=_middle_json(), _model_output={"raw": "model"})+ result = ParseResult(middle_json=_middle_json(), _model_output=_model_json())artifacts = persist_parse_result(result, source, output_root=tmp_path / "output", page_range="")assert artifacts.middle_json_path.read_text(encoding="utf-8").find('"schema_version": "2.0"') >= 0diff --git a/tests/unittest/test_legacy_schema_adapter.py b/tests/unittest/test_legacy_schema_adapter.pyindex bd7229c6..fc749e96 100644--- a/tests/unittest/test_legacy_schema_adapter.py+++ b/tests/unittest/test_legacy_schema_adapter.py@@ -1,387 +1,32 @@-from docvortex.schema import Producer-from mineru.integrations.docvortex import build_metadata-from copy import deepcopy+"""旧协议仅作为明确拒绝的负例保留,不再执行历史块迁移。"""import pytest--from mineru.backend.postprocess.legacy_schema_adapter import legacy_page_to_model_list-from docvortex.postprocess.pages import model_json_to_pages-from mineru.doclib.background.compaction import _normalize_batch_pages-from mineru.types import BlockType, ModelJson---def _legacy_line(content: str, *, bbox: list[int] | None = None) -> dict:- """构造 3.4.5 普通文字行。"""- return {- "bbox": bbox or [10, 10, 90, 20],- "spans": [{"type": "text", "content": content}],- }---def _current_pages(raw_pages: list[list[dict]]) -> list:- """把适配后的 raw pages 送入当前严格后处理。"""- model_json = ModelJson(- pages=raw_pages,- page_index_map=[],- file_suffix="pdf",- producer=Producer(name="mineru", version="3.4.5"),- extensions=build_metadata(effort="medium", parse_mode="txt", mineru_version="3.4.5"),- )- return model_json_to_pages(model_json)---def test_legacy_adapter_converts_inline_spans_without_mutating_input() -> None:- """验证文字标签、公式、链接及危险链接降级,并保证输入不变。"""- page = {- "page_idx": 0,- "page_size": [100, 100],- "preproc_blocks": [- {- "type": "text",- "bbox": [10, 10, 90, 30],- "lines": [- {- "bbox": [10, 10, 90, 30],- "spans": [- {- "type": "text",- "content": (- 'plain <text style="bold,italic">styled</text> '- "<eq>x^2</eq> "- "<hyperlink><text>safe</text><url>https://example.com</url></hyperlink> "- "<hyperlink><text>unsafe</text><url>javascript:alert(1)</url></hyperlink>"- ),- }- ],- }- ],- }- ],- "discarded_blocks": [],- }- original = deepcopy(page)-- raw_page = legacy_page_to_model_list(page)- content = raw_page[0]["content"]-- assert page == original- assert content == [- {"type": "text", "content": "plain ", "styles": []},- {"type": "text", "content": "styled", "styles": ["bold", "italic"]},- {"type": "text", "content": " ", "styles": []},- {"type": "equation_inline", "content": "x^2"},- {"type": "text", "content": " ", "styles": []},- {- "type": "hyperlink",- "url": "https://example.com",- "content": [{"type": "text", "content": "safe", "styles": []}],- },- {"type": "text", "content": " unsafe", "styles": []},- ]- pages = _current_pages([raw_page])- assert pages[0].blocks[0].type == BlockType.TEXT---def test_legacy_adapter_preserves_visual_metadata_and_algorithm_spans() -> None:- """验证视觉父块展平后能由当前后处理恢复表格与算法块。"""- page = {- "page_idx": 0,- "page_size": [100, 100],- "para_blocks": [- {- "type": "table",- "bbox": [10, 10, 90, 60],- "cell_merge": [0, 1],- "blocks": [- {- "type": "table_caption",- "bbox": [10, 10, 90, 20],- "lines": [_legacy_line("表 1")],- },- {- "type": "table_body",- "bbox": [10, 20, 90, 60],- "lines": [- {- "bbox": [10, 20, 90, 60],- "spans": [- {- "type": "table",- "html": "<table><tr><td>x</td></tr></table>",- "image_path": "images/table.png",- }- ],- }- ],- },- ],- },- {- "type": "code",- "sub_type": "algorithm",- "bbox": [10, 70, 90, 90],- "blocks": [- {- "type": "code_body",- "bbox": [10, 70, 90, 90],- "lines": [- {- "bbox": [10, 70, 90, 90],- "spans": [- {"type": "text", "content": "a="},- {"type": "inline_equation", "content": "x"},- ],- }- ],- }- ],- },- ],- "discarded_blocks": [{"type": "header", "bbox": [10, 1, 90, 5], "lines": [_legacy_line("Header")]}],- }-- pages = _current_pages([legacy_page_to_model_list(page)])- table, code, header = pages[0].blocks-- assert table.type == BlockType.TABLE- assert table.cell_merge == [0, 1]- assert table.content[1].image_path == "images/table.png"- assert code.type == BlockType.CODE- assert code.sub_type == "algorithm"- assert code.content[0].type == BlockType.ALGORITHM_BODY- assert [span.type for span in code.content[0].content] == ["text", "equation_inline"]- assert header.type == BlockType.HEADER---def test_legacy_adapter_normalizes_bbox_and_block_aliases() -> None:- """验证像素坐标、越界坐标及旧 discriminator 均转换为当前合法值。"""- page = {- "page_size": [200, 100],- "preproc_blocks": [- {- "type": "title",- "level": 1,- "bbox": [-10, 0, 220, 20],- "lines": [_legacy_line("Document")],- },- {- "type": "interline_equation",- "bbox": [0, 30, 100, 50],- "lines": [- {- "bbox": [0, 30, 100, 50],- "spans": [{"type": "interline_equation", "content": "y=1"}],- }- ],- },- ],- "discarded_blocks": [],- }-- raw_page = legacy_page_to_model_list(page)-- assert raw_page[0]["type"] == "doc_title"- assert raw_page[0]["bbox"] == (0.0, 0.0, 1.0, 0.2)- assert raw_page[1]["type"] == "equation"- assert raw_page[1]["content"] == "y=1"- _current_pages([raw_page])---def test_legacy_adapter_keeps_only_final_middle_json_aliases() -> None:- """验证最终 3.4.5 仍会序列化的三个旧类型继续转换。"""- page = {- "page_size": [100, 100],- "preproc_blocks": [- {"type": "abstract", "bbox": [0, 0, 100, 20], "lines": [_legacy_line("Abstract")]},- {"type": "phonetic", "bbox": [0, 30, 100, 50], "lines": [_legacy_line("phonetic")]},- {- "type": "interline_equation",- "bbox": [0, 60, 100, 80],- "lines": [- {- "bbox": [0, 60, 100, 80],- "spans": [{"type": "interline_equation", "content": "x=1"}],- }- ],- },- ],- "discarded_blocks": [],- }-- raw_page = legacy_page_to_model_list(page)-- assert [block["type"] for block in raw_page] == ["text", "text", "equation"]- assert raw_page[0]["content"][0]["content"] == "Abstract"- assert raw_page[1]["content"][0]["content"] == "phonetic"- assert raw_page[2]["content"] == "x=1"- _current_pages([raw_page])+from mineru.parser import ParseResult@pytest.mark.parametrize(- "block_type",+ "payload",[- "vertical_text",- "formula_number",- "discarded",- "image_block",- "algorithm",- "algorithm_body",- "algorithm_caption",- "algorithm_footnote",- "header_image",- "footer_image",+ {"pdf_info": [{"page_idx": 0, "para_blocks": []}]},+ {"schema_version": "1.0", "pages": [{"page_idx": 0, "blocks": []}]},+ {+ "schema_version": "2.0",+ "pages": [],+ "file_suffix": "pdf",+ "effort": "flash",+ "parse_mode": "txt",+ "mineru_version": "3.4.5",+ },+ {+ "schema": "docvortex.middle",+ "schema_version": "1.0",+ "pages": [],+ "file_suffix": "html",+ "producer": {"name": "docvortex", "version": "0.1.0"},+ },],)-def test_legacy_adapter_rejects_pre_middle_json_block_types(block_type: str) -> None:- """验证 finalize 前布局、raw model 与 Content List 类型要求源文件重解析。"""- block = {- "type": block_type,- "bbox": [0, 0, 100, 20],- "lines": [_legacy_line("internal")],- }- if block_type in {"image_block", "algorithm"}:- block["blocks"] = [{"type": "text", "bbox": [0, 0, 100, 20], "lines": [_legacy_line("nested")]}]- page = {- "page_size": [100, 100],- "preproc_blocks": [block],- "discarded_blocks": [],- }-- with pytest.raises(ValueError, match=rf"{block_type!s}.*source reparse"):- legacy_page_to_model_list(page)---def test_legacy_adapter_preserves_canonical_image_and_auxiliary_blocks() -> None:- """验证规范 image 视觉块及 header/footer 辅助块不会提升为正文图片。"""- page = {- "page_size": [100, 100],- "preproc_blocks": [- {- "type": "image",- "bbox": [10, 20, 90, 80],- "blocks": [- {- "type": "image_body",- "bbox": [10, 20, 90, 80],- "lines": [- {- "bbox": [10, 20, 90, 80],- "spans": [{"type": "image", "image_path": "images/body.png"}],- }- ],- }- ],- }- ],- "discarded_blocks": [- {"type": "header", "bbox": [0, 0, 100, 10], "lines": [_legacy_line("Header")]},- {"type": "footer", "bbox": [0, 90, 100, 100], "lines": [_legacy_line("Footer")]},- ],- }-- pages = _current_pages([legacy_page_to_model_list(page)])-- assert [block.type for block in pages[0].blocks] == [BlockType.IMAGE, BlockType.HEADER, BlockType.FOOTER]- image = pages[0].blocks[0]- assert image.content[0].image_path == "images/body.png"---def test_legacy_adapter_recovers_pdf_list_items_from_line_flags() -> None:- """验证旧 PDF list 的物理行重新生成当前列表文本子块。"""- page = {- "page_size": [100, 100],- "preproc_blocks": [- {- "type": "list",- "bbox": [10, 10, 90, 40],- "lines": [- {- "bbox": [10, 10, 90, 20],- "is_list_start_line": True,- "spans": [{"type": "text", "content": "one"}],- },- {- "bbox": [10, 25, 90, 35],- "is_list_start_line": True,- "spans": [{"type": "text", "content": "two"}],- },- ],- }- ],- "discarded_blocks": [],- }-- pages = _current_pages([legacy_page_to_model_list(page)])- list_block = pages[0].blocks[0]-- assert list_block.type == BlockType.LIST- assert [child.content[0].content for child in list_block.content] == ["one", "two"]---def test_legacy_adapter_separates_office_blocks_with_synthetic_bbox_groups() -> None:- """验证无 page_size 时列表不会吸收相邻 Office 正文块。"""- page = {- "para_blocks": [- {"type": "text", "lines": [{"spans": [{"type": "text", "content": "before"}]}]},- {- "type": "list",- "blocks": [- {"type": "text", "lines": [{"spans": [{"type": "text", "content": "item"}]}]},- ],- },- {"type": "text", "lines": [{"spans": [{"type": "text", "content": "after"}]}]},- ],- "discarded_blocks": [],- }-- pages = _current_pages([legacy_page_to_model_list(page)])-- assert [block.type for block in pages[0].blocks] == [BlockType.TEXT, BlockType.LIST, BlockType.TEXT]- assert pages[0].blocks[0].content[0].content == "before"- assert pages[0].blocks[1].content[0].content[0].content == "item"- assert pages[0].blocks[2].content[0].content == "after"---def test_doclib_compaction_converts_schema_v1_pages_with_current_adapter() -> None:- """验证 Doclib 历史调用点重新通过宿主适配器生成 2.0 page 字典。"""- payload = {- "schema_version": "1.0",- "pages": [- {- "page_idx": 3,- "page_size": [100, 100],- "preproc_blocks": [- {- "type": "text",- "bbox": [10, 10, 90, 20],- "lines": [_legacy_line("cached")],- }- ],- "discarded_blocks": [],- }- ],- }-- pages = _normalize_batch_pages(payload)-- assert pages[0]["page_idx"] == 3- assert pages[0]["blocks"][0]["content"] == [{"type": "text", "content": "cached"}]---def test_doclib_compaction_rejects_unknown_legacy_schema() -> None:- """验证未知 schema 不会被误当成 3.4.5 页面吞掉。"""- with pytest.raises(ValueError, match="source reparse"):- _normalize_batch_pages({"schema_version": "99.0", "pages": [{"page_idx": 0, "blocks": []}]})---def test_doclib_compaction_accepts_current_schema_without_legacy_conversion(monkeypatch: pytest.MonkeyPatch) -> None:- """验证当前 2.0 缓存直接读取页面,不进入旧版转换。"""- from mineru.backend.postprocess import legacy_schema_adapter-- def unexpected_conversion(_page: dict) -> list[dict]:- """当前缓存触发旧版转换时立即报告路由错误。"""- raise AssertionError("current schema must not use legacy conversion")-- monkeypatch.setattr(legacy_schema_adapter, "legacy_page_to_model_list", unexpected_conversion)- pages = [{"page_idx": 0, "blocks": []}]- assert _normalize_batch_pages({"schema_version": "2.0", "pages": pages}) is pages+def test_old_document_protocols_require_reparse(payload: dict) -> None:+ """拒绝旧 DocVortex、MinerU 及无协议标识的数据,不静默生成空页。"""+ with pytest.raises(ValueError, match="reparse"):+ ParseResult.from_dict(payload)diff --git a/tests/unittest/test_llm_aided_postprocess.py b/tests/unittest/test_llm_aided_postprocess.pyindex 4ddca4ad..0e1ad65f 100644--- a/tests/unittest/test_llm_aided_postprocess.py+++ b/tests/unittest/test_llm_aided_postprocess.py@@ -213,9 +213,11 @@ def _middle_json(pages: list[PageInfo], *, is_full_document: bool = True) -> Midreturn MiddleJson(pages=pages,is_full_document=is_full_document,- file_suffix="docx",- producer=Producer(name="mineru", version="test"),- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="test"),+ metadata={"file_suffix": "docx", "producer": Producer(name="mineru", version="test")},+ extensions=build_metadata(+ effort="flash",+ parse_mode="txt",+ ),)@@ -528,9 +530,11 @@ def test_mixed_cell_merge_flows_through_strict_middle_json_and_renderer() -> Nonmiddle_json = MiddleJson(pages=pages,is_full_document=True,- file_suffix="pdf",- producer=Producer(name="mineru", version="test"),- extensions=build_metadata(effort="high", parse_mode="txt", mineru_version="test"),+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version="test")},+ extensions=build_metadata(+ effort="high",+ parse_mode="txt",+ ),)assert _merged_row_texts(middle_json) == [["H1", "H2"], ["AB", "X"], ["", "Y"]]@@ -762,6 +766,6 @@ def test_doc_analyze_does_not_run_llm_for_office(monkeypatch: pytest.MonkeyPatchmiddle_json, _ = analyze.doc_analyze(b"office", file_suffix="pptx")- assert middle_json.file_suffix == "pptx"+ assert middle_json.metadata.file_suffix == "pptx"assert middle_json.is_full_document is Trueassert calls == []diff --git a/tests/unittest/test_markdown_render.py b/tests/unittest/test_markdown_render.pyindex 8caffd52..ad7b2cc7 100644--- a/tests/unittest/test_markdown_render.py+++ b/tests/unittest/test_markdown_render.py@@ -20,9 +20,11 @@ def _middle(*pages: PageInfo, file_suffix: str = "docx") -> MiddleJson:return MiddleJson(pages=list(pages),is_full_document=True,- file_suffix=file_suffix,- producer=Producer(name="mineru", version="test"),- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="test"),+ metadata={"file_suffix": file_suffix, "producer": Producer(name="mineru", version="test")},+ extensions=build_metadata(+ effort="flash",+ parse_mode="txt",+ ),)diff --git a/tests/unittest/test_model_json.py b/tests/unittest/test_model_json.pyindex 19d0685a..3603e4cc 100644--- a/tests/unittest/test_model_json.py+++ b/tests/unittest/test_model_json.py@@ -12,13 +12,8 @@ from mineru import ModelJson as PublicModelJsonfrom mineru.backend.postprocess import documentfrom mineru.backend.postprocess.document import model_json_to_middle_jsonfrom mineru.config import LLMAidedConfig-from mineru.integrations.docvortex import (- build_metadata,- from_mineru_middle,- from_mineru_model,- to_mineru_middle,- to_mineru_model,-)+from mineru.integrations.docvortex import build_metadata+from mineru.types import MiddleJson, ModelJson@@ -31,9 +26,11 @@ def _model_json(return ModelJson(pages=pages if pages is not None else [[{"type": "text", "content": inline("正文")}]],page_index_map=page_index_map if page_index_map is not None else [],- file_suffix="docx",- producer=Producer(name="mineru", version="3.4.0"),- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="3.4.0"),+ metadata={"file_suffix": "docx", "producer": Producer(name="mineru", version="3.4.0")},+ extensions=build_metadata(+ effort="flash",+ parse_mode="txt",+ ),)@@ -41,24 +38,17 @@ def test_model_json_is_public_and_serializes_exact_envelope() -> None:"""验证公开 ModelJson 固定输出六个顶层字段且空映射不会被省略。"""model_json = _model_json()- payload = to_mineru_model(model_json)+ payload = model_json.to_dict()assert PublicModelJson is ModelJson- assert list(payload) == [- "pages",- "page_index_map",- "file_suffix",- "effort",- "parse_mode",- "mineru_version",- ]+ assert set(payload) == {"schema", "schema_version", "metadata", "extensions", "pages", "page_index_map"}assert payload == {"pages": [[{"type": "text", "content": inline("正文")}]],"page_index_map": [],- "file_suffix": "docx",- "effort": "flash",- "parse_mode": "txt",- "mineru_version": "3.4.0",+ "metadata": {"file_suffix": "docx", "producer": {"name": "mineru", "version": "3.4.0"}},+ "schema": "docvortex.model",+ "schema_version": "2.0",+ "extensions": {"mineru": {"tier": "flash", "parse_mode": "txt"}},}assert model_json.is_full_document is Trueassert model_json.resolved_page_indices == [0]@@ -71,43 +61,16 @@ def test_model_json_requires_page_index_map_and_forbids_extra_fields() -> None:"""验证页映射不可省略且 ModelJson 顶层不接受未声明字段。"""payload = {"pages": [],- "file_suffix": "pdf",- "effort": "flash",- "parse_mode": "txt",- "mineru_version": "3.4.0",+ "metadata": {"file_suffix": "pdf", "producer": {"name": "mineru", "version": "3.4.0"}},+ "schema": "docvortex.model",+ "schema_version": "2.0",+ "extensions": {"mineru": {"tier": "flash", "parse_mode": "txt"}},}with pytest.raises(ValidationError, match="page_index_map"):- from_mineru_model(payload)+ ModelJson.from_dict(payload)with pytest.raises(ValidationError, match="extra_forbidden"):- from_mineru_model({**payload, "page_index_map": [], "unexpected": True})---def test_strict_document_models_reject_removed_low_effort() -> None:- """验证 ModelJson 与 MiddleJson 的 schema 2.0 均不再接受 Low effort。"""- with pytest.raises(ValidationError, match="literal_error"):- from_mineru_model(- {- "pages": [],- "page_index_map": [],- "file_suffix": "pdf",- "effort": "low",- "parse_mode": "ocr",- "mineru_version": "3.4.0",- }- )-- with pytest.raises(ValidationError, match="literal_error"):- from_mineru_middle(- {- "pages": [],- "is_full_document": True,- "file_suffix": "pdf",- "effort": "low",- "parse_mode": "ocr",- "mineru_version": "3.4.0",- }- )+ ModelJson.from_dict({**payload, "page_index_map": [], "unexpected": True})def test_middle_json_requires_and_serializes_full_document_flag() -> None:@@ -115,21 +78,16 @@ def test_middle_json_requires_and_serializes_full_document_flag() -> None:middle_json = MiddleJson(pages=[],is_full_document=False,- file_suffix="pdf",- producer=Producer(name="mineru", version="3.4.0"),- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="3.4.0"),+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version="3.4.0")},+ extensions=build_metadata(+ effort="flash",+ parse_mode="txt",+ ),)- payload = to_mineru_middle(middle_json, include_schema_version=False)+ payload = middle_json.to_dict()- assert list(payload) == [- "pages",- "is_full_document",- "file_suffix",- "effort",- "parse_mode",- "mineru_version",- ]+ assert set(payload) == {"schema", "schema_version", "metadata", "extensions", "pages", "is_full_document"}assert payload["is_full_document"] is Falseassert load_middle(json.loads(middle_json.to_json())) == middle_json@@ -141,9 +99,11 @@ def test_model_json_to_middle_json_builds_strict_document_before_pdf_llm(model_json = ModelJson(pages=[[]],page_index_map=[3],- file_suffix="pdf",- producer=Producer(name="mineru", version="3.4.0"),- extensions=build_metadata(effort="xhigh", parse_mode="ocr", mineru_version="3.4.0"),+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version="3.4.0")},+ extensions=build_metadata(+ effort="xhigh",+ parse_mode="ocr",+ ),)observed: list[MiddleJson] = []@@ -158,10 +118,10 @@ def test_model_json_to_middle_json_builds_strict_document_before_pdf_llm(middle_json = model_json_to_middle_json(model_json, llm_aided_config=LLMAidedConfig())assert observed == [middle_json]- assert middle_json.file_suffix == "pdf"- assert middle_json.extensions["mineru"]["effort"] == "xhigh"+ assert middle_json.metadata.file_suffix == "pdf"+ assert middle_json.extensions["mineru"]["tier"] == "advanced"assert middle_json.extensions["mineru"]["parse_mode"] == "ocr"- assert middle_json.extensions["mineru"]["mineru_version"] == "3.4.0"+ assert middle_json.metadata.producer.version == "3.4.0"@pytest.mark.parametrize("invalid_value", [None, 0, 1, "true"])@@ -169,13 +129,13 @@ def test_middle_json_rejects_missing_or_non_boolean_full_document_flag(invalid_v"""验证 MiddleJson 不为整本语义提供缺省值或宽松布尔转换。"""payload = {"pages": [],- "file_suffix": "pdf",- "effort": "flash",- "parse_mode": "txt",- "mineru_version": "3.4.0",+ "metadata": {"file_suffix": "pdf", "producer": {"name": "mineru", "version": "3.4.0"}},+ "schema": "docvortex.middle",+ "schema_version": "2.0",+ "extensions": {"mineru": {"tier": "flash", "parse_mode": "txt"}},}if invalid_value is not None:payload["is_full_document"] = invalid_valuewith pytest.raises(ValidationError, match="is_full_document"):- from_mineru_middle(payload)+ MiddleJson.from_dict(payload)diff --git a/tests/unittest/test_parse_result_contract.py b/tests/unittest/test_parse_result_contract.pyindex fa0c5b07..3e99f2b2 100644--- a/tests/unittest/test_parse_result_contract.py+++ b/tests/unittest/test_parse_result_contract.py@@ -43,9 +43,11 @@ def test_parse_result_from_dict_restores_pages() -> None:)],is_full_document=True,- file_suffix="pdf",- producer=Producer(name="mineru", version=__version__),- extensions=build_metadata(effort="medium", parse_mode="txt", mineru_version=__version__),+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version=__version__)},+ extensions=build_metadata(+ effort="medium",+ parse_mode="txt",+ ),))@@ -62,9 +64,11 @@ def test_parse_result_to_dict_includes_schema_version_without_meta() -> None:middle_json=MiddleJson(pages=[PageInfo(page_idx=0)],is_full_document=True,- file_suffix="pdf",- producer=Producer(name="mineru", version=__version__),- extensions=build_metadata(effort="medium", parse_mode="txt", mineru_version=__version__),+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version=__version__)},+ extensions=build_metadata(+ effort="medium",+ parse_mode="txt",+ ),))@@ -93,9 +97,11 @@ def test_parse_result_roundtrip_preserves_page_footnote_anchor() -> None:)],is_full_document=True,- file_suffix="epub",- producer=Producer(name="mineru", version=__version__),- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version=__version__),+ metadata={"file_suffix": "epub", "producer": Producer(name="mineru", version=__version__)},+ extensions=build_metadata(+ effort="flash",+ parse_mode="txt",+ ),))@@ -114,21 +120,20 @@ def test_parse_result_rejects_low_effort() -> None:"schema_version": MIDDLE_JSON_SCHEMA_VERSION,"pages": [],"is_full_document": True,- "file_suffix": "pdf",- "effort": "low",- "parse_mode": "ocr",- "mineru_version": __version__,+ "metadata": {"file_suffix": "pdf", "producer": {"name": "mineru", "version": __version__}},+ "schema": "docvortex.middle",+ "extensions": {"mineru": {"tier": "low", "parse_mode": "ocr"}},})def test_parse_result_from_dict_rejects_missing_pages() -> None:with pytest.raises(ValueError, match="pages"):- ParseResult.from_dict({"schema_version": MIDDLE_JSON_SCHEMA_VERSION})+ ParseResult.from_dict({"schema": "docvortex.middle", "schema_version": MIDDLE_JSON_SCHEMA_VERSION})-def test_parse_result_from_json_converts_mineru_3_4_5_middle_json() -> None:- """验证 ParseResult 恢复 3.4.5 pdf_info 调用链并生成严格 2.0。"""+def test_parse_result_from_json_rejects_mineru_3_4_5_middle_json() -> None:+ """验证 ParseResult 的真实 JSON 入口拒绝历史 pdf_info 文档。"""data = {"_backend": "hybrid","_effort": "high",@@ -156,22 +161,14 @@ def test_parse_result_from_json_converts_mineru_3_4_5_middle_json() -> None:],}- restored = ParseResult.from_json(json.dumps(data))+ with pytest.raises(ValueError, match="reparse"):+ ParseResult.from_json(json.dumps(data))- assert restored.pages[0].page_idx == 2- assert restored.pages[0].blocks[0].content[0].content == "round trip"- assert restored.middle_json.is_full_document is False- assert restored.middle_json.extensions["mineru"]["effort"] == "high"- assert restored.middle_json.extensions["mineru"]["parse_mode"] == "ocr"- assert restored.middle_json.extensions["mineru"]["mineru_version"] == "3.4.4"--def test_parse_result_accepts_schema_v1_page_wrapper() -> None:- """验证旧调用方的 1.0 pages envelope 继续委托同一适配器。"""- restored = ParseResult.from_dict({"schema_version": "1.0", "pages": []})-- assert restored.pages == []- assert restored.middle_json.is_full_document is True+def test_parse_result_rejects_schema_v1_page_wrapper() -> None:+ """验证旧 1.0 pages 封装明确拒绝,不生成虚假的空文档。"""+ with pytest.raises(ValueError, match="reparse"):+ ParseResult.from_dict({"schema_version": "1.0", "pages": []})@pytest.mark.parametrize("schema_version", [None, "3.0"])@@ -181,7 +178,7 @@ def test_parse_result_rejects_legacy_schema_versions(schema_version: str | None)if schema_version is not None:payload["schema_version"] = schema_version- with pytest.raises(ValueError, match="Reparse the source document"):+ with pytest.raises(ValueError, match="reparse the source document"):ParseResult.from_dict(payload)@@ -210,9 +207,11 @@ def test_parse_result_export_pages_returns_defensive_copy() -> None:middle_json=MiddleJson(pages=[page],is_full_document=True,- file_suffix="pdf",- producer=Producer(name="mineru", version=__version__),- extensions=build_metadata(effort="medium", parse_mode="txt", mineru_version=__version__),+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version=__version__)},+ extensions=build_metadata(+ effort="medium",+ parse_mode="txt",+ ),),)first_export = result.export_pages()diff --git a/tests/unittest/test_parser_api_contract.py b/tests/unittest/test_parser_api_contract.pyindex cb5e05f9..ab004a1f 100644--- a/tests/unittest/test_parser_api_contract.py+++ b/tests/unittest/test_parser_api_contract.py@@ -42,6 +42,7 @@ from mineru.parser.api_server import ()from mineru.parser.base import ParseResultfrom mineru.types import (+ ModelJson,DEFAULT_QUALITY_TIER_SELECTION_ORDER,DEPLOYMENT_TIERS,DeploymentTier,@@ -84,22 +85,34 @@ def _full_middle_json(*pages: PageInfo) -> MiddleJson:return MiddleJson(pages=list(pages),is_full_document=True,- file_suffix="pdf",- producer=Producer(name="mineru", version=__version__),- extensions=build_metadata(effort="medium", parse_mode="txt", mineru_version=__version__),+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version=__version__)},+ extensions=build_metadata(+ effort="medium",+ parse_mode="txt",+ ),)+def _model_payload() -> dict[str, Any]:+ """构造 ZIP 内的完整分析协议,保留原始模型块用于传输回归。"""+ from mineru.types import ModelJson++ return ModelJson(+ pages=[[{"raw": "model"}]],+ page_index_map=[],+ metadata={"file_suffix": "pdf", "producer": {"name": "mineru", "version": __version__}},+ ).to_dict()++def _current_payload(pages: list[dict[str, Any]], **extra: Any) -> dict[str, Any]:"""构造严格 Middle JSON 2.0 API envelope。"""payload: dict[str, Any] = {"schema_version": MIDDLE_JSON_SCHEMA_VERSION,"pages": pages,"is_full_document": True,- "file_suffix": "pdf",- "effort": "medium",- "parse_mode": "txt",- "mineru_version": __version__,+ "metadata": {"file_suffix": "pdf", "producer": {"name": "mineru", "version": __version__}},+ "schema": "docvortex.middle",+ "extensions": {"mineru": {"tier": "basic", "parse_mode": "txt"}},}payload.update(extra)return payload@@ -756,7 +769,7 @@ def test_api_client_downloads_model_output_from_zip(monkeypatch: pytest.MonkeyPaarchive.writestr("middle_json.json", json.dumps(middle_json, ensure_ascii=False))archive.writestr("model_output.json",- json.dumps([[{"raw": "model"}]], ensure_ascii=False, indent=4),+ json.dumps(_model_payload(), ensure_ascii=False, indent=4),)monkeypatch.setattr(@@ -775,11 +788,11 @@ def test_api_client_downloads_model_output_from_zip(monkeypatch: pytest.MonkeyPaparser,)- assert result._model_output == [[{"raw": "model"}]]+ assert result._model_output.pages == [[{"raw": "model"}]]-def test_api_client_accepts_legacy_official_layout_json(monkeypatch: pytest.MonkeyPatch) -> None:- """官方 API zip 中的 legacy pdf_info layout.json 应经 legacy_schema_adapter 迁移为 3.0。"""+def test_api_client_rejects_legacy_official_layout_json(monkeypatch: pytest.MonkeyPatch) -> None:+ """官方 API ZIP 内的历史 pdf_info 文档必须明确拒绝,不再迁移正文。"""parser = MinerUApiParser(api_url="https://mineru.net/api",tier="standard",@@ -813,24 +826,20 @@ def test_api_client_accepts_legacy_official_layout_json(monkeypatch: pytest.Monkwith zipfile.ZipFile(zip_buffer, "w", zipfile.ZIP_DEFLATED) as archive:archive.writestr("layout.json", json.dumps(middle_json, ensure_ascii=False))archive.writestr("images/chart.png", b"chart-bytes")- archive.writestr("ab3a55b0-2017-4e35-8507-ab8e2c012160_model.json", json.dumps([[{"raw": "model"}]]))+ archive.writestr("ab3a55b0-2017-4e35-8507-ab8e2c012160_model.json", json.dumps(_model_payload()))monkeypatch.setattr(api_client, "_download_bytes", lambda _parser, ref: zip_buffer.getvalue() if ref is zip_ref else b"")- result = _parse_result_from_job(- {- "job_id": "job_1",- "status": "completed",- "files": [{"output_files": {"zip": zip_ref}}],- },- "demo.pdf",- parser,- )-- assert len(result.pages) == 1- assert result.pages[0].blocks[0].type.value == "image"- image_body = result.pages[0].blocks[0].content[0]- assert image_body.image_base64 == "data:image/png;base64," + base64.b64encode(b"chart-bytes").decode("ascii") # type: ignore[union-attr]+ with pytest.raises(api_client._V1APIError, match="reparse"):+ _parse_result_from_job(+ {+ "job_id": "job_1",+ "status": "completed",+ "files": [{"output_files": {"zip": zip_ref}}],+ },+ "demo.pdf",+ parser,+ )def test_api_client_async_downloads_model_output_from_zip(monkeypatch: pytest.MonkeyPatch) -> None:@@ -842,7 +851,7 @@ def test_api_client_async_downloads_model_output_from_zip(monkeypatch: pytest.Moarchive.writestr("middle_json.json", json.dumps(middle_json, ensure_ascii=False))archive.writestr("model_output.json",- json.dumps([[{"raw": "model"}]], ensure_ascii=False, indent=4),+ json.dumps(_model_payload(), ensure_ascii=False, indent=4),)async def fake_download_bytes(_parser: object, ref: dict[str, object]) -> bytes:@@ -862,7 +871,7 @@ def test_api_client_async_downloads_model_output_from_zip(monkeypatch: pytest.Mo))- assert result._model_output == [[{"raw": "model"}]]+ assert result._model_output.pages == [[{"raw": "model"}]]def test_api_client_include_images_downloads_single_zip(monkeypatch: pytest.MonkeyPatch) -> None:@@ -898,7 +907,7 @@ def test_api_client_include_images_downloads_single_zip(monkeypatch: pytest.Monk),)archive.writestr("images/chart.png", b"chart-bytes")- archive.writestr("model_output.json", json.dumps([[{"raw": "model"}]], ensure_ascii=False, indent=4))+ archive.writestr("model_output.json", json.dumps(_model_payload(), ensure_ascii=False, indent=4))download_calls: list[dict[str, object]] = []def fake_download_bytes(_parser: object, ref: dict[str, object]) -> bytes:@@ -926,7 +935,7 @@ def test_api_client_include_images_downloads_single_zip(monkeypatch: pytest.Monkassert result.pages[0].page_idx == 0image_body = result.pages[0].blocks[0].content[0]assert image_body.image_base64 == "data:image/png;base64," + base64.b64encode(b"chart-bytes").decode("ascii") # type: ignore[union-attr]- assert result._model_output == [[{"raw": "model"}]]+ assert result._model_output.pages == [[{"raw": "model"}]]def test_api_client_does_not_read_image_cache_from_zip_unless_requested(monkeypatch: pytest.MonkeyPatch) -> None:@@ -937,7 +946,7 @@ def test_api_client_does_not_read_image_cache_from_zip_unless_requested(monkeypawith zipfile.ZipFile(zip_buffer, "w", zipfile.ZIP_DEFLATED) as archive:archive.writestr("middle_json.json", json.dumps(middle_json, ensure_ascii=False))archive.writestr("images/chart.png", b"chart-bytes")- archive.writestr("model_output.json", json.dumps([[{"raw": "model"}]], ensure_ascii=False, indent=4))+ archive.writestr("model_output.json", json.dumps(_model_payload(), ensure_ascii=False, indent=4))monkeypatch.setattr(api_client, "_download_bytes", lambda _parser, ref: zip_buffer.getvalue() if ref is zip_ref else b"")@@ -954,7 +963,7 @@ def test_api_client_does_not_read_image_cache_from_zip_unless_requested(monkeypaimage_body = result.pages[0].blocks[0].content[0]assert image_body.image_base64 is None # type: ignore[union-attr]assert image_body.image_path == "images/chart.png" # type: ignore[union-attr]- assert result._model_output == [[{"raw": "model"}]]+ assert result._model_output.pages == [[{"raw": "model"}]]def test_api_client_async_include_images_downloads_single_zip(monkeypatch: pytest.MonkeyPatch) -> None:@@ -971,7 +980,7 @@ def test_api_client_async_include_images_downloads_single_zip(monkeypatch: pytes"middle_json.json",json.dumps(_current_payload([{"page_idx": 0, "blocks": []}])),)- archive.writestr("model_output.json", json.dumps([[{"raw": "model"}]], ensure_ascii=False, indent=4))+ archive.writestr("model_output.json", json.dumps(_model_payload(), ensure_ascii=False, indent=4))download_calls: list[dict[str, object]] = []async def fake_download_bytes(_parser: object, ref: dict[str, object]) -> bytes:@@ -998,7 +1007,7 @@ def test_api_client_async_include_images_downloads_single_zip(monkeypatch: pytesassert download_calls == [zip_ref]assert result.pages[0].page_idx == 0- assert result._model_output == [[{"raw": "model"}]]+ assert result._model_output.pages == [[{"raw": "model"}]]def test_api_client_include_images_rejects_unsafe_image_entries(monkeypatch: pytest.MonkeyPatch) -> None:@@ -1023,8 +1032,8 @@ def test_api_client_include_images_rejects_unsafe_image_entries(monkeypatch: pyt)-def test_api_client_accepts_remote_pdf_info_middle_json(monkeypatch: pytest.MonkeyPatch) -> None:- """remote 官方 API 返回 3.4.5 pdf_info payload 时应经 legacy 分支迁移而不是拒绝。"""+def test_api_client_rejects_remote_pdf_info_middle_json(monkeypatch: pytest.MonkeyPatch) -> None:+ """远程返回历史 pdf_info 协议时必须提示重新解析,不猜测其版本。"""parser = MinerUApiParser(api_url="https://mineru.net/api", tier="standard")middle_json = {"_backend": "hybrid",@@ -1034,22 +1043,20 @@ def test_api_client_accepts_remote_pdf_info_middle_json(monkeypatch: pytest.Monkmonkeypatch.setattr(api_client, "_download_json", lambda _parser, _outputs: middle_json)- result = _parse_result_from_job(- {- "job_id": "job_1",- "status": "completed",- "files": [{"output_files": {"middle_json": {"file_id": "file-middle-json", "bytes": 10}}}],- },- "demo.pdf",- parser,- )-- assert len(result.pages) == 1- assert result.pages[0].page_idx == 0- assert result.middle_json.extensions["mineru"]["mineru_version"] == "remote"+ with pytest.raises(api_client._V1APIError, match="reparse"):+ _parse_result_from_job(+ {+ "job_id": "job_1",+ "status": "completed",+ "files": [{"output_files": {"middle_json": {"file_id": "file-middle-json", "bytes": 10}}}],+ },+ "demo.pdf",+ parser,+ )-def test_async_api_client_accepts_remote_pdf_info_middle_json(monkeypatch: pytest.MonkeyPatch) -> None:+def test_async_api_client_rejects_remote_pdf_info_middle_json(monkeypatch: pytest.MonkeyPatch) -> None:+ """异步客户端同样拒绝历史 pdf_info 协议,不把空页当成有效结果。"""parser = MinerUApiParser(api_url="https://mineru.net/api", tier="standard")middle_json = {"_backend": "hybrid",@@ -1061,20 +1068,18 @@ def test_async_api_client_accepts_remote_pdf_info_middle_json(monkeypatch: pytesmonkeypatch.setattr(api_client, "_async_download_json", _download_json)- result = asyncio.run(- api_client._async_parse_result_from_job(- {- "job_id": "job_1",- "status": "completed",- "files": [{"output_files": {"middle_json": {"file_id": "file-middle-json", "bytes": 10}}}],- },- "demo.pdf",- parser,+ with pytest.raises(api_client._V1APIError, match="reparse"):+ asyncio.run(+ api_client._async_parse_result_from_job(+ {+ "job_id": "job_1",+ "status": "completed",+ "files": [{"output_files": {"middle_json": {"file_id": "file-middle-json", "bytes": 10}}}],+ },+ "demo.pdf",+ parser,+ ))- )-- assert len(result.pages) == 1- assert result.pages[0].page_idx == 0def test_api_client_rejects_legacy_json_output_file_key() -> None:@@ -1549,7 +1554,7 @@ def test_api_client_accepts_middle_json_current_pages() -> None:)def test_api_client_rejects_unknown_schema_version(payload: dict[str, object]) -> None:"""验证缺失版本与未知版本 payload 仍要求重新解析;pdf_info 走 legacy 迁移单独覆盖。"""- with pytest.raises((ValueError, api_client._V1APIError), match="Reparse the source document"):+ with pytest.raises((ValueError, api_client._V1APIError), match="reparse the source document"):_pages_from_middle_json(payload)@@ -2050,7 +2055,7 @@ def test_api_server_zip_includes_model_output_when_parse_result_has_it() -> None:parse_result = ParseResult(middle_json=_full_middle_json(PageInfo(page_idx=0)),- _model_output=model_output,+ _model_output=ModelJson.from_dict({**_model_payload(), "pages": [model_output]}),)async def fake_parse_async(*args: object, **kwargs: object) -> ParseResult:@@ -2086,7 +2091,7 @@ def test_api_server_zip_includes_model_output_when_parse_result_has_it(assert "model_output.json" in archive.namelist()model_output_text = archive.read("model_output.json").decode("utf-8")payload = json.loads(model_output_text)- assert payload == model_output+ assert payload == ModelJson.from_dict({**_model_payload(), "pages": [model_output]}).to_dict(skip_defaults=False)if model_output:assert "\n " in model_output_textdiff --git a/tests/unittest/test_parser_api_status.py b/tests/unittest/test_parser_api_status.pyindex e79b3ae1..f2ad75e3 100644--- a/tests/unittest/test_parser_api_status.py+++ b/tests/unittest/test_parser_api_status.py@@ -49,9 +49,11 @@ def test_parse_notifies_submission_and_polls_before_result_building(middle_json=MiddleJson(pages=[],is_full_document=True,- file_suffix="pdf",- producer=Producer(name="mineru", version="4.0.0"),- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="4.0.0"),+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version="4.0.0")},+ extensions=build_metadata(+ effort="flash",+ parse_mode="txt",+ ),))diff --git a/tests/unittest/test_pdf_analyze_block_content.py b/tests/unittest/test_pdf_analyze_block_content.pyindex ac206633..5b72b364 100644--- a/tests/unittest/test_pdf_analyze_block_content.py+++ b/tests/unittest/test_pdf_analyze_block_content.py@@ -213,9 +213,11 @@ def test_doc_analyze_converts_vlm_results_before_downstream_processing(expected_middle_json = MiddleJson(pages=[],is_full_document=True,- file_suffix="pdf",- producer=Producer(name="mineru", version="test"),- extensions=build_metadata(effort=effort, parse_mode=parse_mode, mineru_version="test"),+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version="test")},+ extensions=build_metadata(+ effort=effort,+ parse_mode=parse_mode,+ ),)monkeypatch.setattr(postprocess_document, "model_json_to_middle_json", MagicMock(return_value=expected_middle_json))monkeypatch.setattr(pipeline, "clean_memory", MagicMock())@@ -232,14 +234,20 @@ def test_doc_analyze_converts_vlm_results_before_downstream_processing(assert type(model_json.pages[0][0]) is dictassert model_json.pages == [[{"type": BlockType.PAGE_NUMBER, "bbox": [0.45, 0.9, 0.55, 0.95], "content": inline("1")}]]assert model_json.page_index_map == []- assert model_json.file_suffix == "pdf"- assert model_json.extensions["mineru"]["effort"] == effort+ assert model_json.metadata.file_suffix == "pdf"+ assert (+ model_json.extensions["mineru"]["tier"]+ == {"flash": "flash", "medium": "basic", "high": "standard", "xhigh": "advanced"}[effort]+ )assert model_json.extensions["mineru"]["parse_mode"] == parse_modeassert isinstance(middle_json, MiddleJson)assert middle_json is expected_middle_jsonassert middle_json.pages == []assert middle_json.is_full_document is True- assert middle_json.extensions["mineru"]["effort"] == effort+ assert (+ middle_json.extensions["mineru"]["tier"]+ == {"flash": "flash", "medium": "basic", "high": "standard", "xhigh": "advanced"}[effort]+ )assert middle_json.extensions["mineru"]["parse_mode"] == parse_modeassert type(source_page) is ExtractResultassert type(source_block) is VlmContentBlockdiff --git a/tests/unittest/test_pdf_analyze_visual_blocks.py b/tests/unittest/test_pdf_analyze_visual_blocks.pyindex fcc2fca8..92114f18 100644--- a/tests/unittest/test_pdf_analyze_visual_blocks.py+++ b/tests/unittest/test_pdf_analyze_visual_blocks.py@@ -28,7 +28,7 @@ from mineru.types import RAW_ALGORITHM, RAW_CAPTION, RAW_FOOTNOTE, RAW_FORMULA_Nfrom mineru.types import FILE_SUFFIXES, BlockType, FileSuffix, MiddleJson, ModelJsonfrom mineru.version import __version__ as mineru_version-from _span_test_utils import equation, hyperlink, inline, inline_items, inline_text, inline_urls+from _span_test_utils import inline, inline_textJPEG_DATA_URI_PREFIX = "data:image/jpeg;base64,"@@ -1215,18 +1215,18 @@ def test_doc_analyze_office_returns_model_json_without_pdf_processing(assert isinstance(middle_json, MiddleJson)assert len(middle_json.pages) == 1- assert middle_json.file_suffix == file_suffix- assert middle_json.extensions["mineru"]["effort"] == "flash"+ assert middle_json.metadata.file_suffix == file_suffix+ assert middle_json.extensions["mineru"]["tier"] == "flash"assert middle_json.extensions["mineru"]["parse_mode"] == "txt"assert middle_json.is_full_document is Trueassert isinstance(model_json, ModelJson)assert model_json.pages == source_model_listassert model_json.page_index_map == []assert model_json.is_full_document is True- assert model_json.file_suffix == file_suffix- assert model_json.extensions["mineru"]["effort"] == "flash"+ assert model_json.metadata.file_suffix == file_suffix+ assert model_json.extensions["mineru"]["tier"] == "flash"assert model_json.extensions["mineru"]["parse_mode"] == "txt"- assert model_json.extensions["mineru"]["mineru_version"] == mineru_version+ assert model_json.metadata.producer.version == mineru_versionassert inline_text(model_json.pages[0][0]["content"]) == "原始 \\(office\\) 内容"for suffix, model_factory in model_factories.items():assert model_factory.call_count == (1 if suffix == file_suffix else 0)@@ -1596,9 +1596,9 @@ def test_doc_analyze_office_normalizes_product_route(file_suffix: FileSuffix) ->assert isinstance(middle, MiddleJson) and isinstance(model, ModelJson)assert len(model.pages) == len(middle.pages) == 1assert model.is_full_document and middle.is_full_document- assert model.extensions["mineru"]["effort"] == "flash"+ assert model.extensions["mineru"]["tier"] == "flash"assert model.extensions["mineru"]["parse_mode"] == "txt"- assert model.file_suffix == middle.file_suffix == file_suffix+ assert model.metadata.file_suffix == middle.metadata.file_suffix == file_suffixassert middle.pages[0].page_idx == 0assert "Office routing sample" in middle.to_json()@@ -1628,10 +1628,10 @@ def test_doc_analyze_flash_real_pdf_returns_typed_middle_json() -> None:assert all(block.bbox is not None for block in middle_json.pages[0].blocks)assert all("merge_prev" not in block for page in model_json.pages for block in page)assert model_json.page_index_map == []- assert model_json.file_suffix == "pdf"- assert model_json.extensions["mineru"]["effort"] == "flash"+ assert model_json.metadata.file_suffix == "pdf"+ assert model_json.extensions["mineru"]["tier"] == "flash"assert model_json.extensions["mineru"]["parse_mode"] == "txt"- assert model_json.extensions["mineru"]["mineru_version"] == mineru_version+ assert model_json.metadata.producer.version == mineru_versionassert load_middle(json.loads(middle_json.to_json())) == middle_jsonassert load_model(json.loads(model_json.to_json())) == model_jsondiff --git a/tests/unittest/test_render_api.py b/tests/unittest/test_render_api.pyindex b99f604d..16db3048 100644--- a/tests/unittest/test_render_api.py+++ b/tests/unittest/test_render_api.py@@ -36,9 +36,11 @@ def _middle(*pages: PageInfo) -> MiddleJson:return MiddleJson(pages=list(pages),is_full_document=True,- file_suffix="docx",- producer=Producer(name="mineru", version="test"),- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="test"),+ metadata={"file_suffix": "docx", "producer": Producer(name="mineru", version="test")},+ extensions=build_metadata(+ effort="flash",+ parse_mode="txt",+ ),)diff --git a/tests/unittest/test_structured_content_render.py b/tests/unittest/test_structured_content_render.pyindex 2b7a69d5..346475f5 100644--- a/tests/unittest/test_structured_content_render.py+++ b/tests/unittest/test_structured_content_render.py@@ -30,9 +30,11 @@ def _middle(*pages: PageInfo, file_suffix: str = "docx") -> MiddleJson:return MiddleJson(pages=list(pages),is_full_document=True,- file_suffix=file_suffix,- producer=Producer(name="mineru", version="test"),- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="test"),+ metadata={"file_suffix": file_suffix, "producer": Producer(name="mineru", version="test")},+ extensions=build_metadata(+ effort="flash",+ parse_mode="txt",+ ),)@@ -93,8 +95,8 @@ def test_structured_content_preserves_document_tree_without_merging_or_mutation(result = render_structured_content(middle)assert json.loads(json.dumps(result, ensure_ascii=False)) == result- assert result["file_suffix"] == "docx"- assert result["effort"] == "flash"+ assert result["metadata"]["file_suffix"] == "docx"+ assert result["extensions"]["mineru"]["tier"] == "flash"assert [page["page_idx"] for page in result["pages"]] == [0, 1]assert [block["type"] for block in result["pages"][0]["blocks"]] == ["paragraph_title",diff --git a/tests/unittest/test_text_block_anchor.py b/tests/unittest/test_text_block_anchor.pyindex d81fe41d..1b774f92 100644--- a/tests/unittest/test_text_block_anchor.py+++ b/tests/unittest/test_text_block_anchor.py@@ -36,9 +36,11 @@ def _middle_with_text_anchor() -> MiddleJson:),],is_full_document=True,- file_suffix="docx",- producer=Producer(name="mineru", version="test"),- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="test"),+ metadata={"file_suffix": "docx", "producer": Producer(name="mineru", version="test")},+ extensions=build_metadata(+ effort="flash",+ parse_mode="txt",+ ),)diff --git a/tests/unittest/test_vlm_remote.py b/tests/unittest/test_vlm_remote.pyindex ca3e9e44..dbf33cdc 100644--- a/tests/unittest/test_vlm_remote.py+++ b/tests/unittest/test_vlm_remote.py@@ -280,8 +280,8 @@ def test_python_parse_uses_global_remote_vlm(source = tmp_path / "input.png"Image.new("RGB", (300, 200), "white").save(source)result = asyncio.run(parse_async(source, ocr_mode="ocr")) if async_mode else parse(source, ocr_mode="ocr")- assert result.middle_json.file_suffix == "pdf"- assert result.middle_json.extensions["mineru"]["effort"] == "high"+ assert result.middle_json.metadata.file_suffix == "pdf"+ assert result.middle_json.extensions["mineru"]["tier"] == "standard"assert result.middle_json.pages[0].page_idx == 0assert "Remote VLM text" in result.markdown()assert any(request[2] is not None for request in openai_server.requests)@@ -346,7 +346,7 @@ def test_api_job_through_http_vlm_to_middle_json(file_id = job["files"][0]["output_files"]["middle_json"]["file_id"]output = client.get(f"/v1/files/{file_id}/content")assert output.status_code == 200- assert output.json()["effort"] == ("high" if tier == "standard" else "xhigh")+ assert output.json()["extensions"]["mineru"]["tier"] == tierassert "Remote VLM text" in output.textassert all(request[1] == "Bearer test-key" for request in openai_server.requests)