Skip to content

Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.

Update Middle JSON to schema 2.0 with unified metadata handling and remove legacy support

myhlolica15b938262f
diff --git a/docs/next/api/changes.md b/docs/next/api/changes.md
index 4e5f0432..2de23a5b 100644
--- a/docs/next/api/changes.md
+++ b/docs/next/api/changes.md
@@ -1,3 +1,9 @@
+# 共享文档 JSON 2.0
+
+Model/Middle 输出切换为统一 schema 身份及 metadata 分组,产品信息为实际 tier/parse_mode。
+历史 JSON 不再读取,自建 HTTP 服务与客户端需同步升级;请求参数和输出文件名不变。
+详见 [协议](../middle-json/envelope.md) 和 [迁移](../middle-json/migration.md)。
+
# API 变更记录
状态: Implemented
diff --git a/docs/next/code-map.md b/docs/next/code-map.md
index ec283800..e063046f 100644
--- a/docs/next/code-map.md
+++ b/docs/next/code-map.md
@@ -26,7 +26,7 @@
| `mineru/doclib/` | 本地文档库常驻服务:入库、SHA256 去重、解析调度、缓存、搜索、配置。FastAPI + SQLite(WAL) + FTS5,路由→`services/`→`core/` 三层 | `doclib/server.py`(2439 行);`doclib/services/{parse,search,scan,config,cleanup}_svc.py`;`doclib/core/{db,fts,file_io}.py` |
| `mineru/doclib/background/` | 后台 asyncio 任务:ingest(入库)、parse_worker(解析执行)、scan_worker / watch(目录监听)、parse_server_health(parse-server 托管与健康探测)、device_monitor、compaction(批次合并)、telemetry_flush | 各文件顶部 docstring;`background/parse_worker.py:15` |
| `mineru/parser/` | parse-server 与解析门面:`api_server.py`(2495 行,独立进程)、`api_client.py`(HTTP 客户端)、`tier.py`(tier/backend 解析)、`mineru_parser.py`(`MinerUParser` 统一解析入口)、`base.py`(Middle JSON schema) | `parser/mineru_parser.py`、`parser/tier.py` |
-| `mineru/backend/` | 解析引擎:`analyze.py`(`doc_analyze` 统一门面)→ `analysis/`(按格式分派:`pdf/`、`csv.py`、`epub.py`、`html.py`、`ofd.py`、`office.py`)→ `postprocess/`(Middle JSON 组装、段落/标题/列表/表格合并、`llm_aided.py`、`legacy_schema_adapter.py`) | `backend/analyze.py`、`backend/analysis/pdf/pipeline.py` |
+| `mineru/backend/` | 解析引擎:`analyze.py`(`doc_analyze` 统一门面)→ `analysis/`(按格式分派:`pdf/`、`csv.py`、`epub.py`、`html.py`、`ofd.py`、`office.py`)→ `postprocess/`(Middle JSON 组装、段落/标题/列表/表格合并、`llm_aided.py`) | `backend/analyze.py`、`backend/analysis/pdf/pipeline.py` |
| `mineru/model/` | 模型层:`flash/`(纯 CPU 本地结构化解析器族:pdf / office / epub / html / ofd / csv);`runtime/`(`device.py` 设备与 light/full 栈、`hybrid.py` 原子模型分派、`onnx.py`);`layout/ mfr/ ocr/ table/ vlm/`(ML 模型实现);`_internal/pytorchocr` | `model/runtime/device.py:48`、`model/runtime/hybrid.py` |
| `mineru/render/` | 输出渲染:markdown / content_list(v2) / structured_content / docx / epub / html / pdf | `render/api.py`(唯一渲染入口)、`render/contracts.py` |
| `mineru/kit/` | 服务端与 Web 工具 CLI:`parse` / `gradio` / `api-server` / `vlm-server` / `router` / `models`,面向无状态批处理、自部署和多卡部署 | `kit/main.py`、`kit/commands/*.py`、`kit/gradio/` |
diff --git a/docs/next/invariants.md b/docs/next/invariants.md
index 79b01ce8..268e0abc 100644
--- a/docs/next/invariants.md
+++ b/docs/next/invariants.md
@@ -32,7 +32,7 @@ Middle JSON 的对外 schema 固定为 `2.0`(`mineru/parser/base.py:20` `MIDDL
红线:
-- 不输出 `_backend` / `_version_name` / `pdf_info` / `_ocr_enable` / `_vlm_ocr_enable` 等旧字段;旧字段只能作为受支持旧 payload 的迁移输入出现(legacy 读取路径见 `parser/base.py:20-47` 与 `backend/postprocess/legacy_schema_adapter.py`)。
+- 不接受旧 `_backend` / `_version_name` / `pdf_info` 等历史文档字段;加载统一验证 schema 身份、版本和 metadata,不执行旧协议迁移。
- 改动输出结构即破坏性变更:必须走 ADR 流程,并同步 `docs/next/api/changes.md`、`docs/next/middle-json.md` 与 compaction(`doclib/background/compaction.py:27`)。
## 4. 依赖方向:基础层不得反向引用上层
diff --git a/docs/next/middle-json.md b/docs/next/middle-json.md
index 05c77490..817a0b84 100644
--- a/docs/next/middle-json.md
+++ b/docs/next/middle-json.md
@@ -8,7 +8,7 @@
## 当前定位
-Middle JSON 已经收敛为 schema 2.0 的严格 Pydantic 模型:`mineru/types.py` 中的 `MiddleJson`、`PageInfo`、Block 联合与 InlineSpan 联合。自然语言使用结构化 Span,不再使用字符串标签协议。
+Middle JSON 已经收敛为 schema 2.0 的严格 Pydantic 模型:`docvortex.schema` 定义、`mineru/types.py` 重导出的 `MiddleJson`、`PageInfo`、Block 联合与 InlineSpan 联合。自然语言使用结构化 Span,不再使用字符串标签协议。
## 目录
@@ -27,8 +27,7 @@ Middle JSON 已经收敛为 schema 2.0 的严格 Pydantic 模型:`mineru/types
- 以当前代码中的 Pydantic 严格模型为起点(schema 2.0)。
- 区分“已经解决”、“部分解决”、“仍需工作”。
- Agent 引用与稳定 page/block locator 是 P0 目标。
-- MinerU 3.4.5 页面及对应 1.0 pages 包装经 `legacy_schema_adapter` 转换;其它旧 payload
- 和无法识别的缓存视为 stale,必须从源文件重新解析。
+- 统一使用带 `schema` 身份的 2.0 协议及 `metadata`;所有历史文档协议均明确拒绝,旧缓存须重新解析。
- render 统一不能只看入口 facade,还要收敛格式-specific 分支。
## 与其他文档的关系
diff --git a/docs/next/middle-json/README.md b/docs/next/middle-json/README.md
index 44d10ff3..5a77ccba 100644
--- a/docs/next/middle-json/README.md
+++ b/docs/next/middle-json/README.md
@@ -34,7 +34,7 @@ InlineContentBlock
Middle JSON 下一版要达到以下目标:
-1. 统一顶层 envelope,运行时只接受 schema 2.0 `pages` 结构;旧 payload(3.4.5 `pdf_info`、schema 1.0 `pages` 包装)仅通过 `from_dict()` 的 legacy 分支单向迁移,无版本号的裸 `{"pages": []}` 直接拒绝并要求重新解析。
+1. 统一使用带协议身份和 metadata 的 JSON 2.0;所有历史 JSON 必须重新解析,见 [envelope.md](envelope.md)。
2. 明确 `PageInfo` / `Block` / `InlineSpan` 字段契约。
3. 为每个 backend 提供 normalization 任务清单。
4. 定义 Agent 可引用的稳定 locator 规则。
diff --git a/docs/next/middle-json/current-medium.md b/docs/next/middle-json/current-medium.md
index 8fa2ff16..fb049e3a 100644
--- a/docs/next/middle-json/current-medium.md
+++ b/docs/next/middle-json/current-medium.md
@@ -8,7 +8,7 @@
## 标准来源
-事实标准由 `mineru/types.py` 中的严格 Pydantic v2 模型定义:
+事实标准由 `docvortex.schema` 中的严格 Pydantic v2 模型定义,`mineru/types.py` 重导出:
- `ModelJson`、`MiddleJson`、`PageInfo`;
- `Block` / `PageBlock` discriminated union;
@@ -19,9 +19,8 @@
`tuple[MiddleJson, ModelJson]`,`ParseResult.to_dict()` 输出
`schema_version="2.0"`。
-Schema 2.0 是严格的运行时与公开输出契约(3.0 从未发布)。`ParseResult.from_dict()` 直接读取 2.0,
-并仅将可识别的 MinerU 3.4.5 `pdf_info` 或对应 schema 1.0 `pages` 包装经运行时 legacy 分支单向迁移为 2.0;
-其它缺失版本号的 envelope 与未知旧 payload 必须从源文件重新解析。
+读取必须匹配 `schema` 与 `schema_version`,详见[统一外层协议](envelope.md)。
+所有历史文档协议和缺少身份的 JSON 均拒绝,不提供自动迁移。
## ModelJson
@@ -29,10 +28,9 @@ Schema 2.0 是严格的运行时与公开输出契约(3.0 从未发布)。`P
|------|------|------|
| `pages` | `list[list[dict]]` | Analyze 产生的 raw blocks |
| `page_index_map` | `list[int]` | 空列表表示整本;非空时与 pages 等长、唯一且严格递增 |
-| `file_suffix` | `pdf/doc/docx/ppt/pptx/xls/xlsx/rtf/csv/epub/html/ofd/odt/ods/odp` | 必填 |
-| `effort` | `flash/medium/high/xhigh` | 必填 |
-| `parse_mode` | `txt/ocr` | 必填 |
-| `mineru_version` | 非空字符串 | 必填 |
+| `schema` / `schema_version` | 协议标识及版本 | Model/Middle 各自身份,版本 2.0 |
+| `metadata` | DocumentMetadata | 必填 file_suffix 与 producer |
+| `extensions` | JSON 字典 | 可选产品记录,默认输出空对象 |
文本型 raw block 必须直接携带 InlineSpan 列表。校验失败会报告
`pages[页号][块号]`。PDF 的扁平 `list/index` layout block 在对象化前也可以携带
@@ -44,10 +42,9 @@ Span;形成递归容器后,其 `content` 改为子 block 列表。
|------|------|------|
| `pages` | `list[PageInfo]` | `page_idx` 唯一且严格递增 |
| `is_full_document` | `bool` | 必填,不提供默认值 |
-| `file_suffix` | 与 ModelJson 相同 | 必填 |
-| `effort` | `flash/medium/high/xhigh` | 必填 |
-| `parse_mode` | `txt/ocr` | 必填 |
-| `mineru_version` | 非空字符串 | 必填 |
+| `schema` / `schema_version` | 协议标识及版本 | Model/Middle 各自身份,版本 2.0 |
+| `metadata` | DocumentMetadata | 必填 file_suffix 与 producer |
+| `extensions` | JSON 字典 | 可选产品记录,默认输出空对象 |
`PageInfo` 只包含 `page_idx` 与 `blocks`。顶层 block 必须具有唯一且严格递增的
`index`。PDF 与 OFD 属于固定版式输入,其顶层 block 还必须具有有效的 `[0, 1]`
@@ -151,10 +148,7 @@ Markdown、HTML、DOCX 与 Structured Content renderer 按 Span discriminator
分派。HTML versioned wire 使用明确 DOM metadata 往返恢复类型、样式、公式和
链接,不从渲染后的标签字符串猜测语义。
-## 缓存与兼容
+## 缓存与协议
-- `ParseResult.from_dict()` 直接接受 `schema_version="2.0"`;
-- MinerU 3.4.5 `pdf_info` 与对应 1.0 pages 包装经 `legacy_schema_adapter` 在运行时(`from_dict()` 与 doclib compaction)回推;
-- Doclib compaction 使用同一适配器迁移可识别旧 batch,无法识别时仍按 stale 处理;
-- 无版本号的裸 `{"pages": []}` 与其它未知旧版本返回“重新解析源文件”的明确错误;
-- 不提供 `str | list[InlineSpan]` 联合,也不恢复旧版带 bbox/图片职责的 Line/Span。
+只接受当前完整文档协议;旧结果不命中缓存、不参与压缩,并提示从源文件重建。
+压缩不得合并不同来源、扩展或整本标识的数据。Block/Span 类型和内部语义保持不变。
diff --git a/docs/next/middle-json/envelope.md b/docs/next/middle-json/envelope.md
index 6df574b3..4e771aa2 100644
--- a/docs/next/middle-json/envelope.md
+++ b/docs/next/middle-json/envelope.md
@@ -1,221 +1,58 @@
-# Canonical Envelope
+# Model / Middle JSON 统一外层协议
-状态: Draft
-读者: SDK 开发者、API 开发者、doclib 开发者
-范围: Middle JSON 顶层结构、metadata、版本和兼容输入
-来源: 由根目录旧 Middle JSON 底稿迁移整理而来
+状态:Implemented,DocVortex 0.2.0 起直接切换。
-## 目标结构
-
-当前 P0 写出的 `pages` 结构先包含顶层 `schema_version` 和 `pages`,暂不写 `_meta`:
+两边共用 `docvortex.schema` 定义的类型和 JSON 2.0,MinerU 不再维护独立的文档封装。
```json
{
+ "schema": "docvortex.middle",
"schema_version": "2.0",
+ "metadata": {
+ "file_suffix": "pdf",
+ "producer": {"name": "mineru", "version": "4.0.0b1"}
+ },
+ "extensions": {"mineru": {"tier": "standard", "parse_mode": "txt"}},
+ "is_full_document": true,
"pages": []
}
```
-后续 canonical envelope 目标结构:
-
-```json
-{
- "schema_version": "2.0",
- "pages": [],
- "_meta": {
- "mineru_version": "2.x",
- "backend": "hybrid",
- "tier": "basic",
- "file": {
- "sha256": "...",
- "page_count": 12,
- "filename": null
- },
- "features": {
- "ocr_enabled": true,
- "vlm_ocr_enabled": false
- },
- "models": {}
- }
-}
-```
-
-设计选择:
-
-- 新结构使用 `pages`,对应 `ParseResult.pages`。
-- 运行时直接读取 schema 2.0 `pages` 结构;MinerU 3.4.5 `pdf_info` 与 schema 1.0 `pages` 包装由 `ParseResult.from_dict()` 与 doclib compaction 的运行时 legacy 分支单向迁移,不是离线工具专属;无版本号的裸 `{"pages": []}` 被拒绝并要求重新解析。
-- 当前 `ParseResult.to_dict()` 只输出 `schema_version` 与 MiddleJson 顶层字段,不写顶层 `_backend`/`_meta`。
-- 若后续引入 canonical `_meta`,可由 `_meta.backend` 表达 backend 维度;当前 schema 2.0 没有 `_backend` 字段(是否引入 `_meta` 见 [open-questions.md](../open-questions.md))。
-- `schema_version` 放在顶层,便于快速判断 migration。
-- 代码常量定义为 `mineru.parser.MIDDLE_JSON_SCHEMA_VERSION`,由 normalize、validate、writer 和 exporter 统一引用。
-- 当前 P0 写出路径只增加 `schema_version`,不新增 `_meta`;`_meta` 由后续 canonical envelope migration / writer 引入。
-
-## 字段
-
-| 字段 | 类型 | 必带 | 说明 |
-|------|------|:--:|------|
-| `schema_version` | string | 是 | 当前 `"2.0"`;代码常量为 `MIDDLE_JSON_SCHEMA_VERSION`。 |
-| `pages` | list[PageInfo] | 是 | typed pages 的 JSON 表达。 |
-| `is_full_document` | bool | 是 | 是否整本文档解析(空 `page_index_map` 时为 `true`)。 |
-| `file_suffix` | string | 是 | 输入文件类型(`pdf`、`docx`、`pptx`、`epub`、`html`、`ofd` 等)。 |
-| `effort` | string | 是 | 分析强度:`flash`、`medium`、`high`、`xhigh`。 |
-| `parse_mode` | string | 是 | `txt` 或 `ocr`。 |
-| `mineru_version` | string | 是 | 生成该结果的 MinerU 版本。 |
-| `_meta` | object | 后续 | 元数据;当前 P0 写出路径暂不增加。 |
-
-## `_meta`
-
-| 字段 | 类型 | 必带 | 说明 |
-|------|------|:--:|------|
-| `mineru_version` | string | 是 | 生成该结果的 MinerU 版本。 |
-| `backend` | string | 是 | `hybrid`、`office`、`html`、`flash`。 |
-| `tier` | string 或 null | 是 | `flash`、`basic`、`standard`、`advanced` 的解析结果语义;未经过 tier 解析的工具层结果可为 `null`。 |
-| `file` | object | 是 | 文件级信息。 |
-| `features` | object | 是 | 本次解析启用的能力。 |
-| `models` | object | 是 | 实际模型信息。 |
-| `parsed_at` | string 或 null | 否 | 解析时间。默认不参与 locator。 |
-
-## `_meta.file`
-
-| 字段 | 类型 | 必带 | 说明 |
-|------|------|:--:|------|
-| `sha256` | string | 是 | 原文件 SHA-256。Agent locator 严格校验时需要。 |
-| `page_count` | integer | 是 | `len(pages)`。 |
-| `filename` | string 或 null | 否 | 原文件名。存在隐私争议,默认可为 null。 |
-
-如果缺少 `sha256`,可以生成局部 locator,但不能生成可严格校验的跨文档 block reference。migration 可以允许临时为空,但 Agent citation 功能必须明确降级或报错。
-
-## `_meta.features`
-
-建议字段:
-
-| 字段 | 类型 | 说明 |
-|------|------|------|
-| `ocr_enabled` | bool | 是否启用 OCR。 |
-| `vlm_ocr_enabled` | bool | Hybrid 中 VLM OCR 是否启用。 |
-| `image_analysis` | bool | 是否启用图片分析。 |
-
-features 是开放字典。新增字段不破坏 schema。
-
-## `_meta.models`
-
-建议字段:
-
-| 字段 | 类型 | 说明 |
-|------|------|------|
-| `layout` | string | layout 模型。 |
-| `ocr` | string | OCR 模型。 |
-| `formula` | string | 公式模型。 |
-| `table` | string | 表格模型。 |
-| `vlm` | string | VLM 模型。 |
-
-models 也是开放字典。字段粒度可以随 backend 增加。
-
-## 兼容输入
-
-`ParseResult.from_dict()` 在运行时接受 schema 2.0 envelope 与两类可识别 legacy 输入。目标读入函数可支持以下输入:
-
-### 1. Canonical envelope
-
-```json
-{
- "schema_version": "2.0",
- "pages": [],
- "_meta": {}
-}
-```
-
-直接读取。
-
-### 2. 历史 1.0 `pages` 包装
-
-```json
-{
- "schema_version": "1.0",
- "pages": []
-}
-```
-
-`schema_version` 为 `"1.0"` 且 `pages` 为 page dict 列表时,经 `legacy_schema_adapter` 单向回推为 raw model-list 后重走统一后处理,生成 2.0 结果。
-
-### 3. 历史旧 CLI middle_json(MinerU 3.4.5 `pdf_info`)
-
-```json
-{
- "pdf_info": [],
- "_backend": "hybrid",
- "_version_name": "2.x"
-}
-```
-
-`pdf_info` 是 `ParseResult.from_dict()` 的运行时兼容分支(按 envelope 识别,不要求 `_version_name` 作为唯一判据):经 `legacy_schema_adapter` 单向迁移为 2.0。旧产物中的 `_backend: "pipeline"` 等字段不参与迁移语义,产物版本以文件内版本信息为准。
-
-### 4. 被拒绝的输入
-
-无 `schema_version` 且无 `pdf_info` 的裸 `{"pages": []}`、`schema_version: "3.0"` 与其它未知旧版本都会抛出“重新解析源文件”的明确错误,不做静默迁移。
-
-## Migration 函数
-
-目标 API:
-
-```python
-def normalize_middle_json(
- payload: dict | list,
- *,
- sha256: str | None = None,
- backend: str | None = None,
- tier: str | None = None,
- filename: str | None = None,
-) -> dict: ...
-```
-
-规则:
-
-1. 运行时 payload 必须是 dict,且 `pages` 必须是 list。
-2. 历史 `pdf_info` 与 schema 1.0 `pages` 包装由 `ParseResult.from_dict()` / doclib compaction 的运行时 legacy 分支单向迁移;不可识别的旧 payload 标记为 stale 并要求重新解析,不做静默转换。
-3. 如果没有 sha256,则保留 null,但禁用需要严格校验 source identity 的 citation 能力。
-4. 输出必须是 canonical envelope。
-
-## Validator
-
-当前生产代码不提供独立的 envelope validator API;结构合法性由 strict Pydantic 模型(`MiddleJson` 等,`extra="forbid"` + `strict`,`mineru/types.py:1092`)在解析时强制,页面树语义校验仅保留在单测中作为 test-local helper。
-
-```python
-from mineru.parser import MIDDLE_JSON_SCHEMA_VERSION
-```
-
-目标 envelope API:
-
-```python
-def validate_middle_json(payload: dict) -> list[ValidationIssue]: ...
-```
-
-P0 校验:
+Model 使用 `docvortex.model`、`page_index_map` 及原来的逐页 raw block 列表;
+Middle 保留 `is_full_document`、PageInfo、Block 和 Span。默认写出保留协议头、
+完整 metadata 和空 extensions。JSON Schema 由同一组声明字段生成。
-- 有 `schema_version`。
-- 有 `pages` list。
-- 每个 page 有 `page_idx`。
-- 每个 block 有 `index`、`type`;固定版式(`pdf`/`ofd`)顶层 block 必须有 `bbox`。
-- 自然语言 block 的 `content` 是合法的 `InlineSpan` 列表(`TextSpan` / `EquationInlineSpan` / `CodeInlineSpan` / `HyperlinkSpan`),span 只表达行内语义,不携带 bbox。
-- `page_count == len(pages)`。
+Python 使用 `document.metadata.file_suffix` 和 `document.metadata.producer`,
+旧顶层字段和转发属性全部移除。构造器要求明确提供格式和生产者;解析入口参数不变。
+`ModelJson.from_dict/from_json`、`MiddleJson.from_dict/from_json` 是统一读取入口,
+`ParseResult.from_dict/from_json` 直接委托并校验可选的 MinerU 扩展。
-P1 校验:
+## 来源与产品扩展
-- block index 页内唯一。
-- locator 可生成。
-- bbox 为 schema 约定的 0-1 归一化坐标或 `null`(unknown)。
-- 内部字段不出现在 public output。
+`producer` 只表达真实生成来源。加载、渲染、保存不改写,Model 到 Middle 深拷贝
+来源和扩展。没有 MinerU 扩展的 DocVortex 文档可以直接在 MinerU 渲染。
-## 与 `ParseResult`
+`extensions.mineru` 仅含实际 `tier` 和最终 `parse_mode`;
+`flash/medium/high/xhigh` 映射为 `flash/basic/standard/advanced`。
+仅接受 `txt/ocr`,不存请求值或 `auto`,不重复存 `effort/mineru_version`。
+MinerU 校验已存在的自身扩展;DocVortex 不依赖产品枚举,完整保留其他应用扩展。
-`ParseResult.to_dict()` 当前输出 `{"schema_version": "2.0", "pages": ...}` 等 MiddleJson 顶层字段,不包含顶层 `_backend` 或 PDF 页映射内部 metadata。`ParseResult.from_dict()` 当前要求 payload 为 dict,且只接受 schema 2.0 与可识别 legacy 输入(见上文“兼容输入”)。
+## 序列化和素材
-建议:
+DocumentResult 与 ParseResult 继续存在。MinerU 的 PDF ParseResult 序列化仍省略
+递归 image_base64,非 PDF 保持原有图片表示;源对象不被修改。因此协议统一不代表
+所有结果包装的图片字段完全相同。DocVortex 继续使用 AssetStore 和显式素材外置。
+Structured Content 迁移 metadata/extensions,但不带 Model/Middle 协议标识;
+Content List V1/V2 内容契约不变。
+HTTP 请求、输出文件名和 ZIP 布局保持不变;ZIP 内文档必须使用完整新版协议。
-- `ParseResult.to_dict()` 继续输出 `schema_version + pages`。
-- 新增 `ParseResult.to_envelope(meta=...)` 输出 canonical envelope。
-- `ParseResult.from_dict()` 调用 `normalize_middle_json()`。
+## 直接迁移与缓存
-## 未决问题
+读取联合检查 schema 和 schema_version;不接受旧 DocVortex 1.0、缺少 schema 的
+MinerU 2.0、旧 1.0 pages 或 pdf_info。不猜测、不自动转换,也不提供旧格式写出。
+需要从源文件重新解析。Bundle 使用 docvortex.bundle 2.0 和原有素材摘要清单。
-`filename` 是否默认写入、`parsed_at` 是否默认写入,集中维护在 [开放问题清单](../open-questions.md)。
+Doclib 仅将协议有效且覆盖记录页范围的批次视为可用缓存。旧缓存不计入缓存命中、
+可用页范围和默认读取档位。请求解析时重新生成缺失结果;直接读取旧页返回重新解析提示。
+压缩仅合并 metadata、extensions、is_full_document 一致的新批次,重复页仍取最新值;
+遇到旧格式、损坏数据或冲突时跳过压缩并保留源数据。
diff --git a/docs/next/middle-json/migration.md b/docs/next/middle-json/migration.md
index ea76eebb..50acb075 100644
--- a/docs/next/middle-json/migration.md
+++ b/docs/next/middle-json/migration.md
@@ -1,149 +1,15 @@
-# 迁移计划
+# JSON 2.0 直接迁移
-状态: Draft
-读者: 核心开发者、backend 开发者、SDK 开发者
-范围: Middle JSON 统一工作的阶段、任务、验收和风险
-来源: 由根目录旧 Middle JSON 底稿迁移整理而来
+采用[统一外层协议](envelope.md),不保留旧格式转换入口。
-## 目标
+1. 升级 DocVortex 至 0.2.x,MinerU 要求 `docvortex>=0.2.0,<0.3.0`。
+2. 将 Model/Middle 构造器的 file_suffix、producer 放入 DocumentMetadata;
+ Python 访问同步改为 metadata.file_suffix、metadata.producer。
+3. JSON 使用完整 schema/schema_version/metadata/extensions;两边统一 from_dict/from_json。
+4. 产品记录只输出实际 tier 与 txt/ocr;删除旧 effort、顶层 mineru_version 和协议适配调用。
+5. 历史 JSON、结果包及旧缓存需从源文件重建;加载不会改写旧文件。
+6. 同步升级自建 HTTP 服务与客户端。旧远端协议会明确拒绝,不尝试迁移。
-把当前 typed document model 推进为可验证、可迁移、可供 Agent 引用的 Middle JSON 标准。
-
-## Phase 0: 确认事实标准
-
-任务:
-
-1. 明确 `mineru/types.py` 是当前 Middle JSON 基础类型来源。
-2. 将 `PageInfo` / 严格 Pydantic block tree / `InlineSpan` 字段写入文档。
-3. 标注内部字段和 public 字段。
-
-验收:
-
-- 新增文档能回答“当前标准是什么”。
-- backend 开发者知道新增字段应该加在哪里。
-
-## Phase 1: Envelope 与恢复
-
-任务:
-
-1. 定义 canonical envelope。
-2. 已实现 `ParseResult.from_dict()`。
-3. 已实现 `ParseResult.from_json()`。
-4. 已实现 `ParseResult.to_dict()` 输出 `schema_version + pages`。
-5. 待实现 `normalize_middle_json()`。
-6. 当前运行时兼容的输入(`ParseResult.from_dict()`):
- - `{"schema_version": "2.0", "pages": [...], ...}` 直接读取。
- - `{"schema_version": "1.0", "pages": [...]}` 经 legacy 分支单向迁移。
- - `{"pdf_info": [...], "_backend": ..., ...}`(MinerU 3.4.5 产物)按 envelope 识别后经同一 legacy 分支单向迁移。
-7. 无版本号的裸 `{"pages": [...]}` 与其它未知旧 payload 被明确拒绝,要求重新解析,不做静默迁移。
-
-验收:
-
-- API `middle_json` output 可以恢复为 `ParseResult`。
-- doclib 当前缓存 JSON 可以恢复为 `ParseResult`。
-- 历史旧 CLI `pdf_info` 产物经运行时 legacy 分支单向恢复;无法识别的旧 payload 按 stale 处理并要求重新解析。
-
-## Phase 2: Validator
-
-任务:
-
-1. 生产代码暂不提供独立 validator API;结构合法性由 strict Pydantic 模型(`extra="forbid"` + `strict`)在解析时强制。
-2. 单测中保留 `validate_pages()` 与 `ValidationIssue` 作为 test-local helper。
-3. 单测已覆盖 P0 页面树校验:
- - page_idx
- - block index/type/bbox(固定版式顶层 block 强制 bbox)
- - 自然语言 block 的行内 `content` 为合法 InlineSpan 列表
-4. 待补 envelope-level 校验:
- - schema_version
- - pages list
- - page_count
-5. 增加 fixtures。
-
-验收:
-
-- Pipeline/VLM/Hybrid/Office 至少各有一个 fixture。
-- validator 能区分 error / warning。
-- 当前已知 Office unknown bbox 以 warning 处理,不阻塞。
-
-## Phase 3: Agent Locator
-
-任务:
-
-1. (已完成)定义 locator(`mineru/doclib/locators.py`)。
-2. (已完成)实现 block locator(`locator_for_block()` / `block_ref()` / `block_char_ref()`,doclib server 与 CLI 已消费)。
-3. 待实现 citation record helper。
-4. doclib 已持久化 `short_id` 与 `sha256`;`parses` 表暂无 schema_version 列,仍待补。
-
-验收:
-
-- 同一文件重复解析 locator 稳定。
-- 不同 tier 的 block reference 可区分。
-- unknown bbox 输出 `bbox_known=false`。
-- Agent 可以从 block reference 回查 page/block。
-
-## Phase 4: Backend Normalization
-
-任务:
-
-1. Pipeline: 检查 index 全页稳定性。
-2. VLM: 确认 bbox 为 schema 约定的归一化坐标(0-1)且不含内部 metadata。
-3. Hybrid: 将 features/models 进入 `_meta`。
-4. Office: 明确 unknown bbox(`bbox: null`)语义。
-5. HTML: 修正必填字段和 DOM order index。
-6. 所有 backend 输出通过 validator。
-
-验收:
-
-- 每个 backend 生成 canonical envelope。
-- 每个 backend 可生成 locator。
-- `PageInfo` 不携带 `_backend` 等 backend metadata。
-
-## Phase 5: Rendering 收敛
-
-任务:
-
-1. render facade 接受 envelope 或 pages + meta。
-2. 为 markdown 增加可选 Agent marker。
-3. 明确 Structured Content schema。
-4. 把 Office-specific structured_content 逐步收敛到 type-specific helper。
-
-验收:
-
-- Markdown / Content List / Structured Content 均可从 canonical envelope 生成。
-- renderer 不依赖 `pdf_info`。
-- backend-specific dispatch 数量下降。
-
-## Phase 6: 历史数据迁移
-
-任务:
-
-1. 为 doclib 缓存结果提供 lazy migration。
-2. 评估历史 `pdf_info` CLI 输出是否仍需离线批量迁移命令(单文档恢复已由运行时 legacy 分支覆盖),或仅保留重新生成说明。
-3. 对缺少 sha256 的历史数据给出明确错误或要求调用方提供文件 hash。
-
-验收:
-
-- 老 `*_middle.json` 能被新版 SDK 读取。
-- 无 sha256 的数据不能生成可严格校验的跨文档引用,但仍可 render。
-- 迁移不会改变用户可见 markdown 输出。
-
-## 风险
-
-| 风险 | 缓解 |
-|------|------|
-| 强行要求 Office bbox 导致大量无效框 | 使用 `bbox_known=false`,先承认 unknown。 |
-| 改 envelope 破坏旧 CLI | 运行时 legacy 分支单向迁移可识别旧 payload;不可识别的按 stale 处理并重新解析,不静默降级。 |
-| locator 因 index 不稳定而漂移 | 先做 normalization,再生成 locator。 |
-| render 收敛过大 | 分阶段,先 facade,后 type-specific helper。 |
-| filename 泄露隐私 | `_meta.file.filename` 默认可为空。 |
-
-## 首批可执行任务
-
-1. 实现 `normalize_middle_json()`。
-2. 增加 envelope-level validator。
-3. 修正 HTML 解析必填字段。
-4. 定义 `locator_for_block()`。
-5. 给 Pipeline/VLM/Office 各加一个 fixture。
-6. 复核历史 `pdf_info` 迁移路径(运行时 legacy 分支已覆盖单文档恢复,确认离线批量迁移是否仍有必要)。
-
-完成这些任务后,Middle JSON 就可以支撑 API/SDK 的 `middle_json` output、doclib 缓存恢复和 Agent citation 的第一版闭环。
+验证协议 Schema、跨入口往返、真实生产者与扩展保留、旧协议拒绝、ZIP 和素材,
+以及 Doclib 新旧缓存混存、有效页覆盖、重复页与压缩冲突。冻结迁移前页面、正文、
+几何及渲染语义,不把外层迁移变成解析算法调整。
diff --git a/docs/next/pitfalls.md b/docs/next/pitfalls.md
index c33a4e4e..eca6ad68 100644
--- a/docs/next/pitfalls.md
+++ b/docs/next/pitfalls.md
@@ -16,15 +16,12 @@
- 新代码只走 `tier`;`backend_for_tier`(`parser/tier.py:183`)已保证 basic/standard/advanced 共用同一个 hybrid-engine,仅 effort 不同(`HYBRID_EFFORT_BY_TIER` `parser/tier.py:25`)。
- 不要新增依赖 `backend` 入参的分支;`pipeline`/`vlm-*` 仅作为 legacy 别名存在。
-### 1.2 Middle JSON legacy 读取三件套
+### 1.2 共享文档协议与缓存必须同步
-改 Middle JSON 结构时必须同步检查三处,否则旧缓存文档读取会断:
-
-1. legacy 读取:`mineru/parser/base.py:20-47` —— 识别 3.4.5 `pdf_info` 与 schema 1.0 `pages` 包装(`_legacy_raw_pages`),`MIDDLE_JSON_SCHEMA_VERSION = "2.0"`;
-2. legacy 适配:`mineru/backend/postprocess/legacy_schema_adapter.py` —— 旧 payload 单向回推为 raw model-list;
-3. 批次合并:`mineru/doclib/background/compaction.py:23` `_normalize_batch_pages` 会把 schema 2.0、1.0、3.4.5 批次归一化后**跨 schema 合并**(legacy 批次经 `legacy_schema_adapter` 转换参与),并非只合并 2.0 批次;schema 无法识别时抛 stale-cache 错误并放弃本轮压缩(`compaction.py:37`)。
-
-新增或调整 Middle JSON 字段时,先确认 ADR-0020(schema stability boundary)是否覆盖该字段,再同步上述三处与 `docs/next/middle-json.md`。
+改外层结构时同步 DocVortex schema/readers、MinerU ParseResult 和 Doclib 缓存。
+当前使用 schema 身份 + 版本 2.0 + metadata,旧协议全部拒绝;不能仅按版本号识别。
+缓存命中、覆盖与默认读取档位均需检查新版协议;压缩只合并元数据一致的新批次。
+详见 [envelope.md](middle-json/envelope.md)。
## 2. Alpha 高频迭代,公开 API 未稳定
diff --git a/mineru/backend/analyze.py b/mineru/backend/analyze.py
index 8d3cf74e..d90eb962 100644
--- a/mineru/backend/analyze.py
+++ b/mineru/backend/analyze.py
@@ -2,7 +2,7 @@
"""统一 PDF、EPUB、HTML、OFD、CSV 与 Office/RTF 文档分析的稳定公共门面。"""
from __future__ import annotations
-from docvortex.schema import Producer
+from docvortex.schema import DocumentMetadata, Producer
from ..integrations.docvortex import build_metadata
import asyncio
@@ -81,9 +81,8 @@ def doc_analyze(
model_json = ModelJson(
pages=result.model_list,
page_index_map=page_index_map or [],
- file_suffix=file_suffix,
- producer=Producer(name="mineru", version=mineru_version),
- extensions=build_metadata(effort=result.effort, parse_mode=result.parse_mode, mineru_version=mineru_version),
+ metadata=DocumentMetadata(file_suffix=file_suffix, producer=Producer(name="mineru", version=mineru_version)),
+ extensions=build_metadata(effort=result.effort, parse_mode=result.parse_mode),
)
from .postprocess.document import model_json_to_middle_json
diff --git a/mineru/backend/postprocess/document.py b/mineru/backend/postprocess/document.py
index ebef0e3d..d3ddd14e 100644
--- a/mineru/backend/postprocess/document.py
+++ b/mineru/backend/postprocess/document.py
@@ -11,7 +11,7 @@ from .llm_aided import apply_llm_aided_postprocess
def model_json_to_middle_json(model_json: ModelJson, *, llm_aided_config: LLMAidedConfig) -> MiddleJson:
"""只由宿主显式启用 LLM,DocVortex 的默认后处理完整且独立。"""
middle_json = build_middle_json(model_json)
- if model_json.file_suffix == "pdf":
+ if model_json.metadata.file_suffix == "pdf":
apply_llm_aided_postprocess(middle_json, llm_aided_config)
return middle_json
diff --git a/mineru/backend/postprocess/legacy_schema_adapter.py b/mineru/backend/postprocess/legacy_schema_adapter.py
deleted file mode 100644
index d6150f05..00000000
--- a/mineru/backend/postprocess/legacy_schema_adapter.py
+++ /dev/null
@@ -1,727 +0,0 @@
-# Copyright (c) Opendatalab. All rights reserved.
-"""MinerU 3.4.5 Middle JSON page 到当前 raw ModelJson 的适配器。
-
-3.4.5 的页面仍使用 ``preproc_blocks``/``para_blocks``、``lines`` 和
-``spans``,自然语言行内语义则通过旧字符串标签或旧 span 字段表达。本模块只负责
-把单页回推成当前 ``model_json_to_pages`` 能消费的 raw block 列表;文档 envelope
-及当前 MiddleJson 的元数据由调用方负责。
-"""
-
-from __future__ import annotations
-
-import html
-import math
-import re
-from typing import Any
-
-from docvortex.schema import INLINE_STYLE_ORDER, InlineSpan, parse_inline_spans
-from docvortex.foundation.hyperlink import OFFICE_EXTERNAL_HYPERLINK_SCHEMES, sanitize_hyperlink_target
-from docvortex.content.inline import join_inline_spans
-
-# 这些父块在 3.4.5 中通过 ``blocks`` 保存 body/caption/footnote;当前 raw
-# ModelJson 则要求先展平成同级 block,再由统一后处理重新分组。
-_VISUAL_PARENT_TYPES: frozenset[str] = frozenset(
- {
- "image",
- "table",
- "chart",
- "code",
- }
-)
-
-_INLINE_CONTENT_BLOCK_TYPES: frozenset[str] = frozenset(
- {
- "text",
- "ref_text",
- "doc_title",
- "paragraph_title",
- "header",
- "footer",
- "page_number",
- "aside_text",
- "page_footnote",
- "image_caption",
- "image_footnote",
- "table_caption",
- "table_footnote",
- "chart_caption",
- "chart_footnote",
- "code_caption",
- "code_footnote",
- "caption",
- "footnote",
- "phonetic",
- }
-)
-
-_STRING_CONTENT_BLOCK_TYPES: frozenset[str] = frozenset(
- {
- "equation",
- "image_body",
- "table_body",
- "chart_body",
- "code_body",
- }
-)
-
-_BLOCK_TYPE_ALIASES: dict[str, str] = {
- "abstract": "text",
- "phonetic": "text",
- "interline_equation": "equation",
-}
-
-_SUPPORTED_BLOCK_TYPES: frozenset[str] = frozenset(
- _VISUAL_PARENT_TYPES | _INLINE_CONTENT_BLOCK_TYPES | _STRING_CONTENT_BLOCK_TYPES | {"list", "index"}
-)
-
-_INLINE_START_RE = re.compile(
- r"<(?P<tag>eq|code|text|hyperlink|sup|sub|strong|b|em|i|s|u)(?P<attrs>\s[^<>]*?)?>",
- re.IGNORECASE,
-)
-_STYLE_ATTR_RE = re.compile(r"\bstyle\s*=\s*([\"'])(?P<style>.*?)\1", re.IGNORECASE | re.DOTALL)
-_URL_RE = re.compile(r"<url>(?P<url>.*?)</url>", re.IGNORECASE | re.DOTALL)
-_DIRECT_TAG_STYLES: dict[str, str] = {
- "strong": "bold",
- "b": "bold",
- "em": "italic",
- "i": "italic",
- "s": "strikethrough",
- "u": "underline",
- "sup": "superscript",
- "sub": "subscript",
-}
-
-# 3.4.5 PDF 的异常零面积框需要替换后才能通过当前严格 bbox 校验。
-_PLACEHOLDER_BBOX: tuple[float, float, float, float] = (0.0, 0.0, 0.001, 0.001)
-
-
-def legacy_page_to_model_list(page: dict[str, Any]) -> list[dict[str, Any]]:
- """把 MinerU 3.4.5 的单页 Middle JSON 转换为当前 raw model-list。"""
- if not isinstance(page, dict):
- raise ValueError("legacy Middle JSON page must be a dict")
-
- width, height = _parse_page_size(page.get("page_size"))
- source_blocks = page.get("preproc_blocks")
- if source_blocks is None:
- source_blocks = page.get("para_blocks", [])
- if not isinstance(source_blocks, list):
- raise ValueError("legacy Middle JSON page blocks must be a list")
-
- discarded_blocks = page.get("discarded_blocks", [])
- if not isinstance(discarded_blocks, list):
- raise ValueError("legacy Middle JSON discarded_blocks must be a list")
-
- legacy_blocks = [*source_blocks, *discarded_blocks]
- model_list: list[dict[str, Any]] = []
- for group_index, block in enumerate(legacy_blocks):
- if not isinstance(block, dict):
- raise ValueError("legacy Middle JSON block entries must be dicts")
- group_start = len(model_list)
- _collect_blocks(block, width, height, model_list, inherited=None)
- if width <= 0 or height <= 0:
- _assign_synthetic_group_bbox(model_list[group_start:], group_index, len(legacy_blocks))
- return model_list
-
-
-def _parse_page_size(value: Any) -> tuple[float, float]:
- """读取合法 page_size;缺失、非数值或非正值统一返回零值。"""
- if not isinstance(value, (list, tuple)) or len(value) < 2:
- return 0.0, 0.0
- width, height = value[:2]
- if any(isinstance(item, bool) or not isinstance(item, (int, float)) for item in (width, height)):
- return 0.0, 0.0
- parsed_width, parsed_height = float(width), float(height)
- if not all(math.isfinite(item) and item > 0 for item in (parsed_width, parsed_height)):
- return 0.0, 0.0
- return parsed_width, parsed_height
-
-
-def _assign_synthetic_group_bbox(blocks: list[dict[str, Any]], group_index: int, group_count: int) -> None:
- """为无 page_size 文档分配互不重叠的占位框,并保持原顶层块的子项同组。"""
- safe_count = max(group_count, 1)
- y0 = (group_index + 0.2) / safe_count
- y1 = (group_index + 0.8) / safe_count
- bbox = (0.05, y0, 0.95, y1)
- for block in blocks:
- block["bbox"] = bbox
- lines = block.get("lines")
- if isinstance(lines, list):
- for line in lines:
- if isinstance(line, dict):
- line["bbox"] = list(bbox)
-
-
-def _collect_blocks(
- block: dict[str, Any],
- width: float,
- height: float,
- output: list[dict[str, Any]],
- *,
- inherited: dict[str, Any] | None,
-) -> None:
- """递归展平视觉父块,并把普通叶子转换为当前 raw block。"""
- block_type = _normalize_block_type(block)
- nested = block.get("blocks")
- if block_type in _VISUAL_PARENT_TYPES and isinstance(nested, list) and nested:
- child_context = _visual_child_context(block, inherited)
- for child in nested:
- if not isinstance(child, dict):
- raise ValueError("legacy visual block children must be dicts")
- _collect_blocks(child, width, height, output, inherited=child_context)
- return
-
- if block_type == "list" and isinstance(nested, list) and nested:
- output.append(_convert_leaf_block(block, block_type, width, height, inherited))
- output.extend(_flatten_nested_list_items(nested, width, height))
- return
- if block_type == "index" and isinstance(nested, list) and nested:
- output.append(_flatten_nested_index(block, nested, width, height))
- return
- if block_type == "list" and isinstance(block.get("lines"), list) and block["lines"]:
- output.append(_convert_leaf_block(block, block_type, width, height, inherited))
- output.extend(_convert_pdf_list_items(block, width, height))
- return
-
- raw = _convert_leaf_block(block, block_type, width, height, inherited)
- output.append(raw)
-
-
-def _visual_child_context(
- block: dict[str, Any],
- inherited: dict[str, Any] | None,
-) -> dict[str, Any]:
- """收集视觉父块上需要下沉到 body 的元数据。"""
- context = dict(inherited or {})
- context["bbox"] = block.get("bbox", context.get("bbox"))
- sub_type = block.get("sub_type")
- if not isinstance(sub_type, str) or not sub_type.strip():
- sub_type = None
- if sub_type:
- context["sub_type"] = sub_type
- if isinstance(block.get("guess_lang"), str) and block["guess_lang"].strip():
- context["guess_lang"] = block["guess_lang"].strip()
- if "cell_merge" in block:
- context["cell_merge"] = block.get("cell_merge")
- return context
-
-
-def _flatten_nested_list_items(nested: list[Any], width: float, height: float) -> list[dict[str, Any]]:
- """把 3.4.5 Office 嵌套列表叶子展平为当前 PDF list 可关联的文本块。"""
- items: list[dict[str, Any]] = []
- for child in nested:
- if not isinstance(child, dict):
- raise ValueError("legacy list children must be dicts")
- child_type = _normalize_block_type(child)
- child_nested = child.get("blocks")
- if child_type == "list" and isinstance(child_nested, list):
- items.extend(_flatten_nested_list_items(child_nested, width, height))
- continue
- if child_type not in {"text", "ref_text"}:
- child_type = "text"
- items.append(_convert_leaf_block(child, child_type, width, height, inherited=None))
- return items
-
-
-def _flatten_nested_index(
- block: dict[str, Any],
- nested: list[Any],
- width: float,
- height: float,
-) -> dict[str, Any]:
- """把 3.4.5 Office 嵌套目录折叠为当前 PDF index 使用的换行 Span。"""
- contents: list[list[dict[str, Any]]] = []
-
- def collect(children: list[Any]) -> None:
- """按深度优先顺序收集目录叶子的结构化行内内容。"""
- for child in children:
- if not isinstance(child, dict):
- raise ValueError("legacy index children must be dicts")
- child_nested = child.get("blocks")
- if _normalize_block_type(child) == "index" and isinstance(child_nested, list):
- collect(child_nested)
- continue
- converted = _convert_leaf_block(child, "text", width, height, inherited=None)
- content = converted.get("content")
- if isinstance(content, list) and content:
- contents.append([span for span in content if isinstance(span, dict)])
-
- collect(nested)
- joined: list[dict[str, Any]] = []
- for content_index, content in enumerate(contents):
- if content_index:
- _append_text_span(joined, "\n", ())
- _extend_span_dicts(joined, content)
- return {
- "type": "index",
- "bbox": _normalize_bbox(block.get("bbox"), width, height),
- "content": [span.model_dump(mode="json") for span in parse_inline_spans(joined)],
- }
-
-
-def _convert_pdf_list_items(block: dict[str, Any], width: float, height: float) -> list[dict[str, Any]]:
- """把 3.4.5 PDF list 的物理行恢复为当前 bbox 关联需要的文本项。"""
- raw_lines = [line for line in block.get("lines", []) if isinstance(line, dict)]
- if not raw_lines:
- return []
- has_item_flags = any(line.get("is_list_start_line") or line.get("is_list_end_line") for line in raw_lines)
- line_groups: list[list[dict[str, Any]]] = []
- current_group: list[dict[str, Any]] = []
- for line in raw_lines:
- if current_group and (line.get("is_list_start_line") or not has_item_flags):
- line_groups.append(current_group)
- current_group = []
- current_group.append(line)
- if line.get("is_list_end_line"):
- line_groups.append(current_group)
- current_group = []
- if current_group:
- line_groups.append(current_group)
-
- item_type = "ref_text" if block.get("sub_type") == "ref_text" else "text"
- items: list[dict[str, Any]] = []
- for lines in line_groups:
- synthetic_block = {
- "type": item_type,
- "bbox": _union_line_bboxes(lines, block.get("bbox")),
- "lines": lines,
- }
- items.append(_convert_leaf_block(synthetic_block, item_type, width, height, inherited=None))
- return items
-
-
-def _union_line_bboxes(lines: list[dict[str, Any]], fallback: Any) -> Any:
- """合并一个旧列表项的物理行框,缺失时回退列表父框。"""
- bboxes = [line.get("bbox") for line in lines]
- valid = [
- bbox
- for bbox in bboxes
- if isinstance(bbox, (list, tuple))
- and len(bbox) == 4
- and all(isinstance(item, (int, float)) and not isinstance(item, bool) for item in bbox)
- ]
- if not valid:
- return fallback
- return [
- min(float(bbox[0]) for bbox in valid),
- min(float(bbox[1]) for bbox in valid),
- max(float(bbox[2]) for bbox in valid),
- max(float(bbox[3]) for bbox in valid),
- ]
-
-
-def _convert_leaf_block(
- block: dict[str, Any],
- block_type: str,
- width: float,
- height: float,
- inherited: dict[str, Any] | None,
-) -> dict[str, Any]:
- """把一个 3.4.5 叶子块转换为当前 raw block 字典。"""
- inherited = inherited or {}
- raw_bbox = block.get("bbox", inherited.get("bbox"))
- bbox = _normalize_bbox(raw_bbox, width, height)
- lines = block.get("lines")
- legacy_lines = lines if isinstance(lines, list) else []
- effective_sub_type = _effective_sub_type(block, block_type, inherited)
-
- raw: dict[str, Any] = {"type": block_type, "bbox": bbox}
- if _uses_inline_content(block_type, effective_sub_type):
- raw["content"] = _extract_inline_content(block, legacy_lines, preserve_line_breaks=block_type == "index")
- elif block_type in _STRING_CONTENT_BLOCK_TYPES:
- raw["content"] = _extract_string_content(block, legacy_lines, block_type)
-
- image_fields = _extract_image_fields(block, legacy_lines)
- raw.update(image_fields)
-
- normalized_lines = _normalize_lines(legacy_lines, width, height)
- if normalized_lines:
- raw["lines"] = normalized_lines
-
- _copy_leaf_metadata(raw, block, block_type, effective_sub_type, inherited)
- return raw
-
-
-def _effective_sub_type(block: dict[str, Any], block_type: str, inherited: dict[str, Any]) -> str | None:
- """解析 code/image/chart body 的有效 sub_type。"""
- sub_type = block.get("sub_type", inherited.get("sub_type"))
- if isinstance(sub_type, str) and sub_type.strip():
- return sub_type.strip()
- if block_type == "code_body":
- return "code"
- return None
-
-
-def _uses_inline_content(block_type: str, sub_type: str | None) -> bool:
- """判断当前 raw block 是否必须使用结构化 InlineSpan 列表。"""
- return block_type in _INLINE_CONTENT_BLOCK_TYPES or (block_type == "code_body" and sub_type == "algorithm")
-
-
-def _extract_inline_content(
- block: dict[str, Any],
- lines: list[Any],
- *,
- preserve_line_breaks: bool,
-) -> list[dict[str, Any]]:
- """从旧 lines/spans 或直接 content 构造严格 InlineSpan 列表。"""
- line_contents: list[list[InlineSpan]] = []
- for line in lines:
- if not isinstance(line, dict):
- continue
- spans = line.get("spans")
- if not isinstance(spans, list):
- continue
- converted: list[dict[str, Any]] = []
- for span in spans:
- if not isinstance(span, dict):
- continue
- _append_legacy_span(converted, span)
- parsed = parse_inline_spans(converted)
- if parsed:
- line_contents.append(parsed)
-
- if not line_contents:
- direct_content = block.get("content")
- converted = _convert_direct_inline_content(direct_content)
- return [span.model_dump(mode="json") for span in parse_inline_spans(converted)]
-
- if preserve_line_breaks:
- joined: list[dict[str, Any]] = []
- for line_index, content in enumerate(line_contents):
- if line_index:
- _append_text_span(joined, "\n", ())
- _extend_span_dicts(joined, [span.model_dump(mode="json") for span in content])
- return [span.model_dump(mode="json") for span in parse_inline_spans(joined)]
-
- joined_spans = join_inline_spans(line_contents)
- return [span.model_dump(mode="json") for span in joined_spans]
-
-
-def _convert_direct_inline_content(content: Any) -> list[dict[str, Any]]:
- """转换没有 lines 包装的旧字符串或 span 列表。"""
- if isinstance(content, str):
- return _parse_legacy_markup(content)
- if not isinstance(content, list):
- return []
- output: list[dict[str, Any]] = []
- for item in content:
- if isinstance(item, dict):
- _append_legacy_span(output, item)
- elif isinstance(item, str):
- _extend_span_dicts(output, _parse_legacy_markup(item))
- return output
-
-
-def _append_legacy_span(output: list[dict[str, Any]], span: dict[str, Any]) -> None:
- """把一个 3.4.5 span 追加为当前 InlineSpan 字典。"""
- span_type = str(span.get("type", "text") or "text")
- content = span.get("content")
- if span_type in {"inline_equation", "equation_inline", "interline_equation", "equation"}:
- normalized = str(content or "").strip()
- if normalized:
- output.append({"type": "equation_inline", "content": normalized})
- return
- if span_type in {"code", "code_inline"}:
- if isinstance(content, str) and content:
- output.append({"type": "code_inline", "content": content})
- return
- if span_type == "hyperlink":
- _append_legacy_hyperlink(output, span)
- return
- if not isinstance(content, str) or not content:
- return
-
- styles = _normalize_styles(span.get("styles", span.get("style")))
- if styles:
- _append_text_span(output, content, styles)
- else:
- _extend_span_dicts(output, _parse_legacy_markup(content))
-
-
-def _append_legacy_hyperlink(output: list[dict[str, Any]], span: dict[str, Any]) -> None:
- """转换结构化旧 hyperlink;不安全目标降级为可见子内容。"""
- children: list[dict[str, Any]] = []
- raw_children = span.get("children")
- if isinstance(raw_children, list):
- for child in raw_children:
- if isinstance(child, dict):
- _append_legacy_span(children, child)
- elif isinstance(span.get("content"), str):
- styles = _normalize_styles(span.get("styles", span.get("style")))
- if styles:
- _append_text_span(children, span["content"], styles)
- else:
- _extend_span_dicts(children, _parse_legacy_markup(span["content"]))
-
- safe_children = [child for child in children if child.get("type") != "hyperlink"]
- safe_url = sanitize_hyperlink_target(
- html.unescape(str(span.get("url", ""))),
- allowed_schemes=OFFICE_EXTERNAL_HYPERLINK_SCHEMES,
- allow_relative=True,
- allow_fragment=True,
- )
- if safe_url and safe_children:
- output.append({"type": "hyperlink", "url": safe_url, "content": safe_children})
- else:
- _extend_span_dicts(output, safe_children)
-
-
-def _parse_legacy_markup(content: str) -> list[dict[str, Any]]:
- """按 3.4.5 的私有字符串标签协议解析行内内容。"""
- output: list[dict[str, Any]] = []
- cursor = 0
- while match := _INLINE_START_RE.search(content, cursor):
- _append_text_span(output, content[cursor : match.start()], ())
- tag = match.group("tag").lower()
- closing = _find_matching_close(content, tag, match.end())
- if closing is None:
- _append_text_span(output, match.group(0), ())
- cursor = match.end()
- continue
-
- inner = content[match.end() : closing[0]]
- original = content[match.start() : closing[1]]
- parsed = _parse_legacy_element(tag, match.group("attrs") or "", inner)
- if parsed is None:
- _append_text_span(output, original, ())
- else:
- _extend_span_dicts(output, parsed)
- cursor = closing[1]
- _append_text_span(output, content[cursor:], ())
- return output
-
-
-def _find_matching_close(content: str, tag: str, start: int) -> tuple[int, int] | None:
- """查找旧标签的配对结束位置,并兼容同名嵌套。"""
- token_re = re.compile(rf"<(?P<close>/)?{re.escape(tag)}(?:\s[^<>]*?)?>", re.IGNORECASE)
- depth = 1
- for match in token_re.finditer(content, start):
- depth += -1 if match.group("close") else 1
- if depth == 0:
- return match.start(), match.end()
- return None
-
-
-def _parse_legacy_element(tag: str, attrs: str, inner: str) -> list[dict[str, Any]] | None:
- """把一个完整旧标签转换为当前 InlineSpan 字典。"""
- if tag == "eq":
- latex = html.unescape(inner).strip()
- return [{"type": "equation_inline", "content": latex}] if latex else []
- if tag == "code":
- code = html.unescape(inner)
- return [{"type": "code_inline", "content": code}] if code else []
- if tag == "hyperlink":
- return _parse_markup_hyperlink(inner)
-
- children = _parse_legacy_markup(inner)
- styles = _parse_style_attribute(attrs) if tag == "text" else [_DIRECT_TAG_STYLES[tag]]
- return _apply_styles(children, styles)
-
-
-def _parse_markup_hyperlink(inner: str) -> list[dict[str, Any]] | None:
- """解析旧 hyperlink 标签;危险 URL 只移除链接语义。"""
- match = _URL_RE.search(inner)
- if match is None:
- return None
- label_content = f"{inner[: match.start()]}{inner[match.end() :]}"
- children = [span for span in _parse_legacy_markup(label_content) if span.get("type") != "hyperlink"]
- if not children:
- return []
- safe_url = sanitize_hyperlink_target(
- html.unescape(match.group("url")),
- allowed_schemes=OFFICE_EXTERNAL_HYPERLINK_SCHEMES,
- allow_relative=True,
- allow_fragment=True,
- )
- if safe_url is None:
- return children
- return [{"type": "hyperlink", "url": safe_url, "content": children}]
-
-
-def _parse_style_attribute(attrs: str) -> list[str]:
- """读取旧 ``text style`` 属性并过滤为当前白名单样式。"""
- match = _STYLE_ATTR_RE.search(attrs)
- return _normalize_styles(match.group("style") if match else None)
-
-
-def _normalize_styles(value: Any) -> list[str]:
- """按当前固定顺序去重旧样式,并消解同时上下标的非法组合。"""
- if isinstance(value, str):
- candidates = [item.strip().lower() for item in value.split(",")]
- elif isinstance(value, (list, tuple, set)):
- candidates = [str(item).strip().lower() for item in value]
- else:
- candidates = []
- style_set = {item for item in candidates if item in INLINE_STYLE_ORDER}
- if "superscript" in style_set and "subscript" in style_set:
- style_set.remove("subscript")
- return [style for style in INLINE_STYLE_ORDER if style in style_set]
-
-
-def _apply_styles(spans: list[dict[str, Any]], styles: list[str]) -> list[dict[str, Any]]:
- """把旧包装标签的样式递归应用到文字及链接文字。"""
- if not styles:
- return spans
- output: list[dict[str, Any]] = []
- for span in spans:
- span_type = span.get("type")
- if span_type == "text":
- merged_styles = _normalize_styles([*_normalize_styles(span.get("styles")), *styles])
- _append_text_span(output, str(span.get("content", "")), merged_styles)
- elif span_type == "hyperlink" and isinstance(span.get("content"), list):
- children = _apply_styles([child for child in span["content"] if isinstance(child, dict)], styles)
- output.append({**span, "content": children})
- else:
- output.append(dict(span))
- return output
-
-
-def _append_text_span(output: list[dict[str, Any]], content: str, styles: Any) -> None:
- """追加非空文字,并合并相邻同样式 TextSpan。"""
- if not content:
- return
- normalized_styles = _normalize_styles(styles)
- if output and output[-1].get("type") == "text" and output[-1].get("styles", []) == normalized_styles:
- output[-1]["content"] = f"{output[-1].get('content', '')}{content}"
- return
- span: dict[str, Any] = {"type": "text", "content": content}
- if normalized_styles:
- span["styles"] = normalized_styles
- output.append(span)
-
-
-def _extend_span_dicts(output: list[dict[str, Any]], spans: list[dict[str, Any]]) -> None:
- """追加一组 Span,并规范化序列边界上的相邻文字。"""
- for span in spans:
- if span.get("type") == "text" and isinstance(span.get("content"), str):
- _append_text_span(output, span["content"], span.get("styles", []))
- else:
- output.append(dict(span))
-
-
-def _extract_string_content(block: dict[str, Any], lines: list[Any], block_type: str) -> str:
- """从旧 spans 提取公式、视觉主体、表格 HTML 或代码字符串。"""
- parts: list[str] = []
- for line in lines:
- if not isinstance(line, dict):
- continue
- line_parts: list[str] = []
- spans = line.get("spans")
- if not isinstance(spans, list):
- continue
- for span in spans:
- if not isinstance(span, dict):
- continue
- value = span.get("html") if block_type == "table_body" and span.get("html") is not None else span.get("content")
- normalized = _stringify_visual_content(value)
- if normalized:
- line_parts.append(normalized)
- if line_parts:
- parts.append("".join(line_parts))
-
- if not parts:
- direct = block.get("html") if block_type == "table_body" and block.get("html") is not None else block.get("content")
- return _stringify_visual_content(direct)
- return "\n".join(parts) if block_type == "code_body" else "".join(parts)
-
-
-def _stringify_visual_content(value: Any) -> str:
- """把旧视觉识别列表稳定折叠为当前字符串载荷。"""
- if isinstance(value, str):
- return value
- if isinstance(value, list):
- return "\n".join(str(item) for item in value if str(item).strip())
- return "" if value is None else str(value)
-
-
-def _extract_image_fields(block: dict[str, Any], lines: list[Any]) -> dict[str, str]:
- """从 block 或第一个视觉 span 提升当前支持的图片载荷字段。"""
- fields: dict[str, str] = {}
- for field_name in ("image_path", "image_base64", "image_url"):
- value = block.get(field_name)
- if isinstance(value, str) and value:
- fields[field_name] = value
- if fields:
- return fields
- for line in lines:
- if not isinstance(line, dict):
- continue
- spans = line.get("spans")
- if not isinstance(spans, list):
- continue
- for span in spans:
- if not isinstance(span, dict):
- continue
- for field_name in ("image_path", "image_base64", "image_url"):
- value = span.get(field_name)
- if isinstance(value, str) and value and field_name not in fields:
- fields[field_name] = value
- if fields:
- return fields
- return fields
-
-
-def _copy_leaf_metadata(
- raw: dict[str, Any],
- block: dict[str, Any],
- block_type: str,
- sub_type: str | None,
- inherited: dict[str, Any],
-) -> None:
- """复制当前后处理仍会读取、且不会越过严格模型边界的字段。"""
- if block.get("level") is not None:
- raw["level"] = block["level"]
- if sub_type:
- raw["sub_type"] = sub_type
- guess_lang = block.get("guess_lang", inherited.get("guess_lang"))
- if block_type == "code_body" and isinstance(guess_lang, str) and guess_lang.strip():
- raw["guess_lang"] = guess_lang.strip()
- if block_type == "table_body" and "cell_merge" in inherited:
- raw["cell_merge"] = inherited["cell_merge"]
- if block_type in {"doc_title", "paragraph_title", "page_footnote"}:
- anchor = block.get("anchor")
- if isinstance(anchor, str) and anchor.strip():
- raw["anchor"] = anchor.strip()
-
-
-def _normalize_block_type(block: dict[str, Any]) -> str:
- """把最终 3.4.5 discriminator 映射到当前 raw 类型并拒绝内部标签。"""
- block_type = str(block.get("type", "") or "")
- if block_type == "title":
- return "doc_title" if block.get("level") == 1 else "paragraph_title"
- normalized = _BLOCK_TYPE_ALIASES.get(block_type, block_type)
- if normalized not in _SUPPORTED_BLOCK_TYPES:
- raise ValueError(f"unsupported MinerU 3.4.5 block type {block_type!r}; source reparse required")
- return normalized
-
-
-def _normalize_bbox(value: Any, width: float, height: float) -> tuple[float, float, float, float]:
- """把旧 bbox 规范为当前严格的 0-1 非零面积坐标。"""
- if not isinstance(value, (list, tuple)) or len(value) != 4:
- return _PLACEHOLDER_BBOX
- if any(isinstance(item, bool) or not isinstance(item, (int, float)) for item in value):
- return _PLACEHOLDER_BBOX
- coords = tuple(float(item) for item in value)
- if not all(math.isfinite(item) for item in coords):
- return _PLACEHOLDER_BBOX
- if width > 0 and height > 0:
- coords = (coords[0] / width, coords[1] / height, coords[2] / width, coords[3] / height)
- elif not all(0.0 <= item <= 1.0 for item in coords):
- return _PLACEHOLDER_BBOX
- clamped = tuple(min(max(item, 0.0), 1.0) for item in coords)
- if clamped[2] <= clamped[0] or clamped[3] <= clamped[1]:
- return _PLACEHOLDER_BBOX
- return clamped # type: ignore[return-value]
-
-
-def _normalize_lines(lines: list[Any], width: float, height: float) -> list[dict[str, list[float]]]:
- """保留跨页文本合并需要的行框,并转换到当前归一化坐标。"""
- result: list[dict[str, list[float]]] = []
- for line in lines:
- if not isinstance(line, dict):
- continue
- bbox = _normalize_bbox(line.get("bbox"), width, height)
- result.append({"bbox": list(bbox)})
- return result
-
-
-__all__ = ["legacy_page_to_model_list"]
diff --git a/mineru/doclib/background/compaction.py b/mineru/doclib/background/compaction.py
index c5563ae9..5e272414 100644
--- a/mineru/doclib/background/compaction.py
+++ b/mineru/doclib/background/compaction.py
@@ -1,8 +1,7 @@
"""Compaction — merges overlapping / adjacent done parse batches to keep the parses table lean."""
from __future__ import annotations
-from docvortex.schema import Producer
-from ...integrations.docvortex import build_metadata
+from ...parser.base import ParseResult
import asyncio
import json
@@ -12,7 +11,6 @@ import time
from collections.abc import Sequence
from typing import Any, cast
-from ...parser.base import MIDDLE_JSON_SCHEMA_VERSION
from ...parser.page_range import format_page_range
from ...types import Tier
from ..core.db import DatabaseManager
@@ -24,46 +22,8 @@ logger = logging.getLogger("mineru.compaction")
def _normalize_batch_pages(batch_payload: dict[str, Any]) -> list[dict[str, Any]]:
- """统一读取 2.0 batch,或把 3.4.5/1.0 页面转换为当前 page 字典。"""
- schema_version = batch_payload.get("schema_version")
- raw_pages = batch_payload.get("pages")
- if schema_version == MIDDLE_JSON_SCHEMA_VERSION:
- return raw_pages if isinstance(raw_pages, list) else []
-
- if schema_version == "1.0":
- pass
- elif schema_version is None:
- raw_pages = batch_payload.get("pdf_info")
- else:
- raw_pages = None
- if not isinstance(raw_pages, list) or any(not isinstance(page, dict) for page in raw_pages):
- raise ValueError("stale Middle JSON cache requires source reparse")
-
- from ...backend.postprocess.legacy_schema_adapter import legacy_page_to_model_list
- from docvortex.postprocess.pages import model_json_to_pages
- from ...parser.base import (
- _legacy_effort,
- _legacy_file_suffix,
- _legacy_page_index_map,
- _legacy_parse_mode,
- )
- from ...types import ModelJson
- from ...version import __version__ as current_mineru_version
-
- source_version = batch_payload.get("_version_name", batch_payload.get("mineru_version"))
- mineru_version = (
- source_version.strip() if isinstance(source_version, str) and source_version.strip() else current_mineru_version
- )
- model_json = ModelJson(
- pages=[legacy_page_to_model_list(page) for page in raw_pages],
- page_index_map=_legacy_page_index_map(raw_pages),
- file_suffix=_legacy_file_suffix(batch_payload),
- producer=Producer(name="mineru", version=mineru_version),
- extensions=build_metadata(
- effort=_legacy_effort(batch_payload), parse_mode=_legacy_parse_mode(batch_payload), mineru_version=mineru_version
- ),
- )
- return [page.to_dict() for page in model_json_to_pages(model_json)]
+ """仅校验并读取新版批次,拒绝猜测或改写旧协议数据。"""
+ return [page.to_dict() for page in ParseResult.from_dict(batch_payload).pages]
class Compaction:
@@ -199,12 +159,13 @@ class Compaction:
if type(page_idx) is not int or page_idx < 0:
return None
pages_by_page_idx[page_idx] = page
+ current_envelope = {
+ key: value for key, value in ParseResult.from_dict(batch_payload).to_dict().items() if key != "pages"
+ }
if not envelope:
- envelope = {
- key: batch_payload[key]
- for key in ("is_full_document", "file_suffix", "effort", "parse_mode", "mineru_version")
- if key in batch_payload
- }
+ envelope = current_envelope
+ elif json.dumps(envelope, sort_keys=True) != json.dumps(current_envelope, sort_keys=True):
+ return None
return (pages_by_page_idx, envelope) if pages_by_page_idx else None
def _write_compacted_json_files(
@@ -228,8 +189,7 @@ class Compaction:
raise ValueError(f"Compacted page range has no source pages: {page_range}")
final_path = parse_batch_json_path(self.data_dir, sha256, tier, page_range, max_done_at)
temp_path = f"{final_path}.tmp-{time.time_ns()}"
- payload: dict[str, Any] = {"schema_version": MIDDLE_JSON_SCHEMA_VERSION, "pages": json_pages}
- payload.update(envelope)
+ payload = ParseResult.from_dict({**envelope, "pages": json_pages}).to_dict()
with open(temp_path, "w", encoding="utf-8") as f:
json.dump(payload, f, ensure_ascii=False, indent=2)
prepared.append((temp_path, final_path))
diff --git a/mineru/doclib/server.py b/mineru/doclib/server.py
index 706eb860..f9b0aee8 100644
--- a/mineru/doclib/server.py
+++ b/mineru/doclib/server.py
@@ -75,6 +75,7 @@ from .rows import (
WatchTargetRow,
)
from .services.parse_svc import (
+ _json_batch_is_current,
accessible_file_for_sha256,
filter_pages_by_user_range,
load_pages_from_done_batches,
@@ -418,7 +419,13 @@ class DoclibServer(AsyncDoclibInterface):
if doc is None:
raise NotFoundError("doc_not_found", f"Document {resolved_sha256} not found.", "doc_ref")
page_range = _expand_page_range(page_range, doc["page_count"] or 1)
- coverage = _parse_coverage(page_range, rows) if resolved_sha256 and tier and page_range else None
+ coverage_rows = [
+ row
+ for row in rows
+ if row["status"] != PARSE_STATUS_DONE
+ or _json_batch_is_current(_effective_data_dir(self.state), row["sha256"], row["tier"], row)
+ ]
+ coverage = _parse_coverage(page_range, coverage_rows) if resolved_sha256 and tier and page_range else None
return ListParsesResponse(
parses=[_parse_info(row, coverage=coverage) for row in rows],
coverage=coverage,
@@ -717,7 +724,11 @@ class DoclibServer(AsyncDoclibInterface):
tier = cursor.tier or await self._default_read_tier(doc["sha256"])
if tier is None:
- raise NotFoundError("tier_not_cached", f"No parsed tier is cached for document {cursor.short_id}.", "locator")
+ raise NotFoundError(
+ "tier_not_cached",
+ f"No current parsed tier is cached for document {cursor.short_id}; reparse the source document.",
+ "locator",
+ )
if format == "image" and context:
raise InvalidRequestError("context_not_applicable", "context is not supported for image reads.", "context")
if cursor.page_no is None and context:
@@ -867,7 +878,12 @@ class DoclibServer(AsyncDoclibInterface):
if sha256
else []
)
- parsed_tiers = [_tier_parse_info(row) for row in parse_rows if row["status"] == PARSE_STATUS_DONE]
+ parsed_tiers = [
+ _tier_parse_info(row)
+ for row in parse_rows
+ if row["status"] == PARSE_STATUS_DONE
+ and _json_batch_is_current(_effective_data_dir(self.state), row["sha256"], row["tier"], row)
+ ]
active_parses = [
_parse_info(row) for row in parse_rows if row["status"] in {PARSE_STATUS_PENDING, PARSE_STATUS_PARSING}
]
@@ -1221,16 +1237,22 @@ class DoclibServer(AsyncDoclibInterface):
"SELECT page_range, done_at FROM parses WHERE sha256=? AND tier=? AND status=? ORDER BY done_at DESC",
(sha256, tier, PARSE_STATUS_DONE),
)
- loaded_pages = load_pages_from_done_batches(data_dir, sha256, tier, cast(list[ParseBatchRow], rows))
doc = cast(DocRow | None, await self.state.db.fetchone("SELECT * FROM docs WHERE sha256=?", (sha256,)))
if doc is None:
raise NotFoundError("doc_not_found", f"Document {sha256} not found.", "doc_ref")
page_range = normalize_page_range_input(page_range)
if page_range and doc["page_count"] is not None:
- loaded_pages = filter_pages_by_user_range(loaded_pages, _expand_page_range(page_range, doc["page_count"]))
- elif page_range and page_range != "all":
- # 总页数未知时只允许绝对页码,不能用已缓存页数推断文档末页。
- loaded_pages = filter_pages_by_user_range(loaded_pages, page_range)
+ page_range = _expand_page_range(page_range, doc["page_count"])
+ requested = parse_page_range_set(page_range) if page_range and page_range != "all" else None
+ loaded_pages = load_pages_from_done_batches(
+ data_dir,
+ sha256,
+ tier,
+ cast(list[ParseBatchRow], rows),
+ requested_page_numbers=requested,
+ )
+ if requested is not None:
+ loaded_pages = [page for page in loaded_pages if page.page_idx + 1 in requested]
if not loaded_pages:
raise NotFoundError("not_cached", "Requested parsed content is not cached.", "page_range")
output: list[str] = []
@@ -1254,7 +1276,13 @@ class DoclibServer(AsyncDoclibInterface):
"SELECT page_range, done_at FROM parses WHERE sha256=? AND tier=? AND status=? ORDER BY done_at DESC",
(plan.sha256, plan.tier, PARSE_STATUS_DONE),
)
- loaded_pages = load_pages_from_done_batches(data_dir, plan.sha256, plan.tier, cast(list[ParseBatchRow], rows))
+ loaded_pages = load_pages_from_done_batches(
+ data_dir,
+ plan.sha256,
+ plan.tier,
+ cast(list[ParseBatchRow], rows),
+ requested_page_numbers=parse_page_range_set(plan.page_range) if plan.page_range else None,
+ )
if plan.page_range:
loaded_pages = filter_pages_by_user_range(loaded_pages, plan.page_range)
if not loaded_pages:
@@ -1411,11 +1439,17 @@ class DoclibServer(AsyncDoclibInterface):
rows = cast(
list[ParseRow],
await self.state.db.fetchall(
- "SELECT tier FROM parses WHERE sha256=? AND status=? GROUP BY tier",
+ "SELECT * FROM parses WHERE sha256=? AND status=? ORDER BY done_at DESC",
(sha256, PARSE_STATUS_DONE),
),
)
- tiers: set[Tier] = {row["tier"] for row in rows if row["tier"] in TIERS and row["tier"] != "flash"}
+ tiers: set[Tier] = {
+ row["tier"]
+ for row in rows
+ if row["tier"] in TIERS
+ and row["tier"] != "flash"
+ and _json_batch_is_current(_effective_data_dir(self.state), sha256, row["tier"], row)
+ }
if not tiers:
return None
return select_highest_cached_tier(tiers)
diff --git a/mineru/doclib/services/parse_svc.py b/mineru/doclib/services/parse_svc.py
index 39f3caa8..3a351a81 100644
--- a/mineru/doclib/services/parse_svc.py
+++ b/mineru/doclib/services/parse_svc.py
@@ -745,7 +745,7 @@ class ParseService:
),
)
valid_done_batches = [
- batch for batch in done_batches if _json_file_exists_by_batch(self.data_dir, sha256, requested_tier, batch)
+ batch for batch in done_batches if _json_batch_is_current(self.data_dir, sha256, requested_tier, batch)
]
if supports_page_range:
for batch in valid_done_batches:
@@ -1404,7 +1404,9 @@ class ParseService:
tier_set: set[Tier] = {row["tier"] for row in rows}
tiers = sorted(tier_set, key=lambda item: TIER_ORDER.get(item, -1), reverse=True)
for tier in tiers:
- tier_rows = [row for row in rows if row["tier"] == tier]
+ tier_rows = [
+ row for row in rows if row["tier"] == tier and _json_batch_is_current(self.data_dir, sha256, tier, row)
+ ]
pages = load_pages_from_done_batches(self.data_dir, sha256, tier, tier_rows)
if not pages:
continue
@@ -1495,10 +1497,16 @@ def _probe_error_message(prefix: str, error_msg: str | None) -> str:
return prefix
-def _json_file_exists_by_batch(data_dir: str, sha256: str, tier: Tier, batch: ParseRow) -> bool:
- """Check that the JSON result file for a parses batch row actually exists on disk."""
+def _json_batch_is_current(data_dir: str, sha256: str, tier: Tier, batch: ParseRow) -> bool:
+ """仅把协议有效且覆盖记录页范围的新批次视为可命中的缓存。"""
json_path = parse_batch_json_path(data_dir, sha256, tier, batch["page_range"], batch["done_at"])
- return os.path.isfile(json_path)
+ try:
+ with open(json_path, encoding="utf-8") as stream:
+ result = ParseResult.from_dict(json.load(stream))
+ available = {page.page_idx + 1 for page in result.pages}
+ return bool(available) and parse_page_range_set(batch["page_range"]) <= available
+ except (OSError, ValueError, TypeError):
+ return False
def parse_batch_json_path(data_dir: str, sha256: str, tier: Tier, page_range: str, done_at: int | None = 0) -> str:
@@ -1507,9 +1515,17 @@ def parse_batch_json_path(data_dir: str, sha256: str, tier: Tier, page_range: st
return os.path.join(os.path.expanduser(data_dir), "parsed", sha256[:2], sha256, tier, filename)
-def load_pages_from_done_batches(data_dir: str, sha256: str, tier: Tier, done_rows: Sequence[ParseBatchRow]) -> list[PageInfo]:
- """Load valid done JSON batches and keep the newest page for duplicate page_idx values."""
+def load_pages_from_done_batches(
+ data_dir: str,
+ sha256: str,
+ tier: Tier,
+ done_rows: Sequence[ParseBatchRow],
+ *,
+ requested_page_numbers: set[int] | None = None,
+) -> list[PageInfo]:
+ """读取新协议批次并保留最新重复页,所请求的旧缓存需明确重新解析。"""
pages_by_page_idx: dict[int, PageInfo] = {}
+ stale_page_numbers: set[int] = set()
for row in reversed(done_rows):
fpath = parse_batch_json_path(data_dir, sha256, tier, row["page_range"], row["done_at"])
if not os.path.isfile(fpath):
@@ -1520,8 +1536,15 @@ def load_pages_from_done_batches(data_dir: str, sha256: str, tier: Tier, done_ro
parse_result = ParseResult.from_dict(data)
for page in parse_result.pages:
pages_by_page_idx[page.page_idx] = page
- except Exception:
- pass
+ except (OSError, ValueError, TypeError):
+ stale_page_numbers.update(parse_page_range_set(row["page_range"]))
+ uncovered_stale = stale_page_numbers - {index + 1 for index in pages_by_page_idx}
+ if requested_page_numbers is not None:
+ uncovered_stale &= requested_page_numbers
+ if uncovered_stale:
+ raise MineruError(
+ "stale_cache", "Cached document protocol is outdated or invalid; reparse the source document.", "page_range"
+ )
return [pages_by_page_idx[page_idx] for page_idx in sorted(pages_by_page_idx)]
diff --git a/mineru/integrations/docvortex.py b/mineru/integrations/docvortex.py
index 8971b65c..0f1a35bd 100644
--- a/mineru/integrations/docvortex.py
+++ b/mineru/integrations/docvortex.py
@@ -1,102 +1,38 @@
-"""在 DocVortex 共享文档对象与 MinerU 产品协议之间进行显式适配。"""
+"""校验 MinerU 产品扩展,共享文档协议由 DocVortex 统一维护。"""
from __future__ import annotations
-from typing import Any, Literal
-
-from pydantic import BaseModel, ConfigDict, Field, JsonValue
-
-from docvortex.schema import MiddleJson, ModelJson, Producer
+from typing import Literal
+from pydantic import BaseModel, ConfigDict, JsonValue
+from docvortex.schema import MiddleJson, ModelJson
class MinerUMetadata(BaseModel):
- """校验旧协议的产品元数据,禁止将未知档位静默映射。"""
+ """仅记录实际执行的产品档位和最终文字提取模式。"""
model_config = ConfigDict(extra="forbid", strict=True)
- effort: Literal["flash", "medium", "high", "xhigh"]
+ tier: Literal["flash", "basic", "standard", "advanced"]
parse_mode: Literal["txt", "ocr"]
- mineru_version: str = Field(min_length=1)
-def build_metadata(*, effort: str, parse_mode: str, mineru_version: str) -> dict[str, JsonValue]:
- """校验产品参数并生成命名空间扩展,不给公共语义类型添加宿主字段。"""
- metadata = MinerUMetadata.model_validate({"effort": effort, "parse_mode": parse_mode, "mineru_version": mineru_version})
+def build_metadata(*, effort: str, parse_mode: str) -> dict[str, JsonValue]:
+ """将分析结果的实际 effort 映射为产品档位,不记录请求值。"""
+ tiers = {"flash": "flash", "medium": "basic", "high": "standard", "xhigh": "advanced"}
+ if effort not in tiers:
+ raise ValueError(f"Unsupported actual analyze effort: {effort}")
+ metadata = MinerUMetadata(tier=tiers[effort], parse_mode=parse_mode)
return {"mineru": metadata.model_dump(mode="json")}
def with_mineru_metadata(document: ModelJson | MiddleJson, metadata: MinerUMetadata) -> None:
- """由宿主显式附加产品信息,文档本身不猜测 MinerU 版本。"""
+ """附加产品扩展并保留其他应用信息,不修改文档生产者。"""
document.extensions = {**document.extensions, "mineru": metadata.model_dump(mode="json")}
-def _from_payload(payload: dict[str, Any], *, middle: bool) -> ModelJson | MiddleJson:
- """把旧版顶层字段移入扩展,并复用唯一一套语义类型验证。"""
- data = dict(payload)
- data.pop("schema_version", None)
- if "pages" not in data:
- raise ValueError("Missing required pages in MinerU document")
- metadata = MinerUMetadata.model_validate({key: data.pop(key, None) for key in MinerUMetadata.model_fields})
- data["producer"] = Producer(name="mineru", version=metadata.mineru_version)
- data["extensions"] = {"mineru": metadata.model_dump(mode="json")}
- return MiddleJson.model_validate(data) if middle else ModelJson.model_validate(data)
-
-
-def from_mineru_model(payload: dict[str, Any]) -> ModelJson:
- """将当前 MinerU Model JSON 转换成中性分析对象。"""
- result = _from_payload(payload, middle=False)
- assert isinstance(result, ModelJson)
- return result
-
-
-def from_mineru_middle(payload: dict[str, Any]) -> MiddleJson:
- """将当前 MinerU Middle JSON 转换成中性语义对象。"""
- result = _from_payload(payload, middle=True)
- assert isinstance(result, MiddleJson)
- return result
-
-
-def _to_payload(
- document: ModelJson | MiddleJson, *, skip_defaults: bool, exclude_block_fields: set[str] | None = None
-) -> dict[str, Any]:
- """恢复旧版顶层字段,仅接受调用方真实提供的产品元数据。"""
- metadata = MinerUMetadata.model_validate(document.extensions.get("mineru"))
- payload = document.to_dict(skip_defaults=skip_defaults, exclude_block_fields=exclude_block_fields)
- for key in ("schema", "schema_version", "producer", "extensions"):
- payload.pop(key, None)
- return {**payload, **metadata.model_dump(mode="json")}
-
-
-def to_mineru_model(document: ModelJson, *, skip_defaults: bool = False) -> dict[str, Any]:
- """写出与现有 MinerU 相同的 Model JSON 封装。"""
- return _to_payload(document, skip_defaults=skip_defaults)
-
-
-def to_mineru_middle(
- document: MiddleJson,
- *,
- skip_defaults: bool = True,
- include_schema_version: bool = True,
- exclude_block_fields: set[str] | None = None,
-) -> dict[str, Any]:
- """写出 MinerU schema 2.0,允许调用方明确选择素材省略策略。"""
- payload = _to_payload(document, skip_defaults=skip_defaults, exclude_block_fields=exclude_block_fields)
- return {"schema_version": "2.0", **payload} if include_schema_version else payload
-
-
-def to_mineru_structured_content(value: dict[str, Any]) -> dict[str, Any]:
- """将共享 renderer 的中性文档头还原为现有 Structured Content 头。"""
- metadata = MinerUMetadata.model_validate(value.get("extensions", {}).get("mineru"))
- payload = {key: item for key, item in value.items() if key not in {"schema", "schema_version", "producer", "extensions"}}
- return {**payload, **metadata.model_dump(mode="json")}
+def validate_mineru_metadata(document: ModelJson | MiddleJson) -> None:
+ """只校验已存在的 MinerU 扩展,原生及其他生产者文档可直接消费。"""
+ if "mineru" in document.extensions:
+ MinerUMetadata.model_validate(document.extensions["mineru"])
-__all__ = [
- "build_metadata",
- "MinerUMetadata",
- "from_mineru_model",
- "from_mineru_middle",
- "to_mineru_model",
- "to_mineru_middle",
- "to_mineru_structured_content",
- "with_mineru_metadata",
-]
+__all__ = ["MinerUMetadata", "build_metadata", "with_mineru_metadata", "validate_mineru_metadata"]
diff --git a/mineru/kit/gradio/artifacts.py b/mineru/kit/gradio/artifacts.py
index e9c82c35..e7386060 100644
--- a/mineru/kit/gradio/artifacts.py
+++ b/mineru/kit/gradio/artifacts.py
@@ -225,11 +225,7 @@ def persist_parse_result(
shutil.copyfile(source_path, artifacts.source_path)
artifacts.middle_json_path.write_text(result.to_json(), encoding="utf-8")
if result._model_output is not None:
- model_output = (
- result._model_output.to_dict(skip_defaults=False)
- if hasattr(result._model_output, "to_dict")
- else result._model_output
- )
+ model_output = result._model_output.to_dict(skip_defaults=False)
(artifacts.root / "model_output.json").write_text(
json.dumps(model_output, ensure_ascii=False, indent=2),
encoding="utf-8",
diff --git a/mineru/parser/api_client.py b/mineru/parser/api_client.py
index a2857e48..45a27152 100644
--- a/mineru/parser/api_client.py
+++ b/mineru/parser/api_client.py
@@ -818,11 +818,17 @@ def _extract_model_output_from_archive(archive: zipfile.ZipFile) -> Any | None:
raise _V1APIError("invalid_model_output", f"ZIP output contained multiple model outputs: {available}")
model_output_name = legacy_names[0]
+ from ..types import ModelJson
+
raw = archive.read(model_output_name)
try:
- return json.loads(raw)
- except json.JSONDecodeError as exc:
- raise _V1APIError("invalid_model_output", f"model output JSON is not valid JSON: {exc}") from exc
+ model = ModelJson.from_json(raw)
+ from ..integrations.docvortex import validate_mineru_metadata
+
+ validate_mineru_metadata(model)
+ return model
+ except (ValueError, TypeError) as exc:
+ raise _V1APIError("invalid_model_output", f"Invalid Model JSON document: {exc}") from exc
def _download_zip_output(parser: MinerUApiParser, outputs: dict[str, Any]) -> bytes:
@@ -1113,14 +1119,9 @@ def _pages_from_middle_json(mid_json: dict[str, Any] | None) -> list[PageInfo]:
def _parse_result_from_middle_json(mid_json: dict[str, Any]) -> ParseResult:
- """把远端 middle_json 恢复为 ParseResult;legacy pdf_info 交给 ParseResult.from_dict 迁移。"""
+ """把远端 middle_json 恢复为 ParseResult;只接受当前共享文档协议。"""
if not isinstance(mid_json, dict):
raise _V1APIError("invalid_middle_json_output", "middle_json output must be a JSON object")
- if mid_json.get("pages") is None and mid_json.get("pdf_info") is None:
- raise _V1APIError(
- "invalid_middle_json_output",
- "middle_json output must contain a list field named pages or pdf_info",
- )
try:
return ParseResult.from_dict(mid_json)
except ValueError as exc:
diff --git a/mineru/parser/base.py b/mineru/parser/base.py
index 38a5f34a..265ad72e 100644
--- a/mineru/parser/base.py
+++ b/mineru/parser/base.py
@@ -1,83 +1,20 @@
# Copyright (c) Opendatalab. All rights reserved.
from __future__ import annotations
-from ..integrations.docvortex import from_mineru_middle, to_mineru_middle, to_mineru_model
-from docvortex.schema import Producer
-from ..integrations.docvortex import build_metadata
+from ..integrations.docvortex import validate_mineru_metadata
import json
from abc import ABC, abstractmethod
from copy import deepcopy
from dataclasses import dataclass
from pathlib import Path
-from typing import Any, Literal, cast
+from typing import Any
from ..render import render_markdown, render_structured_content
from ..render.contracts import ImageRenderer, RenderMode
-from ..types import FILE_SUFFIXES, FileSuffix, MiddleJson, ModelJson, PageInfo
+from ..types import MiddleJson, ModelJson, PageInfo
from .writer import DataWriter
-MIDDLE_JSON_SCHEMA_VERSION: str = "2.0"
-_LEGACY_SCHEMA_VERSION: str = "1.0"
-_LEGACY_DEFAULT_FILE_SUFFIX: FileSuffix = "pdf"
-_LEGACY_DEFAULT_EFFORT: Literal["flash", "medium", "high", "xhigh"] = "medium"
-_LEGACY_DEFAULT_PARSE_MODE: Literal["txt", "ocr"] = "txt"
-_TO_DICT_EXCLUDED_KEYS: frozenset[str] = frozenset({"schema_version"})
-
-
-def _legacy_raw_pages(payload: dict[str, Any]) -> list[dict[str, Any]] | None:
- """按 envelope 识别 3.4.5 原始 pdf_info 或后续 1.0 pages 包装。
-
- 3.4.5 release tag 内部仍报告 3.4.4,因此不能把 ``_version_name`` 作为
- 唯一判据。
- """
- schema_version = payload.get("schema_version")
- raw_pages = payload.get("pdf_info") if schema_version is None else None
- if raw_pages is None and schema_version == _LEGACY_SCHEMA_VERSION:
- raw_pages = payload.get("pages")
- if not isinstance(raw_pages, list) or any(not isinstance(page, dict) for page in raw_pages):
- return None
- return raw_pages
-
-
-def _legacy_page_index_map(raw_pages: list[dict[str, Any]]) -> list[int]:
- """从旧 page_idx 恢复抽页映射;连续零起始页面仍表示整本文档。"""
- page_indices: list[int] = []
- for fallback_index, page in enumerate(raw_pages):
- page_idx = page.get("page_idx", fallback_index)
- if isinstance(page_idx, bool) or not isinstance(page_idx, int) or page_idx < 0:
- raise ValueError("legacy Middle JSON page_idx values must be non-negative integers")
- page_indices.append(page_idx)
- if len(page_indices) != len(set(page_indices)) or any(
- current <= previous for previous, current in zip(page_indices, page_indices[1:])
- ):
- raise ValueError("legacy Middle JSON page_idx values must be unique and strictly increasing")
- return [] if page_indices == list(range(len(raw_pages))) else page_indices
-
-
-def _legacy_file_suffix(payload: dict[str, Any]) -> FileSuffix:
- """读取旧 payload 可选后缀;缺失时沿用历史 PDF 默认值。"""
- value = payload.get("file_suffix")
- if isinstance(value, str) and value in FILE_SUFFIXES:
- return cast(FileSuffix, value)
- return _LEGACY_DEFAULT_FILE_SUFFIX
-
-
-def _legacy_effort(payload: dict[str, Any]) -> Literal["flash", "medium", "high", "xhigh"]:
- """把 3.4.5 的分析强度映射到当前严格枚举。"""
- value = payload.get("_effort", payload.get("effort"))
- if value in {"flash", "medium", "high", "xhigh"}:
- return cast(Literal["flash", "medium", "high", "xhigh"], value)
- return _LEGACY_DEFAULT_EFFORT
-
-
-def _legacy_parse_mode(payload: dict[str, Any]) -> Literal["txt", "ocr"]:
- """优先读取显式 parse_mode,否则按 3.4.5 的 _ocr_enable 推断。"""
- value = payload.get("parse_mode")
- if value in {"txt", "ocr"}:
- return cast(Literal["txt", "ocr"], value)
- if payload.get("_ocr_enable") is True:
- return "ocr"
- return _LEGACY_DEFAULT_PARSE_MODE
+MIDDLE_JSON_SCHEMA_VERSION: str = MiddleJson.model_fields["schema_version"].default
@dataclass
@@ -89,7 +26,14 @@ class ParseResult:
"""
middle_json: MiddleJson
- _model_output: Any = None
+ _model_output: ModelJson | None = None
+
+ def __post_init__(self) -> None:
+ """要求可选分析结果为共享类型,禁止把任意旧列表重新导出为 Model JSON。"""
+ if self._model_output is not None:
+ if not isinstance(self._model_output, ModelJson):
+ raise TypeError("model output must be a ModelJson document")
+ validate_mineru_metadata(self._model_output)
@property
def pages(self) -> list[PageInfo]:
@@ -98,58 +42,17 @@ class ParseResult:
@staticmethod
def from_dict(d: dict[str, Any]) -> ParseResult:
- if not isinstance(d, dict):
- raise ValueError("ParseResult.from_dict expects a dict.")
-
- schema_version = d.get("schema_version")
-
- if schema_version == MIDDLE_JSON_SCHEMA_VERSION:
- middle_json = ParseResult._build_middle_json_from_current(d)
- elif (raw_pages := _legacy_raw_pages(d)) is not None:
- middle_json = ParseResult._build_middle_json_from_legacy(d, raw_pages)
- else:
- raise ValueError(
- f"Unsupported Middle JSON schema_version={schema_version!r}; "
- f"expected {MIDDLE_JSON_SCHEMA_VERSION!r}. Reparse the source document."
- )
-
+ """只读取统一新版协议,保留真实生产者并校验可选产品扩展。"""
+ middle_json = MiddleJson.from_dict(d)
+ validate_mineru_metadata(middle_json)
return ParseResult(middle_json=middle_json)
- @staticmethod
- def _build_middle_json_from_current(d: dict[str, Any]) -> MiddleJson:
- """从当前版本 schema 直接构造 Span 化 MiddleJson。"""
- payload = {k: v for k, v in d.items() if k not in _TO_DICT_EXCLUDED_KEYS}
- return from_mineru_middle(payload)
-
- @staticmethod
- def _build_middle_json_from_legacy(d: dict[str, Any], raw_pages: list[dict[str, Any]]) -> MiddleJson:
- """把 3.4.5 页面回推为 raw ModelJson,再走当前统一后处理生成 2.0。"""
- from ..backend.postprocess.legacy_schema_adapter import legacy_page_to_model_list
- from ..version import __version__ as current_mineru_version
-
- source_version = d.get("_version_name", d.get("mineru_version"))
- mineru_version = (
- source_version.strip() if isinstance(source_version, str) and source_version.strip() else current_mineru_version
- )
- model_json = ModelJson(
- pages=[legacy_page_to_model_list(page) for page in raw_pages],
- page_index_map=_legacy_page_index_map(raw_pages),
- file_suffix=_legacy_file_suffix(d),
- producer=Producer(name="mineru", version=mineru_version),
- extensions=build_metadata(
- effort=_legacy_effort(d), parse_mode=_legacy_parse_mode(d), mineru_version=mineru_version
- ),
- )
- from docvortex.postprocess.document import model_json_to_middle_json
-
- return model_json_to_middle_json(model_json)
-
def to_dict(self, *, skip_defaults: bool = True) -> dict[str, Any]:
- """保留 schema 2.0 输出及 PDF 图片省略约定,由独立 codec 还原封装。"""
- return to_mineru_middle(
- self.middle_json,
+ """输出共享协议,并保留 PDF 图片省略约定且不修改源对象。"""
+ validate_mineru_metadata(self.middle_json)
+ return self.middle_json.to_dict(
skip_defaults=skip_defaults,
- exclude_block_fields={"image_base64"} if self.middle_json.file_suffix == "pdf" else None,
+ exclude_block_fields={"image_base64"} if self.middle_json.metadata.file_suffix == "pdf" else None,
)
@staticmethod
@@ -192,11 +95,8 @@ class ParseResult:
)
if self._model_output is not None:
- model_output = (
- to_mineru_model(self._model_output, skip_defaults=False)
- if isinstance(self._model_output, ModelJson)
- else self._model_output
- )
+ validate_mineru_metadata(self._model_output)
+ model_output = self._model_output.to_dict(skip_defaults=False)
writer.write_string(
"model_output.json",
json.dumps(model_output, ensure_ascii=False, indent=2),
diff --git a/mineru/render/structured_content.py b/mineru/render/structured_content.py
index f639af70..95d937de 100644
--- a/mineru/render/structured_content.py
+++ b/mineru/render/structured_content.py
@@ -3,7 +3,7 @@
from __future__ import annotations
-from ..integrations.docvortex import to_mineru_structured_content
+from ..integrations.docvortex import validate_mineru_metadata
from typing import Any
from ..types import MiddleJson
@@ -18,8 +18,9 @@ def render_structured_content(
from ..config import config
from docvortex.render.structured_content import render_structured_content as _render_structured_content
- return to_mineru_structured_content(
- _render_structured_content(middle_json, asset_base_url=asset_base_url, latex_delimiters=config.render.latex_delimiters)
+ validate_mineru_metadata(middle_json)
+ return _render_structured_content(
+ middle_json, asset_base_url=asset_base_url, latex_delimiters=config.render.latex_delimiters
)
diff --git a/mineru/types.py b/mineru/types.py
index 0652bd26..90a12066 100644
--- a/mineru/types.py
+++ b/mineru/types.py
@@ -78,6 +78,7 @@ from docvortex.schema import (
parse_block,
Producer,
DocumentModel,
+ DocumentMetadata,
ModelJson,
PageInfo,
MiddleJson,
@@ -299,6 +300,7 @@ __all__ = [
"parse_block",
"Producer",
"DocumentModel",
+ "DocumentMetadata",
"ModelJson",
"PageInfo",
"MiddleJson",
diff --git a/pyproject.toml b/pyproject.toml
index e6095e6b..b3fa82f4 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -38,7 +38,7 @@ classifiers = [
"Programming Language :: Python :: 3.13",
]
dependencies = [
- "docvortex>=0.1.0,<1.0.0",
+ "docvortex>=0.2.0,<0.3.0",
"mineru-vl-utils>=1.2.1,<2",
"metafile-render>=0.3.0,<1.0.0",
"mineru-llama-cpp>=0.1.0,<1",
diff --git a/tests/unittest/test_content_list_ownership.py b/tests/unittest/test_content_list_ownership.py
index 4406713c..bbca7726 100644
--- a/tests/unittest/test_content_list_ownership.py
+++ b/tests/unittest/test_content_list_ownership.py
@@ -51,7 +51,7 @@ def test_migrated_inline_delimiter_cases(version: int, monkeypatch: pytest.Monke
],
)
],
- file_suffix="html",
+ metadata={"file_suffix": "html", "producer": {"name": "docvortex", "version": "0.2.0"}},
is_full_document=True,
)
before = middle.to_dict(skip_defaults=False)
diff --git a/tests/unittest/test_content_list_render.py b/tests/unittest/test_content_list_render.py
index 02f35755..24fb167e 100644
--- a/tests/unittest/test_content_list_render.py
+++ b/tests/unittest/test_content_list_render.py
@@ -35,9 +35,11 @@ def _middle(*pages: PageInfo) -> MiddleJson:
return MiddleJson(
pages=list(pages),
is_full_document=True,
- file_suffix="docx",
- producer=Producer(name="mineru", version="test"),
- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="test"),
+ metadata={"file_suffix": "docx", "producer": Producer(name="mineru", version="test")},
+ extensions=build_metadata(
+ effort="flash",
+ parse_mode="txt",
+ ),
)
diff --git a/tests/unittest/test_content_list_v2_render.py b/tests/unittest/test_content_list_v2_render.py
index a4f02271..81c2793e 100644
--- a/tests/unittest/test_content_list_v2_render.py
+++ b/tests/unittest/test_content_list_v2_render.py
@@ -35,9 +35,11 @@ def _middle(*pages: PageInfo) -> MiddleJson:
return MiddleJson(
pages=list(pages),
is_full_document=True,
- file_suffix="docx",
- producer=Producer(name="mineru", version="test"),
- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="test"),
+ metadata={"file_suffix": "docx", "producer": Producer(name="mineru", version="test")},
+ extensions=build_metadata(
+ effort="flash",
+ parse_mode="txt",
+ ),
)
diff --git a/tests/unittest/test_doclib_cache_semantics.py b/tests/unittest/test_doclib_cache_semantics.py
index 68278ab1..25984c9c 100644
--- a/tests/unittest/test_doclib_cache_semantics.py
+++ b/tests/unittest/test_doclib_cache_semantics.py
@@ -289,10 +289,9 @@ def _write_batch(
"schema_version": MIDDLE_JSON_SCHEMA_VERSION,
"pages": json_pages,
"is_full_document": is_full_document,
- "file_suffix": file_suffix,
- "effort": "medium",
- "parse_mode": "txt",
- "mineru_version": __version__,
+ "metadata": {"file_suffix": file_suffix, "producer": {"name": "mineru", "version": __version__}},
+ "schema": "docvortex.middle",
+ "extensions": {"mineru": {"tier": "basic", "parse_mode": "txt"}},
}
path.write_text(json.dumps(payload, ensure_ascii=False, indent=4), encoding="utf-8")
@@ -476,16 +475,18 @@ def test_parsing_rule_default_tier_allows_flash_fallback(monkeypatch: pytest.Mon
assert _resolve_parsing_rule_default_tier() == "flash"
-def test_read_without_tier_uses_highest_cached_tier() -> None:
- class _DB:
- async def fetchall(self, sql: str, params: tuple[Any, ...]) -> list[dict[str, str]]:
- assert sql == "SELECT tier FROM parses WHERE sha256=? AND status=? GROUP BY tier"
- assert params == ("a" * 64, "done")
- return [{"tier": "flash"}, {"tier": "basic"}, {"tier": "standard"}, {"tier": "advanced"}]
-
- server = DoclibServer(SimpleNamespace(db=_DB()))
-
- assert asyncio.run(server._default_read_tier("a" * 64)) == "advanced"
+def test_read_without_tier_uses_highest_cached_tier(tmp_path: Path) -> None:
+ """默认档位仅从有效协议缓存选择,忽略更高档位的历史结果。"""
+ sha256 = "a" * 64
+ rows = []
+ for tier in ["flash", "basic", "standard", "advanced"]:
+ _write_batch(tmp_path, sha256, tier, "1", 1000, [_text_page(tier).to_dict()])
+ rows.append({"sha256": sha256, "tier": tier, "page_range": "1", "done_at": 1000, "status": "done"})
+ server = DoclibServer(SimpleNamespace(db=_FakeDB(parses=rows, file_row=None), data_dir=str(tmp_path)))
+ assert asyncio.run(server._default_read_tier(sha256)) == "advanced"
+ stale = Path(parse_batch_json_path(str(tmp_path), sha256, "advanced", "1", 1000))
+ stale.write_text('{"schema_version":"1.0","pages":[]}')
+ assert asyncio.run(server._default_read_tier(sha256)) == "standard"
def test_managed_local_parse_server_url_uses_health_managed_url() -> None:
@@ -1223,9 +1224,15 @@ def test_remote_parse_server_default_url_is_declared_once() -> None:
def test_compaction_uses_configured_data_dir(tmp_path: Path) -> None:
sha256 = "b" * 64
tier = "standard"
- older_page = {"page_idx": 0, "content": "old"}
- older_duplicate = {"page_idx": 1, "content": "old"}
- newer_duplicate = {"page_idx": 1, "content": "new"}
+ older_page = {
+ "page_idx": 0,
+ "blocks": [{"type": "text", "index": 0, "bbox": [0.0, 0.0, 0.1, 0.1], "content": _inline("old")}],
+ }
+ older_duplicate = {"page_idx": 1, "blocks": older_page["blocks"]}
+ newer_duplicate = {
+ "page_idx": 1,
+ "blocks": [{"type": "text", "index": 0, "bbox": [0.0, 0.0, 0.1, 0.1], "content": _inline("new")}],
+ }
_write_batch(tmp_path, sha256, tier, "1-2", 1000, [older_page, older_duplicate])
_write_batch(tmp_path, sha256, tier, "2", 2000, [newer_duplicate])
@@ -1352,8 +1359,8 @@ def test_invalidate_deletes_fts_when_no_done_batches_remain(tmp_path: Path) -> N
def test_invalidate_rebuilds_fts_from_highest_remaining_done_tier(tmp_path: Path) -> None:
sha256 = "d" * 64
- _write_batch(tmp_path, sha256, "flash", "1", 1000, [{"page_idx": 1, "blocks": []}])
- _write_batch(tmp_path, sha256, "standard", "1", 2000, [{"page_idx": 1, "blocks": []}])
+ _write_batch(tmp_path, sha256, "flash", "1", 1000, [{"page_idx": 0, "blocks": []}])
+ _write_batch(tmp_path, sha256, "standard", "1", 2000, [{"page_idx": 0, "blocks": []}])
parses = [
{"sha256": sha256, "tier": "flash", "page_range": "1", "status": "done", "done_at": 1000},
{"sha256": sha256, "tier": "standard", "page_range": "1", "status": "done", "done_at": 2000},
@@ -1803,8 +1810,8 @@ def test_csv_doclib_parse_creates_flash_cache_and_rendered_fts(
cached_files = list((tmp_path / "data" / "parsed").rglob("*.json"))
assert len(cached_files) == 1
cached_payload = json.loads(cached_files[0].read_text(encoding="utf-8"))
- assert cached_payload["file_suffix"] == "csv"
- assert cached_payload["effort"] == "flash"
+ assert cached_payload["metadata"]["file_suffix"] == "csv"
+ assert cached_payload["extensions"]["mineru"]["tier"] == "flash"
finally:
await db.close()
@@ -2168,9 +2175,11 @@ def test_remap_api_result_pages_to_non_contiguous_page_range() -> None:
middle_json=MiddleJson(
pages=[PageInfo(page_idx=0), PageInfo(page_idx=1), PageInfo(page_idx=2)],
is_full_document=False,
- file_suffix="pdf",
- producer=Producer(name="mineru", version=__version__),
- extensions=build_metadata(effort="medium", parse_mode="txt", mineru_version=__version__),
+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version=__version__)},
+ extensions=build_metadata(
+ effort="medium",
+ parse_mode="txt",
+ ),
)
)
@@ -2187,10 +2196,9 @@ def test_remap_api_result_pages_remaps_page_indices() -> None:
"schema_version": MIDDLE_JSON_SCHEMA_VERSION,
"pages": [{"page_idx": 0, "blocks": []}],
"is_full_document": False,
- "file_suffix": "pdf",
- "effort": "medium",
- "parse_mode": "txt",
- "mineru_version": __version__,
+ "metadata": {"file_suffix": "pdf", "producer": {"name": "mineru", "version": __version__}},
+ "schema": "docvortex.middle",
+ "extensions": {"mineru": {"tier": "basic", "parse_mode": "txt"}},
}
)
@@ -2206,9 +2214,11 @@ def test_remap_api_result_pages_rejects_count_mismatch() -> None:
middle_json=MiddleJson(
pages=[PageInfo(page_idx=0), PageInfo(page_idx=1)],
is_full_document=False,
- file_suffix="pdf",
- producer=Producer(name="mineru", version=__version__),
- extensions=build_metadata(effort="medium", parse_mode="txt", mineru_version=__version__),
+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version=__version__)},
+ extensions=build_metadata(
+ effort="medium",
+ parse_mode="txt",
+ ),
)
)
@@ -4421,9 +4431,11 @@ def test_process_doc_marks_empty_page_result_failed(tmp_path: Path) -> None:
middle_json=MiddleJson(
pages=[],
is_full_document=False,
- file_suffix="pdf",
- producer=Producer(name="mineru", version=__version__),
- extensions=build_metadata(effort="medium", parse_mode="txt", mineru_version=__version__),
+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version=__version__)},
+ extensions=build_metadata(
+ effort="medium",
+ parse_mode="txt",
+ ),
)
)
@@ -4585,9 +4597,11 @@ def test_parse_via_api_requests_image_cache_only_for_office(
middle_json=MiddleJson(
pages=[],
is_full_document=True,
- file_suffix="pdf",
- producer=Producer(name="mineru", version=__version__),
- extensions=build_metadata(effort="medium", parse_mode="txt", mineru_version=__version__),
+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version=__version__)},
+ extensions=build_metadata(
+ effort="medium",
+ parse_mode="txt",
+ ),
)
)
@@ -4653,9 +4667,11 @@ def test_process_doc_fails_when_batch_json_cannot_be_written(tmp_path: Path) ->
middle_json=MiddleJson(
pages=[PageInfo(page_idx=0)],
is_full_document=False,
- file_suffix="pdf",
- producer=Producer(name="mineru", version=__version__),
- extensions=build_metadata(effort="medium", parse_mode="txt", mineru_version=__version__),
+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version=__version__)},
+ extensions=build_metadata(
+ effort="medium",
+ parse_mode="txt",
+ ),
)
)
@@ -4716,9 +4732,11 @@ def test_process_doc_normalizes_full_document_range_from_actual_pages(tmp_path:
middle_json=MiddleJson(
pages=[PageInfo(page_idx=page_idx) for page_idx in range(12)],
is_full_document=True,
- file_suffix="epub",
- producer=Producer(name="mineru", version=__version__),
- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version=__version__),
+ metadata={"file_suffix": "epub", "producer": Producer(name="mineru", version=__version__)},
+ extensions=build_metadata(
+ effort="flash",
+ parse_mode="txt",
+ ),
)
)
diff --git a/tests/unittest/test_doclib_legacy_page_ranges.py b/tests/unittest/test_doclib_legacy_page_ranges.py
index f1f24f9a..023a9afb 100644
--- a/tests/unittest/test_doclib_legacy_page_ranges.py
+++ b/tests/unittest/test_doclib_legacy_page_ranges.py
@@ -23,10 +23,84 @@ from mineru.doclib.services.parse_svc import (
)
from mineru.parser.base import MIDDLE_JSON_SCHEMA_VERSION
from mineru.version import __version__
+from mineru.errors import MineruError
_SHA256 = "a" * 64
[email protected]("old_version", [None, "1.0"])
+def test_old_protocol_cache_is_not_available_or_compacted(tmp_path: Path, old_version: str | None) -> None:
+ """旧协议不命中、不参与覆盖和压缩,读取旧页明确要求重新解析。"""
+
+ async def verify() -> None:
+ """在真实数据库中混存新旧批次,验证读写和缓存调度边界。"""
+ async with _legacy_store(tmp_path) as (db, service, server):
+ old_path = _cache_path(tmp_path, "1~5", 1000)
+ payload = json.loads(old_path.read_text())
+ if old_version is None:
+ payload.pop("schema")
+ else:
+ payload["schema_version"] = old_version
+ old_path.write_text(json.dumps(payload))
+ await _add_result(db, tmp_path, "4-7", 2000, range(4, 8), "new")
+ rows = await db.fetchall("SELECT * FROM parses ORDER BY done_at DESC")
+ before_files = {path.name: path.read_bytes() for path in old_path.parent.iterdir()}
+ compaction = Compaction(db=db, interval_sec=600, data_dir=str(tmp_path))
+ assert await compaction._compact_doc_tier(_SHA256, "flash") == 0
+ assert {path.name: path.read_bytes() for path in old_path.parent.iterdir()} == before_files
+ with pytest.raises(MineruError, match="reparse"):
+ load_pages_from_done_batches(str(tmp_path), _SHA256, "flash", rows)
+ current = await server._render_doc_content(
+ _SHA256, tier="flash", page_range="4-7", format="markdown", no_marker=True
+ )
+ assert "new page 4" in current and "old page" not in current
+ with pytest.raises(MineruError, match="reparse"):
+ await server._render_doc_content(_SHA256, tier="flash", page_range="1-3", format="markdown", no_marker=True)
+ listing = await server.list_parses(doc_ref="aaaaaaa", tier="flash", page_range="1-7")
+ assert listing.coverage.done_page_range == "4-7"
+ assert listing.coverage.missing_page_range == "1-3"
+ result = await service.request_parse(str(tmp_path / "document.pdf"), tier="flash", page_range="1-7")
+ assert not result.cache_hit and len(result.created_parse_ids) == 1
+ queued = await db.fetchone("SELECT page_range FROM parses WHERE id=?", (result.created_parse_ids[0],))
+ assert queued["page_range"] == "1-3"
+ assert old_path.read_bytes() == before_files[old_path.name]
+
+ asyncio.run(verify())
+
+
[email protected]("field", ["metadata", "extensions", "is_full_document", "extension_value_type"])
+def test_compaction_preserves_batches_with_conflicting_envelopes(tmp_path: Path, field: str) -> None:
+ """不同来源、扩展或整本语义的批次不得被合并为虚假来源的文档。"""
+
+ async def verify() -> None:
+ """逐项注入头部冲突并确认源文件和数据库完全保留。"""
+ async with _legacy_store(tmp_path) as (db, _service, _server):
+ await _add_result(db, tmp_path, "4-7", 2000, range(4, 8), "new")
+ path = _cache_path(tmp_path, "4-7", 2000)
+ payload = json.loads(path.read_text())
+ if field == "metadata":
+ payload[field]["producer"]["version"] = "different"
+ elif field == "extensions":
+ payload[field]["application"] = {"source": "different"}
+ elif field == "extension_value_type":
+ old_path = _cache_path(tmp_path, "1~5", 1000)
+ old_payload = json.loads(old_path.read_text())
+ old_payload["extensions"]["application"] = {"value": True}
+ old_path.write_text(json.dumps(old_payload))
+ payload["extensions"]["application"] = {"value": 1}
+ else:
+ payload[field] = True
+ path.write_text(json.dumps(payload))
+ rows = await db.fetchall("SELECT * FROM parses ORDER BY id")
+ files = {item.name: item.read_bytes() for item in path.parent.iterdir()}
+ compaction = Compaction(db=db, interval_sec=600, data_dir=str(tmp_path))
+ assert await compaction._compact_doc_tier(_SHA256, "flash") == 0
+ assert await db.fetchall("SELECT * FROM parses ORDER BY id") == rows
+ assert {item.name: item.read_bytes() for item in path.parent.iterdir()} == files
+
+ asyncio.run(verify())
+
+
def _cache_path(root: Path, page_range: str, done_at: int) -> Path:
"""按数据库原始范围定位批次文件,保留旧半角波浪号文件名。"""
return Path(parse_batch_json_path(str(root), _SHA256, "flash", page_range, done_at))
@@ -41,10 +115,9 @@ async def _add_result(
payload = {
"schema_version": MIDDLE_JSON_SCHEMA_VERSION,
"is_full_document": False,
- "file_suffix": "pdf",
- "effort": "flash",
- "parse_mode": "txt",
- "mineru_version": __version__,
+ "metadata": {"file_suffix": "pdf", "producer": {"name": "mineru", "version": __version__}},
+ "schema": "docvortex.middle",
+ "extensions": {"mineru": {"tier": "flash", "parse_mode": "txt"}},
"pages": [
{
"page_idx": page_no - 1,
diff --git a/tests/unittest/test_docvortex_protocol.py b/tests/unittest/test_docvortex_protocol.py
index 716c8516..45ead2d3 100644
--- a/tests/unittest/test_docvortex_protocol.py
+++ b/tests/unittest/test_docvortex_protocol.py
@@ -1,116 +1,95 @@
-"""验证 MinerU 独立维护产品封装,DocVortex 只提供共享语义对象。"""
+"""验证宿主与原生文档共享协议,同时保留产品扩展及素材约定。"""
-from collections.abc import Callable
from copy import deepcopy
-from typing import Literal
-
import pytest
from pydantic import ValidationError
-
-from docvortex.schema import EquationBlock, ImageBlock, ImageBodyBlock, MiddleJson, PageInfo, TextBlock, TextSpan
-from mineru.integrations.docvortex import (
- MinerUMetadata,
- build_metadata,
- from_mineru_middle,
- to_mineru_middle,
- with_mineru_metadata,
-)
-from mineru.render.structured_content import render_structured_content
+from docvortex.schema import DocumentMetadata, Producer, MiddleJson, PageInfo, TextBlock, TextSpan, ImageBodyBlock, ImageBlock
+from mineru.integrations.docvortex import MinerUMetadata, build_metadata, validate_mineru_metadata, with_mineru_metadata
from mineru.parser import ParseResult
+from mineru.render.structured_content import render_structured_content
def _document() -> MiddleJson:
- """建立不携带宿主信息的最小语义文档,覆盖文字和公式。"""
+ """建立有真实正文且无 MinerU 运行记录的原生文档。"""
return MiddleJson(
pages=[
- PageInfo(
- page_idx=0,
- blocks=[
- TextBlock(type="text", index=0, content=[TextSpan(type="text", content="Hello 文档")]),
- EquationBlock(type="equation", index=1, content="x^2"),
- ],
- )
+ PageInfo(page_idx=0, blocks=[TextBlock(type="text", index=0, content=[TextSpan(type="text", content="保留正文")])])
],
is_full_document=True,
- file_suffix="html",
+ metadata=DocumentMetadata(file_suffix="html", producer=Producer(name="docvortex", version="0.2.0")),
)
-def test_mineru_envelope_roundtrip() -> None:
- """从引擎迁入的往返契约保证产品字段和页面语义不变。"""
- payload = _document().to_dict(skip_defaults=False)
- for key in ("schema", "schema_version", "producer", "extensions"):
- payload.pop(key, None)
- payload.update(schema_version="2.0", effort="flash", parse_mode="txt", mineru_version="3.4.5")
- before = deepcopy(payload)
-
- assert to_mineru_middle(from_mineru_middle(payload), skip_defaults=False) == payload
- assert payload == before
+def test_native_middle_does_not_lose_text_or_rewrite_producer() -> None:
+ """回归原生 1.0 被误认作宿主历史格式后正文丢失的问题。"""
+ document = _document()
+ before = document.to_dict()
+ result = ParseResult.from_dict(before)
+ assert result.to_dict() == before
+ assert result.pages[0].blocks[0].content[0].content == "保留正文"
+ assert result.middle_json.metadata.producer.name == "docvortex"
+ structured = render_structured_content(document)
+ assert structured["metadata"] == before["metadata"]
+ assert structured["extensions"] == {}
+ assert not {"schema", "schema_version", "schema_id"} & structured.keys()
+ assert structured["pages"][0]["blocks"][0]["content"] == "保留正文"
+ assert document.to_dict() == before
-def test_host_metadata_preserves_other_extensions_and_rendering_does_not_mutate_document() -> None:
- """宿主附加产品信息后仍保留其它扩展,公共渲染只加工输出封装。"""
- document = _document()
- document.extensions = {"application": {"source": "fixture"}}
- metadata = MinerUMetadata(effort="high", parse_mode="ocr", mineru_version="3.4.5")
- with_mineru_metadata(document, metadata)
- before = document.to_dict(skip_defaults=False)
[email protected]("effort,tier", [("flash", "flash"), ("medium", "basic"), ("high", "standard"), ("xhigh", "advanced")])
[email protected]("mode", ["txt", "ocr"])
+def test_actual_effort_maps_to_product_tier(effort: str, tier: str, mode: str) -> None:
+ """所有分析强度均映射为实际公开档位,扩展不重复存放版本。"""
+ assert build_metadata(effort=effort, parse_mode=mode) == {"mineru": {"tier": tier, "parse_mode": mode}}
- payload = to_mineru_middle(document)
- structured = render_structured_content(document)
- assert document.extensions["application"] == {"source": "fixture"}
- assert document.extensions["mineru"] == metadata.model_dump(mode="json")
- assert build_metadata(**metadata.model_dump()) == {"mineru": document.extensions["mineru"]}
- assert payload["schema_version"] == "2.0"
- assert structured["pages"][0]["blocks"][0]["content"] == "Hello 文档"
- for output in (payload, structured):
- assert {key: output[key] for key in MinerUMetadata.model_fields} == metadata.model_dump(mode="json")
- assert not {"schema", "producer", "extensions"} & output.keys()
- assert "schema_version" not in structured
- assert document.to_dict(skip_defaults=False) == before
+def test_extensions_survive_host_rendering_and_roundtrip() -> None:
+ """保留未知应用扩展,产品登记和渲染均不改写真实来源。"""
+ document = _document()
+ document.extensions = {"application": {"items": [None, True, 3, {"text": "中文"}]}}
+ with_mineru_metadata(document, MinerUMetadata(tier="advanced", parse_mode="ocr"))
+ before = deepcopy(document.to_dict())
+ assert ParseResult.from_dict(before).to_dict() == before
+ assert render_structured_content(document)["extensions"] == before["extensions"]
+ assert document.to_dict() == before
[email protected]("operation", [to_mineru_middle, render_structured_content])
-def test_host_output_requires_explicit_mineru_metadata(operation: Callable[[MiddleJson], object]) -> None:
- """没有产品元数据的原生返回值不能被静默伪装为 MinerU 输出。"""
+ "metadata",
+ [
+ {"tier": "low", "parse_mode": "txt"},
+ {"tier": "basic", "parse_mode": "auto"},
+ {"tier": "flash", "parse_mode": "txt", "effort": "flash"},
+ None,
+ ],
+)
+def test_mineru_validates_only_present_product_extension(metadata: object) -> None:
+ """原生扩展可缺省,已存在的产品记录则必须满足严格枚举。"""
+ document = _document()
+ validate_mineru_metadata(document)
+ document.extensions = {"mineru": metadata}
with pytest.raises(ValidationError):
- operation(_document())
+ ParseResult.from_dict(document.to_dict())
@pytest.mark.parametrize("suffix", ["pdf", "docx"])
-def test_parse_result_preserves_format_specific_image_omission(suffix: Literal["pdf", "docx"]) -> None:
- """宿主结果写出时仅省略 PDF 内嵌图片,保留图片路径且不修改语义对象。"""
+def test_format_specific_image_omission_is_preserved(suffix: str) -> None:
+ """PDF 包装继续省略内嵌图片,其他格式保留,且源对象不变。"""
body = ImageBodyBlock(
type="image_body",
index=0,
content="figure",
bbox=(0.1, 0.1, 0.9, 0.9),
- image_path="images/figure.png",
+ image_path="images/a.png",
image_base64="data:image/png;base64,aW1hZ2U=",
)
document = MiddleJson(
pages=[PageInfo(page_idx=0, blocks=[ImageBlock(type="image", index=0, bbox=body.bbox, content=[body])])],
is_full_document=True,
- file_suffix=suffix,
- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="3.4.5"),
+ metadata=DocumentMetadata(file_suffix=suffix, producer=Producer(name="external", version="original")),
)
before = document.to_dict(skip_defaults=False)
-
payload = ParseResult(document).to_dict()
-
- exported_body = payload["pages"][0]["blocks"][0]["content"][0]
- assert ("image_base64" in exported_body) is (suffix != "pdf")
- assert exported_body["image_path"] == "images/figure.png"
+ assert ("image_base64" in payload["pages"][0]["blocks"][0]["content"][0]) is (suffix != "pdf")
+ assert payload["pages"][0]["blocks"][0]["content"][0]["image_path"] == "images/a.png"
assert document.to_dict(skip_defaults=False) == before
-
-
- ("field", "value"),
- [("effort", "low"), ("parse_mode", "auto"), ("mineru_version", ""), ("mineru_version", 345)],
-)
-def test_host_metadata_validation_stays_strict(field: str, value: object) -> None:
- """产品枚举、版本非空约束及类型检查由 MinerU 继续严格执行。"""
- metadata = {"effort": "flash", "parse_mode": "txt", "mineru_version": "3.4.5", field: value}
- with pytest.raises(ValidationError):
- build_metadata(**metadata)
diff --git a/tests/unittest/test_flash_csv.py b/tests/unittest/test_flash_csv.py
index 57ed8e41..b8045700 100644
--- a/tests/unittest/test_flash_csv.py
+++ b/tests/unittest/test_flash_csv.py
@@ -21,8 +21,8 @@ def test_csv_doc_analyze_sync_async_and_render_contracts_match() -> None:
middle, model = doc_analyze(payload, file_suffix="csv")
async_middle, async_model = asyncio.run(aio_doc_analyze(payload, file_suffix="csv"))
- assert model.file_suffix == async_model.file_suffix == "csv"
- assert model.extensions["mineru"]["effort"] == async_model.extensions["mineru"]["effort"] == "flash"
+ assert model.metadata.file_suffix == async_model.metadata.file_suffix == "csv"
+ assert model.extensions["mineru"]["tier"] == async_model.extensions["mineru"]["tier"] == "flash"
assert model.extensions["mineru"]["parse_mode"] == async_model.extensions["mineru"]["parse_mode"] == "txt"
assert middle.model_dump() == async_middle.model_dump()
assert len(middle.pages) == 1
@@ -50,7 +50,7 @@ def test_csv_path_parsing_and_signatureless_detection(tmp_path: Path) -> None:
result = parse(csv_path)
async_result = asyncio.run(parse_async(csv_path))
- assert result.middle_json.file_suffix == async_result.middle_json.file_suffix == "csv"
+ assert result.middle_json.metadata.file_suffix == async_result.middle_json.metadata.file_suffix == "csv"
assert result.markdown() == async_result.markdown()
with pytest.raises(ValueError, match="Unsupported file type: txt"):
parse(text_path)
@@ -95,12 +95,12 @@ def test_csv_parse_server_job_emits_structured_outputs_with_flash_metadata(tmp_p
middle_record = file_store.get_file(parsed_file.output_files.middle_json.file_id)
assert middle_record.sha256sum is not None
middle_payload = json.loads(file_store.read_blob(middle_record.sha256sum))
- assert middle_payload["file_suffix"] == "csv"
- assert middle_payload["effort"] == "flash"
- assert middle_payload["parse_mode"] == "txt"
+ assert middle_payload["metadata"]["file_suffix"] == "csv"
+ assert middle_payload["extensions"]["mineru"]["tier"] == "flash"
+ assert middle_payload["extensions"]["mineru"]["parse_mode"] == "txt"
structured_record = file_store.get_file(parsed_file.output_files.structured_content.file_id)
assert structured_record.sha256sum is not None
structured_payload = json.loads(file_store.read_blob(structured_record.sha256sum))
- assert structured_payload["file_suffix"] == "csv"
+ assert structured_payload["metadata"]["file_suffix"] == "csv"
assert structured_payload["pages"][0]["blocks"][0]["type"] == "table"
diff --git a/tests/unittest/test_flash_docx_equationxml.py b/tests/unittest/test_flash_docx_equationxml.py
index ed44b3f0..110a2ca6 100644
--- a/tests/unittest/test_flash_docx_equationxml.py
+++ b/tests/unittest/test_flash_docx_equationxml.py
@@ -32,7 +32,7 @@ def test_docx_equationxml_analyze_lifecycle_and_strict_contracts() -> None:
assert _equation_contents(pages) == [r"\frac{a}{b}"]
assert isinstance(model, ModelJson)
assert isinstance(middle, MiddleJson)
- assert model.file_suffix == middle.file_suffix == "docx"
+ assert model.metadata.file_suffix == middle.metadata.file_suffix == "docx"
assert middle.pages[0].blocks[0].type == BlockType.EQUATION
assert middle.pages[0].blocks[0].content == r"\frac{a}{b}"
assert async_model == model
diff --git a/tests/unittest/test_flash_epub.py b/tests/unittest/test_flash_epub.py
index 019e2cbd..40b47a0b 100644
--- a/tests/unittest/test_flash_epub.py
+++ b/tests/unittest/test_flash_epub.py
@@ -41,8 +41,8 @@ def test_epub_model_analyze_and_renderers_preserve_structured_content() -> None:
assert explicit_full_model.pages == model_pages
assert explicit_full_middle.is_full_document is True
assert middle.model_dump() == async_middle.model_dump()
- assert model.file_suffix == middle.file_suffix == "epub"
- assert model.extensions["mineru"]["effort"] == middle.extensions["mineru"]["effort"] == "flash"
+ assert model.metadata.file_suffix == middle.metadata.file_suffix == "epub"
+ assert model.extensions["mineru"]["tier"] == middle.extensions["mineru"]["tier"] == "flash"
assert model.extensions["mineru"]["parse_mode"] == middle.extensions["mineru"]["parse_mode"] == "txt"
assert [page.page_idx for page in middle.pages] == [0, 1, 2]
assert middle.pages[0].blocks[0].type == BlockType.DOC_TITLE
@@ -75,7 +75,7 @@ def test_epub_model_analyze_and_renderers_preserve_structured_content() -> None:
assert "hidden secret" not in markdown
assert "Data table" in markdown
assert "<table" in html_output
- assert structured["file_suffix"] == "epub"
+ assert structured["metadata"]["file_suffix"] == "epub"
assert docx.startswith(b"PK")
@@ -180,9 +180,9 @@ def test_epub_local_parse_job_emits_spine_aligned_flash_outputs(tmp_path: Path)
middle_record = file_store.get_file(parsed_file.output_files.middle_json.file_id) # type: ignore[union-attr]
assert middle_record.sha256sum is not None
payload = json.loads(file_store.read_blob(middle_record.sha256sum))
- assert payload["file_suffix"] == "epub"
- assert payload["effort"] == "flash"
- assert payload["parse_mode"] == "txt"
+ assert payload["metadata"]["file_suffix"] == "epub"
+ assert payload["extensions"]["mineru"]["tier"] == "flash"
+ assert payload["extensions"]["mineru"]["parse_mode"] == "txt"
assert [page["page_idx"] for page in payload["pages"]] == [0, 1, 2]
assert payload["pages"][0]["blocks"][0]["type"] == "doc_title"
@@ -306,5 +306,5 @@ def test_epub_middle_json_roundtrip_remains_schema_2() -> None:
"""验证 EPUB 只扩展 file_suffix,不引入新的 schema 或 Block 字段。"""
middle, _ = doc_analyze(build_epub_fixture(), file_suffix="epub")
payload = middle.to_dict(skip_defaults=False)
- assert payload["file_suffix"] == "epub"
- assert json.loads(middle.to_json(skip_defaults=False))["file_suffix"] == "epub"
+ assert payload["metadata"]["file_suffix"] == "epub"
+ assert json.loads(middle.to_json(skip_defaults=False))["metadata"]["file_suffix"] == "epub"
diff --git a/tests/unittest/test_flash_html.py b/tests/unittest/test_flash_html.py
index 1f6d6d79..2fc49658 100644
--- a/tests/unittest/test_flash_html.py
+++ b/tests/unittest/test_flash_html.py
@@ -63,8 +63,8 @@ def test_html_doc_analyze_projects_static_semantics_and_renderers() -> None:
assert middle.model_dump() == async_middle.model_dump()
assert model.pages == async_model.pages
- assert middle.file_suffix == model.file_suffix == "html"
- assert middle.extensions["mineru"]["effort"] == model.extensions["mineru"]["effort"] == "flash"
+ assert middle.metadata.file_suffix == model.metadata.file_suffix == "html"
+ assert middle.extensions["mineru"]["tier"] == model.extensions["mineru"]["tier"] == "flash"
assert middle.extensions["mineru"]["parse_mode"] == model.extensions["mineru"]["parse_mode"] == "txt"
assert middle.is_full_document is True
assert [page.page_idx for page in middle.pages] == [0]
@@ -97,7 +97,7 @@ def test_html_doc_analyze_projects_static_semantics_and_renderers() -> None:
assert "```python" in markdown and "x^2" in markdown
assert "https://cdn.example.com/a.png" in markdown
assert "<table" in render_html(middle)
- assert render_structured_content(middle)["file_suffix"] == "html"
+ assert render_structured_content(middle)["metadata"]["file_suffix"] == "html"
docx = render_docx(middle)
assert docx.startswith(b"PK")
with ZipFile(BytesIO(docx)) as archive:
@@ -285,7 +285,7 @@ def test_html_local_base_images_styles_and_escape_are_bounded(tmp_path: Path) ->
async_result = asyncio.run(parse_async(source))
assert result.middle_json.model_dump() == async_result.middle_json.model_dump()
- assert result.middle_json.file_suffix == "html"
+ assert result.middle_json.metadata.file_suffix == "html"
assert _image_body(result.middle_json).image_base64.startswith("data:image/png;base64,")
markdown = result.markdown()
assert "hidden css" not in markdown
@@ -329,8 +329,8 @@ def test_html_parse_server_local_source_keeps_relative_assets(tmp_path: Path) ->
middle_record = file_store.get_file(parsed_file.output_files.middle_json.file_id)
assert middle_record.sha256sum is not None
middle_payload = json.loads(file_store.read_blob(middle_record.sha256sum))
- assert middle_payload["file_suffix"] == "html"
- assert middle_payload["effort"] == "flash"
+ assert middle_payload["metadata"]["file_suffix"] == "html"
+ assert middle_payload["extensions"]["mineru"]["tier"] == "flash"
image_body = middle_payload["pages"][0]["blocks"][1]["content"][0]
assert image_body["image_base64"].startswith("data:image/png;base64,")
@@ -349,8 +349,8 @@ def test_html_doclib_local_bridge_uses_flash_parser(tmp_path: Path) -> None:
)
)
- assert result.middle_json.file_suffix == "html"
- assert result.middle_json.extensions["mineru"]["effort"] == "flash"
+ assert result.middle_json.metadata.file_suffix == "html"
+ assert result.middle_json.extensions["mineru"]["tier"] == "flash"
assert "Doclib HTML" in result.markdown()
diff --git a/tests/unittest/test_flash_legacy_doc.py b/tests/unittest/test_flash_legacy_doc.py
index 00bf53ca..8cbb4dd0 100644
--- a/tests/unittest/test_flash_legacy_doc.py
+++ b/tests/unittest/test_flash_legacy_doc.py
@@ -20,8 +20,8 @@ def test_doc_analyze_sync_and_async_return_strict_doc_contract() -> None:
assert isinstance(model, ModelJson)
assert isinstance(middle, MiddleJson)
- assert model.file_suffix == middle.file_suffix == "doc"
- assert model.extensions["mineru"]["effort"] == middle.extensions["mineru"]["effort"] == "flash"
+ assert model.metadata.file_suffix == middle.metadata.file_suffix == "doc"
+ assert model.extensions["mineru"]["tier"] == middle.extensions["mineru"]["tier"] == "flash"
assert model.extensions["mineru"]["parse_mode"] == middle.extensions["mineru"]["parse_mode"] == "txt"
assert async_model == model
assert async_middle == middle
@@ -35,5 +35,5 @@ def test_doc_is_supported_by_public_parser(tmp_path: Path) -> None:
result = parse(path, tier="flash")
- assert result.middle_json.file_suffix == "doc"
+ assert result.middle_json.metadata.file_suffix == "doc"
assert inline_text(result.pages[0].blocks[0].content) == "Hello"
diff --git a/tests/unittest/test_flash_legacy_ppt.py b/tests/unittest/test_flash_legacy_ppt.py
index ecadb2c5..2d7660b9 100644
--- a/tests/unittest/test_flash_legacy_ppt.py
+++ b/tests/unittest/test_flash_legacy_ppt.py
@@ -19,9 +19,9 @@ def test_backend_analyze_accepts_ppt_and_async_contract() -> None:
assert isinstance(model_json, ModelJson)
assert isinstance(middle_json, MiddleJson)
- assert model_json.file_suffix == "ppt"
- assert middle_json.file_suffix == "ppt"
- assert model_json.extensions["mineru"]["effort"] == middle_json.extensions["mineru"]["effort"] == "flash"
+ assert model_json.metadata.file_suffix == "ppt"
+ assert middle_json.metadata.file_suffix == "ppt"
+ assert model_json.extensions["mineru"]["tier"] == middle_json.extensions["mineru"]["tier"] == "flash"
assert model_json.extensions["mineru"]["parse_mode"] == middle_json.extensions["mineru"]["parse_mode"] == "txt"
assert model_json.is_full_document is middle_json.is_full_document is True
assert [page.page_idx for page in middle_json.pages] == [0, 1]
@@ -37,5 +37,5 @@ def test_ppt_is_supported_by_public_parser(tmp_path: Path) -> None:
result = parse(path, tier="flash")
- assert result.middle_json.file_suffix == "ppt"
+ assert result.middle_json.metadata.file_suffix == "ppt"
assert len(result.pages) == 2
diff --git a/tests/unittest/test_flash_legacy_xls.py b/tests/unittest/test_flash_legacy_xls.py
index f61ad11b..ae92c48f 100644
--- a/tests/unittest/test_flash_legacy_xls.py
+++ b/tests/unittest/test_flash_legacy_xls.py
@@ -24,8 +24,8 @@ def test_backend_analyze_accepts_xls_and_async_contract() -> None:
assert isinstance(model_json, ModelJson)
assert isinstance(middle_json, MiddleJson)
- assert model_json.file_suffix == middle_json.file_suffix == "xls"
- assert model_json.extensions["mineru"]["effort"] == middle_json.extensions["mineru"]["effort"] == "flash"
+ assert model_json.metadata.file_suffix == middle_json.metadata.file_suffix == "xls"
+ assert model_json.extensions["mineru"]["tier"] == middle_json.extensions["mineru"]["tier"] == "flash"
assert model_json.extensions["mineru"]["parse_mode"] == middle_json.extensions["mineru"]["parse_mode"] == "txt"
assert model_json.is_full_document is middle_json.is_full_document is True
assert async_model == model_json
@@ -40,5 +40,5 @@ def test_xls_is_supported_by_public_parser(tmp_path: Path) -> None:
result = parse(path, tier="flash")
- assert result.middle_json.file_suffix == "xls"
+ assert result.middle_json.metadata.file_suffix == "xls"
assert inline_text(result.pages[0].blocks[0].content) == "value"
diff --git a/tests/unittest/test_flash_legacy_xls_ppt_mtef.py b/tests/unittest/test_flash_legacy_xls_ppt_mtef.py
index ea44ef9f..b03ad084 100644
--- a/tests/unittest/test_flash_legacy_xls_ppt_mtef.py
+++ b/tests/unittest/test_flash_legacy_xls_ppt_mtef.py
@@ -28,7 +28,7 @@ def test_backend_analyze_preserves_native_equations_sync_and_async(
assert isinstance(model, ModelJson)
assert isinstance(middle, MiddleJson)
- assert model.file_suffix == middle.file_suffix == file_suffix
+ assert model.metadata.file_suffix == middle.metadata.file_suffix == file_suffix
assert middle.pages[0].blocks[0].type == BlockType.EQUATION
assert middle.pages[0].blocks[0].content == expected
assert async_model == model
diff --git a/tests/unittest/test_flash_odf.py b/tests/unittest/test_flash_odf.py
index 59102180..42263dd0 100644
--- a/tests/unittest/test_flash_odf.py
+++ b/tests/unittest/test_flash_odf.py
@@ -49,8 +49,8 @@ def test_odf_models_and_analyze_keep_flash_contract(
)
assert model.pages == async_model.pages == model_pages
assert middle.model_dump() == async_middle.model_dump()
- assert model.file_suffix == middle.file_suffix == suffix
- assert model.extensions["mineru"]["effort"] == middle.extensions["mineru"]["effort"] == "flash"
+ assert model.metadata.file_suffix == middle.metadata.file_suffix == suffix
+ assert model.extensions["mineru"]["tier"] == middle.extensions["mineru"]["tier"] == "flash"
assert model.extensions["mineru"]["parse_mode"] == middle.extensions["mineru"]["parse_mode"] == "txt"
@@ -119,7 +119,7 @@ def test_public_parser_handles_odf_sync_and_async(
source.write_bytes(payload)
result = parse(source)
async_result = asyncio.run(parse_async(source))
- assert result.middle_json.file_suffix == async_result.middle_json.file_suffix == suffix
+ assert result.middle_json.metadata.file_suffix == async_result.middle_json.metadata.file_suffix == suffix
assert result.middle_json.model_dump() == async_result.middle_json.model_dump()
@@ -151,9 +151,9 @@ def test_odf_parse_server_job_emits_flash_outputs(tmp_path: Path) -> None:
middle_record = file_store.get_file(parsed_file.output_files.middle_json.file_id) # type: ignore[union-attr]
assert middle_record.sha256sum is not None
payload = json.loads(file_store.read_blob(middle_record.sha256sum))
- assert payload["file_suffix"] == "odt"
- assert payload["effort"] == "flash"
- assert payload["parse_mode"] == "txt"
+ assert payload["metadata"]["file_suffix"] == "odt"
+ assert payload["extensions"]["mineru"]["tier"] == "flash"
+ assert payload["extensions"]["mineru"]["parse_mode"] == "txt"
@pytest.mark.parametrize(
diff --git a/tests/unittest/test_flash_ofd.py b/tests/unittest/test_flash_ofd.py
index dde45289..c48f438e 100644
--- a/tests/unittest/test_flash_ofd.py
+++ b/tests/unittest/test_flash_ofd.py
@@ -48,15 +48,15 @@ def test_ofd_model_analyze_detection_and_renderers(tmp_path: Path) -> None:
assert model.pages == async_model.pages == model_pages
assert middle.model_dump() == async_middle.model_dump()
- assert model.file_suffix == middle.file_suffix == "ofd"
- assert model.extensions["mineru"]["effort"] == middle.extensions["mineru"]["effort"] == "flash"
+ assert model.metadata.file_suffix == middle.metadata.file_suffix == "ofd"
+ assert model.extensions["mineru"]["tier"] == middle.extensions["mineru"]["tier"] == "flash"
assert model.extensions["mineru"]["parse_mode"] == middle.extensions["mineru"]["parse_mode"] == "txt"
assert middle.is_full_document is True
assert middle.pages[0].blocks[0].type == BlockType.TEXT
assert middle.pages[0].blocks[0].bbox is not None
assert "你好,OFD" in render_markdown(middle)
assert "你好,OFD" in render_html(middle)
- assert render_structured_content(middle)["file_suffix"] == "ofd"
+ assert render_structured_content(middle)["metadata"]["file_suffix"] == "ofd"
assert render_docx(middle).startswith(b"PK")
@@ -104,9 +104,9 @@ def test_ofd_parse_server_job_emits_flash_outputs(tmp_path: Path) -> None:
middle_record = file_store.get_file(parsed_file.output_files.middle_json.file_id) # type: ignore[union-attr]
assert middle_record.sha256sum is not None
payload = json.loads(file_store.read_blob(middle_record.sha256sum))
- assert payload["file_suffix"] == "ofd"
- assert payload["effort"] == "flash"
- assert payload["parse_mode"] == "txt"
+ assert payload["metadata"]["file_suffix"] == "ofd"
+ assert payload["extensions"]["mineru"]["tier"] == "flash"
+ assert payload["extensions"]["mineru"]["parse_mode"] == "txt"
assert payload["pages"][0]["blocks"][0]["bbox"]
diff --git a/tests/unittest/test_flash_office_mtef_v5.py b/tests/unittest/test_flash_office_mtef_v5.py
index f9782a5b..b70221b8 100644
--- a/tests/unittest/test_flash_office_mtef_v5.py
+++ b/tests/unittest/test_flash_office_mtef_v5.py
@@ -51,7 +51,7 @@ def test_mtef_v5_runs_through_sync_async_analyze(
assert isinstance(model, ModelJson)
assert isinstance(middle, MiddleJson)
- assert model.file_suffix == middle.file_suffix == file_suffix
+ assert model.metadata.file_suffix == middle.metadata.file_suffix == file_suffix
assert middle.pages[0].blocks[0].type == BlockType.EQUATION
assert middle.pages[0].blocks[0].content == expected
assert async_model == model
diff --git a/tests/unittest/test_flash_ooxml_mtef.py b/tests/unittest/test_flash_ooxml_mtef.py
index f6a592a2..09033d8b 100644
--- a/tests/unittest/test_flash_ooxml_mtef.py
+++ b/tests/unittest/test_flash_ooxml_mtef.py
@@ -32,7 +32,7 @@ def test_ooxml_mtef_backend_analyze_sync_async_contract(file_suffix: str) -> Non
assert isinstance(model, ModelJson)
assert isinstance(middle, MiddleJson)
- assert model.file_suffix == middle.file_suffix == file_suffix
+ assert model.metadata.file_suffix == middle.metadata.file_suffix == file_suffix
assert middle.pages[0].blocks[0].type == BlockType.EQUATION
assert middle.pages[0].blocks[0].content == expected
assert async_model == model
diff --git a/tests/unittest/test_flash_rtf.py b/tests/unittest/test_flash_rtf.py
index c08c3e1e..0981e133 100644
--- a/tests/unittest/test_flash_rtf.py
+++ b/tests/unittest/test_flash_rtf.py
@@ -35,8 +35,8 @@ def test_rtf_doc_analyze_and_renderers_share_strict_metadata() -> None:
aio_doc_analyze(_complex_rtf(), effort="medium", parse_mode="auto", file_suffix="rtf")
)
- assert middle.file_suffix == model.file_suffix == "rtf"
- assert middle.extensions["mineru"]["effort"] == model.extensions["mineru"]["effort"] == "flash"
+ assert middle.metadata.file_suffix == model.metadata.file_suffix == "rtf"
+ assert middle.extensions["mineru"]["tier"] == model.extensions["mineru"]["tier"] == "flash"
assert middle.extensions["mineru"]["parse_mode"] == model.extensions["mineru"]["parse_mode"] == "txt"
assert middle.is_full_document is model.is_full_document is True
assert len(middle.pages) == len(model.pages) == 1
@@ -51,7 +51,7 @@ def test_rtf_doc_analyze_and_renderers_share_strict_metadata() -> None:
assert "Foot body" in markdown
assert "<table" in html
assert docx.startswith(b"PK\x03\x04")
- assert structured["file_suffix"] == "rtf"
+ assert structured["metadata"]["file_suffix"] == "rtf"
def test_public_parser_detects_rtf_content_before_extension(tmp_path: Path) -> None:
@@ -65,7 +65,7 @@ def test_public_parser_detects_rtf_content_before_extension(tmp_path: Path) -> N
parse(source, tier="flash", page_range="99")
assert exc_info.value.code == "page_range_invalid"
- assert result.middle_json.file_suffix == "rtf"
+ assert result.middle_json.metadata.file_suffix == "rtf"
assert result.middle_json.is_full_document is True
assert len(result.pages) == 1
exported = export_middle_json(result.middle_json, tmp_path / "export")
diff --git a/tests/unittest/test_kit_gradio.py b/tests/unittest/test_kit_gradio.py
index 1eb08a62..8dafbbc8 100644
--- a/tests/unittest/test_kit_gradio.py
+++ b/tests/unittest/test_kit_gradio.py
@@ -106,12 +106,19 @@ def _middle_json(
return MiddleJson(
pages=pages,
is_full_document=page_indices == tuple(range(len(page_indices))),
- file_suffix=file_suffix,
- producer=Producer(name="mineru", version=__version__),
- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version=__version__),
+ metadata={"file_suffix": file_suffix, "producer": Producer(name="mineru", version=__version__)},
+ extensions=build_metadata(
+ effort="flash",
+ parse_mode="txt",
+ ),
)
+def _model_json() -> ModelJson:
+ """为真实 API/ZIP 测试提供完整分析协议而不是旧任意字典。"""
+ return ModelJson(pages=[[{"raw": "model"}]], page_index_map=[], metadata=_middle_json().metadata.model_copy(deep=True))
+
+
def _httpx_proxy_for_test_client(
test_client: TestClient,
request_log: list[tuple[str, str, str | None]],
@@ -433,7 +440,7 @@ def test_v1_client_full_asgi_upload_job_poll_and_zip(
async def fake_parse_async(path: str, **kwargs: Any) -> ParseResult:
"""替代模型推理,同时保留 server 的真实 job 和打包流程。"""
parse_calls.append({"path": path, **kwargs})
- return ParseResult(middle_json=_middle_json(with_image=False), _model_output={"raw": "model"})
+ return ParseResult(middle_json=_middle_json(with_image=False), _model_output=_model_json())
monkeypatch.setattr(parser_api_server, "parse_async", fake_parse_async)
api = parser_api_server.create_app(
@@ -458,7 +465,7 @@ def test_v1_client_full_asgi_upload_job_poll_and_zip(
assert any(path.startswith("/v1/files/file-") and path.endswith("/content") for path in paths)
protected_requests = [item for item in request_log if item[1] not in {"/v1/health", "/v1/tiers"}]
assert protected_requests and all(authorization == "Bearer secret" for _method, _path, authorization in protected_requests)
- assert result._model_output == {"raw": "model"}
+ assert result._model_output.pages == [[{"raw": "model"}]]
assert parse_calls[0]["page_range"] == "1"
@@ -471,7 +478,7 @@ def test_gradio_routes_real_v1_jobs_and_isolates_remote_auth(monkeypatch: pytest
async def fake_parse_async(path: str, **kwargs: Any) -> ParseResult:
"""只替换模型推理,保留两套服务各自的上传、任务和 ZIP 打包流程。"""
- return ParseResult(middle_json=_middle_json(with_image=False), _model_output={"raw": "model"})
+ return ParseResult(middle_json=_middle_json(with_image=False), _model_output=_model_json())
monkeypatch.setattr(parser_api_server, "parse_async", fake_parse_async)
remote_api = parser_api_server.create_app(
@@ -493,7 +500,11 @@ def test_gradio_routes_real_v1_jobs_and_isolates_remote_auth(monkeypatch: pytest
headers=dict(request.headers),
content=request.content,
)
- return httpx.Response(response.status_code, headers=response.headers, content=response.content)
+ # TestClient 已解压响应,转交 MockTransport 时同步移除原压缩头。
+ headers = {
+ key: value for key, value in response.headers.items() if key not in {"content-encoding", "content-length"}
+ }
+ return httpx.Response(response.status_code, headers=headers, content=response.content)
def async_client(**kwargs: Any) -> httpx.AsyncClient:
"""为能力发现和 API parser 注入同一双服务传输层。"""
@@ -515,13 +526,13 @@ def test_gradio_routes_real_v1_jobs_and_isolates_remote_auth(monkeypatch: pytest
requests.clear()
flash_result = asyncio.run(routed.parse_file(source, tier="flash", page_range=""))
- assert flash_result._model_output == {"raw": "model"}
+ assert flash_result._model_output.pages == [[{"raw": "model"}]]
assert requests and all(host == "local.test" and auth is None for host, _method, _path, auth in requests)
assert any(method == "POST" and path == "/v1/parse/jobs" for _host, method, path, _auth in requests)
requests.clear()
remote_result = asyncio.run(routed.parse_file(source, tier="standard", page_range="1"))
- assert remote_result._model_output == {"raw": "model"}
+ assert remote_result._model_output.pages == [[{"raw": "model"}]]
assert requests and all(
host == "remote.test" and auth == "Bearer remote-secret" for host, _method, _path, auth in requests
)
@@ -602,10 +613,10 @@ def test_gradio_ocr_reaches_analysis_through_real_v1_jobs(monkeypatch: pytest.Mo
model = ModelJson(
pages=[[]],
page_index_map=[],
- file_suffix="pdf",
- producer=Producer(name="mineru", version=__version__),
+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version=__version__)},
extensions=build_metadata(
- effort="flash", parse_mode=middle.extensions["mineru"]["parse_mode"], mineru_version=__version__
+ effort="flash",
+ parse_mode=middle.extensions["mineru"]["parse_mode"],
),
)
return middle, model
@@ -629,7 +640,7 @@ def test_gradio_ocr_reaches_analysis_through_real_v1_jobs(monkeypatch: pytest.Mo
state = updates[-1][8]
assert state is not None, updates[-1][0]
payload = json.loads(Path(state["middle_json_path"]).read_text())
- assert payload["parse_mode"] == ("ocr" if enabled else "txt")
+ assert payload["extensions"]["mineru"]["parse_mode"] == ("ocr" if enabled else "txt")
default_parser = parser_api_client.MinerUApiParser(api_url="http://testserver", tier="flash")
result = await default_parser.parse_async(source)
assert result.middle_json.extensions["mineru"]["parse_mode"] == "txt"
@@ -834,7 +845,7 @@ def test_persist_and_render_all_gradio_download_formats(tmp_path: Path) -> None:
"""验证严格 Middle JSON、布局 PDF 和六种下载产物。"""
source = tmp_path / "报告.pdf"
source.write_bytes(_pdf_bytes())
- result = ParseResult(middle_json=_middle_json(), _model_output={"raw": "model"})
+ result = ParseResult(middle_json=_middle_json(), _model_output=_model_json())
artifacts = persist_parse_result(result, source, output_root=tmp_path / "output", page_range="")
assert artifacts.middle_json_path.read_text(encoding="utf-8").find('"schema_version": "2.0"') >= 0
diff --git a/tests/unittest/test_legacy_schema_adapter.py b/tests/unittest/test_legacy_schema_adapter.py
index bd7229c6..fc749e96 100644
--- a/tests/unittest/test_legacy_schema_adapter.py
+++ b/tests/unittest/test_legacy_schema_adapter.py
@@ -1,387 +1,32 @@
-from docvortex.schema import Producer
-from mineru.integrations.docvortex import build_metadata
-from copy import deepcopy
+"""旧协议仅作为明确拒绝的负例保留,不再执行历史块迁移。"""
import pytest
-
-from mineru.backend.postprocess.legacy_schema_adapter import legacy_page_to_model_list
-from docvortex.postprocess.pages import model_json_to_pages
-from mineru.doclib.background.compaction import _normalize_batch_pages
-from mineru.types import BlockType, ModelJson
-
-
-def _legacy_line(content: str, *, bbox: list[int] | None = None) -> dict:
- """构造 3.4.5 普通文字行。"""
- return {
- "bbox": bbox or [10, 10, 90, 20],
- "spans": [{"type": "text", "content": content}],
- }
-
-
-def _current_pages(raw_pages: list[list[dict]]) -> list:
- """把适配后的 raw pages 送入当前严格后处理。"""
- model_json = ModelJson(
- pages=raw_pages,
- page_index_map=[],
- file_suffix="pdf",
- producer=Producer(name="mineru", version="3.4.5"),
- extensions=build_metadata(effort="medium", parse_mode="txt", mineru_version="3.4.5"),
- )
- return model_json_to_pages(model_json)
-
-
-def test_legacy_adapter_converts_inline_spans_without_mutating_input() -> None:
- """验证文字标签、公式、链接及危险链接降级,并保证输入不变。"""
- page = {
- "page_idx": 0,
- "page_size": [100, 100],
- "preproc_blocks": [
- {
- "type": "text",
- "bbox": [10, 10, 90, 30],
- "lines": [
- {
- "bbox": [10, 10, 90, 30],
- "spans": [
- {
- "type": "text",
- "content": (
- 'plain <text style="bold,italic">styled</text> '
- "<eq>x^2</eq> "
- "<hyperlink><text>safe</text><url>https://example.com</url></hyperlink> "
- "<hyperlink><text>unsafe</text><url>javascript:alert(1)</url></hyperlink>"
- ),
- }
- ],
- }
- ],
- }
- ],
- "discarded_blocks": [],
- }
- original = deepcopy(page)
-
- raw_page = legacy_page_to_model_list(page)
- content = raw_page[0]["content"]
-
- assert page == original
- assert content == [
- {"type": "text", "content": "plain ", "styles": []},
- {"type": "text", "content": "styled", "styles": ["bold", "italic"]},
- {"type": "text", "content": " ", "styles": []},
- {"type": "equation_inline", "content": "x^2"},
- {"type": "text", "content": " ", "styles": []},
- {
- "type": "hyperlink",
- "url": "https://example.com",
- "content": [{"type": "text", "content": "safe", "styles": []}],
- },
- {"type": "text", "content": " unsafe", "styles": []},
- ]
- pages = _current_pages([raw_page])
- assert pages[0].blocks[0].type == BlockType.TEXT
-
-
-def test_legacy_adapter_preserves_visual_metadata_and_algorithm_spans() -> None:
- """验证视觉父块展平后能由当前后处理恢复表格与算法块。"""
- page = {
- "page_idx": 0,
- "page_size": [100, 100],
- "para_blocks": [
- {
- "type": "table",
- "bbox": [10, 10, 90, 60],
- "cell_merge": [0, 1],
- "blocks": [
- {
- "type": "table_caption",
- "bbox": [10, 10, 90, 20],
- "lines": [_legacy_line("表 1")],
- },
- {
- "type": "table_body",
- "bbox": [10, 20, 90, 60],
- "lines": [
- {
- "bbox": [10, 20, 90, 60],
- "spans": [
- {
- "type": "table",
- "html": "<table><tr><td>x</td></tr></table>",
- "image_path": "images/table.png",
- }
- ],
- }
- ],
- },
- ],
- },
- {
- "type": "code",
- "sub_type": "algorithm",
- "bbox": [10, 70, 90, 90],
- "blocks": [
- {
- "type": "code_body",
- "bbox": [10, 70, 90, 90],
- "lines": [
- {
- "bbox": [10, 70, 90, 90],
- "spans": [
- {"type": "text", "content": "a="},
- {"type": "inline_equation", "content": "x"},
- ],
- }
- ],
- }
- ],
- },
- ],
- "discarded_blocks": [{"type": "header", "bbox": [10, 1, 90, 5], "lines": [_legacy_line("Header")]}],
- }
-
- pages = _current_pages([legacy_page_to_model_list(page)])
- table, code, header = pages[0].blocks
-
- assert table.type == BlockType.TABLE
- assert table.cell_merge == [0, 1]
- assert table.content[1].image_path == "images/table.png"
- assert code.type == BlockType.CODE
- assert code.sub_type == "algorithm"
- assert code.content[0].type == BlockType.ALGORITHM_BODY
- assert [span.type for span in code.content[0].content] == ["text", "equation_inline"]
- assert header.type == BlockType.HEADER
-
-
-def test_legacy_adapter_normalizes_bbox_and_block_aliases() -> None:
- """验证像素坐标、越界坐标及旧 discriminator 均转换为当前合法值。"""
- page = {
- "page_size": [200, 100],
- "preproc_blocks": [
- {
- "type": "title",
- "level": 1,
- "bbox": [-10, 0, 220, 20],
- "lines": [_legacy_line("Document")],
- },
- {
- "type": "interline_equation",
- "bbox": [0, 30, 100, 50],
- "lines": [
- {
- "bbox": [0, 30, 100, 50],
- "spans": [{"type": "interline_equation", "content": "y=1"}],
- }
- ],
- },
- ],
- "discarded_blocks": [],
- }
-
- raw_page = legacy_page_to_model_list(page)
-
- assert raw_page[0]["type"] == "doc_title"
- assert raw_page[0]["bbox"] == (0.0, 0.0, 1.0, 0.2)
- assert raw_page[1]["type"] == "equation"
- assert raw_page[1]["content"] == "y=1"
- _current_pages([raw_page])
-
-
-def test_legacy_adapter_keeps_only_final_middle_json_aliases() -> None:
- """验证最终 3.4.5 仍会序列化的三个旧类型继续转换。"""
- page = {
- "page_size": [100, 100],
- "preproc_blocks": [
- {"type": "abstract", "bbox": [0, 0, 100, 20], "lines": [_legacy_line("Abstract")]},
- {"type": "phonetic", "bbox": [0, 30, 100, 50], "lines": [_legacy_line("phonetic")]},
- {
- "type": "interline_equation",
- "bbox": [0, 60, 100, 80],
- "lines": [
- {
- "bbox": [0, 60, 100, 80],
- "spans": [{"type": "interline_equation", "content": "x=1"}],
- }
- ],
- },
- ],
- "discarded_blocks": [],
- }
-
- raw_page = legacy_page_to_model_list(page)
-
- assert [block["type"] for block in raw_page] == ["text", "text", "equation"]
- assert raw_page[0]["content"][0]["content"] == "Abstract"
- assert raw_page[1]["content"][0]["content"] == "phonetic"
- assert raw_page[2]["content"] == "x=1"
- _current_pages([raw_page])
+from mineru.parser import ParseResult
@pytest.mark.parametrize(
- "block_type",
+ "payload",
[
- "vertical_text",
- "formula_number",
- "discarded",
- "image_block",
- "algorithm",
- "algorithm_body",
- "algorithm_caption",
- "algorithm_footnote",
- "header_image",
- "footer_image",
+ {"pdf_info": [{"page_idx": 0, "para_blocks": []}]},
+ {"schema_version": "1.0", "pages": [{"page_idx": 0, "blocks": []}]},
+ {
+ "schema_version": "2.0",
+ "pages": [],
+ "file_suffix": "pdf",
+ "effort": "flash",
+ "parse_mode": "txt",
+ "mineru_version": "3.4.5",
+ },
+ {
+ "schema": "docvortex.middle",
+ "schema_version": "1.0",
+ "pages": [],
+ "file_suffix": "html",
+ "producer": {"name": "docvortex", "version": "0.1.0"},
+ },
],
)
-def test_legacy_adapter_rejects_pre_middle_json_block_types(block_type: str) -> None:
- """验证 finalize 前布局、raw model 与 Content List 类型要求源文件重解析。"""
- block = {
- "type": block_type,
- "bbox": [0, 0, 100, 20],
- "lines": [_legacy_line("internal")],
- }
- if block_type in {"image_block", "algorithm"}:
- block["blocks"] = [{"type": "text", "bbox": [0, 0, 100, 20], "lines": [_legacy_line("nested")]}]
- page = {
- "page_size": [100, 100],
- "preproc_blocks": [block],
- "discarded_blocks": [],
- }
-
- with pytest.raises(ValueError, match=rf"{block_type!s}.*source reparse"):
- legacy_page_to_model_list(page)
-
-
-def test_legacy_adapter_preserves_canonical_image_and_auxiliary_blocks() -> None:
- """验证规范 image 视觉块及 header/footer 辅助块不会提升为正文图片。"""
- page = {
- "page_size": [100, 100],
- "preproc_blocks": [
- {
- "type": "image",
- "bbox": [10, 20, 90, 80],
- "blocks": [
- {
- "type": "image_body",
- "bbox": [10, 20, 90, 80],
- "lines": [
- {
- "bbox": [10, 20, 90, 80],
- "spans": [{"type": "image", "image_path": "images/body.png"}],
- }
- ],
- }
- ],
- }
- ],
- "discarded_blocks": [
- {"type": "header", "bbox": [0, 0, 100, 10], "lines": [_legacy_line("Header")]},
- {"type": "footer", "bbox": [0, 90, 100, 100], "lines": [_legacy_line("Footer")]},
- ],
- }
-
- pages = _current_pages([legacy_page_to_model_list(page)])
-
- assert [block.type for block in pages[0].blocks] == [BlockType.IMAGE, BlockType.HEADER, BlockType.FOOTER]
- image = pages[0].blocks[0]
- assert image.content[0].image_path == "images/body.png"
-
-
-def test_legacy_adapter_recovers_pdf_list_items_from_line_flags() -> None:
- """验证旧 PDF list 的物理行重新生成当前列表文本子块。"""
- page = {
- "page_size": [100, 100],
- "preproc_blocks": [
- {
- "type": "list",
- "bbox": [10, 10, 90, 40],
- "lines": [
- {
- "bbox": [10, 10, 90, 20],
- "is_list_start_line": True,
- "spans": [{"type": "text", "content": "one"}],
- },
- {
- "bbox": [10, 25, 90, 35],
- "is_list_start_line": True,
- "spans": [{"type": "text", "content": "two"}],
- },
- ],
- }
- ],
- "discarded_blocks": [],
- }
-
- pages = _current_pages([legacy_page_to_model_list(page)])
- list_block = pages[0].blocks[0]
-
- assert list_block.type == BlockType.LIST
- assert [child.content[0].content for child in list_block.content] == ["one", "two"]
-
-
-def test_legacy_adapter_separates_office_blocks_with_synthetic_bbox_groups() -> None:
- """验证无 page_size 时列表不会吸收相邻 Office 正文块。"""
- page = {
- "para_blocks": [
- {"type": "text", "lines": [{"spans": [{"type": "text", "content": "before"}]}]},
- {
- "type": "list",
- "blocks": [
- {"type": "text", "lines": [{"spans": [{"type": "text", "content": "item"}]}]},
- ],
- },
- {"type": "text", "lines": [{"spans": [{"type": "text", "content": "after"}]}]},
- ],
- "discarded_blocks": [],
- }
-
- pages = _current_pages([legacy_page_to_model_list(page)])
-
- assert [block.type for block in pages[0].blocks] == [BlockType.TEXT, BlockType.LIST, BlockType.TEXT]
- assert pages[0].blocks[0].content[0].content == "before"
- assert pages[0].blocks[1].content[0].content[0].content == "item"
- assert pages[0].blocks[2].content[0].content == "after"
-
-
-def test_doclib_compaction_converts_schema_v1_pages_with_current_adapter() -> None:
- """验证 Doclib 历史调用点重新通过宿主适配器生成 2.0 page 字典。"""
- payload = {
- "schema_version": "1.0",
- "pages": [
- {
- "page_idx": 3,
- "page_size": [100, 100],
- "preproc_blocks": [
- {
- "type": "text",
- "bbox": [10, 10, 90, 20],
- "lines": [_legacy_line("cached")],
- }
- ],
- "discarded_blocks": [],
- }
- ],
- }
-
- pages = _normalize_batch_pages(payload)
-
- assert pages[0]["page_idx"] == 3
- assert pages[0]["blocks"][0]["content"] == [{"type": "text", "content": "cached"}]
-
-
-def test_doclib_compaction_rejects_unknown_legacy_schema() -> None:
- """验证未知 schema 不会被误当成 3.4.5 页面吞掉。"""
- with pytest.raises(ValueError, match="source reparse"):
- _normalize_batch_pages({"schema_version": "99.0", "pages": [{"page_idx": 0, "blocks": []}]})
-
-
-def test_doclib_compaction_accepts_current_schema_without_legacy_conversion(monkeypatch: pytest.MonkeyPatch) -> None:
- """验证当前 2.0 缓存直接读取页面,不进入旧版转换。"""
- from mineru.backend.postprocess import legacy_schema_adapter
-
- def unexpected_conversion(_page: dict) -> list[dict]:
- """当前缓存触发旧版转换时立即报告路由错误。"""
- raise AssertionError("current schema must not use legacy conversion")
-
- monkeypatch.setattr(legacy_schema_adapter, "legacy_page_to_model_list", unexpected_conversion)
- pages = [{"page_idx": 0, "blocks": []}]
- assert _normalize_batch_pages({"schema_version": "2.0", "pages": pages}) is pages
+def test_old_document_protocols_require_reparse(payload: dict) -> None:
+ """拒绝旧 DocVortex、MinerU 及无协议标识的数据,不静默生成空页。"""
+ with pytest.raises(ValueError, match="reparse"):
+ ParseResult.from_dict(payload)
diff --git a/tests/unittest/test_llm_aided_postprocess.py b/tests/unittest/test_llm_aided_postprocess.py
index 4ddca4ad..0e1ad65f 100644
--- a/tests/unittest/test_llm_aided_postprocess.py
+++ b/tests/unittest/test_llm_aided_postprocess.py
@@ -213,9 +213,11 @@ def _middle_json(pages: list[PageInfo], *, is_full_document: bool = True) -> Mid
return MiddleJson(
pages=pages,
is_full_document=is_full_document,
- file_suffix="docx",
- producer=Producer(name="mineru", version="test"),
- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="test"),
+ metadata={"file_suffix": "docx", "producer": Producer(name="mineru", version="test")},
+ extensions=build_metadata(
+ effort="flash",
+ parse_mode="txt",
+ ),
)
@@ -528,9 +530,11 @@ def test_mixed_cell_merge_flows_through_strict_middle_json_and_renderer() -> Non
middle_json = MiddleJson(
pages=pages,
is_full_document=True,
- file_suffix="pdf",
- producer=Producer(name="mineru", version="test"),
- extensions=build_metadata(effort="high", parse_mode="txt", mineru_version="test"),
+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version="test")},
+ extensions=build_metadata(
+ effort="high",
+ parse_mode="txt",
+ ),
)
assert _merged_row_texts(middle_json) == [["H1", "H2"], ["AB", "X"], ["", "Y"]]
@@ -762,6 +766,6 @@ def test_doc_analyze_does_not_run_llm_for_office(monkeypatch: pytest.MonkeyPatch
middle_json, _ = analyze.doc_analyze(b"office", file_suffix="pptx")
- assert middle_json.file_suffix == "pptx"
+ assert middle_json.metadata.file_suffix == "pptx"
assert middle_json.is_full_document is True
assert calls == []
diff --git a/tests/unittest/test_markdown_render.py b/tests/unittest/test_markdown_render.py
index 8caffd52..ad7b2cc7 100644
--- a/tests/unittest/test_markdown_render.py
+++ b/tests/unittest/test_markdown_render.py
@@ -20,9 +20,11 @@ def _middle(*pages: PageInfo, file_suffix: str = "docx") -> MiddleJson:
return MiddleJson(
pages=list(pages),
is_full_document=True,
- file_suffix=file_suffix,
- producer=Producer(name="mineru", version="test"),
- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="test"),
+ metadata={"file_suffix": file_suffix, "producer": Producer(name="mineru", version="test")},
+ extensions=build_metadata(
+ effort="flash",
+ parse_mode="txt",
+ ),
)
diff --git a/tests/unittest/test_model_json.py b/tests/unittest/test_model_json.py
index 19d0685a..3603e4cc 100644
--- a/tests/unittest/test_model_json.py
+++ b/tests/unittest/test_model_json.py
@@ -12,13 +12,8 @@ from mineru import ModelJson as PublicModelJson
from mineru.backend.postprocess import document
from mineru.backend.postprocess.document import model_json_to_middle_json
from mineru.config import LLMAidedConfig
-from mineru.integrations.docvortex import (
- build_metadata,
- from_mineru_middle,
- from_mineru_model,
- to_mineru_middle,
- to_mineru_model,
-)
+from mineru.integrations.docvortex import build_metadata
+
from mineru.types import MiddleJson, ModelJson
@@ -31,9 +26,11 @@ def _model_json(
return ModelJson(
pages=pages if pages is not None else [[{"type": "text", "content": inline("正文")}]],
page_index_map=page_index_map if page_index_map is not None else [],
- file_suffix="docx",
- producer=Producer(name="mineru", version="3.4.0"),
- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="3.4.0"),
+ metadata={"file_suffix": "docx", "producer": Producer(name="mineru", version="3.4.0")},
+ extensions=build_metadata(
+ effort="flash",
+ parse_mode="txt",
+ ),
)
@@ -41,24 +38,17 @@ def test_model_json_is_public_and_serializes_exact_envelope() -> None:
"""验证公开 ModelJson 固定输出六个顶层字段且空映射不会被省略。"""
model_json = _model_json()
- payload = to_mineru_model(model_json)
+ payload = model_json.to_dict()
assert PublicModelJson is ModelJson
- assert list(payload) == [
- "pages",
- "page_index_map",
- "file_suffix",
- "effort",
- "parse_mode",
- "mineru_version",
- ]
+ assert set(payload) == {"schema", "schema_version", "metadata", "extensions", "pages", "page_index_map"}
assert payload == {
"pages": [[{"type": "text", "content": inline("正文")}]],
"page_index_map": [],
- "file_suffix": "docx",
- "effort": "flash",
- "parse_mode": "txt",
- "mineru_version": "3.4.0",
+ "metadata": {"file_suffix": "docx", "producer": {"name": "mineru", "version": "3.4.0"}},
+ "schema": "docvortex.model",
+ "schema_version": "2.0",
+ "extensions": {"mineru": {"tier": "flash", "parse_mode": "txt"}},
}
assert model_json.is_full_document is True
assert model_json.resolved_page_indices == [0]
@@ -71,43 +61,16 @@ def test_model_json_requires_page_index_map_and_forbids_extra_fields() -> None:
"""验证页映射不可省略且 ModelJson 顶层不接受未声明字段。"""
payload = {
"pages": [],
- "file_suffix": "pdf",
- "effort": "flash",
- "parse_mode": "txt",
- "mineru_version": "3.4.0",
+ "metadata": {"file_suffix": "pdf", "producer": {"name": "mineru", "version": "3.4.0"}},
+ "schema": "docvortex.model",
+ "schema_version": "2.0",
+ "extensions": {"mineru": {"tier": "flash", "parse_mode": "txt"}},
}
with pytest.raises(ValidationError, match="page_index_map"):
- from_mineru_model(payload)
+ ModelJson.from_dict(payload)
with pytest.raises(ValidationError, match="extra_forbidden"):
- from_mineru_model({**payload, "page_index_map": [], "unexpected": True})
-
-
-def test_strict_document_models_reject_removed_low_effort() -> None:
- """验证 ModelJson 与 MiddleJson 的 schema 2.0 均不再接受 Low effort。"""
- with pytest.raises(ValidationError, match="literal_error"):
- from_mineru_model(
- {
- "pages": [],
- "page_index_map": [],
- "file_suffix": "pdf",
- "effort": "low",
- "parse_mode": "ocr",
- "mineru_version": "3.4.0",
- }
- )
-
- with pytest.raises(ValidationError, match="literal_error"):
- from_mineru_middle(
- {
- "pages": [],
- "is_full_document": True,
- "file_suffix": "pdf",
- "effort": "low",
- "parse_mode": "ocr",
- "mineru_version": "3.4.0",
- }
- )
+ ModelJson.from_dict({**payload, "page_index_map": [], "unexpected": True})
def test_middle_json_requires_and_serializes_full_document_flag() -> None:
@@ -115,21 +78,16 @@ def test_middle_json_requires_and_serializes_full_document_flag() -> None:
middle_json = MiddleJson(
pages=[],
is_full_document=False,
- file_suffix="pdf",
- producer=Producer(name="mineru", version="3.4.0"),
- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="3.4.0"),
+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version="3.4.0")},
+ extensions=build_metadata(
+ effort="flash",
+ parse_mode="txt",
+ ),
)
- payload = to_mineru_middle(middle_json, include_schema_version=False)
+ payload = middle_json.to_dict()
- assert list(payload) == [
- "pages",
- "is_full_document",
- "file_suffix",
- "effort",
- "parse_mode",
- "mineru_version",
- ]
+ assert set(payload) == {"schema", "schema_version", "metadata", "extensions", "pages", "is_full_document"}
assert payload["is_full_document"] is False
assert load_middle(json.loads(middle_json.to_json())) == middle_json
@@ -141,9 +99,11 @@ def test_model_json_to_middle_json_builds_strict_document_before_pdf_llm(
model_json = ModelJson(
pages=[[]],
page_index_map=[3],
- file_suffix="pdf",
- producer=Producer(name="mineru", version="3.4.0"),
- extensions=build_metadata(effort="xhigh", parse_mode="ocr", mineru_version="3.4.0"),
+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version="3.4.0")},
+ extensions=build_metadata(
+ effort="xhigh",
+ parse_mode="ocr",
+ ),
)
observed: list[MiddleJson] = []
@@ -158,10 +118,10 @@ def test_model_json_to_middle_json_builds_strict_document_before_pdf_llm(
middle_json = model_json_to_middle_json(model_json, llm_aided_config=LLMAidedConfig())
assert observed == [middle_json]
- assert middle_json.file_suffix == "pdf"
- assert middle_json.extensions["mineru"]["effort"] == "xhigh"
+ assert middle_json.metadata.file_suffix == "pdf"
+ assert middle_json.extensions["mineru"]["tier"] == "advanced"
assert middle_json.extensions["mineru"]["parse_mode"] == "ocr"
- assert middle_json.extensions["mineru"]["mineru_version"] == "3.4.0"
+ assert middle_json.metadata.producer.version == "3.4.0"
@pytest.mark.parametrize("invalid_value", [None, 0, 1, "true"])
@@ -169,13 +129,13 @@ def test_middle_json_rejects_missing_or_non_boolean_full_document_flag(invalid_v
"""验证 MiddleJson 不为整本语义提供缺省值或宽松布尔转换。"""
payload = {
"pages": [],
- "file_suffix": "pdf",
- "effort": "flash",
- "parse_mode": "txt",
- "mineru_version": "3.4.0",
+ "metadata": {"file_suffix": "pdf", "producer": {"name": "mineru", "version": "3.4.0"}},
+ "schema": "docvortex.middle",
+ "schema_version": "2.0",
+ "extensions": {"mineru": {"tier": "flash", "parse_mode": "txt"}},
}
if invalid_value is not None:
payload["is_full_document"] = invalid_value
with pytest.raises(ValidationError, match="is_full_document"):
- from_mineru_middle(payload)
+ MiddleJson.from_dict(payload)
diff --git a/tests/unittest/test_parse_result_contract.py b/tests/unittest/test_parse_result_contract.py
index fa0c5b07..3e99f2b2 100644
--- a/tests/unittest/test_parse_result_contract.py
+++ b/tests/unittest/test_parse_result_contract.py
@@ -43,9 +43,11 @@ def test_parse_result_from_dict_restores_pages() -> None:
)
],
is_full_document=True,
- file_suffix="pdf",
- producer=Producer(name="mineru", version=__version__),
- extensions=build_metadata(effort="medium", parse_mode="txt", mineru_version=__version__),
+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version=__version__)},
+ extensions=build_metadata(
+ effort="medium",
+ parse_mode="txt",
+ ),
)
)
@@ -62,9 +64,11 @@ def test_parse_result_to_dict_includes_schema_version_without_meta() -> None:
middle_json=MiddleJson(
pages=[PageInfo(page_idx=0)],
is_full_document=True,
- file_suffix="pdf",
- producer=Producer(name="mineru", version=__version__),
- extensions=build_metadata(effort="medium", parse_mode="txt", mineru_version=__version__),
+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version=__version__)},
+ extensions=build_metadata(
+ effort="medium",
+ parse_mode="txt",
+ ),
)
)
@@ -93,9 +97,11 @@ def test_parse_result_roundtrip_preserves_page_footnote_anchor() -> None:
)
],
is_full_document=True,
- file_suffix="epub",
- producer=Producer(name="mineru", version=__version__),
- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version=__version__),
+ metadata={"file_suffix": "epub", "producer": Producer(name="mineru", version=__version__)},
+ extensions=build_metadata(
+ effort="flash",
+ parse_mode="txt",
+ ),
)
)
@@ -114,21 +120,20 @@ def test_parse_result_rejects_low_effort() -> None:
"schema_version": MIDDLE_JSON_SCHEMA_VERSION,
"pages": [],
"is_full_document": True,
- "file_suffix": "pdf",
- "effort": "low",
- "parse_mode": "ocr",
- "mineru_version": __version__,
+ "metadata": {"file_suffix": "pdf", "producer": {"name": "mineru", "version": __version__}},
+ "schema": "docvortex.middle",
+ "extensions": {"mineru": {"tier": "low", "parse_mode": "ocr"}},
}
)
def test_parse_result_from_dict_rejects_missing_pages() -> None:
with pytest.raises(ValueError, match="pages"):
- ParseResult.from_dict({"schema_version": MIDDLE_JSON_SCHEMA_VERSION})
+ ParseResult.from_dict({"schema": "docvortex.middle", "schema_version": MIDDLE_JSON_SCHEMA_VERSION})
-def test_parse_result_from_json_converts_mineru_3_4_5_middle_json() -> None:
- """验证 ParseResult 恢复 3.4.5 pdf_info 调用链并生成严格 2.0。"""
+def test_parse_result_from_json_rejects_mineru_3_4_5_middle_json() -> None:
+ """验证 ParseResult 的真实 JSON 入口拒绝历史 pdf_info 文档。"""
data = {
"_backend": "hybrid",
"_effort": "high",
@@ -156,22 +161,14 @@ def test_parse_result_from_json_converts_mineru_3_4_5_middle_json() -> None:
],
}
- restored = ParseResult.from_json(json.dumps(data))
+ with pytest.raises(ValueError, match="reparse"):
+ ParseResult.from_json(json.dumps(data))
- assert restored.pages[0].page_idx == 2
- assert restored.pages[0].blocks[0].content[0].content == "round trip"
- assert restored.middle_json.is_full_document is False
- assert restored.middle_json.extensions["mineru"]["effort"] == "high"
- assert restored.middle_json.extensions["mineru"]["parse_mode"] == "ocr"
- assert restored.middle_json.extensions["mineru"]["mineru_version"] == "3.4.4"
-
-def test_parse_result_accepts_schema_v1_page_wrapper() -> None:
- """验证旧调用方的 1.0 pages envelope 继续委托同一适配器。"""
- restored = ParseResult.from_dict({"schema_version": "1.0", "pages": []})
-
- assert restored.pages == []
- assert restored.middle_json.is_full_document is True
+def test_parse_result_rejects_schema_v1_page_wrapper() -> None:
+ """验证旧 1.0 pages 封装明确拒绝,不生成虚假的空文档。"""
+ with pytest.raises(ValueError, match="reparse"):
+ ParseResult.from_dict({"schema_version": "1.0", "pages": []})
@pytest.mark.parametrize("schema_version", [None, "3.0"])
@@ -181,7 +178,7 @@ def test_parse_result_rejects_legacy_schema_versions(schema_version: str | None)
if schema_version is not None:
payload["schema_version"] = schema_version
- with pytest.raises(ValueError, match="Reparse the source document"):
+ with pytest.raises(ValueError, match="reparse the source document"):
ParseResult.from_dict(payload)
@@ -210,9 +207,11 @@ def test_parse_result_export_pages_returns_defensive_copy() -> None:
middle_json=MiddleJson(
pages=[page],
is_full_document=True,
- file_suffix="pdf",
- producer=Producer(name="mineru", version=__version__),
- extensions=build_metadata(effort="medium", parse_mode="txt", mineru_version=__version__),
+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version=__version__)},
+ extensions=build_metadata(
+ effort="medium",
+ parse_mode="txt",
+ ),
),
)
first_export = result.export_pages()
diff --git a/tests/unittest/test_parser_api_contract.py b/tests/unittest/test_parser_api_contract.py
index cb5e05f9..ab004a1f 100644
--- a/tests/unittest/test_parser_api_contract.py
+++ b/tests/unittest/test_parser_api_contract.py
@@ -42,6 +42,7 @@ from mineru.parser.api_server import (
)
from mineru.parser.base import ParseResult
from mineru.types import (
+ ModelJson,
DEFAULT_QUALITY_TIER_SELECTION_ORDER,
DEPLOYMENT_TIERS,
DeploymentTier,
@@ -84,22 +85,34 @@ def _full_middle_json(*pages: PageInfo) -> MiddleJson:
return MiddleJson(
pages=list(pages),
is_full_document=True,
- file_suffix="pdf",
- producer=Producer(name="mineru", version=__version__),
- extensions=build_metadata(effort="medium", parse_mode="txt", mineru_version=__version__),
+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version=__version__)},
+ extensions=build_metadata(
+ effort="medium",
+ parse_mode="txt",
+ ),
)
+def _model_payload() -> dict[str, Any]:
+ """构造 ZIP 内的完整分析协议,保留原始模型块用于传输回归。"""
+ from mineru.types import ModelJson
+
+ return ModelJson(
+ pages=[[{"raw": "model"}]],
+ page_index_map=[],
+ metadata={"file_suffix": "pdf", "producer": {"name": "mineru", "version": __version__}},
+ ).to_dict()
+
+
def _current_payload(pages: list[dict[str, Any]], **extra: Any) -> dict[str, Any]:
"""构造严格 Middle JSON 2.0 API envelope。"""
payload: dict[str, Any] = {
"schema_version": MIDDLE_JSON_SCHEMA_VERSION,
"pages": pages,
"is_full_document": True,
- "file_suffix": "pdf",
- "effort": "medium",
- "parse_mode": "txt",
- "mineru_version": __version__,
+ "metadata": {"file_suffix": "pdf", "producer": {"name": "mineru", "version": __version__}},
+ "schema": "docvortex.middle",
+ "extensions": {"mineru": {"tier": "basic", "parse_mode": "txt"}},
}
payload.update(extra)
return payload
@@ -756,7 +769,7 @@ def test_api_client_downloads_model_output_from_zip(monkeypatch: pytest.MonkeyPa
archive.writestr("middle_json.json", json.dumps(middle_json, ensure_ascii=False))
archive.writestr(
"model_output.json",
- json.dumps([[{"raw": "model"}]], ensure_ascii=False, indent=4),
+ json.dumps(_model_payload(), ensure_ascii=False, indent=4),
)
monkeypatch.setattr(
@@ -775,11 +788,11 @@ def test_api_client_downloads_model_output_from_zip(monkeypatch: pytest.MonkeyPa
parser,
)
- assert result._model_output == [[{"raw": "model"}]]
+ assert result._model_output.pages == [[{"raw": "model"}]]
-def test_api_client_accepts_legacy_official_layout_json(monkeypatch: pytest.MonkeyPatch) -> None:
- """官方 API zip 中的 legacy pdf_info layout.json 应经 legacy_schema_adapter 迁移为 3.0。"""
+def test_api_client_rejects_legacy_official_layout_json(monkeypatch: pytest.MonkeyPatch) -> None:
+ """官方 API ZIP 内的历史 pdf_info 文档必须明确拒绝,不再迁移正文。"""
parser = MinerUApiParser(
api_url="https://mineru.net/api",
tier="standard",
@@ -813,24 +826,20 @@ def test_api_client_accepts_legacy_official_layout_json(monkeypatch: pytest.Monk
with zipfile.ZipFile(zip_buffer, "w", zipfile.ZIP_DEFLATED) as archive:
archive.writestr("layout.json", json.dumps(middle_json, ensure_ascii=False))
archive.writestr("images/chart.png", b"chart-bytes")
- archive.writestr("ab3a55b0-2017-4e35-8507-ab8e2c012160_model.json", json.dumps([[{"raw": "model"}]]))
+ archive.writestr("ab3a55b0-2017-4e35-8507-ab8e2c012160_model.json", json.dumps(_model_payload()))
monkeypatch.setattr(api_client, "_download_bytes", lambda _parser, ref: zip_buffer.getvalue() if ref is zip_ref else b"")
- result = _parse_result_from_job(
- {
- "job_id": "job_1",
- "status": "completed",
- "files": [{"output_files": {"zip": zip_ref}}],
- },
- "demo.pdf",
- parser,
- )
-
- assert len(result.pages) == 1
- assert result.pages[0].blocks[0].type.value == "image"
- image_body = result.pages[0].blocks[0].content[0]
- assert image_body.image_base64 == "data:image/png;base64," + base64.b64encode(b"chart-bytes").decode("ascii") # type: ignore[union-attr]
+ with pytest.raises(api_client._V1APIError, match="reparse"):
+ _parse_result_from_job(
+ {
+ "job_id": "job_1",
+ "status": "completed",
+ "files": [{"output_files": {"zip": zip_ref}}],
+ },
+ "demo.pdf",
+ parser,
+ )
def test_api_client_async_downloads_model_output_from_zip(monkeypatch: pytest.MonkeyPatch) -> None:
@@ -842,7 +851,7 @@ def test_api_client_async_downloads_model_output_from_zip(monkeypatch: pytest.Mo
archive.writestr("middle_json.json", json.dumps(middle_json, ensure_ascii=False))
archive.writestr(
"model_output.json",
- json.dumps([[{"raw": "model"}]], ensure_ascii=False, indent=4),
+ json.dumps(_model_payload(), ensure_ascii=False, indent=4),
)
async def fake_download_bytes(_parser: object, ref: dict[str, object]) -> bytes:
@@ -862,7 +871,7 @@ def test_api_client_async_downloads_model_output_from_zip(monkeypatch: pytest.Mo
)
)
- assert result._model_output == [[{"raw": "model"}]]
+ assert result._model_output.pages == [[{"raw": "model"}]]
def test_api_client_include_images_downloads_single_zip(monkeypatch: pytest.MonkeyPatch) -> None:
@@ -898,7 +907,7 @@ def test_api_client_include_images_downloads_single_zip(monkeypatch: pytest.Monk
),
)
archive.writestr("images/chart.png", b"chart-bytes")
- archive.writestr("model_output.json", json.dumps([[{"raw": "model"}]], ensure_ascii=False, indent=4))
+ archive.writestr("model_output.json", json.dumps(_model_payload(), ensure_ascii=False, indent=4))
download_calls: list[dict[str, object]] = []
def fake_download_bytes(_parser: object, ref: dict[str, object]) -> bytes:
@@ -926,7 +935,7 @@ def test_api_client_include_images_downloads_single_zip(monkeypatch: pytest.Monk
assert result.pages[0].page_idx == 0
image_body = result.pages[0].blocks[0].content[0]
assert image_body.image_base64 == "data:image/png;base64," + base64.b64encode(b"chart-bytes").decode("ascii") # type: ignore[union-attr]
- assert result._model_output == [[{"raw": "model"}]]
+ assert result._model_output.pages == [[{"raw": "model"}]]
def test_api_client_does_not_read_image_cache_from_zip_unless_requested(monkeypatch: pytest.MonkeyPatch) -> None:
@@ -937,7 +946,7 @@ def test_api_client_does_not_read_image_cache_from_zip_unless_requested(monkeypa
with zipfile.ZipFile(zip_buffer, "w", zipfile.ZIP_DEFLATED) as archive:
archive.writestr("middle_json.json", json.dumps(middle_json, ensure_ascii=False))
archive.writestr("images/chart.png", b"chart-bytes")
- archive.writestr("model_output.json", json.dumps([[{"raw": "model"}]], ensure_ascii=False, indent=4))
+ archive.writestr("model_output.json", json.dumps(_model_payload(), ensure_ascii=False, indent=4))
monkeypatch.setattr(api_client, "_download_bytes", lambda _parser, ref: zip_buffer.getvalue() if ref is zip_ref else b"")
@@ -954,7 +963,7 @@ def test_api_client_does_not_read_image_cache_from_zip_unless_requested(monkeypa
image_body = result.pages[0].blocks[0].content[0]
assert image_body.image_base64 is None # type: ignore[union-attr]
assert image_body.image_path == "images/chart.png" # type: ignore[union-attr]
- assert result._model_output == [[{"raw": "model"}]]
+ assert result._model_output.pages == [[{"raw": "model"}]]
def test_api_client_async_include_images_downloads_single_zip(monkeypatch: pytest.MonkeyPatch) -> None:
@@ -971,7 +980,7 @@ def test_api_client_async_include_images_downloads_single_zip(monkeypatch: pytes
"middle_json.json",
json.dumps(_current_payload([{"page_idx": 0, "blocks": []}])),
)
- archive.writestr("model_output.json", json.dumps([[{"raw": "model"}]], ensure_ascii=False, indent=4))
+ archive.writestr("model_output.json", json.dumps(_model_payload(), ensure_ascii=False, indent=4))
download_calls: list[dict[str, object]] = []
async def fake_download_bytes(_parser: object, ref: dict[str, object]) -> bytes:
@@ -998,7 +1007,7 @@ def test_api_client_async_include_images_downloads_single_zip(monkeypatch: pytes
assert download_calls == [zip_ref]
assert result.pages[0].page_idx == 0
- assert result._model_output == [[{"raw": "model"}]]
+ assert result._model_output.pages == [[{"raw": "model"}]]
def test_api_client_include_images_rejects_unsafe_image_entries(monkeypatch: pytest.MonkeyPatch) -> None:
@@ -1023,8 +1032,8 @@ def test_api_client_include_images_rejects_unsafe_image_entries(monkeypatch: pyt
)
-def test_api_client_accepts_remote_pdf_info_middle_json(monkeypatch: pytest.MonkeyPatch) -> None:
- """remote 官方 API 返回 3.4.5 pdf_info payload 时应经 legacy 分支迁移而不是拒绝。"""
+def test_api_client_rejects_remote_pdf_info_middle_json(monkeypatch: pytest.MonkeyPatch) -> None:
+ """远程返回历史 pdf_info 协议时必须提示重新解析,不猜测其版本。"""
parser = MinerUApiParser(api_url="https://mineru.net/api", tier="standard")
middle_json = {
"_backend": "hybrid",
@@ -1034,22 +1043,20 @@ def test_api_client_accepts_remote_pdf_info_middle_json(monkeypatch: pytest.Monk
monkeypatch.setattr(api_client, "_download_json", lambda _parser, _outputs: middle_json)
- result = _parse_result_from_job(
- {
- "job_id": "job_1",
- "status": "completed",
- "files": [{"output_files": {"middle_json": {"file_id": "file-middle-json", "bytes": 10}}}],
- },
- "demo.pdf",
- parser,
- )
-
- assert len(result.pages) == 1
- assert result.pages[0].page_idx == 0
- assert result.middle_json.extensions["mineru"]["mineru_version"] == "remote"
+ with pytest.raises(api_client._V1APIError, match="reparse"):
+ _parse_result_from_job(
+ {
+ "job_id": "job_1",
+ "status": "completed",
+ "files": [{"output_files": {"middle_json": {"file_id": "file-middle-json", "bytes": 10}}}],
+ },
+ "demo.pdf",
+ parser,
+ )
-def test_async_api_client_accepts_remote_pdf_info_middle_json(monkeypatch: pytest.MonkeyPatch) -> None:
+def test_async_api_client_rejects_remote_pdf_info_middle_json(monkeypatch: pytest.MonkeyPatch) -> None:
+ """异步客户端同样拒绝历史 pdf_info 协议,不把空页当成有效结果。"""
parser = MinerUApiParser(api_url="https://mineru.net/api", tier="standard")
middle_json = {
"_backend": "hybrid",
@@ -1061,20 +1068,18 @@ def test_async_api_client_accepts_remote_pdf_info_middle_json(monkeypatch: pytes
monkeypatch.setattr(api_client, "_async_download_json", _download_json)
- result = asyncio.run(
- api_client._async_parse_result_from_job(
- {
- "job_id": "job_1",
- "status": "completed",
- "files": [{"output_files": {"middle_json": {"file_id": "file-middle-json", "bytes": 10}}}],
- },
- "demo.pdf",
- parser,
+ with pytest.raises(api_client._V1APIError, match="reparse"):
+ asyncio.run(
+ api_client._async_parse_result_from_job(
+ {
+ "job_id": "job_1",
+ "status": "completed",
+ "files": [{"output_files": {"middle_json": {"file_id": "file-middle-json", "bytes": 10}}}],
+ },
+ "demo.pdf",
+ parser,
+ )
)
- )
-
- assert len(result.pages) == 1
- assert result.pages[0].page_idx == 0
def test_api_client_rejects_legacy_json_output_file_key() -> None:
@@ -1549,7 +1554,7 @@ def test_api_client_accepts_middle_json_current_pages() -> None:
)
def test_api_client_rejects_unknown_schema_version(payload: dict[str, object]) -> None:
"""验证缺失版本与未知版本 payload 仍要求重新解析;pdf_info 走 legacy 迁移单独覆盖。"""
- with pytest.raises((ValueError, api_client._V1APIError), match="Reparse the source document"):
+ with pytest.raises((ValueError, api_client._V1APIError), match="reparse the source document"):
_pages_from_middle_json(payload)
@@ -2050,7 +2055,7 @@ def test_api_server_zip_includes_model_output_when_parse_result_has_it(
) -> None:
parse_result = ParseResult(
middle_json=_full_middle_json(PageInfo(page_idx=0)),
- _model_output=model_output,
+ _model_output=ModelJson.from_dict({**_model_payload(), "pages": [model_output]}),
)
async def fake_parse_async(*args: object, **kwargs: object) -> ParseResult:
@@ -2086,7 +2091,7 @@ def test_api_server_zip_includes_model_output_when_parse_result_has_it(
assert "model_output.json" in archive.namelist()
model_output_text = archive.read("model_output.json").decode("utf-8")
payload = json.loads(model_output_text)
- assert payload == model_output
+ assert payload == ModelJson.from_dict({**_model_payload(), "pages": [model_output]}).to_dict(skip_defaults=False)
if model_output:
assert "\n " in model_output_text
diff --git a/tests/unittest/test_parser_api_status.py b/tests/unittest/test_parser_api_status.py
index e79b3ae1..f2ad75e3 100644
--- a/tests/unittest/test_parser_api_status.py
+++ b/tests/unittest/test_parser_api_status.py
@@ -49,9 +49,11 @@ def test_parse_notifies_submission_and_polls_before_result_building(
middle_json=MiddleJson(
pages=[],
is_full_document=True,
- file_suffix="pdf",
- producer=Producer(name="mineru", version="4.0.0"),
- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="4.0.0"),
+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version="4.0.0")},
+ extensions=build_metadata(
+ effort="flash",
+ parse_mode="txt",
+ ),
)
)
diff --git a/tests/unittest/test_pdf_analyze_block_content.py b/tests/unittest/test_pdf_analyze_block_content.py
index ac206633..5b72b364 100644
--- a/tests/unittest/test_pdf_analyze_block_content.py
+++ b/tests/unittest/test_pdf_analyze_block_content.py
@@ -213,9 +213,11 @@ def test_doc_analyze_converts_vlm_results_before_downstream_processing(
expected_middle_json = MiddleJson(
pages=[],
is_full_document=True,
- file_suffix="pdf",
- producer=Producer(name="mineru", version="test"),
- extensions=build_metadata(effort=effort, parse_mode=parse_mode, mineru_version="test"),
+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version="test")},
+ extensions=build_metadata(
+ effort=effort,
+ parse_mode=parse_mode,
+ ),
)
monkeypatch.setattr(postprocess_document, "model_json_to_middle_json", MagicMock(return_value=expected_middle_json))
monkeypatch.setattr(pipeline, "clean_memory", MagicMock())
@@ -232,14 +234,20 @@ def test_doc_analyze_converts_vlm_results_before_downstream_processing(
assert type(model_json.pages[0][0]) is dict
assert model_json.pages == [[{"type": BlockType.PAGE_NUMBER, "bbox": [0.45, 0.9, 0.55, 0.95], "content": inline("1")}]]
assert model_json.page_index_map == []
- assert model_json.file_suffix == "pdf"
- assert model_json.extensions["mineru"]["effort"] == effort
+ assert model_json.metadata.file_suffix == "pdf"
+ assert (
+ model_json.extensions["mineru"]["tier"]
+ == {"flash": "flash", "medium": "basic", "high": "standard", "xhigh": "advanced"}[effort]
+ )
assert model_json.extensions["mineru"]["parse_mode"] == parse_mode
assert isinstance(middle_json, MiddleJson)
assert middle_json is expected_middle_json
assert middle_json.pages == []
assert middle_json.is_full_document is True
- assert middle_json.extensions["mineru"]["effort"] == effort
+ assert (
+ middle_json.extensions["mineru"]["tier"]
+ == {"flash": "flash", "medium": "basic", "high": "standard", "xhigh": "advanced"}[effort]
+ )
assert middle_json.extensions["mineru"]["parse_mode"] == parse_mode
assert type(source_page) is ExtractResult
assert type(source_block) is VlmContentBlock
diff --git a/tests/unittest/test_pdf_analyze_visual_blocks.py b/tests/unittest/test_pdf_analyze_visual_blocks.py
index fcc2fca8..92114f18 100644
--- a/tests/unittest/test_pdf_analyze_visual_blocks.py
+++ b/tests/unittest/test_pdf_analyze_visual_blocks.py
@@ -28,7 +28,7 @@ from mineru.types import RAW_ALGORITHM, RAW_CAPTION, RAW_FOOTNOTE, RAW_FORMULA_N
from mineru.types import FILE_SUFFIXES, BlockType, FileSuffix, MiddleJson, ModelJson
from mineru.version import __version__ as mineru_version
-from _span_test_utils import equation, hyperlink, inline, inline_items, inline_text, inline_urls
+from _span_test_utils import inline, inline_text
JPEG_DATA_URI_PREFIX = "data:image/jpeg;base64,"
@@ -1215,18 +1215,18 @@ def test_doc_analyze_office_returns_model_json_without_pdf_processing(
assert isinstance(middle_json, MiddleJson)
assert len(middle_json.pages) == 1
- assert middle_json.file_suffix == file_suffix
- assert middle_json.extensions["mineru"]["effort"] == "flash"
+ assert middle_json.metadata.file_suffix == file_suffix
+ assert middle_json.extensions["mineru"]["tier"] == "flash"
assert middle_json.extensions["mineru"]["parse_mode"] == "txt"
assert middle_json.is_full_document is True
assert isinstance(model_json, ModelJson)
assert model_json.pages == source_model_list
assert model_json.page_index_map == []
assert model_json.is_full_document is True
- assert model_json.file_suffix == file_suffix
- assert model_json.extensions["mineru"]["effort"] == "flash"
+ assert model_json.metadata.file_suffix == file_suffix
+ assert model_json.extensions["mineru"]["tier"] == "flash"
assert model_json.extensions["mineru"]["parse_mode"] == "txt"
- assert model_json.extensions["mineru"]["mineru_version"] == mineru_version
+ assert model_json.metadata.producer.version == mineru_version
assert inline_text(model_json.pages[0][0]["content"]) == "原始 \\(office\\) 内容"
for suffix, model_factory in model_factories.items():
assert model_factory.call_count == (1 if suffix == file_suffix else 0)
@@ -1596,9 +1596,9 @@ def test_doc_analyze_office_normalizes_product_route(file_suffix: FileSuffix) ->
assert isinstance(middle, MiddleJson) and isinstance(model, ModelJson)
assert len(model.pages) == len(middle.pages) == 1
assert model.is_full_document and middle.is_full_document
- assert model.extensions["mineru"]["effort"] == "flash"
+ assert model.extensions["mineru"]["tier"] == "flash"
assert model.extensions["mineru"]["parse_mode"] == "txt"
- assert model.file_suffix == middle.file_suffix == file_suffix
+ assert model.metadata.file_suffix == middle.metadata.file_suffix == file_suffix
assert middle.pages[0].page_idx == 0
assert "Office routing sample" in middle.to_json()
@@ -1628,10 +1628,10 @@ def test_doc_analyze_flash_real_pdf_returns_typed_middle_json() -> None:
assert all(block.bbox is not None for block in middle_json.pages[0].blocks)
assert all("merge_prev" not in block for page in model_json.pages for block in page)
assert model_json.page_index_map == []
- assert model_json.file_suffix == "pdf"
- assert model_json.extensions["mineru"]["effort"] == "flash"
+ assert model_json.metadata.file_suffix == "pdf"
+ assert model_json.extensions["mineru"]["tier"] == "flash"
assert model_json.extensions["mineru"]["parse_mode"] == "txt"
- assert model_json.extensions["mineru"]["mineru_version"] == mineru_version
+ assert model_json.metadata.producer.version == mineru_version
assert load_middle(json.loads(middle_json.to_json())) == middle_json
assert load_model(json.loads(model_json.to_json())) == model_json
diff --git a/tests/unittest/test_render_api.py b/tests/unittest/test_render_api.py
index b99f604d..16db3048 100644
--- a/tests/unittest/test_render_api.py
+++ b/tests/unittest/test_render_api.py
@@ -36,9 +36,11 @@ def _middle(*pages: PageInfo) -> MiddleJson:
return MiddleJson(
pages=list(pages),
is_full_document=True,
- file_suffix="docx",
- producer=Producer(name="mineru", version="test"),
- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="test"),
+ metadata={"file_suffix": "docx", "producer": Producer(name="mineru", version="test")},
+ extensions=build_metadata(
+ effort="flash",
+ parse_mode="txt",
+ ),
)
diff --git a/tests/unittest/test_structured_content_render.py b/tests/unittest/test_structured_content_render.py
index 2b7a69d5..346475f5 100644
--- a/tests/unittest/test_structured_content_render.py
+++ b/tests/unittest/test_structured_content_render.py
@@ -30,9 +30,11 @@ def _middle(*pages: PageInfo, file_suffix: str = "docx") -> MiddleJson:
return MiddleJson(
pages=list(pages),
is_full_document=True,
- file_suffix=file_suffix,
- producer=Producer(name="mineru", version="test"),
- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="test"),
+ metadata={"file_suffix": file_suffix, "producer": Producer(name="mineru", version="test")},
+ extensions=build_metadata(
+ effort="flash",
+ parse_mode="txt",
+ ),
)
@@ -93,8 +95,8 @@ def test_structured_content_preserves_document_tree_without_merging_or_mutation(
result = render_structured_content(middle)
assert json.loads(json.dumps(result, ensure_ascii=False)) == result
- assert result["file_suffix"] == "docx"
- assert result["effort"] == "flash"
+ assert result["metadata"]["file_suffix"] == "docx"
+ assert result["extensions"]["mineru"]["tier"] == "flash"
assert [page["page_idx"] for page in result["pages"]] == [0, 1]
assert [block["type"] for block in result["pages"][0]["blocks"]] == [
"paragraph_title",
diff --git a/tests/unittest/test_text_block_anchor.py b/tests/unittest/test_text_block_anchor.py
index d81fe41d..1b774f92 100644
--- a/tests/unittest/test_text_block_anchor.py
+++ b/tests/unittest/test_text_block_anchor.py
@@ -36,9 +36,11 @@ def _middle_with_text_anchor() -> MiddleJson:
),
],
is_full_document=True,
- file_suffix="docx",
- producer=Producer(name="mineru", version="test"),
- extensions=build_metadata(effort="flash", parse_mode="txt", mineru_version="test"),
+ metadata={"file_suffix": "docx", "producer": Producer(name="mineru", version="test")},
+ extensions=build_metadata(
+ effort="flash",
+ parse_mode="txt",
+ ),
)
diff --git a/tests/unittest/test_vlm_remote.py b/tests/unittest/test_vlm_remote.py
index ca3e9e44..dbf33cdc 100644
--- a/tests/unittest/test_vlm_remote.py
+++ b/tests/unittest/test_vlm_remote.py
@@ -280,8 +280,8 @@ def test_python_parse_uses_global_remote_vlm(
source = tmp_path / "input.png"
Image.new("RGB", (300, 200), "white").save(source)
result = asyncio.run(parse_async(source, ocr_mode="ocr")) if async_mode else parse(source, ocr_mode="ocr")
- assert result.middle_json.file_suffix == "pdf"
- assert result.middle_json.extensions["mineru"]["effort"] == "high"
+ assert result.middle_json.metadata.file_suffix == "pdf"
+ assert result.middle_json.extensions["mineru"]["tier"] == "standard"
assert result.middle_json.pages[0].page_idx == 0
assert "Remote VLM text" in result.markdown()
assert any(request[2] is not None for request in openai_server.requests)
@@ -346,7 +346,7 @@ def test_api_job_through_http_vlm_to_middle_json(
file_id = job["files"][0]["output_files"]["middle_json"]["file_id"]
output = client.get(f"/v1/files/{file_id}/content")
assert output.status_code == 200
- assert output.json()["effort"] == ("high" if tier == "standard" else "xhigh")
+ assert output.json()["extensions"]["mineru"]["tier"] == tier
assert "Remote VLM text" in output.text
assert all(request[1] == "Bearer test-key" for request in openai_server.requests)