Skip to content

Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.

Enhance PDF layout handling and documentation for layout geometry integration

myhlolif1f212f89820
diff --git a/docs/en/reference/output_files.md b/docs/en/reference/output_files.md
index 03de9952..b8aea60b 100644
--- a/docs/en/reference/output_files.md
+++ b/docs/en/reference/output_files.md
@@ -24,12 +24,66 @@ A `mineru parse --json` response is not MiddleJson and cannot be passed directly
| `LATEX` | LaTeX | `str` |
| `DOCX` | Word | `bytes` |
| `EPUB` | EPUB | `bytes` |
-| `PDF` | Semantically reflowed PDF | `bytes` |
+| `PDF` | Original block layout or semantic reflow | `bytes` |
| `STRUCTURED_CONTENT` | General structured content | `dict` |
| `CONTENT_LIST` | Content List V1 | `list[dict]` |
| `CONTENT_LIST_V2` | Content List V2 grouped by page | `list[list[dict]]` |
-PDF rendering produces a semantic reflow, not a pixel-identical copy of the input or a layout-debug PDF.
+PDF defaults to `PdfLayout.AUTO`: PDF sources with complete geometry retain their
+original pages and block positions; older results without geometry fall back to
+reflow with diagnostics. `ORIGINAL` requires a PDF source, page sizes and necessary
+bboxes. `REFLOW` explicitly selects semantic reflow. OFD and Office still reflow by
+default. Text remains selectable; tables and charts may use region images.
+
+```python
+from mineru.render import PdfLayout, PdfRenderOptions, RenderFormat, render, render_pdf
+
+pdf_bytes = render_pdf(result.middle_json, layout=PdfLayout.ORIGINAL)
+pdf_bytes = render(result.middle_json, RenderFormat.PDF,
+ options=PdfRenderOptions(layout=PdfLayout.REFLOW))
+```
+
+Original layout keeps body text, captions, code, tables and images independently
+fitted to their source boxes. `continues_prev` does not move text between original
+boxes. Original pages and blank pages remain. Both PDF layouts use CJK character
+wrapping for paragraphs containing Chinese, Japanese or Korean text, including
+mixed-language annotations and table cells. This avoids moving long Chinese
+phrases together at spaces. Pure Latin text and literal code/algorithm blocks
+retain their existing rules; no layout-only characters or hyphens are inserted.
+
+Titles reference actual fitted body fonts: following text in the same page and
+column, then nearest same-column text, then the exported body's median (10.5 pt
+if absent). A chapter type/level shares the median reference plus 2 pt, rounded
+up to 0.1 pt, with individual increases for larger local body text. Document
+titles target 2 pt above the largest chapter target, or body plus 4 pt without
+chapters, retaining local hierarchy. Neither 90% coverage nor the previous style
+cap forces smaller titles.
+
+Titles that do not fit their original box may borrow whitespace above, below and
+to the right in the same column, retaining their left edge and preferably their
+top edge. Column width comes from the matched body; uncertain columns keep the
+original width, while existing spanning titles retain their span. Other source
+boxes remain occupied, with 2 pt clearance and page boundaries enforced.
+Adjacent titles divide their gap at its midpoint. Insufficient space shrinks
+only that title in 0.1 pt steps, then uses the existing below-6-pt fallback.
+Script and formula ink extents are measured; index references and approximate
+parent groups are excluded.
+
+`pdf_title_layout_expanded` records expansion. `pdf_layout_font_exception` records
+local-body increases or space-driven reductions, with reference/target/final
+sizes and original/drawing bounds. Existing overlaps retain original occupancy
+and report `pdf_title_geometry_conflict`; unavailable expansion clearance reports
+`pdf_title_clearance_unavailable`. Only rendering context changes, with no input,
+asset, protocol or public-option changes.
+
+Requires `docvortex>=0.4.2,<1`. DocVortex clears native TXT PDF display equation
+`content` once when producing model output; MinerU Flash TXT uses that output.
+Flash OCR already leaves display equation content empty. MinerU does not clear
+equation content again. Both Flash paths retain the bbox, orientation, image and
+detected number region.
+PDF, Markdown, HTML, DOCX, EPUB and LaTeX use the existing image fallback. Inline
+and non-Flash equations retain their content. Old caches are not rewritten;
+reparse to obtain geometry and image-only Flash display equations.
```python
from pathlib import Path
@@ -47,6 +101,13 @@ Path("report.html").write_text(html, encoding="utf-8")
`metadata` carries the file type, producer, and document properties. `extensions["mineru"]` records the actual `tier` and final `parse_mode`. The producing version is `metadata.producer.version`, not a duplicate product extension field. Pages contain `page_idx` and `blocks`; `page_idx` is zero-based, unlike one-based CLI PDF ranges.
+All PDF tiers also emit `extensions.docvortex_layout` through sync and async
+analysis: `version=1` and `pages` with source `page_idx`, `width_pt`, `height_pt`
+and optional `image_rotations`. Sizes share the bbox orientation. Selected and
+blank pages, multiple windows and cached batch compaction retain geometry by
+source page. Both ModelJson and MiddleJson serialization preserve the extension;
+the main protocol remains 2.0. PDF export requires only MiddleJson and image assets.
+
The following example is serialized from the current public types. `4.0.0` is an illustrative stable producer version. Optional fields may be omitted; use the actual output contract:
```json
diff --git a/docs/zh/reference/output_files.md b/docs/zh/reference/output_files.md
index 09ae4b32..bc0aa00b 100644
--- a/docs/zh/reference/output_files.md
+++ b/docs/zh/reference/output_files.md
@@ -24,12 +24,30 @@
| `LATEX` | LaTeX | `str` |
| `DOCX` | Word | `bytes` |
| `EPUB` | EPUB | `bytes` |
-| `PDF` | 语义重排版 PDF | `bytes` |
+| `PDF` | 原始块布局或语义重排 PDF | `bytes` |
| `STRUCTURED_CONTENT` | 通用结构化内容 | `dict` |
| `CONTENT_LIST` | Content List V1 | `list[dict]` |
| `CONTENT_LIST_V2` | 按页组织的 Content List V2 | `list[list[dict]]` |
-PDF 渲染是语义重排版,不是原 PDF 版式的逐像素复刻,也不是布局调试 PDF。
+PDF 默认采用 `PdfLayout.AUTO`:PDF 来源且几何完整时按原始块布局导出;旧结果缺少几何时整份回退重排并记录诊断。`ORIGINAL` 严格要求 PDF 来源、页面尺寸和必要 bbox;`REFLOW` 显式使用语义重排。OFD、Office 等来源默认仍重排。块内文字可选择和复制,表格、图表允许使用区域图;不承诺逐像素复刻。
+
+```python
+from mineru.render import PdfLayout, PdfRenderOptions, RenderFormat, render, render_pdf
+
+pdf_bytes = render_pdf(result.middle_json, layout=PdfLayout.ORIGINAL)
+pdf_bytes = render(result.middle_json, RenderFormat.PDF,
+ options=PdfRenderOptions(layout=PdfLayout.REFLOW))
+```
+
+固定布局中的正文、图注、代码、表格和图片继续独立适配各自原框;`continues_prev` 不移动原框之间的文字。原页数与空白页保留。原布局与重排 PDF 中,含中日韩文字的自然语言段落、图注及表格单元格使用 CJK 字符断行,避免将空格间的长串中文整体移到下一行;纯英文和代码、算法字面块保留既有规则,不向内容插入排版字符或连字符。
+
+标题参考实际适配后的正文字号:优先同页同栏后续正文,其次同栏最近正文,再回退到导出正文的字号中位数;没有正文时采用 10.5 pt。章节标题按 `type + level` 取参考字号中位数 +2 pt,向上取整至 0.1 pt;附近正文较大时只上调该标题。主标题比最大章节标题目标再大 2 pt,没有章节标题时采用正文 +4 pt,并保留局部层次。取消按 90% 容纳率压低整组字号和旧样式字号上限。
+
+原框放得下则保持位置,否则标题可利用上下及同栏右侧空白,保持左边缘;优先原顶边,必要时向上移动。栏宽由对应正文确定,不可靠时保持原宽,原有跨栏标题保留跨度。其他原框视为占用,与相邻内容保留 2 pt 间距且不越页;相邻标题按间隙中线分配空间。仍放不下时只缩小该标题,低于 6 pt 沿用完整块缩放。上下标和公式的真实外伸范围计入标题测量,目录引用及缺子坐标的近似组合不参与。
+
+`pdf_title_layout_expanded` 记录扩展;`pdf_layout_font_exception` 记录局部正文较大或空间不足的调整原因、参考/目标/最终字号、原框和绘制框。原始几何重叠时不扩大占用并报告 `pdf_title_geometry_conflict`;原始间距过紧、没有安全扩展区域时报告 `pdf_title_clearance_unavailable`。字号和绘制区域仅存在于渲染上下文,不修改 MiddleJson 或素材,不新增接口参数。
+
+使用 `docvortex>=0.4.2,<1`。DocVortex 在生产原生 TXT PDF 模型输出时将行间公式的 `content` 清空一次,MinerU Flash TXT 直接使用该输出;Flash OCR 原本不填充行间公式内容,MinerU 不再重复清空。两条 Flash 路径保留 bbox、方向、图片及检测到的编号区域,PDF、Markdown、HTML、DOCX、EPUB、LaTeX 沿用图片回退。行内公式、非 Flash tier 的公式文本不变。旧缓存不会自动改写,重新解析才获得新几何和空内容公式。
```python
from pathlib import Path
@@ -47,6 +65,8 @@ Path("report.html").write_text(html, encoding="utf-8")
`metadata` 包含文件类型、生产者和文档属性;`extensions["mineru"]` 记录实际执行的 `tier` 与最终 `parse_mode`。版本来自 `metadata.producer.version`,不重复放在产品扩展中。页面包含 `page_idx` 与 `blocks`,`page_idx` 从 0 开始,区别于 CLI 中从 1 开始的 PDF 页码。
+所有 PDF tier 的同步/异步分析同时写入 `extensions.docvortex_layout`:`version=1`,`pages` 包含源 `page_idx`、`width_pt`、`height_pt` 及需要时的 `image_rotations`。尺寸方向与 bbox 一致;选页、空白页、多窗口和分页缓存汇总按源页号保留几何。ModelJson → MiddleJson → 序列化结果不丢失扩展,主协议仍为 2.0。导出只需要 MiddleJson 和图片素材,无需重新打开源 PDF。
+
以下示例由当前公开类型序列化生成,`4.0.0` 是正式版文档的示意生产者版本;可选字段可被省略,实际输出为准:
```json
diff --git a/mineru/backend/analysis/contracts.py b/mineru/backend/analysis/contracts.py
index 5cee4dd6..f5ae069c 100644
--- a/mineru/backend/analysis/contracts.py
+++ b/mineru/backend/analysis/contracts.py
@@ -21,3 +21,4 @@ class AnalysisResult:
effort: AnalyzeEffort
parse_mode: ResolvedParseMode
elapsed: float
+ layout_geometry: dict[str, Any] | None = None
diff --git a/mineru/backend/analysis/pdf/pipeline.py b/mineru/backend/analysis/pdf/pipeline.py
index 118e6273..0a47aa01 100644
--- a/mineru/backend/analysis/pdf/pipeline.py
+++ b/mineru/backend/analysis/pdf/pipeline.py
@@ -8,6 +8,7 @@ from dataclasses import dataclass
from typing import cast
from docvortex.document.pdf import PDFDocument
+from loguru import logger
from ....config import VlmConfig
from ....model.runtime.execution import acquire_document, release_document
@@ -96,11 +97,10 @@ def analyze_pdf(
hybrid_model=state.hybrid_model,
vlm_predictor=state.predictor,
)
- _normalize_pdf_model_list(model_list)
- infer_elapsed = time.perf_counter() - infer_started_at
+ result = _build_pdf_analysis_result(state, model_list, effort, infer_started_at)
finally:
_close_analysis(state)
- return AnalysisResult(model_list=model_list, effort=effort, parse_mode=state.parse_mode, elapsed=infer_elapsed)
+ return result
async def aio_analyze_pdf(
@@ -126,11 +126,29 @@ async def aio_analyze_pdf(
hybrid_model=state.hybrid_model,
vlm_predictor=state.predictor,
)
- await run_sync(_normalize_pdf_model_list, model_list)
- infer_elapsed = time.perf_counter() - infer_started_at
+ result = await run_sync(_build_pdf_analysis_result, state, model_list, effort, infer_started_at)
finally:
await run_sync(_close_analysis, state)
- return AnalysisResult(model_list=model_list, effort=effort, parse_mode=state.parse_mode, elapsed=infer_elapsed)
+ return result
+
+
+def _build_pdf_analysis_result(
+ state: _PDFAnalysis, model_list: list, effort: AnalyzeEffort, started_at: float
+) -> AnalysisResult:
+ """规范化模型结果,并在关闭现有 PDF 之前汇集各档位页面几何与裁图方向。"""
+ from docvortex.document.pdf.layout import extract_layout_geometry, attach_layout_image_rotations
+
+ assert state.document is not None
+ # 规范化会删除 angle 并过滤无效块,先按对象保存裁图方向,再按最终索引登记。
+ angles = {id(block): block.get("angle", 0) for page in model_list for block in page}
+ _normalize_pdf_model_list(model_list)
+ elapsed = time.perf_counter() - started_at
+ geometry, diagnostics = extract_layout_geometry(state.document, None)
+ rotation_pages = [[{**block, "angle": angles.get(id(block), 0)} for block in page] for page in model_list]
+ attach_layout_image_rotations(geometry, rotation_pages, None)
+ for diagnostic in diagnostics:
+ logger.warning("{}: {}", diagnostic.code, diagnostic.message)
+ return AnalysisResult(model_list, effort, state.parse_mode, elapsed, geometry)
__all__ = ["analyze_pdf", "aio_analyze_pdf"]
diff --git a/mineru/backend/analyze.py b/mineru/backend/analyze.py
index 877b062c..f895bb4e 100644
--- a/mineru/backend/analyze.py
+++ b/mineru/backend/analyze.py
@@ -152,6 +152,11 @@ def _build_model_json(
) -> ModelJson:
"""共享模型协议构造,保持生产者、页映射和 MinerU 扩展完全一致。"""
_log_infer_performance(file_suffix, len(result.model_list), result.elapsed)
+ extensions = build_metadata(effort=result.effort, parse_mode=result.parse_mode)
+ if file_suffix == "pdf" and result.layout_geometry is not None:
+ from docvortex.document.pdf.layout import LAYOUT_EXTENSION, remap_layout_geometry
+
+ extensions[LAYOUT_EXTENSION] = remap_layout_geometry(result.layout_geometry, page_index_map)
return ModelJson(
pages=result.model_list,
page_index_map=page_index_map or [],
@@ -160,7 +165,7 @@ def _build_model_json(
producer=Producer(name="mineru", version=mineru_version),
document=source_properties.model_copy(deep=True),
),
- extensions=build_metadata(effort=result.effort, parse_mode=result.parse_mode),
+ extensions=extensions,
)
diff --git a/mineru/doclib/background/compaction.py b/mineru/doclib/background/compaction.py
index 29628072..aa079163 100644
--- a/mineru/doclib/background/compaction.py
+++ b/mineru/doclib/background/compaction.py
@@ -164,8 +164,20 @@ class Compaction:
current_envelope = {key: value for key, value in middle_json.to_dict().items() if key != "pages"}
if not envelope:
envelope = current_envelope
- elif json.dumps(envelope, sort_keys=True) != json.dumps(current_envelope, sort_keys=True):
- return None
+ else:
+ from docvortex.document.pdf.layout import merge_layout_extensions
+
+ # 页面几何随批次变化;其他外层协议和产品信息仍须严格相同。
+ previous_fields = {key: value for key, value in envelope.items() if key != "extensions"}
+ current_fields = {key: value for key, value in current_envelope.items() if key != "extensions"}
+ if json.dumps(previous_fields, sort_keys=True) != json.dumps(current_fields, sort_keys=True):
+ return None
+ try:
+ envelope["extensions"] = merge_layout_extensions(
+ envelope["extensions"], current_envelope["extensions"], [page["page_idx"] for page in batch_pages]
+ )
+ except (KeyError, TypeError, ValueError):
+ return None
return (pages_by_page_idx, envelope) if pages_by_page_idx else None
def _write_compacted_json_files(
diff --git a/mineru/render/__init__.py b/mineru/render/__init__.py
index 2eb6a7a1..06bb2522 100644
--- a/mineru/render/__init__.py
+++ b/mineru/render/__init__.py
@@ -12,6 +12,7 @@ from .contracts import (
LatexRenderOptions,
MarkdownRenderOptions,
PdfRenderOptions,
+ PdfLayout,
RenderFormat,
RenderMode,
RenderOptions,
@@ -40,6 +41,7 @@ __all__ = [
"LatexRenderOptions",
"MarkdownRenderOptions",
"PdfRenderOptions",
+ "PdfLayout",
"RenderFormat",
"RenderMode",
"RenderOptions",
diff --git a/mineru/render/api.py b/mineru/render/api.py
index 356026ca..acfcd6ad 100644
--- a/mineru/render/api.py
+++ b/mineru/render/api.py
@@ -207,6 +207,7 @@ def render(
middle_json,
asset_resolver=resolved_options.asset_resolver,
document_title=resolved_options.document_title,
+ layout=resolved_options.layout,
)
if output_format is RenderFormat.STRUCTURED_CONTENT:
diff --git a/mineru/render/contracts.py b/mineru/render/contracts.py
index 5cc2c330..120a8162 100644
--- a/mineru/render/contracts.py
+++ b/mineru/render/contracts.py
@@ -15,6 +15,7 @@ from docvortex.render.contracts import (
LatexRenderOptions,
MarkdownRenderOptions,
PdfRenderOptions,
+ PdfLayout,
RenderMode,
StructuredContentRenderOptions,
)
@@ -91,6 +92,7 @@ __all__ = [
"LatexRenderOptions",
"MarkdownRenderOptions",
"PdfRenderOptions",
+ "PdfLayout",
"RenderFormat",
"RenderMode",
"RenderOptions",
diff --git a/mineru/render/pdf.py b/mineru/render/pdf.py
index 3a3c982d..a2caa203 100644
--- a/mineru/render/pdf.py
+++ b/mineru/render/pdf.py
@@ -4,7 +4,7 @@
from __future__ import annotations
from ..types import MiddleJson
-from .contracts import AssetResolver
+from .contracts import AssetResolver, PdfLayout
def render_pdf(
@@ -12,6 +12,7 @@ def render_pdf(
*,
asset_resolver: AssetResolver | None = None,
document_title: str | None = None,
+ layout: PdfLayout = PdfLayout.AUTO,
) -> bytes:
"""惰性加载 PDF 实现并渲染严格 MiddleJson。"""
from docvortex.render.pdf import render_pdf as _render_pdf
@@ -20,6 +21,7 @@ def render_pdf(
middle_json,
asset_resolver=asset_resolver,
document_title=document_title,
+ layout=layout,
)
diff --git a/pyproject.toml b/pyproject.toml
index c8336e50..78b27673 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -39,7 +39,7 @@ classifiers = [
"Programming Language :: Python :: 3.14",
]
dependencies = [
- "docvortex>=0.4.0,<1",
+ "docvortex>=0.4.2,<1",
"gradio>=6.8,<7",
"mineru[torch] ; sys_platform == 'darwin' and platform_machine == 'arm64'",
"mineru-vl-utils>=2.0.3,<3",
diff --git a/tests/unittest/test_pdf_analyze_visual_blocks.py b/tests/unittest/test_pdf_analyze_visual_blocks.py
index abbe5de0..fe5c3189 100644
--- a/tests/unittest/test_pdf_analyze_visual_blocks.py
+++ b/tests/unittest/test_pdf_analyze_visual_blocks.py
@@ -1641,7 +1641,7 @@ def test_doc_analyze_flash_real_pdf_returns_typed_middle_json() -> None:
def test_doc_analyze_flash_demo1_uses_canonical_equation_type() -> None:
- """验证真实 demo1.pdf 的两层 Flash 输出统一使用 equation 与 LaTeX tag。"""
+ """验证真实 demo1.pdf 的两层 Flash 输出保留公式块和图片,但不伪造 LaTeX。"""
sample_path = _PROJECT_ROOT / "demo" / "pdfs" / "demo1.pdf"
middle_json, model_json = analyze.doc_analyze(
@@ -1660,14 +1660,7 @@ def test_doc_analyze_flash_demo1_uses_canonical_equation_type() -> None:
assert model_equations
assert len(middle_equations) == len(model_equations)
assert [block.content for block in middle_equations] == [block["content"] for block in model_equations]
- for formula_number in range(1, 8):
- marker = rf"\tag{{{formula_number}}}"
- assert sum(marker in block["content"] for block in model_equations) == 1
- assert not [
- block
- for block in model_equations
- if any(block["content"].rstrip().endswith(f"({formula_number})") for formula_number in range(1, 8))
- ]
+ assert all(block["content"] == "" and block.get("image_base64") for block in model_equations)
assert "interline_equation" not in middle_json.to_json()
@@ -1790,7 +1783,9 @@ def test_doc_analyze_flash_returns_complete_model_json_and_typed_middle_json(mon
assert requested_ranges == [(1, 1), (2, 2)]
assert inline_text(model_json.pages[0][0]["content"]) == "第一页 x+y"
assert [span["content"] for span in model_json.pages[0][0]["content"] if span.get("type") == "equation_inline"] == ["x+y"]
+ # 模拟上游返回非空公式,验证 MinerU 不按 Flash 档位重复清空模型内容。
assert model_json.pages[2][0]["content"] == "z"
+ assert middle_json.pages[2].blocks[0].content == "z"
assert model_json.pages[2][0]["type"] == BlockType.EQUATION
assert middle_json.pages[2].blocks[0].type == BlockType.EQUATION
assert "interline_equation" not in middle_json.to_json()
diff --git a/tests/unittest/test_pdf_layout_integration.py b/tests/unittest/test_pdf_layout_integration.py
new file mode 100644
index 00000000..2848ab72
--- /dev/null
+++ b/tests/unittest/test_pdf_layout_integration.py
@@ -0,0 +1,194 @@
+"""覆盖四档位同步/异步几何传递、Flash 图片公式与缓存汇总。"""
+
+from __future__ import annotations
+
+import asyncio
+import base64
+from copy import deepcopy
+from io import BytesIO
+import json
+from pathlib import Path
+from types import SimpleNamespace
+
+from PIL import Image
+from pypdf import PdfReader, PdfWriter
+import pytest
+from reportlab.pdfgen.canvas import Canvas
+
+from docvortex.document.pdf import PDFDocument
+from docvortex.document.pdf.layout import merge_layout_extensions
+from docvortex.schema import DocumentProperties, MiddleJson, ModelJson
+from mineru.backend import analyze
+from mineru.backend.analysis.pdf import pipeline
+from mineru.backend.postprocess import document as postprocess
+from mineru.doclib.background.compaction import Compaction
+from mineru.doclib.services.parse_svc import parse_batch_json_path
+from mineru.render import PdfLayout, PdfRenderOptions, RenderFormat, render, render_pdf
+
+
+def _source() -> bytes:
+ """生成不同尺寸及空白页的真实 PDF,用于检查几何生命周期。"""
+ output = BytesIO()
+ canvas = Canvas(output)
+ for width, height in [(400, 600), (320, 480), (600, 400)]:
+ canvas.setPageSize((width, height))
+ canvas.showPage()
+ canvas.save()
+ writer = PdfWriter(clone_from=BytesIO(output.getvalue()))
+ writer.pages[1].rotate(90)
+ rotated = BytesIO()
+ writer.write(rotated)
+ return rotated.getvalue()
+
+
+def _model_pages(effort: str) -> list[list[dict]]:
+ """按上游契约提供 Flash 空公式及其他档位公式文本,几何采集与导出仍走真实实现。"""
+ image = BytesIO()
+ Image.new("RGB", (80, 20), "black").save(image, "PNG")
+ return [
+ [
+ {"type": "text", "content": "invalid without line geometry", "bbox": (0.1, 0.1, 0.2, 0.2), "angle": 270},
+ {
+ "type": "equation",
+ "bbox": (0.1, 0.1, 0.9, 0.2),
+ "angle": 90,
+ "content": "" if effort == "flash" else "x^2",
+ "image_base64": "data:image/png;base64," + base64.b64encode(image.getvalue()).decode(),
+ },
+ ],
+ [],
+ [{"type": "text", "bbox": (0.1, 0.1, 0.9, 0.3), "content": [{"type": "text", "content": "last page"}]}],
+ ]
+
+
[email protected]("effort,tier", [("flash", "flash"), ("medium", "basic"), ("high", "standard"), ("xhigh", "advanced")])
[email protected]("parse_mode", ["txt", "ocr"])
[email protected]("asynchronous", [False, True])
+def test_all_tiers_preserve_geometry_to_pdf(
+ monkeypatch: pytest.MonkeyPatch,
+ effort: str,
+ tier: str,
+ parse_mode: str,
+ asynchronous: bool,
+) -> None:
+ """同一结果经过完整同步/异步门面后保留源页映射,并可显式按原布局导出。"""
+ events = []
+
+ def prepare(state: object, data: bytes, requested_effort: str, mode: str, vlm_config: object) -> None:
+ """打开真实 PDF,但用占位上下文代替昂贵推理模型。"""
+ state.document = PDFDocument(data)
+ state.parse_mode = mode
+ state.flash_txt_mode = requested_effort == "flash" and mode == "txt"
+ state.hybrid_model = SimpleNamespace(device="cpu")
+ state.predictor = object()
+
+ def windows(*args: object, **kwargs: object) -> list[list[dict]]:
+ """返回独立 raw pages,模拟三个窗口最终汇总的结果。"""
+ return deepcopy(_model_pages(effort))
+
+ async def async_windows(*args: object, **kwargs: object) -> list[list[dict]]:
+ """覆盖原生异步分支,不把异步 API 误测为纯同步代理。"""
+ events.append("native_async")
+ return windows()
+
+ monkeypatch.setattr(pipeline, "_prepare_analysis", prepare)
+ monkeypatch.setattr(pipeline, "process_pdf_windows", windows)
+ monkeypatch.setattr(pipeline, "aio_process_pdf_windows", async_windows)
+ monkeypatch.setattr(pipeline, "release_document", lambda *args: None)
+ monkeypatch.setattr("mineru.model.vlm.client.uses_native_async_vlm", lambda *args: True)
+ monkeypatch.setattr(postprocess, "apply_llm_aided_postprocess", lambda *args: None)
+
+ async def no_enhancement(*args: object) -> None:
+ """关闭与几何验证无关的外部 LLM 服务。"""
+
+ monkeypatch.setattr(postprocess, "aio_apply_llm_aided_postprocess", no_enhancement)
+ kwargs = {
+ "effort": effort,
+ "parse_mode": parse_mode,
+ "page_index_map": [1, 4, 5],
+ "source_properties": DocumentProperties(),
+ }
+ middle, model = (
+ asyncio.run(analyze.aio_doc_analyze(_source(), **kwargs)) if asynchronous else analyze.doc_analyze(_source(), **kwargs)
+ )
+ assert model.extensions["mineru"] == {"tier": tier, "parse_mode": parse_mode}
+ expected = [
+ {"page_idx": idx, "width_pt": width, "height_pt": height}
+ for idx, (width, height) in zip([1, 4, 5], [(400, 600), (480, 320), (600, 400)], strict=True)
+ ]
+ expected[0]["image_rotations"] = {"0": 90}
+ assert middle.extensions["docvortex_layout"]["pages"] == model.extensions["docvortex_layout"]["pages"] == expected
+ assert model.pages[0][0]["content"] == ("" if effort == "flash" else "x^2")
+ assert ModelJson.from_json(model.to_json()).to_json() == model.to_json()
+ assert MiddleJson.from_json(middle.to_json()).to_json() == middle.to_json()
+ payload = render(middle, RenderFormat.PDF, options=PdfRenderOptions(layout=PdfLayout.ORIGINAL))
+ direct = render_pdf(middle, layout=PdfLayout.ORIGINAL)
+ assert [page.extract_text() for page in PdfReader(BytesIO(payload)).pages] == [
+ page.extract_text() for page in PdfReader(BytesIO(direct)).pages
+ ]
+ assert [tuple(page.mediabox)[2:] for page in PdfReader(BytesIO(payload)).pages] == [(400, 600), (480, 320), (600, 400)]
+ if asynchronous and effort in {"high", "xhigh"}:
+ assert events == ["native_async"]
+
+
+def test_compaction_merges_page_geometry_with_latest_content(tmp_path: Path) -> None:
+ """不同批次页几何可合并,重复页的图片方向与最新正文一同更新。"""
+ sha = "a" * 64
+ for page_range, done_at, indices in [("1-2", 1, [0, 1]), ("2-3", 2, [1, 2])]:
+ model = ModelJson(
+ pages=[[] for _ in indices],
+ page_index_map=indices,
+ metadata={"file_suffix": "pdf", "producer": {"name": "mineru", "version": "test"}},
+ extensions={
+ "mineru": {"tier": "standard", "parse_mode": "txt"},
+ "docvortex_layout": {
+ "version": 1,
+ "pages": [
+ {
+ "page_idx": idx,
+ "width_pt": 400,
+ "height_pt": 600,
+ "image_rotations": {"0": 90 if done_at == 1 else 270},
+ }
+ for idx in indices
+ ],
+ },
+ },
+ )
+ middle = postprocess.build_middle_json(model)
+ path = Path(parse_batch_json_path(str(tmp_path), sha, "standard", page_range, done_at))
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(middle.to_json())
+ compaction = Compaction(db=None, interval_sec=600, data_dir=str(tmp_path))
+ rows = [{"page_range": "2-3", "done_at": 2}, {"page_range": "1-2", "done_at": 1}]
+ loaded = compaction._load_batch_payloads(sha, "standard", rows)
+ assert loaded is not None
+ pages_by_index, envelope = loaded
+ pages = envelope["extensions"]["docvortex_layout"]["pages"]
+ assert [page["page_idx"] for page in pages] == [0, 1, 2]
+ assert [page["image_rotations"]["0"] for page in pages] == [90, 270, 270]
+ assert len(list(tmp_path.rglob("*.json"))) == 2
+ before = deepcopy(envelope)
+ written = compaction._write_compacted_json_files(sha, "standard", ["1-3"], 2, pages_by_index, envelope)
+ assert written is not None and len(written) == 1
+ replayed = MiddleJson.from_json(Path(next(iter(written))).read_text())
+ assert replayed.extensions == envelope["extensions"]
+ assert envelope == before
+ assert len(PdfReader(BytesIO(render_pdf(replayed, layout=PdfLayout.ORIGINAL))).pages) == 3
+
+
+def test_old_replacement_batch_does_not_keep_stale_rotation() -> None:
+ """较新正文没有几何时,不从较旧同页借用可能对应错误块索引的旋转信息。"""
+ old = {
+ "docvortex_layout": {
+ "version": 1,
+ "pages": [
+ {"page_idx": 0, "width_pt": 400, "height_pt": 600},
+ {"page_idx": 1, "width_pt": 400, "height_pt": 600},
+ ],
+ }
+ }
+ original = json.dumps(old)
+ merged = merge_layout_extensions(old, {}, [1])
+ assert [page["page_idx"] for page in merged["docvortex_layout"]["pages"]] == [0]
+ assert json.dumps(old) == original