Skip to content

Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.

feat(api): add recoverable synchronous file parsing service

myhlolif59acf1ffa79
diff --git a/docs/next/api.md b/docs/next/api.md
index 4bc4395e..e74a708d 100644
--- a/docs/next/api.md
+++ b/docs/next/api.md
@@ -23,6 +23,7 @@ Unified API 文档描述两类部署形态下的同一套 API:
4. [Uploads 与 Files](api/uploads-files.md): 上传生命周期、文件对象、产物下载。
5. [Parse Jobs](api/parse-jobs.md): 解析任务、轮询。
[V1 tasks](api/tasks.md): JSON 清单、逐文件上传、独立结果下载链接和 Router 调度。
+ [同步 file_parse](api/file-parse.md): 独立包装服务,单文件直传、同步附件和超时恢复。
6. [Chat 与 Responses](api/chat.md): OpenAI-compatible 文档对话接口。
7. [Webhooks](api/webhooks.md): 异步回调协议。
8. [Usage 与 Limits](api/usage-limits.md): access level、限流、用量查询。
diff --git a/docs/next/api/README.md b/docs/next/api/README.md
index e545afb7..bfbef375 100644
--- a/docs/next/api/README.md
+++ b/docs/next/api/README.md
@@ -68,6 +68,8 @@ ID 使用 OpenAI 风格的前缀加随机串。官方 API 使用 24 字符 base6
JSON、逐文件上传、独立结果链接与 Router 负载规则见 [V1 tasks](tasks.md)。SaaS 可以独立实现同一契约,并返回 OSS 直链;客户端不要求其实现本地 Files 下载路由。
+需要单文件直传和同步附件响应时,可另外启动 [file_parse 包装服务](file-parse.md),它提供自己的 `/v1/file_parse`、查询和下载接口,并通过上面的 Tasks 协议连接任意部署。
+
## 认证
官方 API 使用 Bearer Token:
diff --git a/docs/next/api/file-parse.md b/docs/next/api/file-parse.md
new file mode 100644
index 00000000..1b05ed9d
--- /dev/null
+++ b/docs/next/api/file-parse.md
@@ -0,0 +1,121 @@
+# 单文件同步解析:file_parse
+
+`mineru-kit file-parse-server` 提供独立的同步 HTTP 包装服务:接收一个文件,创建 Tasks、上传、等待,完成后直接返回 ZIP、DOCX 或 HTML 附件。它连接本地 API、Router 或 SaaS;切换部署只修改上游地址和认证。
+
+## 启动
+
+```bash
+mineru-kit file-parse-server --api-url http://127.0.0.1:8000
+
+# 上游基地址可以带路径前缀;不要附加 /v1/tasks。
+MINERU_API_URL=https://mineru.net/api MINERU_API_KEY=your-key \
+ mineru-kit file-parse-server
+```
+
+| 启动参数 | 默认值 | 含义 |
+|---|---|---|
+| `--api-url` | `MINERU_API_URL`,否则必填 | 上游统一 Tasks API 基地址。 |
+| `--api-key` | `MINERU_API_KEY`,否则为空 | 上游 Bearer Token。 |
+| `--host`、`--port` | `127.0.0.1`、`8003` | 包装服务监听地址。 |
+| `--state-dir` | `${MINERU_HOME:-~/.mineru}/file-parse` | 恢复记录及传输临时目录。 |
+| `--retention-seconds` | `86400` | 首次观察终态后保留恢复记录的秒数;`0` 禁用回收。 |
+
+包装入口不提供认证,面向本机或可信网络;部署方可以在反向代理设置访问认证。`--api-key` 只用于连接上游。请求不能覆盖上游地址或认证,入站 Authorization 不会转发到上游。
+
+服务只执行 HTTP 上传、查询、下载,不初始化解析模型。OpenAPI 位于 `/openapi.json`,交互文档位于 `/docs`。这些路由属于独立包装服务,不注册到原 API Server 或 Router。
+
+## 上传与等待
+
+`POST /v1/file_parse` 仅接受 `multipart/form-data`,业务参数放在表单字段中;不接受查询参数、未知字段或重复标量字段。
+
+| 字段 | 默认值 | 说明 |
+|---|---|---|
+| `file` | 必填 | 一个非空文件,保留扩展名;不接受多文件。 |
+| `tier` | 上游默认 | `flash/basic/standard/advanced`。 |
+| `ocr_mode` | `auto` | `auto/txt/ocr`。 |
+| `page_range` | 整本 | 与 Tasks 一致,PDF 支持 `1-5`、`r1`、`all`;格式支持范围由上游校验。 |
+| `output_formats` | `default` | `default/docx/html/latex`;重复同名字段表达列表,重复值按首次出现去重。 |
+| `wait_timeout` | `3600` | 整数秒,范围 `0–3600`。 |
+| `request_id` | 自动生成 | 1–64 个字母、数字、下划线或短横线。提前指定并保存它,可在连接中断后查询。 |
+
+```bash
+# 成功时 response.body 是附件;超时或错误时是 JSON,请同时检查 response.headers。
+curl --max-time 7200 -D response.headers -o response.body \
+ http://127.0.0.1:8003/v1/file_parse \
+ -F 'request_id=report-20261022-001' \
+ -F 'tier=flash' -F 'ocr_mode=txt' \
+ -F 'output_formats=default' -F 'output_formats=docx'
+```
+
+`output_formats` 的列表语义与 [Tasks](tasks.md) 一致;multipart 中不使用 JSON 字符串或逗号分隔字符串。
+
+| 选择 | HTTP 成功附件 |
+|---|---|
+| 单选 `default` | ZIP,包含 Markdown、中间协议、Structured Content、外置图片。 |
+| 单选 `docx` | 图片内嵌的 DOCX。 |
+| 单选 `html` | 图片内嵌的 HTML。 |
+| 单选 `latex` | ZIP,包含 TeX 和外置图片。 |
+| 多个不同格式 | 上游生成的单个 ZIP。 |
+
+成功返回 `200`、正确的 Content-Type 和 `Content-Disposition: attachment`,文件名为 `<输入文件主名>.<扩展名>`,并携带 `X-Request-ID`、`X-Task-ID`。包装服务原样返回验证过的上游产物,不重新渲染或打包。
+
+每次调用在确认上游文件就绪后开始计时,等待包含排队、解析与导出。接收文件、上游上传和结果下载不占此预算,独立使用传输空闲超时。缓存命中跳过上传。
+
+- 正常轮询间隔为 2 秒。
+- 连接、元数据请求、传输空闲超时分别为 10、30、600 秒。
+- `wait_timeout=0` 在文件就绪后直接返回 `202` 查询信息;即使已经完成也不下载附件。
+- 等待到期返回 `202` JSON 和 `Location: <status_url>`,上游继续执行。
+- HTTP 客户端和反向代理的读取超时应覆盖上传、同步等待和结果传输;例如客户端配置 7200 秒,代理按业务文件大小配置相应超时。
+
+## 超时与断连后查询
+
+```json
+{
+ "request_id": "report-20261022-001",
+ "task_id": "task_example",
+ "status": "running",
+ "status_url": "http://127.0.0.1:8003/v1/file_parse/report-20261022-001",
+ "result_url": null,
+ "output_formats": ["default", "docx"],
+ "error": null
+}
+```
+
+`GET /v1/file_parse/{request_id}` 查询状态,完成后 `result_url` 指向包装服务自己的 `/v1/file_parse/{request_id}/result`。调用者不需要上游密钥。
+
+```bash
+curl http://127.0.0.1:8003/v1/file_parse/report-20261022-001
+curl --fail-with-body -o report.zip \
+ http://127.0.0.1:8003/v1/file_parse/report-20261022-001/result
+```
+
+结果未就绪时下载接口返回 `409 result_not_ready`。下载中断可以再次 GET;每次会查询最新下载链接。签名失效时重新查询原任务,最多尝试 3 次;上传失败同样先查询状态,确认已上传成功便不重传。
+
+状态包括接收完成后的 `prepared/submitting`,以及单文件 Tasks 的 `waiting_upload/queued/running/completed/failed/canceled`。`submission_unknown` 表示创建请求可能已被上游接受,但包装服务没有获得有效任务 ID:不会自动重放创建请求,不能承诺找回这个不确定窗口内的任务。
+
+错误响应使用 `error.code/message/upstream_status`。已接受请求的错误还包含请求 ID 和查询地址。参数错误返回 `400`,内容类型错误返回 `415`,ID 冲突或结果未就绪返回 `409`,记录不存在或已过保留期返回 `404`;上游调用或解析失败通常返回 `502` 并保留上游错误码。等待预算耗尽返回 `202`,不是解析失败。
+
+## 去重与持久化边界
+
+完整接收并持久化源文件后,包装服务才接受请求并托管上游上传。此后连接中断不取消上传或解析;文件尚未完整接收时中断不会创建任务。
+
+同一 `request_id`、相同文件内容、逻辑文件名及归一化解析参数复用原任务;不同输入返回 `409 request_id_conflict`。`wait_timeout` 不参与身份比较。没有预先提供 ID 且尚未收到响应的调用者,可能无法自行找回任务。
+
+请求映射使用独立 SQLite 数据库,按上游地址和认证隔离。数据库不保存上游 API Key。单个状态目录只允许一个包装服务进程;服务重启后恢复已知任务,尚待上传且链接有效时继续上传。上游任务记录如果因重启或过期而消失,不会自动重建。
+
+源文件在上游确认就绪后删除,终结失败也会清理源文件。结果只临时下载,发送完成或中断即删除;重启清理遗留传输文件。恢复记录的保留期限不延长上游结果有效期,查询也不会为上游上传链接续期。结果及请求过期后应使用新的请求 ID 发起新的解析。
+
+## Python HTTP 示例
+
+仓库提供 `scripts/http_file_parse_example.py`,依赖 `httpx`,提前打印并保存请求 ID;成功时原子保存附件,`202` 时打印查询信息并以 `124` 退出。
+
+```bash
+python scripts/http_file_parse_example.py report.pdf --output results/report.docx \
+ --output-format docx --request-id report-20261022-002
+
+python scripts/http_file_parse_example.py report.pdf --wait-timeout 0 \
+ --request-id report-20261022-003
+```
+
+后续查询和下载仍使用上面的包装接口,不需要安装 MinerU SDK 或重建 `ParseResult`。
diff --git a/docs/next/api/tasks.md b/docs/next/api/tasks.md
index b51f3c11..1545043c 100644
--- a/docs/next/api/tasks.md
+++ b/docs/next/api/tasks.md
@@ -221,5 +221,6 @@ Router 使用有界、可续租的内部结果租约保护 worker 产物,上
本次协议不保留旧表单、直传或 `markdown/middle_json/structured_content/zip` Tasks 格式别名。
客户端应改用 JSON、`files[].result_url` 及逐文件保存方式。
独立 `/v1/uploads`、`/v1/parse/jobs` 与 SDK 的现有请求及输出协议保持不变。
-`/v1/file_parse`、`/v1/tasks/{task_id}/result` 不恢复。
+API Server 和 Router 不提供 `/v1/file_parse`、`/v1/tasks/{task_id}/result`。
+需要同步单文件调用时,独立启动 [file_parse 包装服务](file-parse.md),由它调用本页的共同 Tasks 契约。
SaaS Go 服务和对象存储配置由部署方按此共同契约实现,不属于本地 Python 服务代码。
diff --git a/docs/next/cli/mineru-kit.md b/docs/next/cli/mineru-kit.md
index 4d0dee53..794624da 100644
--- a/docs/next/cli/mineru-kit.md
+++ b/docs/next/cli/mineru-kit.md
@@ -25,6 +25,7 @@
| `mineru-kit parse` | 无状态文件/目录批处理解析 | [mineru-kit parse](mineru-kit-parse.md) |
| `mineru-kit webui` | 基于 V1 API 的本地文档解析 Web UI | [mineru-kit webui](mineru-kit-webui.md) |
| `mineru-kit api-server` | 启动兼容统一 API 的本地解析服务 | [mineru-kit api-server](mineru-kit-api-server.md) |
+| `mineru-kit file-parse-server` | 单文件直传、同步附件和超时恢复,连接既有 Tasks 服务 | [file_parse](../api/file-parse.md) |
| `mineru-kit vlm-server` | 本地 VLM 服务,兼容 OpenAI Chat Completions 协议 | [mineru-kit vlm-server](mineru-kit-vlm-server.md) |
| `mineru-kit router` | 启动 V1 路由服务,转发到已有 upstream 或管理本地 worker | 仅暴露 `/v1/*` |
@@ -50,7 +51,7 @@ mineru-api --host 127.0.0.1 --port 8000 --tier standard
| 能力 | `mineru` | `mineru-kit` |
|------|----------|--------------|
-| 本地数据库 | 有 | 无 |
+| 本地文档库 | 有 | 无;file-parse-server 仅保存独立请求恢复记录 |
| SHA256 缓存 | 有 | 无 |
| STDOUT 默认阅读 | 有 | 不是默认重点 |
| 目录批处理 | 非首要 | 核心能力 |
diff --git a/mineru/kit/commands/__init__.py b/mineru/kit/commands/__init__.py
index 01f43505..38d00a1e 100644
--- a/mineru/kit/commands/__init__.py
+++ b/mineru/kit/commands/__init__.py
@@ -1,5 +1,5 @@
"""mineru-kit command modules."""
-from . import api_server, models, parse, router, vlm_server, webui
+from . import api_server, file_parse_server, models, parse, router, vlm_server, webui
-__all__ = ["api_server", "models", "parse", "router", "vlm_server", "webui"]
+__all__ = ["api_server", "file_parse_server", "models", "parse", "router", "vlm_server", "webui"]
diff --git a/mineru/kit/commands/file_parse_server.py b/mineru/kit/commands/file_parse_server.py
new file mode 100644
index 00000000..78fcce8f
--- /dev/null
+++ b/mineru/kit/commands/file_parse_server.py
@@ -0,0 +1,39 @@
+"""启动只调用共同 Tasks 契约的同步包装服务。"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import typer
+
+from ...utils.i18n import t
+from ..errors import exit_with_message
+
+
+def file_parse_server_cmd(
+ api_url: str = typer.Option(..., "--api-url", envvar="MINERU_API_URL", help=t("Upstream Tasks API base URL")),
+ api_key: str | None = typer.Option(None, "--api-key", envvar="MINERU_API_KEY", help=t("Upstream Bearer API key")),
+ host: str = typer.Option("127.0.0.1", "--host", help=t("Server host")),
+ port: int = typer.Option(8003, "--port", min=1, max=65535, help=t("Server port")),
+ state_dir: Path | None = typer.Option(None, "--state-dir", help=t("File parse recovery state directory")),
+ retention_seconds: int = typer.Option(
+ 86400, "--retention-seconds", min=0, help=t("Terminal retention seconds (default 86400; 0 disables collection)")
+ ),
+) -> None:
+ """惰性加载包装应用,不启动本地解析模型或继承调用者的认证头。"""
+ import uvicorn
+ from filelock import Timeout
+
+ from ..file_parse import FileParseSettings, create_app
+
+ try:
+ settings = FileParseSettings(api_url, api_key, state_dir, retention_seconds)
+ except ValueError as exc:
+ exit_with_message("invalid_request", str(exc))
+ try:
+ uvicorn.run(create_app(settings), host=host, port=port)
+ except Timeout:
+ exit_with_message("invalid_request", "Another file-parse-server is using this state directory")
+
+
+__all__ = ["file_parse_server_cmd"]
diff --git a/mineru/kit/file_parse/__init__.py b/mineru/kit/file_parse/__init__.py
new file mode 100644
index 00000000..ea0af740
--- /dev/null
+++ b/mineru/kit/file_parse/__init__.py
@@ -0,0 +1,6 @@
+"""独立单文件同步解析服务的显式入口。"""
+
+from .app import create_app
+from .settings import FileParseSettings
+
+__all__ = ["FileParseSettings", "create_app"]
diff --git a/mineru/kit/file_parse/app.py b/mineru/kit/file_parse/app.py
new file mode 100644
index 00000000..b5e0784c
--- /dev/null
+++ b/mineru/kit/file_parse/app.py
@@ -0,0 +1,308 @@
+"""单文件同步 HTTP 包装层;成功返回附件,超时返回可恢复的请求链接。"""
+
+from __future__ import annotations
+
+import asyncio
+import hashlib
+import os
+import re
+import tempfile
+import uuid
+from collections.abc import AsyncIterator
+from contextlib import asynccontextmanager
+from pathlib import Path
+from typing import Any
+
+import httpx
+from fastapi import FastAPI, Request, Response
+from fastapi.responses import FileResponse, JSONResponse
+from pydantic import BaseModel, ValidationError
+from starlette.datastructures import UploadFile
+from starlette.exceptions import HTTPException
+from starlette.requests import ClientDisconnect
+from starlette.types import ASGIApp, Receive, Scope, Send
+
+from ...task_types import TASK_OUTPUT_FORMATS
+from .models import GatewayError, ParseOptions, Record
+from .service import FileParseService
+from .settings import FileParseSettings
+
+REQUEST_ID = re.compile(r"^[A-Za-z0-9_-]{1,64}$")
+MIME_TYPES = {
+ "zip": "application/zip",
+ "docx": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
+ "html": "text/html",
+}
+
+
+class StatusResponse(BaseModel):
+ """查询与超时响应共享结构,创建尚未确认时 task_id 为空。"""
+
+ request_id: str
+ task_id: str | None
+ status: str
+ status_url: str
+ result_url: str | None
+ output_formats: list[str]
+ error: dict[str, Any] | None
+
+
+class ReceiveTimeoutMiddleware:
+ """对接收请求体的每次等待设置独立空闲超时。"""
+
+ def __init__(self, app: ASGIApp, timeout: float) -> None:
+ """记录传输预算,不限制正常持续传输大文件的总时长。"""
+ self.app = app
+ self.timeout = timeout
+
+ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
+ """仅约束 POST 请求体,响应等待期间的断连观察不应用传输超时。"""
+ body_finished = False
+
+ async def timed_receive() -> dict[str, Any]:
+ """在最后一块请求体到达后恢复普通 receive 行为。"""
+ nonlocal body_finished
+ try:
+ if body_finished:
+ return await receive()
+ message = await asyncio.wait_for(receive(), self.timeout)
+ except asyncio.TimeoutError as exc:
+ raise GatewayError("upload_receive_timeout", "Receiving the file timed out", 408) from exc
+ if message["type"] == "http.request" and not message.get("more_body", False):
+ body_finished = True
+ return message
+
+ await self.app(scope, timed_receive if scope["type"] == "http" and scope.get("method") == "POST" else receive, send)
+
+
+class TemporaryFileResponse(FileResponse):
+ """无论发送完成、客户端断连还是取消,都释放已验证的结果临时文件。"""
+
+ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
+ """用 finally 兜底清理,避免只依赖成功发送后的后台回调。"""
+ try:
+ await super().__call__(scope, receive, send)
+ finally:
+ Path(self.path).unlink(missing_ok=True)
+
+
+def _request_schema() -> dict[str, Any]:
+ """为手动严格解析的 multipart 请求提供完整 OpenAPI 描述。"""
+ properties = ParseOptions.model_json_schema()["properties"]
+ properties.update(
+ {
+ "file": {"type": "string", "format": "binary"},
+ "request_id": {"type": "string", "pattern": REQUEST_ID.pattern},
+ "wait_timeout": {"type": "integer", "minimum": 0, "maximum": 3600, "default": 3600},
+ }
+ )
+ properties["output_formats"] = {
+ "type": "array",
+ "items": {"type": "string", "enum": list(TASK_OUTPUT_FORMATS)},
+ "minItems": 1,
+ "default": ["default"],
+ }
+ return {
+ "requestBody": {
+ "required": True,
+ "content": {
+ "multipart/form-data": {
+ "schema": {"type": "object", "properties": properties, "required": ["file"], "additionalProperties": False},
+ "encoding": {"output_formats": {"style": "form", "explode": True}},
+ }
+ },
+ }
+ }
+
+
+async def _receive_file(request: Request, service: FileParseService) -> tuple[Record, int]:
+ """严格接收单文件 multipart 并持久化源字节,完整接受后才启动上游工作。"""
+ if request.query_params:
+ raise GatewayError("invalid_request", "Parameters must be supplied as multipart fields", 400)
+ if request.headers.get("content-type", "").split(";", 1)[0].strip().lower() != "multipart/form-data":
+ raise GatewayError("unsupported_media_type", "Expected multipart/form-data", 415)
+ allowed = {"file", "request_id", "wait_timeout", "tier", "ocr_mode", "page_range", "output_formats"}
+ source: Path | None = None
+ accepted = False
+ try:
+ async with request.form(max_files=1, max_fields=32, max_part_size=65536) as form:
+ if set(form) - allowed or any(len(form.getlist(key)) != 1 for key in form if key != "output_formats"):
+ raise GatewayError("invalid_request", "Unknown or repeated scalar multipart fields", 400)
+ upload = form.get("file")
+ if not isinstance(upload, UploadFile) or any(
+ not isinstance(value, str) for key, value in form.multi_items() if key != "file"
+ ):
+ raise GatewayError("invalid_request", "Exactly one file and textual parameters are required", 400)
+ filename = (upload.filename or "").replace("\\", "/").rsplit("/", 1)[-1]
+ if filename in {"", ".", ".."} or len(filename) > 1024 or any(ord(char) < 32 for char in filename):
+ raise GatewayError("invalid_request", "A valid filename is required", 400)
+ request_id = str(form.get("request_id", uuid.uuid4().hex))
+ if not REQUEST_ID.fullmatch(request_id):
+ raise GatewayError("invalid_request", "Invalid request_id", 400)
+ request.state.request_id = request_id
+ try:
+ wait_timeout = int(str(form.get("wait_timeout", "3600")))
+ if not 0 <= wait_timeout <= 3600:
+ raise ValueError("wait_timeout out of range")
+ options = ParseOptions.model_validate(
+ {
+ **{key: form[key] for key in ("tier", "ocr_mode", "page_range") if key in form},
+ "output_formats": form.getlist("output_formats") if "output_formats" in form else ["default"],
+ }
+ )
+ except (ValueError, ValidationError) as exc:
+ raise GatewayError("invalid_request", "Invalid parsing parameters or wait_timeout", 400) from exc
+ descriptor, name = tempfile.mkstemp(prefix="source-", dir=service.store.sources)
+ source = Path(name)
+ size, hasher = 0, hashlib.sha256()
+ with os.fdopen(descriptor, "wb") as destination:
+ while chunk := await upload.read(1024 * 1024):
+ size += len(chunk)
+ hasher.update(chunk)
+ await asyncio.to_thread(destination.write, chunk)
+ destination.flush()
+ await asyncio.to_thread(os.fsync, destination.fileno())
+ if size == 0:
+ raise GatewayError("invalid_request", "The file must not be empty", 400)
+ operation = asyncio.create_task(service.accept(request_id, filename, size, hasher.hexdigest(), options, source))
+ try:
+ record = await asyncio.shield(operation)
+ except asyncio.CancelledError:
+ await operation
+ accepted = True
+ raise
+ accepted = True
+ return record, wait_timeout
+ except HTTPException as exc:
+ raise GatewayError("invalid_request", "Invalid multipart upload or more than one file", 400) from exc
+ except ClientDisconnect as exc:
+ raise GatewayError("upload_interrupted", "Client disconnected before the file was accepted", 400) from exc
+ except OSError as exc:
+ raise GatewayError("storage_error", "Cannot retain the uploaded file", 500) from exc
+ finally:
+ if source is not None and not accepted:
+ source.unlink(missing_ok=True)
+
+
+async def _attachment(service: FileParseService, record: Record) -> TemporaryFileResponse:
+ """获取已验证产物并设置可直接下载的文件名、类型和任务标识。"""
+ try:
+ path = await service.artifact(record)
+ except OSError as exc:
+ raise GatewayError("storage_error", "Cannot save the downloaded result", 500) from exc
+ formats = record.options["output_formats"]
+ extension = formats[0] if len(formats) == 1 and formats[0] in {"docx", "html"} else "zip"
+ task = record.task
+ assert task is not None
+ return TemporaryFileResponse(
+ path,
+ filename=f"{Path(record.filename).stem}.{extension}",
+ media_type=MIME_TYPES[extension],
+ headers={"X-Request-ID": record.request_id, "X-Task-ID": task.task_id},
+ )
+
+
+async def _finish_request(request: Request, service: FileParseService, record: Record, timeout: int) -> Response:
+ """等待预算只包围排队、解析及导出,结果下载使用独立传输预算。"""
+ if await service.wait(record, timeout):
+ return await _attachment(service, record)
+ payload = service.payload(record, str(request.base_url))
+ headers = {"Location": payload["status_url"], "X-Request-ID": record.request_id, "Retry-After": "2"}
+ if payload["task_id"] is not None:
+ headers["X-Task-ID"] = payload["task_id"]
+ return JSONResponse(payload, status_code=202, headers=headers)
+
+
+async def _until_disconnect(request: Request, operation: asyncio.Task[Response]) -> Response:
+ """客户端离开时只停止该连接的等待或下载,不传播到托管上传任务。"""
+
+ async def disconnected() -> None:
+ """在请求体读取完毕后等待 ASGI 的真实断连消息。"""
+ while (await request.receive())["type"] != "http.disconnect":
+ pass
+
+ watcher = asyncio.create_task(disconnected())
+ try:
+ done, _ = await asyncio.wait({operation, watcher}, return_when=asyncio.FIRST_COMPLETED)
+ if operation in done:
+ return await operation
+ return Response(status_code=499)
+ finally:
+ watcher.cancel()
+ if not operation.done():
+ operation.cancel()
+ await asyncio.gather(watcher, operation, return_exceptions=True)
+
+
+def create_app(settings: FileParseSettings, *, transport: httpx.AsyncBaseTransport | None = None) -> FastAPI:
+ """构建独立包装服务,生命周期内创建数据库和 HTTP 客户端。"""
+
+ @asynccontextmanager
+ async def lifespan(application: FastAPI) -> AsyncIterator[None]:
+ """启动恢复、后台观察和关闭清理均由应用生命周期管理。"""
+ service = FileParseService(settings, transport)
+ application.state.service = service
+ try:
+ await service.start()
+ yield
+ finally:
+ await service.close()
+
+ application = FastAPI(title="MinerU File Parse", version="1", lifespan=lifespan)
+ application.add_middleware(ReceiveTimeoutMiddleware, timeout=settings.transfer_timeout)
+
+ @application.exception_handler(GatewayError)
+ async def gateway_error(request: Request, exc: GatewayError) -> JSONResponse:
+ """错误响应保留已接受请求的查询链接,但不暴露上游凭据及签名链接。"""
+ payload: dict[str, Any] = {}
+ service = request.app.state.service
+ request_id = getattr(request.state, "request_id", None)
+ record = service.records.get(request_id)
+ if record is not None:
+ payload.update(service.payload(record, str(request.base_url)))
+ payload["error"] = exc.detail()
+ return JSONResponse(payload, status_code=exc.status_code)
+
+ @application.post(
+ "/v1/file_parse",
+ openapi_extra=_request_schema(),
+ response_model=None,
+ response_class=Response,
+ responses={
+ 200: {
+ "description": "Result attachment",
+ "content": {mime: {"schema": {"type": "string", "format": "binary"}} for mime in MIME_TYPES.values()},
+ },
+ 202: {"model": StatusResponse, "description": "Accepted; query status_url to continue"},
+ },
+ )
+ async def file_parse(request: Request) -> Response:
+ """接收一个文件并同步等待,调用方可通过 request_id 恢复已有任务。"""
+ service = request.app.state.service
+ record, timeout = await _receive_file(request, service)
+ return await _until_disconnect(request, asyncio.create_task(_finish_request(request, service, record, timeout)))
+
+ @application.get("/v1/file_parse/{request_id}", response_model=StatusResponse)
+ async def status(request_id: str, request: Request) -> dict[str, Any]:
+ """查询已有请求,创建结果未知时返回明确状态而不发起新的任务。"""
+ request.state.request_id = request_id
+ service = request.app.state.service
+ record = service.get(request_id)
+ if record.task is not None:
+ await service.refresh(record)
+ return service.payload(record, str(request.base_url))
+
+ @application.get("/v1/file_parse/{request_id}/result", response_class=FileResponse)
+ async def result(request_id: str, request: Request) -> Response:
+ """使用当前有效的上游链接获取结果,断连时释放本次下载临时文件。"""
+ request.state.request_id = request_id
+ service = request.app.state.service
+ record = service.get(request_id)
+ if record.task is None:
+ service.raise_record_error(record)
+ return await _until_disconnect(request, asyncio.create_task(_attachment(service, record)))
+
+ return application
+
+
+__all__ = ["StatusResponse", "create_app"]
diff --git a/mineru/kit/file_parse/client.py b/mineru/kit/file_parse/client.py
new file mode 100644
index 00000000..d1654bae
--- /dev/null
+++ b/mineru/kit/file_parse/client.py
@@ -0,0 +1,290 @@
+"""只通过共同 Tasks HTTP 契约工作的流式客户端。"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import time
+import zipfile
+from collections.abc import AsyncIterator, Mapping
+from pathlib import Path
+from typing import Any
+from urllib.parse import urljoin
+
+import httpx
+from pydantic import ValidationError
+
+from .models import GatewayError, Record, RemoteTask
+from .settings import FileParseSettings, origin
+
+CHUNK_BYTES = 1024 * 1024
+MAX_JSON_BYTES = 4 * 1024 * 1024
+RETRYABLE_HTTP = frozenset({408, 429, 500, 502, 503, 504})
+REDIRECTS = frozenset({301, 302, 303, 307, 308})
+
+
+async def file_chunks(path: Path) -> AsyncIterator[bytes]:
+ """从磁盘分块读取上传内容,不在事件循环中执行大块文件 I/O。"""
+ with path.open("rb") as source:
+ while chunk := await asyncio.to_thread(source.read, CHUNK_BYTES):
+ yield chunk
+
+
+def validate_artifact(path: Path, formats: list[str]) -> None:
+ """校验下载的基本格式和 ZIP 完整性,不解压文件或重新生成产物。"""
+ try:
+ if path.stat().st_size == 0:
+ raise ValueError("Empty artifact")
+ if formats == ["html"]:
+ with path.open("r", encoding="utf-8-sig") as source:
+ prefix = source.read(65536).lower()
+ if "<html" not in prefix:
+ raise ValueError("Expected an HTML document")
+ else:
+ with zipfile.ZipFile(path) as archive:
+ if archive.testzip() is not None:
+ raise ValueError("Corrupt ZIP member")
+ if formats == ["docx"] and "word/document.xml" not in archive.namelist():
+ raise ValueError("Expected a DOCX document")
+ except (ValueError, UnicodeError, zipfile.BadZipFile, RuntimeError) as exc:
+ raise GatewayError("invalid_result", "Upstream returned an invalid or incomplete result", retryable=True) from exc
+
+
+class TaskClient:
+ """所有请求显式选择认证;创建任务始终只发送一次。"""
+
+ def __init__(self, settings: FileParseSettings, transport: httpx.AsyncBaseTransport | None = None) -> None:
+ """创建轻量连接池,测试可以注入传输而不替换业务实现。"""
+ self.settings = settings
+ self.origin = origin(settings.api_url)
+ self.client = httpx.AsyncClient(
+ timeout=httpx.Timeout(settings.request_timeout, connect=settings.connect_timeout),
+ follow_redirects=False,
+ transport=transport,
+ )
+ self.transfer_timeout = httpx.Timeout(settings.transfer_timeout, connect=settings.connect_timeout)
+ self.retry_delay = 1.0
+
+ async def close(self) -> None:
+ """释放上游连接池,停止服务时不发送任务取消请求。"""
+ await self.client.aclose()
+
+ def resolve(self, url: str, *, base: str | None = None, metadata: bool = False) -> str:
+ """保留签名查询串,元数据查询必须留在已配置的 API 同源地址。"""
+ resolved = urljoin(base or self.settings.api_url + "/", url)
+ try:
+ target = origin(resolved)
+ except ValueError as exc:
+ raise GatewayError("upstream_protocol_error", "Upstream returned an invalid HTTP URL") from exc
+ if metadata and target != self.origin:
+ raise GatewayError("upstream_protocol_error", "Task status URL must use the configured API origin")
+ return resolved
+
+ def headers(self, url: str, supplied: Mapping[str, str] | None = None) -> dict[str, str]:
+ """只在 API 同源请求中添加密钥,保留存储服务显式要求的上传请求头。"""
+ try:
+ headers = httpx.Headers(supplied or {})
+ except (UnicodeError, ValueError) as exc:
+ raise GatewayError("upstream_protocol_error", "Upstream returned invalid upload headers") from exc
+ if any("\r" in key + value or "\n" in key + value for key, value in headers.items()):
+ raise GatewayError("upstream_protocol_error", "Upstream returned invalid upload headers")
+ token = f"Bearer {self.settings.api_key}" if self.settings.api_key else None
+ if origin(url) == self.origin:
+ if token is not None and "authorization" not in headers:
+ headers["Authorization"] = token
+ elif token is not None and headers.get("Authorization") == token:
+ raise GatewayError("upstream_protocol_error", "Refusing to forward the API key to another origin")
+ return dict(headers)
+
+ async def _read_metadata(self, response: httpx.Response) -> bytes:
+ """限制元数据响应的总大小,避免意外把产物当成 JSON 缓冲。"""
+ content = bytearray()
+ async for chunk in response.aiter_bytes():
+ content.extend(chunk)
+ if len(content) > MAX_JSON_BYTES:
+ raise GatewayError("upstream_protocol_error", "Upstream metadata response is too large")
+ return bytes(content)
+
+ def _http_error(self, status: int, body: bytes) -> GatewayError:
+ """保留上游结构化错误码,同时区分配置认证失败和可重试响应。"""
+ code, message = "upstream_http_error", f"Upstream returned HTTP {status}"
+ try:
+ payload = json.loads(body)
+ detail = payload.get("error") if isinstance(payload, dict) else None
+ if isinstance(detail, dict):
+ code = str(detail.get("code") or code)
+ message = str(detail.get("message") or message)
+ except (ValueError, UnicodeError):
+ pass
+ return GatewayError(code, message, retryable=status in RETRYABLE_HTTP, upstream_status=status)
+
+ def _snapshot(self, body: bytes, expected: RemoteTask | None, formats: list[str]) -> RemoteTask:
+ """校验任务 ID、单文件映射和输出格式,拒绝不匹配的任务响应。"""
+ try:
+ task = RemoteTask.model_validate_json(body)
+ if expected is not None and task.task_id != expected.task_id:
+ raise ValueError("Task identity changed")
+ if list(dict.fromkeys(task.output_formats)) != formats:
+ raise ValueError("Task output formats changed")
+ task.status_url = self.resolve(task.status_url, metadata=True)
+ for link in task.uploads:
+ link.upload_url = self.resolve(link.upload_url)
+ if task.files[0].result_url:
+ task.files[0].result_url = self.resolve(task.files[0].result_url)
+ return task
+ except (ValidationError, ValueError) as exc:
+ raise GatewayError("upstream_protocol_error", "Upstream returned inconsistent task metadata") from exc
+
+ async def _json_request(
+ self,
+ method: str,
+ url: str,
+ formats: list[str],
+ *,
+ expected: RemoteTask | None = None,
+ payload: dict[str, Any] | None = None,
+ ) -> RemoteTask:
+ """安全查询有界重试,创建响应丢失或内容不可用则标记结果不确定。"""
+ for attempt in range(1 if method == "POST" else 3):
+ try:
+ async with self.client.stream(method, url, headers=self.headers(url), json=payload) as response:
+ body = await self._read_metadata(response)
+ if response.status_code not in ({202} if method == "POST" else {200}):
+ error = self._http_error(response.status_code, body)
+ if method == "POST" and (response.status_code >= 500 or response.status_code == 408):
+ raise GatewayError("submission_unknown", "Task creation outcome is unknown; not resubmitting")
+ raise error
+ return self._snapshot(body, expected, formats)
+ except (httpx.HTTPError, GatewayError) as exc:
+ error = (
+ exc
+ if isinstance(exc, GatewayError)
+ else GatewayError("upstream_unreachable", "Cannot communicate with the upstream service", retryable=True)
+ )
+ if method == "POST":
+ if isinstance(exc, httpx.HTTPError) or error.code == "upstream_protocol_error":
+ error = GatewayError("submission_unknown", "Task creation outcome is unknown; not resubmitting")
+ raise error from exc
+ if attempt == 2 or not error.retryable:
+ raise error from exc
+ await asyncio.sleep(self.retry_delay * (attempt + 1))
+ raise AssertionError("unreachable")
+
+ async def create(self, record: Record) -> RemoteTask:
+ """只提交一条清单,hash 用于上游缓存预检及上传完整性检查。"""
+ options = {key: value for key, value in record.options.items() if value is not None}
+ payload = {
+ **options,
+ "manifest": [
+ {
+ "client_file_id": "file",
+ "filename": record.filename,
+ "bytes": record.size,
+ "sha256sum": record.sha256sum,
+ }
+ ],
+ }
+ return await self._json_request(
+ "POST", self.settings.api_url + "/v1/tasks", record.options["output_formats"], payload=payload
+ )
+
+ async def query(self, task: RemoteTask) -> RemoteTask:
+ """按服务给出的地址查询原任务,失败不会触发创建操作。"""
+ return await self._json_request("GET", task.status_url, task.output_formats, expected=task)
+
+ async def upload(self, task: RemoteTask, path: Path, size: int) -> RemoteTask:
+ """上传失败先查询确认,最多重传三次,绝不续期或创建替代任务。"""
+ for attempt in range(3):
+ item = task.files[0]
+ if item.upload_status == "ready" or item.status in {"completed", "failed", "canceled"}:
+ return task
+ if not task.uploads:
+ raise GatewayError("upstream_protocol_error", "Pending file has no upload link")
+ link = task.uploads[0]
+ if link.expires_at <= time.time():
+ raise GatewayError("upload_expired", "The task upload link has expired", 410)
+ try:
+ headers = self.headers(link.upload_url, link.upload_headers)
+ headers["content-length"] = str(size)
+ async with self.client.stream(
+ link.upload_method,
+ link.upload_url,
+ headers=headers,
+ content=file_chunks(path),
+ timeout=self.transfer_timeout,
+ ) as response:
+ body = await self._read_metadata(response)
+ if not 200 <= response.status_code < 300:
+ raise self._http_error(response.status_code, body)
+ return await self.query(task)
+ except (httpx.HTTPError, GatewayError) as exc:
+ error = (
+ exc
+ if isinstance(exc, GatewayError)
+ else GatewayError("upload_interrupted", "Upstream upload was interrupted", retryable=True)
+ )
+ task = await self.query(task)
+ item = task.files[0]
+ if item.upload_status == "ready" or item.status in {"completed", "failed", "canceled"}:
+ return task
+ if attempt == 2 or not error.retryable:
+ raise error from exc
+ await asyncio.sleep(self.retry_delay * (attempt + 1))
+ raise AssertionError("unreachable")
+
+ async def _download_once(self, url: str, path: Path) -> None:
+ """逐跳重新决定认证并流式写盘,在完整接收前不向调用者发送成功响应。"""
+ for _ in range(6):
+ async with self.client.stream(
+ "GET",
+ url,
+ headers=self.headers(url),
+ timeout=self.transfer_timeout,
+ ) as response:
+ if response.status_code in REDIRECTS:
+ location = response.headers.get("location")
+ if not location:
+ raise GatewayError("upstream_protocol_error", "Download redirect has no Location")
+ url = self.resolve(location, base=url)
+ continue
+ if response.status_code != 200:
+ raise self._http_error(response.status_code, await self._read_metadata(response))
+ size = 0
+ with path.open("wb") as target:
+ async for chunk in response.aiter_bytes(CHUNK_BYTES):
+ await asyncio.to_thread(target.write, chunk)
+ size += len(chunk)
+ length = response.headers.get("content-length")
+ if length is not None and response.headers.get("content-encoding", "identity") == "identity":
+ try:
+ complete = int(length) == size
+ except ValueError:
+ complete = False
+ if not complete:
+ raise GatewayError("download_incomplete", "Result download is incomplete", retryable=True)
+ return
+ raise GatewayError("upstream_protocol_error", "Too many result download redirects")
+
+ async def download(self, task: RemoteTask, path: Path) -> RemoteTask:
+ """下载错误或签名失效时查询原任务,再对可恢复错误执行有界重试。"""
+ for attempt in range(3):
+ if task.files[0].status != "completed" or not task.files[0].result_url:
+ raise GatewayError("result_not_ready", "The result is not ready", 409)
+ try:
+ await self._download_once(task.files[0].result_url, path)
+ await asyncio.to_thread(validate_artifact, path, task.output_formats)
+ return task
+ except (httpx.HTTPError, GatewayError) as exc:
+ error = (
+ exc
+ if isinstance(exc, GatewayError)
+ else GatewayError("download_interrupted", "Result download was interrupted", retryable=True)
+ )
+ if attempt == 2 or (not error.retryable and error.upstream_status not in {401, 403, 404}):
+ raise error from exc
+ task = await self.query(task)
+ await asyncio.sleep(self.retry_delay * (attempt + 1))
+ raise AssertionError("unreachable")
+
+
+__all__ = ["TaskClient", "validate_artifact"]
diff --git a/mineru/kit/file_parse/models.py b/mineru/kit/file_parse/models.py
new file mode 100644
index 00000000..ec2aef0a
--- /dev/null
+++ b/mineru/kit/file_parse/models.py
@@ -0,0 +1,136 @@
+"""同步包装服务的请求、远端快照和持久化记录。"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from dataclasses import asdict, dataclass
+from typing import Any, Literal
+
+from pydantic import BaseModel, Field, model_validator
+
+from ...task_types import TaskOutputFormat
+
+TERMINAL = frozenset({"completed", "failed", "canceled", "submission_unknown"})
+FileStatus = Literal["waiting_upload", "queued", "running", "completed", "failed", "canceled"]
+
+
+class GatewayError(Exception):
+ """保留结构化错误,避免把含签名地址或密钥的传输异常直接返回。"""
+
+ def __init__(
+ self,
+ code: str,
+ message: str,
+ status_code: int = 502,
+ *,
+ retryable: bool = False,
+ upstream_status: int | None = None,
+ ) -> None:
+ """区分调用参数、远端业务错误与可安全重试的传输错误。"""
+ super().__init__(message)
+ self.code = code
+ self.message = message
+ self.status_code = status_code
+ self.retryable = retryable
+ self.upstream_status = upstream_status
+
+ def detail(self) -> dict[str, Any]:
+ """生成可持久化且不包含底层请求对象的错误信息。"""
+ return {"code": self.code, "message": self.message, "upstream_status": self.upstream_status}
+
+
+class ParseOptions(BaseModel):
+ """保持与 Tasks 一致的解析参数,等待时间不属于解析身份。"""
+
+ tier: Literal["flash", "basic", "standard", "advanced"] | None = None
+ ocr_mode: Literal["auto", "txt", "ocr"] = "auto"
+ page_range: str = ""
+ output_formats: list[TaskOutputFormat] = Field(default_factory=lambda: ["default"], min_length=1)
+
+ @model_validator(mode="after")
+ def normalize(self) -> ParseOptions:
+ """按首次出现去重格式,并统一整本页范围的等价写法。"""
+ self.output_formats = list(dict.fromkeys(self.output_formats))
+ self.page_range = self.page_range.strip()
+ if self.page_range.lower() == "all":
+ self.page_range = ""
+ return self
+
+
+class RemoteFile(BaseModel):
+ """仅消费共同 Tasks 契约,不依赖本地 Files 或 Job 实现。"""
+
+ client_file_id: str
+ status: FileStatus
+ upload_status: Literal["pending", "ready", "not_required", "expired"]
+ result_url: str | None = None
+ error: dict[str, Any] | None = None
+
+
+class UploadLink(BaseModel):
+ """保留原始上传地址、请求头和固定截止时间。"""
+
+ client_file_id: str
+ upload_url: str
+ upload_method: Literal["PUT"]
+ upload_headers: dict[str, str]
+ expires_at: int
+
+
+class RemoteTask(BaseModel):
+ """校验单文件任务的身份和产物契约,允许上游添加其他元数据。"""
+
+ task_id: str = Field(min_length=1)
+ status: Literal["waiting_upload", "queued", "running", "completed", "partial", "failed", "canceled"]
+ status_url: str = Field(min_length=1)
+ files: list[RemoteFile] = Field(min_length=1, max_length=1)
+ uploads: list[UploadLink] = Field(default_factory=list, max_length=1)
+ output_formats: list[TaskOutputFormat] = Field(min_length=1)
+
+ @model_validator(mode="after")
+ def validate_file(self) -> RemoteTask:
+ """确保返回的文件和链接仍对应本次唯一清单项。"""
+ item = self.files[0]
+ if item.client_file_id != "file" or any(link.client_file_id != "file" for link in self.uploads):
+ raise ValueError("Unexpected client_file_id")
+ if item.status == "completed" and not item.result_url:
+ raise ValueError("Completed file has no result_url")
+ if item.status != "completed" and item.result_url is not None:
+ raise ValueError("Unfinished file has a result_url")
+ return self
+
+
+@dataclass
+class Record:
+ """保存恢复所需的最小状态;密钥仅来自启动配置,不写入记录。"""
+
+ request_id: str
+ filename: str
+ size: int
+ sha256sum: str
+ options: dict[str, Any]
+ source_path: str | None
+ fingerprint: str
+ status: str = "prepared"
+ remote: dict[str, Any] | None = None
+ error: dict[str, Any] | None = None
+ terminal_at: float | None = None
+
+ @property
+ def task(self) -> RemoteTask | None:
+ """读取已校验远端快照,空值表示尚未确认创建结果。"""
+ return RemoteTask.model_validate(self.remote) if self.remote is not None else None
+
+ def to_json(self) -> str:
+ """把完整记录序列化为 SQLite 单行快照。"""
+ return json.dumps(asdict(self), ensure_ascii=False)
+
+
+def fingerprint(filename: str, size: int, sha256sum: str, options: ParseOptions) -> str:
+ """用文件内容、逻辑名称与归一化参数区分同 ID 的冲突提交。"""
+ value = {"filename": filename, "size": size, "sha256sum": sha256sum, "options": options.model_dump()}
+ return hashlib.sha256(json.dumps(value, ensure_ascii=False, sort_keys=True).encode()).hexdigest()
+
+
+__all__ = ["GatewayError", "ParseOptions", "Record", "RemoteTask", "TERMINAL", "UploadLink", "fingerprint"]
diff --git a/mineru/kit/file_parse/service.py b/mineru/kit/file_parse/service.py
new file mode 100644
index 00000000..fb63d298
--- /dev/null
+++ b/mineru/kit/file_parse/service.py
@@ -0,0 +1,332 @@
+"""把请求连接与持久化的 Tasks 上传、查询生命周期分离。"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+import os
+import tempfile
+import time
+from dataclasses import replace
+from pathlib import Path
+from typing import Any
+
+import httpx
+
+from .client import TaskClient
+from .models import TERMINAL, GatewayError, ParseOptions, Record, RemoteTask, fingerprint
+from .settings import FileParseSettings
+from .store import RecordStore
+
+logger = logging.getLogger(__name__)
+
+
+class FileParseService:
+ """单进程管理恢复记录;每个请求只有一个独立的上传准备任务。"""
+
+ def __init__(self, settings: FileParseSettings, transport: httpx.AsyncBaseTransport | None = None) -> None:
+ """初始化无副作用的管理状态,网络连接在应用启动后使用。"""
+ self.settings = settings
+ self.store = RecordStore(settings)
+ self.client = TaskClient(settings, transport)
+ self.records: dict[str, Record] = {}
+ self.preparations: dict[str, asyncio.Task[None]] = {}
+ self.changes: dict[str, asyncio.Event] = {}
+ self.query_locks: dict[str, asyncio.Lock] = {}
+ self.state_locks: dict[str, asyncio.Lock] = {}
+ self._accept_lock = asyncio.Lock()
+ self._query_limit = asyncio.Semaphore(16)
+ self._monitor: asyncio.Task[None] | None = None
+
+ async def start(self) -> None:
+ """恢复已有任务;曾经开始但未确认的创建请求不能自动重发。"""
+ for record in await self.store.open():
+ self.records[record.request_id] = record
+ if record.status == "submitting" and record.remote is None:
+ await self._remember_error(
+ record, GatewayError("submission_unknown", "Task creation outcome is unknown; not resubmitting")
+ )
+ elif record.status not in TERMINAL:
+ self._prepare_in_background(record)
+ await self.collect()
+ self._monitor = asyncio.create_task(self._monitor_records(), name="file-parse-monitor")
+
+ async def close(self) -> None:
+ """停止本地协程并保存已确认的任务状态,不取消上游解析。"""
+ tasks = [*self.preparations.values()]
+ if self._monitor is not None:
+ tasks.append(self._monitor)
+ for task in tasks:
+ task.cancel()
+ await asyncio.gather(*tasks, return_exceptions=True)
+ await self.client.close()
+ await self.store.close()
+
+ def _changed(self, record: Record) -> None:
+ """唤醒正在等待该请求状态变化的连接。"""
+ self.changes.setdefault(record.request_id, asyncio.Event()).set()
+
+ def _prepare_in_background(self, record: Record) -> None:
+ """把任务创建及上传托管到应用,连接取消不会传播到这里。"""
+ previous = self.preparations.get(record.request_id)
+ if previous is None or previous.done():
+ self.preparations[record.request_id] = asyncio.create_task(
+ self._prepare(record), name=f"file-parse-prepare-{record.request_id}"
+ )
+
+ async def accept(
+ self,
+ request_id: str,
+ filename: str,
+ size: int,
+ sha256sum: str,
+ options: ParseOptions,
+ source: Path,
+ ) -> Record:
+ """原子接受或复用请求,同 ID 不同内容不会替换原任务或源文件。"""
+ identity = fingerprint(filename, size, sha256sum, options)
+ async with self._accept_lock:
+ existing = self.records.get(request_id)
+ if existing is not None and self._expired(existing):
+ await self._remove_record(existing)
+ existing = None
+ if existing is not None:
+ source.unlink(missing_ok=True)
+ if existing.fingerprint != identity:
+ raise GatewayError("request_id_conflict", "request_id already identifies a different request", 409)
+ if existing.status not in TERMINAL:
+ self._prepare_in_background(existing)
+ return existing
+ record = Record(request_id, filename, size, sha256sum, options.model_dump(), str(source), identity)
+ await self.store.save(record)
+ self.records[request_id] = record
+ self._prepare_in_background(record)
+ return record
+
+ def get(self, request_id: str) -> Record:
+ """仅在当前上游命名空间内查找请求;过期记录等同于不存在。"""
+ record = self.records.get(request_id)
+ if record is None or self._expired(record):
+ raise GatewayError("request_not_found", "Request was not found or its recovery record expired", 404)
+ return record
+
+ def _expired(self, record: Record) -> bool:
+ """终态首次观察时间决定保留期限,重复查询不续期。"""
+ return bool(
+ self.settings.retention_seconds
+ and record.terminal_at is not None
+ and record.terminal_at + self.settings.retention_seconds <= time.time()
+ )
+
+ async def _save_and_cleanup(self, record: Record, updated: Record, *, remove_source: bool) -> None:
+ """持久化和清理完成后才发布新状态,避免并发查询提前观察完成状态。"""
+ source = updated.source_path if remove_source else None
+ if remove_source:
+ updated.source_path = None
+ await self.store.save(updated)
+ try:
+ if source:
+ Path(source).unlink(missing_ok=True)
+ finally:
+ # 即使文件清理失败,也不能丢掉已经持久化的远端 task_id。
+ record.remote = updated.remote
+ record.status = updated.status
+ record.source_path = updated.source_path
+ record.error = updated.error
+ record.terminal_at = updated.terminal_at
+ self._changed(record)
+
+ async def _apply(self, record: Record, task: RemoteTask) -> None:
+ """更新已确认任务快照,并避免旧轮询覆盖较新的终态。"""
+ async with self.state_locks.setdefault(record.request_id, asyncio.Lock()):
+ if self.records.get(record.request_id) is not record:
+ return
+ item = task.files[0]
+ if record.remote is not None and record.status in TERMINAL and item.status not in TERMINAL:
+ return
+ updated = replace(record, remote=task.model_dump(), status=item.status)
+ if item.upload_status == "ready" or item.status in TERMINAL:
+ updated.error = item.error
+ if item.status in {"failed", "canceled"} and updated.error is None:
+ updated.error = {"code": f"task_{item.status}", "message": f"Upstream task {item.status}"}
+ if updated.status in TERMINAL and updated.terminal_at is None:
+ updated.terminal_at = time.time()
+ await self._save_and_cleanup(
+ record, updated, remove_source=item.upload_status == "ready" or item.status in TERMINAL
+ )
+
+ async def _remember_error(self, record: Record, error: GatewayError) -> None:
+ """无任务 ID 的失败是本地终态;已知任务的传输错误保留后续查询能力。"""
+ async with self.state_locks.setdefault(record.request_id, asyncio.Lock()):
+ if self.records.get(record.request_id) is not record:
+ return
+ updated = replace(record, error={**error.detail(), "http_status": error.status_code})
+ remove_source = False
+ if updated.remote is None:
+ updated.status = "submission_unknown" if error.code == "submission_unknown" else "failed"
+ updated.terminal_at = updated.terminal_at or time.time()
+ remove_source = True
+ elif error.upstream_status in {404, 410}:
+ updated.status = "failed"
+ updated.terminal_at = updated.terminal_at or time.time()
+ remove_source = True
+ await self._save_and_cleanup(record, updated, remove_source=remove_source)
+
+ async def _prepare(self, record: Record) -> None:
+ """创建和上传最多执行一条流水线,上传阶段错误仍允许查询已有任务。"""
+ try:
+ record.error = None
+ if record.remote is None:
+ record.status = "submitting"
+ await self.store.save(record)
+ task = await self.client.create(record)
+ await self._apply(record, task)
+ else:
+ task = await self.refresh(record)
+ if task.files[0].upload_status == "pending" and task.files[0].status not in TERMINAL:
+ if record.source_path is None or not Path(record.source_path).is_file():
+ raise GatewayError("source_unavailable", "The retained source is unavailable")
+ task = await self.client.upload(task, Path(record.source_path), record.size)
+ await self._apply(record, task)
+ except GatewayError as exc:
+ await self._remember_error(record, exc)
+ except OSError:
+ await self._remember_error(record, GatewayError("storage_error", "Cannot access the local source storage", 500))
+ finally:
+ self._changed(record)
+
+ async def refresh(self, record: Record) -> RemoteTask:
+ """序列化同一请求的状态查询,限制后台查询总并发。"""
+ async with self.query_locks.setdefault(record.request_id, asyncio.Lock()):
+ task = record.task
+ if task is None:
+ raise GatewayError("task_not_created", "No confirmed upstream task is available", 409)
+ async with self._query_limit:
+ current = await self.client.query(task)
+ await self._apply(record, current)
+ return current
+
+ async def _refresh_safely(self, record: Record) -> None:
+ """后台查询临时失败只更新错误,不使其他请求的轮询停止。"""
+ try:
+ await self.refresh(record)
+ except GatewayError as exc:
+ await self._remember_error(record, exc)
+
+ async def _monitor_records(self) -> None:
+ """持续观察已提交任务,使断连任务也能进入终态和开始保留期。"""
+ while True:
+ await asyncio.sleep(self.settings.poll_interval)
+ records = [
+ record
+ for record in self.records.values()
+ if record.remote is not None
+ and record.status not in TERMINAL
+ and (record.request_id not in self.preparations or self.preparations[record.request_id].done())
+ ]
+ try:
+ results = await asyncio.gather(*(self._refresh_safely(record) for record in records), return_exceptions=True)
+ for result in results:
+ if isinstance(result, Exception):
+ logger.error("File parse reconciliation error: %s", type(result).__name__)
+ await self.collect()
+ except Exception:
+ logger.exception("File parse record reconciliation failed")
+
+ async def collect(self) -> None:
+ """回收超过保留期的终态记录及残留源文件,不触碰上游资源。"""
+ async with self._accept_lock:
+ for request_id, record in list(self.records.items()):
+ operation = self.preparations.get(request_id)
+ if not self._expired(record) or (operation is not None and not operation.done()):
+ continue
+ await self._remove_record(record)
+
+ async def _remove_record(self, record: Record) -> None:
+ """与状态提交串行回收记录,迟到的查询不能重新插入已过期的旧任务。"""
+ request_id = record.request_id
+ async with self.state_locks.setdefault(request_id, asyncio.Lock()):
+ await self.store.delete(request_id)
+ if record.source_path:
+ Path(record.source_path).unlink(missing_ok=True)
+ self.records.pop(request_id, None)
+ self.preparations.pop(request_id, None)
+ self.changes.pop(request_id, None)
+ self.query_locks.pop(request_id, None)
+ self.state_locks.pop(request_id, None)
+
+ async def wait(self, record: Record, timeout: int) -> bool:
+ """上游文件准备完成后才启动单调时钟预算,连接取消不取消上传任务。"""
+ preparation = self.preparations.get(record.request_id)
+ if preparation is not None:
+ await asyncio.shield(preparation)
+ changed = self.changes.setdefault(record.request_id, asyncio.Event())
+ while True:
+ changed.clear()
+ self.raise_record_error(record)
+ task = record.task
+ if record.status in TERMINAL or (task is not None and task.files[0].upload_status == "ready"):
+ break
+ # 对象存储 PUT 成功后,网关可能稍后才确认 ready;这段等待仍属于上传准备。
+ await changed.wait()
+ if timeout == 0:
+ return False
+ deadline = asyncio.get_running_loop().time() + timeout
+ while True:
+ changed.clear()
+ self.raise_record_error(record)
+ if record.status == "completed":
+ return True
+ remaining = deadline - asyncio.get_running_loop().time()
+ if remaining <= 0:
+ return False
+ try:
+ await asyncio.wait_for(changed.wait(), remaining)
+ except asyncio.TimeoutError:
+ return record.status == "completed"
+
+ def raise_record_error(self, record: Record) -> None:
+ """把已知失败或传输异常转成结构化 HTTP 错误,保留记录供查询。"""
+ if record.error is not None:
+ default_status = 409 if record.status in {"canceled", "submission_unknown"} else 502
+ raise GatewayError(
+ str(record.error.get("code") or "task_failed"),
+ str(record.error.get("message") or "Upstream task failed"),
+ int(record.error.get("http_status") or default_status),
+ upstream_status=record.error.get("upstream_status"),
+ )
+
+ def payload(self, record: Record, base_url: str) -> dict[str, Any]:
+ """只返回包装服务的公共链接,隐藏上游上传地址和签名下载地址。"""
+ status_url = f"{base_url.rstrip('/')}/v1/file_parse/{record.request_id}"
+ task = record.task
+ return {
+ "request_id": record.request_id,
+ "task_id": task.task_id if task else None,
+ "status": record.status,
+ "status_url": status_url,
+ "result_url": status_url + "/result" if record.status == "completed" else None,
+ "output_formats": record.options["output_formats"],
+ "error": record.error,
+ }
+
+ async def artifact(self, record: Record) -> Path:
+ """获取最新下载链接并验证临时产物,失败时立即清理临时文件。"""
+ task = await self.refresh(record)
+ if task.files[0].status in {"failed", "canceled"}:
+ self.raise_record_error(record)
+ if task.files[0].status != "completed":
+ raise GatewayError("result_not_ready", "The result is not ready", 409)
+ descriptor, name = tempfile.mkstemp(prefix="result-", dir=self.store.results)
+ os.close(descriptor)
+ path = Path(name)
+ try:
+ task = await self.client.download(task, path)
+ await self._apply(record, task)
+ return path
+ except BaseException:
+ path.unlink(missing_ok=True)
+ raise
+
+
+__all__ = ["FileParseService"]
diff --git a/mineru/kit/file_parse/settings.py b/mineru/kit/file_parse/settings.py
new file mode 100644
index 00000000..1e5d7d1e
--- /dev/null
+++ b/mineru/kit/file_parse/settings.py
@@ -0,0 +1,62 @@
+"""同步包装服务的显式启动配置。"""
+
+from __future__ import annotations
+
+import hashlib
+import math
+import os
+from dataclasses import dataclass
+from pathlib import Path
+from urllib.parse import urlsplit
+
+
+def origin(url: str) -> tuple[str, str, int]:
+ """严格比较协议、主机和端口,并拒绝 URL 内嵌凭据。"""
+ parts = urlsplit(url)
+ if parts.scheme not in {"http", "https"} or not parts.hostname or parts.username or parts.password:
+ raise ValueError("Expected an HTTP(S) URL without embedded credentials")
+ return parts.scheme.lower(), parts.hostname.lower(), parts.port or (443 if parts.scheme == "https" else 80)
+
+
+@dataclass(frozen=True)
+class FileParseSettings:
+ """只在创建配置实例时读取默认目录,不在模块导入时读取环境。"""
+
+ api_url: str
+ api_key: str | None = None
+ state_dir: Path | None = None
+ retention_seconds: int = 86400
+ poll_interval: float = 2.0
+ connect_timeout: float = 10.0
+ request_timeout: float = 30.0
+ transfer_timeout: float = 600.0
+
+ def __post_init__(self) -> None:
+ """校验地址和预算,禁止静默选择官方远程上游。"""
+ normalized = self.api_url.rstrip("/")
+ origin(normalized)
+ parts = urlsplit(normalized)
+ if parts.query or parts.fragment:
+ raise ValueError("api_url must not contain a query or fragment")
+ if self.api_key and any(character in self.api_key for character in "\r\n"):
+ raise ValueError("api_key must not contain line breaks")
+ object.__setattr__(self, "api_url", normalized)
+ if self.state_dir is None:
+ home = Path(os.environ.get("MINERU_HOME") or "~/.mineru").expanduser()
+ object.__setattr__(self, "state_dir", home / "file-parse")
+ else:
+ object.__setattr__(self, "state_dir", Path(self.state_dir).expanduser().resolve())
+ if type(self.retention_seconds) is not int or self.retention_seconds < 0:
+ raise ValueError("retention_seconds must be a non-negative integer")
+ for name in ("poll_interval", "connect_timeout", "request_timeout", "transfer_timeout"):
+ value = getattr(self, name)
+ if not math.isfinite(value) or value <= 0:
+ raise ValueError(f"{name} must be positive and finite")
+
+ @property
+ def namespace(self) -> str:
+ """按上游地址和认证隔离恢复记录,不存储明文密钥。"""
+ return hashlib.sha256(f"{self.api_url}\0{self.api_key or ''}".encode()).hexdigest()
+
+
+__all__ = ["FileParseSettings", "origin"]
diff --git a/mineru/kit/file_parse/store.py b/mineru/kit/file_parse/store.py
new file mode 100644
index 00000000..822ff251
--- /dev/null
+++ b/mineru/kit/file_parse/store.py
@@ -0,0 +1,85 @@
+"""独立于 Doclib 的 SQLite 恢复记录与临时文件管理。"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+
+import aiosqlite
+from filelock import FileLock
+
+from .models import Record
+from .settings import FileParseSettings
+
+
+class RecordStore:
+ """每个状态目录只允许一个进程使用,各上游配置拥有独立数据空间。"""
+
+ def __init__(self, settings: FileParseSettings) -> None:
+ """只保存路径和锁对象,实际创建资源由应用启动阶段负责。"""
+ assert settings.state_dir is not None
+ self.root = settings.state_dir
+ self.directory = self.root / settings.namespace
+ self.sources = self.directory / "sources"
+ self.results = self.directory / "results"
+ self.lock = FileLock(str(self.root / ".lock"), thread_local=False)
+ self.connection: aiosqlite.Connection | None = None
+ self._writes = asyncio.Lock()
+
+ async def open(self) -> list[Record]:
+ """获取独占锁并加载记录,启动时清理上次进程遗留的结果临时文件。"""
+ self.root.mkdir(parents=True, exist_ok=True)
+ self.lock.acquire(timeout=0)
+ try:
+ self.sources.mkdir(parents=True, exist_ok=True)
+ self.results.mkdir(parents=True, exist_ok=True)
+ for path in self.results.iterdir():
+ if path.is_file():
+ path.unlink()
+ self.connection = await aiosqlite.connect(self.directory / "requests.sqlite3")
+ await self.connection.execute("PRAGMA journal_mode=WAL")
+ await self.connection.execute("PRAGMA synchronous=FULL")
+ await self.connection.execute(
+ "CREATE TABLE IF NOT EXISTS requests (request_id TEXT PRIMARY KEY, data TEXT NOT NULL)"
+ )
+ await self.connection.commit()
+ async with self.connection.execute("SELECT data FROM requests") as cursor:
+ records = [Record(**json.loads(row[0])) for row in await cursor.fetchall()]
+ retained = {record.source_path for record in records if record.source_path}
+ for path in self.sources.iterdir():
+ if path.is_file() and str(path) not in retained:
+ path.unlink()
+ return records
+ except BaseException:
+ await self.close()
+ raise
+
+ async def save(self, record: Record) -> None:
+ """原子提交单个记录快照,任务 ID 确认后立即落盘。"""
+ assert self.connection is not None
+ data = record.to_json()
+ async with self._writes:
+ await self.connection.execute(
+ "INSERT INTO requests(request_id, data) VALUES (?, ?) ON CONFLICT(request_id) DO UPDATE SET data=excluded.data",
+ (record.request_id, data),
+ )
+ await self.connection.commit()
+
+ async def delete(self, request_id: str) -> None:
+ """回收过期恢复记录,保留其他请求及其他上游空间。"""
+ assert self.connection is not None
+ async with self._writes:
+ await self.connection.execute("DELETE FROM requests WHERE request_id=?", (request_id,))
+ await self.connection.commit()
+
+ async def close(self) -> None:
+ """关闭数据库后释放目录独占锁,允许下次启动恢复。"""
+ try:
+ if self.connection is not None:
+ await self.connection.close()
+ self.connection = None
+ finally:
+ self.lock.release()
+
+
+__all__ = ["RecordStore"]
diff --git a/mineru/kit/main.py b/mineru/kit/main.py
index 57ebf608..48c59f4a 100644
--- a/mineru/kit/main.py
+++ b/mineru/kit/main.py
@@ -10,12 +10,13 @@ from ..cli.version_command import show_version, version_cmd
from ..utils.i18n import t
from ..utils.logger import configure_global_log_level
from ..utils.stdio import configure_standard_streams
-from .commands import api_server, models, parse, router, vlm_server, webui
+from .commands import api_server, file_parse_server, models, parse, router, vlm_server, webui
TOP_LEVEL_COMMAND_ORDER = [
"parse",
"webui",
"api-server",
+ "file-parse-server",
"vlm-server",
"router",
"models",
@@ -63,6 +64,9 @@ app.command(
help=t("Start the local VLM server with OpenAI-compatible chat completions."),
)(vlm_server.vlm_server_cmd)
app.command("router", help=t("Start a standalone Router service exposing only the MinerU V1 API."))(router.router_cmd)
+app.command("file-parse-server", help=t("Start the synchronous single-file Tasks wrapper service."))(
+ file_parse_server.file_parse_server_cmd
+)
app.command("version", help=t("Print MinerU and Python versions."))(version_cmd)
diff --git a/mineru/parser/api_server.py b/mineru/parser/api_server.py
index 0f200036..fc385db9 100644
--- a/mineru/parser/api_server.py
+++ b/mineru/parser/api_server.py
@@ -80,7 +80,7 @@ from .tier import (
runtime_options_for_tier,
)
from .writer import DataWriter
-from .task_outputs import TASK_OUTPUT_FORMATS, TaskOutputFormat
+from ..task_types import TASK_OUTPUT_FORMATS, TaskOutputFormat
_DEFAULT_API_SERVER_TIER: ServerTier = "standard"
_MAX_INLINE_BYTES_DEFAULT = 1024 * 1024
diff --git a/mineru/parser/task_api.py b/mineru/parser/task_api.py
index 57e8e974..9966f863 100644
--- a/mineru/parser/task_api.py
+++ b/mineru/parser/task_api.py
@@ -22,6 +22,7 @@ from starlette.requests import ClientDisconnect
from ..errors import MineruError
from ..filetypes import is_flash_only_parse_extension
+from ..task_types import TaskOutputFormat
from ..types import Tier
from ..utils.async_utils import drain_future, run_sync
from .api_server import (
@@ -44,7 +45,6 @@ from .api_server import (
submit_parse_job,
)
from .page_range import normalize_page_range_input
-from .task_outputs import TaskOutputFormat
MAX_TASK_FILES = 100
TASK_UPLOAD_TTL = 3600
diff --git a/mineru/parser/task_outputs.py b/mineru/parser/task_outputs.py
index b66894a0..57cf914e 100644
--- a/mineru/parser/task_outputs.py
+++ b/mineru/parser/task_outputs.py
@@ -5,13 +5,10 @@ from __future__ import annotations
import json
import zipfile
from pathlib import Path
-from typing import Literal
+from ..task_types import TASK_OUTPUT_FORMATS, TaskOutputFormat
from .base import ParseResult
-TaskOutputFormat = Literal["default", "docx", "html", "latex"]
-TASK_OUTPUT_FORMATS: tuple[TaskOutputFormat, ...] = ("default", "docx", "html", "latex")
-
def build_task_output(result: ParseResult, formats: list[TaskOutputFormat], directory: Path) -> Path:
"""一次物化并渲染所选格式,全部成功后才返回主产物路径。"""
diff --git a/mineru/task_types.py b/mineru/task_types.py
new file mode 100644
index 00000000..4995fb6b
--- /dev/null
+++ b/mineru/task_types.py
@@ -0,0 +1,8 @@
+"""Tasks 客户端与服务端共用的轻量格式定义,不导入解析或渲染实现。"""
+
+from typing import Literal
+
+TaskOutputFormat = Literal["default", "docx", "html", "latex"]
+TASK_OUTPUT_FORMATS: tuple[TaskOutputFormat, ...] = ("default", "docx", "html", "latex")
+
+__all__ = ["TASK_OUTPUT_FORMATS", "TaskOutputFormat"]
diff --git a/mineru/utils/translations.py b/mineru/utils/translations.py
index 133c6048..fc37fb68 100644
--- a/mineru/utils/translations.py
+++ b/mineru/utils/translations.py
@@ -489,6 +489,10 @@ ZH_MESSAGES: dict[str, str] = {
"Server host": "服务监听主机",
"Server port": "服务监听端口",
"Upload directory": "上传目录",
+ "Start the synchronous single-file Tasks wrapper service.": "启动单文件同步 Tasks 包装服务。",
+ "Upstream Tasks API base URL": "上游 Tasks API 基础 URL",
+ "Upstream Bearer API key": "上游 Bearer API 密钥",
+ "File parse recovery state directory": "同步解析请求恢复目录",
"Server capability tier: flash, basic, or standard": "服务能力档位: flash、basic 或 standard",
"Disable Flash tier advertisement and execution": "禁用 Flash 档位的广播与执行",
"Disable Advanced tier advertisement and execution": "禁用 Advanced 档位的广播与执行",
diff --git a/scripts/http_file_parse_example.py b/scripts/http_file_parse_example.py
new file mode 100644
index 00000000..58af2a04
--- /dev/null
+++ b/scripts/http_file_parse_example.py
@@ -0,0 +1,78 @@
+#!/usr/bin/env python3
+"""同步 HTTP 包装示例:提前记录请求 ID,成功保存附件,超时返回查询链接。"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import tempfile
+import uuid
+from pathlib import Path
+
+import httpx
+
+
+def main() -> int:
+ """演示标准 multipart 列表编码及二进制成功响应与 JSON 状态响应的区分。"""
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("file", type=Path)
+ parser.add_argument("--base-url", default="http://127.0.0.1:8003")
+ parser.add_argument("--output", type=Path)
+ parser.add_argument("--output-format", action="append", choices=("default", "docx", "html", "latex"))
+ parser.add_argument("--request-id", default=uuid.uuid4().hex)
+ parser.add_argument("--wait-timeout", type=int, default=3600)
+ parser.add_argument("--tier", choices=("flash", "basic", "standard", "advanced"))
+ parser.add_argument("--ocr-mode", choices=("auto", "txt", "ocr"), default="auto")
+ parser.add_argument("--page-range", default="")
+ args = parser.parse_args()
+ if not args.file.is_file():
+ parser.error("Input file does not exist")
+ formats = list(dict.fromkeys(args.output_format or ["default"]))
+ extension = formats[0] if formats in (["docx"], ["html"]) else "zip"
+ output = args.output or Path("results") / f"{args.file.stem}.{extension}"
+ if output.exists():
+ parser.error("Output already exists; choose another --output path")
+ print(f"request_id={args.request_id}", flush=True)
+ print(f"status_url={args.base_url.rstrip('/')}/v1/file_parse/{args.request_id}", flush=True)
+ fields = [
+ ("request_id", (None, args.request_id)),
+ ("wait_timeout", (None, str(args.wait_timeout))),
+ ("ocr_mode", (None, args.ocr_mode)),
+ ("page_range", (None, args.page_range)),
+ *(("output_formats", (None, value)) for value in formats),
+ ]
+ if args.tier is not None:
+ fields.append(("tier", (None, args.tier)))
+ temporary: Path | None = None
+ try:
+ with args.file.open("rb") as source, httpx.Client(timeout=httpx.Timeout(7200, connect=10, write=600)) as client:
+ with client.stream(
+ "POST", args.base_url.rstrip("/") + "/v1/file_parse", files=[("file", (args.file.name, source)), *fields]
+ ) as response:
+ if response.status_code != 200:
+ payload = response.read()
+ try:
+ print(json.dumps(json.loads(payload), ensure_ascii=False, indent=2))
+ except ValueError:
+ print(f"HTTP {response.status_code}: {payload[:4096].decode(errors='replace')}")
+ return 124 if response.status_code == 202 else 1
+ output.parent.mkdir(parents=True, exist_ok=True)
+ descriptor, name = tempfile.mkstemp(prefix=f".{output.name}.", suffix=".part", dir=output.parent)
+ temporary = Path(name)
+ with os.fdopen(descriptor, "wb") as destination:
+ for chunk in response.iter_bytes(1024 * 1024):
+ destination.write(chunk)
+ temporary.replace(output)
+ print(f"Saved {output}")
+ return 0
+ except (httpx.HTTPError, OSError) as exc:
+ print(f"Transfer failed ({type(exc).__name__}); use the printed status_url to recover")
+ return 1
+ finally:
+ if temporary is not None:
+ temporary.unlink(missing_ok=True)
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/tests/unittest/test_file_parse.py b/tests/unittest/test_file_parse.py
new file mode 100644
index 00000000..2397ed6e
--- /dev/null
+++ b/tests/unittest/test_file_parse.py
@@ -0,0 +1,654 @@
+"""同步包装服务的参数、恢复、认证和传输故障契约。"""
+
+from __future__ import annotations
+
+import asyncio
+import io
+import itertools
+import json
+import subprocess
+import sys
+import threading
+import time
+import zipfile
+from concurrent.futures import ThreadPoolExecutor
+from pathlib import Path
+from typing import Any
+
+import httpx
+import pytest
+from fastapi.testclient import TestClient
+from filelock import Timeout
+from typer.testing import CliRunner
+
+from mineru.kit.file_parse import FileParseSettings, create_app
+from mineru.kit.file_parse.app import TemporaryFileResponse
+from mineru.kit.file_parse.client import TaskClient
+from mineru.kit.file_parse.models import ParseOptions, Record, fingerprint
+from mineru.kit.file_parse.settings import origin
+from mineru.kit.file_parse.store import RecordStore
+
+
+def artifact_bytes(formats: list[str]) -> bytes:
+ """构造可校验的不同格式附件,下载字节必须原样返回。"""
+ if formats == ["html"]:
+ return b"<!doctype html><html><body>file parse result</body></html>"
+ output = io.BytesIO()
+ with zipfile.ZipFile(output, "w") as archive:
+ archive.writestr("word/document.xml" if formats == ["docx"] else "markdown.md", "file parse result")
+ return output.getvalue()
+
+
+class Upstream:
+ """模拟带路径前缀的 SaaS API 和独立对象存储,不提供 Jobs 或 Files 接口。"""
+
+ def __init__(self) -> None:
+ """初始化可控故障点及全部 HTTP 调用记录。"""
+ self.tasks: dict[str, dict[str, Any]] = {}
+ self.api_url = "https://api.test/api"
+ self.storage_url = "https://store.test"
+ self.calls: list[tuple[str, str, str | None]] = []
+ self.complete = True
+ self.cache_hit = False
+ self.create_failure: str | None = None
+ self.upload_response_lost = False
+ self.upload_failures = 0
+ self.upload_delay = 0.0
+ self.ready_delay = 0.0
+ self.complete_delay = 0.0
+ self.download_expired = False
+ self.download_corrupt = False
+ self.download_interrupted = False
+ self.redirect = False
+ self.gone = False
+ self.uploads = 0
+ self.downloads = 0
+
+ def payload(self, task_id: str) -> dict[str, Any]:
+ """返回共同协议快照,签名中的转义字符用于校验地址未被重新拼装。"""
+ task = self.tasks[task_id]
+ ready = task["ready"] and time.monotonic() >= task["uploaded_at"] + self.ready_delay
+ complete = ready and self.complete and time.monotonic() >= task["uploaded_at"] + self.complete_delay
+ expired = not ready and task["expires_at"] <= time.time()
+ status = "failed" if expired else "completed" if complete else "running" if ready else "waiting_upload"
+ result_url = f"{self.storage_url}/result/{task_id}?signature=a%2Fb&version={task['version']}" if complete else None
+ if result_url and self.redirect:
+ result_url = f"{self.api_url}/redirect/{task_id}"
+ return {
+ "task_id": task_id,
+ "status": status,
+ "status_url": f"{self.api_url}/v1/tasks/{task_id}",
+ "output_formats": task["options"]["output_formats"],
+ "files": [
+ {
+ "client_file_id": "file",
+ "status": status,
+ "upload_status": "expired" if expired else "ready" if ready else "pending",
+ "result_url": result_url,
+ "error": {"code": "upload_expired"} if expired else None,
+ }
+ ],
+ "uploads": []
+ if ready or expired
+ else [
+ {
+ "client_file_id": "file",
+ "upload_method": "PUT",
+ "upload_url": f"{self.storage_url}/upload/{task_id}",
+ "upload_headers": {"x-upload-token": "storage-token"},
+ "expires_at": task["expires_at"],
+ }
+ ],
+ }
+
+ async def handle(self, request: httpx.Request) -> httpx.Response:
+ """执行清单、上传、查询和下载,并在每一跳检查认证边界。"""
+ self.calls.append((request.method, str(request.url), request.headers.get("authorization")))
+ if origin(str(request.url)) == origin(self.api_url):
+ assert request.headers.get("authorization") == "Bearer secret"
+ else:
+ assert origin(str(request.url)) == origin(self.storage_url)
+ assert request.headers.get("authorization") is None
+ if request.method == "POST":
+ assert request.url.path == "/api/v1/tasks"
+ if self.create_failure == "auth":
+ return httpx.Response(401, json={"error": {"code": "invalid_api_key", "message": "Bad upstream key"}})
+ body = json.loads(await request.aread())
+ assert len(body["manifest"]) == 1
+ task_id = f"task_{len(self.tasks) + 1}"
+ self.tasks[task_id] = {
+ "options": body,
+ "ready": self.cache_hit,
+ "uploaded_at": time.monotonic(),
+ "version": 0,
+ "expires_at": int(time.time()) + 3600,
+ }
+ if self.create_failure == "lost":
+ raise httpx.ReadError("creation response lost")
+ payload = self.payload(task_id)
+ if self.create_failure == "malformed":
+ payload["task_id"] = None
+ return httpx.Response(202, json=payload)
+ task_id = request.url.path.rsplit("/", 1)[-1]
+ task = self.tasks[task_id]
+ if request.method == "PUT":
+ self.uploads += 1
+ assert request.headers["x-upload-token"] == "storage-token"
+ content = await request.aread()
+ assert len(content) == task["options"]["manifest"][0]["bytes"]
+ if self.upload_failures:
+ self.upload_failures -= 1
+ return httpx.Response(503)
+ await asyncio.sleep(self.upload_delay)
+ task["ready"] = True
+ task["uploaded_at"] = time.monotonic()
+ return httpx.Response(503 if self.upload_response_lost else 200)
+ if "/v1/tasks/" in request.url.path:
+ return httpx.Response(404) if self.gone else httpx.Response(200, json=self.payload(task_id))
+ if "/redirect/" in request.url.path:
+ return httpx.Response(302, headers={"Location": f"{self.storage_url}/result/{task_id}?signature=a%2Fb"})
+ self.downloads += 1
+ assert "signature=a%2Fb" in str(request.url)
+ if self.download_expired:
+ self.download_expired = False
+ task["version"] += 1
+ return httpx.Response(403)
+ if self.download_corrupt:
+ self.download_corrupt = False
+ return httpx.Response(200, content=b"corrupt result")
+ if self.download_interrupted:
+ self.download_interrupted = False
+ return httpx.Response(200, stream=InterruptedStream())
+ return httpx.Response(200, content=artifact_bytes(task["options"]["output_formats"]))
+
+
+class InterruptedStream(httpx.AsyncByteStream):
+ """模拟结果传输到一半时上游断开连接。"""
+
+ async def __aiter__(self) -> Any:
+ """发出部分 ZIP 头后报错,重试必须从头覆盖临时文件。"""
+ yield b"PK\x03\x04partial"
+ raise httpx.ReadError("download disconnected")
+
+
+def application(tmp_path: Path, upstream: Upstream, **settings: Any) -> Any:
+ """用真实包装层和可控 HTTP 传输构造测试应用。"""
+ return create_app(
+ FileParseSettings("https://api.test/api", "secret", tmp_path, poll_interval=0.01, **settings),
+ transport=httpx.MockTransport(upstream.handle),
+ )
+
+
+def submit(client: TestClient, *, request_id: str = "request_1", content: bytes = b"source", **options: Any) -> httpx.Response:
+ """使用真实 multipart 编码,列表选项通过重复同名字段发送。"""
+ fields: list[tuple[str, Any]] = [("file", ("report.pdf", content, "application/pdf")), ("request_id", (None, request_id))]
+ for name, value in options.items():
+ for item in value if isinstance(value, list) else [value]:
+ fields.append((name, (None, str(item))))
+ return client.post("/v1/file_parse", files=fields)
+
+
+FORMATS = [
+ list(values) for size in range(1, 5) for values in itertools.combinations(("default", "docx", "html", "latex"), size)
+]
+
+
[email protected]("formats", FORMATS)
+def test_formats_are_returned_unchanged(tmp_path: Path, formats: list[str]) -> None:
+ """覆盖全部十五种格式组合及附件字节、文件名、MIME 和临时文件释放。"""
+ upstream = Upstream()
+ app = application(tmp_path, upstream)
+ with TestClient(app) as client:
+ response = submit(client, output_formats=formats)
+ assert response.status_code == 200, response.text
+ assert response.content == artifact_bytes(formats)
+ extension = formats[0] if formats in (["docx"], ["html"]) else "zip"
+ assert f"report.{extension}" in response.headers["content-disposition"]
+ assert response.headers["x-request-id"] == "request_1"
+ assert response.headers["x-task-id"] == "task_1"
+ assert not list(app.state.service.store.sources.iterdir())
+ assert not list(app.state.service.store.results.iterdir())
+
+
+def test_timeout_recovery_deduplication_and_conflict(tmp_path: Path) -> None:
+ """零等待返回包装地址,同 ID 复用任务,参数冲突不会替换原任务。"""
+ upstream = Upstream()
+ upstream.complete = False
+ app = application(tmp_path, upstream)
+ with TestClient(app, root_path="/gateway") as client:
+ response = submit(client, wait_timeout=0, output_formats=["html", "html"])
+ assert response.status_code == 202
+ payload = response.json()
+ assert response.headers["location"] == payload["status_url"]
+ assert payload["status_url"] == "http://testserver/gateway/v1/file_parse/request_1"
+ assert payload["output_formats"] == ["html"]
+ assert payload["result_url"] is None
+ assert client.get(payload["status_url"] + "/result").status_code == 409
+ assert submit(client, wait_timeout=0, output_formats=["html"]).status_code == 202
+ assert len(upstream.tasks) == 1
+ assert submit(client, content=b"different", wait_timeout=0).status_code == 409
+ upstream.complete = True
+ response = client.get(payload["status_url"])
+ assert response.json()["status"] == "completed"
+ assert client.get(response.json()["result_url"]).content == artifact_bytes(["html"])
+ assert not list(app.state.service.store.results.iterdir())
+
+
+def test_wait_budget_excludes_upload(tmp_path: Path) -> None:
+ """慢速上游上传不占一秒等待预算,解析完成仍能直接返回附件。"""
+ upstream = Upstream()
+ upstream.upload_delay = 1.1
+ upstream.complete_delay = 0.15
+ with TestClient(application(tmp_path, upstream)) as client:
+ started = time.monotonic()
+ response = submit(client, wait_timeout=1)
+ assert response.status_code == 200, response.text
+ assert time.monotonic() - started >= 1.1
+
+
+def test_expired_wait_does_not_cancel_upstream(tmp_path: Path) -> None:
+ """等待预算到期返回 202,后台继续观察任务直至完成。"""
+ upstream = Upstream()
+ upstream.complete = False
+ with TestClient(application(tmp_path, upstream)) as client:
+ started = time.monotonic()
+ response = submit(client, wait_timeout=1)
+ assert response.status_code == 202
+ assert 0.9 <= time.monotonic() - started < 3
+ assert not any(method == "DELETE" for method, _, _ in upstream.calls)
+ upstream.complete = True
+ assert client.get(response.json()["status_url"]).json()["status"] == "completed"
+
+
+def test_wait_budget_excludes_storage_notification_delay(tmp_path: Path) -> None:
+ """存储已经接收字节但网关尚未确认 ready 时,不消耗同步解析等待预算。"""
+ upstream = Upstream()
+ upstream.ready_delay = 1.1
+ upstream.complete_delay = 1.2
+ with TestClient(application(tmp_path, upstream)) as client:
+ response = submit(client, wait_timeout=1)
+ assert response.status_code == 200, response.text
+ assert upstream.uploads == 1
+
+
[email protected]("failure", ["lost", "malformed"])
+def test_unknown_creation_is_never_replayed(tmp_path: Path, failure: str) -> None:
+ """上游创建成功但响应丢失或身份缺失时,重试和重启都不能再次 POST。"""
+ upstream = Upstream()
+ upstream.create_failure = failure
+ with TestClient(application(tmp_path, upstream)) as client:
+ response = submit(client)
+ assert response.status_code == 502
+ assert response.json()["status"] == "submission_unknown"
+ assert response.json()["task_id"] is None
+ assert submit(client).status_code in {409, 502}
+ with TestClient(application(tmp_path, upstream)) as client:
+ assert client.get("/v1/file_parse/request_1").json()["status"] == "submission_unknown"
+ assert submit(client).status_code in {409, 502}
+ assert len(upstream.tasks) == 1
+
+
+def test_auth_failure_cleans_source_without_retry(tmp_path: Path) -> None:
+ """上游认证错误是配置错误,不向无认证的包装入口伪报 401。"""
+ upstream = Upstream()
+ upstream.create_failure = "auth"
+ app = application(tmp_path, upstream)
+ with TestClient(app) as client:
+ response = submit(client)
+ assert response.status_code == 502
+ assert response.json()["error"]["code"] == "invalid_api_key"
+ assert response.json()["status_url"]
+ assert not list(app.state.service.store.sources.iterdir())
+ assert len(upstream.calls) == 1
+
+
+def test_upload_response_loss_is_reconciled(tmp_path: Path) -> None:
+ """上传已成功但响应丢失时先查 ready,避免再次发送文件。"""
+ upstream = Upstream()
+ upstream.upload_response_lost = True
+ with TestClient(application(tmp_path, upstream)) as client:
+ assert submit(client).status_code == 200
+ assert upstream.uploads == 1
+
+
+def test_upload_retry_budget_and_explicit_resume(tmp_path: Path) -> None:
+ """自动上传最多尝试三次;调用者重传相同请求可以继续原任务而不再次创建。"""
+ upstream = Upstream()
+ upstream.upload_failures = 3
+ app = application(tmp_path, upstream)
+ with TestClient(app) as client:
+ app.state.service.client.retry_delay = 0
+ response = submit(client)
+ assert response.status_code == 502
+ assert response.json()["task_id"] == "task_1"
+ assert upstream.uploads == 3
+ assert list(app.state.service.store.sources.iterdir())
+ assert submit(client).status_code == 200
+ assert not list(app.state.service.store.sources.iterdir())
+ assert upstream.uploads == 4 and len(upstream.tasks) == 1
+
+
[email protected]("fault", ["download_expired", "download_corrupt", "download_interrupted", "redirect"])
+def test_download_retries_and_cross_origin_headers(tmp_path: Path, fault: str) -> None:
+ """签名失效和损坏结果可以恢复;重定向每跳都验证未向存储发送 API Key。"""
+ upstream = Upstream()
+ setattr(upstream, fault, True)
+ app = application(tmp_path, upstream)
+ with TestClient(app) as client:
+ app.state.service.client.retry_delay = 0
+ response = submit(client)
+ assert response.status_code == 200
+ assert response.content == artifact_bytes(["default"])
+ assert upstream.downloads == (1 if fault == "redirect" else 2)
+ assert not list(app.state.service.store.results.iterdir())
+
+
+def test_cache_hit_avoids_upload(tmp_path: Path) -> None:
+ """源文件 hash 缓存命中时直接等待,不尝试构造或访问上传链接。"""
+ upstream = Upstream()
+ upstream.cache_hit = True
+ with TestClient(application(tmp_path, upstream)) as client:
+ assert submit(client).status_code == 200
+ assert upstream.uploads == 0
+
+
+def test_concurrent_duplicate_requests_share_one_task(tmp_path: Path) -> None:
+ """并发提交相同请求 ID 只创建、上传一次,但每个连接可分别下载。"""
+ upstream = Upstream()
+ upstream.upload_delay = 0.1
+ with TestClient(application(tmp_path, upstream)) as client, ThreadPoolExecutor(max_workers=4) as executor:
+ responses = list(executor.map(lambda _: submit(client), range(4)))
+ assert all(response.status_code == 200 for response in responses)
+ assert len(upstream.tasks) == 1
+ assert upstream.uploads == 1
+
+
+def test_restart_uses_existing_task_and_isolates_upstream(tmp_path: Path) -> None:
+ """进程重启后查询同一任务;更换上游配置不会访问旧记录。"""
+ upstream = Upstream()
+ upstream.complete = False
+ with TestClient(application(tmp_path, upstream)) as client:
+ response = submit(client, wait_timeout=0)
+ assert response.status_code == 202
+ upstream.complete = True
+ with TestClient(application(tmp_path, upstream)) as client:
+ assert client.get("/v1/file_parse/request_1").json()["status"] == "completed"
+ assert client.get("/v1/file_parse/request_1/result").status_code == 200
+ assert len(upstream.tasks) == 1
+ app = create_app(FileParseSettings("https://different.test", state_dir=tmp_path))
+ with TestClient(app) as client:
+ assert client.get("/v1/file_parse/request_1").status_code == 404
+
+
[email protected]("stage", ["prepared", "pending", "expired", "submitting"])
+def test_restart_at_upload_boundaries(tmp_path: Path, stage: str) -> None:
+ """覆盖崩溃前尚未创建、已创建待上传、上传过期和提交不确定四个恢复边界。"""
+ upstream = Upstream()
+ settings = FileParseSettings("https://api.test/api", "secret", tmp_path, poll_interval=0.01)
+
+ async def seed() -> None:
+ """直接落盘一个崩溃时刻的真实恢复记录,不伪造运行中的任务对象。"""
+ store = RecordStore(settings)
+ await store.open()
+ source = store.sources / "retained-source"
+ source.write_bytes(b"source")
+ options = ParseOptions()
+ record = Record(
+ "request_1",
+ "report.pdf",
+ 6,
+ "a" * 64,
+ options.model_dump(),
+ str(source),
+ fingerprint("report.pdf", 6, "a" * 64, options),
+ )
+ if stage in {"pending", "expired"}:
+ remote = TaskClient(settings, httpx.MockTransport(upstream.handle))
+ try:
+ task = await remote.create(record)
+ record.remote = task.model_dump()
+ record.status = "waiting_upload"
+ if stage == "expired":
+ upstream.tasks[task.task_id]["expires_at"] = int(time.time()) - 1
+ finally:
+ await remote.close()
+ elif stage == "submitting":
+ record.status = "submitting"
+ await store.save(record)
+ await store.close()
+
+ asyncio.run(seed())
+ app = application(tmp_path, upstream)
+ with TestClient(app) as client:
+ deadline = time.monotonic() + 3
+ while True:
+ response = client.get("/v1/file_parse/request_1")
+ status = response.json()["status"]
+ if status in {"completed", "failed", "submission_unknown"} or time.monotonic() > deadline:
+ break
+ time.sleep(0.01)
+ assert status == ("submission_unknown" if stage == "submitting" else "failed" if stage == "expired" else "completed")
+ assert len(upstream.tasks) == (0 if stage == "submitting" else 1)
+ assert upstream.uploads == (1 if stage in {"prepared", "pending"} else 0)
+ assert not list(app.state.service.store.sources.iterdir())
+
+
+def test_disconnect_does_not_cancel_upload(tmp_path: Path) -> None:
+ """真实 ASGI 断连消息只能取消连接等待,已经接收的源文件继续上传。"""
+ upstream = Upstream()
+ upstream.upload_delay = 0.1
+ app = application(tmp_path, upstream)
+
+ async def exercise() -> None:
+ """发送完整 multipart 后立即断连,再通过同一请求 ID 检查完成结果。"""
+ async with app.router.lifespan_context(app):
+ encoded = httpx.Request(
+ "POST",
+ "http://wrapper/v1/file_parse",
+ files={
+ "file": ("report.pdf", b"source"),
+ "request_id": (None, "request_1"),
+ },
+ )
+ body = encoded.read()
+ received = False
+
+ async def receive() -> dict[str, Any]:
+ """第一次返回完整文件,后续接收返回真实断连事件。"""
+ nonlocal received
+ if not received:
+ received = True
+ return {"type": "http.request", "body": body, "more_body": False}
+ return {"type": "http.disconnect"}
+
+ async def send(message: dict[str, Any]) -> None:
+ """模拟断开的客户端,不消费任何响应内容。"""
+ assert message["type"] in {"http.response.start", "http.response.body"}
+
+ scope = {
+ "type": "http",
+ "method": "POST",
+ "path": "/v1/file_parse",
+ "root_path": "",
+ "raw_path": b"/v1/file_parse",
+ "query_string": b"",
+ "headers": [(key.lower(), value) for key, value in encoded.headers.raw],
+ "server": ("wrapper", 80),
+ "client": ("client", 123),
+ "scheme": "http",
+ "http_version": "1.1",
+ "asgi": {"version": "3.0", "spec_version": "2.4"},
+ }
+ await app(scope, receive, send)
+ for _ in range(100):
+ record = app.state.service.records["request_1"]
+ if record.status == "completed":
+ break
+ await asyncio.sleep(0.01)
+ assert record.status == "completed"
+ assert upstream.uploads == 1 and len(upstream.tasks) == 1
+ assert not list(app.state.service.store.sources.iterdir())
+
+ asyncio.run(exercise())
+
+
+def test_attachment_cleanup_after_send_disconnect(tmp_path: Path) -> None:
+ """附件发送中断也执行清理,而非只依赖发送成功后的后台回调。"""
+ path = tmp_path / "result.zip"
+ path.write_bytes(artifact_bytes(["default"]))
+
+ async def exercise() -> None:
+ """在响应已经开始后断开下游,验证 finally 释放结果文件。"""
+
+ async def receive() -> dict[str, Any]:
+ """提供断连事件,兼容 ASGI 响应实现。"""
+ return {"type": "http.disconnect"}
+
+ async def send(message: dict[str, Any]) -> None:
+ """在文件主体发送时模拟客户端 socket 关闭。"""
+ if message["type"] == "http.response.body":
+ raise OSError("client disconnected")
+
+ with pytest.raises(OSError):
+ await TemporaryFileResponse(path)(
+ {"type": "http", "method": "GET", "headers": [], "asgi": {"spec_version": "2.4"}}, receive, send
+ )
+
+ asyncio.run(exercise())
+ assert not path.exists()
+
+
+def test_retention_and_state_directory_lock(tmp_path: Path) -> None:
+ """同目录第二进程无法接管;恢复记录按首次终态时间回收。"""
+ upstream = Upstream()
+ app = application(tmp_path, upstream, retention_seconds=1)
+ with TestClient(app) as client:
+ assert submit(client).status_code == 200
+ with pytest.raises(Timeout), TestClient(application(tmp_path, upstream)):
+ pass
+ app.state.service.records["request_1"].terminal_at = time.time() - 2
+ assert client.get("/v1/file_parse/request_1").status_code == 404
+ client.portal.call(app.state.service.collect)
+ assert "request_1" not in app.state.service.records
+
+
+def test_late_query_cannot_replace_reused_request_id(tmp_path: Path) -> None:
+ """记录过期并复用 ID 后,旧任务的迟到查询不能覆盖新任务的持久化映射。"""
+ upstream = Upstream()
+ started, release = threading.Event(), asyncio.Event()
+ pause = False
+
+ async def delayed(request: httpx.Request) -> httpx.Response:
+ """仅延迟旧任务查询,保证新任务的创建和下载仍可正常完成。"""
+ if pause and request.method == "GET" and request.url.path.endswith("/v1/tasks/task_1"):
+ started.set()
+ await release.wait()
+ return await upstream.handle(request)
+
+ app = create_app(
+ FileParseSettings("https://api.test/api", "secret", tmp_path, retention_seconds=1, poll_interval=0.01),
+ transport=httpx.MockTransport(delayed),
+ )
+ with TestClient(app) as client, ThreadPoolExecutor(max_workers=1) as executor:
+ assert submit(client).status_code == 200
+ pause = True
+ pending = executor.submit(client.get, "/v1/file_parse/request_1")
+ assert started.wait(2)
+ app.state.service.records["request_1"].terminal_at = time.time() - 2
+ response = submit(client)
+ assert response.status_code == 200
+ assert response.headers["x-task-id"] == "task_2"
+ client.portal.call(release.set)
+ assert pending.result(timeout=2).status_code == 200
+ assert app.state.service.records["request_1"].task.task_id == "task_2"
+ with TestClient(application(tmp_path, upstream)) as client:
+ assert client.get("/v1/file_parse/request_1").json()["task_id"] == "task_2"
+
+
+ "options",
+ [
+ {"wait_timeout": -1},
+ {"wait_timeout": 3601},
+ {"wait_timeout": "1.5"},
+ {"request_id": "../bad"},
+ {"output_formats": ["markdown"]},
+ {"output_formats": [""]},
+ {"tier": "wrong"},
+ {"ocr_mode": "wrong"},
+ {"api_url": "https://different.test"},
+ ],
+)
+def test_invalid_parameters_do_not_create_tasks(tmp_path: Path, options: dict[str, Any]) -> None:
+ """所有本地可校验参数错误在创建上游任务前返回。"""
+ upstream = Upstream()
+ with TestClient(application(tmp_path, upstream)) as client:
+ assert submit(client, **options).status_code == 400
+ assert not upstream.tasks
+
+
+def test_single_file_content_type_and_scalar_duplicates(tmp_path: Path) -> None:
+ """拒绝 JSON、空文件、重复文件和重复标量,格式列表保持唯一例外。"""
+ upstream = Upstream()
+ with TestClient(application(tmp_path, upstream)) as client:
+ assert client.post("/v1/file_parse", json={}).status_code == 415
+ assert submit(client, content=b"").status_code == 400
+ assert client.post("/v1/file_parse", files=[("file", ("a.pdf", b"a")), ("file", ("b.pdf", b"b"))]).status_code == 400
+ assert (
+ client.post(
+ "/v1/file_parse",
+ files=[
+ ("file", ("a.pdf", b"a")),
+ ("tier", (None, "flash")),
+ ("tier", (None, "flash")),
+ ],
+ ).status_code
+ == 400
+ )
+ assert not upstream.tasks
+
+
+def test_openapi_and_light_imports(tmp_path: Path) -> None:
+ """OpenAPI 说明格式数组及附件,独立导入包装应用不加载解析运行时。"""
+ schema = application(tmp_path, Upstream()).openapi()
+ post = schema["paths"]["/v1/file_parse"]["post"]
+ request = post["requestBody"]["content"]["multipart/form-data"]["schema"]
+ assert request["properties"]["wait_timeout"]["maximum"] == 3600
+ assert request["properties"]["output_formats"]["items"]["enum"] == ["default", "docx", "html", "latex"]
+ assert "application/zip" in post["responses"]["200"]["content"]
+ result = subprocess.run(
+ [
+ sys.executable,
+ "-c",
+ (
+ "import sys; from mineru.kit.file_parse import create_app; "
+ "assert not any(x in sys.modules for x in ('torch', 'transformers', 'mineru.parser.api_server'))"
+ ),
+ ],
+ capture_output=True,
+ text=True,
+ )
+ assert result.returncode == 0, result.stderr
+
+
+def test_cli_configuration_and_environment(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ """真实 CLI 解析环境变量及显式覆盖,把上游配置交给独立应用。"""
+ from mineru.kit.main import app
+
+ captured: list[tuple[Any, dict[str, Any]]] = []
+
+ def run(application: Any, **options: Any) -> None:
+ """捕获启动参数,通过应用 lifespan 验证最终配置。"""
+ captured.append((application, options))
+ with TestClient(application):
+ assert application.state.service.settings.api_url == "https://explicit.test/api"
+ assert application.state.service.settings.api_key == "environment-key"
+
+ monkeypatch.setattr("uvicorn.run", run)
+ monkeypatch.setenv("MINERU_API_URL", "https://environment.test")
+ monkeypatch.setenv("MINERU_API_KEY", "environment-key")
+ result = CliRunner().invoke(
+ app, ["file-parse-server", "--api-url", "https://explicit.test/api", "--state-dir", str(tmp_path)]
+ )
+ assert result.exit_code == 0, result.output
+ assert captured[0][1] == {"host": "127.0.0.1", "port": 8003}
diff --git a/tests/unittest/test_file_parse_http.py b/tests/unittest/test_file_parse_http.py
new file mode 100644
index 00000000..045ed8ff
--- /dev/null
+++ b/tests/unittest/test_file_parse_http.py
@@ -0,0 +1,230 @@
+"""本地 API、Router、SaaS/对象存储的真实 TCP 同步包装验收。"""
+
+from __future__ import annotations
+
+import io
+import socket
+import subprocess
+import sys
+import time
+import zipfile
+from collections.abc import Iterator
+from contextlib import contextmanager
+from pathlib import Path
+
+import httpx
+from fastapi import FastAPI, Request, Response
+from PIL import Image, ImageDraw
+from reportlab.lib.utils import ImageReader
+from reportlab.pdfgen.canvas import Canvas
+from test_file_parse import FORMATS, Upstream, artifact_bytes
+from test_http_api_example_script import _Server
+from test_task_http import HTML, _LiveService, live_service as tasks_live_service
+
+from mineru.kit.file_parse import FileParseSettings, create_app
+
+live_service = tasks_live_service
+
+
+def assert_clean(directory: Path) -> None:
+ """TCP 客户端可能先收到末块数据,给服务器的响应 finally 一个有界收尾窗口。"""
+ deadline = time.monotonic() + 2
+ while list(directory.iterdir()) and time.monotonic() < deadline:
+ time.sleep(0.01)
+ assert not list(directory.iterdir())
+
+
+@contextmanager
+def wrapper(tmp_path: Path, base_url: str, key: str | None) -> Iterator[tuple[httpx.Client, FastAPI]]:
+ """启动真正的包装 HTTP 服务,统一测试只改变上游地址和认证。"""
+ app = create_app(FileParseSettings(base_url, key, tmp_path, poll_interval=0.02))
+ server = _Server(app)
+ server.start()
+ try:
+ with httpx.Client(base_url=server.url, timeout=90, trust_env=False) as client:
+ yield client, app
+ finally:
+ server.stop()
+
+
+def parse_file(
+ client: httpx.Client,
+ request_id: str,
+ filename: str,
+ content: bytes,
+ formats: list[str],
+) -> httpx.Response:
+ """从调用者端发送单文件 multipart,请求格式遵循标准重复字段编码。"""
+ return client.post(
+ "/v1/file_parse",
+ files=[
+ ("file", (filename, content)),
+ ("request_id", (None, request_id)),
+ ("tier", (None, "flash")),
+ ("ocr_mode", (None, "txt")),
+ *(("output_formats", (None, item)) for item in formats),
+ ],
+ )
+
+
+def test_api_and_router_all_formats(live_service: _LiveService, tmp_path: Path) -> None:
+ """真实解析与导出全部格式组合,包装返回的字节必须和上游保存的产物相同。"""
+ with wrapper(tmp_path / "wrapper", live_service.url, "test-key") as (client, app):
+ with httpx.Client(headers={"Authorization": "Bearer test-key"}, timeout=90, trust_env=False) as upstream:
+ for index, formats in enumerate(FORMATS):
+ response = parse_file(client, f"formats_{index}", "source.html", HTML, formats)
+ assert response.status_code == 200, response.text
+ task_id = response.headers["x-task-id"]
+ task = upstream.get(f"{live_service.url}/v1/tasks/{task_id}").json()
+ direct = upstream.get(task["files"][0]["result_url"])
+ assert direct.status_code == 200
+ assert response.content == direct.content
+ assert response.headers["content-type"] == direct.headers["content-type"]
+ assert_clean(app.state.service.store.results)
+ assert_clean(app.state.service.store.sources)
+
+
+def cloud_bridge(upstream: Upstream) -> FastAPI:
+ """通过独立网络端口暴露模拟 SaaS 协议,数据传输仍走真实 HTTP。"""
+ app = FastAPI()
+
+ @app.api_route("/{path:path}", methods=["GET", "POST", "PUT"])
+ async def relay(request: Request) -> Response:
+ """只桥接 HTTP,不为包装客户端增加 Jobs、Files 或存储路径假设。"""
+ response = await upstream.handle(
+ httpx.Request(
+ request.method,
+ str(request.url),
+ headers=request.headers,
+ content=await request.body(),
+ )
+ )
+ return Response(response.content, status_code=response.status_code, headers=dict(response.headers))
+
+ return app
+
+
+def test_saas_storage_all_formats_and_recovery(tmp_path: Path) -> None:
+ """API 路径前缀、独立存储端口、重定向和签名刷新均使用同一包装实现。"""
+ state = Upstream()
+ api, storage = _Server(cloud_bridge(state)), _Server(cloud_bridge(state))
+ state.api_url, state.storage_url = api.url + "/api", storage.url
+ api.start()
+ storage.start()
+ try:
+ with wrapper(tmp_path, state.api_url, "secret") as (client, app):
+ app.state.service.client.retry_delay = 0
+ state.redirect = True
+ for index, formats in enumerate(FORMATS):
+ state.download_expired = True
+ response = parse_file(client, f"cloud_{index}", "source.pdf", b"source", formats)
+ assert response.status_code == 200, response.text
+ assert response.content == artifact_bytes(formats)
+ assert_clean(app.state.service.store.results)
+ assert_clean(app.state.service.store.sources)
+ source = tmp_path / "example.pdf"
+ source.write_bytes(b"source")
+ output = tmp_path / "example.docx"
+ script = Path(__file__).resolve().parents[2] / "scripts" / "http_file_parse_example.py"
+ common = [sys.executable, str(script), str(source), "--base-url", str(client.base_url)]
+ completed = subprocess.run(
+ [*common, "--output-format", "docx", "--output", str(output), "--request-id", "example-success"],
+ capture_output=True,
+ text=True,
+ timeout=10,
+ )
+ assert completed.returncode == 0, completed.stdout + completed.stderr
+ assert output.read_bytes() == artifact_bytes(["docx"])
+ state.complete = False
+ waiting = subprocess.run(
+ [*common, "--wait-timeout", "0", "--request-id", "example-waiting", "--output", str(tmp_path / "pending.zip")],
+ capture_output=True,
+ text=True,
+ timeout=10,
+ )
+ assert waiting.returncode == 124, waiting.stdout + waiting.stderr
+ assert "status_url" in waiting.stdout and not (tmp_path / "pending.zip").exists()
+ assert len(state.tasks) == 17
+ assert all(method != "DELETE" for method, _, _ in state.calls)
+ finally:
+ storage.stop()
+ api.stop()
+
+
+def illustrated_pdf() -> bytes:
+ """生成包含文本、原生图片和图注的真实 PDF,验证附件传输不改变解析产物。"""
+ picture = Image.new("RGB", (240, 140), "white")
+ drawing = ImageDraw.Draw(picture)
+ drawing.rectangle((10, 20, 100, 120), fill="navy")
+ drawing.ellipse((130, 20, 230, 120), fill="orange")
+ buffer = io.BytesIO()
+ canvas = Canvas(buffer, pagesize=(595, 842))
+ canvas.setFont("Helvetica", 18)
+ canvas.drawString(72, 760, "File Parse round trip")
+ canvas.setFont("Helvetica", 12)
+ canvas.drawString(72, 720, "The same document must reach every download client unchanged.")
+ canvas.drawImage(ImageReader(picture), 100, 420, width=360, height=210)
+ canvas.drawString(100, 395, "Figure 1. A blue rectangle and an orange circle.")
+ canvas.drawString(72, 350, "The figure is embedded in the PDF as a native image.")
+ canvas.save()
+ return buffer.getvalue()
+
+
+def test_real_pdf_attachment_matches_upstream(live_service: _LiveService, tmp_path: Path) -> None:
+ """真实 PDF 经 API 与 Router 导出四格式后,附件和上游产物逐字节一致且包含图片。"""
+ pdf = illustrated_pdf()
+ (tmp_path / "illustrated.pdf").write_bytes(pdf)
+ with wrapper(tmp_path / "wrapper", live_service.url, "test-key") as (client, _):
+ response = parse_file(client, "illustrated", "illustrated.pdf", pdf, ["default", "docx", "html", "latex"])
+ assert response.status_code == 200, response.text
+ (tmp_path / "wrapper-result.zip").write_bytes(response.content)
+ with httpx.Client(headers={"Authorization": "Bearer test-key"}, timeout=90, trust_env=False) as upstream:
+ task = upstream.get(f"{live_service.url}/v1/tasks/{response.headers['x-task-id']}").json()
+ direct = upstream.get(task["files"][0]["result_url"])
+ assert response.content == direct.content
+ with zipfile.ZipFile(io.BytesIO(response.content)) as archive:
+ assert "File Parse round trip" in archive.read("markdown.md").decode()
+ assert {"document.docx", "document.html", "document.tex"}.issubset(archive.namelist())
+ assert any(name.startswith("images/") and not name.endswith("/") for name in archive.namelist())
+ assert b"data:image/" in archive.read("document.html")
+
+
+def test_tcp_disconnect_keeps_accepted_request(tmp_path: Path) -> None:
+ """调用者关闭真实 socket 后,完整收到的文件仍上传并可通过请求 ID 找回。"""
+ state = Upstream()
+ state.upload_delay = 0.2
+ api, storage = _Server(cloud_bridge(state)), _Server(cloud_bridge(state))
+ state.api_url, state.storage_url = api.url + "/api", storage.url
+ api.start()
+ storage.start()
+ try:
+ with wrapper(tmp_path, state.api_url, "secret") as (client, app):
+ request = httpx.Request(
+ "POST",
+ str(client.base_url).rstrip("/") + "/v1/file_parse",
+ files={
+ "file": ("source.pdf", b"source"),
+ "request_id": (None, "disconnected"),
+ },
+ )
+ content = request.read()
+ headers = "\r\n".join(f"{key}: {value}" for key, value in request.headers.items()).encode()
+ address = client.base_url
+ with socket.create_connection((address.host, address.port)) as sock:
+ sock.sendall(b"POST /v1/file_parse HTTP/1.1\r\n" + headers + b"\r\n\r\n" + content)
+ deadline = time.monotonic() + 3
+ while "disconnected" not in app.state.service.records and time.monotonic() < deadline:
+ time.sleep(0.01)
+ assert "disconnected" in app.state.service.records
+ deadline = time.monotonic() + 5
+ while True:
+ response = client.get("/v1/file_parse/disconnected")
+ if response.json().get("status") == "completed" or time.monotonic() > deadline:
+ break
+ time.sleep(0.02)
+ assert response.json()["status"] == "completed"
+ assert client.get(response.json()["result_url"]).content == artifact_bytes(["default"])
+ assert state.uploads == 1 and len(state.tasks) == 1
+ finally:
+ storage.stop()
+ api.stop()
diff --git a/tests/unittest/test_kit_commands.py b/tests/unittest/test_kit_commands.py
index 7060850e..5b91ee3a 100644
--- a/tests/unittest/test_kit_commands.py
+++ b/tests/unittest/test_kit_commands.py
@@ -400,6 +400,7 @@ def test_kit_root_commands_keep_product_order() -> None:
"parse",
"webui",
"api-server",
+ "file-parse-server",
"vlm-server",
"router",
"models",