Skip to content

Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.

feat(api)!: make Tasks the only asynchronous parsing API

myhloli7f250e2331d6
diff --git a/docs/en/reference/migration_4.md b/docs/en/reference/migration_4.md
index 86444b3d..1654d03d 100644
--- a/docs/en/reference/migration_4.md
+++ b/docs/en/reference/migration_4.md
@@ -60,7 +60,7 @@ Restart relevant services after configuration changes, and download and verify m
## API and WebUI
-V1 uses `/v1/health`, `/v1/tiers`, `/v1/uploads`, `/v1/parse/jobs`, and `/v1/files`. Legacy `/file_parse` and `/tasks` are not provided. SDK clients and the WebUI must connect to a V1 service. The parsing API and OpenAI-compatible VLM server are different interfaces; their addresses are not interchangeable. API servers and Routers provide `/v1/tasks` form manifests, per-file uploads, and ZIP download links. Local services also accept single-file multipart. Clients poll tasks and use returned URLs; synchronous wrapping and inline-result convenience routes are no longer provided; see [HTTP API](../usage/http_api.md).
+Asynchronous parsing uses `/v1/tasks` JSON manifests, optional uploads and per-file result URLs. Upgrade the SDK and WebUI together with the service. Files downloads and the separate synchronous wrapper remain available; loopback callers also use hash preflight. See [HTTP API](../usage/http_api.md).
Existing Gradio clients also need to follow the new event interface; renaming the command alone is insufficient. Retained third-party integration guides describe their original adaptations and do not imply V1 compatibility.
diff --git a/docs/en/usage/http_api.md b/docs/en/usage/http_api.md
index 543cecc7..e32c90c2 100644
--- a/docs/en/usage/http_api.md
+++ b/docs/en/usage/http_api.md
@@ -1,109 +1,37 @@
# V1 HTTP API
-HTTP clients can use the tasks form-manifest and download-link protocol. This page first covers that workflow, then the existing Uploads/Jobs resource workflow used by the Python SDK and WebUI. For the Python client, see [Python SDK](sdk_api.md).
-
-Start a local service; for SaaS, use a deployment implementing the same tasks contract:
+Tasks is the single asynchronous parsing API. API Server, Router, the SDK and WebUI use a JSON manifest, upload only missing files, poll task metadata and download each file's result. See the [Tasks contract](../../next/api/tasks.md) and [Python SDK](sdk_api.md).
```bash
mineru-kit api-server --host 127.0.0.1 --port 8000 --tier standard
+MINERU_API_URL=http://127.0.0.1:8000 bash scripts/http_task_example.sh a.pdf b.pdf
+OUTPUT_FORMATS=default,docx MINERU_API_URL=http://127.0.0.1:8000 bash scripts/http_api_example.sh a.pdf
```
-## Tasks: manifests, per-file uploads, and ZIP links
-
-API servers, Routers, and SaaS deployments implementing this contract use the same client workflow: submit a manifest, upload missing files, poll status, and download the returned `result_url`.
-SaaS can return an OSS URL; a local service returns its Files URL. Clients use the returned links and do not require SaaS to implement the local Files API.
-See the [complete tasks contract](../../next/api/tasks.md).
-
-```bash
-# One or more files: preflight first and upload only cache misses.
-MINERU_API_URL=http://127.0.0.1:8000 \
- bash scripts/http_task_example.sh a.pdf b.pdf
-
-# Hashes are optional.
-SEND_HASH=0 MINERU_API_URL=http://127.0.0.1:8000 \
- bash scripts/http_task_example.sh a.pdf b.pdf
-
-# Continue the existing task using the original file list and order.
-TASK_ID=job_... MINERU_API_URL=http://127.0.0.1:8000 \
- bash scripts/http_task_example.sh a.pdf b.pdf
-
-# Local multipart accepts one file and still returns an asynchronous task.
-curl --fail-with-body 'http://127.0.0.1:8000/v1/tasks' \
- -F 'tier=flash' -F 'ocr_mode=txt' -F '[email protected]'
-```
-
-All `POST /v1/tasks` parameters belong in the form body. `manifest` is a text field containing a JSON array: each item has a unique `client_file_id`, `filename`, actual `bytes`, and optional `sha256sum` and per-file `page_range`.
-Multi-file requests submit metadata only. Single-file direct uploads may include a one-item manifest to declare a hash.
-Parsing options remain `tier`, `ocr_mode`, `page_range`, and repeated `output_formats`; a final ZIP is always generated.
-POST query parameters return `400`; JSON and raw-byte task creation return `415`. Existing Uploads/Jobs JSON requests remain unchanged.
-
-Creation immediately returns a stable `task_id` and `status_url`. `uploads` contains only missing files, with their upload method, URL, headers, and expiration.
-Parsing starts automatically after every file passes validation; no client completion request is needed. Without a supplied hash, the service receives the file and still computes a hash for storage deduplication.
-Each fully uploaded file becomes `ready` independently. If another upload is interrupted, retry only that file on the same task. This is file-level retry, not resumable chunk upload within one file.
-
-Poll only `status_url`. States are `waiting_upload/queued/running/completed/partial/failed/canceled`.
-`completed/partial` and `result_url` are published only after the ZIP is saved. Status never includes inline document contents; SaaS clients download directly from object storage.
-Batch ZIPs contain numbered directories and a per-file manifest. Packaging failures produce a failed task, not an unusable download link.
-
-The complete client is [`scripts/http_task_example.sh`](https://github.com/opendatalab/MinerU/blob/master/scripts/http_task_example.sh) and its adjacent Python file, requiring only Python 3.10+ standard library.
-`UPLOAD_MODE=direct` accepts one file. `RESULT_FILE` defaults to `result.zip`; `MAX_POLLS/POLL_INTERVAL` control polling, and `CONNECT_TIMEOUT/REQUEST_TIMEOUT` control transfer budgets.
-Each file gets at most three upload attempts. Creation is never automatically retried. After an uncertain upload response, query status before retransmitting; refresh expired links by querying the original task.
-The API Key is attached only to same-origin requests, never to OSS, external downloads, or cross-origin redirects.
-Exit codes are `0` for success, `2` for partial success with a saved ZIP, `1` for failure or cancellation, and `124` when the polling budget is exhausted without canceling the task.
-
-Local services impose no fixed uploaded-file size or PDF page-count limit; the limit remains 100 files per task. Waiting uploads consume no parsing slot or Router worker reservation.
-Router selects by load and reserves before upstream submission; a cache hit cannot pin work to a busy worker. One batch stays on one worker, which produces the ZIP; Router maps resources and streams downloads.
-
-## Existing Uploads / Jobs resource example
-
-For independent Upload, File, and Job resources, continue using the following script. Its protocol and exit codes remain unchanged and are maintained separately from the tasks client above:
-
-> [`scripts/http_api_example.sh`](https://github.com/opendatalab/MinerU/blob/master/scripts/http_api_example.sh)
-
-```bash
-export MINERU_API_URL=http://127.0.0.1:8000
-export MINERU_API_KEY=secret-key # omit for anonymous local access
-./scripts/http_api_example.sh document.pdf
-```
+## Create and upload
-The script's behavior is verified by `tests/unittest/test_http_api_example_script.py` against a real Flash service and a scripted mock, without model downloads. Exit codes are stable for automation:
+Send application/json to POST `/v1/tasks`. Each manifest entry contains a unique client_file_id, filename, bytes for uploaded files, and optional sha256sum/page_range. URL entries may omit bytes. Options are tier, ocr_mode, page_range, output_formats and include_model_output.
-| Code | Meaning |
-| --- | --- |
-| `0` | Job completed; every requested artifact was saved |
-| `1` | Script, transport, protocol, download, or local write failure |
-| `2` | Job partial: artifacts of completed files were saved first |
-| `3` | Job failed |
-| `4` | Job canceled |
-| `124` | Polling budget exhausted; `job_id` printed so polling can resume |
+The response contains task_id, status_url and upload links for missing files. Follow each returned PUT URL and its headers; no completion request is needed. Each ready file queues independently. Waiting for uploads consumes no parsing slot.
-## Resource API authentication and byte uploads
+Loopback requests follow the same protocol. The SDK computes SHA-256 before submission; completed, unexpired cached inputs are not uploaded again. Concurrent first uploads may transfer duplicate bytes, but source storage is deduplicated. Default retention is 24 hours with collection every 5 minutes; active task and transfer references protect their bytes.
-Requests to `/v1/*` carry `Authorization: Bearer $MINERU_API_KEY` when the service is started with `--api-key`; omit the header for anonymous local access.
+Upload links expire one hour after task creation. Queries do not renew them. Uploads accepted before the deadline may finish. After an uncertain upload, query first and skip ready files; retry pending uploads from the beginning while their links remain valid.
-The byte upload uses exactly the method, URL, and headers returned by `POST /v1/uploads`:
+## Status and results
-- A self-hosted service returns a same-origin URL such as `/v1/uploads/{id}/content`. Same-origin means scheme + host + effective port; it is checked by the client, not assumed. Same-origin uploads must carry the API authentication when the service requires it.
-- The official API returns a pre-signed object-storage URL that carries its own authorization. Do **not** attach the MinerU API key to a different-origin URL — that would leak the credential. Headers returned by the service (`upload_headers`) are always kept as-is.
+Poll status_url and match files by client_file_id. Parent states are waiting_upload, queued, running, completed, partial, failed and canceled. Each completed file has one files[].result_url; there is no top-level result_url or combined batch ZIP. Completed files can be downloaded while other files are still pending.
-The example script implements the same rule as the Python SDK (`api_client._same_origin_upload_headers`): resolve relative URLs against the API base, compare scheme + host + effective port, attach the MinerU key only for same-origin uploads. Relative URLs resolve to same-origin; the same host on a different port counts as a different origin.
+Outputs are default (Markdown, Middle JSON, Structured Content and images in a ZIP), docx, html, or latex (TEX and images in a ZIP). Multiple formats share one flat ZIP. DOCX and HTML embed images. include_model_output defaults to false; enabling it requires default and adds model_output.json when raw output exists.
-Downloads go through `GET /v1/files/{file_id}/content`, which may answer with a `302` redirect. The example follows redirects without `--location-trusted`, so credentials are never re-sent to a different origin (including the same host on a different port; requires curl >= 7.83.0).
+GET `/v1/tasks` lists parent tasks with status filters and cursor pagination. DELETE `/v1/tasks/{task_id}` cancels unfinished files. Files metadata, listing, output download and deletion remain available.
-## Resource-script request-cycle details
+Use returned result URLs directly. Local servers and Router return Files URLs; SaaS may return signed storage URLs. Attach the API key only to same-origin API requests and reevaluate credentials at every redirect. Query the task again when a result signature expires.
-- **Upload lifecycle.** Pass `sha256sum` when creating the upload for integrity checking and instant reuse of an already-known file; in that case the response is `completed` and embeds the `file` object directly, so byte upload and completion are skipped.
-- **A 200 response is not automatically valid.** The script validates JSON structure at every step: IDs must be non-empty strings and statuses must belong to the supported sets; anything else fails immediately instead of entering the polling loop with a `null` ID.
-- **Bounded polling.** `MAX_POLLS` limits the number of polling requests (each also spends up to the request timeout and the poll interval); when the budget is exhausted the script exits `124` and prints the `job_id` — the job was not canceled and polling can simply resume.
-- **Terminal states.** `completed`, `partial`, `failed`, and `canceled` are all terminal. `partial` means some files succeeded: the script saves the artifacts of the completed files, prints per-file results including errors, and only then exits `2`. `failed` exits `3`, `canceled` exits `4`.
-- **Atomic downloads.** Artifacts are written to `*.part` first and moved into place on success; a `completed` job with a failing or missing artifact download does not exit `0`.
-- **Outputs.** Every artifact is a `purpose:"parse_output"` file; content always comes from `GET /v1/files/{file_id}/content` using the ids under `files[].output_files`. Treat `GET /v1/health` (`features.sources`, `features.output_formats`) as the source of truth for what the current deployment supports; renderer formats do not imply API support.
+## Client and migration
-## Production notes
+Both shell examples delegate to the standard-library Python Tasks client. RESULT_DIR selects the per-file destination. OUTPUT_FORMATS defaults to default; INCLUDE_MODEL_OUTPUT=1 requests raw model output. SEND_HASH=0 disables the optional preflight hash. Resume with TASK_ID and the same input order.
-- Bind `--host` consciously. The default `127.0.0.1` keeps the service on loopback; `0.0.0.0` exposes it to the network, so combine that with `--api-key` or an authenticating reverse proxy.
-- Storage and restart boundary: file bytes live under `--upload-dir` (a temporary directory that is removed on clean shutdown when not configured), but the upload/file/job resource index is in-process state. Restarting the service does **not** preserve old `upload_id`/`file_id`/job ids, even when the directory is kept. This API is not a recoverable persistent job service; use the document library for persistent reading, indexing, and caching — it is not a drop-in persistent replacement for this upload/job interface.
-- `--concurrency` caps concurrent parse jobs; queue growth beyond it shows up as longer `queued` times, not failures.
-- `GET /v1/usage` reports consumption. Use `/v1/health` and the OpenAPI docs at `/docs` (when enabled) to separate service-side failures from client errors; check server logs before retrying a `failed` job.
+Creation is not automatically retried. MAX_POLLS/POLL_INTERVAL bound polling; CONNECT_TIMEOUT/REQUEST_TIMEOUT bound individual requests. Exit codes are 0 (success), 2 (partial success), 1 (failure or cancellation) and 124 (poll budget exhausted). Completed downloads remain on disk.
-The convenience workflow uses `/v1/tasks` and its returned links, with no synchronous-wait or inline-result route. Legacy root routes remain unavailable; see [migration](../reference/migration_4.md) for older clients.
+The standalone upload and Jobs APIs have been removed. Coordinate server, Router and SDK upgrades; see [migration notes](../../next/api/changes.md). The separate [file_parse wrapper](../../next/api/file-parse.md) provides recoverable synchronous single-file calls.
diff --git a/docs/next/api.md b/docs/next/api.md
index e74a708d..dc0746f6 100644
--- a/docs/next/api.md
+++ b/docs/next/api.md
@@ -20,12 +20,11 @@ Unified API 文档描述两类部署形态下的同一套 API:
1. [API 总览](api/README.md): 资源模型、端点地图、认证、通用约定。
2. [响应与错误](api/responses.md): 成功响应、错误 envelope、请求追踪。
3. [Health、Models 与 Tiers](api/health-models.md): 服务能力、模型列表、解析档位。
-4. [Uploads 与 Files](api/uploads-files.md): 上传生命周期、文件对象、产物下载。
-5. [Parse Jobs](api/parse-jobs.md): 解析任务、轮询。
- [V1 tasks](api/tasks.md): JSON 清单、逐文件上传、独立结果下载链接和 Router 调度。
+4. [Files](api/files.md): 文件元数据、列表、产物下载和删除。
+5. [Tasks](api/tasks.md): JSON 清单、hash 缓存、逐文件上传、列表、轮询和下载。
[同步 file_parse](api/file-parse.md): 独立包装服务,单文件直传、同步附件和超时恢复。
6. [Chat 与 Responses](api/chat.md): OpenAI-compatible 文档对话接口。
-7. [Webhooks](api/webhooks.md): 异步回调协议。
+7. [任务通知](api/webhooks.md): 轮询和 SDK 回调边界。
8. [Usage 与 Limits](api/usage-limits.md): access level、限流、用量查询。
9. [端到端示例](api/examples.md): 常见调用流程。
diff --git a/docs/next/api/README.md b/docs/next/api/README.md
index bfbef375..4fcce138 100644
--- a/docs/next/api/README.md
+++ b/docs/next/api/README.md
@@ -15,17 +15,15 @@ MinerU v1 API 以 `mineru.net` 官方远程 API 为主线,同时让 Local Pars
| 资源 | 含义 |
|------|------|
-| `file` | 平台中的文件,由不透明 `file_id` 标识。可表示用户上传的源文件、Chat/Responses 的图片输入,或解析产物。 |
-| `upload` | OpenAI Uploads API 风格的上传会话,用来管理上传生命周期。 |
-| `job` | 一次解析任务,可包含一个或多个输入文件。 |
+| `file` | 源文件视图或已生成的解析产物。 |
+| `task` | 父解析任务,聚合独立文件状态和结果链接。 |
ID 使用 OpenAI 风格的前缀加随机串。官方 API 使用 24 字符 base62 随机串;本地 server 可以用同长度 hex 随机串降低实现复杂度。ID 是不透明标识,客户端不得从中推断时间、租户或存储路径。
| 资源 | 示例 |
|------|------|
| `file` | `file-r9NSmHLJE6flShV5vQ0Y60Rd` |
-| `upload` | `upload_r9NSmHLJE6flShV5vQ0Y60Rd` |
-| `job` | `job_r9NSmHLJE6flShV5vQ0Y60Rd` |
+| `task` | `job_01HXYZ123ABCDEF`(不透明标识) |
## Base URL
@@ -46,22 +44,15 @@ ID 使用 OpenAI 风格的前缀加随机串。官方 API 使用 24 字符 base6
| `GET` | `/v1/models` | 列出模型 |
| `GET` | `/v1/models/{model}` | 查询单个模型 |
| `GET` | `/v1/tiers` | 列出解析档位 |
-| `POST` | `/v1/uploads` | 创建 upload |
-| `POST` | `/v1/uploads/{upload_id}/complete` | 完成 upload |
-| `POST` | `/v1/uploads/{upload_id}/cancel` | 取消 upload |
-| `GET` | `/v1/uploads/{upload_id}` | 查询 upload |
| `GET` | `/v1/files` | 列出文件 |
| `GET` | `/v1/files/{file_id}` | 查询文件元信息 |
| `GET` | `/v1/files/{file_id}/content` | 下载解析产物 |
| `DELETE` | `/v1/files/{file_id}` | 删除文件 |
| `POST` | `/v1/tasks` | JSON 清单创建任务,返回缺失文件上传链接 |
+| `GET` | `/v1/tasks` | 列出当前调用方的父任务,支持筛选和分页 |
+| `DELETE` | `/v1/tasks/{task_id}` | 取消未完成文件 |
| `GET` | `/v1/tasks/{task_id}` | 查询总状态和逐文件状态,取得每个成功文件的下载链接 |
| `PUT` | 返回的上传链接 | 每个链接上传一个文件 |
-| `DELETE` | `/v1/tasks/{task_id}` | 取消任务 |
-| `POST` | `/v1/parse/jobs` | 创建解析任务 |
-| `GET` | `/v1/parse/jobs/{job_id}` | 查询任务 |
-| `GET` | `/v1/parse/jobs` | 列出任务 |
-| `DELETE` | `/v1/parse/jobs/{job_id}` | 取消任务 |
| `POST` | `/v1/chat/completions` | Chat Completions 文档对话 |
| `POST` | `/v1/responses` | Responses 文档对话 |
| `GET` | `/v1/usage` | 查询用量 |
@@ -101,4 +92,3 @@ Local Parse Server 的目标是复用客户端协议,而不是完整模拟公
- 不支持 Webhook。
- 上传 URL 可以指向本地 server 的临时上传 endpoint,而不是 OSS。
- 文件产物下载可以直接返回 `200 + body`,不必返回 CDN 重定向。
-- 支持 `local` source,但必须通过 `--allow-local-source` 显式开启;客户端应以 `health.features.sources` 判断当前部署是否允许。
diff --git a/docs/next/api/changes.md b/docs/next/api/changes.md
index 2de23a5b..961d776c 100644
--- a/docs/next/api/changes.md
+++ b/docs/next/api/changes.md
@@ -1,48 +1,18 @@
-# 共享文档 JSON 2.0
+# Tasks 唯一异步接口迁移
-Model/Middle 输出切换为统一 schema 身份及 metadata 分组,产品信息为实际 tier/parse_mode。
-历史 JSON 不再读取,自建 HTTP 服务与客户端需同步升级;请求参数和输出文件名不变。
-详见 [协议](../middle-json/envelope.md) 和 [迁移](../middle-json/migration.md)。
+这是破坏性 HTTP 接口升级:独立 `/v1/uploads` 全部路由及 `/v1/parse/jobs` 创建、列表、查询和取消路由已移除,返回 404;不保留别名或客户端自动回退。
-# API 变更记录
+| 旧方式 | 新方式 |
+|---|---|
+| 创建 Upload、PUT、complete | POST `/v1/tasks` 提交清单,只 PUT 缺失文件;上传成功自动排队。 |
+| 创建、查询、取消 Job | POST/GET/DELETE Tasks;GET `/v1/tasks` 列出父任务。 |
+| `files[].output_files` | `files[].result_url`,每个输入一个完整产物链接。 |
+| `markdown/middle_json/structured_content/zip` 输出选项 | `default` 默认 ZIP;另可组合 `docx/html/latex`。 |
+| `local/inline/file_id` 输入 | 文件上传或 URL;同机先计算 hash,缓存未命中也需上传。 |
+| 默认 ZIP 中的模型输出 | 显式 `include_model_output=true`,并包含 `default` 格式。 |
+| Health `output_formats` | `task_output_formats`;sources 为 upload/url,task_model_output 声明附加模型输出能力。 |
-状态: Implemented
-范围: NEXT v1 API 标准的破坏性或需要迁移的行为变化。
+`--allow-local-source` 和 `--max-inline-bytes` 同时删除。Files CRUD、Health、Tiers、Models、Usage 保留;同步 `file_parse` 继续作为独立包装服务运行。
+SDK 保留 `MinerUApiParser` 方法和返回类型,状态类型改为 `ApiTaskStatus`,保留 `ApiJobStatus` Python 别名。直接本地 parser 与 HF CPU/ZeroGPU 内嵌解析不经过 HTTP。
-## 2026-07-08: 明确 Local Parse Server base URL
-
-Local Parse Server 的 base URL 是 `http://localhost:8000`,endpoint 直接挂在 `/v1` 下。`/api` 只属于 `https://mineru.net/api` 这类官方远程 API base URL。
-
-## 2026-07-08: 增加 `features.output_formats`
-
-`GET /v1/health` 的 `features` 增加 `output_formats`,用于声明当前部署实际支持的 parse job 输出格式。客户端应以该字段作为能力发现事实,而不是假设所有部署都支持同一组输出格式。
-
-## 2026-07-08: 增加 `features.sources`
-
-`GET /v1/health` 的 `features` 增加 `sources`,用于声明当前部署允许的 parse job source 类型。客户端应以该字段决定是否可以使用 `local` source;没有声明 `local` 时,应通过 Uploads API 取得 `file_id` 后再创建解析任务。
-
-## 2026-07-08: 收紧 Local Parse Server source 策略
-
-Local Parse Server 对 `local`、`inline` 和 `url` source 增加启动时可配置的安全策略: `local` source 默认关闭,只有显式开启 `--allow-local-source` 后才允许读取 server 进程权限范围内的本地路径;`inline` source 受 `--max-inline-bytes` 限制;`url` source 默认只允许 HTTPS,只有显式开启 `--allow-http-source` 后才允许 HTTP。
-
-## 2026-07-08: 删除 parse job SSE events
-
-NEXT v1 API 不再定义 `GET /v1/parse/jobs/{job_id}/events`,parse job 响应中也不再返回 `links.events`。客户端统一通过 `GET /v1/parse/jobs/{job_id}` 查询任务状态和结果。
-
-线上 API Server 不支持长连接;本地 SSE 实现也只是服务端每秒检查 job 状态后通过长连接转发,相比客户端轮询没有实质效率收益。删除这一路径可以避免线上/本地协议分叉,并让 parse job 保持单一轮询模型。
-
-## 2026-07-08: 删除 parse job `wait` 参数
-
-`POST /v1/parse/jobs` 不再支持 `wait`。创建 parse job 始终返回 `202` 和 job 引用,客户端通过 `GET /v1/parse/jobs/{job_id}` 查询状态和结果。
-
-API 不提供在创建请求中同步等待完成的模式,也不内联返回 Markdown 内容。所有产物内容仍通过 `output_files` 对应的 File API 下载。
-
-这样可以避免线上服务长连接阻塞和本地/线上实现差异,让 parse job 保持单一异步模型。
-
-## 2026-07-08: 删除独立 `images` 输出格式
-
-`images` 不再是 `POST /v1/parse/jobs` 的合法 `output_formats` 值,parse job 响应中也不再返回 `output_files.images`。
-
-图片 sidecar 只通过 `zip` 产物返回。`markdown`、`middle_json`、`structured_content` 输出仍可能包含 `image_path` 引用,但这些引用对应的图片字节不再作为独立 parse output 暴露。
-
-这样可以避免同一批图片同时通过独立 output 和 zip 两条路径返回,减少产物语义重复,并让需要自包含解析结果的客户端统一依赖 `zip`。
+停收旧请求、等待在途任务完成,再协调升级 worker、Router、SDK 和应用。SaaS Go 服务需部署方按同一契约配套更新;进程内任务和缓存索引不保证跨重启恢复。
diff --git a/docs/next/api/chat.md b/docs/next/api/chat.md
index a52afc69..84f4b515 100644
--- a/docs/next/api/chat.md
+++ b/docs/next/api/chat.md
@@ -7,7 +7,7 @@
## 定位
-Chat Completions 和 Responses 面向细粒度文档理解场景,例如单张图片 OCR、单个表格提取或一段文本识别。完整 PDF、DOCX、PPTX 等多页文档解析应使用 [Parse Jobs](parse-jobs.md)。
+Chat Completions 和 Responses 面向细粒度文档理解场景,例如单张图片 OCR、单个表格提取或一段文本识别。完整 PDF、DOCX、PPTX 等多页文档解析应使用 [Tasks](tasks.md)。
两套接口都只支持单轮请求,不维护服务端会话历史。
@@ -49,7 +49,7 @@ Chat Completions 和 Responses 面向细粒度文档理解场景,例如单张
|------|------|:--:|------|
| `messages` | array | 是 | 消息列表。 |
| `model` | string | 是 | 虚拟模型 ID(`mineru-<服务版本>`),来自 `GET /v1/models`,精确匹配当前部署广告值;不用于选择引擎。 |
-| `tier` | string | 否 | MinerU 扩展字段:解析质量档(`flash`/`basic`/`standard`/`advanced`),内部复用 parse job 的 tier 路由与默认选择策略。OpenAI SDK 通过 `extra_body` 传入。 |
+| `tier` | string | 否 | MinerU 扩展字段:解析质量档(`flash`/`basic`/`standard`/`advanced`),内部复用 Tasks 的 tier 路由与默认选择策略。OpenAI SDK 通过 `extra_body` 传入。 |
| `stream` | bool | 否 | 是否流式输出,默认 `false`。 |
| `stream_options.include_usage` | bool | 否 | `stream=true` 时,最终 chunk 是否包含 usage。 |
| `temperature` | number | 否 | 采样温度,范围 `[0, 2]`。 |
@@ -81,7 +81,6 @@ Chat Completions 和 Responses 面向细粒度文档理解场景,例如单张
|------|------|------|
| `text` | `{"type":"text","text":"..."}` | 文本指令。 |
| `image_url` | `{"type":"image_url","image_url":{"url":"..."}}` | 图片 URL 或 base64 data URI。`detail` 可传但忽略。 |
-| `file` | `{"type":"file","file":{"file_id":"file-..."}}` | 引用 `purpose:"input_image"` 的 File。 |
输入限制:
@@ -224,8 +223,7 @@ Content part:
| part | 结构 | 说明 |
|------|------|------|
| `input_text` | `{"type":"input_text","text":"..."}` | 文本。 |
-| `input_image` | `{"type":"input_image","image_url":"..."}` 或 `{"type":"input_image","file_id":"file-..."}` | 图片 URL、data URI 或 File。 |
-| `input_file` | `{"type":"input_file","file_id":"file-..."}` | 仅支持 `purpose:"input_image"` 的 File。 |
+| `input_image` | `{"type":"input_image","image_url":"..."}` | 图片 URL 或 data URI。 |
不支持 assistant role、tool/function item、reasoning item、conversation 状态 item。
@@ -329,3 +327,5 @@ Local Parse Server 可以暴露同样的 Chat 和 Responses endpoint,但实际
- 不得静默切换到 `flash`。
- 本地部署通常不做 anonymous/registered 的功能售卖差异;启用 API Key 后,通过认证的请求可以使用本地 server 暴露的全部模型能力。
- 如果本地不实现 Chat/Responses,应在 `GET /v1/models` 中不要暴露对应模型,并对 endpoint 返回明确错误。
+
+这里的 Chat/Responses 为独立协议说明;Python 解析服务不新增这些路由。独立 Uploads 移除后,图片输入使用 URL 或 data URI,不再承诺 purpose=input_image 的 File 输入。
diff --git a/docs/next/api/examples.md b/docs/next/api/examples.md
index 87e1269c..477586d9 100644
--- a/docs/next/api/examples.md
+++ b/docs/next/api/examples.md
@@ -1,159 +1,20 @@
-# 端到端示例
+# Tasks 使用示例
-状态: Implemented
-读者: API 使用者、SDK 开发者
-范围: 官方 API 与本地 server 的常见调用流程
-来源: 由根目录旧 Unified API 底稿迁移整理而来
-
-## 官方 API: 上传并解析 PDF
-
-```bash
-SHA=$(sha256sum report.pdf | cut -d' ' -f1)
-SIZE=$(wc -c < report.pdf | tr -d ' ')
-
-RESP=$(curl -s -X POST https://mineru.net/api/v1/uploads \
- -H "Authorization: Bearer $MINERU_API_KEY" \
- -H "Content-Type: application/json" \
- -d "{\"filename\":\"report.pdf\",\"bytes\":$SIZE,\"mime_type\":\"application/pdf\",\"purpose\":\"parse\",\"sha256sum\":\"$SHA\"}")
-
-STATUS=$(echo "$RESP" | jq -r '.status')
-
-if [ "$STATUS" = "pending" ]; then
- UPLOAD_ID=$(echo "$RESP" | jq -r '.id')
- URL=$(echo "$RESP" | jq -r '.upload_url')
- METHOD=$(echo "$RESP" | jq -r '.upload_method')
-
- curl -X "$METHOD" "$URL" \
- -H "Content-Type: application/pdf" \
- --data-binary @report.pdf
-
- RESP=$(curl -s -X POST "https://mineru.net/api/v1/uploads/$UPLOAD_ID/complete" \
- -H "Authorization: Bearer $MINERU_API_KEY" \
- -H "Content-Type: application/json" \
- -d "{\"sha256sum\":\"$SHA\"}")
-fi
-
-FILE_ID=$(echo "$RESP" | jq -r '.file.id')
-
-RESP=$(curl -s -X POST https://mineru.net/api/v1/parse/jobs \
- -H "Authorization: Bearer $MINERU_API_KEY" \
- -H "Content-Type: application/json" \
- -d "{
- \"files\": [{
- \"source\": {\"type\":\"file_id\",\"file_id\":\"$FILE_ID\"}
- }],
- \"tier\": \"standard\",
- \"output_formats\": [\"markdown\",\"middle_json\",\"zip\"]
- }")
-```
-
-如果 create upload 返回 `status:"completed"`,说明发生秒传,客户端可以直接使用响应里的 `file.id` 创建 parse job。
-
-创建任务后,保存响应里的 `job_id`:
-
-```bash
-JOB_ID=$(echo "$RESP" | jq -r '.job_id')
-```
-
-## 官方 API: 轮询任务并下载 Markdown
-
-```bash
-while true; do
- JOB=$(curl -s "https://mineru.net/api/v1/parse/jobs/$JOB_ID" \
- -H "Authorization: Bearer $MINERU_API_KEY")
-
- STATUS=$(echo "$JOB" | jq -r '.status')
- if [ "$STATUS" = "completed" ] || [ "$STATUS" = "partial" ] || \
- [ "$STATUS" = "failed" ] || [ "$STATUS" = "canceled" ]; then
- break
- fi
-
- sleep 2
-done
-
-MD_FILE_ID=$(echo "$JOB" | jq -r '.files[0].output_files.markdown.file_id')
-
-curl -L "https://mineru.net/api/v1/files/$MD_FILE_ID/content" \
- -H "Authorization: Bearer $MINERU_API_KEY" \
- -o report.md
-```
-
-实际客户端应使用指数退避轮询,最大间隔建议 30 秒。
-
-## 官方 API: URL 来源
-
-```bash
-curl -X POST https://mineru.net/api/v1/parse/jobs \
- -H "Content-Type: application/json" \
- -d '{
- "files": [{
- "source": {"type":"url","url":"https://example.com/doc.pdf"}
- }],
- "tier": null,
- "output_formats": ["markdown"]
- }'
-```
-
-不传 API Key 时进入 anonymous access level。anonymous 可以使用基础解析能力,但不能使用 callback 或高级导出格式。
-
-## 下载产物
-
-```bash
-curl -L "https://mineru.net/api/v1/files/$MD_FILE_ID/content" \
- -H "Authorization: Bearer $MINERU_API_KEY" \
- -o report.md
-```
-
-官方 API 返回 302,`-L` 会自动跟随重定向。
-
-## 本地 Server 示例
-
-先发现本地能力:
+完整的流式上传、缓存预检、有限重试、恢复和逐文件下载由 `scripts/http_task_example.py` 实现;两个 shell 示例入口都调用它。
```bash
-curl http://localhost:8000/v1/health
-curl http://localhost:8000/v1/tiers
+MINERU_API_URL=http://127.0.0.1:8000 bash scripts/http_task_example.sh report.pdf
+OUTPUT_FORMATS=default,docx MINERU_API_URL=http://127.0.0.1:8000 bash scripts/http_api_example.sh a.pdf b.pdf
+TASK_ID=job_... MINERU_API_URL=http://127.0.0.1:8000 bash scripts/http_task_example.sh a.pdf b.pdf
```
-本地 server 可以直接使用 `local` source:
-
-```bash
-mineru-kit api-server --allow-local-source
+直接提供 URL 时只需提交 JSON:
-curl -X POST http://localhost:8000/v1/parse/jobs \
- -H "Content-Type: application/json" \
- -d '{
- "files": [{
- "source": {"type":"local","path":"/data/docs/report.pdf"}
- }],
- "tier": null,
- "output_formats": ["markdown"]
- }'
+```json
+{"manifest":[{"client_file_id":"report","filename":"report.pdf","url":"https://example.com/report.pdf"}],"output_formats":["default"],"include_model_output":false}
```
-对 PDF/image,省略 `tier` 或传 `null` 时,本地 server 按 [解析 Tier](../tiers.md) 的默认选择策略选择非 `flash` 质量 tier。如果不可用,请求应失败,而不是使用 `flash` 返回解析结果。OFD/EPUB/Office/HTML/MHTML/CSV 按 [ADR-0024](../decisions/0024-file-type-tier-normalization.md) 与 [ADR-0028](../decisions/0028-csv-structured-flash-parsing.md) 的 API Server 批量规则归一为 `flash`,其它 text 不作为解析输入。
-
-## 本地 Server: 复用上传流程
-
-客户端也可以对本地 server 使用与官方 API 相同的上传流程。差异只在于 `upload_url` 指向本地 server:
-
-```bash
-SIZE=$(wc -c < report.pdf | tr -d ' ')
-
-RESP=$(curl -s -X POST http://localhost:8000/v1/uploads \
- -H "Content-Type: application/json" \
- -d "{\"filename\":\"report.pdf\",\"bytes\":$SIZE,\"mime_type\":\"application/pdf\",\"purpose\":\"parse\"}")
-
-UPLOAD_ID=$(echo "$RESP" | jq -r '.id')
-URL=$(echo "$RESP" | jq -r '.upload_url')
-METHOD=$(echo "$RESP" | jq -r '.upload_method')
-
-curl -X "$METHOD" "$URL" \
- -H "Content-Type: application/pdf" \
- --data-binary @report.pdf
-
-RESP=$(curl -s -X POST "http://localhost:8000/v1/uploads/$UPLOAD_ID/complete")
-FILE_ID=$(echo "$RESP" | jq -r '.file.id')
-```
+对 `/v1/tasks` 执行 POST,保存返回的 `task_id/status_url`。本地文件清单包含 `bytes` 和可选 `sha256sum`;仅对响应中的待上传项使用返回的 PUT 地址。
+完成后下载对应 `files[].result_url`。列表使用 `GET /v1/tasks?status=running,queued&limit=20`,取消使用 `DELETE /v1/tasks/{task_id}`。
-这样写的客户端可以在官方 API 和 Local Parse Server 之间复用绝大多数代码。
+详见 [Tasks 契约](tasks.md)、[SDK](../sdk/api-parser.md) 和 [同步包装服务](file-parse.md)。
diff --git a/docs/next/api/file-parse.md b/docs/next/api/file-parse.md
index 1b05ed9d..1b01ea5a 100644
--- a/docs/next/api/file-parse.md
+++ b/docs/next/api/file-parse.md
@@ -35,6 +35,7 @@ MINERU_API_URL=https://mineru.net/api MINERU_API_KEY=your-key \
| `ocr_mode` | `auto` | `auto/txt/ocr`。 |
| `page_range` | 整本 | 与 Tasks 一致,PDF 支持 `1-5`、`r1`、`all`;格式支持范围由上游校验。 |
| `output_formats` | `default` | `default/docx/html/latex`;重复同名字段表达列表,重复值按首次出现去重。 |
+| `include_model_output` | `false` | 附加可用原始模型输出;开启时 output_formats 必须包含 default,参与 request_id 的解析身份。 |
| `wait_timeout` | `3600` | 整数秒,范围 `0–3600`。 |
| `request_id` | 自动生成 | 1–64 个字母、数字、下划线或短横线。提前指定并保存它,可在连接中断后查询。 |
diff --git a/docs/next/api/files.md b/docs/next/api/files.md
new file mode 100644
index 00000000..3afdb21f
--- /dev/null
+++ b/docs/next/api/files.md
@@ -0,0 +1,20 @@
+# Files
+
+Files 保存源文件视图与已生成的结果。异步解析统一使用 [Tasks](tasks.md),客户端按 `files[].result_url` 下载,不自行拼接存储路径。
+
+| 方法 | 地址 | 行为 |
+|---|---|---|
+| GET | `/v1/files` | 按 `purpose/limit/after/order` 查询当前调用方的文件列表。 |
+| GET | `/v1/files/{file_id}` | 返回文件元数据。 |
+| GET | `/v1/files/{file_id}/content` | 流式下载已保存的解析产物;源文件不提供公共下载。 |
+| DELETE | `/v1/files/{file_id}` | 删除文件视图;正在执行和下载的引用继续受保护。 |
+
+元数据包括 `id/object/filename/bytes/created_at/expires_at/purpose/sha256sum`。
+`purpose` 为 `parse`(源文件)或 `parse_output`(结果);删除返回 `id/object/deleted`。
+列表使用 `object: list`、`data`、`first_id`、`last_id`、`has_more`;默认 20 条,上限 100。
+
+源文件通过 Tasks 的 JSON 清单和返回的上传链接创建。相同 hash 的有效缓存免传;不同文件名可以共享一份源字节,文件名和解析参数仍属于各自任务。
+API Server 与 Router 默认保留 86400 秒,每 300 秒扫描。`--retention-seconds` 优先于 `MINERU_API_RETENTION_SECONDS`,0 禁用正常保留期清理。上传链接固定的一小时窗口不随该设置改变。
+
+清理只删除过期视图和无引用字节。排队、执行、上传、下载持有的引用阻止物理删除;结果保留期从父任务完成资源清理后开始。
+Router 映射自身 Files 地址,按调用方隔离缓存与元数据,后台维持 worker 结果租约。SaaS 可以提供对象存储签名地址,不要求实现本地 Files 下载路由。
diff --git a/docs/next/api/health-models.md b/docs/next/api/health-models.md
index 05e7aee8..64f4c813 100644
--- a/docs/next/api/health-models.md
+++ b/docs/next/api/health-models.md
@@ -23,8 +23,9 @@ GET /v1/health HTTP/1.1
"version": "1.0.0",
"features": {
"webhook": true,
- "output_formats": ["markdown", "middle_json", "structured_content", "html", "latex", "docx", "zip"],
- "sources": ["file_id", "url", "inline"]
+ "task_output_formats": ["default", "docx", "html", "latex"],
+ "task_model_output": true,
+ "sources": ["upload", "url"]
}
}
```
@@ -37,8 +38,9 @@ GET /v1/health HTTP/1.1
| `version` | string | 是 | API 服务版本。 |
| `features` | object | 是 | 当前部署支持的外围能力。 |
| `features.webhook` | bool | 是 | 是否支持 Webhook 回调。 |
-| `features.output_formats` | array | 是 | 当前部署支持的 parse job `output_formats` 值。 |
-| `features.sources` | array | 是 | 当前部署允许的 parse job source 类型。 |
+| `features.task_output_formats` | array | 是 | 当前部署支持的 Tasks `output_formats` 值。 |
+| `features.task_model_output` | bool | 是 | 是否支持附加可用原始模型输出。 |
+| `features.sources` | array | 是 | 当前部署允许的 Tasks 输入类型(upload/url)。 |
错误:
@@ -73,7 +75,7 @@ GET /v1/models HTTP/1.1
}
```
-- `/v1/models` 只广告一个**虚拟模型**:id 为 `mineru-<服务版本>`,标识服务端版本,不对应任何可单独调用的引擎。解析质量始终通过 parse job 的 `tier` 字段选择,模型 id 不参与路由。
+- `/v1/models` 只广告一个**虚拟模型**:id 为 `mineru-<服务版本>`,标识服务端版本,不对应任何可单独调用的引擎。解析质量始终通过 Task 的 `tier` 字段选择,模型 id 不参与路由。
- `created` 为该部署的注册时间(本地 server 取进程启动时刻),同一部署内多次请求返回相同值。
- 模型列表与启动 `--tier`、tier 禁用开关无关:档位增减只反映在 `/v1/tiers`,虚拟模型恒在列表中。
- 服务端升级后 id 随版本变化(如 `mineru-4.0.11` → `mineru-4.0.12`)。客户端应重新发现,不要缓存 id 硬编码进请求。
@@ -156,7 +158,7 @@ Tier 对象字段:
v1 协议中面向用户读取文档的质量档位以 [解析 Tier](../tiers.md) 为准。具体服务当前支持哪些 tier,以 `/v1/tiers` 返回为准。
-请求创建 parse job 时,客户端可以省略 `tier` 或传 `null`,表示使用 [解析 Tier](../tiers.md) 中定义的默认选择策略。对 PDF/image 这类支持多 tier 的输入,默认选择不能等价于 `flash`;OFD/EPUB/Office/HTML/MHTML/CSV 按 [ADR-0024](../decisions/0024-file-type-tier-normalization.md) 与 [ADR-0028](../decisions/0028-csv-structured-flash-parsing.md) 的批量规则归一为 `flash`,其它 text 不作为解析输入。
+请求创建 Task 时,客户端可以省略 `tier` 或传 `null`,表示使用 [解析 Tier](../tiers.md) 中定义的默认选择策略。对 PDF/image 这类支持多 tier 的输入,默认选择不能等价于 `flash`;OFD/EPUB/Office/HTML/MHTML/CSV 按 [ADR-0024](../decisions/0024-file-type-tier-normalization.md) 与 [ADR-0028](../decisions/0028-csv-structured-flash-parsing.md) 的批量规则归一为 `flash`,其它 text 不作为解析输入。
更完整产品语义见 [解析 Tier](../tiers.md)。
@@ -170,13 +172,13 @@ Local Parse Server 的 `health` 通常返回:
"version": "1.0.0",
"features": {
"webhook": false,
- "output_formats": ["markdown", "middle_json", "structured_content", "zip"],
- "sources": ["file_id", "url", "inline"]
+ "task_output_formats": ["default", "docx", "html", "latex"],
+ "task_model_output": true,
+ "sources": ["upload", "url"]
}
}
```
-如果 Local Parse Server 启动时开启 `--allow-local-source`,`features.sources` 额外包含 `local`。
本地 `GET /v1/models` 和 `GET /v1/tiers` 必须反映当前 server 实际可用能力。常见组合:
diff --git a/docs/next/api/parse-jobs.md b/docs/next/api/parse-jobs.md
deleted file mode 100644
index da220683..00000000
--- a/docs/next/api/parse-jobs.md
+++ /dev/null
@@ -1,366 +0,0 @@
-# Parse Jobs
-
-页码规范与历史结果兼容说明见 [PDF 页码范围规范](../page-ranges.md)。
-
-状态: Draft
-读者: API 使用者、服务端开发者、SDK 开发者
-范围: 解析任务创建、查询和轮询
-来源: 由根目录旧 Unified API 底稿迁移整理而来
-
-## 任务模型
-
-Parse Job 是一次文档解析请求,可以包含一个或多个输入文件。任务创建后进入队列,服务端按 `tier`、文件类型和当前可用解析能力选择具体引擎。
-
-Job 状态:
-
-| 状态 | 含义 | 终态 |
-|------|------|:--:|
-| `queued` | 已创建,等待调度。 | 否 |
-| `running` | 至少一个文件正在解析。 | 否 |
-| `completed` | 全部文件解析成功。 | 是 |
-| `partial` | 部分文件成功,部分文件失败。 | 是 |
-| `failed` | 全部文件失败。 | 是 |
-| `canceled` | 用户取消。 | 是 |
-
-客户端应把 `completed`、`partial`、`failed`、`canceled` 都视为终态。
-
-一次上传并提交、内联结果与同步包装见 [V1 便捷解析](tasks.md),它们共用本页的 Job 和任务 ID。
-
-## POST `/v1/parse/jobs`
-
-创建解析任务。
-
-```json
-{
- "files": [
- {
- "source": {
- "type": "file_id",
- "file_id": "file-01HXYZ123ABCDEF"
- },
- "page_range": "1-10"
- }
- ],
- "tier": null,
- "ocr_mode": "auto",
- "output_formats": ["markdown", "middle_json", "structured_content", "zip"],
- "callback": {
- "url": "https://your.app/mineru-webhook",
- "secret": "abc123"
- }
-}
-```
-
-顶层字段:
-
-| 字段 | 类型 | 必填 | 默认 | 说明 |
-|------|------|:--:|------|------|
-| `files` | array | 是 | 无 | 至少 1 个,最多由 access level 决定。 |
-| `tier` | string 或 null | 否 | `null` | 当前服务支持的 tier 或 `null`。省略或传 `null` 表示使用默认选择策略;完整 tier 语义见 [解析 Tier](../tiers.md) 与 [ADR-0024](../decisions/0024-file-type-tier-normalization.md)。HTML 输入自动路由到 HTML 解析器。 |
-| `ocr_mode` | string | 否 | `"auto"` | 本次任务的解析模式:`auto` 自动判断、`txt` 使用文本层、`ocr` 强制 OCR。覆盖所有 tier,沿用底层 PDF/图片解析语义;其他格式使用各自的固定解析路径。非法值或 `null` 返回 HTTP 400。 |
-| `output_formats` | array | 否 | `["markdown"]` | 请求产物格式。 |
-| `callback` | object | 否 | `null` | Webhook 回调配置,官方 API registered 用户可用。 |
-
-`ocr_mode` 在同一任务的文件间共用,任务之间独立。省略时固定使用 `auto`;Local Parse Server 不再通过启动参数设置全局 OCR 模式。
-
-HTML job 固定使用静态 `auto` 正文选择,不执行 JavaScript,也不下载页面引用的远程图片。URL source 的来源 URL
-会用于解析相对链接;本地 source 的相对图片仅允许位于源 HTML 的安全根目录内。
-
-### `files[]`
-
-| 字段 | 类型 | 必填 | 说明 |
-|------|------|:--:|------|
-| `source` | object | 是 | 文件来源。 |
-| `page_range` | string | 否 | 仅针对单个 PDF 文件的页码范围;图片及其他非 PDF 输入显式传入时返回 `page_range_invalid`。 |
-
-### Source 类型
-
-`file_id` 引用已经存在的 File:
-
-```json
-{
- "type": "file_id",
- "file_id": "file-01HXYZ123ABCDEF"
-}
-```
-
-`url` 由服务端拉取文件,并在内部转为 File:
-
-```json
-{
- "type": "url",
- "url": "https://example.com/doc.pdf"
-}
-```
-
-`inline` 用于小文件内嵌,`data` 是 base64 字符串:
-
-```json
-{
- "type": "inline",
- "name": "scan.jpg",
- "data": "base64..."
-}
-```
-
-`local` 仅 Local Parse Server 支持:
-
-```json
-{
- "type": "local",
- "path": "/data/docs/report.pdf"
-}
-```
-
-Source 字段约束:
-
-| `type` | 必填字段 | 约束 |
-|--------|----------|------|
-| `file_id` | `file_id` | File 必须存在、属于当前租户,且 `purpose` 适合解析。 |
-| `url` | `url` | 官方 API 只允许公网 URL;拒绝私有网段、链路本地和 cloud metadata 地址。 |
-| `inline` | `name`、`data` | base64 解码后字节数默认不超过 1MB。 |
-| `local` | `path` | 仅本地 server 支持;只有 `features.sources` 包含 `local` 时可用。 |
-
-客户端应以 `GET /v1/health` 返回的 `features.sources` 作为当前部署实际允许的 source 类型。
-
-Local Parse Server 的 source 策略由启动参数决定:
-
-- `--allow-local-source` 控制是否允许 `local` source;未开启时拒绝 `local` source,开启后允许读取 server 进程权限范围内的任意本地路径。
-- `--max-inline-bytes` 控制 `inline` source 解码后的最大字节数,默认 1MB。
-- `--allow-http-source` 控制 `url` source 是否允许 `http://`;默认只允许 `https://`。
-
-### 文件级解析字段
-
-| 字段 | 类型 | 默认 | 说明 |
-|------|------|------|------|
-| `page_range` | string | `null` | PDF 页码范围。省略或传 `null` 表示解析整个文件;图片及其他非 PDF 输入只允许省略或传 `null`。使用 `-` 分隔,如 `1-10`、`1,3,5-7`、`r5-r1`。 |
-
-OCR 策略和图片分析能力由 `tier` 与服务端实际引擎自动决定,客户端不能通过文件级参数单独关闭或开启。
-
-### 输出格式
-
-| 格式 | 产物类型 | 含义 | 需 API Key |
-|------|----------|------|:--:|
-| `markdown` | 文本 | Markdown 文本。 | 否 |
-| `middle_json` | 文本 | 完整 Middle JSON,中间结构和高级调试。 | 否 |
-| `structured_content` | 文本 | 面向 Agent 和新客户端的结构化内容 JSON。 | 否 |
-| `html` | 文本 | HTML 导出。 | 是 |
-| `latex` | 文本 | LaTeX 导出。 | 是 |
-| `docx` | 二进制 | Word 文档导出。 | 是 |
-| `zip` | 二进制 | 自包含解析包;图片 sidecar 只通过 zip 返回。 | 否 |
-
-所有产物都以 `purpose:"parse_output"` 的 File 对象存储。Job 响应只返回产物 File 引用;所有产物内容都通过 `GET /v1/files/{file_id}/content` 获取。
-
-`images` 不是可请求的 `output_formats` 值。需要图片 sidecar 的客户端应请求 `zip`,并从 zip 包内读取 `middle_json` 引用的图片路径。
-
-客户端应以 `GET /v1/health` 返回的 `features.output_formats` 作为当前部署实际支持格式。Local Parse Server 可以只暴露基础格式。
-
-`json`、`content_list_v2` 不是 NEXT 版正式格式名,不进入公开格式集合。命名决策见 [ADR-0001](../decisions/0001-json-output-formats.md)。
-
-### 创建响应
-
-创建成功时返回 `202`。客户端通过 `GET /v1/parse/jobs/{job_id}` 查询任务状态和结果。
-
-```json
-{
- "job_id": "job_01HXYZ123ABCDEF",
- "status": "queued",
- "created_at": "2026-05-21T08:30:00Z",
- "tier": "standard",
- "output_formats": ["markdown", "middle_json", "structured_content", "zip"],
- "access_level": "registered",
- "files": [
- {
- "file_id": "file-01HXYZ123ABCDEF",
- "name": "report.pdf",
- "page_range": "1-12",
- "status": "queued"
- }
- ],
- "links": {
- "self": "/v1/parse/jobs/job_01HXYZ123ABCDEF",
- "cancel": "/v1/parse/jobs/job_01HXYZ123ABCDEF"
- }
-}
-```
-
-创建响应字段:
-
-| 字段 | 类型 | 说明 |
-|------|------|------|
-| `job_id` | string | Job ID。 |
-| `status` | string | 当前 job 状态。 |
-| `created_at` | string | ISO-8601 UTC。 |
-| `tier` | string | 本次 job 的请求/解析 tier。请求省略或传 `null` 时,响应中返回解析后的 job tier;非 PDF/image 文件可能按 ADR-0024 在文件执行时归一为 `flash`,暂不暴露 file-level effective tier。 |
-| `output_formats` | array | 实际接受的输出格式。 |
-| `access_level` | string | `anonymous` 或 `registered`。 |
-| `files[]` | array | 输入文件的队列状态。 |
-| `links.self` | string | 查询任务 URL。 |
-| `links.cancel` | string | 取消任务 URL。 |
-
-### 终态响应
-
-`GET /v1/parse/jobs/{job_id}` 查询到终态任务时返回 `200`。
-
-```json
-{
- "job_id": "job_01HXYZ123ABCDEF",
- "status": "completed",
- "created_at": "2026-05-21T08:30:00Z",
- "started_at": "2026-05-21T08:30:02Z",
- "finished_at": "2026-05-21T08:30:48Z",
- "tier": "standard",
- "output_formats": ["markdown", "middle_json", "zip"],
- "access_level": "registered",
- "progress": {
- "completed": 1,
- "failed": 0,
- "total": 1
- },
- "files": [
- {
- "file_id": "file-01HXYZ123ABCDEF",
- "name": "report.pdf",
- "page_range": "1-12",
- "status": "completed",
- "parse": {
- "model_used": "mineru-4.0.11",
- "duration_ms": 8234,
- "parser_version": "3.1.14"
- },
- "output_files": {
- "markdown": {
- "file_id": "file-01HXYZMD000001",
- "bytes": 51407
- },
- "middle_json": {
- "file_id": "file-01HXYZJSON000002",
- "bytes": 184320
- },
- "zip": {
- "file_id": "file-01HXYZZIP000004",
- "bytes": 286720
- }
- }
- }
- ]
-}
-```
-
-文件结果字段:
-
-| 字段 | 类型 | 说明 |
-|------|------|------|
-| `file_id` | string 或 null | 输入 File ID。`local` source 可以为 `null`。 |
-| `name` | string | 文件显示名。 |
-| `page_range` | string | 服务端规范化后的实际解析页码范围。请求未指定时为 `1-{total}`;请求指定时,服务端先按文件总页数展开倒数页码 `rN`,再去重并合并连续页码区间。例如 `1,2,3,r1,r1` 在 10 页文件中规范化为 `1-3,10`。 |
-| `status` | string | 文件级状态。 |
-| `parse.model_used` | string | 实际模型 ID。仅当该文件 `status="completed"` 时出现。 |
-| `parse.duration_ms` | integer | 单文件从开始处理到输出生成完毕的毫秒耗时,包含解析和结果导出/打包,不包含排队、客户端下载或 WebUI 渲染时间。仅当该文件 `status="completed"` 时出现。 |
-| `parse.parser_version` | string | 解析器版本。仅当该文件 `status="completed"` 时出现。 |
-| `output_files` | object | 各输出格式对应的 File 引用。仅当该文件 `status="completed"` 时出现。 |
-| `error` | object | 文件失败时出现,包含 `code` 和 `message`。仅当该文件 `status="failed"` 时出现。 |
-
-`queued`、`running`、`canceled` 状态下的文件可以不返回 `parse`、`output_files` 和 `error`。`partial` job 中,成功文件按 `completed` 文件返回 `parse` 与 `output_files`,失败文件按 `failed` 文件返回 `error`。
-
-## GET `/v1/parse/jobs/{job_id}`
-
-查询任务状态和结果。终态 job 使用与终态响应相同的结构,但只返回产物 File 引用,不返回产物内容。未完成 job 可以只返回当前 `status`、`progress` 和 `files` 的阶段信息。
-
-轮询建议:
-
-- 初始间隔 2 秒。
-- 使用指数退避。
-- 最大间隔 30 秒。
-- 进入终态后停止轮询。
-
-错误:
-
-| HTTP | code |
-|------|------|
-| 404 | `job_not_found` |
-
-## GET `/v1/parse/jobs`
-
-列出任务。anonymous 用户调用返回 `403 list_requires_api_key`。
-
-查询参数:
-
-| 参数 | 类型 | 默认 | 说明 |
-|------|------|------|------|
-| `status` | string | `null` | 按状态过滤,可逗号分隔。 |
-| `limit` | integer | `20` | 返回数量,范围 `[1, 100]`。 |
-| `after` | string | `null` | 游标,上一页响应的 `last_id`。 |
-| `created_after` | string | `null` | ISO-8601 时间,只返回此时间之后创建的任务。 |
-| `order` | string | `desc` | `asc` 或 `desc`。 |
-
-响应:
-
-```json
-{
- "object": "list",
- "data": [
- {
- "job_id": "job_01HXYZ...",
- "status": "completed",
- "created_at": "2026-05-21T08:30:00Z",
- "file_count": 2
- }
- ],
- "first_id": "job_01HXYZAAA...",
- "last_id": "job_01HXYZBBB...",
- "has_more": true
-}
-```
-
-错误:
-
-| HTTP | code |
-|------|------|
-| 400 | `invalid_request` |
-| 403 | `list_requires_api_key` |
-
-## DELETE `/v1/parse/jobs/{job_id}`
-
-取消未进入终态的任务。
-
-```json
-{
- "job_id": "job_01HXYZ...",
- "status": "canceled",
- "canceled_at": "2026-05-21T08:30:30Z"
-}
-```
-
-错误:
-
-| HTTP | code | 场景 |
-|------|------|------|
-| 404 | `job_not_found` | job 不存在或不属于当前租户。 |
-| 409 | `job_already_terminal` | job 已完成、失败或已取消。 |
-
-## 创建任务错误
-
-`POST /v1/parse/jobs` 常见错误:
-
-| HTTP | code | 场景 |
-|------|------|------|
-| 400 | `invalid_request` | 请求结构非法、source 缺字段、page_range 不合法等。 |
-| 400 | `unsupported_output_format` | 输出格式不支持。 |
-| 400 | `unsupported_source` | 当前部署不支持该 source 类型。 |
-| 403 | `feature_requires_api_key` | anonymous 使用高级输出格式或 callback。 |
-| 404 | `file_not_found` | `file_id` 不存在或不可见。 |
-| 413 | `file_too_large` | 文件超过当前 access level 限制。 |
-| 429 | `rate_limit_exceeded` | 触发 parse 类限流。 |
-| 503 | `quality_tier_unavailable` | 请求的质量 tier 不可用,或 PDF/image 默认选择策略找不到可用的非 `flash` tier。 |
-
-## 本地 Server 差异
-
-Local Parse Server 的任务 API 与官方 API 保持同一结构,但有以下实现差异:
-
-- 不支持 Webhook 时,`health.features.webhook` 必须为 `false`;收到 `callback` 时应返回 `400 invalid_request` 或明确忽略。
-- `health.features.output_formats` 必须反映本地 server 实际支持的输出格式;当前本地实现支持 `markdown`、`middle_json`、`structured_content` 和 `zip`。
-- `health.features.sources` 必须反映本地 server 实际允许的 source 类型;只有启动时开启 `--allow-local-source` 才包含 `local`。
-- `local` source 可以不生成输入 `file_id`;响应中应保留 `name` 和文件级状态。
-- 对 PDF/image,省略 `tier` 或传 `null` 时,只能按默认选择策略选择本地可发现的非 `flash` 质量 tier,不能回退到 `flash`;如果只有 `flash` 可用,应返回 `quality_tier_unavailable`。
-- 对 OFD/EPUB/Office/HTML/MHTML/CSV/TSV,API Server job 按批量规则处理,即使 job tier 是质量 tier,文件实际解析也按 `flash` 语义归一;其它 text 不进入 parse job;暂不新增 file-level effective tier 字段。
diff --git a/docs/next/api/responses.md b/docs/next/api/responses.md
index 8615a94c..bb99a373 100644
--- a/docs/next/api/responses.md
+++ b/docs/next/api/responses.md
@@ -11,7 +11,7 @@
```json
{
- "job_id": "job_01HXYZ123ABCDEF",
+ "task_id": "job_01HXYZ123ABCDEF",
"status": "queued"
}
```
@@ -127,6 +127,5 @@ Idempotency-Key: 7f5b7f7c-5c5a-4b2f-bb89-0c901c9a2d6a
Local Parse Server 使用同一套错误 envelope。差异主要体现在错误来源:
- 未启用 API Key 时,不会产生 `invalid_api_key`。
-- 请求 `local` source 但本地 server 未开启 `--allow-local-source`,应返回 `400 unsupported_source`。
- PDF/image 省略 `tier` 或传 `null` 且无法解析到非 `flash` 质量 tier 时,应返回 tier 相关错误,不能降级到 `flash`。OFD/EPUB/Office/HTML/MHTML/CSV 按 [ADR-0024](../decisions/0024-file-type-tier-normalization.md) 与 [ADR-0028](../decisions/0028-csv-structured-flash-parsing.md) 归一为 `flash`,其它 text 不作为解析输入。
- 解析能力缺失时,应优先返回可操作的 `engine_error`,而不是泛化为 `internal_error`。
diff --git a/docs/next/api/tasks.md b/docs/next/api/tasks.md
index 1545043c..c6fa17bb 100644
--- a/docs/next/api/tasks.md
+++ b/docs/next/api/tasks.md
@@ -17,7 +17,7 @@
创建请求不再接受表单、multipart 或文件字节,其他 Content-Type 返回 `415`。
业务参数必须放在 JSON 主体中,POST 查询参数、未知字段、非法值返回 `400 invalid_request`。
上传文件的 PUT 仍发送原始字节,使用返回的 `upload_headers`;签名 URL 必须原样使用。
-Task ID 是父任务标识,不承诺同时是 `/v1/parse/jobs` 的 Job ID。
+Task ID 是父任务标识;内部逐文件执行 ID 不属于公共契约。
| JSON 字段 | 默认值 | 说明 |
|---|---|---|
@@ -27,6 +27,8 @@ Task ID 是父任务标识,不承诺同时是 `/v1/parse/jobs` 的 Job ID。
| `page_range` | 整本 | 全局 PDF 页范围,支持 `1-5`、`r1`、`all` 等现有语法。 |
| `output_formats` | `["default"]` | 非空列表,值为 `default/docx/html/latex` 中一个或多个;重复值按首次出现去重。 |
+`include_model_output` 默认为 `false`。为 `true` 时必须包含 `default` 输出格式,存在原始模型输出时在结果 ZIP 中增加 `model_output.json`;其他格式组合不满足条件时返回 400。创建和查询响应都携带该选项。
+
每个清单条目包含:
| 字段 | 必填 | 说明 |
@@ -197,6 +199,7 @@ TASK_ID=job_... MINERU_API_URL=http://127.0.0.1:8000 \
| `SEND_HASH` | `1` | `0` 时不计算或提交 hash。 |
| `TASK_ID` | 空 | 恢复已有任务,跳过 ready 文件。 |
| `MINERU_TIER`、`OCR_MODE`、`PAGE_RANGE` | 服务默认 / `auto` / 整本 | 解析参数。 |
+| `INCLUDE_MODEL_OUTPUT` | `0` | `1` 时附加可用模型输出,必须包含 default 格式。 |
| `OUTPUT_FORMATS` | `default` | 逗号分隔,转为 JSON 数组。 |
| `RESULT_DIR` | `results` | 逐文件结果目录。 |
| `MAX_POLLS`、`POLL_INTERVAL` | `300`、`1` | 查询预算和间隔秒数。 |
@@ -209,18 +212,25 @@ TASK_ID=job_... MINERU_API_URL=http://127.0.0.1:8000 \
## 部署、Router 与迁移
本地不设固定客户端上传大小或 PDF 页数上限;`usage.limits.max_file_size_bytes/max_pages_per_file` 为 `null`。
-URL 下载沿用 Jobs 的策略:默认 HTTPS;HTTP 需启用 `--allow-http-source`,下载超时默认 60 秒、大小默认 200 MiB。
+URL 下载沿用服务的来源策略:默认 HTTPS;HTTP 需启用 `--allow-http-source`,下载超时默认 60 秒、大小默认 200 MiB。
任务记录和上传索引仍为进程内状态,重启不保证恢复。
Router 按就绪文件独立选择 worker 并预占负载,同一父 Task 可使用多个 worker。
URL 交给 worker 下载,客户端上传字节按 worker 的缓存预检结果传输。
-健康接口的 `features.task_output_formats` 单独声明 Tasks 格式;原 `features.output_formats` 继续描述 Jobs。
+健康接口的 `features.task_output_formats` 声明 Tasks 格式,`sources` 为 `upload/url`,`task_model_output` 声明模型输出能力。
不声明新 Tasks 能力的旧 worker 不参与 Tasks 调度,更新时应先更新 worker,再更新 Router 和客户端。
Router 使用有界、可续租的内部结果租约保护 worker 产物,上传窗口不受结果租约影响;Router 丢失后租约会自行结束。
本次协议不保留旧表单、直传或 `markdown/middle_json/structured_content/zip` Tasks 格式别名。
客户端应改用 JSON、`files[].result_url` 及逐文件保存方式。
-独立 `/v1/uploads`、`/v1/parse/jobs` 与 SDK 的现有请求及输出协议保持不变。
API Server 和 Router 不提供 `/v1/file_parse`、`/v1/tasks/{task_id}/result`。
需要同步单文件调用时,独立启动 [file_parse 包装服务](file-parse.md),由它调用本页的共同 Tasks 契约。
SaaS Go 服务和对象存储配置由部署方按此共同契约实现,不属于本地 Python 服务代码。
+
+## 父任务列表与缓存
+
+`GET /v1/tasks` 返回当前调用方的父任务摘要。支持 `status`(逗号分隔)、`limit`(默认 20,上限 100)、`after`(上一页 last_id)、`order`(默认 desc)、`created_after`。按 created_at 和 task_id 稳定排序;游标不存在、过期或属于其他调用方时返回 404。
+
+响应为 `object=list/data/first_id/last_id/has_more`;每项包含 `task_id/status/created_at/file_count`,不展示内部子执行。
+
+SDK 即使连接同机也计算 hash 并提交清单。缓存有效且已完成上传的同 hash 后续免传;并发首次上传允许重复传输但源字节去重。默认缓存保留 24 小时、每 5 分钟扫描,沿用 retention 配置;活动任务与传输引用保护字节。缓存过期或服务重启后允许重新上传。
diff --git a/docs/next/api/uploads-files.md b/docs/next/api/uploads-files.md
deleted file mode 100644
index c55b8b3e..00000000
--- a/docs/next/api/uploads-files.md
+++ /dev/null
@@ -1,387 +0,0 @@
-# Uploads 与 Files
-
-状态: Draft
-读者: API 使用者、服务端开发者、SDK 开发者
-范围: 上传生命周期、文件对象、文件列表、产物下载
-来源: 由根目录旧 Unified API 底稿迁移整理而来
-
-## 上传模型
-
-官方 API 使用三段式上传生命周期:
-
-1. `POST /v1/uploads` 创建 upload。
-2. 客户端向返回的 `upload_url` 上传字节。
-3. `POST /v1/uploads/{upload_id}/complete` 完成 upload 并生成 file。
-
-这套流程对齐 OpenAI Uploads API 的生命周期,但字节传输不经过 API 网关。官方 API 的 `upload_url` 指向对象存储预签名 URL;本地 server 的 `upload_url` 可以指向 server 自身的临时上传 endpoint。
-
-## Upload 对象
-
-| 字段 | 类型 | 必带 | 说明 |
-|------|------|:--:|------|
-| `id` | string | 是 | `upload_<random>`,创建时分配。 |
-| `object` | string | 是 | 固定为 `"upload"`。 |
-| `bytes` | integer | 是 | 创建时声明的预期上传字节数。 |
-| `created_at` | integer | 是 | Unix 秒级时间戳。 |
-| `expires_at` | integer | 是 | Upload 过期时间。 |
-| `filename` | string | 是 | 用户传入的文件名。 |
-| `purpose` | string | 是 | `"parse"` 或 `"input_image"`。 |
-| `mime_type` | string | 是 | MIME 类型。 |
-| `sha256sum` | string 或 null | 是 | 64 位小写 hex;未提供时为 `null`。 |
-| `status` | string | 是 | `pending`、`completed`、`cancelled`、`expired`。 |
-| `upload_url` | string 或 null | 条件 | `pending` 时返回;秒传或 complete 后为 `null`。 |
-| `upload_method` | string | 条件 | `pending` 时返回,当前为 `"PUT"`。 |
-| `upload_headers` | object | 条件 | 上传字节时必须携带的 header。 |
-| `file` | object | 条件 | `completed` 时返回 File 对象。 |
-
-Upload 状态机:
-
-| 当前状态 | 可执行操作 | 下一状态 |
-|----------|------------|----------|
-| `pending` | 上传字节后 complete | `completed` |
-| `pending` | cancel | `cancelled` |
-| `pending` | 过期任务清理 | `expired` |
-| `completed` | 无 | 终态 |
-| `cancelled` | 无 | 终态 |
-| `expired` | 无 | 终态 |
-
-## File 对象
-
-| 字段 | 类型 | 必带 | 说明 |
-|------|------|:--:|------|
-| `id` | string | 是 | `file-<random>`,全局唯一不透明标识。 |
-| `object` | string | 是 | 固定为 `"file"`。 |
-| `bytes` | integer | 是 | 文件字节数。 |
-| `created_at` | integer | 是 | Unix 秒级时间戳。 |
-| `expires_at` | integer 或 null | 是 | 文件过期时间;`null` 表示不自动过期。 |
-| `filename` | string | 是 | 文件名。 |
-| `purpose` | string | 是 | `"parse"`、`"input_image"` 或 `"parse_output"`。 |
-| `sha256sum` | string 或 null | 是 | 64 位小写 hex;未知时为 `null`。 |
-
-`purpose:"parse"` 表示待解析源文件。`purpose:"input_image"` 表示 Chat/Responses 输入图片。`purpose:"parse_output"` 表示解析产物,如 markdown、middle_json、structured_content、docx、zip。解析图片 sidecar 不作为独立 parse output;需要图片时从 zip 中读取。
-
-## POST `/v1/uploads`
-
-创建上传会话。请求体:
-
-| 字段 | 类型 | 必填 | 说明 |
-|------|------|:--:|------|
-| `filename` | string | 是 | 文件名。 |
-| `bytes` | integer | 是 | 预期字节数,服务端用它做配额预校验。 |
-| `mime_type` | string | 是 | MIME 类型,如 `application/pdf`。 |
-| `purpose` | string | 是 | `"parse"` 或 `"input_image"`。 |
-| `sha256sum` | string | 否 | 64 位小写 hex。提供时启用秒传和完整性校验。 |
-| `expires_after` | object | 否 | Upload 过期策略。 |
-| `expires_after.anchor` | string | 否 | 当前仅支持 `"created_at"`。 |
-| `expires_after.seconds` | integer | 否 | 过期秒数,范围 `[3600, 2592000]`。 |
-
-示例:
-
-```json
-{
- "filename": "report.pdf",
- "bytes": 1048576,
- "mime_type": "application/pdf",
- "purpose": "parse",
- "sha256sum": "a1b2c3d4e5f6abcd1234567890abcdef1234567890abcdef1234567890abcdef",
- "expires_after": {
- "anchor": "created_at",
- "seconds": 3600
- }
-}
-```
-
-### 秒传响应
-
-如果传入 `sha256sum` 且命中当前租户已有文件,返回 `200`,`status` 为 `completed`,并内嵌 `file`。客户端无需上传字节,也无需调用 complete。
-
-```json
-{
- "id": "upload_01HXYZ123ABCDEF",
- "object": "upload",
- "bytes": 1048576,
- "created_at": 1719184911,
- "expires_at": 1719188511,
- "filename": "report.pdf",
- "purpose": "parse",
- "mime_type": "application/pdf",
- "sha256sum": "a1b2c3d4e5f6abcd1234567890abcdef1234567890abcdef1234567890abcdef",
- "status": "completed",
- "upload_url": null,
- "file": {
- "id": "file-01HXYZ123ABCDEF",
- "object": "file",
- "bytes": 1048576,
- "created_at": 1719184911,
- "expires_at": 1719188511,
- "filename": "report.pdf",
- "purpose": "parse",
- "sha256sum": "a1b2c3d4e5f6abcd1234567890abcdef1234567890abcdef1234567890abcdef"
- }
-}
-```
-
-秒传作用域是租户级。不同租户之间即使内容相同,也不能互相看到对方文件。
-
-### 待上传响应
-
-需要上传字节时返回 `200`,`status` 为 `pending`。
-
-```json
-{
- "id": "upload_01HXYZ123ABCDEF",
- "object": "upload",
- "bytes": 1048576,
- "created_at": 1719184911,
- "expires_at": 1719188511,
- "filename": "report.pdf",
- "purpose": "parse",
- "mime_type": "application/pdf",
- "sha256sum": "a1b2c3d4e5f6abcd1234567890abcdef1234567890abcdef1234567890abcdef",
- "status": "pending",
- "upload_url": "https://oss.example/mineru/uploads/upload_01HXYZ...?Signature=...",
- "upload_method": "PUT",
- "upload_headers": {
- "Content-Type": "application/pdf",
- "x-amz-content-sha256": "a1b2c3d4e5f6..."
- }
-}
-```
-
-客户端必须使用响应中的 `upload_method`、`upload_url` 和 `upload_headers` 上传字节。预签名 URL 自带授权,不需要 Bearer Token。
-
-错误:
-
-| HTTP | code | 场景 |
-|------|------|------|
-| 400 | `invalid_request` | 缺少必填字段或字段类型错误。 |
-| 400 | `invalid_sha256sum` | `sha256sum` 不是 64 位小写 hex。 |
-| 413 | `file_too_large` | 文件超过当前 access level 的大小限制。 |
-| 429 | `rate_limit_exceeded` | 触发 upload 类限流。 |
-
-## PUT `{upload_url}`
-
-上传文件字节。此请求不属于 `/v1` API envelope,由 `POST /v1/uploads` 返回的 URL 和 headers 决定。
-
-```http
-PUT https://oss.example/mineru/uploads/upload_01HXYZ...?Signature=...
-Content-Type: application/pdf
-x-amz-content-sha256: a1b2c3d4e5f6...
-
-<binary file bytes>
-```
-
-成功时对象存储或本地临时上传 endpoint 返回 `200 OK`。客户端不应假设响应体存在。
-
-## POST `/v1/uploads/{upload_id}/complete`
-
-标记 upload 完成并生成 File 对象。请求体可为空;如果提供 `sha256sum`,它只用于校验,不能在 complete 阶段触发秒传。
-
-```json
-{
- "sha256sum": "a1b2c3d4e5f6abcd1234567890abcdef1234567890abcdef1234567890abcdef"
-}
-```
-
-响应为 `status:"completed"` 的 Upload 对象,并内嵌 File。
-
-```json
-{
- "id": "upload_01HXYZ123ABCDEF",
- "object": "upload",
- "bytes": 1048576,
- "created_at": 1719184911,
- "expires_at": 1719188511,
- "filename": "report.pdf",
- "purpose": "parse",
- "mime_type": "application/pdf",
- "sha256sum": "a1b2c3d4e5f6abcd1234567890abcdef1234567890abcdef1234567890abcdef",
- "status": "completed",
- "upload_url": null,
- "file": {
- "id": "file-01HXYZ123ABCDEF",
- "object": "file",
- "bytes": 1048576,
- "created_at": 1719184911,
- "expires_at": 1719188511,
- "filename": "report.pdf",
- "purpose": "parse",
- "sha256sum": "a1b2c3d4e5f6abcd1234567890abcdef1234567890abcdef1234567890abcdef"
- }
-}
-```
-
-错误:
-
-| HTTP | code | 场景 |
-|------|------|------|
-| 400 | `invalid_sha256sum` | complete 请求中的 `sha256sum` 格式错误。 |
-| 400 | `file_hash_mismatch` | 实际字节 SHA-256 与声明不一致。 |
-| 400 | `bytes_mismatch` | 实际字节数与 create 时的 `bytes` 不一致。 |
-| 404 | `upload_not_found` | upload 不存在或不属于当前租户。 |
-| 409 | `upload_not_ready` | 字节尚未上传完成。 |
-| 409 | `upload_already_terminal` | upload 已是终态。 |
-
-## POST `/v1/uploads/{upload_id}/cancel`
-
-取消 `pending` 状态的 upload。响应:
-
-```json
-{
- "id": "upload_01HXYZ123ABCDEF",
- "object": "upload",
- "bytes": 1048576,
- "created_at": 1719184911,
- "expires_at": 1719188511,
- "filename": "report.pdf",
- "purpose": "parse",
- "mime_type": "application/pdf",
- "sha256sum": null,
- "status": "cancelled"
-}
-```
-
-错误:
-
-| HTTP | code | 场景 |
-|------|------|------|
-| 404 | `upload_not_found` | upload 不存在或不属于当前租户。 |
-| 409 | `upload_already_terminal` | upload 已完成、已取消或已过期。 |
-
-## GET `/v1/uploads/{upload_id}`
-
-查询 upload 当前状态。响应结构与 create、complete、cancel 返回的 Upload 对象一致。`completed` 状态必须包含 `file`。
-
-错误:
-
-| HTTP | code |
-|------|------|
-| 404 | `upload_not_found` |
-
-## GET `/v1/files`
-
-列出当前租户文件,使用游标分页。anonymous 用户调用返回 `403 list_requires_api_key`。
-
-查询参数:
-
-| 参数 | 类型 | 默认 | 说明 |
-|------|------|------|------|
-| `after` | string | `null` | 游标,上一页响应的 `last_id`。 |
-| `limit` | integer | `100` | 返回数量,范围 `[1, 1000]`。 |
-| `order` | string | `desc` | `asc` 或 `desc`,按 `created_at` 排序。 |
-| `purpose` | string | `null` | 可选 `parse`、`input_image`、`parse_output`。 |
-
-响应:
-
-```json
-{
- "object": "list",
- "data": [
- {
- "id": "file-01HXYZ123ABCDEF",
- "object": "file",
- "bytes": 1048576,
- "created_at": 1719184911,
- "expires_at": 1719188511,
- "filename": "report.pdf",
- "purpose": "parse",
- "sha256sum": "a1b2c3d4e5f6abcd1234567890abcdef1234567890abcdef1234567890abcdef"
- }
- ],
- "first_id": "file-01HXYZ123ABCDEF",
- "last_id": "file-01HXYZ123ABCDEF",
- "has_more": false
-}
-```
-
-错误:
-
-| HTTP | code | 场景 |
-|------|------|------|
-| 400 | `invalid_request` | 分页参数非法。 |
-| 403 | `list_requires_api_key` | anonymous 用户访问列表。 |
-
-## GET `/v1/files/{file_id}`
-
-查询单个文件元信息。响应为 File 对象。
-
-错误:
-
-| HTTP | code |
-|------|------|
-| 404 | `file_not_found` |
-
-## GET `/v1/files/{file_id}/content`
-
-下载 `purpose:"parse_output"` 的解析产物。源文件 `purpose:"parse"` 不提供内容下载。
-
-官方 API 返回 302,客户端跟随重定向到 CDN 或对象存储签名 URL。
-
-```http
-HTTP/1.1 302 Found
-Location: https://cdn.mineru.net/files/file-01HXYZ.../signed?Signature=...
-```
-
-客户端示例:
-
-```bash
-curl -L "https://mineru.net/api/v1/files/$MD_FILE_ID/content" \
- -H "Authorization: Bearer $MINERU_API_KEY" \
- -o report.md
-```
-
-错误:
-
-| HTTP | code | 场景 |
-|------|------|------|
-| 404 | `file_not_found` | 文件不存在或不属于当前租户。 |
-| 400 | `invalid_request` | 文件不是可下载的 `parse_output`。 |
-
-## DELETE `/v1/files/{file_id}`
-
-从当前租户视图删除文件。已关联的 job 历史结果不受影响。
-
-响应:
-
-```json
-{
- "id": "file-01HXYZ123ABCDEF",
- "object": "file",
- "deleted": true
-}
-```
-
-错误:
-
-| HTTP | code |
-|------|------|
-| 404 | `file_not_found` |
-
-## 本地 Server 差异
-
-Local Parse Server 保持相同 endpoint 和对象结构,但传输实现可以简化:
-
-- `upload_url` 指向本地 server 的临时上传 endpoint,例如 `/v1/uploads/{upload_id}/content`。
-- 字节上传仍由客户端使用 `upload_method`、`upload_url` 和 `upload_headers` 完成;客户端不应硬编码 OSS 行为。
-- `GET /v1/files/{file_id}/content` 直接返回 `200 + body`,不做 CDN 重定向。
-- 本地 server 支持 parse job 的 `local` source;只有启动时开启 `--allow-local-source` 并在 `features.sources` 返回 `local` 时,用户才可跳过 upload 直接引用 server 进程权限范围内的本地路径。
-- 官方 API 必须拒绝 `local` source,返回 `400 invalid_request`。
-- 本地文件默认可以不设置保留期,File 对象的 `expires_at` 可为 `null`。
-
-
-### 自部署 Router 的源字节复用
-
-Router 把完成上传的源字节缓存在私有临时存储,并按调用方 scope 与实际 SHA-256 建立索引。
-本地命中时,`POST /v1/uploads` 复用缓存完成选定 worker 的上传并返回 `completed`,客户端跳过 PUT/complete。
-本地未命中时,即使某个 worker 已保存该 blob,Router 仍接收客户端字节以保留跨 worker 转移能力。
-`/v1/tasks` 的逐文件上传共用此缓存;清单 hash 命中时,只改变源文件名视图不需要再次发送字节。
-删除一个文件别名不会删除其他别名或正在传输的数据。完整行为见 [V1 tasks](tasks.md)。
-
-## Local Server 保留期限
-
-Local Server 的任务及解析产物默认保留 86400 秒,从任务清理完成时间计算。到期后的
-新请求返回 404;已提交任务持有输入字节租约,源文件视图在排队期间到期不影响该任务。
-Upload 的显式 `expires_after` 保持生效,已完成 Upload 到期后也不再允许读取其旧视图。
-源文件由独立文件视图和任务引用共同保护,删除一个同哈希文件不会删除其他引用使用的字节。
-`--retention-seconds 0` 关闭自动回收,但不取消 Upload 的显式期限。
-详见 [服务配置](../cli/mineru-kit-api-server.md)。
diff --git a/docs/next/api/usage-limits.md b/docs/next/api/usage-limits.md
index 18583bad..6360ceb4 100644
--- a/docs/next/api/usage-limits.md
+++ b/docs/next/api/usage-limits.md
@@ -17,11 +17,11 @@
| 并发任务 | 10+ | 10+ |
| 队列优先级 | 默认 | 优先 |
| `tier` | 当前服务可用 tier。`mineru.net/api` 当前为 `standard`;可省略或传 `null` | 当前服务可用 tier。`mineru.net/api` 当前为 `standard`;可省略或传 `null` |
-| 高级输出格式 | 不支持 `html`、`latex`、`docx` | 支持 |
-| `callback` | 不支持 | 支持 |
+| Tasks 输出格式 | default/docx/html/latex | default/docx/html/latex |
+| Tasks callback | 不支持 | 不支持 |
| 产物保留期 | 30 天 | 30 天 |
-anonymous 与 registered 的主要差异是限流、高级输出格式和 callback。请求高级输出格式或 callback 但没有 API Key 时,返回 `403 feature_requires_api_key`。
+Tasks 四种输出格式不设置额外权限门槛;服务启用认证后所有解析请求均需有效 API Key。Tasks 不接受 callback。部署方可独立配置限流。
## 限流
@@ -29,8 +29,7 @@ anonymous 与 registered 的主要差异是限流、高级输出格式和 callba
| 类别 | 范围 | anonymous | registered |
|------|------|-----------|------------|
-| `parse` | `POST /v1/parse/jobs` | 5/min | 30/min |
-| `upload` | `POST /v1/uploads`、complete、cancel | 10/min | 60/min |
+| `parse` | `POST /v1/tasks` | 5/min | 30/min |
| `chat` | `POST /v1/chat/completions`、`POST /v1/responses` | 10/min | 60/min |
| `read` | 其他 GET/DELETE endpoint | 60/min | 300/min |
@@ -141,3 +140,5 @@ Local Parse Server 的 usage 不表达计费周期,而是表达本地进程视
- `access_level` 取决于是否启用并通过 API Key 鉴权。
本地 server 可以不实现公网同等级别的限流。若实现限流,也应复用同一错误 envelope、`Retry-After` 和 `X-RateLimit-*` 响应头。
+
+当前本地服务和 Router 的 jobs_created/max_concurrent_jobs 保持字段名,计数对象是内部逐文件执行,不能作为父 Task 数量;max_files_per_job 对应清单文件数上限。
diff --git a/docs/next/api/webhooks.md b/docs/next/api/webhooks.md
index c9756725..54be6aca 100644
--- a/docs/next/api/webhooks.md
+++ b/docs/next/api/webhooks.md
@@ -1,85 +1,5 @@
-# Webhooks
+# 任务通知
-状态: Draft
-读者: API 使用者、服务端开发者、SDK 开发者
-范围: Parse Job 异步回调协议
-来源: 由根目录旧 Unified API 底稿迁移整理而来
+当前 Tasks 契约采用轮询,不接受 `callback`,也不提供 SSE 或 Webhook。客户端使用返回的 `status_url` 查询;批量任务的单个文件完成即可下载,无需等待整个批次终结。
-## 可用性
-
-Webhook 是官方远程 API 能力。本地 Local Parse Server 默认不支持 Webhook。客户端应先通过 `GET /v1/health` 的 `features.webhook` 判断是否可用。
-
-Webhook 只面向 registered 请求。anonymous 请求传入 `callback` 时,官方 API 返回 `403 feature_requires_api_key`。
-
-## 创建回调
-
-创建 parse job 时传入 `callback`:
-
-```json
-{
- "callback": {
- "url": "https://your.app/mineru-webhook",
- "secret": "abc123"
- }
-}
-```
-
-字段:
-
-| 字段 | 类型 | 必填 | 说明 |
-|------|------|:--:|------|
-| `url` | string | 是 | 接收回调的 HTTP 或 HTTPS URL,必须支持 POST。 |
-| `secret` | string | 否 | 当前 job 的签名密钥,由英文字母、数字、下划线组成,不超过 64 字符。本地实现中可省略;官方 API 可能要求必填。 |
-
-`secret` 与 job 生命周期绑定,不会在 `GET /v1/parse/jobs/{job_id}` 响应中返回。
-
-## 回调请求
-
-任务进入终态后,官方 API 向 `callback.url` 发送 POST:
-
-```http
-POST https://your.app/mineru-webhook
-Content-Type: application/json
-```
-
-```json
-{
- "checksum": "a1b2c3d4e5f6...",
- "content": "{\"event\":\"job.completed\",\"job_id\":\"job_01HXYZ...\",\"status\":\"completed\",\"occurred_at\":\"2026-05-21T08:30:48Z\"}"
-}
-```
-
-字段:
-
-| 字段 | 类型 | 说明 |
-|------|------|------|
-| `checksum` | string | `SHA256(uid + secret + content)`,用于防篡改校验。 |
-| `content` | string | JSON 字符串,接收方需要自行解析。 |
-
-`content` 解析后至少包含:
-
-| 字段 | 说明 |
-|------|------|
-| `event` | `job.completed`、`job.failed`、`job.canceled` 或 `job.partial`。 |
-| `job_id` | Job ID。 |
-| `status` | Job 终态。 |
-| `occurred_at` | 事件发生时间,ISO-8601 UTC。 |
-
-接收端应先校验 `checksum`,再解析 `content`。收到 webhook 后,仍建议调用 `GET /v1/parse/jobs/{job_id}` 拉取完整结果。
-
-## 重试
-
-接收端返回 HTTP 200 视为成功。其他状态码、连接失败或超时都视为失败。
-
-官方 API 最多重试 5 次,使用递增退避间隔。5 次均失败后不再推送。Webhook 失败不改变 job 的终态。
-
-## 本地 Server 差异
-
-Local Parse Server 不需要实现 Webhook。它应在 `health.features.webhook` 中返回 `false`。
-
-如果客户端仍传入 `callback`,本地实现应选择以下一种明确行为:
-
-- 返回 `400 invalid_request`,提示当前部署不支持 webhook。
-- 在兼容模式下忽略 `callback`,但必须在文档和日志中明确说明不会发生回调。
-
-本地 server 不能让用户误以为回调会发生。
+SDK 提供进程内 `status_callback` 与 `duration_callback`,它们不是服务端 Webhook。未来增加推送能力需要另行定义认证、重试和幂等契约。
diff --git a/docs/next/cli/mineru-kit-api-server.md b/docs/next/cli/mineru-kit-api-server.md
index 990ba92c..14336c8e 100644
--- a/docs/next/cli/mineru-kit-api-server.md
+++ b/docs/next/cli/mineru-kit-api-server.md
@@ -87,7 +87,7 @@ mineru-kit api-server --tier standard --disable-image-analysis
未传 `--tier` 时暴露 `flash`、`basic`、`standard`、`advanced`;PDF/image 请求未指定 tier 时默认 `standard`。
-OCR 模式通过每次 `POST /v1/parse/jobs` 请求的 `ocr_mode` 设置,可选 `auto`、`txt`、`ocr`,省略时为 `auto`。启动参数 `--ocr-mode` 已移除,传入会报错;Python `create_app()` 同样不再接受该参数。
+OCR 模式通过每次 `POST /v1/tasks` 请求的 `ocr_mode` 设置,可选 `auto`、`txt`、`ocr`,省略时为 `auto`。启动参数 `--ocr-mode` 已移除,传入会报错;Python `create_app()` 同样不再接受该参数。
模型默认在首次解析时懒加载。`--preload-models` 会在 Basic 或 Standard 服务启动时提前初始化所需模型或 VLM 客户端,并在失败时让能力接口返回明确错误;Flash 没有本地模型,该参数对 Flash 无操作。Doclib managed parse-server 会自动启用模型预加载。
@@ -176,8 +176,6 @@ Python 可使用 `create_app(vlm_config=VlmConfig(...))` 完整覆盖全局配
- upload-dir
- retention-seconds(默认 86400 秒,0 关闭自动回收)
- url-timeout
-- allow-local-source
-- max-inline-bytes
- allow-http-source
### 专家参数
diff --git a/docs/next/cli/mineru-kit-webui.md b/docs/next/cli/mineru-kit-webui.md
index e449150e..1a3acbd6 100644
--- a/docs/next/cli/mineru-kit-webui.md
+++ b/docs/next/cli/mineru-kit-webui.md
@@ -47,7 +47,7 @@ mineru-kit webui \
--api-key "$MINERU_API_KEY"
```
-未指定 `--api-url` 时,Gradio 会启动一个 loopback `mineru-kit api-server`。本地 server 默认使用 Standard 能力上限、开启 Flash 和 Advanced、关闭 `local` source,文件通过 V1 Uploads API 提交。
+未指定 `--api-url` 时,Gradio 会启动一个 loopback `mineru-kit api-server`。本地 server 默认使用 Standard 能力上限、开启 Flash 和 Advanced。所有文件通过 Tasks 提交 hash 清单,缓存未命中时上传,同机也不使用路径免上传。
未指定 `--server-port` 时,界面由 Gradio 从 `7860` 开始寻找空闲端口;可通过 `GRADIO_SERVER_PORT` 设置起始端口、`GRADIO_NUM_PORTS` 设置尝试数量(默认 `100`)。显式指定 `--server-port 7861` 时只尝试该端口,覆盖环境变量;该端口已被占用则启动失败。界面最终访问地址以启动输出为准,与内部 API server 的端口不同。
@@ -101,13 +101,13 @@ OCR 模式由每次 V1 解析任务决定;API Server 不再提供启动时的
Gradio 转换事件保留 `raw_page_range` 字符串接口,仍接受 V1 `page_range` 语法,例如 `1-5,8,r1`。未指定时按当前上限选择前若干页,无上限则全部;显式选区(包括 `all`)超限会返回 `page_range_invalid`,不会截断。提交端按真实页数复核,直接调用事件也不能绕过上限。该限制不改变通用 Python、V1 API 或 Doclib 的选页规则。
-Gradio 首先发现 `/v1/health` 和 `/v1/tiers`,然后通过 `MinerUApiParser` 上传文件、创建 `/v1/parse/jobs`、轮询任务并下载 ZIP 结果。解析结果保存为严格 Middle JSON 2.0。
+Gradio 首先发现 `/v1/health` 和 `/v1/tiers`,然后通过 `MinerUApiParser` 提交 hash 清单到 `/v1/tasks`、按需上传、轮询父任务并下载逐文件默认 ZIP 结果。解析结果保存为严格 Middle JSON 2.0。
左下角使用与 3.4.5 一致的两列步骤卡片,展示准备请求、检查服务、提交任务、排队、解析中、下载结果、整理输出和完成。排队状态来自本地任务槽或 V1 的 `queued` 响应,圆点每秒循环更新;不显示前方任务数。每个界面服务仍一次执行一个转换,等待中的会话也会立即显示排队动画。
收到 V1 的 `running` 后开始解析计时,由浏览器本地刷新;收到任务终态时冻结。API 路径的最终完成文案优先使用响应中已有的 `files[0].parse.duration_ms`,包含服务端解析和结果导出/打包,不包含排队、客户端下载和 WebUI 整理输出,因此即使轮询跳过 `running` 也能显示亚秒耗时。HF 等直接调用解析函数的适配层继续使用实际阶段通知之间的计时;缺少 API 耗时字段时也保留这种计时回退。完成数值按两位小数四舍五入,最低显示 `0.01 秒`。两种路径均不承诺该值等同于日志中的纯 `infer` 耗时。界面动画不增加 API 请求,任务轮询仍为每秒一次。清除或更换文件会停止当前会话的状态流并释放本地等待位置,不向 V1 发送远端取消请求。
-`MinerUApiParser.parse()` 和 `parse_async()` 提供可选的 `status_callback` 参数,接收公开的 `ApiJobStatus`(`queued`、`running`、`completed`、`partial`、`failed`、`canceled`)。回调在任务提交响应及每次轮询响应后同步调用;回调异常只记录日志。这里的 `completed` 表示服务端任务完成,解析方法仍需下载并构造 `ParseResult` 后才返回。可选的 `duration_callback` 在最终成功响应之后、开始下载之前接收既有文件耗时(浮点毫秒);缺失或非法耗时不触发回调,回调异常不影响解析结果。该值属于本次调用,不存放在共享 parser 实例或 `ParseResult` 中。
+`MinerUApiParser.parse()` 和 `parse_async()` 提供可选的 `status_callback` 参数,接收公开的 `ApiTaskStatus`(`waiting_upload`、`queued`、`running`、`completed`、`partial`、`failed`、`canceled`)。回调在任务提交响应及每次轮询响应后同步调用;回调异常只记录日志。这里的 `completed` 表示服务端任务完成,解析方法仍需下载并构造 `ParseResult` 后才返回。可选的 `duration_callback` 在最终成功响应之后、开始下载之前接收既有文件耗时(浮点毫秒);缺失或非法耗时不触发回调,回调异常不影响解析结果。该值属于本次调用,不存放在共享 parser 实例或 `ParseResult` 中。
## 结果与下载
@@ -144,3 +144,5 @@ Office 转换开始、失败及成功后都保留已挂载的源预览。“忽
PDF 和图片会生成与解析范围一致的 `origin.pdf`;布局预览使用 schema 2.0 顶层 block/bbox 生成语义 overlay。无法生成 overlay 时仍保留 origin PDF 预览。
`mineru-webui` 提供为 `mineru-kit webui` 的独立命令名,参数与行为完全相同;旧 HTTP 协议和旧 Gradio 专属参数不再提供。
+
+Gradio 使用 Tasks 能力发现,只在服务声明 `task_model_output` 时请求模型输出;下载菜单仍从保存的 Middle JSON 按需导出。
diff --git a/docs/next/decisions/0001-json-output-formats.md b/docs/next/decisions/0001-json-output-formats.md
index 163ef245..c8d1da83 100644
--- a/docs/next/decisions/0001-json-output-formats.md
+++ b/docs/next/decisions/0001-json-output-formats.md
@@ -2,7 +2,7 @@
状态: Accepted
日期: 2026-06-09
-相关文档: ../cli/mineru-parse.md, ../api/parse-jobs.md, ../sdk/parse-result.md, ../middle-json.md
+相关文档: ../cli/mineru-parse.md, ../api/tasks.md, ../sdk/parse-result.md, ../middle-json.md
## 背景
diff --git a/docs/next/decisions/0024-file-type-tier-normalization.md b/docs/next/decisions/0024-file-type-tier-normalization.md
index 3f05132c..4dd2c155 100644
--- a/docs/next/decisions/0024-file-type-tier-normalization.md
+++ b/docs/next/decisions/0024-file-type-tier-normalization.md
@@ -2,7 +2,7 @@
状态: Accepted,已实现
日期: 2026-07-09
-相关文档: ../tiers.md, ../cli/mineru-parse.md, ../cli/mineru-kit-parse.md, ../api/parse-jobs.md, 0022-doclib-file-type-tier-remote-semantics.md, 0028-csv-structured-flash-parsing.md
+相关文档: ../tiers.md, ../cli/mineru-parse.md, ../cli/mineru-kit-parse.md, ../api/tasks.md, 0022-doclib-file-type-tier-remote-semantics.md, 0028-csv-structured-flash-parsing.md
## 背景
diff --git a/docs/next/decisions/0028-csv-structured-flash-parsing.md b/docs/next/decisions/0028-csv-structured-flash-parsing.md
index 9f43d7e0..e063ed1d 100644
--- a/docs/next/decisions/0028-csv-structured-flash-parsing.md
+++ b/docs/next/decisions/0028-csv-structured-flash-parsing.md
@@ -2,7 +2,7 @@
状态: Accepted,已实现(`.tsv` 独立后缀已由 ADR-0034 实现并修订本文后续动作 1 的 CSV 部分)
日期: 2026-08-26
-相关文档: ../cli/mineru-parse.md, ../cli/mineru-kit-parse.md, ../api/parse-jobs.md, 0022-doclib-file-type-tier-remote-semantics.md, 0024-file-type-tier-normalization.md
+相关文档: ../cli/mineru-parse.md, ../cli/mineru-kit-parse.md, ../api/tasks.md, 0022-doclib-file-type-tier-remote-semantics.md, 0024-file-type-tier-normalization.md
## 背景
diff --git a/docs/next/decisions/0031-html-native-semantic-parsing.md b/docs/next/decisions/0031-html-native-semantic-parsing.md
index 7c165777..6d417421 100644
--- a/docs/next/decisions/0031-html-native-semantic-parsing.md
+++ b/docs/next/decisions/0031-html-native-semantic-parsing.md
@@ -2,7 +2,7 @@
状态: Accepted,已实现
日期: 2026-08-27
-相关文档: ../sdk/parser.md, ../cli/mineru-kit-parse.md, ../api/parse-jobs.md, 0024-file-type-tier-normalization.md
+相关文档: ../sdk/parser.md, ../cli/mineru-kit-parse.md, ../api/tasks.md, 0024-file-type-tier-normalization.md
## 背景
diff --git a/docs/next/glossary.md b/docs/next/glossary.md
index fe2c1524..7feaf6a3 100644
--- a/docs/next/glossary.md
+++ b/docs/next/glossary.md
@@ -533,7 +533,7 @@ Local Parse Server 默认不按公网 access level 做售卖或配额区分。
| `via` | 实际执行路径 |
| `source` | API 输入来源 |
| `file_id` | API file 资源 ID |
-| `job_id` | API job 资源 ID |
+| `task_id` | API 父任务 ID |
## 12. 易混用法对照
diff --git a/docs/next/implementation-plan.md b/docs/next/implementation-plan.md
index f012e0b6..c9478990 100644
--- a/docs/next/implementation-plan.md
+++ b/docs/next/implementation-plan.md
@@ -1175,7 +1175,7 @@ API-backed parser、Doclib SDK、本地 parser 的错误都能被统一捕获,
参考文档:
- [API 总览](api/README.md)
-- [Parse Jobs](api/parse-jobs.md)
+- [Tasks](api/tasks.md)
建议修改范围:
@@ -1187,8 +1187,7 @@ API-backed parser、Doclib SDK、本地 parser 的错误都能被统一捕获,
1. 测试 `GET /v1/health`。
2. 测试 `GET /v1/tiers`。
-3. 测试 `POST /v1/parse/jobs` 的最小本地路径 source。
-4. local source 默认关闭;开启 `--allow-local-source` 后允许 server 进程权限范围内的本地路径。
+3. 测试 `POST /v1/tasks` 的最小本地路径 source。
5. 未实现 webhook 时 health features 应标记 false。
6. 使用同一个 `MinerUApiParser` 调用本地 server。
diff --git a/docs/next/sdk/api-parser.md b/docs/next/sdk/api-parser.md
index 84869c11..02d75461 100644
--- a/docs/next/sdk/api-parser.md
+++ b/docs/next/sdk/api-parser.md
@@ -1,187 +1,39 @@
# API-backed Parser: `MinerUApiParser`
-状态: Draft
-读者: SDK 开发者、parse-server 开发者、集成方
-范围: 通过 v1 API 委托解析的 `DocumentParser` 实现
-来源: 由旧 SDK 底稿迁移整理而来;旧底稿已归档删除
-
-## 定位
-
-`MinerUApiParser` 是 Tool SDK 中的 API-backed parser。它实现 `DocumentParser` 接口,但不直接调用本地解析后端,而是通过 v1 Unified API 创建 upload 和 parse job。
-
-它的价值是让调用方使用同一个 parser 心智:
+`MinerUApiParser` 实现 `DocumentParser`,通过 [Tasks](../api/tasks.md) 调用 API Server、Router 或 SaaS,返回严格 Middle JSON 2.0 的 `ParseResult`。它不使用 Doclib 结果缓存,也不自行选择是否允许远端上传。
```python
from mineru.parser import MinerUApiParser
-parser = MinerUApiParser(
- api_url="https://mineru.net/api",
- api_key="...",
- tier="standard",
-)
-
-result = parser.parse("report.pdf")
+parser = MinerUApiParser(api_url="http://127.0.0.1:8000", tier="flash", ocr_mode="auto", include_images=True)
+result = parser.parse("report.pdf", page_range="1-5")
print(result.markdown())
```
-适用场景:
-
-- Python 调用方希望使用远端 `mineru.net` 解析能力。
-- doclib worker 需要调用 local parse-server 或 remote parse-server。
-- 用户希望复用 `DocumentParser` / `ParseResult` 接口,而不是直接操作 HTTP API。
-
-非目标:
-
-- 不提供 doclib 搜索、watch、配置能力。
-- 不维护本地缓存。
-- 不隐式决定是否可上传远端;调用方传入远端 `api_url`,或通过环境变量配置远端 API,即表示调用方已经允许上传到该 API。
-
-## 构造参数
-
-目标公开签名:
-
-```python
-class MinerUApiParser(DocumentParser):
- def __init__(
- self,
- *,
- api_url: str | None = None,
- api_key: str | None = None,
- tier: str | None = None,
- ocr_mode: Literal["auto", "txt", "ocr"] | None = None,
- include_images: bool = False,
- include_model_output: bool = False,
- ) -> None: ...
-```
-
-字段:
-
-| 参数 | 说明 |
-|------|------|
-| `api_url` | v1 API base URL,如 `https://mineru.net/api` 或 `http://127.0.0.1:16580`。省略时读取 `MINERU_API_URL`,再退回默认官方 API。 |
-| `api_key` | Bearer token。省略时读取 `MINERU_API_KEY`;Local Parse Server 未启用鉴权时可为空。 |
-| `tier` | `flash`、`basic`、`standard`、`advanced` 或 `None`。`None` 表示 SDK 在 HTTP 请求中省略 `tier`,让 v1 API 使用默认选择策略;非 PDF/image 文件按 API Server 批量规则归一,见 [ADR-0024](../decisions/0024-file-type-tier-normalization.md)。 |
-| `include_images` | 是否从 zip 产物读取图片 sidecar,并挂载到 `ParseResult` 图片缓存。 |
-| `ocr_mode` | 请求级 OCR 模式:`auto` 自动判断、`txt` 使用文本层、`ocr` 强制 OCR。默认 `None`,在 HTTP 请求中省略字段,由服务端按 `auto` 处理;显式设置时同步与异步解析均传入任务顶层。 |
-| `include_model_output` | 是否请求并保留模型原始输出;开启时通过 zip 产物读取。 |
-
-## 行为流程
-
-`parse(path)` 的流程:
-
-1. 检查文件存在。
-2. 判断 `api_url` 是否是本地地址。
-3. 本地地址先读取 `GET /v1/health`;只有 `features.sources` 包含 `local` 时才使用 `local` source。
-4. 非本地地址,或本地 server 未允许 `local` source 时,使用 Uploads API: create upload -> PUT upload_url -> complete upload,再以 `file_id` source 创建解析任务。
-5. 调用 `POST /v1/parse/jobs`;默认请求 `output_formats:["middle_json"]`,需要模型输出或图片缓存时请求 `["zip"]`。
-6. 如果 job 未完成,则轮询 `GET /v1/parse/jobs/{job_id}`。
-7. 普通模式下载 `middle_json` output file;zip 模式下载 `zip` output file。
-8. zip 模式从 zip 中读取 middle JSON,并在 `include_images=True` 时读取 middle JSON 引用的图片 sidecar。
-9. 将 middle JSON 转为 `ParseResult`。
-
-### 传输重试
-
-客户端对可安全重放的传输操作执行有限重试:
-
-- health、upload 状态、job 轮询和产物下载使用 `GET`,可以重试。
-- 向 `upload_url` 写入相同文件字节使用 `PUT`,可以重试。
-- upload complete 的响应丢失后,客户端先查询 upload 状态;已经完成则读取其中的 File,仍为 pending 才再次 complete。
-
-`POST /v1/parse/jobs` 在 API 提供幂等键契约前不会自动重试。该请求的响应丢失时,服务端可能已经创建任务;直接重放会产生重复任务和费用。
-
-重试耗尽后,doclib 远端解析分别使用 `remote_timeout` 或 `remote_unreachable`,不把客户端传输错误归类为 `parse_failed`。远端 job 自身返回的结构化 `parse_failed` 保持原语义,客户端不根据错误消息文本猜测错误来源。
-
-本地 source:
-
-Local Parse Server 必须通过 `--allow-local-source` 开启,并在 `features.sources` 返回 `local`,SDK 才会使用该形式。
-
-```json
-{
- "source": {
- "type": "local",
- "path": "/absolute/path/report.pdf"
- }
-}
-```
-
-远端 source:
-
-```json
-{
- "source": {
- "type": "file_id",
- "file_id": "file-..."
- }
-}
-```
-
-## Tier 映射
-
-`MinerUApiParser` 面向 v1 API 时只使用 tier 术语:
-
-| SDK tier | API tier |
-|----------|----------|
-| `None` | 省略 |
-| `flash` | `flash` |
-| `basic` | `basic` |
-| `standard` | `standard` |
-| `advanced` | `advanced` |
-
-它不接受也不保存 backend 参数。backend 是本地 parser 层的高级实现概念,不应出现在 API-backed parser 的公开构造参数、实例属性或请求 payload 中。
-
-## 输出格式
-
-默认只请求 `middle_json`。如果 `include_images=True` 或 `include_model_output=True`,则请求 `zip`,并从 zip 中恢复 `ParseResult` 所需的 middle JSON。API 不支持把 `images` 作为独立 `output_formats` 请求值。
-
-目标可选参数:
+构造参数保留 `api_url/api_key/tier/ocr_mode/include_images/include_model_output`。
+地址默认读取 `MINERU_API_URL`,否则使用 `https://mineru.net/api`;密钥默认读取 `MINERU_API_KEY`。
+`tier=None` 和 `ocr_mode=None` 省略对应请求字段;服务默认 OCR 模式是 `auto`。
+`parse()` 和 `parse_async()` 均接收 `page_range/status_callback/duration_callback`。
-| 参数 | 说明 |
-|------|------|
-| `include_images` | 请求 `zip` 并从其中读取图片 sidecar 到 `ParseResult` 图片缓存。 |
-| `include_model_output` | 请求 `zip` 并尝试保留模型输出。 |
+## 请求与结果
-## 错误映射
+1. 校验文件及页范围,计算大小和 SHA-256,包括同机 HTTP 请求。
+2. 向 `/v1/tasks` 提交一个 `client_file_id="file"` 的 manifest,固定请求 `output_formats=["default"]`。
+3. 缓存命中不上传;未命中按服务返回的 PUT 链接和 headers 流式上传。无需完成确认。
+4. 按返回的同源 `status_url` 每秒查询,轮询预算 3600 次;用 `client_file_id` 关联结果。
+5. 从该文件的 `result_url` 流式下载并校验 ZIP,读取 `middle_json.json`。
+6. `include_images=True` 恢复图片素材;`include_model_output=True` 请求并读取可用的 `model_output.json`。没有原始模型输出时不生成占位数据。
-API error envelope 应映射为 SDK exception:
+HTTP 输入只有上传和 URL;SDK 的路径参数始终代表客户端文件,不再传递服务端路径。上传 HTML 不自动附带同目录资源,应使用自包含文件或 Tasks URL 输入;直接本地 Python parser 仍可读取本地路径。
-| API code | SDK 行为 |
-|----------|----------|
-| `invalid_api_key` | 抛出认证错误。 |
-| `file_too_large` | 抛出用户可修复错误。 |
-| `quality_tier_unavailable` | 抛出引擎不可用错误。 |
-| `rate_limit_exceeded` | 抛出可重试错误,并保留 `Retry-After`。 |
-| `service_unavailable` | 抛出可重试错误。 |
-| `remote_timeout` | 远端传输超时,有限重试后仍未恢复。 |
-| `remote_unreachable` | 远端连接或协议错误,有限重试后仍未恢复。 |
+## 状态和错误
-当前实现有内部 `_V1APIError`。目标公开契约应复用 `mineru.errors.MineruError` 或统一 SDK exception hierarchy。
+`ApiTaskStatus` 包含 `waiting_upload/queued/running/completed/partial/failed/canceled`;`ApiJobStatus` 是保留的 Python 类型别名。
+状态回调在创建和查询后触发,回调异常只记录日志。服务端完成表示开始下载,不等于 SDK 已经返回。
+`duration_callback` 在开始下载前接收服务端单文件 `parse.duration_ms`,包含解析和导出;缺失或非法值不触发回调。
-## 隐私约束
+创建请求没有幂等保证,不自动重试。上传响应丢失后先查状态:已 ready 则继续,仍 pending 才重传。查询、传输和下载有界重试;下载签名失效后重新查询任务。密钥仅附加到 API 同源请求,每次重定向重新判断来源。
-`MinerUApiParser` 是显式远端/本地 API client。它不会自己决定“是否可以 remote”。调用方只要传入 `https://mineru.net/api`,就表示允许上传到该 API。
-
-doclib SDK 或 CLI 在调用它之前必须完成隐私策略判断:
-
-- 用户未显式 remote 时,不应构造指向 `mineru.net` 的 `MinerUApiParser`。
-- 用户指定 local parse-server 时,优先使用本地 URL。
-- API-backed parser 只负责执行请求,不负责产品策略。
-
-## 与 `mineru.parser.api_server`
-
-`mineru.parser.api_server` 是 `MinerUApiParser` 可以连接的服务端实现之一。它提供 v1 Unified API 的 runtime 和 Pydantic contract。
-
-文档关系:
-
-- HTTP 规格见 [Unified API](../api.md)。
-- `MinerUApiParser` 是该 HTTP 规格的 Python parser 封装。
-- `api_server` 是该 HTTP 规格的本地无状态 server 实现。
-
-## 公开入口
-
-`MinerUApiParser` 保持现有命名和位置,公开入口为:
-
-```python
-from mineru.parser import MinerUApiParser
-```
+结构化业务错误保留 code/message/param。传输失败保留 `_APITransportError` 阶段及超时信息,Doclib 继续区分 `remote_timeout`、`remote_unreachable` 与业务 `parse_failed`。
-不新增 `mineru.parser.remote` 迁移路径。低层 `V1ApiClient` 是否公开仍集中维护在 [开放问题清单](../open-questions.md)。API-backed parser 的异常应收敛到统一 `MineruError` 体系。
+旧协议客户端不能连接移除旧接口后的服务,更新 worker、Router 和 SDK 后再切换调用方;参见 [迁移说明](../api/changes.md)。
diff --git a/docs/next/sdk/migration.md b/docs/next/sdk/migration.md
index 14076481..7ee6f95e 100644
--- a/docs/next/sdk/migration.md
+++ b/docs/next/sdk/migration.md
@@ -24,7 +24,7 @@ SDK 迁移的目标不是重写现有 parser,而是稳定已经成形的边界
| `DocumentParser` | 已有 sync/async/batch/context manager 接口。 |
| `ParseResult` | 已有输出方法,`from_dict()` / `from_json()` 已实现;仍需稳定 envelope 兼容边界。 |
| Tier 参数 | `parse()` 已以 `tier` 为主参数(默认 `standard`),并接受 `ocr_mode` / `image_analysis` / `page_range`;`backend` 专家覆盖保留在 `mineru-kit parse` CLI 层。 |
-| API-backed parser | 已有 `MinerUApiParser`,使用 v1 uploads/jobs/files。 |
+| API-backed parser | 已有 `MinerUApiParser`,使用 v1 Tasks 和逐文件结果链接。 |
| parse-server runtime | 已有 `mineru.parser.api_server`。 |
| Doclib SDK | 已有 `DoclibClient`,继承 `DoclibInterface` 并返回 typed Pydantic response。 |
| 错误模型 | 已有 `mineru.errors`;仍需确认 SDK exception 是否暴露 `user_action` 等扩展字段。 |
@@ -110,7 +110,7 @@ SDK 迁移的目标不是重写现有 parser,而是稳定已经成形的边界
- Import 不加载 heavy backend。
- `parse()` 后缀分派。
- `ParseResult` round-trip。
-- `MinerUApiParser` upload/job/files mock。
+- `MinerUApiParser` Tasks/hash/upload/result_url mock。
- `DoclibClient` UDS error 映射。
- 默认选择不回退 `flash`。
diff --git a/docs/next/workflows.md b/docs/next/workflows.md
index b5d310ae..1b3db1f5 100644
--- a/docs/next/workflows.md
+++ b/docs/next/workflows.md
@@ -82,9 +82,9 @@
│
└─ v1 API 调用
-> Official API 或 Local Parse Server
- -> upload/file/job
+ -> Tasks manifest / hash preflight / optional upload
-> parse-server
- -> job result / requested formats
+ -> per-file result_url / requested formats
```
## 5. 流程 A: Watch 自动发现与索引
@@ -396,13 +396,13 @@ marker 不应依赖自然语言提示。具体 marker 格式可在 CLI 或 Agent
### 8.2 Official API 主线
-1. 客户端创建 upload 或提供已有 `file_id`。
-2. 客户端创建 parse job。
+1. 客户端提交 Tasks 清单和可选 hash,得到父任务及缺失文件上传链接。
+2. 客户端仅上传缓存未命中的文件;各文件就绪即独立排队。
3. 请求可以携带 `tier`;省略或传 JSON `null` 时,PDF/image 使用默认选择策略,OFD/EPUB/Office/HTML/MHTML/CSV/TSV 按批量规则归一为 `flash`;其它 text 不作为解析输入。
4. API service 做鉴权、配额和输入校验。
5. 调度到可用解析能力。
6. 解析服务生成 Middle JSON 和请求的输出格式。
-7. job 进入终态。
+7. 父 Task 聚合逐文件终态,成功文件通过 result_url 下载。
8. 客户端通过轮询或 webhook 获取结果。
### 8.3 Local Parse Server 差异
@@ -412,7 +412,7 @@ Local Parse Server 复用同一套客户端协议,但可以简化:
- 鉴权默认可关闭。
- Webhook 可以不实现。
- 上传可以使用本地临时存储。
-- `local` source 只有在 `features.sources` 包含 `local` 时可用;开启后可以引用 server 进程权限范围内的本地路径。
+- 同机 HTTP 请求也使用 hash 预检和按需上传,不传服务端路径。
- 文件下载可以直接返回 body。
### 8.4 与 doclib 的关系
diff --git a/docs/plans/2026-07-10-remote-transport-retry-design.md b/docs/plans/2026-07-10-remote-transport-retry-design.md
index 02ff28f9..1a919bfd 100644
--- a/docs/plans/2026-07-10-remote-transport-retry-design.md
+++ b/docs/plans/2026-07-10-remote-transport-retry-design.md
@@ -13,7 +13,7 @@ The client retries requests only when replay cannot create duplicate parse jobs:
- Retry `GET` requests for health, upload state, job polling, and output downloads.
- Retry the upload content `PUT` with the same bytes and headers.
- After a transport failure from upload completion, query the upload. Return its file when completed, or retry completion while it remains pending.
-- Do not automatically retry `POST /v1/parse/jobs`. The API does not currently provide an idempotency contract, so a lost response may already represent a created and billable job.
+- Do not automatically retry `POST /v1/tasks`. The API does not currently provide an idempotency contract, so a lost response may already represent a created and billable job.
Retries use a small bounded exponential delay. HTTP application errors are not transport retries; they continue through the existing structured response handling.
@@ -31,11 +31,10 @@ If a remote job itself returns `parse_failed`, the client preserves that structu
## Implementation Shape
-`mineru/parser/api_client.py` owns matching synchronous and asynchronous request helpers. The helpers apply the retry policy and raise a typed internal transport error after exhaustion. Upload completion adds state recovery through `GET /v1/uploads/{upload_id}`.
`ParseService` maps typed transport failures according to the selected API target and records the resulting error code on the parse task. Existing `_V1APIError` handling remains responsible for structured API and job errors.
-Job submission idempotency is a separate API change. Once the server supports an idempotency key, the client can safely retry `POST /v1/parse/jobs` with a stable key.
+Job submission idempotency is a separate API change. Once the server supports an idempotency key, the client can safely retry `POST /v1/tasks` with a stable key.
## Tests
diff --git a/docs/zh/reference/migration_4.md b/docs/zh/reference/migration_4.md
index 02e1c27b..ac917e09 100644
--- a/docs/zh/reference/migration_4.md
+++ b/docs/zh/reference/migration_4.md
@@ -60,7 +60,7 @@ model:
## API 与 WebUI
-V1 API 使用 `/v1/health`、`/v1/tiers`、`/v1/uploads`、`/v1/parse/jobs` 和 `/v1/files`;旧 `/file_parse`、`/tasks` 不再提供。SDK 客户端和 WebUI 必须连接 V1 服务。文档解析 API 与 OpenAI 兼容 VLM 服务是不同接口,地址不可互换。 API server 和 Router 提供 `/v1/tasks` 表单清单、逐文件上传和 ZIP 下载链接;本地另支持单文件 multipart。客户端轮询任务并使用返回链接下载,不再提供同步包装或结果正文便捷路由。详见[HTTP API](../usage/http_api.md)。
+异步解析统一使用 `/v1/tasks` 的 JSON 清单、按需上传和逐文件结果链接。SDK 与 WebUI 同步迁移,Files 下载及独立同步包装服务保留。同机也先做 hash 预检;详见 [HTTP API](../usage/http_api.md)。
原 Gradio 客户端也需按新版事件接口调整,不能仅替换命令名称。第三方插件的旧适配文档保留历史说明,不表示已支持 V1 API。
diff --git a/docs/zh/usage/http_api.md b/docs/zh/usage/http_api.md
index fe3b2ffd..78d606ff 100644
--- a/docs/zh/usage/http_api.md
+++ b/docs/zh/usage/http_api.md
@@ -1,109 +1,44 @@
# V1 HTTP API
-推荐 HTTP 客户端使用 tasks 的表单清单与下载链接协议。本页先介绍该流程,再给出 Python SDK、WebUI 使用的既有 Uploads/Jobs 资源工作流。Python 客户端见 [Python SDK](sdk_api.md)。
-
-先启动本地服务;使用 SaaS 时,将客户端指向实现同一 tasks 契约的服务地址:
+Tasks 是唯一异步解析入口。API Server、Router、SDK 和 WebUI 统一提交 JSON 清单、按需上传、查询状态并下载逐文件结果。Python 调用见 [Python SDK](sdk_api.md),完整字段见 [Tasks 契约](../../next/api/tasks.md)。
```bash
mineru-kit api-server --host 127.0.0.1 --port 8000 --tier standard
+MINERU_API_URL=http://127.0.0.1:8000 bash scripts/http_task_example.sh a.pdf b.pdf
+OUTPUT_FORMATS=default,docx MINERU_API_URL=http://127.0.0.1:8000 bash scripts/http_api_example.sh a.pdf
```
-## tasks:清单、逐文件上传与 ZIP 下载
-
-API server、Router 和采用同一契约的 SaaS 使用相同客户端流程:提交清单、上传缺失文件、轮询状态、下载返回的 `result_url`。
-SaaS 可以返回 OSS 直链,本地返回 Files 下载地址;客户端原样使用链接,不拼接下载路径,也不要求云端实现本地 Files API。
-完整契约见 [V1 tasks](../../next/api/tasks.md)。
-
-```bash
-# 单文件或多个文件:默认先预检,只上传未命中缓存的项。
-MINERU_API_URL=http://127.0.0.1:8000 \
- bash scripts/http_task_example.sh a.pdf b.pdf
-
-# hash 可选。
-SEND_HASH=0 MINERU_API_URL=http://127.0.0.1:8000 \
- bash scripts/http_task_example.sh a.pdf b.pdf
-
-# 使用首次提交的文件列表和顺序继续原任务。
-TASK_ID=job_... MINERU_API_URL=http://127.0.0.1:8000 \
- bash scripts/http_task_example.sh a.pdf b.pdf
-
-# 本地仅保留单文件 multipart 快捷提交,仍返回异步任务。
-curl --fail-with-body 'http://127.0.0.1:8000/v1/tasks' \
- -F 'tier=flash' -F 'ocr_mode=txt' -F '[email protected]'
-```
-
-`POST /v1/tasks` 的参数全部放在表单中。`manifest` 是 JSON 数组形式的文本字段,每项包含唯一 `client_file_id`、`filename`、实际 `bytes`,可选 `sha256sum` 与逐文件 `page_range`。
-多文件任务只能提交清单,不能在同一个请求中附带文件字节。单文件直传需要 hash 时,可附带单条 manifest。
-解析选项仍使用 `tier`、`ocr_mode`、`page_range` 和重复的 `output_formats`,ZIP 始终生成。
-POST 查询参数返回 `400`,JSON 和原始字节创建请求返回 `415`。独立 Uploads/Jobs 的现有 JSON 协议不受影响。
-
-创建立即返回稳定的 `task_id` 和 `status_url`;`uploads` 只包含需要上传的项,每项给出上传方法、地址、请求头与有效期。
-所有文件校验通过后自动开始,客户端不再发送完成确认。未提供 hash 时正常上传,服务端仍计算 hash 并去重保存。
-文件完整上传后独立标记为 `ready`;后续文件中断只需重传失败文件,已经成功的文件保留。当前支持文件级重试,不支持单文件分块续传。
-
-客户端只轮询 `status_url`。状态为 `waiting_upload/queued/running/completed/partial/failed/canceled`。
-最终 ZIP 已保存后才返回 `completed/partial` 和 `result_url`;状态接口不内联解析正文,下载不经过 SaaS API 网关。
-批量 ZIP 含编号目录及逐文件 manifest。打包失败是任务失败,不能返回不可用的下载链接。
-
-完整客户端由 [`scripts/http_task_example.sh`](https://github.com/opendatalab/MinerU/blob/master/scripts/http_task_example.sh) 与同目录 Python 文件维护,只需 Python 3.10+ 标准库。
-`UPLOAD_MODE=direct` 只支持一个文件;`RESULT_FILE` 默认 `result.zip`。`MAX_POLLS/POLL_INTERVAL` 控制有界轮询,`CONNECT_TIMEOUT/REQUEST_TIMEOUT` 控制传输预算。
-每个文件最多尝试三次,创建任务不自动重试;响应丢失先查询文件状态,链接过期则查询原任务刷新。
-API Key 只附加到同源请求,OSS、外部下载和跨源重定向不携带该 Key。
-退出码为 `0`(成功)、`2`(部分成功且已保存 ZIP)、`1`(失败或取消)、`124`(等待预算耗尽,保留任务)。
-
-本地不设固定的上传字节数或 PDF 页数上限,每任务最多 100 文件。等待上传不占解析名额,也不预占 Router worker。
-Router 按负载选择 worker,并在向上游提交前预占;缓存命中不固定到繁忙 worker。批量文件仍交给同一个 worker,最终 ZIP 在 worker 生成,Router 只映射资源并流式下载。
-
-## 既有 Uploads / Jobs 资源示例
-
-需要操作独立 Upload、File 与 Job 资源时,继续使用以下脚本;其协议和退出码保持不变,与上面的 tasks 脚本分别维护:
-
-> [`scripts/http_api_example.sh`](https://github.com/opendatalab/MinerU/blob/master/scripts/http_api_example.sh)
-
-```bash
-export MINERU_API_URL=http://127.0.0.1:8000
-export MINERU_API_KEY=secret-key # 匿名本地访问时省略
-./scripts/http_api_example.sh document.pdf
-```
-
-脚本行为由 `tests/unittest/test_http_api_example_script.py` 在真实 Flash 服务和脚本化 mock 服务上验证,无需下载模型。退出码面向自动化保持稳定:
-
-| 码 | 含义 |
-| --- | --- |
-| `0` | 任务 completed,全部请求产物已保存 |
-| `1` | 脚本、传输、协议、下载或本地写入失败 |
-| `2` | 任务 partial:已先保存成功文件的产物 |
-| `3` | 任务 failed |
-| `4` | 任务 canceled |
-| `124` | 轮询预算耗尽;打印 `job_id`,可继续轮询 |
+## 文件清单与上传
-## 资源级接口的鉴权与字节上传
+向 `POST /v1/tasks` 提交 `application/json`:`manifest` 每项包含唯一 `client_file_id`、逻辑 `filename`、上传文件的 `bytes` 和可选 `sha256sum/page_range`。也可以提供 URL,此时 bytes 可省略。解析选项为 `tier/ocr_mode/page_range/output_formats/include_model_output`。
-服务以 `--api-key` 启动时,`/v1/*` 请求携带 `Authorization: Bearer $MINERU_API_KEY`;匿名本地访问可省略该头。
+创建返回稳定的 `task_id/status_url` 和待上传项的链接。客户端只上传缺失文件,使用响应指定的 PUT 地址及请求头,无需 complete。每个文件就绪即独立排队,其他文件等待上传不会阻塞它。
+同机也使用相同流程。SDK 总是计算 hash;已完成上传的有效缓存可以免传。并发首次请求允许重复传输,但源字节去重。缓存默认保留 24 小时、每 5 分钟扫描,正在执行及传输的引用受保护。
-字节上传原样使用 `POST /v1/uploads` 返回的方法、URL 和请求头:
+上传链接固定有效一小时,查询不会续期。截止前已受理上传可继续完成。中断后先查询,ready 文件跳过,其余在有效期内从头重传;过期项需创建新任务。
-- 自部署服务返回同源 URL(如 `/v1/uploads/{id}/content`)。同源指 scheme + host + 有效端口,由客户端判断而非假设;同源上传在服务要求鉴权时必须携带 API 鉴权。
-- 官方 API 返回自带授权的预签名对象存储 URL。**不要**向不同源的 URL 附加 MinerU API Key——那会泄露凭据。服务返回的上传头(`upload_headers`)始终原样保留。
+## 状态、列表与结果
-示例脚本实现了与 Python SDK(`api_client._same_origin_upload_headers`)相同的规则:相对 URL 先按 API 基地址解析,再比较 scheme + host + 有效端口,仅对同源上传附加 MinerU Key。相对 URL 解析后即同源;同一主机的不同端口按不同源处理。
+使用返回的 `status_url` 轮询。父任务状态为 `waiting_upload/queued/running/completed/partial/failed/canceled`;通过 `client_file_id` 对应输入。
+每个成功文件在 `files[].result_url` 提供唯一下载地址,即使父任务尚未完成也可下载。顶层没有 result_url,不生成跨输入文件的总 ZIP。
-下载走 `GET /v1/files/{file_id}/content`,可能返回 `302` 重定向。示例跟随重定向但不使用 `--location-trusted`,凭据不会被重新发送到不同源(包括同主机不同端口;要求 curl >= 7.83.0)。
+| output_formats | 内容 |
+|---|---|
+| default | ZIP:Markdown、Middle JSON、Structured Content、images/ |
+| docx | 图片内嵌的 DOCX |
+| html | 图片内嵌的 HTML |
+| latex | TEX 与 images/ 的 ZIP |
+| 多种 | 对应文件平铺在一个 ZIP 中 |
-## 资源级脚本的请求周期细节
+`include_model_output=true` 要求包含 default,并附加可用的 model_output.json;默认关闭。
+`GET /v1/tasks` 支持状态筛选和游标分页;`DELETE /v1/tasks/{task_id}` 取消未完成文件。Files 的列表、元数据、结果下载和删除保留。
-- **上传生命周期。** 创建上传时传 `sha256sum` 可启用完整性校验和秒传;命中已存在文件时响应直接为 `completed` 并内嵌 `file` 对象,跳过字节上传和完成两步。
-- **HTTP 200 不等于合法响应。** 脚本在每一步校验 JSON 结构:ID 必须是非空字符串、状态必须属于支持集合;否则立即失败,而不是带着 null ID 进入轮询。
-- **有界轮询。** `MAX_POLLS` 限制轮询请求次数(每次还包括请求超时和轮询间隔);预算耗尽时脚本退出 `124` 并打印 `job_id`——任务未被取消,可直接继续轮询。
-- **终态。** `completed`、`partial`、`failed`、`canceled` 都是终态。`partial` 表示部分文件成功:脚本先保存成功文件的产物、打印含错误的逐文件结果,然后才退出 `2`;`failed` 退出 `3`,`canceled` 退出 `4`。
-- **原子下载。** 产物先写 `*.part`,成功后才移到最终文件名;`completed` 任务若有产物下载失败或缺失,不会退出 `0`。
-- **产物。** 所有产物都是 `purpose:"parse_output"` 的文件;内容一律通过 `GET /v1/files/{file_id}/content`、用 `files[].output_files` 下的 id 获取。以 `GET /v1/health`(`features.sources`、`features.output_formats`)为当前部署实际能力的依据;渲染器支持的格式不代表 API 暴露。
+客户端原样使用结果链接:本地/Router 返回 Files 地址,SaaS 可返回对象存储签名地址。只为 API 同源请求附加 API Key,每次重定向重新判断;签名失效可重新查询结果链接。
-## 生产部署注意事项
+## 示例客户端与迁移
-- 谨慎绑定 `--host`。默认 `127.0.0.1` 只监听回环;使用 `0.0.0.0` 暴露到网络时,应配合 `--api-key` 或带鉴权的反向代理。
-- 存储与重启边界:文件字节保存在 `--upload-dir` 下(未配置时为临时目录,正常关闭即删除),但上传/文件/任务的资源索引是进程内状态。即使保留了目录,服务重启后旧的 `upload_id`/`file_id`/任务 ID 也不保证可用。该 API 不是可恢复的持久化任务服务;持久化阅读、索引和缓存需求使用文档库——但文档库不是这套上传/任务接口的透明持久化替代品。
-- `--concurrency` 限制并发解析任务数;超出后表现为 `queued` 时间变长,而不是失败。
-- `GET /v1/usage` 报告用量。用 `/v1/health` 和 `/docs`(启用时)的 OpenAPI 文档区分服务端故障与客户端错误;重试 `failed` 任务前先查看服务端日志。
+两个 shell 示例入口都复用标准库 Python Tasks 客户端。`RESULT_DIR` 指定逐文件目录;`OUTPUT_FORMATS` 默认为 default,`INCLUDE_MODEL_OUTPUT=1` 请求模型输出,`SEND_HASH=0` 可关闭预检 hash。
+使用相同文件顺序设置 `TASK_ID` 可继续原任务。创建请求不自动重试;`MAX_POLLS/POLL_INTERVAL` 限制轮询,`CONNECT_TIMEOUT/REQUEST_TIMEOUT` 限制单次传输。
+退出码为 0(全部成功)、2(部分成功)、1(失败或取消)、124(轮询耗尽)。已下载产物保留。
-新的便捷工作流统一使用 `/v1/tasks` 和返回链接,不提供同步等待或结果正文路由。根路径旧接口不提供,旧客户端迁移见[迁移指南](../reference/migration_4.md)。
+旧独立上传和 Jobs 接口已移除,客户端与服务端需协调升级,详见 [迁移说明](../../next/api/changes.md)。需要同步单文件返回时使用独立的 [file_parse 包装服务](../../next/api/file-parse.md)。
diff --git a/mineru/doclib/background/parse_server_health.py b/mineru/doclib/background/parse_server_health.py
index aeafde2c..81d9a6a7 100644
--- a/mineru/doclib/background/parse_server_health.py
+++ b/mineru/doclib/background/parse_server_health.py
@@ -128,7 +128,6 @@ def api_server_args_for_tier(tier: DeploymentTier, *, host: str, port: int) -> l
host,
"--port",
str(port),
- "--allow-local-source",
"--no-flash",
"--preload-models",
"--log-level",
diff --git a/mineru/kit/commands/api_server.py b/mineru/kit/commands/api_server.py
index 11732c0d..2067d31b 100644
--- a/mineru/kit/commands/api_server.py
+++ b/mineru/kit/commands/api_server.py
@@ -50,8 +50,6 @@ def api_server_cmd(
),
concurrency: int = typer.Option(1, "--concurrency", help=t("Maximum concurrent parse jobs")),
url_timeout: int = typer.Option(60, "--url-timeout", help=t("Timeout for URL source downloads")),
- allow_local_source: bool = typer.Option(False, "--allow-local-source", help=t("Allow local source paths")),
- max_inline_bytes: int = typer.Option(1024 * 1024, "--max-inline-bytes", help=t("Maximum decoded bytes for inline sources")),
allow_http_source: bool = typer.Option(False, "--allow-http-source", help=t("Allow URL sources to use plain HTTP")),
disable_image_analysis: bool = typer.Option(False, "--disable-image-analysis", help=t("Disable image analysis")),
preload_models: bool = typer.Option(
@@ -105,13 +103,10 @@ def api_server_cmd(
str(concurrency),
"--url-timeout",
str(url_timeout),
- "--max-inline-bytes",
- str(max_inline_bytes),
*tier_args,
*vlm_args,
*(["--no-flash"] if no_flash else []),
*(["--no-advanced"] if no_advanced else []),
- *(["--allow-local-source"] if allow_local_source else []),
*(["--allow-http-source"] if allow_http_source else []),
*(["--upload-dir", upload_dir] if upload_dir else []),
*(["--disable-image-analysis"] if disable_image_analysis else []),
diff --git a/mineru/kit/commands/parse.py b/mineru/kit/commands/parse.py
index 9a5e63bc..87b0ae78 100644
--- a/mineru/kit/commands/parse.py
+++ b/mineru/kit/commands/parse.py
@@ -82,7 +82,9 @@ def parse_cmd(
exit_with_message("invalid_request", str(exc))
if api_url:
- parser = MinerUApiParser(api_url=api_url, api_key=api_key, tier=parse_tier, include_images=True)
+ parser = MinerUApiParser(
+ api_url=api_url, api_key=api_key, tier=parse_tier, ocr_mode=parse_ocr_mode, include_images=True
+ )
parse_one = partial(parser.parse, page_range=pages or "")
else:
is_batch = has_directory_input or len(paths) > 1
diff --git a/mineru/kit/file_parse/app.py b/mineru/kit/file_parse/app.py
index b5e0784c..425f2fb1 100644
--- a/mineru/kit/file_parse/app.py
+++ b/mineru/kit/file_parse/app.py
@@ -121,7 +121,7 @@ async def _receive_file(request: Request, service: FileParseService) -> tuple[Re
raise GatewayError("invalid_request", "Parameters must be supplied as multipart fields", 400)
if request.headers.get("content-type", "").split(";", 1)[0].strip().lower() != "multipart/form-data":
raise GatewayError("unsupported_media_type", "Expected multipart/form-data", 415)
- allowed = {"file", "request_id", "wait_timeout", "tier", "ocr_mode", "page_range", "output_formats"}
+ allowed = {"file", "request_id", "wait_timeout", "tier", "ocr_mode", "page_range", "output_formats", "include_model_output"}
source: Path | None = None
accepted = False
try:
@@ -146,7 +146,7 @@ async def _receive_file(request: Request, service: FileParseService) -> tuple[Re
raise ValueError("wait_timeout out of range")
options = ParseOptions.model_validate(
{
- **{key: form[key] for key in ("tier", "ocr_mode", "page_range") if key in form},
+ **{key: form[key] for key in ("tier", "ocr_mode", "page_range", "include_model_output") if key in form},
"output_formats": form.getlist("output_formats") if "output_formats" in form else ["default"],
}
)
diff --git a/mineru/kit/file_parse/models.py b/mineru/kit/file_parse/models.py
index ec2aef0a..c19fab84 100644
--- a/mineru/kit/file_parse/models.py
+++ b/mineru/kit/file_parse/models.py
@@ -47,10 +47,13 @@ class ParseOptions(BaseModel):
ocr_mode: Literal["auto", "txt", "ocr"] = "auto"
page_range: str = ""
output_formats: list[TaskOutputFormat] = Field(default_factory=lambda: ["default"], min_length=1)
+ include_model_output: bool = False
@model_validator(mode="after")
def normalize(self) -> ParseOptions:
"""按首次出现去重格式,并统一整本页范围的等价写法。"""
+ if self.include_model_output and "default" not in self.output_formats:
+ raise ValueError("include_model_output requires default output format")
self.output_formats = list(dict.fromkeys(self.output_formats))
self.page_range = self.page_range.strip()
if self.page_range.lower() == "all":
@@ -129,7 +132,11 @@ class Record:
def fingerprint(filename: str, size: int, sha256sum: str, options: ParseOptions) -> str:
"""用文件内容、逻辑名称与归一化参数区分同 ID 的冲突提交。"""
- value = {"filename": filename, "size": size, "sha256sum": sha256sum, "options": options.model_dump()}
+ normalized = options.model_dump()
+ # 缺省关闭不改变此前已持久化请求的身份;显式启用才引入新的结果选项。
+ if not normalized["include_model_output"]:
+ normalized.pop("include_model_output")
+ value = {"filename": filename, "size": size, "sha256sum": sha256sum, "options": normalized}
return hashlib.sha256(json.dumps(value, ensure_ascii=False, sort_keys=True).encode()).hexdigest()
diff --git a/mineru/kit/gradio/client.py b/mineru/kit/gradio/client.py
index 1195a227..b8dbcf2d 100644
--- a/mineru/kit/gradio/client.py
+++ b/mineru/kit/gradio/client.py
@@ -16,7 +16,7 @@ from typing import Any, Literal, cast
import httpx
-from ...parser.api_client import ApiJobStatus, MinerUApiParser, should_trust_env_for_url
+from ...parser.api_client import ApiTaskStatus, MinerUApiParser, should_trust_env_for_url
from ...parser.base import ParseResult
from ...parser.page_range import normalize_page_range_input
from ...parser.process_control import ManagedProcessControl
@@ -31,6 +31,7 @@ from .status import (
STATUS_PROCESSING_OUTPUT,
STATUS_QUEUED_ON_SERVER,
STATUS_SUBMITTING_TASK,
+ STATUS_UPLOADING_FILE,
ParseStatusUpdate,
)
@@ -52,6 +53,7 @@ class V1ServerCapabilities:
tiers: tuple[str, ...]
output_formats: tuple[str, ...]
sources: tuple[str, ...]
+ model_output: bool = False
def normalize_v1_base_url(value: str) -> str:
@@ -145,16 +147,16 @@ class V1ArtifactClient:
raise V1ArtifactError("V1 health response did not report status=ok", code="invalid_response")
if not isinstance(features, dict):
raise V1ArtifactError("V1 health response is missing features", code="invalid_response")
- output_formats = _string_tuple(features.get("output_formats"), "features.output_formats")
+ output_formats = _string_tuple(features.get("task_output_formats"), "features.task_output_formats")
sources = _string_tuple(features.get("sources"), "features.sources")
- if "zip" not in output_formats:
+ if "default" not in output_formats:
raise V1ArtifactError(
- "The configured V1 API server does not advertise zip output, which is required by Gradio.",
+ "The configured V1 API server does not advertise default task output, which is required by Gradio.",
code="unsupported_output_format",
)
- if "file_id" not in sources:
+ if "upload" not in sources:
raise V1ArtifactError(
- "The configured V1 API server does not advertise file_id uploads, which are required by Gradio.",
+ "The configured V1 API server does not advertise task uploads, which are required by Gradio.",
code="unsupported_source",
)
@@ -174,6 +176,7 @@ class V1ArtifactClient:
tiers=tuple(tiers),
output_formats=output_formats,
sources=sources,
+ model_output=features.get("task_model_output") is True,
)
self._capabilities = capabilities
return capabilities
@@ -209,12 +212,13 @@ class V1ArtifactClient:
tier=cast(Tier, tier),
ocr_mode=ocr_mode,
include_images=True,
- include_model_output=True,
+ include_model_output=self._capabilities.model_output,
)
- def on_job_status(status: ApiJobStatus) -> None:
+ def on_job_status(status: ApiTaskStatus) -> None:
"""将真实 V1 状态映射到卡片阶段,服务端完成仅代表开始下载。"""
- messages: dict[ApiJobStatus, str] = {
+ messages: dict[ApiTaskStatus, str] = {
+ "waiting_upload": STATUS_UPLOADING_FILE,
"queued": STATUS_QUEUED_ON_SERVER,
"running": STATUS_PROCESSING_ON_SERVER,
"completed": STATUS_DOWNLOADING_RESULT,
diff --git a/mineru/kit/gradio/i18n.py b/mineru/kit/gradio/i18n.py
index 01c47b0a..04c957e7 100644
--- a/mineru/kit/gradio/i18n.py
+++ b/mineru/kit/gradio/i18n.py
@@ -122,6 +122,7 @@ MESSAGES: dict[str, tuple[str, str]] = {
"status_step_failed": ("Failed", "失败"),
"preparing_request": ("Preparing request...", "正在准备请求…"),
"checking_server": ("Checking server status...", "正在检查服务状态…"),
+ "uploading_file": ("Uploading file...", "正在上传文件…"),
"submitting_task": ("Submitting task...", "正在提交任务…"),
"queued_locally": ("Queued locally", "本地排队中"),
"queued_on_server": ("Queued on server", "服务端排队中"),
diff --git a/mineru/kit/gradio/status.py b/mineru/kit/gradio/status.py
index 8165b004..9ec1e53c 100644
--- a/mineru/kit/gradio/status.py
+++ b/mineru/kit/gradio/status.py
@@ -14,6 +14,7 @@ DEFAULT_STATUS = "Upload a file and start conversion."
STATUS_PREPARING_REQUEST = "Preparing request..."
STATUS_CHECKING_SERVER = "Checking server status..."
STATUS_SUBMITTING_TASK = "Submitting task..."
+STATUS_UPLOADING_FILE = "Uploading file..."
STATUS_QUEUED_LOCALLY = "Queued locally"
STATUS_QUEUED_ON_SERVER = "Queued on server"
STATUS_PROCESSING_ON_SERVER = "Processing on server..."
@@ -26,6 +27,7 @@ _MESSAGE_STEPS = {
STATUS_PREPARING_REQUEST: 0,
STATUS_CHECKING_SERVER: 1,
STATUS_SUBMITTING_TASK: 2,
+ STATUS_UPLOADING_FILE: 2,
STATUS_QUEUED_LOCALLY: 3,
STATUS_QUEUED_ON_SERVER: 3,
STATUS_PROCESSING_ON_SERVER: 4,
@@ -157,6 +159,7 @@ __all__ = [
"STATUS_QUEUED_LOCALLY",
"STATUS_QUEUED_ON_SERVER",
"STATUS_SUBMITTING_TASK",
+ "STATUS_UPLOADING_FILE",
"ParseStatusUpdate",
"StatusPanelState",
"status_html",
diff --git a/mineru/kit/router/app.py b/mineru/kit/router/app.py
index eca36fb2..7c3643ef 100644
--- a/mineru/kit/router/app.py
+++ b/mineru/kit/router/app.py
@@ -9,41 +9,31 @@ import hashlib
import json
import time
from contextlib import asynccontextmanager
-from pathlib import Path
-from typing import Any, AsyncIterator, Literal, get_args
+from typing import Any, AsyncIterator, Literal
import httpx
from fastapi import FastAPI, Query, Request
from fastapi.responses import JSONResponse, Response
-from ...filetypes import is_flash_only_parse_extension
-from ...parser.api_server import ApiServerError, CreateUploadRequest, OutputFormat
+from ...parser.api_server import ApiServerError
from ...parser.task_api import TaskStore, build_task_router
-from ...types import TIERS, select_default_quality_tier, validate_tier
-from ...utils.async_utils import drain_future
+from ...types import TIERS
from ...version import __version__
from .proxy import (
RouterProxyError,
- copy_file_to_worker,
json_or_error,
passthrough_response,
release_job_load,
request_upstream,
- require_worker_generation,
rewrite_file_payload,
- rewrite_job_payload,
- rewrite_upload_payload,
router_error_response,
stream_upstream,
- upload_stored_source,
)
from .resources import (
- CopiedInputFile,
ResourceKind,
ResourceRegistry,
ResourceRoute,
SourceFileStore,
- stored_file_chunks,
utc_now_iso,
)
from ...utils.retention import RETENTION_SCAN_INTERVAL_SECONDS
@@ -136,49 +126,6 @@ def _list_registered_files(
}
-def _list_registered_jobs(
- registry: ResourceRegistry,
- *,
- owner_scope: str,
- status_filter: str | None,
- after: str | None,
- limit: int,
- order: Literal["asc", "desc"],
- created_after: str | None,
-) -> dict[str, Any]:
- """从 Router 注册表构造 V1 Jobs 列表并应用过滤与分页。"""
- allowed_statuses = set(status_filter.split(",")) if status_filter else None
- payloads = [
- copy.deepcopy(route.metadata["payload"])
- for route in registry.list("job", owner_scope=owner_scope)
- if isinstance(route.metadata.get("payload"), dict)
- ]
- payloads = [
- payload
- for payload in payloads
- if (allowed_statuses is None or payload.get("status") in allowed_statuses)
- and (created_after is None or str(payload.get("created_at") or "") >= created_after)
- ]
- payloads.sort(key=lambda item: str(item.get("created_at") or ""), reverse=order == "desc")
- start = next((index + 1 for index, item in enumerate(payloads) if item.get("job_id") == after), 0) if after else 0
- page = payloads[start : start + limit]
- return {
- "object": "list",
- "data": [
- {
- "job_id": payload["job_id"],
- "status": payload.get("status", "queued"),
- "created_at": payload.get("created_at", ""),
- "file_count": len(payload.get("files") or []),
- }
- for payload in page
- ],
- "first_id": page[0]["job_id"] if page else None,
- "last_id": page[-1]["job_id"] if page else None,
- "has_more": start + limit < len(payloads),
- }
-
-
def _usage_payload(request: Request, registry: ResourceRegistry, pool: WorkerPool, owner_scope: str) -> dict[str, Any]:
"""聚合当前 Router 进程的累计 jobs/files 与 worker 并发上限。"""
return {
@@ -199,96 +146,6 @@ def _usage_payload(request: Request, registry: ResourceRegistry, pool: WorkerPoo
}
-async def _cleanup_copied_files(
- copied_files: list[CopiedInputFile],
- *,
- request: Request | None,
- pool: WorkerPool,
- registry: ResourceRegistry,
- headers: dict[str, str] | None = None,
-) -> list[CopiedInputFile]:
- """幂等删除 cross-worker 输入副本,返回等待后续重试的失败项。"""
- pending: list[CopiedInputFile] = []
- for copied_file in copied_files:
- worker = pool.get(copied_file.worker_id)
- try:
- response = await request_upstream(
- pool,
- worker,
- "DELETE",
- f"/v1/files/{copied_file.upstream_file_id}",
- request=request,
- headers=headers,
- )
- except RouterProxyError:
- pending.append(copied_file)
- continue
- if response.status_code not in {200, 204, 404}:
- pending.append(copied_file)
- continue
- registry.remove_upstream_alias(
- "file",
- owner_scope=copied_file.owner_scope,
- worker_id=copied_file.worker_id,
- upstream_id=copied_file.upstream_file_id,
- )
- return pending
-
-
-async def _hydrate_job_output_files(
- payload: dict[str, Any],
- *,
- request: Request | None,
- pool: WorkerPool,
- registry: ResourceRegistry,
- headers: dict[str, str] | None = None,
-) -> None:
- """读取 Job output files 的完整元数据,使其立即进入 `/v1/files`。"""
- seen_public_ids: set[str] = set()
- for file_result in payload.get("files") or []:
- if not isinstance(file_result, dict) or not isinstance(file_result.get("output_files"), dict):
- continue
- for output in file_result["output_files"].values():
- if not isinstance(output, dict) or not isinstance(output.get("file_id"), str):
- continue
- public_file_id = output["file_id"]
- if public_file_id in seen_public_ids:
- continue
- seen_public_ids.add(public_file_id)
- route = registry.find("file", public_file_id)
- if route is None or isinstance(route.metadata.get("payload"), dict):
- continue
- worker = pool.get(route.worker_id)
- generation = worker.generation
- try:
- response = await request_upstream(
- pool,
- worker,
- "GET",
- f"/v1/files/{route.upstream_id}",
- request=request,
- headers=headers,
- )
- if response.status_code >= 400:
- route.metadata["hydration_error"] = {
- "status_code": response.status_code,
- "message": response.text,
- }
- continue
- metadata = json_or_error(response)
- except RouterProxyError as exc:
- route.metadata["hydration_error"] = {
- "status_code": exc.status_code,
- "code": exc.code,
- "message": exc.message,
- }
- continue
- if worker.generation != generation or registry.find("file", public_file_id) is not route:
- continue
- rewrite_file_payload(metadata, worker, registry, route.owner_scope)
- route.metadata.pop("hydration_error", None)
-
-
async def _finalize_terminal_job(
route: ResourceRoute,
payload: dict[str, Any],
@@ -298,103 +155,29 @@ async def _finalize_terminal_job(
registry: ResourceRegistry,
headers: dict[str, str] | None = None,
) -> None:
- """在 Job 终态补齐 outputs,并回收或保留待重试的输入副本。"""
- if payload.get("status") not in {"completed", "partial", "failed", "canceled"}:
- return
- try:
- await _hydrate_job_output_files(
- payload,
- request=request,
- pool=pool,
- registry=registry,
- headers=headers,
- )
- finally:
- copied_files = list(route.metadata.get("copied_inputs") or [])
- pending = await _cleanup_copied_files(
- copied_files,
- request=request,
- pool=pool,
- registry=registry,
- headers=headers,
- )
- if pending:
- route.metadata["copied_inputs"] = pending
- else:
- route.metadata.pop("copied_inputs", None)
- if not route.metadata.get("task_result_held"):
- route.metadata.pop("upstream_headers", None)
- route.metadata.setdefault("finalized_at", utc_now_iso())
-
-
-def _mark_missing_job(route: ResourceRoute, pool: WorkerPool) -> dict[str, Any]:
- """上游已丢失任务时保存失败快照并归还负载,避免重启后永久占用名额。"""
- payload = route.metadata.get("payload") or {}
- if payload.get("status") not in {"completed", "partial", "failed", "canceled"}:
- payload["status"] = "failed"
- payload["finished_at"] = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
- for file in payload.get("files") or []:
- if file.get("status") != "completed":
- file["status"] = "failed"
- file["error"] = {
- "type": "engine_error",
- "code": "upstream_job_lost",
- "message": "Upstream task no longer exists",
- }
- progress = payload.get("progress")
- if isinstance(progress, dict):
- progress["failed"] = progress.get("total", 0) - progress.get("completed", 0)
- route.metadata["payload"] = payload
- route.metadata["upstream_lost"] = True
- release_job_load(route, pool)
- return payload
+ """终态释放一次执行预占,结果映射和租约由 Tasks 专用路径管理。"""
+ if payload.get("status") in {"completed", "partial", "failed", "canceled"}:
+ release_job_load(route, pool)
+ if not route.metadata.get("task_result_held"):
+ route.metadata.pop("upstream_headers", None)
+ route.metadata.setdefault("finalized_at", utc_now_iso())
async def _reconcile_jobs_once(registry: ResourceRegistry, pool: WorkerPool) -> None:
- """后台查询活动或待清理 Job,并统一执行终态资源收敛。"""
+ """后台仅查询 worker Tasks,HTTP 查询方断连也能归还执行负载。"""
+ from .tasks import read_task_execution
+
for route in registry.list("job", include_expired=True):
- if not route.metadata.get("active_counted") and not route.metadata.get("copied_inputs"):
- continue
- headers = dict(route.metadata.get("upstream_headers") or {})
- if route.metadata.get("upstream_lost"):
- await _finalize_terminal_job(
- route, route.metadata["payload"], request=None, pool=pool, registry=registry, headers=headers
- )
+ if not route.metadata.get("active_counted"):
continue
- worker = pool.get(route.worker_id)
- generation = worker.generation
try:
- if route.metadata.get("task_protocol"):
- from .tasks import read_task_execution
-
- execution = await read_task_execution(route, pool, registry, headers=headers)
- await _finalize_terminal_job(route, execution.job, request=None, pool=pool, registry=registry, headers=headers)
- continue
- response = await request_upstream(
- pool,
- worker,
- "GET",
- f"/v1/parse/jobs/{route.upstream_id}",
- headers=headers,
- )
- if worker.generation != generation or route.metadata.get("upstream_lost"):
- continue
- if response.status_code >= 400:
- if response.status_code == 404:
- cached = _mark_missing_job(route, pool)
- await _finalize_terminal_job(route, cached, request=None, pool=pool, registry=registry, headers=headers)
- continue
- payload = json_or_error(response)
- rewritten = rewrite_job_payload(payload, worker, registry, pool, route.owner_scope)
- route.metadata["payload"] = copy.deepcopy(rewritten)
- await _finalize_terminal_job(
+ execution = await read_task_execution(
route,
- rewritten,
- request=None,
- pool=pool,
- registry=registry,
- headers=headers,
+ pool,
+ registry,
+ headers=dict(route.metadata.get("upstream_headers") or {}),
)
+ await _finalize_terminal_job(route, execution.job, request=None, pool=pool, registry=registry)
except (RouterProxyError, ApiServerError):
continue
@@ -440,11 +223,8 @@ def create_app(
progress = payload.get("progress")
if isinstance(progress, dict):
progress["failed"] = progress.get("total", 0) - progress.get("completed", 0)
- route.metadata.pop("copied_inputs", None)
for route in registry.remove_worker(worker.worker_id):
- if route.kind == "upload":
- source_store.discard_upload(route.public_id)
- elif route.kind == "file":
+ if route.kind == "file":
source_store.delete_file(route.public_id)
pool = WorkerPool(
@@ -541,9 +321,6 @@ def create_app(
if not workers:
raise RouterProxyError(503, "upstream_unavailable", "No healthy MinerU V1 upstream is available")
sources = sorted({source for worker in workers for source in worker.features.get("sources") or []})
- output_formats = sorted(
- {output_format for worker in workers for output_format in worker.features.get("output_formats") or []}
- )
task_output_formats = sorted(
{output_format for worker in workers for output_format in worker.features.get("task_output_formats") or []}
)
@@ -553,8 +330,8 @@ def create_app(
"version": __version__,
"features": {
"webhook": False,
- "output_formats": output_formats,
"task_output_formats": task_output_formats,
+ "task_model_output": any(worker.features.get("task_model_output") is True for worker in workers),
"sources": sources,
},
}
@@ -601,158 +378,6 @@ def create_app(
data.append(tier_info)
return {"object": "list", "data": data}
- @application.post("/v1/uploads")
- async def create_upload(request: Request) -> Response:
- """按负载和上传轮询选择 worker,并复用本调用方已有的源字节。"""
- owner_scope = _caller_scope(request)
- body = await _read_json_object(request)
- try:
- validated = CreateUploadRequest.model_validate(body)
- except ValueError as exc:
- raise RouterProxyError(400, "invalid_request", str(exc)) from exc
- body = validated.model_dump(exclude_none=True)
- worker = pool.select(tier=None, required_sources={"file_id"}, selection_kind="upload")
- if worker is None:
- raise RouterProxyError(503, "upstream_unavailable", "No upstream accepts file uploads")
- cached = source_store.find_hash(owner_scope, body["sha256sum"]) if isinstance(body.get("sha256sum"), str) else None
- if cached is not None:
- if body.get("bytes") != cached.bytes:
- raise RouterProxyError(400, "upload_size_mismatch", "Declared size does not match cached bytes")
- with source_store.pin(cached):
- result = await upload_stored_source(
- cached, str(body.get("filename") or "input.bin"), worker, request=request, pool=pool, declaration=body
- )
- rewritten = rewrite_upload_payload(result, worker, registry, owner_scope)
- source_store.bind_source(
- str(rewritten["file"]["id"]),
- cached,
- owner_scope=owner_scope,
- expires_at=rewritten["file"].get("expires_at"),
- )
- return JSONResponse(rewritten)
- upstream_body = dict(body)
- # 本地未命中时必须收到源字节;仅凭 worker 的 blob 命中无法支持跨 worker 复制。
- upstream_body.pop("sha256sum", None)
- upstream = await request_upstream(pool, worker, "POST", "/v1/uploads", request=request, json_body=upstream_body)
- result = _successful_json(upstream)
- if isinstance(result, Response):
- return result
- rewritten = rewrite_upload_payload(result, worker, registry, owner_scope)
- if isinstance(body.get("sha256sum"), str):
- rewritten["sha256sum"] = body["sha256sum"]
- route = _route_or_404(registry, "upload", str(rewritten["id"]), owner_scope)
- route.metadata["declared"] = copy.deepcopy(body)
- return JSONResponse(rewritten, status_code=upstream.status_code)
-
- @application.get("/v1/uploads/{upload_id}")
- async def get_upload(upload_id: str, request: Request) -> Response:
- """把 Router upload 查询路由到其所属 worker。"""
- owner_scope = _caller_scope(request)
- route = _route_or_404(registry, "upload", upload_id, owner_scope)
- worker = pool.get(route.worker_id)
- upstream = await request_upstream(pool, worker, "GET", f"/v1/uploads/{route.upstream_id}", request=request)
- result = _successful_json(upstream)
- if isinstance(result, Response):
- return result
- return JSONResponse(
- rewrite_upload_payload(result, worker, registry, owner_scope),
- status_code=upstream.status_code,
- )
-
- @application.put("/v1/uploads/{upload_id}/content")
- async def upload_content(upload_id: str, request: Request) -> Response:
- """把上传内容流式暂存到 Router,再写入 upload 所属 worker。"""
- route = _route_or_404(registry, "upload", upload_id, _caller_scope(request))
- with registry.pin_route(route):
- upload_payload = route.metadata.get("payload")
- upload_status = upload_payload.get("status") if isinstance(upload_payload, dict) else None
- if source_store.is_bound_upload(upload_id) or upload_status == "completed":
- raise RouterProxyError(409, "upload_already_completed", f"Upload {upload_id} is already completed")
- if upload_status == "canceled":
- raise RouterProxyError(409, "upload_already_canceled", f"Upload {upload_id} is already canceled")
- worker = pool.get(route.worker_id)
- declared = route.metadata.get("declared") if isinstance(route.metadata.get("declared"), dict) else {}
- mime_type = str(declared.get("mime_type") or request.headers.get("content-type") or "application/octet-stream")
- try:
- stored = await source_store.stage_upload(upload_id, request.stream(), mime_type=mime_type)
- except ValueError as exc:
- raise RouterProxyError(409, "upload_not_writable", str(exc)) from exc
- declared_bytes = declared.get("bytes")
- if isinstance(declared_bytes, int) and stored.bytes != declared_bytes:
- source_store.discard_upload(upload_id)
- raise RouterProxyError(
- 400,
- "upload_size_mismatch",
- f"Expected {declared_bytes} upload bytes, received {stored.bytes}",
- )
- declared_sha256 = declared.get("sha256sum")
- if isinstance(declared_sha256, str) and stored.sha256sum != declared_sha256:
- source_store.discard_upload(upload_id)
- raise RouterProxyError(400, "upload_sha256_mismatch", "Uploaded content SHA256 does not match request")
- upstream = await request_upstream(
- pool,
- worker,
- "PUT",
- f"/v1/uploads/{route.upstream_id}/content",
- request=request,
- content=stored_file_chunks(stored.path),
- headers={"content-type": "application/octet-stream"},
- )
- return passthrough_response(upstream)
-
- @application.post("/v1/uploads/{upload_id}/complete")
- async def complete_upload(upload_id: str, request: Request) -> Response:
- """完成所属 worker 的 upload,并注册返回的 V1 file。"""
- owner_scope = _caller_scope(request)
- route = _route_or_404(registry, "upload", upload_id, owner_scope)
- worker = pool.get(route.worker_id)
- stored = source_store.find_upload(upload_id)
- if stored is None:
- raise RouterProxyError(409, "upload_not_ready", "Upload bytes have not been received by Router")
- body = await _read_json_object(request, allow_empty=True)
- body.setdefault("sha256sum", stored.sha256sum)
- upstream = await request_upstream(
- pool,
- worker,
- "POST",
- f"/v1/uploads/{route.upstream_id}/complete",
- request=request,
- json_body=body,
- )
- result = _successful_json(upstream)
- if isinstance(result, Response):
- return result
- rewritten = rewrite_upload_payload(result, worker, registry, owner_scope)
- file_payload = rewritten.get("file")
- if not isinstance(file_payload, dict) or not isinstance(file_payload.get("id"), str):
- raise RouterProxyError(502, "invalid_upstream_response", "Completed upload did not return a file")
- source_store.bind_file(
- upload_id, file_payload["id"], owner_scope=owner_scope, expires_at=file_payload.get("expires_at")
- )
- return JSONResponse(rewritten, status_code=upstream.status_code)
-
- @application.post("/v1/uploads/{upload_id}/cancel")
- async def cancel_upload(upload_id: str, request: Request) -> Response:
- """取消所属 worker 中尚未完成的 upload。"""
- owner_scope = _caller_scope(request)
- route = _route_or_404(registry, "upload", upload_id, owner_scope)
- worker = pool.get(route.worker_id)
- upstream = await request_upstream(
- pool,
- worker,
- "POST",
- f"/v1/uploads/{route.upstream_id}/cancel",
- request=request,
- )
- result = _successful_json(upstream)
- if isinstance(result, Response):
- return result
- source_store.discard_upload(upload_id)
- return JSONResponse(
- rewrite_upload_payload(result, worker, registry, owner_scope),
- status_code=upstream.status_code,
- )
-
@application.get("/v1/files")
async def list_files(
request: Request,
@@ -808,247 +433,17 @@ def create_app(
result = _successful_json(upstream)
if isinstance(result, Response):
return result
- for job_route in registry.list("job", owner_scope=owner_scope):
- copied_inputs = list(job_route.metadata.get("copied_inputs") or [])
- matching = [item for item in copied_inputs if item.source_public_id == file_id]
- if not matching:
- continue
- job_payload = job_route.metadata.get("payload")
- if not isinstance(job_payload, dict) or job_payload.get("status") not in {
- "completed",
- "partial",
- "failed",
- "canceled",
- }:
- continue
- pending = await _cleanup_copied_files(
- matching,
- request=request,
- pool=pool,
- registry=registry,
- )
- unrelated = [item for item in copied_inputs if item.source_public_id != file_id]
- if unrelated or pending:
- job_route.metadata["copied_inputs"] = [*unrelated, *pending]
- else:
- job_route.metadata.pop("copied_inputs", None)
- job_route.metadata.setdefault("finalized_at", utc_now_iso())
registry.remove("file", file_id)
source_store.delete_file(file_id)
result["id"] = file_id
return JSONResponse(result, status_code=upstream.status_code)
- async def submit_job(body: dict[str, Any], request: Request) -> Response:
- """为既有 V1 Job 选择并预占 worker,继续处理跨 worker 源文件复制。"""
- owner_scope = _caller_scope(request)
- body = copy.deepcopy(body)
- output_formats = body.get("output_formats", ["markdown"])
- if not isinstance(output_formats, list) or any(not isinstance(fmt, str) for fmt in output_formats):
- raise RouterProxyError(400, "invalid_request", "output_formats must be a list of strings", "output_formats")
- for fmt in output_formats:
- if fmt not in get_args(OutputFormat):
- raise RouterProxyError(400, "unsupported_output_format", f"Unknown output format: {fmt}", "output_formats")
- files = body.get("files")
- if not isinstance(files, list) or not files:
- raise RouterProxyError(400, "invalid_request", "Job request must contain at least one file", "files")
- required_sources: set[str] = set()
- file_routes: list[tuple[dict[str, Any], ResourceRoute]] = []
- names: list[str] = []
- for index, entry in enumerate(files):
- source = entry.get("source") if isinstance(entry, dict) else None
- if not isinstance(source, dict) or not isinstance(source.get("type"), str):
- raise RouterProxyError(400, "invalid_request", "Each file must contain a typed source", f"files.{index}.source")
- required_sources.add(source["type"])
- if source["type"] == "file_id":
- public_file_id = source.get("file_id")
- if not isinstance(public_file_id, str):
- raise RouterProxyError(400, "invalid_request", "file_id source requires a string file_id")
- route = _route_or_404(registry, "file", public_file_id, owner_scope)
- file_routes.append((source, route))
- names.append(str((route.metadata.get("payload") or {}).get("filename") or ""))
- else:
- if source["type"] == "url":
- names.append(str(source.get("url") or "").rsplit("/", 1)[-1].split("?")[0] or "unknown")
- elif source["type"] == "local":
- names.append(Path(str(source.get("path") or "")).name)
- else:
- names.append(str(source.get("name") or ""))
- can_default_to_flash = all(
- is_flash_only_parse_extension(name) or (entry["source"]["type"] == "url" and not Path(name).suffix)
- for name, entry in zip(names, files)
- )
- raw_tier = body.get("tier")
- try:
- tier = validate_tier(raw_tier) if raw_tier is not None else select_default_quality_tier(pool.available_tiers())
- except ValueError as exc:
- raise RouterProxyError(400, "invalid_request", str(exc), "tier") from exc
- if tier is None and can_default_to_flash and "flash" in pool.available_tiers():
- tier = "flash"
- if tier is None:
- raise RouterProxyError(503, "quality_tier_unavailable", "No default quality tier is available")
- body["tier"] = tier
- selection_tier = "flash" if can_default_to_flash and tier not in pool.available_tiers() else tier
- owner_ids = {route.worker_id for _, route in file_routes}
- preferred = next(iter(owner_ids)) if len(owner_ids) == 1 else None
- reservation = pool.reserve_job(
- tier=selection_tier,
- required_sources=required_sources,
- preferred_worker_id=preferred,
- required_outputs=set(output_formats),
- require_flash=any(is_flash_only_parse_extension(name) for name in names),
- )
- if reservation is None:
- raise RouterProxyError(
- 503, "quality_tier_unavailable", f"No healthy upstream supports tier '{tier}' and requested formats"
- )
- worker = reservation.worker
- input_aliases: dict[str, str] = {}
- copied_files: list[CopiedInputFile] = []
- transferred = False
-
- async def rollback() -> None:
- """回收未被任务接管的跨 worker 副本和本次创建的源文件视图。"""
- if worker.generation == reservation.generation and worker.base_url:
- await _cleanup_copied_files(copied_files, request=request, pool=pool, registry=registry)
-
- try:
- for source, route in file_routes:
- target_file_id = route.upstream_id
- if route.worker_id != worker.worker_id:
- target_file_id = await copy_file_to_worker(
- route, worker, request=request, pool=pool, registry=registry, source_store=source_store
- )
- copied_files.append(
- CopiedInputFile(
- source_public_id=route.public_id,
- owner_scope=owner_scope,
- worker_id=worker.worker_id,
- upstream_file_id=target_file_id,
- )
- )
- input_aliases[target_file_id] = route.public_id
- source["file_id"] = target_file_id
- upstream = await request_upstream(pool, worker, "POST", "/v1/parse/jobs", request=request, json_body=body)
- require_worker_generation(worker, reservation.generation)
- result = _successful_json(upstream)
- if isinstance(result, Response):
- return result
- upstream_job_id = result.get("job_id")
- if not isinstance(upstream_job_id, str):
- raise RouterProxyError(502, "invalid_upstream_response", "Created job did not return a job_id")
- job_route = registry.register(
- "job", owner_scope=owner_scope, worker_id=worker.worker_id, upstream_id=upstream_job_id
- )
- job_route.metadata.update(
- input_aliases=input_aliases,
- copied_inputs=copied_files,
- active_counted=True,
- reservation=reservation,
- upstream_headers=_authorization_headers(request),
- )
- transferred = True
- rewritten = rewrite_job_payload(result, worker, registry, pool, owner_scope)
- await _finalize_terminal_job(job_route, rewritten, request=request, pool=pool, registry=registry)
- return JSONResponse(rewritten, status_code=upstream.status_code)
- finally:
- if not transferred:
- reservation.release()
- await drain_future(asyncio.create_task(rollback()))
-
- @application.post("/v1/parse/jobs")
- async def create_job(request: Request) -> Response:
- """沿用 V1 请求响应,通过共享服务提交并预占解析负载。"""
- return await submit_job(await _read_json_object(request), request)
-
- @application.get("/v1/parse/jobs")
- async def list_jobs(
- request: Request,
- status_filter: str | None = Query(default=None, alias="status"),
- limit: int = Query(default=20, ge=1, le=100),
- after: str | None = None,
- order: Literal["asc", "desc"] = "desc",
- created_after: str | None = None,
- ) -> dict[str, Any]:
- """列出当前 Router 进程创建的 jobs。"""
- return _list_registered_jobs(
- registry,
- owner_scope=_caller_scope(request),
- status_filter=status_filter,
- after=after,
- limit=limit,
- order=order,
- created_after=created_after,
- )
-
- @application.get("/v1/parse/jobs/{job_id}")
- async def get_job(job_id: str, request: Request) -> Response:
- """查询 Router job 所属 worker 的最新状态并重写所有资源标识。"""
- owner_scope = _caller_scope(request)
- route = _route_or_404(registry, "job", job_id, owner_scope)
- if route.metadata.get("upstream_lost"):
- return JSONResponse(route.metadata["payload"])
- if route.metadata.get("task_protocol"):
- execution = await read_task_execution(route, pool, registry, request=request)
- await _finalize_terminal_job(route, execution.job, request=request, pool=pool, registry=registry)
- return JSONResponse(execution.job)
- worker = pool.get(route.worker_id)
- generation = worker.generation
- upstream = await request_upstream(pool, worker, "GET", f"/v1/parse/jobs/{route.upstream_id}", request=request)
- if worker.generation != generation or route.metadata.get("upstream_lost"):
- return JSONResponse(route.metadata["payload"])
- if upstream.status_code == 404:
- failed = _mark_missing_job(route, pool)
- await _finalize_terminal_job(route, failed, request=request, pool=pool, registry=registry)
- return JSONResponse(failed)
- result = _successful_json(upstream)
- if isinstance(result, Response):
- return result
- rewritten = rewrite_job_payload(result, worker, registry, pool, owner_scope)
- await _finalize_terminal_job(
- route,
- rewritten,
- request=request,
- pool=pool,
- registry=registry,
- )
- if worker.generation != generation or route.metadata.get("upstream_lost"):
- return JSONResponse(route.metadata["payload"])
- return JSONResponse(rewritten, status_code=upstream.status_code)
-
- @application.delete("/v1/parse/jobs/{job_id}")
- async def cancel_job(job_id: str, request: Request) -> Response:
- """取消 Router job,并保持公共 job ID。"""
- route = _route_or_404(registry, "job", job_id, _caller_scope(request))
- if route.metadata.get("upstream_lost"):
- raise RouterProxyError(409, "job_already_terminal", "Job is failed")
- if route.metadata.get("task_protocol"):
- await task_backend(request).cancel(job_id)
- return JSONResponse({"job_id": job_id, "status": "canceled", "canceled_at": utc_now_iso()})
- worker = pool.get(route.worker_id)
- upstream = await request_upstream(pool, worker, "DELETE", f"/v1/parse/jobs/{route.upstream_id}", request=request)
- result = _successful_json(upstream)
- if isinstance(result, Response):
- return result
- result["job_id"] = job_id
- release_job_load(route, pool)
- payload = route.metadata.get("payload")
- if isinstance(payload, dict):
- payload["status"] = "canceled"
- await _finalize_terminal_job(
- route,
- payload,
- request=request,
- pool=pool,
- registry=registry,
- )
- return JSONResponse(result, status_code=upstream.status_code)
-
@application.get("/v1/usage")
async def usage(request: Request) -> dict[str, Any]:
"""返回当前 Router 进程聚合的 V1 usage。"""
return _usage_payload(request, registry, pool, _caller_scope(request))
- from .tasks import RouterTaskBackend, install_task_error_handlers, read_task_execution
+ from .tasks import RouterTaskBackend, install_task_error_handlers
async def finalize_task_job(route: ResourceRoute, payload: dict[str, Any], request: Request | None) -> None:
"""新任务沿用既有产物元数据补齐及负载释放边界。"""
diff --git a/mineru/kit/router/proxy.py b/mineru/kit/router/proxy.py
index 3a701458..c92944d5 100644
--- a/mineru/kit/router/proxy.py
+++ b/mineru/kit/router/proxy.py
@@ -3,17 +3,15 @@
from __future__ import annotations
-import asyncio
import copy
from collections.abc import AsyncIterator, Mapping
-from typing import Any, cast
+from typing import Any
import httpx
from fastapi import Request
from fastapi.responses import JSONResponse, Response, StreamingResponse
-from ...utils.async_utils import drain_future
-from .resources import ResourceRegistry, ResourceRoute, SourceFileStore, StoredSourceFile, stored_file_chunks
+from .resources import ResourceRegistry, ResourceRoute
from .workers import JobReservation, WorkerPool, WorkerState
_HOP_BY_HOP_HEADERS = frozenset(
@@ -202,92 +200,6 @@ def rewrite_file_payload(
return rewritten
-def rewrite_upload_payload(
- payload: dict[str, Any],
- worker: WorkerState,
- registry: ResourceRegistry,
- owner_scope: str,
-) -> dict[str, Any]:
- """注册 upload 与完成后的 file,并重写 upload URL 和资源标识。"""
- rewritten = copy.deepcopy(payload)
- upstream_id = str(rewritten.get("id") or "")
- if not upstream_id:
- return rewritten
- route = registry.register(
- "upload",
- owner_scope=owner_scope,
- worker_id=worker.worker_id,
- upstream_id=upstream_id,
- )
- rewritten["id"] = route.public_id
- if rewritten.get("upload_url") is not None:
- rewritten["upload_url"] = f"/v1/uploads/{route.public_id}/content"
- if isinstance(rewritten.get("file"), dict):
- rewritten["file"] = rewrite_file_payload(rewritten["file"], worker, registry, owner_scope)
- route.metadata["payload"] = copy.deepcopy(rewritten)
- return rewritten
-
-
-def rewrite_job_payload(
- payload: dict[str, Any],
- worker: WorkerState,
- registry: ResourceRegistry,
- pool: WorkerPool,
- owner_scope: str,
-) -> dict[str, Any]:
- """注册 job、输入/输出 files,重写 links,并在终态归还 worker 负载。"""
- rewritten = copy.deepcopy(payload)
- upstream_id = str(rewritten.get("job_id") or "")
- if not upstream_id:
- return rewritten
- route = registry.register(
- "job",
- owner_scope=owner_scope,
- worker_id=worker.worker_id,
- upstream_id=upstream_id,
- )
- rewritten["job_id"] = route.public_id
- input_aliases = cast(dict[str, str], route.metadata.get("input_aliases") or {})
- for file_result in rewritten.get("files") or []:
- if not isinstance(file_result, dict):
- continue
- upstream_file_id = file_result.get("file_id")
- if isinstance(upstream_file_id, str):
- alias = input_aliases.get(upstream_file_id)
- if alias is not None:
- file_result["file_id"] = alias
- else:
- file_route = registry.register(
- "file",
- owner_scope=owner_scope,
- worker_id=worker.worker_id,
- upstream_id=upstream_file_id,
- )
- file_result["file_id"] = file_route.public_id
- output_files = file_result.get("output_files")
- if isinstance(output_files, dict):
- for output in output_files.values():
- if not isinstance(output, dict) or not isinstance(output.get("file_id"), str):
- continue
- output_route = registry.register(
- "file",
- owner_scope=owner_scope,
- worker_id=worker.worker_id,
- upstream_id=output["file_id"],
- )
- output["file_id"] = output_route.public_id
- links = rewritten.get("links")
- if isinstance(links, dict):
- links["self"] = f"/v1/parse/jobs/{route.public_id}"
- if links.get("cancel") is not None:
- links["cancel"] = f"/v1/parse/jobs/{route.public_id}"
- status = str(rewritten.get("status") or "")
- if status in _TERMINAL_JOB_STATUSES:
- release_job_load(route, pool)
- route.metadata["payload"] = copy.deepcopy(rewritten)
- return rewritten
-
-
def release_job_load(route: ResourceRoute, pool: WorkerPool) -> None:
"""所有终态入口共用一次性释放,预占对象校验 worker generation。"""
if route.metadata.pop("active_counted", False):
@@ -298,264 +210,21 @@ def release_job_load(route: ResourceRoute, pool: WorkerPool) -> None:
pool.mark_job_finished(route.worker_id)
-async def upload_stored_source(
- stored: StoredSourceFile,
- filename: str,
- target: WorkerState,
- *,
- request: Request,
- pool: WorkerPool,
- declaration: dict[str, Any] | None = None,
- generation: int | None = None,
-) -> dict[str, Any]:
- """使用已经校验的缓存字节完成 worker 上传,失败和取消均清理中间资源。"""
- generation = target.generation if generation is None else generation
- require_worker_generation(target, generation)
- body = dict(declaration or {})
- body.update(
- filename=filename, bytes=stored.bytes, mime_type=body.get("mime_type") or stored.mime_type, sha256sum=stored.sha256sum
- )
- body.setdefault("purpose", "parse")
- response = await request_upstream(pool, target, "POST", "/v1/uploads", request=request, json_body=body)
- payload = json_or_error(response)
- require_worker_generation(target, generation)
- if isinstance(payload.get("file"), dict):
- return payload
- upload_id = payload.get("id")
- if not isinstance(upload_id, str):
- raise RouterProxyError(502, "invalid_upstream_response", "Upload did not return an ID")
- try:
- response = await request_upstream(
- pool,
- target,
- "PUT",
- f"/v1/uploads/{upload_id}/content",
- request=request,
- content=stored_file_chunks(stored.path),
- headers={"content-type": "application/octet-stream"},
- )
- if response.status_code >= 400:
- json_or_error(response)
- require_worker_generation(target, generation)
- response = await request_upstream(
- pool,
- target,
- "POST",
- f"/v1/uploads/{upload_id}/complete",
- request=request,
- json_body={"sha256sum": stored.sha256sum},
- )
- payload = json_or_error(response)
- require_worker_generation(target, generation)
- if not isinstance(payload.get("file"), dict) or not isinstance(payload["file"].get("id"), str):
- raise RouterProxyError(502, "invalid_upstream_response", "Completed upload did not return a file")
- return payload
- except (Exception, asyncio.CancelledError):
- if target.generation == generation and target.base_url:
- await drain_future(
- asyncio.create_task(
- _cleanup_failed_target_transfer(target, upload_id, request=request, pool=pool, generation=generation)
- )
- )
- raise
-
-
def require_worker_generation(worker: WorkerState, generation: int) -> None:
"""阻止 replacement 期间的迟到响应发布旧进程的资源标识。"""
if worker.generation != generation or not worker.base_url:
raise RouterProxyError(503, "upstream_unavailable", "Worker changed while the request was being submitted")
-async def _cleanup_failed_target_transfer(
- target: WorkerState,
- upload_id: str,
- *,
- request: Request,
- pool: WorkerPool,
- generation: int | None = None,
-) -> None:
- """尽力取消失败 transfer 的 Upload,或删除已完成但响应丢失的 File。"""
- generation = target.generation if generation is None else generation
- if target.generation != generation or not target.base_url:
- return
- try:
- lookup = await request_upstream(
- pool,
- target,
- "GET",
- f"/v1/uploads/{upload_id}",
- request=request,
- )
- if target.generation != generation or not target.base_url:
- return
- if lookup.status_code == 200:
- try:
- payload = lookup.json()
- except ValueError:
- payload = {}
- file_payload = payload.get("file") if isinstance(payload, dict) else None
- if isinstance(file_payload, dict) and isinstance(file_payload.get("id"), str):
- await request_upstream(
- pool,
- target,
- "DELETE",
- f"/v1/files/{file_payload['id']}",
- request=request,
- )
- return
- except RouterProxyError:
- pass
- if target.generation != generation or not target.base_url:
- return
- try:
- await request_upstream(
- pool,
- target,
- "POST",
- f"/v1/uploads/{upload_id}/cancel",
- request=request,
- )
- except RouterProxyError:
- pass
-
-
-async def copy_file_to_worker(
- route: ResourceRoute,
- target: WorkerState,
- *,
- request: Request,
- pool: WorkerPool,
- registry: ResourceRegistry,
- source_store: SourceFileStore,
-) -> str:
- """优先使用 Router 私有暂存输入,通过 Upload API 复制到目标 worker。"""
- generation = target.generation
- source = pool.get(route.worker_id)
- metadata = route.metadata.get("payload")
- if not isinstance(metadata, dict):
- metadata_response = await request_upstream(
- pool,
- source,
- "GET",
- f"/v1/files/{route.upstream_id}",
- request=request,
- )
- metadata = json_or_error(metadata_response)
- stored = source_store.find_file(route.public_id)
- if stored is not None:
- with source_store.pin(stored):
- payload = await upload_stored_source(
- stored,
- str(metadata.get("filename") or "input.bin"),
- target,
- request=request,
- pool=pool,
- generation=generation,
- declaration={
- "purpose": metadata.get("purpose") if metadata.get("purpose") in {"parse", "input_image"} else "parse"
- },
- )
- target_file_id = str(payload["file"]["id"])
- registry.alias_upstream(route, worker_id=target.worker_id, upstream_id=target_file_id)
- return target_file_id
- else:
- if metadata.get("purpose") != "parse_output":
- raise RouterProxyError(
- 409,
- "source_file_unavailable",
- f"Router source bytes for {route.public_id} are no longer available",
- )
- content_response = await request_upstream(
- pool,
- source,
- "GET",
- f"/v1/files/{route.upstream_id}/content",
- request=request,
- )
- if content_response.status_code >= 400:
- json_or_error(content_response)
- content = content_response.content
- content_size = len(content_response.content)
- content_sha256 = str(metadata.get("sha256sum") or "")
- content_type = content_response.headers.get("content-type") or "application/octet-stream"
- purpose = metadata.get("purpose") if metadata.get("purpose") in {"parse", "input_image"} else "parse"
- create_payload = {
- "filename": metadata.get("filename") or "input.bin",
- "bytes": content_size,
- "mime_type": content_type,
- "purpose": purpose,
- **({"sha256sum": content_sha256} if content_sha256 else {}),
- }
- require_worker_generation(target, generation)
- upload_response = await request_upstream(
- pool,
- target,
- "POST",
- "/v1/uploads",
- request=request,
- json_body=create_payload,
- )
- upload_payload = json_or_error(upload_response)
- require_worker_generation(target, generation)
- if isinstance(upload_payload.get("file"), dict):
- target_file_id = str(upload_payload["file"]["id"])
- registry.alias_upstream(route, worker_id=target.worker_id, upstream_id=target_file_id)
- return target_file_id
- upload_id = upload_payload.get("id")
- if not isinstance(upload_id, str):
- raise RouterProxyError(502, "invalid_upstream_response", "Target upload response did not return an upload ID")
- try:
- put_response = await request_upstream(
- pool,
- target,
- "PUT",
- f"/v1/uploads/{upload_id}/content",
- request=request,
- content=content,
- headers={"content-type": "application/octet-stream"},
- )
- if put_response.status_code >= 400:
- json_or_error(put_response)
- complete_response = await request_upstream(
- pool,
- target,
- "POST",
- f"/v1/uploads/{upload_id}/complete",
- request=request,
- json_body={"sha256sum": content_sha256} if content_sha256 else {},
- )
- complete_payload = json_or_error(complete_response)
- require_worker_generation(target, generation)
- file_payload = complete_payload.get("file")
- if not isinstance(file_payload, dict) or not isinstance(file_payload.get("id"), str):
- raise RouterProxyError(502, "invalid_upstream_response", "Completed upload did not return a file")
- target_file_id = file_payload["id"]
- except (Exception, asyncio.CancelledError):
- await _cleanup_failed_target_transfer(
- target,
- upload_id,
- request=request,
- pool=pool,
- generation=generation,
- )
- raise
- registry.alias_upstream(route, worker_id=target.worker_id, upstream_id=target_file_id)
- return target_file_id
-
-
__all__ = [
"RouterProxyError",
- "copy_file_to_worker",
"forwarded_headers",
"json_or_error",
"passthrough_response",
"request_upstream",
"require_worker_generation",
"release_job_load",
- "upload_stored_source",
"rewrite_file_payload",
- "rewrite_job_payload",
- "rewrite_upload_payload",
"router_error_response",
"stream_upstream",
]
diff --git a/mineru/kit/router/resources.py b/mineru/kit/router/resources.py
index cd10d06f..cdc0526f 100644
--- a/mineru/kit/router/resources.py
+++ b/mineru/kit/router/resources.py
@@ -15,10 +15,9 @@ from pathlib import Path
from typing import Any, Iterator, Literal
from ...utils.retention import resolve_retention_seconds
-ResourceKind = Literal["upload", "file", "job"]
+ResourceKind = Literal["file", "job"]
_PUBLIC_ID_PREFIXES: dict[ResourceKind, str] = {
- "upload": "upload_",
"file": "file-",
"job": "job_",
}
@@ -52,16 +51,6 @@ class StoredSourceFile:
mime_type: str
-@dataclass(frozen=True)
-class CopiedInputFile:
- """记录一个 Job 为 cross-worker 执行创建的目标 worker 输入副本。"""
-
- source_public_id: str
- owner_scope: str
- worker_id: str
- upstream_file_id: str
-
-
class SourceFileStore:
"""在 Router 临时目录中保存可供 cross-worker 重传的输入字节。"""
@@ -178,14 +167,6 @@ class SourceFileStore:
return
path.unlink(missing_ok=True)
- def is_bound_upload(self, upload_id: str) -> bool:
- """判断 Upload 的暂存路径是否已经绑定到完成后的公共 File。"""
- return upload_id in self._bound_uploads
-
- def find_upload(self, upload_id: str) -> StoredSourceFile | None:
- """读取公共 Upload 对应的私有暂存输入,不存在时返回 None。"""
- return self._uploads.get(upload_id)
-
def find_file(self, file_id: str) -> StoredSourceFile | None:
"""读取公共 File 对应的私有暂存输入,不存在时返回 None。"""
return self._files.get(file_id)
@@ -232,13 +213,12 @@ async def stored_file_chunks(path: Path, chunk_size: int = 1024 * 1024) -> Async
class ResourceRegistry:
- """维护当前 Router 进程创建或发现的 uploads、files 与 jobs。"""
+ """维护当前 Router 进程的文件与私有执行路由。"""
def __init__(self, retention_seconds: int | None = None) -> None:
"""初始化按公共标识和 upstream 标识建立的双向索引。"""
self.retention_seconds = resolve_retention_seconds(retention_seconds)
self._by_public: dict[ResourceKind, dict[str, ResourceRoute]] = {
- "upload": {},
"file": {},
"job": {},
}
@@ -298,8 +278,6 @@ class ResourceRegistry:
return False
if not self.retention_seconds:
return False
- if route.metadata.get("copied_inputs"):
- return False
origin = route.metadata.setdefault("finalized_at", utc_now_iso())
else:
explicit = payload.get("expires_at")
@@ -319,17 +297,13 @@ class ResourceRegistry:
protected: set[str] = set()
for route in self._by_public["job"].values():
payload = route.metadata.get("payload") or {}
- if payload.get("status") not in {"completed", "partial", "failed", "canceled"} or route.metadata.get(
- "copied_inputs"
- ):
+ if payload.get("status") not in {"completed", "partial", "failed", "canceled"}:
protected.update(file.get("file_id") for file in payload.get("files", []) if file.get("file_id"))
- protected.update(copy.source_public_id for copy in route.metadata.get("copied_inputs", []))
for kind, routes in self._by_public.items():
for public_id, route in tuple(routes.items()):
if (
not self.is_expired(route)
or public_id in protected
- or route.metadata.get("copied_inputs")
or route.metadata.get("pins")
or route.metadata.get("task_result_held")
):
@@ -337,8 +311,6 @@ class ResourceRegistry:
self.remove(kind, public_id)
if kind == "file":
sources.delete_file(public_id)
- elif kind == "upload":
- sources.discard_upload(public_id)
def get(self, kind: ResourceKind, public_id: str) -> ResourceRoute:
"""按公共标识读取资源路由,不存在时抛出 KeyError。"""
@@ -438,7 +410,6 @@ class ResourceRegistry:
__all__ = [
- "CopiedInputFile",
"ResourceKind",
"ResourceRegistry",
"ResourceRoute",
diff --git a/mineru/kit/router/tasks.py b/mineru/kit/router/tasks.py
index 12230e19..09d0052c 100644
--- a/mineru/kit/router/tasks.py
+++ b/mineru/kit/router/tasks.py
@@ -38,7 +38,6 @@ from .proxy import (
request_upstream,
require_worker_generation,
rewrite_file_payload,
- rewrite_job_payload,
)
from .resources import ResourceRegistry, ResourceRoute, SourceFileStore, StoredSourceFile, utc_now_iso
from .workers import WorkerPool, WorkerState
@@ -112,9 +111,9 @@ def _worker_path(worker: WorkerState, url: str) -> str:
return target.path[len(prefix) :]
-def _job_payload(payload: dict[str, Any], upstream_id: str, *, authenticated: bool) -> dict[str, Any]:
- """把 worker 的 tasks 元数据转换成既有 Router Job 视图,不泄露上传或 worker 链接。"""
- file_keys = ("file_id", "name", "page_range", "status", "parse", "output_files", "error")
+def _job_payload(payload: dict[str, Any], upstream_id: str) -> dict[str, Any]:
+ """把 worker Tasks 转换为私有执行快照,不包含旧 Jobs 公开协议。"""
+ file_keys = ("file_id", "name", "page_range", "status", "parse", "error")
files = [{key: item.get(key) for key in file_keys} for item in payload["files"]]
for file in files:
if file["status"] in {"waiting_upload", "canceled"}:
@@ -126,11 +125,8 @@ def _job_payload(payload: dict[str, Any], upstream_id: str, *, authenticated: bo
"started_at": payload.get("started_at"),
"finished_at": payload.get("finished_at"),
"tier": payload["tier"],
- "output_formats": [],
"progress": payload["progress"],
- "access_level": "registered" if authenticated else "anonymous",
"files": files,
- "links": {"self": f"/v1/parse/jobs/{upstream_id}", "cancel": f"/v1/parse/jobs/{upstream_id}"},
}
@@ -214,14 +210,14 @@ async def read_task_execution(
result_route = registry.find("file", result_file_id)
if result_route is not None:
result_route.metadata["task_result_held"] = bool(route.metadata.get("task_result_held"))
- authenticated = bool(request.headers.get("authorization") if request is not None else (headers or {}).get("authorization"))
- rewritten = rewrite_job_payload(
- _job_payload(payload, route.upstream_id, authenticated=authenticated),
- worker,
- registry,
- pool,
- route.owner_scope,
- )
+ rewritten = _job_payload(payload, route.public_id)
+ aliases = route.metadata.get("input_aliases") or {}
+ for item in rewritten["files"]:
+ if item.get("file_id") in aliases:
+ item["file_id"] = aliases[item["file_id"]]
+ if rewritten["status"] in TERMINAL_TASK_STATUSES:
+ release_job_load(route, pool)
+ route.metadata["payload"] = copy.deepcopy(rewritten)
route.metadata["task_result_file_id"] = result_file_id
error = ErrorDetail.model_validate(payload["error"]) if payload.get("error") else None
input_metadata = {
@@ -276,8 +272,9 @@ class RouterTaskBackend:
selection_tier = "flash" if lightweight else options.tier
candidates = self.pool.capable_workers(
tier=selection_tier,
- required_sources={"url" if item.url is not None else "file_id"},
+ required_sources={"url" if item.url is not None else "upload"},
required_task_outputs=set(options.output_formats),
+ require_model_output=options.include_model_output,
require_flash=lightweight,
)
if not candidates:
@@ -334,8 +331,9 @@ class RouterTaskBackend:
tier = "flash" if lightweight else options.tier
reservation = self.pool.reserve_job(
tier=tier,
- required_sources={"url" if item.url is not None else "file_id" for item in manifest},
+ required_sources={"url" if item.url is not None else "upload" for item in manifest},
required_task_outputs=set(options.output_formats),
+ require_model_output=options.include_model_output,
require_flash=any(is_flash_only_parse_extension(item.filename) for item in manifest),
)
if reservation is None:
@@ -365,6 +363,7 @@ class RouterTaskBackend:
"tier": options.tier,
"ocr_mode": options.ocr_mode,
"output_formats": options.output_formats,
+ "include_model_output": options.include_model_output,
},
headers={"x-mineru-task-lease-seconds": "120"},
)
@@ -380,7 +379,6 @@ class RouterTaskBackend:
)
authorization = self.request.headers.get("authorization")
route.metadata.update(
- task_protocol=True,
task_result_held=True,
active_counted=True,
reservation=reservation,
diff --git a/mineru/kit/router/workers.py b/mineru/kit/router/workers.py
index a6a45e37..3d430994 100644
--- a/mineru/kit/router/workers.py
+++ b/mineru/kit/router/workers.py
@@ -518,8 +518,8 @@ class WorkerPool:
tier: Tier | None,
required_sources: set[str] | None = None,
preferred_worker_id: str | None = None,
- required_outputs: set[str] | None = None,
required_task_outputs: set[str] | None = None,
+ require_model_output: bool = False,
require_flash: bool = False,
selection_kind: Literal["upload", "job"] = "job",
) -> WorkerState | None:
@@ -527,8 +527,8 @@ class WorkerPool:
eligible = self.capable_workers(
tier=tier,
required_sources=required_sources,
- required_outputs=required_outputs,
required_task_outputs=required_task_outputs,
+ require_model_output=require_model_output,
require_flash=require_flash,
)
if not eligible:
@@ -554,8 +554,8 @@ class WorkerPool:
*,
tier: Tier | None,
required_sources: set[str] | None = None,
- required_outputs: set[str] | None = None,
required_task_outputs: set[str] | None = None,
+ require_model_output: bool = False,
require_flash: bool = False,
) -> list[WorkerState]:
"""只读过滤完整能力组合,供预检复用;不改变轮询游标或预占负载。"""
@@ -564,8 +564,8 @@ class WorkerPool:
for worker in self.healthy_workers()
if (tier is None or tier in worker.tiers)
and (required_sources or set()).issubset(set(worker.features.get("sources") or []))
- and (required_outputs or set()).issubset(set(worker.features.get("output_formats") or []))
and (required_task_outputs or set()).issubset(set(worker.features.get("task_output_formats") or []))
+ and (not require_model_output or worker.features.get("task_model_output") is True)
and (not require_flash or "flash" in worker.tiers)
]
@@ -575,8 +575,8 @@ class WorkerPool:
tier: Tier,
required_sources: set[str],
preferred_worker_id: str | None = None,
- required_outputs: set[str] | None = None,
required_task_outputs: set[str] | None = None,
+ require_model_output: bool = False,
require_flash: bool = False,
) -> JobReservation | None:
"""在第一次网络等待前同步选择并预占解析 worker。"""
@@ -584,8 +584,8 @@ class WorkerPool:
tier=tier,
required_sources=required_sources,
preferred_worker_id=preferred_worker_id,
- required_outputs=required_outputs,
required_task_outputs=required_task_outputs,
+ require_model_output=require_model_output,
require_flash=require_flash,
)
if worker is None:
diff --git a/mineru/parser/__init__.py b/mineru/parser/__init__.py
index f3fd0447..866e457f 100644
--- a/mineru/parser/__init__.py
+++ b/mineru/parser/__init__.py
@@ -9,12 +9,13 @@ from docvortex.document.contracts import HtmlSourceContext
from ..config import VlmConfig
from ..utils.logger import configure_global_log_level
from ..types import Tier
-from .api_client import ApiJobStatus, MinerUApiParser
+from .api_client import ApiJobStatus, ApiTaskStatus, MinerUApiParser
from .base import MIDDLE_JSON_SCHEMA_VERSION, DocumentParser, ParseResult
from .mineru_parser import MinerUParser
__all__ = [
"ApiJobStatus",
+ "ApiTaskStatus",
"DocumentParser",
"MinerUApiParser",
"MinerUParser",
diff --git a/mineru/parser/api_client.py b/mineru/parser/api_client.py
index 70116a50..9763286a 100644
--- a/mineru/parser/api_client.py
+++ b/mineru/parser/api_client.py
@@ -21,16 +21,16 @@ import os
import random
import re
import time
+import tempfile
import zipfile
-from collections.abc import Callable
+from collections.abc import Callable, Iterator, AsyncIterator
from pathlib import Path
from typing import Any, Literal, cast
-from urllib.parse import unquote, urlparse
+from urllib.parse import unquote, urlparse, urljoin
import httpx
from docvortex.assets import validate_image_sidecar_path
-from ..filetypes import mime_type_for_extension
from ..types import PageInfo, Tier, validate_tier
from .base import DocumentParser, ParseResult
from .page_range import normalize_page_range_input
@@ -52,8 +52,9 @@ _HTML_IMAGE_SOURCE_RE = re.compile(
logger = logging.getLogger("mineru.api_client")
_RequestMethod = Literal["GET", "POST", "PUT"]
-ApiJobStatus = Literal["queued", "running", "completed", "partial", "failed", "canceled"]
-_JOB_STATUSES = {"queued", "running", *_TERMINAL_JOB_STATUSES}
+ApiTaskStatus = Literal["waiting_upload", "queued", "running", "completed", "partial", "failed", "canceled"]
+ApiJobStatus = ApiTaskStatus
+_JOB_STATUSES = {"waiting_upload", "queued", "running", *_TERMINAL_JOB_STATUSES}
def _notify_job_status(job: dict[str, Any], callback: Callable[[ApiJobStatus], None] | None) -> None:
@@ -101,29 +102,7 @@ class _APITransportError(Exception):
class MinerUApiParser(DocumentParser):
- """Parser that delegates to a MinerU v1 API server.
-
- Works with local-server, LAN, and cloud (mineru.net) deployments::
-
- # local deployment — uses ``local`` source only when the server advertises it.
- # Start the local server with --allow-local-source to skip upload.
- parser = MinerUApiParser(api_url="http://localhost:8000", tier="basic")
-
- # cloud (remote)
- parser = MinerUApiParser(
- api_url="https://mineru.net/api", api_key="sk_...",
- tier="standard",
- )
-
- result = parser.parse("report.pdf")
- print(result.markdown())
-
- Constructor parameters:
-
- - ``tier`` → v1 ``tier`` (``"flash"`` / ``"basic"`` / ``"standard"`` / ``"advanced"``); ``None`` omits the field
- - ``ocr_mode`` → v1 request OCR mode; ``None`` omits the field (server default: ``"auto"``)
- - ``page_range`` → per-file v1 ``page_range``
- """
+ """通过唯一 Tasks 协议解析文件;同机也先使用 hash 预检再按需上传。"""
DEFAULT_API_URL = "https://mineru.net/api"
@@ -137,61 +116,242 @@ class MinerUApiParser(DocumentParser):
include_images: bool = False,
include_model_output: bool = False,
) -> None:
- """保存请求配置;未指定 OCR 模式时省略字段,保留既有客户端的请求结构。"""
+ """只保存显式配置,不在构造时联网或读取文件。"""
if ocr_mode is not None and ocr_mode not in ("auto", "txt", "ocr"):
raise ValueError(f"Unsupported OCR mode '{ocr_mode}'")
self._base_url = (api_url or os.environ.get("MINERU_API_URL") or self.DEFAULT_API_URL).rstrip("/")
self._api_key = (api_key if api_key is not None else os.environ.get("MINERU_API_KEY")) or None
- self._local = _is_local_network_url(self._base_url)
self._trust_env = should_trust_env_for_url(self._base_url)
- self._source_features: set[str] | None = None
self.tier = validate_tier(tier) if tier is not None else None
self.ocr_mode = ocr_mode
self.include_images = include_images
self.include_model_output = include_model_output
- # ── DocumentParser interface ─────────────────────────────────────
+ def _headers(self) -> dict[str, str]:
+ """生成仅用于 API 同源地址的认证头。"""
+ return {"Authorization": f"Bearer {self._api_key}"} if self._api_key else {}
+
+ def _resolve(self, url: str, *, metadata: bool = False, base: str | None = None) -> str:
+ """解析链接并保留签名;元数据查询不得转向其他来源。"""
+ result = urljoin(base or self._base_url + "/", url)
+ _http_origin(result)
+ if metadata and _http_origin(result) != _http_origin(self._base_url):
+ raise _V1APIError("invalid_response", "Task metadata URL must have the API origin")
+ return result
+
+ @staticmethod
+ def _check(response: httpx.Response) -> dict[str, Any]:
+ """保留结构化服务错误及远端认证错误,拒绝非对象响应。"""
+ try:
+ payload = response.json()
+ except ValueError:
+ payload = {}
+ if not isinstance(payload, dict):
+ raise _V1APIError("invalid_response", "API response must be a JSON object")
+ if response.status_code >= 400 or _remote_auth_message(payload):
+ _raise_for_http_error(response.status_code, payload, response.text)
+ if not payload:
+ raise _V1APIError("invalid_response", "API returned an empty or invalid JSON response")
+ return payload
+
+ def _payload(self, path: Path, page_range: str) -> dict[str, Any]:
+ """固定本次源文件身份,任何部署都使用相同的清单和默认结果包。"""
+ if not path.is_file():
+ raise FileNotFoundError(path)
+ payload: dict[str, Any] = {
+ "manifest": [
+ {
+ "client_file_id": "file",
+ "filename": path.name,
+ "bytes": path.stat().st_size,
+ "sha256sum": _sha256_file(path),
+ "page_range": page_range,
+ }
+ ],
+ "output_formats": ["default"],
+ "include_model_output": self.include_model_output,
+ }
+ if self.tier is not None:
+ payload["tier"] = self.tier
+ if self.ocr_mode is not None:
+ payload["ocr_mode"] = self.ocr_mode
+ return payload
+
+ def _snapshot(self, response: httpx.Response, expected: dict[str, Any] | None = None) -> dict[str, Any]:
+ """校验稳定任务身份、唯一文件映射和产物选项,不依赖文件名或数组位置。"""
+ task = self._check(response)
+ files = task.get("files")
+ if (
+ not isinstance(task.get("task_id"), str)
+ or not task["task_id"]
+ or not isinstance(task.get("status"), str)
+ or task.get("status") not in _JOB_STATUSES
+ or task.get("output_formats") != ["default"]
+ or task.get("include_model_output", False) != self.include_model_output
+ or not isinstance(files, list)
+ or len(files) != 1
+ or not isinstance(files[0], dict)
+ or files[0].get("client_file_id") != "file"
+ or (expected is not None and task["task_id"] != expected["task_id"])
+ ):
+ raise _V1APIError("invalid_response", "Inconsistent Tasks response")
+ item = files[0]
+ if item.get("upload_status") not in ("pending", "ready", "expired") or item.get("status") not in (
+ "waiting_upload",
+ "queued",
+ "running",
+ "completed",
+ "failed",
+ "canceled",
+ ):
+ raise _V1APIError("invalid_response", "Invalid task file state")
+ if not isinstance(task.get("status_url"), str):
+ raise _V1APIError("invalid_response", "Task status URL is missing")
+ task["status_url"] = self._resolve(task["status_url"], metadata=True)
+ if files[0].get("status") == "completed" and not isinstance(files[0].get("result_url"), str):
+ raise _V1APIError("invalid_response", "Completed file is missing result_url")
+ return task
def parse(
self,
path: str | Path,
*,
page_range: str = "",
- status_callback: Callable[[ApiJobStatus], None] | None = None,
+ status_callback: Callable[[ApiTaskStatus], None] | None = None,
duration_callback: Callable[[float], None] | None = None,
) -> ParseResult:
- """同步解析;可选 duration_callback 在下载前接收服务端含打包的毫秒耗时。"""
- page_range = normalize_page_range_input(page_range)
+ """创建单文件任务、按需上传并下载完整 ZIP,回调对应真实服务端阶段。"""
file_path = Path(path)
- if not file_path.exists():
- raise FileNotFoundError(file_path)
+ payload = self._payload(file_path, normalize_page_range_input(page_range))
+ with httpx.Client(timeout=httpx.Timeout(120, connect=30), trust_env=self._trust_env, follow_redirects=False) as cli:
+ response = _request_once(
+ cli,
+ "POST",
+ f"{self._base_url}/v1/tasks",
+ stage="task submission",
+ headers=self._headers(),
+ json_body=payload,
+ )
+ task = self._snapshot(response)
+ _notify_job_status(task, status_callback)
+ task = self._upload_input(cli, task, file_path, status_callback)
+ for _ in range(_POLL_MAX_ATTEMPTS):
+ if task["status"] in _TERMINAL_JOB_STATUSES:
+ break
+ time.sleep(_POLL_INTERVAL_SECONDS)
+ task = self._query(cli, task)
+ _notify_job_status(task, status_callback)
+ if task["status"] not in _TERMINAL_JOB_STATUSES:
+ raise _V1APIError("timeout", f"Task {task['task_id']} did not complete within timeout")
+ _raise_for_terminal_job_error(task)
+ if task["files"][0].get("status") != "completed":
+ raise _V1APIError("parse_failed", "Task did not complete its input file")
+ _notify_job_duration(task, duration_callback)
+ with tempfile.TemporaryDirectory(prefix="mineru-api-client-") as directory:
+ target = Path(directory) / "result.zip"
+ self._download_task(cli, task, target)
+ return _parse_result_from_zip_bytes(
+ target, include_images=self.include_images, include_model_output=self.include_model_output
+ )
- payload = self._build_payload(self._build_source(file_path), page_range)
- job = self._do_parse(payload, status_callback=status_callback)
- _notify_job_duration(job, duration_callback)
- return self._build_result(job, file_path.name)
+ def _query(self, cli: httpx.Client, task: dict[str, Any]) -> dict[str, Any]:
+ """按服务返回的同源状态链接查询,保留元数据请求的有限重试。"""
+ response = _request_with_retry(
+ cli,
+ "GET",
+ task["status_url"],
+ stage="task polling",
+ headers=self._headers(),
+ )
+ return self._snapshot(response, task)
- async def parse_async(
+ def _upload_input(
self,
- path: str | Path,
- *,
- page_range: str = "",
- status_callback: Callable[[ApiJobStatus], None] | None = None,
- duration_callback: Callable[[float], None] | None = None,
- ) -> ParseResult:
- """异步解析;状态和耗时回调均属于本次调用,不在复用的 parser 实例上保存。"""
- page_range = normalize_page_range_input(page_range)
- file_path = Path(path)
- if not file_path.exists():
- raise FileNotFoundError(file_path)
-
- payload = self._build_payload(await self._async_build_source(file_path), page_range)
- job = await self._async_do_parse(payload, status_callback=status_callback)
- _notify_job_duration(job, duration_callback)
- return await self._async_build_result(job, file_path.name)
+ cli: httpx.Client,
+ task: dict[str, Any],
+ path: Path,
+ callback: Callable[[ApiTaskStatus], None] | None,
+ ) -> dict[str, Any]:
+ """首次上传失败先核实是否已确认,再重新打开文件流;缓存命中完全不传字节。"""
+ for attempt in range(1, _TRANSPORT_MAX_ATTEMPTS + 1):
+ if task["files"][0].get("upload_status") != "pending":
+ return task
+ links = task.get("uploads", [])
+ link = next((item for item in links if isinstance(item, dict) and item.get("client_file_id") == "file"), None)
+ if not link or link.get("upload_method") != "PUT":
+ raise _V1APIError("invalid_response", "Missing task upload link")
+ url = self._resolve(link["upload_url"])
+ headers = _same_origin_upload_headers(self._base_url, url, link.get("upload_headers", {}), self._headers())
+ try:
+ response = _request_once(
+ cli,
+ "PUT",
+ url,
+ stage="upload content",
+ headers=headers,
+ content=_file_chunks(path),
+ )
+ if response.status_code >= 400:
+ self._check(response)
+ except (_APITransportError, _V1APIError) as exc:
+ task = self._query(cli, task)
+ _notify_job_status(task, callback)
+ if task["files"][0].get("upload_status") != "pending":
+ return task
+ if isinstance(exc, _V1APIError) and response.status_code not in {408, 429, 500, 502, 503, 504}:
+ raise
+ if attempt == _TRANSPORT_MAX_ATTEMPTS:
+ raise
+ time.sleep(_transport_retry_delay(attempt))
+ continue
+ task = self._query(cli, task)
+ _notify_job_status(task, callback)
+ if task["files"][0].get("upload_status") != "pending":
+ return task
+ raise _V1APIError("upload_failed", "Upload was not confirmed")
+
+ def _download_task(self, cli: httpx.Client, task: dict[str, Any], target: Path) -> None:
+ """流式下载和校验完整包;重定向逐跳选择认证,签名失效后重新查询任务。"""
+ for attempt in range(1, _TRANSPORT_MAX_ATTEMPTS + 1):
+ url = self._resolve(task["files"][0]["result_url"])
+ try:
+ for _ in range(6):
+ headers = _same_origin_upload_headers(self._base_url, url, {}, self._headers())
+ with cli.stream("GET", url, headers=headers, follow_redirects=False) as response:
+ if response.status_code in {301, 302, 303, 307, 308}:
+ location = response.headers.get("location")
+ if not location:
+ raise _V1APIError("invalid_response", "Download redirect is missing Location")
+ url = self._resolve(location, base=url)
+ continue
+ if response.status_code >= 400:
+ response.read()
+ self._check(response)
+ received = 0
+ with target.open("wb") as output:
+ for chunk in response.iter_bytes():
+ output.write(chunk)
+ received += len(chunk)
+ _check_download_length(response, received)
+ with zipfile.ZipFile(target) as archive:
+ if archive.testzip() is not None:
+ raise zipfile.BadZipFile("Corrupt ZIP member")
+ return
+ raise _V1APIError("invalid_response", "Too many download redirects")
+ except (httpx.TransportError, zipfile.BadZipFile, _V1APIError) as exc:
+ target.unlink(missing_ok=True)
+ if attempt == _TRANSPORT_MAX_ATTEMPTS:
+ if isinstance(exc, httpx.TransportError):
+ raise _APITransportError(stage="output download", method="GET", attempts=attempt, cause=exc) from exc
+ if isinstance(exc, zipfile.BadZipFile):
+ raise _V1APIError("invalid_zip_output", "Invalid or incomplete result ZIP") from exc
+ raise
+ time.sleep(_transport_retry_delay(attempt))
+ task = self._query(cli, task)
+ _raise_for_terminal_job_error(task)
def get_usage(self) -> dict[str, Any]:
- """Return usage and limits for the configured API identity."""
+ """查询现有用量接口,保持同步包装与 Doclib 的返回结构。"""
with httpx.Client(timeout=httpx.Timeout(30, connect=10), trust_env=self._trust_env) as cli:
response = _request_with_retry(
cli,
@@ -202,396 +362,157 @@ class MinerUApiParser(DocumentParser):
)
return self._check(response)
- async def get_usage_async(self) -> dict[str, Any]:
- """Asynchronous version of :meth:`get_usage`."""
- async with httpx.AsyncClient(timeout=httpx.Timeout(30, connect=10), trust_env=self._trust_env) as cli:
- response = await _async_request_with_retry(
- cli,
- "GET",
- f"{self._base_url}/v1/usage",
- stage="usage query",
- headers=self._headers(),
- )
- return self._check(response)
-
- # ── payload construction ─────────────────────────────────────────
-
- def _build_source(self, file_path: Path) -> dict[str, Any]:
- if self._supports_local_source():
- source = {"type": "local", "path": str(file_path)}
- else:
- # upload flow: create → PUT → complete → file_id
- file_id = self._upload(file_path)
- source = {"type": "file_id", "file_id": file_id}
- return source
-
- async def _async_build_source(self, file_path: Path) -> dict[str, Any]:
- if await self._async_supports_local_source():
- source = {"type": "local", "path": str(file_path)}
- else:
- file_id = await self._async_upload(file_path)
- source = {"type": "file_id", "file_id": file_id}
- return source
-
- def _build_payload(self, source: dict[str, Any], page_range: str) -> dict[str, Any]:
- page_range = normalize_page_range_input(page_range)
- file_entry: dict[str, Any] = {"source": source}
- if page_range:
- file_entry["page_range"] = page_range
-
- payload: dict[str, Any] = {
- "files": [file_entry],
- "output_formats": self._output_formats(),
- }
- if self.tier is not None:
- payload["tier"] = self.tier
- if self.ocr_mode is not None:
- payload["ocr_mode"] = self.ocr_mode
- return payload
-
- def _supports_local_source(self) -> bool:
- if not self._local:
- return False
- return "local" in self._get_source_features()
-
- async def _async_supports_local_source(self) -> bool:
- if not self._local:
- return False
- return "local" in await self._async_get_source_features()
-
- def _get_source_features(self) -> set[str]:
- if self._source_features is not None:
- return self._source_features
- with httpx.Client(timeout=httpx.Timeout(30, connect=10), trust_env=self._trust_env) as cli:
- r = _request_with_retry(
- cli,
- "GET",
- f"{self._base_url}/v1/health",
- stage="health discovery",
- headers=self._headers(),
- )
- health = self._check(r)
- self._source_features = _extract_feature_list(health, "sources")
- return self._source_features
-
- async def _async_get_source_features(self) -> set[str]:
- if self._source_features is not None:
- return self._source_features
- async with httpx.AsyncClient(timeout=httpx.Timeout(30, connect=10), trust_env=self._trust_env) as cli:
- r = await _async_request_with_retry(
- cli,
- "GET",
- f"{self._base_url}/v1/health",
- stage="health discovery",
- headers=self._headers(),
- )
- health = self._check(r)
- self._source_features = _extract_feature_list(health, "sources")
- return self._source_features
-
- def _output_formats(self) -> list[str]:
- if self.include_model_output or self.include_images:
- return ["zip"]
- return ["middle_json"]
-
- # ── HTTP helpers ─────────────────────────────────────────────────
-
- def _headers(self) -> dict[str, str]:
- h: dict[str, str] = {}
- if self._api_key:
- h["Authorization"] = f"Bearer {self._api_key}"
- return h
-
- @staticmethod
- def _is_localhost(base: str) -> bool:
- return _is_local_network_url(base)
-
- @staticmethod
- def _check(r: Any) -> dict[str, Any]:
- data: dict[str, Any] = {}
- try:
- loaded = r.json()
- if isinstance(loaded, dict):
- data = loaded
- except Exception:
- data = {}
- if r.status_code >= 400:
- _raise_for_http_error(r.status_code, data, str(getattr(r, "text", "")))
- if not data:
- return {} # e.g. OSS PUT returns 200 with empty body
- if "error" in data:
- err = data["error"]
- if isinstance(err, dict):
- raise _V1APIError(
- str(err.get("code") or "unknown"),
- str(err.get("message") or err),
- param=str(err["param"]) if err.get("param") is not None else None,
- )
- raise _V1APIError("unknown", str(err))
- return data
-
- # ── upload ───────────────────────────────────────────────────────
-
- def _upload(self, file_path: Path) -> str:
- size = file_path.stat().st_size
- sha = _sha256_file(file_path)
-
- with httpx.Client(timeout=httpx.Timeout(120, connect=30), trust_env=self._trust_env) as cli:
- r = _request_once(
+ async def parse_async(
+ self,
+ path: str | Path,
+ *,
+ page_range: str = "",
+ status_callback: Callable[[ApiTaskStatus], None] | None = None,
+ duration_callback: Callable[[float], None] | None = None,
+ ) -> ParseResult:
+ """创建单文件任务、按需上传并下载完整 ZIP,回调对应真实服务端阶段。"""
+ file_path = Path(path)
+ payload = await asyncio.to_thread(self._payload, file_path, normalize_page_range_input(page_range))
+ async with httpx.AsyncClient(
+ timeout=httpx.Timeout(120, connect=30), trust_env=self._trust_env, follow_redirects=False
+ ) as cli:
+ response = await _async_request_once(
cli,
"POST",
- f"{self._base_url}/v1/uploads",
- stage="upload creation",
+ f"{self._base_url}/v1/tasks",
+ stage="task submission",
headers=self._headers(),
- json_body={
- "filename": file_path.name,
- "bytes": size,
- "mime_type": mime_type_for_extension(file_path),
- "purpose": "parse",
- "sha256sum": sha,
- },
- )
- resp = self._check(r)
- if resp.get("status") == "completed":
- return resp["file"]["id"]
-
- upload_url = resp["upload_url"]
- upload_headers = resp.get("upload_headers", {})
- if upload_url.startswith("/"):
- upload_url = f"{self._base_url}{upload_url}"
- request_headers = _same_origin_upload_headers(
- self._base_url,
- upload_url,
- upload_headers,
- self._headers(),
+ json_body=payload,
)
- with file_path.open("rb") as fh:
- r2 = _request_with_retry(
- cli,
- "PUT",
- upload_url,
- stage="upload content",
- headers=request_headers,
- content=fh.read(),
+ task = self._snapshot(response)
+ _notify_job_status(task, status_callback)
+ task = await self._async_upload_input(cli, task, file_path, status_callback)
+ for _ in range(_POLL_MAX_ATTEMPTS):
+ if task["status"] in _TERMINAL_JOB_STATUSES:
+ break
+ await asyncio.sleep(_POLL_INTERVAL_SECONDS)
+ task = await self._async_query(cli, task)
+ _notify_job_status(task, status_callback)
+ if task["status"] not in _TERMINAL_JOB_STATUSES:
+ raise _V1APIError("timeout", f"Task {task['task_id']} did not complete within timeout")
+ _raise_for_terminal_job_error(task)
+ if task["files"][0].get("status") != "completed":
+ raise _V1APIError("parse_failed", "Task did not complete its input file")
+ _notify_job_duration(task, duration_callback)
+ with tempfile.TemporaryDirectory(prefix="mineru-api-client-") as directory:
+ target = Path(directory) / "result.zip"
+ await self._async_download_task(cli, task, target)
+ return _parse_result_from_zip_bytes(
+ target, include_images=self.include_images, include_model_output=self.include_model_output
)
- self._check(r2)
-
- resp3 = self._complete_upload(cli, str(resp["id"]))
- return resp3["file"]["id"]
- async def _async_upload(self, file_path: Path) -> str:
- size = file_path.stat().st_size
- sha = _sha256_file(file_path)
-
- async with httpx.AsyncClient(timeout=httpx.Timeout(120, connect=30), trust_env=self._trust_env) as cli:
- r = await _async_request_once(
- cli,
- "POST",
- f"{self._base_url}/v1/uploads",
- stage="upload creation",
- headers=self._headers(),
- json_body={
- "filename": file_path.name,
- "bytes": size,
- "mime_type": mime_type_for_extension(file_path),
- "purpose": "parse",
- "sha256sum": sha,
- },
- )
- resp = self._check(r)
- if resp.get("status") == "completed":
- return resp["file"]["id"]
-
- upload_url = resp["upload_url"]
- upload_headers = resp.get("upload_headers", {})
- if upload_url.startswith("/"):
- upload_url = f"{self._base_url}{upload_url}"
- request_headers = _same_origin_upload_headers(
- self._base_url,
- upload_url,
- upload_headers,
- self._headers(),
- )
- data = file_path.read_bytes()
- r2 = await _async_request_with_retry(
- cli,
- "PUT",
- upload_url,
- stage="upload content",
- headers=request_headers,
- content=data,
- )
- self._check(r2)
-
- resp3 = await self._async_complete_upload(cli, str(resp["id"]))
- return resp3["file"]["id"]
+ async def _async_query(self, cli: httpx.AsyncClient, task: dict[str, Any]) -> dict[str, Any]:
+ """按服务返回的同源状态链接查询,保留元数据请求的有限重试。"""
+ response = await _async_request_with_retry(
+ cli,
+ "GET",
+ task["status_url"],
+ stage="task polling",
+ headers=self._headers(),
+ )
+ return self._snapshot(response, task)
- def _complete_upload(self, cli: httpx.Client, upload_id: str) -> dict[str, Any]:
- complete_url = f"{self._base_url}/v1/uploads/{upload_id}/complete"
- last_transport_error: _APITransportError | None = None
+ async def _async_upload_input(
+ self,
+ cli: httpx.AsyncClient,
+ task: dict[str, Any],
+ path: Path,
+ callback: Callable[[ApiTaskStatus], None] | None,
+ ) -> dict[str, Any]:
+ """首次上传失败先核实是否已确认,再重新打开文件流;缓存命中完全不传字节。"""
for attempt in range(1, _TRANSPORT_MAX_ATTEMPTS + 1):
+ if task["files"][0].get("upload_status") != "pending":
+ return task
+ links = task.get("uploads", [])
+ link = next((item for item in links if isinstance(item, dict) and item.get("client_file_id") == "file"), None)
+ if not link or link.get("upload_method") != "PUT":
+ raise _V1APIError("invalid_response", "Missing task upload link")
+ url = self._resolve(link["upload_url"])
+ headers = _same_origin_upload_headers(self._base_url, url, link.get("upload_headers", {}), self._headers())
try:
- response = _request_once(
+ response = await _async_request_once(
cli,
- "POST",
- complete_url,
- stage="upload completion",
- headers=self._headers(),
+ "PUT",
+ url,
+ stage="upload content",
+ headers=headers,
+ content=_async_file_chunks(path),
)
- return self._check(response)
- except _APITransportError as exc:
- last_transport_error = exc
- except _V1APIError as exc:
- if exc.code != "upload_already_terminal":
+ if response.status_code >= 400:
+ self._check(response)
+ except (_APITransportError, _V1APIError) as exc:
+ task = await self._async_query(cli, task)
+ _notify_job_status(task, callback)
+ if task["files"][0].get("upload_status") != "pending":
+ return task
+ if isinstance(exc, _V1APIError) and response.status_code not in {408, 429, 500, 502, 503, 504}:
raise
-
- upload = self._get_upload(cli, upload_id)
- if upload.get("status") == "completed":
- return upload
- if attempt < _TRANSPORT_MAX_ATTEMPTS:
- time.sleep(_transport_retry_delay(attempt))
-
- if last_transport_error is not None:
- raise last_transport_error
- raise _V1APIError("upload_not_ready", f"Upload {upload_id} did not reach completed status")
-
- async def _async_complete_upload(self, cli: httpx.AsyncClient, upload_id: str) -> dict[str, Any]:
- complete_url = f"{self._base_url}/v1/uploads/{upload_id}/complete"
- last_transport_error: _APITransportError | None = None
+ if attempt == _TRANSPORT_MAX_ATTEMPTS:
+ raise
+ await asyncio.sleep(_transport_retry_delay(attempt))
+ continue
+ task = await self._async_query(cli, task)
+ _notify_job_status(task, callback)
+ if task["files"][0].get("upload_status") != "pending":
+ return task
+ raise _V1APIError("upload_failed", "Upload was not confirmed")
+
+ async def _async_download_task(self, cli: httpx.AsyncClient, task: dict[str, Any], target: Path) -> None:
+ """流式下载和校验完整包;重定向逐跳选择认证,签名失效后重新查询任务。"""
for attempt in range(1, _TRANSPORT_MAX_ATTEMPTS + 1):
+ url = self._resolve(task["files"][0]["result_url"])
try:
- response = await _async_request_once(
- cli,
- "POST",
- complete_url,
- stage="upload completion",
- headers=self._headers(),
- )
- return self._check(response)
- except _APITransportError as exc:
- last_transport_error = exc
- except _V1APIError as exc:
- if exc.code != "upload_already_terminal":
+ for _ in range(6):
+ headers = _same_origin_upload_headers(self._base_url, url, {}, self._headers())
+ async with cli.stream("GET", url, headers=headers, follow_redirects=False) as response:
+ if response.status_code in {301, 302, 303, 307, 308}:
+ location = response.headers.get("location")
+ if not location:
+ raise _V1APIError("invalid_response", "Download redirect is missing Location")
+ url = self._resolve(location, base=url)
+ continue
+ if response.status_code >= 400:
+ await response.aread()
+ self._check(response)
+ received = 0
+ with target.open("wb") as output:
+ async for chunk in response.aiter_bytes():
+ await asyncio.to_thread(output.write, chunk)
+ received += len(chunk)
+ _check_download_length(response, received)
+ with zipfile.ZipFile(target) as archive:
+ if archive.testzip() is not None:
+ raise zipfile.BadZipFile("Corrupt ZIP member")
+ return
+ raise _V1APIError("invalid_response", "Too many download redirects")
+ except (httpx.TransportError, zipfile.BadZipFile, _V1APIError) as exc:
+ target.unlink(missing_ok=True)
+ if attempt == _TRANSPORT_MAX_ATTEMPTS:
+ if isinstance(exc, httpx.TransportError):
+ raise _APITransportError(stage="output download", method="GET", attempts=attempt, cause=exc) from exc
+ if isinstance(exc, zipfile.BadZipFile):
+ raise _V1APIError("invalid_zip_output", "Invalid or incomplete result ZIP") from exc
raise
-
- upload = await self._async_get_upload(cli, upload_id)
- if upload.get("status") == "completed":
- return upload
- if attempt < _TRANSPORT_MAX_ATTEMPTS:
await asyncio.sleep(_transport_retry_delay(attempt))
+ task = await self._async_query(cli, task)
+ _raise_for_terminal_job_error(task)
- if last_transport_error is not None:
- raise last_transport_error
- raise _V1APIError("upload_not_ready", f"Upload {upload_id} did not reach completed status")
-
- def _get_upload(self, cli: httpx.Client, upload_id: str) -> dict[str, Any]:
- response = _request_with_retry(
- cli,
- "GET",
- f"{self._base_url}/v1/uploads/{upload_id}",
- stage="upload status",
- headers=self._headers(),
- )
- return self._check(response)
-
- async def _async_get_upload(self, cli: httpx.AsyncClient, upload_id: str) -> dict[str, Any]:
- response = await _async_request_with_retry(
- cli,
- "GET",
- f"{self._base_url}/v1/uploads/{upload_id}",
- stage="upload status",
- headers=self._headers(),
- )
- return self._check(response)
-
- # ── parse execution ──────────────────────────────────────────────
-
- def _do_parse(
- self, payload: dict[str, Any], *, status_callback: Callable[[ApiJobStatus], None] | None = None
- ) -> dict[str, Any]:
- """提交同步任务,并从提交响应开始通知实际状态。"""
- with httpx.Client(timeout=httpx.Timeout(120, connect=30), trust_env=self._trust_env) as cli:
- r = _request_once(
- cli,
- "POST",
- f"{self._base_url}/v1/parse/jobs",
- stage="job submission",
- headers=self._headers(),
- json_body=payload,
- )
- job = self._check(r)
- _notify_job_status(job, status_callback)
- if job.get("status") not in _TERMINAL_JOB_STATUSES:
- job = self._poll(cli, job["job_id"], status_callback=status_callback)
- return job
-
- async def _async_do_parse(
- self, payload: dict[str, Any], *, status_callback: Callable[[ApiJobStatus], None] | None = None
- ) -> dict[str, Any]:
- """提交异步任务,并从提交响应开始通知实际状态。"""
- async with httpx.AsyncClient(timeout=httpx.Timeout(120, connect=30), trust_env=self._trust_env) as cli:
- r = await _async_request_once(
- cli,
- "POST",
- f"{self._base_url}/v1/parse/jobs",
- stage="job submission",
- headers=self._headers(),
- json_body=payload,
- )
- job = self._check(r)
- _notify_job_status(job, status_callback)
- if job.get("status") not in _TERMINAL_JOB_STATUSES:
- job = await self._async_poll(cli, job["job_id"], status_callback=status_callback)
- return job
-
- def _poll(self, cli: Any, job_id: str, *, status_callback: Callable[[ApiJobStatus], None] | None = None) -> dict[str, Any]:
- """保持每秒一次的同步轮询,并通知每次响应中的任务状态。"""
- for _ in range(_POLL_MAX_ATTEMPTS): # max 1 hour
- time.sleep(_POLL_INTERVAL_SECONDS)
- r = _request_with_retry(
- cli,
- "GET",
- f"{self._base_url}/v1/parse/jobs/{job_id}",
- stage="job polling",
- headers=self._headers(),
- )
- job = self._check(r)
- _notify_job_status(job, status_callback)
- if job.get("status") in _TERMINAL_JOB_STATUSES:
- return job
- raise _V1APIError("timeout", f"Job {job_id} did not complete within timeout")
-
- async def _async_poll(
- self, cli: Any, job_id: str, *, status_callback: Callable[[ApiJobStatus], None] | None = None
- ) -> dict[str, Any]:
- """保持每秒一次的异步轮询,状态刷新不额外增加网络请求。"""
- for _ in range(_POLL_MAX_ATTEMPTS):
- await asyncio.sleep(_POLL_INTERVAL_SECONDS)
- r = await _async_request_with_retry(
+ async def get_usage_async(self) -> dict[str, Any]:
+ """查询现有用量接口,保持同步包装与 Doclib 的返回结构。"""
+ async with httpx.AsyncClient(timeout=httpx.Timeout(30, connect=10), trust_env=self._trust_env) as cli:
+ response = await _async_request_with_retry(
cli,
"GET",
- f"{self._base_url}/v1/parse/jobs/{job_id}",
- stage="job polling",
+ f"{self._base_url}/v1/usage",
+ stage="usage query",
headers=self._headers(),
)
- job = self._check(r)
- _notify_job_status(job, status_callback)
- if job.get("status") in _TERMINAL_JOB_STATUSES:
- return job
- raise _V1APIError("timeout", f"Job {job_id} did not complete within timeout")
-
- # ── build ParseResult ────────────────────────────────────────────
-
- def _build_result(self, job: dict[str, Any], file_name: str) -> ParseResult:
- return _parse_result_from_job(job, file_name, self)
-
- async def _async_build_result(self, job: dict[str, Any], file_name: str) -> ParseResult:
- return await _async_parse_result_from_job(job, file_name, self)
-
-
-# ═══════════════════════════════════════════════════════════════════════
-# helpers
-# ═══════════════════════════════════════════════════════════════════════
+ return self._check(response)
def _request_once(
@@ -602,7 +523,7 @@ def _request_once(
stage: str,
headers: dict[str, str],
json_body: dict[str, Any] | None = None,
- content: bytes | None = None,
+ content: bytes | Iterator[bytes] | AsyncIterator[bytes] | None = None,
) -> httpx.Response:
return _request(
client,
@@ -624,7 +545,7 @@ def _request_with_retry(
stage: str,
headers: dict[str, str],
json_body: dict[str, Any] | None = None,
- content: bytes | None = None,
+ content: bytes | Iterator[bytes] | AsyncIterator[bytes] | None = None,
) -> httpx.Response:
return _request(
client,
@@ -646,7 +567,7 @@ def _request(
stage: str,
headers: dict[str, str],
json_body: dict[str, Any] | None,
- content: bytes | None,
+ content: bytes | Iterator[bytes] | AsyncIterator[bytes] | None,
max_attempts: int,
) -> httpx.Response:
for attempt in range(1, max_attempts + 1):
@@ -675,7 +596,7 @@ async def _async_request_once(
stage: str,
headers: dict[str, str],
json_body: dict[str, Any] | None = None,
- content: bytes | None = None,
+ content: bytes | Iterator[bytes] | AsyncIterator[bytes] | None = None,
) -> httpx.Response:
return await _async_request(
client,
@@ -697,7 +618,7 @@ async def _async_request_with_retry(
stage: str,
headers: dict[str, str],
json_body: dict[str, Any] | None = None,
- content: bytes | None = None,
+ content: bytes | Iterator[bytes] | AsyncIterator[bytes] | None = None,
) -> httpx.Response:
return await _async_request(
client,
@@ -719,7 +640,7 @@ async def _async_request(
stage: str,
headers: dict[str, str],
json_body: dict[str, Any] | None,
- content: bytes | None,
+ content: bytes | Iterator[bytes] | AsyncIterator[bytes] | None,
max_attempts: int,
) -> httpx.Response:
for attempt in range(1, max_attempts + 1):
@@ -747,7 +668,7 @@ def _send_request(
*,
headers: dict[str, str],
json_body: dict[str, Any] | None,
- content: bytes | None,
+ content: bytes | Iterator[bytes] | AsyncIterator[bytes] | None,
) -> httpx.Response:
if method == "GET":
return client.get(url, headers=headers)
@@ -765,7 +686,7 @@ async def _async_send_request(
*,
headers: dict[str, str],
json_body: dict[str, Any] | None,
- content: bytes | None,
+ content: bytes | Iterator[bytes] | AsyncIterator[bytes] | None,
) -> httpx.Response:
if method == "GET":
return await client.get(url, headers=headers)
@@ -784,60 +705,39 @@ def _transport_retry_delay(attempt: int) -> float:
return exponential * (0.5 + random.random())
-def _sha256_file(path: Path) -> str:
- h = hashlib.sha256()
- with path.open("rb") as fh:
- while chunk := fh.read(65536):
- h.update(chunk)
- return h.hexdigest()
-
-
-def _extract_feature_list(health: dict[str, Any], name: str) -> set[str]:
- features = health.get("features")
- if not isinstance(features, dict):
- return set()
- values = features.get(name)
- if not isinstance(values, list):
- return set()
- return {item for item in values if isinstance(item, str)}
-
-
-def _parse_result_from_job(job: dict[str, Any], file_name: str, parser: MinerUApiParser) -> ParseResult:
- _raise_for_terminal_job_error(job)
- files = job.get("files", [])
- outputs: dict[str, Any] = {}
- if files and files[0].get("output_files"):
- outputs = files[0]["output_files"]
-
- if _uses_zip_output(parser):
- return _parse_result_from_zip_output(parser, outputs)
-
- mid_json = _download_json(parser, outputs)
- result = _parse_result_from_middle_json(mid_json)
- return result
+def _check_download_length(response: httpx.Response, received: int) -> None:
+ """未压缩响应必须完整接收声明的长度,不能交付恰好仍可解压的截断包。"""
+ raw = response.headers.get("content-length")
+ if raw is None or response.headers.get("content-encoding", "identity") != "identity":
+ return
+ try:
+ complete = int(raw) == received
+ except ValueError:
+ complete = False
+ if not complete:
+ raise httpx.RemoteProtocolError("Incomplete result download")
-async def _async_parse_result_from_job(job: dict[str, Any], file_name: str, parser: MinerUApiParser) -> ParseResult:
- _raise_for_terminal_job_error(job)
- files = job.get("files", [])
- outputs: dict[str, Any] = {}
- if files and files[0].get("output_files"):
- outputs = files[0]["output_files"]
+def _file_chunks(path: Path) -> Iterator[bytes]:
+ """每次传输重新打开文件,避免重试复用已消费的流。"""
+ with path.open("rb") as source:
+ while chunk := source.read(1024 * 1024):
+ yield chunk
- if _uses_zip_output(parser):
- return _parse_result_from_zip_bytes(
- await _async_download_zip_output(parser, outputs),
- include_images=parser.include_images,
- include_model_output=parser.include_model_output,
- )
- mid_json = await _async_download_json(parser, outputs)
- result = _parse_result_from_middle_json(mid_json)
- return result
+async def _async_file_chunks(path: Path) -> AsyncIterator[bytes]:
+ """异步上传逐块读取文件,不把整个输入缓冲到内存。"""
+ with path.open("rb") as source:
+ while chunk := await asyncio.to_thread(source.read, 1024 * 1024):
+ yield chunk
-def _uses_zip_output(parser: MinerUApiParser) -> bool:
- return parser.include_model_output or parser.include_images
+def _sha256_file(path: Path) -> str:
+ h = hashlib.sha256()
+ with path.open("rb") as fh:
+ while chunk := fh.read(65536):
+ h.update(chunk)
+ return h.hexdigest()
def _extract_model_output_from_archive(archive: zipfile.ZipFile) -> Any | None:
@@ -868,42 +768,15 @@ def _extract_model_output_from_archive(archive: zipfile.ZipFile) -> Any | None:
raise _V1APIError("invalid_model_output", f"Invalid Model JSON document: {exc}") from exc
-def _download_zip_output(parser: MinerUApiParser, outputs: dict[str, Any]) -> bytes:
- """下载 zip-only 模式的唯一结果包;缺失 zip 时给出明确错误。"""
- zip_ref = outputs.get("zip")
- if not isinstance(zip_ref, dict):
- available = ", ".join(sorted(outputs)) or "none"
- raise _V1APIError("missing_zip_output", f"Parse job did not return zip output; available outputs: {available}")
- return _download_bytes(parser, zip_ref)
-
-
-async def _async_download_zip_output(parser: MinerUApiParser, outputs: dict[str, Any]) -> bytes:
- """异步下载 zip-only 模式的唯一结果包;缺失 zip 时给出明确错误。"""
- zip_ref = outputs.get("zip")
- if not isinstance(zip_ref, dict):
- available = ", ".join(sorted(outputs)) or "none"
- raise _V1APIError("missing_zip_output", f"Parse job did not return zip output; available outputs: {available}")
- return await _async_download_bytes(parser, zip_ref)
-
-
-def _parse_result_from_zip_output(parser: MinerUApiParser, outputs: dict[str, Any]) -> ParseResult:
- """从 v1 API 自包含 zip 输出恢复 ParseResult,避免逐个下载 middle_json 和图片。"""
- return _parse_result_from_zip_bytes(
- _download_zip_output(parser, outputs),
- include_images=parser.include_images,
- include_model_output=parser.include_model_output,
- )
-
-
def _parse_result_from_zip_bytes(
- zip_bytes: bytes,
+ zip_bytes: bytes | Path,
*,
include_images: bool,
include_model_output: bool,
) -> ParseResult:
"""解析自包含 zip 包:读取 middle_json、图片 sidecar 和可选 model_output。"""
try:
- with zipfile.ZipFile(io.BytesIO(zip_bytes)) as archive:
+ with zipfile.ZipFile(zip_bytes if isinstance(zip_bytes, Path) else io.BytesIO(zip_bytes)) as archive:
mid_json = _read_middle_json_from_zip(archive)
if include_images:
images = _read_image_sidecars_from_zip(archive, mid_json)
@@ -1052,78 +925,6 @@ def _inline_image_sidecars(mid_json: Any, images: dict[str, bytes]) -> None:
_inline_image_sidecars(item, images)
-def _download_json(parser: MinerUApiParser, outputs: dict[str, Any]) -> dict[str, Any]:
- ref = outputs.get("middle_json")
- if not ref:
- available = ", ".join(sorted(outputs)) or "none"
- raise _V1APIError(
- "missing_middle_json_output",
- f"Parse job did not return middle_json output; available outputs: {available}",
- )
- raw = _download_bytes(parser, ref)
- try:
- loaded = json.loads(raw)
- except json.JSONDecodeError as exc:
- raise _V1APIError("invalid_middle_json_output", f"middle_json output is not valid JSON: {exc}") from exc
- if not isinstance(loaded, dict):
- raise _V1APIError("invalid_middle_json_output", "middle_json output must be a JSON object with pages or pdf_info")
- return loaded
-
-
-async def _async_download_json(parser: MinerUApiParser, outputs: dict[str, Any]) -> dict[str, Any]:
- ref = outputs.get("middle_json")
- if not ref:
- available = ", ".join(sorted(outputs)) or "none"
- raise _V1APIError(
- "missing_middle_json_output",
- f"Parse job did not return middle_json output; available outputs: {available}",
- )
- raw = await _async_download_bytes(parser, ref)
- try:
- loaded = json.loads(raw)
- except json.JSONDecodeError as exc:
- raise _V1APIError("invalid_middle_json_output", f"middle_json output is not valid JSON: {exc}") from exc
- if not isinstance(loaded, dict):
- raise _V1APIError("invalid_middle_json_output", "middle_json output must be a JSON object with pages or pdf_info")
- return loaded
-
-
-def _download_bytes(parser: MinerUApiParser, ref: dict[str, Any]) -> bytes:
- file_id = ref.get("file_id")
- if not file_id:
- raise _V1APIError("invalid_response", "No file_id in output reference")
-
- with httpx.Client(timeout=httpx.Timeout(120, connect=30), follow_redirects=True, trust_env=parser._trust_env) as cli:
- r = _request_with_retry(
- cli,
- "GET",
- f"{parser._base_url}/v1/files/{file_id}/content",
- stage="output download",
- headers=parser._headers(),
- )
- _check_download_response(r)
- return r.content
-
-
-async def _async_download_bytes(parser: MinerUApiParser, ref: dict[str, Any]) -> bytes:
- file_id = ref.get("file_id")
- if not file_id:
- raise _V1APIError("invalid_response", "No file_id in output reference")
-
- async with httpx.AsyncClient(
- timeout=httpx.Timeout(120, connect=30), follow_redirects=True, trust_env=parser._trust_env
- ) as cli:
- r = await _async_request_with_retry(
- cli,
- "GET",
- f"{parser._base_url}/v1/files/{file_id}/content",
- stage="output download",
- headers=parser._headers(),
- )
- _check_download_response(r)
- return r.content
-
-
def _same_origin_upload_headers(
base_url: str,
upload_url: str,
@@ -1138,6 +939,8 @@ def _same_origin_upload_headers(
headers = dict(upload_headers)
if _http_origin(base_url) == _http_origin(upload_url):
headers.update(auth_headers)
+ elif any(key.lower() == "authorization" and value.strip() in auth_headers.values() for key, value in headers.items()):
+ raise _V1APIError("invalid_response", "Refusing to forward the API key to another origin")
return headers
@@ -1147,9 +950,14 @@ def _http_origin(url: str) -> tuple[str, str, int | None]:
try:
port = parsed.port
except ValueError as exc:
- raise _V1APIError("invalid_response", f"Invalid upload URL: {url}") from exc
- if parsed.scheme not in {"http", "https"} or not parsed.hostname:
- raise _V1APIError("invalid_response", f"Invalid upload URL: {url}")
+ raise _V1APIError("invalid_response", "Invalid HTTP URL in API response") from exc
+ if (
+ parsed.scheme not in {"http", "https"}
+ or not parsed.hostname
+ or parsed.username is not None
+ or parsed.password is not None
+ ):
+ raise _V1APIError("invalid_response", "Invalid HTTP URL in API response")
effective_port = port if port is not None else (443 if parsed.scheme == "https" else 80)
return parsed.scheme.lower(), parsed.hostname.lower(), effective_port
@@ -1187,20 +995,13 @@ def _pages_from_middle_json(mid_json: dict[str, Any] | None) -> list[PageInfo]:
def _parse_result_from_middle_json(mid_json: dict[str, Any]) -> ParseResult:
- """把远端 middle_json 恢复为 ParseResult;优先当前协议,临时兼容 3.x 历史协议。"""
+ """严格恢复共享 Middle JSON 2.0,历史缓存兼容仅属于 Doclib 读取边界。"""
if not isinstance(mid_json, dict):
raise _V1APIError("invalid_middle_json_output", "middle_json output must be a JSON object")
try:
return ParseResult.from_dict(mid_json)
except ValueError as exc:
- strict_error = exc
- # 临时:兼容 3.x remote server 的历史 middle_json(pdf_info / 旧 1.0 / 旧 2.0),待其升级到 4.0 后移除。
- from ..backend.postprocess.legacy_middle_json import read_legacy_middle_json
-
- try:
- return ParseResult(middle_json=read_legacy_middle_json(mid_json))
- except ValueError:
- raise _V1APIError("invalid_middle_json_output", str(strict_error)) from strict_error
+ raise _V1APIError("invalid_middle_json_output", str(exc)) from exc
def _raise_for_terminal_job_error(job: dict[str, Any]) -> None:
@@ -1221,7 +1022,7 @@ def _raise_for_terminal_job_error(job: dict[str, Any]) -> None:
raise _V1APIError(code, message, param=param)
raise _V1APIError(
- str(job.get("status")), f"Parse job {job.get('job_id', '<unknown>')} ended with status {job.get('status')}"
+ str(job.get("status")), f"Parse task {job.get('task_id', '<unknown>')} ended with status {job.get('status')}"
)
diff --git a/mineru/parser/api_server.py b/mineru/parser/api_server.py
index fc385db9..17b445b5 100644
--- a/mineru/parser/api_server.py
+++ b/mineru/parser/api_server.py
@@ -6,8 +6,6 @@ from __future__ import annotations
import asyncio
import base64
import hashlib
-import io
-import json
from importlib.resources import files as package_resources
import logging
import os
@@ -18,7 +16,6 @@ import shutil
import tempfile
import threading
import time
-import zipfile
from contextlib import asynccontextmanager
from copy import deepcopy
from dataclasses import dataclass, field
@@ -29,7 +26,6 @@ from urllib.parse import urlparse
import click
import httpx
import uvicorn
-from docvortex.assets import validate_image_sidecar_path
from fastapi import APIRouter, Depends, FastAPI, HTTPException, Path, Query, Request, Response, status
from fastapi.exceptions import RequestValidationError
from fastapi.openapi.docs import get_redoc_html, get_swagger_ui_html, get_swagger_ui_oauth2_redirect_html
@@ -60,7 +56,6 @@ from ..utils.logger import configure_global_log_level
from ..utils.stdio import configure_standard_streams
from ..version import __version__
from . import parse_async
-from .base import ParseResult
from .page_range import (
PAGE_RANGE_DESCRIPTION,
normalize_page_range_input,
@@ -79,57 +74,13 @@ from .tier import (
ensure_tier_runtime_dependencies,
runtime_options_for_tier,
)
-from .writer import DataWriter
from ..task_types import TASK_OUTPUT_FORMATS, TaskOutputFormat
_DEFAULT_API_SERVER_TIER: ServerTier = "standard"
-_MAX_INLINE_BYTES_DEFAULT = 1024 * 1024
_MAX_FILES_PER_JOB_DEFAULT = 100
_MAX_URL_BYTES_DEFAULT = 209715200
-_LOCAL_PARSE_OUTPUT_FORMATS: tuple[OutputFormat, ...] = (
- "markdown",
- "middle_json",
- "structured_content",
- "zip",
-)
-_BASE_PARSE_SOURCES: tuple[SourceType, ...] = ("file_id", "url", "inline")
-logger = logging.getLogger("mineru.parser.api_server")
-
-
-def _sanitize_surrogates(value: str) -> str:
- return "".join("\ufffd" if 0xD800 <= ord(ch) <= 0xDFFF else ch for ch in value)
-
-
-def _sanitize_json_for_utf8(value: object) -> object:
- if isinstance(value, str):
- return _sanitize_surrogates(value)
- if isinstance(value, list):
- return [_sanitize_json_for_utf8(item) for item in value]
- if isinstance(value, tuple):
- return [_sanitize_json_for_utf8(item) for item in value]
- if isinstance(value, dict):
- return {_sanitize_json_for_utf8(key): _sanitize_json_for_utf8(item) for key, item in value.items()}
- return value
-
-
-def _text_utf8_bytes(value: str) -> bytes:
- return _sanitize_surrogates(value).encode("utf-8")
-
-def _json_utf8_bytes(value: object) -> bytes:
- return json.dumps(_sanitize_json_for_utf8(value), ensure_ascii=False).encode("utf-8")
-
-
-class _ZipDataWriter(DataWriter):
- def __init__(self, archive: zipfile.ZipFile) -> None:
- self._archive = archive
-
- def write(self, path: str, data: bytes) -> None:
- safe_path = validate_image_sidecar_path(path)
- self._archive.writestr(safe_path, data)
-
- def write_string(self, path: str, data: str) -> None:
- self.write(path, _text_utf8_bytes(data))
+logger = logging.getLogger("mineru.parser.api_server")
class ParseServerStartupError(RuntimeError):
@@ -144,31 +95,11 @@ JobStatus = Literal["queued", "running", "completed", "partial", "failed", "canc
FileStatus = Literal["queued", "running", "completed", "failed"]
"""Per-file status within a job."""
-UploadStatus = Literal["pending", "completed", "cancelled", "expired"]
-"""Upload lifecycle states."""
-
-OutputFormat = Literal[
- "markdown",
- "middle_json",
- "structured_content",
- "html",
- "latex",
- "docx",
- "zip",
-]
-"""Output artifact formats."""
-
-OutputFormatReqToken = Literal["html", "latex", "docx"]
-"""Output formats that require a valid API key."""
-
-SourceType = Literal["file_id", "url", "inline", "local"]
-"""File source types."""
-
AccessLevel = Literal["anonymous", "registered"]
"""Account access level determined by API Key."""
-FilePurpose = Literal["parse", "parse_output", "input_image"]
-"""File purpose: source files, parse artifacts, or chat/response input images."""
+FilePurpose = Literal["parse", "parse_output"]
+"""文件用途限定为解析输入或产物。"""
# ── helper ───────────────────────────────────────────────────────────
@@ -302,9 +233,9 @@ def _http_exception_error(exc: HTTPException) -> ErrorDetail:
class HealthFeatures(BaseModel):
model_config = _PYDANTIC_CONFIG
webhook: bool = False
- output_formats: list[OutputFormat] = Field(default_factory=lambda: list(_LOCAL_PARSE_OUTPUT_FORMATS))
task_output_formats: list[TaskOutputFormat] = Field(default_factory=lambda: list(TASK_OUTPUT_FORMATS))
- sources: list[SourceType] = Field(default_factory=lambda: list(_BASE_PARSE_SOURCES))
+ sources: list[Literal["upload", "url"]] = Field(default_factory=lambda: ["upload", "url"])
+ task_model_output: bool = True
class HealthResponse(BaseModel):
@@ -365,45 +296,6 @@ class TierInfoData(TypedDict):
# ── Upload ───────────────────────────────────────────────────────────
-class ExpiresAfter(BaseModel):
- model_config = _PYDANTIC_CONFIG
- anchor: Literal["created_at"] = "created_at"
- seconds: int = Field(default=3600, ge=3600, le=2592000)
-
-
-class CreateUploadRequest(BaseModel):
- model_config = _PYDANTIC_CONFIG
- filename: str = Field(min_length=1)
- bytes: int = Field(gt=0)
- mime_type: str = Field(min_length=1)
- purpose: Literal["parse", "input_image"] = "parse"
- sha256sum: str | None = Field(default=None, pattern=r"^[a-f0-9]{64}$")
- expires_after: ExpiresAfter | None = None
-
-
-class CompleteUploadRequest(BaseModel):
- model_config = _PYDANTIC_CONFIG
- sha256sum: str | None = Field(default=None, pattern=r"^[a-f0-9]{64}$")
-
-
-class UploadResponse(BaseModel):
- model_config = _PYDANTIC_CONFIG
- id: str
- object: Literal["upload"] = "upload"
- bytes: int
- created_at: int
- expires_at: int
- filename: str
- purpose: FilePurpose = "parse"
- mime_type: str
- sha256sum: str | None = None
- status: UploadStatus
- upload_url: str | None = None
- upload_method: Literal["PUT"] | None = None
- upload_headers: dict[str, str] | None = None
- file: FileObjectModel | None = None
-
-
# ── File ─────────────────────────────────────────────────────────────
@@ -440,70 +332,47 @@ class FileListResponse(BaseModel):
# ── Job sources ──────────────────────────────────────────────────────
-class FileIdSource(BaseModel):
+class _StoredSource(BaseModel):
model_config = _PYDANTIC_CONFIG
type: Literal["file_id"] = "file_id"
file_id: str
-class UrlSource(BaseModel):
+class _UrlSource(BaseModel):
model_config = _PYDANTIC_CONFIG
type: Literal["url"] = "url"
url: str
-class InlineSource(BaseModel):
- model_config = _PYDANTIC_CONFIG
- type: Literal["inline"] = "inline"
- name: str = Field(min_length=1)
- data: str = Field(min_length=1)
-
-
-class LocalSource(BaseModel):
- model_config = _PYDANTIC_CONFIG
- type: Literal["local"] = "local"
- path: str = Field(min_length=1)
-
-
-FileSource = Annotated[
- FileIdSource | UrlSource | InlineSource | LocalSource,
+_ExecutionSource = Annotated[
+ _StoredSource | _UrlSource,
Field(discriminator="type"),
]
-class JobFileEntry(BaseModel):
+class _ParseInput(BaseModel):
model_config = _PYDANTIC_CONFIG
- source: FileSource
+ source: _ExecutionSource
page_range: str | None = Field(default=None, description=PAGE_RANGE_DESCRIPTION, examples=["1-5,8,r3-r1", "all"])
-class CallbackConfig(BaseModel):
- model_config = _PYDANTIC_CONFIG
- url: str
- secret: str | None = None
+class _ParseRequest(BaseModel):
+ """Tasks 构造的私有执行请求,不属于公开 HTTP 协议。"""
-
-class CreateJobRequest(BaseModel):
model_config = _PYDANTIC_CONFIG
- files: list[JobFileEntry] = Field(min_length=1, max_length=_MAX_FILES_PER_JOB_DEFAULT)
+ files: list[_ParseInput] = Field(min_length=1, max_length=_MAX_FILES_PER_JOB_DEFAULT)
tier: Tier | None = None
ocr_mode: Literal["auto", "txt", "ocr"] = Field(
default="auto", description="OCR mode for this parse job: auto-detect, native text, or forced OCR."
)
- output_formats: list[str] = ["markdown"]
- callback: CallbackConfig | None = None
+ output_formats: list[TaskOutputFormat] = Field(default_factory=lambda: ["default"])
+ include_model_output: bool = False
# ── Job responses ────────────────────────────────────────────────────
-class JobLinks(BaseModel):
- model_config = _PYDANTIC_CONFIG
- self: str
- cancel: str
-
-
-class JobProgress(BaseModel):
+class TaskProgress(BaseModel):
model_config = _PYDANTIC_CONFIG
completed: int = 0
failed: int = 0
@@ -517,75 +386,16 @@ class FileParseInfo(BaseModel):
parser_version: str | None = None
-class OutputFileRef(BaseModel):
- model_config = _PYDANTIC_CONFIG
- file_id: str
- bytes: int
-
-
-class OutputFiles(BaseModel):
- """Per-file output artifacts. Content is downloaded via the Files API."""
-
- model_config = _PYDANTIC_CONFIG
- markdown: OutputFileRef | None = None
- middle_json: OutputFileRef | None = None
- structured_content: OutputFileRef | None = None
- html: OutputFileRef | None = None
- latex: OutputFileRef | None = None
- docx: OutputFileRef | None = None
- zip: OutputFileRef | None = None
-
-
-class JobFileResult(BaseModel):
+class _ExecutionFile(BaseModel):
model_config = _PYDANTIC_CONFIG
file_id: str | None = None
name: str
page_range: str
status: FileStatus
parse: FileParseInfo | None = None
- output_files: OutputFiles | None = None
error: ErrorDetail | None = None
-class JobAsyncResponse(BaseModel):
- model_config = _PYDANTIC_CONFIG
- job_id: str
- status: JobStatus
- created_at: str # ISO-8601 UTC
- started_at: str | None = None
- finished_at: str | None = None
- tier: Tier
- output_formats: list[OutputFormat]
- access_level: AccessLevel
- progress: JobProgress | None = None
- files: list[JobFileResult]
- links: JobLinks
-
-
-class JobListItem(BaseModel):
- model_config = _PYDANTIC_CONFIG
- job_id: str
- status: JobStatus
- created_at: str
- file_count: int
-
-
-class JobListResponse(BaseModel):
- model_config = _PYDANTIC_CONFIG
- object: Literal["list"] = "list"
- data: list[JobListItem]
- first_id: str | None = None
- last_id: str | None = None
- has_more: bool = False
-
-
-class JobCancelResponse(BaseModel):
- model_config = _PYDANTIC_CONFIG
- job_id: str
- status: Literal["canceled"] = "canceled"
- canceled_at: str
-
-
# ── Usage ────────────────────────────────────────────────────────────
@@ -625,21 +435,6 @@ class UsageResponse(BaseModel):
# ═══════════════════════════════════════════════════════════════════════
-@dataclass
-class _UploadRecord:
- id: str
- filename: str
- bytes: int
- mime_type: str
- created_at: int
- expires_at: int
- purpose: FilePurpose = "parse"
- sha256sum: str | None = None
- status: UploadStatus = "pending"
- file_id: str | None = None # set on complete
- write_lock: asyncio.Lock = field(default_factory=asyncio.Lock)
-
-
@dataclass
class _FileRecord:
id: str
@@ -649,7 +444,6 @@ class _FileRecord:
purpose: FilePurpose
sha256sum: str | None = None
expires_at: int | None = None
- upload_id: str | None = None
job_id: str | None = None
@@ -665,7 +459,6 @@ class FileStore:
self._root = root
self._blobs = root / "blobs"
self._blobs.mkdir(parents=True, exist_ok=True)
- self._uploads: dict[str, _UploadRecord] = {}
self._files: dict[str, _FileRecord] = {}
def claim_directory(self) -> None:
@@ -738,10 +531,6 @@ class FileStore:
for file_id, rec in tuple(self._files.items()):
if rec.expires_at is not None and rec.expires_at <= now:
self._files.pop(file_id)
- for upload_id, rec in tuple(self._uploads.items()):
- if rec.expires_at <= now:
- self._uploads.pop(upload_id)
- (self._blobs / "_uploads" / upload_id).unlink(missing_ok=True)
referenced = {rec.sha256sum for rec in self._files.values()} | set(self._pins)
for prefix in self._blobs.iterdir():
if prefix.is_symlink() or not prefix.is_dir() or not re.fullmatch(r"[0-9a-f]{2}", prefix.name):
@@ -759,23 +548,9 @@ class FileStore:
# 刚写入但尚未注册的文件及当前扫描期间的写入受锁和年龄门槛保护。
if blob.stat().st_mtime + self.retention_seconds <= now:
blob.unlink(missing_ok=True)
- pending = self._blobs / "_uploads"
- if pending.is_dir():
- for blob in pending.iterdir():
- if (
- blob.is_file()
- and re.fullmatch(r"upload_[0-9a-f]{24}", blob.name)
- and blob.name not in self._uploads
- and blob.stat().st_mtime + self.retention_seconds <= now
- ):
- blob.unlink(missing_ok=True)
# ── ID generation ────────────────────────────────────────────────
- @staticmethod
- def _new_upload_id() -> str:
- return "upload_" + secrets.token_hex(12)
-
@staticmethod
def _new_file_id() -> str:
return "file-" + secrets.token_hex(12)
@@ -877,51 +652,6 @@ class FileStore:
self._pins[rec.sha256sum] = self._pins.get(rec.sha256sum, 0) + 1
return self._blob_abs(rec.sha256sum), rec.filename, rec.sha256sum
- async def store_upload_stream(self, upload_id: str, chunks: AsyncIterator[bytes]) -> None:
- """逐块接收独立 Upload,校验完整大小后才原子替换暂存文件。"""
- rec = self.get_upload(upload_id)
- async with rec.write_lock:
- rec = self.get_upload(upload_id)
- if rec.status != "pending":
- _raise_api_error(
- 409, error_type="invalid_request_error", code="upload_already_terminal", message=f"Upload is {rec.status}"
- )
- directory = self._blobs / "_uploads"
- directory.mkdir(parents=True, exist_ok=True)
- temporary = directory / f"{upload_id}.{secrets.token_hex(8)}"
- size = 0
- try:
- with temporary.open("wb") as target:
- async for chunk in chunks:
- size += len(chunk)
- if size > rec.bytes:
- _raise_api_error(
- 413,
- error_type="invalid_request_error",
- code="upload_size_mismatch",
- message=f"Upload expects {rec.bytes} bytes, received more",
- )
- await run_sync(target.write, chunk)
- with self._lock:
- current = self.get_upload(upload_id)
- if current.status != "pending":
- _raise_api_error(
- 409,
- error_type="invalid_request_error",
- code="upload_already_terminal",
- message=f"Upload is {current.status}",
- )
- if size != current.bytes:
- _raise_api_error(
- 413,
- error_type="invalid_request_error",
- code="upload_size_mismatch",
- message=f"Upload expects {current.bytes} bytes, received {size}",
- )
- os.replace(temporary, directory / upload_id)
- finally:
- temporary.unlink(missing_ok=True)
-
def read_blob(self, sha256hex: str) -> bytes:
with self._lock:
p = self._blob_abs(sha256hex)
@@ -929,219 +659,6 @@ class FileStore:
raise FileNotFoundError(sha256hex)
return p.read_bytes()
- # ── uploads ──────────────────────────────────────────────────────
-
- def create_upload(self, req: CreateUploadRequest, *, base_url: str = "") -> UploadResponse:
- with self._lock:
- now = int(time.time())
- expires_seconds = req.expires_after.seconds if req.expires_after else 3600
-
- # sha256sum dedup
- if req.sha256sum and self.blob_exists(req.sha256sum):
- if self._blob_abs(req.sha256sum).stat().st_size != req.bytes:
- _raise_api_error(
- 400,
- error_type="invalid_request_error",
- code="upload_size_mismatch",
- message="Declared size does not match the cached file",
- param="bytes",
- )
- file_id = self._new_file_id()
- self._files[file_id] = _FileRecord(
- id=file_id,
- filename=req.filename,
- bytes=req.bytes,
- created_at=now,
- purpose=req.purpose,
- sha256sum=req.sha256sum,
- expires_at=now + expires_seconds,
- )
- upload_id = self._new_upload_id()
- self._uploads[upload_id] = _UploadRecord(
- id=upload_id,
- filename=req.filename,
- bytes=req.bytes,
- mime_type=req.mime_type,
- created_at=now,
- expires_at=now + expires_seconds,
- purpose=req.purpose,
- sha256sum=req.sha256sum,
- status="completed",
- file_id=file_id,
- )
- return self._make_upload_response(self._uploads[upload_id], base_url=base_url, expose_upload_url=True)
-
- upload_id = self._new_upload_id()
- self._uploads[upload_id] = _UploadRecord(
- id=upload_id,
- filename=req.filename,
- bytes=req.bytes,
- mime_type=req.mime_type,
- created_at=now,
- expires_at=now + expires_seconds,
- purpose=req.purpose,
- sha256sum=req.sha256sum,
- )
- return self._make_upload_response(self._uploads[upload_id], base_url=base_url, expose_upload_url=True)
-
- def get_upload(self, upload_id: str) -> _UploadRecord:
- with self._lock:
- rec = self._uploads.get(upload_id)
- if rec is None:
- _raise_api_error(
- 404,
- error_type="invalid_request_error",
- code="upload_not_found",
- message=f"Upload {upload_id} not found",
- )
- if int(time.time()) >= rec.expires_at:
- rec.status = "expired"
- (self._blobs / "_uploads" / upload_id).unlink(missing_ok=True)
- _raise_api_error(
- 404, error_type="invalid_request_error", code="upload_not_found", message=f"Upload {upload_id} expired"
- )
- return rec
-
- def complete_upload(self, upload_id: str, sha256hex: str | None) -> UploadResponse:
- with self._lock:
- rec = self.get_upload(upload_id)
- if rec.status == "completed":
- _raise_api_error(
- 409,
- error_type="invalid_request_error",
- code="upload_already_terminal",
- message="Upload is already completed",
- )
- if rec.status in ("cancelled", "expired"):
- _raise_api_error(
- 409,
- error_type="invalid_request_error",
- code="upload_already_terminal",
- message=f"Upload is {rec.status}",
- )
-
- # 流式复核已上传内容,不在完成阶段把大文件整体读入内存。
- staged = self._blobs / "_uploads" / upload_id
- if not staged.is_file():
- _raise_api_error(
- 409, error_type="invalid_request_error", code="upload_not_ready", message="No uploaded content"
- )
- actual_size = staged.stat().st_size
- if actual_size != rec.bytes:
- _raise_api_error(
- 413,
- error_type="invalid_request_error",
- code="upload_size_mismatch",
- message=f"Upload expects {rec.bytes} bytes, received {actual_size}",
- )
- hasher = hashlib.sha256()
- with staged.open("rb") as source:
- while chunk := source.read(1024 * 1024):
- hasher.update(chunk)
- actual_sha = hasher.hexdigest()
-
- if sha256hex and sha256hex != actual_sha:
- _raise_api_error(
- 400,
- error_type="invalid_request_error",
- code="file_hash_mismatch",
- message="SHA-256 mismatch",
- )
- if rec.sha256sum and rec.sha256sum != actual_sha:
- _raise_api_error(
- 400,
- error_type="invalid_request_error",
- code="file_hash_mismatch",
- message="SHA-256 mismatch",
- )
-
- # move from upload blob to content-addressed blob
- sha = sha256hex or actual_sha
- if not self.blob_exists(sha):
- destination = self._blob_abs(sha)
- destination.parent.mkdir(parents=True, exist_ok=True)
- os.replace(staged, destination)
- # remove temp upload blob
- (self._blobs / "_uploads" / upload_id).unlink(missing_ok=True)
-
- now = int(time.time())
- file_id = self._new_file_id()
- self._files[file_id] = _FileRecord(
- id=file_id,
- filename=rec.filename,
- bytes=rec.bytes,
- created_at=now,
- purpose=rec.purpose,
- sha256sum=sha,
- expires_at=rec.expires_at,
- upload_id=upload_id,
- )
- rec.status = "completed"
- rec.file_id = file_id
- rec.sha256sum = sha
- return self._make_upload_response(rec)
-
- def cancel_upload(self, upload_id: str) -> UploadResponse:
- with self._lock:
- rec = self.get_upload(upload_id)
- if rec.status in ("completed", "cancelled", "expired"):
- _raise_api_error(
- 409,
- error_type="invalid_request_error",
- code="upload_already_terminal",
- message=f"Upload is {rec.status}",
- )
- rec.status = "cancelled"
- return self._make_upload_response(rec)
-
- def store_upload_data(self, upload_id: str, data: bytes) -> None:
- """Store raw bytes for an upload (before complete)."""
- with self._lock:
- rec = self.get_upload(upload_id)
- if rec.status != "pending":
- _raise_api_error(
- 409,
- error_type="invalid_request_error",
- code="upload_already_terminal",
- message=f"Upload is {rec.status}",
- )
- if len(data) != rec.bytes:
- _raise_api_error(
- 413,
- error_type="invalid_request_error",
- code="upload_size_mismatch",
- message=f"Upload expects {rec.bytes} bytes, received {len(data)}",
- )
- # store temporarily under upload_id in blobs dir
- p = self._blobs / "_uploads" / upload_id
- p.parent.mkdir(parents=True, exist_ok=True)
- p.write_bytes(data)
-
- def _make_upload_response(
- self, rec: _UploadRecord, *, base_url: str = "", expose_upload_url: bool = False
- ) -> UploadResponse:
- with self._lock:
- resp = UploadResponse(
- id=rec.id,
- bytes=rec.bytes,
- created_at=rec.created_at,
- expires_at=rec.expires_at,
- filename=rec.filename,
- mime_type=rec.mime_type,
- purpose=rec.purpose,
- sha256sum=rec.sha256sum,
- status=rec.status,
- )
- if expose_upload_url and rec.status == "pending":
- resp.upload_url = f"{base_url}/v1/uploads/{rec.id}/content"
- resp.upload_method = "PUT"
- resp.upload_headers = {"Content-Type": rec.mime_type}
- elif rec.status == "completed" and rec.file_id and rec.file_id in self._files:
- resp.file = self._make_file_object(self._files[rec.file_id])
- return resp
-
- # ── files ────────────────────────────────────────────────────────
-
def create_file_for_output(
self, filename: str, data: bytes, *, sha256hex: str | None = None, job_id: str | None = None
) -> str:
@@ -1260,14 +777,6 @@ class FileStore:
app_state.file_store = self
-def _build_self_contained_zip_output(result: ParseResult) -> bytes:
- """构建自包含解析 zip,确保只请求 zip 时也能恢复 middle_json、图片和 model_output。"""
- buf = io.BytesIO()
- with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as zf:
- result.save(_ZipDataWriter(zf))
- return buf.getvalue()
-
-
# ═══════════════════════════════════════════════════════════════════════
# DEPENDENCIES
# ═══════════════════════════════════════════════════════════════════════
@@ -1292,63 +801,25 @@ def _resolve_access_level(request: Request) -> AccessLevel:
# ═══════════════════════════════════════════════════════════════════════
-_OUTPUT_FORMATS_LOCAL = set(_LOCAL_PARSE_OUTPUT_FORMATS)
-
-
-def _parse_sources_for_server(*, allow_local_source: bool) -> list[SourceType]:
- sources = list(_BASE_PARSE_SOURCES)
- if allow_local_source:
- sources.append("local")
- return sources
-
-
-def _decode_inline_data(data: str) -> bytes:
- try:
- return base64.b64decode(data, validate=True)
- except Exception as exc:
- raise ValueError("inline source data must be valid base64") from exc
-
-
-def _validate_source_policy(
- source: FileSource,
- *,
- allow_local_source: bool,
- max_inline_bytes: int,
- allow_http_source: bool,
-) -> None:
- if isinstance(source, UrlSource):
- scheme = urlparse(source.url).scheme.lower()
- if scheme == "https":
- return
- if scheme == "http" and allow_http_source:
- return
- if scheme == "http":
- raise ValueError("url source must use https unless --allow-http-source is enabled")
- raise ValueError("url source must use http or https")
- if isinstance(source, InlineSource):
- data = _decode_inline_data(source.data)
- if len(data) > max_inline_bytes:
- raise ValueError(f"inline source exceeds max_inline_bytes ({max_inline_bytes})")
+def _validate_source_policy(source: _ExecutionSource, *, allow_http_source: bool) -> None:
+ """执行层仅接受已校验的源文件引用或遵守服务策略的 URL。"""
+ if isinstance(source, _StoredSource):
return
- if isinstance(source, LocalSource):
- if not allow_local_source:
- raise ValueError("local source is disabled; enable --allow-local-source")
+ scheme = urlparse(source.url).scheme.lower()
+ if scheme == "https" or (scheme == "http" and allow_http_source):
return
+ raise ValueError("url source must use https unless --allow-http-source is enabled")
def _validate_job_source_policy(
- req: CreateJobRequest,
+ req: _ParseRequest,
*,
- allow_local_source: bool,
- max_inline_bytes: int,
allow_http_source: bool,
) -> None:
for index, entry in enumerate(req.files):
try:
_validate_source_policy(
entry.source,
- allow_local_source=allow_local_source,
- max_inline_bytes=max_inline_bytes,
allow_http_source=allow_http_source,
)
except ValueError as exc:
@@ -1388,27 +859,24 @@ class _JobRecord:
started_at: str | None = None
finished_at: str | None = None
tier: Tier = _DEFAULT_API_SERVER_TIER
- output_formats: list[OutputFormat] = None # type: ignore[assignment]
- progress: JobProgress | None = None
- files: list[JobFileResult] = None # type: ignore[assignment]
- links: JobLinks | None = None
+ progress: TaskProgress | None = None
+ files: list[_ExecutionFile] = None # type: ignore[assignment]
input_blobs: dict[str, str] = field(default_factory=dict)
input_specs: dict[int, _JobInputSpec] = field(default_factory=dict)
input_metadata: dict[int, _JobInputMetadata] = field(default_factory=dict)
file_store: FileStore | None = None
usage_counted: bool = False
task_result_file_id: str | None = None
- task_output_formats: list[TaskOutputFormat] | None = None
+ task_output_formats: list[TaskOutputFormat] = field(default_factory=lambda: ["default"])
+ include_model_output: bool = False
retention_held: bool = False
retention_epoch: float | None = None
def __post_init__(self) -> None:
if self.files is None:
self.files = []
- if self.output_formats is None:
- self.output_formats = ["markdown"]
if self.progress is None:
- self.progress = JobProgress()
+ self.progress = TaskProgress()
class JobStore:
@@ -1532,7 +1000,7 @@ class JobStore:
def create(
self,
- req: CreateJobRequest,
+ req: _ParseRequest,
file_store: FileStore,
*,
job_id: str | None = None,
@@ -1540,7 +1008,7 @@ class JobStore:
) -> _JobRecord:
"""创建唯一执行记录,允许 tasks 使用上传准备阶段预先分配的 ID。"""
if req.tier is None:
- raise ValueError("CreateJobRequest.tier must be resolved before job creation")
+ raise ValueError("_ParseRequest.tier must be resolved before job creation")
job_id = job_id or self._new_job_id()
if job_id in self._jobs:
_raise_api_error(409, error_type="invalid_request_error", code="job_already_exists", message="Job already exists")
@@ -1550,18 +1018,15 @@ class JobStore:
status="queued",
created_at=now,
tier=req.tier,
- output_formats=req.output_formats,
+ task_output_formats=req.output_formats,
+ include_model_output=req.include_model_output,
input_specs=dict(input_specs or {}),
- links=JobLinks(
- self=f"/v1/parse/jobs/{job_id}",
- cancel=f"/v1/parse/jobs/{job_id}",
- ),
)
for index, entry in enumerate(req.files):
name = _source_name(entry.source, file_store, input_spec=rec.input_specs.get(index))
rec.files.append(
- JobFileResult(
- file_id=entry.source.file_id if isinstance(entry.source, FileIdSource) else None,
+ _ExecutionFile(
+ file_id=entry.source.file_id if isinstance(entry.source, _StoredSource) else None,
name=name,
page_range=entry.page_range or "",
status="queued",
@@ -1569,7 +1034,7 @@ class JobStore:
)
rec.progress.total = len(rec.files)
rec.input_blobs = file_store.acquire_inputs(
- [entry.source.file_id for entry in req.files if isinstance(entry.source, FileIdSource)]
+ [entry.source.file_id for entry in req.files if isinstance(entry.source, _StoredSource)]
)
rec.file_store = file_store
self._jobs[job_id] = rec
@@ -1627,60 +1092,18 @@ class JobStore:
task.cancel()
return rec
- def list_jobs(
- self,
- *,
- status_filter: str | None,
- limit: int,
- after: str | None,
- created_after: str | None,
- order: str = "desc",
- ) -> JobListResponse:
- self.collect_expired()
- recs = list(self._jobs.values())
- if status_filter:
- allowed = set(status_filter.split(","))
- recs = [r for r in recs if r.status in allowed]
- if created_after:
- recs = [r for r in recs if r.created_at >= created_after]
- reverse = order == "desc"
- recs.sort(key=lambda r: r.created_at, reverse=reverse)
- start = 0
- if after:
- for i, r in enumerate(recs):
- if r.id == after:
- start = i + 1
- break
- page = recs[start : start + limit]
- return JobListResponse(
- data=[
- JobListItem(
- job_id=r.id,
- status=r.status,
- created_at=r.created_at,
- file_count=len(r.files),
- )
- for r in page
- ],
- first_id=page[0].id if page else None,
- last_id=page[-1].id if page else None,
- has_more=(start + limit) < len(recs),
- )
-
- def build_response(self, rec: _JobRecord, *, access_level: AccessLevel) -> JobAsyncResponse:
- return JobAsyncResponse(
- job_id=rec.id,
- status=rec.status,
- created_at=rec.created_at,
- started_at=rec.started_at,
- finished_at=rec.finished_at,
- tier=rec.tier,
- output_formats=rec.output_formats,
- access_level=access_level,
- progress=rec.progress,
- files=rec.files,
- links=rec.links or JobLinks(self="", cancel=""),
- )
+ def snapshot(self, rec: _JobRecord) -> dict[str, Any]:
+ """返回私有执行快照,不包含公开 Jobs 链接或产物协议。"""
+ return {
+ "job_id": rec.id,
+ "status": rec.status,
+ "created_at": rec.created_at,
+ "started_at": rec.started_at,
+ "finished_at": rec.finished_at,
+ "tier": rec.tier,
+ "progress": rec.progress.model_dump(),
+ "files": [item.model_dump() for item in rec.files],
+ }
def _record_usage(self, rec: _JobRecord) -> None:
"""完成计数只累计一次,元数据过期后服务总用量不倒退。"""
@@ -1718,38 +1141,36 @@ class JobStore:
# ── source helpers ───────────────────────────────────────────────────
-def _source_name(source: FileSource, file_store: FileStore | None = None, *, input_spec: _JobInputSpec | None = None) -> str:
- """优先使用 tasks 的逻辑文件名,普通 Jobs 保持既有来源命名规则。"""
+def _source_name(
+ source: _ExecutionSource, file_store: FileStore | None = None, *, input_spec: _JobInputSpec | None = None
+) -> str:
+ """使用 Tasks 的逻辑文件名识别类型,源存储不改变业务文件名。"""
if input_spec is not None:
return input_spec.filename
- if isinstance(source, FileIdSource) and file_store is not None:
+ if isinstance(source, _StoredSource) and file_store is not None:
try:
return file_store.get_file(source.file_id).filename
except Exception:
pass
return source.file_id
- if isinstance(source, UrlSource):
+ if isinstance(source, _UrlSource):
return source.url.rsplit("/", 1)[-1].split("?")[0] or "unknown"
- if isinstance(source, InlineSource):
- return source.name
- if isinstance(source, LocalSource):
- return pathlib.Path(source.path).name
return "unknown"
def _job_can_default_to_flash(
- req: CreateJobRequest, file_store: FileStore, input_specs: dict[int, _JobInputSpec] | None = None
+ req: _ParseRequest, file_store: FileStore, input_specs: dict[int, _JobInputSpec] | None = None
) -> bool:
"""判断整批输入是否已知为 Flash-only,或需下载后识别类型的无扩展名 URL。"""
return all(
- is_flash_only_parse_extension(name) or (isinstance(entry.source, UrlSource) and not pathlib.Path(name).suffix)
+ is_flash_only_parse_extension(name) or (isinstance(entry.source, _UrlSource) and not pathlib.Path(name).suffix)
for index, entry in enumerate(req.files)
for name in [_source_name(entry.source, file_store, input_spec=(input_specs or {}).get(index))]
)
def _job_has_any_flash_only_inputs(
- req: CreateJobRequest, file_store: FileStore, input_specs: dict[int, _JobInputSpec] | None = None
+ req: _ParseRequest, file_store: FileStore, input_specs: dict[int, _JobInputSpec] | None = None
) -> bool:
"""使用与执行一致的逻辑文件名检查批次中是否包含 Flash-only 格式。"""
return any(
@@ -1779,12 +1200,10 @@ class _ExtractedSource:
async def _extract_bytes(
- source: FileSource,
+ source: _ExecutionSource,
file_store: FileStore,
*,
url_timeout: int = 60,
- allow_local_source: bool = False,
- max_inline_bytes: int = _MAX_INLINE_BYTES_DEFAULT,
allow_http_source: bool = False,
max_url_bytes: int = _MAX_URL_BYTES_DEFAULT,
input_blobs: dict[str, str] | None = None,
@@ -1792,18 +1211,16 @@ async def _extract_bytes(
"""按既有来源策略读取字节,并保留不会由临时文件替代的来源上下文。"""
_validate_source_policy(
source,
- allow_local_source=allow_local_source,
- max_inline_bytes=max_inline_bytes,
allow_http_source=allow_http_source,
)
- if isinstance(source, FileIdSource):
+ if isinstance(source, _StoredSource):
sha = (input_blobs or {}).get(source.file_id)
if sha is None:
sha = file_store.get_file(source.file_id).sha256sum
if sha is None:
raise ValueError("File has no content")
return _ExtractedSource(file_store.read_blob(sha))
- if isinstance(source, UrlSource):
+ if isinstance(source, _UrlSource):
async with httpx.AsyncClient(timeout=url_timeout) as cli:
# 流式读取并在超过 max_url_bytes 时中止,避免无上限响应被整包缓冲进内存。
async with cli.stream("GET", source.url) as r:
@@ -1821,11 +1238,6 @@ async def _extract_bytes(
transport_encoding=r.charset_encoding,
media_type=(r.headers.get("content-type") or "").partition(";")[0].strip().casefold() or None,
)
- if isinstance(source, InlineSource):
- return _ExtractedSource(_decode_inline_data(source.data))
- if isinstance(source, LocalSource):
- path = pathlib.Path(source.path).expanduser().resolve(strict=False)
- return _ExtractedSource(path.read_bytes(), source_uri=path.as_uri(), local_resource_root=path.parent)
raise ValueError(f"Unknown source type: {type(source)}")
@@ -1839,13 +1251,11 @@ def _suffix_type(filename: str) -> str:
async def _run_job(
rec: _JobRecord,
- req: CreateJobRequest,
+ req: _ParseRequest,
file_store: FileStore,
*,
image_analysis: bool,
url_timeout: int = 60,
- allow_local_source: bool = False,
- max_inline_bytes: int = _MAX_INLINE_BYTES_DEFAULT,
allow_http_source: bool = False,
max_url_bytes: int = _MAX_URL_BYTES_DEFAULT,
flash_enabled: bool = True,
@@ -1870,15 +1280,13 @@ async def _run_job(
entry.source,
file_store,
url_timeout=url_timeout,
- allow_local_source=allow_local_source,
- max_inline_bytes=max_inline_bytes,
allow_http_source=allow_http_source,
max_url_bytes=max_url_bytes,
input_blobs=rec.input_blobs,
)
data = extracted.data
input_spec = rec.input_specs.get(i)
- if input_spec is not None and isinstance(entry.source, UrlSource):
+ if input_spec is not None and isinstance(entry.source, _UrlSource):
if not data:
raise MineruError("invalid_request", "Empty files are not supported")
if input_spec.bytes is not None and len(data) != input_spec.bytes:
@@ -1889,7 +1297,7 @@ async def _run_job(
rec.input_metadata[i] = _JobInputMetadata(bytes=len(data), sha256sum=sha)
suffix = pathlib.Path(fr.name).suffix
stype = _suffix_type(fr.name)
- if not stype and not suffix and isinstance(entry.source, UrlSource) and extracted.media_type == "text/html":
+ if not stype and not suffix and isinstance(entry.source, _UrlSource) and extracted.media_type == "text/html":
stype = "html"
suffix = ".html"
if not stype:
@@ -1926,68 +1334,29 @@ async def _run_job(
if rec.status in {"canceled", "failed"}:
break
- if rec.task_output_formats is not None:
- from .task_outputs import build_task_output
-
- try:
- artifact = await run_sync(
- build_task_output, result, rec.task_output_formats, pathlib.Path(tmpdir) / f"outputs_{i}"
- )
- if rec.status in {"canceled", "failed"}:
- break
- rec.task_result_file_id = await run_sync(
- file_store.create_file_for_output_path,
- pathlib.Path(fr.name).stem + artifact.suffix,
- artifact,
- job_id=rec.id,
- )
- except Exception as exc:
- raise MineruError("result_packaging_failed", str(exc)) from exc
-
- # Jobs 的原有产物策略与 Tasks 独立,不能向普通 Jobs 暴露 default 等产品选项。
- out_formats = set(rec.output_formats)
- output_files = OutputFiles()
-
- for fmt in (
- "markdown",
- "middle_json",
- "structured_content",
- ):
- if fmt not in out_formats:
- continue
- if fmt == "markdown":
- md = await run_sync(result.markdown)
- content_bytes = _text_utf8_bytes(md or "")
- sha = hashlib.sha256(content_bytes).hexdigest()
- file_store.store_blob(content_bytes, sha256hex=sha)
- fid = file_store.create_file_for_output(f"{fr.name}.md", content_bytes, sha256hex=sha, job_id=rec.id)
- output_files.markdown = OutputFileRef(file_id=fid, bytes=len(content_bytes))
- elif fmt == "middle_json":
- mj = _json_utf8_bytes(result.to_dict(skip_defaults=True))
- sha = hashlib.sha256(mj).hexdigest()
- file_store.store_blob(mj, sha256hex=sha)
- fid = file_store.create_file_for_output(f"{fr.name}.middle.json", mj, sha256hex=sha, job_id=rec.id)
- output_files.middle_json = OutputFileRef(file_id=fid, bytes=len(mj))
- elif fmt == "structured_content":
- cl2 = _json_utf8_bytes(await run_sync(result.structured_content))
- sha = hashlib.sha256(cl2).hexdigest()
- file_store.store_blob(cl2, sha256hex=sha)
- fid = file_store.create_file_for_output(
- f"{fr.name}.structured_content.json", cl2, sha256hex=sha, job_id=rec.id
- )
- output_files.structured_content = OutputFileRef(file_id=fid, bytes=len(cl2))
-
- # zip
- if "zip" in out_formats:
- zip_bytes = await run_sync(_build_self_contained_zip_output, result)
- zip_sha = hashlib.sha256(zip_bytes).hexdigest()
- file_store.store_blob(zip_bytes, sha256hex=zip_sha)
- zip_fid = file_store.create_file_for_output(f"{fr.name}.zip", zip_bytes, sha256hex=zip_sha, job_id=rec.id)
- output_files.zip = OutputFileRef(file_id=zip_fid, bytes=len(zip_bytes))
+ from .task_outputs import build_task_output
+
+ try:
+ artifact = await run_sync(
+ build_task_output,
+ result,
+ rec.task_output_formats,
+ pathlib.Path(tmpdir) / f"outputs_{i}",
+ include_model_output=rec.include_model_output,
+ )
+ if rec.status in {"canceled", "failed"}:
+ break
+ rec.task_result_file_id = await run_sync(
+ file_store.create_file_for_output_path,
+ pathlib.Path(fr.name).stem + artifact.suffix,
+ artifact,
+ job_id=rec.id,
+ )
+ except Exception as exc:
+ raise MineruError("result_packaging_failed", str(exc)) from exc
fr.status = "completed"
fr.page_range = _page_range_from_result_pages(result.pages)
- fr.output_files = output_files
fr.parse = FileParseInfo(
model_used=None,
@@ -2038,9 +1407,7 @@ _OPENAPI_TAGS: list[dict[str, str]] = [
{"name": "Health", "description": "Liveness and server readiness."},
{"name": "Tiers", "description": "Parsing tiers and runtime capability advertised by this server."},
{"name": "Models", "description": "VLM models available for parsing."},
- {"name": "Uploads", "description": "Create and manage upload sessions for source documents."},
{"name": "Files", "description": "Inspect uploaded files and fetch parsed content."},
- {"name": "Jobs", "description": "Submit parse jobs and poll for results."},
{"name": "Usage", "description": "Remote service quota and usage reporting."},
]
@@ -2053,8 +1420,7 @@ as soon as it is ready; other files may still be waiting for upload. Manifest
entries can instead provide a source `url`, with optional `bytes`; workers download
these inputs in the background. Choose one or more of default/docx/html/latex in
`output_formats`. Task status contains metadata only, never inline result bytes.
-The resource workflow remains: create an upload, complete it, submit a parse job for the
-resulting file, then poll the job and fetch its outputs. Health and usage
+Health and usage
endpoints report server readiness and quota state. Tiers (`flash`, `basic`,
`standard`, `advanced`) select the quality/speed trade-off; the set advertised
by `/v1/tiers` is what this server can actually serve.
@@ -2100,7 +1466,7 @@ async def get_health(request: Request) -> HealthResponse:
_require_model_preload_ready(request)
return HealthResponse(
version=__version__,
- features=HealthFeatures(sources=_parse_sources_for_server(allow_local_source=request.app.state.allow_local_source)),
+ features=HealthFeatures(),
)
@@ -2167,98 +1533,6 @@ async def list_tiers(request: Request) -> TierListResponse:
)
-# ── Uploads ──────────────────────────────────────────────────────────
-
-
-@_router.post(
- "/uploads",
- response_model=None,
- status_code=status.HTTP_200_OK,
- responses={200: {"model": UploadResponse}, **_ERR_400, **_ERR_413},
- tags=["Uploads"],
-)
-async def create_upload(
- body: CreateUploadRequest,
- request: Request,
- store: FileStore = Depends(_get_store),
-) -> Response:
- """Create an Upload."""
- base = str(request.base_url).rstrip("/")
- resp = store.create_upload(body, base_url=base)
- if resp.status == "completed":
- return JSONResponse(content=resp.model_dump(by_alias=True), status_code=200)
- return JSONResponse(content=resp.model_dump(by_alias=True), status_code=200)
-
-
-@_router.get(
- "/uploads/{upload_id}",
- response_model=UploadResponse,
- status_code=status.HTTP_200_OK,
- responses={**_ERR_404},
- tags=["Uploads"],
-)
-async def get_upload(
- upload_id: str = Path(description="Upload ID (upload_...)"),
- store: FileStore = Depends(_get_store),
-) -> UploadResponse:
- """Retrieve an Upload by ID."""
- return store._make_upload_response(store.get_upload(upload_id))
-
-
-@_router.put(
- "/uploads/{upload_id}/content",
- response_model=None,
- status_code=status.HTTP_200_OK,
- responses={**_ERR_404, **_ERR_409, **_ERR_413},
- tags=["Uploads"],
- openapi_extra={
- "requestBody": {
- "required": True,
- "content": {"application/octet-stream": {"schema": {"type": "string", "format": "binary"}}},
- }
- },
-)
-async def upload_content(
- request: Request,
- upload_id: str = Path(description="Upload ID"),
- store: FileStore = Depends(_get_store),
-) -> Response:
- """流式接收原始文件字节,保持独立 Upload 的既有 HTTP 协议。"""
- await store.store_upload_stream(upload_id, request.stream())
- return Response(status_code=200)
-
-
-@_router.post(
- "/uploads/{upload_id}/complete",
- response_model=UploadResponse,
- status_code=status.HTTP_200_OK,
- responses={**_ERR_400, **_ERR_404, **_ERR_409, **_ERR_413},
- tags=["Uploads"],
-)
-async def complete_upload(
- upload_id: str = Path(description="Upload ID"),
- body: CompleteUploadRequest | None = None,
- store: FileStore = Depends(_get_store),
-) -> UploadResponse:
- """Complete an upload, creating the File object."""
- return await run_sync(store.complete_upload, upload_id, body.sha256sum if body else None)
-
-
-@_router.post(
- "/uploads/{upload_id}/cancel",
- response_model=UploadResponse,
- status_code=status.HTTP_200_OK,
- responses={**_ERR_404, **_ERR_409},
- tags=["Uploads"],
-)
-async def cancel_upload(
- upload_id: str = Path(description="Upload ID"),
- store: FileStore = Depends(_get_store),
-) -> UploadResponse:
- """Cancel an in-progress upload."""
- return store.cancel_upload(upload_id)
-
-
# ── Files ────────────────────────────────────────────────────────────
@@ -2360,55 +1634,25 @@ async def delete_file(
return FileDeletionResponse(id=file_id, deleted=True)
-# ── Parse Jobs ───────────────────────────────────────────────────────
+# ── 私有解析执行 ─────────────────────────────────────────────────────
-async def submit_parse_job(
- body: CreateJobRequest,
+async def submit_execution(
+ body: _ParseRequest,
request: Request,
file_store: FileStore,
job_store: JobStore,
- access_level: AccessLevel,
*,
job_id: str | None = None,
- finalize: Callable[[_JobRecord], Awaitable[None]] | None = None,
input_specs: dict[int, _JobInputSpec] | None = None,
- task_output_formats: list[TaskOutputFormat] | None = None,
) -> _JobRecord:
- """统一校验并提交后台解析任务,供 V1 Job 与便捷入口共同调用。"""
+ """校验并提交 Tasks 的私有执行单元,复用唯一并发和生命周期管理。"""
for index, entry in enumerate(body.files):
try:
entry.page_range = normalize_page_range_input(entry.page_range) or None
except MineruError as exc:
_raise_api_error(400, error_type=exc.type, code=exc.code, message=str(exc), param=f"files.{index}.page_range")
_require_model_preload_ready(request)
- if body.callback is not None:
- _raise_api_error(
- 400,
- error_type="invalid_request_error",
- code="invalid_request",
- message=(
- "Webhook callback is not supported by this Local Parse Server. Use polling via GET /v1/parse/jobs/{job_id}."
- ),
- )
-
- # validate output formats — advanced formats require API key
- for fmt in body.output_formats:
- if fmt in ("html", "latex", "docx") and access_level == "anonymous":
- _raise_api_error(
- 403,
- error_type="permission_error",
- code="feature_requires_api_key",
- message=f"Output format '{fmt}' requires an API key",
- )
- if fmt not in _OUTPUT_FORMATS_LOCAL:
- _raise_api_error(
- 400,
- error_type="invalid_request_error",
- code="unsupported_output_format",
- message=f"Unknown output format: {fmt}",
- )
-
# 按请求 tier 选择启动时预先解析好的 runtime;全轻量输入可按 ADR-0024 直接归一到 flash 执行。
default_tier: Tier | None = request.app.state.default_tier
can_default_to_flash = _job_can_default_to_flash(body, file_store, input_specs)
@@ -2449,19 +1693,14 @@ async def submit_parse_job(
_validate_job_source_policy(
body,
- allow_local_source=request.app.state.allow_local_source,
- max_inline_bytes=request.app.state.max_inline_bytes,
allow_http_source=request.app.state.allow_http_source,
)
if job_store._closing:
_raise_api_error(503, error_type="engine_error", code="server_shutting_down", message="Server is shutting down")
rec = job_store.create(body, file_store, job_id=job_id, input_specs=input_specs)
- rec.task_output_formats = task_output_formats
- rec.retention_held = task_output_formats is not None
+ rec.retention_held = True
url_timeout_val: int = request.app.state.url_timeout
- allow_local_source_val: bool = request.app.state.allow_local_source
- max_inline_bytes_val: int = request.app.state.max_inline_bytes
allow_http_source_val: bool = request.app.state.allow_http_source
max_url_bytes_val: int = request.app.state.max_url_bytes
flash_enabled_val: bool = request.app.state.flash_enabled
@@ -2476,16 +1715,12 @@ async def submit_parse_job(
file_store,
image_analysis=image_analysis_val,
url_timeout=url_timeout_val,
- allow_local_source=allow_local_source_val,
- max_inline_bytes=max_inline_bytes_val,
allow_http_source=allow_http_source_val,
max_url_bytes=max_url_bytes_val,
flash_enabled=flash_enabled_val,
vlm_config=vlm_config_val,
job_store=job_store,
)
- if finalize is not None and rec.status in {"completed", "partial"}:
- await finalize(rec)
try:
job_store.start_task(rec, _bg_run)
@@ -2497,89 +1732,6 @@ async def submit_parse_job(
return rec
-@_router.post(
- "/parse/jobs",
- response_model=None,
- status_code=status.HTTP_202_ACCEPTED,
- responses={
- 202: {"model": JobAsyncResponse},
- **_ERR_400,
- **_ERR_403,
- **_ERR_429,
- **_ERR_503,
- },
- tags=["Jobs"],
-)
-async def create_job(
- body: CreateJobRequest,
- request: Request,
- file_store: FileStore = Depends(_get_store),
- job_store: JobStore = Depends(_get_job_store),
- access_level: AccessLevel = Depends(_resolve_access_level),
-) -> Response:
- """通过共享任务服务创建 V1 Job,保持既有响应结构。"""
- rec = await submit_parse_job(body, request, file_store, job_store, access_level)
- return JSONResponse(
- content=job_store.build_response(rec, access_level=access_level).model_dump(by_alias=True), status_code=202
- )
-
-
-@_router.get(
- "/parse/jobs/{job_id}",
- response_model=JobAsyncResponse,
- status_code=status.HTTP_200_OK,
- responses={**_ERR_404},
- tags=["Jobs"],
-)
-async def get_job(
- job_id: str = Path(description="Job ID (job_...)"),
- job_store: JobStore = Depends(_get_job_store),
- access_level: AccessLevel = Depends(_resolve_access_level),
-) -> JobAsyncResponse:
- """Retrieve a job's current status and results."""
- return job_store.build_response(job_store.get(job_id), access_level=access_level)
-
-
-@_router.get(
- "/parse/jobs",
- response_model=JobListResponse,
- status_code=status.HTTP_200_OK,
- tags=["Jobs"],
-)
-async def list_jobs(
- status_filter: str | None = Query(default=None, alias="status", description="Filter by status (comma-separated)"),
- limit: int = Query(default=20, ge=1, le=100, description="Items per page"),
- after: str | None = Query(default=None, description="Pagination cursor (last_id from previous page)"),
- order: Literal["asc", "desc"] = Query(default="desc", description="Sort by created_at"),
- created_after: str | None = Query(default=None, description="ISO-8601 lower bound for created_at"),
- job_store: JobStore = Depends(_get_job_store),
-) -> JobListResponse:
- """List jobs for the current tenant."""
- return job_store.list_jobs(
- status_filter=status_filter,
- limit=limit,
- after=after,
- order=order,
- created_after=created_after,
- )
-
-
-@_router.delete(
- "/parse/jobs/{job_id}",
- response_model=JobCancelResponse,
- status_code=status.HTTP_200_OK,
- responses={**_ERR_404, **_ERR_409},
- tags=["Jobs"],
-)
-async def cancel_job(
- job_id: str = Path(description="Job ID"),
- job_store: JobStore = Depends(_get_job_store),
-) -> JobCancelResponse:
- """Cancel a queued or running job."""
- rec = job_store.cancel(job_id)
- return JobCancelResponse(job_id=rec.id, status="canceled", canceled_at=JobStore._now())
-
-
# ── Usage ─────────────────────────────────────────────────────────────
@@ -2783,8 +1935,6 @@ def create_app(
no_advanced: bool = False,
concurrency: int = 1,
url_timeout: int = 60,
- allow_local_source: bool = False,
- max_inline_bytes: int = _MAX_INLINE_BYTES_DEFAULT,
max_url_bytes: int = _MAX_URL_BYTES_DEFAULT,
allow_http_source: bool = False,
api_key: str | None = None,
@@ -2810,10 +1960,6 @@ def create_app(
Maximum concurrent parse jobs (default 1).
url_timeout:
Timeout in seconds for downloading url sources (default 60).
- allow_local_source:
- Whether ``local`` sources may read paths visible to the server process.
- max_inline_bytes:
- Maximum decoded bytes accepted for ``inline`` sources.
max_url_bytes:
Maximum bytes downloaded for ``url`` sources; larger responses are
aborted mid-download instead of being buffered into memory.
@@ -2841,8 +1987,6 @@ def create_app(
_api_key: str | None = api_key or None
_upload_dir = pathlib.Path(upload_dir) if upload_dir else pathlib.Path(tempfile.mkdtemp(prefix="mineru_"))
_upload_dir.mkdir(parents=True, exist_ok=True)
- if max_inline_bytes < 0:
- raise ValueError("max_inline_bytes must be non-negative")
if max_url_bytes < 0:
raise ValueError("max_url_bytes must be non-negative")
@@ -2864,8 +2008,6 @@ def create_app(
application.state.tiers = _tiers
application.state.concurrency = concurrency
application.state.url_timeout = url_timeout
- application.state.allow_local_source = allow_local_source
- application.state.max_inline_bytes = max_inline_bytes
application.state.max_url_bytes = max_url_bytes
application.state.allow_http_source = allow_http_source
application.state.api_key = _api_key
@@ -2995,8 +2137,6 @@ def create_app(
application.state.tiers = _tiers
application.state.concurrency = concurrency
application.state.url_timeout = url_timeout
- application.state.allow_local_source = allow_local_source
- application.state.max_inline_bytes = max_inline_bytes
application.state.max_url_bytes = max_url_bytes
application.state.allow_http_source = allow_http_source
application.state.api_key = _api_key
@@ -3148,17 +2288,6 @@ def _build_server_log_config(log_level: str) -> dict[str, Any]:
type=int,
help="Timeout in seconds for url source downloads (default: 60)",
)
- "--allow-local-source",
- is_flag=True,
- help="Allow local sources to read any path visible to the server process.",
-)
- "--max-inline-bytes",
- default=_MAX_INLINE_BYTES_DEFAULT,
- type=int,
- help=f"Maximum decoded bytes for inline sources (default: {_MAX_INLINE_BYTES_DEFAULT})",
-)
@click.option(
"--max-url-bytes",
default=_MAX_URL_BYTES_DEFAULT,
@@ -3204,8 +2333,6 @@ def main(
no_advanced: bool,
concurrency: int,
url_timeout: int,
- allow_local_source: bool,
- max_inline_bytes: int,
max_url_bytes: int,
allow_http_source: bool,
disable_image_analysis: bool,
@@ -3262,8 +2389,6 @@ def main(
no_advanced=no_advanced,
concurrency=concurrency,
url_timeout=url_timeout,
- allow_local_source=allow_local_source,
- max_inline_bytes=max_inline_bytes,
max_url_bytes=max_url_bytes,
allow_http_source=allow_http_source,
api_key=api_key,
diff --git a/mineru/parser/task_api.py b/mineru/parser/task_api.py
index 9966f863..9f07d2a5 100644
--- a/mineru/parser/task_api.py
+++ b/mineru/parser/task_api.py
@@ -15,7 +15,7 @@ from pathlib import Path
from typing import Any, Literal, NoReturn, Protocol
from urllib.parse import quote
-from fastapi import APIRouter, Request, Response
+from fastapi import APIRouter, Query, Request, Response
from fastapi.responses import JSONResponse
from pydantic import AnyHttpUrl, BaseModel, ConfigDict, Field, ValidationError, model_validator
from starlette.requests import ClientDisconnect
@@ -26,25 +26,24 @@ from ..task_types import TaskOutputFormat
from ..types import Tier
from ..utils.async_utils import drain_future, run_sync
from .api_server import (
- AccessLevel,
ApiServerError,
- CreateJobRequest,
+ _ParseRequest,
ErrorDetail,
- FileIdSource,
+ _StoredSource,
FileParseInfo,
FileStore,
- JobFileEntry,
- JobProgress,
+ _ParseInput,
+ TaskProgress,
JobStore,
- UrlSource,
+ _UrlSource,
_JobInputMetadata,
_JobInputSpec,
_raise_api_error,
_require_model_preload_ready,
_validate_source_policy,
- submit_parse_job,
+ submit_execution,
)
-from .page_range import normalize_page_range_input
+from .page_range import PAGE_RANGE_DESCRIPTION, normalize_page_range_input
MAX_TASK_FILES = 100
TASK_UPLOAD_TTL = 3600
@@ -63,7 +62,7 @@ class TaskManifestItem(BaseModel):
bytes: int | None = Field(default=None, gt=0, strict=True)
url: str | None = None
sha256sum: str | None = Field(default=None, pattern=r"^[a-f0-9]{64}$")
- page_range: str | None = None
+ page_range: str | None = Field(default=None, description=PAGE_RANGE_DESCRIPTION)
@model_validator(mode="after")
def validate_source(self) -> TaskManifestItem:
@@ -84,6 +83,14 @@ class UploadTaskOptions(BaseModel):
ocr_mode: Literal["auto", "txt", "ocr"] = "auto"
page_range: str = ""
output_formats: list[TaskOutputFormat] = Field(default_factory=lambda: ["default"], min_length=1)
+ include_model_output: bool = False
+
+ @model_validator(mode="after")
+ def validate_model_output(self) -> UploadTaskOptions:
+ """模型输出只能附加到 default 结果包,不改变单文件 HTML/DOCX 的交付类型。"""
+ if self.include_model_output and "default" not in self.output_formats:
+ raise ValueError("include_model_output requires default output format")
+ return self
class CreateTaskRequest(UploadTaskOptions):
@@ -125,12 +132,32 @@ class TaskStatusResponse(BaseModel):
finished_at: str | None = None
tier: Tier | None = None
output_formats: list[TaskOutputFormat]
- progress: JobProgress
+ include_model_output: bool = False
+ progress: TaskProgress
files: list[TaskFileStatus]
uploads: list[TaskUploadLink] = Field(default_factory=list)
error: ErrorDetail | None = None
+class TaskListItem(BaseModel):
+ """任务列表只公开父任务摘要,不暴露内部逐文件执行。"""
+
+ task_id: str
+ status: TaskStatus
+ created_at: str
+ file_count: int
+
+
+class TaskListResponse(BaseModel):
+ """保持统一的游标分页结构,列表数据按调用方隔离。"""
+
+ object: Literal["list"] = "list"
+ data: list[TaskListItem]
+ first_id: str | None = None
+ last_id: str | None = None
+ has_more: bool = False
+
+
@dataclass(frozen=True)
class TaskUpload:
"""保存已校验的源字节位置,文件名视图独立于内容去重路径。"""
@@ -281,14 +308,22 @@ async def _read_submission(request: Request) -> tuple[UploadTaskOptions, list[Ta
options = UploadTaskOptions.model_validate(body.model_dump(exclude={"manifest"}))
options.page_range = normalize_page_range_input(options.page_range)
options.output_formats = list(dict.fromkeys(options.output_formats))
- for item in body.manifest:
- item.page_range = normalize_page_range_input(item.page_range if item.page_range is not None else options.page_range)
+ for index, item in enumerate(body.manifest):
+ try:
+ item.page_range = normalize_page_range_input(
+ item.page_range if item.page_range is not None else options.page_range
+ )
+ except MineruError as exc:
+ _invalid(str(exc), code=exc.code, param=f"manifest.{index}.page_range")
return options, body.manifest
except ClientDisconnect:
_invalid("Request interrupted", code="request_interrupted")
except MineruError as exc:
_invalid(str(exc), code=exc.code, param=exc.param)
- except (ValidationError, ValueError) as exc:
+ except ValidationError as exc:
+ location = exc.errors()[0]["loc"]
+ _invalid("Invalid request: " + str(exc), param=".".join(map(str, location)) or None)
+ except ValueError as exc:
_invalid(str(exc))
@@ -427,6 +462,47 @@ class TaskStore:
_invalid(f"Task {task_id} not found", status_code=404, code="task_not_found")
return record
+ def list_tasks(
+ self,
+ owner_scope: str,
+ *,
+ status_filter: str | None,
+ limit: int,
+ after: str | None,
+ order: Literal["asc", "desc"],
+ created_after: str | None,
+ ) -> TaskListResponse:
+ """对父任务应用稳定排序和游标;同秒创建的任务以 ID 明确排序。"""
+ self.collect_expired()
+ allowed = set(status_filter.split(",")) if status_filter else None
+ if allowed is not None and not allowed.issubset({"waiting_upload", "queued", "running", *TERMINAL_TASK_STATUSES}):
+ _invalid("Unknown task status", param="status")
+ records = sorted(
+ (record for record in self._records.values() if record.backend.owner_scope == owner_scope),
+ key=lambda record: (record.created_at, record.id),
+ reverse=order == "desc",
+ )
+ if after:
+ cursor = self.lookup(after, owner_scope)
+ key = (cursor.created_at, cursor.id)
+ records = [
+ record
+ for record in records
+ if ((record.created_at, record.id) > key if order == "asc" else (record.created_at, record.id) < key)
+ ]
+ records = [
+ record
+ for record in records
+ if (allowed is None or record.status in allowed) and (created_after is None or record.created_at >= created_after)
+ ]
+ page = records[:limit]
+ return TaskListResponse(
+ data=[TaskListItem(task_id=r.id, status=r.status, created_at=r.created_at, file_count=len(r.files)) for r in page],
+ first_id=page[0].id if page else None,
+ last_id=page[-1].id if page else None,
+ has_more=len(records) > limit,
+ )
+
def update_lease(self, record: _TaskRecord, seconds: int | None) -> None:
"""续租只保护结果保留期,从不延长上传窗口,也不复活已释放的结果。"""
if seconds is None:
@@ -774,6 +850,7 @@ def task_payload(record: _TaskRecord, request: Request) -> dict[str, Any]:
"finished_at": record.finished_at,
"tier": record.options.tier,
"output_formats": record.options.output_formats,
+ "include_model_output": record.options.include_model_output,
"progress": {
"completed": sum(file.status == "completed" for file in record.files),
"failed": sum(file.status == "failed" for file in record.files),
@@ -808,6 +885,26 @@ def build_task_router(factory: Callable[[Request], TaskBackend]) -> APIRouter:
"""两类部署使用同一任务协议,下载交给已有 Files 入口。"""
router = APIRouter(prefix="/v1", tags=["Tasks"])
+ @router.get("/tasks", response_model=TaskListResponse)
+ async def list_tasks(
+ request: Request,
+ status_filter: str | None = Query(None, alias="status"),
+ limit: int = Query(20, ge=1, le=100),
+ after: str | None = None,
+ order: Literal["asc", "desc"] = "desc",
+ created_after: str | None = None,
+ ) -> TaskListResponse:
+ """读取当前调用方的父任务列表,子任务不进入公开列表。"""
+ backend = factory(request)
+ return request.app.state.task_store.list_tasks(
+ backend.owner_scope,
+ status_filter=status_filter,
+ limit=limit,
+ after=after,
+ order=order,
+ created_after=created_after,
+ )
+
@router.post("/tasks", status_code=202, response_model=TaskStatusResponse, openapi_extra=_request_schema())
async def create_task(request: Request) -> Response:
"""先验证 JSON 再建立稳定任务 ID,HTTP 结束后逐文件工作独立继续。"""
@@ -875,7 +972,6 @@ class ApiTaskBackend:
self.files: FileStore = request.app.state.file_store
self.jobs: JobStore = request.app.state.job_store
self.owner_scope = "local"
- self.access: AccessLevel = "registered" if request.app.state.api_key else "anonymous"
self._pins: list[str] = []
self._owned_files: list[str] = []
@@ -897,9 +993,7 @@ class ApiTaskBackend:
if item.url is not None:
try:
_validate_source_policy(
- UrlSource(url=item.url),
- allow_local_source=False,
- max_inline_bytes=state.max_inline_bytes,
+ _UrlSource(url=item.url),
allow_http_source=state.allow_http_source,
)
except ValueError as exc:
@@ -949,27 +1043,26 @@ class ApiTaskBackend:
await run_sync(
self.files.register_source_file, upload.filename, upload.path, upload.sha256sum, upload.file_id, keep_alive=True
)
- body = CreateJobRequest(
+ body = _ParseRequest(
files=[
- JobFileEntry(
- source=UrlSource(url=item.url) if item.url is not None else FileIdSource(file_id=uploads[index].file_id),
+ _ParseInput(
+ source=_UrlSource(url=item.url) if item.url is not None else _StoredSource(file_id=uploads[index].file_id),
page_range=item.page_range or None,
)
for index, item in enumerate(manifest)
],
tier=options.tier,
ocr_mode=options.ocr_mode,
- output_formats=[],
+ output_formats=options.output_formats,
+ include_model_output=options.include_model_output,
)
- await submit_parse_job(
+ await submit_execution(
body,
self.request,
self.files,
self.jobs,
- self.access,
job_id=task_id,
- task_output_formats=options.output_formats,
input_specs={
index: _JobInputSpec(item.filename, item.bytes, item.sha256sum) for index, item in enumerate(manifest)
},
@@ -980,7 +1073,7 @@ class ApiTaskBackend:
"""只读取 Job 与最终 File 的标识,不在状态请求中访问产物正文。"""
record = self.jobs.get(task_id)
return TaskExecution(
- self.jobs.build_response(record, access_level=self.access).model_dump(by_alias=True),
+ self.jobs.snapshot(record),
record.task_result_file_id,
None,
dict(record.input_metadata),
@@ -1002,6 +1095,8 @@ __all__ = [
"TaskBackend",
"TaskExecution",
"TaskManifestItem",
+ "TaskListItem",
+ "TaskListResponse",
"TaskStatusResponse",
"TaskStore",
"TaskUpload",
diff --git a/mineru/parser/task_outputs.py b/mineru/parser/task_outputs.py
index 57cf914e..83bb299c 100644
--- a/mineru/parser/task_outputs.py
+++ b/mineru/parser/task_outputs.py
@@ -10,7 +10,13 @@ from ..task_types import TASK_OUTPUT_FORMATS, TaskOutputFormat
from .base import ParseResult
-def build_task_output(result: ParseResult, formats: list[TaskOutputFormat], directory: Path) -> Path:
+def build_task_output(
+ result: ParseResult,
+ formats: list[TaskOutputFormat],
+ directory: Path,
+ *,
+ include_model_output: bool = False,
+) -> Path:
"""一次物化并渲染所选格式,全部成功后才返回主产物路径。"""
from docvortex.export import embed_html_images, materialize_middle, validate_materialized_assets
@@ -20,6 +26,15 @@ def build_task_output(result: ParseResult, formats: list[TaskOutputFormat], dire
validate_materialized_assets(middle, assets)
exported = ParseResult(middle_json=middle)
entries: dict[str, bytes] = {}
+ if include_model_output:
+ if "default" not in formats:
+ raise ValueError("include_model_output requires default output format")
+ if result._model_output is not None:
+ entries["model_output.json"] = json.dumps(
+ result._model_output.to_dict(skip_defaults=False),
+ ensure_ascii=False,
+ indent=2,
+ ).encode("utf-8", errors="replace")
if "default" in formats:
entries.update(
{
diff --git a/scripts/http_api_example.sh b/scripts/http_api_example.sh
index ca3cc05e..9924bb3d 100755
--- a/scripts/http_api_example.sh
+++ b/scripts/http_api_example.sh
@@ -1,365 +1,5 @@
#!/usr/bin/env bash
-# mineru-http-example — one complete V1 API cycle with plain HTTP calls:
-# create upload -> upload bytes -> complete -> submit parse job -> bounded poll -> download artifacts.
-#
-# The documentation pages (docs/{en,zh}/usage/http_api.md) show excerpts of this
-# script; keep them in sync by copying from here, never by editing both copies.
-#
-# Usage:
-# MINERU_API_URL=http://127.0.0.1:8000 [MINERU_API_KEY=...] \
-# ./scripts/http_api_example.sh document.html
-#
-# Environment:
-# MINERU_API_URL (required) Service root before /v1.
-# MINERU_API_KEY (optional) Bearer key; attached only to same-origin requests.
-# MINERU_TIER (optional) Requested tier (default: flash, for native documents).
-# OUTPUT_FORMATS (optional) Comma-separated (default: markdown,zip).
-# OUTPUT_DIR (optional) Download directory (default: current directory).
-# PAGE_RANGE (optional) PDF page range such as "1-10"; omit for whole/native documents.
-# CONNECT_TIMEOUT (optional) Connect timeout per request in seconds (default: 10).
-# REQUEST_TIMEOUT (optional) Max seconds per regular request (default: 60).
-# TRANSFER_TIMEOUT (optional) Max seconds per upload/download (default: 300).
-# MAX_POLLS (optional) Polling attempts before giving up (default: 100).
-# POLL_INTERVAL (optional) Seconds between polls (default: 3).
-#
-# Exit codes:
-# 0 Job completed and every requested artifact was saved.
-# 1 Script, transport, protocol, download, or local write failure.
-# 2 Job partial: artifacts of the completed files were saved first.
-# 3 Job failed.
-# 4 Job canceled.
-# 124 Polling budget exhausted; the job keeps running. job_id is printed
-# so the same polling request can be resumed later.
-#
-# Notes:
-# - Requires bash, curl (>= 7.83.0 for cross-port credential isolation), jq,
-# python3, and sha256sum or shasum.
-# - MAX_POLLS bounds the number of polling requests, not the total wall time:
-# each request also spends up to REQUEST_TIMEOUT plus POLL_INTERVAL.
-# - The byte upload uses the method/URL/headers returned by the service. The
-# MinerU API key is attached only when the upload URL is same-origin with
-# the API base (scheme + host + effective port, relative URLs resolved
-# against the base). Service-provided upload headers are always kept
-# as-is. --location-trusted is never used.
+# 统一 HTTP 示例复用 Tasks 客户端;参数、恢复和退出码见 http_task_example.py。
set -euo pipefail
-
-die() {
- echo "error: $*" >&2
- exit 1
-}
-
-command -v curl >/dev/null 2>&1 || die "curl is required"
-command -v jq >/dev/null 2>&1 || die "jq is required"
-command -v python3 >/dev/null 2>&1 || die "python3 is required"
-
-BASE_URL="${MINERU_API_URL:-}"
-API_KEY="${MINERU_API_KEY:-}"
-TIER="${MINERU_TIER:-flash}"
-OUTPUT_FORMATS_CSV="${OUTPUT_FORMATS:-markdown,zip}"
-OUTPUT_DIR="${OUTPUT_DIR:-.}"
-PAGE_RANGE="${PAGE_RANGE:-}"
-CONNECT_TIMEOUT="${CONNECT_TIMEOUT:-10}"
-REQUEST_TIMEOUT="${REQUEST_TIMEOUT:-60}"
-TRANSFER_TIMEOUT="${TRANSFER_TIMEOUT:-300}"
-MAX_POLLS="${MAX_POLLS:-100}"
-POLL_INTERVAL="${POLL_INTERVAL:-3}"
-
-[ -n "$BASE_URL" ] || die "MINERU_API_URL is required"
-BASE_URL="${BASE_URL%/}"
-[ $# -eq 1 ] || die "usage: $0 <input-file>"
-INPUT_FILE=$1
-[ -f "$INPUT_FILE" ] || die "file not found: $INPUT_FILE"
-mkdir -p -- "$OUTPUT_DIR" || die "cannot create output dir: $OUTPUT_DIR"
-
-REQUEST_BODY=$(mktemp)
-RESPONSE=$(mktemp)
-JOB_RESPONSE=$(mktemp)
-trap 'rm -f "$REQUEST_BODY" "$RESPONSE" "$JOB_RESPONSE"' EXIT
-
-auth_args=()
-if [ -n "$API_KEY" ]; then
- auth_args=(-H "Authorization: Bearer $API_KEY")
-fi
-
-# Request expecting a JSON body saved to $RESPONSE. Tool exit codes are
-# converted to script exit code 1; curl/jq codes are never passed through.
-curl_json() { # curl_json <stage> <max-time> <curl-args...>
- local stage=$1 max_time=$2
- shift 2
- local rc=0
- curl --silent --show-error --fail \
- --connect-timeout "$CONNECT_TIMEOUT" --max-time "$max_time" \
- -o "$RESPONSE" "$@" || rc=$?
- if [ "$rc" -ne 0 ]; then
- die "$stage failed (curl exit $rc)"
- fi
-}
-
-# Extract a non-empty string field; jq -e makes null/false/missing exit non-zero.
-json_string() { # json_string <file> <jq-expr> <label>
- local value
- value=$(jq -er "$2 | select(type == \"string\" and length > 0)" "$1") || die "invalid response at $3: field $2 is not a non-empty string"
- printf '%s\n' "$value"
-}
-
-job_status_ok() {
- case "$1" in
- queued | running | completed | partial | failed | canceled) return 0 ;;
- *) return 1 ;;
- esac
-}
-
-is_terminal() {
- case "$1" in
- completed | partial | failed | canceled) return 0 ;;
- *) return 1 ;;
- esac
-}
-
-file_size() {
- stat -c%s "$1" 2>/dev/null || stat -f%z "$1"
-}
-
-file_sha256() {
- if command -v sha256sum >/dev/null 2>&1; then
- sha256sum "$1" | cut -d' ' -f1
- elif command -v shasum >/dev/null 2>&1; then
- shasum -a 256 "$1" | cut -d' ' -f1
- else
- die "sha256sum or shasum is required"
- fi
-}
-
-# Same-origin check aligned with the SDK rule (api_client._same_origin_upload_headers):
-# compare scheme + host + effective port; resolve relative upload URLs against
-# the API base. Prints the resolved absolute URL on stdout. Exit 0 = same
-# origin, 1 = different, 2 = invalid URL.
-resolve_upload_url() { # resolve_upload_url <api-base> <target-url>
- python3 - "$1" "$2" <<'PY'
-import sys
-from urllib.parse import urljoin, urlsplit
-
-
-def origin(url: str):
- parts = urlsplit(url)
- if parts.scheme not in {"http", "https"} or not parts.hostname:
- raise ValueError(f"not an http(s) URL: {url!r}")
- return parts.scheme.lower(), parts.hostname.lower(), parts.port or (443 if parts.scheme == "https" else 80)
-
-
-try:
- base = origin(sys.argv[1])
- resolved = urljoin(sys.argv[1], sys.argv[2])
- target = origin(resolved)
-except ValueError as exc:
- print(exc, file=sys.stderr)
- raise SystemExit(2)
-print(resolved)
-raise SystemExit(0 if base == target else 1)
-PY
-}
-
-# Build UPLOAD_HEADER_ARGS from the service-provided upload_headers mapping.
-# Headers are passed as a bash argument array; values containing newlines are
-# rejected; no eval and no unquoted string expansion.
-UPLOAD_HEADER_ARGS=()
-build_upload_header_args() { # build_upload_header_args <response-file>
- local response=$1 key value
- jq -e '(.upload_headers // {}) | type == "object" and ([to_entries[].value | type] | all(. == "string"))' \
- "$response" >/dev/null || die "upload_headers is not a string mapping"
- while IFS= read -r -d '' key && IFS= read -r -d '' value; do
- case "$key$value" in
- *$'\n'* | *$'\r'*) die "upload_headers contains a newline in '$key'" ;;
- esac
- UPLOAD_HEADER_ARGS+=(-H "$key: $value")
- done < <(jq -j '(.upload_headers // {}) | to_entries[] | (.key, "\u0000", (.value | tostring), "\u0000")' "$response")
-}
-
-ext_for_format() {
- case "$1" in
- markdown) printf '.md' ;;
- middle_json | structured_content) printf '.json' ;;
- zip) printf '.zip' ;;
- html) printf '.html' ;;
- latex) printf '.tex' ;;
- docx) printf '.docx' ;;
- *) printf '.%s' "$1" ;;
- esac
-}
-
-sanitize_name() {
- printf '%s' "$1" | tr -c 'A-Za-z0-9._-' '_'
-}
-
-print_file_results() {
- jq -r '.files[] | " \(.name // "?"): \(.status // "?")\(if .error then " (\(.error.code // "error"): \(.error.message // ""))" else "" end)"' \
- "$JOB_RESPONSE" || true
-}
-
-# Download the requested artifacts of every completed file. Writes to *.part
-# first and moves into place on success, so a half file never looks complete.
-# Sets MISSING_ARTIFACTS=1 when a completed file lacks a requested format.
-MISSING_ARTIFACTS=0
-download_artifacts() {
- local name fmt fid safe dest part
- while IFS=$'\t' read -r name fmt fid; do
- if [ -z "$fid" ]; then
- echo "error: completed file '$name' has no '$fmt' output reference" >&2
- MISSING_ARTIFACTS=1
- continue
- fi
- safe=$(sanitize_name "$name")
- dest="$OUTPUT_DIR/${safe}$(ext_for_format "$fmt")"
- part="$dest.part"
- echo "==> Downloading $fmt -> $dest"
- local download_rc=0
- curl --silent --show-error --fail --location \
- --connect-timeout "$CONNECT_TIMEOUT" --max-time "$TRANSFER_TIMEOUT" \
- ${auth_args[@]+"${auth_args[@]}"} -o "$part" "$BASE_URL/v1/files/$fid/content" || download_rc=$?
- if [ "$download_rc" -ne 0 ]; then
- rm -f -- "$part"
- die "download of $fmt failed (file $fid, curl exit $download_rc)"
- fi
- mv -f -- "$part" "$dest"
- done < <(
- for fmt in ${OUTPUT_FORMATS_CSV//,/ }; do
- jq -r --arg fmt "$fmt" \
- '.files[] | select(.status == "completed") | [(.name // "?"), $fmt, (.output_files[$fmt].file_id // "")] | @tsv' \
- "$JOB_RESPONSE"
- done
- )
-}
-
-# ── 1. Create the upload session ────────────────────────────────────────────
-INPUT_NAME=$(basename -- "$INPUT_FILE")
-INPUT_BYTES=$(file_size "$INPUT_FILE") || die "cannot stat $INPUT_FILE"
-INPUT_SHA256=$(file_sha256 "$INPUT_FILE")
-case "$INPUT_NAME" in
- *.pdf) INPUT_MIME="application/pdf" ;;
- *.html | *.htm) INPUT_MIME="text/html" ;;
- *.docx) INPUT_MIME="application/vnd.openxmlformats-officedocument.wordprocessingml.document" ;;
- *.doc) INPUT_MIME="application/msword" ;;
- *.pptx) INPUT_MIME="application/vnd.openxmlformats-officedocument.presentationml.presentation" ;;
- *.xlsx) INPUT_MIME="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" ;;
- *.epub) INPUT_MIME="application/epub+zip" ;;
- *) INPUT_MIME="application/octet-stream" ;;
-esac
-
-echo "==> Creating upload for $INPUT_NAME ($INPUT_BYTES bytes)"
-jq -n \
- --arg filename "$INPUT_NAME" --argjson bytes "$INPUT_BYTES" \
- --arg mime_type "$INPUT_MIME" --arg sha256sum "$INPUT_SHA256" \
- '{filename: $filename, bytes: $bytes, mime_type: $mime_type, purpose: "parse", sha256sum: $sha256sum}' \
- >"$REQUEST_BODY"
-curl_json "create upload" "$REQUEST_TIMEOUT" -X POST "$BASE_URL/v1/uploads" \
- ${auth_args[@]+"${auth_args[@]}"} -H "Content-Type: application/json" --data-binary "@$REQUEST_BODY"
-
-UPLOAD_STATUS=$(json_string "$RESPONSE" '.status' 'create upload')
-case "$UPLOAD_STATUS" in
- pending | completed) ;;
- *) die "create upload: unsupported status '$UPLOAD_STATUS'" ;;
-esac
-UPLOAD_ID=$(json_string "$RESPONSE" '.id' 'create upload')
-
-# ── 2. Upload the bytes (pending only) ─────────────────────────────────────
-if [ "$UPLOAD_STATUS" = "pending" ]; then
- UPLOAD_URL=$(json_string "$RESPONSE" '.upload_url' 'create upload')
- UPLOAD_METHOD=$(json_string "$RESPONSE" '.upload_method // "PUT"' 'create upload')
- case "$UPLOAD_METHOD" in
- PUT) ;;
- *) die "unsupported upload_method '$UPLOAD_METHOD'" ;;
- esac
- build_upload_header_args "$RESPONSE"
- origin_rc=0
- UPLOAD_URL_ABSOLUTE=$(resolve_upload_url "$BASE_URL" "$UPLOAD_URL") || origin_rc=$?
- [ "$origin_rc" -ne 2 ] || die "service returned an invalid upload_url"
- upload_args=(${UPLOAD_HEADER_ARGS[@]+"${UPLOAD_HEADER_ARGS[@]}"})
- if [ "$origin_rc" -eq 0 ]; then
- upload_args+=(${auth_args[@]+"${auth_args[@]}"})
- fi
- echo "==> Uploading bytes via $UPLOAD_METHOD (same-origin: $([ "$origin_rc" -eq 0 ] && echo yes || echo no))"
- upload_rc=0
- curl --silent --show-error --fail \
- --connect-timeout "$CONNECT_TIMEOUT" --max-time "$TRANSFER_TIMEOUT" \
- -X "$UPLOAD_METHOD" "$UPLOAD_URL_ABSOLUTE" ${upload_args[@]+"${upload_args[@]}"} \
- --data-binary "@$INPUT_FILE" -o /dev/null || upload_rc=$?
- if [ "$upload_rc" -ne 0 ]; then
- die "byte upload failed (curl exit $upload_rc)"
- fi
-
- # ── 3. Complete the upload to obtain the file id ──────────────────────────
- echo "==> Completing upload $UPLOAD_ID"
- curl_json "complete upload" "$REQUEST_TIMEOUT" -X POST "$BASE_URL/v1/uploads/$UPLOAD_ID/complete" \
- ${auth_args[@]+"${auth_args[@]}"}
- COMPLETE_STATUS=$(json_string "$RESPONSE" '.status' 'complete upload')
- [ "$COMPLETE_STATUS" = "completed" ] || die "complete upload: unexpected status '$COMPLETE_STATUS'"
-else
- echo "==> Upload deduplicated; reusing the existing file"
-fi
-FILE_ID=$(json_string "$RESPONSE" '.file.id' 'upload')
-
-# ── 4. Submit the parse job ────────────────────────────────────────────────
-echo "==> Submitting parse job (tier=$TIER, formats=$OUTPUT_FORMATS_CSV)"
-jq -n \
- --arg file_id "$FILE_ID" --arg tier "$TIER" \
- --arg formats_csv "$OUTPUT_FORMATS_CSV" --arg page_range "$PAGE_RANGE" \
- '{
- files: [{source: {type: "file_id", file_id: $file_id}}
- + (if $page_range == "" then {} else {page_range: $page_range} end)],
- tier: $tier,
- output_formats: ($formats_csv | split(",") | map(gsub("^ +| +$"; "")))
- }' >"$REQUEST_BODY"
-curl_json "create job" "$REQUEST_TIMEOUT" -X POST "$BASE_URL/v1/parse/jobs" \
- ${auth_args[@]+"${auth_args[@]}"} -H "Content-Type: application/json" --data-binary "@$REQUEST_BODY"
-JOB_ID=$(json_string "$RESPONSE" '.job_id' 'create job')
-JOB_STATUS=$(json_string "$RESPONSE" '.status' 'create job')
-job_status_ok "$JOB_STATUS" || die "create job: unsupported status '$JOB_STATUS'"
-cp -- "$RESPONSE" "$JOB_RESPONSE"
-echo "==> Job $JOB_ID ($JOB_STATUS)"
-
-# ── 5. Poll until a terminal state (bounded by request count) ───────────────
-polls=0
-while ! is_terminal "$JOB_STATUS"; do
- polls=$((polls + 1))
- if [ "$polls" -gt "$MAX_POLLS" ]; then
- echo "error: polling budget exhausted after $MAX_POLLS attempts; job is still '$JOB_STATUS' and was NOT canceled" >&2
- echo "job_id=$JOB_ID"
- echo "resume with: curl -fsS ${BASE_URL}/v1/parse/jobs/${JOB_ID} -H 'Authorization: Bearer <key>'" >&2
- exit 124
- fi
- sleep "$POLL_INTERVAL"
- curl_json "poll job" "$REQUEST_TIMEOUT" "$BASE_URL/v1/parse/jobs/$JOB_ID" ${auth_args[@]+"${auth_args[@]}"}
- JOB_STATUS=$(json_string "$RESPONSE" '.status' 'poll job')
- job_status_ok "$JOB_STATUS" || die "poll job: unsupported status '$JOB_STATUS'"
- cp -- "$RESPONSE" "$JOB_RESPONSE"
-done
-
-echo "==> Job $JOB_ID finished: $JOB_STATUS"
-print_file_results
-
-# ── 6. Download the artifacts of completed files ────────────────────────────
-case "$JOB_STATUS" in
- completed)
- download_artifacts
- if [ "$MISSING_ARTIFACTS" -ne 0 ]; then
- die "job completed but some requested artifacts were missing"
- fi
- echo "==> Done."
- exit 0
- ;;
- partial)
- # Save the successful artifacts before reporting partial completion.
- download_artifacts
- echo "==> Partial success: some files failed; saved artifacts of the completed files." >&2
- exit 2
- ;;
- failed)
- exit 3
- ;;
- canceled)
- exit 4
- ;;
- *)
- die "unreachable terminal status '$JOB_STATUS'"
- ;;
-esac
+SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+exec bash "$SCRIPT_DIR/http_task_example.sh" "$@"
diff --git a/scripts/http_task_example.py b/scripts/http_task_example.py
index e6ae9c22..e4f4a91c 100644
--- a/scripts/http_task_example.py
+++ b/scripts/http_task_example.py
@@ -310,6 +310,13 @@ class TaskClient:
"page_range": os.environ.get("PAGE_RANGE", ""),
"output_formats": list(dict.fromkeys(formats)),
}
+ model_output = os.environ.get("INCLUDE_MODEL_OUTPUT", "0")
+ if model_output not in {"0", "1"}:
+ raise ClientError("INCLUDE_MODEL_OUTPUT must be 0 or 1")
+ if model_output == "1":
+ if "default" not in formats:
+ raise ClientError("INCLUDE_MODEL_OUTPUT requires default output format")
+ body["include_model_output"] = True
if os.environ.get("MINERU_TIER"):
body["tier"] = os.environ["MINERU_TIER"]
return self.json_request(
diff --git a/tests/unittest/task_test_utils.py b/tests/unittest/task_test_utils.py
new file mode 100644
index 00000000..4163c291
--- /dev/null
+++ b/tests/unittest/task_test_utils.py
@@ -0,0 +1,71 @@
+"""Tasks 回归测试共享的真实上传与产物读取工具,不模拟旧 HTTP 协议。"""
+
+import hashlib
+import io
+import tempfile
+import time
+import zipfile
+from pathlib import Path
+from typing import Any
+
+from fastapi.testclient import TestClient
+from mineru.parser.api_server import FileStore, _JobRecord
+from mineru.parser.base import ParseResult
+from mineru.parser.task_outputs import build_task_output
+
+
+def stored_source(store: FileStore, path: Path) -> dict[str, str]:
+ """为执行层单元测试登记已校验源字节,不创建公开上传资源。"""
+ sha = hashlib.sha256(path.read_bytes()).hexdigest()
+ file_id = store.register_source_file(path.name, path, sha)
+ return {"type": "file_id", "file_id": file_id}
+
+
+def execution_output(store: FileStore, record: _JobRecord, name: str) -> bytes:
+ """从唯一 Tasks 产物读取成员,避免断言已经移除的多产物引用。"""
+ with zipfile.ZipFile(io.BytesIO(store.read_file_data(record.task_result_file_id))) as archive:
+ return archive.read(name)
+
+
+def build_default_zip(result: ParseResult) -> bytes:
+ """使用真实 Tasks 产物构建器,保留可用原始模型输出。"""
+ with tempfile.TemporaryDirectory() as directory:
+ return build_task_output(result, ["default"], Path(directory), include_model_output=True).read_bytes()
+
+
+def submit_task(client: TestClient, path: Path, **options: Any) -> dict[str, Any]:
+ """通过清单和返回链接上传文件,任何失败都保留真实 HTTP 响应供断言。"""
+ data = path.read_bytes()
+ response = client.post(
+ "/v1/tasks",
+ json={
+ "manifest": [
+ {
+ "client_file_id": "file",
+ "filename": path.name,
+ "bytes": len(data),
+ "sha256sum": hashlib.sha256(data).hexdigest(),
+ }
+ ],
+ **options,
+ },
+ )
+ assert response.status_code == 202, response.text
+ task = response.json()
+ for link in task["uploads"]:
+ response = client.put(link["upload_url"], content=data, headers=link["upload_headers"])
+ assert response.status_code < 400, response.text
+ return task
+
+
+def wait_task(client: TestClient, task: dict[str, Any], timeout: float = 5) -> dict[str, Any]:
+ """等待父任务终态,避免使用内部执行 ID 代替公开 task_id。"""
+ deadline = time.monotonic() + timeout
+ while time.monotonic() < deadline:
+ response = client.get(task["status_url"])
+ assert response.status_code == 200, response.text
+ task = response.json()
+ if task["status"] in {"completed", "partial", "failed", "canceled"}:
+ return task
+ time.sleep(0.01)
+ raise AssertionError(f"Task did not finish: {task}")
diff --git a/tests/unittest/test_async_job_lifecycle.py b/tests/unittest/test_async_job_lifecycle.py
index 84c6b792..81a86280 100644
--- a/tests/unittest/test_async_job_lifecycle.py
+++ b/tests/unittest/test_async_job_lifecycle.py
@@ -1,5 +1,8 @@
"""验证解析 API 任务取消、排队和关闭的真实生命周期。"""
+from task_test_utils import stored_source
+
+
import asyncio
from pathlib import Path
@@ -161,18 +164,18 @@ def test_late_parser_result_is_not_published_after_cancel(tmp_path: Path, monkey
monkeypatch.setattr(api_server, "parse_async", parse)
store = JobStore()
files = api_server.FileStore(tmp_path / "storage")
- request = api_server.CreateJobRequest.model_validate(
+ request = api_server._ParseRequest.model_validate(
{
"tier": "standard",
- "output_formats": ["middle_json"],
- "files": [{"source": {"type": "local", "path": str(source)}}],
+ "output_formats": ["default"],
+ "files": [{"source": stored_source(files, source)}],
}
)
rec = store.create(request, files)
async def work() -> None:
"""执行真实任务循环,避免只测试状态赋值。"""
- await api_server._run_job(rec, request, files, image_analysis=True, allow_local_source=True)
+ await api_server._run_job(rec, request, files, image_analysis=True)
store.start_task(rec, work)
await entered.wait()
@@ -180,7 +183,7 @@ def test_late_parser_result_is_not_published_after_cancel(tmp_path: Path, monkey
store.cancel(rec.id)
await task
assert rec.status == "canceled"
- assert rec.files[0].output_files is None
+ assert rec.task_result_file_id is None
assert rec.progress.completed == 0
assert not rendered
await store.shutdown()
diff --git a/tests/unittest/test_doclib_cache_semantics.py b/tests/unittest/test_doclib_cache_semantics.py
index 74a5b24d..44e3404b 100644
--- a/tests/unittest/test_doclib_cache_semantics.py
+++ b/tests/unittest/test_doclib_cache_semantics.py
@@ -333,8 +333,6 @@ def test_load_pages_from_done_batches_keeps_newest_page_idx(tmp_path: Path) -> N
assert inline_text(pages[1].blocks[0].content) == "newer" # type: ignore[union-attr]
-
-
def test_managed_api_server_args_use_tier_and_selected_port_for_process_start() -> None:
assert api_server_args_for_tier("standard", host="127.0.0.1", port=16580) == [
"--tier",
@@ -343,7 +341,6 @@ def test_managed_api_server_args_use_tier_and_selected_port_for_process_start()
"127.0.0.1",
"--port",
"16580",
- "--allow-local-source",
"--no-flash",
"--preload-models",
"--log-level",
@@ -356,7 +353,6 @@ def test_managed_api_server_args_use_tier_and_selected_port_for_process_start()
"127.0.0.2",
"--port",
"16581",
- "--allow-local-source",
"--no-flash",
"--preload-models",
"--log-level",
diff --git a/tests/unittest/test_engine_dead_lifecycle.py b/tests/unittest/test_engine_dead_lifecycle.py
index e16d5e88..453161e6 100644
--- a/tests/unittest/test_engine_dead_lifecycle.py
+++ b/tests/unittest/test_engine_dead_lifecycle.py
@@ -2,8 +2,9 @@
from __future__ import annotations
+from task_test_utils import submit_task, wait_task
+
import asyncio
-import base64
from pathlib import Path
from types import SimpleNamespace
from typing import Any
@@ -60,29 +61,43 @@ def test_api_engine_failure_drains_running_and_queued_jobs(tmp_path: Path, monke
client.portal.call(make_gate)
jobs = []
for name in ("fatal", "running", "queued", "user-cancel"):
- body = {
- "tier": "flash",
- "files": [
- {"source": {"type": "inline", "name": f"{name}.pdf", "data": base64.b64encode(name.encode()).decode()}}
- ],
- }
- response = client.post("/v1/parse/jobs", json=body)
- assert response.status_code == 202, response.text
- jobs.append(response.json()["job_id"])
- assert client.delete(f"/v1/parse/jobs/{jobs[-1]}").status_code == 200
+ source = tmp_path / f"{name}.pdf"
+ source.write_bytes(name.encode())
+ jobs.append(submit_task(client, source, tier="flash"))
+ assert client.delete(jobs[-1]["status_url"]).status_code == 200
client.portal.call(gate.set)
for job in jobs:
- client.portal.call(app.state.job_store.wait_for_terminal, job, 2.0)
+ wait_task(client, job)
+
+ async def drain_tasks() -> None:
+ """等待公开父任务完成资源清理后检查内部执行槽。"""
+ await asyncio.gather(
+ *(
+ app.state.job_store.wait_for_terminal(file.execution_id, 2)
+ for record in app.state.task_store._records.values()
+ for file in record.files
+ )
+ )
+ await asyncio.gather(*(record.operation for record in app.state.task_store._records.values()))
+
+ client.portal.call(drain_tasks)
assert entered == ["fatal", "running"]
assert set(cleanup) == {"fatal", "running"}
- assert [client.get(f"/v1/parse/jobs/{job}").json()["status"] for job in jobs] == [
+ assert [client.get(job["status_url"]).json()["status"] for job in jobs] == [
"failed",
"failed",
"failed",
"canceled",
]
- for path, method in (("/v1/health", "get"), ("/v1/parse/jobs", "post")):
- response = getattr(client, method)(path, **({"json": body} if method == "post" else {}))
+ for path, method in (("/v1/health", "get"), ("/v1/tasks", "post")):
+ response = getattr(client, method)(
+ path,
+ **(
+ {"json": {"tier": "flash", "manifest": [{"client_file_id": "file", "filename": "new.pdf", "bytes": 3}]}}
+ if method == "post"
+ else {}
+ ),
+ )
assert response.status_code == 503 and response.json()["error"]["code"] == "engine_dead"
assert app.state.job_store._semaphore._value == 2
assert not app.state.job_store._tasks
@@ -175,28 +190,20 @@ def test_old_generation_failure_snapshot_remains_queryable() -> None:
"progress": {"total": 1, "completed": 0},
}
client.portal.call(pool.refresh, state)
- response = client.get(f"/v1/parse/jobs/{route.public_id}")
- assert response.status_code == 200
- assert response.json()["status"] == "failed"
- assert response.json()["files"][0]["error"]["code"] == "engine_dead"
- assert client.delete(f"/v1/parse/jobs/{route.public_id}").status_code == 409
+ from mineru.kit.router.tasks import read_task_execution
+
+ async def snapshot() -> Any:
+ """旧执行只读取已保存的失败信息,不请求替换后的 worker。"""
+ return await read_task_execution(route, pool, registry)
+
+ execution = client.portal.call(snapshot)
+ assert execution.job["status"] == "failed"
+ assert execution.job["files"][0]["error"]["code"] == "upstream_task_lost"
pool._workers = {state.worker_id: state}
- created = client.post(
- "/v1/parse/jobs",
- json={
- "tier": "standard",
- "files": [{"source": {"type": "inline", "name": "recovered.pdf", "data": base64.b64encode(b"pdf").decode()}}],
- },
- )
- assert created.status_code == 202, created.text
- new_job = created.json()["job_id"]
- assert registry.get("job", new_job).worker_id == state.worker_id
- completed = client.get(f"/v1/parse/jobs/{new_job}")
- assert completed.json()["status"] == "completed"
- output = completed.json()["files"][0]["output_files"]["markdown"]["file_id"]
- assert client.get(f"/v1/files/{output}/content").content == b"result"
+ reservation = pool.reserve_job(tier="standard", required_sources={"upload"})
+ assert reservation is not None and reservation.worker is state
+ reservation.release()
assert state.active_jobs == 0
- assert client.get(f"/v1/parse/jobs/{route.public_id}").json()["status"] == "failed"
def test_old_generation_engine_dead_response_does_not_kill_replacement() -> None:
diff --git a/tests/unittest/test_flash_csv.py b/tests/unittest/test_flash_csv.py
index 73829f55..8f037b58 100644
--- a/tests/unittest/test_flash_csv.py
+++ b/tests/unittest/test_flash_csv.py
@@ -1,5 +1,7 @@
from __future__ import annotations
+from task_test_utils import stored_source, execution_output
+
import asyncio
import json
from pathlib import Path
@@ -9,7 +11,7 @@ from docvortex.document.detection import guess_suffix_by_bytes, guess_suffix_by_
from mineru.backend.analyze import aio_doc_analyze, doc_analyze
from mineru.parser import api_server, parse, parse_async
-from mineru.parser.api_server import CreateJobRequest, FileStore
+from mineru.parser.api_server import _ParseRequest, FileStore
from mineru.render.html import render_html
from mineru.render.markdown import render_markdown
from mineru.types import TableBlock, TableBodyBlock
@@ -91,11 +93,11 @@ def test_tsv_parse_server_job_emits_structured_outputs_with_independent_suffix(t
source = tmp_path / "sample.tsv"
source.write_text("name\tage\nAlice\t30\n", encoding="utf-8")
file_store = FileStore(tmp_path / "api-files")
- request = CreateJobRequest.model_validate(
+ request = _ParseRequest.model_validate(
{
- "files": [{"source": {"type": "local", "path": str(source)}}],
+ "files": [{"source": stored_source(file_store, source)}],
"tier": "standard",
- "output_formats": ["markdown", "middle_json"],
+ "output_formats": ["default"],
}
)
record = api_server.JobStore().create(request, file_store)
@@ -106,22 +108,18 @@ def test_tsv_parse_server_job_emits_structured_outputs_with_independent_suffix(t
request,
file_store,
image_analysis=True,
- allow_local_source=True,
)
)
parsed_file = record.files[0]
assert parsed_file.status == "completed"
- assert parsed_file.output_files is not None
- assert parsed_file.output_files.markdown is not None
- assert parsed_file.output_files.middle_json is not None
- markdown_record = file_store.get_file(parsed_file.output_files.markdown.file_id)
- markdown = file_store.read_blob(markdown_record.sha256sum).decode()
+ markdown_data = execution_output(file_store, record, "markdown.md")
+ markdown = markdown_data.decode()
assert "| name | age |" in markdown
- middle_record = file_store.get_file(parsed_file.output_files.middle_json.file_id)
- middle_payload = json.loads(file_store.read_blob(middle_record.sha256sum))
+ middle_data = execution_output(file_store, record, "middle_json.json")
+ middle_payload = json.loads(middle_data)
assert middle_payload["metadata"]["file_suffix"] == "tsv"
assert middle_payload["extensions"]["mineru"]["tier"] == "flash"
assert middle_payload["extensions"]["mineru"]["parse_mode"] == "txt"
@@ -132,11 +130,11 @@ def test_csv_parse_server_job_emits_structured_outputs_with_flash_metadata(tmp_p
source = tmp_path / "sample.csv"
source.write_text("name,age\nAlice,30\n", encoding="utf-8")
file_store = FileStore(tmp_path / "api-files")
- request = CreateJobRequest.model_validate(
+ request = _ParseRequest.model_validate(
{
- "files": [{"source": {"type": "local", "path": str(source)}}],
+ "files": [{"source": stored_source(file_store, source)}],
"tier": "standard",
- "output_formats": ["markdown", "middle_json", "structured_content"],
+ "output_formats": ["default"],
}
)
record = api_server.JobStore().create(request, file_store)
@@ -147,31 +145,23 @@ def test_csv_parse_server_job_emits_structured_outputs_with_flash_metadata(tmp_p
request,
file_store,
image_analysis=True,
- allow_local_source=True,
)
)
parsed_file = record.files[0]
assert parsed_file.status == "completed"
- assert parsed_file.output_files is not None
- assert parsed_file.output_files.markdown is not None
- assert parsed_file.output_files.middle_json is not None
- assert parsed_file.output_files.structured_content is not None
-
- markdown_record = file_store.get_file(parsed_file.output_files.markdown.file_id)
- assert markdown_record.sha256sum is not None
- markdown = file_store.read_blob(markdown_record.sha256sum).decode()
+
+ markdown_data = execution_output(file_store, record, "markdown.md")
+ markdown = markdown_data.decode()
assert "| name | age |" in markdown
- middle_record = file_store.get_file(parsed_file.output_files.middle_json.file_id)
- assert middle_record.sha256sum is not None
- middle_payload = json.loads(file_store.read_blob(middle_record.sha256sum))
+ middle_data = execution_output(file_store, record, "middle_json.json")
+ middle_payload = json.loads(middle_data)
assert middle_payload["metadata"]["file_suffix"] == "csv"
assert middle_payload["extensions"]["mineru"]["tier"] == "flash"
assert middle_payload["extensions"]["mineru"]["parse_mode"] == "txt"
- structured_record = file_store.get_file(parsed_file.output_files.structured_content.file_id)
- assert structured_record.sha256sum is not None
- structured_payload = json.loads(file_store.read_blob(structured_record.sha256sum))
+ structured_data = execution_output(file_store, record, "structured_content.json")
+ structured_payload = json.loads(structured_data)
assert structured_payload["metadata"]["file_suffix"] == "csv"
assert structured_payload["pages"][0]["blocks"][0]["type"] == "table"
diff --git a/tests/unittest/test_flash_epub.py b/tests/unittest/test_flash_epub.py
index 224b80cd..9ad1da53 100644
--- a/tests/unittest/test_flash_epub.py
+++ b/tests/unittest/test_flash_epub.py
@@ -1,5 +1,7 @@
from __future__ import annotations
+from task_test_utils import stored_source, execution_output
+
import asyncio
import json
import subprocess
@@ -21,7 +23,7 @@ from mineru.doclib.core.fts import FTSManager
from mineru.doclib.services.parse_svc import ParseService
from mineru.errors import InvalidRequestError
from mineru.parser import MinerUParser, api_server, parse, parse_async
-from mineru.parser.api_server import CreateJobRequest, FileStore
+from mineru.parser.api_server import _ParseRequest, FileStore
from mineru.render import render_docx, render_html, render_markdown, render_structured_content
from mineru.types import BlockType
@@ -159,11 +161,11 @@ def test_epub_local_parse_job_emits_spine_aligned_flash_outputs(tmp_path: Path)
source = tmp_path / "book.epub"
source.write_bytes(build_epub_fixture())
file_store = FileStore(tmp_path / "api-files")
- request = CreateJobRequest.model_validate(
+ request = _ParseRequest.model_validate(
{
- "files": [{"source": {"type": "local", "path": str(source)}}],
+ "files": [{"source": stored_source(file_store, source)}],
"tier": "standard",
- "output_formats": ["markdown", "middle_json", "structured_content", "zip"],
+ "output_formats": ["default"],
}
)
record = api_server.JobStore().create(request, file_store)
@@ -173,16 +175,12 @@ def test_epub_local_parse_job_emits_spine_aligned_flash_outputs(tmp_path: Path)
request,
file_store,
image_analysis=True,
- allow_local_source=True,
)
)
parsed_file = record.files[0]
assert parsed_file.status == "completed"
- assert parsed_file.output_files is not None
- assert parsed_file.output_files.zip is not None
- middle_record = file_store.get_file(parsed_file.output_files.middle_json.file_id) # type: ignore[union-attr]
- assert middle_record.sha256sum is not None
- payload = json.loads(file_store.read_blob(middle_record.sha256sum))
+ middle_data = execution_output(file_store, record, "middle_json.json")
+ payload = json.loads(middle_data)
assert payload["metadata"]["file_suffix"] == "epub"
assert payload["extensions"]["mineru"]["tier"] == "flash"
assert payload["extensions"]["mineru"]["parse_mode"] == "txt"
@@ -195,11 +193,11 @@ def test_epub_local_parse_job_preserves_page_range_error_code(tmp_path: Path) ->
source = tmp_path / "book.epub"
source.write_bytes(build_epub_fixture())
file_store = FileStore(tmp_path / "api-files")
- request = CreateJobRequest.model_validate(
+ request = _ParseRequest.model_validate(
{
- "files": [{"source": {"type": "local", "path": str(source)}, "page_range": "2"}],
+ "files": [{"source": stored_source(file_store, source), "page_range": "2"}],
"tier": "flash",
- "output_formats": ["middle_json"],
+ "output_formats": ["default"],
}
)
record = api_server.JobStore().create(request, file_store)
@@ -209,7 +207,6 @@ def test_epub_local_parse_job_preserves_page_range_error_code(tmp_path: Path) ->
request,
file_store,
image_analysis=True,
- allow_local_source=True,
)
)
assert record.files[0].status == "failed"
diff --git a/tests/unittest/test_flash_html.py b/tests/unittest/test_flash_html.py
index 06d827eb..6e2f65e5 100644
--- a/tests/unittest/test_flash_html.py
+++ b/tests/unittest/test_flash_html.py
@@ -1,5 +1,8 @@
from __future__ import annotations
+from task_test_utils import stored_source, execution_output, build_default_zip
+
+import base64
import asyncio
import json
from io import BytesIO
@@ -22,7 +25,7 @@ from mineru.backend.analyze import aio_doc_analyze, doc_analyze
from mineru.doclib.services.parse_svc import ParseService
from mineru.errors import InvalidRequestError
from mineru.parser import ParseResult, api_server, parse, parse_async
-from mineru.parser.api_server import CreateJobRequest, FileStore
+from mineru.parser.api_server import _ParseRequest, FileStore
from mineru.render.docx import render_docx
from mineru.render.html import render_html
from mineru.render.markdown import render_markdown
@@ -138,11 +141,11 @@ def test_html_parse_server_url_preserves_http_declared_charset(tmp_path: Path, m
client = _url_stream_client(response)
monkeypatch.setattr(api_server, "httpx", SimpleNamespace(AsyncClient=lambda **_: client))
file_store = FileStore(tmp_path / "api-files")
- request = CreateJobRequest.model_validate(
+ request = _ParseRequest.model_validate(
{
"files": [{"source": {"type": "url", "url": url}}],
"tier": "standard",
- "output_formats": ["middle_json"],
+ "output_formats": ["default"],
}
)
record = api_server.JobStore().create(request, file_store)
@@ -158,10 +161,8 @@ def test_html_parse_server_url_preserves_http_declared_charset(tmp_path: Path, m
parsed_file = record.files[0]
assert parsed_file.status == "completed"
- assert parsed_file.output_files is not None and parsed_file.output_files.middle_json is not None
- middle_record = file_store.get_file(parsed_file.output_files.middle_json.file_id)
- assert middle_record.sha256sum is not None
- middle_payload = json.loads(file_store.read_blob(middle_record.sha256sum))
+ middle_data = execution_output(file_store, record, "middle_json.json")
+ middle_payload = json.loads(middle_data)
assert inline_text(middle_payload["pages"][0]["blocks"][0]["content"]) == expected
@@ -186,11 +187,11 @@ def test_html_parse_server_url_accepts_extensionless_text_html(
client = _url_stream_client(response)
monkeypatch.setattr(api_server, "httpx", SimpleNamespace(AsyncClient=lambda **_: client))
file_store = FileStore(tmp_path / "api-files")
- request = CreateJobRequest.model_validate(
+ request = _ParseRequest.model_validate(
{
"files": [{"source": {"type": "url", "url": url}}],
"tier": "standard",
- "output_formats": ["middle_json"],
+ "output_formats": ["default"],
}
)
record = api_server.JobStore().create(request, file_store)
@@ -206,10 +207,8 @@ def test_html_parse_server_url_accepts_extensionless_text_html(
parsed_file = record.files[0]
assert parsed_file.status == "completed"
- assert parsed_file.output_files is not None and parsed_file.output_files.middle_json is not None
- middle_record = file_store.get_file(parsed_file.output_files.middle_json.file_id)
- assert middle_record.sha256sum is not None
- middle_payload = json.loads(file_store.read_blob(middle_record.sha256sum))
+ middle_data = execution_output(file_store, record, "middle_json.json")
+ middle_payload = json.loads(middle_data)
assert inline_text(middle_payload["pages"][0]["blocks"][0]["content"]) == expected
@@ -227,8 +226,8 @@ def test_html_parse_server_flash_only_admits_extensionless_url(
with TestClient(app) as client:
response = client.post(
- "/v1/parse/jobs",
- json={"files": [{"source": {"type": "url", "url": "https://example.com/article"}}]},
+ "/v1/tasks",
+ json={"manifest": [{"client_file_id": "file", "filename": "article.html", "url": "https://example.com/article"}]},
)
assert response.status_code == 202
@@ -252,7 +251,7 @@ def test_html_parse_server_no_flash_rejects_extensionless_text_html_after_fetch(
monkeypatch.setattr(api_server, "httpx", SimpleNamespace(AsyncClient=lambda **_: client))
monkeypatch.setattr(api_server, "parse_async", parse_async_mock)
file_store = FileStore(tmp_path / "api-files")
- request = CreateJobRequest.model_validate(
+ request = _ParseRequest.model_validate(
{
"files": [{"source": {"type": "url", "url": url}}],
"tier": "standard",
@@ -290,11 +289,11 @@ def test_url_source_aborts_when_exceeding_max_url_bytes(tmp_path: Path, monkeypa
client = _url_stream_client(response)
monkeypatch.setattr(api_server, "httpx", SimpleNamespace(AsyncClient=lambda **_: client))
file_store = FileStore(tmp_path / "api-files")
- request = CreateJobRequest.model_validate(
+ request = _ParseRequest.model_validate(
{
"files": [{"source": {"type": "url", "url": url}}],
"tier": "standard",
- "output_formats": ["middle_json"],
+ "output_formats": ["default"],
}
)
record = api_server.JobStore().create(request, file_store)
@@ -348,18 +347,23 @@ def test_html_local_base_images_styles_and_escape_are_bounded(tmp_path: Path) ->
assert _image_body(exported.middle_json).image_path is not None
-def test_html_parse_server_local_source_keeps_relative_assets(tmp_path: Path) -> None:
- """验证 parse-server 本地来源把原目录上下文传入 HTML 模型并输出严格结果。"""
+def test_html_parse_server_upload_keeps_embedded_assets(tmp_path: Path) -> None:
+ """验证上传 HTML 的内嵌图片进入 Tasks 自包含包,不依赖客户端目录。"""
image_path = tmp_path / "pixel.png"
Image.new("RGB", (2, 2), "green").save(image_path)
source = tmp_path / "sample.html"
- source.write_text('<html><body><h1>API HTML</h1><img src="pixel.png" alt="Pixel"></body></html>', encoding="utf-8")
+ source.write_text(
+ '<html><body><h1>API HTML</h1><img src="data:image/png;base64,'
+ + base64.b64encode(image_path.read_bytes()).decode()
+ + '" alt="Pixel"></body></html>',
+ encoding="utf-8",
+ )
file_store = FileStore(tmp_path / "api-files")
- request = CreateJobRequest.model_validate(
+ request = _ParseRequest.model_validate(
{
- "files": [{"source": {"type": "local", "path": str(source)}}],
+ "files": [{"source": stored_source(file_store, source)}],
"tier": "standard",
- "output_formats": ["markdown", "middle_json", "structured_content"],
+ "output_formats": ["default"],
}
)
record = api_server.JobStore().create(request, file_store)
@@ -370,20 +374,17 @@ def test_html_parse_server_local_source_keeps_relative_assets(tmp_path: Path) ->
request,
file_store,
image_analysis=True,
- allow_local_source=True,
)
)
parsed_file = record.files[0]
assert parsed_file.status == "completed"
- assert parsed_file.output_files is not None and parsed_file.output_files.middle_json is not None
- middle_record = file_store.get_file(parsed_file.output_files.middle_json.file_id)
- assert middle_record.sha256sum is not None
- middle_payload = json.loads(file_store.read_blob(middle_record.sha256sum))
+ middle_data = execution_output(file_store, record, "middle_json.json")
+ middle_payload = json.loads(middle_data)
assert middle_payload["metadata"]["file_suffix"] == "html"
assert middle_payload["extensions"]["mineru"]["tier"] == "flash"
image_body = middle_payload["pages"][0]["blocks"][1]["content"][0]
- assert image_body["image_base64"].startswith("data:image/png;base64,")
+ assert image_body["image_path"].startswith("images/")
def test_html_remote_image_never_downloaded_and_zip_keeps_external_link(tmp_path: Path) -> None:
@@ -397,28 +398,27 @@ def test_html_remote_image_never_downloaded_and_zip_keeps_external_link(tmp_path
body = _image_body(result.middle_json)
assert body.image_url == url and not body.image_path and not body.image_base64
- archive_bytes = api_server._build_self_contained_zip_output(result)
+ archive_bytes = build_default_zip(result)
with ZipFile(BytesIO(archive_bytes)) as archive:
names = archive.namelist()
assert "markdown.md" in names and url in archive.read("markdown.md").decode()
- assert not [name for name in names if name.startswith("images/")]
+ assert not [name for name in names if name.startswith("images/") and not name.endswith("/")]
file_store = FileStore(tmp_path / "api-files")
- request = CreateJobRequest.model_validate(
+ request = _ParseRequest.model_validate(
{
- "files": [{"source": {"type": "local", "path": str(source)}}],
+ "files": [{"source": stored_source(file_store, source)}],
"tier": "standard",
- "output_formats": ["markdown", "zip"],
+ "output_formats": ["default"],
}
)
record = api_server.JobStore().create(request, file_store)
- asyncio.run(api_server._run_job(record, request, file_store, image_analysis=True, allow_local_source=True))
+ asyncio.run(api_server._run_job(record, request, file_store, image_analysis=True))
parsed_file = record.files[0]
assert parsed_file.status == "completed"
- assert parsed_file.output_files is not None and parsed_file.output_files.zip is not None
- zip_record = file_store.get_file(parsed_file.output_files.zip.file_id)
+ zip_record = file_store.get_file(record.task_result_file_id)
assert zip_record.sha256sum is not None
with ZipFile(BytesIO(file_store.read_blob(zip_record.sha256sum))) as archive:
assert url in archive.read("markdown.md").decode()
diff --git a/tests/unittest/test_flash_odf.py b/tests/unittest/test_flash_odf.py
index 42263dd0..75cb2b69 100644
--- a/tests/unittest/test_flash_odf.py
+++ b/tests/unittest/test_flash_odf.py
@@ -1,5 +1,7 @@
from __future__ import annotations
+from task_test_utils import stored_source, execution_output
+
import asyncio
import json
from io import BytesIO
@@ -19,7 +21,7 @@ from mineru.doclib.core.fts import FTSManager
from mineru.doclib.services.parse_svc import ParseService
from mineru.errors import InvalidRequestError
from mineru.parser import api_server, parse, parse_async
-from mineru.parser.api_server import CreateJobRequest, FileStore
+from mineru.parser.api_server import _ParseRequest, FileStore
@pytest.mark.parametrize(
@@ -128,11 +130,11 @@ def test_odf_parse_server_job_emits_flash_outputs(tmp_path: Path) -> None:
source = tmp_path / "sample.odt"
source.write_bytes(build_odt_fixture())
file_store = FileStore(tmp_path / "api-files")
- request = CreateJobRequest.model_validate(
+ request = _ParseRequest.model_validate(
{
- "files": [{"source": {"type": "local", "path": str(source)}}],
+ "files": [{"source": stored_source(file_store, source)}],
"tier": "standard",
- "output_formats": ["markdown", "middle_json", "structured_content"],
+ "output_formats": ["default"],
}
)
record = api_server.JobStore().create(request, file_store)
@@ -142,15 +144,12 @@ def test_odf_parse_server_job_emits_flash_outputs(tmp_path: Path) -> None:
request,
file_store,
image_analysis=True,
- allow_local_source=True,
)
)
parsed_file = record.files[0]
assert parsed_file.status == "completed"
- assert parsed_file.output_files is not None
- middle_record = file_store.get_file(parsed_file.output_files.middle_json.file_id) # type: ignore[union-attr]
- assert middle_record.sha256sum is not None
- payload = json.loads(file_store.read_blob(middle_record.sha256sum))
+ middle_data = execution_output(file_store, record, "middle_json.json")
+ payload = json.loads(middle_data)
assert payload["metadata"]["file_suffix"] == "odt"
assert payload["extensions"]["mineru"]["tier"] == "flash"
assert payload["extensions"]["mineru"]["parse_mode"] == "txt"
diff --git a/tests/unittest/test_flash_ofd.py b/tests/unittest/test_flash_ofd.py
index c48f438e..c3f45be7 100644
--- a/tests/unittest/test_flash_ofd.py
+++ b/tests/unittest/test_flash_ofd.py
@@ -1,5 +1,7 @@
from __future__ import annotations
+from task_test_utils import stored_source, execution_output
+
import asyncio
import json
from io import BytesIO
@@ -18,7 +20,7 @@ from mineru.doclib.core.fts import FTSManager
from mineru.doclib.services.parse_svc import ParseService
from mineru.errors import InvalidRequestError
from mineru.parser import MinerUParser, api_server
-from mineru.parser.api_server import CreateJobRequest, FileStore
+from mineru.parser.api_server import _ParseRequest, FileStore
from mineru.render import render_docx, render_html, render_markdown, render_structured_content
from mineru.types import BlockType
@@ -81,11 +83,11 @@ def test_ofd_parse_server_job_emits_flash_outputs(tmp_path: Path) -> None:
source = tmp_path / "sample.ofd"
source.write_bytes(_minimal_payload())
file_store = FileStore(tmp_path / "api-files")
- request = CreateJobRequest.model_validate(
+ request = _ParseRequest.model_validate(
{
- "files": [{"source": {"type": "local", "path": str(source)}}],
+ "files": [{"source": stored_source(file_store, source)}],
"tier": "standard",
- "output_formats": ["markdown", "middle_json", "structured_content", "zip"],
+ "output_formats": ["default"],
}
)
record = api_server.JobStore().create(request, file_store)
@@ -95,15 +97,12 @@ def test_ofd_parse_server_job_emits_flash_outputs(tmp_path: Path) -> None:
request,
file_store,
image_analysis=True,
- allow_local_source=True,
)
)
parsed_file = record.files[0]
assert parsed_file.status == "completed"
- assert parsed_file.output_files is not None
- middle_record = file_store.get_file(parsed_file.output_files.middle_json.file_id) # type: ignore[union-attr]
- assert middle_record.sha256sum is not None
- payload = json.loads(file_store.read_blob(middle_record.sha256sum))
+ middle_data = execution_output(file_store, record, "middle_json.json")
+ payload = json.loads(middle_data)
assert payload["metadata"]["file_suffix"] == "ofd"
assert payload["extensions"]["mineru"]["tier"] == "flash"
assert payload["extensions"]["mineru"]["parse_mode"] == "txt"
diff --git a/tests/unittest/test_http_api_example_script.py b/tests/unittest/test_http_api_example_script.py
index d872cae5..52b6e90e 100644
--- a/tests/unittest/test_http_api_example_script.py
+++ b/tests/unittest/test_http_api_example_script.py
@@ -1,51 +1,15 @@
-"""scripts/http_api_example.sh 的行为契约测试(真实 Flash 服务 + 模拟服务,无模型下载、无 GPU)。
-
-覆盖计划验收点:
-- 鉴权与同源规则(同源附加 Bearer、外部预签名地址不附加 MinerU Key、保留服务返回的上传头)
-- HTTP 200 非法响应(null/缺失 ID、未知状态)不进入轮询
-- 有界轮询(MAX_POLLS 耗尽退出 124 且输出 job_id)
-- 终态退出码(0/2/3/4),partial 先保存成功产物再退出
-- 下载 302 重定向的凭据隔离(跨 host 与同 host 不同端口均不转发 Authorization)
-"""
+"""共享真实 TCP 测试服务及统一 HTTP 示例入口回归。旧协议场景已迁入 test_task_http。"""
from __future__ import annotations
-import os
-import shutil
import socket
-import subprocess
import threading
import time
+import subprocess
+import os
from pathlib import Path
from typing import Any
-import pytest
-
-# Module-level FastAPI imports: the mock routes annotate `request: Request`, and
-# `from __future__ import annotations` resolves string annotations against
-# module globals, so function-local imports would break parameter recognition.
-from fastapi import FastAPI, Request, Response
-from fastapi.responses import JSONResponse, RedirectResponse
-
-SCRIPT = Path(__file__).resolve().parents[2] / "scripts" / "http_api_example.sh"
-
-TERMINAL_STATUSES = {"completed", "partial", "failed", "canceled"}
-
-REQUIRED_TOOLS = ("bash", "curl", "jq", "python3")
-
-pytestmark = pytest.mark.skipif(
- any(shutil.which(tool) is None for tool in REQUIRED_TOOLS),
- reason="bash/curl/jq/python3 are required (CI installs them explicitly)",
-)
-
-HTML_DOC = (
- "<!DOCTYPE html><html><head><meta charset=\"utf-8\"><title>Doc</title></head>"
- "<body><h1>Heading</h1><p>Hello MinerU http example.</p></body></html>"
-)
-
-
-# ─── helpers ────────────────────────────────────────────────────────────────
-
class _Server:
"""Run a FastAPI app on a pre-picked localhost port in a daemon thread."""
@@ -75,486 +39,31 @@ class _Server:
self._thread.join(timeout=timeout)
-def _script_env(tmp_path: Path, **extra: str) -> dict[str, str]:
- """Environment for the script subprocess: isolated from the user's MINERU_* settings."""
- env = {
- "PATH": os.environ["PATH"],
- "HOME": os.environ.get("HOME", str(tmp_path)),
- "TMPDIR": str(tmp_path),
- "MINERU_HOME": str(tmp_path / "mineru-home"),
- # Default to a tmp dir so downloads never leak into the pytest cwd (repo root);
- # an explicit OUTPUT_DIR in extra overrides this.
- "OUTPUT_DIR": str(tmp_path / "out"),
- }
- for key in ("LANG", "LC_ALL"):
- if key in os.environ:
- env[key] = os.environ[key]
- env.update({key: str(value) for key, value in extra.items()})
- return env
-
-
-def _run_script(env: dict[str, str], *args: str, timeout: float = 90.0) -> subprocess.CompletedProcess[str]:
- return subprocess.run(
- ["bash", str(SCRIPT), *args],
- env=env,
- capture_output=True,
- text=True,
- timeout=timeout,
- )
-
-
-def _output(proc: subprocess.CompletedProcess[str]) -> str:
- return proc.stdout + proc.stderr
-
-
-# ─── mock V1 service ────────────────────────────────────────────────────────
-
-
-class _MockState:
- """Configurable responses plus a record of every received request."""
-
- def __init__(self) -> None:
- self.requests: list[dict[str, str | None]] = []
- self.create_upload_response: dict[str, Any] | None = None
- self.put_status: int = 200
- self.create_job_response: dict[str, Any] | None = None
- self.job_statuses: list[str] = ["running", "completed"]
- self.terminal_files: list[dict[str, Any]] | None = None
- self.file_content: bytes = b"# MARKDOWN BYTES\n"
- self.file_redirect: str | None = None
-
- def record(self, method: str, path: str, request: Any) -> None:
- self.requests.append(
- {
- "method": method,
- "path": path,
- "authorization": request.headers.get("authorization"),
- "x_custom_upload": request.headers.get("x-custom-upload"),
- }
- )
-
- def next_job_status(self) -> str:
- if len(self.job_statuses) > 1:
- return self.job_statuses.pop(0)
- return self.job_statuses[0]
-
- def requests_to(self, method: str, prefix: str) -> list[dict[str, str | None]]:
- return [r for r in self.requests if r["method"] == method and str(r["path"]).startswith(prefix)]
-
-
-COMPLETED_FILES: list[dict[str, Any]] = [
- {
- "file_id": "file-1",
- "name": "a.html",
- "status": "completed",
- "output_files": {
- "markdown": {"file_id": "out-md"},
- "zip": {"file_id": "out-zip"},
- },
- }
-]
-
-PARTIAL_FILES: list[dict[str, Any]] = [
- COMPLETED_FILES[0],
- {"file_id": "file-2", "name": "b.html", "status": "failed", "error": {"code": "parse_failed", "message": "boom"}},
-]
-
-
-def _build_mock_app(state: _MockState) -> FastAPI:
- app = FastAPI()
-
- @app.post("/v1/uploads")
- async def create_upload(request: Request) -> Response:
- state.record("POST", "/v1/uploads", request)
- body = state.create_upload_response or {
- "id": "upload_1",
- "status": "completed",
- "upload_url": None,
- "file": {"id": "file-1", "object": "file", "bytes": 16, "filename": "a.html", "purpose": "parse"},
- }
- return JSONResponse(body)
-
- @app.put("/put-target")
- async def put_target(request: Request) -> Response:
- state.record("PUT", "/put-target", request)
- if state.put_status != 200:
- return JSONResponse({"error": {"code": "invalid_api_key"}}, status_code=state.put_status)
- return Response(status_code=200)
-
- @app.post("/v1/uploads/{upload_id}/complete")
- async def complete_upload(request: Request, upload_id: str) -> Response:
- state.record("POST", f"/v1/uploads/{upload_id}/complete", request)
- return JSONResponse(
- {
- "id": upload_id,
- "status": "completed",
- "upload_url": None,
- "file": {"id": "file-1", "object": "file", "bytes": 16, "filename": "a.html", "purpose": "parse"},
- }
- )
-
- @app.post("/v1/parse/jobs")
- async def create_job(request: Request) -> Response:
- state.record("POST", "/v1/parse/jobs", request)
- body = state.create_job_response or {"job_id": "job_1", "status": "queued"}
- return JSONResponse(body, status_code=202)
-
- @app.get("/v1/parse/jobs/{job_id}")
- async def get_job(request: Request, job_id: str) -> Response:
- state.record("GET", f"/v1/parse/jobs/{job_id}", request)
- status = state.next_job_status()
- if status in TERMINAL_STATUSES:
- return JSONResponse(
- {"job_id": job_id, "status": status, "files": state.terminal_files or COMPLETED_FILES}
- )
- return JSONResponse({"job_id": job_id, "status": status})
-
- @app.get("/v1/files/{file_id}/content")
- async def file_content(request: Request, file_id: str) -> Response:
- state.record("GET", f"/v1/files/{file_id}/content", request)
- if state.file_redirect:
- return RedirectResponse(state.file_redirect, status_code=302)
- return Response(content=state.file_content, media_type="text/markdown")
-
- @app.get("/bytes")
- async def redirected_bytes(request: Request) -> Response:
- state.record("GET", "/bytes", request)
- return Response(content=b"REDIRECTED BYTES", media_type="application/octet-stream")
-
- return app
-
-
-def mock_pair():
- """Two mock services: (state_a, state_b, server_a, server_b). Same host, different ports."""
- state_a, state_b = _MockState(), _MockState()
- server_a = _Server(_build_mock_app(state_a))
- server_b = _Server(_build_mock_app(state_b))
- server_a.start()
- server_b.start()
- yield state_a, state_b, server_a, server_b
- server_a.stop()
- server_b.stop()
-
-
-def doc(tmp_path: Path) -> Path:
- path = tmp_path / "doc.html"
- path.write_text(HTML_DOC, encoding="utf-8")
- return path
-
-
-# ─── real Flash service (no model download, no GPU) ─────────────────────────
-
-
-def real_server_factory(tmp_path: Path):
+def test_http_example_delegates_to_tasks(tmp_path: Path) -> None:
+ """原脚本入口执行同一 Tasks 客户端并保留退出码,不残留旧 HTTP 流程。"""
from mineru.parser.api_server import create_app
- servers: list[_Server] = []
-
- def _make(api_key: str | None = None) -> _Server:
- app = create_app(
- tier="flash",
- preload_models=False,
- upload_dir=str(tmp_path / "uploads"),
- api_key=api_key,
+ server = _Server(create_app(upload_dir=str(tmp_path / "api"), tier="flash"))
+ source = tmp_path / "source.html"
+ source.write_text("<h1>Unified HTTP example</h1>")
+ server.start()
+ try:
+ script = Path(__file__).resolve().parents[2] / "scripts/http_api_example.sh"
+ result = subprocess.run(
+ ["bash", str(script), str(source)],
+ text=True,
+ capture_output=True,
+ timeout=30,
+ env={
+ **os.environ,
+ "MINERU_API_URL": server.url,
+ "MINERU_TIER": "flash",
+ "MINERU_API_KEY": "",
+ "RESULT_DIR": str(tmp_path / "result"),
+ "POLL_INTERVAL": "0.01",
+ },
)
- server = _Server(app)
- server.start()
- servers.append(server)
- return server
-
- yield _make
- for server in servers:
+ assert result.returncode == 0, result.stdout + result.stderr
+ assert list((tmp_path / "result").rglob("*.zip"))
+ finally:
server.stop()
-
-
-def test_real_service_anonymous_round_trip(real_server_factory, tmp_path: Path, doc: Path) -> None:
- server = real_server_factory()
- out = tmp_path / "out"
- proc = _run_script(_script_env(tmp_path, MINERU_API_URL=server.url, OUTPUT_DIR=str(out)), str(doc))
- assert proc.returncode == 0, _output(proc)
- # artifacts are saved under the uploaded file name plus the format extension
- assert (out / "doc.html.md").is_file()
- assert (out / "doc.html.zip").is_file()
-
-
-def test_real_service_api_key_same_origin_upload(real_server_factory, tmp_path: Path, doc: Path) -> None:
- server = real_server_factory(api_key="secret-key")
- out = tmp_path / "out"
- proc = _run_script(
- _script_env(
- tmp_path,
- MINERU_API_URL=server.url,
- MINERU_API_KEY="secret-key",
- OUTPUT_DIR=str(out),
- ),
- str(doc),
- )
- assert proc.returncode == 0, _output(proc)
- assert (out / "doc.html.md").is_file()
-
-
-def test_real_service_rejects_unauthenticated_upload_content(real_server_factory, tmp_path: Path, doc: Path) -> None:
- import httpx
-
- server = real_server_factory(api_key="secret-key")
- create = httpx.post(
- f"{server.url}/v1/uploads",
- headers={"Authorization": "Bearer secret-key"},
- json={"filename": "doc.html", "bytes": len(HTML_DOC.encode()), "mime_type": "text/html", "purpose": "parse"},
- timeout=10,
- )
- assert create.status_code == 200
- upload = create.json()
- assert upload["status"] == "pending"
- unauthenticated = httpx.put(upload["upload_url"], content=HTML_DOC.encode(), timeout=10)
- assert unauthenticated.status_code == 401
-
-
-def test_real_service_deduplicated_upload_reused(real_server_factory, tmp_path: Path, doc: Path) -> None:
- server = real_server_factory()
- out = tmp_path / "out"
- env = _script_env(tmp_path, MINERU_API_URL=server.url, OUTPUT_DIR=str(out))
- first = _run_script(env, str(doc))
- assert first.returncode == 0, _output(first)
- second = _run_script(env, str(doc))
- assert second.returncode == 0, _output(second)
- assert "deduplicated" in _output(second)
-
-
-# ─── mock service: protocol and exit-code contract ──────────────────────────
-
-
-def test_mock_upload_401_stops_before_complete_and_jobs(mock_pair, tmp_path: Path, doc: Path) -> None:
- state_a, _, server_a, _ = mock_pair
- state_a.create_upload_response = {
- "id": "upload_1",
- "status": "pending",
- "upload_url": f"{server_a.url}/put-target",
- "upload_method": "PUT",
- "upload_headers": {"Content-Type": "application/octet-stream"},
- }
- state_a.put_status = 401
- proc = _run_script(_script_env(tmp_path, MINERU_API_URL=server_a.url), str(doc))
- assert proc.returncode == 1
- assert "byte upload failed" in _output(proc)
- assert state_a.requests_to("POST", "/v1/uploads/upload_1/complete") == []
- assert state_a.requests_to("POST", "/v1/parse/jobs") == []
- assert state_a.requests_to("GET", "/v1/parse/jobs/") == []
-
-
- ("statuses", "terminal_files", "expected_code"),
- [
- (["completed"], COMPLETED_FILES, 0),
- (["running", "completed"], COMPLETED_FILES, 0),
- (["partial"], PARTIAL_FILES, 2),
- (["failed"], COMPLETED_FILES, 3),
- (["canceled"], COMPLETED_FILES, 4),
- ],
-)
-def test_mock_terminal_exit_codes(
- mock_pair, tmp_path: Path, doc: Path, statuses: list[str], terminal_files: list[dict[str, Any]], expected_code: int
-) -> None:
- state_a, _, server_a, _ = mock_pair
- state_a.job_statuses = list(statuses)
- state_a.terminal_files = terminal_files
- state_a.create_upload_response = {
- "id": "upload_1",
- "status": "completed",
- "upload_url": None,
- "file": {"id": "file-1", "object": "file", "bytes": 16, "filename": "a.html", "purpose": "parse"},
- }
- out = tmp_path / "out"
- proc = _run_script(
- _script_env(tmp_path, MINERU_API_URL=server_a.url, OUTPUT_DIR=str(out), POLL_INTERVAL="0"),
- str(doc),
- )
- assert proc.returncode == expected_code, _output(proc)
- if expected_code in (0, 2):
- # partial must save the artifacts of completed files before exiting
- assert (out / "a.html.md").is_file()
- assert (out / "a.html.zip").is_file()
-
-
-def test_mock_polling_budget_exhaustion(mock_pair, tmp_path: Path, doc: Path) -> None:
- state_a, _, server_a, _ = mock_pair
- state_a.job_statuses = ["running"]
- state_a.create_upload_response = {
- "id": "upload_1",
- "status": "completed",
- "upload_url": None,
- "file": {"id": "file-1", "object": "file", "bytes": 16, "filename": "a.html", "purpose": "parse"},
- }
- proc = _run_script(
- _script_env(
- tmp_path,
- MINERU_API_URL=server_a.url,
- MAX_POLLS="2",
- POLL_INTERVAL="0",
- ),
- str(doc),
- )
- assert proc.returncode == 124
- assert "job_id=job_1" in _output(proc)
- assert "/v1/parse/jobs/job_1" in _output(proc)
-
-
- ("create_job_body",),
- [
- ({"job_id": None, "status": "queued"},),
- ({"status": "queued"},),
- ({"job_id": "job_1", "status": "exploded"},),
- ],
- ids=["null-job-id", "missing-job-id", "unknown-status"],
-)
-def test_mock_invalid_create_job_response_never_polls(
- mock_pair, tmp_path: Path, doc: Path, create_job_body: dict[str, Any]
-) -> None:
- state_a, _, server_a, _ = mock_pair
- state_a.create_upload_response = {
- "id": "upload_1",
- "status": "completed",
- "upload_url": None,
- "file": {"id": "file-1", "object": "file", "bytes": 16, "filename": "a.html", "purpose": "parse"},
- }
- state_a.create_job_response = create_job_body
- proc = _run_script(
- _script_env(tmp_path, MINERU_API_URL=server_a.url, MINERU_API_KEY="secret", POLL_INTERVAL="0"),
- str(doc),
- )
- assert proc.returncode == 1
- assert state_a.requests_to("GET", "/v1/parse/jobs/") == []
-
-
-def test_mock_completed_but_missing_artifact_fails(mock_pair, tmp_path: Path, doc: Path) -> None:
- state_a, _, server_a, _ = mock_pair
- state_a.job_statuses = ["completed"]
- state_a.terminal_files = [
- {"file_id": "file-1", "name": "a.html", "status": "completed", "output_files": {"markdown": {"file_id": "out-md"}}}
- ]
- state_a.create_upload_response = {
- "id": "upload_1",
- "status": "completed",
- "upload_url": None,
- "file": {"id": "file-1", "object": "file", "bytes": 16, "filename": "a.html", "purpose": "parse"},
- }
- proc = _run_script(
- _script_env(
- tmp_path,
- MINERU_API_URL=server_a.url,
- OUTPUT_FORMATS="markdown,zip",
- POLL_INTERVAL="0",
- ),
- str(doc),
- )
- assert proc.returncode == 1
- assert "missing" in _output(proc)
-
-
-# ─── mock service: same-origin credential isolation ─────────────────────────
-
-
-def _pending_upload(upload_url: str) -> dict[str, Any]:
- return {
- "id": "upload_1",
- "status": "pending",
- "upload_url": upload_url,
- "upload_method": "PUT",
- "upload_headers": {"Content-Type": "application/octet-stream", "x-custom-upload": "token123"},
- }
-
-
-def test_mock_external_port_upload_keeps_service_headers_without_api_key(mock_pair, tmp_path: Path, doc: Path) -> None:
- """Same host, different port counts as cross-origin: no MinerU API key, service headers kept."""
- state_a, state_b, server_a, server_b = mock_pair
- state_a.create_upload_response = _pending_upload(f"{server_b.url}/put-target")
- state_a.job_statuses = ["completed"]
- proc = _run_script(
- _script_env(
- tmp_path,
- MINERU_API_URL=server_a.url,
- MINERU_API_KEY="secret",
- OUTPUT_DIR=str(tmp_path / "out"),
- POLL_INTERVAL="0",
- ),
- str(doc),
- )
- assert proc.returncode == 0, _output(proc)
- create_calls = state_a.requests_to("POST", "/v1/uploads")
- assert create_calls and create_calls[0]["authorization"] == "Bearer secret"
- put_calls = state_b.requests_to("PUT", "/put-target")
- assert put_calls, "byte upload did not reach the external origin"
- assert put_calls[0]["authorization"] is None
- assert put_calls[0]["x_custom_upload"] == "token123"
-
-
-def test_mock_external_host_upload_without_api_key(mock_pair, tmp_path: Path, doc: Path) -> None:
- """Different hostname (127.0.0.1 vs localhost) is cross-origin: no MinerU API key."""
- state_a, state_b, server_a, server_b = mock_pair
- state_a.create_upload_response = _pending_upload(f"http://localhost:{server_b.port}/put-target")
- state_a.job_statuses = ["completed"]
- proc = _run_script(
- _script_env(tmp_path, MINERU_API_URL=server_a.url, MINERU_API_KEY="secret", POLL_INTERVAL="0"),
- str(doc),
- )
- assert proc.returncode == 0, _output(proc)
- put_calls = state_b.requests_to("PUT", "/put-target")
- assert put_calls
- assert put_calls[0]["authorization"] is None
- assert put_calls[0]["x_custom_upload"] == "token123"
-
-
-def test_mock_relative_upload_url_is_same_origin(mock_pair, tmp_path: Path, doc: Path) -> None:
- state_a, _, server_a, _ = mock_pair
- state_a.create_upload_response = _pending_upload("/put-target")
- state_a.job_statuses = ["completed"]
- proc = _run_script(
- _script_env(tmp_path, MINERU_API_URL=server_a.url, MINERU_API_KEY="secret", POLL_INTERVAL="0"),
- str(doc),
- )
- assert proc.returncode == 0, _output(proc)
- put_calls = state_a.requests_to("PUT", "/put-target")
- assert put_calls
- assert put_calls[0]["authorization"] == "Bearer secret"
- assert put_calls[0]["x_custom_upload"] == "token123"
-
-
-# ─── mock service: download redirects do not leak credentials ───────────────
-
-
[email protected]("redirect_target", ["cross-port", "cross-host"])
-def test_mock_download_redirect_isolation(mock_pair, tmp_path: Path, doc: Path, redirect_target: str) -> None:
- state_a, state_b, server_a, server_b = mock_pair
- if redirect_target == "cross-port":
- redirect_url = f"{server_b.url}/bytes"
- else:
- redirect_url = f"http://localhost:{server_b.port}/bytes"
- state_a.create_upload_response = {
- "id": "upload_1",
- "status": "completed",
- "upload_url": None,
- "file": {"id": "file-1", "object": "file", "bytes": 16, "filename": "a.html", "purpose": "parse"},
- }
- state_a.job_statuses = ["completed"]
- state_a.file_redirect = redirect_url
- out = tmp_path / "out"
- proc = _run_script(
- _script_env(
- tmp_path,
- MINERU_API_URL=server_a.url,
- MINERU_API_KEY="secret",
- OUTPUT_DIR=str(out),
- OUTPUT_FORMATS="markdown",
- POLL_INTERVAL="0",
- ),
- str(doc),
- )
- assert proc.returncode == 0, _output(proc)
- assert (out / "a.html.md").read_bytes() == b"REDIRECTED BYTES"
- redirected = state_b.requests_to("GET", "/bytes")
- assert redirected
- assert redirected[0]["authorization"] is None
diff --git a/tests/unittest/test_kit_commands.py b/tests/unittest/test_kit_commands.py
index 5b91ee3a..c1b5cefd 100644
--- a/tests/unittest/test_kit_commands.py
+++ b/tests/unittest/test_kit_commands.py
@@ -1047,11 +1047,13 @@ def test_parse_remote_requests_image_cache(monkeypatch: Any, tmp_path: Path) ->
api_key: str | None,
tier: str | None,
include_images: bool,
+ ocr_mode: str,
) -> None:
seen["api_url"] = api_url
seen["api_key"] = api_key
seen["tier"] = tier
seen["include_images"] = include_images
+ seen["ocr_mode"] = ocr_mode
def parse(self, path: Path, *, page_range: str) -> _Result:
seen["path"] = path
@@ -1085,6 +1087,7 @@ def test_parse_remote_requests_image_cache(monkeypatch: Any, tmp_path: Path) ->
"api_key": "test-key",
"tier": "standard",
"include_images": True,
+ "ocr_mode": "auto",
"path": source,
"page_range": "1",
}
diff --git a/tests/unittest/test_kit_gradio.py b/tests/unittest/test_kit_gradio.py
index 94f0c585..a77f122b 100644
--- a/tests/unittest/test_kit_gradio.py
+++ b/tests/unittest/test_kit_gradio.py
@@ -127,49 +127,19 @@ def _httpx_proxy_for_test_client(
) -> SimpleNamespace:
"""把异步 httpx 调用桥接到已启动 lifespan 的 FastAPI TestClient。"""
- class AsyncClientAdapter:
- """为 V1 async client 提供最小的 TestClient 异步适配层。"""
-
- def __init__(self, *_args: object, **_kwargs: object) -> None:
- """忽略真实网络参数并复用当前 TestClient。"""
-
- async def __aenter__(self) -> "AsyncClientAdapter":
- """进入异步上下文。"""
- return self
-
- async def __aexit__(self, *_args: object) -> None:
- """退出异步上下文,不关闭外层 TestClient。"""
-
- async def get(self, url: str, **kwargs: Any) -> httpx.Response:
- """通过 TestClient 执行 GET。"""
- return self._request("GET", url, **kwargs)
-
- async def post(self, url: str, **kwargs: Any) -> httpx.Response:
- """通过 TestClient 执行 POST。"""
- return self._request("POST", url, **kwargs)
-
- async def put(self, url: str, **kwargs: Any) -> httpx.Response:
- """通过 TestClient 执行 PUT。"""
- return self._request("PUT", url, **kwargs)
-
- def _request(self, method: str, url: str, **kwargs: Any) -> httpx.Response:
- """记录鉴权信息并把绝对 URL 转换为 ASGI 路径。"""
- headers = kwargs.get("headers") or {}
- parsed = httpx.URL(url)
- request_log.append((method, parsed.path, headers.get("Authorization")))
- return test_client.request(method, parsed.raw_path.decode("ascii"), **kwargs)
-
- return SimpleNamespace(
- AsyncClient=AsyncClientAdapter,
- Client=httpx.Client,
- HTTPError=httpx.HTTPError,
- InvalidURL=httpx.InvalidURL,
- Response=httpx.Response,
- Timeout=httpx.Timeout,
- TimeoutException=httpx.TimeoutException,
- TransportError=httpx.TransportError,
- URL=httpx.URL,
- )
+ def handle(request: httpx.Request) -> httpx.Response:
+ """使用真实 HTTPX 流协议桥接 ASGI,并记录鉴权和任务路径。"""
+ request_log.append((request.method, request.url.path, request.headers.get("authorization")))
+ response = test_client.request(request.method, str(request.url), headers=dict(request.headers), content=request.content)
+ headers = {key: value for key, value in response.headers.items() if key not in {"content-encoding", "content-length"}}
+ return httpx.Response(response.status_code, headers=headers, content=response.content)
+
+ def async_client(**kwargs: Any) -> httpx.AsyncClient:
+ """保留真正的异步上传与下载方法,只替换传输层。"""
+ kwargs["transport"] = httpx.MockTransport(handle)
+ return httpx.AsyncClient(**kwargs)
+
+ return SimpleNamespace(**{**vars(httpx), "AsyncClient": async_client})
def test_webui_command_is_registered_and_help_is_available() -> None:
@@ -299,9 +269,9 @@ def test_gradio_url_normalization_and_capability_dataclass() -> None:
normalize_v1_base_url("https://user:[email protected]/api")
with pytest.raises(V1ArtifactError, match="Invalid V1 API URL"):
normalize_v1_base_url("http://example.test:not-a-port")
- capabilities = V1ServerCapabilities("http://example.test", ("flash",), ("zip",), ("file_id",))
+ capabilities = V1ServerCapabilities("http://example.test", ("flash",), ("default",), ("upload",))
assert capabilities.tiers == ("flash",)
- assert capabilities.output_formats == ("zip",)
+ assert capabilities.output_formats == ("default",)
def test_v1_client_uses_environment_api_key_when_cli_value_is_omitted(monkeypatch: pytest.MonkeyPatch) -> None:
@@ -325,8 +295,8 @@ def test_v1_client_discovers_health_and_tiers_and_sends_api_key() -> None:
json={
"status": "ok",
"features": {
- "output_formats": ["markdown", "zip"],
- "sources": ["file_id", "inline"],
+ "task_output_formats": ["default", "html"],
+ "sources": ["upload", "url"],
},
},
request=request,
@@ -376,13 +346,13 @@ def test_v1_client_rejects_server_without_zip() -> None:
if request.url.path.endswith("/health"):
return httpx.Response(
200,
- json={"status": "ok", "features": {"output_formats": ["middle_json"], "sources": ["file_id"]}},
+ json={"status": "ok", "features": {"task_output_formats": ["html"], "sources": ["upload"]}},
request=request,
)
return httpx.Response(200, json={"data": [{"id": "flash"}]}, request=request)
client = V1ArtifactClient(api_url="http://example.test", transport=httpx.MockTransport(handler))
- with pytest.raises(V1ArtifactError, match="does not advertise zip"):
+ with pytest.raises(V1ArtifactError, match="does not advertise default"):
asyncio.run(client.discover())
@@ -390,7 +360,7 @@ def test_v1_client_rejects_server_without_zip() -> None:
("sources", "tiers", "error_code"),
[
(("inline", "url"), ("flash",), "unsupported_source"),
- (("file_id",), ("experimental",), "tier_unavailable"),
+ (("upload",), ("experimental",), "tier_unavailable"),
],
)
def test_v1_client_rejects_missing_upload_or_tier_capability(
@@ -405,7 +375,7 @@ def test_v1_client_rejects_missing_upload_or_tier_capability(
if request.url.path.endswith("/health"):
return httpx.Response(
200,
- json={"status": "ok", "features": {"output_formats": ["zip"], "sources": list(sources)}},
+ json={"status": "ok", "features": {"task_output_formats": ["default"], "sources": list(sources)}},
request=request,
)
return httpx.Response(200, json={"data": [{"id": tier} for tier in tiers]}, request=request)
@@ -426,7 +396,7 @@ def test_v1_client_checks_requested_tier_after_first_discovery(tmp_path: Path) -
if request.url.path.endswith("/health"):
return httpx.Response(
200,
- json={"status": "ok", "features": {"output_formats": ["zip"], "sources": ["file_id"]}},
+ json={"status": "ok", "features": {"task_output_formats": ["default"], "sources": ["upload"]}},
request=request,
)
return httpx.Response(200, json={"data": [{"id": "flash"}]}, request=request)
@@ -458,16 +428,18 @@ def test_v1_client_full_asgi_upload_job_poll_and_zip(
clock[0] += 0.25
return ParseResult(middle_json=_middle_json(with_image=False), _model_output=_model_json())
- original_zip = parser_api_server._build_self_contained_zip_output
+ from mineru.parser import task_outputs
+
+ original_zip = task_outputs.build_task_output
- def export(result: ParseResult) -> bytes:
+ def export(result: ParseResult, formats: Any, directory: Path, **kwargs: Any) -> Path:
"""真实生成 ZIP,仅用确定性时钟模拟结果打包耗时。"""
- payload = original_zip(result)
+ payload = original_zip(result, formats, directory, **kwargs)
clock[0] += export_seconds
return payload
monkeypatch.setattr(parser_api_server, "parse_async", fake_parse_async)
- monkeypatch.setattr(parser_api_server, "_build_self_contained_zip_output", export)
+ monkeypatch.setattr(task_outputs, "build_task_output", export)
api = parser_api_server.create_app(
upload_dir=str(tmp_path / "api"),
tier="flash",
@@ -488,9 +460,8 @@ def test_v1_client_full_asgi_upload_job_poll_and_zip(
paths = [path for _method, path, _authorization in request_log]
assert "/v1/health" in paths
assert "/v1/tiers" in paths
- assert "/v1/uploads" in paths
- assert any(path.startswith("/v1/uploads/upload_") and path.endswith("/content") for path in paths)
- assert "/v1/parse/jobs" in paths
+ assert any(path.startswith("/v1/tasks/") and "/uploads/" in path for path in paths)
+ assert "/v1/tasks" in paths
assert any(path.startswith("/v1/files/file-") and path.endswith("/content") for path in paths)
protected_requests = [item for item in request_log if item[1] not in {"/v1/health", "/v1/tiers"}]
assert protected_requests and all(authorization == "Bearer secret" for _method, _path, authorization in protected_requests)
@@ -557,7 +528,7 @@ def test_gradio_routes_real_v1_jobs_and_isolates_remote_auth(monkeypatch: pytest
flash_result = asyncio.run(routed.parse_file(source, tier="flash", page_range=""))
assert flash_result._model_output.pages == [[{"raw": "model"}]]
assert requests and all(host == "local.test" and auth is None for host, _method, _path, auth in requests)
- assert any(method == "POST" and path == "/v1/parse/jobs" for _host, method, path, _auth in requests)
+ assert any(method == "POST" and path == "/v1/tasks" for _host, method, path, _auth in requests)
requests.clear()
remote_result = asyncio.run(routed.parse_file(source, tier="standard", page_range="1"))
@@ -565,7 +536,7 @@ def test_gradio_routes_real_v1_jobs_and_isolates_remote_auth(monkeypatch: pytest
assert requests and all(
host == "remote.test" and auth == "Bearer remote-secret" for host, _method, _path, auth in requests
)
- assert any(method == "POST" and path == "/v1/parse/jobs" for _host, method, path, _auth in requests)
+ assert any(method == "POST" and path == "/v1/tasks" for _host, method, path, _auth in requests)
@pytest.mark.parametrize("failure", [None, V1ArtifactError("remote failed", code="parse_failed"), asyncio.CancelledError()])
@@ -573,7 +544,7 @@ def test_gradio_router_preserves_remote_flash_results_errors_and_cancellation(fa
"""验证远程已有 Flash 时优先远程,并原样传递结果、失败和取消而不重试本地。"""
result = ParseResult(middle_json=_middle_json(with_image=False))
primary = Mock(spec=V1ArtifactClient)
- primary.capabilities = V1ServerCapabilities("http://remote.test", ("standard", "flash"), ("zip",), ("file_id",))
+ primary.capabilities = V1ServerCapabilities("http://remote.test", ("standard", "flash"), ("default",), ("upload",))
primary.parse_file = AsyncMock(return_value=result, side_effect=failure)
local = Mock(spec=V1ArtifactClient)
routed = GradioArtifactClient(primary, local_flash=local)
@@ -600,9 +571,9 @@ def test_gradio_router_preserves_remote_flash_results_errors_and_cancellation(fa
def test_gradio_router_rejects_unavailable_tiers(tier: str, code: str) -> None:
"""验证路由层只补充 Flash,拒绝未声明的 Advanced 及未知档位。"""
primary = Mock(spec=V1ArtifactClient)
- primary.capabilities = V1ServerCapabilities("http://remote.test", ("standard",), ("zip",), ("file_id",))
+ primary.capabilities = V1ServerCapabilities("http://remote.test", ("standard",), ("default",), ("upload",))
local = Mock(spec=V1ArtifactClient)
- local.capabilities = V1ServerCapabilities("http://local.test", ("flash", "advanced"), ("zip",), ("file_id",))
+ local.capabilities = V1ServerCapabilities("http://local.test", ("flash", "advanced"), ("default",), ("upload",))
routed = GradioArtifactClient(primary, local_flash=local)
assert routed.capabilities.tiers == ("flash", "standard")
with pytest.raises(V1ArtifactError) as error:
@@ -682,7 +653,7 @@ def test_gradio_ocr_reaches_analysis_through_real_v1_jobs(monkeypatch: pytest.Mo
asyncio.run(convert_requests())
assert not hasattr(api.state, "ocr_mode")
assert received_modes == ["ocr", "auto", "ocr", "auto"]
- assert sum(method == "POST" and path == "/v1/parse/jobs" for method, path, _ in request_log) == 4
+ assert sum(method == "POST" and path == "/v1/tasks" for method, path, _ in request_log) == 4
def test_v1_client_preserves_v1_error_code(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
@@ -706,7 +677,7 @@ def test_v1_client_preserves_v1_error_code(monkeypatch: pytest.MonkeyPatch, tmp_
monkeypatch.setattr(gradio_client, "MinerUApiParser", FailingParser)
client = V1ArtifactClient(api_url="http://example.test")
- client._capabilities = V1ServerCapabilities("http://example.test", ("flash",), ("zip",), ("file_id",))
+ client._capabilities = V1ServerCapabilities("http://example.test", ("flash",), ("default",), ("upload",))
with pytest.raises(V1ArtifactError) as error:
asyncio.run(client.parse_file(source, tier="flash", page_range="999"))
assert error.value.code == "page_range_invalid"
@@ -751,8 +722,9 @@ def test_v1_client_parse_uses_api_parser_zip_contract(
client._capabilities = V1ServerCapabilities(
"http://example.test",
("standard",),
- ("zip",),
- ("file_id",),
+ ("default",),
+ ("upload",),
+ model_output=True,
)
statuses: list[str | ParseStatusUpdate] = []
result = asyncio.run(
@@ -1074,7 +1046,7 @@ def test_gradio_file_types_page_range_and_header_follow_new_contract(tmp_path: P
def test_gradio_ocr_control_visibility_reset_and_event_binding(tmp_path: Path) -> None:
"""验证 PDF 专属开关的缺省值、文件重置、清除绑定和独立于 tier 的可见性。"""
- capabilities = V1ServerCapabilities("http://127.0.0.1:1", ("flash", "standard"), ("zip",), ("file_id",))
+ capabilities = V1ServerCapabilities("http://127.0.0.1:1", ("flash", "standard"), ("default",), ("upload",))
demo = build_gradio_app(Mock(), capabilities, output_root=tmp_path, enable_example=False)
checkbox = next(block for block in demo.blocks.values() if block.__class__.__name__ == "Checkbox")
assert checkbox.label.key == "mineru.force_ocr" and checkbox.value is False and checkbox.visible is False
@@ -1095,7 +1067,7 @@ def test_gradio_ocr_control_visibility_reset_and_event_binding(tmp_path: Path) -
def test_gradio_examples_render_only_with_local_files(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
"""验证示例块仅在启动目录存在受支持文件时渲染,并把点击绑定到上传组件。"""
- capabilities = V1ServerCapabilities("http://127.0.0.1:1", ("flash", "standard"), ("zip",), ("file_id",))
+ capabilities = V1ServerCapabilities("http://127.0.0.1:1", ("flash", "standard"), ("default",), ("upload",))
def find_examples(demo: Any) -> list[Any]:
return [block for block in demo.blocks.values() if getattr(block, "elem_id", None) == "mineru-kit-examples"]
@@ -1132,7 +1104,7 @@ def test_gradio_non_pdf_ignores_hidden_force_ocr(tmp_path: Path, suffix: str) ->
source = tmp_path / f"source{suffix}"
source.write_bytes(b"placeholder")
client = SimpleNamespace(parse_file=AsyncMock(side_effect=V1ArtifactError("stop after request")))
- capabilities = V1ServerCapabilities("http://127.0.0.1:1", ("flash",), ("zip",), ("file_id",))
+ capabilities = V1ServerCapabilities("http://127.0.0.1:1", ("flash",), ("default",), ("upload",))
demo = build_gradio_app(client, capabilities, output_root=tmp_path / "output", enable_example=False)
handler = next(fn.fn for fn in demo.fns.values() if fn.name == "convert_handler")
@@ -1219,7 +1191,7 @@ def test_gradio_tier_slider_uses_available_tiers_and_preserves_selection(
tmp_path: Path, tiers: tuple[str, ...], default_position: int, default_tier: str, maximum: int
) -> None:
"""验证乱序、缺档及单档位下的原生滑块默认值与文件切换、清除行为。"""
- capabilities = V1ServerCapabilities("http://127.0.0.1:1", tiers, ("zip",), ("file_id",))
+ capabilities = V1ServerCapabilities("http://127.0.0.1:1", tiers, ("default",), ("upload",))
demo = build_gradio_app(
V1ArtifactClient(api_url=capabilities.base_url), capabilities, output_root=tmp_path, enable_example=False
)
@@ -1263,7 +1235,7 @@ def test_gradio_conversion_enforces_file_type_tier(tmp_path: Path, extension: st
source = tmp_path / f"source.{extension.upper() if uppercase else extension}"
source.write_bytes(b"placeholder")
client = SimpleNamespace(parse_file=AsyncMock(side_effect=V1ArtifactError("stop after request")))
- capabilities = V1ServerCapabilities("http://127.0.0.1:1", ("standard", "flash"), ("zip",), ("file_id",))
+ capabilities = V1ServerCapabilities("http://127.0.0.1:1", ("standard", "flash"), ("default",), ("upload",))
demo = build_gradio_app(client, capabilities, output_root=tmp_path / "output", enable_example=False)
handler = next(fn.fn for fn in demo.fns.values() if fn.name == "convert_handler")
@@ -1285,7 +1257,7 @@ def test_gradio_flash_only_input_requires_available_flash(tmp_path: Path, tiers:
source = tmp_path / "source.csv"
source.write_text("name,value\na,1\n", encoding="utf-8")
client = SimpleNamespace(parse_file=AsyncMock())
- capabilities = V1ServerCapabilities("http://127.0.0.1:1", tiers, ("zip",), ("file_id",))
+ capabilities = V1ServerCapabilities("http://127.0.0.1:1", tiers, ("default",), ("upload",))
demo = build_gradio_app(client, capabilities, output_root=tmp_path / "output", enable_example=False)
handler = next(fn.fn for fn in demo.fns.values() if fn.name == "convert_handler")
@@ -1348,7 +1320,7 @@ def test_gradio_conversion_forwards_page_range_and_enables_fresh_downloads(
"""收集 Gradio 普通转换请求的完整输出。"""
return [await handler(str(source), position, " 1 ", force_ocr)]
- capabilities = V1ServerCapabilities("http://127.0.0.1:1", tiers, ("zip",), ("file_id",))
+ capabilities = V1ServerCapabilities("http://127.0.0.1:1", tiers, ("default",), ("upload",))
client = FakeClient()
demo = build_gradio_app(
client, # type: ignore[arg-type]
@@ -1381,7 +1353,7 @@ def test_gradio_epub_conversion_preserves_source_preview(tmp_path: Path) -> None
status_callback("Processing on server...")
return ParseResult(middle_json=_middle_json(with_image=False, file_suffix="epub"))
- capabilities = V1ServerCapabilities("http://127.0.0.1:1", ("flash",), ("zip",), ("file_id",))
+ capabilities = V1ServerCapabilities("http://127.0.0.1:1", ("flash",), ("default",), ("upload",))
demo = build_gradio_app(EpubClient(), capabilities, output_root=tmp_path / "output", enable_example=False)
handler = next(fn.fn for fn in demo.fns.values() if fn.name == "convert_handler")
@@ -1410,7 +1382,7 @@ def test_gradio_conversion_rejects_invalid_tier_position(tmp_path: Path, tiers:
source = tmp_path / "demo.csv"
source.write_text("name,value\na,1\n", encoding="utf-8")
client = SimpleNamespace(parse_file=AsyncMock())
- capabilities = V1ServerCapabilities("http://127.0.0.1:1", tiers, ("zip",), ("file_id",))
+ capabilities = V1ServerCapabilities("http://127.0.0.1:1", tiers, ("default",), ("upload",))
demo = build_gradio_app(client, capabilities, output_root=tmp_path, enable_example=False)
handler = next(fn.fn for fn in demo.fns.values() if fn.name == "convert_handler")
@@ -1442,7 +1414,7 @@ def test_gradio_conversion_failure_clears_previous_downloads(tmp_path: Path) ->
updates = [await handler(str(source), 0, "")]
return updates[-1]
- capabilities = V1ServerCapabilities("http://127.0.0.1:1", ("flash",), ("zip",), ("file_id",))
+ capabilities = V1ServerCapabilities("http://127.0.0.1:1", ("flash",), ("default",), ("upload",))
demo = build_gradio_app(
FailingClient(), # type: ignore[arg-type]
capabilities,
@@ -1487,7 +1459,7 @@ def test_gradio_local_queue_cancellation_releases_slot_and_keeps_sessions_isolat
status_callback(STATUS_DOWNLOADING_RESULT)
return ParseResult(middle_json=_middle_json(with_image=False))
- capabilities = V1ServerCapabilities("http://127.0.0.1:1", ("flash",), ("zip",), ("file_id",))
+ capabilities = V1ServerCapabilities("http://127.0.0.1:1", ("flash",), ("default",), ("upload",))
demo = build_gradio_app(WaitingClient(), capabilities, output_root=tmp_path / "output", enable_example=False)
handler_fn = next(fn for fn in demo.fns.values() if fn.name == "convert_handler")
assert handler_fn.concurrency_limit is None
@@ -1565,7 +1537,7 @@ def test_gradio_output_failure_stops_timer_and_allows_next_conversion(tmp_path:
raise OSError("output disk unavailable")
monkeypatch.setattr(gradio_app, "persist_parse_result", fail_persistence)
- capabilities = V1ServerCapabilities("http://127.0.0.1:1", ("flash",), ("zip",), ("file_id",))
+ capabilities = V1ServerCapabilities("http://127.0.0.1:1", ("flash",), ("default",), ("upload",))
demo = build_gradio_app(CompletedClient(), capabilities, output_root=tmp_path, enable_example=False)
handler = next(fn.fn for fn in demo.fns.values() if fn.name == "convert_handler")
@@ -1680,7 +1652,7 @@ def test_gradio_slow_precheck_keeps_polling_available_and_queues_other_sessions(
monkeypatch.setattr(gradio_app, "_effective_page_range", precheck)
client = SimpleNamespace(parse_file=AsyncMock(side_effect=V1ArtifactError("test finished")))
- capabilities = V1ServerCapabilities("http://127.0.0.1:1", ("flash",), ("zip",), ("file_id",))
+ capabilities = V1ServerCapabilities("http://127.0.0.1:1", ("flash",), ("default",), ("upload",))
demo = build_gradio_app(client, capabilities, output_root=tmp_path / "output", enable_example=False)
handler = next(fn.fn for fn in demo.fns.values() if fn.name == "convert_ui")
poll = next(fn.fn for fn in demo.fns.values() if fn.name == "read_conversion_status")
diff --git a/tests/unittest/test_kit_gradio_image_names.py b/tests/unittest/test_kit_gradio_image_names.py
index 93eb16a4..92d8db2d 100644
--- a/tests/unittest/test_kit_gradio_image_names.py
+++ b/tests/unittest/test_kit_gradio_image_names.py
@@ -22,7 +22,7 @@ from mineru.kit.gradio.artifacts import (
persist_parse_result,
render_download,
)
-from mineru.parser.api_server import _build_self_contained_zip_output
+from task_test_utils import build_default_zip
from mineru.parser.base import ParseResult
from mineru.types import (
AlgorithmBodyBlock,
@@ -140,16 +140,24 @@ def test_parent_names_table_images_and_repeated_materialization(tmp_path: Path)
assert {path.name: path.read_bytes() for path in (artifacts.root / "images").iterdir()} == before
for format_name in ("markdown", "json", "latex"):
with zipfile.ZipFile(render_download(artifacts.as_state(), format_name)) as archive:
- images = {Path(name).name: archive.read(name) for name in archive.namelist() if name.startswith("images/")}
+ images = {
+ Path(name).name: archive.read(name)
+ for name in archive.namelist()
+ if name.startswith("images/") and not name.endswith("/")
+ }
assert images == before
structured = json.loads(artifacts.structured_content_path.read_text())
assert structured["pages"][0]["blocks"][2]["image_source"] == "images/page_0_table_2.png"
# 同一份解析结果经 CLI/API 保存,图片名称、字节和对应正文必须与 Gradio 一致。
cli_zip = tmp_path / "cli.zip"
save_parse_result(ParseResult(middle), cli_zip, "zip")
- for archive_source in (cli_zip, BytesIO(_build_self_contained_zip_output(ParseResult(middle)))):
+ for archive_source in (cli_zip, BytesIO(build_default_zip(ParseResult(middle)))):
with zipfile.ZipFile(archive_source) as archive:
- assert {Path(name).name: archive.read(name) for name in archive.namelist() if name.startswith("images/")} == before
+ assert {
+ Path(name).name: archive.read(name)
+ for name in archive.namelist()
+ if name.startswith("images/") and not name.endswith("/")
+ } == before
assert archive.read("markdown.md").decode() == artifacts.markdown_path.read_text(encoding="utf-8")
assert json.loads(archive.read("structured_content.json")) == structured
assert json.loads(archive.read("middle_json.json")) == json.loads(artifacts.middle_json_path.read_text())
diff --git a/tests/unittest/test_parse_result_assets.py b/tests/unittest/test_parse_result_assets.py
index ad6f907c..095f42f7 100644
--- a/tests/unittest/test_parse_result_assets.py
+++ b/tests/unittest/test_parse_result_assets.py
@@ -2,13 +2,13 @@
from __future__ import annotations
-import asyncio
+from task_test_utils import build_default_zip
+
import base64
import json
import zipfile
from io import BytesIO
from pathlib import Path
-from unittest.mock import AsyncMock
import pytest
from bs4 import BeautifulSoup
@@ -18,7 +18,7 @@ from reportlab.pdfgen.canvas import Canvas
from mineru.kit.common import save_parse_result
from mineru.kit.gradio import artifacts as gradio_artifacts
-from mineru.parser import api_client, api_server
+from mineru.parser import api_client
from mineru.parser.base import ParseResult
from mineru.parser.writer import DataWriter
from mineru.render import PdfLayout, render_pdf
@@ -78,15 +78,15 @@ def _entries(payload: bytes) -> dict[str, bytes]:
"""读取完整 ZIP 文件集合以比较多次保存的实际内容。"""
with zipfile.ZipFile(BytesIO(payload)) as archive:
assert len(archive.namelist()) == len(set(archive.namelist()))
- return {name: archive.read(name) for name in archive.namelist()}
+ return {name: archive.read(name) for name in archive.namelist() if not name.endswith("/")}
def test_save_externalizes_assets_without_changing_json_only_contract(tmp_path: Path) -> None:
"""API 与 CLI 使用相同保存入口,协议与所有消费格式均引用同一份素材。"""
result, image, _ = _result()
before = result.middle_json.to_json()
- entries = _entries(api_server._build_self_contained_zip_output(result))
- assert entries == _entries(api_server._build_self_contained_zip_output(result))
+ entries = _entries(build_default_zip(result))
+ assert entries == _entries(build_default_zip(result))
assert set(entries) == {
"middle_json.json",
"markdown.md",
@@ -118,21 +118,12 @@ def test_server_zip_roundtrip_restores_direct_and_markup_images(
) -> None:
"""同步异步均消费真实服务端 ZIP,恢复 HTML 时保持属性、引号和字节。"""
result, image, markup = _result()
- payload = api_server._build_self_contained_zip_output(result)
- monkeypatch.setattr(api_client, "_download_bytes", lambda *args: payload)
- monkeypatch.setattr(api_client, "_async_download_bytes", AsyncMock(return_value=payload))
- parser = api_client.MinerUApiParser(
- api_url="http://localhost:8000",
- tier="basic",
+ payload = build_default_zip(result)
+ restored = api_client._parse_result_from_zip_bytes(
+ payload,
include_images=include_images,
include_model_output=True,
)
- job = {"status": "completed", "files": [{"output_files": {"zip": {"file_id": "file-test"}}}]}
- restored = (
- asyncio.run(api_client._async_parse_result_from_job(job, "sample.pdf", parser))
- if asynchronous
- else api_client._parse_result_from_job(job, "sample.pdf", parser)
- )
assert restored._model_output is None
assert restored.middle_json.extensions == result.middle_json.extensions
body = restored.pages[0].blocks[0].content[0]
@@ -140,7 +131,7 @@ def test_server_zip_roundtrip_restores_direct_and_markup_images(
assert body.image_path is None
assert base64.b64decode(body.image_base64.split(",", 1)[1]) == image
assert body.content == markup
- assert _entries(api_server._build_self_contained_zip_output(restored)) == _entries(payload)
+ assert _entries(build_default_zip(restored)) == _entries(payload)
else:
assert body.image_base64 is None and body.image_path == "images/page_4_table_2.png"
assert "images/page_4_table_image_2_1.png" in body.content
@@ -155,7 +146,7 @@ def test_gradio_pdf_uses_packaged_images_without_cropping(
"""API 往返及 Gradio 物化后,在源 PDF 不可用时仍保持原布局像素。"""
result, image, _ = _result(angle)
restored = api_client._parse_result_from_zip_bytes(
- api_server._build_self_contained_zip_output(result),
+ build_default_zip(result),
include_images=True,
include_model_output=False,
)
@@ -275,7 +266,7 @@ def test_save_and_restore_leave_code_image_literals_unchanged() -> None:
}
)
)
- packed = api_server._build_self_contained_zip_output(result)
+ packed = build_default_zip(result)
assert len([name for name in _entries(packed) if name.startswith("images/")]) == 2
restored = api_client._parse_result_from_zip_bytes(packed, include_images=True, include_model_output=False)
assert restored.pages[1].blocks[0].content[0].content == literal
diff --git a/tests/unittest/test_parser_api_contract.py b/tests/unittest/test_parser_api_contract.py
index 92390a8b..2d5c98ed 100644
--- a/tests/unittest/test_parser_api_contract.py
+++ b/tests/unittest/test_parser_api_contract.py
@@ -1,22 +1,15 @@
from docvortex.schema import Producer
from mineru.integrations.docvortex import build_metadata
import asyncio
-import base64
import importlib
import inspect
-import io
-import json
import logging
import subprocess
import sys
-import time
-import types
-import zipfile
from pathlib import Path
from typing import Any
from unittest.mock import patch
-import httpx
import pytest
from click.testing import CliRunner
from fastapi.testclient import TestClient
@@ -27,21 +20,17 @@ import mineru.parser.api_server as api_server
import mineru.parser.tier as parser_tier
from mineru.config import VlmConfig
from mineru.parser import MIDDLE_JSON_SCHEMA_VERSION
-from mineru.parser.api_client import MinerUApiParser, _pages_from_middle_json, _parse_result_from_job, should_trust_env_for_url
+from mineru.parser.api_client import MinerUApiParser, _pages_from_middle_json, should_trust_env_for_url
from mineru.parser.api_server import (
- CreateJobRequest,
- CreateUploadRequest,
+ _ParseRequest,
FileParseInfo,
FileStore,
HealthResponse,
- JobLinks,
- JobListItem,
- OutputFileRef,
- OutputFiles,
create_app,
main,
)
from mineru.parser.base import ParseResult
+from task_test_utils import stored_source
from mineru.types import (
ModelJson,
DEFAULT_QUALITY_TIER_SELECTION_ORDER,
@@ -53,9 +42,6 @@ from mineru.types import (
TIER_ORDER,
TIERS,
TIERS_BY_SERVER_TIER,
- BlockType,
- ImageBlock,
- ImageBodyBlock,
MiddleJson,
PageInfo,
ServerTier,
@@ -97,7 +83,6 @@ def _full_middle_json(*pages: PageInfo) -> MiddleJson:
def _model_payload() -> dict[str, Any]:
"""构造 ZIP 内的完整分析协议,保留原始模型块用于传输回归。"""
- from mineru.types import ModelJson
return ModelJson(
pages=[[{"raw": "model"}]],
@@ -448,198 +433,12 @@ def test_implicit_tier_selection_excludes_advanced() -> None:
assert select_parsing_rule_tier(("advanced", "flash")) == "flash"
-def test_api_client_builds_file_page_range_without_options(tmp_path: Path) -> None:
- pdf = tmp_path / "demo.pdf"
- pdf.write_bytes(b"%PDF-1.7\n")
-
- parser = MinerUApiParser(api_url="http://localhost:8000", tier="standard")
- payload = parser._build_payload({"type": "local", "path": str(pdf)}, "1,3-5")
-
- assert payload["output_formats"] == ["middle_json"]
- assert payload["files"] == [{"source": {"type": "local", "path": str(pdf)}, "page_range": "1,3-5"}]
- assert "options" not in payload["files"][0]
-
-
-def test_api_client_uses_file_id_when_local_server_does_not_advertise_local_source(
- tmp_path: Path,
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
- pdf = tmp_path / "demo.pdf"
- pdf.write_bytes(b"%PDF-1.7\n")
- parser = MinerUApiParser(api_url="http://localhost:8000", tier="standard")
- monkeypatch.setattr(parser, "_upload", lambda _path: "file_1")
-
- class _Response:
- status_code = 200
- text = ""
-
- def json(self) -> dict[str, object]:
- return {"features": {"sources": ["file_id", "url", "inline"]}}
-
- class _Client:
- def __init__(self, *, timeout: object, trust_env: bool, **_: object) -> None:
- assert trust_env is False
-
- def __enter__(self) -> "_Client":
- return self
-
- def __exit__(self, *args: object) -> None:
- return None
-
- def get(self, url: str, headers: dict[str, str]) -> _Response:
- assert url == "http://localhost:8000/v1/health"
- assert headers == {}
- return _Response()
-
- monkeypatch.setattr("mineru.parser.api_client.httpx.Client", _Client)
-
- source = parser._build_source(pdf)
-
- assert source == {"type": "file_id", "file_id": "file_1"}
-
-
-def test_api_client_uses_local_source_when_health_advertises_it(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
- pdf = tmp_path / "demo.pdf"
- pdf.write_bytes(b"%PDF-1.7\n")
- parser = MinerUApiParser(api_url="http://localhost:8000", tier="standard")
- monkeypatch.setattr(parser, "_upload", lambda _path: pytest.fail("local source should not upload"))
-
- class _Response:
- status_code = 200
- text = ""
-
- def json(self) -> dict[str, object]:
- return {"features": {"sources": ["file_id", "url", "inline", "local"]}}
-
- class _Client:
- def __init__(self, **_: object) -> None:
- pass
-
- def __enter__(self) -> "_Client":
- return self
-
- def __exit__(self, *args: object) -> None:
- return None
-
- def get(self, url: str, headers: dict[str, str]) -> _Response:
- return _Response()
-
- monkeypatch.setattr("mineru.parser.api_client.httpx.Client", _Client)
-
- source = parser._build_source(pdf)
-
- assert source == {"type": "local", "path": str(pdf)}
-
-
-def test_async_api_client_uses_file_id_when_local_server_does_not_advertise_local_source(
- tmp_path: Path,
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
- pdf = tmp_path / "demo.pdf"
- pdf.write_bytes(b"%PDF-1.7\n")
- parser = MinerUApiParser(api_url="http://localhost:8000", tier="standard")
-
- async def _async_upload(_path: Path) -> str:
- return "file_1"
-
- monkeypatch.setattr(parser, "_async_upload", _async_upload)
-
- class _Response:
- status_code = 200
- text = ""
-
- def json(self) -> dict[str, object]:
- return {"features": {"sources": ["file_id", "url", "inline"]}}
-
- class _AsyncClient:
- def __init__(self, *, timeout: object, trust_env: bool, **_: object) -> None:
- assert trust_env is False
-
- async def __aenter__(self) -> "_AsyncClient":
- return self
-
- async def __aexit__(self, *args: object) -> None:
- return None
-
- async def get(self, url: str, headers: dict[str, str]) -> _Response:
- assert url == "http://localhost:8000/v1/health"
- assert headers == {}
- return _Response()
-
- monkeypatch.setattr("mineru.parser.api_client.httpx.AsyncClient", _AsyncClient)
-
- source = asyncio.run(parser._async_build_source(pdf))
-
- assert source == {"type": "file_id", "file_id": "file_1"}
-
-
-def test_api_client_can_request_zip_for_model_output(tmp_path: Path) -> None:
- pdf = tmp_path / "demo.pdf"
- pdf.write_bytes(b"%PDF-1.7\n")
-
- parser = MinerUApiParser(api_url="http://localhost:8000", tier="standard", include_model_output=True)
- payload = parser._build_payload({"type": "local", "path": str(pdf)}, "")
-
- assert payload["output_formats"] == ["zip"]
-
-
-def test_api_client_can_request_zip_for_image_cache(tmp_path: Path) -> None:
- pdf = tmp_path / "demo.pdf"
- pdf.write_bytes(b"%PDF-1.7\n")
-
- parser = MinerUApiParser(
- api_url="http://localhost:8000",
- tier="standard",
- include_images=True,
- )
- payload = parser._build_payload({"type": "local", "path": str(pdf)}, "")
-
- assert payload["output_formats"] == ["zip"]
-
-
-def test_api_client_uses_zip_for_official_api_when_model_output_requested() -> None:
- parser = MinerUApiParser(
- api_url="https://mineru.net/api",
- tier="advanced",
- include_model_output=True,
- )
-
- assert parser._output_formats() == ["zip"]
-
-
-def test_api_client_uses_zip_for_official_api_when_image_cache_requested() -> None:
- parser = MinerUApiParser(
- api_url="https://mineru.net/api",
- tier="advanced",
- include_images=True,
- )
-
- assert parser._output_formats() == ["zip"]
-
-
-def test_api_client_uses_tier_without_backend_semantics(tmp_path: Path) -> None:
- pdf = tmp_path / "demo.pdf"
- pdf.write_bytes(b"%PDF-1.7\n")
-
- parser = MinerUApiParser(api_url="http://localhost:8000", tier="advanced")
- payload = parser._build_payload({"type": "local", "path": str(pdf)}, "")
-
- assert payload["tier"] == "advanced"
- assert not hasattr(parser, "backend")
-
-
@pytest.mark.parametrize("tier", ["ultra", "experimental"])
def test_api_client_rejects_invalid_tier_names(tier: str) -> None:
with pytest.raises(ValueError, match=f"Unsupported tier '{tier}'"):
MinerUApiParser(api_url="http://localhost:8000", tier=tier) # type: ignore[arg-type]
-def test_api_client_uses_middle_json_format_for_official_api() -> None:
- parser = MinerUApiParser(api_url="https://mineru.net/api", tier="advanced")
-
- assert parser._output_formats() == ["middle_json"]
-
-
def test_api_client_uses_env_api_key_when_no_api_key_argument(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("MINERU_API_KEY", "env-key")
@@ -724,1686 +523,183 @@ def test_api_client_ignores_legacy_detail_error_envelope() -> None:
assert exc_info.value.message.startswith("HTTP 400:")
-def test_api_client_inlines_image_sidecar_from_zip(monkeypatch: pytest.MonkeyPatch) -> None:
- parser = MinerUApiParser(api_url="http://localhost:8000", tier="standard", include_images=True)
- middle_json = _current_payload([_current_image_page("images/chart.png")])
- zip_ref = {"file_id": "file-zip", "bytes": 10}
- zip_buffer = io.BytesIO()
- with zipfile.ZipFile(zip_buffer, "w", zipfile.ZIP_DEFLATED) as archive:
- archive.writestr("middle_json.json", json.dumps(middle_json, ensure_ascii=False))
- archive.writestr("images/chart.png", b"chart-bytes")
-
- monkeypatch.setattr(api_client, "_download_bytes", lambda _parser, ref: zip_buffer.getvalue() if ref is zip_ref else b"")
-
- result = _parse_result_from_job(
- {
- "job_id": "job_1",
- "status": "completed",
- "files": [{"output_files": {"zip": zip_ref}}],
- },
- "demo.pdf",
- parser,
- )
+def test_api_client_disables_env_proxy_for_local_network_urls() -> None:
+ assert should_trust_env_for_url("http://localhost:8000") is False
+ assert should_trust_env_for_url("http://127.0.0.1:8000") is False
+ assert should_trust_env_for_url("http://192.168.1.20:8000/api") is False
+ assert should_trust_env_for_url("https://mineru.net/api") is True
- image_body = result.pages[0].blocks[0].content[0] # type: ignore[union-attr]
- assert image_body.image_base64 == "data:image/png;base64," + base64.b64encode(b"chart-bytes").decode("ascii") # type: ignore[union-attr]
- assert image_body.image_path is None # type: ignore[union-attr]
-
-
-def test_api_client_downloads_model_output_from_zip(monkeypatch: pytest.MonkeyPatch) -> None:
- parser = MinerUApiParser(api_url="http://localhost:8000", tier="standard", include_model_output=True)
- middle_json = _current_payload([{"page_idx": 0, "blocks": []}])
- zip_ref = {"file_id": "file-zip", "bytes": 10}
- zip_buffer = io.BytesIO()
- with zipfile.ZipFile(zip_buffer, "w", zipfile.ZIP_DEFLATED) as archive:
- archive.writestr("middle_json.json", json.dumps(middle_json, ensure_ascii=False))
- archive.writestr(
- "model_output.json",
- json.dumps(_model_payload(), ensure_ascii=False, indent=4),
- )
- monkeypatch.setattr(
- api_client,
- "_download_bytes",
- lambda _parser, ref: zip_buffer.getvalue() if ref is zip_ref else b"",
- )
+def test_api_client_poll_timeout_budget_is_one_hour() -> None:
+ """验证 v1 job 轮询最大等待预算为 1 小时。"""
+ assert api_client._POLL_INTERVAL_SECONDS * api_client._POLL_MAX_ATTEMPTS == 60 * 60
- result = _parse_result_from_job(
- {
- "job_id": "job_1",
- "status": "completed",
- "files": [{"output_files": {"zip": zip_ref}}],
- },
- "demo.pdf",
- parser,
- )
- assert result._model_output.pages == [[{"raw": "model"}]]
+def test_api_client_constructor_exposes_request_ocr_without_legacy_options() -> None:
+ """验证 OCR 使用显式请求参数,且不恢复旧 method 或图片分析选项。"""
+ parameters = inspect.signature(MinerUApiParser).parameters
+ assert parameters["ocr_mode"].default is None
+ assert "method" not in parameters
+ assert "image_analysis" not in parameters
-def test_api_client_accepts_legacy_official_layout_json(monkeypatch: pytest.MonkeyPatch) -> None:
- """官方 API ZIP 内的历史 pdf_info 文档迁移为当前 MiddleJson,不再拒绝。"""
- parser = MinerUApiParser(
- api_url="https://mineru.net/api",
- tier="standard",
- include_images=True,
- include_model_output=True,
- )
- zip_ref = {"file_id": "file-zip", "bytes": 10}
- middle_json = {
- "_backend": "hybrid",
- "pdf_info": [
- {
- "page_idx": 0,
- "page_size": [100, 200],
- "para_blocks": [
- {
- "index": 0,
- "type": "image",
- "bbox": [0, 0, 10, 10],
- "lines": [
- {
- "bbox": [0, 0, 10, 10],
- "spans": [{"type": "image", "bbox": [0, 0, 10, 10], "image_path": "chart.png"}],
- }
- ],
- }
- ],
- }
- ],
- }
- zip_buffer = io.BytesIO()
- with zipfile.ZipFile(zip_buffer, "w", zipfile.ZIP_DEFLATED) as archive:
- archive.writestr("layout.json", json.dumps(middle_json, ensure_ascii=False))
- archive.writestr("images/chart.png", b"chart-bytes")
- archive.writestr("ab3a55b0-2017-4e35-8507-ab8e2c012160_model.json", json.dumps(_model_payload()))
- monkeypatch.setattr(api_client, "_download_bytes", lambda _parser, ref: zip_buffer.getvalue() if ref is zip_ref else b"")
+def test_api_client_accepts_middle_json_current_pages() -> None:
+ """验证 API client 恢复严格 2.0 pages envelope;legacy pdf_info 由 accepts_remote_pdf_info 系列覆盖。"""
+ pages = _pages_from_middle_json(_current_payload([{"page_idx": 2, "blocks": []}]))
- result = _parse_result_from_job(
- {
- "job_id": "job_1",
- "status": "completed",
- "files": [{"output_files": {"zip": zip_ref}}],
- },
- "demo.pdf",
- parser,
- )
+ assert len(pages) == 1
+ assert pages[0].page_idx == 2
- assert isinstance(result, ParseResult)
- assert isinstance(result.pages, list)
+ "payload",
+ [
+ {"pages": [{"page_idx": 0, "blocks": []}]},
+ {"schema_version": "3.0", "pages": [{"page_idx": 0, "blocks": []}]},
+ ],
+)
+def test_api_client_rejects_unknown_schema_version(payload: dict[str, object]) -> None:
+ """验证缺失版本与未知版本 payload 仍要求重新解析;pdf_info 走 legacy 迁移单独覆盖。"""
+ with pytest.raises((ValueError, api_client._V1APIError), match="reparse the source document"):
+ _pages_from_middle_json(payload)
-def test_api_client_async_downloads_model_output_from_zip(monkeypatch: pytest.MonkeyPatch) -> None:
- parser = MinerUApiParser(api_url="http://localhost:8000", tier="standard", include_model_output=True)
- middle_json = _current_payload([{"page_idx": 0, "blocks": []}])
- zip_ref = {"file_id": "file-zip", "bytes": 10}
- zip_buffer = io.BytesIO()
- with zipfile.ZipFile(zip_buffer, "w", zipfile.ZIP_DEFLATED) as archive:
- archive.writestr("middle_json.json", json.dumps(middle_json, ensure_ascii=False))
- archive.writestr(
- "model_output.json",
- json.dumps(_model_payload(), ensure_ascii=False, indent=4),
- )
- async def fake_download_bytes(_parser: object, ref: dict[str, object]) -> bytes:
- return zip_buffer.getvalue() if ref is zip_ref else b""
+def test_api_server_validation_errors_use_error_envelope(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ _stub_api_server_dependency_preflight(monkeypatch)
+ app = create_app(upload_dir=str(tmp_path))
+ client = TestClient(app)
- monkeypatch.setattr(api_client, "_async_download_bytes", fake_download_bytes)
+ response = client.post("/v1/tasks", json={"files": []})
- result = asyncio.run(
- api_client._async_parse_result_from_job(
- {
- "job_id": "job_1",
- "status": "completed",
- "files": [{"output_files": {"zip": zip_ref}}],
- },
- "demo.pdf",
- parser,
- )
- )
+ assert response.status_code == 400
+ body = response.json()
+ assert "detail" not in body
+ assert body["error"]["type"] == "invalid_request_error"
+ assert body["error"]["code"] == "invalid_request"
+ assert "Invalid request" in body["error"]["message"]
- assert result._model_output.pages == [[{"raw": "model"}]]
+def test_api_server_unhandled_exceptions_use_error_envelope(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ _stub_api_server_dependency_preflight(monkeypatch)
+ app = create_app(upload_dir=str(tmp_path))
-def test_api_client_include_images_downloads_single_zip(monkeypatch: pytest.MonkeyPatch) -> None:
- parser = MinerUApiParser(
- api_url="http://localhost:8000",
- tier="standard",
- include_images=True,
- include_model_output=True,
- )
- zip_ref = {"file_id": "file-zip", "bytes": 10}
- image_path = "images/chart.png"
- image_body = ImageBodyBlock(
- type=BlockType.IMAGE_BODY,
- index=0,
- bbox=(0.0, 0.0, 0.1, 0.05),
- content="",
- image_path=image_path,
- )
- image_block = ImageBlock(
- type=BlockType.IMAGE,
- index=0,
- bbox=(0.0, 0.0, 0.1, 0.05),
- content=[image_body],
- )
- page = PageInfo(page_idx=0, blocks=[image_block])
- zip_buffer = io.BytesIO()
- with zipfile.ZipFile(zip_buffer, "w", zipfile.ZIP_DEFLATED) as archive:
- archive.writestr(
- "middle_json.json",
- json.dumps(
- ParseResult(middle_json=_full_middle_json(page)).to_dict(),
- ensure_ascii=False,
- ),
- )
- archive.writestr("images/chart.png", b"chart-bytes")
- archive.writestr("model_output.json", json.dumps(_model_payload(), ensure_ascii=False, indent=4))
- download_calls: list[dict[str, object]] = []
-
- def fake_download_bytes(_parser: object, ref: dict[str, object]) -> bytes:
- download_calls.append(ref)
- return zip_buffer.getvalue() if ref is zip_ref else b""
-
- monkeypatch.setattr(api_client, "_download_bytes", fake_download_bytes)
- monkeypatch.setattr(
- api_client,
- "_download_json",
- lambda *_args, **_kwargs: (_ for _ in ()).throw(AssertionError("zip result must not download middle_json separately")),
- )
+ @app.get("/boom")
+ async def _boom() -> None:
+ raise RuntimeError("boom")
- result = _parse_result_from_job(
- {
- "job_id": "job_1",
- "status": "completed",
- "files": [{"output_files": {"zip": zip_ref}}],
- },
- "demo.pdf",
- parser,
- )
+ client = TestClient(app, raise_server_exceptions=False)
- assert download_calls == [zip_ref]
- assert result.pages[0].page_idx == 0
- image_body = result.pages[0].blocks[0].content[0]
- assert image_body.image_base64 == "data:image/png;base64," + base64.b64encode(b"chart-bytes").decode("ascii") # type: ignore[union-attr]
- assert result._model_output.pages == [[{"raw": "model"}]]
+ response = client.get("/boom")
+ assert response.status_code == 500
+ body = response.json()
+ assert "detail" not in body
+ assert body["error"] == {
+ "type": "api_error",
+ "code": "internal_error",
+ "message": "Internal server error",
+ "param": None,
+ }
-def test_api_client_does_not_read_image_cache_from_zip_unless_requested(monkeypatch: pytest.MonkeyPatch) -> None:
- parser = MinerUApiParser(api_url="http://localhost:8000", tier="standard", include_model_output=True)
- zip_ref = {"file_id": "file-zip", "bytes": 10}
- middle_json = _current_payload([_current_image_page("images/chart.png")])
- zip_buffer = io.BytesIO()
- with zipfile.ZipFile(zip_buffer, "w", zipfile.ZIP_DEFLATED) as archive:
- archive.writestr("middle_json.json", json.dumps(middle_json, ensure_ascii=False))
- archive.writestr("images/chart.png", b"chart-bytes")
- archive.writestr("model_output.json", json.dumps(_model_payload(), ensure_ascii=False, indent=4))
- monkeypatch.setattr(api_client, "_download_bytes", lambda _parser, ref: zip_buffer.getvalue() if ref is zip_ref else b"")
+def test_create_app_does_not_read_runtime_settings_from_env(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ _stub_api_server_dependency_preflight(monkeypatch)
+ monkeypatch.setenv("MINERU_TIER", "advanced")
+ monkeypatch.setenv("MINERU_BACKEND", "hybrid-auto-engine")
+ monkeypatch.setenv("MINERU_CONCURRENCY", "9")
+ monkeypatch.setenv("MINERU_URL_TIMEOUT", "99")
+ monkeypatch.setenv("MINERU_OCR_MODE", "ocr")
+ monkeypatch.setenv("MINERU_EFFORT", "high")
+ monkeypatch.setenv(_REMOVED_TABLE_ENABLE_ENV, "false")
+ monkeypatch.setenv(_REMOVED_FORMULA_ENABLE_ENV, "false")
+ monkeypatch.setenv("MINERU_IMAGE_ANALYSIS", "false")
- result = _parse_result_from_job(
- {
- "job_id": "job_1",
- "status": "completed",
- "files": [{"output_files": {"zip": zip_ref}}],
- },
- "demo.pdf",
- parser,
- )
+ app = create_app(upload_dir=str(tmp_path))
- image_body = result.pages[0].blocks[0].content[0]
- assert image_body.image_base64 is None # type: ignore[union-attr]
- assert image_body.image_path == "images/chart.png" # type: ignore[union-attr]
- assert result._model_output.pages == [[{"raw": "model"}]]
+ assert app.state.tier == "standard"
+ assert app.state.concurrency == 1
+ assert app.state.url_timeout == 60
+ assert app.state.allow_http_source is False
+ assert not hasattr(app.state, "ocr_mode")
+ assert not hasattr(app.state, "language")
+ assert app.state.effort == "high"
+ assert app.state.image_analysis is True
+ assert not hasattr(app.state, _REMOVED_TABLE_ENABLE_PARAM)
+ assert not hasattr(app.state, _REMOVED_FORMULA_ENABLE_PARAM)
-def test_api_client_async_include_images_downloads_single_zip(monkeypatch: pytest.MonkeyPatch) -> None:
- parser = MinerUApiParser(
- api_url="http://localhost:8000",
- tier="standard",
- include_images=True,
- include_model_output=True,
- )
- zip_ref = {"file_id": "file-zip", "bytes": 10}
- zip_buffer = io.BytesIO()
- with zipfile.ZipFile(zip_buffer, "w", zipfile.ZIP_DEFLATED) as archive:
- archive.writestr(
- "middle_json.json",
- json.dumps(_current_payload([{"page_idx": 0, "blocks": []}])),
- )
- archive.writestr("model_output.json", json.dumps(_model_payload(), ensure_ascii=False, indent=4))
- download_calls: list[dict[str, object]] = []
+def test_api_server_cli_no_longer_exposes_reload() -> None:
+ result = runner.invoke(main, ["--reload"])
- async def fake_download_bytes(_parser: object, ref: dict[str, object]) -> bytes:
- download_calls.append(ref)
- return zip_buffer.getvalue() if ref is zip_ref else b""
+ assert result.exit_code != 0
+ assert "No such option" in result.output
+ assert "--reload" in result.output
- async def fail_download_json(*_args: object, **_kwargs: object) -> dict[str, object]:
- raise AssertionError("zip result must not download middle_json separately")
- monkeypatch.setattr(api_client, "_async_download_bytes", fake_download_bytes)
- monkeypatch.setattr(api_client, "_async_download_json", fail_download_json)
+def test_usage_counts_completed_files_and_reports_configured_concurrency(tmp_path: Path) -> None:
+ """files_processed 按完成的文件计数,max_concurrent_jobs 报告配置上限而非剩余信号量。"""
+ file_store = FileStore(tmp_path / "api-files")
+ store = api_server.JobStore(concurrency=2)
- result = asyncio.run(
- api_client._async_parse_result_from_job(
+ def _request(*names: str) -> _ParseRequest:
+ """登记源内容以构造内部执行统计样本。"""
+ for name in names:
+ (tmp_path / name).write_bytes(b"source")
+ return _ParseRequest.model_validate(
{
- "job_id": "job_1",
- "status": "completed",
- "files": [{"output_files": {"zip": zip_ref}}],
- },
- "demo.pdf",
- parser,
+ "files": [{"source": stored_source(file_store, tmp_path / name)} for name in names],
+ "tier": "flash",
+ "output_formats": ["default"],
+ }
)
- )
-
- assert download_calls == [zip_ref]
- assert result.pages[0].page_idx == 0
- assert result._model_output.pages == [[{"raw": "model"}]]
+ rec_a = store.create(_request("a1.html", "a2.html"), file_store)
+ rec_a.files[0].status = "completed"
+ rec_a.files[1].status = "failed"
+ rec_a.status = "partial"
+ rec_b = store.create(_request("b1.html"), file_store)
+ rec_b.files[0].status = "completed"
+ rec_b.status = "completed"
-def test_api_client_include_images_rejects_unsafe_image_entries(monkeypatch: pytest.MonkeyPatch) -> None:
- parser = MinerUApiParser(api_url="http://localhost:8000", tier="standard", include_images=True)
- zip_ref = {"file_id": "file-zip", "bytes": 10}
- middle_json = _current_payload([_current_image_page("../escape.png")])
- zip_buffer = io.BytesIO()
- with zipfile.ZipFile(zip_buffer, "w", zipfile.ZIP_DEFLATED) as archive:
- archive.writestr("middle_json.json", json.dumps(middle_json, ensure_ascii=False))
-
- monkeypatch.setattr(api_client, "_download_bytes", lambda _parser, ref: zip_buffer.getvalue() if ref is zip_ref else b"")
+ async def _report_while_busy() -> "api_server.UsageResponse":
+ await store._semaphore.acquire()
+ return store.usage("anonymous")
- with pytest.raises(api_client._V1APIError, match="Unsafe image sidecar path"):
- _parse_result_from_job(
- {
- "job_id": "job_1",
- "status": "completed",
- "files": [{"output_files": {"zip": zip_ref}}],
- },
- "demo.pdf",
- parser,
- )
+ usage = asyncio.run(_report_while_busy())
+ assert usage.current.jobs_created == 2
+ assert usage.current.files_processed == 2
+ assert usage.limits.max_concurrent_jobs == 2
-def test_api_client_accepts_remote_pdf_info_middle_json(monkeypatch: pytest.MonkeyPatch) -> None:
- """远程返回历史 pdf_info 协议时迁移为当前 MiddleJson,不猜测但也不拒绝。"""
- parser = MinerUApiParser(api_url="https://mineru.net/api", tier="standard")
- middle_json = {
- "_backend": "hybrid",
- "_version_name": "remote",
- "pdf_info": [{"page_idx": 0, "page_size": [100, 200]}],
- }
- monkeypatch.setattr(api_client, "_download_json", lambda _parser, _outputs: middle_json)
+def test_api_server_logs_traceback_when_job_file_fails(
+ monkeypatch: pytest.MonkeyPatch,
+ tmp_path: Path,
+ caplog: pytest.LogCaptureFixture,
+) -> None:
+ async def fake_parse_async(*args: object, **kwargs: object) -> ParseResult:
+ raise RuntimeError("boom")
- result = _parse_result_from_job(
+ monkeypatch.setattr("mineru.parser.api_server.parse_async", fake_parse_async)
+ file_store = FileStore(tmp_path / "api-files")
+ source = tmp_path / "demo.pdf"
+ source.write_bytes(b"%PDF-1.7\n")
+ request = _ParseRequest.model_validate(
{
- "job_id": "job_1",
- "status": "completed",
- "files": [{"output_files": {"middle_json": {"file_id": "file-middle-json", "bytes": 10}}}],
- },
- "demo.pdf",
- parser,
+ "files": [{"source": stored_source(file_store, source)}],
+ "tier": "standard",
+ "output_formats": ["default"],
+ }
)
-
- assert isinstance(result, ParseResult)
- assert isinstance(result.pages, list)
-
-
-def test_async_api_client_accepts_remote_pdf_info_middle_json(monkeypatch: pytest.MonkeyPatch) -> None:
- """异步客户端同样迁移历史 pdf_info 协议。"""
- parser = MinerUApiParser(api_url="https://mineru.net/api", tier="standard")
- middle_json = {
- "_backend": "hybrid",
- "pdf_info": [{"page_idx": 0, "page_size": [100, 200]}],
- }
-
- async def _download_json(*_args: object, **_kwargs: object) -> dict[str, object]:
- return middle_json
-
- monkeypatch.setattr(api_client, "_async_download_json", _download_json)
-
- result = asyncio.run(
- api_client._async_parse_result_from_job(
- {
- "job_id": "job_1",
- "status": "completed",
- "files": [{"output_files": {"middle_json": {"file_id": "file-middle-json", "bytes": 10}}}],
- },
- "demo.pdf",
- parser,
- )
- )
-
- assert isinstance(result, ParseResult)
- assert isinstance(result.pages, list)
-
-
-def test_api_client_rejects_legacy_json_output_file_key() -> None:
- parser = MinerUApiParser(api_url="https://mineru.net/api", tier="standard")
-
- with pytest.raises(api_client._V1APIError) as exc_info:
- _parse_result_from_job(
- {
- "job_id": "job_1",
- "status": "completed",
- "files": [{"output_files": {"json": {"file_id": "file-json", "bytes": 10}}}],
- },
- "demo.pdf",
- parser,
- )
-
- assert exc_info.value.code == "missing_middle_json_output"
- assert "available outputs: json" in exc_info.value.message
-
-
-def test_api_client_rejects_output_reference_without_file_id() -> None:
- parser = MinerUApiParser(api_url="https://mineru.net/api", tier="standard")
-
- with pytest.raises(api_client._V1APIError) as exc_info:
- api_client._download_bytes(parser, {"url": "https://example.invalid/output.json", "bytes": 10})
-
- assert exc_info.value.code == "invalid_response"
- assert exc_info.value.message == "No file_id in output reference"
-
-
-def test_async_api_client_rejects_output_reference_without_file_id() -> None:
- parser = MinerUApiParser(api_url="https://mineru.net/api", tier="standard")
-
- async def _run() -> None:
- await api_client._async_download_bytes(parser, {"url": "https://example.invalid/output.json", "bytes": 10})
-
- with pytest.raises(api_client._V1APIError) as exc_info:
- asyncio.run(_run())
-
- assert exc_info.value.code == "invalid_response"
- assert exc_info.value.message == "No file_id in output reference"
-
-
[email protected]("image_path", ["../escape.png", "/tmp/escape.png", "\\escape.png", "C:\\escape.png"])
-def test_api_client_include_images_rejects_unsafe_zip_image_sidecar_paths(
- monkeypatch: pytest.MonkeyPatch,
- image_path: str,
-) -> None:
- parser = MinerUApiParser(api_url="http://localhost:8000", tier="standard", include_images=True)
- zip_ref = {"file_id": "file-zip", "bytes": 10}
- middle_json = _current_payload([_current_image_page(image_path)])
- zip_buffer = io.BytesIO()
- with zipfile.ZipFile(zip_buffer, "w", zipfile.ZIP_DEFLATED) as archive:
- archive.writestr("middle_json.json", json.dumps(middle_json, ensure_ascii=False))
-
- monkeypatch.setattr(api_client, "_download_bytes", lambda _parser, ref: zip_buffer.getvalue() if ref is zip_ref else b"")
-
- with pytest.raises(api_client._V1APIError, match="Unsafe image sidecar path"):
- _parse_result_from_job(
- {
- "job_id": "job_1",
- "status": "completed",
- "files": [{"output_files": {"zip": zip_ref}}],
- },
- "demo.pdf",
- parser,
- )
-
-
-def test_async_api_client_include_images_rejects_unsafe_zip_image_sidecar_paths(monkeypatch: pytest.MonkeyPatch) -> None:
- parser = MinerUApiParser(api_url="http://localhost:8000", tier="standard", include_images=True)
- zip_ref = {"file_id": "file-zip", "bytes": 10}
- middle_json = _current_payload([_current_image_page("../escape.png")])
- zip_buffer = io.BytesIO()
- with zipfile.ZipFile(zip_buffer, "w", zipfile.ZIP_DEFLATED) as archive:
- archive.writestr("middle_json.json", json.dumps(middle_json, ensure_ascii=False))
-
- async def _download_bytes(_parser: object, ref: dict[str, object]) -> bytes:
- return zip_buffer.getvalue() if ref is zip_ref else b""
-
- monkeypatch.setattr(api_client, "_async_download_bytes", _download_bytes)
-
- with pytest.raises(api_client._V1APIError, match="Unsafe image sidecar path"):
- asyncio.run(
- api_client._async_parse_result_from_job(
- {
- "job_id": "job_1",
- "status": "completed",
- "files": [{"output_files": {"zip": zip_ref}}],
- },
- "demo.pdf",
- parser,
- )
- )
-
-
-def test_api_client_disables_env_proxy_for_local_network_urls() -> None:
- assert should_trust_env_for_url("http://localhost:8000") is False
- assert should_trust_env_for_url("http://127.0.0.1:8000") is False
- assert should_trust_env_for_url("http://192.168.1.20:8000/api") is False
- assert should_trust_env_for_url("https://mineru.net/api") is True
-
-
-def test_api_client_passes_trust_env_from_api_url(monkeypatch: pytest.MonkeyPatch) -> None:
- calls: list[bool] = []
-
- class _Response:
- status_code = 200
- text = ""
-
- def json(self) -> dict[str, str]:
- return {"job_id": "job_1", "status": "completed"}
-
- class _Client:
- def __init__(self, *, timeout: object, trust_env: bool, **_: object) -> None:
- calls.append(trust_env)
-
- def __enter__(self) -> "_Client":
- return self
-
- def __exit__(self, *args: object) -> None:
- return None
-
- def post(self, *args: object, **kwargs: object) -> _Response:
- return _Response()
-
- monkeypatch.setattr("mineru.parser.api_client.httpx.Client", _Client)
-
- MinerUApiParser(api_url="http://127.0.0.1:8000")._do_parse({"files": []})
- MinerUApiParser(api_url="https://mineru.net/api")._do_parse({"files": []})
-
- assert calls == [False, True]
-
-
-def test_api_client_poll_uses_fixed_one_second_interval(monkeypatch: pytest.MonkeyPatch) -> None:
- """验证同步 v1 job 轮询使用固定 1 秒间隔,避免指数退避放大本地等待。"""
- parser = MinerUApiParser(api_url="http://localhost:8000", tier="standard")
- statuses = iter(["pending", "processing", "completed"])
- sleep_delays: list[int] = []
-
- def fake_sleep(delay: int) -> None:
- """记录同步轮询 sleep 参数,避免测试真的等待。"""
- sleep_delays.append(delay)
-
- def fake_get(url: str, headers: dict[str, str]) -> object:
- """按顺序返回 job 状态,模拟第三次轮询完成。"""
- assert url == "http://localhost:8000/v1/parse/jobs/job_1"
- assert headers == {}
- status = next(statuses)
- return types.SimpleNamespace(
- status_code=200,
- text="",
- json=lambda: {"job_id": "job_1", "status": status},
- )
-
- monkeypatch.setattr(api_client.time, "sleep", fake_sleep)
-
- job = parser._poll(types.SimpleNamespace(get=fake_get), "job_1")
-
- assert job["status"] == "completed"
- assert sleep_delays == [1, 1, 1]
-
-
-def test_async_api_client_poll_uses_fixed_one_second_interval(monkeypatch: pytest.MonkeyPatch) -> None:
- """验证异步 v1 job 轮询也使用固定 1 秒间隔,保持 Gradio 路径一致。"""
- parser = MinerUApiParser(api_url="http://localhost:8000", tier="standard")
- statuses = iter(["pending", "processing", "completed"])
- sleep_delays: list[int] = []
-
- async def fake_sleep(delay: int) -> None:
- """记录异步轮询 sleep 参数,避免测试真的等待。"""
- sleep_delays.append(delay)
-
- async def fake_get(url: str, headers: dict[str, str]) -> object:
- """按顺序返回 job 状态,模拟第三次异步轮询完成。"""
- assert url == "http://localhost:8000/v1/parse/jobs/job_1"
- assert headers == {}
- status = next(statuses)
- return types.SimpleNamespace(
- status_code=200,
- text="",
- json=lambda: {"job_id": "job_1", "status": status},
- )
-
- monkeypatch.setattr(api_client.asyncio, "sleep", fake_sleep)
-
- job = asyncio.run(parser._async_poll(types.SimpleNamespace(get=fake_get), "job_1"))
-
- assert job["status"] == "completed"
- assert sleep_delays == [1, 1, 1]
-
-
-def test_api_client_poll_timeout_budget_is_one_hour() -> None:
- """验证 v1 job 轮询最大等待预算为 1 小时。"""
- assert api_client._POLL_INTERVAL_SECONDS * api_client._POLL_MAX_ATTEMPTS == 60 * 60
-
-
-def test_api_client_retries_transient_poll_transport_errors(monkeypatch: pytest.MonkeyPatch) -> None:
- parser = MinerUApiParser(api_url="https://mineru.net/api", tier="standard")
- calls = 0
- sleep_delays: list[float] = []
-
- def fake_get(_url: str, *, headers: dict[str, str]) -> httpx.Response:
- nonlocal calls
- assert headers == {}
- calls += 1
- if calls < 3:
- raise httpx.ReadError("connection closed")
- return httpx.Response(200, json={"job_id": "job_1", "status": "completed"})
-
- monkeypatch.setattr(api_client, "_transport_retry_delay", lambda _attempt: 0.0)
- monkeypatch.setattr(api_client.time, "sleep", sleep_delays.append)
-
- job = parser._poll(types.SimpleNamespace(get=fake_get), "job_1")
-
- assert job["status"] == "completed"
- assert calls == 3
- assert sleep_delays == [1, 0.0, 0.0]
-
-
-def test_async_api_client_retries_transient_poll_transport_errors(monkeypatch: pytest.MonkeyPatch) -> None:
- parser = MinerUApiParser(api_url="https://mineru.net/api", tier="standard")
- calls = 0
- sleep_delays: list[float] = []
-
- async def fake_get(_url: str, *, headers: dict[str, str]) -> httpx.Response:
- nonlocal calls
- assert headers == {}
- calls += 1
- if calls < 3:
- raise httpx.ReadError("connection closed")
- return httpx.Response(200, json={"job_id": "job_1", "status": "completed"})
-
- async def fake_sleep(delay: float) -> None:
- sleep_delays.append(delay)
-
- monkeypatch.setattr(api_client, "_transport_retry_delay", lambda _attempt: 0.0)
- monkeypatch.setattr(api_client.asyncio, "sleep", fake_sleep)
-
- job = asyncio.run(parser._async_poll(types.SimpleNamespace(get=fake_get), "job_1"))
-
- assert job["status"] == "completed"
- assert calls == 3
- assert sleep_delays == [1, 0.0, 0.0]
-
-
-def test_api_client_retries_upload_content_put(monkeypatch: pytest.MonkeyPatch) -> None:
- calls = 0
-
- class _Client:
- def put(self, _url: str, *, headers: dict[str, str], content: bytes) -> httpx.Response:
- nonlocal calls
- assert headers == {"Content-Type": "application/pdf"}
- assert content == b"pdf"
- calls += 1
- if calls == 1:
- raise httpx.WriteError("connection closed")
- return httpx.Response(200)
-
- monkeypatch.setattr(api_client.time, "sleep", lambda _delay: None)
-
- response = api_client._request_with_retry(
- _Client(), # type: ignore[arg-type]
- "PUT",
- "https://upload.example/content",
- stage="upload content",
- headers={"Content-Type": "application/pdf"},
- content=b"pdf",
- )
-
- assert response.status_code == 200
- assert calls == 2
-
-
-def test_api_client_recovers_completed_upload_after_lost_complete_response() -> None:
- parser = MinerUApiParser(api_url="https://mineru.net/api", tier="standard")
- complete_calls = 0
- status_calls = 0
-
- class _Client:
- def post(self, _url: str, *, headers: dict[str, str]) -> httpx.Response:
- nonlocal complete_calls
- assert headers == {}
- complete_calls += 1
- raise httpx.ReadError("response lost")
-
- def get(self, _url: str, *, headers: dict[str, str]) -> httpx.Response:
- nonlocal status_calls
- assert headers == {}
- status_calls += 1
- return httpx.Response(
- 200,
- json={"id": "upload_1", "status": "completed", "file": {"id": "file_1"}},
- )
-
- upload = parser._complete_upload(_Client(), "upload_1") # type: ignore[arg-type]
-
- assert upload["file"]["id"] == "file_1"
- assert complete_calls == 1
- assert status_calls == 1
-
-
-def test_async_api_client_recovers_completed_upload_after_lost_complete_response() -> None:
- parser = MinerUApiParser(api_url="https://mineru.net/api", tier="standard")
- complete_calls = 0
- status_calls = 0
-
- class _AsyncClient:
- async def post(self, _url: str, *, headers: dict[str, str]) -> httpx.Response:
- nonlocal complete_calls
- assert headers == {}
- complete_calls += 1
- raise httpx.ReadError("response lost")
-
- async def get(self, _url: str, *, headers: dict[str, str]) -> httpx.Response:
- nonlocal status_calls
- assert headers == {}
- status_calls += 1
- return httpx.Response(
- 200,
- json={"id": "upload_1", "status": "completed", "file": {"id": "file_1"}},
- )
-
- upload = asyncio.run(parser._async_complete_upload(_AsyncClient(), "upload_1")) # type: ignore[arg-type]
-
- assert upload["file"]["id"] == "file_1"
- assert complete_calls == 1
- assert status_calls == 1
-
-
-def test_api_client_does_not_retry_job_submission_transport_error(monkeypatch: pytest.MonkeyPatch) -> None:
- calls = 0
-
- class _Client:
- def __init__(self, **_: object) -> None:
- pass
-
- def __enter__(self) -> "_Client":
- return self
-
- def __exit__(self, *args: object) -> None:
- return None
-
- def post(self, _url: str, *, headers: dict[str, str], json: dict[str, object]) -> httpx.Response:
- nonlocal calls
- assert headers == {}
- assert json == {"files": []}
- calls += 1
- raise httpx.ReadError("response lost")
-
- monkeypatch.setattr(api_client.httpx, "Client", _Client)
-
- with pytest.raises(api_client._APITransportError) as exc_info:
- MinerUApiParser(api_url="https://mineru.net/api")._do_parse({"files": []})
-
- assert exc_info.value.stage == "job submission"
- assert exc_info.value.attempts == 1
- assert calls == 1
-
-
-def test_api_client_omits_tier_when_unspecified(tmp_path: Path) -> None:
- pdf = tmp_path / "demo.pdf"
- pdf.write_bytes(b"%PDF-1.7\n")
-
- parser = MinerUApiParser(api_url="http://localhost:8000")
- payload = parser._build_payload({"type": "local", "path": str(pdf)}, "")
-
- assert "tier" not in payload
-
-
-def test_api_client_constructor_exposes_request_ocr_without_legacy_options() -> None:
- """验证 OCR 使用显式请求参数,且不恢复旧 method 或图片分析选项。"""
- parameters = inspect.signature(MinerUApiParser).parameters
-
- assert parameters["ocr_mode"].default is None
- assert "method" not in parameters
- assert "image_analysis" not in parameters
-
-
-def test_api_client_omits_page_range_when_unspecified(tmp_path: Path) -> None:
- pdf = tmp_path / "demo.pdf"
- pdf.write_bytes(b"%PDF-1.7\n")
-
- parser = MinerUApiParser(api_url="http://localhost:8000", tier="standard")
- payload = parser._build_payload({"type": "local", "path": str(pdf)}, "")
-
- assert payload["files"] == [{"source": {"type": "local", "path": str(pdf)}}]
-
-
-def test_api_client_surfaces_failed_job_error() -> None:
- parser = MinerUApiParser(api_url="http://localhost:8000", tier="standard")
-
- with pytest.raises(Exception, match="upstream failed"):
- _parse_result_from_job(
- {
- "job_id": "job_1",
- "status": "failed",
- "error": {"code": "remote_failed", "message": "upstream failed"},
- },
- "demo.pdf",
- parser,
- )
-
-
-def test_api_client_surfaces_failed_file_error_when_job_has_no_top_level_error() -> None:
- parser = MinerUApiParser(api_url="http://localhost:8000", tier="standard")
-
- with pytest.raises(api_client._V1APIError) as exc_info:
- _parse_result_from_job(
- {
- "job_id": "job_1",
- "status": "failed",
- "files": [
- {
- "name": "demo.pdf",
- "status": "failed",
- "error": {
- "type": "engine_error",
- "code": "parse_failed",
- "message": "No module named 'torch'",
- },
- }
- ],
- },
- "demo.pdf",
- parser,
- )
-
- assert exc_info.value.code == "parse_failed"
- assert exc_info.value.message == "No module named 'torch'"
-
-
-def test_api_client_rejects_missing_middle_json_output() -> None:
- parser = MinerUApiParser(api_url="http://localhost:8000", tier="standard")
-
- with pytest.raises(Exception, match="did not return middle_json output"):
- _parse_result_from_job(
- {
- "job_id": "job_1",
- "status": "completed",
- "files": [{"output_files": {}}],
- },
- "demo.pdf",
- parser,
- )
-
-
-def test_api_client_rejects_json_underscore_output_alias() -> None:
- parser = MinerUApiParser(api_url="http://localhost:8000", tier="standard")
-
- with pytest.raises(api_client._V1APIError) as exc_info:
- api_client._download_json(parser, {"json_": {"file_id": "file-json", "bytes": 10}})
-
- assert exc_info.value.code == "missing_middle_json_output"
- assert "available outputs: json_" in exc_info.value.message
-
-
-def test_api_client_accepts_middle_json_current_pages() -> None:
- """验证 API client 恢复严格 2.0 pages envelope;legacy pdf_info 由 accepts_remote_pdf_info 系列覆盖。"""
- pages = _pages_from_middle_json(_current_payload([{"page_idx": 2, "blocks": []}]))
-
- assert len(pages) == 1
- assert pages[0].page_idx == 2
-
-
- "payload",
- [
- {"pages": [{"page_idx": 0, "blocks": []}]},
- {"schema_version": "3.0", "pages": [{"page_idx": 0, "blocks": []}]},
- ],
-)
-def test_api_client_rejects_unknown_schema_version(payload: dict[str, object]) -> None:
- """验证缺失版本与未知版本 payload 仍要求重新解析;pdf_info 走 legacy 迁移单独覆盖。"""
- with pytest.raises((ValueError, api_client._V1APIError), match="reparse the source document"):
- _pages_from_middle_json(payload)
-
-
-def test_create_job_request_accepts_new_format_names_and_rejects_options() -> None:
- req = CreateJobRequest.model_validate(
- {
- "files": [
- {
- "source": {"type": "local", "path": "/tmp/demo.pdf"},
- "page_range": None,
- }
- ],
- "output_formats": ["middle_json", "structured_content"],
- }
- )
- assert req.files[0].page_range is None
- assert req.output_formats == ["middle_json", "structured_content"]
-
- unsupported = CreateJobRequest.model_validate(
- {
- "files": [{"source": {"type": "local", "path": "/tmp/demo.pdf"}}],
- "output_formats": ["images"],
- }
- )
- assert unsupported.output_formats == ["images"]
-
- with pytest.raises(ValidationError):
- CreateJobRequest.model_validate(
- {
- "files": [{"source": {"type": "local", "path": "/tmp/demo.pdf"}}],
- "wait": 5,
- }
- )
-
- with pytest.raises(ValidationError):
- CreateJobRequest.model_validate(
- {
- "files": [
- {
- "source": {"type": "local", "path": "/tmp/demo.pdf"},
- "options": {"page_range": "1"},
- }
- ],
- }
- )
-
-
-def test_api_server_rejects_local_source_by_default(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
- _stub_api_server_dependency_preflight(monkeypatch)
- source = tmp_path / "demo.pdf"
- source.write_bytes(b"%PDF-1.7\n")
- app = create_app(upload_dir=str(tmp_path / "api"), tier="flash")
-
- with TestClient(app) as client:
- response = client.post(
- "/v1/parse/jobs",
- json={"tier": "flash", "files": [{"source": {"type": "local", "path": str(source)}}]},
- )
-
- assert response.status_code == 400
- body = response.json()
- assert body["error"]["code"] == "unsupported_source"
- assert body["error"]["param"] == "files.0.source"
- assert "--allow-local-source" in body["error"]["message"]
-
-
-def test_api_server_health_sources_reflect_local_source_flag(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
- _stub_api_server_dependency_preflight(monkeypatch)
- default_app = create_app(upload_dir=str(tmp_path / "default"), tier="flash")
- local_app = create_app(upload_dir=str(tmp_path / "local"), tier="flash", allow_local_source=True)
-
- with TestClient(default_app) as client:
- default_sources = client.get("/v1/health").json()["features"]["sources"]
- with TestClient(local_app) as client:
- local_sources = client.get("/v1/health").json()["features"]["sources"]
-
- assert default_sources == ["file_id", "url", "inline"]
- assert local_sources == ["file_id", "url", "inline", "local"]
-
-
-def test_api_server_rejects_inline_source_over_configured_limit(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
- _stub_api_server_dependency_preflight(monkeypatch)
- app = create_app(upload_dir=str(tmp_path), tier="flash", max_inline_bytes=3)
-
- with TestClient(app) as client:
- response = client.post(
- "/v1/parse/jobs",
- json={
- "tier": "flash",
- "files": [
- {
- "source": {
- "type": "inline",
- "name": "demo.pdf",
- "data": base64.b64encode(b"1234").decode("ascii"),
- }
- }
- ],
- },
- )
-
- assert response.status_code == 400
- body = response.json()
- assert body["error"]["code"] == "unsupported_source"
- assert body["error"]["param"] == "files.0.source"
- assert "max_inline_bytes" in body["error"]["message"]
-
-
-def test_api_server_rejects_plain_http_url_source_by_default(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
- _stub_api_server_dependency_preflight(monkeypatch)
- app = create_app(upload_dir=str(tmp_path), tier="flash")
-
- with TestClient(app) as client:
- response = client.post(
- "/v1/parse/jobs",
- json={"tier": "flash", "files": [{"source": {"type": "url", "url": "http://example.test/demo.pdf"}}]},
- )
-
- assert response.status_code == 400
- body = response.json()
- assert body["error"]["code"] == "unsupported_source"
- assert body["error"]["param"] == "files.0.source"
- assert "https" in body["error"]["message"]
-
-
-def test_api_server_rejects_unsupported_output_format_with_specific_code(
- tmp_path: Path, monkeypatch: pytest.MonkeyPatch
-) -> None:
- _stub_api_server_dependency_preflight(monkeypatch)
- source = tmp_path / "demo.pdf"
- source.write_bytes(b"%PDF-1.7\n")
- app = create_app(upload_dir=str(tmp_path), tier="flash", allow_local_source=True)
-
- with TestClient(app) as client:
- response = client.post(
- "/v1/parse/jobs",
- json={
- "tier": "flash",
- "files": [{"source": {"type": "local", "path": str(source)}}],
- "output_formats": ["images"],
- },
- )
-
- assert response.status_code == 400
- body = response.json()
- assert body["error"]["code"] == "unsupported_output_format"
- assert body["error"]["param"] is None
- assert "Unknown output format: images" in body["error"]["message"]
-
-
-def test_api_server_rejects_unsupported_source_type_with_specific_code(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
- _stub_api_server_dependency_preflight(monkeypatch)
- app = create_app(upload_dir=str(tmp_path), tier="flash")
-
- with TestClient(app) as client:
- response = client.post(
- "/v1/parse/jobs",
- json={
- "tier": "flash",
- "files": [{"source": {"type": "s3", "uri": "s3://bucket/demo.pdf"}}],
- },
- )
-
- assert response.status_code == 400
- body = response.json()
- assert body["error"]["code"] == "unsupported_source"
- assert body["error"]["param"] == "files.0.source"
-
-
-def test_local_parse_server_rejects_callback(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
- _stub_api_server_dependency_preflight(monkeypatch)
- app = create_app(upload_dir=str(tmp_path))
- client = TestClient(app)
-
- response = client.post(
- "/v1/parse/jobs",
- json={
- "files": [{"source": {"type": "local", "path": str(tmp_path / "demo.pdf")}}],
- "callback": {"url": "https://example.com/mineru-webhook", "secret": "secret"},
- },
- )
-
- assert response.status_code == 400
- body = response.json()
- assert "detail" not in body
- assert body["error"]["code"] == "invalid_request"
- assert "Webhook callback is not supported" in body["error"]["message"]
- assert app.state.job_store._jobs == {}
-
-
-def test_api_server_validation_errors_use_error_envelope(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
- _stub_api_server_dependency_preflight(monkeypatch)
- app = create_app(upload_dir=str(tmp_path))
- client = TestClient(app)
-
- response = client.post("/v1/parse/jobs", json={"files": []})
-
- assert response.status_code == 400
- body = response.json()
- assert "detail" not in body
- assert body["error"]["type"] == "invalid_request_error"
- assert body["error"]["code"] == "invalid_request"
- assert "Invalid request" in body["error"]["message"]
-
-
-def test_api_server_unhandled_exceptions_use_error_envelope(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
- _stub_api_server_dependency_preflight(monkeypatch)
- app = create_app(upload_dir=str(tmp_path))
-
- @app.get("/boom")
- async def _boom() -> None:
- raise RuntimeError("boom")
-
- client = TestClient(app, raise_server_exceptions=False)
-
- response = client.get("/boom")
-
- assert response.status_code == 500
- body = response.json()
- assert "detail" not in body
- assert body["error"] == {
- "type": "api_error",
- "code": "internal_error",
- "message": "Internal server error",
- "param": None,
- }
-
-
-def test_create_app_does_not_read_runtime_settings_from_env(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
- _stub_api_server_dependency_preflight(monkeypatch)
- monkeypatch.setenv("MINERU_TIER", "advanced")
- monkeypatch.setenv("MINERU_BACKEND", "hybrid-auto-engine")
- monkeypatch.setenv("MINERU_CONCURRENCY", "9")
- monkeypatch.setenv("MINERU_URL_TIMEOUT", "99")
- monkeypatch.setenv("MINERU_OCR_MODE", "ocr")
- monkeypatch.setenv("MINERU_EFFORT", "high")
- monkeypatch.setenv(_REMOVED_TABLE_ENABLE_ENV, "false")
- monkeypatch.setenv(_REMOVED_FORMULA_ENABLE_ENV, "false")
- monkeypatch.setenv("MINERU_IMAGE_ANALYSIS", "false")
-
- app = create_app(upload_dir=str(tmp_path))
-
- assert app.state.tier == "standard"
- assert app.state.concurrency == 1
- assert app.state.url_timeout == 60
- assert app.state.allow_local_source is False
- assert app.state.max_inline_bytes == 1024 * 1024
- assert app.state.allow_http_source is False
- assert not hasattr(app.state, "ocr_mode")
- assert not hasattr(app.state, "language")
- assert app.state.effort == "high"
- assert app.state.image_analysis is True
- assert not hasattr(app.state, _REMOVED_TABLE_ENABLE_PARAM)
- assert not hasattr(app.state, _REMOVED_FORMULA_ENABLE_PARAM)
-
-
-def test_api_server_cli_no_longer_exposes_reload() -> None:
- result = runner.invoke(main, ["--reload"])
-
- assert result.exit_code != 0
- assert "No such option" in result.output
- assert "--reload" in result.output
-
-
-def test_output_files_expose_new_names_without_inline_content() -> None:
- ref = OutputFileRef(file_id="file-1", bytes=12)
- outputs = OutputFiles(middle_json=ref, structured_content=ref)
-
- assert outputs.model_dump(by_alias=True, exclude_none=True) == {
- "middle_json": {"file_id": "file-1", "bytes": 12},
- "structured_content": {"file_id": "file-1", "bytes": 12},
- }
-
- with pytest.raises(ValidationError):
- OutputFileRef.model_validate({"file_id": "file-1", "bytes": 12, "content": "inline"})
-
-
-def test_job_links_do_not_expose_sse_events() -> None:
- links = JobLinks(self="/v1/parse/jobs/job_1", cancel="/v1/parse/jobs/job_1")
-
- assert links.model_dump() == {
- "self": "/v1/parse/jobs/job_1",
- "cancel": "/v1/parse/jobs/job_1",
- }
-
- with pytest.raises(ValidationError):
- JobLinks.model_validate(
- {
- "self": "/v1/parse/jobs/job_1",
- "events": "/v1/parse/jobs/job_1/events",
- "cancel": "/v1/parse/jobs/job_1",
- }
- )
-
-
- ("output_format", "output_attr"),
- [
- ("markdown", "markdown"),
- ("structured_content", "structured_content"),
- ],
-)
-def test_api_server_rendered_outputs_do_not_return_image_sidecars(
- monkeypatch: pytest.MonkeyPatch,
- tmp_path: Path,
- output_format: str,
- output_attr: str,
-) -> None:
- image_path = "rendered.png"
- image_body = ImageBodyBlock(
- type=BlockType.IMAGE_BODY,
- index=0,
- bbox=(0.0, 0.0, 0.1, 0.1),
- content="",
- image_path=image_path,
- )
- image_block = ImageBlock(
- type=BlockType.IMAGE,
- index=0,
- bbox=(0.0, 0.0, 0.1, 0.1),
- content=[image_body],
- )
- parse_result = ParseResult(
- middle_json=_full_middle_json(
- PageInfo(
- page_idx=0,
- blocks=[image_block],
- )
- ),
- )
-
- async def fake_parse_async(*args: object, **kwargs: object) -> ParseResult:
- return parse_result
-
- monkeypatch.setattr("mineru.parser.api_server.parse_async", fake_parse_async)
- file_store = FileStore(tmp_path / "api-files")
- source = tmp_path / "demo.pdf"
- source.write_bytes(b"%PDF-1.7\n")
- request = CreateJobRequest.model_validate(
- {
- "files": [{"source": {"type": "local", "path": str(source)}}],
- "tier": "standard",
- "output_formats": [output_format],
- }
- )
- job_store = api_server.JobStore()
- rec = job_store.create(request, file_store)
-
- asyncio.run(
- api_server._run_job(
- rec,
- request,
- file_store,
- image_analysis=True,
- allow_local_source=True,
- )
- )
-
- output_files = rec.files[0].output_files
- assert getattr(output_files, output_attr) is not None
- assert "images" not in output_files.model_dump(by_alias=True, exclude_none=True)
-
-
-def test_api_server_run_job_normalizes_lightweight_file_tier_to_flash(
- monkeypatch: pytest.MonkeyPatch,
- tmp_path: Path,
-) -> None:
- calls: list[dict[str, object]] = []
-
- async def fake_parse_async(*args: object, **kwargs: object) -> ParseResult:
- calls.append({"path": args[0], **kwargs})
- return ParseResult(middle_json=_full_middle_json(PageInfo(page_idx=0)))
-
- monkeypatch.setattr("mineru.parser.api_server.parse_async", fake_parse_async)
- file_store = FileStore(tmp_path / "api-files")
- source = tmp_path / "demo.html"
- source.write_text("<p>content</p>", encoding="utf-8")
- request = CreateJobRequest.model_validate(
- {
- "files": [{"source": {"type": "local", "path": str(source)}}],
- "tier": "standard",
- "output_formats": ["middle_json"],
- }
- )
- rec = api_server.JobStore().create(request, file_store)
-
- asyncio.run(
- api_server._run_job(
- rec,
- request,
- file_store,
- image_analysis=True,
- allow_local_source=True,
- )
- )
-
- assert [call["tier"] for call in calls] == ["flash"]
- assert calls[0]["source_context"].source_uri == source.resolve().as_uri() # type: ignore[union-attr]
- assert calls[0]["source_context"].local_resource_root == source.resolve().parent # type: ignore[union-attr]
- assert rec.tier == "standard"
- assert rec.files[0].status == "completed"
-
-
-def test_store_upload_data_rejects_declared_size_mismatch(tmp_path: Path) -> None:
- """声明 bytes 与 PUT 实际大小不一致时在落盘前拒绝。"""
- file_store = FileStore(tmp_path / "api-files")
- upload = file_store.create_upload(
- CreateUploadRequest.model_validate({"filename": "demo.pdf", "bytes": 4, "mime_type": "application/pdf"})
- )
-
- with pytest.raises(api_server.ApiServerError) as exc_info:
- file_store.store_upload_data(upload.id, b"too-many-bytes")
-
- assert exc_info.value.status_code == 413
- assert exc_info.value.error.code == "upload_size_mismatch"
- assert not (tmp_path / "api-files" / "blobs" / "_uploads" / upload.id).is_file()
-
-
-def test_complete_upload_rejects_size_mismatch(tmp_path: Path) -> None:
- """complete 阶段复核实际大小,防止绕过 store_upload_data 的写入路径。"""
- file_store = FileStore(tmp_path / "api-files")
- upload = file_store.create_upload(
- CreateUploadRequest.model_validate({"filename": "demo.pdf", "bytes": 4, "mime_type": "application/pdf"})
- )
- blob = file_store._blobs / "_uploads" / upload.id
- blob.parent.mkdir(parents=True, exist_ok=True)
- blob.write_bytes(b"0123456789")
-
- with pytest.raises(api_server.ApiServerError) as exc_info:
- file_store.complete_upload(upload.id, None)
-
- assert exc_info.value.status_code == 413
- assert exc_info.value.error.code == "upload_size_mismatch"
-
-
-def test_get_upload_expires_pending_records_and_discards_staged_data(tmp_path: Path) -> None:
- """上传到期后的新读取、写入和完成请求统一返回 404,并丢弃暂存数据。"""
- file_store = FileStore(tmp_path / "api-files")
- upload = file_store.create_upload(
- CreateUploadRequest.model_validate({"filename": "demo.pdf", "bytes": 4, "mime_type": "application/pdf"})
- )
- file_store.store_upload_data(upload.id, b"demo")
- staged = tmp_path / "api-files" / "blobs" / "_uploads" / upload.id
- assert staged.is_file()
-
- file_store._uploads[upload.id].expires_at = int(time.time()) - 1
-
- for operation in (
- lambda: file_store.get_upload(upload.id),
- lambda: file_store.store_upload_data(upload.id, b"demo"),
- lambda: file_store.complete_upload(upload.id, None),
- ):
- with pytest.raises(api_server.ApiServerError) as exc_info:
- operation()
- assert exc_info.value.status_code == 404
- assert not staged.is_file()
-
-
-def test_get_upload_hides_completed_records_after_expiry(tmp_path: Path) -> None:
- """已完成上传到期后同样不允许新的读取请求复用该上传视图。"""
- file_store = FileStore(tmp_path / "api-files")
- sha = "00" * 32
- file_store.store_blob(b"demo", sha256hex=sha)
- upload = file_store.create_upload(
- CreateUploadRequest.model_validate(
- {"filename": "demo.pdf", "bytes": 4, "mime_type": "application/pdf", "sha256sum": sha}
- )
- )
- assert upload.status == "completed"
-
- file_store._uploads[upload.id].expires_at = int(time.time()) - 1
-
- with pytest.raises(api_server.ApiServerError) as exc_info:
- file_store.get_upload(upload.id)
- assert exc_info.value.status_code == 404
-
-
-def test_run_job_preserves_input_file_id_and_keeps_source_undownloadable(
- monkeypatch: pytest.MonkeyPatch,
- tmp_path: Path,
-) -> None:
- """完成的 job 保留输入 file_id,不为源文件创建可下载的 parse_output 副本。"""
-
- async def fake_parse_async(*args: object, **kwargs: object) -> ParseResult:
- return ParseResult(middle_json=_full_middle_json(PageInfo(page_idx=0)))
-
- monkeypatch.setattr("mineru.parser.api_server.parse_async", fake_parse_async)
- file_store = FileStore(tmp_path / "api-files")
- payload = b"%PDF-1.7 demo"
- upload = file_store.create_upload(
- CreateUploadRequest.model_validate({"filename": "demo.pdf", "bytes": len(payload), "mime_type": "application/pdf"})
- )
- file_store.store_upload_data(upload.id, payload)
- completed = file_store.complete_upload(upload.id, None)
- assert completed.file is not None
- input_file_id = completed.file.id
-
- request = CreateJobRequest.model_validate(
- {
- "files": [{"source": {"type": "file_id", "file_id": input_file_id}}],
- "tier": "standard",
- "output_formats": ["middle_json"],
- }
- )
- rec = api_server.JobStore().create(request, file_store)
-
- asyncio.run(
- api_server._run_job(
- rec,
- request,
- file_store,
- image_analysis=True,
- )
- )
-
- assert rec.files[0].status == "completed"
- assert rec.files[0].file_id == input_file_id
- output_files = file_store.list_files(after=None, limit=1000, order="asc", purpose="parse_output").data
- assert input_file_id not in [item.id for item in output_files]
- assert [item.filename for item in output_files] == ["demo.pdf.middle.json"]
- with pytest.raises(api_server.ApiServerError) as exc_info:
- file_store.read_file_data(input_file_id)
- assert exc_info.value.status_code == 403
-
-
-def test_create_job_request_rejects_more_than_max_files_per_job(tmp_path: Path) -> None:
- """超过广告的 max_files_per_job 时拒绝创建请求。"""
- source = str(tmp_path / "demo.pdf")
- entries = [{"source": {"type": "local", "path": source}}] * (api_server._MAX_FILES_PER_JOB_DEFAULT + 1)
-
- with pytest.raises(ValidationError):
- CreateJobRequest.model_validate({"files": entries, "tier": "flash"})
-
-
-def test_job_responses_report_resolved_access_level(
- monkeypatch: pytest.MonkeyPatch,
- tmp_path: Path,
-) -> None:
- """job 创建与查询响应携带按 api_key 解析的 access_level,而不是固定 registered。"""
-
- async def fake_parse_async(*args: object, **kwargs: object) -> ParseResult:
- return ParseResult(middle_json=_full_middle_json(PageInfo(page_idx=0)))
-
- monkeypatch.setattr("mineru.parser.api_server.parse_async", fake_parse_async)
- source = tmp_path / "demo.html"
- source.write_text("<p>content</p>", encoding="utf-8")
- payload = {
- "files": [{"source": {"type": "local", "path": str(source)}}],
- "tier": "flash",
- "output_formats": ["middle_json"],
- }
-
- anonymous_app = create_app(upload_dir=str(tmp_path / "api-anonymous"), tier="flash", allow_local_source=True)
- with TestClient(anonymous_app) as client:
- created = client.post("/v1/parse/jobs", json=payload)
- assert created.status_code == 202
- assert created.json()["access_level"] == "anonymous"
- fetched = client.get(f"/v1/parse/jobs/{created.json()['job_id']}")
- assert fetched.status_code == 200
- assert fetched.json()["access_level"] == "anonymous"
-
- registered_app = create_app(
- upload_dir=str(tmp_path / "api-registered"), tier="flash", allow_local_source=True, api_key="secret-key"
- )
- with TestClient(registered_app) as client:
- created = client.post("/v1/parse/jobs", json=payload, headers={"Authorization": "Bearer secret-key"})
- assert created.status_code == 202
- assert created.json()["access_level"] == "registered"
- fetched = client.get(f"/v1/parse/jobs/{created.json()['job_id']}", headers={"Authorization": "Bearer secret-key"})
- assert fetched.status_code == 200
- assert fetched.json()["access_level"] == "registered"
-
-
-def test_usage_counts_completed_files_and_reports_configured_concurrency(tmp_path: Path) -> None:
- """files_processed 按完成的文件计数,max_concurrent_jobs 报告配置上限而非剩余信号量。"""
- file_store = FileStore(tmp_path / "api-files")
- store = api_server.JobStore(concurrency=2)
-
- def _request(*names: str) -> CreateJobRequest:
- return CreateJobRequest.model_validate(
- {
- "files": [{"source": {"type": "local", "path": str(tmp_path / name)}} for name in names],
- "tier": "flash",
- "output_formats": ["middle_json"],
- }
- )
-
- rec_a = store.create(_request("a1.html", "a2.html"), file_store)
- rec_a.files[0].status = "completed"
- rec_a.files[1].status = "failed"
- rec_a.status = "partial"
- rec_b = store.create(_request("b1.html"), file_store)
- rec_b.files[0].status = "completed"
- rec_b.status = "completed"
-
- async def _report_while_busy() -> "api_server.UsageResponse":
- await store._semaphore.acquire()
- return store.usage("anonymous")
-
- usage = asyncio.run(_report_while_busy())
-
- assert usage.current.jobs_created == 2
- assert usage.current.files_processed == 2
- assert usage.limits.max_concurrent_jobs == 2
-
-
[email protected](("request_tier", "response_tier"), [("standard", "standard"), (None, "flash")])
-def test_api_server_accepts_lightweight_job_without_requested_quality_runtime(
- monkeypatch: pytest.MonkeyPatch,
- tmp_path: Path,
- request_tier: str | None,
- response_tier: str,
-) -> None:
- async def fake_parse_async(*args: object, **kwargs: object) -> ParseResult:
- return ParseResult(middle_json=_full_middle_json(PageInfo(page_idx=0)))
-
- monkeypatch.setattr("mineru.parser.api_server.parse_async", fake_parse_async)
- source = tmp_path / "demo.html"
- source.write_text("<p>content</p>", encoding="utf-8")
- app = create_app(upload_dir=str(tmp_path / "api"), tier="flash", allow_local_source=True)
-
- payload: dict[str, object] = {
- "files": [{"source": {"type": "local", "path": str(source)}}],
- "output_formats": ["middle_json"],
- }
- if request_tier is not None:
- payload["tier"] = request_tier
-
- response = TestClient(app).post("/v1/parse/jobs", json=payload)
-
- assert response.status_code == 202
- assert response.json()["tier"] == response_tier
-
-
-def test_api_server_no_flash_rejects_lightweight_inputs(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
- _stub_api_server_dependency_preflight(monkeypatch)
- source = tmp_path / "demo.html"
- source.write_text("<p>content</p>", encoding="utf-8")
- app = create_app(
- upload_dir=str(tmp_path / "api"),
- tier="standard",
- no_flash=True,
- allow_local_source=True,
- )
-
- response = TestClient(app).post(
- "/v1/parse/jobs",
- json={
- "files": [{"source": {"type": "local", "path": str(source)}}],
- "tier": "standard",
- },
- )
-
- assert response.status_code == 400
- assert response.json()["error"] == {
- "type": "invalid_request_error",
- "code": "invalid_request",
- "message": "Flash parsing is disabled in this server, but one or more inputs require the Flash backend",
- "param": "files",
- }
-
-
-def test_api_server_no_flash_rejects_mixed_batch_with_lightweight_input(
- tmp_path: Path,
- monkeypatch: pytest.MonkeyPatch,
-) -> None:
- _stub_api_server_dependency_preflight(monkeypatch)
- pdf_source = tmp_path / "demo.pdf"
- html_source = tmp_path / "demo.html"
- pdf_source.write_bytes(b"%PDF-1.7\n")
- html_source.write_text("<p>content</p>", encoding="utf-8")
- app = create_app(
- upload_dir=str(tmp_path / "api"),
- tier="standard",
- no_flash=True,
- allow_local_source=True,
- )
-
- response = TestClient(app).post(
- "/v1/parse/jobs",
- json={
- "files": [
- {"source": {"type": "local", "path": str(pdf_source)}},
- {"source": {"type": "local", "path": str(html_source)}},
- ],
- "tier": "standard",
- },
- )
-
- assert response.status_code == 400
- assert "one or more inputs require the Flash backend" in response.text
-
-
[email protected]("model_output", [[{"raw": "model"}], []])
-def test_api_server_zip_includes_model_output_when_parse_result_has_it(
- monkeypatch: pytest.MonkeyPatch,
- tmp_path: Path,
- model_output: list[object],
-) -> None:
- parse_result = ParseResult(
- middle_json=_full_middle_json(PageInfo(page_idx=0)),
- _model_output=ModelJson.from_dict({**_model_payload(), "pages": [model_output]}),
- )
-
- async def fake_parse_async(*args: object, **kwargs: object) -> ParseResult:
- return parse_result
-
- monkeypatch.setattr("mineru.parser.api_server.parse_async", fake_parse_async)
- file_store = FileStore(tmp_path / "api-files")
- source = tmp_path / "demo.pdf"
- source.write_bytes(b"%PDF-1.7\n")
- request = CreateJobRequest.model_validate(
- {
- "files": [{"source": {"type": "local", "path": str(source)}}],
- "tier": "standard",
- "output_formats": ["zip"],
- }
- )
- job_store = api_server.JobStore()
- rec = job_store.create(request, file_store)
-
- asyncio.run(
- api_server._run_job(
- rec,
- request,
- file_store,
- image_analysis=True,
- allow_local_source=True,
- )
- )
-
- zip_ref = rec.files[0].output_files.zip
- assert zip_ref is not None
- with zipfile.ZipFile(io.BytesIO(file_store.read_file_data(zip_ref.file_id))) as archive:
- assert "model_output.json" in archive.namelist()
- model_output_text = archive.read("model_output.json").decode("utf-8")
- payload = json.loads(model_output_text)
- assert payload == ModelJson.from_dict({**_model_payload(), "pages": [model_output]}).to_dict(skip_defaults=False)
- if model_output:
- assert "\n " in model_output_text
-
-
-def test_api_server_zip_rejects_unsafe_image_sidecar_path(
- monkeypatch: pytest.MonkeyPatch,
- tmp_path: Path,
-) -> None:
- class _UnsafeResult:
- """模拟非标准解析结果,验证 ZIP writer 的下游路径防护仍生效。"""
-
- def save(self, writer: Any) -> None:
- """尝试写出不安全 sidecar 路径。"""
- writer.write("../escape.png", b"bad-image")
-
- async def fake_parse_async(*args: object, **kwargs: object) -> Any:
- return _UnsafeResult()
-
- monkeypatch.setattr("mineru.parser.api_server.parse_async", fake_parse_async)
- file_store = FileStore(tmp_path / "api-files")
- source = tmp_path / "demo.pdf"
- source.write_bytes(b"%PDF-1.7\n")
- request = CreateJobRequest.model_validate(
- {
- "files": [{"source": {"type": "local", "path": str(source)}}],
- "tier": "standard",
- "output_formats": ["zip"],
- }
- )
- rec = api_server.JobStore().create(request, file_store)
-
- asyncio.run(
- api_server._run_job(
- rec,
- request,
- file_store,
- image_analysis=True,
- allow_local_source=True,
- )
- )
-
- assert rec.files[0].status == "failed"
- assert rec.files[0].error is not None
- assert "Unsafe image sidecar path" in rec.files[0].error.message
-
-
-def test_api_server_zip_skips_model_output_when_parse_result_has_none(
- monkeypatch: pytest.MonkeyPatch,
- tmp_path: Path,
-) -> None:
- parse_result = ParseResult(
- middle_json=_full_middle_json(PageInfo(page_idx=0)),
- _model_output=None,
- )
-
- async def fake_parse_async(*args: object, **kwargs: object) -> ParseResult:
- return parse_result
-
- monkeypatch.setattr("mineru.parser.api_server.parse_async", fake_parse_async)
- file_store = FileStore(tmp_path / "api-files")
- source = tmp_path / "demo.pdf"
- source.write_bytes(b"%PDF-1.7\n")
- request = CreateJobRequest.model_validate(
- {
- "files": [{"source": {"type": "local", "path": str(source)}}],
- "tier": "standard",
- "output_formats": ["zip"],
- }
- )
- job_store = api_server.JobStore()
- rec = job_store.create(request, file_store)
-
- asyncio.run(
- api_server._run_job(
- rec,
- request,
- file_store,
- image_analysis=True,
- allow_local_source=True,
- )
- )
-
- zip_ref = rec.files[0].output_files.zip
- assert zip_ref is not None
- with zipfile.ZipFile(io.BytesIO(file_store.read_file_data(zip_ref.file_id))) as archive:
- assert "model_output.json" not in archive.namelist()
-
-
-def test_api_server_logs_traceback_when_job_file_fails(
- monkeypatch: pytest.MonkeyPatch,
- tmp_path: Path,
- caplog: pytest.LogCaptureFixture,
-) -> None:
- async def fake_parse_async(*args: object, **kwargs: object) -> ParseResult:
- raise RuntimeError("boom")
-
- monkeypatch.setattr("mineru.parser.api_server.parse_async", fake_parse_async)
- file_store = FileStore(tmp_path / "api-files")
- source = tmp_path / "demo.pdf"
- source.write_bytes(b"%PDF-1.7\n")
- request = CreateJobRequest.model_validate(
- {
- "files": [{"source": {"type": "local", "path": str(source)}}],
- "tier": "standard",
- "output_formats": ["middle_json"],
- }
- )
- job_store = api_server.JobStore()
- rec = job_store.create(request, file_store)
- mineru_logger = logging.getLogger("mineru")
- api_logger = logging.getLogger("mineru.parser.api_server")
- monkeypatch.setattr(mineru_logger, "disabled", False)
- monkeypatch.setattr(mineru_logger, "propagate", True)
- monkeypatch.setattr(api_logger, "disabled", False)
- monkeypatch.setattr(api_logger, "propagate", True)
+ job_store = api_server.JobStore()
+ rec = job_store.create(request, file_store)
+ mineru_logger = logging.getLogger("mineru")
+ api_logger = logging.getLogger("mineru.parser.api_server")
+ monkeypatch.setattr(mineru_logger, "disabled", False)
+ monkeypatch.setattr(mineru_logger, "propagate", True)
+ monkeypatch.setattr(api_logger, "disabled", False)
+ monkeypatch.setattr(api_logger, "propagate", True)
with caplog.at_level(logging.ERROR, logger="mineru.parser.api_server"):
asyncio.run(
@@ -2412,7 +708,6 @@ def test_api_server_logs_traceback_when_job_file_fails(
request,
file_store,
image_analysis=True,
- allow_local_source=True,
)
)
@@ -2424,17 +719,6 @@ def test_api_server_logs_traceback_when_job_file_fails(
assert "RuntimeError: boom" in caplog.text
-def test_job_list_item_uses_file_count() -> None:
- item = JobListItem(job_id="job_1", status="queued", created_at="2026-06-10T00:00:00Z", file_count=2)
-
- assert item.model_dump() == {
- "job_id": "job_1",
- "status": "queued",
- "created_at": "2026-06-10T00:00:00Z",
- "file_count": 2,
- }
-
-
def test_parse_info_uses_parser_version_not_backend_version() -> None:
parse = FileParseInfo(model_used="model-a", duration_ms=12, parser_version="3.2.1")
assert parse.model_dump(exclude_none=True) == {
@@ -2663,12 +947,12 @@ def test_api_server_standard_jobs_use_requested_tier(tmp_path: Path, monkeypatch
monkeypatch.setattr("mineru.parser.api_server.parse_async", fake_parse_async)
source = tmp_path / "demo.pdf"
source.write_bytes(b"%PDF-1.7\n")
- app = create_app(upload_dir=str(tmp_path / "api"), tier="standard", allow_local_source=True)
+ app = create_app(upload_dir=str(tmp_path / "api"), tier="standard")
file_store = FileStore(tmp_path / "api-files")
job_store = api_server.JobStore()
async def run_request(payload: dict[str, object]) -> dict[str, object]:
- request = CreateJobRequest.model_validate(payload)
+ request = _ParseRequest.model_validate(payload)
request.tier = request.tier or app.state.default_tier
rec = job_store.create(request, file_store)
await api_server._run_job(
@@ -2676,35 +960,33 @@ def test_api_server_standard_jobs_use_requested_tier(tmp_path: Path, monkeypatch
request,
file_store,
image_analysis=app.state.image_analysis,
- allow_local_source=app.state.allow_local_source,
- max_inline_bytes=app.state.max_inline_bytes,
allow_http_source=app.state.allow_http_source,
)
- return job_store.build_response(rec, access_level="anonymous").model_dump(by_alias=True)
+ return job_store.snapshot(rec)
default_response = asyncio.run(
run_request(
{
- "files": [{"source": {"type": "local", "path": str(source)}}],
- "output_formats": ["middle_json"],
+ "files": [{"source": stored_source(file_store, source)}],
+ "output_formats": ["default"],
}
)
)
basic_response = asyncio.run(
run_request(
{
- "files": [{"source": {"type": "local", "path": str(source)}}],
+ "files": [{"source": stored_source(file_store, source)}],
"tier": "basic",
- "output_formats": ["middle_json"],
+ "output_formats": ["default"],
}
)
)
advanced_response = asyncio.run(
run_request(
{
- "files": [{"source": {"type": "local", "path": str(source)}}],
+ "files": [{"source": stored_source(file_store, source)}],
"tier": "advanced",
- "output_formats": ["middle_json"],
+ "output_formats": ["default"],
}
)
)
@@ -2720,19 +1002,19 @@ def test_api_server_basic_rejects_advanced_request(tmp_path: Path, monkeypatch:
_stub_api_server_dependency_preflight(monkeypatch)
source = tmp_path / "demo.pdf"
source.write_bytes(b"%PDF-1.7\n")
- app = create_app(upload_dir=str(tmp_path / "api"), tier="basic", allow_local_source=True)
+ app = create_app(upload_dir=str(tmp_path / "api"), tier="basic")
with TestClient(app) as client:
response = client.post(
- "/v1/parse/jobs",
+ "/v1/tasks",
json={
- "files": [{"source": {"type": "local", "path": str(source)}}],
+ "manifest": [{"client_file_id": "file", "filename": source.name, "bytes": source.stat().st_size}],
"tier": "advanced",
},
)
assert response.status_code == 400
- assert "Tier 'advanced' not available in this server" in response.text
+ assert "Tier 'advanced' is not available" in response.text
def test_api_server_no_advanced_rejects_advanced_request(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
@@ -2744,20 +1026,19 @@ def test_api_server_no_advanced_rejects_advanced_request(tmp_path: Path, monkeyp
upload_dir=str(tmp_path / "api"),
tier="standard",
no_advanced=True,
- allow_local_source=True,
)
with TestClient(app) as client:
response = client.post(
- "/v1/parse/jobs",
+ "/v1/tasks",
json={
- "files": [{"source": {"type": "local", "path": str(source)}}],
+ "manifest": [{"client_file_id": "file", "filename": source.name, "bytes": source.stat().st_size}],
"tier": "advanced",
},
)
assert response.status_code == 400
- assert "Tier 'advanced' not available in this server" in response.text
+ assert "Tier 'advanced' is not available" in response.text
def test_api_server_no_flash_rejects_flash_request(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
@@ -2768,19 +1049,18 @@ def test_api_server_no_flash_rejects_flash_request(tmp_path: Path, monkeypatch:
upload_dir=str(tmp_path / "api"),
tier="standard",
no_flash=True,
- allow_local_source=True,
)
response = TestClient(app).post(
- "/v1/parse/jobs",
+ "/v1/tasks",
json={
- "files": [{"source": {"type": "local", "path": str(source)}}],
+ "manifest": [{"client_file_id": "file", "filename": source.name, "bytes": source.stat().st_size}],
"tier": "flash",
},
)
assert response.status_code == 400
- assert "Tier 'flash' not available in this server" in response.text
+ assert "Tier 'flash' is not available" in response.text
def test_api_server_rejects_invalid_startup_tier_configuration(tmp_path: Path) -> None:
@@ -2924,8 +1204,8 @@ def test_api_server_cli_exposes_parser_runtime_options() -> None:
assert "--backend" not in option_names
assert "--language" not in option_names
assert "--ocr-mode" not in option_names
- assert "--allow-local-source" in option_names
- assert "--max-inline-bytes" in option_names
+ assert "--allow-local-source" not in option_names
+ assert "--max-inline-bytes" not in option_names
assert "--allow-http-source" in option_names
assert "--effort" not in option_names
assert "--disable-image-analysis" in option_names
diff --git a/tests/unittest/test_parser_api_ocr.py b/tests/unittest/test_parser_api_ocr.py
index 9b2c3c7b..846821fd 100644
--- a/tests/unittest/test_parser_api_ocr.py
+++ b/tests/unittest/test_parser_api_ocr.py
@@ -2,11 +2,13 @@
from __future__ import annotations
+from test_tasks_sdk import task_snapshot, zip_output
+from mineru.parser.task_api import CreateTaskRequest
+
import asyncio
import json
from pathlib import Path
from typing import Any, Literal
-from unittest.mock import AsyncMock, Mock
import httpx
import pytest
@@ -30,14 +32,15 @@ def test_api_parser_serializes_request_ocr_mode(
source = tmp_path / "demo.pdf"
source.write_bytes(b"%PDF-1.7\n")
parser = MinerUApiParser(api_url="http://localhost:8000", tier="flash", ocr_mode=ocr_mode)
- parser._source_features = {"local"}
payloads: list[dict[str, Any]] = []
def respond(request: httpx.Request) -> httpx.Response:
"""记录完整任务请求,在产物下载前返回已完成任务。"""
- assert request.method == "POST" and request.url.path == "/v1/parse/jobs"
+ if request.method == "GET":
+ return httpx.Response(200, content=zip_output())
+ assert request.method == "POST" and request.url.path == "/v1/tasks"
payloads.append(json.loads(request.content))
- return httpx.Response(202, json={"job_id": "job_test", "status": "completed"})
+ return httpx.Response(202, json={**task_snapshot(), "status_url": "http://localhost:8000/v1/tasks/task-one"})
transport = httpx.MockTransport(respond)
sync_client, async_client = httpx.Client, httpx.AsyncClient
@@ -52,17 +55,24 @@ def test_api_parser_serializes_request_ocr_mode(
monkeypatch.setattr(api_client.httpx, "Client", create_sync_client)
monkeypatch.setattr(api_client.httpx, "AsyncClient", create_async_client)
- monkeypatch.setattr(parser, "_build_result", Mock())
- monkeypatch.setattr(parser, "_async_build_result", AsyncMock())
if use_async:
asyncio.run(parser.parse_async(source, page_range="1"))
else:
parser.parse(source, page_range="1")
expected: dict[str, Any] = {
- "files": [{"source": {"type": "local", "path": str(source)}, "page_range": "1"}],
+ "manifest": [
+ {
+ "client_file_id": "file",
+ "filename": source.name,
+ "bytes": source.stat().st_size,
+ "sha256sum": api_client._sha256_file(source),
+ "page_range": "1",
+ }
+ ],
"tier": "flash",
- "output_formats": ["middle_json"],
+ "output_formats": ["default"],
+ "include_model_output": False,
}
if ocr_mode is not None:
expected["ocr_mode"] = ocr_mode
@@ -82,9 +92,9 @@ def test_api_server_rejects_invalid_request_ocr_mode(tmp_path: Path, value: obje
app = api_server.create_app(upload_dir=str(tmp_path), tier="flash")
with TestClient(app) as client:
response = client.post(
- "/v1/parse/jobs",
+ "/v1/tasks",
json={
- "files": [{"source": {"type": "inline", "name": "demo.pdf", "data": "JVBERi0xLjcK"}}],
+ "manifest": [{"client_file_id": "file", "filename": "demo.pdf", "bytes": 9}],
"ocr_mode": value,
},
)
@@ -96,7 +106,7 @@ def test_api_server_rejects_invalid_request_ocr_mode(tmp_path: Path, value: obje
def test_api_server_ocr_defaults_to_request_auto_without_startup_configuration(tmp_path: Path) -> None:
"""请求缺省值固定为 auto,应用生命周期不保存全局 OCR 模式。"""
- request = api_server.CreateJobRequest.model_validate({"files": [{"source": {"type": "local", "path": "/tmp/demo.pdf"}}]})
+ request = CreateTaskRequest.model_validate({"manifest": [{"client_file_id": "file", "filename": "demo.pdf", "bytes": 9}]})
assert request.ocr_mode == "auto"
app = api_server.create_app(upload_dir=str(tmp_path), tier="flash")
assert not hasattr(app.state, "ocr_mode")
diff --git a/tests/unittest/test_parser_api_status.py b/tests/unittest/test_parser_api_status.py
index e2dbb2e0..5f8aafe8 100644
--- a/tests/unittest/test_parser_api_status.py
+++ b/tests/unittest/test_parser_api_status.py
@@ -1,5 +1,8 @@
"""同步和异步 V1 parser 的状态观察回调合同。"""
+from task_test_utils import build_default_zip
+
+
from docvortex.schema import Producer
from mineru.integrations.docvortex import build_metadata
@@ -41,7 +44,6 @@ def test_parse_notifies_submission_and_polls_before_result_building(
source = tmp_path / "source.pdf"
source.write_bytes(b"%PDF-1.7\n")
parser = MinerUApiParser(api_url="http://localhost:8000", tier="flash")
- parser._source_features = {"local"}
notifications: list[ApiJobStatus] = []
durations: list[float] = []
delays: list[float] = []
@@ -61,9 +63,29 @@ def test_parse_notifies_submission_and_polls_before_result_building(
def respond(request: httpx.Request) -> httpx.Response:
"""只模拟创建和轮询任务,保留正式 HTTP 调用及状态传播路径。"""
+ if request.url.path == "/result":
+ assert notifications == list(statuses)
+ assert durations == [250.0]
+ return httpx.Response(200, content=build_default_zip(result))
methods.append(request.method)
+ status = next(replies)
return httpx.Response(
- 200, json={"job_id": "job_test", "status": next(replies), "files": [{"parse": {"duration_ms": 250}}]}
+ 200,
+ json={
+ "task_id": "task_test",
+ "status_url": "/v1/tasks/task_test",
+ "status": status,
+ "output_formats": ["default"],
+ "files": [
+ {
+ "client_file_id": "file",
+ "upload_status": "ready",
+ "status": "completed" if status == "partial" else status,
+ "parse": {"duration_ms": 250},
+ "result_url": "/result",
+ }
+ ],
+ },
)
transport = httpx.MockTransport(respond)
@@ -87,24 +109,10 @@ def test_parse_notifies_submission_and_polls_before_result_building(
if callback_raises:
raise RuntimeError("observer failed")
- def build_result(job: dict[str, Any], file_name: str) -> ParseResult:
- """确认所有状态在开始下载/构建结果之前已经通知。"""
- assert notifications == list(statuses)
- assert durations == ([250.0] if statuses[-1] in {"completed", "partial"} else [])
- assert file_name == source.name
- api_client._raise_for_terminal_job_error(job)
- return result
-
- async def build_result_async(job: dict[str, Any], file_name: str) -> ParseResult:
- """为异步解析复用相同的终态和通知时序断言。"""
- return build_result(job, file_name)
-
monkeypatch.setattr(httpx, "Client", create_sync_client)
monkeypatch.setattr(httpx, "AsyncClient", create_async_client)
monkeypatch.setattr(api_client.time, "sleep", delays.append)
monkeypatch.setattr(api_client.asyncio, "sleep", sleep_async)
- monkeypatch.setattr(parser, "_build_result", build_result)
- monkeypatch.setattr(parser, "_async_build_result", build_result_async)
def parse() -> ParseResult:
"""统一执行两个公共入口,确保回调参数一路透传。"""
@@ -116,7 +124,7 @@ def test_parse_notifies_submission_and_polls_before_result_building(
with pytest.raises(Exception, match=f"ended with status {statuses[-1]}"):
parse()
else:
- assert parse() is result
+ assert parse().middle_json == result.middle_json
assert notifications == list(statuses)
assert methods == ["POST", *(["GET"] * (len(statuses) - 1))]
assert delays == [1] * (len(statuses) - 1)
diff --git a/tests/unittest/test_pdf_page_id.py b/tests/unittest/test_pdf_page_id.py
index 0f85148f..a67a59c3 100644
--- a/tests/unittest/test_pdf_page_id.py
+++ b/tests/unittest/test_pdf_page_id.py
@@ -1,5 +1,7 @@
from __future__ import annotations
+from task_test_utils import stored_source
+
import asyncio
import io
from pathlib import Path
@@ -150,8 +152,7 @@ def test_remote_clients_validate_before_network(tmp_path: Path, monkeypatch: pyt
parser = MinerUApiParser(api_url="http://example.test", tier="flash")
build_source = Mock(side_effect=AssertionError("must not upload"))
async_build_source = AsyncMock(side_effect=AssertionError("must not upload"))
- monkeypatch.setattr(parser, "_build_source", build_source)
- monkeypatch.setattr(parser, "_async_build_source", async_build_source)
+ monkeypatch.setattr(parser, "_payload", build_source)
with pytest.raises(InvalidRequestError):
parser.parse(source, page_range=raw)
with pytest.raises(InvalidRequestError):
@@ -168,25 +169,25 @@ def test_remote_clients_validate_before_network(tmp_path: Path, monkeypatch: pyt
def test_api_rejects_bad_ranges_before_creating_jobs(tmp_path: Path) -> None:
"""直接 HTTP 请求校验共享语法,错误定位到批量请求的具体文件。"""
- app = api_server.create_app(upload_dir=str(tmp_path / "api"), tier="flash", allow_local_source=True)
+ app = api_server.create_app(upload_dir=str(tmp_path / "api"), tier="flash")
with TestClient(app) as client:
for raw in INVALID_RANGES:
response = client.post(
- "/v1/parse/jobs",
+ "/v1/tasks",
json={
"tier": "flash",
- "files": [
- {"source": {"type": "local", "path": "/unused.pdf"}},
- {"source": {"type": "local", "path": "/unused.pdf"}, "page_range": raw},
+ "manifest": [
+ {"client_file_id": "first", "filename": "a.pdf", "bytes": 1},
+ {"client_file_id": "second", "filename": "b.pdf", "bytes": 1, "page_range": raw},
],
},
)
assert response.status_code == 400
error = response.json()["error"]
assert error["code"] == "page_range_invalid"
- assert error["param"] == "files.1.page_range"
+ assert error["param"] == "manifest.1.page_range"
schema = client.get("/openapi.json").json()
- assert "r1" in schema["components"]["schemas"]["JobFileEntry"]["properties"]["page_range"]["description"]
+ assert "r1" in schema["components"]["schemas"]["TaskFileStatus"]["properties"]["page_range"]["description"]
def test_cli_rejects_old_syntax_and_kit_preserves_range_error(tmp_path: Path) -> None:
@@ -251,18 +252,18 @@ def test_async_job_resolves_relative_ranges_and_reports_file_errors(tmp_path: Pa
source = tmp_path / "numbered.pdf"
_numbered_pdf(source)
store = api_server.FileStore(tmp_path / "files")
- request = api_server.CreateJobRequest.model_validate(
+ request = api_server._ParseRequest.model_validate(
{
"tier": "flash",
- "output_formats": ["middle_json"],
+ "output_formats": ["default"],
"files": [
- {"source": {"type": "local", "path": str(source)}, "page_range": "1,r2-r1"},
- {"source": {"type": "local", "path": str(source)}, "page_range": "20"},
+ {"source": stored_source(store, source), "page_range": "1,r2-r1"},
+ {"source": stored_source(store, source), "page_range": "20"},
],
}
)
record = api_server.JobStore().create(request, store)
- asyncio.run(api_server._run_job(record, request, store, image_analysis=True, allow_local_source=True))
+ asyncio.run(api_server._run_job(record, request, store, image_analysis=True))
assert record.status == "partial"
assert record.files[0].page_range == "1,4-5"
assert record.files[0].status == "completed"
diff --git a/tests/unittest/test_resource_retention.py b/tests/unittest/test_resource_retention.py
index 6971defe..2ca93d37 100644
--- a/tests/unittest/test_resource_retention.py
+++ b/tests/unittest/test_resource_retention.py
@@ -2,6 +2,7 @@
from __future__ import annotations
+
import asyncio
import hashlib
import os
@@ -13,7 +14,7 @@ import pytest
from fastapi.testclient import TestClient
from mineru.parser import api_server
-from mineru.parser.api_server import ApiServerError, CreateJobRequest, FileStore, JobStore
+from mineru.parser.api_server import ApiServerError, _ParseRequest, FileStore, JobStore
from mineru.kit.router.resources import ResourceRegistry, SourceFileStore, StoredSourceFile
from mineru.utils.retention import resolve_retention_seconds
@@ -67,7 +68,7 @@ def test_shared_hash_and_queue_leases_survive_expired_views(tmp_path: Path, cloc
store.release_inputs(leases[0])
store.collect_expired()
assert store._blob_abs(sha).is_file()
- data = asyncio.run(api_server._extract_bytes(api_server.FileIdSource(file_id=second), store, input_blobs=leases[1]))
+ data = asyncio.run(api_server._extract_bytes(api_server._StoredSource(file_id=second), store, input_blobs=leases[1]))
assert data.data == b"document"
store.release_inputs(leases[1])
store.collect_expired()
@@ -82,7 +83,7 @@ def test_output_retention_starts_after_cleanup(tmp_path: Path, clock: list[int])
files = FileStore(tmp_path / "files", retention_seconds=10)
source, sha = _source(files, tmp_path)
jobs = JobStore(concurrency=1, retention_seconds=10)
- req = CreateJobRequest.model_validate({"tier": "flash", "files": [{"source": {"type": "file_id", "file_id": source}}]})
+ req = _ParseRequest.model_validate({"tier": "flash", "files": [{"source": {"type": "file_id", "file_id": source}}]})
gate = asyncio.Event()
records = [jobs.create(req, files) for _ in range(2)]
outputs: list[str] = []
@@ -211,7 +212,7 @@ def test_queued_cancel_releases_input_before_execution(tmp_path: Path, clock: li
files = FileStore(tmp_path / "files", retention_seconds=10)
source, sha = _source(files, tmp_path)
jobs = JobStore(retention_seconds=10)
- req = CreateJobRequest.model_validate({"tier": "flash", "files": [{"source": {"type": "file_id", "file_id": source}}]})
+ req = _ParseRequest.model_validate({"tier": "flash", "files": [{"source": {"type": "file_id", "file_id": source}}]})
rec = jobs.create(req, files)
async def forbidden() -> None:
@@ -271,94 +272,40 @@ def test_gc_interleaves_with_blob_writes_and_downloads(tmp_path: Path, clock: li
assert not files._files and not files._pins
-def test_router_pending_copy_cleanup_keeps_terminal_job_visible(clock: list[int]) -> None:
- """副本尚未清理完成时,任务不应开始保留期或从用户列表消失。"""
- from mineru.kit.router.resources import CopiedInputFile
-
+def test_router_task_result_lease_keeps_terminal_execution_visible(clock: list[int]) -> None:
+ """父任务持有结果租约时,内部执行终态不会提前被回收。"""
registry = ResourceRegistry(retention_seconds=10)
- route = registry.register("job", owner_scope="one", worker_id="worker", upstream_id="job-up")
- route.metadata["payload"] = {"status": "failed", "finished_at": JobStore._now()}
- route.metadata["copied_inputs"] = [CopiedInputFile("file-one", "one", "worker", "file-copy")]
- clock[0] += 11
+ route = registry.register("job", owner_scope="one", worker_id="worker", upstream_id="child")
+ route.metadata.update(payload={"status": "completed"}, task_result_held=True)
+ clock[0] += 20
assert registry.find("job", route.public_id) is route
- assert registry.list("job") == [route]
- assert registry.list("job", include_expired=True) == [route]
-
-
-def test_router_retention_starts_after_copy_cleanup_retry(clock: list[int]) -> None:
- """上游早已结束但副本删除失败时,重试成功后仍享有完整保留期。"""
- from test_v1_router import _FakeV1Upstream, _make_router, _upload_file
- from mineru.kit.router.app import _reconcile_jobs_once
-
- first = _FakeV1Upstream("worker-a", ("standard",))
- second = _FakeV1Upstream("worker-b", ("advanced",))
- app, _ = _make_router(first, second)
- app.state.registry.retention_seconds = 10
- headers = {"authorization": "Bearer retained-copy"}
- with TestClient(app) as client:
- source = _upload_file(client, token="retained-copy", content=b"pdf")
- created = client.post(
- "/v1/parse/jobs",
- headers=headers,
- json={"tier": "advanced", "files": [{"source": {"type": "file_id", "file_id": source}}]},
- )
- assert created.status_code == 202
- job_id = created.json()["job_id"]
- route = app.state.registry.get("job", job_id)
- copied = route.metadata["copied_inputs"][0]
- second.fail_file_delete_once.add(copied.upstream_file_id)
- assert client.get(f"/v1/parse/jobs/{job_id}", headers=headers).status_code == 200
- route.metadata["payload"]["finished_at"] = JobStore._now()
- assert route.metadata.get("copied_inputs")
- clock[0] += 11
- app.state.registry.collect_expired(app.state.source_store)
- assert app.state.registry.find("job", job_id) is route
- assert client.get("/v1/parse/jobs", headers=headers).json()["data"][0]["job_id"] == job_id
- client.portal.call(_reconcile_jobs_once, app.state.registry, app.state.worker_pool)
- assert not route.metadata.get("copied_inputs")
- assert copied.upstream_file_id not in second.files
- clock[0] += 9
- assert client.get(f"/v1/parse/jobs/{job_id}", headers=headers).status_code == 200
- clock[0] += 1
- assert client.get(f"/v1/parse/jobs/{job_id}", headers=headers).status_code == 404
+ route.metadata["task_result_retain_until"] = clock[0] + 10
+ clock[0] += 11
+ assert registry.find("job", route.public_id) is None
def test_router_usage_survives_expiry_collection_and_worker_removal(clock: list[int]) -> None:
- """过期隐藏、重复扫描与 worker 移除均不能减少或重复累计调用方用量。"""
- from test_v1_router import _FakeV1Upstream, _make_router
-
- app, _ = _make_router(_FakeV1Upstream("worker-a", ("standard",)))
- app.state.registry.retention_seconds = 10
- headers = {"authorization": "Bearer lifetime"}
- body = {"tier": "standard", "files": [{"source": {"type": "inline", "name": "demo.pdf", "data": "cGRm"}}]}
- with TestClient(app) as client:
- for _ in range(2):
- created = client.post("/v1/parse/jobs", headers=headers, json=body)
- assert created.status_code == 202
- job_id = created.json()["job_id"]
- assert client.get(f"/v1/parse/jobs/{job_id}", headers=headers).status_code == 200
- before = client.get("/v1/usage", headers=headers).json()["current"]
- assert before == {"pages_processed": 0, "files_processed": 2, "jobs_created": 2}
+ """内部逐文件执行计数在过期和 worker 移除后仍保留,并隔离调用方。"""
+ registry = ResourceRegistry(retention_seconds=10)
+ sources = SourceFileStore(retention_seconds=10)
+ try:
+ for index in range(2):
+ route = registry.register("job", owner_scope="owner", worker_id="worker", upstream_id=str(index))
+ route.metadata.update(
+ payload={"status": "completed", "files": [{"status": "completed"}]}, finalized_at=JobStore._now()
+ )
+ before = registry.usage("owner")
+ assert before == {"jobs_created": 2, "files_processed": 2}
clock[0] += 11
- assert client.get("/v1/parse/jobs", headers=headers).json()["data"] == []
- assert client.get("/v1/usage", headers=headers).json()["current"] == before
- for _ in range(2):
- app.state.registry.collect_expired(app.state.source_store)
- assert not app.state.registry.list("job", include_expired=True)
- assert client.get("/v1/usage", headers=headers).json()["current"] == before
- created = client.post("/v1/parse/jobs", headers=headers, json=body)
- assert created.status_code == 202
- route = app.state.registry.get("job", created.json()["job_id"])
- app.state.registry.remove_worker(route.worker_id)
- assert client.get("/v1/usage", headers=headers).json()["current"] == {
- **before,
- "jobs_created": 3,
- }
- assert client.get("/v1/usage").json()["current"] == {
- "pages_processed": 0,
- "files_processed": 0,
- "jobs_created": 0,
- }
+ registry.collect_expired(sources)
+ assert registry.usage("owner") == before and not registry.list("job", include_expired=True)
+ route = registry.register("job", owner_scope="owner", worker_id="worker", upstream_id="new")
+ route.metadata["payload"] = {"status": "running", "files": [{"status": "running"}]}
+ registry.remove_worker("worker")
+ assert registry.usage("owner") == {"jobs_created": 3, "files_processed": 2}
+ assert registry.usage("anonymous") == {"jobs_created": 0, "files_processed": 0}
+ finally:
+ sources.close()
def test_retention_keeps_cumulative_usage(tmp_path: Path, clock: list[int]) -> None:
@@ -366,7 +313,7 @@ def test_retention_keeps_cumulative_usage(tmp_path: Path, clock: list[int]) -> N
files = FileStore(tmp_path / "files", retention_seconds=10)
source, _ = _source(files, tmp_path)
jobs = JobStore(retention_seconds=10)
- request = CreateJobRequest.model_validate({"tier": "flash", "files": [{"source": {"type": "file_id", "file_id": source}}]})
+ request = _ParseRequest.model_validate({"tier": "flash", "files": [{"source": {"type": "file_id", "file_id": source}}]})
rec = jobs.create(request, files)
rec.status = rec.files[0].status = "completed"
rec.files[0].page_range = "1-3"
@@ -396,55 +343,21 @@ def test_kit_cli_forwards_explicit_retention(monkeypatch: pytest.MonkeyPatch) ->
def test_external_worker_lost_task_releases_load_and_keeps_snapshot() -> None:
- """外部 worker 重启丢失内存任务时,Router 不得永久占用名额或丢弃失败历史。"""
+ """worker Tasks 返回 404 时后台收敛负载,并保存失败执行快照。"""
from test_v1_router import _FakeV1Upstream, _make_router
from mineru.kit.router.app import _reconcile_jobs_once
- upstream = _FakeV1Upstream("worker-a", ("standard",))
- app, _ = _make_router(upstream)
+ app, _ = _make_router(_FakeV1Upstream("worker-a", ("standard",)))
with TestClient(app) as client:
- created = client.post(
- "/v1/parse/jobs",
- json={"tier": "standard", "files": [{"source": {"type": "inline", "name": "demo.pdf", "data": "cGRm"}}]},
+ pool = app.state.worker_pool
+ reservation = pool.reserve_job(tier="standard", required_sources={"upload"})
+ route = app.state.registry.register(
+ "job", owner_scope="anonymous", worker_id=reservation.worker.worker_id, upstream_id="lost-task"
)
- assert created.status_code == 202
- job_id = created.json()["job_id"]
- route = app.state.registry.get("job", job_id)
- worker = app.state.worker_pool.get(route.worker_id)
- assert worker.active_jobs == 1
- upstream.jobs.clear()
- client.portal.call(_reconcile_jobs_once, app.state.registry, app.state.worker_pool)
- assert worker.active_jobs == 0
- response = client.get(f"/v1/parse/jobs/{job_id}")
- assert response.status_code == 200 and response.json()["status"] == "failed"
- assert response.json()["files"][0]["error"]["code"] == "upstream_job_lost"
-
-
-def test_lost_task_retries_failed_cross_worker_copy_cleanup() -> None:
- """上游任务丢失后副本删除失败一次,后续后台轮次仍继续回收并释放 alias。"""
- from test_v1_router import _FakeV1Upstream, _make_router, _upload_file
- from mineru.kit.router.app import _reconcile_jobs_once
-
- first = _FakeV1Upstream("worker-a", ("standard",))
- second = _FakeV1Upstream("worker-b", ("advanced",))
- app, _ = _make_router(first, second)
- headers = {"authorization": "Bearer lost-copy"}
- with TestClient(app) as client:
- source = _upload_file(client, token="lost-copy", content=b"pdf")
- response = client.post(
- "/v1/parse/jobs",
- headers=headers,
- json={"tier": "advanced", "files": [{"source": {"type": "file_id", "file_id": source}}]},
+ route.metadata.update(
+ active_counted=True, reservation=reservation, payload={"status": "running", "files": [{"status": "queued"}]}
)
- assert response.status_code == 202
- route = app.state.registry.get("job", response.json()["job_id"])
- copied = route.metadata["copied_inputs"][0]
- second.jobs.clear()
- second.fail_file_delete_once.add(copied.upstream_file_id)
- client.portal.call(_reconcile_jobs_once, app.state.registry, app.state.worker_pool)
- assert route.metadata["upstream_lost"] and route.metadata["copied_inputs"]
- assert second.files.get(copied.upstream_file_id) is not None
- client.portal.call(_reconcile_jobs_once, app.state.registry, app.state.worker_pool)
- assert not route.metadata.get("copied_inputs")
- assert copied.upstream_file_id not in second.files
- assert app.state.registry.find_upstream("file", copied.owner_scope, copied.worker_id, copied.upstream_file_id) is None
+ client.portal.call(_reconcile_jobs_once, app.state.registry, pool)
+ assert reservation.worker.active_jobs == 0
+ assert route.metadata["payload"]["status"] == "failed"
+ assert route.metadata["payload"]["files"][0]["error"]["code"] == "upstream_task_lost"
diff --git a/tests/unittest/test_router_balancing.py b/tests/unittest/test_router_balancing.py
index 3d8032c2..05fc881e 100644
--- a/tests/unittest/test_router_balancing.py
+++ b/tests/unittest/test_router_balancing.py
@@ -11,14 +11,14 @@ from typing import Any
import httpx
import pytest
from fastapi.testclient import TestClient
-from test_v1_router import _FakeV1Upstream, _make_router, _upload_file
+from test_v1_router import _FakeV1Upstream, _make_router
from test_task_api import HTML, _WorkerTransport, _submit, _terminal, _upload
from mineru.parser import api_server
from mineru.kit.router import RouterSettings, create_app
from mineru.kit.router.workers import JobReservation
-BODY = {"tier": "standard", "files": [{"source": {"type": "inline", "name": "source.pdf", "data": "aGVsbG8="}}]}
+BODY = {"tier": "standard", "manifest": [{"client_file_id": "file", "filename": "source.pdf", "bytes": 5}]}
@pytest.mark.parametrize("formats", [["unknown"], "markdown", None, 1, {}, [{}], [1]])
@@ -30,26 +30,15 @@ def test_output_format_validation_precedes_worker_selection(formats: Any, tmp_pa
app, _ = _make_router(upstream)
body = {**BODY, "tier": "flash", "output_formats": formats}
with TestClient(create_api_app(upload_dir=str(tmp_path / "files"), tier="flash")) as direct:
- expected = direct.post("/v1/parse/jobs", json=body)
+ expected = direct.post("/v1/tasks", json=body)
with TestClient(app, raise_server_exceptions=False) as client:
- response = client.post("/v1/parse/jobs", json=body)
+ response = client.post("/v1/tasks", json=body)
assert response.status_code == expected.status_code == 400, response.text
assert response.json()["error"]["code"] == expected.json()["error"]["code"]
- assert upstream.job_counter == 0
+ assert not app.state.registry.list("job")
assert app.state.worker_pool.workers[0].active_jobs == 0
-def test_valid_unavailable_output_format_remains_a_capability_error() -> None:
- """合法格式确实未被 worker 支持时,继续返回 503 能力不可用。"""
- upstream = _FakeV1Upstream("worker-a", ("standard",))
- app, _ = _make_router(upstream)
- with TestClient(app) as client:
- response = client.post("/v1/parse/jobs", json={**BODY, "output_formats": ["structured_content"]})
- assert response.status_code == 503
- assert response.json()["error"]["code"] == "quality_tier_unavailable"
- assert upstream.job_counter == 0 and app.state.worker_pool.workers[0].active_jobs == 0
-
-
def test_task_auth_preflight_uses_a_capable_worker_without_reservation() -> None:
"""待上传任务只在匹配 tier 的 worker 校验凭证,不能被无关 worker 的权限误判。"""
first = _FakeV1Upstream("worker-a", ("flash",))
@@ -106,77 +95,6 @@ def test_task_auth_preflight_uses_a_capable_worker_without_reservation() -> None
assert [worker.active_jobs for worker in app.state.worker_pool.workers] == [0, 0]
-def test_same_caller_uploads_round_robin_without_affinity() -> None:
- """同一 IP 与凭证的独立文件应均衡上传,不再固定在一个 worker。"""
- first, second = _FakeV1Upstream("worker-a", ("standard",)), _FakeV1Upstream("worker-b", ("standard",))
- app, _ = _make_router(first, second)
- with TestClient(app) as client:
- for index in range(10):
- _upload_file(client, token="same-caller", content=f"unique-pdf-{index}".encode())
- assert (first.upload_counter, second.upload_counter) == (5, 5)
-
-
-def test_completed_inline_jobs_rotate_and_reads_do_not_advance_cursors() -> None:
- """逐个完成的 inline 任务也轮询,状态和能力查询不能改变分配顺序。"""
- first, second = _FakeV1Upstream("worker-a", ("standard",)), _FakeV1Upstream("worker-b", ("standard",))
- app, _ = _make_router(first, second)
- with TestClient(app) as client:
- for index in range(10):
- response = client.post("/v1/parse/jobs", json=BODY)
- assert response.status_code == 202, response.text
- task_id = response.json().get("task_id") or response.json()["job_id"]
- assert client.get(f"/v1/parse/jobs/{task_id}").json()["status"] == "completed"
- client.get("/v1/health")
- client.get("/v1/tiers")
- client.get("/v1/models")
- assert [worker.active_jobs for worker in app.state.worker_pool.workers] == [0, 0]
- assert (first.job_counter, second.job_counter) == (5, 5)
-
-
-def test_delayed_concurrent_submission_pre_reserves_worker_load() -> None:
- """上游创建响应延迟时,十个并发请求仍应在网络等待前分摊负载。"""
- first, second = _FakeV1Upstream("worker-a", ("standard",)), _FakeV1Upstream("worker-b", ("standard",))
- app, _ = _make_router(first, second)
- for upstream in (first, second):
- original = upstream._create_job
-
- async def delayed(request: httpx.Request, create: Any = original) -> httpx.Response:
- """让所有选择发生在首个上游成功响应之前。"""
- await asyncio.sleep(0.05)
- return await create(request)
-
- upstream._create_job = delayed # type: ignore[method-assign]
-
- async def run() -> list[httpx.Response]:
- """在 Router 所属事件循环内同时提交新旧入口。"""
- async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://router") as client:
- return await asyncio.gather(*(client.post("/v1/parse/jobs", json=BODY) for index in range(10)))
-
- with TestClient(app) as client:
- responses = client.portal.call(run)
- assert all(response.status_code == 202 for response in responses)
- assert [worker.active_jobs for worker in app.state.worker_pool.workers] == [5, 5]
- assert (first.job_counter, second.job_counter) == (5, 5)
-
-
-def test_file_ownership_cannot_override_lower_load() -> None:
- """同一文件的十个排队任务不能都固定在其所属 worker。"""
- first, second = _FakeV1Upstream("worker-a", ("standard",)), _FakeV1Upstream("worker-b", ("standard",))
- app, _ = _make_router(first, second)
- headers = {"authorization": "Bearer owner"}
- with TestClient(app) as client:
- file_id = _upload_file(client, token="owner", content=b"source")
- for _ in range(10):
- response = client.post(
- "/v1/parse/jobs",
- headers=headers,
- json={"tier": "standard", "files": [{"source": {"type": "file_id", "file_id": file_id}}]},
- )
- assert response.status_code == 202, response.text
- assert (first.job_counter, second.job_counter) == (5, 5)
- assert first.source_download_attempts == second.source_download_attempts == 0
-
-
def test_cached_tasks_still_balance(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""真实 tasks 命中缓存后仍按负载选择 worker,等待上传不预占模型名额。"""
release = threading.Event()
@@ -228,12 +146,12 @@ def test_generation_change_and_duplicate_release_do_not_touch_new_jobs() -> None
app, _ = _make_router(upstream)
with TestClient(app):
pool = app.state.worker_pool
- old = pool.reserve_job(tier="standard", required_sources={"inline"})
+ old = pool.reserve_job(tier="standard", required_sources={"upload"})
assert isinstance(old, JobReservation)
worker = old.worker
worker.generation += 1
worker.active_jobs = 0
- new = pool.reserve_job(tier="standard", required_sources={"inline"})
+ new = pool.reserve_job(tier="standard", required_sources={"upload"})
assert isinstance(new, JobReservation)
old.release()
assert worker.active_jobs == 1
@@ -242,103 +160,6 @@ def test_generation_change_and_duplicate_release_do_not_touch_new_jobs() -> None
assert worker.active_jobs == 0
-def test_late_job_response_after_replacement_is_not_published() -> None:
- """旧进程的创建响应不能在 replacement 后注册为新 worker 的任务。"""
- upstream = _FakeV1Upstream("worker-a", ("standard",))
- app, _ = _make_router(upstream)
- original = upstream._create_job
- new_reservations: list[JobReservation] = []
-
- async def replaced(request: httpx.Request) -> httpx.Response:
- """在响应到达前模拟重启及新 generation 接单。"""
- response = await original(request)
- pool = app.state.worker_pool
- worker = pool.workers[0]
- worker.generation += 1
- worker.active_jobs = 0
- reservation = pool.reserve_job(tier="standard", required_sources={"inline"})
- assert reservation is not None
- new_reservations.append(reservation)
- return response
-
- upstream._create_job = replaced # type: ignore[method-assign]
- with TestClient(app) as client:
- response = client.post("/v1/parse/jobs", json=BODY)
- assert response.status_code == 503, response.text
- assert not app.state.registry.list("job")
- assert app.state.worker_pool.workers[0].active_jobs == 1
- new_reservations[0].release()
-
-
-def test_output_redirect_does_not_fetch_remote_body() -> None:
- """Files 下载只转发重定向,不在 Router 内回源下载外部结果正文。"""
- upstream = _FakeV1Upstream("worker-a", ("standard",))
-
- async def handler(request: httpx.Request) -> httpx.Response:
- """worker 返回外部下载地址时,Router 不能主动请求 CDN 字节。"""
- assert request.url.host != "cdn.example"
- if request.url.path.endswith("/content"):
- return httpx.Response(302, headers={"location": "https://cdn.example/output"}, request=request)
- return await upstream.handle(request)
-
- app = create_app(
- RouterSettings(upstream_urls=("http://worker-a",), local_gpus="none", worker_refresh_interval_seconds=0),
- transport=httpx.MockTransport(handler),
- )
- with TestClient(app) as client:
- headers = {"authorization": "Bearer private-key"}
- created = client.post("/v1/parse/jobs", headers=headers, json=BODY).json()
- result = client.get(f"/v1/parse/jobs/{created['job_id']}", headers=headers).json()
- file_id = result["files"][0]["output_files"]["markdown"]["file_id"]
- response = client.get(f"/v1/files/{file_id}/content", headers=headers, follow_redirects=False)
- assert response.status_code == 302 and response.headers["location"] == "https://cdn.example/output"
-
-
-def test_creation_failure_and_cancelled_copy_release_reservations() -> None:
- """既有 Jobs 创建失败与跨 worker 复制取消仍归还预占,不损坏原源文件。"""
- upstream = _FakeV1Upstream("worker-a", ("standard",), fail_jobs=True)
- app, _ = _make_router(upstream)
- with TestClient(app) as client:
- failed = client.post("/v1/parse/jobs", json=BODY)
- assert failed.status_code == 502
- assert app.state.worker_pool.workers[0].active_jobs == 0
-
- first = _FakeV1Upstream("worker-a", ("standard",))
- second = _FakeV1Upstream("worker-b", ("standard",))
- original = second.handle
-
- async def canceled_put(request: httpx.Request) -> httpx.Response:
- """仅取消目标 worker 的复制上传,原文件及其私有缓存仍可用。"""
- if request.method == "PUT":
- raise asyncio.CancelledError()
- return await original(request)
-
- second.handle = canceled_put # type: ignore[method-assign]
- app, _ = _make_router(first, second)
- with TestClient(app) as client:
- file_id = _upload_file(client, token="owner", content=b"source")
- app.state.worker_pool.workers[0].active_jobs = 1
-
- async def run_cancel() -> None:
- """在 Router 所属事件循环捕获模拟取消,随后核对清理结果。"""
- async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://router") as asynchronous:
- try:
- await asynchronous.post(
- "/v1/parse/jobs",
- headers={"authorization": "Bearer owner"},
- json={"tier": "standard", "files": [{"source": {"type": "file_id", "file_id": file_id}}]},
- )
- except asyncio.CancelledError:
- pass
-
- client.portal.call(run_cancel)
- assert [worker.active_jobs for worker in app.state.worker_pool.workers] == [1, 0]
- assert app.state.source_store.find_file(file_id) is not None
- assert not app.state.source_store._uploads
- assert all(upload["status"] != "pending" for upload in second.uploads.values())
- app.state.worker_pool.workers[0].active_jobs = 0
-
-
def test_task_submission_failure_and_replacement_release_only_owned_load(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
@@ -364,7 +185,7 @@ def test_task_submission_failure_and_replacement_release_only_owned_load(
worker = pool.workers[0]
worker.generation += 1
worker.active_jobs = 0
- reservation = pool.reserve_job(tier="flash", required_sources={"file_id"})
+ reservation = pool.reserve_job(tier="flash", required_sources={"upload"})
assert reservation is not None
new_reservations.append(reservation)
return response
diff --git a/tests/unittest/test_task_api.py b/tests/unittest/test_task_api.py
index e07e2917..5f6def85 100644
--- a/tests/unittest/test_task_api.py
+++ b/tests/unittest/test_task_api.py
@@ -446,7 +446,7 @@ def test_partial_and_packaging_failure_have_honest_results(
assert "markdown.md" in _archive(client, result)
original = task_outputs.build_task_output
- def fail_packaging(result: Any, formats: Any, directory: Path) -> Path:
+ def fail_packaging(result: Any, formats: Any, directory: Path, *, include_model_output: bool = False) -> Path:
"""仅让第二个输入导出失败,验证文件级隔离及诚实的成功状态。"""
if "Second content" in result.markdown():
raise OSError("simulated full disk")
@@ -973,3 +973,113 @@ def test_router_ends_local_result_lease_even_if_worker_rejects_release(tmp_path:
not route.metadata.get("task_result_held") for route in app.state.registry.list("file", include_expired=True)
)
assert result["files"][0]["result_url"]
+
+
[email protected]("order", ["asc", "desc"])
+def test_parent_task_listing_filters_and_stable_pagination(
+ service: tuple[TestClient, FastAPI],
+ monkeypatch: pytest.MonkeyPatch,
+ order: str,
+) -> None:
+ """同秒父任务按 ID 稳定分页,过滤状态和调用方后不泄露子执行。"""
+ client, app = service
+ monkeypatch.setattr(api_server.JobStore, "_now", staticmethod(lambda: "2026-10-22T00:00:00Z"))
+ tasks = [_submit(client, [(f"input-{index}.html", HTML)], hashes=False) for index in range(4)]
+ for task in tasks:
+ app.state.task_store._records[task["task_id"]].created_at = "2026-10-22T00:00:00Z"
+ expected = sorted([task["task_id"] for task in tasks], reverse=order == "desc")
+ first = client.get("/v1/tasks", params={"limit": 2, "order": order, "status": "waiting_upload"}).json()
+ assert [item["task_id"] for item in first["data"]] == expected[:2]
+ assert first["has_more"] is True and first["first_id"] == expected[0]
+ assert all(
+ item["file_count"] == 1 and set(item) == {"task_id", "created_at", "status", "file_count"} for item in first["data"]
+ )
+ second = client.get("/v1/tasks", params={"limit": 2, "order": order, "after": first["last_id"]}).json()
+ assert [item["task_id"] for item in second["data"]] == expected[2:]
+ assert not second["has_more"]
+ assert client.get("/v1/tasks", params={"status": "completed"}).json()["data"] == []
+ assert client.get("/v1/tasks", params={"created_after": "2027-01-01T00:00:00Z"}).json()["data"] == []
+ assert client.get("/v1/tasks", params={"status": "unknown"}).status_code == 400
+ assert client.get("/v1/tasks", params={"after": "missing"}).status_code == 404
+ assert client.get("/v1/tasks", params={"limit": 101}).status_code == 400
+ if hasattr(app.state, "registry"):
+ assert client.get("/v1/tasks", headers={"authorization": "Bearer other"}).json()["data"] == []
+ assert (
+ client.get("/v1/tasks", params={"after": expected[0]}, headers={"authorization": "Bearer other"}).status_code == 404
+ )
+
+
+def test_tasks_only_routes_and_health(service: tuple[TestClient, FastAPI]) -> None:
+ """所有旧入口均消失,嵌套 Task 上传及独立 Files 仍然存在。"""
+ client, _ = service
+ schema = client.get("/openapi.json").json()
+ assert not any(path.startswith(("/v1/uploads", "/v1/parse/jobs")) for path in schema["paths"])
+ for method, path in [
+ ("POST", "/v1/uploads"),
+ ("GET", "/v1/uploads/a"),
+ ("PUT", "/v1/uploads/a/content"),
+ ("POST", "/v1/uploads/a/complete"),
+ ("POST", "/v1/uploads/a/cancel"),
+ ("POST", "/v1/parse/jobs"),
+ ("GET", "/v1/parse/jobs"),
+ ("GET", "/v1/parse/jobs/a"),
+ ("DELETE", "/v1/parse/jobs/a"),
+ ]:
+ assert client.request(method, path).status_code == 404
+ features = client.get("/v1/health").json()["features"]
+ assert set(features["sources"]) == {"upload", "url"}
+ assert "output_formats" not in features and features["task_model_output"] is True
+ assert set(features["task_output_formats"]) == {"default", "html", "docx", "latex"}
+ assert "get" in schema["paths"]["/v1/tasks"]
+
+
[email protected]("enabled", [False, True])
+def test_model_output_option_reaches_worker_and_result(
+ service: tuple[TestClient, FastAPI],
+ monkeypatch: pytest.MonkeyPatch,
+ enabled: bool,
+) -> None:
+ """开关经过 Router 到实际 worker,默认结果包不含原始模型数据。"""
+ from test_tasks_sdk import middle
+ from mineru.parser.base import ParseResult
+ from mineru.types import ModelJson
+
+ async def infer(*args: Any, **kwargs: Any) -> ParseResult:
+ """替代推理但保留真实执行、导出、结果映射与下载。"""
+ document = middle()
+ return ParseResult(
+ middle_json=document,
+ _model_output=ModelJson(
+ pages=[[{"raw": "model"}]],
+ page_index_map=[0],
+ metadata=document.metadata,
+ ),
+ )
+
+ client, _ = service
+ monkeypatch.setattr(api_server, "parse_async", infer)
+ response = client.post(
+ "/v1/tasks",
+ json={
+ "manifest": [{"client_file_id": "file", "filename": "input.html", "bytes": len(HTML)}],
+ "tier": "flash",
+ "include_model_output": enabled,
+ },
+ )
+ assert response.status_code == 202, response.text
+ task = response.json()
+ assert task["include_model_output"] is enabled
+ client.put(task["uploads"][0]["upload_url"], content=HTML)
+ done = _terminal(client, task)
+ assert done["status"] == "completed" and done["include_model_output"] is enabled
+ with zipfile.ZipFile(io.BytesIO(client.get(done["files"][0]["result_url"]).content)) as archive:
+ assert ("model_output.json" in archive.namelist()) is enabled
+ rejected = client.post(
+ "/v1/tasks",
+ json={
+ "manifest": [{"client_file_id": "file", "filename": "input.html", "bytes": len(HTML)}],
+ "output_formats": ["html"],
+ "include_model_output": True,
+ },
+ )
+ assert rejected.status_code == 400
diff --git a/tests/unittest/test_task_http.py b/tests/unittest/test_task_http.py
index c3f9706a..3f922925 100644
--- a/tests/unittest/test_task_http.py
+++ b/tests/unittest/test_task_http.py
@@ -10,6 +10,7 @@ import os
import socket
import subprocess
import sys
+import tempfile
import threading
import time
import zipfile
@@ -350,54 +351,45 @@ def test_large_page_count_pdf_is_not_rejected(live_service: _LiveService, page_c
@pytest.mark.parametrize("document", ["html", "pdf"])
-def test_real_documents_match_jobs_outputs(live_service: _LiveService, document: str) -> None:
- """真实 HTML/PDF 的任务 ZIP 与既有 Jobs 产物逐项一致,不改变中间协议或素材。"""
+def test_real_documents_match_local_parser_outputs(live_service: _LiveService, document: str) -> None:
+ """真实 HTML/PDF 的 Tasks 产物与直接解析逐项一致,检查中间协议和图片字节。"""
if document == "pdf":
- path = Path(__file__).resolve().parents[2] / "examples" / "demo1.pdf"
+ path = Path(__file__).resolve().parents[2] / "demo" / "pdfs" / "demo1.pdf"
if not path.is_file():
- pytest.skip("Local real PDF examples/demo1.pdf is unavailable")
+ pytest.skip("Real PDF demo/pdfs/demo1.pdf is unavailable")
data, name, page_range = path.read_bytes(), path.name, "1-2"
else:
data, name, page_range = HTML, "source.html", ""
- formats = ["markdown", "middle_json", "structured_content", "zip"]
+ with tempfile.TemporaryDirectory() as directory:
+ source = Path(directory) / name
+ source.write_bytes(data)
+ from mineru.parser import parse
+ from mineru.parser.task_outputs import build_task_output
+ from docvortex.document.contracts import HtmlSourceContext
+
+ baseline = parse(
+ source,
+ tier="flash",
+ ocr_mode="txt",
+ page_range=page_range,
+ source_context=HtmlSourceContext() if document == "html" else None,
+ )
+ expected = build_task_output(baseline, ["default"], Path(directory) / "outputs").read_bytes()
with _client(live_service) as client:
task = _submit(client, [(name, data)], page_range=page_range, output_formats=["default"])
assert client.put(task["uploads"][0]["upload_url"], content=data).status_code == 200
result = _wait(client, task)
assert result["status"] == "completed", result
- submitted = client.post(
- "/v1/parse/jobs",
- json={
- "tier": "flash",
- "ocr_mode": "txt",
- "output_formats": formats,
- "files": [
- {"source": {"type": "file_id", "file_id": result["files"][0]["file_id"]}, "page_range": page_range or None}
- ],
- },
- )
- assert submitted.status_code == 202, submitted.text
- deadline = time.monotonic() + 60
- while True:
- baseline = client.get(f"/v1/parse/jobs/{submitted.json()['job_id']}").json()
- if baseline["status"] in TERMINAL:
- break
- assert time.monotonic() < deadline
- time.sleep(0.02)
- assert baseline["status"] == "completed", baseline
- first = client.get(result["files"][0]["result_url"])
- second = client.get(f"/v1/files/{baseline['files'][0]['output_files']['zip']['file_id']}/content")
- assert first.status_code == second.status_code == 200
- assert {k: v for k, v in _members(first.content).items() if k != "images/"} == {
- k: v for k, v in _members(second.content).items() if k != "model_output.json"
- }
+ output = client.get(result["files"][0]["result_url"])
+ assert output.status_code == 200
+ assert _members(output.content) == _members(expected)
@pytest.mark.parametrize("document", ["html", "pdf"])
def test_url_task_over_http_downloads_chunked_source(live_service: _LiveService, document: str) -> None:
"""真实 TCP URL 无需 HEAD 或长度头,保留文件页范围且不向源站发送 API Key。"""
if document == "pdf":
- sample = Path(__file__).resolve().parents[2] / "examples" / "demo1.pdf"
+ sample = Path(__file__).resolve().parents[2] / "demo" / "pdfs" / "demo1.pdf"
if sample.is_file():
data = sample.read_bytes()
else:
@@ -458,28 +450,12 @@ def test_url_task_over_http_downloads_chunked_source(live_service: _LiveService,
assert requests[0].scope["query_string"] == b"signature=a%2Fb&part=1"
assert "authorization" not in requests[0].headers
if document == "html":
- # HTML 的来源地址参与页面标识,因此与保留相同来源的 Jobs URL 路径比较。
- response = client.post(
- "/v1/parse/jobs",
- json={
- "tier": "flash",
- "ocr_mode": "txt",
- "output_formats": ["markdown", "zip"],
- "files": [{"source": {"type": "url", "url": item["url"]}}],
- },
- )
+ # 同一原始 URL 保留 HTML 来源上下文,重复执行的产物应完全一致。
+ response = client.post("/v1/tasks", json={"tier": "flash", "ocr_mode": "txt", "manifest": [item]})
assert response.status_code == 202, response.text
- job_id = response.json()["job_id"]
- deadline = time.monotonic() + 30
- while True:
- baseline_result = client.get(f"/v1/parse/jobs/{job_id}").json()
- if baseline_result["status"] in TERMINAL:
- break
- assert time.monotonic() < deadline
- time.sleep(0.02)
+ baseline_result = _wait(client, response.json())
assert baseline_result["status"] == "completed"
- output_id = baseline_result["files"][0]["output_files"]["zip"]["file_id"]
- baseline_url = f"/v1/files/{output_id}/content"
+ baseline_url = baseline_result["files"][0]["result_url"]
else:
baseline = _submit(client, [(item["filename"], data)], page_range="2")
assert client.put(baseline["uploads"][0]["upload_url"], content=data).status_code == 200
@@ -768,3 +744,52 @@ def test_client_does_not_renew_expired_upload(cloud: _CloudState, tmp_path: Path
assert result.returncode == 1, result.stdout + result.stderr
assert len(cloud.tasks) == 1 and cloud.upload_attempts == {"cloud_1/f1": 1}
assert not (tmp_path / "results/f1/a.zip").exists()
+
+
[email protected]("asynchronous", [False, True])
+def test_real_sdk_hash_cache_skips_second_upload(live_service: _LiveService, tmp_path: Path, asynchronous: bool) -> None:
+ """真实 SDK 经 TCP 解析相同文件两次,第二次命中源缓存且没有任何额外 PUT 字节。"""
+ from mineru.parser import MinerUApiParser
+
+ source = tmp_path / "sdk.html"
+ source.write_bytes(HTML)
+ parser = MinerUApiParser(api_url=live_service.url, api_key="test-key", tier="flash", include_images=True)
+ first = asyncio.run(parser.parse_async(source)) if asynchronous else parser.parse(source)
+ uploaded = sum(live_service.counts.values())
+ second = asyncio.run(parser.parse_async(source)) if asynchronous else parser.parse(source)
+ assert uploaded == len(HTML) and sum(live_service.counts.values()) == uploaded
+ assert first.to_dict() == second.to_dict()
+ assert first.markdown() == second.markdown() and "Same source and output" in first.markdown()
+
+
[email protected]("document", ["pdf", "html"])
+def test_real_gradio_handler_uses_tasks_and_saves_artifacts(live_service: _LiveService, tmp_path: Path, document: str) -> None:
+ """真实界面转换处理器经过 TCP Tasks,检查 PDF 页码、HTML 正文和实际下载文件。"""
+ from mineru.kit.gradio.app import build_gradio_app
+ from mineru.kit.gradio.client import V1ArtifactClient
+ from mineru.kit.gradio.artifacts import render_download
+ from mineru.parser.base import ParseResult
+
+ if document == "pdf":
+ source = Path(__file__).resolve().parents[2] / "demo/pdfs/demo1.pdf"
+ assert source.is_file(), "The real PDF fixture is required for the migration smoke test"
+ else:
+ source = tmp_path / "gradio.html"
+ source.write_bytes(HTML)
+ client = V1ArtifactClient(api_url=live_service.url, api_key="test-key")
+ capabilities = asyncio.run(client.discover())
+ demo = build_gradio_app(client, capabilities, output_root=tmp_path / "gradio", enable_example=False)
+ try:
+ handler = next(function.fn for function in demo.fns.values() if function.name == "convert_handler")
+ converted = asyncio.run(handler(str(source), 0, "1" if document == "pdf" else "", False))
+ state = converted[6]
+ assert state, converted[0]
+ restored = ParseResult.from_json(Path(state["middle_json_path"]).read_text())
+ assert len(restored.pages) == 1 and restored.markdown().strip()
+ assert restored.pages[0].page_idx == 0
+ assert Path(render_download(state, "docx")).is_file()
+ if document == "html":
+ assert "Same source and output" in restored.markdown()
+ assert sum(live_service.counts.values()) == source.stat().st_size
+ finally:
+ demo.close()
diff --git a/tests/unittest/test_tasks_sdk.py b/tests/unittest/test_tasks_sdk.py
new file mode 100644
index 00000000..3439074d
--- /dev/null
+++ b/tests/unittest/test_tasks_sdk.py
@@ -0,0 +1,348 @@
+"""Tasks SDK 的传输、单文件映射、恢复和产物契约;替代旧 Uploads/Jobs 客户端场景。"""
+
+from __future__ import annotations
+
+import asyncio
+import copy
+import hashlib
+import io
+import json
+import zipfile
+from pathlib import Path
+from typing import Any
+
+import httpx
+import pytest
+
+from mineru.parser import ApiJobStatus, ApiTaskStatus, MinerUApiParser
+from mineru.parser import api_client
+from mineru.parser.task_api import CreateTaskRequest
+from mineru.parser.task_outputs import build_task_output
+from mineru.parser.base import ParseResult
+from mineru.types import MiddleJson, ModelJson, PageInfo
+from mineru.integrations.docvortex import build_metadata
+from docvortex.schema import Producer
+
+
+def middle() -> MiddleJson:
+ """生成严格协议数据,不依赖推理模型。"""
+ return MiddleJson(
+ pages=[PageInfo(page_idx=0)],
+ is_full_document=True,
+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version="4.0")},
+ extensions=build_metadata(effort="flash", parse_mode="txt"),
+ )
+
+
+def zip_output() -> bytes:
+ """构造真实 Middle JSON ZIP,供流式下载和损坏包测试使用。"""
+ stream = io.BytesIO()
+ with zipfile.ZipFile(stream, "w") as archive:
+ archive.writestr("middle_json.json", middle().model_dump_json(by_alias=True))
+ return stream.getvalue()
+
+
+def task_snapshot(status: str = "completed", *, upload: bool = False) -> dict[str, Any]:
+ """提供可独立调整状态和上传准备情况的单文件任务快照。"""
+ return {
+ "task_id": "task-one",
+ "status_url": "https://api.test/v1/tasks/task-one",
+ "status": status,
+ "output_formats": ["default"],
+ "include_model_output": False,
+ "files": [
+ {
+ "client_file_id": "file",
+ "status": status,
+ "upload_status": "pending" if upload else "ready",
+ "parse": {"duration_ms": 250},
+ "result_url": "https://storage.test/result?signature=original" if status == "completed" else None,
+ }
+ ],
+ "uploads": (
+ [
+ {
+ "client_file_id": "file",
+ "upload_method": "PUT",
+ "upload_url": "https://storage.test/input",
+ "upload_headers": {},
+ "expires_at": 9999999999,
+ }
+ ]
+ if upload
+ else []
+ ),
+ }
+
+
+def bind_transport(monkeypatch: pytest.MonkeyPatch, handler: Any) -> None:
+ """同时注入同步/异步 HTTP 传输,保留真实 HTTPX 流和响应处理。"""
+ client, async_client = httpx.Client, httpx.AsyncClient
+ transport = httpx.MockTransport(handler)
+ monkeypatch.setattr(api_client.httpx, "Client", lambda **kwargs: client(transport=transport, **kwargs))
+ monkeypatch.setattr(api_client.httpx, "AsyncClient", lambda **kwargs: async_client(transport=transport, **kwargs))
+ monkeypatch.setattr(api_client, "_POLL_INTERVAL_SECONDS", 0)
+ monkeypatch.setattr(api_client, "_transport_retry_delay", lambda _: 0)
+
+
+def parse(parser: MinerUApiParser, source: Path, asynchronous: bool, **kwargs: Any) -> ParseResult:
+ """统一驱动两个 SDK 公开入口,避免只覆盖其中一条传输路径。"""
+ return asyncio.run(parser.parse_async(source, **kwargs)) if asynchronous else parser.parse(source, **kwargs)
+
+
[email protected]("asynchronous", [False, True])
[email protected]("cached", [False, True])
[email protected]("lost_upload_response", [False, True])
+def test_tasks_sdk_upload_cache_and_lost_response(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, asynchronous: bool, cached: bool, lost_upload_response: bool
+) -> None:
+ """同机也提交 hash;命中不 PUT,上传响应丢失后查询 ready 且不重传或重建任务。"""
+ source = tmp_path / "report.pdf"
+ source.write_bytes(b"%PDF-example")
+ calls: list[httpx.Request] = []
+ state = task_snapshot("completed" if cached else "waiting_upload", upload=not cached)
+
+ def handle(request: httpx.Request) -> httpx.Response:
+ """记录线上协议,不接受任何独立上传或 Jobs 路由。"""
+ calls.append(request)
+ if request.method == "POST":
+ assert request.url.path == "/v1/tasks"
+ body = json.loads(request.content)
+ assert body["manifest"][0]["sha256sum"] == hashlib.sha256(source.read_bytes()).hexdigest()
+ assert body["manifest"][0]["page_range"] == "1"
+ assert body["ocr_mode"] == "txt" and body["tier"] == "flash"
+ return httpx.Response(202, json=copy.deepcopy(state))
+ if request.method == "PUT":
+ assert request.content == source.read_bytes()
+ assert "authorization" not in request.headers
+ state.update(task_snapshot())
+ if lost_upload_response:
+ raise httpx.ReadError("response lost")
+ return httpx.Response(200, json=state)
+ if request.url.host == "api.test":
+ return httpx.Response(200, json=state)
+ assert request.url.query == b"signature=original"
+ assert "authorization" not in request.headers
+ return httpx.Response(200, content=zip_output())
+
+ bind_transport(monkeypatch, handle)
+ statuses, durations = [], []
+ result = parse(
+ MinerUApiParser(api_url="https://api.test", api_key="secret", tier="flash", ocr_mode="txt"),
+ source,
+ asynchronous,
+ page_range="1",
+ status_callback=statuses.append,
+ duration_callback=durations.append,
+ )
+ assert len(result.pages) == 1 and durations == [250.0]
+ assert sum(r.method == "POST" for r in calls) == 1
+ assert sum(r.method == "PUT" for r in calls) == (0 if cached else 1)
+ assert statuses == (["completed"] if cached else ["waiting_upload", "completed"])
+
+
[email protected]("asynchronous", [False, True])
+ "failure", ["submission", "query", "upload", "download", "expired_signature", "corrupt_zip", "redirect"]
+)
+def test_tasks_sdk_transport_recovery(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, asynchronous: bool, failure: str
+) -> None:
+ """覆盖创建不重试、查询/传输有界重试、签名刷新、ZIP 校验和跨源重定向认证。"""
+ source = tmp_path / "file.pdf"
+ source.write_bytes(b"pdf")
+ counts = {"POST": 0, "PUT": 0, "query": 0, "download": 0}
+ state = task_snapshot("waiting_upload" if failure == "upload" else "queued", upload=failure == "upload")
+
+ def handle(request: httpx.Request) -> httpx.Response:
+ """在指定阶段只注入一次故障,观测是否重复创建或泄露认证。"""
+ if request.method == "POST":
+ counts["POST"] += 1
+ if failure == "submission":
+ raise httpx.ReadError("unknown submission outcome")
+ return httpx.Response(202, json=copy.deepcopy(state))
+ if request.method == "PUT":
+ counts["PUT"] += 1
+ if counts["PUT"] == 1:
+ raise httpx.WriteError("upload interrupted")
+ state.update(task_snapshot())
+ return httpx.Response(200, json=state)
+ if request.url.host == "api.test":
+ counts["query"] += 1
+ if failure == "query" and counts["query"] == 1:
+ raise httpx.ReadError("query interrupted")
+ if failure != "upload":
+ state.update(task_snapshot())
+ return httpx.Response(200, json=copy.deepcopy(state))
+ counts["download"] += 1
+ assert "authorization" not in request.headers
+ if counts["download"] == 1:
+ if failure == "download":
+ raise httpx.ReadError("download interrupted")
+ if failure == "expired_signature":
+ return httpx.Response(403, json={"error": {"code": "expired", "message": "expired"}})
+ if failure == "corrupt_zip":
+ return httpx.Response(200, content=b"invalid zip")
+ if failure == "redirect":
+ return httpx.Response(302, headers={"location": "https://cdn.test/result"})
+ return httpx.Response(200, content=zip_output())
+
+ bind_transport(monkeypatch, handle)
+ parser = MinerUApiParser(api_url="https://api.test", api_key="secret", tier="flash")
+ if failure == "submission":
+ with pytest.raises(api_client._APITransportError):
+ parse(parser, source, asynchronous)
+ else:
+ assert len(parse(parser, source, asynchronous).pages) == 1
+ assert counts["POST"] == 1
+ if failure == "upload":
+ assert counts["PUT"] == 2
+
+
[email protected]("asynchronous", [False, True])
+ "field,value",
+ [
+ ("task_id", "other"),
+ ("status_url", "https://evil.test/metadata"),
+ ("output_formats", ["html"]),
+ ("include_model_output", True),
+ ],
+)
+def test_tasks_sdk_rejects_changed_identity(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, asynchronous: bool, field: str, value: Any
+) -> None:
+ """拒绝改变身份或输出协议的响应,不尝试读取不匹配产物。"""
+ source = tmp_path / "file.pdf"
+ source.write_bytes(b"pdf")
+
+ def handle(request: httpx.Request) -> httpx.Response:
+ """提交合法,后续查询注入字段变化。"""
+ response = task_snapshot("queued") if request.method == "POST" else {**task_snapshot(), field: value}
+ return httpx.Response(200, json=response)
+
+ bind_transport(monkeypatch, handle)
+ with pytest.raises(api_client._V1APIError, match="invalid_response"):
+ parse(MinerUApiParser(api_url="https://api.test"), source, asynchronous)
+
+
[email protected]("asynchronous", [False, True])
[email protected]("state", ["failed", "canceled"])
+def test_tasks_sdk_preserves_file_errors(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, asynchronous: bool, state: str
+) -> None:
+ """优先保留逐文件错误码与字段,不统一改写为 parse_failed。"""
+ source = tmp_path / "file.pdf"
+ source.write_bytes(b"pdf")
+ snapshot = task_snapshot(state)
+ if state == "failed":
+ snapshot["files"][0]["error"] = {"code": "engine_dead", "message": "engine stopped", "param": "tier"}
+ bind_transport(monkeypatch, lambda _: httpx.Response(202, json=snapshot))
+ with pytest.raises(api_client._V1APIError, match="engine_dead" if state == "failed" else "canceled"):
+ parse(MinerUApiParser(api_url="https://api.test"), source, asynchronous)
+
+
[email protected]("include", [False, True])
[email protected]("has_model", [False, True])
+def test_task_model_output_contract(tmp_path: Path, include: bool, has_model: bool) -> None:
+ """模型输出显式启用且真实存在时才写入,默认包不变。"""
+ model = (
+ ModelJson(
+ pages=[[{"raw": "model"}]],
+ page_index_map=[0],
+ metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version="4.0")},
+ extensions=build_metadata(effort="flash", parse_mode="txt"),
+ )
+ if has_model
+ else None
+ )
+ result = ParseResult(middle_json=middle(), _model_output=model)
+ output = build_task_output(result, ["default"], tmp_path, include_model_output=include)
+ restored = api_client._parse_result_from_zip_bytes(output, include_images=True, include_model_output=include)
+ assert (restored._model_output is not None) == (include and has_model)
+ with zipfile.ZipFile(output) as archive:
+ assert ("model_output.json" in archive.namelist()) == (include and has_model)
+
+
+def test_task_model_output_requires_default() -> None:
+ """不把原本单文件的 HTML/DOCX 请求隐式改成 ZIP。"""
+ with pytest.raises(ValueError, match="requires default"):
+ CreateTaskRequest(
+ manifest=[{"client_file_id": "file", "filename": "a.pdf", "bytes": 1}],
+ output_formats=["html"],
+ include_model_output=True,
+ )
+ assert ApiJobStatus is ApiTaskStatus
+
+
[email protected]("asynchronous", [False, True])
[email protected]("failure", ["query", "download", "length", "corrupt_zip"])
+def test_tasks_sdk_retries_are_bounded(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, asynchronous: bool, failure: str
+) -> None:
+ """持续失败最多三次,不以无限重试掩盖断连或损坏产物。"""
+ source = tmp_path / "file.pdf"
+ source.write_bytes(b"pdf")
+ attempts = []
+
+ def handle(request: httpx.Request) -> httpx.Response:
+ """按阶段产生持续故障,创建请求仍只提交一次。"""
+ if request.method == "POST":
+ return httpx.Response(202, json=task_snapshot("queued" if failure == "query" else "completed"))
+ if request.url.host == "api.test":
+ if failure == "query":
+ attempts.append(request)
+ raise httpx.ReadTimeout("unreachable")
+ return httpx.Response(200, json=task_snapshot())
+ attempts.append(request)
+ if failure == "download":
+ raise httpx.ReadTimeout("unreachable")
+ if failure == "length":
+ return httpx.Response(200, content=zip_output(), headers={"content-length": str(len(zip_output()) + 10)})
+ return httpx.Response(200, content=b"broken zip")
+
+ bind_transport(monkeypatch, handle)
+ with pytest.raises((api_client._APITransportError, api_client._V1APIError)):
+ parse(MinerUApiParser(api_url="https://api.test"), source, asynchronous)
+ assert len(attempts) == 3
+
+
[email protected]("asynchronous", [False, True])
+ "field,value", [("task_id", None), ("status", {}), ("files", []), ("status_url", "file:///tmp/result")]
+)
+def test_tasks_sdk_rejects_malformed_creation(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, asynchronous: bool, field: str, value: Any
+) -> None:
+ """非法成功响应立即失败,不进行上传、轮询或未知地址下载。"""
+ source = tmp_path / "file.pdf"
+ source.write_bytes(b"pdf")
+ calls = []
+
+ def handle(request: httpx.Request) -> httpx.Response:
+ """只允许一次创建请求。"""
+ calls.append(request)
+ return httpx.Response(202, json={**task_snapshot(), field: value})
+
+ bind_transport(monkeypatch, handle)
+ with pytest.raises(api_client._V1APIError, match="invalid_response"):
+ parse(MinerUApiParser(api_url="https://api.test"), source, asynchronous)
+ assert len(calls) == 1
+
+
+def test_task_model_output_changes_wrapper_identity() -> None:
+ """同步包装服务区分模型输出选项,不能复用另一种结果的 request_id。"""
+ from mineru.kit.file_parse.models import ParseOptions, fingerprint
+
+ ordinary = ParseOptions()
+ with_model = ParseOptions(include_model_output=True)
+ assert ordinary.model_dump()["include_model_output"] is False
+ prior_options = ordinary.model_dump(exclude={"include_model_output"})
+ prior_value = {"filename": "a.pdf", "size": 3, "sha256sum": "a" * 64, "options": prior_options}
+ assert (
+ fingerprint("a.pdf", 3, "a" * 64, ordinary)
+ == hashlib.sha256(json.dumps(prior_value, ensure_ascii=False, sort_keys=True).encode()).hexdigest()
+ )
+ assert fingerprint("a.pdf", 3, "a" * 64, ordinary) != fingerprint("a.pdf", 3, "a" * 64, with_model)
+ with pytest.raises(ValueError, match="requires default"):
+ ParseOptions(output_formats=["docx"], include_model_output=True)
diff --git a/tests/unittest/test_v1_router.py b/tests/unittest/test_v1_router.py
index b22f4ea4..b1b8462d 100644
--- a/tests/unittest/test_v1_router.py
+++ b/tests/unittest/test_v1_router.py
@@ -1,8 +1,6 @@
from __future__ import annotations
import asyncio
-import hashlib
-import json
import time
import tomllib
from dataclasses import dataclass, field
@@ -22,254 +20,48 @@ from mineru.kit.router.workers import (
worker_base_url,
worker_connect_host,
)
-from mineru.parser.api_server import create_app as create_api_server_app
@dataclass
class _FakeV1Upstream:
- """提供 Router 回归测试所需的最小内存 V1 API。"""
+ """仅用于 worker 能力和故障隔离测试;完整 Tasks 链路使用真实双服务 fixture。"""
name: str
tiers: tuple[str, ...]
tier_models: dict[str, str] = field(default_factory=dict)
- upload_counter: int = 0
- file_counter: int = 0
- job_counter: int = 0
- uploads: dict[str, dict[str, Any]] = field(default_factory=dict)
- files: dict[str, dict[str, Any]] = field(default_factory=dict)
- jobs: dict[str, dict[str, Any]] = field(default_factory=dict)
- fail_jobs: bool = False
- source_download_attempts: int = 0
- fail_file_delete_once: set[str] = field(default_factory=set)
async def handle(self, request: httpx.Request) -> httpx.Response:
- """按请求 path/method 返回真实 V1 shape 或测试错误。"""
- path = request.url.path
- method = request.method
- if path == "/v1/health":
- return self._response(
- request,
- 200,
- {
- "status": "ok",
- "version": "test",
- "features": {
- "webhook": False,
- "output_formats": ["markdown", "middle_json"],
- "sources": ["file_id", "url", "inline"],
- },
+ """只声明 Tasks 能力,不提供已移除的旧上传和 Jobs 路由。"""
+ if request.url.path == "/v1/health":
+ payload = {
+ "status": "ok",
+ "version": "test",
+ "features": {
+ "webhook": False,
+ "sources": ["upload", "url"],
+ "task_output_formats": ["default", "docx", "html", "latex"],
+ "task_model_output": True,
},
- )
- if path == "/v1/tiers":
- return self._response(
- request,
- 200,
- {
- "object": "list",
- "data": [
- {
- "id": tier,
- "description": f"{self.name}-{tier}",
- "current_model": self.tier_models.get(tier, f"{self.name}-model"),
- }
- for tier in self.tiers
- ],
- },
- )
- if path == "/v1/models":
- return self._response(
- request,
- 200,
- {"object": "list", "data": [{"id": f"{self.name}-model", "object": "model", "created": 1}]},
- )
- if path == "/v1/uploads" and method == "POST":
- self.upload_counter += 1
- upload_id = f"upload_{self.name}_{self.upload_counter}"
- body = await self._json_body(request)
- self.uploads[upload_id] = {**body, "content": b"", "status": "pending"}
- return self._response(request, 200, self._upload_payload(upload_id))
- if path.startswith("/v1/uploads/"):
- return await self._handle_upload(request)
- if path.startswith("/v1/files/"):
- return self._handle_file(request)
- if path == "/v1/parse/jobs" and method == "POST":
- return await self._create_job(request)
- if path.startswith("/v1/parse/jobs/"):
- return self._handle_job(request)
- return self._response(request, 404, self._error("not_found", path))
-
- async def _handle_upload(self, request: httpx.Request) -> httpx.Response:
- """处理 upload 查询、内容写入、完成和取消。"""
- parts = request.url.path.split("/")
- upload_id = parts[3]
- upload = self.uploads.get(upload_id)
- if upload is None:
- return self._response(request, 404, self._error("upload_not_found", upload_id))
- action = parts[4] if len(parts) > 4 else ""
- if request.method == "PUT" and action == "content":
- upload["content"] = await request.aread()
- return httpx.Response(200, request=request)
- if request.method == "POST" and action == "complete":
- upload["status"] = "completed"
- self.file_counter += 1
- file_id = f"file-{self.name}-{self.file_counter}"
- self.files[file_id] = {
- "id": file_id,
- "object": "file",
- "bytes": len(upload["content"]),
- "created_at": self.file_counter,
- "expires_at": None,
- "filename": upload["filename"],
- "purpose": upload.get("purpose", "parse"),
- "sha256sum": upload.get("sha256sum"),
- "content": upload["content"],
- }
- upload["file_id"] = file_id
- return self._response(request, 200, self._upload_payload(upload_id, file_id=file_id))
- if request.method == "POST" and action == "cancel":
- upload["status"] = "canceled"
- return self._response(request, 200, self._upload_payload(upload_id))
- return self._response(request, 200, self._upload_payload(upload_id))
-
- def _handle_file(self, request: httpx.Request) -> httpx.Response:
- """处理 file 元数据、内容读取和删除。"""
- parts = request.url.path.split("/")
- file_id = parts[3]
- file_record = self.files.get(file_id)
- if file_record is None:
- return self._response(request, 404, self._error("file_not_found", file_id))
- if len(parts) > 4 and parts[4] == "content":
- if file_record["purpose"] != "parse_output":
- self.source_download_attempts += 1
- return self._response(
- request,
- 403,
- self._error("feature_requires_api_key", "Source files cannot be downloaded"),
- )
- return httpx.Response(
- 200,
- content=file_record["content"],
- headers={"content-type": "application/octet-stream"},
- request=request,
- )
- if request.method == "DELETE":
- if file_id in self.fail_file_delete_once:
- self.fail_file_delete_once.remove(file_id)
- return self._response(request, 500, self._error("delete_failed", file_id))
- self.files.pop(file_id)
- return self._response(request, 200, {"id": file_id, "object": "file", "deleted": True})
- return self._response(request, 200, {key: value for key, value in file_record.items() if key != "content"})
-
- async def _create_job(self, request: httpx.Request) -> httpx.Response:
- """创建 queued job,并校验请求中的 file_id 已复制到当前 upstream。"""
- if self.fail_jobs:
- raise httpx.ConnectError("simulated disconnect", request=request)
- body = await self._json_body(request)
- for entry in body.get("files") or []:
- source = entry.get("source") or {}
- if source.get("type") == "file_id" and source.get("file_id") not in self.files:
- return self._response(request, 400, self._error("file_not_found", str(source.get("file_id"))))
- self.job_counter += 1
- job_id = f"job_{self.name}_{self.job_counter}"
- self.jobs[job_id] = {"body": body, "status": "queued"}
- return self._response(request, 202, self._job_payload(job_id))
-
- def _handle_job(self, request: httpx.Request) -> httpx.Response:
- """查询时完成 job 并生成输出文件,DELETE 时取消 job。"""
- job_id = request.url.path.rsplit("/", 1)[-1]
- job = self.jobs.get(job_id)
- if job is None:
- return self._response(request, 404, self._error("job_not_found", job_id))
- if request.method == "DELETE":
- job["status"] = "canceled"
- return self._response(request, 200, {"job_id": job_id, "status": "canceled", "canceled_at": "2026-01-01T00:00:00Z"})
- if job["status"] == "queued":
- job["status"] = "completed"
- self.file_counter += 1
- output_id = f"file-{self.name}-{self.file_counter}"
- self.files[output_id] = {
- "id": output_id,
- "object": "file",
- "bytes": 6,
- "created_at": self.file_counter,
- "expires_at": None,
- "filename": "result.md",
- "purpose": "parse_output",
- "sha256sum": None,
- "content": b"result",
}
- job["output_id"] = output_id
- return self._response(request, 200, self._job_payload(job_id))
-
- def _job_payload(self, job_id: str) -> dict[str, Any]:
- """构造包含输入与可选输出 file ID 的 V1 job response。"""
- job = self.jobs[job_id]
- files: list[dict[str, Any]] = []
- for entry in job["body"].get("files") or []:
- source = entry.get("source") or {}
- file_result: dict[str, Any] = {
- "file_id": source.get("file_id"),
- "name": "demo.pdf",
- "page_range": entry.get("page_range") or "",
- "status": "completed" if job["status"] == "completed" else "queued",
+ elif request.url.path == "/v1/tiers":
+ payload = {
+ "object": "list",
+ "data": [
+ {
+ "id": tier,
+ "description": f"{self.name}-{tier}",
+ "current_model": self.tier_models.get(tier, f"{self.name}-model"),
+ }
+ for tier in self.tiers
+ ],
}
- if job.get("output_id"):
- file_result["output_files"] = {
- "markdown": {"file_id": job["output_id"], "bytes": 6},
- }
- files.append(file_result)
- return {
- "job_id": job_id,
- "status": job["status"],
- "created_at": "2026-01-01T00:00:00Z",
- "started_at": None,
- "finished_at": None,
- "tier": job["body"].get("tier", "standard"),
- "output_formats": job["body"].get("output_formats", ["markdown"]),
- "access_level": "anonymous",
- "progress": {"completed": 1 if job["status"] == "completed" else 0, "failed": 0, "total": len(files)},
- "files": files,
- "links": {"self": f"/v1/parse/jobs/{job_id}", "cancel": f"/v1/parse/jobs/{job_id}"},
- }
-
- def _upload_payload(self, upload_id: str, *, file_id: str | None = None) -> dict[str, Any]:
- """构造 pending/completed upload response。"""
- upload = self.uploads[upload_id]
- file_id = file_id or upload.get("file_id")
- payload: dict[str, Any] = {
- "id": upload_id,
- "object": "upload",
- "bytes": upload["bytes"],
- "created_at": 1,
- "expires_at": int(time.time()) + 3600,
- "filename": upload["filename"],
- "purpose": upload.get("purpose", "parse"),
- "mime_type": upload["mime_type"],
- "sha256sum": upload.get("sha256sum"),
- "status": upload["status"],
- "upload_url": f"http://{self.name}/v1/uploads/{upload_id}/content",
- "upload_method": "PUT",
- "upload_headers": {},
- }
- if file_id is not None:
- payload["file"] = {key: value for key, value in self.files[file_id].items() if key != "content"}
- return payload
-
- @staticmethod
- async def _json_body(request: httpx.Request) -> dict[str, Any]:
- """读取 MockTransport 请求中的 JSON object body。"""
- body = await request.aread()
- return json.loads(body.decode("utf-8")) if body else {}
-
- @staticmethod
- def _error(code: str, message: str) -> dict[str, Any]:
- """构造最小 V1 error body。"""
- return {"error": {"type": "invalid_request_error", "code": code, "message": message}}
-
- @staticmethod
- def _response(request: httpx.Request, status_code: int, payload: dict[str, Any]) -> httpx.Response:
- """创建绑定原请求的 JSON response。"""
- return httpx.Response(status_code, json=payload, request=request)
+ elif request.url.path == "/v1/models":
+ payload = {"object": "list", "data": [{"id": f"{self.name}-model", "object": "model", "created": 1}]}
+ elif request.url.path == "/v1/files":
+ payload = {"object": "list", "data": []}
+ else:
+ return httpx.Response(404, json={"error": {"code": "task_not_found", "message": "missing"}}, request=request)
+ return httpx.Response(200, json=payload, request=request)
def _make_router(
@@ -290,126 +82,6 @@ def _make_router(
return create_app(settings, transport=httpx.MockTransport(_handler)), by_host
-def _upload_file(client: TestClient, *, token: str, content: bytes) -> str:
- """通过 Router 完成一次 V1 upload 并返回公共 file ID。"""
- headers = {"authorization": f"Bearer {token}"}
- sha256sum = hashlib.sha256(content).hexdigest()
- create = client.post(
- "/v1/uploads",
- headers=headers,
- json={
- "filename": f"{token}.pdf",
- "bytes": len(content),
- "mime_type": "application/pdf",
- "purpose": "parse",
- "sha256sum": sha256sum,
- },
- )
- assert create.status_code == 200, create.text
- upload_id = create.json()["id"]
- assert create.json()["status"] == "pending"
- assert create.json()["sha256sum"] == sha256sum
- assert create.json()["upload_url"] == f"/v1/uploads/{upload_id}/content"
- assert client.put(f"/v1/uploads/{upload_id}/content", headers=headers, content=content).status_code == 200
- complete = client.post(f"/v1/uploads/{upload_id}/complete", headers=headers, json={})
- assert complete.status_code == 200, complete.text
- return complete.json()["file"]["id"]
-
-
-def _upload_files_on_distinct_workers(
- client: TestClient,
- router_app: Any,
- *,
- token: str,
-) -> list[str]:
- """在同一 caller scope 下轮换 upload worker,获得跨 worker 文件。"""
- files_by_worker: dict[str, str] = {}
- for index in range(2):
- public_file_id = _upload_file(client, token=token, content=f"pdf-{index}".encode())
- route = router_app.state.registry.get("file", public_file_id)
- files_by_worker.setdefault(route.worker_id, public_file_id)
- assert len(files_by_worker) == 2
- return list(files_by_worker.values())
-
-
-def test_v1_router_aggregates_capabilities_and_routes_cross_worker_job() -> None:
- """验证完整 V1 主链、跨 worker 输入复制和公共资源 ID 重写。"""
- first = _FakeV1Upstream("worker-a", ("basic", "standard"))
- second = _FakeV1Upstream("worker-b", ("standard", "advanced"))
- router_app, _ = _make_router(first, second)
- staged_path: Path | None = None
- headers = {"authorization": "Bearer main-cross-worker"}
-
- with TestClient(router_app) as client:
- health = client.get("/v1/health")
- tiers = client.get("/v1/tiers")
- models = client.get("/v1/models")
- assert health.status_code == 200
- assert client.get("/health").status_code == 404
- assert client.get("/tasks").status_code == 404
- assert client.post("/file_parse").status_code == 404
- assert [item["id"] for item in tiers.json()["data"]] == ["basic", "standard", "advanced"]
- assert {item["id"] for item in models.json()["data"]} == {"worker-a-model", "worker-b-model"}
-
- public_file_ids = _upload_files_on_distinct_workers(
- client,
- router_app,
- token="main-cross-worker",
- )
- stored = router_app.state.source_store.find_file(public_file_ids[0])
- assert stored is not None
- staged_path = stored.path
- assert staged_path.is_file()
-
- created = client.post(
- "/v1/parse/jobs",
- headers=headers,
- json={
- "tier": "standard",
- "files": [{"source": {"type": "file_id", "file_id": file_id}} for file_id in public_file_ids],
- "output_formats": ["markdown"],
- },
- )
- assert created.status_code == 202, created.text
- assert [item["file_id"] for item in created.json()["files"]] == public_file_ids
- job_id = created.json()["job_id"]
- job_route = router_app.state.registry.get("job", job_id)
- copied_inputs = list(job_route.metadata.get("copied_inputs") or [])
- assert len(copied_inputs) == 1
- copied_upstream_id = copied_inputs[0].upstream_file_id
- target = first if job_route.worker_id == "remote-1" else second
- assert copied_upstream_id in target.files
-
- completed = client.get(f"/v1/parse/jobs/{job_id}", headers=headers)
- assert completed.status_code == 200
- assert completed.json()["status"] == "completed"
- assert [item["file_id"] for item in completed.json()["files"]] == public_file_ids
- output_id = completed.json()["files"][0]["output_files"]["markdown"]["file_id"]
- assert output_id.startswith("file-")
- assert client.get(f"/v1/files/{output_id}/content", headers=headers).content == b"result"
- assert job_route.metadata.get("copied_inputs") is None
- assert copied_upstream_id not in target.files
-
- job_list = client.get("/v1/parse/jobs", headers=headers).json()
- file_list = client.get("/v1/files", headers=headers).json()
- usage = client.get("/v1/usage", headers=headers).json()
- assert [item["job_id"] for item in job_list["data"]] == [job_id]
- assert {item["id"] for item in file_list["data"]}.issuperset(set(public_file_ids))
- assert output_id in {item["id"] for item in file_list["data"]}
- assert usage["current"]["jobs_created"] == 1
- assert usage["current"]["files_processed"] == 2
-
- selected_job = first.jobs or second.jobs
- submitted_files = [entry["source"]["file_id"] for entry in next(iter(selected_job.values()))["body"]["files"]]
- selected_files = first.files if first.jobs else second.files
- assert copied_upstream_id in submitted_files
- assert copied_upstream_id not in selected_files
- assert first.source_download_attempts == 0
- assert second.source_download_attempts == 0
- assert staged_path is not None
- assert not staged_path.exists()
-
-
def test_v1_router_preserves_upstream_tier_metadata() -> None:
"""验证 Router 不再从全局 models 列表推断各 tier 的 current_model。"""
upstream = _FakeV1Upstream(
@@ -433,400 +105,6 @@ def test_v1_router_preserves_upstream_tier_metadata() -> None:
]
-def test_real_v1_api_rejects_public_download_of_parse_source(tmp_path: Path) -> None:
- """验证真实 api-server 保持普通 parse 输入不可公开下载的安全边界。"""
- data = b"%PDF-1.7"
- api_app = create_api_server_app(upload_dir=tmp_path, tier="flash")
-
- with TestClient(api_app) as client:
- created = client.post(
- "/v1/uploads",
- json={
- "filename": "input.pdf",
- "bytes": len(data),
- "mime_type": "application/pdf",
- "purpose": "parse",
- },
- )
- upload_id = created.json()["id"]
- uploaded = client.put(
- f"/v1/uploads/{upload_id}/content",
- content=data,
- headers={"content-type": "application/octet-stream"},
- )
- completed = client.post(f"/v1/uploads/{upload_id}/complete", json={})
- file_id = completed.json()["file"]["id"]
- downloaded = client.get(f"/v1/files/{file_id}/content")
-
- assert uploaded.status_code == 200
- assert completed.json()["file"]["purpose"] == "parse"
- assert downloaded.status_code == 403
- assert downloaded.json()["error"]["message"] == "Source files cannot be downloaded"
-
-
-def test_router_discards_staged_upload_with_wrong_sha256() -> None:
- """验证 Router 在 SHA256 不匹配时删除暂存输入且不转发字节。"""
- upstream = _FakeV1Upstream("worker-a", ("standard",))
- router_app, _ = _make_router(upstream)
- expected_sha256 = hashlib.sha256(b"ok").hexdigest()
-
- with TestClient(router_app) as client:
- created = client.post(
- "/v1/uploads",
- json={
- "filename": "input.pdf",
- "bytes": 2,
- "mime_type": "application/pdf",
- "purpose": "parse",
- "sha256sum": expected_sha256,
- },
- )
- upload_id = created.json()["id"]
- uploaded = client.put(
- f"/v1/uploads/{upload_id}/content",
- content=b"no",
- headers={"content-type": "application/octet-stream"},
- )
-
- assert uploaded.status_code == 400
- assert uploaded.json()["error"]["code"] == "upload_sha256_mismatch"
- assert router_app.state.source_store.find_upload(upload_id) is None
- assert next(iter(upstream.uploads.values()))["content"] == b""
-
-
-def test_router_maps_empty_malformed_and_non_object_json_to_400() -> None:
- """验证手工 JSON 入口对客户端 body 错误返回稳定 400 而非 500。"""
- upstream = _FakeV1Upstream("worker-a", ("standard",))
- router_app, _ = _make_router(upstream)
-
- with TestClient(router_app, raise_server_exceptions=False) as client:
- for path in ("/v1/uploads", "/v1/parse/jobs"):
- empty = client.post(path, content=b"", headers={"content-type": "application/json"})
- malformed = client.post(path, content=b"{", headers={"content-type": "application/json"})
- non_object = client.post(path, json=[])
- assert empty.status_code == 400
- assert empty.json()["error"]["code"] == "invalid_json"
- assert malformed.status_code == 400
- assert malformed.json()["error"]["code"] == "invalid_json"
- assert non_object.status_code == 400
- assert non_object.json()["error"]["code"] == "invalid_request"
-
- created = client.post(
- "/v1/uploads",
- json={"filename": "input.pdf", "bytes": 3, "mime_type": "application/pdf", "purpose": "parse"},
- )
- upload_id = created.json()["id"]
- client.put(f"/v1/uploads/{upload_id}/content", content=b"pdf")
- malformed_complete = client.post(
- f"/v1/uploads/{upload_id}/complete",
- content=b"{",
- headers={"content-type": "application/json"},
- )
- completed_with_empty_body = client.post(f"/v1/uploads/{upload_id}/complete", content=b"")
-
- assert malformed_complete.status_code == 400
- assert malformed_complete.json()["error"]["code"] == "invalid_json"
- assert completed_with_empty_body.status_code == 200
-
- assert upstream.upload_counter == 1
- assert upstream.job_counter == 0
-
-
-def test_completed_upload_rejects_content_rewrite_without_touching_bound_source() -> None:
- """验证 completed Upload 的后续 PUT 返回 409 且绑定源文件保持不可变。"""
- upstream = _FakeV1Upstream("worker-a", ("standard",))
- router_app, _ = _make_router(upstream)
- original = b"original"
-
- with TestClient(router_app) as client:
- created = client.post(
- "/v1/uploads",
- json={
- "filename": "input.pdf",
- "bytes": len(original),
- "mime_type": "application/pdf",
- "purpose": "parse",
- },
- )
- upload_id = created.json()["id"]
- client.put(f"/v1/uploads/{upload_id}/content", content=original)
- completed = client.post(f"/v1/uploads/{upload_id}/complete", json={})
- file_id = completed.json()["file"]["id"]
- stored = router_app.state.source_store.find_file(file_id)
- assert stored is not None
-
- rewritten = client.put(f"/v1/uploads/{upload_id}/content", content=b"modified")
-
- assert rewritten.status_code == 409
- assert rewritten.json()["error"]["code"] == "upload_already_completed"
- assert stored.path.read_bytes() == original
-
-
-def test_terminal_job_read_survives_deleted_output_metadata() -> None:
- """验证 output 删除后 hydration 失败不会阻断 completed Job 读取。"""
- upstream = _FakeV1Upstream("worker-a", ("standard",))
- router_app, _ = _make_router(upstream)
- headers = {"authorization": "Bearer output-delete"}
-
- with TestClient(router_app) as client:
- file_id = _upload_file(client, token="output-delete", content=b"pdf")
- created = client.post(
- "/v1/parse/jobs",
- headers=headers,
- json={"tier": "standard", "files": [{"source": {"type": "file_id", "file_id": file_id}}]},
- )
- job_id = created.json()["job_id"]
- completed = client.get(f"/v1/parse/jobs/{job_id}", headers=headers)
- output_id = completed.json()["files"][0]["output_files"]["markdown"]["file_id"]
- assert client.delete(f"/v1/files/{output_id}", headers=headers).status_code == 200
-
- reread = client.get(f"/v1/parse/jobs/{job_id}", headers=headers)
-
- assert reread.status_code == 200
- assert reread.json()["status"] == "completed"
- replacement_output_id = reread.json()["files"][0]["output_files"]["markdown"]["file_id"]
- replacement_route = router_app.state.registry.get("file", replacement_output_id)
- assert replacement_route.metadata["hydration_error"]["status_code"] == 404
-
-
-def test_router_scopes_cached_resources_to_bearer_identity() -> None:
- """验证 files、jobs、usage 与单资源访问按 bearer caller scope 隔离。"""
- upstream = _FakeV1Upstream("worker-a", ("standard",))
- router_app, _ = _make_router(upstream)
- owner_headers = {"authorization": "Bearer tenant-a"}
- other_headers = {"authorization": "Bearer tenant-b"}
-
- with TestClient(router_app) as client:
- file_id = _upload_file(client, token="tenant-a", content=b"pdf")
- created = client.post(
- "/v1/parse/jobs",
- headers=owner_headers,
- json={"tier": "standard", "files": [{"source": {"type": "file_id", "file_id": file_id}}]},
- )
- job_id = created.json()["job_id"]
-
- assert [item["id"] for item in client.get("/v1/files", headers=owner_headers).json()["data"]] == [file_id]
- assert [item["job_id"] for item in client.get("/v1/parse/jobs", headers=owner_headers).json()["data"]] == [job_id]
- assert client.get("/v1/files", headers=other_headers).json()["data"] == []
- assert client.get("/v1/parse/jobs", headers=other_headers).json()["data"] == []
- assert client.get("/v1/files").json()["data"] == []
- assert client.get("/v1/parse/jobs").json()["data"] == []
- assert client.get("/v1/usage", headers=other_headers).json()["current"]["jobs_created"] == 0
- assert client.get(f"/v1/files/{file_id}", headers=other_headers).status_code == 404
- assert client.get(f"/v1/parse/jobs/{job_id}", headers=other_headers).status_code == 404
-
-
-def test_background_reconciler_finalizes_unpolled_cross_worker_job() -> None:
- """验证无客户端轮询时后台 reconciliation 仍归还负载并回收副本。"""
- first = _FakeV1Upstream("worker-a", ("standard",))
- second = _FakeV1Upstream("worker-b", ("standard",))
- router_app, _ = _make_router(first, second)
- headers = {"authorization": "Bearer reconcile"}
-
- with TestClient(router_app) as client:
- file_ids = _upload_files_on_distinct_workers(client, router_app, token="reconcile")
- created = client.post(
- "/v1/parse/jobs",
- headers=headers,
- json={
- "tier": "standard",
- "files": [{"source": {"type": "file_id", "file_id": file_id}} for file_id in file_ids],
- },
- )
- job_id = created.json()["job_id"]
- route = router_app.state.registry.get("job", job_id)
- worker = first if route.worker_id == "remote-1" else second
- copied = list(route.metadata["copied_inputs"])
-
- client.portal.call(router_app.state.reconcile_jobs_once)
-
- assert route.metadata["payload"]["status"] == "completed"
- assert route.metadata.get("active_counted") is None
- assert route.metadata.get("copied_inputs") is None
- assert route.metadata.get("upstream_headers") is None
- assert router_app.state.worker_pool.get(route.worker_id).active_jobs == 0
- assert copied[0].upstream_file_id not in worker.files
- listed_ids = {item["id"] for item in client.get("/v1/files", headers=headers).json()["data"]}
- output_id = route.metadata["payload"]["files"][0]["output_files"]["markdown"]["file_id"]
- assert output_id in listed_ids
-
-
-def test_router_reclaims_cross_worker_copy_when_job_is_canceled() -> None:
- """验证取消 Job 时立即删除并解除 cross-worker 输入副本。"""
- first = _FakeV1Upstream("worker-a", ("standard",))
- second = _FakeV1Upstream("worker-b", ("standard",))
- router_app, _ = _make_router(first, second)
- headers = {"authorization": "Bearer cancel-cross-worker"}
-
- with TestClient(router_app) as client:
- public_file_ids = _upload_files_on_distinct_workers(
- client,
- router_app,
- token="cancel-cross-worker",
- )
- created = client.post(
- "/v1/parse/jobs",
- headers=headers,
- json={
- "tier": "standard",
- "files": [{"source": {"type": "file_id", "file_id": file_id}} for file_id in public_file_ids],
- },
- )
- job_id = created.json()["job_id"]
- job_route = router_app.state.registry.get("job", job_id)
- copied = list(job_route.metadata["copied_inputs"])
- assert len(copied) == 1
- target = first if copied[0].worker_id == "remote-1" else second
- assert copied[0].upstream_file_id in target.files
-
- canceled = client.delete(f"/v1/parse/jobs/{job_id}", headers=headers)
-
- assert canceled.status_code == 200
- assert copied[0].upstream_file_id not in target.files
- assert job_route.metadata.get("copied_inputs") is None
- assert (
- router_app.state.registry.find_upstream(
- "file",
- copied[0].owner_scope,
- copied[0].worker_id,
- copied[0].upstream_file_id,
- )
- is None
- )
-
-
-def test_public_source_delete_retries_terminal_copy_cleanup() -> None:
- """验证终态清理失败后,删除公共源文件会重试目标副本回收。"""
- first = _FakeV1Upstream("worker-a", ("standard",))
- second = _FakeV1Upstream("worker-b", ("standard",))
- router_app, _ = _make_router(first, second)
- headers = {"authorization": "Bearer delete-retry"}
-
- with TestClient(router_app) as client:
- public_file_ids = _upload_files_on_distinct_workers(
- client,
- router_app,
- token="delete-retry",
- )
- created = client.post(
- "/v1/parse/jobs",
- headers=headers,
- json={
- "tier": "standard",
- "files": [{"source": {"type": "file_id", "file_id": file_id}} for file_id in public_file_ids],
- },
- )
- job_id = created.json()["job_id"]
- job_route = router_app.state.registry.get("job", job_id)
- copied = list(job_route.metadata["copied_inputs"])
- target = first if copied[0].worker_id == "remote-1" else second
- target.fail_file_delete_once.add(copied[0].upstream_file_id)
-
- completed = client.get(f"/v1/parse/jobs/{job_id}", headers=headers)
- assert completed.status_code == 200
- assert copied[0].upstream_file_id in target.files
- assert job_route.metadata.get("copied_inputs") == copied
-
- deleted = client.delete(f"/v1/files/{copied[0].source_public_id}", headers=headers)
-
- assert deleted.status_code == 200
- assert copied[0].upstream_file_id not in target.files
- assert job_route.metadata.get("copied_inputs") is None
-
-
-def test_v1_router_reports_capability_transport_and_resource_errors() -> None:
- """验证无匹配 tier、upstream 断连和未知资源使用稳定 V1 错误。"""
- upstream = _FakeV1Upstream("worker-a", ("flash",))
- router_app, _ = _make_router(upstream)
-
- with TestClient(router_app) as client:
- unavailable = client.post(
- "/v1/parse/jobs",
- json={"tier": "standard", "files": [{"source": {"type": "inline", "name": "a.pdf", "data": "AAAA"}}]},
- )
- missing = client.get("/v1/files/file-missing")
- assert unavailable.status_code == 503
- assert unavailable.json()["error"]["code"] == "quality_tier_unavailable"
- assert missing.status_code == 404
- assert missing.json()["error"]["code"] == "file_not_found"
-
- failing = _FakeV1Upstream("worker-b", ("standard",), fail_jobs=True)
- failing_app, _ = _make_router(failing)
- with TestClient(failing_app) as client:
- response = client.post(
- "/v1/parse/jobs",
- json={"tier": "standard", "files": [{"source": {"type": "inline", "name": "a.pdf", "data": "AAAA"}}]},
- )
- assert response.status_code == 502
- assert response.json()["error"]["code"] == "upstream_unavailable"
-
-
-def test_cross_worker_copy_cancels_target_upload_after_put_failure() -> None:
- """验证 target PUT 失败时取消已创建的内部 Upload,不留下 pending 状态。"""
- first = _FakeV1Upstream("worker-a", ("standard",))
- second = _FakeV1Upstream("worker-b", ("standard",))
- router_app, _ = _make_router(first, second)
- headers = {"authorization": "Bearer copy-put-failure"}
-
- with TestClient(router_app) as client:
- file_ids = _upload_files_on_distinct_workers(client, router_app, token="copy-put-failure")
- original_handle = first.handle
-
- async def _fail_copy_put(request: httpx.Request) -> httpx.Response:
- """仅让 cross-worker target PUT 返回 500。"""
- if request.method == "PUT" and request.url.path.startswith("/v1/uploads/"):
- return httpx.Response(500, json={"error": {"message": "copy put failed"}}, request=request)
- return await original_handle(request)
-
- first.handle = _fail_copy_put
- response = client.post(
- "/v1/parse/jobs",
- headers=headers,
- json={
- "tier": "standard",
- "files": [{"source": {"type": "file_id", "file_id": file_id}} for file_id in file_ids],
- },
- )
-
- assert response.status_code == 500
- assert not any(upload["status"] == "pending" for upload in first.uploads.values())
- assert sum(upload["status"] == "canceled" for upload in first.uploads.values()) == 1
-
-
-def test_cross_worker_copy_deletes_file_after_completion_timeout() -> None:
- """验证 complete 已成功但响应超时时,通过 Upload 查询删除孤立 File。"""
- first = _FakeV1Upstream("worker-a", ("standard",))
- second = _FakeV1Upstream("worker-b", ("standard",))
- router_app, _ = _make_router(first, second)
- headers = {"authorization": "Bearer copy-complete-timeout"}
-
- with TestClient(router_app) as client:
- file_ids = _upload_files_on_distinct_workers(client, router_app, token="copy-complete-timeout")
- original_file_ids = set(first.files)
- original_handle = first.handle
-
- async def _timeout_after_complete(request: httpx.Request) -> httpx.Response:
- """先完成 target Upload,再模拟 completion response 丢失。"""
- if request.method == "POST" and request.url.path.endswith("/complete"):
- await original_handle(request)
- raise httpx.ReadTimeout("completion response lost", request=request)
- return await original_handle(request)
-
- first.handle = _timeout_after_complete
- response = client.post(
- "/v1/parse/jobs",
- headers=headers,
- json={
- "tier": "standard",
- "files": [{"source": {"type": "file_id", "file_id": file_id}} for file_id in file_ids],
- },
- )
-
- assert response.status_code == 504
- assert set(first.files) == original_file_ids
- assert not any(upload["status"] == "pending" for upload in first.uploads.values())
-
-
def test_managed_router_worker_builds_new_api_server_command(tmp_path: Path) -> None:
"""验证本地 Router worker 仅启动正式 mineru-kit api-server 参数面。"""
settings = RouterSettings(
diff --git a/tests/unittest/test_vlm_remote.py b/tests/unittest/test_vlm_remote.py
index 0e5d7c39..f8d29b1c 100644
--- a/tests/unittest/test_vlm_remote.py
+++ b/tests/unittest/test_vlm_remote.py
@@ -3,7 +3,6 @@ from __future__ import annotations
import asyncio
import json
import threading
-import time
from collections.abc import Iterator
from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass, field
@@ -330,40 +329,29 @@ def test_api_job_through_http_vlm_to_middle_json(
source = _pdf_input(tmp_path)
app = api_server.create_app(
upload_dir=str(tmp_path / "api"),
- allow_local_source=True,
api_key="inbound-key",
no_flash=True,
vlm_config=_settings(openai_server),
)
with TestClient(app, headers={"Authorization": "Bearer inbound-key"}) as client:
- submitted = client.post(
- "/v1/parse/jobs",
- json={
- "files": [{"source": {"type": "local", "path": str(source)}}],
- "tier": tier,
- "ocr_mode": "ocr",
- "output_formats": ["middle_json"],
- },
- )
- assert submitted.status_code == 202, submitted.text
- job_id = submitted.json()["job_id"]
- deadline = time.monotonic() + 15
- while True:
- job = client.get(f"/v1/parse/jobs/{job_id}").json()
- if job["status"] in {"completed", "failed", "partial"}:
- break
- assert time.monotonic() < deadline, job
- time.sleep(0.01)
+ from task_test_utils import submit_task, wait_task
+
+ task = submit_task(client, source, tier=tier, ocr_mode="ocr")
+ job = wait_task(client, task, timeout=15)
if chat_status != 200:
assert job["status"] == "failed", job
assert job["files"][0]["error"]["code"] == "parse_failed"
return
assert job["status"] == "completed", job
- file_id = job["files"][0]["output_files"]["middle_json"]["file_id"]
- output = client.get(f"/v1/files/{file_id}/content")
+ import io
+ import zipfile
+
+ output = client.get(job["files"][0]["result_url"])
assert output.status_code == 200
- assert output.json()["extensions"]["mineru"]["tier"] == tier
- assert "Remote VLM text" in output.text
+ with zipfile.ZipFile(io.BytesIO(output.content)) as archive:
+ middle_text = archive.read("middle_json.json").decode()
+ assert json.loads(middle_text)["extensions"]["mineru"]["tier"] == tier
+ assert "Remote VLM text" in middle_text
assert all(request[1] == "Bearer test-key" for request in openai_server.requests)