Skip to content

Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.

refactor(api): unify task file states and response contracts

myhloli619765f4d89d
diff --git a/docs/en/usage/http_api.md b/docs/en/usage/http_api.md
index 7f646d9f..c785c4ba 100644
--- a/docs/en/usage/http_api.md
+++ b/docs/en/usage/http_api.md
@@ -12,19 +12,19 @@ OUTPUT_FORMATS=default,docx MINERU_API_URL=http://127.0.0.1:8000 bash scripts/ht
Send application/json to POST `/v1/tasks`. Each `files` entry requires `file_name` and, for uploaded files, `bytes`; `sha256sum`, `page_range`, and `client_file_id` are optional. Omitted or null IDs are assigned the smallest available `fN`, starting at `f1`, in input order after reserving all explicit IDs. IDs remain stable in task responses and upload links. URL entries may omit bytes. Options are tier, ocr_mode, page_range and output_formats.
-The response contains task_id, status_url and pending_uploads descriptors for missing files. Uploads always use PUT; upload_method is not returned. Follow each returned PUT URL and its headers; no completion request is needed. Each ready file queues independently. Waiting for uploads consumes no parsing slot.
+The response contains task_id and an upload object on each file that still needs uploading. files[].upload is null otherwise; its fields are url, headers and expires_at. Uploads always use PUT; upload_method is not returned. Follow each returned PUT URL and its headers; no completion request is needed. Each ready file enters pending independently. waiting-file covers upload, server-side download and validation; only a non-null upload object asks the client to send bytes. Input preparation consumes no parsing slot.
Loopback requests follow the same protocol. The SDK computes SHA-256 before submission; completed, unexpired cached inputs are not uploaded again. Concurrent first uploads may transfer duplicate bytes, but source storage is deduplicated. Default retention is 24 hours with collection every 5 minutes; active task and transfer references protect their bytes.
-Upload links expire one hour after task creation. Queries do not renew them. Uploads accepted before the deadline may finish. After an uncertain upload, query first and skip ready files; retry pending uploads from the beginning while their links remain valid.
+Upload links expire one hour after task creation. Queries do not renew them. Uploads accepted before the deadline may finish. After an uncertain upload, query first. pending/running/completed need no upload; waiting-file with upload=null requires polling only. Retry only when an upload target is available, within the transfer retry budget. A successful PUT is not repeated while waiting for confirmation.
## Status and results
-Poll status_url and match files by client_file_id. Parent states are waiting_upload, queued, running, completed, partial, failed and canceled. Each completed file has one files[].result_url; there is no top-level result_url or combined batch ZIP. Completed files can be downloaded while other files are still pending.
+Poll GET `/v1/tasks/{task_id}` using the configured API base, preserving its path prefix and URL-encoding the task ID. Match files by client_file_id. Parent states are waiting-file, pending, running, completed, partial, failed and canceled. Files use the same states except partial; upload_status has been removed. Each completed file has one files[].result_url; there is no top-level result_url or combined batch ZIP. Completed files can be downloaded while other files are still pending.
Outputs are default (Markdown, Middle JSON, Structured Content and images in a ZIP), docx, html, or latex (TEX and images in a ZIP). Multiple formats share one flat ZIP. DOCX and HTML embed images. ModelJson stays inside parsing and postprocessing; API, SDK, CLI and Gradio results do not include model output. The removed include_model_output option is rejected with HTTP 400, including false.
-Task and file listing endpoints have been removed. DELETE `/v1/tasks/{task_id}` cancels unfinished files. Files metadata, output download and deletion by ID remain available. Per-file parse statistics contain only duration_ms and parser_version; tier remains at task level.
+Task and file listing endpoints have been removed. DELETE `/v1/tasks/{task_id}` cancels unfinished files. Files metadata, output download and deletion by ID remain available. Before completion, files[].parse is null. Successful files report their actual tier, duration_ms and parser_version; there is no response-level tier. Responses no longer echo per-file sha256sum, page_range, file_id or name. Request parameters remain available.
Use returned result URLs directly. Local servers and Router return Files URLs; SaaS may return signed storage URLs. Attach the API key only to same-origin API requests and reevaluate credentials at every redirect. Query the task again when a result signature expires.
diff --git a/docs/next/api/changes.md b/docs/next/api/changes.md
index 057828a8..3e9780ed 100644
--- a/docs/next/api/changes.md
+++ b/docs/next/api/changes.md
@@ -15,9 +15,14 @@
| `include_model_output` 与模型输出文件 | 参数已删除,HTTP 提交返回 400;ModelJson 仅在内部解析使用,不进入 API、SDK、CLI 或 Gradio 结果。 |
| Health `output_formats` | `task_output_formats`;sources 为 upload/url;删除 `task_model_output`。 |
| GET 任务列表 / 文件列表 | 已移除;合法认证下 `/v1/tasks` 返回 405,`/v1/files` 返回 404。 |
-| 响应 `uploads` | `pending_uploads`,无旧字段别名;上传路由路径仍含 `/uploads/`。 |
+| 响应 `uploads/pending_uploads` | 改为各个 `files[].upload`,子字段为 `url/headers/expires_at`,不重复携带文件 ID;上传路由路径仍含 `/uploads/`。 |
+| Tasks `status_url` | 删除,客户端使用配置的 API 基址和编码后的 task_id 构造固定查询路由。 |
+| 响应顶层 `tier` | 改为成功文件的 `files[].parse.tier`,取实际结果元数据;请求 tier 仍保留。 |
+| 响应 `files[].sha256sum/page_range/file_id/name` | 删除;请求 hash 和页范围不变,源引用只在内部管理。 |
+| `upload_status` | 删除;仅保留 files[].status。waiting-file 统一输入准备,pending 表示已就绪后的解析排队。 |
+| 状态 `waiting_upload/queued` | 改为 `waiting-file/pending`,不提供旧值别名;等待文件且 upload=null 时只轮询。 |
| `upload_method` | 字段删除,客户端固定使用 PUT,并原样使用 URL 和请求头。 |
-| `files[].parse.model_used` | 删除;仅保留 `duration_ms/parser_version`,不重复返回任务顶层 `tier`。 |
+| `files[].parse.model_used` | 删除;成功后的 parse 包含 `tier/duration_ms/parser_version`,完成前为 null。 |
`--allow-local-source` 和 `--max-inline-bytes` 同时删除。Files 按 ID 查询、下载与删除、Health、Tiers、Models、Usage 保留;同步 `file_parse` 继续作为独立包装服务运行。
SDK 保留 `MinerUApiParser` 方法和返回类型,状态类型改为 `ApiTaskStatus`,保留 `ApiJobStatus` Python 别名。直接本地 parser 与 HF CPU/ZeroGPU 内嵌解析不经过 HTTP。
@@ -25,4 +30,6 @@ SDK 保留 `MinerUApiParser` 方法和返回类型,状态类型改为 `ApiTask
停收旧请求、等待在途任务完成,再协调升级 worker、Router、SDK、file_parse 包装服务和应用。SaaS Go 服务需部署方按同一契约配套更新;进程内任务和缓存索引不保证跨重启恢复。
Router 通过候选 worker 的 `/v1/usage` 校验认证,不再依赖 Files 列表。
-file_parse 恢复状态 namespace 增加 `tasks-v2-` 前缀,旧记录和目录保留但不加载;新空间内继续支持 request_id 幂等和重启恢复。不提供旧客户端或旧 worker 的协议回退。
+file_parse 恢复状态 namespace 增加 `tasks-v4-` 前缀,旧 `tasks-v3-`、`tasks-v2-` 及无前缀记录目录保留但不加载;新空间内继续支持 request_id 幂等和重启恢复。不提供旧客户端或旧 worker 的协议回退。
+
+Router 源缓存与下游源 Files ID 解耦,仍保留结果 Files 映射和结果租约。file_parse 自身基于 request_id 的 status_url 不属于 Tasks 响应,本次保持不变。
diff --git a/docs/next/api/examples.md b/docs/next/api/examples.md
index 19f0e11b..a4e41758 100644
--- a/docs/next/api/examples.md
+++ b/docs/next/api/examples.md
@@ -14,7 +14,7 @@ TASK_ID=job_... MINERU_API_URL=http://127.0.0.1:8000 bash scripts/http_task_exam
{"files":[{"client_file_id":"report","file_name":"report.pdf","url":"https://example.com/report.pdf"}],"output_formats":["default"]}
```
-对 `/v1/tasks` 执行 POST,保存返回的 `task_id/status_url`。本地文件清单包含 `bytes` 和可选 `sha256sum`;仅对响应中的待上传项使用返回的 PUT 地址。
-完成后下载对应 `files[].result_url`。通过返回的 `status_url` 查询任务,取消使用 `DELETE /v1/tasks/{task_id}`;不提供任务列表。
+对 `/v1/tasks` 执行 POST,保存返回的 `task_id`。本地文件清单包含 `bytes` 和可选 `sha256sum`;只对 `files[].upload` 非空的文件,使用其中的 `url/headers` 发送 PUT。
+完成后下载对应 `files[].result_url`。使用 API 基址和编码后的 task_id 构造 `GET /v1/tasks/{task_id}` 查询任务,取消使用 `DELETE /v1/tasks/{task_id}`;不提供任务列表。
详见 [Tasks 契约](tasks.md)、[SDK](../sdk/api-parser.md) 和 [同步包装服务](file-parse.md)。
diff --git a/docs/next/api/file-parse.md b/docs/next/api/file-parse.md
index 4ba89b8e..12a8cb7d 100644
--- a/docs/next/api/file-parse.md
+++ b/docs/next/api/file-parse.md
@@ -100,7 +100,7 @@ X-Task-ID: job_0123456789abcdef01234567
### StatusResponse:查询与 202 的完整 JSON 对象
-以下七个顶层字段均会出现。状态对象不返回 Tasks 的 `files/pending_uploads/progress`,不暴露上游上传链接或签名下载地址。
+以下七个顶层字段均会出现。状态对象不返回 Tasks 的 `files/progress`,不暴露上游上传链接或签名下载地址。
| 字段 | JSON 类型 | 可为 null | 含义 |
|---|---|---|---|
@@ -116,8 +116,8 @@ X-Task-ID: job_0123456789abcdef01234567
|---|---|
| `prepared` | 本地源文件和恢复记录已持久化,尚未开始提交。 |
| `submitting` | 正在向上游创建任务,尚无确认后的 `task_id`。 |
-| `waiting_upload` | 上游源文件尚未确认就绪。 |
-| `queued` | 已进入上游执行队列。 |
+| `waiting-file` | 上游仍在接收、下载、校验或确认输入;upload=null 不表示已经就绪。 |
+| `pending` | 已进入上游执行队列。 |
| `running` | 上游正在解析或导出。 |
| `completed` | 上游完整产物已生成,可通过包装服务下载。 |
| `failed` | 已知失败,读取 `error`。 |
@@ -287,7 +287,7 @@ Content-Type: application/json
## 等待、超时与断连后查询
-每次调用在确认上游文件就绪后开始计时,等待包含排队、解析与导出。接收文件、上游上传和结果下载不占此预算,独立使用传输空闲超时。缓存命中跳过上传。
+每次调用在确认上游文件就绪后开始计时,等待包含排队、解析与导出。接收文件、上游上传、文件下载与校验准备及结果下载不占此预算,独立使用传输空闲超时。缓存命中跳过上传;只有进入 pending/running/completed 或失败/取消终态后,才清理本地保留的源文件。
- 正常轮询间隔为 2 秒。
- 连接、元数据请求、传输空闲超时分别为 10、30、600 秒。
@@ -328,4 +328,4 @@ python scripts/http_file_parse_example.py report.pdf --wait-timeout 0 \
## 升级与恢复记录
本次协议升级要求 API Server、Router、SDK、Gradio 与包装服务同步更新,先停收请求并排空旧任务。
-恢复记录目录使用 `tasks-v2-` 加既有上游地址与认证摘要,不加载、不迁移也不自动删除旧目录。旧 request_id 不再恢复;新空间内继续支持幂等和进程重启恢复。
+恢复记录目录使用 `tasks-v4-` 加既有上游地址与认证摘要,不加载、不迁移也不自动删除旧 `tasks-v3-`、`tasks-v2-` 或无前缀目录。包装服务自身的 `status_url` 与 Location 头保持不变,上游 Tasks 查询改用固定路径。旧 request_id 不再恢复;新空间内继续支持幂等和进程重启恢复。
diff --git a/docs/next/api/files.md b/docs/next/api/files.md
index 089d6c0c..b917913e 100644
--- a/docs/next/api/files.md
+++ b/docs/next/api/files.md
@@ -43,7 +43,7 @@ Files 的时间字段为 Unix 秒级整数,与 Tasks 的 ISO-8601 时间字符
}
```
-示例中的 ID、大小、时间和 hash 为示意值。源文件元数据使用相同结构,`purpose` 为 `parse`,`file_name` 为输入文件名。
+示例中的 ID、大小、时间和 hash 为示意值。本地源文件元数据使用相同结构,`purpose` 为 `parse`,`file_name` 为输入文件名。Tasks 不公开源 Files ID;Router 仅登记结果的公共 Files 映射。
## DELETE /v1/files/{file_id}:删除文件视图
@@ -91,4 +91,4 @@ JSON 字段使用 `file_name`;HTTP 附件头的标准参数仍是 `filename`
API Server 与 Router 默认保留 86400 秒,每 300 秒扫描。`--retention-seconds` 优先于 `MINERU_API_RETENTION_SECONDS`,0 禁用正常保留期清理。上传链接固定的一小时窗口不随该设置改变。
清理只删除过期视图和无引用字节。排队、执行、上传、下载持有的引用阻止物理删除;结果保留期从父任务完成资源清理后开始。
-Router 映射自身 Files 地址,按调用方隔离缓存与元数据,后台维持 worker 结果租约。SaaS 可以提供对象存储签名地址,不要求实现本地 Files 下载路由。
+Router 映射结果 Files 地址,源缓存由按调用方隔离的私有引用管理,不依赖下游源 Files ID。父任务清理后开始私有源缓存保留期,后台独立回收并维持 worker 结果租约。SaaS 可以提供对象存储签名地址,不要求实现本地 Files 下载路由。
diff --git a/docs/next/api/tasks.md b/docs/next/api/tasks.md
index 34cbbb27..393942e6 100644
--- a/docs/next/api/tasks.md
+++ b/docs/next/api/tasks.md
@@ -1,7 +1,7 @@
# V1 Tasks:JSON 清单、逐文件上传与下载
`/v1/tasks` 是本地 API Server、Router 和 SaaS 的共同异步客户端契约。
-客户端用 JSON 创建任务,按返回链接上传缺失文件,再轮询 `status_url`。
+客户端用 JSON 创建任务,按 `files[].upload` 上传缺失文件,再通过固定路径 `GET /v1/tasks/{task_id}` 轮询。
每个文件就绪后立即执行,完成后在 `files[].result_url` 返回独立下载链接。
客户端不拼接存储路径,也不要求 SaaS 实现本地 Files API。
@@ -12,13 +12,13 @@
| 方法与路径 | 行为 |
|---|---|
| `POST /v1/tasks` | 仅接受 `application/json`,返回 `202` 和稳定 `task_id`。 |
-| `PUT` 返回的 `upload_url` | 每个链接上传一个文件;本地使用 `PUT /v1/tasks/{task_id}/uploads/{client_file_id}`。 |
+| `PUT` 返回的 `files[].upload.url` | 每个链接上传一个文件;本地使用 `PUT /v1/tasks/{task_id}/uploads/{client_file_id}`。 |
| `GET /v1/tasks/{task_id}` | 返回 `200`,包含总状态、逐文件状态、尚有效的上传链接和已完成文件的下载链接。 |
| `DELETE /v1/tasks/{task_id}` | 返回 `200` 和取消后的完整任务对象,保留已经完成文件的下载链接。 |
创建请求不再接受表单、multipart 或文件字节,其他 Content-Type 返回 `415`。
业务参数必须放在 JSON 主体中,POST 查询参数、未知字段、非法值返回 `400 invalid_request`。
-上传文件的 PUT 仍发送原始字节,使用返回的 `upload_headers`;签名 URL 必须原样使用。
+上传文件的 PUT 仍发送原始字节,使用返回的 `files[].upload.headers`;签名 URL 必须原样使用。
Task ID 是父任务标识;内部逐文件执行 ID 不属于公共契约。
| JSON 字段 | 默认值 | 说明 |
@@ -43,7 +43,7 @@ ModelJson 仅用于内部解析与后处理,不属于 HTTP 或 SDK 结果。
| `page_range` | 否 | 逐文件 PDF 页范围,未提供时继承全局值;非 PDF 仍只支持整本解析。 |
服务端先预留全部显式 `client_file_id`,再按缺省项的输入顺序,从 `f1` 开始分配最小未占用的 `fN`。例如显式占用了 `f2`,两项缺省 ID 分别生成 `f1`、`f3`。空字符串、非法 ID 和重复显式 ID 返回 400。
-创建响应的 `files` 顺序与输入一致,`files[]` 和 `pending_uploads[]` 始终返回非空 ID;生成后上传、查询及 Router 转发均保持不变。
+创建响应的 `files` 顺序与输入一致,`files[]` 始终返回非空 ID;上传描述直接位于对应文件的 `upload`,不重复携带 ID。生成后的文件标识在上传、查询及 Router 转发中保持不变。
先从真实文件生成 JSON 请求(本例省略 ID,由服务端补全):
@@ -85,23 +85,21 @@ curl --fail-with-body "$MINERU_API_URL/v1/tasks" \
JSON 响应使用 `Content-Type: application/json`,正文为 UTF-8。下表列出的字段在当前本地 API Server 和 Router 响应中均会出现;可空字段输出 `null`,空集合输出 `[]` 或 `{}`,不以省略字段代替。客户端应忽略未来新增的响应字段。
时间字符串使用 ISO-8601 UTC 格式,例如 `2026-10-23T04:00:00Z`;上传链接的 `expires_at` 单独使用 Unix 秒级整数。
-请求中的 `ocr_mode` 和顶层 `page_range` 不在响应顶层回显;逐文件页范围通过 `files[].page_range` 返回。
+请求中的 `tier/ocr_mode/page_range` 与逐文件 `sha256sum/page_range` 不作响应回显;实际执行档位在成功文件的 `parse.tier` 中提供。
+客户端使用配置的 API 基址(包含代理前缀)追加 `/v1/tasks/{URL 编码后的 task_id}` 查询;响应不包含 `status_url`。
### 顶层字段
| 字段 | JSON 类型 | 可为 null | 含义 |
|---|---|---|---|
| `task_id` | string | 否 | 稳定的父任务 ID;是不透明标识,不代表内部逐文件执行 ID。 |
-| `status` | string | 否 | 父任务状态:`waiting_upload/queued/running/completed/partial/failed/canceled`。 |
-| `status_url` | string | 否 | 查询本任务的完整 URL;使用返回地址,不自行拼接。 |
+| `status` | string | 否 | 父任务状态:`waiting-file/pending/running/completed/partial/failed/canceled`。 |
| `created_at` | string | 否 | 父任务创建时间。 |
| `started_at` | string | 是 | 已知的最早逐文件执行开始时间;尚无执行开始记录时为 `null`。 |
| `finished_at` | string | 是 | 父任务进入终态的时间;非终态时为 `null`,取消成功时也会填写。 |
-| `tier` | string | 是 | 任务级档位,非空时为 `flash/basic/standard/advanced`;文件实际采用的流程还受格式约束。 |
| `output_formats` | array of string | 否 | 本次任务采用的输出格式,按首次出现去重。 |
| `progress` | object | 否 | 文件计数,结构见下表。 |
| `files` | array of object | 否 | 完整逐文件快照,顺序始终与创建请求一致,结构见下表。 |
-| `pending_uploads` | array of object | 否 | 仍有效、尚未确认就绪的上传描述;无需上传或任务取消后为 `[]`。 |
| `error` | object | 是 | 父任务级错误,结构为 `ErrorDetail`;逐文件失败应检查 `files[].error`。 |
### progress:文件计数
@@ -122,37 +120,46 @@ JSON 响应使用 `Content-Type: application/json`,正文为 UTF-8。下表列
| `file_name` | string | 否 | 创建时提交的逻辑文件名,保留扩展名;同名文件通过 ID 区分。 |
| `bytes` | integer | 是 | 上传项的声明大小;URL 项可暂为 `null`,完整下载并校验后补齐实际字节数。 |
| `url` | string | 是 | URL 输入保留原始下载地址;客户端上传项为 `null`。 |
-| `sha256sum` | string | 是 | 创建时声明的小写 SHA-256;URL 完整下载并校验后回填实际值,未声明 hash 的上传项可保持 `null`。 |
-| `page_range` | string | 是 | 归一化后的逐文件页范围,初始继承请求设置,整本初始为 `""`;执行后可回填实际页号范围。 |
-| `upload_status` | string | 否 | `pending/ready/not_required/expired`,与解析状态分别表达。 |
-| `file_id` | string | 是 | 本地/Router 上传项的源 Files ID,准备中可能为 `null`;URL 项为 `null`。这是源文件 ID,产物使用 `result_url` 下载。 |
-| `name` | string | 否 | 执行层回传的文件名;当前本地/Router 与 `file_name` 相同,仍保留此字段。 |
-| `status` | string | 否 | 单文件状态:`waiting_upload/queued/running/completed/failed/canceled`;没有 `partial`。 |
-| `parse` | object | 是 | 解析统计,尚无统计时为 `null`;非空结构见下表。 |
+| `upload` | object | 是 | 当前可执行的客户端上传动作;服务端接收、下载、校验、输入已就绪或文件终结时为 `null`,结构见下表。 |
+| `status` | string | 否 | 单文件状态:`waiting-file/pending/running/completed/failed/canceled`;没有 `partial`。 |
+| `parse` | object | 是 | 解析成功完成后返回实际档位和统计;完成前、失败或取消的文件为 `null`。 |
| `result_url` | string | 是 | 结果完整保存后发布的唯一下载 URL;非 `completed` 文件为 `null`。 |
| `error` | object | 是 | 该文件的错误详情;无错误时为 `null`。 |
-`upload_status` 的具体含义:`pending` 表示上传尚未确认,`ready` 表示源文件已经就绪,`not_required` 表示 URL 输入不需要客户端上传,`expired` 表示尚未完成上传的条目已过上传期限。
-取消尚未上传的文件后,其 `status` 为 `canceled`,`upload_status` 仍可能是 `pending`,但不再提供上传链接。
+逐文件仅使用一套 `status`:
+
+| 状态 | 含义 |
+|---|---|
+| `waiting-file` | 文件准备中,包含等待客户端上传、接收文件、对象存储确认、URL 下载及完整性校验。 |
+| `pending` | 输入已就绪,等待解析资源。 |
+| `running` | 解析、导出及保存结果中。 |
+| `completed` | 结果已保存,可通过 result_url 下载。 |
+| `failed` | 本次处理失败,原因见 error。 |
+| `canceled` | 已取消。 |
+
+仅在 `waiting-file` 且 `upload` 非空时执行上传;`waiting-file` 且 `upload=null` 表示当前由服务端准备文件,客户端只轮询。
+成功 PUT 后即使仍为 `waiting-file` 也不重复传输。只有上传失败、查询确认仍需上传且链接有效时,才进行有界重试。
+缓存命中可以直接进入 `pending`。上传过期使用 `failed` 和 `error.code=upload_expired`,不另设过期状态;取消时 upload 为 null。
### files[].parse:解析统计
-仅包含耗时和解析器版本;档位保留在任务顶层 `tier`,逐文件统计不重复提供。
+成功完成后包含实际执行档位、耗时和解析器版本。档位来自结果中的 MinerU 元数据,不以创建请求的 tier 代替;响应顶层不再有 tier。
+PDF 与图片沿用多档位规则,HTML、Office 等原生文档使用 flash。同一批次可出现不同的逐文件档位。
| 字段 | JSON 类型 | 可为 null | 含义 |
|---|---|---|---|
-| `duration_ms` | integer | 是 | 毫秒级文件执行耗时;当前实现包含源读取/下载、解析和导出,不含等待上传及排队时间。 |
+| `tier` | string | 否 | 实际执行档位:`flash/basic/standard/advanced`。 |
+| `duration_ms` | integer | 是 | 毫秒级文件执行耗时;当前实现包含源读取/下载、校验、解析和导出,不含等待客户端上传、准备名额及解析队列的时间。 |
| `parser_version` | string | 是 | 执行 worker 的 MinerU 版本,不保证与 Router 的版本一致。 |
-### pending_uploads[]:待上传描述
+### files[].upload:待上传描述
-每个条目固定使用 **PUT** 上传原始文件字节;响应不包含 `upload_method`。客户端原样使用 `upload_url` 和 `upload_headers`。
+该对象固定使用 **PUT** 上传原始文件字节;不包含 `client_file_id` 或 `upload_method`。客户端原样使用 `url` 和 `headers`,按外层文件标识关联输入。查询时仍返回当前有效的上传描述,便于断线恢复。
| 字段 | JSON 类型 | 含义 |
|---|---|---|
-| `client_file_id` | string | 对应 `files[]` 中同 ID 的输入。 |
-| `upload_url` | string | 上传一个文件的完整 URL;签名地址必须原样使用。 |
-| `upload_headers` | object | 字符串到字符串的映射;上传时发送这些请求头,不假定总是固定头集合。 |
+| `url` | string | 上传一个文件的完整 URL;签名地址必须原样使用。 |
+| `headers` | object | 字符串到字符串的映射;上传时发送这些请求头,不假定总是固定头集合。 |
| `expires_at` | integer | Unix 秒级截止时间,创建任务后固定,查询不续期。 |
### ErrorDetail:父任务或逐文件错误
@@ -174,7 +181,7 @@ JSON 响应使用 `Content-Type: application/json`,正文为 UTF-8。下表列
### POST /v1/tasks:创建任务
成功返回 **`202 Accepted` + `application/json`**,正文为 `TaskStatusResponse`。`202` 只表示任务已接单,不表示文件已经解析成功。
-初始状态通常为 `waiting_upload`;URL 或缓存命中项可以直接进入 `queued`。`pending_uploads` 只列出当前需要客户端上传的文件。
+初始状态通常为 `waiting-file`;URL 项在下载和校验后进入 `pending`,缓存命中项可直接进入 `pending`。需要客户端上传的文件具有非空 `upload`,其他文件的 `upload` 为 `null`。
请求示例:
@@ -194,12 +201,10 @@ JSON 响应使用 `Content-Type: application/json`,正文为 UTF-8。下表列
```json
{
"task_id": "job_0123456789abcdef01234567",
- "status": "waiting_upload",
- "status_url": "http://127.0.0.1:8000/v1/tasks/job_0123456789abcdef01234567",
+ "status": "waiting-file",
"created_at": "2026-10-23T04:00:00Z",
"started_at": null,
"finished_at": null,
- "tier": "flash",
"output_formats": [
"default"
],
@@ -214,12 +219,14 @@ JSON 响应使用 `Content-Type: application/json`,正文为 UTF-8。下表列
"file_name": "report.html",
"bytes": 64,
"url": null,
- "sha256sum": null,
- "page_range": "",
- "upload_status": "pending",
- "file_id": null,
- "name": "report.html",
- "status": "waiting_upload",
+ "upload": {
+ "url": "http://127.0.0.1:8000/v1/tasks/job_0123456789abcdef01234567/uploads/f1",
+ "headers": {
+ "Content-Type": "application/octet-stream"
+ },
+ "expires_at": 1792731600
+ },
+ "status": "waiting-file",
"parse": null,
"result_url": null,
"error": null
@@ -229,35 +236,19 @@ JSON 响应使用 `Content-Type: application/json`,正文为 UTF-8。下表列
"file_name": "notes.html",
"bytes": 64,
"url": null,
- "sha256sum": null,
- "page_range": "",
- "upload_status": "pending",
- "file_id": null,
- "name": "notes.html",
- "status": "waiting_upload",
+ "upload": {
+ "url": "http://127.0.0.1:8000/v1/tasks/job_0123456789abcdef01234567/uploads/f2",
+ "headers": {
+ "Content-Type": "application/octet-stream"
+ },
+ "expires_at": 1792731600
+ },
+ "status": "waiting-file",
"parse": null,
"result_url": null,
"error": null
}
],
- "pending_uploads": [
- {
- "client_file_id": "f1",
- "upload_url": "http://127.0.0.1:8000/v1/tasks/job_0123456789abcdef01234567/uploads/f1",
- "upload_headers": {
- "Content-Type": "application/octet-stream"
- },
- "expires_at": 1792731600
- },
- {
- "client_file_id": "f2",
- "upload_url": "http://127.0.0.1:8000/v1/tasks/job_0123456789abcdef01234567/uploads/f2",
- "upload_headers": {
- "Content-Type": "application/octet-stream"
- },
- "expires_at": 1792731600
- }
- ],
"error": null
}
```
@@ -265,17 +256,15 @@ JSON 响应使用 `Content-Type: application/json`,正文为 UTF-8。下表列
### GET /v1/tasks/{task_id}:查询任务
成功返回 **`200 OK` + `application/json`**,正文始终为 `TaskStatusResponse`,包括任务仍在运行、等待上传、部分失败、全部失败或已取消时。
-以下快照中,第一个文件已完成,可立即下载;第二个文件仍等待上传,所以父任务为 `waiting_upload`,`finished_at` 仍为 `null`。
+以下快照中,第一个文件已完成,可立即下载;第二个文件仍等待上传,所以父任务为 `waiting-file`,`finished_at` 仍为 `null`。
```json
{
"task_id": "job_0123456789abcdef01234567",
- "status": "waiting_upload",
- "status_url": "http://127.0.0.1:8000/v1/tasks/job_0123456789abcdef01234567",
+ "status": "waiting-file",
"created_at": "2026-10-23T04:00:00Z",
"started_at": "2026-10-23T04:00:01Z",
"finished_at": null,
- "tier": "flash",
"output_formats": [
"default"
],
@@ -290,13 +279,10 @@ JSON 响应使用 `Content-Type: application/json`,正文为 UTF-8。下表列
"file_name": "report.html",
"bytes": 64,
"url": null,
- "sha256sum": null,
- "page_range": "1",
- "upload_status": "ready",
- "file_id": "file-source0123456789abcdef",
- "name": "report.html",
+ "upload": null,
"status": "completed",
"parse": {
+ "tier": "flash",
"duration_ms": 125,
"parser_version": "4.0.12"
},
@@ -308,34 +294,26 @@ JSON 响应使用 `Content-Type: application/json`,正文为 UTF-8。下表列
"file_name": "notes.html",
"bytes": 64,
"url": null,
- "sha256sum": null,
- "page_range": "",
- "upload_status": "pending",
- "file_id": null,
- "name": "notes.html",
- "status": "waiting_upload",
+ "upload": {
+ "url": "http://127.0.0.1:8000/v1/tasks/job_0123456789abcdef01234567/uploads/f2",
+ "headers": {
+ "Content-Type": "application/octet-stream"
+ },
+ "expires_at": 1792731600
+ },
+ "status": "waiting-file",
"parse": null,
"result_url": null,
"error": null
}
],
- "pending_uploads": [
- {
- "client_file_id": "f2",
- "upload_url": "http://127.0.0.1:8000/v1/tasks/job_0123456789abcdef01234567/uploads/f2",
- "upload_headers": {
- "Content-Type": "application/octet-stream"
- },
- "expires_at": 1792731600
- }
- ],
"error": null
}
```
### DELETE /v1/tasks/{task_id}:取消任务
-成功返回 **`200 OK` + `application/json`**,正文为取消后的完整 `TaskStatusResponse`,`status` 为 `canceled`,`pending_uploads` 为 `[]`。
+成功返回 **`200 OK` + `application/json`**,正文为取消后的完整 `TaskStatusResponse`,`status` 为 `canceled`,所有文件的 `upload` 均为 `null`。
这个接口取消未完成文件,不删除 Task 或已完成的结果;返回时后台不可中断计算或资源清理可能仍在收尾。
已经成功或失败的文件保持原状态,成功文件继续提供 `result_url`。对任意已终结的父任务再次取消,返回 `409 task_already_terminal`。
@@ -343,11 +321,9 @@ JSON 响应使用 `Content-Type: application/json`,正文为 UTF-8。下表列
{
"task_id": "job_0123456789abcdef01234567",
"status": "canceled",
- "status_url": "http://127.0.0.1:8000/v1/tasks/job_0123456789abcdef01234567",
"created_at": "2026-10-23T04:00:00Z",
"started_at": "2026-10-23T04:00:01Z",
"finished_at": "2026-10-23T04:00:03Z",
- "tier": "flash",
"output_formats": [
"default"
],
@@ -362,13 +338,10 @@ JSON 响应使用 `Content-Type: application/json`,正文为 UTF-8。下表列
"file_name": "report.html",
"bytes": 64,
"url": null,
- "sha256sum": null,
- "page_range": "1",
- "upload_status": "ready",
- "file_id": "file-source0123456789abcdef",
- "name": "report.html",
+ "upload": null,
"status": "completed",
"parse": {
+ "tier": "flash",
"duration_ms": 125,
"parser_version": "4.0.12"
},
@@ -380,18 +353,13 @@ JSON 响应使用 `Content-Type: application/json`,正文为 UTF-8。下表列
"file_name": "notes.html",
"bytes": 64,
"url": null,
- "sha256sum": null,
- "page_range": "",
- "upload_status": "pending",
- "file_id": null,
- "name": "notes.html",
+ "upload": null,
"status": "canceled",
"parse": null,
"result_url": null,
"error": null
}
],
- "pending_uploads": [],
"error": null
}
```
@@ -399,11 +367,11 @@ JSON 响应使用 `Content-Type: application/json`,正文为 UTF-8。下表列
### PUT /v1/tasks/{task_id}/uploads/{client_file_id}:本地上传响应
本地 API Server 和 Router 的上传成功返回 **`200 OK` + `application/json`**,正文仍为上面定义的完整 `TaskStatusResponse`。
-本次上传文件的 `upload_status` 为 `ready`,通常从 `pending_uploads` 中移除并进入 `queued/running`;其他文件和已完成结果也包含在响应中。
+本次文件完成保存及校验后,`upload` 变为 `null` 并进入 `pending/running`;其他文件和已完成结果也包含在响应中。
有效期内重复 PUT 已确认文件同样返回 `200` 和当前快照;上传链接到期后返回 `410`,不会重新读取或覆盖源字节。
SaaS 返回的上传目标可能是独立对象存储:其成功状态码和响应体由该存储服务决定,不要求返回 Tasks JSON。
-通用客户端应接受目标服务的成功响应,随后查询 `status_url` 确认就绪情况,不把对象存储上传响应直接作为任务状态解析。
+通用客户端应接受目标服务的成功响应,随后通过 `GET /v1/tasks/{task_id}` 确认就绪情况,不把对象存储上传响应直接作为任务状态解析。
## 各接口错误响应
@@ -452,25 +420,25 @@ URL 输入同样先创建 JSON 任务:
}
```
-- URL 项的 `upload_status` 为 `not_required`,不生成上传链接、不接受 PUT;源 `file_id` 为 `null`。创建后即进入执行队列,不等待其他上传。
-- URL 保留原始签名地址,使用逻辑 `file_name` 识别类型;不要求 HEAD 或 `Content-Length`。下载后返回实际 `bytes` 和 `sha256sum`,提供的大小/hash 必须匹配。URL 每次下载,不用 hash 缓存跳过下载。
-- 上传项 hash 命中时为 `ready`,不返回上传链接,立即排队;未命中或无 hash 时返回该文件的链接。上传后仍计算 hash、校验大小并对源字节去重。
+- URL 项保持 `waiting-file` 和 `upload=null`,由服务端下载和校验;确认有效后进入 `pending`。不生成客户端上传链接,不接受 PUT,不等待其他文件上传。
+- URL 保留原始签名地址,使用逻辑 `file_name` 识别类型;不要求 HEAD 或 `Content-Length`。下载后回填实际 `bytes`;请求提供的大小/hash 必须匹配,但响应不回显 hash。URL 每次下载,不用 hash 缓存跳过下载。
+- 上传项 hash 命中时直接进入 `pending`,不返回上传链接;未命中或无 hash 时返回该文件的链接。上传后仍计算 hash、校验大小并对源字节去重。
- 缓存只复用源文件,不复用不匹配的解析结果;每个文件使用本次解析参数独立执行。
-- 每个上传项校验完成即排队,不需要客户端发送完成确认。等待上传不占解析并发,也不预占 worker。
+- 每个上传项校验完成即排队,不需要客户端发送完成确认。等待客户端上传不预占 worker;worker 上 URL 下载与校验使用受控准备阶段,不占解析并发。
### 一小时上传窗口
所有上传链接的 `expires_at` 固定为创建时起 3600 秒,采用 Unix 秒。查询不会延长或换发链接。
到期后新 PUT 返回 `410 upload_expired`,包括对已经确认文件的重复 PUT。
-截止前已受理的上传可以继续传完并校验,该文件仍处于 `waiting_upload`,不会被过期扫描中止。
-尚未受理上传的文件到期后变为 `status=failed`、`upload_status=expired`,逐文件错误码为 `upload_expired`。
+截止前已受理的上传可以继续传完并校验,该文件仍处于 `waiting-file`,不会被过期扫描中止。
+尚未受理上传的文件到期后变为 `status=failed`,逐文件错误码为 `upload_expired`。URL 下载不受客户端上传截止时间影响,沿用来源下载超时。
在途上传中断且已经过期时,同样进入失败终态;补交须创建新任务。
上传截止时间不限制已上传文件的排队、解析或导出时间。
有效期内,一个文件中断或校验失败可以从头重传,其他文件的状态不受影响。
重复 PUT 已确认文件不读取或覆盖源字节,并发请求也只提交一次解析。
-上传响应丢失时先查询:已经 `ready` 则跳过;仍待上传且链接有效时再重试。
+上传响应丢失时先查询:进入 `pending/running/completed` 后跳过上传;仍为 `waiting-file` 且 upload 为 null 时只轮询;有有效上传目标时才有限重试。
不支持单文件分块断点续传。
## 总状态与逐文件状态
@@ -481,21 +449,21 @@ URL 输入同样先创建 JSON 任务:
完整字段、类型和可空约束见 [TaskStatusResponse](#taskstatusresponse),下面说明状态聚合规则。
Task 顶层没有 `result_url`;`files[]` 不暴露底层 `output_files`。
-已完成文件可以在父任务仍为 `running/queued/waiting_upload` 时下载。
+已完成文件可以在父任务仍为 `running/pending/waiting-file` 时下载。
总状态优先级如下:
| 条件 | `status` |
|---|---|
| 存在运行中的文件,包括导出阶段 | `running` |
-| 没有运行文件,存在排队文件 | `queued` |
-| 剩余未完成文件全部等待上传 | `waiting_upload` |
+| 没有运行文件,存在排队文件 | `pending` |
+| 剩余未完成文件全部处于文件准备阶段 | `waiting-file` |
| 所有文件成功 | `completed` |
| 所有文件终结,成功与失败并存 | `partial` |
| 所有文件终结,无成功文件 | `failed` |
| 用户取消任务 | `canceled` |
-因此,第一个文件解析完成而第二个文件尚未上传时,总状态可以由 `running` 回到 `waiting_upload`,第一个文件的下载链接仍然有效。
+因此,第一个文件解析完成而第二个文件尚未上传时,总状态可以由 `running` 回到 `waiting-file`,第一个文件的下载链接仍然有效。
取消只停止未完成的文件,已经成功或失败的文件保持原状态;重复取消终态任务返回 `409`。
## 输出格式与下载
@@ -552,7 +520,7 @@ TASK_ID=job_... MINERU_API_URL=http://127.0.0.1:8000 \
| `MINERU_API_URL` | 必填 | API 或 Router 基地址。 |
| `MINERU_API_KEY` | 空 | 服务认证。 |
| `SEND_HASH` | `1` | `0` 时不计算或提交 hash。 |
-| `TASK_ID` | 空 | 恢复已有任务,跳过 ready 文件。 |
+| `TASK_ID` | 空 | 恢复已有任务,只处理仍有上传动作的文件。 |
| `MINERU_TIER`、`OCR_MODE`、`PAGE_RANGE` | 服务默认 / `auto` / 整本 | 解析参数。 |
| `OUTPUT_FORMATS` | `default` | 逗号分隔,转为 JSON 数组。 |
| `RESULT_DIR` | `results` | 逐文件结果目录。 |
@@ -571,13 +539,14 @@ URL 下载沿用服务的来源策略:默认 HTTPS;HTTP 需启用 `--allow-h
Router 按就绪文件独立选择 worker 并预占负载,同一父 Task 可使用多个 worker。
URL 交给 worker 下载,客户端上传字节按 worker 的缓存预检结果传输。
+Router 的源缓存使用按调用方隔离的私有引用,不等待或转发 worker 的源 Files ID;结果仍通过 Files 映射下载。私有源引用在父任务完成清理后开始保留期,过期独立回收,活动传输继续保护字节。
健康接口的 `features.task_output_formats` 声明 Tasks 格式,`sources` 为 `upload/url`。
Router 创建任务前使用候选 worker 的受保护 `/v1/usage` 校验认证,不调用列表接口,不预占解析名额。
不声明新 Tasks 能力的旧 worker 不参与 Tasks 调度,更新时应先更新 worker,再更新 Router 和客户端。
Router 使用有界、可续租的内部结果租约保护 worker 产物,上传窗口不受结果租约影响;Router 丢失后租约会自行结束。已完成任务按 worker 并行续租,每轮网络等待最多 10 秒,避免失联 worker 阻塞其他 worker 的结果保留。
创建请求仅接受 `files` 和 `file_name`,不兼容 `manifest`、`filename`;新旧字段混传同样返回 400。Tasks 响应及 Files 详情统一使用 `file_name`。
-任务和文件列表接口已移除;使用创建响应中的 `task_id/status_url` 恢复任务,按 `files[].result_url` 下载。合法认证下 `GET /v1/tasks` 返回 405,`GET /v1/files` 返回 404。
+任务和文件列表接口已移除;使用创建响应中的 `task_id` 和固定查询路由恢复任务,按 `files[].result_url` 下载。合法认证下 `GET /v1/tasks` 返回 405,`GET /v1/files` 返回 404。
这是同一 `/v1` 下的破坏性变更:先停收请求并排空在途任务,再协调更新 worker、Router、SDK、file_parse 包装服务及其他客户端;SaaS 由部署方同步对齐。
本次协议不保留旧表单、直传或 `markdown/middle_json/structured_content/zip` Tasks 格式别名。
diff --git a/docs/next/api/webhooks.md b/docs/next/api/webhooks.md
index 54be6aca..fa36178b 100644
--- a/docs/next/api/webhooks.md
+++ b/docs/next/api/webhooks.md
@@ -1,5 +1,5 @@
# 任务通知
-当前 Tasks 契约采用轮询,不接受 `callback`,也不提供 SSE 或 Webhook。客户端使用返回的 `status_url` 查询;批量任务的单个文件完成即可下载,无需等待整个批次终结。
+当前 Tasks 契约采用轮询,不接受 `callback`,也不提供 SSE 或 Webhook。客户端使用配置的 API 基址与 `GET /v1/tasks/{task_id}` 查询;批量任务的单个文件完成即可下载,无需等待整个批次终结。
SDK 提供进程内 `status_callback` 与 `duration_callback`,它们不是服务端 Webhook。未来增加推送能力需要另行定义认证、重试和幂等契约。
diff --git a/docs/next/cli/mineru-kit-api-server.md b/docs/next/cli/mineru-kit-api-server.md
index 14336c8e..bfae0fab 100644
--- a/docs/next/cli/mineru-kit-api-server.md
+++ b/docs/next/cli/mineru-kit-api-server.md
@@ -239,3 +239,9 @@ Torch 小模型首次初始化时限制 CPU 线程,并传播到后续工作线
配置优先,否则依次使用 `MINERU_CPU_NUM_THREADS`、`MINERU_INTRA_OP_NUM_THREADS`、8。
按 CPU 亲和性及 cgroup 配额裁剪,且不调高已有更低值。单独设置 OpenBLAS 不视为 Torch
线程限制。纯 ONNX/Flash 的启动路径保持惰性加载。
+
+### 文件准备与解析排队
+
+Tasks 使用 waiting-file 表示等待上传、服务端下载或校验,准备完成后进入 pending,取得解析名额后进入 running。
+URL 输入由独立准备阶段处理,其并发上限沿用 concurrency;等待解析名额的已准备输入也占准备预算,防止源字节无界缓冲。准备阶段不占解析名额,已上传就绪的文件可以正常执行。
+文件 duration_ms 包含实际下载、校验、解析和导出耗时,不包含准备或解析队列等待时间。
diff --git a/docs/next/cli/mineru-kit-webui.md b/docs/next/cli/mineru-kit-webui.md
index f6b3a20f..fb96ab71 100644
--- a/docs/next/cli/mineru-kit-webui.md
+++ b/docs/next/cli/mineru-kit-webui.md
@@ -103,11 +103,11 @@ Gradio 转换事件保留 `raw_page_range` 字符串接口,仍接受 V1 `page_
Gradio 首先发现 `/v1/health` 和 `/v1/tiers`,然后通过 `MinerUApiParser` 提交 hash 清单到 `/v1/tasks`、按需上传、轮询父任务并下载逐文件默认 ZIP 结果。解析结果保存为严格 Middle JSON 2.0。
-左下角使用与 3.4.5 一致的两列步骤卡片,展示准备请求、检查服务、提交任务、排队、解析中、下载结果、整理输出和完成。排队状态来自本地任务槽或 V1 的 `queued` 响应,圆点每秒循环更新;不显示前方任务数。每个界面服务仍一次执行一个转换,等待中的会话也会立即显示排队动画。
+左下角使用与 3.4.5 一致的两列步骤卡片,展示准备请求、检查服务、提交任务、排队、解析中、下载结果、整理输出和完成。文件准备状态来自 V1 的 `waiting-file` 响应;排队状态来自本地任务槽或 V1 的 `pending` 响应,圆点每秒循环更新;不显示前方任务数。每个界面服务仍一次执行一个转换,等待中的会话也会立即显示排队动画。
收到 V1 的 `running` 后开始解析计时,由浏览器本地刷新;收到任务终态时冻结。API 路径的最终完成文案优先使用响应中已有的 `files[0].parse.duration_ms`,包含服务端解析和结果导出/打包,不包含排队、客户端下载和 WebUI 整理输出,因此即使轮询跳过 `running` 也能显示亚秒耗时。HF 等直接调用解析函数的适配层继续使用实际阶段通知之间的计时;缺少 API 耗时字段时也保留这种计时回退。完成数值按两位小数四舍五入,最低显示 `0.01 秒`。两种路径均不承诺该值等同于日志中的纯 `infer` 耗时。界面动画不增加 API 请求,任务轮询仍为每秒一次。清除或更换文件会停止当前会话的状态流并释放本地等待位置,不向 V1 发送远端取消请求。
-`MinerUApiParser.parse()` 和 `parse_async()` 提供可选的 `status_callback` 参数,接收公开的 `ApiTaskStatus`(`waiting_upload`、`queued`、`running`、`completed`、`partial`、`failed`、`canceled`)。回调在任务提交响应及每次轮询响应后同步调用;回调异常只记录日志。这里的 `completed` 表示服务端任务完成,解析方法仍需下载并构造 `ParseResult` 后才返回。可选的 `duration_callback` 在最终成功响应之后、开始下载之前接收既有文件耗时(浮点毫秒);缺失或非法耗时不触发回调,回调异常不影响解析结果。该值属于本次调用,不存放在共享 parser 实例或 `ParseResult` 中。
+`MinerUApiParser.parse()` 和 `parse_async()` 提供可选的 `status_callback` 参数,接收公开的 `ApiTaskStatus`(`waiting-file`、`pending`、`running`、`completed`、`partial`、`failed`、`canceled`)。回调在任务提交响应及每次轮询响应后同步调用;回调异常只记录日志。这里的 `completed` 表示服务端任务完成,解析方法仍需下载并构造 `ParseResult` 后才返回。可选的 `duration_callback` 在最终成功响应之后、开始下载之前接收既有文件耗时(浮点毫秒);缺失或非法耗时不触发回调,回调异常不影响解析结果。该值属于本次调用,不存放在共享 parser 实例或 `ParseResult` 中。
## 结果与下载
@@ -145,4 +145,4 @@ PDF 和图片会生成与解析范围一致的 `origin.pdf`;布局预览使用
`mineru-webui` 提供为 `mineru-kit webui` 的独立命令名,参数与行为完全相同;旧 HTTP 协议和旧 Gradio 专属参数不再提供。
-Gradio 使用 Tasks 能力发现和 `pending_uploads`,固定使用 PUT 上传;仅保存 Middle JSON 和素材,下载菜单从保存的 Middle JSON 按需导出。
+Gradio 使用 Tasks 能力发现和逐文件 `upload` 描述,固定使用 PUT 上传;仅保存 Middle JSON 和素材,下载菜单从保存的 Middle JSON 按需导出。
diff --git a/docs/next/sdk/api-parser.md b/docs/next/sdk/api-parser.md
index 1f0edff0..7dfff3d0 100644
--- a/docs/next/sdk/api-parser.md
+++ b/docs/next/sdk/api-parser.md
@@ -19,8 +19,8 @@ print(result.markdown())
1. 校验文件及页范围,计算大小和 SHA-256,包括同机 HTTP 请求。
2. 向 `/v1/tasks` 提交一个 `client_file_id="file"` 的 files 数组,固定请求 `output_formats=["default"]`。
-3. 缓存命中不上传;未命中遍历 `pending_uploads`,固定使用 PUT 向返回的链接按 headers 流式上传,并按清单大小设置 `Content-Length`。无需完成确认。
-4. 按返回的同源 `status_url` 每秒查询,轮询预算 3600 次;用 `client_file_id` 关联结果。
+3. 缓存命中不上传;未命中遍历 `files[]` 中非空的 `upload`,固定使用 PUT 向返回的链接按 headers 流式上传,并按清单大小设置 `Content-Length`。无需完成确认。
+4. 以配置的 API 基址追加 `/v1/tasks/{编码后的 task_id}` 每秒查询,保留代理路径前缀,轮询预算 3600 次;用 `client_file_id` 关联结果。
5. 从该文件的 `result_url` 流式下载并校验 ZIP,读取 `middle_json.json`。
6. `include_images=True` 恢复图片素材。`ParseResult` 只持有 MiddleJson,不携带或导出 ModelJson。
@@ -28,11 +28,11 @@ HTTP 输入只有上传和 URL;SDK 的路径参数始终代表客户端文件
## 状态和错误
-`ApiTaskStatus` 包含 `waiting_upload/queued/running/completed/partial/failed/canceled`;`ApiJobStatus` 是保留的 Python 类型别名。
+`ApiTaskStatus` 包含 `waiting-file/pending/running/completed/partial/failed/canceled`;`ApiJobStatus` 是保留的 Python 类型别名。
状态回调在创建和查询后触发,回调异常只记录日志。服务端完成表示开始下载,不等于 SDK 已经返回。
-`duration_callback` 在开始下载前接收服务端单文件 `parse.duration_ms`,包含解析和导出;缺失或非法值不触发回调。
+单文件成功后的 `parse` 包含实际 `tier`、`duration_ms` 和 `parser_version`。`duration_callback` 在开始下载前接收服务端单文件 `parse.duration_ms`,包含解析和导出;缺失或非法值不触发回调。
-创建请求没有幂等保证,不自动重试。上传响应丢失后先查状态:已 ready 则继续,仍 pending 才重传。查询、传输和下载有界重试;任务查询和用量查询对 408/429/500/502/503/504 最多请求 3 次,支持 `Retry-After` 秒数和 HTTP 日期。单次建议等待超过 60 秒时返回原错误,不提前重试;无有效建议时使用指数退避。下载签名失效后重新查询任务。密钥仅附加到 API 同源请求,每次重定向重新判断来源。
+创建请求没有幂等保证,不自动重试。上传响应丢失后先查状态:进入 pending/running/completed 后不重传;waiting-file 且 upload=null 时仅轮询;出现上传目标后才在原重试预算内发送。PUT 成功后等待确认,不因状态仍是 waiting-file 而重复上传。查询、传输和下载有界重试;任务查询和用量查询对 408/429/500/502/503/504 最多请求 3 次,支持 `Retry-After` 秒数和 HTTP 日期。单次建议等待超过 60 秒时返回原错误,不提前重试;无有效建议时使用指数退避。下载签名失效后重新查询任务。密钥仅附加到 API 同源请求,每次重定向重新判断来源。
结构化业务错误保留 code/message/param。传输失败保留 `_APITransportError` 阶段及超时信息,Doclib 继续区分 `remote_timeout`、`remote_unreachable` 与业务 `parse_failed`。
diff --git a/docs/zh/usage/http_api.md b/docs/zh/usage/http_api.md
index d48195ee..b950e98c 100644
--- a/docs/zh/usage/http_api.md
+++ b/docs/zh/usage/http_api.md
@@ -14,14 +14,14 @@ Tasks 与 Files JSON 统一使用 `file_name`;旧字段 `manifest`、`filename
向 `POST /v1/tasks` 提交 `application/json`:`files` 每项包含必填逻辑 `file_name`、上传文件的 `bytes` 和可选 `client_file_id/sha256sum/page_range`。`client_file_id` 省略或为 `null` 时,先预留全部显式 ID,再按输入顺序从 `f1` 开始分配最小未占用的 `fN`;创建后上传和查询均保持不变。也可以提供 URL,此时 bytes 可省略。解析选项为 `tier/ocr_mode/page_range/output_formats`。
-创建返回稳定的 `task_id/status_url` 和 `pending_uploads` 待上传描述。客户端固定使用 PUT 上传缺失文件,使用响应指定的 PUT 地址及请求头,无需 complete。每个文件就绪即独立排队,其他文件等待上传不会阻塞它。
+创建返回稳定的 `task_id`;每个文件的 `upload` 为待上传描述或 `null`,包含 `url/headers/expires_at`。客户端固定使用 PUT 上传缺失文件,使用响应指定的 PUT 地址及请求头,无需 complete。每个文件就绪即独立排队,其他文件等待上传不会阻塞它。
同机也使用相同流程。SDK 总是计算 hash;已完成上传的有效缓存可以免传。并发首次请求允许重复传输,但源字节去重。缓存默认保留 24 小时、每 5 分钟扫描,正在执行及传输的引用受保护。
-上传链接固定有效一小时,查询不会续期。截止前已受理上传可继续完成。中断后先查询,ready 文件跳过,其余在有效期内从头重传;过期项需创建新任务。
+上传链接固定有效一小时,查询不会续期。截止前已受理上传可继续完成。中断后先查询,进入 pending/running/completed 后跳过上传;waiting-file 且 upload=null 时只等待,有有效上传目标时才有限重试。过期文件进入 failed,错误码为 upload_expired。
## 状态与结果
-使用返回的 `status_url` 轮询。父任务状态为 `waiting_upload/queued/running/completed/partial/failed/canceled`;通过 `client_file_id` 对应输入。
+使用配置的 API 基址(保留代理前缀)和编码后的 task_id 构造 `GET /v1/tasks/{task_id}` 轮询。父任务状态为 `waiting-file/pending/running/completed/partial/failed/canceled`;通过 `client_file_id` 对应输入。逐文件使用同一组状态但没有 partial,不再返回 upload_status。waiting-file 包括上传、服务端下载和校验,pending 表示输入已就绪并排队等待解析。
每个成功文件在 `files[].result_url` 提供唯一下载地址,即使父任务尚未完成也可下载。顶层没有 result_url,不生成跨输入文件的总 ZIP。
| output_formats | 内容 |
@@ -33,7 +33,7 @@ Tasks 与 Files JSON 统一使用 `file_name`;旧字段 `manifest`、`filename
| 多种 | 对应文件平铺在一个 ZIP 中 |
ModelJson 仅在解析内部使用,API、SDK、CLI 和 Gradio 均不输出模型文件。`include_model_output` 已删除,提交该字段返回 400。
-`DELETE /v1/tasks/{task_id}` 取消未完成文件。任务和文件列表已移除;Files 的按 ID 元数据、结果下载和删除保留。逐文件 `parse` 仅包含耗时与解析器版本,档位位于任务顶层 `tier`。
+`DELETE /v1/tasks/{task_id}` 取消未完成文件。任务和文件列表已移除;Files 的按 ID 元数据、结果下载和删除保留。逐文件 `parse` 在完成前为 `null`,成功后包含实际 `tier`、耗时和解析器版本;响应顶层不再有 tier。逐文件响应不回显 hash、页范围、源 file_id 或重复的 name;请求参数仍保留。
客户端原样使用结果链接:本地/Router 返回 Files 地址,SaaS 可返回对象存储签名地址。只为 API 同源请求附加 API Key,每次重定向重新判断;签名失效可重新查询结果链接。
diff --git a/mineru/kit/file_parse/client.py b/mineru/kit/file_parse/client.py
index cab33446..2b1abe8d 100644
--- a/mineru/kit/file_parse/client.py
+++ b/mineru/kit/file_parse/client.py
@@ -14,6 +14,7 @@ from urllib.parse import urljoin
import httpx
from pydantic import ValidationError
+from ...task_types import task_status_url
from .models import GatewayError, Record, RemoteTask
from .settings import FileParseSettings, origin
@@ -126,9 +127,13 @@ class TaskClient:
raise ValueError("Task identity changed")
if list(dict.fromkeys(task.output_formats)) != formats:
raise ValueError("Task output formats changed")
- task.status_url = self.resolve(task.status_url, metadata=True)
- for link in task.pending_uploads:
- link.upload_url = self.resolve(link.upload_url)
+ if expected is not None and (
+ task.files[0].file_name != expected.files[0].file_name or task.files[0].bytes != expected.files[0].bytes
+ ):
+ raise ValueError("Task input identity changed")
+ for item in task.files:
+ if item.upload is not None:
+ item.upload.url = self.resolve(item.upload.url)
if task.files[0].result_url:
task.files[0].result_url = self.resolve(task.files[0].result_url)
return task
@@ -189,48 +194,53 @@ class TaskClient:
)
async def query(self, task: RemoteTask) -> RemoteTask:
- """按服务给出的地址查询原任务,失败不会触发创建操作。"""
- return await self._json_request("GET", task.status_url, task.output_formats, expected=task)
+ """保留部署前缀并按稳定任务 ID 查询,失败不会触发创建操作。"""
+ return await self._json_request(
+ "GET", task_status_url(self.settings.api_url, task.task_id), task.output_formats, expected=task
+ )
async def upload(self, task: RemoteTask, path: Path, size: int) -> RemoteTask:
- """上传失败先查询确认,最多重传三次,绝不续期或创建替代任务。"""
- for attempt in range(3):
- item = task.files[0]
- if item.upload_status == "ready" or item.status in {"completed", "failed", "canceled"}:
- return task
- if not task.pending_uploads:
- raise GatewayError("upstream_protocol_error", "Pending file has no upload link")
- link = task.pending_uploads[0]
+ """文件准备期间只轮询,明确出现上传目标后才发送,最多重试三次。"""
+ attempts = 0
+ uploaded = False
+ last_error: GatewayError | None = None
+ while task.files[0].status == "waiting-file":
+ link = task.files[0].upload
+ if link is None or uploaded:
+ await asyncio.sleep(self.settings.poll_interval)
+ task = await self.query(task)
+ continue
+ if attempts >= 3:
+ raise last_error or GatewayError("upload_failed", "Upload retry budget exhausted")
if link.expires_at <= time.time():
raise GatewayError("upload_expired", "The task upload link has expired", 410)
+ attempts += 1
try:
- headers = self.headers(link.upload_url, link.upload_headers)
+ headers = self.headers(link.url, link.headers)
headers["content-length"] = str(size)
async with self.client.stream(
- "PUT",
- link.upload_url,
- headers=headers,
- content=file_chunks(path),
- timeout=self.transfer_timeout,
+ "PUT", link.url, headers=headers, content=file_chunks(path), timeout=self.transfer_timeout
) as response:
body = await self._read_metadata(response)
if not 200 <= response.status_code < 300:
raise self._http_error(response.status_code, body)
- return await self.query(task)
+ uploaded = True
+ task = await self.query(task)
except (httpx.HTTPError, GatewayError) as exc:
- error = (
+ last_error = (
exc
if isinstance(exc, GatewayError)
- else GatewayError("upload_interrupted", "Upstream upload was interrupted", retryable=True)
+ else GatewayError("upload_interrupted", "The task upload was interrupted", retryable=True)
)
task = await self.query(task)
- item = task.files[0]
- if item.upload_status == "ready" or item.status in {"completed", "failed", "canceled"}:
+ if task.files[0].status != "waiting-file":
return task
- if attempt == 2 or not error.retryable:
- raise error from exc
- await asyncio.sleep(self.retry_delay * (attempt + 1))
- raise AssertionError("unreachable")
+ if task.files[0].upload is None:
+ continue
+ if attempts == 3 or not last_error.retryable:
+ raise last_error from exc
+ await asyncio.sleep(self.retry_delay * attempts)
+ return task
async def _download_once(self, url: str, path: Path) -> None:
"""逐跳重新决定认证并流式写盘,在完整接收前不向调用者发送成功响应。"""
diff --git a/mineru/kit/file_parse/models.py b/mineru/kit/file_parse/models.py
index 3aedcbee..8ec8a9e4 100644
--- a/mineru/kit/file_parse/models.py
+++ b/mineru/kit/file_parse/models.py
@@ -9,10 +9,10 @@ from typing import Any, Literal
from pydantic import BaseModel, ConfigDict, Field, model_validator
-from ...task_types import TaskOutputFormat
+from ...task_types import TaskFileState, TaskOutputFormat, TaskStatus, task_status_url
TERMINAL = frozenset({"completed", "failed", "canceled", "submission_unknown"})
-FileStatus = Literal["waiting_upload", "queued", "running", "completed", "failed", "canceled"]
+FileStatus = TaskFileState
class GatewayError(Exception):
@@ -60,41 +60,43 @@ class ParseOptions(BaseModel):
return self
+class UploadLink(BaseModel):
+ """保留文件内嵌上传地址、请求头和固定截止时间。"""
+
+ url: str
+ headers: dict[str, str]
+ expires_at: int
+
+
class RemoteFile(BaseModel):
"""仅消费共同 Tasks 契约,不依赖本地 Files 或 Job 实现。"""
client_file_id: str
+ file_name: str
+ bytes: int
status: FileStatus
- upload_status: Literal["pending", "ready", "not_required", "expired"]
+ upload: UploadLink | None
result_url: str | None = None
error: dict[str, Any] | None = None
-class UploadLink(BaseModel):
- """保留原始上传地址、请求头和固定截止时间。"""
-
- client_file_id: str
- upload_url: str
- upload_headers: dict[str, str]
- expires_at: int
-
-
class RemoteTask(BaseModel):
"""校验单文件任务的身份和产物契约,允许上游添加其他元数据。"""
task_id: str = Field(min_length=1)
- status: Literal["waiting_upload", "queued", "running", "completed", "partial", "failed", "canceled"]
- status_url: str = Field(min_length=1)
+ status: TaskStatus
files: list[RemoteFile] = Field(min_length=1, max_length=1)
- pending_uploads: list[UploadLink] = Field(max_length=1)
output_formats: list[TaskOutputFormat] = Field(min_length=1)
@model_validator(mode="after")
def validate_file(self) -> RemoteTask:
"""确保返回的文件和链接仍对应本次唯一清单项。"""
item = self.files[0]
- if item.client_file_id != "file" or any(link.client_file_id != "file" for link in self.pending_uploads):
+ task_status_url("", self.task_id)
+ if item.client_file_id != "file":
raise ValueError("Unexpected client_file_id")
+ if item.upload is not None and item.status != "waiting-file":
+ raise ValueError("Only waiting-file accepts uploads")
if item.status == "completed" and not item.result_url:
raise ValueError("Completed file has no result_url")
if item.status != "completed" and item.result_url is not None:
diff --git a/mineru/kit/file_parse/service.py b/mineru/kit/file_parse/service.py
index fb63d298..53b564eb 100644
--- a/mineru/kit/file_parse/service.py
+++ b/mineru/kit/file_parse/service.py
@@ -145,15 +145,13 @@ class FileParseService:
if record.remote is not None and record.status in TERMINAL and item.status not in TERMINAL:
return
updated = replace(record, remote=task.model_dump(), status=item.status)
- if item.upload_status == "ready" or item.status in TERMINAL:
+ if item.status != "waiting-file":
updated.error = item.error
if item.status in {"failed", "canceled"} and updated.error is None:
updated.error = {"code": f"task_{item.status}", "message": f"Upstream task {item.status}"}
if updated.status in TERMINAL and updated.terminal_at is None:
updated.terminal_at = time.time()
- await self._save_and_cleanup(
- record, updated, remove_source=item.upload_status == "ready" or item.status in TERMINAL
- )
+ await self._save_and_cleanup(record, updated, remove_source=item.status != "waiting-file")
async def _remember_error(self, record: Record, error: GatewayError) -> None:
"""无任务 ID 的失败是本地终态;已知任务的传输错误保留后续查询能力。"""
@@ -183,7 +181,7 @@ class FileParseService:
await self._apply(record, task)
else:
task = await self.refresh(record)
- if task.files[0].upload_status == "pending" and task.files[0].status not in TERMINAL:
+ if task.files[0].status == "waiting-file":
if record.source_path is None or not Path(record.source_path).is_file():
raise GatewayError("source_unavailable", "The retained source is unavailable")
task = await self.client.upload(task, Path(record.source_path), record.size)
@@ -265,9 +263,9 @@ class FileParseService:
changed.clear()
self.raise_record_error(record)
task = record.task
- if record.status in TERMINAL or (task is not None and task.files[0].upload_status == "ready"):
+ if record.status in TERMINAL or (task is not None and task.files[0].status != "waiting-file"):
break
- # 对象存储 PUT 成功后,网关可能稍后才确认 ready;这段等待仍属于上传准备。
+ # 文件准备包括上传确认与服务端下载;进入 pending 后才启动解析等待预算。
await changed.wait()
if timeout == 0:
return False
diff --git a/mineru/kit/file_parse/settings.py b/mineru/kit/file_parse/settings.py
index 6e7083fe..0903a353 100644
--- a/mineru/kit/file_parse/settings.py
+++ b/mineru/kit/file_parse/settings.py
@@ -56,7 +56,7 @@ class FileParseSettings:
@property
def namespace(self) -> str:
"""按新任务协议、上游和认证隔离记录,保留但不加载旧协议状态。"""
- return "tasks-v2-" + hashlib.sha256(f"{self.api_url}\0{self.api_key or ''}".encode()).hexdigest()
+ return "tasks-v4-" + hashlib.sha256(f"{self.api_url}\0{self.api_key or ''}".encode()).hexdigest()
__all__ = ["FileParseSettings", "origin"]
diff --git a/mineru/kit/gradio/client.py b/mineru/kit/gradio/client.py
index 7343a0c8..79409661 100644
--- a/mineru/kit/gradio/client.py
+++ b/mineru/kit/gradio/client.py
@@ -31,7 +31,7 @@ from .status import (
STATUS_PROCESSING_OUTPUT,
STATUS_QUEUED_ON_SERVER,
STATUS_SUBMITTING_TASK,
- STATUS_UPLOADING_FILE,
+ STATUS_PREPARING_FILE,
ParseStatusUpdate,
)
@@ -215,8 +215,8 @@ class V1ArtifactClient:
def on_job_status(status: ApiTaskStatus) -> None:
"""将真实 V1 状态映射到卡片阶段,服务端完成仅代表开始下载。"""
messages: dict[ApiTaskStatus, str] = {
- "waiting_upload": STATUS_UPLOADING_FILE,
- "queued": STATUS_QUEUED_ON_SERVER,
+ "waiting-file": STATUS_PREPARING_FILE,
+ "pending": STATUS_QUEUED_ON_SERVER,
"running": STATUS_PROCESSING_ON_SERVER,
"completed": STATUS_DOWNLOADING_RESULT,
"partial": STATUS_DOWNLOADING_RESULT,
diff --git a/mineru/kit/gradio/i18n.py b/mineru/kit/gradio/i18n.py
index 04c957e7..cc2a5b22 100644
--- a/mineru/kit/gradio/i18n.py
+++ b/mineru/kit/gradio/i18n.py
@@ -122,7 +122,7 @@ MESSAGES: dict[str, tuple[str, str]] = {
"status_step_failed": ("Failed", "失败"),
"preparing_request": ("Preparing request...", "正在准备请求…"),
"checking_server": ("Checking server status...", "正在检查服务状态…"),
- "uploading_file": ("Uploading file...", "正在上传文件…"),
+ "preparing_file": ("Preparing file...", "正在准备文件…"),
"submitting_task": ("Submitting task...", "正在提交任务…"),
"queued_locally": ("Queued locally", "本地排队中"),
"queued_on_server": ("Queued on server", "服务端排队中"),
diff --git a/mineru/kit/gradio/status.py b/mineru/kit/gradio/status.py
index 9ec1e53c..b82ec1b8 100644
--- a/mineru/kit/gradio/status.py
+++ b/mineru/kit/gradio/status.py
@@ -14,7 +14,7 @@ DEFAULT_STATUS = "Upload a file and start conversion."
STATUS_PREPARING_REQUEST = "Preparing request..."
STATUS_CHECKING_SERVER = "Checking server status..."
STATUS_SUBMITTING_TASK = "Submitting task..."
-STATUS_UPLOADING_FILE = "Uploading file..."
+STATUS_PREPARING_FILE = "Preparing file..."
STATUS_QUEUED_LOCALLY = "Queued locally"
STATUS_QUEUED_ON_SERVER = "Queued on server"
STATUS_PROCESSING_ON_SERVER = "Processing on server..."
@@ -27,7 +27,7 @@ _MESSAGE_STEPS = {
STATUS_PREPARING_REQUEST: 0,
STATUS_CHECKING_SERVER: 1,
STATUS_SUBMITTING_TASK: 2,
- STATUS_UPLOADING_FILE: 2,
+ STATUS_PREPARING_FILE: 2,
STATUS_QUEUED_LOCALLY: 3,
STATUS_QUEUED_ON_SERVER: 3,
STATUS_PROCESSING_ON_SERVER: 4,
@@ -159,7 +159,7 @@ __all__ = [
"STATUS_QUEUED_LOCALLY",
"STATUS_QUEUED_ON_SERVER",
"STATUS_SUBMITTING_TASK",
- "STATUS_UPLOADING_FILE",
+ "STATUS_PREPARING_FILE",
"ParseStatusUpdate",
"StatusPanelState",
"status_html",
diff --git a/mineru/kit/router/app.py b/mineru/kit/router/app.py
index 17cec6e8..88e8bd2a 100644
--- a/mineru/kit/router/app.py
+++ b/mineru/kit/router/app.py
@@ -226,6 +226,7 @@ def create_app(
while True:
await asyncio.sleep(RETENTION_SCAN_INTERVAL_SECONDS)
registry.collect_expired(source_store)
+ source_store.collect_expired()
try:
if resolved_settings.retention_seconds:
diff --git a/mineru/kit/router/resources.py b/mineru/kit/router/resources.py
index cdc0526f..76e9133b 100644
--- a/mineru/kit/router/resources.py
+++ b/mineru/kit/router/resources.py
@@ -59,7 +59,8 @@ class SourceFileStore:
self._temp_dir = tempfile.TemporaryDirectory(prefix="mineru-v1-router-sources-")
self._root = Path(self._temp_dir.name)
self.retention_seconds = retention_seconds
- self._file_expiry: dict[str, int | None] = {}
+ self._file_expiry: dict[str, float | None] = {}
+ self._active_references: set[str] = set()
self._uploads: dict[str, StoredSourceFile] = {}
self._files: dict[str, StoredSourceFile] = {}
self._bound_uploads: set[str] = set()
@@ -75,7 +76,7 @@ class SourceFileStore:
mime_type: str,
max_bytes: int | None = None,
) -> StoredSourceFile:
- """流式写入一个公共 Upload 的输入字节,并计算实际大小与 SHA256。"""
+ """流式写入私有暂存输入,并计算实际大小与 SHA256。"""
if upload_id in self._bound_uploads:
raise ValueError(f"Upload {upload_id} is already bound to a completed file")
path = self._root / "uploads" / upload_id
@@ -111,14 +112,14 @@ class SourceFileStore:
def bind_file(
self, upload_id: str, file_id: str, *, owner_scope: str = "anonymous", expires_at: int | None = None
) -> StoredSourceFile:
- """把已完成 Upload 的暂存输入绑定到 Router 公共 File。"""
+ """把已完成上传的暂存输入绑定到 Router 私有源引用。"""
stored = self._uploads.pop(upload_id)
self.bind_source(file_id, stored, owner_scope=owner_scope, expires_at=expires_at)
self._bound_uploads.add(upload_id)
return stored
def bind_source(self, file_id: str, stored: StoredSourceFile, *, owner_scope: str, expires_at: int | None = None) -> None:
- """为公共文件增加缓存引用,并建立调用方隔离的哈希索引。"""
+ """增加私有源缓存引用,并建立调用方隔离的哈希索引。"""
previous = self._files.get(file_id)
if previous is not None:
previous_scope = self._file_scopes.get(file_id, "anonymous")
@@ -138,6 +139,27 @@ class SourceFileStore:
if previous is not None:
self._release_path(previous.path)
+ def hold_reference(self, reference_id: str) -> None:
+ """准备和执行阶段固定源引用,父任务清理前不开始保留期。"""
+ if reference_id not in self._files:
+ raise KeyError(reference_id)
+ self._active_references.add(reference_id)
+ self._file_expiry[reference_id] = None
+
+ def release_reference(self, reference_id: str, finished: float) -> None:
+ """首次解除任务引用时开始缓存保留期,重复释放不延长期限。"""
+ if reference_id not in self._active_references:
+ return
+ self._active_references.remove(reference_id)
+ self._file_expiry[reference_id] = finished + self.retention_seconds if self.retention_seconds else None
+
+ def collect_expired(self) -> None:
+ """独立回收过期私有引用,传输中的字节仍由 pin 保护。"""
+ now = time.time()
+ for reference_id, deadline in tuple(self._file_expiry.items()):
+ if reference_id not in self._active_references and deadline is not None and deadline <= now:
+ self.delete_file(reference_id)
+
def find_hash(self, owner_scope: str, sha256sum: str) -> StoredSourceFile | None:
"""只查当前调用方仍持有的源字节,不使用 worker 的哈希声明。"""
for file_id in sorted(self._hash_files.get((owner_scope, sha256sum), set())):
@@ -168,7 +190,7 @@ class SourceFileStore:
path.unlink(missing_ok=True)
def find_file(self, file_id: str) -> StoredSourceFile | None:
- """读取公共 File 对应的私有暂存输入,不存在时返回 None。"""
+ """按私有源引用读取缓存输入,不存在时返回 None。"""
return self._files.get(file_id)
def discard_upload(self, upload_id: str) -> None:
@@ -181,7 +203,8 @@ class SourceFileStore:
self._release_path(stored.path)
def delete_file(self, file_id: str) -> None:
- """删除公共 File 绑定的私有暂存输入。"""
+ """删除私有源引用,只有最后一个引用及传输租约释放后才删除字节。"""
+ self._active_references.discard(file_id)
self._file_expiry.pop(file_id, None)
stored = self._files.pop(file_id, None)
if stored is not None:
@@ -200,6 +223,7 @@ class SourceFileStore:
self._bound_uploads.clear()
self._file_scopes.clear()
self._file_expiry.clear()
+ self._active_references.clear()
self._hash_files.clear()
self._pins.clear()
self._temp_dir.cleanup()
diff --git a/mineru/kit/router/tasks.py b/mineru/kit/router/tasks.py
index 2293af9b..6cbe0409 100644
--- a/mineru/kit/router/tasks.py
+++ b/mineru/kit/router/tasks.py
@@ -39,7 +39,7 @@ from .proxy import (
require_worker_generation,
rewrite_file_payload,
)
-from .resources import ResourceRegistry, ResourceRoute, SourceFileStore, StoredSourceFile, utc_now_iso
+from .resources import ResourceRegistry, ResourceRoute, SourceFileStore, utc_now_iso
from .workers import WorkerPool, WorkerState
FinalizeJob = Callable[[ResourceRoute, dict[str, Any], Request | None], Awaitable[None]]
@@ -113,18 +113,14 @@ def _worker_path(worker: WorkerState, url: str) -> str:
def _job_payload(payload: dict[str, Any], upstream_id: str) -> dict[str, Any]:
"""把 worker Tasks 转换为私有执行快照,不包含旧 Jobs 公开协议。"""
- file_keys = ("file_id", "name", "page_range", "status", "parse", "error")
+ file_keys = ("status", "parse", "error")
files = [{key: item.get(key) for key in file_keys} for item in payload["files"]]
- for file in files:
- if file["status"] in {"waiting_upload", "canceled"}:
- file["status"] = "queued"
return {
"job_id": upstream_id,
- "status": "queued" if payload["status"] == "waiting_upload" else payload["status"],
+ "status": payload["status"],
"created_at": payload["created_at"],
"started_at": payload.get("started_at"),
"finished_at": payload.get("finished_at"),
- "tier": payload["tier"],
"progress": payload["progress"],
"files": files,
}
@@ -211,19 +207,15 @@ async def read_task_execution(
if result_route is not None:
result_route.metadata["task_result_held"] = bool(route.metadata.get("task_result_held"))
rewritten = _job_payload(payload, route.public_id)
- aliases = route.metadata.get("input_aliases") or {}
- for item in rewritten["files"]:
- if item.get("file_id") in aliases:
- item["file_id"] = aliases[item["file_id"]]
if rewritten["status"] in TERMINAL_TASK_STATUSES:
release_job_load(route, pool)
route.metadata["payload"] = copy.deepcopy(rewritten)
route.metadata["task_result_file_id"] = result_file_id
error = ErrorDetail.model_validate(payload["error"]) if payload.get("error") else None
input_metadata = {
- index: _JobInputMetadata(item["bytes"], item["sha256sum"])
+ index: _JobInputMetadata(item["bytes"])
for index, item in enumerate(payload["files"])
- if item.get("url") is not None and item["bytes"] is not None and item["sha256sum"] is not None
+ if item.get("url") is not None and item["bytes"] is not None
}
route.metadata["task_input_metadata"] = input_metadata
return TaskExecution(rewritten, result_file_id, error, input_metadata)
@@ -241,7 +233,8 @@ class RouterTaskBackend:
self.registry: ResourceRegistry = request.app.state.registry
self.pool: WorkerPool = request.app.state.worker_pool
self._pins = ExitStack()
- self._owned_files: list[str] = []
+ self._prepared_sources: list[str] = []
+ self._task_sources: list[str] = []
def fork(self) -> RouterTaskBackend:
"""每个输入文件拥有独立暂存引用和 worker 调度上下文。"""
@@ -249,6 +242,10 @@ class RouterTaskBackend:
async def release_result(self, task_id: str, finished: float) -> None:
"""父任务完成后记录公开保留期限,后台续租直到该期限再释放 worker。"""
+ # 私有源缓存独立于 worker 路由,即使 worker 已丢失也必须开始回收倒计时。
+ for reference_id in self._task_sources:
+ self.sources.release_reference(reference_id, finished)
+ self._task_sources.clear()
route = self.registry.find("job", task_id)
if route is None:
return
@@ -301,7 +298,7 @@ class RouterTaskBackend:
"""为已校验文件建立准备引用,同一调用方相同内容复用现有暂存字节。"""
async with self.request.app.state.task_store.storage_lock:
private_id = "task-source-" + secrets.token_hex(12)
- self._owned_files.append(private_id)
+ self._prepared_sources.append(private_id)
stored = self.sources.find_hash(self.owner_scope, upload.sha256sum)
if stored is None:
staging_id = "task-" + secrets.token_hex(12)
@@ -312,20 +309,22 @@ class RouterTaskBackend:
self.sources.discard_upload(staging_id)
else:
self.sources.bind_source(private_id, stored, owner_scope=self.owner_scope)
+ self.sources.hold_reference(private_id)
self._pins.enter_context(self.sources.pin(stored))
return replace(upload, path=stored.path, file_id=None)
def release_uploads(self, *, rollback: bool) -> None:
- """关闭准备引用;成功提交后公开 File 绑定继续持有可复用源字节。"""
- for file_id in self._owned_files:
- self.sources.delete_file(file_id)
- self._owned_files.clear()
+ """回滚未交接的源引用并释放传输租约,成功提交的缓存由父任务持有。"""
+ if rollback:
+ for reference_id in self._prepared_sources:
+ self.sources.delete_file(reference_id)
+ self._prepared_sources.clear()
self._pins.close()
- async def _wait_for_input_files(
+ async def _wait_for_uploads_ready(
self, worker: WorkerState, upstream_id: str, generation: int, inputs: list[TaskFileItem]
) -> dict[str, Any]:
- """等待已上传项生成公共 File;级联 Router 的上传就绪可能早于下游引用交接。"""
+ """只等待上传确认,不要求下游暴露源 File 或完成内部引用交接。"""
while True:
response = await _worker_request(
self.pool,
@@ -337,7 +336,7 @@ class RouterTaskBackend:
)
require_worker_generation(worker, generation)
ready = TaskStatusResponse.model_validate(_checked_json(response)).model_dump()
- if ready["task_id"] != upstream_id or len(ready["files"]) != len(inputs) or ready["status"] == "waiting_upload":
+ if ready["task_id"] != upstream_id or len(ready["files"]) != len(inputs):
_raise_api_error(
502, error_type="api_error", code="invalid_upstream_response", message="Worker inputs are not ready"
)
@@ -345,13 +344,15 @@ class RouterTaskBackend:
_raise_api_error(
502, error_type="api_error", code="invalid_upstream_response", message="Worker input order changed"
)
- pending = any(
- source.url is None
- and item["upload_status"] == "ready"
- and item["file_id"] is None
- and item["status"] in {"queued", "running"}
- for item, source in zip(ready["files"], inputs)
- )
+ pending = False
+ for item, source in zip(ready["files"], inputs):
+ if source.url is not None:
+ if item["upload"] is not None:
+ _raise_api_error(
+ 502, error_type="api_error", code="invalid_upstream_response", message="URL input requests upload"
+ )
+ continue
+ pending = pending or item["status"] == "waiting-file"
if not pending:
return ready
# 只等待合法的中间状态;取消、worker replacement 和终态仍走原有清理边界。
@@ -376,7 +377,6 @@ class RouterTaskBackend:
worker = reservation.worker
upstream_id: str | None = None
route: ResourceRoute | None = None
- registered_inputs: list[str] = []
transferred = False
try:
worker_inputs = [
@@ -415,17 +415,24 @@ class RouterTaskBackend:
active_counted=True,
reservation=reservation,
upstream_headers={"authorization": authorization} if authorization else {},
- payload={"job_id": task_id, "status": "queued", "files": []},
+ payload={"job_id": task_id, "status": "pending", "files": []},
)
- by_client_id = {item.client_file_id: index for index, item in enumerate(inputs)}
- for link in payload["pending_uploads"]:
- index = by_client_id.get(link["client_file_id"])
+ if len(payload["files"]) != len(inputs) or any(
+ item["client_file_id"] != source.client_file_id for item, source in zip(payload["files"], inputs)
+ ):
+ _raise_api_error(
+ 502, error_type="api_error", code="invalid_upstream_response", message="Worker input order changed"
+ )
+ for index, item in enumerate(payload["files"]):
+ link = item["upload"]
+ if link is None:
+ continue
if index not in uploads:
_raise_api_error(
502, error_type="api_error", code="invalid_upstream_response", message="Unexpected worker upload"
)
- path = _worker_path(worker, link["upload_url"])
- expected_path = f"/v1/tasks/{quote(upstream_id, safe='')}/uploads/{quote(link['client_file_id'], safe='')}"
+ path = _worker_path(worker, link["url"])
+ expected_path = f"/v1/tasks/{quote(upstream_id, safe='')}/uploads/{quote(item['client_file_id'], safe='')}"
if path != expected_path:
_raise_api_error(
502, error_type="api_error", code="invalid_upstream_response", message="Invalid worker upload path"
@@ -438,7 +445,7 @@ class RouterTaskBackend:
request=self.request,
content=file_chunks(uploads[index].path),
headers={
- **link["upload_headers"],
+ **link["headers"],
"content-length": str(uploads[index].bytes),
"x-mineru-task-lease-seconds": "120",
},
@@ -446,46 +453,10 @@ class RouterTaskBackend:
if response.status_code >= 400:
_checked_json(response)
require_worker_generation(worker, reservation.generation)
- ready = await self._wait_for_input_files(worker, upstream_id, reservation.generation, inputs)
- aliases: dict[str, str] = {}
- for index, item in enumerate(ready["files"]):
- if inputs[index].url is not None:
- if item["upload_status"] != "not_required":
- _raise_api_error(
- 502, error_type="api_error", code="invalid_upstream_response", message="Unexpected URL upload"
- )
- continue
- upstream_file_id = item.get("file_id")
- if not isinstance(upstream_file_id, str):
- _raise_api_error(
- 502, error_type="api_error", code="invalid_upstream_response", message="Missing worker source File"
- )
- upload = uploads[index]
- rewritten = rewrite_file_payload(
- {
- "id": upstream_file_id,
- "object": "file",
- "file_name": upload.filename,
- "bytes": upload.bytes,
- "sha256sum": upload.sha256sum,
- "purpose": "parse",
- "created_at": int(time.time()),
- "expires_at": None,
- },
- worker,
- self.registry,
- self.owner_scope,
- )
- public_id = rewritten["id"]
- registered_inputs.append(public_id)
- self.sources.bind_source(
- public_id,
- StoredSourceFile(upload.path, upload.bytes, upload.sha256sum, upload.mime_type),
- owner_scope=self.owner_scope,
- )
- aliases[upstream_file_id] = public_id
- route.metadata["input_aliases"] = aliases
+ await self._wait_for_uploads_ready(worker, upstream_id, reservation.generation, inputs)
result = await self.get(task_id)
+ self._task_sources.extend(self._prepared_sources)
+ self._prepared_sources.clear()
transferred = True
return result
finally:
@@ -500,9 +471,6 @@ class RouterTaskBackend:
route.metadata["task_result_retain_until"] = time.time()
else:
reservation.release()
- for public_id in registered_inputs:
- self.registry.remove("file", public_id)
- self.sources.delete_file(public_id)
async def _cancel_remote(self, worker: WorkerState, upstream_id: str) -> None:
"""尽力取消已经创建的 worker 任务,清理失败不覆盖最初的提交错误。"""
diff --git a/mineru/parser/api_client.py b/mineru/parser/api_client.py
index 0c3da23e..e781b389 100644
--- a/mineru/parser/api_client.py
+++ b/mineru/parser/api_client.py
@@ -24,6 +24,7 @@ import time
import tempfile
import zipfile
from collections.abc import Callable, Iterator, AsyncIterator
+from dataclasses import dataclass
from email.utils import parsedate_to_datetime
from pathlib import Path
from typing import Any, Literal, cast
@@ -32,6 +33,7 @@ from urllib.parse import unquote, urlparse, urljoin
import httpx
from docvortex.assets import validate_image_sidecar_path
+from ..task_types import TaskStatus, task_status_url
from ..types import PageInfo, Tier, validate_tier
from .base import DocumentParser, ParseResult
from .page_range import normalize_page_range_input
@@ -55,9 +57,17 @@ _HTML_IMAGE_SOURCE_RE = re.compile(
logger = logging.getLogger("mineru.api_client")
_RequestMethod = Literal["GET", "POST", "PUT"]
-ApiTaskStatus = Literal["waiting_upload", "queued", "running", "completed", "partial", "failed", "canceled"]
+ApiTaskStatus = TaskStatus
ApiJobStatus = ApiTaskStatus
-_JOB_STATUSES = {"waiting_upload", "queued", "running", *_TERMINAL_JOB_STATUSES}
+_JOB_STATUSES = {"waiting-file", "pending", "running", *_TERMINAL_JOB_STATUSES}
+
+
+@dataclass
+class _UploadState:
+ """跨轮询保留上传成功标记与重试次数,等待服务确认时不重复发送文件。"""
+
+ attempts: int = 0
+ accepted: bool = False
def _notify_job_status(job: dict[str, Any], callback: Callable[[ApiJobStatus], None] | None) -> None:
@@ -187,7 +197,6 @@ class MinerUApiParser(DocumentParser):
or not isinstance(task.get("status"), str)
or task.get("status") not in _JOB_STATUSES
or task.get("output_formats") != ["default"]
- or not isinstance(task.get("pending_uploads"), list)
or not isinstance(files, list)
or len(files) != 1
or not isinstance(files[0], dict)
@@ -196,18 +205,25 @@ class MinerUApiParser(DocumentParser):
):
raise _V1APIError("invalid_response", "Inconsistent Tasks response")
item = files[0]
- if item.get("upload_status") not in ("pending", "ready", "expired") or item.get("status") not in (
- "waiting_upload",
- "queued",
+ if item.get("status") not in (
+ "waiting-file",
+ "pending",
"running",
"completed",
"failed",
"canceled",
):
raise _V1APIError("invalid_response", "Invalid task file state")
- if not isinstance(task.get("status_url"), str):
- raise _V1APIError("invalid_response", "Task status URL is missing")
- task["status_url"] = self._resolve(task["status_url"], metadata=True)
+ if "upload" not in item or (item["upload"] is not None and not isinstance(item["upload"], dict)):
+ raise _V1APIError("invalid_response", "Invalid task upload descriptor")
+ if item["upload"] is not None and item["status"] != "waiting-file":
+ raise _V1APIError("invalid_response", "Only waiting-file accepts uploads")
+ if expected is not None and any(item.get(key) != expected["files"][0].get(key) for key in ("file_name", "bytes")):
+ raise _V1APIError("invalid_response", "Task input identity changed")
+ try:
+ task_status_url(self._base_url, task["task_id"])
+ except ValueError as exc:
+ raise _V1APIError("invalid_response", str(exc)) from exc
if files[0].get("status") == "completed" and not isinstance(files[0].get("result_url"), str):
raise _V1APIError("invalid_response", "Completed file is missing result_url")
return task
@@ -234,13 +250,19 @@ class MinerUApiParser(DocumentParser):
)
task = self._snapshot(response)
_notify_job_status(task, status_callback)
- task = self._upload_input(cli, task, file_path, status_callback, size=payload["files"][0]["bytes"])
+ upload_state = _UploadState()
+ task = self._upload_input(
+ cli, task, file_path, status_callback, size=payload["files"][0]["bytes"], upload_state=upload_state
+ )
for _ in range(_POLL_MAX_ATTEMPTS):
if task["status"] in _TERMINAL_JOB_STATUSES:
break
time.sleep(_POLL_INTERVAL_SECONDS)
task = self._query(cli, task)
_notify_job_status(task, status_callback)
+ task = self._upload_input(
+ cli, task, file_path, status_callback, size=payload["files"][0]["bytes"], upload_state=upload_state
+ )
if task["status"] not in _TERMINAL_JOB_STATUSES:
raise _V1APIError("timeout", f"Task {task['task_id']} did not complete within timeout")
_raise_for_terminal_job_error(task)
@@ -253,11 +275,11 @@ class MinerUApiParser(DocumentParser):
return _parse_result_from_zip_bytes(target, include_images=self.include_images)
def _query(self, cli: httpx.Client, task: dict[str, Any]) -> dict[str, Any]:
- """按服务返回的同源状态链接查询,保留元数据请求的有限重试。"""
+ """按配置的 API 基址和任务 ID 查询,保留元数据请求的有限重试。"""
response = _request_with_retry(
cli,
"GET",
- task["status_url"],
+ task_status_url(self._base_url, task["task_id"]),
stage="task polling",
headers=self._headers(),
)
@@ -271,47 +293,46 @@ class MinerUApiParser(DocumentParser):
callback: Callable[[ApiTaskStatus], None] | None,
*,
size: int,
+ upload_state: _UploadState,
) -> dict[str, Any]:
- """首次上传失败先核实是否已确认,再重新打开文件流;缓存命中完全不传字节。"""
- for attempt in range(1, _TRANSPORT_MAX_ATTEMPTS + 1):
- if task["files"][0].get("upload_status") != "pending":
+ """只有明确待上传时才发 PUT,上传成功及等待确认均不消耗重复传输。"""
+ if task["files"][0]["status"] != "waiting-file":
+ upload_state.accepted = True
+ if upload_state.accepted or task["files"][0]["upload"] is None:
+ return task
+ while upload_state.attempts < _TRANSPORT_MAX_ATTEMPTS:
+ link = task["files"][0]["upload"]
+ if link is None:
return task
- links = task["pending_uploads"]
- link = next((item for item in links if isinstance(item, dict) and item.get("client_file_id") == "file"), None)
- if not link:
- raise _V1APIError("invalid_response", "Missing task upload link")
- url = self._resolve(link["upload_url"])
- headers = _same_origin_upload_headers(self._base_url, url, link.get("upload_headers", {}), self._headers())
+ url = self._resolve(link["url"])
+ headers = _same_origin_upload_headers(self._base_url, url, link.get("headers", {}), self._headers())
headers = dict(httpx.Headers(headers))
headers.pop("transfer-encoding", None)
headers["content-length"] = str(size)
+ upload_state.attempts += 1
try:
- response = _request_once(
- cli,
- "PUT",
- url,
- stage="upload content",
- headers=headers,
- content=_file_chunks(path),
- )
+ response = _request_once(cli, "PUT", url, stage="upload content", headers=headers, content=_file_chunks(path))
if response.status_code >= 400:
self._check(response)
except (_APITransportError, _V1APIError) as exc:
task = self._query(cli, task)
_notify_job_status(task, callback)
- if task["files"][0].get("upload_status") != "pending":
+ if task["files"][0]["status"] != "waiting-file":
+ upload_state.accepted = True
+ return task
+ if task["files"][0]["upload"] is None:
return task
if isinstance(exc, _V1APIError) and response.status_code not in {408, 429, 500, 502, 503, 504}:
raise
- if attempt == _TRANSPORT_MAX_ATTEMPTS:
+ if upload_state.attempts == _TRANSPORT_MAX_ATTEMPTS:
raise
- time.sleep(_transport_retry_delay(attempt))
+ time.sleep(_transport_retry_delay(upload_state.attempts))
continue
+ upload_state.accepted = True
task = self._query(cli, task)
_notify_job_status(task, callback)
- if task["files"][0].get("upload_status") != "pending":
- return task
- raise _V1APIError("upload_failed", "Upload was not confirmed")
+ return task
+ raise _V1APIError("upload_failed", "Upload retry budget exhausted")
def _download_task(self, cli: httpx.Client, task: dict[str, Any], target: Path) -> None:
"""流式下载和校验完整包;重定向逐跳选择认证,签名失效后重新查询任务。"""
@@ -389,13 +410,19 @@ class MinerUApiParser(DocumentParser):
)
task = self._snapshot(response)
_notify_job_status(task, status_callback)
- task = await self._async_upload_input(cli, task, file_path, status_callback, size=payload["files"][0]["bytes"])
+ upload_state = _UploadState()
+ task = await self._async_upload_input(
+ cli, task, file_path, status_callback, size=payload["files"][0]["bytes"], upload_state=upload_state
+ )
for _ in range(_POLL_MAX_ATTEMPTS):
if task["status"] in _TERMINAL_JOB_STATUSES:
break
await asyncio.sleep(_POLL_INTERVAL_SECONDS)
task = await self._async_query(cli, task)
_notify_job_status(task, status_callback)
+ task = await self._async_upload_input(
+ cli, task, file_path, status_callback, size=payload["files"][0]["bytes"], upload_state=upload_state
+ )
if task["status"] not in _TERMINAL_JOB_STATUSES:
raise _V1APIError("timeout", f"Task {task['task_id']} did not complete within timeout")
_raise_for_terminal_job_error(task)
@@ -408,11 +435,11 @@ class MinerUApiParser(DocumentParser):
return _parse_result_from_zip_bytes(target, include_images=self.include_images)
async def _async_query(self, cli: httpx.AsyncClient, task: dict[str, Any]) -> dict[str, Any]:
- """按服务返回的同源状态链接查询,保留元数据请求的有限重试。"""
+ """按配置的 API 基址和任务 ID 查询,保留元数据请求的有限重试。"""
response = await _async_request_with_retry(
cli,
"GET",
- task["status_url"],
+ task_status_url(self._base_url, task["task_id"]),
stage="task polling",
headers=self._headers(),
)
@@ -426,47 +453,48 @@ class MinerUApiParser(DocumentParser):
callback: Callable[[ApiTaskStatus], None] | None,
*,
size: int,
+ upload_state: _UploadState,
) -> dict[str, Any]:
- """首次上传失败先核实是否已确认,再重新打开文件流;缓存命中完全不传字节。"""
- for attempt in range(1, _TRANSPORT_MAX_ATTEMPTS + 1):
- if task["files"][0].get("upload_status") != "pending":
+ """只有明确待上传时才发 PUT,上传成功及等待确认均不消耗重复传输。"""
+ if task["files"][0]["status"] != "waiting-file":
+ upload_state.accepted = True
+ if upload_state.accepted or task["files"][0]["upload"] is None:
+ return task
+ while upload_state.attempts < _TRANSPORT_MAX_ATTEMPTS:
+ link = task["files"][0]["upload"]
+ if link is None:
return task
- links = task["pending_uploads"]
- link = next((item for item in links if isinstance(item, dict) and item.get("client_file_id") == "file"), None)
- if not link:
- raise _V1APIError("invalid_response", "Missing task upload link")
- url = self._resolve(link["upload_url"])
- headers = _same_origin_upload_headers(self._base_url, url, link.get("upload_headers", {}), self._headers())
+ url = self._resolve(link["url"])
+ headers = _same_origin_upload_headers(self._base_url, url, link.get("headers", {}), self._headers())
headers = dict(httpx.Headers(headers))
headers.pop("transfer-encoding", None)
headers["content-length"] = str(size)
+ upload_state.attempts += 1
try:
response = await _async_request_once(
- cli,
- "PUT",
- url,
- stage="upload content",
- headers=headers,
- content=_async_file_chunks(path),
+ cli, "PUT", url, stage="upload content", headers=headers, content=_async_file_chunks(path)
)
if response.status_code >= 400:
self._check(response)
except (_APITransportError, _V1APIError) as exc:
task = await self._async_query(cli, task)
_notify_job_status(task, callback)
- if task["files"][0].get("upload_status") != "pending":
+ if task["files"][0]["status"] != "waiting-file":
+ upload_state.accepted = True
+ return task
+ if task["files"][0]["upload"] is None:
return task
if isinstance(exc, _V1APIError) and response.status_code not in {408, 429, 500, 502, 503, 504}:
raise
- if attempt == _TRANSPORT_MAX_ATTEMPTS:
+ if upload_state.attempts == _TRANSPORT_MAX_ATTEMPTS:
raise
- await asyncio.sleep(_transport_retry_delay(attempt))
+ await asyncio.sleep(_transport_retry_delay(upload_state.attempts))
continue
+ upload_state.accepted = True
task = await self._async_query(cli, task)
_notify_job_status(task, callback)
- if task["files"][0].get("upload_status") != "pending":
- return task
- raise _V1APIError("upload_failed", "Upload was not confirmed")
+ return task
+ raise _V1APIError("upload_failed", "Upload retry budget exhausted")
async def _async_download_task(self, cli: httpx.AsyncClient, task: dict[str, Any], target: Path) -> None:
"""流式下载和校验完整包;重定向逐跳选择认证,签名失效后重新查询任务。"""
@@ -939,7 +967,7 @@ def _same_origin_upload_headers(
if not isinstance(upload_headers, dict) or any(
not isinstance(key, str) or not isinstance(value, str) for key, value in upload_headers.items()
):
- raise _V1APIError("invalid_response", "upload_headers must be a string mapping")
+ raise _V1APIError("invalid_response", "upload.headers must be a string mapping")
headers = dict(upload_headers)
if _http_origin(base_url) == _http_origin(upload_url):
headers.update(auth_headers)
diff --git a/mineru/parser/api_server.py b/mineru/parser/api_server.py
index 85024fbe..98edd4eb 100644
--- a/mineru/parser/api_server.py
+++ b/mineru/parser/api_server.py
@@ -42,6 +42,7 @@ from starlette.types import Receive, Scope, Send
from ..config import VlmConfig
from ..config import config as mineru_config
from ..errors import MineruError
+from ..integrations.docvortex import MinerUMetadata
from ..filetypes import (
PARSEABLE_EXTENSIONS,
batch_effective_parse_tier,
@@ -74,7 +75,7 @@ from .tier import (
ensure_tier_runtime_dependencies,
runtime_options_for_tier,
)
-from ..task_types import TASK_OUTPUT_FORMATS, TaskOutputFormat
+from ..task_types import TASK_OUTPUT_FORMATS, TaskFileState, TaskOutputFormat, TaskStatus
_DEFAULT_API_SERVER_TIER: ServerTier = "standard"
_MAX_FILES_PER_JOB_DEFAULT = 100
@@ -89,10 +90,10 @@ class ParseServerStartupError(RuntimeError):
# ── literal type aliases ────────────────────────────────────────────
-JobStatus = Literal["queued", "running", "completed", "partial", "failed", "canceled"]
+JobStatus = TaskStatus
"""Job lifecycle states."""
-FileStatus = Literal["queued", "running", "completed", "failed"]
+FileStatus = TaskFileState
"""Per-file status within a job."""
AccessLevel = Literal["anonymous", "registered"]
@@ -370,6 +371,7 @@ class TaskProgress(BaseModel):
class FileParseInfo(BaseModel):
model_config = _PYDANTIC_CONFIG
+ tier: Tier
duration_ms: int | None = None
parser_version: str | None = None
@@ -804,13 +806,12 @@ class _JobInputMetadata:
"""记录完整下载且校验通过的实际输入信息,不扩展公开 Jobs 协议。"""
bytes: int
- sha256sum: str
@dataclass
class _JobRecord:
id: str
- status: JobStatus = "queued"
+ status: JobStatus = "pending"
created_at: str = ""
started_at: str | None = None
finished_at: str | None = None
@@ -820,6 +821,7 @@ class _JobRecord:
input_blobs: dict[str, str] = field(default_factory=dict)
input_specs: dict[int, _JobInputSpec] = field(default_factory=dict)
input_metadata: dict[int, _JobInputMetadata] = field(default_factory=dict)
+ prepared_sources: dict[int, _PreparedInput] = field(default_factory=dict)
file_store: FileStore | None = None
usage_counted: bool = False
task_result_file_id: str | None = None
@@ -843,6 +845,7 @@ class JobStore:
self._jobs: dict[str, _JobRecord] = {}
self._concurrency = max(1, concurrency)
self._semaphore = asyncio.Semaphore(self._concurrency)
+ self._preparation_semaphore = asyncio.Semaphore(self._concurrency)
self._started_at = JobStore._now()
self._tasks: dict[str, asyncio.Task[None]] = {}
self._completion_events: dict[str, asyncio.Event] = {}
@@ -874,7 +877,13 @@ class JobStore:
self.fail_engine(message)
_raise_api_error(503, error_type="engine_error", code="engine_dead", message=message)
- def start_task(self, rec: _JobRecord, operation: Callable[[], Awaitable[None]]) -> None:
+ def start_task(
+ self,
+ rec: _JobRecord,
+ operation: Callable[[], Awaitable[None]],
+ *,
+ prepare: Callable[[], Awaitable[None]] | None = None,
+ ) -> None:
"""保存真实后台任务及所属应用租约,排队取消也不会开始解析。"""
self.require_ready()
if self._closing:
@@ -882,17 +891,32 @@ class JobStore:
self._completion_events[rec.id] = asyncio.Event()
async def run_owned() -> None:
- """在整个任务期间传播应用租约,并直到清理完成才释放 job 名额。"""
+ """准备输入不占解析名额,等待解析时仍限制准备任务数以控制内存。"""
token = runtime_owner.set(self.runtime_owner)
+ acquired = False
try:
- async with self._semaphore:
- if rec.status == "queued":
- await operation()
+ if prepare is not None:
+ async with self._preparation_semaphore:
+ if rec.status not in {"canceled", "failed"}:
+ await prepare()
+ if rec.status != "pending":
+ return
+ # 取得解析名额后才允许下一份输入准备,避免待解析字节无限堆积。
+ await self._semaphore.acquire()
+ acquired = True
+ else:
+ await self._semaphore.acquire()
+ acquired = True
+ if rec.status == "pending":
+ await operation()
except asyncio.CancelledError:
if rec.status != "failed":
rec.status = "canceled"
raise
finally:
+ rec.prepared_sources.clear()
+ if acquired:
+ self._semaphore.release()
runtime_owner.reset(token)
def finished(task: asyncio.Task[None]) -> None:
@@ -970,7 +994,7 @@ class JobStore:
now = self._now()
rec = _JobRecord(
id=job_id,
- status="queued",
+ status="waiting-file" if any(isinstance(entry.source, _UrlSource) for entry in req.files) else "pending",
created_at=now,
tier=req.tier,
task_output_formats=req.output_formats,
@@ -983,7 +1007,7 @@ class JobStore:
file_id=entry.source.file_id if isinstance(entry.source, _StoredSource) else None,
name=name,
page_range=entry.page_range or "",
- status="queued",
+ status="waiting-file" if isinstance(entry.source, _UrlSource) else "pending",
)
)
rec.progress.total = len(rec.files)
@@ -1153,6 +1177,74 @@ class _ExtractedSource:
media_type: str | None = None
+@dataclass(frozen=True, slots=True)
+class _PreparedInput:
+ """暂存已校验的 URL 输入及准备耗时,等待解析队列不计入文件执行耗时。"""
+
+ source: _ExtractedSource
+ duration_seconds: float
+
+
+def _validate_url_input(rec: _JobRecord, index: int, extracted: _ExtractedSource) -> None:
+ """下载完成后验证大小和 hash,只有有效输入才能进入解析队列。"""
+ data = extracted.data
+ if not data:
+ raise MineruError("invalid_request", "Empty files are not supported")
+ spec = rec.input_specs.get(index)
+ if spec is not None:
+ if spec.bytes is not None and len(data) != spec.bytes:
+ raise MineruError("bytes_mismatch", "Declared size does not match downloaded bytes")
+ if spec.sha256sum is not None and hashlib.sha256(data).hexdigest() != spec.sha256sum:
+ raise MineruError("file_hash_mismatch", "SHA-256 mismatch")
+ rec.input_metadata[index] = _JobInputMetadata(bytes=len(data))
+
+
+async def _prepare_job_inputs(
+ rec: _JobRecord,
+ req: _ParseRequest,
+ file_store: FileStore,
+ *,
+ url_timeout: int,
+ allow_http_source: bool,
+ max_url_bytes: int,
+) -> None:
+ """在独立受控阶段下载 URL,保留 HTML 来源信息并逐文件收敛准备失败。"""
+ if rec.status in {"canceled", "failed"}:
+ return
+ rec.status = "waiting-file"
+ rec.started_at = rec.started_at or JobStore._now()
+ for index, entry in enumerate(req.files):
+ if rec.status in {"canceled", "failed"}:
+ return
+ if not isinstance(entry.source, _UrlSource):
+ continue
+ file = rec.files[index]
+ file.status = "waiting-file"
+ started = time.monotonic()
+ try:
+ extracted = await _extract_bytes(
+ entry.source,
+ file_store,
+ url_timeout=url_timeout,
+ allow_http_source=allow_http_source,
+ max_url_bytes=max_url_bytes,
+ )
+ _validate_url_input(rec, index, extracted)
+ rec.prepared_sources[index] = _PreparedInput(extracted, time.monotonic() - started)
+ file.status = "pending"
+ except Exception as exc:
+ logger.exception("Parse-server input preparation failed: job_id=%s file=%r", rec.id, file.name)
+ file.status = "failed"
+ file.error = (
+ ErrorDetail(type=exc.type, code=exc.code, message=str(exc), param=exc.param)
+ if isinstance(exc, MineruError)
+ else ErrorDetail(type="engine_error", code="parse_failed", message=str(exc))
+ )
+ rec.progress.failed += 1
+ if rec.status not in {"canceled", "failed"}:
+ rec.status = "failed" if rec.progress.failed == rec.progress.total else "pending"
+
+
async def _extract_bytes(
source: _ExecutionSource,
file_store: FileStore,
@@ -1220,35 +1312,34 @@ async def _run_job(
if rec.status == "canceled":
return
rec.status = "running"
- rec.started_at = JobStore._now()
+ rec.started_at = rec.started_at or JobStore._now()
with tempfile.TemporaryDirectory(prefix="mineru_job_") as tmpdir:
for i, entry in enumerate(req.files):
if rec.status in {"canceled", "failed"}:
break
fr = rec.files[i]
+ if fr.status in {"failed", "canceled"}:
+ continue
+ prepared = rec.prepared_sources.pop(i, None)
try:
- file_started = time.monotonic()
+ file_started = time.monotonic() - (prepared.duration_seconds if prepared is not None else 0)
fr.status = "running"
- extracted = await _extract_bytes(
- entry.source,
- file_store,
- url_timeout=url_timeout,
- allow_http_source=allow_http_source,
- max_url_bytes=max_url_bytes,
- input_blobs=rec.input_blobs,
+ extracted = (
+ prepared.source
+ if prepared is not None
+ else await _extract_bytes(
+ entry.source,
+ file_store,
+ url_timeout=url_timeout,
+ allow_http_source=allow_http_source,
+ max_url_bytes=max_url_bytes,
+ input_blobs=rec.input_blobs,
+ )
)
data = extracted.data
- input_spec = rec.input_specs.get(i)
- if input_spec is not None and isinstance(entry.source, _UrlSource):
- if not data:
- raise MineruError("invalid_request", "Empty files are not supported")
- if input_spec.bytes is not None and len(data) != input_spec.bytes:
- raise MineruError("bytes_mismatch", "Declared size does not match downloaded bytes")
- sha = hashlib.sha256(data).hexdigest()
- if input_spec.sha256sum is not None and sha != input_spec.sha256sum:
- raise MineruError("file_hash_mismatch", "SHA-256 mismatch")
- rec.input_metadata[i] = _JobInputMetadata(bytes=len(data), sha256sum=sha)
+ if prepared is None and isinstance(entry.source, _UrlSource):
+ _validate_url_input(rec, i, extracted)
suffix = pathlib.Path(fr.name).suffix
stype = _suffix_type(fr.name)
if not stype and not suffix and isinstance(entry.source, _UrlSource) and extracted.media_type == "text/html":
@@ -1288,6 +1379,7 @@ async def _run_job(
if rec.status in {"canceled", "failed"}:
break
+ actual_metadata = MinerUMetadata.model_validate(result.middle_json.extensions.get("mineru"))
from .task_outputs import build_task_output
try:
@@ -1312,6 +1404,7 @@ async def _run_job(
fr.page_range = _page_range_from_result_pages(result.pages)
fr.parse = FileParseInfo(
+ tier=actual_metadata.tier,
duration_ms=int((time.monotonic() - file_started) * 1000),
parser_version=__version__,
)
@@ -1657,8 +1750,21 @@ async def submit_execution(
job_store=job_store,
)
+ async def _bg_prepare() -> None:
+ """复用本次来源配置准备 URL 输入,只有准备成功的文件进入解析队列。"""
+ await _prepare_job_inputs(
+ rec,
+ body,
+ file_store,
+ url_timeout=url_timeout_val,
+ allow_http_source=allow_http_source_val,
+ max_url_bytes=max_url_bytes_val,
+ )
+
try:
- job_store.start_task(rec, _bg_run)
+ job_store.start_task(
+ rec, _bg_run, prepare=_bg_prepare if any(isinstance(entry.source, _UrlSource) for entry in body.files) else None
+ )
except BaseException:
file_store.release_inputs(rec.input_blobs)
job_store._jobs.pop(rec.id, None)
diff --git a/mineru/parser/task_api.py b/mineru/parser/task_api.py
index 5468ad13..bc54f77d 100644
--- a/mineru/parser/task_api.py
+++ b/mineru/parser/task_api.py
@@ -22,7 +22,7 @@ from starlette.requests import ClientDisconnect
from ..errors import MineruError
from ..filetypes import is_flash_only_parse_extension
-from ..task_types import TaskOutputFormat
+from ..task_types import TaskFileState, TaskOutputFormat, TaskStatus
from ..types import Tier
from ..utils.async_utils import drain_future, run_sync
from .api_server import (
@@ -48,8 +48,6 @@ from .page_range import PAGE_RANGE_DESCRIPTION, normalize_page_range_input
MAX_TASK_FILES = 100
TASK_UPLOAD_TTL = 3600
TERMINAL_TASK_STATUSES = frozenset({"completed", "partial", "failed", "canceled"})
-TaskStatus = Literal["waiting_upload", "queued", "running", "completed", "partial", "failed", "canceled"]
-TaskFileState = Literal["waiting_upload", "queued", "running", "completed", "failed", "canceled"]
logger = logging.getLogger(__name__)
@@ -100,38 +98,43 @@ class CreateTaskRequest(UploadTaskOptions):
class TaskUploadLink(BaseModel):
"""描述一个文件的上传目标,客户端原样使用服务端给出的地址和请求头。"""
- client_file_id: str = Field(pattern=r"^[A-Za-z0-9_-]{1,64}$")
- upload_url: str
- upload_headers: dict[str, str]
+ url: str
+ headers: dict[str, str]
expires_at: int
-class TaskFileStatus(TaskFileItem):
- """同时提供上传准备情况与逐文件解析状态,禁止内联解析正文。"""
+class TaskFileStatus(BaseModel):
+ """用统一状态描述文件准备和解析,上传对象仅表示客户端当前可执行的动作。"""
- upload_status: Literal["pending", "ready", "not_required", "expired"]
- file_id: str | None = None
- name: str
+ client_file_id: str = Field(pattern=r"^[A-Za-z0-9_-]{1,64}$")
+ file_name: str
+ bytes: int | None
+ url: str | None
+ upload: TaskUploadLink | None
status: TaskFileState
parse: FileParseInfo | None = None
result_url: str | None = None
error: ErrorDetail | None = None
+ @model_validator(mode="after")
+ def validate_upload_action(self) -> TaskFileStatus:
+ """只有等待客户端文件的条目才能提供上传目标,服务端准备阶段可以为空。"""
+ if self.upload is not None and (self.status != "waiting-file" or self.url is not None):
+ raise ValueError("Only waiting-file without a URL source accepts uploads")
+ return self
+
class TaskStatusResponse(BaseModel):
"""任务状态只携带元数据;每个成功文件提供自己的下载链接。"""
task_id: str
status: TaskStatus
- status_url: str
created_at: str
started_at: str | None = None
finished_at: str | None = None
- tier: Tier | None = None
output_formats: list[TaskOutputFormat]
progress: TaskProgress
files: list[TaskFileStatus]
- pending_uploads: list[TaskUploadLink]
error: ErrorDetail | None = None
@@ -345,7 +348,7 @@ class _TaskFileRecord:
backend: TaskBackend
execution_id: str = field(default_factory=JobStore._new_job_id)
- status: TaskFileState = "waiting_upload"
+ status: TaskFileState = "waiting-file"
upload: TaskUpload | None = None
upload_lock: asyncio.Lock = field(default_factory=asyncio.Lock)
upload_active: bool = False
@@ -370,7 +373,7 @@ class _TaskRecord:
expires_at: int
file_states: list[_TaskFileRecord]
created_at: str = field(default_factory=JobStore._now)
- status: TaskStatus = "waiting_upload"
+ status: TaskStatus = "waiting-file"
refresh_lock: asyncio.Lock = field(default_factory=asyncio.Lock)
changed: asyncio.Event = field(default_factory=asyncio.Event)
operation: asyncio.Task[None] | None = None
@@ -413,10 +416,10 @@ class TaskStore:
states = [file.status for file in record.file_states]
if "running" in states:
record.status = "running"
- elif "queued" in states:
- record.status = "queued"
- elif "waiting_upload" in states:
- record.status = "waiting_upload"
+ elif "pending" in states:
+ record.status = "pending"
+ elif "waiting-file" in states:
+ record.status = "waiting-file"
else:
completed = states.count("completed")
record.status = "completed" if completed == len(states) else ("partial" if completed else "failed")
@@ -430,8 +433,8 @@ class TaskStore:
"""只终结尚未受理上传的文件;截止前已经受理的流允许继续完成。"""
if record.expires_at > time.time() or record.status == "canceled":
return
- for file in record.file_states:
- if file.status == "waiting_upload" and not file.upload_active:
+ for item, file in zip(record.inputs, record.file_states):
+ if item.url is None and file.upload is None and file.status == "waiting-file" and not file.upload_active:
file.expired = True
file.status = "failed"
file.error = ErrorDetail(type="invalid_request_error", code="upload_expired", message="Upload expired")
@@ -511,9 +514,9 @@ class TaskStore:
def _start_file(self, record: _TaskRecord, index: int) -> None:
"""事件循环内同步登记提交句柄,重复 PUT 不能创建第二个子作业。"""
file = record.file_states[index]
- if file.operation is not None or file.status != "waiting_upload" or record.status == "canceled":
+ if file.operation is not None or file.status != "waiting-file" or record.status == "canceled":
return
- file.status = "queued"
+ file.status = "pending" if file.upload is not None else "waiting-file"
record.changed.set()
file.operation = asyncio.create_task(self._dispatch_file(record, index), name=f"mineru-task-file-{file.execution_id}")
self._aggregate(record)
@@ -523,7 +526,7 @@ class TaskStore:
if file.status in TERMINAL_TASK_STATUSES:
return
status = execution.job.get("status")
- if status not in {"queued", "running", "completed", "failed", "canceled"}:
+ if status not in {"waiting-file", "pending", "running", "completed", "failed", "canceled"}:
_invalid("Invalid file status from execution backend", code="invalid_response", status_code=502)
if status == "completed" and execution.result_file_id is None:
_invalid("Completed file has no result", code="invalid_response", status_code=502)
@@ -669,7 +672,7 @@ class TaskStore:
_invalid("Task was canceled", status_code=409, code="task_already_terminal")
if file.upload is not None:
return True
- if file.status != "waiting_upload":
+ if file.status != "waiting-file":
_invalid("File no longer accepts uploads", status_code=409, code="file_already_terminal")
length = request.headers.get("content-length")
if length is not None:
@@ -755,56 +758,54 @@ def _url(request: Request, path: str) -> str:
def task_payload(record: _TaskRecord, request: Request) -> dict[str, Any]:
"""逐文件合并上传与执行元数据,成功文件立即给出独立下载链接。"""
files: list[dict[str, Any]] = []
- links: list[dict[str, Any]] = []
starts: list[str] = []
for item, state in zip(record.inputs, record.file_states):
- file = item.model_dump()
- upload = state.upload
- file.update(
- name=item.file_name,
- upload_status="not_required"
- if item.url is not None
- else ("ready" if upload is not None else ("expired" if state.expired else "pending")),
- status=state.status,
- file_id=upload.file_id if upload else None,
- result_url=None,
- )
+ file: dict[str, Any] = {
+ "client_file_id": item.client_file_id,
+ "file_name": item.file_name,
+ "bytes": item.bytes,
+ "url": item.url,
+ "status": state.status,
+ "upload": None,
+ "parse": None,
+ "result_url": None,
+ "error": None,
+ }
if state.execution is not None:
job = state.execution.job
results = job.get("files") or []
if results:
- file.update({key: results[0].get(key) for key in ("name", "file_id", "page_range", "parse", "error")})
+ file["error"] = results[0].get("error")
+ if state.status == "completed":
+ file["parse"] = results[0].get("parse")
if job.get("started_at"):
starts.append(job["started_at"])
if (metadata := state.execution.input_metadata.get(0)) is not None:
- file.update(bytes=metadata.bytes, sha256sum=metadata.sha256sum)
+ file["bytes"] = metadata.bytes
if state.status == "completed" and state.execution.result_file_id:
file["result_url"] = _url(request, f"/v1/files/{quote(state.execution.result_file_id, safe='')}/content")
if state.error is not None:
file["error"] = state.error.model_dump()
- files.append(file)
if (
item.url is None
- and state.status == "waiting_upload"
+ and state.upload is None
+ and not state.upload_active
+ and state.status == "waiting-file"
and record.status != "canceled"
and record.expires_at > time.time()
):
- links.append(
- {
- "client_file_id": item.client_file_id,
- "upload_url": _url(request, f"/v1/tasks/{record.id}/uploads/{quote(item.client_file_id, safe='')}"),
- "upload_headers": {"Content-Type": "application/octet-stream"},
- "expires_at": record.expires_at,
- }
- )
+ file["upload"] = {
+ "url": _url(request, f"/v1/tasks/{record.id}/uploads/{quote(item.client_file_id, safe='')}"),
+ "headers": {"Content-Type": "application/octet-stream"},
+ "expires_at": record.expires_at,
+ }
+ files.append(file)
payload = {
"task_id": record.id,
"status": record.status,
- "status_url": _url(request, f"/v1/tasks/{record.id}"),
"created_at": record.created_at,
"started_at": min(starts) if starts else None,
"finished_at": record.finished_at,
- "tier": record.options.tier,
"output_formats": record.options.output_formats,
"progress": {
"completed": sum(file.status == "completed" for file in record.file_states),
@@ -812,7 +813,6 @@ def task_payload(record: _TaskRecord, request: Request) -> dict[str, Any]:
"total": len(files),
},
"files": files,
- "pending_uploads": links,
"error": record.error.model_dump() if record.error else None,
}
return TaskStatusResponse.model_validate(payload).model_dump()
diff --git a/mineru/task_types.py b/mineru/task_types.py
index 4995fb6b..2f44b119 100644
--- a/mineru/task_types.py
+++ b/mineru/task_types.py
@@ -1,8 +1,19 @@
-"""Tasks 客户端与服务端共用的轻量格式定义,不导入解析或渲染实现。"""
+"""Tasks 共用的轻量格式与查询路由,不导入解析或渲染实现。"""
from typing import Literal
+from urllib.parse import quote
TaskOutputFormat = Literal["default", "docx", "html", "latex"]
TASK_OUTPUT_FORMATS: tuple[TaskOutputFormat, ...] = ("default", "docx", "html", "latex")
+TaskStatus = Literal["waiting-file", "pending", "running", "completed", "partial", "failed", "canceled"]
+TaskFileState = Literal["waiting-file", "pending", "running", "completed", "failed", "canceled"]
-__all__ = ["TASK_OUTPUT_FORMATS", "TaskOutputFormat"]
+
+def task_status_url(api_url: str, task_id: str) -> str:
+ """保留 API 部署前缀并编码任务标识,拒绝会改变固定路由的点路径。"""
+ if not task_id or task_id in {".", ".."}:
+ raise ValueError("Invalid task_id")
+ return api_url.rstrip("/") + "/v1/tasks/" + quote(task_id, safe="")
+
+
+__all__ = ["TASK_OUTPUT_FORMATS", "TaskFileState", "TaskOutputFormat", "TaskStatus", "task_status_url"]
diff --git a/scripts/http_task_example.py b/scripts/http_task_example.py
index fae615f6..9b9601b9 100644
--- a/scripts/http_task_example.py
+++ b/scripts/http_task_example.py
@@ -20,7 +20,7 @@ from typing import Any
from urllib.parse import quote, urljoin, urlsplit
TERMINAL = frozenset({"completed", "partial", "failed", "canceled"})
-STATUSES = TERMINAL | {"waiting_upload", "queued", "running"}
+STATUSES = TERMINAL | {"waiting-file", "pending", "running"}
RETRYABLE_HTTP = frozenset({408, 429, 500, 502, 503, 504})
REDIRECTS = frozenset({301, 302, 303, 307, 308})
CHUNK_BYTES = 1024 * 1024
@@ -113,8 +113,9 @@ class TaskClient:
if not 0 <= self.poll_interval < float("inf"):
raise ClientError("POLL_INTERVAL must be a non-negative finite number")
self.task_id: str | None = None
- self.status_url: str | None = None
self.saved_ids: set[str] = set()
+ self.uploaded_ids: set[str] = set()
+ self.upload_attempts: dict[str, int] = {}
def resolve(self, url: str) -> str:
"""解析服务返回的绝对或相对链接,保留 OSS 签名查询串。"""
@@ -236,12 +237,15 @@ class TaskClient:
def validate_task(self, payload: dict[str, Any]) -> dict[str, Any]:
"""校验任务身份和文件映射,恢复请求不能误把文件上传给另一项。"""
task_id = payload.get("task_id")
- if not isinstance(task_id, str) or not task_id or (self.task_id is not None and task_id != self.task_id):
+ if (
+ not isinstance(task_id, str)
+ or not task_id
+ or task_id in {".", ".."}
+ or (self.task_id is not None and task_id != self.task_id)
+ ):
raise ClientError("Missing or inconsistent task_id")
if payload.get("status") not in STATUSES:
raise ClientError("Unknown task status")
- if not isinstance(payload.get("status_url"), str) or not payload["status_url"]:
- raise ClientError("Missing status_url")
files = payload.get("files")
if not isinstance(files, list) or len(files) != len(self.files):
raise ClientError("Task file count does not match local inputs")
@@ -256,30 +260,19 @@ class TaskClient:
local = self.by_id[client_id]
if item.get("file_name") != local.path.name or item.get("bytes") != local.size:
raise ClientError("Task files differ from local inputs; use the original file list and order")
- if local.sha256sum is not None and item.get("sha256sum") not in {None, local.sha256sum}:
- raise ClientError("Task hash differs from the local file")
- if item.get("upload_status") not in {"pending", "ready", "expired"}:
- raise ClientError("Unknown upload_status")
- uploads = payload.get("pending_uploads")
- if not isinstance(uploads, list):
- raise ClientError("Missing pending_uploads list")
- seen_uploads: set[str] = set()
- pending = {item["client_file_id"] for item in files if item["upload_status"] == "pending"}
- for link in uploads:
+ if "upload" not in item:
+ raise ClientError("Missing file upload descriptor")
+ link = item["upload"]
+ if link is None:
+ continue
if not isinstance(link, dict):
raise ClientError("Invalid upload descriptor")
- client_id = link.get("client_file_id")
- if not isinstance(client_id, str) or client_id not in self.by_id or client_id in seen_uploads:
- raise ClientError("Invalid or duplicate upload client_file_id")
- seen_uploads.add(client_id)
- if not isinstance(link.get("upload_url"), str) or not link["upload_url"]:
+ if not isinstance(link.get("url"), str) or not link["url"]:
raise ClientError("Invalid upload URL")
- if not isinstance(link.get("upload_headers"), dict):
- raise ClientError("Invalid upload headers")
- if not seen_uploads.issubset(pending):
- raise ClientError("Upload descriptors do not match pending files")
- if payload["status"] in TERMINAL and seen_uploads:
- raise ClientError("Terminal task still returns upload URLs")
+ if not isinstance(link.get("headers"), dict) or type(link.get("expires_at")) is not int:
+ raise ClientError("Invalid upload headers or expiry")
+ if item["status"] != "waiting-file" or payload["status"] in TERMINAL:
+ raise ClientError("Upload descriptor does not match file state")
formats = payload.get("output_formats")
if (
not isinstance(formats, list)
@@ -288,7 +281,7 @@ class TaskClient:
):
raise ClientError("Invalid task output_formats")
for item in files:
- if item.get("status") not in {"waiting_upload", "queued", "running", "completed", "failed", "canceled"}:
+ if item.get("status") not in {"waiting-file", "pending", "running", "completed", "failed", "canceled"}:
raise ClientError("Unknown file status")
if item["status"] == "completed":
if not isinstance(item.get("result_url"), str) or not item["result_url"]:
@@ -296,7 +289,6 @@ class TaskClient:
elif item.get("result_url") is not None:
raise ClientError("Unfinished file has a download URL")
self.task_id = task_id
- self.status_url = self.resolve(payload["status_url"])
return payload
def create(self) -> dict[str, Any]:
@@ -321,9 +313,9 @@ class TaskClient:
def refresh(self) -> dict[str, Any]:
"""只查询已有任务,任何查询失败都不能触发创建重试。"""
- if self.status_url is None:
- raise ClientError("No task status URL")
- return self.json_request("GET", self.status_url)
+ if self.task_id is None or self.task_id in {".", ".."}:
+ raise ClientError("No valid task ID")
+ return self.json_request("GET", self.base + "/v1/tasks/" + quote(self.task_id, safe=""))
def refresh_after_transfer_error(self) -> dict[str, Any]:
"""传输结果不确定时先恢复状态查询,无法确认时不盲目重复发送文件。"""
@@ -337,12 +329,19 @@ class TaskClient:
raise AssertionError("unreachable")
def upload_missing(self, task: dict[str, Any]) -> dict[str, Any]:
- """逐文件上传与恢复,已经 ready 的文件永远不会重新发送。"""
- for original in task["pending_uploads"]:
+ """逐文件上传与恢复,成功 PUT 或已进入后续阶段的文件不重复发送。"""
+ for original in task["files"]:
client_id = original["client_file_id"]
local = self.by_id[client_id]
- link = original
- for attempt in range(3):
+ link = original["upload"]
+ if original["status"] != "waiting-file":
+ self.uploaded_ids.add(client_id)
+ if link is None or client_id in self.uploaded_ids:
+ continue
+ if self.upload_attempts.get(client_id, 0) >= 3:
+ raise ClientError(f"Upload failed after three attempts: {local.path.name}")
+ for attempt in range(self.upload_attempts.get(client_id, 0), 3):
+ self.upload_attempts[client_id] = attempt + 1
def upload_body() -> Generator[bytes, None, None]:
"""每次重试重新打开同一个文件,单文件从头重传。"""
@@ -351,13 +350,14 @@ class TaskClient:
try:
with self.request(
"PUT",
- link["upload_url"],
- headers=link["upload_headers"],
+ link["url"],
+ headers=link["headers"],
body=upload_body,
length=local.size,
) as response:
if not 200 <= response.status < 300:
raise self._http_error(response)
+ self.uploaded_ids.add(client_id)
break
except ClientError as exc:
if not exc.retryable and exc.status not in {403, 410}:
@@ -365,16 +365,13 @@ class TaskClient:
# 请求可能已经落盘但响应丢失;先看状态,再决定是否重传。
current = self.refresh_after_transfer_error()
remote = next(item for item in current["files"] if item["client_file_id"] == client_id)
- if remote["upload_status"] == "ready":
- break
- if remote["status"] in {"failed", "canceled"}:
+ if remote["status"] != "waiting-file":
+ self.uploaded_ids.add(client_id)
break
- replacement = next(
- (item for item in current["pending_uploads"] if item["client_file_id"] == client_id), None
- )
+ replacement = remote["upload"]
if replacement is None:
- raise ClientError("Pending file has no upload URL") from exc
- if exc.status == 403 and replacement["upload_url"] == link["upload_url"]:
+ break
+ if exc.status == 403 and replacement["url"] == link["url"]:
raise
if attempt == 2:
raise ClientError(f"Upload failed after three attempts: {local.path.name}") from exc
@@ -461,17 +458,16 @@ class TaskClient:
resume = os.environ.get("TASK_ID", "")
if resume:
self.task_id = resume
- self.status_url = self.base + "/v1/tasks/" + quote(resume, safe="")
task = self.refresh()
else:
task = self.create()
print(f"task_id={self.task_id}", file=sys.stderr)
self.download_completed(task, output)
- for link in list(task["pending_uploads"]):
- current_links = [item for item in task["pending_uploads"] if item["client_file_id"] == link["client_file_id"]]
- if not current_links:
+ for original in list(task["files"]):
+ current = next(item for item in task["files"] if item["client_file_id"] == original["client_file_id"])
+ if current["upload"] is None:
continue
- task = self.upload_missing({**task, "pending_uploads": current_links})
+ task = self.upload_missing({**task, "files": [current]})
self.download_completed(task, output)
polls = 0
while task["status"] not in TERMINAL:
@@ -486,6 +482,13 @@ class TaskClient:
if not exc.retryable:
raise
continue
+ if any(
+ item["status"] == "waiting-file"
+ and item["upload"] is not None
+ and item["client_file_id"] not in self.uploaded_ids
+ for item in task["files"]
+ ):
+ task = self.upload_missing(task)
self.download_completed(task, output)
if task["status"] in {"failed", "canceled"}:
print(
diff --git a/tests/unittest/task_test_utils.py b/tests/unittest/task_test_utils.py
index 65b96e79..0f43d693 100644
--- a/tests/unittest/task_test_utils.py
+++ b/tests/unittest/task_test_utils.py
@@ -8,12 +8,24 @@ import zipfile
from pathlib import Path
from typing import Any
+import httpx
from fastapi.testclient import TestClient
+from mineru.task_types import task_status_url
from mineru.parser.api_server import FileStore, _JobRecord
from mineru.parser.base import ParseResult
from mineru.parser.task_outputs import build_task_output
+def task_url(client: TestClient | httpx.Client | httpx.AsyncClient, task: dict[str, Any]) -> str:
+ """从客户端部署地址和任务 ID 构造固定查询路径,不依赖响应链接。"""
+ return task_status_url(str(client.base_url), task["task_id"])
+
+
+def pending_files(task: dict[str, Any]) -> list[dict[str, Any]]:
+ """从真实逐文件快照选取待上传项,缺少新 upload 字段时直接失败。"""
+ return [item for item in task["files"] if item["upload"] is not None]
+
+
def stored_source(store: FileStore, path: Path) -> dict[str, str]:
"""为执行层单元测试登记已校验源字节,不创建公开上传资源。"""
sha = hashlib.sha256(path.read_bytes()).hexdigest()
@@ -52,8 +64,8 @@ def submit_task(client: TestClient, path: Path, **options: Any) -> dict[str, Any
)
assert response.status_code == 202, response.text
task = response.json()
- for link in task["pending_uploads"]:
- response = client.put(link["upload_url"], content=data, headers=link["upload_headers"])
+ for link in pending_files(task):
+ response = client.put(link["upload"]["url"], content=data, headers=link["upload"]["headers"])
assert response.status_code < 400, response.text
return task
@@ -62,7 +74,7 @@ def wait_task(client: TestClient, task: dict[str, Any], timeout: float = 5) -> d
"""等待父任务终态,避免使用内部执行 ID 代替公开 task_id。"""
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
- response = client.get(task["status_url"])
+ response = client.get(task_url(client, task))
assert response.status_code == 200, response.text
task = response.json()
if task["status"] in {"completed", "partial", "failed", "canceled"}:
diff --git a/tests/unittest/test_engine_dead_lifecycle.py b/tests/unittest/test_engine_dead_lifecycle.py
index ba129dbc..649586a2 100644
--- a/tests/unittest/test_engine_dead_lifecycle.py
+++ b/tests/unittest/test_engine_dead_lifecycle.py
@@ -2,6 +2,8 @@
from __future__ import annotations
+from task_test_utils import task_url
+
from task_test_utils import submit_task, wait_task
import asyncio
@@ -60,11 +62,11 @@ def test_api_engine_failure_drains_running_and_queued_jobs(tmp_path: Path, monke
client.portal.call(make_gate)
jobs = []
- for name in ("fatal", "running", "queued", "user-cancel"):
+ for name in ("fatal", "running", "pending", "user-cancel"):
source = tmp_path / f"{name}.pdf"
source.write_bytes(name.encode())
jobs.append(submit_task(client, source, tier="flash"))
- assert client.delete(jobs[-1]["status_url"]).status_code == 200
+ assert client.delete(task_url(client, jobs[-1])).status_code == 200
client.portal.call(gate.set)
for job in jobs:
wait_task(client, job)
@@ -83,7 +85,7 @@ def test_api_engine_failure_drains_running_and_queued_jobs(tmp_path: Path, monke
client.portal.call(drain_tasks)
assert entered == ["fatal", "running"]
assert set(cleanup) == {"fatal", "running"}
- assert [client.get(job["status_url"]).json()["status"] for job in jobs] == [
+ assert [client.get(task_url(client, job)).json()["status"] for job in jobs] == [
"failed",
"failed",
"failed",
@@ -186,7 +188,7 @@ def test_old_generation_failure_snapshot_remains_queryable() -> None:
route.metadata["payload"] = {
"job_id": route.public_id,
"status": "running",
- "files": [{"status": "queued"}],
+ "files": [{"status": "pending"}],
"progress": {"total": 1, "completed": 0},
}
client.portal.call(pool.refresh, state)
diff --git a/tests/unittest/test_file_parse.py b/tests/unittest/test_file_parse.py
index 0b5bf360..89b30e52 100644
--- a/tests/unittest/test_file_parse.py
+++ b/tests/unittest/test_file_parse.py
@@ -72,34 +72,31 @@ class Upstream:
ready = task["ready"] and time.monotonic() >= task["uploaded_at"] + self.ready_delay
complete = ready and self.complete and time.monotonic() >= task["uploaded_at"] + self.complete_delay
expired = not ready and task["expires_at"] <= time.time()
- status = "failed" if expired else "completed" if complete else "running" if ready else "waiting_upload"
+ status = "failed" if expired else "completed" if complete else "running" if ready else "waiting-file"
result_url = f"{self.storage_url}/result/{task_id}?signature=a%2Fb&version={task['version']}" if complete else None
if result_url and self.redirect:
result_url = f"{self.api_url}/redirect/{task_id}"
return {
"task_id": task_id,
"status": status,
- "status_url": f"{self.api_url}/v1/tasks/{task_id}",
"output_formats": task["options"]["output_formats"],
"files": [
{
"client_file_id": "file",
+ "file_name": task["options"]["files"][0]["file_name"],
+ "bytes": task["options"]["files"][0]["bytes"],
"status": status,
- "upload_status": "expired" if expired else "ready" if ready else "pending",
+ "upload": None
+ if task["ready"] or expired
+ else {
+ "url": f"{self.storage_url}/upload/{task_id}",
+ "headers": {"x-upload-token": "storage-token"},
+ "expires_at": task["expires_at"],
+ },
"result_url": result_url,
"error": {"code": "upload_expired"} if expired else None,
}
],
- "pending_uploads": []
- if ready or expired
- else [
- {
- "client_file_id": "file",
- "upload_url": f"{self.storage_url}/upload/{task_id}",
- "upload_headers": {"x-upload-token": "storage-token"},
- "expires_at": task["expires_at"],
- }
- ],
}
async def handle(self, request: httpx.Request) -> httpx.Response:
@@ -386,8 +383,8 @@ def test_new_protocol_preserves_but_does_not_restore_legacy_records(tmp_path: Pa
upstream = Upstream()
settings = FileParseSettings(upstream.api_url, "secret", tmp_path)
legacy_namespace = hashlib.sha256(f"{settings.api_url}\0secret".encode()).hexdigest()
- assert settings.namespace == "tasks-v2-" + legacy_namespace
- legacy = tmp_path / legacy_namespace
+ assert settings.namespace == "tasks-v4-" + legacy_namespace
+ legacy = tmp_path / ("tasks-v3-" + legacy_namespace)
source = legacy / "sources" / "old-input"
source.parent.mkdir(parents=True)
source.write_bytes(b"old source")
@@ -457,7 +454,7 @@ def test_restart_at_upload_boundaries(tmp_path: Path, stage: str) -> None:
try:
task = await remote.create(record)
record.remote = task.model_dump()
- record.status = "waiting_upload"
+ record.status = "waiting-file"
if stage == "expired":
upstream.tasks[task.task_id]["expires_at"] = int(time.time()) - 1
finally:
diff --git a/tests/unittest/test_kit_gradio.py b/tests/unittest/test_kit_gradio.py
index e6f20d87..1315c6ac 100644
--- a/tests/unittest/test_kit_gradio.py
+++ b/tests/unittest/test_kit_gradio.py
@@ -703,7 +703,7 @@ def test_v1_client_parse_uses_api_parser_zip_contract(
calls["page_range"] = page_range
assert statuses == ["Preparing request...", "Submitting task..."]
assert status_callback is not None
- status_callback("queued")
+ status_callback("pending")
if observe_running:
status_callback("running")
status_callback("completed")
diff --git a/tests/unittest/test_parser_api_contract.py b/tests/unittest/test_parser_api_contract.py
index 71eb3ccc..12b679d9 100644
--- a/tests/unittest/test_parser_api_contract.py
+++ b/tests/unittest/test_parser_api_contract.py
@@ -720,16 +720,17 @@ def test_api_server_logs_traceback_when_job_file_fails(
def test_parse_info_uses_parser_version_not_backend_version() -> None:
- """解析统计仅保留耗时和解析器版本,不新增逐文件档位。"""
- parse = FileParseInfo(duration_ms=12, parser_version="3.2.1")
+ """解析统计包含实际档位、耗时和解析器版本,不接受已删除的模型字段。"""
+ parse = FileParseInfo(tier="basic", duration_ms=12, parser_version="3.2.1")
assert parse.model_dump(exclude_none=True) == {
+ "tier": "basic",
"duration_ms": 12,
"parser_version": "3.2.1",
}
- for removed in ("backend_version", "model_used", "tier"):
+ for removed in ("backend_version", "model_used"):
with pytest.raises(ValidationError):
- FileParseInfo.model_validate({removed: "unused"})
+ FileParseInfo.model_validate({"tier": "basic", removed: "unused"})
@pytest.mark.parametrize("field", ["backend_version", "parser_version"])
diff --git a/tests/unittest/test_parser_api_ocr.py b/tests/unittest/test_parser_api_ocr.py
index db9c697e..cf5749b6 100644
--- a/tests/unittest/test_parser_api_ocr.py
+++ b/tests/unittest/test_parser_api_ocr.py
@@ -40,7 +40,7 @@ def test_api_parser_serializes_request_ocr_mode(
return httpx.Response(200, content=zip_output())
assert request.method == "POST" and request.url.path == "/v1/tasks"
payloads.append(json.loads(request.content))
- return httpx.Response(202, json={**task_snapshot(), "status_url": "http://localhost:8000/v1/tasks/task-one"})
+ return httpx.Response(202, json=task_snapshot())
transport = httpx.MockTransport(respond)
sync_client, async_client = httpx.Client, httpx.AsyncClient
diff --git a/tests/unittest/test_parser_api_status.py b/tests/unittest/test_parser_api_status.py
index 5f8aafe8..aa1ebe01 100644
--- a/tests/unittest/test_parser_api_status.py
+++ b/tests/unittest/test_parser_api_status.py
@@ -23,12 +23,13 @@ from mineru.types import MiddleJson
@pytest.mark.parametrize(
"statuses",
[
- ("queued", "running", "running", "completed"),
+ ("waiting-file", "pending", "running", "completed"),
+ ("pending", "running", "running", "completed"),
("completed",),
- ("queued", "completed"),
- ("queued", "partial"),
- ("queued", "running", "failed"),
- ("queued", "canceled"),
+ ("pending", "completed"),
+ ("pending", "partial"),
+ ("pending", "running", "failed"),
+ ("pending", "canceled"),
],
)
def test_parse_notifies_submission_and_polls_before_result_building(
@@ -73,13 +74,12 @@ def test_parse_notifies_submission_and_polls_before_result_building(
200,
json={
"task_id": "task_test",
- "status_url": "/v1/tasks/task_test",
"status": status,
"output_formats": ["default"],
"files": [
{
"client_file_id": "file",
- "upload_status": "ready",
+ "upload": None,
"status": "completed" if status == "partial" else status,
"parse": {"duration_ms": 250},
"result_url": "/result",
diff --git a/tests/unittest/test_resource_retention.py b/tests/unittest/test_resource_retention.py
index 225695ad..1579086e 100644
--- a/tests/unittest/test_resource_retention.py
+++ b/tests/unittest/test_resource_retention.py
@@ -355,7 +355,7 @@ def test_external_worker_lost_task_releases_load_and_keeps_snapshot() -> None:
"job", owner_scope="anonymous", worker_id=reservation.worker.worker_id, upstream_id="lost-task"
)
route.metadata.update(
- active_counted=True, reservation=reservation, payload={"status": "running", "files": [{"status": "queued"}]}
+ active_counted=True, reservation=reservation, payload={"status": "running", "files": [{"status": "pending"}]}
)
client.portal.call(_reconcile_jobs_once, app.state.registry, pool)
assert reservation.worker.active_jobs == 0
diff --git a/tests/unittest/test_router_balancing.py b/tests/unittest/test_router_balancing.py
index 2a79e506..8492d5a6 100644
--- a/tests/unittest/test_router_balancing.py
+++ b/tests/unittest/test_router_balancing.py
@@ -2,6 +2,8 @@
from __future__ import annotations
+from task_test_utils import pending_files
+
import asyncio
import threading
import time
@@ -125,7 +127,7 @@ def test_cached_tasks_still_balance(tmp_path: Path, monkeypatch: pytest.MonkeyPa
assert [worker.active_jobs for worker in app.state.worker_pool.workers] == [0, 0]
_upload(client, task, [("source.html", HTML)])
else:
- assert task["pending_uploads"] == []
+ assert pending_files(task) == []
tasks.append(task)
deadline = time.monotonic() + 5
while not app.state.task_store._records[task["task_id"]].file_states[0].submitted:
diff --git a/tests/unittest/test_router_cascade.py b/tests/unittest/test_router_cascade.py
index 89381f70..b3c1654d 100644
--- a/tests/unittest/test_router_cascade.py
+++ b/tests/unittest/test_router_cascade.py
@@ -2,6 +2,8 @@
from __future__ import annotations
+from task_test_utils import pending_files, task_url
+
import asyncio
import hashlib
import io
@@ -103,7 +105,7 @@ async def _terminal(client: httpx.AsyncClient, task: dict[str, Any]) -> dict[str
"""按真实公共链接等待终态,超时不允许测试永久挂起。"""
async with asyncio.timeout(10):
while True:
- response = await client.get(task["status_url"])
+ response = await client.get(task_url(client, task))
assert response.status_code == 200, response.text
result = response.json()
if result["status"] in {"completed", "partial", "failed", "canceled"}:
@@ -120,31 +122,37 @@ async def _check_download(client: httpx.AsyncClient, result: dict[str, Any]) ->
assert b"Preserved content" in archive.read("markdown.md")
-def test_cascade_waits_for_source_file_and_reuses_cached_upload(tmp_path: Path) -> None:
- """上传 ready 但 file_id 尚空时继续等待,交接后正常下载并按 hash 复用源文件。"""
+def test_cascade_accepts_ready_upload_without_source_file_and_reuses_cache(tmp_path: Path) -> None:
+ """下游尚未提交解析也可确认上传,级联只依赖任务身份和私有缓存。"""
async def scenario() -> None:
"""通过提交等待固定竞争窗口,验证内外层均不提前判错。"""
async with _cascade(tmp_path, block_submission=True) as (client, outer, inner, worker, transport):
task = await _submit(client)
- response = await client.put(task["pending_uploads"][0]["upload_url"], content=HTML)
+ response = await client.put(pending_files(task)[0]["upload"]["url"], content=HTML)
assert response.status_code == 200, response.text
await asyncio.wait_for(transport.submission_started.wait(), timeout=5)
inner_record = next(iter(inner.state.task_store._records.values()))
inner_route = outer.state.registry.list("job")[0]
ready = await outer.state.worker_pool.client.get(f"http://inner/v1/tasks/{inner_route.upstream_id}")
- assert ready.json()["files"][0]["upload_status"] == "ready"
- assert ready.json()["files"][0]["file_id"] is None
- pending = (await client.get(task["status_url"])).json()
- assert pending["files"][0]["status"] == "queued", pending
- assert inner_record.status == "queued"
+ assert ready.json()["files"][0]["status"] == "pending"
+ assert ready.json()["files"][0]["upload"] is None
+ assert "file_id" not in ready.json()["files"][0]
+ async with asyncio.timeout(5):
+ while not outer.state.task_store._records[task["task_id"]].file_states[0].submitted:
+ await asyncio.sleep(0.01)
+ assert not outer.state.registry.list("file")
+ assert outer.state.source_store._active_references
+ pending = (await client.get(task_url(client, task))).json()
+ assert pending["files"][0]["status"] == "pending", pending
+ assert inner_record.status == "pending"
transport.allow_submission.set()
result = await _terminal(client, task)
assert result["status"] == "completed", result
- assert result["files"][0]["file_id"] is not None
+ assert "file_id" not in result["files"][0]
await _check_download(client, result)
cached = await _submit(client, filename="renamed.html")
- assert cached["pending_uploads"] == []
+ assert pending_files(cached) == []
result = await _terminal(client, cached)
assert result["status"] == "completed", result
await _check_download(client, result)
@@ -154,17 +162,17 @@ def test_cascade_waits_for_source_file_and_reuses_cached_upload(tmp_path: Path)
asyncio.run(scenario())
-def test_cascade_can_cancel_while_waiting_for_source_file(tmp_path: Path) -> None:
- """等待源 File 的轮询可被取消,内外层预占和准备引用都正常释放。"""
+def test_cascade_can_cancel_while_downstream_submission_is_blocked(tmp_path: Path) -> None:
+ """下游提交阻塞时仍可取消,内外层预占和准备引用都正常释放。"""
async def scenario() -> None:
"""阻塞实际任务提交后取消外层父任务,确认没有遗留解析或缓存租约。"""
async with _cascade(tmp_path, block_submission=True) as (client, outer, inner, worker, transport):
task = await _submit(client)
- response = await client.put(task["pending_uploads"][0]["upload_url"], content=HTML)
+ response = await client.put(pending_files(task)[0]["upload"]["url"], content=HTML)
assert response.status_code == 200, response.text
await asyncio.wait_for(transport.submission_started.wait(), timeout=5)
- response = await client.delete(task["status_url"])
+ response = await client.delete(task_url(client, task))
assert response.status_code == 200, response.text
assert response.json()["status"] == "canceled"
async with asyncio.timeout(5):
diff --git a/tests/unittest/test_router_source_cache.py b/tests/unittest/test_router_source_cache.py
new file mode 100644
index 00000000..9c57ecae
--- /dev/null
+++ b/tests/unittest/test_router_source_cache.py
@@ -0,0 +1,104 @@
+"""私有源缓存不依赖下游 Files 标识,引用和过期由本层任务生命周期管理。"""
+
+from __future__ import annotations
+
+import hashlib
+import time
+from pathlib import Path
+
+import pytest
+from fastapi.testclient import TestClient
+from task_test_utils import task_url
+from test_task_api import HTML, _WorkerTransport, _submit, _terminal, _upload
+
+from mineru.kit.router import RouterSettings, create_app
+from mineru.kit.router.resources import SourceFileStore, StoredSourceFile
+from mineru.parser import api_server
+
+
+def test_private_cache_reference_lifetime_and_transfer_pins(monkeypatch: pytest.MonkeyPatch) -> None:
+ """独立回收只移除到期引用,活跃任务和并发下载均保护同一份字节。"""
+ now = [1000.0]
+ monkeypatch.setattr(time, "time", lambda: now[0])
+ sources = SourceFileStore(retention_seconds=10)
+ try:
+ path = sources._root / "shared"
+ path.write_bytes(b"source")
+ stored = StoredSourceFile(path, 6, hashlib.sha256(b"source").hexdigest(), "application/pdf")
+ for reference in ("task-a", "task-b"):
+ sources.bind_source(reference, stored, owner_scope="owner")
+ sources.hold_reference(reference)
+ assert sources.find_hash("stranger", stored.sha256sum) is None
+ now[0] += 1000
+ sources.collect_expired()
+ assert sources.find_hash("owner", stored.sha256sum) is stored
+ sources.release_reference("task-a", now[0])
+ deadline = sources._file_expiry["task-a"]
+ now[0] += 11
+ sources.release_reference("task-a", now[0])
+ assert sources._file_expiry["task-a"] == deadline
+ sources.collect_expired()
+ assert set(sources._files) == {"task-b"} and path.exists()
+ with sources.pin(stored):
+ sources.release_reference("task-b", now[0])
+ now[0] += 11
+ sources.collect_expired()
+ assert not sources._files and not sources._active_references
+ assert sources.find_hash("owner", stored.sha256sum) is None
+ assert path.exists()
+ assert not path.exists() and not sources._pins
+ finally:
+ sources.close()
+
+
+def test_private_cache_disabled_retention_and_rollback(monkeypatch: pytest.MonkeyPatch) -> None:
+ """禁用保留期仍保留已提交源缓存,准备失败回滚只删除自己的引用。"""
+ sources = SourceFileStore(retention_seconds=0)
+ try:
+ path = sources._root / "source"
+ path.write_bytes(b"source")
+ stored = StoredSourceFile(path, 6, hashlib.sha256(b"source").hexdigest(), "application/pdf")
+ for reference in ("committed", "prepared"):
+ sources.bind_source(reference, stored, owner_scope="owner")
+ sources.hold_reference(reference)
+ sources.delete_file("prepared")
+ sources.delete_file("prepared")
+ sources.release_reference("committed", 1000)
+ monkeypatch.setattr(time, "time", lambda: 10_000_000)
+ sources.collect_expired()
+ assert sources.find_hash("owner", stored.sha256sum) is stored
+ assert set(sources._files) == {"committed"} and not sources._active_references
+ finally:
+ sources.close()
+
+
+def test_parent_cleanup_releases_sources_after_worker_route_disappears(tmp_path: Path) -> None:
+ """执行路由已移除时,父任务仍须结束私有源引用并允许独立缓存回收。"""
+ worker = api_server.create_app(upload_dir=str(tmp_path / "worker"), tier="flash")
+ app = create_app(
+ RouterSettings(upstream_urls=("http://worker",), local_gpus="none", worker_refresh_interval_seconds=0),
+ transport=_WorkerTransport({"worker": worker}),
+ )
+ with TestClient(app) as client:
+ task = _submit(client, [("source.html", HTML)], headers={"x-mineru-task-lease-seconds": "120"})
+ _upload(client, task, [("source.html", HTML)])
+ assert _terminal(client, task)["status"] == "completed"
+ record = app.state.task_store._records[task["task_id"]]
+ sources = app.state.source_store
+ assert sources._active_references
+ client.portal.call(app.state.registry.remove, "job", record.file_states[0].execution_id)
+ assert client.get(task_url(client, task), headers={"x-mineru-task-lease-seconds": "0"}).status_code == 200
+
+ async def drain() -> None:
+ """等待真实父任务清理,确保检查时释放钩子已经执行。"""
+ await record.operation
+
+ client.portal.call(drain)
+ assert not sources._active_references and not sources._pins
+ assert sources._files
+ for reference in sources._files:
+ assert sources._file_expiry[reference] is not None
+ sources._file_expiry[reference] = 0
+ paths = {stored.path for stored in sources._files.values()}
+ client.portal.call(sources.collect_expired)
+ assert not sources._files and all(not path.exists() for path in paths)
diff --git a/tests/unittest/test_task_api.py b/tests/unittest/test_task_api.py
index 9dcfb0fa..8519f291 100644
--- a/tests/unittest/test_task_api.py
+++ b/tests/unittest/test_task_api.py
@@ -2,6 +2,8 @@
from __future__ import annotations
+from task_test_utils import pending_files, task_url
+
import asyncio
import hashlib
import io
@@ -56,6 +58,26 @@ def _execution_apps(app: FastAPI) -> list[FastAPI]:
return list(app.state.test_transport.apps.values()) if hasattr(app.state, "test_transport") else [app]
+def _source_references(app: FastAPI, task: dict[str, Any]) -> set[str]:
+ """从内部存储定位源引用,公共 Tasks 响应不再暴露源文件标识。"""
+ upload = app.state.task_store._records[task["task_id"]].file_states[0].upload
+ assert upload is not None
+ if hasattr(app.state, "source_store"):
+ return {key for key, stored in app.state.source_store._files.items() if stored.path == upload.path}
+ assert upload.file_id is not None
+ return {upload.file_id}
+
+
+def _delete_source_references(client: TestClient, app: FastAPI, references: set[str]) -> None:
+ """删除旧源引用以复现共享字节竞争,Router 私有引用不经过 Files HTTP。"""
+ assert references
+ for reference in references:
+ if hasattr(app.state, "source_store"):
+ client.portal.call(app.state.source_store.delete_file, reference)
+ else:
+ assert client.delete(f"/v1/files/{reference}").status_code == 200
+
+
def _url_manifest(**metadata: Any) -> dict[str, Any]:
"""构造与下载地址名称无关的逻辑文件条目,默认不声明大小或 hash。"""
return {
@@ -135,10 +157,10 @@ def _upload(
client: TestClient, task: dict[str, Any], inputs: list[tuple[str, bytes]], *, headers: dict[str, str] | None = None
) -> None:
"""仅上传返回链接对应的缺失文件,按 client_file_id 保持同名文件的关联。"""
- for link in task["pending_uploads"]:
+ for link in pending_files(task):
index = int(link["client_file_id"][1:]) - 1
response = client.put(
- link["upload_url"], content=inputs[index][1], headers={**link["upload_headers"], **(headers or {})}
+ link["upload"]["url"], content=inputs[index][1], headers={**link["upload"]["headers"], **(headers or {})}
)
assert response.status_code == 200, response.text
@@ -147,7 +169,7 @@ def _terminal(client: TestClient, task: dict[str, Any], *, headers: dict[str, st
"""只轮询状态接口;未完成的文件不能发布 result_url。"""
deadline = time.monotonic() + 20
while time.monotonic() < deadline:
- response = client.get(task["status_url"], headers=headers)
+ response = client.get(task_url(client, task), headers=headers)
assert response.status_code == 200, response.text
payload = response.json()
if payload["status"] in task_api.TERMINAL_TASK_STATUSES:
@@ -161,7 +183,7 @@ def _file_done(client: TestClient, task: dict[str, Any], index: int) -> dict[str
"""只等待指定文件完成,父任务可继续等待其他上传。"""
deadline = time.monotonic() + 20
while time.monotonic() < deadline:
- payload = client.get(task["status_url"]).json()
+ payload = client.get(task_url(client, task)).json()
if payload["files"][index]["status"] in {"completed", "failed", "canceled"}:
return payload
time.sleep(0.01)
@@ -191,13 +213,13 @@ def test_manifest_uploads_and_zip_only_status(service: tuple[TestClient, FastAPI
client, app = service
inputs = [("同名 文档.html", HTML), ("同名 文档.html", SECOND)]
task = _submit(client, inputs, hashes=hashes, output_formats=["default"])
- assert task["status"] == "waiting_upload" and "result_url" not in task
- assert len(task["pending_uploads"]) == 2 and _job_count(app) == 0
- assert client.put(task["pending_uploads"][0]["upload_url"], content=HTML).status_code == 200
+ assert task["status"] == "waiting-file" and "result_url" not in task
+ assert len(pending_files(task)) == 2 and _job_count(app) == 0
+ assert client.put(pending_files(task)[0]["upload"]["url"], content=HTML).status_code == 200
ready = _file_done(client, task, 0)
- assert ready["status"] == "waiting_upload"
- assert [file["upload_status"] for file in ready["files"]] == ["ready", "pending"]
- assert [link["client_file_id"] for link in ready["pending_uploads"]] == ["f2"]
+ assert ready["status"] == "waiting-file"
+ assert [file["status"] for file in ready["files"]] == ["completed", "waiting-file"]
+ assert [link["client_file_id"] for link in pending_files(ready)] == ["f2"]
assert _job_count(app) == 1
assert b"Preserved content" in _archive(client, ready)["markdown.md"]
_upload(client, ready, inputs)
@@ -213,7 +235,7 @@ def test_manifest_uploads_and_zip_only_status(service: tuple[TestClient, FastAPI
assert client.get(f"/v1/tasks/{task['task_id']}/result").status_code == 404
assert client.post("/v1/file_parse").status_code == 404
assert client.post("/tasks").status_code == 404
- assert client.get(f"/v1/files/{result['files'][0]['file_id']}/content").status_code == 403
+ assert all({"file_id", "name", "sha256sum", "page_range"}.isdisjoint(file) for file in result["files"])
def test_direct_and_form_submissions_are_rejected(service: tuple[TestClient, FastAPI]) -> None:
@@ -243,11 +265,12 @@ def test_mixed_cache_hit_survives_source_deletion(service: tuple[TestClient, Fas
initial = _submit(client, [("original.html", HTML)])
_upload(client, initial, [("original.html", HTML)])
original = _terminal(client, initial)
+ old_references = _source_references(app, original)
inputs = [("renamed.html", HTML), ("new.html", SECOND)]
task = _submit(client, inputs)
- assert [link["client_file_id"] for link in task["pending_uploads"]] == ["f2"]
- assert task["files"][0]["upload_status"] == "ready"
- assert client.delete(f"/v1/files/{original['files'][0]['file_id']}").status_code == 200
+ assert [link["client_file_id"] for link in pending_files(task)] == ["f2"]
+ assert task["files"][0]["status"] == "pending" and task["files"][0]["upload"] is None
+ _delete_source_references(client, app, old_references)
duplicate = client.put(
f"/v1/tasks/{task['task_id']}/uploads/f1",
content=_UnreadableBody(),
@@ -256,9 +279,9 @@ def test_mixed_cache_hit_survives_source_deletion(service: tuple[TestClient, Fas
assert duplicate.status_code == 200 and duplicate.headers["connection"] == "close"
_upload(client, task, inputs)
result = _terminal(client, task)
- assert result["status"] == "completed" and result["files"][0]["name"] == "renamed.html"
+ assert result["status"] == "completed" and result["files"][0]["file_name"] == "renamed.html"
cached = _submit(client, inputs)
- assert cached["pending_uploads"] == [] and cached["status"] == "queued"
+ assert pending_files(cached) == [] and cached["status"] == "pending"
assert _terminal(client, cached)["status"] == "completed"
assert _job_count(app) == 5
@@ -270,22 +293,22 @@ def test_no_hash_still_deduplicates_source_storage(service: tuple[TestClient, Fa
_upload(client, first, [("first.html", HTML)])
assert _terminal(client, first)["status"] == "completed"
second = _submit(client, [("second.html", HTML)], hashes=False)
- assert len(second["pending_uploads"]) == 1
+ assert len(pending_files(second)) == 1
_upload(client, second, [("second.html", HTML)])
assert _terminal(client, second)["status"] == "completed"
records = app.state.task_store._records
assert records[first["task_id"]].file_states[0].upload.path == records[second["task_id"]].file_states[0].upload.path
-def test_waiting_upload_retains_deleted_source_views(service: tuple[TestClient, FastAPI]) -> None:
+def test_waiting_file_retains_deleted_source_views(service: tuple[TestClient, FastAPI]) -> None:
"""删除已解析源视图不影响已生成结果,也不阻止其他文件继续上传。"""
client, app = service
inputs = [("a.html", HTML), ("b.html", SECOND)]
task = _submit(client, inputs, hashes=False)
- assert client.put(task["pending_uploads"][0]["upload_url"], content=HTML).status_code == 200
+ assert client.put(pending_files(task)[0]["upload"]["url"], content=HTML).status_code == 200
ready = _file_done(client, task, 0)
- assert client.delete(f"/v1/files/{ready['files'][0]['file_id']}").status_code == 200
- assert client.put(task["pending_uploads"][1]["upload_url"], content=SECOND).status_code == 200
+ _delete_source_references(client, app, _source_references(app, ready))
+ assert client.put(pending_files(task)[1]["upload"]["url"], content=SECOND).status_code == 200
result = _terminal(client, task)
assert result["status"] == "completed", result
assert b"Preserved content" in _archive(client, result)["markdown.md"]
@@ -296,14 +319,15 @@ def test_failed_upload_can_retry_without_reuploading_ready_file(service: tuple[T
client, app = service
inputs = [("a.html", HTML), ("b.html", SECOND)]
task = _submit(client, inputs)
- assert client.put(task["pending_uploads"][0]["upload_url"], content=HTML).status_code == 200
- failed = client.put(task["pending_uploads"][1]["upload_url"], content=SECOND.replace(b"Second", b"Broken"))
+ assert client.put(pending_files(task)[0]["upload"]["url"], content=HTML).status_code == 200
+ failed = client.put(pending_files(task)[1]["upload"]["url"], content=SECOND.replace(b"Second", b"Broken"))
assert failed.status_code == 400 and failed.json()["error"]["code"] == "file_hash_mismatch"
- mismatch = client.put(task["pending_uploads"][1]["upload_url"], content=SECOND + b"x")
+ mismatch = client.put(pending_files(task)[1]["upload"]["url"], content=SECOND + b"x")
assert mismatch.status_code == 400 and mismatch.json()["error"]["code"] == "upload_size_mismatch"
pending = _file_done(client, task, 0)
- assert pending["status"] == "waiting_upload" and _job_count(app) == 1
- assert [file["upload_status"] for file in pending["files"]] == ["ready", "pending"]
+ assert pending["status"] == "waiting-file" and _job_count(app) == 1
+ assert pending["files"][0]["status"] in {"pending", "running", "completed"}
+ assert pending["files"][1]["status"] == "waiting-file"
assert list(app.state.task_store._records[task["task_id"]].directory.iterdir()) == []
_upload(client, pending, inputs)
assert _terminal(client, task)["status"] == "completed" and _job_count(app) == 2
@@ -318,9 +342,9 @@ def test_concurrent_last_uploads_and_duplicate_put_submit_once(service: tuple[Te
"""在服务所属事件循环同时发起两个文件及同一文件的重复请求。"""
async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://testserver") as asynchronous:
return await asyncio.gather(
- asynchronous.put(task["pending_uploads"][0]["upload_url"], content=HTML),
- asynchronous.put(task["pending_uploads"][1]["upload_url"], content=SECOND),
- asynchronous.put(task["pending_uploads"][1]["upload_url"], content=SECOND),
+ asynchronous.put(pending_files(task)[0]["upload"]["url"], content=HTML),
+ asynchronous.put(pending_files(task)[1]["upload"]["url"], content=SECOND),
+ asynchronous.put(pending_files(task)[1]["upload"]["url"], content=SECOND),
)
responses = client.portal.call(upload_all)
@@ -332,29 +356,29 @@ def test_concurrent_last_uploads_and_duplicate_put_submit_once(service: tuple[Te
@pytest.mark.parametrize("expired", [False, True])
-def test_cancel_and_expire_waiting_upload_release_inputs(service: tuple[TestClient, FastAPI], expired: bool) -> None:
+def test_cancel_and_expire_waiting_file_release_inputs(service: tuple[TestClient, FastAPI], expired: bool) -> None:
"""取消或上传过期只终结未完成文件,已生成结果继续可下载。"""
client, app = service
task = _submit(client, [("a.html", HTML), ("b.html", SECOND)])
- assert client.put(task["pending_uploads"][0]["upload_url"], content=HTML).status_code == 200
+ assert client.put(pending_files(task)[0]["upload"]["url"], content=HTML).status_code == 200
ready = _file_done(client, task, 0)
record = app.state.task_store._records[task["task_id"]]
if expired:
record.expires_at = int(time.time()) - 1
- response = client.get(task["status_url"])
+ response = client.get(task_url(client, task))
assert response.json()["files"][1]["error"]["code"] == "upload_expired"
- assert response.json()["files"][1]["upload_status"] == "expired"
+ assert response.json()["files"][1]["status"] == "failed"
else:
- response = client.delete(task["status_url"])
+ response = client.delete(task_url(client, task))
assert response.status_code == 200
assert response.json()["status"] == ("partial" if expired else "canceled")
- assert response.json()["pending_uploads"] == [] and _job_count(app) == 1
+ assert pending_files(response.json()) == [] and _job_count(app) == 1
assert response.json()["files"][0]["result_url"] == ready["files"][0]["result_url"]
assert b"Preserved content" in _archive(client, response.json())["markdown.md"]
- rejected = client.put(task["pending_uploads"][1]["upload_url"], content=SECOND)
+ rejected = client.put(pending_files(task)[1]["upload"]["url"], content=SECOND)
assert rejected.status_code == (410 if expired else 409)
if expired:
- assert client.put(task["pending_uploads"][0]["upload_url"], content=HTML).status_code == 410
+ assert client.put(pending_files(task)[0]["upload"]["url"], content=HTML).status_code == 410
async def wait_cleanup() -> None:
"""等待父任务持有的清理句柄,避免按定时猜测引用是否已释放。"""
@@ -362,7 +386,11 @@ def test_cancel_and_expire_waiting_upload_release_inputs(service: tuple[TestClie
client.portal.call(wait_cleanup)
assert not record.directory.exists()
- assert all(file.backend._owned_files == [] for file in record.file_states)
+ if hasattr(app.state, "source_store"):
+ assert all(not file.backend._prepared_sources and not file.backend._task_sources for file in record.file_states)
+ assert not app.state.source_store._active_references
+ else:
+ assert all(file.backend._owned_files == [] for file in record.file_states)
def test_cancel_running_task_releases_execution_slot(
@@ -381,10 +409,10 @@ def test_cancel_running_task_releases_execution_slot(
task = _submit(client, [("a.html", HTML)])
_upload(client, task, [("a.html", HTML)])
assert started.wait(5)
- response = client.delete(task["status_url"])
+ response = client.delete(task_url(client, task))
assert response.status_code == 200, response.text
assert response.json()["status"] == "canceled"
- assert client.get(task["status_url"]).json()["files"][0]["result_url"] is None
+ assert client.get(task_url(client, task)).json()["files"][0]["result_url"] is None
deadline = time.monotonic() + 5
operation = app.state.task_store._records[task["task_id"]].operation
while operation is not None and not operation.done():
@@ -424,7 +452,7 @@ def test_status_never_reads_or_rebuilds_zip(service: tuple[TestClient, FastAPI],
monkeypatch.setattr(api_server.FileStore, "read_blob", forbid_read)
monkeypatch.setattr(api_server.FileStore, "read_file_data", forbid_read)
for _ in range(3):
- assert client.get(task["status_url"]).json()["files"] == result["files"]
+ assert client.get(task_url(client, task)).json()["files"] == result["files"]
assert "markdown.md" in _archive(client, result)
assert "markdown.md" in _archive(client, result, 1)
assert calls == 2
@@ -505,10 +533,10 @@ def test_input_validation_and_openapi(service: tuple[TestClient, FastAPI]) -> No
assert "file_name" in files_schema["items"]["required"]
assert "client_file_id" not in files_schema["items"]["required"]
assert "filename" not in files_schema["items"]["properties"]
- for model in ("TaskFileStatus", "TaskUploadLink"):
- id_schema = spec["components"]["schemas"][model]
- assert "client_file_id" in id_schema["required"]
- assert id_schema["properties"]["client_file_id"]["type"] == "string"
+ id_schema = spec["components"]["schemas"]["TaskFileStatus"]
+ assert "client_file_id" in id_schema["required"]
+ assert id_schema["properties"]["client_file_id"]["type"] == "string"
+ assert set(spec["components"]["schemas"]["TaskUploadLink"]["properties"]) == {"url", "headers", "expires_at"}
if hasattr(app.state, "job_store"):
metadata_schema = spec["components"]["schemas"]["FileObjectModel"]
assert "file_name" in metadata_schema["required"]
@@ -544,12 +572,12 @@ def test_optional_file_ids_survive_upload_and_router_forwarding(
task = response.json()
for _ in range(2):
assert [item["client_file_id"] for item in task["files"]] == expected_ids
- assert [item["client_file_id"] for item in task["pending_uploads"]] == expected_ids
+ assert [item["client_file_id"] for item in pending_files(task)] == expected_ids
assert all(item["file_name"] == "same.html" and "filename" not in item for item in task["files"])
- task = client.get(task["status_url"]).json()
- for link in task["pending_uploads"]:
+ task = client.get(task_url(client, task)).json()
+ for link in pending_files(task):
for _ in range(2):
- assert client.put(link["upload_url"], content=HTML, headers=link["upload_headers"]).status_code == 200
+ assert client.put(link["upload"]["url"], content=HTML, headers=link["upload"]["headers"]).status_code == 200
completed = _terminal(client, task)
assert completed["status"] == "completed", completed
assert [item["client_file_id"] for item in completed["files"]] == expected_ids
@@ -580,20 +608,25 @@ def test_invalid_explicit_file_ids_fail_before_creation(service: tuple[TestClien
def test_files_metadata_and_attachment_keep_logical_names(service: tuple[TestClient, FastAPI]) -> None:
"""Tasks 和 Files 只返回新文件名字段,源文件与结果附件仍使用各自逻辑名称。"""
- client, _ = service
+ client, app = service
inputs = [("report.v2.html", HTML)]
task = _submit(client, inputs)
_upload(client, task, inputs)
completed = _terminal(client, task)
item = completed["files"][0]
- assert item["file_name"] == item["name"] == "report.v2.html" and "filename" not in item
- source = client.get(f"/v1/files/{item['file_id']}").json()
+ assert item["file_name"] == "report.v2.html" and "name" not in item and "filename" not in item
result = client.get(item["result_url"].removesuffix("/content")).json()
- assert source["file_name"] == "report.v2.html" and "filename" not in source
+ if hasattr(app.state, "file_store"):
+ source_id = next(iter(_source_references(app, completed)))
+ source = client.get(f"/v1/files/{source_id}").json()
+ assert source["file_name"] == "report.v2.html" and "filename" not in source
+ assert client.get(f"/v1/files/{source_id}/content").status_code == 403
+ else:
+ assert all(route.metadata["payload"]["purpose"] == "parse_output" for route in app.state.registry.list("file"))
assert result["file_name"] == "report.v2.zip" and "filename" not in result
- assert set(item["parse"]) == {"duration_ms", "parser_version"}
- assert completed["tier"] == "flash"
- assert completed["pending_uploads"] == []
+ assert set(item["parse"]) == {"tier", "duration_ms", "parser_version"}
+ assert item["parse"]["tier"] == "flash" and "tier" not in completed
+ assert pending_files(completed) == []
assert "uploads" not in completed and "include_model_output" not in completed
download = client.get(item["result_url"])
assert download.status_code == 200
@@ -625,16 +658,16 @@ def test_url_inputs_download_in_worker_and_report_actual_metadata(
task = response.json()
expected_id = item["client_file_id"] or "f1"
assert task["files"][0]["client_file_id"] == expected_id
- assert task["status"] == "queued" and task["pending_uploads"] == []
+ assert task["status"] == "waiting-file" and pending_files(task) == []
assert task["files"][0]["bytes"] == metadata.get("bytes")
- assert task["files"][0]["upload_status"] == "not_required"
+ assert task["files"][0]["status"] == "waiting-file" and task["files"][0]["upload"] is None
result = _terminal(client, task)
assert result["status"] == "completed"
file = result["files"][0]
assert file["client_file_id"] == expected_id
- assert file["file_name"] == file["name"] == "logical.html"
- assert file["bytes"] == len(HTML) and file["sha256sum"] == hashlib.sha256(HTML).hexdigest()
- assert file["upload_status"] == "not_required" and file["file_id"] is None
+ assert file["file_name"] == "logical.html" and "name" not in file
+ assert file["bytes"] == len(HTML) and "sha256sum" not in file
+ assert file["status"] == "completed" and "file_id" not in file and file["upload"] is None
assert file["url"] == item["url"]
assert b"Preserved content" in _archive(client, result)["markdown.md"]
assert len(url_downloads) == 1 and str(url_downloads[0].url) == item["url"]
@@ -661,7 +694,7 @@ def test_url_manifest_validation(service: tuple[TestClient, FastAPI], url_downlo
assert response.status_code == 415
assert _job_count(app) == 0 and url_downloads == []
schema = client.get("/openapi.json").json()
- assert "not_required" in schema["components"]["schemas"]["TaskFileStatus"]["properties"]["upload_status"]["enum"]
+ assert "upload_status" not in schema["components"]["schemas"]["TaskFileStatus"]["properties"]
def test_mixed_url_upload_partial_and_no_early_reservation(
@@ -677,12 +710,12 @@ def test_mixed_url_upload_partial_and_no_early_reservation(
response = client.post("/v1/tasks", json={"files": manifest})
assert response.status_code == 202, response.text
task = response.json()
- assert task["status"] == "queued"
- assert [link["client_file_id"] for link in task["pending_uploads"]] == ["local"]
+ assert task["status"] == "waiting-file"
+ assert [link["client_file_id"] for link in pending_files(task)] == ["local"]
ready = _file_done(client, task, 0)
assert b"Preserved content" in _archive(client, ready)["markdown.md"]
- assert ready["files"][1]["status"] == "waiting_upload"
- assert client.put(task["pending_uploads"][0]["upload_url"], content=SECOND).status_code == 200
+ assert ready["files"][1]["status"] == "waiting-file"
+ assert client.put(pending_files(task)[0]["upload"]["url"], content=SECOND).status_code == 200
result = _terminal(client, task)
assert result["status"] == "partial" and result["progress"] == {"completed": 2, "failed": 1, "total": 3}
assert result["files"][0]["bytes"] == len(HTML)
@@ -794,7 +827,7 @@ def test_url_logical_filename_drives_flash_fallback(
assert response.status_code == 202, response.text
result = _terminal(client, response.json())
assert result["status"] == "completed", result
- assert result["files"][0]["name"] == "logical.html"
+ assert result["files"][0]["file_name"] == "logical.html"
assert b"Preserved content" in _archive(client, result)["markdown.md"]
assert len(url_downloads) == 1
@@ -836,7 +869,7 @@ def test_cancel_url_download_closes_stream_and_releases_worker(
task = response.json()
assert started.wait(5)
before = time.monotonic()
- canceled = client.delete(task["status_url"])
+ canceled = client.delete(task_url(client, task))
assert canceled.status_code == 200 and canceled.json()["status"] == "canceled"
assert time.monotonic() - before < 2
assert closed.wait(5)
@@ -848,7 +881,7 @@ def test_cancel_url_download_closes_stream_and_releases_worker(
while any(worker.active_jobs for worker in app.state.worker_pool.workers):
assert time.monotonic() < deadline, "Canceled URL task retained its worker reservation"
time.sleep(0.01)
- final = client.get(task["status_url"]).json()
+ final = client.get(task_url(client, task)).json()
assert final["status"] == "canceled" and final["files"][0]["result_url"] is None
assert final["files"][0]["bytes"] is None
@@ -865,26 +898,28 @@ def test_router_private_hash_cache_and_task_scope(tmp_path: Path) -> None:
task = _submit(client, [("a.html", HTML)], headers=owner)
_upload(client, task, [("a.html", HTML)], headers=owner)
result = _terminal(client, task, headers=owner)
- assert client.get(task["status_url"], headers=stranger).status_code == 404
+ assert client.get(task_url(client, task), headers=stranger).status_code == 404
assert client.get(result["files"][0]["result_url"], headers=stranger).status_code == 404
other = _submit(client, [("a.html", HTML)], headers=stranger)
- assert len(other["pending_uploads"]) == 1
- assert client.delete(other["status_url"], headers=stranger).status_code == 200
+ assert len(pending_files(other)) == 1
+ assert client.delete(task_url(client, other), headers=stranger).status_code == 200
owned = _submit(client, [("a.html", HTML)], headers=owner)
- assert not owned["pending_uploads"]
+ assert not pending_files(owned)
assert _terminal(client, owned, headers=owner)["status"] == "completed"
@pytest.mark.parametrize("root", ["", "/mineru"])
def test_task_links_respect_proxy_root(tmp_path: Path, root: str) -> None:
- """状态、上传与结果链接全部携带部署前缀,客户端不用重建任何地址。"""
+ """固定查询路由保留客户端部署前缀,上传和结果仍使用服务返回的链接。"""
app = api_server.create_app(upload_dir=str(tmp_path), tier="flash", api_key="secret")
- with TestClient(app, root_path=root, headers={"Authorization": "Bearer secret"}) as client:
+ with TestClient(
+ app, base_url="http://testserver" + root, root_path=root, headers={"Authorization": "Bearer secret"}
+ ) as client:
manifest = [{"client_file_id": "f1", "file_name": "a.html", "bytes": len(HTML)}]
- response = client.post(root + "/v1/tasks", json={"files": manifest, "tier": "flash"})
+ response = client.post("/v1/tasks", json={"files": manifest, "tier": "flash"})
assert response.status_code == 202
task = response.json()
- assert task["status_url"].startswith("http://testserver" + root + "/v1/tasks/")
+ assert task_url(client, task).startswith("http://testserver" + root + "/v1/tasks/")
_upload(client, task, [("a.html", HTML)])
result = _terminal(client, task)
assert result["files"][0]["result_url"].startswith("http://testserver" + root + "/v1/files/")
@@ -899,9 +934,9 @@ def test_inflight_upload_may_finish_after_deadline(service: tuple[TestClient, Fa
client, app = service
task = _submit(client, [("a.html", HTML), ("b.html", SECOND)], hashes=False)
record = app.state.task_store._records[task["task_id"]]
- expires_at = task["pending_uploads"][0]["expires_at"]
+ expires_at = pending_files(task)[0]["upload"]["expires_at"]
assert 3598 <= expires_at - time.time() <= 3600
- assert client.get(task["status_url"]).json()["pending_uploads"][0]["expires_at"] == expires_at
+ assert pending_files(client.get(task_url(client, task)).json())[0]["upload"]["expires_at"] == expires_at
async def cross_deadline() -> httpx.Response:
"""让真实 ASGI 上传跨过时限,并在受理流中途并发查询和尝试新 PUT。"""
@@ -912,17 +947,17 @@ def test_inflight_upload_may_finish_after_deadline(service: tuple[TestClient, Fa
yield HTML[:10]
assert record.file_states[0].upload_active
record.expires_at = int(time.time()) - 1
- state = (await transport.get(task["status_url"])).json()
- assert state["files"][0]["status"] == "waiting_upload"
+ state = (await transport.get(task_url(client, task))).json()
+ assert state["files"][0]["status"] == "waiting-file"
assert state["files"][1]["status"] == "failed"
- assert state["pending_uploads"] == []
- rejected = await transport.put(task["pending_uploads"][1]["upload_url"], content=SECOND)
+ assert pending_files(state) == []
+ rejected = await transport.put(pending_files(task)[1]["upload"]["url"], content=SECOND)
assert rejected.status_code == 410
if interrupted:
raise ClientDisconnect
yield HTML[10:]
- return await transport.put(task["pending_uploads"][0]["upload_url"], content=body())
+ return await transport.put(pending_files(task)[0]["upload"]["url"], content=body())
response = client.portal.call(cross_deadline)
result = _terminal(client, task)
@@ -934,7 +969,7 @@ def test_inflight_upload_may_finish_after_deadline(service: tuple[TestClient, Fa
assert response.status_code == 200
assert result["status"] == "partial"
assert b"Preserved content" in _archive(client, result)["markdown.md"]
- assert client.put(task["pending_uploads"][0]["upload_url"], content=HTML).status_code == 410
+ assert client.put(pending_files(task)[0]["upload"]["url"], content=HTML).status_code == 410
def test_expired_upload_rejects_before_reading_body(service: tuple[TestClient, FastAPI]) -> None:
@@ -942,12 +977,13 @@ def test_expired_upload_rejects_before_reading_body(service: tuple[TestClient, F
client, app = service
task = _submit(client, [("a.html", HTML)], hashes=False)
app.state.task_store._records[task["task_id"]].expires_at = int(time.time()) - 1
- response = client.put(task["pending_uploads"][0]["upload_url"], content=_UnreadableBody())
+ response = client.put(pending_files(task)[0]["upload"]["url"], content=_UnreadableBody())
assert response.status_code == 410 and response.json()["error"]["code"] == "upload_expired"
for _ in range(2):
- state = client.get(task["status_url"]).json()
- assert state["status"] == "failed" and state["pending_uploads"] == []
- assert state["files"][0]["upload_status"] == "expired"
+ state = client.get(task_url(client, task)).json()
+ assert state["status"] == "failed" and pending_files(state) == []
+ assert state["files"][0]["status"] == "failed"
+ assert state["files"][0]["error"]["code"] == "upload_expired"
assert _job_count(app) == 0
@@ -962,16 +998,16 @@ def test_completed_child_retention_starts_after_parent_finishes(service: tuple[T
worker.state.job_store.retention_seconds = 1
worker.state.file_store.retention_seconds = 1
task = _submit(client, [("a.html", HTML), ("b.html", SECOND)], hashes=False)
- assert client.put(task["pending_uploads"][0]["upload_url"], content=HTML).status_code == 200
+ assert client.put(pending_files(task)[0]["upload"]["url"], content=HTML).status_code == 200
ready = _file_done(client, task, 0)
result_url = ready["files"][0]["result_url"]
time.sleep(1.05)
for worker in _execution_apps(app):
worker.state.job_store.collect_expired()
worker.state.file_store.collect_expired()
- assert client.get(task["status_url"]).json()["files"][0]["status"] == "completed"
+ assert client.get(task_url(client, task)).json()["files"][0]["status"] == "completed"
assert client.get(result_url).status_code == 200
- assert client.put(task["pending_uploads"][1]["upload_url"], content=SECOND).status_code == 200
+ assert client.put(pending_files(task)[1]["upload"]["url"], content=SECOND).status_code == 200
assert _terminal(client, task)["status"] == "completed"
record = app.state.task_store._records[task["task_id"]]
@@ -982,7 +1018,7 @@ def test_completed_child_retention_starts_after_parent_finishes(service: tuple[T
client.portal.call(finish_retention)
assert client.get(result_url).status_code == 200
time.sleep(1.05)
- assert client.get(task["status_url"]).status_code == 404
+ assert client.get(task_url(client, task)).status_code == 404
assert client.get(result_url).status_code == 404
@@ -1030,7 +1066,7 @@ def test_worker_upload_renews_result_lease_after_long_transfer(tmp_path: Path) -
async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://testserver") as transport:
response = await transport.put(
- task["pending_uploads"][0]["upload_url"], content=content(), headers={"x-mineru-task-lease-seconds": "120"}
+ pending_files(task)[0]["upload"]["url"], content=content(), headers={"x-mineru-task-lease-seconds": "120"}
)
assert response.status_code == 200
assert record.lease_expires_at > time.time() + 110
@@ -1039,7 +1075,7 @@ def test_worker_upload_renews_result_lease_after_long_transfer(tmp_path: Path) -
result = _terminal(client, task)
assert result["status"] == "completed" and record.retention_epoch is None
assert client.get(result["files"][0]["result_url"]).status_code == 200
- assert client.get(task["status_url"], headers={"x-mineru-task-lease-seconds": "0"}).status_code == 200
+ assert client.get(task_url(client, task), headers={"x-mineru-task-lease-seconds": "0"}).status_code == 200
def test_router_ends_local_result_lease_even_if_worker_rejects_release(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
@@ -1089,13 +1125,13 @@ def test_removed_lists_preserve_task_creation_query_and_cancel(service: tuple[Te
assert client.get("/v1/tasks").status_code == 405
assert client.get("/v1/files").status_code == 404
assert "uploads" not in task and "include_model_output" not in task
- assert task["tier"] == "flash"
- assert set(task["pending_uploads"][0]) == {"client_file_id", "upload_url", "upload_headers", "expires_at"}
- assert "/uploads/" in task["pending_uploads"][0]["upload_url"]
- assert client.get(task["status_url"]).json()["pending_uploads"] == task["pending_uploads"]
- canceled = client.delete(task["status_url"])
+ assert {"tier", "status_url", "pending_uploads"}.isdisjoint(task)
+ assert set(pending_files(task)[0]["upload"]) == {"url", "headers", "expires_at"}
+ assert "/uploads/" in pending_files(task)[0]["upload"]["url"]
+ assert pending_files(client.get(task_url(client, task)).json()) == pending_files(task)
+ canceled = client.delete(task_url(client, task))
assert canceled.status_code == 200
- assert canceled.json()["status"] == "canceled" and canceled.json()["pending_uploads"] == []
+ assert canceled.json()["status"] == "canceled" and pending_files(canceled.json()) == []
def test_tasks_only_routes_and_health(service: tuple[TestClient, FastAPI]) -> None:
@@ -1123,8 +1159,10 @@ def test_tasks_only_routes_and_health(service: tuple[TestClient, FastAPI]) -> No
assert "/v1/files" not in schema["paths"]
assert {"TaskListItem", "TaskListResponse", "FileListResponse"}.isdisjoint(schema["components"]["schemas"])
status_properties = schema["components"]["schemas"]["TaskStatusResponse"]["properties"]
- assert "pending_uploads" in status_properties
- assert {"uploads", "include_model_output"}.isdisjoint(status_properties)
+ assert {"status_url", "tier", "pending_uploads", "uploads", "include_model_output"}.isdisjoint(status_properties)
+ file_properties = schema["components"]["schemas"]["TaskFileStatus"]["properties"]
+ assert {"sha256sum", "page_range", "file_id", "name", "upload_status"}.isdisjoint(file_properties)
+ assert "upload" in schema["components"]["schemas"]["TaskFileStatus"]["required"]
assert "upload_method" not in schema["components"]["schemas"]["TaskUploadLink"]["properties"]
@@ -1215,5 +1253,5 @@ def test_router_lease_timeout_does_not_expire_other_worker_results(tmp_path: Pat
assert worker_record.retention_epoch is None
client.portal.call(check_renewal)
- assert client.get(result["status_url"]).json()["status"] == "completed"
+ assert client.get(task_url(client, result)).json()["status"] == "completed"
assert client.get(result["files"][0]["result_url"]).status_code == 200
diff --git a/tests/unittest/test_task_client_file_states.py b/tests/unittest/test_task_client_file_states.py
new file mode 100644
index 00000000..408abefa
--- /dev/null
+++ b/tests/unittest/test_task_client_file_states.py
@@ -0,0 +1,88 @@
+"""SDK 和同步包装在等待文件但没有上传动作时只轮询,避免重复传输或提前清理。"""
+
+from __future__ import annotations
+
+import time
+from concurrent.futures import ThreadPoolExecutor
+from pathlib import Path
+
+import httpx
+import pytest
+from fastapi.testclient import TestClient
+from test_file_parse import Upstream, application, submit
+from test_tasks_sdk import bind_transport, parse, task_snapshot, zip_output
+
+from mineru.parser import MinerUApiParser
+
+
[email protected]("asynchronous", [False, True])
[email protected]("outcome", ["success", "lost_accepted", "lost_retry"])
+def test_sdk_waits_without_upload_and_preserves_retry_budget(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, asynchronous: bool, outcome: str
+) -> None:
+ """上传目标延后出现、成功后延迟确认和响应丢失均不触发重复创建或无限上传。"""
+ source = tmp_path / "file.pdf"
+ source.write_bytes(b"pdf")
+ puts = 0
+ queries = 0
+ creates = 0
+
+ def handle(request: httpx.Request) -> httpx.Response:
+ """按阶段提供无上传动作和待重传快照,所有结果仍由正式 SDK 消费。"""
+ nonlocal puts, queries, creates
+ if request.method == "POST":
+ creates += 1
+ return httpx.Response(202, json=task_snapshot("waiting-file"))
+ if request.method == "PUT":
+ puts += 1
+ queries = 0
+ assert request.content == b"pdf" and request.headers["content-length"] == "3"
+ if puts == 1 and outcome != "success":
+ raise httpx.ReadError("upload response lost", request=request)
+ return httpx.Response(200)
+ if request.url.host == "storage.test":
+ return httpx.Response(200, content=zip_output())
+ queries += 1
+ if puts == 0:
+ snapshot = task_snapshot("waiting-file", upload=queries >= 2)
+ elif outcome == "lost_retry" and puts == 1:
+ snapshot = task_snapshot("waiting-file", upload=queries >= 3)
+ else:
+ snapshot = task_snapshot("waiting-file" if queries < 3 else "pending" if queries == 3 else "completed")
+ return httpx.Response(200, json=snapshot)
+
+ bind_transport(monkeypatch, handle)
+ result = parse(MinerUApiParser(api_url="https://api.test"), source, asynchronous)
+ assert len(result.pages) == 1 and creates == 1
+ assert puts == (2 if outcome == "lost_retry" else 1)
+
+
+def test_file_parse_keeps_source_until_input_is_ready(tmp_path: Path) -> None:
+ """缓存命中后的服务端准备期间保留本地源文件,不把 upload=null 当成已就绪。"""
+ upstream = Upstream()
+ upstream.cache_hit = True
+ upstream.ready_delay = 1000
+ app = application(tmp_path, upstream)
+ with TestClient(app) as client, ThreadPoolExecutor(max_workers=1) as executor:
+ pending = executor.submit(submit, client, wait_timeout=0)
+ try:
+ deadline = time.monotonic() + 3
+ while True:
+ record = app.state.service.records.get("request_1")
+ if record is not None and record.remote is not None:
+ break
+ assert time.monotonic() < deadline
+ time.sleep(0.01)
+ state = client.get("/v1/file_parse/request_1").json()
+ assert state["status"] == "waiting-file"
+ assert record.task.files[0].upload is None
+ retained = Path(record.source_path)
+ assert retained.is_file() and upstream.uploads == 0
+ assert not pending.done()
+ finally:
+ upstream.ready_delay = 0
+ response = pending.result(timeout=3)
+ assert response.status_code in {200, 202}
+ assert client.get("/v1/file_parse/request_1").json()["status"] == "completed"
+ assert record.source_path is None and not retained.exists()
+ assert upstream.uploads == 0 and len(upstream.tasks) == 1
diff --git a/tests/unittest/test_task_file_states.py b/tests/unittest/test_task_file_states.py
new file mode 100644
index 00000000..45601f13
--- /dev/null
+++ b/tests/unittest/test_task_file_states.py
@@ -0,0 +1,181 @@
+"""统一文件状态的准备、排队、取消和耗时边界,使用真实 API 生命周期验证。"""
+
+from __future__ import annotations
+
+import asyncio
+import threading
+import time
+from collections.abc import AsyncIterator
+from pathlib import Path
+from types import SimpleNamespace
+from typing import Any
+
+import httpx
+import pytest
+from fastapi.testclient import TestClient
+from task_test_utils import task_url
+from test_task_api import HTML, _archive, _submit, _terminal, _upload
+from test_tasks_sdk import middle
+
+from mineru.parser import api_server
+from mineru.parser.base import ParseResult
+
+
+def _install_downloads(monkeypatch: pytest.MonkeyPatch, handler: Any) -> None:
+ """只隔离源站下载,保留客户端到 API 的真实协议和任务调度。"""
+
+ def client(**kwargs: Any) -> httpx.AsyncClient:
+ """为每次源下载创建独立 HTTPX 客户端,不修改全局 HTTPX 对象。"""
+ return httpx.AsyncClient(transport=httpx.MockTransport(handler), **kwargs)
+
+ monkeypatch.setattr(api_server, "httpx", SimpleNamespace(AsyncClient=client))
+
+
+def _url_task(client: TestClient, name: str = "remote") -> dict[str, Any]:
+ """创建无需客户端上传的 URL 文件,初始必须处于文件准备阶段。"""
+ response = client.post(
+ "/v1/tasks", json={"tier": "flash", "files": [{"file_name": name + ".html", "url": "https://source.test/" + name}]}
+ )
+ assert response.status_code == 202, response.text
+ task = response.json()
+ assert task["status"] == "waiting-file"
+ assert task["files"][0]["upload"] is None and "upload_status" not in task["files"][0]
+ return task
+
+
[email protected]("cancel", [False, True])
+def test_url_preparation_does_not_block_ready_files_or_use_upload_expiry(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, cancel: bool
+) -> None:
+ """单解析名额下慢下载不阻塞已上传文件,URL 准备不受客户端上传截止时间影响。"""
+ started, release, closed = threading.Event(), threading.Event(), threading.Event()
+
+ class Source(httpx.AsyncByteStream):
+ """提供可取消的源下载流,确认取消真正关闭了网络输入。"""
+
+ async def __aiter__(self) -> AsyncIterator[bytes]:
+ """等待测试放行后输出有效 HTML,取消必须中断等待。"""
+ started.set()
+ while not release.is_set():
+ await asyncio.sleep(0.01)
+ yield HTML
+
+ async def aclose(self) -> None:
+ """记录源流关闭,区别取消和正常放行。"""
+ closed.set()
+
+ _install_downloads(monkeypatch, lambda _: httpx.Response(200, stream=Source()))
+ app = api_server.create_app(upload_dir=str(tmp_path), tier="flash", concurrency=1)
+ try:
+ with TestClient(app) as client:
+ remote = _url_task(client)
+ assert started.wait(3)
+ app.state.task_store._records[remote["task_id"]].expires_at = 0
+ snapshot = client.get(task_url(client, remote)).json()
+ assert snapshot["status"] == "waiting-file" and snapshot["files"][0]["error"] is None
+ assert snapshot["files"][0]["parse"] is None
+ local = _submit(client, [("local.html", HTML)])
+ _upload(client, local, [("local.html", HTML)])
+ assert _terminal(client, local)["status"] == "completed"
+ assert not release.is_set()
+ if cancel:
+ assert client.delete(task_url(client, remote)).json()["status"] == "canceled"
+ assert closed.wait(3)
+ else:
+ release.set()
+ completed = _terminal(client, remote)
+ assert completed["status"] == "completed"
+ assert completed["files"][0]["bytes"] == len(HTML)
+ assert b"Preserved content" in _archive(client, completed)["markdown.md"]
+ assert all(not job.prepared_sources for job in app.state.job_store._jobs.values())
+ finally:
+ release.set()
+
+
[email protected]("cancel_ready", [False, True])
+def test_ready_url_inputs_wait_in_pending_with_bounded_preparation(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, cancel_ready: bool
+) -> None:
+ """文件下载完成后排队,准备缓冲有界,排队取消会释放缓冲及准备名额。"""
+ parsing, release = threading.Event(), threading.Event()
+ downloads: list[str] = []
+ original = api_server.parse_async
+
+ async def infer(path: str, **kwargs: Any) -> ParseResult:
+ """仅阻塞占位文件的解析,让后续 URL 输入稳定停在排队阶段。"""
+ if b"blocker" in Path(path).read_bytes():
+ parsing.set()
+ while not release.is_set():
+ await asyncio.sleep(0.01)
+ return await original(path, **kwargs)
+
+ def download(request: httpx.Request) -> httpx.Response:
+ """记录实际开始的下载,尚未取得准备名额时不应访问源站。"""
+ downloads.append(request.url.path)
+ return httpx.Response(200, content=HTML)
+
+ monkeypatch.setattr(api_server, "parse_async", infer)
+ _install_downloads(monkeypatch, download)
+ app = api_server.create_app(upload_dir=str(tmp_path), tier="flash", concurrency=1)
+ try:
+ with TestClient(app) as client:
+ blocking = [("blocker.html", b"<html><body>blocker</body></html>")]
+ first = _submit(client, blocking)
+ _upload(client, first, blocking)
+ assert parsing.wait(3)
+ second = _url_task(client, "second")
+ deadline = time.monotonic() + 3
+ while client.get(task_url(client, second)).json()["status"] != "pending":
+ assert time.monotonic() < deadline
+ time.sleep(0.01)
+ record = app.state.task_store._records[second["task_id"]]
+ job = app.state.job_store._jobs[record.file_states[0].execution_id]
+ assert job.prepared_sources and job.files[0].parse is None
+ third = _url_task(client, "third")
+ assert client.get(task_url(client, third)).json()["status"] == "waiting-file"
+ assert downloads == ["/second"]
+ if cancel_ready:
+ assert client.delete(task_url(client, second)).json()["status"] == "canceled"
+ deadline = time.monotonic() + 3
+ while "/third" not in downloads:
+ assert time.monotonic() < deadline
+ time.sleep(0.01)
+ assert not job.prepared_sources
+ release.set()
+ assert _terminal(client, first)["status"] == "completed"
+ assert _terminal(client, second)["status"] == ("canceled" if cancel_ready else "completed")
+ assert _terminal(client, third)["status"] == "completed"
+ assert all(not item.prepared_sources for item in app.state.job_store._jobs.values())
+ finally:
+ release.set()
+
+
+def test_file_duration_includes_preparation_but_excludes_queue_wait(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ """确定性时钟区分两秒准备、千秒排队和三秒解析,最终耗时应为五秒。"""
+ clock = [0.0]
+ monkeypatch.setattr(api_server, "time", SimpleNamespace(time=time.time, monotonic=lambda: clock[0]))
+
+ def download(request: httpx.Request) -> httpx.Response:
+ """源下载推进两秒执行时钟,真实准备逻辑仍负责完整性检查。"""
+ clock[0] += 2
+ return httpx.Response(200, content=HTML)
+
+ async def infer(path: str, **kwargs: Any) -> ParseResult:
+ """解析推进三秒时钟,保留真实导出和统计发布。"""
+ clock[0] += 3
+ return ParseResult(middle())
+
+ _install_downloads(monkeypatch, download)
+ monkeypatch.setattr(api_server, "parse_async", infer)
+ app = api_server.create_app(upload_dir=str(tmp_path), tier="flash", concurrency=1)
+ with TestClient(app) as client:
+ client.portal.call(app.state.job_store._semaphore.acquire)
+ task = _url_task(client)
+ deadline = time.monotonic() + 3
+ while client.get(task_url(client, task)).json()["status"] != "pending":
+ assert time.monotonic() < deadline
+ time.sleep(0.01)
+ clock[0] += 1000
+ client.portal.call(app.state.job_store._semaphore.release)
+ result = _terminal(client, task)
+ assert result["files"][0]["parse"]["duration_ms"] == 5000
diff --git a/tests/unittest/test_task_http.py b/tests/unittest/test_task_http.py
index 3644c5ff..55227040 100644
--- a/tests/unittest/test_task_http.py
+++ b/tests/unittest/test_task_http.py
@@ -2,6 +2,8 @@
from __future__ import annotations
+from task_test_utils import pending_files, task_url
+
import asyncio
import hashlib
import io
@@ -149,7 +151,7 @@ def _wait(client: httpx.Client, task: dict[str, Any], timeout: float = 60) -> di
"""有界轮询任务元数据,不能调用已删除的 result 路由。"""
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
- response = client.get(task["status_url"])
+ response = client.get(task_url(client, task))
assert response.status_code == 200, response.text
payload = response.json()
if payload["status"] in TERMINAL:
@@ -185,10 +187,10 @@ def test_pdf_without_client_id_keeps_metadata_and_cached_results(live_service: _
response = client.post("/v1/tasks", json=payload)
assert response.status_code == 202, response.text
task = response.json()
- assert task["files"][0]["client_file_id"] == task["pending_uploads"][0]["client_file_id"] == "f1"
- link = task["pending_uploads"][0]
+ assert task["files"][0]["client_file_id"] == pending_files(task)[0]["client_file_id"] == "f1"
+ link = pending_files(task)[0]
for _ in range(2):
- upload = client.put(link["upload_url"], content=data, headers=link["upload_headers"])
+ upload = client.put(link["upload"]["url"], content=data, headers=link["upload"]["headers"])
assert upload.status_code == 200, upload.text
completed = _wait(client, task)
assert completed["status"] == "completed", completed
@@ -207,8 +209,8 @@ def test_pdf_without_client_id_keeps_metadata_and_cached_results(live_service: _
cached = client.post("/v1/tasks", json=payload)
assert cached.status_code == 202, cached.text
cached_task = cached.json()
- assert cached_task["pending_uploads"] == [] and cached_task["files"][0]["client_file_id"] == "f1"
- assert cached_task["files"][0]["upload_status"] == "ready"
+ assert pending_files(cached_task) == [] and cached_task["files"][0]["client_file_id"] == "f1"
+ assert cached_task["files"][0]["status"] == "pending" and cached_task["files"][0]["upload"] is None
cached_result = _wait(client, cached_task)
assert cached_result["status"] == "completed", cached_result
cached_download = client.get(cached_result["files"][0]["result_url"])
@@ -247,15 +249,16 @@ def test_router_keeps_tls_proxy_headers_on_the_public_hop_only(tmp_path: Path) -
) as client:
for index in range(2):
task = _submit(client, [(f"document-{index}.html", HTML)])
- assert urlsplit(task["status_url"]).scheme == "https"
- assert len(task["pending_uploads"]) == (1 if index == 0 else 0)
- for upload in task["pending_uploads"]:
+ assert "status_url" not in task
+ assert all(urlsplit(item["upload"]["url"]).scheme == "https" for item in pending_files(task))
+ assert len(pending_files(task)) == (1 if index == 0 else 0)
+ for upload in pending_files(task):
# 以本地 HTTP 模拟 TLS 终止代理到 Router 的后段连接,保留真实入站代理头。
- response = client.put(urlsplit(upload["upload_url"]).path, content=HTML)
+ response = client.put(urlsplit(upload["upload"]["url"]).path, content=HTML)
assert response.status_code == 200, response.text
deadline = time.monotonic() + 10
while time.monotonic() < deadline:
- response = client.get(urlsplit(task["status_url"]).path)
+ response = client.get(urlsplit(task_url(client, task)).path)
assert response.status_code == 200, response.text
result = response.json()
if result["status"] in TERMINAL:
@@ -342,18 +345,18 @@ def test_hash_preflight_sends_only_missing_file_bytes(live_service: _LiveService
"""真实网络计数证明混合命中仅上传缺失文件,全命中不发送任何文件 PUT。"""
with _client(live_service) as client:
first = _submit(client, [("a.html", HTML)])
- assert client.put(first["pending_uploads"][0]["upload_url"], content=HTML).status_code == 200
+ assert client.put(pending_files(first)[0]["upload"]["url"], content=HTML).status_code == 200
assert _wait(client, first)["status"] == "completed"
second_data = b"<h1>second</h1>"
mixed = _submit(client, [("renamed.html", HTML), ("b.html", second_data)])
- assert [link["client_file_id"] for link in mixed["pending_uploads"]] == ["f2"]
- assert client.put(mixed["pending_uploads"][0]["upload_url"], content=second_data).status_code == 200
+ assert [link["client_file_id"] for link in pending_files(mixed)] == ["f2"]
+ assert client.put(pending_files(mixed)[0]["upload"]["url"], content=second_data).status_code == 200
assert _wait(client, mixed)["status"] == "completed"
base = f"/v1/tasks/{mixed['task_id']}/uploads/"
assert live_service.counts.get(base + "f1", 0) == 0
assert live_service.counts[base + "f2"] == len(second_data)
hit = _submit(client, [("a.html", HTML), ("b.html", second_data)])
- assert hit["pending_uploads"] == []
+ assert pending_files(hit) == []
assert _wait(client, hit)["status"] == "completed"
assert not any(hit["task_id"] in path for path in live_service.counts)
denied = client.post(
@@ -376,8 +379,8 @@ def test_disconnect_then_resume_keeps_uploaded_files(live_service: _LiveService,
second_path.write_bytes(second)
with _client(live_service) as client:
task = _submit(client, [(first_path.name, HTML), (second_path.name, second)])
- assert client.put(task["pending_uploads"][0]["upload_url"], content=HTML).status_code == 200
- target = urlsplit(task["pending_uploads"][1]["upload_url"])
+ assert client.put(pending_files(task)[0]["upload"]["url"], content=HTML).status_code == 200
+ target = urlsplit(pending_files(task)[1]["upload"]["url"])
connection = socket.create_connection((target.hostname, target.port), timeout=5)
prefix = second[:4096]
try:
@@ -392,17 +395,18 @@ def test_disconnect_then_resume_keeps_uploaded_files(live_service: _LiveService,
assert live_service.counts[target.path] == len(prefix)
finally:
connection.close()
- pending = client.get(task["status_url"]).json()
- assert pending["status"] in {"waiting_upload", "queued", "running"}
- assert [item["upload_status"] for item in pending["files"]] == ["ready", "pending"]
+ pending = client.get(task_url(client, task)).json()
+ assert pending["status"] in {"waiting-file", "pending", "running"}
+ assert pending["files"][0]["status"] in {"pending", "running", "completed"}
+ assert pending["files"][1]["status"] == "waiting-file"
result = _run_script(_script_env(tmp_path, live_service.url, TASK_ID=task["task_id"]), first_path, second_path)
assert result.returncode == 0, result.stdout + result.stderr
- assert live_service.counts[urlsplit(task["pending_uploads"][0]["upload_url"]).path] == len(HTML)
+ assert live_service.counts[urlsplit(pending_files(task)[0]["upload"]["url"]).path] == len(HTML)
assert live_service.counts[target.path] == len(prefix) + len(second)
assert len(live_service.app.state.task_store._records) == 1
members = _members((tmp_path / "results/f2/b.zip").read_bytes())
assert b"Resumed file" in members["markdown.md"]
- final = client.get(task["status_url"]).json()
+ final = client.get(task_url(client, task)).json()
assert final["task_id"] == task["task_id"] and final["status"] == "completed"
@@ -422,12 +426,12 @@ def test_cancel_response_does_not_wait_for_native_thread_cleanup(
try:
with _client(live_service) as client:
task = _submit(client, [("a.html", HTML)])
- assert client.put(task["pending_uploads"][0]["upload_url"], content=HTML).status_code == 200
+ assert client.put(pending_files(task)[0]["upload"]["url"], content=HTML).status_code == 200
assert entered.wait(5)
- canceled = client.delete(task["status_url"], timeout=2)
+ canceled = client.delete(task_url(client, task), timeout=2)
assert canceled.status_code == 200 and canceled.json()["status"] == "canceled"
assert not release.is_set()
- assert client.get(task["status_url"]).json()["files"][0]["result_url"] is None
+ assert client.get(task_url(client, task)).json()["files"][0]["result_url"] is None
finally:
release.set()
@@ -458,13 +462,13 @@ def test_upload_over_200_mib_is_accepted(live_service: _LiveService, tmp_path: P
task = response.json()
with source.open("rb") as stream:
uploaded = client.put(
- task["pending_uploads"][0]["upload_url"], content=stream, headers={"Content-Length": str(size)}
+ pending_files(task)[0]["upload"]["url"], content=stream, headers={"Content-Length": str(size)}
)
assert uploaded.status_code == 200, uploaded.text
- assert live_service.counts[urlsplit(task["pending_uploads"][0]["upload_url"]).path] == size
+ assert live_service.counts[urlsplit(pending_files(task)[0]["upload"]["url"]).path] == size
assert started.wait(60)
- assert client.get(task["status_url"]).json()["files"][0]["upload_status"] == "ready"
- assert client.delete(task["status_url"]).json()["status"] == "canceled"
+ assert client.get(task_url(client, task)).json()["files"][0]["status"] in {"pending", "running", "completed"}
+ assert client.delete(task_url(client, task)).json()["status"] == "canceled"
@pytest.mark.parametrize("page_count", [601, 1001])
@@ -478,7 +482,7 @@ def test_large_page_count_pdf_is_not_rejected(live_service: _LiveService, page_c
data = buffer.getvalue()
with _client(live_service) as client:
task = _submit(client, [("many-pages.pdf", data)])
- assert client.put(task["pending_uploads"][0]["upload_url"], content=data).status_code == 200
+ assert client.put(pending_files(task)[0]["upload"]["url"], content=data).status_code == 200
result = _wait(client, task, timeout=90)
assert result["status"] == "completed", result
download = client.get(result["files"][0]["result_url"])
@@ -514,7 +518,7 @@ def test_real_documents_match_local_parser_outputs(live_service: _LiveService, d
expected = build_task_output(baseline, ["default"], Path(directory) / "outputs").read_bytes()
with _client(live_service) as client:
task = _submit(client, [(name, data)], page_range=page_range, output_formats=["default"])
- assert client.put(task["pending_uploads"][0]["upload_url"], content=data).status_code == 200
+ assert client.put(pending_files(task)[0]["upload"]["url"], content=data).status_code == 200
result = _wait(client, task)
assert result["status"] == "completed", result
output = client.get(result["files"][0]["result_url"])
@@ -576,11 +580,11 @@ def test_url_task_over_http_downloads_chunked_source(live_service: _LiveService,
)
assert response.status_code == 202, response.text
task = response.json()
- assert task["pending_uploads"] == [] and task["files"][0]["upload_status"] == "not_required"
+ assert pending_files(task) == [] and task["files"][0]["status"] == "waiting-file"
result = _wait(client, task)
assert result["status"] == "completed", result
assert result["files"][0]["bytes"] == len(data)
- assert result["files"][0]["sha256sum"] == hashlib.sha256(data).hexdigest()
+ assert "sha256sum" not in result["files"][0]
members = _members(client.get(result["files"][0]["result_url"]).content)
assert live_service.counts == {}
assert len(requests) == 1 and requests[0].method == "GET"
@@ -595,7 +599,7 @@ def test_url_task_over_http_downloads_chunked_source(live_service: _LiveService,
baseline_url = baseline_result["files"][0]["result_url"]
else:
baseline = _submit(client, [(item["file_name"], data)], page_range="2")
- assert client.put(baseline["pending_uploads"][0]["upload_url"], content=data).status_code == 200
+ assert client.put(pending_files(baseline)[0]["upload"]["url"], content=data).status_code == 200
baseline_result = _wait(client, baseline)
assert baseline_result["status"] == "completed"
baseline_url = baseline_result["files"][0]["result_url"]
@@ -636,7 +640,7 @@ class _CloudState:
files = []
for index, item in enumerate(record["files"]):
client_id = item["client_file_id"]
- status = "completed" if client_id in ready else "waiting_upload"
+ status = "completed" if client_id in ready else "waiting-file"
if client_id in expired:
status = "failed"
if client_id in ready and self.forced_status in {"failed", "running"}:
@@ -650,9 +654,17 @@ class _CloudState:
result_url = f"{self.gateway_url}/redirect/{task_id}/{client_id}"
files.append(
{
- **item,
- "name": item["file_name"],
- "upload_status": "ready" if client_id in ready else ("expired" if client_id in expired else "pending"),
+ "client_file_id": client_id,
+ "file_name": item["file_name"],
+ "bytes": item["bytes"],
+ "url": item.get("url"),
+ "upload": None
+ if client_id in ready | expired
+ else {
+ "url": f"{self.storage_url}/objects/{task_id}/{client_id}?signature=0",
+ "headers": {"Content-Type": "application/octet-stream", "X-Upload-Token": "storage-token"},
+ "expires_at": record["expires_at"],
+ },
"status": status,
"result_url": result_url,
}
@@ -660,30 +672,18 @@ class _CloudState:
states = [item["status"] for item in files]
if "running" in states:
status = "running"
- elif "waiting_upload" in states:
- status = "waiting_upload"
+ elif "waiting-file" in states:
+ status = "waiting-file"
else:
status = (
"completed"
if all(value == "completed" for value in states)
else ("partial" if "completed" in states else "failed")
)
- uploads = [
- {
- "client_file_id": item["client_file_id"],
- "upload_url": f"{self.storage_url}/objects/{task_id}/{item['client_file_id']}?signature=0",
- "upload_headers": {"Content-Type": "application/octet-stream", "X-Upload-Token": "storage-token"},
- "expires_at": record["expires_at"],
- }
- for item in record["files"]
- if item["client_file_id"] not in ready | expired
- ]
payload = {
"task_id": task_id,
"status": status,
- "status_url": f"{self.gateway_url}/v1/tasks/{task_id}",
"files": files,
- "pending_uploads": uploads,
"error": None,
"output_formats": record["output_formats"],
}
diff --git a/tests/unittest/test_task_outputs.py b/tests/unittest/test_task_outputs.py
index 355a9915..8ed8c1c2 100644
--- a/tests/unittest/test_task_outputs.py
+++ b/tests/unittest/test_task_outputs.py
@@ -74,6 +74,7 @@ def _illustrated_result() -> tuple[ParseResult, bytes]:
],
is_full_document=True,
metadata={"file_suffix": "docx", "producer": {"name": "mineru", "version": "test"}},
+ extensions={"mineru": {"tier": "flash", "parse_mode": "txt"}},
)
return ParseResult(middle_json=middle), png
diff --git a/tests/unittest/test_task_response_contract.py b/tests/unittest/test_task_response_contract.py
new file mode 100644
index 00000000..96d710a7
--- /dev/null
+++ b/tests/unittest/test_task_response_contract.py
@@ -0,0 +1,139 @@
+"""跨部署验证逐文件响应与实际档位,阻止内部请求字段再次进入公共协议。"""
+
+from __future__ import annotations
+
+import io
+import json
+from pathlib import Path
+from typing import Any
+
+import pytest
+from docx import Document
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+from PIL import Image
+from reportlab.pdfgen.canvas import Canvas
+from test_task_api import _WorkerTransport, _archive, _submit, _terminal, _upload
+
+from mineru.integrations.docvortex import build_metadata
+from mineru.kit.router import RouterSettings, create_app as create_router
+from mineru.parser import api_server
+from mineru.parser.base import ParseResult
+from mineru.types import MiddleJson, PageInfo, Tier
+
+
+def _mixed_inputs() -> list[tuple[str, bytes]]:
+ """生成有效 PDF、HTML、DOCX 和 PNG,原生格式执行真实解析和导出。"""
+ pdf = io.BytesIO()
+ canvas = Canvas(pdf)
+ canvas.drawString(60, 720, "Mixed task PDF")
+ canvas.save()
+ word = io.BytesIO()
+ document = Document()
+ document.add_paragraph("Mixed task DOCX")
+ document.save(word)
+ image = io.BytesIO()
+ Image.new("RGB", (20, 20), "white").save(image, "PNG")
+ return [
+ ("input.pdf", pdf.getvalue()),
+ ("input.html", b"<html><body><p>Mixed task HTML</p></body></html>"),
+ ("input.docx", word.getvalue()),
+ ("input.png", image.getvalue()),
+ ]
+
+
+def _service(tmp_path: Path, deployment: str) -> FastAPI:
+ """创建完整本地或双 worker Router 服务,复用真实上传和后台执行流程。"""
+ if deployment == "api":
+ return api_server.create_app(upload_dir=str(tmp_path / "api"), tier="standard")
+ workers = {
+ name: api_server.create_app(upload_dir=str(tmp_path / name), tier="standard") for name in ("worker-a", "worker-b")
+ }
+ return create_router(
+ RouterSettings(
+ upstream_urls=tuple(f"http://{name}" for name in workers), local_gpus="none", worker_refresh_interval_seconds=0
+ ),
+ transport=_WorkerTransport(workers),
+ )
+
+
[email protected]("deployment", ["api", "router"])
+ "requested,actual",
+ [("flash", "flash"), ("basic", "basic"), ("standard", "standard"), ("advanced", "advanced"), ("advanced", "basic")],
+)
+def test_mixed_file_tiers_come_from_completed_results(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, deployment: str, requested: Tier, actual: Tier
+) -> None:
+ """混合任务逐文件展示实际档位,最后一组刻意让推理结果与请求不同。"""
+ monkeypatch.setattr(api_server, "_preflight_tier_dependencies", lambda *_args, **_kwargs: None)
+ original = api_server.parse_async
+ calls: list[tuple[str, Tier]] = []
+
+ async def infer(path: str, **kwargs: Any) -> ParseResult:
+ """仅替代 PDF 和图片推理,原生文档仍走真实解析,记录实际路由参数。"""
+ suffix = Path(path).suffix
+ calls.append((suffix, kwargs["tier"]))
+ if suffix in {".html", ".docx"}:
+ return await original(path, **kwargs)
+ effort = {"flash": "flash", "basic": "medium", "standard": "high", "advanced": "xhigh"}[actual]
+ return ParseResult(
+ MiddleJson(
+ pages=[PageInfo(page_idx=0)],
+ is_full_document=True,
+ metadata={"file_suffix": "pdf", "producer": {"name": "mineru", "version": "test"}},
+ extensions=build_metadata(effort=effort, parse_mode="txt"),
+ )
+ )
+
+ monkeypatch.setattr(api_server, "parse_async", infer)
+ inputs = _mixed_inputs()
+ with TestClient(_service(tmp_path, deployment)) as client:
+ task = _submit(client, inputs, tier=requested)
+ assert {"status_url", "tier", "pending_uploads"}.isdisjoint(task)
+ for item in task["files"]:
+ assert set(item) == {
+ "client_file_id",
+ "file_name",
+ "bytes",
+ "url",
+ "upload",
+ "status",
+ "parse",
+ "result_url",
+ "error",
+ }
+ assert item["parse"] is None
+ assert set(item["upload"]) == {"url", "headers", "expires_at"}
+ _upload(client, task, inputs)
+ completed = _terminal(client, task)
+ assert completed["status"] == "completed", completed
+ assert [item["parse"]["tier"] for item in completed["files"]] == [actual, "flash", "flash", actual]
+ assert dict(calls) == {".pdf": requested, ".html": "flash", ".docx": "flash", ".png": requested}
+ for index, item in enumerate(completed["files"]):
+ assert item["upload"] is None
+ assert set(item["parse"]) == {"tier", "duration_ms", "parser_version"}
+ assert item["parse"]["duration_ms"] >= 0
+ middle = json.loads(_archive(client, completed, index)["middle_json.json"])
+ assert middle["extensions"]["mineru"]["tier"] == item["parse"]["tier"]
+ assert b"Mixed task HTML" in _archive(client, completed, 1)["markdown.md"]
+ assert b"Mixed task DOCX" in _archive(client, completed, 2)["markdown.md"]
+
+
+def test_missing_actual_metadata_does_not_fabricate_requested_tier(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ """结果缺少真实档位时解析失败,不能以请求默认值伪造完成统计。"""
+
+ async def infer(path: str, **kwargs: Any) -> ParseResult:
+ """返回缺少产品扩展的结果,覆盖 worker 生产者违反契约的边界。"""
+ return ParseResult(
+ MiddleJson(pages=[], metadata={"file_suffix": "html", "producer": {"name": "test", "version": "test"}})
+ )
+
+ monkeypatch.setattr(api_server, "parse_async", infer)
+ with TestClient(api_server.create_app(upload_dir=str(tmp_path), tier="flash")) as client:
+ inputs = [("input.html", b"<p>missing metadata</p>")]
+ task = _submit(client, inputs)
+ _upload(client, task, inputs)
+ result = _terminal(client, task)
+ assert result["status"] == "failed"
+ assert result["files"][0]["parse"] is None and result["files"][0]["result_url"] is None
diff --git a/tests/unittest/test_tasks_sdk.py b/tests/unittest/test_tasks_sdk.py
index 0fde29a5..25726793 100644
--- a/tests/unittest/test_tasks_sdk.py
+++ b/tests/unittest/test_tasks_sdk.py
@@ -43,33 +43,28 @@ def zip_output() -> bytes:
def task_snapshot(status: str = "completed", *, upload: bool = False) -> dict[str, Any]:
- """提供可独立调整状态和上传准备情况的单文件任务快照。"""
+ """提供逐文件内嵌上传描述,不伪造已移除的查询链接或顶层档位。"""
return {
"task_id": "task-one",
- "status_url": "https://api.test/v1/tasks/task-one",
"status": status,
"output_formats": ["default"],
"files": [
{
"client_file_id": "file",
+ "file_name": "file.pdf",
+ "bytes": 3,
"status": status,
- "upload_status": "pending" if upload else "ready",
- "parse": {"duration_ms": 250},
+ "upload": {
+ "url": "https://storage.test/input",
+ "headers": {},
+ "expires_at": 9999999999,
+ }
+ if upload
+ else None,
+ "parse": {"tier": "flash", "duration_ms": 250, "parser_version": "test"} if status == "completed" else None,
"result_url": "https://storage.test/result?signature=original" if status == "completed" else None,
}
],
- "pending_uploads": (
- [
- {
- "client_file_id": "file",
- "upload_url": "https://storage.test/input",
- "upload_headers": {},
- "expires_at": 9999999999,
- }
- ]
- if upload
- else []
- ),
}
@@ -88,6 +83,50 @@ def parse(parser: MinerUApiParser, source: Path, asynchronous: bool, **kwargs: A
return asyncio.run(parser.parse_async(source, **kwargs)) if asynchronous else parser.parse(source, **kwargs)
[email protected]("asynchronous", [False, True])
[email protected]("prefix", ["", "/gateway/api"])
+def test_task_query_uses_configured_prefix_without_status_url(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, asynchronous: bool, prefix: str
+) -> None:
+ """响应不含查询链接时仍能轮询,代理前缀和编码后的任务 ID 均不丢失。"""
+ source = tmp_path / "file.pdf"
+ source.write_bytes(b"pdf")
+ queries: list[str] = []
+
+ def handle(request: httpx.Request) -> httpx.Response:
+ """区分固定 API 路径和跨源结果下载,禁止向对象存储泄漏凭据。"""
+ if request.url.host == "storage.test":
+ assert request.headers.get("authorization") is None
+ return httpx.Response(200, content=zip_output())
+ assert request.headers.get("authorization") == "Bearer secret"
+ if request.method == "POST":
+ assert request.url.path == prefix + "/v1/tasks"
+ snapshot = task_snapshot("pending")
+ else:
+ queries.append(request.url.raw_path.decode())
+ assert request.url.raw_path.decode() == prefix + "/v1/tasks/task%20one"
+ snapshot = task_snapshot()
+ snapshot["task_id"] = "task one"
+ assert "status_url" not in snapshot
+ return httpx.Response(200, json=snapshot)
+
+ bind_transport(monkeypatch, handle)
+ result = parse(MinerUApiParser(api_url="https://api.test" + prefix, api_key="secret"), source, asynchronous)
+ assert len(result.pages) == 1 and len(queries) == 1
+
+
[email protected]("asynchronous", [False, True])
+def test_tasks_sdk_rejects_missing_nested_upload(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, asynchronous: bool) -> None:
+ """缺少新 upload 字段的旧响应不能被当作缓存命中或触发旧字段回退。"""
+ source = tmp_path / "file.pdf"
+ source.write_bytes(b"pdf")
+ snapshot = task_snapshot("waiting-file", upload=True)
+ del snapshot["files"][0]["upload"]
+ bind_transport(monkeypatch, lambda _: httpx.Response(202, json=snapshot))
+ with pytest.raises(api_client._V1APIError, match="upload descriptor"):
+ parse(MinerUApiParser(api_url="https://api.test"), source, asynchronous)
+
+
@pytest.mark.parametrize("asynchronous", [False, True])
@pytest.mark.parametrize("cached", [False, True])
@pytest.mark.parametrize("lost_upload_response", [False, True])
@@ -98,7 +137,7 @@ def test_tasks_sdk_upload_cache_and_lost_response(
source = tmp_path / "report.pdf"
source.write_bytes(b"%PDF-example")
calls: list[httpx.Request] = []
- state = task_snapshot("completed" if cached else "waiting_upload", upload=not cached)
+ state = task_snapshot("completed" if cached else "waiting-file", upload=not cached)
def handle(request: httpx.Request) -> httpx.Response:
"""记录线上协议,不接受任何独立上传或 Jobs 路由。"""
@@ -138,7 +177,7 @@ def test_tasks_sdk_upload_cache_and_lost_response(
assert len(result.pages) == 1 and durations == [250.0]
assert sum(r.method == "POST" for r in calls) == 1
assert sum(r.method == "PUT" for r in calls) == (0 if cached else 1)
- assert statuses == (["completed"] if cached else ["waiting_upload", "completed"])
+ assert statuses == (["completed"] if cached else ["waiting-file", "completed"])
@pytest.mark.parametrize("asynchronous", [False, True])
@@ -152,7 +191,7 @@ def test_tasks_sdk_transport_recovery(
source = tmp_path / "file.pdf"
source.write_bytes(b"pdf")
counts = {"POST": 0, "PUT": 0, "query": 0, "download": 0}
- state = task_snapshot("waiting_upload" if failure == "upload" else "queued", upload=failure == "upload")
+ state = task_snapshot("waiting-file" if failure == "upload" else "pending", upload=failure == "upload")
def handle(request: httpx.Request) -> httpx.Response:
"""在指定阶段只注入一次故障,观测是否重复创建或泄露认证。"""
@@ -204,9 +243,8 @@ def test_tasks_sdk_transport_recovery(
"field,value",
[
("task_id", "other"),
- ("status_url", "https://evil.test/metadata"),
+ ("task_id", ".."),
("output_formats", ["html"]),
- ("pending_uploads", None),
],
)
def test_tasks_sdk_rejects_changed_identity(
@@ -218,7 +256,7 @@ def test_tasks_sdk_rejects_changed_identity(
def handle(request: httpx.Request) -> httpx.Response:
"""提交合法,后续查询注入字段变化。"""
- response = task_snapshot("queued") if request.method == "POST" else {**task_snapshot(), field: value}
+ response = task_snapshot("pending") if request.method == "POST" else {**task_snapshot(), field: value}
return httpx.Response(200, json=response)
bind_transport(monkeypatch, handle)
@@ -282,7 +320,7 @@ def test_tasks_sdk_retries_are_bounded(
def handle(request: httpx.Request) -> httpx.Response:
"""按阶段产生持续故障,创建请求仍只提交一次。"""
if request.method == "POST":
- return httpx.Response(202, json=task_snapshot("queued" if failure == "query" else "completed"))
+ return httpx.Response(202, json=task_snapshot("pending" if failure == "query" else "completed"))
if request.url.host == "api.test":
if failure == "query":
attempts.append(request)
@@ -302,9 +340,7 @@ def test_tasks_sdk_retries_are_bounded(
@pytest.mark.parametrize("asynchronous", [False, True])
- "field,value", [("task_id", None), ("status", {}), ("files", []), ("status_url", "file:///tmp/result")]
-)
[email protected]("field,value", [("task_id", None), ("status", {}), ("files", []), ("task_id", ".")])
def test_tasks_sdk_rejects_malformed_creation(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, asynchronous: bool, field: str, value: Any
) -> None:
@@ -360,7 +396,7 @@ def test_tasks_sdk_retries_transient_poll_responses(
"""首次创建成功,按预算注入轮询故障,之后返回可读取的完整结果。"""
if request.method == "POST":
counts["post"] += 1
- return httpx.Response(202, json=task_snapshot("queued"))
+ return httpx.Response(202, json=task_snapshot("pending"))
if request.url.host == "api.test":
counts["query"] += 1
if counts["query"] <= failures: