Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.
Python ai4sciencedocument-analysisdocxextract-datalayout-analysisocrparserpdfpdf-converterpdf-extractor-llmpdf-extractor-pretrainpdf-extractor-ragpdf-parserpptxpythonxlsx
feat(router): add dynamic upstream management
myhloli31bea79b7d4e
diff --git a/docs/en/usage/cli_tools.md b/docs/en/usage/cli_tools.mdindex bc592659..f7826e68 100644--- a/docs/en/usage/cli_tools.md+++ b/docs/en/usage/cli_tools.md@@ -106,6 +106,8 @@ mineru-kit vlm-server --engine auto --port 30000`mineru-router` remains a command-name alias for `mineru-kit router`. Router exposes only the V1 API and accepts only its documented worker options.+Set `--admin-api-key` or `MINERU_ROUTER_ADMIN_API_KEY` to enable independently authenticated upstream registration, listing, and safe deregistration. Dynamic membership lasts only for the current process; another Router can be registered as an upstream. See [management APIs and cascading](./tasks.md#dynamic-router-upstreams-and-cascading).+## Environment variables- `MINERU_HOME`: root for MinerU configuration, cache, and document-library state.diff --git a/docs/en/usage/tasks.md b/docs/en/usage/tasks.mdindex 9fe9284b..50bcf4e4 100644--- a/docs/en/usage/tasks.md+++ b/docs/en/usage/tasks.md@@ -153,6 +153,55 @@ PUT raw bytes using the returned `upload.url/headers`; no completion call is reqLocal source caches default to 24-hour retention with collection every five minutes; active tasks and transfers protect source bytes. API/Router task records and indexes are in memory and are not guaranteed to survive restarts.+## Dynamic Router upstreams and cascading++A running Router can register, inspect, and safely deregister remote upstreams. An upstream can be a V1 API Server or another Router. Management affects only the receiving Router; it does not deregister downstream workers or stop remote processes. Locally managed workers are excluded.++Set an independent management key to enable the endpoints. Without a key, these paths return 404. `--admin-api-key` overrides `MINERU_ROUTER_ADMIN_API_KEY` and also works with `--reload`. This key is separate from parsing credentials and is never forwarded downstream; parsing authentication is unchanged.++```bash+# An empty pool can start: health returns 503 until a healthy upstream is added,+# while the management endpoints remain available.+export MINERU_ROUTER_ADMIN_API_KEY='replace-with-a-management-secret'+mineru-kit router --host 127.0.0.1 --port 8002 --local-gpus none++# Set the same key in another terminal before running these requests.+curl -X POST http://127.0.0.1:8002/v1/router/workers \+ -H "Authorization: Bearer $MINERU_ROUTER_ADMIN_API_KEY" \+ -H 'Content-Type: application/json' -d '{"url":"http://worker-a:8000"}'+curl http://127.0.0.1:8002/v1/router/workers \+ -H "Authorization: Bearer $MINERU_ROUTER_ADMIN_API_KEY"+# Use the worker_id returned by registration or listing.+curl -i -X DELETE http://127.0.0.1:8002/v1/router/workers/remote-1 \+ -H "Authorization: Bearer $MINERU_ROUTER_ADMIN_API_KEY"+```++| Endpoint | Response and behavior |+|---|---|+| `GET /v1/router/workers` | `200`, with `{"object":"list","data":[...]}` containing both startup-configured and dynamically registered remote nodes. |+| `POST /v1/router/workers` | Accepts only `url`. Returns `201` for a new URL or `200` for an existing normalized URL. Re-registration preserves its ID and references and resumes a draining node. |+| `DELETE /v1/router/workers/{worker_id}` | Returns `204` after removal if no references remain, or `202` with a draining snapshot otherwise. Unknown and locally managed nodes return `404 worker_not_found`. |++Each node snapshot contains `worker_id`, `url`, `state` (`active/draining`), `healthy`, `tiers`, `active_jobs`, and `retained_routes`. URLs must be HTTP/HTTPS base URLs; path prefixes are supported, while embedded credentials, query strings, and fragments are rejected. Invalid input returns `400 invalid_request`; missing or incorrect management credentials return `401 invalid_api_key`. Errors use `error.code/message` and optional `param`.++Registration performs an initial health and capability probe. An unreachable node or one returning invalid tier data remains registered with `healthy=false` and is retried by the existing periodic checks. Invalid capabilities do not replace the last valid snapshot or break management listing. Only healthy, active nodes participate in new scheduling and health/models/tiers aggregation. A successful health check does not undo draining.++Deregistration stops new reservations immediately. Existing reservations, upload handoffs, queries, cancellation, result leases, and downloads continue. Removal waits for execution reservations, requests/streams, execution and file routes, and leases to be released; `active_jobs=0` alone does not mean the remote service can stop. Results default to 24-hour retention, so a node may remain draining during that period. The five-minute collection loop removes eligible nodes; repeating DELETE also checks eligibility. With `--retention-seconds 0`, permanent result references can prevent removal indefinitely. There is no force-delete operation.++Files that have not yet been dispatched use the remaining eligible nodes. If none qualify, they fail with the existing `quality_tier_unavailable` error; no implicit tier downgrade or new waiting queue is introduced. Removing the last node is allowed. Absence from this Router's list only means this Router has released its references; the downstream service may still serve other callers.++For a two-level deployment, configure the outer Router with inner Router URLs:++```bash+mineru-kit router --local-gpus none \+ --upstream-url http://router-a:8002 \+ --upstream-url http://router-b:8002+```++The same POST endpoint can add Router C at runtime. Use independent worker pools below each inner Router and an acyclic topology. Global load accounting and cross-Router task takeover are not provided.++**Dynamic membership lasts only for the current process.** Registration and deregistration do not modify startup configuration. Restarting restores the `--upstream-url` list; dynamic nodes must be registered again, and worker IDs are not stable across restarts. The outer Router remains a single point of failure, and existing tasks are not guaranteed to recover after any Router restarts.+## States and cancellation| File state | Meaning |diff --git a/docs/next/api/README.md b/docs/next/api/README.mdindex 9316a818..f690aec0 100644--- a/docs/next/api/README.md+++ b/docs/next/api/README.md@@ -13,6 +13,8 @@[4.1 迁移说明](https://opendatalab.github.io/MinerU/zh/reference/migration_4_1/)维护;英文版本在正式用户文档中。本目录保留能力发现、Files、错误与部署边界说明。+Router 可选启用独立认证的[远程 upstream 管理接口](router.md),用于运行期注册、安全注销及级联部署;它不属于普通解析 API 的必选接口。+## 资源模型| 资源 | 含义 |diff --git a/docs/next/api/health-models.md b/docs/next/api/health-models.mdindex 5847c1b4..41e75d38 100644--- a/docs/next/api/health-models.md+++ b/docs/next/api/health-models.md@@ -2,6 +2,7 @@实现:`mineru/parser/api_server.py`、`mineru/kit/router/app.py`。这些发现接口不要求 API Key,但仍可能因预加载失败或无健康 worker 返回 503。+Router 仅聚合健康且管理状态为 active 的节点;draining 节点仍可服务已有任务,但不再贡献对外发现能力。参见 [upstream 管理](router.md)。它们描述当前服务,不能推断公网 SaaS 的套餐或部署版本。## GET `/v1/health`diff --git a/docs/next/api/router.md b/docs/next/api/router.mdnew file mode 100644index 00000000..a6e6f625--- /dev/null+++ b/docs/next/api/router.md@@ -0,0 +1,26 @@+# Router 远程 upstream 管理++实现:`mineru/kit/router/admin.py`、`workers.py`、`resources.py`。该接口是 Router 的可选部署管理能力,不属于普通 API Server 或 SaaS 必须实现的 Tasks 契约。++公开请求字段、响应和 curl/级联示例维护在[中文 Tasks 文档](../../zh/usage/tasks.md#动态-upstream-管理与级联)与[英文 Tasks 文档](../../en/usage/tasks.md#dynamic-router-upstreams-and-cascading)。++## 接口与配置++| 接口 | 语义 |+|---|---|+| `GET /v1/router/workers` | 列出本层启动配置和动态注册的远程节点。 |+| `POST /v1/router/workers` | 按规范化 URL 幂等登记并探测;重复登记恢复 draining 节点。 |+| `DELETE /v1/router/workers/{worker_id}` | 停止新预占;无引用返回 204,等待引用释放返回 202。 |++`--admin-api-key` 优先于 `MINERU_ROUTER_ADMIN_API_KEY`;Python 工厂通过 `RouterSettings(admin_api_key=...)` 配置。reload 工厂恢复相同凭证。未配置时不挂载管理路由,启用后所有操作要求独立 Bearer Key。管理凭证不参与下层探测或解析认证。++## 生命周期边界++- 管理状态 `active/draining` 与健康状态独立。调度、预检及 health/models/tiers 仅聚合健康的 active 节点;按 ID 访问旧任务仍允许 draining 节点。+- 探测阶段校验 tier 数据后才整体发布能力快照;非法响应将节点标记为不健康,保留上一份合法快照,管理查询仍可使用。+- 登记、停止接单与执行预占在首次网络等待前完成,避免并发重复登记或注销后新增预占。已删除 ID 在当前进程内不复用。+- 注销必须等待活动执行、进行中的 HTTP 请求与流式响应、执行/文件路由、反向别名和租约释放。回收沿用既有结果保留期和五分钟扫描;健康检查不会重新激活 draining。+- 永久保留结果可以永久阻止节点删除;不提供强制删除。节点从本层移除不意味着可以关闭仍服务其他调用方的下层进程。+- 待上传或尚未分派的文件继续选择剩余节点;无可用节点使用原有 `quality_tier_unavailable` 失败语义,允许节点池清空后重新注册。+- 动态清单不落盘、不修改启动参数,重启恢复 `--upstream-url`;本地托管 worker 不在接口范围内。+- 级联只复用现有 V1 协议及逐跳结果租约,管理操作不向下传播。不增加全局负载统计、共享状态或故障接管;部署保持无环。diff --git a/docs/next/cli/mineru-kit.md b/docs/next/cli/mineru-kit.mdindex 263fe73e..5a0c4266 100644--- a/docs/next/cli/mineru-kit.md+++ b/docs/next/cli/mineru-kit.md@@ -88,10 +88,13 @@ mineru-api --host 127.0.0.1 --port 8000 --tier standard当前 `mineru-kit router` 已确定:-- 入口参数包括 `--host`、`--port`、`--reload`、`--upstream-url`、`--local-gpus`、`--worker-host`、`--worker-tier`、`--worker-concurrency`、`--preload-models`。+- 入口参数包括 `--host`、`--port`、`--reload`、`--upstream-url`、`--local-gpus`、`--worker-host`、`--worker-tier`、`--worker-concurrency`、`--preload-models`、`--admin-api-key`。- `--upstream-url` 可重复,用于接入已有 MinerU V1 API base URL。- `--local-gpus` 支持 `auto`、`none` 或 GPU CSV,例如 `0,1,2`。- 本地 worker 统一启动 `mineru-kit api-server`;`--worker-tier` 是包含较低档位的上限,可选 flash/basic/standard/advanced,默认 standard;`--worker-concurrency` 默认 `1`。- Router 提供 V1 health、models、tiers、usage、按 ID 操作 Files,以及 `/v1/tasks` JSON 创建、按 ID 查询/取消、逐文件上传和结果下载;不提供任务或文件列表。- Router 只接受显式声明的参数,不再透传未知模型引擎参数;需要额外调优时应部署独立 upstream。- tasks 与 files 的公共路由状态保存在当前进程内,Router 重启后不保证旧公共 ID 继续可用。+- `--admin-api-key` 优先于 `MINERU_ROUTER_ADMIN_API_KEY`,未配置时不启用管理接口;管理凭证不下传,解析认证保持原契约。+- `GET/POST /v1/router/workers` 与 `DELETE /v1/router/workers/{worker_id}` 仅管理本层远程 upstream,包括启动清单;支持接入下层 Router,不管理本地托管 worker。+- 注销进入 draining,停止新预占,保留已有任务、传输和结果租约;全部引用释放后移除。清单仅当前进程有效,重启恢复启动参数。完整接口与级联示例见 [Router 管理](../api/router.md)。diff --git a/docs/zh/usage/cli_tools.md b/docs/zh/usage/cli_tools.mdindex 166f85da..5dae78f2 100644--- a/docs/zh/usage/cli_tools.md+++ b/docs/zh/usage/cli_tools.md@@ -106,6 +106,8 @@ mineru-kit vlm-server --engine auto --port 30000`mineru-router` 保留为 `mineru-kit router` 的命令名别名。Router 只暴露 V1 API,并且只接受文档中声明的 worker 参数。+使用 `--admin-api-key` 或 `MINERU_ROUTER_ADMIN_API_KEY` 启用独立认证的 upstream 注册、查询和安全注销接口。动态清单仅当前进程有效;下层 Router 可作为 upstream 接入。[管理 API 与级联示例](./tasks.md#动态-upstream-管理与级联)。+## 环境变量- `MINERU_HOME`:MinerU 配置、缓存和文档库状态的根目录。diff --git a/docs/zh/usage/tasks.md b/docs/zh/usage/tasks.mdindex f0ed7631..117ae895 100644--- a/docs/zh/usage/tasks.md+++ b/docs/zh/usage/tasks.md@@ -646,6 +646,56 @@ Router 创建任务前使用候选 worker 的受保护 `/v1/usage` 校验认证不声明新 Tasks 能力的旧 worker 不参与 Tasks 调度,更新时应先更新 worker,再更新 Router 和客户端。Router 使用有界、可续租的内部结果租约保护 worker 产物,上传窗口不受结果租约影响;Router 丢失后租约会自行结束。已完成任务按 worker 并行续租,每轮网络等待最多 10 秒,避免失联 worker 阻塞其他 worker 的结果保留。+### 动态 upstream 管理与级联++Router 可以在运行期间注册、查询和安全注销远程 upstream。upstream 可以是独立的 V1 API Server,也可以是另一层 Router;操作只影响接收请求的这一层,不会注销下层的 worker 或关闭远程进程。本地托管 worker 不在管理接口范围内。++配置独立管理 Key 后启用接口,未配置时这些路径返回 404。`--admin-api-key` 优先于 `MINERU_ROUTER_ADMIN_API_KEY`,也支持 `--reload`。管理 Key 不等于解析 API Key,不会被转发给下层;解析请求继续使用原有认证。++```bash+# 空节点池也可启动;注册健康节点前 /v1/health 返回 503,管理接口仍可访问。+export MINERU_ROUTER_ADMIN_API_KEY='replace-with-a-management-secret'+mineru-kit router --host 127.0.0.1 --port 8002 --local-gpus none++# 在另一个终端设置相同管理 Key 后操作。+curl -X POST http://127.0.0.1:8002/v1/router/workers \+ -H "Authorization: Bearer $MINERU_ROUTER_ADMIN_API_KEY" \+ -H 'Content-Type: application/json' -d '{"url":"http://worker-a:8000"}'+curl http://127.0.0.1:8002/v1/router/workers \+ -H "Authorization: Bearer $MINERU_ROUTER_ADMIN_API_KEY"+# 使用注册或列表响应中的 worker_id。+curl -i -X DELETE http://127.0.0.1:8002/v1/router/workers/remote-1 \+ -H "Authorization: Bearer $MINERU_ROUTER_ADMIN_API_KEY"+```++| 接口 | 响应与行为 |+|---|---|+| `GET /v1/router/workers` | `200`,返回 `{"object":"list","data":[...]}`,包含启动参数和运行期注册的远程节点。 |+| `POST /v1/router/workers` | 请求只接受 `url`。新增返回 `201`,同一规范化 URL 重复注册返回 `200`;保留原 ID 和已有引用,draining 节点恢复接单。 |+| `DELETE /v1/router/workers/{worker_id}` | 无引用时删除并返回 `204`;仍有引用时返回 `202` 与 draining 快照;未知或本地托管节点返回 `404 worker_not_found`。 |++节点快照包含 `worker_id`、`url`、`state`(`active/draining`)、`healthy`、`tiers`、`active_jobs` 和 `retained_routes`。注册 URL 须为 HTTP/HTTPS base URL,允许路径前缀,禁止内嵌凭证、查询参数和 fragment。非法输入返回 `400 invalid_request`,缺失或错误管理凭证返回 `401 invalid_api_key`,错误均使用 `error.code/message` 和可选 `param`。++注册后执行一次健康与能力探测,暂时离线或返回非法 tier 数据的节点仍保留并返回 `healthy=false`,后续由周期检查重试。非法能力不替换上一份合法快照,也不影响管理列表查询。只有健康的 active 节点参与新任务调度及 health/models/tiers 能力聚合;健康恢复不会撤销 draining。++注销立即停止新预占,已有预占、上传交接、查询、取消、结果续租与下载继续执行。节点必须等执行预占、请求/流式传输、执行与文件路由及租约全部释放后才会移除;`active_jobs=0` 不能单独作为关闭远程服务的依据。结果默认保留 24 小时,节点可能在这段时间内保持 draining;后台复用五分钟扫描完成回收,也可重复 DELETE 检查。`--retention-seconds 0` 下长期结果引用可以一直阻止移除,无强制删除接口。++尚未分派的文件使用剩余可用节点,没有匹配节点时按原契约返回 `quality_tier_unavailable`,不自动降档或新增等待队列。允许注销最后一个节点。列表中节点消失只表示本层已释放引用;下层可能仍在为其他调用方服务。++两层部署时,顶层直接接入下层 Router 的地址:++```bash+mineru-kit router --local-gpus none \+ --upstream-url http://router-a:8002 \+ --upstream-url http://router-b:8002+```++也可通过上述 POST 动态加入 Router C。建议各下层管理独立 worker 池并保持无环拓扑;本次不提供全局负载统计或跨 Router 任务接管。++**动态清单仅当前进程有效。** 注册和注销不会修改启动配置;重启后恢复 `--upstream-url` 中的节点,动态节点须重新注册,worker ID 不能跨重启作为稳定标识。顶层仍有单点,任一 Router 重启后的旧任务恢复不在保证范围内。++### 协议迁移+创建请求仅接受 `files` 和 `file_name`,不兼容 `manifest`、`filename`;新旧字段混传同样返回 400。Tasks 响应及 Files 详情统一使用 `file_name`。任务和文件列表接口已移除;使用创建响应中的 `task_id` 和固定查询路由恢复任务,按 `files[].result_url` 下载。合法认证下 `GET /v1/tasks` 返回 405,`GET /v1/files` 返回 404。这是同一 `/v1` 下的破坏性变更:先停收请求并排空在途任务,再协调更新 worker、Router、SDK、file_parse 包装服务及其他客户端;SaaS 由部署方同步对齐。diff --git a/mineru/kit/router/admin.py b/mineru/kit/router/admin.pynew file mode 100644index 00000000..b0483b1d--- /dev/null+++ b/mineru/kit/router/admin.py@@ -0,0 +1,113 @@+"""当前 Router 进程的远程 upstream 管理接口,与解析凭证和下层管理操作隔离。"""++from __future__ import annotations++import secrets+from collections.abc import Callable+from typing import Literal++from fastapi import APIRouter, Depends, Response, Security+from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer+from pydantic import BaseModel, ConfigDict, field_validator++from ...types import Tier+from .proxy import RouterProxyError+from .resources import ResourceRegistry+from .workers import WorkerPool, WorkerState, normalize_base_url+++class RegisterWorkerRequest(BaseModel):+ """注册输入只接受 base URL,不接受下层凭证或能力覆盖。"""++ model_config = ConfigDict(extra="forbid")+ url: str++ @field_validator("url")+ @classmethod+ def validate_url(cls, value: str) -> str:+ """与启动配置共用 URL 校验和去重规则。"""+ return normalize_base_url(value)+++class WorkerResponse(BaseModel):+ """提供运维判断可用性和注销进度所需的最小节点快照。"""++ worker_id: str+ url: str+ state: Literal["active", "draining"]+ healthy: bool+ tiers: list[Tier]+ active_jobs: int+ retained_routes: int+++class WorkerListResponse(BaseModel):+ """返回本层远程节点,不递归展开下层 Router 的节点或本地托管进程。"""++ object: Literal["list"] = "list"+ data: list[WorkerResponse]+++def build_admin_router(+ pool: WorkerPool,+ registry: ResourceRegistry,+ *,+ api_key: str,+ collect_resources: Callable[[], None],+) -> APIRouter:+ """为显式启用管理凭证的 Router 构建独立认证及成员管理路由。"""+ bearer = HTTPBearer(auto_error=False, scheme_name="RouterAdmin", description="Independent Router management key")++ async def require_admin(credentials: HTTPAuthorizationCredentials | None = Security(bearer)) -> None:+ """只校验本层管理凭证,不调用或携带该凭证探测任何 upstream。"""+ if credentials is None or not secrets.compare_digest(credentials.credentials.encode(), api_key.encode()):+ raise RouterProxyError(401, "invalid_api_key", "Invalid Router management API key")++ router = APIRouter(prefix="/v1/router/workers", tags=["Router management"], dependencies=[Depends(require_admin)])++ def describe(worker: WorkerState) -> WorkerResponse:+ """只返回公共管理字段,避免序列化内部连接、启动设置和认证信息。"""+ return WorkerResponse(+ worker_id=worker.worker_id,+ url=worker.base_url,+ state=worker.state,+ healthy=worker.healthy,+ tiers=sorted(worker.tiers),+ active_jobs=worker.active_jobs,+ retained_routes=registry.worker_route_count(worker.worker_id),+ )++ @router.get("", response_model=WorkerListResponse)+ async def list_workers() -> WorkerListResponse:+ """列出启动参数与运行期注册的所有远程节点,包括 draining 和不健康节点。"""+ return WorkerListResponse(data=[describe(worker) for worker in pool.workers if worker.source == "remote"])++ @router.post("", response_model=WorkerResponse, status_code=201, responses={200: {"model": WorkerResponse}})+ async def register_worker(body: RegisterWorkerRequest, response: Response) -> WorkerResponse:+ """先同步登记或恢复节点,再用无管理凭证的现有探测更新其健康与能力。"""+ worker, created = pool.register_remote(body.url)+ response.status_code = 201 if created else 200+ await pool.refresh(worker)+ return describe(worker)++ @router.delete(+ "/{worker_id}",+ response_model=WorkerResponse,+ status_code=202,+ responses={204: {"description": "Upstream removed after all references were released"}},+ )+ async def unregister_worker(worker_id: str) -> WorkerResponse | Response:+ """停止新分配后检查全部引用;不取消任务、不缩短保留期、不管理下层节点。"""+ try:+ worker = pool.drain_remote(worker_id)+ except KeyError as exc:+ raise RouterProxyError(404, "worker_not_found", "Remote worker not found", "worker_id") from exc+ collect_resources()+ if worker not in pool.workers:+ return Response(status_code=204)+ return describe(worker)++ return router+++__all__ = ["RegisterWorkerRequest", "WorkerListResponse", "WorkerResponse", "build_admin_router"]diff --git a/mineru/kit/router/app.py b/mineru/kit/router/app.pyindex 446450cf..a011416f 100644--- a/mineru/kit/router/app.py+++ b/mineru/kit/router/app.py@@ -40,6 +40,7 @@ from .resources import ()from ...utils.retention import RETENTION_SCAN_INTERVAL_SECONDSfrom .workers import RouterSettings, WorkerPool, WorkerState+from .admin import build_admin_routerdef _caller_scope(request: Request) -> str:@@ -203,6 +204,13 @@ def create_app(on_local_worker_replaced=_invalidate_replaced_worker,)+ def collect_resources_once() -> None:+ """先按原保留契约回收路由,再移除无执行、传输或结果引用的 draining 节点。"""+ if resolved_settings.retention_seconds:+ registry.collect_expired(source_store)+ source_store.collect_expired()+ pool.remove_drained_workers(registry.worker_route_count)+@asynccontextmanagerasync def _lifespan(application: FastAPI) -> AsyncIterator[None]:"""启动 worker pool,并在应用关闭时释放全部网络和子进程资源。"""@@ -220,15 +228,13 @@ def create_app(await maintain_task_result_leases(registry, pool)async def collection_loop() -> None:- """每五分钟同步回收过期公共路由与无引用源文件缓存。"""+ """复用五分钟扫描回收路由和节点;永久保留模式仍可回收无引用节点。"""while True:await asyncio.sleep(RETENTION_SCAN_INTERVAL_SECONDS)- registry.collect_expired(source_store)- source_store.collect_expired()+ collect_resources_once()try:- if resolved_settings.retention_seconds:- collection_task = asyncio.create_task(collection_loop(), name="mineru-router-retention")+ collection_task = asyncio.create_task(collection_loop(), name="mineru-router-retention")await pool.start()task_store.start()lease_task = asyncio.create_task(task_lease_loop(), name="mineru-router-task-result-leases")@@ -272,6 +278,7 @@ def create_app(application.state.source_store = source_storeapplication.state.task_store = task_storeapplication.state.worker_pool = pool+ application.state.collect_resources_once = collect_resources_onceapplication.state.started_at = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())async def _reconcile_once_for_state() -> None:@@ -392,6 +399,15 @@ def create_app(install_task_error_handlers(application)application.include_router(build_task_router(task_backend))+ if resolved_settings.admin_api_key:+ application.include_router(+ build_admin_router(+ pool,+ registry,+ api_key=resolved_settings.admin_api_key,+ collect_resources=collect_resources_once,+ )+ )return applicationdiff --git a/mineru/kit/router/cli.py b/mineru/kit/router/cli.pyindex 6c003351..eaa29a3f 100644--- a/mineru/kit/router/cli.py+++ b/mineru/kit/router/cli.py@@ -40,6 +40,12 @@ def router_cmd(help=t("Terminal retention seconds (default 86400; 0 disables collection)"),),preload_models: bool = typer.Option(False, "--preload-models", help=t("Preload models in managed workers")),+ admin_api_key: str | None = typer.Option(+ None,+ "--admin-api-key",+ envvar="MINERU_ROUTER_ADMIN_API_KEY",+ help=t("Bearer key enabling Router upstream management APIs"),+ ),) -> None:"""启动只暴露 MinerU V1 API 的独立 Router 服务。"""if worker_tier not in SERVER_TIERS:@@ -61,6 +67,7 @@ def router_cmd(worker_concurrency=worker_concurrency,preload_models=preload_models,retention_seconds=retention_seconds,+ admin_api_key=admin_api_key,)except ValueError as exc:exit_with_message("invalid_request", str(exc))diff --git a/mineru/kit/router/proxy.py b/mineru/kit/router/proxy.pyindex 2485df5e..1b4f38d9 100644--- a/mineru/kit/router/proxy.py+++ b/mineru/kit/router/proxy.py@@ -7,13 +7,15 @@ import copyfrom collections.abc import AsyncIterator, Mappingfrom typing import Any+import anyioimport httpxfrom fastapi import Requestfrom fastapi.responses import JSONResponse, Response, StreamingResponse+from starlette.types import Receive, Scope, Sendfrom ...tasks.contracts import ErrorResponsefrom .resources import ResourceRegistry-from .workers import WorkerPool, WorkerState+from .workers import WorkerPool, WorkerRequest, WorkerState_HOP_BY_HOP_HEADERS = frozenset({@@ -91,6 +93,7 @@ async def request_upstream(if json_body is not None:outgoing_headers["content-type"] = "application/json"outgoing_headers.update(headers or {})+ reference = worker.acquire_request()try:response = await pool.client.request(method,@@ -111,6 +114,42 @@ async def request_upstream(raise RouterProxyError(504, "upstream_timeout", f"Upstream {worker.worker_id} timed out") from excexcept httpx.HTTPError as exc:raise RouterProxyError(502, "upstream_unavailable", f"Upstream {worker.worker_id} is unavailable") from exc+ finally:+ reference.release()+++class _UpstreamStreamingResponse(StreamingResponse):+ """把节点引用持有到 ASGI 响应结束,响应头发送失败或 body 未启动也能回收。"""++ def __init__(self, upstream: httpx.Response, reference: WorkerRequest) -> None:+ """接管已打开的 HTTP 流及其节点引用,避免下载过程中注销节点。"""+ self._upstream = upstream+ self._reference = reference+ super().__init__(+ self._body(),+ status_code=upstream.status_code,+ headers=response_headers(upstream.headers),+ media_type=upstream.headers.get("content-type"),+ )++ async def _body(self) -> AsyncIterator[bytes]:+ """兼容已缓冲的测试响应,并直接转发真实 upstream 原始字节流。"""+ if self._upstream.is_stream_consumed:+ yield self._upstream.content+ return+ async for chunk in self._upstream.aiter_raw():+ yield chunk++ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:+ """无论正常完成、断连或发送异常,都关闭 upstream 并归还节点引用。"""+ try:+ await super().__call__(scope, receive, send)+ finally:+ try:+ with anyio.CancelScope(shield=True):+ await self._upstream.aclose()+ finally:+ self._reference.release()async def stream_upstream(@@ -127,36 +166,29 @@ async def stream_upstream(f"{worker.base_url}{path}",headers=forwarded_headers(request),)+ reference = worker.acquire_request()+ upstream: httpx.Response | None = None+ transferred = Falsetry:upstream = await pool.client.send(upstream_request, stream=True)if upstream.status_code >= 400:- try:- await upstream.aread()- return passthrough_response(upstream)- finally:- await upstream.aclose()+ await upstream.aread()+ return passthrough_response(upstream)+ response = _UpstreamStreamingResponse(upstream, reference)+ transferred = True+ return responseexcept httpx.TimeoutException as exc:raise RouterProxyError(504, "upstream_timeout", f"Upstream {worker.worker_id} timed out") from excexcept httpx.HTTPError as exc:raise RouterProxyError(502, "upstream_unavailable", f"Upstream {worker.worker_id} is unavailable") from exc-- async def _body() -> AsyncIterator[bytes]:- """逐块读取 upstream body,并确保响应最终关闭。"""- try:- if upstream.is_stream_consumed:- yield upstream.content- return- async for chunk in upstream.aiter_raw():- yield chunk- finally:- await upstream.aclose()-- return StreamingResponse(- _body(),- status_code=upstream.status_code,- headers=response_headers(upstream.headers),- media_type=upstream.headers.get("content-type"),- )+ finally:+ if not transferred:+ try:+ if upstream is not None:+ with anyio.CancelScope(shield=True):+ await upstream.aclose()+ finally:+ reference.release()def passthrough_response(upstream: httpx.Response) -> Response:diff --git a/mineru/kit/router/resources.py b/mineru/kit/router/resources.pyindex d46bdfc5..cfbaacd1 100644--- a/mineru/kit/router/resources.py+++ b/mineru/kit/router/resources.py@@ -545,6 +545,14 @@ class ResourceRegistry:self._by_upstream.pop(key, None)return removed+ def worker_route_count(self, worker_id: str) -> int:+ """计算正反索引引用的不同路由,已过期但尚未回收或仍有租约的记录也阻止注销。"""+ route_ids = {+ id(route) for routes in self._by_public.values() for route in routes.values() if route.worker_id == worker_id+ }+ route_ids.update(id(route) for key, route in self._by_upstream.items() if key[2] == worker_id)+ return len(route_ids)+def usage(self, owner_scope: str) -> dict[str, int]:"""按调用方聚合存活与已移除任务的累计用量,到期隐藏不影响统计。"""totals = {"jobs_created": 0, "files_processed": 0, **self._retired_usage.get(owner_scope, {})}diff --git a/mineru/kit/router/tasks.py b/mineru/kit/router/tasks.pyindex c80814eb..23ea1abd 100644--- a/mineru/kit/router/tasks.py+++ b/mineru/kit/router/tasks.py@@ -97,12 +97,21 @@ def _worker_path(worker: WorkerState, url: str) -> str:"""按协议、主机及有效端口比较来源,端口不同也属于跨源。"""return item.scheme, item.hostname, item.port or (443 if item.scheme == "https" else 80)- prefix = base.path.rstrip("/")+ def normalized_path(path: str) -> str:+ """逐段规范化 UTF-8 与转义大小写,保留标识内部编码斜杠的路径段边界。"""+ return "/".join(quote(unquote(segment, errors="strict"), safe="") for segment in path.split("/"))+if origin(base) != origin(target) or target.username or target.password or target.query or target.fragment:raise_api_error(502, code="invalid_upstream_response", message="Invalid worker resource URL")- if prefix and not target.path.startswith(prefix + "/"):+ # URI 转义大小写和原始 Unicode 不影响同一前缀的身份;只规范化路径,不放宽来源校验。+ try:+ prefix = normalized_path(base.path).rstrip("/")+ target_path = normalized_path(target.path)+ except UnicodeDecodeError:+ raise_api_error(502, code="invalid_upstream_response", message="Invalid worker resource path")+ if prefix and not target_path.startswith(prefix + "/"):raise_api_error(502, code="invalid_upstream_response", message="Invalid worker resource path")- return target.path[len(prefix) :]+ return target_path[len(prefix) :]def _lost_execution(route: ExecutionRoute) -> TaskExecution:diff --git a/mineru/kit/router/workers.py b/mineru/kit/router/workers.pyindex 36caf6ea..f12e3895 100644--- a/mineru/kit/router/workers.py+++ b/mineru/kit/router/workers.py@@ -15,13 +15,14 @@ from collections.abc import Awaitable, Callablefrom dataclasses import dataclass, fieldfrom pathlib import Pathfrom typing import Any, Literal, cast+from urllib.parse import urlsplitimport httpxfrom loguru import loggerfrom ...model.runtime.device import get_devicefrom ...parser.process_control import ManagedProcessControl-from ...types import SERVER_TIERS, ServerTier, Tier+from ...types import SERVER_TIERS, TIERS, ServerTier, Tierfrom ...utils.stdio import utf8_subprocess_envfrom ...utils.retention import resolve_retention_seconds@@ -31,11 +32,38 @@ DEFAULT_WORKER_SHUTDOWN_TIMEOUT_SECONDS = 10.0def normalize_base_url(url: str) -> str:- """规范化 upstream base URL 并拒绝空地址。"""- normalized = str(url).strip().rstrip("/")- if not normalized:- raise ValueError("Router upstream URL must not be empty")- return normalized+ """统一校验并规范化 HTTP base URL,保留路径前缀且禁止凭证、查询和片段。"""+ normalized = url.strip()+ try:+ parts = urlsplit(normalized)+ parsed = httpx.URL(normalized)+ valid = (+ parsed.scheme in {"http", "https"}+ and bool(parsed.host)+ and parts.username is None+ and parts.password is None+ and "?" not in normalized+ and "#" not in normalized+ and not any(char.isspace() or ord(char) < 32 for char in normalized)+ and "\\" not in normalized+ )+ except (ValueError, httpx.InvalidURL):+ valid = False+ if not valid:+ raise ValueError("Upstream must be an HTTP(S) base URL without credentials, query, or fragment")+ return str(parsed).rstrip("/")+++def _read_tier_metadata(payload: Any) -> dict[Tier, dict[str, Any]]:+ """在发布探测结果前校验全部 tier,拒绝未知 ID 和畸形条目,保留合法附加元数据。"""+ if not isinstance(payload, dict) or not isinstance(payload.get("data"), list):+ raise ValueError("Invalid upstream tiers response: expected a data array")+ metadata: dict[Tier, dict[str, Any]] = {}+ for item in payload["data"]:+ if not isinstance(item, dict) or item.get("id") not in TIERS:+ raise ValueError("Invalid upstream tiers response: unsupported tier ID")+ metadata[cast(Tier, item["id"])] = dict(item)+ return metadatadef worker_connect_host(bind_host: str) -> str:@@ -137,10 +165,13 @@ class RouterSettings:preload_models: bool = Falseretention_seconds: int | None = Noneworker_refresh_interval_seconds: float = DEFAULT_WORKER_REFRESH_INTERVAL_SECONDS+ admin_api_key: str | None = field(default=None, repr=False)def __post_init__(self) -> None:"""校验 worker tier、并发数和刷新间隔。"""object.__setattr__(self, "retention_seconds", resolve_retention_seconds(self.retention_seconds))+ object.__setattr__(self, "upstream_urls", tuple(dict.fromkeys(normalize_base_url(url) for url in self.upstream_urls)))+ object.__setattr__(self, "admin_api_key", (self.admin_api_key or "").strip() or None)if self.worker_tier not in SERVER_TIERS:raise ValueError(f"Unsupported worker tier: {self.worker_tier}")if self.worker_concurrency <= 0:@@ -166,6 +197,7 @@ class RouterSettings:worker_tier=cast(ServerTier, tier),worker_concurrency=int(os.getenv("MINERU_ROUTER_WORKER_CONCURRENCY", "1")),preload_models=os.getenv("MINERU_ROUTER_PRELOAD_MODELS", "0").lower() in {"1", "true", "yes"},+ admin_api_key=os.getenv("MINERU_ROUTER_ADMIN_API_KEY"),)def apply_to_env(self) -> None:@@ -177,6 +209,10 @@ class RouterSettings:os.environ["MINERU_ROUTER_WORKER_TIER"] = self.worker_tieros.environ["MINERU_ROUTER_WORKER_CONCURRENCY"] = str(self.worker_concurrency)os.environ["MINERU_ROUTER_PRELOAD_MODELS"] = "1" if self.preload_models else "0"+ if self.admin_api_key:+ os.environ["MINERU_ROUTER_ADMIN_API_KEY"] = self.admin_api_key+ else:+ os.environ.pop("MINERU_ROUTER_ADMIN_API_KEY", None)@dataclass@@ -322,12 +358,33 @@ class WorkerState:restart_failures: int = 0retry_at: float = 0.0refresh_lock: asyncio.Lock = field(default_factory=asyncio.Lock)+ state: Literal["active", "draining"] = "active"+ active_requests: int = 0@propertydef tiers(self) -> set[Tier]:"""返回当前 generation 保留的 tier 标识集合。"""return set(self.tier_metadata)+ def acquire_request(self) -> WorkerRequest:+ """在首次网络等待前保护节点,普通请求及流式响应结束后归还引用。"""+ self.active_requests += 1+ return WorkerRequest(self)+++@dataclass+class WorkerRequest:+ """持有一个请求期间的节点引用,与执行预占和结果租约分别计数。"""++ worker: WorkerState+ released: bool = False++ def release(self) -> None:+ """幂等释放请求引用,取消或响应失败也不能遗留 draining 阻塞。"""+ if not self.released:+ self.released = True+ self.worker.active_requests -= 1+@dataclassclass JobReservation:@@ -360,6 +417,8 @@ class WorkerPool:self.client = httpx.AsyncClient(timeout=httpx.Timeout(60.0, connect=10.0), transport=transport)self._on_local_worker_replaced = on_local_worker_replacedself._workers: dict[str, WorkerState] = {}+ self._remote_urls: dict[str, WorkerState] = {}+ self._next_remote_id = 1self._monitor_task: asyncio.Task[None] | None = Noneself._built = Falseself._selection_cursors: dict[str, str] = {}@@ -368,12 +427,8 @@ class WorkerPool:"""在 lifespan 启动阶段解析设备并创建 remote/local worker 状态。"""if self._built:return- for index, url in enumerate(- dict.fromkeys(normalize_base_url(item) for item in self.settings.upstream_urls),- start=1,- ):- worker_id = f"remote-{index}"- self._workers[worker_id] = WorkerState(worker_id=worker_id, base_url=url, source="remote")+ for url in self.settings.upstream_urls:+ self.register_remote(url)for index, gpu in enumerate(parse_local_gpus(self.settings.local_gpus), start=1):worker_id = f"local-{gpu if gpu is not None else 'cpu'}-{index}"local_worker = ManagedLocalWorker(worker_id, self.settings.worker_host, gpu, self.settings)@@ -395,6 +450,41 @@ class WorkerPool:"""按稳定 worker 标识读取状态。"""return self._workers[worker_id]+ def register_remote(self, url: str) -> tuple[WorkerState, bool]:+ """同步登记或恢复远程节点;首次 await 前完成去重,删除后不复用 ID。"""+ normalized = normalize_base_url(url)+ existing = self._remote_urls.get(normalized)+ if existing is not None:+ existing.state = "active"+ return existing, False+ worker_id = f"remote-{self._next_remote_id}"+ self._next_remote_id += 1+ worker = WorkerState(worker_id=worker_id, base_url=normalized, source="remote")+ self._workers[worker_id] = worker+ self._remote_urls[normalized] = worker+ return worker, True++ def drain_remote(self, worker_id: str) -> WorkerState:+ """同步停止远程节点的新预占,保留已有执行和资源访问使用的对象。"""+ worker = self.get(worker_id)+ if worker.source != "remote":+ raise KeyError(worker_id)+ worker.state = "draining"+ return worker++ def remove_drained_workers(self, route_count: Callable[[str], int]) -> None:+ """仅删除无预占、请求和任何路由引用的远程 draining 节点。"""+ for worker in self.workers:+ if (+ worker.source == "remote"+ and worker.state == "draining"+ and worker.active_jobs == 0+ and worker.active_requests == 0+ and route_count(worker.worker_id) == 0+ ):+ del self._workers[worker.worker_id]+ del self._remote_urls[worker.base_url]+async def start(self) -> None:"""启动本地 worker、刷新全部能力并开启后台健康检查。"""self._build_workers()@@ -445,8 +535,14 @@ class WorkerPool:async def refresh(self, worker: WorkerState) -> None:"""序列化单 worker 的健康刷新,确保同一 generation 只替换一次。"""- async with worker.refresh_lock:- await self._refresh_worker(worker)+ if self._workers.get(worker.worker_id) is not worker:+ return+ reference = worker.acquire_request()+ try:+ async with worker.refresh_lock:+ await self._refresh_worker(worker)+ finally:+ reference.release()async def _refresh_worker(self, worker: WorkerState) -> None:"""复用启停超时重建死亡进程,启动失败按 5/10/20/30 秒退避。"""@@ -493,11 +589,13 @@ class WorkerPool:tiers.raise_for_status()models.raise_for_status()health_payload, tier_payload, model_payload = health.json(), tiers.json(), models.json()- worker.features = dict(health_payload.get("features") or {})- worker.tier_metadata = {- cast(Tier, item["id"]): dict(item) for item in tier_payload.get("data", []) if item.get("id")- }- worker.models = [dict(item) for item in model_payload.get("data", [])]+ # 先完整读取并校验,失败时保留上一份合法快照,避免局部更新污染管理响应。+ tier_metadata = _read_tier_metadata(tier_payload)+ features = dict(health_payload.get("features") or {})+ model_metadata = [dict(item) for item in model_payload.get("data", [])]+ worker.features = features+ worker.tier_metadata = tier_metadata+ worker.models = model_metadataworker.healthy = health_payload.get("status") == "ok"worker.last_error = None if worker.healthy else health.textexcept Exception as exc:@@ -505,8 +603,8 @@ class WorkerPool:worker.last_error = str(exc)def healthy_workers(self) -> list[WorkerState]:- """返回当前健康且具有 base URL 的 worker。"""- return [worker for worker in self.workers if worker.healthy and worker.base_url]+ """仅返回允许新任务接入的健康节点;旧任务仍可通过 get 访问 draining。"""+ return [worker for worker in self.workers if worker.state == "active" and worker.healthy and worker.base_url]def available_tiers(self) -> set[Tier]:"""聚合全部健康 worker 暴露的请求 tier。"""@@ -588,6 +686,7 @@ __all__ = ["ManagedLocalWorker","RouterSettings","WorkerPool",+ "WorkerRequest","WorkerState","normalize_base_url","parse_local_gpus",diff --git a/mineru/tasks/http.py b/mineru/tasks/http.pyindex 480543f2..5cefd4e1 100644--- a/mineru/tasks/http.py+++ b/mineru/tasks/http.py@@ -80,8 +80,9 @@ def _retention_lease(request: Request) -> int | None:def _url(request: Request, path: str) -> str:- """生成包含反向代理 root_path 的本部署地址,客户端无需自行拼接路径。"""- return str(request.base_url).rstrip("/") + path+ """编码当前挂载的已解码 root_path;资源路径本身已编码,不能再次编码其中的标识。"""+ root_path = quote(request.scope.get("root_path", "").rstrip("/"), safe="/")+ return str(request.base_url.replace(path=root_path + path))def _upload_descriptor(record: TaskRecord, index: int, request: Request) -> TaskUploadLink | None:diff --git a/tests/unittest/test_api_error_contract.py b/tests/unittest/test_api_error_contract.pyindex ea878e2c..ba405435 100644--- a/tests/unittest/test_api_error_contract.py+++ b/tests/unittest/test_api_error_contract.py@@ -19,6 +19,7 @@ from test_task_tiers import _deploymentfrom mineru.errors import MineruErrorfrom mineru.kit.router.proxy import RouterProxyError, passthrough_response, stream_upstream+from mineru.kit.router.workers import WorkerStatefrom mineru.kit.router.tasks import _checked_jsonfrom mineru.parser import api_serverfrom mineru.parser.base import ParseResult@@ -106,7 +107,7 @@ def test_download_proxy_validates_errors_and_closes_upstream(outcome: str) -> Notransport = httpx.MockTransport(lambda _: httpx.Response(404, stream=ErrorStream()))async with httpx.AsyncClient(transport=transport) as client:pool = SimpleNamespace(client=client)- worker = SimpleNamespace(base_url="http://worker", worker_id="worker")+ worker = WorkerState(worker_id="worker", base_url="http://worker", source="remote")request = Request({"type": "http", "headers": []})if outcome != "valid":with pytest.raises(RouterProxyError) as error:@@ -125,6 +126,7 @@ def test_download_proxy_validates_errors_and_closes_upstream(outcome: str) -> Noassert response.status_code == 404assert json.loads(response.body) == {"error": {"code": "file_not_found", "message": "Expired"}}assert closed == [True]+ assert worker.active_requests == 0asyncio.run(scenario())diff --git a/tests/unittest/test_router_admin.py b/tests/unittest/test_router_admin.pynew file mode 100644index 00000000..d29326dd--- /dev/null+++ b/tests/unittest/test_router_admin.py@@ -0,0 +1,627 @@+"""验证远程 upstream 动态注册、独立认证和不破坏既有任务的安全注销。"""++from __future__ import annotations++import asyncio+import time+from collections.abc import AsyncIterator+from contextlib import AsyncExitStack, asynccontextmanager+from functools import partial+from pathlib import Path+from typing import Any+from urllib.parse import unquote++import httpx+import pytest+from fastapi import FastAPI, Request+from fastapi.testclient import TestClient+from starlette.types import Message++from mineru.kit.router import RouterSettings, create_app+from mineru.kit.router.proxy import RouterProxyError, request_upstream, stream_upstream+from mineru.kit.router.tasks import _worker_path, maintain_task_result_leases+from mineru.kit.router.workers import WorkerState, normalize_base_url+from mineru.parser import api_server+from mineru.tasks.errors import ApiServerError+from test_router_cascade import HTML, _WorkerTransport, _check_download, _submit, _terminal+from test_v1_router import _FakeV1Upstream++ADMIN = {"authorization": "Bearer management-only-secret"}+WORKERS = "/v1/router/workers"+++def _settings(*urls: str, retention: int = 86400, enabled: bool = True) -> RouterSettings:+ """构造禁用本地模型与周期探测的管理测试配置,固定保留期避免环境变量干扰。"""+ return RouterSettings(+ upstream_urls=urls,+ local_gpus="none",+ worker_refresh_interval_seconds=0,+ retention_seconds=retention,+ admin_api_key="management-only-secret" if enabled else None,+ )+++def _fake_app(settings: RouterSettings) -> tuple[FastAPI, list[httpx.Request]]:+ """只模拟无认证的能力端点,记录管理凭证是否意外传给下层。"""+ requests: list[httpx.Request] = []++ async def handle(request: httpx.Request) -> httpx.Response:+ """所有测试地址均返回相同的合法能力,不替换 Router 自身的行为。"""+ requests.append(request)+ return await _FakeV1Upstream("worker", ("flash",)).handle(request)++ return create_app(settings, transport=httpx.MockTransport(handle)), requests+++@asynccontextmanager+async def _client(app: FastAPI) -> AsyncIterator[httpx.AsyncClient]:+ """在一个事件循环中运行完整 Router 生命周期,便于控制请求之间的竞争窗口。"""+ async with app.router.lifespan_context(app):+ async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://router") as client:+ yield client++[email protected]("enabled", [False, True])+def test_management_authentication_and_openapi(enabled: bool) -> None:+ """未配置 Key 时接口不存在;启用后所有管理操作均使用独立认证且不向下传递。"""+ app, requests = _fake_app(_settings(enabled=enabled))+ with TestClient(app) as client:+ for method, path in (("GET", WORKERS), ("POST", WORKERS), ("DELETE", f"{WORKERS}/remote-1")):+ for headers in ({}, {"authorization": "Bearer parse-key"}):+ response = client.request(method, path, headers=headers, json={"url": "http://worker"})+ assert response.status_code == (401 if enabled else 404), response.text+ if enabled:+ assert response.json()["error"]["code"] == "invalid_api_key"+ schema = client.get("/openapi.json").json()+ assert (WORKERS in schema["paths"]) is enabled+ if enabled:+ response = client.post(WORKERS, json={"url": "http://worker"}, headers=ADMIN)+ assert response.status_code == 201, response.text+ assert response.json()["healthy"] is True+ assert requests and all("authorization" not in request.headers for request in requests)+ assert "management-only-secret" not in response.text + str(schema) + repr(app.state.settings)+ else:+ assert client.get(WORKERS, headers=ADMIN).status_code == 404+ assert not requests+++ "body",+ [+ {},+ {"url": 123},+ {"url": ""},+ {"url": "ftp://worker"},+ {"url": "//worker"},+ {"url": "http://user:secret@worker"},+ {"url": "http://@worker"},+ {"url": "http://worker?"},+ {"url": "http://worker#"},+ {"url": "http://worker?x=1"},+ {"url": "http://worker/#part"},+ {"url": "http://work er"},+ {"url": "http://worker:bad"},+ {"url": "http://worker/\\path"},+ {"url": "http://worker", "healthy": True},+ ],+)+def test_registration_rejects_invalid_input_before_probe(body: dict[str, Any]) -> None:+ """非法地址和额外字段必须在节点入池及网络探测前拒绝。"""+ app, requests = _fake_app(_settings())+ with TestClient(app) as client:+ response = client.post(WORKERS, json=body, headers=ADMIN)+ assert response.status_code == 400, response.text+ assert response.json()["error"]["code"] == "invalid_request"+ assert not app.state.worker_pool.workers+ assert not requests+++def test_registration_drain_resume_and_non_reused_ids() -> None:+ """启动节点与动态节点共用身份;draining 停止新调度,重复注册恢复,最终删除不复用 ID。"""+ app, _ = _fake_app(_settings("HTTP://WORKER:80/"))+ with TestClient(app) as client:+ first = client.post(WORKERS, json={"url": "http://worker/"}, headers=ADMIN)+ assert first.status_code == 200+ worker_id = first.json()["worker_id"]+ pool = app.state.worker_pool+ reservation = client.portal.call(partial(pool.reserve_job, tier="flash", required_sources={"upload"}))+ assert reservation is not None+ drained = client.delete(f"{WORKERS}/{worker_id}", headers=ADMIN)+ assert drained.status_code == 202+ assert drained.json()["active_jobs"] == 1+ client.portal.call(pool.refresh, reservation.worker)+ assert reservation.worker.healthy+ assert reservation.worker.state == "draining"+ assert client.get("/v1/health").status_code == 503+ assert client.get("/v1/tiers").status_code == 503+ assert client.get("/v1/models").status_code == 503+ assert pool.select(tier="flash") is None+ restored = client.post(WORKERS, json={"url": "http://worker"}, headers=ADMIN)+ assert restored.status_code == 200+ assert restored.json()["worker_id"] == worker_id+ assert restored.json()["active_jobs"] == 1+ client.portal.call(reservation.release)+ assert client.delete(f"{WORKERS}/{worker_id}", headers=ADMIN).status_code == 204+ assert client.delete(f"{WORKERS}/{worker_id}", headers=ADMIN).status_code == 404+ assert client.get(WORKERS, headers=ADMIN).json()["data"] == []+ replacement = client.post(WORKERS, json={"url": "http://worker"}, headers=ADMIN)+ assert replacement.status_code == 201+ assert replacement.json()["worker_id"] != worker_id+++def test_restarts_restore_only_startup_upstreams_and_exclude_managed_workers() -> None:+ """运行期注销不改启动配置;本地托管节点不能经管理接口列出或删除。"""+ settings = _settings("http://configured")+ app, _ = _fake_app(settings)+ with TestClient(app) as client:+ assert client.delete(f"{WORKERS}/remote-1", headers=ADMIN).status_code == 204+ assert client.post(WORKERS, json={"url": "http://dynamic"}, headers=ADMIN).status_code == 201+ local = WorkerState("local-test", "http://local", "local")+ app.state.worker_pool._workers[local.worker_id] = local+ assert client.delete(f"{WORKERS}/{local.worker_id}", headers=ADMIN).status_code == 404+ assert [item["url"] for item in client.get(WORKERS, headers=ADMIN).json()["data"]] == ["http://dynamic"]+ restarted, _ = _fake_app(settings)+ with TestClient(restarted) as client:+ assert [item["url"] for item in client.get(WORKERS, headers=ADMIN).json()["data"]] == ["http://configured"]+++def test_concurrent_registration_and_drain_during_initial_probe() -> None:+ """探测等待期间重复注册只创建一个节点,并发注销不会被迟到的健康结果撤销。"""++ async def scenario() -> None:+ """固定两次注册都持有请求引用的窗口,再发起注销并放行探测。"""+ release = asyncio.Event()++ async def handle(request: httpx.Request) -> httpx.Response:+ """延迟所有探测请求以观察节点登记与 HTTP 完成之间的状态。"""+ await release.wait()+ return await _FakeV1Upstream("worker", ("flash",)).handle(request)++ app = create_app(_settings(), transport=httpx.MockTransport(handle))+ async with _client(app) as client:+ operations = [+ asyncio.create_task(client.post(WORKERS, json={"url": url}, headers=ADMIN))+ for url in ("http://worker", "http://worker/")+ ]+ try:+ async with asyncio.timeout(5):+ while not app.state.worker_pool.workers or app.state.worker_pool.workers[0].active_requests < 2:+ await asyncio.sleep(0.01)+ assert len(app.state.worker_pool.workers) == 1+ drained = await client.delete(f"{WORKERS}/remote-1", headers=ADMIN)+ assert drained.status_code == 202+ finally:+ release.set()+ responses = await asyncio.gather(*operations)+ assert sorted(response.status_code for response in responses) == [200, 201]+ assert all(response.json()["state"] == "draining" for response in responses)+ assert app.state.worker_pool.workers[0].active_requests == 0+ app.state.collect_resources_once()+ assert not app.state.worker_pool.workers++ asyncio.run(scenario())+++def test_failed_probe_recovers_without_reregistration() -> None:+ """注册暂时离线的节点仍保留记录,后续现有健康检查可恢复其可调度状态。"""++ async def scenario() -> None:+ """切换模拟 upstream 可达性,验证探测失败与管理状态互相独立。"""+ healthy = False++ async def handle(request: httpx.Request) -> httpx.Response:+ """离线时使用真实 httpx 连接异常,恢复后返回标准能力。"""+ if not healthy:+ raise httpx.ConnectError("offline", request=request)+ return await _FakeV1Upstream("worker", ("flash",)).handle(request)++ app = create_app(_settings(), transport=httpx.MockTransport(handle))+ async with _client(app) as client:+ response = await client.post(WORKERS, json={"url": "http://worker"}, headers=ADMIN)+ assert response.status_code == 201+ assert response.json()["healthy"] is False+ assert response.json()["state"] == "active"+ healthy = True+ await app.state.worker_pool.refresh_all()+ assert (await client.get("/v1/health")).status_code == 200++ asyncio.run(scenario())+++ "invalid_tiers",+ [+ {"data": [{"id": "unknown-tier"}]},+ {"data": [{"id": "flash"}, {"id": "unknown-tier"}]},+ {"data": [{"id": ["flash"]}]},+ {"data": [{}]},+ {"data": ["flash"]},+ {"data": None},+ {"data": {}},+ {},+ ],+)+def test_invalid_tiers_do_not_poison_management_or_replace_valid_capabilities(invalid_tiers: dict[str, Any]) -> None:+ """异常节点与错误刷新不能使整个管理列表返回 500,也不能发布未校验的部分能力。"""+ invalid = True++ async def handle(request: httpx.Request) -> httpx.Response:+ """仅破坏一个节点的 tier,并修改其 health 内容以检查失败刷新是否原子化。"""+ response = await _FakeV1Upstream("worker", ("flash",)).handle(request)+ if request.url.host == "bad" and invalid:+ if request.url.path == "/v1/tiers":+ return httpx.Response(200, json=invalid_tiers)+ if request.url.path == "/v1/health":+ return httpx.Response(200, json={"status": "ok", "features": {"sources": ["url"]}})+ return response++ app = create_app(_settings("http://good"), transport=httpx.MockTransport(handle))+ with TestClient(app, raise_server_exceptions=False) as client:+ created = client.post(WORKERS, json={"url": "http://bad"}, headers=ADMIN)+ assert created.status_code == 201, created.text+ assert created.json()["healthy"] is False+ assert created.json()["tiers"] == []+ worker = app.state.worker_pool.get(created.json()["worker_id"])+ assert worker.features == {} and worker.models == []+ listed = client.get(WORKERS, headers=ADMIN)+ assert listed.status_code == 200+ assert len(listed.json()["data"]) == 2+ assert [item.worker_id for item in app.state.worker_pool.capable_workers(tier="flash")] == ["remote-1"]+ invalid = False+ client.portal.call(app.state.worker_pool.refresh, worker)+ assert worker.healthy and worker.tiers == {"flash"}+ features = dict(worker.features)+ invalid = True+ client.portal.call(app.state.worker_pool.refresh, worker)+ assert worker.healthy is False+ assert worker.tiers == {"flash"} and worker.features == features+ assert client.get(WORKERS, headers=ADMIN).status_code == 200+ assert client.delete(f"{WORKERS}/{worker.worker_id}", headers=ADMIN).status_code == 204++[email protected]("cancel", [False, True])+def test_pending_request_blocks_removal_and_releases_on_failure(cancel: bool) -> None:+ """即使没有执行预占和路由,进行中的请求也阻止删除,异常或取消后可正常回收。"""++ async def scenario() -> None:+ """在请求已发出但尚未返回时注销节点,分别模拟断网与调用方取消。"""+ started, release = asyncio.Event(), asyncio.Event()++ async def handle(request: httpx.Request) -> httpx.Response:+ """能力探测正常,仅业务请求在指定窗口失败。"""+ if request.url.path == "/blocked":+ started.set()+ await release.wait()+ raise httpx.ConnectError("disconnected", request=request)+ return await _FakeV1Upstream("worker", ("flash",)).handle(request)++ app = create_app(_settings("http://worker"), transport=httpx.MockTransport(handle))+ async with _client(app) as client:+ pool = app.state.worker_pool+ worker = pool.workers[0]+ operation = asyncio.create_task(request_upstream(pool, worker, "GET", "/blocked"))+ await asyncio.wait_for(started.wait(), 5)+ assert (await client.delete(f"{WORKERS}/{worker.worker_id}", headers=ADMIN)).status_code == 202+ if cancel:+ operation.cancel()+ else:+ release.set()+ with pytest.raises(asyncio.CancelledError if cancel else RouterProxyError):+ await operation+ assert worker.active_requests == 0+ app.state.collect_resources_once()+ assert not pool.workers++ asyncio.run(scenario())++[email protected]("ending", ["complete", "cancel", "headers_fail"])+def test_stream_holds_worker_until_asgi_response_finishes(ending: str) -> None:+ """流式下载持有节点直到整个 ASGI 响应结束,含 body 未启动与中途取消两种失败。"""++ async def scenario() -> None:+ """使用真实 httpx 流配合 ASGI send 控制下载窗口,验证关闭与引用释放。"""+ body_started, release = asyncio.Event(), asyncio.Event()++ class Stream(httpx.AsyncByteStream):+ """记录流是否被关闭,并把正文传输停在第一块之后。"""++ closed = False++ async def __aiter__(self) -> AsyncIterator[bytes]:+ """先返回一块正文,再等待测试控制结束。"""+ yield b"first"+ body_started.set()+ await release.wait()+ yield b"last"++ async def aclose(self) -> None:+ """记录上游流清理,即使正文未消费也应调用。"""+ self.closed = True++ stream = Stream()++ async def handle(request: httpx.Request) -> httpx.Response:+ """下载端点使用阻塞字节流,其他请求使用标准能力响应。"""+ if request.url.path == "/download":+ return httpx.Response(200, stream=stream, request=request)+ return await _FakeV1Upstream("worker", ("flash",)).handle(request)++ async def receive() -> Message:+ """ASGI 2.4 的响应不读取请求体,保留合法 receive 协议。"""+ return {"type": "http.request", "body": b"", "more_body": False}++ async def send(message: Message) -> None:+ """模拟响应头发送失败,其他模式允许正文正常发送。"""+ if ending == "headers_fail" and message["type"] == "http.response.start":+ raise OSError("client disconnected before body")++ app = create_app(_settings("http://worker"), transport=httpx.MockTransport(handle))+ async with _client(app) as client:+ pool = app.state.worker_pool+ worker = pool.workers[0]+ scope = {"type": "http", "method": "GET", "headers": [], "asgi": {"spec_version": "2.4"}}+ response = await stream_upstream(pool, worker, "GET", "/download", request=Request(scope))+ assert (await client.delete(f"{WORKERS}/{worker.worker_id}", headers=ADMIN)).status_code == 202+ operation = asyncio.create_task(response(scope, receive, send))+ if ending != "headers_fail":+ await asyncio.wait_for(body_started.wait(), 5)+ app.state.collect_resources_once()+ assert pool.workers == [worker]+ if ending == "cancel":+ operation.cancel()+ else:+ release.set()+ if ending == "complete":+ await operation+ elif ending == "cancel":+ with pytest.raises(asyncio.CancelledError):+ await operation+ else:+ # Starlette 将 ASGI send 的 OSError 包装为 ClientDisconnect。+ from starlette.requests import ClientDisconnect++ with pytest.raises(ClientDisconnect):+ await operation+ assert stream.closed+ assert worker.active_requests == 0+ app.state.collect_resources_once()+ assert not pool.workers++ asyncio.run(scenario())+++def test_reverse_alias_and_route_lease_block_removal() -> None:+ """删除条件覆盖反向别名及过期但仍持有租约的路由,不能只统计正向归属。"""+ app, _ = _fake_app(_settings("http://worker"))+ with TestClient(app) as client:+ registry = app.state.registry+ route = registry.register_file(owner_scope="anonymous", worker_id="other", upstream_id="file-other")+ registry.alias_upstream(route, worker_id="remote-1", upstream_id="alias")+ route.payload = {"expires_at": time.time() - 10}+ route.lease.held = True+ route.lease.retain_until = time.time() - 1+ response = client.delete(f"{WORKERS}/remote-1", headers=ADMIN)+ assert response.status_code == 202+ assert response.json()["retained_routes"] == 1+ route.lease.held = False+ client.portal.call(app.state.collect_resources_once)+ assert client.get(WORKERS, headers=ADMIN).json()["data"] == []++[email protected]("retention", [0, 60])[email protected]("prefix", ["", "/service", "/模型", "/worker%20one", "/%e6%a8%a1%e5%9e%8b", "/50%25done"])+def test_dynamic_cascade_preserves_results_and_routes_new_tasks_elsewhere(tmp_path: Path, retention: int, prefix: str) -> None:+ """动态接入下层 Router,注销后旧结果仍可下载,新任务进入另一节点且注销不会向下传播。"""++ async def scenario() -> None:+ """完整启动两级 Router 和解析 API,检查真实 ZIP、逐跳租约与结果保留后移除。"""+ worker = api_server.create_app(upload_dir=str(tmp_path / "worker"), tiers=("flash",), retention_seconds=1)+ inner = create_app(_settings("http://worker", retention=retention), transport=httpx.ASGITransport(app=worker))+ entry = FastAPI()+ entry.mount(unquote(prefix) or "/", inner)+ apps = {"inner": entry, "worker": worker}++ async def dispatch(request: httpx.Request) -> httpx.Response:+ """顶层可连接下层 Router 或 API,转发仍由实际应用执行。"""+ return await httpx.ASGITransport(app=apps[request.url.host]).handle_async_request(request)++ outer = create_app(_settings(retention=retention), transport=httpx.MockTransport(dispatch))+ async with AsyncExitStack() as stack:+ for app in (worker, inner):+ await stack.enter_async_context(app.router.lifespan_context(app))+ client = await stack.enter_async_context(_client(outer))+ assert (await client.get("/v1/health")).status_code == 503+ registered = await client.post(WORKERS, json={"url": f"http://inner{prefix}"}, headers=ADMIN)+ assert registered.status_code == 201 and registered.json()["healthy"]+ task = await _submit(client)+ detail = (await client.get(f"/v1/tasks/{task['task_id']}")).json()+ uploaded = await client.put(detail["files"][0]["upload"]["url"], content=HTML)+ assert uploaded.status_code == 200+ result = await _terminal(client, task)+ await outer.state.task_store._records[task["task_id"]].operation+ assert result["status"] == "completed", str(result)+ await _check_download(client, result)+ worker_id = registered.json()["worker_id"]+ drained = await client.delete(f"{WORKERS}/{worker_id}", headers=ADMIN)+ assert drained.status_code == 202 and drained.json()["retained_routes"] >= 2+ outer.state.collect_resources_once()+ await maintain_task_result_leases(outer.state.registry, outer.state.worker_pool)+ await maintain_task_result_leases(inner.state.registry, inner.state.worker_pool)+ await _check_download(client, result)+ assert inner.state.worker_pool.workers[0].state == "active"+ assert (await client.post(WORKERS, json={"url": "http://worker"}, headers=ADMIN)).status_code == 201+ second = await _submit(client, filename="another.html")+ second_result = await _terminal(client, second)+ assert second_result["status"] == "completed"+ await _check_download(client, second_result)+ assert len(inner.state.task_store._records) == 1+ routes = [route for route in outer.state.registry.list("job") if route.worker_id == worker_id]+ assert len(routes) == 1+ if retention == 0:+ assert routes[0].lease.retain_until is None+ outer.state.collect_resources_once()+ assert outer.state.worker_pool.get(worker_id).state == "draining"+ else:+ # 模拟结果保留期到期,先正常释放下游租约再执行既有回收循环。+ routes[0].lease.retain_until = time.time() - 1+ routes[0].finalized_at = "2000-01-01T00:00:00Z"+ for route in outer.state.registry.list("file", include_expired=True):+ if route.worker_id == worker_id:+ route.payload["expires_at"] = time.time() - 1+ await maintain_task_result_leases(outer.state.registry, outer.state.worker_pool)+ outer.state.collect_resources_once()+ assert [item.worker_id for item in outer.state.worker_pool.workers] == ["remote-2"]++ asyncio.run(scenario())+++ ("base", "target", "expected"),+ [+ ("/模型", "/模型/v1/tasks/task/uploads/file", "/v1/tasks/task/uploads/file"),+ (+ "/模型",+ "/%e6%a8%a1%e5%9e%8b/v1/tasks/task%2fpart/uploads/file%20name",+ "/v1/tasks/task%2Fpart/uploads/file%20name",+ ),+ ("/worker%20one", "/worker one/v1/files/file/content", "/v1/files/file/content"),+ ("/50%25done", "/50%25done/v1/files/file/content", "/v1/files/file/content"),+ ],+)+def test_worker_resource_paths_normalize_equivalent_encodings(base: str, target: str, expected: str) -> None:+ """原始 Unicode、百分号转义和大小写差异可匹配,资源 ID 内部的编码斜杠保持不变。"""+ worker = WorkerState("worker", normalize_base_url("http://worker" + base), "remote")+ assert _worker_path(worker, "http://worker" + target) == expected+++ "target",+ [+ "http://other/模型/v1/files/file/content",+ "http://worker:8080/模型/v1/files/file/content",+ "https://worker/模型/v1/files/file/content",+ "http://user:pass@worker/模型/v1/files/file/content",+ "http://worker/模型-other/v1/files/file/content",+ "http://worker/模型%2Foutside/v1/files/file/content",+ "http://worker/模型/v1/files/file/content?token=secret",+ "http://worker/模型/v1/files/file/content#fragment",+ "http://worker/模型/%FF/v1/files/file/content",+ ],+)+def test_worker_resource_path_normalization_preserves_origin_and_prefix_checks(target: str) -> None:+ """编码兼容不放宽同源、端口或路径前缀限制,非法 UTF-8 仍返回结构化代理错误。"""+ worker = WorkerState("worker", normalize_base_url("http://worker/模型"), "remote")+ with pytest.raises(ApiServerError) as error:+ _worker_path(worker, target)+ assert error.value.status_code == 502+ assert error.value.error.code == "invalid_upstream_response"++[email protected]("cancel", [False, True])+def test_draining_preserves_reserved_submission_and_cancel(tmp_path: Path, cancel: bool) -> None:+ """注销前已预占但尚未返回下游任务 ID 的请求可继续提交,也可沿原链路取消。"""++ async def scenario() -> None:+ """阻塞真实 API 创建之前的窗口,注销后再选择完成或取消原任务。"""+ worker = api_server.create_app(upload_dir=str(tmp_path / "worker"), tiers=("flash",))+ transport = _WorkerTransport(worker, block_submission=True)+ router = create_app(_settings("http://worker"), transport=transport)+ async with worker.router.lifespan_context(worker), _client(router) as client:+ try:+ task = await _submit(client)+ path = f"/v1/tasks/{task['task_id']}"+ detail = (await client.get(path)).json()+ assert (await client.put(detail["files"][0]["upload"]["url"], content=HTML)).status_code == 200+ await asyncio.wait_for(transport.submission_started.wait(), 5)+ drained = await client.delete(f"{WORKERS}/remote-1", headers=ADMIN)+ assert drained.status_code == 202+ assert drained.json()["active_jobs"] == 1+ assert drained.json()["retained_routes"] == 0+ assert (await client.get(path)).status_code == 200+ if cancel:+ canceled = await client.post(path + "/cancel")+ assert canceled.status_code == 200 and canceled.json()["status"] == "canceled"+ transport.allow_submission.set()+ terminal = await _terminal(client, task)+ assert terminal["status"] == ("canceled" if cancel else "completed")+ await router.state.task_store._records[task["task_id"]].operation+ assert router.state.worker_pool.get("remote-1").active_jobs == 0+ if cancel:+ assert not worker.state.execution_store._executions+ router.state.collect_resources_once()+ assert not router.state.worker_pool.workers+ else:+ await _check_download(client, terminal)+ finally:+ transport.allow_submission.set()++ asyncio.run(scenario())+++def test_waiting_upload_fails_explicitly_when_last_worker_removed(tmp_path: Path) -> None:+ """允许移除最后一个空闲节点;未分派文件随后上传时明确失败,不改档或隐式排队。"""++ async def scenario() -> None:+ """先受理待上传任务再移除唯一节点,确认既有失败契约与管理入口仍有效。"""+ worker = api_server.create_app(upload_dir=str(tmp_path / "worker"), tiers=("flash",))+ router = create_app(_settings("http://worker"), transport=httpx.ASGITransport(app=worker))+ async with worker.router.lifespan_context(worker), _client(router) as client:+ task = await _submit(client)+ detail = (await client.get(f"/v1/tasks/{task['task_id']}")).json()+ assert (await client.delete(f"{WORKERS}/remote-1", headers=ADMIN)).status_code == 204+ assert (await client.put(detail["files"][0]["upload"]["url"], content=HTML)).status_code == 200+ result = await _terminal(client, task)+ assert result["status"] == "failed"+ assert result["files"][0]["error"]["code"] == "quality_tier_unavailable"+ assert (await client.get(WORKERS, headers=ADMIN)).status_code == 200++ asyncio.run(scenario())+++def test_background_collection_removes_drained_node_without_retention(monkeypatch: pytest.MonkeyPatch) -> None:+ """永久保留模式仍运行节点回收循环,无资源节点的请求引用释放后可以自动移除。"""+ from mineru.kit.router import app as router_app++ monkeypatch.setattr(router_app, "RETENTION_SCAN_INTERVAL_SECONDS", 0.01)++ async def scenario() -> None:+ """仅持有请求引用触发 draining,释放后等待实际后台循环删除节点。"""+ app, _ = _fake_app(_settings("http://worker", retention=0))+ async with _client(app) as client:+ worker = app.state.worker_pool.workers[0]+ reference = worker.acquire_request()+ assert (await client.delete(f"{WORKERS}/{worker.worker_id}", headers=ADMIN)).status_code == 202+ reference.release()+ async with asyncio.timeout(5):+ while app.state.worker_pool.workers:+ await asyncio.sleep(0.01)++ asyncio.run(scenario())+++def test_cli_admin_key_precedence_and_reload(monkeypatch: pytest.MonkeyPatch) -> None:+ """显式管理 Key 优先于环境变量,reload 工厂恢复相同设置且不改变运行期清单语义。"""+ from typer.testing import CliRunner++ from mineru.kit.main import app as cli_app+ from mineru.kit.router import cli++ monkeypatch.setenv("MINERU_ROUTER_ADMIN_API_KEY", "environment-key")+ seen: list[RouterSettings] = []++ def run(target: FastAPI | str, **kwargs: Any) -> None:+ """截获启动设置,reload 模式从真实环境恢复工厂配置。"""+ settings = RouterSettings.from_env() if isinstance(target, str) else target.state.settings+ seen.append(settings)+ assert kwargs["reload"] is True++ monkeypatch.setattr(cli.uvicorn, "run", run)+ runner = CliRunner()+ result = runner.invoke(cli_app, ["router", "--reload", "--local-gpus", "none"])+ assert result.exit_code == 0, result.output+ assert seen[-1].admin_api_key == "environment-key"+ result = runner.invoke(cli_app, ["router", "--reload", "--local-gpus", "none", "--admin-api-key", "explicit-key"])+ assert result.exit_code == 0, result.output+ assert seen[-1].admin_api_key == "explicit-key"+ assert "explicit-key" not in result.output + repr(seen[-1])