Skip to content

Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.

refactor: clarify task, document and model lifecycle boundaries

myhloliafda7189de1d
diff --git a/docs/next/architecture.md b/docs/next/architecture.md
index 7d6b151f..f8f1ed90 100644
--- a/docs/next/architecture.md
+++ b/docs/next/architecture.md
@@ -23,7 +23,9 @@ MinerU OCR/VLM/Hybrid ─────────────────→ Mod
PDF 只有 `auto` 模式调用共享 classify;显式 `txt` 不分类,显式 `ocr` 保留已有推理路线。原生分析只由 Flash 分支调用。DocVortex 不反向依赖 MinerU。
-DocVortex 内存对象和 JSON 使用中性 producer/extensions;MinerU 对外继续输出 schema 2.0,由兼容 codec 恢复产品元数据。所有持久化边界应使用 ParseResult 或明确的 MinerU codec,而不是直接写出 DocVortex 对象的原生封装。
+MinerU 直接消费 DocVortex 的共享 schema 2.0,`ParseResult` 持有 `MiddleJson`,产品档位和解析模式通过 `extensions["mineru"]` 保存并校验。旧协议适配仅保留在 Doclib 的历史缓存读取边界,不扩展到通用 SDK/API。
+
+启动配置类型可直接导入,不读取配置文件或校验运行环境。CLI 和默认调用路径通过 `get_config()` 在首次使用时装载并缓存进程默认配置;配置来源和文件路径查询共享同一份装载结果。需要独立配置时使用 `load_config()`,其返回值包含配置对象、来源和文件路径,且不替换进程默认配置。依赖 `MINERU_HOME` 的默认路径在构造配置对象时解析。Doclib 应用工厂复制传入的启动配置,日志默认级别也取自该应用配置。
## 1. 系统上下文
@@ -87,6 +89,7 @@ Routes
| Service | 职责 |
|---------|------|
| `ParseService` | 接收解析请求、同步入库、查缓存、创建解析任务 |
+| `ContentReadService` | 解析内容定位、读取计划、渐进 Markdown、图片读取与文件导出 |
| `SearchService` | 文件名和内容搜索 |
| `ConfigService` | KV 配置、watch 目录、parsing-rules、exclude 规则 |
| `CleanupService` | 孤儿文档、删除文件和解析缓存清理 |
@@ -108,6 +111,63 @@ Routes
所有后台组件都是 asyncio task,随 server 启停。
+内容读取由 `DoclibServer` 委托给 `ContentReadService`。读取服务只接收数据库与产物目录,
+不持有 HTTP 请求、应用全局状态或解析调度器;接口层保留参数绑定、响应和遥测。
+`doclib/core/parse_artifacts.py` 统一维护批次路径、缓存读写、有效性检查、页选择和覆盖计算,
+供解析服务、内容读取及后台 compaction 共用。历史协议适配仅从
+`doclib/core/middle_json.py:read_cached_middle_json()` 进入。
+
+### Tasks 服务边界
+
+本地解析服务和 Router 共用 Tasks 协议与生命周期,通过显式执行协议接入各自部署:
+
+| 模块 | 职责 |
+|------|------|
+| `tasks/contracts.py` | 请求、响应和公共字段校验;不依赖 HTTP 框架或解析实现 |
+| `tasks/execution.py` | 上传信息、具名单文件执行快照与 `TaskBackend` 协议 |
+| `tasks/service.py` | 上传校验、逐文件状态、任务聚合、取消与保留期 |
+| `tasks/http.py` | HTTP 输入、错误映射、响应序列化和路由 |
+| `parser/task_backend.py` | 将 Tasks 接入本地 `FileStore`、`ExecutionStore` 和解析执行 |
+| `kit/router/tasks.py` | 将 Tasks 接入 worker 选择、上游请求和资源租约 |
+
+共享 Tasks 模块不导入本地 API server。Router 使用共同协议和执行接口,不依赖本地
+`ApiTaskBackend`;本地作业并发、模型预热和执行清理仍由解析服务负责。
+SDK 与 `file_parse` 复用文件身份、上传描述及解析选项。SDK 保留原始响应中的扩展字段;
+`file_parse` 的单文件限制、创建回执恢复上下文和持久化状态仍由包装服务维护。
+
+每个执行只接收一个文件和可选上传引用,返回 `TaskExecution`,不再传递包含 `files[]`
+的内部 Job 字典。公开 `file_id` 与源文件存储引用分别保存;成功产物继续使用预留的公开身份。
+父 Task 负责多文件进度和 `partial` 聚合,本地执行负责单文件准备、推理、打包及清理。
+准备限流与解析信号量保持独立,取消后仍先排空同步工作,再释放名额和输入引用。
+公开用量字段 `jobs_created`、`max_concurrent_jobs` 保留原名称和按文件执行统计的含义。
+
+### 模型资源生命周期
+
+`model/runtime/ownership.py` 定义应用持有者及文档使用范围。每个解析服务应用拥有一个
+`RuntimeOwner`;PDF 分析的准备、推理、回填与清理都处于同一个文档范围内。
+应用关闭先停止接单并排空任务,然后拒绝新文档、等待已接纳文档退出,再释放各模型池。
+已接纳文档在退出前仍可完成惰性模型加载,不强制中断正在执行的同步推理。
+模型初始化也持有应用租约;关闭等待构造及失败清理结束,失效应用的迟到结果不会发布到缓存。
+
+小模型缓存显式记录应用、Hybrid 上下文和原子模型的依赖关系。共享上下文的
+`cached_property` 不需要为每次属性访问重新登记应用;最后一个持有者退出才撤销相应缓存引用。
+初始化失败会回滚未被其它调用方持有的依赖。独立 Python/CLI 调用继续固定进程级缓存。
+
+VLM 的 `RuntimeHandle` 在创建时绑定客户端、必要的串行锁和确定的后端关闭动作。
+关闭时不再遍历第三方对象猜测方法,也不向客户端动态附加资源字段;外部关闭动作在缓存锁外执行,
+同一缓存键的重新创建等待前一次关闭完成。同步执行开始后等待其退出,异步排队取消不会启动迟到推理。
+关闭回调不能阻塞等待正在初始化或关闭的运行时,重入时明确报错;异步关闭被取消后仍完成连接及引擎清理。
+
+### Gradio 转换编排
+
+`kit/gradio/conversion_service.py` 接收解析客户端、档位、输出目录、页数限制和执行槽,
+负责输入校验、排队、解析、素材持久化及预览准备,返回具名成功、失败或取消结果。
+`conversion_ui.py` 将这些结果转换为现有 Gradio 组件更新与浏览器回执,文件 URL 校验也位于该适配层。
+应用工厂负责控件和事件组装,服务层不接收 Gradio 请求对象或生成组件更新。
+
+公开转换 API 的四个输入、十六项输出及浏览器十五项回执保持不变。会话只缓存状态与素材路径,
+不持有 `ParseResult`;同步输出工作完成清理之后才归还执行槽,迟到响应继续由会话和任务身份过滤。
+
## 3. 数据模型
doclib 的核心数据模型围绕三张表展开:
@@ -197,7 +257,7 @@ metadata 更新规则:入库阶段写基础 metadata;解析完成后,只
| `config` | SQLite KV 配置 |
| `_migrations` | schema 版本追踪 |
-SQLite 运行在 WAL 模式,使用 FTS5 提供搜索能力。每次 DB 操作打开独立 `aiosqlite` 连接,提交后关闭,避免跨 worker 游标冲突。WAL mode 只在初始化时设置;运行期连接配置显式 busy timeout。单 doclib 进程内的写操作通过共享写锁串行化,普通读取保持并发;`SQLITE_BUSY` / `SQLITE_LOCKED` 使用有上限的指数退避重试,耗尽后对外返回可重试的 `server_busy`。
+SQLite 运行在 WAL 模式,使用 FTS5 提供搜索能力。每次 DB 操作打开独立 `aiosqlite` 连接,提交后关闭,避免跨 worker 游标冲突。WAL mode 只在初始化时设置;每个新连接均应用 busy timeout、外键约束,以及 synchronous、mmap、cache、temp store、自动 checkpoint 和 journal 大小限制。单 doclib 进程内的写操作通过共享写锁串行化,普通读取保持并发;`SQLITE_BUSY` / `SQLITE_LOCKED` 使用有上限的指数退避重试,耗尽后对外返回可重试的 `server_busy`。
## 4. 数据流
diff --git a/docs/next/config.md b/docs/next/config.md
index 4ec9a121..154244ea 100644
--- a/docs/next/config.md
+++ b/docs/next/config.md
@@ -28,6 +28,22 @@ MinerU 有两类配置:
- 默认 DB、日志和 UDS socket 从 `MINERU_HOME` 派生;默认 `data_dir` 是 `$MINERU_HOME/doclib`,其下包含 `parsed/` 和 `temp/`。
- `MINERU_CONFIG` 仍可显式指定配置文件路径;`MINERU_HOME` 只影响默认 home 和默认配置文件位置。
+Python 模块导入不装载启动配置。默认调用路径使用 `get_config()`:第一次访问时读取
+文件和环境并校验,之后复用进程默认配置;配置文件路径、存在状态和字段来源查询也使用
+同一份首次装载结果。环境和文件变更不会自动刷新这份配置。
+
+```python
+from mineru.config import get_config, load_config
+
+settings = get_config() # 读取并缓存进程默认配置
+loaded = load_config() # 按当前环境独立装载,不替换进程默认配置
+```
+
+原来的模块级 `config` 实例已移除,内部调用统一使用显式访问函数。`load_config()`
+返回配置对象及其来源信息;`Config()` 只构造模型默认值,不读取 YAML 或应用环境覆盖。
+依赖 `MINERU_HOME` 的默认路径在每次构造对象时解析。Doclib 的
+`create_app(cfg)` 会保存传入配置的独立副本,并使用其中的全局日志级别补全服务日志设置。
+
### 远程 VLM
`model.vlm` 是所有本地解析入口的 VLM 默认配置,适用于 API server、Python
diff --git a/mineru/backend/analysis/pdf/pipeline.py b/mineru/backend/analysis/pdf/pipeline.py
index efeca1be..01cd1b4c 100644
--- a/mineru/backend/analysis/pdf/pipeline.py
+++ b/mineru/backend/analysis/pdf/pipeline.py
@@ -14,6 +14,7 @@ from ....utils.timing import stage_timer
from ....config import VlmConfig
from ....model.runtime.execution import acquire_document, release_document
+from ....model.runtime.ownership import runtime_document_scope
from ....utils.async_utils import run_sync
from ....model.vlm.contracts import VlmPredictor
from ....model.runtime.memory import clean_memory, trim_process_heap
@@ -90,26 +91,27 @@ def analyze_pdf(
vlm_config: VlmConfig | None = None,
) -> AnalysisResult:
"""使用共享资源生命周期与同步窗口编排生产 PDF 模型结果。"""
- state = _PDFAnalysis()
- try:
- with stage_timer("pdf.prepare"):
- _prepare_analysis(state, file_bytes, effort, parse_mode, vlm_config)
- infer_started_at = time.perf_counter()
- model_list = process_pdf_windows(
- file_bytes,
- state.document,
- effort=effort,
- parse_mode=state.parse_mode,
- image_analysis=image_analysis,
- flash_txt_mode=state.flash_txt_mode,
- hybrid_model=state.hybrid_model,
- vlm_predictor=state.predictor,
- )
- result = _build_pdf_analysis_result(state, model_list, effort, infer_started_at)
- finally:
- with stage_timer("pdf.cleanup"):
- _close_analysis(state)
- return result
+ with runtime_document_scope():
+ state = _PDFAnalysis()
+ try:
+ with stage_timer("pdf.prepare"):
+ _prepare_analysis(state, file_bytes, effort, parse_mode, vlm_config)
+ infer_started_at = time.perf_counter()
+ model_list = process_pdf_windows(
+ file_bytes,
+ state.document,
+ effort=effort,
+ parse_mode=state.parse_mode,
+ image_analysis=image_analysis,
+ flash_txt_mode=state.flash_txt_mode,
+ hybrid_model=state.hybrid_model,
+ vlm_predictor=state.predictor,
+ )
+ result = _build_pdf_analysis_result(state, model_list, effort, infer_started_at)
+ finally:
+ with stage_timer("pdf.cleanup"):
+ _close_analysis(state)
+ return result
async def aio_analyze_pdf(
@@ -120,25 +122,26 @@ async def aio_analyze_pdf(
vlm_config: VlmConfig | None = None,
) -> AnalysisResult:
"""原生异步调度 VLM;同步准备、回填与清理使用取消安全的线程边界。"""
- state = _PDFAnalysis()
- try:
- await run_sync(_prepare_analysis, state, file_bytes, effort, parse_mode, vlm_config)
- if state.document is None or state.hybrid_model is None or state.predictor is None:
- raise ValueError("Native async PDF analysis requires a high/xhigh VLM pipeline")
- infer_started_at = time.perf_counter()
- model_list = await aio_process_pdf_windows(
- file_bytes,
- state.document,
- effort=effort,
- parse_mode=state.parse_mode,
- image_analysis=image_analysis,
- hybrid_model=state.hybrid_model,
- vlm_predictor=state.predictor,
- )
- result = await run_sync(_build_pdf_analysis_result, state, model_list, effort, infer_started_at)
- finally:
- await run_sync(_close_analysis, state)
- return result
+ with runtime_document_scope():
+ state = _PDFAnalysis()
+ try:
+ await run_sync(_prepare_analysis, state, file_bytes, effort, parse_mode, vlm_config)
+ if state.document is None or state.hybrid_model is None or state.predictor is None:
+ raise ValueError("Native async PDF analysis requires a high/xhigh VLM pipeline")
+ infer_started_at = time.perf_counter()
+ model_list = await aio_process_pdf_windows(
+ file_bytes,
+ state.document,
+ effort=effort,
+ parse_mode=state.parse_mode,
+ image_analysis=image_analysis,
+ hybrid_model=state.hybrid_model,
+ vlm_predictor=state.predictor,
+ )
+ result = await run_sync(_build_pdf_analysis_result, state, model_list, effort, infer_started_at)
+ finally:
+ await run_sync(_close_analysis, state)
+ return result
def _build_pdf_analysis_result(
diff --git a/mineru/backend/analyze.py b/mineru/backend/analyze.py
index cce37351..de139b60 100644
--- a/mineru/backend/analyze.py
+++ b/mineru/backend/analyze.py
@@ -9,7 +9,7 @@ from docvortex.document.contracts import HtmlSourceContext
from docvortex.schema import DocumentMetadata, DocumentProperties, Producer
from loguru import logger
-from ..config import VlmConfig, config
+from ..config import VlmConfig, get_config
from ..integrations.docvortex import build_metadata, read_source_properties
from ..types import FILE_SUFFIXES, FileSuffix, MiddleJson, ModelJson
from ..utils.async_utils import run_sync
@@ -87,7 +87,7 @@ def doc_analyze(
middle_json = model_json_to_middle_json(
model_json,
- llm_aided_config=config.llm_aided,
+ llm_aided_config=get_config().llm_aided,
)
return middle_json, model_json
@@ -137,7 +137,7 @@ async def aio_doc_analyze(
vlm_config=vlm_config,
)
model_json = await run_sync(_build_model_json, result, file_suffix, page_index_map, source_properties)
- middle_json = await aio_model_json_to_middle_json(model_json, llm_aided_config=config.llm_aided)
+ middle_json = await aio_model_json_to_middle_json(model_json, llm_aided_config=get_config().llm_aided)
return middle_json, model_json
diff --git a/mineru/cli/commands/server.py b/mineru/cli/commands/server.py
index 5f57f65b..5226b633 100644
--- a/mineru/cli/commands/server.py
+++ b/mineru/cli/commands/server.py
@@ -16,7 +16,7 @@ from rich.panel import Panel
from rich.table import Table
from rich.text import Text
-from ...config import config
+from ...config import get_config
from ...doclib.endpoint import read_endpoint_file
from ...doclib.instance_lock import DoclibLockUnavailable, build_doclib_home_owned_message, doclib_home_lock
from ...doclib.types import ServerStatusResponse, TCPServerStatus
@@ -39,23 +39,23 @@ class _ServerStartingStatus:
def _socket_path() -> str:
- return config.doclib.uds.path
+ return get_config().doclib.uds.path
def _endpoint_path() -> str:
- return config.doclib.endpoint_path
+ return get_config().doclib.endpoint_path
def _server_log_path() -> str:
- return os.path.expanduser(config.doclib.log.resolved_app_path)
+ return os.path.expanduser(get_config().doclib.log.resolved_app_path)
def _server_stdout_log_path() -> str:
- return os.path.expanduser(config.doclib.log.resolved_stdout_path)
+ return os.path.expanduser(get_config().doclib.log.resolved_stdout_path)
def _server_stderr_log_path() -> str:
- return os.path.expanduser(config.doclib.log.resolved_stderr_path)
+ return os.path.expanduser(get_config().doclib.log.resolved_stderr_path)
def _server_start_lock_path() -> str:
@@ -567,12 +567,12 @@ def _not_running_status() -> ServerStatusResponse:
version=__version__,
python_version=sys.version.split()[0],
socket_path=_socket_path(),
- data_dir=os.path.expanduser(config.doclib.data_dir),
- sqlite_path=os.path.expanduser(config.doclib.sqlite.path),
- log_path=os.path.expanduser(config.doclib.log.resolved_app_path),
- access_log_path=os.path.expanduser(config.doclib.log.resolved_access_path),
- stdout_log_path=os.path.expanduser(config.doclib.log.resolved_stdout_path),
- stderr_log_path=os.path.expanduser(config.doclib.log.resolved_stderr_path),
+ data_dir=os.path.expanduser(get_config().doclib.data_dir),
+ sqlite_path=os.path.expanduser(get_config().doclib.sqlite.path),
+ log_path=os.path.expanduser(get_config().doclib.log.resolved_app_path),
+ access_log_path=os.path.expanduser(get_config().doclib.log.resolved_access_path),
+ stdout_log_path=os.path.expanduser(get_config().doclib.log.resolved_stdout_path),
+ stderr_log_path=os.path.expanduser(get_config().doclib.log.resolved_stderr_path),
tcp=TCPServerStatus(enabled=False, host=None, port=None),
)
diff --git a/mineru/config.py b/mineru/config.py
index eec299ec..da904c5b 100644
--- a/mineru/config.py
+++ b/mineru/config.py
@@ -9,6 +9,7 @@ import os
import re
import socket
import tempfile
+import threading
from collections.abc import Sequence
from dataclasses import dataclass
from typing import Any, Literal, get_args, get_origin
@@ -257,16 +258,39 @@ def _load_effective_config() -> LoadedConfig:
)
+def load_config() -> LoadedConfig:
+ """按当前文件和环境独立装载配置及来源,不替换进程默认配置。"""
+ return _load_effective_config()
+
+
+def _get_loaded_config() -> LoadedConfig:
+ """首次使用时在线程锁内装载进程默认配置,后续复用同一份来源记录。"""
+ global _loaded_config
+ if _loaded_config is None:
+ with _config_load_lock:
+ if _loaded_config is None:
+ _loaded_config = load_config()
+ return _loaded_config
+
+
+def get_config() -> Config:
+ """返回首次使用时装载并缓存的进程默认配置。"""
+ return _get_loaded_config().config
+
+
def get_config_source(path: str | Sequence[str]) -> ConfigSource:
- return _loaded_config.sources.get(_normalize_config_source_path(path), "default")
+ """从进程默认配置的同一份装载结果查询字段来源。"""
+ return _get_loaded_config().sources.get(_normalize_config_source_path(path), "default")
def get_config_file_path() -> str:
- return _loaded_config.config_file
+ """返回进程默认配置首次装载时确定的文件路径。"""
+ return _get_loaded_config().config_file
def get_config_file_exists() -> bool:
- return _loaded_config.config_file_exists
+ """返回进程默认配置首次装载时的文件存在状态。"""
+ return _get_loaded_config().config_file_exists
def update_config_file(patch: dict[str, Any], *, only_if_onnx_auto: bool = False) -> bool:
@@ -311,7 +335,7 @@ def read_onnx_device_from_file() -> OnnxDevice:
class UDSConfig(BaseModel):
enabled: AutoBool = "auto"
- path: str = _default_path("doclib.sock")
+ path: str = Field(default_factory=lambda: _default_path("doclib.sock"))
permission: int = 0o600
@@ -338,7 +362,7 @@ class GlobalLogConfig(BaseModel):
class LogConfig(BaseModel):
- dir: str = _default_path("logs")
+ dir: str = Field(default_factory=lambda: _default_path("logs"))
app_path: str | None = None
access_path: str | None = None
stdout_path: str | None = None
@@ -379,7 +403,7 @@ class LogConfig(BaseModel):
class SQLiteConfig(BaseModel):
- path: str = _default_path("doclib.db")
+ path: str = Field(default_factory=lambda: _default_path("doclib.db"))
busy_timeout_ms: int = Field(default=5000, ge=0)
lock_retry_attempts: int = Field(default=3, ge=0)
lock_retry_base_delay_ms: int = Field(default=50, ge=0)
@@ -514,7 +538,7 @@ class VlmConfig(BaseModel):
class ModelConfig(BaseModel):
- base_dir: str = _default_path("models")
+ base_dir: str = Field(default_factory=lambda: _default_path("models"))
source: str = "auto"
small_backend: Literal["auto", "onnx", "torch"] = "auto"
onnx_device: OnnxDevice = "auto"
@@ -541,8 +565,8 @@ class DoclibConfig(BaseModel):
tcp: TCPConfig = Field(default_factory=TCPConfig)
log: LogConfig = Field(default_factory=LogConfig)
sqlite: SQLiteConfig = Field(default_factory=SQLiteConfig)
- endpoint_path: str = _default_path("doclib.endpoint.json")
- data_dir: str = _default_path("doclib")
+ endpoint_path: str = Field(default_factory=lambda: _default_path("doclib.endpoint.json"))
+ data_dir: str = Field(default_factory=lambda: _default_path("doclib"))
managed_parse_server: ManagedParseServerConfig = Field(default_factory=ManagedParseServerConfig)
ingest_workers: int = 2
parse_workers: int = 2
@@ -581,12 +605,13 @@ class Config(BaseModel):
model: ModelConfig = Field(default_factory=ModelConfig)
-_loaded_config = _load_effective_config()
-config = _loaded_config.config
+_loaded_config: LoadedConfig | None = None
+_config_load_lock = threading.Lock()
def PatchedConfig(**kwargs: Any) -> Config:
- merged = _deep_merge(to_jsonable_python(config), kwargs)
+ """在进程默认配置的独立副本上应用局部覆盖并重新校验。"""
+ merged = _deep_merge(to_jsonable_python(get_config()), kwargs)
return Config(**merged)
@@ -601,7 +626,8 @@ __all__ = [
"ModelConfig",
"VlmConfig",
"ModelSource",
- "config",
+ "get_config",
+ "load_config",
"Config",
"DoclibConfig",
"LLMAidedConfig",
diff --git a/mineru/doclib/app.py b/mineru/doclib/app.py
index 6978b06a..5565146d 100644
--- a/mineru/doclib/app.py
+++ b/mineru/doclib/app.py
@@ -12,14 +12,14 @@ import uuid
from collections.abc import Callable
from contextlib import asynccontextmanager
from logging.handlers import RotatingFileHandler
-from typing import TYPE_CHECKING, Any, AsyncIterator, cast
+from typing import TYPE_CHECKING, Any, AsyncIterator, Literal, cast
import uvicorn
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
from loguru import logger as loguru_logger
-from ..config import Config, LogConfig, _mineru_home, config, get_config_source
+from ..config import Config, ConfigSource, LogConfig, LogLevel, _mineru_home, get_config, get_config_source
from ..errors import MineruError, error_response, http_status_for
from ..utils.logger import configure_global_log_level
from ..utils.stdio import configure_standard_streams
@@ -37,11 +37,11 @@ logger = logging.getLogger("mineru.doclib.app")
def create_app(cfg: Config | None = None) -> FastAPI:
- """Create the new interface-backed FastAPI app."""
- if cfg is None:
- cfg = config
- _setup_logging(cfg.doclib.log)
+ """使用独立启动配置快照创建应用,显式配置不触发进程默认配置加载。"""
+ cfg = (get_config() if cfg is None else cfg).model_copy(deep=True)
+ _setup_logging(cfg.doclib.log, global_level=cfg.log.level)
state = AppState()
+ state.config = cfg
app = DoclibServer(state).app
app.title = "MinerU DocLib"
app.version = "1.0.0"
@@ -202,7 +202,6 @@ def create_app(cfg: Config | None = None) -> FastAPI:
state.socket_path = os.path.expanduser(cfg.doclib.uds.path)
state.tcp_enabled = cfg.doclib.resolved_tcp_enabled
state.tcp_host = cfg.doclib.tcp.host
- state.config = cfg
async def shutdown() -> None:
from .background.parse_server_health import get_health, stop_managed_parse_server
@@ -446,13 +445,15 @@ def main() -> None:
configure_global_log_level()
try:
with doclib_home_lock():
- _run_server(config)
+ cfg = get_config()
+ log_source = get_config_source("doclib.log.level" if cfg.doclib.log.level is not None else "log.level")
+ _run_server(cfg, log_source=log_source)
except DoclibLockUnavailable:
raise SystemExit(build_doclib_home_owned_message()) from None
-def _run_server(cfg: Config) -> None:
- """Run the doclib server while the caller holds the home ownership lock."""
+def _run_server(cfg: Config, *, log_source: ConfigSource | Literal["provided"] = "provided") -> None:
+ """在持有目录锁时按显式配置启动服务,配置来源由入口提供。"""
uds_path = os.path.expanduser(cfg.doclib.uds.path)
endpoint_path = os.path.expanduser(cfg.doclib.endpoint_path)
uds_enabled = cfg.doclib.resolved_uds_enabled
@@ -468,16 +469,9 @@ def _run_server(cfg: Config) -> None:
app = create_app(cfg)
server_id = app.state.doclib_state.server_id
- log_source = (
- get_config_source("doclib.log.level")
- if cfg is config and cfg.doclib.log.level is not None
- else get_config_source("log.level")
- if cfg is config
- else "provided"
- )
logger.info(
"Doclib logging configured level=%s source=%s path=%s",
- logging.getLevelName(_resolve_log_level(cfg.doclib.log)),
+ logging.getLevelName(_resolve_log_level(cfg.doclib.log, global_level=cfg.log.level)),
log_source,
os.path.expanduser(cfg.doclib.log.resolved_app_path),
)
@@ -637,13 +631,15 @@ def _remove_owned_uds_path(uds_path: str, expected_identity: tuple[int, int]) ->
logger.debug("Removed owned doclib UDS path=%s identity=%s", uds_path, expected_identity)
-def _resolve_log_level(log_cfg: LogConfig) -> int:
- level = log_cfg.level or config.log.level
+def _resolve_log_level(log_cfg: LogConfig, *, global_level: LogLevel) -> int:
+ """以当前应用的全局级别补全未设置的服务日志级别。"""
+ level = log_cfg.level or global_level
return getattr(logging, level.upper(), logging.INFO)
-def _setup_logging(log_cfg: LogConfig) -> None:
- level = _resolve_log_level(log_cfg)
+def _setup_logging(log_cfg: LogConfig, *, global_level: LogLevel) -> None:
+ """按当前应用配置安装日志处理器,不读取默认配置。"""
+ level = _resolve_log_level(log_cfg, global_level=global_level)
log_path = os.path.expanduser(log_cfg.resolved_app_path)
access_log_path = os.path.expanduser(log_cfg.resolved_access_path)
_ensure_log_dir(log_path)
diff --git a/mineru/doclib/background/compaction.py b/mineru/doclib/background/compaction.py
index cf6c0c0d..7d9e90b2 100644
--- a/mineru/doclib/background/compaction.py
+++ b/mineru/doclib/background/compaction.py
@@ -11,22 +11,21 @@ import time
from collections.abc import Sequence
from typing import Any, cast
-from ...parser.page_range import format_page_range
+from ...parser.page_range import format_page_range, parse_page_range_set
from ...types import Tier
from ..core.db import DatabaseManager
-from ...backend.postprocess.legacy_middle_json import read_legacy_middle_json
+from ..core.middle_json import read_cached_middle_json
from ..rows import ParseBatchRow, ParseGroupRow, ParseRow
-from ..services.parse_svc import parse_batch_json_path, parse_page_range_set
+from ..core.parse_artifacts import (
+ parse_artifact_directory,
+ parse_batch_json_path,
+ write_parse_payload,
+)
from ..types import PARSE_STATUS_DONE, PARSE_STATUS_SUPERSEDED
logger = logging.getLogger("mineru.compaction")
-def _normalize_batch_pages(batch_payload: dict[str, Any]) -> list[dict[str, Any]]:
- """读取当前和受支持的历史批次,仅在内存中统一页面结构。"""
- return [page.to_dict() for page in read_legacy_middle_json(batch_payload).pages]
-
-
class Compaction:
def __init__(self, db: DatabaseManager, interval_sec: int, data_dir: str) -> None:
self.db = db
@@ -148,9 +147,7 @@ class Compaction:
if not os.path.isfile(fpath):
return None
try:
- with open(fpath, encoding="utf-8") as f:
- batch_payload = json.load(f)
- middle_json = read_legacy_middle_json(batch_payload)
+ middle_json = read_cached_middle_json(fpath)
batch_pages = [page.to_dict() for page in middle_json.pages]
except Exception:
return None
@@ -202,8 +199,7 @@ class Compaction:
final_path = parse_batch_json_path(self.data_dir, sha256, tier, page_range, max_done_at)
temp_path = f"{final_path}.tmp-{time.time_ns()}"
payload = ParseResult.from_dict({**envelope, "pages": json_pages}).to_dict()
- with open(temp_path, "w", encoding="utf-8") as f:
- json.dump(payload, f, ensure_ascii=False, indent=2)
+ write_parse_payload(temp_path, payload)
prepared.append((temp_path, final_path))
for temp_path, final_path in prepared:
os.replace(temp_path, final_path)
@@ -218,7 +214,7 @@ class Compaction:
def _delete_obsolete_json_files(self, sha256: str, tier: Tier, keep_paths: set[str]) -> None:
"""仅在新缓存和数据库均就绪后删除已被替代的 JSON。"""
- tier_dir = os.path.join(self.data_dir, "parsed", sha256[:2], sha256, tier)
+ tier_dir = parse_artifact_directory(self.data_dir, sha256, tier)
for fname in os.listdir(tier_dir):
path = os.path.join(tier_dir, fname)
if fname.endswith(".json") and path not in keep_paths:
diff --git a/mineru/doclib/background/parse_server_health.py b/mineru/doclib/background/parse_server_health.py
index 81d9a6a7..f2254dd9 100644
--- a/mineru/doclib/background/parse_server_health.py
+++ b/mineru/doclib/background/parse_server_health.py
@@ -16,7 +16,7 @@ from typing import IO, TYPE_CHECKING, Final, Iterator, cast
import httpx
-from ...config import LogConfig, ManagedParseServerConfig, config
+from ...config import LogConfig, ManagedParseServerConfig, get_config
from ...parser.api_client import should_trust_env_for_url
from ...types import DEPLOYMENT_TIERS, TIERS, DeploymentTier, Tier
from ...parser.process_control import ManagedProcessControl
@@ -159,11 +159,11 @@ def select_available_managed_port(host: str, port: int, *, strict_port: bool, po
def get_parse_server_stdout_log_path() -> str:
- return os.path.expanduser(config.doclib.log.resolved_parse_server_stdout_path)
+ return os.path.expanduser(get_config().doclib.log.resolved_parse_server_stdout_path)
def get_parse_server_stderr_log_path() -> str:
- return os.path.expanduser(config.doclib.log.resolved_parse_server_stderr_path)
+ return os.path.expanduser(get_config().doclib.log.resolved_parse_server_stderr_path)
def _parse_server_stdout_log_path(log_cfg: LogConfig | None) -> str:
@@ -335,10 +335,10 @@ class ParseServerHealthCheck:
self.probe_timeout_sec = probe_timeout_sec
self.startup_grace_sec = startup_grace_sec
self.startup_timeout_sec = (
- startup_timeout_sec if startup_timeout_sec is not None else config.doclib.parse_server_startup_timeout_sec
+ startup_timeout_sec if startup_timeout_sec is not None else get_config().doclib.parse_server_startup_timeout_sec
)
self.stop_timeout_sec = stop_timeout_sec
- self.managed_parse_server = managed_parse_server or config.doclib.managed_parse_server
+ self.managed_parse_server = managed_parse_server or get_config().doclib.managed_parse_server
self.log_cfg = log_cfg
self.running = False
self._wakeup_event = asyncio.Event()
diff --git a/mineru/doclib/core/__init__.py b/mineru/doclib/core/__init__.py
index f4553b15..4146cfef 100644
--- a/mineru/doclib/core/__init__.py
+++ b/mineru/doclib/core/__init__.py
@@ -3,9 +3,30 @@
from .db import DatabaseManager
from .file_io import FileStat, MetadataExtractionError, compute_sha256, extract_metadata, get_file_stat
from .fts import FTSManager, strip_sep, tokenize_for_index, tokenize_for_query
+from .middle_json import read_cached_middle_json
+
+from .parse_artifacts import (
+ accessible_file_for_sha256,
+ filter_pages_by_user_range,
+ load_pages_from_done_batches,
+ parse_artifact_directory,
+ parse_batch_is_readable,
+ parse_batch_json_path,
+ parse_coverage,
+ write_parse_payload,
+)
__all__ = [
"DatabaseManager",
+ "accessible_file_for_sha256",
+ "filter_pages_by_user_range",
+ "load_pages_from_done_batches",
+ "parse_artifact_directory",
+ "parse_batch_is_readable",
+ "parse_batch_json_path",
+ "parse_coverage",
+ "read_cached_middle_json",
+ "write_parse_payload",
"FTSManager",
"FileStat",
"MetadataExtractionError",
diff --git a/mineru/doclib/core/db.py b/mineru/doclib/core/db.py
index 3e6e52ec..9662947b 100644
--- a/mineru/doclib/core/db.py
+++ b/mineru/doclib/core/db.py
@@ -19,9 +19,6 @@ _MIGRATIONS_DIR = Path(__file__).resolve().parent.parent / "migrations"
_T = TypeVar("_T")
_CREATE_TABLES_SQL: list[str] = [] # filled by _load_init_sql()
-_DEFAULT_BUSY_TIMEOUT_MS = 5000
-_DEFAULT_LOCK_RETRY_ATTEMPTS = 3
-_DEFAULT_LOCK_RETRY_BASE_DELAY_MS = 50
def _load_init_sql() -> list[str]:
@@ -58,7 +55,7 @@ class DatabaseManager:
def __init__(self, db_path: str, sqlite_cfg: SQLiteConfig | None = None) -> None:
self.db_path = db_path
- self.sqlite_cfg = sqlite_cfg # SQLiteConfig or None
+ self.sqlite_cfg = sqlite_cfg if sqlite_cfg is not None else SQLiteConfig(path=db_path)
self._write_lock = asyncio.Lock()
# ── lifecycle ──────────────────────────────────────────────────
@@ -68,23 +65,9 @@ class DatabaseManager:
os.makedirs(os.path.dirname(self.db_path), exist_ok=True)
conn = await self._connect()
try:
+ # WAL 是数据库持久设置,仅在初始化时切换,避免运行期连接重复争抢模式锁。
await conn.execute("PRAGMA journal_mode=WAL")
- if self.sqlite_cfg:
- await conn.execute(f"PRAGMA synchronous={self.sqlite_cfg.synchronous}")
- await conn.execute(f"PRAGMA mmap_size={self.sqlite_cfg.mmap_size}")
- await conn.execute(f"PRAGMA cache_size={self.sqlite_cfg.cache_size}")
- await conn.execute(f"PRAGMA temp_store={self.sqlite_cfg.temp_store}")
- await conn.execute(f"PRAGMA wal_autocheckpoint={self.sqlite_cfg.wal_autocheckpoint}")
- await conn.execute(f"PRAGMA journal_size_limit={self.sqlite_cfg.journal_size_limit}")
- else:
- await conn.execute("PRAGMA synchronous=NORMAL")
- await conn.execute("PRAGMA mmap_size=268435456")
- await conn.execute("PRAGMA cache_size=-20000")
- await conn.execute("PRAGMA temp_store=MEMORY")
- await conn.execute("PRAGMA wal_autocheckpoint=1000")
- await conn.execute("PRAGMA journal_size_limit=33554432")
-
for sql in _CREATE_TABLES_SQL:
await conn.execute(sql)
@@ -121,11 +104,18 @@ class DatabaseManager:
await conn.close()
async def _connect(self) -> aiosqlite.Connection:
- busy_timeout_ms = self.sqlite_cfg.busy_timeout_ms if self.sqlite_cfg else _DEFAULT_BUSY_TIMEOUT_MS
+ """为每个新连接应用运行参数,确保初始化、普通操作和重试使用相同配置。"""
+ busy_timeout_ms = self.sqlite_cfg.busy_timeout_ms
conn = await aiosqlite.connect(self.db_path, timeout=busy_timeout_ms / 1000)
try:
await conn.execute(f"PRAGMA busy_timeout={busy_timeout_ms}")
await conn.execute("PRAGMA foreign_keys=ON")
+ await conn.execute(f"PRAGMA synchronous={self.sqlite_cfg.synchronous}")
+ await conn.execute(f"PRAGMA mmap_size={self.sqlite_cfg.mmap_size}")
+ await conn.execute(f"PRAGMA cache_size={self.sqlite_cfg.cache_size}")
+ await conn.execute(f"PRAGMA temp_store={self.sqlite_cfg.temp_store}")
+ await conn.execute(f"PRAGMA wal_autocheckpoint={self.sqlite_cfg.wal_autocheckpoint}")
+ await conn.execute(f"PRAGMA journal_size_limit={self.sqlite_cfg.journal_size_limit}")
conn.row_factory = aiosqlite.Row
return conn
except Exception:
@@ -149,8 +139,8 @@ class DatabaseManager:
return await self._run_with_retry(operation)
async def _run_with_retry(self, operation: Callable[[aiosqlite.Connection], Awaitable[_T]]) -> _T:
- retry_attempts = self.sqlite_cfg.lock_retry_attempts if self.sqlite_cfg else _DEFAULT_LOCK_RETRY_ATTEMPTS
- base_delay_ms = self.sqlite_cfg.lock_retry_base_delay_ms if self.sqlite_cfg else _DEFAULT_LOCK_RETRY_BASE_DELAY_MS
+ retry_attempts = self.sqlite_cfg.lock_retry_attempts
+ base_delay_ms = self.sqlite_cfg.lock_retry_base_delay_ms
for attempt in range(retry_attempts + 1):
conn: aiosqlite.Connection | None = None
diff --git a/mineru/doclib/core/middle_json.py b/mineru/doclib/core/middle_json.py
new file mode 100644
index 00000000..cc191ff1
--- /dev/null
+++ b/mineru/doclib/core/middle_json.py
@@ -0,0 +1,18 @@
+"""Doclib 持久化缓存的 Middle JSON 协议读取边界。"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+from ...backend.postprocess.legacy_middle_json import read_legacy_middle_json
+from ...types import MiddleJson
+
+
+def read_cached_middle_json(path: str | Path) -> MiddleJson:
+ """仅在 Doclib 持久化缓存读取边界接受受支持的历史协议。"""
+ with open(path, encoding="utf-8") as stream:
+ return read_legacy_middle_json(json.load(stream))
+
+
+__all__ = ["read_cached_middle_json"]
diff --git a/mineru/doclib/core/parse_artifacts.py b/mineru/doclib/core/parse_artifacts.py
new file mode 100644
index 00000000..1097f800
--- /dev/null
+++ b/mineru/doclib/core/parse_artifacts.py
@@ -0,0 +1,129 @@
+"""Doclib 持久化解析产物:路径、协议读取、页选择和覆盖计算。"""
+
+from __future__ import annotations
+
+import json
+import os
+from collections.abc import Sequence
+from pathlib import Path
+from typing import Any, cast
+
+from ...errors import MineruError
+from ...parser.page_range import format_page_range as _page_numbers_to_range_str, parse_page_range_set
+from ...types import PageInfo, Tier
+from ..rows import FileRow, ParseBatchRow, ParseRow
+from ..types import FILE_STATUS_ACTIVE, PARSE_STATUS_DONE, PARSE_STATUS_PENDING, PARSE_STATUS_PARSING, ParseCoverage
+from .db import DatabaseManager
+from .middle_json import read_cached_middle_json
+
+
+def parse_artifact_directory(data_dir: str, sha256: str, tier: Tier) -> str:
+ """统一解析批次目录,供任务写出、内容读取和后台压缩使用。"""
+ return os.path.join(os.path.expanduser(data_dir), "parsed", sha256[:2], sha256, tier)
+
+
+def parse_batch_json_path(data_dir: str, sha256: str, tier: Tier, page_range: str, done_at: int | None = 0) -> str:
+ """生成持久化批次路径,保持现有页范围与完成时间文件名约定。"""
+ filename = f"{page_range}_{done_at}.json" if done_at else page_range
+ return os.path.join(parse_artifact_directory(data_dir, sha256, tier), filename)
+
+
+def write_parse_payload(path: str | Path, payload: dict[str, Any]) -> None:
+ """按现有 JSON 编码写出批次;目录、原子替换和事务时机由调用者协调。"""
+ with open(path, "w", encoding="utf-8") as stream:
+ json.dump(payload, stream, ensure_ascii=False, indent=2)
+
+
+async def accessible_file_for_sha256(db: DatabaseManager, sha256: str) -> FileRow | None:
+ """按更新时间寻找仍可访问的同内容源文件,跳过尚未刷新的失效路径。"""
+ rows = cast(
+ list[FileRow],
+ await db.fetchall(
+ "SELECT * FROM files WHERE sha256=? AND status=? ORDER BY updated_at DESC",
+ (sha256, FILE_STATUS_ACTIVE),
+ ),
+ )
+ for row in rows:
+ if os.path.exists(row["path"]):
+ return row
+ return None
+
+
+def filter_pages_by_user_range(pages: list[PageInfo], page_range: str) -> list[PageInfo]:
+ """按用户的一基页码筛选缓存中的零基页面。"""
+ requested_page_numbers = parse_page_range_set(page_range)
+ return [page for page in pages if page.page_idx + 1 in requested_page_numbers]
+
+
+def parse_batch_is_readable(data_dir: str, sha256: str, tier: Tier, batch: ParseRow) -> bool:
+ """将可转换为当前协议且覆盖记录页范围的持久化批次视为有效缓存。"""
+ json_path = parse_batch_json_path(data_dir, sha256, tier, batch["page_range"], batch["done_at"])
+ try:
+ result = read_cached_middle_json(json_path)
+ available = {page.page_idx + 1 for page in result.pages}
+ return bool(available) and parse_page_range_set(batch["page_range"]) <= available
+ except (OSError, ValueError, TypeError):
+ return False
+
+
+def load_pages_from_done_batches(
+ data_dir: str,
+ sha256: str,
+ tier: Tier,
+ done_rows: Sequence[ParseBatchRow],
+ *,
+ requested_page_numbers: set[int] | None = None,
+) -> list[PageInfo]:
+ """读取当前或受支持的历史批次,保留最新重复页;损坏或未知缓存须重新解析。"""
+ pages_by_page_idx: dict[int, PageInfo] = {}
+ stale_page_numbers: set[int] = set()
+ for row in reversed(done_rows):
+ fpath = parse_batch_json_path(data_dir, sha256, tier, row["page_range"], row["done_at"])
+ if not os.path.isfile(fpath):
+ continue
+ try:
+ parse_result = read_cached_middle_json(fpath)
+ for page in parse_result.pages:
+ pages_by_page_idx[page.page_idx] = page
+ except (OSError, ValueError, TypeError):
+ stale_page_numbers.update(parse_page_range_set(row["page_range"]))
+ uncovered_stale = stale_page_numbers - {index + 1 for index in pages_by_page_idx}
+ if requested_page_numbers is not None:
+ uncovered_stale &= requested_page_numbers
+ if uncovered_stale:
+ raise MineruError(
+ "stale_cache", "Cached document protocol is outdated or invalid; reparse the source document.", "page_range"
+ )
+ return [pages_by_page_idx[page_idx] for page_idx in sorted(pages_by_page_idx)]
+
+
+def parse_coverage(request_page_range: str, rows: list[ParseRow]) -> ParseCoverage:
+ """统一计算请求页范围内的已完成、处理中和缺失覆盖。"""
+ requested_page_numbers = parse_page_range_set(request_page_range)
+ done_page_numbers: set[int] = set()
+ active_page_numbers: set[int] = set()
+ for row in rows:
+ row_page_numbers = parse_page_range_set(row["page_range"]) & requested_page_numbers
+ if row["status"] == PARSE_STATUS_DONE:
+ done_page_numbers |= row_page_numbers
+ elif row["status"] in {PARSE_STATUS_PENDING, PARSE_STATUS_PARSING}:
+ active_page_numbers |= row_page_numbers
+ active_page_numbers -= done_page_numbers
+ missing_page_numbers = requested_page_numbers - done_page_numbers - active_page_numbers
+ return ParseCoverage(
+ done_page_range=_page_numbers_to_range_str(done_page_numbers),
+ active_page_range=_page_numbers_to_range_str(active_page_numbers),
+ missing_page_range=_page_numbers_to_range_str(missing_page_numbers),
+ )
+
+
+__all__ = [
+ "accessible_file_for_sha256",
+ "filter_pages_by_user_range",
+ "load_pages_from_done_batches",
+ "parse_artifact_directory",
+ "parse_batch_is_readable",
+ "parse_batch_json_path",
+ "parse_coverage",
+ "write_parse_payload",
+]
diff --git a/mineru/doclib/endpoint.py b/mineru/doclib/endpoint.py
index 3428affe..ab68fddd 100644
--- a/mineru/doclib/endpoint.py
+++ b/mineru/doclib/endpoint.py
@@ -10,7 +10,7 @@ from dataclasses import dataclass
from pathlib import Path
from typing import Any, Literal
-from ..config import Config, config
+from ..config import Config, get_config
logger = logging.getLogger("mineru.doclib.endpoint")
@@ -46,7 +46,7 @@ def uds_available() -> bool:
def default_endpoint_path(cfg: Config | None = None) -> str:
- cfg = config if cfg is None else cfg
+ cfg = get_config() if cfg is None else cfg
return os.path.expanduser(cfg.doclib.endpoint_path)
diff --git a/mineru/doclib/server.py b/mineru/doclib/server.py
index bcb9f42a..2a781318 100644
--- a/mineru/doclib/server.py
+++ b/mineru/doclib/server.py
@@ -3,15 +3,10 @@
from __future__ import annotations
import asyncio
-import math
import os
-import re
import signal
import sys
import time
-from collections.abc import Callable
-from dataclasses import dataclass, replace
-from pathlib import Path
from typing import Any, cast
from urllib.parse import urlparse
@@ -19,41 +14,28 @@ from fastapi import APIRouter, FastAPI, Request
from fastapi.responses import JSONResponse
from pydantic import ValidationError
-from ..config import config
+from ..config import get_config
from ..errors import InvalidRequestError, MineruError, NotFoundError, error_response, http_status_for
-from ..filetypes import IMAGE_EXTENSIONS, TEXT_EXTENSIONS, TEXT_FILE_TYPES, is_page_range_parse_extension
+from ..filetypes import TEXT_EXTENSIONS, TEXT_FILE_TYPES
from ..parser.page_range import (
- count_pages_in_range,
expand_page_range as _expand_page_range,
- format_page_range as _page_numbers_to_range_str,
normalize_page_range_input,
normalize_result_page_range,
+ parse_page_range_set,
)
from ..parser.tier import TierDependencyError, ensure_tier_runtime_dependencies
-from docvortex.render.markdown import build_markdown_image
-from docvortex.render.markdown import render_single_block
-from docvortex.assets import parse_image_data_uri_strict, transcode_image
-from docvortex.content.tree import iter_image_payloads
from ..types import (
DEPLOYMENT_TIERS,
- TIERS,
- BlockBase,
- BlockType,
DeploymentTier,
- PageBlock,
- PageInfo,
Tier,
- select_highest_cached_tier,
)
from ..model.download import verify_model_repo
from ..model.registry import ModelRepo, model_repos_for_tier
-from docvortex.document.pdf import PDFDocument
from ..version import __version__
from .background.parse_server_health import get_health, get_managed_parse_server_tier
from .base import AsyncDoclibInterface
from .config_schema import validate_config_value
from .core.db import DatabaseManager
-from .locators import ContentCursor, block_char_ref, block_ref, page_ref, parse_content_cursor
from .remote_api import REMOTE_API_KEY_CONFIG, resolve_remote_api_key
from .rows import (
ContentSearchResultRow,
@@ -62,7 +44,6 @@ from .rows import (
ExcludeRuleRow,
FilenameSearchResultRow,
FileRow,
- ParseBatchRow,
ParseRow,
ParsingRuleRow,
RecentScanRow,
@@ -72,13 +53,8 @@ from .rows import (
WatchStatsFileRow,
WatchTargetRow,
)
-from .services.parse_svc import (
- _json_batch_is_readable,
- accessible_file_for_sha256,
- filter_pages_by_user_range,
- load_pages_from_done_batches,
- parse_page_range_set,
-)
+from .core.parse_artifacts import parse_batch_is_readable, parse_coverage
+from .services.content_read_svc import ContentReadService
from .telemetry.buckets import pages_bucket, results_bucket
from .types import (
FILE_STATUS_ACTIVE,
@@ -105,11 +81,7 @@ from .types import (
ConfigSource,
ConfigUnsetResponse,
ConfigValueResponse,
- ContentAsset,
ContentFormat,
- ContentNextRequest,
- ContentRange,
- ContentRequestScope,
DocContentExportRequest,
DocContentExportResponse,
DocContentResponse,
@@ -175,59 +147,8 @@ from .types import (
from .utils.route_utils import get_route_info, has_route_info, route
-@dataclass(frozen=True)
-class _RenderedContent:
- content: str
- content_ranges: list[ContentRange]
- truncated: bool
- last_page_no: int
- next_page_no: int | None = None
- cut_inside_page: bool = False
-
-
-@dataclass(frozen=True)
-class _ReadPlan:
- sha256: str
- short_id: str
- tier: Tier
- page_range: str | None
- after: str | None
- locator: str | None
- context: int
- limit: int
- format: ContentFormat
- no_marker: bool
- image_format: ImageFormat = "jpeg"
- target: ContentCursor | None = None
- next_mode: str = "parse"
-
-
_NO_MATCHING_DOC_SHA256 = "__mineru_no_matching_doc__"
-_VISUAL_BLOCK_LABELS = {
- BlockType.IMAGE: "Image block",
- BlockType.TABLE: "Table block",
- BlockType.CHART: "Chart block",
- BlockType.EQUATION: "Formula block",
-}
-
-
-@dataclass(frozen=True)
-class _BlockImageSource:
- kind: str
- data_uri: str | None = None
-
-
-@dataclass(frozen=True)
-class _LocatorParts:
- short_id: str
- tier: Tier | None = None
- page_no: int | None = None
- block_no: int | None = None
- char_offset: int | None = None
- page_range: str | None = None
- range_selection: bool = False
-
class DoclibServer(AsyncDoclibInterface):
"""Async doclib server whose routes are declared on interface methods."""
@@ -244,6 +165,10 @@ class DoclibServer(AsyncDoclibInterface):
self.app.add_exception_handler(Exception, _unexpected_error_handler)
self.app.include_router(self._build_router())
+ def _content_read_service(self) -> ContentReadService:
+ """将当前应用已初始化的数据库与产物目录传给内容服务。"""
+ return ContentReadService(self.state.db, data_dir=_effective_data_dir(self.state))
+
def _build_router(self) -> APIRouter:
endpoints = []
for attr_name in dir(self):
@@ -423,9 +348,9 @@ class DoclibServer(AsyncDoclibInterface):
row
for row in rows
if row["status"] != PARSE_STATUS_DONE
- or _json_batch_is_readable(_effective_data_dir(self.state), row["sha256"], row["tier"], row)
+ or parse_batch_is_readable(_effective_data_dir(self.state), row["sha256"], row["tier"], row)
]
- coverage = _parse_coverage(page_range, coverage_rows) if resolved_sha256 and tier and page_range else None
+ coverage = parse_coverage(page_range, coverage_rows) if resolved_sha256 and tier and page_range else None
return ListParsesResponse(
parses=[_parse_info(row, coverage=coverage) for row in rows],
coverage=coverage,
@@ -608,16 +533,14 @@ class DoclibServer(AsyncDoclibInterface):
dims = {"content_mode": "read", "output_format": _telemetry_output_format(format), "tier": tier}
await _record_telemetry_count(self.state, "content.request.count", dimensions=dims)
try:
- response = await self._execute_read_plan(
- await self._build_read_plan_from_parse(
- doc_ref,
- tier=tier,
- page_range=page_range,
- after=after,
- limit=limit,
- format=format,
- no_marker=no_marker,
- )
+ response = await self._content_read_service().read_document(
+ doc_ref,
+ tier=tier,
+ page_range=page_range,
+ after=after,
+ limit=limit,
+ format=format,
+ no_marker=no_marker,
)
finished_dims = dims | {"status": "succeeded"}
await _record_telemetry_count(self.state, "content.finished.count", dimensions=finished_dims)
@@ -646,16 +569,14 @@ class DoclibServer(AsyncDoclibInterface):
dims = {"content_mode": "read", "output_format": _telemetry_output_format(format), "tier": "unknown"}
await _record_telemetry_count(self.state, "content.request.count", dimensions=dims)
try:
- response = await self._execute_read_plan(
- await self._build_read_plan_from_locator(
- locator,
- after=after,
- context=context,
- limit=limit,
- format=format,
- image_format=image_format,
- no_marker=no_marker,
- )
+ response = await self._content_read_service().read_locator(
+ locator,
+ after=after,
+ context=context,
+ limit=limit,
+ format=format,
+ image_format=image_format,
+ no_marker=no_marker,
)
finished_dims = dims | {"status": "succeeded", "tier": response.tier}
await _record_telemetry_count(self.state, "content.finished.count", dimensions=finished_dims)
@@ -667,148 +588,17 @@ class DoclibServer(AsyncDoclibInterface):
await _record_telemetry_duration(self.state, "content.duration_bucket.count", start_ms, dimensions=finished_dims)
raise
- async def _build_read_plan_from_parse(
- self,
- doc_ref: str,
- *,
- tier: Tier,
- page_range: str | None,
- after: str | None,
- limit: int,
- format: str,
- no_marker: bool,
- ) -> _ReadPlan:
- if format != "markdown":
- raise InvalidRequestError("invalid_request", "Only markdown is currently implemented for parse content.", "format")
- limit = max(1, limit)
- doc = await self._doc_for_ref(doc_ref, param="doc_ref")
-
- # 非 PDF 的省略范围表示整本,逻辑分节数不应触发 PDF 的前十页默认限制。
- full_document = not is_page_range_parse_extension(doc["file_type"]) and not normalize_page_range_input(page_range)
- normalized_page_range = None if full_document else _normalize_content_page_range(page_range, after, doc)
- after_cursor = _parse_after_cursor(after)
- if after_cursor:
- _validate_cursor_for_doc(after_cursor, doc, tier, normalized_page_range)
- return _ReadPlan(
- sha256=doc["sha256"],
- tier=tier,
- short_id=doc["short_id"],
- page_range=normalized_page_range,
- after=after,
- limit=limit,
- format="markdown",
- no_marker=no_marker,
- locator=None,
- context=0,
- target=None,
- next_mode="parse",
- )
-
- async def _build_read_plan_from_locator(
- self,
- locator: str,
- *,
- context: int,
- limit: int,
- format: ContentFormat,
- image_format: ImageFormat,
- no_marker: bool,
- after: str | None = None,
- ) -> _ReadPlan:
- """将页选择解析为实际页范围,同时保持单点块和图片的定位约束。"""
- limit = max(1, limit)
- context = max(0, context)
- cursor = _parse_doc_locator(locator)
- doc = cast(DocRow | None, await self.state.db.fetchone("SELECT * FROM docs WHERE short_id=?", (cursor.short_id,)))
- if doc is None:
- raise NotFoundError("doc_not_found", f"Document {cursor.short_id} not found.", "locator")
- if doc["file_type"] in TEXT_FILE_TYPES:
- raise InvalidRequestError(
- "parse_not_required",
- "Text files do not require MinerU parsing. Read the file directly.",
- "locator",
- )
- cursor = _resolve_locator_pages(cursor, doc)
-
- tier = cursor.tier or await self._default_read_tier(doc["sha256"])
- if tier is None:
- raise NotFoundError(
- "tier_not_cached",
- f"No current parsed tier is cached for document {cursor.short_id}; reparse the source document.",
- "locator",
- )
- if format == "image" and context:
- raise InvalidRequestError("context_not_applicable", "context is not supported for image reads.", "context")
- if cursor.page_range and count_pages_in_range(cursor.page_range) > 1:
- if cursor.block_no is not None:
- raise InvalidRequestError("invalid_locator", "Block and char locators require a single page.", "locator")
- if format == "image":
- raise InvalidRequestError("multi_page_image_not_supported", "image format supports only one page.", "locator")
- if context:
- raise InvalidRequestError("context_not_applicable", "context requires a single page.", "context")
- if cursor.page_no is None and context:
- raise InvalidRequestError("context_not_applicable", "context requires a page, block, or char locator.", "context")
-
- page_range = _locator_page_range(cursor, doc, context)
- if after is not None:
- if cursor.page_range is None or cursor.block_no is not None or context or format != "markdown":
- raise InvalidRequestError(
- "invalid_request", "after requires a Markdown page selection without context.", "after"
- )
- after_cursor = _parse_after_cursor(after)
- if after_cursor is None:
- raise InvalidRequestError("invalid_locator", "after must be a non-empty content cursor.", "after")
- _validate_cursor_for_doc(after_cursor, doc, tier, page_range)
- return _ReadPlan(
- sha256=doc["sha256"],
- short_id=doc["short_id"],
- tier=tier,
- page_range=page_range,
- after=after if after is not None else _locator_after(cursor, tier),
- locator=_canonical_locator(doc["short_id"], tier, cursor),
- context=context,
- limit=limit,
- format=format,
- image_format=image_format,
- no_marker=no_marker,
- target=ContentCursor(
- short_id=doc["short_id"],
- tier=tier,
- page_no=cursor.page_no or 1,
- block_no=cursor.block_no,
- char_offset=cursor.char_offset,
- )
- if cursor.page_no is not None
- else None,
- next_mode="range"
- if (cursor.range_selection and cursor.block_no is None and not context) or after is not None
- else "read",
- )
-
@route("POST", "/docs/{doc_ref}/exports", tags=("docs",))
async def export_doc_content(self, doc_ref: str, request: DocContentExportRequest) -> DocContentExportResponse:
start_ms = _now_ms()
dims = {"content_mode": "export", "output_format": _telemetry_output_format(request.format), "tier": request.tier}
await _record_telemetry_count(self.state, "content.request.count", dimensions=dims)
try:
- doc = await self._doc_for_ref(doc_ref, param="doc_ref")
- content = await self._render_doc_content(
- doc["sha256"],
- tier=request.tier,
- page_range=request.page_range,
- format=request.format,
- no_marker=request.no_marker,
- )
- output_path = os.path.abspath(request.output)
- os.makedirs(os.path.dirname(output_path), exist_ok=True)
- with open(output_path, "w", encoding="utf-8") as f:
- f.write(content)
+ response = await self._content_read_service().export_document(doc_ref, request)
finished_dims = dims | {"status": "succeeded"}
await _record_telemetry_count(self.state, "content.finished.count", dimensions=finished_dims)
await _record_telemetry_duration(self.state, "content.duration_bucket.count", start_ms, dimensions=finished_dims)
- return DocContentExportResponse(
- sha256=doc["sha256"], short_id=doc["short_id"], tier=request.tier, output=output_path
- )
+ return response
except Exception:
finished_dims = dims | {"status": "failed"}
await _record_telemetry_count(self.state, "content.finished.count", dimensions=finished_dims)
@@ -910,7 +700,7 @@ class DoclibServer(AsyncDoclibInterface):
_tier_parse_info(row)
for row in parse_rows
if row["status"] == PARSE_STATUS_DONE
- and _json_batch_is_readable(_effective_data_dir(self.state), row["sha256"], row["tier"], row)
+ and parse_batch_is_readable(_effective_data_dir(self.state), row["sha256"], row["tier"], row)
]
active_parses = [
_parse_info(row) for row in parse_rows if row["status"] in {PARSE_STATUS_PENDING, PARSE_STATUS_PARSING}
@@ -1267,181 +1057,6 @@ class DoclibServer(AsyncDoclibInterface):
total = total_row["cnt"] if total_row else 0
return rows, total, limit, offset
- async def _render_doc_content(
- self,
- sha256: str,
- *,
- tier: Tier,
- page_range: str | None,
- format: str,
- no_marker: bool,
- ) -> str:
- if format != "markdown":
- raise InvalidRequestError("invalid_request", "Only markdown is currently implemented.", "format")
- data_dir = _effective_data_dir(self.state)
- rows = await self.state.db.fetchall(
- "SELECT page_range, done_at FROM parses WHERE sha256=? AND tier=? AND status=? ORDER BY done_at DESC",
- (sha256, tier, PARSE_STATUS_DONE),
- )
- doc = cast(DocRow | None, await self.state.db.fetchone("SELECT * FROM docs WHERE sha256=?", (sha256,)))
- if doc is None:
- raise NotFoundError("doc_not_found", f"Document {sha256} not found.", "doc_ref")
- page_range = normalize_page_range_input(page_range)
- if page_range and doc["page_count"] is not None:
- page_range = _expand_page_range(page_range, doc["page_count"])
- requested = parse_page_range_set(page_range) if page_range and page_range != "all" else None
- loaded_pages = load_pages_from_done_batches(
- data_dir,
- sha256,
- tier,
- cast(list[ParseBatchRow], rows),
- requested_page_numbers=requested,
- )
- if requested is not None:
- loaded_pages = [page for page in loaded_pages if page.page_idx + 1 in requested]
- if not loaded_pages:
- raise NotFoundError("not_cached", "Requested parsed content is not cached.", "page_range")
- output: list[str] = []
- page_count = doc["page_count"]
- for page in loaded_pages:
- if not no_marker:
- output.append(_page_marker(page.page_idx + 1, page_count))
- output.extend(
- text
- for _, text in _page_markdown_blocks(
- page,
- short_id=doc["short_id"],
- tier=tier,
- )
- )
- return "\n\n".join(output)
-
- async def _execute_read_plan(self, plan: _ReadPlan) -> DocContentResponse:
- data_dir = _effective_data_dir(self.state)
- rows = await self.state.db.fetchall(
- "SELECT page_range, done_at FROM parses WHERE sha256=? AND tier=? AND status=? ORDER BY done_at DESC",
- (plan.sha256, plan.tier, PARSE_STATUS_DONE),
- )
- loaded_pages = load_pages_from_done_batches(
- data_dir,
- plan.sha256,
- plan.tier,
- cast(list[ParseBatchRow], rows),
- requested_page_numbers=parse_page_range_set(plan.page_range) if plan.page_range else None,
- )
- if plan.page_range:
- loaded_pages = filter_pages_by_user_range(loaded_pages, plan.page_range)
- if not loaded_pages:
- raise NotFoundError("not_cached", "Requested parsed content is not cached.", "page_range")
-
- doc = cast(DocRow | None, await self.state.db.fetchone("SELECT * FROM docs WHERE sha256=?", (plan.sha256,)))
- if plan.format == "image":
- return await self._render_image_response(plan, loaded_pages)
-
- pages_for_render = _select_context_pages(loaded_pages, plan.target, plan.context)
- rendered = _render_progressive_markdown(
- pages_for_render,
- short_id=plan.short_id,
- tier=plan.tier,
- after=_parse_after_cursor(plan.after),
- limit=plan.limit,
- add_markers=not plan.no_marker,
- target=plan.target,
- context=plan.context,
- page_count=doc["page_count"] if doc else None,
- )
- if not rendered.content_ranges:
- raise NotFoundError("not_cached", "Requested parsed content is not cached after cursor.", "after")
-
- page_count = doc["page_count"] if doc else None
- paginated = _is_paginated_doc(doc) if doc else True
- if (
- plan.next_mode == "parse"
- and plan.page_range is None
- and doc
- and not is_page_range_parse_extension(doc["file_type"])
- ):
- # 整本格式仅用内容游标续读,避免生成它不接受的 --pages 解析参数。
- paginated = False
- if plan.next_mode == "range":
- next_request = _next_range_read_request(rendered, plan.locator)
- elif plan.next_mode == "read":
- next_request = _next_read_request(rendered, plan.short_id, plan.tier, page_count)
- else:
- next_request = _next_content_request(
- rendered=rendered,
- request_page_range=plan.page_range,
- after=plan.after,
- page_count=page_count,
- paginated=paginated,
- )
- return DocContentResponse(
- sha256=plan.sha256,
- short_id=plan.short_id,
- tier=plan.tier,
- format="markdown",
- content=rendered.content,
- request_scope=ContentRequestScope(
- page_range=plan.page_range,
- after=plan.after,
- limit=plan.limit,
- locator=plan.locator,
- context=plan.context,
- ),
- content_ranges=rendered.content_ranges,
- truncated=rendered.truncated,
- next_request=next_request,
- )
-
- async def _render_image_response(
- self,
- plan: _ReadPlan,
- loaded_pages: list[PageInfo],
- ) -> DocContentResponse:
- if plan.target is None or plan.target.page_no is None:
- raise InvalidRequestError("format_not_supported", "image format requires a page or block locator.", "format")
- if plan.page_range and len(parse_page_range_set(plan.page_range)) != 1:
- raise InvalidRequestError("multi_page_image_not_supported", "image format supports only one page.", "locator")
- page = _find_page(loaded_pages, plan.target.page_no)
- if page is None:
- raise NotFoundError("page_not_cached", f"Page {plan.target.page_no} is not cached.", "locator")
-
- if plan.target.block_no is None:
- asset = await self._render_source_image_asset(plan, page)
- else:
- block = _find_block_by_no(page, plan.target.block_no)
- if block is None:
- raise NotFoundError("block_not_found", f"Block {plan.target.block_no} not found.", "locator")
- source = _resolve_block_image_source(block)
- if source is None:
- raise NotFoundError("asset_not_available", "Block image asset is not available.", "locator")
- if source.kind == "source_bbox":
- asset = await self._render_source_image_asset(plan, page)
- else:
- assert source.data_uri is not None
- asset = self._render_base64_image_asset(plan, source.data_uri)
-
- target_ref = plan.locator or page_ref(plan.short_id, plan.tier, plan.target.page_no)
- return DocContentResponse(
- sha256=plan.sha256,
- short_id=plan.short_id,
- tier=plan.tier,
- format="image",
- content="",
- request_scope=ContentRequestScope(
- page_range=plan.page_range,
- after=plan.after,
- limit=plan.limit,
- locator=plan.locator,
- context=plan.context,
- image_format=plan.image_format,
- ),
- content_ranges=[ContentRange(page_range=str(plan.target.page_no), start=target_ref, end=target_ref)],
- truncated=False,
- next_request=None,
- asset=asset,
- )
-
async def _sha256_for_path(self, path: str) -> str | None:
row = cast(
Sha256Row | None,
@@ -1456,10 +1071,8 @@ class DoclibServer(AsyncDoclibInterface):
return row
async def _maybe_doc_for_ref(self, doc_ref: str) -> DocRow | None:
- row = cast(DocRow | None, await self.state.db.fetchone("SELECT * FROM docs WHERE sha256=?", (doc_ref,)))
- if row is not None:
- return row
- return cast(DocRow | None, await self.state.db.fetchone("SELECT * FROM docs WHERE short_id=?", (doc_ref,)))
+ """通过内容服务统一解析完整和短文档标识。"""
+ return await self._content_read_service().find_document(doc_ref)
async def _resolve_doc_filter_sha256(self, doc_ref: str | None) -> str | None:
if doc_ref is None:
@@ -1490,180 +1103,6 @@ class DoclibServer(AsyncDoclibInterface):
)
return [_file_info(row) for row in rows]
- async def _default_read_tier(self, sha256: str) -> Tier | None:
- rows = cast(
- list[ParseRow],
- await self.state.db.fetchall(
- "SELECT * FROM parses WHERE sha256=? AND status=? ORDER BY done_at DESC",
- (sha256, PARSE_STATUS_DONE),
- ),
- )
- tiers: set[Tier] = {
- row["tier"]
- for row in rows
- if row["tier"] in TIERS
- and row["tier"] != "flash"
- and _json_batch_is_readable(_effective_data_dir(self.state), sha256, row["tier"], row)
- }
- if not tiers:
- return None
- return select_highest_cached_tier(tiers)
-
- async def _render_source_image_asset(self, plan: _ReadPlan, page: PageInfo) -> ContentAsset:
- """查找源文件并调用公共图像输出,保留 Doclib 定位与错误协议。"""
- if plan.target is None:
- raise InvalidRequestError("format_not_supported", "image format requires a page or block locator.", "format")
- file_row = await accessible_file_for_sha256(self.state.db, plan.sha256)
- if file_row is None:
- raise NotFoundError("no_accessible_file", "No active source file found for this document.", "locator")
- source_path = file_row["path"]
- source_ext = str(file_row.get("ext") or Path(source_path).suffix).lower().lstrip(".")
- source_bytes = Path(source_path).read_bytes()
- if source_ext == "pdf":
- doc = PDFDocument(source_bytes)
- elif source_ext in IMAGE_EXTENSIONS:
- doc = PDFDocument.from_image(source_bytes)
- else:
- raise InvalidRequestError(
- "format_not_supported",
- "Page and bbox image output is only supported for PDF and image sources.",
- "format",
- )
- with doc:
- bbox = None
- if plan.target.block_no is not None:
- block = _find_block_by_no(page, plan.target.block_no)
- if block is None:
- raise NotFoundError("block_not_found", f"Block {plan.target.block_no} not found.", "locator")
- if block.bbox is None or _is_empty_bbox(block.bbox):
- raise InvalidRequestError("bbox_not_available", "Block bbox is not available for image output.", "locator")
- bbox = block.bbox
- artifact = doc.render_image(plan.target.page_no - 1, bbox=bbox, image_format=plan.image_format)
- return _write_temp_asset(
- self.state.data_dir,
- plan.short_id,
- artifact.extension,
- artifact.data,
- mime_type=artifact.mime_type,
- width=artifact.width,
- height=artifact.height,
- )
-
- def _render_base64_image_asset(self, plan: _ReadPlan, data_uri: str) -> ContentAsset:
- """通过公共素材接口解码并转码,Doclib 只负责临时资产写出。"""
- image_bytes, _extension = parse_image_data_uri_strict(data_uri)
- artifact = transcode_image(image_bytes, image_format=plan.image_format)
- return _write_temp_asset(
- self.state.data_dir,
- plan.short_id,
- artifact.extension,
- artifact.data,
- mime_type=artifact.mime_type,
- width=artifact.width,
- height=artifact.height,
- )
-
-
-_READ_LOCATOR_RE = re.compile(
- r"^doc:(?P<short_id>[0-9a-fA-F]+)"
- r"(?:/tier:(?P<tier>flash|basic|standard|advanced)"
- r"(?:/page:(?P<page_range>[^/]+)"
- r"(?:/block:(?P<block_no>[1-9][0-9]*)(?:/char:(?P<char_offset>0|[1-9][0-9]*))?)?)?)?$"
-)
-
-
-def _parse_doc_locator(locator: str) -> _LocatorParts:
- """只解析输入语法,倒数页和范围在获取文档页数后统一求值。"""
- match = _READ_LOCATOR_RE.match(locator)
- if match is None:
- raise InvalidRequestError("invalid_locator", f"Invalid doclib locator: {locator}", "locator")
- page_range = match.group("page_range")
- if page_range is not None:
- if not page_range.strip():
- raise InvalidRequestError("invalid_locator", "Page selection cannot be empty.", "locator")
- try:
- normalized_range = normalize_page_range_input(page_range)
- except InvalidRequestError as error:
- raise InvalidRequestError("invalid_locator", error.message, "locator") from error
- else:
- normalized_range = None
- block_no = match.group("block_no")
- char_offset = match.group("char_offset")
- return _LocatorParts(
- short_id=match.group("short_id"),
- tier=cast(Tier | None, match.group("tier")),
- page_no=int(normalized_range) if normalized_range and normalized_range.isdecimal() else None,
- block_no=int(block_no) if block_no is not None else None,
- char_offset=int(char_offset) if char_offset is not None else None,
- page_range=normalized_range,
- range_selection=page_range is not None and (page_range.strip() == "all" or "," in page_range or "-" in page_range),
- )
-
-
-def _resolve_locator_pages(locator: _LocatorParts, doc: DocRow) -> _LocatorParts:
- """复用公共页范围算法求值;倒数页和 all 禁止用部分缓存猜测总页数。"""
- if locator.page_range is None:
- return locator
- page_count = doc.get("page_count")
- if (page_count is None or page_count < 1) and ("r" in locator.page_range or locator.page_range == "all"):
- raise InvalidRequestError(
- "invalid_locator", "Cannot resolve all/rN because the document page count is unavailable.", "locator"
- )
- resolved_range = _expand_page_range(locator.page_range, page_count or 1)
- single_page = count_pages_in_range(resolved_range) == 1
- return replace(locator, page_range=resolved_range, page_no=int(resolved_range) if single_page else None)
-
-
-def _canonical_locator(short_id: str, tier: Tier, locator: _LocatorParts) -> str:
- if locator.range_selection and locator.page_range is not None and locator.block_no is None:
- return f"doc:{short_id}/tier:{tier}/page:{locator.page_range}"
- if locator.page_no is None:
- return f"doc:{short_id}/tier:{tier}"
- if locator.block_no is None:
- return page_ref(short_id, tier, locator.page_no)
- if locator.char_offset is None:
- return block_ref(short_id, tier, locator.page_no, locator.block_no)
- return block_char_ref(short_id, tier, locator.page_no, locator.block_no, locator.char_offset)
-
-
-def _locator_after(locator: _LocatorParts, tier: Tier) -> str | None:
- if locator.page_no is None:
- return None
- if locator.block_no is None:
- return None
- return _canonical_locator(locator.short_id, tier, locator) if locator.char_offset is not None else None
-
-
-def _locator_page_range(locator: _LocatorParts, doc: DocRow, context: int) -> str | None:
- page_count = doc.get("page_count") or 1
- if locator.page_range is not None and not context:
- return locator.page_range
- if locator.page_no is None:
- return _normalize_content_page_range(None, None, doc)
- if locator.page_no > page_count:
- return _normalize_page_range(str(locator.page_no), page_count)
- start = max(1, locator.page_no - context)
- end = min(page_count, locator.page_no + context)
- return _range_str(start, end)
-
-
-def _parse_after_cursor(after: str | None) -> ContentCursor | None:
- if not after:
- return None
- try:
- return parse_content_cursor(after)
- except ValueError as exc:
- raise InvalidRequestError("invalid_locator", str(exc), "after") from None
-
-
-def _validate_cursor_for_doc(cursor: ContentCursor, doc: DocRow, tier: Tier, page_range: str | None) -> None:
- if cursor.short_id != doc["short_id"]:
- raise InvalidRequestError("invalid_request", "after cursor does not belong to this document.", "after")
- if cursor.tier != tier:
- raise InvalidRequestError("invalid_request", "after cursor tier does not match request tier.", "after")
- if page_range and cursor.page_no not in parse_page_range_set(page_range):
- raise InvalidRequestError("invalid_request", "after cursor is outside requested page_range.", "after")
-
def _mask_config(config: dict[str, str]) -> dict[str, str]:
return {key: _mask_config_value(key, value) for key, value in config.items()}
@@ -1816,411 +1255,15 @@ async def _sqlite_journal_mode(db: DatabaseManager) -> str | None:
return cast(str | None, row.get("journal_mode"))
-def _is_paginated_doc(doc: DocRow) -> bool:
- page_count = doc.get("page_count")
- return page_count is not None and page_count > 1
-
-
-def _normalize_content_page_range(page_range: str | None, after: str | None, doc: DocRow) -> str | None:
- page_range = normalize_page_range_input(page_range)
- page_count = doc.get("page_count") or 1
- if _is_paginated_doc(doc):
- if page_range and page_range.strip() == "all":
- return _expand_page_range("all", page_count)
- if page_range:
- return _normalize_page_range(page_range, page_count)
- if after:
- cursor = _parse_after_cursor(after)
- assert cursor is not None
- end = min(page_count, cursor.page_no + 9)
- return _range_str(cursor.page_no, end)
- end = min(page_count, 10)
- return _range_str(1, end)
- if page_range and page_range.strip() == "all":
- return None
- return _normalize_page_range(page_range, page_count) if page_range else None
-
-
-def _normalize_page_range(page_range: str, page_count: int) -> str:
- return _expand_page_range(page_range, page_count)
-
-
-def _select_context_pages(pages: list[PageInfo], target: ContentCursor | None, context: int) -> list[PageInfo]:
- if target is None or target.page_no is None or context <= 0:
- return pages
- start = target.page_no - context
- end = target.page_no + context
- return [page for page in pages if start <= page.page_idx + 1 <= end]
-
-
-def _find_page(pages: list[PageInfo], page_no: int) -> PageInfo | None:
- return next((page for page in pages if page.page_idx + 1 == page_no), None)
-
-
-def _find_block_by_no(page: PageInfo, block_no: int) -> PageBlock | None:
- target_index = block_no - 1
- return next((block for block in page.blocks if block.index == target_index), None)
-
-
-def _is_empty_bbox(bbox: object) -> bool:
- if not isinstance(bbox, (tuple, list)) or len(bbox) != 4:
- return True
- try:
- values = [float(v) for v in bbox]
- except (TypeError, ValueError):
- return True
- if not all(math.isfinite(value) for value in values):
- return True
- return values[0] >= values[2] or values[1] >= values[3]
-
-
-def _resolve_block_image_source(block: BlockBase) -> _BlockImageSource | None:
- if not _is_empty_bbox(block.bbox):
- return _BlockImageSource(kind="source_bbox")
- for payload in iter_image_payloads(block):
- if payload.image_base64:
- return _BlockImageSource(kind="base64", data_uri=payload.image_base64)
- return None
-
-
-def _make_doclib_image_renderer(
- *,
- short_id: str,
- tier: Tier,
- page_no: int,
-) -> Callable[[BlockBase], str]:
- def _render(block: BlockBase) -> str:
- label = _VISUAL_BLOCK_LABELS.get(block.type, "Image block")
- source = _resolve_block_image_source(block)
- if source is not None:
- block_no = block.index + 1 if block.index is not None else 0 # TODO: change after block.index's type changed.
- return build_markdown_image(block_ref(short_id, tier, page_no, block_no), label)
- remote_url = next((payload.image_url for payload in iter_image_payloads(block) if payload.image_url), None)
- return build_markdown_image(remote_url, label) if remote_url is not None else f"![{label}]()"
-
- return _render
-
-
-def _write_temp_asset(
- data_dir: str,
- short_id: str,
- ext: str,
- image_bytes: bytes,
- *,
- mime_type: str,
- width: int | None,
- height: int | None,
-) -> ContentAsset:
- safe_ext = ext.lower().lstrip(".") or "png"
- output_dir = Path(os.path.expanduser(data_dir)) / "temp" / "read-assets"
- output_dir.mkdir(parents=True, exist_ok=True)
- path = output_dir / f"{short_id}-{int(time.time() * 1000)}.{safe_ext}"
- path.write_bytes(image_bytes)
- return ContentAsset(
- path=str(path),
- mime_type=mime_type,
- size_bytes=len(image_bytes),
- width=width,
- height=height,
- )
-
-
-def _render_progressive_markdown(
- pages: list[PageInfo],
- *,
- short_id: str,
- tier: Tier,
- after: Any,
- limit: int,
- add_markers: bool,
- target: ContentCursor | None = None,
- context: int = 0,
- page_count: int | None = None,
-) -> _RenderedContent:
- output: list[str] = []
- ranges: list[ContentRange] = []
- started = after is None
- last_page_no = 0
- next_page_no: int | None = None
- cut_inside_page = False
-
- for page in pages:
- page_no = page.page_idx + 1
- if after and page_no < after.page_no:
- continue
- page_blocks = _page_markdown_blocks(
- page,
- short_id=short_id,
- tier=tier,
- )
- if target and target.block_no is not None and page_no == target.page_no:
- start_block_no = max(1, target.block_no - context)
- end_block_no = target.block_no + context
- page_blocks = [(index, text) for index, text in page_blocks if start_block_no <= index + 1 <= end_block_no]
- page_output: list[str] = []
- page_start_cursor = page_ref(short_id, tier, page_no)
- page_end_cursor = page_ref(short_id, tier, page_no)
- page_started = False
- empty_page_selected = False
-
- for block_index, block_text in page_blocks:
- block_no = block_index + 1
- if after and page_no == after.page_no:
- if after.block_no is None:
- continue
- if block_no < after.block_no:
- continue
- if block_no == after.block_no and after.char_offset is None:
- continue
- started = True
- text = block_text
- block_start_offset = 0
- if after and page_no == after.page_no and after.block_no == block_no and after.char_offset is not None:
- block_start_offset = after.char_offset
- text = text[block_start_offset:]
- if not text:
- continue
- block_start_cursor = (
- block_char_ref(short_id, tier, page_no, block_no, block_start_offset)
- if block_start_offset
- else block_ref(short_id, tier, page_no, block_no)
- )
- if not page_started:
- page_start_cursor = block_start_cursor if after and after.block_no is not None else page_start_cursor
- page_started = True
- candidate_page = [*page_output, text]
- candidate_content = _join_markdown([*output, *candidate_page])
- if len(candidate_content) <= limit:
- page_output.append(text)
- page_end_cursor = block_ref(short_id, tier, page_no, block_no)
- continue
-
- if page_output and not output:
- cut_inside_page = True
- break
- if output:
- next_page_no = page_no
- return _RenderedContent(
- content=_join_markdown(output),
- content_ranges=_merge_content_ranges(ranges),
- truncated=True,
- last_page_no=last_page_no,
- next_page_no=next_page_no,
- cut_inside_page=False,
- )
-
- cut = _soft_cut(text, limit)
- if cut <= 0:
- cut = min(len(text), limit)
- page_output.append(text[:cut])
- end_offset = block_start_offset + cut
- page_end_cursor = block_char_ref(short_id, tier, page_no, block_no, end_offset)
- cut_inside_page = True
- break
-
- if not page_output and after and page_no == after.page_no:
- continue
- if not page_output and (target is None or target.block_no is None):
- started = True
- empty_page_selected = True
- if add_markers:
- page_output.append(_page_marker(page_no, page_count))
-
- if not started or (not page_output and not empty_page_selected):
- continue
- if add_markers:
- marker = _page_marker(page_no, page_count)
- if not page_output or page_output[0] != marker:
- page_output.insert(0, marker)
- if page_output:
- output.extend(page_output)
- last_page_no = page_no
- ranges.append(ContentRange(page_range=str(page_no), start=page_start_cursor, end=page_end_cursor))
- if cut_inside_page:
- next_page_no = page_no
- return _RenderedContent(
- content=_join_markdown(output),
- content_ranges=_merge_content_ranges(ranges),
- truncated=True,
- last_page_no=last_page_no,
- next_page_no=next_page_no,
- cut_inside_page=True,
- )
- # Keep whole pages as the normal truncation boundary for paginated docs.
- if len(_join_markdown(output)) >= limit:
- next_page_no = page_no + 1
- return _RenderedContent(
- content=_join_markdown(output),
- content_ranges=_merge_content_ranges(ranges),
- truncated=next_page_no <= (pages[-1].page_idx + 1),
- last_page_no=last_page_no,
- next_page_no=next_page_no,
- cut_inside_page=False,
- )
-
- return _RenderedContent(
- content=_join_markdown(output),
- content_ranges=_merge_content_ranges(ranges),
- truncated=False,
- last_page_no=last_page_no,
- next_page_no=None,
- cut_inside_page=False,
- )
-
-
-def _page_marker(page_no: int, page_count: int | None) -> str:
- if page_count is None:
- return f"<!-- page {page_no} -->"
- return f"<!-- page {page_no} of {page_count} -->"
-
-
-def _page_markdown_blocks(
- page: PageInfo,
- *,
- short_id: str = "",
- tier: Tier = "standard",
-) -> list[tuple[int, str]]:
- from ..config import config
-
- result: list[tuple[int, str]] = []
- image_renderer = _make_doclib_image_renderer(
- short_id=short_id,
- tier=tier,
- page_no=page.page_idx + 1,
- )
- delimiters = config.render.latex_delimiters
- for block in page.blocks:
- rendered = render_single_block(
- block,
- delimiters=delimiters,
- asset_base_url="",
- image_renderer=image_renderer,
- )
- text = rendered.strip() if rendered else ""
- if text:
- # TODO: simplify after BlockBase.index type narrows to int (top-level blocks always have index)
- block_index = block.index if block.index is not None else -1
- result.append((block_index, text))
- return result
-
-
-def _join_markdown(parts: list[str]) -> str:
- return "\n\n".join(part for part in parts if part)
-
-
-def _soft_cut(text: str, limit: int) -> int:
- if len(text) <= limit:
- return len(text)
- window = text[:limit]
- for pattern in ("\n\n", "\n", "。", ". ", " "):
- pos = window.rfind(pattern)
- if pos > max(0, limit // 2):
- return pos + len(pattern)
- return limit
-
-
-def _merge_content_ranges(ranges: list[ContentRange]) -> list[ContentRange]:
- if not ranges:
- return []
- merged: list[ContentRange] = []
- current = ranges[0]
- for item in ranges[1:]:
- if current.page_range and item.page_range:
- current_page_numbers = parse_page_range_set(current.page_range)
- item_page_numbers = parse_page_range_set(item.page_range)
- if max(current_page_numbers) + 1 == min(item_page_numbers):
- current = ContentRange(
- page_range=_page_numbers_to_range_str(current_page_numbers | item_page_numbers),
- start=current.start,
- end=item.end,
- )
- continue
- merged.append(current)
- current = item
- merged.append(current)
- return merged
-
-
-def _next_content_request(
- *,
- rendered: _RenderedContent,
- request_page_range: str | None,
- after: str | None,
- page_count: int | None,
- paginated: bool,
-) -> ContentNextRequest | None:
- if not rendered.content_ranges:
- return None
- if not paginated:
- if rendered.truncated:
- return ContentNextRequest(after=rendered.content_ranges[-1].end)
- return None
-
- total_page_count = page_count or rendered.last_page_no
- if rendered.truncated:
- if rendered.cut_inside_page:
- start = rendered.next_page_no or rendered.last_page_no
- end = _last_requested_page(request_page_range) or start
- return ContentNextRequest(page_range=_range_str(start, end), after=rendered.content_ranges[-1].end)
- start = rendered.next_page_no or (rendered.last_page_no + 1)
- end = _last_requested_page(request_page_range) or min(total_page_count, start + 9)
- if start <= end:
- return ContentNextRequest(page_range=_range_str(start, end))
- if after:
- return ContentNextRequest(page_range=request_page_range, after=rendered.content_ranges[-1].end)
- return None
-
- start = rendered.last_page_no + 1
- if start > total_page_count:
- return None
- end = min(total_page_count, start + 9)
- return ContentNextRequest(page_range=_range_str(start, end))
-
-
-def _next_read_request(
- rendered: _RenderedContent,
- short_id: str,
- tier: Tier,
- page_count: int | None,
-) -> ContentNextRequest | None:
- if not rendered.content_ranges:
- return None
- if rendered.truncated:
- return ContentNextRequest(locator=rendered.content_ranges[-1].end)
- total_page_count = page_count or rendered.last_page_no
- next_page_no = rendered.last_page_no + 1
- if next_page_no > total_page_count:
- return None
- return ContentNextRequest(locator=page_ref(short_id, tier, next_page_no))
-
-
-def _next_range_read_request(rendered: _RenderedContent, locator: str | None) -> ContentNextRequest | None:
- """保留范围定位器并推进实际页码游标,避免续读扩入未选择的页面。"""
- if not rendered.truncated or not rendered.content_ranges or locator is None:
- return None
- return ContentNextRequest(locator=locator, after=rendered.content_ranges[-1].end)
-
-
-def _last_requested_page(page_range: str | None) -> int | None:
- if not page_range:
- return None
- page_numbers = parse_page_range_set(page_range)
- return max(page_numbers) if page_numbers else None
-
-
-def _range_str(start: int, end: int) -> str:
- return _page_numbers_to_range_str(range(start, end + 1))
-
-
def _effective_data_dir(state: Any) -> str:
data_dir = getattr(state, "data_dir", "")
if data_dir:
return os.path.expanduser(data_dir)
- return os.path.expanduser(config.doclib.data_dir)
+ return os.path.expanduser(get_config().doclib.data_dir)
def _tail_log(log_path: str, lines: int = 100) -> list[str]:
- log_path = os.path.expanduser(log_path) if log_path else os.path.expanduser(config.doclib.log.resolved_app_path)
+ log_path = os.path.expanduser(log_path) if log_path else os.path.expanduser(get_config().doclib.log.resolved_app_path)
if not os.path.isfile(log_path):
return []
with open(log_path, encoding="utf-8", errors="replace") as fh:
@@ -2430,22 +1473,3 @@ def _search_result(row: ContentSearchResultRow) -> SearchResult:
def _find_result(row: FilenameSearchResultRow) -> FindResult:
return FindResult.model_validate(row)
-
-
-def _parse_coverage(request_page_range: str, rows: list[ParseRow]) -> ParseCoverage:
- requested_page_numbers = parse_page_range_set(request_page_range)
- done_page_numbers: set[int] = set()
- active_page_numbers: set[int] = set()
- for row in rows:
- row_page_numbers = parse_page_range_set(row["page_range"]) & requested_page_numbers
- if row["status"] == PARSE_STATUS_DONE:
- done_page_numbers |= row_page_numbers
- elif row["status"] in {PARSE_STATUS_PENDING, PARSE_STATUS_PARSING}:
- active_page_numbers |= row_page_numbers
- active_page_numbers -= done_page_numbers
- missing_page_numbers = requested_page_numbers - done_page_numbers - active_page_numbers
- return ParseCoverage(
- done_page_range=_page_numbers_to_range_str(done_page_numbers),
- active_page_range=_page_numbers_to_range_str(active_page_numbers),
- missing_page_range=_page_numbers_to_range_str(missing_page_numbers),
- )
diff --git a/mineru/doclib/services/__init__.py b/mineru/doclib/services/__init__.py
index 1e19a727..f44055ea 100644
--- a/mineru/doclib/services/__init__.py
+++ b/mineru/doclib/services/__init__.py
@@ -2,6 +2,7 @@
from .cleanup_svc import CleanupService
from .config_svc import ConfigService
+from .content_read_svc import ContentReadService
from .parse_svc import (
FileRefreshResult,
FileRefreshStatus,
@@ -10,11 +11,8 @@ from .parse_svc import (
default_parse_range,
ensure_doc_record,
expand_page_range,
- filter_pages_by_user_range,
- load_pages_from_done_batches,
page_range_covered,
page_range_uncovered,
- parse_batch_json_path,
parse_page_range_set,
)
from .scan_svc import ScanService
@@ -23,6 +21,7 @@ from .search_svc import SearchService
__all__ = [
"CleanupService",
"ConfigService",
+ "ContentReadService",
"FileRefreshResult",
"FileRefreshStatus",
"ParseFailure",
@@ -32,10 +31,7 @@ __all__ = [
"default_parse_range",
"ensure_doc_record",
"expand_page_range",
- "filter_pages_by_user_range",
- "load_pages_from_done_batches",
"page_range_covered",
"page_range_uncovered",
- "parse_batch_json_path",
"parse_page_range_set",
]
diff --git a/mineru/doclib/services/content_read_svc.py b/mineru/doclib/services/content_read_svc.py
new file mode 100644
index 00000000..27366f7c
--- /dev/null
+++ b/mineru/doclib/services/content_read_svc.py
@@ -0,0 +1,1083 @@
+"""Doclib 内容读取服务:定位计划、渐进 Markdown、源图像和文件导出。"""
+
+from __future__ import annotations
+
+import math
+import os
+import re
+import time
+from collections.abc import Callable
+from dataclasses import dataclass, replace
+from pathlib import Path
+from typing import cast
+
+from docvortex.assets import parse_image_data_uri_strict, transcode_image
+from docvortex.content.tree import iter_image_payloads
+from docvortex.document.pdf import PDFDocument
+from docvortex.render.markdown import build_markdown_image, render_single_block
+
+from ...config import get_config
+from ...errors import InvalidRequestError, NotFoundError
+from ...filetypes import IMAGE_EXTENSIONS, TEXT_FILE_TYPES, is_page_range_parse_extension
+from ...parser.page_range import (
+ count_pages_in_range,
+ expand_page_range as _expand_page_range,
+ format_page_range as _page_numbers_to_range_str,
+ normalize_page_range_input,
+ parse_page_range_set,
+)
+from ...types import TIERS, BlockBase, BlockType, PageBlock, PageInfo, Tier, select_highest_cached_tier
+from ..core.db import DatabaseManager
+from ..core.parse_artifacts import (
+ accessible_file_for_sha256,
+ filter_pages_by_user_range,
+ load_pages_from_done_batches,
+ parse_batch_is_readable,
+)
+from ..locators import ContentCursor, block_char_ref, block_ref, page_ref, parse_content_cursor
+from ..rows import DocRow, ParseBatchRow, ParseRow
+from ..types import (
+ PARSE_STATUS_DONE,
+ ContentAsset,
+ ContentFormat,
+ ContentNextRequest,
+ ContentRange,
+ ContentRequestScope,
+ DocContentExportRequest,
+ DocContentExportResponse,
+ DocContentResponse,
+ ImageFormat,
+)
+
+
+@dataclass(frozen=True)
+class _RenderedContent:
+ content: str
+ content_ranges: list[ContentRange]
+ truncated: bool
+ last_page_no: int
+ next_page_no: int | None = None
+ cut_inside_page: bool = False
+
+
+@dataclass(frozen=True)
+class _ReadPlan:
+ sha256: str
+ short_id: str
+ tier: Tier
+ page_range: str | None
+ after: str | None
+ locator: str | None
+ context: int
+ limit: int
+ format: ContentFormat
+ no_marker: bool
+ image_format: ImageFormat = "jpeg"
+ target: ContentCursor | None = None
+ next_mode: str = "parse"
+
+
+@dataclass(frozen=True)
+class _BlockImageSource:
+ kind: str
+ data_uri: str | None = None
+
+
+@dataclass(frozen=True)
+class _LocatorParts:
+ short_id: str
+ tier: Tier | None = None
+ page_no: int | None = None
+ block_no: int | None = None
+ char_offset: int | None = None
+ page_range: str | None = None
+ range_selection: bool = False
+
+
+_VISUAL_BLOCK_LABELS = {
+ BlockType.IMAGE: "Image block",
+ BlockType.TABLE: "Table block",
+ BlockType.CHART: "Chart block",
+ BlockType.EQUATION: "Formula block",
+}
+
+
+class ContentReadService:
+ """协调缓存内容读取,不持有 HTTP 状态或后台任务生命周期。"""
+
+ def __init__(self, db: DatabaseManager, *, data_dir: str) -> None:
+ """显式接收数据库和产物目录,便于不同应用实例独立读取。"""
+ self.db = db
+ self.data_dir = os.path.expanduser(data_dir)
+
+ async def read_document(
+ self,
+ doc_ref: str,
+ *,
+ tier: Tier,
+ page_range: str | None = None,
+ after: str | None = None,
+ limit: int = 30000,
+ format: str = "markdown",
+ no_marker: bool = False,
+ ) -> DocContentResponse:
+ """读取文档解析结果,并保留页范围、截断及续读语义。"""
+ plan = await self._build_read_plan_from_parse(
+ doc_ref,
+ tier=tier,
+ page_range=page_range,
+ after=after,
+ limit=limit,
+ format=format,
+ no_marker=no_marker,
+ )
+ return await self._execute_read_plan(plan)
+
+ async def read_locator(
+ self,
+ locator: str,
+ *,
+ after: str | None = None,
+ context: int = 0,
+ limit: int = 30000,
+ format: ContentFormat = "markdown",
+ image_format: ImageFormat = "jpeg",
+ no_marker: bool = False,
+ ) -> DocContentResponse:
+ """读取文档、页、块或字符定位符,可返回文本或图片素材。"""
+ plan = await self._build_read_plan_from_locator(
+ locator,
+ after=after,
+ context=context,
+ limit=limit,
+ format=format,
+ image_format=image_format,
+ no_marker=no_marker,
+ )
+ return await self._execute_read_plan(plan)
+
+ async def export_document(self, doc_ref: str, request: DocContentExportRequest) -> DocContentExportResponse:
+ """从缓存生成完整内容并写入用户指定的导出文件。"""
+ doc = await self.get_document(doc_ref, param="doc_ref")
+ content = await self._render_doc_content(
+ doc["sha256"],
+ tier=request.tier,
+ page_range=request.page_range,
+ format=request.format,
+ no_marker=request.no_marker,
+ )
+ output_path = os.path.abspath(request.output)
+ os.makedirs(os.path.dirname(output_path), exist_ok=True)
+ with open(output_path, "w", encoding="utf-8") as stream:
+ stream.write(content)
+ return DocContentExportResponse(sha256=doc["sha256"], short_id=doc["short_id"], tier=request.tier, output=output_path)
+
+ async def get_document(self, doc_ref: str, *, param: str) -> DocRow:
+ """按完整或短文档标识查找记录,缺失时返回统一业务错误。"""
+ row = await self.find_document(doc_ref)
+ if row is None:
+ raise NotFoundError("doc_not_found", f"Document {doc_ref} not found.", param)
+ return row
+
+ async def find_document(self, doc_ref: str) -> DocRow | None:
+ """按完整或短文档标识查询记录,允许调用者处理不存在的文档。"""
+ row = cast(DocRow | None, await self.db.fetchone("SELECT * FROM docs WHERE sha256=?", (doc_ref,)))
+ if row is not None:
+ return row
+ return cast(DocRow | None, await self.db.fetchone("SELECT * FROM docs WHERE short_id=?", (doc_ref,)))
+
+ async def _build_read_plan_from_parse(
+ self,
+ doc_ref: str,
+ *,
+ tier: Tier,
+ page_range: str | None,
+ after: str | None,
+ limit: int,
+ format: str,
+ no_marker: bool,
+ ) -> _ReadPlan:
+ """校验解析内容请求并生成统一读取计划。"""
+ if format != "markdown":
+ raise InvalidRequestError("invalid_request", "Only markdown is currently implemented for parse content.", "format")
+ limit = max(1, limit)
+ doc = await self.get_document(doc_ref, param="doc_ref")
+
+ # 非 PDF 的省略范围表示整本,逻辑分节数不应触发 PDF 的前十页默认限制。
+ full_document = not is_page_range_parse_extension(doc["file_type"]) and not normalize_page_range_input(page_range)
+ normalized_page_range = None if full_document else _normalize_content_page_range(page_range, after, doc)
+ after_cursor = _parse_after_cursor(after)
+ if after_cursor:
+ _validate_cursor_for_doc(after_cursor, doc, tier, normalized_page_range)
+ return _ReadPlan(
+ sha256=doc["sha256"],
+ tier=tier,
+ short_id=doc["short_id"],
+ page_range=normalized_page_range,
+ after=after,
+ limit=limit,
+ format="markdown",
+ no_marker=no_marker,
+ locator=None,
+ context=0,
+ target=None,
+ next_mode="parse",
+ )
+
+ async def _build_read_plan_from_locator(
+ self,
+ locator: str,
+ *,
+ context: int,
+ limit: int,
+ format: ContentFormat,
+ image_format: ImageFormat,
+ no_marker: bool,
+ after: str | None = None,
+ ) -> _ReadPlan:
+ """将页选择解析为实际页范围,同时保持单点块和图片的定位约束。"""
+ limit = max(1, limit)
+ context = max(0, context)
+ cursor = _parse_doc_locator(locator)
+ doc = cast(DocRow | None, await self.db.fetchone("SELECT * FROM docs WHERE short_id=?", (cursor.short_id,)))
+ if doc is None:
+ raise NotFoundError("doc_not_found", f"Document {cursor.short_id} not found.", "locator")
+ if doc["file_type"] in TEXT_FILE_TYPES:
+ raise InvalidRequestError(
+ "parse_not_required",
+ "Text files do not require MinerU parsing. Read the file directly.",
+ "locator",
+ )
+ cursor = _resolve_locator_pages(cursor, doc)
+
+ tier = cursor.tier or await self._default_read_tier(doc["sha256"])
+ if tier is None:
+ raise NotFoundError(
+ "tier_not_cached",
+ f"No current parsed tier is cached for document {cursor.short_id}; reparse the source document.",
+ "locator",
+ )
+ if format == "image" and context:
+ raise InvalidRequestError("context_not_applicable", "context is not supported for image reads.", "context")
+ if cursor.page_range and count_pages_in_range(cursor.page_range) > 1:
+ if cursor.block_no is not None:
+ raise InvalidRequestError("invalid_locator", "Block and char locators require a single page.", "locator")
+ if format == "image":
+ raise InvalidRequestError("multi_page_image_not_supported", "image format supports only one page.", "locator")
+ if context:
+ raise InvalidRequestError("context_not_applicable", "context requires a single page.", "context")
+ if cursor.page_no is None and context:
+ raise InvalidRequestError("context_not_applicable", "context requires a page, block, or char locator.", "context")
+
+ page_range = _locator_page_range(cursor, doc, context)
+ if after is not None:
+ if cursor.page_range is None or cursor.block_no is not None or context or format != "markdown":
+ raise InvalidRequestError(
+ "invalid_request", "after requires a Markdown page selection without context.", "after"
+ )
+ after_cursor = _parse_after_cursor(after)
+ if after_cursor is None:
+ raise InvalidRequestError("invalid_locator", "after must be a non-empty content cursor.", "after")
+ _validate_cursor_for_doc(after_cursor, doc, tier, page_range)
+ return _ReadPlan(
+ sha256=doc["sha256"],
+ short_id=doc["short_id"],
+ tier=tier,
+ page_range=page_range,
+ after=after if after is not None else _locator_after(cursor, tier),
+ locator=_canonical_locator(doc["short_id"], tier, cursor),
+ context=context,
+ limit=limit,
+ format=format,
+ image_format=image_format,
+ no_marker=no_marker,
+ target=ContentCursor(
+ short_id=doc["short_id"],
+ tier=tier,
+ page_no=cursor.page_no or 1,
+ block_no=cursor.block_no,
+ char_offset=cursor.char_offset,
+ )
+ if cursor.page_no is not None
+ else None,
+ next_mode="range"
+ if (cursor.range_selection and cursor.block_no is None and not context) or after is not None
+ else "read",
+ )
+
+ async def _render_doc_content(
+ self,
+ sha256: str,
+ *,
+ tier: Tier,
+ page_range: str | None,
+ format: str,
+ no_marker: bool,
+ ) -> str:
+ """读取有效缓存页并生成完整导出 Markdown。"""
+ if format != "markdown":
+ raise InvalidRequestError("invalid_request", "Only markdown is currently implemented.", "format")
+ data_dir = self.data_dir
+ rows = await self.db.fetchall(
+ "SELECT page_range, done_at FROM parses WHERE sha256=? AND tier=? AND status=? ORDER BY done_at DESC",
+ (sha256, tier, PARSE_STATUS_DONE),
+ )
+ doc = cast(DocRow | None, await self.db.fetchone("SELECT * FROM docs WHERE sha256=?", (sha256,)))
+ if doc is None:
+ raise NotFoundError("doc_not_found", f"Document {sha256} not found.", "doc_ref")
+ page_range = normalize_page_range_input(page_range)
+ if page_range and doc["page_count"] is not None:
+ page_range = _expand_page_range(page_range, doc["page_count"])
+ requested = parse_page_range_set(page_range) if page_range and page_range != "all" else None
+ loaded_pages = load_pages_from_done_batches(
+ data_dir,
+ sha256,
+ tier,
+ cast(list[ParseBatchRow], rows),
+ requested_page_numbers=requested,
+ )
+ if requested is not None:
+ loaded_pages = [page for page in loaded_pages if page.page_idx + 1 in requested]
+ if not loaded_pages:
+ raise NotFoundError("not_cached", "Requested parsed content is not cached.", "page_range")
+ output: list[str] = []
+ page_count = doc["page_count"]
+ for page in loaded_pages:
+ if not no_marker:
+ output.append(_page_marker(page.page_idx + 1, page_count))
+ output.extend(
+ text
+ for _, text in _page_markdown_blocks(
+ page,
+ short_id=doc["short_id"],
+ tier=tier,
+ )
+ )
+ return "\n\n".join(output)
+
+ async def _execute_read_plan(self, plan: _ReadPlan) -> DocContentResponse:
+ """执行读取计划并构造分页、续读和素材响应。"""
+ data_dir = self.data_dir
+ rows = await self.db.fetchall(
+ "SELECT page_range, done_at FROM parses WHERE sha256=? AND tier=? AND status=? ORDER BY done_at DESC",
+ (plan.sha256, plan.tier, PARSE_STATUS_DONE),
+ )
+ loaded_pages = load_pages_from_done_batches(
+ data_dir,
+ plan.sha256,
+ plan.tier,
+ cast(list[ParseBatchRow], rows),
+ requested_page_numbers=parse_page_range_set(plan.page_range) if plan.page_range else None,
+ )
+ if plan.page_range:
+ loaded_pages = filter_pages_by_user_range(loaded_pages, plan.page_range)
+ if not loaded_pages:
+ raise NotFoundError("not_cached", "Requested parsed content is not cached.", "page_range")
+
+ doc = cast(DocRow | None, await self.db.fetchone("SELECT * FROM docs WHERE sha256=?", (plan.sha256,)))
+ if plan.format == "image":
+ return await self._render_image_response(plan, loaded_pages)
+
+ pages_for_render = _select_context_pages(loaded_pages, plan.target, plan.context)
+ rendered = _render_progressive_markdown(
+ pages_for_render,
+ short_id=plan.short_id,
+ tier=plan.tier,
+ after=_parse_after_cursor(plan.after),
+ limit=plan.limit,
+ add_markers=not plan.no_marker,
+ target=plan.target,
+ context=plan.context,
+ page_count=doc["page_count"] if doc else None,
+ )
+ if not rendered.content_ranges:
+ raise NotFoundError("not_cached", "Requested parsed content is not cached after cursor.", "after")
+
+ page_count = doc["page_count"] if doc else None
+ paginated = _is_paginated_doc(doc) if doc else True
+ if (
+ plan.next_mode == "parse"
+ and plan.page_range is None
+ and doc
+ and not is_page_range_parse_extension(doc["file_type"])
+ ):
+ # 整本格式仅用内容游标续读,避免生成它不接受的 --pages 解析参数。
+ paginated = False
+ if plan.next_mode == "range":
+ next_request = _next_range_read_request(rendered, plan.locator)
+ elif plan.next_mode == "read":
+ next_request = _next_read_request(rendered, plan.short_id, plan.tier, page_count)
+ else:
+ next_request = _next_content_request(
+ rendered=rendered,
+ request_page_range=plan.page_range,
+ after=plan.after,
+ page_count=page_count,
+ paginated=paginated,
+ )
+ return DocContentResponse(
+ sha256=plan.sha256,
+ short_id=plan.short_id,
+ tier=plan.tier,
+ format="markdown",
+ content=rendered.content,
+ request_scope=ContentRequestScope(
+ page_range=plan.page_range,
+ after=plan.after,
+ limit=plan.limit,
+ locator=plan.locator,
+ context=plan.context,
+ ),
+ content_ranges=rendered.content_ranges,
+ truncated=rendered.truncated,
+ next_request=next_request,
+ )
+
+ async def _render_image_response(
+ self,
+ plan: _ReadPlan,
+ loaded_pages: list[PageInfo],
+ ) -> DocContentResponse:
+ """解析页或块的图片来源并构造素材响应。"""
+ if plan.target is None or plan.target.page_no is None:
+ raise InvalidRequestError("format_not_supported", "image format requires a page or block locator.", "format")
+ if plan.page_range and len(parse_page_range_set(plan.page_range)) != 1:
+ raise InvalidRequestError("multi_page_image_not_supported", "image format supports only one page.", "locator")
+ page = _find_page(loaded_pages, plan.target.page_no)
+ if page is None:
+ raise NotFoundError("page_not_cached", f"Page {plan.target.page_no} is not cached.", "locator")
+
+ if plan.target.block_no is None:
+ asset = await self._render_source_image_asset(plan, page)
+ else:
+ block = _find_block_by_no(page, plan.target.block_no)
+ if block is None:
+ raise NotFoundError("block_not_found", f"Block {plan.target.block_no} not found.", "locator")
+ source = _resolve_block_image_source(block)
+ if source is None:
+ raise NotFoundError("asset_not_available", "Block image asset is not available.", "locator")
+ if source.kind == "source_bbox":
+ asset = await self._render_source_image_asset(plan, page)
+ else:
+ assert source.data_uri is not None
+ asset = self._render_base64_image_asset(plan, source.data_uri)
+
+ target_ref = plan.locator or page_ref(plan.short_id, plan.tier, plan.target.page_no)
+ return DocContentResponse(
+ sha256=plan.sha256,
+ short_id=plan.short_id,
+ tier=plan.tier,
+ format="image",
+ content="",
+ request_scope=ContentRequestScope(
+ page_range=plan.page_range,
+ after=plan.after,
+ limit=plan.limit,
+ locator=plan.locator,
+ context=plan.context,
+ image_format=plan.image_format,
+ ),
+ content_ranges=[ContentRange(page_range=str(plan.target.page_no), start=target_ref, end=target_ref)],
+ truncated=False,
+ next_request=None,
+ asset=asset,
+ )
+
+ async def _default_read_tier(self, sha256: str) -> Tier | None:
+ """从可读取的已完成缓存中选择最高质量档位。"""
+ rows = cast(
+ list[ParseRow],
+ await self.db.fetchall(
+ "SELECT * FROM parses WHERE sha256=? AND status=? ORDER BY done_at DESC",
+ (sha256, PARSE_STATUS_DONE),
+ ),
+ )
+ tiers: set[Tier] = {
+ row["tier"]
+ for row in rows
+ if row["tier"] in TIERS
+ and row["tier"] != "flash"
+ and parse_batch_is_readable(self.data_dir, sha256, row["tier"], row)
+ }
+ if not tiers:
+ return None
+ return select_highest_cached_tier(tiers)
+
+ async def _render_source_image_asset(self, plan: _ReadPlan, page: PageInfo) -> ContentAsset:
+ """查找源文件并调用公共图像输出,保留 Doclib 定位与错误协议。"""
+ if plan.target is None:
+ raise InvalidRequestError("format_not_supported", "image format requires a page or block locator.", "format")
+ file_row = await accessible_file_for_sha256(self.db, plan.sha256)
+ if file_row is None:
+ raise NotFoundError("no_accessible_file", "No active source file found for this document.", "locator")
+ source_path = file_row["path"]
+ source_ext = str(file_row.get("ext") or Path(source_path).suffix).lower().lstrip(".")
+ source_bytes = Path(source_path).read_bytes()
+ if source_ext == "pdf":
+ doc = PDFDocument(source_bytes)
+ elif source_ext in IMAGE_EXTENSIONS:
+ doc = PDFDocument.from_image(source_bytes)
+ else:
+ raise InvalidRequestError(
+ "format_not_supported",
+ "Page and bbox image output is only supported for PDF and image sources.",
+ "format",
+ )
+ with doc:
+ bbox = None
+ if plan.target.block_no is not None:
+ block = _find_block_by_no(page, plan.target.block_no)
+ if block is None:
+ raise NotFoundError("block_not_found", f"Block {plan.target.block_no} not found.", "locator")
+ if block.bbox is None or _is_empty_bbox(block.bbox):
+ raise InvalidRequestError("bbox_not_available", "Block bbox is not available for image output.", "locator")
+ bbox = block.bbox
+ artifact = doc.render_image(plan.target.page_no - 1, bbox=bbox, image_format=plan.image_format)
+ return _write_temp_asset(
+ self.data_dir,
+ plan.short_id,
+ artifact.extension,
+ artifact.data,
+ mime_type=artifact.mime_type,
+ width=artifact.width,
+ height=artifact.height,
+ )
+
+ def _render_base64_image_asset(self, plan: _ReadPlan, data_uri: str) -> ContentAsset:
+ """通过公共素材接口解码并转码,Doclib 只负责临时资产写出。"""
+ image_bytes, _extension = parse_image_data_uri_strict(data_uri)
+ artifact = transcode_image(image_bytes, image_format=plan.image_format)
+ return _write_temp_asset(
+ self.data_dir,
+ plan.short_id,
+ artifact.extension,
+ artifact.data,
+ mime_type=artifact.mime_type,
+ width=artifact.width,
+ height=artifact.height,
+ )
+
+
+_READ_LOCATOR_RE = re.compile(
+ r"^doc:(?P<short_id>[0-9a-fA-F]+)"
+ r"(?:/tier:(?P<tier>flash|basic|standard|advanced)"
+ r"(?:/page:(?P<page_range>[^/]+)"
+ r"(?:/block:(?P<block_no>[1-9][0-9]*)(?:/char:(?P<char_offset>0|[1-9][0-9]*))?)?)?)?$"
+)
+
+
+def _parse_doc_locator(locator: str) -> _LocatorParts:
+ """只解析输入语法,倒数页和范围在获取文档页数后统一求值。"""
+ match = _READ_LOCATOR_RE.match(locator)
+ if match is None:
+ raise InvalidRequestError("invalid_locator", f"Invalid doclib locator: {locator}", "locator")
+ page_range = match.group("page_range")
+ if page_range is not None:
+ if not page_range.strip():
+ raise InvalidRequestError("invalid_locator", "Page selection cannot be empty.", "locator")
+ try:
+ normalized_range = normalize_page_range_input(page_range)
+ except InvalidRequestError as error:
+ raise InvalidRequestError("invalid_locator", error.message, "locator") from error
+ else:
+ normalized_range = None
+ block_no = match.group("block_no")
+ char_offset = match.group("char_offset")
+ return _LocatorParts(
+ short_id=match.group("short_id"),
+ tier=cast(Tier | None, match.group("tier")),
+ page_no=int(normalized_range) if normalized_range and normalized_range.isdecimal() else None,
+ block_no=int(block_no) if block_no is not None else None,
+ char_offset=int(char_offset) if char_offset is not None else None,
+ page_range=normalized_range,
+ range_selection=page_range is not None and (page_range.strip() == "all" or "," in page_range or "-" in page_range),
+ )
+
+
+def _resolve_locator_pages(locator: _LocatorParts, doc: DocRow) -> _LocatorParts:
+ """复用公共页范围算法求值;倒数页和 all 禁止用部分缓存猜测总页数。"""
+ if locator.page_range is None:
+ return locator
+ page_count = doc.get("page_count")
+ if (page_count is None or page_count < 1) and ("r" in locator.page_range or locator.page_range == "all"):
+ raise InvalidRequestError(
+ "invalid_locator", "Cannot resolve all/rN because the document page count is unavailable.", "locator"
+ )
+ resolved_range = _expand_page_range(locator.page_range, page_count or 1)
+ single_page = count_pages_in_range(resolved_range) == 1
+ return replace(locator, page_range=resolved_range, page_no=int(resolved_range) if single_page else None)
+
+
+def _canonical_locator(short_id: str, tier: Tier, locator: _LocatorParts) -> str:
+ """将已解析的定位信息编码为统一的文档引用。"""
+ if locator.range_selection and locator.page_range is not None and locator.block_no is None:
+ return f"doc:{short_id}/tier:{tier}/page:{locator.page_range}"
+ if locator.page_no is None:
+ return f"doc:{short_id}/tier:{tier}"
+ if locator.block_no is None:
+ return page_ref(short_id, tier, locator.page_no)
+ if locator.char_offset is None:
+ return block_ref(short_id, tier, locator.page_no, locator.block_no)
+ return block_char_ref(short_id, tier, locator.page_no, locator.block_no, locator.char_offset)
+
+
+def _locator_after(locator: _LocatorParts, tier: Tier) -> str | None:
+ """仅为字符级定位构造块内续读游标。"""
+ if locator.page_no is None:
+ return None
+ if locator.block_no is None:
+ return None
+ return _canonical_locator(locator.short_id, tier, locator) if locator.char_offset is not None else None
+
+
+def _locator_page_range(locator: _LocatorParts, doc: DocRow, context: int) -> str | None:
+ """按定位页和上下文范围确定实际读取页码。"""
+ page_count = doc.get("page_count") or 1
+ if locator.page_range is not None and not context:
+ return locator.page_range
+ if locator.page_no is None:
+ return _normalize_content_page_range(None, None, doc)
+ if locator.page_no > page_count:
+ return _normalize_page_range(str(locator.page_no), page_count)
+ start = max(1, locator.page_no - context)
+ end = min(page_count, locator.page_no + context)
+ return _range_str(start, end)
+
+
+def _parse_after_cursor(after: str | None) -> ContentCursor | None:
+ """解析续读游标并转换为 Doclib 的参数错误。"""
+ if not after:
+ return None
+ try:
+ return parse_content_cursor(after)
+ except ValueError as exc:
+ raise InvalidRequestError("invalid_locator", str(exc), "after") from None
+
+
+def _validate_cursor_for_doc(cursor: ContentCursor, doc: DocRow, tier: Tier, page_range: str | None) -> None:
+ """验证续读游标属于指定文档、档位和页范围。"""
+ if cursor.short_id != doc["short_id"]:
+ raise InvalidRequestError("invalid_request", "after cursor does not belong to this document.", "after")
+ if cursor.tier != tier:
+ raise InvalidRequestError("invalid_request", "after cursor tier does not match request tier.", "after")
+ if page_range and cursor.page_no not in parse_page_range_set(page_range):
+ raise InvalidRequestError("invalid_request", "after cursor is outside requested page_range.", "after")
+
+
+def _is_paginated_doc(doc: DocRow) -> bool:
+ """根据已知页面总数判断文档是否需要分页读取。"""
+ page_count = doc.get("page_count")
+ return page_count is not None and page_count > 1
+
+
+def _normalize_content_page_range(page_range: str | None, after: str | None, doc: DocRow) -> str | None:
+ """统一内容页范围并保留默认前十页与游标续读规则。"""
+ page_range = normalize_page_range_input(page_range)
+ page_count = doc.get("page_count") or 1
+ if _is_paginated_doc(doc):
+ if page_range and page_range.strip() == "all":
+ return _expand_page_range("all", page_count)
+ if page_range:
+ return _normalize_page_range(page_range, page_count)
+ if after:
+ cursor = _parse_after_cursor(after)
+ assert cursor is not None
+ end = min(page_count, cursor.page_no + 9)
+ return _range_str(cursor.page_no, end)
+ end = min(page_count, 10)
+ return _range_str(1, end)
+ if page_range and page_range.strip() == "all":
+ return None
+ return _normalize_page_range(page_range, page_count) if page_range else None
+
+
+def _normalize_page_range(page_range: str, page_count: int) -> str:
+ """复用公共页范围算法获得规范化的一基页码范围。"""
+ return _expand_page_range(page_range, page_count)
+
+
+def _select_context_pages(pages: list[PageInfo], target: ContentCursor | None, context: int) -> list[PageInfo]:
+ """按目标页与上下文数量筛选已加载页面。"""
+ if target is None or target.page_no is None or context <= 0:
+ return pages
+ start = target.page_no - context
+ end = target.page_no + context
+ return [page for page in pages if start <= page.page_idx + 1 <= end]
+
+
+def _find_page(pages: list[PageInfo], page_no: int) -> PageInfo | None:
+ """从缓存页面中查找指定的一基页码。"""
+ return next((page for page in pages if page.page_idx + 1 == page_no), None)
+
+
+def _find_block_by_no(page: PageInfo, block_no: int) -> PageBlock | None:
+ """将一基块号映射为页面中的顶层块。"""
+ target_index = block_no - 1
+ return next((block for block in page.blocks if block.index == target_index), None)
+
+
+def _is_empty_bbox(bbox: object) -> bool:
+ """验证边框是否为有限数值组成的有效矩形。"""
+ if not isinstance(bbox, (tuple, list)) or len(bbox) != 4:
+ return True
+ try:
+ values = [float(v) for v in bbox]
+ except (TypeError, ValueError):
+ return True
+ if not all(math.isfinite(value) for value in values):
+ return True
+ return values[0] >= values[2] or values[1] >= values[3]
+
+
+def _resolve_block_image_source(block: BlockBase) -> _BlockImageSource | None:
+ """优先选择原文矩形裁图,其次使用已物化的内嵌图片。"""
+ if not _is_empty_bbox(block.bbox):
+ return _BlockImageSource(kind="source_bbox")
+ for payload in iter_image_payloads(block):
+ if payload.image_base64:
+ return _BlockImageSource(kind="base64", data_uri=payload.image_base64)
+ return None
+
+
+def _make_doclib_image_renderer(
+ *,
+ short_id: str,
+ tier: Tier,
+ page_no: int,
+) -> Callable[[BlockBase], str]:
+ """构造输出 Doclib 素材定位符的 Markdown 图片渲染器。"""
+
+ def _render(block: BlockBase) -> str:
+ """将可读取图片转换为块定位符,并保留远端图片链接。"""
+ label = _VISUAL_BLOCK_LABELS.get(block.type, "Image block")
+ source = _resolve_block_image_source(block)
+ if source is not None:
+ block_no = block.index + 1 if block.index is not None else 0 # TODO: change after block.index's type changed.
+ return build_markdown_image(block_ref(short_id, tier, page_no, block_no), label)
+ remote_url = next((payload.image_url for payload in iter_image_payloads(block) if payload.image_url), None)
+ return build_markdown_image(remote_url, label) if remote_url is not None else f"![{label}]()"
+
+ return _render
+
+
+def _write_temp_asset(
+ data_dir: str,
+ short_id: str,
+ ext: str,
+ image_bytes: bytes,
+ *,
+ mime_type: str,
+ width: int | None,
+ height: int | None,
+) -> ContentAsset:
+ """写出临时图片文件并返回包含尺寸与媒体类型的素材信息。"""
+ safe_ext = ext.lower().lstrip(".") or "png"
+ output_dir = Path(os.path.expanduser(data_dir)) / "temp" / "read-assets"
+ output_dir.mkdir(parents=True, exist_ok=True)
+ path = output_dir / f"{short_id}-{int(time.time() * 1000)}.{safe_ext}"
+ path.write_bytes(image_bytes)
+ return ContentAsset(
+ path=str(path),
+ mime_type=mime_type,
+ size_bytes=len(image_bytes),
+ width=width,
+ height=height,
+ )
+
+
+def _render_progressive_markdown(
+ pages: list[PageInfo],
+ *,
+ short_id: str,
+ tier: Tier,
+ after: ContentCursor | None,
+ limit: int,
+ add_markers: bool,
+ target: ContentCursor | None = None,
+ context: int = 0,
+ page_count: int | None = None,
+) -> _RenderedContent:
+ """在页面与块边界渐进生成 Markdown,并记录截断位置。"""
+ output: list[str] = []
+ ranges: list[ContentRange] = []
+ started = after is None
+ last_page_no = 0
+ next_page_no: int | None = None
+ cut_inside_page = False
+
+ for page in pages:
+ page_no = page.page_idx + 1
+ if after and page_no < after.page_no:
+ continue
+ page_blocks = _page_markdown_blocks(
+ page,
+ short_id=short_id,
+ tier=tier,
+ )
+ if target and target.block_no is not None and page_no == target.page_no:
+ start_block_no = max(1, target.block_no - context)
+ end_block_no = target.block_no + context
+ page_blocks = [(index, text) for index, text in page_blocks if start_block_no <= index + 1 <= end_block_no]
+ page_output: list[str] = []
+ page_start_cursor = page_ref(short_id, tier, page_no)
+ page_end_cursor = page_ref(short_id, tier, page_no)
+ page_started = False
+ empty_page_selected = False
+
+ for block_index, block_text in page_blocks:
+ block_no = block_index + 1
+ if after and page_no == after.page_no:
+ if after.block_no is None:
+ continue
+ if block_no < after.block_no:
+ continue
+ if block_no == after.block_no and after.char_offset is None:
+ continue
+ started = True
+ text = block_text
+ block_start_offset = 0
+ if after and page_no == after.page_no and after.block_no == block_no and after.char_offset is not None:
+ block_start_offset = after.char_offset
+ text = text[block_start_offset:]
+ if not text:
+ continue
+ block_start_cursor = (
+ block_char_ref(short_id, tier, page_no, block_no, block_start_offset)
+ if block_start_offset
+ else block_ref(short_id, tier, page_no, block_no)
+ )
+ if not page_started:
+ page_start_cursor = block_start_cursor if after and after.block_no is not None else page_start_cursor
+ page_started = True
+ candidate_page = [*page_output, text]
+ candidate_content = _join_markdown([*output, *candidate_page])
+ if len(candidate_content) <= limit:
+ page_output.append(text)
+ page_end_cursor = block_ref(short_id, tier, page_no, block_no)
+ continue
+
+ if page_output and not output:
+ cut_inside_page = True
+ break
+ if output:
+ next_page_no = page_no
+ return _RenderedContent(
+ content=_join_markdown(output),
+ content_ranges=_merge_content_ranges(ranges),
+ truncated=True,
+ last_page_no=last_page_no,
+ next_page_no=next_page_no,
+ cut_inside_page=False,
+ )
+
+ cut = _soft_cut(text, limit)
+ if cut <= 0:
+ cut = min(len(text), limit)
+ page_output.append(text[:cut])
+ end_offset = block_start_offset + cut
+ page_end_cursor = block_char_ref(short_id, tier, page_no, block_no, end_offset)
+ cut_inside_page = True
+ break
+
+ if not page_output and after and page_no == after.page_no:
+ continue
+ if not page_output and (target is None or target.block_no is None):
+ started = True
+ empty_page_selected = True
+ if add_markers:
+ page_output.append(_page_marker(page_no, page_count))
+
+ if not started or (not page_output and not empty_page_selected):
+ continue
+ if add_markers:
+ marker = _page_marker(page_no, page_count)
+ if not page_output or page_output[0] != marker:
+ page_output.insert(0, marker)
+ if page_output:
+ output.extend(page_output)
+ last_page_no = page_no
+ ranges.append(ContentRange(page_range=str(page_no), start=page_start_cursor, end=page_end_cursor))
+ if cut_inside_page:
+ next_page_no = page_no
+ return _RenderedContent(
+ content=_join_markdown(output),
+ content_ranges=_merge_content_ranges(ranges),
+ truncated=True,
+ last_page_no=last_page_no,
+ next_page_no=next_page_no,
+ cut_inside_page=True,
+ )
+ # Keep whole pages as the normal truncation boundary for paginated docs.
+ if len(_join_markdown(output)) >= limit:
+ next_page_no = page_no + 1
+ return _RenderedContent(
+ content=_join_markdown(output),
+ content_ranges=_merge_content_ranges(ranges),
+ truncated=next_page_no <= (pages[-1].page_idx + 1),
+ last_page_no=last_page_no,
+ next_page_no=next_page_no,
+ cut_inside_page=False,
+ )
+
+ return _RenderedContent(
+ content=_join_markdown(output),
+ content_ranges=_merge_content_ranges(ranges),
+ truncated=False,
+ last_page_no=last_page_no,
+ next_page_no=None,
+ cut_inside_page=False,
+ )
+
+
+def _page_marker(page_no: int, page_count: int | None) -> str:
+ """按已知总页数生成页标记。"""
+ if page_count is None:
+ return f"<!-- page {page_no} -->"
+ return f"<!-- page {page_no} of {page_count} -->"
+
+
+def _page_markdown_blocks(
+ page: PageInfo,
+ *,
+ short_id: str = "",
+ tier: Tier = "standard",
+) -> list[tuple[int, str]]:
+ """使用共享渲染器和宿主公式配置生成顶层块内容。"""
+ result: list[tuple[int, str]] = []
+ image_renderer = _make_doclib_image_renderer(
+ short_id=short_id,
+ tier=tier,
+ page_no=page.page_idx + 1,
+ )
+ delimiters = get_config().render.latex_delimiters
+ for block in page.blocks:
+ rendered = render_single_block(
+ block,
+ delimiters=delimiters,
+ asset_base_url="",
+ image_renderer=image_renderer,
+ )
+ text = rendered.strip() if rendered else ""
+ if text:
+ # TODO: simplify after BlockBase.index type narrows to int (top-level blocks always have index)
+ block_index = block.index if block.index is not None else -1
+ result.append((block_index, text))
+ return result
+
+
+def _join_markdown(parts: list[str]) -> str:
+ """以段落分隔符连接非空 Markdown 片段。"""
+ return "\n\n".join(part for part in parts if part)
+
+
+def _soft_cut(text: str, limit: int) -> int:
+ """在字符预算内优先选择自然文本分隔位置。"""
+ if len(text) <= limit:
+ return len(text)
+ window = text[:limit]
+ for pattern in ("\n\n", "\n", "。", ". ", " "):
+ pos = window.rfind(pattern)
+ if pos > max(0, limit // 2):
+ return pos + len(pattern)
+ return limit
+
+
+def _merge_content_ranges(ranges: list[ContentRange]) -> list[ContentRange]:
+ """合并相邻页面的内容引用范围。"""
+ if not ranges:
+ return []
+ merged: list[ContentRange] = []
+ current = ranges[0]
+ for item in ranges[1:]:
+ if current.page_range and item.page_range:
+ current_page_numbers = parse_page_range_set(current.page_range)
+ item_page_numbers = parse_page_range_set(item.page_range)
+ if max(current_page_numbers) + 1 == min(item_page_numbers):
+ current = ContentRange(
+ page_range=_page_numbers_to_range_str(current_page_numbers | item_page_numbers),
+ start=current.start,
+ end=item.end,
+ )
+ continue
+ merged.append(current)
+ current = item
+ merged.append(current)
+ return merged
+
+
+def _next_content_request(
+ *,
+ rendered: _RenderedContent,
+ request_page_range: str | None,
+ after: str | None,
+ page_count: int | None,
+ paginated: bool,
+) -> ContentNextRequest | None:
+ """结合页范围与截断位置生成解析内容的续读请求。"""
+ if not rendered.content_ranges:
+ return None
+ if not paginated:
+ if rendered.truncated:
+ return ContentNextRequest(after=rendered.content_ranges[-1].end)
+ return None
+
+ total_page_count = page_count or rendered.last_page_no
+ if rendered.truncated:
+ if rendered.cut_inside_page:
+ start = rendered.next_page_no or rendered.last_page_no
+ end = _last_requested_page(request_page_range) or start
+ return ContentNextRequest(page_range=_range_str(start, end), after=rendered.content_ranges[-1].end)
+ start = rendered.next_page_no or (rendered.last_page_no + 1)
+ end = _last_requested_page(request_page_range) or min(total_page_count, start + 9)
+ if start <= end:
+ return ContentNextRequest(page_range=_range_str(start, end))
+ if after:
+ return ContentNextRequest(page_range=request_page_range, after=rendered.content_ranges[-1].end)
+ return None
+
+ start = rendered.last_page_no + 1
+ if start > total_page_count:
+ return None
+ end = min(total_page_count, start + 9)
+ return ContentNextRequest(page_range=_range_str(start, end))
+
+
+def _next_read_request(
+ rendered: _RenderedContent,
+ short_id: str,
+ tier: Tier,
+ page_count: int | None,
+) -> ContentNextRequest | None:
+ """为定位符读取生成后续块或下一页请求。"""
+ if not rendered.content_ranges:
+ return None
+ if rendered.truncated:
+ return ContentNextRequest(locator=rendered.content_ranges[-1].end)
+ total_page_count = page_count or rendered.last_page_no
+ next_page_no = rendered.last_page_no + 1
+ if next_page_no > total_page_count:
+ return None
+ return ContentNextRequest(locator=page_ref(short_id, tier, next_page_no))
+
+
+def _next_range_read_request(rendered: _RenderedContent, locator: str | None) -> ContentNextRequest | None:
+ """保留范围定位器并推进实际页码游标,避免续读扩入未选择的页面。"""
+ if not rendered.truncated or not rendered.content_ranges or locator is None:
+ return None
+ return ContentNextRequest(locator=locator, after=rendered.content_ranges[-1].end)
+
+
+def _last_requested_page(page_range: str | None) -> int | None:
+ """返回请求范围中的最后一页。"""
+ if not page_range:
+ return None
+ page_numbers = parse_page_range_set(page_range)
+ return max(page_numbers) if page_numbers else None
+
+
+def _range_str(start: int, end: int) -> str:
+ """将起止页码转换为规范化范围字符串。"""
+ return _page_numbers_to_range_str(range(start, end + 1))
+
+
+__all__ = ["ContentReadService"]
diff --git a/mineru/doclib/services/parse_svc.py b/mineru/doclib/services/parse_svc.py
index 05ff4684..e8dbe061 100644
--- a/mineru/doclib/services/parse_svc.py
+++ b/mineru/doclib/services/parse_svc.py
@@ -3,11 +3,9 @@
from __future__ import annotations
import asyncio
-import json
import logging
import os
import time
-from collections.abc import Sequence
from dataclasses import dataclass
from pathlib import Path
from typing import Literal, cast
@@ -28,7 +26,6 @@ from ...filetypes import (
)
from ...parser.api_client import _APITransportError, _V1APIError
from ...parser.base import ParseResult
-from ...backend.postprocess.legacy_middle_json import read_legacy_middle_json
from ...parser.page_range import (
expand_page_range,
format_page_range as _page_numbers_to_range_str,
@@ -36,8 +33,17 @@ from ...parser.page_range import (
normalize_result_page_range,
parse_page_range_set,
)
-from ...types import QUALITY_TIERS, TIER_ORDER, PageInfo, Tier, select_default_quality_tier, select_parsing_rule_tier
+from ...types import QUALITY_TIERS, TIER_ORDER, Tier, select_default_quality_tier, select_parsing_rule_tier
from ..core.db import DatabaseManager
+from ..core.parse_artifacts import (
+ accessible_file_for_sha256,
+ load_pages_from_done_batches,
+ parse_artifact_directory,
+ parse_batch_is_readable,
+ parse_batch_json_path,
+ parse_coverage,
+ write_parse_payload,
+)
from ..core.file_io import (
FileStat,
MetadataExtractionError,
@@ -48,7 +54,7 @@ from ..core.file_io import (
)
from ..core.fts import FTSManager
from ..remote_api import resolve_remote_api_key
-from ..rows import DocRow, FileRow, ParseBatchRow, ParseRow, Sha256Row, ShortIdRow, WatchTargetRow
+from ..rows import DocRow, FileRow, ParseRow, Sha256Row, ShortIdRow, WatchTargetRow
from ..types import (
FILE_STATUS_ACTIVE,
FILE_STATUS_DELETED,
@@ -85,26 +91,6 @@ MAX_FTS_CHARS = 30_000
FTS_HEAD_HALF = 15_000
-async def accessible_file_for_sha256(db: DatabaseManager, sha256: str) -> FileRow | None:
- """Find the newest active file row whose path still exists.
-
- The same content (sha256) may be tracked at several paths; rows whose
- source file was moved away stay active until a refresh marks them
- deleted, so existence must be checked before trusting a path.
- """
- rows = cast(
- list[FileRow],
- await db.fetchall(
- "SELECT * FROM files WHERE sha256=? AND status=? ORDER BY updated_at DESC",
- (sha256, FILE_STATUS_ACTIVE),
- ),
- )
- for row in rows:
- if os.path.exists(row["path"]):
- return row
- return None
-
-
FileRefreshStatus = Literal["known", "new", "changed", "missing", "deleted", "unreachable", "unsupported", "error"]
@@ -139,12 +125,6 @@ def _raise_refresh_error(refreshed: FileRefreshResult) -> None:
# ── range helpers ──────────────────────────────────────────────────
-def filter_pages_by_user_range(pages: list[PageInfo], page_range: str) -> list[PageInfo]:
- """Filter 0-based PageInfo objects by a user-facing 1-based page range."""
- requested_page_numbers = parse_page_range_set(page_range)
- return [page for page in pages if page.page_idx + 1 in requested_page_numbers]
-
-
def page_range_covered(request_page_range: str, done_batches: list[ParseRow]) -> bool:
"""Check whether request_page_range is fully covered by done batches.
Batches are sorted by done_at DESC so newer batches take precedence."""
@@ -758,7 +738,7 @@ class ParseService:
),
)
valid_done_batches = [
- batch for batch in done_batches if _json_batch_is_readable(self.data_dir, sha256, requested_tier, batch)
+ batch for batch in done_batches if parse_batch_is_readable(self.data_dir, sha256, requested_tier, batch)
]
if supports_page_range:
for batch in valid_done_batches:
@@ -915,7 +895,7 @@ class ParseService:
return False
file_ext = file_row.get("ext") or Path(file_row["path"]).suffix.lower().lstrip(".")
- output_dir = os.path.join(self.data_dir, "parsed", sha256[:2], sha256, tier)
+ output_dir = parse_artifact_directory(self.data_dir, sha256, tier)
# route parse based on tier
via = "local"
@@ -998,12 +978,11 @@ class ParseService:
# save per-batch JSON (markdown is generated on read from /docs/{doc_ref}/content)
done_at_ms = _now_ms()
- json_path = os.path.join(output_dir, _safe_filename(persisted_page_range, done_at_ms))
+ json_path = parse_batch_json_path(self.data_dir, sha256, tier, persisted_page_range, done_at_ms)
write_start_ms = _now_ms()
try:
os.makedirs(output_dir, exist_ok=True)
- with open(json_path, "w", encoding="utf-8") as f:
- json.dump(export_payload, f, ensure_ascii=False, indent=2)
+ write_parse_payload(json_path, export_payload)
except Exception as exc:
logger.error(
"Parse post-write finalization failed for task_id=%s path=%s tier=%s page_range=%s: %s",
@@ -1346,7 +1325,7 @@ class ParseService:
doc = await self.db.fetchone("SELECT * FROM docs WHERE sha256=?", (sha256,))
if doc is None:
raise InvalidRequestError("doc_not_found", f"Document {sha256} not found.", "doc_ref")
- result["coverage"] = _parse_coverage(expand_page_range(page_range, doc["page_count"] or 1), rows)
+ result["coverage"] = parse_coverage(expand_page_range(page_range, doc["page_count"] or 1), rows).model_dump()
return result
async def invalidate(self, sha256: str, tier: Tier | None = None) -> int:
@@ -1441,16 +1420,16 @@ class ParseService:
tiers = sorted(tier_set, key=lambda item: TIER_ORDER.get(item, -1), reverse=True)
for tier in tiers:
tier_rows = [
- row for row in rows if row["tier"] == tier and _json_batch_is_readable(self.data_dir, sha256, tier, row)
+ row for row in rows if row["tier"] == tier and parse_batch_is_readable(self.data_dir, sha256, tier, row)
]
pages = load_pages_from_done_batches(self.data_dir, sha256, tier, tier_rows)
if not pages:
continue
- from ...config import config
+ from ...config import get_config
from docvortex.render.markdown import render_single_block
- delimiters = config.render.latex_delimiters
+ delimiters = get_config().render.latex_delimiters
parts: list[str] = []
for page in pages:
for block in page.blocks:
@@ -1533,57 +1512,6 @@ def _probe_error_message(prefix: str, error_msg: str | None) -> str:
return prefix
-def _json_batch_is_readable(data_dir: str, sha256: str, tier: Tier, batch: ParseRow) -> bool:
- """将可转换为当前协议且覆盖记录页范围的持久化批次视为有效缓存。"""
- json_path = parse_batch_json_path(data_dir, sha256, tier, batch["page_range"], batch["done_at"])
- try:
- with open(json_path, encoding="utf-8") as stream:
- result = read_legacy_middle_json(json.load(stream))
- available = {page.page_idx + 1 for page in result.pages}
- return bool(available) and parse_page_range_set(batch["page_range"]) <= available
- except (OSError, ValueError, TypeError):
- return False
-
-
-def parse_batch_json_path(data_dir: str, sha256: str, tier: Tier, page_range: str, done_at: int | None = 0) -> str:
- """Return the persisted Middle JSON path for one parse batch."""
- filename = _safe_filename(page_range, done_at or 0)
- return os.path.join(os.path.expanduser(data_dir), "parsed", sha256[:2], sha256, tier, filename)
-
-
-def load_pages_from_done_batches(
- data_dir: str,
- sha256: str,
- tier: Tier,
- done_rows: Sequence[ParseBatchRow],
- *,
- requested_page_numbers: set[int] | None = None,
-) -> list[PageInfo]:
- """读取当前或受支持的历史批次,保留最新重复页;损坏或未知缓存须重新解析。"""
- pages_by_page_idx: dict[int, PageInfo] = {}
- stale_page_numbers: set[int] = set()
- for row in reversed(done_rows):
- fpath = parse_batch_json_path(data_dir, sha256, tier, row["page_range"], row["done_at"])
- if not os.path.isfile(fpath):
- continue
- try:
- with open(fpath, encoding="utf-8") as f:
- data = json.load(f)
- parse_result = read_legacy_middle_json(data)
- for page in parse_result.pages:
- pages_by_page_idx[page.page_idx] = page
- except (OSError, ValueError, TypeError):
- stale_page_numbers.update(parse_page_range_set(row["page_range"]))
- uncovered_stale = stale_page_numbers - {index + 1 for index in pages_by_page_idx}
- if requested_page_numbers is not None:
- uncovered_stale &= requested_page_numbers
- if uncovered_stale:
- raise MineruError(
- "stale_cache", "Cached document protocol is outdated or invalid; reparse the source document.", "page_range"
- )
- return [pages_by_page_idx[page_idx] for page_idx in sorted(pages_by_page_idx)]
-
-
def _done_response(sha256: str, short_id: str | None, tier: Tier, page_range: str) -> ParseResponse:
return ParseResponse(
sha256=sha256,
@@ -1653,25 +1581,6 @@ def _remap_api_result_pages_to_page_range(result: ParseResult, page_range: str)
extensions[LAYOUT_EXTENSION] = geometry
-def _parse_coverage(request_page_range: str, rows: list[ParseRow]) -> dict:
- requested_page_numbers = parse_page_range_set(request_page_range)
- done_page_numbers: set[int] = set()
- active_page_numbers: set[int] = set()
- for row in rows:
- row_page_numbers = parse_page_range_set(row["page_range"]) & requested_page_numbers
- if row["status"] == PARSE_STATUS_DONE:
- done_page_numbers |= row_page_numbers
- elif row["status"] in (PARSE_STATUS_PENDING, PARSE_STATUS_PARSING):
- active_page_numbers |= row_page_numbers
- active_page_numbers -= done_page_numbers
- missing_page_numbers = requested_page_numbers - done_page_numbers - active_page_numbers
- return {
- "done_page_range": _page_numbers_to_range_str(done_page_numbers),
- "active_page_range": _page_numbers_to_range_str(active_page_numbers),
- "missing_page_range": _page_numbers_to_range_str(missing_page_numbers),
- }
-
-
def _local_parse_server_url(mode: str, health: object) -> str | None:
"""Resolve local parse-server URL from mode and health state."""
if mode == "managed":
@@ -1696,11 +1605,6 @@ def _file_info(row: FileRow | None) -> FileInfo | None:
return FileInfo.model_validate(row) if row is not None else None
-def _safe_filename(page_range: str, done_at: int) -> str:
- """Convert a page_range string + done_at to a filename."""
- return f"{page_range}_{done_at}.json" if done_at else page_range
-
-
def _unsupported_file_type_message(ext_or_name: str) -> str:
display = ext_or_name if ext_or_name.startswith(".") else Path(ext_or_name).name
return f"File type is not supported for parsing: {display}"
diff --git a/mineru/kit/commands/models.py b/mineru/kit/commands/models.py
index ca720115..b5e1fcc9 100644
--- a/mineru/kit/commands/models.py
+++ b/mineru/kit/commands/models.py
@@ -4,7 +4,7 @@ import os
import typer
-from ...config import config, get_config_file_exists, get_config_file_path, get_config_source
+from ...config import get_config, get_config_file_exists, get_config_file_path, get_config_source
from ...model.download import (
DOWNLOAD_MODEL_SOURCES,
MODEL_SOURCE_ENV_VAR,
@@ -132,7 +132,7 @@ def show_cmd(
try:
effective_small_backend = resolve_small_model_backend(small_backend)
effective_vlm_engine = (
- "http-client" if vlm_engine is None and config.model.vlm.server_url else resolve_vlm_engine(vlm_engine)
+ "http-client" if vlm_engine is None and get_config().model.vlm.server_url else resolve_vlm_engine(vlm_engine)
)
except ValueError as exc:
exit_with_message("invalid_request", str(exc), "backend")
@@ -142,15 +142,15 @@ def show_cmd(
t("Config: {path}", path=config_file),
t("Config exists: {value}", value=str(get_config_file_exists()).lower()),
f"MINERU_MODEL_SOURCE={os.getenv(MODEL_SOURCE_ENV_VAR, '') or '(unset)'}",
- f"model.base_dir: {config.model.base_dir}",
+ f"model.base_dir: {get_config().model.base_dir}",
f"model.base_dir.source: {get_config_source('model.base_dir')}",
- f"model.source: {config.model.source}",
+ f"model.source: {get_config().model.source}",
f"model.source.source: {get_config_source('model.source')}",
- f"model.small_backend: {config.model.small_backend}",
+ f"model.small_backend: {get_config().model.small_backend}",
f"model.small_backend.source: {get_config_source('model.small_backend')}",
- f"model.onnx_device: {config.model.onnx_device}",
+ f"model.onnx_device: {get_config().model.onnx_device}",
f"model.onnx_device.source: {get_config_source('model.onnx_device')}",
- f"model.vlm.engine: {config.model.vlm.engine}",
+ f"model.vlm.engine: {get_config().model.vlm.engine}",
f"model.vlm.engine.source: {get_config_source('model.vlm.engine')}",
t("Effective small backend: {backend}", backend=effective_small_backend),
t("Effective VLM engine: {engine}", engine=effective_vlm_engine),
diff --git a/mineru/kit/file_parse/models.py b/mineru/kit/file_parse/models.py
index 1524f1a1..2daec9b6 100644
--- a/mineru/kit/file_parse/models.py
+++ b/mineru/kit/file_parse/models.py
@@ -5,14 +5,14 @@ from __future__ import annotations
import hashlib
import json
from dataclasses import asdict, dataclass
-from typing import Any, Literal
+from typing import Any
-from pydantic import BaseModel, ConfigDict, Field, model_validator
+from pydantic import BaseModel, Field, model_validator
-from ...task_types import TaskFileState, TaskOutputFormat, TaskStatus, task_status_url
+from ...task_types import TaskOutputFormat, TaskStatus, task_status_url
+from ...tasks.contracts import TaskFileIdentity, TaskUploadLink, UploadTaskOptions
TERMINAL = frozenset({"completed", "failed", "canceled", "submission_unknown"})
-FileStatus = TaskFileState
class GatewayError(Exception):
@@ -40,16 +40,9 @@ class GatewayError(Exception):
return {"code": self.code, "message": self.message, "upstream_status": self.upstream_status}
-class ParseOptions(BaseModel):
+class ParseOptions(UploadTaskOptions):
"""保持与 Tasks 一致的解析参数,等待时间不属于解析身份。"""
- model_config = ConfigDict(extra="forbid")
-
- tier: Literal["flash", "basic", "standard", "advanced"] | None = None
- ocr_mode: Literal["auto", "txt", "ocr"] = "auto"
- page_range: str = ""
- output_formats: list[TaskOutputFormat] = Field(default_factory=lambda: ["default"], min_length=1)
-
@model_validator(mode="after")
def normalize(self) -> ParseOptions:
"""按首次出现去重格式,并统一整本页范围的等价写法。"""
@@ -60,22 +53,11 @@ class ParseOptions(BaseModel):
return self
-class UploadLink(BaseModel):
- """保留文件内嵌上传地址、请求头和固定截止时间。"""
-
- url: str
- headers: dict[str, str]
- expires_at: int
-
-
-class RemoteFile(BaseModel):
+class RemoteFile(TaskFileIdentity):
"""仅消费共同 Tasks 契约,不依赖本地 Files 或 Job 实现。"""
- file_id: str = Field(min_length=1)
- file_name: str
bytes: int
- status: FileStatus
- upload: UploadLink | None = None
+ upload: TaskUploadLink | None = None
result_url: str | None = None
error: dict[str, Any] | None = None
@@ -103,13 +85,10 @@ class RemoteTask(BaseModel):
return self
-class RemoteCreatedFile(BaseModel):
+class RemoteCreatedFile(TaskFileIdentity):
"""只消费创建回执中的文件身份、状态与上传描述。"""
- file_id: str = Field(min_length=1)
- file_name: str
- status: FileStatus
- upload: UploadLink | None = None
+ upload: TaskUploadLink | None = None
class RemoteCreatedTask(BaseModel):
@@ -158,4 +137,4 @@ def fingerprint(filename: str, size: int, sha256sum: str, options: ParseOptions)
return hashlib.sha256(json.dumps(value, ensure_ascii=False, sort_keys=True).encode()).hexdigest()
-__all__ = ["GatewayError", "ParseOptions", "Record", "RemoteCreatedTask", "RemoteTask", "TERMINAL", "UploadLink", "fingerprint"]
+__all__ = ["GatewayError", "ParseOptions", "Record", "RemoteCreatedTask", "RemoteTask", "TERMINAL", "fingerprint"]
diff --git a/mineru/kit/gradio/app.py b/mineru/kit/gradio/app.py
index 1a21d7a1..d7f9c721 100644
--- a/mineru/kit/gradio/app.py
+++ b/mineru/kit/gradio/app.py
@@ -6,50 +6,47 @@ import asyncio
import html
import json
import os
-import time
import uuid
+from collections.abc import Callable
from contextlib import suppress
from pathlib import Path
-from collections.abc import Callable
from typing import Any
from urllib.parse import quote
-from loguru import logger
-
-from ...errors import MineruError
from ...filetypes import (
FLASH_ONLY_PARSE_EXTENSIONS,
HTML_EXTENSIONS,
- MHTML_EXTENSIONS,
IMAGE_EXTENSIONS,
+ MHTML_EXTENSIONS,
OFFICE_EXTENSIONS,
PARSEABLE_EXTENSIONS,
PDF_EXTENSIONS,
)
-from ...types import TIERS, Tier
+from ...types import TIERS
from ...utils.logger import configure_global_log_level
from ...utils.stdio import configure_standard_streams
-from .artifacts import RunArtifacts, persist_parse_result, render_download, render_html_preview
+from .artifacts import RunArtifacts, render_download
from .client import (
GradioArtifactClient,
ManagedLocalApiServer,
V1ArtifactClient,
V1ServerCapabilities,
)
+from .conversion import ConversionInput, SessionConversions, await_task_completion
+from .conversion_service import ConversionService
+from .conversion_ui import build_conversion_receipt, conversion_updates, download_updates
from .epub_preview import register_epub_preview_resources
-from .conversion import ConversionRun, SessionConversions, await_task_completion, run_sync_output
from .i18n import MESSAGES, localized_text, preview_placeholder, translations
-from .page_range import effective_page_range as _effective_page_range
from .page_range import pdf_page_metadata, validate_max_pages
from .pdf_preview import pdf_preview_js, register_pdf_preview_resources
from .source_preview import prepare_source_preview
from .status import (
DEFAULT_STATUS as _DEFAULT_STATUS,
- STATUS_COMPLETED,
+)
+from .status import (
STATUS_PREPARING_REQUEST,
- STATUS_PROCESSING_OUTPUT,
- STATUS_QUEUED_LOCALLY,
- ParseStatusUpdate,
+)
+from .status import (
status_html as _status_html,
)
@@ -319,13 +316,6 @@ def _default_tier(capabilities: V1ServerCapabilities) -> str:
return capabilities.tiers[0]
-def _tier_for_position(position: int | float, tier_choices: list[Tier]) -> Tier:
- """把离散滑块位置映射为服务支持的 tier,并拒绝越界或非整数位置。"""
- if isinstance(position, bool) or position not in range(len(tier_choices)):
- raise ValueError("Invalid tier slider position")
- return tier_choices[int(position)]
-
-
def _file_suffix(path_value: str | Path | None) -> str:
"""读取上传文件的小写后缀,供预览、选页与 OCR 控件判断。"""
if not path_value:
@@ -354,11 +344,6 @@ def _pdf_preview_update(gr: Any, value: str | None) -> Any:
return gr.update(value=value, visible=False)
-def _download_updates(gr: Any, *, interactive: bool, run_id: str = "") -> tuple[Any, ...]:
- """同步当前结果标识,并恢复全部下载按钮的标签与交互状态。"""
- return (run_id, *(gr.update(value=label, interactive=interactive) for _format_name, label in _DOWNLOAD_FORMATS))
-
-
def build_gradio_app(
client: V1ArtifactClient | GradioArtifactClient,
capabilities: V1ServerCapabilities,
@@ -395,6 +380,13 @@ def build_gradio_app(
)
# 等待限制放在生成器内部,使其他会话也能立即显示本地排队状态。
conversions = SessionConversions()
+ conversion_service = ConversionService(
+ client,
+ tier_choices=tier_choices,
+ output_root=output_root,
+ max_pages=max_pages,
+ slot=conversions.slot,
+ )
with gr.Blocks() as demo:
gr.HTML(
@@ -599,7 +591,13 @@ def build_gradio_app(
def update_file_preview(file_path: str | None, request: object | None = None) -> tuple[Any, ...]:
"""切换源文件预览,并清除上一份文档的结果与下载状态。"""
- reset_result = (_status_html(_DEFAULT_STATUS), "", None, *_download_updates(gr, interactive=False), "")
+ reset_result = (
+ _status_html(_DEFAULT_STATUS),
+ "",
+ None,
+ *download_updates(_DOWNLOAD_FORMATS, interactive=False),
+ "",
+ )
if not file_path:
return (
_pdf_preview_update(gr, None),
@@ -801,139 +799,24 @@ def build_gradio_app(
**private_event_kwargs,
)
- async def execute_conversion(
- run: ConversionRun,
- file_path: str | None,
- tier_position: int | float,
- raw_page_range: str,
- force_ocr: bool,
- request: object | None,
- ) -> tuple[Any, ...]:
- """执行一次转换,只返回完整终态;进度写入独立的会话快照。"""
- reset_result = (
- _status_html(_DEFAULT_STATUS),
- "",
- *(gr.skip() for _ in range(4)),
- None,
- *_download_updates(gr, interactive=False),
- "",
- )
-
- def failure(message: str) -> tuple[Any, ...]:
- """把输入或输出错误收敛成可轮询的失败终态。"""
- run.publish(f"Failed: {message}", final_failure=True)
- return (run.state.render(), *reset_result[1:])
-
- if not file_path:
- return failure("No input file")
- source_path = Path(file_path).resolve()
- if not source_path.is_file():
- return failure("input file does not exist")
- suffix = _file_suffix(source_path)
- if suffix not in PARSEABLE_EXTENSIONS:
- return failure(f"unsupported file type '.{suffix}'")
- try:
- selected_tier = _tier_for_position(tier_position, tier_choices)
- except ValueError as exc:
- return failure(str(exc))
- if suffix in FLASH_ONLY_PARSE_EXTENSIONS:
- if "flash" not in tier_choices:
- return failure("tier_unavailable: 该格式仅支持 Flash,当前服务不可用")
- selected_tier = "flash"
- loop = asyncio.get_running_loop()
-
- def emit(message: str | ParseStatusUpdate) -> None:
- """在通知发生时记录时间,再由事件循环串行更新会话状态。"""
- at = time.monotonic()
- loop.call_soon_threadsafe(lambda: run.publish(message, at=at))
-
- async def run_conversion() -> tuple[Any, ...]:
- """实际同步工作退出前一直持有执行槽,避免取消后与新任务重叠。"""
- if conversions.slot.locked():
- run.publish(STATUS_QUEUED_LOCALLY)
- async with conversions.slot:
- run.publish(STATUS_PREPARING_REQUEST)
- page_text = await run_sync_output(_effective_page_range, source_path, raw_page_range, max_pages=max_pages)
- result = await client.parse_file(
- source_path,
- tier=selected_tier,
- page_range=page_text,
- ocr_mode="ocr" if suffix in PDF_EXTENSIONS and force_ocr else "auto",
- status_callback=emit,
- )
- # 先消费同一轮已发出的解析通知,保持阶段和解析耗时的顺序。
- await asyncio.sleep(0)
- run.publish(STATUS_PROCESSING_OUTPUT)
- output_started = time.monotonic()
- artifacts = await run_sync_output(
- persist_parse_result, result, source_path, output_root=output_root, page_range=page_text
- )
- rendered_html = await run_sync_output(
- render_html_preview, artifacts, public_base_url=_gradio_public_base_url(request)
- )
- structured_json = await run_sync_output(artifacts.structured_content_path.read_text, encoding="utf-8")
- preview_path = artifacts.layout_pdf_path or artifacts.origin_pdf_path
- generic_html = "" if preview_path else preview_placeholder("result_ready")
- show_image_preview = suffix in IMAGE_EXTENSIONS and preview_path is None
- result_preview_updates = (
- _pdf_preview_update(gr, str(preview_path) if preview_path else None),
- gr.update(value=str(artifacts.source_path) if show_image_preview else None, visible=show_image_preview),
- gr.update(value="", visible=False),
- gr.update(value=generic_html, visible=bool(generic_html)),
- )
- if _is_office(source_path) or suffix in {"ofd", "epub"} or suffix in HTML_EXTENSIONS | MHTML_EXTENSIONS:
- result_preview_updates = tuple(gr.skip() for _ in range(4))
- run.artifacts = artifacts.as_state()
- logger.debug(
- "WebUI output ready run_id={} artifacts={} elapsed={:.3f}s html_bytes={} json_bytes={} ready_at={:.3f}",
- run.run_id,
- artifacts.root.name,
- time.monotonic() - output_started,
- len(rendered_html.encode("utf-8")),
- len(structured_json.encode("utf-8")),
- time.time(),
- )
- return (
- rendered_html,
- *result_preview_updates,
- run.artifacts,
- *_download_updates(gr, interactive=True, run_id=artifacts.root.name),
- structured_json,
- )
-
- task = asyncio.create_task(run_conversion())
- run.task = task
- try:
- result_outputs = await asyncio.shield(task)
- if run.cancelled:
- return tuple(gr.skip() for _ in reset_result)
- run.publish(STATUS_COMPLETED)
- return (run.state.render(), *result_outputs)
- except asyncio.CancelledError:
- run.cancel()
- with suppress(asyncio.CancelledError, Exception):
- await await_task_completion(task)
- return tuple(gr.skip() for _ in reset_result)
- except Exception as exc:
- logger.exception("WebUI conversion failed run_id={}", run.run_id)
- message = f"{exc.code}: {exc}" if isinstance(exc, MineruError) else str(exc)
- return failure(message)
- finally:
- run.task = None
-
async def convert_handler(
file_path: str | None,
tier_position: int | float,
raw_page_range: str,
force_ocr: bool = False,
request: object | None = None,
- ) -> tuple[Any, ...]:
+ ) -> tuple[object, ...]:
"""保留公开转换 API 的四个输入和原生组件输出,普通响应一次返回结果。"""
session = getattr(request, "session_hash", None) or uuid.uuid4().hex
run = conversions.start(session, uuid.uuid4().hex)
assert run is not None
try:
- return await execute_conversion(run, file_path, tier_position, raw_page_range, force_ocr, request)
+ outcome = await conversion_service.execute(
+ run,
+ ConversionInput(file_path, tier_position, raw_page_range, force_ocr),
+ public_base_url=_gradio_public_base_url(request),
+ )
+ return conversion_updates(outcome, status_html=run.state.render(), download_formats=_DOWNLOAD_FORMATS)
finally:
conversions.cancel(session, run.run_id)
@@ -1011,8 +894,6 @@ def build_gradio_app(
request: object | None = None,
) -> str:
"""用完整回执传递浏览器结果,使迟到响应无法直接覆盖可见组件。"""
- from gradio.data_classes import FileData
-
try:
identity = json.loads(ticket)
run_id = uuid.UUID(identity["run_id"]).hex
@@ -1025,29 +906,22 @@ def build_gradio_app(
run = conversions.start(session, run_id, revision=revision)
if run is None:
return ""
- result = await execute_conversion(run, file_path, tier_position, raw_page_range, force_ocr, request)
+ public_base_url = _gradio_public_base_url(request)
+ outcome = await conversion_service.execute(
+ run,
+ ConversionInput(file_path, tier_position, raw_page_range, force_ocr),
+ public_base_url=public_base_url,
+ )
if conversions.current(session, run_id) is not run:
return ""
- values = list(result)
- # 文件来自输出根目录的不可变任务目录,使用现有 allowed_paths 文件路由。
- for index in (2, 3):
- update = values[index]
- if isinstance(update, dict) and update.get("value"):
- path = Path(update["value"]).resolve()
- path.relative_to(output_root.resolve())
- file_url = f"{_gradio_public_base_url(request)}/gradio_api/file={quote(path.as_posix(), safe='/')}"
- values[index] = {
- **update,
- "value": FileData(path=str(path), url=file_url, orig_name=path.name).model_dump(mode="json"),
- }
- return json.dumps(
- {
- "run_id": run_id,
- "sequence": run.state.sequence,
- "ready_at": time.time(),
- "outputs": values[:6] + values[7:],
- },
- ensure_ascii=False,
+ return build_conversion_receipt(
+ outcome,
+ status_html=run.state.render(),
+ run_id=run_id,
+ sequence=run.state.sequence,
+ output_root=output_root,
+ public_base_url=public_base_url,
+ download_formats=_DOWNLOAD_FORMATS,
)
for callback in (read_conversion_status, cancel_ui_conversion, unload_session, convert_ui):
@@ -1169,7 +1043,7 @@ def build_gradio_app(
gr.update(value="", visible=False),
gr.update(value=preview_placeholder("empty_preview"), visible=True),
None,
- *_download_updates(gr, interactive=False),
+ *download_updates(_DOWNLOAD_FORMATS, interactive=False),
"",
)
diff --git a/mineru/kit/gradio/conversion.py b/mineru/kit/gradio/conversion.py
index ba9d0e5b..371e1880 100644
--- a/mineru/kit/gradio/conversion.py
+++ b/mineru/kit/gradio/conversion.py
@@ -6,14 +6,70 @@ import asyncio
import json
from collections.abc import Callable
from dataclasses import dataclass, field
-from typing import Any, ParamSpec, TypeVar
+from pathlib import Path
+from typing import TYPE_CHECKING, Any, ParamSpec, TypeVar
from .status import STATUS_COMPLETED, STATUS_PREPARING_REQUEST, ParseStatusUpdate, StatusPanelState
+if TYPE_CHECKING:
+ from .artifacts import RunArtifacts
+
P = ParamSpec("P")
T = TypeVar("T")
+@dataclass(frozen=True)
+class ConversionInput:
+ """描述一次转换输入,不依赖 Gradio 请求或组件。"""
+
+ file_path: str | None
+ tier_position: int | float
+ page_range: str
+ force_ocr: bool = False
+
+
+@dataclass(frozen=True)
+class PreviewPlan:
+ """描述结果预览所需路径与占位状态,由界面层决定组件更新方式。"""
+
+ preserve_source: bool
+ pdf_path: Path | None = None
+ image_path: Path | None = None
+ show_ready_placeholder: bool = False
+
+
+@dataclass(frozen=True)
+class ConversionOutput:
+ """保存已物化的转换输出,不保留解析语义树。"""
+
+ artifacts: RunArtifacts
+ html: str
+ structured_json: str
+ preview: PreviewPlan
+
+
+@dataclass(frozen=True)
+class ConversionSuccess:
+ """表示解析及所有输出准备均已完成。"""
+
+ output: ConversionOutput
+
+
+@dataclass(frozen=True)
+class ConversionFailure:
+ """记录最终错误,由界面层保持既有失败响应格式。"""
+
+ message: str
+
+
+@dataclass(frozen=True)
+class ConversionCancelled:
+ """表示任务已失效且同步工作已退出,不再应用界面结果。"""
+
+
+ConversionOutcome = ConversionSuccess | ConversionFailure | ConversionCancelled
+
+
@dataclass
class ConversionRun:
"""只保留当前任务的状态和素材路径,不在会话缓存中持有解析结果。"""
@@ -23,7 +79,7 @@ class ConversionRun:
snapshot: str = ""
cancelled: bool = False
terminal: bool = False
- task: asyncio.Task[tuple[Any, ...]] | None = None
+ task: asyncio.Task[ConversionOutput] | None = None
artifacts: dict[str, Any] | None = None
def __post_init__(self) -> None:
@@ -48,7 +104,7 @@ class ConversionRun:
ensure_ascii=False,
)
- def cancel(self) -> asyncio.Task[tuple[Any, ...]] | None:
+ def cancel(self) -> asyncio.Task[ConversionOutput] | None:
"""立即失效界面回执,实际任务在同步工作退出后释放执行槽。"""
self.cancelled = True
self.artifacts = None
@@ -84,7 +140,7 @@ class SessionConversions:
run = self.runs.get(session)
return run if run is not None and run.run_id == run_id and not run.cancelled else None
- def cancel(self, session: str, run_id: str | None = None) -> asyncio.Task[tuple[Any, ...]] | None:
+ def cancel(self, session: str, run_id: str | None = None) -> asyncio.Task[ConversionOutput] | None:
"""按回执身份取消任务,迟到的文件切换请求不能取消新任务。"""
run = self.runs.get(session)
if run is None or (run_id is not None and run.run_id != run_id):
@@ -115,4 +171,16 @@ async def run_sync_output(function: Callable[P, T], *args: P.args, **kwargs: P.k
return await await_task_completion(asyncio.create_task(asyncio.to_thread(function, *args, **kwargs)))
-__all__ = ["ConversionRun", "SessionConversions", "await_task_completion", "run_sync_output"]
+__all__ = [
+ "ConversionCancelled",
+ "ConversionFailure",
+ "ConversionInput",
+ "ConversionOutcome",
+ "ConversionOutput",
+ "ConversionRun",
+ "ConversionSuccess",
+ "PreviewPlan",
+ "SessionConversions",
+ "await_task_completion",
+ "run_sync_output",
+]
diff --git a/mineru/kit/gradio/conversion_service.py b/mineru/kit/gradio/conversion_service.py
new file mode 100644
index 00000000..abf5e65f
--- /dev/null
+++ b/mineru/kit/gradio/conversion_service.py
@@ -0,0 +1,228 @@
+"""WebUI 转换业务:输入校验、解析排队、素材持久化和预览准备。"""
+
+from __future__ import annotations
+
+import asyncio
+import time
+from collections.abc import Callable, Sequence
+from contextlib import suppress
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Literal, Protocol
+
+from loguru import logger
+
+from ...errors import MineruError
+from ...filetypes import (
+ FLASH_ONLY_PARSE_EXTENSIONS,
+ HTML_EXTENSIONS,
+ IMAGE_EXTENSIONS,
+ MHTML_EXTENSIONS,
+ OFFICE_EXTENSIONS,
+ PARSEABLE_EXTENSIONS,
+ PDF_EXTENSIONS,
+)
+from ...parser.base import ParseResult
+from ...types import Tier
+from .artifacts import RunArtifacts, persist_parse_result, render_html_preview
+from .conversion import (
+ ConversionCancelled,
+ ConversionFailure,
+ ConversionInput,
+ ConversionOutcome,
+ ConversionOutput,
+ ConversionRun,
+ ConversionSuccess,
+ PreviewPlan,
+ await_task_completion,
+ run_sync_output,
+)
+from .page_range import effective_page_range, validate_max_pages
+from .status import (
+ STATUS_COMPLETED,
+ STATUS_PREPARING_REQUEST,
+ STATUS_PROCESSING_OUTPUT,
+ STATUS_QUEUED_LOCALLY,
+ ParseStatusUpdate,
+)
+
+
+class ParseClient(Protocol):
+ """转换流程所需的最小解析接口,兼容远端与进程内客户端。"""
+
+ async def parse_file(
+ self,
+ path: Path,
+ *,
+ tier: str,
+ page_range: str,
+ ocr_mode: Literal["auto", "txt", "ocr"] = "auto",
+ status_callback: Callable[[str | ParseStatusUpdate], None] | None = None,
+ ) -> ParseResult:
+ """解析输入文件,进度通过回调传递,取消遵守客户端已有执行边界。"""
+ ...
+
+
+@dataclass(frozen=True)
+class _ValidatedInput:
+ """保存已校验的源文件与实际档位,避免排队后重复判断输入。"""
+
+ source_path: Path
+ suffix: str
+ tier: Tier
+
+
+class ConversionService:
+ """执行转换并返回具名结果,不接收 Gradio 请求或生成组件更新。"""
+
+ def __init__(
+ self,
+ client: ParseClient,
+ *,
+ tier_choices: Sequence[Tier],
+ output_root: Path,
+ max_pages: int | None,
+ slot: asyncio.Semaphore,
+ ) -> None:
+ """显式接收解析能力、产物目录、页数限制和共享执行槽。"""
+ validate_max_pages(max_pages)
+ self.client = client
+ self.tier_choices = tuple(tier_choices)
+ self.output_root = output_root
+ self.max_pages = max_pages
+ self.slot = slot
+
+ async def execute(
+ self,
+ run: ConversionRun,
+ inputs: ConversionInput,
+ *,
+ public_base_url: str,
+ ) -> ConversionOutcome:
+ """返回成功、失败或取消结果,并在所有退出路径回收后台任务句柄。"""
+ try:
+ validated = self._validate_input(inputs)
+ except ValueError as exc:
+ return self._failure(run, str(exc))
+ loop = asyncio.get_running_loop()
+
+ def emit(message: str | ParseStatusUpdate) -> None:
+ """记录通知实际发生时间,再由事件循环串行更新会话快照。"""
+ at = time.monotonic()
+ loop.call_soon_threadsafe(lambda: run.publish(message, at=at))
+
+ task = asyncio.create_task(self._convert(run, inputs, validated, emit, public_base_url=public_base_url))
+ run.task = task
+ try:
+ output = await asyncio.shield(task)
+ if run.cancelled:
+ return ConversionCancelled()
+ run.publish(STATUS_COMPLETED)
+ return ConversionSuccess(output)
+ except asyncio.CancelledError:
+ run.cancel()
+ with suppress(asyncio.CancelledError, Exception):
+ await await_task_completion(task)
+ return ConversionCancelled()
+ except Exception as exc:
+ logger.exception("WebUI conversion failed run_id={}", run.run_id)
+ message = f"{exc.code}: {exc}" if isinstance(exc, MineruError) else str(exc)
+ return self._failure(run, message)
+ finally:
+ run.task = None
+
+ def _validate_input(self, inputs: ConversionInput) -> _ValidatedInput:
+ """在进入队列前校验文件和滑块位置,并应用原生格式的 Flash 限制。"""
+ if not inputs.file_path:
+ raise ValueError("No input file")
+ source_path = Path(inputs.file_path).resolve()
+ if not source_path.is_file():
+ raise ValueError("input file does not exist")
+ suffix = source_path.suffix.lower().lstrip(".")
+ if suffix not in PARSEABLE_EXTENSIONS:
+ raise ValueError(f"unsupported file type '.{suffix}'")
+ position = inputs.tier_position
+ if isinstance(position, bool) or position not in range(len(self.tier_choices)):
+ raise ValueError("Invalid tier slider position")
+ selected_tier = self.tier_choices[int(position)]
+ if suffix in FLASH_ONLY_PARSE_EXTENSIONS:
+ if "flash" not in self.tier_choices:
+ raise ValueError("tier_unavailable: 该格式仅支持 Flash,当前服务不可用")
+ selected_tier = "flash"
+ return _ValidatedInput(source_path, suffix, selected_tier)
+
+ async def _convert(
+ self,
+ run: ConversionRun,
+ inputs: ConversionInput,
+ validated: _ValidatedInput,
+ emit: Callable[[str | ParseStatusUpdate], None],
+ *,
+ public_base_url: str,
+ ) -> ConversionOutput:
+ """同步工作退出前始终持有执行槽,输出完成后只缓存素材路径。"""
+ if self.slot.locked():
+ run.publish(STATUS_QUEUED_LOCALLY)
+ async with self.slot:
+ run.publish(STATUS_PREPARING_REQUEST)
+ page_text = await run_sync_output(
+ effective_page_range, validated.source_path, inputs.page_range, max_pages=self.max_pages
+ )
+ result = await self.client.parse_file(
+ validated.source_path,
+ tier=validated.tier,
+ page_range=page_text,
+ ocr_mode="ocr" if validated.suffix in PDF_EXTENSIONS and inputs.force_ocr else "auto",
+ status_callback=emit,
+ )
+ # 先消费本轮已发出的解析通知,保持阶段和解析耗时的顺序。
+ await asyncio.sleep(0)
+ run.publish(STATUS_PROCESSING_OUTPUT)
+ output_started = time.monotonic()
+ artifacts = await run_sync_output(
+ persist_parse_result,
+ result,
+ validated.source_path,
+ output_root=self.output_root,
+ page_range=page_text,
+ )
+ rendered_html = await run_sync_output(render_html_preview, artifacts, public_base_url=public_base_url)
+ structured_json = await run_sync_output(artifacts.structured_content_path.read_text, encoding="utf-8")
+ run.artifacts = artifacts.as_state()
+ logger.debug(
+ "WebUI output ready run_id={} artifacts={} elapsed={:.3f}s html_bytes={} json_bytes={} ready_at={:.3f}",
+ run.run_id,
+ artifacts.root.name,
+ time.monotonic() - output_started,
+ len(rendered_html.encode("utf-8")),
+ len(structured_json.encode("utf-8")),
+ time.time(),
+ )
+ return ConversionOutput(
+ artifacts=artifacts,
+ html=rendered_html,
+ structured_json=structured_json,
+ preview=_prepare_preview(validated.suffix, artifacts),
+ )
+
+ @staticmethod
+ def _failure(run: ConversionRun, message: str) -> ConversionFailure:
+ """将最终错误补充到失败快照,保留已有成功和取消终态保护。"""
+ run.publish(f"Failed: {message}", final_failure=True)
+ return ConversionFailure(message)
+
+
+def _prepare_preview(suffix: str, artifacts: RunArtifacts) -> PreviewPlan:
+ """仅描述结果预览数据,保留已有 Office、OFD、EPUB 和网页源预览。"""
+ if suffix in OFFICE_EXTENSIONS | HTML_EXTENSIONS | MHTML_EXTENSIONS | {"ofd", "epub"}:
+ return PreviewPlan(preserve_source=True)
+ preview_path = artifacts.layout_pdf_path or artifacts.origin_pdf_path
+ return PreviewPlan(
+ preserve_source=False,
+ pdf_path=preview_path,
+ image_path=artifacts.source_path if suffix in IMAGE_EXTENSIONS and preview_path is None else None,
+ show_ready_placeholder=preview_path is None,
+ )
+
+
+__all__ = ["ConversionService", "ParseClient"]
diff --git a/mineru/kit/gradio/conversion_ui.py b/mineru/kit/gradio/conversion_ui.py
new file mode 100644
index 00000000..716b287a
--- /dev/null
+++ b/mineru/kit/gradio/conversion_ui.py
@@ -0,0 +1,102 @@
+"""转换结果的 Gradio 组件更新与浏览器完整回执适配。"""
+
+from __future__ import annotations
+
+import json
+import time
+from collections.abc import Sequence
+from pathlib import Path
+from urllib.parse import quote
+
+from .conversion import ConversionCancelled, ConversionFailure, ConversionOutcome
+from .i18n import preview_placeholder
+
+
+def download_updates(formats: Sequence[tuple[str, str]], *, interactive: bool, run_id: str = "") -> tuple[object, ...]:
+ """同步当前素材标识,并恢复全部下载按钮的标签与交互状态。"""
+ import gradio as gr
+
+ return (run_id, *(gr.update(value=label, interactive=interactive) for _format_name, label in formats))
+
+
+def conversion_updates(
+ outcome: ConversionOutcome,
+ *,
+ status_html: str,
+ download_formats: Sequence[tuple[str, str]],
+) -> tuple[object, ...]:
+ """将具名转换结果统一编码为既有公开 API 的组件输出顺序。"""
+ import gradio as gr
+
+ if isinstance(outcome, ConversionCancelled):
+ return tuple(gr.skip() for _ in range(9 + len(download_formats)))
+ if isinstance(outcome, ConversionFailure):
+ return (
+ status_html,
+ "",
+ *(gr.skip() for _ in range(4)),
+ None,
+ *download_updates(download_formats, interactive=False),
+ "",
+ )
+ output = outcome.output
+ preview = output.preview
+ if preview.preserve_source:
+ preview_updates = tuple(gr.skip() for _ in range(4))
+ else:
+ generic_html = preview_placeholder("result_ready") if preview.show_ready_placeholder else ""
+ preview_updates = (
+ gr.update(value=str(preview.pdf_path) if preview.pdf_path else None, visible=False),
+ gr.update(value=str(preview.image_path) if preview.image_path else None, visible=preview.image_path is not None),
+ gr.update(value="", visible=False),
+ gr.update(value=generic_html, visible=bool(generic_html)),
+ )
+ return (
+ status_html,
+ output.html,
+ *preview_updates,
+ output.artifacts.as_state(),
+ *download_updates(download_formats, interactive=True, run_id=output.artifacts.root.name),
+ output.structured_json,
+ )
+
+
+def build_conversion_receipt(
+ outcome: ConversionOutcome,
+ *,
+ status_html: str,
+ run_id: str,
+ sequence: int,
+ output_root: Path,
+ public_base_url: str,
+ download_formats: Sequence[tuple[str, str]],
+) -> str:
+ """校验结果文件归属并生成既有浏览器回执,不向浏览器发送内部素材状态。"""
+ from gradio.data_classes import FileData
+
+ if isinstance(outcome, ConversionCancelled):
+ return ""
+ values = list(conversion_updates(outcome, status_html=status_html, download_formats=download_formats))
+ # 两个文件组件使用既有输出根目录路由,输出索引仅保留在此序列化边界。
+ for index in (2, 3):
+ update = values[index]
+ if isinstance(update, dict) and update.get("value"):
+ path = Path(update["value"]).resolve()
+ path.relative_to(output_root.resolve())
+ file_url = f"{public_base_url}/gradio_api/file={quote(path.as_posix(), safe='/')}"
+ values[index] = {
+ **update,
+ "value": FileData(path=str(path), url=file_url, orig_name=path.name).model_dump(mode="json"),
+ }
+ return json.dumps(
+ {
+ "run_id": run_id,
+ "sequence": sequence,
+ "ready_at": time.time(),
+ "outputs": values[:6] + values[7:],
+ },
+ ensure_ascii=False,
+ )
+
+
+__all__ = ["build_conversion_receipt", "conversion_updates", "download_updates"]
diff --git a/mineru/kit/router/app.py b/mineru/kit/router/app.py
index a6ae0b80..5799b853 100644
--- a/mineru/kit/router/app.py
+++ b/mineru/kit/router/app.py
@@ -15,8 +15,10 @@ import httpx
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse, Response
-from ...parser.api_server import ApiServerError
-from ...parser.task_api import TaskStore, build_task_router
+from ...tasks.errors import ApiServerError
+from ...tasks.execution import TaskExecution
+from ...tasks.http import build_task_router
+from ...tasks.service import TaskStore
from ...types import TIERS
from ...version import __version__
from .proxy import (
@@ -120,7 +122,7 @@ def _usage_payload(request: Request, registry: ResourceRegistry, pool: WorkerPoo
async def _finalize_terminal_job(
route: ResourceRoute,
- payload: dict[str, Any],
+ execution: TaskExecution,
*,
request: Request | None,
pool: WorkerPool,
@@ -128,7 +130,7 @@ async def _finalize_terminal_job(
headers: dict[str, str] | None = None,
) -> None:
"""终态释放一次执行预占,结果映射和租约由 Tasks 专用路径管理。"""
- if payload.get("status") in {"completed", "partial", "failed", "canceled"}:
+ if execution.status in {"completed", "failed", "canceled"}:
release_job_load(route, pool)
if not route.metadata.get("task_result_held"):
route.metadata.pop("upstream_headers", None)
@@ -149,7 +151,7 @@ async def _reconcile_jobs_once(registry: ResourceRegistry, pool: WorkerPool) ->
registry,
headers=dict(route.metadata.get("upstream_headers") or {}),
)
- await _finalize_terminal_job(route, execution.job, request=None, pool=pool, registry=registry)
+ await _finalize_terminal_job(route, execution, request=None, pool=pool, registry=registry)
except (RouterProxyError, ApiServerError):
continue
@@ -384,9 +386,9 @@ def create_app(
from .tasks import RouterTaskBackend, install_task_error_handlers
- async def finalize_task_job(route: ResourceRoute, payload: dict[str, Any], request: Request | None) -> None:
+ async def finalize_task_job(route: ResourceRoute, execution: TaskExecution, request: Request | None) -> None:
"""新任务沿用既有产物元数据补齐及负载释放边界。"""
- await _finalize_terminal_job(route, payload, request=request, pool=pool, registry=registry)
+ await _finalize_terminal_job(route, execution, request=request, pool=pool, registry=registry)
def task_backend(request: Request) -> RouterTaskBackend:
"""显式组装 Router 适配器,便捷入口不通过 HTTP 请求自身。"""
diff --git a/mineru/kit/router/tasks.py b/mineru/kit/router/tasks.py
index b5a670b2..a2b62b5e 100644
--- a/mineru/kit/router/tasks.py
+++ b/mineru/kit/router/tasks.py
@@ -19,21 +19,19 @@ from fastapi.responses import JSONResponse
from pydantic import ValidationError
from ...filetypes import is_flash_only_parse_extension
-from ...parser.api_server import ApiServerError, ErrorDetail, _JobInputMetadata, _raise_api_error
-from ...parser.task_api import (
- TERMINAL_TASK_STATUSES,
- CreateTaskResponse,
+from ...tasks.contracts import (
CancelTaskResponse,
- TaskExecution,
+ CreateTaskResponse,
+ ErrorDetail,
+ TaskDetailResponse,
TaskFileInput,
TaskFileItem,
- TaskDetailResponse,
- TaskUpload,
UploadFileResponse,
UploadTaskOptions,
- file_chunks,
- resolve_task_tier,
)
+from ...tasks.errors import ApiServerError, raise_api_error
+from ...tasks.execution import TaskExecution, TaskInputMetadata, TaskUpload
+from ...tasks.service import TERMINAL_TASK_STATUSES, file_chunks, resolve_task_tier
from ...types import select_default_quality_tier
from ...utils.async_utils import drain_future
from .proxy import (
@@ -46,7 +44,7 @@ from .proxy import (
from .resources import ResourceRegistry, ResourceRoute, SourceFileStore, utc_now_iso
from .workers import WorkerPool, WorkerState
-FinalizeJob = Callable[[ResourceRoute, dict[str, Any], Request | None], Awaitable[None]]
+FinalizeExecution = Callable[[ResourceRoute, TaskExecution, Request | None], Awaitable[None]]
def _checked_json(response: httpx.Response) -> dict[str, Any]:
@@ -56,7 +54,7 @@ def _checked_json(response: httpx.Response) -> dict[str, Any]:
if not isinstance(payload, dict):
raise ValueError("Expected an object")
except ValueError:
- _raise_api_error(
+ raise_api_error(
response.status_code if response.status_code >= 400 else 502,
error_type="api_error",
code="invalid_upstream_response",
@@ -95,7 +93,7 @@ async def _worker_request(
pool, worker, method, path, request=request, content=content, json_body=json_body, headers=headers
)
except RouterProxyError as exc:
- _raise_api_error(exc.status_code, error_type="api_error", code=exc.code, message=exc.message, param=exc.param)
+ raise_api_error(exc.status_code, error_type="api_error", code=exc.code, message=exc.message, param=exc.param)
def _worker_path(worker: WorkerState, url: str) -> str:
@@ -109,38 +107,25 @@ def _worker_path(worker: WorkerState, url: str) -> str:
prefix = base.path.rstrip("/")
if origin(base) != origin(target) or target.username or target.password or target.query or target.fragment:
- _raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message="Invalid worker resource URL")
+ raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message="Invalid worker resource URL")
if prefix and not target.path.startswith(prefix + "/"):
- _raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message="Invalid worker resource path")
+ raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message="Invalid worker resource path")
return target.path[len(prefix) :]
-def _job_payload(payload: dict[str, Any], upstream_id: str) -> dict[str, Any]:
- """把 worker Tasks 转换为私有执行快照,不包含旧 Jobs 公开协议。"""
- file_keys = ("status", "parse", "error")
- files = [{key: item.get(key) for key in file_keys} for item in payload["files"]]
- return {
- "job_id": upstream_id,
- "status": payload["status"],
- "created_at": payload["created_at"],
- "started_at": payload.get("started_at"),
- "finished_at": payload.get("finished_at"),
- "progress": payload["progress"],
- "files": files,
- }
-
-
def _lost_execution(route: ResourceRoute, pool: WorkerPool) -> TaskExecution:
- """worker 丢失时收敛失败和负载,保留已经成功文件的诊断信息。"""
- payload = copy.deepcopy(route.metadata.get("payload") or {})
- payload.update(job_id=route.public_id, status="failed", finished_at=utc_now_iso())
+ """worker 丢失时终结有类型快照,并保留既有资源计数与诊断缓存。"""
+ previous: TaskExecution = route.metadata["execution"]
error = ErrorDetail(type="engine_error", code="upstream_task_lost", message="Upstream task no longer exists")
+ execution = replace(previous, status="failed", finished_at=utc_now_iso(), result_file_id=None, error=error)
+ payload = copy.deepcopy(route.metadata.get("payload") or {})
+ payload.update(status="failed", finished_at=execution.finished_at)
for file in payload.get("files") or []:
if file.get("status") != "completed":
file.update(status="failed", error=error.model_dump())
- route.metadata.update(payload=payload, upstream_lost=True)
+ route.metadata.update(payload=payload, execution=execution, upstream_lost=True)
release_job_load(route, pool)
- return TaskExecution(payload, error=error, input_metadata=dict(route.metadata.get("task_input_metadata") or {}))
+ return execution
async def read_task_execution(
@@ -151,7 +136,7 @@ async def read_task_execution(
request: Request | None = None,
headers: Mapping[str, str] | None = None,
) -> TaskExecution:
- """查询 worker 的轻量任务状态,并把单文件产物登记为 Router 的公共 File。"""
+ """只在 HTTP 边界拆出 worker 的唯一文件,内部传递有类型的执行快照。"""
if route.metadata.get("upstream_lost"):
return _lost_execution(route, pool)
worker = pool.get(route.worker_id)
@@ -170,31 +155,36 @@ async def read_task_execution(
if response.status_code == 404 or worker.generation != generation or route.metadata.get("upstream_lost"):
return _lost_execution(route, pool)
try:
- payload = TaskDetailResponse.model_validate(_checked_json(response)).model_dump()
+ payload = TaskDetailResponse.model_validate(_checked_json(response))
except ValidationError as exc:
- _raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message=str(exc))
- if payload["task_id"] != route.upstream_id:
- _raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message="Upstream task ID changed")
+ raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message=str(exc))
+ if payload.task_id != route.upstream_id:
+ raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message="Upstream task ID changed")
+ if len(payload.files) != 1:
+ raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message="Expected one task file")
+ if payload.status == "partial":
+ raise_api_error(
+ 502, error_type="api_error", code="invalid_upstream_response", message="Invalid single-file task status"
+ )
+ (remote_file,) = payload.files
+ if remote_file.file_id != route.metadata["upstream_file_id"] or remote_file.file_name != route.metadata["file_name"]:
+ raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message="Worker file identity changed")
+ upstream_file_id = remote_file.file_id
+ public_file_id: str = route.metadata["file_id"]
result_file_id: str | None = None
- if len(payload["files"]) != 1:
- _raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message="Expected one task file")
- upstream_file_id = payload["files"][0]["file_id"]
- public_file_id = route.metadata["file_ids"].get(upstream_file_id)
- if public_file_id is None or payload["files"][0]["file_name"] != route.metadata["file_names"].get(upstream_file_id):
- _raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message="Worker file identity changed")
- if payload["files"][0]["status"] == "completed":
- result_url = payload["files"][0].get("result_url")
+ if remote_file.status == "completed":
+ result_url = remote_file.result_url
if not isinstance(result_url, str):
- _raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message="Missing result URL")
- path = _worker_path(worker, result_url)
- parts = path.strip("/").split("/")
+ raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message="Missing result URL")
+ resource_path = _worker_path(worker, result_url)
+ parts = resource_path.strip("/").split("/")
if len(parts) != 4 or parts[:2] != ["v1", "files"] or parts[-1] != "content":
- _raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message="Invalid result File URL")
+ raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message="Invalid result File URL")
if unquote(parts[2]) != upstream_file_id:
- _raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message="Result file ID changed")
+ raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message="Result file ID changed")
existing = registry.find_upstream("file", route.owner_scope, worker.worker_id, upstream_file_id)
if existing is None and not registry.is_file_reserved(public_file_id):
- # 取消、删除或过期的结果不因迟到响应重新登记;任务保持已发布的终态。
+ # 取消、删除或过期的结果不因迟到响应重新登记;父 Task 保持已经发布的终态。
result_file_id = public_file_id
elif existing is None or not isinstance(existing.metadata.get("payload"), dict):
metadata = _checked_json(
@@ -209,38 +199,45 @@ async def read_task_execution(
)
require_worker_generation(worker, generation)
if metadata.get("id") != upstream_file_id or metadata.get("purpose") != "parse_output":
- _raise_api_error(
+ raise_api_error(
502, error_type="api_error", code="invalid_upstream_response", message="Invalid result File metadata"
)
result_file_id = rewrite_file_payload(metadata, worker, registry, route.owner_scope, public_id=public_file_id)["id"]
else:
if existing.public_id != public_file_id:
- _raise_api_error(
- 502, error_type="api_error", code="invalid_upstream_response", message="Public file ID changed"
- )
+ raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message="Public file ID changed")
result_file_id = existing.public_id
result_route = registry.find("file", result_file_id)
if result_route is not None:
result_route.metadata["task_result_held"] = bool(route.metadata.get("task_result_held"))
- rewritten = _job_payload(payload, route.public_id)
- if rewritten["status"] in TERMINAL_TASK_STATUSES:
+ previous: TaskExecution = route.metadata["execution"]
+ execution = TaskExecution(
+ execution_id=route.public_id,
+ file_id=public_file_id,
+ file_name=remote_file.file_name,
+ status=payload.status,
+ created_at=payload.created_at,
+ started_at=payload.started_at,
+ finished_at=payload.finished_at,
+ page_range=previous.page_range,
+ parse=remote_file.parse,
+ result_file_id=result_file_id,
+ error=payload.error or remote_file.error,
+ input_metadata=TaskInputMetadata(remote_file.bytes)
+ if remote_file.url is not None and remote_file.bytes is not None
+ else None,
+ )
+ if execution.status in TERMINAL_TASK_STATUSES:
release_job_load(route, pool)
- route.metadata["payload"] = copy.deepcopy(rewritten)
- route.metadata["task_result_file_id"] = result_file_id
- error = ErrorDetail.model_validate(payload["error"]) if payload.get("error") else None
- input_metadata = {
- index: _JobInputMetadata(item["bytes"])
- for index, item in enumerate(payload["files"])
- if item.get("url") is not None and item["bytes"] is not None
- }
- route.metadata["task_input_metadata"] = input_metadata
- return TaskExecution(rewritten, result_file_id, error, input_metadata)
+ # 既有 Registry 仍消费 HTTP 状态缓存做回收和统计;执行协议不再经过 Job 字典转换。
+ route.metadata.update(payload=payload.model_dump(), execution=execution, task_result_file_id=result_file_id)
+ return execution
class RouterTaskBackend:
"""客户端上传项就绪后才选择 worker;URL 原样交由 worker 下载。"""
- def __init__(self, request: Request, owner_scope: str, finalize: FinalizeJob) -> None:
+ def __init__(self, request: Request, owner_scope: str, finalize: FinalizeExecution) -> None:
"""保存轻量上下文与显式收尾回调,不跨请求保存上传主体。"""
self.request = Request(dict(request.scope))
self.owner_scope = owner_scope
@@ -302,7 +299,7 @@ class RouterTaskBackend:
require_flash=lightweight,
)
if not candidates:
- _raise_api_error(
+ raise_api_error(
503,
error_type="api_error",
code="quality_tier_unavailable",
@@ -349,10 +346,10 @@ class RouterTaskBackend:
self._prepared_sources.clear()
self._pins.close()
- async def _wait_for_uploads_ready(
- self, worker: WorkerState, upstream_id: str, generation: int, inputs: list[TaskFileItem], file_ids: dict[str, str]
- ) -> dict[str, Any]:
- """只等待上传确认,不要求下游暴露源 File 或完成内部引用交接。"""
+ async def _wait_for_upload_ready(
+ self, worker: WorkerState, upstream_id: str, generation: int, item: TaskFileItem, upstream_file_id: str
+ ) -> TaskDetailResponse:
+ """等待唯一上传项被接受,URL 输入仍由下游在独立准备阶段下载。"""
while True:
response = await _worker_request(
self.pool,
@@ -363,46 +360,40 @@ class RouterTaskBackend:
headers={"x-mineru-task-lease-seconds": "120"},
)
require_worker_generation(worker, generation)
- ready = TaskDetailResponse.model_validate(_checked_json(response)).model_dump()
- if ready["task_id"] != upstream_id or len(ready["files"]) != len(inputs):
- _raise_api_error(
+ ready = TaskDetailResponse.model_validate(_checked_json(response))
+ if ready.task_id != upstream_id or len(ready.files) != 1:
+ raise_api_error(
502, error_type="api_error", code="invalid_upstream_response", message="Worker inputs are not ready"
)
- if any(
- file_ids.get(item["file_id"]) != source.file_id or item["file_name"] != source.file_name
- for item, source in zip(ready["files"], inputs)
- ):
- _raise_api_error(
+ (remote_file,) = ready.files
+ if remote_file.file_id != upstream_file_id or remote_file.file_name != item.file_name:
+ raise_api_error(
502, error_type="api_error", code="invalid_upstream_response", message="Worker input order changed"
)
- pending = False
- for item, source in zip(ready["files"], inputs):
- if source.url is not None:
- if item["upload"] is not None:
- _raise_api_error(
- 502, error_type="api_error", code="invalid_upstream_response", message="URL input requests upload"
- )
- continue
- pending = pending or item["status"] == "waiting-file"
- if not pending:
+ if item.url is not None:
+ if remote_file.upload is not None:
+ raise_api_error(
+ 502, error_type="api_error", code="invalid_upstream_response", message="URL input requests upload"
+ )
+ return ready
+ if remote_file.status != "waiting-file":
return ready
- # 只等待合法的中间状态;取消、worker replacement 和终态仍走原有清理边界。
await asyncio.sleep(1)
async def submit(
- self, task_id: str, options: UploadTaskOptions, inputs: list[TaskFileItem], uploads: dict[int, TaskUpload]
+ self, execution_id: str, options: UploadTaskOptions, item: TaskFileItem, upload: TaskUpload | None
) -> TaskExecution:
- """预占后提交 worker 任务,只向 worker 上传它实际缺失的文件。"""
- lightweight = all(is_flash_only_parse_extension(item.file_name) for item in inputs)
+ """为单文件预占 worker,并把公开文件身份与下游身份一一关联。"""
+ lightweight = is_flash_only_parse_extension(item.file_name)
tier = "flash" if lightweight else options.tier
reservation = self.pool.reserve_job(
tier=tier,
- required_sources={"url" if item.url is not None else "upload" for item in inputs},
+ required_sources={"url" if item.url is not None else "upload"},
required_task_outputs=set(options.output_formats),
- require_flash=any(is_flash_only_parse_extension(item.file_name) for item in inputs),
+ require_flash=lightweight,
)
if reservation is None:
- _raise_api_error(
+ raise_api_error(
503, error_type="api_error", code="quality_tier_unavailable", message="No capable worker is available"
)
worker = reservation.worker
@@ -410,12 +401,11 @@ class RouterTaskBackend:
route: ResourceRoute | None = None
transferred = False
try:
- worker_inputs = [
- item.model_dump(exclude_none=True, exclude={"file_id"})
- if item.url is not None
- else {**item.model_dump(exclude_none=True, exclude={"file_id"}), "sha256sum": uploads[index].sha256sum}
- for index, item in enumerate(inputs)
- ]
+ worker_input = item.model_dump(exclude_none=True, exclude={"file_id"})
+ if item.url is None:
+ if upload is None:
+ raise ValueError("Prepared source file is missing")
+ worker_input["sha256sum"] = upload.sha256sum
response = await _worker_request(
self.pool,
worker,
@@ -423,7 +413,7 @@ class RouterTaskBackend:
"/v1/tasks",
request=self.request,
json_body={
- "files": worker_inputs,
+ "files": [worker_input],
"tier": options.tier,
"ocr_mode": options.ocr_mode,
"output_formats": options.output_formats,
@@ -431,74 +421,84 @@ class RouterTaskBackend:
headers={"x-mineru-task-lease-seconds": "120"},
)
require_worker_generation(worker, reservation.generation)
- payload = CreateTaskResponse.model_validate(_checked_json(response)).model_dump()
- upstream_id = payload["task_id"]
+ created = CreateTaskResponse.model_validate(_checked_json(response))
+ upstream_id = created.task_id
route = self.registry.register(
"job",
owner_scope=self.owner_scope,
worker_id=worker.worker_id,
upstream_id=upstream_id,
- public_id=task_id,
+ public_id=execution_id,
)
authorization = self.request.headers.get("authorization")
+ initial = TaskExecution(
+ execution_id=execution_id,
+ file_id=item.file_id,
+ file_name=item.file_name,
+ status="pending",
+ created_at=route.created_at,
+ page_range=item.page_range or "",
+ )
route.metadata.update(
- file_ids={item["file_id"]: source.file_id for item, source in zip(payload["files"], inputs)},
- file_names={item["file_id"]: source.file_name for item, source in zip(payload["files"], inputs)},
+ file_id=item.file_id,
+ file_name=item.file_name,
+ upstream_file_id=None,
+ execution=initial,
task_result_held=True,
active_counted=True,
reservation=reservation,
upstream_headers={"authorization": authorization} if authorization else {},
- payload={"job_id": task_id, "status": "pending", "files": []},
+ payload={"task_id": upstream_id, "status": "pending", "files": []},
)
- if (
- len(payload["files"]) != len(inputs)
- or len(route.metadata["file_ids"]) != len(inputs)
- or any(item["file_name"] != source.file_name for item, source in zip(payload["files"], inputs))
- ):
- _raise_api_error(
+ if len(created.files) != 1:
+ raise_api_error(
502, error_type="api_error", code="invalid_upstream_response", message="Worker input order changed"
)
- for index, item in enumerate(payload["files"]):
- link = item["upload"]
- if link is None:
- continue
- if index not in uploads:
- _raise_api_error(
+ (remote_file,) = created.files
+ if remote_file.file_name != item.file_name:
+ raise_api_error(
+ 502, error_type="api_error", code="invalid_upstream_response", message="Worker input order changed"
+ )
+ route.metadata["upstream_file_id"] = remote_file.file_id
+ link = remote_file.upload
+ if link is not None:
+ if upload is None:
+ raise_api_error(
502, error_type="api_error", code="invalid_upstream_response", message="Unexpected worker upload"
)
- path = _worker_path(worker, link["url"])
- expected_path = f"/v1/tasks/{quote(upstream_id, safe='')}/uploads/{quote(item['file_id'], safe='')}"
- if path != expected_path:
- _raise_api_error(
+ resource_path = _worker_path(worker, link.url)
+ expected_path = f"/v1/tasks/{quote(upstream_id, safe='')}/uploads/{quote(remote_file.file_id, safe='')}"
+ if resource_path != expected_path:
+ raise_api_error(
502, error_type="api_error", code="invalid_upstream_response", message="Invalid worker upload path"
)
response = await _worker_request(
self.pool,
worker,
"PUT",
- path,
+ resource_path,
request=self.request,
- content=file_chunks(uploads[index].path),
+ content=file_chunks(upload.path),
headers={
- **link["headers"],
- "content-length": str(uploads[index].bytes),
+ **link.headers,
+ "content-length": str(upload.bytes),
"x-mineru-task-lease-seconds": "120",
},
)
try:
acknowledged = UploadFileResponse.model_validate(_checked_json(response))
except ValidationError as exc:
- _raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message=str(exc))
- if acknowledged.task_id != upstream_id or acknowledged.file_id != item["file_id"]:
- _raise_api_error(
+ raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message=str(exc))
+ if acknowledged.task_id != upstream_id or acknowledged.file_id != remote_file.file_id:
+ raise_api_error(
502,
error_type="api_error",
code="invalid_upstream_response",
message="Upload acknowledgement ID changed",
)
require_worker_generation(worker, reservation.generation)
- await self._wait_for_uploads_ready(worker, upstream_id, reservation.generation, inputs, route.metadata["file_ids"])
- result = await self.get(task_id)
+ await self._wait_for_upload_ready(worker, upstream_id, reservation.generation, item, remote_file.file_id)
+ result = await self.get(execution_id)
self._task_sources.extend(self._prepared_sources)
self._prepared_sources.clear()
transferred = True
@@ -509,8 +509,11 @@ class RouterTaskBackend:
await drain_future(asyncio.create_task(self._cancel_remote(worker, upstream_id)))
if route is not None:
release_job_load(route, self.pool)
+ previous: TaskExecution = route.metadata["execution"]
route.metadata.update(
- payload={"job_id": task_id, "status": "failed", "files": []}, finalized_at=utc_now_iso()
+ payload={"task_id": upstream_id, "status": "failed", "files": []},
+ execution=replace(previous, status="failed", finished_at=utc_now_iso()),
+ finalized_at=utc_now_iso(),
)
route.metadata["task_result_retain_until"] = time.time()
else:
@@ -530,20 +533,18 @@ class RouterTaskBackend:
"""检查本调用方的公共执行映射,不能用其他调用方的任务探测 worker。"""
route = self.registry.find("job", task_id)
if route is None or route.owner_scope != self.owner_scope:
- _raise_api_error(
- 404, error_type="invalid_request_error", code="task_not_found", message=f"Task {task_id} not found"
- )
+ raise_api_error(404, error_type="invalid_request_error", code="task_not_found", message=f"Task {task_id} not found")
return route
async def get(self, task_id: str) -> TaskExecution:
- """只查询 worker 元数据并登记结果引用,产物字节走已有 Files 流式代理。"""
+ """查询有类型单文件快照,产物字节继续经过现有 Files 流式代理。"""
route = self._route(task_id)
execution = await read_task_execution(route, self.pool, self.registry, request=self.request)
- await self.finalize(route, execution.job, self.request)
+ await self.finalize(route, execution, self.request)
return execution
async def cancel(self, task_id: str) -> None:
- """通过 worker tasks 取消准备或执行,成功后沿用一次性负载释放。"""
+ """核对唯一文件身份后取消下游任务,并沿原有边界释放执行预占。"""
route = self.registry.find("job", task_id)
if route is None:
return
@@ -558,17 +559,21 @@ class RouterTaskBackend:
try:
canceled = CancelTaskResponse.model_validate(_checked_json(response))
except ValidationError as exc:
- _raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message=str(exc))
+ raise_api_error(502, error_type="api_error", code="invalid_upstream_response", message=str(exc))
+ if canceled.task_id != route.upstream_id or len(canceled.files) != 1:
+ raise_api_error(
+ 502, error_type="api_error", code="invalid_upstream_response", message="Cancellation file identity changed"
+ )
+ (remote_file,) = canceled.files
if (
- canceled.task_id != route.upstream_id
- or [item.file_id for item in canceled.files] != list(route.metadata["file_ids"])
- or any(item.file_name != route.metadata["file_names"].get(item.file_id) for item in canceled.files)
+ remote_file.file_id != route.metadata["upstream_file_id"]
+ or remote_file.file_name != route.metadata["file_name"]
):
- _raise_api_error(
+ raise_api_error(
502, error_type="api_error", code="invalid_upstream_response", message="Cancellation file identity changed"
)
execution = await self.get(task_id)
- if execution.job["status"] in TERMINAL_TASK_STATUSES:
+ if execution.status in TERMINAL_TASK_STATUSES:
release_job_load(route, self.pool)
diff --git a/mineru/model/download.py b/mineru/model/download.py
index e942c6ec..18840803 100644
--- a/mineru/model/download.py
+++ b/mineru/model/download.py
@@ -15,7 +15,7 @@ from huggingface_hub import snapshot_download as hf_snapshot_download
from huggingface_hub.utils import filter_repo_objects
from loguru import logger
-from ..config import config, get_config_source, update_config_file
+from ..config import get_config, get_config_source, update_config_file
if TYPE_CHECKING:
from ..config import ModelSource
@@ -98,12 +98,12 @@ class ModelRepo:
def local_dir(self) -> Path:
"""返回仓库在 MinerU 模型目录中的本地根路径。"""
- return Path(config.model.base_dir).expanduser() / self.local_name
+ return Path(get_config().model.base_dir).expanduser() / self.local_name
def lock_path(self) -> Path:
"""返回仓库下载互斥锁的稳定路径。"""
safe_name = re.sub(r"[^A-Za-z0-9_.-]+", "_", self.local_name).strip("._") or "model"
- return Path(config.model.base_dir).expanduser() / ".locks" / f"{safe_name}.lock"
+ return Path(get_config().model.base_dir).expanduser() / ".locks" / f"{safe_name}.lock"
def ensure(self, *, source: "ModelSource | None" = None) -> Path:
"""确保完整仓库可用并返回本地根路径。"""
@@ -184,11 +184,11 @@ def resolve_model_source(
"""Resolve the effective model source.
Explicit ``model_source`` is CLI input and is validated strictly. When no
- explicit source is supplied, ``config.model.source`` may come from config
+ explicit source is supplied, ``get_config().model.source`` may come from config
file, environment, or defaults; invalid values are treated as auto.
"""
explicit_source = model_source is not None
- raw_source = model_source if explicit_source else config.model.source
+ raw_source = model_source if explicit_source else get_config().model.source
normalized_source = _normalize_source(raw_source, strict=explicit_source)
if explicit_source and normalized_source == "local":
diff --git a/mineru/model/registry.py b/mineru/model/registry.py
index e368fdcc..384204be 100644
--- a/mineru/model/registry.py
+++ b/mineru/model/registry.py
@@ -138,7 +138,7 @@ def model_repos_for_tier(
vlm_config: VlmConfig | None = None,
) -> tuple[ModelRepo, ...]:
"""组合档位需要的资源;远程 VLM 不要求本地权重,显式引擎可覆盖远程配置。"""
- from ..config import config
+ from ..config import get_config
resolved_tier = validate_model_tier(tier)
if vlm_engine is not None:
@@ -147,7 +147,7 @@ def model_repos_for_tier(
# basic 不加载 VLM,但仍拒绝模型管理命令中的非法显式引擎名称。
vlm_engine = resolve_vlm_engine(vlm_engine)
repos = (small_model_repo(small_backend),)
- settings = vlm_config if vlm_config is not None else config.model.vlm
+ settings = vlm_config if vlm_config is not None else get_config().model.vlm
if resolved_tier == "standard" and (vlm_engine is not None or not settings.server_url):
repos += (vlm_model_repo(vlm_engine if vlm_engine is not None else settings.engine),)
return repos
diff --git a/mineru/model/runtime/closing.py b/mineru/model/runtime/closing.py
new file mode 100644
index 00000000..bce23411
--- /dev/null
+++ b/mineru/model/runtime/closing.py
@@ -0,0 +1,28 @@
+# Copyright (c) Opendatalab. All rights reserved.
+"""模型关闭回调的上下文,防止清理过程中等待正在关闭的资源。"""
+
+from __future__ import annotations
+
+from collections.abc import Iterator
+from contextlib import contextmanager
+from contextvars import ContextVar
+
+_runtime_closing: ContextVar[bool] = ContextVar("mineru_runtime_closing", default=False)
+
+
+@contextmanager
+def runtime_close_scope() -> Iterator[None]:
+ """标记当前关闭调用链,调度到异步运行时后仍保留回调身份。"""
+ token = _runtime_closing.set(True)
+ try:
+ yield
+ finally:
+ _runtime_closing.reset(token)
+
+
+def in_runtime_close_scope() -> bool:
+ """判断当前是否处于资源关闭回调,供缓存入口拒绝阻塞式重入。"""
+ return _runtime_closing.get()
+
+
+__all__ = ["in_runtime_close_scope", "runtime_close_scope"]
diff --git a/mineru/model/runtime/device.py b/mineru/model/runtime/device.py
index 7e980a39..7e8326a0 100644
--- a/mineru/model/runtime/device.py
+++ b/mineru/model/runtime/device.py
@@ -63,9 +63,9 @@ def module_available(name: str) -> bool:
def resolve_small_model_backend(backend: str | None = None) -> Literal["onnx", "torch"]:
"""独立解析小模型后端;显式选择可用于离线下载,依赖由运行入口预检。"""
- from ...config import config
+ from ...config import get_config
- selected = config.model.small_backend if backend is None else backend
+ selected = get_config().model.small_backend if backend is None else backend
if selected == "onnx":
return "onnx"
if selected == "torch":
diff --git a/mineru/model/runtime/hybrid.py b/mineru/model/runtime/hybrid.py
index 7b03b3c3..69eb7f49 100644
--- a/mineru/model/runtime/hybrid.py
+++ b/mineru/model/runtime/hybrid.py
@@ -17,6 +17,15 @@ from ..ocr.language import normalize_ocr_model_lang
from ..ocr.resources import PPOCRV6_DICT_PATH
from ..registry import MINERU_4_MODELS_ONNX, small_model_repo
from .contracts import AtomicModelName
+from .local_cache import (
+ LOCAL_MODEL_CACHE_LOCK,
+ LocalModelResource,
+ current_local_resource,
+ get_local_model,
+ local_resource_scope,
+ release_local_owner,
+)
+from .ownership import RuntimeOwner
if TYPE_CHECKING:
from ..table.cls.mineru_table_ori_cls import MineruTableOrientationClsModel
@@ -29,7 +38,7 @@ if TYPE_CHECKING:
from .device import get_device, resolve_small_model_backend
-LOCAL_MODEL_INIT_LOCK = threading.RLock()
+LOCAL_MODEL_INIT_LOCK = LOCAL_MODEL_CACHE_LOCK
# 这些锁保护 Hybrid medium/high/xhigh 共享的 atom model/native 模型推理调用,避免多线程同时进入同一个模型对象。
LOCAL_MODEL_LAYOUT_INFERENCE_LOCK = threading.RLock()
LOCAL_MODEL_MFR_INFERENCE_LOCK = threading.RLock()
@@ -198,7 +207,7 @@ class AtomModelSingleton:
"""按模型配置缓存并复用本地原子模型实例。"""
_instance: AtomModelSingleton | None = None
- _models: dict[object, object] = {}
+ _models: dict[object, LocalModelResource] = {}
_lock: threading.RLock = LOCAL_MODEL_INIT_LOCK
def __new__(cls, *args: Any, **kwargs: Any) -> AtomModelSingleton:
@@ -243,10 +252,11 @@ class AtomModelSingleton:
from .onnx_device import resolve_onnx_device
key = (*key, resolve_onnx_device())
- with self._lock:
- if key not in self._models:
- self._models[key] = atom_model_init(model_name=atom_model_name, **kwargs)
- return self._models[key]
+ return get_local_model(self._models, key, lambda: atom_model_init(model_name=atom_model_name, **kwargs))
+
+ def release_owner(self, owner: RuntimeOwner) -> None:
+ """释放应用直接持有的原子模型,仍被其它上下文引用的依赖继续保留。"""
+ release_local_owner(self._models, owner)
@stage_timer("model.initialize")
@@ -349,7 +359,7 @@ class HybridLocalModelContextSingleton:
"""全局缓存并复用 Hybrid 本地模型上下文。"""
_instance: HybridLocalModelContextSingleton | None = None
- _models: dict[tuple[object, ...], HybridLocalModelContext] = {}
+ _models: dict[object, LocalModelResource] = {}
_lock: threading.RLock = LOCAL_MODEL_INIT_LOCK
def __new__(cls, *args: Any, **kwargs: Any) -> HybridLocalModelContextSingleton:
@@ -365,13 +375,14 @@ class HybridLocalModelContextSingleton:
"""按小模型后端与实际设备缓存上下文,避免切换配置后混用模型。"""
small_backend = resolve_small_model_backend()
device = "cpu" if small_backend == "onnx" else get_device()
- from ...config import config
+ from ...config import get_config
- key = (small_backend, device, config.model.onnx_device if small_backend == "onnx" else None)
- with self._lock:
- if key not in self._models:
- self._models[key] = HybridLocalModelContext(device=device, small_backend=small_backend)
- return self._models[key]
+ key = (small_backend, device, get_config().model.onnx_device if small_backend == "onnx" else None)
+ return get_local_model(self._models, key, lambda: HybridLocalModelContext(device=device, small_backend=small_backend))
+
+ def release_owner(self, owner: RuntimeOwner) -> None:
+ """释放应用对上下文的租约,并按依赖关系回收无人持有的原子模型。"""
+ release_local_owner(self._models, owner)
def ocr_det_batch_setting() -> bool:
@@ -406,6 +417,7 @@ class HybridLocalModelContext:
small_backend: str | None = None,
) -> None:
"""初始化 Hybrid 基础运行时,其他模型在首次访问对应属性时加载。"""
+ self._resource = current_local_resource()
self.small_backend = resolve_small_model_backend(small_backend)
self.device = "cpu" if self.small_backend == "onnx" else (device or get_device())
self.enable_ocr_det_batch = True if self.small_backend == "onnx" else ocr_det_batch_setting()
@@ -430,6 +442,11 @@ class HybridLocalModelContext:
# 初始化layout模型,用于提供行内公式检测框和Hybrid标题拆分
self.layout_model = self.get_layout_model()
+ def _get_atom_model(self, atom_model_name: str, **kwargs: Any) -> Any:
+ """把惰性模型登记为上下文依赖,复用 cached_property 时无需重复登记应用。"""
+ with local_resource_scope(self._resource):
+ return self.atom_model_manager.get_atom_model(atom_model_name=atom_model_name, **kwargs)
+
@cached_property
def mfr_model(self) -> FormulaRecognizer:
"""首次访问时加载公式识别模型,并在当前 Context 内复用。"""
@@ -467,7 +484,7 @@ class HybridLocalModelContext:
enable_merge_det_boxes: bool = True,
) -> "PytorchPaddleOCR":
"""获取 OCR 原子模型,默认使用当前 Hybrid 本地上下文语言并复用 singleton 缓存。"""
- return self.atom_model_manager.get_atom_model(
+ return self._get_atom_model(
small_backend=self.small_backend,
device=self.device,
atom_model_name=AtomicModelName.OCR,
@@ -479,7 +496,7 @@ class HybridLocalModelContext:
def get_layout_model(self) -> "PPDocLayoutV2LayoutModel":
"""获取 Layout 原子模型,供 Hybrid 本地 layout、标题拆分和公式框检测复用。"""
- return self.atom_model_manager.get_atom_model(
+ return self._get_atom_model(
small_backend=self.small_backend,
device=self.device,
atom_model_name=AtomicModelName.Layout,
@@ -487,7 +504,7 @@ class HybridLocalModelContext:
def get_mfr_model(self) -> "FormulaRecognizer":
"""获取公式识别原子模型,统一复用当前公式模型配置和设备。"""
- return self.atom_model_manager.get_atom_model(
+ return self._get_atom_model(
small_backend=self.small_backend,
device=self.device,
atom_model_name=AtomicModelName.MFR,
@@ -495,7 +512,7 @@ class HybridLocalModelContext:
def get_wireless_table_model(self) -> PaddleTableModel:
"""获取无线表格识别原子模型。"""
- return self.atom_model_manager.get_atom_model(
+ return self._get_atom_model(
small_backend=self.small_backend,
device=self.device,
atom_model_name=AtomicModelName.WirelessTable,
@@ -504,7 +521,7 @@ class HybridLocalModelContext:
def get_wired_table_model(self) -> UnetTableModel:
"""获取有线表格识别原子模型。"""
- return self.atom_model_manager.get_atom_model(
+ return self._get_atom_model(
small_backend=self.small_backend,
device=self.device,
atom_model_name=AtomicModelName.WiredTable,
@@ -513,7 +530,7 @@ class HybridLocalModelContext:
def get_table_cls_model(self) -> PaddleTableClsModel:
"""获取表格分类原子模型。"""
- return self.atom_model_manager.get_atom_model(
+ return self._get_atom_model(
small_backend=self.small_backend,
device=self.device,
atom_model_name=AtomicModelName.TableCls,
@@ -521,7 +538,7 @@ class HybridLocalModelContext:
def get_table_orientation_cls_model(self) -> MineruTableOrientationClsModel:
"""获取表格方向分类原子模型。"""
- return self.atom_model_manager.get_atom_model(
+ return self._get_atom_model(
small_backend=self.small_backend,
device=self.device,
atom_model_name=AtomicModelName.TableOrientationCls,
@@ -529,7 +546,7 @@ class HybridLocalModelContext:
def get_seal_ocr_model(self) -> PytorchPaddleOCR:
"""获取印章识别 OCR 原子模型"""
- return self.atom_model_manager.get_atom_model(
+ return self._get_atom_model(
small_backend=self.small_backend,
device=self.device,
atom_model_name=AtomicModelName.OCR,
diff --git a/mineru/model/runtime/local_cache.py b/mineru/model/runtime/local_cache.py
new file mode 100644
index 00000000..4db10440
--- /dev/null
+++ b/mineru/model/runtime/local_cache.py
@@ -0,0 +1,120 @@
+# Copyright (c) Opendatalab. All rights reserved.
+"""本地模型缓存的显式依赖租约,保护共享上下文及嵌套原子模型。"""
+
+from __future__ import annotations
+
+import gc
+import threading
+from collections.abc import Callable, Iterator
+from contextlib import contextmanager
+from contextvars import ContextVar
+from dataclasses import dataclass, field
+from typing import Any
+
+from .ownership import RuntimeOwner, current_runtime_owner
+
+LOCAL_MODEL_CACHE_LOCK = threading.RLock()
+
+
+@dataclass(eq=False)
+class LocalModelResource:
+ """记录缓存对象及应用、父模型和子模型之间的真实持有关系。"""
+
+ cache: dict[object, LocalModelResource]
+ key: object
+ value: Any = None
+ owners: set[RuntimeOwner] = field(default_factory=set)
+ parents: set[LocalModelResource] = field(default_factory=set)
+ dependencies: set[LocalModelResource] = field(default_factory=set)
+ pinned: bool = False
+ closed: bool = False
+
+
+_current_resource: ContextVar[LocalModelResource | None] = ContextVar("mineru_local_model_resource", default=None)
+
+
+def current_local_resource() -> LocalModelResource | None:
+ """让新建 Hybrid 上下文保留其缓存节点,供后续惰性加载登记依赖。"""
+ return _current_resource.get()
+
+
+@contextmanager
+def local_resource_scope(resource: LocalModelResource | None) -> Iterator[None]:
+ """将后续原子模型借用登记为指定上下文或模型的依赖。"""
+ token = _current_resource.set(resource)
+ try:
+ yield
+ finally:
+ _current_resource.reset(token)
+
+
+def _collect_unused(resource: LocalModelResource, discarded: list[object]) -> None:
+ """在缓存锁内移除无人持有节点,延迟到锁外才释放对象强引用。"""
+ if resource.closed or resource.pinned or resource.owners or resource.parents:
+ return
+ resource.closed = True
+ if resource.cache.get(resource.key) is resource:
+ del resource.cache[resource.key]
+ if resource.value is not None:
+ discarded.append(resource.value)
+ resource.value = None
+ dependencies = tuple(resource.dependencies)
+ resource.dependencies.clear()
+ for dependency in dependencies:
+ dependency.parents.discard(resource)
+ _collect_unused(dependency, discarded)
+
+
+def get_local_model(cache: dict[object, LocalModelResource], key: object, factory: Callable[[], Any]) -> Any:
+ """沿现有缓存键构造或复用模型,初始化失败时回滚其未被共享的依赖。"""
+ discarded: list[object] = []
+ try:
+ with LOCAL_MODEL_CACHE_LOCK:
+ owner = current_runtime_owner()
+ parent = _current_resource.get()
+ if parent is not None and parent.closed:
+ raise RuntimeError("Local model context is closed")
+ resource = cache.get(key)
+ if resource is None:
+ resource = LocalModelResource(cache, key)
+ try:
+ with local_resource_scope(resource):
+ resource.value = factory()
+ except BaseException:
+ _collect_unused(resource, discarded)
+ raise
+ cache[key] = resource
+ if parent is not None:
+ parent.dependencies.add(resource)
+ resource.parents.add(parent)
+ elif owner is None:
+ resource.pinned = True
+ else:
+ resource.owners.add(owner)
+ return resource.value
+ finally:
+ # ORT/Torch 小模型没有统一 close API;释放宿主持有引用即可让各自对象回收资源。
+ discarded.clear()
+
+
+def release_local_owner(cache: dict[object, LocalModelResource], owner: RuntimeOwner) -> None:
+ """撤销应用租约并递归释放无其它持有者的模型,垃圾回收不持有初始化锁。"""
+ owner.close()
+ discarded: list[object] = []
+ with LOCAL_MODEL_CACHE_LOCK:
+ for resource in tuple(cache.values()):
+ resource.owners.discard(owner)
+ _collect_unused(resource, discarded)
+ if discarded:
+ discarded.clear()
+ gc.collect()
+
+
+__all__ = [
+ "LOCAL_MODEL_CACHE_LOCK",
+ "LocalModelResource",
+ "current_local_resource",
+ "get_local_model",
+ "local_resource_scope",
+ "release_local_owner",
+]
diff --git a/mineru/model/runtime/onnx_device.py b/mineru/model/runtime/onnx_device.py
index 2bcff873..add2c50b 100644
--- a/mineru/model/runtime/onnx_device.py
+++ b/mineru/model/runtime/onnx_device.py
@@ -128,9 +128,9 @@ def resolve_onnx_device(
*, intra_op_num_threads: int = 0, inter_op_num_threads: int = 0, allow_download: bool = True
) -> OnnxSelection:
"""Resolve at first model use; imports and diagnostic commands never call this function."""
- from ...config import config, get_config_file_path, get_config_source
+ from ...config import get_config, get_config_file_path, get_config_source
- requested = config.model.onnx_device
+ requested = get_config().model.onnx_device
source = get_config_source("model.onnx_device")
path = get_config_file_path()
key = (os.getpid(), path, requested, source, allow_download)
@@ -170,9 +170,9 @@ def confirm_gpu_session(selection: OnnxSelection) -> None:
"""Persist a discrete-GPU auto choice only after an actual DML session succeeded."""
if selection.reason != "discrete_gpu":
return
- from ...config import config, get_config_source, update_config_file
+ from ...config import get_config, get_config_source, update_config_file
- if config.model.onnx_device == "auto" and get_config_source("model.onnx_device") != "env":
+ if get_config().model.onnx_device == "auto" and get_config_source("model.onnx_device") != "env":
try:
update_config_file({"model": {"onnx_device": "gpu"}}, only_if_onnx_auto=True)
except OSError as error:
diff --git a/mineru/model/runtime/onnx_rec.py b/mineru/model/runtime/onnx_rec.py
index 26156d90..a3f4b433 100644
--- a/mineru/model/runtime/onnx_rec.py
+++ b/mineru/model/runtime/onnx_rec.py
@@ -99,9 +99,9 @@ def _reduce_graph(source: bytes) -> tuple[bytes, int]:
def prepare_rec_model(model_path: str, *, cache_dir: Path | None = None) -> CompactRecModel:
"""Keep the source immutable and atomically cache a checked, compact-output graph."""
if cache_dir is None:
- from ...config import config
+ from ...config import get_config
- cache_dir = Path(config.model.base_dir).expanduser() / ".derived" / "ocr-rec"
+ cache_dir = Path(get_config().model.base_dir).expanduser() / ".derived" / "ocr-rec"
source = Path(model_path).read_bytes()
source_sha = hashlib.sha256(source).hexdigest()
target = cache_dir / f"{REC_OUTPUT_FORMAT}-{source_sha}.onnx"
diff --git a/mineru/model/runtime/ownership.py b/mineru/model/runtime/ownership.py
new file mode 100644
index 00000000..9d300e2e
--- /dev/null
+++ b/mineru/model/runtime/ownership.py
@@ -0,0 +1,155 @@
+# Copyright (c) Opendatalab. All rights reserved.
+"""应用与文档的模型租约,统一协调共享推理资源的释放。"""
+
+from __future__ import annotations
+
+import threading
+from collections.abc import Iterator
+from contextlib import contextmanager
+from contextvars import ContextVar
+
+
+class RuntimeOwner:
+ """标识应用持有的模型租约,并等待已接纳的文档自然完成。"""
+
+ def __init__(self) -> None:
+ """初始化关闭状态、引擎错误及受保护的在途文档计数。"""
+ self.closed = False
+ self.engine_error: str | None = None
+ self._documents = 0
+ self._initializations = 0
+ self._condition = threading.Condition()
+
+ def acquire_document(self) -> None:
+ """接纳新文档;关闭开始后不允许新的模型使用范围进入。"""
+ with self._condition:
+ if self.closed:
+ raise RuntimeError("Runtime owner is closed")
+ self._documents += 1
+
+ def release_document(self) -> None:
+ """文档全部清理后归还租约,唤醒等待应用资源回收的线程。"""
+ with self._condition:
+ self._documents -= 1
+ self._condition.notify_all()
+
+ def close(self) -> None:
+ """拒绝新文档并等待既有文档退出,不强停同步模型工作。"""
+ lease = _document_lease.get()
+ if lease is not None and lease.active and lease.owner is self:
+ raise RuntimeError("Cannot release a runtime owner from its active document")
+ initialization = _initialization_lease.get()
+ if initialization is not None and initialization.active and initialization.owner is self:
+ raise RuntimeError("Cannot release a runtime owner from its active initialization")
+ with self._condition:
+ self.closed = True
+ while self._documents or self._initializations:
+ self._condition.wait()
+
+ def acquire_initialization(self) -> None:
+ """接纳一次模型获取或初始化,使应用关闭等待构造及失败清理完整结束。"""
+ with self._condition:
+ if current_runtime_owner() is not self:
+ raise RuntimeError("Runtime initialization owner does not match the current context")
+ self._initializations += 1
+
+ def release_initialization(self) -> None:
+ """构造结果已发布或清理完毕后归还初始化租约。"""
+ with self._condition:
+ self._initializations -= 1
+ self._condition.notify_all()
+
+
+class _RuntimeLease:
+ """让复制到工作线程的上下文共享同一个有效期标记。"""
+
+ def __init__(self, owner: RuntimeOwner) -> None:
+ """保留接纳当前操作的应用身份,并标记操作尚未退出。"""
+ self.owner = owner
+ self.active = True
+
+
+runtime_owner: ContextVar[RuntimeOwner | None] = ContextVar("mineru_runtime_owner", default=None)
+_document_lease: ContextVar[_RuntimeLease | None] = ContextVar("mineru_runtime_document", default=None)
+_initialization_lease: ContextVar[_RuntimeLease | None] = ContextVar("mineru_runtime_initialization", default=None)
+
+
+def current_runtime_owner() -> RuntimeOwner | None:
+ """校验借用权限;关闭中的应用仅允许仍有效的既有文档惰性加载模型。"""
+ owner = runtime_owner.get()
+ lease = _document_lease.get()
+ if lease is not None and lease.owner is owner and not lease.active:
+ raise RuntimeError("Runtime document lease has ended")
+ initialization = _initialization_lease.get()
+ if initialization is not None and initialization.owner is owner and not initialization.active:
+ raise RuntimeError("Runtime initialization lease has ended")
+ if owner is not None and owner.closed and not (lease is not None and lease.owner is owner and lease.active):
+ raise RuntimeError("Runtime owner is closed")
+ return owner
+
+
+@contextmanager
+def runtime_document_scope() -> Iterator[None]:
+ """为整份文档保留应用模型,异步线程复制上下文时也沿用同一租约。"""
+ owner = runtime_owner.get()
+ if owner is None:
+ yield
+ return
+ owner.acquire_document()
+ lease = _RuntimeLease(owner)
+ token = _document_lease.set(lease)
+ try:
+ yield
+ finally:
+ lease.active = False
+ _document_lease.reset(token)
+ owner.release_document()
+
+
+@contextmanager
+def runtime_initialization_scope() -> Iterator[None]:
+ """先校验应用再解析模型路径;构造期间关闭应用时,等待清理而不发布迟到实例。"""
+ owner = current_runtime_owner()
+ if owner is None:
+ yield
+ return
+ owner.acquire_initialization()
+ lease = _RuntimeLease(owner)
+ token = _initialization_lease.set(lease)
+ try:
+ yield
+ finally:
+ lease.active = False
+ _initialization_lease.reset(token)
+ owner.release_initialization()
+
+
+def release_runtime_owner(owner: RuntimeOwner) -> None:
+ """排空已接纳文档后释放所有模型池,某个池失败也继续清理其余资源。"""
+ owner.close()
+ from .hybrid import AtomModelSingleton, HybridLocalModelContextSingleton
+ from ..vlm.runtime import ModelSingleton
+
+ first_error: BaseException | None = None
+ for release in (
+ ModelSingleton().release_owner,
+ HybridLocalModelContextSingleton().release_owner,
+ AtomModelSingleton().release_owner,
+ ):
+ try:
+ release(owner)
+ except BaseException as exc:
+ if first_error is None:
+ first_error = exc
+ if first_error is not None:
+ raise first_error
+
+
+__all__ = [
+ "RuntimeOwner",
+ "current_runtime_owner",
+ "release_runtime_owner",
+ "runtime_document_scope",
+ "runtime_initialization_scope",
+ "runtime_owner",
+]
diff --git a/mineru/model/vlm/async_runtime.py b/mineru/model/vlm/async_runtime.py
index 6a4cbc1f..38cfe49e 100644
--- a/mineru/model/vlm/async_runtime.py
+++ b/mineru/model/vlm/async_runtime.py
@@ -7,32 +7,20 @@ import asyncio
import concurrent.futures
import threading
from collections.abc import Awaitable, Callable, Sequence
-from contextvars import ContextVar
from typing import TYPE_CHECKING, Any, TypeVar
from ...utils.async_utils import drain_future, run_sync
+from ..runtime import ownership
from .errors import EngineDeadError, is_fatal_engine_error
+from .handles import RuntimeHandle
if TYPE_CHECKING:
- from mineru_vl_utils import MinerUClient
from mineru_vl_utils.structs import ContentBlock, ExtractResult
from PIL.Image import Image
T = TypeVar("T")
-class RuntimeOwner:
- """标识应用持有的模型租约;独立解析使用进程级缓存租约。"""
-
- def __init__(self) -> None:
- """记录关闭状态,阻止已关闭应用重新取得模型。"""
- self.closed = False
- self.engine_error: str | None = None
-
-
-runtime_owner: ContextVar[RuntimeOwner | None] = ContextVar("mineru_vlm_runtime_owner", default=None)
-
-
class _Submission:
"""分离请求结果与清理完成信号,避免 Future 取消被误认为推理退出。"""
@@ -56,8 +44,7 @@ class AsyncVlmPredictor:
def __init__(
self,
- factory: Callable[[], MinerUClient],
- dispose: Callable[[MinerUClient], None],
+ factory: Callable[[], RuntimeHandle],
*,
backend: str,
max_concurrency: int,
@@ -67,10 +54,9 @@ class AsyncVlmPredictor:
raise ValueError("VLM max_concurrency must be positive")
self.backend = backend
self.max_concurrency = max_concurrency
- self.owners: set[RuntimeOwner] = set()
+ self.owners: set[ownership.RuntimeOwner] = set()
self.pinned = False
self._factory = factory
- self._dispose = dispose
self._ready: concurrent.futures.Future[None] = concurrent.futures.Future()
self._state_lock = threading.Lock()
self._closing = False
@@ -82,8 +68,13 @@ class AsyncVlmPredictor:
async def _initialize(self) -> None:
"""在固定循环创建引擎和共享请求信号量。"""
- self._predictor = self._factory()
- self._semaphore = asyncio.Semaphore(self.max_concurrency)
+ self._runtime_handle = self._factory()
+ try:
+ self._predictor = self._runtime_handle.predictor
+ self._semaphore = asyncio.Semaphore(self.max_concurrency)
+ except BaseException:
+ await self._runtime_handle.aclose()
+ raise
def _serve(self) -> None:
"""运行独立事件循环,退出前回收异步生成器及线程池工作。"""
@@ -136,7 +127,7 @@ class AsyncVlmPredictor:
self._fatal_error = exc
for owner in tuple(self.owners):
owner.engine_error = str(exc)
- owner = runtime_owner.get()
+ owner = ownership.runtime_owner.get()
if owner is not None:
owner.engine_error = str(exc)
raise
@@ -281,9 +272,10 @@ class AsyncVlmPredictor:
submission.cancel()
await asyncio.gather(*(asyncio.wrap_future(item.finished) for item in submissions))
try:
- await self._predictor.aclose()
+ await self._runtime_handle.aclose()
finally:
- self._dispose(self._predictor)
+ self._predictor = None
+ self._runtime_handle = None
def shutdown(self) -> None:
"""幂等关闭运行时,等待线程退出后允许重新初始化。"""
@@ -311,4 +303,4 @@ class AsyncVlmPredictor:
await run_sync(self.shutdown)
-__all__ = ["AsyncVlmPredictor", "RuntimeOwner", "runtime_owner"]
+__all__ = ["AsyncVlmPredictor"]
diff --git a/mineru/model/vlm/client.py b/mineru/model/vlm/client.py
index 67147765..0e2ce720 100644
--- a/mineru/model/vlm/client.py
+++ b/mineru/model/vlm/client.py
@@ -4,17 +4,15 @@ from __future__ import annotations
from typing import TYPE_CHECKING
-from ...config import VlmConfig, config
+from ...config import VlmConfig, get_config
if TYPE_CHECKING:
- from mineru_vl_utils import MinerUClient
+ from .contracts import VlmPredictor
- from .async_runtime import AsyncVlmPredictor
-
-def get_vlm_predictor(vlm_config: VlmConfig | None = None) -> tuple[MinerUClient | AsyncVlmPredictor, str]:
+def get_vlm_predictor(vlm_config: VlmConfig | None = None) -> tuple[VlmPredictor, str]:
"""按连接配置选择远程客户端或本地引擎,预加载与真实解析复用相同缓存。"""
- settings = (vlm_config if vlm_config is not None else config.model.vlm).model_copy(deep=True)
+ settings = (vlm_config if vlm_config is not None else get_config().model.vlm).model_copy(deep=True)
settings.validate_environment()
from .runtime import ModelSingleton
@@ -43,7 +41,7 @@ def get_vlm_predictor(vlm_config: VlmConfig | None = None) -> tuple[MinerUClient
def uses_native_async_vlm(vlm_config: VlmConfig | None = None) -> bool:
"""不加载模型即可判断当前连接是否支持本期原生异步编排。"""
- settings = vlm_config if vlm_config is not None else config.model.vlm
+ settings = vlm_config if vlm_config is not None else get_config().model.vlm
if settings.server_url:
return True
from .selector import resolve_vlm_engine
diff --git a/mineru/model/vlm/handles.py b/mineru/model/vlm/handles.py
new file mode 100644
index 00000000..6c9aee43
--- /dev/null
+++ b/mineru/model/vlm/handles.py
@@ -0,0 +1,229 @@
+# Copyright (c) Opendatalab. All rights reserved.
+"""宿主持有的 VLM 客户端与确定的后端关闭动作。"""
+
+from __future__ import annotations
+
+import asyncio
+import threading
+from collections.abc import AsyncIterator, Callable, Iterator, Sequence
+from contextlib import asynccontextmanager, contextmanager
+from typing import TYPE_CHECKING
+
+from ...utils.async_utils import drain_future, run_sync
+from ..runtime.closing import runtime_close_scope
+from ..runtime.ownership import RuntimeOwner
+
+if TYPE_CHECKING:
+ from mineru_vl_utils import MinerUClient
+ from mineru_vl_utils.structs import ContentBlock, ExtractResult
+ from PIL.Image import Image
+
+
+class RuntimeHandle:
+ """将客户端、串行执行锁和创建时确定的资源释放动作放在同一宿主对象中。"""
+
+ def __init__(
+ self,
+ predictor: MinerUClient,
+ *,
+ backend: str,
+ close_engine: Callable[[], None] | None = None,
+ ) -> None:
+ """保存已创建资源,不修改第三方客户端的属性或推测其内部结构。"""
+ self.backend = backend
+ self.owners: set[RuntimeOwner] = set()
+ self.pinned = False
+ self.execution_lock = threading.Lock() if backend == "mlx-engine" else None
+ self._predictor: MinerUClient | None = predictor
+ self._close_engine = close_engine
+ self._condition = threading.Condition()
+ self._active = 0
+ self._closing = False
+ self._closed = threading.Event()
+
+ @property
+ def predictor(self) -> MinerUClient:
+ """只向托管异步运行时交付仍有效的底层客户端。"""
+ if self._predictor is None:
+ raise RuntimeError("VLM runtime is closed")
+ return self._predictor
+
+ @property
+ def is_closed(self) -> bool:
+ """让缓存识别正在关闭或已经关闭的宿主句柄。"""
+ return self._closing
+
+ @contextmanager
+ def _use(self, *, acquire_lock: bool = True) -> Iterator[MinerUClient]:
+ """等待同步推理自然返回后才允许关闭引擎,MLX 使用宿主串行锁。"""
+ with self._condition:
+ if self._closing:
+ raise RuntimeError("VLM runtime is shutting down")
+ self._active += 1
+ predictor = self.predictor
+ try:
+ if self.execution_lock is None or not acquire_lock:
+ yield predictor
+ else:
+ with self.execution_lock:
+ yield predictor
+ finally:
+ with self._condition:
+ self._active -= 1
+ self._condition.notify_all()
+
+ @asynccontextmanager
+ async def _async_use(self) -> AsyncIterator[MinerUClient]:
+ """可取消地等待串行锁,排队期间取消不得让迟到工作线程开始推理。"""
+ lock = self.execution_lock
+ if lock is not None:
+ while True:
+ with self._condition:
+ if self._closing:
+ raise RuntimeError("VLM runtime is shutting down")
+ if lock.acquire(blocking=False):
+ break
+ await asyncio.sleep(0.01)
+ try:
+ with self._use(acquire_lock=False) as predictor:
+ yield predictor
+ finally:
+ if lock is not None:
+ lock.release()
+
+ def batch_extract_with_layout(
+ self,
+ images: list[Image],
+ blocks_list: Sequence[Sequence[ContentBlock | dict]],
+ *,
+ not_extract_list: list[str] | None = None,
+ image_analysis: bool | None = None,
+ ) -> list[ExtractResult]:
+ """同步抽取期间持有客户端,关闭操作等待本次调用完整退出。"""
+ with self._use() as predictor:
+ return predictor.batch_extract_with_layout(
+ images, blocks_list, not_extract_list=not_extract_list, image_analysis=image_analysis
+ )
+
+ def batch_two_step_extract(
+ self,
+ images: list[Image],
+ *,
+ not_extract_list: list[str] | None = None,
+ image_analysis: bool | None = None,
+ ) -> list[ExtractResult]:
+ """同步两阶段推理沿用同一资源使用范围,不强制中断 CPU 执行。"""
+ with self._use() as predictor:
+ return predictor.batch_two_step_extract(images, not_extract_list=not_extract_list, image_analysis=image_analysis)
+
+ async def aio_batch_extract_with_layout(
+ self,
+ images: list[Image],
+ blocks_list: Sequence[Sequence[ContentBlock | dict]],
+ *,
+ not_extract_list: list[str] | None = None,
+ image_analysis: bool | None = None,
+ ) -> list[ExtractResult]:
+ """同步后端的异步调用等待工作线程排空,取消不会提前关闭资源。"""
+ async with self._async_use() as predictor:
+ return await run_sync(
+ predictor.batch_extract_with_layout,
+ images,
+ blocks_list,
+ not_extract_list=not_extract_list,
+ image_analysis=image_analysis,
+ )
+
+ async def aio_batch_two_step_extract(
+ self,
+ images: list[Image],
+ *,
+ not_extract_list: list[str] | None = None,
+ image_analysis: bool | None = None,
+ ) -> list[ExtractResult]:
+ """在线程中运行同步两阶段推理,并保留取消后的资源清理边界。"""
+ async with self._async_use() as predictor:
+ return await run_sync(
+ predictor.batch_two_step_extract, images, not_extract_list=not_extract_list, image_analysis=image_analysis
+ )
+
+ def _begin_close(self) -> bool:
+ """只让一个调用方执行关闭,其余调用方等待同一个完成信号。"""
+ with self._condition:
+ if self._closing:
+ return False
+ self._closing = True
+ return True
+
+ def _wait_for_calls(self) -> None:
+ """在缓存锁之外等待所有同步调用退出。"""
+ with self._condition:
+ while self._active:
+ self._condition.wait()
+
+ def _finish_close(self, first_error: BaseException | None = None) -> None:
+ """执行唯一确定的后端关闭动作,并在失败时同样释放宿主强引用。"""
+ close_engine, self._close_engine = self._close_engine, None
+ try:
+ if close_engine is not None:
+ with runtime_close_scope():
+ close_engine()
+ except BaseException as exc:
+ if first_error is None:
+ first_error = exc
+ finally:
+ self._predictor = None
+ self._closed.set()
+ if first_error is not None:
+ raise first_error
+
+ def shutdown(self) -> None:
+ """同步后端排空调用后关闭;HTTP 句柄的连接关闭仍交给其公开异步接口。"""
+ if not self._begin_close():
+ self._closed.wait()
+ return
+ error: BaseException | None = None
+ try:
+ with runtime_close_scope():
+ self._wait_for_calls()
+ if self.backend == "http-client":
+ asyncio.run(self.predictor.aclose())
+ except BaseException as exc:
+ error = exc
+ self._finish_close(error)
+
+ async def _close_client(self) -> None:
+ """在受保护的任务中排空推理并关闭客户端,等待异常也不跳过连接清理。"""
+ error: BaseException | None = None
+ with runtime_close_scope():
+ try:
+ await run_sync(self._wait_for_calls)
+ except BaseException as exc:
+ error = exc
+ try:
+ await self.predictor.aclose()
+ except BaseException as exc:
+ if error is None:
+ error = exc
+ if error is not None:
+ raise error
+
+ async def aclose(self) -> None:
+ """取消时仍排空客户端关闭,再释放引擎并传播最早的取消或清理异常。"""
+ if not self._begin_close():
+ await run_sync(self._closed.wait)
+ return
+ work = asyncio.create_task(self._close_client())
+ error: BaseException | None = None
+ try:
+ await asyncio.shield(work)
+ except BaseException as exc:
+ error = exc
+ try:
+ await drain_future(work)
+ except BaseException:
+ pass
+ self._finish_close(error)
+
+
+__all__ = ["RuntimeHandle"]
diff --git a/mineru/model/vlm/runtime.py b/mineru/model/vlm/runtime.py
index 968e9c40..c74cff8b 100644
--- a/mineru/model/vlm/runtime.py
+++ b/mineru/model/vlm/runtime.py
@@ -1,7 +1,6 @@
# Copyright (c) Opendatalab. All rights reserved.
from __future__ import annotations
-import asyncio
import atexit
import gc
import hashlib
@@ -9,10 +8,12 @@ import json
import os
import threading
import time
-from contextlib import asynccontextmanager, contextmanager
+from collections.abc import Callable
+from contextvars import ContextVar, copy_context
+from dataclasses import dataclass, field
from pathlib import Path
from functools import partial
-from typing import Any, AsyncIterator, Generator, Iterator, Literal
+from typing import Any, Literal
from loguru import logger
from mineru_vl_utils import MinerUClient
@@ -20,7 +21,10 @@ from mineru_vl_utils import MinerUClient
from ..registry import MINERU_2_5_PRO_2605_1_2B, vlm_model_repo
from ..runtime.device import get_device
from ..runtime.platform import is_mac_os_version_supported
-from .async_runtime import AsyncVlmPredictor, RuntimeOwner, runtime_owner
+from ..runtime.closing import in_runtime_close_scope, runtime_close_scope
+from ..runtime.ownership import RuntimeOwner, current_runtime_owner, runtime_initialization_scope
+from .async_runtime import AsyncVlmPredictor
+from .handles import RuntimeHandle
from .engine_utils import (
enable_custom_logits_processors,
mod_kwargs_by_device_type,
@@ -30,9 +34,21 @@ from .engine_utils import (
)
+@dataclass(eq=False)
+class _RuntimeInitialization:
+ """保存同键初始化屏障,直到构造发布或失败清理完整结束。"""
+
+ finished: threading.Event = field(default_factory=threading.Event)
+
+
+_initializing_runtime: ContextVar[_RuntimeInitialization | None] = ContextVar("mineru_initializing_vlm", default=None)
+
+
class ModelSingleton:
_instance = None
- _models = {}
+ _models: dict[tuple[object, ...], RuntimeHandle | AsyncVlmPredictor] = {}
+ _closing: dict[tuple[object, ...], threading.Event] = {}
+ _initializing: dict[tuple[object, ...], _RuntimeInitialization] = {}
_lock = threading.RLock()
_shutdown_registered = False
@@ -58,64 +74,154 @@ class ModelSingleton:
*,
model_name: str | None = None,
**kwargs: Any,
- ) -> MinerUClient | AsyncVlmPredictor:
- """缓存模型实例,远程客户端按模型、凭据与连接选项隔离。"""
- # 文档同步与异步入口统一使用一个原生异步 vLLM 实例。
- if backend == "vllm-engine":
- backend = "vllm-async-engine"
- native_async = backend in {"http-client", "vllm-async-engine"}
- options = {"model_name": model_name, **kwargs}
- if backend == "vllm-async-engine":
- options["device"] = get_device()
- model_path = str(Path(model_path).expanduser().resolve()) if model_path else str(MINERU_2_5_PRO_2605_1_2B.ensure())
- fingerprint = hashlib.sha256(json.dumps(options, sort_keys=True, default=str).encode()).hexdigest()
- cache_key = (backend, model_path, server_url, fingerprint)
- owner = runtime_owner.get()
- with self._lock:
- if owner is not None and owner.closed:
- raise RuntimeError("VLM runtime owner is closed")
- if not self._shutdown_registered:
- atexit.register(shutdown_cached_models)
- type(self)._shutdown_registered = True
- cached = self._models.get(cache_key)
- if isinstance(cached, AsyncVlmPredictor) and cached.is_closed:
- cached.shutdown()
- del self._models[cache_key]
- if cache_key not in self._models:
- factory = partial(
- self._create_model,
- backend,
- model_path,
- server_url,
- model_name=model_name,
- **kwargs,
- )
- self._models[cache_key] = (
- AsyncVlmPredictor(
- factory,
- _shutdown_predictor_runtime,
- backend=backend,
- max_concurrency=kwargs.get("max_concurrency", 100),
- )
- if native_async
- else factory()
+ ) -> RuntimeHandle | AsyncVlmPredictor:
+ """先接纳应用操作,再沿原缓存键复用运行时;路径解析和构造都不持有缓存锁。"""
+ with runtime_initialization_scope():
+ if backend == "vllm-engine":
+ backend = "vllm-async-engine"
+ native_async = backend in {"http-client", "vllm-async-engine"}
+ options = {"model_name": model_name, **kwargs}
+ if backend == "vllm-async-engine":
+ options["device"] = get_device()
+ model_path = (
+ str(Path(model_path).expanduser().resolve()) if model_path else str(MINERU_2_5_PRO_2605_1_2B.ensure())
)
- predictor = self._models[cache_key]
- if isinstance(predictor, AsyncVlmPredictor):
- if owner is None:
- predictor.pinned = True
+ fingerprint = hashlib.sha256(json.dumps(options, sort_keys=True, default=str).encode()).hexdigest()
+ cache_key = (backend, model_path, server_url, fingerprint)
+ factory = partial(self._create_model, backend, model_path, server_url, model_name=model_name, **kwargs)
+ return self._get_or_initialize(
+ cache_key,
+ factory,
+ native_async=native_async,
+ backend=backend,
+ max_concurrency=kwargs.get("max_concurrency", 100),
+ )
+
+ def _retain(self, predictor: RuntimeHandle | AsyncVlmPredictor) -> None:
+ """发布或复用前重新校验应用,防止关闭后的预热结果进入缓存。"""
+ owner = current_runtime_owner()
+ if owner is None:
+ predictor.pinned = True
+ else:
+ predictor.owners.add(owner)
+
+ def _get_or_initialize(
+ self,
+ key: tuple[object, ...],
+ factory: Callable[[], RuntimeHandle],
+ *,
+ native_async: bool,
+ backend: str,
+ max_concurrency: int,
+ ) -> RuntimeHandle | AsyncVlmPredictor:
+ """同键共享初始化或关闭屏障,外部构造和关闭动作始终在全局锁之外。"""
+ while True:
+ detached = []
+ reservation = None
+ with self._lock:
+ current_runtime_owner()
+ closing = self._closing.get(key)
+ initializing = self._initializing.get(key)
+ cached = self._models.get(key)
+ if in_runtime_close_scope() and (
+ closing is not None or initializing is not None or (cached is not None and cached.is_closed)
+ ):
+ raise RuntimeError("Cannot acquire a closing VLM runtime from a close callback")
+ if initializing is not None and initializing is _initializing_runtime.get():
+ raise RuntimeError("Cannot recursively acquire a VLM runtime during its initialization")
+ if closing is not None:
+ pending = closing
+ elif initializing is not None:
+ pending = initializing.finished
+ elif cached is not None and cached.is_closed:
+ pending = None
+ detached.append(self._detach(key, cached))
+ elif cached is not None:
+ self._retain(cached)
+ return cached
else:
- predictor.owners.add(owner)
- if isinstance(predictor, AsyncVlmPredictor):
- try:
+ pending = None
+ reservation = _RuntimeInitialization()
+ self._initializing[key] = reservation
+ if not self._shutdown_registered:
+ atexit.register(shutdown_cached_models)
+ type(self)._shutdown_registered = True
+ if pending is not None:
+ pending.wait()
+ elif detached:
+ self._close_detached(detached)
+ elif reservation is not None:
+ return self._initialize_model(
+ key, reservation, factory, native_async=native_async, backend=backend, max_concurrency=max_concurrency
+ )
+
+ def _initialize_model(
+ self,
+ key: tuple[object, ...],
+ reservation: _RuntimeInitialization,
+ factory: Callable[[], RuntimeHandle],
+ *,
+ native_async: bool,
+ backend: str,
+ max_concurrency: int,
+ ) -> RuntimeHandle | AsyncVlmPredictor:
+ """锁外构造并等待就绪,失效应用或失败构造完整清理后才解除同键屏障。"""
+ predictor: RuntimeHandle | AsyncVlmPredictor | None = None
+ token = _initializing_runtime.set(reservation)
+ try:
+ if native_async:
+ factory_context = copy_context()
+ predictor = AsyncVlmPredictor(
+ partial(factory_context.run, factory), backend=backend, max_concurrency=max_concurrency
+ )
predictor.wait_ready()
- except BaseException:
- with self._lock:
- if self._models.get(cache_key) is predictor:
- del self._models[cache_key]
+ else:
+ predictor = factory()
+ with self._lock:
+ self._retain(predictor)
+ self._models[key] = predictor
+ return predictor
+ except BaseException as initialization_error:
+ if predictor is not None:
+ try:
+ with runtime_close_scope():
+ predictor.shutdown()
+ except BaseException as cleanup_error:
+ raise initialization_error from cleanup_error
+ raise
+ finally:
+ _initializing_runtime.reset(token)
+ with self._lock:
+ self._initializing.pop(key, None)
+ reservation.finished.set()
+
+ def _detach(
+ self, key: tuple[object, ...], predictor: RuntimeHandle | AsyncVlmPredictor
+ ) -> tuple[tuple[object, ...], RuntimeHandle | AsyncVlmPredictor, threading.Event]:
+ """在缓存锁内撤销条目并保留关闭屏障,防止相同权重在关闭完成前重新创建。"""
+ del self._models[key]
+ finished = threading.Event()
+ self._closing[key] = finished
+ return key, predictor, finished
+
+ def _close_detached(
+ self, entries: list[tuple[tuple[object, ...], RuntimeHandle | AsyncVlmPredictor, threading.Event]]
+ ) -> None:
+ """锁外逐个关闭已撤销条目,某个后端失败也释放其余条目和等待屏障。"""
+ first_error: BaseException | None = None
+ for key, predictor, finished in entries:
+ try:
+ with runtime_close_scope():
predictor.shutdown()
- raise
- return predictor
+ except BaseException as exc:
+ if first_error is None:
+ first_error = exc
+ finally:
+ with self._lock:
+ self._closing.pop(key, None)
+ finished.set()
+ if first_error is not None:
+ raise first_error
def _create_model(
self,
@@ -125,7 +231,7 @@ class ModelSingleton:
*,
model_name: str | None = None,
**kwargs: Any,
- ) -> MinerUClient:
+ ) -> RuntimeHandle:
"""只构造一个底层客户端,不操作缓存;原生异步引擎由所属运行时调用。"""
start_time = time.time()
model = None
@@ -133,6 +239,7 @@ class ModelSingleton:
vllm_llm = None
lmdeploy_engine = None
vllm_async_llm = None
+ close_engine = None
llama_cpp_engine = None
batch_size = kwargs.get("batch_size", 0) # 本地引擎批次大小;MLX 的 0 由客户端解析为保守默认值。
max_concurrency = kwargs.get("max_concurrency", 100) # 原生异步运行时在所有文档间共用该额度。
@@ -183,6 +290,7 @@ class ModelSingleton:
raise ImportError("Please install mineru-llama-cpp to use the llama-cpp-engine backend.")
with llama_engine_scope(selection):
llama_cpp_engine = Engine(model_gguf, mmproj_gguf, **engine_kwargs)
+ close_engine = llama_cpp_engine.close
elif backend == "transformers":
from ..runtime.cpu_threads import initialize_torch_threads
@@ -211,32 +319,7 @@ class ModelSingleton:
if os.getenv("OMP_NUM_THREADS") is None:
os.environ["OMP_NUM_THREADS"] = "1"
- if backend == "vllm-engine":
- try:
- import vllm
- except ImportError:
- raise ImportError("Please install vllm to use the vllm-engine backend.")
-
- kwargs = mod_kwargs_by_device_type(kwargs, vllm_mode="sync_engine")
-
- if "compilation_config" in kwargs:
- if isinstance(kwargs["compilation_config"], str):
- try:
- kwargs["compilation_config"] = json.loads(kwargs["compilation_config"])
- except json.JSONDecodeError:
- logger.warning(f"Failed to parse compilation_config as JSON: {kwargs['compilation_config']}")
- del kwargs["compilation_config"]
- if "gpu_memory_utilization" not in kwargs:
- kwargs["gpu_memory_utilization"] = set_default_gpu_memory_utilization()
- if "model" not in kwargs:
- kwargs["model"] = model_path
- if enable_custom_logits_processors() and ("logits_processors" not in kwargs):
- from mineru_vl_utils import MinerULogitsProcessor
-
- kwargs["logits_processors"] = [MinerULogitsProcessor]
- # 使用kwargs为 vllm初始化参数
- vllm_llm = vllm.LLM(**kwargs)
- elif backend == "vllm-async-engine":
+ if backend == "vllm-async-engine":
try:
from vllm.config import CompilationConfig
from vllm.engine.arg_utils import AsyncEngineArgs
@@ -268,6 +351,7 @@ class ModelSingleton:
kwargs["logits_processors"] = [MinerULogitsProcessor]
# 使用kwargs为 vllm初始化参数
vllm_async_llm = AsyncLLM.from_engine_args(AsyncEngineArgs(**kwargs))
+ close_engine = vllm_async_llm.shutdown
elif backend == "lmdeploy-engine":
try:
from lmdeploy import PytorchEngineConfig, TurbomindEngineConfig, pipeline
@@ -315,6 +399,7 @@ class ModelSingleton:
backend_config=backend_config,
log_level=log_level,
)
+ close_engine = lmdeploy_engine.close
try:
predictor = MinerUClient(
backend=backend,
@@ -336,49 +421,46 @@ class ModelSingleton:
image_analysis=True,
enable_cross_page_table_merge=False,
)
- except BaseException:
- # 客户端或分词器构造失败时,已经成功创建的引擎仍由 MinerU 负责关闭。
- for handle in (vllm_async_llm, vllm_llm, lmdeploy_engine, llama_cpp_engine):
- if handle is not None:
- _shutdown_runtime_handle(handle)
+ except BaseException as initialization_error:
+ # 客户端构造失败后,宿主仍负责关闭已经成功创建的引擎。
+ if close_engine is not None:
+ try:
+ with runtime_close_scope():
+ close_engine()
+ except BaseException as cleanup_error:
+ raise initialization_error from cleanup_error
raise
- predictor._mineru_runtime_handles = {
- "backend": backend,
- "model": model,
- "processor": processor,
- "vllm_llm": vllm_llm,
- "vllm_async_llm": vllm_async_llm,
- "lmdeploy_engine": lmdeploy_engine,
- "llama_cpp_engine": llama_cpp_engine,
- }
- _maybe_enable_serial_execution(predictor, backend)
elapsed = round(time.time() - start_time, 2)
logger.info(f"get {backend} predictor cost: {elapsed}s")
- return predictor
+ return RuntimeHandle(predictor, backend=backend, close_engine=close_engine)
def release_owner(self, owner: RuntimeOwner) -> None:
- """释放应用租约,只关闭未被其他应用或独立解析持有的原生运行时。"""
+ """排空该应用文档后撤销全部后端租约,保留其它应用和独立解析持有的实例。"""
+ owner.close()
with self._lock:
- owner.closed = True
closing = []
- for key, predictor in list(self._models.items()):
- if not isinstance(predictor, AsyncVlmPredictor):
- continue
+ for key, predictor in tuple(self._models.items()):
predictor.owners.discard(owner)
if not predictor.owners and not predictor.pinned:
- del self._models[key]
- closing.append(predictor)
- _shutdown_predictors(closing)
+ closing.append(self._detach(key, predictor))
+ self._close_detached(closing)
def shutdown(self) -> None:
- """阻止关闭期间创建第二份权重,逐个等待缓存运行时完整退出。"""
- with self._lock:
- predictors = list(self._models.values())
- self._models.clear()
- try:
- _shutdown_predictors(predictors)
- finally:
- gc.collect()
+ """等待已开始的初始化收尾,撤销缓存后在锁外排空运行时。"""
+ while True:
+ with self._lock:
+ pending = tuple(self._initializing.values())
+ if not pending:
+ closing = [self._detach(key, predictor) for key, predictor in tuple(self._models.items())]
+ break
+ if _initializing_runtime.get() in pending or in_runtime_close_scope():
+ raise RuntimeError("Cannot wait for VLM initialization from its cleanup callback")
+ for item in pending:
+ item.finished.wait()
+ try:
+ self._close_detached(closing)
+ finally:
+ gc.collect()
async def _get_model_async(
@@ -394,7 +476,7 @@ async def _get_model_async(
model_path: str | None,
server_url: str | None,
**kwargs: Any,
-) -> MinerUClient:
+) -> RuntimeHandle | AsyncVlmPredictor:
"""模型初始化被取消时等待后台加载结束,防止清理后又写入迟到的模型句柄。"""
from mineru_vl_utils.vlm_client.utils import run_in_thread_until_complete
@@ -407,148 +489,9 @@ async def _get_model_async(
)
-def _iter_shutdown_candidates(predictor: MinerUClient) -> Generator[object, None, None]:
- runtime_handles = getattr(predictor, "_mineru_runtime_handles", {})
- client = getattr(predictor, "client", None)
-
- seen_ids = set()
-
- def _yield_candidate(candidate: Any) -> Iterator[Any]:
- if candidate is None:
- return
- candidate_id = id(candidate)
- if candidate_id in seen_ids:
- return
- seen_ids.add(candidate_id)
- yield candidate
-
- for key in ("vllm_llm", "vllm_async_llm", "lmdeploy_engine", "llama_cpp_engine", "model"):
- yield from _yield_candidate(runtime_handles.get(key))
-
- if client is not None:
- for key in ("vllm_llm", "vllm_async_llm", "lmdeploy_engine", "llama_cpp_engine", "model"):
- yield from _yield_candidate(getattr(client, key, None))
-
-
-def _call_nested_shutdown(target: object, method_path: str, label: str) -> bool:
- current = target
- for attr in method_path.split("."):
- current = getattr(current, attr, None)
- if current is None:
- return False
-
- if not callable(current):
- return False
-
- try:
- current()
- logger.debug(f"Shutdown {label} via `{method_path}`")
- return True
- except TypeError:
- logger.debug(f"Skip unsupported shutdown call {label}.{method_path}")
- return False
- except Exception as exc:
- logger.debug(f"Failed to shutdown {label} via `{method_path}`: {exc}")
- return False
-
-
-def _shutdown_runtime_handle(handle: object) -> None:
- for method_path in (
- "shutdown",
- "close",
- "stop",
- "terminate",
- "destroy",
- "engine.shutdown",
- "engine.close",
- "engine_core.shutdown",
- "engine_core.close",
- "llm_engine.shutdown",
- "llm_engine.close",
- "llm_engine.model_executor.shutdown",
- "llm_engine.model_executor.close",
- "model_executor.shutdown",
- "model_executor.close",
- ):
- if _call_nested_shutdown(handle, method_path, type(handle).__name__):
- return
-
-
-def _clear_predictor_references(predictor: MinerUClient) -> None:
- runtime_handles = getattr(predictor, "_mineru_runtime_handles", {})
- for key in tuple(runtime_handles.keys()):
- runtime_handles[key] = None
-
- client = getattr(predictor, "client", None)
- if client is not None:
- for attr in ("vllm_llm", "vllm_async_llm", "lmdeploy_engine", "llama_cpp_engine", "model", "processor"):
- if hasattr(client, attr):
- setattr(client, attr, None)
-
-
-def _shutdown_predictor_runtime(predictor: MinerUClient | AsyncVlmPredictor) -> None:
- """关闭托管运行时或其拥有的底层模型句柄。"""
- if isinstance(predictor, AsyncVlmPredictor):
- predictor.shutdown()
- return
- for handle in _iter_shutdown_candidates(predictor):
- _shutdown_runtime_handle(handle)
- _clear_predictor_references(predictor)
-
-
-def _shutdown_predictors(predictors: list[MinerUClient | AsyncVlmPredictor]) -> None:
- """某个客户端关闭失败时仍关闭其余实例,最后再传播最初的异常。"""
- first_error: BaseException | None = None
- for predictor in predictors:
- try:
- _shutdown_predictor_runtime(predictor)
- except BaseException as exc:
- if first_error is None:
- first_error = exc
- if first_error is not None:
- raise first_error
-
-
def shutdown_cached_models() -> None:
+ """进程结束时关闭包含独立解析固定租约在内的全部 VLM 运行时。"""
ModelSingleton().shutdown()
-def _predictor_uses_mlx(predictor: MinerUClient, backend: str | None = None) -> bool:
- if backend == "mlx-engine":
- return True
- client = getattr(predictor, "client", None)
- return type(client).__module__.endswith(".mlx_client")
-
-
-def _maybe_enable_serial_execution(predictor: MinerUClient, backend: str | None = None) -> MinerUClient:
- if _predictor_uses_mlx(predictor, backend) and not hasattr(predictor, "_mineru_execution_lock"):
- setattr(predictor, "_mineru_execution_lock", threading.Lock())
- return predictor
-
-
-@contextmanager
-def predictor_execution_guard(predictor: MinerUClient) -> Iterator[None]:
- lock = getattr(predictor, "_mineru_execution_lock", None)
- if lock is None:
- yield
- return
- with lock:
- yield
-
-
-@asynccontextmanager
-async def aio_predictor_execution_guard(predictor: MinerUClient) -> AsyncIterator[None]:
- """可取消地等待串行模型锁,避免被取消的后台 acquire 在线程中遗留持锁者。"""
- lock = getattr(predictor, "_mineru_execution_lock", None)
- if lock is None:
- yield
- return
- while not lock.acquire(blocking=False):
- await asyncio.sleep(0.01)
- try:
- yield
- finally:
- lock.release()
-
-
-__all__ = ["ModelSingleton", "shutdown_cached_models", "predictor_execution_guard", "aio_predictor_execution_guard"]
+__all__ = ["ModelSingleton", "shutdown_cached_models"]
diff --git a/mineru/model/vlm/selector.py b/mineru/model/vlm/selector.py
index 35f67148..024bf86c 100644
--- a/mineru/model/vlm/selector.py
+++ b/mineru/model/vlm/selector.py
@@ -27,9 +27,9 @@ VLM_REQUIRED_MODULES: dict[VlmEngineName, tuple[str, ...]] = {
def resolve_vlm_engine(engine: str | None = None) -> VlmEngineName:
"""按显式配置或平台能力选择引擎;显式名称不要求下载端安装引擎。"""
- from ...config import config
+ from ...config import get_config
- selected = config.model.vlm.engine if engine is None else engine
+ selected = get_config().model.vlm.engine if engine is None else engine
if selected == "llama-cpp":
return "llama-cpp"
if selected == "vllm":
diff --git a/mineru/parser/api_client.py b/mineru/parser/api_client.py
index 767f374b..0d68f9f3 100644
--- a/mineru/parser/api_client.py
+++ b/mineru/parser/api_client.py
@@ -32,8 +32,10 @@ from urllib.parse import unquote, urlparse, urljoin
import httpx
from docvortex.assets import validate_image_sidecar_path
+from pydantic import ValidationError
from ..task_types import TaskStatus, task_status_url
+from ..tasks.contracts import TaskFileIdentity
from ..types import PageInfo, Tier, validate_tier
from .base import DocumentParser, ParseResult
from .page_range import normalize_page_range_input
@@ -200,15 +202,17 @@ class MinerUApiParser(DocumentParser):
or not isinstance(files, list)
or len(files) != 1
or not isinstance(files[0], dict)
- or not isinstance(files[0].get("file_id"), str)
- or not files[0]["file_id"]
- or not isinstance(files[0].get("file_name"), str)
or (expected is not None and task["task_id"] != expected["task_id"])
):
raise _V1APIError("invalid_response", "Inconsistent Tasks response")
item = files[0]
- if item.get("status") not in ("waiting-file", "pending", "running", "completed", "failed", "canceled"):
- raise _V1APIError("invalid_response", "Invalid task file state")
+ try:
+ # 只校验共同身份字段,继续保留原始快照中的服务扩展和可选字段。
+ TaskFileIdentity.model_validate(item, strict=True)
+ except ValidationError as exc:
+ identity_invalid = any(error["loc"] != ("status",) for error in exc.errors())
+ message = "Inconsistent Tasks response" if identity_invalid else "Invalid task file state"
+ raise _V1APIError("invalid_response", message) from exc
link = item.get("upload")
if link is not None and (not isinstance(link, dict) or item["status"] != "waiting-file"):
raise _V1APIError("invalid_response", "Invalid task upload descriptor")
diff --git a/mineru/parser/api_server.py b/mineru/parser/api_server.py
index a5ab21d8..86bbe3dd 100644
--- a/mineru/parser/api_server.py
+++ b/mineru/parser/api_server.py
@@ -19,8 +19,8 @@ import time
from contextlib import asynccontextmanager
from copy import deepcopy
from dataclasses import dataclass, field
-from datetime import datetime, timezone
-from typing import Annotated, Any, AsyncIterator, Awaitable, Callable, Literal, NoReturn, TypedDict
+from datetime import datetime
+from typing import Annotated, Any, AsyncIterator, Awaitable, Callable, Literal, TypedDict
from urllib.parse import urlparse
import click
@@ -40,7 +40,7 @@ from starlette.datastructures import State
from starlette.types import Receive, Scope, Send
from ..config import VlmConfig
-from ..config import config as mineru_config
+from ..config import get_config
from ..errors import MineruError
from ..integrations.docvortex import MinerUMetadata
from ..filetypes import (
@@ -51,7 +51,7 @@ from ..filetypes import (
)
from ..types import SERVER_TIERS, TIERS_BY_SERVER_TIER, DeploymentTier, PageInfo, ServerTier, Tier, select_default_quality_tier
from ..utils.async_utils import drain_future, run_sync
-from ..model.vlm.async_runtime import RuntimeOwner, runtime_owner
+from ..model.runtime.ownership import RuntimeOwner, runtime_owner
from ..model.vlm.errors import is_fatal_engine_error
from ..utils.logger import configure_global_log_level
from ..utils.stdio import configure_standard_streams
@@ -75,10 +75,14 @@ from .tier import (
ensure_tier_runtime_dependencies,
runtime_options_for_tier,
)
-from ..task_types import TASK_OUTPUT_FORMATS, TaskFileState, TaskOutputFormat, TaskStatus
+from ..task_types import TASK_OUTPUT_FORMATS, TaskFileState, TaskOutputFormat
+from ..tasks.contracts import MAX_TASK_FILES, ErrorDetail, ErrorResponse, FileParseInfo
+from ..tasks.errors import ApiServerError, raise_api_error
+from ..tasks.execution import TaskExecution, TaskInputMetadata, execution_timestamp, new_execution_id
+from ..tasks.http import build_task_router
+from ..tasks.service import TaskStore
_DEFAULT_API_SERVER_TIER: ServerTier = "standard"
-_MAX_FILES_PER_JOB_DEFAULT = 100
_MAX_URL_BYTES_DEFAULT = 209715200
logger = logging.getLogger("mineru.parser.api_server")
@@ -90,12 +94,6 @@ class ParseServerStartupError(RuntimeError):
# ── literal type aliases ────────────────────────────────────────────
-JobStatus = TaskStatus
-"""Job lifecycle states."""
-
-FileStatus = TaskFileState
-"""Per-file status within a job."""
-
AccessLevel = Literal["anonymous", "registered"]
"""Account access level determined by API Key."""
@@ -122,47 +120,6 @@ _PYDANTIC_CONFIG = ConfigDict(extra="forbid")
# ── Error ────────────────────────────────────────────────────────────
-class ErrorDetail(BaseModel):
- model_config = _PYDANTIC_CONFIG
- type: str
- code: str | None = None
- message: str
- param: str | None = None
-
-
-class ErrorResponse(BaseModel):
- model_config = _PYDANTIC_CONFIG
- error: ErrorDetail
-
-
-class ApiServerError(Exception):
- """Structured API error raised by parser API business logic."""
-
- def __init__(self, status_code: int, error: ErrorDetail) -> None:
- super().__init__(error.message)
- self.status_code = status_code
- self.error = error
-
-
-def _raise_api_error(
- status_code: int,
- *,
- error_type: str,
- code: str | None,
- message: str,
- param: str | None = None,
-) -> NoReturn:
- raise ApiServerError(
- status_code,
- ErrorDetail(
- type=error_type,
- code=code,
- message=message,
- param=param,
- ),
- )
-
-
def _error_response(status_code: int, error: ErrorDetail) -> JSONResponse:
return JSONResponse(
status_code=status_code,
@@ -334,51 +291,24 @@ _ExecutionSource = Annotated[
]
-class _ParseInput(BaseModel):
+class _ExecutionRequest(BaseModel):
+ """本地单文件执行请求,公开结果身份与内部源文件引用分别保存。"""
+
model_config = _PYDANTIC_CONFIG
source: _ExecutionSource
+ file_id: str
+ file_name: str
+ bytes: int | None = None
+ sha256sum: str | None = None
page_range: str | None = Field(default=None, description=PAGE_RANGE_DESCRIPTION, examples=["1-5,8,r3-r1", "all"])
-
-
-class _ParseRequest(BaseModel):
- """Tasks 构造的私有执行请求,不属于公开 HTTP 协议。"""
-
- model_config = _PYDANTIC_CONFIG
- files: list[_ParseInput] = Field(min_length=1, max_length=_MAX_FILES_PER_JOB_DEFAULT)
tier: Tier | None = None
- ocr_mode: Literal["auto", "txt", "ocr"] = Field(
- default="auto", description="OCR mode for this parse job: auto-detect, native text, or forced OCR."
- )
+ ocr_mode: Literal["auto", "txt", "ocr"] = "auto"
output_formats: list[TaskOutputFormat] = Field(default_factory=lambda: ["default"])
# ── Job responses ────────────────────────────────────────────────────
-class TaskProgress(BaseModel):
- model_config = _PYDANTIC_CONFIG
- completed: int = 0
- failed: int = 0
- total: int = 0
-
-
-class FileParseInfo(BaseModel):
- model_config = _PYDANTIC_CONFIG
- tier: Tier
- duration_ms: int | None = None
- parser_version: str | None = None
-
-
-class _ExecutionFile(BaseModel):
- model_config = _PYDANTIC_CONFIG
- file_id: str | None = None
- name: str
- page_range: str
- status: FileStatus
- parse: FileParseInfo | None = None
- error: ErrorDetail | None = None
-
-
# ── Usage ────────────────────────────────────────────────────────────
@@ -399,7 +329,7 @@ class UsageLimits(BaseModel):
model_config = _PYDANTIC_CONFIG
max_pages_per_file: int | None = None
max_file_size_bytes: int | None = None
- max_files_per_job: int = _MAX_FILES_PER_JOB_DEFAULT
+ max_files_per_job: int = MAX_TASK_FILES
max_concurrent_jobs: int = 1
max_file_retention_days: int | None = None
@@ -689,14 +619,14 @@ class FileStore:
with self._lock:
rec = self.get_file(file_id)
if rec.purpose != "parse_output":
- _raise_api_error(
+ raise_api_error(
403,
error_type="permission_error",
code="feature_requires_api_key",
message="Source files cannot be downloaded",
)
if rec.sha256sum is None or not self._blob_abs(rec.sha256sum).is_file():
- _raise_api_error(404, error_type="invalid_request_error", code="file_not_found", message="Output expired")
+ raise_api_error(404, error_type="invalid_request_error", code="file_not_found", message="Output expired")
self._pins[rec.sha256sum] = self._pins.get(rec.sha256sum, 0) + 1
return self._blob_abs(rec.sha256sum), rec.filename, rec.sha256sum
@@ -737,7 +667,7 @@ class FileStore:
with self._lock:
rec = self._files.get(file_id)
if rec is None or (rec.expires_at is not None and rec.expires_at <= int(time.time())):
- _raise_api_error(
+ raise_api_error(
404,
error_type="invalid_request_error",
code="file_not_found",
@@ -761,14 +691,14 @@ class FileStore:
with self._lock:
rec = self.get_file(file_id)
if rec.purpose != "parse_output":
- _raise_api_error(
+ raise_api_error(
403,
error_type="permission_error",
code="feature_requires_api_key",
message="Source files cannot be downloaded",
)
if rec.sha256sum is None:
- _raise_api_error(
+ raise_api_error(
500,
error_type="api_error",
code="internal_error",
@@ -779,7 +709,7 @@ class FileStore:
def delete_file(self, file_id: str) -> None:
with self._lock:
if file_id not in self._files:
- _raise_api_error(
+ raise_api_error(
404,
error_type="invalid_request_error",
code="file_not_found",
@@ -804,8 +734,9 @@ def _get_store(request: Request) -> FileStore:
return request.app.state.file_store
-def _get_job_store(request: Request) -> JobStore:
- return request.app.state.job_store
+def _get_execution_store(request: Request) -> ExecutionStore:
+ """读取当前应用持有的单文件执行存储。"""
+ return request.app.state.execution_store
def _resolve_access_level(request: Request) -> AccessLevel:
@@ -829,115 +760,76 @@ def _validate_source_policy(source: _ExecutionSource, *, allow_http_source: bool
raise ValueError("url source must use https unless --allow-http-source is enabled")
-def _validate_job_source_policy(
- req: _ParseRequest,
- *,
- allow_http_source: bool,
-) -> None:
- for index, entry in enumerate(req.files):
- try:
- _validate_source_policy(
- entry.source,
- allow_http_source=allow_http_source,
- )
- except ValueError as exc:
- message = str(exc)
- code = "unsupported_source" if "source" in message else "invalid_request"
- _raise_api_error(
- 400,
- error_type="invalid_request_error",
- code=code,
- message=message,
- param=f"files.{index}.source",
- )
-
-
-@dataclass(frozen=True)
-class _JobInputSpec:
- """仅在服务内部传递 tasks 的逻辑文件名及可选完整性声明。"""
-
- filename: str
- bytes: int | None = None
- sha256sum: str | None = None
- result_file_id: str | None = None
-
-
-@dataclass(frozen=True)
-class _JobInputMetadata:
- """记录完整下载且校验通过的实际输入信息,不扩展公开 Jobs 协议。"""
-
- bytes: int
+def _validate_execution_source_policy(req: _ExecutionRequest, *, allow_http_source: bool) -> None:
+ """校验单个来源并保留 Tasks 对外错误参数的位置约定。"""
+ try:
+ _validate_source_policy(req.source, allow_http_source=allow_http_source)
+ except ValueError as exc:
+ message = str(exc)
+ code = "unsupported_source" if "source" in message else "invalid_request"
+ raise_api_error(400, error_type="invalid_request_error", code=code, message=message, param="files.0.source")
@dataclass
-class _JobRecord:
+class _ExecutionRecord:
+ """保存一次单文件执行的状态和资源租约,批次进度只由父 Task 聚合。"""
+
id: str
- status: JobStatus = "pending"
+ request: _ExecutionRequest
+ status: TaskFileState = "pending"
created_at: str = ""
started_at: str | None = None
finished_at: str | None = None
- tier: Tier = _DEFAULT_API_SERVER_TIER
- progress: TaskProgress | None = None
- files: list[_ExecutionFile] = None # type: ignore[assignment]
+ page_range: str = ""
+ parse: FileParseInfo | None = None
+ error: ErrorDetail | None = None
input_blobs: dict[str, str] = field(default_factory=dict)
- input_specs: dict[int, _JobInputSpec] = field(default_factory=dict)
- input_metadata: dict[int, _JobInputMetadata] = field(default_factory=dict)
- prepared_sources: dict[int, _PreparedInput] = field(default_factory=dict)
+ input_metadata: TaskInputMetadata | None = None
+ prepared_input: _PreparedInput | None = None
file_store: FileStore | None = None
usage_counted: bool = False
- task_result_file_id: str | None = None
- task_output_formats: list[TaskOutputFormat] = field(default_factory=lambda: ["default"])
+ result_file_id: str | None = None
retention_held: bool = False
retention_epoch: float | None = None
- def __post_init__(self) -> None:
- if self.files is None:
- self.files = []
- if self.progress is None:
- self.progress = TaskProgress()
+def _discard_unfinished_output(rec: _ExecutionRecord) -> None:
+ """失败或取消时撤销预留结果,阻止尚在排空的后台线程发布迟到产物。"""
+ if rec.file_store is not None and rec.status != "completed":
+ rec.file_store.discard_file_id(rec.request.file_id)
-def _discard_unfinished_outputs(rec: _JobRecord) -> None:
- """执行取消或失败时立即撤销未完成产物,防止后台线程晚于终态发布文件。"""
- if rec.file_store is None:
- return
- for index, spec in rec.input_specs.items():
- if spec.result_file_id is not None and rec.files[index].status != "completed":
- rec.file_store.discard_file_id(spec.result_file_id)
+class ExecutionStore:
+ """管理本地单文件执行、并发名额及结果保留期。"""
-class JobStore:
def __init__(self, concurrency: int = 1, retention_seconds: int = 86400) -> None:
+ """初始化本应用执行状态、准备限流与解析信号量。"""
self.retention_seconds = retention_seconds
self._jobs_created = 0
self._files_processed = 0
self._pages_processed = 0
- self._jobs: dict[str, _JobRecord] = {}
+ self._executions: dict[str, _ExecutionRecord] = {}
self._concurrency = max(1, concurrency)
self._semaphore = asyncio.Semaphore(self._concurrency)
self._preparation_semaphore = asyncio.Semaphore(self._concurrency)
- self._started_at = JobStore._now()
+ self._started_at = execution_timestamp()
self._tasks: dict[str, asyncio.Task[None]] = {}
self._completion_events: dict[str, asyncio.Event] = {}
self._closing = False
self.runtime_owner = RuntimeOwner()
def fail_engine(self, message: str) -> None:
- """锁定失败状态并取消未结束任务,用户取消的任务保持原终态。"""
+ """标记本应用未结束执行的引擎错误,并沿原边界取消和排空任务。"""
if self.runtime_owner.engine_error is None:
self.runtime_owner.engine_error = message
current = asyncio.current_task()
- for job_id, task in tuple(self._tasks.items()):
- rec = self._jobs[job_id]
+ for execution_id, task in tuple(self._tasks.items()):
+ rec = self._executions[execution_id]
if task.done() or rec.status == "canceled":
continue
rec.status = "failed"
- for file in rec.files:
- if file.status != "completed":
- file.status = "failed"
- file.error = ErrorDetail(type="engine_error", code="engine_dead", message=message)
- rec.progress.failed = rec.progress.total - rec.progress.completed
- _discard_unfinished_outputs(rec)
+ rec.error = ErrorDetail(type="engine_error", code="engine_dead", message=message)
+ _discard_unfinished_output(rec)
if task is not current and not task.cancelling():
task.cancel()
@@ -946,11 +838,11 @@ class JobStore:
message = self.runtime_owner.engine_error
if message is not None:
self.fail_engine(message)
- _raise_api_error(503, error_type="engine_error", code="engine_dead", message=message)
+ raise_api_error(503, error_type="engine_error", code="engine_dead", message=message)
def start_task(
self,
- rec: _JobRecord,
+ rec: _ExecutionRecord,
operation: Callable[[], Awaitable[None]],
*,
prepare: Callable[[], Awaitable[None]] | None = None,
@@ -958,7 +850,7 @@ class JobStore:
"""保存真实后台任务及所属应用租约,排队取消也不会开始解析。"""
self.require_ready()
if self._closing:
- _raise_api_error(503, error_type="engine_error", code="server_shutting_down", message="Server is shutting down")
+ raise_api_error(503, error_type="engine_error", code="server_shutting_down", message="Server is shutting down")
self._completion_events[rec.id] = asyncio.Event()
async def run_owned() -> None:
@@ -985,7 +877,7 @@ class JobStore:
rec.status = "canceled"
raise
finally:
- rec.prepared_sources.clear()
+ rec.prepared_input = None
if acquired:
self._semaphore.release()
runtime_owner.reset(token)
@@ -993,7 +885,7 @@ class JobStore:
def finished(task: asyncio.Task[None]) -> None:
"""回收句柄并读取异常,覆盖任务首次执行前就被取消的路径。"""
self._tasks.pop(rec.id, None)
- rec.finished_at = self._now()
+ rec.finished_at = execution_timestamp()
if rec.file_store is not None:
if not rec.retention_held:
rec.file_store.finalize_outputs(rec.id, int(rec.retention_epoch or time.time()))
@@ -1004,7 +896,7 @@ class JobStore:
logger.error("Parse job task failed: %s", rec.id, exc_info=(type(error), error, error.__traceback__))
if rec.status != "canceled":
rec.status = "failed"
- _discard_unfinished_outputs(rec)
+ _discard_unfinished_output(rec)
self._record_usage(rec)
self._completion_events[rec.id].set()
@@ -1012,7 +904,7 @@ class JobStore:
self._tasks[rec.id] = task
task.add_done_callback(finished)
- async def wait_for_terminal(self, job_id: str, timeout: float) -> _JobRecord:
+ async def wait_for_terminal(self, job_id: str, timeout: float) -> _ExecutionRecord:
"""只等待完成通知,超时或等待方取消不会传播到后台解析任务。"""
rec = self.get(job_id)
event = self._completion_events.get(job_id)
@@ -1029,7 +921,7 @@ class JobStore:
tasks = list(self._tasks.items())
for job_id, task in tasks:
if not task.done():
- rec = self._jobs[job_id]
+ rec = self._executions[job_id]
if rec.status not in {"canceled", "failed"}:
rec.status = "canceled"
if not task.cancelling():
@@ -1037,57 +929,27 @@ class JobStore:
try:
await drain_future(asyncio.gather(*(task for _, task in tasks), return_exceptions=True))
finally:
- from ..model.vlm.runtime import ModelSingleton
-
- await run_sync(ModelSingleton().release_owner, self.runtime_owner)
-
- @staticmethod
- def _new_job_id() -> str:
- return "job_" + secrets.token_hex(12)
+ from ..model.runtime.ownership import release_runtime_owner
- @staticmethod
- def _now() -> str:
- return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+ await run_sync(release_runtime_owner, self.runtime_owner)
- def create(
- self,
- req: _ParseRequest,
- file_store: FileStore,
- *,
- job_id: str | None = None,
- input_specs: dict[int, _JobInputSpec] | None = None,
- ) -> _JobRecord:
- """创建唯一执行记录,允许 tasks 使用上传准备阶段预先分配的 ID。"""
+ def create(self, req: _ExecutionRequest, file_store: FileStore, *, execution_id: str | None = None) -> _ExecutionRecord:
+ """登记一个已确定档位的文件执行,持有源字节引用而不创建内部批次。"""
if req.tier is None:
- raise ValueError("_ParseRequest.tier must be resolved before job creation")
- job_id = job_id or self._new_job_id()
- if job_id in self._jobs:
- _raise_api_error(409, error_type="invalid_request_error", code="job_already_exists", message="Job already exists")
- now = self._now()
- rec = _JobRecord(
- id=job_id,
- status="waiting-file" if any(isinstance(entry.source, _UrlSource) for entry in req.files) else "pending",
- created_at=now,
- tier=req.tier,
- task_output_formats=req.output_formats,
- input_specs=dict(input_specs or {}),
+ raise ValueError("_ExecutionRequest.tier must be resolved before execution creation")
+ execution_id = execution_id or new_execution_id()
+ if execution_id in self._executions:
+ raise_api_error(409, error_type="invalid_request_error", code="job_already_exists", message="Job already exists")
+ rec = _ExecutionRecord(
+ id=execution_id,
+ request=req,
+ status="waiting-file" if isinstance(req.source, _UrlSource) else "pending",
+ created_at=execution_timestamp(),
+ page_range=req.page_range or "",
+ file_store=file_store,
+ input_blobs=file_store.acquire_inputs([req.source.file_id] if isinstance(req.source, _StoredSource) else []),
)
- for index, entry in enumerate(req.files):
- name = _source_name(entry.source, file_store, input_spec=rec.input_specs.get(index))
- rec.files.append(
- _ExecutionFile(
- file_id=entry.source.file_id if isinstance(entry.source, _StoredSource) else None,
- name=name,
- page_range=entry.page_range or "",
- status="waiting-file" if isinstance(entry.source, _UrlSource) else "pending",
- )
- )
- rec.progress.total = len(rec.files)
- rec.input_blobs = file_store.acquire_inputs(
- [entry.source.file_id for entry in req.files if isinstance(entry.source, _StoredSource)]
- )
- rec.file_store = file_store
- self._jobs[job_id] = rec
+ self._executions[execution_id] = rec
self._jobs_created += 1
return rec
@@ -1096,18 +958,18 @@ class JobStore:
if not self.retention_seconds:
return
now = time.time()
- for job_id, rec in tuple(self._jobs.items()):
+ for job_id, rec in tuple(self._executions.items()):
if job_id in self._tasks or rec.finished_at is None or rec.retention_held:
continue
finished = rec.retention_epoch or datetime.fromisoformat(rec.finished_at.replace("Z", "+00:00")).timestamp()
if finished + self.retention_seconds <= now:
self._record_usage(rec)
- self._jobs.pop(job_id)
+ self._executions.pop(job_id)
self._completion_events.pop(job_id, None)
def release_task_result(self, job_id: str, finished: float) -> None:
"""父 Task 收尾后才开始子作业与产物的保留期,不更改真实完成时间。"""
- rec = self._jobs.get(job_id)
+ rec = self._executions.get(job_id)
if rec is None:
return
rec.retention_held = False
@@ -1115,11 +977,12 @@ class JobStore:
if rec.file_store is not None:
rec.file_store.finalize_outputs(job_id, int(finished))
- def get(self, job_id: str) -> _JobRecord:
+ def get(self, job_id: str) -> _ExecutionRecord:
+ """读取仍在保留期内的执行,未知标识沿用已有错误合同。"""
self.collect_expired()
- rec = self._jobs.get(job_id)
+ rec = self._executions.get(job_id)
if rec is None:
- _raise_api_error(
+ raise_api_error(
404,
error_type="invalid_request_error",
code="job_not_found",
@@ -1127,56 +990,57 @@ class JobStore:
)
return rec
- def cancel(self, job_id: str) -> _JobRecord:
+ def cancel(self, job_id: str) -> _ExecutionRecord:
+ """取消尚未完成的执行并撤销结果预留,不强制中断同步工作。"""
rec = self.get(job_id)
- if rec.status in ("completed", "partial", "failed", "canceled"):
- _raise_api_error(
+ if rec.status in ("completed", "failed", "canceled"):
+ raise_api_error(
409,
error_type="invalid_request_error",
code="job_already_terminal",
message=f"Job is {rec.status}",
)
rec.status = "canceled"
- _discard_unfinished_outputs(rec)
+ _discard_unfinished_output(rec)
task = self._tasks.get(job_id)
if task is not None and not task.done():
task.cancel()
return rec
- def snapshot(self, rec: _JobRecord) -> dict[str, Any]:
- """返回私有执行快照,不包含公开 Jobs 链接或产物协议。"""
- return {
- "job_id": rec.id,
- "status": rec.status,
- "created_at": rec.created_at,
- "started_at": rec.started_at,
- "finished_at": rec.finished_at,
- "tier": rec.tier,
- "progress": rec.progress.model_dump(),
- "files": [item.model_dump() for item in rec.files],
- }
+ def snapshot(self, rec: _ExecutionRecord) -> TaskExecution:
+ """返回独立的有类型快照,公开文件身份始终取预留结果 ID。"""
+ return TaskExecution(
+ execution_id=rec.id,
+ file_id=rec.request.file_id,
+ file_name=rec.request.file_name,
+ status=rec.status,
+ created_at=rec.created_at,
+ started_at=rec.started_at,
+ finished_at=rec.finished_at,
+ page_range=rec.page_range,
+ parse=rec.parse.model_copy(deep=True) if rec.parse is not None else None,
+ result_file_id=rec.result_file_id,
+ error=rec.error.model_copy(deep=True) if rec.error is not None else None,
+ input_metadata=rec.input_metadata,
+ )
- def _record_usage(self, rec: _JobRecord) -> None:
- """完成计数只累计一次,元数据过期后服务总用量不倒退。"""
+ def _record_usage(self, rec: _ExecutionRecord) -> None:
+ """每个执行只计入一次成功文件和实际页面,用量不随记录到期倒退。"""
if not rec.usage_counted:
- self._files_processed += sum(file.status == "completed" for file in rec.files)
- self._pages_processed += sum(
- _count_pages_in_range(file.page_range) for file in rec.files if file.status == "completed"
- )
+ if rec.status == "completed":
+ self._files_processed += 1
+ self._pages_processed += _count_pages_in_range(rec.page_range)
rec.usage_counted = True
def usage(self, access_level: AccessLevel) -> UsageResponse:
- pending = [rec for rec in self._jobs.values() if not rec.usage_counted]
- files_processed = self._files_processed + sum(1 for rec in pending for file in rec.files if file.status == "completed")
- processed_page_count = self._pages_processed + sum(
- _count_pages_in_range(file.page_range) for rec in pending for file in rec.files if file.status == "completed"
- )
+ """保留公开 jobs_created 与并发上限字段,按真实单文件执行累计用量。"""
+ pending = [rec for rec in self._executions.values() if not rec.usage_counted and rec.status == "completed"]
return UsageResponse(
access_level=access_level,
billing_period=UsageBillingPeriod(start=self._started_at),
current=UsageCurrent(
- pages_processed=processed_page_count,
- files_processed=files_processed,
+ pages_processed=self._pages_processed + sum(_count_pages_in_range(rec.page_range) for rec in pending),
+ files_processed=self._files_processed + len(pending),
jobs_created=self._jobs_created,
),
limits=UsageLimits(
@@ -1186,48 +1050,23 @@ class JobStore:
)
def install(self, app_state: State) -> None:
- app_state.job_store = self
+ """把执行存储显式安装到当前应用实例。"""
+ app_state.execution_store = self
# ── source helpers ───────────────────────────────────────────────────
-def _source_name(
- source: _ExecutionSource, file_store: FileStore | None = None, *, input_spec: _JobInputSpec | None = None
-) -> str:
- """使用 Tasks 的逻辑文件名识别类型,源存储不改变业务文件名。"""
- if input_spec is not None:
- return input_spec.filename
- if isinstance(source, _StoredSource) and file_store is not None:
- try:
- return file_store.get_file(source.file_id).filename
- except Exception:
- pass
- return source.file_id
- if isinstance(source, _UrlSource):
- return source.url.rsplit("/", 1)[-1].split("?")[0] or "unknown"
- return "unknown"
-
-
-def _job_can_default_to_flash(
- req: _ParseRequest, file_store: FileStore, input_specs: dict[int, _JobInputSpec] | None = None
-) -> bool:
- """判断整批输入是否已知为 Flash-only,或需下载后识别类型的无扩展名 URL。"""
- return all(
- is_flash_only_parse_extension(name) or (isinstance(entry.source, _UrlSource) and not pathlib.Path(name).suffix)
- for index, entry in enumerate(req.files)
- for name in [_source_name(entry.source, file_store, input_spec=(input_specs or {}).get(index))]
+def _execution_can_default_to_flash(req: _ExecutionRequest) -> bool:
+ """原生格式或等待下载后识别类型的无扩展名 URL 可使用 Flash 回退。"""
+ return is_flash_only_parse_extension(req.file_name) or (
+ isinstance(req.source, _UrlSource) and not pathlib.Path(req.file_name).suffix
)
-def _job_has_any_flash_only_inputs(
- req: _ParseRequest, file_store: FileStore, input_specs: dict[int, _JobInputSpec] | None = None
-) -> bool:
- """使用与执行一致的逻辑文件名检查批次中是否包含 Flash-only 格式。"""
- return any(
- is_flash_only_parse_extension(_source_name(entry.source, file_store, input_spec=(input_specs or {}).get(index)))
- for index, entry in enumerate(req.files)
- )
+def _execution_requires_flash(req: _ExecutionRequest) -> bool:
+ """按清单声明的逻辑文件名判断单文件是否必须使用原生解析。"""
+ return is_flash_only_parse_extension(req.file_name)
def _page_range_from_result_pages(pages: list[PageInfo]) -> str:
@@ -1258,64 +1097,54 @@ class _PreparedInput:
duration_seconds: float
-def _validate_url_input(rec: _JobRecord, index: int, extracted: _ExtractedSource) -> None:
- """下载完成后验证大小和 hash,只有有效输入才能进入解析队列。"""
+def _validate_url_input(rec: _ExecutionRecord, extracted: _ExtractedSource) -> None:
+ """下载完成后核对声明大小与哈希,实际输入元数据直接归属于本次执行。"""
data = extracted.data
if not data:
raise MineruError("invalid_request", "Empty files are not supported")
- spec = rec.input_specs.get(index)
- if spec is not None:
- if spec.bytes is not None and len(data) != spec.bytes:
- raise MineruError("bytes_mismatch", "Declared size does not match downloaded bytes")
- if spec.sha256sum is not None and hashlib.sha256(data).hexdigest() != spec.sha256sum:
- raise MineruError("file_hash_mismatch", "SHA-256 mismatch")
- rec.input_metadata[index] = _JobInputMetadata(bytes=len(data))
-
-
-async def _prepare_job_inputs(
- rec: _JobRecord,
- req: _ParseRequest,
+ if rec.request.bytes is not None and len(data) != rec.request.bytes:
+ raise MineruError("bytes_mismatch", "Declared size does not match downloaded bytes")
+ if rec.request.sha256sum is not None and hashlib.sha256(data).hexdigest() != rec.request.sha256sum:
+ raise MineruError("file_hash_mismatch", "SHA-256 mismatch")
+ rec.input_metadata = TaskInputMetadata(bytes=len(data))
+
+
+async def _prepare_execution_input(
+ rec: _ExecutionRecord,
file_store: FileStore,
*,
url_timeout: int,
allow_http_source: bool,
max_url_bytes: int,
) -> None:
- """在独立受控阶段下载 URL,保留 HTML 来源信息并逐文件收敛准备失败。"""
- if rec.status in {"canceled", "failed"}:
+ """在准备限流内下载一个 URL,保留来源及耗时,完成后再等待解析名额。"""
+ if rec.status in {"canceled", "failed"} or not isinstance(rec.request.source, _UrlSource):
return
rec.status = "waiting-file"
- rec.started_at = rec.started_at or JobStore._now()
- for index, entry in enumerate(req.files):
+ rec.started_at = rec.started_at or execution_timestamp()
+ started = time.monotonic()
+ try:
+ extracted = await _extract_bytes(
+ rec.request.source,
+ file_store,
+ url_timeout=url_timeout,
+ allow_http_source=allow_http_source,
+ max_url_bytes=max_url_bytes,
+ )
+ _validate_url_input(rec, extracted)
+ rec.prepared_input = _PreparedInput(extracted, time.monotonic() - started)
+ if rec.status not in {"canceled", "failed"}:
+ rec.status = "pending"
+ except Exception as exc:
+ logger.exception("Parse-server input preparation failed: job_id=%s file=%r", rec.id, rec.request.file_name)
if rec.status in {"canceled", "failed"}:
return
- if not isinstance(entry.source, _UrlSource):
- continue
- file = rec.files[index]
- file.status = "waiting-file"
- started = time.monotonic()
- try:
- extracted = await _extract_bytes(
- entry.source,
- file_store,
- url_timeout=url_timeout,
- allow_http_source=allow_http_source,
- max_url_bytes=max_url_bytes,
- )
- _validate_url_input(rec, index, extracted)
- rec.prepared_sources[index] = _PreparedInput(extracted, time.monotonic() - started)
- file.status = "pending"
- except Exception as exc:
- logger.exception("Parse-server input preparation failed: job_id=%s file=%r", rec.id, file.name)
- file.status = "failed"
- file.error = (
- ErrorDetail(type=exc.type, code=exc.code, message=str(exc), param=exc.param)
- if isinstance(exc, MineruError)
- else ErrorDetail(type="engine_error", code="parse_failed", message=str(exc))
- )
- rec.progress.failed += 1
- if rec.status not in {"canceled", "failed"}:
- rec.status = "failed" if rec.progress.failed == rec.progress.total else "pending"
+ rec.status = "failed"
+ rec.error = (
+ ErrorDetail(type=exc.type, code=exc.code, message=str(exc), param=exc.param)
+ if isinstance(exc, MineruError)
+ else ErrorDetail(type="engine_error", code="parse_failed", message=str(exc))
+ )
async def _extract_bytes(
@@ -1368,9 +1197,8 @@ def _suffix_type(filename: str) -> str:
return "" if file_type == "unknown" else file_type
-async def _run_job(
- rec: _JobRecord,
- req: _ParseRequest,
+async def _run_execution(
+ rec: _ExecutionRecord,
file_store: FileStore,
*,
image_analysis: bool,
@@ -1379,142 +1207,117 @@ async def _run_job(
max_url_bytes: int = _MAX_URL_BYTES_DEFAULT,
flash_enabled: bool = True,
vlm_config: VlmConfig | None = None,
- job_store: JobStore | None = None,
+ execution_store: ExecutionStore | None = None,
) -> None:
- """执行解析任务,使用所属服务的 VLM 配置并记录每个文件的成功或失败。"""
- if rec.status == "canceled":
+ """执行唯一输入并发布同一公开文件身份的产物,取消时排空已有同步阶段。"""
+ if rec.status in {"canceled", "failed"}:
return
rec.status = "running"
- rec.started_at = rec.started_at or JobStore._now()
+ rec.started_at = rec.started_at or execution_timestamp()
+ req = rec.request
+ prepared = rec.prepared_input
+ rec.prepared_input = None
with tempfile.TemporaryDirectory(prefix="mineru_job_") as tmpdir:
- for i, entry in enumerate(req.files):
- if rec.status in {"canceled", "failed"}:
- break
- fr = rec.files[i]
- if fr.status in {"failed", "canceled"}:
- continue
- prepared = rec.prepared_sources.pop(i, None)
- try:
- file_started = time.monotonic() - (prepared.duration_seconds if prepared is not None else 0)
- fr.status = "running"
- extracted = (
- prepared.source
- if prepared is not None
- else await _extract_bytes(
- entry.source,
- file_store,
- url_timeout=url_timeout,
- allow_http_source=allow_http_source,
- max_url_bytes=max_url_bytes,
- input_blobs=rec.input_blobs,
- )
+ try:
+ file_started = time.monotonic() - (prepared.duration_seconds if prepared is not None else 0)
+ extracted = (
+ prepared.source
+ if prepared is not None
+ else await _extract_bytes(
+ req.source,
+ file_store,
+ url_timeout=url_timeout,
+ allow_http_source=allow_http_source,
+ max_url_bytes=max_url_bytes,
+ input_blobs=rec.input_blobs,
)
- data = extracted.data
- if prepared is None and isinstance(entry.source, _UrlSource):
- _validate_url_input(rec, i, extracted)
- suffix = pathlib.Path(fr.name).suffix
- stype = _suffix_type(fr.name)
- if not stype and not suffix and isinstance(entry.source, _UrlSource) and extracted.media_type == "text/html":
- stype = "html"
- suffix = ".html"
- if not stype:
- raise ValueError(f"Unsupported file type: {fr.name}")
- if not flash_enabled and is_flash_only_parse_extension(stype):
- raise ValueError("Flash parsing is disabled in this server, but this input requires the Flash backend")
-
- # write to temp file for parsers that require a path
- suffix = suffix or ".pdf"
- tmp_path = pathlib.Path(tmpdir) / f"input_{i}{suffix}"
- tmp_path.write_bytes(data)
-
- page_range = entry.page_range or ""
- effective_tier = batch_effective_parse_tier(rec.tier, suffix)
-
- source_context = None
- if stype == "html":
- from docvortex.document.contracts import HtmlSourceContext
-
- source_context = HtmlSourceContext(
- source_uri=extracted.source_uri,
- local_resource_root=extracted.local_resource_root,
- transport_encoding=extracted.transport_encoding,
- )
- result = await parse_async(
- str(tmp_path),
- tier=effective_tier,
- ocr_mode=req.ocr_mode,
- image_analysis=image_analysis,
- page_range=page_range,
- source_context=source_context,
- vlm_config=vlm_config,
+ )
+ data = extracted.data
+ if prepared is None and isinstance(req.source, _UrlSource):
+ _validate_url_input(rec, extracted)
+ suffix = pathlib.Path(req.file_name).suffix
+ stype = _suffix_type(req.file_name)
+ if not stype and not suffix and isinstance(req.source, _UrlSource) and extracted.media_type == "text/html":
+ stype = "html"
+ suffix = ".html"
+ if not stype:
+ raise ValueError(f"Unsupported file type: {req.file_name}")
+ if not flash_enabled and is_flash_only_parse_extension(stype):
+ raise ValueError("Flash parsing is disabled in this server, but this input requires the Flash backend")
+
+ suffix = suffix or ".pdf"
+ tmp_path = pathlib.Path(tmpdir) / f"input_0{suffix}"
+ tmp_path.write_bytes(data)
+ effective_tier = batch_effective_parse_tier(req.tier, suffix)
+ source_context = None
+ if stype == "html":
+ from docvortex.document.contracts import HtmlSourceContext
+
+ source_context = HtmlSourceContext(
+ source_uri=extracted.source_uri,
+ local_resource_root=extracted.local_resource_root,
+ transport_encoding=extracted.transport_encoding,
)
+ result = await parse_async(
+ str(tmp_path),
+ tier=effective_tier,
+ ocr_mode=req.ocr_mode,
+ image_analysis=image_analysis,
+ page_range=req.page_range or "",
+ source_context=source_context,
+ vlm_config=vlm_config,
+ )
+ if rec.status in {"canceled", "failed"}:
+ return
+ actual_metadata = MinerUMetadata.model_validate(result.middle_json.extensions.get("mineru"))
+ from .task_outputs import build_task_output
+ try:
+ artifact = await run_sync(build_task_output, result, req.output_formats, pathlib.Path(tmpdir) / "outputs_0")
if rec.status in {"canceled", "failed"}:
- break
- actual_metadata = MinerUMetadata.model_validate(result.middle_json.extensions.get("mineru"))
- from .task_outputs import build_task_output
-
- try:
- artifact = await run_sync(
- build_task_output,
- result,
- rec.task_output_formats,
- pathlib.Path(tmpdir) / f"outputs_{i}",
- )
- if rec.status in {"canceled", "failed"}:
- break
- rec.task_result_file_id = await run_sync(
- file_store.create_file_for_output_path,
- pathlib.Path(fr.name).stem + artifact.suffix,
- artifact,
- job_id=rec.id,
- file_id=rec.input_specs[i].result_file_id if i in rec.input_specs else None,
- )
- if rec.status in {"canceled", "failed"}:
- file_store.discard_file_id(rec.task_result_file_id)
- break
- except Exception as exc:
- raise MineruError("result_packaging_failed", str(exc)) from exc
-
- fr.status = "completed"
- fr.page_range = _page_range_from_result_pages(result.pages)
-
- fr.parse = FileParseInfo(
- tier=actual_metadata.tier,
- duration_ms=int((time.monotonic() - file_started) * 1000),
- parser_version=__version__,
+ return
+ rec.result_file_id = await run_sync(
+ file_store.create_file_for_output_path,
+ pathlib.Path(req.file_name).stem + artifact.suffix,
+ artifact,
+ job_id=rec.id,
+ file_id=req.file_id,
)
- rec.progress.completed += 1
-
- except Exception as exc:
- if job_store is not None and is_fatal_engine_error(exc):
- job_store.fail_engine(str(exc))
- raise
if rec.status in {"canceled", "failed"}:
- break
- logger.exception(
- "Parse-server job file failed: job_id=%s file=%r tier=%s page_range=%r",
- rec.id,
- fr.name,
- rec.tier,
- fr.page_range,
- )
- fr.status = "failed"
- if isinstance(exc, MineruError):
- fr.error = ErrorDetail(type=exc.type, code=exc.code, message=str(exc), param=exc.param)
- else:
- fr.error = ErrorDetail(type="engine_error", code="parse_failed", message=str(exc))
- rec.progress.failed += 1
+ file_store.discard_file_id(rec.result_file_id)
+ return
+ except Exception as exc:
+ raise MineruError("result_packaging_failed", str(exc)) from exc
- if rec.status not in {"canceled", "failed"}:
- if rec.progress.failed == rec.progress.total:
- rec.status = "failed"
- elif rec.progress.failed > 0:
- rec.status = "partial"
- else:
+ rec.page_range = _page_range_from_result_pages(result.pages)
+ rec.parse = FileParseInfo(
+ tier=actual_metadata.tier,
+ duration_ms=int((time.monotonic() - file_started) * 1000),
+ parser_version=__version__,
+ )
rec.status = "completed"
- rec.finished_at = JobStore._now()
+ except Exception as exc:
+ if execution_store is not None and is_fatal_engine_error(exc):
+ execution_store.fail_engine(str(exc))
+ raise
+ if rec.status in {"canceled", "failed"}:
+ return
+ logger.exception(
+ "Parse-server job file failed: job_id=%s file=%r tier=%s page_range=%r",
+ rec.id,
+ req.file_name,
+ req.tier,
+ rec.page_range,
+ )
+ rec.status = "failed"
+ rec.error = (
+ ErrorDetail(type=exc.type, code=exc.code, message=str(exc), param=exc.param)
+ if isinstance(exc, MineruError)
+ else ErrorDetail(type="engine_error", code="parse_failed", message=str(exc))
+ )
+ finally:
+ rec.finished_at = execution_timestamp()
# ═══════════════════════════════════════════════════════════════════════
@@ -1564,11 +1367,11 @@ _ERR_503: _ErrorResponseMap = {503: {"model": ErrorResponse}}
def _require_model_preload_ready(request: Request) -> None:
- request.app.state.job_store.require_ready()
+ request.app.state.execution_store.require_ready()
preload_error: _ModelPreloadError | None = request.app.state.model_preload_error
if preload_error is None:
return
- _raise_api_error(
+ raise_api_error(
503,
error_type="engine_error",
code=preload_error.code,
@@ -1627,7 +1430,7 @@ async def get_model(
_require_model_preload_ready(request)
model_ids: list[str] = request.app.state.model_ids
if model not in model_ids:
- _raise_api_error(
+ raise_api_error(
404,
error_type="invalid_request_error",
code="model_not_found",
@@ -1727,26 +1530,23 @@ async def get_file_content(
async def submit_execution(
- body: _ParseRequest,
+ body: _ExecutionRequest,
request: Request,
file_store: FileStore,
- job_store: JobStore,
+ execution_store: ExecutionStore,
*,
- job_id: str | None = None,
- input_specs: dict[int, _JobInputSpec] | None = None,
-) -> _JobRecord:
- """校验并提交 Tasks 的私有执行单元,复用唯一并发和生命周期管理。"""
- for index, entry in enumerate(body.files):
- try:
- entry.page_range = normalize_page_range_input(entry.page_range) or None
- except MineruError as exc:
- _raise_api_error(400, error_type=exc.type, code=exc.code, message=str(exc), param=f"files.{index}.page_range")
+ execution_id: str | None = None,
+) -> _ExecutionRecord:
+ """校验单文件执行请求并接入唯一并发队列,不再创建内部批次或按序号映射。"""
+ try:
+ body.page_range = normalize_page_range_input(body.page_range) or None
+ except MineruError as exc:
+ raise_api_error(400, error_type=exc.type, code=exc.code, message=str(exc), param="files.0.page_range")
_require_model_preload_ready(request)
- # 按请求 tier 选择启动时预先解析好的 runtime;全轻量输入可按 ADR-0024 直接归一到 flash 执行。
default_tier: Tier | None = request.app.state.default_tier
- can_default_to_flash = _job_can_default_to_flash(body, file_store, input_specs)
- if not request.app.state.flash_enabled and _job_has_any_flash_only_inputs(body, file_store, input_specs):
- _raise_api_error(
+ can_default_to_flash = _execution_can_default_to_flash(body)
+ if not request.app.state.flash_enabled and _execution_requires_flash(body):
+ raise_api_error(
400,
error_type="invalid_request_error",
code="invalid_request",
@@ -1757,7 +1557,7 @@ async def submit_execution(
if can_default_to_flash:
body.tier = "flash"
else:
- _raise_api_error(
+ raise_api_error(
503,
error_type="engine_error",
code="quality_tier_unavailable",
@@ -1773,21 +1573,16 @@ async def submit_execution(
if runtime is None and can_default_to_flash:
runtime = runtime_options.get("flash") or runtime_options_for_tier("flash")
if runtime is None:
- _raise_api_error(
+ raise_api_error(
400,
error_type="invalid_request_error",
code="invalid_request",
message=f"Tier '{body.tier}' not available in this server",
)
-
- _validate_job_source_policy(
- body,
- allow_http_source=request.app.state.allow_http_source,
- )
-
- if job_store._closing:
- _raise_api_error(503, error_type="engine_error", code="server_shutting_down", message="Server is shutting down")
- rec = job_store.create(body, file_store, job_id=job_id, input_specs=input_specs)
+ _validate_execution_source_policy(body, allow_http_source=request.app.state.allow_http_source)
+ if execution_store._closing:
+ raise_api_error(503, error_type="engine_error", code="server_shutting_down", message="Server is shutting down")
+ rec = execution_store.create(body, file_store, execution_id=execution_id)
rec.retention_held = True
url_timeout_val: int = request.app.state.url_timeout
allow_http_source_val: bool = request.app.state.allow_http_source
@@ -1797,10 +1592,9 @@ async def submit_execution(
vlm_config_val: VlmConfig = request.app.state.vlm_config
async def _bg_run() -> None:
- """运行任务内容,排队和租约由 JobStore 统一管理。"""
- await _run_job(
+ """运行单文件内容,排队、资源租约和取消排空仍由执行存储管理。"""
+ await _run_execution(
rec,
- body,
file_store,
image_analysis=image_analysis_val,
url_timeout=url_timeout_val,
@@ -1808,14 +1602,13 @@ async def submit_execution(
max_url_bytes=max_url_bytes_val,
flash_enabled=flash_enabled_val,
vlm_config=vlm_config_val,
- job_store=job_store,
+ execution_store=execution_store,
)
async def _bg_prepare() -> None:
- """复用本次来源配置准备 URL 输入,只有准备成功的文件进入解析队列。"""
- await _prepare_job_inputs(
+ """沿用来源配置准备 URL 输入,完整下载并校验后才进入解析队列。"""
+ await _prepare_execution_input(
rec,
- body,
file_store,
url_timeout=url_timeout_val,
allow_http_source=allow_http_source_val,
@@ -1823,13 +1616,11 @@ async def submit_execution(
)
try:
- job_store.start_task(
- rec, _bg_run, prepare=_bg_prepare if any(isinstance(entry.source, _UrlSource) for entry in body.files) else None
- )
+ execution_store.start_task(rec, _bg_run, prepare=_bg_prepare if isinstance(body.source, _UrlSource) else None)
except BaseException:
file_store.release_inputs(rec.input_blobs)
- job_store._jobs.pop(rec.id, None)
- job_store._jobs_created -= 1
+ execution_store._executions.pop(rec.id, None)
+ execution_store._jobs_created -= 1
raise
return rec
@@ -1845,11 +1636,11 @@ async def submit_execution(
)
async def get_usage(
request: Request,
- job_store: JobStore = Depends(_get_job_store),
+ execution_store: ExecutionStore = Depends(_get_execution_store),
access_level: AccessLevel = Depends(_resolve_access_level),
) -> UsageResponse:
"""Query current usage and limits."""
- return job_store.usage(access_level)
+ return execution_store.usage(access_level)
# ═══════════════════════════════════════════════════════════════════════
@@ -1934,7 +1725,7 @@ def _preflight_tier_dependencies(tier: ServerTier, vlm_config: VlmConfig | None
"""远程 VLM 仅预检本地 Hybrid 依赖,避免要求安装本地 VLM serving 引擎。"""
if tier == "flash":
return
- settings = vlm_config if vlm_config is not None else mineru_config.model.vlm
+ settings = vlm_config if vlm_config is not None else get_config().model.vlm
if tier == "standard":
settings.validate_environment()
try:
@@ -2084,7 +1875,7 @@ def create_app(
tier_runtime_options = _runtime_options_for_server_tiers(server_tiers)
default_tier = select_default_quality_tier(tier_runtime_options)
effort = effort_for_tier(tier)
- vlm_config = (vlm_config if vlm_config is not None else mineru_config.model.vlm).model_copy(deep=True)
+ vlm_config = (vlm_config if vlm_config is not None else get_config().model.vlm).model_copy(deep=True)
_preflight_tier_dependencies(tier, vlm_config)
_api_key: str | None = api_key or None
_upload_dir = pathlib.Path(upload_dir) if upload_dir else pathlib.Path(tempfile.mkdtemp(prefix="mineru_"))
@@ -2118,7 +1909,7 @@ def create_app(
application.state.preload_models = preload_models
application.state.vlm_config = vlm_config
application.state.model_preload_error = None
- job_store: JobStore = application.state.job_store
+ execution_store: ExecutionStore = application.state.execution_store
file_store: FileStore = application.state.file_store
file_store.claim_directory()
application.state.task_store.start()
@@ -2128,12 +1919,12 @@ def create_app(
"""每五分钟扫描,先清理终态元数据,再释放无引用文件。"""
while True:
await asyncio.sleep(RETENTION_SCAN_INTERVAL_SECONDS)
- job_store.collect_expired()
+ execution_store.collect_expired()
await run_sync(file_store.collect_expired)
if retention_seconds:
collection_task = asyncio.create_task(collect_loop(), name="mineru-api-retention")
- token = runtime_owner.set(job_store.runtime_owner)
+ token = runtime_owner.set(execution_store.runtime_owner)
try:
file_store.collect_expired()
if _preload_tier is not None:
@@ -2147,7 +1938,7 @@ def create_app(
except Exception as exc:
error_code, error_msg = _classify_model_preload_error(exc)
if error_code == "engine_dead":
- job_store.fail_engine(error_msg)
+ execution_store.fail_engine(error_msg)
application.state.model_preload_error = _ModelPreloadError(
code=error_code,
message=error_msg,
@@ -2167,7 +1958,7 @@ def create_app(
await asyncio.gather(collection_task, return_exceptions=True)
try:
await application.state.task_store.shutdown()
- await job_store.shutdown()
+ await execution_store.shutdown()
finally:
file_store.release_directory()
if not upload_dir and _upload_dir.exists():
@@ -2248,7 +2039,7 @@ def create_app(
application.state.vlm_config = vlm_config
application.state.model_preload_error = None
FileStore(_upload_dir, retention_seconds=retention_seconds).install(application.state)
- JobStore(concurrency=concurrency, retention_seconds=retention_seconds).install(application.state)
+ ExecutionStore(concurrency=concurrency, retention_seconds=retention_seconds).install(application.state)
application.add_middleware(GZipMiddleware, minimum_size=1000)
@application.exception_handler(ApiServerError)
@@ -2310,7 +2101,7 @@ def create_app(
application.middleware("http")(_auth_middleware)
application.include_router(_build_v1_router())
- from .task_api import ApiTaskBackend, TaskStore, build_task_router
+ from .task_backend import ApiTaskBackend
application.state.task_store = TaskStore(retention_seconds)
application.include_router(build_task_router(ApiTaskBackend))
@@ -2480,7 +2271,7 @@ def main(
}
vlm_settings = VlmConfig.model_validate(
{
- **mineru_config.model.vlm.model_dump(),
+ **get_config().model.vlm.model_dump(),
**{key: value for key, value in vlm_overrides.items() if value is not None},
}
)
diff --git a/mineru/parser/mineru_parser.py b/mineru/parser/mineru_parser.py
index 7208cd8d..054ab926 100644
--- a/mineru/parser/mineru_parser.py
+++ b/mineru/parser/mineru_parser.py
@@ -13,7 +13,7 @@ from docvortex.document.contracts import HtmlSourceContext
from docvortex.schema import DocumentProperties
from ..backend.analyze import aio_doc_analyze, doc_analyze
-from ..config import VlmConfig, config
+from ..config import VlmConfig, get_config
from ..errors import InvalidRequestError
from ..filetypes import IMAGE_EXTENSIONS, PAGE_RANGE_PARSE_EXTENSIONS
from ..integrations.docvortex import read_source_properties
@@ -62,7 +62,7 @@ class MinerUParser(DocumentParser):
self.effort: _Effort = effort_for_tier(tier) # type: ignore[assignment]
self.parse_mode: _ParseMode = parse_mode
self.image_analysis: bool = image_analysis
- self.vlm_config = (vlm_config if vlm_config is not None else config.model.vlm).model_copy(deep=True)
+ self.vlm_config = (vlm_config if vlm_config is not None else get_config().model.vlm).model_copy(deep=True)
def parse(
self,
diff --git a/mineru/parser/task_api.py b/mineru/parser/task_api.py
deleted file mode 100644
index f90d1b80..00000000
--- a/mineru/parser/task_api.py
+++ /dev/null
@@ -1,1167 +0,0 @@
-"""本地服务与 Router 共用的 JSON 任务、逐文件上传、执行和结果链接协议。"""
-
-from __future__ import annotations
-
-import asyncio
-import hashlib
-import logging
-import secrets
-import shutil
-import tempfile
-import time
-from collections.abc import AsyncIterator, Callable
-from dataclasses import dataclass, field, replace
-from pathlib import Path
-from typing import Any, Literal, NoReturn, Protocol
-from urllib.parse import quote
-
-from fastapi import APIRouter, Request, Response
-from fastapi.responses import JSONResponse
-from pydantic import AnyHttpUrl, BaseModel, ConfigDict, Field, ValidationError, model_validator
-from starlette.requests import ClientDisconnect
-
-from ..errors import MineruError
-from ..filetypes import is_flash_only_parse_extension
-from ..task_types import TaskFileState, TaskOutputFormat, TaskStatus
-from ..types import Tier
-from ..utils.async_utils import drain_future, run_sync
-from .api_server import (
- ApiServerError,
- _ParseRequest,
- ErrorDetail,
- _StoredSource,
- FileParseInfo,
- FileStore,
- _ParseInput,
- TaskProgress,
- JobStore,
- _UrlSource,
- _JobInputMetadata,
- _JobInputSpec,
- _raise_api_error,
- _require_model_preload_ready,
- _validate_source_policy,
- submit_execution,
-)
-from .page_range import PAGE_RANGE_DESCRIPTION, normalize_page_range_input
-
-MAX_TASK_FILES = 100
-TASK_UPLOAD_TTL = 3600
-TERMINAL_TASK_STATUSES = frozenset({"completed", "partial", "failed", "canceled"})
-logger = logging.getLogger(__name__)
-
-
-class TaskFileInput(BaseModel):
- """仅声明输入,文件标识由服务端分配,不接受任何客户端 ID。"""
-
- model_config = ConfigDict(extra="forbid")
- file_name: str = Field(min_length=1, max_length=1024)
- bytes: int | None = Field(default=None, gt=0, strict=True)
- url: str | None = None
- sha256sum: str | None = Field(default=None, pattern=r"^[a-f0-9]{64}$")
- page_range: str | None = Field(default=None, description=PAGE_RANGE_DESCRIPTION)
-
- @model_validator(mode="after")
- def validate_source(self) -> TaskFileInput:
- """上传项必须声明大小;URL 只校验合法性,保留签名地址的原始字符串。"""
- if self.url is None:
- if self.bytes is None:
- raise ValueError("bytes is required when url is not supplied")
- else:
- AnyHttpUrl(self.url)
- return self
-
-
-class TaskFileItem(TaskFileInput):
- """接单前补全文件标识,内部执行和状态响应只消费完整文件条目。"""
-
- file_id: str
-
-
-class UploadTaskOptions(BaseModel):
- """解析选项仅来自 JSON 主体,文件元数据独立保存在 files 中。"""
-
- model_config = ConfigDict(extra="forbid")
- tier: Tier | None = None
- ocr_mode: Literal["auto", "txt", "ocr"] = "auto"
- page_range: str = ""
- output_formats: list[TaskOutputFormat] = Field(default_factory=lambda: ["default"], min_length=1)
-
-
-class CreateTaskRequest(UploadTaskOptions):
- """严格 JSON 创建请求,清单是数组而不是编码后的字符串。"""
-
- files: list[TaskFileInput] = Field(min_length=1, max_length=MAX_TASK_FILES)
-
-
-class TaskUploadLink(BaseModel):
- """描述一个文件的上传目标,客户端原样使用服务端给出的地址和请求头。"""
-
- url: str
- headers: dict[str, str]
- expires_at: int
-
-
-class CreatedTaskFile(BaseModel):
- """创建回执只提供输入映射、当前状态和可执行的上传动作。"""
-
- file_id: str = Field(min_length=1)
- file_name: str
- status: TaskFileState
- upload: TaskUploadLink | None = None
-
- @model_validator(mode="after")
- def validate_upload_action(self) -> CreatedTaskFile:
- """创建回执的上传动作只能属于仍在等待文件的条目。"""
- if self.upload is not None and self.status != "waiting-file":
- raise ValueError("Only waiting-file accepts uploads")
- return self
-
-
-class CreateTaskResponse(BaseModel):
- """创建回执独立于查询详情,后续详情扩展不会进入创建响应。"""
-
- task_id: str
- status: TaskStatus
- files: list[CreatedTaskFile]
-
-
-class TaskFileDetail(BaseModel):
- """用统一状态描述文件准备和解析,上传对象仅表示客户端当前可执行的动作。"""
-
- file_id: str = Field(min_length=1)
- file_name: str
- bytes: int | None = None
- url: str | None = None
- upload: TaskUploadLink | None = None
- status: TaskFileState
- parse: FileParseInfo | None = None
- result_url: str | None = None
- error: ErrorDetail | None = None
-
- @model_validator(mode="after")
- def validate_upload_action(self) -> TaskFileDetail:
- """只有等待客户端文件的条目才能提供上传目标,服务端准备阶段可以为空。"""
- if self.upload is not None and (self.status != "waiting-file" or self.url is not None):
- raise ValueError("Only waiting-file without a URL source accepts uploads")
- return self
-
-
-class TaskDetailResponse(BaseModel):
- """任务状态只携带元数据;每个成功文件提供自己的下载链接。"""
-
- task_id: str
- status: TaskStatus
- created_at: str
- started_at: str | None = None
- finished_at: str | None = None
- output_formats: list[TaskOutputFormat]
- progress: TaskProgress
- files: list[TaskFileDetail]
- error: ErrorDetail | None = None
-
-
-class CanceledTaskFile(BaseModel):
- """取消回执保留每个文件的终态、可下载产物和失败原因。"""
-
- file_id: str = Field(min_length=1)
- file_name: str
- status: Literal["completed", "failed", "canceled"]
- parse: FileParseInfo | None = None
- result_url: str | None = None
- error: ErrorDetail | None = None
-
-
-class CancelTaskResponse(BaseModel):
- """取消回执只描述本次取消后的确定终态,不携带上传或查询扩展字段。"""
-
- task_id: str
- status: Literal["canceled"]
- finished_at: str
- files: list[CanceledTaskFile]
-
-
-class UploadFileResponse(BaseModel):
- """确认一次文件上传,仅返回对应文件的身份和当前状态。"""
-
- task_id: str
- file_id: str = Field(min_length=1)
- status: TaskFileState
-
-
-@dataclass(frozen=True)
-class TaskUpload:
- """保存已校验的源字节位置,文件名视图独立于内容去重路径。"""
-
- path: Path
- filename: str
- bytes: int
- sha256sum: str
- mime_type: str = "application/octet-stream"
- file_id: str | None = None
-
-
-@dataclass(frozen=True)
-class TaskExecution:
- """在适配器之间传递单文件执行快照、实际输入元数据及本部署的产物标识。"""
-
- job: dict[str, Any]
- result_file_id: str | None = None
- error: ErrorDetail | None = None
- input_metadata: dict[int, _JobInputMetadata] = field(default_factory=dict)
-
-
-class TaskBackend(Protocol):
- """上传准备复用各部署的存储,执行仍交给已有 Job 服务。"""
-
- owner_scope: str
-
- def reserve_file_id(self) -> str:
- """在本层 Files 命名空间内预留唯一产物 ID。"""
- ...
-
- def discard_file_id(self, file_id: str) -> None:
- """撤销失败或取消文件的预留及尚未发布的结果。"""
- ...
-
- def forget_file_id(self, file_id: str) -> None:
- """任务回收后释放预留索引,不删除仍有效的产物资源。"""
- ...
-
- def fork(self) -> TaskBackend:
- """为每个文件创建独立的引用和执行上下文。"""
- ...
-
- async def release_result(self, task_id: str, finished: float) -> None:
- """父任务收尾后解除子结果租约,并开始正常保留期。"""
- ...
-
- async def validate(self, options: UploadTaskOptions, inputs: list[TaskFileInput]) -> None:
- """在接单之前校验部署能力,不能预占解析名额。"""
- ...
-
- async def find_upload(self, sha256sum: str, filename: str) -> TaskUpload | None:
- """原子查找并固定当前调用方可以复用的源文件。"""
- ...
-
- async def retain_upload(self, upload: TaskUpload) -> TaskUpload:
- """持久保存已校验的单个输入,并持有跨请求准备阶段的引用。"""
- ...
-
- def release_uploads(self, *, rollback: bool) -> None:
- """提交后交接输入引用;未提交的取消或失败同时清理文件视图。"""
- ...
-
- async def submit(
- self, task_id: str, options: UploadTaskOptions, inputs: list[TaskFileItem], uploads: dict[int, TaskUpload]
- ) -> TaskExecution:
- """使用预先分配的公开 ID 提交一次后台解析。"""
- ...
-
- async def get(self, task_id: str) -> TaskExecution:
- """只查询执行和产物元数据,不读取结果文件内容。"""
- ...
-
- async def cancel(self, task_id: str) -> None:
- """取消对应后台工作,并收敛已有资源和预占。"""
- ...
-
-
-def _invalid(message: str, *, code: str = "invalid_request", status_code: int = 400, param: str | None = None) -> NoReturn:
- """把协议校验错误转换成两类服务共用的 V1 error envelope。"""
- _raise_api_error(
- status_code,
- error_type="api_error" if status_code >= 500 else "invalid_request_error",
- code=code,
- message=message,
- param=param,
- )
-
-
-def resolve_task_tier(options: UploadTaskOptions, inputs: list[TaskFileInput], tiers: set[Tier], default: Tier | None) -> Tier:
- """按现有轻量格式回退规则解析 tier,不触发 worker 选择或模型执行。"""
- lightweight = [is_flash_only_parse_extension(item.file_name) for item in inputs]
- if any(lightweight) and "flash" not in tiers:
- _invalid("Flash parsing is disabled, but an input requires it", param="files")
- tier = options.tier or default
- if tier is None and all(lightweight) and "flash" in tiers:
- tier = "flash"
- if tier is None:
- _invalid("No default quality tier is available; pass tier explicitly", code="quality_tier_unavailable", status_code=503)
- if tier not in tiers and not (all(lightweight) and "flash" in tiers):
- _invalid(f"Tier '{tier}' is not available", param="tier")
- return tier
-
-
-async def file_chunks(path: Path) -> AsyncIterator[bytes]:
- """以固定大小从磁盘读取,避免阻塞事件循环和完整文件缓冲。"""
- with path.open("rb") as source:
- while chunk := await run_sync(source.read, 1024 * 1024):
- yield chunk
-
-
-async def _save_upload(
- chunks: AsyncIterator[bytes],
- path: Path,
- filename: str,
- mime_type: str,
- *,
- expected: TaskFileItem | None = None,
- canceled: Callable[[], bool] | None = None,
-) -> TaskUpload:
- """流式落盘并校验完整性;声明大小用于一致性检查,不是上传大小上限。"""
- hasher = hashlib.sha256()
- size = 0
- try:
- with path.open("wb") as target:
- async for chunk in chunks:
- if canceled is not None and canceled():
- _invalid("Task was canceled during upload", status_code=409, code="task_already_terminal")
- size += len(chunk)
- if expected is not None and expected.bytes is not None and size > expected.bytes:
- _invalid("Declared size does not match uploaded bytes", code="upload_size_mismatch")
- await run_sync(target.write, chunk)
- hasher.update(chunk)
- if size == 0:
- _invalid("Empty files are not supported", param="files")
- if expected is not None and expected.bytes is not None and size != expected.bytes:
- _invalid("Declared size does not match uploaded bytes", code="upload_size_mismatch")
- sha = hasher.hexdigest()
- if expected is not None and expected.sha256sum is not None and sha != expected.sha256sum:
- _invalid("SHA-256 mismatch", code="file_hash_mismatch")
- return TaskUpload(path, filename, size, sha, mime_type)
- except BaseException:
- path.unlink(missing_ok=True)
- raise
-
-
-def _normalize_task_files(inputs: list[TaskFileInput], backend: TaskBackend) -> list[TaskFileItem]:
- """按请求顺序为每个输入预留公共产物 ID,失败时撤销已分配的预留。"""
- normalized: list[TaskFileItem] = []
- try:
- for item in inputs:
- normalized.append(TaskFileItem(**item.model_dump(), file_id=backend.reserve_file_id()))
- return normalized
- except BaseException:
- for item in normalized:
- backend.discard_file_id(item.file_id)
- backend.forget_file_id(item.file_id)
- raise
-
-
-async def _read_submission(request: Request) -> tuple[UploadTaskOptions, list[TaskFileInput]]:
- """只读取 JSON 清单,严格拒绝表单、直传字节和 URL 查询参数。"""
- if request.query_params:
- _invalid("POST parameters must be supplied in the JSON body, not the URL")
- media_type = request.headers.get("content-type", "").split(";", 1)[0].strip().lower()
- if media_type != "application/json":
- _invalid("Use application/json", status_code=415)
- try:
- body = CreateTaskRequest.model_validate(await request.json())
- inputs = body.files
- options = UploadTaskOptions.model_validate(body.model_dump(exclude={"files"}))
- options.page_range = normalize_page_range_input(options.page_range)
- options.output_formats = list(dict.fromkeys(options.output_formats))
- for index, item in enumerate(inputs):
- try:
- item.page_range = normalize_page_range_input(
- item.page_range if item.page_range is not None else options.page_range
- )
- except MineruError as exc:
- _invalid(str(exc), code=exc.code, param=f"files.{index}.page_range")
- return options, inputs
- except ClientDisconnect:
- _invalid("Request interrupted", code="request_interrupted")
- except MineruError as exc:
- _invalid(str(exc), code=exc.code, param=exc.param)
- except ValidationError as exc:
- location = exc.errors()[0]["loc"]
- _invalid("Invalid request: " + str(exc), param=".".join(map(str, location)) or None)
- except ValueError as exc:
- _invalid(str(exc))
-
-
-def _retention_lease(request: Request) -> int | None:
- """读取 Router 的有界结果租约,避免父任务结束前 worker 提前清理子文件。"""
- value = request.headers.get("x-mineru-task-lease-seconds")
- if value is None:
- return None
- try:
- seconds = int(value)
- if not 0 <= seconds <= TASK_UPLOAD_TTL:
- raise ValueError
- except ValueError:
- _invalid("Task result lease must be between 0 and 3600 seconds")
- return seconds
-
-
-@dataclass
-class _TaskFileRecord:
- """每个文件独立拥有准备引用、执行 ID 和一次性提交句柄。"""
-
- backend: TaskBackend
- file_id: str
- execution_id: str = field(default_factory=JobStore._new_job_id)
- status: TaskFileState = "waiting-file"
- upload: TaskUpload | None = None
- upload_lock: asyncio.Lock = field(default_factory=asyncio.Lock)
- upload_active: bool = False
- expired: bool = False
- execution: TaskExecution | None = None
- operation: asyncio.Task[None] | None = None
- submitted: bool = False
- finished: asyncio.Event = field(default_factory=asyncio.Event)
- lease_refresh_at: float = 0
- error: ErrorDetail | None = None
-
-
-@dataclass
-class _TaskRecord:
- """父 Task 只聚合文件状态,独立子 Job 继续使用现有解析并发。"""
-
- id: str
- backend: TaskBackend
- options: UploadTaskOptions
- inputs: list[TaskFileItem]
- directory: Path
- expires_at: int
- file_states: list[_TaskFileRecord]
- created_at: str = field(default_factory=JobStore._now)
- status: TaskStatus = "waiting-file"
- refresh_lock: asyncio.Lock = field(default_factory=asyncio.Lock)
- changed: asyncio.Event = field(default_factory=asyncio.Event)
- operation: asyncio.Task[None] | None = None
- finished_at: str | None = None
- finished_epoch: float | None = None
- retention_epoch: float | None = None
- lease_expires_at: float = 0
- error: ErrorDetail | None = None
-
-
-class TaskStore:
- """管理固定上传窗口和逐文件执行,不建立第二个解析队列。"""
-
- def __init__(self, retention_seconds: int) -> None:
- """只在创建应用时分配暂存目录,模块导入不产生副作用。"""
- self.retention_seconds = retention_seconds
- self._temporary = tempfile.TemporaryDirectory(prefix="mineru-tasks-")
- self._root = Path(self._temporary.name)
- self._records: dict[str, _TaskRecord] = {}
- self.storage_lock = asyncio.Lock()
- self._collector: asyncio.Task[None] | None = None
- self._closing = False
-
- def start(self) -> None:
- """启动过期扫描,关闭正常保留期也仍然执行上传截止时间。"""
- if self._collector is None and not self._closing:
- self._collector = asyncio.create_task(self._collect_loop(), name="mineru-task-retention")
-
- async def _collect_loop(self) -> None:
- """定期清理过期元数据;上传与状态请求也执行同一截止检查。"""
- while True:
- await asyncio.sleep(60)
- self.collect_expired()
-
- def _aggregate(self, record: _TaskRecord) -> None:
- """按运行、排队、等待的优先级聚合,所有文件终结后再发布成功或部分成功。"""
- if record.status == "canceled":
- return
- previous = record.status
- states = [file.status for file in record.file_states]
- if "running" in states:
- record.status = "running"
- elif "pending" in states:
- record.status = "pending"
- elif "waiting-file" in states:
- record.status = "waiting-file"
- else:
- completed = states.count("completed")
- record.status = "completed" if completed == len(states) else ("partial" if completed else "failed")
- if record.status in TERMINAL_TASK_STATUSES and record.finished_epoch is None:
- record.finished_at = JobStore._now()
- record.finished_epoch = time.time()
- if record.status != previous:
- record.changed.set()
-
- def _expire_uploads(self, record: _TaskRecord) -> None:
- """只终结尚未受理上传的文件;截止前已经受理的流允许继续完成。"""
- if record.expires_at > time.time() or record.status == "canceled":
- return
- for item, file in zip(record.inputs, record.file_states):
- if item.url is None and file.upload is None and file.status == "waiting-file" and not file.upload_active:
- file.expired = True
- file.status = "failed"
- file.error = ErrorDetail(type="invalid_request_error", code="upload_expired", message="Upload expired")
- self._aggregate(record)
-
- def collect_expired(self) -> None:
- """上传到期按文件处理,记录仅在清理结束并超过父任务保留期后移除。"""
- now = time.time()
- for task_id, record in tuple(self._records.items()):
- self._expire_uploads(record)
- if (
- self.retention_seconds
- and record.retention_epoch is not None
- and record.retention_epoch + self.retention_seconds <= now
- and record.operation is not None
- and record.operation.done()
- ):
- self._records.pop(task_id, None)
- for item in record.inputs:
- record.backend.forget_file_id(item.file_id)
- shutil.rmtree(record.directory, ignore_errors=True)
-
- def lookup(self, task_id: str, owner_scope: str) -> _TaskRecord:
- """未知与越权任务使用相同错误,避免跨调用方探测。"""
- self.collect_expired()
- record = self._records.get(task_id)
- if record is None or record.backend.owner_scope != owner_scope:
- _invalid(f"Task {task_id} not found", status_code=404, code="task_not_found")
- return record
-
- def update_lease(self, record: _TaskRecord, seconds: int | None) -> None:
- """续租只保护结果保留期,从不延长上传窗口,也不复活已释放的结果。"""
- if seconds is None:
- return
- if seconds and record.retention_epoch is not None:
- _invalid("Task result lease has already ended", status_code=409, code="task_lease_ended")
- record.lease_expires_at = time.time() + seconds if seconds else 0
- record.changed.set()
-
- async def create(
- self, backend: TaskBackend, options: UploadTaskOptions, inputs: list[TaskFileInput], *, lease_seconds: int = 0
- ) -> _TaskRecord:
- """先完整校验清单并固定缓存,再为各个就绪文件独立提交执行。"""
- if self._closing:
- _invalid("Server is shutting down", status_code=503, code="server_shutting_down")
- await backend.validate(options, inputs)
- inputs = _normalize_task_files(inputs, backend)
- task_id = JobStore._new_job_id()
- record = _TaskRecord(
- task_id,
- backend,
- options,
- inputs,
- self._root / task_id,
- int(time.time()) + TASK_UPLOAD_TTL,
- [_TaskFileRecord(backend.fork(), file_id=item.file_id) for item in inputs],
- )
- self.update_lease(record, lease_seconds)
- try:
- for index, item in enumerate(inputs):
- file = record.file_states[index]
- if item.url is None and item.sha256sum:
- upload = await file.backend.find_upload(item.sha256sum, item.file_name)
- if upload is not None:
- if upload.bytes != item.bytes:
- _invalid("Declared size does not match cached bytes", code="upload_size_mismatch")
- file.upload = await file.backend.retain_upload(upload)
- self._records[task_id] = record
- self.start()
- for index, item in enumerate(inputs):
- if item.url is not None or record.file_states[index].upload is not None:
- self._start_file(record, index)
- record.operation = asyncio.create_task(self._monitor(record), name=f"mineru-task-{task_id}")
- return record
- except BaseException:
- for item in inputs:
- backend.discard_file_id(item.file_id)
- backend.forget_file_id(item.file_id)
- for file in record.file_states:
- file.backend.release_uploads(rollback=True)
- raise
-
- def _start_file(self, record: _TaskRecord, index: int) -> None:
- """事件循环内同步登记提交句柄,重复 PUT 不能创建第二个子作业。"""
- file = record.file_states[index]
- if file.operation is not None or file.status != "waiting-file" or record.status == "canceled":
- return
- file.status = "pending" if file.upload is not None else "waiting-file"
- record.changed.set()
- file.operation = asyncio.create_task(self._dispatch_file(record, index), name=f"mineru-task-file-{file.execution_id}")
- self._aggregate(record)
-
- def _apply_execution(self, record: _TaskRecord, file: _TaskFileRecord, execution: TaskExecution) -> None:
- """只接受单文件执行快照,取消和已发布终态不被迟到响应覆盖。"""
- if file.status in TERMINAL_TASK_STATUSES:
- return
- status = execution.job.get("status")
- if status not in {"waiting-file", "pending", "running", "completed", "failed", "canceled"}:
- _invalid("Invalid file status from execution backend", code="invalid_response", status_code=502)
- if status == "completed" and execution.result_file_id is None:
- _invalid("Completed file has no result", code="invalid_response", status_code=502)
- if len(execution.job.get("files") or []) != 1:
- _invalid("Expected one file from execution backend", code="invalid_response", status_code=502)
- if status != file.status:
- record.changed.set()
- file.execution = execution
- file.status = status
- file.error = execution.error
- if status in TERMINAL_TASK_STATUSES:
- file.finished.set()
- file.lease_refresh_at = time.monotonic() + 30
- self._aggregate(record)
-
- async def refresh(self, record: _TaskRecord) -> None:
- """并发刷新独立文件;成功文件仍续租 worker 结果直到父任务收尾。"""
- async with record.refresh_lock:
- self._expire_uploads(record)
- if record.status in TERMINAL_TASK_STATUSES:
- return
- targets = [
- file
- for file in record.file_states
- if file.submitted
- and file.status not in {"failed", "canceled"}
- and (file.status != "completed" or file.lease_refresh_at <= time.monotonic())
- ]
- results = await asyncio.gather(*(file.backend.get(file.execution_id) for file in targets), return_exceptions=True)
- for file, result in zip(targets, results):
- if isinstance(result, BaseException):
- if (
- isinstance(result, ApiServerError)
- and result.status_code in {408, 429, 500, 502, 503, 504}
- and result.error.code not in {"invalid_response", "invalid_upstream_response"}
- ):
- logger.warning("Task file refresh failed: %s: %s", file.execution_id, result.error.code)
- continue
- if file.status not in TERMINAL_TASK_STATUSES:
- file.status = "failed"
- file.backend.discard_file_id(file.file_id)
- file.error = (
- result.error
- if isinstance(result, ApiServerError)
- else ErrorDetail(type="api_error", code="task_refresh_failed", message=str(result))
- )
- if file.operation is not None:
- file.operation.cancel()
- continue
- try:
- self._apply_execution(record, file, result)
- if file.status == "completed":
- file.lease_refresh_at = time.monotonic() + 30
- except ApiServerError as exc:
- file.status = "failed"
- file.backend.discard_file_id(file.file_id)
- file.error = exc.error
- if file.operation is not None:
- file.operation.cancel()
- self._aggregate(record)
-
- async def _dispatch_file(self, record: _TaskRecord, index: int) -> None:
- """每个文件独立提交,失败或取消只清理自己的作业与源文件租约。"""
- file = record.file_states[index]
- try:
- execution = await file.backend.submit(
- file.execution_id, record.options, [record.inputs[index]], {0: file.upload} if file.upload is not None else {}
- )
- file.submitted = True
- self._apply_execution(record, file, execution)
- file.backend.release_uploads(rollback=False)
- if file.status not in TERMINAL_TASK_STATUSES:
- await file.finished.wait()
- except asyncio.CancelledError:
- if file.status != "completed":
- file.backend.discard_file_id(file.file_id)
- try:
- await drain_future(asyncio.create_task(file.backend.cancel(file.execution_id)))
- except Exception:
- logger.exception("Task file cancellation failed: %s", file.execution_id)
- if file.status not in TERMINAL_TASK_STATUSES:
- file.status = "canceled"
- except Exception as exc:
- if file.status not in TERMINAL_TASK_STATUSES:
- file.status = "failed"
- file.error = (
- exc.error
- if isinstance(exc, ApiServerError)
- else ErrorDetail(type="api_error", code="task_submission_failed", message=str(exc))
- )
- if file.status != "completed":
- file.backend.discard_file_id(file.file_id)
- try:
- await file.backend.cancel(file.execution_id)
- except Exception:
- logger.exception("Task file failure cleanup failed: %s", file.execution_id)
- finally:
- file.finished.set()
- record.changed.set()
- file.backend.release_uploads(rollback=not file.submitted)
- self._aggregate(record)
-
- async def _monitor(self, record: _TaskRecord) -> None:
- """应用持有父任务监控;HTTP 断连不影响执行,所有子文件清理后才开始保留期。"""
- try:
- while record.status not in TERMINAL_TASK_STATUSES:
- record.changed.clear()
- await self.refresh(record)
- if record.status not in TERMINAL_TASK_STATUSES:
- await self._wait_for_change(record)
- operations = [file.operation for file in record.file_states if file.operation is not None]
- if operations:
- await asyncio.gather(*operations, return_exceptions=True)
- for item, file in zip(record.inputs, record.file_states):
- if file.status != "completed":
- file.backend.discard_file_id(item.file_id)
- while any(file.upload_active for file in record.file_states) or record.lease_expires_at > time.time():
- record.changed.clear()
- await self._wait_for_change(record)
- finished = time.time()
- for file in record.file_states:
- await file.backend.release_result(file.execution_id, finished)
- file.backend.release_uploads(rollback=not file.submitted)
- record.retention_epoch = finished
- except Exception:
- logger.exception("Task result cleanup failed: %s", record.id)
- finally:
- if not any(file.upload_active for file in record.file_states):
- shutil.rmtree(record.directory, ignore_errors=True)
-
- async def _wait_for_change(self, record: _TaskRecord) -> None:
- """状态变更即时唤醒;没有事件时每秒检查执行与到期边界。"""
- try:
- await asyncio.wait_for(record.changed.wait(), timeout=1)
- except asyncio.TimeoutError:
- pass
-
- async def upload(self, record: _TaskRecord, file_id: str, request: Request) -> bool:
- """在接收主体前检查固定期限,已受理流不因父任务状态变化中止。"""
- index = next((i for i, item in enumerate(record.inputs) if item.file_id == file_id), None)
- if index is None:
- _invalid("Unknown file_id", status_code=404, code="upload_not_found")
- expected = record.inputs[index]
- if expected.url is not None:
- _invalid("URL inputs do not accept uploads")
- file = record.file_states[index]
- async with file.upload_lock:
- if record.expires_at <= time.time():
- self._expire_uploads(record)
- _invalid("Upload link expired", status_code=410, code="upload_expired")
- if record.status == "canceled":
- _invalid("Task was canceled", status_code=409, code="task_already_terminal")
- if file.upload is not None:
- return True
- if file.status != "waiting-file":
- _invalid("File no longer accepts uploads", status_code=409, code="file_already_terminal")
- length = request.headers.get("content-length")
- if length is not None:
- try:
- declared = int(length)
- except ValueError:
- _invalid("Invalid Content-Length", code="upload_size_mismatch")
- if declared != expected.bytes:
- _invalid("Content-Length does not match declared file size", code="upload_size_mismatch")
- file.upload_active = True
- record.directory.mkdir(parents=True, exist_ok=True)
- path = record.directory / secrets.token_hex(16)
-
- def was_canceled() -> bool:
- """只在本文件被取消时中止已受理上传,不把上传截止时间当作传输超时。"""
- return record.status == "canceled" or file.status == "canceled"
-
- try:
- upload = await _save_upload(
- request.stream(),
- path,
- expected.file_name,
- request.headers.get("content-type", "application/octet-stream"),
- expected=expected,
- canceled=was_canceled,
- )
- if was_canceled():
- _invalid("Task was canceled during upload", status_code=409, code="task_already_terminal")
- retained = await file.backend.retain_upload(upload)
- if was_canceled():
- file.backend.release_uploads(rollback=True)
- _invalid("Task was canceled during upload", status_code=409, code="task_already_terminal")
- file.upload = retained
- self._start_file(record, index)
- return False
- except ClientDisconnect:
- _invalid("Upload interrupted; retry this file on the same task", code="upload_interrupted")
- finally:
- path.unlink(missing_ok=True)
- file.upload_active = False
- record.changed.set()
- self._expire_uploads(record)
-
- async def cancel(self, record: _TaskRecord, *, wait: bool = True) -> None:
- """保留已完成结果,只取消未完成文件并等待真实子作业清理。"""
- if record.status in TERMINAL_TASK_STATUSES:
- _invalid(f"Task is {record.status}", status_code=409, code="task_already_terminal")
- record.status = "canceled"
- record.finished_at = JobStore._now()
- record.finished_epoch = time.time()
- record.changed.set()
- for item, file in zip(record.inputs, record.file_states):
- if file.status not in TERMINAL_TASK_STATUSES:
- file.status = "canceled"
- file.backend.discard_file_id(item.file_id)
- if file.operation is not None:
- file.operation.cancel()
- if wait and record.operation is not None:
- record.lease_expires_at = 0
- record.changed.set()
- await drain_future(record.operation)
-
- async def shutdown(self) -> None:
- """停止接单并排空全部文件执行和结果租约,最后释放临时目录。"""
- self._closing = True
- if self._collector is not None:
- self._collector.cancel()
- await asyncio.gather(self._collector, return_exceptions=True)
- for record in tuple(self._records.values()):
- record.lease_expires_at = 0
- if record.status not in TERMINAL_TASK_STATUSES:
- await self.cancel(record, wait=False)
- await asyncio.gather(
- *(record.operation for record in self._records.values() if record.operation is not None), return_exceptions=True
- )
- self._temporary.cleanup()
-
-
-def _url(request: Request, path: str) -> str:
- """生成包含反向代理 root_path 的本部署地址,客户端无需自行拼接路径。"""
- return str(request.base_url).rstrip("/") + path
-
-
-def _upload_descriptor(record: _TaskRecord, index: int, request: Request) -> TaskUploadLink | None:
- """上传描述仅表示当前可执行的动作,不将服务端准备误判为文件就绪。"""
- item, state = record.inputs[index], record.file_states[index]
- if (
- item.url is None
- and state.upload is None
- and not state.upload_active
- and state.status == "waiting-file"
- and record.status != "canceled"
- and record.expires_at > time.time()
- ):
- return TaskUploadLink(
- url=_url(request, f"/v1/tasks/{record.id}/uploads/{quote(item.file_id, safe='')}"),
- headers={"Content-Type": "application/octet-stream"},
- expires_at=record.expires_at,
- )
- return None
-
-
-def _file_payload(record: _TaskRecord, index: int, request: Request) -> dict[str, Any]:
- """从内部执行记录提取文件信息,各响应模型显式限定自身公开字段。"""
- item, state = record.inputs[index], record.file_states[index]
- payload: dict[str, Any] = {
- "file_id": item.file_id,
- "file_name": item.file_name,
- "bytes": item.bytes,
- "url": item.url,
- "status": state.status,
- "upload": _upload_descriptor(record, index, request),
- }
- if state.execution is not None:
- results = state.execution.job.get("files") or []
- if results:
- payload["error"] = results[0].get("error")
- if state.status == "completed":
- payload["parse"] = results[0].get("parse")
- if (metadata := state.execution.input_metadata.get(0)) is not None:
- payload["bytes"] = metadata.bytes
- if state.status == "completed" and state.execution.result_file_id:
- if state.execution.result_file_id != item.file_id:
- raise ValueError("Result file identity differs from the reserved task file ID")
- payload["result_url"] = _url(request, f"/v1/files/{quote(item.file_id, safe='')}/content")
- if state.error is not None:
- payload["error"] = state.error
- return payload
-
-
-def create_task_payload(record: _TaskRecord, request: Request) -> dict[str, Any]:
- """创建响应只确认服务端文件标识和上传入口,不依赖 GET 详情模型。"""
- return CreateTaskResponse(
- task_id=record.id,
- status=record.status,
- files=[
- CreatedTaskFile(
- file_id=item.file_id,
- file_name=item.file_name,
- status=state.status,
- upload=_upload_descriptor(record, index, request),
- )
- for index, (item, state) in enumerate(zip(record.inputs, record.file_states))
- ],
- ).model_dump(exclude_none=True)
-
-
-def task_payload(record: _TaskRecord, request: Request) -> dict[str, Any]:
- """查询返回当前完整详情,递归省略空值且保留进度中的零。"""
- starts = [
- state.execution.job["started_at"]
- for state in record.file_states
- if state.execution is not None and state.execution.job.get("started_at")
- ]
- return TaskDetailResponse(
- task_id=record.id,
- status=record.status,
- created_at=record.created_at,
- started_at=min(starts) if starts else None,
- finished_at=record.finished_at,
- output_formats=record.options.output_formats,
- progress=TaskProgress(
- completed=sum(state.status == "completed" for state in record.file_states),
- failed=sum(state.status == "failed" for state in record.file_states),
- total=len(record.inputs),
- ),
- files=[TaskFileDetail.model_validate(_file_payload(record, index, request)) for index in range(len(record.inputs))],
- error=record.error,
- ).model_dump(exclude_none=True)
-
-
-def cancel_task_payload(record: _TaskRecord, request: Request) -> dict[str, Any]:
- """取消响应保留全部逐文件终态和成功产物,不携带上传或其他详情字段。"""
- return CancelTaskResponse(
- task_id=record.id,
- status=record.status,
- finished_at=record.finished_at,
- files=[CanceledTaskFile.model_validate(_file_payload(record, index, request)) for index in range(len(record.inputs))],
- ).model_dump(exclude_none=True)
-
-
-def _request_schema() -> dict[str, Any]:
- """OpenAPI 使用与运行时相同的 JSON 清单和格式约束。"""
- schema = CreateTaskRequest.model_json_schema()
- definitions = schema.pop("$defs", {})
-
- def expand(value: Any) -> Any:
- """内联局部模型引用,避免 OpenAPI 把 JSON Schema 的 $defs 当成全局路径。"""
- if isinstance(value, dict):
- if "$ref" in value:
- return expand(definitions[value["$ref"].rsplit("/", 1)[-1]])
- return {key: expand(item) for key, item in value.items()}
- if isinstance(value, list):
- return [expand(item) for item in value]
- return value
-
- return {"requestBody": {"required": True, "content": {"application/json": {"schema": expand(schema)}}}}
-
-
-def build_task_router(factory: Callable[[Request], TaskBackend]) -> APIRouter:
- """两类部署使用同一任务协议,下载交给已有 Files 入口。"""
- router = APIRouter(prefix="/v1", tags=["Tasks"])
-
- @router.post("/tasks", status_code=202, response_model=CreateTaskResponse, openapi_extra=_request_schema())
- async def create_task(request: Request) -> Response:
- """先验证 JSON 再建立稳定任务 ID,HTTP 结束后逐文件工作独立继续。"""
- backend = factory(request)
- store: TaskStore = request.app.state.task_store
- try:
- options, inputs = await _read_submission(request)
- record = await store.create(backend, options, inputs, lease_seconds=_retention_lease(request) or 0)
- except BaseException:
- backend.release_uploads(rollback=True)
- raise
- return JSONResponse(create_task_payload(record, request), status_code=202)
-
- @router.get("/tasks/{task_id}", response_model=TaskDetailResponse)
- async def get_task(task_id: str, request: Request) -> Response:
- """读取逐文件状态与结果地址,上传链接截止时间保持不变。"""
- store: TaskStore = request.app.state.task_store
- record = store.lookup(task_id, factory(request).owner_scope)
- store.update_lease(record, _retention_lease(request))
- await store.refresh(record)
- return JSONResponse(task_payload(record, request))
-
- @router.put(
- "/tasks/{task_id}/uploads/{file_id}",
- response_model=UploadFileResponse,
- openapi_extra={
- "requestBody": {
- "required": True,
- "content": {"application/octet-stream": {"schema": {"type": "string", "format": "binary"}}},
- }
- },
- )
- async def upload_task_file(task_id: str, file_id: str, request: Request) -> Response:
- """每条上传链接只接收一个文件,该文件就绪后立即提交执行。"""
- store: TaskStore = request.app.state.task_store
- record = store.lookup(task_id, factory(request).owner_scope)
- lease_seconds = _retention_lease(request)
- store.update_lease(record, lease_seconds)
- skipped = await store.upload(record, file_id, request)
- # 大文件转传可超过租约长度;受理完成时续租,保护紧接着发布的结果。
- store.update_lease(record, lease_seconds)
- return JSONResponse(
- UploadFileResponse(
- task_id=record.id,
- file_id=file_id,
- status=next(state.status for item, state in zip(record.inputs, record.file_states) if item.file_id == file_id),
- ).model_dump(),
- headers={"Connection": "close"} if skipped and request.scope.get("http_version") == "1.1" else None,
- )
-
- @router.post("/tasks/{task_id}/cancel", response_model=CancelTaskResponse)
- async def cancel_task(task_id: str, request: Request) -> Response:
- """统一取消等待上传与正在执行的任务,不重建后台作业。"""
- store: TaskStore = request.app.state.task_store
- record = store.lookup(task_id, factory(request).owner_scope)
- await store.refresh(record)
- await store.cancel(record, wait=False)
- return JSONResponse(cancel_task_payload(record, request))
-
- return router
-
-
-class ApiTaskBackend:
- """把上传准备接入 API server 的唯一文件存储和 JobStore。"""
-
- def __init__(self, request: Request) -> None:
- """保留轻量请求上下文,不跨请求持有上传主体。"""
- self.request = Request(dict(request.scope))
- self.files: FileStore = request.app.state.file_store
- self.jobs: JobStore = request.app.state.job_store
- self.owner_scope = "local"
- self._pins: list[str] = []
- self._owned_files: list[str] = []
-
- def fork(self) -> ApiTaskBackend:
- """为单个文件隔离源字节引用,其他文件提交不释放本文件资源。"""
- return ApiTaskBackend(self.request)
-
- def reserve_file_id(self) -> str:
- """在 API Server 的 Files 存储中预留统一的公共产物 ID。"""
- return self.files.reserve_file_id()
-
- def discard_file_id(self, file_id: str) -> None:
- """取消或失败时使结果预留失效,不影响共享的源文件缓存。"""
- self.files.discard_file_id(file_id)
-
- def forget_file_id(self, file_id: str) -> None:
- """任务记录过期后释放 ID 预留索引,Files 自身仍遵循原保留期。"""
- self.files.forget_file_id(file_id)
-
- async def release_result(self, task_id: str, finished: float) -> None:
- """由父 Task 统一开始子作业与结果文件的保留期。"""
- self.jobs.release_task_result(task_id, finished)
-
- async def validate(self, options: UploadTaskOptions, inputs: list[TaskFileInput]) -> None:
- """复用引擎就绪状态,提前校验显式 tier 与轻量格式能力。"""
- _require_model_preload_ready(self.request)
- if self.jobs._closing:
- _invalid("Server is shutting down", code="server_shutting_down", status_code=503)
- state = self.request.app.state
- for index, item in enumerate(inputs):
- if item.url is not None:
- try:
- _validate_source_policy(
- _UrlSource(url=item.url),
- allow_http_source=state.allow_http_source,
- )
- except ValueError as exc:
- _invalid(str(exc), param=f"files.{index}.url")
- options.tier = resolve_task_tier(options, inputs, set(state.tier_runtime_options), state.default_tier)
-
- async def find_upload(self, sha256sum: str, filename: str) -> TaskUpload | None:
- """原子查找并取得缓存租约,准备阶段不会因原文件视图到期丢失字节。"""
- path = self.files.pin_cached_blob(sha256sum)
- if path is None:
- return None
- self._pins.append(sha256sum)
- return TaskUpload(path, filename, path.stat().st_size, sha256sum)
-
- async def retain_upload(self, upload: TaskUpload) -> TaskUpload:
- """立即按内容去重并登记独立文件视图,后续文件失败不丢弃已成功上传项。"""
- file_id = self.files._allocate_file_id()
- self._owned_files.append(file_id)
- await run_sync(
- self.files.register_source_file, upload.filename, upload.path, upload.sha256sum, file_id, keep_alive=True
- )
- path = self.files.pin_cached_blob(upload.sha256sum)
- if path is None:
- raise RuntimeError("New source file disappeared")
- self._pins.append(upload.sha256sum)
- return replace(upload, path=path, file_id=file_id)
-
- def release_uploads(self, *, rollback: bool) -> None:
- """成功提交后交接输入;准备失败时删除本任务视图但保留其他任务引用。"""
- if rollback:
- for file_id in self._owned_files:
- if file_id in self.files._files:
- self.files.delete_file(file_id)
- self.files.forget_file_id(file_id)
- else:
- self.files.finalize_source_views(self._owned_files)
- self._owned_files.clear()
- for sha in self._pins:
- self.files.unpin_cached_blob(sha)
- self._pins.clear()
-
- async def submit(
- self, task_id: str, options: UploadTaskOptions, inputs: list[TaskFileItem], uploads: dict[int, TaskUpload]
- ) -> TaskExecution:
- """提交单文件子作业,在同一次解析中生成专用产物并持有结果租约。"""
- # 接单时已持有源字节;用户删除文件视图也不能破坏尚在等待其他文件的任务。
- for upload in uploads.values():
- await run_sync(
- self.files.register_source_file, upload.filename, upload.path, upload.sha256sum, upload.file_id, keep_alive=True
- )
- body = _ParseRequest(
- files=[
- _ParseInput(
- source=_UrlSource(url=item.url) if item.url is not None else _StoredSource(file_id=uploads[index].file_id),
- page_range=item.page_range or None,
- )
- for index, item in enumerate(inputs)
- ],
- tier=options.tier,
- ocr_mode=options.ocr_mode,
- output_formats=options.output_formats,
- )
-
- await submit_execution(
- body,
- self.request,
- self.files,
- self.jobs,
- job_id=task_id,
- input_specs={
- index: _JobInputSpec(item.file_name, item.bytes, item.sha256sum, result_file_id=item.file_id)
- for index, item in enumerate(inputs)
- },
- )
- return await self.get(task_id)
-
- async def get(self, task_id: str) -> TaskExecution:
- """只读取 Job 与最终 File 的标识,不在状态请求中访问产物正文。"""
- record = self.jobs.get(task_id)
- return TaskExecution(
- self.jobs.snapshot(record),
- record.task_result_file_id,
- None,
- dict(record.input_metadata),
- )
-
- async def cancel(self, task_id: str) -> None:
- """沿用 JobStore 取消及线程清理边界,终态竞争不影响已经完成的任务。"""
- record = self.jobs._jobs.get(task_id)
- if record is None:
- return
- if record.status not in TERMINAL_TASK_STATUSES:
- self.jobs.cancel(task_id)
- await self.jobs.wait_for_terminal(task_id, 3600)
-
-
-__all__ = [
- "ApiTaskBackend",
- "CreateTaskRequest",
- "TaskBackend",
- "TaskExecution",
- "TaskFileItem",
- "TaskFileInput",
- "CreateTaskResponse",
- "TaskDetailResponse",
- "CancelTaskResponse",
- "UploadFileResponse",
- "TaskStore",
- "TaskUpload",
- "UploadTaskOptions",
- "build_task_router",
- "file_chunks",
- "resolve_task_tier",
- "task_payload",
-]
diff --git a/mineru/parser/task_backend.py b/mineru/parser/task_backend.py
new file mode 100644
index 00000000..53b41c03
--- /dev/null
+++ b/mineru/parser/task_backend.py
@@ -0,0 +1,153 @@
+# Copyright (c) Opendatalab. All rights reserved.
+"""把共享 Tasks 执行协议接入本地解析服务的存储、并发与资源生命周期。"""
+
+from __future__ import annotations
+
+from dataclasses import replace
+
+from fastapi import Request
+
+from ..tasks.contracts import TaskFileInput, TaskFileItem, UploadTaskOptions
+from ..tasks.errors import invalid_task_request
+from ..tasks.execution import TaskExecution, TaskUpload
+from ..tasks.service import TERMINAL_TASK_STATUSES, resolve_task_tier
+from ..utils.async_utils import run_sync
+from .api_server import (
+ FileStore,
+ ExecutionStore,
+ _ExecutionRequest,
+ _StoredSource,
+ _UrlSource,
+ _require_model_preload_ready,
+ _validate_source_policy,
+ submit_execution,
+)
+
+
+class ApiTaskBackend:
+ """把上传准备接入 API server 的唯一文件存储和 ExecutionStore。"""
+
+ def __init__(self, request: Request) -> None:
+ """保留轻量请求上下文,不跨请求持有上传主体。"""
+ self.request = Request(dict(request.scope))
+ self.files: FileStore = request.app.state.file_store
+ self.executions: ExecutionStore = request.app.state.execution_store
+ self.owner_scope = "local"
+ self._pins: list[str] = []
+ self._owned_files: list[str] = []
+
+ def fork(self) -> ApiTaskBackend:
+ """为单个文件隔离源字节引用,其他文件提交不释放本文件资源。"""
+ return ApiTaskBackend(self.request)
+
+ def reserve_file_id(self) -> str:
+ """在 API Server 的 Files 存储中预留统一的公共产物 ID。"""
+ return self.files.reserve_file_id()
+
+ def discard_file_id(self, file_id: str) -> None:
+ """取消或失败时使结果预留失效,不影响共享的源文件缓存。"""
+ self.files.discard_file_id(file_id)
+
+ def forget_file_id(self, file_id: str) -> None:
+ """任务记录过期后释放 ID 预留索引,Files 自身仍遵循原保留期。"""
+ self.files.forget_file_id(file_id)
+
+ async def release_result(self, task_id: str, finished: float) -> None:
+ """由父 Task 统一开始子作业与结果文件的保留期。"""
+ self.executions.release_task_result(task_id, finished)
+
+ async def validate(self, options: UploadTaskOptions, inputs: list[TaskFileInput]) -> None:
+ """复用引擎就绪状态,提前校验显式 tier 与轻量格式能力。"""
+ _require_model_preload_ready(self.request)
+ if self.executions._closing:
+ invalid_task_request("Server is shutting down", code="server_shutting_down", status_code=503)
+ state = self.request.app.state
+ for index, item in enumerate(inputs):
+ if item.url is not None:
+ try:
+ _validate_source_policy(
+ _UrlSource(url=item.url),
+ allow_http_source=state.allow_http_source,
+ )
+ except ValueError as exc:
+ invalid_task_request(str(exc), param=f"files.{index}.url")
+ options.tier = resolve_task_tier(options, inputs, set(state.tier_runtime_options), state.default_tier)
+
+ async def find_upload(self, sha256sum: str, filename: str) -> TaskUpload | None:
+ """原子查找并取得缓存租约,准备阶段不会因原文件视图到期丢失字节。"""
+ path = self.files.pin_cached_blob(sha256sum)
+ if path is None:
+ return None
+ self._pins.append(sha256sum)
+ return TaskUpload(path, filename, path.stat().st_size, sha256sum)
+
+ async def retain_upload(self, upload: TaskUpload) -> TaskUpload:
+ """立即按内容去重并登记独立文件视图,后续文件失败不丢弃已成功上传项。"""
+ file_id = self.files._allocate_file_id()
+ self._owned_files.append(file_id)
+ await run_sync(
+ self.files.register_source_file, upload.filename, upload.path, upload.sha256sum, file_id, keep_alive=True
+ )
+ path = self.files.pin_cached_blob(upload.sha256sum)
+ if path is None:
+ raise RuntimeError("New source file disappeared")
+ self._pins.append(upload.sha256sum)
+ return replace(upload, path=path, file_id=file_id)
+
+ def release_uploads(self, *, rollback: bool) -> None:
+ """成功提交后交接输入;准备失败时删除本任务视图但保留其他任务引用。"""
+ if rollback:
+ for file_id in self._owned_files:
+ if file_id in self.files._files:
+ self.files.delete_file(file_id)
+ self.files.forget_file_id(file_id)
+ else:
+ self.files.finalize_source_views(self._owned_files)
+ self._owned_files.clear()
+ for sha in self._pins:
+ self.files.unpin_cached_blob(sha)
+ self._pins.clear()
+
+ async def submit(
+ self, execution_id: str, options: UploadTaskOptions, item: TaskFileItem, upload: TaskUpload | None
+ ) -> TaskExecution:
+ """提交唯一文件并保留公开结果身份,源文件视图只用于内部读取和租约。"""
+ if upload is not None:
+ await run_sync(
+ self.files.register_source_file, upload.filename, upload.path, upload.sha256sum, upload.file_id, keep_alive=True
+ )
+ if item.url is not None:
+ source = _UrlSource(url=item.url)
+ elif upload is not None and upload.file_id is not None:
+ source = _StoredSource(file_id=upload.file_id)
+ else:
+ raise ValueError("Prepared source file is missing")
+ body = _ExecutionRequest(
+ source=source,
+ file_id=item.file_id,
+ file_name=item.file_name,
+ bytes=item.bytes,
+ sha256sum=item.sha256sum,
+ page_range=item.page_range or None,
+ tier=options.tier,
+ ocr_mode=options.ocr_mode,
+ output_formats=options.output_formats,
+ )
+ await submit_execution(body, self.request, self.files, self.executions, execution_id=execution_id)
+ return await self.get(execution_id)
+
+ async def get(self, task_id: str) -> TaskExecution:
+ """返回单文件执行快照,状态查询不读取产物内容。"""
+ return self.executions.snapshot(self.executions.get(task_id))
+
+ async def cancel(self, task_id: str) -> None:
+ """沿用执行取消及线程清理边界,终态竞争不影响已经完成的任务。"""
+ record = self.executions._executions.get(task_id)
+ if record is None:
+ return
+ if record.status not in TERMINAL_TASK_STATUSES:
+ self.executions.cancel(task_id)
+ await self.executions.wait_for_terminal(task_id, 3600)
+
+
+__all__ = ["ApiTaskBackend"]
diff --git a/mineru/parser/tier.py b/mineru/parser/tier.py
index ff53481b..e6a61466 100644
--- a/mineru/parser/tier.py
+++ b/mineru/parser/tier.py
@@ -62,7 +62,7 @@ def runtime_options_for_tier(tier: Tier) -> ParserRuntimeOptions:
def required_modules_for_tier(tier: DeploymentTier, *, vlm_config: VlmConfig | None = None) -> list[str]:
"""按独立后端组合预检依赖,远程 VLM 不要求本地引擎。"""
- from ..config import config
+ from ..config import get_config
from ..model.runtime.device import TORCH_REQUIRED_MODULES, resolve_small_model_backend
from ..model.vlm.selector import VLM_REQUIRED_MODULES, resolve_vlm_engine
@@ -71,7 +71,7 @@ def required_modules_for_tier(tier: DeploymentTier, *, vlm_config: VlmConfig | N
modules = ["onnxruntime"]
if resolve_small_model_backend() == "torch":
modules.extend(TORCH_REQUIRED_MODULES)
- settings = vlm_config if vlm_config is not None else config.model.vlm
+ settings = vlm_config if vlm_config is not None else get_config().model.vlm
if tier == "standard" and not settings.server_url:
modules.extend(VLM_REQUIRED_MODULES[resolve_vlm_engine(settings.engine)])
return list(dict.fromkeys(modules))
diff --git a/mineru/render/content_list.py b/mineru/render/content_list.py
index 7ca406f7..aa34154c 100644
--- a/mineru/render/content_list.py
+++ b/mineru/render/content_list.py
@@ -14,10 +14,12 @@ def render_content_list(
asset_base_url: str = "",
) -> list[dict[str, Any]]:
"""惰性加载 Content List V1 实现并渲染严格 MiddleJson。"""
- from ..config import config
+ from ..config import get_config
from ._internal.content_list.v1 import render_content_list as _render_content_list
- return _render_content_list(middle_json, asset_base_url=asset_base_url, latex_delimiters=config.render.latex_delimiters)
+ return _render_content_list(
+ middle_json, asset_base_url=asset_base_url, latex_delimiters=get_config().render.latex_delimiters
+ )
__all__ = ["render_content_list"]
diff --git a/mineru/render/content_list_v2.py b/mineru/render/content_list_v2.py
index aca910a3..16d5cf79 100644
--- a/mineru/render/content_list_v2.py
+++ b/mineru/render/content_list_v2.py
@@ -14,10 +14,12 @@ def render_content_list_v2(
asset_base_url: str = "",
) -> list[list[dict[str, Any]]]:
"""惰性加载 Content List V2 实现并渲染严格 MiddleJson。"""
- from ..config import config
+ from ..config import get_config
from ._internal.content_list.v2 import render_content_list_v2 as _render_content_list_v2
- return _render_content_list_v2(middle_json, asset_base_url=asset_base_url, latex_delimiters=config.render.latex_delimiters)
+ return _render_content_list_v2(
+ middle_json, asset_base_url=asset_base_url, latex_delimiters=get_config().render.latex_delimiters
+ )
__all__ = ["render_content_list_v2"]
diff --git a/mineru/render/markdown.py b/mineru/render/markdown.py
index e5fb3c22..66053df9 100644
--- a/mineru/render/markdown.py
+++ b/mineru/render/markdown.py
@@ -15,7 +15,7 @@ def render_markdown(
image_renderer: ImageRenderer | None = None,
) -> str:
"""惰性加载 Markdown 实现并渲染严格 MiddleJson。"""
- from ..config import config
+ from ..config import get_config
from docvortex.render.markdown import render_markdown as _render_markdown
return _render_markdown(
@@ -23,7 +23,7 @@ def render_markdown(
mode=mode,
asset_base_url=asset_base_url,
image_renderer=image_renderer,
- latex_delimiters=config.render.latex_delimiters,
+ latex_delimiters=get_config().render.latex_delimiters,
)
diff --git a/mineru/render/structured_content.py b/mineru/render/structured_content.py
index 95d937de..f3e6e503 100644
--- a/mineru/render/structured_content.py
+++ b/mineru/render/structured_content.py
@@ -15,12 +15,12 @@ def render_structured_content(
asset_base_url: str = "",
) -> dict[str, Any]:
"""惰性加载 Structured Content 实现并渲染严格 MiddleJson。"""
- from ..config import config
+ from ..config import get_config
from docvortex.render.structured_content import render_structured_content as _render_structured_content
validate_mineru_metadata(middle_json)
return _render_structured_content(
- middle_json, asset_base_url=asset_base_url, latex_delimiters=config.render.latex_delimiters
+ middle_json, asset_base_url=asset_base_url, latex_delimiters=get_config().render.latex_delimiters
)
diff --git a/mineru/tasks/__init__.py b/mineru/tasks/__init__.py
new file mode 100644
index 00000000..a0076624
--- /dev/null
+++ b/mineru/tasks/__init__.py
@@ -0,0 +1,3 @@
+"""Tasks 的共享契约、执行边界及生命周期组件。"""
+
+__all__: list[str] = []
diff --git a/mineru/tasks/contracts.py b/mineru/tasks/contracts.py
new file mode 100644
index 00000000..506fd7cd
--- /dev/null
+++ b/mineru/tasks/contracts.py
@@ -0,0 +1,201 @@
+"""Tasks 共享协议模型,仅定义字段与校验,不依赖 HTTP 或本地执行服务。"""
+
+from __future__ import annotations
+
+from typing import Literal
+
+from docvortex.document.page_range import PAGE_RANGE_DESCRIPTION
+from pydantic import AnyHttpUrl, BaseModel, ConfigDict, Field, model_validator
+
+from ..task_types import TaskFileState, TaskOutputFormat, TaskStatus
+from ..types import Tier
+
+MAX_TASK_FILES = 100
+_PYDANTIC_CONFIG = ConfigDict(extra="forbid")
+
+
+class ErrorDetail(BaseModel):
+ model_config = _PYDANTIC_CONFIG
+ type: str
+ code: str | None = None
+ message: str
+ param: str | None = None
+
+
+class ErrorResponse(BaseModel):
+ model_config = _PYDANTIC_CONFIG
+ error: ErrorDetail
+
+
+class TaskProgress(BaseModel):
+ model_config = _PYDANTIC_CONFIG
+ completed: int = 0
+ failed: int = 0
+ total: int = 0
+
+
+class FileParseInfo(BaseModel):
+ model_config = _PYDANTIC_CONFIG
+ tier: Tier
+ duration_ms: int | None = None
+ parser_version: str | None = None
+
+
+class TaskFileIdentity(BaseModel):
+ """共享文件身份和状态,部署响应及客户端可分别扩展各自上下文。"""
+
+ file_id: str = Field(min_length=1)
+ file_name: str
+ status: TaskFileState
+
+
+class TaskFileInput(BaseModel):
+ """仅声明输入,文件标识由服务端分配,不接受任何客户端 ID。"""
+
+ model_config = ConfigDict(extra="forbid")
+ file_name: str = Field(min_length=1, max_length=1024)
+ bytes: int | None = Field(default=None, gt=0, strict=True)
+ url: str | None = None
+ sha256sum: str | None = Field(default=None, pattern=r"^[a-f0-9]{64}$")
+ page_range: str | None = Field(default=None, description=PAGE_RANGE_DESCRIPTION)
+
+ @model_validator(mode="after")
+ def validate_source(self) -> TaskFileInput:
+ """上传项必须声明大小;URL 只校验合法性,保留签名地址的原始字符串。"""
+ if self.url is None:
+ if self.bytes is None:
+ raise ValueError("bytes is required when url is not supplied")
+ else:
+ AnyHttpUrl(self.url)
+ return self
+
+
+class TaskFileItem(TaskFileInput):
+ """接单前补全文件标识,内部执行和状态响应只消费完整文件条目。"""
+
+ file_id: str
+
+
+class UploadTaskOptions(BaseModel):
+ """解析选项仅来自 JSON 主体,文件元数据独立保存在 files 中。"""
+
+ model_config = ConfigDict(extra="forbid")
+ tier: Tier | None = None
+ ocr_mode: Literal["auto", "txt", "ocr"] = "auto"
+ page_range: str = ""
+ output_formats: list[TaskOutputFormat] = Field(default_factory=lambda: ["default"], min_length=1)
+
+
+class CreateTaskRequest(UploadTaskOptions):
+ """严格 JSON 创建请求,清单是数组而不是编码后的字符串。"""
+
+ files: list[TaskFileInput] = Field(min_length=1, max_length=MAX_TASK_FILES)
+
+
+class TaskUploadLink(BaseModel):
+ """描述一个文件的上传目标,客户端原样使用服务端给出的地址和请求头。"""
+
+ url: str
+ headers: dict[str, str]
+ expires_at: int
+
+
+class CreatedTaskFile(TaskFileIdentity):
+ """创建回执只提供输入映射、当前状态和可执行的上传动作。"""
+
+ upload: TaskUploadLink | None = None
+
+ @model_validator(mode="after")
+ def validate_upload_action(self) -> CreatedTaskFile:
+ """创建回执的上传动作只能属于仍在等待文件的条目。"""
+ if self.upload is not None and self.status != "waiting-file":
+ raise ValueError("Only waiting-file accepts uploads")
+ return self
+
+
+class CreateTaskResponse(BaseModel):
+ """创建回执独立于查询详情,后续详情扩展不会进入创建响应。"""
+
+ task_id: str
+ status: TaskStatus
+ files: list[CreatedTaskFile]
+
+
+class TaskFileDetail(TaskFileIdentity):
+ """用统一状态描述文件准备和解析,上传对象仅表示客户端当前可执行的动作。"""
+
+ bytes: int | None = None
+ url: str | None = None
+ upload: TaskUploadLink | None = None
+ parse: FileParseInfo | None = None
+ result_url: str | None = None
+ error: ErrorDetail | None = None
+
+ @model_validator(mode="after")
+ def validate_upload_action(self) -> TaskFileDetail:
+ """只有等待客户端文件的条目才能提供上传目标,服务端准备阶段可以为空。"""
+ if self.upload is not None and (self.status != "waiting-file" or self.url is not None):
+ raise ValueError("Only waiting-file without a URL source accepts uploads")
+ return self
+
+
+class TaskDetailResponse(BaseModel):
+ """任务状态只携带元数据;每个成功文件提供自己的下载链接。"""
+
+ task_id: str
+ status: TaskStatus
+ created_at: str
+ started_at: str | None = None
+ finished_at: str | None = None
+ output_formats: list[TaskOutputFormat]
+ progress: TaskProgress
+ files: list[TaskFileDetail]
+ error: ErrorDetail | None = None
+
+
+class CanceledTaskFile(TaskFileIdentity):
+ """取消回执保留每个文件的终态、可下载产物和失败原因。"""
+
+ status: Literal["completed", "failed", "canceled"]
+ parse: FileParseInfo | None = None
+ result_url: str | None = None
+ error: ErrorDetail | None = None
+
+
+class CancelTaskResponse(BaseModel):
+ """取消回执只描述本次取消后的确定终态,不携带上传或查询扩展字段。"""
+
+ task_id: str
+ status: Literal["canceled"]
+ finished_at: str
+ files: list[CanceledTaskFile]
+
+
+class UploadFileResponse(BaseModel):
+ """确认一次文件上传,仅返回对应文件的身份和当前状态。"""
+
+ task_id: str
+ file_id: str = Field(min_length=1)
+ status: TaskFileState
+
+
+__all__ = [
+ "MAX_TASK_FILES",
+ "ErrorDetail",
+ "ErrorResponse",
+ "TaskProgress",
+ "FileParseInfo",
+ "TaskFileIdentity",
+ "TaskFileInput",
+ "TaskFileItem",
+ "UploadTaskOptions",
+ "CreateTaskRequest",
+ "TaskUploadLink",
+ "CreatedTaskFile",
+ "CreateTaskResponse",
+ "TaskFileDetail",
+ "TaskDetailResponse",
+ "CanceledTaskFile",
+ "CancelTaskResponse",
+ "UploadFileResponse",
+]
diff --git a/mineru/tasks/errors.py b/mineru/tasks/errors.py
new file mode 100644
index 00000000..9a5b5f3b
--- /dev/null
+++ b/mineru/tasks/errors.py
@@ -0,0 +1,53 @@
+"""Tasks 与本地执行服务共用的结构化错误,不依赖 HTTP 框架。"""
+
+from __future__ import annotations
+
+from typing import NoReturn
+
+from .contracts import ErrorDetail
+
+
+class ApiServerError(Exception):
+ """在共享任务业务边界中携带 HTTP 状态和结构化错误内容。"""
+
+ def __init__(self, status_code: int, error: ErrorDetail) -> None:
+ """保存调用方的状态与错误字段,异常文本沿用公开错误信息。"""
+ super().__init__(error.message)
+ self.status_code = status_code
+ self.error = error
+
+
+def raise_api_error(
+ status_code: int,
+ *,
+ error_type: str,
+ code: str | None,
+ message: str,
+ param: str | None = None,
+) -> NoReturn:
+ """统一构造并抛出已有错误信封,不改变字段或状态码。"""
+ raise ApiServerError(
+ status_code,
+ ErrorDetail(
+ type=error_type,
+ code=code,
+ message=message,
+ param=param,
+ ),
+ )
+
+
+def invalid_task_request(
+ message: str, *, code: str = "invalid_request", status_code: int = 400, param: str | None = None
+) -> NoReturn:
+ """把协议校验错误转换成两类服务共用的 V1 error envelope。"""
+ raise_api_error(
+ status_code,
+ error_type="api_error" if status_code >= 500 else "invalid_request_error",
+ code=code,
+ message=message,
+ param=param,
+ )
+
+
+__all__ = ["ApiServerError", "invalid_task_request", "raise_api_error"]
diff --git a/mineru/tasks/execution.py b/mineru/tasks/execution.py
new file mode 100644
index 00000000..5ae37d6e
--- /dev/null
+++ b/mineru/tasks/execution.py
@@ -0,0 +1,118 @@
+"""部署无关的上传与执行协议,隔离本地 ExecutionStore 和 Router 实现。"""
+
+from __future__ import annotations
+
+import secrets
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Protocol
+
+from ..task_types import TaskFileState
+from .contracts import ErrorDetail, FileParseInfo, TaskFileInput, TaskFileItem, UploadTaskOptions
+
+
+@dataclass(frozen=True)
+class TaskInputMetadata:
+ """记录完整下载且校验通过的实际输入信息,供父 Task 更新文件元数据。"""
+
+ bytes: int
+
+
+@dataclass(frozen=True)
+class TaskUpload:
+ """保存已校验的源字节位置,文件名视图独立于内容去重路径。"""
+
+ path: Path
+ filename: str
+ bytes: int
+ sha256sum: str
+ mime_type: str = "application/octet-stream"
+ file_id: str | None = None
+
+
+@dataclass(frozen=True)
+class TaskExecution:
+ """记录单文件执行快照;file_id 固定为公开结果身份,不代表内部源文件引用。"""
+
+ execution_id: str
+ file_id: str
+ file_name: str
+ status: TaskFileState
+ created_at: str
+ started_at: str | None = None
+ finished_at: str | None = None
+ page_range: str = ""
+ parse: FileParseInfo | None = None
+ result_file_id: str | None = None
+ error: ErrorDetail | None = None
+ input_metadata: TaskInputMetadata | None = None
+
+
+class TaskBackend(Protocol):
+ """上传准备复用各部署的存储,单文件执行由本地或 Router 适配器负责。"""
+
+ owner_scope: str
+
+ def reserve_file_id(self) -> str:
+ """在本层 Files 命名空间内预留唯一产物 ID。"""
+ ...
+
+ def discard_file_id(self, file_id: str) -> None:
+ """撤销失败或取消文件的预留及尚未发布的结果。"""
+ ...
+
+ def forget_file_id(self, file_id: str) -> None:
+ """任务回收后释放预留索引,不删除仍有效的产物资源。"""
+ ...
+
+ def fork(self) -> TaskBackend:
+ """为每个文件创建独立的引用和执行上下文。"""
+ ...
+
+ async def release_result(self, task_id: str, finished: float) -> None:
+ """父任务收尾后解除子结果租约,并开始正常保留期。"""
+ ...
+
+ async def validate(self, options: UploadTaskOptions, inputs: list[TaskFileInput]) -> None:
+ """在接单之前校验部署能力,不能预占解析名额。"""
+ ...
+
+ async def find_upload(self, sha256sum: str, filename: str) -> TaskUpload | None:
+ """原子查找并固定当前调用方可以复用的源文件。"""
+ ...
+
+ async def retain_upload(self, upload: TaskUpload) -> TaskUpload:
+ """持久保存已校验的单个输入,并持有跨请求准备阶段的引用。"""
+ ...
+
+ def release_uploads(self, *, rollback: bool) -> None:
+ """提交后交接输入引用;未提交的取消或失败同时清理文件视图。"""
+ ...
+
+ async def submit(
+ self, execution_id: str, options: UploadTaskOptions, item: TaskFileItem, upload: TaskUpload | None
+ ) -> TaskExecution:
+ """使用预留的公开文件身份提交一次单文件执行,不承载批次聚合。"""
+ ...
+
+ async def get(self, task_id: str) -> TaskExecution:
+ """只查询执行和产物元数据,不读取结果文件内容。"""
+ ...
+
+ async def cancel(self, task_id: str) -> None:
+ """取消对应后台工作,并收敛已有资源和预占。"""
+ ...
+
+
+def new_execution_id() -> str:
+ """保持现有任务和执行标识格式,生成不可预测的独立 ID。"""
+ return "job_" + secrets.token_hex(12)
+
+
+def execution_timestamp() -> str:
+ """按现有秒级 UTC 格式记录任务和本地执行的生命周期时间。"""
+ return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+
+
+__all__ = ["TaskBackend", "TaskExecution", "TaskInputMetadata", "TaskUpload", "execution_timestamp", "new_execution_id"]
diff --git a/mineru/tasks/http.py b/mineru/tasks/http.py
new file mode 100644
index 00000000..480543f2
--- /dev/null
+++ b/mineru/tasks/http.py
@@ -0,0 +1,274 @@
+"""Tasks 的 HTTP 输入、响应序列化与路由,生命周期交给共享服务管理。"""
+
+from __future__ import annotations
+
+import time
+from collections.abc import Callable
+from typing import Any
+from urllib.parse import quote
+
+from docvortex.document.page_range import normalize_page_range_input
+from docvortex.errors import InvalidRequestError
+from fastapi import APIRouter, Request, Response
+from fastapi.responses import JSONResponse
+from pydantic import ValidationError
+from starlette.requests import ClientDisconnect
+
+from .contracts import (
+ CancelTaskResponse,
+ CanceledTaskFile,
+ CreateTaskRequest,
+ CreateTaskResponse,
+ CreatedTaskFile,
+ TaskDetailResponse,
+ TaskFileDetail,
+ TaskFileInput,
+ TaskProgress,
+ TaskUploadLink,
+ UploadFileResponse,
+ UploadTaskOptions,
+)
+from .errors import invalid_task_request
+from .execution import TaskBackend
+from .service import TASK_UPLOAD_TTL, TaskRecord, TaskStore
+
+
+async def _read_submission(request: Request) -> tuple[UploadTaskOptions, list[TaskFileInput]]:
+ """只读取 JSON 清单,严格拒绝表单、直传字节和 URL 查询参数。"""
+ if request.query_params:
+ invalid_task_request("POST parameters must be supplied in the JSON body, not the URL")
+ media_type = request.headers.get("content-type", "").split(";", 1)[0].strip().lower()
+ if media_type != "application/json":
+ invalid_task_request("Use application/json", status_code=415)
+ try:
+ body = CreateTaskRequest.model_validate(await request.json())
+ inputs = body.files
+ options = UploadTaskOptions.model_validate(body.model_dump(exclude={"files"}))
+ options.page_range = normalize_page_range_input(options.page_range)
+ options.output_formats = list(dict.fromkeys(options.output_formats))
+ for index, item in enumerate(inputs):
+ try:
+ item.page_range = normalize_page_range_input(
+ item.page_range if item.page_range is not None else options.page_range
+ )
+ except InvalidRequestError as exc:
+ invalid_task_request(str(exc), code=exc.code, param=f"files.{index}.page_range")
+ return options, inputs
+ except ClientDisconnect:
+ invalid_task_request("Request interrupted", code="request_interrupted")
+ except InvalidRequestError as exc:
+ invalid_task_request(str(exc), code=exc.code, param=exc.param)
+ except ValidationError as exc:
+ location = exc.errors()[0]["loc"]
+ invalid_task_request("Invalid request: " + str(exc), param=".".join(map(str, location)) or None)
+ except ValueError as exc:
+ invalid_task_request(str(exc))
+
+
+def _retention_lease(request: Request) -> int | None:
+ """读取 Router 的有界结果租约,避免父任务结束前 worker 提前清理子文件。"""
+ value = request.headers.get("x-mineru-task-lease-seconds")
+ if value is None:
+ return None
+ try:
+ seconds = int(value)
+ if not 0 <= seconds <= TASK_UPLOAD_TTL:
+ raise ValueError
+ except ValueError:
+ invalid_task_request("Task result lease must be between 0 and 3600 seconds")
+ return seconds
+
+
+def _url(request: Request, path: str) -> str:
+ """生成包含反向代理 root_path 的本部署地址,客户端无需自行拼接路径。"""
+ return str(request.base_url).rstrip("/") + path
+
+
+def _upload_descriptor(record: TaskRecord, index: int, request: Request) -> TaskUploadLink | None:
+ """上传描述仅表示当前可执行的动作,不将服务端准备误判为文件就绪。"""
+ item, state = record.inputs[index], record.file_states[index]
+ if (
+ item.url is None
+ and state.upload is None
+ and not state.upload_active
+ and state.status == "waiting-file"
+ and record.status != "canceled"
+ and record.expires_at > time.time()
+ ):
+ return TaskUploadLink(
+ url=_url(request, f"/v1/tasks/{record.id}/uploads/{quote(item.file_id, safe='')}"),
+ headers={"Content-Type": "application/octet-stream"},
+ expires_at=record.expires_at,
+ )
+ return None
+
+
+def _file_payload(record: TaskRecord, index: int, request: Request) -> dict[str, Any]:
+ """从内部执行记录提取文件信息,各响应模型显式限定自身公开字段。"""
+ item, state = record.inputs[index], record.file_states[index]
+ payload: dict[str, Any] = {
+ "file_id": item.file_id,
+ "file_name": item.file_name,
+ "bytes": item.bytes,
+ "url": item.url,
+ "status": state.status,
+ "upload": _upload_descriptor(record, index, request),
+ }
+ if state.execution is not None:
+ payload["error"] = state.execution.error
+ if state.status == "completed":
+ payload["parse"] = state.execution.parse
+ if (metadata := state.execution.input_metadata) is not None:
+ payload["bytes"] = metadata.bytes
+ if state.status == "completed" and state.execution.result_file_id:
+ if state.execution.result_file_id != item.file_id:
+ raise ValueError("Result file identity differs from the reserved task file ID")
+ payload["result_url"] = _url(request, f"/v1/files/{quote(item.file_id, safe='')}/content")
+ if state.error is not None:
+ payload["error"] = state.error
+ return payload
+
+
+def create_task_payload(record: TaskRecord, request: Request) -> dict[str, Any]:
+ """创建响应只确认服务端文件标识和上传入口,不依赖 GET 详情模型。"""
+ return CreateTaskResponse(
+ task_id=record.id,
+ status=record.status,
+ files=[
+ CreatedTaskFile(
+ file_id=item.file_id,
+ file_name=item.file_name,
+ status=state.status,
+ upload=_upload_descriptor(record, index, request),
+ )
+ for index, (item, state) in enumerate(zip(record.inputs, record.file_states))
+ ],
+ ).model_dump(exclude_none=True)
+
+
+def task_payload(record: TaskRecord, request: Request) -> dict[str, Any]:
+ """查询返回当前完整详情,递归省略空值且保留进度中的零。"""
+ starts = [
+ state.execution.started_at for state in record.file_states if state.execution is not None and state.execution.started_at
+ ]
+ return TaskDetailResponse(
+ task_id=record.id,
+ status=record.status,
+ created_at=record.created_at,
+ started_at=min(starts) if starts else None,
+ finished_at=record.finished_at,
+ output_formats=record.options.output_formats,
+ progress=TaskProgress(
+ completed=sum(state.status == "completed" for state in record.file_states),
+ failed=sum(state.status == "failed" for state in record.file_states),
+ total=len(record.inputs),
+ ),
+ files=[TaskFileDetail.model_validate(_file_payload(record, index, request)) for index in range(len(record.inputs))],
+ error=record.error,
+ ).model_dump(exclude_none=True)
+
+
+def cancel_task_payload(record: TaskRecord, request: Request) -> dict[str, Any]:
+ """取消响应保留全部逐文件终态和成功产物,不携带上传或其他详情字段。"""
+ return CancelTaskResponse(
+ task_id=record.id,
+ status=record.status,
+ finished_at=record.finished_at,
+ files=[CanceledTaskFile.model_validate(_file_payload(record, index, request)) for index in range(len(record.inputs))],
+ ).model_dump(exclude_none=True)
+
+
+def _request_schema() -> dict[str, Any]:
+ """OpenAPI 使用与运行时相同的 JSON 清单和格式约束。"""
+ schema = CreateTaskRequest.model_json_schema()
+ definitions = schema.pop("$defs", {})
+
+ def expand(value: Any) -> Any:
+ """内联局部模型引用,避免 OpenAPI 把 JSON Schema 的 $defs 当成全局路径。"""
+ if isinstance(value, dict):
+ if "$ref" in value:
+ return expand(definitions[value["$ref"].rsplit("/", 1)[-1]])
+ return {key: expand(item) for key, item in value.items()}
+ if isinstance(value, list):
+ return [expand(item) for item in value]
+ return value
+
+ return {"requestBody": {"required": True, "content": {"application/json": {"schema": expand(schema)}}}}
+
+
+def build_task_router(factory: Callable[[Request], TaskBackend]) -> APIRouter:
+ """两类部署使用同一任务协议,下载交给已有 Files 入口。"""
+ router = APIRouter(prefix="/v1", tags=["Tasks"])
+
+ @router.post("/tasks", status_code=202, response_model=CreateTaskResponse, openapi_extra=_request_schema())
+ async def create_task(request: Request) -> Response:
+ """先验证 JSON 再建立稳定任务 ID,HTTP 结束后逐文件工作独立继续。"""
+ backend = factory(request)
+ store: TaskStore = request.app.state.task_store
+ try:
+ options, inputs = await _read_submission(request)
+ record = await store.create(backend, options, inputs, lease_seconds=_retention_lease(request) or 0)
+ except BaseException:
+ backend.release_uploads(rollback=True)
+ raise
+ return JSONResponse(create_task_payload(record, request), status_code=202)
+
+ @router.get("/tasks/{task_id}", response_model=TaskDetailResponse)
+ async def get_task(task_id: str, request: Request) -> Response:
+ """读取逐文件状态与结果地址,上传链接截止时间保持不变。"""
+ store: TaskStore = request.app.state.task_store
+ record = store.lookup(task_id, factory(request).owner_scope)
+ store.update_lease(record, _retention_lease(request))
+ await store.refresh(record)
+ return JSONResponse(task_payload(record, request))
+
+ @router.put(
+ "/tasks/{task_id}/uploads/{file_id}",
+ response_model=UploadFileResponse,
+ openapi_extra={
+ "requestBody": {
+ "required": True,
+ "content": {"application/octet-stream": {"schema": {"type": "string", "format": "binary"}}},
+ }
+ },
+ )
+ async def upload_task_file(task_id: str, file_id: str, request: Request) -> Response:
+ """每条上传链接只接收一个文件,该文件就绪后立即提交执行。"""
+ store: TaskStore = request.app.state.task_store
+ record = store.lookup(task_id, factory(request).owner_scope)
+ lease_seconds = _retention_lease(request)
+ store.update_lease(record, lease_seconds)
+ try:
+ skipped = await store.upload(
+ record,
+ file_id,
+ chunks=request.stream(),
+ content_length=request.headers.get("content-length"),
+ mime_type=request.headers.get("content-type", "application/octet-stream"),
+ )
+ except ClientDisconnect:
+ invalid_task_request("Upload interrupted; retry this file on the same task", code="upload_interrupted")
+ # 大文件转传可超过租约长度;受理完成时续租,保护紧接着发布的结果。
+ store.update_lease(record, lease_seconds)
+ return JSONResponse(
+ UploadFileResponse(
+ task_id=record.id,
+ file_id=file_id,
+ status=next(state.status for item, state in zip(record.inputs, record.file_states) if item.file_id == file_id),
+ ).model_dump(),
+ headers={"Connection": "close"} if skipped and request.scope.get("http_version") == "1.1" else None,
+ )
+
+ @router.post("/tasks/{task_id}/cancel", response_model=CancelTaskResponse)
+ async def cancel_task(task_id: str, request: Request) -> Response:
+ """统一取消等待上传与正在执行的任务,不重建后台作业。"""
+ store: TaskStore = request.app.state.task_store
+ record = store.lookup(task_id, factory(request).owner_scope)
+ await store.refresh(record)
+ await store.cancel(record, wait=False)
+ return JSONResponse(cancel_task_payload(record, request))
+
+ return router
+
+
+__all__ = ["build_task_router", "cancel_task_payload", "create_task_payload", "task_payload"]
diff --git a/mineru/tasks/service.py b/mineru/tasks/service.py
new file mode 100644
index 00000000..61b30ee5
--- /dev/null
+++ b/mineru/tasks/service.py
@@ -0,0 +1,546 @@
+"""Tasks 生命周期与上传存储编排,不依赖 HTTP 请求或本地解析实现。"""
+
+from __future__ import annotations
+
+import asyncio
+import hashlib
+import logging
+import secrets
+import shutil
+import tempfile
+import time
+from collections.abc import AsyncIterator, Callable
+from dataclasses import dataclass, field
+from pathlib import Path
+
+from ..filetypes import is_flash_only_parse_extension
+from ..task_types import TaskFileState, TaskStatus
+from ..types import Tier
+from ..utils.async_utils import drain_future, run_sync
+from .contracts import ErrorDetail, TaskFileInput, TaskFileItem, UploadTaskOptions
+from .errors import ApiServerError, invalid_task_request
+from .execution import TaskBackend, TaskExecution, TaskUpload, execution_timestamp, new_execution_id
+
+TASK_UPLOAD_TTL = 3600
+TERMINAL_TASK_STATUSES = frozenset({"completed", "partial", "failed", "canceled"})
+logger = logging.getLogger(__name__)
+
+
+def resolve_task_tier(options: UploadTaskOptions, inputs: list[TaskFileInput], tiers: set[Tier], default: Tier | None) -> Tier:
+ """按现有轻量格式回退规则解析 tier,不触发 worker 选择或模型执行。"""
+ lightweight = [is_flash_only_parse_extension(item.file_name) for item in inputs]
+ if any(lightweight) and "flash" not in tiers:
+ invalid_task_request("Flash parsing is disabled, but an input requires it", param="files")
+ tier = options.tier or default
+ if tier is None and all(lightweight) and "flash" in tiers:
+ tier = "flash"
+ if tier is None:
+ invalid_task_request(
+ "No default quality tier is available; pass tier explicitly", code="quality_tier_unavailable", status_code=503
+ )
+ if tier not in tiers and not (all(lightweight) and "flash" in tiers):
+ invalid_task_request(f"Tier '{tier}' is not available", param="tier")
+ return tier
+
+
+async def file_chunks(path: Path) -> AsyncIterator[bytes]:
+ """以固定大小从磁盘读取,避免阻塞事件循环和完整文件缓冲。"""
+ with path.open("rb") as source:
+ while chunk := await run_sync(source.read, 1024 * 1024):
+ yield chunk
+
+
+async def _save_upload(
+ chunks: AsyncIterator[bytes],
+ path: Path,
+ filename: str,
+ mime_type: str,
+ *,
+ expected: TaskFileItem | None = None,
+ canceled: Callable[[], bool] | None = None,
+) -> TaskUpload:
+ """流式落盘并校验完整性;声明大小用于一致性检查,不是上传大小上限。"""
+ hasher = hashlib.sha256()
+ size = 0
+ try:
+ with path.open("wb") as target:
+ async for chunk in chunks:
+ if canceled is not None and canceled():
+ invalid_task_request("Task was canceled during upload", status_code=409, code="task_already_terminal")
+ size += len(chunk)
+ if expected is not None and expected.bytes is not None and size > expected.bytes:
+ invalid_task_request("Declared size does not match uploaded bytes", code="upload_size_mismatch")
+ await run_sync(target.write, chunk)
+ hasher.update(chunk)
+ if size == 0:
+ invalid_task_request("Empty files are not supported", param="files")
+ if expected is not None and expected.bytes is not None and size != expected.bytes:
+ invalid_task_request("Declared size does not match uploaded bytes", code="upload_size_mismatch")
+ sha = hasher.hexdigest()
+ if expected is not None and expected.sha256sum is not None and sha != expected.sha256sum:
+ invalid_task_request("SHA-256 mismatch", code="file_hash_mismatch")
+ return TaskUpload(path, filename, size, sha, mime_type)
+ except BaseException:
+ path.unlink(missing_ok=True)
+ raise
+
+
+def _normalize_task_files(inputs: list[TaskFileInput], backend: TaskBackend) -> list[TaskFileItem]:
+ """按请求顺序为每个输入预留公共产物 ID,失败时撤销已分配的预留。"""
+ normalized: list[TaskFileItem] = []
+ try:
+ for item in inputs:
+ normalized.append(TaskFileItem(**item.model_dump(), file_id=backend.reserve_file_id()))
+ return normalized
+ except BaseException:
+ for item in normalized:
+ backend.discard_file_id(item.file_id)
+ backend.forget_file_id(item.file_id)
+ raise
+
+
+@dataclass
+class TaskFileRecord:
+ """每个文件独立拥有准备引用、执行 ID 和一次性提交句柄。"""
+
+ backend: TaskBackend
+ file_id: str
+ file_name: str
+ execution_id: str = field(default_factory=new_execution_id)
+ status: TaskFileState = "waiting-file"
+ upload: TaskUpload | None = None
+ upload_lock: asyncio.Lock = field(default_factory=asyncio.Lock)
+ upload_active: bool = False
+ expired: bool = False
+ execution: TaskExecution | None = None
+ operation: asyncio.Task[None] | None = None
+ submitted: bool = False
+ finished: asyncio.Event = field(default_factory=asyncio.Event)
+ lease_refresh_at: float = 0
+ error: ErrorDetail | None = None
+
+
+@dataclass
+class TaskRecord:
+ """父 Task 独占批次聚合,独立单文件执行继续使用现有解析并发。"""
+
+ id: str
+ backend: TaskBackend
+ options: UploadTaskOptions
+ inputs: list[TaskFileItem]
+ directory: Path
+ expires_at: int
+ file_states: list[TaskFileRecord]
+ created_at: str = field(default_factory=execution_timestamp)
+ status: TaskStatus = "waiting-file"
+ refresh_lock: asyncio.Lock = field(default_factory=asyncio.Lock)
+ changed: asyncio.Event = field(default_factory=asyncio.Event)
+ operation: asyncio.Task[None] | None = None
+ finished_at: str | None = None
+ finished_epoch: float | None = None
+ retention_epoch: float | None = None
+ lease_expires_at: float = 0
+ error: ErrorDetail | None = None
+
+
+class TaskStore:
+ """管理固定上传窗口和逐文件执行,不建立第二个解析队列。"""
+
+ def __init__(self, retention_seconds: int) -> None:
+ """只在创建应用时分配暂存目录,模块导入不产生副作用。"""
+ self.retention_seconds = retention_seconds
+ self._temporary = tempfile.TemporaryDirectory(prefix="mineru-tasks-")
+ self._root = Path(self._temporary.name)
+ self._records: dict[str, TaskRecord] = {}
+ self.storage_lock = asyncio.Lock()
+ self._collector: asyncio.Task[None] | None = None
+ self._closing = False
+
+ def start(self) -> None:
+ """启动过期扫描,关闭正常保留期也仍然执行上传截止时间。"""
+ if self._collector is None and not self._closing:
+ self._collector = asyncio.create_task(self._collect_loop(), name="mineru-task-retention")
+
+ async def _collect_loop(self) -> None:
+ """定期清理过期元数据;上传与状态请求也执行同一截止检查。"""
+ while True:
+ await asyncio.sleep(60)
+ self.collect_expired()
+
+ def _aggregate(self, record: TaskRecord) -> None:
+ """按运行、排队、等待的优先级聚合,所有文件终结后再发布成功或部分成功。"""
+ if record.status == "canceled":
+ return
+ previous = record.status
+ states = [file.status for file in record.file_states]
+ if "running" in states:
+ record.status = "running"
+ elif "pending" in states:
+ record.status = "pending"
+ elif "waiting-file" in states:
+ record.status = "waiting-file"
+ else:
+ completed = states.count("completed")
+ record.status = "completed" if completed == len(states) else ("partial" if completed else "failed")
+ if record.status in TERMINAL_TASK_STATUSES and record.finished_epoch is None:
+ record.finished_at = execution_timestamp()
+ record.finished_epoch = time.time()
+ if record.status != previous:
+ record.changed.set()
+
+ def _expire_uploads(self, record: TaskRecord) -> None:
+ """只终结尚未受理上传的文件;截止前已经受理的流允许继续完成。"""
+ if record.expires_at > time.time() or record.status == "canceled":
+ return
+ for item, file in zip(record.inputs, record.file_states):
+ if item.url is None and file.upload is None and file.status == "waiting-file" and not file.upload_active:
+ file.expired = True
+ file.status = "failed"
+ file.error = ErrorDetail(type="invalid_request_error", code="upload_expired", message="Upload expired")
+ self._aggregate(record)
+
+ def collect_expired(self) -> None:
+ """上传到期按文件处理,记录仅在清理结束并超过父任务保留期后移除。"""
+ now = time.time()
+ for task_id, record in tuple(self._records.items()):
+ self._expire_uploads(record)
+ if (
+ self.retention_seconds
+ and record.retention_epoch is not None
+ and record.retention_epoch + self.retention_seconds <= now
+ and record.operation is not None
+ and record.operation.done()
+ ):
+ self._records.pop(task_id, None)
+ for item in record.inputs:
+ record.backend.forget_file_id(item.file_id)
+ shutil.rmtree(record.directory, ignore_errors=True)
+
+ def lookup(self, task_id: str, owner_scope: str) -> TaskRecord:
+ """未知与越权任务使用相同错误,避免跨调用方探测。"""
+ self.collect_expired()
+ record = self._records.get(task_id)
+ if record is None or record.backend.owner_scope != owner_scope:
+ invalid_task_request(f"Task {task_id} not found", status_code=404, code="task_not_found")
+ return record
+
+ def update_lease(self, record: TaskRecord, seconds: int | None) -> None:
+ """续租只保护结果保留期,从不延长上传窗口,也不复活已释放的结果。"""
+ if seconds is None:
+ return
+ if seconds and record.retention_epoch is not None:
+ invalid_task_request("Task result lease has already ended", status_code=409, code="task_lease_ended")
+ record.lease_expires_at = time.time() + seconds if seconds else 0
+ record.changed.set()
+
+ async def create(
+ self, backend: TaskBackend, options: UploadTaskOptions, inputs: list[TaskFileInput], *, lease_seconds: int = 0
+ ) -> TaskRecord:
+ """先完整校验清单并固定缓存,再为各个就绪文件独立提交执行。"""
+ if self._closing:
+ invalid_task_request("Server is shutting down", status_code=503, code="server_shutting_down")
+ await backend.validate(options, inputs)
+ inputs = _normalize_task_files(inputs, backend)
+ task_id = new_execution_id()
+ record = TaskRecord(
+ task_id,
+ backend,
+ options,
+ inputs,
+ self._root / task_id,
+ int(time.time()) + TASK_UPLOAD_TTL,
+ [TaskFileRecord(backend.fork(), file_id=item.file_id, file_name=item.file_name) for item in inputs],
+ )
+ self.update_lease(record, lease_seconds)
+ try:
+ for index, item in enumerate(inputs):
+ file = record.file_states[index]
+ if item.url is None and item.sha256sum:
+ upload = await file.backend.find_upload(item.sha256sum, item.file_name)
+ if upload is not None:
+ if upload.bytes != item.bytes:
+ invalid_task_request("Declared size does not match cached bytes", code="upload_size_mismatch")
+ file.upload = await file.backend.retain_upload(upload)
+ self._records[task_id] = record
+ self.start()
+ for index, item in enumerate(inputs):
+ if item.url is not None or record.file_states[index].upload is not None:
+ self._start_file(record, index)
+ record.operation = asyncio.create_task(self._monitor(record), name=f"mineru-task-{task_id}")
+ return record
+ except BaseException:
+ for item in inputs:
+ backend.discard_file_id(item.file_id)
+ backend.forget_file_id(item.file_id)
+ for file in record.file_states:
+ file.backend.release_uploads(rollback=True)
+ raise
+
+ def _start_file(self, record: TaskRecord, index: int) -> None:
+ """事件循环内同步登记提交句柄,重复 PUT 不能创建第二个子作业。"""
+ file = record.file_states[index]
+ if file.operation is not None or file.status != "waiting-file" or record.status == "canceled":
+ return
+ file.status = "pending" if file.upload is not None else "waiting-file"
+ record.changed.set()
+ file.operation = asyncio.create_task(self._dispatch_file(record, index), name=f"mineru-task-file-{file.execution_id}")
+ self._aggregate(record)
+
+ def _apply_execution(self, record: TaskRecord, file: TaskFileRecord, execution: TaskExecution) -> None:
+ """只接受单文件执行快照,取消和已发布终态不被迟到响应覆盖。"""
+ if file.status in TERMINAL_TASK_STATUSES:
+ return
+ status = execution.status
+ if status not in {"waiting-file", "pending", "running", "completed", "failed", "canceled"}:
+ invalid_task_request("Invalid file status from execution backend", code="invalid_response", status_code=502)
+ if status == "completed" and execution.result_file_id is None:
+ invalid_task_request("Completed file has no result", code="invalid_response", status_code=502)
+ if (
+ execution.execution_id != file.execution_id
+ or execution.file_id != file.file_id
+ or execution.file_name != file.file_name
+ ):
+ invalid_task_request("Execution file identity changed", code="invalid_response", status_code=502)
+ if execution.result_file_id is not None and execution.result_file_id != file.file_id:
+ invalid_task_request("Execution result identity changed", code="invalid_response", status_code=502)
+ if status != file.status:
+ record.changed.set()
+ file.execution = execution
+ file.status = status
+ file.error = execution.error
+ if status in TERMINAL_TASK_STATUSES:
+ file.finished.set()
+ file.lease_refresh_at = time.monotonic() + 30
+ self._aggregate(record)
+
+ async def refresh(self, record: TaskRecord) -> None:
+ """并发刷新独立文件;成功文件仍续租 worker 结果直到父任务收尾。"""
+ async with record.refresh_lock:
+ self._expire_uploads(record)
+ if record.status in TERMINAL_TASK_STATUSES:
+ return
+ targets = [
+ file
+ for file in record.file_states
+ if file.submitted
+ and file.status not in {"failed", "canceled"}
+ and (file.status != "completed" or file.lease_refresh_at <= time.monotonic())
+ ]
+ results = await asyncio.gather(*(file.backend.get(file.execution_id) for file in targets), return_exceptions=True)
+ for file, result in zip(targets, results):
+ if isinstance(result, BaseException):
+ if (
+ isinstance(result, ApiServerError)
+ and result.status_code in {408, 429, 500, 502, 503, 504}
+ and result.error.code not in {"invalid_response", "invalid_upstream_response"}
+ ):
+ logger.warning("Task file refresh failed: %s: %s", file.execution_id, result.error.code)
+ continue
+ if file.status not in TERMINAL_TASK_STATUSES:
+ file.status = "failed"
+ file.backend.discard_file_id(file.file_id)
+ file.error = (
+ result.error
+ if isinstance(result, ApiServerError)
+ else ErrorDetail(type="api_error", code="task_refresh_failed", message=str(result))
+ )
+ if file.operation is not None:
+ file.operation.cancel()
+ continue
+ try:
+ self._apply_execution(record, file, result)
+ if file.status == "completed":
+ file.lease_refresh_at = time.monotonic() + 30
+ except ApiServerError as exc:
+ file.status = "failed"
+ file.backend.discard_file_id(file.file_id)
+ file.error = exc.error
+ if file.operation is not None:
+ file.operation.cancel()
+ self._aggregate(record)
+
+ async def _dispatch_file(self, record: TaskRecord, index: int) -> None:
+ """每个文件独立提交,失败或取消只清理自己的作业与源文件租约。"""
+ file = record.file_states[index]
+ try:
+ execution = await file.backend.submit(file.execution_id, record.options, record.inputs[index], file.upload)
+ file.submitted = True
+ self._apply_execution(record, file, execution)
+ file.backend.release_uploads(rollback=False)
+ if file.status not in TERMINAL_TASK_STATUSES:
+ await file.finished.wait()
+ except asyncio.CancelledError:
+ if file.status != "completed":
+ file.backend.discard_file_id(file.file_id)
+ try:
+ await drain_future(asyncio.create_task(file.backend.cancel(file.execution_id)))
+ except Exception:
+ logger.exception("Task file cancellation failed: %s", file.execution_id)
+ if file.status not in TERMINAL_TASK_STATUSES:
+ file.status = "canceled"
+ except Exception as exc:
+ if file.status not in TERMINAL_TASK_STATUSES:
+ file.status = "failed"
+ file.error = (
+ exc.error
+ if isinstance(exc, ApiServerError)
+ else ErrorDetail(type="api_error", code="task_submission_failed", message=str(exc))
+ )
+ if file.status != "completed":
+ file.backend.discard_file_id(file.file_id)
+ try:
+ await file.backend.cancel(file.execution_id)
+ except Exception:
+ logger.exception("Task file failure cleanup failed: %s", file.execution_id)
+ finally:
+ file.finished.set()
+ record.changed.set()
+ file.backend.release_uploads(rollback=not file.submitted)
+ self._aggregate(record)
+
+ async def _monitor(self, record: TaskRecord) -> None:
+ """应用持有父任务监控;HTTP 断连不影响执行,所有子文件清理后才开始保留期。"""
+ try:
+ while record.status not in TERMINAL_TASK_STATUSES:
+ record.changed.clear()
+ await self.refresh(record)
+ if record.status not in TERMINAL_TASK_STATUSES:
+ await self._wait_for_change(record)
+ operations = [file.operation for file in record.file_states if file.operation is not None]
+ if operations:
+ await asyncio.gather(*operations, return_exceptions=True)
+ for item, file in zip(record.inputs, record.file_states):
+ if file.status != "completed":
+ file.backend.discard_file_id(item.file_id)
+ while any(file.upload_active for file in record.file_states) or record.lease_expires_at > time.time():
+ record.changed.clear()
+ await self._wait_for_change(record)
+ finished = time.time()
+ for file in record.file_states:
+ await file.backend.release_result(file.execution_id, finished)
+ file.backend.release_uploads(rollback=not file.submitted)
+ record.retention_epoch = finished
+ except Exception:
+ logger.exception("Task result cleanup failed: %s", record.id)
+ finally:
+ if not any(file.upload_active for file in record.file_states):
+ shutil.rmtree(record.directory, ignore_errors=True)
+
+ async def _wait_for_change(self, record: TaskRecord) -> None:
+ """状态变更即时唤醒;没有事件时每秒检查执行与到期边界。"""
+ try:
+ await asyncio.wait_for(record.changed.wait(), timeout=1)
+ except asyncio.TimeoutError:
+ pass
+
+ async def upload(
+ self,
+ record: TaskRecord,
+ file_id: str,
+ *,
+ chunks: AsyncIterator[bytes],
+ content_length: str | None,
+ mime_type: str,
+ ) -> bool:
+ """在接收主体前检查固定期限,已受理流不因父任务状态变化中止。"""
+ index = next((i for i, item in enumerate(record.inputs) if item.file_id == file_id), None)
+ if index is None:
+ invalid_task_request("Unknown file_id", status_code=404, code="upload_not_found")
+ expected = record.inputs[index]
+ if expected.url is not None:
+ invalid_task_request("URL inputs do not accept uploads")
+ file = record.file_states[index]
+ async with file.upload_lock:
+ if record.expires_at <= time.time():
+ self._expire_uploads(record)
+ invalid_task_request("Upload link expired", status_code=410, code="upload_expired")
+ if record.status == "canceled":
+ invalid_task_request("Task was canceled", status_code=409, code="task_already_terminal")
+ if file.upload is not None:
+ return True
+ if file.status != "waiting-file":
+ invalid_task_request("File no longer accepts uploads", status_code=409, code="file_already_terminal")
+ length = content_length
+ if length is not None:
+ try:
+ declared = int(length)
+ except ValueError:
+ invalid_task_request("Invalid Content-Length", code="upload_size_mismatch")
+ if declared != expected.bytes:
+ invalid_task_request("Content-Length does not match declared file size", code="upload_size_mismatch")
+ file.upload_active = True
+ record.directory.mkdir(parents=True, exist_ok=True)
+ path = record.directory / secrets.token_hex(16)
+
+ def was_canceled() -> bool:
+ """只在本文件被取消时中止已受理上传,不把上传截止时间当作传输超时。"""
+ return record.status == "canceled" or file.status == "canceled"
+
+ try:
+ upload = await _save_upload(
+ chunks,
+ path,
+ expected.file_name,
+ mime_type,
+ expected=expected,
+ canceled=was_canceled,
+ )
+ if was_canceled():
+ invalid_task_request("Task was canceled during upload", status_code=409, code="task_already_terminal")
+ retained = await file.backend.retain_upload(upload)
+ if was_canceled():
+ file.backend.release_uploads(rollback=True)
+ invalid_task_request("Task was canceled during upload", status_code=409, code="task_already_terminal")
+ file.upload = retained
+ self._start_file(record, index)
+ return False
+ finally:
+ path.unlink(missing_ok=True)
+ file.upload_active = False
+ record.changed.set()
+ self._expire_uploads(record)
+
+ async def cancel(self, record: TaskRecord, *, wait: bool = True) -> None:
+ """保留已完成结果,只取消未完成文件并等待真实子作业清理。"""
+ if record.status in TERMINAL_TASK_STATUSES:
+ invalid_task_request(f"Task is {record.status}", status_code=409, code="task_already_terminal")
+ record.status = "canceled"
+ record.finished_at = execution_timestamp()
+ record.finished_epoch = time.time()
+ record.changed.set()
+ for item, file in zip(record.inputs, record.file_states):
+ if file.status not in TERMINAL_TASK_STATUSES:
+ file.status = "canceled"
+ file.backend.discard_file_id(item.file_id)
+ if file.operation is not None:
+ file.operation.cancel()
+ if wait and record.operation is not None:
+ record.lease_expires_at = 0
+ record.changed.set()
+ await drain_future(record.operation)
+
+ async def shutdown(self) -> None:
+ """停止接单并排空全部文件执行和结果租约,最后释放临时目录。"""
+ self._closing = True
+ if self._collector is not None:
+ self._collector.cancel()
+ await asyncio.gather(self._collector, return_exceptions=True)
+ for record in tuple(self._records.values()):
+ record.lease_expires_at = 0
+ if record.status not in TERMINAL_TASK_STATUSES:
+ await self.cancel(record, wait=False)
+ await asyncio.gather(
+ *(record.operation for record in self._records.values() if record.operation is not None), return_exceptions=True
+ )
+ self._temporary.cleanup()
+
+
+__all__ = [
+ "TASK_UPLOAD_TTL",
+ "TERMINAL_TASK_STATUSES",
+ "TaskFileRecord",
+ "TaskRecord",
+ "TaskStore",
+ "file_chunks",
+ "resolve_task_tier",
+]
diff --git a/mineru/utils/logger.py b/mineru/utils/logger.py
index 38373648..72670deb 100644
--- a/mineru/utils/logger.py
+++ b/mineru/utils/logger.py
@@ -26,7 +26,7 @@ def _active_managed_sink_ids() -> set[int]:
def configure_global_log_level(level: LogLevel | str | None = None) -> LogLevel:
"""配置当前进程的 Loguru 默认 stderr sink,并返回归一化后的级别。
- ``level`` 为 ``None`` 时使用 ``config.log.level``;显式传入后作为进程级
+ ``level`` 为 ``None`` 时使用 ``get_config().log.level``;显式传入后作为进程级
局部覆盖,后续默认调用不会把它改回全局配置。函数只移除 Loguru 初始
sink 和 MinerU 自己创建的 sink,不会影响宿主程序显式添加的自定义 sink。
级别同时写入 ``LOGURU_LEVEL`` 环境变量(覆盖外部预设),使之后 spawn 的
@@ -35,13 +35,13 @@ def configure_global_log_level(level: LogLevel | str | None = None) -> LogLevel:
"""
from loguru import logger
- from ..config import GlobalLogConfig, config
+ from ..config import GlobalLogConfig, get_config
global _configured_level, _explicit_level
with _lock:
if level is None:
- normalized = _explicit_level or config.log.level
+ normalized = _explicit_level or get_config().log.level
else:
normalized = GlobalLogConfig(level=level).level
_explicit_level = normalized
diff --git a/scripts/benchmark_async_analysis.py b/scripts/benchmark_async_analysis.py
index 4b1a7139..89f4280b 100644
--- a/scripts/benchmark_async_analysis.py
+++ b/scripts/benchmark_async_analysis.py
@@ -60,10 +60,10 @@ def save_result(result: Any, path: Path, directory: Path) -> None:
async def measure(args: argparse.Namespace, concurrency: int) -> dict[str, Any]:
"""以固定并发解析同一组文件,同时采集延迟、内存、线程和事件循环响应。"""
- from mineru.config import config
+ from mineru.config import get_config
from mineru.parser import parse, parse_async
- settings = config.model.vlm.model_copy(deep=True)
+ settings = get_config().model.vlm.model_copy(deep=True)
if args.server_url:
settings.server_url = args.server_url
if args.engine:
@@ -71,8 +71,8 @@ async def measure(args: argparse.Namespace, concurrency: int) -> dict[str, Any]:
settings.max_concurrency = args.request_limit
settings = type(settings).model_validate(settings.model_dump())
if not args.llm:
- config.llm_aided.features.title_leveling = False
- config.llm_aided.features.cross_page_table_cell_merge = False
+ get_config().llm_aided.features.title_leveling = False
+ get_config().llm_aided.features.cross_page_table_cell_merge = False
gate = asyncio.Semaphore(concurrency)
process = psutil.Process()
lag_ms: list[float] = []
@@ -212,7 +212,7 @@ def main() -> None:
args.files = [path.resolve(strict=True) for path in args.files]
import mineru
import mineru_vl_utils
- from mineru.config import config
+ from mineru.config import get_config
report = {
"source": str(Path(mineru.__file__).resolve()),
@@ -221,10 +221,10 @@ def main() -> None:
"tier": args.tier,
"ocr_mode": args.ocr_mode,
"request_limit": args.request_limit,
- "remote_vlm": bool(args.server_url or config.model.vlm.server_url),
+ "remote_vlm": bool(args.server_url or get_config().model.vlm.server_url),
"vl_utils_source_version": mineru_vl_utils.__version__,
"vl_utils_source": str(Path(mineru_vl_utils.__file__).resolve()),
- "small_backend": config.model.small_backend,
+ "small_backend": get_config().model.small_backend,
"device_mode": os.getenv("MINERU_DEVICE_MODE", "auto"),
"onnx_intra_threads": os.getenv("MINERU_INTRA_OP_NUM_THREADS"),
"onnx_inter_threads": os.getenv("MINERU_INTER_OP_NUM_THREADS"),
diff --git a/scripts/validate_detection_batching.py b/scripts/validate_detection_batching.py
index 3f29d792..da892205 100644
--- a/scripts/validate_detection_batching.py
+++ b/scripts/validate_detection_batching.py
@@ -220,7 +220,7 @@ def integration(args: argparse.Namespace) -> dict[str, Any]:
def main() -> None:
"""显式选择组件对照或真实 PDF 集成验证,并保存配置与内存指标。"""
- from mineru.config import config
+ from mineru.config import get_config
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("kind", choices=["components", "parse"])
@@ -233,9 +233,13 @@ def main() -> None:
args = parser.parse_args()
args.small_backend, args.tier, args.rounds = "light", "basic", 1
args.output_dir.mkdir(parents=True, exist_ok=True)
- config.model.base_dir, config.model.source, config.model.small_backend = str(args.models_dir.resolve()), "local", "onnx"
- config.llm_aided.features.title_leveling = False
- config.llm_aided.features.cross_page_table_cell_merge = False
+ get_config().model.base_dir, get_config().model.source, get_config().model.small_backend = (
+ str(args.models_dir.resolve()),
+ "local",
+ "onnx",
+ )
+ get_config().llm_aided.features.title_leveling = False
+ get_config().llm_aided.features.cross_page_table_cell_merge = False
report = components(args) if args.kind == "components" else integration(args)
report["peak_rss_bytes"] = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss * (1 if sys.platform == "darwin" else 1024)
report["environment"] = {"python": sys.version.split()[0], "onnxruntime": version("onnxruntime")}
diff --git a/scripts/validate_onnx_models.py b/scripts/validate_onnx_models.py
index 47ef16f3..52b65224 100644
--- a/scripts/validate_onnx_models.py
+++ b/scripts/validate_onnx_models.py
@@ -261,7 +261,7 @@ def parse_documents(args: argparse.Namespace) -> dict[str, Any]:
def main() -> None:
"""配置隔离的模型目录和本地来源,运行指定验证场景并写出报告。"""
- from mineru.config import config
+ from mineru.config import get_config
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("kind", choices=["ocr", "seal", "mfr", "layout", "parse"])
@@ -275,12 +275,12 @@ def main() -> None:
parser.add_argument("--rounds", type=int, default=1)
args = parser.parse_args()
args.output_dir.mkdir(parents=True, exist_ok=True)
- config.model.base_dir = str(args.models_dir.resolve())
- config.model.source = "local"
+ get_config().model.base_dir = str(args.models_dir.resolve())
+ get_config().model.source = "local"
# 组件对照需要同时加载 Torch;完整 Light 解析则严格使用 Light 配置。
- config.model.small_backend = args.small_backend if args.kind == "parse" else "torch"
- config.llm_aided.features.title_leveling = False
- config.llm_aided.features.cross_page_table_cell_merge = False
+ get_config().model.small_backend = args.small_backend if args.kind == "parse" else "torch"
+ get_config().llm_aided.features.title_leveling = False
+ get_config().llm_aided.features.cross_page_table_cell_merge = False
operations = {
"ocr": ocr_comparison,
"seal": ocr_comparison,
diff --git a/scripts/validate_transformers_migration.py b/scripts/validate_transformers_migration.py
index 7f13559e..56a7f776 100644
--- a/scripts/validate_transformers_migration.py
+++ b/scripts/validate_transformers_migration.py
@@ -172,12 +172,12 @@ def _operation(args: argparse.Namespace, manifest: dict[str, Any]) -> Callable[[
detections = [[{"label": "display_formula", "bbox": [0, 0, image.shape[1], image.shape[0]]}] for image in images]
return lambda: model.batch_predict(detections, images, batch_size=2)
- from mineru.config import VlmConfig, config
+ from mineru.config import VlmConfig, get_config
from mineru.parser.mineru_parser import MinerUParser
- config.model.small_backend = "torch"
- config.llm_aided.features.title_leveling = False
- config.llm_aided.features.cross_page_table_cell_merge = False
+ get_config().model.small_backend = "torch"
+ get_config().llm_aided.features.title_leveling = False
+ get_config().llm_aided.features.cross_page_table_cell_merge = False
parser = MinerUParser(tier=args.tier, parse_mode="ocr", vlm_config=VlmConfig())
def parse() -> list[dict[str, Any]]:
diff --git a/tests/benchmarks/opencv_paths.py b/tests/benchmarks/opencv_paths.py
index d90346fc..47959d56 100644
--- a/tests/benchmarks/opencv_paths.py
+++ b/tests/benchmarks/opencv_paths.py
@@ -51,13 +51,13 @@ def main() -> None:
os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["MINERU_LANG"] = "en"
args.out.mkdir(parents=True, exist_ok=True)
- from mineru.config import VlmConfig, config
+ from mineru.config import VlmConfig, get_config
import mineru
assert Path(mineru.__file__).resolve().parent == (args.root / "mineru").resolve()
- config.llm_aided.features.title_leveling = False
- config.llm_aided.features.cross_page_table_cell_merge = False
+ get_config().llm_aided.features.title_leveling = False
+ get_config().llm_aided.features.cross_page_table_cell_merge = False
def fingerprint(value: Any) -> Any:
"""完整记录模型输入数组及相关参数,不使用有损数值归一化。"""
diff --git a/tests/conftest.py b/tests/conftest.py
index 4306a01d..db3db0dd 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -31,7 +31,7 @@ def isolate_optional_llm_configuration(request: pytest.FixtureRequest, monkeypat
if request.node.get_closest_marker("remote") or request.node.get_closest_marker("full_stack"):
return
module = importlib.import_module("mineru.config")
- configuration = module.config.llm_aided.model_copy(deep=True)
+ configuration = module.get_config().llm_aided.model_copy(deep=True)
configuration.features.title_leveling = False
configuration.features.cross_page_table_cell_merge = False
- monkeypatch.setattr(module.config, "llm_aided", configuration)
+ monkeypatch.setattr(module.get_config(), "llm_aided", configuration)
diff --git a/tests/unittest/conftest.py b/tests/unittest/conftest.py
index e2991dc9..2e38e96c 100644
--- a/tests/unittest/conftest.py
+++ b/tests/unittest/conftest.py
@@ -8,8 +8,8 @@ import pytest
@pytest.fixture(autouse=True)
def isolate_onnx_device_selection(monkeypatch: pytest.MonkeyPatch) -> None:
"""Individual policy tests explicitly override this CPU default with isolated files."""
- from mineru.config import config
+ from mineru.config import get_config
from mineru.model.runtime import onnx_device
- monkeypatch.setattr(config.model, "onnx_device", "cpu")
+ monkeypatch.setattr(get_config().model, "onnx_device", "cpu")
monkeypatch.setattr(onnx_device, "_selections", {})
diff --git a/tests/unittest/task_test_utils.py b/tests/unittest/task_test_utils.py
index 5cb2bdb1..521b7cde 100644
--- a/tests/unittest/task_test_utils.py
+++ b/tests/unittest/task_test_utils.py
@@ -6,12 +6,13 @@ import tempfile
import time
import zipfile
from pathlib import Path
-from typing import Any
+from typing import Any, Literal
import httpx
from fastapi.testclient import TestClient
-from mineru.task_types import task_status_url
-from mineru.parser.api_server import FileStore, _JobRecord
+from mineru.task_types import TaskOutputFormat, task_status_url
+from mineru.types import Tier
+from mineru.parser.api_server import FileStore, _ExecutionRecord, _ExecutionRequest
from mineru.parser.base import ParseResult
from mineru.parser.task_outputs import build_task_output
@@ -33,9 +34,37 @@ def stored_source(store: FileStore, path: Path) -> dict[str, str]:
return {"type": "file_id", "file_id": file_id}
-def execution_output(store: FileStore, record: _JobRecord, name: str) -> bytes:
+def execution_request(
+ store: FileStore,
+ source: dict[str, str],
+ *,
+ tier: Tier | None = None,
+ page_range: str | None = None,
+ ocr_mode: Literal["auto", "txt", "ocr"] = "auto",
+ output_formats: list[TaskOutputFormat] | None = None,
+ file_name: str | None = None,
+) -> _ExecutionRequest:
+ """为单文件执行测试分配结果身份,源 ID 与结果 ID 始终独立。"""
+ if file_name is None:
+ file_name = (
+ store.get_file(source["file_id"]).filename
+ if source["type"] == "file_id"
+ else source["url"].rsplit("/", 1)[-1].split("?")[0] or "unknown"
+ )
+ return _ExecutionRequest(
+ source=source,
+ file_id=store.reserve_file_id(),
+ file_name=file_name,
+ tier=tier,
+ page_range=page_range,
+ ocr_mode=ocr_mode,
+ output_formats=output_formats or ["default"],
+ )
+
+
+def execution_output(store: FileStore, record: _ExecutionRecord, name: str) -> bytes:
"""从唯一 Tasks 产物读取成员,避免断言已经移除的多产物引用。"""
- with zipfile.ZipFile(io.BytesIO(store.read_file_data(record.task_result_file_id))) as archive:
+ with zipfile.ZipFile(io.BytesIO(store.read_file_data(record.result_file_id))) as archive:
return archive.read(name)
diff --git a/tests/unittest/test_api_model_ownership.py b/tests/unittest/test_api_model_ownership.py
new file mode 100644
index 00000000..e17be767
--- /dev/null
+++ b/tests/unittest/test_api_model_ownership.py
@@ -0,0 +1,112 @@
+"""通过真实 Tasks API 和微型 CPU ONNX 会话验证应用模型租约。"""
+
+from __future__ import annotations
+
+import gc
+import time
+import weakref
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+import onnx
+import onnxruntime
+import pytest
+from fastapi.testclient import TestClient
+
+from mineru.config import get_config
+from mineru.integrations.docvortex import build_metadata
+from mineru.model.runtime import hybrid
+from mineru.model.runtime.ownership import runtime_document_scope
+from mineru.parser import api_server
+from mineru.parser.base import ParseResult
+from mineru.types import MiddleJson, PageInfo
+
+
+def _identity_model() -> bytes:
+ """生成无需下载权重的 ONNX 模型,让实际原生会话参与租约验证。"""
+ value = onnx.helper.make_tensor_value_info("value", onnx.TensorProto.FLOAT, [1])
+ output = onnx.helper.make_tensor_value_info("output", onnx.TensorProto.FLOAT, [1])
+ graph = onnx.helper.make_graph([onnx.helper.make_node("Identity", ["value"], ["output"])], "identity", [value], [output])
+ model = onnx.helper.make_model(graph, opset_imports=[onnx.helper.make_opsetid("", 17)])
+ model.ir_version = 9
+ return model.SerializeToString()
+
+
+def _parse_task(client: TestClient, content: bytes) -> None:
+ """经过真实创建、上传、后台执行和查询流程,确认单文件产物成功发布。"""
+ created = client.post(
+ "/v1/tasks",
+ json={"files": [{"file_name": "input.html", "bytes": len(content)}], "tier": "flash"},
+ )
+ assert created.status_code == 202, created.text
+ receipt = created.json()
+ upload = receipt["files"][0]["upload"]
+ response = client.put(upload["url"], content=content, headers=upload["headers"])
+ assert response.status_code == 200, response.text
+ deadline = time.monotonic() + 5
+ while True:
+ response = client.get(f"/v1/tasks/{receipt['task_id']}")
+ assert response.status_code == 200, response.text
+ task = response.json()
+ if task["status"] in {"completed", "failed", "canceled", "partial"}:
+ assert task["status"] == "completed", task
+ assert task["files"][0]["parse"]["tier"] == "flash"
+ assert client.get(task["files"][0]["result_url"]).status_code == 200
+ return
+ assert time.monotonic() < deadline, task
+ time.sleep(0.01)
+
+
+def test_api_apps_share_native_sessions_until_last_owner_closes(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ """一个应用关闭后其它应用仍可推理,最后应用退出后会话可回收且后续重新创建。"""
+ monkeypatch.setattr(hybrid.AtomModelSingleton, "_models", {})
+ monkeypatch.setattr(hybrid.HybridLocalModelContextSingleton, "_models", {})
+ monkeypatch.setattr(get_config().model, "small_backend", "onnx")
+ model_bytes = _identity_model()
+ sessions: list[weakref.ReferenceType[onnxruntime.InferenceSession]] = []
+
+ def create_session(model_name: str, **_options: Any) -> onnxruntime.InferenceSession:
+ """替换权重装载,每个真实缓存条目仍持有可执行的原生 ORT 会话。"""
+ options = onnxruntime.SessionOptions()
+ options.intra_op_num_threads = 1
+ options.inter_op_num_threads = 1
+ session = onnxruntime.InferenceSession(model_bytes, options, providers=["CPUExecutionProvider"])
+ sessions.append(weakref.ref(session))
+ return session
+
+ async def parse_with_shared_models(_path: str, **_options: Any) -> ParseResult:
+ """在真实执行所属应用中访问上下文和惰性模型,不保留模型或文档引用。"""
+ with runtime_document_scope():
+ context = hybrid.HybridLocalModelContextSingleton().get_model()
+ output = context.mfr_model.run(None, {"value": np.array([7], dtype=np.float32)})
+ np.testing.assert_array_equal(output[0], [7])
+ return ParseResult(
+ MiddleJson(
+ pages=[PageInfo(page_idx=0)],
+ is_full_document=True,
+ metadata={"file_suffix": "html", "producer": {"name": "mineru", "version": "test"}},
+ extensions=build_metadata(effort="flash", parse_mode="txt"),
+ )
+ )
+
+ monkeypatch.setattr(hybrid, "atom_model_init", create_session)
+ monkeypatch.setattr(api_server, "parse_async", parse_with_shared_models)
+
+ with TestClient(api_server.create_app(upload_dir=str(tmp_path / "second"), tier="flash")) as second:
+ with TestClient(api_server.create_app(upload_dir=str(tmp_path / "first"), tier="flash")) as first:
+ _parse_task(first, b"<p>first</p>")
+ _parse_task(second, b"<p>second</p>")
+ assert len(sessions) == 3 # OCR、Layout 与惰性 MFR 只创建一份。
+ gc.collect()
+ assert all(reference() is not None for reference in sessions)
+ _parse_task(second, b"<p>still active</p>")
+ assert len(sessions) == 3
+ gc.collect()
+ assert all(reference() is None for reference in sessions)
+
+ with TestClient(api_server.create_app(upload_dir=str(tmp_path / "replacement"), tier="flash")) as replacement:
+ _parse_task(replacement, b"<p>replacement</p>")
+ assert len(sessions) == 6
+ gc.collect()
+ assert all(reference() is None for reference in sessions)
diff --git a/tests/unittest/test_api_server_logging.py b/tests/unittest/test_api_server_logging.py
index 22788305..057ba6ae 100644
--- a/tests/unittest/test_api_server_logging.py
+++ b/tests/unittest/test_api_server_logging.py
@@ -40,7 +40,7 @@ def test_log_level_reaches_server(monkeypatch: pytest.MonkeyPatch, entrypoint: s
expected_server_level = (level or "debug").lower()
configure_global = MagicMock(return_value="debug")
server = SimpleNamespace(run=MagicMock(), should_exit=False)
- monkeypatch.setattr(api_server.mineru_config.log, "level", "debug")
+ monkeypatch.setattr(api_server.get_config().log, "level", "debug")
monkeypatch.setattr(api_server, "create_app", create_app)
monkeypatch.setattr(api_server, "configure_global_log_level", configure_global)
monkeypatch.setattr(kit_api, "configure_global_log_level", MagicMock(return_value="debug"))
diff --git a/tests/unittest/test_async_job_lifecycle.py b/tests/unittest/test_async_job_lifecycle.py
index 81a86280..8e54919a 100644
--- a/tests/unittest/test_async_job_lifecycle.py
+++ b/tests/unittest/test_async_job_lifecycle.py
@@ -1,5 +1,6 @@
"""验证解析 API 任务取消、排队和关闭的真实生命周期。"""
+from task_test_utils import execution_request
from task_test_utils import stored_source
@@ -8,7 +9,21 @@ from pathlib import Path
import pytest
-from mineru.parser.api_server import ApiServerError, JobStore, _JobRecord
+from mineru.parser.api_server import ExecutionStore, _ExecutionRecord, _ExecutionRequest
+from mineru.tasks.errors import ApiServerError
+
+
+def _record(execution_id: str) -> _ExecutionRecord:
+ """为纯生命周期场景构造一个具有独立公开身份的单文件执行。"""
+ return _ExecutionRecord(
+ execution_id,
+ _ExecutionRequest(
+ source={"type": "url", "url": f"https://example.test/{execution_id}.pdf"},
+ file_id=f"file-{execution_id}",
+ file_name=f"{execution_id}.pdf",
+ tier="flash",
+ ),
+ )
def test_queued_job_cancel_never_starts_and_does_not_cancel_other_job() -> None:
@@ -16,9 +31,9 @@ def test_queued_job_cancel_never_starts_and_does_not_cancel_other_job() -> None:
async def run() -> None:
"""在一个并发名额下验证队列取消和状态不可被覆盖。"""
- store = JobStore(concurrency=1)
- first, queued = _JobRecord("first"), _JobRecord("queued")
- store._jobs = {first.id: first, queued.id: queued}
+ store = ExecutionStore(concurrency=1)
+ first, queued = _record("first"), _record("queued")
+ store._executions = {first.id: first, queued.id: queued}
entered, release = asyncio.Event(), asyncio.Event()
queued_started = False
@@ -57,9 +72,9 @@ def test_running_cancel_keeps_slot_until_cleanup_and_shutdown_waits() -> None:
async def run() -> None:
"""用受控清理阶段验证取消、排队和服务关闭顺序。"""
- store = JobStore(1)
- first, second = _JobRecord("first"), _JobRecord("second")
- store._jobs = {first.id: first, second.id: second}
+ store = ExecutionStore(1)
+ first, second = _record("first"), _record("second")
+ store._executions = {first.id: first, second.id: second}
entered, cleaning, finish_cleanup, second_started = (asyncio.Event() for _ in range(4))
async def work() -> None:
@@ -150,7 +165,7 @@ def test_late_parser_result_is_not_published_after_cancel(tmp_path: Path, monkey
return {"pages": []}
async def run() -> None:
- """取消实际 JobStore 任务,并通过真实文件处理逻辑检查结果发布边界。"""
+ """取消实际 ExecutionStore 任务,并通过真实文件处理逻辑检查结果发布边界。"""
entered = asyncio.Event()
async def parse(*args: object, **kwargs: object) -> Result:
@@ -162,20 +177,14 @@ def test_late_parser_result_is_not_published_after_cancel(tmp_path: Path, monkey
return Result()
monkeypatch.setattr(api_server, "parse_async", parse)
- store = JobStore()
+ store = ExecutionStore()
files = api_server.FileStore(tmp_path / "storage")
- request = api_server._ParseRequest.model_validate(
- {
- "tier": "standard",
- "output_formats": ["default"],
- "files": [{"source": stored_source(files, source)}],
- }
- )
+ request = execution_request(files, stored_source(files, source), tier="standard", output_formats=["default"])
rec = store.create(request, files)
async def work() -> None:
"""执行真实任务循环,避免只测试状态赋值。"""
- await api_server._run_job(rec, request, files, image_analysis=True)
+ await api_server._run_execution(rec, files, image_analysis=True)
store.start_task(rec, work)
await entered.wait()
@@ -183,8 +192,8 @@ def test_late_parser_result_is_not_published_after_cancel(tmp_path: Path, monkey
store.cancel(rec.id)
await task
assert rec.status == "canceled"
- assert rec.task_result_file_id is None
- assert rec.progress.completed == 0
+ assert rec.result_file_id is None
+ assert rec.parse is None
assert not rendered
await store.shutdown()
diff --git a/tests/unittest/test_backend_architecture.py b/tests/unittest/test_backend_architecture.py
index 0402b972..dc3b10f0 100644
--- a/tests/unittest/test_backend_architecture.py
+++ b/tests/unittest/test_backend_architecture.py
@@ -2,6 +2,7 @@ from __future__ import annotations
import ast
import importlib.util
+import os
import subprocess
import sys
from pathlib import Path
@@ -131,7 +132,9 @@ def test_target_python_files_have_copyright_header() -> None:
if "_internal/pytorchocr" not in path.as_posix() and path.name != "cli_parser.py"
]
offenders = [
- str(path.relative_to(_PROJECT_ROOT)) for path in paths if path.read_text(encoding="utf-8").splitlines()[0] != _COPYRIGHT_HEADER
+ str(path.relative_to(_PROJECT_ROOT))
+ for path in paths
+ if path.read_text(encoding="utf-8").splitlines()[0] != _COPYRIGHT_HEADER
]
assert not offenders
@@ -290,6 +293,41 @@ assert before_env == dict(os.environ)
assert result.returncode == 0, result.stderr
+def test_public_facades_import_without_loading_runtime_configuration(tmp_path: Path) -> None:
+ """配置路径无效时仍能导入 SDK 和服务入口,配置校验延后到实际使用。"""
+ script = """
+import importlib
+
+settings = importlib.import_module("mineru.config")
+for name in (
+ "mineru.parser",
+ "mineru.render",
+ "mineru.backend.analyze",
+ "mineru.model.download",
+ "mineru.model.vlm.client",
+ "mineru.doclib.app",
+ "mineru.cli.main",
+ "mineru.kit.main",
+ "mineru.parser.api_server",
+ "mineru.kit.router.app",
+ "mineru.kit.file_parse.app",
+ "mineru.kit.gradio.app",
+):
+ importlib.import_module(name)
+ assert settings._loaded_config is None, name
+"""
+ result = subprocess.run(
+ [sys.executable, "-c", script],
+ cwd=_PROJECT_ROOT,
+ env={**os.environ, "MINERU_CONFIG": str(tmp_path / "missing.yaml"), "MINERU_MODEL_STACK": "removed"},
+ capture_output=True,
+ text=True,
+ timeout=30,
+ check=False,
+ )
+ assert result.returncode == 0, result.stderr
+
+
def test_table_merge_public_contract_remains_callable() -> None:
"""验证本地入口和 mineru-vl-utils 使用的七个公开函数保持可调用。"""
function_names = [
diff --git a/tests/unittest/test_cli_next_command_design.py b/tests/unittest/test_cli_next_command_design.py
index a9facdba..f7d58359 100644
--- a/tests/unittest/test_cli_next_command_design.py
+++ b/tests/unittest/test_cli_next_command_design.py
@@ -2810,12 +2810,12 @@ def test_server_status_json_not_running_returns_state_json(monkeypatch: Any) ->
monkeypatch.setattr(server, "_server_running", lambda: False)
monkeypatch.setattr(server, "_doclib_lock_available", lambda: True)
monkeypatch.setattr(server, "_socket_path", lambda: "/tmp/doclib.sock")
- monkeypatch.setattr(server.config.doclib, "data_dir", "~/.mineru")
- monkeypatch.setattr(server.config.doclib.sqlite, "path", "~/.mineru/doclib.db")
- monkeypatch.setattr(server.config.doclib.log, "app_path", "~/.mineru/logs/doclib.log")
- monkeypatch.setattr(server.config.doclib.log, "access_path", "~/.mineru/logs/doclib.access.log")
- monkeypatch.setattr(server.config.doclib.log, "stdout_path", "~/.mineru/logs/doclib.stdout.log")
- monkeypatch.setattr(server.config.doclib.log, "stderr_path", "~/.mineru/logs/doclib.stderr.log")
+ monkeypatch.setattr(server.get_config().doclib, "data_dir", "~/.mineru")
+ monkeypatch.setattr(server.get_config().doclib.sqlite, "path", "~/.mineru/doclib.db")
+ monkeypatch.setattr(server.get_config().doclib.log, "app_path", "~/.mineru/logs/doclib.log")
+ monkeypatch.setattr(server.get_config().doclib.log, "access_path", "~/.mineru/logs/doclib.access.log")
+ monkeypatch.setattr(server.get_config().doclib.log, "stdout_path", "~/.mineru/logs/doclib.stdout.log")
+ monkeypatch.setattr(server.get_config().doclib.log, "stderr_path", "~/.mineru/logs/doclib.stderr.log")
result = runner.invoke(app, ["server", "status", "--json"])
diff --git a/tests/unittest/test_config_loading.py b/tests/unittest/test_config_loading.py
new file mode 100644
index 00000000..5a4a68ea
--- /dev/null
+++ b/tests/unittest/test_config_loading.py
@@ -0,0 +1,156 @@
+"""验证配置装载时机、默认路径及进程缓存边界。"""
+
+from __future__ import annotations
+
+import importlib
+import os
+import subprocess
+import sys
+import threading
+from concurrent.futures import ThreadPoolExecutor
+from pathlib import Path
+
+import pytest
+
+from mineru.config import Config, LoadedConfig
+
+config_module = importlib.import_module("mineru.config")
+
+
[email protected](autouse=True)
+def isolate_default_configuration(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ """每项装载测试使用独立环境和空缓存,结束后恢复调用方原有默认配置。"""
+ for name in tuple(os.environ):
+ if name.startswith("MINERU_"):
+ monkeypatch.delenv(name)
+ monkeypatch.setenv("MINERU_HOME", str(tmp_path))
+ monkeypatch.setattr(config_module, "_loaded_config", None)
+
+
+def test_import_does_not_open_configuration_file(tmp_path: Path) -> None:
+ """通过独立进程审计文件读取,保证导入配置模块不会装载配置文件。"""
+ config_path = tmp_path / "config.yaml"
+ config_path.write_text("model:\n source: local\n", encoding="utf-8")
+ script = """
+import importlib
+import sys
+
+def reject_configuration_read(event: str, args: tuple[object, ...]) -> None:
+ \"\"\"配置文件只允许显式装载时读取。\"\"\"
+ if event == "open" and args[0] == sys.argv[1]:
+ raise AssertionError("configuration file read during import")
+
+sys.addaudithook(reject_configuration_read)
+module = importlib.import_module("mineru.config")
+assert module._loaded_config is None
+assert not hasattr(module, "config")
+"""
+ env = {**os.environ, "MINERU_CONFIG": str(config_path), "MINERU_MODEL_STACK": "removed"}
+ completed = subprocess.run(
+ [sys.executable, "-c", script, str(config_path)],
+ cwd=Path(__file__).resolve().parents[2],
+ env=env,
+ capture_output=True,
+ text=True,
+ check=False,
+ )
+ assert completed.returncode == 0, completed.stderr
+
+
[email protected]("accessor", ["get_config", "get_config_file_path", "get_config_file_exists", "get_config_source"])
+def test_first_configuration_access_validates_missing_file(
+ accessor: str, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """任何默认配置或来源查询首次执行时都报告显式配置路径错误。"""
+ monkeypatch.setenv("MINERU_CONFIG", str(tmp_path / "missing.yaml"))
+ arguments = ("model.source",) if accessor == "get_config_source" else ()
+ with pytest.raises(FileNotFoundError, match="missing.yaml"):
+ getattr(config_module, accessor)(*arguments)
+ assert config_module._loaded_config is None
+
+
+def test_failed_default_load_can_retry_after_configuration_is_repaired(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ """初始化失败不缓存半成品,修复配置后同一进程可以重新装载。"""
+ config_path = tmp_path / "config.yaml"
+ monkeypatch.setenv("MINERU_CONFIG", str(config_path))
+ with pytest.raises(FileNotFoundError):
+ config_module.get_config()
+ config_path.write_text("log:\n level: debug\n", encoding="utf-8")
+ assert config_module.get_config().log.level == "debug"
+
+
+def test_default_paths_use_environment_at_construction(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ """所有依赖 MINERU_HOME 的默认路径在构造配置时解析,且不修改既有实例。"""
+ first_home, second_home = tmp_path / "first", tmp_path / "second"
+ monkeypatch.setenv("MINERU_HOME", str(first_home))
+ first = Config()
+ monkeypatch.setenv("MINERU_HOME", str(second_home))
+ second = Config()
+
+ for configuration, root in ((first, first_home), (second, second_home)):
+ assert Path(configuration.doclib.uds.path) == root / "doclib.sock"
+ assert Path(configuration.doclib.log.dir) == root / "logs"
+ assert Path(configuration.doclib.sqlite.path) == root / "doclib.db"
+ assert Path(configuration.doclib.endpoint_path) == root / "doclib.endpoint.json"
+ assert Path(configuration.doclib.data_dir) == root / "doclib"
+ assert Path(configuration.model.base_dir) == root / "models"
+ assert config_module._loaded_config is None
+
+
+def test_independent_loads_do_not_replace_default_configuration(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ """独立装载读取当前文件和环境,同时保持默认实例及其配置来源稳定。"""
+ config_path = tmp_path / "config.yaml"
+ config_path.write_text("model:\n source: local\n", encoding="utf-8")
+ default = config_module.get_config()
+ assert default.model.source == "local"
+ assert config_module.get_config_source("model.source") == "file"
+
+ config_path.write_text("model:\n source: huggingface\n", encoding="utf-8")
+ monkeypatch.setenv("MINERU_MODEL_SOURCE", "modelscope")
+ loaded = config_module.load_config()
+ assert loaded.config.model.source == "modelscope"
+ assert loaded.sources[("model", "source")] == "env"
+ assert loaded.config_file == str(config_path)
+ assert loaded.config_file_exists is True
+ assert config_module.load_config().config is not loaded.config
+ assert config_module.get_config() is default
+ assert default.model.source == "local"
+ assert config_module.get_config_source("model.source") == "file"
+
+
+def test_configuration_queries_share_first_loaded_snapshot(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ """路径查询触发的首次装载与后续配置及来源查询共享同一快照。"""
+ config_path = tmp_path / "config.yaml"
+ config_path.write_text("model:\n source: local\n", encoding="utf-8")
+ assert config_module.get_config_file_path() == str(config_path)
+ monkeypatch.setenv("MINERU_CONFIG", str(tmp_path / "missing.yaml"))
+ assert config_module.get_config().model.source == "local"
+ assert config_module.get_config_file_path() == str(config_path)
+ assert config_module.get_config_file_exists() is True
+ assert config_module.get_config_source(("model", "source")) == "file"
+
+
+def test_concurrent_first_access_loads_default_once(monkeypatch: pytest.MonkeyPatch) -> None:
+ """并发首次访问只能执行一次装载,并向所有线程返回同一配置实例。"""
+ entered, release = threading.Event(), threading.Event()
+ loaded_results: list[LoadedConfig] = []
+ independent_loader = config_module.load_config
+
+ def load_after_release() -> LoadedConfig:
+ """阻塞首次装载,让其他线程在同一初始化锁上等待。"""
+ entered.set()
+ assert release.wait(5)
+ loaded = independent_loader()
+ loaded_results.append(loaded)
+ return loaded
+
+ monkeypatch.setattr(config_module, "load_config", load_after_release)
+ with ThreadPoolExecutor(max_workers=8) as executor:
+ futures = [executor.submit(config_module.get_config) for _ in range(8)]
+ try:
+ assert entered.wait(5)
+ finally:
+ release.set()
+ results = [future.result(timeout=5) for future in futures]
+ assert len(loaded_results) == 1
+ assert all(configuration is loaded_results[0].config for configuration in results)
diff --git a/tests/unittest/test_content_list_ownership.py b/tests/unittest/test_content_list_ownership.py
index bbca7726..be923c43 100644
--- a/tests/unittest/test_content_list_ownership.py
+++ b/tests/unittest/test_content_list_ownership.py
@@ -11,7 +11,7 @@ from docvortex.render import RenderFormat as EngineFormat
from docvortex.render.contracts import MarkdownRenderOptions as EngineMarkdownOptions
from docvortex.schema import MiddleJson, PageInfo, TextBlock, TextSpan, EquationInlineSpan, ChartBlock, ChartBodyBlock
-from mineru.config import config
+from mineru.config import get_config
from mineru.render import RenderFormat, MarkdownRenderOptions, render_content_list, render_content_list_v2
from mineru.types import ContentType, ContentTypeV2
@@ -56,7 +56,7 @@ def test_migrated_inline_delimiter_cases(version: int, monkeypatch: pytest.Monke
)
before = middle.to_dict(skip_defaults=False)
delimiters = LatexDelimitersConfig(inline=LatexDelimiterConfig(left="\\(", right="\\)"))
- monkeypatch.setattr(config.render, "latex_delimiters", delimiters)
+ monkeypatch.setattr(get_config().render, "latex_delimiters", delimiters)
value = render_content_list(middle) if version == 1 else render_content_list_v2(middle)
def strings(item: Any) -> list[str]:
diff --git a/tests/unittest/test_doclib_api_page_remap.py b/tests/unittest/test_doclib_api_page_remap.py
index 047df54c..5332d7a5 100644
--- a/tests/unittest/test_doclib_api_page_remap.py
+++ b/tests/unittest/test_doclib_api_page_remap.py
@@ -11,7 +11,8 @@ from docvortex.document.pdf.layout import LAYOUT_EXTENSION
from docvortex.schema import MiddleJson, PageInfo, TextBlock, TextSpan
from pypdf import PdfReader
-from mineru.doclib.services.parse_svc import ParseFailure, _remap_api_result_pages_to_page_range, parse_batch_json_path
+from mineru.doclib.services.parse_svc import ParseFailure, _remap_api_result_pages_to_page_range
+from mineru.doclib.core.parse_artifacts import parse_batch_json_path
from mineru.integrations.docvortex import build_metadata
from mineru.parser.base import ParseResult
from mineru.render import PdfLayout, render_pdf
diff --git a/tests/unittest/test_doclib_app_startup.py b/tests/unittest/test_doclib_app_startup.py
index 22511d31..612147a2 100644
--- a/tests/unittest/test_doclib_app_startup.py
+++ b/tests/unittest/test_doclib_app_startup.py
@@ -11,11 +11,12 @@ from fastapi.testclient import TestClient
from loguru import logger as loguru_logger
from packaging.requirements import Requirement
-from mineru.config import LogConfig, PatchedConfig, config as mineru_config
+from mineru.config import Config, LogConfig, PatchedConfig, get_config
from mineru.doclib import app as doclib_app
from mineru.doclib.app import _assert_required_schema
from mineru.doclib.core.db import DatabaseManager
-from mineru.doclib.server import _tail_log, _write_temp_asset
+from mineru.doclib.server import _tail_log
+from mineru.doclib.services.content_read_svc import _write_temp_asset
from mineru.model import download as model_download
from mineru.parser import tier as parser_tier
from mineru.version import __version__
@@ -79,7 +80,7 @@ def test_torch_extra_includes_preflight_runtime_dependencies(monkeypatch: pytest
"transformers": "transformers",
}
- monkeypatch.setattr(mineru_config.model, "small_backend", "torch")
+ monkeypatch.setattr(get_config().model, "small_backend", "torch")
missing = [
module_name
@@ -136,9 +137,9 @@ def test_config_set_managed_tier_rejects_missing_models(monkeypatch: pytest.Monk
monkeypatch.setattr(doclib_app, "_create_background_task", _skip_background_task)
monkeypatch.setattr("mineru.doclib.server.ensure_tier_runtime_dependencies", lambda tier: None)
- monkeypatch.setattr(model_download.config.model, "base_dir", str(tmp_path / "models"))
+ monkeypatch.setattr(model_download.get_config().model, "base_dir", str(tmp_path / "models"))
# 无 torch 环境会把小模型后端解析为 onnx,显式固定以保持仓库名断言确定。
- monkeypatch.setattr(model_download.config.model, "small_backend", "torch")
+ monkeypatch.setattr(model_download.get_config().model, "small_backend", "torch")
cfg = PatchedConfig(doclib={"data_dir": str(tmp_path), "sqlite": {"path": str(tmp_path / "doclib.db")}})
with TestClient(doclib_app.create_app(cfg)) as client:
@@ -161,7 +162,7 @@ def test_config_set_managed_advanced_is_rejected(monkeypatch: pytest.MonkeyPatch
monkeypatch.setattr(doclib_app, "_create_background_task", _skip_background_task)
monkeypatch.setattr("mineru.doclib.server.ensure_tier_runtime_dependencies", lambda tier: None)
- monkeypatch.setattr(model_download.config.model, "base_dir", str(tmp_path / "models"))
+ monkeypatch.setattr(model_download.get_config().model, "base_dir", str(tmp_path / "models"))
cfg = PatchedConfig(doclib={"data_dir": str(tmp_path), "sqlite": {"path": str(tmp_path / "doclib.db")}})
with TestClient(doclib_app.create_app(cfg)) as client:
@@ -181,7 +182,7 @@ def test_config_set_managed_mode_rejects_missing_models_for_current_tier(
monkeypatch.setattr(doclib_app, "_create_background_task", _skip_background_task)
monkeypatch.setattr("mineru.doclib.server.ensure_tier_runtime_dependencies", lambda tier: None)
- monkeypatch.setattr(model_download.config.model, "base_dir", str(tmp_path / "models"))
+ monkeypatch.setattr(model_download.get_config().model, "base_dir", str(tmp_path / "models"))
cfg = PatchedConfig(doclib={"data_dir": str(tmp_path), "sqlite": {"path": str(tmp_path / "doclib.db")}})
with TestClient(doclib_app.create_app(cfg)) as client:
@@ -294,12 +295,32 @@ def test_bind_tcp_socket_strict_port_does_not_probe(monkeypatch: pytest.MonkeyPa
assert bind_calls == [("127.0.0.1", 15980)]
-def test_doclib_log_level_inherits_global_and_accepts_local_override(monkeypatch: pytest.MonkeyPatch) -> None:
- """验证 Doclib 未设置级别时继承全局配置,显式设置时局部覆盖。"""
- monkeypatch.setattr(doclib_app.config.log, "level", "debug")
+def test_doclib_log_level_inherits_global_and_accepts_local_override() -> None:
+ """验证 Doclib 未设置级别时继承当前应用的全局配置,显式设置时局部覆盖。"""
+ assert doclib_app._resolve_log_level(LogConfig(), global_level="debug") == logging.DEBUG
+ assert doclib_app._resolve_log_level(LogConfig(level="warning"), global_level="debug") == logging.WARNING
- assert doclib_app._resolve_log_level(LogConfig()) == logging.DEBUG
- assert doclib_app._resolve_log_level(LogConfig(level="warning")) == logging.WARNING
+
+def test_explicit_app_configuration_is_copied_without_loading_defaults(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ """显式启动配置独立于默认配置,创建后修改调用方对象不会影响应用快照。"""
+ cfg = Config(doclib={"log": {"dir": str(tmp_path)}}, log={"level": "debug"})
+
+ def reject_default_configuration() -> Config:
+ """显式配置路径不得再次装载进程默认配置。"""
+ pytest.fail("explicit app configuration must not load defaults")
+
+ monkeypatch.setattr(doclib_app, "get_config", reject_default_configuration)
+ try:
+ app = doclib_app.create_app(cfg)
+ snapshot = app.state.doclib_state.config
+ assert snapshot is not cfg
+ assert logging.getLogger("mineru").level == logging.DEBUG
+ cfg.doclib.parse_workers = 7
+ cfg.log.level = "error"
+ assert snapshot.doclib.parse_workers == 2
+ assert snapshot.log.level == "debug"
+ finally:
+ _clear_test_loggers()
def test_setup_logging_routes_application_logs_to_rotating_file_without_stderr_duplication(tmp_path: Path) -> None:
@@ -307,7 +328,7 @@ def test_setup_logging_routes_application_logs_to_rotating_file_without_stderr_d
log_path = tmp_path / "doclib.log"
access_log_path = tmp_path / "doclib.access.log"
try:
- doclib_app._setup_logging(LogConfig(app_path=str(log_path), access_path=str(access_log_path)))
+ doclib_app._setup_logging(LogConfig(app_path=str(log_path), access_path=str(access_log_path)), global_level="info")
mineru_logger = logging.getLogger("mineru")
uvicorn_error_logger = logging.getLogger("uvicorn.error")
@@ -351,7 +372,7 @@ def test_setup_logging_routes_loguru_records_to_application_log(tmp_path: Path)
log_path = tmp_path / "doclib.log"
access_log_path = tmp_path / "doclib.access.log"
try:
- doclib_app._setup_logging(LogConfig(app_path=str(log_path), access_path=str(access_log_path)))
+ doclib_app._setup_logging(LogConfig(app_path=str(log_path), access_path=str(access_log_path)), global_level="info")
patched_logger = loguru_logger.patch(lambda record: record.update(name="mineru.test_loguru"))
patched_logger.error("loguru bridged message")
@@ -773,6 +794,6 @@ def test_write_temp_asset_uses_temp_read_assets_directory(tmp_path: Path) -> Non
def test_tail_log_uses_current_config_default_when_data_dir_missing(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
log_path = tmp_path / "doclib.log"
log_path.write_text("line-1\nline-2\n", encoding="utf-8")
- monkeypatch.setattr(mineru_config.doclib, "log", LogConfig(app_path=str(log_path)))
+ monkeypatch.setattr(get_config().doclib, "log", LogConfig(app_path=str(log_path)))
assert _tail_log("") == ["line-1\n", "line-2\n"]
diff --git a/tests/unittest/test_doclib_architecture.py b/tests/unittest/test_doclib_architecture.py
new file mode 100644
index 00000000..14ffc7eb
--- /dev/null
+++ b/tests/unittest/test_doclib_architecture.py
@@ -0,0 +1,51 @@
+"""保护 Doclib 内容服务、产物访问和历史协议读取之间的依赖边界。"""
+
+from __future__ import annotations
+
+import ast
+import importlib.util
+from pathlib import Path
+
+_ROOT = Path(__file__).resolve().parents[2]
+
+
+def _imports(path: Path) -> set[str]:
+ """解析模块及符号导入,包括函数内和类型检查分支中的相对引用。"""
+ package = ".".join(path.relative_to(_ROOT).parts[:-1])
+ modules: set[str] = set()
+ for node in ast.walk(ast.parse(path.read_text(encoding="utf-8"))):
+ if isinstance(node, ast.Import):
+ modules.update(alias.name for alias in node.names)
+ elif isinstance(node, ast.ImportFrom):
+ module = importlib.util.resolve_name("." * node.level + (node.module or ""), package)
+ modules.add(module)
+ modules.update(f"{module}.{alias.name}" for alias in node.names)
+ return modules
+
+
+def test_doclib_core_does_not_depend_on_application_services() -> None:
+ """产物访问和数据库基础层不得反向借用服务或 HTTP 接口实现。"""
+ forbidden = ("mineru.doclib.services", "mineru.doclib.server", "mineru.doclib.app")
+ offenders = {
+ str(path.relative_to(_ROOT)): sorted(module for module in _imports(path) if module.startswith(forbidden))
+ for path in (_ROOT / "mineru/doclib/core").glob("*.py")
+ }
+ assert not {path: imports for path, imports in offenders.items() if imports}
+
+
+def test_content_read_service_is_independent_of_http_and_parse_scheduling() -> None:
+ """内容读取使用显式数据库和目录,不导入路由、应用状态或解析调度服务。"""
+ forbidden = ("fastapi", "starlette", "mineru.doclib.server", "mineru.doclib.app", "mineru.doclib.services.parse_svc")
+ imports = _imports(_ROOT / "mineru/doclib/services/content_read_svc.py")
+ assert not sorted(module for module in imports if module.startswith(forbidden))
+
+
+def test_legacy_protocol_conversion_stays_at_cached_middle_json_boundary() -> None:
+ """Doclib 仅由指定缓存读取模块进入旧协议转换,不让业务层自行解析旧格式。"""
+ boundary = _ROOT / "mineru/doclib/core/middle_json.py"
+ consumers = {
+ path
+ for path in (_ROOT / "mineru/doclib").rglob("*.py")
+ if any(module.startswith("mineru.backend.postprocess.legacy_") for module in _imports(path))
+ }
+ assert consumers == {boundary}
diff --git a/tests/unittest/test_doclib_cache_semantics.py b/tests/unittest/test_doclib_cache_semantics.py
index 44e3404b..67c1e419 100644
--- a/tests/unittest/test_doclib_cache_semantics.py
+++ b/tests/unittest/test_doclib_cache_semantics.py
@@ -1,4 +1,5 @@
from __future__ import annotations
+from mineru.doclib.services.content_read_svc import ContentReadService
from docvortex.schema import Producer
from mineru.integrations.docvortex import build_metadata
from _span_test_utils import inline as _inline, inline_text
@@ -38,8 +39,8 @@ from mineru.doclib.core.db import DatabaseManager
from mineru.doclib.core.file_io import FileStat, get_file_stat
from mineru.doclib.core.fts import FTSManager
from mineru.doclib.locators import ContentCursor
-from mineru.doclib.server import (
- DoclibServer,
+from mineru.doclib.server import DoclibServer
+from mineru.doclib.services.content_read_svc import (
_find_block_by_no,
_LocatorParts,
_make_doclib_image_renderer,
@@ -47,6 +48,7 @@ from mineru.doclib.server import (
_locator_after,
)
from mineru.doclib.services import parse_svc as parse_svc_module
+from mineru.doclib.core import parse_artifacts as parse_artifacts_module
from mineru.doclib.services.cleanup_svc import CleanupService
from mineru.doclib.services.config_svc import ConfigService
from mineru.doclib.services.parse_svc import (
@@ -57,10 +59,8 @@ from mineru.doclib.services.parse_svc import (
_resolve_default_tier,
_resolve_parsing_rule_default_tier,
expand_page_range,
- filter_pages_by_user_range,
- load_pages_from_done_batches,
- parse_batch_json_path,
)
+from mineru.doclib.core.parse_artifacts import filter_pages_by_user_range, load_pages_from_done_batches, parse_batch_json_path
from mineru.doclib.services.scan_svc import ScanService
from mineru.doclib.services.search_svc import SearchService
from mineru.doclib.types import DocContentExportRequest, FileInfo, InvalidateRequest, ParseResponse, WatchRequest
@@ -475,10 +475,16 @@ def test_read_without_tier_uses_highest_cached_tier(tmp_path: Path) -> None:
_write_batch(tmp_path, sha256, tier, "1", 1000, [_text_page(tier).to_dict()])
rows.append({"sha256": sha256, "tier": tier, "page_range": "1", "done_at": 1000, "status": "done"})
server = DoclibServer(SimpleNamespace(db=_FakeDB(parses=rows, file_row=None), data_dir=str(tmp_path)))
- assert asyncio.run(server._default_read_tier(sha256)) == "advanced"
+ assert (
+ asyncio.run(ContentReadService(server.state.db, data_dir=server.state.data_dir)._default_read_tier(sha256))
+ == "advanced"
+ )
stale = Path(parse_batch_json_path(str(tmp_path), sha256, "advanced", "1", 1000))
stale.write_text('{"schema_version":"1.0","pages":[]}')
- assert asyncio.run(server._default_read_tier(sha256)) == "standard"
+ assert (
+ asyncio.run(ContentReadService(server.state.db, data_dir=server.state.data_dir)._default_read_tier(sha256))
+ == "standard"
+ )
def test_managed_local_parse_server_url_uses_health_managed_url() -> None:
@@ -1590,7 +1596,7 @@ def test_ingest_moved_file_retargets_to_new_path(tmp_path: Path, monkeypatch: py
assert second_row["path"] == str(new_dir / "doc.pdf")
stale = await db.fetchone("SELECT status FROM files WHERE id=?", (first_row["id"],))
assert stale["status"] == "deleted"
- accessible = await parse_svc_module.accessible_file_for_sha256(db, second_row["sha256"])
+ accessible = await parse_artifacts_module.accessible_file_for_sha256(db, second_row["sha256"])
assert accessible is not None
assert accessible["path"] == str(new_dir / "doc.pdf")
@@ -1617,12 +1623,12 @@ def test_accessible_file_for_sha256_skips_missing_paths(tmp_path: Path) -> None:
await db.execute(insert_sql, (str(missing), "moved.pdf", "pdf", 10, 10, sha256, 2000, 2000))
await db.execute(insert_sql, (str(existing), "kept.pdf", "pdf", 10, 10, sha256, 1000, 1000))
- row = await parse_svc_module.accessible_file_for_sha256(db, sha256)
+ row = await parse_artifacts_module.accessible_file_for_sha256(db, sha256)
assert row is not None
assert row["path"] == str(existing)
existing.unlink()
- assert await parse_svc_module.accessible_file_for_sha256(db, sha256) is None
+ assert await parse_artifacts_module.accessible_file_for_sha256(db, sha256) is None
asyncio.run(_run())
@@ -4977,7 +4983,7 @@ def test_progressive_markdown_uses_readable_pdf_block_locator(tmp_path: Path) ->
no_marker=False,
)
- response = asyncio.run(server._execute_read_plan(plan))
+ response = asyncio.run(ContentReadService(server.state.db, data_dir=server.state.data_dir)._execute_read_plan(plan))
assert "![Image block](doc:eeeeeee/tier:standard/page:1/block:1)" in response.content
assert "figures/rendered.jpg" not in response.content
@@ -5005,7 +5011,7 @@ def test_doc_content_marker_uses_document_page_count_only(
server = DoclibServer(SimpleNamespace(data_dir=str(tmp_path), db=db))
content = asyncio.run(
- server._render_doc_content(
+ ContentReadService(server.state.db, data_dir=server.state.data_dir)._render_doc_content(
sha256,
tier=tier,
page_range="5",
@@ -5117,7 +5123,7 @@ def test_progressive_markdown_uses_base64_availability_for_bboxless_blocks(
no_marker=False,
)
- response = asyncio.run(server._execute_read_plan(plan))
+ response = asyncio.run(ContentReadService(server.state.db, data_dir=server.state.data_dir)._execute_read_plan(plan))
assert f"![Image block]({expected_target})" in response.content
@@ -5148,7 +5154,9 @@ def test_doclib_office_image_asset_reads_base64(tmp_path: Path) -> None:
target=ContentCursor(short_id="fffffff", tier=tier, page_no=1, block_no=1),
)
- asset = asyncio.run(server._render_image_response(plan, [page])).asset
+ asset = asyncio.run(
+ ContentReadService(server.state.db, data_dir=server.state.data_dir)._render_image_response(plan, [page])
+ ).asset
assert asset is not None
assert asset.mime_type == "image/png"
@@ -5200,7 +5208,9 @@ def test_doclib_visual_block_asset_reads_base64(
target=ContentCursor(short_id="fffffff", tier=tier, page_no=1, block_no=1),
)
- asset = asyncio.run(server._render_image_response(plan, [page])).asset
+ asset = asyncio.run(
+ ContentReadService(server.state.db, data_dir=server.state.data_dir)._render_image_response(plan, [page])
+ ).asset
assert asset is not None
assert asset.mime_type == "image/png"
@@ -5235,7 +5245,9 @@ def test_doclib_pdf_block_locator_crops_source_pdf(tmp_path: Path) -> None:
target=ContentCursor(short_id="aaaaaaa", tier=tier, page_no=1, block_no=1),
)
- asset = asyncio.run(server._render_image_response(plan, [page])).asset
+ asset = asyncio.run(
+ ContentReadService(server.state.db, data_dir=server.state.data_dir)._render_image_response(plan, [page])
+ ).asset
assert asset is not None
assert asset.mime_type == "image/png"
@@ -5274,7 +5286,9 @@ def test_doclib_image_block_locator_crops_source_image_at_original_resolution(tm
target=ContentCursor(short_id="bbbbbbb", tier=tier, page_no=1, block_no=1),
)
- asset = asyncio.run(server._render_image_response(plan, [page])).asset
+ asset = asyncio.run(
+ ContentReadService(server.state.db, data_dir=server.state.data_dir)._render_image_response(plan, [page])
+ ).asset
assert asset is not None
assert asset.mime_type == "image/png"
@@ -5308,7 +5322,9 @@ def test_doclib_office_image_asset_transcodes_to_requested_format(tmp_path: Path
target=ContentCursor(short_id="fffffff", tier=tier, page_no=1, block_no=1),
)
- asset = asyncio.run(server._render_image_response(plan, [page])).asset
+ asset = asyncio.run(
+ ContentReadService(server.state.db, data_dir=server.state.data_dir)._render_image_response(plan, [page])
+ ).asset
assert asset is not None
assert asset.mime_type == "image/jpeg"
@@ -5340,7 +5356,7 @@ def test_doclib_office_image_asset_missing_payload_reports_asset_not_available(t
)
with pytest.raises(NotFoundError) as exc_info:
- asyncio.run(server._render_image_response(plan, [page]))
+ asyncio.run(ContentReadService(server.state.db, data_dir=server.state.data_dir)._render_image_response(plan, [page]))
assert exc_info.value.code == "asset_not_available"
assert exc_info.value.param == "locator"
@@ -5385,10 +5401,14 @@ def test_doclib_bbox_precedes_embedded_image(
)
if not source_available:
with pytest.raises(NotFoundError) as error:
- asyncio.run(server._render_image_response(plan, [page]))
+ asyncio.run(
+ ContentReadService(server.state.db, data_dir=server.state.data_dir)._render_image_response(plan, [page])
+ )
assert error.value.code == "no_accessible_file"
return
- asset = asyncio.run(server._render_image_response(plan, [page])).asset
+ asset = asyncio.run(
+ ContentReadService(server.state.db, data_dir=server.state.data_dir)._render_image_response(plan, [page])
+ ).asset
assert asset is not None
assert asset.mime_type == f"image/{image_format}"
with Image.open(BytesIO(Path(asset.path).read_bytes())) as actual:
diff --git a/tests/unittest/test_doclib_config.py b/tests/unittest/test_doclib_config.py
index 0d5df46e..94604f42 100644
--- a/tests/unittest/test_doclib_config.py
+++ b/tests/unittest/test_doclib_config.py
@@ -14,6 +14,7 @@ from mineru.config import (
_interpolate_env,
_load_config,
_load_effective_config,
+ get_config,
update_config_file,
)
from mineru.doclib.config_defaults import CONFIG_DEFAULTS
@@ -438,10 +439,15 @@ def test_transport_enabled_accepts_auto_from_env(monkeypatch: pytest.MonkeyPatch
def test_patched_config_returns_validated_deep_patch() -> None:
+ """验证局部覆盖经过类型校验,且不会修改默认配置中的嵌套对象。"""
+ default = get_config()
+ original = default.model_dump()
cfg = PatchedConfig(doclib={"tcp": {"port": "16000"}, "sqlite": {"cache_size": "-1"}})
assert cfg.doclib.tcp.port == 16000
assert cfg.doclib.sqlite.cache_size == -1
+ assert cfg.doclib is not default.doclib
+ assert default.model_dump() == original
def test_tcp_config_exposes_port_probe_count() -> None:
@@ -553,12 +559,10 @@ def test_log_config_dir_override_derives_paths_in_deep_patches(monkeypatch: pyte
assert Path(env_cfg.doclib.log.resolved_stdout_path).as_posix() == "/tmp/env-logs/doclib.stdout.log"
assert Path(env_cfg.doclib.log.resolved_stderr_path).as_posix() == "/tmp/env-logs/doclib.stderr.log"
assert (
- Path(env_cfg.doclib.log.resolved_parse_server_stdout_path).as_posix()
- == "/tmp/env-logs/doclib.parse-server.stdout.log"
+ Path(env_cfg.doclib.log.resolved_parse_server_stdout_path).as_posix() == "/tmp/env-logs/doclib.parse-server.stdout.log"
)
assert (
- Path(env_cfg.doclib.log.resolved_parse_server_stderr_path).as_posix()
- == "/tmp/env-logs/doclib.parse-server.stderr.log"
+ Path(env_cfg.doclib.log.resolved_parse_server_stderr_path).as_posix() == "/tmp/env-logs/doclib.parse-server.stderr.log"
)
assert patched_cfg.doclib.log.app_path is None
assert patched_cfg.doclib.log.access_path is None
diff --git a/tests/unittest/test_doclib_db.py b/tests/unittest/test_doclib_db.py
index e1da1a24..5aeb6bf1 100644
--- a/tests/unittest/test_doclib_db.py
+++ b/tests/unittest/test_doclib_db.py
@@ -76,6 +76,61 @@ def test_initialize_enables_wal_mode(tmp_path: Path) -> None:
asyncio.run(_run())
[email protected]("use_custom_config", [False, True], ids=["defaults", "custom"])
+def test_operational_connections_apply_sqlite_settings(tmp_path: Path, use_custom_config: bool) -> None:
+ """验证初始化结束后,新建运行连接仍应用默认或自定义参数。"""
+
+ async def _run() -> None:
+ """通过公开查询入口检查多个独立连接以及重新创建的数据库管理器。"""
+ sqlite_cfg = (
+ SQLiteConfig(
+ busy_timeout_ms=4321,
+ synchronous="FULL",
+ mmap_size=4_194_304,
+ cache_size=-4096,
+ temp_store="FILE",
+ wal_autocheckpoint=64,
+ journal_size_limit=1_048_576,
+ )
+ if use_custom_config
+ else None
+ )
+ expected = (
+ {
+ "busy_timeout": 4321,
+ "synchronous": 2,
+ "mmap_size": 4_194_304,
+ "cache_size": -4096,
+ "temp_store": 1,
+ "wal_autocheckpoint": 64,
+ "journal_size_limit": 1_048_576,
+ "foreign_keys": 1,
+ }
+ if use_custom_config
+ else {
+ "busy_timeout": 5000,
+ "synchronous": 1,
+ "mmap_size": 268_435_456,
+ "cache_size": -20000,
+ "temp_store": 2,
+ "wal_autocheckpoint": 1000,
+ "journal_size_limit": 33_554_432,
+ "foreign_keys": 1,
+ }
+ )
+ db_path = str(tmp_path / "doclib.db")
+ initialized_db = DatabaseManager(db_path, sqlite_cfg)
+ await initialized_db.initialize()
+
+ for db in (initialized_db, DatabaseManager(db_path, sqlite_cfg)):
+ for setting, value in expected.items():
+ row = await db.fetchone(f"PRAGMA {setting}")
+ assert row is not None and list(row.values()) == [value], setting
+ assert await db.fetchone("PRAGMA journal_mode") == {"journal_mode": "wal"}
+
+ asyncio.run(_run())
+
+
def test_operational_connection_sets_busy_timeout_without_resetting_wal(monkeypatch: pytest.MonkeyPatch) -> None:
async def _run() -> None:
connection = _FakeConnection()
@@ -91,7 +146,9 @@ def test_operational_connection_sets_busy_timeout_without_resetting_wal(monkeypa
await db._connect()
assert connect_timeouts == [4.321]
- assert connection.statements == ["PRAGMA busy_timeout=4321", "PRAGMA foreign_keys=ON"]
+ assert "PRAGMA busy_timeout=4321" in connection.statements
+ assert "PRAGMA foreign_keys=ON" in connection.statements
+ assert not any(statement.startswith("PRAGMA journal_mode") for statement in connection.statements)
asyncio.run(_run())
diff --git a/tests/unittest/test_doclib_instance_lock.py b/tests/unittest/test_doclib_instance_lock.py
index 7c0894b0..8849bf3b 100644
--- a/tests/unittest/test_doclib_instance_lock.py
+++ b/tests/unittest/test_doclib_instance_lock.py
@@ -80,7 +80,7 @@ def test_doclib_main_exits_before_startup_when_home_lock_is_held(monkeypatch: py
"uds": {"path": str(socket_path)},
}
)
- monkeypatch.setattr(doclib_app, "config", cfg)
+ monkeypatch.setattr(doclib_app, "get_config", lambda: cfg)
monkeypatch.setattr(instance_lock, "default_endpoint_path", lambda: str(endpoint_path))
with instance_lock.doclib_home_lock():
diff --git a/tests/unittest/test_doclib_legacy_page_ranges.py b/tests/unittest/test_doclib_legacy_page_ranges.py
index 2d738772..c7830d51 100644
--- a/tests/unittest/test_doclib_legacy_page_ranges.py
+++ b/tests/unittest/test_doclib_legacy_page_ranges.py
@@ -1,4 +1,5 @@
from __future__ import annotations
+from mineru.doclib.services.content_read_svc import ContentReadService
import asyncio
import json
@@ -15,12 +16,8 @@ from mineru.doclib.core.db import DatabaseManager
from mineru.doclib.core.fts import FTSManager
from mineru.doclib.server import DoclibServer, _parse_info, _tier_parse_info
from mineru.doclib.services.config_svc import ConfigService
-from mineru.doclib.services.parse_svc import (
- ParseService,
- _parse_record_response,
- load_pages_from_done_batches,
- parse_batch_json_path,
-)
+from mineru.doclib.services.parse_svc import ParseService, _parse_record_response
+from mineru.doclib.core.parse_artifacts import load_pages_from_done_batches, parse_batch_json_path
from mineru.parser.base import MIDDLE_JSON_SCHEMA_VERSION
from mineru.version import __version__
from mineru.errors import MineruError
@@ -86,7 +83,10 @@ def test_historical_cache_rebuilds_fts_and_default_tier(tmp_path: Path, family:
standard_path.write_bytes(path.read_bytes())
await db.execute("UPDATE parses SET tier='standard'")
before = standard_path.read_bytes()
- assert await server._default_read_tier(_SHA256) == "standard"
+ assert (
+ await ContentReadService(server.state.db, data_dir=server.state.data_dir)._default_read_tier(_SHA256)
+ == "standard"
+ )
await service._rebuild_fts_after_invalidate(_SHA256)
assert await service.fts.search("old")
assert await service.fts.get_tier(_SHA256) == "standard"
@@ -120,7 +120,7 @@ def test_all_supported_formats_compact_to_current_protocol(tmp_path: Path, legac
assert "file_suffix" not in payload and "effort" not in payload
assert payload["metadata"]["producer"]["version"] == __version__
assert [page["page_idx"] for page in payload["pages"]] == [2, 3, 4, 5]
- content = await server._render_doc_content(
+ content = await ContentReadService(server.state.db, data_dir=server.state.data_dir)._render_doc_content(
_SHA256, tier="flash", page_range="3-6", format="markdown", no_marker=True
)
assert "old page 3" in content and "intermediate page 4" in content and "latest page 5" in content
@@ -152,12 +152,14 @@ def test_old_protocol_cache_is_not_available_or_compacted(tmp_path: Path, old_ve
assert {path.name: path.read_bytes() for path in old_path.parent.iterdir()} == before_files
with pytest.raises(MineruError, match="reparse"):
load_pages_from_done_batches(str(tmp_path), _SHA256, "flash", rows)
- current = await server._render_doc_content(
+ current = await ContentReadService(server.state.db, data_dir=server.state.data_dir)._render_doc_content(
_SHA256, tier="flash", page_range="4-7", format="markdown", no_marker=True
)
assert "new page 4" in current and "old page" not in current
with pytest.raises(MineruError, match="reparse"):
- await server._render_doc_content(_SHA256, tier="flash", page_range="1-3", format="markdown", no_marker=True)
+ await ContentReadService(server.state.db, data_dir=server.state.data_dir)._render_doc_content(
+ _SHA256, tier="flash", page_range="1-3", format="markdown", no_marker=True
+ )
listing = await server.list_parses(doc_ref="aaaaaaa", tier="flash", page_range="1-7")
assert listing.coverage.done_page_range == "4-7"
assert listing.coverage.missing_page_range == "1-3"
@@ -302,7 +304,7 @@ def test_old_cache_hits_exports_and_serializes_without_mutating_storage(tmp_path
assert _tier_parse_info(public_row).page_range == "1-5"
assert _parse_record_response(public_row)["page_range"] == "1-5"
assert public_row["page_range"] == "1~5"
- exported = await server._render_doc_content(
+ exported = await ContentReadService(server.state.db, data_dir=server.state.data_dir)._render_doc_content(
_SHA256, tier="flash", page_range="1-5", format="markdown", no_marker=True
)
assert "old page 1" in exported and "old page 5" in exported
@@ -354,7 +356,7 @@ def test_mixed_cache_compaction_keeps_latest_pages_and_writes_new_ranges(tmp_pat
assert await db.fetchall("SELECT page_range, done_at FROM parses") == [{"page_range": "1-7", "done_at": 2000}]
path = _cache_path(tmp_path, "1-7", 2000)
assert sorted(p.name for p in path.parent.glob("*.json")) == [path.name]
- exported = await server._render_doc_content(
+ exported = await ContentReadService(server.state.db, data_dir=server.state.data_dir)._render_doc_content(
_SHA256, tier="flash", page_range="1-7", format="markdown", no_marker=True
)
assert "old page 3" in exported and "new page 4" in exported
diff --git a/tests/unittest/test_doclib_legacy_schema_adapter.py b/tests/unittest/test_doclib_legacy_schema_adapter.py
index 0d594510..15944191 100644
--- a/tests/unittest/test_doclib_legacy_schema_adapter.py
+++ b/tests/unittest/test_doclib_legacy_schema_adapter.py
@@ -1,12 +1,14 @@
from docvortex.schema import Producer
from mineru.integrations.docvortex import build_metadata
from copy import deepcopy
+import json
+from pathlib import Path
import pytest
from mineru.backend.postprocess.legacy_schema_adapter import legacy_page_to_model_list
from docvortex.postprocess.pages import model_json_to_pages
-from mineru.doclib.background.compaction import _normalize_batch_pages
+from mineru.doclib.core.middle_json import read_cached_middle_json
from mineru.types import BlockType, MiddleJson, ModelJson
@@ -341,7 +343,7 @@ def test_legacy_adapter_separates_office_blocks_with_synthetic_bbox_groups() ->
assert pages[0].blocks[2].content[0].content == "after"
-def test_doclib_compaction_converts_schema_v1_pages_with_current_adapter() -> None:
+def test_doclib_cache_converts_schema_v1_pages_with_current_adapter(tmp_path: Path) -> None:
"""验证 Doclib 历史调用点重新通过宿主适配器生成 2.0 page 字典。"""
payload = {
"schema_version": "1.0",
@@ -361,19 +363,23 @@ def test_doclib_compaction_converts_schema_v1_pages_with_current_adapter() -> No
],
}
- pages = _normalize_batch_pages(payload)
+ cache_path = tmp_path / "legacy.json"
+ cache_path.write_text(json.dumps(payload), encoding="utf-8")
+ pages = [page.to_dict() for page in read_cached_middle_json(cache_path).pages]
assert pages[0]["page_idx"] == 3
assert pages[0]["blocks"][0]["content"] == [{"type": "text", "content": "cached"}]
-def test_doclib_compaction_rejects_unknown_legacy_schema() -> None:
+def test_doclib_cache_rejects_unknown_legacy_schema(tmp_path: Path) -> None:
"""验证未知 schema 不会被误当成 3.4.5 页面吞掉。"""
+ cache_path = tmp_path / "unknown.json"
+ cache_path.write_text(json.dumps({"schema_version": "99.0", "pages": [{"page_idx": 0, "blocks": []}]}), encoding="utf-8")
with pytest.raises(ValueError, match="source reparse"):
- _normalize_batch_pages({"schema_version": "99.0", "pages": [{"page_idx": 0, "blocks": []}]})
+ read_cached_middle_json(cache_path)
-def test_doclib_compaction_accepts_current_schema_without_legacy_conversion(monkeypatch: pytest.MonkeyPatch) -> None:
+def test_doclib_cache_accepts_current_schema_without_legacy_conversion(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""验证当前 2.0 缓存直接读取页面,不进入旧版转换。"""
from mineru.backend.postprocess import legacy_schema_adapter
@@ -388,4 +394,6 @@ def test_doclib_compaction_accepts_current_schema_without_legacy_conversion(monk
is_full_document=True,
metadata={"file_suffix": "pdf", "producer": Producer(name="mineru", version="test")},
)
- assert _normalize_batch_pages(document.to_dict()) == document.to_dict()["pages"]
+ cache_path = tmp_path / "current.json"
+ cache_path.write_text(json.dumps(document.to_dict()), encoding="utf-8")
+ assert [page.to_dict() for page in read_cached_middle_json(cache_path).pages] == document.to_dict()["pages"]
diff --git a/tests/unittest/test_doclib_page_selection.py b/tests/unittest/test_doclib_page_selection.py
index e0a1c416..40c46610 100644
--- a/tests/unittest/test_doclib_page_selection.py
+++ b/tests/unittest/test_doclib_page_selection.py
@@ -18,8 +18,9 @@ from mineru.cli.main import app
from mineru.doclib.client import DoclibClient
from mineru.doclib.core.db import DatabaseManager
from mineru.doclib.locators import parse_content_cursor
-from mineru.doclib.server import DoclibServer, _ReadPlan
-from mineru.doclib.services.parse_svc import parse_batch_json_path
+from mineru.doclib.server import DoclibServer
+from mineru.doclib.services.content_read_svc import ContentReadService, _ReadPlan
+from mineru.doclib.core.parse_artifacts import parse_batch_json_path
from mineru.doclib.types import ContentAsset, ContentNextRequest, ContentRequestScope, DocContentResponse
from mineru.errors import InvalidRequestError, NotFoundError
from mineru.parser.page_range import format_page_range
@@ -207,13 +208,13 @@ def test_relative_page_image_uses_resolved_page(tmp_path: Path, monkeypatch: pyt
db, server = await _cached_library(tmp_path)
captured: list[int] = []
- async def render_asset(plan: _ReadPlan, page: PageInfo) -> ContentAsset:
+ async def render_asset(self: ContentReadService, plan: _ReadPlan, page: PageInfo) -> ContentAsset:
"""记录素材调用目标,不初始化 PDFium。"""
assert plan.target is not None
captured.append(plan.target.page_no)
return ContentAsset(path=str(tmp_path / "page.png"), mime_type="image/png")
- monkeypatch.setattr(server, "_render_source_image_asset", render_asset)
+ monkeypatch.setattr(ContentReadService, "_render_source_image_asset", render_asset)
try:
result = await server.read_content(f"{_DOC_REF}/page:r1", format="image")
assert captured == [12]
diff --git a/tests/unittest/test_doclib_progressive_reading.py b/tests/unittest/test_doclib_progressive_reading.py
index 81eef5e5..c2584cd7 100644
--- a/tests/unittest/test_doclib_progressive_reading.py
+++ b/tests/unittest/test_doclib_progressive_reading.py
@@ -1,7 +1,7 @@
from _span_test_utils import inline as _inline
import pytest
-from mineru.doclib.server import (
+from mineru.doclib.services.content_read_svc import (
_next_content_request,
_normalize_content_page_range,
_page_markdown_blocks,
diff --git a/tests/unittest/test_engine_dead_lifecycle.py b/tests/unittest/test_engine_dead_lifecycle.py
index f515dd8f..c86be219 100644
--- a/tests/unittest/test_engine_dead_lifecycle.py
+++ b/tests/unittest/test_engine_dead_lifecycle.py
@@ -75,7 +75,7 @@ def test_api_engine_failure_drains_running_and_queued_jobs(tmp_path: Path, monke
"""等待公开父任务完成资源清理后检查内部执行槽。"""
await asyncio.gather(
*(
- app.state.job_store.wait_for_terminal(file.execution_id, 2)
+ app.state.execution_store.wait_for_terminal(file.execution_id, 2)
for record in app.state.task_store._records.values()
for file in record.file_states
)
@@ -97,8 +97,8 @@ def test_api_engine_failure_drains_running_and_queued_jobs(tmp_path: Path, monke
**({"json": {"tier": "flash", "files": [{"file_name": "new.pdf", "bytes": 3}]}} if method == "post" else {}),
)
assert response.status_code == 503 and response.json().get("error")["code"] == "engine_dead"
- assert app.state.job_store._semaphore._value == 2
- assert not app.state.job_store._tasks
+ assert app.state.execution_store._semaphore._value == 2
+ assert not app.state.execution_store._tasks
assert not app.state.file_store._reserved_output_ids
for job in jobs:
file_id = job["files"][0]["file_id"]
@@ -186,11 +186,20 @@ def test_old_generation_failure_snapshot_remains_queryable() -> None:
pool._workers[state.worker_id] = state
route = registry.register("job", owner_scope="anonymous", worker_id=state.worker_id, upstream_id="job-old")
route.metadata["payload"] = {
- "job_id": route.public_id,
+ "task_id": "job-old",
"status": "running",
"files": [{"status": "pending"}],
"progress": {"total": 1, "completed": 0},
}
+ from mineru.tasks.execution import TaskExecution
+
+ route.metadata["execution"] = TaskExecution(
+ execution_id=route.public_id,
+ file_id="file-old",
+ file_name="old.pdf",
+ status="pending",
+ created_at=route.created_at,
+ )
client.portal.call(pool.refresh, state)
from mineru.kit.router.tasks import read_task_execution
@@ -199,8 +208,8 @@ def test_old_generation_failure_snapshot_remains_queryable() -> None:
return await read_task_execution(route, pool, registry)
execution = client.portal.call(snapshot)
- assert execution.job["status"] == "failed"
- assert execution.job["files"][0].get("error")["code"] == "upstream_task_lost"
+ assert execution.status == "failed"
+ assert execution.error is not None and execution.error.code == "upstream_task_lost"
pool._workers = {state.worker_id: state}
reservation = pool.reserve_job(tier="standard", required_sources={"upload"})
assert reservation is not None and reservation.worker is state
diff --git a/tests/unittest/test_flash_csv.py b/tests/unittest/test_flash_csv.py
index 8f037b58..8e65b742 100644
--- a/tests/unittest/test_flash_csv.py
+++ b/tests/unittest/test_flash_csv.py
@@ -1,5 +1,6 @@
from __future__ import annotations
+from task_test_utils import execution_request
from task_test_utils import stored_source, execution_output
import asyncio
@@ -11,7 +12,7 @@ from docvortex.document.detection import guess_suffix_by_bytes, guess_suffix_by_
from mineru.backend.analyze import aio_doc_analyze, doc_analyze
from mineru.parser import api_server, parse, parse_async
-from mineru.parser.api_server import _ParseRequest, FileStore
+from mineru.parser.api_server import FileStore
from mineru.render.html import render_html
from mineru.render.markdown import render_markdown
from mineru.types import TableBlock, TableBodyBlock
@@ -93,25 +94,12 @@ def test_tsv_parse_server_job_emits_structured_outputs_with_independent_suffix(t
source = tmp_path / "sample.tsv"
source.write_text("name\tage\nAlice\t30\n", encoding="utf-8")
file_store = FileStore(tmp_path / "api-files")
- request = _ParseRequest.model_validate(
- {
- "files": [{"source": stored_source(file_store, source)}],
- "tier": "standard",
- "output_formats": ["default"],
- }
- )
- record = api_server.JobStore().create(request, file_store)
-
- asyncio.run(
- api_server._run_job(
- record,
- request,
- file_store,
- image_analysis=True,
- )
- )
-
- parsed_file = record.files[0]
+ request = execution_request(file_store, stored_source(file_store, source), tier="standard", output_formats=["default"])
+ record = api_server.ExecutionStore().create(request, file_store)
+
+ asyncio.run(api_server._run_execution(record, file_store, image_analysis=True))
+
+ parsed_file = record
assert parsed_file.status == "completed"
markdown_data = execution_output(file_store, record, "markdown.md")
@@ -130,25 +118,12 @@ def test_csv_parse_server_job_emits_structured_outputs_with_flash_metadata(tmp_p
source = tmp_path / "sample.csv"
source.write_text("name,age\nAlice,30\n", encoding="utf-8")
file_store = FileStore(tmp_path / "api-files")
- request = _ParseRequest.model_validate(
- {
- "files": [{"source": stored_source(file_store, source)}],
- "tier": "standard",
- "output_formats": ["default"],
- }
- )
- record = api_server.JobStore().create(request, file_store)
-
- asyncio.run(
- api_server._run_job(
- record,
- request,
- file_store,
- image_analysis=True,
- )
- )
-
- parsed_file = record.files[0]
+ request = execution_request(file_store, stored_source(file_store, source), tier="standard", output_formats=["default"])
+ record = api_server.ExecutionStore().create(request, file_store)
+
+ asyncio.run(api_server._run_execution(record, file_store, image_analysis=True))
+
+ parsed_file = record
assert parsed_file.status == "completed"
markdown_data = execution_output(file_store, record, "markdown.md")
diff --git a/tests/unittest/test_flash_epub.py b/tests/unittest/test_flash_epub.py
index 9ad1da53..67031766 100644
--- a/tests/unittest/test_flash_epub.py
+++ b/tests/unittest/test_flash_epub.py
@@ -1,5 +1,6 @@
from __future__ import annotations
+from task_test_utils import execution_request
from task_test_utils import stored_source, execution_output
import asyncio
@@ -23,7 +24,7 @@ from mineru.doclib.core.fts import FTSManager
from mineru.doclib.services.parse_svc import ParseService
from mineru.errors import InvalidRequestError
from mineru.parser import MinerUParser, api_server, parse, parse_async
-from mineru.parser.api_server import _ParseRequest, FileStore
+from mineru.parser.api_server import FileStore
from mineru.render import render_docx, render_html, render_markdown, render_structured_content
from mineru.types import BlockType
@@ -161,23 +162,10 @@ def test_epub_local_parse_job_emits_spine_aligned_flash_outputs(tmp_path: Path)
source = tmp_path / "book.epub"
source.write_bytes(build_epub_fixture())
file_store = FileStore(tmp_path / "api-files")
- request = _ParseRequest.model_validate(
- {
- "files": [{"source": stored_source(file_store, source)}],
- "tier": "standard",
- "output_formats": ["default"],
- }
- )
- record = api_server.JobStore().create(request, file_store)
- asyncio.run(
- api_server._run_job(
- record,
- request,
- file_store,
- image_analysis=True,
- )
- )
- parsed_file = record.files[0]
+ request = execution_request(file_store, stored_source(file_store, source), tier="standard", output_formats=["default"])
+ record = api_server.ExecutionStore().create(request, file_store)
+ asyncio.run(api_server._run_execution(record, file_store, image_analysis=True))
+ parsed_file = record
assert parsed_file.status == "completed"
middle_data = execution_output(file_store, record, "middle_json.json")
payload = json.loads(middle_data)
@@ -193,25 +181,14 @@ def test_epub_local_parse_job_preserves_page_range_error_code(tmp_path: Path) ->
source = tmp_path / "book.epub"
source.write_bytes(build_epub_fixture())
file_store = FileStore(tmp_path / "api-files")
- request = _ParseRequest.model_validate(
- {
- "files": [{"source": stored_source(file_store, source), "page_range": "2"}],
- "tier": "flash",
- "output_formats": ["default"],
- }
- )
- record = api_server.JobStore().create(request, file_store)
- asyncio.run(
- api_server._run_job(
- record,
- request,
- file_store,
- image_analysis=True,
- )
+ request = execution_request(
+ file_store, stored_source(file_store, source), page_range="2", tier="flash", output_formats=["default"]
)
- assert record.files[0].status == "failed"
- assert record.files[0].error is not None
- assert record.files[0].error.code == "page_range_invalid"
+ record = api_server.ExecutionStore().create(request, file_store)
+ asyncio.run(api_server._run_execution(record, file_store, image_analysis=True))
+ assert record.status == "failed"
+ assert record.error is not None
+ assert record.error.code == "page_range_invalid"
def test_doclib_ingests_epub_as_local_flash_with_spine_page_count(tmp_path: Path) -> None:
diff --git a/tests/unittest/test_flash_html.py b/tests/unittest/test_flash_html.py
index faa62f5a..8a3a593b 100644
--- a/tests/unittest/test_flash_html.py
+++ b/tests/unittest/test_flash_html.py
@@ -1,5 +1,6 @@
from __future__ import annotations
+from task_test_utils import execution_request
from task_test_utils import stored_source, execution_output, build_default_zip
import base64
@@ -25,7 +26,7 @@ from mineru.backend.analyze import aio_doc_analyze, doc_analyze
from mineru.doclib.services.parse_svc import ParseService
from mineru.errors import InvalidRequestError
from mineru.parser import ParseResult, api_server, parse, parse_async
-from mineru.parser.api_server import _ParseRequest, FileStore
+from mineru.parser.api_server import FileStore
from mineru.render.docx import render_docx
from mineru.render.html import render_html
from mineru.render.markdown import render_markdown
@@ -141,25 +142,12 @@ def test_html_parse_server_url_preserves_http_declared_charset(tmp_path: Path, m
client = _url_stream_client(response)
monkeypatch.setattr(api_server, "httpx", SimpleNamespace(AsyncClient=lambda **_: client))
file_store = FileStore(tmp_path / "api-files")
- request = _ParseRequest.model_validate(
- {
- "files": [{"source": {"type": "url", "url": url}}],
- "tier": "standard",
- "output_formats": ["default"],
- }
- )
- record = api_server.JobStore().create(request, file_store)
-
- asyncio.run(
- api_server._run_job(
- record,
- request,
- file_store,
- image_analysis=True,
- )
- )
+ request = execution_request(file_store, {"type": "url", "url": url}, tier="standard", output_formats=["default"])
+ record = api_server.ExecutionStore().create(request, file_store)
- parsed_file = record.files[0]
+ asyncio.run(api_server._run_execution(record, file_store, image_analysis=True))
+
+ parsed_file = record
assert parsed_file.status == "completed"
middle_data = execution_output(file_store, record, "middle_json.json")
middle_payload = json.loads(middle_data)
@@ -187,25 +175,12 @@ def test_html_parse_server_url_accepts_extensionless_text_html(
client = _url_stream_client(response)
monkeypatch.setattr(api_server, "httpx", SimpleNamespace(AsyncClient=lambda **_: client))
file_store = FileStore(tmp_path / "api-files")
- request = _ParseRequest.model_validate(
- {
- "files": [{"source": {"type": "url", "url": url}}],
- "tier": "standard",
- "output_formats": ["default"],
- }
- )
- record = api_server.JobStore().create(request, file_store)
-
- asyncio.run(
- api_server._run_job(
- record,
- request,
- file_store,
- image_analysis=True,
- )
- )
+ request = execution_request(file_store, {"type": "url", "url": url}, tier="standard", output_formats=["default"])
+ record = api_server.ExecutionStore().create(request, file_store)
+
+ asyncio.run(api_server._run_execution(record, file_store, image_analysis=True))
- parsed_file = record.files[0]
+ parsed_file = record
assert parsed_file.status == "completed"
middle_data = execution_output(file_store, record, "middle_json.json")
middle_payload = json.loads(middle_data)
@@ -221,7 +196,7 @@ def test_html_parse_server_flash_only_admits_extensionless_url(
async def fake_run_job(*args: object, **kwargs: object) -> None:
"""跳过后台解析,仅验证 admission 已选择 Flash tier。"""
- monkeypatch.setattr(api_server, "_run_job", fake_run_job)
+ monkeypatch.setattr(api_server, "_run_execution", fake_run_job)
app = api_server.create_app(upload_dir=str(tmp_path / "api"), tier="flash")
with TestClient(app) as client:
@@ -251,29 +226,14 @@ def test_html_parse_server_no_flash_rejects_extensionless_text_html_after_fetch(
monkeypatch.setattr(api_server, "httpx", SimpleNamespace(AsyncClient=lambda **_: client))
monkeypatch.setattr(api_server, "parse_async", parse_async_mock)
file_store = FileStore(tmp_path / "api-files")
- request = _ParseRequest.model_validate(
- {
- "files": [{"source": {"type": "url", "url": url}}],
- "tier": "standard",
- }
- )
- record = api_server.JobStore().create(request, file_store)
-
- asyncio.run(
- api_server._run_job(
- record,
- request,
- file_store,
- image_analysis=True,
- flash_enabled=False,
- )
- )
+ request = execution_request(file_store, {"type": "url", "url": url}, tier="standard")
+ record = api_server.ExecutionStore().create(request, file_store)
+
+ asyncio.run(api_server._run_execution(record, file_store, image_analysis=True, flash_enabled=False))
assert record.status == "failed"
- assert record.files[0].error is not None
- assert record.files[0].error.message == (
- "Flash parsing is disabled in this server, but this input requires the Flash backend"
- )
+ assert record.error is not None
+ assert record.error.message == ("Flash parsing is disabled in this server, but this input requires the Flash backend")
parse_async_mock.assert_not_awaited()
@@ -289,29 +249,15 @@ def test_url_source_aborts_when_exceeding_max_url_bytes(tmp_path: Path, monkeypa
client = _url_stream_client(response)
monkeypatch.setattr(api_server, "httpx", SimpleNamespace(AsyncClient=lambda **_: client))
file_store = FileStore(tmp_path / "api-files")
- request = _ParseRequest.model_validate(
- {
- "files": [{"source": {"type": "url", "url": url}}],
- "tier": "standard",
- "output_formats": ["default"],
- }
- )
- record = api_server.JobStore().create(request, file_store)
-
- asyncio.run(
- api_server._run_job(
- record,
- request,
- file_store,
- image_analysis=True,
- max_url_bytes=64,
- )
- )
+ request = execution_request(file_store, {"type": "url", "url": url}, tier="standard", output_formats=["default"])
+ record = api_server.ExecutionStore().create(request, file_store)
+
+ asyncio.run(api_server._run_execution(record, file_store, image_analysis=True, max_url_bytes=64))
assert record.status == "failed"
- assert record.files[0].status == "failed"
- assert record.files[0].error is not None
- assert record.files[0].error.message == "url source exceeds max_url_bytes (64)"
+ assert record.status == "failed"
+ assert record.error is not None
+ assert record.error.message == "url source exceeds max_url_bytes (64)"
def test_html_local_base_images_styles_and_escape_are_bounded(tmp_path: Path) -> None:
@@ -359,25 +305,12 @@ def test_html_parse_server_upload_keeps_embedded_assets(tmp_path: Path) -> None:
encoding="utf-8",
)
file_store = FileStore(tmp_path / "api-files")
- request = _ParseRequest.model_validate(
- {
- "files": [{"source": stored_source(file_store, source)}],
- "tier": "standard",
- "output_formats": ["default"],
- }
- )
- record = api_server.JobStore().create(request, file_store)
-
- asyncio.run(
- api_server._run_job(
- record,
- request,
- file_store,
- image_analysis=True,
- )
- )
+ request = execution_request(file_store, stored_source(file_store, source), tier="standard", output_formats=["default"])
+ record = api_server.ExecutionStore().create(request, file_store)
- parsed_file = record.files[0]
+ asyncio.run(api_server._run_execution(record, file_store, image_analysis=True))
+
+ parsed_file = record
assert parsed_file.status == "completed"
middle_data = execution_output(file_store, record, "middle_json.json")
middle_payload = json.loads(middle_data)
@@ -405,20 +338,14 @@ def test_html_remote_image_never_downloaded_and_zip_keeps_external_link(tmp_path
assert not [name for name in names if name.startswith("images/") and not name.endswith("/")]
file_store = FileStore(tmp_path / "api-files")
- request = _ParseRequest.model_validate(
- {
- "files": [{"source": stored_source(file_store, source)}],
- "tier": "standard",
- "output_formats": ["default"],
- }
- )
- record = api_server.JobStore().create(request, file_store)
+ request = execution_request(file_store, stored_source(file_store, source), tier="standard", output_formats=["default"])
+ record = api_server.ExecutionStore().create(request, file_store)
- asyncio.run(api_server._run_job(record, request, file_store, image_analysis=True))
+ asyncio.run(api_server._run_execution(record, file_store, image_analysis=True))
- parsed_file = record.files[0]
+ parsed_file = record
assert parsed_file.status == "completed"
- zip_record = file_store.get_file(record.task_result_file_id)
+ zip_record = file_store.get_file(record.result_file_id)
assert zip_record.sha256sum is not None
with ZipFile(BytesIO(file_store.read_blob(zip_record.sha256sum))) as archive:
assert url in archive.read("markdown.md").decode()
diff --git a/tests/unittest/test_flash_odf.py b/tests/unittest/test_flash_odf.py
index 75cb2b69..20395188 100644
--- a/tests/unittest/test_flash_odf.py
+++ b/tests/unittest/test_flash_odf.py
@@ -1,5 +1,6 @@
from __future__ import annotations
+from task_test_utils import execution_request
from task_test_utils import stored_source, execution_output
import asyncio
@@ -21,7 +22,7 @@ from mineru.doclib.core.fts import FTSManager
from mineru.doclib.services.parse_svc import ParseService
from mineru.errors import InvalidRequestError
from mineru.parser import api_server, parse, parse_async
-from mineru.parser.api_server import _ParseRequest, FileStore
+from mineru.parser.api_server import FileStore
@pytest.mark.parametrize(
@@ -130,23 +131,10 @@ def test_odf_parse_server_job_emits_flash_outputs(tmp_path: Path) -> None:
source = tmp_path / "sample.odt"
source.write_bytes(build_odt_fixture())
file_store = FileStore(tmp_path / "api-files")
- request = _ParseRequest.model_validate(
- {
- "files": [{"source": stored_source(file_store, source)}],
- "tier": "standard",
- "output_formats": ["default"],
- }
- )
- record = api_server.JobStore().create(request, file_store)
- asyncio.run(
- api_server._run_job(
- record,
- request,
- file_store,
- image_analysis=True,
- )
- )
- parsed_file = record.files[0]
+ request = execution_request(file_store, stored_source(file_store, source), tier="standard", output_formats=["default"])
+ record = api_server.ExecutionStore().create(request, file_store)
+ asyncio.run(api_server._run_execution(record, file_store, image_analysis=True))
+ parsed_file = record
assert parsed_file.status == "completed"
middle_data = execution_output(file_store, record, "middle_json.json")
payload = json.loads(middle_data)
diff --git a/tests/unittest/test_flash_ofd.py b/tests/unittest/test_flash_ofd.py
index c3f45be7..0a6b5e78 100644
--- a/tests/unittest/test_flash_ofd.py
+++ b/tests/unittest/test_flash_ofd.py
@@ -1,5 +1,6 @@
from __future__ import annotations
+from task_test_utils import execution_request
from task_test_utils import stored_source, execution_output
import asyncio
@@ -20,7 +21,7 @@ from mineru.doclib.core.fts import FTSManager
from mineru.doclib.services.parse_svc import ParseService
from mineru.errors import InvalidRequestError
from mineru.parser import MinerUParser, api_server
-from mineru.parser.api_server import _ParseRequest, FileStore
+from mineru.parser.api_server import FileStore
from mineru.render import render_docx, render_html, render_markdown, render_structured_content
from mineru.types import BlockType
@@ -83,23 +84,10 @@ def test_ofd_parse_server_job_emits_flash_outputs(tmp_path: Path) -> None:
source = tmp_path / "sample.ofd"
source.write_bytes(_minimal_payload())
file_store = FileStore(tmp_path / "api-files")
- request = _ParseRequest.model_validate(
- {
- "files": [{"source": stored_source(file_store, source)}],
- "tier": "standard",
- "output_formats": ["default"],
- }
- )
- record = api_server.JobStore().create(request, file_store)
- asyncio.run(
- api_server._run_job(
- record,
- request,
- file_store,
- image_analysis=True,
- )
- )
- parsed_file = record.files[0]
+ request = execution_request(file_store, stored_source(file_store, source), tier="standard", output_formats=["default"])
+ record = api_server.ExecutionStore().create(request, file_store)
+ asyncio.run(api_server._run_execution(record, file_store, image_analysis=True))
+ parsed_file = record
assert parsed_file.status == "completed"
middle_data = execution_output(file_store, record, "middle_json.json")
payload = json.loads(middle_data)
diff --git a/tests/unittest/test_flash_pdf_render_scheduling.py b/tests/unittest/test_flash_pdf_render_scheduling.py
index a604885c..a950b575 100644
--- a/tests/unittest/test_flash_pdf_render_scheduling.py
+++ b/tests/unittest/test_flash_pdf_render_scheduling.py
@@ -17,7 +17,7 @@ from docvortex.document.pdf import PDFDocument, images
from mineru.backend.analysis.pdf import window
from mineru.backend.postprocess import document as postprocess_document
from mineru.backend.postprocess import llm_aided
-from mineru.config import LLMAidedConfig, LLMAidedFeaturesConfig, config
+from mineru.config import LLMAidedConfig, LLMAidedFeaturesConfig, get_config
from mineru.parser import parse, parse_async
from mineru.types import MiddleJson, ModelJson
@@ -107,7 +107,7 @@ def test_flash_sparse_render_keeps_page_mapping_and_title_gate(
monkeypatch.setattr(llm_aided, "LLMAidedClient", MagicMock(return_value=client))
monkeypatch.setattr(llm_aided, "apply_llm_title_leveling", title)
monkeypatch.setattr(
- config,
+ get_config(),
"llm_aided",
LLMAidedConfig(
api_key="isolated-test",
diff --git a/tests/unittest/test_kit_commands.py b/tests/unittest/test_kit_commands.py
index c1b5cefd..b8e0ed05 100644
--- a/tests/unittest/test_kit_commands.py
+++ b/tests/unittest/test_kit_commands.py
@@ -497,8 +497,8 @@ def test_router_startup_with_no_servers_fails_health_check() -> None:
def test_models_download_tier_basic(monkeypatch: Any) -> None:
- monkeypatch.setattr(models.config.model, "small_backend", "torch")
- monkeypatch.setattr(models.config.model.vlm, "engine", "vllm")
+ monkeypatch.setattr(models.get_config().model, "small_backend", "torch")
+ monkeypatch.setattr(models.get_config().model.vlm, "engine", "vllm")
captured: list[str] = []
def fake_download_model_repo(repo: Any, *, source: str | None = None, local_as_auto: bool = False) -> Path:
@@ -515,8 +515,8 @@ def test_models_download_tier_basic(monkeypatch: Any) -> None:
def test_models_download_tier_standard(monkeypatch: Any) -> None:
- monkeypatch.setattr(models.config.model, "small_backend", "torch")
- monkeypatch.setattr(models.config.model.vlm, "engine", "vllm")
+ monkeypatch.setattr(models.get_config().model, "small_backend", "torch")
+ monkeypatch.setattr(models.get_config().model.vlm, "engine", "vllm")
captured: list[str] = []
def fake_download_model_repo(repo: Any, *, source: str | None = None, local_as_auto: bool = False) -> Path:
@@ -571,9 +571,9 @@ def test_models_download_repo_uses_explicit_source(monkeypatch: Any) -> None:
def test_models_show_and_verify(tmp_path: Path, monkeypatch: Any) -> None:
base_dir = tmp_path / "models"
- monkeypatch.setattr(models.config.model, "base_dir", str(base_dir))
- monkeypatch.setattr(models.config.model, "small_backend", "torch")
- monkeypatch.setattr(models.config.model.vlm, "engine", "vllm")
+ monkeypatch.setattr(models.get_config().model, "base_dir", str(base_dir))
+ monkeypatch.setattr(models.get_config().model, "small_backend", "torch")
+ monkeypatch.setattr(models.get_config().model.vlm, "engine", "vllm")
for repo in models.MODEL_REPOS:
if repo.download_mode == "full":
repo.local_dir().mkdir(parents=True, exist_ok=True)
@@ -1203,9 +1203,9 @@ def test_models_download_repo_ignores_backend_options(monkeypatch: Any) -> None:
def test_models_show_displays_backend_fields(tmp_path: Path, monkeypatch: Any) -> None:
base_dir = tmp_path / "models"
- monkeypatch.setattr(models.config.model, "base_dir", str(base_dir))
- monkeypatch.setattr(models.config.model, "small_backend", "torch")
- monkeypatch.setattr(models.config.model.vlm, "engine", "vllm")
+ monkeypatch.setattr(models.get_config().model, "base_dir", str(base_dir))
+ monkeypatch.setattr(models.get_config().model, "small_backend", "torch")
+ monkeypatch.setattr(models.get_config().model.vlm, "engine", "vllm")
result = runner.invoke(app, ["models", "show"])
@@ -1219,9 +1219,9 @@ def test_models_show_displays_backend_fields(tmp_path: Path, monkeypatch: Any) -
def test_models_show_with_onnx_backend_filter(tmp_path: Path, monkeypatch: Any) -> None:
"""显式 ONNX 覆盖当前命令的小模型选择,不修改全局配置。"""
base_dir = tmp_path / "models"
- monkeypatch.setattr(models.config.model, "base_dir", str(base_dir))
- monkeypatch.setattr(models.config.model, "small_backend", "torch")
- monkeypatch.setattr(models.config.model.vlm, "engine", "vllm")
+ monkeypatch.setattr(models.get_config().model, "base_dir", str(base_dir))
+ monkeypatch.setattr(models.get_config().model, "small_backend", "torch")
+ monkeypatch.setattr(models.get_config().model.vlm, "engine", "vllm")
result = runner.invoke(app, ["models", "show", "--small-backend", "onnx", "--vlm-engine", "llama-cpp"])
@@ -1241,9 +1241,9 @@ def test_models_show_rejects_invalid_small_backend() -> None:
def test_models_verify_filters_by_effective_backends(tmp_path: Path, monkeypatch: Any) -> None:
"""默认校验只检查当前 Torch 与 VLM 引擎实际需要的仓库。"""
base_dir = tmp_path / "models"
- monkeypatch.setattr(models.config.model, "base_dir", str(base_dir))
- monkeypatch.setattr(models.config.model, "small_backend", "torch")
- monkeypatch.setattr(models.config.model.vlm, "engine", "vllm")
+ monkeypatch.setattr(models.get_config().model, "base_dir", str(base_dir))
+ monkeypatch.setattr(models.get_config().model, "small_backend", "torch")
+ monkeypatch.setattr(models.get_config().model.vlm, "engine", "vllm")
# 仅准备当前后端所需仓库,其余仓库保持缺失。
for repo in models.MODEL_REPOS:
@@ -1272,7 +1272,7 @@ def test_models_verify_filters_by_effective_backends(tmp_path: Path, monkeypatch
def test_models_verify_with_onnx_backend(tmp_path: Path, monkeypatch: Any) -> None:
"""显式 ONNX 只校验 ONNX 与所选 VLM 权重。"""
base_dir = tmp_path / "models"
- monkeypatch.setattr(models.config.model, "base_dir", str(base_dir))
+ monkeypatch.setattr(models.get_config().model, "base_dir", str(base_dir))
result = runner.invoke(app, ["models", "verify", "--small-backend", "onnx", "--vlm-engine", "llama-cpp"])
@@ -1284,7 +1284,7 @@ def test_models_verify_with_onnx_backend(tmp_path: Path, monkeypatch: Any) -> No
def test_models_verify_repo_ignores_backend_options(tmp_path: Path, monkeypatch: Any) -> None:
"""传具体 repo 名时后端选项被忽略。"""
base_dir = tmp_path / "models"
- monkeypatch.setattr(models.config.model, "base_dir", str(base_dir))
+ monkeypatch.setattr(models.get_config().model, "base_dir", str(base_dir))
# 聚合 ONNX 仓库必须具有每个必需文件,根目录标记不能替代文件检查。
repo = next(r for r in models.MODEL_REPOS if r.name == "MinerU-4_models_onnx")
diff --git a/tests/unittest/test_kit_gradio.py b/tests/unittest/test_kit_gradio.py
index fa41351b..4ae4b394 100644
--- a/tests/unittest/test_kit_gradio.py
+++ b/tests/unittest/test_kit_gradio.py
@@ -26,6 +26,8 @@ from mineru.filetypes import FLASH_ONLY_PARSE_EXTENSIONS, IMAGE_EXTENSIONS, PARS
from mineru.integrations.docvortex import build_metadata
from mineru.kit.commands import webui as webui_command
from mineru.kit.gradio import app as gradio_app
+from mineru.kit.gradio import conversion_service
+from mineru.kit.gradio.page_range import effective_page_range
from mineru.kit.gradio import client as gradio_client
from mineru.kit.gradio.app import build_gradio_app
from mineru.kit.gradio.artifacts import create_run_artifacts, persist_parse_result, render_download
@@ -1043,8 +1045,8 @@ def test_gradio_file_types_page_range_and_header_follow_new_contract(tmp_path: P
assert set(gradio_app._supported_file_types()) == {f".{extension}" for extension in PARSEABLE_EXTENSIONS}
source = tmp_path / "report.PDF"
source.write_bytes(_pdf_bytes(5))
- assert gradio_app._effective_page_range(source, " 1-3,r1 ") == "1-3,r1"
- assert gradio_app._effective_page_range("report.docx", "1-3") == ""
+ assert effective_page_range(source, " 1-3,r1 ") == "1-3,r1"
+ assert effective_page_range("report.docx", "1-3") == ""
header = gradio_app._render_header()
assert "mineru-demo-header" in header
assert "mineru-header-popover mineru-model-popover" in header
@@ -1544,7 +1546,7 @@ def test_gradio_output_failure_stops_timer_and_allows_next_conversion(tmp_path:
"""模拟产物磁盘写入失败。"""
raise OSError("output disk unavailable")
- monkeypatch.setattr(gradio_app, "persist_parse_result", fail_persistence)
+ monkeypatch.setattr(conversion_service, "persist_parse_result", fail_persistence)
capabilities = V1ServerCapabilities("http://127.0.0.1:1", ("flash",), ("default",), ("upload",))
demo = build_gradio_app(CompletedClient(), capabilities, output_root=tmp_path, enable_example=False)
handler = next(fn.fn for fn in demo.fns.values() if fn.name == "convert_handler")
@@ -1658,7 +1660,7 @@ def test_gradio_slow_precheck_keeps_polling_available_and_queues_other_sessions(
assert release.wait(10)
return "1"
- monkeypatch.setattr(gradio_app, "_effective_page_range", precheck)
+ monkeypatch.setattr(conversion_service, "effective_page_range", precheck)
client = SimpleNamespace(parse_file=AsyncMock(side_effect=V1ArtifactError("test finished")))
capabilities = V1ServerCapabilities("http://127.0.0.1:1", ("flash",), ("default",), ("upload",))
demo = build_gradio_app(client, capabilities, output_root=tmp_path / "output", enable_example=False)
diff --git a/tests/unittest/test_kit_gradio_conversion.py b/tests/unittest/test_kit_gradio_conversion.py
index cb492961..7e728273 100644
--- a/tests/unittest/test_kit_gradio_conversion.py
+++ b/tests/unittest/test_kit_gradio_conversion.py
@@ -16,6 +16,7 @@ from urllib.parse import unquote
import pytest
+from mineru.kit.gradio import conversion_service
from mineru.kit.gradio import app as gradio_app
from mineru.kit.gradio.client import V1ArtifactError, V1ServerCapabilities
from mineru.kit.gradio.conversion import ConversionRun, SessionConversions
@@ -280,7 +281,7 @@ def test_cancelled_sync_output_keeps_slot_until_thread_exits(tmp_path: Path, mon
source = tmp_path / "source.pdf"
source.write_bytes(_pdf_bytes())
started, release = threading.Event(), threading.Event()
- persist = gradio_app.persist_parse_result
+ persist = conversion_service.persist_parse_result
client = SimpleNamespace(parse_file=AsyncMock(return_value=ParseResult(middle_json=_middle_json(with_image=False))))
calls = 0
@@ -293,7 +294,7 @@ def test_cancelled_sync_output_keeps_slot_until_thread_exits(tmp_path: Path, mon
assert release.wait(10)
return persist(*args, **kwargs)
- monkeypatch.setattr(gradio_app, "persist_parse_result", slow_output)
+ monkeypatch.setattr(conversion_service, "persist_parse_result", slow_output)
app = _application(tmp_path, client)
convert, poll, cancel = (_callback(app, name) for name in ("convert_ui", "read_conversion_status", "cancel_ui_conversion"))
request = SimpleNamespace(session_hash="same-session")
diff --git a/tests/unittest/test_kit_gradio_conversion_service.py b/tests/unittest/test_kit_gradio_conversion_service.py
new file mode 100644
index 00000000..de7264c5
--- /dev/null
+++ b/tests/unittest/test_kit_gradio_conversion_service.py
@@ -0,0 +1,139 @@
+"""不构建 Gradio 界面的转换服务行为与结果生命周期回归。"""
+
+from __future__ import annotations
+
+import asyncio
+import gc
+import weakref
+from collections.abc import Callable
+from pathlib import Path
+from typing import Literal
+
+from mineru.kit.gradio.conversion import (
+ ConversionCancelled,
+ ConversionFailure,
+ ConversionInput,
+ ConversionRun,
+ ConversionSuccess,
+)
+from mineru.kit.gradio.conversion_service import ConversionService
+from mineru.kit.gradio.status import STATUS_COMPLETED, STATUS_PROCESSING_ON_SERVER, ParseStatusUpdate
+from mineru.parser.base import ParseResult
+from tests.unittest.test_kit_gradio import _middle_json
+
+
+class _ParseClient:
+ """记录服务实际选择的参数,只通过弱引用观察解析结果生命周期。"""
+
+ def __init__(self, *, cancel: bool = False) -> None:
+ """保存可控取消行为与调用记录,不缓存解析结果。"""
+ self.cancel = cancel
+ self.calls: list[tuple[Path, str, str, str]] = []
+ self.result_ref: weakref.ReferenceType[ParseResult] | None = None
+
+ async def parse_file(
+ self,
+ path: Path,
+ *,
+ tier: str,
+ page_range: str,
+ ocr_mode: Literal["auto", "txt", "ocr"] = "auto",
+ status_callback: Callable[[str | ParseStatusUpdate], None] | None = None,
+ ) -> ParseResult:
+ """构造一次性解析结果,避免模拟客户端自身延长结果生命周期。"""
+ self.calls.append((path, tier, page_range, ocr_mode))
+ if self.cancel:
+ raise asyncio.CancelledError
+ if status_callback is not None:
+ status_callback(STATUS_PROCESSING_ON_SERVER)
+ result = ParseResult(middle_json=_middle_json(with_image=False, file_suffix="csv"))
+ self.result_ref = weakref.ref(result)
+ return result
+
+
+def test_service_returns_materialized_output_without_retaining_parse_result(tmp_path: Path) -> None:
+ """直接转换 CSV,验证实际档位、具名结果和会话缓存均不保留解析树。"""
+ source = tmp_path / "source.csv"
+ source.write_text("name,value\na,1\n", encoding="utf-8")
+ client = _ParseClient()
+ run = ConversionRun("direct-conversion")
+
+ async def scenario() -> None:
+ """原生格式强制 Flash,忽略 PDF 选页与 OCR,并物化可下载结果。"""
+ service = ConversionService(
+ client,
+ tier_choices=("flash", "basic"),
+ output_root=tmp_path / "output",
+ max_pages=10,
+ slot=asyncio.Semaphore(1),
+ )
+ outcome = await service.execute(
+ run,
+ ConversionInput(str(source), 1, "1-3", True),
+ public_base_url="https://example.test/mineru",
+ )
+ assert isinstance(outcome, ConversionSuccess)
+ assert client.calls == [(source.resolve(), "flash", "", "auto")]
+ output = outcome.output
+ assert output.artifacts.middle_json_path.is_file()
+ assert output.artifacts.structured_content_path.read_text(encoding="utf-8") == output.structured_json
+ assert output.html
+ assert output.preview.show_ready_placeholder and not output.preview.preserve_source
+ assert output.preview.pdf_path is None and output.preview.image_path is None
+ assert run.artifacts == output.artifacts.as_state()
+ assert run.task is None and run.state.message == STATUS_COMPLETED
+ await asyncio.sleep(0)
+ gc.collect()
+ assert client.result_ref is not None and client.result_ref() is None
+
+ asyncio.run(scenario())
+
+
+def test_service_rejects_input_before_waiting_for_slot(tmp_path: Path) -> None:
+ """无文件请求直接形成失败结果,不等待已占用的解析槽。"""
+ client = _ParseClient()
+ run = ConversionRun("invalid-input")
+
+ async def scenario() -> None:
+ """使用没有空位的执行槽,确保输入错误不进入排队和解析。"""
+ service = ConversionService(
+ client,
+ tier_choices=("flash",),
+ output_root=tmp_path,
+ max_pages=None,
+ slot=asyncio.Semaphore(0),
+ )
+ outcome = await asyncio.wait_for(
+ service.execute(run, ConversionInput(None, 0, ""), public_base_url="https://example.test"),
+ timeout=1,
+ )
+ assert isinstance(outcome, ConversionFailure) and outcome.message == "No input file"
+ assert run.terminal and run.state.message == "Failed: No input file"
+ assert run.task is None and run.artifacts is None and not client.calls
+
+ asyncio.run(scenario())
+
+
+def test_service_returns_cancelled_outcome_and_releases_slot(tmp_path: Path) -> None:
+ """解析取消只返回取消结果,不能生成成功素材或留下执行槽。"""
+ source = tmp_path / "source.csv"
+ source.write_text("a,b\n1,2\n", encoding="utf-8")
+ run = ConversionRun("cancelled-input")
+
+ async def scenario() -> None:
+ """等待客户端取消传播完成后检查任务句柄、素材状态和槽位。"""
+ slot = asyncio.Semaphore(1)
+ service = ConversionService(
+ _ParseClient(cancel=True),
+ tier_choices=("flash",),
+ output_root=tmp_path / "output",
+ max_pages=None,
+ slot=slot,
+ )
+ outcome = await service.execute(run, ConversionInput(str(source), 0, ""), public_base_url="https://example.test")
+ assert isinstance(outcome, ConversionCancelled)
+ assert run.cancelled and run.task is None and run.artifacts is None
+ assert not slot.locked()
+ assert not (tmp_path / "output").exists()
+
+ asyncio.run(scenario())
diff --git a/tests/unittest/test_kit_gradio_i18n.py b/tests/unittest/test_kit_gradio_i18n.py
index e7cc8a80..909ecd2d 100644
--- a/tests/unittest/test_kit_gradio_i18n.py
+++ b/tests/unittest/test_kit_gradio_i18n.py
@@ -91,7 +91,7 @@ def test_conversion_preserves_all_preview_components_until_success(
outcome: str,
) -> None:
"""暂停普通转换请求,确认处理期间不发送预览更新且失败保留已有四种预览。"""
- from mineru.kit.gradio import app as app_module
+ from mineru.kit.gradio import conversion_service
source = tmp_path / "document.csv"
source.write_text("a,b\n1,2\n", encoding="utf-8")
@@ -115,7 +115,7 @@ def test_conversion_preserves_all_preview_components_until_success(
raise OSError("test output failure")
if outcome == "output_failure":
- monkeypatch.setattr(app_module, "persist_parse_result", fail_output)
+ monkeypatch.setattr(conversion_service, "persist_parse_result", fail_output)
demo = build_gradio_app(
SimpleNamespace(parse_file=parse_file),
V1ServerCapabilities("http://localhost:1", ("flash",), ("zip",), ("file_id",)),
diff --git a/tests/unittest/test_kit_gradio_office_preview.py b/tests/unittest/test_kit_gradio_office_preview.py
index 7e1b4299..b3a31c5e 100644
--- a/tests/unittest/test_kit_gradio_office_preview.py
+++ b/tests/unittest/test_kit_gradio_office_preview.py
@@ -14,6 +14,7 @@ import pytest
from mineru.filetypes import OFFICE_EXTENSIONS
from mineru.kit.gradio import app as app_module
+from mineru.kit.gradio import conversion_service
from mineru.kit.gradio.client import V1ServerCapabilities
from mineru.kit.gradio.i18n import MESSAGES
from mineru.parser.base import ParseResult
@@ -115,7 +116,7 @@ def test_office_conversion_never_replaces_uploaded_preview(
raise OSError("output unavailable")
if outcome == "output_failure":
- monkeypatch.setattr(app_module, "persist_parse_result", fail_output)
+ monkeypatch.setattr(conversion_service, "persist_parse_result", fail_output)
demo = app_module.build_gradio_app(
SimpleNamespace(parse_file=parse),
V1ServerCapabilities("http://localhost:1", ("flash",), ("zip",), ("file_id",)),
diff --git a/tests/unittest/test_markdown_render.py b/tests/unittest/test_markdown_render.py
index ad7b2cc7..b2b469f0 100644
--- a/tests/unittest/test_markdown_render.py
+++ b/tests/unittest/test_markdown_render.py
@@ -43,7 +43,7 @@ def test_equation_uses_content_then_image_fallback(monkeypatch: pytest.MonkeyPat
}
}
)
- monkeypatch.setattr("mineru.config.config", configured)
+ monkeypatch.setattr("mineru.config.get_config", lambda: configured)
middle = _middle(
_page(
0,
diff --git a/tests/unittest/test_model_backend_selection.py b/tests/unittest/test_model_backend_selection.py
index ca20eacf..7b16ad1e 100644
--- a/tests/unittest/test_model_backend_selection.py
+++ b/tests/unittest/test_model_backend_selection.py
@@ -12,7 +12,7 @@ from packaging.requirements import Requirement
from pydantic import ValidationError
from typer.testing import CliRunner
-from mineru.config import Config, VlmConfig, _load_effective_config, config
+from mineru.config import Config, VlmConfig, _load_effective_config, get_config
from mineru.kit.commands.models import app
from mineru.model import registry
from mineru.model.runtime import device
@@ -106,8 +106,8 @@ def test_old_cli_stack_option_is_removed() -> None:
@pytest.mark.parametrize("engine", ["llama-cpp", "vllm", "lmdeploy", "mlx"])
def test_preflight_uses_supplied_vlm_config(monkeypatch: pytest.MonkeyPatch, engine: str) -> None:
"""显式 VLM 配置同时决定预检依赖与资源集合,不被全局设置覆盖。"""
- monkeypatch.setattr(config.model, "small_backend", "torch")
- monkeypatch.setattr(config.model.vlm, "engine", "auto")
+ monkeypatch.setattr(get_config().model, "small_backend", "torch")
+ monkeypatch.setattr(get_config().model.vlm, "engine", "auto")
settings = VlmConfig(engine=engine)
modules = parser_tier.required_modules_for_tier("standard", vlm_config=settings)
assert set(modules) == {"onnxruntime", *device.TORCH_REQUIRED_MODULES, *selector.VLM_REQUIRED_MODULES[engine]}
@@ -126,7 +126,7 @@ def test_preflight_uses_supplied_vlm_config(monkeypatch: pytest.MonkeyPatch, eng
def test_remote_vlm_needs_no_local_engine_or_weights(monkeypatch: pytest.MonkeyPatch) -> None:
"""远程地址优先,不因未安装本地 MLX 而阻断服务或下载无用 VLM 权重。"""
- monkeypatch.setattr(config.model, "small_backend", "onnx")
+ monkeypatch.setattr(get_config().model, "small_backend", "onnx")
settings = VlmConfig(server_url="http://localhost:9000", engine="mlx")
assert parser_tier.required_modules_for_tier("standard", vlm_config=settings) == ["onnxruntime"]
assert registry.model_repos_for_tier("standard", vlm_config=settings) == (registry.MINERU_4_MODELS_ONNX,)
@@ -136,7 +136,7 @@ def test_remote_vlm_needs_no_local_engine_or_weights(monkeypatch: pytest.MonkeyP
def test_explicit_missing_engine_reports_dependency(monkeypatch: pytest.MonkeyPatch) -> None:
"""显式引擎缺失时报错,不改用可用的 llama 掩盖配置问题。"""
- monkeypatch.setattr(config.model, "small_backend", "onnx")
+ monkeypatch.setattr(get_config().model, "small_backend", "onnx")
def missing_engine(name: str) -> object:
"""只模拟选定引擎缺失,避免加载真实 GPU 模块。"""
@@ -170,8 +170,8 @@ def test_standard_dependency_errors_survive_startup_wrappers(
from mineru.parser import api_server
settings = VlmConfig(engine=engine)
- monkeypatch.setattr(config.model, "small_backend", small_backend)
- monkeypatch.setattr(config.model, "vlm", settings)
+ monkeypatch.setattr(get_config().model, "small_backend", small_backend)
+ monkeypatch.setattr(get_config().model, "vlm", settings)
monkeypatch.setattr(parser_tier, "installed_distribution_name", lambda: "mineru")
def import_without_dependency(name: str) -> object:
diff --git a/tests/unittest/test_models_download_utils.py b/tests/unittest/test_models_download_utils.py
index be4b81ca..1275f3e2 100644
--- a/tests/unittest/test_models_download_utils.py
+++ b/tests/unittest/test_models_download_utils.py
@@ -10,7 +10,7 @@ from mineru.model.registry import MODEL_COMPLETE_MARKER, MINERU_2_5_PRO_2605_1_2
def test_resolve_model_source_does_not_persist_env_auto(monkeypatch: pytest.MonkeyPatch) -> None:
- monkeypatch.setattr(models_download_utils.config.model, "source", "auto")
+ monkeypatch.setattr(models_download_utils.get_config().model, "source", "auto")
monkeypatch.setattr(models_download_utils, "get_config_source", lambda path: "env")
monkeypatch.setattr(models_download_utils, "_resolve_auto_model_source", lambda: "modelscope")
@@ -25,7 +25,7 @@ def test_resolve_model_source_does_not_persist_env_auto(monkeypatch: pytest.Monk
def test_resolve_model_source_can_treat_config_local_as_auto_for_explicit_download(
monkeypatch: pytest.MonkeyPatch,
) -> None:
- monkeypatch.setattr(models_download_utils.config.model, "source", "local")
+ monkeypatch.setattr(models_download_utils.get_config().model, "source", "local")
monkeypatch.setattr(models_download_utils, "_resolve_auto_model_source", lambda: "huggingface")
assert models_download_utils.resolve_model_source(allow_auto=True) == "local"
@@ -50,7 +50,7 @@ def test_download_model_repo_uses_required_path_patterns(
)
captured: dict[str, object] = {}
- monkeypatch.setattr(models_download_utils.config.model, "base_dir", str(tmp_path / "models"))
+ monkeypatch.setattr(models_download_utils.get_config().model, "base_dir", str(tmp_path / "models"))
monkeypatch.setattr(models_download_utils, "resolve_model_source", lambda *args, **kwargs: "modelscope")
def fake_snapshot_download(model_source: str, target_repo: ModelRepo, patterns: list[str] | None) -> str:
@@ -87,7 +87,7 @@ def test_download_model_repo_keeps_full_download_as_default(
)
captured: dict[str, object] = {}
- monkeypatch.setattr(models_download_utils.config.model, "base_dir", str(tmp_path / "models"))
+ monkeypatch.setattr(models_download_utils.get_config().model, "base_dir", str(tmp_path / "models"))
monkeypatch.setattr(models_download_utils, "resolve_model_source", lambda *args, **kwargs: "huggingface")
def fake_snapshot_download(model_source: str, target_repo: ModelRepo, patterns: list[str] | None) -> str:
@@ -113,7 +113,7 @@ def test_download_model_repo_keeps_full_download_as_default(
def test_vlm_repo_verify_requires_root_completion_marker(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
- monkeypatch.setattr(models_download_utils.config.model, "base_dir", str(tmp_path / "models"))
+ monkeypatch.setattr(models_download_utils.get_config().model, "base_dir", str(tmp_path / "models"))
repo_dir = MINERU_2_5_PRO_2605_1_2B.local_dir()
repo_dir.mkdir(parents=True)
@@ -142,7 +142,7 @@ def test_model_path_exists_requires_completion_marker_for_directory(
repos={"huggingface": "owner/repo", "modelscope": "Owner/repo"},
paths={"weights": "models/weights", "config": "config.json"},
)
- monkeypatch.setattr(models_download_utils.config.model, "base_dir", str(tmp_path / "models"))
+ monkeypatch.setattr(models_download_utils.get_config().model, "base_dir", str(tmp_path / "models"))
weights_dir = repo.weights.local_path()
weights_dir.mkdir(parents=True)
@@ -172,7 +172,7 @@ def test_model_path_ensure_reuses_completed_directory_without_resolving_source(
repos={"huggingface": "owner/repo", "modelscope": "Owner/repo"},
paths={"weights": "models/weights"},
)
- monkeypatch.setattr(models_download_utils.config.model, "base_dir", str(tmp_path / "models"))
+ monkeypatch.setattr(models_download_utils.get_config().model, "base_dir", str(tmp_path / "models"))
weights_dir = repo.weights.local_path()
weights_dir.mkdir(parents=True)
(weights_dir / "model.bin").write_bytes(b"weights")
@@ -196,7 +196,7 @@ def test_model_repo_ensure_reuses_completed_full_repo_without_resolving_source(
repos={"huggingface": "owner/repo", "modelscope": "Owner/repo"},
paths={"config": "config.json", "weights": "model.bin"},
)
- monkeypatch.setattr(models_download_utils.config.model, "base_dir", str(tmp_path / "models"))
+ monkeypatch.setattr(models_download_utils.get_config().model, "base_dir", str(tmp_path / "models"))
repo.local_dir().mkdir(parents=True)
repo.config.local_path().write_text("{}", encoding="utf-8")
repo.weights.local_path().write_bytes(b"weights")
@@ -220,7 +220,7 @@ def test_model_path_ensure_repairs_unmarked_directory_and_writes_marker(
repos={"huggingface": "owner/repo", "modelscope": "Owner/repo"},
paths={"weights": "models/weights"},
)
- monkeypatch.setattr(models_download_utils.config.model, "base_dir", str(tmp_path / "models"))
+ monkeypatch.setattr(models_download_utils.get_config().model, "base_dir", str(tmp_path / "models"))
weights_dir = repo.weights.local_path()
weights_dir.mkdir(parents=True)
(weights_dir / "existing.bin").write_bytes(b"existing")
@@ -251,8 +251,8 @@ def test_local_source_rejects_unmarked_directory(
repos={"huggingface": "owner/repo", "modelscope": "Owner/repo"},
paths={"weights": "models/weights"},
)
- monkeypatch.setattr(models_download_utils.config.model, "base_dir", str(tmp_path / "models"))
- monkeypatch.setattr(models_download_utils.config.model, "source", "local")
+ monkeypatch.setattr(models_download_utils.get_config().model, "base_dir", str(tmp_path / "models"))
+ monkeypatch.setattr(models_download_utils.get_config().model, "source", "local")
weights_dir = repo.weights.local_path()
weights_dir.mkdir(parents=True)
(weights_dir / "partial.bin").write_bytes(b"partial")
@@ -274,7 +274,7 @@ def test_download_failure_removes_stale_directory_marker(
paths={"weights": "models/weights"},
download_mode="required_paths",
)
- monkeypatch.setattr(models_download_utils.config.model, "base_dir", str(tmp_path / "models"))
+ monkeypatch.setattr(models_download_utils.get_config().model, "base_dir", str(tmp_path / "models"))
weights_dir = repo.weights.local_path()
weights_dir.mkdir(parents=True)
marker = weights_dir / MODEL_COMPLETE_MARKER
@@ -301,7 +301,7 @@ def test_full_download_failure_removes_root_marker(
repos={"huggingface": "owner/repo", "modelscope": "Owner/repo"},
paths={"config": "config.json"},
)
- monkeypatch.setattr(models_download_utils.config.model, "base_dir", str(tmp_path / "models"))
+ monkeypatch.setattr(models_download_utils.get_config().model, "base_dir", str(tmp_path / "models"))
repo.local_dir().mkdir(parents=True)
marker = repo.local_dir() / MODEL_COMPLETE_MARKER
marker.touch()
@@ -327,7 +327,7 @@ def test_nested_download_failure_invalidates_parent_directory_marker(
repos={"huggingface": "owner/repo", "modelscope": "Owner/repo"},
paths={"weights": "models/weights"},
)
- monkeypatch.setattr(models_download_utils.config.model, "base_dir", str(tmp_path / "models"))
+ monkeypatch.setattr(models_download_utils.get_config().model, "base_dir", str(tmp_path / "models"))
nested_weights = repo.weights.path("det")
nested_weights.local_path().mkdir(parents=True)
parent_marker = repo.weights.local_path() / MODEL_COMPLETE_MARKER
@@ -354,7 +354,7 @@ def test_nested_download_success_restores_parent_directory_marker(
repos={"huggingface": "owner/repo", "modelscope": "Owner/repo"},
paths={"weights": "models/weights"},
)
- monkeypatch.setattr(models_download_utils.config.model, "base_dir", str(tmp_path / "models"))
+ monkeypatch.setattr(models_download_utils.get_config().model, "base_dir", str(tmp_path / "models"))
nested_weights = repo.weights.path("det")
nested_weights.local_path().mkdir(parents=True)
parent_marker = repo.weights.local_path() / MODEL_COMPLETE_MARKER
@@ -386,7 +386,7 @@ def test_download_does_not_mark_metadata_only_directory(
paths={"weights": "models/weights"},
download_mode="required_paths",
)
- monkeypatch.setattr(models_download_utils.config.model, "base_dir", str(tmp_path / "models"))
+ monkeypatch.setattr(models_download_utils.get_config().model, "base_dir", str(tmp_path / "models"))
weights_dir = repo.weights.local_path()
def fake_snapshot_download(model_source: str, target_repo: ModelRepo, patterns: list[str] | None) -> str:
@@ -416,7 +416,7 @@ def test_huggingface_snapshot_rejects_missing_expected_files(
repos={"huggingface": "owner/repo", "modelscope": "Owner/repo"},
paths={"weights": "models/weights"},
)
- monkeypatch.setattr(models_download_utils.config.model, "base_dir", str(tmp_path / "models"))
+ monkeypatch.setattr(models_download_utils.get_config().model, "base_dir", str(tmp_path / "models"))
class FakeApi:
def list_repo_files(self, repo_id: str) -> list[str]:
@@ -453,9 +453,9 @@ def test_resolve_small_model_backend_explicit() -> None:
def test_resolve_small_model_backend_auto(monkeypatch: pytest.MonkeyPatch) -> None:
"""自动模式同时检查小模型依赖和可用设备。"""
from mineru.model.runtime import device
- from mineru.config import config
+ from mineru.config import get_config
- monkeypatch.setattr(config.model, "small_backend", "auto")
+ monkeypatch.setattr(get_config().model, "small_backend", "auto")
monkeypatch.setattr(device, "module_available", lambda name: True)
monkeypatch.setattr(device, "get_device", lambda: "cpu")
assert device.resolve_small_model_backend() == "onnx"
diff --git a/tests/unittest/test_native_async_runtime.py b/tests/unittest/test_native_async_runtime.py
index b936838e..e410c076 100644
--- a/tests/unittest/test_native_async_runtime.py
+++ b/tests/unittest/test_native_async_runtime.py
@@ -9,7 +9,9 @@ from typing import Any
import pytest
-from mineru.model.vlm.async_runtime import AsyncVlmPredictor, RuntimeOwner, runtime_owner
+from mineru.model.runtime.ownership import RuntimeOwner, runtime_owner
+from mineru.model.vlm.async_runtime import AsyncVlmPredictor
+from mineru.model.vlm.handles import RuntimeHandle
from mineru.utils.async_utils import run_sync
@@ -66,7 +68,7 @@ class _AsyncClient:
@pytest.fixture
def native_runtime() -> Any:
"""创建可跨线程使用的原生异步运行时并保证用例后关闭。"""
- runtime = AsyncVlmPredictor(_AsyncClient, lambda _: None, backend="test", max_concurrency=2)
+ runtime = AsyncVlmPredictor(lambda: RuntimeHandle(_AsyncClient(), backend="test"), backend="test", max_concurrency=2)
runtime.wait_ready()
try:
yield runtime
@@ -114,14 +116,16 @@ def test_repeated_cancel_waits_for_engine_cleanup(native_runtime: AsyncVlmPredic
def test_shutdown_drains_active_requests_and_rejects_new(native_runtime: AsyncVlmPredictor) -> None:
"""关闭时等待在途请求真正退出,然后拒绝旧代理继续提交。"""
- native_runtime._predictor.block = True
+ client = native_runtime._predictor
+ client.block = True
with ThreadPoolExecutor() as executor:
future = executor.submit(native_runtime.batch_two_step_extract, [1])
assert native_runtime._predictor.entered.wait(2)
native_runtime.shutdown()
with pytest.raises(asyncio.CancelledError):
future.result()
- assert native_runtime._predictor.closed
+ assert client.closed
+ assert native_runtime._predictor is None
assert not native_runtime._thread.is_alive()
with pytest.raises(RuntimeError, match="shutting down"):
native_runtime.batch_two_step_extract([2])
@@ -143,7 +147,9 @@ def test_application_owners_share_and_release_runtime(monkeypatch: pytest.Monkey
from mineru.model.vlm.runtime import ModelSingleton
monkeypatch.setattr(ModelSingleton, "_models", {})
- monkeypatch.setattr(ModelSingleton, "_create_model", lambda *args, **kwargs: _AsyncClient())
+ monkeypatch.setattr(
+ ModelSingleton, "_create_model", lambda *args, **kwargs: RuntimeHandle(_AsyncClient(), backend="http-client")
+ )
manager = ModelSingleton()
first, second = RuntimeOwner(), RuntimeOwner()
@@ -157,12 +163,14 @@ def test_application_owners_share_and_release_runtime(monkeypatch: pytest.Monkey
try:
predictor = get(first)
+ client = predictor._predictor
assert get(second) is predictor
manager.release_owner(first)
assert predictor.batch_two_step_extract([1]) == [1]
assert not predictor._predictor.closed
manager.release_owner(second)
- assert predictor._predictor.closed
+ assert client.closed
+ assert predictor._predictor is None
replacement = get(None)
assert replacement is not predictor
third = RuntimeOwner()
@@ -182,13 +190,13 @@ def test_cancelled_initialization_is_drained_before_owner_release(monkeypatch: p
entered, release = threading.Event(), threading.Event()
clients = []
- def factory(*args: Any, **kwargs: Any) -> _AsyncClient:
+ def factory(*args: Any, **kwargs: Any) -> RuntimeHandle:
"""模拟无法强行中断的模型初始化。"""
entered.set()
assert release.wait(3)
client = _AsyncClient()
clients.append(client)
- return client
+ return RuntimeHandle(client, backend="http-client")
monkeypatch.setattr(ModelSingleton, "_models", {})
monkeypatch.setattr(ModelSingleton, "_create_model", factory)
@@ -231,7 +239,9 @@ def test_failed_initialization_does_not_poison_cache(monkeypatch: pytest.MonkeyP
with pytest.raises(ValueError, match="initialization failed"):
manager.get_model("http-client", None, "http://unused")
assert not manager._models
- monkeypatch.setattr(ModelSingleton, "_create_model", lambda *args, **kwargs: _AsyncClient())
+ monkeypatch.setattr(
+ ModelSingleton, "_create_model", lambda *args, **kwargs: RuntimeHandle(_AsyncClient(), backend="http-client")
+ )
try:
assert manager.get_model("http-client", None, "http://unused").batch_two_step_extract([1]) == [1]
finally:
@@ -244,11 +254,11 @@ def test_vllm_sync_and_async_names_resolve_to_one_async_engine(monkeypatch: pyte
calls = []
- def factory(self: Any, backend: str, *args: Any, **kwargs: Any) -> _AsyncClient:
+ def factory(self: Any, backend: str, *args: Any, **kwargs: Any) -> RuntimeHandle:
"""记录实际请求构造的后端,并拒绝离线同步引擎。"""
assert backend == "vllm-async-engine"
calls.append(backend)
- return _AsyncClient()
+ return RuntimeHandle(_AsyncClient(), backend=backend)
monkeypatch.setattr(runtime.ModelSingleton, "_models", {})
monkeypatch.setattr(runtime.ModelSingleton, "_create_model", factory)
@@ -339,13 +349,13 @@ def test_owner_cleanup_failure_still_closes_other_runtimes(monkeypatch: pytest.M
"""模拟底层传输关闭异常。"""
raise RuntimeError("close failed")
- def factory(*args: Any, **kwargs: Any) -> _AsyncClient:
+ def factory(*args: Any, **kwargs: Any) -> RuntimeHandle:
"""仅第一个客户端关闭时失败。"""
client = _AsyncClient()
if not clients:
client.aclose = fail_close
clients.append(client)
- return client
+ return RuntimeHandle(client, backend="http-client")
monkeypatch.setattr(ModelSingleton, "_models", {})
monkeypatch.setattr(ModelSingleton, "_create_model", factory)
@@ -392,7 +402,9 @@ def test_vllm_runtime_uses_standard_progress_label(
monkeypatch.setattr(backend, "aio_predict", predict)
monkeypatch.setattr("mineru_vl_utils.mineru_client.new_vlm_client", lambda **kwargs: backend)
- runtime = AsyncVlmPredictor(create, lambda _: None, backend="vllm-async-engine", max_concurrency=2)
+ runtime = AsyncVlmPredictor(
+ lambda: RuntimeHandle(create(), backend="vllm-async-engine"), backend="vllm-async-engine", max_concurrency=2
+ )
image = Image.new("RGB", (32, 32))
blocks = [[ContentBlock("text", [0, 0, 1, 1])]]
try:
diff --git a/tests/unittest/test_onnx_device_selection.py b/tests/unittest/test_onnx_device_selection.py
index a8cb0602..56d676fd 100644
--- a/tests/unittest/test_onnx_device_selection.py
+++ b/tests/unittest/test_onnx_device_selection.py
@@ -29,7 +29,7 @@ def config_file(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
target.write_text("model:\n onnx_device: auto\n source: local\n", encoding="utf-8")
monkeypatch.setattr(settings, "get_config_file_path", lambda: str(target))
monkeypatch.setattr(settings, "get_config_source", lambda key: "file")
- monkeypatch.setattr(settings.config.model, "onnx_device", "auto")
+ monkeypatch.setattr(settings.get_config().model, "onnx_device", "auto")
monkeypatch.setattr(policy, "sys", SimpleNamespace(platform="win32"))
monkeypatch.setattr(policy, "_selections", {})
monkeypatch.setattr(policy, "ONNX_GPU_TIME_RATIO_THRESHOLDS", {"dml": 0.75, "webgpu": 0.8})
@@ -84,7 +84,7 @@ def test_gpu_mode_uses_performance_order_without_creating_devices() -> None:
def test_gpu_mode_requests_fast_probe(config_file: Path, monkeypatch: pytest.MonkeyPatch) -> None:
- monkeypatch.setattr(settings.config.model, "onnx_device", "gpu")
+ monkeypatch.setattr(settings.get_config().model, "onnx_device", "gpu")
discover = Mock(return_value=(adapter(uma=None),))
monkeypatch.setattr(policy, "discover_gpus", discover)
assert policy.resolve_onnx_device().device == "gpu"
@@ -103,7 +103,7 @@ def test_invalid_timings(values: tuple[float, float, float]) -> None:
def test_cpu_never_discovers_or_benchmarks(config_file: Path, monkeypatch: pytest.MonkeyPatch) -> None:
- monkeypatch.setattr(settings.config.model, "onnx_device", "cpu")
+ monkeypatch.setattr(settings.get_config().model, "onnx_device", "cpu")
discover = Mock(side_effect=AssertionError("discovery"))
monkeypatch.setattr(policy, "discover_gpus", discover)
assert policy.resolve_onnx_device().device == "cpu"
@@ -203,7 +203,7 @@ def test_discovery_and_unavailable_states(config_file: Path, monkeypatch: pytest
def test_manual_gpu_retained_without_gpu(config_file: Path, monkeypatch: pytest.MonkeyPatch) -> None:
- monkeypatch.setattr(settings.config.model, "onnx_device", "gpu")
+ monkeypatch.setattr(settings.get_config().model, "onnx_device", "gpu")
config_file.write_text("model:\n onnx_device: gpu\n", encoding="utf-8")
monkeypatch.setattr(policy, "discover_gpus", lambda **_: ())
assert policy.resolve_onnx_device().device == "cpu"
@@ -266,7 +266,7 @@ def test_unet_first_model_shares_selection_and_persistence(
from mineru.model.runtime import onnx, onnx_calibration
from mineru.model.table.rec.unet_table.table_structure_unet import TSRUnet
- monkeypatch.setattr(settings.config.model, "onnx_device", requested)
+ monkeypatch.setattr(settings.get_config().model, "onnx_device", requested)
config_file.write_text(f"model:\n onnx_device: {requested}\n source: local\n", encoding="utf-8")
monkeypatch.setenv("MINERU_TABLE_DEVICE", "cpu" if expected == "gpu" else "cuda")
discover = Mock(return_value=(adapter(index=7, uma=uma),))
@@ -323,7 +323,7 @@ def test_linux_calibrates_both_gpu_types_and_uses_own_threshold(
def test_linux_explicit_modes_do_not_calibrate(linux_config: Path, monkeypatch: pytest.MonkeyPatch, requested: str) -> None:
from mineru.model.runtime import onnx_calibration
- monkeypatch.setattr(settings.config.model, "onnx_device", requested)
+ monkeypatch.setattr(settings.get_config().model, "onnx_device", requested)
assert policy.resolve_onnx_device().device == requested
onnx_calibration.run_calibration.assert_not_called()
assert policy.discover_webgpu_devices.call_count == (0 if requested == "cpu" else 1)
diff --git a/tests/unittest/test_onnx_model_stack.py b/tests/unittest/test_onnx_model_stack.py
index 50b4f470..fdef9ddd 100644
--- a/tests/unittest/test_onnx_model_stack.py
+++ b/tests/unittest/test_onnx_model_stack.py
@@ -40,7 +40,7 @@ def test_unpublished_source_does_not_invalidate_existing_payload(
whole_repo: bool,
) -> None:
"""用固定的单来源仓库验证拒绝不可用来源时,已有模型和完整标记均不受影响。"""
- monkeypatch.setattr(download.config.model, "base_dir", str(tmp_path))
+ monkeypatch.setattr(download.get_config().model, "base_dir", str(tmp_path))
# 不依赖生产仓库是否已发布到 ModelScope,避免新增来源使负向测试失效。
repo = download.ModelRepo(
name="test-huggingface-only",
@@ -71,7 +71,7 @@ def test_unpublished_source_does_not_invalidate_existing_payload(
def test_onnx_required_paths_cannot_be_satisfied_by_legacy_cache(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
"""旧缓存与根标记均不能替代新 ONNX 仓库的实际文件。"""
- monkeypatch.setattr(download.config.model, "base_dir", str(tmp_path))
+ monkeypatch.setattr(download.get_config().model, "base_dir", str(tmp_path))
(tmp_path / "PDF-Extract-Kit-1.0").mkdir()
repo = registry.MINERU_4_MODELS_ONNX
repo.local_dir().mkdir()
diff --git a/tests/unittest/test_opencv_boundaries.py b/tests/unittest/test_opencv_boundaries.py
index 3f59beab..8166dba9 100644
--- a/tests/unittest/test_opencv_boundaries.py
+++ b/tests/unittest/test_opencv_boundaries.py
@@ -86,13 +86,13 @@ def test_real_flash_pdf_parse_and_all_renderers_without_opencv(tmp_path: Path, a
import asyncio
from pathlib import Path
from typing import Any
-from mineru.config import config
+from mineru.config import get_config
from mineru.parser import parse, parse_async
from mineru.parser.writer import FileBasedDataWriter
from mineru.render import RenderFormat, render
-config.llm_aided.features.title_leveling = False
-config.llm_aided.features.cross_page_table_cell_merge = False
+get_config().llm_aided.features.title_leveling = False
+get_config().llm_aided.features.cross_page_table_cell_merge = False
options = dict(tier="flash", ocr_mode=sys.argv[2], page_range="1,3")
result = asyncio.run(parse_async(sys.argv[1], **options)) if sys.argv[3] == "True" else parse(sys.argv[1], **options)
assert len(result.pages) == 2
diff --git a/tests/unittest/test_parser_api_contract.py b/tests/unittest/test_parser_api_contract.py
index 1d0a94a4..a8475ff1 100644
--- a/tests/unittest/test_parser_api_contract.py
+++ b/tests/unittest/test_parser_api_contract.py
@@ -19,17 +19,18 @@ import mineru.parser.api_client as api_client
import mineru.parser.api_server as api_server
import mineru.parser.tier as parser_tier
from mineru.config import VlmConfig
+from mineru.tasks.contracts import FileParseInfo
+from mineru.tasks.execution import TaskExecution
from mineru.parser import MIDDLE_JSON_SCHEMA_VERSION
from mineru.parser.api_client import MinerUApiParser, _pages_from_middle_json, should_trust_env_for_url
from mineru.parser.api_server import (
- _ParseRequest,
- FileParseInfo,
FileStore,
HealthResponse,
create_app,
main,
)
from mineru.parser.base import ParseResult
+from task_test_utils import execution_request
from task_test_utils import stored_source
from mineru.types import (
ModelJson,
@@ -45,6 +46,7 @@ from mineru.types import (
MiddleJson,
PageInfo,
ServerTier,
+ Tier,
select_default_quality_tier,
select_parsing_rule_tier,
validate_tier,
@@ -264,15 +266,17 @@ def test_hybrid_context_does_not_resolve_full_weights_before_stack_selection() -
"""初始化原子模型请求列表。"""
self.calls: list[dict[str, Any]] = []
- def get_atom_model(self, **kwargs: Any) -> dict[str, Any]:
+ def get_atom_model(self, atom_model_name: str, **kwargs: Any) -> dict[str, Any]:
"""记录原子模型请求并原样返回。"""
- self.calls.append(dict(kwargs))
- return dict(kwargs)
+ arguments = {"atom_model_name": atom_model_name, **kwargs}
+ self.calls.append(arguments)
+ return dict(arguments)
context = object.__new__(HybridLocalModelContext)
manager = _RecordingAtomManager()
context.device = "cpu"
context.small_backend = "onnx"
+ context._resource = None
context.atom_model_manager = manager # type: ignore[assignment]
with patch.object(ModelPath, "ensure", side_effect=AssertionError("full weight path resolved too early")):
@@ -638,37 +642,25 @@ def test_api_server_cli_no_longer_exposes_reload() -> None:
def test_usage_counts_completed_files_and_reports_configured_concurrency(tmp_path: Path) -> None:
- """files_processed 按完成的文件计数,max_concurrent_jobs 报告配置上限而非剩余信号量。"""
+ """每个真实文件执行计入 jobs_created,成功文件与并发上限保持已有公开语义。"""
file_store = FileStore(tmp_path / "api-files")
- store = api_server.JobStore(concurrency=2)
-
- def _request(*names: str) -> _ParseRequest:
- """登记源内容以构造内部执行统计样本。"""
- for name in names:
- (tmp_path / name).write_bytes(b"source")
- return _ParseRequest.model_validate(
- {
- "files": [{"source": stored_source(file_store, tmp_path / name)} for name in names],
- "tier": "flash",
- "output_formats": ["default"],
- }
- )
-
- rec_a = store.create(_request("a1.html", "a2.html"), file_store)
- rec_a.files[0].status = "completed"
- rec_a.files[1].status = "failed"
- rec_a.status = "partial"
- rec_b = store.create(_request("b1.html"), file_store)
- rec_b.files[0].status = "completed"
- rec_b.status = "completed"
+ store = api_server.ExecutionStore(concurrency=2)
+ records = []
+ for name in ("a1.html", "a2.html", "b1.html"):
+ source = tmp_path / name
+ source.write_bytes(b"source")
+ records.append(store.create(execution_request(file_store, stored_source(file_store, source), tier="flash"), file_store))
+ completed, failed, other = records
+ completed.status = other.status = "completed"
+ failed.status = "failed"
async def _report_while_busy() -> "api_server.UsageResponse":
+ """占用一个名额后确认公开限制仍报告配置上限。"""
await store._semaphore.acquire()
return store.usage("anonymous")
usage = asyncio.run(_report_while_busy())
-
- assert usage.current.jobs_created == 2
+ assert usage.current.jobs_created == 3
assert usage.current.files_processed == 2
assert usage.limits.max_concurrent_jobs == 2
@@ -685,15 +677,9 @@ def test_api_server_logs_traceback_when_job_file_fails(
file_store = FileStore(tmp_path / "api-files")
source = tmp_path / "demo.pdf"
source.write_bytes(b"%PDF-1.7\n")
- request = _ParseRequest.model_validate(
- {
- "files": [{"source": stored_source(file_store, source)}],
- "tier": "standard",
- "output_formats": ["default"],
- }
- )
- job_store = api_server.JobStore()
- rec = job_store.create(request, file_store)
+ request = execution_request(file_store, stored_source(file_store, source), tier="standard", output_formats=["default"])
+ execution_store = api_server.ExecutionStore()
+ rec = execution_store.create(request, file_store)
mineru_logger = logging.getLogger("mineru")
api_logger = logging.getLogger("mineru.parser.api_server")
monkeypatch.setattr(mineru_logger, "disabled", False)
@@ -702,18 +688,11 @@ def test_api_server_logs_traceback_when_job_file_fails(
monkeypatch.setattr(api_logger, "propagate", True)
with caplog.at_level(logging.ERROR, logger="mineru.parser.api_server"):
- asyncio.run(
- api_server._run_job(
- rec,
- request,
- file_store,
- image_analysis=True,
- )
- )
+ asyncio.run(api_server._run_execution(rec, file_store, image_analysis=True))
assert rec.status == "failed"
- assert rec.files[0].error is not None
- assert rec.files[0].error.message == "boom"
+ assert rec.error is not None
+ assert rec.error.message == "boom"
assert "Parse-server job file failed" in caplog.text
assert f"job_id={rec.id}" in caplog.text
assert "RuntimeError: boom" in caplog.text
@@ -937,65 +916,42 @@ def test_api_server_model_preload_is_opt_in_and_ignored_for_flash(tmp_path: Path
def test_api_server_standard_jobs_use_requested_tier(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
- """校验 Standard server 把每个 job 请求的 tier 传给统一 parser。"""
+ """Standard 服务按每个文件请求选择档位,并从实际解析结果记录执行信息。"""
_stub_api_server_dependency_preflight(monkeypatch)
calls: list[dict[str, object]] = []
async def fake_parse_async(*args: object, **kwargs: object) -> ParseResult:
- """记录 API server 传给 parser 的 runtime 参数,并返回最小解析结果。"""
+ """记录生产解析调用,并返回相应档位的最小有效协议。"""
calls.append(dict(kwargs))
- return ParseResult(middle_json=_full_middle_json(PageInfo(page_idx=0)))
+ middle_json = _full_middle_json(PageInfo(page_idx=0))
+ middle_json.extensions["mineru"] = {"tier": kwargs["tier"], "parse_mode": "txt"}
+ return ParseResult(middle_json=middle_json)
monkeypatch.setattr("mineru.parser.api_server.parse_async", fake_parse_async)
source = tmp_path / "demo.pdf"
source.write_bytes(b"%PDF-1.7\n")
app = create_app(upload_dir=str(tmp_path / "api"), tier="standard")
file_store = FileStore(tmp_path / "api-files")
- job_store = api_server.JobStore()
+ execution_store = api_server.ExecutionStore()
- async def run_request(payload: dict[str, object]) -> dict[str, object]:
- request = _ParseRequest.model_validate(payload)
- request.tier = request.tier or app.state.default_tier
- rec = job_store.create(request, file_store)
- await api_server._run_job(
+ async def run_request(tier: Tier | None = None) -> TaskExecution:
+ """构造并运行一个真实内部执行请求,返回有类型快照。"""
+ request = execution_request(file_store, stored_source(file_store, source), tier=tier or app.state.default_tier)
+ rec = execution_store.create(request, file_store)
+ await api_server._run_execution(
rec,
- request,
file_store,
image_analysis=app.state.image_analysis,
allow_http_source=app.state.allow_http_source,
)
- return job_store.snapshot(rec)
-
- default_response = asyncio.run(
- run_request(
- {
- "files": [{"source": stored_source(file_store, source)}],
- "output_formats": ["default"],
- }
- )
- )
- basic_response = asyncio.run(
- run_request(
- {
- "files": [{"source": stored_source(file_store, source)}],
- "tier": "basic",
- "output_formats": ["default"],
- }
- )
- )
- advanced_response = asyncio.run(
- run_request(
- {
- "files": [{"source": stored_source(file_store, source)}],
- "tier": "advanced",
- "output_formats": ["default"],
- }
- )
- )
-
- assert default_response["tier"] == "standard"
- assert basic_response["tier"] == "basic"
- assert advanced_response["tier"] == "advanced"
+ return execution_store.snapshot(rec)
+
+ default_response = asyncio.run(run_request())
+ basic_response = asyncio.run(run_request("basic"))
+ advanced_response = asyncio.run(run_request("advanced"))
+ assert default_response.parse is not None and default_response.parse.tier == "standard"
+ assert basic_response.parse is not None and basic_response.parse.tier == "basic"
+ assert advanced_response.parse is not None and advanced_response.parse.tier == "advanced"
assert [call["tier"] for call in calls] == ["standard", "basic", "advanced"]
@@ -1093,8 +1049,8 @@ def test_api_server_preflights_basic_tier_dependencies(monkeypatch: pytest.Monke
return object()
monkeypatch.setattr(importlib, "import_module", fake_import_module)
- monkeypatch.setattr(api_server.mineru_config.model, "small_backend", "torch")
- monkeypatch.setattr(api_server.mineru_config.model.vlm, "engine", "llama-cpp")
+ monkeypatch.setattr(api_server.get_config().model, "small_backend", "torch")
+ monkeypatch.setattr(api_server.get_config().model.vlm, "engine", "llama-cpp")
create_app(upload_dir=str(tmp_path), tier="basic")
@@ -1116,8 +1072,8 @@ def test_api_server_preflights_standard_tier_dependencies_for_platform(monkeypat
return object()
monkeypatch.setattr(importlib, "import_module", fake_import_module)
- monkeypatch.setattr(api_server.mineru_config.model, "small_backend", "torch")
- monkeypatch.setattr(api_server.mineru_config.model.vlm, "engine", "llama-cpp")
+ monkeypatch.setattr(api_server.get_config().model, "small_backend", "torch")
+ monkeypatch.setattr(api_server.get_config().model.vlm, "engine", "llama-cpp")
create_app(upload_dir=str(tmp_path), tier="standard")
@@ -1142,8 +1098,8 @@ def test_api_server_preflights_standard_tier_dependencies_skip_mlx_on_intel_maco
return object()
monkeypatch.setattr(importlib, "import_module", fake_import_module)
- monkeypatch.setattr(api_server.mineru_config.model, "small_backend", "torch")
- monkeypatch.setattr(api_server.mineru_config.model.vlm, "engine", "llama-cpp")
+ monkeypatch.setattr(api_server.get_config().model, "small_backend", "torch")
+ monkeypatch.setattr(api_server.get_config().model.vlm, "engine", "llama-cpp")
create_app(upload_dir=str(tmp_path), tier="standard")
@@ -1165,7 +1121,7 @@ def test_api_server_preflight_rejects_missing_tier_dependency(monkeypatch: pytes
return object()
monkeypatch.setattr(importlib, "import_module", fake_import_module)
- monkeypatch.setattr(api_server.mineru_config.model, "small_backend", "torch")
+ monkeypatch.setattr(api_server.get_config().model, "small_backend", "torch")
monkeypatch.setattr(parser_tier.importlib_metadata, "packages_distributions", lambda: {"mineru": ["mineru"]})
with pytest.raises(api_server.ParseServerStartupError, match="tier 'basic'.*torch.*mineru\\[torch\\]"):
diff --git a/tests/unittest/test_parser_api_ocr.py b/tests/unittest/test_parser_api_ocr.py
index 604c317c..3098cfae 100644
--- a/tests/unittest/test_parser_api_ocr.py
+++ b/tests/unittest/test_parser_api_ocr.py
@@ -3,7 +3,7 @@
from __future__ import annotations
from test_tasks_sdk import task_snapshot, zip_output
-from mineru.parser.task_api import CreateTaskRequest
+from mineru.tasks.contracts import CreateTaskRequest
import asyncio
import json
@@ -101,7 +101,7 @@ def test_api_server_rejects_invalid_request_ocr_mode(tmp_path: Path, value: obje
assert response.status_code == 400
assert response.json().get("error")["code"] == "invalid_request"
assert response.json().get("error")["param"] == "ocr_mode"
- assert app.state.job_store._jobs == {}
+ assert app.state.execution_store._executions == {}
def test_api_server_ocr_defaults_to_request_auto_without_startup_configuration(tmp_path: Path) -> None:
diff --git a/tests/unittest/test_parser_model_preload.py b/tests/unittest/test_parser_model_preload.py
index f9c7b187..512ac643 100644
--- a/tests/unittest/test_parser_model_preload.py
+++ b/tests/unittest/test_parser_model_preload.py
@@ -104,7 +104,7 @@ def test_local_preload_and_parse_reuse_llama_predictor(monkeypatch: pytest.Monke
api_server._preload_server_models("standard", vlm_config=VlmConfig())
predictor, engine = get_vlm_predictor(VlmConfig())
- assert predictor is predictor_factory.return_value
+ assert predictor.predictor is predictor_factory.return_value
assert engine == "llama-cpp-engine"
predictor_factory.assert_called_once()
engine_factory.assert_called_once()
diff --git a/tests/unittest/test_pdf_layout_integration.py b/tests/unittest/test_pdf_layout_integration.py
index 958f0722..f71f8511 100644
--- a/tests/unittest/test_pdf_layout_integration.py
+++ b/tests/unittest/test_pdf_layout_integration.py
@@ -22,7 +22,7 @@ from mineru.backend import analyze
from mineru.backend.analysis.pdf import pipeline
from mineru.backend.postprocess import document as postprocess
from mineru.doclib.background.compaction import Compaction
-from mineru.doclib.services.parse_svc import parse_batch_json_path
+from mineru.doclib.core.parse_artifacts import parse_batch_json_path
from mineru.render import PdfLayout, PdfRenderOptions, RenderFormat, render, render_pdf
diff --git a/tests/unittest/test_pdf_page_id.py b/tests/unittest/test_pdf_page_id.py
index ec748b3e..2403c00e 100644
--- a/tests/unittest/test_pdf_page_id.py
+++ b/tests/unittest/test_pdf_page_id.py
@@ -1,6 +1,7 @@
from __future__ import annotations
+from mineru.doclib.services.content_read_svc import ContentReadService
-from task_test_utils import stored_source
+from task_test_utils import wait_task
import asyncio
import io
@@ -16,11 +17,11 @@ from typer.testing import CliRunner
from mineru.cli.commands.parse import _validate_page_range_input
from mineru.cli.main import app as cli_app
-from mineru.doclib.server import _normalize_content_page_range
+from mineru.doclib.services.content_read_svc import _normalize_content_page_range
from mineru.doclib.services.config_svc import ConfigService
from mineru.doclib.services.parse_svc import default_parse_range
from mineru.errors import InvalidRequestError, MineruError
-from mineru.kit.gradio.app import _effective_page_range
+from mineru.kit.gradio.page_range import effective_page_range as _effective_page_range
from mineru.kit.gradio.artifacts import persist_parse_result
from mineru.kit.gradio.client import V1ArtifactClient
from mineru.kit.main import app as kit_app
@@ -253,27 +254,36 @@ def test_real_pdf_selection_preserves_original_indices_and_artifact_order(tmp_pa
def test_async_job_resolves_relative_ranges_and_reports_file_errors(tmp_path: Path) -> None:
- """后台任务完成后返回规范实际范围,依赖总页数的错误保留文件级错误码。"""
+ """父 Task 聚合独立成功和失败执行,实际范围与文件错误均保留到对应快照。"""
source = tmp_path / "numbered.pdf"
_numbered_pdf(source)
- store = api_server.FileStore(tmp_path / "files")
- request = api_server._ParseRequest.model_validate(
- {
- "tier": "flash",
- "output_formats": ["default"],
- "files": [
- {"source": stored_source(store, source), "page_range": "1,r2-r1"},
- {"source": stored_source(store, source), "page_range": "20"},
- ],
- }
- )
- record = api_server.JobStore().create(request, store)
- asyncio.run(api_server._run_job(record, request, store, image_analysis=True))
- assert record.status == "partial"
- assert record.files[0].page_range == "1,4-5"
- assert record.files[0].status == "completed"
- assert record.files[1].error.code == "page_range_invalid"
- assert record.files[1].error.param == "page_range"
+ data = source.read_bytes()
+ app = api_server.create_app(upload_dir=str(tmp_path / "api-files"), tier="flash")
+ with TestClient(app) as client:
+ response = client.post(
+ "/v1/tasks",
+ json={
+ "tier": "flash",
+ "output_formats": ["default"],
+ "files": [
+ {"file_name": source.name, "bytes": len(data), "page_range": "1,r2-r1"},
+ {"file_name": source.name, "bytes": len(data), "page_range": "20"},
+ ],
+ },
+ )
+ assert response.status_code == 202
+ created = response.json()
+ for item in created["files"]:
+ upload = item["upload"]
+ assert client.put(upload["url"], content=data, headers=upload["headers"]).status_code == 200
+ ready = wait_task(client, created)
+ assert ready["status"] == "partial"
+ succeeded, failed = app.state.task_store._records[created["task_id"]].file_states
+ assert succeeded.execution.page_range == "1,4-5"
+ assert succeeded.execution.status == "completed"
+ assert failed.execution.status == "failed"
+ assert failed.execution.error.code == "page_range_invalid"
+ assert failed.execution.error.param == "page_range"
def test_doclib_client_validates_requests_before_transport(monkeypatch: pytest.MonkeyPatch) -> None:
@@ -339,7 +349,7 @@ def test_doclib_relative_selection_coverage_and_export(tmp_path: Path) -> None:
from mineru.doclib.core.db import DatabaseManager
from mineru.doclib.server import DoclibServer
- from mineru.doclib.services.parse_svc import parse_batch_json_path
+ from mineru.doclib.core.parse_artifacts import parse_batch_json_path
source = tmp_path / "numbered.pdf"
_numbered_pdf(source)
@@ -368,9 +378,13 @@ def test_doclib_relative_selection_coverage_and_export(tmp_path: Path) -> None:
listing = await server.list_parses(doc_ref="aaaaaaa", tier="flash", page_range="r3-r1")
assert listing.coverage.done_page_range == "3,5"
assert listing.coverage.missing_page_range == "4"
- text = await server._render_doc_content(sha256, tier="flash", page_range="r1", format="markdown", no_marker=True)
+ text = await ContentReadService(server.state.db, data_dir=server.state.data_dir)._render_doc_content(
+ sha256, tier="flash", page_range="r1", format="markdown", no_marker=True
+ )
assert "Source page 5" in text and "Source page 3" not in text
- text = await server._render_doc_content(sha256, tier="flash", page_range="all", format="markdown", no_marker=True)
+ text = await ContentReadService(server.state.db, data_dir=server.state.data_dir)._render_doc_content(
+ sha256, tier="flash", page_range="all", format="markdown", no_marker=True
+ )
assert "Source page 1" in text and "Source page 5" in text
finally:
await db.close()
diff --git a/tests/unittest/test_resource_retention.py b/tests/unittest/test_resource_retention.py
index 0956fcbb..c8bf88cd 100644
--- a/tests/unittest/test_resource_retention.py
+++ b/tests/unittest/test_resource_retention.py
@@ -6,15 +6,20 @@ from __future__ import annotations
import asyncio
import hashlib
import os
-from datetime import datetime, timezone
+from datetime import datetime
from pathlib import Path
+from types import SimpleNamespace
from typing import Any
import pytest
from fastapi.testclient import TestClient
+from task_test_utils import execution_request
from mineru.parser import api_server
-from mineru.parser.api_server import ApiServerError, _ParseRequest, FileStore, JobStore
+from mineru.parser.api_server import FileStore, ExecutionStore
+from mineru.tasks.errors import ApiServerError
+from mineru.tasks import execution
+from mineru.tasks.execution import TaskExecution, execution_timestamp
from mineru.kit.router.resources import ResourceRegistry, SourceFileStore, StoredSourceFile
from mineru.utils.retention import resolve_retention_seconds
@@ -24,9 +29,7 @@ def clock(monkeypatch: pytest.MonkeyPatch) -> list[int]:
"""只替换墙上时间,不影响事件循环超时和任务调度。"""
value = [2_000_000_000]
monkeypatch.setattr(api_server.time, "time", lambda: value[0])
- monkeypatch.setattr(
- JobStore, "_now", staticmethod(lambda: datetime.fromtimestamp(value[0], timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"))
- )
+ monkeypatch.setattr(execution, "datetime", SimpleNamespace(now=lambda zone: datetime.fromtimestamp(value[0], zone)))
return value
@@ -82,10 +85,11 @@ def test_output_retention_starts_after_cleanup(tmp_path: Path, clock: list[int])
"""手动控制运行、排队、完成和回收的交错顺序。"""
files = FileStore(tmp_path / "files", retention_seconds=10)
source, sha = _source(files, tmp_path)
- jobs = JobStore(concurrency=1, retention_seconds=10)
- req = _ParseRequest.model_validate({"tier": "flash", "files": [{"source": {"type": "file_id", "file_id": source}}]})
+ jobs = ExecutionStore(concurrency=1, retention_seconds=10)
gate = asyncio.Event()
- records = [jobs.create(req, files) for _ in range(2)]
+ records = [
+ jobs.create(execution_request(files, {"type": "file_id", "file_id": source}, tier="flash"), files) for _ in range(2)
+ ]
outputs: list[str] = []
async def work(rec: Any) -> None:
@@ -94,7 +98,7 @@ def test_output_retention_starts_after_cleanup(tmp_path: Path, clock: list[int])
output = files.create_file_for_output("out.md", b"result", job_id=rec.id)
outputs.append(output)
await gate.wait()
- extracted = await api_server._extract_bytes(req.files[0].source, files, input_blobs=rec.input_blobs)
+ extracted = await api_server._extract_bytes(rec.request.source, files, input_blobs=rec.input_blobs)
assert extracted.data == b"document"
rec.status = "completed"
@@ -104,7 +108,7 @@ def test_output_retention_starts_after_cleanup(tmp_path: Path, clock: list[int])
clock[0] += 100
jobs.collect_expired()
files.collect_expired()
- assert len(jobs._jobs) == 2 and len(outputs) == 1 and files._blob_abs(sha).exists()
+ assert len(jobs._executions) == 2 and len(outputs) == 1 and files._blob_abs(sha).exists()
assert files.get_file(outputs[0]).expires_at is None
gate.set()
for rec in records:
@@ -118,7 +122,7 @@ def test_output_retention_starts_after_cleanup(tmp_path: Path, clock: list[int])
assert exc.value.status_code == 404
jobs.collect_expired()
files.collect_expired()
- assert not jobs._jobs and not jobs._completion_events and not files._files
+ assert not jobs._executions and not jobs._completion_events and not files._files
assert not any(files._blobs.glob("??/*"))
await jobs.shutdown()
@@ -193,7 +197,7 @@ def test_router_expiry_cache_pins_and_alias_cleanup(tmp_path: Path, clock: list[
assert sources.find_hash("one", stored.sha256sum) is None
assert path.exists() and registry.find("job", job.public_id) is job
job.metadata["payload"]["status"] = "failed"
- job.metadata["payload"]["finished_at"] = JobStore._now()
+ job.metadata["payload"]["finished_at"] = execution_timestamp()
registry.collect_expired(sources)
assert registry.find_upstream("file", "one", "copy", "copy-file") is None
assert path.exists()
@@ -211,8 +215,8 @@ def test_queued_cancel_releases_input_before_execution(tmp_path: Path, clock: li
"""取消早于 run_owned 的首次执行,覆盖没有 finally 执行机会的分支。"""
files = FileStore(tmp_path / "files", retention_seconds=10)
source, sha = _source(files, tmp_path)
- jobs = JobStore(retention_seconds=10)
- req = _ParseRequest.model_validate({"tier": "flash", "files": [{"source": {"type": "file_id", "file_id": source}}]})
+ jobs = ExecutionStore(retention_seconds=10)
+ req = execution_request(files, {"type": "file_id", "file_id": source}, tier="flash")
rec = jobs.create(req, files)
async def forbidden() -> None:
@@ -247,7 +251,7 @@ def test_hash_preflight_rejection_releases_request_lease(tmp_path: Path, monkeyp
)
assert response.status_code == 400
assert response.json().get("error")["code"] == "upload_size_mismatch"
- assert not app.state.file_store._pins and not app.state.job_store._jobs
+ assert not app.state.file_store._pins and not app.state.execution_store._executions
def test_gc_interleaves_with_blob_writes_and_downloads(tmp_path: Path, clock: list[int]) -> None:
@@ -292,7 +296,7 @@ def test_router_usage_survives_expiry_collection_and_worker_removal(clock: list[
for index in range(2):
route = registry.register("job", owner_scope="owner", worker_id="worker", upstream_id=str(index))
route.metadata.update(
- payload={"status": "completed", "files": [{"status": "completed"}]}, finalized_at=JobStore._now()
+ payload={"status": "completed", "files": [{"status": "completed"}]}, finalized_at=execution_timestamp()
)
before = registry.usage("owner")
assert before == {"jobs_created": 2, "files_processed": 2}
@@ -312,12 +316,12 @@ def test_retention_keeps_cumulative_usage(tmp_path: Path, clock: list[int]) -> N
"""元数据回收后已完成的文件、页数与任务累计值不减少。"""
files = FileStore(tmp_path / "files", retention_seconds=10)
source, _ = _source(files, tmp_path)
- jobs = JobStore(retention_seconds=10)
- request = _ParseRequest.model_validate({"tier": "flash", "files": [{"source": {"type": "file_id", "file_id": source}}]})
+ jobs = ExecutionStore(retention_seconds=10)
+ request = execution_request(files, {"type": "file_id", "file_id": source}, tier="flash")
rec = jobs.create(request, files)
- rec.status = rec.files[0].status = "completed"
- rec.files[0].page_range = "1-3"
- rec.finished_at = JobStore._now()
+ rec.status = "completed"
+ rec.page_range = "1-3"
+ rec.finished_at = execution_timestamp()
before = jobs.usage("anonymous").current
files.release_inputs(rec.input_blobs)
clock[0] += 11
@@ -355,7 +359,16 @@ def test_external_worker_lost_task_releases_load_and_keeps_snapshot() -> None:
"job", owner_scope="anonymous", worker_id=reservation.worker.worker_id, upstream_id="lost-task"
)
route.metadata.update(
- active_counted=True, reservation=reservation, payload={"status": "running", "files": [{"status": "pending"}]}
+ active_counted=True,
+ reservation=reservation,
+ payload={"status": "running", "files": [{"status": "pending"}]},
+ execution=TaskExecution(
+ execution_id=route.public_id,
+ file_id="file-lost",
+ file_name="lost.pdf",
+ status="pending",
+ created_at=route.created_at,
+ ),
)
client.portal.call(_reconcile_jobs_once, app.state.registry, pool)
assert reservation.worker.active_jobs == 0
diff --git a/tests/unittest/test_router_balancing.py b/tests/unittest/test_router_balancing.py
index 69b45e6a..51940c0d 100644
--- a/tests/unittest/test_router_balancing.py
+++ b/tests/unittest/test_router_balancing.py
@@ -130,11 +130,11 @@ def test_cached_tasks_still_balance(tmp_path: Path, monkeypatch: pytest.MonkeyPa
tasks.append(task)
# submitted 只表示 Router 开始转发;等待上游实际建好 job,包含仍在排队的任务。
deadline = time.monotonic() + 5
- while sum(len(worker.state.job_store._jobs) for worker in workers.values()) < len(tasks):
+ while sum(len(worker.state.execution_store._executions) for worker in workers.values()) < len(tasks):
assert time.monotonic() < deadline
time.sleep(0.01)
assert [worker.active_jobs for worker in app.state.worker_pool.workers] == [5, 5]
- assert [len(worker.state.job_store._jobs) for worker in workers.values()] == [5, 5]
+ assert [len(worker.state.execution_store._executions) for worker in workers.values()] == [5, 5]
assert sum(method == "PUT" for _, method, _ in transport.calls) == 2
finally:
release.set()
diff --git a/tests/unittest/test_router_cascade.py b/tests/unittest/test_router_cascade.py
index 24c5b04c..7f3a9939 100644
--- a/tests/unittest/test_router_cascade.py
+++ b/tests/unittest/test_router_cascade.py
@@ -157,7 +157,7 @@ def test_cascade_accepts_ready_upload_without_source_file_and_reuses_cache(tmp_p
result = await _terminal(client, cached)
assert result["status"] == "completed", result
await _check_download(client, result)
- assert len(worker.state.job_store._jobs) == 2
+ assert len(worker.state.execution_store._executions) == 2
assert all(item.active_jobs == 0 for app in (outer, inner) for item in app.state.worker_pool.workers)
asyncio.run(scenario())
@@ -181,7 +181,7 @@ def test_cascade_can_cancel_while_downstream_submission_is_blocked(tmp_path: Pat
for record in inner.state.task_store._records.values():
# 下游仍有结果租约,取消清理只需等待实际子文件退出。
await asyncio.gather(*(file.operation for file in record.file_states if file.operation is not None))
- assert not worker.state.job_store._jobs
+ assert not worker.state.execution_store._executions
for app in (outer, inner):
assert all(item.active_jobs == 0 for item in app.state.worker_pool.workers)
assert not app.state.source_store._pins
diff --git a/tests/unittest/test_runtime_ownership.py b/tests/unittest/test_runtime_ownership.py
new file mode 100644
index 00000000..f228548d
--- /dev/null
+++ b/tests/unittest/test_runtime_ownership.py
@@ -0,0 +1,583 @@
+"""验证应用、文档和模型依赖的租约,不加载真实模型权重。"""
+
+from __future__ import annotations
+
+import asyncio
+import threading
+from collections.abc import Iterator
+from concurrent.futures import ThreadPoolExecutor, TimeoutError
+from contextlib import contextmanager
+from contextvars import copy_context
+from dataclasses import dataclass
+from typing import Any
+
+import pytest
+
+from mineru.config import get_config
+from mineru.model.runtime import cpu_threads, hybrid, local_cache, onnx_device
+from mineru.model.runtime.ownership import (
+ RuntimeOwner,
+ current_runtime_owner,
+ release_runtime_owner,
+ runtime_document_scope,
+ runtime_owner,
+)
+from mineru.model.vlm.handles import RuntimeHandle
+from mineru.model.vlm.runtime import ModelSingleton
+
+
+@dataclass
+class _Atom:
+ """保留嵌套模型引用的轻量替身,用于验证实际缓存依赖关系。"""
+
+ name: str
+ dependency: _Atom | None = None
+
+
+@contextmanager
+def _owned(owner: RuntimeOwner | None) -> Iterator[None]:
+ """在明确的应用上下文中调用真实模型缓存。"""
+ token = runtime_owner.set(owner)
+ try:
+ yield
+ finally:
+ runtime_owner.reset(token)
+
+
+def local_models(monkeypatch: pytest.MonkeyPatch) -> list[_Atom]:
+ """只替换实际模型构造,保留上下文、缓存键和嵌套借用的生产逻辑。"""
+ created: list[_Atom] = []
+ monkeypatch.setattr(hybrid.AtomModelSingleton, "_models", {})
+ monkeypatch.setattr(hybrid.HybridLocalModelContextSingleton, "_models", {})
+ monkeypatch.setattr(ModelSingleton, "_models", {})
+ monkeypatch.setattr(ModelSingleton, "_closing", {})
+ monkeypatch.setattr(ModelSingleton, "_initializing", {})
+ monkeypatch.setattr(hybrid, "resolve_small_model_backend", lambda backend=None: backend or "onnx")
+ monkeypatch.setattr(hybrid, "get_device", lambda: "cpu")
+ monkeypatch.setattr(hybrid, "ocr_det_batch_setting", lambda: True)
+ monkeypatch.setattr(cpu_threads, "initialize_torch_threads", lambda: 2)
+ monkeypatch.setattr(onnx_device, "resolve_onnx_device", lambda: "cpu")
+
+ def create(model_name: str, **options: Any) -> _Atom:
+ """表格模型像生产构造器一样借用不同配置的 OCR,形成真实二级依赖。"""
+ dependency = None
+ if model_name == "wireless_table":
+ dependency = hybrid.AtomModelSingleton().get_atom_model(
+ "ocr", small_backend=options["small_backend"], device=options["device"], det_db_unclip_ratio=1.6
+ )
+ model = _Atom(model_name, dependency)
+ created.append(model)
+ return model
+
+ monkeypatch.setattr(hybrid, "atom_model_init", create)
+ return created
+
+
+def test_shared_context_keeps_cached_properties_until_last_owner(local_models: list[_Atom]) -> None:
+ """A 首次加载惰性模型、B 复用同一上下文后,关闭 A 不得清理 B 的依赖。"""
+ first, second = RuntimeOwner(), RuntimeOwner()
+ manager = hybrid.HybridLocalModelContextSingleton()
+ with _owned(first):
+ context = manager.get_model()
+ formula = context.mfr_model
+ table = context.wireless_table_model
+ with _owned(second):
+ shared = manager.get_model()
+ assert shared is context
+ assert shared.mfr_model is formula
+ assert shared.wireless_table_model is table
+ release_runtime_owner(first)
+ assert shared.mfr_model is formula
+ assert shared.wireless_table_model.dependency is table.dependency
+ assert len(local_models) == 5
+ assert len(hybrid.AtomModelSingleton._models) == 5
+ release_runtime_owner(second)
+ assert not manager._models
+ assert not hybrid.AtomModelSingleton._models
+
+
+def test_distinct_contexts_preserve_shared_atom_dependencies(
+ local_models: list[_Atom], monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """不同 ONNX 配置上下文解析到同一设备时,共享的表格及其 OCR 依赖按最后持有者释放。"""
+ first, second = RuntimeOwner(), RuntimeOwner()
+ manager = hybrid.HybridLocalModelContextSingleton()
+ monkeypatch.setattr(get_config().model, "onnx_device", "auto")
+ with _owned(first):
+ left = manager.get_model()
+ table = left.wireless_table_model
+ monkeypatch.setattr(get_config().model, "onnx_device", "cpu")
+ with _owned(second):
+ right = manager.get_model()
+ assert right is not left
+ assert right.ocr_model is left.ocr_model
+ assert right.wireless_table_model is table
+ release_runtime_owner(first)
+ assert right.wireless_table_model.dependency is table.dependency
+ assert len(local_models) == 4
+ assert len(hybrid.AtomModelSingleton._models) == 4
+ release_runtime_owner(second)
+ assert not manager._models
+ assert not hybrid.AtomModelSingleton._models
+
+
+def test_backend_and_device_changes_release_only_their_context(
+ local_models: list[_Atom], monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """后端或设备切换仍沿原缓存键隔离,关闭一个配置不会回收另一个配置。"""
+ first, second = RuntimeOwner(), RuntimeOwner()
+ manager = hybrid.HybridLocalModelContextSingleton()
+ with _owned(first):
+ onnx_context = manager.get_model()
+ monkeypatch.setattr(hybrid, "resolve_small_model_backend", lambda backend=None: backend or "torch")
+ monkeypatch.setattr(hybrid, "get_device", lambda: "cuda:0")
+ with _owned(second):
+ torch_context = manager.get_model()
+ assert torch_context.ocr_model is not onnx_context.ocr_model
+ release_runtime_owner(first)
+ assert len(manager._models) == 1
+ assert len(hybrid.AtomModelSingleton._models) == 2
+ with _owned(second):
+ assert manager.get_model() is torch_context
+ release_runtime_owner(second)
+ assert not manager._models
+ assert not hybrid.AtomModelSingleton._models
+
+
+def test_cli_pin_survives_application_release(local_models: list[_Atom]) -> None:
+ """独立 CLI 借用固定进程缓存,后续应用退出不能撤销该固定租约。"""
+ manager = hybrid.HybridLocalModelContextSingleton()
+ with _owned(None):
+ pinned = manager.get_model()
+ owner = RuntimeOwner()
+ with _owned(owner):
+ assert manager.get_model() is pinned
+ release_runtime_owner(owner)
+ with _owned(None):
+ assert manager.get_model() is pinned
+ assert len(hybrid.AtomModelSingleton._models) == 2
+
+
+def test_empty_and_shared_owner_release_do_not_force_full_gc(
+ local_models: list[_Atom], monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """空应用和仍被共享的模型无需全量 GC,最后上下文释放依赖后只回收一次。"""
+ collections: list[bool] = []
+
+ def collect() -> int:
+ """记录显式全量回收,避免测试本身依赖真实内存回收时机。"""
+ collections.append(True)
+ return 0
+
+ monkeypatch.setattr(local_cache.gc, "collect", collect)
+ release_runtime_owner(RuntimeOwner())
+ assert not collections
+ first, second = RuntimeOwner(), RuntimeOwner()
+ with _owned(first):
+ hybrid.HybridLocalModelContextSingleton().get_model()
+ with _owned(second):
+ hybrid.HybridLocalModelContextSingleton().get_model()
+ release_runtime_owner(first)
+ assert not collections
+ release_runtime_owner(second)
+ assert collections == [True]
+
+
[email protected]("shared", [False, True])
+def test_failed_context_initialization_rolls_back_unshared_dependencies(
+ local_models: list[_Atom], monkeypatch: pytest.MonkeyPatch, shared: bool
+) -> None:
+ """OCR 已初始化而版面模型失败时,失败上下文及其孤立 OCR 租约必须一并撤销。"""
+ create = hybrid.atom_model_init
+
+ def fail_layout(model_name: str, **options: Any) -> _Atom:
+ """只让第二个必需阶段失败,真实执行先前依赖的登记与回滚。"""
+ if model_name == "layout":
+ raise ValueError("layout initialization failed")
+ return create(model_name, **options)
+
+ monkeypatch.setattr(hybrid, "atom_model_init", fail_layout)
+ existing_owner = RuntimeOwner()
+ if shared:
+ with _owned(existing_owner):
+ hybrid.AtomModelSingleton().get_atom_model("ocr")
+ owner = RuntimeOwner()
+ with _owned(owner), pytest.raises(ValueError, match="layout initialization failed"):
+ hybrid.HybridLocalModelContextSingleton().get_model()
+ assert len(local_models) == 1
+ assert not hybrid.HybridLocalModelContextSingleton._models
+ assert len(hybrid.AtomModelSingleton._models) == int(shared)
+ release_runtime_owner(existing_owner)
+ assert not hybrid.AtomModelSingleton._models
+
+
+def test_owner_close_waits_for_document_and_allows_its_lazy_models(local_models: list[_Atom]) -> None:
+ """关闭应用时等待已接纳文档继续惰性加载并结束,同时拒绝新文档进入。"""
+ owner = RuntimeOwner()
+ entered, continue_document, lazy_loaded = threading.Event(), threading.Event(), threading.Event()
+
+ def document() -> None:
+ """模拟正在同步执行的文档,在关闭开始后才首次读取公式模型。"""
+ with _owned(owner), runtime_document_scope():
+ context = hybrid.HybridLocalModelContextSingleton().get_model()
+ entered.set()
+ assert continue_document.wait(3)
+ assert context.mfr_model.name == "mfr"
+ lazy_loaded.set()
+
+ with ThreadPoolExecutor(max_workers=2) as executor:
+ work = executor.submit(document)
+ try:
+ assert entered.wait(3)
+ closing = executor.submit(release_runtime_owner, owner)
+ with pytest.raises(TimeoutError):
+ closing.result(timeout=0.05)
+ assert owner.closed
+ with _owned(owner), pytest.raises(RuntimeError, match="owner is closed"), runtime_document_scope():
+ raise AssertionError("New document must not enter")
+ finally:
+ continue_document.set()
+ work.result(timeout=3)
+ closing.result(timeout=3)
+ assert lazy_loaded.is_set()
+ assert len(local_models) == 3
+ assert not hybrid.HybridLocalModelContextSingleton._models
+ assert not hybrid.AtomModelSingleton._models
+
+
+def test_owner_release_inside_document_is_rejected_without_deadlock() -> None:
+ """文档内部同步释放自己的应用时明确报错,不能等待当前文档自身退出。"""
+ owner = RuntimeOwner()
+ with _owned(owner), runtime_document_scope():
+ with pytest.raises(RuntimeError, match="active document"):
+ release_runtime_owner(owner)
+ assert not owner.closed
+ assert current_runtime_owner() is owner
+ owner.close()
+ assert owner.closed
+
+
+def test_expired_document_context_cannot_acquire_models(local_models: list[_Atom]) -> None:
+ """已退出文档复制到迟到线程的上下文不能重新借用或初始化模型。"""
+ owner = RuntimeOwner()
+ with _owned(owner), runtime_document_scope():
+ delayed = copy_context()
+ with pytest.raises(RuntimeError, match="lease has ended"):
+ delayed.run(hybrid.HybridLocalModelContextSingleton().get_model)
+ assert not local_models
+
+
+class _SyncClient:
+ """记录同步执行边界的微型客户端,不启动模型或网络连接。"""
+
+ def __init__(self) -> None:
+ """创建可以由测试显式放行的同步执行门。"""
+ self.entered = threading.Event()
+ self.release = threading.Event()
+ self.calls: list[list[int]] = []
+
+ def batch_two_step_extract(self, images: list[int], **kwargs: Any) -> list[int]:
+ """模拟无法被 asyncio 取消强行打断的 CPU 推理。"""
+ self.calls.append(images)
+ self.entered.set()
+ assert self.release.wait(3)
+ return images
+
+ async def aclose(self) -> None:
+ """为异步托管测试提供客户端关闭接口,本替身没有额外连接资源。"""
+
+
+def test_sync_backend_owners_share_until_last_application_release(
+ local_models: list[_Atom], monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """同步后端同样按应用租约共享,A 关闭后 B 可继续推理,最后关闭才释放引擎。"""
+ closed = threading.Event()
+ client = _SyncClient()
+ client.release.set()
+ monkeypatch.setattr(
+ ModelSingleton,
+ "_create_model",
+ lambda *args, **kwargs: RuntimeHandle(client, backend="transformers", close_engine=closed.set),
+ )
+ first, second = RuntimeOwner(), RuntimeOwner()
+ with _owned(first):
+ predictor = ModelSingleton().get_model("transformers", "shared", None)
+ with _owned(second):
+ assert ModelSingleton().get_model("transformers", "shared", None) is predictor
+ release_runtime_owner(first)
+ assert not closed.is_set()
+ with _owned(second):
+ assert predictor.batch_two_step_extract([1]) == [1]
+ release_runtime_owner(second)
+ assert closed.is_set()
+ assert not ModelSingleton._models
+
+
+def test_sync_runtime_shutdown_waits_for_inference() -> None:
+ """宿主句柄关闭前等待同步推理自然完成,不提前调用底层引擎关闭方法。"""
+ client = _SyncClient()
+ closed = threading.Event()
+ handle = RuntimeHandle(client, backend="transformers", close_engine=closed.set)
+ with ThreadPoolExecutor(max_workers=2) as executor:
+ inference = executor.submit(handle.batch_two_step_extract, [1])
+ try:
+ assert client.entered.wait(3)
+ shutdown = executor.submit(handle.shutdown)
+ with pytest.raises(TimeoutError):
+ shutdown.result(timeout=0.05)
+ assert not closed.is_set()
+ finally:
+ client.release.set()
+ assert inference.result(timeout=3) == [1]
+ shutdown.result(timeout=3)
+ assert closed.is_set()
+ with pytest.raises(RuntimeError, match="closed"):
+ _ = handle.predictor
+
+
+def test_handle_closes_engine_after_client_error_and_preserves_first_error() -> None:
+ """客户端和引擎关闭都失败时仍执行两个动作,并保留最早的异常。"""
+ closed: list[str] = []
+
+ class Client:
+ """只提供可观察关闭错误的客户端替身。"""
+
+ async def aclose(self) -> None:
+ """模拟自有连接释放失败。"""
+ closed.append("client")
+ raise ValueError("client close failed")
+
+ def close_engine() -> None:
+ """模拟引擎释放同样失败,不能覆盖前面的客户端异常。"""
+ closed.append("engine")
+ raise RuntimeError("engine close failed")
+
+ handle = RuntimeHandle(Client(), backend="http-client", close_engine=close_engine)
+ with pytest.raises(ValueError, match="client close failed"):
+ asyncio.run(handle.aclose())
+ assert closed == ["client", "engine"]
+ assert handle.is_closed
+ with pytest.raises(RuntimeError, match="closed"):
+ _ = handle.predictor
+
+
+def test_mlx_queued_async_cancel_never_starts_delayed_inference() -> None:
+ """同步推理占用 MLX 锁时取消异步排队者,不得在锁释放后执行已取消请求。"""
+ client = _SyncClient()
+ handle = RuntimeHandle(client, backend="mlx-engine")
+
+ async def run() -> None:
+ """交错真实句柄的同步与异步入口,并验证取消后下一请求仍能获得锁。"""
+ with ThreadPoolExecutor(max_workers=1) as executor:
+ first = executor.submit(handle.batch_two_step_extract, [1])
+ try:
+ assert await asyncio.to_thread(client.entered.wait, 3)
+ queued = asyncio.create_task(handle.aio_batch_two_step_extract([2]))
+ await asyncio.sleep(0.03)
+ queued.cancel()
+ with pytest.raises(asyncio.CancelledError):
+ await queued
+ assert client.calls == [[1]]
+ finally:
+ client.release.set()
+ assert await asyncio.wrap_future(first) == [1]
+ assert await asyncio.wait_for(handle.aio_batch_two_step_extract([3]), timeout=1) == [3]
+
+ try:
+ asyncio.run(run())
+ assert client.calls == [[1], [3]]
+ finally:
+ handle.shutdown()
+
+
+def test_mlx_running_async_cancel_drains_before_next_sync_call() -> None:
+ """异步入口已开始 CPU 推理后,取消必须排空线程再向同步等待者交出 MLX 锁。"""
+ client = _SyncClient()
+ handle = RuntimeHandle(client, backend="mlx-engine")
+
+ async def run() -> None:
+ """取消正在执行的异步调用,并在同一锁上安排后续同步调用。"""
+ first = asyncio.create_task(handle.aio_batch_two_step_extract([1]))
+ assert await asyncio.to_thread(client.entered.wait, 3)
+ with ThreadPoolExecutor(max_workers=1) as executor:
+ second = executor.submit(handle.batch_two_step_extract, [2])
+ try:
+ first.cancel()
+ await asyncio.sleep(0.02)
+ assert not first.done()
+ assert client.calls == [[1]]
+ finally:
+ client.release.set()
+ with pytest.raises(asyncio.CancelledError):
+ await first
+ assert await asyncio.wait_for(asyncio.wrap_future(second), timeout=1) == [2]
+
+ try:
+ asyncio.run(run())
+ assert client.calls == [[1], [2]]
+ finally:
+ handle.shutdown()
+
+
+def test_cleanup_error_does_not_skip_other_runtimes_or_local_models(
+ local_models: list[_Atom], monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """一个 VLM 关闭失败时仍清理其它 VLM 与全部本地依赖,最终传播最初的异常。"""
+ closed: list[str] = []
+
+ def create(self: ModelSingleton, backend: str, path: str, *args: Any, **kwargs: Any) -> RuntimeHandle:
+ """为每个配置绑定唯一关闭动作,首个动作故意失败。"""
+
+ def close() -> None:
+ """记录关闭顺序并模拟一个后端释放失败。"""
+ closed.append(path)
+ if path == "first":
+ raise ValueError("first close failed")
+
+ return RuntimeHandle(_SyncClient(), backend=backend, close_engine=close)
+
+ monkeypatch.setattr(ModelSingleton, "_create_model", create)
+ owner = RuntimeOwner()
+ with _owned(owner):
+ hybrid.HybridLocalModelContextSingleton().get_model()
+ first = ModelSingleton().get_model("transformers", "first", None)
+ second = ModelSingleton().get_model("transformers", "second", None)
+ with pytest.raises(ValueError, match="first close failed"):
+ release_runtime_owner(owner)
+ assert closed == ["first", "second"]
+ assert first.is_closed and second.is_closed
+ assert not ModelSingleton._models
+ assert not ModelSingleton._closing
+ assert not hybrid.HybridLocalModelContextSingleton._models
+ assert not hybrid.AtomModelSingleton._models
+
+
+def test_backend_close_runs_outside_global_cache_lock(local_models: list[_Atom], monkeypatch: pytest.MonkeyPatch) -> None:
+ """关闭回调可等待另一个线程获取不同模型,证明外部关闭不持有全局缓存锁。"""
+ first, second = RuntimeOwner(), RuntimeOwner()
+ entered = threading.Event()
+
+ def borrow_other() -> None:
+ """在另一个应用中构造不同缓存键的模型。"""
+ with _owned(second):
+ ModelSingleton().get_model("transformers", "other", None)
+ entered.set()
+
+ def close() -> None:
+ """真实等待另一个线程完成模型借用,持锁实现会在此超时。"""
+ with ThreadPoolExecutor(max_workers=1) as executor:
+ executor.submit(borrow_other).result(timeout=2)
+
+ def create(self: ModelSingleton, backend: str, path: str, *args: Any, **kwargs: Any) -> RuntimeHandle:
+ """只给首个模型安装跨线程关闭回调。"""
+ return RuntimeHandle(_SyncClient(), backend=backend, close_engine=close if path == "first" else None)
+
+ monkeypatch.setattr(ModelSingleton, "_create_model", create)
+ with _owned(first):
+ ModelSingleton().get_model("transformers", "first", None)
+ release_runtime_owner(first)
+ assert entered.is_set()
+ release_runtime_owner(second)
+
+
+def test_same_cache_key_waits_for_previous_engine_close(local_models: list[_Atom], monkeypatch: pytest.MonkeyPatch) -> None:
+ """旧引擎关闭期间同键借用必须等待,不能并行创建第二份权重。"""
+ first, second = RuntimeOwner(), RuntimeOwner()
+ closing, release_close = threading.Event(), threading.Event()
+ created: list[RuntimeHandle] = []
+
+ def close() -> None:
+ """让真实缓存的关闭阶段停留在已撤销但资源尚未释放的状态。"""
+ closing.set()
+ assert release_close.wait(3)
+
+ def create(self: ModelSingleton, backend: str, *args: Any, **kwargs: Any) -> RuntimeHandle:
+ """记录同键工厂调用次数,只有首个实例需要阻塞关闭。"""
+ handle = RuntimeHandle(_SyncClient(), backend=backend, close_engine=close if not created else None)
+ created.append(handle)
+ return handle
+
+ def borrow() -> RuntimeHandle:
+ """在另一应用的上下文中请求正在关闭的同一模型。"""
+ with _owned(second):
+ return ModelSingleton().get_model("transformers", "same", None)
+
+ monkeypatch.setattr(ModelSingleton, "_create_model", create)
+ with _owned(first):
+ initial = ModelSingleton().get_model("transformers", "same", None)
+ with ThreadPoolExecutor(max_workers=2) as executor:
+ shutdown = executor.submit(release_runtime_owner, first)
+ try:
+ assert closing.wait(3)
+ replacement = executor.submit(borrow)
+ with pytest.raises(TimeoutError):
+ replacement.result(timeout=0.05)
+ assert len(created) == 1
+ finally:
+ release_close.set()
+ shutdown.result(timeout=3)
+ assert replacement.result(timeout=3) is not initial
+ assert len(created) == 2
+ release_runtime_owner(second)
+
+
[email protected]("backend", ["transformers", "http-client"])
+def test_same_cache_key_initializes_once_for_concurrent_owners(
+ local_models: list[_Atom], monkeypatch: pytest.MonkeyPatch, backend: str
+) -> None:
+ """同步和原生异步后端的同键并发首次获取只构造一次,并分别保留两个应用租约。"""
+ first, second = RuntimeOwner(), RuntimeOwner()
+ entered, release = threading.Event(), threading.Event()
+ created: list[RuntimeHandle] = []
+
+ def create(self: ModelSingleton, backend: str, *args: Any, **kwargs: Any) -> RuntimeHandle:
+ """延迟真实缓存工厂的完成,迫使第二应用等待同一个初始化屏障。"""
+ entered.set()
+ assert release.wait(3)
+ handle = RuntimeHandle(_SyncClient(), backend=backend)
+ created.append(handle)
+ return handle
+
+ def borrow(owner: RuntimeOwner) -> Any:
+ """从指定应用首次请求同一模型键。"""
+ with _owned(owner):
+ return ModelSingleton().get_model(backend, "same", "https://unused.test")
+
+ monkeypatch.setattr(ModelSingleton, "_create_model", create)
+ with ThreadPoolExecutor(max_workers=2) as executor:
+ left = executor.submit(borrow, first)
+ try:
+ assert entered.wait(3)
+ right = executor.submit(borrow, second)
+ with pytest.raises(TimeoutError):
+ right.result(timeout=0.05)
+ finally:
+ release.set()
+ predictor = left.result(timeout=3)
+ assert right.result(timeout=3) is predictor
+ assert len(created) == 1
+ assert not ModelSingleton._initializing
+ release_runtime_owner(first)
+ assert not predictor.is_closed
+ release_runtime_owner(second)
+ assert predictor.is_closed
+ assert not ModelSingleton._models
+
+
+def test_document_scope_propagates_to_async_worker_thread(local_models: list[_Atom]) -> None:
+ """异步文档在线程准备阶段继续使用同一租约,退出后原线程上下文不再有效。"""
+ owner = RuntimeOwner()
+
+ async def run() -> None:
+ """沿生产中的 to_thread 传播方式借用模型并捕获线程上下文。"""
+ with _owned(owner), runtime_document_scope():
+ context = await asyncio.to_thread(hybrid.HybridLocalModelContextSingleton().get_model)
+ delayed = await asyncio.to_thread(copy_context)
+ assert await asyncio.to_thread(lambda: context.mfr_model.name) == "mfr"
+ with pytest.raises(RuntimeError, match="lease has ended"):
+ delayed.run(current_runtime_owner)
+
+ asyncio.run(run())
+ release_runtime_owner(owner)
+ assert not hybrid.AtomModelSingleton._models
diff --git a/tests/unittest/test_runtime_shutdown_regressions.py b/tests/unittest/test_runtime_shutdown_regressions.py
new file mode 100644
index 00000000..1f368a30
--- /dev/null
+++ b/tests/unittest/test_runtime_shutdown_regressions.py
@@ -0,0 +1,445 @@
+"""用有界独立进程回归运行时关闭再入、失败构造及取消清理,不加载真实模型。"""
+
+from __future__ import annotations
+
+import subprocess
+import sys
+import textwrap
+from pathlib import Path
+
+import pytest
+
+
+_BOOTSTRAP = '''
+import asyncio
+import threading
+import time
+from contextlib import contextmanager
+from types import SimpleNamespace
+
+from mineru.model.runtime.ownership import RuntimeOwner, runtime_owner
+from mineru.model.vlm import runtime
+from mineru.model.vlm.handles import RuntimeHandle
+
+manager = runtime.ModelSingleton()
+
+class Client:
+ """不创建网络连接或模型的客户端,记录自有关闭动作。"""
+
+ def __init__(self) -> None:
+ """初始化关闭计数。"""
+ self.close_calls = 0
+
+ async def aclose(self) -> None:
+ """完成一次可观察的连接清理。"""
+ self.close_calls += 1
+
+def borrow(owner, backend, model_path, server_url):
+ """给真实缓存调用设置明确的应用身份。"""
+ token = runtime_owner.set(owner)
+ try:
+ return manager.get_model(backend, model_path, server_url)
+ finally:
+ runtime_owner.reset(token)
+'''
+
+
+def _run_isolated(script: str) -> None:
+ """把死锁上限限定在单个子进程,失败不遗留线程或污染其它模型缓存测试。"""
+ try:
+ completed = subprocess.run(
+ [sys.executable, "-c", _BOOTSTRAP + "\n" + textwrap.dedent(script)],
+ cwd=Path(__file__).resolve().parents[2],
+ capture_output=True,
+ text=True,
+ timeout=10,
+ check=False,
+ )
+ except subprocess.TimeoutExpired as exc:
+ pytest.fail(f"Runtime shutdown did not complete within 10 seconds: {exc.stdout!r}\n{exc.stderr!r}", pytrace=False)
+ assert completed.returncode == 0, completed.stdout + completed.stderr
+
+
+def test_sync_close_callback_rejects_another_key_in_its_closing_batch() -> None:
+ """A 的关闭回调不能等待同批尚未关闭的 B,同时仍允许另一应用借用未关闭键。"""
+ _run_isolated('''
+ first_owner, other_owner = RuntimeOwner(), RuntimeOwner()
+ closed, rejected = [], []
+
+ def create(self, backend, model_path, server_url, **kwargs):
+ """仅第一个模型的关闭动作尝试借用其它缓存键。"""
+ def close() -> None:
+ """非关闭键仍可复用,当前关闭批次中的键必须及时拒绝。"""
+ closed.append(model_path)
+ if model_path != "first":
+ return
+ assert borrow(other_owner, "transformers", "other", None) is other
+ try:
+ borrow(other_owner, "transformers", "second", None)
+ except RuntimeError as error:
+ rejected.append(str(error))
+ else:
+ raise AssertionError("A close callback acquired another key from its closing batch")
+ return RuntimeHandle(Client(), backend=backend, close_engine=close)
+
+ runtime.ModelSingleton._create_model = create
+ other = borrow(other_owner, "transformers", "other", None)
+ first = borrow(first_owner, "transformers", "first", None)
+ second = borrow(first_owner, "transformers", "second", None)
+ manager.release_owner(first_owner)
+ assert closed == ["first", "second"]
+ assert len(rejected) == 1
+ assert first.is_closed and second.is_closed and not other.is_closed
+ manager.release_owner(other_owner)
+ assert closed == ["first", "second", "other"]
+ manager.shutdown()
+ ''')
+
+
+def test_async_runtime_close_callback_rejects_its_own_cache_key() -> None:
+ """关闭再入保护必须传播到专属异步运行时线程,不能只比较外部关闭线程 ID。"""
+ _run_isolated('''
+ owner, other_owner = RuntimeOwner(), RuntimeOwner()
+ closed, rejected, clients = [], [], []
+
+ def create(self, backend, model_path, server_url, **kwargs):
+ """由真实 AsyncVlmPredictor 在所属循环内构造一个轻量 HTTP 句柄。"""
+ client = Client()
+ clients.append(client)
+ def close() -> None:
+ """异步运行时关闭回调可借用其它键,但不能等待当前键自己的屏障。"""
+ closed.append(backend)
+ if backend != "http-client":
+ return
+ borrow(other_owner, "transformers", "other", None)
+ try:
+ borrow(other_owner, "http-client", None, "http://example.test")
+ except RuntimeError as error:
+ rejected.append(str(error))
+ else:
+ raise AssertionError("An async close callback acquired its own closing key")
+ return RuntimeHandle(client, backend=backend, close_engine=close)
+
+ runtime.ModelSingleton._create_model = create
+ predictor = borrow(owner, "http-client", None, "http://example.test")
+ manager.release_owner(owner)
+ assert predictor.is_closed and closed == ["http-client"]
+ assert clients[0].close_calls == 1 and len(rejected) == 1
+ manager.release_owner(other_owner)
+ assert closed == ["http-client", "transformers"]
+ manager.shutdown()
+ ''')
+
+
[email protected]("asynchronous", [False, True])
+def test_direct_cached_handle_close_rejects_same_key_reentry(asynchronous: bool) -> None:
+ """直接关闭仍在缓存中的句柄时,同键再入不能绕过 cached.is_closed 分支的保护。"""
+ _run_isolated(
+ f"asynchronous = {asynchronous!r}\n"
+ + '''
+owner, other_owner = RuntimeOwner(), RuntimeOwner()
+created, closed, rejected = [], [], []
+
+def create(self, backend, model_path, server_url, **kwargs):
+ """首个句柄在其关闭回调中故意重新请求同一缓存键。"""
+ identity = len(created)
+ def close() -> None:
+ """直接句柄关闭未提前撤销缓存,同键请求仍应立即失败。"""
+ closed.append(identity)
+ if identity != 0:
+ return
+ try:
+ borrow(other_owner, "transformers", "same", None)
+ except RuntimeError as error:
+ rejected.append(str(error))
+ else:
+ raise AssertionError("Direct handle close allowed same-key reentry")
+ handle = RuntimeHandle(Client(), backend=backend, close_engine=close)
+ created.append(handle)
+ return handle
+
+runtime.ModelSingleton._create_model = create
+handle = borrow(owner, "transformers", "same", None)
+if asynchronous:
+ asyncio.run(handle.aclose())
+else:
+ handle.shutdown()
+assert handle.is_closed and closed == [0] and len(rejected) == 1
+replacement = borrow(other_owner, "transformers", "same", None)
+assert replacement is not handle and len(created) == 2
+manager.release_owner(owner)
+assert not replacement.is_closed
+manager.release_owner(other_owner)
+assert closed == [0, 1]
+manager.shutdown()
+'''
+ )
+
+
+def test_close_callback_does_not_wait_for_initializer_that_needs_closing_key() -> None:
+ """A 的关闭回调不能等待正在构造且反向等待 A 的 B,避免关闭与初始化互等。"""
+ _run_isolated('''
+ first_owner, second_owner = RuntimeOwner(), RuntimeOwner()
+ b_entered, allow_b = threading.Event(), threading.Event()
+ a_handles, b_results, errors, rejected = [], [], [], []
+
+ def create(self, backend, model_path, server_url, **kwargs):
+ """让 B 的构造等待 A 已开始关闭,再请求 A 的替代实例。"""
+ if model_path == "B":
+ b_entered.set()
+ assert allow_b.wait(3)
+ borrow(second_owner, "transformers", "A", None)
+ return RuntimeHandle(Client(), backend=backend)
+ identity = len(a_handles)
+ def close() -> None:
+ """仅旧 A 的关闭回调尝试获取仍在普通构造阶段的 B。"""
+ if identity != 0:
+ return
+ allow_b.set()
+ try:
+ borrow(second_owner, "transformers", "B", None)
+ except RuntimeError as error:
+ rejected.append(str(error))
+ else:
+ raise AssertionError("A close callback waited for a pending initializer")
+ handle = RuntimeHandle(Client(), backend=backend, close_engine=close)
+ a_handles.append(handle)
+ return handle
+
+ def initialize_b() -> None:
+ """在独立应用中执行真实同键初始化等待流程。"""
+ try:
+ b_results.append(borrow(second_owner, "transformers", "B", None))
+ except BaseException as error:
+ errors.append(error)
+
+ runtime.ModelSingleton._create_model = create
+ initial_a = borrow(first_owner, "transformers", "A", None)
+ worker = threading.Thread(target=initialize_b, daemon=True)
+ worker.start()
+ assert b_entered.wait(2)
+ manager.release_owner(first_owner)
+ worker.join(2)
+ assert not worker.is_alive() and not errors
+ assert initial_a.is_closed and len(rejected) == 1
+ assert len(a_handles) == 2 and len(b_results) == 1
+ assert not a_handles[1].is_closed and not b_results[0].is_closed
+ manager.release_owner(second_owner)
+ assert a_handles[1].is_closed and b_results[0].is_closed
+ manager.shutdown()
+ ''')
+
+
+def test_failed_real_factory_closes_engine_outside_global_cache_lock() -> None:
+ """真实工厂的客户端构造失败清理也必须允许其它线程加载不同模型键。"""
+ _run_isolated('''
+ import sys
+ from mineru.model.runtime import llama_device
+
+ owner, other_owner = RuntimeOwner(), RuntimeOwner()
+ finished = threading.Event()
+ waited, worker_errors, workers = [], [], []
+
+ @contextmanager
+ def engine_scope(selection):
+ """跳过真实设备环境操作,保留工厂内部的引擎创建顺序。"""
+ yield
+
+ def client_factory(**kwargs):
+ """只让 llama 高层客户端失败,其它键正常创建轻量客户端。"""
+ if kwargs["backend"] == "llama-cpp-engine":
+ raise ValueError("client initialization failed")
+ return Client()
+
+ class Engine:
+ """不读取权重的引擎替身,使用真实工厂调用其关闭动作。"""
+
+ def __init__(self, *args, **kwargs) -> None:
+ """接收真实工厂的路径和选项但不分配模型。"""
+
+ def close(self) -> None:
+ """在清理回调中等待不同线程完成另一个缓存键的构造。"""
+ def load_other() -> None:
+ """给另一应用构造轻量异步客户端,并记录可能的异常。"""
+ try:
+ borrow(other_owner, "http-client", None, "http://other.test")
+ except BaseException as error:
+ worker_errors.append(error)
+ finally:
+ finished.set()
+ worker = threading.Thread(target=load_other, daemon=True)
+ workers.append(worker)
+ worker.start()
+ waited.append(finished.wait(2))
+
+ llama_device.resolve_llama_device = lambda **kwargs: SimpleNamespace(device="cpu")
+ llama_device.llama_engine_scope = engine_scope
+ runtime.MinerUClient = client_factory
+ sys.modules["mineru_llama_cpp"] = SimpleNamespace(Engine=Engine)
+ try:
+ borrow(owner, "llama-cpp-engine", "/tmp/fake-model", None)
+ except ValueError as error:
+ assert str(error) == "client initialization failed"
+ else:
+ raise AssertionError("The original initialization error was lost")
+ assert finished.wait(2)
+ for worker in workers:
+ worker.join(2)
+ manager.release_owner(owner)
+ manager.release_owner(other_owner)
+ manager.shutdown()
+ assert not worker_errors
+ assert waited == [True], "Failed initialization cleanup still holds the global model lock"
+ ''')
+
+
[email protected]("stage", ["inference_drain", "client_close"])
+def test_cancelled_handle_close_drains_client_and_engine_once(stage: str) -> None:
+ """等待推理或关闭客户端期间重复取消,都必须完成清理再向调用方传播取消。"""
+ _run_isolated(
+ f"stage = {stage!r}\n"
+ + '''
+class BlockingClient(Client):
+ """分别阻塞同步推理与异步关闭,使两个取消时序可以稳定重现。"""
+
+ def __init__(self) -> None:
+ """在调用方事件循环中创建关闭门,线程只使用线程事件。"""
+ super().__init__()
+ self.entered = threading.Event()
+ self.release = threading.Event()
+ self.close_started = asyncio.Event()
+ self.finish_close = asyncio.Event()
+ self.close_completed = False
+
+ def batch_two_step_extract(self, images, **kwargs):
+ """模拟无法被协程取消强制中断的同步模型调用。"""
+ self.entered.set()
+ assert self.release.wait(3)
+ return images
+
+ async def aclose(self) -> None:
+ """记录关闭开始与真正完成,不能把一次调用误认为资源已经释放。"""
+ self.close_calls += 1
+ self.close_started.set()
+ await self.finish_close.wait()
+ self.close_completed = True
+
+async def check() -> None:
+ """重复取消真实句柄的关闭入口,并验证后续幂等关闭不会重复释放。"""
+ client = BlockingClient()
+ engine_closed = []
+ handle = RuntimeHandle(client, backend="http-client", close_engine=lambda: engine_closed.append(True))
+ work = None
+ if stage == "inference_drain":
+ client.finish_close.set()
+ work = asyncio.create_task(handle.aio_batch_two_step_extract([1]))
+ assert await asyncio.to_thread(client.entered.wait, 2)
+ closing = asyncio.create_task(handle.aclose())
+ try:
+ if stage == "client_close":
+ await asyncio.wait_for(client.close_started.wait(), 2)
+ else:
+ await asyncio.sleep(0.03)
+ closing.cancel()
+ await asyncio.sleep(0)
+ closing.cancel()
+ await asyncio.sleep(0.02)
+ assert not closing.done(), "Closing returned before the blocked resource finished"
+ finally:
+ client.release.set()
+ client.finish_close.set()
+ if work is not None:
+ assert await work == [1]
+ try:
+ await closing
+ except asyncio.CancelledError:
+ pass
+ else:
+ raise AssertionError("Caller cancellation was swallowed")
+ assert client.close_calls == 1 and client.close_completed
+ assert engine_closed == [True] and handle.is_closed
+ await handle.aclose()
+ assert client.close_calls == 1 and engine_closed == [True]
+
+asyncio.run(check())
+'''
+ )
+
+
+def test_closed_owner_is_rejected_before_device_probe_or_model_download() -> None:
+ """已关闭应用在任何缺省模型路径解析或设备探测前就必须被拒绝。"""
+ _run_isolated("""
+ calls = []
+ runtime.get_device = lambda: calls.append("device") or "cpu"
+ runtime.MINERU_2_5_PRO_2605_1_2B = SimpleNamespace(ensure=lambda: calls.append("ensure") or "/tmp/mock-model")
+ owner = RuntimeOwner()
+ owner.close()
+ try:
+ borrow(owner, "vllm-engine", None, None)
+ except RuntimeError:
+ pass
+ else:
+ raise AssertionError("A closed owner started model loading")
+ assert calls == []
+ """)
+
+
+def test_owner_release_waits_for_inflight_synchronous_initialization() -> None:
+ """锁外初始化与应用关闭竞争时,关闭等待构造收尾并禁止迟到发布缓存。"""
+ _run_isolated('''
+ owner, other_owner = RuntimeOwner(), RuntimeOwner()
+ entered, release, close_done = threading.Event(), threading.Event(), threading.Event()
+ created, engine_closed, borrow_errors, close_errors = [], [], [], []
+
+ def create(self, backend, model_path, server_url, **kwargs):
+ """只阻塞第一个实例,后续应用仍可重新构造同键模型。"""
+ identity = len(created)
+ handle = RuntimeHandle(Client(), backend=backend, close_engine=lambda: engine_closed.append(identity))
+ created.append(handle)
+ if identity == 0:
+ entered.set()
+ assert release.wait(3)
+ return handle
+
+ def initialize() -> None:
+ """在原应用中发起没有文档范围的预热式模型构造。"""
+ try:
+ borrow(owner, "transformers", "same", None)
+ except BaseException as error:
+ borrow_errors.append(error)
+
+ def close() -> None:
+ """记录关闭真正返回的时刻,不能只验证缓存最终为空。"""
+ try:
+ manager.release_owner(owner)
+ except BaseException as error:
+ close_errors.append(error)
+ finally:
+ close_done.set()
+
+ runtime.ModelSingleton._create_model = create
+ initializing = threading.Thread(target=initialize, daemon=True)
+ initializing.start()
+ assert entered.wait(2)
+ closing = threading.Thread(target=close, daemon=True)
+ closing.start()
+ try:
+ deadline = time.monotonic() + 2
+ while not owner.closed and time.monotonic() < deadline:
+ time.sleep(0.005)
+ assert owner.closed
+ assert not close_done.wait(0.05), "Owner release skipped its inflight initializer"
+ finally:
+ release.set()
+ initializing.join(2)
+ closing.join(2)
+ assert not initializing.is_alive() and not closing.is_alive()
+ assert not close_errors
+ assert len(borrow_errors) == 1 and isinstance(borrow_errors[0], RuntimeError)
+ assert created[0].is_closed and engine_closed == [0]
+ replacement = borrow(other_owner, "transformers", "same", None)
+ assert replacement is not created[0] and len(created) == 2
+ manager.release_owner(other_owner)
+ assert engine_closed == [0, 1]
+ manager.shutdown()
+ ''')
diff --git a/tests/unittest/test_structured_content_render.py b/tests/unittest/test_structured_content_render.py
index 346475f5..31e8053a 100644
--- a/tests/unittest/test_structured_content_render.py
+++ b/tests/unittest/test_structured_content_render.py
@@ -137,7 +137,7 @@ def test_structured_content_keeps_chart_content_separate_from_base64_source(
}
}
)
- monkeypatch.setattr("mineru.config.config", configured)
+ monkeypatch.setattr("mineru.config.get_config", lambda: configured)
chart = ChartBlock(
type="chart",
index=0,
@@ -183,7 +183,7 @@ def test_structured_content_renders_equation_as_raw_latex_with_single_image_sour
}
}
)
- monkeypatch.setattr("mineru.config.config", configured)
+ monkeypatch.setattr("mineru.config.get_config", lambda: configured)
middle = _middle(
_page(
0,
diff --git a/tests/unittest/test_task_api.py b/tests/unittest/test_task_api.py
index 6db1aa51..134cb711 100644
--- a/tests/unittest/test_task_api.py
+++ b/tests/unittest/test_task_api.py
@@ -23,7 +23,8 @@ from fastapi.testclient import TestClient
from mineru.kit.router import RouterSettings
from mineru.kit.router import create_app as create_router
-from mineru.parser import api_server, task_api
+from mineru.parser import ParseResult, api_server
+from mineru.tasks.service import TERMINAL_TASK_STATUSES
HTML = b"<!doctype html><html><body><h1>Shared tasks</h1><p>Preserved content.</p></body></html>"
SECOND = b"<h1>Second document</h1><p>Second content.</p>"
@@ -105,7 +106,7 @@ class _WorkerTransport(httpx.AsyncBaseTransport):
"""等待任务与 Job 清理完成,再释放测试 worker 的解析运行时。"""
for app in self.apps.values():
await app.state.task_store.shutdown()
- await app.state.job_store.shutdown()
+ await app.state.execution_store.shutdown()
@pytest.fixture(params=["api", "router"])
@@ -175,7 +176,7 @@ def _terminal(client: TestClient, task: dict[str, Any], *, headers: dict[str, st
response = client.get(task_url(client, task), headers=headers)
assert response.status_code == 200, response.text
payload = response.json()
- if payload["status"] in task_api.TERMINAL_TASK_STATUSES:
+ if payload["status"] in TERMINAL_TASK_STATUSES:
return payload
assert all(file.get("result_url") is None for file in payload["files"] if file["status"] != "completed")
time.sleep(0.01)
@@ -205,8 +206,8 @@ def _archive(client: TestClient, task: dict[str, Any], index: int = 0) -> dict[s
def _job_count(app: FastAPI) -> int:
"""观察唯一执行队列,准备任务不应该产生 Job 或 worker 预占。"""
- if hasattr(app.state, "job_store"):
- return len(app.state.job_store._jobs)
+ if hasattr(app.state, "execution_store"):
+ return len(app.state.execution_store._executions)
return len(app.state.registry.list("job"))
@@ -424,7 +425,7 @@ def test_cancel_running_task_releases_execution_slot(
if hasattr(app.state, "worker_pool"):
assert sum(worker.active_jobs for worker in app.state.worker_pool.workers) == 0
else:
- assert not app.state.job_store._tasks
+ assert not app.state.execution_store._tasks
def test_status_never_reads_or_rebuilds_zip(service: tuple[TestClient, FastAPI], monkeypatch: pytest.MonkeyPatch) -> None:
@@ -539,7 +540,7 @@ def test_input_validation_and_openapi(service: tuple[TestClient, FastAPI]) -> No
assert "file_id" in id_schema["required"]
assert id_schema["properties"]["file_id"]["type"] == "string"
assert set(spec["components"]["schemas"]["TaskUploadLink"]["properties"]) == {"url", "headers", "expires_at"}
- if hasattr(app.state, "job_store"):
+ if hasattr(app.state, "execution_store"):
metadata_schema = spec["components"]["schemas"]["FileObjectModel"]
assert "file_name" in metadata_schema["required"]
assert "filename" not in metadata_schema["properties"]
@@ -563,6 +564,14 @@ def test_server_file_ids_survive_upload_and_router_forwarding(service: tuple[Tes
completed = _terminal(client, task)
assert [item["file_id"] for item in completed["files"]] == expected_ids
assert all(item["result_url"].endswith(f"/{item['file_id']}/content") for item in completed["files"])
+ record = app.state.task_store._records[task["task_id"]]
+ for item, state in zip(completed["files"], record.file_states):
+ assert state.execution.file_id == item["file_id"]
+ assert state.execution.result_file_id == item["file_id"]
+ assert state.execution.execution_id == state.execution_id
+ if hasattr(app.state, "execution_store"):
+ for execution in app.state.execution_store._executions.values():
+ assert execution.request.source.file_id != execution.request.file_id
if hasattr(app.state, "test_transport"):
worker_ids = {
item.file_id
@@ -573,6 +582,35 @@ def test_server_file_ids_survive_upload_and_router_forwarding(service: tuple[Tes
assert len(worker_ids) == count and worker_ids.isdisjoint(expected_ids)
+def test_parent_task_aggregates_single_file_executions_and_usage(
+ service: tuple[TestClient, FastAPI], monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """父 Task 独占部分成功状态,用量仍按两个真实执行和一个成功文件累计。"""
+ client, app = service
+ original = api_server.parse_async
+
+ async def infer(path: str, **kwargs: Any) -> ParseResult:
+ """只让指定文件失败,其余文件经过真实 Flash 解析及产物保存。"""
+ if b"intentional-failure" in Path(path).read_bytes():
+ raise ValueError("intentional file failure")
+ return await original(path, **kwargs)
+
+ monkeypatch.setattr(api_server, "parse_async", infer)
+ before = client.get("/v1/usage").json()["current"]
+ inputs = [("good.html", HTML), ("bad.html", b"<html><body>intentional-failure</body></html>")]
+ created = _submit(client, inputs, hashes=False)
+ _upload(client, created, inputs)
+ completed = _terminal(client, created)
+ assert completed["status"] == "partial"
+ assert completed["progress"] == {"completed": 1, "failed": 1, "total": 2}
+ assert [item["status"] for item in completed["files"]] == ["completed", "failed"]
+ states = app.state.task_store._records[created["task_id"]].file_states
+ assert [state.execution.status for state in states] == ["completed", "failed"]
+ after = client.get("/v1/usage").json()["current"]
+ assert after["jobs_created"] - before["jobs_created"] == 2
+ assert after["files_processed"] - before["files_processed"] == 1
+
+
@pytest.mark.parametrize("file_id", ["", "with space", "bad/id", "x" * 65, 123, False])
def test_invalid_explicit_file_ids_fail_before_creation(service: tuple[TestClient, FastAPI], file_id: object) -> None:
"""可选标识不放宽显式值校验,错误仍定位到新 files 字段且不创建任务。"""
@@ -850,7 +888,7 @@ def test_cancel_url_download_closes_stream_and_releases_worker(
assert time.monotonic() - before < 2
assert closed.wait(5)
deadline = time.monotonic() + 5
- while any(worker.state.job_store._tasks for worker in _execution_apps(app)):
+ while any(worker.state.execution_store._tasks for worker in _execution_apps(app)):
assert time.monotonic() < deadline, "Canceled URL job retained its execution slot"
time.sleep(0.01)
if hasattr(app.state, "worker_pool"):
@@ -971,7 +1009,7 @@ def test_completed_child_retention_starts_after_parent_finishes(service: tuple[T
app.state.registry.retention_seconds = 1
for worker in _execution_apps(app):
worker.state.task_store.retention_seconds = 1
- worker.state.job_store.retention_seconds = 1
+ worker.state.execution_store.retention_seconds = 1
worker.state.file_store.retention_seconds = 1
task = _submit(client, [("a.html", HTML), ("b.html", SECOND)], hashes=False)
assert client.put(pending_files(task)[0].get("upload").get("url"), content=HTML).status_code == 200
@@ -979,7 +1017,7 @@ def test_completed_child_retention_starts_after_parent_finishes(service: tuple[T
result_url = ready["files"][0].get("result_url")
time.sleep(1.05)
for worker in _execution_apps(app):
- worker.state.job_store.collect_expired()
+ worker.state.execution_store.collect_expired()
worker.state.file_store.collect_expired()
assert client.get(task_url(client, task)).json()["files"][0]["status"] == "completed"
assert client.get(result_url).status_code == 200
diff --git a/tests/unittest/test_task_file_states.py b/tests/unittest/test_task_file_states.py
index c112538c..70e7218a 100644
--- a/tests/unittest/test_task_file_states.py
+++ b/tests/unittest/test_task_file_states.py
@@ -87,7 +87,7 @@ def test_url_preparation_does_not_block_ready_files_or_use_upload_expiry(
assert completed["status"] == "completed"
assert completed["files"][0].get("bytes") == len(HTML)
assert b"Preserved content" in _archive(client, completed)["markdown.md"]
- assert all(not job.prepared_sources for job in app.state.job_store._jobs.values())
+ assert all(not job.prepared_input for job in app.state.execution_store._executions.values())
finally:
release.set()
@@ -129,8 +129,8 @@ def test_ready_url_inputs_wait_in_pending_with_bounded_preparation(
assert time.monotonic() < deadline
time.sleep(0.01)
record = app.state.task_store._records[second["task_id"]]
- job = app.state.job_store._jobs[record.file_states[0].execution_id]
- assert job.prepared_sources and job.files[0].parse is None
+ job = app.state.execution_store._executions[record.file_states[0].execution_id]
+ assert job.prepared_input and job.parse is None
third = _url_task(client, "third")
assert client.get(task_url(client, third)).json()["status"] == "waiting-file"
assert downloads == ["/second"]
@@ -140,12 +140,12 @@ def test_ready_url_inputs_wait_in_pending_with_bounded_preparation(
while "/third" not in downloads:
assert time.monotonic() < deadline
time.sleep(0.01)
- assert not job.prepared_sources
+ assert not job.prepared_input
release.set()
assert _terminal(client, first)["status"] == "completed"
assert _terminal(client, second)["status"] == ("canceled" if cancel_ready else "completed")
assert _terminal(client, third)["status"] == "completed"
- assert all(not item.prepared_sources for item in app.state.job_store._jobs.values())
+ assert all(not item.prepared_input for item in app.state.execution_store._executions.values())
finally:
release.set()
@@ -169,13 +169,13 @@ def test_file_duration_includes_preparation_but_excludes_queue_wait(tmp_path: Pa
monkeypatch.setattr(api_server, "parse_async", infer)
app = api_server.create_app(upload_dir=str(tmp_path), tier="flash", concurrency=1)
with TestClient(app) as client:
- client.portal.call(app.state.job_store._semaphore.acquire)
+ client.portal.call(app.state.execution_store._semaphore.acquire)
task = _url_task(client)
deadline = time.monotonic() + 3
while client.get(task_url(client, task)).json()["status"] != "pending":
assert time.monotonic() < deadline
time.sleep(0.01)
clock[0] += 1000
- client.portal.call(app.state.job_store._semaphore.release)
+ client.portal.call(app.state.execution_store._semaphore.release)
result = _terminal(client, task)
assert result["files"][0].get("parse")["duration_ms"] == 5000
diff --git a/tests/unittest/test_tasks_architecture.py b/tests/unittest/test_tasks_architecture.py
new file mode 100644
index 00000000..87284428
--- /dev/null
+++ b/tests/unittest/test_tasks_architecture.py
@@ -0,0 +1,66 @@
+"""保护 Tasks 共享契约、业务服务与具体部署实现之间的依赖边界。"""
+
+from __future__ import annotations
+
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+
[email protected]("module", ["contracts", "errors", "execution", "service"])
+def test_shared_tasks_import_without_http_or_parser_runtime(module: str, tmp_path: Path) -> None:
+ """独立导入共享层时不得装载 HTTP、本地解析器或推理运行时。"""
+ script = """
+import importlib
+import sys
+
+importlib.import_module(sys.argv[1])
+for name in (
+ "fastapi", "starlette", "torch", "transformers", "pypdfium2",
+ "mineru.parser", "mineru.backend", "mineru.model", "mineru.config",
+):
+ assert name not in sys.modules, name
+"""
+ completed = subprocess.run(
+ [sys.executable, "-c", script, f"mineru.tasks.{module}"],
+ cwd=Path(__file__).resolve().parents[2],
+ env={**os.environ, "MINERU_CONFIG": str(tmp_path / "missing.yaml")},
+ capture_output=True,
+ text=True,
+ check=False,
+ )
+ assert completed.returncode == 0, completed.stderr
+
+
+def test_router_import_does_not_load_local_api_server(tmp_path: Path) -> None:
+ """Router 通过共享 Tasks 层组装路由,不借用本地 API Server 的实现。"""
+ script = """
+import sys
+
+from mineru.kit.router.app import create_app
+from mineru.kit.router.workers import RouterSettings
+
+for name in (
+ "mineru.parser.api_server", "mineru.parser.task_backend", "mineru.parser.task_api",
+ "torch", "transformers", "pypdfium2",
+):
+ assert name not in sys.modules, name
+"""
+ completed = subprocess.run(
+ [sys.executable, "-c", script],
+ cwd=Path(__file__).resolve().parents[2],
+ env={**os.environ, "MINERU_CONFIG": str(tmp_path / "missing.yaml")},
+ capture_output=True,
+ text=True,
+ check=False,
+ )
+ assert completed.returncode == 0, completed.stderr
+
+
+def test_removed_private_task_api_module_has_no_forwarding_shell() -> None:
+ """旧私有模块必须彻底移除,所有内部消费者显式依赖新的职责归属。"""
+ old_module = Path(__file__).resolve().parents[2] / "mineru" / "parser" / "task_api.py"
+ assert not old_module.exists()
diff --git a/tests/unittest/test_tasks_client_contract.py b/tests/unittest/test_tasks_client_contract.py
new file mode 100644
index 00000000..62201ef1
--- /dev/null
+++ b/tests/unittest/test_tasks_client_contract.py
@@ -0,0 +1,199 @@
+"""共享 Tasks 字段后仍保留 SDK 与同步网关各自的响应消费边界。"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+from collections.abc import Iterator
+from pathlib import Path
+from typing import Any
+
+import httpx
+import pytest
+
+from mineru.kit.file_parse.client import TaskClient
+from mineru.kit.file_parse.models import GatewayError, ParseOptions, RemoteTask, fingerprint
+from mineru.kit.file_parse.settings import FileParseSettings
+from mineru.parser.api_client import MinerUApiParser, _V1APIError
+
+
+def _pending_task() -> dict[str, Any]:
+ """构造客户端所需的最小单文件详情,不要求它携带服务端管理字段。"""
+ return {
+ "task_id": "task-one",
+ "status": "pending",
+ "output_formats": ["default"],
+ "files": [{"file_id": "file-one", "file_name": "file.pdf", "bytes": 3, "status": "pending"}],
+ }
+
+
+def clients(tmp_path: Path) -> Iterator[tuple[MinerUApiParser, TaskClient]]:
+ """使用禁止联网的传输构造真实客户端,并在每项测试结束后关闭连接池。"""
+
+ def reject_request(request: httpx.Request) -> httpx.Response:
+ """响应契约测试只能消费给定快照,不允许发送任何请求。"""
+ raise AssertionError(f"Unexpected network request: {request.method}")
+
+ api_url = "https://api.test/gateway"
+ gateway = TaskClient(
+ FileParseSettings(api_url=api_url, state_dir=tmp_path),
+ transport=httpx.MockTransport(reject_request),
+ )
+ try:
+ yield MinerUApiParser(api_url=api_url), gateway
+ finally:
+ asyncio.run(gateway.close())
+
+
+ "task_patch,file_patch,sdk_accepts,gateway_accepts",
+ [
+ pytest.param({}, {}, True, True, id="minimal-detail"),
+ pytest.param({}, {"bytes": "3"}, False, True, id="numeric-string-size"),
+ pytest.param({"output_formats": ["default", "default"]}, {}, False, True, id="duplicate-formats"),
+ pytest.param({}, {"error": {"code": "remote-code"}}, True, True, id="partial-error-details"),
+ pytest.param(
+ {"status": "completed"},
+ {"status": "completed", "result_url": ""},
+ True,
+ False,
+ id="empty-completed-result-url",
+ ),
+ pytest.param({}, {"result_url": "https://storage.test/early"}, True, False, id="premature-result-url"),
+ pytest.param(
+ {"status": "waiting-file"},
+ {"status": "waiting-file", "upload": {"url": "https://storage.test/input"}},
+ True,
+ False,
+ id="partial-upload-descriptor",
+ ),
+ pytest.param(
+ {"status": "waiting-file"},
+ {
+ "status": "waiting-file",
+ "url": "https://source.test/file.pdf",
+ "upload": {"url": "https://storage.test/input", "headers": {}, "expires_at": 9999999999},
+ },
+ True,
+ True,
+ id="unconsumed-source-url",
+ ),
+ ],
+)
+def test_snapshot_consumers_keep_existing_acceptance_boundaries(
+ clients: tuple[MinerUApiParser, TaskClient],
+ task_patch: dict[str, Any],
+ file_patch: dict[str, Any],
+ sdk_accepts: bool,
+ gateway_accepts: bool,
+) -> None:
+ """实际快照入口保留各自宽容度,不把服务端完整响应模型强加给消费者。"""
+ sdk, gateway = clients
+ expected = _pending_task()
+ payload = _pending_task()
+ payload.update(task_patch)
+ payload["files"][0].update(file_patch)
+ response = httpx.Response(200, json=payload)
+ if sdk_accepts:
+ assert sdk._snapshot(response, expected) == payload
+ else:
+ with pytest.raises(_V1APIError) as sdk_error:
+ sdk._snapshot(response, expected)
+ assert sdk_error.value.code == "invalid_response"
+
+ if gateway_accepts:
+ actual = gateway._snapshot(response.content, RemoteTask.model_validate(expected), ["default"])
+ assert actual.task_id == expected["task_id"]
+ assert actual.files[0].bytes == 3
+ assert actual.output_formats == payload["output_formats"]
+ assert actual.files[0].error == payload["files"][0].get("error")
+ else:
+ with pytest.raises(GatewayError) as gateway_error:
+ gateway._snapshot(response.content, RemoteTask.model_validate(expected), ["default"])
+ assert gateway_error.value.code == "upstream_protocol_error"
+ assert gateway_error.value.status_code == 502
+
+
+def test_sdk_preserves_response_extensions_and_does_not_reuse_missing_fields(
+ clients: tuple[MinerUApiParser, TaskClient],
+) -> None:
+ """身份投影只负责校验,SDK 仍返回完整新快照且不会补回上次的可选字段。"""
+ sdk, _ = clients
+ expected = _pending_task()
+ expected["files"][0]["error"] = {"code": "stale-error"}
+ payload = _pending_task()
+ payload["provider_extension"] = {"queue": "priority"}
+ payload["files"][0]["provider_extension"] = {"attempt": 2}
+ payload["files"][0]["parse"] = {"duration_ms": 25, "provider_timing": "accepted"}
+
+ actual = sdk._snapshot(httpx.Response(200, json=payload), expected)
+
+ assert actual == payload
+ assert "error" not in actual["files"][0]
+ assert "upload" not in actual["files"][0]
+
+
+ "file_patch,message",
+ [
+ ({"file_id": "", "status": "unknown"}, "Inconsistent Tasks response"),
+ ({"file_name": 7, "status": "unknown"}, "Inconsistent Tasks response"),
+ ({"file_id": 7}, "Inconsistent Tasks response"),
+ ({"status": "partial"}, "Invalid task file state"),
+ ({"status": "unknown", "upload": [], "file_id": "changed"}, "Invalid task file state"),
+ ({"status": "running", "upload": {}, "file_id": "changed"}, "Invalid task upload descriptor"),
+ ({"file_id": "changed", "bytes": 4}, "Task input identity changed"),
+ ],
+)
+def test_sdk_preserves_identity_error_priority(
+ clients: tuple[MinerUApiParser, TaskClient], file_patch: dict[str, Any], message: str
+) -> None:
+ """同时存在多个响应错误时,仍先报告原有身份、状态和上传边界中的首个错误。"""
+ sdk, _ = clients
+ payload = _pending_task()
+ payload["files"][0].update(file_patch)
+ with pytest.raises(_V1APIError) as error:
+ sdk._snapshot(httpx.Response(200, json=payload), _pending_task())
+ assert error.value.code == "invalid_response"
+ assert error.value.message == message
+
+
[email protected]("status", ["pending", "completed"])
+def test_creation_receipt_keeps_local_recovery_context(clients: tuple[MinerUApiParser, TaskClient], status: str) -> None:
+ """短创建回执从请求补齐大小和格式,已完成回执也可先保存再获取详情。"""
+ sdk, gateway = clients
+ receipt = {
+ "task_id": "task-one",
+ "status": status,
+ "files": [{"file_id": "file-one", "file_name": "file.pdf", "status": status}],
+ }
+ request = {"files": [{"file_name": "file.pdf", "bytes": 3}], "output_formats": ["default"]}
+ sdk_task = sdk._created_task(httpx.Response(202, json=receipt), request)
+ gateway_task = gateway._created_snapshot(json.dumps(receipt).encode(), request)
+
+ assert sdk_task["files"][0]["bytes"] == 3
+ assert "result_url" not in sdk_task["files"][0]
+ restored = RemoteTask.model_validate_json(gateway_task.model_dump_json())
+ assert restored.task_id == "task-one"
+ assert restored.files[0].bytes == 3
+ assert restored.output_formats == ["default"]
+ assert restored.files[0].result_url is None
+ assert restored.details_loaded is False
+
+
[email protected]("page_range,normalized_range", [(" ALL ", ""), (" 1-3 ", "1-3")])
+def test_parse_options_normalization_preserves_request_identity(page_range: str, normalized_range: str) -> None:
+ """共享选项模型后仍保留格式去重顺序和页范围归一化,等价请求指纹相同。"""
+ options = ParseOptions(page_range=page_range, output_formats=["html", "default", "html", "docx", "default"])
+ normalized = ParseOptions(page_range=normalized_range, output_formats=["html", "default", "docx"])
+ assert options.page_range == normalized_range
+ assert options.output_formats == ["html", "default", "docx"]
+ assert fingerprint("file.pdf", 3, "a" * 64, options) == fingerprint("file.pdf", 3, "a" * 64, normalized)
+
+
[email protected]("extra", [{"wait_timeout": 10}, {"include_model_output": False}])
+def test_parse_options_rejects_fields_outside_parse_identity(extra: dict[str, Any]) -> None:
+ """等待预算和已移除参数不能混入解析身份或被继承模型静默忽略。"""
+ with pytest.raises(ValueError, match="extra_forbidden"):
+ ParseOptions(**extra)
diff --git a/tests/unittest/test_tasks_sdk.py b/tests/unittest/test_tasks_sdk.py
index 580944b0..d9d0d63b 100644
--- a/tests/unittest/test_tasks_sdk.py
+++ b/tests/unittest/test_tasks_sdk.py
@@ -16,7 +16,7 @@ import pytest
from mineru.parser import ApiJobStatus, ApiTaskStatus, MinerUApiParser
from mineru.parser import api_client
-from mineru.parser.task_api import CreateTaskRequest
+from mineru.tasks.contracts import CreateTaskRequest
from mineru.parser.task_outputs import build_task_output
from mineru.parser.base import ParseResult
from mineru.types import MiddleJson, PageInfo
diff --git a/tests/unittest/test_transformers5_runtime.py b/tests/unittest/test_transformers5_runtime.py
index 92174ecb..2e0bc763 100644
--- a/tests/unittest/test_transformers5_runtime.py
+++ b/tests/unittest/test_transformers5_runtime.py
@@ -71,42 +71,11 @@ def test_runtime_constructs_caches_and_closes_pipeline(monkeypatch: pytest.Monke
assert calls[0]["backend_config"].device_type == "cuda"
assert calls[0]["backend_config"].cache_max_entry_count == 0.2
assert calls[0]["log_level"] == "ERROR"
- handle = client._mineru_runtime_handles["lmdeploy_engine"]
- runtime._shutdown_runtime_handle(handle)
- assert handle.closed == 1
-
-
-def test_cancelled_mlx_guard_does_not_leave_a_background_lock_owner() -> None:
- """等待模型锁时取消请求,不得有后台线程随后取得锁并永远阻塞后续请求。"""
- import asyncio
- import threading
-
- from mineru.model.vlm.runtime import aio_predictor_execution_guard
-
- lock = threading.Lock()
- predictor = SimpleNamespace(_mineru_execution_lock=lock)
-
- async def guarded() -> None:
- """模拟进入共享 MLX 模型的推理范围。"""
- async with aio_predictor_execution_guard(predictor):
- await asyncio.sleep(0)
-
- async def run() -> None:
- """先占用锁,取消等待者,再验证下一请求仍可正常进入。"""
- lock.acquire()
- task = asyncio.create_task(guarded())
- await asyncio.sleep(0.03)
- task.cancel()
- try:
- await task
- except asyncio.CancelledError:
- pass
- finally:
- lock.release()
- await asyncio.wait_for(guarded(), timeout=1)
- assert not lock.locked()
-
- asyncio.run(run())
+ engine = client.predictor.lmdeploy_engine
+ singleton.shutdown()
+ assert engine.closed == 1
+ assert client.is_closed
+ assert not singleton._models
def test_cancelled_model_initialization_waits_before_cleanup(monkeypatch: pytest.MonkeyPatch) -> None:
diff --git a/tests/unittest/test_vlm_config.py b/tests/unittest/test_vlm_config.py
index 9e988141..cae39514 100644
--- a/tests/unittest/test_vlm_config.py
+++ b/tests/unittest/test_vlm_config.py
@@ -133,7 +133,7 @@ def test_vlm_options_forward_including_empty_values(monkeypatch: pytest.MonkeyPa
def test_api_cli_merges_vlm_without_mutating_global(monkeypatch: pytest.MonkeyPatch, url: str) -> None:
"""底层 CLI 按字段覆盖全局设置,并保持 API Key、全局配置和环境变量隔离。"""
settings = VlmConfig(server_url="http://global.test", api_key="global-key", model="global-model", max_concurrency=7)
- monkeypatch.setattr(api_server.mineru_config.model, "vlm", settings)
+ monkeypatch.setattr(api_server.get_config().model, "vlm", settings)
seen: dict[str, Any] = {}
def capture_app(**kwargs: Any) -> object:
@@ -159,7 +159,7 @@ def test_vlm_app_and_parser_snapshot_configuration(tmp_path: Path, monkeypatch:
"""应用及解析器捕获独立副本,显式空配置完整覆盖全局远程设置。"""
monkeypatch.setattr(api_server, "ensure_tier_runtime_dependencies", lambda tier, **kwargs: None)
settings = VlmConfig(server_url="http://remote.test", model="alias")
- monkeypatch.setattr(api_server.mineru_config.model, "vlm", settings)
+ monkeypatch.setattr(api_server.get_config().model, "vlm", settings)
app = api_server.create_app(upload_dir=str(tmp_path))
parser = MinerUParser()
local_parser = MinerUParser(vlm_config=VlmConfig())
diff --git a/tests/unittest/test_vlm_remote.py b/tests/unittest/test_vlm_remote.py
index f8d29b1c..6e0c95c8 100644
--- a/tests/unittest/test_vlm_remote.py
+++ b/tests/unittest/test_vlm_remote.py
@@ -18,7 +18,7 @@ from PIL import Image
from reportlab.pdfgen import canvas
from typer.testing import CliRunner
-from mineru.config import VlmConfig, config
+from mineru.config import VlmConfig, get_config
from mineru.model.vlm.client import get_vlm_predictor
from mineru.parser import api_server, parse, parse_async
@@ -44,9 +44,9 @@ def openai_server(monkeypatch: pytest.MonkeyPatch) -> Iterator[_OpenAIServer]:
monkeypatch.delenv("MINERU_VL_API_KEY", raising=False)
monkeypatch.delenv("MINERU_VL_MODEL_NAME", raising=False)
- monkeypatch.setattr(config.model, "vlm", VlmConfig())
- monkeypatch.setattr(config.llm_aided.features, "title_leveling", False)
- monkeypatch.setattr(config.llm_aided.features, "cross_page_table_cell_merge", False)
+ monkeypatch.setattr(get_config().model, "vlm", VlmConfig())
+ monkeypatch.setattr(get_config().llm_aided.features, "title_leveling", False)
+ monkeypatch.setattr(get_config().llm_aided.features, "cross_page_table_cell_merge", False)
monkeypatch.setattr(ModelSingleton, "_models", {})
state = _OpenAIServer()
@@ -289,7 +289,7 @@ def test_python_parse_uses_global_remote_vlm(
image_input: bool,
) -> None:
"""同步及异步入口读取全局 VLM 配置,PDF 和图片转 PDF 均产生真实 Middle JSON。"""
- monkeypatch.setattr(config.model, "vlm", _settings(openai_server))
+ monkeypatch.setattr(get_config().model, "vlm", _settings(openai_server))
source = _pdf_input(tmp_path)
if image_input:
source = tmp_path / "input.png"
@@ -308,7 +308,7 @@ def test_kit_parse_uses_global_remote_vlm(
"""实际执行 mineru-kit parse 命令,验证全局 VLM 配置进入解析并生成 Markdown 文件。"""
from mineru.kit.main import app
- monkeypatch.setattr(config.model, "vlm", _settings(openai_server))
+ monkeypatch.setattr(get_config().model, "vlm", _settings(openai_server))
output = tmp_path / "result.md"
result = CliRunner().invoke(app, ["parse", str(_pdf_input(tmp_path)), "-o", str(output), "--ocr-mode", "ocr"])
assert result.exit_code == 0, result.output
@@ -396,7 +396,7 @@ def test_remote_parse_inference_progress(
from mineru.model.vlm import runtime
monkeypatch.setattr(runtime, "MinerUClient", partial(MinerUClient, use_tqdm=enabled))
- monkeypatch.setattr(config.model, "vlm", _settings(openai_server))
+ monkeypatch.setattr(get_config().model, "vlm", _settings(openai_server))
source = _pdf_input(tmp_path)
result = (
asyncio.run(parse_async(source, tier=tier, ocr_mode="ocr")) if async_mode else parse(source, tier=tier, ocr_mode="ocr")
@@ -458,7 +458,7 @@ def test_standard_progress_label_survives_pdf_mode_routing(
"""TXT/OCR 及 auto 的两个分类分支均保留同名推理进度,并走真实 PDF 和 HTTP 流程。"""
from mineru.backend.analysis.pdf import pipeline
- monkeypatch.setattr(config.model, "vlm", _settings(openai_server))
+ monkeypatch.setattr(get_config().model, "vlm", _settings(openai_server))
monkeypatch.setattr(pipeline.PDFDocument, "classify", lambda self: resolved_mode)
context = hybrid_runtime.HybridLocalModelContextSingleton().get_model()
# 公式块在 TXT 下仍需 VLM 抽取,避免测试被原生正文跳过逻辑短路。
diff --git a/tests/unittest/test_webgpu_architecture.py b/tests/unittest/test_webgpu_architecture.py
index 41db8e50..34833ad5 100644
--- a/tests/unittest/test_webgpu_architecture.py
+++ b/tests/unittest/test_webgpu_architecture.py
@@ -101,7 +101,7 @@ def test_blocked_policy_never_downloads_measures_or_saves(
path = tmp_path / "config.yaml"
path.write_text(f"model:\n onnx_device: {requested}\n source: local\n")
before = path.read_bytes()
- monkeypatch.setattr(settings.config.model, "onnx_device", requested)
+ monkeypatch.setattr(settings.get_config().model, "onnx_device", requested)
monkeypatch.setattr(settings, "get_config_file_path", lambda: str(path))
monkeypatch.setattr(settings, "get_config_source", lambda _: "file")
monkeypatch.setattr(onnx_device, "_selections", {})
diff --git a/tools/benchmark_basic_nnpack.py b/tools/benchmark_basic_nnpack.py
index 91478f8e..df71a5c0 100644
--- a/tools/benchmark_basic_nnpack.py
+++ b/tools/benchmark_basic_nnpack.py
@@ -98,14 +98,14 @@ def worker(args: argparse.Namespace) -> None:
import torch
from loguru import logger
- from mineru.config import config
+ from mineru.config import get_config
from mineru.model.runtime.cpu_threads import initialize_torch_threads
from mineru.parser import parse
from mineru.parser.writer import FileBasedDataWriter
- config.model.small_backend = "torch"
- config.llm_aided.features.title_leveling = False
- config.llm_aided.features.cross_page_table_cell_merge = False
+ get_config().model.small_backend = "torch"
+ get_config().llm_aided.features.title_leveling = False
+ get_config().llm_aided.features.cross_page_table_cell_merge = False
initialize_torch_threads()
source_before = det_source_fingerprint()
torch.backends.nnpack.set_flags(args.policy == "on")
@@ -182,7 +182,7 @@ def worker(args: argparse.Namespace) -> None:
"nnpack_enabled": torch._C._get_nnpack_enabled(),
"device": os.environ["MINERU_DEVICE_MODE"],
"table_device": os.environ["MINERU_TABLE_DEVICE"],
- "small_backend": config.model.small_backend,
+ "small_backend": get_config().model.small_backend,
"torch": torch.__version__,
"torch_commit": torch.version.git_version,
"python": platform.python_version(),
diff --git a/tools/benchmark_mps_det_service.py b/tools/benchmark_mps_det_service.py
index 795a8608..1fe64fef 100644
--- a/tools/benchmark_mps_det_service.py
+++ b/tools/benchmark_mps_det_service.py
@@ -288,13 +288,13 @@ def serve(args: argparse.Namespace) -> None:
os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1"
import uvicorn
from loguru import logger
- from mineru.config import config
+ from mineru.config import get_config
from mineru.backend.analysis.pdf import pipeline
from mineru.model.runtime.hybrid import AtomModelSingleton, HybridLocalModelContextSingleton
from mineru.model.runtime.memory import clean_memory
from mineru.parser.api_server import create_app
- config.model.small_backend = "torch"
+ get_config().model.small_backend = "torch"
output = args.output / args.policy
output.mkdir(parents=True, exist_ok=True)
observer = Observer(args.policy, output)
diff --git a/tools/benchmark_torch_cpu_threads.py b/tools/benchmark_torch_cpu_threads.py
index 1f12d6c3..8b3ef89d 100644
--- a/tools/benchmark_torch_cpu_threads.py
+++ b/tools/benchmark_torch_cpu_threads.py
@@ -80,13 +80,13 @@ def main() -> None:
if not args.pdf.is_file():
parser.error("PDF does not exist")
os.environ["MINERU_DEVICE_MODE"] = "cpu"
- from mineru.config import config
+ from mineru.config import get_config
import mineru
import docvortex
from mineru import version
import torch
- config.model.small_backend = "torch"
+ get_config().model.small_backend = "torch"
warmup = parse_once(args.pdf, args.tier)
rounds = [measure_round(args.pdf, args.tier, concurrency) for _ in range(args.rounds) for concurrency in (1, 3)]
report = {
diff --git a/tools/validate_docvortex_boundary.py b/tools/validate_docvortex_boundary.py
index a4650c4e..99e66c3a 100644
--- a/tools/validate_docvortex_boundary.py
+++ b/tools/validate_docvortex_boundary.py
@@ -21,13 +21,13 @@ def capture(args: argparse.Namespace) -> None:
import mineru
from mineru.backend.analyze import doc_analyze
- from mineru.config import config
+ from mineru.config import get_config
from mineru.render import RenderFormat, render
from mineru.utils.logger import configure_global_log_level
configure_global_log_level()
- config.llm_aided.features.title_leveling = False
- config.llm_aided.features.cross_page_table_cell_merge = False
+ get_config().llm_aided.features.title_leveling = False
+ get_config().llm_aided.features.cross_page_table_cell_merge = False
args.output.mkdir(parents=True, exist_ok=True)
manifest = {"imports": {"mineru": mineru.__file__, "docvortex": docvortex.__file__}, "documents": []}
for name in args.documents: