Skip to content

Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.

fix(model): validate cached payloads and migrate legacy completion markers

myhlolib9f257cb74a4
diff --git a/docs/next/cli/mineru-kit-models.md b/docs/next/cli/mineru-kit-models.md
index c9c5fb95..f6984d9d 100644
--- a/docs/next/cli/mineru-kit-models.md
+++ b/docs/next/cli/mineru-kit-models.md
@@ -158,7 +158,12 @@ mineru-kit models verify --small-backend <auto|onnx|torch> --vlm-engine <auto|ll
- repo 位置参数与 `--tier` 互斥
- `--tier` 只接受 `basic` 和 `standard`;按 `--tier` 校验时使用独立后端选项解析出的资源组合
- 不是单纯目录存在性检查,还会检查 registry 中声明的关键路径
-- 第一阶段不做 hash 级完整性校验
+- 整仓和目录资源会读取 `.mineru_complete` 文件清单,检查预期文件是否存在、是否为有效文件,以及大小是否一致;正常校验无需联网,不读取大权重计算 hash
+- 兼容旧版本的零字节完成标记:检查本地非空权重、registry 声明的必需路径以及权重索引引用的分片,通过后可直接离线复用
+- 首次使用旧模型时后台尝试升级标记,包括 `model.source: local`:按缓存中的旧版本查询远端文件清单,存在性和大小全部匹配后原子发布新版清单;只查询元数据,不下载、更新或删除模型文件,也不改写来源配置
+- 无网、源不可达、旧版本无法确定、核验不匹配或目录不可写时保留旧标记,不阻塞当前使用;失败后冷却 5 分钟,下次使用可重试。`models show/verify` 保持只读,不启动迁移
+- 后续实际下载或显式执行 `mineru-kit models download <repo>` 时,由 provider 确认请求范围并生成新版清单;旧标记的兼容检查不具备新版清单的逐文件大小校验能力
+- 文件缺失、被截断或完成清单无效时,校验失败并列出未通过检查的路径
示例:
diff --git a/docs/next/decisions/0025-model-download-local-dir.md b/docs/next/decisions/0025-model-download-local-dir.md
index fb24ecc1..53703a5e 100644
--- a/docs/next/decisions/0025-model-download-local-dir.md
+++ b/docs/next/decisions/0025-model-download-local-dir.md
@@ -272,7 +272,7 @@ repo 同时声明默认下载策略:
- `download_mode="full"`: `download_model_repo(repo)` 下载完整仓库。
- `download_mode="required_paths"`: `download_model_repo(repo)` 只下载 `repo.required_paths()` 对应的文件/目录。
-当前 `PDF-Extract-Kit-1.0` 使用 `required_paths`,避免拉取仓库中未被 runtime 使用的大量历史/可选权重。`MinerU2.5-Pro-2605-1.2B` 保持 `full`,因为 VLM 模型加载通常需要完整 tokenizer/config/weights 文件集;整仓 readiness 由仓库根目录的 `.mineru_complete` 表示,不再维护逐文件 paths 清单。
+当前小模型仓库使用 `required_paths`,避免拉取仓库中未被 runtime 使用的大量历史/可选权重。VLM 仓库保持 `full`,因为模型加载通常需要完整 tokenizer/config/weights 文件集;整仓 readiness 由仓库根目录的 `.mineru_complete` 文件清单验证,不在 registry 中手工维护逐文件 paths 清单。
### 7. 下载 API
@@ -347,17 +347,25 @@ def verify_model_tier(tier: Tier) -> ModelReadyResult:
校验规则:
-- `model.source=local` 时禁止下载,只做 ready 检查。
+- `model.source=local` 时禁止模型文件下载或更新;旧标记自动迁移可以后台查询远端元数据,来源配置仍保持 `local`。
- `mineru-kit models verify <repo>` 检查指定 repo 的完整 required paths。
- `mineru-kit models verify --tier <tier>` 只接受 Basic/Standard,并检查对应 repo 集合。
- managed local parse server 只检查 Basic 或 Standard 启动能力对应的 `REPOS_FOR_TIER`;Standard 模型集同时覆盖 Advanced 请求。
- partial download 只验证本次请求 paths,不要求整仓 ready。
-- 最终文件存在时视为 ready;Hugging Face 和 ModelScope 都先写临时文件,完成后再移动到最终路径。
-- `full` 模式在 ModelRepo 根目录写入空白 `.mineru_complete`,表示整仓 ready;`required_paths` 模式在每个目录型 ModelPath 下分别写入 marker,文件型 ModelPath 仍按最终文件是否存在判断。
-- marker 只由 MinerU 在 provider 下载成功返回且下载后 payload 检查通过后创建。
-- 下载或更新目录前删除旧 marker;下载失败、payload 缺失或进程中断时不创建 marker。
+- 独立文件必须存在且非空;有对应完成清单时还需满足清单中的大小要求。Hugging Face 和 ModelScope 都先写临时文件,完成后再移动到最终路径。
+- `full` 模式在 ModelRepo 根目录写入 `.mineru_complete`;`required_paths` 模式在每个目录型 ModelPath 下分别写入 marker。标记是带 `version: 1` 的 JSON,其中 `files` 保存相对于标记目录的文件路径和字节大小。
+- marker 只由 MinerU 在 provider 下载成功或旧标记在线迁移核验成功后原子发布,要求远程预期文件全部落盘且大小一致。清单由 provider 文件列表生成,不能通过扫描现有本地文件自行认证完整性。
+- 仓库和目录路径共用就绪检查。目录继承祖先 marker 时,只检查清单中属于当前请求范围的文件;清单未覆盖的目录不能凭祖先 marker 通过检查。正常复用无需联网。
+- 下载或更新文件、目录前删除相关目录及祖先 marker;下载失败、文件缺失或进程中断时不创建 marker。局部更新成功后合并原父清单并重新验证全部成员,其他成员仍缺失时不恢复父目录 marker。
+- provider 可能依赖自己的元数据跳过已有文件,下载前先移除与远程预期大小不符的失效文件,保证增量修复生效;完整文件继续由 provider 复用。
- provider 的 `.cache/huggingface`、`._____temp`、`.msc`、`.mdl` 和 `.mv` 等元数据或临时内容不能单独作为模型 payload。
-- 旧版本留下但没有 marker 的目录按未确认完整处理;远端 source 会通过 provider 增量补齐并创建 marker,`local` source 则报告 not ready。
+- 旧版本的零字节 marker 保持兼容:通过本地非空权重、registry 必需路径和权重索引分片检查后,仓库和目录入口均可直接离线复用。`verify_model_repo` 和 `model_path_exists` 仅本地只读检查,不启动迁移。
+- `ModelRepo.ensure()` 和 `ModelPath.ensure()` 成功复用旧模型后,在下载锁外安排守护线程尝试升级。任务使用冻结的目录和仓库信息,按本地 HF 缓存中的 commit 查询旧版本清单;ModelScope 按缓存版本查询并逐项比对 `.msc` 中的文件 commit,允许正常的逐文件多 commit 缓存。请求使用短超时,不调用下载器、不删除或改写模型文件、不修改来源配置。
+- 在线核验成功后,任务通过非阻塞仓库锁重新检查旧标记和文件大小,再原子写入对应范围的新清单。每仓库同时最多一个任务,进程同时最多两个;查询失败、锁竞争、版本不明、文件不匹配或写入失败时保留旧标记,冷却 5 分钟后可在下次使用重试。
+- 无 marker、损坏 JSON 或不支持版本的清单仍按未确认完整处理,不降级为旧标记。有祖先新版清单时,旧子标记不能绕过其严格校验。
+- 实际下载或显式执行 `mineru-kit models download <repo>` 后,由 provider 确认请求范围并发布新版清单。局部更新后的旧父标记只在本地必要文件检查仍通过时恢复为空标记,不伪造整仓新版清单。
+- 旧标记不包含原始文件集合及大小,兼容校验无法识别所有未声明文件的丢失或非空文件的截断;新版清单继续提供逐文件存在性和大小检查。
+- 校验范围为文件缺失、失效软链接和大小变化,不读取大权重计算 hash;`missing_paths` 返回未通过检查的具体文件或完成清单路径。
### 10. 并发安全
diff --git a/mineru/model/_download_manifest.py b/mineru/model/_download_manifest.py
new file mode 100644
index 00000000..eee95cc7
--- /dev/null
+++ b/mineru/model/_download_manifest.py
@@ -0,0 +1,176 @@
+# Copyright (c) Opendatalab. All rights reserved.
+"""模型下载完成清单的本地读写与文件校验。"""
+
+from __future__ import annotations
+
+import json
+import os
+import stat
+import tempfile
+from collections.abc import Mapping
+from pathlib import Path, PurePosixPath, PureWindowsPath
+
+MODEL_COMPLETE_MARKER = ".mineru_complete"
+_MANIFEST_VERSION = 1
+_PROVIDER_METADATA_DIR_NAMES = {".cache", "._____temp"}
+_PROVIDER_METADATA_FILE_NAMES = {MODEL_COMPLETE_MARKER, ".msc", ".mdl", ".mv"}
+_WEIGHT_SUFFIXES = {".safetensors", ".bin", ".pth", ".pt", ".onnx", ".gguf", ".pdiparams"}
+
+
+def is_payload_path(value: str) -> bool:
+ """只接受仓库内规范的相对文件路径,并排除下载器元数据。"""
+ path = PurePosixPath(value)
+ return bool(
+ value
+ and path.parts
+ and "\\" not in value
+ and not any(ord(character) < 32 for character in value)
+ and not path.is_absolute()
+ and not PureWindowsPath(value).drive
+ and path.as_posix() == value
+ and ".." not in path.parts
+ and path.name not in _PROVIDER_METADATA_FILE_NAMES
+ and not any(part in _PROVIDER_METADATA_DIR_NAMES for part in path.parts)
+ and not path.name.startswith(f"{MODEL_COMPLETE_MARKER}.")
+ )
+
+
+def manifest_files_for_path(files: Mapping[str, int], relative_path: str) -> dict[str, int]:
+ """选取请求文件或目录覆盖的清单条目,保持原有相对路径。"""
+ if not relative_path:
+ return dict(files)
+ prefix = f"{relative_path}/"
+ return {name: size for name, size in files.items() if name == relative_path or name.startswith(prefix)}
+
+
+def read_manifest(marker: Path) -> dict[str, int] | None:
+ """读取版本化清单;没有新版清单时由调用者单独判断是否为旧空标记。"""
+ try:
+ value = json.loads(marker.read_text(encoding="utf-8"))
+ except (OSError, ValueError):
+ return None
+ if not isinstance(value, dict) or type(value.get("version")) is not int or value["version"] != _MANIFEST_VERSION:
+ return None
+ files = value.get("files")
+ if not isinstance(files, dict) or not files:
+ return None
+ if any(
+ not isinstance(name, str) or not is_payload_path(name) or type(size) is not int or size < 0
+ for name, size in files.items()
+ ):
+ return None
+ return files
+
+
+def is_legacy_marker(marker: Path) -> bool:
+ """仅将历史下载器写入的零字节常规文件识别为旧完成标记。"""
+ try:
+ info = marker.stat()
+ except OSError:
+ return False
+ return stat.S_ISREG(info.st_mode) and info.st_size == 0
+
+
+def _file_has_content(path: Path) -> bool:
+ """通过文件类型和大小排除空权重、目录和失效软链接。"""
+ try:
+ info = path.stat()
+ except OSError:
+ return False
+ return stat.S_ISREG(info.st_mode) and info.st_size > 0
+
+
+def _legacy_index_missing(index: Path) -> list[Path]:
+ """利用旧模型自带的权重索引检查全部分片,不依赖远端或扫描生成的清单。"""
+ try:
+ value = json.loads(index.read_text(encoding="utf-8"))
+ except (OSError, ValueError):
+ return [index]
+ weight_map = value.get("weight_map") if isinstance(value, dict) else None
+ if not isinstance(weight_map, dict) or not weight_map:
+ return [index]
+ shards = list(weight_map.values())
+ if any(not isinstance(name, str) or not is_payload_path(name) for name in shards):
+ return [index]
+ return [index.parent / name for name in dict.fromkeys(shards) if not _file_has_content(index.parent / name)]
+
+
+def legacy_payload_missing(path: Path) -> list[Path]:
+ """兼容旧标记时检查真实权重和索引分片;README、配置及缓存不能单独证明可用。"""
+ if path.is_file():
+ return [] if _file_has_content(path) else [path]
+ if not path.is_dir():
+ return [path]
+ weights: list[Path] = []
+ missing: list[Path] = []
+ for root, directories, names in os.walk(path):
+ directories[:] = sorted(name for name in directories if name not in _PROVIDER_METADATA_DIR_NAMES)
+ for name in sorted(names):
+ candidate = Path(root) / name
+ if name == MODEL_COMPLETE_MARKER and candidate.parent != path:
+ files = read_manifest(candidate)
+ if files is not None:
+ missing.extend(candidate.parent / item for item in missing_manifest_files(candidate.parent, files))
+ elif not is_legacy_marker(candidate):
+ missing.append(candidate)
+ continue
+ if not is_payload_path(candidate.relative_to(path).as_posix()):
+ continue
+ if candidate.suffix.lower() in _WEIGHT_SUFFIXES:
+ weights.append(candidate)
+ if not _file_has_content(candidate):
+ missing.append(candidate)
+ elif name.endswith((".safetensors.index.json", ".bin.index.json")):
+ missing.extend(_legacy_index_missing(candidate))
+ if not weights and not missing:
+ missing.append(path)
+ return list(dict.fromkeys(missing))
+
+
+def missing_manifest_files(root: Path, files: Mapping[str, int]) -> list[str]:
+ """返回清单中缺失、不是有效文件或大小发生变化的路径。"""
+ missing: list[str] = []
+ for name, size in files.items():
+ try:
+ info = (root / name).stat()
+ except OSError:
+ missing.append(name)
+ continue
+ if not stat.S_ISREG(info.st_mode) or info.st_size != size:
+ missing.append(name)
+ return missing
+
+
+def write_manifest(marker: Path, files: Mapping[str, int]) -> None:
+ """在同一目录原子发布清单,避免中断时留下半写入的完成标记。"""
+ temporary_path: Path | None = None
+ try:
+ with tempfile.NamedTemporaryFile(
+ mode="w",
+ encoding="utf-8",
+ dir=marker.parent,
+ prefix=f"{MODEL_COMPLETE_MARKER}.",
+ suffix=".tmp",
+ delete=False,
+ ) as temporary:
+ temporary_path = Path(temporary.name)
+ json.dump({"version": _MANIFEST_VERSION, "files": dict(files)}, temporary, ensure_ascii=False, sort_keys=True)
+ temporary.write("\n")
+ temporary.flush()
+ os.fsync(temporary.fileno())
+ temporary_path.replace(marker)
+ finally:
+ if temporary_path is not None:
+ temporary_path.unlink(missing_ok=True)
+
+
+__all__ = [
+ "MODEL_COMPLETE_MARKER",
+ "is_legacy_marker",
+ "is_payload_path",
+ "legacy_payload_missing",
+ "manifest_files_for_path",
+ "missing_manifest_files",
+ "read_manifest",
+ "write_manifest",
+]
diff --git a/mineru/model/_manifest_upgrade.py b/mineru/model/_manifest_upgrade.py
new file mode 100644
index 00000000..fe894d48
--- /dev/null
+++ b/mineru/model/_manifest_upgrade.py
@@ -0,0 +1,258 @@
+# Copyright (c) Opendatalab. All rights reserved.
+"""旧完成标记的尽力而为后台迁移,只查询元数据并原子更新标记。"""
+
+from __future__ import annotations
+
+import io
+import os
+import pickle
+import re
+import threading
+from dataclasses import dataclass
+from pathlib import Path
+from time import monotonic
+from urllib.parse import quote
+
+import requests
+from filelock import FileLock
+from huggingface_hub import HfApi
+from loguru import logger
+
+from ._download_manifest import (
+ is_legacy_marker,
+ is_payload_path,
+ legacy_payload_missing,
+ manifest_files_for_path,
+ missing_manifest_files,
+ write_manifest,
+)
+
+_QUERY_CONNECT_TIMEOUT = 2.0
+_QUERY_TIMEOUT = 5.0
+_RETRY_COOLDOWN = 300.0
+_MAX_ACTIVE_UPGRADES = 2
+_MAX_METADATA_BYTES = 8 * 1024 * 1024
+_COMMIT_PATTERN = re.compile(r"[0-9a-f]{40,64}\Z")
+
+
+@dataclass(frozen=True)
+class ManifestUpgrade:
+ """冻结模型目录及仓库信息,后台任务不再读取调用线程的配置上下文。"""
+
+ root: Path
+ lock_path: Path
+ repos: dict[str, str]
+ markers: tuple[Path, ...]
+
+
+@dataclass(frozen=True)
+class _CachedRevision:
+ """记录查询版本及缓存文件版本,兼容 ModelScope 按文件保存不同 commit 的格式。"""
+
+ revision: str
+ files: dict[str, str]
+
+
+class _CacheUnpickler(pickle.Unpickler):
+ """仅兼容 ModelScope 缓存中的基础容器,禁止反序列化任意 Python 对象。"""
+
+ def find_class(self, module: str, name: str) -> object:
+ """拒绝缓存中引用的全局对象和可执行构造器。"""
+ raise pickle.UnpicklingError("Model cache metadata must contain only plain values.")
+
+
+def _cached_revisions(request: ManifestUpgrade) -> dict[str, _CachedRevision]:
+ """读取旧下载版本;ModelScope 多 commit 时必须额外比对每个文件的缓存版本。"""
+ scopes = [marker.parent.relative_to(request.root).as_posix() for marker in request.markers]
+ scopes = ["" if value == "." else value for value in scopes]
+ files: dict[str, dict[str, str]] = {"huggingface": {}, "modelscope": {}}
+ metadata_root = request.root / ".cache/huggingface/download"
+ if "huggingface" in request.repos and metadata_root.is_dir():
+ for metadata in metadata_root.rglob("*.metadata"):
+ name = metadata.relative_to(metadata_root).as_posix().removesuffix(".metadata")
+ if not is_payload_path(name) or not any(manifest_files_for_path({name: 0}, scope) for scope in scopes):
+ continue
+ try:
+ with metadata.open(encoding="utf-8") as stream:
+ revision = stream.readline(256).strip()
+ except (OSError, ValueError):
+ continue
+ if _COMMIT_PATTERN.fullmatch(revision):
+ files["huggingface"][name] = revision
+ cache = request.root / ".msc"
+ if "modelscope" in request.repos and cache.is_file():
+ try:
+ with cache.open("rb") as stream:
+ content = stream.read(_MAX_METADATA_BYTES + 1)
+ records = _CacheUnpickler(io.BytesIO(content)).load() if len(content) <= _MAX_METADATA_BYTES else None
+ if isinstance(records, list):
+ for record in records:
+ if not isinstance(record, dict):
+ continue
+ name, revision = record.get("Path"), record.get("Revision")
+ if (
+ isinstance(name, str)
+ and is_payload_path(name)
+ and isinstance(revision, str)
+ and _COMMIT_PATTERN.fullmatch(revision)
+ and any(manifest_files_for_path({name: 0}, scope) for scope in scopes)
+ ):
+ files["modelscope"][name] = revision
+ except (OSError, ValueError, EOFError, pickle.UnpicklingError):
+ pass
+ revisions: dict[str, _CachedRevision] = {}
+ hf_versions = set(files["huggingface"].values())
+ if len(hf_versions) == 1:
+ revisions["huggingface"] = _CachedRevision(next(iter(hf_versions)), files["huggingface"])
+ ms_versions = set(files["modelscope"].values())
+ if ms_versions:
+ revision = next(iter(ms_versions)) if len(ms_versions) == 1 else "master"
+ if len(ms_versions) > 1:
+ try:
+ with (request.root / ".mv").open(encoding="utf-8") as stream:
+ cached_version = stream.readline(1024)
+ match = re.match(r"Revision:([^,\r\n]+)", cached_version)
+ if match:
+ revision = match.group(1)
+ except (OSError, ValueError):
+ pass
+ revisions["modelscope"] = _CachedRevision(revision, files["modelscope"])
+ return revisions
+
+
+def _remote_manifest(
+ source: str, repo_id: str, revision: str, *, cached_file_revisions: dict[str, str] | None = None
+) -> dict[str, int]:
+ """用短超时查询指定旧版本的远程清单,不创建目录、不下载或删除模型文件。"""
+ if source == "huggingface":
+ info = HfApi().model_info(repo_id, revision=revision, files_metadata=True, timeout=_QUERY_TIMEOUT)
+ if info.sha != revision:
+ return {}
+ files = {item.rfilename: item.size for item in info.siblings or ()}
+ elif source == "modelscope":
+ # 迁移只访问公开仓库元数据,避免导入 ModelScope 推理 SDK 或触发其下载流程。
+ url = f"https://modelscope.cn/api/v1/models/{quote(repo_id, safe='/')}/repo/files"
+ with requests.get(
+ url,
+ params={"Revision": revision, "Recursive": "True"},
+ timeout=(_QUERY_CONNECT_TIMEOUT, _QUERY_TIMEOUT),
+ ) as response:
+ response.raise_for_status()
+ value = response.json()
+ if not isinstance(value, dict) or value.get("Code") != 200:
+ return {}
+ data = value.get("Data")
+ entries = data.get("Files") if isinstance(data, dict) else None
+ # 服务端可能截断大型递归列表;无法证明列表完整时不执行自动迁移。
+ if not isinstance(entries, list) or len(entries) >= 1000:
+ return {}
+ if cached_file_revisions:
+ remote_revisions = {
+ item["Path"]: item.get("Revision") for item in entries if isinstance(item, dict) and item.get("Type") != "tree"
+ }
+ if any(remote_revisions.get(name) != value for name, value in cached_file_revisions.items()):
+ return {}
+ files = {item["Path"]: item["Size"] for item in entries if isinstance(item, dict) and item.get("Type") != "tree"}
+ else:
+ return {}
+ payload: dict[str, int] = {}
+ for name, size in files.items():
+ if not isinstance(name, str) or not is_payload_path(name):
+ continue
+ if type(size) is not int or size < 0:
+ return {}
+ payload[name] = size
+ return payload
+
+
+def _upgrade_markers(request: ManifestUpgrade) -> None:
+ """联网取得历史预期集合,核验成功后在下载锁内再次复核并原子替换旧标记。"""
+ for source, cached in _cached_revisions(request).items():
+ repo_id = request.repos.get(source)
+ if repo_id is None:
+ continue
+ try:
+ if source == "modelscope":
+ files = _remote_manifest(source, repo_id, cached.revision, cached_file_revisions=cached.files)
+ else:
+ files = _remote_manifest(source, repo_id, cached.revision)
+ except Exception as exc:
+ logger.debug(f"Model manifest upgrade query failed for {repo_id}: {exc}")
+ continue
+ if not files:
+ continue
+ for marker in request.markers:
+ scope = marker.parent.relative_to(request.root).as_posix() if marker.parent != request.root else ""
+ selected = manifest_files_for_path(files, scope)
+ prefix = f"{scope}/" if scope else ""
+ manifest = {name.removeprefix(prefix): size for name, size in selected.items()}
+ if not manifest or missing_manifest_files(marker.parent, manifest):
+ continue
+ # 网络查询不占用下载锁;锁竞争时让位给正常使用,后续冷却结束后再尝试。
+ with FileLock(str(request.lock_path), timeout=0):
+ if not is_legacy_marker(marker):
+ continue
+ if legacy_payload_missing(marker.parent) or missing_manifest_files(marker.parent, manifest):
+ continue
+ write_manifest(marker, manifest)
+
+
+class _UpgradeScheduler:
+ """限制并发和失败重试频率,使用守护线程保证离线查询不拖慢退出。"""
+
+ def __init__(self) -> None:
+ """只准备进程内同步状态,创建实例不会联网或启动线程。"""
+ self.lock = threading.Lock()
+ self.active: dict[Path, threading.Thread] = {}
+ self.retry_after: dict[Path, float] = {}
+ self.fork_handler_registered = False
+
+ def _after_fork(self) -> None:
+ """子进程重建锁和任务状态,避免继承已消失的后台线程。"""
+ self.lock = threading.Lock()
+ self.active = {}
+ self.retry_after = {}
+
+ def schedule(self, request: ManifestUpgrade) -> threading.Thread | None:
+ """每仓库同时启动一个任务,失败后冷却,所有启动错误均留给旧协议正常使用。"""
+ with self.lock:
+ if request.root in self.active or monotonic() < self.retry_after.get(request.root, 0):
+ return None
+ if len(self.active) >= _MAX_ACTIVE_UPGRADES:
+ return None
+ if not self.fork_handler_registered and hasattr(os, "register_at_fork"):
+ os.register_at_fork(after_in_child=self._after_fork)
+ self.fork_handler_registered = True
+ worker = threading.Thread(target=self._run, args=(request,), name="mineru-manifest-upgrade", daemon=True)
+ self.active[request.root] = worker
+ try:
+ worker.start()
+ except Exception:
+ self.active.pop(request.root, None)
+ self.retry_after[request.root] = monotonic() + _RETRY_COOLDOWN
+ return None
+ return worker
+
+ def _run(self, request: ManifestUpgrade) -> None:
+ """后台迁移异常只记录调试信息,始终释放并发配额且保留失败冷却。"""
+ try:
+ _upgrade_markers(request)
+ except Exception as exc:
+ logger.debug(f"Model manifest upgrade skipped for {request.root.name}: {exc}")
+ finally:
+ with self.lock:
+ self.active.pop(request.root, None)
+ self.retry_after[request.root] = monotonic() + _RETRY_COOLDOWN
+
+
+_SCHEDULER = _UpgradeScheduler()
+
+
+def schedule_manifest_upgrade(request: ManifestUpgrade) -> threading.Thread | None:
+ """仅在存在旧标记时安排迁移,不使用或改写 model.source,包括 local 配置。"""
+ if not any(is_legacy_marker(marker) for marker in request.markers):
+ return None
+ return _SCHEDULER.schedule(request)
+
+
+__all__ = ["ManifestUpgrade", "schedule_manifest_upgrade"]
diff --git a/mineru/model/download.py b/mineru/model/download.py
index 18840803..e96f739b 100644
--- a/mineru/model/download.py
+++ b/mineru/model/download.py
@@ -1,7 +1,6 @@
# Copyright (c) Opendatalab. All rights reserved.
from __future__ import annotations
-import os
import re
from collections.abc import Sequence
from dataclasses import dataclass, field
@@ -16,12 +15,23 @@ from huggingface_hub.utils import filter_repo_objects
from loguru import logger
from ..config import get_config, get_config_source, update_config_file
+from ._download_manifest import (
+ MODEL_COMPLETE_MARKER,
+ is_legacy_marker,
+ is_payload_path,
+ legacy_payload_missing,
+ manifest_files_for_path,
+ missing_manifest_files,
+ read_manifest,
+ write_manifest,
+)
+from ._manifest_upgrade import ManifestUpgrade, schedule_manifest_upgrade
if TYPE_CHECKING:
from ..config import ModelSource
DownloadMode = Literal["full", "required_paths"]
-MODEL_COMPLETE_MARKER = ".mineru_complete"
+_RemovedMarker = dict[str, int] | Literal["legacy"] | None
def _join_model_path(*parts: str) -> str:
@@ -56,8 +66,9 @@ class ModelPath:
return self.repo.local_dir() / self.relative_path
def ensure(self, *, source: "ModelSource | None" = None) -> Path:
- """确保当前模型路径已下载并返回本地位置。"""
+ """确保当前模型路径可用,并在返回前安排旧标记的后台迁移。"""
download_model_files(self.repo, [self], source=source, reuse_ready=True)
+ _schedule_legacy_upgrade(self.repo, [self])
return self.local_path()
@@ -106,8 +117,10 @@ class ModelRepo:
return Path(get_config().model.base_dir).expanduser() / ".locks" / f"{safe_name}.lock"
def ensure(self, *, source: "ModelSource | None" = None) -> Path:
- """确保完整仓库可用并返回本地根路径。"""
- return download_model_repo(self, source=source, reuse_ready=True)
+ """确保完整仓库可用并返回根目录,旧标记的联网迁移不阻塞当前使用。"""
+ root = download_model_repo(self, source=source, reuse_ready=True)
+ _schedule_legacy_upgrade(self, _repo_completion_paths(self))
+ return root
MODEL_SOURCE_ENV_VAR = "MINERU_MODEL_SOURCE"
@@ -118,8 +131,6 @@ _REMOTE_MODEL_SOURCES = ("huggingface", "modelscope")
DOWNLOAD_MODEL_SOURCES = ("auto", *_REMOTE_MODEL_SOURCES)
ResolvedRemoteModelSource = Literal["huggingface", "modelscope"]
ResolvedModelSource = Literal["huggingface", "modelscope", "local"]
-_PROVIDER_METADATA_DIR_NAMES = {".cache", "._____temp"}
-_PROVIDER_METADATA_FILE_NAMES = {MODEL_COMPLETE_MARKER, ".msc", ".mdl", ".mv"}
def ms_snapshot_download(repo_id: str, *, local_dir: str, allow_patterns: list[str] | None = None) -> str:
@@ -129,6 +140,15 @@ def ms_snapshot_download(repo_id: str, *, local_dir: str, allow_patterns: list[s
return snapshot_download(repo_id, local_dir=local_dir, allow_patterns=allow_patterns)
+def ms_model_files(repo_id: str) -> dict[str, int]:
+ """按下载器默认版本读取 ModelScope 预期文件,仅在使用该来源时加载 SDK。"""
+ from modelscope.hub.api import HubApi
+ from modelscope.utils.constant import DEFAULT_MODEL_REVISION
+
+ files = HubApi().get_model_files(repo_id, revision=DEFAULT_MODEL_REVISION, recursive=True)
+ return {item["Path"]: item["Size"] for item in files if item.get("Type") != "tree"}
+
+
@dataclass(frozen=True)
class ModelReadyResult:
ready: bool
@@ -221,30 +241,47 @@ def _source_repo_id(repo: ModelRepo, model_source: str) -> str:
return repo.repos[model_source]
-def _snapshot_download(model_source: ResolvedRemoteModelSource, repo: ModelRepo, patterns: list[str] | None) -> str:
- """下载已配置来源的快照,并验证远程清单中的文件均已落盘。"""
+def _snapshot_download(model_source: ResolvedRemoteModelSource, repo: ModelRepo, patterns: list[str] | None) -> dict[str, int]:
+ """按远程预期清单下载并校验快照,返回可用于本地复用检查的文件及大小。"""
repo_id = _source_repo_id(repo, model_source)
local_dir = repo.local_dir()
local_dir.parent.mkdir(parents=True, exist_ok=True)
- kwargs = {"local_dir": str(local_dir)}
- if patterns is not None:
- kwargs["allow_patterns"] = patterns
if model_source == "huggingface":
- # Hugging Face may return a non-empty local_dir when the remote repo is unavailable, so verify the expected
- # remote manifest before treating the snapshot as complete.
- expected_files = list(filter_repo_objects(HfApi().list_repo_files(repo_id), allow_patterns=patterns))
- if not expected_files:
- raise FileNotFoundError(f"Hugging Face snapshot for {repo.name} has no files matching the requested paths.")
- root = hf_snapshot_download(repo_id, **kwargs)
- missing_files = [path for path in expected_files if not (local_dir / path).is_file()]
- if missing_files:
- missing = ", ".join(missing_files)
- raise FileNotFoundError(f"Hugging Face snapshot for {repo.name} is incomplete; missing: {missing}")
- return root
- return ms_snapshot_download(repo_id, **kwargs)
-
-
-def _path_patterns(paths: list[str]) -> list[str]:
+ info = HfApi().model_info(repo_id, files_metadata=True)
+ files = {item.rfilename: item.size for item in info.siblings or ()}
+ else:
+ files = ms_model_files(repo_id)
+ expected_files: dict[str, int] = {}
+ for name in filter_repo_objects(files, allow_patterns=patterns):
+ if not is_payload_path(name):
+ continue
+ size = files[name]
+ if type(size) is not int or size < 0:
+ raise ValueError(f"Missing or invalid model file size from {model_source}: {name!r}")
+ expected_files[name] = size
+ if not expected_files:
+ raise FileNotFoundError(f"{model_source} snapshot for {repo.name} has no files matching the requested paths.")
+ # 下载器可能凭借自己的缓存元数据跳过已有文件;先移除大小不符的失效文件,保证增量修复实际执行。
+ for name in missing_manifest_files(local_dir, expected_files):
+ target = local_dir / name
+ if target.is_file() or target.is_symlink():
+ target.unlink()
+ requested_files = list(expected_files)
+ if model_source == "huggingface":
+ hf_snapshot_download(repo_id, local_dir=str(local_dir), allow_patterns=requested_files, revision=info.sha)
+ else:
+ ms_snapshot_download(repo_id, local_dir=str(local_dir), allow_patterns=requested_files)
+ missing_files = missing_manifest_files(local_dir, expected_files)
+ if missing_files:
+ missing = ", ".join(missing_files)
+ raise FileNotFoundError(f"{model_source} snapshot for {repo.name} is incomplete; missing or invalid: {missing}")
+ return expected_files
+
+
+def _path_patterns(paths: list[str]) -> list[str] | None:
+ """将请求路径展开为下载模式;仓库根路径表示完整快照。"""
+ if "" in paths:
+ return None
patterns: list[str] = []
for path in paths:
normalized = path.strip("/")
@@ -267,31 +304,82 @@ def _relative_paths(repo: ModelRepo, paths: Sequence[str | ModelPath]) -> list[s
def _verify_paths(repo: ModelRepo, paths: Sequence[str | ModelPath]) -> ModelReadyResult:
+ """统一检查文件和目录,并保留清单给出的具体失效文件路径。"""
model_paths = [path if isinstance(path, ModelPath) else repo.path(path) for path in paths]
- missing = [path.relative_path for path in model_paths if not model_path_exists(path)]
+ missing = list(dict.fromkeys(name for path in model_paths for name in _model_path_missing(path)))
return ModelReadyResult(ready=not missing, root=repo.local_dir(), missing_paths=missing)
-def _directory_has_payload(path: Path) -> bool:
- for _root, dir_names, file_names in os.walk(path):
- dir_names[:] = [name for name in dir_names if name not in _PROVIDER_METADATA_DIR_NAMES]
- if any(name not in _PROVIDER_METADATA_FILE_NAMES for name in file_names):
- return True
- return False
-
-
-def _payload_path_exists(path: ModelPath) -> bool:
+def _model_path_missing(path: ModelPath) -> list[str]:
+ """验证请求范围内的完成清单,目录继承祖先清单时也必须包含自身文件。"""
local_path = path.local_path()
- if local_path.is_file():
- return True
- if not local_path.is_dir():
- return False
- return _directory_has_payload(local_path)
+ repo_root = path.repo.local_dir()
+ try:
+ local_path.relative_to(repo_root)
+ except ValueError:
+ return [path.relative_path]
+ is_file = local_path.is_file()
+ if local_path == repo_root and is_file:
+ return [MODEL_COMPLETE_MARKER]
+ directory = local_path.parent if is_file else local_path
+ legacy_marker_found = False
+ for parent in (directory, *directory.parents):
+ marker = parent / MODEL_COMPLETE_MARKER
+ if marker.is_file():
+ files = read_manifest(marker)
+ marker_name = marker.relative_to(repo_root).as_posix()
+ if files is None:
+ if is_legacy_marker(marker):
+ # 先查找祖先的新版清单,避免旧子标记绕过其中记录的严格文件检查。
+ legacy_marker_found = True
+ elif not is_file:
+ # 损坏或未知版本的清单不能降级为旧标记;独立文件继续遵守文件路径契约。
+ return [marker_name if local_path.is_dir() or local_path == repo_root else path.relative_path]
+ else:
+ selected = manifest_files_for_path(
+ files, local_path.relative_to(parent).as_posix() if parent != local_path else ""
+ )
+ if selected:
+ return [
+ (parent / name).relative_to(repo_root).as_posix() for name in missing_manifest_files(parent, selected)
+ ]
+ if not is_file:
+ return [path.relative_path or marker_name]
+ if parent == repo_root:
+ break
+ if legacy_marker_found:
+ return _legacy_model_path_missing(path)
+ if is_file:
+ try:
+ if local_path.stat().st_size > 0:
+ return []
+ except OSError:
+ pass
+ return [path.relative_path]
+ if local_path == repo_root:
+ return [MODEL_COMPLETE_MARKER]
+ return [path.relative_path]
+
+
+def _legacy_model_path_missing(path: ModelPath) -> list[str]:
+ """旧标记按本地必要路径和真实权重只读检查,迁移由 ensure 返回前另行安排。"""
+ root = path.repo.local_dir()
+ local_path = path.local_path()
+ missing = legacy_payload_missing(local_path)
+ if local_path.is_dir():
+ for required in path.repo.required_paths():
+ if required.local_path() != local_path and required.local_path().is_relative_to(local_path):
+ missing.extend(legacy_payload_missing(required.local_path()))
+ return list(dict.fromkeys(item.relative_to(root).as_posix() if item != root else "<payload>" for item in missing))
-def _verify_downloaded_paths(repo: ModelRepo, paths: Sequence[str | ModelPath]) -> ModelReadyResult:
+def _verify_downloaded_paths(repo: ModelRepo, paths: Sequence[str | ModelPath], files: dict[str, int]) -> ModelReadyResult:
+ """下载后要求每个请求路径均被远程清单覆盖,已有本地残留不能替代缺失文件。"""
model_paths = [path if isinstance(path, ModelPath) else repo.path(path) for path in paths]
- missing = [path.relative_path for path in model_paths if not _payload_path_exists(path)]
+ missing = missing_manifest_files(repo.local_dir(), files)
+ missing.extend(
+ path.relative_path or "<payload>" for path in model_paths if not manifest_files_for_path(files, path.relative_path)
+ )
return ModelReadyResult(ready=not missing, root=repo.local_dir(), missing_paths=missing)
@@ -301,38 +389,113 @@ def _repo_completion_paths(repo: ModelRepo) -> list[ModelPath]:
return list(repo.required_paths())
-def _remove_directory_markers(paths: Sequence[ModelPath]) -> list[Path]:
- removed_markers: list[Path] = []
+def _schedule_legacy_upgrade(repo: ModelRepo, paths: Sequence[ModelPath]) -> None:
+ """使用后捕获旧标记范围,在下载锁外安排元数据迁移,不影响当前模型加载。"""
+ try:
+ root = repo.local_dir().absolute()
+ markers: set[Path] = set()
+ for path in (*paths, *repo.required_paths()):
+ local_path = path.local_path().absolute()
+ directory = local_path if local_path.is_dir() else local_path.parent
+ if not directory.is_relative_to(root):
+ continue
+ for parent in (directory, *directory.parents):
+ marker = parent / MODEL_COMPLETE_MARKER
+ if is_legacy_marker(marker):
+ markers.add(marker)
+ if parent == root:
+ break
+ # 有父标记时按完整父范围迁移,子目录请求不会被误记成整仓完整清单。
+ selected = tuple(
+ marker
+ for marker in sorted(markers)
+ if not any(marker != other and marker.is_relative_to(other.parent) for other in markers)
+ )
+ if selected:
+ schedule_manifest_upgrade(
+ ManifestUpgrade(root=root, lock_path=repo.lock_path().absolute(), repos=dict(repo.repos), markers=selected)
+ )
+ except Exception as exc:
+ logger.debug(f"Could not schedule legacy model manifest upgrade for {repo.name}: {exc}")
+
+
+def _remove_directory_markers(paths: Sequence[ModelPath]) -> dict[Path, _RemovedMarker]:
+ """下载前撤销请求范围及祖先标记,并保存可在成功后重新验证的原清单。"""
+ removed_markers: dict[Path, _RemovedMarker] = {}
for path in paths:
local_path = path.local_path()
- if not local_path.is_dir():
- continue
repo_root = path.repo.local_dir()
- for directory in (local_path, *local_path.parents):
+ directory = local_path if local_path.is_dir() or local_path == repo_root else local_path.parent
+ for directory in (directory, *directory.parents):
marker = directory / MODEL_COMPLETE_MARKER
if marker.is_file():
+ removed_markers[marker] = "legacy" if is_legacy_marker(marker) else read_manifest(marker)
marker.unlink()
- removed_markers.append(marker)
if directory == repo_root:
break
return removed_markers
-def _mark_directories_complete(paths: Sequence[ModelPath]) -> None:
+def _mark_directories_complete(paths: Sequence[ModelPath], files: dict[str, int]) -> None:
+ """仅用本次下载确认的文件清单原子发布请求目录的完成标记。"""
+ manifests: dict[Path, dict[str, int]] = {}
for path in paths:
local_path = path.local_path()
if local_path.is_dir():
- (local_path / MODEL_COMPLETE_MARKER).touch(exist_ok=True)
-
-
-def _restore_directory_markers(markers: Sequence[Path]) -> None:
- for marker in markers:
- marker.touch(exist_ok=True)
+ selected = manifest_files_for_path(files, path.relative_path)
+ prefix = f"{path.relative_path}/" if path.relative_path else ""
+ manifests[local_path] = {name.removeprefix(prefix): size for name, size in selected.items()}
+ # 整个目录已被重新确认时,其文件祖先中的旧子清单应由新清单接管,避免离线加载仍命中旧标记。
+ removed: set[Path] = set()
+ for root, manifest in manifests.items():
+ for name in manifest:
+ directory = (root / name).parent
+ while directory != root:
+ marker = directory / MODEL_COMPLETE_MARKER
+ if marker not in removed and marker.is_file():
+ marker.unlink()
+ removed.add(marker)
+ directory = directory.parent
+ for root, manifest in manifests.items():
+ write_manifest(root / MODEL_COMPLETE_MARKER, manifest)
+
+
+def _restore_directory_markers(
+ repo: ModelRepo,
+ markers: dict[Path, _RemovedMarker],
+ paths: Sequence[ModelPath],
+ files: dict[str, int],
+) -> None:
+ """合并局部更新后重新验证父清单,其他成员缺失时不恢复父目录就绪状态。"""
+ for marker, previous in markers.items():
+ if marker.is_file() or previous is None:
+ continue
+ root = marker.parent
+ root_name = root.relative_to(repo.local_dir()).as_posix() if root != repo.local_dir() else ""
+ if previous == "legacy":
+ if not _legacy_model_path_missing(repo.path(root_name)):
+ marker.touch(exist_ok=True)
+ continue
+ prefix = f"{root_name}/" if root_name else ""
+ updated = dict(previous)
+ for path in paths:
+ if not path.local_path().is_relative_to(root):
+ continue
+ relative_path = path.local_path().relative_to(root).as_posix() if path.local_path() != root else ""
+ for name in manifest_files_for_path(updated, relative_path):
+ del updated[name]
+ updated.update({name.removeprefix(prefix): size for name, size in manifest_files_for_path(files, root_name).items()})
+ if updated and not missing_manifest_files(root, updated):
+ write_manifest(marker, updated)
def _raise_not_ready(repo: ModelRepo, result: ModelReadyResult) -> None:
+ """离线校验失败时报告真实路径,并提示修复文件或完成标记。"""
missing = ", ".join(result.missing_paths)
- raise FileNotFoundError(f"Model repo {repo.name} is not ready under {result.root}; missing: {missing}")
+ raise FileNotFoundError(
+ f"Model repo {repo.name} is not ready under {result.root}; missing or invalid: {missing}. "
+ f"Run: mineru-kit models download {repo.name} to repair files or completion markers."
+ )
def download_model_repo(
@@ -342,6 +505,7 @@ def download_model_repo(
local_as_auto: bool = False,
reuse_ready: bool = False,
) -> Path:
+ """在仓库锁内验证或补齐请求资源,成功后发布可离线验证的完成清单。"""
completion_paths = _repo_completion_paths(repo)
relative_paths = [path.relative_path for path in completion_paths] if repo.download_mode == "required_paths" else None
lock_path = repo.lock_path()
@@ -361,12 +525,12 @@ def download_model_repo(
_source_repo_id(repo, resolved_source)
removed_markers = _remove_directory_markers(completion_paths)
- _snapshot_download(resolved_source, repo, None if relative_paths is None else _path_patterns(relative_paths))
- downloaded_result = _verify_downloaded_paths(repo, completion_paths)
+ files = _snapshot_download(resolved_source, repo, None if relative_paths is None else _path_patterns(relative_paths))
+ downloaded_result = _verify_downloaded_paths(repo, completion_paths, files)
if not downloaded_result.ready:
_raise_not_ready(repo, downloaded_result)
- _mark_directories_complete(completion_paths)
- _restore_directory_markers(removed_markers)
+ _mark_directories_complete(completion_paths, files)
+ _restore_directory_markers(repo, removed_markers, completion_paths, files)
return downloaded_result.root
@@ -378,6 +542,7 @@ def download_model_files(
local_as_auto: bool = False,
reuse_ready: bool = False,
) -> Path:
+ """按请求范围增量下载,并与整仓入口使用相同的就绪判定和标记生命周期。"""
if not paths:
return repo.local_dir()
@@ -400,45 +565,23 @@ def download_model_files(
_source_repo_id(repo, resolved_source)
removed_markers = _remove_directory_markers(model_paths)
- _snapshot_download(resolved_source, repo, _path_patterns(relative_paths))
- downloaded_result = _verify_downloaded_paths(repo, model_paths)
+ files = _snapshot_download(resolved_source, repo, _path_patterns(relative_paths))
+ downloaded_result = _verify_downloaded_paths(repo, model_paths, files)
if not downloaded_result.ready:
_raise_not_ready(repo, downloaded_result)
- _mark_directories_complete(model_paths)
- _restore_directory_markers(removed_markers)
+ _mark_directories_complete(model_paths, files)
+ _restore_directory_markers(repo, removed_markers, model_paths, files)
return downloaded_result.root
def verify_model_repo(repo: ModelRepo) -> ModelReadyResult:
- if repo.download_mode == "full":
- marker = repo.local_dir() / MODEL_COMPLETE_MARKER
- ready = marker.is_file()
- return ModelReadyResult(
- ready=ready,
- root=repo.local_dir(),
- missing_paths=[] if ready else [MODEL_COMPLETE_MARKER],
- )
- return _verify_paths(repo, list(repo.required_paths()))
+ """仅通过本地文件与完成清单检查仓库,不触发网络请求或修改模型目录。"""
+ return _verify_paths(repo, _repo_completion_paths(repo))
def model_path_exists(path: ModelPath) -> bool:
- """判断模型路径存在且所属目录已写入完整标记。"""
- local_path = path.local_path()
- if local_path.is_file():
- return True
- if not local_path.is_dir():
- return False
- repo_root = path.repo.local_dir()
- try:
- local_path.relative_to(repo_root)
- except ValueError:
- return False
- for directory in (local_path, *local_path.parents):
- if (directory / MODEL_COMPLETE_MARKER).is_file():
- return True
- if directory == repo_root:
- break
- return False
+ """与仓库检查一致地判断请求文件或目录是否就绪。"""
+ return not _model_path_missing(path)
__all__ = [
diff --git a/tests/unittest/test_kit_commands.py b/tests/unittest/test_kit_commands.py
index ac92f8ce..52a1ba0f 100644
--- a/tests/unittest/test_kit_commands.py
+++ b/tests/unittest/test_kit_commands.py
@@ -23,7 +23,7 @@ from mineru.kit.commands import api_server, models, parse, router, vlm_server, w
from mineru.kit.main import app
from mineru.kit.vlm_server import mlx_vlm_server
from mineru.parser.base import ParseResult
-from mineru.model.registry import MODEL_COMPLETE_MARKER
+from mineru.model.registry import MODEL_COMPLETE_MARKER, ModelRepo
from mineru.types import Tier
from mineru.version import __version__
@@ -33,6 +33,36 @@ _REMOVED_DISABLE_TABLE_OPTION = "--disable-" + "table"
_REMOVED_DISABLE_FORMULA_OPTION = "--disable-" + "formula"
+def _prepare_model_directory(directory: Path, files: list[str], *, legacy: bool = False) -> None:
+ """为命令行测试准备真实文件,以及旧空标记或独立构造的新版完成清单。"""
+ for name in files:
+ target = directory / name
+ target.parent.mkdir(parents=True, exist_ok=True)
+ target.write_bytes(b"x")
+ if legacy:
+ (directory / MODEL_COMPLETE_MARKER).touch()
+ return
+ (directory / MODEL_COMPLETE_MARKER).write_text(
+ json.dumps({"version": 1, "files": dict.fromkeys(files, 1)}), encoding="utf-8"
+ )
+
+
+def _prepare_model_repo(repo: ModelRepo, *, legacy: bool = False) -> None:
+ """按实际仓库的下载范围准备就绪资源,目录均需有效完成清单。"""
+ if repo.download_mode == "full":
+ _prepare_model_directory(
+ repo.local_dir(), list(repo.paths.values()) or ["config.json", "model.safetensors"], legacy=legacy
+ )
+ return
+ for model_path in repo.required_paths():
+ target = model_path.local_path()
+ if Path(model_path.relative_path).suffix:
+ target.parent.mkdir(parents=True, exist_ok=True)
+ target.write_bytes(b"x")
+ else:
+ _prepare_model_directory(target, ["model.bin"], legacy=legacy)
+
+
def test_kit_main_configures_standard_streams_before_running_app(monkeypatch: pytest.MonkeyPatch) -> None:
"""控制台入口先设置编码,再把日志初始化标记交给实际命令。"""
calls: list[str] = []
@@ -590,18 +620,7 @@ def test_models_show_and_verify(tmp_path: Path, monkeypatch: Any) -> None:
monkeypatch.setattr(models.get_config().model, "small_backend", "torch")
monkeypatch.setattr(models.get_config().model.vlm, "engine", "vllm")
for repo in models.MODEL_REPOS:
- if repo.download_mode == "full":
- repo.local_dir().mkdir(parents=True, exist_ok=True)
- (repo.local_dir() / MODEL_COMPLETE_MARKER).touch()
- continue
- for model_path in repo.required_paths():
- target = repo.local_dir() / model_path.relative_path
- if Path(model_path.relative_path).suffix:
- target.parent.mkdir(parents=True, exist_ok=True)
- target.write_text("x", encoding="utf-8")
- continue
- target.mkdir(parents=True, exist_ok=True)
- (target / MODEL_COMPLETE_MARKER).touch()
+ _prepare_model_repo(repo)
show_result = runner.invoke(app, ["models", "show"])
verify_result = runner.invoke(app, ["models", "verify"])
@@ -620,6 +639,39 @@ def test_models_show_and_verify(tmp_path: Path, monkeypatch: Any) -> None:
assert "MinerU2.5-Pro-2605-1.2B: ok" in verify_result.output
+def test_models_commands_accept_complete_legacy_markers_offline(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+ """原有模型的旧空标记可通过查看、校验和 Doclib 托管能力检查,且无需联网迁移。"""
+ from mineru.doclib import server
+ from mineru.model import download
+
+ monkeypatch.setattr(models.get_config().model, "base_dir", str(tmp_path / "models"))
+ monkeypatch.setattr(models.get_config().model, "small_backend", "torch")
+ monkeypatch.setattr(models.get_config().model, "source", "local")
+ monkeypatch.setattr(models.get_config().model.vlm, "engine", "vllm")
+ monkeypatch.setattr(models.get_config().model.vlm, "server_url", "")
+ for repo in models.MODEL_REPOS:
+ _prepare_model_repo(repo, legacy=True)
+
+ def fail_resolve(*args: object, **kwargs: object) -> str:
+ """任何尝试联网解析来源的行为均说明旧标记兼容失败。"""
+ raise AssertionError("legacy readiness checks must stay offline")
+
+ monkeypatch.setattr(download, "resolve_model_source", fail_resolve)
+
+ show = runner.invoke(app, ["models", "show"])
+ verify = runner.invoke(app, ["models", "verify"])
+ server._ensure_managed_parse_server_models_ready("standard", "parse_server.local.managed_tier")
+
+ assert show.exit_code == 0
+ assert "MinerU-4_models_torch: ready" in show.output
+ assert "MinerU2.5-Pro-2605-1.2B: ready" in show.output
+ assert verify.exit_code == 0
+ assert "MinerU-4_models_torch: ok" in verify.output
+ assert "MinerU2.5-Pro-2605-1.2B: ok" in verify.output
+ for repo in models.MODEL_REPOS:
+ assert all(marker.read_bytes() == b"" for marker in repo.local_dir().rglob(MODEL_COMPLETE_MARKER))
+
+
def test_api_server_rejects_backend_and_effort_options() -> None:
backend_result = runner.invoke(app, ["api-server", "--backend", "hybrid-engine"])
effort_result = runner.invoke(app, ["api-server", "--effort", "high"])
@@ -1349,18 +1401,7 @@ def test_models_verify_filters_by_effective_backends(tmp_path: Path, monkeypatch
# 仅准备当前后端所需仓库,其余仓库保持缺失。
for repo in models.MODEL_REPOS:
if repo in models.model_repos_for_tier("standard"):
- if repo.download_mode == "full":
- repo.local_dir().mkdir(parents=True, exist_ok=True)
- (repo.local_dir() / MODEL_COMPLETE_MARKER).touch()
- continue
- for model_path in repo.required_paths():
- target = repo.local_dir() / model_path.relative_path
- if Path(model_path.relative_path).suffix:
- target.parent.mkdir(parents=True, exist_ok=True)
- target.write_text("x", encoding="utf-8")
- else:
- target.mkdir(parents=True, exist_ok=True)
- (target / MODEL_COMPLETE_MARKER).touch()
+ _prepare_model_repo(repo)
result = runner.invoke(app, ["models", "verify"])
diff --git a/tests/unittest/test_model_manifest_upgrade.py b/tests/unittest/test_model_manifest_upgrade.py
new file mode 100644
index 00000000..4d6d20f5
--- /dev/null
+++ b/tests/unittest/test_model_manifest_upgrade.py
@@ -0,0 +1,506 @@
+"""旧模型完成标记的后台联网迁移与离线兼容回归。"""
+
+from __future__ import annotations
+
+import json
+import pickle
+import threading
+from collections.abc import Iterator
+from pathlib import Path
+from types import SimpleNamespace
+from unittest.mock import MagicMock, Mock
+
+import pytest
+import requests
+from filelock import FileLock
+
+from mineru.model import _manifest_upgrade as upgrade
+from mineru.model import download
+
+_REVISION = "a" * 40
+_FILES = {"config.json": 2, "model.safetensors": 7}
+
+
+def _write_model(root: Path, *, scope: str = "", source: str = "huggingface") -> None:
+ """准备旧标记、真实文件与下载器缓存记录,不依赖实际网络或模型权重。"""
+ directory = root / scope
+ directory.mkdir(parents=True, exist_ok=True)
+ (directory / "config.json").write_bytes(b"{}")
+ (directory / "model.safetensors").write_bytes(b"weights")
+ (directory / download.MODEL_COMPLETE_MARKER).touch()
+ names = [f"{scope}/{name}" if scope else name for name in _FILES]
+ if source == "huggingface":
+ for name in names:
+ metadata = root / ".cache/huggingface/download" / f"{name}.metadata"
+ metadata.parent.mkdir(parents=True, exist_ok=True)
+ metadata.write_text(f"{_REVISION}\n{'b' * 64}\n1000\n", encoding="utf-8")
+ else:
+ (root / ".msc").write_bytes(pickle.dumps([{"Path": name, "Revision": _REVISION} for name in names]))
+
+
+def _request(root: Path, *, scope: str = "") -> upgrade.ManifestUpgrade:
+ """构造目录范围明确的后台迁移请求。"""
+ lock_path = root.parent / ".locks" / f"{root.name}.lock"
+ lock_path.parent.mkdir(exist_ok=True)
+ return upgrade.ManifestUpgrade(
+ root=root,
+ lock_path=lock_path,
+ repos={"huggingface": "owner/repo", "modelscope": "Owner/repo"},
+ markers=(root / scope / download.MODEL_COMPLETE_MARKER,),
+ )
+
+
+def _join(workers: list[threading.Thread]) -> None:
+ """等待已安排的测试线程结束,避免后台访问已清理的临时目录。"""
+ for worker in workers:
+ worker.join(timeout=3)
+ assert not worker.is_alive()
+
+
+def environment(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> Iterator[tuple[download.ModelRepo, list[threading.Thread]]]:
+ """隔离仓库、调度状态和线程列表,所有测试在退出前回收自身任务。"""
+ monkeypatch.setattr(download.get_config().model, "base_dir", str(tmp_path))
+ monkeypatch.setattr(download.get_config().model, "source", "local")
+ monkeypatch.setattr(upgrade, "_SCHEDULER", upgrade._UpgradeScheduler())
+ repo = download.ModelRepo(name="repo", repos={"huggingface": "owner/repo", "modelscope": "Owner/repo"})
+ workers: list[threading.Thread] = []
+
+ def schedule(request: upgrade.ManifestUpgrade) -> threading.Thread | None:
+ """记录真实后台线程,同时保留生产调度器的去重和冷却行为。"""
+ worker = upgrade.schedule_manifest_upgrade(request)
+ if worker is not None:
+ workers.append(worker)
+ return worker
+
+ monkeypatch.setattr(download, "schedule_manifest_upgrade", schedule)
+ yield repo, workers
+ _join(workers)
+
+
[email protected]("source", ["auto", "huggingface", "modelscope", "local"])
[email protected]("entry", ["repo", "path"])
+def test_online_legacy_model_upgrades_without_downloading_payload(
+ environment: tuple[download.ModelRepo, list[threading.Thread]],
+ monkeypatch: pytest.MonkeyPatch,
+ source: str,
+ entry: str,
+) -> None:
+ """任意来源配置均可后台升级旧标记,local 也只放行元数据请求。"""
+ repo, workers = environment
+ _write_model(repo.local_dir())
+ monkeypatch.setattr(download.get_config().model, "source", source)
+ query = Mock(return_value=_FILES)
+ snapshot = Mock(side_effect=AssertionError("manifest migration must not download model files"))
+ resolve = Mock(side_effect=AssertionError("manifest migration must not change model source"))
+ monkeypatch.setattr(upgrade, "_remote_manifest", query)
+ monkeypatch.setattr(download, "_snapshot_download", snapshot)
+ monkeypatch.setattr(download, "resolve_model_source", resolve)
+ before = {name: (repo.local_dir() / name).read_bytes() for name in _FILES}
+
+ target = repo if entry == "repo" else repo.path("")
+ assert target.ensure() == repo.local_dir()
+ _join(workers)
+
+ marker = repo.local_dir() / download.MODEL_COMPLETE_MARKER
+ assert json.loads(marker.read_text()) == {"version": 1, "files": _FILES}
+ assert before == {name: (repo.local_dir() / name).read_bytes() for name in _FILES}
+ assert download.get_config().model.source == source
+ query.assert_called_once_with("huggingface", "owner/repo", _REVISION)
+ snapshot.assert_not_called()
+ resolve.assert_not_called()
+
+
+ "error", [requests.ConnectionError("offline"), requests.Timeout("timeout"), ValueError("bad response")]
+)
+def test_offline_or_failed_query_preserves_usable_legacy_model(
+ environment: tuple[download.ModelRepo, list[threading.Thread]], monkeypatch: pytest.MonkeyPatch, error: Exception
+) -> None:
+ """无网、超时或元数据损坏均不影响旧模型使用,且短期内不会反复请求。"""
+ repo, workers = environment
+ _write_model(repo.local_dir())
+ query = Mock(side_effect=error)
+ monkeypatch.setattr(upgrade, "_remote_manifest", query)
+
+ assert repo.ensure() == repo.local_dir()
+ _join(workers)
+ assert repo.ensure() == repo.local_dir()
+ _join(workers)
+
+ assert download.verify_model_repo(repo).ready
+ assert (repo.local_dir() / download.MODEL_COMPLETE_MARKER).read_bytes() == b""
+ query.assert_called_once()
+
+
+def test_offline_device_can_upgrade_after_retry_cooldown(
+ environment: tuple[download.ModelRepo, list[threading.Thread]], monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """设备从无网恢复联网后,冷却结束的下一次使用会重新尝试迁移。"""
+ repo, workers = environment
+ _write_model(repo.local_dir())
+ now = [1000.0]
+ monkeypatch.setattr(upgrade, "monotonic", lambda: now[0])
+ query = Mock(side_effect=[requests.ConnectionError("offline"), _FILES])
+ monkeypatch.setattr(upgrade, "_remote_manifest", query)
+
+ repo.ensure()
+ _join(workers)
+ assert (repo.local_dir() / download.MODEL_COMPLETE_MARKER).read_bytes() == b""
+ now[0] += upgrade._RETRY_COOLDOWN + 1
+ repo.ensure()
+ _join(workers)
+
+ assert query.call_count == 2
+ assert json.loads((repo.local_dir() / download.MODEL_COMPLETE_MARKER).read_text())["files"] == _FILES
+
+
+def test_pending_metadata_query_does_not_block_model_use(
+ environment: tuple[download.ModelRepo, list[threading.Thread]], monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """远端查询被阻塞时,模型加载仍可返回,重复调用也只保留一个后台任务。"""
+ repo, workers = environment
+ _write_model(repo.local_dir())
+ entered, release, returned = threading.Event(), threading.Event(), threading.Event()
+
+ def query(source: str, repo_id: str, revision: str) -> dict[str, int]:
+ """将查询保持在进行中,以检测调用方是否错误等待网络。"""
+ entered.set()
+ assert release.wait(timeout=3)
+ return _FILES
+
+ remote = Mock(side_effect=query)
+ monkeypatch.setattr(upgrade, "_remote_manifest", remote)
+
+ def load() -> None:
+ """模拟正常使用线程,通过事件报告 ensure 已经返回。"""
+ repo.ensure()
+ returned.set()
+
+ caller = threading.Thread(target=load)
+ caller.start()
+ try:
+ assert entered.wait(timeout=1)
+ assert returned.wait(timeout=1)
+ assert repo.ensure() == repo.local_dir()
+ assert remote.call_count == 1
+ assert all(worker.daemon for worker in workers)
+ finally:
+ release.set()
+ caller.join(timeout=3)
+ _join(workers)
+
+
+def test_background_upgrade_uses_captured_model_directory(
+ environment: tuple[download.ModelRepo, list[threading.Thread]], monkeypatch: pytest.MonkeyPatch, tmp_path: Path
+) -> None:
+ """启动任务后配置上下文切换,也只更新最初使用的目录,不串到其他模型目录。"""
+ repo, workers = environment
+ root = repo.local_dir()
+ _write_model(root)
+ entered, release = threading.Event(), threading.Event()
+
+ def query(source: str, repo_id: str, revision: str) -> dict[str, int]:
+ """在主线程切换配置之前暂停元数据返回。"""
+ entered.set()
+ assert release.wait(timeout=3)
+ return _FILES
+
+ monkeypatch.setattr(upgrade, "_remote_manifest", query)
+ repo.ensure()
+ try:
+ assert entered.wait(timeout=1)
+ monkeypatch.setattr(download.get_config().model, "base_dir", str(tmp_path / "other-context"))
+ finally:
+ release.set()
+ _join(workers)
+
+ assert json.loads((root / download.MODEL_COMPLETE_MARKER).read_text())["files"] == _FILES
+ assert not (tmp_path / "other-context").exists()
+
+
+def test_concurrent_repositories_are_bounded_and_can_retry_when_slot_is_free(
+ environment: tuple[download.ModelRepo, list[threading.Thread]], monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """多个仓库同时使用时迁移有并发上限,被让位的仓库可在后续使用时再次安排。"""
+ repo, workers = environment
+ repos = [download.ModelRepo(name=f"repo-{index}", repos=repo.repos) for index in range(3)]
+ for target in repos:
+ _write_model(target.local_dir())
+ release = threading.Event()
+
+ def query(source: str, repo_id: str, revision: str) -> dict[str, int]:
+ """暂不完成现有请求,验证新仓库不会无限创建后台线程。"""
+ assert release.wait(timeout=3)
+ return _FILES
+
+ monkeypatch.setattr(upgrade, "_remote_manifest", query)
+ try:
+ for target in repos:
+ target.ensure()
+ assert len(workers) == 2
+ assert (repos[2].local_dir() / download.MODEL_COMPLETE_MARKER).read_bytes() == b""
+ finally:
+ release.set()
+ _join(workers)
+ repos[2].ensure()
+ _join(workers)
+
+ assert len(workers) == 3
+ assert all(download.verify_model_repo(target).ready for target in repos)
+ assert json.loads((repos[2].local_dir() / download.MODEL_COMPLETE_MARKER).read_text())["files"] == _FILES
+
+
+def test_background_thread_start_failure_does_not_break_model_use(
+ environment: tuple[download.ModelRepo, list[threading.Thread]], monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """解释器退出或系统无法创建线程时,旧模型仍正常返回并释放任务配额。"""
+ repo, workers = environment
+ _write_model(repo.local_dir())
+ monkeypatch.setattr(threading.Thread, "start", Mock(side_effect=RuntimeError("cannot start thread")))
+
+ assert repo.ensure() == repo.local_dir()
+ assert not workers
+ assert not upgrade._SCHEDULER.active
+ assert (repo.local_dir() / download.MODEL_COMPLETE_MARKER).read_bytes() == b""
+
+
[email protected]("files", [{"config.json": 2, "model.safetensors": 99}, {**_FILES, "missing.json": 10}])
+def test_mismatched_remote_manifest_keeps_old_marker(
+ environment: tuple[download.ModelRepo, list[threading.Thread]], monkeypatch: pytest.MonkeyPatch, files: dict[str, int]
+) -> None:
+ """远端多出文件或大小不一致时保留旧标记,不删除、补下载或改写已有模型。"""
+ repo, workers = environment
+ _write_model(repo.local_dir())
+ monkeypatch.setattr(upgrade, "_remote_manifest", Mock(return_value=files))
+
+ repo.ensure()
+ _join(workers)
+
+ assert (repo.local_dir() / download.MODEL_COMPLETE_MARKER).read_bytes() == b""
+ assert (repo.local_dir() / "model.safetensors").read_bytes() == b"weights"
+ assert not (repo.local_dir() / "missing.json").exists()
+
+
+def test_partial_directory_migration_does_not_publish_root_completion(
+ environment: tuple[download.ModelRepo, list[threading.Thread]], monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """局部目录只升级自己的完整范围,不将单个模型目录当作整个仓库。"""
+ repo, workers = environment
+ _write_model(repo.local_dir(), scope="weights")
+ remote = {f"weights/{name}": size for name, size in _FILES.items()}
+ monkeypatch.setattr(upgrade, "_remote_manifest", Mock(return_value={**remote, "other/model.bin": 100}))
+
+ repo.path("weights").ensure()
+ _join(workers)
+
+ assert json.loads((repo.local_dir() / "weights" / download.MODEL_COMPLETE_MARKER).read_text())["files"] == _FILES
+ assert not (repo.local_dir() / download.MODEL_COMPLETE_MARKER).exists()
+ assert not download.verify_model_repo(repo).ready
+
+
[email protected]("change", ["marker", "payload"])
+def test_concurrent_update_is_not_overwritten_by_migration(
+ environment: tuple[download.ModelRepo, list[threading.Thread]], monkeypatch: pytest.MonkeyPatch, change: str
+) -> None:
+ """请求网络期间其他操作更新标记或模型文件时,后台迁移不能覆盖新状态。"""
+ repo, workers = environment
+ _write_model(repo.local_dir())
+ marker = repo.local_dir() / download.MODEL_COMPLETE_MARKER
+ other = {"version": 1, "files": {"model.safetensors": 7}}
+
+ def query(source: str, repo_id: str, revision: str) -> dict[str, int]:
+ """在网络返回之前模拟另一个下载或更新操作。"""
+ if change == "marker":
+ marker.write_text(json.dumps(other), encoding="utf-8")
+ else:
+ (repo.local_dir() / "model.safetensors").write_bytes(b"changed payload")
+ return _FILES
+
+ monkeypatch.setattr(upgrade, "_remote_manifest", query)
+ repo.ensure()
+ _join(workers)
+
+ if change == "marker":
+ assert json.loads(marker.read_text()) == other
+ else:
+ assert marker.read_bytes() == b""
+ assert (repo.local_dir() / "model.safetensors").read_bytes() == b"changed payload"
+
+
+def test_payload_is_rechecked_inside_repository_lock(
+ environment: tuple[download.ModelRepo, list[threading.Thread]], monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """网络返回后的第一次检查通过,也不能省略锁内对文件变更的第二次检查。"""
+ repo, workers = environment
+ _write_model(repo.local_dir())
+ monkeypatch.setattr(upgrade, "_remote_manifest", Mock(return_value=_FILES))
+ verify = upgrade.missing_manifest_files
+ calls = []
+
+ def check(root: Path, files: dict[str, int]) -> list[str]:
+ """在进入锁内复核前模拟大小发生变化。"""
+ calls.append(root)
+ if len(calls) == 2:
+ (root / "model.safetensors").write_bytes(b"changed payload")
+ return verify(root, files)
+
+ monkeypatch.setattr(upgrade, "missing_manifest_files", check)
+ repo.ensure()
+ _join(workers)
+
+ assert len(calls) == 2
+ assert (repo.local_dir() / download.MODEL_COMPLETE_MARKER).read_bytes() == b""
+
+
[email protected]("failure", ["lock", "write"])
+def test_busy_or_readonly_repository_keeps_legacy_marker(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, failure: str) -> None:
+ """锁被正常下载占用或目录不允许发布清单时,迁移安静让位并保留旧文件。"""
+ root = tmp_path / "repo"
+ _write_model(root)
+ request = _request(root)
+ monkeypatch.setattr(upgrade, "_SCHEDULER", upgrade._UpgradeScheduler())
+ monkeypatch.setattr(upgrade, "_remote_manifest", Mock(return_value=_FILES))
+ if failure == "write":
+ monkeypatch.setattr(upgrade, "write_manifest", Mock(side_effect=PermissionError("read only")))
+ worker = upgrade.schedule_manifest_upgrade(request)
+ assert worker is not None
+ _join([worker])
+ else:
+ with FileLock(str(request.lock_path)):
+ worker = upgrade.schedule_manifest_upgrade(request)
+ assert worker is not None
+ _join([worker])
+
+ assert (root / download.MODEL_COMPLETE_MARKER).read_bytes() == b""
+ assert (root / "model.safetensors").read_bytes() == b"weights"
+
+
[email protected]("cache", ["missing", "mixed", "unsafe_pickle"])
+def test_unidentified_cached_revision_is_not_replaced_with_latest(
+ environment: tuple[download.ModelRepo, list[threading.Thread]], monkeypatch: pytest.MonkeyPatch, cache: str
+) -> None:
+ """缺少可信旧版本、混合版本或包含可执行对象的缓存均跳过迁移。"""
+ repo, workers = environment
+ _write_model(repo.local_dir())
+ metadata = repo.local_dir() / ".cache/huggingface/download"
+ if cache == "mixed":
+ (metadata / "config.json.metadata").write_text(f"{'c' * 40}\netag\n1000\n", encoding="utf-8")
+ else:
+ for path in metadata.rglob("*.metadata"):
+ path.unlink()
+ if cache == "unsafe_pickle":
+ (repo.local_dir() / ".msc").write_bytes(b"cos\nsystem\n.")
+ query = Mock(side_effect=AssertionError("unknown cached revisions must not query latest"))
+ monkeypatch.setattr(upgrade, "_remote_manifest", query)
+
+ repo.ensure()
+ _join(workers)
+
+ assert (repo.local_dir() / download.MODEL_COMPLETE_MARKER).read_bytes() == b""
+ query.assert_not_called()
+
+
+def test_modelscope_cache_can_upgrade_using_metadata_only(
+ environment: tuple[download.ModelRepo, list[threading.Thread]], monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """ModelScope 的旧缓存也能在 local 模式下通过带超时的元数据 GET 完成升级。"""
+ repo, workers = environment
+ _write_model(repo.local_dir(), source="modelscope")
+ response = MagicMock()
+ response.__enter__.return_value = response
+ response.json.return_value = {
+ "Code": 200,
+ "Data": {
+ "Files": [{"Path": name, "Size": size, "Type": "blob", "Revision": _REVISION} for name, size in _FILES.items()]
+ },
+ }
+ get = Mock(return_value=response)
+ monkeypatch.setattr(upgrade.requests, "get", get)
+ snapshot = Mock(side_effect=AssertionError("metadata upgrade cannot invoke snapshot_download"))
+ monkeypatch.setattr(download, "ms_snapshot_download", snapshot)
+
+ repo.ensure()
+ _join(workers)
+
+ assert json.loads((repo.local_dir() / download.MODEL_COMPLETE_MARKER).read_text())["files"] == _FILES
+ get.assert_called_once_with(
+ "https://modelscope.cn/api/v1/models/Owner/repo/repo/files",
+ params={"Revision": _REVISION, "Recursive": "True"},
+ timeout=(upgrade._QUERY_CONNECT_TIMEOUT, upgrade._QUERY_TIMEOUT),
+ )
+ snapshot.assert_not_called()
+
+
[email protected]("changed", [False, True])
+def test_modelscope_mixed_file_commits_must_match_cached_versions(
+ environment: tuple[download.ModelRepo, list[threading.Thread]], monkeypatch: pytest.MonkeyPatch, changed: bool
+) -> None:
+ """MS 正常的逐文件多 commit 缓存可升级;远端任一文件版本变化时保留旧标记。"""
+ repo, workers = environment
+ _write_model(repo.local_dir(), source="modelscope")
+ revisions = {"config.json": "c" * 40, "model.safetensors": _REVISION}
+ (repo.local_dir() / ".msc").write_bytes(
+ pickle.dumps([{"Path": name, "Revision": revision} for name, revision in revisions.items()])
+ )
+ (repo.local_dir() / ".mv").write_text("Revision:v1.0,CreatedAt:1000", encoding="utf-8")
+ response = MagicMock()
+ response.__enter__.return_value = response
+ response.json.return_value = {
+ "Code": 200,
+ "Data": {
+ "Files": [
+ {
+ "Path": name,
+ "Size": size,
+ "Type": "blob",
+ "Revision": "d" * 40 if changed and name == "model.safetensors" else revisions[name],
+ }
+ for name, size in _FILES.items()
+ ]
+ },
+ }
+ get = Mock(return_value=response)
+ monkeypatch.setattr(upgrade.requests, "get", get)
+
+ repo.ensure()
+ _join(workers)
+
+ assert get.call_args.kwargs["params"]["Revision"] == "v1.0"
+ marker = repo.local_dir() / download.MODEL_COMPLETE_MARKER
+ if changed:
+ assert marker.read_bytes() == b""
+ else:
+ assert json.loads(marker.read_text())["files"] == _FILES
+
+
+def test_huggingface_query_pins_cached_revision_and_timeout(monkeypatch: pytest.MonkeyPatch) -> None:
+ """HF 请求明确固定旧 commit 并开启文件大小字段,避免隐式访问最新版或无限等待。"""
+ api = Mock()
+ api.model_info.return_value = SimpleNamespace(
+ sha=_REVISION, siblings=[SimpleNamespace(rfilename=name, size=size) for name, size in _FILES.items()]
+ )
+ monkeypatch.setattr(upgrade, "HfApi", Mock(return_value=api))
+
+ assert upgrade._remote_manifest("huggingface", "owner/repo", _REVISION) == _FILES
+ api.model_info.assert_called_once_with(
+ "owner/repo", revision=_REVISION, files_metadata=True, timeout=upgrade._QUERY_TIMEOUT
+ )
+ api.model_info.return_value.sha = "c" * 40
+ assert upgrade._remote_manifest("huggingface", "owner/repo", _REVISION) == {}
+
+
+def test_readonly_verification_does_not_schedule_network_upgrade(
+ environment: tuple[download.ModelRepo, list[threading.Thread]], monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """show/verify 使用的只读检查不启动后台任务、不修改旧标记。"""
+ repo, workers = environment
+ _write_model(repo.local_dir())
+ schedule = Mock(side_effect=AssertionError("readonly verification cannot schedule migration"))
+ monkeypatch.setattr(download, "schedule_manifest_upgrade", schedule)
+
+ assert download.verify_model_repo(repo).ready
+ assert download.model_path_exists(repo.path(""))
+ assert not workers
+ assert (repo.local_dir() / download.MODEL_COMPLETE_MARKER).read_bytes() == b""
+ schedule.assert_not_called()
diff --git a/tests/unittest/test_model_readiness.py b/tests/unittest/test_model_readiness.py
new file mode 100644
index 00000000..08e9d17b
--- /dev/null
+++ b/tests/unittest/test_model_readiness.py
@@ -0,0 +1,546 @@
+"""模型文件被清理、损坏及局部更新后的真实就绪行为回归。"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from types import SimpleNamespace
+from typing import Literal, NoReturn
+from unittest.mock import Mock
+
+import pytest
+
+from mineru.model import download
+
+
+def repo(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> download.ModelRepo:
+ """将下载和离线检查隔离在临时模型目录中。"""
+ monkeypatch.setattr(download.get_config().model, "base_dir", str(tmp_path))
+ return download.ModelRepo(
+ name="test-model",
+ repos={"huggingface": "owner/repo", "modelscope": "Owner/repo"},
+ paths={"config": "config.json", "weights": "model.safetensors"},
+ )
+
+
+def _write_files(root: Path, files: dict[str, bytes]) -> dict[str, int]:
+ """准备实际模型文件,并返回与字节内容独立对应的预期大小。"""
+ for name, content in files.items():
+ target = root / name
+ target.parent.mkdir(parents=True, exist_ok=True)
+ if target.is_symlink():
+ target.unlink()
+ target.write_bytes(content)
+ return {name: len(content) for name, content in files.items()}
+
+
+def _write_marker(root: Path, files: dict[str, int]) -> None:
+ """直接按协议构造完成标记,使回归测试不依赖生产清单写入函数。"""
+ (root / download.MODEL_COMPLETE_MARKER).write_text(json.dumps({"version": 1, "files": files}), encoding="utf-8")
+
+
+ "remaining",
+ [
+ {},
+ {"README.md": b"readme"},
+ {"config.json": b"{}"},
+ {"config.json": b"{}", "model.safetensors": b""},
+ {".cache/huggingface/model.safetensors": b"cached", ".msc": b"metadata"},
+ {"._____temp/model.safetensors": b"partial"},
+ ],
+)
+def test_legacy_marker_cannot_confirm_remaining_files(
+ repo: download.ModelRepo, monkeypatch: pytest.MonkeyPatch, remaining: dict[str, bytes]
+) -> None:
+ """旧空标记及少量残留文件均不能在离线模式中被认定为完整模型。"""
+ root = repo.local_dir()
+ root.mkdir()
+ _write_files(root, remaining)
+ marker = root / download.MODEL_COMPLETE_MARKER
+ marker.touch()
+ monkeypatch.setattr(download.get_config().model, "source", "local")
+ remote = Mock(side_effect=AssertionError("offline checks must not download"))
+ monkeypatch.setattr(download, "_snapshot_download", remote)
+
+ result = download.verify_model_repo(repo)
+
+ assert result.ready is False
+ assert result.missing_paths
+ assert not download.model_path_exists(repo.path(""))
+ for target in (repo, repo.path("")):
+ with pytest.raises(FileNotFoundError, match="repair files"):
+ target.ensure()
+ assert marker.read_bytes() == b""
+ remote.assert_not_called()
+
+
[email protected]("source", ["auto", "huggingface", "modelscope", "local"])
+def test_complete_legacy_model_stays_offline_without_marker_upgrade(
+ repo: download.ModelRepo, monkeypatch: pytest.MonkeyPatch, source: str
+) -> None:
+ """完整旧模型在任意来源配置下均直接复用,仓库及目录加载不联网也不改写标记。"""
+ _write_files(repo.local_dir(), {"config.json": b"{}", "model.safetensors": b"weights", "ocr/inference.pth": b"detector"})
+ marker = repo.local_dir() / download.MODEL_COMPLETE_MARKER
+ marker.touch()
+ monkeypatch.setattr(download.get_config().model, "source", source)
+ resolve = Mock(side_effect=AssertionError("complete legacy models must stay offline"))
+ monkeypatch.setattr(download, "resolve_model_source", resolve)
+
+ assert download.verify_model_repo(repo).ready
+ assert repo.ensure() == repo.local_dir()
+ assert repo.path("").ensure() == repo.local_dir()
+ assert repo.path("ocr").ensure() == repo.local_dir() / "ocr"
+ assert repo.weights.ensure() == repo.weights.local_path()
+ assert marker.read_bytes() == b""
+ assert not (repo.local_dir() / "ocr" / download.MODEL_COMPLETE_MARKER).exists()
+ resolve.assert_not_called()
+
+
+def test_legacy_marker_still_requires_declared_config(repo: download.ModelRepo) -> None:
+ """权重存在也不能掩盖 registry 已声明的必要配置文件缺失。"""
+ _write_files(repo.local_dir(), {"model.safetensors": b"weights"})
+ (repo.local_dir() / download.MODEL_COMPLETE_MARKER).touch()
+
+ assert download.verify_model_repo(repo).missing_paths == ["config.json"]
+
+
[email protected]("damage", ["intact", "deleted", "empty", "broken_link"])
+def test_legacy_weight_index_checks_all_shards(repo: download.ModelRepo, monkeypatch: pytest.MonkeyPatch, damage: str) -> None:
+ """原始 VLM 没有静态逐文件清单时,仍通过自带权重索引检查每一个分片。"""
+ raw_repo = download.ModelRepo(name=repo.name, repos=repo.repos)
+ index = {"weight_map": {"layer.0": "model-00001.safetensors", "layer.1": "model-00002.safetensors"}}
+ _write_files(
+ raw_repo.local_dir(),
+ {
+ "config.json": b"{}",
+ "model.safetensors.index.json": json.dumps(index).encode(),
+ "model-00001.safetensors": b"first shard",
+ "model-00002.safetensors": b"second shard",
+ },
+ )
+ (raw_repo.local_dir() / download.MODEL_COMPLETE_MARKER).touch()
+ second = raw_repo.local_dir() / "model-00002.safetensors"
+ if damage != "intact":
+ second.unlink()
+ if damage == "empty":
+ second.touch()
+ elif damage == "broken_link":
+ second.symlink_to("missing-target")
+ resolve = Mock(side_effect=AssertionError("legacy verification must stay offline"))
+ monkeypatch.setattr(download, "resolve_model_source", resolve)
+
+ result = download.verify_model_repo(raw_repo)
+
+ if damage == "intact":
+ assert result.ready
+ assert raw_repo.ensure() == raw_repo.local_dir()
+ else:
+ assert not result.ready
+ assert result.missing_paths == ["model-00002.safetensors"]
+ resolve.assert_not_called()
+
+
[email protected]("index", ["{", "[]", '{"weight_map": {}}', '{"weight_map": {"layer": "../outside"}}'])
+def test_legacy_broken_weight_index_is_rejected(repo: download.ModelRepo, index: str) -> None:
+ """旧标记不能绕过损坏、空或包含不安全分片路径的权重索引。"""
+ _write_files(repo.local_dir(), {"config.json": b"{}", "model.safetensors": b"weights"})
+ (repo.local_dir() / "model.safetensors.index.json").write_text(index, encoding="utf-8")
+ (repo.local_dir() / download.MODEL_COMPLETE_MARKER).touch()
+
+ assert download.verify_model_repo(repo).missing_paths == ["model.safetensors.index.json"]
+
+
+def test_legacy_gguf_requires_both_registered_weights(repo: download.ModelRepo) -> None:
+ """GGUF 旧目录必须同时具备主模型与投影文件,保留其中一个不能判为就绪。"""
+ from mineru.model.registry import MINERU_2_5_PRO_2605_1_2B_GGUF
+
+ gguf = MINERU_2_5_PRO_2605_1_2B_GGUF
+ _write_files(gguf.local_dir(), {gguf.paths["main"]: b"main model"})
+ (gguf.local_dir() / download.MODEL_COMPLETE_MARKER).touch()
+
+ assert download.verify_model_repo(gguf).missing_paths == [gguf.paths["mmproj"]]
+
+ _write_files(gguf.local_dir(), {gguf.paths["mmproj"]: b"projector"})
+ assert download.verify_model_repo(gguf).ready
+
+
+def test_legacy_child_marker_cannot_override_new_parent_manifest(repo: download.ModelRepo) -> None:
+ """祖先已有新版清单时,旧子标记不能把缺少一个权重的目录降级放行。"""
+ files = _write_files(repo.local_dir(), {"det/first.onnx": b"first", "det/second.onnx": b"second"})
+ _write_marker(repo.local_dir(), files)
+ (repo.local_dir() / "det" / download.MODEL_COMPLETE_MARKER).touch()
+ (repo.local_dir() / "det/second.onnx").unlink()
+
+ result = download._verify_paths(repo, [repo.path("det")])
+
+ assert not result.ready
+ assert result.missing_paths == ["det/second.onnx"]
+ assert not download.model_path_exists(repo.path("det"))
+
+
+def test_legacy_parent_still_checks_new_child_manifest(repo: download.ModelRepo) -> None:
+ """旧父目录复用时也必须尊重已确认的新版子清单,不能忽略后来丢失的成员。"""
+ _write_files(repo.local_dir(), {"config.json": b"{}", "model.safetensors": b"weights"})
+ files = _write_files(repo.local_dir() / "det", {"first.onnx": b"first", "second.onnx": b"second"})
+ _write_marker(repo.local_dir() / "det", files)
+ (repo.local_dir() / download.MODEL_COMPLETE_MARKER).touch()
+ (repo.local_dir() / "det/second.onnx").unlink()
+
+ assert download.verify_model_repo(repo).missing_paths == ["det/second.onnx"]
+ assert not download.model_path_exists(repo.path("det"))
+
+
[email protected]("damage", ["deleted", "truncated", "empty", "broken_link"])
[email protected]("entry", ["repo", "path"])
+def test_missing_payload_is_reported_offline_and_repaired_online(
+ repo: download.ModelRepo,
+ monkeypatch: pytest.MonkeyPatch,
+ damage: str,
+ entry: str,
+) -> None:
+ """已完成模型缺失或截断时,两种入口均报告具体文件并通过一次下载恢复。"""
+ contents = {"config.json": b"{}", "model.safetensors": b"complete model weights"}
+ files = _write_files(repo.local_dir(), contents)
+ _write_marker(repo.local_dir(), files)
+ weight = repo.weights.local_path()
+ weight.unlink()
+ if damage == "truncated":
+ weight.write_bytes(b"short")
+ elif damage == "empty":
+ weight.touch()
+ elif damage == "broken_link":
+ weight.symlink_to("missing-target")
+ target = repo if entry == "repo" else repo.path("")
+ monkeypatch.setattr(download.get_config().model, "source", "local")
+ remote = Mock(side_effect=AssertionError("offline checks must not download"))
+ monkeypatch.setattr(download, "_snapshot_download", remote)
+
+ assert download.verify_model_repo(repo).missing_paths == ["model.safetensors"]
+ assert not download.model_path_exists(repo.weights)
+ with pytest.raises(FileNotFoundError, match="model.safetensors"):
+ target.ensure()
+ remote.assert_not_called()
+
+ def repair(source: str, target_repo: download.ModelRepo, patterns: list[str] | None) -> dict[str, int]:
+ """模拟 provider 增量修复,并确认根目录请求没有被展开成空过滤器。"""
+ assert source == "modelscope"
+ assert target_repo is repo
+ assert patterns is None
+ assert not (repo.local_dir() / download.MODEL_COMPLETE_MARKER).exists()
+ return _write_files(repo.local_dir(), contents)
+
+ remote.side_effect = repair
+ assert target.ensure(source="modelscope") == repo.local_dir()
+ assert target.ensure(source="modelscope") == repo.local_dir()
+ assert remote.call_count == 1
+ assert download.verify_model_repo(repo).ready
+ assert download.model_path_exists(repo.path(""))
+
+
+ "marker_text",
+ [
+ " ",
+ "{",
+ "[]",
+ '{"version": 2, "files": {"model.safetensors": 7}}',
+ '{"version": true, "files": {"model.safetensors": 7}}',
+ '{"version": 1, "files": {}}',
+ '{"version": 1, "files": {"../outside": 7}}',
+ '{"version": 1, "files": {"/outside": 7}}',
+ '{"version": 1, "files": {"C:/outside": 7}}',
+ '{"version": 1, "files": {"bad\\u0000path": 7}}',
+ '{"version": 1, "files": {".cache/metadata": 7}}',
+ '{"version": 1, "files": {"model.safetensors": -1}}',
+ '{"version": 1, "files": {"model.safetensors": true}}',
+ ],
+)
+def test_invalid_completion_manifest_is_not_ready(repo: download.ModelRepo, marker_text: str) -> None:
+ """损坏或不受支持的清单不能通过本地目录存在性检查。"""
+ _write_files(repo.local_dir(), {"model.safetensors": b"weights"})
+ (repo.local_dir() / download.MODEL_COMPLETE_MARKER).write_text(marker_text, encoding="utf-8")
+
+ assert not download.verify_model_repo(repo).ready
+ assert not download.model_path_exists(repo.path(""))
+
+
+def test_legacy_marker_is_upgraded_after_provider_confirmation(
+ repo: download.ModelRepo, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """旧目录只有经 provider 确认完整后才升级,不能扫描残留文件自行认证。"""
+ _write_files(repo.local_dir(), {"config.json": b"{}"})
+ marker = repo.local_dir() / download.MODEL_COMPLETE_MARKER
+ marker.touch()
+
+ def repair(source: str, target_repo: download.ModelRepo, patterns: list[str] | None) -> dict[str, int]:
+ """下载器补齐原目录缺少的权重,并返回完整预期集合。"""
+ assert not marker.exists()
+ return _write_files(target_repo.local_dir(), {"config.json": b"{}", "model.safetensors": b"weights"})
+
+ remote = Mock(side_effect=repair)
+ monkeypatch.setattr(download, "_snapshot_download", remote)
+
+ repo.ensure(source="modelscope")
+
+ assert remote.call_count == 1
+ assert json.loads(marker.read_text())["files"] == {"config.json": 2, "model.safetensors": 7}
+ assert download.verify_model_repo(repo).ready
+
+
+def test_ready_model_reuses_local_manifest_without_source_resolution(
+ repo: download.ModelRepo, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """正常模型在整仓、根目录和子目录入口均通过纯本地校验复用。"""
+ files = _write_files(repo.local_dir(), {"model/config.json": b"{}", "model/model.safetensors": b"weights"})
+ _write_marker(repo.local_dir(), files)
+ resolve = Mock(side_effect=AssertionError("ready models must not resolve a remote source"))
+ monkeypatch.setattr(download, "resolve_model_source", resolve)
+
+ assert repo.ensure() == repo.local_dir()
+ assert repo.path("").ensure() == repo.local_dir()
+ assert repo.path("model").ensure() == repo.local_dir() / "model"
+ resolve.assert_not_called()
+
+
[email protected]("damage", ["legacy", "deleted", "truncated"])
+def test_required_directory_verifies_and_repairs_all_members(
+ repo: download.ModelRepo, monkeypatch: pytest.MonkeyPatch, damage: str
+) -> None:
+ """小模型目录保留配置时,必需权重缺失也必须通过一次增量下载补齐。"""
+ directory_repo = download.ModelRepo(
+ name=repo.name, repos=repo.repos, paths={"weights": "weights"}, download_mode="required_paths"
+ )
+ files = _write_files(directory_repo.local_dir() / "weights", {"config.json": b"{}", "model.onnx": b"weights"})
+ marker = directory_repo.weights.local_path() / download.MODEL_COMPLETE_MARKER
+ if damage == "legacy":
+ marker.touch()
+ (directory_repo.weights.local_path() / "model.onnx").unlink()
+ else:
+ _write_marker(directory_repo.weights.local_path(), files)
+ (directory_repo.weights.local_path() / "model.onnx").unlink()
+ if damage == "truncated":
+ (directory_repo.weights.local_path() / "model.onnx").write_bytes(b"short")
+ monkeypatch.setattr(download.get_config().model, "source", "local")
+
+ assert not download.verify_model_repo(directory_repo).ready
+ assert not download.model_path_exists(directory_repo.weights)
+ with pytest.raises(FileNotFoundError, match="weights"):
+ directory_repo.weights.ensure()
+
+ def repair(source: str, target_repo: download.ModelRepo, patterns: list[str] | None) -> dict[str, int]:
+ """只确认和修复所请求的小模型目录。"""
+ assert patterns == ["weights", "weights/*"]
+ return _write_files(target_repo.local_dir(), {"weights/config.json": b"{}", "weights/model.onnx": b"weights"})
+
+ remote = Mock(side_effect=repair)
+ monkeypatch.setattr(download, "_snapshot_download", remote)
+ directory_repo.weights.ensure(source="modelscope")
+ directory_repo.ensure(source="modelscope")
+
+ assert remote.call_count == 1
+ assert download.verify_model_repo(directory_repo).ready
+
+
+def test_partial_repair_does_not_hide_missing_sibling(repo: download.ModelRepo, monkeypatch: pytest.MonkeyPatch) -> None:
+ """检测器下载成功不能恢复仍缺少识别器的父目录完成状态。"""
+ files = _write_files(repo.local_dir(), {"det/model.onnx": b"detector", "rec/model.onnx": b"recognizer"})
+ _write_marker(repo.local_dir(), files)
+ (repo.local_dir() / "rec/model.onnx").unlink()
+
+ def repair(source: str, target_repo: download.ModelRepo, patterns: list[str] | None) -> dict[str, int]:
+ """模拟仅确认请求的检测器目录。"""
+ assert patterns == ["det", "det/*"]
+ return _write_files(target_repo.local_dir(), {"det/model.onnx": b"new detector"})
+
+ monkeypatch.setattr(download, "_snapshot_download", repair)
+ download.download_model_files(repo, [repo.path("det")], source="modelscope")
+
+ assert download.model_path_exists(repo.path("det"))
+ assert not (repo.local_dir() / download.MODEL_COMPLETE_MARKER).exists()
+ assert not download.verify_model_repo(repo).ready
+
+
[email protected]("fail", [False, True])
+def test_single_file_update_invalidates_and_revalidates_parent_manifest(
+ repo: download.ModelRepo, monkeypatch: pytest.MonkeyPatch, fail: bool
+) -> None:
+ """单文件更新也撤销父标记,成功后合并新大小,失败后保持未完成。"""
+ files = _write_files(repo.local_dir(), {"config.json": b"{}", "model.safetensors": b"weights"})
+ _write_marker(repo.local_dir(), files)
+ marker = repo.local_dir() / download.MODEL_COMPLETE_MARKER
+
+ def update(source: str, target_repo: download.ModelRepo, patterns: list[str] | None) -> dict[str, int]:
+ """验证标记在 provider 改写文件之前已经撤销。"""
+ assert not marker.exists()
+ if fail:
+ raise RuntimeError("download interrupted")
+ return _write_files(target_repo.local_dir(), {"model.safetensors": b"updated model weights"})
+
+ monkeypatch.setattr(download, "_snapshot_download", update)
+ if fail:
+ with pytest.raises(RuntimeError, match="download interrupted"):
+ download.download_model_files(repo, [repo.weights], source="modelscope")
+ assert not marker.exists()
+ else:
+ download.download_model_files(repo, [repo.weights], source="modelscope")
+ assert download.verify_model_repo(repo).ready
+ assert json.loads(marker.read_text())["files"] == {"config.json": 2, "model.safetensors": 21}
+
+
+def test_multiple_directory_updates_preserve_each_parent_manifest(
+ repo: download.ModelRepo, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """一次下载多个分支时,各父目录只合并自身请求,且完整根清单同步更新。"""
+ files = _write_files(repo.local_dir(), {"a/nested/model.bin": b"a", "b/nested/model.bin": b"b"})
+ _write_marker(repo.local_dir(), files)
+ _write_marker(repo.local_dir() / "a", {"nested/model.bin": 1})
+ _write_marker(repo.local_dir() / "b", {"nested/model.bin": 1})
+
+ def update(source: str, target_repo: download.ModelRepo, patterns: list[str] | None) -> dict[str, int]:
+ """模拟分别补齐两个请求目录。"""
+ return _write_files(target_repo.local_dir(), {"a/nested/model.bin": b"new a", "b/nested/model.bin": b"new b"})
+
+ monkeypatch.setattr(download, "_snapshot_download", update)
+ download.download_model_files(repo, [repo.path("a/nested"), repo.path("b/nested")], source="modelscope")
+
+ assert download.verify_model_repo(repo).ready
+ assert download.model_path_exists(repo.path("a"))
+ assert download.model_path_exists(repo.path("b"))
+
+
+def test_local_subdirectory_download_preserves_complete_legacy_parent(
+ repo: download.ModelRepo, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """局部更新后可恢复通过本地必要文件检查的旧父标记,且不伪造新版清单。"""
+ _write_files(repo.local_dir(), {"config.json": b"{}", "model.safetensors": b"weights", "det/model.onnx": b"detector"})
+ (repo.local_dir() / download.MODEL_COMPLETE_MARKER).touch()
+
+ def update(source: str, target_repo: download.ModelRepo, patterns: list[str] | None) -> dict[str, int]:
+ """只确认检测器,父目录保留原有兼容语义。"""
+ return {"det/model.onnx": 8}
+
+ monkeypatch.setattr(download, "_snapshot_download", update)
+ download.download_model_files(repo, [repo.path("det")], source="modelscope")
+
+ assert download.model_path_exists(repo.path("det"))
+ assert download.verify_model_repo(repo).ready
+ assert (repo.local_dir() / download.MODEL_COMPLETE_MARKER).read_bytes() == b""
+
+
+def test_full_download_supersedes_stale_child_marker(repo: download.ModelRepo, monkeypatch: pytest.MonkeyPatch) -> None:
+ """整仓确认后,旧子目录标记不能阻止新清单在离线加载中生效。"""
+ _write_files(repo.local_dir(), {"nested/model.safetensors": b"old"})
+ child_marker = repo.local_dir() / "nested" / download.MODEL_COMPLETE_MARKER
+ child_marker.touch()
+
+ def update(source: str, target_repo: download.ModelRepo, patterns: list[str] | None) -> dict[str, int]:
+ """整仓下载同时确认子目录的新权重。"""
+ return _write_files(target_repo.local_dir(), {"nested/model.safetensors": b"new weights"})
+
+ monkeypatch.setattr(download, "_snapshot_download", update)
+ repo.ensure(source="modelscope")
+ resolve = Mock(side_effect=AssertionError("confirmed child paths must stay offline"))
+ monkeypatch.setattr(download, "resolve_model_source", resolve)
+
+ assert not child_marker.exists()
+ assert repo.path("nested").ensure() == repo.local_dir() / "nested"
+ assert download.verify_model_repo(repo).ready
+ resolve.assert_not_called()
+
+
[email protected]("source", ["huggingface", "modelscope"])
+def test_provider_repair_replaces_wrong_size_cached_payload(
+ repo: download.ModelRepo,
+ monkeypatch: pytest.MonkeyPatch,
+ source: Literal["huggingface", "modelscope"],
+) -> None:
+ """SDK 根据缓存跳过已有文件时,失效权重必须先移除,完整文件继续复用。"""
+ _write_files(repo.local_dir(), {"config.json": b"{}", "model.safetensors": b"short"})
+ expected = {"config.json": 2, "model.safetensors": 7}
+ api = Mock()
+ api.model_info.return_value = SimpleNamespace(
+ sha="commit-id",
+ siblings=[SimpleNamespace(rfilename=name, size=size) for name, size in expected.items()],
+ )
+ monkeypatch.setattr(download, "HfApi", Mock(return_value=api))
+ monkeypatch.setattr(download, "ms_model_files", Mock(return_value=expected))
+
+ def snapshot(repo_id: str, **kwargs: object) -> str:
+ """模拟增量下载只补齐缺失文件,保留已验证的配置。"""
+ assert repo.config.local_path().read_bytes() == b"{}"
+ assert not repo.weights.local_path().exists()
+ repo.weights.local_path().write_bytes(b"weights")
+ return str(repo.local_dir())
+
+ monkeypatch.setattr(download, "hf_snapshot_download", snapshot)
+ monkeypatch.setattr(download, "ms_snapshot_download", snapshot)
+
+ repo.ensure(source=source)
+
+ assert download.verify_model_repo(repo).ready
+ assert repo.weights.local_path().read_bytes() == b"weights"
+
+
[email protected]("source", ["huggingface", "modelscope"])
[email protected]("damage", ["deleted", "truncated", "intact"])
+def test_provider_manifest_checks_expected_payload_before_marking_complete(
+ repo: download.ModelRepo,
+ monkeypatch: pytest.MonkeyPatch,
+ source: Literal["huggingface", "modelscope"],
+ damage: str,
+) -> None:
+ """两种来源均按远程大小检查预期文件,配置或 README 不能掩盖权重未下载。"""
+ expected = {"config.json": 2, "README.md": 6, "model.safetensors": 7}
+ api = Mock()
+ api.model_info.return_value = SimpleNamespace(
+ sha="commit-id",
+ siblings=[SimpleNamespace(rfilename=name, size=size) for name, size in expected.items()],
+ )
+ monkeypatch.setattr(download, "HfApi", Mock(return_value=api))
+ monkeypatch.setattr(download, "ms_model_files", Mock(return_value=expected))
+
+ def snapshot(repo_id: str, **kwargs: object) -> str:
+ """模拟 SDK 返回成功但权重可能缺失或未完整落盘。"""
+ contents = {"config.json": b"{}", "README.md": b"readme"}
+ if damage != "deleted":
+ contents["model.safetensors"] = b"short" if damage == "truncated" else b"weights"
+ _write_files(repo.local_dir(), contents)
+ if source == "huggingface":
+ assert kwargs["revision"] == "commit-id"
+ return str(repo.local_dir())
+
+ monkeypatch.setattr(download, "hf_snapshot_download", snapshot)
+ monkeypatch.setattr(download, "ms_snapshot_download", snapshot)
+
+ if damage == "intact":
+ download.download_model_repo(repo, source=source)
+ assert download.verify_model_repo(repo).ready
+ assert json.loads((repo.local_dir() / download.MODEL_COMPLETE_MARKER).read_text())["files"] == expected
+ else:
+ with pytest.raises(FileNotFoundError, match="model.safetensors"):
+ download.download_model_repo(repo, source=source)
+ assert not (repo.local_dir() / download.MODEL_COMPLETE_MARKER).exists()
+
+
+def test_manifest_publication_failure_does_not_leave_completion_marker(
+ repo: download.ModelRepo, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """原子替换失败时不留下完成标记或临时清单,避免后续误判。"""
+
+ def snapshot(source: str, target_repo: download.ModelRepo, patterns: list[str] | None) -> dict[str, int]:
+ """先完成下载,使失败只发生在清单发布阶段。"""
+ return _write_files(target_repo.local_dir(), {"model.safetensors": b"weights"})
+
+ def fail_replace(path: Path, target: Path) -> NoReturn:
+ """模拟进程无法原子发布文件。"""
+ raise OSError("cannot publish completion manifest")
+
+ monkeypatch.setattr(download, "_snapshot_download", snapshot)
+ monkeypatch.setattr(Path, "replace", fail_replace)
+ with pytest.raises(OSError, match="cannot publish"):
+ repo.ensure(source="modelscope")
+
+ assert not download.verify_model_repo(repo).ready
+ assert sorted(path.name for path in repo.local_dir().iterdir()) == ["model.safetensors"]
diff --git a/tests/unittest/test_models_download_utils.py b/tests/unittest/test_models_download_utils.py
index 1275f3e2..5c457ba5 100644
--- a/tests/unittest/test_models_download_utils.py
+++ b/tests/unittest/test_models_download_utils.py
@@ -1,6 +1,8 @@
from __future__ import annotations
+import json
from pathlib import Path
+from types import SimpleNamespace
from typing import Any, NoReturn
import pytest
@@ -9,6 +11,11 @@ from mineru.model import download as models_download_utils
from mineru.model.registry import MODEL_COMPLETE_MARKER, MINERU_2_5_PRO_2605_1_2B, ModelRepo, model_path_exists
+def _write_marker(directory: Path, files: dict[str, int]) -> None:
+ """按持久化协议构造已确认清单,文件是否完整由各测试独立安排。"""
+ (directory / MODEL_COMPLETE_MARKER).write_text(json.dumps({"version": 1, "files": files}), encoding="utf-8")
+
+
def test_resolve_model_source_does_not_persist_env_auto(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(models_download_utils.get_config().model, "source", "auto")
monkeypatch.setattr(models_download_utils, "get_config_source", lambda path: "env")
@@ -53,14 +60,15 @@ def test_download_model_repo_uses_required_path_patterns(
monkeypatch.setattr(models_download_utils.get_config().model, "base_dir", str(tmp_path / "models"))
monkeypatch.setattr(models_download_utils, "resolve_model_source", lambda *args, **kwargs: "modelscope")
- def fake_snapshot_download(model_source: str, target_repo: ModelRepo, patterns: list[str] | None) -> str:
+ def fake_snapshot_download(model_source: str, target_repo: ModelRepo, patterns: list[str] | None) -> dict[str, int]:
+ """模拟 provider 确认局部仓库的预期权重文件。"""
captured["source"] = model_source
captured["repo"] = target_repo.name
captured["patterns"] = patterns
target = target_repo.local_dir() / "models/weights"
target.mkdir(parents=True)
(target / "model.bin").write_text("x", encoding="utf-8")
- return str(target_repo.local_dir())
+ return {"models/weights/model.bin": 1}
monkeypatch.setattr(models_download_utils, "_snapshot_download", fake_snapshot_download)
@@ -90,14 +98,15 @@ def test_download_model_repo_keeps_full_download_as_default(
monkeypatch.setattr(models_download_utils.get_config().model, "base_dir", str(tmp_path / "models"))
monkeypatch.setattr(models_download_utils, "resolve_model_source", lambda *args, **kwargs: "huggingface")
- def fake_snapshot_download(model_source: str, target_repo: ModelRepo, patterns: list[str] | None) -> str:
+ def fake_snapshot_download(model_source: str, target_repo: ModelRepo, patterns: list[str] | None) -> dict[str, int]:
+ """模拟 provider 返回整仓文件清单。"""
captured["source"] = model_source
captured["repo"] = target_repo.name
captured["patterns"] = patterns
target = target_repo.local_dir() / "config.json"
target.parent.mkdir(parents=True)
target.write_text("{}", encoding="utf-8")
- return str(target_repo.local_dir())
+ return {"config.json": 2}
monkeypatch.setattr(models_download_utils, "_snapshot_download", fake_snapshot_download)
@@ -109,7 +118,10 @@ def test_download_model_repo_keeps_full_download_as_default(
"repo": "full-repo",
"patterns": None,
}
- assert (repo.local_dir() / MODEL_COMPLETE_MARKER).read_bytes() == b""
+ assert json.loads((repo.local_dir() / MODEL_COMPLETE_MARKER).read_text()) == {
+ "version": 1,
+ "files": {"config.json": 2},
+ }
def test_vlm_repo_verify_requires_root_completion_marker(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
@@ -126,6 +138,10 @@ def test_vlm_repo_verify_requires_root_completion_marker(monkeypatch: pytest.Mon
(repo_dir / MODEL_COMPLETE_MARKER).touch()
+ assert models_download_utils.verify_model_repo(MINERU_2_5_PRO_2605_1_2B).ready is False
+ (repo_dir / "model.safetensors").write_bytes(b"weights")
+ _write_marker(repo_dir, {"model.safetensors": 7})
+
ready = models_download_utils.verify_model_repo(MINERU_2_5_PRO_2605_1_2B)
assert ready.ready is True
@@ -149,12 +165,14 @@ def test_model_path_exists_requires_completion_marker_for_directory(
(weights_dir / "model.bin").write_bytes(b"weights")
assert model_path_exists(repo.weights) is False
- (weights_dir / MODEL_COMPLETE_MARKER).touch()
+ _write_marker(weights_dir, {"model.bin": 7})
assert model_path_exists(repo.weights) is True
nested_weights = repo.weights.path("det")
nested_weights.local_path().mkdir()
(nested_weights.local_path() / "det.bin").write_bytes(b"weights")
+ assert model_path_exists(nested_weights) is False
+ _write_marker(weights_dir, {"model.bin": 7, "det/det.bin": 7})
assert model_path_exists(nested_weights) is True
config_path = repo.config.local_path()
@@ -176,7 +194,7 @@ def test_model_path_ensure_reuses_completed_directory_without_resolving_source(
weights_dir = repo.weights.local_path()
weights_dir.mkdir(parents=True)
(weights_dir / "model.bin").write_bytes(b"weights")
- (weights_dir / MODEL_COMPLETE_MARKER).touch()
+ _write_marker(weights_dir, {"model.bin": 7})
def fail_resolve(*args: object, **kwargs: object) -> NoReturn:
raise AssertionError("ready model paths must not resolve a remote source")
@@ -200,7 +218,7 @@ def test_model_repo_ensure_reuses_completed_full_repo_without_resolving_source(
repo.local_dir().mkdir(parents=True)
repo.config.local_path().write_text("{}", encoding="utf-8")
repo.weights.local_path().write_bytes(b"weights")
- (repo.local_dir() / MODEL_COMPLETE_MARKER).touch()
+ _write_marker(repo.local_dir(), {"config.json": 2, "model.bin": 7})
def fail_resolve(*args: object, **kwargs: object) -> NoReturn:
raise AssertionError("ready model repos must not resolve a remote source")
@@ -226,13 +244,14 @@ def test_model_path_ensure_repairs_unmarked_directory_and_writes_marker(
(weights_dir / "existing.bin").write_bytes(b"existing")
calls = 0
- def fake_snapshot_download(model_source: str, target_repo: ModelRepo, patterns: list[str] | None) -> str:
+ def fake_snapshot_download(model_source: str, target_repo: ModelRepo, patterns: list[str] | None) -> dict[str, int]:
+ """模拟增量补齐并返回完整请求目录的文件清单。"""
nonlocal calls
calls += 1
assert model_source == "modelscope"
assert patterns == ["models/weights", "models/weights/*"]
(weights_dir / "missing.bin").write_bytes(b"repaired")
- return str(target_repo.local_dir())
+ return {"models/weights/existing.bin": 8, "models/weights/missing.bin": 8}
monkeypatch.setattr(models_download_utils, "_snapshot_download", fake_snapshot_download)
@@ -358,20 +377,25 @@ def test_nested_download_success_restores_parent_directory_marker(
nested_weights = repo.weights.path("det")
nested_weights.local_path().mkdir(parents=True)
parent_marker = repo.weights.local_path() / MODEL_COMPLETE_MARKER
- parent_marker.touch()
+ (nested_weights.local_path() / "det.bin").write_bytes(b"old")
+ (repo.weights.local_path() / "rec.bin").write_bytes(b"recognizer")
+ _write_marker(repo.weights.local_path(), {"det/det.bin": 3, "rec.bin": 10})
- def fake_snapshot_download(model_source: str, target_repo: ModelRepo, patterns: list[str] | None) -> str:
+ def fake_snapshot_download(model_source: str, target_repo: ModelRepo, patterns: list[str] | None) -> dict[str, int]:
+ """模拟只更新检测器,父清单中的识别器仍必须保留。"""
assert model_source == "modelscope"
assert target_repo is repo
assert patterns == ["models/weights/det", "models/weights/det/*"]
(nested_weights.local_path() / "det.bin").write_bytes(b"weights")
- return str(repo.local_dir())
+ return {"models/weights/det/det.bin": 7}
monkeypatch.setattr(models_download_utils, "_snapshot_download", fake_snapshot_download)
models_download_utils.download_model_files(repo, [nested_weights], source="modelscope")
assert parent_marker.is_file()
+ assert json.loads(parent_marker.read_text())["files"] == {"det/det.bin": 7, "rec.bin": 10}
+ assert model_path_exists(repo.weights)
assert (nested_weights.local_path() / MODEL_COMPLETE_MARKER).is_file()
@@ -389,14 +413,15 @@ def test_download_does_not_mark_metadata_only_directory(
monkeypatch.setattr(models_download_utils.get_config().model, "base_dir", str(tmp_path / "models"))
weights_dir = repo.weights.local_path()
- def fake_snapshot_download(model_source: str, target_repo: ModelRepo, patterns: list[str] | None) -> str:
+ def fake_snapshot_download(model_source: str, target_repo: ModelRepo, patterns: list[str] | None) -> dict[str, int]:
+ """模拟下载器只有元数据、没有任何已确认模型文件。"""
assert model_source == "modelscope"
assert target_repo is repo
assert patterns == ["models/weights", "models/weights/*"]
(weights_dir / ".cache/huggingface").mkdir(parents=True)
(weights_dir / ".cache/huggingface/metadata").write_text("metadata", encoding="utf-8")
(weights_dir / ".msc").write_text("metadata", encoding="utf-8")
- return str(repo.local_dir())
+ return {}
monkeypatch.setattr(models_download_utils, "_snapshot_download", fake_snapshot_download)
@@ -419,9 +444,17 @@ def test_huggingface_snapshot_rejects_missing_expected_files(
monkeypatch.setattr(models_download_utils.get_config().model, "base_dir", str(tmp_path / "models"))
class FakeApi:
- def list_repo_files(self, repo_id: str) -> list[str]:
+ def model_info(self, repo_id: str, *, files_metadata: bool) -> SimpleNamespace:
+ """提供与下载提交一致的远程文件名和大小。"""
assert repo_id == "owner/repo"
- return ["models/weights/config.json", "models/weights/model.bin"]
+ assert files_metadata
+ return SimpleNamespace(
+ sha="commit",
+ siblings=[
+ SimpleNamespace(rfilename="models/weights/config.json", size=2),
+ SimpleNamespace(rfilename="models/weights/model.bin", size=7),
+ ],
+ )
monkeypatch.setattr(models_download_utils, "HfApi", FakeApi)