Peach: Local-first video metadata scraper and media manager.
chore(tasks): 保存长时任务续跑实现进度
longmeidao5a33935a7db4
diff --git a/README.en.md b/README.en.mdindex dce33aa25..f786a03be 100644--- a/README.en.md+++ b/README.en.md@@ -45,6 +45,8 @@ https://github.com/user-attachments/assets/a5610874-e611-41bb-be22-7d99f0d64b62Watch history, favorites and settings stay on your computer. Scanning leaves original files untouched; manual organization offers a preview and an undo for the last batch.+Scanning, follow updates and new-release metadata collection save their progress and continue unfinished items after the app restarts. See [task recovery](docs/OPERATIONS.md#长时任务断点续跑) for recovery boundaries.+The combined "Send version statistics and error diagnostics (recommended)" option is checked by default. Clear it during installation, first setup or in settings to disable both. Saved opt-out choices are preserved. While enabled, it sends the listed technical fields to Sentry; full logs, diagnostic ZIPs, media details and watch history are excluded. See [remote diagnostics](docs/DIAGNOSTICS.md#可选远程诊断).Sharing library and taste details for population research is a separate option, off by default. Review the fields and full local preview in settings, then explicitly consent to weekly snapshots of works, tags, ratings and derived preferences. Identity fields and raw browsing records are excluded, but detailed preferences cannot be guaranteed anonymous.diff --git a/README.md b/README.mdindex e4619cab1..fe42af760 100644--- a/README.md+++ b/README.md@@ -45,6 +45,8 @@ https://github.com/user-attachments/assets/a5610874-e611-41bb-be22-7d99f0d64b62观看记录、收藏和设置保存在本机。扫描不动原文件;手动整理先预览,可撤回上一批。+扫描、关注更新和新作资料采集会保存进度,应用重启后继续处理未完成的项目。恢复范围见[断点续跑](docs/OPERATIONS.md#长时任务断点续跑)。+「发送版本统计与错误诊断(推荐)」默认勾选,可在安装、首次设置或设置页取消。两类技术事件由同一开关控制,已保存的关闭选择会保留。启用后向 Sentry 发送页面列出的技术字段,不自动上传完整日志、诊断 ZIP、媒体信息或观看记录。见[远程诊断说明](docs/DIAGNOSTICS.md#可选远程诊断)。馆藏与口味群体研究另设独立选项,默认关闭。先在设置页查看字段和完整数据预览,再明确同意,才会每周发送作品、标签、评分与派生偏好明细;不含身份标识或逐条浏览记录。明细可以反映个人偏好,不能保证匿名。diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.mdindex 64089e760..17ea7d28c 100644--- a/docs/OPERATIONS.md+++ b/docs/OPERATIONS.md@@ -93,6 +93,27 @@ Windows 测试包默认使用 `%LOCALAPPDATA%\Peach\peach-data`。首次设置- 给了根目录时,它必须在某个声明的目录之内(macOS 上是挂载点之内),否则会被拒绝,因为登记进去的路径无法换算回本机路径。- 扫描跳过 `#recycle` 和名称以 `@` 开头的 NAS 系统目录;同名前缀的普通文件仍按文件类型处理。+### 长时任务断点续跑++后台任务把任务标识、处理范围、完成集合和计数保存在本机。服务重启时接回同一条任务记录,继续未完成的部分。用户暂停的任务保持暂停;明确停止会取消恢复。失败、限流或未取得的结果保留原有重试与冷却规则,不作为成功项目跳过。++| 任务 | 保存与恢复的单位 |+| --- | --- |+| 扫描文件 | 来源根、待遍历目录和目录内的已提交文件批次;保留文件数、字节数与字幕配对资料 |+| 读取配套资料、采集元数据与预览 | 已读取的本地证据、已落盘候选与已登记预览;保留完成影片和问题记录 |+| 拉取关注更新、历史回抓 | 固定来源集合、成功来源与已提交页码;从头回抓任务沿用本次保存的页码 |+| 新作订阅、想要资料、发现卡资料 | 已完成来源或番号、已保存资料与封面;发现卡同时保留已消耗请求与流量预算 |+| 后继任务、下载后处理 | 原有持久队列与同一条任务记录;批量资料与样张保存原始番号集合和成功项目 |+| 时间轴缩略图、媒体修复 | 已核验产物、带文件身份的检查结论与累计计数 |+| 链接检查、关注来源查找、口味读取 | 已完成链接、检索行或浏览器资料;保留结果和处理范围 |+| 文件整理与回滚 | 固定整理计划、同一批次的逐项日志与待回滚记录 |+| 资源检查、资源清理、链接清理 | 已完成来源、已处理记录与清理回执;删除前仍复核当前状态 |+| amane 桥环境准备 | `uv` 缓存和已安装依赖;中断的安装命令重新核对锁文件 |++检查点在结果成功保存后提交。中断时尚未提交的文件批次、来源页、单部影片或写入事务可以重做,已提交结果继续复用。外部请求本身无法从半个响应恢复。检查点损坏或结果文件未取得时会报告问题;此前没有保存的进度无法追溯恢复。++命令行扫描中断后,用相同来源和根目录再次运行即可接回断点;完整扫描结束会清除该断点,下一次扫描重新检查目录。`scripts/scrape_codes.py` 默认续跑同一批次,候选 CSV 和旁边的 `.resume.json` 一起保留;指定 `--out` 时沿用原输出路径。批次参数必须一致,`--no-resume` 用于开始新批次。其它探测、拼图和封面批处理继续按已保存的测量值、产物或成功日志选取待办。+### 媒体来源状态服务在后台读取挂载目录的首个条目,空目录可读也算在线。正常来源每 600 秒检测,异常来源每 60 秒检测;首次失败隔一秒复查,连续两次失败才确认故障。提示分别说明没有权限读取、目录不存在、目标不是目录或目录读取失败。diff --git a/docs/REUSE.md b/docs/REUSE.mdindex 24eb6281e..9b301b595 100644--- a/docs/REUSE.md+++ b/docs/REUSE.md@@ -664,6 +664,10 @@ Python 复用 MIT 许可的 [Ruff](https://docs.astral.sh/ruff/) 0.17.0,版本## 馆藏处理控制与耗时预估+长时任务续跑复用 `job_recovery` 的本机原子记录、`BackgroundJob.checkpoint` 与 `TaskRunHandle.checkpoint`。主任务保留原 `job_id`、`run_id`、开始时间、处理范围和领域检查点;后继重排保留原进度。扫描复用现有 `os.scandir`、SQLite 批次提交、`filelock` 和 `fsutil.atomic_write_text`,保存遍历栈、文件名游标及字幕配对资料。本地资料按影片保存小份证据边车,候选成功落盘后才提交元数据完成集合;整理逐项原子保存同一批次的日志。`scrape_codes.py` 复用流式 CSV 与原始来源快照,续跑只保留已确认完成的番号,未完成番号可以重放。++当前树、扫描与采集的 Git 历史,以及现有任务队列均已核对;不引入新的调度或工作流依赖。项目已固定的 APScheduler 3.11.3(MIT)负责派发,其持久 job store 记录调度任务,不能保存函数执行到哪一条影片或来源页,见[上游调度说明](https://apscheduler.readthedocs.io/en/3.x/userguide.html#adding-jobs)。结果替换沿用 Python 3.14 标准库 [`os.replace`](https://docs.python.org/3.14/library/os.html#os.replace)(PSF),锁沿用项目固定的 `filelock`,没有新增运行时体积或平台要求。隔离实证覆盖目录与目录内批次中断、关注回抓游标、固定来源集合、原任务标识与计数、候选 CSV 保留及同批整理日志;不读取真实媒体、不访问外部采集来源,也不写真实 ledger。+扫描、探测、资料请求和九宫格复用 `process_library`、`scan_location`、`SourceTransport`、`frame_capture` 与已有影片检查点;任务恢复沿用 `BackgroundJob`、`job_recovery` 和 `task_run`,用户暂停意图单独持久保存。当前树及 `library_processing.py` 的 Git 历史已核对,安装暂停只用于安装事务,不能直接代表用户暂停。Python 标准库 `threading.Event` 与 `contextvars.ContextVar` 提供等待唤醒和当前任务隔离,沿用项目 Python 版本,跨 Windows 与 macOS,无额外依赖体积;许可证为 PSF,官方文档与实现见 [threading](https://docs.python.org/3/library/threading.html) 和 [contextvars](https://docs.python.org/3/library/contextvars.html)。现有 APScheduler 3.11.3 用于定时派发,不能暂停已进入媒体读取的函数,因此不承担安全停点。隔离测试使用临时账本、暂停线程与模拟时钟验证安全停点、跨重启意图、低占用间隔和暂停预算补偿,不读取真实媒体、不发外部请求。`processing_control` 保留 Peach 的阶段、模式与估时策略:启动仅汇总账本文件数和大小,优先使用本机同类完成耗时;未知新增明确不在初始范围内。整趟剩余时间合计当前阶段和待执行阶段;当前阶段采用最近 20 次完成增量,后续阶段使用保守粗估。无番号、无 NFO 的文件完成同目录检查后不计入外部采集数量;任务停滞时读接口撤下旧估时。处理强度为低占用、标准和高强度:低占用采用单线程 FFmpeg 与较大单项间隔,标准采用两个解码、滤镜和编码线程与短间隔,高强度采用工具自动线程且没有额外间隔。amane 低占用逐站查询,其余沿用现有调度;各档均遵守来源限流,单资产流水线与暂停检查点共用。强度只作用于当前任务,历史耗时按档位系数归一,初始与切换时的范围属于粗估,实际样本继续校准。持久状态不修改 ledger schema,不改变生产入口。diff --git a/scripts/scrape_codes.py b/scripts/scrape_codes.pyindex 77bda96f6..249223b52 100644--- a/scripts/scrape_codes.py+++ b/scripts/scrape_codes.py@@ -12,6 +12,7 @@ from __future__ import annotationsimport argparseimport csvimport hashlib+from contextlib import ExitStackimport jsonimport osimport random@@ -50,7 +51,8 @@ from peach.metadata import (from peach.metadata_policy import (PEACH_FIELDS, POLICY_VERSION, SOURCE_SPECS, field_rank,preferred_tag_source, sort_candidates)from peach.platform import system_volume-from peach.review_csv import ENCODING, write_rows+from peach.review_csv import ENCODING, read_rows, write_rows+from peach.fsutil import atomic_write_textfrom peach.sources import Sessionfrom peach.sources.seesaa import SEESAA, WIKI_SOURCES as WIKI_SITES, WikiPages@@ -510,6 +512,8 @@ def build_parser() -> argparse.ArgumentParser:help="系统盘最低可用 GiB;运行中每隔一段时间复查")parser.add_argument("--disk-check-secs", type=float, default=20.0)parser.add_argument("--refresh", action="store_true", help="ignore reusable raw snapshots")+ parser.add_argument('--resume', action=argparse.BooleanOptionalAction, default=True,+ help='复用同一批次已保存的成功结果;--no-resume 开始新批次')parser.add_argument("--wiki-pages-file", type=Path, help="预取 Wiki 作品目录页,每行一个 URL")parser.add_argument("--wiki-max-requests", type=int, default=80,help="每个 Wiki 本批 HTTP 请求上限;每个请求最多 4 MiB,同主机几个 Wiki 合计间隔至少 2 秒")@@ -889,15 +893,16 @@ def main(argv: list[str] | None = None, *, provider=None) -> int:"""parser = build_parser()args = parser.parse_args(argv)- with cli_run("scrape-codes", args.db, label="番号资料刮削") as handle:- code = _scrape(parser, args, handle, provider=provider)+ with cli_run("scrape-codes", args.db, label="番号资料刮削") as handle, ExitStack() as resources:+ resources.callback(close_log)+ code = _scrape(parser, args, handle, provider=provider, resources=resources)if code:handle.finish("cancelled", summary={"exit_code": code},error=f"提前收工,退出码 {code}")return code-def _scrape(parser, args, handle, *, provider=None) -> int:+def _scrape(parser, args, handle, *, provider=None, resources) -> int:sources = Noneif args.sources is not None:try:@@ -905,7 +910,20 @@ def _scrape(parser, args, handle, *, provider=None) -> int:except ValueError as error:parser.error(str(error))profile = CUSTOM_PROFILE if sources is not None else args.profile- output = args.out or _default_output()+ scope = dict(db=str(args.db.resolve()), profile=profile, sources=sources,+ english_title_only=args.english_title_only, refresh=args.refresh, limit=args.limit,+ codes=_requested_codes(args.codes_file) if args.codes_file else None,+ wiki_pages=args.wiki_pages_file.read_text(encoding=ENCODING) if args.wiki_pages_file else None)+ signature = hashlib.sha256(json.dumps(scope, sort_keys=True).encode()).hexdigest()+ active_path = args.log_dir / 'scrape-codes-active.json'+ active = json.loads(active_path.read_text(encoding='utf-8')) if args.resume and active_path.is_file() else {}+ output = args.out or (Path(active['output']) if active.get('signature') == signature+ and active.get('status') == 'active' else _default_output())+ checkpoint_path = output.with_suffix('.resume.json')+ saved = json.loads(checkpoint_path.read_text(encoding='utf-8')) if args.resume and checkpoint_path.is_file() else {}+ completed = set(saved.get('completed') or [])+ if completed and not output.is_file():+ parser.error('续跑候选文件未取得,请恢复原文件或使用 --no-resume')error_name = output.name.replace("metadata-field-candidates-", "metadata-source-errors-", 1)if error_name == output.name:error_name = output.stem + "-errors.csv"@@ -926,27 +944,52 @@ def _scrape(parser, args, handle, *, provider=None) -> int:# 健康表覆盖这一趟可能问到的全部来源:点名的就是那几家,走链的是所有链的并集。covered = sources or ((WIKI_SOURCE,) if profile == WIKI_PROFILE else CHAIN_SOURCES)health = _health_rows(covered, profile)+ health.update(saved.get('health') or {})unmapped_genres: dict[tuple[str, str], list] = {}+ unmapped_genres.update({tuple(key): value for key, value in saved.get('unmapped', [])})adapter = _build_adapter(args, covered, provider)+ resources.callback(adapter.close)connection = open_readonly(args.db)+ resources.callback(connection.close)# 用户在复核页收录过的 genre 这一批就当已知词,不再作为未收录回来问一遍。genre_decisions = load_genre_decisions(connection)codes = _load_codes(connection, args, parser)+ if saved and saved.get('signature') != signature:+ connection.close()+ close_log()+ parser.error('续跑参数与原批次不一致,请使用原参数或 --no-resume')+ if saved.get('codes'):+ by_code = {normalise_code_key(row[0]): row for row in codes}+ codes = [by_code[code] for code in saved['codes'] if code in by_code]+ def save_progress(status='active'):+ record = dict(signature=signature, output=str(output.resolve()), status=status,+ completed=sorted(completed), codes=[normalise_code_key(row[0]) for row in codes],+ health=health, unmapped=list(unmapped_genres.items()))+ atomic_write_text(checkpoint_path, json.dumps(record, ensure_ascii=False))+ atomic_write_text(active_path, json.dumps(record, ensure_ascii=False))+ previous_rows = read_rows(output) if saved and output.is_file() else []+ previous_rows = [row for row in previous_rows if row['query'] in completed]+ write_rows(output, FIELDS, previous_rows, atomic=True)+ previous_errors = read_rows(errors_path) if saved and errors_path.is_file() else []+ write_rows(errors_path, ERROR_FIELDS, previous_errors, atomic=True)+ save_progress()log(f"字段候选批次:profile {profile},番号 {len(codes)},来源 {','.join(covered)};只读查询,不写 ledger")throttle = _Throttle(health)- groups_written = errors_written = 0+ groups_written, errors_written = len(previous_rows), len(previous_errors)stopped: JobPolicyError | None = None# 流式写:两个文件同时开着,行在长循环里边跑边落盘,中途还有 guard.check()# 打断点。write_rows 收的是完整行集合,改用它等于「跑完才落盘」,被打断就全丢。- with output.open("w", encoding=ENCODING, newline="") as candidate_handle, \- errors_path.open("w", encoding=ENCODING, newline="") as error_handle:+ with output.open("a", encoding='utf-8', newline="") as candidate_handle, \+ errors_path.open("a", encoding='utf-8', newline="") as error_handle:candidate_writer = csv.DictWriter(candidate_handle, fieldnames=FIELDS)error_writer = csv.DictWriter(error_handle, fieldnames=ERROR_FIELDS)- candidate_writer.writeheader(); error_writer.writeheader()for index, row in enumerate(codes, 1):code, size_gb, videos = row[0], row[1], row[2]+ if normalise_code_key(code) in completed:+ handle.progress(index, len(codes), normalise_code_key(code))+ continuetry:guard.check()except JobPolicyError as error:@@ -979,13 +1022,16 @@ def _scrape(parser, args, handle, *, provider=None) -> int:})groups_written += 1candidate_handle.flush()+ error_handle.flush()+ if by_field and not errors:+ completed.add(query)+ save_progress()_write_health(health_path, health)_write_unmapped(unmapped_path, unmapped_genres)if index % 25 == 0:log(f"{index}/{len(codes)}:已落 {groups_written} 个字段组,错误 {errors_written}")handle.progress(index, len(codes), f"{query}")- connection.close()- adapter.close()+ save_progress('complete' if len(completed) == len(codes) else 'active')_write_health(health_path, health)_write_unmapped(unmapped_path, unmapped_genres)log(f"完成:{groups_written} 个字段候选组 → {output}")diff --git a/src/peach/download_postprocess.py b/src/peach/download_postprocess.pyindex 3ab53c1be..35d5186c0 100644--- a/src/peach/download_postprocess.py+++ b/src/peach/download_postprocess.py@@ -2,6 +2,7 @@from __future__ import annotationsimport time+import jsonfrom pathlib import Pathfrom . import organize, settings_file@@ -101,10 +102,15 @@ def _collect(contract, main: int, handle) -> dict:if not config.configured or Path(contract.db_path).resolve() != (config.directory("database") / "ledger.db").resolve():raise ValueError("当前馆藏与媒体配置不一致,不能采集下载资料")+ state_file = config.directory('state') / 'download-processing' / f'{main}.json'+ try:+ previous = json.loads(state_file.read_text(encoding='utf-8'))+ except FileNotFoundError:+ previous = Nonestate = process_library(config, contract.db_path, contract.candidate_root, contract.cover_root,retry_ids=[main], stage=COLLECT_STAGE, database=contract.database,- state_file=config.directory('state') / 'download-processing' / f'{main}.json',+ state_file=state_file, resume_state=previous,report=lambda state: handle.progress(current=int(state.get("checked") or 0), total=int(state.get("total") or 0),label=str(state.get("stage") or TASK_LABEL)))diff --git a/src/peach/feed_followup.py b/src/peach/feed_followup.pyindex fd6043f12..898069035 100644--- a/src/peach/feed_followup.py+++ b/src/peach/feed_followup.py@@ -289,8 +289,14 @@ def run(contract, key: str, handle) -> dict:codes = parse_key(key)provider = None- fetched = missed = covers = unreachable = 0+ saved = getattr(handle, 'saved', {})+ saved = saved if isinstance(saved, dict) else {}+ completed = set(saved.get('completed_codes') or [])+ fetched, covers = saved.get('fetched', 0), saved.get('covers', 0)+ missed = unreachable = 0for index, code in enumerate(codes):+ if code in completed:+ continuehandle.progress(current=index, total=len(codes),label=f"{TASK_LABEL}:{code}", throttle=0)with contract.database.read_connection() as connection:@@ -300,10 +306,12 @@ def run(contract, key: str, handle) -> dict:if row is None:# 壳被删了,或者这个番号这中间已经入库。两种都不该再取一遍。continue+ metadata_ok = row['scraped_at'] is not None and row['scrape_error'] is Noneif row["scraped_at"] is None or row["scrape_error"] is not None:provider = provider or LibraryMetadataProvider(contract.follow_secrets_root)if _scrape_fields(contract, provider, row):fetched += 1+ metadata_ok = Trueelse:missed += 1# 封面地址在取资料那一步才写上壳,所以取完再读一遍,不用循环开头那份。@@ -311,7 +319,10 @@ def run(contract, key: str, handle) -> dict:cover_url = connection.execute("SELECT cover_url FROM feed_discovery WHERE id=?", (int(row["id"]),)).fetchone()try:- covers += ensure_cover(contract, code, cover_url[0] if cover_url else None)+ covered = ensure_cover(contract, code, cover_url[0] if cover_url else None)+ covers += covered+ if metadata_ok and covered:+ completed.add(code)except (CoverConnectError, httpx.TransportError):unreachable += 1except Exception: # noqa: BLE001 - 取不到封面这一部照样有资料,下一轮再试@@ -320,6 +331,7 @@ def run(contract, key: str, handle) -> dict:with contract.database.write_transaction() as connection:connection.execute("UPDATE feed_discovery SET scraped_at=? WHERE id=?",(feeds.stamp(), int(row["id"])))+ handle.checkpoint(completed_codes=sorted(completed), fetched=fetched, covers=covers)handle.progress(current=len(codes), total=len(codes), label=TASK_LABEL, throttle=0)contract.cache_bust()summary = {"total": len(codes), "ok": fetched, "miss": missed, "covers": covers}diff --git a/src/peach/jav_discovery_metadata.py b/src/peach/jav_discovery_metadata.pyindex 6eda5034f..08857dd33 100644--- a/src/peach/jav_discovery_metadata.py+++ b/src/peach/jav_discovery_metadata.py@@ -49,6 +49,21 @@ class DiscoveryMetadata:self.last_job = Noneself.records = OrderedDict()self._load()+ self.job_path = self.path.with_suffix('.job.json')+ try:+ saved = json.loads(self.job_path.read_text(encoding='utf-8'))+ except (OSError, ValueError):+ saved = {}+ if saved.get('status') == 'running':+ self.ensure(saved.get('codes') or [], _resume=saved)+ elif saved.get('job_id'):+ self.last_job = saved++ def _save_job(self):+ with self.lock:+ state = self.job.snapshot()+ if state:+ atomic_write_text(self.job_path, json.dumps(state, ensure_ascii=False))def _load(self):try:@@ -104,8 +119,13 @@ class DiscoveryMetadata:return movie, {"metadata_state": state, "metadata_fields": fields,"metadata_message": record.get("error", ""), "metadata_checked_at": record.get("checked_at")}- def ensure(self, codes, *, refresh=False, job_id=None):- snapshot = self.job.snapshot() or self.last_job+ def ensure(self, codes, *, refresh=False, job_id=None, _resume=None):+ snapshot = None if _resume else self.job.snapshot() or self.last_job+ if _resume:+ refresh = bool(_resume.get('refresh'))+ successful = set((_resume or {}).get('completed_codes') or [])+ attempted = set((_resume or {}).get('attempted_codes') or [])+ failures = set((_resume or {}).get('failed_codes') or [])# 同一页的轮询只观察固定集合,完成后不会自行领下一批。if snapshot and (snapshot["status"] == "running" or(not refresh and snapshot.get('status') != 'complete')):@@ -114,19 +134,25 @@ class DiscoveryMetadata:codes = snapshot['codes']codes = list(dict.fromkeys(normalise_code_key(code) for code in codes if code))[:MAX_ITEMS]with self.lock:- pending = [code for code in codes if self.records.get(code, {}).get('retry_after', 0) <= self.clock()+ pending = [code for code in codes if code not in successful and self.records.get(code, {}).get('retry_after', 0) <= self.clock()and (refresh or not cache_fresh(self.records.get(code, {}).get('checked_at', -1),self.records.get(code, {}).get('movie', {}).get('release_date'), self.clock()))]if snapshot and not pending:return snapshotbudget = PublicRequestBudget()- initial = {"total": len(codes), "completed": len(codes) - len(pending), "failed": 0,- "requests": 0, "bytes": 0, "stop_reason": "", "codes": codes,+ budget.requests = (_resume or {}).get('requests', 0)+ budget.bytes = (_resume or {}).get('bytes', 0)+ previous_elapsed = (_resume or {}).get('active_seconds', 0)+ budget.max_seconds = max(0, budget.max_seconds - previous_elapsed)+ started = time.monotonic()+ initial = {"total": len(codes), "completed": (_resume or {}).get('completed', len(codes) - len(pending)),+ "failed": len(failures), 'refresh': refresh,+ "requests": budget.requests, "bytes": budget.bytes, "stop_reason": "", "codes": codes,"budget": {"max_items": MAX_ITEMS, "max_requests": budget.max_requests,"max_bytes": budget.max_bytes, "max_seconds": budget.max_seconds}}def work(current_id):- completed, failed = initial["completed"], 0+ completed = initial['completed']stop_reason = ""with public_cache_scope(self.path.parent / 'jav-public'), self.api.request_budget(budget):for code in pending:@@ -137,27 +163,43 @@ class DiscoveryMetadata:self.remember(code, movie, checked_at=cache_checked_at('movie:' + code, exact=True))if cache_error('movie:' + code):raise PublicSourceUnavailable(cache_error('movie:' + code))- completed += 1+ successful.add(code)+ failures.discard(code)except PublicBudgetExceeded:stop_reason = budget.stop_reasonbreakexcept PublicSourceUnavailable as error:self.remember(code, error=str(error))- failed += 1- completed += 1+ failures.add(code)if self.api.paused_until > self.api.clock():stop_reason = "source_cooldown"- if not self.job.update(current_id, completed=completed, failed=failed,- requests=budget.requests, bytes=budget.bytes, batch=(completed + 9) // 10):+ if code not in attempted:+ completed += 1+ attempted.add(code)+ if not self.job.update(current_id, completed=completed, failed=len(failures),+ completed_codes=sorted(successful), attempted_codes=sorted(attempted), failed_codes=sorted(failures),+ active_seconds=previous_elapsed + time.monotonic() - started,+ requests=budget.requests, bytes=budget.bytes, batch=(completed + 9) // 10):return+ self._save_job()if stop_reason:breakself.job.update(current_id, status="stopped" if stop_reason else "complete", stop_reason=stop_reason,- completed=completed, failed=failed, requests=budget.requests, bytes=budget.bytes)+ completed=completed, failed=len(failures), requests=budget.requests, bytes=budget.bytes)+ self._save_job()- return self.job.start(work, initial=initial, restart=bool(snapshot))+ from .job_recovery import current_job_id+ token = current_job_id.set(_resume.get('job_id')) if _resume else None+ try:+ result = self.job.start(work, initial=initial, restart=bool(snapshot))+ self._save_job()+ return result+ finally:+ if token is not None:+ current_job_id.reset(token)def stop(self):+ self._save_job()self.job.stop()def cancel(self, job_id=None):@@ -167,6 +209,8 @@ class DiscoveryMetadata:self.job.stop()snapshot.update(status="stopped", stop_reason="cancelled")self.last_job = snapshot+ with self.lock:+ atomic_write_text(self.job_path, json.dumps(snapshot, ensure_ascii=False))return snapshotdiff --git a/src/peach/job_recovery.py b/src/peach/job_recovery.pyindex 787a09ce8..f19c755e3 100644--- a/src/peach/job_recovery.py+++ b/src/peach/job_recovery.py@@ -32,9 +32,21 @@ def read(db: Path, key: str) -> dict:def save(db: Path, key: str, job_id: str, request: dict, *, run_id=None, waiting=False) -> None:+ previous = read(db, key)atomic_write_text(_path(db, key), json.dumps({"job_id": job_id, "pid": -1 if waiting else os.getpid(), "status": "active",- "request": request, 'run_id': run_id}, ensure_ascii=False))+ "request": request, 'run_id': run_id,+ 'checkpoint': previous.get('checkpoint', {}) if previous.get('job_id') == job_id else {},+ }, ensure_ascii=False))+++def checkpoint(db: Path, key: str, job_id: str, fields: dict) -> None:+ """结果落盘后保存进度;只更新同一任务的恢复记录。"""+ with _mutex(state_for(db)):+ data = read(db, key)+ if data.get('job_id') == job_id and data.get('status') == 'active':+ data['checkpoint'] = {**data.get('checkpoint', {}), **fields}+ atomic_write_text(_path(db, key), json.dumps(data, ensure_ascii=False))def finish(db: Path, key: str, job_id: str, status: str) -> None:diff --git a/src/peach/jobs.py b/src/peach/jobs.pyindex 9855bcb33..4aa2ad973 100644--- a/src/peach/jobs.py+++ b/src/peach/jobs.py@@ -321,6 +321,12 @@ class BackgroundJob:#: 摘 run id 与把那一行写成结束是同一件事:只摘不写完就放手的话,另一个线程#: 看见 id 没了以为已经收过尾,紧接着开的下一轮会撞上那一行还没释放的互斥键。self._close_lock = threading.Lock()+ if (db := self._database_path()) is not None:+ from .job_recovery import read+ previous = read(db, self.task_key)+ saved = previous.get('checkpoint') or {}+ if previous.get('status') in ('complete', 'failed') and saved.get(self.id_key):+ self.state = copy.deepcopy(saved)def snapshot(self) -> dict | None:"""当前状态的深拷贝;一次都没跑过返回 None。"""@@ -339,6 +345,15 @@ class BackgroundJob:state.update(fields)return True+ def checkpoint(self, job_id: str, **fields: object) -> bool:+ """领域结果已保存后,原子登记可跨进程恢复的完成集合与计数。"""+ if not self.update(job_id, **fields):+ return False+ if job_id in self._recoverable and (db := self._database_path()) is not None:+ from .job_recovery import checkpoint+ checkpoint(db, self.task_key, job_id, fields)+ return True+@contextmanagerdef editing(self, job_id: str) -> Iterator[dict | None]:"""拿住锁产出仍属于 `job_id` 的状态;被顶掉则产出 None。@@ -462,6 +477,7 @@ class BackgroundJob:job_id = current_job_id.get() or uuid.uuid4().hexstate = {self.id_key: job_id, "status": "running", "run_id": None,"started_at": time.time(), "error": "", **(initial or {})}+ state.update(self._resuming_fields(job_id, initial))self.state = statethread = threading.Thread(target=self._run, args=(fn, job_id),daemon=True, name=self.name)@@ -489,6 +505,8 @@ class BackgroundJob:from .job_recovery import savesave(db, self.task_key, job_id, recovery, run_id=self._run_ids.get(job_id))self._recoverable.add(job_id)+ from .job_recovery import checkpoint+ checkpoint(db, self.task_key, job_id, self.snapshot() or snapshot)except Exception:self._close_run(job_id, 'failed', error='任务未能启动')activity = self._activity_files.pop(job_id, None)@@ -503,6 +521,20 @@ class BackgroundJob:thread.start()return self.snapshot() or snapshot+ def _resuming_fields(self, job_id, initial):+ """恢复原任务的持久进度,当前请求仍使用自己的同步回执标识。"""+ from .job_recovery import current_job_id, read+ db = self._database_path()+ if not current_job_id.get() or db is None:+ return {}+ saved = read(db, self.task_key)+ if saved.get('job_id') != job_id:+ return {}+ fields = {**(saved.get('checkpoint') or {}), 'status': 'running', 'error': ''}+ if initial and 'request_id' in initial:+ fields['request_id'] = initial['request_id']+ return fields+def _open_run(self, job_id: str, trigger: str, state: dict) -> None:if self.runs is None:return@@ -534,8 +566,8 @@ class BackgroundJob:往一个没人读的状态里写进度。等不到就不等——一次外部 HTTP 探测可以是十几秒,关停不该被它拖住。- 安装暂停期间保留可恢复任务与原运行记录,线程退出后交回恢复入口。- 普通关停的在途任务记成 `cancelled`,租约恢复不把有序退出计为故障。+ `resume=True` 保留可恢复任务与原运行记录,线程退出后交回恢复入口。+ `resume=False` 取消任务并删除恢复记录。"""from .update_tasks import _mutex, state_fordb = self._database_path()@@ -556,15 +588,11 @@ class BackgroundJob:def _stop_run(self, job_id, *, resume):if resume and job_id in self._recoverable:- from .job_recovery import read- from .processing_control import user_paused- if (read(self._database_path(), self.task_key).get('pause_requested') or- self.task_key == 'library-processing' and user_paused(self._database_path(), job_id)):- self._paused_stops.add(job_id)- run_id = self._run_ids.pop(job_id, None)- if self.runs is not None and run_id is not None:- self._suspend_run(job_id, run_id)- return+ self._paused_stops.add(job_id)+ run_id = self._run_ids.pop(job_id, None)+ if self.runs is not None and run_id is not None:+ self._suspend_run(job_id, run_id)+ returnself._close_run(job_id, 'cancelled', error='服务关停,这一轮没有跑完')def _suspend_run(self, job_id, run_id):@@ -631,7 +659,8 @@ class BackgroundJob:state = self.snapshot() or {}if (db is not None and job_id in self._recoverableand state.get(self.id_key) == job_id and not state.get('waiting_for_update')):- from .job_recovery import finish+ from .job_recovery import checkpoint, finish+ checkpoint(db, self.task_key, job_id, state)finish(db, self.task_key, job_id,'failed' if state.get('status') == 'failed' else 'complete')finally:diff --git a/src/peach/library_processing.py b/src/peach/library_processing.pyindex fcf315e4b..613dbdbfe 100644--- a/src/peach/library_processing.py+++ b/src/peach/library_processing.py@@ -34,6 +34,7 @@ from .metadata_policy import SOURCE_SPECS, preferred_tag_sourcefrom .platform import root_online, translate_ledger_pathfrom .review_csv import read_rows, write_rowsfrom .scan import scan_location+from .fsutil import atomic_write_textfrom .ffmpeg import FFmpegResolverfrom .jobs import DiskGuardfrom .media_probe import probe_unmeasured@@ -1742,9 +1743,11 @@ class _SourceScan:scanned_bytes: int = 0-def _scan_sources(config, db_path, *, locations, asset_ids, stage, guard, update, issue):+def _scan_sources(config, db_path, *, locations, asset_ids, stage, guard, update, issue,+ scan_checkpoints=None, save_checkpoint=lambda: None, initial_probed=0):"""发现本趟可访问的来源根,按阶段登记文件并探测新视频。"""- result = _SourceScan([])+ result = _SourceScan([], probed=initial_probed)+ scan_checkpoints = scan_checkpoints if scan_checkpoints is not None else {}chosen_paths = _chosen_asset_paths(db_path, asset_ids)scan = asset_ids is None and stage in (ALL_STAGES, SCAN_STAGE)mounts = {key: tuple(str(value) for value in values) for key, values in config.mounts.items()}@@ -1760,8 +1763,13 @@ def _scan_sources(config, db_path, *, locations, asset_ids, stage, guard, updatecontinueresult.online_roots.append(translated_root)if scan:+ key = f'{source}:{root}'+ def remember(value, key=key):+ scan_checkpoints[key] = value+ save_checkpoint()registered = scan_location(db_path, source, root, declared_roots=config.locations,mounts=mounts, report=lambda line: None,+ resume_state=scan_checkpoints.get(key), save_checkpoint=remember,progress=lambda files, size: update(stage='扫描文件', scanned=result.scanned + files,scanned_bytes=result.scanned_bytes + size,@@ -1771,19 +1779,21 @@ def _scan_sources(config, db_path, *, locations, asset_ids, stage, guard, updateif stage == SCAN_STAGE:result.probed += probe_unmeasured(db_path, ffprobe, source, root, config=config,- report=lambda done, total: update(stage='读取时长与分辨率', checked=done, total=total))+ report=lambda done, total: update(stage='读取时长与分辨率',+ checked=result.probed + done, total=result.probed + total, probed=result.probed + done))reattach_in(db_path, registered.new_ids)update(scanned=result.scanned, probed=result.probed)return result-def _probe_sources(config, db_path, locations, online_roots, asset_ids, update, *, total_pending=0):+def _probe_sources(config, db_path, locations, online_roots, asset_ids, update, *, total_pending=0, initial_probed=0):"""已登记视频按可访问来源逐项读取媒体属性。"""- probed = 0- checked = 0+ probed = initial_probed+ checked = initial_probedffprobe = _ffprobe_path(config)def report(done, total):- update(stage='读取时长与分辨率', checked=checked + done, total=max(total_pending, checked + total))+ update(stage='读取时长与分辨率', checked=checked + done, probed=checked + done,+ total=max(initial_probed + total_pending, checked + total))for source, roots in locations.items():for root in roots:if translate_ledger_path(root) in online_roots:@@ -1818,10 +1828,11 @@ def _collection_rows(db_path, *, locations, online_roots, asset_ids):return rows, genre_decisions-def _processing_issue_log(config, path, job_id, isolated):+def _processing_issue_log(config, path, job_id, isolated, *, resume=False):"""每趟任务保留独立问题清单,仅初始化本趟文件。"""log_path = path.with_suffix('.issues.jsonl') if isolated else issues_path(config, job_id)- log_path.unlink(missing_ok=True)+ if not resume:+ log_path.unlink(missing_ok=True)return log_path@@ -1945,27 +1956,29 @@ def _generate_preview(config, row, database, resolver, *, progress):def _generate_previews(config, rows, database, *, update, issue, guard,- completed=(), checkpoint=lambda row: None):+ completed=(), checkpoint=lambda row: None, initial_made=0):"""只读本机视频;来源失败与抽帧失败各自保留可重试项目。"""from .previews import PreviewUnavailableif database is None:return 0- rows = [row for row in rows if row['location'] == 'local' and not skip_background_read(row['path'])- and _checkpoint_key(row) not in completed]+ rows = [row for row in rows if row['location'] == 'local' and not skip_background_read(row['path'])]+ base_done = sum(_checkpoint_key(row) in completed for row in rows)+ rows = [row for row in rows if _checkpoint_key(row) not in completed]resolver = FFmpegResolver(config.directory('tools') / 'ffmpeg')- made = 0+ made = initial_madefor index, row in enumerate(rows):if metadata_only(config.directory('state')):breakdef progress(row=row, index=index, made=made):guard.check()- update(stage='生成预览图', checked=index, total=len(rows), previews=made,+ update(stage='生成预览图', checked=base_done + index, total=base_done + len(rows), previews=made,current_asset_id=row['id'], current_asset_name=Path(str(row['name'] or '')).name,current_action='generating_preview', current_started_at=time.time(),current_deadline_at=time.time() + 90)progress()try:made += int(_generate_preview(config, row, database, resolver, progress=progress))+ update(previews=made)checkpoint(row)except InterruptedError:raise@@ -1977,7 +1990,7 @@ def _generate_previews(config, rows, database, *, update, issue, guard,'生成预览图失败,请检查 FFmpeg 与目录权限后重试')details = {'diagnostics': error.diagnostics} if isinstance(error, _PreviewCaptureError) else {}issue(row, message, action='generating_preview', retryable=True, **details)- update(checked=index + 1, previews=made, current_asset_id=None, current_asset_name='',+ update(checked=base_done + index + 1, previews=made, current_asset_id=None, current_asset_name='',current_action='', current_started_at=None, current_deadline_at=None)return made@@ -2138,6 +2151,54 @@ def _processing_workload(config, rows, stage, database, completed_previews):return pending+def _restore_processing_counts(state, previous):+ """同一处理任务沿用累计数、问题清单和已保存证据索引。"""+ previous = previous or {}+ for key in ('started_at', 'identified', 'covers', 'previews', 'probed', 'issue_count', 'paused_count',+ 'issue_preview', 'issues_truncated', 'notes', 'retryable_asset_ids'):+ if key in previous:+ state[key] = previous[key]+ for key in ('scan_checkpoints', 'local_metadata'):+ state[key] = dict(previous.get(key) or {})+ for key in ('auto_applied', 'auto_rejected'):+ state[key] = previous.get(key, 0)+++class _LocalEvidenceCache:+ """保存已经读出的本地证据与待处理图片,供同一影片的后续阶段复用。"""++ def __init__(self, config, db_path, cover_root, listing, state, path):+ self.config, self.db_path, self.cover_root = config, db_path, cover_root+ self.listing, self.state, self.path = listing, state, path++ def read(self, row, update, issue, poster_jobs):+ key = _checkpoint_key(row)+ cached_path = self.state['local_metadata'].get(key)+ try:+ cached = json.loads(Path(cached_path).read_text(encoding='utf-8')) if cached_path else None+ except (OSError, ValueError):+ cached = None+ if cached is not None:+ code, verify, payload, entries = cached['local']+ poster_jobs.extend((row, Path(source), name, Path(target))+ for source, name, target in cached['posters'])+ return code, verify, payload, [(source, document, Path(path)) for source, document, path in entries]+ offset, issues_before = len(poster_jobs), self.state['issue_count']+ local = _read_local_metadata(self.config, self.db_path, row, self.cover_root, self.listing,+ self.state, update, issue, poster_jobs)+ if local is not None and self.state['issue_count'] == issues_before:+ code, verify, payload, entries = local+ record = dict(local=[code, verify, payload, [(source, document, str(path)) for source, document, path in entries]],+ posters=[(str(source), name, str(target)) for _, source, name, target in poster_jobs[offset:]])+ encoded = json.dumps(record, ensure_ascii=False)+ digest = hashlib.sha256(encoded.encode()).hexdigest()+ item_path = self.path.parent / (self.path.stem + '-items') / f"{row['id']}-{digest}.json"+ if not item_path.is_file():+ atomic_write_text(item_path, encoded)+ self.state['local_metadata'][key] = str(item_path)+ return local++def process_library(config, db_path, candidate_root, cover_root, *, location='configured',report=lambda state: None, provider_factory=None, job_id=None,retry_ids=None, active=lambda: True, stage=ALL_STAGES,@@ -2194,13 +2255,15 @@ def process_library(config, db_path, candidate_root, cover_root, *, location='costate.update(initial_estimate(config, db_path, stage, scope_ids, speed_mode))state['steps'] = _processing_steps(stage, scope_ids,metadata_only_enabled=metadata_only(config.directory('state')))+ _restore_processing_counts(state, resume_state)estimate = ProcessingEstimate(config, state, control)# 实体表的水位在开工前记一次:比它大的实体就是这一轮建出来的(`_entity_followups`)。checkpoints = _LibraryCheckpoints(state, resume_state, database)entity_watermark = checkpoints.entity_watermark# 每行这一轮换上了几张封面,`(asset_id, 张数)`:换上了的作品,女优可能截得出更清楚的脸。covered = []- log_path = _processing_issue_log(config, path, state['job_id'], state_file is not None)+ log_path = _processing_issue_log(config, path, state['job_id'], state_file is not None,+ resume=bool(resume_state))# 界面只展示前 20 条,完整清单在这个文件里;地址跟着状态一起给出,# 不让人按 job_id 自己去拼路径。state['issues_log'] = str(log_path)@@ -2258,7 +2321,9 @@ def process_library(config, db_path, candidate_root, cover_root, *, location='coguard.check(force=True)locations = config.locations if location == 'configured' else {location: config.locations[location]}sources = _scan_sources(config, db_path, locations=locations, asset_ids=scope_ids,- stage=stage, guard=guard, update=update, issue=issue)+ stage=stage, guard=guard, update=update, issue=issue,+ scan_checkpoints=state['scan_checkpoints'],+ save_checkpoint=lambda: flush_state(force=True), initial_probed=state['probed'])state.update(scanned=sources.scanned, scanned_bytes=sources.scanned_bytes, probed=sources.probed)finish_step('sources')if stage == SCAN_STAGE:@@ -2274,6 +2339,7 @@ def process_library(config, db_path, candidate_root, cover_root, *, location='coif stage == PREVIEWS_STAGE:_generate_previews(config, rows, database, update=update, issue=issue, guard=guard,completed=checkpoints.completed['completed_previews'],+ initial_made=state['previews'],checkpoint=lambda row: checkpoints.mark('completed_previews', row, flush_state))finish_step('media')update(status='failed' if state['issue_count'] else 'complete', stage='处理结束',@@ -2283,9 +2349,9 @@ def process_library(config, db_path, candidate_root, cover_root, *, location='cogroups = {row['item_key']: row for row in read_rows(output, missing_ok=True)}scoped_groups = lambda: _candidate_scope(groups, scope_ids)listing = _DirectoryIndex()+ local_evidence = _LocalEvidenceCache(config, db_path, cover_root, listing, state, path)csv_dirty = Falsecsv_written_at = time.monotonic()- state.update(auto_applied=0, auto_rejected=0)def flush_candidates(force=False):nonlocal csv_dirty, csv_written_at@@ -2316,14 +2382,15 @@ def process_library(config, db_path, candidate_root, cover_root, *, location='coprepared = []poster_jobs = []local_rows = list(checkpoints.remaining_metadata(rows))+ completed_count = len(rows) - len(local_rows)for index, (_, row) in enumerate(local_rows):guard.check()issues_before = state['issue_count']- update(stage='检查配套文件', checked=index, total=len(local_rows),+ update(stage='检查配套文件', checked=completed_count + index, total=len(rows),current_asset_id=row['id'], current_asset_name=Path(str(row['name'] or '')).name,current_action='reading_local', current_started_at=time.time(),current_deadline_at=None)- local = _read_local_metadata(config, db_path, row, cover_root, listing, state, update, issue, poster_jobs)+ local = local_evidence.read(row, update, issue, poster_jobs)if local is None:continuecode, verify_existing, payload, entries = local@@ -2332,7 +2399,7 @@ def process_library(config, db_path, candidate_root, cover_root, *, location='coif code or payload:prepared.append((row, code, verify_existing, local_fields, entries,state['issue_count'] == issues_before))- update(checked=index + 1)+ update(checked=completed_count + index + 1)flush_candidates(force=True)pending.pop('local')pending['metadata'] = len(prepared)@@ -2348,12 +2415,13 @@ def process_library(config, db_path, candidate_root, cover_root, *, location='cofinish_step('local')if stage in (ALL_STAGES, COLLECT_STAGE):_probe_sources(config, db_path, locations, sources.online_roots, scope_ids, update,- total_pending=pending.get('probe', 0))+ total_pending=pending.get('probe', 0), initial_probed=state['probed'])pending.pop('probe', None)update(estimate_pending=dict(pending), stage='准备生成预览', checked=0, total=0)if not metadata_only(config.directory('state')):_generate_previews(config, rows, database, update=update, issue=issue, guard=guard,completed=checkpoints.completed['completed_previews'],+ initial_made=state['previews'],checkpoint=lambda row: checkpoints.mark('completed_previews', row, flush_state))pending.pop('previews', None)update(estimate_pending=dict(pending), stage='准备采集外部资料', checked=0, total=0)@@ -2362,7 +2430,8 @@ def process_library(config, db_path, candidate_root, cover_root, *, location='colocal_ok = local_ok and row['id'] not in poster_failedguard.check()issues_before = state['issue_count']- update(stage='采集缺失资料', checked=position, total=len(prepared), current_asset_id=row['id'],+ update(stage='采集缺失资料', checked=completed_count + position,+ total=completed_count + len(prepared), current_asset_id=row['id'],current_asset_name=Path(str(row['name'] or '')).name)target_key = f"asset:{row['id']}"missing = list(COLLECTED_FIELDS) if verify_existing else _missing_fields(row, local_fields)@@ -2376,7 +2445,7 @@ def process_library(config, db_path, candidate_root, cover_root, *, location='cocurrent_started_at=time.time(), current_deadline_at=None)checkpoints.queue_metadata(row, local_entries + found if local_ok else [], issues_before)flush_candidates()- update(checked=position + 1, candidates=len(scoped_groups()),+ update(checked=completed_count + position + 1, candidates=len(scoped_groups()),current_asset_id=None, current_asset_name='', current_action='',current_started_at=None, current_deadline_at=None)flush_candidates(force=True)diff --git a/src/peach/organize.py b/src/peach/organize.pyindex 7f8a66b83..081bf0450 100644--- a/src/peach/organize.py+++ b/src/peach/organize.py@@ -27,6 +27,7 @@ from .organize_templates import MAX_PATH, TemplateError, render, validate_templafrom .platform import (is_windows_path, location_mounts, location_roots, resolve_root, root_online,translate_ledger_path, within_root)from .review_csv import write_rows+from .fsutil import atomic_write_text#: 计划 CSV 的列。`reason` 只在 `action=skip` 时有值。PLAN_FIELDS = ["asset_id", "location", "current_path", "target_path", "action", "reason"]@@ -370,28 +371,37 @@ def apply_plan(connection: sqlite3.Connection, rows: Sequence[dict], *,"""movable = [row for row in rows if row.get("action") in {"rename", "move"}]refuse_unverified(path for row in movable for path in (row["current_path"], row["target_path"]))- entries: list[dict] = []- failures: list[dict] = []log_path = batch_log_dir(generated_root) / (f"organize-{stamp or time.strftime('%Y%m%d-%H%M%S')}.json")log_path.parent.mkdir(parents=True, exist_ok=True)+ previous = json.loads(log_path.read_text(encoding='utf-8')) if stamp and log_path.exists() else {}+ entries = list(previous.get('entries') or [])+ failures: list[dict] = []+ completed = {entry['asset_id'] for entry in entries}+ def save():+ atomic_write_text(log_path, json.dumps(+ {'stamp': stamp or time.strftime('%Y%m%d-%H%M%S'), 'entries': entries, 'failures': failures},+ ensure_ascii=False, indent=1))try:for index, row in enumerate(movable, 1):+ if int(row['asset_id']) in completed:+ continueold_path, new_path = str(row["current_path"]), str(row["target_path"])- outcome = _move_one(connection, int(row["asset_id"]), old_path, new_path)+ current = connection.execute('SELECT path FROM asset WHERE id=?', (int(row['asset_id']),)).fetchone()+ landed = (current is not None and current[0] == new_path+ and translate_ledger_path(new_path).is_file() and not translate_ledger_path(old_path).exists())+ outcome = 'ok' if landed else _move_one(connection, int(row["asset_id"]), old_path, new_path)if outcome == "ok":entries.append({"asset_id": int(row["asset_id"]),"old_path": old_path, "new_path": new_path})else:failures.append({"asset_id": int(row["asset_id"]),"path": old_path, "reason": outcome})+ save()if progress:progress(checked=index, total=len(movable))finally:- log_path.write_text(json.dumps(- {"stamp": stamp or time.strftime("%Y%m%d-%H%M%S"),- "entries": entries, "failures": failures},- ensure_ascii=False, indent=1), encoding="utf-8")+ save()return {"moved": len(entries), "failed": len(failures), "failures": failures[:50],"batch": log_path.name, "log_path": str(log_path)}@@ -402,28 +412,36 @@ def rollback_batch(connection: sqlite3.Connection, log_path: Path | str, *,path = Path(log_path)payload = json.loads(path.read_text(encoding="utf-8"))entries = list(payload.get("entries") or [])+ previous_restored = int(payload.get('restored_count') or 0)refuse_unverified(path for entry in entries for path in (entry["new_path"], entry["old_path"]))restored: list[dict] = []failures: list[dict] = []for index, entry in enumerate(reversed(entries), 1):- outcome = _move_one(connection, int(entry["asset_id"]),- str(entry["new_path"]), str(entry["old_path"]))+ current = connection.execute('SELECT path FROM asset WHERE id=?', (int(entry['asset_id']),)).fetchone()+ landed = (current is not None and current[0] == entry['old_path']+ and translate_ledger_path(entry['old_path']).is_file()+ and not translate_ledger_path(entry['new_path']).exists())+ outcome = 'ok' if landed else _move_one(connection, int(entry["asset_id"]),+ str(entry["new_path"]), str(entry["old_path"]))if outcome == "ok":restored.append(entry)else:failures.append({"asset_id": int(entry["asset_id"]),"path": str(entry["new_path"]), "reason": outcome})+ payload['entries'] = [row for row in entries if row not in restored]+ payload['restored_count'] = previous_restored + len(restored)+ atomic_write_text(path, json.dumps(payload, ensure_ascii=False, indent=1))if progress:- progress(checked=index, total=len(entries))+ progress(checked=previous_restored + index, total=previous_restored + len(entries))remaining = [entry for entry in entriesif entry not in restored]if remaining:payload["entries"] = remaining- path.write_text(json.dumps(payload, ensure_ascii=False, indent=1), encoding="utf-8")+ atomic_write_text(path, json.dumps(payload, ensure_ascii=False, indent=1))else:# 整批都退回去了,这个批次就不该再出现在「回滚上一批」里。path.unlink(missing_ok=True)- return {"restored": len(restored), "failed": len(failures), "failures": failures[:50],+ return {"restored": previous_restored + len(restored), "failed": len(failures), "failures": failures[:50],"batch": path.name}diff --git a/src/peach/runtime_prepare.py b/src/peach/runtime_prepare.pyindex 792b8d9ec..3b512392e 100644--- a/src/peach/runtime_prepare.py+++ b/src/peach/runtime_prepare.py@@ -336,7 +336,11 @@ def task_gate(db: Path, *, host: str, alive: Callable[[int], bool]) -> TaskGate:for row in rows:run = ActiveRun(int(row[0]), str(row[1]), str(row[2]), str(row[3]),None if row[4] is None else int(row[4]), row[5])- if run.followup_key or run.trigger in RESUMABLE_TRIGGERS:+ from .job_recovery import read+ recovery = read(db, run.task_key)+ can_resume = (recovery.get('status') == 'active' and recovery.get('run_id') == run.id+ and isinstance(recovery.get('request'), dict))+ if run.followup_key or run.trigger in RESUMABLE_TRIGGERS or can_resume:resumable.append(run)elif run.pid is not None and not alive(run.pid):stale.append(run)diff --git a/src/peach/sample_followup.py b/src/peach/sample_followup.pyindex 67231e4d0..839db2bf8 100644--- a/src/peach/sample_followup.py+++ b/src/peach/sample_followup.py@@ -64,6 +64,27 @@ def _missing(error: Exception) -> bool:isinstance(error, SourceFailure) and error.reason == FailureReason.NOT_FOUND)+class _SampleCheckpoint:+ """固定本批次番号,保留已经落库的样张与累计产物数。"""++ def __init__(self, contract, handle, misses, snapshots):+ self.handle = handle+ saved = getattr(handle, 'saved', {}) if handle is not None else {}+ saved = saved if isinstance(saved, dict) else {}+ self.completed = set(saved.get('completed_codes') or [])+ self.landed, self.images = saved.get('landed', 0), saved.get('images', 0)+ with contract.database.read_connection() as connection:+ self.codes = saved.get('codes') or sample_images.pending(+ connection, contract.has_cover, misses, snapshots, limit=SCAN)+ if handle is not None:+ handle.checkpoint(codes=self.codes)++ def mark(self, code, landed, images):+ self.completed.add(code)+ if self.handle is not None:+ self.handle.checkpoint(completed_codes=sorted(self.completed), landed=landed, images=images)++def run(contract, key: str, handle, *, transport=None, clock=time.monotonic) -> dict:"""跑一条补样张后继,返回活动页上那一行的摘要。"""from .jav_cover_fetch import DeadlineExceeded, HostLimitedTransport@@ -72,8 +93,8 @@ def run(contract, key: str, handle, *, transport=None, clock=time.monotonic) ->generated = Path(contract.candidate_root)misses = sample_images.Misses(sample_images.misses_path(generated))snapshots = sample_images.snapshot_index(Path(contract.follow_sources_root))- with contract.database.read_connection() as connection:- codes = sample_images.pending(connection, contract.has_cover, misses, snapshots, limit=SCAN)+ checkpoint = _SampleCheckpoint(contract, handle, misses, snapshots)+ codes = checkpoint.codesif not codes:return {"outcome": "没有要补的番号", "codes": 0}batch = sample_images.batch_for(getattr(handle, "run_id", None))@@ -83,7 +104,8 @@ def run(contract, key: str, handle, *, transport=None, clock=time.monotonic) ->SourceTransport(Path(contract.follow_secrets_root), max_requests=BATCH * 3,max_bytes=64 * 1024 * 1024, max_seconds=BUDGET_SECONDS), 2.0)deadline = clock() + BUDGET_SECONDS- landed = images = absent = failed = deferred = asked = 0+ landed, images = checkpoint.landed, checkpoint.images+ absent = failed = deferred = asked = 0#: 快照第一次看过、里面没有样张的部数:下一轮改为联网问,也算这一轮往前推了。probed = 0#: 没问站就跳过的部数,按原因分:快照里没样张又没有可问的官方站、那一站近期说过没有、那一站冷却中。@@ -91,8 +113,10 @@ def run(contract, key: str, handle, *, transport=None, clock=time.monotonic) ->paused: set[str] = set()try:for index, code in enumerate(codes):+ if code in checkpoint.completed:+ continueif handle is not None:- handle.progress(current=index, total=len(codes), label=f"{TASK_LABEL}:{code}")+ handle.progress(current=max(index, len(checkpoint.completed)), total=len(codes), label=f"{TASK_LABEL}:{code}")site, urls = "", []snapshot = sample_images.snapshot_samples(snapshots.get(code, ()))if snapshot:@@ -144,6 +168,7 @@ def run(contract, key: str, handle, *, transport=None, clock=time.monotonic) ->if count:landed += 1images += count+ checkpoint.mark(code, landed, images)finally:misses.save()if owned:diff --git a/src/peach/scan.py b/src/peach/scan.pyindex 3824e4a82..46d08ed8f 100644--- a/src/peach/scan.py+++ b/src/peach/scan.py@@ -31,16 +31,20 @@from __future__ import annotationsimport os+import hashlib+import jsonimport reimport sqlite3import timefrom collections.abc import Callable, Mapping, Sequence-from dataclasses import dataclass+from dataclasses import asdict, dataclassfrom pathlib import Path, PureWindowsPath+from filelock import FileLockfrom . import folder_rules, record_rehome, subtitles, wantsfrom .platform import is_windows_path, resolve_location, resolve_rootfrom .processing_control import checkpoint+from .fsutil import atomic_write_textVIDEO = {".mp4", ".m4v", ".mkv", ".avi", ".wmv", ".mov", ".ts", ".flv", ".rmvb", ".mpg",".m2ts", ".strm", ".iso"}@@ -336,14 +340,14 @@ class ScanResult:f"字幕 {self.subtitles:,} 条(孤立 {self.orphan_subtitles:,} 条)")-def _walk(top: Path):+def _walk(top: Path, pending: list[str] | None = None):"""自顶向下遍历,每个目录给出它里面的非目录条目(`os.DirEntry`)。与 `os.walk` 的差别只在一处:条目的大小与时间从 `DirEntry.stat()` 拿。Windows 的目录列表自带这两项,不必再对每个文件发一次 `os.stat`——网盘挂载上那一次就是一趟往返,几万个文件就是几万趟。读不了的目录跳过;符号链接指向的目录列出但不进入。"""- stack = [os.fspath(top)]+ stack = pending if pending is not None else [os.fspath(top)]while stack:checkpoint('scan')directory = stack.pop()@@ -368,7 +372,8 @@ def _walk(top: Path):else:files.append(entry)# 倒着压栈,弹出来就是字典序,进度行读起来和目录里看到的一致。- stack.extend(sorted(subdirectories, reverse=True))+ queued = set(stack)+ stack.extend(path for path in sorted(subdirectories, reverse=True) if path not in queued)yield directory, sorted(files, key=lambda entry: entry.name)@@ -385,11 +390,39 @@ class _ScanProgress:self.reported_at = now-def scan_location(+def scan_location(db_path, location, root, **kwargs) -> ScanResult:+ """命令行按来源与根目录保存断点,组合任务可提供自己的持久检查点。"""+ if 'save_checkpoint' in kwargs or 'resume_state' in kwargs:+ return _scan_location(db_path, location, root, **kwargs)+ from .update_tasks import state_for+ key = hashlib.sha256(f'{location}:{root}'.encode()).hexdigest()+ path = state_for(Path(db_path)).parent / 'scan-checkpoints' / f'{key}.json'+ path.parent.mkdir(parents=True, exist_ok=True)+ with FileLock(str(path) + '.lock', timeout=0):+ previous = json.loads(path.read_text(encoding='utf-8')) if path.is_file() else {}+ result = _scan_location(db_path, location, root, **kwargs, resume_state=previous,+ save_checkpoint=lambda state: atomic_write_text(path, json.dumps(state, ensure_ascii=False)))+ path.unlink(missing_ok=True)+ return result+++def _pending_directories(previous, walk_root):+ """恢复目录栈,并核对本机挂载仍属于当前扫描根。"""+ pending = list(previous.get('pending', [os.fspath(walk_root)]))+ if previous.get('current_directory'):+ pending.append(previous['current_directory'])+ if any(not Path(value).is_relative_to(walk_root) for value in pending):+ raise ScanTargetError('扫描目录与保存的断点不一致,请核对挂载设置')+ return pending+++def _scan_location(db_path: str | os.PathLike[str], location: str, root: str, *,declared_roots: Mapping[str, str], mounts: Mapping[str, str | Path] | None = None,report: Callable[[str], None] = print, windows: bool | None = None,progress: Callable[[int, int], None] = lambda files, total_bytes: None,+ resume_state: dict | None = None,+ save_checkpoint: Callable[[dict], None] = lambda state: None,) -> ScanResult:"""遍历 `root`(账本口径)对应的本机目录,把文件元数据 upsert 进 `asset`。@@ -399,27 +432,41 @@ def scan_location(check_scan_target(location, root, declared_roots=declared_roots)walk_root = walk_root_for(location, root, declared_roots=declared_roots, mounts=mounts or {}, windows=windows)+ previous = resume_state or {}+ if previous.get('result'):+ return ScanResult(**{**previous['result'], 'new_ids': tuple(previous['result']['new_ids'])})ledger_root = PureWindowsPath(root)- now = time.strftime("%Y-%m-%d %H:%M:%S")+ now = previous.get('scan_time') or time.strftime("%Y-%m-%d %H:%M:%S")started = time.time()progress = _ScanProgress(progress)- files = 0- total = 0- skipped = 0- video_dirs: set[str] = set()+ files = previous.get('files', 0)+ total = previous.get('total_bytes', 0)+ skipped = previous.get('sidecars', 0)+ tracks = previous.get('subtitles', 0)+ orphans = previous.get('orphan_subtitles', 0)+ video_dirs = set(previous.get('video_dirs', []))+ pending = _pending_directories(previous, walk_root)batch: list[tuple] = []- # 字幕要等正片的行落库之后才能按 `(location, path)` 查到 asset_id,所以先攒着,- # 遍历完再一次登记。sidecar 按定义与正片同目录,配对只看当前这一个目录。- sidecars: list[subtitles.Sidecar] = []+ # 文件批次提交后保存文件名游标;同目录的配对资料随断点保留。+ def save_directory(directory=None, cursor=None, here=None):+ artwork_parents = {os.path.dirname(value) for value in [*pending, *([directory] if directory else [])]+ if os.path.basename(value).casefold() in ARTWORK_DIRS}+ video_dirs.intersection_update(artwork_parents)+ save_checkpoint(dict(scan_time=now, pending=list(pending), files=files,+ total_bytes=total, sidecars=skipped, subtitles=tracks,+ orphan_subtitles=orphans, video_dirs=sorted(video_dirs),+ current_directory=directory, cursor=cursor, here=dict(here or {})))# 写事务只在每批 upsert 到 commit 之间打开,遍历与 stat 都在两次 commit 之间,不持锁。# 忙等时长与 `LedgerDatabase.connect` 一致。connection = sqlite3.connect(db_path, timeout=30)try:connection.execute("PRAGMA journal_mode=WAL")- for directory, entries in _walk(walk_root):+ for directory, entries in _walk(walk_root, pending):checkpoint('scan')relative = Path(directory).relative_to(walk_root).parts- here: dict[str, tuple[int, str]] = {}+ continuing = directory == previous.get('current_directory')+ here = dict(previous.get('here') or {}) if continuing else {}+ cursor = previous.get('cursor') if continuing else Nonevideos = video_stems(entry.name for entry in entries)if videos:video_dirs.add(directory)@@ -429,6 +476,8 @@ def scan_location(for entry in entries:checkpoint('scan')name = entry.name+ if cursor is not None and name <= cursor:+ continueif is_sidecar(name, videos, artwork_dir=artwork_dir):skipped += 1continue@@ -448,16 +497,23 @@ def scan_location(connection.executemany(_UPSERT, batch)connection.commit()batch.clear()+ save_directory(directory, name, here)report(f" {time.time() - started:5.0f}s {files:,} 文件 "f"{total / 1024 ** 4:.2f} TB")- sidecars.extend(subtitles.directory_sidecars(+ sidecars = subtitles.directory_sidecars(ledger_root.joinpath(*relative), here,- [name for name in here if medium_of(name) == "video"]))+ [name for name in here if medium_of(name) == "video"])+ if batch:+ connection.executemany(_UPSERT, batch)+ batch.clear()+ directory_tracks, directory_orphans = subtitles.record(connection, location, sidecars, now)+ connection.commit()+ tracks += directory_tracks+ orphans += directory_orphans+ save_directory()if batch:connection.executemany(_UPSERT, batch)connection.commit()- tracks, orphans = subtitles.record(connection, location, sidecars, now)- connection.commit()# 只有 INSERT 那一支写 `first_seen`,等于本次时刻的就是这一轮新建的行。new_ids = tuple(row[0] for row in connection.execute("SELECT id FROM asset WHERE location=? AND first_seen=? ORDER BY id", (location, now)))@@ -472,6 +528,7 @@ def scan_location(connection.close()result = ScanResult(location, root, files, total, time.time() - started, gone,tracks, orphans, skipped, new_ids, reattached, acquired)+ save_checkpoint({'result': asdict(result)})progress(files, total, force=True)report(result.summary())return resultdiff --git a/src/peach/task_runs.py b/src/peach/task_runs.pyindex 5b7461dbe..0c6618c72 100644--- a/src/peach/task_runs.py+++ b/src/peach/task_runs.py@@ -546,7 +546,7 @@ class TaskRunStore:return Nonerun_id = int(row["id"])cursor = connection.execute(- "UPDATE task_run SET status='running',pid=?,host=?,started_at=?,"+ "UPDATE task_run SET status='running',pid=?,host=?,started_at=COALESCE(started_at,?),""heartbeat_at=? WHERE id=? AND status='pending'",(os.getpid(), self.host, moment, moment, run_id))return self.get(run_id) if cursor.rowcount else None@@ -567,8 +567,8 @@ class TaskRunStore:if not rows:return []connection.execute(- "UPDATE task_run SET status='pending',started_at=NULL,heartbeat_at=?,"- "progress_current=0 WHERE followup_key IS NOT NULL AND status='running'",+ "UPDATE task_run SET status='pending',heartbeat_at=? "+ "WHERE followup_key IS NOT NULL AND status='running'",(stamp(),))return [int(row["id"]) for row in rows]@@ -635,6 +635,12 @@ class TaskRunStore:expired = beat is None or beat < deadlineif alive or (not local and not expired):continue+ if local:+ from .job_recovery import read+ recovery = read(self.database.db_path, run.task_key)+ if recovery.get('status') == 'active' and recovery.get('run_id') == run.id:+ self.suspend(run.id, label='等待服务启动后继续')+ continueif self.finish(run.id, "interrupted",error="服务重启,这一轮没有跑完"):recovered.append(run.id)@@ -823,6 +829,25 @@ class TaskRunHandle:run_id: int | Nonesettled: bool = False+ @property+ def saved(self) -> dict:+ """同一条持久任务已提交的领域检查点。"""+ run = self.store.get(self.run_id) if self.run_id is not None else None+ return dict(run.result_summary.get('checkpoint') or {}) if run else {}++ def checkpoint(self, **fields) -> None:+ """领域结果提交后保存完成集合;后继重新领取时沿用。"""+ if self.run_id is None or not self.store.enabled:+ return+ with self.store.database.write_transaction(notify=False) as connection:+ row = connection.execute('SELECT result_summary FROM task_run WHERE id=?', (self.run_id,)).fetchone()+ if row is None:+ return+ summary = json.loads(row[0] or '{}')+ summary['checkpoint'] = {**summary.get('checkpoint', {}), **fields}+ connection.execute("UPDATE task_run SET result_summary=? WHERE id=? AND status='running'",+ (json.dumps(summary, ensure_ascii=False), self.run_id))+def progress(self, current: int | None = None, total: int | None = None,label: str | None = None, *,throttle: float = PROGRESS_INTERVAL) -> None:diff --git a/src/peach/timeline_sheets.py b/src/peach/timeline_sheets.pyindex 8ec25a9f6..62ac7ea09 100644--- a/src/peach/timeline_sheets.py+++ b/src/peach/timeline_sheets.py@@ -209,7 +209,7 @@ def pending(db_path: Path | str, interval: int, *, limit: int = 0) -> list[tupledef generate_library(db_path: Path | str, root: Path, interval: int, *, ffmpeg: str,report=lambda **fields: None, active=lambda: True,- guard=None, limit: int = 0) -> dict:+ guard=None, limit: int = 0, resume_state=None, checkpoint=lambda **fields: None) -> dict:"""按队列逐部生成,返回这一趟的计数。失败不中断:一部片子解不出来(源坏了、时长记错、编码器不认)是它自己的事,@@ -219,8 +219,16 @@ def generate_library(db_path: Path | str, root: Path, interval: int, *, ffmpeg:tasks = pending(db_path, interval, limit=limit)state = {"total": len(tasks), "done": 0, "made": 0, "skipped": 0,"failed": 0, "stopped": ""}+ completed = set((resume_state or {}).get('completed_assets') or [])+ for key in ('done', 'made', 'skipped', 'failed'):+ state[key] = (resume_state or {}).get(key, 0)+ def save(asset_id):+ completed.add(asset_id)+ checkpoint(completed_assets=sorted(completed), **state)report(**state)for asset_id, path, duration in tasks:+ if asset_id in completed and is_current(root, asset_id, interval):+ continuefrom .strm import skip_background_readif skip_background_read(path):state['done'] += 1@@ -239,6 +247,7 @@ def generate_library(db_path: Path | str, root: Path, interval: int, *, ffmpeg:state["done"] += 1if is_current(root, asset_id, interval):state["skipped"] += 1+ save(asset_id)report(**state)continue# 账本路径是 Windows 形态,而挂载层可能是大小写敏感的:这一步同时管翻译和@@ -251,6 +260,7 @@ def generate_library(db_path: Path | str, root: Path, interval: int, *, ffmpeg:if build(ffmpeg, str(source), float(duration or 0), root, int(asset_id),interval, active=active):state["made"] += 1+ save(asset_id)else:state["failed"] += 1report(**state)diff --git a/src/peach/web_feeds.py b/src/peach/web_feeds.pyindex dec162814..bf80e8abc 100644--- a/src/peach/web_feeds.py+++ b/src/peach/web_feeds.py@@ -157,10 +157,11 @@ def _backfill_solo_works(contract, transport, source, report: dict) -> None:report["codes"] = list(report.get("codes") or []) + codes-def _execute_check(contract, body, job_id: str) -> dict:- only = body.get("sources")+def _check_sources(contract, body, previous):+ """首次按请求选取到期来源,续跑只读取本批次冻结的来源集合。"""+ only = previous.get('source_ids') if 'source_ids' in previous else body.get("sources")with contract.database.read_connection() as connection:- if isinstance(only, list) and only:+ if isinstance(only, list):wanted = {int(value) for value in only}rows = [row for row in connection.execute("SELECT * FROM feed_source ORDER BY id") if int(row["id"]) in wanted]@@ -169,16 +170,32 @@ def _execute_check(contract, body, job_id: str) -> dict:"SELECT * FROM feed_source WHERE enabled=1 ORDER BY id"))else:rows = feeds.due_sources(connection)- rows = rows[:MAX_SOURCES_PER_RUN]+ return rows[:MAX_SOURCES_PER_RUN]+++def _execute_check(contract, body, job_id: str) -> dict:+ previous = contract.feed_job.snapshot() or {}+ rows = _check_sources(contract, body, previous)transport = _transport(contract)- results, codes = [], []+ results = list(previous.get('results') or [])+ completed = {row['source'] for row in results if row.get('ok') and not row.get('backfill_error')}+ codes = [code for row in results for code in row.get('codes') or []]+ if not contract.feed_job.checkpoint(job_id, source_ids=[int(row['id']) for row in rows]):+ raise InterruptedError('订阅检查已停止')try:for index, source in enumerate(rows):- contract.feed_job.update(job_id, checked=index, total=len(rows),- current=source["name"] or source["url"])+ if int(source['id']) in completed:+ continue+ if not contract.feed_job.update(job_id, checked=len(results), total=len(rows),+ current=source["name"] or source["url"]):+ raise InterruptedError('订阅检查已停止')report = poll_source(contract, transport, source)+ results = [row for row in results if row['source'] != report['source']]results.append(report)codes.extend(report.get("codes") or [])+ if not contract.feed_job.checkpoint(job_id, results=results, checked=len(results),+ total=len(rows), added=len(set(codes))):+ raise InterruptedError('订阅检查已停止')finally:close = getattr(transport, "close", None)if close:diff --git a/src/peach/web_follow.py b/src/peach/web_follow.pyindex 09c73a3e8..1dcdd4a6b 100644--- a/src/peach/web_follow.py+++ b/src/peach/web_follow.py@@ -2040,7 +2040,7 @@ def w_follow_check(contract, body) -> dict:# 轮询撞上在跑的那一轮记成跳过,手动撞上外部占用则由 API 回 409。started = contract.follow_job.start(work, restart=True,trigger="scheduled" if body.get("automatic") else "manual",- recovery={**{key: body[key] for key in ('source', 'sources', 'older', 'backfill_all', 'automatic') if key in body},+ recovery={**{key: body[key] for key in ('source', 'sources', 'older', 'backfill_all', 'rewind', 'automatic') if key in body},'background': True},initial={"ok": True, "checked": 0, "total": 0, "results": [],"request_id": request_id, "older": bool(body.get("older")), "current": None})@@ -2143,6 +2143,7 @@ def _check_payload(result) -> dict:def _run_follow_check(contract, body, job_id=None) -> dict:+ previous = (contract.follow_job.snapshot() or {}) if job_id else {}requested = body.get("source")source_id = requested if isinstance(requested, int) else None# 往回抓一页。这是**显式的、一次一页**的动作:常规检查永远只看第一页,@@ -2161,15 +2162,23 @@ def _run_follow_check(contract, body, job_id=None) -> dict:backfill_providers=_BACKFILL_PROVIDERS)if "sources" in body:rows = [row for row in rows if row["id"] in body["sources"]]+ if 'source_ids' in previous:+ rows = [row for row in rows if row['id'] in previous['source_ids']]+ results = list(previous.get('results') or [])+ completed = {row['source'] for row in results if row.get('ok')}+ pages = dict(previous.get('backfill_pages') or {})if rewind and older:for row in rows:- row["backfill_page"] = 0+ row["backfill_page"] = pages.get(str(row['id']), 0)writer = _check_writer(contract)cooldown = getattr(contract, "follow_media_resolver", None)- results = []+ if job_id and not contract.follow_job.checkpoint(job_id, source_ids=[row['id'] for row in rows]):+ raise InterruptedError('关注检查已停止')for row in rows:+ if row['id'] in completed:+ continueif job_id and contract.follow_job.snapshot() is None:- break+ raise InterruptedError('关注检查已停止')current = {"source": row["id"], "label": author_display_name(row),"provider": PROVIDER_LABELS.get(row["provider"], row["provider"])}def progress(*, current=current, **fields):@@ -2182,6 +2191,12 @@ def _run_follow_check(contract, body, job_id=None) -> dict:row, credentials=credentials, writer=writer,connector_factory=build_connector, older=older, progress=progress,initial_days=initial_days, cooldown=cooldown))+ def remember_page():+ if result.get('ok') and isinstance(result.get('page'), int):+ pages[str(row['id'])] = result['page']+ if job_id and not contract.follow_job.checkpoint(job_id, backfill_pages=dict(pages)):+ raise InterruptedError('关注检查已停止')+ remember_page()if older and body.get("backfill_all"):# 全量回抓:一页记完接着抓下一页,直到站点说没有更多、某一页失败,# 或满 500 轮。每轮把结果页写回工作行,`run_check` 从它加一起算,页码@@ -2190,7 +2205,7 @@ def _run_follow_check(contract, body, job_id=None) -> dict:while (result.get("ok") and not result.get("exhausted")and isinstance(result.get("page"), int) and rounds < 500):if job_id and contract.follow_job.snapshot() is None:- break+ raise InterruptedError('关注检查已停止')row["backfill_page"] = result["page"]# 首页重放只该有一次:`run_check` 已把标记写回 ledger,工作行里的旧# metadata 还带着它,不摘掉的话每一轮都判成重放,第 0 页连抓到轮数上限。@@ -2202,8 +2217,12 @@ def _run_follow_check(contract, body, job_id=None) -> dict:connector_factory=build_connector, older=True, progress=progress,initial_days=initial_days, cooldown=cooldown))rounds += 1+ remember_page()result["author"] = author_display_name(row)+ results = [item for item in results if item['source'] != row['id']]results.append(result)+ if job_id and not contract.follow_job.checkpoint(job_id, results=results.copy(), checked=len(results), total=len(rows)):+ raise InterruptedError('关注检查已停止')progress()# 还没建档的关注作者交给建档后继(ADR-0096)。后继由结果声明、调度端统一派# (ADR-0040);这里判不出来只是这一轮不派,检查本身的结论不受影响。@@ -2464,7 +2483,7 @@ def _external_search_payload(search) -> dict:"url": search.url, "evidence": search.evidence}-def w_follow_resolve(contract, body, *, progress=None) -> dict:+def w_follow_resolve(contract, body, *, progress=None, checkpoint=None, resume_state=None) -> dict:"""把粘进来的每一行解析成「可以添加什么」,但**不添加**。一行是链接就直接认;不是链接就当成名字或 id 拿去各来源查一遍。两种都只返回结果,@@ -2475,9 +2494,11 @@ def w_follow_resolve(contract, body, *, progress=None) -> dict:job = contract.follow_resolve_jobdef work(job_id):result = w_follow_resolve(contract, {**body, "background": False},- progress=lambda **fields: job.update(job_id, **fields))+ progress=lambda **fields: job.update(job_id, **fields),+ checkpoint=lambda **fields: job.checkpoint(job_id, **fields), resume_state=job.snapshot())job.update(job_id, **result, status="complete", completed_at=time.time())- return job.start(work, restart=True, initial={"message": "正在准备查找关注来源"})+ return job.start(work, restart=True, initial={"message": "正在准备查找关注来源"},+ recovery={'lines': body.get('lines'), 'background': True})raw = body.get("lines")if isinstance(raw, str):raw = raw.splitlines()@@ -2547,11 +2568,17 @@ def w_follow_resolve(contract, body, *, progress=None) -> dict:"known": (parsed.provider, canonical_source_ref(parsed.provider, parsed.ref)) in known}]}- results = []+ results = list((resume_state or {}).get('results') or [])+ completed = {row['line'] for row in results if row.get('kind') != 'error' and not row.get('failures')}for index, line in enumerate(lines):# 分母是 plan_units 算出来的,分子就照同一把尺子加:一行查失败也占掉了它那# 几个来源,不加进去的话后面每一行都低这么多,环最后停在半路。- results.append(resolve_line(index, line))+ if line not in completed:+ result = resolve_line(index, line)+ results = [row for row in results if row['line'] != line]+ results.append(result)+ if checkpoint and not checkpoint(results=results.copy(), checked=done_units + plan_units(line), total=total_units):+ raise InterruptedError('关注来源查找已停止')done_units += plan_units(line)if progress is not None and total_units:# 每个来源是开查前报一次自己的序号,最后一个报的是 total-1;收尾这一下才走满。diff --git a/src/peach/web_job_recovery.py b/src/peach/web_job_recovery.pyindex 7ef418330..e8b032ee2 100644--- a/src/peach/web_job_recovery.py+++ b/src/peach/web_job_recovery.py@@ -1,11 +1,16 @@"""Web 组合入口为可恢复任务装配领域处理器。"""from .job_recovery import resumefrom .web_library_processing import w_library_processing-from .web_scraping import w_scraping_cover+from .web_scraping import w_scraping_cover, w_scraping_amane_rebuildfrom .web_timeline_thumbnails import w_thumbnail_jobs-from .web_follow import w_follow_check+from .web_follow import w_follow_check, w_follow_resolvefrom .web_feeds import w_feed_checkfrom .web_wants import start_scrape+from .web_links import w_links_check, w_links_prune+from .web_resource_sync import w_resource_sync_scan, w_resource_sync_apply+from .web_stats import w_taste_refresh+from .web_media_repair import w_media_repair+from .web_organize import w_organize_apply, w_organize_rollbackfrom apscheduler.schedulers.background import BackgroundScheduler@@ -17,6 +22,16 @@ def restore(contract):"follow-check": lambda body: w_follow_check(contract, body),"feed-check": lambda body: w_feed_check(contract, body),"want-scrape": lambda _body: start_scrape(contract),+ "link-check": lambda body: w_links_check(contract, {**body, 'restart': True}),+ "resource-scan": lambda body: w_resource_sync_scan(contract, {**body, 'background': True}),+ "resource-apply": lambda body: w_resource_sync_apply(contract, body),+ "link-prune": lambda body: w_links_prune(contract, body),+ "follow-resolve": lambda body: w_follow_resolve(contract, body),+ "taste-refresh": lambda body: w_taste_refresh(contract, body),+ "media-repair": lambda body: w_media_repair(contract, body),+ "amane-bridge": lambda body: w_scraping_amane_rebuild(contract, body),+ "organize": lambda body: (w_organize_rollback(contract, body) if body.get('action') == 'rollback'+ else w_organize_apply(contract, body)),})diff --git a/src/peach/web_links.py b/src/peach/web_links.pyindex a24c042b0..61bb8da61 100644--- a/src/peach/web_links.py+++ b/src/peach/web_links.py@@ -137,6 +137,10 @@ def _run_link_check(contract: LinkContract, check_id: str,给了 `link_ids` 就只验点名的那几条,别的链接的判定由调用方原样带进初始状态。"""job = contract.link_check+ previous = job.snapshot() or {}+ completed = set(previous.get('completed_links') or [])+ if 'link_ids' in previous:+ link_ids = previous['link_ids']# 已标记失效的不再验:结论已经落账,每次都去敲一个停放域名只会招来杀毒软件告警。query = ("SELECT l.id, l.link_kind, l.label, l.url, e.canonical_name AS entity ""FROM entity_link l JOIN entity e ON e.id=l.entity_id "@@ -153,13 +157,23 @@ def _run_link_check(contract: LinkContract, check_id: str,if state is None:returnstate["total"] = len(rows)+ if not job.checkpoint(check_id, link_ids=[row['id'] for row in rows]):+ raise InterruptedError('链接检查已停止')for row in rows:+ if row['id'] in completed:+ continue+ if job.snapshot() is None:+ raise InterruptedError('链接检查已停止')status, note = _probe(row["url"])verdict = link_verdict(status, note)with job.editing(check_id) as state:if state is None:return # 被新的检查顶掉了,安静收工- state["checked"] += 1+ state['gone'] = [item for item in state['gone'] if item['id'] != row['id']]+ state['unclear'] = [item for item in state['unclear'] if item['id'] != row['id']]+ if verdict != 'unclear':+ completed.add(row['id'])+ state["checked"] = len(completed)if verdict != "ok":state["gone" if verdict == "gone" else "unclear"].append({"id": row["id"], "entity": row["entity"],@@ -167,8 +181,12 @@ def _run_link_check(contract: LinkContract, check_id: str,"url": row["url"],"note": _note(status, note),})+ snapshot = job.snapshot() or {}+ if not job.checkpoint(check_id, completed_links=sorted(completed), checked=len(completed),+ total=len(rows), gone=snapshot.get('gone', []), unclear=snapshot.get('unclear', [])):+ raise InterruptedError('链接检查已停止')time.sleep(CHECK_INTERVAL)- job.update(check_id, status="complete", completed_at=time.time())+ job.update(check_id, checked=len(rows), status="complete", completed_at=time.time())def _retry_state(state: dict, link_ids: list[int]) -> dict:@@ -193,6 +211,13 @@ def w_links_check(contract: LinkContract, body=None):`BackgroundJob` 的后台线程 + 可轮询状态。"""body = body or {}+ from .job_recovery import current_job_id, read+ if current_job_id.get() and not body.get('status_only'):+ saved = read(contract.db_path, 'link-check').get('checkpoint') or {}+ return _check_public(contract.link_check.start(+ lambda check_id: _run_link_check(contract, check_id, saved.get('link_ids')),+ initial={'checked': 0, 'total': 0, 'gone': [], 'unclear': [], 'scope': 'all'},+ restart=True, recovery=body))if body.get("status_only") is True:state = contract.link_check.snapshot()if state is None:@@ -217,12 +242,14 @@ def w_links_check(contract: LinkContract, body=None):return _check_public(contract.link_check.start(lambda check_id: _run_link_check(contract, check_id, link_ids),initial=_retry_state(state, link_ids), restart=True,+ recovery={'retry': link_ids},))return _check_public(contract.link_check.start(lambda check_id: _run_link_check(contract, check_id),initial={"checked": 0, "total": 0, "gone": [], "unclear": [], "scope": "all"},restart=body.get("restart") is True,+ recovery={},))@@ -239,7 +266,7 @@ def _prune_selection(planned: list[dict], body: dict) -> list[dict]:return [item for item in planned if item["id"] in requested]-def w_links_prune(contract: LinkContract, body, *, progress=None):+def w_links_prune(contract: LinkContract, body, *, progress=None, checkpoint=None, resume_state=None):"""删掉上一次检查判定为 gone 的链接。只接受**刚跑完的那一次**检查的 `check_id`:拿一份放了半天的清单去删,删的可能是@@ -252,9 +279,12 @@ def w_links_prune(contract: LinkContract, body, *, progress=None):job = contract.link_prune_jobdef work(job_id):result = w_links_prune(contract, {**body, "background": False},- progress=lambda **fields: job.update(job_id, **fields))+ progress=lambda **fields: job.update(job_id, **fields),+ checkpoint=lambda **fields: job.checkpoint(job_id, **fields), resume_state=job.snapshot())job.update(job_id, **result, status="complete" if result.get("ok") else "failed", completed_at=time.time())- return job.start(work, restart=True, initial={"message": "正在核对失效链接清单"})+ return job.start(work, restart=True, initial={"message": "正在核对失效链接清单"}, recovery=body)+ if (resume_state or {}).get('prune_receipt'):+ return resume_state['prune_receipt']state = contract.link_check.snapshot()if state is None or state["status"] != "complete":return {"ok": False, "error": "没有已完成的检查结果"}@@ -268,12 +298,15 @@ def w_links_prune(contract: LinkContract, body, *, progress=None):confirmed, recovered = [], []for index, item in enumerate(planned):if progress:- progress(checked=index, total=len(planned), message=f"重验失效链接:已检查 {index} / {len(planned)} 条")+ progress(checked=max(index, (resume_state or {}).get('checked', 0)), total=len(planned),+ message=f"重验失效链接:已检查 {index} / {len(planned)} 条")status, note = _probe(item["url"])if link_verdict(status, note) == "gone":confirmed.append({**item, "note": _note(status, note)})else:recovered.append(item)+ if checkpoint and not checkpoint(checked=max(index + 1, (resume_state or {}).get('checked', 0)), total=len(planned)):+ raise InterruptedError('链接清理已停止')removed = marked = 0if progress:@@ -287,5 +320,8 @@ def w_links_prune(contract: LinkContract, body, *, progress=None):if state is not None:processed = {item["id"] for item in planned}state["gone"] = [item for item in state["gone"] if item["id"] not in processed]- return {"ok": True, "removed": removed, "marked": marked, "recovered": len(recovered),- "entities": len({item["entity"] for item in confirmed})}+ receipt = {"ok": True, "removed": removed, "marked": marked, "recovered": len(recovered),+ "entities": len({item["entity"] for item in confirmed})}+ if checkpoint:+ checkpoint(prune_receipt=receipt)+ return receiptdiff --git a/src/peach/web_media_repair.py b/src/peach/web_media_repair.pyindex 6b7187b7a..94155cbfa 100644--- a/src/peach/web_media_repair.py+++ b/src/peach/web_media_repair.py@@ -31,7 +31,7 @@ from .platform import translate_ledger_path#: 判定结论的落盘位置,放在边车自己的目录里:它们同生同灭,清缓存时一起走。SCAN_CACHE_NAME = "mp4-repair-scan.json"-#: 只有这两个结论值得记:需要修的当场就修了,修完有边车或新文件为证。+#: 文件身份对应的判定缓存;修复产物由边车或新文件核验。CLEAN = "clean"UNAVAILABLE = "unavailable"@@ -116,14 +116,18 @@ def _survey(contract, job, job_id: str, rows, cache: dict[str, str]) -> list[tupif store.lookup(asset_id, source) is not None or cache.get(key) in (CLEAN, UNAVAILABLE):_advance(job, job_id, index, skipped=True)continue- try:- kind = _diagnose(contract, source, duration)- except OSError:- kind = None+ kind = cache.get(key)+ if kind not in (HEADER, INDEX):+ try:+ kind = _diagnose(contract, source, duration)+ except OSError:+ continueif kind:found.append((asset_id, raw, kind))- else:- cache[key] = CLEAN+ cache[key] = kind or CLEAN+ _save_cache(contract, cache)+ if not job.checkpoint(job_id, survey_checked=index, survey_cache=dict(cache)):+ raise InterruptedError('媒体检查已停止')_advance(job, job_id, index, skipped=False)return found@@ -132,8 +136,8 @@ def _advance(job, job_id: str, checked: int, *, skipped: bool) -> None:with job.editing(job_id) as state:if state is None:return- state["checked"] = checked- if skipped:+ state["checked"] = max(state.get('checked', 0), checked)+ if skipped and checked > state.get('survey_checked', 0):state["skipped"] = state.get("skipped", 0) + 1@@ -201,7 +205,9 @@ def _repair(contract, job, job_id: str, found, cache: dict[str, str]) -> None:"""逐个修。一部片要整片读一遍,所以这里刻意不并发。"""store = contract.header_repairstools = _recovery_tools(contract) if any(kind == INDEX for *_, kind in found) else None- job.update(job_id, stage="修复", checked=0, total=len(found), repaired=0, failed=0)+ previous = job.snapshot() or {}+ job.update(job_id, stage="修复", checked=previous.get('repaired', 0),+ total=len(found) + previous.get('repaired', 0))for index, (asset_id, raw, kind) in enumerate(found, start=1):if job.snapshot() is None:return@@ -221,20 +227,26 @@ def _repair(contract, job, job_id: str, found, cache: dict[str, str]) -> None:with job.editing(job_id) as state:if state is None:return- state["checked"] = index+ state["checked"] = previous.get('repaired', 0) + indexstate["repaired"] = state.get("repaired", 0) + (1 if made else 0)state["failed"] = state.get("failed", 0) + (0 if made else 1)state["missing_tool"] = state.get("missing_tool", 0) + (1 if made is None else 0)state["message"] = source.name[:80]_save_cache(contract, cache)+ state = job.snapshot() or {}+ if not job.checkpoint(job_id, **{key: state.get(key, 0) for key in+ ('checked', 'total', 'repaired', 'failed', 'missing_tool')}):+ raise InterruptedError('媒体修复已停止')def run_media_repair(contract, job_id: str, scope: tuple[str, list]) -> None:"""一轮完整的批量修复。异常由 `BackgroundJob` 变成可轮询的失败状态。"""job = contract.media_repair_jobcache = _load_cache(contract)+ cache.update((job.snapshot() or {}).get('survey_cache') or {})rows = _candidates(contract, scope)- job.update(job_id, stage="扫描", total=len(rows), checked=0)+ previous = job.snapshot() or {}+ job.update(job_id, stage="扫描", total=len(rows), checked=previous.get('survey_checked', 0))found = _survey(contract, job, job_id, rows, cache)_save_cache(contract, cache)if job.snapshot() is None:@@ -276,7 +288,7 @@ def w_media_repair(contract, body=None):body = body or {}job = contract.media_repair_jobif body.get("stop") is True:- job.stop()+ job.stop(resume=False)return _public(None)if body.get("status_only") is True:return _public(job.snapshot())@@ -291,4 +303,5 @@ def w_media_repair(contract, body=None):initial={"library": library, "stage": "扫描", "total": 0, "checked": 0, "found": 0,"repaired": 0, "failed": 0, "missing_tool": 0, "skipped": 0, "message": ""},restart=body.get("restart") is True,+ recovery={'library': library, 'restart': True},))diff --git a/src/peach/web_organize.py b/src/peach/web_organize.pyindex 5e1ad85e9..ae052f76b 100644--- a/src/peach/web_organize.py+++ b/src/peach/web_organize.py@@ -46,7 +46,7 @@ def q_organize(contract, _args=None) -> dict:latest = organize.latest_batch(contract.candidate_root)declared = location_roots()return {- **state,+ **{key: value for key, value in state.items() if key not in ('plan', 'batch_stamp')},"status": state.get("status", "idle"),"locations": [{"location": location, "roots": list(declared.get(location, ()))}for location in ORGANIZE_LOCATIONS if declared.get(location)],@@ -108,37 +108,56 @@ def w_organize_apply(contract, body) -> dict:job = contract.organize_jobdef work(job_id: str) -> None:- with contract.read_connection() as connection:- plan = organize.build_plan(connection, location=location,- file_template=file_template,- dir_template=dir_template)+ batch_stamp = (job.snapshot() or {}).get('batch_stamp') or f"{time.strftime('%Y%m%d-%H%M%S')}-{job_id}"+ plan = (job.snapshot() or {}).get('plan')+ if plan is None:+ with contract.read_connection() as connection:+ plan = organize.build_plan(connection, location=location,+ file_template=file_template,+ dir_template=dir_template)+ if not job.checkpoint(job_id, plan=plan, batch_stamp=batch_stamp):+ raise InterruptedError('整理已停止')job.update(job_id, total=plan["counts"]["change"], stage="正在整理")result = _run_write(contract, job_id, lambda connection: organize.apply_plan(connection, plan["rows"], generated_root=contract.candidate_root,- progress=lambda **fields: job.update(job_id, **fields)))+ stamp=batch_stamp,+ progress=lambda **fields: job.checkpoint(job_id, **fields)))job.update(job_id, **result, counts=plan["counts"], status="complete",completed_at=time.time())- return job.start(work, restart=True,- initial={"stage": "正在核对计划", "location": location})+ started = job.start(work, restart=True,+ initial={"stage": "正在核对计划", "location": location},+ recovery={'confirm': True, 'location': location, 'file_template': file_template,+ 'dir_template': dir_template, 'action': 'apply'})+ return {key: value for key, value in started.items() if key not in ('plan', 'batch_stamp')}def w_organize_rollback(contract, body) -> dict:"""把上一批整理原样退回去。"""if body.get("confirm") is not True:raise ValueError("回滚需要确认")- latest = organize.latest_batch(contract.candidate_root)+ from .job_recovery import current_job_id, read+ saved = read(contract.db_path, 'organize') if current_job_id.get() else {}+ batch = (saved.get('checkpoint') or {}).get('rollback_batch')+ latest = (organize.batch_log_dir(contract.candidate_root) / batch if batch+ else organize.latest_batch(contract.candidate_root))if latest is None:raise ValueError("没有可以回滚的批次")job = contract.organize_jobdef work(job_id: str) -> None:+ if not job.checkpoint(job_id, rollback_batch=latest.name):+ raise InterruptedError('整理回滚已停止')+ if not latest.exists():+ job.update(job_id, status='complete', restored=0, failed=0)+ returnresult = _run_write(contract, job_id, lambda connection: organize.rollback_batch(- connection, latest, progress=lambda **fields: job.update(job_id, **fields)))+ connection, latest, progress=lambda **fields: job.checkpoint(job_id, **fields)))job.update(job_id, **result, status="complete", completed_at=time.time())return job.start(work, restart=True,- initial={"stage": "正在回滚上一批", "batch": latest.name})+ initial={"stage": "正在回滚上一批", "batch": latest.name},+ recovery={'confirm': True, 'action': 'rollback'})__all__ = ["ORGANIZE_LOCATIONS", "TemplateError", "q_organize", "w_organize_apply",diff --git a/src/peach/web_resource_sync.py b/src/peach/web_resource_sync.pyindex 7075016ae..1617514f8 100644--- a/src/peach/web_resource_sync.py+++ b/src/peach/web_resource_sync.py@@ -268,33 +268,44 @@ def _keeps_records(contract: ResourceSyncContract, rows: Sequence) -> set[int]:def _scan_sources(contract: ResourceSyncContract,progress: Callable[[dict], None] | None = None,+ *, resume_state=None, checkpoint=None,) -> dict:"""只读核对每个在线来源:文件已不在盘上的行(分删除与标已消失两档)、空文件夹、读不了的目录。"""with contract.read_connection() as connection:totals = {row[0]: int(row[1]) for row in connection.execute("SELECT location,count(*) FROM asset WHERE path IS NOT NULL ""AND COALESCE(disposal,'')<>? GROUP BY location", (VANISHED,))}- vanished_ids: list[int] = []- keep_ids: set[int] = set()- empty_dirs: dict[str, list[str]] = {}- sources = []+ previous = (resume_state or {}).get('scan_checkpoint') or {}+ parts = dict(previous.get('parts') or {})+ sources = [part['source'] for part in parts.values()]+ completed = {row['location'] for row in sources if row['online'] and not row['unreadable']}for location in configured_resource_locations():+ if location in completed:+ continueonline = source_is_online(location)source = {"location": location, "online": online, "total": totals.get(location, 0),"missing": 0, "vanish": 0, "empty": 0, "unreadable": 0}+ part = dict(source=source, vanished_ids=[], keep_ids=[], empty_dirs=[])if online:unreadable: set[str] = set()gone = vanished_asset_rows(contract, location, unreadable_dirs=unreadable)empties = _empty_directories(_source_roots(location), unreadable)- vanished_ids.extend(int(row["id"]) for row in gone)+ part['vanished_ids'] = [int(row['id']) for row in gone]keep = _keeps_records(contract, gone)- keep_ids.update(keep)- empty_dirs[location] = [os.fspath(path) for path in empties]+ part['keep_ids'] = sorted(keep)+ part['empty_dirs'] = [os.fspath(path) for path in empties]source.update(missing=len(gone), vanish=len(keep), empty=len(empties),unreadable=len(unreadable))- sources.append(source)+ sources = [row for row in sources if row['location'] != location] + [source]+ parts[location] = partif progress is not None:progress(source)+ if checkpoint and not checkpoint(scan_checkpoint={'parts': parts}, sources=sources):+ raise InterruptedError('资源检查已停止')+ sources = [part['source'] for part in parts.values()]+ vanished_ids = sorted({value for part in parts.values() for value in part['vanished_ids']})+ keep_ids = {value for part in parts.values() for value in part['keep_ids']}+ empty_dirs = {location: part['empty_dirs'] for location, part in parts.items() if part['source']['online']}return {"sources": sources, "vanished_ids": vanished_ids, "keep_ids": keep_ids,"empty_dirs": empty_dirs}@@ -451,13 +462,14 @@ def _resource_orphan_plan(}-def clean_resource_orphans(contract: ResourceSyncContract, *, progress=None) -> dict:+def clean_resource_orphans(contract: ResourceSyncContract, *, progress=None, checkpoint=None, resume_state=None) -> dict:if progress:progress(checked=0, total=None, message="正在核对可重建缓存清单")plan = _resource_orphan_plan(contract)- removed = 0- reclaimed = 0- blocked = []+ previous = (resume_state or {}).get('cache_checkpoint') or {}+ removed = previous.get('cache_removed', 0)+ reclaimed = previous.get('bytes_reclaimed', 0)+ blocked = list(previous.get('cache_blocked') or [])for index, (kind, path, size) in enumerate(plan["files"]):if progress:progress(checked=index, total=len(plan["files"]), message=f"清理缓存:已处理 {index} / {len(plan['files'])} 个")@@ -468,6 +480,9 @@ def clean_resource_orphans(contract: ResourceSyncContract, *, progress=None) ->except OSError as error:blocked.append({"kind": kind, "name": path.name,"reason": error.strerror or str(error)})+ if checkpoint and not checkpoint(cache_checkpoint=dict(cache_removed=removed, bytes_reclaimed=reclaimed,+ cache_blocked=blocked)):+ raise InterruptedError('缓存清理已停止')if progress:progress(checked=0, total=None, message="正在整理缓存目录并汇总释放空间")for directory in sorted(plan["dirs"], key=lambda item: len(item.parts), reverse=True):@@ -505,9 +520,11 @@ def _run_resource_scan(contract: ResourceSyncContract, scan_id: str) -> None:def progress(source: dict) -> None:with job.editing(scan_id) as state:if state is not None:+ state['sources'] = [row for row in state['sources'] if row['location'] != source['location']]state["sources"].append(dict(source))- scan = _scan_sources(contract, progress)+ scan = _scan_sources(contract, progress, resume_state=job.snapshot(),+ checkpoint=lambda **fields: job.checkpoint(scan_id, **fields))# 候选名单和空目录路径只留在任务状态里,不进 `result`:公开投影整个下发 `result`,# 物理路径不出服务端。job.update(scan_id, status="complete", result=_scan_result(contract, scan),@@ -538,6 +555,7 @@ def _background_resource_scan(contract: ResourceSyncContract, restart: bool = Falambda scan_id: _run_resource_scan(contract, scan_id),initial={"sources": [], "result": None, "vanished_ids": [], "empty_dirs": {}},restart=restart,+ recovery={'restart': True},))@@ -640,7 +658,7 @@ def _mark_record_holders(contract: ResourceSyncContract,return marked, reattached, [row for row in rows if int(row["id"]) not in kept]-def w_resource_sync_apply(contract: ResourceSyncContract, body, *, progress=None):+def w_resource_sync_apply(contract: ResourceSyncContract, body, *, progress=None, checkpoint=None, resume_state=None):"""处理这一轮检查报出的失效记录与空文件夹,再清孤儿缓存。文件已不在盘上的行不进回收站:在库且带个人记录的标「已消失」,记录留着等接回@@ -653,24 +671,44 @@ def w_resource_sync_apply(contract: ResourceSyncContract, body, *, progress=Noneif body.get("confirm") is not True:raise ValueError("resource sync requires confirmation")scan_id = str(body.get("scan_id") or "")- _completed_scan(contract, scan_id)+ from .job_recovery import current_job_id, read+ recovering = (read(contract.db_path, 'resource-apply').get('checkpoint') or {}) if current_job_id.get() else {}+ if not recovering.get('apply_receipt'):+ _completed_scan(contract, scan_id)if body.get("background"):job = contract.resource_apply_jobdef work(job_id):result = w_resource_sync_apply(contract, {**body, "background": False},- progress=lambda **fields: job.update(job_id, **fields))+ progress=lambda **fields: job.update(job_id, **fields),+ checkpoint=lambda **fields: job.checkpoint(job_id, **fields), resume_state=job.snapshot())job.update(job_id, **result, status="complete", completed_at=time.time())- return job.start(work, restart=True, initial={"message": "正在逐条复核失效记录"})+ return job.start(work, restart=True, initial={"message": "正在逐条复核失效记录"}, recovery=body)+ if (resume_state or {}).get('apply_receipt'):+ return resume_state['apply_receipt']state = _completed_scan(contract, scan_id)report = progress or (lambda **_fields: None)- report(checked=0, total=None, message="正在逐条复核失效记录")- marked, reattached, confirmed = _mark_record_holders(- contract, _recheck_vanished(contract, state["vanished_ids"]))- purge = {"purged": 0, "blocked": [], "empty_dirs_removed": 0}- if confirmed:- report(checked=0, total=None, message=f"正在永久删除 {len(confirmed)} 条失效记录")- from .web_batch import purge_vanished_rows # web_batch 在模块顶部 import 本模块- purge = purge_vanished_rows(contract, confirmed)+ saved = (resume_state or {}).get('apply_checkpoint') or {}+ processed = set(saved.get('processed') or [])+ report(checked=len(processed), total=len(state['vanished_ids']), message="正在逐条复核失效记录")+ marked, reattached = list(saved.get('marked') or []), list(saved.get('reattached') or [])+ purge = dict(saved.get('purge') or {'purged': 0, 'blocked': [], 'empty_dirs_removed': 0})+ pending = [value for value in state['vanished_ids'] if value not in processed]+ for offset in range(0, len(pending), 100):+ selected = pending[offset:offset + 100]+ added, attached, confirmed = _mark_record_holders(contract, _recheck_vanished(contract, selected))+ marked.extend(added)+ reattached.extend(attached)+ if confirmed:+ from .web_batch import purge_vanished_rows+ result = purge_vanished_rows(contract, confirmed)+ for field in ('purged', 'empty_dirs_removed'):+ purge[field] += int(result.get(field) or 0)+ purge['blocked'].extend(result.get('blocked') or [])+ processed.update(selected)+ report(checked=len(processed), total=len(state['vanished_ids']), message='正在处理失效记录')+ if checkpoint and not checkpoint(apply_checkpoint=dict(processed=sorted(processed), marked=marked,+ reattached=reattached, purge=purge)):+ raise InterruptedError('资源清理已停止')report(checked=0, total=None, message="正在删除空文件夹")dirs_removed, dir_errors = 0, 0for location, paths in state["empty_dirs"].items():@@ -681,14 +719,14 @@ def w_resource_sync_apply(contract: ResourceSyncContract, body, *, progress=Nonedirs_removed += removeddir_errors += errorscontract.cache_bust()- cleanup = clean_resource_orphans(contract, progress=progress) if body.get("clean_cache", True) else {+ cleanup = clean_resource_orphans(contract, progress=progress, checkpoint=checkpoint,+ resume_state=resume_state) if body.get("clean_cache", True) else {"cache_removed": 0, "bytes_reclaimed": 0, "cache_blocked": [],}blocked = [{"id": item["id"], "name": PureWindowsPath(item["path"]).name,"reason": item["reason"]} for item in purge["blocked"]]# 这一轮的清单已经照着清过:读数不再是盘上的现状,结果区换成清理回执。- contract.resource_scan.update(scan_id, applied_at=time.time())- return {+ receipt = {"ok": True, "sources": state["result"]["sources"],"purged": int(purge["purged"]), "vanished": len(marked) - len(reattached),"reattached": len(reattached),@@ -697,6 +735,10 @@ def w_resource_sync_apply(contract: ResourceSyncContract, body, *, progress=None"dir_errors": dir_errors, "unreadable": int(state["result"].get("unreadable") or 0),**cleanup,}+ if checkpoint:+ checkpoint(apply_receipt=receipt)+ contract.resource_scan.update(scan_id, applied_at=time.time())+ return receiptdef w_purge_missing(contract: ResourceSyncContract, body):diff --git a/src/peach/web_scraping.py b/src/peach/web_scraping.pyindex d053d577a..46933122c 100644--- a/src/peach/web_scraping.py+++ b/src/peach/web_scraping.py@@ -265,4 +265,4 @@ def w_scraping_amane_check(contract, body):def w_scraping_amane_rebuild(contract, body):"""按锁重建桥的 venv。首次要下载约 98 MB,放后台跑,页面轮询 `q_scraping_amane_bridge` 那条读接口。"""from . import metadata_amane- return contract.amane_bridge_job.start_result(lambda: metadata_amane.rebuild(contract.tools_root))+ return contract.amane_bridge_job.start_result(lambda: metadata_amane.rebuild(contract.tools_root), recovery={})diff --git a/src/peach/web_state.py b/src/peach/web_state.pyindex 26717eaa6..0d6cb2552 100644--- a/src/peach/web_state.py+++ b/src/peach/web_state.py@@ -346,6 +346,9 @@ class WebContract:self.link_check.stop()self.feed_job.stop()self.want_job.stop()+ self.organize_job.stop()+ self.media_repair_job.stop()+ self.amane_bridge_job.stop()if hasattr(self, "jav_discovery_metadata"):self.jav_discovery_metadata.stop()self.followups.stop()diff --git a/src/peach/web_stats.py b/src/peach/web_stats.pyindex 75681c1a8..fd8a852bc 100644--- a/src/peach/web_stats.py+++ b/src/peach/web_stats.py@@ -135,20 +135,45 @@ def q_taste(contract: WebContract, args=None):return payload-def w_taste_refresh(contract: WebContract, body, *, progress=None):+def w_taste_refresh(contract: WebContract, body, *, progress=None, checkpoint=None, resume_state=None):if body.get("background"):job = contract.taste_refresh_jobdef work(job_id):result = w_taste_refresh(contract, {**body, "background": False},- progress=lambda **fields: job.update(job_id, **fields))+ progress=lambda **fields: job.update(job_id, **fields),+ checkpoint=lambda **fields: job.checkpoint(job_id, **fields), resume_state=job.snapshot())job.update(job_id, **result, status="complete", message="部分浏览记录未取得" if result["warnings"] else "口味分析已更新", completed_at=time.time())return job.start(work, restart=True, initial={- "stage": "discovering", "message": "正在查找 Peach 主机上的浏览器资料"})+ "stage": "discovering", "message": "正在查找 Peach 主机上的浏览器资料"},+ recovery={'background': True, 'window': body.get('window') or 'all'})window = str(body.get("window") or "all")sources = enabled_history_sources(contract.taste_history_store)- results = refresh_history(sources, contract.taste_history_store, progress=progress) if sources else []- results.extend(refresh_bookmarks(enabled_bookmark_sources(contract.taste_history_store),- contract.taste_history_store, progress=progress))+ results = list((resume_state or {}).get('refresh_results') or [])+ completed = set((resume_state or {}).get('completed_profiles') or [])+ plans = (+ ('history', sources, refresh_history),+ ('bookmarks', enabled_bookmark_sources(contract.taste_history_store), refresh_bookmarks))+ profile_keys = (resume_state or {}).get('profile_keys')+ if profile_keys is None:+ profile_keys = [f'{kind}:{source.browser}:{source.profile}:{source.path}'+ for kind, selected, _refresh in plans for source in selected]+ if checkpoint and not checkpoint(profile_keys=profile_keys):+ raise InterruptedError('口味读取已停止')+ def profile_progress(**fields):+ if progress:+ progress(**{**fields, 'checked': len(completed), 'total': len(profile_keys)})+ for kind, selected, refresh in plans:+ for source in selected:+ key = f'{kind}:{source.browser}:{source.profile}:{source.path}'+ if key in completed or key not in profile_keys:+ continue+ rows = refresh([source], contract.taste_history_store, progress=profile_progress)+ results.extend(rows)+ if rows and all(row.get('status') != 'failed' for row in rows):+ completed.add(key)+ if checkpoint and not checkpoint(refresh_results=results.copy(), completed_profiles=sorted(completed),+ checked=len(completed), total=len(profile_keys)):+ raise InterruptedError('口味读取已停止')if contract.taste_history_store.is_file():analysis = analyze_history(contract.taste_history_store, contract.taste_history_root, progress=progress)write_manifest(contract.taste_history_manifest, results, analysis)diff --git a/src/peach/web_timeline_thumbnails.py b/src/peach/web_timeline_thumbnails.pyindex bd35f300c..062764262 100644--- a/src/peach/web_timeline_thumbnails.py+++ b/src/peach/web_timeline_thumbnails.py@@ -111,7 +111,8 @@ def w_thumbnail_jobs(contract, body):active=lambda: ((contract.thumbnail_job.snapshot() or {}).get('job_id') == job_idand not metadata_only(contract.follow_state_root)),report=lambda **fields: contract.thumbnail_job.update(job_id, **_with_progress(fields)),- guard=guard)+ guard=guard, resume_state=contract.thumbnail_job.snapshot(),+ checkpoint=lambda **fields: contract.thumbnail_job.checkpoint(job_id, **fields))# 磁盘触线停下来的那一轮不能记成完成:队列后面还剩几千部,而页面上「完成」# 和「跑完了」是同一个意思。按停是用户自己做的,不算故障。stopped = str(result.get('stopped') or '')diff --git a/src/peach/web_wants.py b/src/peach/web_wants.pyindex 2c3376c0c..150c7c7ca 100644--- a/src/peach/web_wants.py+++ b/src/peach/web_wants.py@@ -97,8 +97,10 @@ def _provider(contract):def scrape(contract, job_id: str | None = None) -> dict:"""把积压的想要取一遍。一轮里新登记的也会接着取到:每取完一批再读一次积压。"""- done: set[int] = set()- fetched = covers = 0+ previous = (contract.want_job.snapshot() or {}) if job_id else {}+ done = set(previous.get('completed_wants') or [])+ completed = set(done)+ fetched, covers = previous.get('fetched', 0), previous.get('covers', 0)provider = _provider(contract)while len(done) < MAX_PER_RUN:with contract.database.read_connection() as connection:@@ -107,12 +109,20 @@ def scrape(contract, job_id: str | None = None) -> dict:breakfor want_id in batch:if job_id:- contract.want_job.update(job_id, checked=len(done), total=len(done) + len(batch),- current=want_id)+ if not contract.want_job.update(job_id, checked=len(done), total=len(done) + len(batch),+ current=want_id):+ raise InterruptedError('想要资料任务已停止')got, covered = _scrape_one(contract, provider, want_id)fetched += gotcovers += covereddone.add(want_id)+ with contract.database.read_connection() as connection:+ saved = wants.get(connection, want_id)+ if covered and saved and saved['scraped_at'] is not None and saved['scrape_error'] is None:+ completed.add(want_id)+ if job_id and not contract.want_job.checkpoint(job_id, completed_wants=sorted(completed),+ checked=len(done), fetched=fetched, covers=covers):+ raise InterruptedError('想要资料任务已停止')contract.cache_bust()return {"ok": True, "total": len(done), "fetched": fetched, "covers": covers}@@ -120,7 +130,8 @@ def scrape(contract, job_id: str | None = None) -> dict:def start_scrape(contract, *, trigger: str = "manual") -> dict | None:"""有积压就起一轮 `want_job`;没有就不起,任务中心里不留空跑的一行。"""with contract.database.read_connection() as connection:- if not backlog(connection, contract.cover_root):+ from .job_recovery import current_job_id+ if not current_job_id.get() and not backlog(connection, contract.cover_root):return Nonedef work(job_id):diff --git a/tests/test_feeds_web.py b/tests/test_feeds_web.pyindex d98d4eb66..6b17c1a96 100644--- a/tests/test_feeds_web.py+++ b/tests/test_feeds_web.py@@ -129,6 +129,29 @@ class FeedWebFixture(unittest.TestCase):class FeedWebTest(FeedWebFixture):+ def test_recovery_keeps_source_selection_progress_and_discovered_codes(self):+ from peach import job_recovery+ first = self._add()+ self.url = 'https://javdb.com/actors/Smp2'+ second = self._add()+ run = self.contract.task_runs.start('feed-check', trigger='manual', total=2)+ self.contract.task_runs.suspend(run.id)+ job_recovery.save(self.db_path, 'feed-check', 'original', {'background': True}, run_id=run.id, waiting=True)+ job_recovery.checkpoint(self.db_path, 'feed-check', 'original', dict(source_ids=[first, second],+ checked=1, total=2, added=1, results=[{'source': first, 'ok': True, 'codes': ['AAA-001']}]))+ seen = []+ def poll(contract, transport, source):+ seen.append((source['id'], contract.feed_job.snapshot()['checked']))+ return {'source': source['id'], 'ok': True, 'codes': ['BBB-002']}+ with patch.object(web_feeds, 'poll_source', side_effect=poll), \+ patch.object(web_feeds, '_transport', return_value=self.transport):+ job_recovery.resume(self.db_path, {'feed-check': lambda body: web_feeds.w_feed_check(self.contract, body)})+ self.contract.feed_job.thread.join(5)+ final = self.contract.feed_job.snapshot()+ self.assertEqual(seen, [(second, 1)])+ self.assertEqual((final['job_id'], final['run_id'], final['checked'], final['added']), ('original', run.id, 2, 2))+ self.assertEqual({row['source'] for row in final['results']}, {first, second})+def test_javdb_provenance_is_unavailable_to_cover_readers(self):from PIL import Imagetarget = self.contract.cover_root / 'SSIS-950.jpg'diff --git a/tests/test_follow_web.py b/tests/test_follow_web.pyindex b6c064b72..25cc1745f 100644--- a/tests/test_follow_web.py+++ b/tests/test_follow_web.py@@ -35,6 +35,47 @@ MOMENT = datetime(2026, 8, 25, 9, 0, tzinfo=timezone.utc)class FollowContractTests(unittest.TestCase):+ def test_service_recovery_skips_completed_sources_with_the_same_progress(self):+ from peach import job_recovery+ first = self._seed(ref='first')+ second = self._seed(ref='second')+ job = self.contract.follow_job+ run = self.contract.task_runs.start('follow-check', trigger='manual', total=2)+ self.contract.task_runs.suspend(run.id)+ job_recovery.save(self.db, 'follow-check', 'original', {'background': True}, run_id=run.id, waiting=True)+ job_recovery.checkpoint(self.db, 'follow-check', 'original', dict(+ source_ids=[first, second], checked=1, total=2, results=[{'source': first, 'ok': True, 'added': 4}]))+ seen = []+ def check(row, **kwargs):+ seen.append((row['id'], job.snapshot()['checked'], job.snapshot()['run_id']))+ return {'source': row['id'], 'ok': True, 'page': 0}+ with mock.patch.object(web_follow, 'run_check', side_effect=check), \+ mock.patch.object(web_follow, '_check_payload', side_effect=lambda result: result):+ self.assertEqual(job_recovery.resume(self.db, {'follow-check':+ lambda body: web_follow.w_follow_check(self.contract, body)}), ['follow-check'])+ job.thread.join(5)+ self.assertEqual(seen, [(second, 1, run.id)])+ self.assertEqual(job.snapshot()['job_id'], 'original')+ self.assertEqual(job.snapshot()['checked'], 2)+ self.assertEqual(job.snapshot()['results'][0]['added'], 4)++ def test_rewind_recovery_uses_the_saved_page_cursor(self):+ from peach import job_recovery+ source = self._seed()+ job_recovery.save(self.db, 'follow-check', 'original',+ {'background': True, 'older': True, 'rewind': True, 'backfill_all': True}, waiting=True)+ job_recovery.checkpoint(self.db, 'follow-check', 'original',+ {'source_ids': [source], 'backfill_pages': {str(source): 7}})+ seen = []+ def check(row, **kwargs):+ seen.append(row['backfill_page'])+ return {'source': row['id'], 'ok': True, 'page': 8, 'exhausted': True}+ with mock.patch.object(web_follow, 'run_check', side_effect=check), \+ mock.patch.object(web_follow, '_check_payload', side_effect=lambda result: result):+ job_recovery.resume(self.db, {'follow-check': lambda body: web_follow.w_follow_check(self.contract, body)})+ self.contract.follow_job.thread.join(5)+ self.assertEqual(seen, [7])+def test_background_check_exposes_progress_and_deduplicates(self):self._seed()entered, release = threading.Event(), threading.Event()diff --git a/tests/test_followups.py b/tests/test_followups.pyindex d9b86e855..526d42cb3 100644--- a/tests/test_followups.py+++ b/tests/test_followups.py@@ -260,7 +260,7 @@ class RestartTests(LedgerTestCase):self.assertEqual(self.store.requeue_followups(), queued)run = self.store.get(queued[0])self.assertEqual(run.status, "pending")- self.assertIsNone(run.started_at)+ self.assertEqual(run.started_at, '2026-09-22T00:00:00.000Z')def test_a_queued_followup_survives_and_still_blocks_its_key(self):parent = self.parent()diff --git a/tests/test_jav_discovery_metadata.py b/tests/test_jav_discovery_metadata.pyindex 10169bd57..513c40a6d 100644--- a/tests/test_jav_discovery_metadata.py+++ b/tests/test_jav_discovery_metadata.py@@ -1,5 +1,6 @@"""发现资料边车、固定批次、预算和失败状态的行为契约。"""import tempfile+import jsonimport unittestimport threadingfrom pathlib import Path@@ -11,6 +12,20 @@ from peach.web_jav_discovery import _metadata_responseclass DiscoveryMetadataTests(unittest.TestCase):+ def test_restart_retains_batch_identity_budget_and_completed_codes(self):+ self.cache.remember('EXM-001', {'number': 'EXM-001', 'tags': [], 'score': 4, 'duration': 120})+ self.path.with_suffix('.job.json').write_text(json.dumps(dict(+ job_id='original', status='running', codes=['EXM-001', 'EXM-002'], refresh=True,+ completed=1, completed_codes=['EXM-001'], attempted_codes=['EXM-001'], requests=3, bytes=120,+ active_seconds=10)), encoding='utf-8')+ with patch.object(self.api, 'movie', return_value={'number': 'EXM-002', 'tags': [], 'score': 4, 'duration': 120}) as movie:+ restarted = DiscoveryMetadata(self.path, api=self.api, clock=lambda: self.now[0])+ self.addCleanup(restarted.stop)+ restarted.job.thread.join(5)+ final = restarted.job.snapshot()+ movie.assert_called_once_with('EXM-002', refresh=True)+ self.assertEqual((final['job_id'], final['completed'], final['requests'], final['bytes']), ('original', 2, 3, 120))+def setUp(self):temporary = tempfile.TemporaryDirectory()self.addCleanup(temporary.cleanup)diff --git a/tests/test_metadata_library.py b/tests/test_metadata_library.pyindex 781ddb4fd..3e6b38af6 100644--- a/tests/test_metadata_library.py+++ b/tests/test_metadata_library.py@@ -2985,6 +2985,7 @@ class LibraryPreviewCompletionTests(unittest.TestCase):result = self.run_processing(stage='previews', asset_ids=[1, 5], resume_state=saved)self.assertEqual(seen, [5])self.assertEqual(result['status'], 'complete')+ self.assertEqual(result['previews'], 2)def test_real_local_video_produces_a_snapshot_and_card_poster(self):import subprocessdiff --git a/tests/test_organize.py b/tests/test_organize.pyindex 8fa802f95..04019051e 100644--- a/tests/test_organize.py+++ b/tests/test_organize.py@@ -2,6 +2,7 @@from __future__ import annotationsimport sqlite3+import jsonimport tempfileimport unittestfrom contextlib import closing@@ -221,6 +222,23 @@ class PlanTests(unittest.TestCase):class ApplyTests(unittest.TestCase):"""执行与回滚都真的动文件:`translate_ledger_path` 指到临时目录,两个平台一致。"""+ def test_committed_moves_keep_one_journal_and_resume_without_renaming_twice(self):+ for index in (1, 2):+ self._file(f'old/{index}.mp4')+ _seed(self.db_path, [dict(id=index, path=f'R:\\media\\old\\{index}.mp4',+ name=f'{index}.mp4', code=f'ABC-00{index}') for index in (1, 2)])+ plan = _plan(self.db_path, file_template='{number}')+ def stop(**fields):+ raise InterruptedError('模拟整理中断')+ with closing(sqlite3.connect(self.db_path)) as connection, self.assertRaises(InterruptedError):+ organize.apply_plan(connection, plan['rows'], generated_root=self.holder, stamp='resume', progress=stop)+ with closing(sqlite3.connect(self.db_path)) as connection:+ result = organize.apply_plan(connection, plan['rows'], generated_root=self.holder, stamp='resume')+ self.assertEqual((result['moved'], result['failed']), (2, 0))+ journal = json.loads(Path(result['log_path']).read_text(encoding='utf-8'))+ self.assertEqual(len(journal['entries']), 2)+ self.assertEqual(len(list(organize.batch_log_dir(self.holder).glob('*.json'))), 1)+def setUp(self):self.holder = Path(tempfile.mkdtemp(prefix="peach-organize-apply-")).resolve()self.addCleanup(lambda: __import__("shutil").rmtree(self.holder, ignore_errors=True))diff --git a/tests/test_sample_images.py b/tests/test_sample_images.pyindex 0ef188c23..a65f329e2 100644--- a/tests/test_sample_images.py+++ b/tests/test_sample_images.py@@ -114,7 +114,8 @@ class LedgerCase(unittest.TestCase):"SELECT code,position,url,site,source FROM code_sample_image ORDER BY code,position")]def run_followup(self, fetchers: dict, run_id: int = 7) -> dict:- handle = SimpleNamespace(run_id=run_id, progress=lambda **_kwargs: None)+ handle = SimpleNamespace(run_id=run_id, progress=lambda **_kwargs: None,+ saved={}, checkpoint=lambda **_kwargs: None)with mock.patch.dict(sample_images.FETCHERS, fetchers, clear=True):return sample_followup.run(self.contract, sample_followup.FOLLOWUP_KEY, handle,transport=object())diff --git a/tests/test_scan.py b/tests/test_scan.pyindex 039dd5eda..a260cf9d1 100644--- a/tests/test_scan.py+++ b/tests/test_scan.py@@ -10,12 +10,14 @@ POSIX 形态那条平台无关,哪台机器上都跑。Windows 形态那条只from __future__ import annotationsimport io+import jsonimport osimport sqlite3import tempfileimport unittestfrom contextlib import redirect_stdoutfrom pathlib import Path, PureWindowsPath+from unittest.mock import patchfrom peach import scanfrom peach.migrations import upgrade@@ -104,6 +106,56 @@ class WindowsShapeTests(_ScanCase):class PosixShapeTests(_ScanCase):"""macOS:账本里写 `R:\\media\\...`,遍历的是挂载点,读取侧按挂载表翻回去。"""+ def test_resume_skips_committed_directories_and_retains_subtitles_and_new_ids(self):+ (self.media / '创作者/sub/a.srt').write_text('subtitle', encoding='utf-8')+ saved = {}+ def save(state):+ saved.clear()+ saved.update(json.loads(json.dumps(state)))+ if state.get('files') == 2:+ raise InterruptedError('模拟服务退出')+ options = dict(declared_roots={'local': ('R:\\media',)},+ mounts={'local': (self.media,)}, windows=False)+ with self.assertRaises(InterruptedError):+ self._scan('R:\\media', save_checkpoint=save, **options)+ visited = []+ scandir = os.scandir+ def walk(path):+ visited.append(Path(path))+ return scandir(path)+ with patch.object(scan.os, 'scandir', side_effect=walk):+ result, _ = self._scan('R:\\media', resume_state=saved, **options)+ self.assertNotIn(self.media, visited)+ self.assertEqual((result.files, result.total_bytes, result.subtitles), (4, 22, 1))+ self.assertEqual(len(result.new_ids), 4)+ self.assertEqual(len(self._paths()), 4)++ def test_finished_root_checkpoint_does_not_enumerate_the_tree(self):+ saved = []+ options = dict(declared_roots={'local': ('R:\\media',)}, mounts={'local': (self.media,)}, windows=False)+ first, _ = self._scan('R:\\media', save_checkpoint=lambda value: saved.append(value), **options)+ with patch.object(scan.os, 'scandir', side_effect=AssertionError('已完成的根不应再次枚举')):+ second, _ = self._scan('R:\\media', resume_state=json.loads(json.dumps(saved[-1])), **options)+ self.assertEqual(second, first)++ def test_file_batch_cursor_resumes_inside_a_directory(self):+ saved = {}+ def save(state):+ saved.update(json.loads(json.dumps(state)))+ if state.get('cursor'):+ raise InterruptedError('模拟文件批次提交后退出')+ options = dict(declared_roots={'local': ('R:\\media',)}, mounts={'local': (self.media,)}, windows=False)+ with patch.object(scan, 'BATCH_SIZE', 1), self.assertRaises(InterruptedError):+ self._scan('R:\\media', save_checkpoint=save, **options)+ self.assertEqual(saved['files'], 1)+ (self.media / saved['cursor']).write_bytes(b'changed after checkpoint')+ progress = []+ result, _ = self._scan('R:\\media', resume_state=saved,+ progress=lambda count, size: progress.append((count, size)), **options)+ self.assertEqual((result.files, result.total_bytes), (3, 14))+ self.assertTrue(all(count >= 1 for count, _ in progress))+ self.assertEqual(len(self._paths()), 3)+def test_structured_progress_reports_discovered_files_and_bytes(self):progress = []result, _ = self._scan('R:\\media', declared_roots={'local': ('R:\\media',)},diff --git a/tests/test_scripts.py b/tests/test_scripts.pyindex 5d2ed6292..ff757f0df 100644--- a/tests/test_scripts.py+++ b/tests/test_scripts.py@@ -1268,6 +1268,39 @@ class LibraryDirectoryTests(unittest.TestCase):class OperationalScriptTests(unittest.TestCase):+ def test_scrape_codes_resumes_committed_codes_and_retains_candidate_rows(self):+ with tempfile.TemporaryDirectory() as temporary:+ root = Path(temporary).resolve()+ db = self._chain_ledger(root, ['AAA-001', 'BBB-002'])+ output = root / 'candidates.csv'+ arguments = ['--db', str(db), '--out', str(output), '--raw-dir', str(root / 'raw'),+ '--log-dir', str(root / 'logs'), '--sources', 'r18dev', '--delay', '0', '--min-free', '0']+ class Provider:+ calls = []+ interrupt = True+ def query(instance, code, source):+ instance.calls.append(code)+ if code == 'BBB-002' and instance.interrupt:+ raise KeyboardInterrupt('模拟进程退出')+ return OperationalScriptTests._full_payload(code, source)+ provider = Provider()+ with redirect_stdout(io.StringIO()), self.assertRaises(KeyboardInterrupt):+ self.scrape_codes.main(arguments, provider=provider)+ saved = json.loads(output.with_suffix('.resume.json').read_text(encoding='utf-8'))+ self.assertEqual(saved['completed'], ['AAA-001'])+ provider.calls.clear()+ provider.interrupt = False+ try:+ with redirect_stdout(io.StringIO()):+ self.assertEqual(self.scrape_codes.main(arguments, provider=provider), 0)+ finally:+ self.scrape_codes.close_log()+ self.assertEqual(provider.calls, ['BBB-002'])+ with output.open(encoding='utf-8-sig', newline='') as file:+ rows = list(csv.DictReader(file))+ self.assertEqual({row['query'] for row in rows}, {'AAA-001', 'BBB-002'})+ self.assertEqual(len(rows), len({row['item_key'] for row in rows}))+def setUp(self):policy = mock.patch.object(self.sheets, "metadata_only", return_value=False)policy.start()diff --git a/tests/test_timeline_sheets.py b/tests/test_timeline_sheets.pyindex dcbb66114..d8549c445 100644--- a/tests/test_timeline_sheets.py+++ b/tests/test_timeline_sheets.py@@ -153,6 +153,16 @@ class QueueTests(unittest.TestCase):class GenerateLibraryTests(unittest.TestCase):+ def test_resume_keeps_counters_and_does_not_build_committed_assets(self):+ saved = dict(completed_assets=[1], done=1, made=1, skipped=0, failed=0)+ built = []+ with patch.object(timeline_sheets, 'is_current', side_effect=lambda root, asset_id, interval: asset_id == 1), \+ patch.object(timeline_sheets, 'build', side_effect=lambda ffmpeg, path, duration, root, asset_id, interval, **kwargs:+ built.append(asset_id) or True):+ state = self._run(resume_state=saved)+ self.assertEqual(built, [2])+ self.assertEqual((state['made'], state['done'], state['total']), (2, 3, 3))+def setUp(self):self.root = Path(tempfile.mkdtemp(prefix="timeline-run-"))self.addCleanup(lambda: shutil.rmtree(self.root, ignore_errors=True))diff --git a/tests/test_update_tasks.py b/tests/test_update_tasks.pyindex 31e2b28ac..e35c59866 100644--- a/tests/test_update_tasks.py+++ b/tests/test_update_tasks.py@@ -26,6 +26,69 @@ class UpdateTaskTests(unittest.TestCase):fresh_ledger(self.db.parent)self.store = TaskRunStore(self.db)+ def test_restart_keeps_the_run_identity_and_committed_progress(self):+ entered, release = threading.Event(), threading.Event()+ first = BackgroundJob('resume-test', runs=self.store, task_key='feed-check')+ def initial(job_id):+ first.checkpoint(job_id, completed_sources=[1], checked=1, total=3, added=7)+ entered.set()+ release.wait(5)+ original = first.start(initial, recovery={'all': True}, initial={'checked': 0, 'total': 3})+ worker = first.thread+ self.assertTrue(entered.wait(5))+ first.stop(timeout=0)+ release.set()+ worker.join(5)+ self.assertFalse(worker.is_alive())+ self.assertEqual(self.store.get(original['run_id']).status, 'pending')+ second = BackgroundJob('resume-test', runs=self.store, task_key='feed-check')+ observed, processed = [], []+ def resumed(job_id):+ observed.append(second.snapshot())+ for source in (1, 2, 3):+ if source not in observed[0]['completed_sources']:+ processed.append(source)+ second.update(job_id, checked=3, status='complete')+ result = job_recovery.resume(self.db, {'feed-check': lambda body: second.start(+ resumed, restart=True, recovery=body, initial={'checked': 0, 'total': 0})})+ second.thread.join(5)+ self.assertEqual(result, ['feed-check'])+ self.assertEqual(processed, [2, 3])+ self.assertEqual((observed[0]['job_id'], observed[0]['run_id']),+ (original['job_id'], original['run_id']))+ self.assertEqual((observed[0]['checked'], observed[0]['total'], observed[0]['added']), (1, 3, 7))++ def test_dead_process_keeps_a_recoverable_run_pending(self):+ run = self.store.start('follow-check', trigger='manual', total=5)+ self.store.progress(run.id, current=3, total=5, throttle=0)+ job_recovery.save(self.db, 'follow-check', 'original', {'older': True}, run_id=run.id, waiting=True)+ self.assertNotIn(run.id, self.store.recover_interrupted())+ pending = self.store.get(run.id)+ self.assertEqual((pending.status, pending.progress_current, pending.progress_total), ('pending', 3, 5))++ def test_terminal_checkpoint_is_readable_by_a_new_job_instance(self):+ first = BackgroundJob('resume-test', runs=self.store, task_key='link-check')+ original = first.start(lambda identity: first.update(identity, status='complete', checked=2, total=2), recovery={})+ first.thread.join(5)+ second = BackgroundJob('resume-test', runs=self.store, task_key='link-check')+ self.assertEqual((second.snapshot()['job_id'], second.snapshot()['checked']), (original['job_id'], 2))++ def test_explicit_stop_removes_the_resume_record(self):+ entered, release = threading.Event(), threading.Event()+ job = BackgroundJob('resume-test', runs=self.store, task_key='feed-check')+ def work(identity):+ job.checkpoint(identity, checked=1)+ entered.set()+ release.wait(5)+ original = job.start(work, recovery={})+ worker = job.thread+ self.assertTrue(entered.wait(5))+ job.stop(timeout=0, resume=False)+ release.set()+ worker.join(5)+ self.assertEqual(job_recovery.read(self.db, 'feed-check'), {})+ self.assertEqual(self.store.get(original['run_id']).status, 'cancelled')+def test_running_tasks_receive_a_pause_request_when_installation_is_accepted(self):for trigger in ('manual', 'scheduled'):with self.subTest(trigger=trigger):