Peach: Local-first video metadata scraper and media manager.
fix(media): 统一垃圾检查并补齐媒体处理顺序
longmeidaoa317e56717e3
diff --git a/README.en.md b/README.en.mdindex 27d294586..b9cd340b1 100644--- a/README.en.md+++ b/README.en.md@@ -46,6 +46,8 @@ https://github.com/user-attachments/assets/a5610874-e611-41bb-be22-7d99f0d64b62Watch history, favorites and settings stay on your computer. Scanning leaves original files untouched; manual organization offers a preview and an undo for the last batch.+Scanning, cloud downloads and media processing share junk checks. Attachments with sufficient evidence go to the recoverable trash; uncertain items await review. You can restore an item and mark it as “not junk”. See [junk files](docs/OPERATIONS.md#垃圾文件) for processing order and protections.+Scanning, follow updates and new-release metadata collection save their progress and continue unfinished items after the app restarts. See [task recovery](docs/OPERATIONS.md#长时任务断点续跑) for recovery boundaries.Scanning and collection show video and recognizable JAV counts, with incremental checks for new folders. Traffic estimates use pending work, local caches, deduplicated release codes, and local samples of similar operations. Library settings read and save CloudDrive API and data proxies; traffic totals follow each linked folder's connection setting. Proxy protection limits automatic video reads. See [configuration](docs/OPERATIONS.md#配置页) for measurement coverage and estimation limits.diff --git a/README.md b/README.mdindex a6c5809f7..733462276 100644--- a/README.md+++ b/README.md@@ -46,6 +46,8 @@ https://github.com/user-attachments/assets/a5610874-e611-41bb-be22-7d99f0d64b62观看记录、收藏和设置保存在本机。扫描不动原文件;手动整理先预览,可撤回上一批。+扫描、云下载和媒体处理共用垃圾检查:证据充分的附件自动进回收站,疑似项留待复核;可还原,也可标记「不是垃圾」。处理顺序和保护条件见[垃圾文件](docs/OPERATIONS.md#垃圾文件)。+扫描、关注更新和新作资料采集会保存进度,应用重启后继续处理未完成的项目。恢复范围见[断点续跑](docs/OPERATIONS.md#长时任务断点续跑)。扫描与采集显示视频与可识别 JAV 数量,首次目录分批预检;流量按待处理资料、缓存、同番号去重和本机同类样本估算,运行中显示已用流量和预计总流量。媒体库设置可读取和保存 CloudDrive 的 API/数据代理,按对应目录的连接方式汇总流量;“代理保护”限制自动读取视频。统计范围与估算限制见[运行与配置](docs/OPERATIONS.md#配置页)。diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.mdindex 002ffd81a..470aef54c 100644--- a/docs/OPERATIONS.md+++ b/docs/OPERATIONS.md@@ -397,10 +397,23 @@ untrunc 按相同顺序查找,对应目录为 `tools/untrunc`。独立程序### 垃圾文件-「垃圾文件」复核覆盖 `local`、`115`、`pikpak` 全部来源的各种文件类型,在线网址不在其中。Windows 的 `.url` 快捷方式直接交给人工判断。+「垃圾文件」复核覆盖已配置物理媒体来源的各种文件类型,在线网址不在其中。检查复用同一套评分,先核对原文件名、目录、时长、体积与相邻正片,再决定是否隔离。++| 流程 | 检查与处理顺序 |+| --- | --- |+| 添加媒体库、全量扫描 | 登记与配套文件识别 → 按本次目录检查 → 允许自动读取的媒体探测 → 重判 |+| 推送发现、单文件登记 | 登记与配套文件识别 → 检查该资产 → 允许自动读取的媒体探测 → 重判 |+| 云下载 | 落地并绑定正片 → 检查下载范围 → 探测并重判 → 采集资料与预览 → 重判 → 按模板整理 → 派发后继任务 |+| 补全资料、补全预览 | 检查当前范围 → 探测并重判 → 读取本地资料与生成预览 → 刷新活动资产 → 联网采集适用资料 |+| 仅采集资料 | 检查已有证据 → 读取配套资料 → 重判 → 联网采集;不读取视频或抽帧 |+| 手动整理、维护脚本 | 执行整理和生成九宫格前检查;独立探测脚本在探测前后检查;整理预览与只读盘点保持只读 |++探测沿用来源的计费与代理保护,不为垃圾检查额外读取媒体。探测后先接回能确定归属的个人记录,再重判垃圾。没有有效时长的视频只进复核。仅补预览限定本地来源;完整处理在关闭视频预览时跳过抽帧。失败任务保留检查点与问题记录,重试复用已登记资产、测量值与整理日志;资料或整理失败时不派发该下载的后继任务。- 判断为垃圾的文件只会被移入回收站,不会直接删除。- 点「不是垃圾」会记下这个文件的判断(可以撤销),它会出现在「已排除」视图里。这个判断只针对这一个文件,不会把整个域名或来源列入白名单。+- 自动隔离要求达到自动线,并保护人工保留、喜欢、观看或播放列表等个人记录、已配对字幕、文件自身的作品番号和本次绑定的下载正片。写入前再次核对当前资产;已离开馆藏的条目不进入后续采集或整理。+- 无关键词的极短附件可凭同目录长正片、数量及跨目录测量值进入复核;这类结构证据与「域名加序号」命名不能单独触发自动隔离。还原后确认要保留的文件,可同时标记「不是垃圾」。维护脚本 `scripts/trash_junk.py` 默认生成候选清单;永久删除使用 `--purge --review-csv`,复核 CSV 每行包含 `id`、`path`、`size`、`decision`、`confidence`、`why`。diff --git a/docs/REUSE.md b/docs/REUSE.mdindex 4e7513d2e..4e4c2c80e 100644--- a/docs/REUSE.md+++ b/docs/REUSE.md@@ -423,7 +423,8 @@ MGS 专属发行先查自营商品目录,Prestige 番号复用 `metadata_route- 非 Windows 表单的内部路径由 `media_configuration.prepare_sources` 补齐,复用既有来源校验、配置写入和 `platform` 路径转换;不改数据库路径模型或执行迁移。标准库 [PureWindowsPath / PurePosixPath](https://docs.python.org/3.14/library/pathlib.html#pure-paths) 用于比较两种路径,[hashlib.sha256](https://docs.python.org/3.14/library/hashlib.html) 按来源与实际目录生成稳定候选前缀,已有配置和显式输入优先,冲突时换盘符。媒体库名称使用实际目录名,前端复用现有 `Disclosure` 和 `Input`,无新增依赖。- JAV 入库资源查询复用 `sources.javdb` 的身份核对、`SourceTransport` 的 Cookie 与冷却、`HostLimitedTransport` 的 3 秒间隔、`beautifulsoup4==4.15.0` 和 `downloads.parse_magnet`。JavPack 的 `JavDB.magnet.user.js` 0.0.2(GPL-3.0,仓库 revision `b546e1881147e15796c697e8e086db9d3ac0c5be`)只作页面字段参考;它依赖浏览器用户脚本 API,不能直接作为 Python 服务端依赖。Peach 负责评论链接、协议分流、去重、60 秒缓存和最多三页评论的预算,不新增依赖。2026-10-02 单作品只读实测取得 4 条磁链和含 ed2k 的评论页;评论入口从 `.review-tab[data-url]` 读取,受限评论保留来源限制提示。磁链可交现有云下载服务,ed2k 与其他网页链接提供复制。- 磁力优先链复用同一套 `SourceTransport`、`HostLimitedTransport`、`downloads.parse_magnet` 与 Torznab 后备,不引入新的抓取运行时。manko.fun 的页面请求实际落到 `healertanker.com/swx`,响应 `data` 是 basE91 包装的 JSON;实现只移植该公开响应形态与编号精确匹配,不复用会话脚本。Sukebei 直接读搜索表格;JavBus、16mag、OneJAV、Free JAV Torrent 由现有自配 Prowlarr/Jackett 索引器续接。Akiba Online 与 Reddit 只作线索,不自动取磁力。候选按分辨率、做种、推广附件数量与正片体积排序;总文件数不再是一票否决,manko.fun 候选绕过水印门槛,其他来源只能标为待到盘检查。-- 云下载后处理复用 `task_run` 后继串行、`process_library(retry_ids=...)`、垃圾复核的 `AUTO_TRASH_SCORE`、`organize.build_plan(asset_ids=...)`、SQLite backup 与整理批次日志。没有另造第二套刮削、垃圾词库或移动器。+- 云下载后处理复用 `task_run` 后继串行、`process_library(retry_ids=...)`、`junk.isolate`、`organize.build_plan(asset_ids=...)`、SQLite backup 与整理批次日志。没有另造第二套刮削、垃圾词库或移动器。+- `junk.scored_candidates` 统一复核与自动隔离的证据;`junk.isolate` 要求显式资产或来源目录,保护人工决定、个人记录、字幕与正片身份,写前重查证据,仅写可撤销的回收站状态。扫描、推送、媒体探测、资料处理、云下载与维护脚本共用该入口;页面只补展示成本与缩略图字段。复用当前树的词库、目录关系、`record_holders`、`LedgerDatabase` 与历史评分回归,使用 Python 标准库和现有 SQLite JSON 支持,无新增依赖、协议或媒体读取。流程顺序见[垃圾文件](OPERATIONS.md#垃圾文件)。- 抓取入口的复用缺口见 [来源采集](SOURCING.md) ;跨用户安装、来源网络、Cookie GUI、最高可得画质与图像清单按 [ADR-0024](adr/0024-mark-manifest-not-bundled-bytes.md)。私有后缀判断用 tldextract 5.4.0(BSD-3-Clause),随包 PSL、`suffix_list_urls=()`、`cache_dir=None`、`include_psl_private_domains=True`;Instaloader 4.15.3 与 browser_cookie3 不进正式依赖。HTTPX(BSD-3-Clause)、curl_cffi(MIT)、Pillow、amane 桥与现有候选缓存是正式基础,请求节拍用 `scripting.RateLimiter`/`HostLimiter`。- 采集 GUI 用 React island、CredentialStore、HTTPX、Pillow 与 BackgroundJob,`jav_cover_fetch` 同时服务界面与 CLI;Peach 保留域内凭据、来源路由、预算、冷却、番号身份与高清替换策略。Cookie 文本用标准库 SimpleCookie/MozillaCookieJar 解析,不导入 pickle。- 删除失效链接与资源同步的执行阶段用 `BackgroundJob.start_result` 存终态回执;刷新只查状态,写入不自动重放,确认与检查结果过期门槛照旧有效。@@ -482,7 +483,7 @@ PikPak WebDAV 同来源移动由 `organize_clouddrive.py` 提供,目录入口A 盘同父目录改名复用 CloudDrive 官方 `RenameFile`;跨父且改名的操作拆为同父 `RenameFile` 和保留名称的官方 `MoveFile`,中间路径进入恢复回执并禁止覆盖。每步核对完整成员与体积;成功应答不替代路径核验。路径中的 `@` 保留为 URI 路径字符,其他字符使用 UTF-8 百分号编码。认证样本已移动时,最多另查两条馆藏内图片样本,仅提取既有认证,不读取媒体内容。-推广网页资源目录中的横幅、标志和导航图片由 `web_batch._attachment_junk_reason` 进入复核队列。判据同时要求 `_files`、网址推广套话、域名、导航元素文件名和体积上限;继承的作品番号不豁免网页导航图片,作品图片、字幕、NFO 与普通 `_files` 目录保留。固定命名的推广附件(安装包、宣传视频、二维码卡片、下载站横幅)按扩展名、体积上限与整名收在词库 `resources/junk/promo_attachment_names.json`,由 `web_batch._promo_name_reason` 读取;命中只计复核档 `JUNK_REVIEW_SCORE`,未探测时长视频的推广目录、推广创作者位与体积证据同样只到复核档,这类分数单独凑不到 `trash_junk.py` 默认的自动移入回收站门槛 `AUTO_TRASH_SCORE`,不凭名称移入回收站或删除。永久删除仍走 `trash_junk.py --purge --review-csv` 和 `purge_assets`。+推广网页资源目录中的横幅、标志和导航图片由 `junk._attachment_junk_reason` 进入复核队列。判据同时要求 `_files`、网址推广套话、域名、导航元素文件名和体积上限;继承的作品番号不豁免网页导航图片,作品图片、字幕、NFO 与普通 `_files` 目录保留。固定命名的推广附件(安装包、宣传视频、二维码卡片、下载站横幅)按扩展名、体积上限与整名收在词库 `resources/junk/promo_attachment_names.json`,由 `junk._promo_name_reason` 读取;命中只计复核档 `JUNK_REVIEW_SCORE`,未探测时长视频的推广目录、推广创作者位与体积证据同样只到复核档,这类分数单独凑不到 `trash_junk.py` 默认的自动移入回收站门槛 `AUTO_TRASH_SCORE`,不凭名称移入回收站或删除。永久删除仍走 `trash_junk.py --purge --review-csv` 和 `purge_assets`。极短视频附件复用同一垃圾复核评分、`promo_free_key` 的名称比较和 `release_code_from_filename` 的自身番号识别。发行目录须有与目录同名、至少 20 分钟且至少 120 MiB 的活动物理正片;附件须有有效测量值、不超过 30 秒与 32 MiB,正片至少为附件时长的 40 倍、体积的 10 倍。同目录至少 3 条此类附件,或相同体积、毫秒时长与有效分辨率的附件分布在至少 3 个发行目录,才计结构证据。真实创作者归属、自身番号、分卷、独立短动画与普通混合目录受保护;目录继承的番号不能证明附件身份。结构证据只够复核,短、小或同测量值都不能单独证明是广告,默认自动隔离线不接收仅由这些证据凑分的候选。查询只读账本,不读取云盘媒体,不新增依赖。diff --git a/scripts/organize_media.py b/scripts/organize_media.pyindex 91e5b8278..096f86eff 100644--- a/scripts/organize_media.py+++ b/scripts/organize_media.py@@ -24,10 +24,11 @@ SRC_DIR = PROJECT_ROOT / "src"if str(SRC_DIR) not in sys.path:sys.path.insert(0, str(SRC_DIR))-from peach import organize+from peach import junk, organizefrom peach.config import DATABASE_PATH, GENERATED_DIRfrom peach.media_sources import SCANNABLE_LOCATIONSfrom peach.organize_templates import PLACEHOLDERS, TemplateError+from peach.platform import location_rootsfrom peach.scripting import add_ledger_write_args, open_for_write, verify_after_write@@ -58,6 +59,7 @@ def _report_plan(plan: dict) -> None:def run(args: argparse.Namespace) -> int:connection = open_for_write(args)try:+ baseline = {tuple(row) for row in connection.execute('PRAGMA foreign_key_check')}if args.rollback_last:latest = organize.latest_batch(args.generated_root)if latest is None:@@ -72,6 +74,10 @@ def run(args: argparse.Namespace) -> int:for failure in result["failures"]:print(f" {failure['path']}:{failure['reason']}")else:+ if args.apply:+ checked = junk.isolate(args.db, roots=[(args.location, root)+ for root in location_roots().get(args.location, ())])+ print(f"垃圾检查:隔离 {checked['isolated']} 个,待复核 {checked['review']} 个")plan = organize.build_plan(connection, location=args.location,file_template=args.file_template, dir_template=args.dir_template)@@ -88,10 +94,12 @@ def run(args: argparse.Namespace) -> int:for failure in result["failures"]:print(f" {failure['path']}:{failure['reason']}")print(f"批次日志:{result['log_path']}")- integrity, violations = verify_after_write(connection)- if integrity != "ok" or violations:- print(f"写入后 ledger 校验失败:integrity={integrity} foreign_keys={violations}")+ integrity, _ = verify_after_write(connection)+ introduced = {tuple(row) for row in connection.execute('PRAGMA foreign_key_check')} - baseline+ if integrity != "ok" or introduced:+ print(f"写入后 ledger 校验失败:integrity={integrity} new_foreign_keys={len(introduced)}")return 2+ print(f"已核对 ledger,存量外键异常 {len(baseline)} 条")return 0finally:connection.close()diff --git a/scripts/probe.py b/scripts/probe.pyindex ef3d3dafb..dedf4c3c7 100644--- a/scripts/probe.py+++ b/scripts/probe.py@@ -19,6 +19,7 @@ SRC_DIR = PROJECT_ROOT / "src"if str(SRC_DIR) not in sys.path:sys.path.insert(0, str(SRC_DIR))+from peach import junk, record_rehomefrom peach.config import DATABASE_PATH, FFMPEG_DIR, LOG_DIR, STATE_DIRfrom peach.ffmpeg import FFmpegResolverfrom peach.jobs import (@@ -133,6 +134,8 @@ def run(args: argparse.Namespace) -> int:from peach.strm import background_rowstasks = background_rows(connection.execute(sql, parameters).fetchall(), 1)connection.close()+ isolated = junk.isolate(args.db, asset_ids=[row[0] for row in tasks])+ tasks = [row for row in tasks if row[0] not in isolated['ids']]total = len(tasks)log(f"待处理 {total:,} 个视频 workers={args.workers} interval={args.interval}s 日志={log_path}")if not total:@@ -143,6 +146,7 @@ def run(args: argparse.Namespace) -> int:for task in tasks:pending.put(task)counters = {"done": 0, "failed": 0}+ measured_ids = []started = time.time()# 探测本身产物很小,但它触发的云盘块缓存会落在系统盘;边跑边看,不只查起跑线。guard = DiskGuard(system_volume(), args.min_free, args.disk_check_secs)@@ -202,13 +206,19 @@ def run(args: argparse.Namespace) -> int:if len(buffer) >= 25:connection.executemany(update, buffer)connection.commit()+ measured_ids.extend(row[-1] for row in buffer)buffer.clear()if buffer:connection.executemany(update, buffer)connection.commit()+ measured_ids.extend(row[-1] for row in buffer)elapsed = time.time() - startedlog(f"完成 {counters['done']:,} 个,失败 {counters['failed']},耗时 {elapsed/60:.1f} 分钟")+ record_rehome.reattach(connection, measured_ids)+ connection.commit()connection.close()+ checked = junk.isolate(args.db, asset_ids=measured_ids)+ log(f"垃圾检查:隔离 {isolated['isolated'] + checked['isolated']} 个,待复核 {checked['review']} 个")if stop_reason:# 已探测的部分都已入库;中止必须体现在退出码上,不能报成正常完成。log(f"磁盘闸门中止:{stop_reason[0]}")diff --git a/scripts/sheets.py b/scripts/sheets.pyindex b8aab4273..5b35a0e86 100644--- a/scripts/sheets.py+++ b/scripts/sheets.py@@ -19,7 +19,7 @@ SRC_DIR = PROJECT_ROOT / "src"if str(SRC_DIR) not in sys.path:sys.path.insert(0, str(SRC_DIR))-from peach import frame_capture+from peach import frame_capture, junkfrom peach.config import DATABASE_PATH, FFMPEG_DIR, GENERATED_DIR, LOG_DIR, STATE_DIRfrom peach.ffmpeg import FFmpegResolverfrom peach.jobs import (@@ -194,6 +194,9 @@ def run(args: argparse.Namespace) -> int:parameters += (args.limit,)tasks = background_rows(connection.execute(sql, parameters).fetchall(), 2)connection.close()+ checked = junk.isolate(args.db, asset_ids=[row[0] for row in tasks])+ tasks = [row for row in tasks if row[0] not in checked['ids']]+ log(f"垃圾检查:隔离 {checked['isolated']} 个,待复核 {checked['review']} 个")total = len(tasks)log(f"待处理 {total:,} 个视频 workers={args.workers} frames={args.frames} 日志={log_path}")if not total:diff --git a/src/peach/api.py b/src/peach/api.pyindex 8a66e3678..c4522dbba 100644--- a/src/peach/api.py+++ b/src/peach/api.py@@ -34,7 +34,7 @@ from fastapi.staticfiles import StaticFilesfrom starlette.exceptions import HTTPException as StarletteHTTPExceptionfrom starlette.middleware.gzip import DEFAULT_EXCLUDED_CONTENT_TYPES, GZipMiddleware-from . import (__version__, browser_transport, feeds, tunnel, web_batch, web_contract,+from . import (__version__, browser_transport, feeds, tunnel, web_contract,web_feeds, web_follow, web_settings)from . import routes_api, routes_auth, routes_configuration, routes_jav_browser, routes_media, routes_pages, routes_subtitlesfrom .buildinfo import frozen_build@@ -233,7 +233,6 @@ def create_app(taste_history_manifest=settings.taste_history_manifest,database=database,)- contract.download_junk_candidates = lambda: web_batch.automatic_junk_candidates(contract)contract.download_organize_templates = lambda: web_settings.organize_templates(contract)repository = LedgerRepository(database)from .mount_reachability import source_rootsdiff --git a/src/peach/download_postprocess.py b/src/peach/download_postprocess.pyindex 58b947762..758ab7035 100644--- a/src/peach/download_postprocess.py+++ b/src/peach/download_postprocess.py@@ -5,14 +5,15 @@ import timeimport jsonfrom pathlib import Path-from . import organize, settings_file+from . import junk, organize, settings_filefrom .catalog_rules import same_release_codefrom .downloads import (INGESTED, POSTPROCESS_COMPLETE, POSTPROCESS_FAILED,POSTPROCESS_PENDING, POSTPROCESS_QUEUED, POSTPROCESS_RUNNING,DownloadStore, iso)from .field_owners import SCAN_FILENAME, write_owned_fieldsfrom .followups import FollowupType, register-from .library_processing import COLLECT_STAGE, process_library+from .library_processing import COLLECT_STAGE, _ffprobe_path, process_library+from .media_probe import probe_unmeasuredfrom .migrations import sqlite_backupfrom .task_runs import declared_rows@@ -85,16 +86,17 @@ def _bind_main(contract, task) -> tuple[int, str, list[int]]:def _isolate_ads(contract, main: int, folder_ids: list[int]) -> int:"""只自动隔离既有垃圾规则达到自动线的附属资产;文件数本身不作为证据。"""- candidates = [int(row["id"]) for row in contract.download_junk_candidates()- if int(row["id"]) in folder_ids and int(row["id"]) != main]- if not candidates:- return 0- marks = ",".join("?" * len(candidates))- with contract.database.write_transaction() as connection:- cursor = connection.execute(- f"UPDATE asset SET disposal='trash',disposal_reason='junk',feedback_at=? WHERE id IN ({marks}) "- "AND disposal IS NULL", (time.time(), *candidates))- return int(cursor.rowcount)+ return junk.isolate(contract.database, asset_ids=folder_ids, protected_ids=[main])['isolated']+++def _probe_folder(contract, task, main, folder_ids, *, on_junk=lambda result: None):+ """只为该下载范围补允许自动读取的媒体属性,正片身份在整个阶段都受保护。"""+ config = settings_file.active()+ if not config.configured or Path(contract.db_path).resolve() != (+ config.directory('database') / 'ledger.db').resolve():+ raise ValueError('当前馆藏与媒体配置不一致,不能探测下载媒体')+ return probe_unmeasured(contract.db_path, _ffprobe_path(config), task.location, task.ledger_path,+ asset_ids=folder_ids, config=config, protected_ids=[main], on_junk=on_junk)def _collect(contract, main: int, handle) -> dict:@@ -110,13 +112,10 @@ def _collect(contract, main: int, handle) -> dict:state = process_library(config, contract.db_path, contract.candidate_root, contract.cover_root,retry_ids=[main], stage=COLLECT_STAGE, database=contract.database,- state_file=state_file, resume_state=previous,+ state_file=state_file, resume_state=previous, protected_ids=[main],report=lambda state: handle.progress(current=int(state.get("checked") or 0), total=int(state.get("total") or 0),label=str(state.get("stage") or TASK_LABEL)))- rows = declared_rows(state.get("followups"))- if rows and handle.run_id is not None:- contract.task_runs.enqueue_followups(handle.run_id, rows)if state.get("status") == "failed":raise RuntimeError(str(state.get("error") or "资料采集失败"))return state@@ -172,11 +171,21 @@ def run(contract, key: str, handle) -> dict:main, _path, folder_ids = _bind_main(contract, task)handle.progress(label="隔离推广附件", throttle=0)isolated = _isolate_ads(contract, main, folder_ids)+ handle.progress(label='读取下载媒体属性', throttle=0)+ measured_junk = []+ probed = _probe_folder(contract, task, main, folder_ids, on_junk=measured_junk.append)+ isolated += sum(result['isolated'] for result in measured_junk)+ isolated += _isolate_ads(contract, main, folder_ids)handle.progress(label="采集作品资料", throttle=0)metadata = _collect(contract, main, handle)+ isolated += int(metadata.get('junk_isolated') or 0)+ isolated += _isolate_ads(contract, main, folder_ids)task = store.get(task_id)handle.progress(label="按模板整理文件", throttle=0)organized = _organize(contract, task, main)+ rows = declared_rows(metadata.get('followups'))+ if rows and getattr(handle, 'run_id', None) is not None:+ contract.task_runs.enqueue_followups(handle.run_id, rows)except Exception as error:store.postprocess(task_id, POSTPROCESS_FAILED,detail=f"{type(error).__name__}: {error}")@@ -186,7 +195,7 @@ def run(contract, key: str, handle) -> dict:if organized.get('skipped') else "正片已绑定,资料已采集并完成整理"),finished=iso(time.time()))contract.cache_bust()- return {"task_id": task_id, "asset_id": main, "ads_isolated": isolated,+ return {"task_id": task_id, "asset_id": main, "ads_isolated": isolated, 'probed': probed,"metadata_candidates": int(metadata.get("candidates") or 0),"moved": int(organized.get("moved") or 0),'existing_foreign_keys': int(organized.get('existing_foreign_keys') or 0),diff --git a/src/peach/junk.py b/src/peach/junk.pynew file mode 100644index 000000000..b93cbda5d--- /dev/null+++ b/src/peach/junk.py@@ -0,0 +1,601 @@+"""物理媒体垃圾证据与复核评分;扫描、后处理和页面共用同一套规则。"""+from __future__ import annotations++import json+import re+import time+from functools import cache+from pathlib import PureWindowsPath++from .catalog_rules import is_jav_code, promo_free_key, release_code_from_filename+from .media_sources import SCANNABLE_LOCATIONS+from .settings_file import PROJECT_ROOT+from .personal_records import record_holders+from .repository import LedgerDatabase++# 只认联系方式与站点形态的推广套话。「微信」「成人游戏」这类词单独出现不算:+# 实测正片标题里就有(「还要微信跟老公汇报战果」是剧情,不是联系方式)。+PROMO_PHRASE = re.compile(+ r"(扫码|扫一扫|掃一掃|扫描|掃描|QR ?CODE|二维码|二維碼|加微信|加微|威信\d|微信号|"+ r"微信\s*[::]|免费看|免费玩|福利群|最新地址|"+ r"永久(?:域名|地址|发布)|地址发布器|防屏蔽|资源获取|点击(?:观看|下载|进入)|下载APP|下载|签到|代币|领取|"+ r"强力推荐|国产大片|在线视频|線上影片每天火熱更新|手机看片|手機影城|"+ r"大饱眼福|房间火爆|澳门|赌场|博彩|棋牌|加我|"+ r"GAMES?\d*|APP)", re.I)+# 只降低残留、自己不构成命中的套话。+#+# 手游插页整批绕过了上面的判据:`爱姬远征-免费18禁手游-扫码安装.jpg` 命中「扫码」,+# 可剥完还剩「爱姬远征免费禁手游安装」12 个字,落在 6~14 的中间档只拿 30 分,够不上+# 40 的门槛;`三国志侵略版-免费18禁手游-扫码安装.png` 多两字残留就到 14,连 30 分都没有。+# 残留越多分越低,而这些残留里一个内容字都没有——广告名写得越长反而越安全。+# 游戏名(爱姬远征、天下布魔)永远进不了词表,能剥的只有它旁边那圈固定套话。+# 这些词单独出现说明不了什么(真片名里也有「免费」),所以只参与残留计算,不作命中依据。+PROMO_FILLER = re.compile(+ r"(免费\d*禁|免费|免費|手游|手遊|成人遊?戲|成人游戏|下载安装|安装|安裝|访问|訪問)",+ re.I)+# 结尾不能用 \b:`uuc82.com_2` 里 `m` 和 `_` 都是词字符,构不成边界,域名会漏掉。+PROMO_DOMAIN = re.compile(+ r"(?:https?://)?(?:www\.)?[\w-]{2,}\.(?:com|net|me|la|xyz|cc|tv|top|vip|club|"+ r"info|org|pw|cn|app|site|online|shop)(?![a-z0-9])", re.I)+# 真番号带厂牌前缀和连字号(ABW-153、259LUXU-1141)。RAIKUN325 这类没有连字号的+# 是被误填进 code 的创作者账号名,不能拿来做「同番号有完整版」的比较,+# 判据与 `is_jav_code` 同源:分隔符正是番号与账号名的唯一线索。+REAL_CODE = re.compile(r"^(?:\d{2,3})?[A-Za-z]{2,8}-\d{2,5}$|^FC2", re.I)+PART_MARK = re.compile(r"(CD\d|part\d|分卷|-\d{1,2}$|\(\d+\)$)", re.I)+# 推广站目录的两种形态,与 `scripts/find_ads.py` 的判据 D/E 同源:+# 创作者位是旧导入器的目录名投影,`bbsxv.xyz-DOCP-324` 这类广告包会直接落在那里;+# 裸域名目录(98T.la@账号、huachishe.com@系列)是转载水印,不是广告,不能进判据。+AD_DOMAIN = re.compile(+ r"\b[0-9a-z][-0-9a-z]{1,20}\.(?:cc|xyz|xxx|com|net|la|me|top|vip|club|app|cn|pw|tv|gg)\b", re.I)+AD_DIRPACK = re.compile(+ r"[0-9a-z][-0-9a-z]{1,20}\.[a-z]{2,10}[ \-_]+\[?[A-Za-z]{2,6}-?\d{2,5}", re.I)+#: 同一目录里有这么多资产,它就是成套下载的资源包,不是塞进别人目录的插页。+#:+#: `jitumi.pw(77).gif` 与 `【91狼友之家91home.cc】19.jpg` 剥完只剩域名,是最硬的+#: 「整名推广语」信号,可它们分别躺在 `森萝财团 X-019 肉丝换白丝 [103P1V-1.39GB]`+#: (同目录 107 项)和 `赠送-稀缺整合【91home.cc】\海角\新建文件夹`(同目录 24 项)里:+#: 域名在这里是打包渠道给整包起的名,几十上百张成套的图本身就是要留的资源。+#: 广告插页反过来,是一两张挤在别人的番号目录里——WAAA-415 那个目录只有 13 项。+BUNDLE_DIR_ASSETS = 20+#: 同一目录里有这么多同类推广名,它们就是成群塞进来的插页。+#:+#: 与 `BUNDLE_DIR_ASSETS` 正好对称:资源是成套的正片与图集,插页是成群的同一段套话。+#: `三国志侵略版-免费18禁手游-扫码安装.png` 剥完仍剩 6 个字的游戏名,单看名字只够+#: 中间档;但同目录另外 10 个文件挂着一模一样的推广尾巴,谁都不是正片的一部分。+PROMO_CLUSTER_FILES = 3+#: 缩略图与封面相对正片的固定尾巴(`MIAD573_02.wmv` ↔ `MIAD573_02_s.jpg`,+#: `LXVS006-BD.iso` ↔ `lxvs006pl.jpg`)。`pl`/`ps` 是 JAV 大小海报的通用写法。+#: 不剥数字:剥了 `MIAD573_02_s` 会退到 `MIAD573`,那已经不是同一条正片。+SHOT_TAIL = re.compile(r"[-_. ]?(?:s|t|pl|ps|thumb|poster|cover|fanart|big|small)$", re.I)+#: 正片自己带的画质与载体尾巴,只在配对时剥,用来让海报对上原盘。+DISC_TAIL = re.compile(r"[-_. ]?(?:bd|bdmv|bdiso|bdrip|dvd|dvdiso|iso|uhd|fhd|hd)$", re.I)+#: 超过这个体积的文件自己就是内容,不必再看目录名。+#:+#: `javme.me_LXVS-006-BD\LXVS006-BD.iso` 是 19.8 GB 的蓝光原盘,却因为住在+#: 「域名+番号」的目录里被判 45 分。广告不会有大文件:整包插页加起来都不到 10 MB,+#: 一个上 GB 的文件是资源站给自己的资源改了目录名,不是资源站塞进来的推广。+CONTENT_BYTES = 1024 ** 3+INTERNET_SHORTCUT_SUFFIXES = frozenset({".url"})+#: 保存下来的网页。`.mhtml` 是 Edge 的整页存档,媒体目录里的这份没有内容价值;+#: 它和 `.url` 一样是「导航页」而不是媒体,不给体积兜底留机会。+PAGE_ARCHIVE_SUFFIXES = frozenset({".mhtml", ".mht", ".htm", ".html"})+#: 推广品牌词。它们单独出现在视频名里不构成证据——`麻豆传媒`、`外围`、`直播` 都可能是+#: 真片名的一部分——只有叠加推广形态、装饰符或推广目录上下文时才计分。+#: 名单来自 2026-09 用户实例 `B:\xxr\0208 (23)` 里自曝身份的推广站与推广 APP。+AD_BRAND = re.compile(+ r"(91(?:短视频|视频|国产|AV|約炮|约炮)|成人抖音|快手直播|杏吧|草榴|色中色|成人头条|"+ r"泡芙短视频|含羞草|她趣|台湾UU|全国外围|外围楼凤|楼凤|麻豆传媒映画|蜜桃影像|"+ r"成人游戏|169BBS|SEX169)", re.I)+#: 与品牌词同现才算推广名的形态词。`视频`、`直播` 这类过常见的词不进这张表:+#: `麻豆传媒映画APP-限时免费体验` 是推广包,`麻豆传媒 某作品` 是资源。+AD_BRAND_FORM = re.compile(+ r"(APP|一键|导航|聊天室|免费体验|限时免费|扫码|资源获取|网址|发布页)", re.I)+#: 品牌推广图的体积上限。`蜜桃影像传媒.png` 18 KB、`東方秋白…169BBS…` 75 KB,+#: 而任何一张内容图都在 MB 量级;品牌名 + 这种体积才是广告卡,不是作品套图。+BRAND_TINY_BYTES = 300 * 1024+#: 装饰符名(`❤91短视频❤.jpg`):两侧都挂或挂多个爱心的小图,名字里没有实质描述。+DECOR_MARKS = "❤♥❥♡💕💗💖💘💝"+#: 装饰图体积上限。`❤草榴视频❤.jpg` 705 KB 是广告卡;同名配套封面(2 MB 级)不算,+#: 它走 `has_sibling_original` 的正片配对豁免。+DECORATED_MAX_BYTES = 1536 * 1024+#: 目录名自曝是推广包的形态:`-APP`、`一键约炮`、`論壇文宣`、聊天室等。+#: 只匹配目录分量,不看文件名;命中后目录里不构成内容的文件都进复核。+AD_DIR_FORM = re.compile(+ r"(論壇文宣|论坛文宣|文宣|宣傳|一键约炮|全国外围|外围楼凤|楼凤|聊天室|"+ r"免费体验|限时免费)", re.I)+#: `APP` 按词匹配:`Happy` 里也藏着 `app`,裸子串会把正常目录卷进来;+#: `.app` 是域名后缀,`bkm9.app` 那类推广目录靠目录里其他证据,不靠这一条。+AD_DIR_APP = re.compile(r"(?:^|[^A-Za-z.])APP(?:[^A-Za-z.]|$)", re.I)+#: 品牌名只在目录里还带媒体平台词时才算推广目录;`草榴视频` 是推广 APP,+#: `麻豆传媒` 单独出现可能是这个厂牌的资源目录。+AD_DIR_BRAND_MEDIA = re.compile(r"(?:视频|視頻|影视|影視|传媒|傳媒|直播|短视频|短視頻)", re.I)+JUNK_KINDS = frozenset({"video", "image", "audio", "archive", "url", "other"})+#: 达到这个体积的视频不再当广告残留。这个值与原有「小于 120 MB」证据同源;+#: 较大的同番号短版本属于重复清理问题,由 ``q_duplicates`` 承接。+JUNK_VIDEO_MAX_BYTES = 120 * 1024**2+MEDIA_SIDECAR_SUFFIXES = frozenset({".nfo", ".srt", ".ass", ".ssa", ".vtt", ".sub", ".idx"})+INSTALLER_SUFFIXES = frozenset({".apk", ".exe", ".msi"})+#: 按固定命名识别的推广附件。名字和体积只能说明「像」,内容要人看过才算数。+PROMO_NAMES_FILE = PROJECT_ROOT / "resources" / "junk" / "promo_attachment_names.json"+#: 进入垃圾复核队列的分数线。+JUNK_REVIEW_SCORE = 40+#: `scripts/trash_junk.py --apply` 默认的自动移入回收站分数线。只能复核的证据+#: (固定命名、未探测时长视频的目录与体积证据)单独凑不到这一档。+AUTO_TRASH_SCORE = 60+#: 极短附件的结构证据只进入复核:目录中须有同名长正片,且附件相对它足够短、小。+SHORT_ATTACHMENT_SECONDS = 30+SHORT_ATTACHMENT_BYTES = 32 * 1024**2+SHORT_ATTACHMENT_CLUSTER = 3+#: 「来源站名 + 序号」也用于正常短片;站点水印不能成为自动隔离依据。+NUMBERED_SOURCE_VIDEO = re.compile(r"^(?:\[[^\]]+\]|【[^】]+】)[\s._-]*\d+(?:[\s._-]*\(\d+\))?$")+PAGE_COMPONENT_SUFFIXES = frozenset({".js", ".css", ".aspx"})+PAGE_CHROME_IMAGE = re.compile(r"^(?:banner(?:_[bs])?|1000x90(?:yunding)?|count\d*|logo|lan\d*|you|zuo|zuoxia)$", re.I)+PAGE_PROMO_CLAIM = re.compile(r"(?:记住|記住).{0,12}(?:网址|網址)|(?:网址|網址).{0,8}(?:发布|發布)")+++def promo_residue(name: str) -> int:+ """剥掉域名和推广套话后,还剩多少实质描述字符。++ 这是区分「广告」与「正片被打了站点水印」的关键:+ `点击观看 房间火爆` 剥完什么都不剩;+ `236953.xyz 推特新晋4年绿帽美腿淫妻网黄「一个ren」…` 剥完仍有大段内容描述。++ 剥掉的词替换成空串而不是空格:空格自己也在计数字符集里,换成空格等于每剥一个+ 词就补回一个残留。`爱姬远征-免费18禁手游-扫码安装` 剥掉四段套话后只剩四个字,+ 却因为补进来的四个空格数成 8,正好卡在门槛外——剥得越干净残留越高。+ """+ text = PROMO_DOMAIN.sub("", name or "")+ text = PROMO_PHRASE.sub("", text)+ text = PROMO_FILLER.sub("", text)+ # 只数中日韩文字与字母,忽略编号、扩展名和标点。+ return len(re.findall(r"[一-鿿-ヿ가- A-Za-z]", text))+++def _folder_index(connection, *, roots=None) -> tuple[dict[str, int], dict[str, set[str]]]:+ """每个目录里有多少资产,以及其中正片的文件名主体。++ 两条判据都要看候选的邻居而不只是它自己:成套资源看目录规模,封面与截图看+ 同目录有没有同名正片。`rsplit` 而不是 `PureWindowsPath`:ledger 路径统统是+ Windows 形态,这里要跑全表七万多行,只取目录名不值得为每行造一个路径对象。++ 正片不限于 `medium='video'`:蓝光原盘在账本里是 `other`,它同样是海报要配的那份+ 内容。用体积兜住这一类,不去猜扩展名——`.iso`、`.mds`、`BDMV` 之外还有什么,+ 库里下一份原盘才知道。+ """+ counts: dict[str, int] = {}+ videos: dict[str, set[str]] = {}+ scope, parameters = _scope_clause(roots=roots)+ for path, medium, size in connection.execute(+ "SELECT path,medium,size FROM asset WHERE path IS NOT NULL AND path<>''" + scope, parameters):+ folder, _, filename = str(path).rpartition("\\")+ counts[folder] = counts.get(folder, 0) + 1+ if medium == "video" or (size or 0) >= CONTENT_BYTES:+ stem = filename.rsplit(".", 1)[0] if "." in filename else filename+ key = stem.casefold()+ videos.setdefault(folder, set()).add(key)+ trimmed = DISC_TAIL.sub("", key)+ if trimmed:+ videos[folder].add(trimmed)+ return counts, videos+++def has_sibling_original(stem: str, sibling_videos: set[str]) -> bool:+ """True 表示同目录里有一条与它同名的正片,它是那条正片的封面或截图。"""+ key = str(stem or "").casefold()+ if not key:+ return False+ return key in sibling_videos or SHOT_TAIL.sub("", key) in sibling_videos+++def is_decorated_name(stem: str) -> bool:+ """名字挂着两个以上装饰符,没有任何实质描述。++ 样本是 `B:\\xxr\\0208 (23)` 里那批 7~20 KB 的 `❤91短视频❤.png`。+ 只挂一个爱心不算:Telegram 导出与正常标题里单个爱心常见,两个以上才是装饰性命名。+ """+ return sum(str(stem or "").count(mark) for mark in DECOR_MARKS) >= 2+++def _promo_directory(parts) -> bool:+ """路径里有任意一层目录自曝是推广包。++ 三种形态:`APP` 词形(`_含羞草APP`)、促销目录名(`一键约炮`、`論壇文宣`)、+ 品牌名叠加媒体词(`_草榴视频`)。只看目录分量,文件名自己是不是推广由调用方判。+ """+ for part in parts:+ text = str(part)+ if AD_DIR_APP.search(text) or AD_DIR_FORM.search(text):+ return True+ if AD_BRAND.search(text) and AD_DIR_BRAND_MEDIA.search(text):+ return True+ return False+++def _scope_clause(*, asset_ids=None, roots=None):+ """显式空范围只匹配空集,目录比较包含来源及分隔符边界。"""+ if asset_ids is not None:+ ids = tuple(dict.fromkeys(int(value) for value in asset_ids))+ return " AND id IN (SELECT value FROM json_each(?))", (json.dumps(ids),)+ if roots is None:+ return "", ()+ clauses, values = [], []+ for location, root in roots:+ root = str(PureWindowsPath(root)).rstrip("\\")+ prefix = root + "\\"+ clauses.append("(location=? AND (path=? COLLATE NOCASE OR substr(path,1,?)=? COLLATE NOCASE))")+ values.extend((location, root, len(prefix), prefix))+ return (" AND (" + " OR ".join(clauses) + ")", tuple(values)) if clauses else (" AND 0", ())+++def _junk_rows(connection, *, asset_ids=None, roots=None):+ """读取能进入垃圾复核的物理文件;字幕、NFO 与大视频在最外层排除。"""+ marks = ",".join("?" for _ in SCANNABLE_LOCATIONS)+ scope, parameters = _scope_clause(asset_ids=asset_ids, roots=roots)+ rows = connection.execute(+ "SELECT id,location,name,medium,creator,code,size,duration,width,height,snapshot_path,"+ "feedback,disposal,play_count,leave_ratio,o_count,studio,ctx_orient,path "+ f"FROM asset WHERE location IN ({marks}) AND disposal IS NULL "+ "AND (COALESCE(medium,'other')<>'video' OR (size < ? "+ "AND (duration IS NULL OR duration <= 1200)))" + scope,+ (*SCANNABLE_LOCATIONS, JUNK_VIDEO_MAX_BYTES, *parameters)).fetchall()+ # NFO 是媒体资料边车,不是等待清理的物理内容;即使名字或所在目录带推广词,+ # 也应由资料读取报告解析问题,不能进入会把文件移入回收站的垃圾队列。+ return [row for row in rows if _junk_file_eligible(row)]+++def _junk_file_eligible(row) -> bool:+ """资料边车和文件自身带番号的分卷不属于垃圾附件。"""+ item = PureWindowsPath(row["name"] or row["path"] or "")+ if item.suffix.casefold() in MEDIA_SIDECAR_SUFFIXES:+ return False+ identity = PART_MARK.sub("", PROMO_DOMAIN.sub("", item.stem)).strip("@._- ")+ return not (row["medium"] == "video" and PART_MARK.search(item.stem)+ and release_code_from_filename(identity))+++def _promo_neighbour_counts(rows) -> dict[str, int]:+ """统计同目录中缺少内容描述的推广名。"""+ counts: dict[str, int] = {}+ for row in rows:+ name = row["name"] or PureWindowsPath(row["path"] or "").name+ stem = PureWindowsPath(name).stem+ if not (PROMO_PHRASE.search(stem) or PROMO_DOMAIN.search(stem)):+ continue+ if promo_residue(stem) >= 14:+ continue+ folder = str(row["path"] or name).rpartition("\\")[0]+ counts[folder] = counts.get(folder, 0) + 1+ return counts+++@cache+def promo_name_rules() -> tuple[dict, ...]:+ """读取 `PROMO_NAMES_FILE` 的推广附件命名规则;文件缺失或格式不对直接报错。"""+ data = json.loads(PROMO_NAMES_FILE.read_text(encoding="utf-8"))+ rules = []+ for entry in data["rules"]:+ names = frozenset(re.sub(r"\s", "", name).casefold() for name in entry.get("names", ()))+ patterns = tuple(re.compile(pattern, re.I) for pattern in entry.get("patterns", ()))+ if not entry["reason"] or not (names or patterns) or int(entry["max_bytes"]) <= 0:+ raise ValueError(f"推广附件命名规则不完整:{entry}")+ rules.append({+ "reason": str(entry["reason"]),+ "suffixes": frozenset(str(suffix).casefold() for suffix in entry["suffixes"]),+ "max_bytes": int(entry["max_bytes"]),+ "parent_is_name": bool(entry.get("parent_is_name")),+ "names": names,+ "patterns": patterns,+ })+ return tuple(rules)+++def _promo_name_reason(suffix: str, path: str, size: int) -> str:+ """命中词库里推广附件固定命名的理由;只够进复核队列,见 `AUTO_TRASH_SCORE`。"""+ item = PureWindowsPath(path)+ stem = item.stem+ for rule in promo_name_rules():+ if suffix not in rule["suffixes"] or not 0 < size <= rule["max_bytes"]:+ continue+ if rule["parent_is_name"] and item.parent.name != stem:+ continue+ if (re.sub(r"\s", "", stem).casefold() in rule["names"]+ or any(pattern.fullmatch(stem) for pattern in rule["patterns"])):+ return rule["reason"]+ return ""+++def _attachment_junk_reason(suffix: str, path: str, size: int) -> str:+ """区分安装附件与网页存档组件。"""+ if suffix in INSTALLER_SUFFIXES and size < 64 * 1024**2:+ return "媒体目录中的安装附件"+ if suffix in PAGE_COMPONENT_SUFFIXES and any(+ part.casefold().endswith("_files") for part in PureWindowsPath(path).parent.parts):+ return "网页存档的脚本或样式附件"+ parent = PureWindowsPath(path).parent.name+ if (parent.casefold().endswith('_files') and PAGE_PROMO_CLAIM.search(parent)+ and AD_DOMAIN.search(parent[:-len('_files')]) and suffix in {'.gif', '.jpg', '.png'}+ and size < 2 * 1024**2+ and (PAGE_CHROME_IMAGE.fullmatch(PureWindowsPath(path).stem)+ or (PureWindowsPath(path).name.casefold() == '1.gif' and size <= 4096))):+ return "推广网页存档的横幅、标志或导航图片"+ return ""+++def _short_attachment(row) -> bool:+ """有测量值、没有自身作品身份或分卷标记的极短视频。"""+ stem = PureWindowsPath(row["name"] or row["path"] or "").stem+ owner = row["creator"] or ""+ return (row["medium"] == "video"+ and 0 < (row["duration"] or 0) <= SHORT_ATTACHMENT_SECONDS+ and 0 < (row["size"] or 0) <= SHORT_ATTACHMENT_BYTES+ and not (owner and not AD_DOMAIN.search(owner))+ and not PART_MARK.search(stem)+ and not release_code_from_filename(stem))+++def _release_main_videos(connection, *, roots=None) -> dict[str, list[tuple[float, int]]]:+ """按活动物理文件建立发行目录的长正片索引;普通混合目录不算发行目录。"""+ mains: dict[str, list[tuple[float, int]]] = {}+ marks = ",".join("?" for _ in SCANNABLE_LOCATIONS)+ scope, parameters = _scope_clause(roots=roots)+ for path, duration, size in connection.execute(+ "SELECT path,duration,size FROM asset WHERE disposal IS NULL AND medium='video' "+ f"AND location IN ({marks}) AND duration>=1200 AND size>=? AND path IS NOT NULL" + scope,+ (*SCANNABLE_LOCATIONS, JUNK_VIDEO_MAX_BYTES, *parameters)):+ item = PureWindowsPath(path)+ key = promo_free_key(item.stem)+ if key and key == promo_free_key(item.parent.name):+ mains.setdefault(str(item.parent).casefold(), []).append((duration, size))+ return mains+++def _short_attachment_reasons(connection, rows, *, roots=None) -> dict[int, str]:+ """从正片与附件的量级关系、同目录成群和跨发行目录扩散生成复核证据。++ 不读媒体,也不按题材、游戏名或推广词命中。相同测量值只是扩散线索,不能证明内容相同。+ """+ mains = _release_main_videos(connection, roots=roots)+ folders: dict[str, set[str]] = {}+ signatures: dict[tuple, set[str]] = {}+ candidates = []+ for row in rows:+ if not _short_attachment(row):+ continue+ path = PureWindowsPath(row["path"] or "")+ folder = str(path.parent).casefold()+ if not any(duration >= row["duration"] * 40 and size >= row["size"] * 10+ for duration, size in mains.get(folder, ())):+ continue+ signature = (row["size"], round(row["duration"], 3), row["width"], row["height"])+ folders.setdefault(folder, set()).add(str(path).casefold())+ if (row["width"] or 0) > 0 and (row["height"] or 0) > 0:+ signatures.setdefault(signature, set()).add(folder)+ candidates.append((row["id"], folder, signature))+ reasons = {}+ for asset_id, folder, signature in candidates:+ count = len(folders[folder])+ spread = len(signatures.get(signature, ()))+ if count >= SHORT_ATTACHMENT_CLUSTER:+ reasons[asset_id] = f"同目录长正片旁有 {count} 条极短小附件,须核验内容"+ elif spread >= SHORT_ATTACHMENT_CLUSTER:+ reasons[asset_id] = f"同体积、时长与分辨率的极短附件散布在 {spread} 个发行目录,须核验内容"+ return reasons+++def _video_evidence(row, stem: str, longer: dict, promo_dir: bool,+ directory_points: int, attachment_reason: str = "") -> tuple[int, int, list[str]]:+ """视频自己的证据:同番号完整版、时长、体积与推广目录。++ 返回(分数,其中只够复核的分数,理由)。时长没探测到的视频可能就是正片:+ 推广创作者位或「域名+番号」目录(`directory_points`)、推广目录与体积只说明它住在哪、+ 有多大,说明不了它是广告,这些分数只够进复核队列。+ """+ score, why = 0, []+ code = (row["code"] or "").strip()+ duration = row["duration"]+ longest = longer.get(code)+ if longest and REAL_CODE.match(code) and duration is not None and 0 < duration < longest * 0.2 \+ and not PART_MARK.search(stem):+ # 分卷已排除,真番号下不到两成时长基本就是片段/预告,单独即可入队复核。+ # 用户标记的 `反抗不如享受.mp4`(ABW-220,244 秒)正好卡在旧的 35 分门外。+ score += 40; why.append(f"同番号有 {longest/60:.0f} 分完整版")+ if duration is not None and 0 < duration < 240:+ score += 15; why.append("不足 4 分钟")+ context = directory_points+ if (row["size"] or 0) < JUNK_VIDEO_MAX_BYTES:+ score += 10; context += 10; why.append("小于 120 MB")+ if promo_dir:+ # 30 分单独不构成删片理由:正片也可能躺在别人起错名的目录里,+ # 要再叠一条时长或体积证据才到门槛。+ score += 30; context += 30; why.append("住在推广目录")+ probed = duration is not None and duration > 0+ review_only = 0 if probed else context+ if NUMBERED_SOURCE_VIDEO.fullmatch(stem) and PROMO_DOMAIN.search(stem):+ review_only += AUTO_TRASH_SCORE+ why.append("编号视频可能只有来源水印,须核验内容")+ if attachment_reason:+ score += JUNK_REVIEW_SCORE; review_only += JUNK_REVIEW_SCORE; why.append(attachment_reason)+ return score, review_only, why+++def _judgement_inputs(database, *, asset_ids, roots):+ """读取同一账本快照中的候选、用户决定、作品版本与目录上下文。"""+ with database.read_connection() as c:+ rows = _junk_rows(c, asset_ids=asset_ids, roots=roots)+ if not rows:+ return [], {}, frozenset(), {}, {}, {}+ # 同番号是否存在明显更长的版本;只在 code 是真番号时才有意义。+ longer = {r[0]: r[1] for r in c.execute(+ "SELECT code, max(duration) FROM asset WHERE medium='video' AND code IS NOT NULL "+ "AND code<>'' AND duration IS NOT NULL GROUP BY code")}+ dismissed_keys = [str(row[0]) for row in c.execute(+ "SELECT item_key FROM review_decision "+ "WHERE category='junk_file' AND status='rejected'"+ )]+ dismissed_ids = {int(key) for key in dismissed_keys if key.isdigit()}+ # 显式资产可以横跨全库;邻居一次读取,避免把成千目录拼成 SQL OR 表达式。+ context = None if asset_ids is not None else roots+ folder_assets, folder_videos = _folder_index(c, roots=context)+ context_rows = _junk_rows(c, roots=context) if asset_ids is not None else rows+ short_attachments = _short_attachment_reasons(c, context_rows, roots=context)+ return rows, longer, frozenset(dismissed_ids), folder_assets, folder_videos, short_attachments+++def scored_candidates(database, *, asset_ids=None, roots=None) -> tuple[list[dict], frozenset[int]]:+ """候选按嫌疑分从高到低排序,连同用户已确认不是垃圾的 id。"""+ rows, longer, dismissed_ids, folder_assets, folder_videos, short_attachments = _judgement_inputs(+ database, asset_ids=asset_ids, roots=roots)+ # 每个目录里挂着推广名的候选有多少个;插页判据要看它有没有同伙。+ promo_neighbours = _promo_neighbour_counts(rows)+ out = []+ for r in rows:+ d = dict(r)+ s, why = 0, []+ name = d.get("name") or PureWindowsPath(d.get("path") or "").name+ name_path = PureWindowsPath(name)+ nm = name_path.stem+ suffix = name_path.suffix.casefold()+ d["junk_kind"] = (+ "url" if suffix in INTERNET_SHORTCUT_SUFFIXES+ else (d.get("medium") if d.get("medium") in JUNK_KINDS else "other")+ )+ residue = promo_residue(nm)+ promo = bool(PROMO_PHRASE.search(nm) or PROMO_DOMAIN.search(nm))+ page_archive = suffix in PAGE_ARCHIVE_SUFFIXES+ if suffix in INTERNET_SHORTCUT_SUFFIXES:+ s += 60; why.append("网址快捷方式")+ elif page_archive:+ s += 60; why.append("网页存档")+ attachment_reason = _attachment_junk_reason(suffix, d.get("path") or name, d.get("size") or 0)+ if attachment_reason:+ s += 60; why.append(attachment_reason)+ # 只够进复核队列的分数:固定命名,和未探测时长视频的目录与体积证据。+ review_only = 0+ name_reason = _promo_name_reason(suffix, d.get("path") or name, d.get("size") or 0)+ if name_reason:+ s += JUNK_REVIEW_SCORE; review_only += JUNK_REVIEW_SCORE; why.append(name_reason)+ # 目录维度的证据:广告包的文件名往往干净(`极道世界.mp4`),唯一线索在旧导入器+ # 从目录名投影出来的创作者位或路径里。creator 位本身是推广站域名时,它就不再是+ # 「有归属所以是正片」的证据,下面两处对 creator 的信任都必须先排除这种情况。+ owner = d.get("creator") or ""+ owner_is_promo = bool(AD_DOMAIN.search(owner))+ real_owner = bool(owner) and not owner_is_promo+ # ledger 路径在两个平台都是 Windows 形态。`rpartition` 没有平台依赖,+ # 键也与 `_folder_index` 完全同源;`os.path.dirname` 在 macOS 会把整条+ # 反斜杠路径当成文件名。+ folder = str(d.get("path") or name).rpartition("\\")[0]+ # 成套下载的资源包里,域名是打包渠道给整包起的名,不是插页的自我暴露。+ bundled = folder_assets.get(folder, 0) >= BUNDLE_DIR_ASSETS+ # 目录名带推广站域名只说明「从哪个站下的」,说明不了这个文件是广告。+ # 文件自己是真番号、自己就有内容级的体积、或者是同目录正片的封面/截图时,+ # 目录证据就不成立。+ #+ # 只放非视频:广告包里的视频本来就叫 `极道世界.mp4`,它的 code 是旧导入器从+ # `bbsxv.xyz-DOCP-324` 目录名投影出来的真番号形状,而它自己就在同目录的正片+ # 名单里——两条豁免对视频都会自动成立,`AD_DIRPACK` 这条判据就没了。+ self_evident = d.get("medium") != "video" and (+ is_jav_code(d.get("code"))+ or (d.get("size") or 0) >= CONTENT_BYTES+ or has_sibling_original(nm, folder_videos.get(folder, frozenset()))+ )+ decorated = (d.get("medium") != "video" and is_decorated_name(nm)+ and (d.get("size") or 0) <= DECORATED_MAX_BYTES)+ brand_hit = bool(AD_BRAND.search(nm)) and bool(+ AD_BRAND_FORM.search(nm)+ or (d.get("medium") == "image"+ and (d.get("size") or 0) <= BRAND_TINY_BYTES))+ promo_dir = _promo_directory(PureWindowsPath(d.get("path") or name).parent.parts)+ if promo and residue < 6 and not bundled:+ # 名字剥完只剩广告本身,这是最硬的信号。+ s += 60; why.append("整个名字都是推广语")+ elif promo and residue < 14 and not real_owner and not bundled:+ s += 30; why.append("推广语占了名字主体")+ neighbours = promo_neighbours.get(folder, 0)+ if neighbours >= PROMO_CLUSTER_FILES:+ s += 30+ why.append(f"同目录另有 {neighbours - 1} 个同类推广名")+ directory_points = 0+ if owner_is_promo:+ directory_points = 50; why.append("创作者位是推广站域名")+ elif AD_DIRPACK.search(folder) and not self_evident:+ directory_points = 45; why.append("目录是「域名+番号」的推广打包")+ s += directory_points+ # 非视频的推广形态:装饰符小图、品牌推广名、住在推广目录。`self_evident`+ # (真番号、内容级体积、正片配套图)与网页存档、网址快捷方式已单独计分,+ # 这里不重复叠加。+ if (d.get("medium") != "video" and not self_evident and not page_archive+ and suffix not in INTERNET_SHORTCUT_SUFFIXES):+ if decorated:+ s += 60; why.append("装饰符小图")+ elif brand_hit:+ s += 60; why.append("推广品牌名")+ elif promo_dir:+ s += 50; why.append("住在推广目录")+ if d.get("medium") == "video":+ points, weak, reasons = _video_evidence(+ d, nm, longer, promo_dir, directory_points, short_attachments.get(d["id"], ""))+ s += points; review_only += weak; why.extend(reasons)+ # 有真实创作者归属、且名字剥完仍有实质描述的,是被打了水印的正片,不是广告。+ if real_owner and residue >= 14 and not (attachment_reason or name_reason):+ s -= 45+ if review_only and s - review_only < AUTO_TRASH_SCORE:+ s = min(s, AUTO_TRASH_SCORE - 1)+ if s >= JUNK_REVIEW_SCORE:+ d["score"] = s; d["why"] = " · ".join(why)+ out.append(d)+ out.sort(key=lambda x: (-x["score"], -(x["size"] or 0)))+ return out, frozenset(dismissed_ids)+++def isolate(database, *, asset_ids=None, roots=None, protected_ids=()) -> dict:+ """将明确范围内的高分垃圾移入可还原回收站,不读取或删除媒体。++ 人工保留、个人记录、文件自身作品身份和调用方点名的正片受保护。写入前重查评分依据,+ 扫描或人工操作在计算期间改变了资产时留待下一次判定。+ """+ if asset_ids is None and roots is None:+ raise ValueError("自动垃圾检查必须指定资产或来源目录")+ if not isinstance(database, LedgerDatabase):+ database = LedgerDatabase(database)+ scored, dismissed = scored_candidates(database, asset_ids=asset_ids, roots=roots)+ protected = set(protected_ids) | set(dismissed)+ chosen = [row for row in scored if row['score'] >= AUTO_TRASH_SCORE and row['id'] not in protected]+ isolated = []+ if not chosen:+ return {'candidates': len(scored), 'review': sum(row['id'] not in dismissed for row in scored),+ 'isolated': 0, 'ids': []}+ with database.write_transaction() as connection:+ connection.execute('BEGIN IMMEDIATE')+ holders = record_holders(connection, [row['id'] for row in chosen])+ for row in chosen:+ current = connection.execute("SELECT * FROM asset WHERE id=?", (row['id'],)).fetchone()+ if not _can_isolate(connection, current, row, holders):+ continue+ connection.execute(+ "UPDATE asset SET disposal='trash',disposal_reason='junk',feedback_at=? WHERE id=? AND disposal IS NULL",+ (time.time(), row['id']))+ isolated.append(row['id'])+ return {'candidates': len(scored), 'review': sum(row['id'] not in dismissed for row in scored+ if row['id'] not in isolated),+ 'isolated': len(isolated), 'ids': isolated}+++def _can_isolate(connection, current, evidence, holders) -> bool:+ """当前资产仍符合已算出的证据,且没有用户保留意图或自身作品身份。"""+ if current is None or current['disposal'] is not None or current['id'] in holders:+ return False+ if current['feedback'] == 'like' or any(current[key] != evidence[key] for key in+ ('location', 'path', 'name', 'size', 'duration', 'creator', 'code', 'medium')):+ return False+ if connection.execute("SELECT 1 FROM review_decision WHERE category='junk_file' AND status='rejected' "+ "AND item_key=?", (str(current['id']),)).fetchone():+ return False+ if connection.execute("SELECT 1 FROM asset_subtitle WHERE asset_id=?", (current['id'],)).fetchone():+ return False+ if current['medium'] == 'video' and (current['duration'] is None or current['duration'] <= 0):+ return False+ stem = PureWindowsPath(current['name'] or current['path'] or '').stem+ identity = PART_MARK.sub('', PROMO_DOMAIN.sub('', stem)).strip('@._- ')+ return not (current['medium'] == 'video' and release_code_from_filename(identity))++diff --git a/src/peach/library_processing.py b/src/peach/library_processing.pyindex d24c3ceb2..873191800 100644--- a/src/peach/library_processing.py+++ b/src/peach/library_processing.py@@ -30,6 +30,7 @@ from .library_nfo import directory_files, read_nfo, sidecars, local_art, Sidecarfrom .genre_decisions import load_genre_decisionsfrom .metadata import extract_catalog_evidence, extract_peach_fields, validate_provider_codefrom . import metadata_routes+from . import junkfrom . import sample_followupfrom .metadata_policy import SOURCE_SPECS, preferred_tag_sourcefrom .platform import root_online, translate_ledger_path@@ -1743,12 +1744,15 @@ class _SourceScan:scanned: int = 0probed: int = 0scanned_bytes: int = 0+ junk_isolated: int = 0def _scan_sources(config, db_path, *, locations, asset_ids, stage, guard, update, issue,scan_checkpoints=None, save_checkpoint=lambda: None, initial_probed=0):"""发现本趟可访问的来源根,按阶段登记文件并探测新视频。"""result = _SourceScan([], probed=initial_probed)+ def count_junk(checked):+ result.junk_isolated += checked['isolated']scan_checkpoints = scan_checkpoints if scan_checkpoints is not None else {}chosen_paths = _chosen_asset_paths(db_path, asset_ids)scan = asset_ids is None and stage in (ALL_STAGES, SCAN_STAGE)@@ -1778,9 +1782,11 @@ def _scan_sources(config, db_path, *, locations, asset_ids, stage, guard, updatechecked=0, total=0, current_action='scanning_files'))result.scanned += registered.filesresult.scanned_bytes += registered.total_bytes+ result.junk_isolated += registered.junk_isolatedif stage == SCAN_STAGE:result.probed += probe_unmeasured(db_path, ffprobe, source, root, config=config,+ on_junk=count_junk,report=lambda done, total: update(stage='读取时长与分辨率',checked=result.probed + done, total=result.probed + total, probed=result.probed + done))reattach_in(db_path, registered.new_ids)@@ -1788,7 +1794,8 @@ def _scan_sources(config, db_path, *, locations, asset_ids, stage, guard, updatereturn result-def _probe_sources(config, db_path, locations, online_roots, asset_ids, update, *, total_pending=0, initial_probed=0):+def _probe_sources(config, db_path, locations, online_roots, asset_ids, update, *, total_pending=0,+ initial_probed=0, protected_ids=(), on_junk=lambda result: None):"""已登记视频按可访问来源逐项读取媒体属性。"""probed = initial_probedchecked = initial_probed@@ -1801,12 +1808,49 @@ def _probe_sources(config, db_path, locations, online_roots, asset_ids, update,if translate_ledger_path(root) in online_roots:probed += probe_unmeasured(db_path, ffprobe, source, root, asset_ids=asset_ids, config=config,- report=report)+ report=report, protected_ids=protected_ids, on_junk=on_junk)checked = probedupdate(probed=probed)return probed+def _check_processing_junk(database, locations, online_roots, asset_ids, protected_ids, state, update):+ """只检查本趟可访问目录或显式选择,并把隔离与复核计数记进可恢复状态。"""+ roots = [(source, root) for source, paths in locations.items() for root in paths+ if translate_ledger_path(root) in online_roots]+ checked = junk.isolate(database, asset_ids=asset_ids, roots=roots, protected_ids=protected_ids)+ update(junk_isolated=state.get('junk_isolated', 0) + checked['isolated'],+ junk_review=checked['review'])+++def _prepare_processing_media(config, db_path, database, locations, sources, asset_ids,+ protected_ids, stage, state, update):+ """在配套资料、预览和联网采集前检查已知证据,补媒体测量后重新检查。"""+ update(stage='检查垃圾文件', checked=0, total=0)+ _check_processing_junk(database, locations, sources.online_roots, asset_ids, protected_ids, state, update)+ if stage in (ALL_STAGES, COLLECT_STAGE, PREVIEWS_STAGE):+ _probe_sources(config, db_path, locations, sources.online_roots, asset_ids, update,+ initial_probed=state['probed'], protected_ids=protected_ids,+ on_junk=lambda result: update(junk_isolated=state['junk_isolated'] + result['isolated']))+ update(stage='复查垃圾文件', checked=0, total=0)+ _check_processing_junk(database, locations, sources.online_roots, asset_ids, protected_ids, state, update)+++def _active_prepared(database, prepared):+ """媒体处理后的活动视频携带最新测量值,已隔离或被用户移出的条目不继续采集。"""+ if not prepared:+ return []+ ids = [row[0]['id'] for row in prepared]+ with database.read_connection() as connection:+ current = {}+ for start in range(0, len(ids), 500):+ batch = ids[start:start + 500]+ current.update((row['id'], dict(row)) for row in connection.execute(+ "SELECT * FROM asset WHERE disposal IS NULL AND id IN (" + ','.join('?' for _ in batch) + ')', batch))+ return [(dict(item[0], **current[item[0]['id']]), *item[1:]) for item in prepared+ if item[0]['id'] in current]++def _collection_rows(db_path, *, locations, online_roots, asset_ids):"""读取可访问视频与采集所需的演员、内容标签和已收录 genre。"""multi = ("(SELECT group_concat(entity.canonical_name) FROM asset_entity"@@ -2113,11 +2157,13 @@ def _processing_steps(stage, asset_ids, *, metadata_only_enabled):"""按本趟真实处理范围构造有序阶段清单。"""scanning = asset_ids is None and stage in (ALL_STAGES, SCAN_STAGE)steps = [('sources', '扫描文件' if scanning else '检查媒体目录')]+ steps.append(('inspect', '检查垃圾文件' if stage in (SCAN_STAGE, PREVIEWS_STAGE, METADATA_STAGE)+ else '读取媒体属性与检查垃圾'))if stage not in (SCAN_STAGE, PREVIEWS_STAGE):steps.append(('local', '检查配套文件'))if stage == PREVIEWS_STAGE or stage in (ALL_STAGES, COLLECT_STAGE):- media_label = ('生成视频预览' if stage == PREVIEWS_STAGE else- '读取媒体属性' if metadata_only_enabled else '读取媒体属性与生成预览')+ media_label = ('生成视频预览' if stage == PREVIEWS_STAGE or not metadata_only_enabled+ else '检查媒体预览设置')steps.append(('media', media_label))if stage not in (SCAN_STAGE, PREVIEWS_STAGE):steps.append(('collect', '采集外部资料与封面'))@@ -2157,7 +2203,7 @@ def _restore_processing_counts(state, previous):"""同一处理任务沿用累计数、问题清单和已保存证据索引。"""previous = previous or {}for key in ('started_at', 'identified', 'covers', 'previews', 'probed', 'issue_count', 'paused_count',- 'issue_preview', 'issues_truncated', 'notes', 'retryable_asset_ids'):+ 'issue_preview', 'issues_truncated', 'notes', 'retryable_asset_ids', 'junk_isolated', 'junk_review'):if key in previous:state[key] = previous[key]for key in ('scan_checkpoints', 'local_metadata'):@@ -2205,7 +2251,7 @@ def process_library(config, db_path, candidate_root, cover_root, *, location='coreport=lambda state: None, provider_factory=None, job_id=None,retry_ids=None, active=lambda: True, stage=ALL_STAGES,database=None, route_overrides=None, state_file=None, asset_ids=None,- resume_state=None, speed_mode=None):+ resume_state=None, speed_mode=None, protected_ids=()):"""登记文件与确定的番号,外部资料保留为可复核候选。`retry_ids` 为 `None` 时处理整个馆藏;给定时只处理这些项目(上一任务记录的@@ -2216,13 +2262,16 @@ def process_library(config, db_path, candidate_root, cover_root, *, location='co`asset_ids` 指定仍在馆藏中的视频,不扫描来源目录;空选择无效。`database` 是调用方已经在用的 `LedgerDatabase`。给了它才做收尾的自动落库与人物- 资料补齐与本地预览登记,并且与调用方共用同一把进程内写锁;不给就只采集候选。+ 资料补齐与本地预览登记,并且与调用方共用同一把进程内写锁。所有模式都会按范围+ 检查垃圾并隔离到回收站;不给 `database` 时外部资料只采集候选。`route_overrides` 覆盖某几种内容类型的来源链,形状与判据见`metadata_routes.parse_route_overrides`;不给就用内建那张表。`state_file` 给后台定向任务保存独立进度,共享候选写入仍使用全库处理锁。"""selected_ids = _validate_processing_request(config, db_path, stage, asset_ids, retry_ids, database)+ from .repository import LedgerDatabase+ judgement_database = database if database is not None else LedgerDatabase(db_path)routes = metadata_routes.parse_route_overrides(route_overrides)path = Path(state_file) if state_file is not None else state_path(config)path.parent.mkdir(parents=True, exist_ok=True)@@ -2257,6 +2306,7 @@ def process_library(config, db_path, candidate_root, cover_root, *, location='cocurrent_asset_id=None, current_asset_name='', current_action='',current_started_at=None, current_deadline_at=None,followups=[],+ junk_isolated=0, junk_review=0,started_at=time.time(), error='')state.update(initial_estimate(config, db_path, stage, scope_ids, speed_mode))state['steps'] = _processing_steps(stage, scope_ids,@@ -2340,9 +2390,13 @@ def process_library(config, db_path, candidate_root, cover_root, *, location='coscan_checkpoints=state['scan_checkpoints'],save_checkpoint=lambda: flush_state(force=True), initial_probed=state['probed'])state.update(scanned=sources.scanned, scanned_bytes=sources.scanned_bytes, probed=sources.probed)+ state['junk_isolated'] = max(state['junk_isolated'], sources.junk_isolated)traffic.unknown_files = Falsetraffic.refresh_plan()finish_step('sources')+ _prepare_processing_media(config, db_path, judgement_database, locations, sources,+ scope_ids, protected_ids, stage, state, update)+ finish_step('inspect')if stage == SCAN_STAGE:# `checked`/`total` 在探时长时借给进度条用过;只扫描这一段没有采集,读数归零。update(status='failed' if state['issue_count'] else 'complete',@@ -2359,6 +2413,8 @@ def process_library(config, db_path, candidate_root, cover_root, *, location='cocompleted=checkpoints.completed['completed_previews'],initial_made=state['previews'],checkpoint=lambda row: checkpoints.mark('completed_previews', row, flush_state))+ _check_processing_junk(judgement_database, locations, sources.online_roots,+ scope_ids, protected_ids, state, update)finish_step('media')update(status='failed' if state['issue_count'] else 'complete', stage='处理结束',completed_at=time.time(), error=issue_summary(state['issue_count'], state['paused_count']))@@ -2433,8 +2489,6 @@ def process_library(config, db_path, candidate_root, cover_root, *, location='cofinish_step('local')traffic.prepare_metadata(prepared)if stage in (ALL_STAGES, COLLECT_STAGE):- _probe_sources(config, db_path, locations, sources.online_roots, scope_ids, update,- total_pending=pending.get('probe', 0), initial_probed=state['probed'])pending.pop('probe', None)update(estimate_pending=dict(pending), stage='准备生成预览', checked=0, total=0)if not metadata_only(config.directory('state')):@@ -2445,6 +2499,9 @@ def process_library(config, db_path, candidate_root, cover_root, *, location='copending.pop('previews', None)update(estimate_pending=dict(pending), stage='准备采集外部资料', checked=0, total=0)finish_step('media')+ _check_processing_junk(judgement_database, locations, sources.online_roots,+ scope_ids, protected_ids, state, update)+ prepared = _active_prepared(judgement_database, prepared)for position, (row, code, verify_existing, local_fields, local_entries, local_ok) in enumerate(prepared):local_ok = local_ok and row['id'] not in poster_failedguard.check()diff --git a/src/peach/media_probe.py b/src/peach/media_probe.pyindex 0bb8a8f96..b7c4bbea2 100644--- a/src/peach/media_probe.py+++ b/src/peach/media_probe.py@@ -27,6 +27,7 @@ from .media import resolve_case_insensitivefrom .media_sources import cost_forfrom .strm import skip_background_readfrom .processing_control import CURRENT, checkpoint+from . import junk, record_rehome#: 一次探测最多等多久。ffprobe 只读文件头,网盘上实测一两秒。TIMEOUT_SECONDS = 20.0@@ -118,14 +119,15 @@ def probe_unmeasured(db_path, ffprobe: str | None, location: str, root: str, *,report: Callable[[int, int], None] = lambda done, total: None,asset_ids: Sequence[int] | None = None,- config=None,+ config=None, protected_ids=(), on_junk=lambda result: None,) -> int:"""`root`(账本口径)下还没探过的视频逐个探一遍,返回探了几个。"""if not ffprobe:return 0- prefix = str(PureWindowsPath(root)).rstrip("\\") + "\\"+ exact = str(PureWindowsPath(root)).rstrip("\\")+ prefix = exact + "\\"selected = ''- parameters = (location, len(prefix), prefix)+ parameters = (location, exact, len(prefix), prefix)if asset_ids is not None:if not asset_ids:return 0@@ -133,10 +135,10 @@ def probe_unmeasured(parameters += tuple(asset_ids)with closing(sqlite3.connect(db_path, timeout=30)) as connection:rows = connection.execute(- f"SELECT id,path FROM asset WHERE {_UNMEASURED} AND substr(path,1,?)=?{selected} ORDER BY id",+ f"SELECT id,path FROM asset WHERE {_UNMEASURED} AND (path=? OR substr(path,1,?)=?){selected} ORDER BY id",parameters).fetchall()rows = [row for row in rows if cost_for(location, row[1], config) == 'free']- return _record(db_path, ffprobe, rows, report)+ return _record(db_path, ffprobe, rows, report, protected_ids=protected_ids, on_junk=on_junk)def probe_path(db_path, ffprobe: str | None, location: str, path: str, *, config=None) -> int:@@ -151,7 +153,8 @@ def probe_path(db_path, ffprobe: str | None, location: str, path: str, *, configreturn _record(db_path, ffprobe, rows, lambda done, total: None)-def _record(db_path, ffprobe: str, rows: Sequence, report: Callable[[int, int], None]) -> int:+def _record(db_path, ffprobe: str, rows: Sequence, report: Callable[[int, int], None], *,+ protected_ids=(), on_junk=lambda result: None) -> int:"""探完一条写一条。`report` 抛出来(任务被停)时,还没开始的那些不再探。"""rows = [row for row in rows if not skip_background_read(row[1])]if not rows:@@ -164,6 +167,9 @@ def _record(db_path, ffprobe: str, rows: Sequence, report: Callable[[int, int],connection.commit()done += 1report(done, len(rows))+ record_rehome.reattach(connection, [row[0] for row in rows])+ connection.commit()+ on_junk(junk.isolate(db_path, asset_ids=[row[0] for row in rows], protected_ids=protected_ids))return donediff --git a/src/peach/organize.py b/src/peach/organize.pyindex 4e742492b..b265cdd75 100644--- a/src/peach/organize.py+++ b/src/peach/organize.py@@ -422,10 +422,13 @@ def apply_plan(connection: sqlite3.Connection, rows: Sequence[dict], *,if int(row['asset_id']) in completed:continueold_path, new_path = str(row["current_path"]), str(row["target_path"])- current = connection.execute('SELECT path FROM asset WHERE id=?', (int(row['asset_id']),)).fetchone()+ current = connection.execute('SELECT path,disposal FROM asset WHERE id=?', (int(row['asset_id']),)).fetchone()landed = (current is not None and current[0] == new_pathand translate_ledger_path(new_path).is_file() and not translate_ledger_path(old_path).exists())- outcome = 'ok' if landed else _move_one(connection, int(row["asset_id"]), old_path, new_path)+ if current is None or current[1] is not None:+ outcome = '资产已不在馆藏,未执行整理'+ else:+ outcome = 'ok' if landed else _move_one(connection, int(row["asset_id"]), old_path, new_path)if outcome == "ok":entries.append({"asset_id": int(row["asset_id"]),"old_path": old_path, "new_path": new_path})diff --git a/src/peach/scan.py b/src/peach/scan.pyindex 2219f6151..770ee0891 100644--- a/src/peach/scan.py+++ b/src/peach/scan.py@@ -41,7 +41,7 @@ from dataclasses import asdict, dataclassfrom pathlib import Path, PureWindowsPathfrom filelock import FileLock-from . import folder_rules, record_rehome, subtitles, wants+from . import folder_rules, junk, record_rehome, subtitles, wantsfrom .platform import is_windows_path, resolve_location, resolve_rootfrom .processing_control import checkpointfrom .fsutil import atomic_write_text@@ -308,6 +308,7 @@ def ingest_path(connection.commit()finally:connection.close()+ junk.isolate(db_path, asset_ids=[asset_id])return IngestResult(location, str(ledger_path), True, stat.st_size, tracks,asset_id=asset_id, new=new, reattached=reattached, acquired=acquired)@@ -332,12 +333,15 @@ class ScanResult:reattached: int = 0#: 本次让几条「想要」标了已入库。acquired: int = 0+ junk_isolated: int = 0+ junk_review: int = 0def summary(self) -> str:return (f"✓ {self.location}: {self.files:,} 文件 / "f"{self.total_bytes / 1024 ** 4:.2f} TB / 耗时 {self.seconds:.0f}s;"f"附属文件未登记 {self.sidecars:,} 个;清单中已消失 {self.gone:,} 个;"- f"字幕 {self.subtitles:,} 条(孤立 {self.orphan_subtitles:,} 条)")+ f"字幕 {self.subtitles:,} 条(孤立 {self.orphan_subtitles:,} 条);"+ f"垃圾移入回收站 {self.junk_isolated:,} 个,待复核 {self.junk_review:,} 个")def _walk(top: Path, pending: list[str] | None = None, *, onerror=lambda error: None):@@ -527,8 +531,10 @@ def _scan_location((location, now)).fetchone()[0]finally:connection.close()+ checked = junk.isolate(db_path, roots=[(location, root)])result = ScanResult(location, root, files, total, time.time() - started, gone,- tracks, orphans, skipped, new_ids, reattached, acquired)+ tracks, orphans, skipped, new_ids, reattached, acquired,+ checked['isolated'], checked['review'])save_checkpoint({'result': asdict(result)})progress(files, total, force=True)report(result.summary())diff --git a/src/peach/web_batch.py b/src/peach/web_batch.pyindex 109a0259f..371dcced1 100644--- a/src/peach/web_batch.py+++ b/src/peach/web_batch.py@@ -1,9 +1,7 @@-"""数据清理:广告残留判定、批量操作、物理清除与回收站。+"""垃圾复核展示、批量决定、物理清除与回收站。-放在同一个模块是因为它们是同一条链上的四段:`q_ads` 判定哪些文件是广告残留,-`w_batch` 把用户的批量决定落库,`purge_assets` 真正动磁盘,`w_empty_trash` 收尾。-判定与执行分家的话,「命中推广词」和「可以删」之间那道界线就没人守了——-剥掉推广词后还剩内容的文件不是广告,这条只有把判据和删除放在一起看才成立。+`junk` 提供各流程共用的证据与隔离规则;本模块将评分转成页面数据,并执行用户的+批量决定。自动隔离只进入回收站,永久删除需要独立的确认与取证流程。物理删除要清的引用表只写在 `personal_records.ASSET_REFERENCE_TABLES` 一处;文件隔离加数据库失败回滚(`_restore_staged_media`)是这个模块最不能出错的部分:删错的文件找不回来。@@ -13,27 +11,22 @@ from __future__ import annotationsimport hashlibimport jsonimport os-import reimport shutilimport tempfileimport timeimport uuid-from functools import cachefrom pathlib import Path, PureWindowsPathfrom typing import Sequence-from .catalog_rules import (- duration_clusters, is_jav_code, normalise_code_key, promo_free_key,- release_code_from_filename,-)+from . import junk+from .catalog_rules import duration_clusters, is_jav_code, normalise_code_keyfrom .config import GENERATED_DIR, LOCATION_ROOT_DECLARATIONSfrom .field_owners import USER_MANUAL, write_owned_fieldsfrom .media_sources import SCANNABLE_LOCATIONSfrom .personal_records import ASSET_REFERENCE_TABLES, VANISHEDfrom .platform import is_unmapped, root_online, translate_ledger_path, within_rootfrom .regions import normalize_region-from .settings_file import PROJECT_ROOTfrom .task_runs import TaskRunHandlefrom .web_activity import DEFAULT_PROFILE_IDfrom .web_catalog import attach_card_performers@@ -41,6 +34,28 @@ from .media_sources import cost_forfrom .web_resource_sync import clean_resource_orphansfrom .web_state import WebContract+from .junk import (+ AD_DIRPACK as AD_DIRPACK,+ AD_DOMAIN as AD_DOMAIN,+ AUTO_TRASH_SCORE as AUTO_TRASH_SCORE,+ BUNDLE_DIR_ASSETS as BUNDLE_DIR_ASSETS,+ CONTENT_BYTES as CONTENT_BYTES,+ INTERNET_SHORTCUT_SUFFIXES as INTERNET_SHORTCUT_SUFFIXES,+ JUNK_KINDS as JUNK_KINDS,+ JUNK_REVIEW_SCORE as JUNK_REVIEW_SCORE,+ JUNK_VIDEO_MAX_BYTES as JUNK_VIDEO_MAX_BYTES,+ MEDIA_SIDECAR_SUFFIXES as MEDIA_SIDECAR_SUFFIXES,+ PART_MARK as PART_MARK,+ PROMO_CLUSTER_FILES as PROMO_CLUSTER_FILES,+ PROMO_DOMAIN as PROMO_DOMAIN,+ PROMO_FILLER as PROMO_FILLER,+ PROMO_PHRASE as PROMO_PHRASE,+ REAL_CODE as REAL_CODE,+ has_sibling_original as has_sibling_original,+ promo_residue as promo_residue,+)++#: 批量操作在活动页上的名字。表里存 `operation`,人看的是这一列。BATCH_LABELS = {"like": "批量标记喜欢", "seen": "批量标记看过", "later": "批量加入稍后看",@@ -50,228 +65,6 @@ BATCH_LABELS = {}-# 只认联系方式与站点形态的推广套话。「微信」「成人游戏」这类词单独出现不算:-# 实测正片标题里就有(「还要微信跟老公汇报战果」是剧情,不是联系方式)。-PROMO_PHRASE = re.compile(- r"(扫码|扫一扫|掃一掃|扫描|掃描|QR ?CODE|二维码|二維碼|加微信|加微|威信\d|微信号|"- r"微信\s*[::]|免费看|免费玩|福利群|最新地址|"- r"永久(?:域名|地址|发布)|地址发布器|防屏蔽|资源获取|点击(?:观看|下载|进入)|下载APP|下载|签到|代币|领取|"- r"强力推荐|国产大片|在线视频|線上影片每天火熱更新|手机看片|手機影城|"- r"大饱眼福|房间火爆|澳门|赌场|博彩|棋牌|加我|"- r"GAMES?\d*|APP)", re.I)-# 只降低残留、自己不构成命中的套话。-#-# 手游插页整批绕过了上面的判据:`爱姬远征-免费18禁手游-扫码安装.jpg` 命中「扫码」,-# 可剥完还剩「爱姬远征免费禁手游安装」12 个字,落在 6~14 的中间档只拿 30 分,够不上-# 40 的门槛;`三国志侵略版-免费18禁手游-扫码安装.png` 多两字残留就到 14,连 30 分都没有。-# 残留越多分越低,而这些残留里一个内容字都没有——广告名写得越长反而越安全。-# 游戏名(爱姬远征、天下布魔)永远进不了词表,能剥的只有它旁边那圈固定套话。-# 这些词单独出现说明不了什么(真片名里也有「免费」),所以只参与残留计算,不作命中依据。-PROMO_FILLER = re.compile(- r"(免费\d*禁|免费|免費|手游|手遊|成人遊?戲|成人游戏|下载安装|安装|安裝|访问|訪問)",- re.I)-# 结尾不能用 \b:`uuc82.com_2` 里 `m` 和 `_` 都是词字符,构不成边界,域名会漏掉。-PROMO_DOMAIN = re.compile(- r"(?:https?://)?(?:www\.)?[\w-]{2,}\.(?:com|net|me|la|xyz|cc|tv|top|vip|club|"- r"info|org|pw|cn|app|site|online|shop)(?![a-z0-9])", re.I)-# 真番号带厂牌前缀和连字号(ABW-153、259LUXU-1141)。RAIKUN325 这类没有连字号的-# 是被误填进 code 的创作者账号名,不能拿来做「同番号有完整版」的比较,-# 判据与 `is_jav_code` 同源:分隔符正是番号与账号名的唯一线索。-REAL_CODE = re.compile(r"^(?:\d{2,3})?[A-Za-z]{2,8}-\d{2,5}$|^FC2", re.I)-PART_MARK = re.compile(r"(CD\d|part\d|分卷|-\d{1,2}$|\(\d+\)$)", re.I)-# 推广站目录的两种形态,与 `scripts/find_ads.py` 的判据 D/E 同源:-# 创作者位是旧导入器的目录名投影,`bbsxv.xyz-DOCP-324` 这类广告包会直接落在那里;-# 裸域名目录(98T.la@账号、huachishe.com@系列)是转载水印,不是广告,不能进判据。-AD_DOMAIN = re.compile(- r"\b[0-9a-z][-0-9a-z]{1,20}\.(?:cc|xyz|xxx|com|net|la|me|top|vip|club|app|cn|pw|tv|gg)\b", re.I)-AD_DIRPACK = re.compile(- r"[0-9a-z][-0-9a-z]{1,20}\.[a-z]{2,10}[ \-_]+\[?[A-Za-z]{2,6}-?\d{2,5}", re.I)-#: 同一目录里有这么多资产,它就是成套下载的资源包,不是塞进别人目录的插页。-#:-#: `jitumi.pw(77).gif` 与 `【91狼友之家91home.cc】19.jpg` 剥完只剩域名,是最硬的-#: 「整名推广语」信号,可它们分别躺在 `森萝财团 X-019 肉丝换白丝 [103P1V-1.39GB]`-#: (同目录 107 项)和 `赠送-稀缺整合【91home.cc】\海角\新建文件夹`(同目录 24 项)里:-#: 域名在这里是打包渠道给整包起的名,几十上百张成套的图本身就是要留的资源。-#: 广告插页反过来,是一两张挤在别人的番号目录里——WAAA-415 那个目录只有 13 项。-BUNDLE_DIR_ASSETS = 20-#: 同一目录里有这么多同类推广名,它们就是成群塞进来的插页。-#:-#: 与 `BUNDLE_DIR_ASSETS` 正好对称:资源是成套的正片与图集,插页是成群的同一段套话。-#: `三国志侵略版-免费18禁手游-扫码安装.png` 剥完仍剩 6 个字的游戏名,单看名字只够-#: 中间档;但同目录另外 10 个文件挂着一模一样的推广尾巴,谁都不是正片的一部分。-PROMO_CLUSTER_FILES = 3-#: 缩略图与封面相对正片的固定尾巴(`MIAD573_02.wmv` ↔ `MIAD573_02_s.jpg`,-#: `LXVS006-BD.iso` ↔ `lxvs006pl.jpg`)。`pl`/`ps` 是 JAV 大小海报的通用写法。-#: 不剥数字:剥了 `MIAD573_02_s` 会退到 `MIAD573`,那已经不是同一条正片。-SHOT_TAIL = re.compile(r"[-_. ]?(?:s|t|pl|ps|thumb|poster|cover|fanart|big|small)$", re.I)-#: 正片自己带的画质与载体尾巴,只在配对时剥,用来让海报对上原盘。-DISC_TAIL = re.compile(r"[-_. ]?(?:bd|bdmv|bdiso|bdrip|dvd|dvdiso|iso|uhd|fhd|hd)$", re.I)-#: 超过这个体积的文件自己就是内容,不必再看目录名。-#:-#: `javme.me_LXVS-006-BD\LXVS006-BD.iso` 是 19.8 GB 的蓝光原盘,却因为住在-#: 「域名+番号」的目录里被判 45 分。广告不会有大文件:整包插页加起来都不到 10 MB,-#: 一个上 GB 的文件是资源站给自己的资源改了目录名,不是资源站塞进来的推广。-CONTENT_BYTES = 1024 ** 3-INTERNET_SHORTCUT_SUFFIXES = frozenset({".url"})-#: 保存下来的网页。`.mhtml` 是 Edge 的整页存档,媒体目录里的这份没有内容价值;-#: 它和 `.url` 一样是「导航页」而不是媒体,不给体积兜底留机会。-PAGE_ARCHIVE_SUFFIXES = frozenset({".mhtml", ".mht", ".htm", ".html"})-#: 推广品牌词。它们单独出现在视频名里不构成证据——`麻豆传媒`、`外围`、`直播` 都可能是-#: 真片名的一部分——只有叠加推广形态、装饰符或推广目录上下文时才计分。-#: 名单来自 2026-09 用户实例 `B:\xxr\0208 (23)` 里自曝身份的推广站与推广 APP。-AD_BRAND = re.compile(- r"(91(?:短视频|视频|国产|AV|約炮|约炮)|成人抖音|快手直播|杏吧|草榴|色中色|成人头条|"- r"泡芙短视频|含羞草|她趣|台湾UU|全国外围|外围楼凤|楼凤|麻豆传媒映画|蜜桃影像|"- r"成人游戏|169BBS|SEX169)", re.I)-#: 与品牌词同现才算推广名的形态词。`视频`、`直播` 这类过常见的词不进这张表:-#: `麻豆传媒映画APP-限时免费体验` 是推广包,`麻豆传媒 某作品` 是资源。-AD_BRAND_FORM = re.compile(- r"(APP|一键|导航|聊天室|免费体验|限时免费|扫码|资源获取|网址|发布页)", re.I)-#: 品牌推广图的体积上限。`蜜桃影像传媒.png` 18 KB、`東方秋白…169BBS…` 75 KB,-#: 而任何一张内容图都在 MB 量级;品牌名 + 这种体积才是广告卡,不是作品套图。-BRAND_TINY_BYTES = 300 * 1024-#: 装饰符名(`❤91短视频❤.jpg`):两侧都挂或挂多个爱心的小图,名字里没有实质描述。-DECOR_MARKS = "❤♥❥♡💕💗💖💘💝"-#: 装饰图体积上限。`❤草榴视频❤.jpg` 705 KB 是广告卡;同名配套封面(2 MB 级)不算,-#: 它走 `has_sibling_original` 的正片配对豁免。-DECORATED_MAX_BYTES = 1536 * 1024-#: 目录名自曝是推广包的形态:`-APP`、`一键约炮`、`論壇文宣`、聊天室等。-#: 只匹配目录分量,不看文件名;命中后目录里不构成内容的文件都进复核。-AD_DIR_FORM = re.compile(- r"(論壇文宣|论坛文宣|文宣|宣傳|一键约炮|全国外围|外围楼凤|楼凤|聊天室|"- r"免费体验|限时免费)", re.I)-#: `APP` 按词匹配:`Happy` 里也藏着 `app`,裸子串会把正常目录卷进来;-#: `.app` 是域名后缀,`bkm9.app` 那类推广目录靠目录里其他证据,不靠这一条。-AD_DIR_APP = re.compile(r"(?:^|[^A-Za-z.])APP(?:[^A-Za-z.]|$)", re.I)-#: 品牌名只在目录里还带媒体平台词时才算推广目录;`草榴视频` 是推广 APP,-#: `麻豆传媒` 单独出现可能是这个厂牌的资源目录。-AD_DIR_BRAND_MEDIA = re.compile(r"(?:视频|視頻|影视|影視|传媒|傳媒|直播|短视频|短視頻)", re.I)-JUNK_KINDS = frozenset({"video", "image", "audio", "archive", "url", "other"})-#: 达到这个体积的视频不再当广告残留。这个值与原有「小于 120 MB」证据同源;-#: 较大的同番号短版本属于重复清理问题,由 ``q_duplicates`` 承接。-JUNK_VIDEO_MAX_BYTES = 120 * 1024**2-MEDIA_SIDECAR_SUFFIXES = frozenset({".nfo", ".srt", ".ass", ".ssa", ".vtt", ".sub", ".idx"})-INSTALLER_SUFFIXES = frozenset({".apk", ".exe", ".msi"})-#: 按固定命名识别的推广附件。名字和体积只能说明「像」,内容要人看过才算数。-PROMO_NAMES_FILE = PROJECT_ROOT / "resources" / "junk" / "promo_attachment_names.json"-#: 进入垃圾复核队列的分数线。-JUNK_REVIEW_SCORE = 40-#: `scripts/trash_junk.py --apply` 默认的自动移入回收站分数线。只能复核的证据-#: (固定命名、未探测时长视频的目录与体积证据)单独凑不到这一档。-AUTO_TRASH_SCORE = 60-#: 极短附件的结构证据只进入复核:目录中须有同名长正片,且附件相对它足够短、小。-SHORT_ATTACHMENT_SECONDS = 30-SHORT_ATTACHMENT_BYTES = 32 * 1024**2-SHORT_ATTACHMENT_CLUSTER = 3-#: 「来源站名 + 序号」也用于正常短片;站点水印不能成为自动隔离依据。-NUMBERED_SOURCE_VIDEO = re.compile(r"^(?:\[[^\]]+\]|【[^】]+】)[\s._-]*\d+(?:[\s._-]*\(\d+\))?$")-PAGE_COMPONENT_SUFFIXES = frozenset({".js", ".css", ".aspx"})-PAGE_CHROME_IMAGE = re.compile(r"^(?:banner(?:_[bs])?|1000x90(?:yunding)?|count\d*|logo|lan\d*|you|zuo|zuoxia)$", re.I)-PAGE_PROMO_CLAIM = re.compile(r"(?:记住|記住).{0,12}(?:网址|網址)|(?:网址|網址).{0,8}(?:发布|發布)")---def promo_residue(name: str) -> int:- """剥掉域名和推广套话后,还剩多少实质描述字符。-- 这是区分「广告」与「正片被打了站点水印」的关键:- `点击观看 房间火爆` 剥完什么都不剩;- `236953.xyz 推特新晋4年绿帽美腿淫妻网黄「一个ren」…` 剥完仍有大段内容描述。-- 剥掉的词替换成空串而不是空格:空格自己也在计数字符集里,换成空格等于每剥一个- 词就补回一个残留。`爱姬远征-免费18禁手游-扫码安装` 剥掉四段套话后只剩四个字,- 却因为补进来的四个空格数成 8,正好卡在门槛外——剥得越干净残留越高。- """- text = PROMO_DOMAIN.sub("", name or "")- text = PROMO_PHRASE.sub("", text)- text = PROMO_FILLER.sub("", text)- # 只数中日韩文字与字母,忽略编号、扩展名和标点。- return len(re.findall(r"[一-鿿-ヿ가- A-Za-z]", text))---def _folder_index(connection) -> tuple[dict[str, int], dict[str, set[str]]]:- """每个目录里有多少资产,以及其中正片的文件名主体。-- 两条判据都要看候选的邻居而不只是它自己:成套资源看目录规模,封面与截图看- 同目录有没有同名正片。`rsplit` 而不是 `PureWindowsPath`:ledger 路径统统是- Windows 形态,这里要跑全表七万多行,只取目录名不值得为每行造一个路径对象。-- 正片不限于 `medium='video'`:蓝光原盘在账本里是 `other`,它同样是海报要配的那份- 内容。用体积兜住这一类,不去猜扩展名——`.iso`、`.mds`、`BDMV` 之外还有什么,- 库里下一份原盘才知道。- """- counts: dict[str, int] = {}- videos: dict[str, set[str]] = {}- for path, medium, size in connection.execute(- "SELECT path,medium,size FROM asset WHERE path IS NOT NULL AND path<>''"):- folder, _, filename = str(path).rpartition("\\")- counts[folder] = counts.get(folder, 0) + 1- if medium == "video" or (size or 0) >= CONTENT_BYTES:- stem = filename.rsplit(".", 1)[0] if "." in filename else filename- key = stem.casefold()- videos.setdefault(folder, set()).add(key)- trimmed = DISC_TAIL.sub("", key)- if trimmed:- videos[folder].add(trimmed)- return counts, videos---def has_sibling_original(stem: str, sibling_videos: set[str]) -> bool:- """True 表示同目录里有一条与它同名的正片,它是那条正片的封面或截图。"""- key = str(stem or "").casefold()- if not key:- return False- return key in sibling_videos or SHOT_TAIL.sub("", key) in sibling_videos---def is_decorated_name(stem: str) -> bool:- """名字挂着两个以上装饰符,没有任何实质描述。-- 样本是 `B:\\xxr\\0208 (23)` 里那批 7~20 KB 的 `❤91短视频❤.png`。- 只挂一个爱心不算:Telegram 导出与正常标题里单个爱心常见,两个以上才是装饰性命名。- """- return sum(str(stem or "").count(mark) for mark in DECOR_MARKS) >= 2---def _promo_directory(parts) -> bool:- """路径里有任意一层目录自曝是推广包。-- 三种形态:`APP` 词形(`_含羞草APP`)、促销目录名(`一键约炮`、`論壇文宣`)、- 品牌名叠加媒体词(`_草榴视频`)。只看目录分量,文件名自己是不是推广由调用方判。- """- for part in parts:- text = str(part)- if AD_DIR_APP.search(text) or AD_DIR_FORM.search(text):- return True- if AD_BRAND.search(text) and AD_DIR_BRAND_MEDIA.search(text):- return True- return False---def _junk_rows(connection):- """读取能进入垃圾复核的物理文件;字幕、NFO 与大视频在最外层排除。"""- marks = ",".join("?" for _ in SCANNABLE_LOCATIONS)- rows = connection.execute(- "SELECT id,location,name,medium,creator,code,size,duration,width,height,snapshot_path,"- "feedback,disposal,play_count,leave_ratio,o_count,studio,ctx_orient,path "- f"FROM asset WHERE location IN ({marks}) AND disposal IS NULL "- "AND (COALESCE(medium,'other')<>'video' OR (size < ? "- "AND (duration IS NULL OR duration <= 1200)))",- (*SCANNABLE_LOCATIONS, JUNK_VIDEO_MAX_BYTES)).fetchall()- # NFO 是媒体资料边车,不是等待清理的物理内容;即使名字或所在目录带推广词,- # 也应由资料读取报告解析问题,不能进入会把文件移入回收站的垃圾队列。- return [row for row in rows if _junk_file_eligible(row)]---def _junk_file_eligible(row) -> bool:- """资料边车和文件自身带番号的分卷不属于垃圾附件。"""- item = PureWindowsPath(row["name"] or row["path"] or "")- if item.suffix.casefold() in MEDIA_SIDECAR_SUFFIXES:- return False- identity = PART_MARK.sub("", PROMO_DOMAIN.sub("", item.stem)).strip("@._- ")- return not (row["medium"] == "video" and PART_MARK.search(item.stem)- and release_code_from_filename(identity))--def q_ads(contract: WebContract, limit=200, offset=0, kind="", status="pending"):"""疑似垃圾复核队列 —— **不自动删**,只排队让人看证据确认。@@ -332,300 +125,14 @@ def q_ads(contract: WebContract, limit=200, offset=0, kind="", status="pending")}-def _promo_neighbour_counts(rows) -> dict[str, int]:- """统计同目录中缺少内容描述的推广名。"""- counts: dict[str, int] = {}- for row in rows:- name = row["name"] or PureWindowsPath(row["path"] or "").name- stem = PureWindowsPath(name).stem- if not (PROMO_PHRASE.search(stem) or PROMO_DOMAIN.search(stem)):- continue- if promo_residue(stem) >= 14:- continue- folder = str(row["path"] or name).rpartition("\\")[0]- counts[folder] = counts.get(folder, 0) + 1- return counts---@cache-def promo_name_rules() -> tuple[dict, ...]:- """读取 `PROMO_NAMES_FILE` 的推广附件命名规则;文件缺失或格式不对直接报错。"""- data = json.loads(PROMO_NAMES_FILE.read_text(encoding="utf-8"))- rules = []- for entry in data["rules"]:- names = frozenset(re.sub(r"\s", "", name).casefold() for name in entry.get("names", ()))- patterns = tuple(re.compile(pattern, re.I) for pattern in entry.get("patterns", ()))- if not entry["reason"] or not (names or patterns) or int(entry["max_bytes"]) <= 0:- raise ValueError(f"推广附件命名规则不完整:{entry}")- rules.append({- "reason": str(entry["reason"]),- "suffixes": frozenset(str(suffix).casefold() for suffix in entry["suffixes"]),- "max_bytes": int(entry["max_bytes"]),- "parent_is_name": bool(entry.get("parent_is_name")),- "names": names,- "patterns": patterns,- })- return tuple(rules)---def _promo_name_reason(suffix: str, path: str, size: int) -> str:- """命中词库里推广附件固定命名的理由;只够进复核队列,见 `AUTO_TRASH_SCORE`。"""- item = PureWindowsPath(path)- stem = item.stem- for rule in promo_name_rules():- if suffix not in rule["suffixes"] or not 0 < size <= rule["max_bytes"]:- continue- if rule["parent_is_name"] and item.parent.name != stem:- continue- if (re.sub(r"\s", "", stem).casefold() in rule["names"]- or any(pattern.fullmatch(stem) for pattern in rule["patterns"])):- return rule["reason"]- return ""---def _attachment_junk_reason(suffix: str, path: str, size: int) -> str:- """区分安装附件与网页存档组件。"""- if suffix in INSTALLER_SUFFIXES and size < 64 * 1024**2:- return "媒体目录中的安装附件"- if suffix in PAGE_COMPONENT_SUFFIXES and any(- part.casefold().endswith("_files") for part in PureWindowsPath(path).parent.parts):- return "网页存档的脚本或样式附件"- parent = PureWindowsPath(path).parent.name- if (parent.casefold().endswith('_files') and PAGE_PROMO_CLAIM.search(parent)- and AD_DOMAIN.search(parent[:-len('_files')]) and suffix in {'.gif', '.jpg', '.png'}- and size < 2 * 1024**2- and (PAGE_CHROME_IMAGE.fullmatch(PureWindowsPath(path).stem)- or (PureWindowsPath(path).name.casefold() == '1.gif' and size <= 4096))):- return "推广网页存档的横幅、标志或导航图片"- return ""---def _short_attachment(row) -> bool:- """有测量值、没有自身作品身份或分卷标记的极短视频。"""- stem = PureWindowsPath(row["name"] or row["path"] or "").stem- owner = row["creator"] or ""- return (row["medium"] == "video"- and 0 < (row["duration"] or 0) <= SHORT_ATTACHMENT_SECONDS- and 0 < (row["size"] or 0) <= SHORT_ATTACHMENT_BYTES- and not (owner and not AD_DOMAIN.search(owner))- and not PART_MARK.search(stem)- and not release_code_from_filename(stem))---def _release_main_videos(connection) -> dict[str, list[tuple[float, int]]]:- """按活动物理文件建立发行目录的长正片索引;普通混合目录不算发行目录。"""- mains: dict[str, list[tuple[float, int]]] = {}- marks = ",".join("?" for _ in SCANNABLE_LOCATIONS)- for path, duration, size in connection.execute(- "SELECT path,duration,size FROM asset WHERE disposal IS NULL AND medium='video' "- f"AND location IN ({marks}) AND duration>=1200 AND size>=? AND path IS NOT NULL",- (*SCANNABLE_LOCATIONS, JUNK_VIDEO_MAX_BYTES)):- item = PureWindowsPath(path)- key = promo_free_key(item.stem)- if key and key == promo_free_key(item.parent.name):- mains.setdefault(str(item.parent).casefold(), []).append((duration, size))- return mains---def _short_attachment_reasons(connection, rows) -> dict[int, str]:- """从正片与附件的量级关系、同目录成群和跨发行目录扩散生成复核证据。-- 不读媒体,也不按题材、游戏名或推广词命中。相同测量值只是扩散线索,不能证明内容相同。- """- mains = _release_main_videos(connection)- folders: dict[str, set[str]] = {}- signatures: dict[tuple, set[str]] = {}- candidates = []- for row in rows:- if not _short_attachment(row):- continue- path = PureWindowsPath(row["path"] or "")- folder = str(path.parent).casefold()- if not any(duration >= row["duration"] * 40 and size >= row["size"] * 10- for duration, size in mains.get(folder, ())):- continue- signature = (row["size"], round(row["duration"], 3), row["width"], row["height"])- folders.setdefault(folder, set()).add(str(path).casefold())- if (row["width"] or 0) > 0 and (row["height"] or 0) > 0:- signatures.setdefault(signature, set()).add(folder)- candidates.append((row["id"], folder, signature))- reasons = {}- for asset_id, folder, signature in candidates:- count = len(folders[folder])- spread = len(signatures.get(signature, ()))- if count >= SHORT_ATTACHMENT_CLUSTER:- reasons[asset_id] = f"同目录长正片旁有 {count} 条极短小附件,须核验内容"- elif spread >= SHORT_ATTACHMENT_CLUSTER:- reasons[asset_id] = f"同体积、时长与分辨率的极短附件散布在 {spread} 个发行目录,须核验内容"- return reasons---def _video_evidence(row, stem: str, longer: dict, promo_dir: bool,- directory_points: int, attachment_reason: str = "") -> tuple[int, int, list[str]]:- """视频自己的证据:同番号完整版、时长、体积与推广目录。-- 返回(分数,其中只够复核的分数,理由)。时长没探测到的视频可能就是正片:- 推广创作者位或「域名+番号」目录(`directory_points`)、推广目录与体积只说明它住在哪、- 有多大,说明不了它是广告,这些分数只够进复核队列。- """- score, why = 0, []- code = (row["code"] or "").strip()- duration = row["duration"]- longest = longer.get(code)- if longest and REAL_CODE.match(code) and duration is not None and 0 < duration < longest * 0.2 \- and not PART_MARK.search(stem):- # 分卷已排除,真番号下不到两成时长基本就是片段/预告,单独即可入队复核。- # 用户标记的 `反抗不如享受.mp4`(ABW-220,244 秒)正好卡在旧的 35 分门外。- score += 40; why.append(f"同番号有 {longest/60:.0f} 分完整版")- if duration is not None and 0 < duration < 240:- score += 15; why.append("不足 4 分钟")- context = directory_points- if (row["size"] or 0) < JUNK_VIDEO_MAX_BYTES:- score += 10; context += 10; why.append("小于 120 MB")- if promo_dir:- # 30 分单独不构成删片理由:正片也可能躺在别人起错名的目录里,- # 要再叠一条时长或体积证据才到门槛。- score += 30; context += 30; why.append("住在推广目录")- probed = duration is not None and duration > 0- review_only = 0 if probed else context- if NUMBERED_SOURCE_VIDEO.fullmatch(stem) and PROMO_DOMAIN.search(stem):- review_only += AUTO_TRASH_SCORE- why.append("编号视频可能只有来源水印,须核验内容")- if attachment_reason:- score += JUNK_REVIEW_SCORE; review_only += JUNK_REVIEW_SCORE; why.append(attachment_reason)- return score, review_only, why--def _scored_junk(contract: WebContract) -> tuple[list[dict], frozenset[int]]:- """全部候选按嫌疑分从高到低,连同用户已确认不是垃圾的 id。-- 与请求的类型、状态和分页无关:整份算一次(真实账本上两三秒),`q_ads` 按参数切。- """- with contract.read_connection() as c:- rows = _junk_rows(c)- # 同番号是否存在明显更长的版本;只在 code 是真番号时才有意义。- longer = {r[0]: r[1] for r in c.execute(- "SELECT code, max(duration) FROM asset WHERE medium='video' AND code IS NOT NULL "- "AND code<>'' AND duration IS NOT NULL GROUP BY code")}- dismissed_keys = [str(row[0]) for row in c.execute(- "SELECT item_key FROM review_decision "- "WHERE category='junk_file' AND status='rejected'"- )]- dismissed_ids = {int(key) for key in dismissed_keys if key.isdigit()}- folder_assets, folder_videos = _folder_index(c)- short_attachments = _short_attachment_reasons(c, rows)- # 每个目录里挂着推广名的候选有多少个;插页判据要看它有没有同伙。- promo_neighbours = _promo_neighbour_counts(rows)- out = []- for r in rows:- d = dict(r)- s, why = 0, []- name = d.get("name") or PureWindowsPath(d.get("path") or "").name- name_path = PureWindowsPath(name)- nm = name_path.stem- suffix = name_path.suffix.casefold()- d["junk_kind"] = (- "url" if suffix in INTERNET_SHORTCUT_SUFFIXES- else (d.get("medium") if d.get("medium") in JUNK_KINDS else "other")- )- residue = promo_residue(nm)- promo = bool(PROMO_PHRASE.search(nm) or PROMO_DOMAIN.search(nm))- page_archive = suffix in PAGE_ARCHIVE_SUFFIXES- if suffix in INTERNET_SHORTCUT_SUFFIXES:- s += 60; why.append("网址快捷方式")- elif page_archive:- s += 60; why.append("网页存档")- attachment_reason = _attachment_junk_reason(suffix, d.get("path") or name, d.get("size") or 0)- if attachment_reason:- s += 60; why.append(attachment_reason)- # 只够进复核队列的分数:固定命名,和未探测时长视频的目录与体积证据。- review_only = 0- name_reason = _promo_name_reason(suffix, d.get("path") or name, d.get("size") or 0)- if name_reason:- s += JUNK_REVIEW_SCORE; review_only += JUNK_REVIEW_SCORE; why.append(name_reason)- # 目录维度的证据:广告包的文件名往往干净(`极道世界.mp4`),唯一线索在旧导入器- # 从目录名投影出来的创作者位或路径里。creator 位本身是推广站域名时,它就不再是- # 「有归属所以是正片」的证据,下面两处对 creator 的信任都必须先排除这种情况。- owner = d.get("creator") or ""- owner_is_promo = bool(AD_DOMAIN.search(owner))- real_owner = bool(owner) and not owner_is_promo- # ledger 路径在两个平台都是 Windows 形态。`rpartition` 没有平台依赖,- # 键也与 `_folder_index` 完全同源;`os.path.dirname` 在 macOS 会把整条- # 反斜杠路径当成文件名。- folder = str(d.get("path") or name).rpartition("\\")[0]- # 成套下载的资源包里,域名是打包渠道给整包起的名,不是插页的自我暴露。- bundled = folder_assets.get(folder, 0) >= BUNDLE_DIR_ASSETS- # 目录名带推广站域名只说明「从哪个站下的」,说明不了这个文件是广告。- # 文件自己是真番号、自己就有内容级的体积、或者是同目录正片的封面/截图时,- # 目录证据就不成立。- #- # 只放非视频:广告包里的视频本来就叫 `极道世界.mp4`,它的 code 是旧导入器从- # `bbsxv.xyz-DOCP-324` 目录名投影出来的真番号形状,而它自己就在同目录的正片- # 名单里——两条豁免对视频都会自动成立,`AD_DIRPACK` 这条判据就没了。- self_evident = d.get("medium") != "video" and (- is_jav_code(d.get("code"))- or (d.get("size") or 0) >= CONTENT_BYTES- or has_sibling_original(nm, folder_videos.get(folder, frozenset()))- )- decorated = (d.get("medium") != "video" and is_decorated_name(nm)- and (d.get("size") or 0) <= DECORATED_MAX_BYTES)- brand_hit = bool(AD_BRAND.search(nm)) and bool(- AD_BRAND_FORM.search(nm)- or (d.get("medium") == "image"- and (d.get("size") or 0) <= BRAND_TINY_BYTES))- promo_dir = _promo_directory(PureWindowsPath(d.get("path") or name).parent.parts)- if promo and residue < 6 and not bundled:- # 名字剥完只剩广告本身,这是最硬的信号。- s += 60; why.append("整个名字都是推广语")- elif promo and residue < 14 and not real_owner and not bundled:- s += 30; why.append("推广语占了名字主体")- neighbours = promo_neighbours.get(folder, 0)- if neighbours >= PROMO_CLUSTER_FILES:- s += 30- why.append(f"同目录另有 {neighbours - 1} 个同类推广名")- directory_points = 0- if owner_is_promo:- directory_points = 50; why.append("创作者位是推广站域名")- elif AD_DIRPACK.search(folder) and not self_evident:- directory_points = 45; why.append("目录是「域名+番号」的推广打包")- s += directory_points- # 非视频的推广形态:装饰符小图、品牌推广名、住在推广目录。`self_evident`- # (真番号、内容级体积、正片配套图)与网页存档、网址快捷方式已单独计分,- # 这里不重复叠加。- if (d.get("medium") != "video" and not self_evident and not page_archive- and suffix not in INTERNET_SHORTCUT_SUFFIXES):- if decorated:- s += 60; why.append("装饰符小图")- elif brand_hit:- s += 60; why.append("推广品牌名")- elif promo_dir:- s += 50; why.append("住在推广目录")- if d.get("medium") == "video":- points, weak, reasons = _video_evidence(- d, nm, longer, promo_dir, directory_points, short_attachments.get(d["id"], ""))- s += points; review_only += weak; why.extend(reasons)- # 有真实创作者归属、且名字剥完仍有实质描述的,是被打了水印的正片,不是广告。- if real_owner and residue >= 14 and not (attachment_reason or name_reason):- s -= 45- if review_only and s - review_only < AUTO_TRASH_SCORE:- s = min(s, AUTO_TRASH_SCORE - 1)- if s >= JUNK_REVIEW_SCORE:- d["score"] = s; d["why"] = " · ".join(why)- d["cost"] = cost_for(d['location'], d.get('path'))- d["has_thumb"] = contract.has_snapshot(d["snapshot_path"])- d.pop("snapshot_path", None)- d.pop("path", None)- out.append(d)- out.sort(key=lambda x: (-x["score"], -(x["size"] or 0)))- return out, frozenset(dismissed_ids)---def automatic_junk_candidates(contract: WebContract) -> list[dict]:- """返回达到自动隔离线且未被用户驳回的候选,供组合根接入下载后处理。"""- scored, dismissed = contract.cached_until_changed(- "junk-scored", lambda: _scored_junk(contract))- return [row for row in scored- if int(row["id"]) not in dismissed and int(row["score"]) >= AUTO_TRASH_SCORE]+ """领域评分加上页面需要的成本与缩略图状态,路径留在服务内部。"""+ out, dismissed = junk.scored_candidates(contract)+ for item in out:+ item["cost"] = cost_for(item['location'], item.get('path'))+ item["has_thumb"] = contract.has_snapshot(item.pop("snapshot_path", None))+ item.pop("path", None)+ return out, dismisseddef _restore_staged_media(staged):diff --git a/src/peach/web_organize.py b/src/peach/web_organize.pyindex 7eacdf11b..b4b2c3ca1 100644--- a/src/peach/web_organize.py+++ b/src/peach/web_organize.py@@ -12,7 +12,7 @@ import sqlite3import timefrom pathlib import Path-from . import organize+from . import junk, organizefrom .migrations import sqlite_backupfrom .media_sources import SCANNABLE_LOCATIONSfrom .organize_templates import PLACEHOLDERS, TemplateError, example_path, validate_template@@ -93,15 +93,20 @@ def _run_write(contract, job_id: str, work) -> dict:connection = sqlite3.connect(str(contract.db_path), timeout=30)connection.row_factory = sqlite3.Rowtry:+ before, _ = verify_after_write(connection)+ if before != 'ok':+ raise RuntimeError(f'写入前 ledger 校验失败:integrity={before}')+ baseline = {tuple(row) for row in connection.execute('PRAGMA foreign_key_check')}result = work(connection)- integrity, violations = verify_after_write(connection)+ integrity, _ = verify_after_write(connection)+ introduced = {tuple(row) for row in connection.execute('PRAGMA foreign_key_check')} - baselinefinally:connection.close()contract.cache_bust()- if integrity != "ok" or violations:+ if integrity != "ok" or introduced:raise RuntimeError(- f"写入后 ledger 校验失败:integrity={integrity} foreign_keys={violations}")- return {**result, "backup": str(backup)}+ f"写入后 ledger 校验失败:integrity={integrity} new_foreign_keys={len(introduced)}")+ return {**result, "backup": str(backup), 'existing_foreign_keys': len(baseline)}def w_organize_apply(contract, body) -> dict:@@ -116,6 +121,9 @@ def w_organize_apply(contract, body) -> dict:def work(job_id: str) -> None:batch_stamp = (job.snapshot() or {}).get('batch_stamp') or f"{time.strftime('%Y%m%d-%H%M%S')}-{job_id}"+ roots = [(location, path_scope)] if path_scope else [(location, root) for root in location_roots()[location]]+ checked = junk.isolate(contract.database, roots=roots)+ job.update(job_id, junk_isolated=checked['isolated'], junk_review=checked['review'])plan = (job.snapshot() or {}).get('plan')if plan is None:with contract.read_connection() as connection:diff --git a/src/peach/web_state.py b/src/peach/web_state.pyindex 0d6cb2552..aa650f0ed 100644--- a/src/peach/web_state.py+++ b/src/peach/web_state.py@@ -174,9 +174,7 @@ class WebContract:#: 任务中心。每个后台任务的开始、进度与结束都写这一张表,`/tasks` 只读它。#: 只读端由 `api` 关掉写入:那边的账本是复制来的,写一行就是一处合不回去的分叉。self.task_runs = TaskRunStore(self.database)- #: 云下载后处理复用 Web 层的垃圾评分与整理设置,但核心后继不反向依赖 Web。- #: `api` 作为组合根接入真实实现;裸契约用于单元测试时保持无副作用。- self.download_junk_candidates = lambda: []+ #: 整理设置由组合根注入;垃圾检查由扫描和后处理共用领域规则。self.download_organize_templates = lambda: {}#: 后继调度(ADR-0040)。导入 `avatar_followup` 就是它的登记动作:调度器本身#: 不认识任何一种后继,登记表在 `followups.REGISTRY` 里。diff --git a/tests/test_ad_judgement.py b/tests/test_ad_judgement.pyindex 30e5af826..a93900aca 100644--- a/tests/test_ad_judgement.py+++ b/tests/test_ad_judgement.py@@ -10,7 +10,7 @@ from contextlib import closingfrom pathlib import Pathfrom unittest import mock-from peach import web_batch+from peach import junk, web_batchfrom peach.web_contract import (BUNDLE_DIR_ASSETS,CONTENT_BYTES,@@ -174,6 +174,60 @@ class ResourceJunkQueueTests(unittest.TestCase):finally:connection.close()+ def test_automatic_isolation_requires_a_scope_and_observes_source_and_directory_boundaries(self):+ for aid, location, folder in ((1, 'local', 'media'), (2, 'local', 'media2'), (3, '115', 'media')):+ self.add(aid, location, rf'R:\{folder}\点击进入.url', 'other')+ with self.assertRaises(ValueError):+ junk.isolate(self.db_path)+ self.assertEqual(junk.isolate(self.db_path, asset_ids=[])['isolated'], 0)+ self.assertEqual(junk.isolate(self.db_path, roots=[])['isolated'], 0)+ checked = junk.isolate(self.db_path, roots=[('local', r'r:\MEDIA')])+ self.assertEqual(checked['ids'], [1])+ self.assertEqual(junk.isolate(self.db_path, roots=[('local', r'R:\media')])['isolated'], 0)+ with self.contract.read_connection() as connection:+ self.assertEqual([tuple(row) for row in connection.execute(+ 'SELECT id,disposal,disposal_reason FROM asset ORDER BY id')],+ [(1, 'trash', 'junk'), (2, None, None), (3, None, None)])++ def test_automatic_isolation_preserves_user_choices_records_subtitles_and_main_identity(self):+ for aid in range(1, 8):+ self.add(aid, 'local', rf'R:\media\{aid}\点击进入.url', 'other')+ self.add(8, 'local', r'R:\media\123.com-ABC-123.mp4', 'video', 2000, 5)+ with self.contract.database.write_transaction() as connection:+ connection.execute("UPDATE asset SET feedback='like' WHERE id=1")+ connection.execute('UPDATE asset SET play_count=1 WHERE id=2')+ connection.execute("INSERT INTO watch_queue(profile_id,asset_id,added_at) VALUES('local-default',3,'now')")+ connection.execute("INSERT INTO asset_subtitle(asset_id,location,path,name,format,pairing,first_seen,last_seen) "+ "VALUES(4,'local','R:\\media\\sub.srt','sub.srt','srt','exact','now','now')")+ w_batch(self.contract, {'ids': [5], 'operation': 'dismiss-junk'})+ checked = junk.isolate(self.contract.database, asset_ids=list(range(1, 9)), protected_ids=[6])+ self.assertEqual(checked['ids'], [7])+ w_batch(self.contract, {'ids': [7], 'operation': 'restore'})+ w_batch(self.contract, {'ids': [7], 'operation': 'dismiss-junk'})+ self.assertEqual(junk.isolate(self.db_path, asset_ids=[7])['isolated'], 0)++ def test_isolation_rechecks_evidence_after_scoring(self):+ self.add(1, 'local', r'R:\media\点击进入.url', 'other')+ original = junk.scored_candidates+ def score_then_change(*args, **options):+ result = original(*args, **options)+ with self.contract.database.write_transaction() as connection:+ connection.execute("UPDATE asset SET name='notes.txt' WHERE id=1")+ return result+ with mock.patch.object(junk, 'scored_candidates', side_effect=score_then_change):+ self.assertEqual(junk.isolate(self.db_path, asset_ids=[1])['isolated'], 0)++ def test_unmeasured_and_failed_videos_await_review(self):+ for aid, duration in enumerate((None, 0, -1), 1):+ self.add(aid, 'local', rf'R:\media\{aid}\点击进入.mp4', 'video', 1000, duration)+ result = junk.isolate(self.db_path, asset_ids=[1, 2, 3])+ self.assertEqual((result['isolated'], result['review']), (0, 3))++ def test_short_attachment_structure_stays_in_review_during_automatic_processing(self):+ self.release_with_shorts(r'B:\发行物\正片标题')+ checked = junk.isolate(self.db_path, roots=[('115', r'B:\发行物')])+ self.assertEqual((checked['isolated'], checked['review']), (0, 3))+def test_video_image_audio_archive_and_url_shortcut_are_all_judged(self):self.add(1, "115", r"B:\广告\tuu26.com.mp4", "video",10 * 1024**2, 60)@@ -218,7 +272,7 @@ class ResourceJunkQueueTests(unittest.TestCase):self.assertIn("游戏下载站推广视频", row["why"])self.assertGreaterEqual(row["score"], web_batch.JUNK_REVIEW_SCORE)self.assertLess(row["score"], web_batch.AUTO_TRASH_SCORE)- self.assertEqual(web_batch.automatic_junk_candidates(self.contract), [])+ self.assertEqual(junk.isolate(self.db_path, asset_ids=list(candidates))['isolated'], 0)def release_with_shorts(self, base, first_id=300, names=None, **short_values):self.add(first_id, "115", base + "\\" + base.rsplit("\\", 1)[-1] + ".mp4",@@ -236,7 +290,7 @@ class ResourceJunkQueueTests(unittest.TestCase):self.assertTrue(all("6 条极短小附件" in row["why"] for row in items))self.assertTrue(all(web_batch.JUNK_REVIEW_SCORE <= row["score"] < web_batch.AUTO_TRASH_SCOREfor row in items))- self.assertEqual(web_batch.automatic_junk_candidates(self.contract), [])+ self.assertEqual(junk.isolate(self.db_path, asset_ids=[row['id'] for row in items])['isolated'], 0)def test_a_pair_of_shorts_and_standalone_animation_loops_stay_out(self):self.release_with_shorts(r"B:\发行物\正片标题", names=("附件甲", "附件乙"))@@ -283,7 +337,7 @@ class ResourceJunkQueueTests(unittest.TestCase):items = q_ads(self.contract)["items"]self.assertEqual({row["id"] for row in items}, {601, 611, 621})self.assertTrue(all("3 个发行目录" in row["why"] for row in items))- self.assertEqual(web_batch.automatic_junk_candidates(self.contract), [])+ self.assertEqual(junk.isolate(self.db_path, asset_ids=[row['id'] for row in items])['isolated'], 0)def test_cross_release_spread_requires_full_matching_measurements(self):for index in range(3):@@ -308,7 +362,7 @@ class ResourceJunkQueueTests(unittest.TestCase):for asset_id, name in enumerate(names, 700):self.add(asset_id, "115", "B:\\发行物\\" + name, "video", 10 * 1024**2, 400)self.assertEqual(q_ads(self.contract)["items"], [])- self.assertEqual(web_batch.automatic_junk_candidates(self.contract), [])+ self.assertEqual(junk.isolate(self.db_path, asset_ids=range(700, 704))['isolated'], 0)def test_numbered_source_watermarks_only_enter_review(self):for asset_id, name in enumerate(("【来源站example.com】1", "[example.com]84 (2)"), 710):@@ -316,7 +370,7 @@ class ResourceJunkQueueTests(unittest.TestCase):items = q_ads(self.contract)["items"]self.assertEqual({row["id"] for row in items}, {710, 711})self.assertTrue(all(row["score"] < web_batch.AUTO_TRASH_SCORE for row in items))- self.assertEqual(web_batch.automatic_junk_candidates(self.contract), [])+ self.assertEqual(junk.isolate(self.db_path, asset_ids=[row['id'] for row in items])['isolated'], 0)def test_non_video_trash_items_remain_visible_and_recoverable(self):self.add(10, "115", r"B:\番号\广告图.jpg", "image", disposal="trash")@@ -515,8 +569,8 @@ class ResourceJunkQueueTests(unittest.TestCase):items = q_ads(self.contract, limit=200)['items']self.assertEqual([item['id'] for item in items], [211])self.assertIn('推广安装包', items[0]['why'])- self.assertEqual(web_batch._promo_name_reason('.rar', path, 128 * 1024), '')- self.assertEqual(web_batch._promo_name_reason('.rar', r'B:\MVP\n1025.rar', 4111), '')+ self.assertEqual(junk._promo_name_reason('.rar', path, 128 * 1024), '')+ self.assertEqual(junk._promo_name_reason('.rar', r'B:\MVP\n1025.rar', 4111), '')def test_promotional_page_navigation_images_with_inherited_release_code(self):base = r'B:\MVP\1pon-092415_001-fhd'@@ -531,10 +585,10 @@ class ResourceJunkQueueTests(unittest.TestCase):items = q_ads(self.contract, limit=200)['items']self.assertEqual([item['id'] for item in items], [201])self.assertIn('导航图片', items[0]['why'])- self.assertEqual(web_batch._attachment_junk_reason('.gif', page + r'\1.gif', 974),+ self.assertEqual(junk._attachment_junk_reason('.gif', page + r'\1.gif', 974),'推广网页存档的横幅、标志或导航图片')- self.assertEqual(web_batch._attachment_junk_reason('.gif', page + r'\1.gif', 6000), '')- self.assertTrue(web_batch._attachment_junk_reason('.gif', page + r'\1000x90yunding.gif', 97843))+ self.assertEqual(junk._attachment_junk_reason('.gif', page + r'\1.gif', 6000), '')+ self.assertTrue(junk._attachment_junk_reason('.gif', page + r'\1000x90yunding.gif', 97843))if __name__ == "__main__":diff --git a/tests/test_complexity_ratchet.py b/tests/test_complexity_ratchet.pyindex 9fb4a4837..c8cf8bcf9 100644--- a/tests/test_complexity_ratchet.py+++ b/tests/test_complexity_ratchet.py@@ -36,7 +36,7 @@ LIMIT = 30#: 键是 `<仓库相对路径>:<限定名>`,值是当前的分支数,只能往下改。BASELINE: dict[str, int] = {- "src/peach/web_batch.py:_scored_junk": 62,+ "src/peach/junk.py:scored_candidates": 57,"scripts/localize_performer_names.py:collect": 63,"src/peach/web_resource_sync.py:_resource_orphan_plan": 50,"src/peach/web_review.py:_review_rows": 43,@@ -49,13 +49,11 @@ BASELINE: dict[str, int] = {"scripts/fetch_studio_avatar_candidates.py:main": 44,"src/peach/web_catalog.py:q_item": 44,"src/peach/web_follow.py:q_follow_authors": 40,- "scripts/sheets.py:run": 40,"src/peach/taste_history.py:_taste_analysis": 40,"src/peach/web_catalog.py:catalog_filter": 40,"scripts/audit_domain_codes.py:collect": 39,"src/peach/web_catalog.py:q_items": 37,"scripts/harvest_social_avatars.py:harvest_entity": 37,- "scripts/probe.py:run": 36,"scripts/merge_duplicate_identities.py:collect_repeated_projections": 34,"src/peach/fc2_similarity.py:media_evidence": 34,"scripts/clean_names.py:run": 33,@@ -71,12 +69,18 @@ ACCEPTED: dict[str, tuple[int, str]] = {42, "应用工厂:按启动顺序构造数据库、各项后台服务(含可选分享)与路由并接成一个 app,每个可注入的""依赖各带一个缺省;挂载探测按配置状态启动。拆开会把同一条构造顺序分散到几处"),"src/peach/library_processing.py:process_library": (- 47, "顺序流水线:本地资料、本地预览、联网采集、写候选、收尾共用一份进度状态"+ 48, "顺序流水线:垃圾检查、媒体属性、本地资料、本地预览、联网采集、写候选、收尾共用一份进度状态""与问题记录;来源扫描、资产查询与外部会话分别归 `_scan_sources`、""`_collection_rows`、`_RemoteSession`,剩下的是任务编排与逐项目资料处理"),"src/peach/jav_cover_fetch.py:run": (50, "批处理的主循环:每个番号依次量本机尺寸、择优、比较、落盘、记日志,统计、日志行""和出错后换新的连接池在整轮里共用,逐段拆开只是把这几样来回传"),+ "scripts/probe.py:run": (+ 41, "媒体批处理:选择允许读取的资产、检查垃圾、并行探测、串行提交与重判共用日志、"+ "停止标志和磁盘闸门;提交后的资产列表必须跨缓冲批次保留,拆散会割裂中止时的收尾"),+ "scripts/sheets.py:run": (+ 43, "媒体批处理:按来源选择资产并检查垃圾、并行抽帧、串行登记共用日志、计数和停止标志;"+ "已有产物复用与磁盘闸门属于同一工作循环,拆开会把退出与失败统计分散到多处"),}#: 一句理由至少要说清「是哪种形状、拆开差在哪」,短于这个长度的多半只是占位。diff --git a/tests/test_downloads_postprocess.py b/tests/test_downloads_postprocess.pyindex 4c7e75136..38562a819 100644--- a/tests/test_downloads_postprocess.py+++ b/tests/test_downloads_postprocess.py@@ -3,7 +3,7 @@ import tempfileimport unittestfrom pathlib import Pathfrom types import SimpleNamespace-from unittest.mock import patch+from unittest.mock import Mock, patchfrom peach import download_postprocess as postfrom peach.downloads import DownloadStore, DownloadTask, INGESTED, POSTPROCESS_PENDING@@ -13,7 +13,9 @@ from support.ledger import fresh_ledgerclass DownloadPostprocessTests(unittest.TestCase):def test_isolated_ads_record_the_junk_reason(self):- self.contract.download_junk_candidates = lambda: [{"id": self.asset_id}]+ with self.database.write_transaction() as connection:+ connection.execute("UPDATE asset SET medium='other',name='推广.url',path=? WHERE id=?",+ (str(self.root / '推广.url'), self.asset_id))self.assertEqual(post._isolate_ads(self.contract, -1, [self.asset_id]), 1)with self.database.read_connection() as connection:self.assertEqual(tuple(connection.execute("SELECT disposal,disposal_reason FROM asset WHERE id=?",@@ -40,7 +42,7 @@ class DownloadPostprocessTests(unittest.TestCase):postprocess_state=POSTPROCESS_PENDING))self.contract = SimpleNamespace(database=self.database, db_path=self.database.db_path,- candidate_root=self.root / 'generated', download_junk_candidates=lambda: [],+ candidate_root=self.root / 'generated',download_organize_templates=lambda: {'115': {'file': '{code}'}}, cache_bust=lambda: None)self.handle = SimpleNamespace(progress=lambda **_: None)plan = {'rows': [{'asset_id': self.asset_id, 'action': 'move',@@ -51,6 +53,9 @@ class DownloadPostprocessTests(unittest.TestCase):collect = patch.object(post, '_collect', return_value={'candidates': 0})collect.start()self.addCleanup(collect.stop)+ probe = patch.object(post, '_probe_folder', return_value=0)+ probe.start()+ self.addCleanup(probe.stop)def run_task(self):return post.run(self.contract, post.followup_key(self.task.id), self.handle)@@ -104,3 +109,34 @@ class DownloadPostprocessTests(unittest.TestCase):self.assertRaisesRegex(RuntimeError, '整理'):self.run_task()self.assertTrue(self.source.exists())++ def test_followups_are_declared_only_after_successful_organization(self):+ events = []+ self.handle.run_id = 123+ self.contract.task_runs = Mock()+ rows = [{'task_key': 'avatar', 'followup_key': 'asset:1'}]+ self.contract.task_runs.enqueue_followups.side_effect = lambda *_: events.append('followup')+ original = post._organize+ def organize(*args):+ events.append('organize')+ return original(*args)+ with patch.object(post, '_isolate_ads', side_effect=lambda *_: events.append('check') or 0), \+ patch.object(post, '_probe_folder', side_effect=lambda *_, **options: events.append('probe') or 1), \+ patch.object(post, '_collect', side_effect=lambda *_: events.append('collect') or {'followups': rows}), \+ patch.object(post, '_organize', side_effect=organize), \+ patch.object(post, 'declared_rows', return_value=rows):+ result = self.run_task()+ self.assertEqual(events, ['check', 'probe', 'check', 'collect', 'check', 'organize', 'followup'])+ self.assertEqual(result['probed'], 1)+ self.assertEqual(self.run_task(), {'task_id': self.task.id, 'already_complete': True})+ self.contract.task_runs.enqueue_followups.assert_called_once()++ def test_metadata_or_organization_failure_never_dispatches_followups(self):+ self.handle.run_id = 123+ self.contract.task_runs = Mock()+ for failed_stage in ('_collect', '_organize'):+ with self.subTest(stage=failed_stage), patch.object(post, failed_stage, side_effect=RuntimeError('模拟阶段失败')):+ with self.assertRaisesRegex(RuntimeError, '模拟阶段失败'):+ self.run_task()+ self.assertEqual(self.store.get(self.task.id).postprocess_state, 'failed')+ self.contract.task_runs.enqueue_followups.assert_not_called()diff --git a/tests/test_media_probe.py b/tests/test_media_probe.pyindex 62a6f924b..a73072b60 100644--- a/tests/test_media_probe.py+++ b/tests/test_media_probe.py@@ -88,6 +88,43 @@ class ProbeUnmeasuredTests(unittest.TestCase):self.assertEqual(self.probe(ffprobe=None), 0)self.assertIsNone(self.row(pending)[0])+ def test_file_scope_reads_exactly_that_file(self):+ self.add(ROOT + r'\a.mp4')+ self.add(ROOT + r'\b.mp4')+ self.assertEqual(self.probe(root=ROOT + r'\a.mp4'), 1)+ self.assertEqual(self.probed, ['a.mp4'])++ def test_measured_short_promotion_is_isolated_and_bound_main_is_protected(self):+ from peach.repository import LedgerDatabase+ for aid in (1, 2):+ self.add(ROOT + rf'\bbsxv.xyz-ABC-123\附件{aid}.mp4')+ with LedgerDatabase(self.db).write_transaction() as connection:+ connection.execute("UPDATE asset SET size=1048576,creator='bbsxv.xyz',code='ABC-123'")+ with mock.patch.object(media_probe, 'probe_file', return_value=(10, 1280, 720, 'h264', 30, None)), \+ mock.patch.object(media_probe, 'resolve_case_insensitive', str):+ count = media_probe.probe_unmeasured(self.db, 'ffprobe', 'local', ROOT, protected_ids=[1])+ self.assertEqual(count, 2)+ with LedgerDatabase(self.db).read_connection() as connection:+ self.assertEqual([tuple(row) for row in connection.execute(+ 'SELECT id,duration,disposal FROM asset ORDER BY id')], [(1, 10, None), (2, 10, 'trash')])++ def test_personal_records_reattach_before_measured_junk_isolation(self):+ from peach.repository import LedgerDatabase+ fresh = self.add(ROOT + r'\bbsxv.xyz\点击进入.mp4')+ with LedgerDatabase(self.db).write_transaction() as connection:+ connection.execute("UPDATE asset SET size=1000,creator='bbsxv.xyz' WHERE id=?", (fresh,))+ connection.execute("INSERT INTO asset(location,path,name,medium,size,creator,duration,play_count,disposal) "+ "VALUES('local','R:\\gone\\点击进入.mp4','点击进入.mp4','video',1000,'bbsxv.xyz',10,1,'vanished')")+ checked = []+ with mock.patch.object(media_probe, 'probe_file', return_value=(10, 1280, 720, 'h264', 30, None)), \+ mock.patch.object(media_probe, 'resolve_case_insensitive', str):+ media_probe.probe_unmeasured(self.db, 'ffprobe', 'local', ROOT, on_junk=checked.append)+ with LedgerDatabase(self.db).read_connection() as connection:+ self.assertEqual(tuple(connection.execute('SELECT disposal,play_count FROM asset WHERE id=?',+ (fresh,)).fetchone()), (None, 1))+ self.assertEqual(connection.execute('SELECT count(*) FROM record_rehome').fetchone()[0], 1)+ self.assertEqual(checked[0]['isolated'], 0)+def test_an_unreadable_file_is_marked_failed_so_the_next_scan_skips_it(self):broken = self.add(ROOT + r"\坏.mp4")diff --git a/tests/test_metadata_library.py b/tests/test_metadata_library.pyindex 9ec8f90ad..ee77ad7cc 100644--- a/tests/test_metadata_library.py+++ b/tests/test_metadata_library.py@@ -2804,7 +2804,7 @@ class LibraryPreviewCompletionTests(unittest.TestCase):finish.assert_not_called()factory.assert_not_called()self.assertEqual([(step['id'], step['status']) for step in result['steps']],- [('sources', 'complete'), ('media', 'complete')])+ [('sources', 'complete'), ('inspect', 'complete'), ('media', 'complete')])def test_missing_ffmpeg_is_retryable_and_collection_can_finish_its_work(self):self.resolver.ffmpeg.return_value = None@@ -2859,6 +2859,48 @@ class LibraryPreviewCompletionTests(unittest.TestCase):path = connection.execute('SELECT snapshot_path FROM asset WHERE id=1').fetchone()[0]self.assertTrue(Path(path).is_file())+ def test_inspection_uses_measured_rows_and_removes_ads_before_preview_or_metadata(self):+ events = []+ with self.database.write_transaction() as connection:+ connection.execute("UPDATE asset SET duration=NULL,code='ABW-358' WHERE id=1")+ connection.execute("INSERT INTO asset(id,location,path,name,medium,size,duration,creator,code) "+ "VALUES(5,'local',?,'附件.mp4','video',1048576,NULL,'bbsxv.xyz','ABC-123')",+ (str(self.media / 'bbsxv.xyz-ABC-123' / '附件.mp4'),))+ def probe(_ffprobe, path, _timeout):+ events.append('probe')+ return (10 if path.endswith('附件.mp4') else 1800), 1920, 1080, 'h264', 30, None+ def previews(_config, rows, *_args, **_options):+ events.append('preview')+ self.assertEqual([(row['id'], row['duration']) for row in rows], [(1, 1800)])+ def collect(*_args, **_options):+ events.append('metadata')+ return [], 0+ self.resolver.ffprobe.return_value = SimpleNamespace(path=Path('test-ffprobe'))+ with patch('peach.media_probe.probe_file', side_effect=probe), \+ patch('peach.media_probe.resolve_case_insensitive', str), \+ patch('peach.library_processing.FFmpegResolver', return_value=self.resolver), \+ patch('peach.library_processing._generate_previews', side_effect=previews), \+ patch('peach.library_processing._RemoteSession.collect', side_effect=collect), \+ patch('peach.library_processing._finish_collection', return_value={}):+ result = self.run_processing(stage='collect', asset_ids=[1, 5])+ self.assertEqual(events, ['probe', 'probe', 'preview', 'metadata'])+ self.assertEqual((result['status'], result['junk_isolated']), ('complete', 1))+ with self.database.read_connection() as connection:+ self.assertEqual(connection.execute('SELECT disposal FROM asset WHERE id=5').fetchone()[0], 'trash')++ def test_metadata_stage_inspects_known_evidence_without_reading_video(self):+ with self.database.write_transaction() as connection:+ connection.execute("UPDATE asset SET name='点击进入.mp4',size=1000 WHERE id=1")+ with patch('peach.library_processing.probe_unmeasured') as probe, \+ patch('peach.library_processing._generate_previews') as previews, \+ patch('peach.library_processing._RemoteSession.collect') as collect, \+ patch('peach.library_processing._finish_collection', return_value={}):+ result = self.run_processing(stage='metadata', asset_ids=[1])+ self.assertEqual((result['status'], result['junk_isolated']), ('complete', 1))+ probe.assert_not_called()+ previews.assert_not_called()+ collect.assert_not_called()+def test_failed_tiling_leaves_no_registered_or_partial_snapshot(self):import subprocesswith patch('peach.library_processing.FFmpegResolver', return_value=self.resolver), \diff --git a/tests/test_organize.py b/tests/test_organize.pyindex ee144697f..991bec6ef 100644--- a/tests/test_organize.py+++ b/tests/test_organize.py@@ -350,6 +350,17 @@ class ApplyTests(unittest.TestCase):row = connection.execute("SELECT path,name FROM asset WHERE id=1").fetchone()self.assertEqual(row, ("R:\\media\\厂牌\\ABC-123.mp4", "ABC-123.mp4"))+ def test_a_disposed_asset_in_a_saved_plan_is_left_on_disk(self):+ original = self._file('old/title.mp4')+ _seed(self.db_path, [dict(id=1, path=r'R:\media\old\title.mp4', name='title.mp4', code='ABC-123')])+ plan = _plan(self.db_path, file_template='{number}')+ with closing(sqlite3.connect(self.db_path)) as connection:+ connection.execute("UPDATE asset SET disposal='trash' WHERE id=1")+ connection.commit()+ result = self._apply(plan)+ self.assertEqual((result['moved'], result['failed']), (0, 1))+ self.assertTrue(original.is_file())+def test_scoped_symbol_cleanup_and_rollback_preserve_unselected_files(self):self._file('合集/★标题⧹⧹片段.mp4')self._file('合集2/★标题.mp4')@@ -492,6 +503,19 @@ class EndpointTests(unittest.TestCase):self.assertEqual(result["rows"][0]["target_path"], "R:\\media\\旧\\ABC-123 标题.mp4")self.assertTrue(Path(result["plan_csv"]).is_file())+ def test_write_reports_existing_foreign_keys_and_rejects_new_violations(self):+ from peach.web_organize import _run_write+ with closing(sqlite3.connect(self.db_path)) as connection:+ connection.execute("INSERT INTO media_binding(asset_id,backend,external_id) VALUES(99999,'test','old')")+ connection.commit()+ self.assertEqual(_run_write(self.contract, 'test', lambda _: {})['existing_foreign_keys'], 1)+ def introduce(connection):+ connection.execute("INSERT INTO media_binding(asset_id,backend,external_id) VALUES(99998,'test','new')")+ connection.commit()+ return {}+ with self.assertRaisesRegex(RuntimeError, 'new_foreign_keys=1'):+ _run_write(self.contract, 'test-new', introduce)+def test_apply_without_confirmation_is_refused(self):from peach.web_organize import w_organize_applydiff --git a/tests/test_processing_control.py b/tests/test_processing_control.pyindex b1629881f..2005fdc79 100644--- a/tests/test_processing_control.py+++ b/tests/test_processing_control.py@@ -499,7 +499,7 @@ class ProcessingControlTests(unittest.TestCase):def scan(*_args, **kwargs):kwargs['progress'](1, 10)kwargs['progress'](2, 30)- return SimpleNamespace(files=2, total_bytes=30, new_ids=[])+ return SimpleNamespace(files=2, total_bytes=30, new_ids=[], junk_isolated=1)with patch('peach.library_processing.scan_location', side_effect=scan), \patch('peach.library_processing._online_source_root', return_value=self.root), \patch('peach.library_processing._ffprobe_path', return_value=None), \@@ -513,3 +513,4 @@ class ProcessingControlTests(unittest.TestCase):self.assertEqual([row['scanned_bytes'] for row in discovering], [10, 30, 40, 60])self.assertTrue(all(row['total'] == 0 for row in discovering))self.assertEqual((result.scanned, result.scanned_bytes), (4, 60))+ self.assertEqual(result.junk_isolated, 2)diff --git a/tests/test_scan.py b/tests/test_scan.pyindex 8775acd2b..f864ada94 100644--- a/tests/test_scan.py+++ b/tests/test_scan.py@@ -15,7 +15,7 @@ import osimport sqlite3import tempfileimport unittest-from contextlib import redirect_stdout+from contextlib import closing, redirect_stdoutfrom pathlib import Path, PureWindowsPathfrom unittest.mock import patch@@ -73,6 +73,23 @@ class WindowsShapeTests(_ScanCase):for path in rows:self.assertTrue(Path(path).is_file(), path)+ def test_scan_isolates_shortcuts_without_removing_files_or_media_sidecars(self):+ ad = self.media / '点击进入.url'+ ad.write_bytes(b'link')+ video = self.media / 'ABC-123.mp4'+ video.write_bytes(b'video')+ (self.media / 'ABC-123.nfo').write_text('<movie/>', encoding='utf-8')+ kwargs = dict(declared_roots={'local': (str(self.media),)}, windows=True)+ first, _ = self._scan(str(self.media), **kwargs)+ second, _ = self._scan(str(self.media), **kwargs)+ self.assertEqual((first.junk_isolated, second.junk_isolated), (1, 0))+ self.assertTrue(ad.is_file())+ with closing(sqlite3.connect(self.db)) as connection:+ self.assertEqual(connection.execute('SELECT disposal,disposal_reason FROM asset WHERE path=?',+ (str(ad),)).fetchone(), ('trash', 'junk'))+ self.assertEqual(connection.execute('SELECT disposal FROM asset WHERE path=?',+ (str(video),)).fetchone(), (None,))+def test_scanned_video_is_visible_without_a_thumbnail_filter(self):from support.web import route_clientfrom peach.api import create_appdiff --git a/tests/test_scripts.py b/tests/test_scripts.pyindex cdc5de40e..ed842dca9 100644--- a/tests/test_scripts.py+++ b/tests/test_scripts.py@@ -3036,14 +3036,11 @@ class OperationalScriptTests(unittest.TestCase):sheets = self.sheetswith tempfile.TemporaryDirectory() as tmp:root = Path(tmp)- db = root / "ledger.db"+ db = fresh_ledger(root)connection = sqlite3.connect(db)connection.execute(- "CREATE TABLE asset(id INTEGER PRIMARY KEY,location TEXT,path TEXT,"- "medium TEXT,duration REAL,size INTEGER,snapshot_path TEXT,disposal TEXT)"- )- connection.execute(- "INSERT INTO asset VALUES(18349,'local',?,'video',752.24,1000,NULL,NULL)",+ "INSERT INTO asset(id,location,path,medium,duration,size,snapshot_path,disposal) "+ "VALUES(18349,'local',?,'video',752.24,1000,NULL,NULL)",(str(root / "one.mp4"),),)connection.commit()@@ -3072,19 +3069,17 @@ class OperationalScriptTests(unittest.TestCase):sheets = self.sheetswith tempfile.TemporaryDirectory() as tmp:root = Path(tmp)- db = root / "ledger.db"+ db = fresh_ledger(root)connection = sqlite3.connect(db)- connection.execute(- "CREATE TABLE asset(id INTEGER PRIMARY KEY,location TEXT,path TEXT,"- "medium TEXT,duration REAL,size INTEGER,snapshot_path TEXT,disposal TEXT)"- )# 18349 已有陈旧产物且已登记;1 是同来源的另一条待抽项,不该被顺带带走。connection.execute(- "INSERT INTO asset VALUES(18349,'local',?,'video',110.87,2000,'stale.jpg',NULL)",+ "INSERT INTO asset(id,location,path,medium,duration,size,snapshot_path,disposal) "+ "VALUES(18349,'local',?,'video',110.87,2000,'stale.jpg',NULL)",(str(root / "one.mp4"),),)connection.execute(- "INSERT INTO asset VALUES(1,'local',?,'video',600.0,1000,NULL,NULL)",+ "INSERT INTO asset(id,location,path,medium,duration,size,snapshot_path,disposal) "+ "VALUES(1,'local',?,'video',600.0,1000,NULL,NULL)",(str(root / "two.mp4"),),)connection.commit()@@ -3150,18 +3145,16 @@ class OperationalScriptTests(unittest.TestCase):for named, expected in ((False, ["keep.mp4"]), (True, ["trashed.mp4"])):with self.subTest(named=named), tempfile.TemporaryDirectory() as tmp:root = Path(tmp)- db = root / "ledger.db"+ db = fresh_ledger(root)connection = sqlite3.connect(db)connection.execute(- "CREATE TABLE asset(id INTEGER PRIMARY KEY,location TEXT,path TEXT,"- "medium TEXT,duration REAL,size INTEGER,snapshot_path TEXT,disposal TEXT)"- )- connection.execute(- "INSERT INTO asset VALUES(1,'local',?,'video',600.0,2000,NULL,'trash')",+ "INSERT INTO asset(id,location,path,medium,duration,size,snapshot_path,disposal) "+ "VALUES(1,'local',?,'video',600.0,2000,NULL,'trash')",(str(root / "trashed.mp4"),),)connection.execute(- "INSERT INTO asset VALUES(2,'local',?,'video',600.0,1000,NULL,NULL)",+ "INSERT INTO asset(id,location,path,medium,duration,size,snapshot_path,disposal) "+ "VALUES(2,'local',?,'video',600.0,1000,NULL,NULL)",(str(root / "keep.mp4"),),)connection.commit()@@ -3194,15 +3187,12 @@ class OperationalScriptTests(unittest.TestCase):sheets = self.sheetswith tempfile.TemporaryDirectory() as tmp:root = Path(tmp)- db = root / "ledger.db"+ db = fresh_ledger(root)connection = sqlite3.connect(db)- connection.execute(- "CREATE TABLE asset(id INTEGER PRIMARY KEY,location TEXT,path TEXT,"- "medium TEXT,duration REAL,size INTEGER,snapshot_path TEXT,disposal TEXT)"- )for asset_id in range(1, 61):connection.execute(- "INSERT INTO asset VALUES(?,?,?,?,?,?,NULL,NULL)",+ "INSERT INTO asset(id,location,path,medium,duration,size,snapshot_path,disposal) "+ "VALUES(?,?,?,?,?,?,NULL,NULL)",(asset_id, "local", str(root / f"{asset_id}.mp4"), "video", 600.0, 1000),)connection.commit()@@ -3301,14 +3291,8 @@ class OperationalScriptTests(unittest.TestCase):probe = self.probewith tempfile.TemporaryDirectory() as tmp:root = Path(tmp)- db = root / "ledger.db"+ db = fresh_ledger(root)connection = sqlite3.connect(db)- connection.execute(- "CREATE TABLE asset(id INTEGER PRIMARY KEY,location TEXT,path TEXT,"- "medium TEXT,duration REAL,size INTEGER,width INTEGER,height INTEGER,"- "vcodec TEXT,fps REAL,has_audio INTEGER,ctx_length TEXT,ctx_orient TEXT,"- "ctx_quality TEXT,disposal TEXT)"- )for asset_id, duration in ((18349, 752.24), (1, None)):connection.execute("INSERT INTO asset(id,location,path,medium,duration,size) "diff --git a/tests/test_trash_junk.py b/tests/test_trash_junk.pyindex 6de99a542..90395d10a 100644--- a/tests/test_trash_junk.py+++ b/tests/test_trash_junk.py@@ -326,7 +326,7 @@ class TrashJunkTests(unittest.TestCase):self.assertFalse(scan.is_sidecar(name,frozenset()))def test_download_site_navigation_images_keep_content_attachments(self):- from peach.web_batch import _promo_name_reason+ from peach.junk import _promo_name_reasonself.assertIn('须核验视频内容', _promo_name_reason('.mp4', 'A:\\作品\\社 區 最 新 情 報.mp4', 15089802))self.assertEqual(_promo_name_reason('.mp4', 'A:\\作品\\社区最新情报合集.mp4', 15089802), '')self.assertEqual(_promo_name_reason('.mp4', 'A:\\作品\\社區最新情報.mp4', 32 * 1024**2), '')@@ -340,7 +340,7 @@ class TrashJunkTests(unittest.TestCase):self.assertEqual(_promo_name_reason(Path(name).suffix, 'B:\\作品\\' + name, 50000), '')def test_tiny_promotion_cards_require_exact_names_and_content_review(self):- from peach.web_batch import _promo_name_reason+ from peach.junk import _promo_name_reasonfor name in ('51风流', '代开实习证明', '扫码约炮', '探花社区'):self.assertIn('须核验图片内容', _promo_name_reason('.png', 'A:\\作品\\'+name+'.png', 1400))self.assertFalse(_promo_name_reason('.png', 'A:\\作品\\'+name+'.png', 4097))@@ -348,7 +348,7 @@ class TrashJunkTests(unittest.TestCase):self.assertFalse(_promo_name_reason('.mp4', 'A:\\作品\\'+name+'.mp4', 1400))def test_site_cards_require_matching_parent_and_small_png_content_review(self):- from peach.web_batch import _promo_name_reason+ from peach.junk import _promo_name_reasonname='|91porn|真实国产原创亚洲最火成人网站,强势回归|'source='A:\\待确认\\My Pack\\213\\'+name+'\\'+name+'.png'self.assertIn('须核验图片内容', _promo_name_reason('.png', source, 49130))@@ -360,7 +360,7 @@ class TrashJunkTests(unittest.TestCase):self.assertFalse(_promo_name_reason(suffix,path,size))def test_information_wmv_requires_exact_name_size_and_content_review(self):- from peach.web_batch import JUNK_VIDEO_MAX_BYTES, _promo_name_reason+ from peach.junk import JUNK_VIDEO_MAX_BYTES, _promo_name_reasonfor name in ('最新情報', '最 新 情 报'):source = 'B:\\云下载\\DOCP-324\\' + name + '.wmv'self.assertIn('须核验视频内容', _promo_name_reason('.wmv', source, 90867046))@@ -419,7 +419,7 @@ class TrashJunkTests(unittest.TestCase):self.assertEqual(trashable, {4, 5})def test_promotion_names_come_from_the_lexicon_file(self):- from peach import web_batch+ from peach import junk as web_batchlexicon = self.root / 'promo_attachment_names.json'lexicon.write_text('{"rules": [{"reason": "测试推广卡,须核验图片内容", "suffixes": [".png"], ''"max_bytes": 4096, "names": ["测试推广卡"]}]}', encoding='utf-8')