Peach: Local-first video metadata scraper and media manager.
fix(media): 按待处理项与缓存估算采集流量
longmeidao8a83310ba5a1
diff --git a/README.en.md b/README.en.mdindex 30049e47d..73726762e 100644--- a/README.en.md+++ b/README.en.md@@ -47,7 +47,7 @@ Watch history, favorites and settings stay on your computer. Scanning leaves oriScanning, follow updates and new-release metadata collection save their progress and continue unfinished items after the app restarts. See [task recovery](docs/OPERATIONS.md#长时任务断点续跑) for recovery boundaries.-Scanning and collection show a traffic estimate before starting, then usage and estimated totals grouped by each media folder's proxy traffic protection setting. See [configuration](docs/OPERATIONS.md#配置页) for measurement coverage and estimation limits.+Scanning and collection estimate traffic from pending work and local caches, deduplicating downloads for the same release code. Usage and estimated totals follow each media folder's proxy traffic protection setting. See [configuration](docs/OPERATIONS.md#配置页) for measurement coverage and estimation limits.The combined "Send version statistics and error diagnostics (recommended)" option is checked by default. Clear it during installation, first setup or in settings to disable both. Saved opt-out choices are preserved. While enabled, it sends the listed technical fields to Sentry; full logs, diagnostic ZIPs, media details and watch history are excluded. See [remote diagnostics](docs/DIAGNOSTICS.md#可选远程诊断).diff --git a/README.md b/README.mdindex 7d5a3d313..eeddf5f0d 100644--- a/README.md+++ b/README.md@@ -47,7 +47,7 @@ https://github.com/user-attachments/assets/a5610874-e611-41bb-be22-7d99f0d64b62扫描、关注更新和新作资料采集会保存进度,应用重启后继续处理未完成的项目。恢复范围见[断点续跑](docs/OPERATIONS.md#长时任务断点续跑)。-扫描与采集启动前可查看流量粗估,运行中显示已用流量和预计总流量,按各盘的代理流量保护设置分别汇总。统计范围与估算限制见[运行与配置](docs/OPERATIONS.md#配置页)。+扫描与采集按待处理资料、缓存和同番号去重估算流量,运行中显示已用流量和预计总流量,按各盘的代理流量保护设置分别汇总。统计范围与估算限制见[运行与配置](docs/OPERATIONS.md#配置页)。「发送版本统计与错误诊断(推荐)」默认勾选,可在安装、首次设置或设置页取消。两类技术事件由同一开关控制,已保存的关闭选择会保留。启用后向 Sentry 发送页面列出的技术字段,不自动上传完整日志、诊断 ZIP、媒体信息或观看记录。见[远程诊断说明](docs/DIAGNOSTICS.md#可选远程诊断)。diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.mdindex c964422fc..095a0fc48 100644--- a/docs/OPERATIONS.md+++ b/docs/OPERATIONS.md@@ -196,7 +196,7 @@ Peach 配置页地址是 `/configuration`。从管理菜单或设置弹窗的「媒体修复在「数据管理」页,订阅源在「关注管理」页的「订阅源」页签(分工见 [ADR-0050](adr/0050-settings-modal-holds-values-pages-hold-work.md))。-「扫描与采集」启动前显示流量粗估;运行中的任务面显示已用流量与预计总流量,可展开查看带网盘图标的各盘明细,连接方式显示「直连」或「代理」,区间使用 `6-12MB` 格式。媒体目录按各自的「代理流量保护」设置分别汇总,开启的目录计入代理流量估算。本地磁盘读取不计入网络流量;资料站下载按来源连接设置另列。挂载读取量受缓存和预读影响,浏览器与外部采集器的流量也无法完整统计,读数作估算参考。新目录尚未登记的文件在扫描后更新预估,暂停和应用重启保留同一任务的累计值。+「扫描与采集」启动前按待处理项显示流量约值;运行中的任务面显示已用流量与预计总流量,可展开查看带网盘图标的各盘明细,连接方式显示「直连」或「代理」。资料下载按番号去重,扣除已齐全的字段、可用封面、有效缓存和来源近期确认无结果的请求,并采用现有 HTTP 下载预算;外部采集器另行估算。配套文件检查完毕后,按实际取得的字段和核查范围更新预估,完成的工作从待下载量中扣除。媒体目录按各自的「代理流量保护」设置分别汇总,开启的目录计入代理流量估算。本地磁盘读取不计入网络流量;资料站下载按来源连接设置另列。挂载读取量受缓存和预读影响,浏览器与外部采集器的流量也无法完整统计,读数作估算参考。新目录尚未登记的文件在扫描后更新预估,暂停和应用重启保留同一任务的累计值。常用的本机设置:diff --git a/docs/REUSE.md b/docs/REUSE.mdindex 7a39932fb..bb6f70440 100644--- a/docs/REUSE.md+++ b/docs/REUSE.md@@ -108,7 +108,9 @@ Folia(`chthollyphile/folia-major`,AGPL-3.0,revision `9a01e788ed52d25ea01b1## 馆藏处理流量-馆藏处理的流量分账复用 `media_sources.cost_for` 的最长路径前缀与逐目录保护设置、`ContextVar`、现有任务快照和断点恢复。HTTP 下载复用固定版 HTTPX 0.28.1(BSD-3-Clause)的 [`num_bytes_downloaded`](https://www.python-httpx.org/advanced/clients/#monitoring-download-progress),按压缩传输字节累计,失败的部分响应保留读数;不使用解压后的正文大小替代下载量。FFmpeg 输入读取复用 verbose 级别的 AVIO Statistics,NFO 与配套图片累计实际应用读取量。挂载缓存、预读与系统协议开销无法由这些读数推导,媒体读数仅作流量估算;浏览器、amane 与 curl 指纹传输保留未观测标记,不填零。预估只查询账本,不遍历或下载媒体;资料按每部 0.25–8 MiB、配套图片至多 2 MiB、待探测媒体 1–32 MiB、九帧至多 288 MiB 粗估,文件大小限制媒体项,保护开启的目录不计自动探测。估算是 Peach 的任务策略,不是上游流量保证;读数按任务保存,逐目录明细与代理、全部合计共用一份状态。界面复用 React Aria Slider、现有 `ProcessingEffort` 与 `formModal`;无新增依赖。HTTPX 官方接口已用压缩响应和中断流做隔离实证,未接入机器全局流量监控,以免混入其他应用与任务。+馆藏处理的流量分账复用 `media_sources.cost_for` 的最长路径前缀与逐目录保护设置、`ContextVar`、现有任务快照和断点恢复。HTTP 下载复用固定版 HTTPX 0.28.1(BSD-3-Clause)的 [`num_bytes_downloaded`](https://www.python-httpx.org/advanced/clients/#monitoring-download-progress),按压缩传输字节累计,失败的部分响应保留读数。FFmpeg 输入读取复用 verbose 级别的 AVIO Statistics,NFO 与配套图片累计实际应用读取量;挂载缓存、预读、浏览器与外部采集器使读数仅能作为估算。++`processing_traffic_plan` 是 Peach 的工作量策略,复用 `_missing_fields`、`_scrapes_as_jav`、`_RemoteSession` 的来源链、7 天缓存、`_MissCache` 与 `cover_settled`,按番号合并资料请求;只有实际待补全、未命中缓存的来源进入计划。HTTP 下载共用 `MAX_SOURCE_BYTES`,amane 计划不受该 HTTP 预算裁切。按来源连接设置拆分直连与代理;配套文件检查完毕后,用实际字段与核查范围重算,并将完成阶段和番号持久到任务快照,恢复时继续扣除已完成工作。预检只访问账本和本机缓存目录,至多采样 256 张封面与 256 份 NFO 的大小中位数;无样本时采用封面 512 KiB、NFO 32 KiB,待探测媒体 8 MiB、九帧 72 MiB,媒体项受文件大小限制。资料每站按两次、每次约 250 KiB 的采集器工作量基准估算。以上是默认计划值,不是实际下载保证。界面以 `estimated_bytes` / `metered_estimated_bytes` 显示约值,逐盘、代理与全部合计共用一份状态;已用量保留实际累计。领域工作量不引入外部估算库,通用统计、目录索引与数据库复用 Python 标准库;无新增依赖。明细开合复用设置页 `Disclosure` 与 `setCollapseOpen`,可由任务提示同步收起;任务正文的网格高度与同一枚箭头沿用 Collapse 的 200ms 节奏。全局减少动态效果设置关闭这些过渡。网盘标识复用 `SourceLabel`,115、PikPak 等使用已登记的站标。diff --git a/frontend/e2e/library-processing-controls.test.ts b/frontend/e2e/library-processing-controls.test.tsindex 2fa8cf6dd..69d329521 100644--- a/frontend/e2e/library-processing-controls.test.ts+++ b/frontend/e2e/library-processing-controls.test.ts@@ -17,12 +17,13 @@ describe('扫描与采集控制', () => {current_asset_name: '', current_action: '', stage: '',stalled: false, waited_seconds: 0,traffic: { used_bytes: 0, estimated_min_bytes: 1048576, estimated_max_bytes: 8388608,+ estimated_bytes: 6291456, metered_estimated_bytes: 3145728,metered_used_bytes: 0, metered_estimated_min_bytes: 0, metered_estimated_max_bytes: 4194304,partial: true, unknown_files: true,sources: [{ key: 'drive-a', label: '示例网盘 A', root: 'A:\\Media', location: '115', metered: true,- used_bytes: 0, estimated_min_bytes: 0, estimated_max_bytes: 4194304, unobserved: ['挂载缓存与预读未观测'] },+ used_bytes: 0, estimated_bytes: 3145728, estimated_min_bytes: 0, estimated_max_bytes: 4194304, unobserved: ['挂载缓存与预读未观测'] },{ key: 'drive-b', label: '示例网盘 B', root: 'B:\\Media', location: 'pikpak', metered: false,- used_bytes: 0, estimated_min_bytes: 1048576, estimated_max_bytes: 4194304, unobserved: ['挂载缓存与预读未观测'] }] },+ used_bytes: 0, estimated_bytes: 3145728, estimated_min_bytes: 1048576, estimated_max_bytes: 4194304, unobserved: ['挂载缓存与预读未观测'] }] },eta_kind: 'historical', eta_initial: true, eta_seconds_min: 1200, eta_seconds_max: 2400 };const writes: Record<string, unknown>[] = [];const opened = await visit(browser, '/data-cleanup', viewport, async page => {@@ -77,7 +78,7 @@ describe('扫描与采集控制', () => {assert.match(await sources.nth(1).innerText(), /示例网盘 B[\s\S]*直连/);assert.equal(await sources.locator('svg, img').count(), 2);assert.equal((await dialog.innerText()).includes('未观测'), false);- assert.match(await dialog.innerText(), /1-8MB/);+ assert.match(await dialog.innerText(), /约 6MB/);const collapse = dialog.locator('.ui-fcollapse');const transition = await collapse.evaluate(el => ({ property: getComputedStyle(el).transitionProperty, duration: getComputedStyle(el).transitionDuration }));assert.equal(transition.property, 'height');diff --git a/frontend/src/react/library-processing/library-processing.ts b/frontend/src/react/library-processing/library-processing.tsindex 794bf7bd0..e34cd3bb7 100644--- a/frontend/src/react/library-processing/library-processing.ts+++ b/frontend/src/react/library-processing/library-processing.ts@@ -72,11 +72,13 @@ export interface LibraryProcessingData {}export interface TrafficSource {+ estimated_bytes?: number;key: string; label: string; root: string; location: string; metered: boolean;used_bytes: number; estimated_min_bytes: number; estimated_max_bytes: number;unobserved: string[];}export interface ProcessingTraffic {+ estimated_bytes?: number; metered_estimated_bytes?: number;used_bytes: number; estimated_min_bytes: number; estimated_max_bytes: number;metered_used_bytes: number; metered_estimated_min_bytes: number; metered_estimated_max_bytes: number;sources: TrafficSource[]; partial: boolean; unknown_files: boolean;diff --git a/frontend/src/react/library-processing/processing-traffic.tsx b/frontend/src/react/library-processing/processing-traffic.tsxindex c89fb99cc..7d34ce420 100644--- a/frontend/src/react/library-processing/processing-traffic.tsx+++ b/frontend/src/react/library-processing/processing-traffic.tsx@@ -11,6 +11,8 @@ const range = (low: number, high: number) => {const [end, endUnit] = fmtSize(high).split(' ');return startUnit === endUnit ? `${start}-${end}${endUnit}` : `${size(low)}-${size(high)}`;};+const forecast = (expected: number | undefined, low: number, high: number) => expected === undefined+ ? range(low, high) : `${expected > 0 ? '约 ' : ''}${size(expected)}`;/** 确认弹窗、任务卡和管理页共用逐目录分账。 */export function TrafficProgress({ traffic, initial = false, detailsOpen, onDetailsChange }: {@@ -21,8 +23,8 @@ export function TrafficProgress({ traffic, initial = false, detailsOpen, onDetaireturn <div data-processing-traffic="" className="flex min-w-0 flex-col gap-2 text-caption-1-regular text-text-secondary tabular-nums wrap-anywhere"><dl className="inline-grid grid-cols-2 items-baseline gap-x-4 gap-y-1">{!initial ? <><dt>已用流量</dt><dd className="text-right text-text-primary">{traffic.partial ? '约 ' : ''}{size(traffic.used_bytes)}</dd></> : null}- <dt>预计总流量</dt><dd className="text-right text-body-2-medium text-text-primary">{range(traffic.estimated_min_bytes, traffic.estimated_max_bytes)}</dd>- <dt>其中代理流量</dt><dd className="text-right">{initial ? '' : `已用 ${size(traffic.metered_used_bytes)} / 预计 `}{range(traffic.metered_estimated_min_bytes, traffic.metered_estimated_max_bytes)}</dd>+ <dt>预计总流量</dt><dd className="text-right text-body-2-medium text-text-primary">{forecast(traffic.estimated_bytes, traffic.estimated_min_bytes, traffic.estimated_max_bytes)}</dd>+ <dt>其中代理流量</dt><dd className="text-right">{initial ? '' : `已用 ${size(traffic.metered_used_bytes)} / 预计 `}{forecast(traffic.metered_estimated_bytes, traffic.metered_estimated_min_bytes, traffic.metered_estimated_max_bytes)}</dd></dl><Disclosure summary="查看明细" trailing isOpen={detailsOpen} onOpenChange={onDetailsChange}><div className="relative min-w-0">@@ -35,7 +37,7 @@ export function TrafficProgress({ traffic, initial = false, detailsOpen, onDetai</div><div className="flex flex-wrap justify-between gap-x-4 text-text-tertiary">{!initial ? <span>已用{source.unobserved.length ? '约' : ''} {size(source.used_bytes)}</span> : null}- <span>预计 {range(source.estimated_min_bytes, source.estimated_max_bytes)}</span>+ <span>预计 {forecast(source.estimated_bytes, source.estimated_min_bytes, source.estimated_max_bytes)}</span></div></li>)}</ul>diff --git a/src/peach/library_processing.py b/src/peach/library_processing.pyindex 6f1610648..0bb3a0480 100644--- a/src/peach/library_processing.py+++ b/src/peach/library_processing.py@@ -2322,6 +2322,7 @@ def process_library(config, db_path, candidate_root, cover_root, *, location='codef finish_step(key):"""阶段成功返回后发布完成状态;异常退出的阶段保留当前状态。"""+ traffic.finish_work(key)update(steps=_complete_processing_step(state['steps'], key))update()@@ -2429,6 +2430,7 @@ def process_library(config, db_path, candidate_root, cover_root, *, location='cocurrent_asset_id=None, current_asset_name='', current_action='',current_started_at=None, current_deadline_at=None)finish_step('local')+ traffic.prepare_metadata(prepared)if stage in (ALL_STAGES, COLLECT_STAGE):_probe_sources(config, db_path, locations, sources.online_roots, scope_ids, update,total_pending=pending.get('probe', 0), initial_probed=state['probed'])@@ -2455,6 +2457,7 @@ def process_library(config, db_path, candidate_root, cover_root, *, location='coanswered=() if verify_existing else _answered_sources(groups, target_key, missing),received=lambda entries, row=row, code=code, fields=local_fields, verify=verify_existing:publish(row, code, fields, verify, entries))+ traffic.finish_work('metadata', code)state['covers'] += coverscovered.append((row['id'], covers))update(stage='保存待确认资料', current_action='writing_candidates',diff --git a/src/peach/processing_traffic.py b/src/peach/processing_traffic.pyindex e5d2baee2..43aca46ac 100644--- a/src/peach/processing_traffic.py+++ b/src/peach/processing_traffic.py@@ -3,10 +3,9 @@ from __future__ import annotationsimport mathimport re-import sqlite3import threadingimport time-from contextlib import closing, contextmanager+from contextlib import contextmanagerfrom contextvars import ContextVarfrom pathlib import PureWindowsPath@@ -92,6 +91,9 @@ class ProcessingTraffic:('label', 'root', 'location', 'metered', 'network', 'used_bytes', 'unobserved')}self.rows[old['key']].update(remaining_min=0, remaining_max=0)self.unknown_files = stage in ('all', 'scan') and asset_ids is None+ from .processing_traffic_plan import TrafficPlan+ self.plan = TrafficPlan(config, stage, self._media_key)+ self.plan.completed.update(previous.get('completed_work', []))self.refresh_plan()@contextmanager@@ -120,43 +122,23 @@ class ProcessingTraffic:def refresh_plan(self, rows=None):"""账本统计不访问媒体目录;保护开启的目录不预估自动探测。"""- query = "SELECT location,path,size,duration,snapshot_path FROM asset WHERE medium='video' AND disposal IS NULL"- params = []- if self.asset_ids is not None:- query += ' AND id IN (' + ','.join('?' for _ in self.asset_ids) + ')'- params += self.asset_idsif rows is None:- with closing(sqlite3.connect(self.db_path, timeout=30)) as connection:- assets = connection.execute(query, params).fetchall()- else:- assets = [(row['location'], row['path'], row['size'], row['duration'], row['snapshot_path']) for row in rows]+ from .processing_traffic_plan import ledger_rows+ rows = ledger_rows(self.db_path, self.asset_ids)with self.lock:+ self.plan.refresh(rows)for row in self.rows.values():- row.update(remaining_min=0, remaining_max=0)- for location, path, size, duration, preview in assets:- key = self._media_key(path, location)- if key is None or self.stage == 'previews' and location != 'local':- continue- row = self.rows[key]- size = max(0, size or 0)- from .strm import skip_background_read- readable = not skip_background_read(path)- if self.stage != 'previews' and self.stage != 'scan':- # 资料和配套图片的工作量只用于粗估,不当作实测下载。- self.rows['metadata']['remaining_min'] += MIB // 4- self.rows['metadata']['remaining_max'] += 8 * MIB- if row['network']:- row['remaining_max'] += 2 * MIB- if self.stage != 'metadata' and duration is None and not row['metered'] and row['network'] and readable:- row['remaining_min'] += min(size, MIB)- row['remaining_max'] += min(size, 32 * MIB)- if self.stage in ('all', 'collect', 'previews') and location == 'local' and row['network'] and not preview and readable:- from .media_processing_policy import metadata_only- if not metadata_only(self.config.directory('state')):- row['remaining_max'] += min(size, 9 * 32 * MIB)- if row['network']:+ if row['root'] and row['network']:self._reason(row, '挂载缓存与预读未观测')+ def finish_work(self, stage, code=None):+ with self.lock:+ self.plan.finish(stage, code)++ def prepare_metadata(self, prepared):+ with self.lock:+ self.plan.prepare_metadata(prepared)+@staticmethoddef _reason(row, reason):if reason not in row['unobserved']:@@ -174,9 +156,6 @@ class ProcessingTraffic:if not row['network']:returnrow['used_bytes'] += int(size)- remaining = self.rows['metadata'] if key.startswith('metadata-') else row- for field in ('remaining_min', 'remaining_max'):- remaining[field] = max(0, remaining[field] - int(size))if threading.get_ident() != self.owner or time.monotonic() - self.published_at < 0.5:returnself.published_at = time.monotonic()@@ -194,16 +173,20 @@ class ProcessingTraffic:def snapshot(self, *, finished=False):with self.lock:sources = []+ remaining = self.plan.remaining(self.rows)for key, row in self.rows.items():- if not row['root'] and not row['used_bytes'] and not row['remaining_max'] and not row['unobserved']:+ amount = 0 if finished else remaining[key]+ if not row['root'] and not row['used_bytes'] and not amount and not row['unobserved']:continuesource = {**row, 'unobserved': list(row['unobserved'])}sources.append(dict(key=key, **source,- estimated_min_bytes=row['used_bytes'] + (0 if finished else row['remaining_min']),- estimated_max_bytes=row['used_bytes'] + (0 if finished else row['remaining_max'])))+ estimated_bytes=row['used_bytes'] + amount,+ estimated_min_bytes=row['used_bytes'],+ estimated_max_bytes=row['used_bytes'] + amount))result = dict(sources=sources, unknown_files=self.unknown_files and not finished,- partial=any(row['unobserved'] for row in sources))- for field in ('used_bytes', 'estimated_min_bytes', 'estimated_max_bytes'):+ partial=any(row['unobserved'] for row in sources),+ completed_work=sorted(self.plan.completed))+ for field in ('used_bytes', 'estimated_bytes', 'estimated_min_bytes', 'estimated_max_bytes'):result[field] = sum(row[field] for row in sources)result['metered_' + field] = sum(row[field] for row in sources if row['metered'])return resultdiff --git a/src/peach/processing_traffic_plan.py b/src/peach/processing_traffic_plan.pynew file mode 100644index 000000000..78fc502be--- /dev/null+++ b/src/peach/processing_traffic_plan.py@@ -0,0 +1,207 @@+"""从本地账本、资料缓存与下载预算规划待执行的流量。"""+from __future__ import annotations++from itertools import islice+import os+import sqlite3+from statistics import median+from contextlib import closing++MIB = 1024 ** 2+++def cached_names(directory):+ try:+ with os.scandir(directory) as entries:+ return {entry.name.casefold() for entry in entries}+ except OSError:+ return set()+++def ledger_rows(db_path, asset_ids=None):+ query = """SELECT asset.*,+ EXISTS(SELECT 1 FROM asset_entity ae JOIN entity e ON ae.entity_id=e.id+ WHERE ae.asset_id=asset.id AND e.kind='performer') AS performers,+ EXISTS(SELECT 1 FROM asset_tag t WHERE t.asset_id=asset.id AND t.tag NOT LIKE '演员:%') AS tags+ FROM asset WHERE medium='video' AND disposal IS NULL"""+ parameters = []+ if asset_ids is not None:+ query += ' AND id IN (' + ','.join('?' for _ in asset_ids) + ')'+ parameters = asset_ids+ with closing(sqlite3.connect(db_path, timeout=30)) as connection:+ connection.row_factory = sqlite3.Row+ return [dict(row) for row in connection.execute(query, parameters)]+++def typical_size(directory, pattern, fallback):+ """只在本机缓存抽取至多 256 个文件大小,不访问媒体挂载。"""+ sizes = []+ for path in islice(directory.glob(pattern), 256):+ try:+ size = path.stat().st_size+ if size > 0:+ sizes.append(size)+ except OSError:+ pass+ return int(median(sizes)) if sizes else fallback+++class TrafficPlan:+ def __init__(self, config, stage, media_key):+ from .library_processing import _MissCache, _RemoteSession, misses_path+ self.config, self.stage, self.media_key = config, stage, media_key+ self.remote = _RemoteSession(config, None, _MissCache(misses_path(config)), retrying=False)+ self.cover_root = config.directory('generated') / 'covers'+ self.cover_bytes = typical_size(self.cover_root, '*.jpg', MIB // 2)+ self.nfo_bytes = typical_size(config.directory('sources') / 'library-metadata', '*.nfo', 32 * 1024)+ self.units = {}+ self.outside_budget = {}+ self.completed = set()+ self.cover_ready = {}+ self.cover_names = cached_names(self.cover_root)+ self.evidence_names = cached_names(config.directory('sources') / 'library-metadata')+ self.proxy = {}++ def _add(self, token, key, amount, *, budgeted=True):+ if amount <= 0 or token in self.completed or token.partition(':')[0] + ':*' in self.completed:+ return+ unit = self.units.setdefault(token, {})+ unit[key] = unit.get(key, 0) + int(amount)+ if not budgeted:+ outside = self.outside_budget.setdefault(token, {})+ outside[key] = outside.get(key, 0) + int(amount)++ def _ready(self, code):+ from .library_processing import cover_settled+ if code not in self.cover_ready:+ self.cover_ready[code] = ((code + '.jpg').casefold() in self.cover_names+ and cover_settled(self.cover_root / (code + '.jpg')))+ return self.cover_ready[code]++ def _network(self, source):+ from . import peach_proxy+ from .scraping_access import values_for+ if source not in self.proxy:+ root = self.config.directory('secrets')+ if values_for(root, source).get('network') == 'direct':+ self.proxy[source] = False+ else:+ try:+ options = peach_proxy.client_options(root)+ self.proxy[source] = bool(options.get('proxy') or options.get('trust_env', True))+ except ValueError:+ self.proxy[source] = True+ return 'metadata-proxy' if self.proxy[source] else 'metadata-direct'++ def refresh(self, rows):+ from .library_processing import _provider_code, _scrapes_as_jav, _missing_fields+ from .catalog_rules import release_code_from_filename+ self.assets = list(rows)+ self.units = {}+ self.outside_budget = {}+ titles = {}+ for row in self.assets:+ if self.stage == 'previews' and row['location'] != 'local':+ continue+ key = self.media_key(row['path'], row['location'])+ if key is None:+ continue+ code = _provider_code(row.get('code')) or release_code_from_filename(row.get('name') or '')+ eligible = bool(code and _scrapes_as_jav(row, code))+ path = str(row['path'])+ if self.stage not in ('scan', 'previews'):+ # 无发行番号的创作者文件只检查配套文件,不进入 JAV 下载链。+ if eligible:+ self._add('local:' + path, key, self.nfo_bytes)+ if not self._ready(code):+ self._add('local:' + path, key, self.cover_bytes)+ title = titles.setdefault(code, dict(row=row, missing=set()))+ title['missing'].update(_missing_fields(row))+ self._media(row, key)+ for code, title in titles.items():+ self._downloads(code, title)++ def _media(self, row, key):+ from .media_processing_policy import metadata_only+ from .media_sources import cost_for+ from .strm import skip_background_read+ path = str(row['path'])+ if skip_background_read(path):+ return+ size = max(0, row.get('size') or 0)+ if (self.stage != 'metadata' and row.get('duration') is None+ and cost_for(row['location'], path, self.config) == 'free'):+ self._add('media:' + path, key, min(size, 8 * MIB))+ if (self.stage in ('all', 'collect', 'previews') and row['location'] == 'local'+ and not row.get('snapshot_path') and not metadata_only(self.config.directory('state'))):+ self._add('media:' + path, key, min(size, 9 * 8 * MIB))++ def _downloads(self, code, title):+ from .library_processing import _studio_evidence+ from .metadata_routes import stage_name+ chain = self.remote._chain(code, *_studio_evidence(title['row']))+ askable = self.remote._askable(chain, code)+ names = [source for source in askable if f'{code}-{source}.json'.casefold() in self.evidence_names]+ cached = {source for source, _, _ in self.remote._cached_evidence(names, code)}+ pending = [source for source in askable if source not in cached]+ if title['missing']:+ for source in pending:+ # 每站一至两次请求,沿用采集器约 250KB/次的工作量基准。+ self._add('metadata:' + code, self._network(source), 2 * 250 * 1024,+ budgeted=stage_name(source) not in ('amane', 'amane_official'))+ if (not self._ready(code) and chain+ and not self.remote._fresh_miss('cover', code, chain)):+ self._add('metadata:' + code, self._network(chain[0]), self.cover_bytes)++ def prepare_metadata(self, prepared):+ """配套文件读完后,使用任务实际取得的字段和核查范围。"""+ from .library_processing import COLLECTED_FIELDS, _missing_fields, _scrapes_as_jav+ for token in list(self.units):+ if token.startswith('metadata:'):+ self.units.pop(token)+ self.outside_budget.pop(token, None)+ titles = {}+ for row, code, verify, local_fields, *_ in prepared:+ if code and _scrapes_as_jav(row, code):+ title = titles.setdefault(code, dict(row=row, missing=set()))+ title['missing'].update(COLLECTED_FIELDS if verify else _missing_fields(row, local_fields))+ for code, title in titles.items():+ self._downloads(code, title)++ def finish(self, stage, code=None):+ from .library_processing import _provider_code+ if stage == 'metadata':+ release = _provider_code(code)+ if not release:+ return+ tokens = ['metadata:' + release]+ self.completed.update(tokens)+ else:+ tokens = [token for token in self.units if token.startswith(stage + ':')]+ self.completed.add(stage + ':*')+ for token in tokens:+ self.units.pop(token, None)+ self.outside_budget.pop(token, None)+ if stage == 'local':+ self.cover_ready.clear()+ self.cover_names = cached_names(self.cover_root)+ self.refresh(self.assets)++ def remaining(self, rows):+ from .library_processing import MAX_SOURCE_BYTES+ planned = dict.fromkeys(rows, 0)+ for unit in self.units.values():+ for key, amount in unit.items():+ if rows[key]['network']:+ planned[key] += amount+ download_keys = [key for key, row in rows.items() if not row['root']]+ outside = {key: sum(unit.get(key, 0) for unit in self.outside_budget.values()) for key in download_keys}+ wanted = sum(planned[key] - outside[key] for key in download_keys)+ budget = max(0, MAX_SOURCE_BYTES - sum(rows[key]['used_bytes'] for key in download_keys))+ if wanted > budget:+ remaining = budget+ for key in download_keys:+ value = min(remaining, budget * (planned[key] - outside[key]) // wanted)+ planned[key] = outside[key] + value+ remaining -= value+ return planneddiff --git a/tests/test_processing_traffic.py b/tests/test_processing_traffic.pyindex e524ab9b0..a2052819f 100644--- a/tests/test_processing_traffic.py+++ b/tests/test_processing_traffic.py@@ -1,5 +1,6 @@"""馆藏任务按媒体目录的流量保护选项分账。"""import gzip+import jsonimport sqlite3import subprocessimport tempfile@@ -136,11 +137,110 @@ class ProcessingTrafficTests(unittest.TestCase):traffic = ProcessingTraffic(self.config, self.db, 'scan')rows = [dict(location='mounted', path='B:\\Cloud\\a.mp4', size=1024**3,duration=None, snapshot_path=None)]- with patch('peach.processing_traffic.sqlite3.connect', side_effect=AssertionError('重复查询')):+ with patch('peach.processing_traffic_plan.sqlite3.connect', side_effect=AssertionError('重复查询')):traffic.refresh_plan(rows)self.assertGreater(traffic.snapshot()['estimated_max_bytes'], 0)self.assertEqual(traffic.snapshot()['metered_estimated_max_bytes'], 0)+ def _video(self, **values):+ return dict(id=1, location='mounted', path='A:\\Cloud\\ABW-001.mp4', name='ABW-001.mp4',+ code='ABW-001', size=1024**3, duration=3600, snapshot_path=None, **values)++ def test_download_plan_deduplicates_releases_and_ignores_creator_files(self):+ traffic = ProcessingTraffic(self.config, self.db, 'collect')+ with patch.object(traffic.plan, '_ready', return_value=False):+ traffic.refresh_plan([self._video()])+ single = sum(row['estimated_bytes'] for row in traffic.snapshot()['sources'] if not row['root'])+ rows = [{**self._video(), 'id': index, 'path': f'A:\\Cloud\\copy{index}.mp4'} for index in range(20)]+ traffic.refresh_plan(rows)+ repeated = sum(row['estimated_bytes'] for row in traffic.snapshot()['sources'] if not row['root'])+ self.assertGreater(single, 0)+ self.assertEqual(single, repeated)+ traffic.refresh_plan([{**self._video(), 'code': None, 'name': '创作者作品.mp4'}])+ self.assertEqual(sum(row['estimated_bytes'] for row in traffic.snapshot()['sources'] if not row['root']), 0)++ def test_complete_fields_and_local_cover_have_no_download_work(self):+ traffic = ProcessingTraffic(self.config, self.db, 'collect')+ row = self._video(catalog_title='影片', performers='演员', studio='厂牌', release_date='2026-01-01', tags='类别')+ with patch.object(traffic.plan, '_ready', return_value=True):+ traffic.refresh_plan([row])+ self.assertEqual(sum(source['estimated_bytes'] for source in traffic.snapshot()['sources'] if not source['root']), 0)++ def test_local_evidence_recalculates_actual_metadata_verification_work(self):+ traffic = ProcessingTraffic(self.config, self.db, 'collect')+ row = self._video()+ fields = dict(title='影片', performers='演员', studio='厂牌', release_date='2026-01-01', tags='类别')+ with patch.object(traffic.plan, '_ready', return_value=True):+ traffic.prepare_metadata([(row, 'ABW-001', False, fields, [], True)])+ self.assertEqual(sum(source['estimated_bytes'] for source in traffic.snapshot()['sources'] if not source['root']), 0)+ traffic.prepare_metadata([(row, 'ABW-001', True, fields, [], True)])+ self.assertGreater(sum(source['estimated_bytes'] for source in traffic.snapshot()['sources'] if not source['root']), 0)++ def test_fresh_source_cache_and_misses_do_not_plan_network_requests(self):+ directory = self.config.directory('sources') / 'library-metadata'+ directory.mkdir(parents=True)+ for source in ('r18dev', 'javdb'):+ (directory / f'ABW-001-{source}.json').write_text(json.dumps({'title': '影片'}), encoding='utf-8')+ traffic = ProcessingTraffic(self.config, self.db, 'collect')+ with patch.object(traffic.plan.remote, '_chain', return_value=('r18dev', 'javdb')), patch.object(traffic.plan, '_ready', return_value=True):+ traffic.refresh_plan([self._video()])+ self.assertEqual(sum(row['estimated_bytes'] for row in traffic.snapshot()['sources'] if not row['root']), 0)+ traffic.plan.evidence_names.clear()+ with patch.object(traffic.plan.remote, '_chain', return_value=('r18dev', 'javdb')), patch.object(traffic.plan.remote.misses, 'fresh', return_value=True), patch.object(traffic.plan, '_ready', return_value=True):+ traffic.refresh_plan([self._video()])+ self.assertEqual(sum(row['estimated_bytes'] for row in traffic.snapshot()['sources'] if not row['root']), 0)++ def test_download_budget_is_shared_and_finished_work_leaves_used_bytes(self):+ from peach.library_processing import MAX_SOURCE_BYTES+ traffic = ProcessingTraffic(self.config, self.db, 'collect')+ traffic.plan.units = {'metadata:ABW-001': {'metadata-proxy': MAX_SOURCE_BYTES * 3, 'metadata-direct': MAX_SOURCE_BYTES * 2}}+ result = traffic.snapshot()+ self.assertLessEqual(result['estimated_bytes'], MAX_SOURCE_BYTES)+ self.assertEqual(result['estimated_bytes'], sum(row['estimated_bytes'] for row in result['sources']))+ self.assertEqual(result['metered_estimated_bytes'], sum(row['estimated_bytes'] for row in result['sources'] if row['metered']))+ with traffic.bind(), download_route(True):+ record_download(100)+ traffic.finish_work('metadata', 'ABW-001')+ self.assertEqual(traffic.snapshot()['estimated_bytes'], 100)+ restored = ProcessingTraffic(self.config, self.db, 'collect', resume_state={'traffic': traffic.snapshot()})+ restored.refresh_plan([self._video()])+ self.assertEqual(sum(row['estimated_bytes'] for row in restored.snapshot()['sources'] if not row['root']), 100)++ def test_source_network_choice_splits_pending_downloads(self):+ traffic = ProcessingTraffic(self.config, self.db, 'collect')+ with patch.object(traffic.plan.remote, '_chain', return_value=('r18dev', 'javdb')), patch.object(traffic.plan, '_ready', return_value=True), patch('peach.scraping_access.values_for', side_effect=lambda root, source: {'network': 'direct' if source == 'javdb' else 'peach'}), patch('peach.peach_proxy.client_options', return_value={'proxy': 'http://example.invalid', 'trust_env': False}):+ traffic.refresh_plan([self._video()])+ sources = {row['key']: row for row in traffic.snapshot()['sources']}+ self.assertEqual(sources['metadata-direct']['estimated_bytes'], 500 * 1024)+ self.assertEqual(sources['metadata-proxy']['estimated_bytes'], 500 * 1024)++ def test_external_collector_plan_is_separate_from_http_download_budget(self):+ from peach.library_processing import MAX_SOURCE_BYTES+ traffic = ProcessingTraffic(self.config, self.db, 'collect')+ traffic.plan._add('metadata:ABW-001', 'metadata-proxy', MAX_SOURCE_BYTES * 2)+ traffic.plan._add('metadata:ABW-001', 'metadata-proxy', 1000, budgeted=False)+ self.assertEqual(traffic.snapshot()['estimated_bytes'], MAX_SOURCE_BYTES + 1000)++ def test_non_release_metadata_completion_leaves_other_releases_pending(self):+ traffic = ProcessingTraffic(self.config, self.db, 'collect')+ traffic.plan._add('metadata:ABW-001', 'metadata-proxy', 1000)+ traffic.finish_work('metadata', None)+ self.assertEqual(traffic.snapshot()['estimated_bytes'], 1000)++ def test_local_cache_file_sizes_inform_sidecar_plan_and_local_stage_settles(self):+ evidence = self.config.directory('sources') / 'library-metadata'+ evidence.mkdir(parents=True)+ (evidence / 'sample.nfo').write_bytes(b'x' * 1234)+ traffic = ProcessingTraffic(self.config, self.db, 'collect')+ with patch.object(traffic.plan, '_ready', return_value=True):+ traffic.refresh_plan([self._video()])+ root = next(row for row in traffic.snapshot()['sources'] if row['root'] == 'A:\\Cloud')+ self.assertEqual(root['estimated_bytes'], 1234)+ traffic.finish_work('local')+ root = next(row for row in traffic.snapshot()['sources'] if row['root'] == 'A:\\Cloud')+ self.assertEqual(root['estimated_bytes'], 0)+ self.assertEqual(traffic.snapshot()['completed_work'], ['local:*'])+def test_real_ffmpeg_probe_and_frame_report_input_reads(self):from peach.ffmpeg import FFmpegResolverfrom peach.frame_capture import capture_frame