Peach: Local-first video metadata scraper and media manager.
fix(follow): 修复快照冲突并补齐来源故障取证
longmeidao631f4bc377f7
diff --git a/docs/DIAGNOSTICS.md b/docs/DIAGNOSTICS.mdindex 0f574a733..93fea7160 100644--- a/docs/DIAGNOSTICS.md+++ b/docs/DIAGNOSTICS.md@@ -20,6 +20,7 @@| 抽帧、拼图、探测 | 保留退出码、超时、采样位置、时长、工具来源、色彩重试和有界 stderr;`media-probe`、`preview`、`frame-capture`、`frame-tile` 与 `library-preview` 覆盖各入口 || 图片有效性 | Pillow 完整解码确认图片;纯色小 JPEG 可用,残片、非零退出和陈旧输出不能算生成成功 || 外部来源、下载 | 来源授权、限流、冷却与未命中使用领域状态;意外失败使用现有异常日志和任务回执;ZIP 包含来源健康状态与时间,来源响应原文留在本机 |+| 关注检查、媒体代理 | `follow-check` 与 `follow-media` 本机事件记录来源、来源及条目编号、失败阶段、操作和可用的上游状态码;媒体事件区分帖子来源与资源来源,异常链与代码位置随支持包导出。地址、凭据和私人路径统一脱敏,不记录上游响应正文,不扩大远程诊断字段 || 更新与回滚 | 安装、回滚失败保留 traceback;ZIP 包含启动准备、更新状态、源码更新日志、托盘重启日志及近期迁移日志 || 隧道 | ZIP 包含 `cloudflare-tunnel.log` 与脱敏运行状态 || 浏览器 | 保存消息、堆栈、脚本文件名、行列、页面与浏览器标识;查询参数不进入页面字段;同源、有效登录、每分钟条数和请求体积门槛有回归 |diff --git a/docs/REUSE.md b/docs/REUSE.mdindex d2a20cc56..18023296a 100644--- a/docs/REUSE.md+++ b/docs/REUSE.md@@ -665,6 +665,10 @@ A 盘垃圾隔离使用短名称 `peach-purge-<32 位 UUID>.peach-quarantine`,- 关注的作者头像与来源图标是元数据:`follow_assets` 取回落在 `generated/follow-assets/`,经 `/follow-avatar`、`/source-icon` 给页面;地址只从固定表或固定主机拼,字节先认成图再写盘,到期重取失败继续用旧的并退避一小时,缓存有效期与 `/link-mark` 共用 `web_settings.metadata_refresh_seconds`。视频与图片不存本机。官方头像先认 FANBOX,没有时 `follow_avatar.profile_avatar_tiers` 取名片上的 X 与 Patreon,`follow_assets.largest_image` 留像素最大的那张。- 厂牌标识由契约位 `has_logo` 决定出不出图:没装标识的厂牌一个 `<img>` 都不发,改用首字母底板,不靠 404 摘。标识 198 张随仓库分发(ADR-0026)。- 关注检查分两阶段:列表阶段落 partial 行,详情补全按 provider 额度只补新行和未补齐行。「补齐过」由连接器的 `ENRICHED_MARK` 声明、判据登记在 `follow_store._ENRICHED_PREDICATES`。+- 关注原始证据复用 Python 标准库 `hashlib.sha256` 与既有 `write_immutable`。快照文件名由检查时间和完整说明的摘要组成;说明含正文 SHA-256、脱敏请求地址及候选数量,因此相同正文的不同游标或筛选结果各有独立证据。相同快照可重复保存,已有文件内容冲突仍拒绝覆盖;历史证据按账本保存的相对路径读取,无迁移或新增依赖。临时账本回归遍历登记表中的全部追更来源,覆盖正文、地址及候选数量的独立变化,并经 FANBOX 连接器验证相同正文的历史补足和重复检查。+- 单来源失败边界覆盖取回、整页存储和历史补足;证据内容冲突保留错误,已提交的条目继续保留,其他来源继续检查。Web 与 CLI 共用 `follow_check.run_check` 的失败结果,不把证据冲突记成成功。+- 关注检查失败在本机 `follow-check` 事件中保留来源、来源编号、失败阶段、请求页及异常链;只记录登记来源键与技术状态,不复制作者名称或来源地址。单来源错误仍按既有任务结果返回。+- 关注媒体的本机诊断复用 `error_reporting.event` 与支持包脱敏入口,分别记录来源、资源来源、来源及条目编号、播放/图片/清晰度/大小操作、解析或回源阶段和上游状态码。上游响应正文不进入诊断字段;异常中的地址和凭据由统一入口脱敏,远程诊断字段范围保持既有约定。隔离 API 和支持包回归覆盖 403、429、异常链、无效媒体清单及脱敏保真,无新增依赖。- 存量行重抓(连接器新学到图片宽高、封面这类字段时,旧行不会自己补上)分两步:把 `ENRICHED_MARK` 换成新字段落库后才有的键并登记判据;再 `POST /api/follow/check` 发 `{"older":true,"backfill_all":true,"rewind":true,"background":true}`(可加 `"sources":[id,…]`),从第 1 页走到站点说没有更多,进度读 `GET /api/follow/check`。`backfill_all`、`rewind` 离开 `older` 被拒收、不进界面,ledger 的回填游标只进不退;长跑任务,重启前按 `peach-batch-jobs` 先查。- 图片的固有宽高只问文件头:`follow_image_dims.probe_image_dims` 发 `Range: bytes=0-65535`,`dims_from_header` 认 PNG/GIF/WebP(VP8、VP8L、VP8X)/JPEG(跳过 EXIF 到 SOF);连接器、`backfill_follow_image_dims.py` 与 `/api/follow/image-dims` 落库前都经 `positive_dims` 归一,`FollowStore.set_image_dims` 只补空缺。宽高只为图片视图的卡面预留比例,回填每个条目只问卡面那张;原文件主机拦脚本(pawchive 的 `file.` 子域挂 ddos-guard)时按缩略图量,详情原图取不到也退回缩略图。- 归档站(kemono、coomer、pawchive)多图或视频的帖子由 `KemonoConnector._media_items` 列进 `media_items`(交付文件排第一、按路径去重),详情轮播与 `/follow-stream?media=N` 按条目自己站点的主机白名单取。diff --git a/src/peach/follow_check.py b/src/peach/follow_check.pyindex 24918082a..9022b3201 100644--- a/src/peach/follow_check.py+++ b/src/peach/follow_check.py@@ -14,7 +14,7 @@ from dataclasses import dataclass, replacefrom datetime import datetime, timedelta, timezonefrom typing import Mapping-from . import follow_providers+from . import error_reporting, follow_providersfrom .follow import FollowHistoryEnd, FollowSourceError, FollowSourceRateLimitedfrom .follow_secrets import CredentialErrorfrom .follow_sources import SourceFetch, enrichment_mark, history_window, published_stamp@@ -126,6 +126,7 @@ def run_check(row: Mapping, *, credentials, writer, connector_factory,cutoff, floor = _history_limits(row, metadata, writer, moment,older=older, initial_days=initial_days)recorder = _PageRecorder(row, writer, moment, cutoff)+ phase = "fetch"try:connector = build_connector_for(provider, credentials, connector_factory,@@ -142,33 +143,36 @@ def run_check(row: Mapping, *, credentials, writer, connector_factory,last_modified=None if force else row["last_modified"],page=page,)+ candidates = history_window(fetch.candidates, cutoff, floor)+ history_skipped = getattr(connector, 'history_skipped', 0) + len(fetch.candidates) - len(candidates)+ fetch = replace(fetch, candidates=candidates)+ phase = "record"+ with writer() as store:+ if not fetch.not_modified:+ if replay_first:+ store.merge_source_metadata(source_id, {'initial_history_first_page_pending': False}, moment)+ elif history_skipped and page == 0 and 'initial_history_first_page_pending' not in metadata:+ store.merge_source_metadata(source_id, {'initial_history_first_page_pending': True}, moment)+ outcome = recorder.record_in(store, fetch, page)+ if floor and not fetch.not_modified:+ phase = "history"+ recorder.cutoff = None+ fetch, outcome = _fill_floor(row, connector, writer, fetch, outcome, cutoff, floor, moment,+ recorder=recorder)+ with writer() as store:+ phase = "identity"+ learned = store.learn_official_author_alias(provider, ref, fetch.candidates)+ return CheckResult(**base, fetch=fetch, outcome=outcome,+ author_alias_learned=learned, history_skipped=history_skipped)except FollowHistoryEnd:with writer() as store:store.record_history_end(source_id, moment)return CheckResult(**base, exhausted=True, message="没有更多历史内容")except CredentialError as error:- return _record_failure(writer, base, error, moment, "unauthorized")+ return _record_failure(writer, base, error, moment, "unauthorized", phase=phase)except FollowSourceError as error:_pause_if_rate_limited(cooldown, provider, error)- return _record_failure(writer, base, error, moment, "error")- candidates = history_window(fetch.candidates, cutoff, floor)- history_skipped = getattr(connector, 'history_skipped', 0) + len(fetch.candidates) - len(candidates)- fetch = replace(fetch, candidates=candidates)- with writer() as store:- if not fetch.not_modified:- if replay_first:- store.merge_source_metadata(source_id, {'initial_history_first_page_pending': False}, moment)- elif history_skipped and page == 0 and 'initial_history_first_page_pending' not in metadata:- store.merge_source_metadata(source_id, {'initial_history_first_page_pending': True}, moment)- outcome = recorder.record_in(store, fetch, page)- if floor and not fetch.not_modified:- recorder.cutoff = None- fetch, outcome = _fill_floor(row, connector, writer, fetch, outcome, cutoff, floor, moment,- recorder=recorder)- with writer() as store:- learned = store.learn_official_author_alias(provider, ref, fetch.candidates)- return CheckResult(**base, fetch=fetch, outcome=outcome,- author_alias_learned=learned, history_skipped=history_skipped)+ return _record_failure(writer, base, error, moment, "error", phase=phase)def _pause_if_rate_limited(cooldown, provider: str, error: FollowSourceError) -> None:@@ -237,9 +241,12 @@ def _fill_floor(row, connector, writer, fetch, outcome, cutoff, floor, moment, *def _record_failure(writer, base: dict, error: Exception, moment: datetime,- status: str) -> CheckResult:+ status: str, *, phase: str) -> CheckResult:with writer() as store:store.record_error(base["source_id"], str(error), moment, status=status)+ error_reporting.event("follow-check", error, provider=base["provider"],+ source_id=base["source_id"], phase=phase,+ page=base["page"], older=base["older"], status=status)return CheckResult(**base, ok=False, status=status, error=str(error))diff --git a/src/peach/follow_store.py b/src/peach/follow_store.pyindex 8d17af366..c5bbaff90 100644--- a/src/peach/follow_store.py+++ b/src/peach/follow_store.py@@ -641,16 +641,19 @@ class FollowStore:key = hashlib.sha256(f"{fetch.provider}\n{fetch.ref}".encode()).hexdigest()[:20]directory = self.sources_root / fetch.provider / keystamp = reference.astimezone(timezone.utc).strftime("%Y%m%dT%H%M%S.%fZ")- path = directory / f"{stamp}-{digest[:12]}.raw"+ sidecar = (json.dumps({+ "provider": fetch.provider, "ref": fetch.ref,+ # 脱敏后的请求 URL:凭据永不进证据目录。+ "request_url": fetch.request_url, "sha256": digest,+ "checked_at": _now_text(reference), "candidates": len(fetch.candidates),+ }, ensure_ascii=False, indent=2) + "\n").encode("utf-8")+ # 快照身份包含正文摘要与说明;相同正文可来自不同游标或候选筛选。+ snapshot_digest = hashlib.sha256(sidecar).hexdigest()+ path = directory / f"{stamp}-{snapshot_digest}.raw"try:directory.mkdir(parents=True, exist_ok=True)write_immutable(path, fetch.raw_body)- write_immutable(path.with_suffix(".json"), (json.dumps({- "provider": fetch.provider, "ref": fetch.ref,- # 脱敏后的请求 URL:凭据永不进证据目录。- "request_url": fetch.request_url, "sha256": digest,- "checked_at": _now_text(reference), "candidates": len(fetch.candidates),- }, ensure_ascii=False, indent=2) + "\n").encode("utf-8"))+ write_immutable(path.with_suffix(".json"), sidecar)except OSError as error:return None, f"证据未取得({self.sources_root} 不可写:{error.strerror or error})"return str(path.relative_to(self.sources_root.parent)), Nonediff --git a/src/peach/follow_stream.py b/src/peach/follow_stream.pyindex dba34f071..e0986fba8 100644--- a/src/peach/follow_stream.py+++ b/src/peach/follow_stream.py@@ -25,7 +25,9 @@ from .http import HttpRequest, HttpResponse, HttpTransport, public_https_url, reLOGGER = logging.getLogger(__name__)class FollowMediaUnavailable(RuntimeError):- pass+ def __init__(self, message: str = "", *, status_code: int | None = None):+ super().__init__(message)+ self.status_code = status_codeclass FollowProxyError(RuntimeError):@@ -35,6 +37,10 @@ class FollowProxyError(RuntimeError):媒体是 404,上游这一次没给出可转发的响应是 502。"""+ def __init__(self, message: str, *, status_code: int | None = None):+ super().__init__(message)+ self.status_code = status_code+@dataclass(frozen=True)class ResolvedFollowMedia:@@ -337,7 +343,7 @@ class FollowMediaResolver:LOGGER.warning("rule34video detail page for follow item %s unavailable: HTTP %s",item.id, response.status)self._remember_miss(item.id)- raise FollowMediaUnavailable("Rule34Video 详情页未取得")+ raise FollowMediaUnavailable("Rule34Video 详情页未取得", status_code=response.status)text = response.body.decode("utf-8", errors="replace")found: list[str] = []for matched in _VIDEO_URL_RE.finditer(text):@@ -443,5 +449,5 @@ def open_upstream(client: httpx.Client, method: str, target: ResolvedFollowMediaif not 200 <= response.status_code < 300 and response.status_code not in accepted_statuses:status = response.status_coderesponse.close()- raise FollowProxyError(f"上游返回 HTTP {status}")+ raise FollowProxyError(f"上游返回 HTTP {status}", status_code=status)return responsediff --git a/src/peach/routes_media.py b/src/peach/routes_media.pyindex f176ea6cb..d311324d2 100644--- a/src/peach/routes_media.py+++ b/src/peach/routes_media.py@@ -32,7 +32,8 @@ from starlette.staticfiles import StaticFilesfrom starlette.requests import ClientDisconnectfrom . import (- avatar_face, avatar_online, avatar_picker, avatar_provider, follow_assets, follow_providers, images,+ avatar_face, avatar_online, avatar_picker, avatar_provider, error_reporting,+ follow_assets, follow_providers, images,jav_poster_crop, link_marks, link_status, scraping_access, site_icons, social_links, subtitles,taste_history, timeline_sheets, web_follow,)@@ -937,7 +938,25 @@ def work_icon(request: Request, work: str = "",return _asset_response(request, path)-def _follow_media_size(state, item_id: int, target: ResolvedFollowMedia) -> int | None:+def _report_follow_media_failure(item, error: Exception, *, operation: str, phase: str,+ media_index: int | None = None) -> None:+ """媒体失败按来源、资源来源和操作定位;地址与凭据交给统一脱敏入口。"""+ rows = web_follow._raw_media_items(item)+ index = media_index if media_index is not None else 0+ resource_provider = item.provider+ if ((operation != "image" or media_index is not None) and 0 <= index < len(rows)+ and isinstance(rows[index], dict)):+ resource_provider = str(rows[index].get("resource_provider") or item.provider)+ if resource_provider not in follow_providers.PROVIDERS:+ resource_provider = "unknown"+ error_reporting.event(+ "follow-media", error, provider=item.provider, source_id=getattr(item, "source_id", None),+ item_id=item.id, resource_provider=resource_provider, operation=operation,+ phase=phase, media_index=media_index, upstream_status=getattr(error, "status_code", None),+ )+++def _follow_media_size(state, item, target: ResolvedFollowMedia) -> int | None:"""正片的字节数,取不到就是 None。只能问上游:来源列表里没有这一项,账本里也只有落盘资源才有 size。一次 HEAD 就够,@@ -950,7 +969,7 @@ def _follow_media_size(state, item_id: int, target: ResolvedFollowMedia) -> inttry:response = open_upstream(state.http_transport.client, "HEAD", target, incoming={})except (FollowProxyError, OSError, httpx.HTTPError) as error:- LOGGER.warning("follow media size unavailable for item %s: %s", item_id, error)+ _report_follow_media_failure(item, error, operation="size", phase="upstream")return Nonetry:size = int(response.headers.get("content-length") or 0)@@ -984,7 +1003,8 @@ def follow_qualities(request: Request, id: int, args: dict[str, str] = Depends(rreturn JSONResponse({"error": "no such follow item"}, status_code=404)try:resolved = state.follow_media_resolver.resolve(item)- except FollowMediaUnavailable:+ except FollowMediaUnavailable as error:+ _report_follow_media_failure(item, error, operation="qualities", phase="resolve")return {"qualities": [], "size": None}# 档位数量不写死:正则匹配 video_url 与任意编号的 video_alt_urlN,# 站点给几档就是几档(实测同一作者下有 4 档也有 5 档的条目)。@@ -992,7 +1012,7 @@ def follow_qualities(request: Request, id: int, args: dict[str, str] = Depends(rreturn {"qualities": [{"height": height, "label": "4K" if height >= 2160 else f"{height}p"}for height, _ in resolved.qualities if height- ], "size": _follow_media_size(state, id, resolved)}+ ], "size": _follow_media_size(state, item, resolved)}@router.api_route("/follow-stream", methods=["GET", "HEAD"])@@ -1005,6 +1025,7 @@ def follow_stream(request: Request, id: int, media: int | None = None,item = FollowStore(lambda: connection).item(id)if item is None:return JSONResponse({"error": "no such follow item"}, status_code=404)+ phase = "resolve"try:key = f"follow:{item.id}:media:{media}:quality:{quality}"cache = cache_for(state.web_contract)@@ -1013,6 +1034,7 @@ def follow_stream(request: Request, id: int, media: int | None = None,if cached is not None:return cachedtarget = state.follow_media_resolver.resolve(item, media, quality)+ phase = "upstream"# 跟重定向、逐跳过白名单、非 2xx 一律不转发,都在 follow_stream 里;# 这里只负责把 Peach 的请求接到那个流上。extra = {"content-disposition": _attachment_disposition(item.title, target.url)} if download else None@@ -1020,14 +1042,16 @@ def follow_stream(request: Request, id: int, media: int | None = None,lambda: open_upstream(state.http_transport.client, request.method, target, incoming=request.headers),headers=extra)except FollowMediaUnavailable as error:+ _report_follow_media_failure(item, error, operation="stream", phase=phase, media_index=media)return JSONResponse({"error": str(error)}, status_code=404)except FollowProxyError as error:# 上游的状态码与正文不转发:403 页面、限流提示、错误 JSON 交给播放器# 没有用处,还会把上游主机和提示语抄给浏览器。对外统一是 502。- LOGGER.warning("follow media proxy refused for item %s: %s", id, error)+ _report_follow_media_failure(item, error, operation="stream", phase=phase, media_index=media)return JSONResponse({"error": "follow media unavailable"}, status_code=502)- except (OSError, httpx.HTTPError):- LOGGER.exception("follow media proxy failed for item %s", id)+ except (OSError, httpx.HTTPError) as error:+ LOGGER.exception("follow media proxy failed for provider %s item %s", item.provider, id)+ _report_follow_media_failure(item, error, operation="stream", phase=phase, media_index=media)return JSONResponse({"error": "follow media unavailable"}, status_code=502)@router.api_route("/follow-image", methods=["GET", "HEAD"])@@ -1039,6 +1063,7 @@ def follow_image(request: Request, id: int, media: int | None = None,if item is None:return Response(status_code=404)cache = cache_for(state.web_contract)+ phase = "resolve"try:if media is None:url = web_follow._remote_thumb_url(item)@@ -1060,9 +1085,11 @@ def follow_image(request: Request, id: int, media: int | None = None,resolved = state.follow_media_resolver.resolve(item, media)if resolved.headers:target = replace(resolved, url=url)+ phase = "upstream"return cache.stream(key, request, lambda: open_upstream(state.http_transport.client, request.method, target, incoming=request.headers))- except (ValueError, FollowMediaUnavailable, FollowProxyError, OSError, httpx.HTTPError):+ except (ValueError, FollowMediaUnavailable, FollowProxyError, OSError, httpx.HTTPError) as error:+ _report_follow_media_failure(item, error, operation="image", phase=phase, media_index=media)return Response(status_code=404, headers={"cache-control": "no-store"})diff --git a/tests/test_error_reporting.py b/tests/test_error_reporting.pyindex e77c97504..aa33c5e87 100644--- a/tests/test_error_reporting.py+++ b/tests/test_error_reporting.py@@ -319,6 +319,53 @@ class ErrorReportingTests(unittest.TestCase):for secret in ("private-cookie", "private-cid", "private-token"):self.assertNotIn(secret, content)+ def test_follow_media_report_keeps_source_context_and_redacts_upstream_details(self):+ from types import SimpleNamespace+ from peach.follow_stream import FollowProxyError+ from peach.routes_media import _report_follow_media_failure++ item = SimpleNamespace(provider="fanbox", source_id=3, id=7,+ metadata={"media_items": [{"resource_provider": "gofile"}]})+ try:+ raise FollowProxyError(+ "upstream https://media.gofile.io/private-file?token=private-token "+ "Cookie: private-cookie", status_code=403)+ except FollowProxyError as error:+ _report_follow_media_failure(item, error, operation="stream", phase="upstream",+ media_index=0)+ with zipfile.ZipFile(io.BytesIO(support_bundle.bundle_bytes())) as archive:+ payload = "\n".join(archive.read(name).decode() for name in archive.namelist()+ if "errors-" in name and name.endswith(".log"))+ row = next(json.loads(line) for line in payload.splitlines()+ if json.loads(line).get("stage") == "follow-media")+ self.assertEqual({key: row[key] for key in (+ "provider", "source_id", "item_id", "resource_provider", "operation", "phase",+ "media_index", "upstream_status")}, {+ "provider": "fanbox", "source_id": 3, "item_id": 7, "resource_provider": "gofile",+ "operation": "stream", "phase": "upstream", "media_index": 0, "upstream_status": 403})+ self.assertEqual(row["exception"][0]["type"], "FollowProxyError")+ self.assertTrue(row["frames"])+ for secret in ("private-file", "private-token", "private-cookie", "media.gofile.io"):+ self.assertNotIn(secret, payload)++ def test_follow_media_report_handles_invalid_media_without_masking_the_failure(self):+ from types import SimpleNamespace+ from peach.routes_media import _report_follow_media_failure++ item = SimpleNamespace(provider="kemono", source_id=4, id=8,+ metadata={"media_items": [None, {"resource_provider": "private-name"}]})+ for index, expected in ((0, "kemono"), (1, "unknown")):+ with self.subTest(index=index):+ with mock.patch.object(error_reporting, "event") as emit:+ _report_follow_media_failure(item, ValueError("媒体条目格式不符"),+ operation="stream", phase="resolve", media_index=index)+ self.assertEqual(emit.call_args.kwargs["resource_provider"], expected)+ del item.source_id+ with mock.patch.object(error_reporting, "event") as emit:+ _report_follow_media_failure(item, ValueError("图片来源不可用"),+ operation="image", phase="resolve")+ self.assertIsNone(emit.call_args.kwargs["source_id"])+def test_failure_dialog_points_to_an_existing_report_and_cli_needs_no_service(self):message = error_reporting.failure_details("tray-handoff", OSError("desktop unavailable"))self.assertIn("诊断报告", message)diff --git a/tests/test_fastapi_api.py b/tests/test_fastapi_api.pyindex 2aeea514c..36c0ec548 100644--- a/tests/test_fastapi_api.py+++ b/tests/test_fastapi_api.py@@ -2461,6 +2461,72 @@ class FastApiContractTests(unittest.IsolatedAsyncioTestCase):self.assertEqual(response.headers["content-range"], "bytes 2-5/10")self.assertNotIn("img.kemono.cr", response.text)+ async def test_follow_media_failures_record_source_phase_and_status(self):+ from peach import error_reporting+ from peach.follow_stream import FollowMediaUnavailable, ResolvedFollowMedia++ with closing(sqlite3.connect(self.db)) as connection, connection:+ connection.executescript((ROOT / "migrations" / "0018_online_follow.sql").read_text(+ encoding="utf-8"))+ connection.execute(+ "INSERT INTO follow_source(id,provider,ref,label,url,semantics,created_at,updated_at)"+ " VALUES(3,'fanbox','creator','Creator','https://creator.fanbox.cc/','work','x','x')")+ connection.execute(+ "INSERT INTO follow_item(id,source_id,external_id,title,url,release_key,metadata_json,"+ "first_seen_at,last_seen_at) VALUES(7,3,'7','Remote',"+ "'https://creator.fanbox.cc/posts/7','remote',?,'x','x')",+ (json.dumps({"media_items": [{"url": "https://media.gofile.io/7.mp4",+ "thumb_url": "https://media.gofile.io/7.jpg",+ "resource_provider": "gofile"}]}),))+ target = ResolvedFollowMedia("https://media.gofile.io/7.mp4", allowed_hosts=("gofile.io",))+ original = self.app.state.http_transport.client+ fake = httpx.Client(transport=httpx.MockTransport(+ lambda request: httpx.Response(403, request=request, content=b"private upstream error")))+ self.app.state.http_transport.client = fake+ try:+ with patch.object(self.app.state.follow_media_resolver, "resolve", return_value=target), \+ patch("peach.follow_stream.resolves_publicly", return_value=True), \+ patch.object(error_reporting, "event") as emit:+ for path, operation, status in (+ ("/follow-stream?id=7&media=0&t=secret", "stream", 502),+ ("/follow-image?id=7&media=0&t=secret", "image", 404),+ ("/follow-qualities?id=7&t=secret", "size", 200),+ ):+ with self.subTest(operation=operation):+ emit.reset_mock()+ response = await self.client.get(path)+ self.assertEqual(response.status_code, status)+ fields = emit.call_args.kwargs+ self.assertEqual(fields["provider"], "fanbox")+ self.assertEqual(fields["resource_provider"], "gofile")+ self.assertEqual(fields["item_id"], 7)+ self.assertEqual(fields["source_id"], 3)+ self.assertEqual(fields["operation"], operation)+ self.assertEqual(fields["phase"], "upstream")+ self.assertEqual(fields["upstream_status"], 403)+ self.assertNotIn("private upstream error", response.text)+ with patch.object(self.app.state.follow_media_resolver, "resolve",+ side_effect=FollowMediaUnavailable("详情未取得", status_code=429)), \+ patch.object(error_reporting, "event") as emit:+ response = await self.client.get("/follow-qualities?id=7&t=secret")+ self.assertEqual(response.json(), {"qualities": [], "size": None})+ self.assertEqual(emit.call_args.kwargs["operation"], "qualities")+ self.assertEqual(emit.call_args.kwargs["phase"], "resolve")+ self.assertEqual(emit.call_args.kwargs["upstream_status"], 429)+ with patch.object(self.app.state.follow_media_resolver, "resolve", return_value=target), \+ patch("peach.routes_media.open_upstream", side_effect=httpx.ConnectError("network failure")), \+ patch.object(error_reporting, "event") as emit, \+ self.assertLogs("peach.routes_media", level="ERROR") as captured:+ response = await self.client.get("/follow-stream?id=7&media=0&t=secret")+ self.assertEqual(response.status_code, 502)+ self.assertEqual(emit.call_args.kwargs["phase"], "upstream")+ self.assertIsNone(emit.call_args.kwargs["upstream_status"])+ self.assertIn("fanbox", captured.output[0])+ self.assertIsNotNone(captured.records[0].exc_info)+ finally:+ self.app.state.http_transport.client = original+ fake.close()+async def test_follow_stream_only_offers_a_download_when_asked(self):"""下载是显式动作,不是播放的副作用。diff --git a/tests/test_follow_check.py b/tests/test_follow_check.pyindex ca725d854..e4648aafb 100644--- a/tests/test_follow_check.py+++ b/tests/test_follow_check.py@@ -11,13 +11,15 @@ import unittestfrom dataclasses import replacefrom datetime import datetime, timedelta, timezonefrom pathlib import Path+from unittest import mockfrom peach.follow import FollowHistoryEnd, FollowSourceError, FollowSourceRateLimitedfrom peach.follow_check import plan_check, run_checkfrom peach.follow_stream import FollowMediaResolverfrom peach.follow_secrets import CredentialError-from peach.follow_sources import FollowCandidate, SourceFetch, _BaseConnector+from peach.follow_sources import FanboxConnector, FollowCandidate, SourceFetch, _BaseConnectorfrom peach.follow_store import FollowStore+from peach.http import HttpResponsefrom support.ledger import fresh_ledger@@ -567,6 +569,66 @@ class InitialHistoryTests(_CheckCase):self.assertEqual(json.loads(row['metadata_json'])['initial_history_after'],(start - timedelta(days=40 + 11)).isoformat())+ def test_fanbox_initial_history_archives_identical_posts_from_distinct_urls(self):+ source = self._register()+ history_url = 'https://api.fanbox.cc/post.listCreator?creatorId=ffxivinitiala&firstId=2'+ body = json.dumps({'body': {'posts': [+ {'id': str(index), 'title': f'作品 {index}', 'feeRequired': 0,+ 'publishedDatetime': '2026-09-02T00:00:00Z'} for index in (2, 1)+ ]}}).encode()+ seen = []++ def transport(request, timeout, max_bytes):+ seen.append(request.url)+ if 'paginateCreator' in request.url:+ return HttpResponse(200, {}, json.dumps(+ {'body': {'pageUrls': [history_url]}}).encode())+ return HttpResponse(200, {}, body)++ connector = FanboxConnector(transport=transport, enrich_budget=0)+ result = self._run(source, connector, initial_days=30)+ self.assertTrue(result.ok)+ self.assertEqual(result.outcome.added, 2)+ self.assertEqual(len(self.store.items(source_id=source)), 2)+ row = self._row(source)+ self.assertEqual(row['last_status'], 'ok')+ self.assertEqual(row['backfill_page'], 1)+ self.assertEqual(json.loads(row['metadata_json'])['initial_history_floor'], 0)+ snapshots = list((self.root / 'sources').rglob('*.raw'))+ self.assertEqual(len(snapshots), 2)+ self.assertTrue(all(path.read_bytes() == body for path in snapshots))+ urls = {json.loads(path.with_suffix('.json').read_text(encoding='utf-8'))['request_url']+ for path in snapshots}+ self.assertEqual(urls, {seen[0], history_url})+ again = self._run(source, connector, initial_days=30)+ self.assertTrue(again.ok)+ self.assertEqual(again.outcome.added, 0)+ self.assertEqual(len(self.store.items(source_id=source)), 2)+ self.assertEqual(len(list((self.root / 'sources').rglob('*.raw'))), 2)++ def test_a_history_evidence_conflict_records_a_source_failure_and_keeps_saved_items(self):+ source = self._register()+ pages = {page: _fetch(request_url=f'https://api.fanbox.cc/page/{page}',+ raw_body=f'page {page}'.encode(),+ candidates=_dated(f'p{page}', 2, MOMENT, provider='fanbox'))+ for page in (0, 1)}+ persist = self.store._persist_evidence++ def evidence(fetch, moment):+ if fetch.request_url.endswith('/1'):+ raise FollowSourceError('immutable follow snapshot collision')+ return persist(fetch, moment)++ with mock.patch.object(self.store, '_persist_evidence', evidence):+ result = self._run(source, _PagedConnector(pages), initial_days=30)+ self.assertFalse(result.ok)+ self.assertEqual(result.status, 'error')+ self.assertIn('immutable follow snapshot collision', result.error)+ self.assertEqual({item.external_id for item in self.store.items(source_id=source)},+ {'p0-0', 'p0-1'})+ self.assertEqual(self._row(source)['last_status'], 'error')+ self.assertEqual(self._row(source)['backfill_page'], 0)+def test_the_top_up_stops_quietly_at_the_end_of_history(self):source = self._register(provider='kemono', ref='fanbox/1', url='https://kemono.cr/fanbox/user/1')page = _fetch(provider='kemono', ref='fanbox/1',diff --git a/tests/test_follow_store.py b/tests/test_follow_store.pyindex e0941543b..ea60d4ed0 100644--- a/tests/test_follow_store.py+++ b/tests/test_follow_store.py@@ -7,6 +7,7 @@ from dataclasses import replacefrom datetime import datetime, timedelta, timezonefrom pathlib import Path+from peach import follow_providersfrom peach.follow import FollowSourceErrorfrom peach.follow_sources import FollowCandidate, SourceFetchfrom peach.follow_store import (@@ -300,6 +301,56 @@ class RecordTests(_StoreCase):self.assertIsNotNone(outcome.evidence_path)self.assertIsNone(outcome.evidence_error)+ def test_identical_evidence_is_reused_without_overwriting_files(self):+ source_id = self._source()+ fetch = _fetch([_candidate("1", "A")])+ first = self.store.record(source_id, fetch, moment=MOMENT)+ evidence = self.root / "sources" / Path(first.evidence_path)+ sidecar = evidence.with_suffix(".json")+ before = [(path.read_bytes(), path.stat().st_mtime_ns) for path in (evidence, sidecar)]+ again = self.store.record(source_id, fetch, moment=MOMENT)+ self.assertEqual(again.evidence_path, first.evidence_path)+ self.assertEqual(again.added, 0)+ self.assertEqual([(path.read_bytes(), path.stat().st_mtime_ns)+ for path in (evidence, sidecar)], before)++ def test_shared_response_body_keeps_distinct_snapshot_metadata(self):+ for provider, spec in follow_providers.PROVIDERS.items():+ if not spec.source_url:+ continue+ with self.subTest(provider=provider):+ source_id = self._source(provider=provider, semantics=spec.semantics)+ fetch = _fetch([_candidate("1", "A", provider=provider)],+ provider=provider, semantics=spec.semantics)+ variants = (fetch, replace(fetch, request_url=fetch.request_url + "?firstId=1"),+ replace(fetch, candidates=()), replace(fetch, raw_body=b"different body"))+ paths = []+ for variant in variants:+ outcome = self.store.record(source_id, variant, moment=MOMENT)+ self.assertIsNone(outcome.evidence_error)+ paths.append(self.root / "sources" / Path(outcome.evidence_path))+ self.assertEqual(len(set(paths)), 4)+ for path, variant in zip(paths, variants):+ self.assertEqual(path.read_bytes(), variant.raw_body)+ sidecar = json.loads(path.with_suffix(".json").read_text(encoding="utf-8"))+ self.assertEqual(sidecar["provider"], provider)+ self.assertEqual(sidecar["request_url"], variant.request_url)+ self.assertEqual(sidecar["candidates"], len(variant.candidates))++ def test_conflicting_evidence_still_rejects_overwriting(self):+ source_id = self._source()+ fetch = _fetch([_candidate("1", "A")])+ outcome = self.store.record(source_id, fetch, moment=MOMENT)+ evidence = self.root / "sources" / Path(outcome.evidence_path)+ sidecar = evidence.with_suffix(".json")+ original = sidecar.read_bytes()+ sidecar.write_bytes(b"conflicting evidence")+ with self.assertRaisesRegex(FollowSourceError, "immutable follow snapshot collision"):+ self.store.record(source_id, fetch, moment=MOMENT)+ self.assertEqual(sidecar.read_bytes(), b"conflicting evidence")+ self.assertEqual(evidence.read_bytes(), fetch.raw_body)+ sidecar.write_bytes(original)+def test_approximate_precision_is_carried_through_from_the_connector(self):source_id = self._source()self.store.record(source_id, _fetch([diff --git a/tests/test_follow_stream.py b/tests/test_follow_stream.pyindex 135cf7b3d..8444cccb0 100644--- a/tests/test_follow_stream.py+++ b/tests/test_follow_stream.py@@ -410,6 +410,7 @@ class ProxyUpstreamTests(unittest.TestCase):with self.assertRaises(FollowProxyError) as raised:open_upstream(self._client(handler), "GET", target, incoming={})self.assertIn("403", str(raised.exception))+ self.assertEqual(raised.exception.status_code, 403)self.assertNotIn("shield", str(raised.exception))def test_a_redirect_loop_is_bounded(self):diff --git a/tests/test_follow_web.py b/tests/test_follow_web.pyindex 3cf6f110e..ad4f1c9b8 100644--- a/tests/test_follow_web.py+++ b/tests/test_follow_web.py@@ -2218,6 +2218,43 @@ class FollowContractTests(unittest.TestCase):self.assertIn("证据未取得", outcome["evidence_error"])self.assertEqual(self._get()["sources"][0]["last_status"], "ok")+ def test_one_source_evidence_conflict_does_not_abort_the_other_sources(self):+ self._seed()+ self._seed(provider="rule34xxx", ref="lazyprocrastinator", candidates=())+ persist = FollowStore._persist_evidence++ def evidence(store, fetch, moment):+ if fetch.provider == "rule34video":+ raise FollowSourceError("immutable follow snapshot collision")+ return persist(store, fetch, moment)++ def factory(provider, **kwargs):+ connector = mock.Mock()+ connector.fetch.return_value = SourceFetch(+ provider=provider, ref="lazyprocrastinator",+ request_url=f"https://{provider}.test/", semantics="work", raw_body=b"page",+ candidates=(FollowCandidate(provider=provider, external_id="999", title="作品"),))+ return connector++ with mock.patch.object(web_follow, "build_connector", factory), \+ mock.patch.object(FollowStore, "_persist_evidence", evidence), \+ mock.patch("peach.error_reporting.event") as report:+ result = self._post("/api/follow/check", {})+ failure = next(call for call in report.call_args_list if call.args[0] == "follow-check")+ self.assertEqual(failure.kwargs["provider"], "rule34video")+ self.assertEqual(failure.kwargs["phase"], "record")+ self.assertEqual(failure.kwargs["status"], "error")+ self.assertEqual(result["checked"], 2)+ outcomes = {row["provider"]: row for row in result["results"]}+ self.assertFalse(outcomes["rule34video"]["ok"])+ self.assertEqual(outcomes["rule34video"]["status"], "error")+ self.assertIn("immutable follow snapshot collision", outcomes["rule34video"]["error"])+ self.assertTrue(outcomes["rule34xxx"]["ok"])+ self.assertEqual(outcomes["rule34xxx"]["added"], 1)+ sources = {row["provider"]: row for row in self._get()["sources"]}+ self.assertEqual(sources["rule34video"]["last_status"], "error")+ self.assertEqual(sources["rule34xxx"]["last_status"], "ok")+def test_source_errors_are_recorded_for_later_inspection(self):self._seed()with mock.patch.object(web_follow, "build_connector") as factory: