Skip to content

armgit / peach

Public Imported from GitHub
0

Peach: Local-first video metadata scraper and media manager.

fix(follow): 修复快照冲突并补齐来源故障取证

longmeidao631f4bc377f7
diff --git a/docs/DIAGNOSTICS.md b/docs/DIAGNOSTICS.md
index 0f574a733..93fea7160 100644
--- a/docs/DIAGNOSTICS.md
+++ b/docs/DIAGNOSTICS.md
@@ -20,6 +20,7 @@
| 抽帧、拼图、探测 | 保留退出码、超时、采样位置、时长、工具来源、色彩重试和有界 stderr;`media-probe`、`preview`、`frame-capture`、`frame-tile` 与 `library-preview` 覆盖各入口 |
| 图片有效性 | Pillow 完整解码确认图片;纯色小 JPEG 可用,残片、非零退出和陈旧输出不能算生成成功 |
| 外部来源、下载 | 来源授权、限流、冷却与未命中使用领域状态;意外失败使用现有异常日志和任务回执;ZIP 包含来源健康状态与时间,来源响应原文留在本机 |
+| 关注检查、媒体代理 | `follow-check` 与 `follow-media` 本机事件记录来源、来源及条目编号、失败阶段、操作和可用的上游状态码;媒体事件区分帖子来源与资源来源,异常链与代码位置随支持包导出。地址、凭据和私人路径统一脱敏,不记录上游响应正文,不扩大远程诊断字段 |
| 更新与回滚 | 安装、回滚失败保留 traceback;ZIP 包含启动准备、更新状态、源码更新日志、托盘重启日志及近期迁移日志 |
| 隧道 | ZIP 包含 `cloudflare-tunnel.log` 与脱敏运行状态 |
| 浏览器 | 保存消息、堆栈、脚本文件名、行列、页面与浏览器标识;查询参数不进入页面字段;同源、有效登录、每分钟条数和请求体积门槛有回归 |
diff --git a/docs/REUSE.md b/docs/REUSE.md
index d2a20cc56..18023296a 100644
--- a/docs/REUSE.md
+++ b/docs/REUSE.md
@@ -665,6 +665,10 @@ A 盘垃圾隔离使用短名称 `peach-purge-<32 位 UUID>.peach-quarantine`,
- 关注的作者头像与来源图标是元数据:`follow_assets` 取回落在 `generated/follow-assets/`,经 `/follow-avatar`、`/source-icon` 给页面;地址只从固定表或固定主机拼,字节先认成图再写盘,到期重取失败继续用旧的并退避一小时,缓存有效期与 `/link-mark` 共用 `web_settings.metadata_refresh_seconds`。视频与图片不存本机。官方头像先认 FANBOX,没有时 `follow_avatar.profile_avatar_tiers` 取名片上的 X 与 Patreon,`follow_assets.largest_image` 留像素最大的那张。
- 厂牌标识由契约位 `has_logo` 决定出不出图:没装标识的厂牌一个 `<img>` 都不发,改用首字母底板,不靠 404 摘。标识 198 张随仓库分发(ADR-0026)。
- 关注检查分两阶段:列表阶段落 partial 行,详情补全按 provider 额度只补新行和未补齐行。「补齐过」由连接器的 `ENRICHED_MARK` 声明、判据登记在 `follow_store._ENRICHED_PREDICATES`。
+- 关注原始证据复用 Python 标准库 `hashlib.sha256` 与既有 `write_immutable`。快照文件名由检查时间和完整说明的摘要组成;说明含正文 SHA-256、脱敏请求地址及候选数量,因此相同正文的不同游标或筛选结果各有独立证据。相同快照可重复保存,已有文件内容冲突仍拒绝覆盖;历史证据按账本保存的相对路径读取,无迁移或新增依赖。临时账本回归遍历登记表中的全部追更来源,覆盖正文、地址及候选数量的独立变化,并经 FANBOX 连接器验证相同正文的历史补足和重复检查。
+- 单来源失败边界覆盖取回、整页存储和历史补足;证据内容冲突保留错误,已提交的条目继续保留,其他来源继续检查。Web 与 CLI 共用 `follow_check.run_check` 的失败结果,不把证据冲突记成成功。
+- 关注检查失败在本机 `follow-check` 事件中保留来源、来源编号、失败阶段、请求页及异常链;只记录登记来源键与技术状态,不复制作者名称或来源地址。单来源错误仍按既有任务结果返回。
+- 关注媒体的本机诊断复用 `error_reporting.event` 与支持包脱敏入口,分别记录来源、资源来源、来源及条目编号、播放/图片/清晰度/大小操作、解析或回源阶段和上游状态码。上游响应正文不进入诊断字段;异常中的地址和凭据由统一入口脱敏,远程诊断字段范围保持既有约定。隔离 API 和支持包回归覆盖 403、429、异常链、无效媒体清单及脱敏保真,无新增依赖。
- 存量行重抓(连接器新学到图片宽高、封面这类字段时,旧行不会自己补上)分两步:把 `ENRICHED_MARK` 换成新字段落库后才有的键并登记判据;再 `POST /api/follow/check` 发 `{"older":true,"backfill_all":true,"rewind":true,"background":true}`(可加 `"sources":[id,…]`),从第 1 页走到站点说没有更多,进度读 `GET /api/follow/check`。`backfill_all`、`rewind` 离开 `older` 被拒收、不进界面,ledger 的回填游标只进不退;长跑任务,重启前按 `peach-batch-jobs` 先查。
- 图片的固有宽高只问文件头:`follow_image_dims.probe_image_dims` 发 `Range: bytes=0-65535`,`dims_from_header` 认 PNG/GIF/WebP(VP8、VP8L、VP8X)/JPEG(跳过 EXIF 到 SOF);连接器、`backfill_follow_image_dims.py` 与 `/api/follow/image-dims` 落库前都经 `positive_dims` 归一,`FollowStore.set_image_dims` 只补空缺。宽高只为图片视图的卡面预留比例,回填每个条目只问卡面那张;原文件主机拦脚本(pawchive 的 `file.` 子域挂 ddos-guard)时按缩略图量,详情原图取不到也退回缩略图。
- 归档站(kemono、coomer、pawchive)多图或视频的帖子由 `KemonoConnector._media_items` 列进 `media_items`(交付文件排第一、按路径去重),详情轮播与 `/follow-stream?media=N` 按条目自己站点的主机白名单取。
diff --git a/src/peach/follow_check.py b/src/peach/follow_check.py
index 24918082a..9022b3201 100644
--- a/src/peach/follow_check.py
+++ b/src/peach/follow_check.py
@@ -14,7 +14,7 @@ from dataclasses import dataclass, replace
from datetime import datetime, timedelta, timezone
from typing import Mapping
-from . import follow_providers
+from . import error_reporting, follow_providers
from .follow import FollowHistoryEnd, FollowSourceError, FollowSourceRateLimited
from .follow_secrets import CredentialError
from .follow_sources import SourceFetch, enrichment_mark, history_window, published_stamp
@@ -126,6 +126,7 @@ def run_check(row: Mapping, *, credentials, writer, connector_factory,
cutoff, floor = _history_limits(row, metadata, writer, moment,
older=older, initial_days=initial_days)
recorder = _PageRecorder(row, writer, moment, cutoff)
+ phase = "fetch"
try:
connector = build_connector_for(
provider, credentials, connector_factory,
@@ -142,33 +143,36 @@ def run_check(row: Mapping, *, credentials, writer, connector_factory,
last_modified=None if force else row["last_modified"],
page=page,
)
+ candidates = history_window(fetch.candidates, cutoff, floor)
+ history_skipped = getattr(connector, 'history_skipped', 0) + len(fetch.candidates) - len(candidates)
+ fetch = replace(fetch, candidates=candidates)
+ phase = "record"
+ with writer() as store:
+ if not fetch.not_modified:
+ if replay_first:
+ store.merge_source_metadata(source_id, {'initial_history_first_page_pending': False}, moment)
+ elif history_skipped and page == 0 and 'initial_history_first_page_pending' not in metadata:
+ store.merge_source_metadata(source_id, {'initial_history_first_page_pending': True}, moment)
+ outcome = recorder.record_in(store, fetch, page)
+ if floor and not fetch.not_modified:
+ phase = "history"
+ recorder.cutoff = None
+ fetch, outcome = _fill_floor(row, connector, writer, fetch, outcome, cutoff, floor, moment,
+ recorder=recorder)
+ with writer() as store:
+ phase = "identity"
+ learned = store.learn_official_author_alias(provider, ref, fetch.candidates)
+ return CheckResult(**base, fetch=fetch, outcome=outcome,
+ author_alias_learned=learned, history_skipped=history_skipped)
except FollowHistoryEnd:
with writer() as store:
store.record_history_end(source_id, moment)
return CheckResult(**base, exhausted=True, message="没有更多历史内容")
except CredentialError as error:
- return _record_failure(writer, base, error, moment, "unauthorized")
+ return _record_failure(writer, base, error, moment, "unauthorized", phase=phase)
except FollowSourceError as error:
_pause_if_rate_limited(cooldown, provider, error)
- return _record_failure(writer, base, error, moment, "error")
- candidates = history_window(fetch.candidates, cutoff, floor)
- history_skipped = getattr(connector, 'history_skipped', 0) + len(fetch.candidates) - len(candidates)
- fetch = replace(fetch, candidates=candidates)
- with writer() as store:
- if not fetch.not_modified:
- if replay_first:
- store.merge_source_metadata(source_id, {'initial_history_first_page_pending': False}, moment)
- elif history_skipped and page == 0 and 'initial_history_first_page_pending' not in metadata:
- store.merge_source_metadata(source_id, {'initial_history_first_page_pending': True}, moment)
- outcome = recorder.record_in(store, fetch, page)
- if floor and not fetch.not_modified:
- recorder.cutoff = None
- fetch, outcome = _fill_floor(row, connector, writer, fetch, outcome, cutoff, floor, moment,
- recorder=recorder)
- with writer() as store:
- learned = store.learn_official_author_alias(provider, ref, fetch.candidates)
- return CheckResult(**base, fetch=fetch, outcome=outcome,
- author_alias_learned=learned, history_skipped=history_skipped)
+ return _record_failure(writer, base, error, moment, "error", phase=phase)
def _pause_if_rate_limited(cooldown, provider: str, error: FollowSourceError) -> None:
@@ -237,9 +241,12 @@ def _fill_floor(row, connector, writer, fetch, outcome, cutoff, floor, moment, *
def _record_failure(writer, base: dict, error: Exception, moment: datetime,
- status: str) -> CheckResult:
+ status: str, *, phase: str) -> CheckResult:
with writer() as store:
store.record_error(base["source_id"], str(error), moment, status=status)
+ error_reporting.event("follow-check", error, provider=base["provider"],
+ source_id=base["source_id"], phase=phase,
+ page=base["page"], older=base["older"], status=status)
return CheckResult(**base, ok=False, status=status, error=str(error))
diff --git a/src/peach/follow_store.py b/src/peach/follow_store.py
index 8d17af366..c5bbaff90 100644
--- a/src/peach/follow_store.py
+++ b/src/peach/follow_store.py
@@ -641,16 +641,19 @@ class FollowStore:
key = hashlib.sha256(f"{fetch.provider}\n{fetch.ref}".encode()).hexdigest()[:20]
directory = self.sources_root / fetch.provider / key
stamp = reference.astimezone(timezone.utc).strftime("%Y%m%dT%H%M%S.%fZ")
- path = directory / f"{stamp}-{digest[:12]}.raw"
+ sidecar = (json.dumps({
+ "provider": fetch.provider, "ref": fetch.ref,
+ # 脱敏后的请求 URL:凭据永不进证据目录。
+ "request_url": fetch.request_url, "sha256": digest,
+ "checked_at": _now_text(reference), "candidates": len(fetch.candidates),
+ }, ensure_ascii=False, indent=2) + "\n").encode("utf-8")
+ # 快照身份包含正文摘要与说明;相同正文可来自不同游标或候选筛选。
+ snapshot_digest = hashlib.sha256(sidecar).hexdigest()
+ path = directory / f"{stamp}-{snapshot_digest}.raw"
try:
directory.mkdir(parents=True, exist_ok=True)
write_immutable(path, fetch.raw_body)
- write_immutable(path.with_suffix(".json"), (json.dumps({
- "provider": fetch.provider, "ref": fetch.ref,
- # 脱敏后的请求 URL:凭据永不进证据目录。
- "request_url": fetch.request_url, "sha256": digest,
- "checked_at": _now_text(reference), "candidates": len(fetch.candidates),
- }, ensure_ascii=False, indent=2) + "\n").encode("utf-8"))
+ write_immutable(path.with_suffix(".json"), sidecar)
except OSError as error:
return None, f"证据未取得({self.sources_root} 不可写:{error.strerror or error})"
return str(path.relative_to(self.sources_root.parent)), None
diff --git a/src/peach/follow_stream.py b/src/peach/follow_stream.py
index dba34f071..e0986fba8 100644
--- a/src/peach/follow_stream.py
+++ b/src/peach/follow_stream.py
@@ -25,7 +25,9 @@ from .http import HttpRequest, HttpResponse, HttpTransport, public_https_url, re
LOGGER = logging.getLogger(__name__)
class FollowMediaUnavailable(RuntimeError):
- pass
+ def __init__(self, message: str = "", *, status_code: int | None = None):
+ super().__init__(message)
+ self.status_code = status_code
class FollowProxyError(RuntimeError):
@@ -35,6 +37,10 @@ class FollowProxyError(RuntimeError):
媒体是 404,上游这一次没给出可转发的响应是 502。
"""
+ def __init__(self, message: str, *, status_code: int | None = None):
+ super().__init__(message)
+ self.status_code = status_code
+
@dataclass(frozen=True)
class ResolvedFollowMedia:
@@ -337,7 +343,7 @@ class FollowMediaResolver:
LOGGER.warning("rule34video detail page for follow item %s unavailable: HTTP %s",
item.id, response.status)
self._remember_miss(item.id)
- raise FollowMediaUnavailable("Rule34Video 详情页未取得")
+ raise FollowMediaUnavailable("Rule34Video 详情页未取得", status_code=response.status)
text = response.body.decode("utf-8", errors="replace")
found: list[str] = []
for matched in _VIDEO_URL_RE.finditer(text):
@@ -443,5 +449,5 @@ def open_upstream(client: httpx.Client, method: str, target: ResolvedFollowMedia
if not 200 <= response.status_code < 300 and response.status_code not in accepted_statuses:
status = response.status_code
response.close()
- raise FollowProxyError(f"上游返回 HTTP {status}")
+ raise FollowProxyError(f"上游返回 HTTP {status}", status_code=status)
return response
diff --git a/src/peach/routes_media.py b/src/peach/routes_media.py
index f176ea6cb..d311324d2 100644
--- a/src/peach/routes_media.py
+++ b/src/peach/routes_media.py
@@ -32,7 +32,8 @@ from starlette.staticfiles import StaticFiles
from starlette.requests import ClientDisconnect
from . import (
- avatar_face, avatar_online, avatar_picker, avatar_provider, follow_assets, follow_providers, images,
+ avatar_face, avatar_online, avatar_picker, avatar_provider, error_reporting,
+ follow_assets, follow_providers, images,
jav_poster_crop, link_marks, link_status, scraping_access, site_icons, social_links, subtitles,
taste_history, timeline_sheets, web_follow,
)
@@ -937,7 +938,25 @@ def work_icon(request: Request, work: str = "",
return _asset_response(request, path)
-def _follow_media_size(state, item_id: int, target: ResolvedFollowMedia) -> int | None:
+def _report_follow_media_failure(item, error: Exception, *, operation: str, phase: str,
+ media_index: int | None = None) -> None:
+ """媒体失败按来源、资源来源和操作定位;地址与凭据交给统一脱敏入口。"""
+ rows = web_follow._raw_media_items(item)
+ index = media_index if media_index is not None else 0
+ resource_provider = item.provider
+ if ((operation != "image" or media_index is not None) and 0 <= index < len(rows)
+ and isinstance(rows[index], dict)):
+ resource_provider = str(rows[index].get("resource_provider") or item.provider)
+ if resource_provider not in follow_providers.PROVIDERS:
+ resource_provider = "unknown"
+ error_reporting.event(
+ "follow-media", error, provider=item.provider, source_id=getattr(item, "source_id", None),
+ item_id=item.id, resource_provider=resource_provider, operation=operation,
+ phase=phase, media_index=media_index, upstream_status=getattr(error, "status_code", None),
+ )
+
+
+def _follow_media_size(state, item, target: ResolvedFollowMedia) -> int | None:
"""正片的字节数,取不到就是 None。
只能问上游:来源列表里没有这一项,账本里也只有落盘资源才有 size。一次 HEAD 就够,
@@ -950,7 +969,7 @@ def _follow_media_size(state, item_id: int, target: ResolvedFollowMedia) -> int
try:
response = open_upstream(state.http_transport.client, "HEAD", target, incoming={})
except (FollowProxyError, OSError, httpx.HTTPError) as error:
- LOGGER.warning("follow media size unavailable for item %s: %s", item_id, error)
+ _report_follow_media_failure(item, error, operation="size", phase="upstream")
return None
try:
size = int(response.headers.get("content-length") or 0)
@@ -984,7 +1003,8 @@ def follow_qualities(request: Request, id: int, args: dict[str, str] = Depends(r
return JSONResponse({"error": "no such follow item"}, status_code=404)
try:
resolved = state.follow_media_resolver.resolve(item)
- except FollowMediaUnavailable:
+ except FollowMediaUnavailable as error:
+ _report_follow_media_failure(item, error, operation="qualities", phase="resolve")
return {"qualities": [], "size": None}
# 档位数量不写死:正则匹配 video_url 与任意编号的 video_alt_urlN,
# 站点给几档就是几档(实测同一作者下有 4 档也有 5 档的条目)。
@@ -992,7 +1012,7 @@ def follow_qualities(request: Request, id: int, args: dict[str, str] = Depends(r
return {"qualities": [
{"height": height, "label": "4K" if height >= 2160 else f"{height}p"}
for height, _ in resolved.qualities if height
- ], "size": _follow_media_size(state, id, resolved)}
+ ], "size": _follow_media_size(state, item, resolved)}
@router.api_route("/follow-stream", methods=["GET", "HEAD"])
@@ -1005,6 +1025,7 @@ def follow_stream(request: Request, id: int, media: int | None = None,
item = FollowStore(lambda: connection).item(id)
if item is None:
return JSONResponse({"error": "no such follow item"}, status_code=404)
+ phase = "resolve"
try:
key = f"follow:{item.id}:media:{media}:quality:{quality}"
cache = cache_for(state.web_contract)
@@ -1013,6 +1034,7 @@ def follow_stream(request: Request, id: int, media: int | None = None,
if cached is not None:
return cached
target = state.follow_media_resolver.resolve(item, media, quality)
+ phase = "upstream"
# 跟重定向、逐跳过白名单、非 2xx 一律不转发,都在 follow_stream 里;
# 这里只负责把 Peach 的请求接到那个流上。
extra = {"content-disposition": _attachment_disposition(item.title, target.url)} if download else None
@@ -1020,14 +1042,16 @@ def follow_stream(request: Request, id: int, media: int | None = None,
lambda: open_upstream(state.http_transport.client, request.method, target, incoming=request.headers),
headers=extra)
except FollowMediaUnavailable as error:
+ _report_follow_media_failure(item, error, operation="stream", phase=phase, media_index=media)
return JSONResponse({"error": str(error)}, status_code=404)
except FollowProxyError as error:
# 上游的状态码与正文不转发:403 页面、限流提示、错误 JSON 交给播放器
# 没有用处,还会把上游主机和提示语抄给浏览器。对外统一是 502。
- LOGGER.warning("follow media proxy refused for item %s: %s", id, error)
+ _report_follow_media_failure(item, error, operation="stream", phase=phase, media_index=media)
return JSONResponse({"error": "follow media unavailable"}, status_code=502)
- except (OSError, httpx.HTTPError):
- LOGGER.exception("follow media proxy failed for item %s", id)
+ except (OSError, httpx.HTTPError) as error:
+ LOGGER.exception("follow media proxy failed for provider %s item %s", item.provider, id)
+ _report_follow_media_failure(item, error, operation="stream", phase=phase, media_index=media)
return JSONResponse({"error": "follow media unavailable"}, status_code=502)
@router.api_route("/follow-image", methods=["GET", "HEAD"])
@@ -1039,6 +1063,7 @@ def follow_image(request: Request, id: int, media: int | None = None,
if item is None:
return Response(status_code=404)
cache = cache_for(state.web_contract)
+ phase = "resolve"
try:
if media is None:
url = web_follow._remote_thumb_url(item)
@@ -1060,9 +1085,11 @@ def follow_image(request: Request, id: int, media: int | None = None,
resolved = state.follow_media_resolver.resolve(item, media)
if resolved.headers:
target = replace(resolved, url=url)
+ phase = "upstream"
return cache.stream(key, request, lambda: open_upstream(
state.http_transport.client, request.method, target, incoming=request.headers))
- except (ValueError, FollowMediaUnavailable, FollowProxyError, OSError, httpx.HTTPError):
+ except (ValueError, FollowMediaUnavailable, FollowProxyError, OSError, httpx.HTTPError) as error:
+ _report_follow_media_failure(item, error, operation="image", phase=phase, media_index=media)
return Response(status_code=404, headers={"cache-control": "no-store"})
diff --git a/tests/test_error_reporting.py b/tests/test_error_reporting.py
index e77c97504..aa33c5e87 100644
--- a/tests/test_error_reporting.py
+++ b/tests/test_error_reporting.py
@@ -319,6 +319,53 @@ class ErrorReportingTests(unittest.TestCase):
for secret in ("private-cookie", "private-cid", "private-token"):
self.assertNotIn(secret, content)
+ def test_follow_media_report_keeps_source_context_and_redacts_upstream_details(self):
+ from types import SimpleNamespace
+ from peach.follow_stream import FollowProxyError
+ from peach.routes_media import _report_follow_media_failure
+
+ item = SimpleNamespace(provider="fanbox", source_id=3, id=7,
+ metadata={"media_items": [{"resource_provider": "gofile"}]})
+ try:
+ raise FollowProxyError(
+ "upstream https://media.gofile.io/private-file?token=private-token "
+ "Cookie: private-cookie", status_code=403)
+ except FollowProxyError as error:
+ _report_follow_media_failure(item, error, operation="stream", phase="upstream",
+ media_index=0)
+ with zipfile.ZipFile(io.BytesIO(support_bundle.bundle_bytes())) as archive:
+ payload = "\n".join(archive.read(name).decode() for name in archive.namelist()
+ if "errors-" in name and name.endswith(".log"))
+ row = next(json.loads(line) for line in payload.splitlines()
+ if json.loads(line).get("stage") == "follow-media")
+ self.assertEqual({key: row[key] for key in (
+ "provider", "source_id", "item_id", "resource_provider", "operation", "phase",
+ "media_index", "upstream_status")}, {
+ "provider": "fanbox", "source_id": 3, "item_id": 7, "resource_provider": "gofile",
+ "operation": "stream", "phase": "upstream", "media_index": 0, "upstream_status": 403})
+ self.assertEqual(row["exception"][0]["type"], "FollowProxyError")
+ self.assertTrue(row["frames"])
+ for secret in ("private-file", "private-token", "private-cookie", "media.gofile.io"):
+ self.assertNotIn(secret, payload)
+
+ def test_follow_media_report_handles_invalid_media_without_masking_the_failure(self):
+ from types import SimpleNamespace
+ from peach.routes_media import _report_follow_media_failure
+
+ item = SimpleNamespace(provider="kemono", source_id=4, id=8,
+ metadata={"media_items": [None, {"resource_provider": "private-name"}]})
+ for index, expected in ((0, "kemono"), (1, "unknown")):
+ with self.subTest(index=index):
+ with mock.patch.object(error_reporting, "event") as emit:
+ _report_follow_media_failure(item, ValueError("媒体条目格式不符"),
+ operation="stream", phase="resolve", media_index=index)
+ self.assertEqual(emit.call_args.kwargs["resource_provider"], expected)
+ del item.source_id
+ with mock.patch.object(error_reporting, "event") as emit:
+ _report_follow_media_failure(item, ValueError("图片来源不可用"),
+ operation="image", phase="resolve")
+ self.assertIsNone(emit.call_args.kwargs["source_id"])
+
def test_failure_dialog_points_to_an_existing_report_and_cli_needs_no_service(self):
message = error_reporting.failure_details("tray-handoff", OSError("desktop unavailable"))
self.assertIn("诊断报告", message)
diff --git a/tests/test_fastapi_api.py b/tests/test_fastapi_api.py
index 2aeea514c..36c0ec548 100644
--- a/tests/test_fastapi_api.py
+++ b/tests/test_fastapi_api.py
@@ -2461,6 +2461,72 @@ class FastApiContractTests(unittest.IsolatedAsyncioTestCase):
self.assertEqual(response.headers["content-range"], "bytes 2-5/10")
self.assertNotIn("img.kemono.cr", response.text)
+ async def test_follow_media_failures_record_source_phase_and_status(self):
+ from peach import error_reporting
+ from peach.follow_stream import FollowMediaUnavailable, ResolvedFollowMedia
+
+ with closing(sqlite3.connect(self.db)) as connection, connection:
+ connection.executescript((ROOT / "migrations" / "0018_online_follow.sql").read_text(
+ encoding="utf-8"))
+ connection.execute(
+ "INSERT INTO follow_source(id,provider,ref,label,url,semantics,created_at,updated_at)"
+ " VALUES(3,'fanbox','creator','Creator','https://creator.fanbox.cc/','work','x','x')")
+ connection.execute(
+ "INSERT INTO follow_item(id,source_id,external_id,title,url,release_key,metadata_json,"
+ "first_seen_at,last_seen_at) VALUES(7,3,'7','Remote',"
+ "'https://creator.fanbox.cc/posts/7','remote',?,'x','x')",
+ (json.dumps({"media_items": [{"url": "https://media.gofile.io/7.mp4",
+ "thumb_url": "https://media.gofile.io/7.jpg",
+ "resource_provider": "gofile"}]}),))
+ target = ResolvedFollowMedia("https://media.gofile.io/7.mp4", allowed_hosts=("gofile.io",))
+ original = self.app.state.http_transport.client
+ fake = httpx.Client(transport=httpx.MockTransport(
+ lambda request: httpx.Response(403, request=request, content=b"private upstream error")))
+ self.app.state.http_transport.client = fake
+ try:
+ with patch.object(self.app.state.follow_media_resolver, "resolve", return_value=target), \
+ patch("peach.follow_stream.resolves_publicly", return_value=True), \
+ patch.object(error_reporting, "event") as emit:
+ for path, operation, status in (
+ ("/follow-stream?id=7&media=0&t=secret", "stream", 502),
+ ("/follow-image?id=7&media=0&t=secret", "image", 404),
+ ("/follow-qualities?id=7&t=secret", "size", 200),
+ ):
+ with self.subTest(operation=operation):
+ emit.reset_mock()
+ response = await self.client.get(path)
+ self.assertEqual(response.status_code, status)
+ fields = emit.call_args.kwargs
+ self.assertEqual(fields["provider"], "fanbox")
+ self.assertEqual(fields["resource_provider"], "gofile")
+ self.assertEqual(fields["item_id"], 7)
+ self.assertEqual(fields["source_id"], 3)
+ self.assertEqual(fields["operation"], operation)
+ self.assertEqual(fields["phase"], "upstream")
+ self.assertEqual(fields["upstream_status"], 403)
+ self.assertNotIn("private upstream error", response.text)
+ with patch.object(self.app.state.follow_media_resolver, "resolve",
+ side_effect=FollowMediaUnavailable("详情未取得", status_code=429)), \
+ patch.object(error_reporting, "event") as emit:
+ response = await self.client.get("/follow-qualities?id=7&t=secret")
+ self.assertEqual(response.json(), {"qualities": [], "size": None})
+ self.assertEqual(emit.call_args.kwargs["operation"], "qualities")
+ self.assertEqual(emit.call_args.kwargs["phase"], "resolve")
+ self.assertEqual(emit.call_args.kwargs["upstream_status"], 429)
+ with patch.object(self.app.state.follow_media_resolver, "resolve", return_value=target), \
+ patch("peach.routes_media.open_upstream", side_effect=httpx.ConnectError("network failure")), \
+ patch.object(error_reporting, "event") as emit, \
+ self.assertLogs("peach.routes_media", level="ERROR") as captured:
+ response = await self.client.get("/follow-stream?id=7&media=0&t=secret")
+ self.assertEqual(response.status_code, 502)
+ self.assertEqual(emit.call_args.kwargs["phase"], "upstream")
+ self.assertIsNone(emit.call_args.kwargs["upstream_status"])
+ self.assertIn("fanbox", captured.output[0])
+ self.assertIsNotNone(captured.records[0].exc_info)
+ finally:
+ self.app.state.http_transport.client = original
+ fake.close()
+
async def test_follow_stream_only_offers_a_download_when_asked(self):
"""下载是显式动作,不是播放的副作用。
diff --git a/tests/test_follow_check.py b/tests/test_follow_check.py
index ca725d854..e4648aafb 100644
--- a/tests/test_follow_check.py
+++ b/tests/test_follow_check.py
@@ -11,13 +11,15 @@ import unittest
from dataclasses import replace
from datetime import datetime, timedelta, timezone
from pathlib import Path
+from unittest import mock
from peach.follow import FollowHistoryEnd, FollowSourceError, FollowSourceRateLimited
from peach.follow_check import plan_check, run_check
from peach.follow_stream import FollowMediaResolver
from peach.follow_secrets import CredentialError
-from peach.follow_sources import FollowCandidate, SourceFetch, _BaseConnector
+from peach.follow_sources import FanboxConnector, FollowCandidate, SourceFetch, _BaseConnector
from peach.follow_store import FollowStore
+from peach.http import HttpResponse
from support.ledger import fresh_ledger
@@ -567,6 +569,66 @@ class InitialHistoryTests(_CheckCase):
self.assertEqual(json.loads(row['metadata_json'])['initial_history_after'],
(start - timedelta(days=40 + 11)).isoformat())
+ def test_fanbox_initial_history_archives_identical_posts_from_distinct_urls(self):
+ source = self._register()
+ history_url = 'https://api.fanbox.cc/post.listCreator?creatorId=ffxivinitiala&firstId=2'
+ body = json.dumps({'body': {'posts': [
+ {'id': str(index), 'title': f'作品 {index}', 'feeRequired': 0,
+ 'publishedDatetime': '2026-09-02T00:00:00Z'} for index in (2, 1)
+ ]}}).encode()
+ seen = []
+
+ def transport(request, timeout, max_bytes):
+ seen.append(request.url)
+ if 'paginateCreator' in request.url:
+ return HttpResponse(200, {}, json.dumps(
+ {'body': {'pageUrls': [history_url]}}).encode())
+ return HttpResponse(200, {}, body)
+
+ connector = FanboxConnector(transport=transport, enrich_budget=0)
+ result = self._run(source, connector, initial_days=30)
+ self.assertTrue(result.ok)
+ self.assertEqual(result.outcome.added, 2)
+ self.assertEqual(len(self.store.items(source_id=source)), 2)
+ row = self._row(source)
+ self.assertEqual(row['last_status'], 'ok')
+ self.assertEqual(row['backfill_page'], 1)
+ self.assertEqual(json.loads(row['metadata_json'])['initial_history_floor'], 0)
+ snapshots = list((self.root / 'sources').rglob('*.raw'))
+ self.assertEqual(len(snapshots), 2)
+ self.assertTrue(all(path.read_bytes() == body for path in snapshots))
+ urls = {json.loads(path.with_suffix('.json').read_text(encoding='utf-8'))['request_url']
+ for path in snapshots}
+ self.assertEqual(urls, {seen[0], history_url})
+ again = self._run(source, connector, initial_days=30)
+ self.assertTrue(again.ok)
+ self.assertEqual(again.outcome.added, 0)
+ self.assertEqual(len(self.store.items(source_id=source)), 2)
+ self.assertEqual(len(list((self.root / 'sources').rglob('*.raw'))), 2)
+
+ def test_a_history_evidence_conflict_records_a_source_failure_and_keeps_saved_items(self):
+ source = self._register()
+ pages = {page: _fetch(request_url=f'https://api.fanbox.cc/page/{page}',
+ raw_body=f'page {page}'.encode(),
+ candidates=_dated(f'p{page}', 2, MOMENT, provider='fanbox'))
+ for page in (0, 1)}
+ persist = self.store._persist_evidence
+
+ def evidence(fetch, moment):
+ if fetch.request_url.endswith('/1'):
+ raise FollowSourceError('immutable follow snapshot collision')
+ return persist(fetch, moment)
+
+ with mock.patch.object(self.store, '_persist_evidence', evidence):
+ result = self._run(source, _PagedConnector(pages), initial_days=30)
+ self.assertFalse(result.ok)
+ self.assertEqual(result.status, 'error')
+ self.assertIn('immutable follow snapshot collision', result.error)
+ self.assertEqual({item.external_id for item in self.store.items(source_id=source)},
+ {'p0-0', 'p0-1'})
+ self.assertEqual(self._row(source)['last_status'], 'error')
+ self.assertEqual(self._row(source)['backfill_page'], 0)
+
def test_the_top_up_stops_quietly_at_the_end_of_history(self):
source = self._register(provider='kemono', ref='fanbox/1', url='https://kemono.cr/fanbox/user/1')
page = _fetch(provider='kemono', ref='fanbox/1',
diff --git a/tests/test_follow_store.py b/tests/test_follow_store.py
index e0941543b..ea60d4ed0 100644
--- a/tests/test_follow_store.py
+++ b/tests/test_follow_store.py
@@ -7,6 +7,7 @@ from dataclasses import replace
from datetime import datetime, timedelta, timezone
from pathlib import Path
+from peach import follow_providers
from peach.follow import FollowSourceError
from peach.follow_sources import FollowCandidate, SourceFetch
from peach.follow_store import (
@@ -300,6 +301,56 @@ class RecordTests(_StoreCase):
self.assertIsNotNone(outcome.evidence_path)
self.assertIsNone(outcome.evidence_error)
+ def test_identical_evidence_is_reused_without_overwriting_files(self):
+ source_id = self._source()
+ fetch = _fetch([_candidate("1", "A")])
+ first = self.store.record(source_id, fetch, moment=MOMENT)
+ evidence = self.root / "sources" / Path(first.evidence_path)
+ sidecar = evidence.with_suffix(".json")
+ before = [(path.read_bytes(), path.stat().st_mtime_ns) for path in (evidence, sidecar)]
+ again = self.store.record(source_id, fetch, moment=MOMENT)
+ self.assertEqual(again.evidence_path, first.evidence_path)
+ self.assertEqual(again.added, 0)
+ self.assertEqual([(path.read_bytes(), path.stat().st_mtime_ns)
+ for path in (evidence, sidecar)], before)
+
+ def test_shared_response_body_keeps_distinct_snapshot_metadata(self):
+ for provider, spec in follow_providers.PROVIDERS.items():
+ if not spec.source_url:
+ continue
+ with self.subTest(provider=provider):
+ source_id = self._source(provider=provider, semantics=spec.semantics)
+ fetch = _fetch([_candidate("1", "A", provider=provider)],
+ provider=provider, semantics=spec.semantics)
+ variants = (fetch, replace(fetch, request_url=fetch.request_url + "?firstId=1"),
+ replace(fetch, candidates=()), replace(fetch, raw_body=b"different body"))
+ paths = []
+ for variant in variants:
+ outcome = self.store.record(source_id, variant, moment=MOMENT)
+ self.assertIsNone(outcome.evidence_error)
+ paths.append(self.root / "sources" / Path(outcome.evidence_path))
+ self.assertEqual(len(set(paths)), 4)
+ for path, variant in zip(paths, variants):
+ self.assertEqual(path.read_bytes(), variant.raw_body)
+ sidecar = json.loads(path.with_suffix(".json").read_text(encoding="utf-8"))
+ self.assertEqual(sidecar["provider"], provider)
+ self.assertEqual(sidecar["request_url"], variant.request_url)
+ self.assertEqual(sidecar["candidates"], len(variant.candidates))
+
+ def test_conflicting_evidence_still_rejects_overwriting(self):
+ source_id = self._source()
+ fetch = _fetch([_candidate("1", "A")])
+ outcome = self.store.record(source_id, fetch, moment=MOMENT)
+ evidence = self.root / "sources" / Path(outcome.evidence_path)
+ sidecar = evidence.with_suffix(".json")
+ original = sidecar.read_bytes()
+ sidecar.write_bytes(b"conflicting evidence")
+ with self.assertRaisesRegex(FollowSourceError, "immutable follow snapshot collision"):
+ self.store.record(source_id, fetch, moment=MOMENT)
+ self.assertEqual(sidecar.read_bytes(), b"conflicting evidence")
+ self.assertEqual(evidence.read_bytes(), fetch.raw_body)
+ sidecar.write_bytes(original)
+
def test_approximate_precision_is_carried_through_from_the_connector(self):
source_id = self._source()
self.store.record(source_id, _fetch([
diff --git a/tests/test_follow_stream.py b/tests/test_follow_stream.py
index 135cf7b3d..8444cccb0 100644
--- a/tests/test_follow_stream.py
+++ b/tests/test_follow_stream.py
@@ -410,6 +410,7 @@ class ProxyUpstreamTests(unittest.TestCase):
with self.assertRaises(FollowProxyError) as raised:
open_upstream(self._client(handler), "GET", target, incoming={})
self.assertIn("403", str(raised.exception))
+ self.assertEqual(raised.exception.status_code, 403)
self.assertNotIn("shield", str(raised.exception))
def test_a_redirect_loop_is_bounded(self):
diff --git a/tests/test_follow_web.py b/tests/test_follow_web.py
index 3cf6f110e..ad4f1c9b8 100644
--- a/tests/test_follow_web.py
+++ b/tests/test_follow_web.py
@@ -2218,6 +2218,43 @@ class FollowContractTests(unittest.TestCase):
self.assertIn("证据未取得", outcome["evidence_error"])
self.assertEqual(self._get()["sources"][0]["last_status"], "ok")
+ def test_one_source_evidence_conflict_does_not_abort_the_other_sources(self):
+ self._seed()
+ self._seed(provider="rule34xxx", ref="lazyprocrastinator", candidates=())
+ persist = FollowStore._persist_evidence
+
+ def evidence(store, fetch, moment):
+ if fetch.provider == "rule34video":
+ raise FollowSourceError("immutable follow snapshot collision")
+ return persist(store, fetch, moment)
+
+ def factory(provider, **kwargs):
+ connector = mock.Mock()
+ connector.fetch.return_value = SourceFetch(
+ provider=provider, ref="lazyprocrastinator",
+ request_url=f"https://{provider}.test/", semantics="work", raw_body=b"page",
+ candidates=(FollowCandidate(provider=provider, external_id="999", title="作品"),))
+ return connector
+
+ with mock.patch.object(web_follow, "build_connector", factory), \
+ mock.patch.object(FollowStore, "_persist_evidence", evidence), \
+ mock.patch("peach.error_reporting.event") as report:
+ result = self._post("/api/follow/check", {})
+ failure = next(call for call in report.call_args_list if call.args[0] == "follow-check")
+ self.assertEqual(failure.kwargs["provider"], "rule34video")
+ self.assertEqual(failure.kwargs["phase"], "record")
+ self.assertEqual(failure.kwargs["status"], "error")
+ self.assertEqual(result["checked"], 2)
+ outcomes = {row["provider"]: row for row in result["results"]}
+ self.assertFalse(outcomes["rule34video"]["ok"])
+ self.assertEqual(outcomes["rule34video"]["status"], "error")
+ self.assertIn("immutable follow snapshot collision", outcomes["rule34video"]["error"])
+ self.assertTrue(outcomes["rule34xxx"]["ok"])
+ self.assertEqual(outcomes["rule34xxx"]["added"], 1)
+ sources = {row["provider"]: row for row in self._get()["sources"]}
+ self.assertEqual(sources["rule34video"]["last_status"], "error")
+ self.assertEqual(sources["rule34xxx"]["last_status"], "ok")
+
def test_source_errors_are_recorded_for_later_inspection(self):
self._seed()
with mock.patch.object(web_follow, "build_connector") as factory: