Peach: Local-first video metadata scraper and media manager.
fix(follow): 保留 Iwara 标签类型并接入通用词表
longmeidaod795bd1026c9
diff --git a/docs/REUSE.md b/docs/REUSE.mdindex 12a6b220d..0a1ba6ca7 100644--- a/docs/REUSE.md+++ b/docs/REUSE.md@@ -649,7 +649,7 @@ A 盘垃圾隔离使用短名称 `peach-purge-<32 位 UUID>.peach-quarantine`,- 关注的作者头像与来源图标是元数据:`follow_assets` 取回落在 `generated/follow-assets/`,经 `/follow-avatar`、`/source-icon` 给页面;地址只从固定表或固定主机拼,字节先认成图再写盘,到期重取失败继续用旧的并退避一小时,缓存有效期与 `/link-mark` 共用 `web_settings.metadata_refresh_seconds`。视频与图片不存本机。官方头像先认 FANBOX,没有时 `follow_avatar.profile_avatar_tiers` 取名片上的 X 与 Patreon,`follow_assets.largest_image` 留像素最大的那张。- 厂牌标识由契约位 `has_logo` 决定出不出图:没装标识的厂牌一个 `<img>` 都不发,改用首字母底板,不靠 404 摘。标识 198 张随仓库分发(ADR-0026)。- Iwara 视频作者订阅复用 [yt-dlp 的 Iwara 提取器](https://github.com/yt-dlp/yt-dlp/blob/9b58cb33d0cd3e823c8d8732f6adce8faa1f37e3/yt_dlp/extractor/iwara.py)(Unlicense,固定 revision `9b58cb33d0cd3e823c8d8732f6adce8faa1f37e3`)的 profile→用户 ID→videos 分页模型与 `X-Version` 文件接口签名;[gallery-dl 的 Iwara 提取器](https://github.com/mikf/gallery-dl/blob/882c8d0bb5c5a3ed4e88726d2f42a1cb7db59bf2/gallery_dl/extractor/iwara.py)(GPL-2.0,固定 revision `882c8d0bb5c5a3ed4e88726d2f42a1cb7db59bf2`)用于交叉核对 `apiq.iwara.tv`、视频文件字段与格式清单,不复制其实现。两者在 2026-10-23 获取时均维护中,Python 客户端可跨平台运行;完整下载器包含 Peach 不使用的下载、账户缓存与大量站点能力,因此只复用公开协议模型和 Unlicense 签名算法,不新增依赖。HTTP 复用现有 `curl_cffi`:匿名只读 POC 中普通 HTTP 对 API 返回 403,Chrome 传输取得作者资料、视频列表两页、详情与三档格式,五次请求约 4.9 KB、4.7 秒,未读取凭据或下载媒体。Peach 保留 DTO、关注状态、历史游标、凭据隔离及媒体代理;可选 `access_token` 仅发送到固定 API,不同步。列表只保存稳定视频页地址,短效文件地址与清晰度仅在播放缓存中保留;外部播放器、权限不足、空历史页、限流与结构错误分别处理。用法见 [Iwara 视频作者订阅](SOURCING.md#iwara-视频作者订阅)。-- Iwara 封面复用 `/follow-image` 的同源缓存,官方缩略图经正式 HTTPX 传输实测取得 200 `image/jpeg`;来源图标采用官网声明且实测为 PNG 的 `/logo.png`。官方头像专用解析路径未取得,作者头像沿用首字母回退;作者主页的用户名与署名复用既有官方别名学习。部分迁移视频没有可用时长或原画分辨率,字段保留未知,播放仍优先选择 `Source` 原画。+- Iwara 封面复用 `/follow-image` 的同源缓存,官方缩略图经正式 HTTPX 传输实测取得 200 `image/jpeg`;来源图标采用官网声明且实测为 PNG 的 `/logo.png`。官方头像专用解析路径未取得,作者头像沿用首字母回退;作者主页的用户名与署名复用既有官方别名学习。部分迁移视频没有可用时长或原画分辨率,字段保留未知,播放仍优先选择 `Source` 原画。标签模型另核对 [IwaraTool API 字段说明](https://github.com/Moeary/IwaraTool/blob/1e98c96c422a169699c0d16bcab49b8a7f5238d5/docs/API_zh.md)(MIT,固定 revision `1e98c96c422a169699c0d16bcab49b8a7f5238d5`,仅核对协议,不复制代码);匿名 `/tags` 与八条视频的 28 个标签均为 `id/type/sensitive` 模型。仅 `type=general` 明确进入 Peach 的通用词表,其他类型与缺少类型的原词保留,原始类型另存 `source_tag_types`,不按词形推断。- 关注检查分两阶段:列表阶段落 partial 行,详情补全按 provider 额度只补新行和未补齐行。「补齐过」由连接器的 `ENRICHED_MARK` 声明、判据登记在 `follow_store._ENRICHED_PREDICATES`。- 存量行重抓(连接器新学到图片宽高、封面这类字段时,旧行不会自己补上)分两步:把 `ENRICHED_MARK` 换成新字段落库后才有的键并登记判据;再 `POST /api/follow/check` 发 `{"older":true,"backfill_all":true,"rewind":true,"background":true}`(可加 `"sources":[id,…]`),从第 1 页走到站点说没有更多,进度读 `GET /api/follow/check`。`backfill_all`、`rewind` 离开 `older` 被拒收、不进界面,ledger 的回填游标只进不退;长跑任务,重启前按 `peach-batch-jobs` 先查。- 图片的固有宽高只问文件头:`follow_image_dims.probe_image_dims` 发 `Range: bytes=0-65535`,`dims_from_header` 认 PNG/GIF/WebP(VP8、VP8L、VP8X)/JPEG(跳过 EXIF 到 SOF);连接器、`backfill_follow_image_dims.py` 与 `/api/follow/image-dims` 落库前都经 `positive_dims` 归一,`FollowStore.set_image_dims` 只补空缺。宽高只为图片视图的卡面预留比例,回填每个条目只问卡面那张;原文件主机拦脚本(pawchive 的 `file.` 子域挂 ddos-guard)时按缩略图量,详情原图取不到也退回缩略图。diff --git a/docs/SOURCING.md b/docs/SOURCING.mdindex a5e01c0c6..73588a76b 100644--- a/docs/SOURCING.md+++ b/docs/SOURCING.md@@ -40,6 +40,8 @@在关注管理粘贴 Iwara 作者主页或作者的视频列表链接,或用 `provider=iwara` 与作者用户名登记来源。订阅按发布时间读取作者视频,支持检查更新和抓取更早内容;图片、播放列表和全站视频列表不属于这个来源。条目保留标题、作者、时间、标签、时长和封面;使用外部播放器的条目提示打开来源页面。+通用标签词表与筛选仅接纳 Iwara 明确标为 `general` 的标签。`category`、`source`、未知及未提供类型的标签保留来源原词;分类对应关系未确认时不推断为题材。+公开作者视频无需登录。有访问限制的视频须由具备访问权的账号提供有效 `access_token`,在「来源与订阅」设置中填写。令牌只保存在本机凭据文件,不进账本或同步;失效时重新登录并按设置页说明复制。Peach 不求解浏览器验证。播放时按需解析短效视频地址,默认选择最高分辨率,可切换来源提供的清晰度;私有或已删除视频、接口结构变化与限流会显示具体失败。没有更早内容是正常的历史终点。## 采集判据diff --git a/src/peach/follow_sources.py b/src/peach/follow_sources.pyindex 1ef1df208..430b31f29 100644--- a/src/peach/follow_sources.py+++ b/src/peach/follow_sources.py@@ -873,10 +873,7 @@ class IwaraConnector(_BaseConnector):if re.fullmatch(r"[A-Za-z0-9-]{1,80}", file_id):thumbnail = f"https://files.iwara.tv/image/thumbnail/{file_id}/thumbnail-00.jpg"url = f"https://www.iwara.tv/video/{video_id}"- tags = row.get("tags") if isinstance(row.get("tags"), list) else []- extra = {"media_kind": "video", "tags": [- tag["id"] for tag in tags- if isinstance(tag, dict) and isinstance(tag.get("id"), str)]}+ extra = {"media_kind": "video", **self._tag_metadata(row.get("tags"))}if row.get("embedUrl"):extra["media_error"] = "Iwara 此条目使用外部播放器,请打开来源页面观看"return FollowCandidate(@@ -888,6 +885,18 @@ class IwaraConnector(_BaseConnector):author=plain_text(str(user.get("name") or handle)),summary=plain_text(str(row.get("body") or "")) or None, extra=extra)+ @staticmethod+ def _tag_metadata(value):+ """仅接纳上游明确的 general 类型,其他类型保留原值。"""+ rows = value if isinstance(value, list) else []+ tags = [tag for tag in rows if isinstance(tag, dict)+ and isinstance(tag.get("id"), str)]+ source_types = {tag["id"]: tag["type"] for tag in tags+ if isinstance(tag.get("type"), str) and tag["type"]}+ return {"tags": [tag["id"] for tag in tags], "source_tag_types": source_types,+ "tag_types": {name: "general" for name, kind in source_types.items()+ if kind == "general"}}+@staticmethoddef _duration(value):try:diff --git a/tests/test_follow_sources.py b/tests/test_follow_sources.pyindex 4a22be8f1..db290bc85 100644--- a/tests/test_follow_sources.py+++ b/tests/test_follow_sources.py@@ -2066,7 +2066,7 @@ class ParseSourceUrlTests(unittest.TestCase):class IwaraConnectorTests(unittest.TestCase):PROFILE = {"user": {"id": "user-id", "name": "Test Creator", "username": "creator"}}VIDEO = {"id": "Video123", "title": "Test animation", "body": "Description",- "createdAt": "2026-10-01T12:00:00Z", "tags": [{"id": "animation"}],+ "createdAt": "2026-10-01T12:00:00Z", "tags": [{"id": "animation", "type": "general"}],"file": {"id": "file-id", "height": 1080, "duration": 20.5}}def connector(self, payloads, **kwargs):@@ -2104,7 +2104,9 @@ class IwaraConnectorTests(unittest.TestCase):"https://files.iwara.tv/image/thumbnail/file-id/thumbnail-00.jpg")self.assertEqual((item.author, item.duration), ("Test Creator", 20.5))self.assertEqual(item.published_at, "2026-10-01T12:00:00Z")- self.assertEqual(item.extra, {"media_kind": "video", "tags": ["animation"]})+ self.assertEqual(item.extra, {"media_kind": "video", "tags": ["animation"],+ "tag_types": {"animation": "general"},+ "source_tag_types": {"animation": "general"}})self.assertNotIn("If-None-Match", requests[0].headers)self.assertEqual(requests[1].headers["If-None-Match"], "old-version")query = urllib.parse.parse_qs(urllib.parse.urlsplit(requests[1].url).query)diff --git a/tests/test_follow_store.py b/tests/test_follow_store.pyindex e86913574..c586eac51 100644--- a/tests/test_follow_store.py+++ b/tests/test_follow_store.py@@ -8,7 +8,7 @@ from datetime import datetime, timedelta, timezonefrom pathlib import Pathfrom peach.follow import FollowSourceError-from peach.follow_sources import FollowCandidate, SourceFetch+from peach.follow_sources import FollowCandidate, IwaraConnector, SourceFetchfrom peach.follow_store import (FollowStore, ReleaseGroup, author_display_text, f95_author_handles,f95_author_name, normalized_author_name,@@ -123,6 +123,27 @@ class RegistrationTests(_StoreCase):class RecordTests(_StoreCase):+ def test_iwara_general_tags_enter_the_inventory_without_guessing_other_types(self):+ from peach.web_follow import _item_tags, online_tag_inventory++ candidate = IwaraConnector()._candidate({+ "id": "Video123", "title": "Test animation", "tags": [+ {"id": "dance", "type": "general"},+ {"id": "blender", "type": "category"},+ {"id": "series_title", "type": "source"},+ {"id": "pose", "type": "unknown"},+ {"id": "animation"}]}, {"name": "Test Creator"}, "creator")+ source_id = self._source(provider="iwara", ref="creator")+ self.store.record(source_id, _fetch([candidate], provider="iwara", ref="creator"),+ moment=MOMENT)+ rows = online_tag_inventory(self.connection)+ self.assertEqual([(row["k"], row["cat"]) for row in rows], [("dance", "general")])+ item = self.store.items()[0]+ self.assertEqual(_item_tags(item), ["dance"])+ self.assertEqual(item.metadata["source_tag_types"], {+ "dance": "general", "blender": "category", "series_title": "source", "pose": "unknown"})+ self.assertEqual(item.metadata["tags"], ["dance", "blender", "series_title", "pose", "animation"])+def test_booru_source_only_becomes_a_profile_when_the_handle_matches(self):source_id = self._source(provider="rule34xxx", ref="auxtasy", label="Auxtasy",diff --git a/tests/test_follow_stream.py b/tests/test_follow_stream.pyindex 41ce072bc..76c5b6dce 100644--- a/tests/test_follow_stream.py+++ b/tests/test_follow_stream.py@@ -67,6 +67,25 @@ class FollowMediaResolverTests(unittest.TestCase):self.assertEqual(transport.call_count, 1)self.assertEqual(resolver.cooling("iwara"), 30)+ def test_iwara_unknown_original_height_preserves_default_and_explicit_selection(self):+ transport = mock.Mock(side_effect=[+ HttpResponse(200, {}, json.dumps({+ "fileUrl": "https://files.iwara.tv/file/id?expires=2000000000",+ "file": {"height": 0}}).encode()),+ HttpResponse(200, {}, json.dumps([+ {"name": "540", "src": {"view": "//hime.iwara.tv/540.mp4"}},+ {"name": "Source", "src": {"view": "//mikoto.iwara.tv/original.mp4"}}]).encode())])+ resolver = FollowMediaResolver(transport)+ item = SimpleNamespace(id=91, provider="iwara", external_id="Video123",+ url="https://www.iwara.tv/video/Video123", metadata={})+ original = resolver.resolve(item)+ self.assertEqual(original.url, "https://mikoto.iwara.tv/original.mp4")+ self.assertEqual(original.qualities[0][0], 0)+ self.assertEqual(resolver.resolve(item, height=540).url, "https://hime.iwara.tv/540.mp4")+ self.assertEqual(resolver.resolve(item, height=0), original)+ self.assertEqual(resolver.resolve(item, height=9999), original)+ self.assertEqual(transport.call_count, 2)+def test_rule34video_resolves_the_current_signed_url_and_caches_it(self):requests = []