Skip to content

armgit / peach

Public Imported from GitHub
0

Peach: Local-first video metadata scraper and media manager.

fix(feeds): 补新作封面单独排队,在队番号不再重复派

longmeidao49ecac8588ab
diff --git a/docs/REUSE.md b/docs/REUSE.md
index ee78017b4..72ad628b3 100644
--- a/docs/REUSE.md
+++ b/docs/REUSE.md
@@ -431,7 +431,7 @@ MGS 专属发行先查自营商品目录,Prestige 番号复用 `metadata_route
传统正封完整居中,比例留白复用同图派生档的模糊补底,新作与视频流卡片使用同一份取景规则。整幅大封使用卡片纯色底,竖版按主脸填满,横版完整放入容器;图片切换时清理残留背景属性。
-取景由 `poster-crop-v13` 管理。Blu-ray 来源标题、明确的版次标签与图片比例共同决定 0.85 正封比例,标题随正封框保留,重算人脸与正封框均读取同一版次;有效宽正封框在没有版次标题的卡片上也完整显示。VR 标记、IENEE、IENFE、SDHS 数字发行拼图,以及可信主脸跨过 DVD 正封先验边界的大封保留整幅画面,由 `cover_frame.layout=still` 传给卡片。两条满高接缝夹出居中正封且两侧纹理均低于正封四分之一时,只显示中间正封。大封在竖版大图按主脸居中填满,横版小图完整显示,均不使用模糊补边;JPEG 软边的书脊识别结合突出度、左忙右静的方向与正封侧基线。手工框保持用户取景。重试队列按最早尝试时间轮转,收起的合集沿用设置跳过;逐番号结果保存在 `feed-cover-attempts/`。复用现有 HTTP、Pillow、OpenCV 与文件锁,无新增依赖。
+取景由 `poster-crop-v13` 管理。Blu-ray 来源标题、明确的版次标签与图片比例共同决定 0.85 正封比例,标题随正封框保留,重算人脸与正封框均读取同一版次;有效宽正封框在没有版次标题的卡片上也完整显示。VR 标记、IENEE、IENFE、SDHS 数字发行拼图,以及可信主脸跨过 DVD 正封先验边界的大封保留整幅画面,由 `cover_frame.layout=still` 传给卡片。两条满高接缝夹出居中正封且两侧纹理均低于正封四分之一时,只显示中间正封。大封在竖版大图按主脸居中填满,横版小图完整显示,均不使用模糊补边;JPEG 软边的书脊识别结合突出度、左忙右静的方向与正封侧基线。手工框保持用户取景。重试队列按最早尝试时间轮转,收起的合集沿用设置跳过;逐番号结果保存在 `feed-cover-attempts/`。资料已齐、只缺封面的新作另派 `feed-cover`(补新作封面)批次,只取图、不写账本,走独立后继通道;排在队里或正在跑的番号不再进下一批。复用现有 HTTP、Pillow、OpenCV 与文件锁,无新增依赖。
馆藏与新作取图共用 `cover_artwork.metadata_snapshots` 和 `best_cover`,片商官网、发行方、R18 与 DMM 的官方候选均参与尺寸比较。馆藏采集将长边 1600px 作为高清重探界限,社区转存图可由官方封面替换;同类官方图按像素面积升级,FC2 保留商品图的来源保护。`scripts/audit_cover_artwork.py` 逐张核对尺寸、来源、版次与取景框,输出 CSV;`scripts/upgrade_audited_covers.py` 按已有官方快照逐番号重探,`--live` 经同一官方链在线补查,保存原件备份与结果检查点。两者只处理可重建图片和边车,不写 ledger。
diff --git a/docs/adr/0042-feeds-discover-codes-not-files.md b/docs/adr/0042-feeds-discover-codes-not-files.md
index 258bbea03..f36c972de 100644
--- a/docs/adr/0042-feeds-discover-codes-not-files.md
+++ b/docs/adr/0042-feeds-discover-codes-not-files.md
@@ -61,8 +61,10 @@ sukebei 两个响应头一个都不给(2026-09-22 实测)。304 只是省一
取消忽略只恢复可见性,不自动派刮削。两列都不参与任何唯一约束。
**六、刮削走现有来源链,由后继机制派。** 一轮拉取按 ADR-0040 声明一条后继
-(`feed-scrape:<番号>,<番号>,…`),带上这一轮新建的壳,余下名额给还缺资料或本机封面、
-距上次尝试已过 12 小时的旧壳(手动点「检查」的那一轮不等这 12 小时);活动页上一轮订阅
+(`feed-scrape:<番号>,<番号>,…`),带上这一轮新建的壳,余下名额给没取到资料、
+距上次尝试已过 12 小时的旧壳(手动点「检查」的那一轮不等这 12 小时)。资料已齐、只缺本机封面的
+旧壳另成一条 `feed-cover:<番号>,…`,按上次取图由远到近排,只取图、不写账本,走自己的后继通道;
+已在队里或正在跑的番号不再进下一批;活动页上一轮订阅
因此是一行,不是一部一行。后继内部用
`metadata_routes.route_for_code` 算链、用 `library_processing.LibraryMetadataProvider` 发请求,
取到的标题、女优、厂牌、发行日与封面地址写回壳自己的列。**不复用 `process_library`**:
diff --git a/src/peach/feed_followup.py b/src/peach/feed_followup.py
index 898069035..e9848ef5b 100644
--- a/src/peach/feed_followup.py
+++ b/src/peach/feed_followup.py
@@ -6,6 +6,11 @@
**一轮拉取一条后继。** 同一轮发现的番号在活动页上是一件事,拆成一部一条,一次订阅
就是二十张一模一样的卡片;键里带着这一批的番号,重跑、去重都按这一批算。
+**补封面单独排队。** 资料已经齐、只缺本机封面的壳不进取资料那一批,而是另派一条
+`feed-cover`(补新作封面):它只取图、不问资料来源、不写账本,走自己的通道。资料来源
+连不上时取资料每部要等一轮超时,补封面若跟在后面,旧壳会被源源不断的新番号挤到轮不上。
+排在队里或正在跑的番号(`in_flight`)不再进下一批,两种批次也互不重复。
+
**不复用 `process_library`**:那条路以 `asset` 行为单位、要写候选 CSV、要按字段优先级链
结算到真相字段,而壳没有真相字段可写。壳上的每个值都只是「某个来源这么说」,用户复核
的时机是文件真的到手、建成 `asset` 之后。所以这里只借来源链本身
@@ -35,12 +40,16 @@ from .metadata import extract_peach_fields
TASK_KEY = "feed-scrape"
TASK_LABEL = "取新作资料"
+COVER_TASK_KEY = "feed-cover"
+COVER_TASK_LABEL = "补新作封面"
+LABELS = {TASK_KEY: TASK_LABEL, COVER_TASK_KEY: COVER_TASK_LABEL}
#: 一条资料够用就停。四个标量都拿到就不再问下一家——Feed 是后台跑的,
#: 多问一家换来的是配额,而壳上的值本来就不是真相字段。
WANTED = ("title", "performers", "studio", "release_date")
-#: 一批最多带几部。新发现的先排,余下的名额给还缺资料或封面、到了重试时间的旧壳。
+#: 一批最多带几部。取资料那一批新发现的先排,余下的名额给上次没取到资料、到了重试时间的
+#: 旧壳;补封面那一批按上次尝试由远到近排。
MAX_BATCH = 24
#: 缺封面或上次没取到资料的壳,定时拉取隔多久再试一次。新片的封面常常晚于番号出现,
@@ -48,34 +57,52 @@ MAX_BATCH = 24
#: 点「检查」本身就是要现在取。
RETRY_AFTER = timedelta(hours=12)
-#: 一部封面最多等多久。后继在写账本那条串行通道上,一部卡住不能拖住后面整批。
+#: 一部封面最多等多久。一部卡住不能拖住后面整批。
COVER_SECONDS = 30
-def followup_key(codes) -> str:
- return f"{TASK_KEY}:{','.join(codes)}"
+def followup_key(codes, task_key: str = TASK_KEY) -> str:
+ return f"{task_key}:{','.join(codes)}"
-def parse_key(key: str) -> list[str]:
+def parse_key(key: str, task_key: str = TASK_KEY) -> list[str]:
prefix, _, body = str(key).partition(":")
codes = [code for code in body.split(",") if code]
- if prefix != TASK_KEY or not codes:
- raise ValueError(f"认不出这条取资料后继:{key}")
+ if prefix != task_key or not codes:
+ raise ValueError(f"认不出这条{LABELS.get(task_key, task_key)}后继:{key}")
return codes
-def _label(codes: list[str]) -> str:
+def _label(codes: list[str], task_key: str = TASK_KEY) -> str:
+ name = LABELS[task_key]
if len(codes) == 1:
- return f"{TASK_LABEL}:{codes[0]}"
- return f"{TASK_LABEL}:{codes[0]} 等 {len(codes)} 部"
+ return f"{name}:{codes[0]}"
+ return f"{name}:{codes[0]} 等 {len(codes)} 部"
-def plan(codes) -> list[Followup]:
+def plan(codes, task_key: str = TASK_KEY) -> list[Followup]:
"""一批番号合成一条后继;一部都没有就不派。"""
batch = list(dict.fromkeys(code for code in codes if code))[:MAX_BATCH]
if not batch:
return []
- return [Followup(key=followup_key(batch), task_key=TASK_KEY, label=_label(batch))]
+ return [Followup(key=followup_key(batch, task_key), task_key=task_key,
+ label=_label(batch, task_key))]
+
+
+def plan_covers(codes) -> list[Followup]:
+ return plan(codes, COVER_TASK_KEY)
+
+
+def in_flight(task_runs) -> set[str]:
+ """排在队里或正在跑的取资料、补封面后继里的番号键。下一批不再带它们。"""
+ codes: set[str] = set()
+ for task_key in (TASK_KEY, COVER_TASK_KEY):
+ for run in task_runs.query(status="active", task_key=task_key, limit=500):
+ try:
+ codes.update(normalise_code_key(code) for code in parse_key(run.followup_key, task_key))
+ except ValueError:
+ continue
+ return codes
def has_local_cover(cover_root: Path, code: str) -> bool:
@@ -87,36 +114,72 @@ def has_local_cover(cover_root: Path, code: str) -> bool:
def backlog(connection, cover_root: Path, *, exclude=(), now: datetime | None = None,
limit: int = MAX_BATCH, retry_after: timedelta = RETRY_AFTER,
hidden: frozenset[str] = feeds.DEFAULT_HIDDEN_COMPILATIONS) -> list[str]:
- """还缺资料或本机封面、上次尝试已经过了 `retry_after` 的壳。
+ """还没取过资料,或上次没取到、已经过了 `retry_after` 的壳。资料齐了只缺封面的归
+ `cover_backlog`。
已忽略的不算:人说了不想看,就不再替它花配额。已经入库的壳本来就不在表里
(入库即从列表里消失,ADR-0042 第三条),这里同样用番号键排除掉。取回标题后认出是
- 设置里收起的那类合集(`hidden`)也不算:页面上不列它,替它补封面的配额是白花的;
+ 设置里收起的那类合集(`hidden`)也不算:页面上不列它,替它补资料的配额是白花的;
开关打开后下一轮就轮到它。
"""
moment = now or datetime.now(timezone.utc)
cutoff = feeds.stamp(moment - retry_after)
- skip = set(exclude)
- rows = connection.execute(
- "SELECT d.code,d.title,d.performers,d.studio,d.scraped_at,d.scrape_error FROM feed_discovery d"
- " WHERE d.ignored_at IS NULL AND (d.scraped_at IS NULL OR d.scraped_at<?)"
+ rows = _candidates(connection,
+ "(d.scraped_at IS NULL OR (d.scrape_error IS NOT NULL AND d.scraped_at<?))",
+ (cutoff,), "COALESCE(d.scraped_at,''), ")
+ found: list[str] = []
+ for row in _visible(rows, exclude, hidden):
+ found.append(row["code"])
+ if len(found) >= limit:
+ break
+ return found
+
+
+def cover_backlog(connection, cover_root: Path, *, exclude=(), now: datetime | None = None,
+ limit: int = MAX_BATCH, retry_after: timedelta = RETRY_AFTER,
+ hidden: frozenset[str] = feeds.DEFAULT_HIDDEN_COMPILATIONS) -> list[str]:
+ """资料已经齐、本机还没有封面、上次取图已经过了 `retry_after` 的壳。
+
+ 上次取图的时间读逐番号的取图记录(`record_cover_attempt`),没试过的排最前,其余由远到近;
+ 同样远的新片在前。排除规则与 `backlog` 相同。
+ """
+ from .catalog_rules import is_korean_mib_code
+
+ cutoff = (now or datetime.now(timezone.utc)).timestamp() - retry_after.total_seconds()
+ rows = _candidates(connection, "d.scraped_at IS NOT NULL AND d.scrape_error IS NULL", (), "")
+ due: list[tuple[float, int, str]] = []
+ for position, row in enumerate(_visible(rows, exclude, hidden)):
+ code = row["code"]
+ key = normalise_code_key(code)
+ if not key or is_korean_mib_code(key) or has_local_cover(cover_root, code):
+ continue
+ try:
+ attempted = cover_attempt_path(cover_root, key).stat().st_mtime
+ except OSError:
+ attempted = 0.0
+ if attempted < cutoff:
+ due.append((attempted, position, code))
+ return [code for _, _, code in sorted(due)[:limit]]
+
+
+def _candidates(connection, condition: str, values: tuple, order: str):
+ return connection.execute(
+ "SELECT d.code,d.title,d.performers,d.studio FROM feed_discovery d"
+ f" WHERE d.ignored_at IS NULL AND {condition}"
" AND NOT EXISTS (SELECT 1 FROM asset a WHERE a.code IS NOT NULL"
" AND COALESCE(a.disposal,'')<>'vanished'"
" AND normalise_code_key(a.code)=normalise_code_key(d.code))"
- " ORDER BY COALESCE(d.scraped_at,''), COALESCE(d.release_date,d.discovered_at) DESC, d.id DESC",
- (cutoff,)).fetchall()
- found: list[str] = []
+ f" ORDER BY {order}COALESCE(d.release_date,d.discovered_at) DESC, d.id DESC",
+ values).fetchall()
+
+
+def _visible(rows, exclude, hidden):
+ skip = {normalise_code_key(code) for code in exclude}
for row in rows:
- code = row["code"]
- if code in skip or feeds.compilation_kind(
+ if normalise_code_key(row["code"]) in skip or feeds.compilation_kind(
row["title"], row["performers"], row["studio"]) in hidden:
continue
- needs_fields = row["scraped_at"] is None or row["scrape_error"] is not None
- if needs_fields or not has_local_cover(cover_root, code):
- found.append(code)
- if len(found) >= limit:
- break
- return found
+ yield row
def _cover(payload: dict) -> str | None:
@@ -195,10 +258,15 @@ def fetch_cover(contract, code: str,
return data, size, evidence
+def cover_attempt_path(cover_root: Path, code: str) -> Path:
+ return Path(cover_root).parent / 'feed-cover-attempts' / (normalise_code_key(code) + '.json')
+
+
def record_cover_attempt(contract, code: str, *, ok: bool, error=None):
- """逐番号保存封面处理结果;不记录请求凭据或来源返回正文。"""
+ """逐番号保存封面处理结果;不记录请求凭据或来源返回正文。文件时间就是 `cover_backlog`
+ 判重试的上次取图时间。"""
from .fsutil import atomic_write_text
- path = contract.cover_root.parent / 'feed-cover-attempts' / (normalise_code_key(code) + '.json')
+ path = cover_attempt_path(contract.cover_root, code)
path.parent.mkdir(parents=True, exist_ok=True)
record = {'code': code, 'checked_at': time.time(), 'ok': ok}
if error is not None:
@@ -327,7 +395,7 @@ def run(contract, key: str, handle) -> dict:
unreachable += 1
except Exception: # noqa: BLE001 - 取不到封面这一部照样有资料,下一轮再试
pass
- # 封面取没取到都记下这一次尝试,`backlog` 按它隔 `RETRY_AFTER` 再试。
+ # 记下这一次尝试,没取到资料的 `backlog` 按它隔 `RETRY_AFTER` 再试;封面的重试另看取图记录。
with contract.database.write_transaction() as connection:
connection.execute("UPDATE feed_discovery SET scraped_at=? WHERE id=?",
(feeds.stamp(), int(row["id"])))
@@ -340,6 +408,48 @@ def run(contract, key: str, handle) -> dict:
return summary
+def run_covers(contract, key: str, handle) -> dict:
+ """跑一批补封面:只取图装进本机封面目录,不问资料来源、不写账本。摘要同取资料那一行,
+ 连不上图片主机的几部同样另记 `cover_network`。"""
+ import httpx
+
+ from .jav_cover_fetch import CoverConnectError
+
+ codes = parse_key(key, COVER_TASK_KEY)
+ saved = getattr(handle, 'saved', {})
+ saved = saved if isinstance(saved, dict) else {}
+ done = set(saved.get('done_codes') or [])
+ covers = saved.get('covers', 0)
+ unreachable = saved.get('cover_network', 0)
+ for index, code in enumerate(codes):
+ if code in done:
+ continue
+ handle.progress(current=index, total=len(codes),
+ label=f"{COVER_TASK_LABEL}:{code}", throttle=0)
+ with contract.database.read_connection() as connection:
+ row = connection.execute(
+ "SELECT cover_url FROM feed_discovery WHERE code=? AND ignored_at IS NULL",
+ (code,)).fetchone()
+ if row is not None:
+ try:
+ covers += ensure_cover(contract, code, row["cover_url"])
+ except (CoverConnectError, httpx.TransportError):
+ unreachable += 1
+ except Exception: # noqa: BLE001 - 取图记录里有原因,到了重试时间再排
+ pass
+ done.add(code)
+ handle.checkpoint(done_codes=sorted(done), covers=covers, cover_network=unreachable)
+ handle.progress(current=len(codes), total=len(codes), label=COVER_TASK_LABEL, throttle=0)
+ contract.cache_bust()
+ summary = {"total": len(codes), "covers": covers}
+ if unreachable:
+ summary["cover_network"] = unreachable
+ return summary
+
+
#: 写账本:壳与实体关联都在 `ledger.db` 里,所以它走串行那条通道。
TYPE = register(FollowupType(task_key=TASK_KEY, label=TASK_LABEL,
writes_ledger=True, run=run))
+#: 补封面只写封面目录与取图记录,走自己的通道,不排在写账本的后继后面。
+COVER_TYPE = register(FollowupType(task_key=COVER_TASK_KEY, label=COVER_TASK_LABEL,
+ writes_ledger=False, run=run_covers))
diff --git a/src/peach/task_runs.py b/src/peach/task_runs.py
index 2f0313a53..64f9c452a 100644
--- a/src/peach/task_runs.py
+++ b/src/peach/task_runs.py
@@ -105,6 +105,7 @@ TASK_LABELS = {
"code-samples": "补番号样张",
"feed-check": "订阅源拉取",
"feed-scrape": "取新作资料",
+ "feed-cover": "补新作封面",
"want-scrape": "取想要的资料",
"timeline-thumbnails": "视频缩略图采集",
"download-postprocess": "云下载后处理",
diff --git a/src/peach/web_feeds.py b/src/peach/web_feeds.py
index f3e193acb..7b679c48d 100644
--- a/src/peach/web_feeds.py
+++ b/src/peach/web_feeds.py
@@ -201,15 +201,20 @@ def _execute_check(contract, body, job_id: str) -> dict:
if close:
close()
contract.cache_bust()
- # 新发现的先排,余下名额给还缺资料或封面的旧壳:同一批一条后继。定时那一轮只带
- # 到了重试时间的,手动点的那一轮全带。
+ # 取资料:新发现的先排,余下名额给没取到资料的旧壳,同一批一条后继。只缺封面的另成一批
+ # 补封面。定时那一轮只带到了重试时间的,手动点的那一轮全带;已在队里的番号都不再带。
retry_after = feed_followup.RETRY_AFTER if body.get("automatic") else timedelta(0)
hidden = web_settings.hidden_compilations(contract.database)
+ busy = feed_followup.in_flight(contract.task_runs)
+ fresh = [code for code in dict.fromkeys(codes) if normalise_code_key(code) not in busy]
with contract.database.read_connection() as connection:
- batch = codes + feed_followup.backlog(
- connection, contract.cover_root, exclude=codes, retry_after=retry_after,
- limit=max(0, feed_followup.MAX_BATCH - len(codes)), hidden=hidden)
- planned = feed_followup.plan(batch)
+ batch = fresh + feed_followup.backlog(
+ connection, contract.cover_root, exclude=[*codes, *busy], retry_after=retry_after,
+ limit=max(0, feed_followup.MAX_BATCH - len(fresh)), hidden=hidden)
+ covers = feed_followup.cover_backlog(
+ connection, contract.cover_root, exclude=[*codes, *batch, *busy],
+ retry_after=retry_after, hidden=hidden)
+ planned = feed_followup.plan(batch) + feed_followup.plan_covers(covers)
# 余下的名额给缺头像的存量女优(ADR-0097)。资料处理收尾也派这一份,但没有新片
# 入库的日子它一轮都不跑,存量就一直停在原地;定时拉取每天都跑。
planned += avatar_followup.stock(
@@ -556,16 +561,13 @@ def q_feed_discoveries(contract, args) -> dict:
wanted = wants.wanted_keys(connection, (normalise_code_key(row["code"]) for row in rows[:limit]))
performers = _discovery_performers(connection, contract, rows[:limit])
more = len(rows) > limit
- # 最近一轮取新作资料里封面断在连接上的部数。只看最近这一轮:换了线路之后下一轮取到,
- # 提示就该跟着消失,而不是让一次旧故障一直挂在页面上。
- last = contract.task_runs.query(status="succeeded", task_key=feed_followup.TASK_KEY, limit=1)
- cover_network = int(last[0].result_summary.get("cover_network") or 0) if last else 0
- fetching = set()
- for run in contract.task_runs.query(status="active", task_key=feed_followup.TASK_KEY, limit=500):
- try:
- fetching.update(normalise_code_key(code) for code in feed_followup.parse_key(run.followup_key))
- except ValueError:
- continue
+ # 最近一轮取新作资料或补新作封面里封面断在连接上的部数。只看最近这一轮:换了线路之后
+ # 下一轮取到,提示就该跟着消失,而不是让一次旧故障一直挂在页面上。
+ last = [run for task_key in (feed_followup.TASK_KEY, feed_followup.COVER_TASK_KEY)
+ for run in contract.task_runs.query(status="succeeded", task_key=task_key, limit=1)]
+ latest = max(last, key=lambda run: run.finished_at or "", default=None)
+ cover_network = int(latest.result_summary.get("cover_network") or 0) if latest else 0
+ fetching = feed_followup.in_flight(contract.task_runs)
return {"ok": True, "more": more, "cover_network": cover_network, "items": [{
"id": int(row["id"]),
"code": row["code"],
diff --git a/tests/test_feeds_web.py b/tests/test_feeds_web.py
index 0d38585df..98e5aaeac 100644
--- a/tests/test_feeds_web.py
+++ b/tests/test_feeds_web.py
@@ -3,11 +3,12 @@ from __future__ import annotations
import io
import json
+import os
import sys
import tempfile
import time
import unittest
-from datetime import datetime, timedelta, timezone
+from datetime import timedelta
from pathlib import Path
from unittest.mock import patch
@@ -629,15 +630,66 @@ class FeedWebTest(FeedWebFixture):
target.write_text(json.dumps({'layout': 'still', 'face': None}), encoding='utf-8')
self.assertEqual(self.contract._cover_focus(str(target)), {'layout': 'still'})
+ def _attempted(self, code: str, seconds_ago: float) -> None:
+ path = feed_followup.cover_attempt_path(self.contract.cover_root, code)
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(json.dumps({"code": code, "ok": False}), encoding="utf-8")
+ moment = time.time() - seconds_ago
+ os.utime(path, (moment, moment))
+
+ def _cover_backlog(self, **options) -> list[str]:
+ with self.contract.database.read_connection() as connection:
+ return feed_followup.cover_backlog(connection, self.contract.cover_root, hidden=frozenset(),
+ retry_after=timedelta(0), **options)
+
def test_cover_backlog_prioritises_the_least_recent_attempt(self):
self._add()
self._check()
with self.contract.database.write_transaction() as connection:
connection.execute("UPDATE feed_discovery SET scraped_at='2026-01-01T00:00:00Z',release_date='2026-10-01' WHERE code='SSIS-950'")
- connection.execute("UPDATE feed_discovery SET scraped_at='2025-01-01T00:00:00Z',release_date='2025-01-01' WHERE code='HMN-071'")
+ connection.execute("UPDATE feed_discovery SET scraped_at='2026-01-01T00:00:00Z',release_date='2025-01-01' WHERE code='HMN-071'")
+ self._attempted('SSIS-950', 3600)
+ self._attempted('HMN-071', 86400)
+ self.assertEqual(self._cover_backlog(limit=1), ['HMN-071'])
+ # 资料没取到的壳归取资料那一批,不进补封面。
+ with self.contract.database.write_transaction() as connection:
+ connection.execute("UPDATE feed_discovery SET scrape_error='来源连接未取得' WHERE code='HMN-071'")
+ self.assertEqual(self._cover_backlog(), ['SSIS-950'])
with self.contract.database.read_connection() as connection:
- result = feed_followup.backlog(connection, self.contract.cover_root, hidden=frozenset(), limit=1)
- self.assertEqual(result, ['HMN-071'])
+ self.assertEqual(feed_followup.backlog(connection, self.contract.cover_root, hidden=frozenset(),
+ retry_after=timedelta(0)), ['HMN-071'])
+
+ def test_codes_already_queued_are_not_planned_again(self):
+ """在队里的那一批还没跑完,下一轮检查只带新的番号,不把同一批换个组合再派一条。"""
+ self._add()
+ self.assertEqual([item["key"] for item in self._check()["followups"]],
+ ["feed-scrape:SSIS-950,HMN-071"])
+ with self.contract.database.write_transaction() as connection:
+ connection.execute("INSERT INTO feed_discovery(code,source_id,title,link,release_date,discovered_at)"
+ " VALUES('ABP-123',1,'',NULL,'2026-09-22',?)", (feeds.stamp(),))
+ self.transport.responses[self.url] = HttpResponse(304, {}, b"", self.url)
+ thread = self.contract.feed_job.thread
+ if thread is not None:
+ thread.join(5)
+ self.assertEqual([item["key"] for item in self._check()["followups"]], ["feed-scrape:ABP-123"])
+
+ def test_a_missing_cover_gets_its_own_batch_outside_the_ledger_lane(self):
+ """资料齐了只缺封面的壳另派补封面:只取图,不再问资料来源,也不排在写账本的后继后面。"""
+ self.cover_misses.add("HMN-071")
+ self._add()
+ self._check()
+ self._drain(1)
+ self.cover_misses.clear()
+ self.transport.responses[self.url] = HttpResponse(304, {}, b"", self.url)
+ asked = []
+ feed_followup.collect = lambda provider, code: asked.append(code)
+ self.assertEqual([item["key"] for item in self._check()["followups"]], ["feed-cover:HMN-071"])
+ self.assertFalse(feed_followup.COVER_TYPE.writes_ledger)
+ self._drain(1)
+ self.assertEqual(asked, [])
+ self.assertTrue(feed_followup.has_local_cover(self.contract.cover_root, "HMN-071"))
+ [run] = self.contract.task_runs.query(task_key=feed_followup.COVER_TASK_KEY)
+ self.assertEqual(run.result_summary, {"total": 1, "covers": 1})
def test_covers_cut_off_at_the_connection_are_counted_for_the_new_release_row(self):
"""连不上图片主机的几部单独记数,列表把最近一轮的数带给页面去提示换线路;
@@ -670,12 +722,9 @@ class FeedWebTest(FeedWebFixture):
self.transport.responses[self.url] = HttpResponse(304, {}, b"", self.url)
# 定时那一轮:刚试过的不再排;已经有封面、资料也齐的那部永远不再排。
self.assertEqual(self._check(automatic=True)["followups"], [])
- earlier = feeds.stamp(datetime.now(timezone.utc)
- - feed_followup.RETRY_AFTER - timedelta(minutes=1))
- with self.contract.database.write_transaction() as connection:
- connection.execute("UPDATE feed_discovery SET scraped_at=?", (earlier,))
+ self._attempted("HMN-071", feed_followup.RETRY_AFTER.total_seconds() + 60)
self.assertEqual([item["key"] for item in self._check(automatic=True)["followups"]],
- ["feed-scrape:HMN-071"])
+ ["feed-cover:HMN-071"])
# 人说了不想看的,不再替它花配额。
first = {row["code"]: row for row in dispatch_api_get(
self.contract, "/api/feeds/discoveries", {})["items"]}["HMN-071"]
@@ -694,7 +743,7 @@ class FeedWebTest(FeedWebFixture):
attempt.read_text(encoding='utf-8') if attempt.exists() else '未取得封面处理记录')
self.transport.responses[self.url] = HttpResponse(304, {}, b"", self.url)
self.assertEqual([item["key"] for item in self._check()["followups"]],
- ["feed-scrape:HMN-071"])
+ ["feed-cover:HMN-071"])
def test_every_check_carries_performers_still_missing_an_avatar(self):
"""资料处理不跑的日子,缺头像的存量女优跟着定时拉取补(ADR-0097)。试过的不再带。"""