Skip to content

armgit / peach

Public Imported from GitHub
0

Peach: Local-first video metadata scraper and media manager.

fix(queues): 取资料限时、旧壳保底、下载后处理先领、来源限速共用

longmeidaof87c0e6dfe04
diff --git a/docs/SOURCING.md b/docs/SOURCING.md
index ad4e81f35..0c4e9a027 100644
--- a/docs/SOURCING.md
+++ b/docs/SOURCING.md
@@ -404,7 +404,8 @@ minnano-av 也在 Cloudflare 后面,但对 HTTP 客户端是间歇拦截:202
### 库内采集的节奏
- javdb 的主机间隔由用户定:`library_processing.SOURCE_INTERVALS` 里 `javdb.com` 与 `jdbstatic.com` 都是 3 秒,
- 两个主机要一起改。
+ 两个主机要一起改。这份间隔由 `SOURCE_LIMITER` 整个进程共用:资料处理、取想要的资料、取新作资料与订阅检查
+ 同时跑时,问同一主机仍按一条队排。
- **FC2 的商品号在三家里只问 javdb**(`community_catalog.community_sources_for`)。本机来源证据里 AVBase 与 JavBus
对 FC2 一份都没给过;javdb 能补约一半的演员与发行日,厂牌、标签、封面不给,所以它问得慢却砍不掉。
- 一轮采集的长短约等于 javdb 请求数乘这个间隔(实测相差约一成)。`HostLimiter` 等的是「距上次满 N 秒」,别家的
diff --git a/docs/adr/0042-feeds-discover-codes-not-files.md b/docs/adr/0042-feeds-discover-codes-not-files.md
index f36c972de..a46910a53 100644
--- a/docs/adr/0042-feeds-discover-codes-not-files.md
+++ b/docs/adr/0042-feeds-discover-codes-not-files.md
@@ -61,8 +61,9 @@ sukebei 两个响应头一个都不给(2026-09-22 实测)。304 只是省一
取消忽略只恢复可见性,不自动派刮削。两列都不参与任何唯一约束。
**六、刮削走现有来源链,由后继机制派。** 一轮拉取按 ADR-0040 声明一条后继
-(`feed-scrape:<番号>,<番号>,…`),带上这一轮新建的壳,余下名额给没取到资料、
-距上次尝试已过 12 小时的旧壳(手动点「检查」的那一轮不等这 12 小时)。资料已齐、只缺本机封面的
+(`feed-scrape:<番号>,<番号>,…`),一批 24 部:这一轮新建的壳与还没取过资料的壳在前,
+没取到资料、距上次尝试已过 12 小时的旧壳在后,并且至少留 6 个名额给后者(手动点「检查」的那一轮
+不等这 12 小时)。一部最多问 90 秒,与媒体库资料处理同一份预算。资料已齐、只缺本机封面的
旧壳另成一条 `feed-cover:<番号>,…`,按上次取图由远到近排,只取图、不写账本,走自己的后继通道;
已在队里或正在跑的番号不再进下一批;活动页上一轮订阅
因此是一行,不是一部一行。后继内部用
diff --git a/docs/adr/0114-followups-a-download-goes-first-in-the-ledger-lane.md b/docs/adr/0114-followups-a-download-goes-first-in-the-ledger-lane.md
new file mode 100644
index 000000000..bcb0f9a90
--- /dev/null
+++ b/docs/adr/0114-followups-a-download-goes-first-in-the-ledger-lane.md
@@ -0,0 +1,26 @@
+# ADR-0114:云下载后处理在账本通道里先领
+
+- 状态:已采纳
+- 日期:2026-10-22
+- 修订:ADR-0040「未决」里的后继优先级
+
+## 背景
+
+写账本的后继共用一条 `ledger` 通道,一次只跑一条(ADR-0040 第二条),按排队先后领取。
+这条通道上有九种后继,其中取新作资料(ADR-0042 第六条)一批 24 部,来源连不上时每部要等满资料预算,
+排着几批时要跑一到几个小时。云下载后处理排在它们后面,下好的文件要等这些后台批次跑完才认正片、
+写番号、整理入库,而这正是用户刚做完、在等结果的那件事。
+
+## 决策
+
+**一、后继类型可以声明 `jumps_queue`。** 带这一项的后继在自己的通道里先于别的种类领取,
+同一档里仍按 id 先排先跑。在途那一条不打断,插队只决定下一条领谁。
+
+**二、目前只有云下载后处理插队。** 判据是「人刚做完一件事、在等它收尾」;补资料、补头像、
+补样张这类后台存量不插队。新增插队的种类要能说出用户在等什么。
+
+## 后果
+
+- 下载后处理最多等通道上在途的那一条跑完,不再等整队后台批次。
+- 云下载连续涌入时,后台批次会被推后;下载是用户一部部发起的,量有上限,不会把后台批次饿死。
+- 通道仍是串行,账本写锁的约束不变。
diff --git a/src/peach/download_postprocess.py b/src/peach/download_postprocess.py
index 758ab7035..bc6095568 100644
--- a/src/peach/download_postprocess.py
+++ b/src/peach/download_postprocess.py
@@ -203,4 +203,4 @@ def run(contract, key: str, handle) -> dict:
TYPE = register(FollowupType(task_key=TASK_KEY, label=TASK_LABEL,
- writes_ledger=True, run=run))
+ writes_ledger=True, run=run, jumps_queue=True))
diff --git a/src/peach/feed_followup.py b/src/peach/feed_followup.py
index e9848ef5b..a85f1934d 100644
--- a/src/peach/feed_followup.py
+++ b/src/peach/feed_followup.py
@@ -52,6 +52,10 @@ WANTED = ("title", "performers", "studio", "release_date")
#: 旧壳;补封面那一批按上次尝试由远到近排。
MAX_BATCH = 24
+#: 取资料那一批至少留给重试旧壳的名额。订阅一轮能发现几十部新番号,不留就一部旧壳也
+#: 轮不到:没取到资料的那些会一直停在页面上没有标题。
+RETRY_SHARE = 6
+
#: 缺封面或上次没取到资料的壳,定时拉取隔多久再试一次。新片的封面常常晚于番号出现,
#: 所以不是试一次就放弃;但也不该每拉一轮源就把同一批再问一遍。人手点的那一次不等:
#: 点「检查」本身就是要现在取。
@@ -111,11 +115,12 @@ def has_local_cover(cover_root: Path, code: str) -> bool:
return bool(key) and cover_settled(Path(cover_root) / f"{key}.jpg")
-def backlog(connection, cover_root: Path, *, exclude=(), now: datetime | None = None,
- limit: int = MAX_BATCH, retry_after: timedelta = RETRY_AFTER,
- hidden: frozenset[str] = feeds.DEFAULT_HIDDEN_COMPILATIONS) -> list[str]:
- """还没取过资料,或上次没取到、已经过了 `retry_after` 的壳。资料齐了只缺封面的归
- `cover_backlog`。
+def scrape_batch(connection, fresh=(), *, exclude=(), now: datetime | None = None,
+ retry_after: timedelta = RETRY_AFTER,
+ hidden: frozenset[str] = feeds.DEFAULT_HIDDEN_COMPILATIONS) -> list[str]:
+ """下一批取资料的番号:这一轮新发现的 `fresh` 与还没取过资料的壳在前,上次没取到、
+ 已经过了 `retry_after` 的壳在后,并且至少留 `RETRY_SHARE` 个名额给后者。资料齐了只缺
+ 封面的归 `cover_backlog`。
已忽略的不算:人说了不想看,就不再替它花配额。已经入库的壳本来就不在表里
(入库即从列表里消失,ADR-0042 第三条),这里同样用番号键排除掉。取回标题后认出是
@@ -124,9 +129,17 @@ def backlog(connection, cover_root: Path, *, exclude=(), now: datetime | None =
"""
moment = now or datetime.now(timezone.utc)
cutoff = feeds.stamp(moment - retry_after)
- rows = _candidates(connection,
- "(d.scraped_at IS NULL OR (d.scrape_error IS NOT NULL AND d.scraped_at<?))",
- (cutoff,), "COALESCE(d.scraped_at,''), ")
+ fresh = list(dict.fromkeys(code for code in fresh if code))
+ skip = [*exclude, *fresh]
+ untried = _first(_candidates(connection, "d.scraped_at IS NULL", (), ""),
+ skip, hidden, MAX_BATCH)
+ retries = _first(_candidates(connection, "d.scrape_error IS NOT NULL AND d.scraped_at<?",
+ (cutoff,), "d.scraped_at, "), skip, hidden, MAX_BATCH)
+ head = [*fresh, *untried][:MAX_BATCH - min(len(retries), RETRY_SHARE)]
+ return head + retries[:MAX_BATCH - len(head)]
+
+
+def _first(rows, exclude, hidden, limit: int) -> list[str]:
found: list[str] = []
for row in _visible(rows, exclude, hidden):
found.append(row["code"])
@@ -141,7 +154,7 @@ def cover_backlog(connection, cover_root: Path, *, exclude=(), now: datetime | N
"""资料已经齐、本机还没有封面、上次取图已经过了 `retry_after` 的壳。
上次取图的时间读逐番号的取图记录(`record_cover_attempt`),没试过的排最前,其余由远到近;
- 同样远的新片在前。排除规则与 `backlog` 相同。
+ 同样远的新片在前。排除规则与 `scrape_batch` 相同。
"""
from .catalog_rules import is_korean_mib_code
@@ -202,10 +215,24 @@ def collect(provider, code: str) -> tuple[dict, str | None]:
问哪几家由 `metadata_routes` 决定,与刮削链同一份判据;这里没有本机证据可给,
所以只传番号本身。一家都没给就抛——调用方把原因写进壳的 `scrape_error`,
页面上那一行因此能说清是「还没取」还是「取不到」。
+
+ 一部最多问多久与媒体库资料处理同一份预算(`ACTION_BUDGETS['querying_metadata']`):
+ 来源连不上时每家要等满超时再退避重试,不设上限一部能耗掉几分钟,整批和排在后面的
+ 后继都跟着等。
"""
+ from .jav_cover_fetch import DeadlineExceeded
+ from .library_processing import ACTION_BUDGETS
+ from .processing_control import deadline_after
+
chain = metadata_routes.route_for_code(code)
- found = provider.community(
- code, route=metadata_routes.community_route(code))
+ budget = ACTION_BUDGETS["querying_metadata"]
+ try:
+ found = provider.community(
+ code, deadline=deadline_after(budget),
+ route=metadata_routes.community_route(code))
+ except DeadlineExceeded as error:
+ raise DeadlineExceeded(
+ f"来源链 {'、'.join(chain)} 在 {budget:g} 秒内没有答复") from error
fields: dict[str, object] = {}
cover: str | None = None
for _source, payload in found:
@@ -395,7 +422,7 @@ def run(contract, key: str, handle) -> dict:
unreachable += 1
except Exception: # noqa: BLE001 - 取不到封面这一部照样有资料,下一轮再试
pass
- # 记下这一次尝试,没取到资料的 `backlog` 按它隔 `RETRY_AFTER` 再试;封面的重试另看取图记录。
+ # 记下这一次尝试,没取到资料的由 `scrape_batch` 按它隔 `RETRY_AFTER` 再试;封面的重试另看取图记录。
with contract.database.write_transaction() as connection:
connection.execute("UPDATE feed_discovery SET scraped_at=? WHERE id=?",
(feeds.stamp(), int(row["id"])))
diff --git a/src/peach/followups.py b/src/peach/followups.py
index de6ac31d7..243e726f6 100644
--- a/src/peach/followups.py
+++ b/src/peach/followups.py
@@ -100,12 +100,16 @@ class FollowupType:
`run(contract, key, handle)` 返回一份摘要字典,异常一律算这一条失败——后继失败
不回滚也不改判父任务(ADR-0040 第四条),所以这里不往上抛。
+
+ `jumps_queue` 的后继在自己的通道里先于别的种类领取(ADR-0114):给人刚做完的那件事
+ 收尾的后继,人在等它;补资料、补头像这类后台存量不急。
"""
task_key: str
label: str
writes_ledger: bool
run: Callable[[object, str, TaskRunHandle], dict]
+ jumps_queue: bool = False
@property
def lane(self) -> str:
@@ -133,6 +137,11 @@ def lanes() -> dict[str, tuple[str, ...]]:
return {lane: tuple(sorted(keys)) for lane, keys in grouped.items()}
+def _ahead(keys) -> tuple[str, ...]:
+ """这几类里插队先领的那些(`FollowupType.jumps_queue`)。"""
+ return tuple(key for key in keys if key in REGISTRY and REGISTRY[key].jumps_queue)
+
+
class FollowupRunner:
"""把库里排着的后继跑掉。每条通道一个线程,通道内串行。
@@ -198,7 +207,7 @@ class FollowupRunner:
done = 0
for keys in lanes().values():
while True:
- run = self.store.claim_followup(keys)
+ run = self.store.claim_followup(keys, first=_ahead(keys))
if run is None:
break
self._execute(run)
@@ -209,7 +218,7 @@ class FollowupRunner:
def _work(self, keys: tuple[str, ...]) -> None:
while not self._stopping:
- run = self.store.claim_followup(keys)
+ run = self.store.claim_followup(keys, first=_ahead(keys))
if run is None:
from .update_tasks import is_pausing
db = getattr(getattr(self.store, 'database', None), 'db_path', None)
diff --git a/src/peach/jav_cover_fetch.py b/src/peach/jav_cover_fetch.py
index 69630615b..3d6d4e8e4 100644
--- a/src/peach/jav_cover_fetch.py
+++ b/src/peach/jav_cover_fetch.py
@@ -191,17 +191,21 @@ class CoverConnectError(Unavailable):
class HostLimitedTransport:
- """把请求间隔按主机分别计算;不同官方站点互不阻塞。"""
+ """把请求间隔按主机分别计算;不同官方站点互不阻塞。
+
+ 给了 `limiter` 就按它等:几个传输共用一份间隔,同时跑的任务问同一主机也不会叠在一起。
+ """
def __init__(self, inner: HttpxTransport, interval: float, *,
intervals: dict[str, float] | None = None,
- clock=time.monotonic, sleeper=time.sleep) -> None:
+ clock=time.monotonic, sleeper=time.sleep,
+ limiter: HostLimiter | None = None) -> None:
self.inner = inner
self.interval = max(0.0, interval)
self.clock = clock
self.sleeper = sleeper
- self.limiter = HostLimiter(dict(intervals or {}), default_interval=self.interval,
- clock=clock, sleeper=sleeper)
+ self.limiter = limiter or HostLimiter(dict(intervals or {}), default_interval=self.interval,
+ clock=clock, sleeper=sleeper)
def __call__(self, request: HttpRequest, timeout: float, limit: int):
self.limiter.wait(request.url)
diff --git a/src/peach/library_processing.py b/src/peach/library_processing.py
index 873191800..a5563ba3a 100644
--- a/src/peach/library_processing.py
+++ b/src/peach/library_processing.py
@@ -42,6 +42,7 @@ from .jobs import DiskGuard
from .media_probe import probe_unmeasured
from .media_sources import cost_for
from .media_processing_policy import metadata_only
+from .scripting import HostLimiter
from .strm import skip_background_read
from .record_rehome import reattach_in
from .processing_traffic import ProcessingTraffic
@@ -78,6 +79,10 @@ MISS_TTL_SECONDS = 7 * 24 * 3600
#: 那条批量线仍按 5 秒跑。撞上 403 不再是盲等 24 小时——`scraping_access.FIRST_BLOCKED_PAUSE`
#: 先停 15 分钟,连着再撞才翻倍,所以这一档收紧的代价是有限且可观测的(docs/SOURCING.md)。
SOURCE_INTERVALS = {'javdb.com': 3.0, 'jdbstatic.com': 3.0}
+#: 按 `SOURCE_INTERVALS` 等的那一份间隔,整个进程共用。取想要的资料、取新作资料、资料处理
+#: 各起各的 provider,间隔若各算各的,几条同时跑时问同一主机的请求就叠在一起,配额照样按
+#: 出口 IP 一份算。
+SOURCE_LIMITER = HostLimiter(SOURCE_INTERVALS, default_interval=2.0)
SOURCE_LABELS = {'r18dev': 'r18.dev', 'dmm': 'DMM / FANZA', 'avbase': 'AVBase', 'javbus': 'JavBus', 'javdb': 'javdb',
'fc2': 'FC2', 'fc2cmadb': 'FC2CMADB', 'fc2ppvdb': 'FC2PPV-DB', 'javten': 'JAVten',
'javarchive': 'JavArchive',
@@ -316,7 +321,7 @@ class LibraryMetadataProvider:
self.transport = HostLimitedTransport(
SourceTransport(secrets_root, max_requests=MAX_SOURCE_REQUESTS,
max_bytes=MAX_SOURCE_BYTES, max_seconds=MAX_SOURCE_SECONDS),
- 2.0, intervals=SOURCE_INTERVALS)
+ 2.0, limiter=SOURCE_LIMITER)
#: 由本机浏览器过验证的来源累计等人点验证的秒数(`_excuse`)。调用方拿它前后相减,把这段时间
#: 从单项动作预算里扣掉:一个站第一次弹验证窗口时,那一问要等 `min(40, timeout) + 120` 秒,
diff --git a/src/peach/task_runs.py b/src/peach/task_runs.py
index 64f9c452a..dbd3a425d 100644
--- a/src/peach/task_runs.py
+++ b/src/peach/task_runs.py
@@ -521,30 +521,32 @@ class TaskRunStore:
return None
return int(cursor.lastrowid)
- def claim_followup(self, task_keys) -> TaskRun | None:
+ def claim_followup(self, task_keys, first=()) -> TaskRun | None:
from .update_tasks import admission, UpdateBusy
try:
with admission(self.database.db_path):
- return self._claim_followup(task_keys)
+ return self._claim_followup(task_keys, first)
except UpdateBusy:
return None
- def _claim_followup(self, task_keys) -> TaskRun | None:
+ def _claim_followup(self, task_keys, first=()) -> TaskRun | None:
"""领走这几类里最早的一条待跑后继并标 `running`;没有就返回 None。
- `task_keys` 是调用方这条通道负责的任务种类。按 id 升序领:后继之间没有优先级
- (ADR-0040 未决),先声明的先跑是唯一说得清的顺序。
+ `task_keys` 是调用方这条通道负责的任务种类,`first` 是其中插队先领的那几类
+ (ADR-0114)。同一档里按 id 升序领,先声明的先跑。
"""
keys = tuple(dict.fromkeys(str(key) for key in task_keys or ()))
if not self.enabled or not keys:
return None
+ ahead = tuple(dict.fromkeys(str(key) for key in first or ())) or ("",)
marks = ",".join("?" * len(keys))
moment = stamp()
with self.database.write_transaction(notify=False) as connection:
row = connection.execute(
f"SELECT id FROM task_run WHERE followup_key IS NOT NULL "
- f"AND status='pending' AND task_key IN ({marks}) ORDER BY id LIMIT 1",
- keys).fetchone()
+ f"AND status='pending' AND task_key IN ({marks}) "
+ f"ORDER BY task_key IN ({','.join('?' * len(ahead))}) DESC, id LIMIT 1",
+ (*keys, *ahead)).fetchone()
if row is None:
return None
run_id = int(row["id"])
diff --git a/src/peach/web_feeds.py b/src/peach/web_feeds.py
index 7b679c48d..385ecffee 100644
--- a/src/peach/web_feeds.py
+++ b/src/peach/web_feeds.py
@@ -38,11 +38,11 @@ _NOT_MODIFIED = 304
def _transport(contract):
from .jav_cover_fetch import HostLimitedTransport
- from .library_processing import SOURCE_INTERVALS
+ from .library_processing import SOURCE_LIMITER
from .scraping_access import SourceTransport
return HostLimitedTransport(
- SourceTransport(contract.follow_secrets_root), 2.0, intervals=SOURCE_INTERVALS)
+ SourceTransport(contract.follow_secrets_root), 2.0, limiter=SOURCE_LIMITER)
def fetch(transport, url: str, *, etag: str | None = None,
@@ -201,16 +201,15 @@ def _execute_check(contract, body, job_id: str) -> dict:
if close:
close()
contract.cache_bust()
- # 取资料:新发现的先排,余下名额给没取到资料的旧壳,同一批一条后继。只缺封面的另成一批
+ # 取资料:新发现的先排,没取到资料的旧壳留着保底名额,同一批一条后继。只缺封面的另成一批
# 补封面。定时那一轮只带到了重试时间的,手动点的那一轮全带;已在队里的番号都不再带。
retry_after = feed_followup.RETRY_AFTER if body.get("automatic") else timedelta(0)
hidden = web_settings.hidden_compilations(contract.database)
busy = feed_followup.in_flight(contract.task_runs)
fresh = [code for code in dict.fromkeys(codes) if normalise_code_key(code) not in busy]
with contract.database.read_connection() as connection:
- batch = fresh + feed_followup.backlog(
- connection, contract.cover_root, exclude=[*codes, *busy], retry_after=retry_after,
- limit=max(0, feed_followup.MAX_BATCH - len(fresh)), hidden=hidden)
+ batch = feed_followup.scrape_batch(
+ connection, fresh, exclude=[*codes, *busy], retry_after=retry_after, hidden=hidden)
covers = feed_followup.cover_backlog(
connection, contract.cover_root, exclude=[*codes, *batch, *busy],
retry_after=retry_after, hidden=hidden)
diff --git a/tests/test_feeds_web.py b/tests/test_feeds_web.py
index 98e5aaeac..0f14b0691 100644
--- a/tests/test_feeds_web.py
+++ b/tests/test_feeds_web.py
@@ -409,7 +409,7 @@ class FeedWebTest(FeedWebFixture):
def _backlog(self) -> list[str]:
hidden = web_settings.hidden_compilations(self.contract.database)
with self.contract.database.read_connection() as connection:
- return feed_followup.backlog(connection, self.contract.cover_root, hidden=hidden)
+ return feed_followup.scrape_batch(connection, hidden=hidden)
def test_a_group_compilation_is_neither_listed_counted_nor_rescheduled(self):
self._compilation("波多野結衣、篠田ゆう")
@@ -656,8 +656,45 @@ class FeedWebTest(FeedWebFixture):
connection.execute("UPDATE feed_discovery SET scrape_error='来源连接未取得' WHERE code='HMN-071'")
self.assertEqual(self._cover_backlog(), ['SSIS-950'])
with self.contract.database.read_connection() as connection:
- self.assertEqual(feed_followup.backlog(connection, self.contract.cover_root, hidden=frozenset(),
- retry_after=timedelta(0)), ['HMN-071'])
+ self.assertEqual(feed_followup.scrape_batch(connection, hidden=frozenset(),
+ retry_after=timedelta(0)), ['HMN-071'])
+
+ def test_failed_old_rows_keep_a_share_of_a_batch_full_of_new_codes(self):
+ """一轮发现的新番号比一批还多时,上次没取到资料的旧壳仍有保底名额,不被挤到轮不上。"""
+ fresh = [f"NEW-{number:03d}" for number in range(feed_followup.MAX_BATCH + 10)]
+ self._add()
+ with self.contract.database.write_transaction() as connection:
+ for number in range(feed_followup.RETRY_SHARE + 2):
+ connection.execute(
+ "INSERT INTO feed_discovery(code,source_id,title,link,release_date,discovered_at,"
+ "scraped_at,scrape_error) VALUES(?,1,'',NULL,'2026-09-22',?,?,'来源连接未取得')",
+ (f"OLD-{number:03d}", feeds.stamp(), f"2026-01-{number + 1:02d}T00:00:00Z"))
+ with self.contract.database.read_connection() as connection:
+ batch = feed_followup.scrape_batch(connection, fresh, hidden=frozenset(),
+ retry_after=timedelta(0))
+ self.assertEqual(len(batch), feed_followup.MAX_BATCH)
+ self.assertEqual(batch[:feed_followup.MAX_BATCH - feed_followup.RETRY_SHARE],
+ fresh[:feed_followup.MAX_BATCH - feed_followup.RETRY_SHARE])
+ # 保底名额给最久没试过的那几部。
+ self.assertEqual(batch[-feed_followup.RETRY_SHARE:],
+ [f"OLD-{number:03d}" for number in range(feed_followup.RETRY_SHARE)])
+
+ def test_scraping_one_code_stops_at_the_metadata_budget(self):
+ """来源连不上时一部最多问到资料预算为止,原因写成哪几家在多少秒内没答复。"""
+ from peach.jav_cover_fetch import DeadlineExceeded
+ from peach.library_processing import ACTION_BUDGETS
+
+ seen = {}
+
+ class Silent:
+ def community(self, code, *, deadline=None, route=None):
+ seen["remaining"] = float(deadline) - time.monotonic()
+ raise DeadlineExceeded("动作预算已用尽")
+
+ with self.assertRaisesRegex(DeadlineExceeded, "秒内没有答复"):
+ self._collect(Silent(), "SSIS-950")
+ self.assertLessEqual(seen["remaining"], ACTION_BUDGETS["querying_metadata"])
+ self.assertGreater(seen["remaining"], ACTION_BUDGETS["querying_metadata"] - 5)
def test_codes_already_queued_are_not_planned_again(self):
"""在队里的那一批还没跑完,下一轮检查只带新的番号,不把同一批换个组合再派一条。"""
diff --git a/tests/test_followups.py b/tests/test_followups.py
index b804425ee..c7d260017 100644
--- a/tests/test_followups.py
+++ b/tests/test_followups.py
@@ -210,6 +210,29 @@ class RunnerTests(LedgerTestCase):
self.assertEqual(
{row.status for row in self.store.children(parent.id)}, {"succeeded"})
+ def test_a_type_that_jumps_the_queue_is_claimed_before_earlier_ones_in_its_lane(self):
+ """下载后处理这类人在等的后继,排在已经在排的后台批次前面;同一档里仍是先排先跑。"""
+ order = []
+
+ def record(_contract, key, _handle):
+ order.append(key)
+ return {"outcome": key}
+
+ urgent = FollowupType(task_key="urgent", label="urgent", writes_ledger=True,
+ run=record, jumps_queue=True)
+ parent = self.parent()
+ with registered(echo_type("backlog", run=record), urgent):
+ self.store.enqueue_followups(parent.id, [
+ ("backlog:1", "backlog", "甲"), ("backlog:2", "backlog", "乙"),
+ ("urgent:1", "urgent", "丙"), ("urgent:2", "urgent", "丁")])
+ self.assertEqual(FollowupRunner(self.contract()).drain(), 4)
+ self.assertEqual(order, ["urgent:1", "urgent:2", "backlog:1", "backlog:2"])
+
+ def test_the_download_postprocess_jumps_the_ledger_lane(self):
+ from peach import download_postprocess
+ self.assertTrue(download_postprocess.TYPE.writes_ledger)
+ self.assertTrue(download_postprocess.TYPE.jumps_queue)
+
def test_types_that_do_not_write_the_ledger_get_their_own_lane(self):
with registered(echo_type("ledger-a"), echo_type("cheap", writes_ledger=False)):
self.assertEqual(sorted(lanes()), ["cheap", followups_module.LEDGER_LANE])
diff --git a/tests/test_metadata_library.py b/tests/test_metadata_library.py
index 8c67c06de..71fbfa6ad 100644
--- a/tests/test_metadata_library.py
+++ b/tests/test_metadata_library.py
@@ -1312,6 +1312,17 @@ class LibraryNfoTests(unittest.TestCase):
from peach.library_processing import SOURCE_INTERVALS
self.assertEqual(SOURCE_INTERVALS, {'javdb.com': 3.0, 'jdbstatic.com': 3.0})
+ def test_every_provider_waits_on_one_shared_host_interval(self):
+ """取想要的资料、取新作资料与资料处理各起一个 provider;同时跑时问 javdb 也要排同一条队。"""
+ from peach.library_processing import SOURCE_LIMITER, LibraryMetadataProvider
+ from peach.web_feeds import _transport
+ with tempfile.TemporaryDirectory() as root:
+ first, second = LibraryMetadataProvider(root), LibraryMetadataProvider(root)
+ feed = _transport(SimpleNamespace(follow_secrets_root=Path(root)))
+ self.assertIs(first.transport.limiter, SOURCE_LIMITER)
+ self.assertIs(second.transport.limiter, SOURCE_LIMITER)
+ self.assertIs(feed.limiter, SOURCE_LIMITER)
+
def test_community_sources_are_asked_once_per_code_and_say_why_they_failed(self):
"""资料和封面两步都要社区来源的结果,javdb 的配额经不起同一部片问两遍。"""
from peach.jav_cover_fetch import NotFound, Unavailable