Skip to content

armgit / peach

Public Imported from GitHub
0

Peach: Local-first video metadata scraper and media manager.

fix(config): 外部请求统一遵循 Peach 代理设置

longmeidaoc2b68ce8adc0
diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md
index bd058f590..8c7ae369b 100644
--- a/docs/OPERATIONS.md
+++ b/docs/OPERATIONS.md
@@ -255,7 +255,7 @@ Peach 配置页地址是 `/configuration`。从管理菜单或设置弹窗的「
常用的本机设置:
- 「通用 → 开机自启」:开机后启动 Peach、静默启动(开机后只显示托盘图标,不打开网页)、在桌面创建快捷方式。
-- 「网络与访问 → Peach 代理」:选系统代理、直连或自定义地址。采集来源选「Peach 代理」时共用这里的设置,选「直接连接」则不走代理。保存后,新的采集按新设置运行。
+- 「网络与访问 → Peach 代理」:选系统代理、直连或自定义地址。默认外网 HTTP 请求与采集来源选「Peach 代理」时共用这里的设置,来源选「直接连接」则使用直连。保存后新建客户端按新设置运行,已有任务与长连接沿用创建时的配置。覆盖范围及本机、网盘和独立工具边界见[来源连接设置](SOURCING.md)。
### CloudDrive 配置
diff --git a/docs/REUSE.md b/docs/REUSE.md
index 7ac1ae4d2..ee78017b4 100644
--- a/docs/REUSE.md
+++ b/docs/REUSE.md
@@ -12,9 +12,11 @@ GitGud 双存储副本复用同一版 glab 的原生 `packages upload` 与官方
`fetch_release_tools.py` 从固定版本官方发行站读取 SHA-256 清单,仅提取指定二进制。Python 标准库负责制品清单、版本与提交判据,不另实现 GitLab 认证、S3 签名或多段上传协议。Windows 与 macOS 的 CI 入口复用既有测试、构建和独立制品验收脚本。下载后端及费用边界见[运维说明](OPERATIONS.md#gitlab-发布与下载后端)。
-CI 大产物中转复用相同的 rclone 1.75.2 与官方 [R2 S3 适配](https://developers.cloudflare.com/r2/examples/rclone/),覆盖 Linux amd64、macOS arm64/x86_64 与 Windows amd64;MIT 许可,仅构建期使用。传输、校验和多段上传由 rclone 负责,Peach 只维护流水线身份、输出路径白名单、SHA-256 清单与桶容量预算。macOS 只传 DMG 和构建清单,展开应用保留在构建机器。GitLab job artifacts 只传 JSON 清单;Generic Package 的单文件配额不能保证前置代理接受同等大小的请求。官方 rclone 归档名称与校验清单按实际输入核验,大文件往返使用独立临时仓库验证,不将传输测试当作正式制品验收。无新增 Python 依赖。源码获取复用共享 User-Agent 与标准库 urllib,针对连接中断、超时及暂时性 HTTP 错误最多尝试三次,完成 SHA-256 校验后才登记缓存;对应源码包包含共享请求标识及其许可证,可脱离开发环境执行。
+CI 大产物中转复用相同的 rclone 1.75.2 与官方 [R2 S3 适配](https://developers.cloudflare.com/r2/examples/rclone/),覆盖 Linux amd64、macOS arm64/x86_64 与 Windows amd64;MIT 许可,仅构建期使用。传输、校验和多段上传由 rclone 负责,Peach 只维护流水线身份、输出路径白名单、SHA-256 清单与桶容量预算。macOS 只传 DMG 和构建清单,展开应用保留在构建机器。GitLab job artifacts 只传 JSON 清单;Generic Package 的单文件配额不能保证前置代理接受同等大小的请求。官方 rclone 归档名称与校验清单按实际输入核验,大文件往返使用独立临时仓库验证,不将传输测试当作正式制品验收。无新增 Python 依赖。源码获取复用共享 User-Agent 与 Peach HTTPX 客户端,针对连接中断、超时及暂时性 HTTP 错误最多尝试三次,完成 SHA-256 校验后才登记缓存;可独立构建的对应源码包使用标准库 urllib,包含共享请求标识及其许可证。
-细粒度凭据缺少公开 CI 元数据的读取权限时,发版查询可在官方 API 上进行一次匿名 GET,使用标准库 HTTPS 与项目共用 User-Agent;不扩大凭据权限,也不携带本机认证头。私有资源仍须通过服务端权限检查。
+细粒度凭据缺少公开 CI 元数据的读取权限时,发版查询可在官方 API 上进行一次匿名 GET,复用 HTTPX、Peach 代理与项目共用 User-Agent;不扩大凭据权限,也不携带本机认证头。私有资源仍须通过服务端权限检查。
+
+外部 HTTP 请求的连接策略复用 `peach_proxy.client_options`,客户端使用固定的 HTTPX 0.28.1(BSD-3-Clause)与 curl_cffi 0.16.3(MIT),没有新增依赖。共享 `HttpxTransport`、浏览器指纹传输、字幕、PikPak、更新、模型与维护脚本使用同一策略;libcurl 通过 `NOPROXY` 明确表达直连及自定义代理,避免继承相反的环境例外。Sentry 2.72.0 的事件、限流与回执仍由 SDK 维护,最终请求复用 HTTPX,保持测试禁发、无重试和无重定向。临时本机代理验证两种传输的实际请求路径;下载大小、SHA-256、重试次数和独立源码归档继续由行为回归核对。无需引入额外代理库或全局改写第三方网络函数。来源显式直连、网盘挂载、本机控制接口和独立工具的边界见[来源连接设置](SOURCING.md)。
npm 安装沿用 Node.js 24 LTS 附带的官方 [`npm ci`](https://docs.npmjs.com/cli/v11/commands/npm-ci/),锁文件与完整性校验由 npm 维护。`npm_ci.py` 复用既有 `npm_audit.py` 的有限重试方式及 Python 3.14 标准库 `subprocess`(PSF 许可,无额外安装体积),只为明确的 `ECONNRESET`、`ETIMEDOUT`、`EAI_AGAIN` 下载错误重试,最多三次;锁文件、完整性与生命周期错误保留失败。Linux、Windows、macOS CI 和 Windows 打包入口共用此策略,不重试测试失败、不变更包版本。真实锁定安装与注入失败回归覆盖成功、重试耗尽和非网络错误,无新增应用依赖、生产入口或账本改动。
diff --git a/docs/SOURCING.md b/docs/SOURCING.md
index 98be6dfe9..ad4e81f35 100644
--- a/docs/SOURCING.md
+++ b/docs/SOURCING.md
@@ -25,7 +25,11 @@
「同步媒体库」与 `scripts/scrape_codes.py` 共用正式来源链。来源设为「直连」时使用直连客户端;设为跟随 Peach 代理时,使用配置页保存的直连、自定义代理或跟随环境设置。在线封面使用同一来源传输,amane 桥把代理及逐站直连设置传给子进程。浏览器取页也遵循显式直连和自定义代理;代理地址带用户名时退回 HTTP 客户端。
-跟随环境时,HTTP 客户端读取进程代理环境变量,浏览器使用系统代理;两者不保证采用相同出口。应用直连不能排除操作系统 TUN。部分独立维护脚本直接使用 `HttpxTransport` 或 `page_cache.Site`,只读取环境代理,未接入页面保存的 Peach 代理,不能据此声称所有脚本设置一致。
+默认外网 HTTP 请求共用「Peach 代理」,覆盖发现页与作品详情资料、关注媒体、封面与头像、字幕搜索、PikPak API、更新检查与下载、模型下载和远程诊断。独立采集、链接检查、参考资料与发行下载脚本也复用这份设置。来源显式选择「直接连接」、自定义元数据的独立连接策略和注入的传输继续优先。
+
+跟随环境时,HTTP 客户端读取进程代理环境变量,浏览器使用系统代理;两者不保证采用相同出口。应用直连不能排除操作系统 TUN。设置对新建客户端生效,已经运行的任务或长连接沿用创建时的设置。浏览器直接加载的外站图片与用户打开的外链由浏览器自身网络设置决定。
+
+本机健康检查、浏览器控制接口、writer 镜像、用户自配 Torznab 索引器和 CloudDrive 挂载连接使用各自的网络边界。CloudDrive 的 API/数据代理仍在媒体库设置中管理;Git、glab、rclone、cloudflared 与构建工具的联网由各工具配置负责。可独立构建的 FFmpeg 对应源码归档不包含 Peach 应用,归档内下载脚本使用系统环境代理。
网盘媒体的探测、九宫格、时间轴和播放会读取视频数据;挂载路径的网络连接由 CloudDrive2、OpenList 对应挂载工具负责。Peach 的目录「媒体读取策略」限制后台读取和确认边界,不修改挂载工具的代理。联网采集作品文字和封面使用上述来源设置,与视频所在网盘分别配置。
diff --git a/scripts/check_reference_updates.py b/scripts/check_reference_updates.py
index e46938c56..f2bd98e3e 100644
--- a/scripts/check_reference_updates.py
+++ b/scripts/check_reference_updates.py
@@ -46,8 +46,8 @@ def fetch_bytes(url: str, *, timeout: float = 20.0) -> bytes:
"User-Agent": USER_AGENT,
},
)
- with urllib.request.urlopen(request, timeout=timeout) as response:
- return response.read()
+ from peach.peach_proxy import read_url
+ return read_url(request.full_url, headers=dict(request.header_items()), timeout=timeout)
def resolve_git_revision(source: dict, *, timeout: float = 20.0) -> str | None:
diff --git a/scripts/demo_portraits.py b/scripts/demo_portraits.py
index 7772bd878..3ce9474b1 100644
--- a/scripts/demo_portraits.py
+++ b/scripts/demo_portraits.py
@@ -22,6 +22,7 @@ if str(SRC_DIR) not in sys.path:
sys.path.insert(0, str(SRC_DIR))
from peach.user_agent import USER_AGENT # noqa: E402
+from peach.peach_proxy import read_url
MANIFEST_PATH = Path(__file__).resolve().parent / "demo-portraits.json"
GFRIENDS_RAW = "https://raw.githubusercontent.com/gfriends/gfriends/master/Content/"
@@ -77,7 +78,7 @@ def load_manifest(path: Path | None = None) -> list[Portrait]:
return [Portrait(**entry) for entry in document["entries"]]
-def fetch(portrait: Portrait, cache_dir: Path, *, opener=urllib.request.urlopen) -> Path:
+def fetch(portrait: Portrait, cache_dir: Path, *, opener=None) -> Path:
"""取一张图到缓存,校验哈希。缓存命中就不再联网。
哈希不符是硬错误,不是可以将就的降级:清单的全部意义就是「取到的是我看过的那张」,
@@ -88,8 +89,11 @@ def fetch(portrait: Portrait, cache_dir: Path, *, opener=urllib.request.urlopen)
if target.is_file() and hashlib.sha256(target.read_bytes()).hexdigest() == portrait.sha256:
return target
request = urllib.request.Request(portrait.url, headers={"User-Agent": USER_AGENT})
- with opener(request, timeout=60) as response:
- body = response.read(MAX_IMAGE_BYTES + 1)
+ if opener is None:
+ body = read_url(portrait.url, headers=dict(request.header_items()), max_bytes=MAX_IMAGE_BYTES)
+ else:
+ with opener(request, timeout=60) as response:
+ body = response.read(MAX_IMAGE_BYTES + 1)
if len(body) > MAX_IMAGE_BYTES:
raise RuntimeError(f"{portrait.name} 的图超过 {MAX_IMAGE_BYTES} 字节,取错了东西")
digest = hashlib.sha256(body).hexdigest()
diff --git a/scripts/fetch_cloudflared_macos.py b/scripts/fetch_cloudflared_macos.py
index d350ca034..5acfe301a 100644
--- a/scripts/fetch_cloudflared_macos.py
+++ b/scripts/fetch_cloudflared_macos.py
@@ -5,11 +5,14 @@ import argparse
import json
from pathlib import Path
import subprocess
+import sys
import tarfile
-import urllib.request
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
from fetch_ffmpeg_sources import digest, download_source
from build_macos_media import audit_binary
+from peach.peach_proxy import read_url
def fetch(destination: Path, architecture: str) -> Path:
@@ -31,8 +34,7 @@ def fetch(destination: Path, architecture: str) -> Path:
version = subprocess.check_output([str(binary), "--version"], text=True, encoding="utf-8")
if lock["version"] not in version:
raise ValueError("cloudflared 版本不符")
- with urllib.request.urlopen(lock["license_url"], timeout=60) as response:
- license_text = response.read().decode("utf-8")
+ license_text = read_url(lock["license_url"], timeout=60).decode("utf-8")
if "Apache License" not in license_text:
raise ValueError("cloudflared 许可证未取得")
(destination / "LICENSE-cloudflared.txt").write_text(license_text, encoding="utf-8")
diff --git a/scripts/fetch_ffmpeg_sources.py b/scripts/fetch_ffmpeg_sources.py
index 7287f9ac9..bfd13f157 100644
--- a/scripts/fetch_ffmpeg_sources.py
+++ b/scripts/fetch_ffmpeg_sources.py
@@ -19,6 +19,14 @@ import zipfile
SCRIPT_DIR = Path(__file__).resolve().parent
sys.path.insert(0, str(SCRIPT_DIR.parent / "src"))
from peach.user_agent import USER_AGENT
+if (SCRIPT_DIR.parent / "src/peach/peach_proxy.py").is_file():
+ import httpx
+ from peach.peach_proxy import http_client
+ HTTP_ERRORS = (httpx.HTTPError,)
+else:
+ # 对应源码归档可独立构建,归档不包含 Peach 应用或它的代理配置。
+ http_client = None
+ HTTP_ERRORS = ()
LOCK_PATH = SCRIPT_DIR / "ffmpeg-sources.json"
@@ -35,14 +43,22 @@ def download_source(source: dict, archive: Path) -> None:
for attempt in range(3):
try:
print(f"下载固定源码:{source['name']},第 {attempt + 1}/3 次", flush=True)
- with urllib.request.urlopen(request, timeout=60) as response, temporary.open("wb") as stream:
- shutil.copyfileobj(response, stream)
+ if http_client is None:
+ with urllib.request.urlopen(request, timeout=60) as response, temporary.open("wb") as stream:
+ shutil.copyfileobj(response, stream)
+ else:
+ with http_client(follow_redirects=True, timeout=60) as client, client.stream(
+ "GET", request.full_url, headers=dict(request.header_items())) as response, temporary.open("wb") as stream:
+ response.raise_for_status()
+ for chunk in response.iter_bytes(256 * 1024):
+ stream.write(chunk)
if digest(temporary) != source["sha256"]:
raise ValueError(f"源码 SHA256 不匹配: {source['name']}")
temporary.replace(archive)
return
- except (TimeoutError, ConnectionError, http.client.IncompleteRead, urllib.error.URLError) as error:
- retryable = not isinstance(error, urllib.error.HTTPError) or error.code in {408, 429, 500, 502, 503, 504}
+ except (TimeoutError, ConnectionError, http.client.IncompleteRead, urllib.error.URLError, *HTTP_ERRORS) as error:
+ status = getattr(getattr(error, "response", None), "status_code", getattr(error, "code", None))
+ retryable = status is None or status in {408, 429, 500, 502, 503, 504}
if not retryable or attempt == 2:
raise
time.sleep(2 ** (attempt + 1))
@@ -62,7 +78,7 @@ def obtain_source(source: dict, archive: Path) -> None:
try:
download_source(cached, archive)
return
- except (TimeoutError, ConnectionError, http.client.IncompleteRead, urllib.error.URLError):
+ except (TimeoutError, ConnectionError, http.client.IncompleteRead, urllib.error.URLError, *HTTP_ERRORS):
print(f"源码缓存未取得,尝试固定上游:{source['name']}", flush=True)
download_source(source, archive)
diff --git a/scripts/fetch_release_tools.py b/scripts/fetch_release_tools.py
index d7b012493..143587ec2 100644
--- a/scripts/fetch_release_tools.py
+++ b/scripts/fetch_release_tools.py
@@ -9,11 +9,11 @@ import re
import platform
import sys
import tarfile
-from urllib.request import Request, urlopen
import zipfile
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
from peach.user_agent import USER_AGENT
+from peach.peach_proxy import read_url
TOOLS = (
@@ -23,8 +23,7 @@ TOOLS = (
def download(url: str) -> bytes:
- with urlopen(Request(url, headers={"User-Agent": USER_AGENT}), timeout=120) as response:
- return response.read()
+ return read_url(url, headers={"User-Agent": USER_AGENT}, timeout=120)
def tool_specs(*, rclone_only: bool = False, system: str | None = None,
diff --git a/scripts/harvest_directory_links.py b/scripts/harvest_directory_links.py
index 60cf4acf0..21d4c6813 100644
--- a/scripts/harvest_directory_links.py
+++ b/scripts/harvest_directory_links.py
@@ -77,8 +77,6 @@ FIELDS = ("entity_id", "kind", "name", "link_kind", "label", "url", "evidence",
PORTRAIT_FIELDS = ("entity_id", "kind", "name", "matched_name", "portrait_url",
"source", "page", "verdict")
SOURCES = ("laoshi", "bstar", "jae", "javdb")
-#: 直连不通的来源:jae 被对端重置(`WinError 10054`),javdb 直接连接超时。
-PROXY_SOURCES = ("jae", "javdb")
#: 压过 `--interval` 的按来源最小间隔(秒)。javdb 按出口 IP 自己封速率,实测
#: 2026-09-04 那一轮:03:22:01 起 8 分钟取到 124 页,中位间隔 1.23 秒、最快 0.76 秒,
#: 逐分钟成功数 44 / 15 / 6 / 0 / 0 / 0 / 7 / 50 / 2,之后整站每条路径都回 403,
@@ -846,8 +844,7 @@ def run(args) -> int:
sources = SOURCES if args.source == "all" else (args.source,)
for source in sources:
interval = max(args.interval, SOURCE_INTERVAL.get(source, 0.0))
- site = Site(args.cache_dir / source, interval, args.timeout, refresh=args.refresh,
- via_proxy=source in PROXY_SOURCES)
+ site = Site(args.cache_dir / source, interval, args.timeout, refresh=args.refresh)
collector = COLLECTORS.get(source)
try:
site_links, pages = (collector(site, args.limit) if collector
diff --git a/scripts/harvest_studio_icons.py b/scripts/harvest_studio_icons.py
index e99b0660a..1266995b1 100644
--- a/scripts/harvest_studio_icons.py
+++ b/scripts/harvest_studio_icons.py
@@ -12,7 +12,6 @@ import sqlite3
import sys
from pathlib import Path
-import httpx
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
@@ -47,7 +46,8 @@ def run(args: argparse.Namespace) -> dict[str, object]:
targets = {key: value for key, value in targets.items() if key in wanted}
faces = icons.FaceGate(not args.no_face_gate)
- client = httpx.Client(trust_env=True, follow_redirects=True)
+ from peach.peach_proxy import http_client
+ client = http_client(follow_redirects=True)
try:
rows = icons.harvest(targets, links, icons.Fetcher(client, args.timeout, args.interval),
args.candidate_dir.resolve(), faces,
diff --git a/scripts/install_entity_links.py b/scripts/install_entity_links.py
index 7ae6275a2..3d97aa1f4 100644
--- a/scripts/install_entity_links.py
+++ b/scripts/install_entity_links.py
@@ -152,7 +152,8 @@ def resolves(url: str, timeout: float = 12.0,
eltra.jp 2026-09-05 连着两趟这样被判死,而每次重试一下就 200。404 重试三次还是 404。
"""
def get(target: str) -> httpx.Response:
- with httpx.Client(follow_redirects=True, timeout=timeout,
+ from peach.peach_proxy import http_client
+ with http_client(follow_redirects=True, timeout=timeout,
limits=httpx.Limits(max_connections=4,
max_keepalive_connections=0)) as client:
return client.get(target, headers={"User-Agent": USER_AGENT})
diff --git a/scripts/publish_gitlab_release.py b/scripts/publish_gitlab_release.py
index 6a305351f..f7f789edf 100644
--- a/scripts/publish_gitlab_release.py
+++ b/scripts/publish_gitlab_release.py
@@ -10,7 +10,6 @@ import subprocess
import sys
import time
from urllib.parse import quote, urlencode, urlsplit
-from urllib.request import Request, urlopen
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "src"))
@@ -18,6 +17,7 @@ if not __package__:
sys.path.insert(0, str(ROOT))
__package__ = "scripts"
from peach.user_agent import USER_AGENT
+from peach.peach_proxy import read_url
def verify_local_assets(root: Path, assets: list[dict]) -> None:
@@ -117,8 +117,7 @@ def publish(root: Path = Path('release')) -> None:
if backend in {"r2", "both"}:
subprocess.run([sys.executable, "scripts/release_r2.py", "--root", str(root), "--version", tag[1:], "--apply"], check=True)
base = os.environ["PEACH_R2_PUBLIC_BASE"].rstrip("/")
- with urlopen(Request(f"{base}/{tag}/release.json", headers={"User-Agent": USER_AGENT}), timeout=60) as response:
- release = json.load(response)
+ release = json.loads(read_url(f"{base}/{tag}/release.json", headers={"User-Agent": USER_AGENT}, timeout=60))
if (release["tag_name"] != tag or release.get("commit_sha") != sha or not release["assets"]
or any(item["state"] != "uploaded" or not item["browser_download_url"].startswith(f"{base}/{tag}/")
for item in release["assets"])):
diff --git a/scripts/rediscover_entity_links.py b/scripts/rediscover_entity_links.py
index 5c11f3b0f..07e2b0e15 100644
--- a/scripts/rediscover_entity_links.py
+++ b/scripts/rediscover_entity_links.py
@@ -42,7 +42,8 @@ FIELDS = ("entity_id", "kind", "name", "link_kind", "label", "url", "evidence",
def fetch(url: str, timeout: float = 12.0) -> tuple[int, str, str]:
"""(status, 正文, 最终地址)。每个请求新建 client——失败的连接会漏池槽。"""
try:
- with httpx.Client(follow_redirects=True, timeout=timeout,
+ from peach.peach_proxy import http_client
+ with http_client(follow_redirects=True, timeout=timeout,
limits=httpx.Limits(max_connections=4,
max_keepalive_connections=0)) as client:
response = client.get(url, headers={"User-Agent": USER_AGENT})
diff --git a/scripts/release_forge.py b/scripts/release_forge.py
index 24481c307..4fa61b9aa 100644
--- a/scripts/release_forge.py
+++ b/scripts/release_forge.py
@@ -9,10 +9,11 @@ import subprocess
import sys
from pathlib import Path
from urllib.parse import quote, urlencode
-from urllib.request import Request, urlopen
+from urllib.request import Request
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
from peach.user_agent import USER_AGENT
+from peach.peach_proxy import read_url
REQUIRED_TEST_JOBS = frozenset({"verify-linux", "verify-windows", "verify-macos", "wheel-smoke"})
@@ -40,8 +41,7 @@ class GitLabForge:
# 公开项目的只读元数据可匿名取得,不扩大本机细粒度凭据权限。
request = Request(f"https://{self.host}/api/v4/{path}",
headers={"Accept": "application/json", "User-Agent": USER_AGENT})
- with urlopen(request, timeout=60) as response:
- return json.load(response)
+ return json.loads(read_url(request.full_url, headers=dict(request.header_items()), timeout=60))
def remote_url(self, repo: str) -> str:
# 使用已配置的 Git 认证;PAT 和 Runner token 不进入命令参数。
diff --git a/scripts/release_r2.py b/scripts/release_r2.py
index 2698a517f..dd5e476f5 100644
--- a/scripts/release_r2.py
+++ b/scripts/release_r2.py
@@ -12,10 +12,10 @@ import subprocess
import sys
import tempfile
from urllib.parse import quote, urlsplit
-from urllib.request import Request, urlopen
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
from peach.user_agent import USER_AGENT
+from peach.peach_proxy import http_client
def inventory(root: Path, version: str) -> list[dict]:
@@ -40,10 +40,10 @@ def inventory(root: Path, version: str) -> list[dict]:
def verify_download(url: str, item: dict) -> None:
checksum = hashlib.sha256()
size = 0
- with urlopen(Request(url, headers={"User-Agent": USER_AGENT}), timeout=120) as response:
- if response.status != 200:
+ with http_client(follow_redirects=True, timeout=120) as client, client.stream("GET", url, headers={"User-Agent": USER_AGENT}) as response:
+ if response.status_code != 200:
raise ValueError(f"公开下载未返回完整文件:{item['name']}")
- while chunk := response.read(1024 * 1024):
+ for chunk in response.iter_bytes(1024 * 1024):
size += len(chunk)
checksum.update(chunk)
if size != item["size"] or checksum.hexdigest() != item["sha256"]:
diff --git a/scripts/release_review.py b/scripts/release_review.py
index 8abcdecfb..225a122c3 100644
--- a/scripts/release_review.py
+++ b/scripts/release_review.py
@@ -8,7 +8,7 @@ from pathlib import Path
import shutil
import subprocess
import sys
-from urllib.request import Request, urlopen
+from urllib.request import Request
if not __package__:
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
@@ -31,8 +31,8 @@ def public_github_api(endpoint: str):
"""公开上游元数据使用匿名官方 API,不携带本机账号凭据。"""
request = Request(f"https://api.github.com/{endpoint}",
headers={"Accept": "application/vnd.github+json", "User-Agent": references.USER_AGENT})
- with urlopen(request, timeout=60) as response:
- return json.load(response)
+ from peach.peach_proxy import read_url
+ return json.loads(read_url(request.full_url, headers=dict(request.header_items()), timeout=60))
def public_dependabot(repo: str) -> list[dict]:
diff --git a/scripts/repair_entity_links.py b/scripts/repair_entity_links.py
index ce05e9948..28b975dd2 100644
--- a/scripts/repair_entity_links.py
+++ b/scripts/repair_entity_links.py
@@ -151,7 +151,8 @@ class Fetcher:
# 每个请求一个 client。共用一个池时,连不上的那些主机会把池槽漏掉,几十条
# 之后所有请求一律 `PoolTimeout`——那看起来像整批站点集体挂了,而实际上
# 只是没有空槽可用了。这一批本来就按域名限速在跑,握手的开销无关紧要。
- transport = HttpxTransport(httpx.Client(
+ from peach.peach_proxy import http_client
+ transport = HttpxTransport(http_client(
verify=self._context, headers={"User-Agent": USER_AGENT},
limits=httpx.Limits(max_connections=4, max_keepalive_connections=0)),
owns_client=True)
diff --git a/src/peach/avatar_watermark.py b/src/peach/avatar_watermark.py
index 5a391a472..d8178a15e 100644
--- a/src/peach/avatar_watermark.py
+++ b/src/peach/avatar_watermark.py
@@ -25,7 +25,8 @@ r"""头像上的站点水印:检出、裁掉或修掉。
from __future__ import annotations
import hashlib
-import urllib.request
+import httpx
+from .peach_proxy import read_url
from dataclasses import dataclass
from pathlib import Path
@@ -111,9 +112,8 @@ def ensure_model(path: Path | None = None, *, allow_download: bool = True) -> Pa
target.parent.mkdir(parents=True, exist_ok=True)
partial = target.with_suffix(target.suffix + ".part")
try:
- with urllib.request.urlopen(MODEL_URL, timeout=60) as response:
- partial.write_bytes(response.read())
- except OSError as error:
+ partial.write_bytes(read_url(MODEL_URL, timeout=60))
+ except (OSError, httpx.HTTPError) as error:
partial.unlink(missing_ok=True)
raise WatermarkModelUnavailable(f"取文字检测模型失败:{error}") from error
digest = _digest(partial)
diff --git a/src/peach/browser_transport.py b/src/peach/browser_transport.py
index c638ebf1f..018a923cc 100644
--- a/src/peach/browser_transport.py
+++ b/src/peach/browser_transport.py
@@ -136,7 +136,8 @@ def proxy_flags(*, direct: bool, proxy: str) -> tuple[str, ...] | None:
parsed = urlsplit(proxy)
if parsed.username or parsed.password:
return None
- return (f"--proxy-server={urlunsplit((parsed.scheme, parsed.netloc, '', '', ''))}",)
+ scheme = 'socks5' if parsed.scheme == 'socks5h' else parsed.scheme
+ return (f"--proxy-server={urlunsplit((scheme, parsed.netloc, '', '', ''))}",)
class _WebSocket:
diff --git a/src/peach/custom_metadata.py b/src/peach/custom_metadata.py
index 71337f7ca..d9717f108 100644
--- a/src/peach/custom_metadata.py
+++ b/src/peach/custom_metadata.py
@@ -172,8 +172,8 @@ def fetch_json(url: str, timeout: float, *, client_options=None):
# curl_cffi 0.16.3 的 CONNECT_TO 接收原生 slist;Session 关闭后才释放。
target = ffi.gc(lib.curl_slist_append(ffi.NULL, f'{host}:443:{address}:443'.encode()), lib.curl_slist_free_all)
curl_options = {CurlOpt.CONNECT_TO: target}
- if not options.get('proxy') and not options.get('trust_env'):
- curl_options[CurlOpt.NOPROXY] = '*'
+ if not options.get('trust_env'):
+ curl_options[CurlOpt.NOPROXY] = '' if options.get('proxy') else '*'
with requests.Session(curl_options=curl_options, **options) as client:
try:
with client.stream('GET', url, timeout=_remaining_seconds(expires), allow_redirects=False,
diff --git a/src/peach/downloads_pikpak.py b/src/peach/downloads_pikpak.py
index 92cc54963..70d070645 100644
--- a/src/peach/downloads_pikpak.py
+++ b/src/peach/downloads_pikpak.py
@@ -162,7 +162,8 @@ class PikPakClient:
self.profile = profile_for(self.values.get("client_id"))
self.persist = persist
self.clock = clock
- self.http = httpx.Client(timeout=TIMEOUT, transport=transport, follow_redirects=False)
+ from .peach_proxy import http_client
+ self.http = http_client(timeout=TIMEOUT, transport=transport, follow_redirects=False)
def close(self) -> None:
self.http.close()
diff --git a/src/peach/downloads_pikpak_browser.py b/src/peach/downloads_pikpak_browser.py
index 63eac6b9c..9135137d1 100644
--- a/src/peach/downloads_pikpak_browser.py
+++ b/src/peach/downloads_pikpak_browser.py
@@ -107,7 +107,14 @@ def parse_capture(payload: dict) -> dict[str, str] | None:
def _attended_browser(executable: str, profile: Path) -> _Browser:
- return _Browser(executable, profile, (), attended=True)
+ from . import peach_proxy
+ from .browser_transport import proxy_flags
+ options = peach_proxy.client_options(profile.parent)
+ flags = proxy_flags(direct=not options.get('trust_env') and not options.get('proxy'),
+ proxy=options.get('proxy', ''))
+ if flags is None:
+ raise BrowserUnavailable('浏览器登录不支持带凭据的代理地址,请选择系统代理或无需认证的本机代理')
+ return _Browser(executable, profile, flags, attended=True)
class BrowserLogin:
@@ -184,8 +191,9 @@ class BrowserLogin:
self._state = {"id": run_id, "state": state, "message": message}
def _run(self, run_id: str, executable: str, cancel: threading.Event) -> None:
- browser = self._browser(executable, self.profile)
+ browser = None
try:
+ browser = self._browser(executable, self.profile)
browser.start()
assert browser.socket is not None
browser.socket.call("Page.addScriptToEvaluateOnNewDocument", source=_FORGET_ON_FIRST_DOCUMENT)
@@ -199,7 +207,8 @@ class BrowserLogin:
LOGGER.error("PikPak 浏览器登录出错:%s", type(error).__name__)
self._finish(run_id, FAILED, f"浏览器登录出错:{type(error).__name__}")
finally:
- browser.close()
+ if browser is not None:
+ browser.close()
if self.status()["state"] == DONE and self.on_done is not None:
self.on_done()
diff --git a/src/peach/face_detect.py b/src/peach/face_detect.py
index aa2250615..7d27bee5c 100644
--- a/src/peach/face_detect.py
+++ b/src/peach/face_detect.py
@@ -23,7 +23,8 @@ YuNet 给的是框 + 5 个关键点 + 置信度,可以按分数卡,不必靠
from __future__ import annotations
import hashlib
-import urllib.request
+import httpx
+from .peach_proxy import read_url
from dataclasses import dataclass, replace
from pathlib import Path
@@ -177,9 +178,8 @@ def fetch_model(target: Path, url: str, sha256: str, *, allow_download: bool,
target.parent.mkdir(parents=True, exist_ok=True)
temporary = target.with_suffix(".part")
try:
- with urllib.request.urlopen(url, timeout=timeout) as response:
- payload = response.read()
- except OSError as error:
+ payload = read_url(url, timeout=timeout)
+ except (OSError, httpx.HTTPError) as error:
raise FaceModelUnavailable(f"取{label}失败:{error}") from error
digest = hashlib.sha256(payload).hexdigest()
if digest != sha256:
diff --git a/src/peach/http.py b/src/peach/http.py
index 303a4d965..67bd01b26 100644
--- a/src/peach/http.py
+++ b/src/peach/http.py
@@ -11,6 +11,7 @@ from .user_agent import USER_AGENT
import ipaddress
import re
import socket
+import threading
import urllib.parse
from dataclasses import dataclass
from typing import Mapping, Protocol
@@ -18,6 +19,7 @@ from typing import Mapping, Protocol
import httpx
from curl_cffi import requests as curl_requests
from curl_cffi.requests.exceptions import CurlError
+from . import peach_proxy
#: 只在本机或局域网里有意义的名字后缀。外面递进来的地址落到它们上,只可能是想让
#: Peach 替人去探内网。
@@ -188,12 +190,28 @@ class HttpxTransport:
MAX_REDIRECTS = 5
def __init__(self, client: httpx.Client | None = None, *, owns_client: bool = False):
- self.client = client or httpx.Client(
- limits=httpx.Limits(max_connections=32, max_keepalive_connections=16),
- headers={"User-Agent": USER_AGENT},
- )
+ self._client = client
+ self._client_lock = threading.Lock()
+ self._closed = False
self._owns_client = client is None or owns_client
+ @property
+ def client(self):
+ with self._client_lock:
+ if self._closed and self._client is None:
+ raise RuntimeError("HTTP 客户端已关闭")
+ if self._client is None:
+ self._client = httpx.Client(
+ **peach_proxy.client_options(),
+ limits=httpx.Limits(max_connections=32, max_keepalive_connections=16),
+ headers={"User-Agent": USER_AGENT})
+ return self._client
+
+ @client.setter
+ def client(self, value):
+ with self._client_lock:
+ self._client = value
+
def __call__(
self,
request: HttpRequest,
@@ -238,7 +256,10 @@ class HttpxTransport:
def close(self) -> None:
if self._owns_client:
- self.client.close()
+ with self._client_lock:
+ self._closed = True
+ if self._client is not None:
+ self._client.close()
def _download_chunks(response):
@@ -277,9 +298,22 @@ class CurlCffiTransport:
def __init__(self, *, impersonate: str,
session: curl_requests.Session | None = None):
- self.session = session or curl_requests.Session(impersonate=impersonate)
+ self._session = session
+ self._impersonate = impersonate
+ self._session_lock = threading.Lock()
+ self._closed = False
self._owns_session = session is None
+ @property
+ def session(self):
+ with self._session_lock:
+ if self._closed and self._session is None:
+ raise RuntimeError("浏览器指纹客户端已关闭")
+ if self._session is None:
+ self._session = curl_requests.Session(impersonate=self._impersonate,
+ **peach_proxy.curl_options())
+ return self._session
+
def __call__(
self,
request: HttpRequest,
@@ -321,4 +355,7 @@ class CurlCffiTransport:
def close(self) -> None:
if self._owns_session:
- self.session.close()
+ with self._session_lock:
+ self._closed = True
+ if self._session is not None:
+ self._session.close()
diff --git a/src/peach/page_cache.py b/src/peach/page_cache.py
index 9fa49f019..e358813f3 100644
--- a/src/peach/page_cache.py
+++ b/src/peach/page_cache.py
@@ -36,12 +36,15 @@ class Site:
"""
def __init__(self, cache_dir: Path, interval: float, timeout: float, *,
- refresh: bool = False, via_proxy: bool = False, transport=None,
+ refresh: bool = False, via_proxy: bool | None = None, transport=None,
cookies: dict[str, str] | None = None, retries: int = 2,
backoff: float = 2.0):
self.cache_dir, self.interval, self.timeout, self.refresh = cache_dir, interval, timeout, refresh
+ from .peach_proxy import http_client
+ options = {"trust_env": False} if via_proxy is False else {}
self.transport = transport or HttpxTransport(
- httpx.Client(trust_env=via_proxy, follow_redirects=True, cookies=cookies or {}),
+ (httpx.Client if via_proxy is False else http_client)(
+ **options, follow_redirects=True, cookies=cookies or {}),
owns_client=True)
self.retries, self.backoff = max(0, retries), backoff
self._limiter = RateLimiter(interval)
diff --git a/src/peach/peach_proxy.py b/src/peach/peach_proxy.py
index ffcb4feed..cb3747616 100644
--- a/src/peach/peach_proxy.py
+++ b/src/peach/peach_proxy.py
@@ -1,4 +1,4 @@
-"""采集来源共用的 Peach 连接策略与本机代理凭据。"""
+"""外部请求共用的 Peach 连接策略与本机代理凭据。"""
from __future__ import annotations
import json
@@ -55,7 +55,13 @@ def save(root: Path, body: dict) -> dict:
return describe(root)
-def client_options(root: Path) -> dict:
+def secrets_root() -> Path:
+ from .settings_file import active
+ return active().directory("secrets")
+
+
+def client_options(root: Path | None = None) -> dict:
+ root = secrets_root() if root is None else root
saved = values(root)
if saved.get("conflict"):
raise ValueError("来源代理设置不同,请先在配置页选择 Peach 代理")
@@ -63,3 +69,32 @@ def client_options(root: Path) -> dict:
if mode == "proxy":
return {"trust_env": False, "proxy": saved["proxy"]}
return {"trust_env": mode == "environment"}
+
+
+def http_client(*, root: Path | None = None, **kwargs):
+ """外网 HTTPX 客户端;注入的传输独占请求,不额外挂载代理。"""
+ import httpx
+ options = {"trust_env": False} if kwargs.get("transport") is not None else client_options(root)
+ return httpx.Client(**{**options, **kwargs})
+
+
+def curl_options(root: Path | None = None) -> dict:
+ """libcurl 的直连与自定义代理显式覆盖环境 NO_PROXY。"""
+ from curl_cffi import CurlOpt
+ options = client_options(root)
+ if not options.get("trust_env"):
+ options["curl_options"] = {CurlOpt.NOPROXY: "" if options.get("proxy") else "*"}
+ return options
+
+
+def read_url(url: str, *, timeout: float = 60, headers=None, max_bytes: int | None = None) -> bytes:
+ """读取模型、清单等完整响应,沿用 Peach 的 HTTP 与 SOCKS 代理。"""
+ import httpx
+ from .http import _bounded_body
+ try:
+ with http_client(follow_redirects=True, timeout=timeout) as client, client.stream("GET", url, headers=headers) as response:
+ response.raise_for_status()
+ return response.read() if max_bytes is None else _bounded_body(response.iter_bytes(), max_bytes)
+ except httpx.HTTPError as error:
+ kind = f"HTTP {error.response.status_code}" if isinstance(error, httpx.HTTPStatusError) else type(error).__name__
+ raise OSError(f"外部请求未取得({kind})") from error
diff --git a/src/peach/preview_transport.py b/src/peach/preview_transport.py
index 3bb9f70b2..d60c0bd53 100644
--- a/src/peach/preview_transport.py
+++ b/src/peach/preview_transport.py
@@ -31,9 +31,7 @@ class _CurlStream(httpx.SyncByteStream):
class _CurlClient:
def __init__(self, root):
- options = peach_proxy.client_options(root)
- self.session = requests.Session(impersonate='chrome', proxy=options.get('proxy'),
- trust_env=options.get('trust_env', True))
+ self.session = requests.Session(impersonate='chrome', **peach_proxy.curl_options(root))
def build_request(self, method, url, *, headers):
return httpx.Request(method, url, headers=headers)
diff --git a/src/peach/reporting_transport.py b/src/peach/reporting_transport.py
index 617d491a6..b5e8f33dd 100644
--- a/src/peach/reporting_transport.py
+++ b/src/peach/reporting_transport.py
@@ -3,6 +3,9 @@ from __future__ import annotations
import os
import threading
+from types import SimpleNamespace
+
+from .peach_proxy import http_client
def transport_class(gate, callback):
@@ -43,8 +46,11 @@ def transport_class(gate, callback):
def _request(self, method, endpoint_type, body, headers):
if os.environ.get("PEACH_REPORTING_TEST_MODE") == "1":
raise OSError("测试进程禁止远程诊断外发")
- return self._pool.request(method, self._auth.get_api_url(endpoint_type), body=body,
- headers=headers, retries=False, redirect=False)
+ with http_client(timeout=self.TIMEOUT, follow_redirects=False) as client:
+ response = client.request(method, self._auth.get_api_url(endpoint_type),
+ content=body, headers=headers)
+ return SimpleNamespace(status=response.status_code, headers=response.headers,
+ data=response.content, close=response.close)
def _send_envelope(self, envelope):
self._context(envelope)
diff --git a/src/peach/standalone_update.py b/src/peach/standalone_update.py
index f16c063e0..7313c9af5 100644
--- a/src/peach/standalone_update.py
+++ b/src/peach/standalone_update.py
@@ -102,9 +102,10 @@ def extract(archive: Path, destination: Path, progress, *, tool_files: set[str]
def download(url: str, archive: Path, size: int, digest: str, progress) -> None:
+ from .peach_proxy import http_client
checksum = hashlib.sha256()
received = 0
- with httpx.Client(follow_redirects=True, timeout=30) as client, client.stream("GET", url) as response:
+ with http_client(follow_redirects=True, timeout=30) as client, client.stream("GET", url) as response:
response.raise_for_status()
with archive.open("wb") as output:
for chunk in response.iter_bytes(256 * 1024):
diff --git a/src/peach/studio_followup.py b/src/peach/studio_followup.py
index 5c66acffe..604d0d7ff 100644
--- a/src/peach/studio_followup.py
+++ b/src/peach/studio_followup.py
@@ -192,12 +192,12 @@ def _land_site(contract, entity_id: int, name: str, aliases, batch: str) -> dict
def _land_mark(contract, entity_id: int, name: str, batch: str) -> dict:
"""按账本里这家的链接取图,可装的直接落盘。"""
- import httpx
+ from .peach_proxy import http_client
with contract.database.read_connection() as connection:
entries = studio_icons.entity_links(connection, entity_id, name)
candidate_dir = contract.candidate_root / "provider-cache" / "studio-icons"
- client = httpx.Client(trust_env=True, follow_redirects=True)
+ client = http_client(root=contract.follow_secrets_root, follow_redirects=True)
try:
landed = studio_icons.land_one(
name, entries, contract.logo_root, candidate_dir,
diff --git a/src/peach/studio_sites.py b/src/peach/studio_sites.py
index 4813f8e34..0f47c1f70 100644
--- a/src/peach/studio_sites.py
+++ b/src/peach/studio_sites.py
@@ -411,7 +411,8 @@ def crawler_client() -> "httpx.Client":
代价只是每次多建一个 client,在 0.8 秒的请求间隔下可以忽略。不改共享默认值——
那会伤到真正需要复用连接的连接器。
"""
- return httpx.Client(
+ from .peach_proxy import http_client
+ return http_client(
follow_redirects=True,
limits=httpx.Limits(max_connections=4, max_keepalive_connections=0),
headers={"User-Agent": USER_AGENT},
diff --git a/src/peach/subtitle_search.py b/src/peach/subtitle_search.py
index 7842dc010..85f4e4485 100644
--- a/src/peach/subtitle_search.py
+++ b/src/peach/subtitle_search.py
@@ -58,7 +58,8 @@ def _get(client: httpx.Client, url: str, *, suffix: str = "", params: dict | Non
def _client() -> httpx.Client:
- return httpx.Client(timeout=8, headers={"User-Agent": USER_AGENT})
+ from .peach_proxy import http_client
+ return http_client(timeout=8, headers={"User-Agent": USER_AGENT})
def _id(url: str) -> str:
diff --git a/src/peach/web_links.py b/src/peach/web_links.py
index a2126cd9d..389d2fe1c 100644
--- a/src/peach/web_links.py
+++ b/src/peach/web_links.py
@@ -54,7 +54,8 @@ def _probe(url: str, timeout: float = CHECK_TIMEOUT) -> tuple[int, str]:
连不上,而失败的连接会在共享池里漏掉槽位,几十个请求之后一切都变成 PoolTimeout。
"""
def get(target: str) -> httpx.Response:
- with httpx.Client(follow_redirects=True, timeout=timeout,
+ from .peach_proxy import http_client
+ with http_client(follow_redirects=True, timeout=timeout,
limits=httpx.Limits(max_connections=4,
max_keepalive_connections=0)) as client:
return client.get(target, headers={"User-Agent": USER_AGENT})
diff --git a/tests/test_browser_transport.py b/tests/test_browser_transport.py
index 076a280aa..5848772f3 100644
--- a/tests/test_browser_transport.py
+++ b/tests/test_browser_transport.py
@@ -627,6 +627,7 @@ class BrowserLookupTests(unittest.TestCase):
self.assertEqual(proxy_flags(direct=True, proxy="http://x:1"), ("--no-proxy-server",))
self.assertEqual(proxy_flags(direct=False, proxy=""), ())
self.assertEqual(proxy_flags(direct=False, proxy="socks5://127.0.0.1:7890/"), ("--proxy-server=socks5://127.0.0.1:7890",))
+ self.assertEqual(proxy_flags(direct=False, proxy="socks5h://127.0.0.1:7890/"), ("--proxy-server=socks5://127.0.0.1:7890",))
self.assertIsNone(proxy_flags(direct=False, proxy="http://user:[email protected]:7890"), "命令行带不了代理凭据")
diff --git a/tests/test_custom_metadata.py b/tests/test_custom_metadata.py
index 025d2c472..08803b310 100644
--- a/tests/test_custom_metadata.py
+++ b/tests/test_custom_metadata.py
@@ -135,6 +135,7 @@ class CustomMetadataTests(unittest.TestCase):
self.assertEqual(custom.fetch_json('https://metadata.example.com/ABW-001', 3,
client_options={'proxy': 'http://127.0.0.1:7890', 'trust_env': False}), {'ok': True})
options = session.call_args.kwargs
+ self.assertEqual(options['curl_options'][custom.CurlOpt.NOPROXY], '')
target = options['curl_options'][custom.CurlOpt.CONNECT_TO]
self.assertEqual(custom.ffi.string(target.data), b'metadata.example.com:443:1.1.1.1:443')
stream = session.return_value.__enter__.return_value.stream
diff --git a/tests/test_downloads_pikpak_browser.py b/tests/test_downloads_pikpak_browser.py
index 6e3f45dd1..0a5c80cd9 100644
--- a/tests/test_downloads_pikpak_browser.py
+++ b/tests/test_downloads_pikpak_browser.py
@@ -8,6 +8,7 @@ import json
import threading
import time
import unittest
+from unittest.mock import patch
from pathlib import Path
from peach import downloads_pikpak_browser as pb
@@ -107,6 +108,22 @@ class BrowserLoginTests(unittest.TestCase):
self.assertFalse(thread.is_alive())
return login.status()
+ def test_proxy_setup_failure_finishes_the_login_without_a_browser(self):
+ login, _ = self.make(FakeBrowser([]))
+ with patch.object(login, '_browser', side_effect=BrowserUnavailable('代理配置不可用')):
+ login.start()
+ status = self.finish(login)
+ self.assertEqual(status['state'], pb.FAILED)
+ self.assertIn('代理配置不可用', status['message'])
+
+ def test_browser_login_uses_the_saved_proxy_from_its_secrets_root(self):
+ from peach import peach_proxy
+ with patch.object(peach_proxy, 'client_options', return_value={'trust_env': False, 'proxy': 'http://127.0.0.1:17890'}) as options, \
+ patch.object(pb, '_Browser') as factory:
+ pb._attended_browser('chrome.exe', Path('C:/fake/secrets/browser-pikpak'))
+ options.assert_called_once_with(Path('C:/fake/secrets'))
+ self.assertEqual(factory.call_args.args[2], ('--proxy-server=http://127.0.0.1:17890',))
+
def test_the_page_credentials_are_stored_with_the_client_from_the_key_and_then_forgotten(self):
browser = FakeBrowser([None, RuntimeError("Execution context was destroyed"), json.dumps(CAPTURE)])
login, made = self.make(browser)
diff --git a/tests/test_face_detect.py b/tests/test_face_detect.py
index afeb529b9..09154b245 100644
--- a/tests/test_face_detect.py
+++ b/tests/test_face_detect.py
@@ -22,7 +22,7 @@ class EnsureModelTests(unittest.TestCase):
target.write_bytes(payload)
with patch.object(face_detect, "MODEL_SHA256",
hashlib.sha256(payload).hexdigest()), \
- patch.object(face_detect.urllib.request, "urlopen") as opener:
+ patch.object(face_detect, "read_url") as opener:
self.assertEqual(ensure_model(target), target)
opener.assert_not_called()
@@ -40,20 +40,9 @@ class EnsureModelTests(unittest.TestCase):
self.assertFalse(target.exists(), "校验不过的文件必须删掉,不能留着下次再骗一次")
def test_a_download_whose_digest_differs_is_refused(self):
- class _Response:
- def read(self):
- return b"not-the-model"
-
- def __enter__(self):
- return self
-
- def __exit__(self, *_exc):
- return False
-
with tempfile.TemporaryDirectory() as temp:
target = Path(temp).resolve() / "yunet.onnx"
- with patch.object(face_detect.urllib.request, "urlopen",
- return_value=_Response()):
+ with patch.object(face_detect, "read_url", return_value=b"not-the-model"):
with self.assertRaises(FaceModelUnavailable) as caught:
ensure_model(target)
self.assertIn("校验不符", str(caught.exception))
diff --git a/tests/test_face_match.py b/tests/test_face_match.py
index 564abb9e9..655ddf50d 100644
--- a/tests/test_face_match.py
+++ b/tests/test_face_match.py
@@ -19,7 +19,7 @@ class EnsureModelTests(unittest.TestCase):
target.write_bytes(payload)
with patch.object(face_match, "MODEL_SHA256",
hashlib.sha256(payload).hexdigest()), \
- patch.object(face_detect.urllib.request, "urlopen") as opener:
+ patch.object(face_detect, "read_url") as opener:
self.assertEqual(face_match.ensure_model(target), target)
opener.assert_not_called()
diff --git a/tests/test_ffmpeg_build.py b/tests/test_ffmpeg_build.py
index 20488bcf0..32617a501 100644
--- a/tests/test_ffmpeg_build.py
+++ b/tests/test_ffmpeg_build.py
@@ -14,6 +14,8 @@ import unittest
from unittest import mock
import urllib.error
import zipfile
+from contextlib import nullcontext
+import httpx
SCRIPT = Path(__file__).resolve().parents[1] / "scripts/fetch_ffmpeg_sources.py"
@@ -24,23 +26,33 @@ SPEC.loader.exec_module(builder)
class FFmpegSourceBuildTests(unittest.TestCase):
+ def setUp(self):
+ self.opened = mock.Mock()
+ client = mock.MagicMock()
+ client.__enter__.return_value = client
+ def stream(method, url, **kwargs):
+ result = self.opened(method, url, **kwargs)
+ return nullcontext(httpx.Response(200, content=result.read(), request=httpx.Request(method, url)))
+ client.stream.side_effect = stream
+ self.enterContext(mock.patch.object(builder, "http_client", return_value=client))
+
def test_source_cache_falls_back_to_pinned_upstream_on_transport_failure(self):
import hashlib
data = b"pinned source bytes"
source = {"name": "ffmpeg", "url": "https://upstream.example/source.tar.xz", "archive": "source.tar.xz",
"sha256": hashlib.sha256(data).hexdigest()}
with tempfile.TemporaryDirectory() as temporary, mock.patch.dict(os.environ, {"PEACH_SOURCE_CACHE_BASE": "https://cache.example/_sources"}), \
- mock.patch.object(builder.urllib.request, "urlopen", side_effect=[
+ mock.patch.object(self, "opened", side_effect=[
urllib.error.HTTPError("https://cache.example", 404, "missing", {}, None), io.BytesIO(data)]) as opened:
archive = Path(temporary).resolve() / "source.tar.xz"
builder.obtain_source(source, archive)
self.assertEqual(archive.read_bytes(), data)
- self.assertEqual([call.args[0].full_url for call in opened.call_args_list],
+ self.assertEqual([call.args[1] for call in opened.call_args_list],
[f"https://cache.example/_sources/{source['sha256']}/source.tar.xz", source["url"]])
def test_source_cache_hash_mismatch_is_fatal(self):
with tempfile.TemporaryDirectory() as temporary, mock.patch.dict(os.environ, {"PEACH_SOURCE_CACHE_BASE": "https://cache.example/_sources"}), \
- mock.patch.object(builder.urllib.request, "urlopen", return_value=io.BytesIO(b"corrupt")) as opened:
+ mock.patch.object(self, "opened", return_value=io.BytesIO(b"corrupt")) as opened:
archive = Path(temporary).resolve() / "source.tar.xz"
with self.assertRaisesRegex(ValueError, "SHA256"):
builder.obtain_source({"name": "ffmpeg", "url": "https://upstream.example/source.tar.xz",
@@ -55,7 +67,7 @@ class FFmpegSourceBuildTests(unittest.TestCase):
"sha256": hashlib.sha256(data).hexdigest()}
with tempfile.TemporaryDirectory() as temporary:
archive = Path(temporary).resolve() / "source.tar.xz"
- with mock.patch.object(builder.urllib.request, "urlopen", side_effect=[TimeoutError(), io.BytesIO(data)]) as opened, \
+ with mock.patch.object(self, "opened", side_effect=[TimeoutError(), io.BytesIO(data)]) as opened, \
mock.patch.object(builder.time, "sleep") as sleep:
builder.download_source(source, archive)
self.assertEqual(archive.read_bytes(), data)
@@ -67,7 +79,7 @@ class FFmpegSourceBuildTests(unittest.TestCase):
for failure in (None, urllib.error.HTTPError("https://upstream.example", 403, "Forbidden", {}, None)):
with self.subTest(failure=failure), tempfile.TemporaryDirectory() as temporary:
archive = Path(temporary).resolve() / "source.tar.xz"
- with mock.patch.object(builder.urllib.request, "urlopen", side_effect=failure,
+ with mock.patch.object(self, "opened", side_effect=failure,
return_value=io.BytesIO(b"corrupt")) as opened, \
mock.patch.object(builder.time, "sleep") as sleep:
with self.assertRaises((ValueError, urllib.error.HTTPError)):
@@ -81,7 +93,7 @@ class FFmpegSourceBuildTests(unittest.TestCase):
def test_download_stops_after_three_transport_attempts(self):
with tempfile.TemporaryDirectory() as temporary:
archive = Path(temporary).resolve() / "source.tar.xz"
- with mock.patch.object(builder.urllib.request, "urlopen", side_effect=TimeoutError()) as opened, \
+ with mock.patch.object(self, "opened", side_effect=TimeoutError()) as opened, \
mock.patch.object(builder.time, "sleep") as sleep:
with self.assertRaises(TimeoutError):
builder.download_source({"name": "ffmpeg", "url": "https://upstream.example/source.tar.xz",
diff --git a/tests/test_http.py b/tests/test_http.py
index 83d351df3..29c595508 100644
--- a/tests/test_http.py
+++ b/tests/test_http.py
@@ -1,12 +1,120 @@
import unittest
import ast
+import tempfile
+import threading
+import os
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
+from unittest.mock import patch
import httpx
from peach.http import (CurlCffiTransport, HttpRequest, HttpResponse, HttpxTransport,
body_text, response_text)
from peach.user_agent import USER_AGENT
+from peach import peach_proxy
+
+
+class PeachProxyTests(unittest.TestCase):
+ def test_proxy_conflict_is_reported_on_outbound_request_and_leaves_local_startup_available(self):
+ with patch.object(peach_proxy, 'client_options', side_effect=ValueError('请选择 Peach 代理')):
+ for transport in (HttpxTransport(), CurlCffiTransport(impersonate='chrome')):
+ try:
+ with self.assertRaisesRegex(ValueError, '请选择 Peach 代理'):
+ transport(HttpRequest('GET', 'https://example.invalid', {}), 2, 100)
+ finally:
+ transport.close()
+
+ def test_http_and_browser_transports_reach_the_configured_proxy(self):
+ seen = []
+ class Proxy(BaseHTTPRequestHandler):
+ def do_GET(self):
+ seen.append(self.path)
+ self.send_response(200)
+ self.send_header("Content-Length", "5")
+ self.end_headers()
+ self.wfile.write(b"proxy")
+ def log_message(self, *_args):
+ pass
+ server = ThreadingHTTPServer(("127.0.0.1", 0), Proxy)
+ thread = threading.Thread(target=server.serve_forever, daemon=True)
+ thread.start()
+ try:
+ with tempfile.TemporaryDirectory() as folder:
+ root = Path(folder).resolve()
+ peach_proxy.save(root, {"mode": "proxy", "proxy": f"http://127.0.0.1:{server.server_port}"})
+ with patch.object(peach_proxy, "secrets_root", return_value=root), patch.dict(os.environ, {"NO_PROXY": "*", "no_proxy": "*"}):
+ for transport in (HttpxTransport(), CurlCffiTransport(impersonate="chrome")):
+ try:
+ self.assertEqual(transport(HttpRequest("GET", "http://peach-proxy.invalid/check", {}), 3, 20).body, b"proxy")
+ finally:
+ transport.close()
+ peach_proxy.save(root, {"mode": "direct"})
+ with patch.object(peach_proxy, "secrets_root", return_value=root), patch.dict(os.environ, {
+ "HTTP_PROXY": "http://127.0.0.1:1", "http_proxy": "http://127.0.0.1:1", "NO_PROXY": "", "no_proxy": ""}):
+ for transport in (HttpxTransport(), CurlCffiTransport(impersonate="chrome")):
+ try:
+ self.assertEqual(transport(HttpRequest("GET", f"http://127.0.0.1:{server.server_port}/direct", {}), 3, 20).body, b"proxy")
+ finally:
+ transport.close()
+ self.assertEqual(seen, ["http://peach-proxy.invalid/check"] * 2 + ["/direct"] * 2)
+ finally:
+ server.shutdown()
+ server.server_close()
+ thread.join(timeout=3)
+
+ def test_environment_mode_inherits_process_proxy(self):
+ with tempfile.TemporaryDirectory() as folder:
+ root = Path(folder).resolve()
+ peach_proxy.save(root, {"mode": "environment"})
+ self.assertEqual(peach_proxy.client_options(root), {"trust_env": True})
+
+ def test_read_url_preserves_script_network_failure_handling_and_exception_chain(self):
+ failure = httpx.ConnectError('private proxy address')
+ with patch.object(peach_proxy, 'http_client', side_effect=failure):
+ with self.assertRaises(OSError) as caught:
+ peach_proxy.read_url('https://example.invalid')
+ self.assertIs(caught.exception.__cause__, failure)
+ self.assertIn('ConnectError', str(caught.exception))
+ self.assertNotIn('private proxy address', str(caught.exception))
+
+ def test_default_transport_uses_saved_proxy_without_environment_fallback(self):
+ with tempfile.TemporaryDirectory() as folder:
+ root = Path(folder).resolve()
+ peach_proxy.save(root, {"mode": "proxy", "proxy": "socks5h://127.0.0.1:17890"})
+ with patch.object(peach_proxy, "secrets_root", return_value=root), patch("peach.http.httpx.Client") as factory:
+ transport = HttpxTransport()
+ _ = transport.client
+ self.assertEqual(factory.call_args.kwargs["proxy"], "socks5h://127.0.0.1:17890")
+ self.assertFalse(factory.call_args.kwargs["trust_env"])
+ transport.close()
+
+ def test_direct_client_ignores_environment_proxy(self):
+ with tempfile.TemporaryDirectory() as folder:
+ root = Path(folder).resolve()
+ peach_proxy.save(root, {"mode": "direct"})
+ with patch("httpx.Client") as factory:
+ peach_proxy.http_client(root=root, timeout=8)
+ self.assertEqual(factory.call_args.kwargs, {"trust_env": False, "timeout": 8})
+
+ def test_injected_transport_is_used_without_a_proxy_mount(self):
+ with tempfile.TemporaryDirectory() as folder:
+ root = Path(folder).resolve()
+ peach_proxy.save(root, {"mode": "proxy", "proxy": "http://127.0.0.1:1"})
+ with peach_proxy.http_client(root=root, transport=httpx.MockTransport(
+ lambda request: httpx.Response(200, content=b"injected"))) as client:
+ self.assertEqual(client.get("https://example.invalid").content, b"injected")
+
+ def test_browser_fingerprint_transport_uses_saved_proxy(self):
+ with tempfile.TemporaryDirectory() as folder:
+ root = Path(folder).resolve()
+ peach_proxy.save(root, {"mode": "proxy", "proxy": "http://127.0.0.1:17890"})
+ with patch.object(peach_proxy, "secrets_root", return_value=root), patch("peach.http.curl_requests.Session") as factory:
+ transport = CurlCffiTransport(impersonate="chrome")
+ _ = transport.session
+ self.assertEqual(factory.call_args.kwargs["proxy"], "http://127.0.0.1:17890")
+ self.assertFalse(factory.call_args.kwargs["trust_env"])
+ transport.close()
class UserAgentTests(unittest.TestCase):
diff --git a/tests/test_library_sharing.py b/tests/test_library_sharing.py
index 0e85597d3..f33e4789d 100644
--- a/tests/test_library_sharing.py
+++ b/tests/test_library_sharing.py
@@ -399,10 +399,12 @@ class LibrarySharingTests(unittest.TestCase):
(200, {'x-sentry-rate-limits': '60:error:organization'}, 'unknown'), (None, {}, 'unknown')):
with self.subTest(status=status, headers=headers):
sharing._state_path(self.config).unlink(missing_ok=True)
- response = SimpleNamespace(status=status, headers=headers, close=mock.Mock(), data=b'private')
+ response = SimpleNamespace(status_code=status, headers=headers, close=mock.Mock(), content=b'private')
with mock.patch.dict(os.environ, {'PEACH_REPORTING_TEST_MODE': '0'}), \
- mock.patch('urllib3.PoolManager.request', return_value=response,
- side_effect=OSError('private proxy') if status is None else None) as request:
+ mock.patch('peach.reporting_transport.http_client') as factory:
+ request = factory.return_value.__enter__.return_value.request
+ request.return_value = response
+ request.side_effect = OSError('private proxy') if status is None else None
self.service.attempt()
self.service.attempt()
request.assert_called_once()
diff --git a/tests/test_reporting_transport.py b/tests/test_reporting_transport.py
index 37a8b5cb8..29a861557 100644
--- a/tests/test_reporting_transport.py
+++ b/tests/test_reporting_transport.py
@@ -43,16 +43,18 @@ class ReportingTransportTests(unittest.TestCase):
with self.subTest(status=status, headers=headers):
outcomes = []
client = self.client(outcomes)
- response = SimpleNamespace(status=status, headers=headers, close=mock.Mock(), data=b"private")
- with mock.patch('urllib3.PoolManager.request', return_value=response) as request:
+ response = SimpleNamespace(status_code=status, headers=headers, close=mock.Mock(), content=b"private")
+ with mock.patch('peach.reporting_transport.http_client') as factory:
+ request = factory.return_value.__enter__.return_value.request
+ request.return_value = response
client.transport._send_envelope(self.envelope())
- self.assertIs(request.call_args.kwargs['retries'], False)
- self.assertIs(request.call_args.kwargs['redirect'], False)
+ self.assertFalse(factory.call_args.kwargs['follow_redirects'])
+ self.assertEqual(request.call_count, 1)
self.assertEqual(outcomes[-1]["status"], expected)
self.assertEqual(outcomes[-1]["status_code"], status)
self.assertNotIn("private", str(outcomes))
outcomes = []
- with mock.patch('urllib3.PoolManager.request', side_effect=OSError("private proxy")):
+ with mock.patch('peach.reporting_transport.http_client', side_effect=OSError("private proxy")):
with self.assertRaises(OSError):
self.client(outcomes).transport._send_envelope(self.envelope())
self.assertEqual(outcomes, [{"status": "unknown", "reason": "network"}])
@@ -61,7 +63,7 @@ class ReportingTransportTests(unittest.TestCase):
outcomes = []
client = self.client(outcomes)
with mock.patch.dict(os.environ, {"PEACH_REPORTING_TEST_MODE": "1"}), \
- mock.patch('urllib3.PoolManager.request') as network:
+ mock.patch('peach.reporting_transport.http_client') as network:
with self.assertRaises(OSError):
client.transport._send_envelope(self.envelope())
network.assert_not_called()
@@ -127,7 +129,7 @@ class ReportingTransportTests(unittest.TestCase):
outcomes = []
client = self.client(outcomes)
client.transport._disabled_until[None] = datetime.now(timezone.utc) + timedelta(seconds=60)
- with mock.patch('urllib3.PoolManager.request') as network:
+ with mock.patch('peach.reporting_transport.http_client') as network:
client.transport._send_envelope(self.envelope())
network.assert_not_called()
self.assertEqual(outcomes, [{'status': 'failed', 'reason': 'rate_limit'}])
diff --git a/tests/test_scripts.py b/tests/test_scripts.py
index 93fc430f5..b904a3805 100644
--- a/tests/test_scripts.py
+++ b/tests/test_scripts.py
@@ -4943,14 +4943,17 @@ class R2ReleaseTests(unittest.TestCase):
data = b"Peach public artifact"
response = mock.MagicMock()
response.__enter__.return_value = response
- response.status = 200
- response.read.side_effect = [data[:5], data[5:], b""]
+ response.status_code = 200
+ response.iter_bytes.return_value = iter([data[:5], data[5:]])
expected = {"name": "Peach.zip", "size": len(data), "sha256": hashlib.sha256(data).hexdigest()}
- with mock.patch.object(module, "urlopen", return_value=response):
- module.verify_download("https://downloads.example/Peach.zip", expected)
- response.read.side_effect = [data[:-1], b""]
- with mock.patch.object(module, "urlopen", return_value=response), self.assertRaisesRegex(ValueError, "SHA-256"):
+ with mock.patch.object(module, "http_client") as factory:
+ factory.return_value.__enter__.return_value.stream.return_value = response
module.verify_download("https://downloads.example/Peach.zip", expected)
+ response.iter_bytes.return_value = iter([data[:-1]])
+ with mock.patch.object(module, "http_client") as factory:
+ factory.return_value.__enter__.return_value.stream.return_value = response
+ with self.assertRaisesRegex(ValueError, "SHA-256"):
+ module.verify_download("https://downloads.example/Peach.zip", expected)
def test_inventory_requires_complete_platforms_and_does_not_upload_helpers(self):
module = load_script("release_r2")
@@ -5082,7 +5085,7 @@ class GitGudPackageMirrorTests(unittest.TestCase):
'PEACH_R2_PUBLIC_BASE': 'https://downloads.example'}), \
mock.patch.object(module.subprocess, 'check_output', return_value=sha), \
mock.patch.object(module.subprocess, 'run') as run, \
- mock.patch.object(module, 'urlopen', return_value=response), \
+ mock.patch.object(module, 'read_url', return_value=json.dumps(release).encode()), \
mock.patch('scripts.release_r2.inventory', return_value=[asset]), \
mock.patch.object(module, 'gitgud_package_links', return_value=[backup]) as mirror:
module.publish(root)
@@ -5107,15 +5110,15 @@ class GitLabReleaseTests(unittest.TestCase):
for status in (401, 403):
forge.command.side_effect = subprocess.CalledProcessError(1, ["glab"], stderr=f"glab: HTTP {status}")
with self.subTest(status=status), mock.patch.dict(os.environ, {"PEACH_GLAB": "glab"}), \
- mock.patch.object(release_forge, "urlopen", return_value=response) as fetch:
+ mock.patch.object(release_forge, "read_url", return_value=b'[{"id":7}]') as fetch:
self.assertEqual(forge.api("owner/repo", "pipelines"), [{"id": 7}])
- request = fetch.call_args.args[0]
- self.assertEqual(request.full_url, "https://gitgud.io/api/v4/projects/owner%2Frepo/pipelines")
- self.assertNotIn("Private-token", request.headers)
- self.assertNotIn("Authorization", request.headers)
+ self.assertEqual(fetch.call_args.args[0], "https://gitgud.io/api/v4/projects/owner%2Frepo/pipelines")
+ headers = {key.lower(): value for key, value in fetch.call_args.kwargs['headers'].items()}
+ self.assertNotIn("private-token", headers)
+ self.assertNotIn("authorization", headers)
forge.command.side_effect = subprocess.CalledProcessError(1, ["glab"], stderr="network unavailable")
with mock.patch.dict(os.environ, {"PEACH_GLAB": "glab"}), \
- mock.patch.object(release_forge, "urlopen") as fetch, self.assertRaises(subprocess.CalledProcessError):
+ mock.patch.object(release_forge, "read_url") as fetch, self.assertRaises(subprocess.CalledProcessError):
forge.api("owner/repo", "pipelines")
fetch.assert_not_called()
diff --git a/tests/test_standalone_update.py b/tests/test_standalone_update.py
index a6e5445c8..b73921f3c 100644
--- a/tests/test_standalone_update.py
+++ b/tests/test_standalone_update.py
@@ -194,7 +194,7 @@ class StandaloneUpdateTests(unittest.TestCase):
with patch.object(update.sys, "executable", str(target / "Peach.exe")), patch.object(
update.release_updates, "check", return_value=release), patch.object(
update.httpx, "Client", side_effect=lambda **kwargs: client_type(
- **kwargs, transport=update.httpx.MockTransport(response))):
+ **{**kwargs, 'transport': update.httpx.MockTransport(response), 'trust_env': False, 'proxy': None})):
update.prepare({"id": "split-fixture"}, lock)
lock.release.assert_called_once()
return update.read(), requests
@@ -273,7 +273,7 @@ class StandaloneUpdateTests(unittest.TestCase):
return httpx.Response(200, content=(output / name).read_bytes())
def client(*args, **kwargs):
- return original_client(*args, **kwargs, transport=httpx.MockTransport(respond), trust_env=False)
+ return original_client(*args, **{**kwargs, 'transport': httpx.MockTransport(respond), 'trust_env': False, 'proxy': None})
with patch('peach.__version__', '1.0.0'), patch.object(update.httpx, 'Client', side_effect=client):
self.assertEqual(prepare_install_tools(target, self.root / 'empty-cache', error, stage=stage), 0)
@@ -301,7 +301,7 @@ class StandaloneUpdateTests(unittest.TestCase):
return httpx.Response(200, content=b'corrupt' if bad else (output / name).read_bytes())
with patch('peach.__version__', '1.0.0'), patch.object(update.httpx, 'Client', side_effect=lambda **kwargs:
- original_client(**kwargs, transport=httpx.MockTransport(respond), trust_env=False)):
+ original_client(**{**kwargs, 'transport': httpx.MockTransport(respond), 'trust_env': False, 'proxy': None})):
result = prepare_install_tools(target, self.root / 'empty-cache', error, stage=stage)
self.assertFalse(target.exists())
return result, requests, manifest, stage, error
diff --git a/tests/test_studio_followup.py b/tests/test_studio_followup.py
index 5ae943393..cdb85109f 100644
--- a/tests/test_studio_followup.py
+++ b/tests/test_studio_followup.py
@@ -65,6 +65,7 @@ class StudioFollowupCase(unittest.TestCase):
self.contract = SimpleNamespace(
database=self.database, logo_root=self.logos,
candidate_root=self.root / "generated",
+ follow_secrets_root=self.root / "secrets",
cache_bust=lambda: self.busted.append(1))
def entity(self, kind: str, name: str) -> int: