Skip to content

armgit / peach

Public Imported from GitHub
0

Peach: Local-first video metadata scraper and media manager.

fix(ci): 核对媒体缓存身份并均衡 Windows 测试分片

longmeidao87c206ba64aa
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index cea2e43de..a580c4d94 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -51,27 +51,20 @@ jobs:
timeout-minutes: 40
steps:
- uses: actions/checkout@v7
- - name: 复用固定源码的媒体工具与对应源码包
- id: media-cache
- uses: actions/[email protected]
- with:
- key: windows-media-v1-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('scripts/build_ffmpeg_windows.sh', 'scripts/dav1d-mingw64.ini', 'scripts/build_untrunc_windows.sh', 'scripts/fetch_ffmpeg_sources.py', 'scripts/fetch_untrunc_sources.py', 'scripts/ffmpeg-sources.json', 'scripts/untrunc-sources.json') }}
- path: |
- build/ffmpeg-windows/ffmpeg
- build/ffmpeg-windows/ffmpeg-corresponding-source.zip
- build/untrunc-windows/untrunc
- build/untrunc-windows/untrunc-corresponding-source.zip
- name: 安装交叉编译工具
- if: steps.media-cache.outputs.cache-hit != 'true'
run: |
sudo apt-get update
sudo apt-get install -y build-essential gcc-mingw-w64-x86-64-posix g++-mingw-w64-x86-64-posix mingw-w64-x86-64-dev nasm pkg-config python3 clang meson ninja-build cmake ca-certificates curl xz-utils
- - name: 构建固定版本媒体工具
- if: steps.media-cache.outputs.cache-hit != 'true'
- run: bash scripts/build_ffmpeg_windows.sh build/ffmpeg-windows
- - name: 构建固定版本索引修复工具
- if: steps.media-cache.outputs.cache-hit != 'true'
- run: bash scripts/build_untrunc_windows.sh build/untrunc-windows
+ - name: 核对媒体构建身份
+ id: media-identity
+ run: echo "key=$(python3 scripts/media_build_cache.py windows --github-key)" >> "$GITHUB_OUTPUT"
+ - name: 复用固定源码的媒体工具与对应源码包
+ uses: actions/[email protected]
+ with:
+ key: ${{ steps.media-identity.outputs.key }}
+ path: build/media-cache/windows-x86_64
+ - name: 校验或构建媒体工具与对应源码包
+ run: python3 scripts/media_build_cache.py windows
- uses: actions/upload-artifact@v7
with:
name: ffmpeg-windows
@@ -303,17 +296,18 @@ jobs:
uv sync --locked --all-extras --python python
uv pip check --python .venv/bin/python
npm ci --prefix frontend
- # 与 scripts/ci_macos.sh 相同的媒体工具准备:缓存按 media_build_cache.py 的输入与工具链身份核对,
- # 不一致时在本机重新编译。
- - name: 复用固定源码的 macOS 媒体工具与对应源码包
- uses: actions/[email protected]
- with:
- key: macos-media-v1-${{ matrix.arch }}-${{ runner.os }}-${{ hashFiles('scripts/ffmpeg-sources.json', 'scripts/untrunc-sources.json', 'scripts/build_macos_media.py', 'scripts/fetch_ffmpeg_sources.py', 'scripts/fetch_untrunc_sources.py', 'scripts/setup_macos_media_tools.py', 'scripts/media_build_cache.py', 'src/peach/user_agent.py', 'LICENSE') }}
- path: build/media-cache/macos-${{ matrix.arch }}
- name: 安装固定版本的媒体编译工具
run: |
.venv/bin/python scripts/setup_macos_media_tools.py --root build/macos-media-toolchain
echo "$PWD/build/macos-media-toolchain/env/bin" >> "$GITHUB_PATH"
+ - name: 核对媒体构建身份
+ id: media-identity
+ run: echo "key=$(.venv/bin/python scripts/media_build_cache.py macos --arch '${{ matrix.arch }}' --github-key)" >> "$GITHUB_OUTPUT"
+ - name: 复用固定源码的 macOS 媒体工具与对应源码包
+ uses: actions/[email protected]
+ with:
+ key: ${{ steps.media-identity.outputs.key }}
+ path: build/media-cache/macos-${{ matrix.arch }}
- name: 构建或复用媒体工具并取得 cloudflared
run: |
for tool in ffmpeg untrunc; do
diff --git a/docs/TESTING.md b/docs/TESTING.md
index fd59fa16e..61bdf2152 100644
--- a/docs/TESTING.md
+++ b/docs/TESTING.md
@@ -94,7 +94,7 @@ Python 静态检查使用开发依赖 Ruff,随每个正式测试域扫描全
| wheel | Windows 与 macOS 各一次 Python 3.14 独立最小安装 |
| Release | 复用同 SHA 主线成功 CI;Windows 独立 EXE 与 Apple Silicon、Intel DMG 分别构建和验收 |
-Python 全量按文件稳定分片,Windows 四片、macOS 两片。`auto` 按影响域选择文件;唯一选测文件达到 150 个时,Windows 复用四片分配,macOS 仍用单片;更小的 `auto` 两端均用单片。每片使用独立 runner,仍按同一基线和影响域运行,汇总任务要求所有分片和计划选择的任务成功。分片子进程不签发记录。
+Python 全量按文件稳定分片,Windows 四片、macOS 两片。`auto` 按影响域选择文件;唯一选测文件达到 150 个时,Windows 复用四片分配,macOS 仍用单片;更小的 `auto` 两端均用单片。Windows 独立 Python 分片按 `scripts/ci-python-timings.json` 的成功日志耗时分配,不计已经交给浏览器任务的本机耗时先验;macOS 保持文件名轮转。每片使用独立 runner,仍按同一基线和影响域运行,汇总任务要求所有分片和计划选择的任务成功。分片子进程不签发记录。
Test 工作流的 `web-bundle` 在 Ubuntu 上统一执行 tsc、lint、Vitest、构建及产物差异检查;lint 同时检查无诊断且扫描文件数大于零。CI 的 Python 任务把这三个工具用例交给前端任务,把两个真实浏览器用例交给浏览器矩阵;同模块的契约测试仍执行。任务身份由固定工作流与 job 名识别,未知任务和本机保留完整测试。承担部分 CI 职责的运行不签发本机测试记录。
@@ -108,7 +108,7 @@ Python 分片上传完整测试输出;浏览器分片上传 TAP、服务日志
GitGud 流水线(`.gitlab-ci.yml`)的 `verify-linux`、`verify-windows`、`verify-macos` 在分支与合并请求上跑改动域加本平台的平台敏感用例:基线是最近一条成功的主线流水线里、作为本提交祖先的那个提交,由 `scripts/ci_scope.py` 判定;先用正式入口跑 `auto --base <基线>`,选域与退回全量的规则同本机 `auto`,再用 `-Focus` 跑 `PLATFORM_TESTS` 里本平台的用例。取不到基线(首次推送、强推、接口不可达)、改了 `.gitlab-ci.yml` 本身,或改动落到必须全量的面时只跑一次全量。版本提交(`chore(release): 版本 X`)的主线流水线是发布演练:三平台全量、媒体构建、桌面安装升级与镜像验收都成功后登记候选;`--ship` 等演练成功才打标签。标签核对同一提交、验证任务和产物 SHA-256,发布已验收的相同文件与镜像,不重新构建。候选身份、保留与回收见 [ADR-0127](adr/0127-verified-release-artifact-promotion.md),全量范围见 [ADR-0119](adr/0119-mainline-changed-scope-release-full-suite.md)。
-演练先读取发布桶真实占用,媒体编译可与正式回归并行;构建清单齐备后在安装冒烟前核对包含暂存与公开副本的精确峰值,候选登记与正式上传仍复查。媒体缓存身份绑定固定源码、构建脚本、工具链、平台与架构,二进制和对应源码均逐文件校验;缓存不代替独立制品验收。Docker 发布核对演练记录的 image ID、提交与版本,在桌面包和 Release 就绪后推送。版本发布与覆盖升级验收不取消进行中的同目标运行;覆盖升级需明确选择目标版本及完整构建提交。
+演练先读取发布桶真实占用,媒体编译可与正式回归并行;构建清单齐备后在安装冒烟前核对包含暂存与公开副本的精确峰值,候选登记与正式上传仍复查。媒体缓存身份绑定固定源码、构建脚本、工具链、平台与架构;GitHub 外层键和包内回执使用同一身份,两平台二进制和对应源码均逐文件校验。缓存未取得或校验不符时重新构建,缓存不代替独立制品验收。Docker 发布核对演练记录的 image ID、提交与版本,在桌面包和 Release 就绪后推送。版本发布与覆盖升级验收不取消进行中的同目标运行;覆盖升级需明确选择目标版本及完整构建提交。
浏览器用例覆盖 `frontend/e2e` 下全部 `*.test.ts`,包括子目录。`setup.test.ts` 由 `SetupE2ETests` 在首次设置服务上执行,其余文件在已配置的演示库服务上按文件并发、文件内串行。并发数取 `PEACH_E2E_CONCURRENCY`;未设时本机取逻辑核数的四分之一、上限 4,CI(`GITHUB_ACTIONS=true`)为 1。每个文件单独起一次 Node,限时 600 秒,取得自己的完整 TAP;空出的通道马上领下一个文件,没有批与批之间的等待。并发时文件按 `scripts/ci-browser-timings.json` 的实测秒数先长后短排队(没测过的按 30 秒算),测量动画、密度切换与等待态替换的文件(`tests/test_web_e2e.py` 的 `TIMING_SENSITIVE_SUITES`)在并发文件全部结束后逐个串行;并发为 1 时按设计决定、交互回归和路由冒烟的顺序逐个执行。任一文件失败或超时,整轮验证失败。并发的文件共用一个服务和演示库:用例触发的写请求只能落在别的用例不断言的状态上,否则用 `page.route` 拦下或归进时序敏感的串行文件。控制台在每个文件结束时输出成败与耗时;每个文件的 TAP 日志 `e2e-<文件名>.tap`、首启的 `setup.tap` 与服务访问日志 `serve.log` 保存在 `build/agent-verification/browser/`,CI 在测试结束后上传其中的 TAP 日志。
diff --git a/scripts/ci-python-timings.json b/scripts/ci-python-timings.json
new file mode 100644
index 000000000..6fab2fb54
--- /dev/null
+++ b/scripts/ci-python-timings.json
@@ -0,0 +1,299 @@
+{
+ "source_run": "https://github.com/peach-mitao/peach/actions/runs/41017888332",
+ "method": "通过分片日志相邻用例时间戳估计文件耗时;取消分片不纳入",
+ "seconds": {
+ "test_access": 3.718,
+ "test_ad_judgement": 10.673,
+ "test_agency_entity": 57.69,
+ "test_agency_reject": 17.371,
+ "test_agency_roster_harvest": 22.81,
+ "test_agent_worktree": 199.801,
+ "test_auth": 1.49,
+ "test_automatic_updates": 1.743,
+ "test_avatar_face": 0.252,
+ "test_avatar_picker": 11.128,
+ "test_avatar_provider": 0.2,
+ "test_avatar_watermark": 0.105,
+ "test_babepedia_match": 0.005,
+ "test_background_issues": 1.382,
+ "test_bif": 0.066,
+ "test_brand_marks": 2.305,
+ "test_browser_transport": 0.553,
+ "test_buildinfo": 0.009,
+ "test_certs": 4.63,
+ "test_changelog": 0.033,
+ "test_clear_camera_filename_codes": 1.55,
+ "test_cli": 41.691,
+ "test_clouddrive_proxy": 0.425,
+ "test_cloudflared_packaging": 1.255,
+ "test_code_creators": 0.568,
+ "test_community_catalog": 0.673,
+ "test_company_profiles": 3.123,
+ "test_complexity_ratchet": 2.058,
+ "test_composite_name_split": 18.669,
+ "test_config": 0.153,
+ "test_configuration_sources": 0.091,
+ "test_content_region": 2.642,
+ "test_context_budget": 0.045,
+ "test_copy_final_state": 10.695,
+ "test_creator_identity_followup": 1.845,
+ "test_custom_metadata": 0.034,
+ "test_data_storage": 1.017,
+ "test_demo_dataset": 21.071,
+ "test_dependency_policy": 10.963,
+ "test_deploy_windows_tray": 0.029,
+ "test_desktop_installer": 5.64,
+ "test_desktop_settings": 16.381,
+ "test_dev_copy": 0.207,
+ "test_diagnostics": 6.482,
+ "test_directory_link_harvest": 0.069,
+ "test_downloads": 4.477,
+ "test_downloads_channels": 0.024,
+ "test_downloads_clouddrive": 8.515,
+ "test_downloads_clouddrive_organize": 0.027,
+ "test_downloads_jav_browser": 21.406,
+ "test_downloads_pikpak": 0.022,
+ "test_downloads_pikpak_browser": 0.028,
+ "test_downloads_postprocess": 1.721,
+ "test_downloads_resource_search": 0.062,
+ "test_duplicate_identity_merge": 75.204,
+ "test_endcard": 0.396,
+ "test_entity_classification": 6.183,
+ "test_entity_link_install": 3.003,
+ "test_entity_merge": 61.58,
+ "test_entity_redirect": 3.781,
+ "test_entry_links": 0.028,
+ "test_error_reporting": 7.869,
+ "test_external_playback": 0.016,
+ "test_face_detect": 0.048,
+ "test_face_match": 0.018,
+ "test_face_models": 5.342,
+ "test_face_models_bundle": 0.118,
+ "test_fastapi_api": 291.916,
+ "test_fc2_cast_review": 0.344,
+ "test_fc2_metadata": 0.102,
+ "test_fc2_sellers": 2.942,
+ "test_fc2_similarity": 3.607,
+ "test_feeds": 19.801,
+ "test_feeds_web": 37.74,
+ "test_ffmpeg_build": 0.237,
+ "test_ffmpeg_bundle": 0.117,
+ "test_field_owners": 18.239,
+ "test_folder_picker": 0.004,
+ "test_follow_assets": 0.429,
+ "test_follow_avatar": 0.008,
+ "test_follow_check": 0.96,
+ "test_follow_connector_base": 0.068,
+ "test_follow_covers": 0.274,
+ "test_follow_creators": 4.994,
+ "test_follow_discovery": 0.154,
+ "test_follow_faces": 0.065,
+ "test_follow_fanbox": 0.003,
+ "test_follow_gofile": 0.002,
+ "test_follow_image_dims": 0.003,
+ "test_follow_media_cache": 0.624,
+ "test_follow_providers": 0.007,
+ "test_follow_scheduler": 0.025,
+ "test_follow_sources": 0.171,
+ "test_follow_store": 2.038,
+ "test_follow_stream": 0.013,
+ "test_follow_variants": 0.01,
+ "test_follow_web": 50.957,
+ "test_followups": 20.13,
+ "test_frame_capture": 0.99,
+ "test_frontend_build": 0.595,
+ "test_fsutil": 0.018,
+ "test_genre_taxonomy": 0.031,
+ "test_guangya_sources": 5.278,
+ "test_hardware_transcode": 0.323,
+ "test_http": 2.901,
+ "test_interaction": 0.006,
+ "test_jav_cache": 1.556,
+ "test_jav_code_domain": 28.613,
+ "test_jav_cover_artwork": 0.368,
+ "test_jav_covers": 0.723,
+ "test_jav_discovery_metadata": 2.879,
+ "test_jav_official_preview": 0.238,
+ "test_jav_poster_crop": 4.261,
+ "test_javdb_cn_names": 2.494,
+ "test_javdb_public": 23.409,
+ "test_job_status": 0.143,
+ "test_jobs": 0.051,
+ "test_label_maker": 40.845,
+ "test_ledger_backups": 19.731,
+ "test_ledger_revision": 2.285,
+ "test_legacy_shell_routes": 0.016,
+ "test_library_diagnostics": 27.404,
+ "test_library_sharing": 9.195,
+ "test_link_label_owner": 18.822,
+ "test_link_marks": 0.972,
+ "test_link_rediscovery": 0.048,
+ "test_link_repair": 1.704,
+ "test_local_poster": 1.818,
+ "test_log_retention": 0.019,
+ "test_logo_provider": 0.199,
+ "test_macos_media": 0.066,
+ "test_macos_standalone": 2.292,
+ "test_magnet_chain": 0.011,
+ "test_maker_directory_harvest": 0.515,
+ "test_mdns": 0.012,
+ "test_media": 0.03,
+ "test_media_build_cache": 0.119,
+ "test_media_folders": 29.755,
+ "test_media_io": 1.434,
+ "test_media_libraries": 0.044,
+ "test_media_probe": 2.718,
+ "test_media_sources": 0.016,
+ "test_media_tool_import": 0.797,
+ "test_metadata": 0.006,
+ "test_metadata_1pondo": 0.004,
+ "test_metadata_amane": 5.213,
+ "test_metadata_amane_bridge": 0.987,
+ "test_metadata_creator_attributions": 1.191,
+ "test_metadata_dmm": 0.008,
+ "test_metadata_fc2": 0.061,
+ "test_metadata_kmib": 0.392,
+ "test_metadata_library": 202.638,
+ "test_metadata_performer_profiles": 1.172,
+ "test_metadata_policy": 0.003,
+ "test_metadata_r18dev": 0.007,
+ "test_metadata_routes": 0.029,
+ "test_metadata_seesaa": 0.264,
+ "test_metadata_seesaa_name_wikis": 4.493,
+ "test_metadata_sources": 0.026,
+ "test_migrations": 102.767,
+ "test_minnano_av": 0.003,
+ "test_module_layering": 4.169,
+ "test_mount": 0.004,
+ "test_mp4recover": 0.034,
+ "test_mp4repair": 0.103,
+ "test_netwatch": 0.042,
+ "test_network_reachability": 1.022,
+ "test_onboarding": 59.599,
+ "test_organize": 19.797,
+ "test_outbound_proxy_policy": 1.665,
+ "test_performer_agency_resync": 31.068,
+ "test_performer_alias_followup": 4.926,
+ "test_performer_entry_id_backfill": 46.082,
+ "test_performer_header": 0.729,
+ "test_performer_link_harvest": 0.146,
+ "test_performer_localization": 17.764,
+ "test_performer_name_harvest": 0.709,
+ "test_performer_name_variant_link": 0.002,
+ "test_performer_portrait_audit": 3.491,
+ "test_performer_profile_followup": 8.57,
+ "test_platform": 0.269,
+ "test_portrait_artwork": 0.728,
+ "test_portrait_gaps": 4.197,
+ "test_preview_transport": 0.004,
+ "test_previews": 3.447,
+ "test_process_job": 0.376,
+ "test_processing_control": 2.249,
+ "test_processing_inventory": 1.061,
+ "test_processing_traffic": 0.474,
+ "test_providers": 0.001,
+ "test_push_discovery": 5.111,
+ "test_python_lint": 0.251,
+ "test_r18_machine_translation_repair": 18.267,
+ "test_readme_contracts": 0.006,
+ "test_record_rehome": 53.226,
+ "test_reference_updates": 0.037,
+ "test_related": 3.164,
+ "test_release_candidate": 1.123,
+ "test_release_due": 15.744,
+ "test_release_updates": 0.004,
+ "test_remote_reporting": 2.404,
+ "test_repo_hygiene": 7.827,
+ "test_reporting_fault_fixes": 0.355,
+ "test_reporting_projection": 0.178,
+ "test_reporting_transport": 0.498,
+ "test_reproject_snapshot_tags": 1.705,
+ "test_resource_identification": 0.488,
+ "test_restart_windows_tray": 0.177,
+ "test_review_csv": 0.538,
+ "test_review_mirror": 0.017,
+ "test_rm_web": 41.869,
+ "test_runner_launcher": 0.006,
+ "test_runtime_consistency": 0.216,
+ "test_runtime_prepare": 0.405,
+ "test_sample_images": 2.117,
+ "test_scan": 50.691,
+ "test_scraping_access": 0.366,
+ "test_script_policy": 3.778,
+ "test_scripts": 87.213,
+ "test_search_suggest": 5.139,
+ "test_seed_corrections": 3.149,
+ "test_seed_followup": 2.169,
+ "test_seed_pack": 2.222,
+ "test_segments": 0.232,
+ "test_series_localization": 7.055,
+ "test_server": 0.163,
+ "test_site_icons": 0.012,
+ "test_site_logos": 0.014,
+ "test_smb_media": 3.195,
+ "test_smb_transport": 0.188,
+ "test_social_avatar_harvest": 0.541,
+ "test_source_assertion_ratchet": 4.236,
+ "test_stale_candidates": 0.011,
+ "test_standalone_update": 21.171,
+ "test_star_history": 0.009,
+ "test_stream_lifecycle": 1.84,
+ "test_streaming": 0.338,
+ "test_strm": 3.154,
+ "test_studio_followup": 1.514,
+ "test_studio_icon_variants": 18.506,
+ "test_studio_name_localization": 0.354,
+ "test_studio_name_localization_apply": 1.418,
+ "test_studio_name_variant_merge": 1.537,
+ "test_studio_site_harvest": 1.657,
+ "test_subprocess_encoding": 6.088,
+ "test_subtitle_memory": 0.969,
+ "test_subtitle_search": 3.376,
+ "test_subtitles": 1.777,
+ "test_sync": 3.647,
+ "test_sync_dirty": 1.84,
+ "test_tag_management": 6.15,
+ "test_tag_renames": 3.836,
+ "test_task_center_integration": 6.504,
+ "test_task_runs": 48.612,
+ "test_taste_history": 6.276,
+ "test_taste_preferences": 2.072,
+ "test_test_collection": 1.409,
+ "test_test_environment": 1.762,
+ "test_test_evidence": 124.824,
+ "test_test_history": 0.191,
+ "test_test_planning": 12.857,
+ "test_timeline_sheets": 1.633,
+ "test_timeline_sources": 0.273,
+ "test_transcodes": 0.058,
+ "test_trash_junk": 3.006,
+ "test_tray": 1.043,
+ "test_tray_detached": 0.607,
+ "test_tray_diagnostics": 2.335,
+ "test_tray_lifecycle": 3.564,
+ "test_tunnel": 4.463,
+ "test_untrunc_build": 0.033,
+ "test_update_tasks": 1.961,
+ "test_upgrade_repairs": 7.873,
+ "test_version_bump": 8.208,
+ "test_versioning": 10.857,
+ "test_wants": 27.953,
+ "test_wants_web": 3.295,
+ "test_web_downloads": 3.169,
+ "test_web_e2e": 3.788,
+ "test_web_js": 2.25,
+ "test_web_library_shortcuts": 2.482,
+ "test_web_links": 0.738,
+ "test_web_media_repair": 6.029,
+ "test_web_orphan_records": 0.305,
+ "test_web_perf": 1.172,
+ "test_web_resource_sync": 6.418,
+ "test_web_review": 52.2,
+ "test_web_settings": 1.324,
+ "test_web_timeline_thumbnails": 0.031,
+ "test_web_ui": 4.951,
+ "test_western_artwork": 0.195,
+ "test_western_profiles": 0.349,
+ "test_windows_update": 0.122
+ }
+}
diff --git a/scripts/media_build_cache.py b/scripts/media_build_cache.py
index 083701a47..d0b1c899d 100644
--- a/scripts/media_build_cache.py
+++ b/scripts/media_build_cache.py
@@ -38,6 +38,12 @@ def input_identity(root: Path, target: str, arch: str, *, toolchain: dict | None
"toolchain": toolchain, "inputs": {name: digest(root / name) for name in INPUTS}}
+def github_cache_key(identity: dict) -> str:
+ """Actions 外层缓存与包内回执采用同一构建身份。"""
+ encoded = json.dumps(identity, sort_keys=True, separators=(",", ":")).encode("utf-8")
+ return f"{identity['target']}-media-v2-{identity['arch']}-{hashlib.sha256(encoded).hexdigest()}"
+
+
def remove_owned(root: Path, path: Path) -> None:
base = (root / "build").resolve()
if not base.is_relative_to(root.resolve()) or path.is_symlink() or not path.resolve().is_relative_to(base) or path.resolve() == base:
@@ -121,10 +127,14 @@ def main() -> None:
parser.add_argument("target", choices=("windows", "macos"))
parser.add_argument("--arch", choices=("arm64", "x86_64"), default="x86_64")
parser.add_argument("--tool", choices=("ffmpeg", "untrunc"))
+ parser.add_argument("--github-key", action="store_true", help="只输出包含工具链身份的 Actions 缓存键")
args = parser.parse_args()
if args.target == "windows" and args.arch != "x86_64":
parser.error("Windows 媒体工具只构建 x86_64")
- print(json.dumps(build(Path.cwd(), args.target, args.arch, args.tool)))
+ if args.github_key:
+ print(github_cache_key(input_identity(Path.cwd(), args.target, args.arch)))
+ else:
+ print(json.dumps(build(Path.cwd(), args.target, args.arch, args.tool)))
if __name__ == "__main__":
diff --git a/scripts/test_runner.py b/scripts/test_runner.py
index a27f793a2..ee27574d7 100644
--- a/scripts/test_runner.py
+++ b/scripts/test_runner.py
@@ -661,12 +661,12 @@ def without_ci_frontend_tests(suite: unittest.TestSuite) -> unittest.TestSuite:
def build_suite(*scopes: str, shard_index: int = 0, shard_count: int = 1,
assigned_names: Iterable[str] | None = None) -> unittest.TestSuite:
- """本片的用例。`assigned_names` 是父进程按耗时划好的文件名;不给时按文件名轮转(CI 分片)。"""
+ """本片的用例;本机父进程或 Windows CI 按耗时分配,其余按文件名轮转。"""
loader = unittest.defaultTestLoader
suite = unittest.TestSuite()
files = sorted({path for scope in scopes for path in selected_files(scope)})
if assigned_names is None:
- assigned = set(files[shard_index::shard_count])
+ assigned = assigned_files(files, shard_index, shard_count)
else:
assigned = {TESTS / name for name in assigned_names}
if not assigned <= set(files):
@@ -686,6 +686,17 @@ def build_suite(*scopes: str, shard_index: int = 0, shard_count: int = 1,
return without_ci_frontend_tests(suite) if ci_frontend_delegated() else suite
+def assigned_files(files: Iterable[Path], index: int, count: int) -> set[Path]:
+ """Windows 独立 CI 分片共用固定耗时表,本机与未知任务保持原分配。"""
+ files = sorted(files)
+ if (count > 1 and ci_frontend_delegated() and os.environ.get("GITHUB_JOB") == "python"
+ and os.environ.get("RUNNER_OS") == "Windows"):
+ seconds = json.loads((ROOT / "scripts/ci-python-timings.json").read_text(encoding="utf-8"))["seconds"]
+ groups = shard_plan((path.name for path in files), count, seconds, prior_seconds={})
+ return {TESTS / name for name in groups[index]} if index < len(groups) else set()
+ return set(files[index::count])
+
+
class TimedResult(unittest.TextTestResult):
"""逐用例耗时,外加逐文件墙钟。
@@ -844,7 +855,7 @@ PRIOR_SECONDS = {"test_web_e2e.py": 900.0}
def shard_plan(names: Iterable[str], shard_count: int,
- seconds: dict[str, float]) -> list[list[str]]:
+ seconds: dict[str, float], *, prior_seconds: dict[str, float] | None = None) -> list[list[str]]:
"""按预计秒数把测试文件装进分片:先长后短,每个文件放进当前最空的一片。
返回的分片按预计总时长降序排列,派发按这个顺序,最长的先开跑。等负载时按文件数择空。
@@ -854,7 +865,8 @@ def shard_plan(names: Iterable[str], shard_count: int,
names = sorted(names)
known = sorted(value for value in seconds.values() if value > 0)
default = known[len(known) // 2] if known else 1.0
- weight = {name: max(seconds.get(name.removesuffix(".py"), default), PRIOR_SECONDS.get(name, 0.0))
+ priors = PRIOR_SECONDS if prior_seconds is None else prior_seconds
+ weight = {name: max(seconds.get(name.removesuffix(".py"), default), priors.get(name, 0.0))
for name in names}
groups: list[list[str]] = [[] for _ in range(max(1, min(shard_count, len(names))))]
totals = [0.0] * len(groups)
@@ -1175,7 +1187,7 @@ def run_shard_child(args, requested, scopes, files) -> int:
temporary.replace(args.timings)
if args.shard_plan is None:
# CI 的独立分片没有父进程替它汇总,自己写一次历史。
- names = [path.name for path in sorted(files)][args.shard_index::args.shard_count]
+ names = sorted(path.name for path in assigned_files(files, args.shard_index, args.shard_count))
test_history.append(ROOT, scope=f"{' '.join(requested)}:{' '.join(scopes)} "
f"shard {args.shard_index + 1}/{args.shard_count}",
rows=rows, files=[name.removesuffix(".py") for name in names])
diff --git a/tests/test_media_build_cache.py b/tests/test_media_build_cache.py
index 20615e9e4..8382b0a66 100644
--- a/tests/test_media_build_cache.py
+++ b/tests/test_media_build_cache.py
@@ -75,3 +75,22 @@ class MediaBuildCacheTests(unittest.TestCase):
self.assertNotEqual(first, cache.input_identity(root, "macos", "arm64", toolchain={"sdk": "two"}))
(root / cache.INPUTS[0]).write_text("different source revision")
self.assertNotEqual(first, cache.input_identity(root, "macos", "arm64", toolchain={"sdk": "one"}))
+
+ def test_actions_cache_identity_tracks_toolchain_and_source_changes(self):
+ with tempfile.TemporaryDirectory() as directory:
+ root = Path(directory).resolve()
+ for name in cache.INPUTS:
+ path = root / name
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text("input")
+ first = cache.input_identity(root, "macos", "arm64", toolchain={"sdk": "one", "clang": "one"})
+ key = cache.github_cache_key(first)
+ reordered = cache.input_identity(root, "macos", "arm64", toolchain={"clang": "one", "sdk": "one"})
+ self.assertEqual(key, cache.github_cache_key(reordered))
+ for target, arch, tools in (("macos", "x86_64", first["toolchain"]),
+ ("macos", "arm64", {"sdk": "two", "clang": "one"}),
+ ("windows", "x86_64", first["toolchain"])):
+ with self.subTest(target=target, arch=arch, tools=tools):
+ self.assertNotEqual(key, cache.github_cache_key(cache.input_identity(root, target, arch, toolchain=tools)))
+ (root / cache.INPUTS[0]).write_text("new source")
+ self.assertNotEqual(key, cache.github_cache_key(cache.input_identity(root, "macos", "arm64", toolchain=first["toolchain"])))
diff --git a/tests/test_test_planning.py b/tests/test_test_planning.py
index 6caa3b599..9356c2abc 100644
--- a/tests/test_test_planning.py
+++ b/tests/test_test_planning.py
@@ -293,6 +293,27 @@ class TestPlanningTests(unittest.TestCase):
costs = [row['estimated_seconds'] for row in matrix]
self.assertLess(max(costs) - min(costs), 60)
+ def test_windows_python_partitions_balance_recorded_cost_and_keep_every_case(self):
+ seconds = json.loads((runner.ROOT / 'scripts/ci-python-timings.json').read_text(encoding='utf-8'))['seconds']
+ files = runner.selected_files('full')
+ with patch.dict(os.environ, {**CIFrontendOwnershipTests.CI, 'RUNNER_OS': 'Windows'}):
+ groups = [runner.assigned_files(files, index, 4) for index in range(4)]
+ serial = list(CIFrontendOwnershipTests.ids(runner.build_suite('full')))
+ collected = [name for index in range(4) for name in CIFrontendOwnershipTests.ids(
+ runner.build_suite('full', shard_index=index, shard_count=4))]
+ self.assertCountEqual(serial, collected)
+ self.assertEqual(len(collected), len(set(collected)))
+ self.assertCountEqual(files, [path for group in groups for path in group])
+ costs = [sum(seconds.get(path.stem, 1) for path in group) for group in groups]
+ self.assertLess(max(costs) - min(costs), 5)
+ self.assertGreater(len(groups[0]), 1)
+
+ def test_unassigned_ci_and_macos_keep_filename_partitions(self):
+ files = [runner.TESTS / f'test_selected_{index}.py' for index in range(8)]
+ for environment in ({'RUNNER_OS': 'Windows'}, {**CIFrontendOwnershipTests.CI, 'RUNNER_OS': 'macOS'}):
+ with self.subTest(environment=environment), patch.dict(os.environ, environment, clear=True):
+ self.assertEqual(runner.assigned_files(files, 1, 4), set(files[1::4]))
+
def test_broad_auto_matrix_keeps_each_selected_test_once(self):
for paths in (['src/peach/mp4repair.py', 'src/peach/web_media_repair.py'],
['src/peach/web_catalog.py']):