Free fork of NotProton: Windows Steam games in Mac Steam, on CodeWeavers' Wine and Apple's Game Porting Toolkit
Swift apple-silicond3dmetaldxmtdxvkgame-porting-toolkitgptkmac-gamingmacosmetalfxrosetta-2steamsteam-playwinewinecx
build(local): seed new worktrees from the main checkout's build caches
raymondchinscbf67e9aa54c
diff --git a/docs/BUILD-LOCAL.md b/docs/BUILD-LOCAL.mdindex b3d849a..8c427c7 100644--- a/docs/BUILD-LOCAL.md+++ b/docs/BUILD-LOCAL.md@@ -86,9 +86,27 @@ disk. All of these paths are gitignored:`register-components.diff` changes, the script prints a warning. To rebuild fresh the wayCI would, run `rm -rf scratch/wine scratch/wine-build-dual scratch/wine-build-arm64 scratch/.wine-key`.-The Wine build trees record their source tree by absolute path (`srcdir = <repo>/scratch/wine`).-To reuse them from another checkout, symlink that checkout's `scratch` to this one. Don't-copy them. Two builds sharing one tree must not run at the same time.+The bridge phase is skipped when `scratch/.bridge-key` (written after each bridge build)+matches `scripts/bridge-key.sh`, a hash of `bridge/`, `lsteamclient/` and `steam-shim/`, the+same sources CI keys its bridge cache on.++### A new worktree++The Wine build trees record their source tree by absolute path (`srcdir = <repo>/scratch/wine`),+so they can't be copied, and two checkouts building in one shared tree would overwrite each+other's bridge. Seed the worktree instead, then build:++```sh+scripts/worktree-seed.sh # from the main worktree; or pass another checkout+scripts/build-local.sh+```++The seed clones (APFS `cp -c`: instant, no extra disk) Dobby, the fonts and their stamp, the+Proton tarball and, when both checkouts have the same bridge sources, the five bridge outputs.+The font venv is linked. Measured on the M5 Max: the seed takes 1–2s and the first build 45s,+instead of about 15m. When the bridge sources differ, the Wine source is cloned instead and the+bridge is built in the worktree without the 8m30s clone: 3m21s, 4m05s for the whole build.+`app/.build` is not seeded: SwiftPM records absolute paths, and a cold app build is 35s.## Local deviation from CIdiff --git a/scripts/bridge-key.sh b/scripts/bridge-key.shnew file mode 100644index 0000000..e1e7800--- /dev/null+++ b/scripts/bridge-key.sh@@ -0,0 +1,21 @@+#!/bin/sh+# The bridge build's cache key, shared by build-local.sh and worktree-seed.sh.+# scripts/bridge-key.sh a hash of the sources app.yml keys its bridge cache on+# (bridge/, lsteamclient/, steam-shim/), uncommitted edits included+# scripts/bridge-key.sh --outputs the bridge outputs, the paths app.yml's bridge cache saves+set -eu+cd "$(dirname "$0")/.."++if [ "${1:-}" = --outputs ]; then+ cat <<'EOF'+scratch/wine-build-dual/programs/steam.exe/x86_64-windows/steam.exe+scratch/wine-build-dual/dlls/lsteamclient/x86_64-windows/lsteamclient.dll+scratch/wine-build-dual/dlls/lsteamclient/lsteamclient.so+scratch/wine-build-dual/dlls/lsteamclient/i386-windows/lsteamclient.dll+scratch/wine-build-arm64/dlls/lsteamclient/lsteamclient.so+EOF+ exit 0+fi++git ls-files -co --exclude-standard -z bridge lsteamclient steam-shim \+ | xargs -0 shasum -a 256 2>/dev/null | shasum -a 256 | cut -c1-16diff --git a/scripts/build-local.sh b/scripts/build-local.shindex a2fee7b..4cce85f 100755--- a/scripts/build-local.sh+++ b/scripts/build-local.sh@@ -13,6 +13,8 @@# scratch/wine-build-arm64 arm64 tree (FEX unix half)# scratch/proton-lsc Proton tarball for lsteamclient# app/.build SwiftPM build and the Sparkle checkout+# The bridge phase is skipped when scratch/.bridge-key matches the bridge sources.+# A new worktree: scripts/worktree-seed.sh first, then this.## Overridable: DEVELOPER_DIR (default Xcode-beta), JOBS (default hw.ncpu)# A beta build: BRIDGEPLAY_BETA=N BRIDGEPLAY_BETA_BUILD=B [BRIDGEPLAY_BETA_PLANNED=X.Y.Z], see docs/RELEASING.md@@ -172,9 +174,24 @@ if [ -f "$wine_stamp" ] && [ "$(cat "$wine_stamp")" != "$wine_key" ]; thenecho " rm -rf scratch/wine scratch/wine-build-dual scratch/wine-build-arm64 $wine_stamp"fi-# The wine-tree scripts call plain make, which joins this make's jobserver.-phase bridge make -j"$JOBS" bridge-[ -f "$wine_stamp" ] || echo "$wine_key" >"$wine_stamp"+# Like CI's bridge cache: outputs built from these exact sources are reused, which is how a+# worktree seeded by worktree-seed.sh skips the Wine clone and build.+bridge_key=$(sh scripts/bridge-key.sh)+bridge_stamp=scratch/.bridge-key+bridge_built=0+if [ "$(cat "$bridge_stamp" 2>/dev/null)" = "$bridge_key" ]; then+ bridge_built=1+ for f in $(sh scripts/bridge-key.sh --outputs); do [ -f "$f" ] || bridge_built=0; done+fi+if [ "$bridge_built" -eq 1 ]; then+ echo "==> [bridge] skipped: already built from these sources ($bridge_key)"+else+ rm -f "$bridge_stamp"+ # The wine-tree scripts call plain make, which joins this make's jobserver.+ phase bridge make -j"$JOBS" bridge+ [ -f "$wine_stamp" ] || echo "$wine_key" >"$wine_stamp"+ echo "$bridge_key" >"$bridge_stamp"+fi# --- fonts -------------------------------------------------------------------------@@ -185,6 +202,8 @@ for tool in fontforge gpatch; do|| { echo "==> $tool is missing, run: brew install fontforge gpatch" >&2; exit 1; }doneif [ ! -x scratch/fontvenv/bin/makeotfexe ] || [ "$(cat scratch/fontvenv/.tools 2>/dev/null)" != "$font_tools" ]; then+ # A venv shared by worktree-seed.sh belongs to the other checkout: replace the link, not that venv.+ [ ! -L scratch/fontvenv ] || rm scratch/fontvenvpython3 -m venv scratch/fontvenv# shellcheck disable=SC2086 # FONT_TOOLS is a space-separated requirement listscratch/fontvenv/bin/pip install -q $font_toolsdiff --git a/scripts/tests/cicheck.sh b/scripts/tests/cicheck.shindex f87a3fb..cb53f26 100644--- a/scripts/tests/cicheck.sh+++ b/scripts/tests/cicheck.sh@@ -372,5 +372,10 @@ yes "scope: runs the scope script into the job output" has ' run: sh scriyes "scope: output is docs_only" has ' docs_only: ${{ steps.scope.outputs.docs_only }}' "$scope_job"yes "build-local.sh reuses runs through ci-run-passed.sh" grep -q 'ci-run-passed.sh' "$root/scripts/build-local.sh"+# bridge-key.sh: the outputs build-local.sh and worktree-seed.sh reuse are the ones CI's bridge cache saves+ci_bridge=$(sed -n '/^ key: bridge-/q;/^ path: |$/,$p' "$app" | sed -n 's/^ \(scratch\/.*\)$/\1/p' | sort)+eq "bridge-key.sh --outputs = app.yml bridge cache" "$ci_bridge" "$(sh "$root/scripts/bridge-key.sh" --outputs | sort)"+yes "bridge key is a 16-digit hash" sh -c '[ "$(sh "$1/scripts/bridge-key.sh" | tr -d 0-9a-f)" = "" ] && [ "$(sh "$1/scripts/bridge-key.sh" | wc -c)" -eq 17 ]' _ "$root"+echo "cicheck: $passed passed, $failed failed"[ "$failed" -eq 0 ]diff --git a/scripts/worktree-seed.sh b/scripts/worktree-seed.shnew file mode 100644index 0000000..490af9a--- /dev/null+++ b/scripts/worktree-seed.sh@@ -0,0 +1,54 @@+#!/bin/sh+# Seed a fresh git worktree with another checkout's build caches, so its first+# scripts/build-local.sh takes about a minute instead of about 15.+#+# scripts/worktree-seed.sh [SOURCE] SOURCE defaults to the main worktree+#+# Copies are APFS clones (cp -c): instant and no extra disk; other volumes get a plain copy.+# Only caches that don't record their checkout's path come over. The Wine build trees do+# (srcdir = <repo>/scratch/wine), so the bridge comes over as its outputs, and only when+# both checkouts have the same bridge sources. Otherwise the Wine source comes over, and+# build-local.sh builds the bridge here without cloning Wine. Nothing in SOURCE is written.+set -eu+repo=$(cd "$(dirname "$0")/.." && pwd)+src=${1:-$(git -C "$repo" worktree list --porcelain | sed -n '1s/^worktree //p')}+src=$(cd "$src" && pwd)+[ "$src" != "$repo" ] || { echo "==> seed: this is the source checkout; run it in another worktree" >&2; exit 1; }+cd "$repo"+echo "==> seeding $repo from $src"++# clone PATH: from SOURCE, unless it is already here or SOURCE has none+clone() {+ if [ -e "$1" ] || [ -L "$1" ]; then echo " $1: already here"; return 0; fi+ if [ ! -e "$src/$1" ]; then echo " $1: not in the source"; return 0; fi+ mkdir -p "$(dirname "$1")"+ cp -c -R "$src/$1" "$1" 2>/dev/null || { rm -rf "$1"; cp -R "$src/$1" "$1"; }+ echo " $1"+}++clone vendor/dobby+clone build/dobby+clone build/fonts+clone build/fonts.stamp+clone scratch/proton-lsc++# The font venv's scripts name its path, so it is shared, not copied, while both checkouts+# want the same tools; build-local.sh replaces the link with its own venv when they don't.+tools=$(sed -n 's/^ *FONT_TOOLS: "\(.*\)"$/\1/p' .github/workflows/app.yml)+if [ ! -e scratch/fontvenv ] && [ -n "$tools" ] && [ "$(cat "$src/scratch/fontvenv/.tools" 2>/dev/null)" = "$tools" ]; then+ mkdir -p scratch+ ln -s "$src/scratch/fontvenv" scratch/fontvenv+ echo " scratch/fontvenv (link)"+fi++key=$(sh scripts/bridge-key.sh)+if [ "$(cat "$src/scratch/.bridge-key" 2>/dev/null)" = "$key" ] && [ "$(sh "$src/scripts/bridge-key.sh")" = "$key" ]; then+ for f in $(sh scripts/bridge-key.sh --outputs); do clone "$f"; done+ echo "$key" >scratch/.bridge-key+ echo "==> bridge: same sources as the source checkout, its build is reused"+else+ clone scratch/wine+ clone scratch/.wine-key+ echo "==> bridge: sources differ from the source checkout's last bridge build;"+ echo " build-local.sh builds it here from the cloned Wine source"+fi