Skip to content

Free fork of NotProton: Windows Steam games in Mac Steam, on CodeWeavers' Wine and Apple's Game Porting Toolkit

build(local): seed new worktrees from the main checkout's build caches

raymondchinscbf67e9aa54c
diff --git a/docs/BUILD-LOCAL.md b/docs/BUILD-LOCAL.md
index b3d849a..8c427c7 100644
--- a/docs/BUILD-LOCAL.md
+++ b/docs/BUILD-LOCAL.md
@@ -86,9 +86,27 @@ disk. All of these paths are gitignored:
`register-components.diff` changes, the script prints a warning. To rebuild fresh the way
CI would, run `rm -rf scratch/wine scratch/wine-build-dual scratch/wine-build-arm64 scratch/.wine-key`.
-The Wine build trees record their source tree by absolute path (`srcdir = <repo>/scratch/wine`).
-To reuse them from another checkout, symlink that checkout's `scratch` to this one. Don't
-copy them. Two builds sharing one tree must not run at the same time.
+The bridge phase is skipped when `scratch/.bridge-key` (written after each bridge build)
+matches `scripts/bridge-key.sh`, a hash of `bridge/`, `lsteamclient/` and `steam-shim/`, the
+same sources CI keys its bridge cache on.
+
+### A new worktree
+
+The Wine build trees record their source tree by absolute path (`srcdir = <repo>/scratch/wine`),
+so they can't be copied, and two checkouts building in one shared tree would overwrite each
+other's bridge. Seed the worktree instead, then build:
+
+```sh
+scripts/worktree-seed.sh # from the main worktree; or pass another checkout
+scripts/build-local.sh
+```
+
+The seed clones (APFS `cp -c`: instant, no extra disk) Dobby, the fonts and their stamp, the
+Proton tarball and, when both checkouts have the same bridge sources, the five bridge outputs.
+The font venv is linked. Measured on the M5 Max: the seed takes 1–2s and the first build 45s,
+instead of about 15m. When the bridge sources differ, the Wine source is cloned instead and the
+bridge is built in the worktree without the 8m30s clone: 3m21s, 4m05s for the whole build.
+`app/.build` is not seeded: SwiftPM records absolute paths, and a cold app build is 35s.
## Local deviation from CI
diff --git a/scripts/bridge-key.sh b/scripts/bridge-key.sh
new file mode 100644
index 0000000..e1e7800
--- /dev/null
+++ b/scripts/bridge-key.sh
@@ -0,0 +1,21 @@
+#!/bin/sh
+# The bridge build's cache key, shared by build-local.sh and worktree-seed.sh.
+# scripts/bridge-key.sh a hash of the sources app.yml keys its bridge cache on
+# (bridge/, lsteamclient/, steam-shim/), uncommitted edits included
+# scripts/bridge-key.sh --outputs the bridge outputs, the paths app.yml's bridge cache saves
+set -eu
+cd "$(dirname "$0")/.."
+
+if [ "${1:-}" = --outputs ]; then
+ cat <<'EOF'
+scratch/wine-build-dual/programs/steam.exe/x86_64-windows/steam.exe
+scratch/wine-build-dual/dlls/lsteamclient/x86_64-windows/lsteamclient.dll
+scratch/wine-build-dual/dlls/lsteamclient/lsteamclient.so
+scratch/wine-build-dual/dlls/lsteamclient/i386-windows/lsteamclient.dll
+scratch/wine-build-arm64/dlls/lsteamclient/lsteamclient.so
+EOF
+ exit 0
+fi
+
+git ls-files -co --exclude-standard -z bridge lsteamclient steam-shim \
+ | xargs -0 shasum -a 256 2>/dev/null | shasum -a 256 | cut -c1-16
diff --git a/scripts/build-local.sh b/scripts/build-local.sh
index a2fee7b..4cce85f 100755
--- a/scripts/build-local.sh
+++ b/scripts/build-local.sh
@@ -13,6 +13,8 @@
# scratch/wine-build-arm64 arm64 tree (FEX unix half)
# scratch/proton-lsc Proton tarball for lsteamclient
# app/.build SwiftPM build and the Sparkle checkout
+# The bridge phase is skipped when scratch/.bridge-key matches the bridge sources.
+# A new worktree: scripts/worktree-seed.sh first, then this.
#
# Overridable: DEVELOPER_DIR (default Xcode-beta), JOBS (default hw.ncpu)
# A beta build: BRIDGEPLAY_BETA=N BRIDGEPLAY_BETA_BUILD=B [BRIDGEPLAY_BETA_PLANNED=X.Y.Z], see docs/RELEASING.md
@@ -172,9 +174,24 @@ if [ -f "$wine_stamp" ] && [ "$(cat "$wine_stamp")" != "$wine_key" ]; then
echo " rm -rf scratch/wine scratch/wine-build-dual scratch/wine-build-arm64 $wine_stamp"
fi
-# The wine-tree scripts call plain make, which joins this make's jobserver.
-phase bridge make -j"$JOBS" bridge
-[ -f "$wine_stamp" ] || echo "$wine_key" >"$wine_stamp"
+# Like CI's bridge cache: outputs built from these exact sources are reused, which is how a
+# worktree seeded by worktree-seed.sh skips the Wine clone and build.
+bridge_key=$(sh scripts/bridge-key.sh)
+bridge_stamp=scratch/.bridge-key
+bridge_built=0
+if [ "$(cat "$bridge_stamp" 2>/dev/null)" = "$bridge_key" ]; then
+ bridge_built=1
+ for f in $(sh scripts/bridge-key.sh --outputs); do [ -f "$f" ] || bridge_built=0; done
+fi
+if [ "$bridge_built" -eq 1 ]; then
+ echo "==> [bridge] skipped: already built from these sources ($bridge_key)"
+else
+ rm -f "$bridge_stamp"
+ # The wine-tree scripts call plain make, which joins this make's jobserver.
+ phase bridge make -j"$JOBS" bridge
+ [ -f "$wine_stamp" ] || echo "$wine_key" >"$wine_stamp"
+ echo "$bridge_key" >"$bridge_stamp"
+fi
# --- fonts -------------------------------------------------------------------------
@@ -185,6 +202,8 @@ for tool in fontforge gpatch; do
|| { echo "==> $tool is missing, run: brew install fontforge gpatch" >&2; exit 1; }
done
if [ ! -x scratch/fontvenv/bin/makeotfexe ] || [ "$(cat scratch/fontvenv/.tools 2>/dev/null)" != "$font_tools" ]; then
+ # A venv shared by worktree-seed.sh belongs to the other checkout: replace the link, not that venv.
+ [ ! -L scratch/fontvenv ] || rm scratch/fontvenv
python3 -m venv scratch/fontvenv
# shellcheck disable=SC2086 # FONT_TOOLS is a space-separated requirement list
scratch/fontvenv/bin/pip install -q $font_tools
diff --git a/scripts/tests/cicheck.sh b/scripts/tests/cicheck.sh
index f87a3fb..cb53f26 100644
--- a/scripts/tests/cicheck.sh
+++ b/scripts/tests/cicheck.sh
@@ -372,5 +372,10 @@ yes "scope: runs the scope script into the job output" has ' run: sh scri
yes "scope: output is docs_only" has ' docs_only: ${{ steps.scope.outputs.docs_only }}' "$scope_job"
yes "build-local.sh reuses runs through ci-run-passed.sh" grep -q 'ci-run-passed.sh' "$root/scripts/build-local.sh"
+# bridge-key.sh: the outputs build-local.sh and worktree-seed.sh reuse are the ones CI's bridge cache saves
+ci_bridge=$(sed -n '/^ key: bridge-/q;/^ path: |$/,$p' "$app" | sed -n 's/^ \(scratch\/.*\)$/\1/p' | sort)
+eq "bridge-key.sh --outputs = app.yml bridge cache" "$ci_bridge" "$(sh "$root/scripts/bridge-key.sh" --outputs | sort)"
+yes "bridge key is a 16-digit hash" sh -c '[ "$(sh "$1/scripts/bridge-key.sh" | tr -d 0-9a-f)" = "" ] && [ "$(sh "$1/scripts/bridge-key.sh" | wc -c)" -eq 17 ]' _ "$root"
+
echo "cicheck: $passed passed, $failed failed"
[ "$failed" -eq 0 ]
diff --git a/scripts/worktree-seed.sh b/scripts/worktree-seed.sh
new file mode 100644
index 0000000..490af9a
--- /dev/null
+++ b/scripts/worktree-seed.sh
@@ -0,0 +1,54 @@
+#!/bin/sh
+# Seed a fresh git worktree with another checkout's build caches, so its first
+# scripts/build-local.sh takes about a minute instead of about 15.
+#
+# scripts/worktree-seed.sh [SOURCE] SOURCE defaults to the main worktree
+#
+# Copies are APFS clones (cp -c): instant and no extra disk; other volumes get a plain copy.
+# Only caches that don't record their checkout's path come over. The Wine build trees do
+# (srcdir = <repo>/scratch/wine), so the bridge comes over as its outputs, and only when
+# both checkouts have the same bridge sources. Otherwise the Wine source comes over, and
+# build-local.sh builds the bridge here without cloning Wine. Nothing in SOURCE is written.
+set -eu
+repo=$(cd "$(dirname "$0")/.." && pwd)
+src=${1:-$(git -C "$repo" worktree list --porcelain | sed -n '1s/^worktree //p')}
+src=$(cd "$src" && pwd)
+[ "$src" != "$repo" ] || { echo "==> seed: this is the source checkout; run it in another worktree" >&2; exit 1; }
+cd "$repo"
+echo "==> seeding $repo from $src"
+
+# clone PATH: from SOURCE, unless it is already here or SOURCE has none
+clone() {
+ if [ -e "$1" ] || [ -L "$1" ]; then echo " $1: already here"; return 0; fi
+ if [ ! -e "$src/$1" ]; then echo " $1: not in the source"; return 0; fi
+ mkdir -p "$(dirname "$1")"
+ cp -c -R "$src/$1" "$1" 2>/dev/null || { rm -rf "$1"; cp -R "$src/$1" "$1"; }
+ echo " $1"
+}
+
+clone vendor/dobby
+clone build/dobby
+clone build/fonts
+clone build/fonts.stamp
+clone scratch/proton-lsc
+
+# The font venv's scripts name its path, so it is shared, not copied, while both checkouts
+# want the same tools; build-local.sh replaces the link with its own venv when they don't.
+tools=$(sed -n 's/^ *FONT_TOOLS: "\(.*\)"$/\1/p' .github/workflows/app.yml)
+if [ ! -e scratch/fontvenv ] && [ -n "$tools" ] && [ "$(cat "$src/scratch/fontvenv/.tools" 2>/dev/null)" = "$tools" ]; then
+ mkdir -p scratch
+ ln -s "$src/scratch/fontvenv" scratch/fontvenv
+ echo " scratch/fontvenv (link)"
+fi
+
+key=$(sh scripts/bridge-key.sh)
+if [ "$(cat "$src/scratch/.bridge-key" 2>/dev/null)" = "$key" ] && [ "$(sh "$src/scripts/bridge-key.sh")" = "$key" ]; then
+ for f in $(sh scripts/bridge-key.sh --outputs); do clone "$f"; done
+ echo "$key" >scratch/.bridge-key
+ echo "==> bridge: same sources as the source checkout, its build is reused"
+else
+ clone scratch/wine
+ clone scratch/.wine-key
+ echo "==> bridge: sources differ from the source checkout's last bridge build;"
+ echo " build-local.sh builds it here from the cloned Wine source"
+fi