Skip to content

Play your own Switch games on iPhone/iPad — one-click macOS app, no terminal. | 把 Switch 游戏装进 iPhone:图形界面一键构建,不用命令行。Bring your own game.

Experimental builder: bundle the pinned nxbuild release with the app (#749)

v0lumehib37a935ee529
diff --git a/.github/scripts/build_wanted.py b/.github/scripts/build_wanted.py
index 5a037315..8cf7fb86 100644
--- a/.github/scripts/build_wanted.py
+++ b/.github/scripts/build_wanted.py
@@ -35,6 +35,7 @@ PACKAGING = (
'frontend/Package.swift',
'frontend/Package.resolved',
'frontend/engine-signing.pub',
+ 'frontend/nxbuild.pin',
'frontend/Resources/',
'frontend/AppIcon.icon/',
'frontend/bundled-python/',
diff --git a/.github/workflows/build-mac-app.yml b/.github/workflows/build-mac-app.yml
index 7f372155..da049995 100644
--- a/.github/workflows/build-mac-app.yml
+++ b/.github/workflows/build-mac-app.yml
@@ -341,6 +341,9 @@ jobs:
# The build tools and the Python go into Resources before the bundle is signed (bundle_tools.sh signs each nested Mach-O file
# itself: --deep below does not reach code that sits in Resources).
tools/bundle_tools.sh "$APP/Contents/Resources" "$RUNNER_TEMP/bundle-tools-work"
+ # The experimental builder (#749): the nxbuild release nxbuild.pin names, its SHA-256 checked before it is unpacked, nxb signed ad hoc into
+ # Resources/nxbuild. A placeholder digest (no release yet) bundles nothing: the launcher's setting then needs a chosen nxb.
+ python3 tools/nxbuild.py fetch "$APP/Contents/Resources" "$RUNNER_TEMP/nxbuild-work"
# The engine tree (Contents/ is read-only once signed: the launcher copies it to ~/Library/Application Support/NX2Apple/ on first
# use). --deep does not reach code in Resources, but its Mach-O files (the shader translator, spirv-cross) are kept as upstream built
# them, already signed: re-signing rewrote spirv-cross, which dmsh_chain.lock pins by its bytes, and stopped every shader build from
diff --git a/frontend/README.md b/frontend/README.md
index 1e2d7536..928a0eb1 100644
--- a/frontend/README.md
+++ b/frontend/README.md
@@ -279,14 +279,15 @@ The build log says which stage nxbuild ran and why a stage fell back. nxbuild's
Where `nxb` comes from (`ExperimentalBuilder` in `NXCore`):
1. The path chosen in the row under the switch, the `nxb` binary or the folder that holds it. A chosen path that does not work is reported, never replaced by another binary.
-2. Otherwise a downloaded release in `~/Library/Application Support/NS Port Builder/nxbuild/<version>/`. No nxbuild release is published yet, so `ExperimentalBuilder.releaseURL` and `releaseSHA256` are nil and nothing is downloaded. Once a release exists (a macOS arm64 zip of the binaries, signed ad hoc), both are pinned together in that file. The download goes to a temporary file, its SHA-256 is checked against `releaseSHA256` before anything is unpacked, and a zip whose digest differs is deleted and never run.
+2. Otherwise the app's own copy, `Contents/Resources/nxbuild/nxb`. `frontend/nxbuild.pin` names the nxbuild release (version, SHA-256, URL pattern). `build-mac-app.yml` and `tools/try-local` run `frontend/tools/nxbuild.py fetch`: it downloads the zip, checks its SHA-256 against the pin before anything is unpacked, puts `nxb` into Resources and signs it ad hoc. A zip whose digest differs fails the app build. A pin with `PLACEHOLDER` as its digest bundles nothing. `frontend/tools/nxbuild.py bump <version>` pins a new release with the digest from its `SHA256SUMS`. Players download nothing at run time.
With neither, the switch only writes a note in the build log and the standard builder runs.
-To try it with a local nxbuild:
+To try a local nxbuild:
```bash
cd <nxbuild checkout> && cargo build --release # the binaries land in its target folder, release/
+codesign --force --sign - <target>/release/nxb* # ad hoc, as the app's copy
```
Then turn the switch on and choose `release/nxb` (or the folder). From a terminal, the same works without the launcher:
@@ -295,6 +296,8 @@ Then turn the switch on and choose `release/nxb` (or the folder). From a termina
DM_EXPERIMENTAL_BUILDER=1 NXB_BIN=<target>/release/nxb python3 engine/scripts/build_ipa_from_nsp.py <NSP folder> -k <prod.keys> --platform mac-app
```
+The script takes a stage binary beside `nxb` (`nxb-extract`, `nxb-split`) when there is one, else the subcommand of `nxb` (`nxb extract`, `nxb split`).
+
The Qt front end has no switch for it: nxbuild is built for macOS on Apple silicon only, and on a Mac the launcher above is the one to use.
## Run
diff --git a/frontend/Sources/NXCore/ExperimentalBuilder.swift b/frontend/Sources/NXCore/ExperimentalBuilder.swift
index 62091942..8678eb32 100644
--- a/frontend/Sources/NXCore/ExperimentalBuilder.swift
+++ b/frontend/Sources/NXCore/ExperimentalBuilder.swift
@@ -2,26 +2,16 @@ import Foundation
/// The experimental builder (LNPANDY/NX2Apple#749): nxbuild, a build tool in Rust that reproduces stages of the NSP script byte for byte and faster.
/// The script stays the driver. With `DM_EXPERIMENTAL_BUILDER=1` and `NXB_BIN` it hands the stages nxbuild supports (the extraction and the shader split today)
-/// to the binaries beside `nxb`, and runs every other case the standard way. An engine without that patch ignores both variables.
+/// to nxbuild, and runs every other case the standard way. An engine without that patch ignores both variables.
///
/// Where `nxb` comes from:
/// 1. the path the player chose in Settings (the `nxb` binary, or the folder that holds it), for a local build of nxbuild;
-/// 2. a release unpacked into `installFolder` by a download on demand. That download needs a published release: `releaseURL` and `releaseSHA256` are
-/// pinned here together once it exists (a macOS arm64 zip of the binaries, signed ad hoc). Until then both are nil and only the chosen path is used.
-/// The download writes the zip to a temporary file, checks its SHA-256 against `releaseSHA256` before anything is unpacked, and unpacks it into
-/// `installFolder/<version>`. A zip whose digest differs is deleted and never run.
+/// 2. the app's own copy, `Contents/Resources/nxbuild/nxb`: the release `frontend/nxbuild.pin` names, downloaded when the app is built, its SHA-256
+/// checked against the pin before it is unpacked (`frontend/tools/nxbuild.py fetch`). Players download nothing at run time.
package enum ExperimentalBuilder {
- /// The nxbuild release the launcher downloads on demand. nil: none is published yet.
- package static let releaseURL: URL? = nil
- /// The SHA-256 (lowercase hex) of the zip at `releaseURL`, checked before it is unpacked. nil with it.
- package static let releaseSHA256: String? = nil
-
- /// The binaries the NSP script runs, all beside `nxb`. A missing stage binary makes that stage run the standard way.
- package static let stageTools = ["nxb-extract", "nxb-split"]
-
- /// Where a downloaded release goes: `<support>/nxbuild`, beside the venv in the launcher's data folder.
- package static func installFolder(support: String) -> String {
- (support as NSString).appendingPathComponent("nxbuild")
+ /// The app's copy of nxb under a Resources folder.
+ package static func bundled(resources: String?) -> String? {
+ resources.map { ($0 as NSString).appendingPathComponent("nxbuild/nxb") }
}
/// The `nxb` a chosen path names: the file itself, or `nxb` inside a folder. Empty for an empty path.
@@ -31,16 +21,14 @@ package enum ExperimentalBuilder {
return isDirectory(p) ? (p as NSString).appendingPathComponent("nxb") : p
}
- /// The `nxb` the build takes: the chosen path when it is an executable `nxb`, else the newest release in `installFolder`. nil when there is none.
- package static func resolve(customPath: String, support: String, isExecutable: (String) -> Bool = { FileManager.default.isExecutableFile(atPath: $0) },
- isDirectory: (String) -> Bool = directoryExists, list: (String) -> [String] = listing) -> String? {
+ /// The `nxb` the build takes: the chosen path when it is an executable `nxb`, else the app's copy. nil when there is none.
+ package static func resolve(customPath: String, bundled: String?, isExecutable: (String) -> Bool = { FileManager.default.isExecutableFile(atPath: $0) },
+ isDirectory: (String) -> Bool = directoryExists) -> String? {
let chosen = binary(at: customPath, isDirectory: isDirectory)
if !chosen.isEmpty {
- return isExecutable(chosen) ? chosen : nil // a chosen path that does not work is not silently replaced by another binary
+ return isExecutable(chosen) ? chosen : nil // a chosen path that does not work is reported, not silently replaced by the app's copy
}
- let folder = installFolder(support: support)
- let versions = list(folder).sorted { $0.compare($1, options: .numeric) == .orderedDescending }
- return versions.lazy.map { ((folder as NSString).appendingPathComponent($0) as NSString).appendingPathComponent("nxb") }.first(where: isExecutable)
+ return bundled.flatMap { isExecutable($0) ? $0 : nil }
}
/// The variables the build gets: the switch and the path of `nxb`.
@@ -48,18 +36,8 @@ package enum ExperimentalBuilder {
["DM_EXPERIMENTAL_BUILDER": "1", "NXB_BIN": nxb]
}
- /// Whether a downloaded zip is the pinned release: its SHA-256 (lowercase hex, as `shasum -a 256` prints it) against `releaseSHA256`.
- package static func matchesRelease(sha256 digest: String, expected: String? = releaseSHA256) -> Bool {
- guard let expected, expected.count == 64 else { return false }
- return digest.lowercased() == expected.lowercased()
- }
-
package static func directoryExists(_ path: String) -> Bool {
var dir: ObjCBool = false
return FileManager.default.fileExists(atPath: path, isDirectory: &dir) && dir.boolValue
}
-
- package static func listing(_ path: String) -> [String] {
- (try? FileManager.default.contentsOfDirectory(atPath: path)) ?? []
- }
}
diff --git a/frontend/Sources/NXModel/BuildStore+Builder.swift b/frontend/Sources/NXModel/BuildStore+Builder.swift
index c9879b59..18f285c6 100644
--- a/frontend/Sources/NXModel/BuildStore+Builder.swift
+++ b/frontend/Sources/NXModel/BuildStore+Builder.swift
@@ -5,12 +5,9 @@ import NXCore
// The rules (where nxb comes from, the variables) are ExperimentalBuilder in NXCore.
extension BuildStore {
- /// The launcher's data folder, where a downloaded nxbuild release goes (`home` is a temporary folder in tests).
- package var builderSupportFolder: String { home + "/Library/Application Support/NS Port Builder" }
-
/// The `nxb` the next build takes, nil when there is none (the switch is then only a note in the log).
package var experimentalBuilderBinary: String? {
- ExperimentalBuilder.resolve(customPath: experimentalBuilderPath, support: builderSupportFolder)
+ ExperimentalBuilder.resolve(customPath: experimentalBuilderPath, bundled: bundledBuilder)
}
/// The variables the build gets for the experimental builder, empty when it is off or no nxb is found. Says in the log which it is.
diff --git a/frontend/Sources/NXModel/BuildStore.swift b/frontend/Sources/NXModel/BuildStore.swift
index 33d079d9..3248de74 100644
--- a/frontend/Sources/NXModel/BuildStore.swift
+++ b/frontend/Sources/NXModel/BuildStore.swift
@@ -133,10 +133,12 @@ func hop(_ body: @escaping @MainActor () -> Void) {
@Published package var experimentalBuilder: Bool {
didSet { defaults.set(experimentalBuilder, forKey: Self.experimentalBuilderKey) }
}
- /// The `nxb` binary the player chose (or its folder), empty for the downloaded release (ExperimentalBuilder.resolve).
+ /// The `nxb` binary the player chose (or its folder), empty for the app's own copy (ExperimentalBuilder.resolve).
@Published package var experimentalBuilderPath: String {
didSet { defaults.set(experimentalBuilderPath, forKey: Self.experimentalBuilderPathKey) }
}
+ /// The app's own nxb (Contents/Resources/nxbuild/nxb, frontend/nxbuild.pin), nil in tests and in a build without a pinned release.
+ package var bundledBuilder: String? = ExperimentalBuilder.bundled(resources: Bundle.main.resourcePath)
@Published package var archivePath = "" // the romfs folder the pack wrote
/// Where this build's folder actually is: --work-dir when the checkout has it, else the script's
/// default under the input folder (issue #5 — an upstream checkout has no --work-dir).
diff --git a/frontend/Tests/NXCoreTests/ExperimentalBuilderTests.swift b/frontend/Tests/NXCoreTests/ExperimentalBuilderTests.swift
index cc49d554..70a08028 100644
--- a/frontend/Tests/NXCoreTests/ExperimentalBuilderTests.swift
+++ b/frontend/Tests/NXCoreTests/ExperimentalBuilderTests.swift
@@ -3,7 +3,7 @@ import XCTest
/// `ExperimentalBuilder` (#749): which nxb the build takes and what it passes. Paths are strings checked by stand-in closures, nothing is run.
final class ExperimentalBuilderTests: XCTestCase {
- private let support = "/Users/p/Library/Application Support/NS Port Builder"
+ private let app = "/Applications/NX2Apple.app/Contents/Resources/nxbuild/nxb"
func testAChosenFileOrFolderNamesTheNxbBinary() {
XCTAssertEqual(ExperimentalBuilder.binary(at: "/opt/nxbuild/nxb", isDirectory: { _ in false }), "/opt/nxbuild/nxb")
@@ -12,37 +12,22 @@ final class ExperimentalBuilderTests: XCTestCase {
XCTAssertEqual(ExperimentalBuilder.binary(at: "~/nxb", isDirectory: { _ in false }), NSHomeDirectory() + "/nxb")
}
- func testTheChosenPathWinsAndIsNeverSwappedForAnother() {
- let installed = support + "/nxbuild/0.1.0/nxb"
- let ok: (String) -> Bool = { [installed, "/opt/nxb"].contains($0) }
- let list: (String) -> [String] = { _ in ["0.1.0"] }
- XCTAssertEqual(ExperimentalBuilder.resolve(customPath: "/opt/nxb", support: support, isExecutable: ok, isDirectory: { _ in false }, list: list), "/opt/nxb")
- XCTAssertNil(ExperimentalBuilder.resolve(customPath: "/gone/nxb", support: support, isExecutable: ok, isDirectory: { _ in false }, list: list),
- "a chosen path that does not work is reported, not replaced by the downloaded release")
- XCTAssertEqual(ExperimentalBuilder.resolve(customPath: "", support: support, isExecutable: ok, isDirectory: { _ in false }, list: list), installed)
+ func testTheAppsCopyIsInResources() {
+ XCTAssertEqual(ExperimentalBuilder.bundled(resources: "/Applications/NX2Apple.app/Contents/Resources"), app)
+ XCTAssertNil(ExperimentalBuilder.bundled(resources: nil))
}
- func testTheNewestDownloadedReleaseIsTaken() {
- let folder = ExperimentalBuilder.installFolder(support: support)
- XCTAssertEqual(folder, support + "/nxbuild")
- let have = Set(["0.2.0", "0.10.0", "0.9.1"].map { folder + "/" + $0 + "/nxb" })
- let nxb = ExperimentalBuilder.resolve(customPath: "", support: support, isExecutable: { have.contains($0) }, isDirectory: { _ in false },
- list: { _ in ["0.2.0", "0.10.0", "0.9.1", "notes.txt"] })
- XCTAssertEqual(nxb, folder + "/0.10.0/nxb", "versions compare as numbers")
- XCTAssertNil(ExperimentalBuilder.resolve(customPath: "", support: support, isExecutable: { _ in false }, isDirectory: { _ in false }, list: { _ in [] }))
+ func testTheChosenPathWinsAndIsNeverSwappedForTheAppsCopy() {
+ let ok: (String) -> Bool = { [self.app, "/opt/nxb"].contains($0) }
+ XCTAssertEqual(ExperimentalBuilder.resolve(customPath: "/opt/nxb", bundled: app, isExecutable: ok, isDirectory: { _ in false }), "/opt/nxb")
+ XCTAssertNil(ExperimentalBuilder.resolve(customPath: "/gone/nxb", bundled: app, isExecutable: ok, isDirectory: { _ in false }),
+ "a chosen path that does not work is reported, not replaced")
+ XCTAssertEqual(ExperimentalBuilder.resolve(customPath: "", bundled: app, isExecutable: ok, isDirectory: { _ in false }), app)
+ XCTAssertNil(ExperimentalBuilder.resolve(customPath: "", bundled: app, isExecutable: { _ in false }, isDirectory: { _ in false }),
+ "an app built without a pinned release has no copy")
}
func testTheBuildGetsTheSwitchAndThePath() {
XCTAssertEqual(ExperimentalBuilder.environment(nxb: "/opt/nxb"), ["DM_EXPERIMENTAL_BUILDER": "1", "NXB_BIN": "/opt/nxb"])
}
-
- func testNoReleaseIsPinnedYetAndADownloadIsOnlyTakenWithItsDigest() {
- XCTAssertNil(ExperimentalBuilder.releaseURL, "no nxbuild release is published yet: nothing is downloaded")
- XCTAssertNil(ExperimentalBuilder.releaseSHA256)
- let digest = String(repeating: "ab", count: 32)
- XCTAssertFalse(ExperimentalBuilder.matchesRelease(sha256: digest), "without a pinned digest nothing matches")
- XCTAssertTrue(ExperimentalBuilder.matchesRelease(sha256: digest.uppercased(), expected: digest))
- XCTAssertFalse(ExperimentalBuilder.matchesRelease(sha256: String(repeating: "cd", count: 32), expected: digest))
- XCTAssertFalse(ExperimentalBuilder.matchesRelease(sha256: "ab", expected: "ab"), "a digest is 64 hex digits")
- }
}
diff --git a/frontend/Tests/NXModelTests/BuildStoreTests.swift b/frontend/Tests/NXModelTests/BuildStoreTests.swift
index b51f0304..01f5f694 100644
--- a/frontend/Tests/NXModelTests/BuildStoreTests.swift
+++ b/frontend/Tests/NXModelTests/BuildStoreTests.swift
@@ -162,7 +162,7 @@ import XCTest
func testTheBuildGetsTheChosenNxbOrANoteThatTheStandardBuilderRuns() throws {
let store = makeStore()
- store.home = temporary.path
+ store.bundledBuilder = nil
store.experimentalBuilder = true
store.experimentalBuilderPath = temporary.path + "/gone/nxb"
XCTAssertNil(store.experimentalBuilderBinary)
diff --git a/frontend/nxbuild.pin b/frontend/nxbuild.pin
new file mode 100644
index 00000000..42445360
--- /dev/null
+++ b/frontend/nxbuild.pin
@@ -0,0 +1,7 @@
+# The nxbuild release the app bundles as its experimental builder (Contents/Resources/nxbuild/nxb, LNPANDY/NX2Apple#749).
+# build-mac-app.yml and tools/try-local run `frontend/tools/nxbuild.py fetch`: download the zip, check its SHA-256 against the line
+# below, unpack nxb and sign it ad hoc. `frontend/tools/nxbuild.py bump <version>` writes a new version with its digest from the
+# release's SHA256SUMS. A sha256 of PLACEHOLDER (no release yet) bundles nothing, and the setting then needs a chosen nxb.
+version 0.1.0
+sha256 PLACEHOLDER
+url https://github.com/v0lumehi/nxbuild/releases/download/v{version}/nxb-v{version}-macos-arm64.zip
diff --git a/frontend/tools/nxbuild.py b/frontend/tools/nxbuild.py
new file mode 100755
index 00000000..00c4f839
--- /dev/null
+++ b/frontend/tools/nxbuild.py
@@ -0,0 +1,81 @@
+#!/usr/bin/env python3
+"""The nxbuild release the app bundles (frontend/nxbuild.pin, LNPANDY/NX2Apple#749).
+ nxbuild.py fetch <Resources dir> <work dir> download the pinned zip, check its SHA-256 before anything is unpacked, put nxb into <Resources>/nxbuild/
+ and sign it ad hoc (the app's codesign --deep does not reach code in Resources). A placeholder digest
+ bundles nothing and says so (exit 0): the launcher's setting then needs a chosen nxb.
+ nxbuild.py bump <version> pin a new release: its digest from the release's SHA256SUMS
+Nothing here runs nxb: a zip whose digest differs is deleted and the step fails."""
+import hashlib, os, re, subprocess, sys, urllib.request, zipfile
+from pathlib import Path
+
+PIN = Path(__file__).resolve().parents[1] / 'nxbuild.pin'
+
+
+def read(pin=PIN):
+ """{'version', 'sha256', 'url'} with the version put into the URL; 'sha256' is None for a placeholder"""
+ kv = dict(ln.split('\t', 1) for ln in pin.read_text().splitlines() if ln.strip() and not ln.startswith('#'))
+ if set(kv) != {'version', 'sha256', 'url'} or not re.fullmatch(r'\d+\.\d+\.\d+', kv['version']):
+ raise SystemExit(f'{pin}: needs version (x.y.z), sha256 and url lines')
+ sha = kv['sha256'].strip().lower()
+ return {'version': kv['version'], 'url': kv['url'].replace('{version}', kv['version']), 'sha256': sha if re.fullmatch(r'[0-9a-f]{64}', sha) else None}
+
+
+def digest(path):
+ h = hashlib.sha256()
+ with open(path, 'rb') as f:
+ for b in iter(lambda: f.read(1 << 20), b''): h.update(b)
+ return h.hexdigest()
+
+
+def unpack(zip_path, expected, dest):
+ """Check the zip against expected, then copy its nxb (wherever it sits in the zip) to dest/nxb. Returns that path"""
+ got = digest(zip_path)
+ if got != expected:
+ os.unlink(zip_path)
+ raise SystemExit(f'[RED] nxbuild zip has SHA-256 {got}, the pin says {expected}: not bundled')
+ with zipfile.ZipFile(zip_path) as z:
+ names = [n for n in z.namelist() if Path(n).name == 'nxb' and not n.endswith('/')]
+ if len(names) != 1: raise SystemExit(f'[RED] nxbuild zip holds {len(names)} files named nxb, expected one')
+ dest.mkdir(parents=True, exist_ok=True)
+ out = dest / 'nxb'
+ out.write_bytes(z.read(names[0]))
+ out.chmod(0o755)
+ return out
+
+
+def fetch(resources, work):
+ pin = read()
+ if not pin['sha256']:
+ print(f'nxbuild {pin["version"]}: no digest pinned yet (PLACEHOLDER in {PIN.name}), the app bundles no nxb')
+ return None
+ work.mkdir(parents=True, exist_ok=True)
+ z = work / Path(pin['url']).name
+ urllib.request.urlretrieve(pin['url'], z)
+ nxb = unpack(z, pin['sha256'], resources / 'nxbuild')
+ subprocess.run(['codesign', '--force', '--sign', '-', str(nxb)], check=True)
+ print(f'nxbuild {pin["version"]}: {nxb} ({pin["sha256"]})')
+ return nxb
+
+
+def sums_digest(text, name): # the digest of name in a SHA256SUMS text ("<hex> <name>" lines, a '*' before binary names)
+ for ln in text.splitlines():
+ m = re.fullmatch(r'([0-9a-fA-F]{64})\s+\*?(\S+)', ln.strip())
+ if m and Path(m[2]).name == name: return m[1].lower()
+ raise SystemExit(f'SHA256SUMS lists no {name}')
+
+
+def bump(version, pin=PIN):
+ old = pin.read_text()
+ url = read(pin)['url'].replace(read(pin)['version'], version)
+ with urllib.request.urlopen(url.rsplit('/', 1)[0] + '/SHA256SUMS') as r: sums = r.read().decode()
+ sha = sums_digest(sums, Path(url).name)
+ new = re.sub(r'^version\t.*$', f'version\t{version}', old, flags=re.M)
+ pin.write_text(re.sub(r'^sha256\t.*$', f'sha256\t{sha}', new, flags=re.M))
+ print(f'pinned nxbuild {version}: {sha}')
+
+
+if __name__ == '__main__':
+ a = sys.argv[1:]
+ if a[:1] == ['fetch'] and len(a) == 3: fetch(Path(a[1]), Path(a[2]))
+ elif a[:1] == ['bump'] and len(a) == 2: bump(a[1])
+ else: raise SystemExit(__doc__)
diff --git a/patches/0073-nsp-script-hand-the-extraction-and-the-shader-sp.patch b/patches/0073-nsp-script-hand-the-extraction-and-the-shader-sp.patch
index 8f0662de..fc402af8 100644
--- a/patches/0073-nsp-script-hand-the-extraction-and-the-shader-sp.patch
+++ b/patches/0073-nsp-script-hand-the-extraction-and-the-shader-sp.patch
@@ -22,20 +22,19 @@ Without the variable nothing changes.
Upstream-Status: forwarded seedds/nxnative#1325
Reason: the launcher's opt-in experimental builder (nxbuild) runs the extraction and the shader split
Issue: LNPANDY/NX2Apple#749
-
---
- .../tests/test_nsp_experimental_builder.py | 145 ++++++++++++++++++
+ .../tests/test_nsp_experimental_builder.py | 156 ++++++++++++++++++
docs/env_switches.tsv | 1 +
- scripts/build_ipa_from_nsp.py | 114 +++++++++++++-
- 3 files changed, 256 insertions(+), 4 deletions(-)
+ scripts/build_ipa_from_nsp.py | 116 ++++++++++++-
+ 3 files changed, 269 insertions(+), 4 deletions(-)
create mode 100644 dm/tools/tests/test_nsp_experimental_builder.py
diff --git a/dm/tools/tests/test_nsp_experimental_builder.py b/dm/tools/tests/test_nsp_experimental_builder.py
new file mode 100644
-index 0000000..859b72a
+index 0000000..9005508
--- /dev/null
+++ b/dm/tools/tests/test_nsp_experimental_builder.py
-@@ -0,0 +1,145 @@
+@@ -0,0 +1,156 @@
+"""NSP script, the experimental builder (scripts/build_ipa_from_nsp.py, DM_EXPERIMENTAL_BUILDER and NXB_BIN): which nxbuild binary a stage takes, and that a stage it does not
+support, fails or gets wrong runs the standard way. The nxbuild binaries are stand-in scripts in a temporary folder: no game files, keys or nxbuild needed."""
+import importlib.util
@@ -89,17 +88,28 @@ index 0000000..859b72a
+ def test_off_by_default_and_with_another_value(self):
+ for value in (None, '0', 'yes'):
+ env = {k: v for k, v in os.environ.items() if k != 'DM_EXPERIMENTAL_BUILDER'} | ({'DM_EXPERIMENTAL_BUILDER': value} if value else {})
-+ with self.subTest(value=value): self.assertIsNone(nsp.nxb_tool('nxb', env))
++ with self.subTest(value=value): self.assertIsNone(nsp.nxb_tool('extract', env))
+ self.assertEqual(self.said, [], 'off says nothing')
+
-+ def test_the_binaries_beside_nxb_or_in_the_folder(self):
-+ self.assertEqual(nsp.nxb_tool('nxb'), self.nxb)
-+ self.assertEqual(nsp.nxb_tool('nxb', dict(os.environ, NXB_BIN=str(self.bin))), self.nxb, 'NXB_BIN may name the folder')
-+ self.assertIsNone(nsp.nxb_tool('nxb-split'))
-+ self.assertIn('nxb-split is missing: the standard builder runs this stage', self.text())
-+ self.assertIsNone(nsp.nxb_tool('nxb', dict(os.environ, NXB_BIN=str(self.t / 'gone'))))
++ def test_a_stage_binary_beside_nxb_else_its_subcommand(self):
++ self.assertEqual(nsp.nxb_tool('split'), [self.nxb, 'split'], 'one binary: the stage is its subcommand')
++ split = tool(self.bin, 'nxb-split', 'sys.exit(0)')
++ self.assertEqual(nsp.nxb_tool('split'), [split], 'a stage binary of its own goes first')
++ self.assertEqual(nsp.nxb_tool('split', dict(os.environ, NXB_BIN=str(self.bin))), [split], 'NXB_BIN may name the folder')
++ self.assertEqual(self.said, [])
++ self.assertIsNone(nsp.nxb_tool('split', dict(os.environ, NXB_BIN=str(self.t / 'gone'))))
+ self.assertIn('NXB_BIN does not name the nxb binary', self.text())
-+ self.said.clear(); nsp.nxb_tool('nxb-split'); self.assertEqual(self.said, [], 'a note once per build')
++ empty = self.t / 'empty'; empty.mkdir()
++ self.assertIsNone(nsp.nxb_tool('split', dict(os.environ, NXB_BIN=str(empty))))
++ self.assertIn('the standard builder runs every stage', self.text())
++ self.said.clear(); nsp.nxb_tool('split', dict(os.environ, NXB_BIN=str(empty))); self.assertEqual(self.said, [], 'a note once per build')
++
++ def test_a_subcommand_nxb_does_not_have_falls_back(self): # the one-binary nxb of a release without the stage: its usage error, exit 2
++ tool(self.bin, 'nxb', "print('nxb: extract: not implemented yet', file=sys.stderr); sys.exit(2)")
++ tmp = self.t / 'v0.partial'
++ self.assertFalse(nsp.nxb_extract(self.t / 'k', pfs0(self.t / 'g.nsp', ['a.cnmt.nca']), None, tmp, self.logs, 'v0', APP, None))
++ self.assertIn('exit code 2', self.text())
++ self.assertIn('the standard builder runs it', self.text())
+
+ def extraction(self, body, marker='marker.txt'):
+ src = self.t / 'src'; src.mkdir(exist_ok=True); (src / 'marker.txt').write_text('x')
@@ -194,7 +204,7 @@ index 772479e..aceb18e 100644
DM_FIXED_TIME switch `=<seconds>`: the user clock always returns that time and the host time zone is pinned to UTC, for deterministic comparison
DM_FOCUS_KEEP switch `1` keeps the game in focus while its Mac window is up, whichever app is active (minimizing still counts); `scripts/regress.py` sets it, so another app taking the key window during a run does not pause a game that stops out of focus
diff --git a/scripts/build_ipa_from_nsp.py b/scripts/build_ipa_from_nsp.py
-index d7ebdb5..33ebbe9 100755
+index d7ebdb5..0b1768b 100755
--- a/scripts/build_ipa_from_nsp.py
+++ b/scripts/build_ipa_from_nsp.py
@@ -33,6 +33,7 @@
@@ -205,13 +215,13 @@ index d7ebdb5..33ebbe9 100755
import ast, collections, contextlib, ctypes, errno, getpass, pwd, hashlib, importlib.util, json, os, re, shlex, shutil, struct, subprocess, sys, tempfile, threading, time, types
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "dm/tools"))
-@@ -422,6 +423,102 @@ def free_gb(p):
+@@ -422,6 +423,104 @@ def free_gb(p):
return shutil.disk_usage(p).free / 2**30
+# ---- The experimental builder (nxbuild) ----
+# DM_EXPERIMENTAL_BUILDER=1 with NXB_BIN=<the nxb binary, or its folder>: the stages nxbuild reproduces byte for byte run through its binaries beside nxb (the extraction:
-+# nxb-extract; the shader split: nxb-split). This script stays the driver: the stage names, the events, the work folder and the reuse keys are the same. A stage nxbuild does
++# nxb-extract, or nxb extract; the shader split: nxb-split, or nxb split). This script stays the driver: the stage names, the events, the work folder and the reuse keys are the same. A stage nxbuild does
+# not support (exit code NXB_UNSUPPORTED, or any other failure), a case it is not given (XCI, NSZ, merged packages, DLC, captures, a provisional or derived profile) and an
+# output that fails the checks below run the standard way, with a note in the log. Without the variable nothing changes.
+NXB_UNSUPPORTED = 4 # nxbuild's exit code for "not supported yet": nothing it wrote is an output (nxbuild's docs/integration-requests.md)
@@ -222,15 +232,17 @@ index d7ebdb5..33ebbe9 100755
+ if msg not in _nxb_noted: _nxb_noted.add(msg); say(f'experimental builder: {msg}')
+
+
-+def nxb_tool(name, env=None):
-+ """The nxbuild binary name (nxb-extract, nxb-split) beside NXB_BIN when the experimental builder is on, else None"""
++def nxb_tool(sub, env=None):
++ """The command of nxbuild's stage sub (extract, split) when the experimental builder is on, else None: its own binary nxb-<sub> beside NXB_BIN when there is one
++ (nxbuild's first releases), else the nxb binary's subcommand ([nxb, sub], one binary for every stage)"""
+ env = os.environ if env is None else env
+ if env.get('DM_EXPERIMENTAL_BUILDER') != '1': return None
+ where = Path(os.path.expanduser(env.get('NXB_BIN', '')))
+ if not env.get('NXB_BIN') or not where.exists(): nxb_note(f'NXB_BIN does not name the nxb binary ({env.get("NXB_BIN", "")!r}): the standard builder runs every stage'); return None
-+ tool = (where if where.is_dir() else where.parent) / name
-+ if tool.is_file() and os.access(tool, os.X_OK): return tool
-+ nxb_note(f'{tool} is missing: the standard builder runs this stage'); return None
++ folder = where if where.is_dir() else where.parent
++ for cmd in ([folder / f'nxb-{sub}'], [folder / 'nxb', sub]):
++ if cmd[0].is_file() and os.access(cmd[0], os.X_OK): return cmd
++ nxb_note(f'no nxb in {folder}: the standard builder runs every stage'); return None
+
+
+def nxb_run(cmd, log, what):
@@ -253,7 +265,7 @@ index d7ebdb5..33ebbe9 100755
+def nxb_extract(keys, base, upd, tmp, logs, name, app, marker):
+ """The extraction of one version through nxb-extract into tmp (the layout nx_extract writes: ExeFS/, RomFS.img, control.nacp, icon.jpg). True when its output passed the checks;
+ False leaves no tmp, for nx_extract. Only plain NSPs of one title: an XCI, NSZ or merged package, another program or a damaged NSP go the standard way"""
-+ tool = nxb_tool('nxb-extract')
++ tool = nxb_tool('extract')
+ if not tool: return False
+ def plain(p):
+ try: return p.suffix.lower() == '.nsp' and is_package(p) and not is_merged(p) and not any(n.lower().endswith('.ncz') for n in package_names(p))
@@ -262,7 +274,7 @@ index d7ebdb5..33ebbe9 100755
+ nxb_note(f'extract {name}: an XCI, NSZ or merged package, --program or --allow-damaged is not supported yet, the standard builder runs it'); return False
+ log, scratch = logs / f'nxb_extract_{name}.log', tmp.with_name(tmp.name + '.nxb-base')
+ rmtree(tmp); rmtree(scratch)
-+ cmd = [tool, '--keys', keys, '--base', base] + (['--out', scratch, '--update', upd, tmp, '--updates-only'] if upd and upd != base else ['--out', tmp])
++ cmd = [*tool, '--keys', keys, '--base', base] + (['--out', scratch, '--update', upd, tmp, '--updates-only'] if upd and upd != base else ['--out', tmp])
+ ok = nxb_run(cmd, log, f'extract {name}')
+ rmtree(scratch)
+ if ok:
@@ -283,13 +295,13 @@ index d7ebdb5..33ebbe9 100755
+ """The shader split through nxb-split into tmp/pkg and tmp/tables (shader_split.py's outputs). Returns the text the caller reads from shader_split.py's log (its 'shaders: N' line),
+ or None for shader_split.py, with tmp emptied again. Taken only when the tables it wrote have the data rows of the profile's (pso_pairs.tsv too where the profile has one): a reader
+ nxbuild lacks shows there. The caller passes only builds with a registered profile and no DLC, captures, NROs or decoder"""
-+ tool = nxb_tool('nxb-split')
++ tool = nxb_tool('split')
+ if not tool: return None
+ dirs = [Path(m).parent.parent for m in mains]
+ if any(Path(r) != d / 'RomFS.img' or Path(m) != d / 'ExeFS' / 'main' for r, m, d in zip(roms, mains, dirs)) or not (g.dir / 'game.tsv').is_file():
+ nxb_note('shader split: the inputs are not RomFS images of extractions, the standard builder runs it'); return None
+ log, t = logs / 'nxb_split.log', tmp / 'tables'
-+ ok = nxb_run([tool, '--game', g.dir / 'game.tsv', '--pkg', tmp / 'pkg', '--map', t / names[0], '--facts', t / names[1], '--pairs', t / names[2], '--pso-pairs', t / 'pso_pairs.tsv', *dirs],
++ ok = nxb_run([*tool, '--game', g.dir / 'game.tsv', '--pkg', tmp / 'pkg', '--map', t / names[0], '--facts', t / names[1], '--pairs', t / names[2], '--pso-pairs', t / 'pso_pairs.tsv', *dirs],
+ log, 'shader split')
+ if ok:
+ bad = [ln for ln in log.read_text(errors='replace').splitlines() if ln.startswith('[RED]')]
@@ -308,7 +320,7 @@ index d7ebdb5..33ebbe9 100755
def configure_dotnet(env):
"""Let standalone .NET apphosts find the runtime used by dotnet (Homebrew installs it outside the registered locations)."""
if env.get('DOTNET_ROOT'): return
-@@ -1183,7 +1280,8 @@ def extract(nx, keys, ex, name, base, upd, cmp_dir, logs, g, app): # the build
+@@ -1183,7 +1282,8 @@ def extract(nx, keys, ex, name, base, upd, cmp_dir, logs, g, app): # the build
def poll():
n, b = tree_size(tmp); return {'done': b, 'note': f'{n:,} files'}
PROG.stage(f'extract {name}', poll=poll, unit='B', total=last[1] if last else base.stat().st_size + (upd.stat().st_size if upd and upd != base else 0), est=not last)
@@ -318,7 +330,7 @@ index d7ebdb5..33ebbe9 100755
swap_in(tmp, dst)
n, b = extracted(log) or (0, 0)
PROG.end(f'{n:,} files, {amount(b, "B")}')
-@@ -2107,9 +2205,17 @@ def split_shaders(py, repo, roms, mains, pkg, env, logs, g, provisional=False, o
+@@ -2107,9 +2207,17 @@ def split_shaders(py, repo, roms, mains, pkg, env, logs, g, provisional=False, o
tmp = pkg.with_name(pkg.name + '.partial'); rmtree(tmp); (tmp / 'tables').mkdir(parents=True)
def counted(ln): # shader_split.py's 'progress: <done>/<total>'
if m := re.fullmatch(r'progress: (\d+)/(\d+)', ln): PROG.set(done=int(m[1]), total=int(m[2]), unit='files')
diff --git a/tools/tests/test_nxbuild_pin.py b/tools/tests/test_nxbuild_pin.py
new file mode 100644
index 00000000..9dcd4c3c
--- /dev/null
+++ b/tools/tests/test_nxbuild_pin.py
@@ -0,0 +1,53 @@
+"""frontend/tools/nxbuild.py and frontend/nxbuild.pin (LNPANDY/NX2Apple#749): the pin's form, the digest check before anything is unpacked, and a
+release's SHA256SUMS. No network: zips are made in a temporary folder."""
+import hashlib, importlib.util, tempfile, unittest, zipfile
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+spec = importlib.util.spec_from_file_location('nxbuild', ROOT / 'frontend/tools/nxbuild.py')
+nxbuild = importlib.util.module_from_spec(spec); spec.loader.exec_module(nxbuild)
+
+
+class Pin(unittest.TestCase):
+ def test_the_repository_pin_reads(self):
+ pin = nxbuild.read()
+ self.assertRegex(pin['version'], r'^\d+\.\d+\.\d+$')
+ self.assertTrue(pin['url'].startswith('https://github.com/v0lumehi/nxbuild/releases/download/v' + pin['version'] + '/'))
+ self.assertTrue(pin['url'].endswith('-macos-arm64.zip'))
+ self.assertTrue(pin['sha256'] is None or len(pin['sha256']) == 64)
+
+ def test_a_placeholder_bundles_nothing(self):
+ with tempfile.TemporaryDirectory() as t:
+ p = Path(t, 'pin'); p.write_text('# c\nversion\t0.1.0\nsha256\tPLACEHOLDER\nurl\thttps://x/v{version}/nxb-v{version}.zip\n')
+ self.assertEqual(nxbuild.read(p), {'version': '0.1.0', 'sha256': None, 'url': 'https://x/v0.1.0/nxb-v0.1.0.zip'})
+ p.write_text('version\t0.1\nsha256\tx\nurl\tu\n')
+ with self.assertRaises(SystemExit): nxbuild.read(p)
+
+ def zip(self, t, files):
+ z = Path(t, 'nxb.zip')
+ with zipfile.ZipFile(z, 'w') as f:
+ for n, b in files.items(): f.writestr(n, b)
+ return z
+
+ def test_the_digest_is_checked_before_unpacking(self):
+ with tempfile.TemporaryDirectory() as t:
+ z = self.zip(t, {'nxb-v0.1.0/nxb': b'binary', 'nxb-v0.1.0/README': b'r'})
+ good = hashlib.sha256(z.read_bytes()).hexdigest()
+ out = nxbuild.unpack(z, good, Path(t, 'res/nxbuild'))
+ self.assertEqual(out.read_bytes(), b'binary')
+ self.assertTrue(out.stat().st_mode & 0o111)
+ z = self.zip(t, {'nxb': b'other'})
+ with self.assertRaises(SystemExit) as e: nxbuild.unpack(z, good, Path(t, 'res2'))
+ self.assertIn('not bundled', str(e.exception))
+ self.assertFalse(z.exists(), 'a zip whose digest differs is deleted')
+ self.assertFalse(Path(t, 'res2').exists(), 'nothing is unpacked')
+
+ def test_sha256sums(self):
+ text = f'{"a" * 64} nxb-v0.1.0-macos-arm64.zip\n{"B" * 64} *other.zip\n'
+ self.assertEqual(nxbuild.sums_digest(text, 'nxb-v0.1.0-macos-arm64.zip'), 'a' * 64)
+ self.assertEqual(nxbuild.sums_digest(text, 'other.zip'), 'b' * 64)
+ with self.assertRaises(SystemExit): nxbuild.sums_digest(text, 'missing.zip')
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/tools/try-local b/tools/try-local
index 48624458..3fa3f703 100755
--- a/tools/try-local
+++ b/tools/try-local
@@ -136,6 +136,7 @@ cp frontend/.build/release/NX2Apple "$app/Contents/MacOS/NSPortBuilder"
if [ "$have_icon" = 1 ]; then cp "$icon_cache/Assets.car" "$icon_cache/AppIcon.icns" "$app/Contents/Resources/"; fi
ditto "$tools_cache/res/tools" "$app/Contents/Resources/tools"
ditto "$tools_cache/res/python" "$app/Contents/Resources/python"
+python3 frontend/tools/nxbuild.py fetch "$app/Contents/Resources" "$work/nxbuild" # the pinned experimental builder (#749), nothing without a digest
ditto "$work/engine-tree" "$app/Contents/Resources/engine"
frontend/tools/sign_engine_tree.sh "$app/Contents/Resources/engine" "$work/engine-tree"
rsync -a --exclude .build --exclude .swiftpm frontend-ios/ "$app/Contents/Resources/frontend-ios/"