Play your own Switch games on iPhone/iPad — one-click macOS app, no terminal. | 把 Switch 游戏装进 iPhone:图形界面一键构建,不用命令行。Bring your own game.
Swift apple-siliconcustom-formathomebrewiosipaipadipadosiphonemetalnative-portnintendo-switchnxnativeromfsswitch-games
Experimental builder: bundle the pinned nxbuild release with the app (#749)
v0lumehib37a935ee529
diff --git a/.github/scripts/build_wanted.py b/.github/scripts/build_wanted.pyindex 5a037315..8cf7fb86 100644--- a/.github/scripts/build_wanted.py+++ b/.github/scripts/build_wanted.py@@ -35,6 +35,7 @@ PACKAGING = ('frontend/Package.swift','frontend/Package.resolved','frontend/engine-signing.pub',+ 'frontend/nxbuild.pin','frontend/Resources/','frontend/AppIcon.icon/','frontend/bundled-python/',diff --git a/.github/workflows/build-mac-app.yml b/.github/workflows/build-mac-app.ymlindex 7f372155..da049995 100644--- a/.github/workflows/build-mac-app.yml+++ b/.github/workflows/build-mac-app.yml@@ -341,6 +341,9 @@ jobs:# The build tools and the Python go into Resources before the bundle is signed (bundle_tools.sh signs each nested Mach-O file# itself: --deep below does not reach code that sits in Resources).tools/bundle_tools.sh "$APP/Contents/Resources" "$RUNNER_TEMP/bundle-tools-work"+ # The experimental builder (#749): the nxbuild release nxbuild.pin names, its SHA-256 checked before it is unpacked, nxb signed ad hoc into+ # Resources/nxbuild. A placeholder digest (no release yet) bundles nothing: the launcher's setting then needs a chosen nxb.+ python3 tools/nxbuild.py fetch "$APP/Contents/Resources" "$RUNNER_TEMP/nxbuild-work"# The engine tree (Contents/ is read-only once signed: the launcher copies it to ~/Library/Application Support/NX2Apple/ on first# use). --deep does not reach code in Resources, but its Mach-O files (the shader translator, spirv-cross) are kept as upstream built# them, already signed: re-signing rewrote spirv-cross, which dmsh_chain.lock pins by its bytes, and stopped every shader build fromdiff --git a/frontend/README.md b/frontend/README.mdindex 1e2d7536..928a0eb1 100644--- a/frontend/README.md+++ b/frontend/README.md@@ -279,14 +279,15 @@ The build log says which stage nxbuild ran and why a stage fell back. nxbuild'sWhere `nxb` comes from (`ExperimentalBuilder` in `NXCore`):1. The path chosen in the row under the switch, the `nxb` binary or the folder that holds it. A chosen path that does not work is reported, never replaced by another binary.-2. Otherwise a downloaded release in `~/Library/Application Support/NS Port Builder/nxbuild/<version>/`. No nxbuild release is published yet, so `ExperimentalBuilder.releaseURL` and `releaseSHA256` are nil and nothing is downloaded. Once a release exists (a macOS arm64 zip of the binaries, signed ad hoc), both are pinned together in that file. The download goes to a temporary file, its SHA-256 is checked against `releaseSHA256` before anything is unpacked, and a zip whose digest differs is deleted and never run.+2. Otherwise the app's own copy, `Contents/Resources/nxbuild/nxb`. `frontend/nxbuild.pin` names the nxbuild release (version, SHA-256, URL pattern). `build-mac-app.yml` and `tools/try-local` run `frontend/tools/nxbuild.py fetch`: it downloads the zip, checks its SHA-256 against the pin before anything is unpacked, puts `nxb` into Resources and signs it ad hoc. A zip whose digest differs fails the app build. A pin with `PLACEHOLDER` as its digest bundles nothing. `frontend/tools/nxbuild.py bump <version>` pins a new release with the digest from its `SHA256SUMS`. Players download nothing at run time.With neither, the switch only writes a note in the build log and the standard builder runs.-To try it with a local nxbuild:+To try a local nxbuild:```bashcd <nxbuild checkout> && cargo build --release # the binaries land in its target folder, release/+codesign --force --sign - <target>/release/nxb* # ad hoc, as the app's copy```Then turn the switch on and choose `release/nxb` (or the folder). From a terminal, the same works without the launcher:@@ -295,6 +296,8 @@ Then turn the switch on and choose `release/nxb` (or the folder). From a terminaDM_EXPERIMENTAL_BUILDER=1 NXB_BIN=<target>/release/nxb python3 engine/scripts/build_ipa_from_nsp.py <NSP folder> -k <prod.keys> --platform mac-app```+The script takes a stage binary beside `nxb` (`nxb-extract`, `nxb-split`) when there is one, else the subcommand of `nxb` (`nxb extract`, `nxb split`).+The Qt front end has no switch for it: nxbuild is built for macOS on Apple silicon only, and on a Mac the launcher above is the one to use.## Rundiff --git a/frontend/Sources/NXCore/ExperimentalBuilder.swift b/frontend/Sources/NXCore/ExperimentalBuilder.swiftindex 62091942..8678eb32 100644--- a/frontend/Sources/NXCore/ExperimentalBuilder.swift+++ b/frontend/Sources/NXCore/ExperimentalBuilder.swift@@ -2,26 +2,16 @@ import Foundation/// The experimental builder (LNPANDY/NX2Apple#749): nxbuild, a build tool in Rust that reproduces stages of the NSP script byte for byte and faster./// The script stays the driver. With `DM_EXPERIMENTAL_BUILDER=1` and `NXB_BIN` it hands the stages nxbuild supports (the extraction and the shader split today)-/// to the binaries beside `nxb`, and runs every other case the standard way. An engine without that patch ignores both variables.+/// to nxbuild, and runs every other case the standard way. An engine without that patch ignores both variables.////// Where `nxb` comes from:/// 1. the path the player chose in Settings (the `nxb` binary, or the folder that holds it), for a local build of nxbuild;-/// 2. a release unpacked into `installFolder` by a download on demand. That download needs a published release: `releaseURL` and `releaseSHA256` are-/// pinned here together once it exists (a macOS arm64 zip of the binaries, signed ad hoc). Until then both are nil and only the chosen path is used.-/// The download writes the zip to a temporary file, checks its SHA-256 against `releaseSHA256` before anything is unpacked, and unpacks it into-/// `installFolder/<version>`. A zip whose digest differs is deleted and never run.+/// 2. the app's own copy, `Contents/Resources/nxbuild/nxb`: the release `frontend/nxbuild.pin` names, downloaded when the app is built, its SHA-256+/// checked against the pin before it is unpacked (`frontend/tools/nxbuild.py fetch`). Players download nothing at run time.package enum ExperimentalBuilder {- /// The nxbuild release the launcher downloads on demand. nil: none is published yet.- package static let releaseURL: URL? = nil- /// The SHA-256 (lowercase hex) of the zip at `releaseURL`, checked before it is unpacked. nil with it.- package static let releaseSHA256: String? = nil-- /// The binaries the NSP script runs, all beside `nxb`. A missing stage binary makes that stage run the standard way.- package static let stageTools = ["nxb-extract", "nxb-split"]-- /// Where a downloaded release goes: `<support>/nxbuild`, beside the venv in the launcher's data folder.- package static func installFolder(support: String) -> String {- (support as NSString).appendingPathComponent("nxbuild")+ /// The app's copy of nxb under a Resources folder.+ package static func bundled(resources: String?) -> String? {+ resources.map { ($0 as NSString).appendingPathComponent("nxbuild/nxb") }}/// The `nxb` a chosen path names: the file itself, or `nxb` inside a folder. Empty for an empty path.@@ -31,16 +21,14 @@ package enum ExperimentalBuilder {return isDirectory(p) ? (p as NSString).appendingPathComponent("nxb") : p}- /// The `nxb` the build takes: the chosen path when it is an executable `nxb`, else the newest release in `installFolder`. nil when there is none.- package static func resolve(customPath: String, support: String, isExecutable: (String) -> Bool = { FileManager.default.isExecutableFile(atPath: $0) },- isDirectory: (String) -> Bool = directoryExists, list: (String) -> [String] = listing) -> String? {+ /// The `nxb` the build takes: the chosen path when it is an executable `nxb`, else the app's copy. nil when there is none.+ package static func resolve(customPath: String, bundled: String?, isExecutable: (String) -> Bool = { FileManager.default.isExecutableFile(atPath: $0) },+ isDirectory: (String) -> Bool = directoryExists) -> String? {let chosen = binary(at: customPath, isDirectory: isDirectory)if !chosen.isEmpty {- return isExecutable(chosen) ? chosen : nil // a chosen path that does not work is not silently replaced by another binary+ return isExecutable(chosen) ? chosen : nil // a chosen path that does not work is reported, not silently replaced by the app's copy}- let folder = installFolder(support: support)- let versions = list(folder).sorted { $0.compare($1, options: .numeric) == .orderedDescending }- return versions.lazy.map { ((folder as NSString).appendingPathComponent($0) as NSString).appendingPathComponent("nxb") }.first(where: isExecutable)+ return bundled.flatMap { isExecutable($0) ? $0 : nil }}/// The variables the build gets: the switch and the path of `nxb`.@@ -48,18 +36,8 @@ package enum ExperimentalBuilder {["DM_EXPERIMENTAL_BUILDER": "1", "NXB_BIN": nxb]}- /// Whether a downloaded zip is the pinned release: its SHA-256 (lowercase hex, as `shasum -a 256` prints it) against `releaseSHA256`.- package static func matchesRelease(sha256 digest: String, expected: String? = releaseSHA256) -> Bool {- guard let expected, expected.count == 64 else { return false }- return digest.lowercased() == expected.lowercased()- }-package static func directoryExists(_ path: String) -> Bool {var dir: ObjCBool = falsereturn FileManager.default.fileExists(atPath: path, isDirectory: &dir) && dir.boolValue}-- package static func listing(_ path: String) -> [String] {- (try? FileManager.default.contentsOfDirectory(atPath: path)) ?? []- }}diff --git a/frontend/Sources/NXModel/BuildStore+Builder.swift b/frontend/Sources/NXModel/BuildStore+Builder.swiftindex c9879b59..18f285c6 100644--- a/frontend/Sources/NXModel/BuildStore+Builder.swift+++ b/frontend/Sources/NXModel/BuildStore+Builder.swift@@ -5,12 +5,9 @@ import NXCore// The rules (where nxb comes from, the variables) are ExperimentalBuilder in NXCore.extension BuildStore {- /// The launcher's data folder, where a downloaded nxbuild release goes (`home` is a temporary folder in tests).- package var builderSupportFolder: String { home + "/Library/Application Support/NS Port Builder" }-/// The `nxb` the next build takes, nil when there is none (the switch is then only a note in the log).package var experimentalBuilderBinary: String? {- ExperimentalBuilder.resolve(customPath: experimentalBuilderPath, support: builderSupportFolder)+ ExperimentalBuilder.resolve(customPath: experimentalBuilderPath, bundled: bundledBuilder)}/// The variables the build gets for the experimental builder, empty when it is off or no nxb is found. Says in the log which it is.diff --git a/frontend/Sources/NXModel/BuildStore.swift b/frontend/Sources/NXModel/BuildStore.swiftindex 33d079d9..3248de74 100644--- a/frontend/Sources/NXModel/BuildStore.swift+++ b/frontend/Sources/NXModel/BuildStore.swift@@ -133,10 +133,12 @@ func hop(_ body: @escaping @MainActor () -> Void) {@Published package var experimentalBuilder: Bool {didSet { defaults.set(experimentalBuilder, forKey: Self.experimentalBuilderKey) }}- /// The `nxb` binary the player chose (or its folder), empty for the downloaded release (ExperimentalBuilder.resolve).+ /// The `nxb` binary the player chose (or its folder), empty for the app's own copy (ExperimentalBuilder.resolve).@Published package var experimentalBuilderPath: String {didSet { defaults.set(experimentalBuilderPath, forKey: Self.experimentalBuilderPathKey) }}+ /// The app's own nxb (Contents/Resources/nxbuild/nxb, frontend/nxbuild.pin), nil in tests and in a build without a pinned release.+ package var bundledBuilder: String? = ExperimentalBuilder.bundled(resources: Bundle.main.resourcePath)@Published package var archivePath = "" // the romfs folder the pack wrote/// Where this build's folder actually is: --work-dir when the checkout has it, else the script's/// default under the input folder (issue #5 — an upstream checkout has no --work-dir).diff --git a/frontend/Tests/NXCoreTests/ExperimentalBuilderTests.swift b/frontend/Tests/NXCoreTests/ExperimentalBuilderTests.swiftindex cc49d554..70a08028 100644--- a/frontend/Tests/NXCoreTests/ExperimentalBuilderTests.swift+++ b/frontend/Tests/NXCoreTests/ExperimentalBuilderTests.swift@@ -3,7 +3,7 @@ import XCTest/// `ExperimentalBuilder` (#749): which nxb the build takes and what it passes. Paths are strings checked by stand-in closures, nothing is run.final class ExperimentalBuilderTests: XCTestCase {- private let support = "/Users/p/Library/Application Support/NS Port Builder"+ private let app = "/Applications/NX2Apple.app/Contents/Resources/nxbuild/nxb"func testAChosenFileOrFolderNamesTheNxbBinary() {XCTAssertEqual(ExperimentalBuilder.binary(at: "/opt/nxbuild/nxb", isDirectory: { _ in false }), "/opt/nxbuild/nxb")@@ -12,37 +12,22 @@ final class ExperimentalBuilderTests: XCTestCase {XCTAssertEqual(ExperimentalBuilder.binary(at: "~/nxb", isDirectory: { _ in false }), NSHomeDirectory() + "/nxb")}- func testTheChosenPathWinsAndIsNeverSwappedForAnother() {- let installed = support + "/nxbuild/0.1.0/nxb"- let ok: (String) -> Bool = { [installed, "/opt/nxb"].contains($0) }- let list: (String) -> [String] = { _ in ["0.1.0"] }- XCTAssertEqual(ExperimentalBuilder.resolve(customPath: "/opt/nxb", support: support, isExecutable: ok, isDirectory: { _ in false }, list: list), "/opt/nxb")- XCTAssertNil(ExperimentalBuilder.resolve(customPath: "/gone/nxb", support: support, isExecutable: ok, isDirectory: { _ in false }, list: list),- "a chosen path that does not work is reported, not replaced by the downloaded release")- XCTAssertEqual(ExperimentalBuilder.resolve(customPath: "", support: support, isExecutable: ok, isDirectory: { _ in false }, list: list), installed)+ func testTheAppsCopyIsInResources() {+ XCTAssertEqual(ExperimentalBuilder.bundled(resources: "/Applications/NX2Apple.app/Contents/Resources"), app)+ XCTAssertNil(ExperimentalBuilder.bundled(resources: nil))}- func testTheNewestDownloadedReleaseIsTaken() {- let folder = ExperimentalBuilder.installFolder(support: support)- XCTAssertEqual(folder, support + "/nxbuild")- let have = Set(["0.2.0", "0.10.0", "0.9.1"].map { folder + "/" + $0 + "/nxb" })- let nxb = ExperimentalBuilder.resolve(customPath: "", support: support, isExecutable: { have.contains($0) }, isDirectory: { _ in false },- list: { _ in ["0.2.0", "0.10.0", "0.9.1", "notes.txt"] })- XCTAssertEqual(nxb, folder + "/0.10.0/nxb", "versions compare as numbers")- XCTAssertNil(ExperimentalBuilder.resolve(customPath: "", support: support, isExecutable: { _ in false }, isDirectory: { _ in false }, list: { _ in [] }))+ func testTheChosenPathWinsAndIsNeverSwappedForTheAppsCopy() {+ let ok: (String) -> Bool = { [self.app, "/opt/nxb"].contains($0) }+ XCTAssertEqual(ExperimentalBuilder.resolve(customPath: "/opt/nxb", bundled: app, isExecutable: ok, isDirectory: { _ in false }), "/opt/nxb")+ XCTAssertNil(ExperimentalBuilder.resolve(customPath: "/gone/nxb", bundled: app, isExecutable: ok, isDirectory: { _ in false }),+ "a chosen path that does not work is reported, not replaced")+ XCTAssertEqual(ExperimentalBuilder.resolve(customPath: "", bundled: app, isExecutable: ok, isDirectory: { _ in false }), app)+ XCTAssertNil(ExperimentalBuilder.resolve(customPath: "", bundled: app, isExecutable: { _ in false }, isDirectory: { _ in false }),+ "an app built without a pinned release has no copy")}func testTheBuildGetsTheSwitchAndThePath() {XCTAssertEqual(ExperimentalBuilder.environment(nxb: "/opt/nxb"), ["DM_EXPERIMENTAL_BUILDER": "1", "NXB_BIN": "/opt/nxb"])}-- func testNoReleaseIsPinnedYetAndADownloadIsOnlyTakenWithItsDigest() {- XCTAssertNil(ExperimentalBuilder.releaseURL, "no nxbuild release is published yet: nothing is downloaded")- XCTAssertNil(ExperimentalBuilder.releaseSHA256)- let digest = String(repeating: "ab", count: 32)- XCTAssertFalse(ExperimentalBuilder.matchesRelease(sha256: digest), "without a pinned digest nothing matches")- XCTAssertTrue(ExperimentalBuilder.matchesRelease(sha256: digest.uppercased(), expected: digest))- XCTAssertFalse(ExperimentalBuilder.matchesRelease(sha256: String(repeating: "cd", count: 32), expected: digest))- XCTAssertFalse(ExperimentalBuilder.matchesRelease(sha256: "ab", expected: "ab"), "a digest is 64 hex digits")- }}diff --git a/frontend/Tests/NXModelTests/BuildStoreTests.swift b/frontend/Tests/NXModelTests/BuildStoreTests.swiftindex b51f0304..01f5f694 100644--- a/frontend/Tests/NXModelTests/BuildStoreTests.swift+++ b/frontend/Tests/NXModelTests/BuildStoreTests.swift@@ -162,7 +162,7 @@ import XCTestfunc testTheBuildGetsTheChosenNxbOrANoteThatTheStandardBuilderRuns() throws {let store = makeStore()- store.home = temporary.path+ store.bundledBuilder = nilstore.experimentalBuilder = truestore.experimentalBuilderPath = temporary.path + "/gone/nxb"XCTAssertNil(store.experimentalBuilderBinary)diff --git a/frontend/nxbuild.pin b/frontend/nxbuild.pinnew file mode 100644index 00000000..42445360--- /dev/null+++ b/frontend/nxbuild.pin@@ -0,0 +1,7 @@+# The nxbuild release the app bundles as its experimental builder (Contents/Resources/nxbuild/nxb, LNPANDY/NX2Apple#749).+# build-mac-app.yml and tools/try-local run `frontend/tools/nxbuild.py fetch`: download the zip, check its SHA-256 against the line+# below, unpack nxb and sign it ad hoc. `frontend/tools/nxbuild.py bump <version>` writes a new version with its digest from the+# release's SHA256SUMS. A sha256 of PLACEHOLDER (no release yet) bundles nothing, and the setting then needs a chosen nxb.+version 0.1.0+sha256 PLACEHOLDER+url https://github.com/v0lumehi/nxbuild/releases/download/v{version}/nxb-v{version}-macos-arm64.zipdiff --git a/frontend/tools/nxbuild.py b/frontend/tools/nxbuild.pynew file mode 100755index 00000000..00c4f839--- /dev/null+++ b/frontend/tools/nxbuild.py@@ -0,0 +1,81 @@+#!/usr/bin/env python3+"""The nxbuild release the app bundles (frontend/nxbuild.pin, LNPANDY/NX2Apple#749).+ nxbuild.py fetch <Resources dir> <work dir> download the pinned zip, check its SHA-256 before anything is unpacked, put nxb into <Resources>/nxbuild/+ and sign it ad hoc (the app's codesign --deep does not reach code in Resources). A placeholder digest+ bundles nothing and says so (exit 0): the launcher's setting then needs a chosen nxb.+ nxbuild.py bump <version> pin a new release: its digest from the release's SHA256SUMS+Nothing here runs nxb: a zip whose digest differs is deleted and the step fails."""+import hashlib, os, re, subprocess, sys, urllib.request, zipfile+from pathlib import Path++PIN = Path(__file__).resolve().parents[1] / 'nxbuild.pin'+++def read(pin=PIN):+ """{'version', 'sha256', 'url'} with the version put into the URL; 'sha256' is None for a placeholder"""+ kv = dict(ln.split('\t', 1) for ln in pin.read_text().splitlines() if ln.strip() and not ln.startswith('#'))+ if set(kv) != {'version', 'sha256', 'url'} or not re.fullmatch(r'\d+\.\d+\.\d+', kv['version']):+ raise SystemExit(f'{pin}: needs version (x.y.z), sha256 and url lines')+ sha = kv['sha256'].strip().lower()+ return {'version': kv['version'], 'url': kv['url'].replace('{version}', kv['version']), 'sha256': sha if re.fullmatch(r'[0-9a-f]{64}', sha) else None}+++def digest(path):+ h = hashlib.sha256()+ with open(path, 'rb') as f:+ for b in iter(lambda: f.read(1 << 20), b''): h.update(b)+ return h.hexdigest()+++def unpack(zip_path, expected, dest):+ """Check the zip against expected, then copy its nxb (wherever it sits in the zip) to dest/nxb. Returns that path"""+ got = digest(zip_path)+ if got != expected:+ os.unlink(zip_path)+ raise SystemExit(f'[RED] nxbuild zip has SHA-256 {got}, the pin says {expected}: not bundled')+ with zipfile.ZipFile(zip_path) as z:+ names = [n for n in z.namelist() if Path(n).name == 'nxb' and not n.endswith('/')]+ if len(names) != 1: raise SystemExit(f'[RED] nxbuild zip holds {len(names)} files named nxb, expected one')+ dest.mkdir(parents=True, exist_ok=True)+ out = dest / 'nxb'+ out.write_bytes(z.read(names[0]))+ out.chmod(0o755)+ return out+++def fetch(resources, work):+ pin = read()+ if not pin['sha256']:+ print(f'nxbuild {pin["version"]}: no digest pinned yet (PLACEHOLDER in {PIN.name}), the app bundles no nxb')+ return None+ work.mkdir(parents=True, exist_ok=True)+ z = work / Path(pin['url']).name+ urllib.request.urlretrieve(pin['url'], z)+ nxb = unpack(z, pin['sha256'], resources / 'nxbuild')+ subprocess.run(['codesign', '--force', '--sign', '-', str(nxb)], check=True)+ print(f'nxbuild {pin["version"]}: {nxb} ({pin["sha256"]})')+ return nxb+++def sums_digest(text, name): # the digest of name in a SHA256SUMS text ("<hex> <name>" lines, a '*' before binary names)+ for ln in text.splitlines():+ m = re.fullmatch(r'([0-9a-fA-F]{64})\s+\*?(\S+)', ln.strip())+ if m and Path(m[2]).name == name: return m[1].lower()+ raise SystemExit(f'SHA256SUMS lists no {name}')+++def bump(version, pin=PIN):+ old = pin.read_text()+ url = read(pin)['url'].replace(read(pin)['version'], version)+ with urllib.request.urlopen(url.rsplit('/', 1)[0] + '/SHA256SUMS') as r: sums = r.read().decode()+ sha = sums_digest(sums, Path(url).name)+ new = re.sub(r'^version\t.*$', f'version\t{version}', old, flags=re.M)+ pin.write_text(re.sub(r'^sha256\t.*$', f'sha256\t{sha}', new, flags=re.M))+ print(f'pinned nxbuild {version}: {sha}')+++if __name__ == '__main__':+ a = sys.argv[1:]+ if a[:1] == ['fetch'] and len(a) == 3: fetch(Path(a[1]), Path(a[2]))+ elif a[:1] == ['bump'] and len(a) == 2: bump(a[1])+ else: raise SystemExit(__doc__)diff --git a/patches/0073-nsp-script-hand-the-extraction-and-the-shader-sp.patch b/patches/0073-nsp-script-hand-the-extraction-and-the-shader-sp.patchindex 8f0662de..fc402af8 100644--- a/patches/0073-nsp-script-hand-the-extraction-and-the-shader-sp.patch+++ b/patches/0073-nsp-script-hand-the-extraction-and-the-shader-sp.patch@@ -22,20 +22,19 @@ Without the variable nothing changes.Upstream-Status: forwarded seedds/nxnative#1325Reason: the launcher's opt-in experimental builder (nxbuild) runs the extraction and the shader splitIssue: LNPANDY/NX2Apple#749----- .../tests/test_nsp_experimental_builder.py | 145 +++++++++++++++++++ .../tests/test_nsp_experimental_builder.py | 156 ++++++++++++++++++docs/env_switches.tsv | 1 +- scripts/build_ipa_from_nsp.py | 114 +++++++++++++-- 3 files changed, 256 insertions(+), 4 deletions(-)+ scripts/build_ipa_from_nsp.py | 116 ++++++++++++-+ 3 files changed, 269 insertions(+), 4 deletions(-)create mode 100644 dm/tools/tests/test_nsp_experimental_builder.pydiff --git a/dm/tools/tests/test_nsp_experimental_builder.py b/dm/tools/tests/test_nsp_experimental_builder.pynew file mode 100644-index 0000000..859b72a+index 0000000..9005508--- /dev/null+++ b/dm/tools/tests/test_nsp_experimental_builder.py-@@ -0,0 +1,145 @@+@@ -0,0 +1,156 @@+"""NSP script, the experimental builder (scripts/build_ipa_from_nsp.py, DM_EXPERIMENTAL_BUILDER and NXB_BIN): which nxbuild binary a stage takes, and that a stage it does not+support, fails or gets wrong runs the standard way. The nxbuild binaries are stand-in scripts in a temporary folder: no game files, keys or nxbuild needed."""+import importlib.util@@ -89,17 +88,28 @@ index 0000000..859b72a+ def test_off_by_default_and_with_another_value(self):+ for value in (None, '0', 'yes'):+ env = {k: v for k, v in os.environ.items() if k != 'DM_EXPERIMENTAL_BUILDER'} | ({'DM_EXPERIMENTAL_BUILDER': value} if value else {})-+ with self.subTest(value=value): self.assertIsNone(nsp.nxb_tool('nxb', env))++ with self.subTest(value=value): self.assertIsNone(nsp.nxb_tool('extract', env))+ self.assertEqual(self.said, [], 'off says nothing')+-+ def test_the_binaries_beside_nxb_or_in_the_folder(self):-+ self.assertEqual(nsp.nxb_tool('nxb'), self.nxb)-+ self.assertEqual(nsp.nxb_tool('nxb', dict(os.environ, NXB_BIN=str(self.bin))), self.nxb, 'NXB_BIN may name the folder')-+ self.assertIsNone(nsp.nxb_tool('nxb-split'))-+ self.assertIn('nxb-split is missing: the standard builder runs this stage', self.text())-+ self.assertIsNone(nsp.nxb_tool('nxb', dict(os.environ, NXB_BIN=str(self.t / 'gone'))))++ def test_a_stage_binary_beside_nxb_else_its_subcommand(self):++ self.assertEqual(nsp.nxb_tool('split'), [self.nxb, 'split'], 'one binary: the stage is its subcommand')++ split = tool(self.bin, 'nxb-split', 'sys.exit(0)')++ self.assertEqual(nsp.nxb_tool('split'), [split], 'a stage binary of its own goes first')++ self.assertEqual(nsp.nxb_tool('split', dict(os.environ, NXB_BIN=str(self.bin))), [split], 'NXB_BIN may name the folder')++ self.assertEqual(self.said, [])++ self.assertIsNone(nsp.nxb_tool('split', dict(os.environ, NXB_BIN=str(self.t / 'gone'))))+ self.assertIn('NXB_BIN does not name the nxb binary', self.text())-+ self.said.clear(); nsp.nxb_tool('nxb-split'); self.assertEqual(self.said, [], 'a note once per build')++ empty = self.t / 'empty'; empty.mkdir()++ self.assertIsNone(nsp.nxb_tool('split', dict(os.environ, NXB_BIN=str(empty))))++ self.assertIn('the standard builder runs every stage', self.text())++ self.said.clear(); nsp.nxb_tool('split', dict(os.environ, NXB_BIN=str(empty))); self.assertEqual(self.said, [], 'a note once per build')++++ def test_a_subcommand_nxb_does_not_have_falls_back(self): # the one-binary nxb of a release without the stage: its usage error, exit 2++ tool(self.bin, 'nxb', "print('nxb: extract: not implemented yet', file=sys.stderr); sys.exit(2)")++ tmp = self.t / 'v0.partial'++ self.assertFalse(nsp.nxb_extract(self.t / 'k', pfs0(self.t / 'g.nsp', ['a.cnmt.nca']), None, tmp, self.logs, 'v0', APP, None))++ self.assertIn('exit code 2', self.text())++ self.assertIn('the standard builder runs it', self.text())++ def extraction(self, body, marker='marker.txt'):+ src = self.t / 'src'; src.mkdir(exist_ok=True); (src / 'marker.txt').write_text('x')@@ -194,7 +204,7 @@ index 772479e..aceb18e 100644DM_FIXED_TIME switch `=<seconds>`: the user clock always returns that time and the host time zone is pinned to UTC, for deterministic comparisonDM_FOCUS_KEEP switch `1` keeps the game in focus while its Mac window is up, whichever app is active (minimizing still counts); `scripts/regress.py` sets it, so another app taking the key window during a run does not pause a game that stops out of focusdiff --git a/scripts/build_ipa_from_nsp.py b/scripts/build_ipa_from_nsp.py-index d7ebdb5..33ebbe9 100755+index d7ebdb5..0b1768b 100755--- a/scripts/build_ipa_from_nsp.py+++ b/scripts/build_ipa_from_nsp.py@@ -33,6 +33,7 @@@@ -205,13 +215,13 @@ index d7ebdb5..33ebbe9 100755import ast, collections, contextlib, ctypes, errno, getpass, pwd, hashlib, importlib.util, json, os, re, shlex, shutil, struct, subprocess, sys, tempfile, threading, time, typesfrom pathlib import Pathsys.path.insert(0, str(Path(__file__).resolve().parents[1] / "dm/tools"))-@@ -422,6 +423,102 @@ def free_gb(p):+@@ -422,6 +423,104 @@ def free_gb(p):return shutil.disk_usage(p).free / 2**30+# ---- The experimental builder (nxbuild) ----+# DM_EXPERIMENTAL_BUILDER=1 with NXB_BIN=<the nxb binary, or its folder>: the stages nxbuild reproduces byte for byte run through its binaries beside nxb (the extraction:-+# nxb-extract; the shader split: nxb-split). This script stays the driver: the stage names, the events, the work folder and the reuse keys are the same. A stage nxbuild does++# nxb-extract, or nxb extract; the shader split: nxb-split, or nxb split). This script stays the driver: the stage names, the events, the work folder and the reuse keys are the same. A stage nxbuild does+# not support (exit code NXB_UNSUPPORTED, or any other failure), a case it is not given (XCI, NSZ, merged packages, DLC, captures, a provisional or derived profile) and an+# output that fails the checks below run the standard way, with a note in the log. Without the variable nothing changes.+NXB_UNSUPPORTED = 4 # nxbuild's exit code for "not supported yet": nothing it wrote is an output (nxbuild's docs/integration-requests.md)@@ -222,15 +232,17 @@ index d7ebdb5..33ebbe9 100755+ if msg not in _nxb_noted: _nxb_noted.add(msg); say(f'experimental builder: {msg}')++-+def nxb_tool(name, env=None):-+ """The nxbuild binary name (nxb-extract, nxb-split) beside NXB_BIN when the experimental builder is on, else None"""++def nxb_tool(sub, env=None):++ """The command of nxbuild's stage sub (extract, split) when the experimental builder is on, else None: its own binary nxb-<sub> beside NXB_BIN when there is one++ (nxbuild's first releases), else the nxb binary's subcommand ([nxb, sub], one binary for every stage)"""+ env = os.environ if env is None else env+ if env.get('DM_EXPERIMENTAL_BUILDER') != '1': return None+ where = Path(os.path.expanduser(env.get('NXB_BIN', '')))+ if not env.get('NXB_BIN') or not where.exists(): nxb_note(f'NXB_BIN does not name the nxb binary ({env.get("NXB_BIN", "")!r}): the standard builder runs every stage'); return None-+ tool = (where if where.is_dir() else where.parent) / name-+ if tool.is_file() and os.access(tool, os.X_OK): return tool-+ nxb_note(f'{tool} is missing: the standard builder runs this stage'); return None++ folder = where if where.is_dir() else where.parent++ for cmd in ([folder / f'nxb-{sub}'], [folder / 'nxb', sub]):++ if cmd[0].is_file() and os.access(cmd[0], os.X_OK): return cmd++ nxb_note(f'no nxb in {folder}: the standard builder runs every stage'); return None+++def nxb_run(cmd, log, what):@@ -253,7 +265,7 @@ index d7ebdb5..33ebbe9 100755+def nxb_extract(keys, base, upd, tmp, logs, name, app, marker):+ """The extraction of one version through nxb-extract into tmp (the layout nx_extract writes: ExeFS/, RomFS.img, control.nacp, icon.jpg). True when its output passed the checks;+ False leaves no tmp, for nx_extract. Only plain NSPs of one title: an XCI, NSZ or merged package, another program or a damaged NSP go the standard way"""-+ tool = nxb_tool('nxb-extract')++ tool = nxb_tool('extract')+ if not tool: return False+ def plain(p):+ try: return p.suffix.lower() == '.nsp' and is_package(p) and not is_merged(p) and not any(n.lower().endswith('.ncz') for n in package_names(p))@@ -262,7 +274,7 @@ index d7ebdb5..33ebbe9 100755+ nxb_note(f'extract {name}: an XCI, NSZ or merged package, --program or --allow-damaged is not supported yet, the standard builder runs it'); return False+ log, scratch = logs / f'nxb_extract_{name}.log', tmp.with_name(tmp.name + '.nxb-base')+ rmtree(tmp); rmtree(scratch)-+ cmd = [tool, '--keys', keys, '--base', base] + (['--out', scratch, '--update', upd, tmp, '--updates-only'] if upd and upd != base else ['--out', tmp])++ cmd = [*tool, '--keys', keys, '--base', base] + (['--out', scratch, '--update', upd, tmp, '--updates-only'] if upd and upd != base else ['--out', tmp])+ ok = nxb_run(cmd, log, f'extract {name}')+ rmtree(scratch)+ if ok:@@ -283,13 +295,13 @@ index d7ebdb5..33ebbe9 100755+ """The shader split through nxb-split into tmp/pkg and tmp/tables (shader_split.py's outputs). Returns the text the caller reads from shader_split.py's log (its 'shaders: N' line),+ or None for shader_split.py, with tmp emptied again. Taken only when the tables it wrote have the data rows of the profile's (pso_pairs.tsv too where the profile has one): a reader+ nxbuild lacks shows there. The caller passes only builds with a registered profile and no DLC, captures, NROs or decoder"""-+ tool = nxb_tool('nxb-split')++ tool = nxb_tool('split')+ if not tool: return None+ dirs = [Path(m).parent.parent for m in mains]+ if any(Path(r) != d / 'RomFS.img' or Path(m) != d / 'ExeFS' / 'main' for r, m, d in zip(roms, mains, dirs)) or not (g.dir / 'game.tsv').is_file():+ nxb_note('shader split: the inputs are not RomFS images of extractions, the standard builder runs it'); return None+ log, t = logs / 'nxb_split.log', tmp / 'tables'-+ ok = nxb_run([tool, '--game', g.dir / 'game.tsv', '--pkg', tmp / 'pkg', '--map', t / names[0], '--facts', t / names[1], '--pairs', t / names[2], '--pso-pairs', t / 'pso_pairs.tsv', *dirs],++ ok = nxb_run([*tool, '--game', g.dir / 'game.tsv', '--pkg', tmp / 'pkg', '--map', t / names[0], '--facts', t / names[1], '--pairs', t / names[2], '--pso-pairs', t / 'pso_pairs.tsv', *dirs],+ log, 'shader split')+ if ok:+ bad = [ln for ln in log.read_text(errors='replace').splitlines() if ln.startswith('[RED]')]@@ -308,7 +320,7 @@ index d7ebdb5..33ebbe9 100755def configure_dotnet(env):"""Let standalone .NET apphosts find the runtime used by dotnet (Homebrew installs it outside the registered locations)."""if env.get('DOTNET_ROOT'): return-@@ -1183,7 +1280,8 @@ def extract(nx, keys, ex, name, base, upd, cmp_dir, logs, g, app): # the build+@@ -1183,7 +1282,8 @@ def extract(nx, keys, ex, name, base, upd, cmp_dir, logs, g, app): # the builddef poll():n, b = tree_size(tmp); return {'done': b, 'note': f'{n:,} files'}PROG.stage(f'extract {name}', poll=poll, unit='B', total=last[1] if last else base.stat().st_size + (upd.stat().st_size if upd and upd != base else 0), est=not last)@@ -318,7 +330,7 @@ index d7ebdb5..33ebbe9 100755swap_in(tmp, dst)n, b = extracted(log) or (0, 0)PROG.end(f'{n:,} files, {amount(b, "B")}')-@@ -2107,9 +2205,17 @@ def split_shaders(py, repo, roms, mains, pkg, env, logs, g, provisional=False, o+@@ -2107,9 +2207,17 @@ def split_shaders(py, repo, roms, mains, pkg, env, logs, g, provisional=False, otmp = pkg.with_name(pkg.name + '.partial'); rmtree(tmp); (tmp / 'tables').mkdir(parents=True)def counted(ln): # shader_split.py's 'progress: <done>/<total>'if m := re.fullmatch(r'progress: (\d+)/(\d+)', ln): PROG.set(done=int(m[1]), total=int(m[2]), unit='files')diff --git a/tools/tests/test_nxbuild_pin.py b/tools/tests/test_nxbuild_pin.pynew file mode 100644index 00000000..9dcd4c3c--- /dev/null+++ b/tools/tests/test_nxbuild_pin.py@@ -0,0 +1,53 @@+"""frontend/tools/nxbuild.py and frontend/nxbuild.pin (LNPANDY/NX2Apple#749): the pin's form, the digest check before anything is unpacked, and a+release's SHA256SUMS. No network: zips are made in a temporary folder."""+import hashlib, importlib.util, tempfile, unittest, zipfile+from pathlib import Path++ROOT = Path(__file__).resolve().parents[2]+spec = importlib.util.spec_from_file_location('nxbuild', ROOT / 'frontend/tools/nxbuild.py')+nxbuild = importlib.util.module_from_spec(spec); spec.loader.exec_module(nxbuild)+++class Pin(unittest.TestCase):+ def test_the_repository_pin_reads(self):+ pin = nxbuild.read()+ self.assertRegex(pin['version'], r'^\d+\.\d+\.\d+$')+ self.assertTrue(pin['url'].startswith('https://github.com/v0lumehi/nxbuild/releases/download/v' + pin['version'] + '/'))+ self.assertTrue(pin['url'].endswith('-macos-arm64.zip'))+ self.assertTrue(pin['sha256'] is None or len(pin['sha256']) == 64)++ def test_a_placeholder_bundles_nothing(self):+ with tempfile.TemporaryDirectory() as t:+ p = Path(t, 'pin'); p.write_text('# c\nversion\t0.1.0\nsha256\tPLACEHOLDER\nurl\thttps://x/v{version}/nxb-v{version}.zip\n')+ self.assertEqual(nxbuild.read(p), {'version': '0.1.0', 'sha256': None, 'url': 'https://x/v0.1.0/nxb-v0.1.0.zip'})+ p.write_text('version\t0.1\nsha256\tx\nurl\tu\n')+ with self.assertRaises(SystemExit): nxbuild.read(p)++ def zip(self, t, files):+ z = Path(t, 'nxb.zip')+ with zipfile.ZipFile(z, 'w') as f:+ for n, b in files.items(): f.writestr(n, b)+ return z++ def test_the_digest_is_checked_before_unpacking(self):+ with tempfile.TemporaryDirectory() as t:+ z = self.zip(t, {'nxb-v0.1.0/nxb': b'binary', 'nxb-v0.1.0/README': b'r'})+ good = hashlib.sha256(z.read_bytes()).hexdigest()+ out = nxbuild.unpack(z, good, Path(t, 'res/nxbuild'))+ self.assertEqual(out.read_bytes(), b'binary')+ self.assertTrue(out.stat().st_mode & 0o111)+ z = self.zip(t, {'nxb': b'other'})+ with self.assertRaises(SystemExit) as e: nxbuild.unpack(z, good, Path(t, 'res2'))+ self.assertIn('not bundled', str(e.exception))+ self.assertFalse(z.exists(), 'a zip whose digest differs is deleted')+ self.assertFalse(Path(t, 'res2').exists(), 'nothing is unpacked')++ def test_sha256sums(self):+ text = f'{"a" * 64} nxb-v0.1.0-macos-arm64.zip\n{"B" * 64} *other.zip\n'+ self.assertEqual(nxbuild.sums_digest(text, 'nxb-v0.1.0-macos-arm64.zip'), 'a' * 64)+ self.assertEqual(nxbuild.sums_digest(text, 'other.zip'), 'b' * 64)+ with self.assertRaises(SystemExit): nxbuild.sums_digest(text, 'missing.zip')+++if __name__ == '__main__':+ unittest.main()diff --git a/tools/try-local b/tools/try-localindex 48624458..3fa3f703 100755--- a/tools/try-local+++ b/tools/try-local@@ -136,6 +136,7 @@ cp frontend/.build/release/NX2Apple "$app/Contents/MacOS/NSPortBuilder"if [ "$have_icon" = 1 ]; then cp "$icon_cache/Assets.car" "$icon_cache/AppIcon.icns" "$app/Contents/Resources/"; fiditto "$tools_cache/res/tools" "$app/Contents/Resources/tools"ditto "$tools_cache/res/python" "$app/Contents/Resources/python"+python3 frontend/tools/nxbuild.py fetch "$app/Contents/Resources" "$work/nxbuild" # the pinned experimental builder (#749), nothing without a digestditto "$work/engine-tree" "$app/Contents/Resources/engine"frontend/tools/sign_engine_tree.sh "$app/Contents/Resources/engine" "$work/engine-tree"rsync -a --exclude .build --exclude .swiftpm frontend-ios/ "$app/Contents/Resources/frontend-ios/"