Play your own Switch games on iPhone/iPad — one-click macOS app, no terminal. | 把 Switch 游戏装进 iPhone:图形界面一键构建,不用命令行。Bring your own game.
Swift apple-siliconcustom-formathomebrewiosipaipadipadosiphonemetalnative-portnintendo-switchnxnativeromfsswitch-games
Experimental builder: an opt-in setting that hands stages to nxbuild (#749)
v0lumehi2280f436bb03
diff --git a/frontend/README.md b/frontend/README.mdindex 2c74484a..1e2d7536 100644--- a/frontend/README.md+++ b/frontend/README.md@@ -266,6 +266,37 @@ bundle), a bundle under `.build/` or `DerivedData/`, a bundle beside`Package.swift` (the `NX2Apple.app` made here), or one whose Info.plist versionis not this binary's release number. `AppUpdate.swift` holds these rules.+## Experimental builder++Settings, Build tools, **Use experimental builder** (off by default, LNPANDY/NX2Apple#749). nxbuild is a build tool in Rust that reproduces stages of the NSP script byte for byte, and faster. With the switch on, the build gets `DM_EXPERIMENTAL_BUILDER=1` and `NXB_BIN` (the path of the `nxb` binary). The script stays the driver. It runs the extraction (`nxb-extract`) and the shader split (`nxb-split`) through nxbuild where nxbuild supports the case, with the same outputs, stages and events. Everything else runs the standard way:++- an XCI, an NSZ, a merged package, `--program`, `--allow-damaged`, DLC, run-time captures, NROs, a game decoder, a provisional or derived profile+- an nxbuild exit code other than 0 (4 means "not supported yet")+- an output that fails the checks: for an extraction the title, `ExeFS/main`, `RomFS.img` and the profile's marker, for the split the data rows of the profile's shader tables++The build log says which stage nxbuild ran and why a stage fell back. nxbuild's own log is `logs/nxb_extract_<version>.log` or `logs/nxb_split.log` in the build folder. The shader step, the pipeline archive and the Mac app are not handed over yet. The NSP script needs the engine patch that reads the two variables. An engine without it ignores them.++Where `nxb` comes from (`ExperimentalBuilder` in `NXCore`):++1. The path chosen in the row under the switch, the `nxb` binary or the folder that holds it. A chosen path that does not work is reported, never replaced by another binary.+2. Otherwise a downloaded release in `~/Library/Application Support/NS Port Builder/nxbuild/<version>/`. No nxbuild release is published yet, so `ExperimentalBuilder.releaseURL` and `releaseSHA256` are nil and nothing is downloaded. Once a release exists (a macOS arm64 zip of the binaries, signed ad hoc), both are pinned together in that file. The download goes to a temporary file, its SHA-256 is checked against `releaseSHA256` before anything is unpacked, and a zip whose digest differs is deleted and never run.++With neither, the switch only writes a note in the build log and the standard builder runs.++To try it with a local nxbuild:++```bash+cd <nxbuild checkout> && cargo build --release # the binaries land in its target folder, release/+```++Then turn the switch on and choose `release/nxb` (or the folder). From a terminal, the same works without the launcher:++```bash+DM_EXPERIMENTAL_BUILDER=1 NXB_BIN=<target>/release/nxb python3 engine/scripts/build_ipa_from_nsp.py <NSP folder> -k <prod.keys> --platform mac-app+```++The Qt front end has no switch for it: nxbuild is built for macOS on Apple silicon only, and on a Mac the launcher above is the one to use.+## Run```bashdiff --git a/frontend/Sources/NX2Apple/SettingsPage.swift b/frontend/Sources/NX2Apple/SettingsPage.swiftindex e8a34025..11ca887f 100644--- a/frontend/Sources/NX2Apple/SettingsPage.swift+++ b/frontend/Sources/NX2Apple/SettingsPage.swift@@ -33,7 +33,7 @@ struct SettingsPage: View {startInConsole: $startInConsole, onOpenControls: onOpenControls, onOpenControllers: onOpenControllers,onOpenHomeSettings: HomeButtonSettings.available ? { HomeButtonSettings.open() } : nil) // HomeButtonSettings.swift (#336)ToolsCard(repo: repo, engine: engine, dependencies: dependencies, buildRunning: build.running,- shellCache: { ShellCacheRow(build: build) }) {+ shellCache: { ShellCacheRow(build: build) }, builder: { ExperimentalBuilderRow(build: build) }) {DiagnosticsButton(report: diagnostics, game: library.selected)}AppCard(updates: updates, buildRunning: build.running, notify: $notifyBuilds, notifyAvailable: BuildNotice.hasBundleID,diff --git a/frontend/Sources/NXCore/ExperimentalBuilder.swift b/frontend/Sources/NXCore/ExperimentalBuilder.swiftnew file mode 100644index 00000000..62091942--- /dev/null+++ b/frontend/Sources/NXCore/ExperimentalBuilder.swift@@ -0,0 +1,65 @@+import Foundation++/// The experimental builder (LNPANDY/NX2Apple#749): nxbuild, a build tool in Rust that reproduces stages of the NSP script byte for byte and faster.+/// The script stays the driver. With `DM_EXPERIMENTAL_BUILDER=1` and `NXB_BIN` it hands the stages nxbuild supports (the extraction and the shader split today)+/// to the binaries beside `nxb`, and runs every other case the standard way. An engine without that patch ignores both variables.+///+/// Where `nxb` comes from:+/// 1. the path the player chose in Settings (the `nxb` binary, or the folder that holds it), for a local build of nxbuild;+/// 2. a release unpacked into `installFolder` by a download on demand. That download needs a published release: `releaseURL` and `releaseSHA256` are+/// pinned here together once it exists (a macOS arm64 zip of the binaries, signed ad hoc). Until then both are nil and only the chosen path is used.+/// The download writes the zip to a temporary file, checks its SHA-256 against `releaseSHA256` before anything is unpacked, and unpacks it into+/// `installFolder/<version>`. A zip whose digest differs is deleted and never run.+package enum ExperimentalBuilder {+ /// The nxbuild release the launcher downloads on demand. nil: none is published yet.+ package static let releaseURL: URL? = nil+ /// The SHA-256 (lowercase hex) of the zip at `releaseURL`, checked before it is unpacked. nil with it.+ package static let releaseSHA256: String? = nil++ /// The binaries the NSP script runs, all beside `nxb`. A missing stage binary makes that stage run the standard way.+ package static let stageTools = ["nxb-extract", "nxb-split"]++ /// Where a downloaded release goes: `<support>/nxbuild`, beside the venv in the launcher's data folder.+ package static func installFolder(support: String) -> String {+ (support as NSString).appendingPathComponent("nxbuild")+ }++ /// The `nxb` a chosen path names: the file itself, or `nxb` inside a folder. Empty for an empty path.+ package static func binary(at path: String, isDirectory: (String) -> Bool = directoryExists) -> String {+ let p = (path as NSString).expandingTildeInPath+ if p.isEmpty { return "" }+ return isDirectory(p) ? (p as NSString).appendingPathComponent("nxb") : p+ }++ /// The `nxb` the build takes: the chosen path when it is an executable `nxb`, else the newest release in `installFolder`. nil when there is none.+ package static func resolve(customPath: String, support: String, isExecutable: (String) -> Bool = { FileManager.default.isExecutableFile(atPath: $0) },+ isDirectory: (String) -> Bool = directoryExists, list: (String) -> [String] = listing) -> String? {+ let chosen = binary(at: customPath, isDirectory: isDirectory)+ if !chosen.isEmpty {+ return isExecutable(chosen) ? chosen : nil // a chosen path that does not work is not silently replaced by another binary+ }+ let folder = installFolder(support: support)+ let versions = list(folder).sorted { $0.compare($1, options: .numeric) == .orderedDescending }+ return versions.lazy.map { ((folder as NSString).appendingPathComponent($0) as NSString).appendingPathComponent("nxb") }.first(where: isExecutable)+ }++ /// The variables the build gets: the switch and the path of `nxb`.+ package static func environment(nxb: String) -> [String: String] {+ ["DM_EXPERIMENTAL_BUILDER": "1", "NXB_BIN": nxb]+ }++ /// Whether a downloaded zip is the pinned release: its SHA-256 (lowercase hex, as `shasum -a 256` prints it) against `releaseSHA256`.+ package static func matchesRelease(sha256 digest: String, expected: String? = releaseSHA256) -> Bool {+ guard let expected, expected.count == 64 else { return false }+ return digest.lowercased() == expected.lowercased()+ }++ package static func directoryExists(_ path: String) -> Bool {+ var dir: ObjCBool = false+ return FileManager.default.fileExists(atPath: path, isDirectory: &dir) && dir.boolValue+ }++ package static func listing(_ path: String) -> [String] {+ (try? FileManager.default.contentsOfDirectory(atPath: path)) ?? []+ }+}diff --git a/frontend/Sources/NXLauncher/ExperimentalBuilderRow.swift b/frontend/Sources/NXLauncher/ExperimentalBuilderRow.swiftnew file mode 100644index 00000000..68ddc9de--- /dev/null+++ b/frontend/Sources/NXLauncher/ExperimentalBuilderRow.swift@@ -0,0 +1,35 @@+import SwiftUI+import AppKit+import NXCore+import NXModel++/// Settings, "Build tools": the switch of the experimental builder (#749) and, when it is on, the nxb binary it uses with a button to choose it.+/// A view of its own because it observes the build store, which keeps the setting with the other build settings. ToolsCard takes it as a view.+package struct ExperimentalBuilderRow: View {+ @ObservedObject var build: BuildStore+ @EnvironmentObject var loc: Localizer++ package init(build: BuildStore) { self.build = build }++ package var body: some View {+ CardRow(loc.t("builder.title"), detail: loc.t("builder.note")) {+ Toggle("", isOn: $build.experimentalBuilder).labelsHidden().toggleStyle(LauncherSwitchStyle()).disabled(build.running)+ }+ if build.experimentalBuilder {+ CardDivider()+ let nxb = build.experimentalBuilderBinary+ CardRow(loc.t("builder.program"), detail: nxb.map { RepoLayout.shortPath($0) } ?? loc.t("builder.notFound"),+ dot: nxb == nil ? LauncherPalette.building : LauncherPalette.ready, help: nxb ?? "") {+ Button(loc.t("builder.choose")) { pick() }.disabled(build.running)+ .padTarget("set.builderChoose", scope: "main") { if !build.running { pick() } }+ }+ }+ }++ private func pick() {+ let p = NSOpenPanel()+ p.canChooseFiles = true; p.canChooseDirectories = true; p.allowsMultipleSelection = false+ p.message = loc.t("builder.pickMessage")+ if p.runModal() == .OK, let u = p.url { build.experimentalBuilderPath = u.path }+ }+}diff --git a/frontend/Sources/NXLauncher/ToolsCard.swift b/frontend/Sources/NXLauncher/ToolsCard.swiftindex e6446b42..6a03e242 100644--- a/frontend/Sources/NXLauncher/ToolsCard.swift+++ b/frontend/Sources/NXLauncher/ToolsCard.swift@@ -5,23 +5,26 @@ import NXModel/// Settings, "Tools": the repository or the release app's engine, the dependency rows and the diagnostics row./// Observes the repository, engine sync and dependency stores. What it needs from the build store comes as values:-/// whether a build runs. The cache row (ShellCacheRow, which observes the build store) and the diagnostics button come as views.-package struct ToolsCard<Diagnostics: View, ShellCache: View>: View {+/// whether a build runs. The cache row (ShellCacheRow, which observes the build store), the experimental builder's rows (ExperimentalBuilderRow, the same)+/// and the diagnostics button come as views.+package struct ToolsCard<Diagnostics: View, ShellCache: View, Builder: View>: View {@ObservedObject var repo: RepoStore@ObservedObject var engine: EngineSyncStore@ObservedObject var dependencies: DependencyStorelet buildRunning: Boollet shellCache: ShellCache+ let builder: Builderlet diagnostics: Diagnostics@EnvironmentObject var loc: Localizerpackage init(repo: RepoStore, engine: EngineSyncStore, dependencies: DependencyStore, buildRunning: Bool,- @ViewBuilder shellCache: () -> ShellCache, @ViewBuilder diagnostics: () -> Diagnostics) {+ @ViewBuilder shellCache: () -> ShellCache, @ViewBuilder builder: () -> Builder, @ViewBuilder diagnostics: () -> Diagnostics) {self.repo = repoself.engine = engineself.dependencies = dependenciesself.buildRunning = buildRunningself.shellCache = shellCache()+ self.builder = builder()self.diagnostics = diagnostics()}@@ -128,6 +131,8 @@ package struct ToolsCard<Diagnostics: View, ShellCache: View>: View {}}CardDivider()+ builder+ CardDivider()CardRow(loc.t("diag.title"), detail: loc.t("diag.note")) { diagnostics }}}diff --git a/frontend/Sources/NXModel/BuildStore+Builder.swift b/frontend/Sources/NXModel/BuildStore+Builder.swiftnew file mode 100644index 00000000..c9879b59--- /dev/null+++ b/frontend/Sources/NXModel/BuildStore+Builder.swift@@ -0,0 +1,26 @@+import Foundation+import NXCore++// The experimental builder (#749): Settings' switch and the nxb binary it uses, and what the build gets from them.+// The rules (where nxb comes from, the variables) are ExperimentalBuilder in NXCore.++extension BuildStore {+ /// The launcher's data folder, where a downloaded nxbuild release goes (`home` is a temporary folder in tests).+ package var builderSupportFolder: String { home + "/Library/Application Support/NS Port Builder" }++ /// The `nxb` the next build takes, nil when there is none (the switch is then only a note in the log).+ package var experimentalBuilderBinary: String? {+ ExperimentalBuilder.resolve(customPath: experimentalBuilderPath, support: builderSupportFolder)+ }++ /// The variables the build gets for the experimental builder, empty when it is off or no nxb is found. Says in the log which it is.+ package func builderEnvironment() -> [String: String] {+ guard experimentalBuilder else { return [:] }+ guard let nxb = experimentalBuilderBinary else {+ say(t("msg.builderMissing"))+ return [:]+ }+ say(String(format: t("msg.builderOn"), nxb))+ return ExperimentalBuilder.environment(nxb: nxb)+ }+}diff --git a/frontend/Sources/NXModel/BuildStore+Run.swift b/frontend/Sources/NXModel/BuildStore+Run.swiftindex 72f1eb6c..633cd9aa 100644--- a/frontend/Sources/NXModel/BuildStore+Run.swift+++ b/frontend/Sources/NXModel/BuildStore+Run.swift@@ -187,6 +187,7 @@ extension BuildStore {// (the packages are compiled at bundle time), and a user site must not shadow its pinned packages.denv.merge(BundledTools.environment(for: py)) { $1 }if !oneGameIPAPath.isEmpty { denv.merge(OneGameIPA.environment) { $1 } }+ denv.merge(builderEnvironment()) { $1 } // the experimental builder, when it is on (BuildStore+Builder.swift, #749)let launch = ProcessLaunch(executable: py, arguments: args, environment: denv, directory: engine, groupPython: py)let onExit: @Sendable (Int32) -> Void = { [weak self] status inhop {diff --git a/frontend/Sources/NXModel/BuildStore.swift b/frontend/Sources/NXModel/BuildStore.swiftindex 3b741293..33d079d9 100644--- a/frontend/Sources/NXModel/BuildStore.swift+++ b/frontend/Sources/NXModel/BuildStore.swift@@ -72,6 +72,8 @@ func hop(_ body: @escaping @MainActor () -> Void) {nonisolated package static let outputKey = "buildOutputDirectory"nonisolated package static let targetKey = "buildTarget"nonisolated package static let bcOnCPUKey = "bcOnCPU"+ nonisolated package static let experimentalBuilderKey = "experimentalBuilder"+ nonisolated package static let experimentalBuilderPathKey = "experimentalBuilderPath"@Published package var gameDir = ""@Published package var slots: [Slot] = [@@ -126,6 +128,15 @@ func hop(_ body: @escaping @MainActor () -> Void) {@Published package var bcOnCPU: Bool {didSet { defaults.set(bcOnCPU, forKey: Self.bcOnCPUKey) }}+ /// Settings, "Use experimental builder" (#749): the NSP script hands the stages nxbuild supports to it, and runs the rest the standard way.+ /// Off by default. BuildStore+Builder.swift passes it to the build.+ @Published package var experimentalBuilder: Bool {+ didSet { defaults.set(experimentalBuilder, forKey: Self.experimentalBuilderKey) }+ }+ /// The `nxb` binary the player chose (or its folder), empty for the downloaded release (ExperimentalBuilder.resolve).+ @Published package var experimentalBuilderPath: String {+ didSet { defaults.set(experimentalBuilderPath, forKey: Self.experimentalBuilderPathKey) }+ }@Published package var archivePath = "" // the romfs folder the pack wrote/// Where this build's folder actually is: --work-dir when the checkout has it, else the script's/// default under the input folder (issue #5 — an upstream checkout has no --work-dir).@@ -207,6 +218,8 @@ func hop(_ body: @escaping @MainActor () -> Void) {outDir = defaults.string(forKey: Self.outputKey) ?? ""target = BuildTarget.saved(defaults)bcOnCPU = defaults.bool(forKey: Self.bcOnCPUKey)+ experimentalBuilder = defaults.bool(forKey: Self.experimentalBuilderKey)+ experimentalBuilderPath = defaults.string(forKey: Self.experimentalBuilderPathKey) ?? ""}/// The language and the text this store writes itself: views take strings from the environment.diff --git a/frontend/Tests/NX2AppleTests/ExperimentalBuilderCaptureTests.swift b/frontend/Tests/NX2AppleTests/ExperimentalBuilderCaptureTests.swiftnew file mode 100644index 00000000..547b39a9--- /dev/null+++ b/frontend/Tests/NX2AppleTests/ExperimentalBuilderCaptureTests.swift@@ -0,0 +1,62 @@+import XCTest+import SwiftUI+import AppKit+@testable import NX2Apple+@testable import NXCore+@testable import NXModel+@testable import NXLauncher++/// Visual QA of the experimental builder setting, LNPANDY/NX2Apple#749 (matched by --filter testRenderExperimentalBuilderForVisualQA). The Tools card of the+/// release app and of a checkout: the switch off, on with an nxb found, and on without one. The nxb is a stand-in file in a temporary folder.+final class ExperimentalBuilderCaptureTests: IsolatedDefaultsTestCase {+ @MainActor func testRenderExperimentalBuilderForVisualQA() throws {+ guard let directory = ProcessInfo.processInfo.environment["NX_LAUNCHER_CAPTURE_DIR"] else { throw XCTSkip("Set NX_LAUNCHER_CAPTURE_DIR to capture native views") }+ let output = URL(fileURLWithPath: directory)+ try FileManager.default.createDirectory(at: output, withIntermediateDirectories: true)+ let lang = CaptureLanguage.current(.en)+ let folder = NSTemporaryDirectory() + "nxbuild-capture-" + UUID().uuidString+ try FileManager.default.createDirectory(atPath: folder, withIntermediateDirectories: true)+ defer { try? FileManager.default.removeItem(atPath: folder) }+ let nxb = folder + "/nxb"+ FileManager.default.createFile(atPath: nxb, contents: Data("#!/bin/sh\n".utf8), attributes: [.posixPermissions: 0o755])++ let stores = LauncherStores(defaults: testDefaults); let model = stores.build+ stores.localizer.lang = lang+ stores.engineSync.status = ""+ MainActor.assumeIsolated { stores.dependencies.checked = true }+ MainActor.assumeIsolated { stores.dependencies.deps = [Dep(id: "test", name: "Xcode", hint: "", present: true, installCmd: [])] }++ func shot(_ name: String) throws {+ let card = ToolsCard(repo: stores.repo, engine: stores.engineSync, dependencies: stores.dependencies, buildRunning: model.running,+ shellCache: { ShellCacheRow(build: model) }, builder: { ExperimentalBuilderRow(build: model) }) { DiagnosticsButton(report: { _, _ in }) }+ let framed = card.environmentObject(stores.localizer).environmentObject(PadFocus()).padding(24).frame(width: 860).background(Color(white: 0.09)).environment(\.colorScheme, .dark)+ NSApplication.shared.setActivationPolicy(.accessory)+ let host = NSHostingView(rootView: framed)+ host.frame = NSRect(origin: .zero, size: host.fittingSize)+ host.wantsLayer = true+ host.layer?.backgroundColor = NSColor(white: 0.09, alpha: 1).cgColor+ let window = OffscreenWindow(size: host.frame.size)+ window.contentView = host+ window.orderFront(nil)+ RunLoop.main.run(until: Date().addingTimeInterval(0.4))+ host.layoutSubtreeIfNeeded()+ let rep = try XCTUnwrap(host.bitmapImageRepForCachingDisplay(in: host.bounds))+ host.cacheDisplay(in: host.bounds, to: rep)+ try XCTUnwrap(rep.representation(using: .png, properties: [:])).write(to: output.appendingPathComponent("builder-\(name)-\(lang.code).png"))+ window.orderOut(nil)+ }++ stores.engineSync.bundle = URL(fileURLWithPath: "/Applications/NX2Apple.app/Contents/Resources/engine") // the release app's rows+ model.experimentalBuilder = false+ try shot("release-0-off")+ model.experimentalBuilder = true; model.experimentalBuilderPath = nxb+ try shot("release-1-on")+ model.experimentalBuilderPath = folder + "/gone/nxb"+ try shot("release-2-not-found")+ stores.engineSync.bundle = nil // a checkout's rows+ model.experimentalBuilder = false+ try shot("checkout-0-off")+ model.experimentalBuilder = true; model.experimentalBuilderPath = nxb+ try shot("checkout-1-on")+ }+}diff --git a/frontend/Tests/NX2AppleTests/ShellCacheCaptureTests.swift b/frontend/Tests/NX2AppleTests/ShellCacheCaptureTests.swiftindex 6b37904c..73565969 100644--- a/frontend/Tests/NX2AppleTests/ShellCacheCaptureTests.swift+++ b/frontend/Tests/NX2AppleTests/ShellCacheCaptureTests.swift@@ -26,7 +26,7 @@ final class ShellCacheCaptureTests: IsolatedDefaultsTestCase {func shot(_ name: String) throws {let card = ToolsCard(repo: stores.repo, engine: stores.engineSync, dependencies: stores.dependencies, buildRunning: model.running,- shellCache: { ShellCacheRow(build: model) }) { DiagnosticsButton(report: { _, _ in }) }+ shellCache: { ShellCacheRow(build: model) }, builder: { ExperimentalBuilderRow(build: model) }) { DiagnosticsButton(report: { _, _ in }) }let framed = card.environmentObject(stores.localizer).environmentObject(PadFocus()).padding(24).frame(width: 860).background(Color(white: 0.09)).environment(\.colorScheme, .dark)NSApplication.shared.setActivationPolicy(.accessory)let host = NSHostingView(rootView: framed)diff --git a/frontend/Tests/NXCoreTests/ExperimentalBuilderTests.swift b/frontend/Tests/NXCoreTests/ExperimentalBuilderTests.swiftnew file mode 100644index 00000000..cc49d554--- /dev/null+++ b/frontend/Tests/NXCoreTests/ExperimentalBuilderTests.swift@@ -0,0 +1,48 @@+import XCTest+@testable import NXCore++/// `ExperimentalBuilder` (#749): which nxb the build takes and what it passes. Paths are strings checked by stand-in closures, nothing is run.+final class ExperimentalBuilderTests: XCTestCase {+ private let support = "/Users/p/Library/Application Support/NS Port Builder"++ func testAChosenFileOrFolderNamesTheNxbBinary() {+ XCTAssertEqual(ExperimentalBuilder.binary(at: "/opt/nxbuild/nxb", isDirectory: { _ in false }), "/opt/nxbuild/nxb")+ XCTAssertEqual(ExperimentalBuilder.binary(at: "/opt/nxbuild", isDirectory: { $0 == "/opt/nxbuild" }), "/opt/nxbuild/nxb")+ XCTAssertEqual(ExperimentalBuilder.binary(at: "", isDirectory: { _ in true }), "")+ XCTAssertEqual(ExperimentalBuilder.binary(at: "~/nxb", isDirectory: { _ in false }), NSHomeDirectory() + "/nxb")+ }++ func testTheChosenPathWinsAndIsNeverSwappedForAnother() {+ let installed = support + "/nxbuild/0.1.0/nxb"+ let ok: (String) -> Bool = { [installed, "/opt/nxb"].contains($0) }+ let list: (String) -> [String] = { _ in ["0.1.0"] }+ XCTAssertEqual(ExperimentalBuilder.resolve(customPath: "/opt/nxb", support: support, isExecutable: ok, isDirectory: { _ in false }, list: list), "/opt/nxb")+ XCTAssertNil(ExperimentalBuilder.resolve(customPath: "/gone/nxb", support: support, isExecutable: ok, isDirectory: { _ in false }, list: list),+ "a chosen path that does not work is reported, not replaced by the downloaded release")+ XCTAssertEqual(ExperimentalBuilder.resolve(customPath: "", support: support, isExecutable: ok, isDirectory: { _ in false }, list: list), installed)+ }++ func testTheNewestDownloadedReleaseIsTaken() {+ let folder = ExperimentalBuilder.installFolder(support: support)+ XCTAssertEqual(folder, support + "/nxbuild")+ let have = Set(["0.2.0", "0.10.0", "0.9.1"].map { folder + "/" + $0 + "/nxb" })+ let nxb = ExperimentalBuilder.resolve(customPath: "", support: support, isExecutable: { have.contains($0) }, isDirectory: { _ in false },+ list: { _ in ["0.2.0", "0.10.0", "0.9.1", "notes.txt"] })+ XCTAssertEqual(nxb, folder + "/0.10.0/nxb", "versions compare as numbers")+ XCTAssertNil(ExperimentalBuilder.resolve(customPath: "", support: support, isExecutable: { _ in false }, isDirectory: { _ in false }, list: { _ in [] }))+ }++ func testTheBuildGetsTheSwitchAndThePath() {+ XCTAssertEqual(ExperimentalBuilder.environment(nxb: "/opt/nxb"), ["DM_EXPERIMENTAL_BUILDER": "1", "NXB_BIN": "/opt/nxb"])+ }++ func testNoReleaseIsPinnedYetAndADownloadIsOnlyTakenWithItsDigest() {+ XCTAssertNil(ExperimentalBuilder.releaseURL, "no nxbuild release is published yet: nothing is downloaded")+ XCTAssertNil(ExperimentalBuilder.releaseSHA256)+ let digest = String(repeating: "ab", count: 32)+ XCTAssertFalse(ExperimentalBuilder.matchesRelease(sha256: digest), "without a pinned digest nothing matches")+ XCTAssertTrue(ExperimentalBuilder.matchesRelease(sha256: digest.uppercased(), expected: digest))+ XCTAssertFalse(ExperimentalBuilder.matchesRelease(sha256: String(repeating: "cd", count: 32), expected: digest))+ XCTAssertFalse(ExperimentalBuilder.matchesRelease(sha256: "ab", expected: "ab"), "a digest is 64 hex digits")+ }+}diff --git a/frontend/Tests/NXModelTests/BuildStoreTests.swift b/frontend/Tests/NXModelTests/BuildStoreTests.swiftindex 21358fa1..b51f0304 100644--- a/frontend/Tests/NXModelTests/BuildStoreTests.swift+++ b/frontend/Tests/NXModelTests/BuildStoreTests.swift@@ -145,6 +145,39 @@ import XCTestXCTAssertEqual(makeStore().outDir, "/out", "a new store starts from what was saved")}+ // MARK: The experimental builder (#749)++ func testTheExperimentalBuilderIsOffByDefaultAndItsSettingIsSaved() {+ let store = makeStore()+ XCTAssertFalse(store.experimentalBuilder)+ XCTAssertEqual(store.builderEnvironment(), [:], "off: the build gets nothing")+ XCTAssertFalse(store.log.text.contains(store.t("msg.builderMissing")), "and the log says nothing")+ store.experimentalBuilder = true; store.experimentalBuilderPath = "/opt/nxbuild"+ XCTAssertTrue(defaults.bool(forKey: BuildStore.experimentalBuilderKey))+ XCTAssertEqual(defaults.string(forKey: BuildStore.experimentalBuilderPathKey), "/opt/nxbuild")+ let again = makeStore()+ XCTAssertTrue(again.experimentalBuilder)+ XCTAssertEqual(again.experimentalBuilderPath, "/opt/nxbuild")+ }++ func testTheBuildGetsTheChosenNxbOrANoteThatTheStandardBuilderRuns() throws {+ let store = makeStore()+ store.home = temporary.path+ store.experimentalBuilder = true+ store.experimentalBuilderPath = temporary.path + "/gone/nxb"+ XCTAssertNil(store.experimentalBuilderBinary)+ XCTAssertEqual(store.builderEnvironment(), [:])+ XCTAssertTrue(store.log.text.contains(store.t("msg.builderMissing")))+ let nxb = try file("nxb")+ try FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: nxb)+ store.experimentalBuilderPath = temporary.path // the folder that holds it+ XCTAssertEqual(store.experimentalBuilderBinary, nxb)+ XCTAssertEqual(store.builderEnvironment(), ["DM_EXPERIMENTAL_BUILDER": "1", "NXB_BIN": nxb])+ XCTAssertTrue(store.log.text.contains(String(format: store.t("msg.builderOn"), nxb)))+ store.experimentalBuilder = false+ XCTAssertEqual(store.builderEnvironment(), [:], "off again: nothing, whatever the path")+ }+func testTakingFilesFillsTheRowAndNamesTheFirstFileThatDoesNotFit() {let store = makeStore()store.take(picked: ["/g/Game [0100000000001000].nsp"], kind: .base)diff --git a/frontend/Tests/NXModelTests/GameFolderStoreTests.swift b/frontend/Tests/NXModelTests/GameFolderStoreTests.swiftindex 6806e397..73de40e0 100644--- a/frontend/Tests/NXModelTests/GameFolderStoreTests.swift+++ b/frontend/Tests/NXModelTests/GameFolderStoreTests.swift@@ -121,6 +121,22 @@ import XCTestXCTAssertTrue(args.contains("mac-app"))}+ /// #749: the script's command stays the same with the experimental builder on, and the log names the nxb the build gets.+ func testTheExperimentalBuilderChangesNoArgumentAndIsNamedInTheLog() {+ let (store, game) = makeStore()+ store.scan(folder: game)+ store.startBuild()+ let standard = store.lastBuildCommand+ XCTAssertFalse(store.log.text.contains("nxb"))+ let nxb = root + "/nxbuild/nxb"+ write(nxb)+ try? FileManager.default.setAttributes([.posixPermissions: 0o755], ofItemAtPath: nxb)+ store.experimentalBuilder = true; store.experimentalBuilderPath = nxb+ store.startBuild()+ XCTAssertEqual(store.lastBuildCommand, standard, "the variables carry it, not the command line")+ XCTAssertTrue(store.log.text.contains(String(format: store.t("msg.builderOn"), nxb)))+ }+func testWithoutAProfileTheFolderIsRefusedWithTheTitleID() {let (store, game) = makeStore(profile: false)store.scan(folder: game)diff --git a/frontend/l10n/de.json b/frontend/l10n/de.jsonindex eab599e3..e4e5d6da 100644--- a/frontend/l10n/de.json+++ b/frontend/l10n/de.json@@ -61,6 +61,12 @@"build.showLibrary": "Spiel ansehen","build.start": "Build starten","build.startGame": "Spiel bauen",+ "builder.choose": "Wählen …",+ "builder.notFound": "Nicht gefunden, wähle das Programm nxb",+ "builder.note": "Schneller, unterstützt vielleicht nicht jedes Spiel und wechselt bei Bedarf von selbst zum Standard-Builder",+ "builder.pickMessage": "Wähle das Programm nxb von nxbuild oder den Ordner, der es enthält",+ "builder.program": "Builder-Programm",+ "builder.title": "Experimentellen Builder verwenden","cheats.alwaysOn": "Immer an","cheats.broken": "Fehlerhaft, läuft nicht","cheats.buildLabel": "Build-ID dieses Spiels",@@ -1057,6 +1063,8 @@"msg.bcOn": "Modus für ältere Geräte: BC-Texturen werden auf der CPU dekodiert","msg.buildFailed": "Build fehlgeschlagen (Exit-Code %d). Siehe das Protokoll unten","msg.buildFailedReason": "Build fehlgeschlagen. Siehe das Protokoll unten. Letzte Zeile: %@",+ "msg.builderMissing": "Der experimentelle Builder ist an, aber es wurde kein Programm nxb gefunden. Der Standard-Builder läuft.",+ "msg.builderOn": "Experimenteller Builder: %@","msg.done": "Fertig","msg.exitCode": "Exit-Code %d","msg.installingNsz": "Der gewählte Dump ist NSZ / XCZ. nsz wird in das Build-Python installiert …",diff --git a/frontend/l10n/en.json b/frontend/l10n/en.jsonindex c1f54631..882767f0 100644--- a/frontend/l10n/en.json+++ b/frontend/l10n/en.json@@ -59,6 +59,12 @@"build.showLibrary": "View game","build.start": "Start build","build.startGame": "Build game",+ "builder.choose": "Choose…",+ "builder.notFound": "Not found, choose the nxb program",+ "builder.note": "Faster, may not support every game, and falls back to the standard builder by itself",+ "builder.pickMessage": "Choose the nxb program of nxbuild, or the folder that holds it",+ "builder.program": "Builder program",+ "builder.title": "Use experimental builder","cheats.alwaysOn": "Always on","cheats.broken": "Malformed, will not run","cheats.buildLabel": "Build ID of this game",@@ -1055,6 +1061,8 @@"msg.bcOn": "Older-device mode: BC textures will be decoded on the CPU","msg.buildFailed": "Build failed (exit code %d) — see the log below","msg.buildFailedReason": "Build failed — see the log below. Last line: %@",+ "msg.builderMissing": "The experimental builder is on, but no nxb program was found. The standard builder runs.",+ "msg.builderOn": "Experimental builder: %@","msg.done": "Done","msg.exitCode": "exit code %d","msg.installingNsz": "The selected dump is NSZ / XCZ — installing nsz into the build's Python…",diff --git a/frontend/l10n/it.json b/frontend/l10n/it.jsonindex 1c5fecde..ea4f7eab 100644--- a/frontend/l10n/it.json+++ b/frontend/l10n/it.json@@ -60,6 +60,12 @@"build.showLibrary": "Vedi gioco","build.start": "Avvia la build","build.startGame": "Compila gioco",+ "builder.choose": "Scegli…",+ "builder.notFound": "Non trovato, scegli il programma nxb",+ "builder.note": "Più veloce, potrebbe non supportare ogni gioco e torna da solo al builder standard",+ "builder.pickMessage": "Scegli il programma nxb di nxbuild o la cartella che lo contiene",+ "builder.program": "Programma del builder",+ "builder.title": "Usa il builder sperimentale","cheats.alwaysOn": "Sempre attivo","cheats.broken": "Non valido, non verrà eseguito","cheats.buildLabel": "Build ID di questo gioco",@@ -1056,6 +1062,8 @@"msg.bcOn": "Modalità per dispositivi meno recenti: le texture BC verranno decodificate sulla CPU","msg.buildFailed": "Build non riuscita (codice di uscita %d): vedi il log qui sotto","msg.buildFailedReason": "Build non riuscita: vedi il log qui sotto. Ultima riga: %@",+ "msg.builderMissing": "Il builder sperimentale è attivo, ma non è stato trovato nessun programma nxb. Si usa il builder standard.",+ "msg.builderOn": "Builder sperimentale: %@","msg.done": "Fatto","msg.exitCode": "codice di uscita %d","msg.installingNsz": "Il dump selezionato è NSZ / XCZ: installazione di nsz nel Python della build…",diff --git a/frontend/l10n/sv.json b/frontend/l10n/sv.jsonindex ab2b496d..e5b30f42 100644--- a/frontend/l10n/sv.json+++ b/frontend/l10n/sv.json@@ -60,6 +60,12 @@"build.showLibrary": "Visa spel","build.start": "Starta bygget","build.startGame": "Bygg spel",+ "builder.choose": "Välj …",+ "builder.notFound": "Hittades inte, välj programmet nxb",+ "builder.note": "Snabbare, stöder kanske inte alla spel och går automatiskt tillbaka till standardbyggaren",+ "builder.pickMessage": "Välj programmet nxb från nxbuild eller mappen som innehåller det",+ "builder.program": "Byggprogram",+ "builder.title": "Använd experimentell byggare","cheats.alwaysOn": "Alltid på","cheats.broken": "Felformaterad, körs inte","cheats.buildLabel": "Spelets bygg-ID",@@ -896,6 +902,8 @@"msg.bcOn": "Läget för äldre enheter: BC-texturer avkodas på processorn","msg.buildFailed": "Bygget misslyckades med slutkod %d. Se loggen nedan.","msg.buildFailedReason": "Bygget misslyckades. Se loggen nedan. Sista raden: %@",+ "msg.builderMissing": "Den experimentella byggaren är på, men inget program nxb hittades. Standardbyggaren körs.",+ "msg.builderOn": "Experimentell byggare: %@","msg.done": "Klart","msg.exitCode": "slutkod %d","msg.installingNsz": "Det valda filformatet är komprimerat NSZ/XCZ. Installerar nsz i Python-miljön som används för bygget …",diff --git a/frontend/l10n/zh.json b/frontend/l10n/zh.jsonindex 77b3ebea..1bb7540f 100644--- a/frontend/l10n/zh.json+++ b/frontend/l10n/zh.json@@ -60,6 +60,12 @@"build.showLibrary": "查看游戏","build.start": "开始构建","build.startGame": "构建游戏",+ "builder.choose": "选择…",+ "builder.notFound": "未找到,请选择 nxb 程序",+ "builder.note": "更快,但可能不支持所有游戏,不支持时会自动改用标准构建器",+ "builder.pickMessage": "选择 nxbuild 的 nxb 程序,或包含它的文件夹",+ "builder.program": "构建器程序",+ "builder.title": "使用实验性构建器","cheats.alwaysOn": "始终开启","cheats.broken": "格式有误,不会运行","cheats.buildLabel": "这个游戏版本的构建 ID",@@ -1056,6 +1062,8 @@"msg.bcOn": "兼容老设备:BC 纹理将在 CPU 上解压","msg.buildFailed": "构建失败(退出码 %d),见下方日志","msg.buildFailedReason": "构建失败,见下方日志。最后一行:%@",+ "msg.builderMissing": "实验性构建器已开启,但未找到 nxb 程序。将使用标准构建器。",+ "msg.builderOn": "实验性构建器:%@","msg.done": "完成","msg.exitCode": "退出码 %d","msg.installingNsz": "选中的是 NSZ / XCZ 压缩格式,正在给构建用的 Python 安装 nsz…",diff --git a/patches/0073-nsp-script-hand-the-extraction-and-the-shader-sp.patch b/patches/0073-nsp-script-hand-the-extraction-and-the-shader-sp.patchnew file mode 100644index 00000000..3ea4fd54--- /dev/null+++ b/patches/0073-nsp-script-hand-the-extraction-and-the-shader-sp.patch@@ -0,0 +1,341 @@+From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001+From: v0lumehi <[email protected]>+Date: Thu, 5 Nov 2026 13:03:51 +0100+Subject: [PATCH] NSP script: hand the extraction and the shader split to+ nxbuild when asked++nxbuild (a build tool in Rust) reproduces the extraction and the shader split+byte for byte and much faster: about 15 s for both versions of Wonder against+2:27 for nx_extract and 0:43 for shader_split.py.++With DM_EXPERIMENTAL_BUILDER=1 and NXB_BIN (the nxb binary or its folder) the+script runs nxb-extract and nxb-split for these stages. It stays the driver:+stage names, events, work folder and reuse keys do not change. A case nxbuild+is not given (XCI, NSZ, merged packages, --program, --allow-damaged, DLC,+captures, NROs, a decoder, a provisional or derived profile), an nxbuild exit+code other than 0 (4 is "not supported yet") and an output that fails the+checks (the title, ExeFS/main, RomFS.img and the marker for an extraction,+the profile's table rows for the split) run the standard step, with a note.+nxbuild's own red lines stay in its log, since the build did not fail.+Without the variable nothing changes.++Upstream-Status: pending+Reason: the launcher's opt-in experimental builder (nxbuild) runs the extraction and the shader split+Issue: LNPANDY/NX2Apple#749++---+ .../tests/test_nsp_experimental_builder.py | 145 +++++++++++++++++++ docs/env_switches.tsv | 1 ++ scripts/build_ipa_from_nsp.py | 114 +++++++++++++-+ 3 files changed, 256 insertions(+), 4 deletions(-)+ create mode 100644 dm/tools/tests/test_nsp_experimental_builder.py++diff --git a/dm/tools/tests/test_nsp_experimental_builder.py b/dm/tools/tests/test_nsp_experimental_builder.py+new file mode 100644+index 0000000..859b72a+--- /dev/null++++ b/dm/tools/tests/test_nsp_experimental_builder.py+@@ -0,0 +1,145 @@++"""NSP script, the experimental builder (scripts/build_ipa_from_nsp.py, DM_EXPERIMENTAL_BUILDER and NXB_BIN): which nxbuild binary a stage takes, and that a stage it does not++support, fails or gets wrong runs the standard way. The nxbuild binaries are stand-in scripts in a temporary folder: no game files, keys or nxbuild needed."""++import importlib.util++import os++from pathlib import Path++import struct++import sys++import tempfile++import unittest++from unittest import mock++++REPO = Path(__file__).resolve().parents[3]++spec = importlib.util.spec_from_file_location('build_ipa_from_nsp', REPO / 'scripts/build_ipa_from_nsp.py')++assert spec is not None and spec.loader is not None++nsp = importlib.util.module_from_spec(spec); spec.loader.exec_module(nsp)++spec = importlib.util.spec_from_file_location('romfs_image', REPO / 'dm/tools/romfs_image.py')++assert spec is not None and spec.loader is not None++romfs_image = importlib.util.module_from_spec(spec); spec.loader.exec_module(romfs_image)++++APP = 0x010015100B514000++++++def pfs0(path, names): # a PFS0 (NSP) with empty files of these names++ strs = b''.join(n.encode() + b'\0' for n in names)++ ent, off = b'', 0++ for n in names: ent += struct.pack('<QQII', 0, 0, off, 0); off += len(n) + 1++ path.write_bytes(b'PFS0' + struct.pack('<III', len(names), len(strs), 0) + ent + strs)++ return path++++++def tool(folder, name, body): # an executable stand-in for an nxbuild binary: Python, run with this interpreter++ p = folder / name++ p.write_text(f'#!{sys.executable}\nimport os, shutil, sys\nfrom pathlib import Path\n{body}\n')++ p.chmod(0o755)++ return p++++++class Builder(unittest.TestCase):++ def setUp(self):++ self.t = Path(tempfile.mkdtemp()); self.addCleanup(lambda: nsp.rmtree(self.t))++ self.bin = self.t / 'bin'; self.bin.mkdir()++ self.nxb = tool(self.bin, 'nxb', 'sys.exit(0)')++ self.logs = self.t / 'logs'++ self.said = []++ for cm in (mock.patch.object(nsp.PROG, 'say', side_effect=self.said.append), mock.patch.object(nsp, '_nxb_noted', set()),++ mock.patch.dict(os.environ, {'DM_EXPERIMENTAL_BUILDER': '1', 'NXB_BIN': str(self.nxb)})):++ cm.__enter__(); self.addCleanup(cm.__exit__, None, None, None)++++ def text(self): return '\n'.join(self.said)++++ def test_off_by_default_and_with_another_value(self):++ for value in (None, '0', 'yes'):++ env = {k: v for k, v in os.environ.items() if k != 'DM_EXPERIMENTAL_BUILDER'} | ({'DM_EXPERIMENTAL_BUILDER': value} if value else {})++ with self.subTest(value=value): self.assertIsNone(nsp.nxb_tool('nxb', env))++ self.assertEqual(self.said, [], 'off says nothing')++++ def test_the_binaries_beside_nxb_or_in_the_folder(self):++ self.assertEqual(nsp.nxb_tool('nxb'), self.nxb)++ self.assertEqual(nsp.nxb_tool('nxb', dict(os.environ, NXB_BIN=str(self.bin))), self.nxb, 'NXB_BIN may name the folder')++ self.assertIsNone(nsp.nxb_tool('nxb-split'))++ self.assertIn('nxb-split is missing: the standard builder runs this stage', self.text())++ self.assertIsNone(nsp.nxb_tool('nxb', dict(os.environ, NXB_BIN=str(self.t / 'gone'))))++ self.assertIn('NXB_BIN does not name the nxb binary', self.text())++ self.said.clear(); nsp.nxb_tool('nxb-split'); self.assertEqual(self.said, [], 'a note once per build')++++ def extraction(self, body, marker='marker.txt'):++ src = self.t / 'src'; src.mkdir(exist_ok=True); (src / 'marker.txt').write_text('x')++ romfs_image.pack(src, self.t / 'image.img')++ tool(self.bin, 'nxb-extract', body.replace('IMAGE', repr(str(self.t / 'image.img'))))++ base = pfs0(self.t / 'Game [010015100B514000].nsp', ['a.nca', 'a.cnmt.nca'])++ tmp = self.t / 'extract' / 'v0.partial'; tmp.parent.mkdir(exist_ok=True)++ return nsp.nxb_extract(self.t / 'prod.keys', base, None, tmp, self.logs, 'v0', APP, marker), tmp++++ WRITES = ("out = Path(sys.argv[sys.argv.index('--out') + 1]); (out / 'ExeFS').mkdir(parents=True); (out / 'ExeFS/main').write_bytes(b'NSO0')\n"++ "shutil.copyfile(IMAGE, out / 'RomFS.img'); print('title {title}'); print('ExeFS: 4 files 1.0 MB, 0 identical to the compare dir (cloned), 1.0 MB newly written')")++++ def test_extract_takes_a_whole_output_of_the_game(self):++ ok, tmp = self.extraction(self.WRITES.format(title=f'{APP:016x}'))++ self.assertTrue(ok, self.text())++ self.assertTrue((tmp / 'RomFS.img').is_file())++ self.assertEqual(nsp.extracted(self.logs / 'extract_v0.log'), (4, 2**20), 'the summary nx_extract writes, for the next estimate')++ self.assertIn('extracted v0 with nxb-extract', self.text())++++ def test_extract_falls_back_on_unsupported_failure_or_a_bad_output(self):++ for body, why in (('sys.exit(4)', 'not supported yet'), ("print('[RED] not supported yet: sparse NCA'); sys.exit(1)", 'not supported yet'),++ ('sys.exit(3)', 'exit code 3'), (self.WRITES.format(title='0100000000001000'), 'failed the checks'),++ (self.WRITES.format(title=f'{APP:016x}').replace("print('title", "Path(out / 'RomFS.img').unlink(); print('title"), 'failed the checks')):++ with self.subTest(why=why, body=body[:30]):++ self.said.clear(); nsp._nxb_noted.clear()++ ok, tmp = self.extraction(body)++ self.assertFalse(ok)++ self.assertFalse(tmp.exists(), 'nx_extract starts from an empty folder')++ self.assertIn(why, self.text())++ self.assertIn('the standard builder runs it', self.text())++ self.assertNotIn('[RED]', self.text(), 'nxbuild\'s red lines stay in its log: the build did not fail')++++ def test_extract_leaves_packages_it_does_not_read_to_nx_extract(self):++ tool(self.bin, 'nxb-extract', f"Path({str(self.t / 'ran')!r}).write_text('x')")++ tmp = self.t / 'v0.partial'++ for names in (['a.cnmt.nca', 'b.cnmt.nca'], ['a.ncz', 'a.cnmt.nca']): # merged, NSZ++ with self.subTest(names=names):++ base = pfs0(self.t / 'g.nsp', names)++ self.assertFalse(nsp.nxb_extract(self.t / 'k', base, None, tmp, self.logs, 'v0', APP, None))++ xci = self.t / 'g.xci'; xci.write_bytes(b'\0' * 0x100 + b'HEAD')++ self.assertFalse(nsp.nxb_extract(self.t / 'k', xci, None, tmp, self.logs, 'v0', APP, None))++ with mock.patch.object(nsp, 'PROGRAM_INDEX', 1):++ self.assertFalse(nsp.nxb_extract(self.t / 'k', pfs0(self.t / 'g.nsp', ['a.cnmt.nca']), None, tmp, self.logs, 'v0', APP + 1, None))++ self.assertIn('not supported yet', self.text())++ self.assertFalse((self.t / 'ran').exists(), 'nxb-extract is not run for them')++++ def test_an_update_is_extracted_alone(self):++ tool(self.bin, 'nxb-extract', f"Path({str(self.t / 'args')!r}).write_text(' '.join(sys.argv[1:])); sys.exit(4)")++ base, upd = pfs0(self.t / 'b.nsp', ['a.cnmt.nca']), pfs0(self.t / 'u.nsp', ['u.cnmt.nca'])++ tmp = self.t / 'v1.partial'++ self.assertFalse(nsp.nxb_extract(self.t / 'k', base, upd, tmp, self.logs, 'v1', APP, None))++ args = (self.t / 'args').read_text()++ self.assertIn(f'--update {upd} {tmp} --updates-only', args)++ self.assertFalse(tmp.with_name('v1.partial.nxb-base').exists(), 'the base\'s scratch folder goes')++++ def split(self, rows):++ tool(self.bin, 'nxb-split', "a = sys.argv; o = lambda k: Path(a[a.index(k) + 1])\n"++ "o('--pkg').mkdir(parents=True)\n"++ f"for k in ('--map', '--facts', '--pairs', '--pso-pairs'): o(k).write_text({rows!r})")++ game = self.t / 'game'; game.mkdir(exist_ok=True); (game / 'game.tsv').write_text('name\tx\n')++ for n in ('shader_map.tsv', 'shader_facts.tsv', 'shader_pairs.tsv'): (game / n).write_text('# header of the profile\nrow\t1\n')++ g = mock.Mock(dir=game, table=lambda n: game / n)++ d = self.t / 'extract' / 'v0'++ tmp = self.t / 'pkg.partial'; (tmp / 'tables').mkdir(parents=True, exist_ok=True)++ return nsp.nxb_split([d / 'RomFS.img'], [d / 'ExeFS' / 'main'], tmp, ('shader_map.tsv', 'shader_facts.tsv', 'shader_pairs.tsv'), self.logs, g), tmp++++ def test_split_is_taken_when_its_tables_are_the_profiles(self):++ text, tmp = self.split('# another header\nrow\t1\n')++ self.assertEqual(text, 'shaders: 1 (nxb-split)\n')++ self.assertTrue((tmp / 'pkg').is_dir())++++ def test_split_with_other_tables_runs_the_standard_way(self):++ text, tmp = self.split('row\t2\n')++ self.assertIsNone(text)++ self.assertEqual(sorted(p.name for p in tmp.iterdir()), ['tables'], 'shader_split.py starts from an empty folder')++ self.assertEqual(list((tmp / 'tables').iterdir()), [])++ self.assertIn('differs from the profile\'s tables', self.text())++++++if __name__ == '__main__':++ unittest.main()+diff --git a/docs/env_switches.tsv b/docs/env_switches.tsv+index 772479e..aceb18e 100644+--- a/docs/env_switches.tsv++++ b/docs/env_switches.tsv+@@ -36,6 +36,7 @@ DM_ENV_FILE path `=<file>` makes `build_ipa.sh` bake its `KEY=VALUE` lines into+ DM_ENV_FILE_KEYS switch Set by the iOS app and the Mac launcher at launch to the space-separated names an env file set; the settings screen keeps those values and shows them greyed out+ DM_ENV_LAUNCH_KEYS switch Set at launch by the Mac launcher (the launcher settings already in the environment) and the iOS app (every DM_* variable already in it) to the space-separated names; the settings screen keeps those values and shows them greyed out+ DM_EXEC_OUT path File `oe::ExecuteProgram` records the next program and its parameters in, then the app ends (the iOS app of a game of several programs sets it to `Documents/program.bin`)++DM_EXPERIMENTAL_BUILDER switch `1` makes the NSP script run the extraction and the shader split through nxbuild's binaries beside `NXB_BIN` (the path of its `nxb` binary) where nxbuild supports the case, with the same outputs; any other case, an unsupported stage or an output that fails the checks runs the standard way (the Mac launcher's Use experimental builder setting)+ DM_FD_LOG diagnostic `=<seconds>` logs, every that many seconds, the number and kinds of file descriptors the process has open+ DM_FIXED_TIME switch `=<seconds>`: the user clock always returns that time and the host time zone is pinned to UTC, for deterministic comparison+ DM_FOCUS_KEEP switch `1` keeps the game in focus while its Mac window is up, whichever app is active (minimizing still counts); `scripts/regress.py` sets it, so another app taking the key window during a run does not pause a game that stops out of focus+diff --git a/scripts/build_ipa_from_nsp.py b/scripts/build_ipa_from_nsp.py+index d7ebdb5..33ebbe9 100755+--- a/scripts/build_ipa_from_nsp.py++++ b/scripts/build_ipa_from_nsp.py+@@ -33,6 +33,7 @@+ # unless ALLOW_OTHER_METAL / --allow-other-metal. Issue #242 reported corruption with another compiler; the cause is still unconfirmed.+ # Python 3.12+ is required for bounded archive mappings inside images. The Python running this file needs lz4, capstone 5.x and fonttools (and backports.zstd before Python 3.14); also needed: Xcode, xcodegen, .NET SDK 10, zstd. SPIRV-Cross is the in-repo build. A first run takes 2 (Metroid Dread) to 10 minutes (Breath of the Wild: 5 of them the shader split), longer when shader libraries must be compiled; a repeat run with unchanged inputs takes about 20 s (Metroid Dread), mostly packing the IPA. Key values are never printed.+ # The console shows one line per stage ([k/n] stage, with a bar, done/total, percent, elapsed time and ETA where the work can be counted); the tools' own output goes only to logs/ (PROGRESS / --progress). --progress json writes newline-delimited JSON events to stdout.++# DM_EXPERIMENTAL_BUILDER=1 with NXB_BIN=<path of nxbuild's nxb binary>: the extraction and the shader split run through nxbuild's binaries where it supports the case, with the same outputs and events; anything else runs as always (nxb_tool).+ import ast, collections, contextlib, ctypes, errno, getpass, pwd, hashlib, importlib.util, json, os, re, shlex, shutil, struct, subprocess, sys, tempfile, threading, time, types+ from pathlib import Path+ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "dm/tools"))+@@ -422,6 +423,102 @@ def free_gb(p):+ return shutil.disk_usage(p).free / 2**30++++# ---- The experimental builder (nxbuild) ----++# DM_EXPERIMENTAL_BUILDER=1 with NXB_BIN=<the nxb binary, or its folder>: the stages nxbuild reproduces byte for byte run through its binaries beside nxb (the extraction:++# nxb-extract; the shader split: nxb-split). This script stays the driver: the stage names, the events, the work folder and the reuse keys are the same. A stage nxbuild does++# not support (exit code NXB_UNSUPPORTED, or any other failure), a case it is not given (XCI, NSZ, merged packages, DLC, captures, a provisional or derived profile) and an++# output that fails the checks below run the standard way, with a note in the log. Without the variable nothing changes.++NXB_UNSUPPORTED = 4 # nxbuild's exit code for "not supported yet": nothing it wrote is an output (nxbuild's docs/integration-requests.md)++_nxb_noted = set()++++++def nxb_note(msg): # one note per message and build: a missing binary is named once, not per version++ if msg not in _nxb_noted: _nxb_noted.add(msg); say(f'experimental builder: {msg}')++++++def nxb_tool(name, env=None):++ """The nxbuild binary name (nxb-extract, nxb-split) beside NXB_BIN when the experimental builder is on, else None"""++ env = os.environ if env is None else env++ if env.get('DM_EXPERIMENTAL_BUILDER') != '1': return None++ where = Path(os.path.expanduser(env.get('NXB_BIN', '')))++ if not env.get('NXB_BIN') or not where.exists(): nxb_note(f'NXB_BIN does not name the nxb binary ({env.get("NXB_BIN", "")!r}): the standard builder runs every stage'); return None++ tool = (where if where.is_dir() else where.parent) / name++ if tool.is_file() and os.access(tool, os.X_OK): return tool++ nxb_note(f'{tool} is missing: the standard builder runs this stage'); return None++++++def nxb_run(cmd, log, what):++ """Run an nxbuild binary with its output in log only (its [RED] lines are not the build's: a stage it cannot do runs the standard way). True when it exited 0"""++ log.parent.mkdir(parents=True, exist_ok=True)++ err = ''++ try:++ with open(log, 'w') as f: p = subprocess.Popen([str(c) for c in cmd], stdout=f, stderr=subprocess.STDOUT)++ try: code = p.wait()++ except KeyboardInterrupt: p.terminate(); p.wait(); raise++ except OSError as e: code, err = None, str(e)++ if code == 0: return True++ try: unsupported = code == NXB_UNSUPPORTED or 'not supported yet' in log.read_text(errors='replace')++ except OSError: unsupported = False++ why = f'could not start ({err})' if code is None else 'not supported yet' if unsupported else f'exit code {code}'++ nxb_note(f'{what}: {why} (see {log}), the standard builder runs it')++ return False++++++def nxb_extract(keys, base, upd, tmp, logs, name, app, marker):++ """The extraction of one version through nxb-extract into tmp (the layout nx_extract writes: ExeFS/, RomFS.img, control.nacp, icon.jpg). True when its output passed the checks;++ False leaves no tmp, for nx_extract. Only plain NSPs of one title: an XCI, NSZ or merged package, another program or a damaged NSP go the standard way"""++ tool = nxb_tool('nxb-extract')++ if not tool: return False++ def plain(p):++ try: return p.suffix.lower() == '.nsp' and is_package(p) and not is_merged(p) and not any(n.lower().endswith('.ncz') for n in package_names(p))++ except (OSError, ValueError, struct.error, BuildError): return False++ if PROGRAM_INDEX or ALLOW_DAMAGED or not all(plain(p) for p in (base, upd) if p):++ nxb_note(f'extract {name}: an XCI, NSZ or merged package, --program or --allow-damaged is not supported yet, the standard builder runs it'); return False++ log, scratch = logs / f'nxb_extract_{name}.log', tmp.with_name(tmp.name + '.nxb-base')++ rmtree(tmp); rmtree(scratch)++ cmd = [tool, '--keys', keys, '--base', base] + (['--out', scratch, '--update', upd, tmp, '--updates-only'] if upd and upd != base else ['--out', tmp])++ ok = nxb_run(cmd, log, f'extract {name}')++ rmtree(scratch)++ if ok:++ text = log.read_text(errors='replace')++ title = re.search(r'^title ([0-9a-f]{16})', text, re.M)++ try: whole = (tmp / 'ExeFS' / 'main').is_file() and (tmp / 'RomFS.img').is_file() and (not marker or (rfs.source(tmp) / marker).exists())++ except (OSError, ValueError, struct.error): whole = False++ if whole and title and int(title[1], 16) == app - PROGRAM_INDEX:++ shutil.copyfile(log, logs / f'extract_{name}.log') # the next build's size estimate reads the summary lines there, as nx_extract writes them++ say(f'experimental builder: extracted {name} with nxb-extract')++ return True++ nxb_note(f'extract {name}: the output of nxb-extract failed the checks (see {log}), the standard builder runs it')++ rmtree(tmp)++ return False++++++def nxb_split(roms, mains, tmp, names, logs, g):++ """The shader split through nxb-split into tmp/pkg and tmp/tables (shader_split.py's outputs). Returns the text the caller reads from shader_split.py's log (its 'shaders: N' line),++ or None for shader_split.py, with tmp emptied again. Taken only when the tables it wrote have the data rows of the profile's (pso_pairs.tsv too where the profile has one): a reader++ nxbuild lacks shows there. The caller passes only builds with a registered profile and no DLC, captures, NROs or decoder"""++ tool = nxb_tool('nxb-split')++ if not tool: return None++ dirs = [Path(m).parent.parent for m in mains]++ if any(Path(r) != d / 'RomFS.img' or Path(m) != d / 'ExeFS' / 'main' for r, m, d in zip(roms, mains, dirs)) or not (g.dir / 'game.tsv').is_file():++ nxb_note('shader split: the inputs are not RomFS images of extractions, the standard builder runs it'); return None++ log, t = logs / 'nxb_split.log', tmp / 'tables'++ ok = nxb_run([tool, '--game', g.dir / 'game.tsv', '--pkg', tmp / 'pkg', '--map', t / names[0], '--facts', t / names[1], '--pairs', t / names[2], '--pso-pairs', t / 'pso_pairs.tsv', *dirs],++ log, 'shader split')++ if ok:++ bad = [ln for ln in log.read_text(errors='replace').splitlines() if ln.startswith('[RED]')]++ check = names + (('pso_pairs.tsv',) if g.table('pso_pairs.tsv').is_file() else ())++ try: same = (tmp / 'pkg').is_dir() and (t / 'pso_pairs.tsv').is_file() and all(data_rows((t / n).read_bytes()) == data_rows(g.table(n).read_bytes()) for n in check)++ except OSError: same = False++ if not bad and same:++ rows = [ln for ln in (t / names[0]).read_text(errors='replace').splitlines() if ln.strip() and not ln.startswith(('#', 'package\t'))]++ say('experimental builder: split the shaders with nxb-split')++ return f'shaders: {len(rows)} (nxb-split)\n'++ nxb_note('shader split: the output of nxb-split ' + ('could not read every file' if bad else 'differs from the profile\'s tables') + f' (see {log}), the standard builder runs it')++ rmtree(tmp); t.mkdir(parents=True)++ return None+++++ def configure_dotnet(env):+ """Let standalone .NET apphosts find the runtime used by dotnet (Homebrew installs it outside the registered locations)."""+ if env.get('DOTNET_ROOT'): return+@@ -1183,7 +1280,8 @@ def extract(nx, keys, ex, name, base, upd, cmp_dir, logs, g, app): # the build+ def poll():+ n, b = tree_size(tmp); return {'done': b, 'note': f'{n:,} files'}+ PROG.stage(f'extract {name}', poll=poll, unit='B', total=last[1] if last else base.stat().st_size + (upd.stat().st_size if upd and upd != base else 0), est=not last)+- run([nx, keys, base, upd or '-', tmp, '--romfs-image'] + extract_flags(app), log)++ if not nxb_extract(keys, base, upd, tmp, logs, name, app, marker): # the experimental builder, when it is on and can (else nothing happens there)++ run([nx, keys, base, upd or '-', tmp, '--romfs-image'] + extract_flags(app), log)+ swap_in(tmp, dst)+ n, b = extracted(log) or (0, 0)+ PROG.end(f'{n:,} files, {amount(b, "B")}')+@@ -2107,9 +2205,17 @@ def split_shaders(py, repo, roms, mains, pkg, env, logs, g, provisional=False, o+ tmp = pkg.with_name(pkg.name + '.partial'); rmtree(tmp); (tmp / 'tables').mkdir(parents=True)+ def counted(ln): # shader_split.py's 'progress: <done>/<total>'+ if m := re.fullmatch(r'progress: (\d+)/(\d+)', ln): PROG.set(done=int(m[1]), total=int(m[2]), unit='files')+- run([py, repo / 'dm/gfx/tools/shader_split.py', ','.join(map(str, roms)), ','.join(map(str, mains)), tmp / 'pkg'] + [tmp / 'tables' / n for n in names + ('pso_pairs.tsv',)] + ([','.join(map(str, dlc_roms))] if dlc_roms else []),+- logs / 'shader_split.log', env=env, on_line=counted)+- text = (logs / 'shader_split.log').read_text(errors='replace')++ # the experimental builder takes a registered profile's split of the game files alone; DLC, run-time captures, NROs and a game decoder stay with shader_split.py++ captured = env.get('DM_NRO_DIR') or any(env.get(s) or next((Path(r).parent / f).glob(p), None) for r in roms for s, f, p in++ (('DM_GLSLC_CAPTURE', 'glslc_capture', '*.glslc'), ('DM_UE_SHADERS', 'ue_shaders', '*.ushaderbytecode')))++ plain = not (provisional or oodle or dlc_roms or captured)++ if not plain and os.environ.get('DM_EXPERIMENTAL_BUILDER') == '1':++ nxb_note('shader split: a provisional or derived profile, DLC, run-time captures or a game decoder is not supported yet, the standard builder runs it')++ text = nxb_split(roms, mains, tmp, names, logs, g) if plain else None++ if text is None:++ run([py, repo / 'dm/gfx/tools/shader_split.py', ','.join(map(str, roms)), ','.join(map(str, mains)), tmp / 'pkg'] + [tmp / 'tables' / n for n in names + ('pso_pairs.tsv',)] + ([','.join(map(str, dlc_roms))] if dlc_roms else []),++ logs / 'shader_split.log', env=env, on_line=counted)++ text = (logs / 'shader_split.log').read_text(errors='replace')+ m = re.search(r'shaders: (\d+) \(', text)+ bad = re.search(r'^\[RED\] (\d+) file\(s\) could not be read.*$', text, re.M) # the split leaves out a file no reader reads (its own red line per file in the log) and goes on+ if bad and not provisional: # the profile's tables were split from every file: one that does not read now is a reader changediff --git a/patches/series b/patches/seriesindex dcf51b4e..cc46df43 100644--- a/patches/series+++ b/patches/series@@ -40,3 +40,4 @@0070-ios-touch-layout.patch0071-ios-shell-restore-native-duo-orientation.patch0072-nsp-script-the-first-dotnet-with-an-sdk-10-goes.patch+0073-nsp-script-hand-the-extraction-and-the-shader-sp.patch