Skip to content

Play your own Switch games on iPhone/iPad — one-click macOS app, no terminal. | 把 Switch 游戏装进 iPhone:图形界面一键构建,不用命令行。Bring your own game.

Release carries the engine tree again, signed latest.json, experimental channel

v0lumehi1a487bdeeb15
diff --git a/.github/workflows/build-mac-app.yml b/.github/workflows/build-mac-app.yml
index 610350c3..4d833422 100644
--- a/.github/workflows/build-mac-app.yml
+++ b/.github/workflows/build-mac-app.yml
@@ -8,7 +8,12 @@
# "up to date" — a silent failure of the update path for every existing install. Copies from before
# the app was renamed (NS Port Builder) look for NS-Port-Builder-macOS-arm64.zip, so that one goes up too.
#
-# The app carries no engine (issue #214): the launcher runs `tools/engine.py sync` on the first start and after updates, from GitHub. It carries its build tools (zstd, glslangValidator, spirv-cross, xcodegen) and a Python with the packages the build scripts import,
+# The app carries the engine (issues #214 and #227): `tools/engine.py tree` builds the engine tree (seedds/nxnative at the pin plus the patch
+# series, no git history) into Contents/Resources/engine, and the launcher copies it to Application Support on first use. The same tree goes up
+# as a release asset (NX2Apple-engine.zip) together with latest.json and its Ed25519 signature latest.json.sig (tools/release_manifest.py), which
+# is what the launcher reads to update the engine without an app release. The signing key is the repository secret ENGINE_SIGNING_KEY: a tag build
+# fails at its first step when the secret is missing or is not the key in frontend/engine-signing.pub (compiled into the launcher). Other builds
+# run the same steps as a dry run with a throwaway key and upload nothing. It also carries its build tools (zstd, glslangValidator, spirv-cross, xcodegen) and a Python with the packages the build scripts import,
# so a player installs neither Homebrew nor Python (frontend/tools/bundle_tools.sh builds them from pinned, checksummed sources; check_bundle.sh
# runs each from the assembled bundle with a clean PATH, checks the load paths and the signature).
#
@@ -21,7 +26,7 @@ on:
push:
tags: ['v*']
pull_request:
- paths: ['frontend/**', 'tools/engine.py', '.github/workflows/build-mac-app.yml']
+ paths: ['frontend/**', 'tools/engine.py', 'tools/release_manifest.py', 'tools/requirements-release.txt', 'patches/**', '.upstream-ref', '.github/workflows/build-mac-app.yml']
workflow_dispatch:
permissions:
@@ -33,6 +38,30 @@ jobs:
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+ - uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
+ with:
+ python-version: '3.14'
+ - name: Install the signing tool's package
+ run: python -m pip install -r tools/requirements-release.txt
+
+ - name: Signing key (a release without a matching key must not ship)
+ if: startsWith(github.ref, 'refs/tags/')
+ env:
+ ENGINE_SIGNING_KEY: ${{ secrets.ENGINE_SIGNING_KEY }}
+ run: |
+ # Fails when the secret is missing, or when its public key is not the one in frontend/engine-signing.pub, the value the launcher
+ # compiles in (EngineManifest.verify): a manifest signed with any other key would be rejected by every installed app.
+ python tools/release_manifest.py check-key --pub frontend/engine-signing.pub
+
+ - name: Engine tree (tools/engine.py tree)
+ shell: bash
+ run: |
+ set -euo pipefail
+ python tools/engine.py sync
+ # The tree as it goes into the app, and its zip, byte for byte the same on every run (the SHA-256 in latest.json).
+ python tools/engine.py tree --out "$RUNNER_TEMP/engine-tree" --zip "$RUNNER_TEMP/engine-out/NX2Apple-engine.zip"
+ du -sh "$RUNNER_TEMP/engine-tree" "$RUNNER_TEMP/engine-out/NX2Apple-engine.zip"
+
- name: Version from the tag
id: ver
shell: bash
@@ -73,6 +102,13 @@ jobs:
# The build tools and the Python go into Resources before the bundle is signed (bundle_tools.sh signs each nested Mach-O file
# itself: --deep below does not reach code that sits in Resources).
tools/bundle_tools.sh "$APP/Contents/Resources" "$RUNNER_TEMP/bundle-tools-work"
+ # The engine tree (Contents/ is read-only once signed: the launcher copies it to ~/Library/Application Support/NX2Apple/ on first
+ # use). Its Mach-O files (the shader translator, spirv-cross) are signed on their own like the bundled tools: --deep does not reach
+ # code in Resources. The zip asset keeps them as upstream built them.
+ ditto "$RUNNER_TEMP/engine-tree" "$APP/Contents/Resources/engine"
+ while IFS= read -r -d '' f; do
+ if file -b "$f" | grep -q 'Mach-O'; then codesign --force --sign - "$f" 2> /dev/null; fi
+ done < <(find "$APP/Contents/Resources/engine" -type f -perm -u+x -print0)
V="${{ steps.ver.outputs.version }}"
cat > "$APP/Contents/Info.plist" <<PLIST
<?xml version="1.0" encoding="UTF-8"?>
@@ -108,6 +144,13 @@ jobs:
# lz4, capstone, fontTools, zbic (system 22.0.0+ modules) and nsz, so a release without one of them fails here; no Mach-O file links into /opt/homebrew; the signature verifies.
tools/check_bundle.sh NX2Apple.app
codesign -v --deep --strict NX2Apple.app
+ # The engine tree inside the bundle is the one the zip asset holds: same identity, the files the launcher runs by path.
+ E="NX2Apple.app/Contents/Resources/engine"
+ cmp "$E/.engine-id" "$RUNNER_TEMP/engine-tree/.engine-id"
+ cmp "$E/.upstream-ref" ../.upstream-ref
+ test -f "$E/scripts/build_ipa_from_nsp.py" && test -f "$E/dm/tools/romfs_image.py" && test -f "$E/dm/fork_patches.txt"
+ test ! -e "$E/.git"
+ echo "engine in the bundle: $(cat "$E/.engine-id")"
- name: Package
working-directory: frontend
@@ -153,3 +196,35 @@ jobs:
frontend/NX2Apple-macOS-arm64.zip \
frontend/nxnative-macOS-arm64.zip \
frontend/NS-Port-Builder-macOS-arm64.zip --clobber
+
+ - name: Publish the engine (NX2Apple-engine.zip, latest.json, latest.json.sig)
+ if: startsWith(github.ref, 'refs/tags/')
+ shell: bash
+ env:
+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ ENGINE_SIGNING_KEY: ${{ secrets.ENGINE_SIGNING_KEY }}
+ run: |
+ # latest.json: version, pin, engine_id, the engine and app assets with SHA-256 and size, sources (this release's download URL),
+ # built, channel stable. latest.json.sig: the Ed25519 signature over its exact bytes. The zip goes first, the signature last.
+ python tools/release_manifest.py publish \
+ --engine-zip "$RUNNER_TEMP/engine-out/NX2Apple-engine.zip" --app-zip frontend/NX2Apple-macOS-arm64.zip \
+ --version "${{ steps.ver.outputs.version }}" --channel stable --tag "$GITHUB_REF_NAME" \
+ --out "$RUNNER_TEMP/publish" --upload
+
+ - name: Engine publishing dry run (a throwaway key, nothing uploaded)
+ if: ${{ !startsWith(github.ref, 'refs/tags/') }}
+ shell: bash
+ run: |
+ python tools/release_manifest.py publish \
+ --engine-zip "$RUNNER_TEMP/engine-out/NX2Apple-engine.zip" --app-zip frontend/NX2Apple-macOS-arm64.zip \
+ --version "${{ steps.ver.outputs.version }}" --channel stable --tag "v${{ steps.ver.outputs.version }}" \
+ --out "$RUNNER_TEMP/publish" --ephemeral-key
+ cat "$RUNNER_TEMP/publish/latest.json"
+ - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
+ if: ${{ !startsWith(github.ref, 'refs/tags/') }}
+ with:
+ name: engine-latest-json-dry-run
+ path: |
+ ${{ runner.temp }}/publish/latest.json
+ ${{ runner.temp }}/publish/latest.json.sig
+ if-no-files-found: error
diff --git a/.github/workflows/sync-upstream.yml b/.github/workflows/sync-upstream.yml
index f73a9ce3..09c1f6d3 100644
--- a/.github/workflows/sync-upstream.yml
+++ b/.github/workflows/sync-upstream.yml
@@ -10,12 +10,24 @@
# A PR opened with the default GITHUB_TOKEN does not start other workflows (GitHub's rule), so this job
# runs the checks itself before it opens the PR. A repository secret SYNC_TOKEN (a fine-grained token
# with contents, pull requests and issues write) makes the PR start tests.yml as well.
+#
+# The experimental engine channel (LNPANDY/NX2Apple#227): after the sync job passed its checks (the series applies, the tests, the host build)
+# the publish-experimental job builds the engine tree of the bumped branch (tools/engine.py tree), writes latest.json with channel
+# "experimental" and "built", signs it with the repository secret ENGINE_SIGNING_KEY (tools/release_manifest.py) and replaces the assets
+# (NX2Apple-engine.zip, latest.json, latest.json.sig) of the pre-release tag engine-experimental, which it creates when missing and keeps a
+# pre-release that is never "latest". A workflow artifact cannot serve: the launcher has no token. It only publishes after a green sync job,
+# and its failure (a missing or wrong key) never opens the "sync blocked" issue. `force_experimental` publishes main's engine by hand.
name: Sync upstream
on:
schedule:
- cron: '30 21 * * *' # 21:30 UTC = 5:30 Beijing
workflow_dispatch:
+ inputs:
+ force_experimental:
+ description: "Also publish the engine of main to the engine-experimental tag (the first time, or after the tag was removed)"
+ type: boolean
+ default: false
permissions:
contents: write
@@ -25,6 +37,8 @@ permissions:
jobs:
sync:
runs-on: macos-latest # the host build needs clang + macOS headers
+ outputs:
+ branch: ${{ steps.pr.outputs.branch }} # the pushed sync branch, empty when nothing changed
env:
GH_TOKEN: ${{ secrets.SYNC_TOKEN || secrets.GITHUB_TOKEN }}
steps:
@@ -64,6 +78,7 @@ jobs:
make -C engine/dm/host GAME=$GAME dmhost -j4 2>&1 | tail -20
- name: Commit to a sync branch and open a pull request
+ id: pr
if: steps.bump.outputs.changed == 'yes'
# main is protected (a pull request is required, force pushes and deletions are
# blocked, and the rules apply to everyone including the owner and this bot), so the
@@ -79,6 +94,7 @@ jobs:
git checkout -b "sync/upstream-$SHORT"
git commit -m "Sync upstream $SHORT"
git push -u origin "sync/upstream-$SHORT"
+ echo "branch=sync/upstream-$SHORT" >> "$GITHUB_OUTPUT"
{
echo "The patch queue re-applied on upstream \`${NEW:0:12}\` (was \`${OLD:0:12}\`): every patch applied, and \`tools/engine.py check --tree\`, the tests and a macOS host build passed. Merging brings seedds/nxnative to \`$NEW\`."
echo
@@ -113,3 +129,50 @@ jobs:
else
gh issue create -R "${{ github.repository }}" --title "$TITLE" --body-file issue.md || echo "issue creation failed; the log is in the workflow run"
fi
+
+ publish-experimental:
+ # Only after a green sync job (its checks are the "CI is green" of this channel). Separate from it so that a publishing problem
+ # (no key, no release permission) does not read as a blocked sync above.
+ needs: sync
+ if: needs.sync.result == 'success' && (needs.sync.outputs.branch != '' || inputs.force_experimental)
+ runs-on: ubuntu-latest
+ timeout-minutes: 20
+ concurrency: engine-experimental # two runs must not replace the assets at the same time
+ permissions:
+ contents: write
+ actions: read
+ env:
+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ ENGINE_SIGNING_KEY: ${{ secrets.ENGINE_SIGNING_KEY }}
+ steps:
+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+ with:
+ ref: ${{ needs.sync.outputs.branch || github.event.repository.default_branch }}
+ persist-credentials: false
+ - uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
+ with:
+ python-version: '3.14'
+ - run: python -m pip install -r tools/requirements-release.txt
+
+ - name: Signing key (fails when the secret is missing or is not the key in frontend/engine-signing.pub)
+ run: python tools/release_manifest.py check-key --pub frontend/engine-signing.pub
+
+ - name: "Main's tests must be green (a bump branch was just checked, a hand-started publish of main was not)"
+ if: needs.sync.outputs.branch == ''
+ run: |
+ c=$(gh run list -R "${{ github.repository }}" --workflow tests.yml --branch "${{ github.event.repository.default_branch }}" --event push --limit 1 --json conclusion --jq '.[0].conclusion')
+ echo "latest tests.yml on main: ${c:-none}"
+ [ "$c" = "success" ] || { echo "::error::tests.yml on main is not green: not publishing"; exit 1; }
+
+ - name: Build the engine tree
+ run: |
+ python tools/engine.py sync
+ python tools/engine.py tree --out "$RUNNER_TEMP/engine-tree" --zip "$RUNNER_TEMP/engine-out/NX2Apple-engine.zip"
+
+ - name: Publish to the engine-experimental pre-release
+ run: |
+ APPV=$(sed -n 's/.*static let appVersion = "\(.*\)".*/\1/p' frontend/Sources/NSPortBuilder/ContentView.swift)
+ python tools/release_manifest.py publish \
+ --engine-zip "$RUNNER_TEMP/engine-out/NX2Apple-engine.zip" \
+ --version "$APPV-experimental" --channel experimental --tag engine-experimental \
+ --out "$RUNNER_TEMP/publish" --upload
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index c08f3f18..b7ba7f7d 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -160,8 +160,10 @@ jobs:
}
- name: Install ripgrep (the engine/ search test)
run: sudo apt-get install -y -qq ripgrep > /dev/null
- - name: Patch tool tests
- run: python -m unittest -v tools/tests/test_patches.py tools/tests/test_engine.py
+ - name: Install the signing tool's package (tools/release_manifest.py)
+ run: python -m pip install -r tools/requirements-release.txt
+ - name: Patch tool and release tool tests
+ run: python -m unittest -v tools/tests/test_patches.py tools/tests/test_engine.py tools/tests/test_release_manifest.py
- name: Dry bump to seedds main (report only)
run: python tools/engine.py bump --dry --to main || true
diff --git a/frontend-qt/nspb_qt/engine.py b/frontend-qt/nspb_qt/engine.py
index d3ddeab6..0d530534 100644
--- a/frontend-qt/nspb_qt/engine.py
+++ b/frontend-qt/nspb_qt/engine.py
@@ -2,8 +2,8 @@
Since #214 the engine is built into `<checkout>/engine/` (seedds/nxnative at `.upstream-ref` plus `patches/`). Until the
committed copy goes (step 2), a checkout has both, and one from before the switch has only the committed copy. The rule, the
-same as the Mac launcher's (frontend/Sources/NSPortBuilder/Engine.swift). A release carries no engine: the sync downloads it from
-GitHub on the first start and after updates, and a failed one becomes a failure row (`failure_kind`):
+same as the Mac launcher's (frontend/Sources/NSPortBuilder/Engine.swift). The sync is for checkouts (a release carries the engine tree
+instead, #227): it downloads the pinned commit from GitHub, and a failed one becomes a failure row (`failure_kind`):
- engine/ when it is synced and current,
- else the committed tree (dm/, scripts/ at the checkout's root) when the checkout has one,
diff --git a/frontend/Sources/NSPortBuilder/BuildFailureView.swift b/frontend/Sources/NSPortBuilder/BuildFailureView.swift
index 4ca1e3c0..c228e1d5 100644
--- a/frontend/Sources/NSPortBuilder/BuildFailureView.swift
+++ b/frontend/Sources/NSPortBuilder/BuildFailureView.swift
@@ -82,7 +82,7 @@ struct BuildFailureCard: View {
}
}
-/// Above every page of the launcher while the engine is fetched and patched (tools/engine.py sync, run on the first start and after
+/// Above every page of the launcher while the engine is fetched and patched (tools/engine.py sync, run for a checkout at launch and after
/// updates), and its failure row when that stops: what happened, what to do, and Try again.
struct EngineSyncBanner: View {
@ObservedObject var m: BuildModel
diff --git a/frontend/Sources/NSPortBuilder/ContentView.swift b/frontend/Sources/NSPortBuilder/ContentView.swift
index c804cb77..89daa93f 100644
--- a/frontend/Sources/NSPortBuilder/ContentView.swift
+++ b/frontend/Sources/NSPortBuilder/ContentView.swift
@@ -629,7 +629,7 @@ final class BuildModel: ObservableObject {
@Published var engineFailure: BuildFailure?
/// `tools/engine.py sync` when engine/ is stale (#214, Engine.swift): at launch, after a clone, a pull or an adoption, and before a
- /// build that has no other engine. A release app carries no engine, so the first start downloads it here. A no-op for a checkout
+ /// build that has no other engine. Checkouts only: a release app carries its engine tree (#227) and never syncs. A no-op for a checkout
/// from before #214 or a current engine/. `then(ok)` runs on the main queue.
func syncEngine(then: ((Bool) -> Void)? = nil) {
if engineSyncWaiting != nil { engineSyncAgain = true; if let then { engineSyncWaiting?.append(then) }; return }
diff --git a/frontend/Sources/NSPortBuilder/Engine.swift b/frontend/Sources/NSPortBuilder/Engine.swift
index bbeae806..f9914505 100644
--- a/frontend/Sources/NSPortBuilder/Engine.swift
+++ b/frontend/Sources/NSPortBuilder/Engine.swift
@@ -12,8 +12,10 @@ import Foundation
/// - else `engine/` as it is (stale or missing: the build syncs it first).
///
/// `stampKey` is tools/engine.py's digest of the pin and the series, compared with the stamp sync wrote into engine/.git, so the
-/// launcher tells a stale engine without starting Python. A release app carries no engine: the launcher runs the sync on the first
-/// start and after updates, which downloads it from GitHub (about 80 MB), and shows a failure row when that does not work.
+/// launcher tells a stale engine without starting Python. The sync is for a repository checkout (a developer's,
+/// or one a launcher cloned): it downloads the pinned engine from GitHub (about 80 MB) and shows a failure row when that does not work.
+/// A release app carries the engine tree instead (`Contents/Resources/engine`, built by `tools/engine.py tree`, #227) and takes updates
+/// from the release's `latest.json`: no git, no sync.
enum EngineLayout {
/// The file that tells an engine tree (BuildModel.isRepo always looked for it).
static let marker = "dm/tools/nx_extract/Program.cs"
diff --git a/frontend/engine-signing.pub b/frontend/engine-signing.pub
new file mode 100644
index 00000000..72b5ab9b
--- /dev/null
+++ b/frontend/engine-signing.pub
@@ -0,0 +1,3 @@
+# The public half of the key that signs latest.json (Ed25519, base64 of the 32 raw bytes), one line below. Empty until the key exists.
+# The launcher (EngineManifest.verify's publicKey) compiles in the same value, and the release workflow fails when the repository
+# secret ENGINE_SIGNING_KEY does not belong to it (tools/release_manifest.py check-key). Lines starting with # are comments.
diff --git a/patches/README.md b/patches/README.md
index 3462880f..69803617 100644
--- a/patches/README.md
+++ b/patches/README.md
@@ -107,6 +107,23 @@ leaves the conflict in a worktree.
The nightly workflow does the first step only: it opens a pull request on `sync/upstream` when every patch applies, and an issue
naming the blocking patch when one does not. Whoever owns the patch refreshes it.
+## The engine tree and how it is published (#227)
+
+Players get no git: a release carries the engine as a plain tree. `python3 tools/engine.py tree --out DIR [--zip FILE]` writes it from
+a synced `engine/` (`dm/`, `scripts/`, `docs/`, `ref/`, `AGENTS.md`, `LICENSE`, the patches applied, `dm/fork_patches.txt`, no history) plus
+`.upstream-ref` and `.engine-id`, the tree's identity `<pin>-<first 12 hex of the series digest>` (the series changes without the pin moving, so
+the pin alone does not name a tree). The zip is deterministic (sorted, fixed timestamps), so its SHA-256 is reproducible.
+
+`python3 tools/release_manifest.py publish ...` writes `latest.json` (version, pin, `engine_id`, channel, built, the engine and app assets with
+SHA-256 and size, sources) and `latest.json.sig` (a detached Ed25519 signature over its exact bytes, base64), and with `--upload` puts
+`NX2Apple-engine.zip`, `latest.json` and `latest.json.sig` on a GitHub release. Without `--upload` it is a dry run. The key is the repository
+secret `ENGINE_SIGNING_KEY` (base64 of the 32 raw private bytes), checked against `frontend/engine-signing.pub`, which the launcher compiles in
+(`python3 tools/release_manifest.py keygen` makes a pair). `build-mac-app.yml` does this for a release tag (stable channel, the tree also goes
+into `Contents/Resources/engine`), `sync-upstream.yml` for the pre-release tag `engine-experimental` after a green nightly bump.
+
+Locally: `python3 tools/engine.py sync`, then `tree --out /tmp/t --zip /tmp/e/NX2Apple-engine.zip`, then `publish --engine-zip /tmp/e/NX2Apple-engine.zip
+--version 0.0.0 --channel experimental --tag engine-experimental --out /tmp/p --ephemeral-key` runs every step except the upload.
+
## Patch policy
Upstream's compatibility is the baseline. A patch does not change it silently.
diff --git a/tools/engine.py b/tools/engine.py
index 71313782..bc5cd726 100755
--- a/tools/engine.py
+++ b/tools/engine.py
@@ -16,6 +16,7 @@ Subcommands (Python 3.9+, stdlib and git):
export [REV] turn commits made in engine/ (default: everything after the last sync) into patches with the
Upstream-Status / Reason / Issue header, and add them to patches/series
status the patches (Upstream-Status, Reason, age) and the state of engine/
+ tree --out DIR [--zip FILE] the engine tree for players, the release's engine (see below)
check [--tree] headers valid, no patch touches the shader chain or the generated files, the series applies
cleanly to the pin with no empty patch. --tree also compares the committed vendored tree with
pin + series, as long as the repository still commits it (step 1 of #214)
@@ -24,8 +25,14 @@ Subcommands (Python 3.9+, stdlib and git):
refresh WORKTREE --to SHA after a conflicting bump was resolved by hand in WORKTREE: its commits become the series
The upstream URL is https://github.com/seedds/nxnative.git, or $NX2APPLE_ENGINE_URL (a mirror, or a local clone that holds the
-pin, which needs no network). A release app carries no engine: the launcher runs `sync` on the first start and after updates.
+pin, which needs no network). `sync` is for developers, CI and checkouts: a release carries the engine tree (`tree`) instead.
Scratch worktrees go to $PATCHES_TMP (or --tmp), else the system's temporary folder.
+
+The engine tree (`tree`): what a player's launcher runs the build from, with no git. DIR gets the files of engine/ at its last sync
+(TREE_PATHS, the patches applied, dm/fork_patches.txt included), `.upstream-ref` and `.engine-id`, the engine's identity
+`<pin>-<first 12 hex of the series digest>`: the pin alone does not name a tree, since the series changes without the pin moving.
+The zip lists its entries sorted, with fixed timestamps and permissions (0755 or 0644), so the same tree gives the same bytes
+(for one zlib). tools/release_manifest.py publishes it.
"""
import argparse
import contextlib
@@ -34,6 +41,7 @@ import importlib
import io
import os
import re
+import shutil
import subprocess
import sys
import time
@@ -236,6 +244,97 @@ def cmd_hook(a):
return 0
+# ---- tree: the engine for players ----
+
+TREE_PATHS = ('dm', 'scripts', 'docs', 'ref', 'AGENTS.md', 'LICENSE') # what a player's build reads, of what engine/ tracks
+ENGINE_ID_FILE = '.engine-id'
+ZIP_TIME = (2020, 1, 1, 0, 0, 0) # fixed, so the zip's bytes depend on the content only
+
+
+def engine_id(root):
+ """`<pin>-<first 12 hex of the series digest>`: names a tree (the series changes without the pin moving). [0-9a-f-], 53 characters."""
+ return f'{pin(root)}-{stamp_key(root)[:12]}'
+
+
+def extract_tree(eng, out):
+ """The files of engine/'s HEAD under TREE_PATHS into `out`: `git archive`, so no .git and nothing untracked or ignored."""
+ import tarfile
+ names = [n for n in TREE_PATHS if git('cat-file', '-e', f'HEAD:{n}', cwd=eng, check=False).returncode == 0]
+ if not names: raise Fail(f'{eng} holds none of {", ".join(TREE_PATHS)}: not an engine checkout')
+ proc = subprocess.Popen(['git', 'archive', '--format=tar', 'HEAD', '--', *names], cwd=eng, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ count = 0
+ with tarfile.open(fileobj=proc.stdout, mode='r|') as tar:
+ for m in tar:
+ dest = out / m.name
+ if m.isdir(): dest.mkdir(parents=True, exist_ok=True)
+ elif m.isreg():
+ dest.parent.mkdir(parents=True, exist_ok=True)
+ with tar.extractfile(m) as src, open(dest, 'wb') as f: shutil.copyfileobj(src, f)
+ dest.chmod(0o755 if m.mode & 0o111 else 0o644)
+ count += 1
+ elif m.issym():
+ dest.parent.mkdir(parents=True, exist_ok=True)
+ os.symlink(m.linkname, dest)
+ count += 1
+ proc.stdout.close()
+ if proc.wait(): raise Fail(f'git archive failed: {proc.stderr.read().decode(errors="replace").strip()[-500:]}')
+ return count
+
+
+def write_zip(tree, zpath):
+ """Every file of `tree` (the tree itself is the archive's root), sorted by path, fixed time, 0755 or 0644, deflate level 6."""
+ import stat
+ import zipfile
+ entries = []
+ for dirpath, dirnames, files in os.walk(tree):
+ dirnames.sort()
+ for n in files + [d for d in dirnames if os.path.islink(os.path.join(dirpath, d))]:
+ full = os.path.join(dirpath, n)
+ entries.append((Path(full).relative_to(tree).as_posix(), full))
+ entries.sort()
+ zpath = Path(zpath)
+ zpath.parent.mkdir(parents=True, exist_ok=True)
+ tmp = zpath.with_name(zpath.name + '.part')
+ with zipfile.ZipFile(tmp, 'w', zipfile.ZIP_DEFLATED, compresslevel=6) as z:
+ for rel, full in entries:
+ zi = zipfile.ZipInfo(rel, ZIP_TIME)
+ zi.create_system = 3
+ if os.path.islink(full):
+ zi.external_attr = (stat.S_IFLNK | 0o755) << 16
+ z.writestr(zi, os.readlink(full))
+ continue
+ zi.compress_type = zipfile.ZIP_DEFLATED
+ zi.external_attr = (stat.S_IFREG | (0o755 if os.access(full, os.X_OK) else 0o644)) << 16
+ with open(full, 'rb') as f: z.writestr(zi, f.read(), compresslevel=6)
+ os.replace(tmp, zpath)
+ return len(entries)
+
+
+def cmd_tree(a):
+ root = a.root
+ eng = engine_dir(root)
+ reason = why_stale(root)
+ if reason: raise Fail(f'engine/ is not a current sync ({reason}): run `python3 tools/engine.py sync` first')
+ work = local_work(eng, read_stamp(eng))
+ if work: raise Fail('engine/ has changes the release must not carry: ' + '; '.join(work) + '. Export them as patches or sync --force.')
+ out = Path(a.out).resolve()
+ if out.exists() and any(out.iterdir()):
+ if not (out / ENGINE_ID_FILE).is_file(): raise Fail(f'{out} exists and is not an engine tree made by this command: choose an empty folder')
+ shutil.rmtree(out)
+ out.mkdir(parents=True, exist_ok=True)
+ n = extract_tree(eng, out)
+ eid = engine_id(root)
+ (out / '.upstream-ref').write_text(pin(root) + '\n')
+ (out / ENGINE_ID_FILE).write_text(eid + '\n')
+ if not (out / P.FORK_PATCHES).is_file(): raise Fail(f'{P.FORK_PATCHES} is missing from engine/: sync again')
+ say(f'engine tree: {eid} ({n} files) in {out}')
+ if a.zip:
+ k = write_zip(out, a.zip)
+ h = hashlib.sha256(Path(a.zip).read_bytes()).hexdigest()
+ say(f'engine zip: {a.zip} ({k} entries, {Path(a.zip).stat().st_size} bytes, sha256 {h})')
+ return 0
+
+
# ---- export ----
def cmd_export(a):
@@ -423,6 +522,7 @@ def main(argv=None):
s = sub.add_parser('hook'); s.set_defaults(f=cmd_hook, force=False, quiet=True)
s = sub.add_parser('export'); s.add_argument('rev', nargs='?', help='a commit or a range in engine/ (default: everything after the last sync)')
s.add_argument('--status', default='pending'); s.add_argument('--reason', default='TODO: why the fork carries this'); s.add_argument('--issue', default='TODO: LNPANDY/NX2Apple#<n>'); s.set_defaults(f=cmd_export)
+ s = sub.add_parser('tree'); s.add_argument('--out', required=True, help='the folder for the tree (emptied when it is an earlier tree)'); s.add_argument('--zip', help='also write the deterministic zip of the tree'); s.set_defaults(f=cmd_tree)
sub.add_parser('status').set_defaults(f=cmd_status)
s = sub.add_parser('check'); s.add_argument('--tree', action='store_true', help='also compare the committed vendored tree (step 1 of #214)'); s.set_defaults(f=cmd_check)
s = sub.add_parser('bump'); s.add_argument('--to', required=True, help='the new upstream commit, or a branch such as main'); s.add_argument('--dry', action='store_true'); s.set_defaults(f=cmd_bump)
diff --git a/tools/release_manifest.py b/tools/release_manifest.py
new file mode 100755
index 00000000..9904b784
--- /dev/null
+++ b/tools/release_manifest.py
@@ -0,0 +1,256 @@
+#!/usr/bin/env python3
+"""latest.json for the engine, signed (LNPANDY/NX2Apple#227): what the launcher reads to update the engine without an app release.
+
+The manifest has the shape of EngineManifest.Manifest (frontend/Sources/NSPortBuilder/EngineManifest.swift), plus `engine_id`:
+
+ {"version": "0.8.0", "pin": "<seedds commit>", "engine_id": "<pin>-<12 hex of the series digest>", "channel": "stable",
+ "built": "2026-10-31T02:00:00Z",
+ "engine": {"asset": "NX2Apple-engine.zip", "sha256": "<64 hex>", "size": 91553652},
+ "app": {"asset": "NX2Apple-macOS-arm64.zip", "sha256": "...", "size": 1234}, (stable only, optional)
+ "sources": ["https://github.com/LNPANDY/NX2Apple/releases/download/v0.8.0/"]}
+
+`pin` and `engine_id` come from the engine zip itself (its .upstream-ref and .engine-id, written by `tools/engine.py tree`), so the
+manifest cannot disagree with the asset it describes. The signature is a detached Ed25519 signature over the exact bytes of
+latest.json, base64, in latest.json.sig next to it (EngineManifest.verify(manifest:signature:publicKey:)).
+
+The private key is the repository secret ENGINE_SIGNING_KEY: base64 of the 32 raw bytes. Its public key must equal the one in
+frontend/engine-signing.pub, which the launcher compiles in. `check-key` fails when the secret is missing or belongs to another key,
+so a release can never ship with signing off or with a key the launcher does not trust.
+
+Subcommands (Python 3.9+, `pip install -r tools/requirements-release.txt`):
+ keygen print a new key pair (private base64 for the secret, public base64 for engine-signing.pub)
+ check-key [--pub FILE] exit 0 when $ENGINE_SIGNING_KEY is set and matches the public key file
+ publish --engine-zip Z --version V --channel stable|experimental --tag T --out DIR [--app-zip A] [--upload]
+ write latest.json and latest.json.sig (and copies of the zips) to DIR and verify them. With --upload,
+ put the zip, latest.json and latest.json.sig on the GitHub release TAG (gh). Without it nothing is
+ uploaded: that is the dry run. --ephemeral-key signs with a throwaway key (dry runs only, never uploads)
+ verify --manifest F --sig F [--pub FILE] check a signature the way the launcher does
+"""
+import argparse
+import base64
+import datetime
+import hashlib
+import json
+import os
+import re
+import shutil
+import subprocess
+import sys
+import zipfile
+from pathlib import Path
+
+REPO = Path(__file__).resolve().parent.parent
+PUB_FILE = REPO / 'frontend' / 'engine-signing.pub'
+KEY_ENV = 'ENGINE_SIGNING_KEY'
+MANIFEST = 'latest.json'
+SIGNATURE = 'latest.json.sig'
+GITHUB_REPO = 'LNPANDY/NX2Apple'
+EXPERIMENTAL_TAG = 'engine-experimental'
+ENGINE_ID = re.compile(r'[0-9a-f]{40}-[0-9a-f]{12}')
+
+
+class Fail(Exception):
+ pass
+
+
+def say(*a):
+ print(*a, flush=True)
+
+
+def ed25519():
+ try:
+ from cryptography.hazmat.primitives.asymmetric import ed25519
+ from cryptography.exceptions import InvalidSignature
+ except ImportError:
+ raise Fail('the Python package `cryptography` is missing: python3 -m pip install -r tools/requirements-release.txt')
+ return ed25519, InvalidSignature
+
+
+def raw32(b64, what):
+ try: data = base64.b64decode(b64.strip(), validate=True)
+ except Exception: raise Fail(f'{what} is not base64')
+ if len(data) != 32: raise Fail(f'{what} must be the base64 of 32 raw bytes (is {len(data)} bytes)')
+ return data
+
+
+def private_key(b64):
+ return ed25519()[0].Ed25519PrivateKey.from_private_bytes(raw32(b64, KEY_ENV))
+
+
+def public_b64(priv):
+ from cryptography.hazmat.primitives import serialization
+ raw = priv.public_key().public_bytes(serialization.Encoding.Raw, serialization.PublicFormat.Raw)
+ return base64.b64encode(raw).decode()
+
+
+def read_pub(path):
+ """The first line of the file that is not blank and not a comment."""
+ path = Path(path)
+ if not path.is_file(): raise Fail(f'{path} is missing: it holds the public key the launcher compiles in')
+ for line in path.read_text().splitlines():
+ line = line.strip()
+ if line and not line.startswith('#'):
+ raw32(line, str(path))
+ return line
+ raise Fail(f'{path} holds no public key yet: generate a key pair (`python3 tools/release_manifest.py keygen`), store the private '
+ f'half as the repository secret {KEY_ENV}, put the public half in this file and compile the same value into the launcher')
+
+
+def signing_key(pub_file, ephemeral=False):
+ """The private key to sign with, after checking it belongs to the committed public key. (key, its public base64)."""
+ if ephemeral:
+ ed, _ = ed25519()
+ k = ed.Ed25519PrivateKey.generate()
+ return k, public_b64(k)
+ secret = os.environ.get(KEY_ENV, '').strip()
+ if not secret: raise Fail(f'the repository secret {KEY_ENV} is not set: a release must be signed')
+ key = private_key(secret)
+ want = read_pub(pub_file)
+ have = public_b64(key)
+ if have != want:
+ raise Fail(f'{KEY_ENV} does not belong to the public key in {pub_file} (its public key is {have}, the file has {want}): '
+ f'the launcher would reject every manifest')
+ return key, have
+
+
+def verify_signature(manifest_bytes, sig_text, pub_b64):
+ ed, Invalid = ed25519()
+ key = ed.Ed25519PublicKey.from_public_bytes(raw32(pub_b64, 'the public key'))
+ try: sig = base64.b64decode(sig_text.strip(), validate=True)
+ except Exception: return False
+ try: key.verify(sig, manifest_bytes)
+ except Invalid: return False
+ return True
+
+
+# ---- the manifest ----
+
+def sha256_file(path):
+ h = hashlib.sha256()
+ with open(path, 'rb') as f:
+ for chunk in iter(lambda: f.read(1 << 20), b''): h.update(chunk)
+ return h.hexdigest()
+
+
+def asset(path):
+ path = Path(path)
+ if not path.is_file(): raise Fail(f'{path} is missing')
+ return {'asset': path.name, 'sha256': sha256_file(path), 'size': path.stat().st_size}
+
+
+def engine_identity(zip_path):
+ """(pin, engine_id) as the engine zip itself says."""
+ try:
+ with zipfile.ZipFile(zip_path) as z:
+ eid = z.read('.engine-id').decode().strip()
+ pin = z.read('.upstream-ref').decode().strip()
+ except (KeyError, zipfile.BadZipFile, OSError) as e:
+ raise Fail(f'{zip_path} is not an engine tree zip from `tools/engine.py tree` (.engine-id or .upstream-ref missing: {e})')
+ if not ENGINE_ID.fullmatch(eid): raise Fail(f'.engine-id in {zip_path} is not <pin>-<12 hex>: {eid!r}')
+ if eid[:40] != pin: raise Fail(f'.engine-id ({eid}) and .upstream-ref ({pin}) in {zip_path} disagree about the pin')
+ return pin, eid
+
+
+def build_manifest(engine_zip, version, channel, tag, app_zip=None, repo=GITHUB_REPO, built=None, extra_sources=()):
+ if channel not in ('stable', 'experimental'): raise Fail(f'channel must be stable or experimental, not {channel!r}')
+ if not version.strip(): raise Fail('--version is empty')
+ pin, eid = engine_identity(engine_zip)
+ built = built or datetime.datetime.now(datetime.timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ')
+ m = {'version': version, 'pin': pin, 'engine_id': eid, 'channel': channel, 'built': built, 'engine': asset(engine_zip)}
+ if app_zip: m['app'] = asset(app_zip)
+ m['sources'] = [f'https://github.com/{repo}/releases/download/{tag}/', *extra_sources]
+ return (json.dumps(m, indent=2) + '\n').encode()
+
+
+# ---- publishing ----
+
+def gh(*a, check=True):
+ r = subprocess.run(['gh', *a], capture_output=True, text=True)
+ if check and r.returncode: raise Fail(f'gh {" ".join(a[:3])} failed: {(r.stderr or r.stdout).strip()[-600:]}')
+ return r
+
+
+def upload(files, tag, repo, channel, notes):
+ """Engine zip first, the manifest and its signature last, so a launcher that reads the new manifest finds its zip."""
+ exists = gh('release', 'view', tag, '-R', repo, check=False).returncode == 0
+ if channel == 'experimental':
+ # A pre-release under a fixed tag, never "latest", so it never becomes what the app's own update looks at.
+ if exists: gh('release', 'edit', tag, '-R', repo, '--prerelease', '--latest=false', '--notes', notes)
+ else: gh('release', 'create', tag, '-R', repo, '--prerelease', '--latest=false', '--title', 'Engine (experimental)', '--notes', notes)
+ elif not exists:
+ raise Fail(f'the release {tag} does not exist: the release workflow creates it before it publishes the engine')
+ for f in files:
+ gh('release', 'upload', tag, str(f), '-R', repo, '--clobber')
+ say(f'uploaded {Path(f).name} to {repo} {tag}')
+
+
+def cmd_publish(a):
+ if a.upload and a.ephemeral_key: raise Fail('--ephemeral-key signs with a throwaway key: it never uploads')
+ key, pub = signing_key(a.pub, a.ephemeral_key)
+ out = Path(a.out)
+ out.mkdir(parents=True, exist_ok=True)
+ zsrc = Path(a.engine_zip)
+ data = build_manifest(zsrc, a.version, a.channel, a.tag, a.app_zip, a.repo, a.built, a.source or ())
+ files = [out / zsrc.name, out / MANIFEST, out / SIGNATURE]
+ if zsrc.resolve() != files[0].resolve(): shutil.copyfile(zsrc, files[0])
+ files[1].write_bytes(data)
+ files[2].write_text(base64.b64encode(key.sign(data)).decode()) # no newline: Data(base64Encoded:) rejects one
+ if not verify_signature(files[1].read_bytes(), files[2].read_text(), pub): raise Fail('the signature just made does not verify')
+ m = json.loads(data)
+ say(f'{MANIFEST}: {m["channel"]} {m["version"]} engine_id {m["engine_id"]} built {m["built"]}, signed ({"ephemeral key" if a.ephemeral_key else "ENGINE_SIGNING_KEY"})')
+ if a.upload:
+ notes = (f'Engine {m["engine_id"]} (seedds/nxnative {m["pin"][:12]} + the patch series), built {m["built"]}. '
+ + ('Experimental: built by the nightly sync and checked by CI only, not played on hardware. The launcher offers it as a manual, opt-in update.'
+ if a.channel == 'experimental' else 'The engine of this release.'))
+ upload(files, a.tag, a.repo, a.channel, notes)
+ else:
+ say(f'dry run: nothing uploaded; the files are in {out}')
+ return 0
+
+
+def cmd_check_key(a):
+ key, pub = signing_key(a.pub)
+ say(f'{KEY_ENV} matches {a.pub} ({pub})')
+ return 0
+
+
+def cmd_verify(a):
+ pub = read_pub(a.pub)
+ ok = verify_signature(Path(a.manifest).read_bytes(), Path(a.sig).read_text(), pub)
+ say('signature valid' if ok else 'signature INVALID')
+ return 0 if ok else 1
+
+
+def cmd_keygen(a):
+ ed, _ = ed25519()
+ from cryptography.hazmat.primitives import serialization
+ k = ed.Ed25519PrivateKey.generate()
+ raw = k.private_bytes(serialization.Encoding.Raw, serialization.PrivateFormat.Raw, serialization.NoEncryption())
+ say(f'private (the repository secret {KEY_ENV}, keep it out of the repository): {base64.b64encode(raw).decode()}')
+ say(f'public (frontend/engine-signing.pub and the launcher): {public_b64(k)}')
+ return 0
+
+
+def main(argv=None):
+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ sub = ap.add_subparsers(dest='cmd', required=True)
+ sub.add_parser('keygen').set_defaults(f=cmd_keygen)
+ s = sub.add_parser('check-key'); s.add_argument('--pub', default=str(PUB_FILE)); s.set_defaults(f=cmd_check_key)
+ s = sub.add_parser('verify'); s.add_argument('--manifest', required=True); s.add_argument('--sig', required=True); s.add_argument('--pub', default=str(PUB_FILE)); s.set_defaults(f=cmd_verify)
+ s = sub.add_parser('publish')
+ s.add_argument('--engine-zip', required=True); s.add_argument('--app-zip'); s.add_argument('--version', required=True)
+ s.add_argument('--channel', required=True, choices=('stable', 'experimental')); s.add_argument('--tag', required=True)
+ s.add_argument('--out', required=True); s.add_argument('--repo', default=os.environ.get('GITHUB_REPOSITORY') or GITHUB_REPO)
+ s.add_argument('--built', help='UTC timestamp, 2026-10-31T02:00:00Z (default: now)'); s.add_argument('--source', action='append', help='another base URL for the assets (a mirror), tried after GitHub')
+ s.add_argument('--pub', default=str(PUB_FILE)); s.add_argument('--upload', action='store_true'); s.add_argument('--ephemeral-key', action='store_true')
+ s.set_defaults(f=cmd_publish)
+ a = ap.parse_args(argv)
+ try:
+ return a.f(a)
+ except Fail as e:
+ print(f'release_manifest.py: {e}', file=sys.stderr)
+ return 2
+
+
+if __name__ == '__main__':
+ sys.exit(main())
diff --git a/tools/requirements-release.txt b/tools/requirements-release.txt
new file mode 100644
index 00000000..4e65bf54
--- /dev/null
+++ b/tools/requirements-release.txt
@@ -0,0 +1,2 @@
+# What tools/release_manifest.py needs to sign latest.json (Ed25519). Only the release and the nightly publish jobs install it.
+cryptography==46.0.3
diff --git a/tools/tests/test_engine.py b/tools/tests/test_engine.py
index bdeac34f..82f812d2 100644
--- a/tools/tests/test_engine.py
+++ b/tools/tests/test_engine.py
@@ -255,6 +255,80 @@ class Search(unittest.TestCase):
self.assertEqual(r.stdout.strip(), './engine/dm/needle.c', r.stderr)
+class Tree(Fixture):
+ """The engine tree for players: engine/ at its last sync, no git, with the identity and a zip that is the same every time."""
+ def make(self, name='out', zipname=None):
+ out = self.tmp / name
+ args = ['tree', '--out', str(out)] + (['--zip', str(self.tmp / zipname)] if zipname else [])
+ return out, self.tool(*args)
+
+ def test_the_tree_is_the_patched_engine_without_git_and_names_its_identity(self):
+ self.tool('sync')
+ out, said = self.make()
+ self.assertEqual((out / 'dm/a.c').read_text(), 'line1\nline2 fork\nline3\nline4\nline5\nline6\nline7\n') # the patch is in
+ self.assertEqual((out / '.upstream-ref').read_text(), self.up1 + '\n')
+ eid = (out / '.engine-id').read_text().strip()
+ self.assertEqual(eid, self.up1 + '-' + E.stamp_key(self.r)[:12])
+ self.assertRegex(eid, r'^[0-9a-f]{40}-[0-9a-f]{12}$')
+ self.assertEqual(eid, E.engine_id(self.r)); self.assertIn(eid, said)
+ self.assertIn('line-2', (out / 'dm/fork_patches.txt').read_text())
+ self.assertTrue((out / 'AGENTS.md').is_file())
+ self.assertFalse((out / '.git').exists()); self.assertFalse((out / '.gitignore').exists()) # no history, no repository files
+ self.assertFalse((out / 'tools').exists()); self.assertFalse((out / 'patches').exists()) # nothing of the fork
+
+ def test_a_series_change_without_a_pin_move_changes_the_identity(self):
+ self.tool('sync'); out, _ = self.make('one')
+ self.add_patch('0002-b.patch', 'dm/b.c', 'b1\nb2 fork\n', 'Change b2', header=HEADER.replace('#1', '#2'))
+ self.tool('sync'); out2, _ = self.make('two')
+ a, b = (out / '.engine-id').read_text().strip(), (out2 / '.engine-id').read_text().strip()
+ self.assertEqual(a[:40], b[:40]); self.assertNotEqual(a, b)
+
+ def test_the_zip_is_deterministic_and_unpacks_to_the_tree(self):
+ import hashlib, time, zipfile
+ self.tool('sync')
+ out, _ = self.make('one', 'one.zip')
+ time.sleep(1.1) # other modification times, same bytes
+ for f in out.rglob('*'): os.utime(f, (time.time(), time.time()))
+ out2, _ = self.make('two', 'two.zip')
+ h = lambda n: hashlib.sha256((self.tmp / n).read_bytes()).hexdigest()
+ self.assertEqual(h('one.zip'), h('two.zip'))
+ with zipfile.ZipFile(self.tmp / 'one.zip') as z:
+ names = z.namelist()
+ self.assertEqual(names, sorted(names)); self.assertIn('.engine-id', names); self.assertIn('dm/a.c', names)
+ self.assertEqual({i.date_time for i in z.infolist()}, {E.ZIP_TIME})
+ self.assertEqual(z.read('.engine-id').decode(), (out / '.engine-id').read_text())
+
+ def test_executable_bits_and_symlinks_survive(self):
+ import stat, zipfile
+ self.wu('dm/run.sh', '#!/bin/sh\n'); os.chmod(self.up / 'dm/run.sh', 0o755)
+ os.symlink('a.c', self.up / 'dm/link.c')
+ up2 = self.up_commit('dm/c.c', 'c\n', 'upstream 2')
+ (self.r / '.upstream-ref').write_text(up2 + '\n')
+ self.tool('sync'); out, _ = self.make('out', 'e.zip')
+ self.assertTrue(os.access(out / 'dm/run.sh', os.X_OK)); self.assertFalse(os.access(out / 'dm/a.c', os.X_OK))
+ self.assertTrue((out / 'dm/link.c').is_symlink())
+ with zipfile.ZipFile(self.tmp / 'e.zip') as z:
+ self.assertEqual(stat.S_IMODE(z.getinfo('dm/run.sh').external_attr >> 16), 0o755)
+ self.assertTrue(stat.S_ISLNK(z.getinfo('dm/link.c').external_attr >> 16))
+
+ def test_a_stale_or_dirty_engine_makes_no_tree(self):
+ self.tool('tree', '--out', str(self.tmp / 'o'), expect=2) # no engine/ at all
+ self.tool('sync')
+ self.add_patch('0002-b.patch', 'dm/b.c', 'b1\nb2 fork\n', 'Change b2', header=HEADER.replace('#1', '#2'))
+ self.assertIn('not a current sync', self.tool('tree', '--out', str(self.tmp / 'o'), expect=2))
+ self.tool('sync')
+ (self.eng / 'dm/a.c').write_text('local edit\n')
+ self.assertIn('changes the release must not carry', self.tool('tree', '--out', str(self.tmp / 'o'), expect=2))
+ self.assertFalse((self.tmp / 'o/.engine-id').exists())
+
+ def test_it_will_not_empty_a_folder_that_is_not_an_earlier_tree(self):
+ self.tool('sync')
+ keep = self.tmp / 'precious'; keep.mkdir(); (keep / 'x').write_text('mine')
+ self.assertIn('not an engine tree', self.tool('tree', '--out', str(keep), expect=2))
+ self.assertEqual((keep / 'x').read_text(), 'mine')
+ self.make('again'); self.make('again') # an earlier tree is replaced
+
+
class StampKey(Fixture):
def test_the_qt_front_end_computes_the_same_key(self):
sys.path.insert(0, str(REPO / 'frontend-qt'))
diff --git a/tools/tests/test_release_manifest.py b/tools/tests/test_release_manifest.py
new file mode 100644
index 00000000..7a6ad6bd
--- /dev/null
+++ b/tools/tests/test_release_manifest.py
@@ -0,0 +1,228 @@
+"""tools/release_manifest.py (LNPANDY/NX2Apple#227): latest.json has the shape EngineManifest.Manifest decodes, the signature is a
+detached Ed25519 signature over its exact bytes, the release fails without the right key, and uploads go in the right order with a
+fake `gh`. The keys are generated here: nothing secret is used and nothing touches the network."""
+import base64
+import hashlib
+import importlib
+import json
+import os
+import stat
+import subprocess
+import sys
+import tempfile
+import unittest
+import zipfile
+from pathlib import Path
+
+TOOLS = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(TOOLS))
+R = importlib.import_module('release_manifest')
+
+try:
+ from cryptography.hazmat.primitives import serialization
+ from cryptography.hazmat.primitives.asymmetric import ed25519
+except ImportError:
+ ed25519 = None
+
+PIN = 'ee6648cb2bb46c19255c1d04f88e9c40d36ef6b4'
+EID = PIN + '-d4abe1fa2485'
+
+
+def new_key():
+ k = ed25519.Ed25519PrivateKey.generate()
+ raw = k.private_bytes(serialization.Encoding.Raw, serialization.PrivateFormat.Raw, serialization.NoEncryption())
+ return k, base64.b64encode(raw).decode(), R.public_b64(k)
+
+
[email protected](ed25519 is None, 'needs `cryptography` (tools/requirements-release.txt)')
+class Manifest(unittest.TestCase):
+ def setUp(self):
+ self.tmp = Path(tempfile.mkdtemp(prefix='test-manifest-'))
+ self.addCleanup(__import__('shutil').rmtree, self.tmp, True)
+ self.key, self.secret, self.pub = new_key()
+ self.pubfile = self.tmp / 'engine-signing.pub'
+ self.pubfile.write_text(f'# comment\n\n{self.pub}\n')
+ self.engine = self.zip('NX2Apple-engine.zip', {'.engine-id': EID + '\n', '.upstream-ref': PIN + '\n', 'dm/a.c': 'x'})
+ self.app = self.tmp / 'NX2Apple-macOS-arm64.zip'; self.app.write_bytes(b'app bytes')
+ self.env = dict(os.environ, **{R.KEY_ENV: self.secret})
+ self.old_env = dict(os.environ)
+ os.environ[R.KEY_ENV] = self.secret
+ self.addCleanup(self.restore)
+
+ def restore(self):
+ os.environ.clear(); os.environ.update(self.old_env)
+
+ def zip(self, name, files):
+ p = self.tmp / name
+ with zipfile.ZipFile(p, 'w') as z:
+ for n, t in files.items(): z.writestr(n, t)
+ return p
+
+ def publish(self, *extra, channel='stable', tag='v0.8.0', out='out', expect=0):
+ args = ['publish', '--engine-zip', str(self.engine), '--version', '0.8.0', '--channel', channel, '--tag', tag,
+ '--out', str(self.tmp / out), '--pub', str(self.pubfile), '--built', '2026-10-31T02:00:00Z', *extra]
+ self.assertEqual(R.main(args), expect)
+ return self.tmp / out
+
+ # ---- the shape the launcher decodes ----
+
+ def test_the_manifest_has_every_field_the_launcher_decodes(self):
+ out = self.publish('--app-zip', str(self.app))
+ m = json.loads((out / 'latest.json').read_text())
+ self.assertEqual(m['version'], '0.8.0'); self.assertEqual(m['pin'], PIN); self.assertEqual(m['engine_id'], EID)
+ self.assertEqual(m['channel'], 'stable'); self.assertEqual(m['built'], '2026-10-31T02:00:00Z')
+ self.assertEqual(m['sources'], ['https://github.com/LNPANDY/NX2Apple/releases/download/v0.8.0/'])
+ self.assertEqual(set(m['engine']), {'asset', 'sha256', 'size'})
+ self.assertEqual(m['engine']['asset'], 'NX2Apple-engine.zip')
+ self.assertEqual(m['engine']['sha256'], hashlib.sha256(self.engine.read_bytes()).hexdigest())
+ self.assertEqual(m['engine']['size'], self.engine.stat().st_size)
+ self.assertEqual(m['app'], {'asset': 'NX2Apple-macOS-arm64.zip', 'sha256': hashlib.sha256(b'app bytes').hexdigest(), 'size': 9})
+ for a in (m['engine'], m['app']): self.assertRegex(a['sha256'], r'^[0-9a-f]{64}$') # EngineManifest.isDigest
+ self.assertRegex(m['engine_id'], r'^[0-9a-f]{40}-[0-9a-f]{12}$')
+
+ def test_the_experimental_channel_has_no_app_and_its_own_source(self):
+ out = self.publish(channel='experimental', tag='engine-experimental')
+ m = json.loads((out / 'latest.json').read_text())
+ self.assertEqual(m['channel'], 'experimental'); self.assertNotIn('app', m)
+ self.assertEqual(m['sources'], ['https://github.com/LNPANDY/NX2Apple/releases/download/engine-experimental/'])
+
+ def test_the_manifest_bytes_are_the_same_for_the_same_input(self):
+ a = (self.publish(out='a') / 'latest.json').read_bytes(); b = (self.publish(out='b') / 'latest.json').read_bytes()
+ self.assertEqual(a, b)
+
+ def test_a_mirror_is_added_as_a_source_after_github(self):
+ out = self.publish('--source', 'https://mirror.example/engine/')
+ self.assertEqual(json.loads((out / 'latest.json').read_text())['sources'][1], 'https://mirror.example/engine/')
+
+ def test_a_zip_that_is_not_an_engine_tree_is_refused(self):
+ self.engine = self.zip('bad.zip', {'dm/a.c': 'x'})
+ self.publish(expect=2)
+ self.engine = self.zip('bad2.zip', {'.engine-id': 'ABC\n', '.upstream-ref': PIN + '\n'})
+ self.publish(expect=2)
+ self.engine = self.zip('bad3.zip', {'.engine-id': PIN[:-1] + '0-d4abe1fa2485\n', '.upstream-ref': PIN + '\n'})
+ self.publish(expect=2)
+
+ # ---- the signature ----
+
+ def test_the_signature_is_detached_ed25519_over_the_exact_bytes(self):
+ out = self.publish()
+ data = (out / 'latest.json').read_bytes(); sig = (out / 'latest.json.sig').read_text()
+ self.assertFalse(sig.endswith('\n')) # Data(base64Encoded:) rejects a newline
+ raw = base64.b64decode(sig, validate=True); self.assertEqual(len(raw), 64)
+ self.key.public_key().verify(raw, data) # what CryptoKit's isValidSignature checks
+ self.assertTrue(R.verify_signature(data, sig, self.pub))
+ self.assertFalse(R.verify_signature(data + b' ', sig, self.pub)) # one byte more: invalid
+ self.assertFalse(R.verify_signature(data.replace(b'0.8.0', b'0.8.1'), sig, self.pub))
+ other = new_key()[2]
+ self.assertFalse(R.verify_signature(data, sig, other))
+ self.assertFalse(R.verify_signature(data, 'not base64!', self.pub))
+
+ def test_verify_subcommand(self):
+ out = self.publish()
+ args = ['verify', '--manifest', str(out / 'latest.json'), '--sig', str(out / 'latest.json.sig'), '--pub', str(self.pubfile)]
+ self.assertEqual(R.main(args), 0)
+ (out / 'latest.json').write_bytes((out / 'latest.json').read_bytes().replace(b'stable', b'rolling'))
+ self.assertEqual(R.main(args), 1)
+
+ # ---- the key check ----
+
+ def test_a_release_fails_without_the_secret(self):
+ del os.environ[R.KEY_ENV]
+ self.assertEqual(R.main(['check-key', '--pub', str(self.pubfile)]), 2)
+ self.publish(expect=2)
+ os.environ[R.KEY_ENV] = ' '
+ self.assertEqual(R.main(['check-key', '--pub', str(self.pubfile)]), 2)
+
+ def test_a_release_fails_when_the_key_is_not_the_one_in_the_repository(self):
+ os.environ[R.KEY_ENV] = new_key()[1]
+ self.assertEqual(R.main(['check-key', '--pub', str(self.pubfile)]), 2)
+ self.publish(expect=2)
+ self.assertFalse((self.tmp / 'out' / 'latest.json').exists())
+
+ def test_a_release_fails_while_the_repository_has_no_public_key(self):
+ self.pubfile.write_text('# nothing yet\n')
+ self.assertEqual(R.main(['check-key', '--pub', str(self.pubfile)]), 2)
+ self.pubfile.unlink()
+ self.assertEqual(R.main(['check-key', '--pub', str(self.pubfile)]), 2)
+
+ def test_malformed_keys_are_refused(self):
+ os.environ[R.KEY_ENV] = 'not base64!'
+ self.assertEqual(R.main(['check-key', '--pub', str(self.pubfile)]), 2)
+ os.environ[R.KEY_ENV] = base64.b64encode(b'short').decode()
+ self.assertEqual(R.main(['check-key', '--pub', str(self.pubfile)]), 2)
+ os.environ[R.KEY_ENV] = self.secret
+ self.pubfile.write_text(base64.b64encode(b'short').decode() + '\n')
+ self.assertEqual(R.main(['check-key', '--pub', str(self.pubfile)]), 2)
+
+ def test_the_matching_key_passes(self):
+ self.assertEqual(R.main(['check-key', '--pub', str(self.pubfile)]), 0)
+
+ def test_a_dry_run_can_sign_with_a_throwaway_key_but_never_upload(self):
+ del os.environ[R.KEY_ENV]
+ self.pubfile.write_text('# none\n')
+ out = self.publish('--ephemeral-key')
+ data = (out / 'latest.json').read_bytes(); sig = (out / 'latest.json.sig').read_text()
+ self.assertFalse(R.verify_signature(data, sig, self.pub)) # not the repository's key
+ self.publish('--ephemeral-key', '--upload', out='out2', expect=2)
+
+ def test_keygen_makes_a_pair_the_check_accepts(self):
+ import contextlib, io
+ buf = io.StringIO()
+ with contextlib.redirect_stdout(buf): self.assertEqual(R.main(['keygen']), 0)
+ lines = buf.getvalue().splitlines()
+ priv, pub = lines[0].rsplit(': ', 1)[1], lines[1].rsplit(': ', 1)[1]
+ os.environ[R.KEY_ENV] = priv; self.pubfile.write_text(pub + '\n')
+ self.assertEqual(R.main(['check-key', '--pub', str(self.pubfile)]), 0)
+
+ # ---- upload, with a fake gh ----
+
+ def fake_gh(self, exists):
+ """A `gh` that logs its arguments and answers `release view` as asked."""
+ d = self.tmp / 'bin'; d.mkdir(exist_ok=True)
+ log = self.tmp / 'gh.log'
+ gh = d / 'gh'
+ gh.write_text(f'#!/bin/sh\necho "$@" >> "{log}"\n[ "$1 $2" = "release view" ] && exit {0 if exists else 1}\nexit 0\n')
+ gh.chmod(gh.stat().st_mode | stat.S_IXUSR)
+ os.environ['PATH'] = f'{d}:{os.environ["PATH"]}'
+ return log
+
+ def calls(self, log):
+ return [l.split() for l in log.read_text().splitlines()]
+
+ def test_the_stable_release_uploads_the_zip_first_and_the_signature_last(self):
+ log = self.fake_gh(exists=True)
+ self.publish('--app-zip', str(self.app), '--upload', '--repo', 'LNPANDY/NX2Apple')
+ ups = [c for c in self.calls(log) if c[:2] == ['release', 'upload']]
+ self.assertEqual([Path(c[3]).name for c in ups], ['NX2Apple-engine.zip', 'latest.json', 'latest.json.sig'])
+ self.assertTrue(all(c[2] == 'v0.8.0' and '--clobber' in c and c[4:6] == ['-R', 'LNPANDY/NX2Apple'] for c in ups))
+ self.assertFalse([c for c in self.calls(log) if c[:2] in (['release', 'create'], ['release', 'edit'])])
+
+ def test_the_stable_release_needs_its_release_to_exist(self):
+ log = self.fake_gh(exists=False)
+ self.publish('--upload', expect=2)
+ self.assertFalse([c for c in self.calls(log) if c[:2] == ['release', 'upload']])
+
+ def test_the_experimental_tag_is_created_as_a_prerelease_that_is_never_latest(self):
+ log = self.fake_gh(exists=False)
+ self.publish('--upload', channel='experimental', tag='engine-experimental')
+ create = [c for c in self.calls(log) if c[:2] == ['release', 'create']]
+ self.assertEqual(len(create), 1)
+ self.assertEqual(create[0][2], 'engine-experimental'); self.assertIn('--prerelease', create[0]); self.assertIn('--latest=false', create[0])
+ self.assertEqual(len([c for c in self.calls(log) if c[:2] == ['release', 'upload']]), 3)
+
+ def test_an_existing_experimental_release_is_kept_a_prerelease_and_its_assets_replaced(self):
+ log = self.fake_gh(exists=True)
+ self.publish('--upload', channel='experimental', tag='engine-experimental')
+ edit = [c for c in self.calls(log) if c[:2] == ['release', 'edit']]
+ self.assertEqual(len(edit), 1); self.assertIn('--prerelease', edit[0]); self.assertIn('--latest=false', edit[0])
+ self.assertFalse([c for c in self.calls(log) if c[:2] == ['release', 'create']])
+ self.assertTrue(all('--clobber' in c for c in self.calls(log) if c[:2] == ['release', 'upload']))
+
+ def test_a_dry_run_calls_gh_never(self):
+ log = self.fake_gh(exists=True)
+ self.publish()
+ self.assertFalse(log.exists())
+
+
+if __name__ == '__main__':
+ unittest.main()