Skip to content

armgit / ente

Public Imported from GitHub
0

πŸ’š End-to-end encrypted cloud for everything.

single post-install script

laurenspriem8b12485d977f
diff --git a/desktop/.gitignore b/desktop/.gitignore
index 33dcd9a63e..98133c7f9b 100644
--- a/desktop/.gitignore
+++ b/desktop/.gitignore
@@ -15,7 +15,7 @@ dist/
# We download it on demand, if needed for the particular OS/arch.
build/vips*
-# Downloaded by scripts/ort.js for the particular OS/arch.
+# Downloaded by scripts/ml-native.js for the particular OS/arch.
build/onnxruntime/
# Rust ML addons staged for packaging by scripts/ml-native.js.
diff --git a/desktop/docs/dependencies.md b/desktop/docs/dependencies.md
index 7117a14b3d..7b20e462d1 100644
--- a/desktop/docs/dependencies.md
+++ b/desktop/docs/dependencies.md
@@ -76,7 +76,7 @@ On macOS, we use the `sips` CLI tool for these tasks, but that is already availa
The ML pipeline (natural language search using CLIP, and face detection and recognition) is implemented by the Rust crate shared with the mobile apps (`rust/crates/photos`), exposed to us as a Node native addon built from source by `cargo codegen napi` (see "[Note: ML with Rust]").
-The addon loads [ONNX Runtime](https://onnxruntime.ai) dynamically at runtime, using Ente's pinned custom builds (CoreML-enabled on macOS) that `scripts/ort.js` downloads into the extra resources (`build`) folder. See "[Note: ONNX Runtime binaries]".
+The addon loads [ONNX Runtime](https://onnxruntime.ai) dynamically at runtime, using Ente's pinned custom builds (CoreML-enabled on macOS) that `scripts/ml-native.js` downloads into the extra resources (`build`) folder. See "[Note: ONNX Runtime binaries]".
### ZIP
diff --git a/desktop/docs/dev.md b/desktop/docs/dev.md
index 41b40cfb0b..60cf9c1afc 100644
--- a/desktop/docs/dev.md
+++ b/desktop/docs/dev.md
@@ -51,7 +51,7 @@ Installs skip dependency lifecycle scripts (`ignore-scripts` in `.npmrc`) so tha
- Downloads the `vips` binary used during development (see `scripts/vips.js`).
-- Downloads the ONNX Runtime library used by the ML pipeline (see `scripts/ort.js`).
+- Downloads the ONNX Runtime library used by the ML pipeline (see `scripts/ml-native.js`).
- Builds the Rust ML addon from source by running `cargo codegen napi`, which places the built addon and its generated TypeScript declarations in the gitignored `rust-bindings/` directory. Rerun this codegen (from `../rust`) after changing the Rust ML code.
diff --git a/desktop/scripts/beforeBuild.js b/desktop/scripts/beforeBuild.js
index 75240efa5a..df78cdf0a2 100755
--- a/desktop/scripts/beforeBuild.js
+++ b/desktop/scripts/beforeBuild.js
@@ -1,6 +1,5 @@
const fsp = require("fs/promises");
-const { stageMLAddons } = require("./ml-native");
-const { ensureONNXRuntime } = require("./ort");
+const { ensureONNXRuntime, stageMLAddons } = require("./ml-native");
/**
* This hook is invoked during the initial build (e.g. when triggered by
diff --git a/desktop/scripts/ml-native.js b/desktop/scripts/ml-native.js
index 1572b3ccfa..a7876508e5 100644
--- a/desktop/scripts/ml-native.js
+++ b/desktop/scripts/ml-native.js
@@ -14,9 +14,131 @@
* corresponding Rust target to be installed (`rustup target add ...`).
*/
-const fsp = require("fs/promises");
-const path = require("path");
-const { execSync } = require("child_process");
+/**
+ * [Note: ONNX Runtime binaries]
+ *
+ * The Rust ML addon (see [Note: ML with Rust]) loads the ONNX Runtime
+ * dynamic library at runtime. We use Ente's pinned custom builds β€” CoreML
+ * enabled on macOS, WebGPU-capable elsewhere β€” published at
+ * https://github.com/laurens-pilot/ort-packaging/releases
+ *
+ * This script downloads the archive for the relevant OS/arch combination,
+ * verifies it against the SHA-256 checksums pinned below, and extracts it
+ * into `build/onnxruntime/<arch>`. From there it gets used during
+ * development, and copied into the packaged app's resources by the
+ * "extraFiles" clause in `electron-builder.yml`.
+ *
+ * On macOS both architectures are downloaded since the packaged app is a
+ * universal binary. Like `vips.js`, this script runs during "npm install"
+ * (via postinstall), while `beforeBuild.js` additionally invokes it for the
+ * specific architecture being packaged (which on CI might differ from the
+ * architecture we're running on).
+ */
+
+const crypto = require("node:crypto");
+const fsp = require("node:fs/promises");
+const path = require("node:path");
+const { execFileSync, execSync } = require("node:child_process");
+
+const version = "1.27.0-r4";
+
+const releaseURL = `https://github.com/laurens-pilot/ort-packaging/releases/download/ort-${version}`;
+
+/**
+ * SHA-256 checksums of the release assets, pinned here so that the library we
+ * package cannot change without a corresponding (reviewed) change to this
+ * file. When bumping {@link version}, update these with the values from the
+ * release's `.sha256` sidecar files after verifying them against the built
+ * archives.
+ */
+const assetSHA256s = {
+ "darwin-arm64":
+ "dbb243c3b43963fda6c9475a978d0954b69490d386bc3ecb54a471b2a13ba043",
+ "darwin-x64":
+ "d655686380c1976a9067a48d82645a381e4bc4bdfcc44c88f300befc24c4ae8e",
+ "linux-arm64":
+ "0b0367bddd96fb0263781022f715ec531a9856e14d8ed91beea80da8d9bb6a2e",
+ "linux-x64":
+ "d417b8d92498f3e754e23fbeac0751f13375061c6c31cc83e869f33a37fca8ad",
+ "win32-arm64":
+ "9e3179a985e08700b37f15af7049bca4fc007cb4d5a7e2ff1d81df10e330544e",
+ "win32-x64":
+ "0534f35981fe3174379c7e80f93977edd990ce67c40f6cbdde1e6890eb798911",
+};
+
+const assetName = (platform, arch) => {
+ switch (platform) {
+ case "darwin":
+ return `onnxruntime-coreml-macos-${arch}-${version}.tar.gz`;
+ case "linux":
+ return `onnxruntime-webgpu-linux-${arch}-${version}.tar.gz`;
+ case "win32":
+ return `onnxruntime-webgpu-windows-${arch}-${version}.zip`;
+ default:
+ throw new Error(`Unsupported platform: ${platform}`);
+ }
+};
+
+/**
+ * Download and extract the ONNX Runtime library for the given OS/arch into
+ * `<appDir>/build/onnxruntime/<arch>`, no-oping if it is already there.
+ */
+const downloadONNXRuntimeIfNeeded = async (platform, arch, appDir) => {
+ const asset = assetName(platform, arch);
+ const outDir = path.join(appDir, "build", "onnxruntime", arch);
+ const stampPath = path.join(outDir, ".ente-ort-stamp");
+
+ try {
+ if ((await fsp.readFile(stampPath, "utf8")) == asset) return;
+ } catch {}
+
+ const expected = assetSHA256s[`${platform}-${arch}`];
+ if (!expected) throw new Error(`No pinned SHA-256 for ${platform}-${arch}`);
+
+ console.log(`Downloading ${asset}`);
+ const res = await fetch(`${releaseURL}/${asset}`);
+ if (!res.ok)
+ throw new Error(`Failed to download ${asset}: HTTP ${res.status}`);
+ const archive = Buffer.from(await res.arrayBuffer());
+
+ const actual = crypto.createHash("sha256").update(archive).digest("hex");
+ if (actual != expected)
+ throw new Error(
+ `SHA-256 mismatch for ${asset}: expected ${expected}, got ${actual}`,
+ );
+
+ await fsp.rm(outDir, { recursive: true, force: true });
+ await fsp.mkdir(outDir, { recursive: true });
+ const archivePath = path.join(outDir, asset);
+ await fsp.writeFile(archivePath, archive);
+ // tar is available on all the platforms we build on (bsdtar on macOS and
+ // Windows, GNU tar on Linux), and handles both .tar.gz and .zip.
+ execFileSync("tar", ["-xf", asset], { cwd: outDir });
+ await fsp.rm(archivePath);
+ await fsp.writeFile(stampPath, asset);
+};
+
+/**
+ * Ensure `<appDir>/build/onnxruntime` contains exactly the ONNX Runtime
+ * libraries needed for the given OS/arch combination.
+ *
+ * On macOS the packaged app is universal, so both architectures are kept. On
+ * Linux and Windows only the given arch is kept: unlike vips (a single file
+ * that gets overwritten in place per arch), the libraries live in per-arch
+ * directories, so directories left over from earlier passes for other
+ * architectures need to be removed lest they also get copied into the
+ * package.
+ */
+const ensureONNXRuntime = async (platform, arch, appDir) => {
+ const arches = platform == "darwin" ? ["arm64", "x64"] : [arch];
+ for (const a of arches)
+ await downloadONNXRuntimeIfNeeded(platform, a, appDir);
+
+ const outDir = path.join(appDir, "build", "onnxruntime");
+ for (const entry of await fsp.readdir(outDir))
+ if (!arches.includes(entry))
+ await fsp.rm(path.join(outDir, entry), { recursive: true });
+};
/** napi-cli's name for the addon built for the given platform-arch. */
const napiTriple = (platform, arch) => {
@@ -123,4 +245,15 @@ const stageMLAddons = async (appDir, platform, arch) => {
await fsp.rm(path.join(stageDir, entry), { recursive: true });
};
-module.exports = { napiTriple, stageMLAddons };
+const main = async () => {
+ await ensureONNXRuntime(process.platform, process.arch, ".");
+ execSync("cargo codegen napi", { cwd: "../rust", stdio: "inherit" });
+};
+
+module.exports = { ensureONNXRuntime, napiTriple, stageMLAddons };
+
+if (require.main === module)
+ main().catch((e) => {
+ console.error(e);
+ process.exit(1);
+ });
diff --git a/desktop/scripts/ort.js b/desktop/scripts/ort.js
deleted file mode 100644
index 072fe35c40..0000000000
--- a/desktop/scripts/ort.js
+++ /dev/null
@@ -1,135 +0,0 @@
-/**
- * [Note: ONNX Runtime binaries]
- *
- * The Rust ML addon (see [Note: ML with Rust]) loads the ONNX Runtime
- * dynamic library at runtime. We use Ente's pinned custom builds β€” CoreML
- * enabled on macOS, WebGPU-capable elsewhere β€” published at
- * https://github.com/laurens-pilot/ort-packaging/releases
- *
- * This script downloads the archive for the relevant OS/arch combination,
- * verifies it against the SHA-256 checksums pinned below, and extracts it
- * into `build/onnxruntime/<arch>`. From there it gets used during
- * development, and copied into the packaged app's resources by the
- * "extraFiles" clause in `electron-builder.yml`.
- *
- * On macOS both architectures are downloaded since the packaged app is a
- * universal binary. Like `vips.js`, this script runs during "npm install"
- * (via postinstall), while `beforeBuild.js` additionally invokes it for the
- * specific architecture being packaged (which on CI might differ from the
- * architecture we're running on).
- */
-
-const crypto = require("node:crypto");
-const fsp = require("node:fs/promises");
-const path = require("node:path");
-const { execFileSync } = require("node:child_process");
-
-const version = "1.27.0-r4";
-
-const releaseURL = `https://github.com/laurens-pilot/ort-packaging/releases/download/ort-${version}`;
-
-/**
- * SHA-256 checksums of the release assets, pinned here so that the library we
- * package cannot change without a corresponding (reviewed) change to this
- * file. When bumping {@link version}, update these with the values from the
- * release's `.sha256` sidecar files after verifying them against the built
- * archives.
- */
-const assetSHA256s = {
- "darwin-arm64":
- "dbb243c3b43963fda6c9475a978d0954b69490d386bc3ecb54a471b2a13ba043",
- "darwin-x64":
- "d655686380c1976a9067a48d82645a381e4bc4bdfcc44c88f300befc24c4ae8e",
- "linux-arm64":
- "0b0367bddd96fb0263781022f715ec531a9856e14d8ed91beea80da8d9bb6a2e",
- "linux-x64":
- "d417b8d92498f3e754e23fbeac0751f13375061c6c31cc83e869f33a37fca8ad",
- "win32-arm64":
- "9e3179a985e08700b37f15af7049bca4fc007cb4d5a7e2ff1d81df10e330544e",
- "win32-x64":
- "0534f35981fe3174379c7e80f93977edd990ce67c40f6cbdde1e6890eb798911",
-};
-
-const assetName = (platform, arch) => {
- switch (platform) {
- case "darwin":
- return `onnxruntime-coreml-macos-${arch}-${version}.tar.gz`;
- case "linux":
- return `onnxruntime-webgpu-linux-${arch}-${version}.tar.gz`;
- case "win32":
- return `onnxruntime-webgpu-windows-${arch}-${version}.zip`;
- default:
- throw new Error(`Unsupported platform: ${platform}`);
- }
-};
-
-/**
- * Download and extract the ONNX Runtime library for the given OS/arch into
- * `<appDir>/build/onnxruntime/<arch>`, no-oping if it is already there.
- */
-const downloadONNXRuntimeIfNeeded = async (platform, arch, appDir) => {
- const asset = assetName(platform, arch);
- const outDir = path.join(appDir, "build", "onnxruntime", arch);
- const stampPath = path.join(outDir, ".ente-ort-stamp");
-
- try {
- if ((await fsp.readFile(stampPath, "utf8")) == asset) return;
- } catch {}
-
- const expected = assetSHA256s[`${platform}-${arch}`];
- if (!expected) throw new Error(`No pinned SHA-256 for ${platform}-${arch}`);
-
- console.log(`Downloading ${asset}`);
- const res = await fetch(`${releaseURL}/${asset}`);
- if (!res.ok)
- throw new Error(`Failed to download ${asset}: HTTP ${res.status}`);
- const archive = Buffer.from(await res.arrayBuffer());
-
- const actual = crypto.createHash("sha256").update(archive).digest("hex");
- if (actual != expected)
- throw new Error(
- `SHA-256 mismatch for ${asset}: expected ${expected}, got ${actual}`,
- );
-
- await fsp.rm(outDir, { recursive: true, force: true });
- await fsp.mkdir(outDir, { recursive: true });
- const archivePath = path.join(outDir, asset);
- await fsp.writeFile(archivePath, archive);
- // tar is available on all the platforms we build on (bsdtar on macOS and
- // Windows, GNU tar on Linux), and handles both .tar.gz and .zip.
- execFileSync("tar", ["-xf", asset], { cwd: outDir });
- await fsp.rm(archivePath);
- await fsp.writeFile(stampPath, asset);
-};
-
-/**
- * Ensure `<appDir>/build/onnxruntime` contains exactly the ONNX Runtime
- * libraries needed for the given OS/arch combination.
- *
- * On macOS the packaged app is universal, so both architectures are kept. On
- * Linux and Windows only the given arch is kept: unlike vips (a single file
- * that gets overwritten in place per arch), the libraries live in per-arch
- * directories, so directories left over from earlier passes for other
- * architectures need to be removed lest they also get copied into the
- * package.
- */
-const ensureONNXRuntime = async (platform, arch, appDir) => {
- const arches = platform == "darwin" ? ["arm64", "x64"] : [arch];
- for (const a of arches)
- await downloadONNXRuntimeIfNeeded(platform, a, appDir);
-
- const outDir = path.join(appDir, "build", "onnxruntime");
- for (const entry of await fsp.readdir(outDir))
- if (!arches.includes(entry))
- await fsp.rm(path.join(outDir, entry), { recursive: true });
-};
-
-const main = () => ensureONNXRuntime(process.platform, process.arch, ".");
-
-module.exports = { ensureONNXRuntime };
-
-if (require.main === module)
- main().catch((e) => {
- console.error(e);
- process.exit(1);
- });
diff --git a/desktop/scripts/postinstall.mjs b/desktop/scripts/postinstall.mjs
index 0e8f9e0072..a93b81a920 100644
--- a/desktop/scripts/postinstall.mjs
+++ b/desktop/scripts/postinstall.mjs
@@ -40,10 +40,6 @@ const run = (cmd) => {
run("npm rebuild --ignore-scripts=false ffmpeg-static electron-winstaller");
run("npm exec -- electron-builder install-app-deps");
run("node scripts/vips.js");
-run("node scripts/ort.js");
-// Build the Rust ML addon from source (requires a Rust toolchain). See
-// [Note: ML with Rust].
-console.log("> cargo codegen napi");
-execSync("cargo codegen napi", { cwd: "../rust", stdio: "inherit" });
+run("node scripts/ml-native.js");
writeFileSync(stamp, treeHash());
diff --git a/desktop/src/main/services/ml-native.ts b/desktop/src/main/services/ml-native.ts
index 23b016e910..9ad41dff88 100644
--- a/desktop/src/main/services/ml-native.ts
+++ b/desktop/src/main/services/ml-native.ts
@@ -11,7 +11,7 @@
* (rust/crates/photos). The desktop app talks to it through a Node native
* addon (rust/bindings/napi/photos), which loads ONNX Runtime dynamically at
* runtime from Ente's pinned custom builds (see [Note: ONNX Runtime binaries]
- * in scripts/ort.js).
+ * in scripts/ml-native.js).
*
* During development, `cargo codegen napi` (invoked by our postinstall)
* builds the addon into the gitignored `rust-bindings/` directory. When
diff --git a/infra/ml/test/run_ml_parity_tests.sh b/infra/ml/test/run_ml_parity_tests.sh
index 9cdd215187..5951ddd4f3 100755
--- a/infra/ml/test/run_ml_parity_tests.sh
+++ b/infra/ml/test/run_ml_parity_tests.sh
@@ -580,7 +580,7 @@ run_preflight_checks() {
preflight_errors+=("desktop ML addon not found under desktop/rust-bindings (run 'npm ci && npm run postinstall' in desktop/, or 'cargo codegen napi' from rust/)")
fi
if [[ ! -d "$ROOT_DIR/desktop/build/onnxruntime" ]]; then
- preflight_errors+=("ONNX Runtime library not found under desktop/build/onnxruntime (run 'node scripts/ort.js' in desktop/)")
+ preflight_errors+=("ONNX Runtime library not found under desktop/build/onnxruntime (run 'node scripts/ml-native.js' in desktop/)")
fi
;;
android|ios)
diff --git a/infra/ml/test/tools/desktop_parity_runner.js b/infra/ml/test/tools/desktop_parity_runner.js
index 7c93960645..0b10fc8d5e 100755
--- a/infra/ml/test/tools/desktop_parity_runner.js
+++ b/infra/ml/test/tools/desktop_parity_runner.js
@@ -16,7 +16,7 @@
* The desktop build artifacts it needs are produced by the desktop
* postinstall: the addon in `desktop/rust-bindings/` (via `cargo codegen
* napi`) and the ONNX Runtime library in `desktop/build/onnxruntime/` (via
- * `node scripts/ort.js`).
+ * `node scripts/ml-native.js`).
*/
const fsp = require("node:fs/promises");
@@ -62,7 +62,7 @@ const loadMLAddon = () => {
};
/**
- * The ONNX Runtime dynamic library downloaded by desktop's `scripts/ort.js`
+ * The ONNX Runtime dynamic library downloaded by desktop's `scripts/ml-native.js`
* for the current architecture, located by scanning so that this script does
* not duplicate the pinned library version.
*/
@@ -82,7 +82,7 @@ const onnxRuntimeLibraryPath = async () => {
);
if (!library)
throw new Error(
- `No ONNX Runtime library found in ${libraryDir} (run "node scripts/ort.js" in desktop/)`,
+ `No ONNX Runtime library found in ${libraryDir} (run "node scripts/ml-native.js" in desktop/)`,
);
return path.join(libraryDir, library.name);
};