Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
build: faster Rust builds with a shared compile cache, gated cloud SDKs and a lighter dev profile (#6023)
ddupont1d528b1789a2
diff --git a/docs/contributing/Development.md b/docs/contributing/Development.mdindex f95db14cef..6fc8f7d3cb 100644--- a/docs/contributing/Development.md+++ b/docs/contributing/Development.md@@ -67,6 +67,51 @@ API keys are not required for ordinary builds and deterministic tests. Addcredentials only for a test or example that explicitly calls an externalprovider, and never commit them.+## Fast Rust Builds++The Rust workspaces compile several hundred crates. Share one compile cache+between every checkout and worktree, so a crate compiled once is reused+everywhere:++```bash+brew install sccache # or: cargo install sccache --locked+export RUSTC_WRAPPER=sccache # route rustc through the cache+export SCCACHE_CACHE_SIZE=30G # least recently used objects are evicted beyond this+```++Put the exports in your shell profile. Every `cargo` command then reads and+fills the cache in `~/Library/Caches/Mozilla.sccache` (macOS) or+`~/.cache/sccache` (Linux); `sccache --show-stats` reports the hit rate. A new+worktree with an empty `target/` compiles its dependencies from the cache in a+fraction of the cold time. Dependencies are shared between all checkouts;+the workspace's own crates are cached per checkout path, and only when+incremental compilation is off. Scripted and agent builds should set+`CARGO_INCREMENTAL=0` (smaller `target/`, every crate cached); keep the+default when you iterate on one crate, where incremental rebuilds are faster.++Other defaults that keep builds small:++* The dev and test profile of `libs/cua` keeps line tables only for workspace+ crates and no debug info for dependencies; set+ `CARGO_PROFILE_DEV_DEBUG=full` for a debugger session.+* The cloud SDKs (AWS, Google Cloud, Hetzner, Modal) compile only with the+ `byoc` feature. `cargo build -p cua-cli` and `cargo test -p cua-cli` do not+ need them; the release builds turn the feature on.+* Build one package at a time (`-p <crate>`) and cap parallelism on a laptop+ (`CARGO_BUILD_JOBS=4`); the largest crates take about 1 GB of RAM each.+* The linkers are already fast: Xcode's linker on macOS and `rust-lld`, the+ default on x86-64 Linux, so the repository sets none.+* Building `cua-cli` and then `cua-sdk` in one `target/` compiles about 100+ dependencies a second time with other features (`cua-sdk` turns on `byoc`).+ This happens once per `target/` and both variants are cached, so there is+ no workspace-hack crate: it would make every build compile the cloud SDKs.++`node libs/cua/scripts/generate-uniffi-bindings.mjs` builds `cua-sdk` without+the cloud SDKs, and caches the generated bindings by a fingerprint of the+interface sources: when the `cua-sdk` crate has not changed, a rerun from any+worktree writes them without compiling. `--library=PATH` reads a+`libcua_sdk` you already built instead.+## Root Formatting HooksInstall the root Python development tools and optional Git hooks with:diff --git a/libs/cua/Cargo.toml b/libs/cua/Cargo.tomlindex e92e1b1869..8f3f3db69a 100644--- a/libs/cua/Cargo.toml+++ b/libs/cua/Cargo.toml@@ -225,6 +225,19 @@ cua-media-client = { path = "crates/cua-media-client", default-features = false[profile.release]strip = "symbols"+# Dev and test builds: line tables only for the workspace (backtraces keep+# file:line; full variable info is what made target dirs tens of GB), and+# no debug info for dependencies. macOS already keeps debug info unpacked+# (split-debuginfo = "unpacked" is cargo's default there), so linking never+# copies it. For a debugger session set CARGO_PROFILE_DEV_DEBUG=full (the+# workspace crates; dependencies stay without). The release profile is+# unchanged.+[profile.dev]+debug = "line-tables-only"++[profile.dev.package."*"]+debug = false+# SHA-256 over multi-GiB transfers is unusably slow unoptimised; keep test# builds fast without making every dependency release-mode.[profile.dev.package.sha2]diff --git a/libs/cua/crates/cua-cli/Cargo.toml b/libs/cua/crates/cua-cli/Cargo.tomlindex ca19df6cef..9ba0bb97e6 100644--- a/libs/cua/crates/cua-cli/Cargo.toml+++ b/libs/cua/crates/cua-cli/Cargo.toml@@ -99,8 +99,10 @@ cua-host = { workspace = true, features = ["testing"] }cua-daemon = { workspace = true, features = ["server", "client", "test-fixtures"] }cua-spacesd-client = { workspace = true, features = ["testing"] }cua-sandbox-core = { workspace = true, features = ["testing"] }-# The scripted cloud for `cua cloud add aws`'s create-all-or-none tests.-cua-byoc = { workspace = true, features = ["aws", "testing"] }+# The scripted cloud for `cua cloud add aws`'s create-all-or-none tests+# (built only with `byoc`, which brings the AWS provider itself, so plain+# `cargo test -p cua-cli` does not compile the AWS SDK).+cua-byoc = { workspace = true, features = ["testing"] }tempfile.workspace = true# The default log filter test emits events.tracing.workspace = truediff --git a/libs/cua/scripts/generate-uniffi-bindings.mjs b/libs/cua/scripts/generate-uniffi-bindings.mjsindex 573e4039ea..f976b3a86a 100644--- a/libs/cua/scripts/generate-uniffi-bindings.mjs+++ b/libs/cua/scripts/generate-uniffi-bindings.mjs@@ -25,11 +25,29 @@* other (dev profile) cargo steps. Pass `--release` to read* `target/release` instead, e.g. with `--skip-build` after a release build.*+ * The library is built without the `byoc` feature: it only forwards to the+ * daemon's cloud providers (the AWS SDK alone is minutes of compile time)+ * and adds nothing to the UniFFI interface.+ *+ * Avoiding the build altogether:+ *+ * - `--library=PATH` (or `CUA_SDK_LIBRARY=PATH`) reads an already built,+ * unstripped library (UniFFI reads the interface from its symbol table,+ * so CI's stripped release artifacts do not work);+ * - otherwise the generated files are cached by a fingerprint of everything+ * the interface is made from (the cua-sdk crate, VERSION, the pinned UniFFI+ * and UBRN, these scripts). A later run with the same fingerprint, from+ * any worktree, writes the cached files without compiling anything.+ * The cache lives in `$CUA_BUILD_CACHE_DIR/uniffi` (default+ * `~/.cache/cua-build`); `--no-cache` ignores it, and CI never uses it.+ ** Usage: node generate-uniffi-bindings.mjs [--check] [--skip-build]* [--release] [--only=python,swift,kotlin,typescript]+ * [--library=PATH] [--no-cache]*/import { spawnSync } from "node:child_process"+import { createHash } from "node:crypto"import {existsSync,lstatSync,@@ -42,8 +60,8 @@ import {unlinkSync,writeFileSync,} from "node:fs"-import { tmpdir } from "node:os"-import { dirname, isAbsolute, join, resolve } from "node:path"+import { homedir, tmpdir } from "node:os"+import { dirname, isAbsolute, join, relative, resolve } from "node:path"import { fileURLToPath } from "node:url"import { postprocessPython } from "./uniffi-python-postprocess.mjs"@@ -63,6 +81,8 @@ const languages = new Set(.map((language) => language.trim()).filter(Boolean),)+const libraryArgument = process.argv.find((argument) => argument.startsWith("--library="))+const useCache = !process.argv.includes("--no-cache") && !process.env.CIconst INVENTORY = ".cua-uniffi-generated-files"const CRATE = "cua_sdk"@@ -127,9 +147,56 @@ function assertOwnedPath(path) {return path}+// Every input the generated files are a function of. The interface is the+// metadata UniFFI's proc macros embed in cua-sdk (the only crate of this+// library that uses UniFFI), so other crates cannot change it.+function fingerprint() {+ const hash = createHash("sha256")+ const add = (path) => {+ hash.update(relative(cuaRoot, path))+ hash.update("\0")+ hash.update(readFileSync(path, "utf8").replaceAll("\r\n", "\n"))+ hash.update("\0")+ }+ const walk = (directory) => {+ for (const name of readdirSync(directory).sort()) {+ const path = join(directory, name)+ if (lstatSync(path).isDirectory()) walk(path)+ else add(path)+ }+ }+ walk(join(cuaRoot, "crates/cua-sdk"))+ walk(join(cuaRoot, "crates/cua-bindgen"))+ for (const path of [+ "VERSION",+ "scripts/generate-uniffi-bindings.mjs",+ "scripts/uniffi-python-postprocess.mjs",+ "typescript/package-lock.json",+ ]) {+ add(join(cuaRoot, path))+ }+ // The pinned UniFFI crates (the rest of Cargo.lock does not matter).+ const lock = readFileSync(join(cuaRoot, "Cargo.lock"), "utf8").replaceAll("\r\n", "\n")+ for (const entry of lock.split("\n[[package]]\n")) {+ if (/^name = "(uniffi[^"]*|ubrn[^"]*)"/m.test(entry)) hash.update(entry)+ }+ hash.update(`${[...languages].sort().join(",")}\0${process.env.CUA_SDK_RELEASE_VERSION ?? ""}`)+ return hash.digest("hex").slice(0, 32)+}++function cacheDirectory(key) {+ const base =+ process.env.CUA_BUILD_CACHE_DIR ??+ join(process.env.XDG_CACHE_HOME ?? join(homedir(), ".cache"), "cua-build")+ return join(base, "uniffi", key)+}++// The generated files of every group, in generation order, for the cache.+const generated = []const stale = []function applyGroup(root, files) {+ generated.push({ root: relative(cuaRoot, root), files })const inventory = join(root, INVENTORY)const previous = existsSync(inventory)? readFileSync(inventory, "utf8")@@ -197,105 +264,150 @@ function bindgen(library, language, outDir) {)}+// cua-sdk's default features without `byoc` (see the header comment).+function buildFeatures() {+ const manifest = readFileSync(join(cuaRoot, "crates/cua-sdk/Cargo.toml"), "utf8")+ const match = manifest.match(/^default\s*=\s*\[([^\]]*)\]/m)+ if (!match) throw new Error("no default features in crates/cua-sdk/Cargo.toml")+ return [...match[1].matchAll(/"([^"]+)"/g)].map((m) => m[1]).filter((f) => f !== "byoc")+}++const external = Boolean(libraryArgument || process.env.CUA_SDK_LIBRARY)+// A library someone else built (or an old one) may not match the sources:+// only a library built here fills the cache.+const key = useCache && !skipBuild && !external ? fingerprint() : null+const cached = key ? join(cacheDirectory(key), "files.json") : nullconst temporaryRoot = mkdtempSync(join(tmpdir(), "cua-uniffi-"))try {- if (!skipBuild) {- run("cargo", ["build", "--locked", ...(release ? ["--release"] : []), "-p", "cua-sdk"], cuaRoot)- }- const library = nativeLibraryPath()- if (!existsSync(library)) {- throw new Error(- `missing built SDK library ${library}` +- (release ? "" : " (pass --release to use a release build)"),- )- }-- if (languages.has("python")) {- const out = join(temporaryRoot, "python")- bindgen(library, "python", out)- applyGroup(pythonRoot, {- "src/cua/_native.py": postprocessPython(- normalizeWhitespace(readFileSync(join(out, "cua_sdk.py"), "utf8")),- ),- })- }+ if (cached && existsSync(cached)) {+ for (const group of JSON.parse(readFileSync(cached, "utf8"))) {+ applyGroup(join(cuaRoot, group.root), group.files)+ }+ console.log(`UniFFI interface unchanged (cache ${key}); nothing compiled.`)+ } else {+ let library = libraryArgument+ ? resolve(libraryArgument.slice("--library=".length))+ : process.env.CUA_SDK_LIBRARY+ ? resolve(process.env.CUA_SDK_LIBRARY)+ : null+ if (!library && !skipBuild) {+ run(+ "cargo",+ [+ "build",+ "--locked",+ ...(release ? ["--release"] : []),+ "-p",+ "cua-sdk",+ "--no-default-features",+ "--features",+ buildFeatures().join(","),+ ],+ cuaRoot,+ )+ }+ library ??= nativeLibraryPath()+ if (!existsSync(library)) {+ throw new Error(+ `missing built SDK library ${library}` ++ (release ? "" : " (pass --release to use a release build)"),+ )+ }- if (languages.has("swift")) {- const out = join(temporaryRoot, "swift")- bindgen(library, "swift", out)- applyGroup(swiftRoot, {- "Sources/CuaSDK/CuaSDK.swift": normalizeWhitespace(- readFileSync(join(out, "CuaSDK.swift"), "utf8"),- ),- "Sources/cua_sdkFFI/include/cua_sdkFFI.h": normalizeWhitespace(- readFileSync(join(out, "cua_sdkFFI.h"), "utf8"),- ),- // SwiftPM requires the map to be named `module.modulemap` and to point- // at the header by its path inside the system-library target. This is- // the one place generated output is edited.- "Sources/cua_sdkFFI/module.modulemap": normalizeWhitespace(- readFileSync(join(out, "cua_sdkFFI.modulemap"), "utf8").replace(- 'header "cua_sdkFFI.h"',- 'header "include/cua_sdkFFI.h"',+ if (languages.has("python")) {+ const out = join(temporaryRoot, "python")+ bindgen(library, "python", out)+ applyGroup(pythonRoot, {+ "src/cua/_native.py": postprocessPython(+ normalizeWhitespace(readFileSync(join(out, "cua_sdk.py"), "utf8")),),- ),- })- }+ })+ }- if (languages.has("kotlin")) {- const out = join(temporaryRoot, "kotlin")- bindgen(library, "kotlin", out)- const relative = "ai/cua/sdk/cua_sdk.kt"- applyGroup(kotlinRoot, {- [`src/main/kotlin/${relative}`]: normalizeWhitespace(- readFileSync(join(out, relative), "utf8"),- ),- })- }+ if (languages.has("swift")) {+ const out = join(temporaryRoot, "swift")+ bindgen(library, "swift", out)+ applyGroup(swiftRoot, {+ "Sources/CuaSDK/CuaSDK.swift": normalizeWhitespace(+ readFileSync(join(out, "CuaSDK.swift"), "utf8"),+ ),+ "Sources/cua_sdkFFI/include/cua_sdkFFI.h": normalizeWhitespace(+ readFileSync(join(out, "cua_sdkFFI.h"), "utf8"),+ ),+ // SwiftPM requires the map to be named `module.modulemap` and to point+ // at the header by its path inside the system-library target. This is+ // the one place generated output is edited.+ "Sources/cua_sdkFFI/module.modulemap": normalizeWhitespace(+ readFileSync(join(out, "cua_sdkFFI.modulemap"), "utf8").replace(+ 'header "cua_sdkFFI.h"',+ 'header "include/cua_sdkFFI.h"',+ ),+ ),+ })+ }- if (languages.has("typescript")) {- const out = join(temporaryRoot, "typescript")- const ubrn = join(- typescriptRoot,- "node_modules",- ".bin",- process.platform === "win32" ? "ubrn.cmd" : "ubrn",- )- if (!existsSync(ubrn)) {- throw new Error("missing pinned UBRN generator; run npm ci in libs/cua/typescript")+ if (languages.has("kotlin")) {+ const out = join(temporaryRoot, "kotlin")+ bindgen(library, "kotlin", out)+ const relative = "ai/cua/sdk/cua_sdk.kt"+ applyGroup(kotlinRoot, {+ [`src/main/kotlin/${relative}`]: normalizeWhitespace(+ readFileSync(join(out, relative), "utf8"),+ ),+ })}- run(- ubrn,- [- "generate",- "napi",- "bindings",- "--library",- library,- "--ts-dir",- out,- "--lib-package-base",- "@trycua/cua",- "--lib-node-triple",- "--no-format",- ],- // UBRN shells out to `cargo metadata` in its working directory.- cuaRoot,- )- const files = {}- const names = readdirSync(out).filter((name) => name.endsWith(".ts"))- for (const name of names) {- if (name === "index.ts") {- // UBRN's initializer.- files["src/native/index.ts"] = normalizeTypeScript(name, readFileSync(join(out, name), "utf8"))- continue++ if (languages.has("typescript")) {+ const out = join(temporaryRoot, "typescript")+ const ubrn = join(+ typescriptRoot,+ "node_modules",+ ".bin",+ process.platform === "win32" ? "ubrn.cmd" : "ubrn",+ )+ if (!existsSync(ubrn)) {+ throw new Error("missing pinned UBRN generator; run npm ci in libs/cua/typescript")+ }+ run(+ ubrn,+ [+ "generate",+ "napi",+ "bindings",+ "--library",+ library,+ "--ts-dir",+ out,+ "--lib-package-base",+ "@trycua/cua",+ "--lib-node-triple",+ "--no-format",+ ],+ // UBRN shells out to `cargo metadata` in its working directory.+ cuaRoot,+ )+ const files = {}+ const names = readdirSync(out).filter((name) => name.endsWith(".ts"))+ for (const name of names) {+ if (name === "index.ts") {+ // UBRN's initializer.+ files["src/native/index.ts"] = normalizeTypeScript(name, readFileSync(join(out, name), "utf8"))+ continue+ }+ // Only the cua_sdk namespace (see the header comment).+ if (!name.startsWith(`${CRATE}.`) && !name.startsWith(`${CRATE}-`)) continue+ files[`src/native/${name}`] = normalizeTypeScript(name, readFileSync(join(out, name), "utf8"))}- // Only the cua_sdk namespace (see the header comment).- if (!name.startsWith(`${CRATE}.`) && !name.startsWith(`${CRATE}-`)) continue- files[`src/native/${name}`] = normalizeTypeScript(name, readFileSync(join(out, name), "utf8"))+ if (!files[`src/native/${CRATE}.ts`]) throw new Error("UBRN produced no cua_sdk.ts")+ applyGroup(typescriptRoot, files)+ }++ if (cached) {+ mkdirSync(dirname(cached), { recursive: true })+ const temporary = `${cached}.tmp.${process.pid}`+ writeFileSync(temporary, JSON.stringify(generated))+ renameSync(temporary, cached)}- if (!files[`src/native/${CRATE}.ts`]) throw new Error("UBRN produced no cua_sdk.ts")- applyGroup(typescriptRoot, files)}if (languages.has("typescript")) {