Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

build: faster Rust builds with a shared compile cache, gated cloud SDKs and a lighter dev profile (#6023)

ddupont1d528b1789a2
diff --git a/docs/contributing/Development.md b/docs/contributing/Development.md
index f95db14cef..6fc8f7d3cb 100644
--- a/docs/contributing/Development.md
+++ b/docs/contributing/Development.md
@@ -67,6 +67,51 @@ API keys are not required for ordinary builds and deterministic tests. Add
credentials only for a test or example that explicitly calls an external
provider, and never commit them.
+## Fast Rust Builds
+
+The Rust workspaces compile several hundred crates. Share one compile cache
+between every checkout and worktree, so a crate compiled once is reused
+everywhere:
+
+```bash
+brew install sccache # or: cargo install sccache --locked
+export RUSTC_WRAPPER=sccache # route rustc through the cache
+export SCCACHE_CACHE_SIZE=30G # least recently used objects are evicted beyond this
+```
+
+Put the exports in your shell profile. Every `cargo` command then reads and
+fills the cache in `~/Library/Caches/Mozilla.sccache` (macOS) or
+`~/.cache/sccache` (Linux); `sccache --show-stats` reports the hit rate. A new
+worktree with an empty `target/` compiles its dependencies from the cache in a
+fraction of the cold time. Dependencies are shared between all checkouts;
+the workspace's own crates are cached per checkout path, and only when
+incremental compilation is off. Scripted and agent builds should set
+`CARGO_INCREMENTAL=0` (smaller `target/`, every crate cached); keep the
+default when you iterate on one crate, where incremental rebuilds are faster.
+
+Other defaults that keep builds small:
+
+* The dev and test profile of `libs/cua` keeps line tables only for workspace
+ crates and no debug info for dependencies; set
+ `CARGO_PROFILE_DEV_DEBUG=full` for a debugger session.
+* The cloud SDKs (AWS, Google Cloud, Hetzner, Modal) compile only with the
+ `byoc` feature. `cargo build -p cua-cli` and `cargo test -p cua-cli` do not
+ need them; the release builds turn the feature on.
+* Build one package at a time (`-p <crate>`) and cap parallelism on a laptop
+ (`CARGO_BUILD_JOBS=4`); the largest crates take about 1 GB of RAM each.
+* The linkers are already fast: Xcode's linker on macOS and `rust-lld`, the
+ default on x86-64 Linux, so the repository sets none.
+* Building `cua-cli` and then `cua-sdk` in one `target/` compiles about 100
+ dependencies a second time with other features (`cua-sdk` turns on `byoc`).
+ This happens once per `target/` and both variants are cached, so there is
+ no workspace-hack crate: it would make every build compile the cloud SDKs.
+
+`node libs/cua/scripts/generate-uniffi-bindings.mjs` builds `cua-sdk` without
+the cloud SDKs, and caches the generated bindings by a fingerprint of the
+interface sources: when the `cua-sdk` crate has not changed, a rerun from any
+worktree writes them without compiling. `--library=PATH` reads a
+`libcua_sdk` you already built instead.
+
## Root Formatting Hooks
Install the root Python development tools and optional Git hooks with:
diff --git a/libs/cua/Cargo.toml b/libs/cua/Cargo.toml
index e92e1b1869..8f3f3db69a 100644
--- a/libs/cua/Cargo.toml
+++ b/libs/cua/Cargo.toml
@@ -225,6 +225,19 @@ cua-media-client = { path = "crates/cua-media-client", default-features = false
[profile.release]
strip = "symbols"
+# Dev and test builds: line tables only for the workspace (backtraces keep
+# file:line; full variable info is what made target dirs tens of GB), and
+# no debug info for dependencies. macOS already keeps debug info unpacked
+# (split-debuginfo = "unpacked" is cargo's default there), so linking never
+# copies it. For a debugger session set CARGO_PROFILE_DEV_DEBUG=full (the
+# workspace crates; dependencies stay without). The release profile is
+# unchanged.
+[profile.dev]
+debug = "line-tables-only"
+
+[profile.dev.package."*"]
+debug = false
+
# SHA-256 over multi-GiB transfers is unusably slow unoptimised; keep test
# builds fast without making every dependency release-mode.
[profile.dev.package.sha2]
diff --git a/libs/cua/crates/cua-cli/Cargo.toml b/libs/cua/crates/cua-cli/Cargo.toml
index ca19df6cef..9ba0bb97e6 100644
--- a/libs/cua/crates/cua-cli/Cargo.toml
+++ b/libs/cua/crates/cua-cli/Cargo.toml
@@ -99,8 +99,10 @@ cua-host = { workspace = true, features = ["testing"] }
cua-daemon = { workspace = true, features = ["server", "client", "test-fixtures"] }
cua-spacesd-client = { workspace = true, features = ["testing"] }
cua-sandbox-core = { workspace = true, features = ["testing"] }
-# The scripted cloud for `cua cloud add aws`'s create-all-or-none tests.
-cua-byoc = { workspace = true, features = ["aws", "testing"] }
+# The scripted cloud for `cua cloud add aws`'s create-all-or-none tests
+# (built only with `byoc`, which brings the AWS provider itself, so plain
+# `cargo test -p cua-cli` does not compile the AWS SDK).
+cua-byoc = { workspace = true, features = ["testing"] }
tempfile.workspace = true
# The default log filter test emits events.
tracing.workspace = true
diff --git a/libs/cua/scripts/generate-uniffi-bindings.mjs b/libs/cua/scripts/generate-uniffi-bindings.mjs
index 573e4039ea..f976b3a86a 100644
--- a/libs/cua/scripts/generate-uniffi-bindings.mjs
+++ b/libs/cua/scripts/generate-uniffi-bindings.mjs
@@ -25,11 +25,29 @@
* other (dev profile) cargo steps. Pass `--release` to read
* `target/release` instead, e.g. with `--skip-build` after a release build.
*
+ * The library is built without the `byoc` feature: it only forwards to the
+ * daemon's cloud providers (the AWS SDK alone is minutes of compile time)
+ * and adds nothing to the UniFFI interface.
+ *
+ * Avoiding the build altogether:
+ *
+ * - `--library=PATH` (or `CUA_SDK_LIBRARY=PATH`) reads an already built,
+ * unstripped library (UniFFI reads the interface from its symbol table,
+ * so CI's stripped release artifacts do not work);
+ * - otherwise the generated files are cached by a fingerprint of everything
+ * the interface is made from (the cua-sdk crate, VERSION, the pinned UniFFI
+ * and UBRN, these scripts). A later run with the same fingerprint, from
+ * any worktree, writes the cached files without compiling anything.
+ * The cache lives in `$CUA_BUILD_CACHE_DIR/uniffi` (default
+ * `~/.cache/cua-build`); `--no-cache` ignores it, and CI never uses it.
+ *
* Usage: node generate-uniffi-bindings.mjs [--check] [--skip-build]
* [--release] [--only=python,swift,kotlin,typescript]
+ * [--library=PATH] [--no-cache]
*/
import { spawnSync } from "node:child_process"
+import { createHash } from "node:crypto"
import {
existsSync,
lstatSync,
@@ -42,8 +60,8 @@ import {
unlinkSync,
writeFileSync,
} from "node:fs"
-import { tmpdir } from "node:os"
-import { dirname, isAbsolute, join, resolve } from "node:path"
+import { homedir, tmpdir } from "node:os"
+import { dirname, isAbsolute, join, relative, resolve } from "node:path"
import { fileURLToPath } from "node:url"
import { postprocessPython } from "./uniffi-python-postprocess.mjs"
@@ -63,6 +81,8 @@ const languages = new Set(
.map((language) => language.trim())
.filter(Boolean),
)
+const libraryArgument = process.argv.find((argument) => argument.startsWith("--library="))
+const useCache = !process.argv.includes("--no-cache") && !process.env.CI
const INVENTORY = ".cua-uniffi-generated-files"
const CRATE = "cua_sdk"
@@ -127,9 +147,56 @@ function assertOwnedPath(path) {
return path
}
+// Every input the generated files are a function of. The interface is the
+// metadata UniFFI's proc macros embed in cua-sdk (the only crate of this
+// library that uses UniFFI), so other crates cannot change it.
+function fingerprint() {
+ const hash = createHash("sha256")
+ const add = (path) => {
+ hash.update(relative(cuaRoot, path))
+ hash.update("\0")
+ hash.update(readFileSync(path, "utf8").replaceAll("\r\n", "\n"))
+ hash.update("\0")
+ }
+ const walk = (directory) => {
+ for (const name of readdirSync(directory).sort()) {
+ const path = join(directory, name)
+ if (lstatSync(path).isDirectory()) walk(path)
+ else add(path)
+ }
+ }
+ walk(join(cuaRoot, "crates/cua-sdk"))
+ walk(join(cuaRoot, "crates/cua-bindgen"))
+ for (const path of [
+ "VERSION",
+ "scripts/generate-uniffi-bindings.mjs",
+ "scripts/uniffi-python-postprocess.mjs",
+ "typescript/package-lock.json",
+ ]) {
+ add(join(cuaRoot, path))
+ }
+ // The pinned UniFFI crates (the rest of Cargo.lock does not matter).
+ const lock = readFileSync(join(cuaRoot, "Cargo.lock"), "utf8").replaceAll("\r\n", "\n")
+ for (const entry of lock.split("\n[[package]]\n")) {
+ if (/^name = "(uniffi[^"]*|ubrn[^"]*)"/m.test(entry)) hash.update(entry)
+ }
+ hash.update(`${[...languages].sort().join(",")}\0${process.env.CUA_SDK_RELEASE_VERSION ?? ""}`)
+ return hash.digest("hex").slice(0, 32)
+}
+
+function cacheDirectory(key) {
+ const base =
+ process.env.CUA_BUILD_CACHE_DIR ??
+ join(process.env.XDG_CACHE_HOME ?? join(homedir(), ".cache"), "cua-build")
+ return join(base, "uniffi", key)
+}
+
+// The generated files of every group, in generation order, for the cache.
+const generated = []
const stale = []
function applyGroup(root, files) {
+ generated.push({ root: relative(cuaRoot, root), files })
const inventory = join(root, INVENTORY)
const previous = existsSync(inventory)
? readFileSync(inventory, "utf8")
@@ -197,105 +264,150 @@ function bindgen(library, language, outDir) {
)
}
+// cua-sdk's default features without `byoc` (see the header comment).
+function buildFeatures() {
+ const manifest = readFileSync(join(cuaRoot, "crates/cua-sdk/Cargo.toml"), "utf8")
+ const match = manifest.match(/^default\s*=\s*\[([^\]]*)\]/m)
+ if (!match) throw new Error("no default features in crates/cua-sdk/Cargo.toml")
+ return [...match[1].matchAll(/"([^"]+)"/g)].map((m) => m[1]).filter((f) => f !== "byoc")
+}
+
+const external = Boolean(libraryArgument || process.env.CUA_SDK_LIBRARY)
+// A library someone else built (or an old one) may not match the sources:
+// only a library built here fills the cache.
+const key = useCache && !skipBuild && !external ? fingerprint() : null
+const cached = key ? join(cacheDirectory(key), "files.json") : null
const temporaryRoot = mkdtempSync(join(tmpdir(), "cua-uniffi-"))
try {
- if (!skipBuild) {
- run("cargo", ["build", "--locked", ...(release ? ["--release"] : []), "-p", "cua-sdk"], cuaRoot)
- }
- const library = nativeLibraryPath()
- if (!existsSync(library)) {
- throw new Error(
- `missing built SDK library ${library}` +
- (release ? "" : " (pass --release to use a release build)"),
- )
- }
-
- if (languages.has("python")) {
- const out = join(temporaryRoot, "python")
- bindgen(library, "python", out)
- applyGroup(pythonRoot, {
- "src/cua/_native.py": postprocessPython(
- normalizeWhitespace(readFileSync(join(out, "cua_sdk.py"), "utf8")),
- ),
- })
- }
+ if (cached && existsSync(cached)) {
+ for (const group of JSON.parse(readFileSync(cached, "utf8"))) {
+ applyGroup(join(cuaRoot, group.root), group.files)
+ }
+ console.log(`UniFFI interface unchanged (cache ${key}); nothing compiled.`)
+ } else {
+ let library = libraryArgument
+ ? resolve(libraryArgument.slice("--library=".length))
+ : process.env.CUA_SDK_LIBRARY
+ ? resolve(process.env.CUA_SDK_LIBRARY)
+ : null
+ if (!library && !skipBuild) {
+ run(
+ "cargo",
+ [
+ "build",
+ "--locked",
+ ...(release ? ["--release"] : []),
+ "-p",
+ "cua-sdk",
+ "--no-default-features",
+ "--features",
+ buildFeatures().join(","),
+ ],
+ cuaRoot,
+ )
+ }
+ library ??= nativeLibraryPath()
+ if (!existsSync(library)) {
+ throw new Error(
+ `missing built SDK library ${library}` +
+ (release ? "" : " (pass --release to use a release build)"),
+ )
+ }
- if (languages.has("swift")) {
- const out = join(temporaryRoot, "swift")
- bindgen(library, "swift", out)
- applyGroup(swiftRoot, {
- "Sources/CuaSDK/CuaSDK.swift": normalizeWhitespace(
- readFileSync(join(out, "CuaSDK.swift"), "utf8"),
- ),
- "Sources/cua_sdkFFI/include/cua_sdkFFI.h": normalizeWhitespace(
- readFileSync(join(out, "cua_sdkFFI.h"), "utf8"),
- ),
- // SwiftPM requires the map to be named `module.modulemap` and to point
- // at the header by its path inside the system-library target. This is
- // the one place generated output is edited.
- "Sources/cua_sdkFFI/module.modulemap": normalizeWhitespace(
- readFileSync(join(out, "cua_sdkFFI.modulemap"), "utf8").replace(
- 'header "cua_sdkFFI.h"',
- 'header "include/cua_sdkFFI.h"',
+ if (languages.has("python")) {
+ const out = join(temporaryRoot, "python")
+ bindgen(library, "python", out)
+ applyGroup(pythonRoot, {
+ "src/cua/_native.py": postprocessPython(
+ normalizeWhitespace(readFileSync(join(out, "cua_sdk.py"), "utf8")),
),
- ),
- })
- }
+ })
+ }
- if (languages.has("kotlin")) {
- const out = join(temporaryRoot, "kotlin")
- bindgen(library, "kotlin", out)
- const relative = "ai/cua/sdk/cua_sdk.kt"
- applyGroup(kotlinRoot, {
- [`src/main/kotlin/${relative}`]: normalizeWhitespace(
- readFileSync(join(out, relative), "utf8"),
- ),
- })
- }
+ if (languages.has("swift")) {
+ const out = join(temporaryRoot, "swift")
+ bindgen(library, "swift", out)
+ applyGroup(swiftRoot, {
+ "Sources/CuaSDK/CuaSDK.swift": normalizeWhitespace(
+ readFileSync(join(out, "CuaSDK.swift"), "utf8"),
+ ),
+ "Sources/cua_sdkFFI/include/cua_sdkFFI.h": normalizeWhitespace(
+ readFileSync(join(out, "cua_sdkFFI.h"), "utf8"),
+ ),
+ // SwiftPM requires the map to be named `module.modulemap` and to point
+ // at the header by its path inside the system-library target. This is
+ // the one place generated output is edited.
+ "Sources/cua_sdkFFI/module.modulemap": normalizeWhitespace(
+ readFileSync(join(out, "cua_sdkFFI.modulemap"), "utf8").replace(
+ 'header "cua_sdkFFI.h"',
+ 'header "include/cua_sdkFFI.h"',
+ ),
+ ),
+ })
+ }
- if (languages.has("typescript")) {
- const out = join(temporaryRoot, "typescript")
- const ubrn = join(
- typescriptRoot,
- "node_modules",
- ".bin",
- process.platform === "win32" ? "ubrn.cmd" : "ubrn",
- )
- if (!existsSync(ubrn)) {
- throw new Error("missing pinned UBRN generator; run npm ci in libs/cua/typescript")
+ if (languages.has("kotlin")) {
+ const out = join(temporaryRoot, "kotlin")
+ bindgen(library, "kotlin", out)
+ const relative = "ai/cua/sdk/cua_sdk.kt"
+ applyGroup(kotlinRoot, {
+ [`src/main/kotlin/${relative}`]: normalizeWhitespace(
+ readFileSync(join(out, relative), "utf8"),
+ ),
+ })
}
- run(
- ubrn,
- [
- "generate",
- "napi",
- "bindings",
- "--library",
- library,
- "--ts-dir",
- out,
- "--lib-package-base",
- "@trycua/cua",
- "--lib-node-triple",
- "--no-format",
- ],
- // UBRN shells out to `cargo metadata` in its working directory.
- cuaRoot,
- )
- const files = {}
- const names = readdirSync(out).filter((name) => name.endsWith(".ts"))
- for (const name of names) {
- if (name === "index.ts") {
- // UBRN's initializer.
- files["src/native/index.ts"] = normalizeTypeScript(name, readFileSync(join(out, name), "utf8"))
- continue
+
+ if (languages.has("typescript")) {
+ const out = join(temporaryRoot, "typescript")
+ const ubrn = join(
+ typescriptRoot,
+ "node_modules",
+ ".bin",
+ process.platform === "win32" ? "ubrn.cmd" : "ubrn",
+ )
+ if (!existsSync(ubrn)) {
+ throw new Error("missing pinned UBRN generator; run npm ci in libs/cua/typescript")
+ }
+ run(
+ ubrn,
+ [
+ "generate",
+ "napi",
+ "bindings",
+ "--library",
+ library,
+ "--ts-dir",
+ out,
+ "--lib-package-base",
+ "@trycua/cua",
+ "--lib-node-triple",
+ "--no-format",
+ ],
+ // UBRN shells out to `cargo metadata` in its working directory.
+ cuaRoot,
+ )
+ const files = {}
+ const names = readdirSync(out).filter((name) => name.endsWith(".ts"))
+ for (const name of names) {
+ if (name === "index.ts") {
+ // UBRN's initializer.
+ files["src/native/index.ts"] = normalizeTypeScript(name, readFileSync(join(out, name), "utf8"))
+ continue
+ }
+ // Only the cua_sdk namespace (see the header comment).
+ if (!name.startsWith(`${CRATE}.`) && !name.startsWith(`${CRATE}-`)) continue
+ files[`src/native/${name}`] = normalizeTypeScript(name, readFileSync(join(out, name), "utf8"))
}
- // Only the cua_sdk namespace (see the header comment).
- if (!name.startsWith(`${CRATE}.`) && !name.startsWith(`${CRATE}-`)) continue
- files[`src/native/${name}`] = normalizeTypeScript(name, readFileSync(join(out, name), "utf8"))
+ if (!files[`src/native/${CRATE}.ts`]) throw new Error("UBRN produced no cua_sdk.ts")
+ applyGroup(typescriptRoot, files)
+ }
+
+ if (cached) {
+ mkdirSync(dirname(cached), { recursive: true })
+ const temporary = `${cached}.tmp.${process.pid}`
+ writeFileSync(temporary, JSON.stringify(generated))
+ renameSync(temporary, cached)
}
- if (!files[`src/native/${CRATE}.ts`]) throw new Error("UBRN produced no cua_sdk.ts")
- applyGroup(typescriptRoot, files)
}
if (languages.has("typescript")) {