The open-source app everyone uses to manage agents at work
TypeScript acpxagent-orchestrationagent-skillsagentic-aiai-agentsai-engineeringai-governanceai-toolsautonomous-agentsclaudeclaude-codeclaude-skillscodexhermesllmmcpmulti-agentopenclawproject-managementworkflow-automation
ci: cache compiled Docker Rust dependencies separately from source (#13329)
Devin Foley7435b2ee9cc6
diff --git a/.github/workflows/docker-runner-check.yml b/.github/workflows/docker-runner-check.ymlindex e9705c3b5e..7c3b7820b5 100644--- a/.github/workflows/docker-runner-check.yml+++ b/.github/workflows/docker-runner-check.yml@@ -6,6 +6,7 @@ on:- .github/workflows/docker-runner-check.yml- Dockerfile- .dockerignore+ - scripts/check-docker-runner-cache.sh- packages/paperclip-runner/rust-toolchain.toml- packages/paperclip-runner/runner/**- packages/paperclip-runner/protocol/**@@ -20,7 +21,7 @@ jobs:runner:name: Compile isolated native Runnerruns-on: ubuntu-latest- timeout-minutes: 15+ timeout-minutes: 20permissions:contents: readsteps:@@ -30,8 +31,8 @@ jobs:persist-credentials: false- name: Set up Docker Buildxuses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4- # Compile the real target with the real .dockerignore. This catches new- # Cargo or embedded protocol inputs that the isolated COPY set omits.- # No registry credentials, cache imports/exports, or image publication.- - name: Compile the Runner from its isolated Docker context- run: docker buildx build --target runner-build --progress plain .+ # Compile the real target, then change source in a disposable context.+ # Require dependency-layer reuse and changed metadata from the real binary.+ # No registry credentials, external cache, or image publication.+ - name: Verify native build and dependency cache reuse+ run: bash scripts/check-docker-runner-cache.shdiff --git a/Dockerfile b/Dockerfileindex b349251238..a3d046c96e 100644--- a/Dockerfile+++ b/Dockerfile@@ -88,7 +88,27 @@ RUN set -eux; \COPY packages/paperclip-runner/rust-toolchain.toml /tmp/runner-toolchain/rust-toolchain.tomlRUN cd /tmp/runner-toolchain && rustup show-FROM rust-toolchain AS runner-build+# Pin the recipe generator and its dependency lockfile. It is a build-only tool+# and uses the same package-owned compiler as both native build stages.+FROM rust-toolchain AS rust-chef+RUN cd /tmp/runner-toolchain && cargo install cargo-chef --version 0.1.73 --locked++FROM rust-chef AS runner-plan+WORKDIR /app/packages/paperclip-runner+COPY packages/paperclip-runner/rust-toolchain.toml ./+COPY packages/paperclip-runner/runner ./runner+RUN cd runner && cargo chef prepare --recipe-path /tmp/runner-recipe.json++FROM rust-chef AS runner-deps+WORKDIR /app/packages/paperclip-runner/runner+COPY packages/paperclip-runner/rust-toolchain.toml ../+# The recipe changes only when dependency manifests, the lockfile, or target+# metadata change. Source edits can reuse this compiled dependency layer.+COPY --from=runner-plan /tmp/runner-recipe.json /tmp/runner-recipe.json+RUN cargo chef cook --release --locked --package paperclip-runner-core --bin paperclip-runnerd --recipe-path /tmp/runner-recipe.json \+ && find . -mindepth 1 -maxdepth 1 ! -name target -exec rm -rf {} +++FROM runner-deps AS runner-buildWORKDIR /app/packages/paperclip-runner# Rust embeds protocol schemas and fixtures with include_str!. Keep those# alongside the complete Cargo workspace so every compile-time input keysdiff --git a/doc/DOCKER.md b/doc/DOCKER.mdindex 757a2c3094..2905bfe957 100644--- a/doc/DOCKER.md+++ b/doc/DOCKER.md@@ -342,11 +342,20 @@ Notes:## Native Runner build cacheThe image compiles the native Runner in `runner-build`, before copying the-application source. That stage includes the pinned Rust compiler, the complete-Cargo workspace and lockfile, and the protocol schemas and fixtures embedded-by Rust. Changes to those inputs rebuild the native binary. Ordinary server or-UI changes can reuse it through the existing registry cache (`mode=max`). Each-platform gets its own native build; no cross-architecture binary is reused.+application source. A pinned `cargo-chef` generates a dependency recipe in+`runner-plan`. The separate `runner-deps` stage compiles that recipe with the+package-owned Rust compiler. Both the dependency build and the real binary use+the release profile and locked Cargo dependencies. The recipe stage never+modifies source in the checkout.++Changes to Rust source or embedded protocol inputs rebuild the real binary but+can reuse compiled dependencies when the recipe is unchanged. Dependency+manifests, the Cargo lockfile, target metadata, or compiler changes invalidate+the relevant cache. Ordinary server or UI changes can reuse the entire native+build through the existing registry cache (`mode=max`). Each platform gets its+own native build; no cross-architecture binary is reused. No additional GitHub+Actions cache is created. A cold build also installs the recipe generator and+compiles dependencies, so the savings apply after those layers are available.The application build inherits that stage and still runs the normal serverbuild, including Cargo, binary staging, and generated-contract checks. Rust@@ -357,6 +366,13 @@ directory as before. Cache misses only cost compilation time.Pull requests that change the Dockerfile, Docker ignore rules, or Runner nativeinputs also build the isolated `runner-build` target in `Docker Runner check`.-This compiles against the actual reduced context and catches missing embedded-inputs before the post-merge image build. It uses a GitHub-hosted runner with-read-only repository access and does not publish images or cache artifacts.+The check runs `bash scripts/check-docker-runner-cache.sh` against a disposable+copy of tracked source and the actual Docker ignore rules. It compiles a baseline,+changes a Rust metadata constant, and rebuilds. It requires a cached dependency+build, an unchanged dependency recipe, and changed metadata from the real binary.+It also verifies that a dependency declaration change alters the recipe. The+probe exports only small metadata files, avoiding a large image import into the+Docker daemon. It catches missing embedded inputs before the post-merge build.+It uses a GitHub-hosted runner with read-only repository access and does not+publish images or cache artifacts. Allow up to 20 minutes for its cold build and+source rebuild.diff --git a/scripts/check-docker-runner-cache.sh b/scripts/check-docker-runner-cache.shnew file mode 100644index 0000000000..0c8caf81d5--- /dev/null+++ b/scripts/check-docker-runner-cache.sh@@ -0,0 +1,60 @@+#!/usr/bin/env bash+# Build the real Docker target twice in a disposable copy of tracked source.+# Export only metadata, avoiding a multi-gigabyte test image in the daemon.+set -euo pipefail+repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"+probe_dir="$(mktemp -d "${TMPDIR:-/tmp}/paperclip-runner-cache.XXXXXX")"+trap 'rm -rf "$probe_dir"' EXIT+mkdir "$probe_dir/context"+cd "$repo_root"+git ls-files -z | tar -cf - --null -T - | tar -xf - -C "$probe_dir/context"+cd "$probe_dir/context"+export PROBE_DIR="$probe_dir"+cp Dockerfile "$probe_dir/cache-probe.Dockerfile"+cat >> "$probe_dir/cache-probe.Dockerfile" <<'DOCKER'+FROM runner-build AS cache-proof+RUN ./runner/target/release/paperclip-runnerd --build-metadata > /metadata.json+FROM scratch AS cache-proof-export+COPY --from=cache-proof /metadata.json /metadata.json+COPY --from=runner-plan /tmp/runner-recipe.json /recipe.json+FROM scratch AS recipe-proof-export+COPY --from=runner-plan /tmp/runner-recipe.json /recipe.json+DOCKER+build_proof() {+ docker buildx build --file "$probe_dir/cache-probe.Dockerfile" --target cache-proof-export --output "type=local,dest=$probe_dir/$1" --progress plain . 2>&1 | tee "$probe_dir/$1.log"+}+build_proof baseline+python3 - <<'CHECK'+from pathlib import Path+p=Path('packages/paperclip-runner/runner/crates/runner-core/src/bin/paperclip-runnerd.rs')+s=p.read_text(); needle='paperclip-runner/runnerd-build-metadata/v1'+assert s.count(needle)==1+p.write_text(s.replace(needle,needle+'-cache-probe'))+CHECK+build_proof source-change+python3 - <<'CHECK'+import os,json,re+from pathlib import Path+root=Path(os.environ['PROBE_DIR'])+before=json.loads((root/'baseline/metadata.json').read_text())+after=json.loads((root/'source-change/metadata.json').read_text())+assert before['schema']=='paperclip-runner/runnerd-build-metadata/v1'+assert after['schema']==before['schema']+'-cache-probe'+assert (root/'baseline/recipe.json').read_bytes()==(root/'source-change/recipe.json').read_bytes()+log=(root/'source-change.log').read_text()+step=re.search(r'#(\d+) \[runner-deps[^\n]+ RUN cargo chef cook',log)[1]+assert f'#{step} CACHED' in log+assert 'Compiling paperclip-runner-core' in log+print('PASS: unchanged dependency recipe and cached cook layer; real binary changed.')+p=Path('packages/paperclip-runner/runner/Cargo.toml')+s=p.read_text(); assert 'serde_json = "1.0"' in s+p.write_text(s.replace('serde_json = "1.0"','serde_json = ">=1.0.0, <2.0.0"'))+CHECK+docker buildx build --file "$probe_dir/cache-probe.Dockerfile" --target recipe-proof-export --output "type=local,dest=$probe_dir/manifest-change" --progress plain .+python3 - <<'CHECK'+from pathlib import Path+import os+root=Path(os.environ['PROBE_DIR'])+assert (root/'source-change/recipe.json').read_bytes()!=(root/'manifest-change/recipe.json').read_bytes()+print('PASS: dependency declaration change invalidates the recipe.')+CHECKdiff --git a/server/src/__tests__/docker-build-stamp.test.ts b/server/src/__tests__/docker-build-stamp.test.tsindex 4794989feb..76862952c8 100644--- a/server/src/__tests__/docker-build-stamp.test.ts+++ b/server/src/__tests__/docker-build-stamp.test.ts@@ -76,3 +76,32 @@ describe("docker build-stamp wiring", () => {).toBeGreaterThanOrEqual(2);});});+++describe("Docker Rust dependency cache", () => {+ it("caches the locked dependency recipe separately from source and per-build metadata", () => {+ const chef = stageBody(dockerfile, "rust-chef");+ const planner = stageBody(dockerfile, "runner-plan");+ const dependencies = stageBody(dockerfile, "runner-deps");+ expect(chef).toContain("FROM rust-toolchain AS rust-chef");+ expect(chef).toMatch(/cargo install cargo-chef --version \d+\.\d+\.\d+ --locked/);+ expect(planner).toContain("COPY packages/paperclip-runner/runner ./runner");+ expect(planner).toContain("cargo chef prepare --recipe-path /tmp/runner-recipe.json");+ expect(dependencies).toContain("FROM rust-chef AS runner-deps");+ expect(dependencies).toContain("COPY --from=runner-plan /tmp/runner-recipe.json /tmp/runner-recipe.json");+ expect(dependencies).toContain("cargo chef cook --release --locked --package paperclip-runner-core --bin paperclip-runnerd");+ expect(dependencies).not.toMatch(/COPY .*\.\/runner|COPY .*\.\/protocol|COPY \. \.|PAPERCLIP_BUILD_COMMIT/);+ });++ it("rebuilds real workspace code and embedded protocol inputs after cooking dependencies", () => {+ const native = stageBody(dockerfile, "runner-build");+ expect(native).toContain("FROM runner-deps AS runner-build");+ for (const source of ["runner", "protocol"]) {+ expect(native.indexOf(`COPY packages/paperclip-runner/${source} ./${source}`))+ .toBeLessThan(native.indexOf("cargo build --release"));+ expect(native).toContain(`COPY packages/paperclip-runner/${source} ./${source}`);+ }+ expect(native).toContain("cargo build --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd");+ expect(stageBody(dockerfile, "build")).toContain("FROM runner-build AS build");+ });+});