Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
feat(cloud): drop Fly preview and run the app from Vite (#5863)
teoe77b5e6bba5d
diff --git a/.claude/skills/run-app/SKILL.md b/.claude/skills/run-app/SKILL.mdnew file mode 100644index 0000000000..3a5df8e1e7--- /dev/null+++ b/.claude/skills/run-app/SKILL.md@@ -0,0 +1,82 @@+---+name: run-app+description: >-+ Run the Macro app on Cursor Cloud and pick up edits. Use when asked to run+ the app, open it in the browser, test a frontend or backend change, hot+ reload, or before touching stack.sh, rebuild.sh, frontend.sh, or just stack.+---++# Run the app++The app is http://localhost:3000/app. Open it in the VM browser. The user's laptop cannot reach these URLs.++| Goal | Command |+| --- | --- |+| Start | `bash .cursor/stack.sh` |+| After a Rust edit | `bash .cursor/rebuild.sh` |+| After an `apps/web` edit | save the file |++The `.cursor` scripts re-enter the pinned nix shell. For any other `docker`, `just`, `bun`, or `doppler` command, prefix with `nix develop /workspace --command`.++Do not run `just run_local`. That is the laptop TUI.++## Start++```bash+bash .cursor/stack.sh+```++Let the first run finish. It compiles whatever changed since the environment bake. Do not kill the Nix build, copy binaries out of `/nix/store`, or replace this with hand-rolled `docker compose`.++Log in with any email. The login API returns the code. Codes also land in Mailpit at http://localhost:8025.++## After a Rust edit++```bash+bash .cursor/rebuild.sh+```++Volumes stay. If the log says binaries are unchanged, the mounts were left alone.++Do not run `just stack up` or `stack.sh --fresh` to pick up an edit. `up` deletes the stack and recreates it. Use `--fresh` only when you mean to wipe.++`cargo build` and `just build` do not remount containers. `rebuild.sh` is the remount path.++## After an `apps/web` edit++Save the file. Vite watches `apps/web`. If the page does not update, read `~/.cursor-cloud/frontend-dev.log` for `hmr update`.++If http://localhost:3000/app stops responding, restart the dev server:++```bash+bash .cursor/frontend.sh stop && bash .cursor/frontend.sh+```++Use the same restart after a frontend dependency or Vite config change.++Do not run `just stack update --frontend`. That builds a static bundle. This Cloud stack does not serve one. On a `--no-frontend` stack that flag then tells you to re-run `just stack up`, which wipes volumes.++## Show the change++Bring the stack up, log in, and demonstrate the change in the UI. Unit tests and SQL probes are not a walkthrough.++## If login fails++If the page says "unable to lookup identity providers", FusionAuth is in maintenance mode. `docker ps` shows `macro-fusionauth-1` as unhealthy. `curl localhost:9011/api/status` returns 302.++```bash+docker restart macro-fusionauth-1+```++Wait until the container is healthy.++If the page loads and API calls fail, the backend is down. Run `just stack status --json`, then `bash .cursor/stack.sh`.++Do not chase `agent_harness_service` restart loops when `DOPPLER_TOKEN` is absent. That loop is expected.++## Status++```bash+just stack status --json+docker compose -p macro logs -f <service>+```diff --git a/.cursor/cloud-lib.sh b/.cursor/cloud-lib.shindex de6074de93..e00ddd7b66 100755--- a/.cursor/cloud-lib.sh+++ b/.cursor/cloud-lib.sh@@ -235,7 +235,7 @@ ensure_dockerd() {# Populate `doppler_args` for `just stack up`. The doppler CLI is already on# PATH from `nix develop` — this only chooses whether to pass `--no-doppler`.# A config-scoped service token (local / lcl_preview) is enough; no login.-# DOPPLER_PREVIEW_TOKEN is the Fly/CI name for the same token.+# DOPPLER_PREVIEW_TOKEN is the CI name for the same token.stack_doppler_args() {if [ -z "${DOPPLER_TOKEN:-}" ] && [ -n "${DOPPLER_PREVIEW_TOKEN:-}" ]; thenexport DOPPLER_TOKEN="${DOPPLER_PREVIEW_TOKEN}"@@ -249,6 +249,20 @@ stack_doppler_args() {fi}+# Run a command while re-applying bridge forwarding every few seconds.+# Compose creates networks mid-`stack up`; a service booting on a bridge+# before its FORWARD rules exist times out reaching its database (FusionAuth+# wedged in maintenance mode this way). The watcher closes that window.+run_with_bridge_forwarding() {+ "$@" &+ local cmd_pid=$!+ while kill -0 "${cmd_pid}" 2>/dev/null; do+ ensure_docker_bridge_forwarding+ sleep 3+ done+ wait "${cmd_pid}"+}+in_pinned_nix_shell() {[ "${MACRO_CLOUD_PINNED_NIX:-}" = "1" ]}diff --git a/.cursor/frontend.sh b/.cursor/frontend.shnew file mode 100755index 0000000000..ab28d89a06--- /dev/null+++ b/.cursor/frontend.sh@@ -0,0 +1,78 @@+#!/usr/bin/env bash+set -euo pipefail++SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"++# THE frontend on Cloud: a hot-reloading Vite dev server against the running+# backend. stack.sh starts it; edits under apps/web apply on save. There is+# no static bundle — http://localhost:3000/app is the only app URL.+#+# bash .cursor/frontend.sh # start (no-op if already serving)+# bash .cursor/frontend.sh stop # stop the dev server++# shellcheck source=cloud-lib.sh+source "${SCRIPT_DIR}/cloud-lib.sh"++if ! in_pinned_nix_shell; then+ ensure_nix_daemon+ reenter_pinned_nix_shell "${SCRIPT_DIR}/frontend.sh" "$@"+fi++FRONTEND_PORT=3000+PROXY_ORIGIN='http://localhost:8090'+DEV_URL="http://localhost:${FRONTEND_PORT}/app"+PID_FILE="${LOG_DIR}/frontend-dev.pid"+DEV_LOG="${LOG_DIR}/frontend-dev.log"++if [ "${1:-}" = "stop" ]; then+ if [ -f "${PID_FILE}" ]; then+ pgid="$(cat "${PID_FILE}")"+ kill -- "-${pgid}" 2>/dev/null || true+ rm -f "${PID_FILE}"+ echo "cursor-cloud frontend: stopped"+ else+ echo "cursor-cloud frontend: not running (no pid file)"+ fi+ exit 0+fi++if curl -fsS --max-time 2 "${DEV_URL}" >/dev/null 2>&1; then+ echo "cursor-cloud frontend: already serving at ${DEV_URL}"+ exit 0+fi++if ! curl -fsS --max-time 3 "${PROXY_ORIGIN}/auth/health" >/dev/null 2>&1; then+ echo "cursor-cloud frontend: stack is not running — run: bash .cursor/stack.sh" >&2+ exit 1+fi++: >"${DEV_LOG}"+\cd "${WORKSPACE_ROOT}/apps/web"+setsid env \+ PORT="${FRONTEND_PORT}" \+ VITE_LOCAL_SERVERS=ALL \+ VITE_LOCAL_BACKEND_ORIGIN="${PROXY_ORIGIN}" \+ VITE_AI_EDITING_WORKER_URL="${PROXY_ORIGIN}/ai-editing" \+ bun run --bun dev >>"${DEV_LOG}" 2>&1 </dev/null &+echo "$!" >"${PID_FILE}"++# Cold starts build wasm packages before Vite binds; be patient once.+n=0+while [ "${n}" -lt 300 ]; do+ if ! kill -0 "$(cat "${PID_FILE}")" 2>/dev/null; then+ echo "cursor-cloud frontend: dev server exited during startup" >&2+ tail -40 "${DEV_LOG}" >&2+ rm -f "${PID_FILE}"+ exit 1+ fi+ if curl -fsS --max-time 2 "${DEV_URL}" >/dev/null 2>&1; then+ echo "cursor-cloud frontend: hot-reloading dev server at ${DEV_URL}"+ echo "cursor-cloud frontend: logs at ${DEV_LOG}"+ exit 0+ fi+ n=$((n + 1))+ sleep 1+done+echo "cursor-cloud frontend: dev server did not become ready" >&2+tail -40 "${DEV_LOG}" >&2+exit 1diff --git a/.cursor/rebuild.sh b/.cursor/rebuild.shindex 2eb9b197d9..e4e9b2b666 100755--- a/.cursor/rebuild.sh+++ b/.cursor/rebuild.sh@@ -3,9 +3,9 @@ set -euo pipefailSCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"-# Backend-edit flow, as one verb: rebuild the Nix stack binaries and recycle-# the stack onto them. This replaces `just stack update`, which cannot write-# into the read-only Nix binaries directory.+# Backend-edit flow: rebuild Nix stack binaries and adopt them into the+# running stack. Volumes stay. If nothing is recorded yet, `stack update`+# bootstraps through `stack up`.# shellcheck source=cloud-lib.shsource "${SCRIPT_DIR}/cloud-lib.sh"@@ -18,5 +18,11 @@ fi/usr/bin/bash "${SCRIPT_DIR}/infra.sh"\cd "${WORKSPACE_ROOT}"-just stack down-exec /usr/bin/bash "${SCRIPT_DIR}/stack.sh"+build_local_stack_binaries+stack_doppler_args+# The watcher matters on the bootstrap path (update -> up creates networks).+run_with_bridge_forwarding \+ just stack update "${doppler_args[@]}" --binaries-dir "${LOCAL_STACK_BINS}/bin"+ensure_docker_bridge_forwarding++echo "cursor-cloud stack: binaries remounted"diff --git a/.cursor/stack.sh b/.cursor/stack.shindex 831c9a1b0f..eb01c26eec 100755--- a/.cursor/stack.sh+++ b/.cursor/stack.sh@@ -3,8 +3,13 @@ set -euo pipefailSCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"-# On-demand product stack. Boot (start.sh) starts nothing; infra.sh brings up-# dockerd and the databases first. After backend edits, run rebuild.sh.+# On-demand product stack: backend containers behind the proxy (8090) plus+# the hot-reloading frontend dev server (3000). There is no static frontend+# on Cloud — the dev server IS the frontend, so edits under apps/web apply+# on save. A healthy running backend is left alone — `stack up` is+# full-delete/full-create, so a blind re-run would wipe volumes (logins,+# documents). Pass --fresh to wipe and recreate deliberately.+# After backend edits, run rebuild.sh: it remounts new Nix binaries in place.# shellcheck source=cloud-lib.shsource "${SCRIPT_DIR}/cloud-lib.sh"@@ -17,9 +22,22 @@ fi/usr/bin/bash "${SCRIPT_DIR}/infra.sh"\cd "${WORKSPACE_ROOT}"-build_local_stack_binaries-stack_doppler_args-just stack up "${doppler_args[@]}" --build-aux-services --binaries-dir "${LOCAL_STACK_BINS}/bin"-ensure_docker_bridge_forwarding-echo "cursor-cloud stack: app ready"+if [ "${1:-}" != "--fresh" ] \+ && curl -fsS --max-time 3 http://localhost:8090/auth/health >/dev/null 2>&1; then+ echo "cursor-cloud stack: backend already running"+else+ build_local_stack_binaries+ stack_doppler_args+ run_with_bridge_forwarding \+ just stack up "${doppler_args[@]}" --no-frontend --build-aux-services \+ --binaries-dir "${LOCAL_STACK_BINS}/bin"+ ensure_docker_bridge_forwarding+fi++/usr/bin/bash "${SCRIPT_DIR}/frontend.sh"++echo "cursor-cloud stack: app ready at http://localhost:3000/app"+echo "cursor-cloud stack: backend edits -> bash .cursor/rebuild.sh"+echo "cursor-cloud stack: frontend edits apply on save (no command)"+echo "cursor-cloud stack: wipe + restart -> bash .cursor/stack.sh --fresh"diff --git a/.github/workflows/preview-fly-cleanup.yml b/.github/workflows/preview-fly-cleanup.ymldeleted file mode 100644index ebe911e0bb..0000000000--- a/.github/workflows/preview-fly-cleanup.yml+++ /dev/null@@ -1,21 +0,0 @@-# DO NOT EDIT — regenerate with `cargo x workflows` (from the repository root).-# Source: tooling/xtask/crates/xtask_workflows/src/workflows/preview_fly.rs-name: Fly Preview Cleanup-'on':- pull_request:- types:- - closed- - unlabeled-jobs:- destroy:- if: github.event.action == 'closed' || github.event.label.name == 'preview'- runs-on: namespace-profile-linux-tiny-no-cache- permissions:- contents: read- env:- FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}- steps:- - name: Setup flyctl- uses: superfly/flyctl-actions/setup-flyctl@ed8efb33836e8b2096c7fd3ba1c8afe303ebbff1- - name: Destroy app- run: flyctl apps destroy "macro-pr-${{ github.event.pull_request.number }}" --yes || truediff --git a/.github/workflows/preview-fly.yml b/.github/workflows/preview-fly.ymldeleted file mode 100644index 632159d6cf..0000000000--- a/.github/workflows/preview-fly.yml+++ /dev/null@@ -1,767 +0,0 @@-# DO NOT EDIT — regenerate with `cargo x workflows` (from the repository root).-# Source: tooling/xtask/crates/xtask_workflows/src/workflows/preview_fly.rs-name: Fly Preview-'on':- pull_request:- types:- - opened- - reopened- - synchronize- - labeled-jobs:- deploy:- if: github.event.pull_request.head.repo.full_name == github.repository && contains(github.event.pull_request.labels.*.name, 'preview')- runs-on: namespace-profile-linux-mid;overrides.cache-tag=fly-preview- permissions:- contents: read- pull-requests: write- env:- FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}- APP_NAME: macro-pr-${{ github.event.pull_request.number }}- TEMPLATE_APP: macro-preview-template- MACRO_STACK_SNAPSHOT_DIR: /home/runner/.cache/macro-preview-snapshots- CARGO_INCREMENTAL: '0'- timeout-minutes: 90- steps:- - name: Checkout- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd- with:- clean: 'false'- - name: Mount Namespace cache volume- uses: namespacelabs/nscloud-cache-action@15799a6b54e5765f85b2aac25b3f0df43ed571c0- with:- cache: nix- path: |-- ${{ github.workspace }}/target- /home/runner/.cache/macro-preview-snapshots- /home/runner/.cargo/registry- /home/runner/.cargo/git- continue-on-error: true- - name: Prune stale incremental caches- run: rm -rf target/*/debug/incremental target/debug/incremental- - name: Set up Namespace Docker builder- uses: namespacelabs/nscloud-setup-buildx-action@d059ed7184f0bc7c8b27e8810cea153d02bcc6dd- - name: Setup Nix- uses: './.github/actions/setup-nix'- - name: Setup dev shell + web deps- uses: './.github/actions/setup-reqs-web'- - name: Configure Namespace remote sccache- if: (github.event_name != 'pull_request' && github.event_name != 'pull_request_target') || github.event.pull_request.head.repo.full_name == github.repository- run: |-- set -euo pipefail- env_file="$(mktemp "$RUNNER_TEMP/namespace-sccache.XXXXXX")"- trap 'rm -f "$env_file"' EXIT- if nsc cache sccache setup --cache_name fly-preview > "$env_file"; then- # Register credential values as masked BEFORE exporting: GitHub only- # masks `secrets.*`, so without this SCCACHE_WEBDAV_TOKEN — a broad,- # ~24h Namespace workspace token (registry + cache write) — printed- # verbatim in every subsequent step's env dump. Masking is selective by- # key name: masking non-secrets like the endpoint URL or key prefix- # would redact those strings everywhere in the logs.- while IFS= read -r line; do- k="${line%%=*}"- v="${line#*=}"- [ -n "$v" ] && [ "$k" != "$line" ] || continue- case "$k" in- *TOKEN*|*SECRET*|*PASSWORD*) echo "::add-mask::$v" ;;- esac- done < "$env_file"- cat "$env_file" >> "$GITHUB_ENV"- # Force the next compiler invocation to start a server with the new remote- # backend even if a setup hook happened to launch one already.- sccache --stop-server >/dev/null 2>&1 || true- else- echo "::warning::Namespace remote sccache setup failed; using local cache fallback"- fi- - name: Setup flyctl- uses: superfly/flyctl-actions/setup-flyctl@ed8efb33836e8b2096c7fd3ba1c8afe303ebbff1- - name: Build artifacts (parallel lanes)- run: |- set -euo pipefail- lanes="$RUNNER_TEMP/lanes"- mkdir -p "$lanes"-- cat > "$lanes/cargo.sh" <<'LANE'- set -euo pipefail- cargo x zigbuild- sccache --show-stats- cargo zigbuild --release --target x86_64-unknown-linux-gnu.2.36 -p xtask_local- LANE-- cat > "$lanes/web.sh" <<'LANE'- set -euo pipefail- cd apps/web- export MODE=development NODE_ENV=production- export VITE_LOCAL_SERVERS=ALL VITE_LOCAL_BACKEND_ORIGIN=same-origin- # Same ceiling as `just -f apps/web/justfile build-dev`: vite's- # chunk-render peaks at node's default ~4GB heap and aborts with- # "Ineffective mark-compacts near heap limit".- export NODE_OPTIONS=--max-old-space-size=6144- bun run --bun build- LANE-- # Instant lane: everything Fly-side that needs no build output.- # The template fork must fire within seconds of the job starting —- # its ~2 min server-side hydration then finishes inside the compose- # build. (It previously sat after the premirror pushes, which put- # it at the END of the build phase and re-gated the deploy.)- # Entirely best-effort: everything in the fly lane (app create,- # fork kick, token mint) has an authoritative fallback in the- # deploy step, so a transient Fly API failure must not fail the- # build. A child `bash -e` (not a `|| `-guarded function: errexit- # is suppressed inside those) keeps its internal fail-fast so a- # failed app_id fetch can't mint a garbage token.- cat > "$lanes/fly.sh" <<'LANE'- set -euo pipefail- bash -euo pipefail "$(dirname "$0")/fly-prep.sh" \- || echo "fly prep failed (non-fatal): the deploy step falls back" >&2- LANE-- cat > "$lanes/fly-prep.sh" <<'LANE'- flyctl status --app "$APP_NAME" >/dev/null 2>&1 \- || flyctl apps create "$APP_NAME" --org "$FLY_ORG" \- || flyctl status --app "$APP_NAME" >/dev/null- # Kick the template-volume fork (see the deploy step's- # volume-state block, which stays the authoritative fallback).- if ! flyctl volumes list --app "$APP_NAME" --json 2>/dev/null \- | jq -e 'map(select(.name == "docker_data")) | length > 0' >/dev/null; then- tpl_src=$(flyctl volumes list --app "$TEMPLATE_APP" --json 2>/dev/null \- | jq -r '[.[] | select((.name | startswith("tpl")) and .state == "created" and .attached_machine_id == null)]- | sort_by(.created_at) | last | .id // empty' 2>/dev/null) || tpl_src=- if [ -n "$tpl_src" ]; then- curl -sf -X POST "https://api.machines.dev/v1/apps/$APP_NAME/volumes" \- -H "Authorization: Bearer $FLY_API_TOKEN" \- -H "Content-Type: application/json" \- -d "{\"name\":\"docker_data\",\"region\":\"ewr\",\"source_volume_id\":\"$tpl_src\"}" \- | jq -r '"early template fork: \(.id // "failed") (\(.state // "-"))"' \- || echo "early template fork failed; deploy step will fall back" >&2- fi- fi- # Mint the machine's app-scoped read-only registry pull token- # (macaroon attenuation, see the deploy step for the why) so the- # deploy preamble doesn't pay the round-trips serially.- app_id=$(curl -sf https://api.fly.io/graphql \- -H "Authorization: Bearer $FLY_API_TOKEN" \- -H "content-type: application/json" \- -d "{\"query\":\"{ app(name: \\\"$APP_NAME\\\") { internalNumericId } }\"}" \- | jq -re '.data.app.internalNumericId')- now=$(date +%s)- printf '[{"type":"Apps","body":{"apps":{"%s":"r"}}},{"type":"ValidityWindow","body":{"not_before":%d,"not_after":%d}}]' \- "$app_id" $((now - 60)) $((now + 604800)) \- | flyctl tokens attenuate > "$RUNNER_TEMP/registry-pull-token.next"- chmod 600 "$RUNNER_TEMP/registry-pull-token.next"- mv "$RUNNER_TEMP/registry-pull-token.next" "$RUNNER_TEMP/registry-pull-token"- LANE-- cat > "$lanes/images.sh" <<'LANE'- set -euo pipefail- docker compose --project-directory . -p macro -f docker/docker-compose.yml build \- search sync_service websocket_service lexical_service ai_editing_worker analytics_proxy- # Not a compose service, so `config --images` never lists it.- # BuildKit cache on this runner is the freshness check.- docker build --tag "$LOCAL_SANDBOX_IMAGE" crates/agent_harness/container- # The image BUILD above is fatal; the premirror below is a pure- # optimization with an authoritative fallback (the deploy step's- # mirror loop), so its failure only costs a slower push later.- bash -euo pipefail "$(dirname "$0")/premirror.sh" \- || echo "premirror failed (non-fatal): the deploy step mirrors authoritatively" >&2- LANE-- # Premirror: content-addressed refs, identical scheme to the deploy- # step's mirror loop, so its manifest-existence check skips these.- # Best effort by design — anything skipped here (images only the- # bake produces, like macro-local-runtime:dev) is pushed there.- # The fly lane created the app long ago; the guard is a fallback.- cat > "$lanes/premirror.sh" <<'LANE'- flyctl status --app "$APP_NAME" >/dev/null 2>&1 \- || flyctl apps create "$APP_NAME" --org "$FLY_ORG" \- || flyctl status --app "$APP_NAME" >/dev/null- flyctl auth docker- registry="registry.fly.io/$APP_NAME"- for img in $(docker compose --project-directory . -p macro \- -f docker/docker-compose.yml config --images | sort -u) alpine:3 "$LOCAL_SANDBOX_IMAGE"; do- if ! docker image inspect "$img" >/dev/null 2>&1 && ! docker pull "$img"; then- echo "premirror: skipping $img (not local, not pullable)"- continue- fi- id=$(docker image inspect -f '{{.Id}}' "$img")- ref="$registry:img-$(echo "$img" | tr '/:' '__')-$(echo "$id" | cut -c8-19)"- docker tag "$img" "$ref"- if ! docker manifest inspect "$ref" >/dev/null 2>&1; then- for _ in 1 2 3; do- docker push "$ref" && break- echo "premirror push of $ref failed, retrying" >&2- sleep 15- done- fi- done- LANE-- # A cold `stack up --infra-only` runs the real init (migrate,- # kickstart, Kafka topics, indices) and saves the content-addressed- # snapshot the VM restores from. Infra only: the app services need- # the Doppler-sourced env to boot, which this runner deliberately- # lacks — the snapshot captures only the infra volumes anyway. Via- # `just` (not the bare `cargo x` alias) because the justfile- # enables the `local-stack` feature the Kafka provisioning needs.- # The store lives on the cache volume for a zero-copy hit; every- # snapshot is also stored as one tar in Namespace artifact storage- # keyed by the same hash, so cache-volume onboarding misses- # download the exact snapshot instead of paying for a full init.- # Artifact access is an optimization: failures fall back to the- # volume/cold-bake behavior. The final `snapshot --json` line lands- # in $RUNNER_TEMP for the stage step.- cat > "$lanes/bake.sh" <<'LANE'- set -euo pipefail- export CARGO_TARGET_DIR="$PWD/target/bake"- mkdir -p "$MACRO_STACK_SNAPSHOT_DIR"- status=$(just stack snapshot --json | tail -n 1)- key=$(echo "$status" | jq -r '.key')- root=$(echo "$status" | jq -r '.root')- artifact="macro-preview/init-snapshots/${key}.tar"- archive="$RUNNER_TEMP/init-snapshot-${key}.tar"- artifact_hit=-- # The Namespace cache-volume pool has an onboarding period where a- # runner can receive an empty fork. Artifact storage is the durable- # fallback: restore into a temporary directory and only publish it to- # the live store after validating the embedded manifest through xtask.- if ! echo "$status" | jq -e '.present' >/dev/null; then- restored="$RUNNER_TEMP/restored-init-snapshot"- rm -rf "$restored"- mkdir -p "$restored"- if nsc artifact download "$artifact" "$archive" \- && tar -xf "$archive" -C "$restored" \- && [ -f "$restored/$key/manifest.json" ]; then- rm -rf "$root/$key"- cp -a "$restored/$key" "$root/$key"- status=$(just stack snapshot --json | tail -n 1)- if echo "$status" | jq -e '.present' >/dev/null; then- artifact_hit=1- echo "init snapshot artifact hit: $status"- else- rm -rf "$root/$key"- echo "downloaded init snapshot failed validation; baking cold" >&2- fi- else- echo "init snapshot artifact miss; baking cold" >&2- fi- fi-- if echo "$status" | jq -e '.present' >/dev/null; then- echo "init snapshot cache hit: $status"- cargo run --quiet --manifest-path Cargo.toml \- -p xtask_local --features local-stack -- gen-compose- # The cold path builds the runtime image inside `stack up`; on a- # hit nothing else does, and the registry mirror needs it in the- # local daemon (its pull fallback only covers public images —- # macro-local-runtime:dev is not on Docker Hub).- cargo run --quiet --manifest-path Cargo.toml \- -p xtask_local --features local-stack -- runtime-image- else- just stack up --infra-only --no-doppler --no-build- status=$(just stack snapshot --json | tail -n 1)- echo "$status" | jq -e '.present' >/dev/null- fi-- # Seed artifact storage from either a volume hit or a cold bake. Avoid- # uploading another version when the artifact already exists; a race- # between concurrent PRs is harmless and must not fail the preview.- if [ -z "$artifact_hit" ] && ! nsc artifact describe "$artifact" >/dev/null 2>&1; then- snap_dir=$(echo "$status" | jq -r '.dir')- tar -cf "$archive" -C "$root" "$(basename "$snap_dir")"- nsc artifact upload "$archive" "$artifact" --expires_in 720h \- || echo "warning: failed to upload init snapshot artifact" >&2- fi- echo "$status" > "$RUNNER_TEMP/preview-snapshot.json"- LANE-- declare -a pids=() names=()- for name in fly cargo web images bake; do- # setsid: each lane leads its own process group so fail-fast- # can kill the whole tree (cargo/docker/bun children), not just- # the wrapper shell.- setsid bash "$lanes/$name.sh" > "$lanes/$name.log" 2>&1 &- pids+=($!)- names+=("$name")- done- fail=- for _ in "${pids[@]}"; do- # Bare `wait -n`: any child; explicit pids would error once a- # job has already been reaped.- if ! wait -n; then- fail=1- break- fi- done- if [ -n "$fail" ]; then- # Negative pids: signal each lane's whole process group.- kill -- "${pids[@]/#/-}" 2>/dev/null || true- wait || true- fi- for i in "${!names[@]}"; do- echo "::group::lane: ${names[$i]}"- cat "$lanes/${names[$i]}.log" || true- echo "::endgroup::"- done- if [ -n "$fail" ]; then- echo "a build lane failed (first failure aborts the rest); see lane logs above" >&2- exit 1- fi- env:- FLY_ORG: ${{ vars.FLY_ORG || secrets.FLY_ORG }}- LOCAL_SANDBOX_IMAGE: macro-agent-harness:latest- - name: Dump stack diagnostics- if: failure()- run: |- docker compose -p macro ps --all || true- for svc in authentication-service proxy fusionauth postgres kafka \- connection_gateway document_storage_service email_service \- notification_service contacts_service; do- echo "==================== $svc ===================="- docker compose -p macro logs --no-color --tail 80 "$svc" 2>&1 || true- done- - name: Stage preview build context- run: |- set -euo pipefail- # The dev-shell env (BASH_ENV) exports a Nix LD_LIBRARY_PATH; host- # binaries like rsync then resolve Nix-store libs whose deps drag in- # Nix glibc mid-process and crash on glibc symbol versions. This step- # is pure file shuffling — drop it.- unset LD_LIBRARY_PATH- ctx=preview-ctx- rm -rf "$ctx"- mkdir -p "$ctx/repo" "$ctx/artifacts/binaries" "$ctx/bin"- # The minimal repo layout xtask reads at runtime — including every- # snapshot-key input, byte-identical to this checkout, so the VM's- # key matches the snapshot baked above.- rsync -a --relative --exclude node_modules \- docker/docker-compose.yml \- docker/docker-compose-databases.yml \- infra/stacks/fusionauth-instance/docker-compose.yml \- infra/local/nginx \- infra/local/opensearch \- infra/stacks/opensearch/helpers \- crates/macro_db_client/migrations \- "$ctx/repo/"- # Service binaries only — the target dir also holds gigabytes of- # build intermediates. `-p` everywhere: preserved mtimes mean files- # cargo didn't relink produce byte-identical Docker layers, so the- # registry push/pull skips them (the Dockerfile orders its COPYs- # stable → volatile for the same reason).- find target/x86_64-unknown-linux-gnu/debug \- -maxdepth 1 -type f -executable ! -name '*.d' ! -name '*.so' \- -exec cp -p {} "$ctx/artifacts/binaries/" \;- cp -a apps/web/dist "$ctx/artifacts/frontend-dist"- # Only the current key's snapshot — the volume store accumulates keys.- snap_dir=$(jq -r '.dir' "$RUNNER_TEMP/preview-snapshot.json")- mkdir -p "$ctx/artifacts/snapshots"- cp -a "$snap_dir" "$ctx/artifacts/snapshots/"- snapshot_key=$(jq -r '.key' "$RUNNER_TEMP/preview-snapshot.json")- runtime_key=$(cd "$ctx/repo" \- && find . -type f -print0 \- | sort -z \- | xargs -0 sha256sum \- | sha256sum \- | cut -d ' ' -f 1)- frontend_key=$(cd "$ctx/artifacts/frontend-dist" \- && find . -type f -print0 \- | sort -z \- | xargs -0 sha256sum \- | sha256sum \- | cut -d ' ' -f 1)- jq -n \- --arg snapshot_key "$snapshot_key" \- --arg runtime_key "$runtime_key" \- --arg frontend_key "$frontend_key" \- --arg commit "${{ github.sha }}" \- '{format: 1, snapshot_key: $snapshot_key, runtime_key: $runtime_key, frontend_key: $frontend_key, commit: $commit}' \- > "$ctx/deployment.json"- cp -p target/x86_64-unknown-linux-gnu/release/xtask_local "$ctx/bin/xtask"- cp -p infra/preview/hot-update.sh "$ctx/bin/hot-update"- cp -p infra/preview/entrypoint.sh "$ctx/entrypoint.sh"- cp -p infra/preview/Dockerfile "$ctx/Dockerfile"- cp -p infra/preview/update.Dockerfile "$ctx/update.Dockerfile"- - name: Deploy to Fly- run: |- set -euo pipefail- if [ -z "$DOPPLER_PREVIEW_TOKEN" ]; then- echo "DOPPLER_PREVIEW_TOKEN secret is not set (a Doppler service token scoped to the preview config)" >&2- exit 1- fi- if [ -z "$FLY_ORG" ]; then- echo "FLY_ORG is not set (repo variable or secret with the Fly org slug)" >&2- exit 1- fi-- app_existed=- if flyctl status --app "$APP_NAME" >/dev/null 2>&1; then- app_existed=1- else- # A transient status error must not turn an existing app into a- # hard create failure: create, then prove the app is reachable.- flyctl apps create "$APP_NAME" --org "$FLY_ORG" \- || flyctl status --app "$APP_NAME" >/dev/null- fi-- # Seed a first boot's volume from the newest template volume — a- # warm /var/lib/docker layer store published by a previous- # successful deploy (see the "Publish template volume" step).- # Measured cold, an empty volume costs ~11 min of image pulls at- # boot. The images lane normally kicked the fork minutes ago (its- # ~2 min server-side hydration overlaps the whole build phase);- # this block is the authoritative fallback for a missing or- # failed early fork. Cross-app forks need the Machines API:- # flyctl resolves volume ids app-locally. The boot manifest check- # makes the volume a pure cache, so every failure path here just- # falls back to the old empty-volume behavior.- vol_state=$(flyctl volumes list --app "$APP_NAME" --json 2>/dev/null \- | jq -r '[.[] | select(.name == "docker_data")][0].state // empty') || vol_state=- if [ -z "$vol_state" ]; then- tpl_src=$(flyctl volumes list --app "$TEMPLATE_APP" --json 2>/dev/null \- | jq -r '[.[] | select((.name | startswith("tpl")) and .state == "created" and .attached_machine_id == null)]- | sort_by(.created_at) | last | .id // empty' 2>/dev/null) || tpl_src=- if [ -n "$tpl_src" ]; then- vol_state=$(curl -sf -X POST "https://api.machines.dev/v1/apps/$APP_NAME/volumes" \- -H "Authorization: Bearer $FLY_API_TOKEN" \- -H "Content-Type: application/json" \- -d "{\"name\":\"docker_data\",\"region\":\"ewr\",\"source_volume_id\":\"$tpl_src\"}" \- | jq -r '.state // empty') || vol_state=- if [ -n "$vol_state" ]; then- echo "seeding docker_data from template volume $tpl_src (state: $vol_state)"- else- echo "template fork failed; falling back to an empty volume" >&2- fi- else- echo "no template volume available; first boot will pull cold"- fi- fi-- # A read-only, app-scoped, time-boxed pull token so the machine's- # inner dockerd can pull the mirrored stack images from this app's- # registry repo — and nothing else (PR code can read machine- # secrets, so scope matters). The org deploy token can't mint new- # tokens (createLimitedAccessToken is not authorized), but macaroon- # attenuation is pure client-side crypto: append an Apps caveat- # (numeric app id, mask "r") and a validity window to our own- # token. The images lane normally minted it already; fall back to- # minting inline when its file is absent.- if [ -s "$RUNNER_TEMP/registry-pull-token" ]; then- pull_token=$(cat "$RUNNER_TEMP/registry-pull-token")- else- app_id=$(curl -sf https://api.fly.io/graphql \- -H "Authorization: Bearer $FLY_API_TOKEN" \- -H "content-type: application/json" \- -d "{\"query\":\"{ app(name: \\\"$APP_NAME\\\") { internalNumericId } }\"}" \- | jq -re '.data.app.internalNumericId')- now=$(date +%s)- pull_token=$(printf '[{"type":"Apps","body":{"apps":{"%s":"r"}}},{"type":"ValidityWindow","body":{"not_before":%d,"not_after":%d}}]' \- "$app_id" $((now - 60)) $((now + 604800)) | flyctl tokens attenuate)- fi-- snapshot_key=$(jq -r '.key' "$RUNNER_TEMP/preview-snapshot.json")- runtime_key=$(jq -r '.runtime_key' preview-ctx/deployment.json)- mode=bootstrap- machine_id=- if [ -n "$app_existed" ]; then- mode=rehydrate- machine_id=$(flyctl machine list --app "$APP_NAME" --json \- | jq -r '[.[] | select((.config.mounts // []) | length > 0)][0].id // empty')- if [ -n "$machine_id" ]; then- # A suspended preview needs demand before SSH is reachable.- curl -s -o /dev/null --max-time 10 "https://$APP_NAME.fly.dev/" || true- for _ in $(seq 1 12); do- # `--command` is exec'd directly, NOT run through a shell —- # a bare `if [ -f ...` dies with `exec: "if": not found`, so- # the marker never parsed and every push silently rehydrated- # (~7 min) instead of going hot (~3 min). Hence the sh -c.- marker=$(flyctl ssh console --app "$APP_NAME" --machine "$machine_id" \- --quiet --command 'sh -c "if [ -f /var/lib/docker/.macro-preview/deployment.json ]; then cat /var/lib/docker/.macro-preview/deployment.json; else echo __NO_MARKER__; fi"' \- 2>/dev/null || true)- if echo "$marker" | jq -e \- --arg key "$snapshot_key" \- --arg runtime "$runtime_key" \- '.format == 1 and .snapshot_key == $key and .runtime_key == $runtime and (.frontend_key | type == "string")' \- >/dev/null 2>&1; then- mode=hot- break- fi- # SSH succeeded and returned a marker (or proved it absent):- # incompatibility is real, not a wake-up race.- if [ "$marker" = __NO_MARKER__ ] \- || echo "$marker" | jq -e '.format != null' >/dev/null 2>&1; then- break- fi- sleep 5- done- fi- fi- echo "preview deployment mode: $mode (snapshot $snapshot_key)"-- # Keep the machine in demand for the WHOLE deploy, hot path- # included: fly-proxy suspends a machine ~7 min after its last- # proxied request, and SSH via hallpass is not proxy traffic — a- # hot update with no browser traffic got its VM suspended- # mid-session (the frozen `flyctl ssh console` then ate 52 min of- # runner until the job timeout). Any response status counts as- # traffic, and a request also resumes a suspended machine.- (while :; do- curl -s -o /dev/null --max-time 10 "https://$APP_NAME.fly.dev/" || true- sleep 15- done) &- keepalive=$!- trap 'kill "$keepalive" 2>/dev/null || true' EXIT-- flyctl auth docker- registry="registry.fly.io/$APP_NAME"- # Mirror every stack image into the app's registry repo instead of- # baking docker-save tars into the VM image (which made every- # deploy re-ship ~5GB the machine's volume already had). Pushes- # dedup at the layer level against the repo, so a redeploy only- # uploads layers that actually changed; the entrypoint pulls with- # the same dedup against the machine's persistent layer store. The- # manifest maps image ID -> local tag -> registry ref so boots can- # skip images already present. alpine:3 rides along for the- # snapshot-restore helper container.- # On a snapshot cache hit only gen-compose ran, which doesn't write- # the env file; compose hard-fails on a missing --env-file, and the- # image names don't interpolate env vars anyway.- envfile=infra/local/generated/macro/local.generated.env- [ -f "$envfile" ] || : > "$envfile"- images=$(docker compose --project-directory . -p macro \- -f docker/docker-compose.yml \- -f infra/local/generated/macro/docker-compose.override.yml \- --env-file "$envfile" \- config --images | sort -u)- echo "mirroring images:" $images- docker image inspect alpine:3 >/dev/null 2>&1 || docker pull alpine:3- mkdir -p preview-ctx/preload- : > preview-ctx/preload/manifest.txt- for img in $images alpine:3 "$LOCAL_SANDBOX_IMAGE"; do- # Images the bake didn't leave in the daemon (snapshot cache hit,- # or app-layer-only images like proxy/mailpit) get pulled here.- # Compose `config --images` also lists locally-built tags- # (analytics_proxy, sdk-webhook-relay) that are not on Docker Hub;- # premirror already skips those — do the same so a missing local- # tag cannot fail the deploy.- if ! docker image inspect "$img" >/dev/null 2>&1 && ! docker pull "$img"; then- echo "mirror: skipping $img (not local, not pullable)"- continue- fi- id=$(docker image inspect -f '{{.Id}}' "$img")- # Content-addressed tag: same image content = same ref, so a- # redeploy's push is skippable by a manifest existence check and- # concurrent PRs' deploys can never clobber each other's tags.- ref="$registry:img-$(echo "$img" | tr '/:' '__')-$(echo "$id" | cut -c8-19)"- docker tag "$img" "$ref"- echo "$id $img $ref" >> preview-ctx/preload/manifest.txt- done- # The existence probes are pure network round-trips (and after the- # premirror lane, almost all of them hit) — run them in parallel- # instead of ~1s each serially.- awk '{print $3}' preview-ctx/preload/manifest.txt \- | xargs -P 8 -I {} sh -c \- 'docker manifest inspect "{}" >/dev/null 2>&1 || echo "{}"' \- > "$RUNNER_TEMP/push-refs.txt"- echo "pushing $(wc -l < "$RUNNER_TEMP/push-refs.txt") of $(wc -l < preview-ctx/preload/manifest.txt) images"- # The Fly registry occasionally aborts large uploads ("s3aws:- # append to zero-size path unsupported") — retry; layers that made- # it are reused. Four pushes at a time keeps the first (cold) push- # moving without saturating the registry.- if [ -s "$RUNNER_TEMP/push-refs.txt" ]; then- xargs -P 4 -I {} sh -c '- for _ in 1 2 3; do- docker push "{}" && exit 0- echo "push of {} failed, retrying" >&2- sleep 15- done- exit 1' < "$RUNNER_TEMP/push-refs.txt"- fi-- push_image() {- image_to_push=$1- for _ in 1 2 3; do- if docker push "$image_to_push"; then return 0; fi- echo "docker push failed for $image_to_push, retrying" >&2- sleep 15- done- return 1- }-- if [ "$mode" = hot ]; then- update_image="$registry:update-${{ github.sha }}"- docker build -f preview-ctx/update.Dockerfile -t "$update_image" preview-ctx- push_image "$update_image"- token_file="$RUNNER_TEMP/registry-pull-token"- printf '%s' "$pull_token" > "$token_file"- chmod 600 "$token_file"- hot_rc=0- flyctl ssh sftp put "$token_file" /tmp/macro-registry-token \- --app "$APP_NAME" --machine "$machine_id" --mode 0600 \- || hot_rc=$?- if [ "$hot_rc" = 0 ]; then- # timeout backstop: a suspended-mid-session VM freezes the ssh- # client forever (measured: 52 min of hang). 10 min is triple- # a normal hot apply; on expiry we fall back to rehydrate.- hot_command="/srv/macro/bin/hot-update '$update_image' /tmp/macro-registry-token"- timeout 600 flyctl ssh console --app "$APP_NAME" --machine "$machine_id" \- --command "$hot_command" || hot_rc=$?- fi- if [ "$hot_rc" = 0 ]; then- echo "hot update completed without restarting the Fly machine"- exit 0- fi- echo "hot update failed (exit $hot_rc); falling back to full rehydrate" >&2- mode=rehydrate- fi-- # Bootstrap/rehydrate refreshes app secrets and reconciles the- # persistent Docker volume before replacing the machine image.- flyctl secrets set --app "$APP_NAME" --stage \- "DOPPLER_TOKEN=$DOPPLER_PREVIEW_TOKEN" \- "REGISTRY_PULL_TOKEN=$pull_token"- if [ -z "$vol_state" ]; then- # No volume and no fork landed (no template, or both fork- # attempts failed): the old empty-volume path.- flyctl volumes create docker_data --app "$APP_NAME" --region ewr --size 40 --yes- elif [ "$vol_state" != created ]; then- # A fork is still hydrating (usually already finished: the- # images lane kicked it before the cargo lane even ended). The- # volume must reach "created" before a machine can mount it; a- # fork that never lands is replaced by an empty volume rather- # than failing the deploy.- for _ in $(seq 1 60); do- vol_state=$(flyctl volumes list --app "$APP_NAME" --json \- | jq -r '[.[] | select(.name == "docker_data")][0].state // empty')- [ "$vol_state" = created ] && break- sleep 10- done- echo "seeded volume state: $vol_state"- if [ "$vol_state" != created ]; then- echo "template fork stuck (state: $vol_state); replacing with an empty volume" >&2- vol_id=$(flyctl volumes list --app "$APP_NAME" --json \- | jq -r '[.[] | select(.name == "docker_data")][0].id // empty')- [ -z "$vol_id" ] || flyctl volumes destroy "$vol_id" --app "$APP_NAME" --yes || true- flyctl volumes create docker_data --app "$APP_NAME" --region ewr --size 40 --yes- fi- fi- # Machines created before the volume existed can't take the mounts- # config update. A destroyed/partial machine is recreated by deploy.- flyctl machine list --app "$APP_NAME" --json \- | jq -r '.[] | select((.config.mounts // []) | length == 0) | .id' \- | while read -r id; do- [ -z "$id" ] || flyctl machine destroy "$id" --app "$APP_NAME" --force || true- done-- image="$registry:${{ github.sha }}"- docker build -t "$image" preview-ctx- push_image "$image"- # First boot does real work before 8090 opens (pulling the stack- # images, snapshot restore, compose up, FusionAuth's JVM) —- # give the health check more runway than flyctl's default wait.- # The step-wide keepalive above holds the machine out of suspend.- flyctl deploy --app "$APP_NAME" \- --config infra/preview/fly.toml \- --image "$image" \- --wait-timeout 1800 \- --yes- env:- FLY_ORG: ${{ vars.FLY_ORG || secrets.FLY_ORG }}- DOPPLER_PREVIEW_TOKEN: ${{ secrets.DOPPLER_PREVIEW_TOKEN }}- LOCAL_SANDBOX_IMAGE: macro-agent-harness:latest- - name: Dump Fly diagnostics- if: failure()- run: |- flyctl machine list --app "$APP_NAME" || true- # `flyctl logs --no-tail` can hang forever on its NATS fetch- # (observed: a healthy deploy stuck 1h53m in the timings dump) —- # never call it without a timeout.- timeout 60 flyctl logs --app "$APP_NAME" --no-tail || true- - name: Dump boot timings- run: |- timeout 120 flyctl ssh console --app "$APP_NAME" --quiet \- --command 'cat /var/lib/docker/.macro-preview/boot-timings.log' || true- - name: Publish template volume- run: |- set -euo pipefail- key=$(awk '{print $1, $2}' preview-ctx/preload/manifest.txt \- | LC_ALL=C sort | sha256sum | cut -c1-24)- vol_name="tpl$key"- flyctl status --app "$TEMPLATE_APP" >/dev/null 2>&1 \- || flyctl apps create "$TEMPLATE_APP" --org "$FLY_ORG"- vols=$(flyctl volumes list --app "$TEMPLATE_APP" --json 2>/dev/null || echo '[]')- if echo "$vols" | jq -e --arg n "$vol_name" \- 'map(select(.name == $n)) | length > 0' >/dev/null; then- echo "template $vol_name already exists; image set unchanged"- else- src=$(flyctl volumes list --app "$APP_NAME" --json \- | jq -r '[.[] | select(.name == "docker_data")][0].id // empty')- if [ -n "$src" ]; then- # Fire-and-forget: hydration is a server-side block copy;- # nothing in this run depends on it finishing.- curl -sf -X POST "https://api.machines.dev/v1/apps/$TEMPLATE_APP/volumes" \- -H "Authorization: Bearer $FLY_API_TOKEN" \- -H "Content-Type: application/json" \- -d "{\"name\":\"$vol_name\",\"region\":\"ewr\",\"source_volume_id\":\"$src\"}" \- | jq -r '"published \(.name) (\(.id), state: \(.state))"' \- || echo "template publish failed (non-fatal)" >&2- fi- fi- # Prune superseded templates: keep the 3 newest, and never touch- # anything younger than 2h — it may still be hydrating or serving- # as a concurrent deploy's fork source.- cutoff=$(date -u -d '2 hours ago' +%Y-%m-%dT%H:%M:%SZ)- echo "$vols" | jq -r --arg cutoff "$cutoff" \- '[.[] | select((.name | startswith("tpl")) and .attached_machine_id == null)]- | sort_by(.created_at) | reverse | .[3:]- | .[] | select(.created_at < $cutoff) | .id' \- | while read -r old; do- [ -z "$old" ] || flyctl volumes destroy "$old" --app "$TEMPLATE_APP" --yes || true- done- env:- FLY_ORG: ${{ vars.FLY_ORG || secrets.FLY_ORG }}- continue-on-error: true- - name: Comment preview URL- uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b- with:- github-token: ${{ secrets.GITHUB_TOKEN }}- script: |- const app = process.env.APP_NAME;- const url = `https://${app}.fly.dev`;- const marker = '<!-- fly-preview -->';- const body = [- marker,- `🚀 **Full-stack preview**: ${url}`,- '',- `- Suspended when idle — the first request wakes it (a few seconds).`,- `- Log in with any email; the passwordless code lands in [Mailpit](${url}/mailpit/).`,- `- Deployed from ${context.payload.pull_request.head.sha.slice(0, 7)}.`,- ].join('\n');- const { data: comments } = await github.rest.issues.listComments({- owner: context.repo.owner,- repo: context.repo.repo,- issue_number: context.issue.number,- per_page: 100,- });- const existing = comments.find(c => c.body && c.body.startsWith(marker));- if (existing) {- await github.rest.issues.updateComment({- owner: context.repo.owner,- repo: context.repo.repo,- comment_id: existing.id,- body,- });- } else {- await github.rest.issues.createComment({- owner: context.repo.owner,- repo: context.repo.repo,- issue_number: context.issue.number,- body,- });- }- - name: Teardown Nix- if: always()- uses: './.github/actions/teardown-nix'-concurrency:- group: fly-preview-${{ github.event.pull_request.number }}- cancel-in-progress: truediff --git a/CLAUDE.md b/CLAUDE.mdindex 2e3aa1412b..4152c6f537 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -317,7 +317,7 @@ don't inject it directly into the error message.## Cursor Cloud specific instructions-`.cursor/install.sh` prepares the durable caches, databases, test dependencies, frontend dependencies, service binaries, and stack init snapshot, then stops dockerd and nix-daemon so the Cloud bake can exit. `.cursor/start.sh` runs at boot and starts nothing beyond the nix daemon, so sessions and subagents are usable immediately. `.cursor/infra.sh` starts Docker, Postgres, and Redis on demand. `.cursor/stack.sh` starts the on-demand product stack. `.cursor/rebuild.sh` rebuilds the stack after backend edits. These scripts are the only supported entry points; each re-enters the pinned nix shell itself, so run them with plain `bash` from any environment.+`.cursor/install.sh` prepares the durable caches, databases, test dependencies, frontend dependencies, service binaries, and stack init snapshot, then stops dockerd and nix-daemon so the Cloud bake can exit. `.cursor/start.sh` runs at boot and starts nothing beyond the nix daemon, so sessions and subagents are usable immediately. `.cursor/infra.sh` starts Docker, Postgres, and Redis on demand. `.cursor/stack.sh` starts the on-demand product: backend containers behind the proxy (8090) plus the hot-reloading frontend dev server — the app is at http://localhost:3000/app and frontend edits apply on save (idempotent; a healthy stack is left alone). `.cursor/rebuild.sh` remounts new backend binaries after Rust edits. These scripts are the only supported entry points; each re-enters the pinned nix shell itself, so run them with plain `bash` from any environment. To run the app and see your edits, follow the `run-app` skill (`.claude/skills/run-app/SKILL.md`).Nix is the only host dependency. The pinned dev shell supplies the Docker CLI and daemon, Compose, `fuse-overlayfs`, and OpenSSH. `ssh-keygen` must come from this shell.@@ -329,8 +329,8 @@ Set `DOPPLER_TOKEN` as a Cursor environment **runtime** secret: a Doppler servicNothing runs after boot. Before DB-backed `cargo test -p <crate>`, run `bash .cursor/infra.sh` once — it brings up Docker, Postgres, and Redis in seconds because install baked the images and volumes. Pure-logic crate tests need nothing. Run `bash .cursor/stack.sh` for a product-ready environment.-After backend edits, run `bash .cursor/rebuild.sh`. Do not use `just stack update` with the read-only Nix binary directory.+After backend edits, run `bash .cursor/rebuild.sh`. That nix-builds the stack binaries and runs `just stack update --binaries-dir`, which remounts them without wiping volumes.-No seeding or OTP is needed to log in: passwordless login auto-creates a user for any email, and the stack's auth service is built with `return_passwordless_code`, so the login API returns the code in its response (codes are also visible at the proxy's `/mailpit/` UI). `just seed-scenario apply --file seed/scenarios/team-perms.json` is optional, for multi-user team/permission fixtures. The `agent_harness_service` restart loop is expected when AI provider keys are missing; with `DOPPLER_TOKEN` those keys come from `local`/`lcl_preview`.+No seeding or OTP is needed to log in: passwordless login auto-creates a user for any email, and the stack's auth service is built with `return_passwordless_code`, so the login API returns the code in its response (codes are also visible in Mailpit at http://localhost:8025). `just seed-scenario apply --file seed/scenarios/team-perms.json` is optional, for multi-user team/permission fixtures. The `agent_harness_service` restart loop is expected when AI provider keys are missing; with `DOPPLER_TOKEN` those keys come from `local`/`lcl_preview`.Leave `SQLX_OFFLINE` unset for `cargo test`. If SQLx reports missing cached query data, run `just prepare_db` instead of enabling offline mode. Run crate tests from the repository root with `cargo test -p <crate>`.diff --git a/apps/web/scripts/keep-import-meta-dev.test.ts b/apps/web/scripts/keep-import-meta-dev.test.tsindex 600f6835e5..b5c191a57a 100644--- a/apps/web/scripts/keep-import-meta-dev.test.ts+++ b/apps/web/scripts/keep-import-meta-dev.test.ts@@ -2,7 +2,7 @@ import { describe, expect, it } from 'vitest';import { keepImportMetaDev } from './keep-import-meta-dev';describe('keepImportMetaDev', () => {- it('keeps DEV for local-backend static builds (stack up, Fly preview)', () => {+ it('keeps DEV for local-backend static builds (stack up)', () => {expect(keepImportMetaDev({command: 'build',diff --git a/apps/web/scripts/keep-import-meta-dev.ts b/apps/web/scripts/keep-import-meta-dev.tsindex 3bfd74bc93..05b7670853 100644--- a/apps/web/scripts/keep-import-meta-dev.ts+++ b/apps/web/scripts/keep-import-meta-dev.ts@@ -3,7 +3,7 @@** Vite compiles DEV from NODE_ENV, not MODE, so* `MODE=development NODE_ENV=production vite build` would otherwise ship- * DEV=false. Local-backend bundles (`just stack up`, Fly preview) already set+ * DEV=false. Local-backend bundles (`just stack up`) already set* `VITE_LOCAL_BACKEND_ORIGIN`; hosted `just build-dev` / staging / prod do not.** `vite serve` already has DEV=true, so this only applies to `build`.diff --git a/apps/web/vite.base.ts b/apps/web/vite.base.tsindex 1f57061906..06b6be950b 100644--- a/apps/web/vite.base.ts+++ b/apps/web/vite.base.ts@@ -231,7 +231,7 @@ function getAssetsPath(mode: string, command: string): string {function defineEnv(mode: string, command: string) {// `vite build` compiles DEV from NODE_ENV, not MODE. Local-backend static- // bundles already set VITE_LOCAL_BACKEND_ORIGIN (stack up, Fly preview);+ // bundles already set VITE_LOCAL_BACKEND_ORIGIN (stack up);// keep DEV so those artifacts match `just run_local` (vite serve). Hosted// `just build-dev` does not set the origin, so DEV stays false.const keepDev = keepImportMetaDev({diff --git a/docs/RUNNING_LOCALLY.md b/docs/RUNNING_LOCALLY.mdindex 85e35e7104..db2f6dfd74 100644--- a/docs/RUNNING_LOCALLY.md+++ b/docs/RUNNING_LOCALLY.md@@ -256,12 +256,13 @@ just stack up # bring everything up, print URLs, returnjust stack status --json # machine-readable state (containers, health, URLs)just stack update # rebuild and reload only the changed services (the `r` hotkey)just stack update --frontend # also rebuild the frontend bundle+just stack update --binaries-dir <dir> # remount a prebuilt set; volumes stayjust stack down # remove containers, volumes, and state```-All the `run_local` flags apply to `stack` too. This includes `--instance`, `--no-doppler`, `--no-build`, and `--binaries-dir`. CI can pass a prebuilt bundle with `--frontend-dist`.+All the `run_local` flags apply to `stack` too. This includes `--instance`, `--no-doppler`, `--no-build`, and `--binaries-dir`.-The app is served at `<proxy>/app/`. The bundle resolves its backend from the origin it is served on. The same stack works on localhost or behind a preview hostname without a rebuild.+The app is served at `<proxy>/app/`. The bundle resolves its backend from the origin it is served on. The same stack works on localhost or behind any hostname without a rebuild.### Init Snapshots@@ -276,11 +277,10 @@ It saves these volumes as an init snapshot. The snapshot is content-addressed anUseful commands:```bash-just stack snapshot # show the current snapshot keyjust stack up --no-snapshot # skip the snapshot cache```-CI bakes the snapshot into the preview image. This is what makes Fly previews boot fast. See `infra/preview/README.md`.+Cursor Cloud bakes the snapshot during environment install. Later `stack up` restores it.## Common Commandsdiff --git a/infra/preview/Dockerfile b/infra/preview/Dockerfiledeleted file mode 100644index d131641c65..0000000000--- a/infra/preview/Dockerfile+++ /dev/null@@ -1,59 +0,0 @@-# The Fly preview machine image: the entire local stack inside one VM.-#-# Built by .github/workflows/preview-fly.yml from a STAGED context (not the-# repo root) — CI lays out everything the machine needs and this file just-# copies it in:-#-# repo/ minimal repo layout xtask reads (compose files, migrations,-# index helpers) — MACRO_REPO_ROOT points here-# artifacts/ binaries/ (zigbuild output), frontend-dist/ (same-origin-# bundle), snapshots/ (baked init snapshot)-# bin/ prebuilt orchestrator + the in-place hot-update driver-# preload/ manifest of every image the stack runs (image ID -> local tag-# -> registry ref); the entrypoint pulls them from the app's Fly-# registry repo at boot, skipping whatever the machine's-# persistent layer store already has-#-# Fly machines are full microVMs, so an inner Docker daemon runs the same-# compose topology as `just run_local` — the preview IS the local stack.-FROM ubuntu:24.04--RUN apt-get update && apt-get install -y --no-install-recommends \- docker.io \- docker-compose-v2 \- docker-buildx \- ca-certificates \- curl \- gnupg \- jq \- && rm -rf /var/lib/apt/lists/*--# Doppler CLI: the machine boots with a config-scoped service token-# (DOPPLER_TOKEN, set as a Fly secret by the workflow) and the stack's env-# layer pulls the preview secrets with it.-RUN curl -Ls --tlsv1.2 --proto "=https" https://cli.doppler.com/install.sh | sh--# One COPY per artifact, ordered stable → volatile: the staging step preserves-# mtimes, so a layer whose inputs didn't change this deploy is byte-identical-# to the previous image's — the registry push and the Fly host's pull both-# skip it. Everything after the first changed layer re-ships, so the pieces-# that churn every deploy (frontend bundle, service binaries) go last.-COPY entrypoint.sh /srv/macro/entrypoint.sh-COPY deployment.json /srv/macro/deployment.json-COPY repo/ /srv/macro/repo/-COPY bin/ /srv/macro/bin/-COPY artifacts/snapshots/ /srv/macro/artifacts/snapshots/-COPY preload/ /srv/macro/preload/-COPY artifacts/frontend-dist/ /srv/macro/artifacts/frontend-dist/-COPY artifacts/binaries/ /srv/macro/artifacts/binaries/-RUN chmod +x /srv/macro/entrypoint.sh /srv/macro/bin/xtask /srv/macro/bin/hot-update--# Point the prebuilt xtask at the staged layout (its compiled-in paths refer to-# the CI checkout) and at the baked snapshot store.-ENV MACRO_REPO_ROOT=/srv/macro/repo \- MACRO_STACK_SNAPSHOT_DIR=/srv/macro/artifacts/snapshots--# The instance's single-origin proxy (fly.toml's internal_port).-EXPOSE 8090--ENTRYPOINT ["/srv/macro/entrypoint.sh"]diff --git a/infra/preview/README.md b/infra/preview/README.mddeleted file mode 100644index f195a94d7e..0000000000--- a/infra/preview/README.md+++ /dev/null@@ -1,105 +0,0 @@-# Full-stack PR previews on Fly--Label a PR `preview` and CI deploys the **entire stack** — every service,-Postgres, OpenSearch, FusionAuth, LocalStack, the frontend — as one Fly app at-`https://macro-pr-<N>.fly.dev`. The app suspends when idle (≈zero cost) and the-fly-proxy wakes it on the next request. Closing the PR (or removing the label)-destroys it.--## How it works--The preview is not a parallel deployment system: it is the local stack-(`just stack up`, see `RUNNING_LOCALLY.md`) running inside one Fly machine.-A Fly machine is a full microVM, so an inner Docker daemon runs the exact-compose topology `run_local` uses. What differs is that **everything expensive-is baked in CI** and the machine only restores:--| Baked in CI | Restored on the machine |-|---|---|-| Service binaries (`cargo x zigbuild`) | bind-mounted into the runtime image |-| Frontend bundle (`same-origin` build) | served statically by the instance Caddy |-| Init snapshot (migrations + FusionAuth kickstart + search indices) | volumes restored, init skipped |-| Every Docker image, mirrored to the app's Fly registry repo | `docker pull` at boot — layer-dedup'd against the machine's persistent layer store, nothing built |--Boot on a fresh machine ≈ image pulls + snapshot restore + JVM startup-(a couple of minutes); wake from suspend ≈ seconds. Pushes to the PR choose one-of three paths from the durable deployment marker on the Docker volume:--- **bootstrap** — no app/machine yet: create everything and restore the- snapshot. The new volume is forked (Machines API `source_volume_id`) from- the newest **template volume** on `macro-preview-template` — a warm- `/var/lib/docker` layer store published by a previous successful deploy —- so the first boot pulls only image deltas instead of ~6GB cold (measured:- ~11 min of a ~19 min cold boot was image pulls). Templates are- content-addressed by the preload manifest's (image id, tag) pairs: after- any successful deploy, if no template exists for the current image set,- the deploy publishes its own volume as one (`tpl<hash>`), and old- templates are pruned. The template volume is a pure cache — the boot-time- manifest check pulls whatever a stale or missing template lacks, and every- fork failure falls back to the old empty-volume path;-- **hot update** — the snapshot key, staged-runtime hash, and update protocol- are unchanged: pull a layer-deduplicated artifact carrier, atomically replace- changed binaries/frontend, and restart only affected containers without- restarting the Fly machine or infra;-- **rehydrate** — compatibility changed, or a hot update fails: fall back to- the immutable full-image deploy and snapshot restore.--The URL remains stable across all three paths.--The whole product sits behind the stack's single-origin Caddy proxy-(`:8090`), so one `internal_port` covers the frontend, APIs, WebSockets, and-Mailpit.--## Logging in--Login is passwordless (email codes) and preview email is captured, not sent:-enter any address on the login screen, then open-`https://macro-pr-<N>.fly.dev/mailpit/` to read the code. (Routing login codes-through scoped SES so teammates get real emails is a planned follow-up.)--## Secrets (Doppler)--The stack's local plumbing (DBs, queues, auth identity) is code-owned and-dummy, but integration secrets (AI keys etc.) come from Doppler, exactly like-`run_local`. The workflow stages a **config-scoped service token** onto each-preview app as the Fly secret `DOPPLER_TOKEN`; at boot the env layer detects-the token and pulls that config (no `doppler login` involved), then overlays-the code-owned `LocalEnv` on top — so Doppler supplies integration keys while-local plumbing stays local. A preview machine can only ever read the one-config its token is scoped to.--## One-time setup--1. Create a Fly org deploy token: `fly tokens create org -o <org>` →- repo secret `FLY_API_TOKEN`.-2. Set the repo variable `FLY_ORG` to the org slug.-3. In Doppler, create a preview config in the `local` project (e.g.- `lcl_preview` — start from `lcl_personal`, keep only what previews should- have) and a service token for it → repo secret `DOPPLER_PREVIEW_TOKEN`.-4. Create the `preview` label.--## Security posture--- Previews only build from same-repo branches (the workflow refuses forks).-- No real AWS: S3/SQS/DynamoDB are LocalStack, email is Mailpit. The- cloud-reachable credentials on a machine are the preview Doppler config's- contents — scope that config accordingly (assume PR code can read it) — and- `REGISTRY_PULL_TOKEN`, the CI deploy token attenuated (client-side macaroon- caveats) to read-only on this one preview app with a 7-day validity window:- it can pull the app's registry repo, nothing else.-- URLs are public. Anyone with the link can use the preview (and read its- Mailpit). Don't put sensitive data in one. Edge auth (oauth2-proxy in front- of Caddy) is a planned hardening step.--## Known costs & future optimizations--- The stack images are mirrored into the per-PR app's registry repo, so the- first deploy of a PR pushes the full set; after that pushes move only- changed layers. The **pull** side of that cost is gone (template volume- forks, above), but there is still no cross-PR sharing on the push side —- a shared registry (GHCR/Namespace) would fix that at the cost of new- org-level credentials.-- The aux images (sync/websocket/lexical) rebuild in CI on every preview- deploy. Runner-level Docker layer caching covers most of it.-- `cpus = 8, memory = 16gb` (shared) is a deliberate over-provision; measure- and shrink once a few previews have run.diff --git a/infra/preview/entrypoint.sh b/infra/preview/entrypoint.shdeleted file mode 100644index 8234216251..0000000000--- a/infra/preview/entrypoint.sh+++ /dev/null@@ -1,129 +0,0 @@-#!/usr/bin/env bash-# Boot for the Fly preview machine: start the inner Docker daemon, pull the-# stack images from the app's registry repo, bring the stack up from the baked-# artifacts, then idle while the containers serve. Everything expensive-# (compile, bundle, init) happened in CI — this script only restores and-# starts.-set -euo pipefail--state_dir=/var/lib/docker/.macro-preview-mkdir -p "$state_dir"-# Fly's log stream is short-retention and `flyctl logs --no-tail` can hang-# forever, so the boot breakdown also lands in a volume-backed file that CI-# reads back with a plain `ssh cat`. Truncated here, not appended: the file-# always describes the most recent real boot.-timings="$state_dir/boot-timings.log"-: > "$timings"-log() { echo "[preview] $*" | tee -a "$timings"; }--t_boot=$(date +%s)-# A hard loss during a hot update cannot run its EXIT trap. The full immutable-# boot is the recovery boundary, so clear any abandoned volume-backed lock.-rmdir "$state_dir/update.lock" 2>/dev/null || true-log "starting inner dockerd"-dockerd >/var/log/dockerd.log 2>&1 &--for _ in $(seq 1 120); do- docker info >/dev/null 2>&1 && break- sleep 1-done-if ! docker info >/dev/null 2>&1; then- log "dockerd failed to start:"- tail -n 50 /var/log/dockerd.log- exit 1-fi-log "dockerd ready in $(($(date +%s) - t_boot))s"--# A template-forked volume carries the source deploy's containers; any with a-# restart policy auto-start with dockerd and can answer :8090 with the OLD-# stack — fly-proxy then reports the machine healthy before the real stack-# exists, and a visitor gets the stale app followed by a teardown blackout.-# Kill them before anything can serve; `stack up` recreates properly.-stale=$(docker ps -q)-if [ -n "$stale" ]; then- log "killing $(echo "$stale" | wc -l) auto-started containers from the volume source"- echo "$stale" | xargs -r docker kill >/dev/null 2>&1 || true-fi--# Pull the stack images from the app's Fly registry repo (CI mirrored them-# there at deploy time). /var/lib/docker is a persistent volume, so the-# manifest (image ID per tag) lets redeploys skip every image already in the-# store, and pulls dedup at the layer level — only layers that actually-# changed cross the network. Suspend/resume never re-runs this; only real-# boots do. REGISTRY_PULL_TOKEN is an app-scoped deploy token staged as a Fly-# secret by the workflow.-if [ -f /srv/macro/preload/manifest.txt ]; then- if [ -n "${REGISTRY_PULL_TOKEN:-}" ]; then- docker login registry.fly.io -u x --password-stdin <<<"$REGISTRY_PULL_TOKEN"- fi- log "pulling stack images (parallel, warm store aware)"- t0=$(date +%s)- # The manifest's IDs are the RUNNER's view of each image (for hub images- # under the containerd store, the source registry's index digest). Pulling- # the mirror yields a different manifest and therefore a different local- # .Id, so the inspect check below never matches mirrored images and every- # boot re-pulled all 16 (66s of pure round-trips on a fully warm volume).- # The receipt is our own volume-persisted record of which (runner id, tag)- # pairs this store already satisfied — written only after a fully- # successful pass, carried along by template-volume forks, and at worst- # stale in the direction of one redundant pull.- receipt="$state_dir/pulled.txt"- pids=""- while read -r id tag ref; do- have=$(docker image inspect -f '{{.Id}}' "$tag" 2>/dev/null || true)- if [ "$have" = "$id" ]; then- log "already present: $tag"- continue- fi- if [ -n "$have" ] && grep -qxF "$id $tag" "$receipt" 2>/dev/null; then- log "already present (receipt): $tag"- continue- fi- (docker pull -q "$ref" && docker tag "$ref" "$tag" && docker rmi "$ref" >/dev/null) &- pids="$pids $!"- done < /srv/macro/preload/manifest.txt- rc=0- for pid in $pids; do wait "$pid" || rc=1; done- [ "$rc" = 0 ] || { log "image pull failed"; exit 1; }- awk '{print $1, $2}' /srv/macro/preload/manifest.txt > "$receipt.next" \- && mv "$receipt.next" "$receipt"- log "images pulled in $(($(date +%s) - t0))s"-fi--# With a DOPPLER_TOKEN (a config-scoped service token, injected as a Fly-# secret) the env layer pulls the preview secrets; without one the stack runs-# on the code-owned local dummies only.-doppler_args=(--no-doppler)-if [ -n "${DOPPLER_TOKEN:-}" ]; then- log "DOPPLER_TOKEN present — pulling preview secrets"- doppler_args=()-fi--log "bringing the stack up (snapshot restore)"-t0=$(date +%s)-# tee the per-stage timings into the boot log; pipefail keeps a stack-up-# failure fatal through the pipe.-/srv/macro/bin/xtask stack up \- "${doppler_args[@]}" \- --no-build \- --binaries-dir /srv/macro/artifacts/binaries \- --frontend-dist /srv/macro/artifacts/frontend-dist \- --json 2>&1 | tee -a "$timings"--log "stack up took $(($(date +%s) - t0))s (boot total $(($(date +%s) - t_boot))s)"-# The backend health gate is the longest stack-up phase; surface the auth-# service's step timings plus its timestamped tail so the wait is attributable-# from the CI boot-timings step (which ssh-cats the timings file).-docker logs -t macro-authentication-service-1 2>&1 \- | grep 'authentication startup step' \- | sed 's/^/[preview][auth-startup] /' | tee -a "$timings" || true-docker logs -t --tail 40 macro-authentication-service-1 2>&1 \- | sed 's/^/[preview][auth] /' | tee -a "$timings" || true--# Commit the compatibility marker only after the restored stack is healthy.-# It lives beside (not inside) Docker's own data on the persistent volume, so-# CI can distinguish a hot-updatable machine from an absent/partial/old deploy.-install -m 0644 /srv/macro/deployment.json "$state_dir/deployment.json.next"-mv -f "$state_dir/deployment.json.next" "$state_dir/deployment.json"-log "stack ready — proxy serving on :8090"-exec sleep infinitydiff --git a/infra/preview/fly.toml b/infra/preview/fly.tomldeleted file mode 100644index 2cd6e60366..0000000000--- a/infra/preview/fly.toml+++ /dev/null@@ -1,46 +0,0 @@-# Fly config for per-PR preview apps. The workflow overrides the app name-# (`--app macro-pr-<N>`) and supplies the image (`--image`); this file carries-# everything else. See infra/preview/README.md.-app = "macro-preview-template"-primary_region = "ewr"--# Everything (frontend, APIs, WebSockets, Mailpit) is behind the stack's-# single-origin Caddy proxy.-[http_service]- internal_port = 8090- force_https = true- # The whole point: an idle preview suspends to ~zero cost and the fly-proxy- # wakes it on the next request.- auto_stop_machines = "suspend"- auto_start_machines = true- min_machines_running = 0-- [[http_service.checks]]- interval = "15s"- timeout = "5s"- # First boot loads the baked images and restores the init snapshot before- # the backend answers — give it room.- grace_period = "5m"- method = "GET"- path = "/auth/health"--# The inner Docker store must NOT live on the machine rootfs: Fly's rootfs is-# an overlay (nested overlayfs forbids the whiteout device nodes layer-# unpacking needs — every Debian-based image failed with EPERM) and has only-# a few GB of headroom (ENOSPC mid-load). A volume is real ext4, properly-# sized, and persists across redeploys — so later pushes reuse the layer-# store. The deploy step creates the volume if it doesn't exist.-[[mounts]]- source = "docker_data"- destination = "/var/lib/docker"--[[vm]]- # performance (dedicated) cores: boot is JVM/startup-latency bound and- # shared cores throttle exactly then. ~16x the CPU $/hr of shared, but- # compute only bills while awake and previews suspend when idle.- cpu_kind = "performance"- cpus = 8- memory = "16gb"--[[restart]]- policy = "on-failure"diff --git a/infra/preview/hot-update.sh b/infra/preview/hot-update.shdeleted file mode 100644index 6239a6453b..0000000000--- a/infra/preview/hot-update.sh+++ /dev/null@@ -1,136 +0,0 @@-#!/usr/bin/env bash-# Apply a registry-carried artifact update to an already-running preview VM.-# Exit 42 asks CI to fall back to a full image deploy/stack rehydrate.-set -euo pipefail--readonly REHYDRATE_EXIT=42-readonly STATE_DIR=/var/lib/docker/.macro-preview-readonly STATE_FILE="$STATE_DIR/deployment.json"--log() { echo "[preview][hot-update] $*"; }-rehydrate() { log "$*; requesting full rehydrate" >&2; exit "$REHYDRATE_EXIT"; }--if [ "$#" -ne 2 ]; then- echo "usage: hot-update <registry-image> <registry-token-file>" >&2- exit 2-fi-update_image=$1-token_file=$2--mkdir -p "$STATE_DIR"-if ! mkdir "$STATE_DIR/update.lock" 2>/dev/null; then- echo "another preview update is already running" >&2- exit 1-fi--staging=-container=-cleanup() {- rm -f "$token_file"- [ -z "$container" ] || docker rm -f "$container" >/dev/null 2>&1 || true- [ -z "$staging" ] || rm -rf "$staging"- rmdir "$STATE_DIR/update.lock" 2>/dev/null || true-}-trap cleanup EXIT--[ -s "$token_file" ] || rehydrate "registry pull token is missing"-[ -f "$STATE_FILE" ] || rehydrate "deployment state is missing"--log "pulling artifact image $update_image"-docker login registry.fly.io -u x --password-stdin < "$token_file" >/dev/null-docker pull -q "$update_image"-container=$(docker create "$update_image")-staging=$(mktemp -d /srv/macro/.hot-update.XXXXXX)-docker cp "$container:/update/." "$staging/"-docker rm "$container" >/dev/null-container=--new_format=$(jq -er '.format' "$staging/deployment.json")-new_snapshot=$(jq -er '.snapshot_key' "$staging/deployment.json")-new_runtime=$(jq -er '.runtime_key' "$staging/deployment.json")-new_frontend=$(jq -er '.frontend_key' "$staging/deployment.json")-current_format=$(jq -er '.format' "$STATE_FILE")-current_snapshot=$(jq -er '.snapshot_key' "$STATE_FILE")-current_runtime=$(jq -er '.runtime_key' "$STATE_FILE")-current_frontend=$(jq -er '.frontend_key' "$STATE_FILE")-[ "$new_format" = "1" ] || rehydrate "unsupported update format $new_format"-[ "$current_format" = "$new_format" ] || rehydrate "deployment format changed"-[ "$current_snapshot" = "$new_snapshot" ] || rehydrate "snapshot key changed"-[ "$current_runtime" = "$new_runtime" ] || rehydrate "runtime configuration changed"--# CI mirrors every compose image and puts the desired IDs in the carrier.-# Docker-built leaf services are safe to recreate in place. Any other image-# change is infrastructure/runtime drift and must use the full rehydrate path.-aux_changed=-unsafe_images=-while read -r expected_id tag registry_ref; do- current_id=$(docker image inspect -f '{{.Id}}' "$tag" 2>/dev/null || true)- [ "$current_id" = "$expected_id" ] && continue- # Mirrored images never match by .Id (the manifest carries the runner's- # source-registry digest; this store holds the mirror's) — same mismatch- # the entrypoint's pull loop handles. Honor the boot pull receipt before- # classifying an image as changed, or every hot update sees all 12 infra- # images as "non-hot-updatable drift" and needlessly rehydrates.- if [ -n "$current_id" ] \- && grep -qxF "$expected_id $tag" "$STATE_DIR/pulled.txt" 2>/dev/null; then- continue- fi- case "$tag" in- *-ai_editing_worker|*-lexical_service|*-sync_service|*-websocket_service)- log "refreshing auxiliary image $tag"- docker pull -q "$registry_ref"- pulled_id=$(docker image inspect -f '{{.Id}}' "$registry_ref")- [ "$pulled_id" = "$expected_id" ] \- || rehydrate "registry image ID mismatch for $tag"- docker tag "$registry_ref" "$tag"- docker rmi "$registry_ref" >/dev/null- aux_changed=1- ;;- *) unsafe_images="$unsafe_images $tag" ;;- esac-done < "$staging/manifest.txt"-[ -z "$unsafe_images" ] || rehydrate "non-hot-updatable images changed:$unsafe_images"--# SSH sessions don't inherit app secrets. Recover only the config-scoped-# Doppler token from PID 1 without printing it, then let xtask regenerate the-# local env file before recreating/restarting containers.-doppler_token=$(tr '\0' '\n' < /proc/1/environ | sed -n 's/^DOPPLER_TOKEN=//p' | head -n 1)-[ -z "$doppler_token" ] || export DOPPLER_TOKEN="$doppler_token"-export MACRO_REPO_ROOT=/srv/macro/repo-export MACRO_STACK_SNAPSHOT_DIR=/srv/macro/artifacts/snapshots--apply_args=(- stack update- --binaries-dir "$staging/binaries"- --json-)-[ "$current_frontend" = "$new_frontend" ] \- || apply_args+=(--frontend-dist "$staging/frontend-dist")-[ -z "$aux_changed" ] || apply_args+=(--recreate-aux-services)--log "applying prebuilt artifacts"-"$staging/xtask" "${apply_args[@]}"--# Publish the new orchestrator and compatibility marker only after the stack is-# healthy. A failed apply leaves the old marker, so the next run rehydrates.-install -m 0755 "$staging/xtask" /srv/macro/bin/xtask.next-mv -f /srv/macro/bin/xtask.next /srv/macro/bin/xtask-install -m 0644 "$staging/deployment.json" "$STATE_FILE.next"-mv -f "$STATE_FILE.next" "$STATE_FILE"-# The aux refreshes above satisfied the carrier manifest, so record it in the-# boot pull receipt — otherwise the next real boot re-pulls what this update-# already delivered (see the receipt logic in entrypoint.sh).-awk '{print $1, $2}' "$staging/manifest.txt" > "$STATE_DIR/pulled.txt.next" \- && mv "$STATE_DIR/pulled.txt.next" "$STATE_DIR/pulled.txt"--# Retain the current carrier's layers for the next incremental pull, but remove-# older carrier tags and their now-unreferenced layers from the persistent store.-repository=${update_image%:*}-while read -r old; do- case "$old" in- "$repository":update-*) [ "$old" = "$update_image" ] || docker rmi "$old" >/dev/null || true ;;- esac-done < <(docker images "$repository" --format '{{.Repository}}:{{.Tag}}')-docker image prune -f >/dev/null || true--log "update complete"diff --git a/infra/preview/update.Dockerfile b/infra/preview/update.Dockerfiledeleted file mode 100644index 28084f0e25..0000000000--- a/infra/preview/update.Dockerfile+++ /dev/null@@ -1,13 +0,0 @@-# Registry-carried artifact bundle for a running preview machine. The image is-# never started: hot-update.sh creates a stopped container and copies /update-# out of it. Scratch keeps every layer limited to actual application artifacts,-# and stable COPY ordering maximizes registry/inner-Docker dedup across pushes.-FROM scratch--COPY bin/xtask /update/xtask-COPY preload/manifest.txt /update/manifest.txt-COPY deployment.json /update/deployment.json-COPY artifacts/frontend-dist/ /update/frontend-dist/-COPY artifacts/binaries/ /update/binaries/--CMD ["/update/xtask"]diff --git a/tooling/just/xtask.just b/tooling/just/xtask.justindex 5e3ecc5df3..3ca878e212 100644--- a/tooling/just/xtask.just+++ b/tooling/just/xtask.just@@ -36,9 +36,9 @@ destroy_local *ARGS:stop_local *ARGS:{{ xtask }} stop-local "$@"-# Headless stack orchestration for previews, agents, and CI.+# Headless stack orchestration for agents and CI.# just stack up # full stack; static frontend served by the proxy-# just stack update # rebuild + reload only what changed (the `r` hotkey)+# just stack update # adopt a new build; volumes stay# just stack status --json # machine-readable state# just stack down # reclaim everythingstack *ARGS:diff --git a/tooling/xtask/crates/xtask_local/src/local.rs b/tooling/xtask/crates/xtask_local/src/local.rsindex 012bd1786a..3da21dff31 100644--- a/tooling/xtask/crates/xtask_local/src/local.rs+++ b/tooling/xtask/crates/xtask_local/src/local.rs@@ -189,16 +189,6 @@ const LOCAL_RECREATE_SERVICE_IMAGES: &[&str] = &[/// separately by [`build::ensure_runtime_image`].const LOCAL_PULL_SERVICE_IMAGES: &[&str] = &["proxy", "mailpit", "static_file_cdn"];-/// The artifact-driven preview updater can refresh every Docker-built app-/// service. Keep this separate so enabling local aux rebuilds does not make the-/// existing `run_local` edit loop rebuild the AI worker too.-const PREVIEW_AUX_SERVICE_IMAGES: &[&str] = &[- "websocket_service",- "sync_service",- "lexical_service",- "ai_editing_worker",-];-/// Bring up a Local or Dev stack and (unless `--no-frontend`) the frontend.pub fn run_stack(mode: Mode, args: &cli::RunArgs) -> Result<()> {if mode.spec().runs_local_infra {@@ -601,17 +591,6 @@ fn recreate_aux_service_containers(stage.run("Recreating auxiliary service containers", &mut up)}-fn recreate_preview_aux_service_containers(- stage: &Stage,- instance: &Instance,- env: &env_layer::ResolvedEnv,-) -> Result<()> {- let mut up = compose_cmd(instance, env);- up.args(["up", "-d", "--force-recreate", "--no-deps"])- .args(PREVIEW_AUX_SERVICE_IMAGES);- stage.run("Recreating preview auxiliary services", &mut up)-}-/// How the local infra reaches its initialized state on bring-up.#[derive(Clone, Copy, PartialEq, Eq)]enum InfraInit {@@ -680,8 +659,8 @@ fn bring_up_infra(// equivalent of the MSK topic provisioning driven by the generated// `.github/kafka-cluster-topics.json`. Restored volumes already carry// the topics (they live in the broker's data dir), so only a full init- // provisions — which also means a snapshot-restoring `stack up` (e.g.- // the preview VM) never needs the rdkafka-backed `local-stack` feature.+ // provisions — which also means a snapshot-restoring `stack up`+ // never needs the rdkafka-backed `local-stack` feature.if init == InfraInit::Full {stage.run_step("Creating Kafka topics", || kafka::provision(instance))?;}diff --git a/tooling/xtask/crates/xtask_local/src/local/build/binaries_dir.rs b/tooling/xtask/crates/xtask_local/src/local/build/binaries_dir.rsindex 3cabc1f91d..f878d88470 100644--- a/tooling/xtask/crates/xtask_local/src/local/build/binaries_dir.rs+++ b/tooling/xtask/crates/xtask_local/src/local/build/binaries_dir.rs@@ -3,6 +3,7 @@//! mount, not how the binaries were produced.use std::path::{Path, PathBuf};+use std::process::Command;use anyhow::{Context, Result, bail};@@ -17,6 +18,15 @@ pub enum BinariesDir {NixStore(PathBuf),}+/// How a running stack adopts a newly located [`BinariesDir`].+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum Adoption {+ /// Same canonical host dir. Content-addressed Nix paths are byte-identical.+ Unchanged,+ /// Different host dir. Rewrite compose mounts and recreate Rust containers.+ Remount,+}+impl BinariesDir {/// Classify a directory by whether it lives under `/nix/store`.pub fn classify(dir: &Path) -> Result<Self> {@@ -78,4 +88,97 @@ impl BinariesDir {}Ok(())}++ /// Compare this set to the dir currently bind-mounted at `/app/out`.+ pub fn adoption_from(&self, mounted_dir: &Path) -> Adoption {+ let next = canonicalize_or_clone(self.host_dir());+ let mounted = canonicalize_or_clone(mounted_dir);+ if next == mounted {+ Adoption::Unchanged+ } else {+ Adoption::Remount+ }+ }++ /// Same as [`Self::adoption_from`], except a missing record means remount.+ /// Legacy `stack.json` rows have no `binaries_dir`.+ pub fn adoption_from_recorded(&self, recorded: Option<&Path>) -> Adoption {+ match recorded {+ Some(mounted) => self.adoption_from(mounted),+ None => Adoption::Remount,+ }+ }++ /// Keep a Nix store path alive while containers still mount it.+ ///+ /// `nix build --out-link` on Cloud flips the caller's link to the new+ /// path. Pin each generation under `roots_dir` with its own `--out-link`.+ /// Never rename those links: Nix registers the path, not the inode, so a+ /// rename drops the root.+ pub fn pin_gc_root(&self, roots_dir: &Path) -> Result<()> {+ let Self::NixStore(dir) = self else {+ return Ok(());+ };+ std::fs::create_dir_all(roots_dir)+ .with_context(|| format!("creating {}", roots_dir.display()))?;+ let store_output = store_output_dir(dir);+ let current = roots_dir.join("nix-binaries");+ if current.exists()+ && canonicalize_or_clone(¤t) == canonicalize_or_clone(store_output)+ {+ return Ok(());+ }+ if current.exists() {+ publish_gc_root(+ &roots_dir.join("nix-binaries.prev"),+ &canonicalize_or_clone(¤t),+ )?;+ }+ publish_gc_root(¤t, store_output)+ }++ /// Drop the previous generation's pin after the new mount is recorded.+ pub fn release_previous_gc_root(roots_dir: &Path) {+ let _ = std::fs::remove_file(roots_dir.join("nix-binaries.prev"));+ }+}++/// Register `link` as a Nix GC root for `store_output`. Writes `--out-link`+/// at `link` itself so the auto-root path stays stable.+fn publish_gc_root(link: &Path, store_output: &Path) -> Result<()> {+ let status = Command::new("nix")+ .args(["build", "--out-link"])+ .arg(link)+ .arg(store_output)+ .status()+ .with_context(|| {+ format!(+ "running nix build --out-link {} {}",+ link.display(),+ store_output.display()+ )+ })?;+ if status.success() {+ return Ok(());+ }+ bail!(+ "nix build --out-link {} {} failed with {status}",+ link.display(),+ store_output.display()+ );}++fn canonicalize_or_clone(path: &Path) -> PathBuf {+ path.canonicalize().unwrap_or_else(|_| path.to_path_buf())+}++fn store_output_dir(bin_dir: &Path) -> &Path {+ if bin_dir.file_name().and_then(|n| n.to_str()) == Some("bin") {+ bin_dir.parent().unwrap_or(bin_dir)+ } else {+ bin_dir+ }+}++#[cfg(test)]+mod test;diff --git a/tooling/xtask/crates/xtask_local/src/local/build/binaries_dir/test.rs b/tooling/xtask/crates/xtask_local/src/local/build/binaries_dir/test.rsnew file mode 100644index 0000000000..fde8387df7--- /dev/null+++ b/tooling/xtask/crates/xtask_local/src/local/build/binaries_dir/test.rs@@ -0,0 +1,83 @@+use super::*;++fn scratch(name: &str) -> PathBuf {+ let root = std::env::temp_dir().join(format!(+ "macro-binaries-{}-{}-{name}",+ std::process::id(),+ std::time::SystemTime::now()+ .duration_since(std::time::UNIX_EPOCH)+ .unwrap()+ .as_nanos()+ ));+ let _ = std::fs::remove_dir_all(&root);+ std::fs::create_dir_all(&root).unwrap();+ root+}++#[test]+fn adoption_is_unchanged_for_the_same_path() {+ let root = scratch("same");+ let set = BinariesDir::TargetDir(root.clone());+ assert_eq!(set.adoption_from(&root), Adoption::Unchanged);+ let _ = std::fs::remove_dir_all(&root);+}++#[test]+fn adoption_is_remount_when_the_host_dir_changes() {+ let a = PathBuf::from("/tmp/macro-binaries-a");+ let b = PathBuf::from("/tmp/macro-binaries-b");+ let set = BinariesDir::TargetDir(a);+ assert_eq!(set.adoption_from(&b), Adoption::Remount);+}++#[test]+fn missing_record_is_a_remount() {+ let set = BinariesDir::TargetDir(PathBuf::from("/tmp/macro-binaries-recorded"));+ assert_eq!(set.adoption_from_recorded(None), Adoption::Remount);+ assert_eq!(+ set.adoption_from_recorded(Some(Path::new("/tmp/macro-binaries-recorded"))),+ Adoption::Unchanged+ );+}++#[test]+fn classify_rejects_a_missing_dir() {+ let err = BinariesDir::classify(Path::new("/tmp/macro-does-not-exist-binaries"))+ .expect_err("missing dir");+ assert!(err.to_string().contains("does not exist"));+}++#[test]+fn target_dir_pin_is_a_noop() {+ let root = scratch("target-pin");+ let set = BinariesDir::TargetDir(root.join("bins"));+ set.pin_gc_root(&root.join("roots")).unwrap();+ assert!(!root.join("roots").join("nix-binaries").exists());+ let _ = std::fs::remove_dir_all(&root);+}++#[test]+fn nix_pin_fails_when_out_link_cannot_register() {+ let root = scratch("nix-pin-fail");+ let out = root.join("out-a");+ std::fs::create_dir_all(out.join("bin")).unwrap();+ let err = BinariesDir::NixStore(out.join("bin"))+ .pin_gc_root(&root.join("roots"))+ .expect_err("nix build --out-link must register a real GC root");+ let msg = format!("{err:#}");+ assert!(+ msg.contains("nix build --out-link"),+ "unexpected pin error: {msg}"+ );+ let _ = std::fs::remove_dir_all(&root);+}++#[test]+fn release_previous_gc_root_deletes_the_prev_link() {+ let root = scratch("release-prev");+ let prev = root.join("nix-binaries.prev");+ std::fs::write(&prev, "").unwrap();+ BinariesDir::release_previous_gc_root(&root);+ assert!(!prev.exists());+ let _ = std::fs::remove_dir_all(&root);+}diff --git a/tooling/xtask/crates/xtask_local/src/local/build/mod.rs b/tooling/xtask/crates/xtask_local/src/local/build/mod.rsindex 6c52a5e078..f94993c3d2 100644--- a/tooling/xtask/crates/xtask_local/src/local/build/mod.rs+++ b/tooling/xtask/crates/xtask_local/src/local/build/mod.rs@@ -9,7 +9,7 @@ mod binaries_dir;mod runtime_image;mod zigbuild;-pub use binaries_dir::BinariesDir;+pub use binaries_dir::{Adoption, BinariesDir};pub use runtime_image::{RUNTIME_IMAGE_TAG, ensure_runtime_image};use anyhow::Result;diff --git a/tooling/xtask/crates/xtask_local/src/local/cli.rs b/tooling/xtask/crates/xtask_local/src/local/cli.rsindex 168bd88b32..f08888d90e 100644--- a/tooling/xtask/crates/xtask_local/src/local/cli.rs+++ b/tooling/xtask/crates/xtask_local/src/local/cli.rs@@ -51,8 +51,8 @@ enum Cmd {DestroyLocal(InstanceArgs),/// Start, seed, and test an isolated local E2E stack.LocalE2e(super::e2e::LocalE2eArgs),- /// Headless stack orchestration (previews, agents, CI) — no TTY, no- /// attached dev server; the proxy serves a static frontend bundle.+ /// Headless stack orchestration (agents, CI) — no TTY, no attached+ /// dev server; the proxy serves a static frontend bundle.#[command(subcommand)]Stack(StackCmd),}@@ -61,14 +61,12 @@ enum Cmd {pub enum StackCmd {/// Bring a full local stack up and return (only containers keep running).Up(super::stack::UpArgs),- /// Rebuild binaries (and optionally the frontend) and reload what changed.+ /// Adopt a new build into the running stack. Data survives.Update(super::stack::UpdateArgs),/// Report the instance's containers, health, and URLs (`--json` for machines).Status(super::stack::StatusArgs),/// Tear the instance down: containers, volumes, and state.Down(super::stack::DownArgs),- /// Report the init-snapshot key/location for this instance.- Snapshot(super::stack::SnapshotArgs),}#[derive(Args, Clone, Default)]@@ -259,7 +257,6 @@ fn run(cli: Cli) -> Result<()> {StackCmd::Update(a) => super::stack::update(&a),StackCmd::Status(a) => super::stack::status(&a),StackCmd::Down(a) => super::stack::down(&a),- StackCmd::Snapshot(a) => super::stack::snapshot_status(&a),},}}diff --git a/tooling/xtask/crates/xtask_local/src/local/cli/test.rs b/tooling/xtask/crates/xtask_local/src/local/cli/test.rsindex 225380b960..663810ecfb 100644--- a/tooling/xtask/crates/xtask_local/src/local/cli/test.rs+++ b/tooling/xtask/crates/xtask_local/src/local/cli/test.rs@@ -25,6 +25,35 @@ fn local_e2e_accepts_suite_and_trailing_test_arguments() {);}+#[test]+fn stack_update_accepts_binaries_dir_with_build_aux_services() {+ let cli = Cli::try_parse_from([+ "cargo-x",+ "stack",+ "update",+ "--binaries-dir",+ "/tmp/bins",+ "--build-aux-services",+ ])+ .unwrap();+ let Cmd::Stack(StackCmd::Update(args)) = cli.command else {+ panic!("expected stack update");+ };+ assert_eq!(+ args.binaries_dir.as_deref(),+ Some(std::path::Path::new("/tmp/bins"))+ );+ assert!(args.build_aux_services);+}++#[test]+fn stack_has_no_snapshot_verb() {+ match Cli::try_parse_from(["cargo-x", "stack", "snapshot"]) {+ Err(err) => assert_eq!(err.kind(), clap::error::ErrorKind::InvalidSubcommand),+ Ok(_) => panic!("stack snapshot must not parse"),+ }+}+#[test]fn seed_scenario_accepts_instance_and_trailing_scenario_arguments() {let cli = Cli::try_parse_from([diff --git a/tooling/xtask/crates/xtask_local/src/local/env_layer.rs b/tooling/xtask/crates/xtask_local/src/local/env_layer.rsindex ef6bd86675..5489e5a50a 100644--- a/tooling/xtask/crates/xtask_local/src/local/env_layer.rs+++ b/tooling/xtask/crates/xtask_local/src/local/env_layer.rs@@ -199,7 +199,7 @@ fn pull_aws_credentials(env: &mut BTreeMap<String, String>) {/// Pull Doppler secrets as a flat env overlay (mirrors the old/// `just get_environment`). Two modes:///-/// - `DOPPLER_TOKEN` in the process env (CI, Fly preview machines) is a+/// - `DOPPLER_TOKEN` in the process env (CI, Cloud agents) is a/// config-scoped service token: download WITHOUT `--project`/`--config` (the/// token pins them), and treat failure as fatal — whoever provisioned the/// token explicitly asked for these secrets, so proceeding without themdiff --git a/tooling/xtask/crates/xtask_local/src/local/frontend.rs b/tooling/xtask/crates/xtask_local/src/local/frontend.rsindex 87f2982cfb..3e2b7f6550 100644--- a/tooling/xtask/crates/xtask_local/src/local/frontend.rs+++ b/tooling/xtask/crates/xtask_local/src/local/frontend.rs@@ -4,7 +4,6 @@use std::io::Read;use std::os::unix::process::CommandExt;-use std::path::Path;use std::process::{Child, Command, Stdio};use std::sync::{Arc, Mutex};use std::thread::JoinHandle;@@ -40,40 +39,30 @@ pub fn static_dir(instance: &Instance) -> std::path::PathBuf {}/// Build the app bundle for headless serving and stage it into the instance-/// dir. `prebuilt` skips the build and stages an existing dist (CI hands the-/// binaries-style artifact straight in). The build mirrors `just build-dev`+/// dir. The build mirrors `just build-dev`/// (dev-mode bundle, production optimizations), except the backend origin is/// the `same-origin` sentinel — resolved from `location.origin` at runtime — so-/// the one bundle works on localhost and through any tunnel/preview hostname.+/// the one bundle works on localhost and through any tunneled hostname.////// Setting `VITE_LOCAL_BACKEND_ORIGIN` also keeps `import.meta.env.DEV` true/// in the static bundle (see `keepImportMetaDev` in apps/web/scripts). `vite build`/// otherwise compiles DEV from `NODE_ENV=production`, which drops local-only/// paths such as passwordless auto-login (`just run_local` uses `vite serve`,-/// where DEV is already true). Fly preview sets the same origin env, so it-/// gets the override without a second flag.-pub fn build_static(- stage: &Stage,- instance: &Instance,- mode: Mode,- prebuilt: Option<&Path>,-) -> Result<()> {- let dist = match prebuilt {- Some(dir) => dir.to_owned(),- None => {- let mut cmd = Command::new("bun");- cmd.current_dir(app_dir())- .args(["run", "--bun", "build"])- .env("MODE", "development")- .env("NODE_ENV", "production")- .env("VITE_LOCAL_SERVERS", "ALL")- .env("VITE_LOCAL_BACKEND_ORIGIN", "same-origin");- if mode.spec().runs_local_infra {- cmd.env("VITE_AI_EDITING_WORKER_URL", "/ai-editing");- }- stage.run("Building frontend bundle", &mut cmd)?;- app_dir().join("dist")+/// where DEV is already true). Headless `stack up` sets the same origin env.+pub fn build_static(stage: &Stage, instance: &Instance, mode: Mode) -> Result<()> {+ let dist = {+ let mut cmd = Command::new("bun");+ cmd.current_dir(app_dir())+ .args(["run", "--bun", "build"])+ .env("MODE", "development")+ .env("NODE_ENV", "production")+ .env("VITE_LOCAL_SERVERS", "ALL")+ .env("VITE_LOCAL_BACKEND_ORIGIN", "same-origin");+ if mode.spec().runs_local_infra {+ cmd.env("VITE_AI_EDITING_WORKER_URL", "/ai-editing");}+ stage.run("Building frontend bundle", &mut cmd)?;+ app_dir().join("dist")};if stage.is_dry_run() {return Ok(());diff --git a/tooling/xtask/crates/xtask_local/src/local/fusionauth.rs b/tooling/xtask/crates/xtask_local/src/local/fusionauth.rsindex 43c78ee9c0..f276d3d10d 100644--- a/tooling/xtask/crates/xtask_local/src/local/fusionauth.rs+++ b/tooling/xtask/crates/xtask_local/src/local/fusionauth.rs@@ -66,9 +66,12 @@ pub fn wait_ready(stage: &Stage, instance: &Instance) -> Result<()> {instance.port(Port::FusionAuth),identity::APPLICATION_ID,);+ // Require an actual 200: `curl -f` only fails on 400+, so FusionAuth's+ // maintenance-mode 302 (e.g. after a boot-time DB connect failure) would+ // otherwise pass as ready and every later login would 500.let script = format!(- "for i in $(seq 1 120); do curl -fsS --max-time 3 -H 'Authorization: {key}' {url} >/dev/null 2>&1 && exit 0; sleep 2; done; \- echo 'timed out waiting for the FusionAuth kickstart'; exit 1",+ "for i in $(seq 1 120); do [ \"$(curl -sS -o /dev/null -w '%{{http_code}}' --max-time 3 -H 'Authorization: {key}' {url} 2>/dev/null)\" = 200 ] && exit 0; sleep 2; done; \+ echo 'timed out waiting for the FusionAuth kickstart (a 302 here means maintenance mode: FusionAuth could not reach its db)'; exit 1",key = identity::FUSIONAUTH_API_KEY,);let mut cmd = Command::new("bash");diff --git a/tooling/xtask/crates/xtask_local/src/local/mailpit.rs b/tooling/xtask/crates/xtask_local/src/local/mailpit.rsindex 24915e3349..c1ecd8aec7 100644--- a/tooling/xtask/crates/xtask_local/src/local/mailpit.rs+++ b/tooling/xtask/crates/xtask_local/src/local/mailpit.rs@@ -9,7 +9,7 @@ pub fn direct_ui_url(instance: &Instance) -> String {format!("http://localhost:{}", instance.port(Port::MailpitUi))}-/// The single-origin Mailpit route used by headless stacks and previews.+/// The single-origin Mailpit route used by headless stacks.pub fn proxy_ui_url(instance: &Instance) -> String {format!("{}/mailpit/", super::proxy::url(instance))}diff --git a/tooling/xtask/crates/xtask_local/src/local/proxy.rs b/tooling/xtask/crates/xtask_local/src/local/proxy.rsindex 3522116416..32df334812 100644--- a/tooling/xtask/crates/xtask_local/src/local/proxy.rs+++ b/tooling/xtask/crates/xtask_local/src/local/proxy.rs@@ -140,7 +140,7 @@ const SPECIAL_ROUTES: &str = r#" @websocket path /websocket /websocket/*"#;const MAILPIT_ROUTE: &str = r#" # Mailpit serves itself under /mailpit (MP_WEBROOT), so no prefix strip —- # this is how a headless/preview stack reads its passwordless login codes.+ # this is how a headless stack reads its passwordless login codes.handle /mailpit/* {reverse_proxy mailpit:8025}diff --git a/tooling/xtask/crates/xtask_local/src/local/sandbox_image.rs b/tooling/xtask/crates/xtask_local/src/local/sandbox_image.rsindex c8bc5edc82..71e1d51ee8 100644--- a/tooling/xtask/crates/xtask_local/src/local/sandbox_image.rs+++ b/tooling/xtask/crates/xtask_local/src/local/sandbox_image.rs@@ -6,10 +6,8 @@//! flake exposes `x86_64-linux` and `aarch64-linux`; Apple Silicon and ARM//! Linux therefore bake natively instead of qemu. BuildKit cache is the//! freshness check: unchanged `container/` is a no-op rebuild, a Dockerfile or-//! flake change pays the two nix shells. `--no-build` (Fly preview VM boot, CI-//! snapshot bake) skips that: the image is already on the daemon from preload /-//! the images lane, and the Fly VM's staged repo does not even include-//! `crates/agent_harness/container`.+//! flake change pays the two nix shells. `--no-build` (CI snapshot bake)+//! skips that: the image is already on the daemon from preload.use std::collections::BTreeMap;use std::path::Path;@@ -22,8 +20,7 @@ use super::stage::Stage;#[cfg(test)]mod test;-/// Local Docker tag `just run_local` / Fly previews load. Keep in sync with-/// `xtask_workflows::workflows::vars::AGENT_HARNESS_LOCAL_IMAGE`.+/// Local Docker tag `just run_local` loads.pub const DEFAULT_LOCAL_TAG: &str = "macro-agent-harness:latest";/// Build context matching `just -f crates/agent_harness/justfile build-local`.@@ -68,7 +65,7 @@ pub(crate) fn build_args(tag: &str, context: &Path) -> Vec<String> {/// `docker build` the sandbox image when local sandboxes are on.///-/// `no_build` skips the invocation so Fly / `--no-build` stack-up can use a+/// `no_build` skips the invocation so `--no-build` stack-up can use a/// preloaded tag. Dry-run notes the plan and does not invoke Docker.pub fn ensure(stage: &Stage, env: &BTreeMap<String, String>, no_build: bool) -> Result<()> {let Some(plan) = EnsurePlan::from_env(env) else {diff --git a/tooling/xtask/crates/xtask_local/src/local/sandbox_image/test.rs b/tooling/xtask/crates/xtask_local/src/local/sandbox_image/test.rsindex 40c08331b5..a46837305d 100644--- a/tooling/xtask/crates/xtask_local/src/local/sandbox_image/test.rs+++ b/tooling/xtask/crates/xtask_local/src/local/sandbox_image/test.rs@@ -57,7 +57,7 @@ fn docker_build_args_match_the_cli() {#[test]fn no_build_still_plans_the_tag() {- // `--no-build` is an ensure() argument (Fly VM / CI bake), not an env flag.+ // `--no-build` is an ensure() argument (CI bake / Cloud install), not an env flag.// The plan is unchanged so stack-up still documents which preloaded tag it// expects; docker is simply not invoked.assert_eq!(diff --git a/tooling/xtask/crates/xtask_local/src/local/snapshot.rs b/tooling/xtask/crates/xtask_local/src/local/snapshot.rsindex c25b362c46..1014c0aeed 100644--- a/tooling/xtask/crates/xtask_local/src/local/snapshot.rs+++ b/tooling/xtask/crates/xtask_local/src/local/snapshot.rs@@ -14,7 +14,7 @@//!//! Snapshots are tarballs of the Docker volumes, written by a throwaway helper//! container, stored under `infra/local/generated/.snapshots/<key>/` (override-//! with `MACRO_STACK_SNAPSHOT_DIR` — CI bakes this dir into preview images).+//! with `MACRO_STACK_SNAPSHOT_DIR` — CI and Cloud install can bake this dir).use std::path::PathBuf;use std::process::Command;@@ -61,8 +61,8 @@ fn archives(instance: &Instance) -> [(&'static str, String); 5] {/// archives are written so the files are consistent.const STATEFUL_SERVICES: &[&str] = &["postgres", "search", "kafka", "fusionauth", "db"];-/// Where snapshots live. `MACRO_STACK_SNAPSHOT_DIR` overrides for CI bakes and-/// preview images; the default sits inside the gitignored generated dir.+/// Where snapshots live. `MACRO_STACK_SNAPSHOT_DIR` overrides for CI and Cloud+/// install bakes; the default sits inside the gitignored generated dir.// xtask is host tooling, not a service reading APP_SECRETS_JSON, so reading the// process environment directly is correct here.#[allow(clippy::disallowed_methods)]@@ -104,7 +104,7 @@ impl Plan {// cross-arch restore a structural cache miss instead of a latent bug// if snapshots are ever shared between machines (e.g. an S3 cache).// Today each snapshot store is machine-local or CI-baked for- // same-arch Fly machines, so this is insurance, not a behavior change.+ // same-arch hosts, so this is insurance, not a behavior change.h.update(super::arch::detect()?.docker_platform.as_bytes());// Infra image pins + topology.diff --git a/tooling/xtask/crates/xtask_local/src/local/stack.rs b/tooling/xtask/crates/xtask_local/src/local/stack.rsindex a11791b3b5..f3fded1a9f 100644--- a/tooling/xtask/crates/xtask_local/src/local/stack.rs+++ b/tooling/xtask/crates/xtask_local/src/local/stack.rs@@ -1,19 +1,14 @@//! Headless stack orchestration: `cargo x stack up|update|status|down`.//!-//! The preview/agent/CI surface. Same bring-up as `run_local`, but no TTY-//! hotkey loop and no attached dev server: the frontend is a static bundle-//! served by the instance's Caddy, so a finished `up` leaves only Docker-//! containers running — nothing to babysit, and the whole product lives behind-//! the ONE proxy origin that an agent drives with a browser and a preview-//! deploy fronts with a real hostname.+//! Same bring-up as `run_local`, but no TTY hotkey loop and no attached dev+//! server. The frontend is a static bundle served by Caddy. A finished `up`+//! leaves only Docker containers running behind one proxy origin.//!-//! `up` is full-delete/full-create like `run_local` (unconditionally-//! idempotent); `update` is the `r`-hotkey as a one-shot verb (rebuild, restart-//! only what changed); `status` is machine-readable state; `down` reclaims-//! everything.+//! `up` is full-delete/full-create. `update` adopts a new binary directory+//! without touching volumes. If nothing is recorded yet, `update` bootstraps+//! through `up`. `status` is machine-readable state. `down` reclaims everything.-use std::io::{BufReader, Read};-use std::path::{Path, PathBuf};+use std::path::PathBuf;use std::process::Command;use anyhow::{Context, Result, bail};@@ -30,9 +25,6 @@ use super::{Mode, arch, env_layer, frontend, mailpit, proxy, sdk_webhook, snapshpub struct UpArgs {#[command(flatten)]pub run: RunArgs,- /// Stage this prebuilt frontend dist instead of building (CI artifact reuse).- #[arg(long)]- pub frontend_dist: Option<PathBuf>,/// Neither restore from nor save an init snapshot — always run the full/// migrate/kickstart/index init.#[arg(long)]@@ -49,15 +41,6 @@ pub struct UpArgs {pub json: bool,}-#[derive(Args, Clone, Default)]-pub struct SnapshotArgs {- #[command(flatten)]- pub instance: InstanceArgs,- /// Print machine-readable JSON (key, dir, present).- #[arg(long)]- pub json: bool,-}-#[derive(Args, Clone, Default)]pub struct UpdateArgs {#[command(flatten)]@@ -67,23 +50,17 @@ pub struct UpdateArgs {/// Also rebuild + restage the static frontend and recreate the proxy.#[arg(long)]pub frontend: bool,- /// Stage this prebuilt frontend dist instead of building (implies --frontend).+ /// Adopt this complete prebuilt binary set instead of invoking Cargo.#[arg(long)]- pub frontend_dist: Option<PathBuf>,- /// Apply this complete prebuilt binary set instead of invoking Cargo.- #[arg(long, conflicts_with = "build_aux_services")]pub binaries_dir: Option<PathBuf>,- /// Rebuild every repository-built local Docker service.+ /// Rebuild every repository-built local Docker service and recreate it.#[arg(long)]pub build_aux_services: bool,- /// Recreate registry-refreshed preview auxiliary services without building.- #[arg(long, hide = true, requires = "binaries_dir")]- pub recreate_aux_services: bool,/// Stream subprocess output and show per-step timings.#[arg(long, short)]pub verbose: bool,/// Print a machine-readable summary as the final line.- #[arg(long, hide = true, requires = "binaries_dir")]+ #[arg(long)]pub json: bool,}@@ -114,8 +91,7 @@ struct StackState {mode: String,/// `static` (Caddy serves the bundle) or `none` (`--no-frontend`).frontend: String,- /// Stable directory bind-mounted into every Rust service. Prebuilt updates- /// replace files here atomically so existing container mounts see them.+ /// Host directory currently bind-mounted at `/app/out`.#[serde(default)]binaries_dir: Option<PathBuf>,}@@ -197,18 +173,12 @@ pub fn up(mode: Mode, args: &UpArgs) -> Result<Instance> {// joined just before `bring_up_app` creates that container. (Dry run:// synchronously, so the command preview prints in order.)if static_frontend && stage.is_dry_run() {- frontend::build_static(&stage, &instance, mode, args.frontend_dist.as_deref())?;+ frontend::build_static(&stage, &instance, mode)?;}let fe_build = (static_frontend && !stage.is_dry_run()).then(|| {let instance = instance.clone();- let prebuilt = args.frontend_dist.clone();std::thread::spawn(move || {- frontend::build_static(- &Stage::from_env().quiet(),- &instance,- mode,- prebuilt.as_deref(),- )+ frontend::build_static(&Stage::from_env().quiet(), &instance, mode)})});@@ -272,9 +242,9 @@ pub fn up(mode: Mode, args: &UpArgs) -> Result<Instance> {.binaries_dir.clone().unwrap_or_else(|| super::workspace_root().join(target.debug_dir()));- let active_binaries = super::build::BinariesDir::classify(&configured_binaries)?- .host_dir()- .to_path_buf();+ let binaries = super::build::BinariesDir::classify(&configured_binaries)?;+ binaries.pin_gc_root(&instance.artifact_dir())?;+ let active_binaries = binaries.host_dir().to_path_buf();super::bring_up_app(&stage, mode, &instance, &env)?;let _sdk_webhook_tunnel = (mode == Mode::Local && !stage.is_dry_run()).then(|| sdk_webhook::start(&instance))@@ -295,6 +265,7 @@ pub fn up(mode: Mode, args: &UpArgs) -> Result<Instance> {binaries_dir: Some(active_binaries),},)?;+ super::build::BinariesDir::release_previous_gc_root(&instance.artifact_dir());}let frontend_url = if static_frontend {@@ -321,18 +292,41 @@ pub fn up(mode: Mode, args: &UpArgs) -> Result<Instance> {Ok(instance)}-/// `cargo x stack update` — the `r` hotkey as a one-shot verb: rebuild the-/// binaries (and optionally the frontend bundle) and restart only what changed.+/// `cargo x stack update` — adopt a new build into the running stack.+/// Volumes stay. With no recorded stack, this bootstraps through [`up`].pub fn update(args: &UpdateArgs) -> Result<()> {- let stage = Stage::from_env_cli(args.verbose);let instance = Instance::derive(args.instance.instance.as_deref(), args.instance.port_base)?;- let Some(state) = read_state(&instance) else {- bail!(- "no stack state at {} — bring the stack up first (`just stack up{}`)",- state_path(&instance).display(),- instance_suffix(&instance)- );+ if read_state(&instance).is_none() {+ return bootstrap_from_update(args);+ }+ update_running(args)+}++fn bootstrap_from_update(args: &UpdateArgs) -> Result<()> {+ let up_args = UpArgs {+ run: super::cli::RunArgs {+ instance: args.instance.clone(),+ env: args.env.clone(),+ build: super::cli::BuildArgs {+ no_build: args.binaries_dir.is_some(),+ build_aux_services: args.build_aux_services,+ binaries_dir: args.binaries_dir.clone(),+ },+ no_frontend: false,+ verbose: args.verbose,+ traces: None,+ },+ no_snapshot: false,+ infra_only: false,+ json: args.json,};+ up(Mode::Local, &up_args).map(|_| ())+}++fn update_running(args: &UpdateArgs) -> Result<()> {+ let stage = Stage::from_env_cli(args.verbose);+ let instance = Instance::derive(args.instance.instance.as_deref(), args.instance.port_base)?;+ let state = read_state(&instance).expect("update_running requires stack state");let mode = mode_from_label(&state.mode)?;stage.section(&format!("macro {} stack update — instance {}",@@ -347,15 +341,20 @@ pub fn update(args: &UpdateArgs) -> Result<()> {args.env.env_file.as_deref(),state.frontend == "static",)?;- let changed = if let Some(source) = args.binaries_dir.as_deref() {- let changed = apply_prebuilt_binaries(mode, &state, source)?;- if args.recreate_aux_services {- super::recreate_preview_aux_service_containers(&stage, &instance, &env)?;- }- if !changed.is_empty() {- super::reload_services(&stage, &instance, &changed)?;+ let remounted = if let Some(source) = args.binaries_dir.as_deref() {+ let new = super::build::BinariesDir::classify(source)?;+ new.validate(&super::inventory::local_binaries())?;+ new.pin_gc_root(&instance.artifact_dir())?;+ match new.adoption_from_recorded(state.binaries_dir.as_deref()) {+ super::build::Adoption::Unchanged => {+ stage.note("binaries unchanged — mounts left as-is");+ false+ }+ super::build::Adoption::Remount => {+ remount(&stage, mode, &instance, &env, &new, &state)?;+ true+ }}- changed} else {let target = arch::detect()?;super::rebuild_and_reload(@@ -366,18 +365,16 @@ pub fn update(args: &UpdateArgs) -> Result<()> {target,args.build_aux_services,)?;- Vec::new()+ false};- if args.frontend || args.frontend_dist.is_some() {- reload_static_frontend(- &stage,- &instance,- mode,- &env,- &state,- args.frontend_dist.as_deref(),- )?;+ if args.binaries_dir.is_some() && args.build_aux_services {+ super::build_aux_service_images(&stage, &instance, &env)?;+ super::recreate_aux_service_containers(&stage, &instance, &env)?;+ }++ if args.frontend {+ reload_static_frontend(&stage, &instance, mode, &env, &state)?;}if mode.spec().wait_backend_before_frontend {frontend::wait_backend_ready(&stage, &instance)?;@@ -386,35 +383,59 @@ pub fn update(args: &UpdateArgs) -> Result<()> {println!("{}",serde_json::to_string(&json!({- "changed_services": changed.iter().map(|s| s.compose_name).collect::<Vec<_>>(),- "frontend_updated": args.frontend || args.frontend_dist.is_some(),- "aux_services_recreated": args.recreate_aux_services,+ "remounted": remounted,+ "frontend_updated": args.frontend,+ "aux_services_rebuilt": args.build_aux_services,}))?);}Ok(())}-fn apply_prebuilt_binaries(+fn remount(+ stage: &Stage,mode: Mode,+ instance: &Instance,+ env: &env_layer::ResolvedEnv,+ new: &super::build::BinariesDir,state: &StackState,- binaries_dir: &Path,-) -> Result<Vec<&'static super::inventory::RustService>> {- let destination = state.binaries_dir.as_ref().ok_or_else(|| {- anyhow::anyhow!(- "stack state predates prebuilt updates — perform a full stack up before updating"- )- })?;- let source = super::build::BinariesDir::classify(binaries_dir)?;- source.validate(&super::inventory::local_binaries())?;- let mut changed = Vec::new();- for service in super::inventory::services_for_mode(mode) {- if replace_binary_if_changed(source.host_dir(), destination, service.cargo_bin)? {- changed.push(service);- }+) -> Result<()> {+ let gmail_forwarder = env+ .merged+ .get("GMAIL_FORWARDER_SA_KEY")+ .is_some_and(|key| !key.trim().is_empty());+ super::gen_compose::generate(+ mode,+ instance,+ new,+ state.frontend == "static",+ gmail_forwarder,+ )?;+ let mut up = super::compose_cmd(instance, env);+ up.args([+ "up",+ "-d",+ "--force-recreate",+ "--no-deps",+ "--remove-orphans",+ ]);+ for svc in super::inventory::services_for_mode(mode) {+ up.arg(svc.compose_name);}-- Ok(changed)+ if gmail_forwarder {+ up.arg("gmail_forwarder");+ }+ stage.run("Remounting Rust services", &mut up)?;+ write_state(+ instance,+ &StackState {+ mode: state.mode.clone(),+ frontend: state.frontend.clone(),+ binaries_dir: Some(new.host_dir().to_path_buf()),+ },+ )?;+ super::build::BinariesDir::release_previous_gc_root(&instance.artifact_dir());+ Ok(())}fn reload_static_frontend(@@ -423,7 +444,6 @@ fn reload_static_frontend(mode: Mode,env: &env_layer::ResolvedEnv,state: &StackState,- frontend_dist: Option<&Path>,) -> Result<()> {if state.frontend != "static" {bail!(@@ -431,7 +451,7 @@ fn reload_static_frontend(re-run `just stack up` to serve one");}- frontend::build_static(stage, instance, mode, frontend_dist)?;+ frontend::build_static(stage, instance, mode)?;// build_static replaces the staged directory, so the container must be// recreated to establish a bind mount to the new inode.let mut up = super::compose_cmd(instance, env);@@ -439,66 +459,6 @@ fn reload_static_frontend(stage.run("Recreating proxy (frontend bundle)", &mut up)}-fn replace_binary_if_changed(- source_dir: &Path,- destination_dir: &Path,- name: &str,-) -> Result<bool> {- let source = source_dir.join(name);- let destination = destination_dir.join(name);- if files_equal(&source, &destination)? {- return Ok(false);- }- std::fs::create_dir_all(destination_dir)- .with_context(|| format!("creating {}", destination_dir.display()))?;- let temporary = destination_dir.join(format!(".{name}.hot-update-{}", std::process::id()));- let _ = std::fs::remove_file(&temporary);- std::fs::copy(&source, &temporary).with_context(|| {- format!(- "copying prebuilt binary {} to {}",- source.display(),- temporary.display()- )- })?;- std::fs::rename(&temporary, &destination).with_context(|| {- format!(- "atomically replacing {} with {}",- destination.display(),- source.display()- )- })?;- Ok(true)-}--fn files_equal(left: &Path, right: &Path) -> Result<bool> {- let left_meta = std::fs::metadata(left)- .with_context(|| format!("reading binary metadata for {}", left.display()))?;- let Ok(right_meta) = std::fs::metadata(right) else {- return Ok(false);- };- if left_meta.len() != right_meta.len() {- return Ok(false);- }- let mut left = BufReader::new(- std::fs::File::open(left).with_context(|| format!("opening {}", left.display()))?,- );- let mut right = BufReader::new(- std::fs::File::open(right).with_context(|| format!("opening {}", right.display()))?,- );- let mut left_buf = [0_u8; 64 * 1024];- let mut right_buf = [0_u8; 64 * 1024];- loop {- let left_read = left.read(&mut left_buf)?;- let right_read = right.read(&mut right_buf)?;- if left_read != right_read || left_buf[..left_read] != right_buf[..right_read] {- return Ok(false);- }- if left_read == 0 {- return Ok(true);- }- }-}-/// `cargo x stack status` — container states + health through the proxy, as a/// human summary or `--json` for machines.pub fn status(args: &StatusArgs) -> Result<()> {@@ -571,44 +531,6 @@ pub fn down(args: &DownArgs) -> Result<()> {Ok(())}-/// `cargo x stack snapshot` — report the instance's init-snapshot key and-/// whether a snapshot exists for it. CI uses `--json` to find the directory to-/// bake into preview images.-pub fn snapshot_status(args: &SnapshotArgs) -> Result<()> {- let instance = Instance::derive(args.instance.instance.as_deref(), args.instance.port_base)?;- // The key hashes the generated kickstart; (re)write it so the verb works- // before any `up` has run. The kickstart depends on the resolved env (the- // optional Google IdP client), so resolve it the same way `up` does —- // otherwise the key reported here could differ from the key `up` computes.- let env = env_layer::resolve(Mode::Local, &instance, false, None, true)?;- let google = super::kickstart::GoogleIdp::from_env(&env.merged);- super::fusionauth::write_kickstart(&instance, google.as_ref())?;- let plan = snapshot::Plan::compute(&instance)?;- if args.json {- println!(- "{}",- serde_json::to_string(&json!({- "key": plan.key,- "dir": plan.dir,- "present": plan.exists(),- "root": snapshot::root_dir(),- }))?- );- return Ok(());- }- println!("key {}", plan.key);- println!("dir {}", plan.dir.display());- println!(- "present {}",- if plan.exists() {- "yes"- } else {- "no (next cold `stack up` will save it)"- }- );- Ok(())-}-/// The machine-readable endpoint block `up --json` and `status --json` share.fn summary_json(mode: Mode, instance: &Instance, static_frontend: bool) -> serde_json::Value {let mailpit_url = mode.spec().runs_local_infra.then(|| {diff --git a/tooling/xtask/crates/xtask_local/src/local/stack/test.rs b/tooling/xtask/crates/xtask_local/src/local/stack/test.rsindex 0d948adee0..c3f9f0e114 100644--- a/tooling/xtask/crates/xtask_local/src/local/stack/test.rs+++ b/tooling/xtask/crates/xtask_local/src/local/stack/test.rs@@ -44,18 +44,3 @@ fn clearing_state_invalidates_a_previous_headless_stack() {clear_state(&instance).unwrap();let _ = std::fs::remove_dir_all(instance.artifact_dir());}--#[test]-fn compares_binary_contents_not_just_size() {- let root = std::env::temp_dir().join(format!("macro-stack-test-{}", std::process::id()));- let _ = std::fs::remove_dir_all(&root);- std::fs::create_dir_all(&root).unwrap();- let left = root.join("left");- let right = root.join("right");- std::fs::write(&left, b"same").unwrap();- std::fs::write(&right, b"same").unwrap();- assert!(files_equal(&left, &right).unwrap());- std::fs::write(&right, b"diff").unwrap();- assert!(!files_equal(&left, &right).unwrap());- std::fs::remove_dir_all(root).unwrap();-}diff --git a/tooling/xtask/crates/xtask_paths/src/lib.rs b/tooling/xtask/crates/xtask_paths/src/lib.rsindex 70139a0005..7a8e254dd8 100644--- a/tooling/xtask/crates/xtask_paths/src/lib.rs+++ b/tooling/xtask/crates/xtask_paths/src/lib.rs@@ -56,7 +56,7 @@ pub fn repo_root() -> PathBuf {/// Read the runtime repository root used by packaged xtask binaries.////// Environment variables are appropriate here because this tooling crate runs-/// both inside and outside the service runtime, including in the preview VM.+/// both inside and outside the service runtime, including packaged binaries.#[allow(clippy::disallowed_methods)]fn repo_root_override() -> Option<PathBuf> {std::env::var_os("MACRO_REPO_ROOT").map(PathBuf::from)diff --git a/tooling/xtask/crates/xtask_workflows/src/workflows.rs b/tooling/xtask/crates/xtask_workflows/src/workflows.rsindex 45c69eac34..d94c071d61 100644--- a/tooling/xtask/crates/xtask_workflows/src/workflows.rs+++ b/tooling/xtask/crates/xtask_workflows/src/workflows.rs@@ -36,7 +36,6 @@ mod deploy_web_app;mod docs_check;mod ensure_daytona_snapshot;mod path_validation;-mod preview_fly;mod pulumi_preview_pr;mod push_local_stack_binaries;mod reusable_deploy_service;@@ -279,16 +278,6 @@ const WORKFLOWS: &[WorkflowFile] = &[)},},- WorkflowFile {- slug: "preview_fly",- file_name: "preview-fly.yml",- render_yaml: || render_gh_workflow(preview_fly::preview_fly)(),- },- WorkflowFile {- slug: "preview_fly",- file_name: "preview-fly-cleanup.yml",- render_yaml: || render_gh_workflow(preview_fly::preview_fly_cleanup)(),- },WorkflowFile {slug: "check_node_modules_nix",file_name: "check_node_modules_nix.yml",diff --git a/tooling/xtask/crates/xtask_workflows/src/workflows/ensure_daytona_snapshot.rs b/tooling/xtask/crates/xtask_workflows/src/workflows/ensure_daytona_snapshot.rsindex 953f468f02..93d35d82f1 100644--- a/tooling/xtask/crates/xtask_workflows/src/workflows/ensure_daytona_snapshot.rs+++ b/tooling/xtask/crates/xtask_workflows/src/workflows/ensure_daytona_snapshot.rs@@ -2,7 +2,7 @@//! it is missing, and publishes the same image to GHCR. Generated into//! `ensure_daytona_snapshot.yml`.//!-//! Local stacks and Fly previews `docker build` this image themselves (BuildKit+//! Local stacks `docker build` this image themselves (BuildKit//! cache is the freshness check). GHCR is the published copy for `main`//! (`:latest`) and for verifying the image on PRs (`:$SHA` only).@@ -96,7 +96,7 @@ fn ensure_snapshot() -> Job {/// GHCR publish. Runs on Mid with Namespace remote buildx: the Dockerfile/// bakes two nix shells, which is too large for the Small Daytona job's local-/// daemon. linux/amd64 only — Daytona snapshots and Fly preview machines are+/// daemon. linux/amd64 only — Daytona snapshots are/// amd64. Local `docker build` is unpinned so Apple Silicon / ARM Linux bake/// native `aarch64-linux` from the same flake. PRs push `:$SHA` only so they/// cannot clobber `:latest`.diff --git a/tooling/xtask/crates/xtask_workflows/src/workflows/preview_fly.rs b/tooling/xtask/crates/xtask_workflows/src/workflows/preview_fly.rsdeleted file mode 100644index bdd408fe3e..0000000000--- a/tooling/xtask/crates/xtask_workflows/src/workflows/preview_fly.rs+++ /dev/null@@ -1,946 +0,0 @@-//! `Fly Preview` — full-stack preview deploys on Fly Machines, plus the-//! cleanup workflow that destroys the app when the PR closes.-//!-//! Opt-in via the `preview` label. The deploy job builds every artifact on the-//! runner (service binaries, same-origin frontend bundle, init snapshot),-//! mirrors every stack image into the per-PR app's Fly registry repo, stages-//! the rest into a slim VM image (`infra/preview/`), and deploys it as the-//! per-PR Fly app `macro-pr-<N>` — which suspends when idle and wakes on-//! request. The machine pulls the stack images at boot (layer-level dedup-//! against its persistent /var/lib/docker volume) instead of shipping them-//! inside the VM image. A brand-new PR's volume is forked from the newest-//! template volume on `macro-preview-template` (a warm layer store published-//! by a previous successful deploy, content-addressed by the image-//! manifest), so first boots pull only deltas instead of ~6GB cold. Runtime-//! secrets reach the machine as a config-scoped Doppler service token-//! (`DOPPLER_PREVIEW_TOKEN` → Fly secret `DOPPLER_TOKEN`), which the-//! stack's env layer pulls at boot.-//!-//! NOTE: never a required status check — previews are opt-in by design.--use gh_workflow::{- Concurrency, Event, Expression, Job, Level, Permissions, PullRequest, PullRequestType, Run,- Step, Use, Workflow,-};--use crate::workflows::{runners, steps, vars};--#[cfg(test)]-mod test;--/// The per-PR Fly app name; also the preview hostname (`<app>.fly.dev`).-const APP_NAME: &str = "macro-pr-${{ github.event.pull_request.number }}";--/// The app holding template volumes: warm /var/lib/docker layer stores that-/// new PR apps fork instead of pulling ~6GB of images onto an empty volume.-/// Machine-less — it exists purely as a volume namespace. Templates are-/// published by successful deploys and content-addressed by the preload-/// manifest, so nothing runs when the image set hasn't changed.-const TEMPLATE_APP: &str = "macro-preview-template";--/// Build the deploy workflow.-pub fn preview_fly() -> Workflow {- Workflow::new("Fly Preview")- .on(Event::default().pull_request(- PullRequest::default()- .add_type(PullRequestType::Opened)- .add_type(PullRequestType::Reopened)- .add_type(PullRequestType::Synchronize)- .add_type(PullRequestType::Labeled),- ))- .concurrency(- Concurrency::new(Expression::new(- "fly-preview-${{ github.event.pull_request.number }}",- ))- .cancel_in_progress(true),- )- .add_job("deploy", deploy())-}--/// Build the cleanup workflow (destroys the app on close / label removal).-pub fn preview_fly_cleanup() -> Workflow {- Workflow::new("Fly Preview Cleanup")- .on(Event::default().pull_request(- PullRequest::default()- .add_type(PullRequestType::Closed)- .add_type(PullRequestType::Unlabeled),- ))- .add_job("destroy", destroy())-}--fn deploy() -> Job {- Job::default()- // Same-repo branches only: the preview image embeds CI-built artifacts- // and the app serves under our Fly org — never build it from fork code.- .cond(Expression::new(- "github.event.pull_request.head.repo.full_name == github.repository && \- contains(github.event.pull_request.labels.*.name, 'preview')",- ))- // linux-mid for the extra cores: five build lanes (zigbuild, compose- // image builds, frontend, snapshot bake) genuinely contend on the- // 8-core rust-ci shape. The cache volume follows the explicit tag,- // not the profile, so this keeps the dedicated preview pool (nix- // store + cargo target + snapshots) it has always used: see- // PREVIEW_CACHE_TAG for why sharing the check/test jobs' pool was- // measured cold on both layers (different --target = disjoint- // sccache keys, and their volumes never carry a cargo target dir).- .runs_on(runners::Runner::Mid.with_cache_tag(vars::PREVIEW_CACHE_TAG))- // Backstop against hangs: worst honest case is ~15 min cold builds +- // a cold `crates/agent_harness/container` bake (two nix shells) +- // ~6 min cold mirror push + the 30 min flyctl wait cap. 90m leaves- // room for that extra image; anything past that is a stuck step (a- // hung log fetch once burned 2h26m of runner before someone cancelled- // it by hand).- .timeout_minutes(90u32)- .permissions(- Permissions::default()- .contents(Level::Read)- .pull_requests(Level::Write),- )- .add_env(("FLY_API_TOKEN", vars::FLY_API_TOKEN))- .add_env(("APP_NAME", APP_NAME))- .add_env(("TEMPLATE_APP", TEMPLATE_APP))- // Init snapshots live on the cache volume: an unchanged key skips the- // whole infra bake and keeps snapshot bytes (→ VM image layer) stable.- .add_env((- "MACRO_STACK_SNAPSHOT_DIR",- vars::PREVIEW_SNAPSHOT_VOLUME_DIR,- ))- // No incremental in CI: incremental caches are what bloated the cache- // volume to 41G (making it the workspace's eviction victim — observed- // volume loss between runs), and sccache refuses to cache incremental- // compiles, so they also locked workspace crates out of sccache.- // Cargo's fingerprint reuse (what makes warm runs fast) is unaffected.- // Local run_local keeps incremental for the edit-rebuild loop.- .add_env(("CARGO_INCREMENTAL", "0"))- .add_step(steps::checkout(false, true))- .add_step(steps::mount_cache_volume_with_cargo_target())- // CARGO_INCREMENTAL=0 stops incremental caches being written, but the- // ~25G already on older volumes never gets cleaned by cargo — it's- // exactly the weight that made this pool the workspace quota's- // eviction victim. Deleting is idempotent and instant when clean.- .add_step(- Step::new("Prune stale incremental caches")- .run("rm -rf target/*/debug/incremental target/debug/incremental"),- )- // Namespace remote builder: persistent BuildKit layer cache across- // runs, same as the deploy workflows use. The aux-image builds and the- // preview-image build both go through it.- .add_step(steps::setup_namespace_buildx())- .add_step(steps::setup_nix())- .add_step(steps::setup_reqs_web("Setup dev shell + web deps", false))- .add_step(steps::configure_namespace_sccache(- vars::PREVIEW_SCCACHE_NAME,- ))- .add_step(setup_flyctl())- .add_step(build_artifacts())- .add_step(dump_stack_diagnostics())- .add_step(stage_context())- .add_step(deploy_to_fly())- .add_step(dump_fly_diagnostics())- .add_step(dump_boot_timings())- .add_step(publish_template_volume())- .add_step(comment_preview_url())- .add_step(steps::teardown_nix())-}--/// When the machine never passes its health check, the answer is inside the-/// VM (dockerd, image load, `stack up`) — dump its logs and state so a failed-/// deploy is diagnosable from CI output alone.-fn dump_fly_diagnostics() -> Step<Run> {- Step::new("Dump Fly diagnostics")- .run(indoc::indoc! {r#"- flyctl machine list --app "$APP_NAME" || true- # `flyctl logs --no-tail` can hang forever on its NATS fetch- # (observed: a healthy deploy stuck 1h53m in the timings dump) —- # never call it without a timeout.- timeout 60 flyctl logs --app "$APP_NAME" --no-tail || true- "#})- .if_condition(Expression::new("failure()"))-}--/// The entrypoint tees every boot-phase timing (image pulls, per-stage stack-/// up, auth startup) into a volume-backed file precisely so CI can read it-/// deterministically after a healthy deploy. `flyctl logs --no-tail` used to-/// serve this and both hung (unbounded NATS fetch) and silently lost the-/// early boot lines to Fly's short log retention. On a hot update the machine-/// never reboots, so the file describes the last real boot — the hot path's-/// own output already streams into the deploy step via `ssh console`.-fn dump_boot_timings() -> Step<Run> {- Step::new("Dump boot timings").run(indoc::indoc! {r#"- timeout 120 flyctl ssh console --app "$APP_NAME" --quiet \- --command 'cat /var/lib/docker/.macro-preview/boot-timings.log' || true- "#})-}--/// Every build artifact in one step, five lanes in parallel. The lanes are-/// independent by construction:-///-/// - `fly`: app create, template-volume fork kick, and pull-token mint —-/// nothing here needs build output, and the fork's ~2 min server-side-/// hydration must start within seconds to finish inside the compose-/// build instead of gating machine creation.-/// - `cargo`: service binaries then xtask_local, sequential because they-/// share the workspace target dir (cargo's build-dir lock would serialize-/// them anyway). xtask_local deliberately WITHOUT the `local-stack`-/// feature (the VM restores the baked snapshot so it never provisions-/// Kafka; skipping rdkafka means no dynamically-linked librdkafka in the-/// VM) and via zigbuild, not a host build (the nix dev shell links the-/// /nix/store ELF interpreter, which doesn't exist in the VM image).-/// - `web`: the same-origin frontend bundle.-/// - `images`: the compose-built app images (the most fragile part of a-/// fresh bring-up — a broken Dockerfile still fails the step early via-/// fail-fast wait), then a best-effort premirror push to the app's Fly-/// registry so the network transfer overlaps the cargo lane instead of-/// serializing after it. The deploy step's mirror loop stays-/// authoritative: its manifest-existence check skips whatever was-/// premirrored here and pushes the rest (e.g. the bake-built runtime-/// image).-/// - `bake`: the init snapshot (see the lane script header for details),-/// under its own CARGO_TARGET_DIR so its host xtask build doesn't take-/// the workspace build-dir lock the cargo lane holds.-///-/// Lanes log to files and are dumped as collapsible groups afterwards;-/// `wait -n` fails the step on the first lane failure and kills the rest.-fn build_artifacts() -> Step<Run> {- Step::new("Build artifacts (parallel lanes)")- .run(indoc::indoc! {r#"- set -euo pipefail- lanes="$RUNNER_TEMP/lanes"- mkdir -p "$lanes"-- cat > "$lanes/cargo.sh" <<'LANE'- set -euo pipefail- cargo x zigbuild- sccache --show-stats- cargo zigbuild --release --target x86_64-unknown-linux-gnu.2.36 -p xtask_local- LANE-- cat > "$lanes/web.sh" <<'LANE'- set -euo pipefail- cd apps/web- export MODE=development NODE_ENV=production- export VITE_LOCAL_SERVERS=ALL VITE_LOCAL_BACKEND_ORIGIN=same-origin- # Same ceiling as `just -f apps/web/justfile build-dev`: vite's- # chunk-render peaks at node's default ~4GB heap and aborts with- # "Ineffective mark-compacts near heap limit".- export NODE_OPTIONS=--max-old-space-size=6144- bun run --bun build- LANE-- # Instant lane: everything Fly-side that needs no build output.- # The template fork must fire within seconds of the job starting —- # its ~2 min server-side hydration then finishes inside the compose- # build. (It previously sat after the premirror pushes, which put- # it at the END of the build phase and re-gated the deploy.)- # Entirely best-effort: everything in the fly lane (app create,- # fork kick, token mint) has an authoritative fallback in the- # deploy step, so a transient Fly API failure must not fail the- # build. A child `bash -e` (not a `|| `-guarded function: errexit- # is suppressed inside those) keeps its internal fail-fast so a- # failed app_id fetch can't mint a garbage token.- cat > "$lanes/fly.sh" <<'LANE'- set -euo pipefail- bash -euo pipefail "$(dirname "$0")/fly-prep.sh" \- || echo "fly prep failed (non-fatal): the deploy step falls back" >&2- LANE-- cat > "$lanes/fly-prep.sh" <<'LANE'- flyctl status --app "$APP_NAME" >/dev/null 2>&1 \- || flyctl apps create "$APP_NAME" --org "$FLY_ORG" \- || flyctl status --app "$APP_NAME" >/dev/null- # Kick the template-volume fork (see the deploy step's- # volume-state block, which stays the authoritative fallback).- if ! flyctl volumes list --app "$APP_NAME" --json 2>/dev/null \- | jq -e 'map(select(.name == "docker_data")) | length > 0' >/dev/null; then- tpl_src=$(flyctl volumes list --app "$TEMPLATE_APP" --json 2>/dev/null \- | jq -r '[.[] | select((.name | startswith("tpl")) and .state == "created" and .attached_machine_id == null)]- | sort_by(.created_at) | last | .id // empty' 2>/dev/null) || tpl_src=- if [ -n "$tpl_src" ]; then- curl -sf -X POST "https://api.machines.dev/v1/apps/$APP_NAME/volumes" \- -H "Authorization: Bearer $FLY_API_TOKEN" \- -H "Content-Type: application/json" \- -d "{\"name\":\"docker_data\",\"region\":\"ewr\",\"source_volume_id\":\"$tpl_src\"}" \- | jq -r '"early template fork: \(.id // "failed") (\(.state // "-"))"' \- || echo "early template fork failed; deploy step will fall back" >&2- fi- fi- # Mint the machine's app-scoped read-only registry pull token- # (macaroon attenuation, see the deploy step for the why) so the- # deploy preamble doesn't pay the round-trips serially.- app_id=$(curl -sf https://api.fly.io/graphql \- -H "Authorization: Bearer $FLY_API_TOKEN" \- -H "content-type: application/json" \- -d "{\"query\":\"{ app(name: \\\"$APP_NAME\\\") { internalNumericId } }\"}" \- | jq -re '.data.app.internalNumericId')- now=$(date +%s)- printf '[{"type":"Apps","body":{"apps":{"%s":"r"}}},{"type":"ValidityWindow","body":{"not_before":%d,"not_after":%d}}]' \- "$app_id" $((now - 60)) $((now + 604800)) \- | flyctl tokens attenuate > "$RUNNER_TEMP/registry-pull-token.next"- chmod 600 "$RUNNER_TEMP/registry-pull-token.next"- mv "$RUNNER_TEMP/registry-pull-token.next" "$RUNNER_TEMP/registry-pull-token"- LANE-- cat > "$lanes/images.sh" <<'LANE'- set -euo pipefail- docker compose --project-directory . -p macro -f docker/docker-compose.yml build \- search sync_service websocket_service lexical_service ai_editing_worker analytics_proxy- # Not a compose service, so `config --images` never lists it.- # BuildKit cache on this runner is the freshness check.- docker build --tag "$LOCAL_SANDBOX_IMAGE" crates/agent_harness/container- # The image BUILD above is fatal; the premirror below is a pure- # optimization with an authoritative fallback (the deploy step's- # mirror loop), so its failure only costs a slower push later.- bash -euo pipefail "$(dirname "$0")/premirror.sh" \- || echo "premirror failed (non-fatal): the deploy step mirrors authoritatively" >&2- LANE-- # Premirror: content-addressed refs, identical scheme to the deploy- # step's mirror loop, so its manifest-existence check skips these.- # Best effort by design — anything skipped here (images only the- # bake produces, like macro-local-runtime:dev) is pushed there.- # The fly lane created the app long ago; the guard is a fallback.- cat > "$lanes/premirror.sh" <<'LANE'- flyctl status --app "$APP_NAME" >/dev/null 2>&1 \- || flyctl apps create "$APP_NAME" --org "$FLY_ORG" \- || flyctl status --app "$APP_NAME" >/dev/null- flyctl auth docker- registry="registry.fly.io/$APP_NAME"- for img in $(docker compose --project-directory . -p macro \- -f docker/docker-compose.yml config --images | sort -u) alpine:3 "$LOCAL_SANDBOX_IMAGE"; do- if ! docker image inspect "$img" >/dev/null 2>&1 && ! docker pull "$img"; then- echo "premirror: skipping $img (not local, not pullable)"- continue- fi- id=$(docker image inspect -f '{{.Id}}' "$img")- ref="$registry:img-$(echo "$img" | tr '/:' '__')-$(echo "$id" | cut -c8-19)"- docker tag "$img" "$ref"- if ! docker manifest inspect "$ref" >/dev/null 2>&1; then- for _ in 1 2 3; do- docker push "$ref" && break- echo "premirror push of $ref failed, retrying" >&2- sleep 15- done- fi- done- LANE-- # A cold `stack up --infra-only` runs the real init (migrate,- # kickstart, Kafka topics, indices) and saves the content-addressed- # snapshot the VM restores from. Infra only: the app services need- # the Doppler-sourced env to boot, which this runner deliberately- # lacks — the snapshot captures only the infra volumes anyway. Via- # `just` (not the bare `cargo x` alias) because the justfile- # enables the `local-stack` feature the Kafka provisioning needs.- # The store lives on the cache volume for a zero-copy hit; every- # snapshot is also stored as one tar in Namespace artifact storage- # keyed by the same hash, so cache-volume onboarding misses- # download the exact snapshot instead of paying for a full init.- # Artifact access is an optimization: failures fall back to the- # volume/cold-bake behavior. The final `snapshot --json` line lands- # in $RUNNER_TEMP for the stage step.- cat > "$lanes/bake.sh" <<'LANE'- set -euo pipefail- export CARGO_TARGET_DIR="$PWD/target/bake"- mkdir -p "$MACRO_STACK_SNAPSHOT_DIR"- status=$(just stack snapshot --json | tail -n 1)- key=$(echo "$status" | jq -r '.key')- root=$(echo "$status" | jq -r '.root')- artifact="macro-preview/init-snapshots/${key}.tar"- archive="$RUNNER_TEMP/init-snapshot-${key}.tar"- artifact_hit=-- # The Namespace cache-volume pool has an onboarding period where a- # runner can receive an empty fork. Artifact storage is the durable- # fallback: restore into a temporary directory and only publish it to- # the live store after validating the embedded manifest through xtask.- if ! echo "$status" | jq -e '.present' >/dev/null; then- restored="$RUNNER_TEMP/restored-init-snapshot"- rm -rf "$restored"- mkdir -p "$restored"- if nsc artifact download "$artifact" "$archive" \- && tar -xf "$archive" -C "$restored" \- && [ -f "$restored/$key/manifest.json" ]; then- rm -rf "$root/$key"- cp -a "$restored/$key" "$root/$key"- status=$(just stack snapshot --json | tail -n 1)- if echo "$status" | jq -e '.present' >/dev/null; then- artifact_hit=1- echo "init snapshot artifact hit: $status"- else- rm -rf "$root/$key"- echo "downloaded init snapshot failed validation; baking cold" >&2- fi- else- echo "init snapshot artifact miss; baking cold" >&2- fi- fi-- if echo "$status" | jq -e '.present' >/dev/null; then- echo "init snapshot cache hit: $status"- cargo run --quiet --manifest-path Cargo.toml \- -p xtask_local --features local-stack -- gen-compose- # The cold path builds the runtime image inside `stack up`; on a- # hit nothing else does, and the registry mirror needs it in the- # local daemon (its pull fallback only covers public images —- # macro-local-runtime:dev is not on Docker Hub).- cargo run --quiet --manifest-path Cargo.toml \- -p xtask_local --features local-stack -- runtime-image- else- just stack up --infra-only --no-doppler --no-build- status=$(just stack snapshot --json | tail -n 1)- echo "$status" | jq -e '.present' >/dev/null- fi-- # Seed artifact storage from either a volume hit or a cold bake. Avoid- # uploading another version when the artifact already exists; a race- # between concurrent PRs is harmless and must not fail the preview.- if [ -z "$artifact_hit" ] && ! nsc artifact describe "$artifact" >/dev/null 2>&1; then- snap_dir=$(echo "$status" | jq -r '.dir')- tar -cf "$archive" -C "$root" "$(basename "$snap_dir")"- nsc artifact upload "$archive" "$artifact" --expires_in 720h \- || echo "warning: failed to upload init snapshot artifact" >&2- fi- echo "$status" > "$RUNNER_TEMP/preview-snapshot.json"- LANE-- declare -a pids=() names=()- for name in fly cargo web images bake; do- # setsid: each lane leads its own process group so fail-fast- # can kill the whole tree (cargo/docker/bun children), not just- # the wrapper shell.- setsid bash "$lanes/$name.sh" > "$lanes/$name.log" 2>&1 &- pids+=($!)- names+=("$name")- done- fail=- for _ in "${pids[@]}"; do- # Bare `wait -n`: any child; explicit pids would error once a- # job has already been reaped.- if ! wait -n; then- fail=1- break- fi- done- if [ -n "$fail" ]; then- # Negative pids: signal each lane's whole process group.- kill -- "${pids[@]/#/-}" 2>/dev/null || true- wait || true- fi- for i in "${!names[@]}"; do- echo "::group::lane: ${names[$i]}"- cat "$lanes/${names[$i]}.log" || true- echo "::endgroup::"- done- if [ -n "$fail" ]; then- echo "a build lane failed (first failure aborts the rest); see lane logs above" >&2- exit 1- fi- "#})- .add_env(("FLY_ORG", "${{ vars.FLY_ORG || secrets.FLY_ORG }}"))- .add_env(("LOCAL_SANDBOX_IMAGE", vars::AGENT_HARNESS_LOCAL_IMAGE))-}--/// When the bake dies (typically the backend health gate), the answer is in-/// the service logs — dump container states and recent logs so a failed run is-/// diagnosable from CI output alone.-fn dump_stack_diagnostics() -> Step<Run> {- Step::new("Dump stack diagnostics")- .run(indoc::indoc! {r#"- docker compose -p macro ps --all || true- for svc in authentication-service proxy fusionauth postgres kafka \- connection_gateway document_storage_service email_service \- notification_service contacts_service; do- echo "==================== $svc ===================="- docker compose -p macro logs --no-color --tail 80 "$svc" 2>&1 || true- done- "#})- .if_condition(Expression::new("failure()"))-}--fn stage_context() -> Step<Run> {- Step::new("Stage preview build context").run(indoc::indoc! {r#"- set -euo pipefail- # The dev-shell env (BASH_ENV) exports a Nix LD_LIBRARY_PATH; host- # binaries like rsync then resolve Nix-store libs whose deps drag in- # Nix glibc mid-process and crash on glibc symbol versions. This step- # is pure file shuffling — drop it.- unset LD_LIBRARY_PATH- ctx=preview-ctx- rm -rf "$ctx"- mkdir -p "$ctx/repo" "$ctx/artifacts/binaries" "$ctx/bin"- # The minimal repo layout xtask reads at runtime — including every- # snapshot-key input, byte-identical to this checkout, so the VM's- # key matches the snapshot baked above.- rsync -a --relative --exclude node_modules \- docker/docker-compose.yml \- docker/docker-compose-databases.yml \- infra/stacks/fusionauth-instance/docker-compose.yml \- infra/local/nginx \- infra/local/opensearch \- infra/stacks/opensearch/helpers \- crates/macro_db_client/migrations \- "$ctx/repo/"- # Service binaries only — the target dir also holds gigabytes of- # build intermediates. `-p` everywhere: preserved mtimes mean files- # cargo didn't relink produce byte-identical Docker layers, so the- # registry push/pull skips them (the Dockerfile orders its COPYs- # stable → volatile for the same reason).- find target/x86_64-unknown-linux-gnu/debug \- -maxdepth 1 -type f -executable ! -name '*.d' ! -name '*.so' \- -exec cp -p {} "$ctx/artifacts/binaries/" \;- cp -a apps/web/dist "$ctx/artifacts/frontend-dist"- # Only the current key's snapshot — the volume store accumulates keys.- snap_dir=$(jq -r '.dir' "$RUNNER_TEMP/preview-snapshot.json")- mkdir -p "$ctx/artifacts/snapshots"- cp -a "$snap_dir" "$ctx/artifacts/snapshots/"- snapshot_key=$(jq -r '.key' "$RUNNER_TEMP/preview-snapshot.json")- runtime_key=$(cd "$ctx/repo" \- && find . -type f -print0 \- | sort -z \- | xargs -0 sha256sum \- | sha256sum \- | cut -d ' ' -f 1)- frontend_key=$(cd "$ctx/artifacts/frontend-dist" \- && find . -type f -print0 \- | sort -z \- | xargs -0 sha256sum \- | sha256sum \- | cut -d ' ' -f 1)- jq -n \- --arg snapshot_key "$snapshot_key" \- --arg runtime_key "$runtime_key" \- --arg frontend_key "$frontend_key" \- --arg commit "${{ github.sha }}" \- '{format: 1, snapshot_key: $snapshot_key, runtime_key: $runtime_key, frontend_key: $frontend_key, commit: $commit}' \- > "$ctx/deployment.json"- cp -p target/x86_64-unknown-linux-gnu/release/xtask_local "$ctx/bin/xtask"- cp -p infra/preview/hot-update.sh "$ctx/bin/hot-update"- cp -p infra/preview/entrypoint.sh "$ctx/entrypoint.sh"- cp -p infra/preview/Dockerfile "$ctx/Dockerfile"- cp -p infra/preview/update.Dockerfile "$ctx/update.Dockerfile"- "#})-}--fn setup_flyctl() -> Step<Use> {- Step::new("Setup flyctl").uses(- "superfly",- "flyctl-actions/setup-flyctl",- "ed8efb33836e8b2096c7fd3ba1c8afe303ebbff1",- ) // 1.6-}--/// Select bootstrap / hot update / rehydrate from the running machine's durable-/// compatibility marker. All modes mirror content-addressed stack images. A-/// compatible machine pulls a scratch artifact carrier and applies it in place;-/// an absent, incompatible, or failed hot update falls through to the immutable-/// full-image deploy and snapshot restore.-fn deploy_to_fly() -> Step<Run> {- Step::new("Deploy to Fly")- .run(indoc::indoc! {r#"- set -euo pipefail- if [ -z "$DOPPLER_PREVIEW_TOKEN" ]; then- echo "DOPPLER_PREVIEW_TOKEN secret is not set (a Doppler service token scoped to the preview config)" >&2- exit 1- fi- if [ -z "$FLY_ORG" ]; then- echo "FLY_ORG is not set (repo variable or secret with the Fly org slug)" >&2- exit 1- fi-- app_existed=- if flyctl status --app "$APP_NAME" >/dev/null 2>&1; then- app_existed=1- else- # A transient status error must not turn an existing app into a- # hard create failure: create, then prove the app is reachable.- flyctl apps create "$APP_NAME" --org "$FLY_ORG" \- || flyctl status --app "$APP_NAME" >/dev/null- fi-- # Seed a first boot's volume from the newest template volume — a- # warm /var/lib/docker layer store published by a previous- # successful deploy (see the "Publish template volume" step).- # Measured cold, an empty volume costs ~11 min of image pulls at- # boot. The images lane normally kicked the fork minutes ago (its- # ~2 min server-side hydration overlaps the whole build phase);- # this block is the authoritative fallback for a missing or- # failed early fork. Cross-app forks need the Machines API:- # flyctl resolves volume ids app-locally. The boot manifest check- # makes the volume a pure cache, so every failure path here just- # falls back to the old empty-volume behavior.- vol_state=$(flyctl volumes list --app "$APP_NAME" --json 2>/dev/null \- | jq -r '[.[] | select(.name == "docker_data")][0].state // empty') || vol_state=- if [ -z "$vol_state" ]; then- tpl_src=$(flyctl volumes list --app "$TEMPLATE_APP" --json 2>/dev/null \- | jq -r '[.[] | select((.name | startswith("tpl")) and .state == "created" and .attached_machine_id == null)]- | sort_by(.created_at) | last | .id // empty' 2>/dev/null) || tpl_src=- if [ -n "$tpl_src" ]; then- vol_state=$(curl -sf -X POST "https://api.machines.dev/v1/apps/$APP_NAME/volumes" \- -H "Authorization: Bearer $FLY_API_TOKEN" \- -H "Content-Type: application/json" \- -d "{\"name\":\"docker_data\",\"region\":\"ewr\",\"source_volume_id\":\"$tpl_src\"}" \- | jq -r '.state // empty') || vol_state=- if [ -n "$vol_state" ]; then- echo "seeding docker_data from template volume $tpl_src (state: $vol_state)"- else- echo "template fork failed; falling back to an empty volume" >&2- fi- else- echo "no template volume available; first boot will pull cold"- fi- fi-- # A read-only, app-scoped, time-boxed pull token so the machine's- # inner dockerd can pull the mirrored stack images from this app's- # registry repo — and nothing else (PR code can read machine- # secrets, so scope matters). The org deploy token can't mint new- # tokens (createLimitedAccessToken is not authorized), but macaroon- # attenuation is pure client-side crypto: append an Apps caveat- # (numeric app id, mask "r") and a validity window to our own- # token. The images lane normally minted it already; fall back to- # minting inline when its file is absent.- if [ -s "$RUNNER_TEMP/registry-pull-token" ]; then- pull_token=$(cat "$RUNNER_TEMP/registry-pull-token")- else- app_id=$(curl -sf https://api.fly.io/graphql \- -H "Authorization: Bearer $FLY_API_TOKEN" \- -H "content-type: application/json" \- -d "{\"query\":\"{ app(name: \\\"$APP_NAME\\\") { internalNumericId } }\"}" \- | jq -re '.data.app.internalNumericId')- now=$(date +%s)- pull_token=$(printf '[{"type":"Apps","body":{"apps":{"%s":"r"}}},{"type":"ValidityWindow","body":{"not_before":%d,"not_after":%d}}]' \- "$app_id" $((now - 60)) $((now + 604800)) | flyctl tokens attenuate)- fi-- snapshot_key=$(jq -r '.key' "$RUNNER_TEMP/preview-snapshot.json")- runtime_key=$(jq -r '.runtime_key' preview-ctx/deployment.json)- mode=bootstrap- machine_id=- if [ -n "$app_existed" ]; then- mode=rehydrate- machine_id=$(flyctl machine list --app "$APP_NAME" --json \- | jq -r '[.[] | select((.config.mounts // []) | length > 0)][0].id // empty')- if [ -n "$machine_id" ]; then- # A suspended preview needs demand before SSH is reachable.- curl -s -o /dev/null --max-time 10 "https://$APP_NAME.fly.dev/" || true- for _ in $(seq 1 12); do- # `--command` is exec'd directly, NOT run through a shell —- # a bare `if [ -f ...` dies with `exec: "if": not found`, so- # the marker never parsed and every push silently rehydrated- # (~7 min) instead of going hot (~3 min). Hence the sh -c.- marker=$(flyctl ssh console --app "$APP_NAME" --machine "$machine_id" \- --quiet --command 'sh -c "if [ -f /var/lib/docker/.macro-preview/deployment.json ]; then cat /var/lib/docker/.macro-preview/deployment.json; else echo __NO_MARKER__; fi"' \- 2>/dev/null || true)- if echo "$marker" | jq -e \- --arg key "$snapshot_key" \- --arg runtime "$runtime_key" \- '.format == 1 and .snapshot_key == $key and .runtime_key == $runtime and (.frontend_key | type == "string")' \- >/dev/null 2>&1; then- mode=hot- break- fi- # SSH succeeded and returned a marker (or proved it absent):- # incompatibility is real, not a wake-up race.- if [ "$marker" = __NO_MARKER__ ] \- || echo "$marker" | jq -e '.format != null' >/dev/null 2>&1; then- break- fi- sleep 5- done- fi- fi- echo "preview deployment mode: $mode (snapshot $snapshot_key)"-- # Keep the machine in demand for the WHOLE deploy, hot path- # included: fly-proxy suspends a machine ~7 min after its last- # proxied request, and SSH via hallpass is not proxy traffic — a- # hot update with no browser traffic got its VM suspended- # mid-session (the frozen `flyctl ssh console` then ate 52 min of- # runner until the job timeout). Any response status counts as- # traffic, and a request also resumes a suspended machine.- (while :; do- curl -s -o /dev/null --max-time 10 "https://$APP_NAME.fly.dev/" || true- sleep 15- done) &- keepalive=$!- trap 'kill "$keepalive" 2>/dev/null || true' EXIT-- flyctl auth docker- registry="registry.fly.io/$APP_NAME"- # Mirror every stack image into the app's registry repo instead of- # baking docker-save tars into the VM image (which made every- # deploy re-ship ~5GB the machine's volume already had). Pushes- # dedup at the layer level against the repo, so a redeploy only- # uploads layers that actually changed; the entrypoint pulls with- # the same dedup against the machine's persistent layer store. The- # manifest maps image ID -> local tag -> registry ref so boots can- # skip images already present. alpine:3 rides along for the- # snapshot-restore helper container.- # On a snapshot cache hit only gen-compose ran, which doesn't write- # the env file; compose hard-fails on a missing --env-file, and the- # image names don't interpolate env vars anyway.- envfile=infra/local/generated/macro/local.generated.env- [ -f "$envfile" ] || : > "$envfile"- images=$(docker compose --project-directory . -p macro \- -f docker/docker-compose.yml \- -f infra/local/generated/macro/docker-compose.override.yml \- --env-file "$envfile" \- config --images | sort -u)- echo "mirroring images:" $images- docker image inspect alpine:3 >/dev/null 2>&1 || docker pull alpine:3- mkdir -p preview-ctx/preload- : > preview-ctx/preload/manifest.txt- for img in $images alpine:3 "$LOCAL_SANDBOX_IMAGE"; do- # Images the bake didn't leave in the daemon (snapshot cache hit,- # or app-layer-only images like proxy/mailpit) get pulled here.- # Compose `config --images` also lists locally-built tags- # (analytics_proxy, sdk-webhook-relay) that are not on Docker Hub;- # premirror already skips those — do the same so a missing local- # tag cannot fail the deploy.- if ! docker image inspect "$img" >/dev/null 2>&1 && ! docker pull "$img"; then- echo "mirror: skipping $img (not local, not pullable)"- continue- fi- id=$(docker image inspect -f '{{.Id}}' "$img")- # Content-addressed tag: same image content = same ref, so a- # redeploy's push is skippable by a manifest existence check and- # concurrent PRs' deploys can never clobber each other's tags.- ref="$registry:img-$(echo "$img" | tr '/:' '__')-$(echo "$id" | cut -c8-19)"- docker tag "$img" "$ref"- echo "$id $img $ref" >> preview-ctx/preload/manifest.txt- done- # The existence probes are pure network round-trips (and after the- # premirror lane, almost all of them hit) — run them in parallel- # instead of ~1s each serially.- awk '{print $3}' preview-ctx/preload/manifest.txt \- | xargs -P 8 -I {} sh -c \- 'docker manifest inspect "{}" >/dev/null 2>&1 || echo "{}"' \- > "$RUNNER_TEMP/push-refs.txt"- echo "pushing $(wc -l < "$RUNNER_TEMP/push-refs.txt") of $(wc -l < preview-ctx/preload/manifest.txt) images"- # The Fly registry occasionally aborts large uploads ("s3aws:- # append to zero-size path unsupported") — retry; layers that made- # it are reused. Four pushes at a time keeps the first (cold) push- # moving without saturating the registry.- if [ -s "$RUNNER_TEMP/push-refs.txt" ]; then- xargs -P 4 -I {} sh -c '- for _ in 1 2 3; do- docker push "{}" && exit 0- echo "push of {} failed, retrying" >&2- sleep 15- done- exit 1' < "$RUNNER_TEMP/push-refs.txt"- fi-- push_image() {- image_to_push=$1- for _ in 1 2 3; do- if docker push "$image_to_push"; then return 0; fi- echo "docker push failed for $image_to_push, retrying" >&2- sleep 15- done- return 1- }-- if [ "$mode" = hot ]; then- update_image="$registry:update-${{ github.sha }}"- docker build -f preview-ctx/update.Dockerfile -t "$update_image" preview-ctx- push_image "$update_image"- token_file="$RUNNER_TEMP/registry-pull-token"- printf '%s' "$pull_token" > "$token_file"- chmod 600 "$token_file"- hot_rc=0- flyctl ssh sftp put "$token_file" /tmp/macro-registry-token \- --app "$APP_NAME" --machine "$machine_id" --mode 0600 \- || hot_rc=$?- if [ "$hot_rc" = 0 ]; then- # timeout backstop: a suspended-mid-session VM freezes the ssh- # client forever (measured: 52 min of hang). 10 min is triple- # a normal hot apply; on expiry we fall back to rehydrate.- hot_command="/srv/macro/bin/hot-update '$update_image' /tmp/macro-registry-token"- timeout 600 flyctl ssh console --app "$APP_NAME" --machine "$machine_id" \- --command "$hot_command" || hot_rc=$?- fi- if [ "$hot_rc" = 0 ]; then- echo "hot update completed without restarting the Fly machine"- exit 0- fi- echo "hot update failed (exit $hot_rc); falling back to full rehydrate" >&2- mode=rehydrate- fi-- # Bootstrap/rehydrate refreshes app secrets and reconciles the- # persistent Docker volume before replacing the machine image.- flyctl secrets set --app "$APP_NAME" --stage \- "DOPPLER_TOKEN=$DOPPLER_PREVIEW_TOKEN" \- "REGISTRY_PULL_TOKEN=$pull_token"- if [ -z "$vol_state" ]; then- # No volume and no fork landed (no template, or both fork- # attempts failed): the old empty-volume path.- flyctl volumes create docker_data --app "$APP_NAME" --region ewr --size 40 --yes- elif [ "$vol_state" != created ]; then- # A fork is still hydrating (usually already finished: the- # images lane kicked it before the cargo lane even ended). The- # volume must reach "created" before a machine can mount it; a- # fork that never lands is replaced by an empty volume rather- # than failing the deploy.- for _ in $(seq 1 60); do- vol_state=$(flyctl volumes list --app "$APP_NAME" --json \- | jq -r '[.[] | select(.name == "docker_data")][0].state // empty')- [ "$vol_state" = created ] && break- sleep 10- done- echo "seeded volume state: $vol_state"- if [ "$vol_state" != created ]; then- echo "template fork stuck (state: $vol_state); replacing with an empty volume" >&2- vol_id=$(flyctl volumes list --app "$APP_NAME" --json \- | jq -r '[.[] | select(.name == "docker_data")][0].id // empty')- [ -z "$vol_id" ] || flyctl volumes destroy "$vol_id" --app "$APP_NAME" --yes || true- flyctl volumes create docker_data --app "$APP_NAME" --region ewr --size 40 --yes- fi- fi- # Machines created before the volume existed can't take the mounts- # config update. A destroyed/partial machine is recreated by deploy.- flyctl machine list --app "$APP_NAME" --json \- | jq -r '.[] | select((.config.mounts // []) | length == 0) | .id' \- | while read -r id; do- [ -z "$id" ] || flyctl machine destroy "$id" --app "$APP_NAME" --force || true- done-- image="$registry:${{ github.sha }}"- docker build -t "$image" preview-ctx- push_image "$image"- # First boot does real work before 8090 opens (pulling the stack- # images, snapshot restore, compose up, FusionAuth's JVM) —- # give the health check more runway than flyctl's default wait.- # The step-wide keepalive above holds the machine out of suspend.- flyctl deploy --app "$APP_NAME" \- --config infra/preview/fly.toml \- --image "$image" \- --wait-timeout 1800 \- --yes- "#})- // Accept the org slug from either a repo variable or a repo secret —- // it's not sensitive, but people reasonably reach for secrets first.- .add_env(("FLY_ORG", "${{ vars.FLY_ORG || secrets.FLY_ORG }}"))- .add_env(("DOPPLER_PREVIEW_TOKEN", vars::DOPPLER_PREVIEW_TOKEN))- .add_env(("LOCAL_SANDBOX_IMAGE", vars::AGENT_HARNESS_LOCAL_IMAGE))-}--/// After a healthy deploy this PR's volume is, by construction, a fully-/// warmed layer store for exactly the image set in the preload manifest-/// (bootstrap pulled it, hot-update reconciled it). Publish it as the-/// org-wide template that future first boots fork, content-addressed by the-/// manifest's (image id, tag) pairs — registry refs are excluded because-/// they embed the per-PR app name. An existing volume under the same key-/// means the image set is unchanged and nothing runs; there is no scheduled-/// refresh because the cache warms itself on exactly the deploys that-/// change it. Purely an optimization, hence continue-on-error: the boot-/// manifest check keeps forks of any template correct, stale or fresh.-fn publish_template_volume() -> Step<Run> {- Step::new("Publish template volume")- .run(indoc::indoc! {r#"- set -euo pipefail- key=$(awk '{print $1, $2}' preview-ctx/preload/manifest.txt \- | LC_ALL=C sort | sha256sum | cut -c1-24)- vol_name="tpl$key"- flyctl status --app "$TEMPLATE_APP" >/dev/null 2>&1 \- || flyctl apps create "$TEMPLATE_APP" --org "$FLY_ORG"- vols=$(flyctl volumes list --app "$TEMPLATE_APP" --json 2>/dev/null || echo '[]')- if echo "$vols" | jq -e --arg n "$vol_name" \- 'map(select(.name == $n)) | length > 0' >/dev/null; then- echo "template $vol_name already exists; image set unchanged"- else- src=$(flyctl volumes list --app "$APP_NAME" --json \- | jq -r '[.[] | select(.name == "docker_data")][0].id // empty')- if [ -n "$src" ]; then- # Fire-and-forget: hydration is a server-side block copy;- # nothing in this run depends on it finishing.- curl -sf -X POST "https://api.machines.dev/v1/apps/$TEMPLATE_APP/volumes" \- -H "Authorization: Bearer $FLY_API_TOKEN" \- -H "Content-Type: application/json" \- -d "{\"name\":\"$vol_name\",\"region\":\"ewr\",\"source_volume_id\":\"$src\"}" \- | jq -r '"published \(.name) (\(.id), state: \(.state))"' \- || echo "template publish failed (non-fatal)" >&2- fi- fi- # Prune superseded templates: keep the 3 newest, and never touch- # anything younger than 2h — it may still be hydrating or serving- # as a concurrent deploy's fork source.- cutoff=$(date -u -d '2 hours ago' +%Y-%m-%dT%H:%M:%SZ)- echo "$vols" | jq -r --arg cutoff "$cutoff" \- '[.[] | select((.name | startswith("tpl")) and .attached_machine_id == null)]- | sort_by(.created_at) | reverse | .[3:]- | .[] | select(.created_at < $cutoff) | .id' \- | while read -r old; do- [ -z "$old" ] || flyctl volumes destroy "$old" --app "$TEMPLATE_APP" --yes || true- done- "#})- .add_env(("FLY_ORG", "${{ vars.FLY_ORG || secrets.FLY_ORG }}"))- .continue_on_error(true)-}--fn comment_preview_url() -> Step<Use> {- Step::new("Comment preview URL")- .uses(- "actions",- "github-script",- "f28e40c7f34bde8b3046d885e986cb6290c5673b",- )- .add_with(("github-token", "${{ secrets.GITHUB_TOKEN }}"))- .add_with((- "script",- indoc::indoc! {r#"- const app = process.env.APP_NAME;- const url = `https://${app}.fly.dev`;- const marker = '<!-- fly-preview -->';- const body = [- marker,- `🚀 **Full-stack preview**: ${url}`,- '',- `- Suspended when idle — the first request wakes it (a few seconds).`,- `- Log in with any email; the passwordless code lands in [Mailpit](${url}/mailpit/).`,- `- Deployed from ${context.payload.pull_request.head.sha.slice(0, 7)}.`,- ].join('\n');- const { data: comments } = await github.rest.issues.listComments({- owner: context.repo.owner,- repo: context.repo.repo,- issue_number: context.issue.number,- per_page: 100,- });- const existing = comments.find(c => c.body && c.body.startsWith(marker));- if (existing) {- await github.rest.issues.updateComment({- owner: context.repo.owner,- repo: context.repo.repo,- comment_id: existing.id,- body,- });- } else {- await github.rest.issues.createComment({- owner: context.repo.owner,- repo: context.repo.repo,- issue_number: context.issue.number,- body,- });- }- "#},- ))-}--fn destroy() -> Job {- Job::default()- // Closed PRs always attempt cleanup (destroy is a no-op if no app);- // unlabeled only cleans up when the removed label is `preview`.- .cond(Expression::new(- "github.event.action == 'closed' || github.event.label.name == 'preview'",- ))- .runs_on(runners::Runner::TinyNoCache.to_string())- .permissions(Permissions::default().contents(Level::Read))- .add_env(("FLY_API_TOKEN", vars::FLY_API_TOKEN))- .add_step(setup_flyctl())- .add_step(- Step::new("Destroy app")- .run(format!("flyctl apps destroy \"{APP_NAME}\" --yes || true")),- )-}diff --git a/tooling/xtask/crates/xtask_workflows/src/workflows/preview_fly/test.rs b/tooling/xtask/crates/xtask_workflows/src/workflows/preview_fly/test.rsdeleted file mode 100644index d497198130..0000000000--- a/tooling/xtask/crates/xtask_workflows/src/workflows/preview_fly/test.rs+++ /dev/null@@ -1,87 +0,0 @@-use super::*;--fn rendered() -> String {- preview_fly()- .to_string()- .expect("workflow should serialize")-}--#[test]-fn deploy_does_not_need_ghcr_packages() {- let yaml = rendered();- assert!(- !yaml.contains("packages: read"),- "preview builds the sandbox image locally: {yaml}"- );-}--#[test]-fn both_image_loops_include_the_sandbox_tag() {- let yaml = rendered();- let tag = vars::AGENT_HARNESS_LOCAL_IMAGE;- assert!(- yaml.contains(&format!("LOCAL_SANDBOX_IMAGE: {tag}")),- "sandbox tag should be injected as step env: {yaml}"- );- assert!(- yaml.contains("alpine:3 \"$LOCAL_SANDBOX_IMAGE\""),- "premirror loop should append the sandbox tag: {yaml}"- );- assert!(- yaml.contains("$images alpine:3 \"$LOCAL_SANDBOX_IMAGE\""),- "deploy loop should append the sandbox tag: {yaml}"- );-}--#[test]-fn deploy_timeout_covers_a_cold_sandbox_image_bake() {- let yaml = rendered();- assert!(- yaml.contains("timeout-minutes: 90"),- "sandbox image bake can exceed the old 60m backstop: {yaml}"- );-}--#[test]-fn images_lane_always_builds_the_sandbox_image() {- let yaml = rendered();- assert!(- yaml.contains("docker build --tag \"$LOCAL_SANDBOX_IMAGE\" crates/agent_harness/container"),- "{yaml}"- );- assert!(- !yaml.contains("PREVIEW_BASE_SHA"),- "git-diff vs base is unnecessary when we always build: {yaml}"- );- assert!(- !yaml.contains("SANDBOX_GHCR_IMAGE"),- "preview should not pull GHCR: {yaml}"- );-}--#[test]-fn web_lane_raises_the_node_heap() {- let yaml = rendered();- assert!(- yaml.contains("NODE_OPTIONS=--max-old-space-size=6144"),- "vite OOM is the documented CI flap without this: {yaml}"- );-}--#[test]-fn images_lane_builds_analytics_proxy() {- let yaml = rendered();- assert!(- yaml.contains("ai_editing_worker analytics_proxy"),- "analytics_proxy is a default compose service; omitting it makes deploy pull docker.io/library/macro-analytics_proxy: {yaml}"- );-}--#[test]-fn deploy_mirror_skips_unpullable_local_tags() {- let yaml = rendered();- assert!(- yaml.contains("mirror: skipping $img (not local, not pullable)"),- "deploy must skip compose-built tags the way premirror already does: {yaml}"- );-}diff --git a/tooling/xtask/crates/xtask_workflows/src/workflows/steps.rs b/tooling/xtask/crates/xtask_workflows/src/workflows/steps.rsindex 635c30ae36..c3ddd0d0d1 100644--- a/tooling/xtask/crates/xtask_workflows/src/workflows/steps.rs+++ b/tooling/xtask/crates/xtask_workflows/src/workflows/steps.rs@@ -48,9 +48,9 @@ pub(crate) fn uses_local(name: &str, path: RepoDir<'_>) -> Step<Use> {step}-/// Namespace remote BuildKit, pinned. Multi-arch `docker buildx --push` and the-/// Fly preview image builds both go through it so nix-in-Docker work does not-/// land on a Small runner's local daemon.+/// Namespace remote BuildKit, pinned. Multi-arch `docker buildx --push` goes+/// through it so nix-in-Docker work does not land on a Small runner's local+/// daemon.pub fn setup_namespace_buildx() -> Step<Use> {Step::new("Set up Namespace Docker builder").uses("namespacelabs",@@ -201,31 +201,10 @@ fn nscloud_cache_action(name: &str) -> Step<Use> {.continue_on_error(true)}-/// [`mount_cache_volume`] plus the checkout's cargo target dir and the init-/// snapshot store. Persisting `target/` is what makes the preview job's-/// zigbuild incremental — cargo's own fingerprints carry across runs, where-/// remote sccache alone leaves build scripts, native (cmake/zig) compiles, and-/// linking cold every time. Persisting the snapshot store gives the bake step-/// a zero-copy fast path; Namespace artifact storage is its durable fallback.-/// The volume is a block-device mount, so multi-GB trees cost nothing to save-/// or restore when it hits.-pub fn mount_cache_volume_with_cargo_target() -> Step<Use> {- nscloud_cache_action("Mount Namespace cache volume")- .add_with(("cache", "nix"))- .add_with((- "path",- format!(- "${{{{ github.workspace }}}}/target\n{}\n/home/runner/.cargo/registry\n/home/runner/.cargo/git",- vars::PREVIEW_SNAPSHOT_VOLUME_DIR,- ),- ))-}-/// [`mount_cache_volume`] for the wasm worker build. `target/` is listed-/// explicitly for the same reason [`mount_cache_volume_with_cargo_target`] does-/// it — `cache: rust` alone is only relied on for the registry/git here. The-/// last two paths are what wrangler's `[build]` line otherwise redoes every-/// run: the pinned `worker-build` binary and wasm-pack's downloaded `wasm-opt`.+/// explicitly so cargo fingerprints survive across runs. The last two paths+/// are what wrangler's `[build]` line otherwise redoes every run: the pinned+/// `worker-build` binary and wasm-pack's downloaded `wasm-opt`.pub fn mount_wasm_cache_volume() -> Step<Use> {nscloud_cache_action("Mount Namespace cache volume").add_with(("cache", "rust"))diff --git a/tooling/xtask/crates/xtask_workflows/src/workflows/vars.rs b/tooling/xtask/crates/xtask_workflows/src/workflows/vars.rsindex f63828e6de..49fa63c6bf 100644--- a/tooling/xtask/crates/xtask_workflows/src/workflows/vars.rs+++ b/tooling/xtask/crates/xtask_workflows/src/workflows/vars.rs@@ -17,9 +17,7 @@ secret!(CLOUDFLARE_API_TOKEN);secret!(DAYTONA_API_KEY);secret!(DD_API_KEY);secret!(DD_APP_KEY);-secret!(DOPPLER_PREVIEW_TOKEN);secret!(DOPPLER_TOKEN);-secret!(FLY_API_TOKEN);secret!(MACOS_DEVELOPER_ID_CERTIFICATE_BASE64);secret!(MACOS_DEVELOPER_ID_CERTIFICATE_PASSWORD);secret!(NIX_CACHE_SIGNING_KEY);@@ -61,23 +59,6 @@ pub const CI_CACHE_TAG: &str = "sccache-ci";/// the workspace dependency checks.pub const CI_SCCACHE_NAME: &str = "sccache-ci";-/// Namespace cache tag for the Fly preview deploy job. Its own pool, NOT-/// [`CI_CACHE_TAG`]: sharing looked economical but was measured cold both ways-/// (run 28968155599: sccache 2.65% hits, cargo target dir absent). The-/// check/test jobs compile for the host while this job zigbuilds with-/// `--target x86_64-unknown-linux-gnu.2.36`, so their sccache entries hash-/// differently and never serve this job — and they don't persist the cargo-/// target dir at all, so a volume from the shared pool almost never carries-/// one. A dedicated low-concurrency pool gives the job's target dir and init-/// snapshots the best chance of a zero-copy hit; Rust objects use the durable-/// remote sccache below instead of depending on this volume's placement.-pub const PREVIEW_CACHE_TAG: &str = "fly-preview";--/// Durable Namespace remote sccache shared by Fly preview jobs. Unlike the-/// local cache volume, this has an artifact-backed cold tier, so a job placed-/// on a runner that has never seen the volume can still reuse Rust objects.-pub const PREVIEW_SCCACHE_NAME: &str = "fly-preview";-/// Namespace cache tag for the web-app jobs (PR checks + preview deploys)./// Cache volumes are keyed workspace-wide by tag alone, so a dedicated tag/// gives the frontend its own volume — isolated both from the Rust CI volume@@ -99,20 +80,10 @@ pub const SYNC_SERVICE_CACHE_TAG: &str = "sync-service-deploy";/// the Nix dev shell and is not available to Namespace's cache planner yet.pub const BUN_CACHE_VOLUME_DIR: &str = "/home/runner/.bun/install/cache";-/// Init-snapshot store for the preview job (`MACRO_STACK_SNAPSHOT_DIR`). Lives-/// on the preview cache volume for the zero-copy fast path; the workflow also-/// backs each content-addressed snapshot up to Namespace artifact storage so a-/// cache-volume miss does not force another infra bake.-pub const PREVIEW_SNAPSHOT_VOLUME_DIR: &str = "/home/runner/.cache/macro-preview-snapshots";-/// GHCR repository for the agent-harness sandbox image (the same Dockerfile/// Daytona snapshots). Pushed as `:$SHA` on PRs and `:$SHA` + `:latest` on main.pub const AGENT_HARNESS_GHCR_IMAGE: &str = "ghcr.io/macro-inc/macro-agent-harness";-/// Local Docker tag `just run_local` / Fly previews load. Keep in sync with-/// `xtask_local::local::sandbox_image::DEFAULT_LOCAL_TAG`.-pub const AGENT_HARNESS_LOCAL_IMAGE: &str = "macro-agent-harness:latest";-/// The repo-wide env block (mirrors the original top-level `env:`). Defaults the/// linker to `lld`; the heavy jobs override `RUSTFLAGS` to use `mold`.pub fn with_global_env(workflow: Workflow) -> Workflow {