Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

feat(cloud): drop Fly preview and run the app from Vite (#5863)

teoe77b5e6bba5d
diff --git a/.claude/skills/run-app/SKILL.md b/.claude/skills/run-app/SKILL.md
new file mode 100644
index 0000000000..3a5df8e1e7
--- /dev/null
+++ b/.claude/skills/run-app/SKILL.md
@@ -0,0 +1,82 @@
+---
+name: run-app
+description: >-
+ Run the Macro app on Cursor Cloud and pick up edits. Use when asked to run
+ the app, open it in the browser, test a frontend or backend change, hot
+ reload, or before touching stack.sh, rebuild.sh, frontend.sh, or just stack.
+---
+
+# Run the app
+
+The app is http://localhost:3000/app. Open it in the VM browser. The user's laptop cannot reach these URLs.
+
+| Goal | Command |
+| --- | --- |
+| Start | `bash .cursor/stack.sh` |
+| After a Rust edit | `bash .cursor/rebuild.sh` |
+| After an `apps/web` edit | save the file |
+
+The `.cursor` scripts re-enter the pinned nix shell. For any other `docker`, `just`, `bun`, or `doppler` command, prefix with `nix develop /workspace --command`.
+
+Do not run `just run_local`. That is the laptop TUI.
+
+## Start
+
+```bash
+bash .cursor/stack.sh
+```
+
+Let the first run finish. It compiles whatever changed since the environment bake. Do not kill the Nix build, copy binaries out of `/nix/store`, or replace this with hand-rolled `docker compose`.
+
+Log in with any email. The login API returns the code. Codes also land in Mailpit at http://localhost:8025.
+
+## After a Rust edit
+
+```bash
+bash .cursor/rebuild.sh
+```
+
+Volumes stay. If the log says binaries are unchanged, the mounts were left alone.
+
+Do not run `just stack up` or `stack.sh --fresh` to pick up an edit. `up` deletes the stack and recreates it. Use `--fresh` only when you mean to wipe.
+
+`cargo build` and `just build` do not remount containers. `rebuild.sh` is the remount path.
+
+## After an `apps/web` edit
+
+Save the file. Vite watches `apps/web`. If the page does not update, read `~/.cursor-cloud/frontend-dev.log` for `hmr update`.
+
+If http://localhost:3000/app stops responding, restart the dev server:
+
+```bash
+bash .cursor/frontend.sh stop && bash .cursor/frontend.sh
+```
+
+Use the same restart after a frontend dependency or Vite config change.
+
+Do not run `just stack update --frontend`. That builds a static bundle. This Cloud stack does not serve one. On a `--no-frontend` stack that flag then tells you to re-run `just stack up`, which wipes volumes.
+
+## Show the change
+
+Bring the stack up, log in, and demonstrate the change in the UI. Unit tests and SQL probes are not a walkthrough.
+
+## If login fails
+
+If the page says "unable to lookup identity providers", FusionAuth is in maintenance mode. `docker ps` shows `macro-fusionauth-1` as unhealthy. `curl localhost:9011/api/status` returns 302.
+
+```bash
+docker restart macro-fusionauth-1
+```
+
+Wait until the container is healthy.
+
+If the page loads and API calls fail, the backend is down. Run `just stack status --json`, then `bash .cursor/stack.sh`.
+
+Do not chase `agent_harness_service` restart loops when `DOPPLER_TOKEN` is absent. That loop is expected.
+
+## Status
+
+```bash
+just stack status --json
+docker compose -p macro logs -f <service>
+```
diff --git a/.cursor/cloud-lib.sh b/.cursor/cloud-lib.sh
index de6074de93..e00ddd7b66 100755
--- a/.cursor/cloud-lib.sh
+++ b/.cursor/cloud-lib.sh
@@ -235,7 +235,7 @@ ensure_dockerd() {
# Populate `doppler_args` for `just stack up`. The doppler CLI is already on
# PATH from `nix develop` — this only chooses whether to pass `--no-doppler`.
# A config-scoped service token (local / lcl_preview) is enough; no login.
-# DOPPLER_PREVIEW_TOKEN is the Fly/CI name for the same token.
+# DOPPLER_PREVIEW_TOKEN is the CI name for the same token.
stack_doppler_args() {
if [ -z "${DOPPLER_TOKEN:-}" ] && [ -n "${DOPPLER_PREVIEW_TOKEN:-}" ]; then
export DOPPLER_TOKEN="${DOPPLER_PREVIEW_TOKEN}"
@@ -249,6 +249,20 @@ stack_doppler_args() {
fi
}
+# Run a command while re-applying bridge forwarding every few seconds.
+# Compose creates networks mid-`stack up`; a service booting on a bridge
+# before its FORWARD rules exist times out reaching its database (FusionAuth
+# wedged in maintenance mode this way). The watcher closes that window.
+run_with_bridge_forwarding() {
+ "$@" &
+ local cmd_pid=$!
+ while kill -0 "${cmd_pid}" 2>/dev/null; do
+ ensure_docker_bridge_forwarding
+ sleep 3
+ done
+ wait "${cmd_pid}"
+}
+
in_pinned_nix_shell() {
[ "${MACRO_CLOUD_PINNED_NIX:-}" = "1" ]
}
diff --git a/.cursor/frontend.sh b/.cursor/frontend.sh
new file mode 100755
index 0000000000..ab28d89a06
--- /dev/null
+++ b/.cursor/frontend.sh
@@ -0,0 +1,78 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+# THE frontend on Cloud: a hot-reloading Vite dev server against the running
+# backend. stack.sh starts it; edits under apps/web apply on save. There is
+# no static bundle — http://localhost:3000/app is the only app URL.
+#
+# bash .cursor/frontend.sh # start (no-op if already serving)
+# bash .cursor/frontend.sh stop # stop the dev server
+
+# shellcheck source=cloud-lib.sh
+source "${SCRIPT_DIR}/cloud-lib.sh"
+
+if ! in_pinned_nix_shell; then
+ ensure_nix_daemon
+ reenter_pinned_nix_shell "${SCRIPT_DIR}/frontend.sh" "$@"
+fi
+
+FRONTEND_PORT=3000
+PROXY_ORIGIN='http://localhost:8090'
+DEV_URL="http://localhost:${FRONTEND_PORT}/app"
+PID_FILE="${LOG_DIR}/frontend-dev.pid"
+DEV_LOG="${LOG_DIR}/frontend-dev.log"
+
+if [ "${1:-}" = "stop" ]; then
+ if [ -f "${PID_FILE}" ]; then
+ pgid="$(cat "${PID_FILE}")"
+ kill -- "-${pgid}" 2>/dev/null || true
+ rm -f "${PID_FILE}"
+ echo "cursor-cloud frontend: stopped"
+ else
+ echo "cursor-cloud frontend: not running (no pid file)"
+ fi
+ exit 0
+fi
+
+if curl -fsS --max-time 2 "${DEV_URL}" >/dev/null 2>&1; then
+ echo "cursor-cloud frontend: already serving at ${DEV_URL}"
+ exit 0
+fi
+
+if ! curl -fsS --max-time 3 "${PROXY_ORIGIN}/auth/health" >/dev/null 2>&1; then
+ echo "cursor-cloud frontend: stack is not running — run: bash .cursor/stack.sh" >&2
+ exit 1
+fi
+
+: >"${DEV_LOG}"
+\cd "${WORKSPACE_ROOT}/apps/web"
+setsid env \
+ PORT="${FRONTEND_PORT}" \
+ VITE_LOCAL_SERVERS=ALL \
+ VITE_LOCAL_BACKEND_ORIGIN="${PROXY_ORIGIN}" \
+ VITE_AI_EDITING_WORKER_URL="${PROXY_ORIGIN}/ai-editing" \
+ bun run --bun dev >>"${DEV_LOG}" 2>&1 </dev/null &
+echo "$!" >"${PID_FILE}"
+
+# Cold starts build wasm packages before Vite binds; be patient once.
+n=0
+while [ "${n}" -lt 300 ]; do
+ if ! kill -0 "$(cat "${PID_FILE}")" 2>/dev/null; then
+ echo "cursor-cloud frontend: dev server exited during startup" >&2
+ tail -40 "${DEV_LOG}" >&2
+ rm -f "${PID_FILE}"
+ exit 1
+ fi
+ if curl -fsS --max-time 2 "${DEV_URL}" >/dev/null 2>&1; then
+ echo "cursor-cloud frontend: hot-reloading dev server at ${DEV_URL}"
+ echo "cursor-cloud frontend: logs at ${DEV_LOG}"
+ exit 0
+ fi
+ n=$((n + 1))
+ sleep 1
+done
+echo "cursor-cloud frontend: dev server did not become ready" >&2
+tail -40 "${DEV_LOG}" >&2
+exit 1
diff --git a/.cursor/rebuild.sh b/.cursor/rebuild.sh
index 2eb9b197d9..e4e9b2b666 100755
--- a/.cursor/rebuild.sh
+++ b/.cursor/rebuild.sh
@@ -3,9 +3,9 @@ set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-# Backend-edit flow, as one verb: rebuild the Nix stack binaries and recycle
-# the stack onto them. This replaces `just stack update`, which cannot write
-# into the read-only Nix binaries directory.
+# Backend-edit flow: rebuild Nix stack binaries and adopt them into the
+# running stack. Volumes stay. If nothing is recorded yet, `stack update`
+# bootstraps through `stack up`.
# shellcheck source=cloud-lib.sh
source "${SCRIPT_DIR}/cloud-lib.sh"
@@ -18,5 +18,11 @@ fi
/usr/bin/bash "${SCRIPT_DIR}/infra.sh"
\cd "${WORKSPACE_ROOT}"
-just stack down
-exec /usr/bin/bash "${SCRIPT_DIR}/stack.sh"
+build_local_stack_binaries
+stack_doppler_args
+# The watcher matters on the bootstrap path (update -> up creates networks).
+run_with_bridge_forwarding \
+ just stack update "${doppler_args[@]}" --binaries-dir "${LOCAL_STACK_BINS}/bin"
+ensure_docker_bridge_forwarding
+
+echo "cursor-cloud stack: binaries remounted"
diff --git a/.cursor/stack.sh b/.cursor/stack.sh
index 831c9a1b0f..eb01c26eec 100755
--- a/.cursor/stack.sh
+++ b/.cursor/stack.sh
@@ -3,8 +3,13 @@ set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-# On-demand product stack. Boot (start.sh) starts nothing; infra.sh brings up
-# dockerd and the databases first. After backend edits, run rebuild.sh.
+# On-demand product stack: backend containers behind the proxy (8090) plus
+# the hot-reloading frontend dev server (3000). There is no static frontend
+# on Cloud — the dev server IS the frontend, so edits under apps/web apply
+# on save. A healthy running backend is left alone — `stack up` is
+# full-delete/full-create, so a blind re-run would wipe volumes (logins,
+# documents). Pass --fresh to wipe and recreate deliberately.
+# After backend edits, run rebuild.sh: it remounts new Nix binaries in place.
# shellcheck source=cloud-lib.sh
source "${SCRIPT_DIR}/cloud-lib.sh"
@@ -17,9 +22,22 @@ fi
/usr/bin/bash "${SCRIPT_DIR}/infra.sh"
\cd "${WORKSPACE_ROOT}"
-build_local_stack_binaries
-stack_doppler_args
-just stack up "${doppler_args[@]}" --build-aux-services --binaries-dir "${LOCAL_STACK_BINS}/bin"
-ensure_docker_bridge_forwarding
-echo "cursor-cloud stack: app ready"
+if [ "${1:-}" != "--fresh" ] \
+ && curl -fsS --max-time 3 http://localhost:8090/auth/health >/dev/null 2>&1; then
+ echo "cursor-cloud stack: backend already running"
+else
+ build_local_stack_binaries
+ stack_doppler_args
+ run_with_bridge_forwarding \
+ just stack up "${doppler_args[@]}" --no-frontend --build-aux-services \
+ --binaries-dir "${LOCAL_STACK_BINS}/bin"
+ ensure_docker_bridge_forwarding
+fi
+
+/usr/bin/bash "${SCRIPT_DIR}/frontend.sh"
+
+echo "cursor-cloud stack: app ready at http://localhost:3000/app"
+echo "cursor-cloud stack: backend edits -> bash .cursor/rebuild.sh"
+echo "cursor-cloud stack: frontend edits apply on save (no command)"
+echo "cursor-cloud stack: wipe + restart -> bash .cursor/stack.sh --fresh"
diff --git a/.github/workflows/preview-fly-cleanup.yml b/.github/workflows/preview-fly-cleanup.yml
deleted file mode 100644
index ebe911e0bb..0000000000
--- a/.github/workflows/preview-fly-cleanup.yml
+++ /dev/null
@@ -1,21 +0,0 @@
-# DO NOT EDIT — regenerate with `cargo x workflows` (from the repository root).
-# Source: tooling/xtask/crates/xtask_workflows/src/workflows/preview_fly.rs
-name: Fly Preview Cleanup
-'on':
- pull_request:
- types:
- - closed
- - unlabeled
-jobs:
- destroy:
- if: github.event.action == 'closed' || github.event.label.name == 'preview'
- runs-on: namespace-profile-linux-tiny-no-cache
- permissions:
- contents: read
- env:
- FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
- steps:
- - name: Setup flyctl
- uses: superfly/flyctl-actions/setup-flyctl@ed8efb33836e8b2096c7fd3ba1c8afe303ebbff1
- - name: Destroy app
- run: flyctl apps destroy "macro-pr-${{ github.event.pull_request.number }}" --yes || true
diff --git a/.github/workflows/preview-fly.yml b/.github/workflows/preview-fly.yml
deleted file mode 100644
index 632159d6cf..0000000000
--- a/.github/workflows/preview-fly.yml
+++ /dev/null
@@ -1,767 +0,0 @@
-# DO NOT EDIT — regenerate with `cargo x workflows` (from the repository root).
-# Source: tooling/xtask/crates/xtask_workflows/src/workflows/preview_fly.rs
-name: Fly Preview
-'on':
- pull_request:
- types:
- - opened
- - reopened
- - synchronize
- - labeled
-jobs:
- deploy:
- if: github.event.pull_request.head.repo.full_name == github.repository && contains(github.event.pull_request.labels.*.name, 'preview')
- runs-on: namespace-profile-linux-mid;overrides.cache-tag=fly-preview
- permissions:
- contents: read
- pull-requests: write
- env:
- FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
- APP_NAME: macro-pr-${{ github.event.pull_request.number }}
- TEMPLATE_APP: macro-preview-template
- MACRO_STACK_SNAPSHOT_DIR: /home/runner/.cache/macro-preview-snapshots
- CARGO_INCREMENTAL: '0'
- timeout-minutes: 90
- steps:
- - name: Checkout
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
- with:
- clean: 'false'
- - name: Mount Namespace cache volume
- uses: namespacelabs/nscloud-cache-action@15799a6b54e5765f85b2aac25b3f0df43ed571c0
- with:
- cache: nix
- path: |-
- ${{ github.workspace }}/target
- /home/runner/.cache/macro-preview-snapshots
- /home/runner/.cargo/registry
- /home/runner/.cargo/git
- continue-on-error: true
- - name: Prune stale incremental caches
- run: rm -rf target/*/debug/incremental target/debug/incremental
- - name: Set up Namespace Docker builder
- uses: namespacelabs/nscloud-setup-buildx-action@d059ed7184f0bc7c8b27e8810cea153d02bcc6dd
- - name: Setup Nix
- uses: './.github/actions/setup-nix'
- - name: Setup dev shell + web deps
- uses: './.github/actions/setup-reqs-web'
- - name: Configure Namespace remote sccache
- if: (github.event_name != 'pull_request' && github.event_name != 'pull_request_target') || github.event.pull_request.head.repo.full_name == github.repository
- run: |-
- set -euo pipefail
- env_file="$(mktemp "$RUNNER_TEMP/namespace-sccache.XXXXXX")"
- trap 'rm -f "$env_file"' EXIT
- if nsc cache sccache setup --cache_name fly-preview > "$env_file"; then
- # Register credential values as masked BEFORE exporting: GitHub only
- # masks `secrets.*`, so without this SCCACHE_WEBDAV_TOKEN — a broad,
- # ~24h Namespace workspace token (registry + cache write) — printed
- # verbatim in every subsequent step's env dump. Masking is selective by
- # key name: masking non-secrets like the endpoint URL or key prefix
- # would redact those strings everywhere in the logs.
- while IFS= read -r line; do
- k="${line%%=*}"
- v="${line#*=}"
- [ -n "$v" ] && [ "$k" != "$line" ] || continue
- case "$k" in
- *TOKEN*|*SECRET*|*PASSWORD*) echo "::add-mask::$v" ;;
- esac
- done < "$env_file"
- cat "$env_file" >> "$GITHUB_ENV"
- # Force the next compiler invocation to start a server with the new remote
- # backend even if a setup hook happened to launch one already.
- sccache --stop-server >/dev/null 2>&1 || true
- else
- echo "::warning::Namespace remote sccache setup failed; using local cache fallback"
- fi
- - name: Setup flyctl
- uses: superfly/flyctl-actions/setup-flyctl@ed8efb33836e8b2096c7fd3ba1c8afe303ebbff1
- - name: Build artifacts (parallel lanes)
- run: |
- set -euo pipefail
- lanes="$RUNNER_TEMP/lanes"
- mkdir -p "$lanes"
-
- cat > "$lanes/cargo.sh" <<'LANE'
- set -euo pipefail
- cargo x zigbuild
- sccache --show-stats
- cargo zigbuild --release --target x86_64-unknown-linux-gnu.2.36 -p xtask_local
- LANE
-
- cat > "$lanes/web.sh" <<'LANE'
- set -euo pipefail
- cd apps/web
- export MODE=development NODE_ENV=production
- export VITE_LOCAL_SERVERS=ALL VITE_LOCAL_BACKEND_ORIGIN=same-origin
- # Same ceiling as `just -f apps/web/justfile build-dev`: vite's
- # chunk-render peaks at node's default ~4GB heap and aborts with
- # "Ineffective mark-compacts near heap limit".
- export NODE_OPTIONS=--max-old-space-size=6144
- bun run --bun build
- LANE
-
- # Instant lane: everything Fly-side that needs no build output.
- # The template fork must fire within seconds of the job starting —
- # its ~2 min server-side hydration then finishes inside the compose
- # build. (It previously sat after the premirror pushes, which put
- # it at the END of the build phase and re-gated the deploy.)
- # Entirely best-effort: everything in the fly lane (app create,
- # fork kick, token mint) has an authoritative fallback in the
- # deploy step, so a transient Fly API failure must not fail the
- # build. A child `bash -e` (not a `|| `-guarded function: errexit
- # is suppressed inside those) keeps its internal fail-fast so a
- # failed app_id fetch can't mint a garbage token.
- cat > "$lanes/fly.sh" <<'LANE'
- set -euo pipefail
- bash -euo pipefail "$(dirname "$0")/fly-prep.sh" \
- || echo "fly prep failed (non-fatal): the deploy step falls back" >&2
- LANE
-
- cat > "$lanes/fly-prep.sh" <<'LANE'
- flyctl status --app "$APP_NAME" >/dev/null 2>&1 \
- || flyctl apps create "$APP_NAME" --org "$FLY_ORG" \
- || flyctl status --app "$APP_NAME" >/dev/null
- # Kick the template-volume fork (see the deploy step's
- # volume-state block, which stays the authoritative fallback).
- if ! flyctl volumes list --app "$APP_NAME" --json 2>/dev/null \
- | jq -e 'map(select(.name == "docker_data")) | length > 0' >/dev/null; then
- tpl_src=$(flyctl volumes list --app "$TEMPLATE_APP" --json 2>/dev/null \
- | jq -r '[.[] | select((.name | startswith("tpl")) and .state == "created" and .attached_machine_id == null)]
- | sort_by(.created_at) | last | .id // empty' 2>/dev/null) || tpl_src=
- if [ -n "$tpl_src" ]; then
- curl -sf -X POST "https://api.machines.dev/v1/apps/$APP_NAME/volumes" \
- -H "Authorization: Bearer $FLY_API_TOKEN" \
- -H "Content-Type: application/json" \
- -d "{\"name\":\"docker_data\",\"region\":\"ewr\",\"source_volume_id\":\"$tpl_src\"}" \
- | jq -r '"early template fork: \(.id // "failed") (\(.state // "-"))"' \
- || echo "early template fork failed; deploy step will fall back" >&2
- fi
- fi
- # Mint the machine's app-scoped read-only registry pull token
- # (macaroon attenuation, see the deploy step for the why) so the
- # deploy preamble doesn't pay the round-trips serially.
- app_id=$(curl -sf https://api.fly.io/graphql \
- -H "Authorization: Bearer $FLY_API_TOKEN" \
- -H "content-type: application/json" \
- -d "{\"query\":\"{ app(name: \\\"$APP_NAME\\\") { internalNumericId } }\"}" \
- | jq -re '.data.app.internalNumericId')
- now=$(date +%s)
- printf '[{"type":"Apps","body":{"apps":{"%s":"r"}}},{"type":"ValidityWindow","body":{"not_before":%d,"not_after":%d}}]' \
- "$app_id" $((now - 60)) $((now + 604800)) \
- | flyctl tokens attenuate > "$RUNNER_TEMP/registry-pull-token.next"
- chmod 600 "$RUNNER_TEMP/registry-pull-token.next"
- mv "$RUNNER_TEMP/registry-pull-token.next" "$RUNNER_TEMP/registry-pull-token"
- LANE
-
- cat > "$lanes/images.sh" <<'LANE'
- set -euo pipefail
- docker compose --project-directory . -p macro -f docker/docker-compose.yml build \
- search sync_service websocket_service lexical_service ai_editing_worker analytics_proxy
- # Not a compose service, so `config --images` never lists it.
- # BuildKit cache on this runner is the freshness check.
- docker build --tag "$LOCAL_SANDBOX_IMAGE" crates/agent_harness/container
- # The image BUILD above is fatal; the premirror below is a pure
- # optimization with an authoritative fallback (the deploy step's
- # mirror loop), so its failure only costs a slower push later.
- bash -euo pipefail "$(dirname "$0")/premirror.sh" \
- || echo "premirror failed (non-fatal): the deploy step mirrors authoritatively" >&2
- LANE
-
- # Premirror: content-addressed refs, identical scheme to the deploy
- # step's mirror loop, so its manifest-existence check skips these.
- # Best effort by design — anything skipped here (images only the
- # bake produces, like macro-local-runtime:dev) is pushed there.
- # The fly lane created the app long ago; the guard is a fallback.
- cat > "$lanes/premirror.sh" <<'LANE'
- flyctl status --app "$APP_NAME" >/dev/null 2>&1 \
- || flyctl apps create "$APP_NAME" --org "$FLY_ORG" \
- || flyctl status --app "$APP_NAME" >/dev/null
- flyctl auth docker
- registry="registry.fly.io/$APP_NAME"
- for img in $(docker compose --project-directory . -p macro \
- -f docker/docker-compose.yml config --images | sort -u) alpine:3 "$LOCAL_SANDBOX_IMAGE"; do
- if ! docker image inspect "$img" >/dev/null 2>&1 && ! docker pull "$img"; then
- echo "premirror: skipping $img (not local, not pullable)"
- continue
- fi
- id=$(docker image inspect -f '{{.Id}}' "$img")
- ref="$registry:img-$(echo "$img" | tr '/:' '__')-$(echo "$id" | cut -c8-19)"
- docker tag "$img" "$ref"
- if ! docker manifest inspect "$ref" >/dev/null 2>&1; then
- for _ in 1 2 3; do
- docker push "$ref" && break
- echo "premirror push of $ref failed, retrying" >&2
- sleep 15
- done
- fi
- done
- LANE
-
- # A cold `stack up --infra-only` runs the real init (migrate,
- # kickstart, Kafka topics, indices) and saves the content-addressed
- # snapshot the VM restores from. Infra only: the app services need
- # the Doppler-sourced env to boot, which this runner deliberately
- # lacks — the snapshot captures only the infra volumes anyway. Via
- # `just` (not the bare `cargo x` alias) because the justfile
- # enables the `local-stack` feature the Kafka provisioning needs.
- # The store lives on the cache volume for a zero-copy hit; every
- # snapshot is also stored as one tar in Namespace artifact storage
- # keyed by the same hash, so cache-volume onboarding misses
- # download the exact snapshot instead of paying for a full init.
- # Artifact access is an optimization: failures fall back to the
- # volume/cold-bake behavior. The final `snapshot --json` line lands
- # in $RUNNER_TEMP for the stage step.
- cat > "$lanes/bake.sh" <<'LANE'
- set -euo pipefail
- export CARGO_TARGET_DIR="$PWD/target/bake"
- mkdir -p "$MACRO_STACK_SNAPSHOT_DIR"
- status=$(just stack snapshot --json | tail -n 1)
- key=$(echo "$status" | jq -r '.key')
- root=$(echo "$status" | jq -r '.root')
- artifact="macro-preview/init-snapshots/${key}.tar"
- archive="$RUNNER_TEMP/init-snapshot-${key}.tar"
- artifact_hit=
-
- # The Namespace cache-volume pool has an onboarding period where a
- # runner can receive an empty fork. Artifact storage is the durable
- # fallback: restore into a temporary directory and only publish it to
- # the live store after validating the embedded manifest through xtask.
- if ! echo "$status" | jq -e '.present' >/dev/null; then
- restored="$RUNNER_TEMP/restored-init-snapshot"
- rm -rf "$restored"
- mkdir -p "$restored"
- if nsc artifact download "$artifact" "$archive" \
- && tar -xf "$archive" -C "$restored" \
- && [ -f "$restored/$key/manifest.json" ]; then
- rm -rf "$root/$key"
- cp -a "$restored/$key" "$root/$key"
- status=$(just stack snapshot --json | tail -n 1)
- if echo "$status" | jq -e '.present' >/dev/null; then
- artifact_hit=1
- echo "init snapshot artifact hit: $status"
- else
- rm -rf "$root/$key"
- echo "downloaded init snapshot failed validation; baking cold" >&2
- fi
- else
- echo "init snapshot artifact miss; baking cold" >&2
- fi
- fi
-
- if echo "$status" | jq -e '.present' >/dev/null; then
- echo "init snapshot cache hit: $status"
- cargo run --quiet --manifest-path Cargo.toml \
- -p xtask_local --features local-stack -- gen-compose
- # The cold path builds the runtime image inside `stack up`; on a
- # hit nothing else does, and the registry mirror needs it in the
- # local daemon (its pull fallback only covers public images —
- # macro-local-runtime:dev is not on Docker Hub).
- cargo run --quiet --manifest-path Cargo.toml \
- -p xtask_local --features local-stack -- runtime-image
- else
- just stack up --infra-only --no-doppler --no-build
- status=$(just stack snapshot --json | tail -n 1)
- echo "$status" | jq -e '.present' >/dev/null
- fi
-
- # Seed artifact storage from either a volume hit or a cold bake. Avoid
- # uploading another version when the artifact already exists; a race
- # between concurrent PRs is harmless and must not fail the preview.
- if [ -z "$artifact_hit" ] && ! nsc artifact describe "$artifact" >/dev/null 2>&1; then
- snap_dir=$(echo "$status" | jq -r '.dir')
- tar -cf "$archive" -C "$root" "$(basename "$snap_dir")"
- nsc artifact upload "$archive" "$artifact" --expires_in 720h \
- || echo "warning: failed to upload init snapshot artifact" >&2
- fi
- echo "$status" > "$RUNNER_TEMP/preview-snapshot.json"
- LANE
-
- declare -a pids=() names=()
- for name in fly cargo web images bake; do
- # setsid: each lane leads its own process group so fail-fast
- # can kill the whole tree (cargo/docker/bun children), not just
- # the wrapper shell.
- setsid bash "$lanes/$name.sh" > "$lanes/$name.log" 2>&1 &
- pids+=($!)
- names+=("$name")
- done
- fail=
- for _ in "${pids[@]}"; do
- # Bare `wait -n`: any child; explicit pids would error once a
- # job has already been reaped.
- if ! wait -n; then
- fail=1
- break
- fi
- done
- if [ -n "$fail" ]; then
- # Negative pids: signal each lane's whole process group.
- kill -- "${pids[@]/#/-}" 2>/dev/null || true
- wait || true
- fi
- for i in "${!names[@]}"; do
- echo "::group::lane: ${names[$i]}"
- cat "$lanes/${names[$i]}.log" || true
- echo "::endgroup::"
- done
- if [ -n "$fail" ]; then
- echo "a build lane failed (first failure aborts the rest); see lane logs above" >&2
- exit 1
- fi
- env:
- FLY_ORG: ${{ vars.FLY_ORG || secrets.FLY_ORG }}
- LOCAL_SANDBOX_IMAGE: macro-agent-harness:latest
- - name: Dump stack diagnostics
- if: failure()
- run: |
- docker compose -p macro ps --all || true
- for svc in authentication-service proxy fusionauth postgres kafka \
- connection_gateway document_storage_service email_service \
- notification_service contacts_service; do
- echo "==================== $svc ===================="
- docker compose -p macro logs --no-color --tail 80 "$svc" 2>&1 || true
- done
- - name: Stage preview build context
- run: |
- set -euo pipefail
- # The dev-shell env (BASH_ENV) exports a Nix LD_LIBRARY_PATH; host
- # binaries like rsync then resolve Nix-store libs whose deps drag in
- # Nix glibc mid-process and crash on glibc symbol versions. This step
- # is pure file shuffling — drop it.
- unset LD_LIBRARY_PATH
- ctx=preview-ctx
- rm -rf "$ctx"
- mkdir -p "$ctx/repo" "$ctx/artifacts/binaries" "$ctx/bin"
- # The minimal repo layout xtask reads at runtime — including every
- # snapshot-key input, byte-identical to this checkout, so the VM's
- # key matches the snapshot baked above.
- rsync -a --relative --exclude node_modules \
- docker/docker-compose.yml \
- docker/docker-compose-databases.yml \
- infra/stacks/fusionauth-instance/docker-compose.yml \
- infra/local/nginx \
- infra/local/opensearch \
- infra/stacks/opensearch/helpers \
- crates/macro_db_client/migrations \
- "$ctx/repo/"
- # Service binaries only — the target dir also holds gigabytes of
- # build intermediates. `-p` everywhere: preserved mtimes mean files
- # cargo didn't relink produce byte-identical Docker layers, so the
- # registry push/pull skips them (the Dockerfile orders its COPYs
- # stable → volatile for the same reason).
- find target/x86_64-unknown-linux-gnu/debug \
- -maxdepth 1 -type f -executable ! -name '*.d' ! -name '*.so' \
- -exec cp -p {} "$ctx/artifacts/binaries/" \;
- cp -a apps/web/dist "$ctx/artifacts/frontend-dist"
- # Only the current key's snapshot — the volume store accumulates keys.
- snap_dir=$(jq -r '.dir' "$RUNNER_TEMP/preview-snapshot.json")
- mkdir -p "$ctx/artifacts/snapshots"
- cp -a "$snap_dir" "$ctx/artifacts/snapshots/"
- snapshot_key=$(jq -r '.key' "$RUNNER_TEMP/preview-snapshot.json")
- runtime_key=$(cd "$ctx/repo" \
- && find . -type f -print0 \
- | sort -z \
- | xargs -0 sha256sum \
- | sha256sum \
- | cut -d ' ' -f 1)
- frontend_key=$(cd "$ctx/artifacts/frontend-dist" \
- && find . -type f -print0 \
- | sort -z \
- | xargs -0 sha256sum \
- | sha256sum \
- | cut -d ' ' -f 1)
- jq -n \
- --arg snapshot_key "$snapshot_key" \
- --arg runtime_key "$runtime_key" \
- --arg frontend_key "$frontend_key" \
- --arg commit "${{ github.sha }}" \
- '{format: 1, snapshot_key: $snapshot_key, runtime_key: $runtime_key, frontend_key: $frontend_key, commit: $commit}' \
- > "$ctx/deployment.json"
- cp -p target/x86_64-unknown-linux-gnu/release/xtask_local "$ctx/bin/xtask"
- cp -p infra/preview/hot-update.sh "$ctx/bin/hot-update"
- cp -p infra/preview/entrypoint.sh "$ctx/entrypoint.sh"
- cp -p infra/preview/Dockerfile "$ctx/Dockerfile"
- cp -p infra/preview/update.Dockerfile "$ctx/update.Dockerfile"
- - name: Deploy to Fly
- run: |
- set -euo pipefail
- if [ -z "$DOPPLER_PREVIEW_TOKEN" ]; then
- echo "DOPPLER_PREVIEW_TOKEN secret is not set (a Doppler service token scoped to the preview config)" >&2
- exit 1
- fi
- if [ -z "$FLY_ORG" ]; then
- echo "FLY_ORG is not set (repo variable or secret with the Fly org slug)" >&2
- exit 1
- fi
-
- app_existed=
- if flyctl status --app "$APP_NAME" >/dev/null 2>&1; then
- app_existed=1
- else
- # A transient status error must not turn an existing app into a
- # hard create failure: create, then prove the app is reachable.
- flyctl apps create "$APP_NAME" --org "$FLY_ORG" \
- || flyctl status --app "$APP_NAME" >/dev/null
- fi
-
- # Seed a first boot's volume from the newest template volume — a
- # warm /var/lib/docker layer store published by a previous
- # successful deploy (see the "Publish template volume" step).
- # Measured cold, an empty volume costs ~11 min of image pulls at
- # boot. The images lane normally kicked the fork minutes ago (its
- # ~2 min server-side hydration overlaps the whole build phase);
- # this block is the authoritative fallback for a missing or
- # failed early fork. Cross-app forks need the Machines API:
- # flyctl resolves volume ids app-locally. The boot manifest check
- # makes the volume a pure cache, so every failure path here just
- # falls back to the old empty-volume behavior.
- vol_state=$(flyctl volumes list --app "$APP_NAME" --json 2>/dev/null \
- | jq -r '[.[] | select(.name == "docker_data")][0].state // empty') || vol_state=
- if [ -z "$vol_state" ]; then
- tpl_src=$(flyctl volumes list --app "$TEMPLATE_APP" --json 2>/dev/null \
- | jq -r '[.[] | select((.name | startswith("tpl")) and .state == "created" and .attached_machine_id == null)]
- | sort_by(.created_at) | last | .id // empty' 2>/dev/null) || tpl_src=
- if [ -n "$tpl_src" ]; then
- vol_state=$(curl -sf -X POST "https://api.machines.dev/v1/apps/$APP_NAME/volumes" \
- -H "Authorization: Bearer $FLY_API_TOKEN" \
- -H "Content-Type: application/json" \
- -d "{\"name\":\"docker_data\",\"region\":\"ewr\",\"source_volume_id\":\"$tpl_src\"}" \
- | jq -r '.state // empty') || vol_state=
- if [ -n "$vol_state" ]; then
- echo "seeding docker_data from template volume $tpl_src (state: $vol_state)"
- else
- echo "template fork failed; falling back to an empty volume" >&2
- fi
- else
- echo "no template volume available; first boot will pull cold"
- fi
- fi
-
- # A read-only, app-scoped, time-boxed pull token so the machine's
- # inner dockerd can pull the mirrored stack images from this app's
- # registry repo — and nothing else (PR code can read machine
- # secrets, so scope matters). The org deploy token can't mint new
- # tokens (createLimitedAccessToken is not authorized), but macaroon
- # attenuation is pure client-side crypto: append an Apps caveat
- # (numeric app id, mask "r") and a validity window to our own
- # token. The images lane normally minted it already; fall back to
- # minting inline when its file is absent.
- if [ -s "$RUNNER_TEMP/registry-pull-token" ]; then
- pull_token=$(cat "$RUNNER_TEMP/registry-pull-token")
- else
- app_id=$(curl -sf https://api.fly.io/graphql \
- -H "Authorization: Bearer $FLY_API_TOKEN" \
- -H "content-type: application/json" \
- -d "{\"query\":\"{ app(name: \\\"$APP_NAME\\\") { internalNumericId } }\"}" \
- | jq -re '.data.app.internalNumericId')
- now=$(date +%s)
- pull_token=$(printf '[{"type":"Apps","body":{"apps":{"%s":"r"}}},{"type":"ValidityWindow","body":{"not_before":%d,"not_after":%d}}]' \
- "$app_id" $((now - 60)) $((now + 604800)) | flyctl tokens attenuate)
- fi
-
- snapshot_key=$(jq -r '.key' "$RUNNER_TEMP/preview-snapshot.json")
- runtime_key=$(jq -r '.runtime_key' preview-ctx/deployment.json)
- mode=bootstrap
- machine_id=
- if [ -n "$app_existed" ]; then
- mode=rehydrate
- machine_id=$(flyctl machine list --app "$APP_NAME" --json \
- | jq -r '[.[] | select((.config.mounts // []) | length > 0)][0].id // empty')
- if [ -n "$machine_id" ]; then
- # A suspended preview needs demand before SSH is reachable.
- curl -s -o /dev/null --max-time 10 "https://$APP_NAME.fly.dev/" || true
- for _ in $(seq 1 12); do
- # `--command` is exec'd directly, NOT run through a shell —
- # a bare `if [ -f ...` dies with `exec: "if": not found`, so
- # the marker never parsed and every push silently rehydrated
- # (~7 min) instead of going hot (~3 min). Hence the sh -c.
- marker=$(flyctl ssh console --app "$APP_NAME" --machine "$machine_id" \
- --quiet --command 'sh -c "if [ -f /var/lib/docker/.macro-preview/deployment.json ]; then cat /var/lib/docker/.macro-preview/deployment.json; else echo __NO_MARKER__; fi"' \
- 2>/dev/null || true)
- if echo "$marker" | jq -e \
- --arg key "$snapshot_key" \
- --arg runtime "$runtime_key" \
- '.format == 1 and .snapshot_key == $key and .runtime_key == $runtime and (.frontend_key | type == "string")' \
- >/dev/null 2>&1; then
- mode=hot
- break
- fi
- # SSH succeeded and returned a marker (or proved it absent):
- # incompatibility is real, not a wake-up race.
- if [ "$marker" = __NO_MARKER__ ] \
- || echo "$marker" | jq -e '.format != null' >/dev/null 2>&1; then
- break
- fi
- sleep 5
- done
- fi
- fi
- echo "preview deployment mode: $mode (snapshot $snapshot_key)"
-
- # Keep the machine in demand for the WHOLE deploy, hot path
- # included: fly-proxy suspends a machine ~7 min after its last
- # proxied request, and SSH via hallpass is not proxy traffic — a
- # hot update with no browser traffic got its VM suspended
- # mid-session (the frozen `flyctl ssh console` then ate 52 min of
- # runner until the job timeout). Any response status counts as
- # traffic, and a request also resumes a suspended machine.
- (while :; do
- curl -s -o /dev/null --max-time 10 "https://$APP_NAME.fly.dev/" || true
- sleep 15
- done) &
- keepalive=$!
- trap 'kill "$keepalive" 2>/dev/null || true' EXIT
-
- flyctl auth docker
- registry="registry.fly.io/$APP_NAME"
- # Mirror every stack image into the app's registry repo instead of
- # baking docker-save tars into the VM image (which made every
- # deploy re-ship ~5GB the machine's volume already had). Pushes
- # dedup at the layer level against the repo, so a redeploy only
- # uploads layers that actually changed; the entrypoint pulls with
- # the same dedup against the machine's persistent layer store. The
- # manifest maps image ID -> local tag -> registry ref so boots can
- # skip images already present. alpine:3 rides along for the
- # snapshot-restore helper container.
- # On a snapshot cache hit only gen-compose ran, which doesn't write
- # the env file; compose hard-fails on a missing --env-file, and the
- # image names don't interpolate env vars anyway.
- envfile=infra/local/generated/macro/local.generated.env
- [ -f "$envfile" ] || : > "$envfile"
- images=$(docker compose --project-directory . -p macro \
- -f docker/docker-compose.yml \
- -f infra/local/generated/macro/docker-compose.override.yml \
- --env-file "$envfile" \
- config --images | sort -u)
- echo "mirroring images:" $images
- docker image inspect alpine:3 >/dev/null 2>&1 || docker pull alpine:3
- mkdir -p preview-ctx/preload
- : > preview-ctx/preload/manifest.txt
- for img in $images alpine:3 "$LOCAL_SANDBOX_IMAGE"; do
- # Images the bake didn't leave in the daemon (snapshot cache hit,
- # or app-layer-only images like proxy/mailpit) get pulled here.
- # Compose `config --images` also lists locally-built tags
- # (analytics_proxy, sdk-webhook-relay) that are not on Docker Hub;
- # premirror already skips those — do the same so a missing local
- # tag cannot fail the deploy.
- if ! docker image inspect "$img" >/dev/null 2>&1 && ! docker pull "$img"; then
- echo "mirror: skipping $img (not local, not pullable)"
- continue
- fi
- id=$(docker image inspect -f '{{.Id}}' "$img")
- # Content-addressed tag: same image content = same ref, so a
- # redeploy's push is skippable by a manifest existence check and
- # concurrent PRs' deploys can never clobber each other's tags.
- ref="$registry:img-$(echo "$img" | tr '/:' '__')-$(echo "$id" | cut -c8-19)"
- docker tag "$img" "$ref"
- echo "$id $img $ref" >> preview-ctx/preload/manifest.txt
- done
- # The existence probes are pure network round-trips (and after the
- # premirror lane, almost all of them hit) — run them in parallel
- # instead of ~1s each serially.
- awk '{print $3}' preview-ctx/preload/manifest.txt \
- | xargs -P 8 -I {} sh -c \
- 'docker manifest inspect "{}" >/dev/null 2>&1 || echo "{}"' \
- > "$RUNNER_TEMP/push-refs.txt"
- echo "pushing $(wc -l < "$RUNNER_TEMP/push-refs.txt") of $(wc -l < preview-ctx/preload/manifest.txt) images"
- # The Fly registry occasionally aborts large uploads ("s3aws:
- # append to zero-size path unsupported") — retry; layers that made
- # it are reused. Four pushes at a time keeps the first (cold) push
- # moving without saturating the registry.
- if [ -s "$RUNNER_TEMP/push-refs.txt" ]; then
- xargs -P 4 -I {} sh -c '
- for _ in 1 2 3; do
- docker push "{}" && exit 0
- echo "push of {} failed, retrying" >&2
- sleep 15
- done
- exit 1' < "$RUNNER_TEMP/push-refs.txt"
- fi
-
- push_image() {
- image_to_push=$1
- for _ in 1 2 3; do
- if docker push "$image_to_push"; then return 0; fi
- echo "docker push failed for $image_to_push, retrying" >&2
- sleep 15
- done
- return 1
- }
-
- if [ "$mode" = hot ]; then
- update_image="$registry:update-${{ github.sha }}"
- docker build -f preview-ctx/update.Dockerfile -t "$update_image" preview-ctx
- push_image "$update_image"
- token_file="$RUNNER_TEMP/registry-pull-token"
- printf '%s' "$pull_token" > "$token_file"
- chmod 600 "$token_file"
- hot_rc=0
- flyctl ssh sftp put "$token_file" /tmp/macro-registry-token \
- --app "$APP_NAME" --machine "$machine_id" --mode 0600 \
- || hot_rc=$?
- if [ "$hot_rc" = 0 ]; then
- # timeout backstop: a suspended-mid-session VM freezes the ssh
- # client forever (measured: 52 min of hang). 10 min is triple
- # a normal hot apply; on expiry we fall back to rehydrate.
- hot_command="/srv/macro/bin/hot-update '$update_image' /tmp/macro-registry-token"
- timeout 600 flyctl ssh console --app "$APP_NAME" --machine "$machine_id" \
- --command "$hot_command" || hot_rc=$?
- fi
- if [ "$hot_rc" = 0 ]; then
- echo "hot update completed without restarting the Fly machine"
- exit 0
- fi
- echo "hot update failed (exit $hot_rc); falling back to full rehydrate" >&2
- mode=rehydrate
- fi
-
- # Bootstrap/rehydrate refreshes app secrets and reconciles the
- # persistent Docker volume before replacing the machine image.
- flyctl secrets set --app "$APP_NAME" --stage \
- "DOPPLER_TOKEN=$DOPPLER_PREVIEW_TOKEN" \
- "REGISTRY_PULL_TOKEN=$pull_token"
- if [ -z "$vol_state" ]; then
- # No volume and no fork landed (no template, or both fork
- # attempts failed): the old empty-volume path.
- flyctl volumes create docker_data --app "$APP_NAME" --region ewr --size 40 --yes
- elif [ "$vol_state" != created ]; then
- # A fork is still hydrating (usually already finished: the
- # images lane kicked it before the cargo lane even ended). The
- # volume must reach "created" before a machine can mount it; a
- # fork that never lands is replaced by an empty volume rather
- # than failing the deploy.
- for _ in $(seq 1 60); do
- vol_state=$(flyctl volumes list --app "$APP_NAME" --json \
- | jq -r '[.[] | select(.name == "docker_data")][0].state // empty')
- [ "$vol_state" = created ] && break
- sleep 10
- done
- echo "seeded volume state: $vol_state"
- if [ "$vol_state" != created ]; then
- echo "template fork stuck (state: $vol_state); replacing with an empty volume" >&2
- vol_id=$(flyctl volumes list --app "$APP_NAME" --json \
- | jq -r '[.[] | select(.name == "docker_data")][0].id // empty')
- [ -z "$vol_id" ] || flyctl volumes destroy "$vol_id" --app "$APP_NAME" --yes || true
- flyctl volumes create docker_data --app "$APP_NAME" --region ewr --size 40 --yes
- fi
- fi
- # Machines created before the volume existed can't take the mounts
- # config update. A destroyed/partial machine is recreated by deploy.
- flyctl machine list --app "$APP_NAME" --json \
- | jq -r '.[] | select((.config.mounts // []) | length == 0) | .id' \
- | while read -r id; do
- [ -z "$id" ] || flyctl machine destroy "$id" --app "$APP_NAME" --force || true
- done
-
- image="$registry:${{ github.sha }}"
- docker build -t "$image" preview-ctx
- push_image "$image"
- # First boot does real work before 8090 opens (pulling the stack
- # images, snapshot restore, compose up, FusionAuth's JVM) —
- # give the health check more runway than flyctl's default wait.
- # The step-wide keepalive above holds the machine out of suspend.
- flyctl deploy --app "$APP_NAME" \
- --config infra/preview/fly.toml \
- --image "$image" \
- --wait-timeout 1800 \
- --yes
- env:
- FLY_ORG: ${{ vars.FLY_ORG || secrets.FLY_ORG }}
- DOPPLER_PREVIEW_TOKEN: ${{ secrets.DOPPLER_PREVIEW_TOKEN }}
- LOCAL_SANDBOX_IMAGE: macro-agent-harness:latest
- - name: Dump Fly diagnostics
- if: failure()
- run: |
- flyctl machine list --app "$APP_NAME" || true
- # `flyctl logs --no-tail` can hang forever on its NATS fetch
- # (observed: a healthy deploy stuck 1h53m in the timings dump) —
- # never call it without a timeout.
- timeout 60 flyctl logs --app "$APP_NAME" --no-tail || true
- - name: Dump boot timings
- run: |
- timeout 120 flyctl ssh console --app "$APP_NAME" --quiet \
- --command 'cat /var/lib/docker/.macro-preview/boot-timings.log' || true
- - name: Publish template volume
- run: |
- set -euo pipefail
- key=$(awk '{print $1, $2}' preview-ctx/preload/manifest.txt \
- | LC_ALL=C sort | sha256sum | cut -c1-24)
- vol_name="tpl$key"
- flyctl status --app "$TEMPLATE_APP" >/dev/null 2>&1 \
- || flyctl apps create "$TEMPLATE_APP" --org "$FLY_ORG"
- vols=$(flyctl volumes list --app "$TEMPLATE_APP" --json 2>/dev/null || echo '[]')
- if echo "$vols" | jq -e --arg n "$vol_name" \
- 'map(select(.name == $n)) | length > 0' >/dev/null; then
- echo "template $vol_name already exists; image set unchanged"
- else
- src=$(flyctl volumes list --app "$APP_NAME" --json \
- | jq -r '[.[] | select(.name == "docker_data")][0].id // empty')
- if [ -n "$src" ]; then
- # Fire-and-forget: hydration is a server-side block copy;
- # nothing in this run depends on it finishing.
- curl -sf -X POST "https://api.machines.dev/v1/apps/$TEMPLATE_APP/volumes" \
- -H "Authorization: Bearer $FLY_API_TOKEN" \
- -H "Content-Type: application/json" \
- -d "{\"name\":\"$vol_name\",\"region\":\"ewr\",\"source_volume_id\":\"$src\"}" \
- | jq -r '"published \(.name) (\(.id), state: \(.state))"' \
- || echo "template publish failed (non-fatal)" >&2
- fi
- fi
- # Prune superseded templates: keep the 3 newest, and never touch
- # anything younger than 2h — it may still be hydrating or serving
- # as a concurrent deploy's fork source.
- cutoff=$(date -u -d '2 hours ago' +%Y-%m-%dT%H:%M:%SZ)
- echo "$vols" | jq -r --arg cutoff "$cutoff" \
- '[.[] | select((.name | startswith("tpl")) and .attached_machine_id == null)]
- | sort_by(.created_at) | reverse | .[3:]
- | .[] | select(.created_at < $cutoff) | .id' \
- | while read -r old; do
- [ -z "$old" ] || flyctl volumes destroy "$old" --app "$TEMPLATE_APP" --yes || true
- done
- env:
- FLY_ORG: ${{ vars.FLY_ORG || secrets.FLY_ORG }}
- continue-on-error: true
- - name: Comment preview URL
- uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b
- with:
- github-token: ${{ secrets.GITHUB_TOKEN }}
- script: |
- const app = process.env.APP_NAME;
- const url = `https://${app}.fly.dev`;
- const marker = '<!-- fly-preview -->';
- const body = [
- marker,
- `🚀 **Full-stack preview**: ${url}`,
- '',
- `- Suspended when idle — the first request wakes it (a few seconds).`,
- `- Log in with any email; the passwordless code lands in [Mailpit](${url}/mailpit/).`,
- `- Deployed from ${context.payload.pull_request.head.sha.slice(0, 7)}.`,
- ].join('\n');
- const { data: comments } = await github.rest.issues.listComments({
- owner: context.repo.owner,
- repo: context.repo.repo,
- issue_number: context.issue.number,
- per_page: 100,
- });
- const existing = comments.find(c => c.body && c.body.startsWith(marker));
- if (existing) {
- await github.rest.issues.updateComment({
- owner: context.repo.owner,
- repo: context.repo.repo,
- comment_id: existing.id,
- body,
- });
- } else {
- await github.rest.issues.createComment({
- owner: context.repo.owner,
- repo: context.repo.repo,
- issue_number: context.issue.number,
- body,
- });
- }
- - name: Teardown Nix
- if: always()
- uses: './.github/actions/teardown-nix'
-concurrency:
- group: fly-preview-${{ github.event.pull_request.number }}
- cancel-in-progress: true
diff --git a/CLAUDE.md b/CLAUDE.md
index 2e3aa1412b..4152c6f537 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -317,7 +317,7 @@ don't inject it directly into the error message.
## Cursor Cloud specific instructions
-`.cursor/install.sh` prepares the durable caches, databases, test dependencies, frontend dependencies, service binaries, and stack init snapshot, then stops dockerd and nix-daemon so the Cloud bake can exit. `.cursor/start.sh` runs at boot and starts nothing beyond the nix daemon, so sessions and subagents are usable immediately. `.cursor/infra.sh` starts Docker, Postgres, and Redis on demand. `.cursor/stack.sh` starts the on-demand product stack. `.cursor/rebuild.sh` rebuilds the stack after backend edits. These scripts are the only supported entry points; each re-enters the pinned nix shell itself, so run them with plain `bash` from any environment.
+`.cursor/install.sh` prepares the durable caches, databases, test dependencies, frontend dependencies, service binaries, and stack init snapshot, then stops dockerd and nix-daemon so the Cloud bake can exit. `.cursor/start.sh` runs at boot and starts nothing beyond the nix daemon, so sessions and subagents are usable immediately. `.cursor/infra.sh` starts Docker, Postgres, and Redis on demand. `.cursor/stack.sh` starts the on-demand product: backend containers behind the proxy (8090) plus the hot-reloading frontend dev server — the app is at http://localhost:3000/app and frontend edits apply on save (idempotent; a healthy stack is left alone). `.cursor/rebuild.sh` remounts new backend binaries after Rust edits. These scripts are the only supported entry points; each re-enters the pinned nix shell itself, so run them with plain `bash` from any environment. To run the app and see your edits, follow the `run-app` skill (`.claude/skills/run-app/SKILL.md`).
Nix is the only host dependency. The pinned dev shell supplies the Docker CLI and daemon, Compose, `fuse-overlayfs`, and OpenSSH. `ssh-keygen` must come from this shell.
@@ -329,8 +329,8 @@ Set `DOPPLER_TOKEN` as a Cursor environment **runtime** secret: a Doppler servic
Nothing runs after boot. Before DB-backed `cargo test -p <crate>`, run `bash .cursor/infra.sh` once — it brings up Docker, Postgres, and Redis in seconds because install baked the images and volumes. Pure-logic crate tests need nothing. Run `bash .cursor/stack.sh` for a product-ready environment.
-After backend edits, run `bash .cursor/rebuild.sh`. Do not use `just stack update` with the read-only Nix binary directory.
+After backend edits, run `bash .cursor/rebuild.sh`. That nix-builds the stack binaries and runs `just stack update --binaries-dir`, which remounts them without wiping volumes.
-No seeding or OTP is needed to log in: passwordless login auto-creates a user for any email, and the stack's auth service is built with `return_passwordless_code`, so the login API returns the code in its response (codes are also visible at the proxy's `/mailpit/` UI). `just seed-scenario apply --file seed/scenarios/team-perms.json` is optional, for multi-user team/permission fixtures. The `agent_harness_service` restart loop is expected when AI provider keys are missing; with `DOPPLER_TOKEN` those keys come from `local`/`lcl_preview`.
+No seeding or OTP is needed to log in: passwordless login auto-creates a user for any email, and the stack's auth service is built with `return_passwordless_code`, so the login API returns the code in its response (codes are also visible in Mailpit at http://localhost:8025). `just seed-scenario apply --file seed/scenarios/team-perms.json` is optional, for multi-user team/permission fixtures. The `agent_harness_service` restart loop is expected when AI provider keys are missing; with `DOPPLER_TOKEN` those keys come from `local`/`lcl_preview`.
Leave `SQLX_OFFLINE` unset for `cargo test`. If SQLx reports missing cached query data, run `just prepare_db` instead of enabling offline mode. Run crate tests from the repository root with `cargo test -p <crate>`.
diff --git a/apps/web/scripts/keep-import-meta-dev.test.ts b/apps/web/scripts/keep-import-meta-dev.test.ts
index 600f6835e5..b5c191a57a 100644
--- a/apps/web/scripts/keep-import-meta-dev.test.ts
+++ b/apps/web/scripts/keep-import-meta-dev.test.ts
@@ -2,7 +2,7 @@ import { describe, expect, it } from 'vitest';
import { keepImportMetaDev } from './keep-import-meta-dev';
describe('keepImportMetaDev', () => {
- it('keeps DEV for local-backend static builds (stack up, Fly preview)', () => {
+ it('keeps DEV for local-backend static builds (stack up)', () => {
expect(
keepImportMetaDev({
command: 'build',
diff --git a/apps/web/scripts/keep-import-meta-dev.ts b/apps/web/scripts/keep-import-meta-dev.ts
index 3bfd74bc93..05b7670853 100644
--- a/apps/web/scripts/keep-import-meta-dev.ts
+++ b/apps/web/scripts/keep-import-meta-dev.ts
@@ -3,7 +3,7 @@
*
* Vite compiles DEV from NODE_ENV, not MODE, so
* `MODE=development NODE_ENV=production vite build` would otherwise ship
- * DEV=false. Local-backend bundles (`just stack up`, Fly preview) already set
+ * DEV=false. Local-backend bundles (`just stack up`) already set
* `VITE_LOCAL_BACKEND_ORIGIN`; hosted `just build-dev` / staging / prod do not.
*
* `vite serve` already has DEV=true, so this only applies to `build`.
diff --git a/apps/web/vite.base.ts b/apps/web/vite.base.ts
index 1f57061906..06b6be950b 100644
--- a/apps/web/vite.base.ts
+++ b/apps/web/vite.base.ts
@@ -231,7 +231,7 @@ function getAssetsPath(mode: string, command: string): string {
function defineEnv(mode: string, command: string) {
// `vite build` compiles DEV from NODE_ENV, not MODE. Local-backend static
- // bundles already set VITE_LOCAL_BACKEND_ORIGIN (stack up, Fly preview);
+ // bundles already set VITE_LOCAL_BACKEND_ORIGIN (stack up);
// keep DEV so those artifacts match `just run_local` (vite serve). Hosted
// `just build-dev` does not set the origin, so DEV stays false.
const keepDev = keepImportMetaDev({
diff --git a/docs/RUNNING_LOCALLY.md b/docs/RUNNING_LOCALLY.md
index 85e35e7104..db2f6dfd74 100644
--- a/docs/RUNNING_LOCALLY.md
+++ b/docs/RUNNING_LOCALLY.md
@@ -256,12 +256,13 @@ just stack up # bring everything up, print URLs, return
just stack status --json # machine-readable state (containers, health, URLs)
just stack update # rebuild and reload only the changed services (the `r` hotkey)
just stack update --frontend # also rebuild the frontend bundle
+just stack update --binaries-dir <dir> # remount a prebuilt set; volumes stay
just stack down # remove containers, volumes, and state
```
-All the `run_local` flags apply to `stack` too. This includes `--instance`, `--no-doppler`, `--no-build`, and `--binaries-dir`. CI can pass a prebuilt bundle with `--frontend-dist`.
+All the `run_local` flags apply to `stack` too. This includes `--instance`, `--no-doppler`, `--no-build`, and `--binaries-dir`.
-The app is served at `<proxy>/app/`. The bundle resolves its backend from the origin it is served on. The same stack works on localhost or behind a preview hostname without a rebuild.
+The app is served at `<proxy>/app/`. The bundle resolves its backend from the origin it is served on. The same stack works on localhost or behind any hostname without a rebuild.
### Init Snapshots
@@ -276,11 +277,10 @@ It saves these volumes as an init snapshot. The snapshot is content-addressed an
Useful commands:
```bash
-just stack snapshot # show the current snapshot key
just stack up --no-snapshot # skip the snapshot cache
```
-CI bakes the snapshot into the preview image. This is what makes Fly previews boot fast. See `infra/preview/README.md`.
+Cursor Cloud bakes the snapshot during environment install. Later `stack up` restores it.
## Common Commands
diff --git a/infra/preview/Dockerfile b/infra/preview/Dockerfile
deleted file mode 100644
index d131641c65..0000000000
--- a/infra/preview/Dockerfile
+++ /dev/null
@@ -1,59 +0,0 @@
-# The Fly preview machine image: the entire local stack inside one VM.
-#
-# Built by .github/workflows/preview-fly.yml from a STAGED context (not the
-# repo root) — CI lays out everything the machine needs and this file just
-# copies it in:
-#
-# repo/ minimal repo layout xtask reads (compose files, migrations,
-# index helpers) — MACRO_REPO_ROOT points here
-# artifacts/ binaries/ (zigbuild output), frontend-dist/ (same-origin
-# bundle), snapshots/ (baked init snapshot)
-# bin/ prebuilt orchestrator + the in-place hot-update driver
-# preload/ manifest of every image the stack runs (image ID -> local tag
-# -> registry ref); the entrypoint pulls them from the app's Fly
-# registry repo at boot, skipping whatever the machine's
-# persistent layer store already has
-#
-# Fly machines are full microVMs, so an inner Docker daemon runs the same
-# compose topology as `just run_local` — the preview IS the local stack.
-FROM ubuntu:24.04
-
-RUN apt-get update && apt-get install -y --no-install-recommends \
- docker.io \
- docker-compose-v2 \
- docker-buildx \
- ca-certificates \
- curl \
- gnupg \
- jq \
- && rm -rf /var/lib/apt/lists/*
-
-# Doppler CLI: the machine boots with a config-scoped service token
-# (DOPPLER_TOKEN, set as a Fly secret by the workflow) and the stack's env
-# layer pulls the preview secrets with it.
-RUN curl -Ls --tlsv1.2 --proto "=https" https://cli.doppler.com/install.sh | sh
-
-# One COPY per artifact, ordered stable → volatile: the staging step preserves
-# mtimes, so a layer whose inputs didn't change this deploy is byte-identical
-# to the previous image's — the registry push and the Fly host's pull both
-# skip it. Everything after the first changed layer re-ships, so the pieces
-# that churn every deploy (frontend bundle, service binaries) go last.
-COPY entrypoint.sh /srv/macro/entrypoint.sh
-COPY deployment.json /srv/macro/deployment.json
-COPY repo/ /srv/macro/repo/
-COPY bin/ /srv/macro/bin/
-COPY artifacts/snapshots/ /srv/macro/artifacts/snapshots/
-COPY preload/ /srv/macro/preload/
-COPY artifacts/frontend-dist/ /srv/macro/artifacts/frontend-dist/
-COPY artifacts/binaries/ /srv/macro/artifacts/binaries/
-RUN chmod +x /srv/macro/entrypoint.sh /srv/macro/bin/xtask /srv/macro/bin/hot-update
-
-# Point the prebuilt xtask at the staged layout (its compiled-in paths refer to
-# the CI checkout) and at the baked snapshot store.
-ENV MACRO_REPO_ROOT=/srv/macro/repo \
- MACRO_STACK_SNAPSHOT_DIR=/srv/macro/artifacts/snapshots
-
-# The instance's single-origin proxy (fly.toml's internal_port).
-EXPOSE 8090
-
-ENTRYPOINT ["/srv/macro/entrypoint.sh"]
diff --git a/infra/preview/README.md b/infra/preview/README.md
deleted file mode 100644
index f195a94d7e..0000000000
--- a/infra/preview/README.md
+++ /dev/null
@@ -1,105 +0,0 @@
-# Full-stack PR previews on Fly
-
-Label a PR `preview` and CI deploys the **entire stack** — every service,
-Postgres, OpenSearch, FusionAuth, LocalStack, the frontend — as one Fly app at
-`https://macro-pr-<N>.fly.dev`. The app suspends when idle (≈zero cost) and the
-fly-proxy wakes it on the next request. Closing the PR (or removing the label)
-destroys it.
-
-## How it works
-
-The preview is not a parallel deployment system: it is the local stack
-(`just stack up`, see `RUNNING_LOCALLY.md`) running inside one Fly machine.
-A Fly machine is a full microVM, so an inner Docker daemon runs the exact
-compose topology `run_local` uses. What differs is that **everything expensive
-is baked in CI** and the machine only restores:
-
-| Baked in CI | Restored on the machine |
-|---|---|
-| Service binaries (`cargo x zigbuild`) | bind-mounted into the runtime image |
-| Frontend bundle (`same-origin` build) | served statically by the instance Caddy |
-| Init snapshot (migrations + FusionAuth kickstart + search indices) | volumes restored, init skipped |
-| Every Docker image, mirrored to the app's Fly registry repo | `docker pull` at boot — layer-dedup'd against the machine's persistent layer store, nothing built |
-
-Boot on a fresh machine ≈ image pulls + snapshot restore + JVM startup
-(a couple of minutes); wake from suspend ≈ seconds. Pushes to the PR choose one
-of three paths from the durable deployment marker on the Docker volume:
-
-- **bootstrap** — no app/machine yet: create everything and restore the
- snapshot. The new volume is forked (Machines API `source_volume_id`) from
- the newest **template volume** on `macro-preview-template` — a warm
- `/var/lib/docker` layer store published by a previous successful deploy —
- so the first boot pulls only image deltas instead of ~6GB cold (measured:
- ~11 min of a ~19 min cold boot was image pulls). Templates are
- content-addressed by the preload manifest's (image id, tag) pairs: after
- any successful deploy, if no template exists for the current image set,
- the deploy publishes its own volume as one (`tpl<hash>`), and old
- templates are pruned. The template volume is a pure cache — the boot-time
- manifest check pulls whatever a stale or missing template lacks, and every
- fork failure falls back to the old empty-volume path;
-- **hot update** — the snapshot key, staged-runtime hash, and update protocol
- are unchanged: pull a layer-deduplicated artifact carrier, atomically replace
- changed binaries/frontend, and restart only affected containers without
- restarting the Fly machine or infra;
-- **rehydrate** — compatibility changed, or a hot update fails: fall back to
- the immutable full-image deploy and snapshot restore.
-
-The URL remains stable across all three paths.
-
-The whole product sits behind the stack's single-origin Caddy proxy
-(`:8090`), so one `internal_port` covers the frontend, APIs, WebSockets, and
-Mailpit.
-
-## Logging in
-
-Login is passwordless (email codes) and preview email is captured, not sent:
-enter any address on the login screen, then open
-`https://macro-pr-<N>.fly.dev/mailpit/` to read the code. (Routing login codes
-through scoped SES so teammates get real emails is a planned follow-up.)
-
-## Secrets (Doppler)
-
-The stack's local plumbing (DBs, queues, auth identity) is code-owned and
-dummy, but integration secrets (AI keys etc.) come from Doppler, exactly like
-`run_local`. The workflow stages a **config-scoped service token** onto each
-preview app as the Fly secret `DOPPLER_TOKEN`; at boot the env layer detects
-the token and pulls that config (no `doppler login` involved), then overlays
-the code-owned `LocalEnv` on top — so Doppler supplies integration keys while
-local plumbing stays local. A preview machine can only ever read the one
-config its token is scoped to.
-
-## One-time setup
-
-1. Create a Fly org deploy token: `fly tokens create org -o <org>` →
- repo secret `FLY_API_TOKEN`.
-2. Set the repo variable `FLY_ORG` to the org slug.
-3. In Doppler, create a preview config in the `local` project (e.g.
- `lcl_preview` — start from `lcl_personal`, keep only what previews should
- have) and a service token for it → repo secret `DOPPLER_PREVIEW_TOKEN`.
-4. Create the `preview` label.
-
-## Security posture
-
-- Previews only build from same-repo branches (the workflow refuses forks).
-- No real AWS: S3/SQS/DynamoDB are LocalStack, email is Mailpit. The
- cloud-reachable credentials on a machine are the preview Doppler config's
- contents — scope that config accordingly (assume PR code can read it) — and
- `REGISTRY_PULL_TOKEN`, the CI deploy token attenuated (client-side macaroon
- caveats) to read-only on this one preview app with a 7-day validity window:
- it can pull the app's registry repo, nothing else.
-- URLs are public. Anyone with the link can use the preview (and read its
- Mailpit). Don't put sensitive data in one. Edge auth (oauth2-proxy in front
- of Caddy) is a planned hardening step.
-
-## Known costs & future optimizations
-
-- The stack images are mirrored into the per-PR app's registry repo, so the
- first deploy of a PR pushes the full set; after that pushes move only
- changed layers. The **pull** side of that cost is gone (template volume
- forks, above), but there is still no cross-PR sharing on the push side —
- a shared registry (GHCR/Namespace) would fix that at the cost of new
- org-level credentials.
-- The aux images (sync/websocket/lexical) rebuild in CI on every preview
- deploy. Runner-level Docker layer caching covers most of it.
-- `cpus = 8, memory = 16gb` (shared) is a deliberate over-provision; measure
- and shrink once a few previews have run.
diff --git a/infra/preview/entrypoint.sh b/infra/preview/entrypoint.sh
deleted file mode 100644
index 8234216251..0000000000
--- a/infra/preview/entrypoint.sh
+++ /dev/null
@@ -1,129 +0,0 @@
-#!/usr/bin/env bash
-# Boot for the Fly preview machine: start the inner Docker daemon, pull the
-# stack images from the app's registry repo, bring the stack up from the baked
-# artifacts, then idle while the containers serve. Everything expensive
-# (compile, bundle, init) happened in CI — this script only restores and
-# starts.
-set -euo pipefail
-
-state_dir=/var/lib/docker/.macro-preview
-mkdir -p "$state_dir"
-# Fly's log stream is short-retention and `flyctl logs --no-tail` can hang
-# forever, so the boot breakdown also lands in a volume-backed file that CI
-# reads back with a plain `ssh cat`. Truncated here, not appended: the file
-# always describes the most recent real boot.
-timings="$state_dir/boot-timings.log"
-: > "$timings"
-log() { echo "[preview] $*" | tee -a "$timings"; }
-
-t_boot=$(date +%s)
-# A hard loss during a hot update cannot run its EXIT trap. The full immutable
-# boot is the recovery boundary, so clear any abandoned volume-backed lock.
-rmdir "$state_dir/update.lock" 2>/dev/null || true
-log "starting inner dockerd"
-dockerd >/var/log/dockerd.log 2>&1 &
-
-for _ in $(seq 1 120); do
- docker info >/dev/null 2>&1 && break
- sleep 1
-done
-if ! docker info >/dev/null 2>&1; then
- log "dockerd failed to start:"
- tail -n 50 /var/log/dockerd.log
- exit 1
-fi
-log "dockerd ready in $(($(date +%s) - t_boot))s"
-
-# A template-forked volume carries the source deploy's containers; any with a
-# restart policy auto-start with dockerd and can answer :8090 with the OLD
-# stack — fly-proxy then reports the machine healthy before the real stack
-# exists, and a visitor gets the stale app followed by a teardown blackout.
-# Kill them before anything can serve; `stack up` recreates properly.
-stale=$(docker ps -q)
-if [ -n "$stale" ]; then
- log "killing $(echo "$stale" | wc -l) auto-started containers from the volume source"
- echo "$stale" | xargs -r docker kill >/dev/null 2>&1 || true
-fi
-
-# Pull the stack images from the app's Fly registry repo (CI mirrored them
-# there at deploy time). /var/lib/docker is a persistent volume, so the
-# manifest (image ID per tag) lets redeploys skip every image already in the
-# store, and pulls dedup at the layer level — only layers that actually
-# changed cross the network. Suspend/resume never re-runs this; only real
-# boots do. REGISTRY_PULL_TOKEN is an app-scoped deploy token staged as a Fly
-# secret by the workflow.
-if [ -f /srv/macro/preload/manifest.txt ]; then
- if [ -n "${REGISTRY_PULL_TOKEN:-}" ]; then
- docker login registry.fly.io -u x --password-stdin <<<"$REGISTRY_PULL_TOKEN"
- fi
- log "pulling stack images (parallel, warm store aware)"
- t0=$(date +%s)
- # The manifest's IDs are the RUNNER's view of each image (for hub images
- # under the containerd store, the source registry's index digest). Pulling
- # the mirror yields a different manifest and therefore a different local
- # .Id, so the inspect check below never matches mirrored images and every
- # boot re-pulled all 16 (66s of pure round-trips on a fully warm volume).
- # The receipt is our own volume-persisted record of which (runner id, tag)
- # pairs this store already satisfied — written only after a fully
- # successful pass, carried along by template-volume forks, and at worst
- # stale in the direction of one redundant pull.
- receipt="$state_dir/pulled.txt"
- pids=""
- while read -r id tag ref; do
- have=$(docker image inspect -f '{{.Id}}' "$tag" 2>/dev/null || true)
- if [ "$have" = "$id" ]; then
- log "already present: $tag"
- continue
- fi
- if [ -n "$have" ] && grep -qxF "$id $tag" "$receipt" 2>/dev/null; then
- log "already present (receipt): $tag"
- continue
- fi
- (docker pull -q "$ref" && docker tag "$ref" "$tag" && docker rmi "$ref" >/dev/null) &
- pids="$pids $!"
- done < /srv/macro/preload/manifest.txt
- rc=0
- for pid in $pids; do wait "$pid" || rc=1; done
- [ "$rc" = 0 ] || { log "image pull failed"; exit 1; }
- awk '{print $1, $2}' /srv/macro/preload/manifest.txt > "$receipt.next" \
- && mv "$receipt.next" "$receipt"
- log "images pulled in $(($(date +%s) - t0))s"
-fi
-
-# With a DOPPLER_TOKEN (a config-scoped service token, injected as a Fly
-# secret) the env layer pulls the preview secrets; without one the stack runs
-# on the code-owned local dummies only.
-doppler_args=(--no-doppler)
-if [ -n "${DOPPLER_TOKEN:-}" ]; then
- log "DOPPLER_TOKEN present — pulling preview secrets"
- doppler_args=()
-fi
-
-log "bringing the stack up (snapshot restore)"
-t0=$(date +%s)
-# tee the per-stage timings into the boot log; pipefail keeps a stack-up
-# failure fatal through the pipe.
-/srv/macro/bin/xtask stack up \
- "${doppler_args[@]}" \
- --no-build \
- --binaries-dir /srv/macro/artifacts/binaries \
- --frontend-dist /srv/macro/artifacts/frontend-dist \
- --json 2>&1 | tee -a "$timings"
-
-log "stack up took $(($(date +%s) - t0))s (boot total $(($(date +%s) - t_boot))s)"
-# The backend health gate is the longest stack-up phase; surface the auth
-# service's step timings plus its timestamped tail so the wait is attributable
-# from the CI boot-timings step (which ssh-cats the timings file).
-docker logs -t macro-authentication-service-1 2>&1 \
- | grep 'authentication startup step' \
- | sed 's/^/[preview][auth-startup] /' | tee -a "$timings" || true
-docker logs -t --tail 40 macro-authentication-service-1 2>&1 \
- | sed 's/^/[preview][auth] /' | tee -a "$timings" || true
-
-# Commit the compatibility marker only after the restored stack is healthy.
-# It lives beside (not inside) Docker's own data on the persistent volume, so
-# CI can distinguish a hot-updatable machine from an absent/partial/old deploy.
-install -m 0644 /srv/macro/deployment.json "$state_dir/deployment.json.next"
-mv -f "$state_dir/deployment.json.next" "$state_dir/deployment.json"
-log "stack ready — proxy serving on :8090"
-exec sleep infinity
diff --git a/infra/preview/fly.toml b/infra/preview/fly.toml
deleted file mode 100644
index 2cd6e60366..0000000000
--- a/infra/preview/fly.toml
+++ /dev/null
@@ -1,46 +0,0 @@
-# Fly config for per-PR preview apps. The workflow overrides the app name
-# (`--app macro-pr-<N>`) and supplies the image (`--image`); this file carries
-# everything else. See infra/preview/README.md.
-app = "macro-preview-template"
-primary_region = "ewr"
-
-# Everything (frontend, APIs, WebSockets, Mailpit) is behind the stack's
-# single-origin Caddy proxy.
-[http_service]
- internal_port = 8090
- force_https = true
- # The whole point: an idle preview suspends to ~zero cost and the fly-proxy
- # wakes it on the next request.
- auto_stop_machines = "suspend"
- auto_start_machines = true
- min_machines_running = 0
-
- [[http_service.checks]]
- interval = "15s"
- timeout = "5s"
- # First boot loads the baked images and restores the init snapshot before
- # the backend answers — give it room.
- grace_period = "5m"
- method = "GET"
- path = "/auth/health"
-
-# The inner Docker store must NOT live on the machine rootfs: Fly's rootfs is
-# an overlay (nested overlayfs forbids the whiteout device nodes layer
-# unpacking needs — every Debian-based image failed with EPERM) and has only
-# a few GB of headroom (ENOSPC mid-load). A volume is real ext4, properly
-# sized, and persists across redeploys — so later pushes reuse the layer
-# store. The deploy step creates the volume if it doesn't exist.
-[[mounts]]
- source = "docker_data"
- destination = "/var/lib/docker"
-
-[[vm]]
- # performance (dedicated) cores: boot is JVM/startup-latency bound and
- # shared cores throttle exactly then. ~16x the CPU $/hr of shared, but
- # compute only bills while awake and previews suspend when idle.
- cpu_kind = "performance"
- cpus = 8
- memory = "16gb"
-
-[[restart]]
- policy = "on-failure"
diff --git a/infra/preview/hot-update.sh b/infra/preview/hot-update.sh
deleted file mode 100644
index 6239a6453b..0000000000
--- a/infra/preview/hot-update.sh
+++ /dev/null
@@ -1,136 +0,0 @@
-#!/usr/bin/env bash
-# Apply a registry-carried artifact update to an already-running preview VM.
-# Exit 42 asks CI to fall back to a full image deploy/stack rehydrate.
-set -euo pipefail
-
-readonly REHYDRATE_EXIT=42
-readonly STATE_DIR=/var/lib/docker/.macro-preview
-readonly STATE_FILE="$STATE_DIR/deployment.json"
-
-log() { echo "[preview][hot-update] $*"; }
-rehydrate() { log "$*; requesting full rehydrate" >&2; exit "$REHYDRATE_EXIT"; }
-
-if [ "$#" -ne 2 ]; then
- echo "usage: hot-update <registry-image> <registry-token-file>" >&2
- exit 2
-fi
-update_image=$1
-token_file=$2
-
-mkdir -p "$STATE_DIR"
-if ! mkdir "$STATE_DIR/update.lock" 2>/dev/null; then
- echo "another preview update is already running" >&2
- exit 1
-fi
-
-staging=
-container=
-cleanup() {
- rm -f "$token_file"
- [ -z "$container" ] || docker rm -f "$container" >/dev/null 2>&1 || true
- [ -z "$staging" ] || rm -rf "$staging"
- rmdir "$STATE_DIR/update.lock" 2>/dev/null || true
-}
-trap cleanup EXIT
-
-[ -s "$token_file" ] || rehydrate "registry pull token is missing"
-[ -f "$STATE_FILE" ] || rehydrate "deployment state is missing"
-
-log "pulling artifact image $update_image"
-docker login registry.fly.io -u x --password-stdin < "$token_file" >/dev/null
-docker pull -q "$update_image"
-container=$(docker create "$update_image")
-staging=$(mktemp -d /srv/macro/.hot-update.XXXXXX)
-docker cp "$container:/update/." "$staging/"
-docker rm "$container" >/dev/null
-container=
-
-new_format=$(jq -er '.format' "$staging/deployment.json")
-new_snapshot=$(jq -er '.snapshot_key' "$staging/deployment.json")
-new_runtime=$(jq -er '.runtime_key' "$staging/deployment.json")
-new_frontend=$(jq -er '.frontend_key' "$staging/deployment.json")
-current_format=$(jq -er '.format' "$STATE_FILE")
-current_snapshot=$(jq -er '.snapshot_key' "$STATE_FILE")
-current_runtime=$(jq -er '.runtime_key' "$STATE_FILE")
-current_frontend=$(jq -er '.frontend_key' "$STATE_FILE")
-[ "$new_format" = "1" ] || rehydrate "unsupported update format $new_format"
-[ "$current_format" = "$new_format" ] || rehydrate "deployment format changed"
-[ "$current_snapshot" = "$new_snapshot" ] || rehydrate "snapshot key changed"
-[ "$current_runtime" = "$new_runtime" ] || rehydrate "runtime configuration changed"
-
-# CI mirrors every compose image and puts the desired IDs in the carrier.
-# Docker-built leaf services are safe to recreate in place. Any other image
-# change is infrastructure/runtime drift and must use the full rehydrate path.
-aux_changed=
-unsafe_images=
-while read -r expected_id tag registry_ref; do
- current_id=$(docker image inspect -f '{{.Id}}' "$tag" 2>/dev/null || true)
- [ "$current_id" = "$expected_id" ] && continue
- # Mirrored images never match by .Id (the manifest carries the runner's
- # source-registry digest; this store holds the mirror's) — same mismatch
- # the entrypoint's pull loop handles. Honor the boot pull receipt before
- # classifying an image as changed, or every hot update sees all 12 infra
- # images as "non-hot-updatable drift" and needlessly rehydrates.
- if [ -n "$current_id" ] \
- && grep -qxF "$expected_id $tag" "$STATE_DIR/pulled.txt" 2>/dev/null; then
- continue
- fi
- case "$tag" in
- *-ai_editing_worker|*-lexical_service|*-sync_service|*-websocket_service)
- log "refreshing auxiliary image $tag"
- docker pull -q "$registry_ref"
- pulled_id=$(docker image inspect -f '{{.Id}}' "$registry_ref")
- [ "$pulled_id" = "$expected_id" ] \
- || rehydrate "registry image ID mismatch for $tag"
- docker tag "$registry_ref" "$tag"
- docker rmi "$registry_ref" >/dev/null
- aux_changed=1
- ;;
- *) unsafe_images="$unsafe_images $tag" ;;
- esac
-done < "$staging/manifest.txt"
-[ -z "$unsafe_images" ] || rehydrate "non-hot-updatable images changed:$unsafe_images"
-
-# SSH sessions don't inherit app secrets. Recover only the config-scoped
-# Doppler token from PID 1 without printing it, then let xtask regenerate the
-# local env file before recreating/restarting containers.
-doppler_token=$(tr '\0' '\n' < /proc/1/environ | sed -n 's/^DOPPLER_TOKEN=//p' | head -n 1)
-[ -z "$doppler_token" ] || export DOPPLER_TOKEN="$doppler_token"
-export MACRO_REPO_ROOT=/srv/macro/repo
-export MACRO_STACK_SNAPSHOT_DIR=/srv/macro/artifacts/snapshots
-
-apply_args=(
- stack update
- --binaries-dir "$staging/binaries"
- --json
-)
-[ "$current_frontend" = "$new_frontend" ] \
- || apply_args+=(--frontend-dist "$staging/frontend-dist")
-[ -z "$aux_changed" ] || apply_args+=(--recreate-aux-services)
-
-log "applying prebuilt artifacts"
-"$staging/xtask" "${apply_args[@]}"
-
-# Publish the new orchestrator and compatibility marker only after the stack is
-# healthy. A failed apply leaves the old marker, so the next run rehydrates.
-install -m 0755 "$staging/xtask" /srv/macro/bin/xtask.next
-mv -f /srv/macro/bin/xtask.next /srv/macro/bin/xtask
-install -m 0644 "$staging/deployment.json" "$STATE_FILE.next"
-mv -f "$STATE_FILE.next" "$STATE_FILE"
-# The aux refreshes above satisfied the carrier manifest, so record it in the
-# boot pull receipt — otherwise the next real boot re-pulls what this update
-# already delivered (see the receipt logic in entrypoint.sh).
-awk '{print $1, $2}' "$staging/manifest.txt" > "$STATE_DIR/pulled.txt.next" \
- && mv "$STATE_DIR/pulled.txt.next" "$STATE_DIR/pulled.txt"
-
-# Retain the current carrier's layers for the next incremental pull, but remove
-# older carrier tags and their now-unreferenced layers from the persistent store.
-repository=${update_image%:*}
-while read -r old; do
- case "$old" in
- "$repository":update-*) [ "$old" = "$update_image" ] || docker rmi "$old" >/dev/null || true ;;
- esac
-done < <(docker images "$repository" --format '{{.Repository}}:{{.Tag}}')
-docker image prune -f >/dev/null || true
-
-log "update complete"
diff --git a/infra/preview/update.Dockerfile b/infra/preview/update.Dockerfile
deleted file mode 100644
index 28084f0e25..0000000000
--- a/infra/preview/update.Dockerfile
+++ /dev/null
@@ -1,13 +0,0 @@
-# Registry-carried artifact bundle for a running preview machine. The image is
-# never started: hot-update.sh creates a stopped container and copies /update
-# out of it. Scratch keeps every layer limited to actual application artifacts,
-# and stable COPY ordering maximizes registry/inner-Docker dedup across pushes.
-FROM scratch
-
-COPY bin/xtask /update/xtask
-COPY preload/manifest.txt /update/manifest.txt
-COPY deployment.json /update/deployment.json
-COPY artifacts/frontend-dist/ /update/frontend-dist/
-COPY artifacts/binaries/ /update/binaries/
-
-CMD ["/update/xtask"]
diff --git a/tooling/just/xtask.just b/tooling/just/xtask.just
index 5e3ecc5df3..3ca878e212 100644
--- a/tooling/just/xtask.just
+++ b/tooling/just/xtask.just
@@ -36,9 +36,9 @@ destroy_local *ARGS:
stop_local *ARGS:
{{ xtask }} stop-local "$@"
-# Headless stack orchestration for previews, agents, and CI.
+# Headless stack orchestration for agents and CI.
# just stack up # full stack; static frontend served by the proxy
-# just stack update # rebuild + reload only what changed (the `r` hotkey)
+# just stack update # adopt a new build; volumes stay
# just stack status --json # machine-readable state
# just stack down # reclaim everything
stack *ARGS:
diff --git a/tooling/xtask/crates/xtask_local/src/local.rs b/tooling/xtask/crates/xtask_local/src/local.rs
index 012bd1786a..3da21dff31 100644
--- a/tooling/xtask/crates/xtask_local/src/local.rs
+++ b/tooling/xtask/crates/xtask_local/src/local.rs
@@ -189,16 +189,6 @@ const LOCAL_RECREATE_SERVICE_IMAGES: &[&str] = &[
/// separately by [`build::ensure_runtime_image`].
const LOCAL_PULL_SERVICE_IMAGES: &[&str] = &["proxy", "mailpit", "static_file_cdn"];
-/// The artifact-driven preview updater can refresh every Docker-built app
-/// service. Keep this separate so enabling local aux rebuilds does not make the
-/// existing `run_local` edit loop rebuild the AI worker too.
-const PREVIEW_AUX_SERVICE_IMAGES: &[&str] = &[
- "websocket_service",
- "sync_service",
- "lexical_service",
- "ai_editing_worker",
-];
-
/// Bring up a Local or Dev stack and (unless `--no-frontend`) the frontend.
pub fn run_stack(mode: Mode, args: &cli::RunArgs) -> Result<()> {
if mode.spec().runs_local_infra {
@@ -601,17 +591,6 @@ fn recreate_aux_service_containers(
stage.run("Recreating auxiliary service containers", &mut up)
}
-fn recreate_preview_aux_service_containers(
- stage: &Stage,
- instance: &Instance,
- env: &env_layer::ResolvedEnv,
-) -> Result<()> {
- let mut up = compose_cmd(instance, env);
- up.args(["up", "-d", "--force-recreate", "--no-deps"])
- .args(PREVIEW_AUX_SERVICE_IMAGES);
- stage.run("Recreating preview auxiliary services", &mut up)
-}
-
/// How the local infra reaches its initialized state on bring-up.
#[derive(Clone, Copy, PartialEq, Eq)]
enum InfraInit {
@@ -680,8 +659,8 @@ fn bring_up_infra(
// equivalent of the MSK topic provisioning driven by the generated
// `.github/kafka-cluster-topics.json`. Restored volumes already carry
// the topics (they live in the broker's data dir), so only a full init
- // provisions — which also means a snapshot-restoring `stack up` (e.g.
- // the preview VM) never needs the rdkafka-backed `local-stack` feature.
+ // provisions — which also means a snapshot-restoring `stack up`
+ // never needs the rdkafka-backed `local-stack` feature.
if init == InfraInit::Full {
stage.run_step("Creating Kafka topics", || kafka::provision(instance))?;
}
diff --git a/tooling/xtask/crates/xtask_local/src/local/build/binaries_dir.rs b/tooling/xtask/crates/xtask_local/src/local/build/binaries_dir.rs
index 3cabc1f91d..f878d88470 100644
--- a/tooling/xtask/crates/xtask_local/src/local/build/binaries_dir.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/build/binaries_dir.rs
@@ -3,6 +3,7 @@
//! mount, not how the binaries were produced.
use std::path::{Path, PathBuf};
+use std::process::Command;
use anyhow::{Context, Result, bail};
@@ -17,6 +18,15 @@ pub enum BinariesDir {
NixStore(PathBuf),
}
+/// How a running stack adopts a newly located [`BinariesDir`].
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum Adoption {
+ /// Same canonical host dir. Content-addressed Nix paths are byte-identical.
+ Unchanged,
+ /// Different host dir. Rewrite compose mounts and recreate Rust containers.
+ Remount,
+}
+
impl BinariesDir {
/// Classify a directory by whether it lives under `/nix/store`.
pub fn classify(dir: &Path) -> Result<Self> {
@@ -78,4 +88,97 @@ impl BinariesDir {
}
Ok(())
}
+
+ /// Compare this set to the dir currently bind-mounted at `/app/out`.
+ pub fn adoption_from(&self, mounted_dir: &Path) -> Adoption {
+ let next = canonicalize_or_clone(self.host_dir());
+ let mounted = canonicalize_or_clone(mounted_dir);
+ if next == mounted {
+ Adoption::Unchanged
+ } else {
+ Adoption::Remount
+ }
+ }
+
+ /// Same as [`Self::adoption_from`], except a missing record means remount.
+ /// Legacy `stack.json` rows have no `binaries_dir`.
+ pub fn adoption_from_recorded(&self, recorded: Option<&Path>) -> Adoption {
+ match recorded {
+ Some(mounted) => self.adoption_from(mounted),
+ None => Adoption::Remount,
+ }
+ }
+
+ /// Keep a Nix store path alive while containers still mount it.
+ ///
+ /// `nix build --out-link` on Cloud flips the caller's link to the new
+ /// path. Pin each generation under `roots_dir` with its own `--out-link`.
+ /// Never rename those links: Nix registers the path, not the inode, so a
+ /// rename drops the root.
+ pub fn pin_gc_root(&self, roots_dir: &Path) -> Result<()> {
+ let Self::NixStore(dir) = self else {
+ return Ok(());
+ };
+ std::fs::create_dir_all(roots_dir)
+ .with_context(|| format!("creating {}", roots_dir.display()))?;
+ let store_output = store_output_dir(dir);
+ let current = roots_dir.join("nix-binaries");
+ if current.exists()
+ && canonicalize_or_clone(&current) == canonicalize_or_clone(store_output)
+ {
+ return Ok(());
+ }
+ if current.exists() {
+ publish_gc_root(
+ &roots_dir.join("nix-binaries.prev"),
+ &canonicalize_or_clone(&current),
+ )?;
+ }
+ publish_gc_root(&current, store_output)
+ }
+
+ /// Drop the previous generation's pin after the new mount is recorded.
+ pub fn release_previous_gc_root(roots_dir: &Path) {
+ let _ = std::fs::remove_file(roots_dir.join("nix-binaries.prev"));
+ }
+}
+
+/// Register `link` as a Nix GC root for `store_output`. Writes `--out-link`
+/// at `link` itself so the auto-root path stays stable.
+fn publish_gc_root(link: &Path, store_output: &Path) -> Result<()> {
+ let status = Command::new("nix")
+ .args(["build", "--out-link"])
+ .arg(link)
+ .arg(store_output)
+ .status()
+ .with_context(|| {
+ format!(
+ "running nix build --out-link {} {}",
+ link.display(),
+ store_output.display()
+ )
+ })?;
+ if status.success() {
+ return Ok(());
+ }
+ bail!(
+ "nix build --out-link {} {} failed with {status}",
+ link.display(),
+ store_output.display()
+ );
}
+
+fn canonicalize_or_clone(path: &Path) -> PathBuf {
+ path.canonicalize().unwrap_or_else(|_| path.to_path_buf())
+}
+
+fn store_output_dir(bin_dir: &Path) -> &Path {
+ if bin_dir.file_name().and_then(|n| n.to_str()) == Some("bin") {
+ bin_dir.parent().unwrap_or(bin_dir)
+ } else {
+ bin_dir
+ }
+}
+
+#[cfg(test)]
+mod test;
diff --git a/tooling/xtask/crates/xtask_local/src/local/build/binaries_dir/test.rs b/tooling/xtask/crates/xtask_local/src/local/build/binaries_dir/test.rs
new file mode 100644
index 0000000000..fde8387df7
--- /dev/null
+++ b/tooling/xtask/crates/xtask_local/src/local/build/binaries_dir/test.rs
@@ -0,0 +1,83 @@
+use super::*;
+
+fn scratch(name: &str) -> PathBuf {
+ let root = std::env::temp_dir().join(format!(
+ "macro-binaries-{}-{}-{name}",
+ std::process::id(),
+ std::time::SystemTime::now()
+ .duration_since(std::time::UNIX_EPOCH)
+ .unwrap()
+ .as_nanos()
+ ));
+ let _ = std::fs::remove_dir_all(&root);
+ std::fs::create_dir_all(&root).unwrap();
+ root
+}
+
+#[test]
+fn adoption_is_unchanged_for_the_same_path() {
+ let root = scratch("same");
+ let set = BinariesDir::TargetDir(root.clone());
+ assert_eq!(set.adoption_from(&root), Adoption::Unchanged);
+ let _ = std::fs::remove_dir_all(&root);
+}
+
+#[test]
+fn adoption_is_remount_when_the_host_dir_changes() {
+ let a = PathBuf::from("/tmp/macro-binaries-a");
+ let b = PathBuf::from("/tmp/macro-binaries-b");
+ let set = BinariesDir::TargetDir(a);
+ assert_eq!(set.adoption_from(&b), Adoption::Remount);
+}
+
+#[test]
+fn missing_record_is_a_remount() {
+ let set = BinariesDir::TargetDir(PathBuf::from("/tmp/macro-binaries-recorded"));
+ assert_eq!(set.adoption_from_recorded(None), Adoption::Remount);
+ assert_eq!(
+ set.adoption_from_recorded(Some(Path::new("/tmp/macro-binaries-recorded"))),
+ Adoption::Unchanged
+ );
+}
+
+#[test]
+fn classify_rejects_a_missing_dir() {
+ let err = BinariesDir::classify(Path::new("/tmp/macro-does-not-exist-binaries"))
+ .expect_err("missing dir");
+ assert!(err.to_string().contains("does not exist"));
+}
+
+#[test]
+fn target_dir_pin_is_a_noop() {
+ let root = scratch("target-pin");
+ let set = BinariesDir::TargetDir(root.join("bins"));
+ set.pin_gc_root(&root.join("roots")).unwrap();
+ assert!(!root.join("roots").join("nix-binaries").exists());
+ let _ = std::fs::remove_dir_all(&root);
+}
+
+#[test]
+fn nix_pin_fails_when_out_link_cannot_register() {
+ let root = scratch("nix-pin-fail");
+ let out = root.join("out-a");
+ std::fs::create_dir_all(out.join("bin")).unwrap();
+ let err = BinariesDir::NixStore(out.join("bin"))
+ .pin_gc_root(&root.join("roots"))
+ .expect_err("nix build --out-link must register a real GC root");
+ let msg = format!("{err:#}");
+ assert!(
+ msg.contains("nix build --out-link"),
+ "unexpected pin error: {msg}"
+ );
+ let _ = std::fs::remove_dir_all(&root);
+}
+
+#[test]
+fn release_previous_gc_root_deletes_the_prev_link() {
+ let root = scratch("release-prev");
+ let prev = root.join("nix-binaries.prev");
+ std::fs::write(&prev, "").unwrap();
+ BinariesDir::release_previous_gc_root(&root);
+ assert!(!prev.exists());
+ let _ = std::fs::remove_dir_all(&root);
+}
diff --git a/tooling/xtask/crates/xtask_local/src/local/build/mod.rs b/tooling/xtask/crates/xtask_local/src/local/build/mod.rs
index 6c52a5e078..f94993c3d2 100644
--- a/tooling/xtask/crates/xtask_local/src/local/build/mod.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/build/mod.rs
@@ -9,7 +9,7 @@ mod binaries_dir;
mod runtime_image;
mod zigbuild;
-pub use binaries_dir::BinariesDir;
+pub use binaries_dir::{Adoption, BinariesDir};
pub use runtime_image::{RUNTIME_IMAGE_TAG, ensure_runtime_image};
use anyhow::Result;
diff --git a/tooling/xtask/crates/xtask_local/src/local/cli.rs b/tooling/xtask/crates/xtask_local/src/local/cli.rs
index 168bd88b32..f08888d90e 100644
--- a/tooling/xtask/crates/xtask_local/src/local/cli.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/cli.rs
@@ -51,8 +51,8 @@ enum Cmd {
DestroyLocal(InstanceArgs),
/// Start, seed, and test an isolated local E2E stack.
LocalE2e(super::e2e::LocalE2eArgs),
- /// Headless stack orchestration (previews, agents, CI) — no TTY, no
- /// attached dev server; the proxy serves a static frontend bundle.
+ /// Headless stack orchestration (agents, CI) — no TTY, no attached
+ /// dev server; the proxy serves a static frontend bundle.
#[command(subcommand)]
Stack(StackCmd),
}
@@ -61,14 +61,12 @@ enum Cmd {
pub enum StackCmd {
/// Bring a full local stack up and return (only containers keep running).
Up(super::stack::UpArgs),
- /// Rebuild binaries (and optionally the frontend) and reload what changed.
+ /// Adopt a new build into the running stack. Data survives.
Update(super::stack::UpdateArgs),
/// Report the instance's containers, health, and URLs (`--json` for machines).
Status(super::stack::StatusArgs),
/// Tear the instance down: containers, volumes, and state.
Down(super::stack::DownArgs),
- /// Report the init-snapshot key/location for this instance.
- Snapshot(super::stack::SnapshotArgs),
}
#[derive(Args, Clone, Default)]
@@ -259,7 +257,6 @@ fn run(cli: Cli) -> Result<()> {
StackCmd::Update(a) => super::stack::update(&a),
StackCmd::Status(a) => super::stack::status(&a),
StackCmd::Down(a) => super::stack::down(&a),
- StackCmd::Snapshot(a) => super::stack::snapshot_status(&a),
},
}
}
diff --git a/tooling/xtask/crates/xtask_local/src/local/cli/test.rs b/tooling/xtask/crates/xtask_local/src/local/cli/test.rs
index 225380b960..663810ecfb 100644
--- a/tooling/xtask/crates/xtask_local/src/local/cli/test.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/cli/test.rs
@@ -25,6 +25,35 @@ fn local_e2e_accepts_suite_and_trailing_test_arguments() {
);
}
+#[test]
+fn stack_update_accepts_binaries_dir_with_build_aux_services() {
+ let cli = Cli::try_parse_from([
+ "cargo-x",
+ "stack",
+ "update",
+ "--binaries-dir",
+ "/tmp/bins",
+ "--build-aux-services",
+ ])
+ .unwrap();
+ let Cmd::Stack(StackCmd::Update(args)) = cli.command else {
+ panic!("expected stack update");
+ };
+ assert_eq!(
+ args.binaries_dir.as_deref(),
+ Some(std::path::Path::new("/tmp/bins"))
+ );
+ assert!(args.build_aux_services);
+}
+
+#[test]
+fn stack_has_no_snapshot_verb() {
+ match Cli::try_parse_from(["cargo-x", "stack", "snapshot"]) {
+ Err(err) => assert_eq!(err.kind(), clap::error::ErrorKind::InvalidSubcommand),
+ Ok(_) => panic!("stack snapshot must not parse"),
+ }
+}
+
#[test]
fn seed_scenario_accepts_instance_and_trailing_scenario_arguments() {
let cli = Cli::try_parse_from([
diff --git a/tooling/xtask/crates/xtask_local/src/local/env_layer.rs b/tooling/xtask/crates/xtask_local/src/local/env_layer.rs
index ef6bd86675..5489e5a50a 100644
--- a/tooling/xtask/crates/xtask_local/src/local/env_layer.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/env_layer.rs
@@ -199,7 +199,7 @@ fn pull_aws_credentials(env: &mut BTreeMap<String, String>) {
/// Pull Doppler secrets as a flat env overlay (mirrors the old
/// `just get_environment`). Two modes:
///
-/// - `DOPPLER_TOKEN` in the process env (CI, Fly preview machines) is a
+/// - `DOPPLER_TOKEN` in the process env (CI, Cloud agents) is a
/// config-scoped service token: download WITHOUT `--project`/`--config` (the
/// token pins them), and treat failure as fatal — whoever provisioned the
/// token explicitly asked for these secrets, so proceeding without them
diff --git a/tooling/xtask/crates/xtask_local/src/local/frontend.rs b/tooling/xtask/crates/xtask_local/src/local/frontend.rs
index 87f2982cfb..3e2b7f6550 100644
--- a/tooling/xtask/crates/xtask_local/src/local/frontend.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/frontend.rs
@@ -4,7 +4,6 @@
use std::io::Read;
use std::os::unix::process::CommandExt;
-use std::path::Path;
use std::process::{Child, Command, Stdio};
use std::sync::{Arc, Mutex};
use std::thread::JoinHandle;
@@ -40,40 +39,30 @@ pub fn static_dir(instance: &Instance) -> std::path::PathBuf {
}
/// Build the app bundle for headless serving and stage it into the instance
-/// dir. `prebuilt` skips the build and stages an existing dist (CI hands the
-/// binaries-style artifact straight in). The build mirrors `just build-dev`
+/// dir. The build mirrors `just build-dev`
/// (dev-mode bundle, production optimizations), except the backend origin is
/// the `same-origin` sentinel — resolved from `location.origin` at runtime — so
-/// the one bundle works on localhost and through any tunnel/preview hostname.
+/// the one bundle works on localhost and through any tunneled hostname.
///
/// Setting `VITE_LOCAL_BACKEND_ORIGIN` also keeps `import.meta.env.DEV` true
/// in the static bundle (see `keepImportMetaDev` in apps/web/scripts). `vite build`
/// otherwise compiles DEV from `NODE_ENV=production`, which drops local-only
/// paths such as passwordless auto-login (`just run_local` uses `vite serve`,
-/// where DEV is already true). Fly preview sets the same origin env, so it
-/// gets the override without a second flag.
-pub fn build_static(
- stage: &Stage,
- instance: &Instance,
- mode: Mode,
- prebuilt: Option<&Path>,
-) -> Result<()> {
- let dist = match prebuilt {
- Some(dir) => dir.to_owned(),
- None => {
- let mut cmd = Command::new("bun");
- cmd.current_dir(app_dir())
- .args(["run", "--bun", "build"])
- .env("MODE", "development")
- .env("NODE_ENV", "production")
- .env("VITE_LOCAL_SERVERS", "ALL")
- .env("VITE_LOCAL_BACKEND_ORIGIN", "same-origin");
- if mode.spec().runs_local_infra {
- cmd.env("VITE_AI_EDITING_WORKER_URL", "/ai-editing");
- }
- stage.run("Building frontend bundle", &mut cmd)?;
- app_dir().join("dist")
+/// where DEV is already true). Headless `stack up` sets the same origin env.
+pub fn build_static(stage: &Stage, instance: &Instance, mode: Mode) -> Result<()> {
+ let dist = {
+ let mut cmd = Command::new("bun");
+ cmd.current_dir(app_dir())
+ .args(["run", "--bun", "build"])
+ .env("MODE", "development")
+ .env("NODE_ENV", "production")
+ .env("VITE_LOCAL_SERVERS", "ALL")
+ .env("VITE_LOCAL_BACKEND_ORIGIN", "same-origin");
+ if mode.spec().runs_local_infra {
+ cmd.env("VITE_AI_EDITING_WORKER_URL", "/ai-editing");
}
+ stage.run("Building frontend bundle", &mut cmd)?;
+ app_dir().join("dist")
};
if stage.is_dry_run() {
return Ok(());
diff --git a/tooling/xtask/crates/xtask_local/src/local/fusionauth.rs b/tooling/xtask/crates/xtask_local/src/local/fusionauth.rs
index 43c78ee9c0..f276d3d10d 100644
--- a/tooling/xtask/crates/xtask_local/src/local/fusionauth.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/fusionauth.rs
@@ -66,9 +66,12 @@ pub fn wait_ready(stage: &Stage, instance: &Instance) -> Result<()> {
instance.port(Port::FusionAuth),
identity::APPLICATION_ID,
);
+ // Require an actual 200: `curl -f` only fails on 400+, so FusionAuth's
+ // maintenance-mode 302 (e.g. after a boot-time DB connect failure) would
+ // otherwise pass as ready and every later login would 500.
let script = format!(
- "for i in $(seq 1 120); do curl -fsS --max-time 3 -H 'Authorization: {key}' {url} >/dev/null 2>&1 && exit 0; sleep 2; done; \
- echo 'timed out waiting for the FusionAuth kickstart'; exit 1",
+ "for i in $(seq 1 120); do [ \"$(curl -sS -o /dev/null -w '%{{http_code}}' --max-time 3 -H 'Authorization: {key}' {url} 2>/dev/null)\" = 200 ] && exit 0; sleep 2; done; \
+ echo 'timed out waiting for the FusionAuth kickstart (a 302 here means maintenance mode: FusionAuth could not reach its db)'; exit 1",
key = identity::FUSIONAUTH_API_KEY,
);
let mut cmd = Command::new("bash");
diff --git a/tooling/xtask/crates/xtask_local/src/local/mailpit.rs b/tooling/xtask/crates/xtask_local/src/local/mailpit.rs
index 24915e3349..c1ecd8aec7 100644
--- a/tooling/xtask/crates/xtask_local/src/local/mailpit.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/mailpit.rs
@@ -9,7 +9,7 @@ pub fn direct_ui_url(instance: &Instance) -> String {
format!("http://localhost:{}", instance.port(Port::MailpitUi))
}
-/// The single-origin Mailpit route used by headless stacks and previews.
+/// The single-origin Mailpit route used by headless stacks.
pub fn proxy_ui_url(instance: &Instance) -> String {
format!("{}/mailpit/", super::proxy::url(instance))
}
diff --git a/tooling/xtask/crates/xtask_local/src/local/proxy.rs b/tooling/xtask/crates/xtask_local/src/local/proxy.rs
index 3522116416..32df334812 100644
--- a/tooling/xtask/crates/xtask_local/src/local/proxy.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/proxy.rs
@@ -140,7 +140,7 @@ const SPECIAL_ROUTES: &str = r#" @websocket path /websocket /websocket/*
"#;
const MAILPIT_ROUTE: &str = r#" # Mailpit serves itself under /mailpit (MP_WEBROOT), so no prefix strip —
- # this is how a headless/preview stack reads its passwordless login codes.
+ # this is how a headless stack reads its passwordless login codes.
handle /mailpit/* {
reverse_proxy mailpit:8025
}
diff --git a/tooling/xtask/crates/xtask_local/src/local/sandbox_image.rs b/tooling/xtask/crates/xtask_local/src/local/sandbox_image.rs
index c8bc5edc82..71e1d51ee8 100644
--- a/tooling/xtask/crates/xtask_local/src/local/sandbox_image.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/sandbox_image.rs
@@ -6,10 +6,8 @@
//! flake exposes `x86_64-linux` and `aarch64-linux`; Apple Silicon and ARM
//! Linux therefore bake natively instead of qemu. BuildKit cache is the
//! freshness check: unchanged `container/` is a no-op rebuild, a Dockerfile or
-//! flake change pays the two nix shells. `--no-build` (Fly preview VM boot, CI
-//! snapshot bake) skips that: the image is already on the daemon from preload /
-//! the images lane, and the Fly VM's staged repo does not even include
-//! `crates/agent_harness/container`.
+//! flake change pays the two nix shells. `--no-build` (CI snapshot bake)
+//! skips that: the image is already on the daemon from preload.
use std::collections::BTreeMap;
use std::path::Path;
@@ -22,8 +20,7 @@ use super::stage::Stage;
#[cfg(test)]
mod test;
-/// Local Docker tag `just run_local` / Fly previews load. Keep in sync with
-/// `xtask_workflows::workflows::vars::AGENT_HARNESS_LOCAL_IMAGE`.
+/// Local Docker tag `just run_local` loads.
pub const DEFAULT_LOCAL_TAG: &str = "macro-agent-harness:latest";
/// Build context matching `just -f crates/agent_harness/justfile build-local`.
@@ -68,7 +65,7 @@ pub(crate) fn build_args(tag: &str, context: &Path) -> Vec<String> {
/// `docker build` the sandbox image when local sandboxes are on.
///
-/// `no_build` skips the invocation so Fly / `--no-build` stack-up can use a
+/// `no_build` skips the invocation so `--no-build` stack-up can use a
/// preloaded tag. Dry-run notes the plan and does not invoke Docker.
pub fn ensure(stage: &Stage, env: &BTreeMap<String, String>, no_build: bool) -> Result<()> {
let Some(plan) = EnsurePlan::from_env(env) else {
diff --git a/tooling/xtask/crates/xtask_local/src/local/sandbox_image/test.rs b/tooling/xtask/crates/xtask_local/src/local/sandbox_image/test.rs
index 40c08331b5..a46837305d 100644
--- a/tooling/xtask/crates/xtask_local/src/local/sandbox_image/test.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/sandbox_image/test.rs
@@ -57,7 +57,7 @@ fn docker_build_args_match_the_cli() {
#[test]
fn no_build_still_plans_the_tag() {
- // `--no-build` is an ensure() argument (Fly VM / CI bake), not an env flag.
+ // `--no-build` is an ensure() argument (CI bake / Cloud install), not an env flag.
// The plan is unchanged so stack-up still documents which preloaded tag it
// expects; docker is simply not invoked.
assert_eq!(
diff --git a/tooling/xtask/crates/xtask_local/src/local/snapshot.rs b/tooling/xtask/crates/xtask_local/src/local/snapshot.rs
index c25b362c46..1014c0aeed 100644
--- a/tooling/xtask/crates/xtask_local/src/local/snapshot.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/snapshot.rs
@@ -14,7 +14,7 @@
//!
//! Snapshots are tarballs of the Docker volumes, written by a throwaway helper
//! container, stored under `infra/local/generated/.snapshots/<key>/` (override
-//! with `MACRO_STACK_SNAPSHOT_DIR` — CI bakes this dir into preview images).
+//! with `MACRO_STACK_SNAPSHOT_DIR` — CI and Cloud install can bake this dir).
use std::path::PathBuf;
use std::process::Command;
@@ -61,8 +61,8 @@ fn archives(instance: &Instance) -> [(&'static str, String); 5] {
/// archives are written so the files are consistent.
const STATEFUL_SERVICES: &[&str] = &["postgres", "search", "kafka", "fusionauth", "db"];
-/// Where snapshots live. `MACRO_STACK_SNAPSHOT_DIR` overrides for CI bakes and
-/// preview images; the default sits inside the gitignored generated dir.
+/// Where snapshots live. `MACRO_STACK_SNAPSHOT_DIR` overrides for CI and Cloud
+/// install bakes; the default sits inside the gitignored generated dir.
// xtask is host tooling, not a service reading APP_SECRETS_JSON, so reading the
// process environment directly is correct here.
#[allow(clippy::disallowed_methods)]
@@ -104,7 +104,7 @@ impl Plan {
// cross-arch restore a structural cache miss instead of a latent bug
// if snapshots are ever shared between machines (e.g. an S3 cache).
// Today each snapshot store is machine-local or CI-baked for
- // same-arch Fly machines, so this is insurance, not a behavior change.
+ // same-arch hosts, so this is insurance, not a behavior change.
h.update(super::arch::detect()?.docker_platform.as_bytes());
// Infra image pins + topology.
diff --git a/tooling/xtask/crates/xtask_local/src/local/stack.rs b/tooling/xtask/crates/xtask_local/src/local/stack.rs
index a11791b3b5..f3fded1a9f 100644
--- a/tooling/xtask/crates/xtask_local/src/local/stack.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/stack.rs
@@ -1,19 +1,14 @@
//! Headless stack orchestration: `cargo x stack up|update|status|down`.
//!
-//! The preview/agent/CI surface. Same bring-up as `run_local`, but no TTY
-//! hotkey loop and no attached dev server: the frontend is a static bundle
-//! served by the instance's Caddy, so a finished `up` leaves only Docker
-//! containers running — nothing to babysit, and the whole product lives behind
-//! the ONE proxy origin that an agent drives with a browser and a preview
-//! deploy fronts with a real hostname.
+//! Same bring-up as `run_local`, but no TTY hotkey loop and no attached dev
+//! server. The frontend is a static bundle served by Caddy. A finished `up`
+//! leaves only Docker containers running behind one proxy origin.
//!
-//! `up` is full-delete/full-create like `run_local` (unconditionally
-//! idempotent); `update` is the `r`-hotkey as a one-shot verb (rebuild, restart
-//! only what changed); `status` is machine-readable state; `down` reclaims
-//! everything.
+//! `up` is full-delete/full-create. `update` adopts a new binary directory
+//! without touching volumes. If nothing is recorded yet, `update` bootstraps
+//! through `up`. `status` is machine-readable state. `down` reclaims everything.
-use std::io::{BufReader, Read};
-use std::path::{Path, PathBuf};
+use std::path::PathBuf;
use std::process::Command;
use anyhow::{Context, Result, bail};
@@ -30,9 +25,6 @@ use super::{Mode, arch, env_layer, frontend, mailpit, proxy, sdk_webhook, snapsh
pub struct UpArgs {
#[command(flatten)]
pub run: RunArgs,
- /// Stage this prebuilt frontend dist instead of building (CI artifact reuse).
- #[arg(long)]
- pub frontend_dist: Option<PathBuf>,
/// Neither restore from nor save an init snapshot — always run the full
/// migrate/kickstart/index init.
#[arg(long)]
@@ -49,15 +41,6 @@ pub struct UpArgs {
pub json: bool,
}
-#[derive(Args, Clone, Default)]
-pub struct SnapshotArgs {
- #[command(flatten)]
- pub instance: InstanceArgs,
- /// Print machine-readable JSON (key, dir, present).
- #[arg(long)]
- pub json: bool,
-}
-
#[derive(Args, Clone, Default)]
pub struct UpdateArgs {
#[command(flatten)]
@@ -67,23 +50,17 @@ pub struct UpdateArgs {
/// Also rebuild + restage the static frontend and recreate the proxy.
#[arg(long)]
pub frontend: bool,
- /// Stage this prebuilt frontend dist instead of building (implies --frontend).
+ /// Adopt this complete prebuilt binary set instead of invoking Cargo.
#[arg(long)]
- pub frontend_dist: Option<PathBuf>,
- /// Apply this complete prebuilt binary set instead of invoking Cargo.
- #[arg(long, conflicts_with = "build_aux_services")]
pub binaries_dir: Option<PathBuf>,
- /// Rebuild every repository-built local Docker service.
+ /// Rebuild every repository-built local Docker service and recreate it.
#[arg(long)]
pub build_aux_services: bool,
- /// Recreate registry-refreshed preview auxiliary services without building.
- #[arg(long, hide = true, requires = "binaries_dir")]
- pub recreate_aux_services: bool,
/// Stream subprocess output and show per-step timings.
#[arg(long, short)]
pub verbose: bool,
/// Print a machine-readable summary as the final line.
- #[arg(long, hide = true, requires = "binaries_dir")]
+ #[arg(long)]
pub json: bool,
}
@@ -114,8 +91,7 @@ struct StackState {
mode: String,
/// `static` (Caddy serves the bundle) or `none` (`--no-frontend`).
frontend: String,
- /// Stable directory bind-mounted into every Rust service. Prebuilt updates
- /// replace files here atomically so existing container mounts see them.
+ /// Host directory currently bind-mounted at `/app/out`.
#[serde(default)]
binaries_dir: Option<PathBuf>,
}
@@ -197,18 +173,12 @@ pub fn up(mode: Mode, args: &UpArgs) -> Result<Instance> {
// joined just before `bring_up_app` creates that container. (Dry run:
// synchronously, so the command preview prints in order.)
if static_frontend && stage.is_dry_run() {
- frontend::build_static(&stage, &instance, mode, args.frontend_dist.as_deref())?;
+ frontend::build_static(&stage, &instance, mode)?;
}
let fe_build = (static_frontend && !stage.is_dry_run()).then(|| {
let instance = instance.clone();
- let prebuilt = args.frontend_dist.clone();
std::thread::spawn(move || {
- frontend::build_static(
- &Stage::from_env().quiet(),
- &instance,
- mode,
- prebuilt.as_deref(),
- )
+ frontend::build_static(&Stage::from_env().quiet(), &instance, mode)
})
});
@@ -272,9 +242,9 @@ pub fn up(mode: Mode, args: &UpArgs) -> Result<Instance> {
.binaries_dir
.clone()
.unwrap_or_else(|| super::workspace_root().join(target.debug_dir()));
- let active_binaries = super::build::BinariesDir::classify(&configured_binaries)?
- .host_dir()
- .to_path_buf();
+ let binaries = super::build::BinariesDir::classify(&configured_binaries)?;
+ binaries.pin_gc_root(&instance.artifact_dir())?;
+ let active_binaries = binaries.host_dir().to_path_buf();
super::bring_up_app(&stage, mode, &instance, &env)?;
let _sdk_webhook_tunnel = (mode == Mode::Local && !stage.is_dry_run())
.then(|| sdk_webhook::start(&instance))
@@ -295,6 +265,7 @@ pub fn up(mode: Mode, args: &UpArgs) -> Result<Instance> {
binaries_dir: Some(active_binaries),
},
)?;
+ super::build::BinariesDir::release_previous_gc_root(&instance.artifact_dir());
}
let frontend_url = if static_frontend {
@@ -321,18 +292,41 @@ pub fn up(mode: Mode, args: &UpArgs) -> Result<Instance> {
Ok(instance)
}
-/// `cargo x stack update` — the `r` hotkey as a one-shot verb: rebuild the
-/// binaries (and optionally the frontend bundle) and restart only what changed.
+/// `cargo x stack update` — adopt a new build into the running stack.
+/// Volumes stay. With no recorded stack, this bootstraps through [`up`].
pub fn update(args: &UpdateArgs) -> Result<()> {
- let stage = Stage::from_env_cli(args.verbose);
let instance = Instance::derive(args.instance.instance.as_deref(), args.instance.port_base)?;
- let Some(state) = read_state(&instance) else {
- bail!(
- "no stack state at {} — bring the stack up first (`just stack up{}`)",
- state_path(&instance).display(),
- instance_suffix(&instance)
- );
+ if read_state(&instance).is_none() {
+ return bootstrap_from_update(args);
+ }
+ update_running(args)
+}
+
+fn bootstrap_from_update(args: &UpdateArgs) -> Result<()> {
+ let up_args = UpArgs {
+ run: super::cli::RunArgs {
+ instance: args.instance.clone(),
+ env: args.env.clone(),
+ build: super::cli::BuildArgs {
+ no_build: args.binaries_dir.is_some(),
+ build_aux_services: args.build_aux_services,
+ binaries_dir: args.binaries_dir.clone(),
+ },
+ no_frontend: false,
+ verbose: args.verbose,
+ traces: None,
+ },
+ no_snapshot: false,
+ infra_only: false,
+ json: args.json,
};
+ up(Mode::Local, &up_args).map(|_| ())
+}
+
+fn update_running(args: &UpdateArgs) -> Result<()> {
+ let stage = Stage::from_env_cli(args.verbose);
+ let instance = Instance::derive(args.instance.instance.as_deref(), args.instance.port_base)?;
+ let state = read_state(&instance).expect("update_running requires stack state");
let mode = mode_from_label(&state.mode)?;
stage.section(&format!(
"macro {} stack update — instance {}",
@@ -347,15 +341,20 @@ pub fn update(args: &UpdateArgs) -> Result<()> {
args.env.env_file.as_deref(),
state.frontend == "static",
)?;
- let changed = if let Some(source) = args.binaries_dir.as_deref() {
- let changed = apply_prebuilt_binaries(mode, &state, source)?;
- if args.recreate_aux_services {
- super::recreate_preview_aux_service_containers(&stage, &instance, &env)?;
- }
- if !changed.is_empty() {
- super::reload_services(&stage, &instance, &changed)?;
+ let remounted = if let Some(source) = args.binaries_dir.as_deref() {
+ let new = super::build::BinariesDir::classify(source)?;
+ new.validate(&super::inventory::local_binaries())?;
+ new.pin_gc_root(&instance.artifact_dir())?;
+ match new.adoption_from_recorded(state.binaries_dir.as_deref()) {
+ super::build::Adoption::Unchanged => {
+ stage.note("binaries unchanged — mounts left as-is");
+ false
+ }
+ super::build::Adoption::Remount => {
+ remount(&stage, mode, &instance, &env, &new, &state)?;
+ true
+ }
}
- changed
} else {
let target = arch::detect()?;
super::rebuild_and_reload(
@@ -366,18 +365,16 @@ pub fn update(args: &UpdateArgs) -> Result<()> {
target,
args.build_aux_services,
)?;
- Vec::new()
+ false
};
- if args.frontend || args.frontend_dist.is_some() {
- reload_static_frontend(
- &stage,
- &instance,
- mode,
- &env,
- &state,
- args.frontend_dist.as_deref(),
- )?;
+ if args.binaries_dir.is_some() && args.build_aux_services {
+ super::build_aux_service_images(&stage, &instance, &env)?;
+ super::recreate_aux_service_containers(&stage, &instance, &env)?;
+ }
+
+ if args.frontend {
+ reload_static_frontend(&stage, &instance, mode, &env, &state)?;
}
if mode.spec().wait_backend_before_frontend {
frontend::wait_backend_ready(&stage, &instance)?;
@@ -386,35 +383,59 @@ pub fn update(args: &UpdateArgs) -> Result<()> {
println!(
"{}",
serde_json::to_string(&json!({
- "changed_services": changed.iter().map(|s| s.compose_name).collect::<Vec<_>>(),
- "frontend_updated": args.frontend || args.frontend_dist.is_some(),
- "aux_services_recreated": args.recreate_aux_services,
+ "remounted": remounted,
+ "frontend_updated": args.frontend,
+ "aux_services_rebuilt": args.build_aux_services,
}))?
);
}
Ok(())
}
-fn apply_prebuilt_binaries(
+fn remount(
+ stage: &Stage,
mode: Mode,
+ instance: &Instance,
+ env: &env_layer::ResolvedEnv,
+ new: &super::build::BinariesDir,
state: &StackState,
- binaries_dir: &Path,
-) -> Result<Vec<&'static super::inventory::RustService>> {
- let destination = state.binaries_dir.as_ref().ok_or_else(|| {
- anyhow::anyhow!(
- "stack state predates prebuilt updates — perform a full stack up before updating"
- )
- })?;
- let source = super::build::BinariesDir::classify(binaries_dir)?;
- source.validate(&super::inventory::local_binaries())?;
- let mut changed = Vec::new();
- for service in super::inventory::services_for_mode(mode) {
- if replace_binary_if_changed(source.host_dir(), destination, service.cargo_bin)? {
- changed.push(service);
- }
+) -> Result<()> {
+ let gmail_forwarder = env
+ .merged
+ .get("GMAIL_FORWARDER_SA_KEY")
+ .is_some_and(|key| !key.trim().is_empty());
+ super::gen_compose::generate(
+ mode,
+ instance,
+ new,
+ state.frontend == "static",
+ gmail_forwarder,
+ )?;
+ let mut up = super::compose_cmd(instance, env);
+ up.args([
+ "up",
+ "-d",
+ "--force-recreate",
+ "--no-deps",
+ "--remove-orphans",
+ ]);
+ for svc in super::inventory::services_for_mode(mode) {
+ up.arg(svc.compose_name);
}
-
- Ok(changed)
+ if gmail_forwarder {
+ up.arg("gmail_forwarder");
+ }
+ stage.run("Remounting Rust services", &mut up)?;
+ write_state(
+ instance,
+ &StackState {
+ mode: state.mode.clone(),
+ frontend: state.frontend.clone(),
+ binaries_dir: Some(new.host_dir().to_path_buf()),
+ },
+ )?;
+ super::build::BinariesDir::release_previous_gc_root(&instance.artifact_dir());
+ Ok(())
}
fn reload_static_frontend(
@@ -423,7 +444,6 @@ fn reload_static_frontend(
mode: Mode,
env: &env_layer::ResolvedEnv,
state: &StackState,
- frontend_dist: Option<&Path>,
) -> Result<()> {
if state.frontend != "static" {
bail!(
@@ -431,7 +451,7 @@ fn reload_static_frontend(
re-run `just stack up` to serve one"
);
}
- frontend::build_static(stage, instance, mode, frontend_dist)?;
+ frontend::build_static(stage, instance, mode)?;
// build_static replaces the staged directory, so the container must be
// recreated to establish a bind mount to the new inode.
let mut up = super::compose_cmd(instance, env);
@@ -439,66 +459,6 @@ fn reload_static_frontend(
stage.run("Recreating proxy (frontend bundle)", &mut up)
}
-fn replace_binary_if_changed(
- source_dir: &Path,
- destination_dir: &Path,
- name: &str,
-) -> Result<bool> {
- let source = source_dir.join(name);
- let destination = destination_dir.join(name);
- if files_equal(&source, &destination)? {
- return Ok(false);
- }
- std::fs::create_dir_all(destination_dir)
- .with_context(|| format!("creating {}", destination_dir.display()))?;
- let temporary = destination_dir.join(format!(".{name}.hot-update-{}", std::process::id()));
- let _ = std::fs::remove_file(&temporary);
- std::fs::copy(&source, &temporary).with_context(|| {
- format!(
- "copying prebuilt binary {} to {}",
- source.display(),
- temporary.display()
- )
- })?;
- std::fs::rename(&temporary, &destination).with_context(|| {
- format!(
- "atomically replacing {} with {}",
- destination.display(),
- source.display()
- )
- })?;
- Ok(true)
-}
-
-fn files_equal(left: &Path, right: &Path) -> Result<bool> {
- let left_meta = std::fs::metadata(left)
- .with_context(|| format!("reading binary metadata for {}", left.display()))?;
- let Ok(right_meta) = std::fs::metadata(right) else {
- return Ok(false);
- };
- if left_meta.len() != right_meta.len() {
- return Ok(false);
- }
- let mut left = BufReader::new(
- std::fs::File::open(left).with_context(|| format!("opening {}", left.display()))?,
- );
- let mut right = BufReader::new(
- std::fs::File::open(right).with_context(|| format!("opening {}", right.display()))?,
- );
- let mut left_buf = [0_u8; 64 * 1024];
- let mut right_buf = [0_u8; 64 * 1024];
- loop {
- let left_read = left.read(&mut left_buf)?;
- let right_read = right.read(&mut right_buf)?;
- if left_read != right_read || left_buf[..left_read] != right_buf[..right_read] {
- return Ok(false);
- }
- if left_read == 0 {
- return Ok(true);
- }
- }
-}
-
/// `cargo x stack status` — container states + health through the proxy, as a
/// human summary or `--json` for machines.
pub fn status(args: &StatusArgs) -> Result<()> {
@@ -571,44 +531,6 @@ pub fn down(args: &DownArgs) -> Result<()> {
Ok(())
}
-/// `cargo x stack snapshot` — report the instance's init-snapshot key and
-/// whether a snapshot exists for it. CI uses `--json` to find the directory to
-/// bake into preview images.
-pub fn snapshot_status(args: &SnapshotArgs) -> Result<()> {
- let instance = Instance::derive(args.instance.instance.as_deref(), args.instance.port_base)?;
- // The key hashes the generated kickstart; (re)write it so the verb works
- // before any `up` has run. The kickstart depends on the resolved env (the
- // optional Google IdP client), so resolve it the same way `up` does —
- // otherwise the key reported here could differ from the key `up` computes.
- let env = env_layer::resolve(Mode::Local, &instance, false, None, true)?;
- let google = super::kickstart::GoogleIdp::from_env(&env.merged);
- super::fusionauth::write_kickstart(&instance, google.as_ref())?;
- let plan = snapshot::Plan::compute(&instance)?;
- if args.json {
- println!(
- "{}",
- serde_json::to_string(&json!({
- "key": plan.key,
- "dir": plan.dir,
- "present": plan.exists(),
- "root": snapshot::root_dir(),
- }))?
- );
- return Ok(());
- }
- println!("key {}", plan.key);
- println!("dir {}", plan.dir.display());
- println!(
- "present {}",
- if plan.exists() {
- "yes"
- } else {
- "no (next cold `stack up` will save it)"
- }
- );
- Ok(())
-}
-
/// The machine-readable endpoint block `up --json` and `status --json` share.
fn summary_json(mode: Mode, instance: &Instance, static_frontend: bool) -> serde_json::Value {
let mailpit_url = mode.spec().runs_local_infra.then(|| {
diff --git a/tooling/xtask/crates/xtask_local/src/local/stack/test.rs b/tooling/xtask/crates/xtask_local/src/local/stack/test.rs
index 0d948adee0..c3f9f0e114 100644
--- a/tooling/xtask/crates/xtask_local/src/local/stack/test.rs
+++ b/tooling/xtask/crates/xtask_local/src/local/stack/test.rs
@@ -44,18 +44,3 @@ fn clearing_state_invalidates_a_previous_headless_stack() {
clear_state(&instance).unwrap();
let _ = std::fs::remove_dir_all(instance.artifact_dir());
}
-
-#[test]
-fn compares_binary_contents_not_just_size() {
- let root = std::env::temp_dir().join(format!("macro-stack-test-{}", std::process::id()));
- let _ = std::fs::remove_dir_all(&root);
- std::fs::create_dir_all(&root).unwrap();
- let left = root.join("left");
- let right = root.join("right");
- std::fs::write(&left, b"same").unwrap();
- std::fs::write(&right, b"same").unwrap();
- assert!(files_equal(&left, &right).unwrap());
- std::fs::write(&right, b"diff").unwrap();
- assert!(!files_equal(&left, &right).unwrap());
- std::fs::remove_dir_all(root).unwrap();
-}
diff --git a/tooling/xtask/crates/xtask_paths/src/lib.rs b/tooling/xtask/crates/xtask_paths/src/lib.rs
index 70139a0005..7a8e254dd8 100644
--- a/tooling/xtask/crates/xtask_paths/src/lib.rs
+++ b/tooling/xtask/crates/xtask_paths/src/lib.rs
@@ -56,7 +56,7 @@ pub fn repo_root() -> PathBuf {
/// Read the runtime repository root used by packaged xtask binaries.
///
/// Environment variables are appropriate here because this tooling crate runs
-/// both inside and outside the service runtime, including in the preview VM.
+/// both inside and outside the service runtime, including packaged binaries.
#[allow(clippy::disallowed_methods)]
fn repo_root_override() -> Option<PathBuf> {
std::env::var_os("MACRO_REPO_ROOT").map(PathBuf::from)
diff --git a/tooling/xtask/crates/xtask_workflows/src/workflows.rs b/tooling/xtask/crates/xtask_workflows/src/workflows.rs
index 45c69eac34..d94c071d61 100644
--- a/tooling/xtask/crates/xtask_workflows/src/workflows.rs
+++ b/tooling/xtask/crates/xtask_workflows/src/workflows.rs
@@ -36,7 +36,6 @@ mod deploy_web_app;
mod docs_check;
mod ensure_daytona_snapshot;
mod path_validation;
-mod preview_fly;
mod pulumi_preview_pr;
mod push_local_stack_binaries;
mod reusable_deploy_service;
@@ -279,16 +278,6 @@ const WORKFLOWS: &[WorkflowFile] = &[
)
},
},
- WorkflowFile {
- slug: "preview_fly",
- file_name: "preview-fly.yml",
- render_yaml: || render_gh_workflow(preview_fly::preview_fly)(),
- },
- WorkflowFile {
- slug: "preview_fly",
- file_name: "preview-fly-cleanup.yml",
- render_yaml: || render_gh_workflow(preview_fly::preview_fly_cleanup)(),
- },
WorkflowFile {
slug: "check_node_modules_nix",
file_name: "check_node_modules_nix.yml",
diff --git a/tooling/xtask/crates/xtask_workflows/src/workflows/ensure_daytona_snapshot.rs b/tooling/xtask/crates/xtask_workflows/src/workflows/ensure_daytona_snapshot.rs
index 953f468f02..93d35d82f1 100644
--- a/tooling/xtask/crates/xtask_workflows/src/workflows/ensure_daytona_snapshot.rs
+++ b/tooling/xtask/crates/xtask_workflows/src/workflows/ensure_daytona_snapshot.rs
@@ -2,7 +2,7 @@
//! it is missing, and publishes the same image to GHCR. Generated into
//! `ensure_daytona_snapshot.yml`.
//!
-//! Local stacks and Fly previews `docker build` this image themselves (BuildKit
+//! Local stacks `docker build` this image themselves (BuildKit
//! cache is the freshness check). GHCR is the published copy for `main`
//! (`:latest`) and for verifying the image on PRs (`:$SHA` only).
@@ -96,7 +96,7 @@ fn ensure_snapshot() -> Job {
/// GHCR publish. Runs on Mid with Namespace remote buildx: the Dockerfile
/// bakes two nix shells, which is too large for the Small Daytona job's local
-/// daemon. linux/amd64 only — Daytona snapshots and Fly preview machines are
+/// daemon. linux/amd64 only — Daytona snapshots are
/// amd64. Local `docker build` is unpinned so Apple Silicon / ARM Linux bake
/// native `aarch64-linux` from the same flake. PRs push `:$SHA` only so they
/// cannot clobber `:latest`.
diff --git a/tooling/xtask/crates/xtask_workflows/src/workflows/preview_fly.rs b/tooling/xtask/crates/xtask_workflows/src/workflows/preview_fly.rs
deleted file mode 100644
index bdd408fe3e..0000000000
--- a/tooling/xtask/crates/xtask_workflows/src/workflows/preview_fly.rs
+++ /dev/null
@@ -1,946 +0,0 @@
-//! `Fly Preview` — full-stack preview deploys on Fly Machines, plus the
-//! cleanup workflow that destroys the app when the PR closes.
-//!
-//! Opt-in via the `preview` label. The deploy job builds every artifact on the
-//! runner (service binaries, same-origin frontend bundle, init snapshot),
-//! mirrors every stack image into the per-PR app's Fly registry repo, stages
-//! the rest into a slim VM image (`infra/preview/`), and deploys it as the
-//! per-PR Fly app `macro-pr-<N>` — which suspends when idle and wakes on
-//! request. The machine pulls the stack images at boot (layer-level dedup
-//! against its persistent /var/lib/docker volume) instead of shipping them
-//! inside the VM image. A brand-new PR's volume is forked from the newest
-//! template volume on `macro-preview-template` (a warm layer store published
-//! by a previous successful deploy, content-addressed by the image
-//! manifest), so first boots pull only deltas instead of ~6GB cold. Runtime
-//! secrets reach the machine as a config-scoped Doppler service token
-//! (`DOPPLER_PREVIEW_TOKEN` → Fly secret `DOPPLER_TOKEN`), which the
-//! stack's env layer pulls at boot.
-//!
-//! NOTE: never a required status check — previews are opt-in by design.
-
-use gh_workflow::{
- Concurrency, Event, Expression, Job, Level, Permissions, PullRequest, PullRequestType, Run,
- Step, Use, Workflow,
-};
-
-use crate::workflows::{runners, steps, vars};
-
-#[cfg(test)]
-mod test;
-
-/// The per-PR Fly app name; also the preview hostname (`<app>.fly.dev`).
-const APP_NAME: &str = "macro-pr-${{ github.event.pull_request.number }}";
-
-/// The app holding template volumes: warm /var/lib/docker layer stores that
-/// new PR apps fork instead of pulling ~6GB of images onto an empty volume.
-/// Machine-less — it exists purely as a volume namespace. Templates are
-/// published by successful deploys and content-addressed by the preload
-/// manifest, so nothing runs when the image set hasn't changed.
-const TEMPLATE_APP: &str = "macro-preview-template";
-
-/// Build the deploy workflow.
-pub fn preview_fly() -> Workflow {
- Workflow::new("Fly Preview")
- .on(Event::default().pull_request(
- PullRequest::default()
- .add_type(PullRequestType::Opened)
- .add_type(PullRequestType::Reopened)
- .add_type(PullRequestType::Synchronize)
- .add_type(PullRequestType::Labeled),
- ))
- .concurrency(
- Concurrency::new(Expression::new(
- "fly-preview-${{ github.event.pull_request.number }}",
- ))
- .cancel_in_progress(true),
- )
- .add_job("deploy", deploy())
-}
-
-/// Build the cleanup workflow (destroys the app on close / label removal).
-pub fn preview_fly_cleanup() -> Workflow {
- Workflow::new("Fly Preview Cleanup")
- .on(Event::default().pull_request(
- PullRequest::default()
- .add_type(PullRequestType::Closed)
- .add_type(PullRequestType::Unlabeled),
- ))
- .add_job("destroy", destroy())
-}
-
-fn deploy() -> Job {
- Job::default()
- // Same-repo branches only: the preview image embeds CI-built artifacts
- // and the app serves under our Fly org — never build it from fork code.
- .cond(Expression::new(
- "github.event.pull_request.head.repo.full_name == github.repository && \
- contains(github.event.pull_request.labels.*.name, 'preview')",
- ))
- // linux-mid for the extra cores: five build lanes (zigbuild, compose
- // image builds, frontend, snapshot bake) genuinely contend on the
- // 8-core rust-ci shape. The cache volume follows the explicit tag,
- // not the profile, so this keeps the dedicated preview pool (nix
- // store + cargo target + snapshots) it has always used: see
- // PREVIEW_CACHE_TAG for why sharing the check/test jobs' pool was
- // measured cold on both layers (different --target = disjoint
- // sccache keys, and their volumes never carry a cargo target dir).
- .runs_on(runners::Runner::Mid.with_cache_tag(vars::PREVIEW_CACHE_TAG))
- // Backstop against hangs: worst honest case is ~15 min cold builds +
- // a cold `crates/agent_harness/container` bake (two nix shells) +
- // ~6 min cold mirror push + the 30 min flyctl wait cap. 90m leaves
- // room for that extra image; anything past that is a stuck step (a
- // hung log fetch once burned 2h26m of runner before someone cancelled
- // it by hand).
- .timeout_minutes(90u32)
- .permissions(
- Permissions::default()
- .contents(Level::Read)
- .pull_requests(Level::Write),
- )
- .add_env(("FLY_API_TOKEN", vars::FLY_API_TOKEN))
- .add_env(("APP_NAME", APP_NAME))
- .add_env(("TEMPLATE_APP", TEMPLATE_APP))
- // Init snapshots live on the cache volume: an unchanged key skips the
- // whole infra bake and keeps snapshot bytes (→ VM image layer) stable.
- .add_env((
- "MACRO_STACK_SNAPSHOT_DIR",
- vars::PREVIEW_SNAPSHOT_VOLUME_DIR,
- ))
- // No incremental in CI: incremental caches are what bloated the cache
- // volume to 41G (making it the workspace's eviction victim — observed
- // volume loss between runs), and sccache refuses to cache incremental
- // compiles, so they also locked workspace crates out of sccache.
- // Cargo's fingerprint reuse (what makes warm runs fast) is unaffected.
- // Local run_local keeps incremental for the edit-rebuild loop.
- .add_env(("CARGO_INCREMENTAL", "0"))
- .add_step(steps::checkout(false, true))
- .add_step(steps::mount_cache_volume_with_cargo_target())
- // CARGO_INCREMENTAL=0 stops incremental caches being written, but the
- // ~25G already on older volumes never gets cleaned by cargo — it's
- // exactly the weight that made this pool the workspace quota's
- // eviction victim. Deleting is idempotent and instant when clean.
- .add_step(
- Step::new("Prune stale incremental caches")
- .run("rm -rf target/*/debug/incremental target/debug/incremental"),
- )
- // Namespace remote builder: persistent BuildKit layer cache across
- // runs, same as the deploy workflows use. The aux-image builds and the
- // preview-image build both go through it.
- .add_step(steps::setup_namespace_buildx())
- .add_step(steps::setup_nix())
- .add_step(steps::setup_reqs_web("Setup dev shell + web deps", false))
- .add_step(steps::configure_namespace_sccache(
- vars::PREVIEW_SCCACHE_NAME,
- ))
- .add_step(setup_flyctl())
- .add_step(build_artifacts())
- .add_step(dump_stack_diagnostics())
- .add_step(stage_context())
- .add_step(deploy_to_fly())
- .add_step(dump_fly_diagnostics())
- .add_step(dump_boot_timings())
- .add_step(publish_template_volume())
- .add_step(comment_preview_url())
- .add_step(steps::teardown_nix())
-}
-
-/// When the machine never passes its health check, the answer is inside the
-/// VM (dockerd, image load, `stack up`) — dump its logs and state so a failed
-/// deploy is diagnosable from CI output alone.
-fn dump_fly_diagnostics() -> Step<Run> {
- Step::new("Dump Fly diagnostics")
- .run(indoc::indoc! {r#"
- flyctl machine list --app "$APP_NAME" || true
- # `flyctl logs --no-tail` can hang forever on its NATS fetch
- # (observed: a healthy deploy stuck 1h53m in the timings dump) —
- # never call it without a timeout.
- timeout 60 flyctl logs --app "$APP_NAME" --no-tail || true
- "#})
- .if_condition(Expression::new("failure()"))
-}
-
-/// The entrypoint tees every boot-phase timing (image pulls, per-stage stack
-/// up, auth startup) into a volume-backed file precisely so CI can read it
-/// deterministically after a healthy deploy. `flyctl logs --no-tail` used to
-/// serve this and both hung (unbounded NATS fetch) and silently lost the
-/// early boot lines to Fly's short log retention. On a hot update the machine
-/// never reboots, so the file describes the last real boot — the hot path's
-/// own output already streams into the deploy step via `ssh console`.
-fn dump_boot_timings() -> Step<Run> {
- Step::new("Dump boot timings").run(indoc::indoc! {r#"
- timeout 120 flyctl ssh console --app "$APP_NAME" --quiet \
- --command 'cat /var/lib/docker/.macro-preview/boot-timings.log' || true
- "#})
-}
-
-/// Every build artifact in one step, five lanes in parallel. The lanes are
-/// independent by construction:
-///
-/// - `fly`: app create, template-volume fork kick, and pull-token mint —
-/// nothing here needs build output, and the fork's ~2 min server-side
-/// hydration must start within seconds to finish inside the compose
-/// build instead of gating machine creation.
-/// - `cargo`: service binaries then xtask_local, sequential because they
-/// share the workspace target dir (cargo's build-dir lock would serialize
-/// them anyway). xtask_local deliberately WITHOUT the `local-stack`
-/// feature (the VM restores the baked snapshot so it never provisions
-/// Kafka; skipping rdkafka means no dynamically-linked librdkafka in the
-/// VM) and via zigbuild, not a host build (the nix dev shell links the
-/// /nix/store ELF interpreter, which doesn't exist in the VM image).
-/// - `web`: the same-origin frontend bundle.
-/// - `images`: the compose-built app images (the most fragile part of a
-/// fresh bring-up — a broken Dockerfile still fails the step early via
-/// fail-fast wait), then a best-effort premirror push to the app's Fly
-/// registry so the network transfer overlaps the cargo lane instead of
-/// serializing after it. The deploy step's mirror loop stays
-/// authoritative: its manifest-existence check skips whatever was
-/// premirrored here and pushes the rest (e.g. the bake-built runtime
-/// image).
-/// - `bake`: the init snapshot (see the lane script header for details),
-/// under its own CARGO_TARGET_DIR so its host xtask build doesn't take
-/// the workspace build-dir lock the cargo lane holds.
-///
-/// Lanes log to files and are dumped as collapsible groups afterwards;
-/// `wait -n` fails the step on the first lane failure and kills the rest.
-fn build_artifacts() -> Step<Run> {
- Step::new("Build artifacts (parallel lanes)")
- .run(indoc::indoc! {r#"
- set -euo pipefail
- lanes="$RUNNER_TEMP/lanes"
- mkdir -p "$lanes"
-
- cat > "$lanes/cargo.sh" <<'LANE'
- set -euo pipefail
- cargo x zigbuild
- sccache --show-stats
- cargo zigbuild --release --target x86_64-unknown-linux-gnu.2.36 -p xtask_local
- LANE
-
- cat > "$lanes/web.sh" <<'LANE'
- set -euo pipefail
- cd apps/web
- export MODE=development NODE_ENV=production
- export VITE_LOCAL_SERVERS=ALL VITE_LOCAL_BACKEND_ORIGIN=same-origin
- # Same ceiling as `just -f apps/web/justfile build-dev`: vite's
- # chunk-render peaks at node's default ~4GB heap and aborts with
- # "Ineffective mark-compacts near heap limit".
- export NODE_OPTIONS=--max-old-space-size=6144
- bun run --bun build
- LANE
-
- # Instant lane: everything Fly-side that needs no build output.
- # The template fork must fire within seconds of the job starting —
- # its ~2 min server-side hydration then finishes inside the compose
- # build. (It previously sat after the premirror pushes, which put
- # it at the END of the build phase and re-gated the deploy.)
- # Entirely best-effort: everything in the fly lane (app create,
- # fork kick, token mint) has an authoritative fallback in the
- # deploy step, so a transient Fly API failure must not fail the
- # build. A child `bash -e` (not a `|| `-guarded function: errexit
- # is suppressed inside those) keeps its internal fail-fast so a
- # failed app_id fetch can't mint a garbage token.
- cat > "$lanes/fly.sh" <<'LANE'
- set -euo pipefail
- bash -euo pipefail "$(dirname "$0")/fly-prep.sh" \
- || echo "fly prep failed (non-fatal): the deploy step falls back" >&2
- LANE
-
- cat > "$lanes/fly-prep.sh" <<'LANE'
- flyctl status --app "$APP_NAME" >/dev/null 2>&1 \
- || flyctl apps create "$APP_NAME" --org "$FLY_ORG" \
- || flyctl status --app "$APP_NAME" >/dev/null
- # Kick the template-volume fork (see the deploy step's
- # volume-state block, which stays the authoritative fallback).
- if ! flyctl volumes list --app "$APP_NAME" --json 2>/dev/null \
- | jq -e 'map(select(.name == "docker_data")) | length > 0' >/dev/null; then
- tpl_src=$(flyctl volumes list --app "$TEMPLATE_APP" --json 2>/dev/null \
- | jq -r '[.[] | select((.name | startswith("tpl")) and .state == "created" and .attached_machine_id == null)]
- | sort_by(.created_at) | last | .id // empty' 2>/dev/null) || tpl_src=
- if [ -n "$tpl_src" ]; then
- curl -sf -X POST "https://api.machines.dev/v1/apps/$APP_NAME/volumes" \
- -H "Authorization: Bearer $FLY_API_TOKEN" \
- -H "Content-Type: application/json" \
- -d "{\"name\":\"docker_data\",\"region\":\"ewr\",\"source_volume_id\":\"$tpl_src\"}" \
- | jq -r '"early template fork: \(.id // "failed") (\(.state // "-"))"' \
- || echo "early template fork failed; deploy step will fall back" >&2
- fi
- fi
- # Mint the machine's app-scoped read-only registry pull token
- # (macaroon attenuation, see the deploy step for the why) so the
- # deploy preamble doesn't pay the round-trips serially.
- app_id=$(curl -sf https://api.fly.io/graphql \
- -H "Authorization: Bearer $FLY_API_TOKEN" \
- -H "content-type: application/json" \
- -d "{\"query\":\"{ app(name: \\\"$APP_NAME\\\") { internalNumericId } }\"}" \
- | jq -re '.data.app.internalNumericId')
- now=$(date +%s)
- printf '[{"type":"Apps","body":{"apps":{"%s":"r"}}},{"type":"ValidityWindow","body":{"not_before":%d,"not_after":%d}}]' \
- "$app_id" $((now - 60)) $((now + 604800)) \
- | flyctl tokens attenuate > "$RUNNER_TEMP/registry-pull-token.next"
- chmod 600 "$RUNNER_TEMP/registry-pull-token.next"
- mv "$RUNNER_TEMP/registry-pull-token.next" "$RUNNER_TEMP/registry-pull-token"
- LANE
-
- cat > "$lanes/images.sh" <<'LANE'
- set -euo pipefail
- docker compose --project-directory . -p macro -f docker/docker-compose.yml build \
- search sync_service websocket_service lexical_service ai_editing_worker analytics_proxy
- # Not a compose service, so `config --images` never lists it.
- # BuildKit cache on this runner is the freshness check.
- docker build --tag "$LOCAL_SANDBOX_IMAGE" crates/agent_harness/container
- # The image BUILD above is fatal; the premirror below is a pure
- # optimization with an authoritative fallback (the deploy step's
- # mirror loop), so its failure only costs a slower push later.
- bash -euo pipefail "$(dirname "$0")/premirror.sh" \
- || echo "premirror failed (non-fatal): the deploy step mirrors authoritatively" >&2
- LANE
-
- # Premirror: content-addressed refs, identical scheme to the deploy
- # step's mirror loop, so its manifest-existence check skips these.
- # Best effort by design — anything skipped here (images only the
- # bake produces, like macro-local-runtime:dev) is pushed there.
- # The fly lane created the app long ago; the guard is a fallback.
- cat > "$lanes/premirror.sh" <<'LANE'
- flyctl status --app "$APP_NAME" >/dev/null 2>&1 \
- || flyctl apps create "$APP_NAME" --org "$FLY_ORG" \
- || flyctl status --app "$APP_NAME" >/dev/null
- flyctl auth docker
- registry="registry.fly.io/$APP_NAME"
- for img in $(docker compose --project-directory . -p macro \
- -f docker/docker-compose.yml config --images | sort -u) alpine:3 "$LOCAL_SANDBOX_IMAGE"; do
- if ! docker image inspect "$img" >/dev/null 2>&1 && ! docker pull "$img"; then
- echo "premirror: skipping $img (not local, not pullable)"
- continue
- fi
- id=$(docker image inspect -f '{{.Id}}' "$img")
- ref="$registry:img-$(echo "$img" | tr '/:' '__')-$(echo "$id" | cut -c8-19)"
- docker tag "$img" "$ref"
- if ! docker manifest inspect "$ref" >/dev/null 2>&1; then
- for _ in 1 2 3; do
- docker push "$ref" && break
- echo "premirror push of $ref failed, retrying" >&2
- sleep 15
- done
- fi
- done
- LANE
-
- # A cold `stack up --infra-only` runs the real init (migrate,
- # kickstart, Kafka topics, indices) and saves the content-addressed
- # snapshot the VM restores from. Infra only: the app services need
- # the Doppler-sourced env to boot, which this runner deliberately
- # lacks — the snapshot captures only the infra volumes anyway. Via
- # `just` (not the bare `cargo x` alias) because the justfile
- # enables the `local-stack` feature the Kafka provisioning needs.
- # The store lives on the cache volume for a zero-copy hit; every
- # snapshot is also stored as one tar in Namespace artifact storage
- # keyed by the same hash, so cache-volume onboarding misses
- # download the exact snapshot instead of paying for a full init.
- # Artifact access is an optimization: failures fall back to the
- # volume/cold-bake behavior. The final `snapshot --json` line lands
- # in $RUNNER_TEMP for the stage step.
- cat > "$lanes/bake.sh" <<'LANE'
- set -euo pipefail
- export CARGO_TARGET_DIR="$PWD/target/bake"
- mkdir -p "$MACRO_STACK_SNAPSHOT_DIR"
- status=$(just stack snapshot --json | tail -n 1)
- key=$(echo "$status" | jq -r '.key')
- root=$(echo "$status" | jq -r '.root')
- artifact="macro-preview/init-snapshots/${key}.tar"
- archive="$RUNNER_TEMP/init-snapshot-${key}.tar"
- artifact_hit=
-
- # The Namespace cache-volume pool has an onboarding period where a
- # runner can receive an empty fork. Artifact storage is the durable
- # fallback: restore into a temporary directory and only publish it to
- # the live store after validating the embedded manifest through xtask.
- if ! echo "$status" | jq -e '.present' >/dev/null; then
- restored="$RUNNER_TEMP/restored-init-snapshot"
- rm -rf "$restored"
- mkdir -p "$restored"
- if nsc artifact download "$artifact" "$archive" \
- && tar -xf "$archive" -C "$restored" \
- && [ -f "$restored/$key/manifest.json" ]; then
- rm -rf "$root/$key"
- cp -a "$restored/$key" "$root/$key"
- status=$(just stack snapshot --json | tail -n 1)
- if echo "$status" | jq -e '.present' >/dev/null; then
- artifact_hit=1
- echo "init snapshot artifact hit: $status"
- else
- rm -rf "$root/$key"
- echo "downloaded init snapshot failed validation; baking cold" >&2
- fi
- else
- echo "init snapshot artifact miss; baking cold" >&2
- fi
- fi
-
- if echo "$status" | jq -e '.present' >/dev/null; then
- echo "init snapshot cache hit: $status"
- cargo run --quiet --manifest-path Cargo.toml \
- -p xtask_local --features local-stack -- gen-compose
- # The cold path builds the runtime image inside `stack up`; on a
- # hit nothing else does, and the registry mirror needs it in the
- # local daemon (its pull fallback only covers public images —
- # macro-local-runtime:dev is not on Docker Hub).
- cargo run --quiet --manifest-path Cargo.toml \
- -p xtask_local --features local-stack -- runtime-image
- else
- just stack up --infra-only --no-doppler --no-build
- status=$(just stack snapshot --json | tail -n 1)
- echo "$status" | jq -e '.present' >/dev/null
- fi
-
- # Seed artifact storage from either a volume hit or a cold bake. Avoid
- # uploading another version when the artifact already exists; a race
- # between concurrent PRs is harmless and must not fail the preview.
- if [ -z "$artifact_hit" ] && ! nsc artifact describe "$artifact" >/dev/null 2>&1; then
- snap_dir=$(echo "$status" | jq -r '.dir')
- tar -cf "$archive" -C "$root" "$(basename "$snap_dir")"
- nsc artifact upload "$archive" "$artifact" --expires_in 720h \
- || echo "warning: failed to upload init snapshot artifact" >&2
- fi
- echo "$status" > "$RUNNER_TEMP/preview-snapshot.json"
- LANE
-
- declare -a pids=() names=()
- for name in fly cargo web images bake; do
- # setsid: each lane leads its own process group so fail-fast
- # can kill the whole tree (cargo/docker/bun children), not just
- # the wrapper shell.
- setsid bash "$lanes/$name.sh" > "$lanes/$name.log" 2>&1 &
- pids+=($!)
- names+=("$name")
- done
- fail=
- for _ in "${pids[@]}"; do
- # Bare `wait -n`: any child; explicit pids would error once a
- # job has already been reaped.
- if ! wait -n; then
- fail=1
- break
- fi
- done
- if [ -n "$fail" ]; then
- # Negative pids: signal each lane's whole process group.
- kill -- "${pids[@]/#/-}" 2>/dev/null || true
- wait || true
- fi
- for i in "${!names[@]}"; do
- echo "::group::lane: ${names[$i]}"
- cat "$lanes/${names[$i]}.log" || true
- echo "::endgroup::"
- done
- if [ -n "$fail" ]; then
- echo "a build lane failed (first failure aborts the rest); see lane logs above" >&2
- exit 1
- fi
- "#})
- .add_env(("FLY_ORG", "${{ vars.FLY_ORG || secrets.FLY_ORG }}"))
- .add_env(("LOCAL_SANDBOX_IMAGE", vars::AGENT_HARNESS_LOCAL_IMAGE))
-}
-
-/// When the bake dies (typically the backend health gate), the answer is in
-/// the service logs — dump container states and recent logs so a failed run is
-/// diagnosable from CI output alone.
-fn dump_stack_diagnostics() -> Step<Run> {
- Step::new("Dump stack diagnostics")
- .run(indoc::indoc! {r#"
- docker compose -p macro ps --all || true
- for svc in authentication-service proxy fusionauth postgres kafka \
- connection_gateway document_storage_service email_service \
- notification_service contacts_service; do
- echo "==================== $svc ===================="
- docker compose -p macro logs --no-color --tail 80 "$svc" 2>&1 || true
- done
- "#})
- .if_condition(Expression::new("failure()"))
-}
-
-fn stage_context() -> Step<Run> {
- Step::new("Stage preview build context").run(indoc::indoc! {r#"
- set -euo pipefail
- # The dev-shell env (BASH_ENV) exports a Nix LD_LIBRARY_PATH; host
- # binaries like rsync then resolve Nix-store libs whose deps drag in
- # Nix glibc mid-process and crash on glibc symbol versions. This step
- # is pure file shuffling — drop it.
- unset LD_LIBRARY_PATH
- ctx=preview-ctx
- rm -rf "$ctx"
- mkdir -p "$ctx/repo" "$ctx/artifacts/binaries" "$ctx/bin"
- # The minimal repo layout xtask reads at runtime — including every
- # snapshot-key input, byte-identical to this checkout, so the VM's
- # key matches the snapshot baked above.
- rsync -a --relative --exclude node_modules \
- docker/docker-compose.yml \
- docker/docker-compose-databases.yml \
- infra/stacks/fusionauth-instance/docker-compose.yml \
- infra/local/nginx \
- infra/local/opensearch \
- infra/stacks/opensearch/helpers \
- crates/macro_db_client/migrations \
- "$ctx/repo/"
- # Service binaries only — the target dir also holds gigabytes of
- # build intermediates. `-p` everywhere: preserved mtimes mean files
- # cargo didn't relink produce byte-identical Docker layers, so the
- # registry push/pull skips them (the Dockerfile orders its COPYs
- # stable → volatile for the same reason).
- find target/x86_64-unknown-linux-gnu/debug \
- -maxdepth 1 -type f -executable ! -name '*.d' ! -name '*.so' \
- -exec cp -p {} "$ctx/artifacts/binaries/" \;
- cp -a apps/web/dist "$ctx/artifacts/frontend-dist"
- # Only the current key's snapshot — the volume store accumulates keys.
- snap_dir=$(jq -r '.dir' "$RUNNER_TEMP/preview-snapshot.json")
- mkdir -p "$ctx/artifacts/snapshots"
- cp -a "$snap_dir" "$ctx/artifacts/snapshots/"
- snapshot_key=$(jq -r '.key' "$RUNNER_TEMP/preview-snapshot.json")
- runtime_key=$(cd "$ctx/repo" \
- && find . -type f -print0 \
- | sort -z \
- | xargs -0 sha256sum \
- | sha256sum \
- | cut -d ' ' -f 1)
- frontend_key=$(cd "$ctx/artifacts/frontend-dist" \
- && find . -type f -print0 \
- | sort -z \
- | xargs -0 sha256sum \
- | sha256sum \
- | cut -d ' ' -f 1)
- jq -n \
- --arg snapshot_key "$snapshot_key" \
- --arg runtime_key "$runtime_key" \
- --arg frontend_key "$frontend_key" \
- --arg commit "${{ github.sha }}" \
- '{format: 1, snapshot_key: $snapshot_key, runtime_key: $runtime_key, frontend_key: $frontend_key, commit: $commit}' \
- > "$ctx/deployment.json"
- cp -p target/x86_64-unknown-linux-gnu/release/xtask_local "$ctx/bin/xtask"
- cp -p infra/preview/hot-update.sh "$ctx/bin/hot-update"
- cp -p infra/preview/entrypoint.sh "$ctx/entrypoint.sh"
- cp -p infra/preview/Dockerfile "$ctx/Dockerfile"
- cp -p infra/preview/update.Dockerfile "$ctx/update.Dockerfile"
- "#})
-}
-
-fn setup_flyctl() -> Step<Use> {
- Step::new("Setup flyctl").uses(
- "superfly",
- "flyctl-actions/setup-flyctl",
- "ed8efb33836e8b2096c7fd3ba1c8afe303ebbff1",
- ) // 1.6
-}
-
-/// Select bootstrap / hot update / rehydrate from the running machine's durable
-/// compatibility marker. All modes mirror content-addressed stack images. A
-/// compatible machine pulls a scratch artifact carrier and applies it in place;
-/// an absent, incompatible, or failed hot update falls through to the immutable
-/// full-image deploy and snapshot restore.
-fn deploy_to_fly() -> Step<Run> {
- Step::new("Deploy to Fly")
- .run(indoc::indoc! {r#"
- set -euo pipefail
- if [ -z "$DOPPLER_PREVIEW_TOKEN" ]; then
- echo "DOPPLER_PREVIEW_TOKEN secret is not set (a Doppler service token scoped to the preview config)" >&2
- exit 1
- fi
- if [ -z "$FLY_ORG" ]; then
- echo "FLY_ORG is not set (repo variable or secret with the Fly org slug)" >&2
- exit 1
- fi
-
- app_existed=
- if flyctl status --app "$APP_NAME" >/dev/null 2>&1; then
- app_existed=1
- else
- # A transient status error must not turn an existing app into a
- # hard create failure: create, then prove the app is reachable.
- flyctl apps create "$APP_NAME" --org "$FLY_ORG" \
- || flyctl status --app "$APP_NAME" >/dev/null
- fi
-
- # Seed a first boot's volume from the newest template volume — a
- # warm /var/lib/docker layer store published by a previous
- # successful deploy (see the "Publish template volume" step).
- # Measured cold, an empty volume costs ~11 min of image pulls at
- # boot. The images lane normally kicked the fork minutes ago (its
- # ~2 min server-side hydration overlaps the whole build phase);
- # this block is the authoritative fallback for a missing or
- # failed early fork. Cross-app forks need the Machines API:
- # flyctl resolves volume ids app-locally. The boot manifest check
- # makes the volume a pure cache, so every failure path here just
- # falls back to the old empty-volume behavior.
- vol_state=$(flyctl volumes list --app "$APP_NAME" --json 2>/dev/null \
- | jq -r '[.[] | select(.name == "docker_data")][0].state // empty') || vol_state=
- if [ -z "$vol_state" ]; then
- tpl_src=$(flyctl volumes list --app "$TEMPLATE_APP" --json 2>/dev/null \
- | jq -r '[.[] | select((.name | startswith("tpl")) and .state == "created" and .attached_machine_id == null)]
- | sort_by(.created_at) | last | .id // empty' 2>/dev/null) || tpl_src=
- if [ -n "$tpl_src" ]; then
- vol_state=$(curl -sf -X POST "https://api.machines.dev/v1/apps/$APP_NAME/volumes" \
- -H "Authorization: Bearer $FLY_API_TOKEN" \
- -H "Content-Type: application/json" \
- -d "{\"name\":\"docker_data\",\"region\":\"ewr\",\"source_volume_id\":\"$tpl_src\"}" \
- | jq -r '.state // empty') || vol_state=
- if [ -n "$vol_state" ]; then
- echo "seeding docker_data from template volume $tpl_src (state: $vol_state)"
- else
- echo "template fork failed; falling back to an empty volume" >&2
- fi
- else
- echo "no template volume available; first boot will pull cold"
- fi
- fi
-
- # A read-only, app-scoped, time-boxed pull token so the machine's
- # inner dockerd can pull the mirrored stack images from this app's
- # registry repo — and nothing else (PR code can read machine
- # secrets, so scope matters). The org deploy token can't mint new
- # tokens (createLimitedAccessToken is not authorized), but macaroon
- # attenuation is pure client-side crypto: append an Apps caveat
- # (numeric app id, mask "r") and a validity window to our own
- # token. The images lane normally minted it already; fall back to
- # minting inline when its file is absent.
- if [ -s "$RUNNER_TEMP/registry-pull-token" ]; then
- pull_token=$(cat "$RUNNER_TEMP/registry-pull-token")
- else
- app_id=$(curl -sf https://api.fly.io/graphql \
- -H "Authorization: Bearer $FLY_API_TOKEN" \
- -H "content-type: application/json" \
- -d "{\"query\":\"{ app(name: \\\"$APP_NAME\\\") { internalNumericId } }\"}" \
- | jq -re '.data.app.internalNumericId')
- now=$(date +%s)
- pull_token=$(printf '[{"type":"Apps","body":{"apps":{"%s":"r"}}},{"type":"ValidityWindow","body":{"not_before":%d,"not_after":%d}}]' \
- "$app_id" $((now - 60)) $((now + 604800)) | flyctl tokens attenuate)
- fi
-
- snapshot_key=$(jq -r '.key' "$RUNNER_TEMP/preview-snapshot.json")
- runtime_key=$(jq -r '.runtime_key' preview-ctx/deployment.json)
- mode=bootstrap
- machine_id=
- if [ -n "$app_existed" ]; then
- mode=rehydrate
- machine_id=$(flyctl machine list --app "$APP_NAME" --json \
- | jq -r '[.[] | select((.config.mounts // []) | length > 0)][0].id // empty')
- if [ -n "$machine_id" ]; then
- # A suspended preview needs demand before SSH is reachable.
- curl -s -o /dev/null --max-time 10 "https://$APP_NAME.fly.dev/" || true
- for _ in $(seq 1 12); do
- # `--command` is exec'd directly, NOT run through a shell —
- # a bare `if [ -f ...` dies with `exec: "if": not found`, so
- # the marker never parsed and every push silently rehydrated
- # (~7 min) instead of going hot (~3 min). Hence the sh -c.
- marker=$(flyctl ssh console --app "$APP_NAME" --machine "$machine_id" \
- --quiet --command 'sh -c "if [ -f /var/lib/docker/.macro-preview/deployment.json ]; then cat /var/lib/docker/.macro-preview/deployment.json; else echo __NO_MARKER__; fi"' \
- 2>/dev/null || true)
- if echo "$marker" | jq -e \
- --arg key "$snapshot_key" \
- --arg runtime "$runtime_key" \
- '.format == 1 and .snapshot_key == $key and .runtime_key == $runtime and (.frontend_key | type == "string")' \
- >/dev/null 2>&1; then
- mode=hot
- break
- fi
- # SSH succeeded and returned a marker (or proved it absent):
- # incompatibility is real, not a wake-up race.
- if [ "$marker" = __NO_MARKER__ ] \
- || echo "$marker" | jq -e '.format != null' >/dev/null 2>&1; then
- break
- fi
- sleep 5
- done
- fi
- fi
- echo "preview deployment mode: $mode (snapshot $snapshot_key)"
-
- # Keep the machine in demand for the WHOLE deploy, hot path
- # included: fly-proxy suspends a machine ~7 min after its last
- # proxied request, and SSH via hallpass is not proxy traffic — a
- # hot update with no browser traffic got its VM suspended
- # mid-session (the frozen `flyctl ssh console` then ate 52 min of
- # runner until the job timeout). Any response status counts as
- # traffic, and a request also resumes a suspended machine.
- (while :; do
- curl -s -o /dev/null --max-time 10 "https://$APP_NAME.fly.dev/" || true
- sleep 15
- done) &
- keepalive=$!
- trap 'kill "$keepalive" 2>/dev/null || true' EXIT
-
- flyctl auth docker
- registry="registry.fly.io/$APP_NAME"
- # Mirror every stack image into the app's registry repo instead of
- # baking docker-save tars into the VM image (which made every
- # deploy re-ship ~5GB the machine's volume already had). Pushes
- # dedup at the layer level against the repo, so a redeploy only
- # uploads layers that actually changed; the entrypoint pulls with
- # the same dedup against the machine's persistent layer store. The
- # manifest maps image ID -> local tag -> registry ref so boots can
- # skip images already present. alpine:3 rides along for the
- # snapshot-restore helper container.
- # On a snapshot cache hit only gen-compose ran, which doesn't write
- # the env file; compose hard-fails on a missing --env-file, and the
- # image names don't interpolate env vars anyway.
- envfile=infra/local/generated/macro/local.generated.env
- [ -f "$envfile" ] || : > "$envfile"
- images=$(docker compose --project-directory . -p macro \
- -f docker/docker-compose.yml \
- -f infra/local/generated/macro/docker-compose.override.yml \
- --env-file "$envfile" \
- config --images | sort -u)
- echo "mirroring images:" $images
- docker image inspect alpine:3 >/dev/null 2>&1 || docker pull alpine:3
- mkdir -p preview-ctx/preload
- : > preview-ctx/preload/manifest.txt
- for img in $images alpine:3 "$LOCAL_SANDBOX_IMAGE"; do
- # Images the bake didn't leave in the daemon (snapshot cache hit,
- # or app-layer-only images like proxy/mailpit) get pulled here.
- # Compose `config --images` also lists locally-built tags
- # (analytics_proxy, sdk-webhook-relay) that are not on Docker Hub;
- # premirror already skips those — do the same so a missing local
- # tag cannot fail the deploy.
- if ! docker image inspect "$img" >/dev/null 2>&1 && ! docker pull "$img"; then
- echo "mirror: skipping $img (not local, not pullable)"
- continue
- fi
- id=$(docker image inspect -f '{{.Id}}' "$img")
- # Content-addressed tag: same image content = same ref, so a
- # redeploy's push is skippable by a manifest existence check and
- # concurrent PRs' deploys can never clobber each other's tags.
- ref="$registry:img-$(echo "$img" | tr '/:' '__')-$(echo "$id" | cut -c8-19)"
- docker tag "$img" "$ref"
- echo "$id $img $ref" >> preview-ctx/preload/manifest.txt
- done
- # The existence probes are pure network round-trips (and after the
- # premirror lane, almost all of them hit) — run them in parallel
- # instead of ~1s each serially.
- awk '{print $3}' preview-ctx/preload/manifest.txt \
- | xargs -P 8 -I {} sh -c \
- 'docker manifest inspect "{}" >/dev/null 2>&1 || echo "{}"' \
- > "$RUNNER_TEMP/push-refs.txt"
- echo "pushing $(wc -l < "$RUNNER_TEMP/push-refs.txt") of $(wc -l < preview-ctx/preload/manifest.txt) images"
- # The Fly registry occasionally aborts large uploads ("s3aws:
- # append to zero-size path unsupported") — retry; layers that made
- # it are reused. Four pushes at a time keeps the first (cold) push
- # moving without saturating the registry.
- if [ -s "$RUNNER_TEMP/push-refs.txt" ]; then
- xargs -P 4 -I {} sh -c '
- for _ in 1 2 3; do
- docker push "{}" && exit 0
- echo "push of {} failed, retrying" >&2
- sleep 15
- done
- exit 1' < "$RUNNER_TEMP/push-refs.txt"
- fi
-
- push_image() {
- image_to_push=$1
- for _ in 1 2 3; do
- if docker push "$image_to_push"; then return 0; fi
- echo "docker push failed for $image_to_push, retrying" >&2
- sleep 15
- done
- return 1
- }
-
- if [ "$mode" = hot ]; then
- update_image="$registry:update-${{ github.sha }}"
- docker build -f preview-ctx/update.Dockerfile -t "$update_image" preview-ctx
- push_image "$update_image"
- token_file="$RUNNER_TEMP/registry-pull-token"
- printf '%s' "$pull_token" > "$token_file"
- chmod 600 "$token_file"
- hot_rc=0
- flyctl ssh sftp put "$token_file" /tmp/macro-registry-token \
- --app "$APP_NAME" --machine "$machine_id" --mode 0600 \
- || hot_rc=$?
- if [ "$hot_rc" = 0 ]; then
- # timeout backstop: a suspended-mid-session VM freezes the ssh
- # client forever (measured: 52 min of hang). 10 min is triple
- # a normal hot apply; on expiry we fall back to rehydrate.
- hot_command="/srv/macro/bin/hot-update '$update_image' /tmp/macro-registry-token"
- timeout 600 flyctl ssh console --app "$APP_NAME" --machine "$machine_id" \
- --command "$hot_command" || hot_rc=$?
- fi
- if [ "$hot_rc" = 0 ]; then
- echo "hot update completed without restarting the Fly machine"
- exit 0
- fi
- echo "hot update failed (exit $hot_rc); falling back to full rehydrate" >&2
- mode=rehydrate
- fi
-
- # Bootstrap/rehydrate refreshes app secrets and reconciles the
- # persistent Docker volume before replacing the machine image.
- flyctl secrets set --app "$APP_NAME" --stage \
- "DOPPLER_TOKEN=$DOPPLER_PREVIEW_TOKEN" \
- "REGISTRY_PULL_TOKEN=$pull_token"
- if [ -z "$vol_state" ]; then
- # No volume and no fork landed (no template, or both fork
- # attempts failed): the old empty-volume path.
- flyctl volumes create docker_data --app "$APP_NAME" --region ewr --size 40 --yes
- elif [ "$vol_state" != created ]; then
- # A fork is still hydrating (usually already finished: the
- # images lane kicked it before the cargo lane even ended). The
- # volume must reach "created" before a machine can mount it; a
- # fork that never lands is replaced by an empty volume rather
- # than failing the deploy.
- for _ in $(seq 1 60); do
- vol_state=$(flyctl volumes list --app "$APP_NAME" --json \
- | jq -r '[.[] | select(.name == "docker_data")][0].state // empty')
- [ "$vol_state" = created ] && break
- sleep 10
- done
- echo "seeded volume state: $vol_state"
- if [ "$vol_state" != created ]; then
- echo "template fork stuck (state: $vol_state); replacing with an empty volume" >&2
- vol_id=$(flyctl volumes list --app "$APP_NAME" --json \
- | jq -r '[.[] | select(.name == "docker_data")][0].id // empty')
- [ -z "$vol_id" ] || flyctl volumes destroy "$vol_id" --app "$APP_NAME" --yes || true
- flyctl volumes create docker_data --app "$APP_NAME" --region ewr --size 40 --yes
- fi
- fi
- # Machines created before the volume existed can't take the mounts
- # config update. A destroyed/partial machine is recreated by deploy.
- flyctl machine list --app "$APP_NAME" --json \
- | jq -r '.[] | select((.config.mounts // []) | length == 0) | .id' \
- | while read -r id; do
- [ -z "$id" ] || flyctl machine destroy "$id" --app "$APP_NAME" --force || true
- done
-
- image="$registry:${{ github.sha }}"
- docker build -t "$image" preview-ctx
- push_image "$image"
- # First boot does real work before 8090 opens (pulling the stack
- # images, snapshot restore, compose up, FusionAuth's JVM) —
- # give the health check more runway than flyctl's default wait.
- # The step-wide keepalive above holds the machine out of suspend.
- flyctl deploy --app "$APP_NAME" \
- --config infra/preview/fly.toml \
- --image "$image" \
- --wait-timeout 1800 \
- --yes
- "#})
- // Accept the org slug from either a repo variable or a repo secret —
- // it's not sensitive, but people reasonably reach for secrets first.
- .add_env(("FLY_ORG", "${{ vars.FLY_ORG || secrets.FLY_ORG }}"))
- .add_env(("DOPPLER_PREVIEW_TOKEN", vars::DOPPLER_PREVIEW_TOKEN))
- .add_env(("LOCAL_SANDBOX_IMAGE", vars::AGENT_HARNESS_LOCAL_IMAGE))
-}
-
-/// After a healthy deploy this PR's volume is, by construction, a fully
-/// warmed layer store for exactly the image set in the preload manifest
-/// (bootstrap pulled it, hot-update reconciled it). Publish it as the
-/// org-wide template that future first boots fork, content-addressed by the
-/// manifest's (image id, tag) pairs — registry refs are excluded because
-/// they embed the per-PR app name. An existing volume under the same key
-/// means the image set is unchanged and nothing runs; there is no scheduled
-/// refresh because the cache warms itself on exactly the deploys that
-/// change it. Purely an optimization, hence continue-on-error: the boot
-/// manifest check keeps forks of any template correct, stale or fresh.
-fn publish_template_volume() -> Step<Run> {
- Step::new("Publish template volume")
- .run(indoc::indoc! {r#"
- set -euo pipefail
- key=$(awk '{print $1, $2}' preview-ctx/preload/manifest.txt \
- | LC_ALL=C sort | sha256sum | cut -c1-24)
- vol_name="tpl$key"
- flyctl status --app "$TEMPLATE_APP" >/dev/null 2>&1 \
- || flyctl apps create "$TEMPLATE_APP" --org "$FLY_ORG"
- vols=$(flyctl volumes list --app "$TEMPLATE_APP" --json 2>/dev/null || echo '[]')
- if echo "$vols" | jq -e --arg n "$vol_name" \
- 'map(select(.name == $n)) | length > 0' >/dev/null; then
- echo "template $vol_name already exists; image set unchanged"
- else
- src=$(flyctl volumes list --app "$APP_NAME" --json \
- | jq -r '[.[] | select(.name == "docker_data")][0].id // empty')
- if [ -n "$src" ]; then
- # Fire-and-forget: hydration is a server-side block copy;
- # nothing in this run depends on it finishing.
- curl -sf -X POST "https://api.machines.dev/v1/apps/$TEMPLATE_APP/volumes" \
- -H "Authorization: Bearer $FLY_API_TOKEN" \
- -H "Content-Type: application/json" \
- -d "{\"name\":\"$vol_name\",\"region\":\"ewr\",\"source_volume_id\":\"$src\"}" \
- | jq -r '"published \(.name) (\(.id), state: \(.state))"' \
- || echo "template publish failed (non-fatal)" >&2
- fi
- fi
- # Prune superseded templates: keep the 3 newest, and never touch
- # anything younger than 2h — it may still be hydrating or serving
- # as a concurrent deploy's fork source.
- cutoff=$(date -u -d '2 hours ago' +%Y-%m-%dT%H:%M:%SZ)
- echo "$vols" | jq -r --arg cutoff "$cutoff" \
- '[.[] | select((.name | startswith("tpl")) and .attached_machine_id == null)]
- | sort_by(.created_at) | reverse | .[3:]
- | .[] | select(.created_at < $cutoff) | .id' \
- | while read -r old; do
- [ -z "$old" ] || flyctl volumes destroy "$old" --app "$TEMPLATE_APP" --yes || true
- done
- "#})
- .add_env(("FLY_ORG", "${{ vars.FLY_ORG || secrets.FLY_ORG }}"))
- .continue_on_error(true)
-}
-
-fn comment_preview_url() -> Step<Use> {
- Step::new("Comment preview URL")
- .uses(
- "actions",
- "github-script",
- "f28e40c7f34bde8b3046d885e986cb6290c5673b",
- )
- .add_with(("github-token", "${{ secrets.GITHUB_TOKEN }}"))
- .add_with((
- "script",
- indoc::indoc! {r#"
- const app = process.env.APP_NAME;
- const url = `https://${app}.fly.dev`;
- const marker = '<!-- fly-preview -->';
- const body = [
- marker,
- `🚀 **Full-stack preview**: ${url}`,
- '',
- `- Suspended when idle — the first request wakes it (a few seconds).`,
- `- Log in with any email; the passwordless code lands in [Mailpit](${url}/mailpit/).`,
- `- Deployed from ${context.payload.pull_request.head.sha.slice(0, 7)}.`,
- ].join('\n');
- const { data: comments } = await github.rest.issues.listComments({
- owner: context.repo.owner,
- repo: context.repo.repo,
- issue_number: context.issue.number,
- per_page: 100,
- });
- const existing = comments.find(c => c.body && c.body.startsWith(marker));
- if (existing) {
- await github.rest.issues.updateComment({
- owner: context.repo.owner,
- repo: context.repo.repo,
- comment_id: existing.id,
- body,
- });
- } else {
- await github.rest.issues.createComment({
- owner: context.repo.owner,
- repo: context.repo.repo,
- issue_number: context.issue.number,
- body,
- });
- }
- "#},
- ))
-}
-
-fn destroy() -> Job {
- Job::default()
- // Closed PRs always attempt cleanup (destroy is a no-op if no app);
- // unlabeled only cleans up when the removed label is `preview`.
- .cond(Expression::new(
- "github.event.action == 'closed' || github.event.label.name == 'preview'",
- ))
- .runs_on(runners::Runner::TinyNoCache.to_string())
- .permissions(Permissions::default().contents(Level::Read))
- .add_env(("FLY_API_TOKEN", vars::FLY_API_TOKEN))
- .add_step(setup_flyctl())
- .add_step(
- Step::new("Destroy app")
- .run(format!("flyctl apps destroy \"{APP_NAME}\" --yes || true")),
- )
-}
diff --git a/tooling/xtask/crates/xtask_workflows/src/workflows/preview_fly/test.rs b/tooling/xtask/crates/xtask_workflows/src/workflows/preview_fly/test.rs
deleted file mode 100644
index d497198130..0000000000
--- a/tooling/xtask/crates/xtask_workflows/src/workflows/preview_fly/test.rs
+++ /dev/null
@@ -1,87 +0,0 @@
-use super::*;
-
-fn rendered() -> String {
- preview_fly()
- .to_string()
- .expect("workflow should serialize")
-}
-
-#[test]
-fn deploy_does_not_need_ghcr_packages() {
- let yaml = rendered();
- assert!(
- !yaml.contains("packages: read"),
- "preview builds the sandbox image locally: {yaml}"
- );
-}
-
-#[test]
-fn both_image_loops_include_the_sandbox_tag() {
- let yaml = rendered();
- let tag = vars::AGENT_HARNESS_LOCAL_IMAGE;
- assert!(
- yaml.contains(&format!("LOCAL_SANDBOX_IMAGE: {tag}")),
- "sandbox tag should be injected as step env: {yaml}"
- );
- assert!(
- yaml.contains("alpine:3 \"$LOCAL_SANDBOX_IMAGE\""),
- "premirror loop should append the sandbox tag: {yaml}"
- );
- assert!(
- yaml.contains("$images alpine:3 \"$LOCAL_SANDBOX_IMAGE\""),
- "deploy loop should append the sandbox tag: {yaml}"
- );
-}
-
-#[test]
-fn deploy_timeout_covers_a_cold_sandbox_image_bake() {
- let yaml = rendered();
- assert!(
- yaml.contains("timeout-minutes: 90"),
- "sandbox image bake can exceed the old 60m backstop: {yaml}"
- );
-}
-
-#[test]
-fn images_lane_always_builds_the_sandbox_image() {
- let yaml = rendered();
- assert!(
- yaml.contains("docker build --tag \"$LOCAL_SANDBOX_IMAGE\" crates/agent_harness/container"),
- "{yaml}"
- );
- assert!(
- !yaml.contains("PREVIEW_BASE_SHA"),
- "git-diff vs base is unnecessary when we always build: {yaml}"
- );
- assert!(
- !yaml.contains("SANDBOX_GHCR_IMAGE"),
- "preview should not pull GHCR: {yaml}"
- );
-}
-
-#[test]
-fn web_lane_raises_the_node_heap() {
- let yaml = rendered();
- assert!(
- yaml.contains("NODE_OPTIONS=--max-old-space-size=6144"),
- "vite OOM is the documented CI flap without this: {yaml}"
- );
-}
-
-#[test]
-fn images_lane_builds_analytics_proxy() {
- let yaml = rendered();
- assert!(
- yaml.contains("ai_editing_worker analytics_proxy"),
- "analytics_proxy is a default compose service; omitting it makes deploy pull docker.io/library/macro-analytics_proxy: {yaml}"
- );
-}
-
-#[test]
-fn deploy_mirror_skips_unpullable_local_tags() {
- let yaml = rendered();
- assert!(
- yaml.contains("mirror: skipping $img (not local, not pullable)"),
- "deploy must skip compose-built tags the way premirror already does: {yaml}"
- );
-}
diff --git a/tooling/xtask/crates/xtask_workflows/src/workflows/steps.rs b/tooling/xtask/crates/xtask_workflows/src/workflows/steps.rs
index 635c30ae36..c3ddd0d0d1 100644
--- a/tooling/xtask/crates/xtask_workflows/src/workflows/steps.rs
+++ b/tooling/xtask/crates/xtask_workflows/src/workflows/steps.rs
@@ -48,9 +48,9 @@ pub(crate) fn uses_local(name: &str, path: RepoDir<'_>) -> Step<Use> {
step
}
-/// Namespace remote BuildKit, pinned. Multi-arch `docker buildx --push` and the
-/// Fly preview image builds both go through it so nix-in-Docker work does not
-/// land on a Small runner's local daemon.
+/// Namespace remote BuildKit, pinned. Multi-arch `docker buildx --push` goes
+/// through it so nix-in-Docker work does not land on a Small runner's local
+/// daemon.
pub fn setup_namespace_buildx() -> Step<Use> {
Step::new("Set up Namespace Docker builder").uses(
"namespacelabs",
@@ -201,31 +201,10 @@ fn nscloud_cache_action(name: &str) -> Step<Use> {
.continue_on_error(true)
}
-/// [`mount_cache_volume`] plus the checkout's cargo target dir and the init
-/// snapshot store. Persisting `target/` is what makes the preview job's
-/// zigbuild incremental — cargo's own fingerprints carry across runs, where
-/// remote sccache alone leaves build scripts, native (cmake/zig) compiles, and
-/// linking cold every time. Persisting the snapshot store gives the bake step
-/// a zero-copy fast path; Namespace artifact storage is its durable fallback.
-/// The volume is a block-device mount, so multi-GB trees cost nothing to save
-/// or restore when it hits.
-pub fn mount_cache_volume_with_cargo_target() -> Step<Use> {
- nscloud_cache_action("Mount Namespace cache volume")
- .add_with(("cache", "nix"))
- .add_with((
- "path",
- format!(
- "${{{{ github.workspace }}}}/target\n{}\n/home/runner/.cargo/registry\n/home/runner/.cargo/git",
- vars::PREVIEW_SNAPSHOT_VOLUME_DIR,
- ),
- ))
-}
-
/// [`mount_cache_volume`] for the wasm worker build. `target/` is listed
-/// explicitly for the same reason [`mount_cache_volume_with_cargo_target`] does
-/// it — `cache: rust` alone is only relied on for the registry/git here. The
-/// last two paths are what wrangler's `[build]` line otherwise redoes every
-/// run: the pinned `worker-build` binary and wasm-pack's downloaded `wasm-opt`.
+/// explicitly so cargo fingerprints survive across runs. The last two paths
+/// are what wrangler's `[build]` line otherwise redoes every run: the pinned
+/// `worker-build` binary and wasm-pack's downloaded `wasm-opt`.
pub fn mount_wasm_cache_volume() -> Step<Use> {
nscloud_cache_action("Mount Namespace cache volume")
.add_with(("cache", "rust"))
diff --git a/tooling/xtask/crates/xtask_workflows/src/workflows/vars.rs b/tooling/xtask/crates/xtask_workflows/src/workflows/vars.rs
index f63828e6de..49fa63c6bf 100644
--- a/tooling/xtask/crates/xtask_workflows/src/workflows/vars.rs
+++ b/tooling/xtask/crates/xtask_workflows/src/workflows/vars.rs
@@ -17,9 +17,7 @@ secret!(CLOUDFLARE_API_TOKEN);
secret!(DAYTONA_API_KEY);
secret!(DD_API_KEY);
secret!(DD_APP_KEY);
-secret!(DOPPLER_PREVIEW_TOKEN);
secret!(DOPPLER_TOKEN);
-secret!(FLY_API_TOKEN);
secret!(MACOS_DEVELOPER_ID_CERTIFICATE_BASE64);
secret!(MACOS_DEVELOPER_ID_CERTIFICATE_PASSWORD);
secret!(NIX_CACHE_SIGNING_KEY);
@@ -61,23 +59,6 @@ pub const CI_CACHE_TAG: &str = "sccache-ci";
/// the workspace dependency checks.
pub const CI_SCCACHE_NAME: &str = "sccache-ci";
-/// Namespace cache tag for the Fly preview deploy job. Its own pool, NOT
-/// [`CI_CACHE_TAG`]: sharing looked economical but was measured cold both ways
-/// (run 28968155599: sccache 2.65% hits, cargo target dir absent). The
-/// check/test jobs compile for the host while this job zigbuilds with
-/// `--target x86_64-unknown-linux-gnu.2.36`, so their sccache entries hash
-/// differently and never serve this job — and they don't persist the cargo
-/// target dir at all, so a volume from the shared pool almost never carries
-/// one. A dedicated low-concurrency pool gives the job's target dir and init
-/// snapshots the best chance of a zero-copy hit; Rust objects use the durable
-/// remote sccache below instead of depending on this volume's placement.
-pub const PREVIEW_CACHE_TAG: &str = "fly-preview";
-
-/// Durable Namespace remote sccache shared by Fly preview jobs. Unlike the
-/// local cache volume, this has an artifact-backed cold tier, so a job placed
-/// on a runner that has never seen the volume can still reuse Rust objects.
-pub const PREVIEW_SCCACHE_NAME: &str = "fly-preview";
-
/// Namespace cache tag for the web-app jobs (PR checks + preview deploys).
/// Cache volumes are keyed workspace-wide by tag alone, so a dedicated tag
/// gives the frontend its own volume — isolated both from the Rust CI volume
@@ -99,20 +80,10 @@ pub const SYNC_SERVICE_CACHE_TAG: &str = "sync-service-deploy";
/// the Nix dev shell and is not available to Namespace's cache planner yet.
pub const BUN_CACHE_VOLUME_DIR: &str = "/home/runner/.bun/install/cache";
-/// Init-snapshot store for the preview job (`MACRO_STACK_SNAPSHOT_DIR`). Lives
-/// on the preview cache volume for the zero-copy fast path; the workflow also
-/// backs each content-addressed snapshot up to Namespace artifact storage so a
-/// cache-volume miss does not force another infra bake.
-pub const PREVIEW_SNAPSHOT_VOLUME_DIR: &str = "/home/runner/.cache/macro-preview-snapshots";
-
/// GHCR repository for the agent-harness sandbox image (the same Dockerfile
/// Daytona snapshots). Pushed as `:$SHA` on PRs and `:$SHA` + `:latest` on main.
pub const AGENT_HARNESS_GHCR_IMAGE: &str = "ghcr.io/macro-inc/macro-agent-harness";
-/// Local Docker tag `just run_local` / Fly previews load. Keep in sync with
-/// `xtask_local::local::sandbox_image::DEFAULT_LOCAL_TAG`.
-pub const AGENT_HARNESS_LOCAL_IMAGE: &str = "macro-agent-harness:latest";
-
/// The repo-wide env block (mirrors the original top-level `env:`). Defaults the
/// linker to `lld`; the heavy jobs override `RUSTFLAGS` to use `mold`.
pub fn with_global_env(workflow: Workflow) -> Workflow {