Skip to content

The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu

ci: boot-check smoke images before publishing, keep rollbacks, prune and report (#1247)

kyle-elliott-asymptotea39afa3563eb
diff --git a/.github/workflows/smoke-images.yml b/.github/workflows/smoke-images.yml
index 3aec7495..f95a192c 100644
--- a/.github/workflows/smoke-images.yml
+++ b/.github/workflows/smoke-images.yml
@@ -8,8 +8,13 @@ name: Smoke images
# - weekly, every image is rebuilt from a freshly pulled base under its existing tag, so the
# smoke tests keep running on a patched base without a pull request to bump anything.
#
-# Until the first publish of a recipe, the smoke scripts build it themselves, as on a developer
-# machine, so a pull request that changes a Dockerfile still tests the change.
+# An image is published only once it boots, and every publish also carries a dated tag to roll back
+# to (see smoke/image.sh). Until the first publish of a recipe, the smoke scripts build it
+# themselves, as on a developer machine, so a pull request that changes a Dockerfile still tests it.
+#
+# Cost: the packages are public and so is this repository, so GHCR storage and transfer and these
+# Actions minutes are free. The prune job bounds each package anyway (smoke/prune.sh), so that a
+# change of visibility could not leave a year of weekly images to bill for.
on:
push:
branches:
@@ -28,6 +33,9 @@ concurrency:
group: ${{ github.workflow }}
cancel-in-progress: false
+env:
+ IMAGES: systemd package-deb package-rpm
+
jobs:
publish:
runs-on: ubuntu-latest
@@ -45,7 +53,65 @@ jobs:
run: echo "$GITHUB_TOKEN" | docker login ghcr.io -u "$GITHUB_ACTOR" --password-stdin
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- - name: Build and push ${{ matrix.image }}
+ - name: Build, boot-check and push ${{ matrix.image }}
run: |
BEACON_SMOKE_REGISTRY="ghcr.io/$(echo "$GITHUB_REPOSITORY_OWNER" | tr '[:upper:]' '[:lower:]')" \
sh packaging/linux/smoke/image.sh --publish "${{ matrix.image }}"
+
+ # Deletes versions CI and a rollback no longer need; see smoke/prune.sh for what is kept. Only on
+ # main: what it keeps is decided by the Dockerfiles in the checkout, and a branch's are not the ones
+ # main's CI pulls. Runs after a failed publish too, since what it keeps does not depend on what
+ # was just pushed. A version it cannot delete is a warning, not a failure.
+ prune:
+ needs: publish
+ if: ${{ !cancelled() && github.ref == 'refs/heads/main' }}
+ runs-on: ubuntu-latest
+ timeout-minutes: 5
+ permissions:
+ contents: read
+ packages: write
+ steps:
+ - uses: actions/checkout@v7
+ - name: Delete versions no longer needed
+ env:
+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ # Word splitting of $IMAGES is the point: one argument per image.
+ run: |
+ # shellcheck disable=SC2086
+ sh packaging/linux/smoke/prune.sh "$GITHUB_REPOSITORY_OWNER" $IMAGES
+
+ # A failed publish on main has no reviewer watching, and GitHub mails a failed scheduled run only
+ # to whoever last edited the cron line. One open issue tracks it instead: opened or commented on
+ # when a publish fails, closed by the next run that publishes. A run dispatched from a branch
+ # reports nothing, since what it publishes is the branch's business.
+ report:
+ needs: [publish, prune]
+ if: ${{ !cancelled() && github.ref == 'refs/heads/main' }}
+ runs-on: ubuntu-latest
+ timeout-minutes: 5
+ permissions:
+ issues: write
+ env:
+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ GH_REPO: ${{ github.repository }}
+ TITLE: "Smoke images: publish failing"
+ RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+ FAILED: ${{ needs.publish.result != 'success' }}
+ steps:
+ - name: Open, update or close the tracking issue
+ run: |
+ set -eu
+ open="$(gh issue list --state open --search "in:title \"$TITLE\"" --json number,title \
+ --jq "map(select(.title == \"$TITLE\")) | .[0].number // empty")"
+ if [ "$FAILED" = true ]; then
+ body="The smoke image workflow failed: $RUN_URL
+
+ CI keeps pulling the last published images, so pull requests are not affected until a Dockerfile changes, but the images are not getting base updates. A distribution mirror outage is the usual cause; re-run the workflow once it is over."
+ if [ -n "$open" ]; then
+ gh issue comment "$open" --body "Failed again: $RUN_URL"
+ else
+ gh issue create --title "$TITLE" --body "$body"
+ fi
+ elif [ -n "$open" ]; then
+ gh issue close "$open" --comment "Published successfully: $RUN_URL"
+ fi
diff --git a/CLAUDE.md b/CLAUDE.md
index 11689293..bf8eca33 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -744,7 +744,7 @@ developer mode.
- `terraform` (ubuntu): `terraform fmt -check`, `validate`, and `terraform test` (mocked providers) for every module under `terraform/`, and `validate` for each example.
- `cross-build` (ubuntu): builds all five release targets and asserts the Linux binaries are statically linked (no `PT_INTERP`).
- `package-smoke` (macOS): `packaging/macos/test-endpoint-scripts.sh`, a `build-pkg.sh` payload check, and `packaging/macos/smoke-endpoint.sh`.
-- `linux-systemd-smoke` and `linux-package-smoke` (ubuntu): the real systemd install path, and installing the built `.deb` and `.rpm` in a systemd container. The containers come prebuilt from GHCR (`ghcr.io/asymptote-labs/beacon-smoke-*`), published by `smoke-images.yml` from `packaging/linux/smoke/*.Dockerfile` on a change to `main` and weekly from a fresh base, and tagged by a digest of the Dockerfile; `packaging/linux/smoke/image.sh` pulls the tag and builds locally when it is not published yet. So these jobs never run apt or dnf against a distribution mirror.
+- `linux-systemd-smoke` and `linux-package-smoke` (ubuntu): the real systemd install path, and installing the built `.deb` and `.rpm` in a systemd container. The containers come prebuilt from GHCR (`ghcr.io/asymptote-labs/beacon-smoke-*`, public), published by `smoke-images.yml` from `packaging/linux/smoke/*.Dockerfile` on a change to `main` and weekly from a fresh base, and tagged by a digest of the Dockerfile; `packaging/linux/smoke/image.sh` pulls the tag and builds locally when it is not published yet. So these jobs never run apt or dnf against a distribution mirror. A publish moves the tag only after the image boots systemd, and also pushes `<digest>-YYYYMMDD` to roll back to (`image.sh` shows the command). On `main` only, the workflow prunes by recipe (`packaging/linux/smoke/prune.sh`: the newest four versions of the checked-out Dockerfile's digest, and other digests or untagged versions after 30 days) and tracks a failed publish in one open issue titled `Smoke images: publish failing`.
- `windows-package-smoke` (windows-2025): `packaging/windows/test-endpoint-scripts.ps1`, then fetching the pinned Vector, building the MSI and installing it.
The Go jobs build the embedded hooks binary first (`make build-hooks-current`); `cli/beacon` tests
diff --git a/packaging/linux/smoke/image.sh b/packaging/linux/smoke/image.sh
index b69ad389..35aa0d4f 100755
--- a/packaging/linux/smoke/image.sh
+++ b/packaging/linux/smoke/image.sh
@@ -2,7 +2,8 @@
# Resolves the container image a Linux smoke test runs in, and prints its reference on stdout.
#
# sh packaging/linux/smoke/image.sh NAME # NAME is a Dockerfile here: systemd, package-deb, package-rpm
-# sh packaging/linux/smoke/image.sh --publish NAME # build with a fresh base and push to the registry
+# sh packaging/linux/smoke/image.sh --publish NAME # build with a fresh base, boot-check, push to the registry
+# sh packaging/linux/smoke/image.sh --digest NAME # print the tag NAME's Dockerfile is published under
#
# The tag is a digest of the Dockerfile, so an image is only ever reused for the recipe it was built
# from. With BEACON_SMOKE_REGISTRY set (CI sets ghcr.io/<owner>), the prebuilt image is pulled from
@@ -12,16 +13,26 @@
# is built here, as it is on a developer machine. BEACON_SMOKE_BASE replaces the base image and always
# builds locally, since the registry holds only the default bases.
#
+# --publish moves the tag CI pulls only after the new image has booted systemd to a running state, so
+# a broken base update stays out of every smoke job instead of failing them all until someone
+# notices. Each publish also carries a dated tag, <digest>-YYYYMMDD, which is what a rollback points
+# the plain tag back at:
+#
+# docker buildx imagetools create -t ghcr.io/<owner>/beacon-smoke-NAME:<digest> \
+# ghcr.io/<owner>/beacon-smoke-NAME:<digest>-<date>
+#
# The build output goes to stderr in full, so a mirror failure reads as one rather than as a bare
# `exit code: 100`.
set -eu
+mode=resolve
+case "${1:-}" in
+ --publish) mode=publish; shift ;;
+ --digest) mode=digest; shift ;;
+esac
publish=0
-if [ "${1:-}" = "--publish" ]; then
- publish=1
- shift
-fi
-name="${1:?usage: image.sh [--publish] NAME}"
+[ "$mode" = publish ] && publish=1
+name="${1:?usage: image.sh [--publish|--digest] NAME}"
dir="$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)"
dockerfile="$dir/$name.Dockerfile"
[ -f "$dockerfile" ] || { echo "image.sh: no $dockerfile" >&2; exit 1; }
@@ -31,21 +42,52 @@ if command -v sha256sum >/dev/null 2>&1; then
else
digest="$(shasum -a 256 "$dockerfile" | cut -c1-16)"
fi
+if [ "$mode" = digest ]; then
+ echo "$digest"
+ exit 0
+fi
repo="beacon-smoke-$name"
registry="${BEACON_SMOKE_REGISTRY:-}"
base="${BEACON_SMOKE_BASE:-}"
+# boot_check IMAGE: systemd comes up as PID 1 in IMAGE and reaches running (or degraded, which a
+# container without every kernel facility always is), the state each smoke test waits for first.
+boot_check() {
+ check="beacon-smoke-check-$$"
+ docker rm -f "$check" >/dev/null 2>&1 || true
+ docker run -d --name "$check" --platform "$platform" --privileged --cgroupns=host \
+ -v /sys/fs/cgroup:/sys/fs/cgroup:rw --tmpfs /run --tmpfs /run/lock "$1" >/dev/null
+ i=0 state=
+ while [ "$i" -lt 60 ]; do
+ state="$(docker exec "$check" systemctl is-system-running 2>/dev/null || true)"
+ case "$state" in running|degraded) break ;; esac
+ i=$((i+1)); sleep 1
+ done
+ pid1="$(docker exec "$check" cat /proc/1/comm 2>/dev/null || true)"
+ docker rm -f "$check" >/dev/null 2>&1 || true
+ case "$state" in running|degraded) ;; *)
+ echo "image.sh: $1 did not boot (systemd state '$state', PID 1 '$pid1'); not publishing it" >&2
+ return 1 ;;
+ esac
+ [ "$pid1" = systemd ] || { echo "image.sh: $1 has PID 1 '$pid1', not systemd; not publishing it" >&2; return 1; }
+}
+
if [ "$publish" = 1 ]; then
[ -n "$registry" ] || { echo "image.sh: --publish needs BEACON_SMOKE_REGISTRY" >&2; exit 1; }
ref="$registry/$repo:$digest"
+ dated="$ref-$(date -u +%Y%m%d)"
+ # linux/amd64 is what the CI runners are, whatever machine publishes.
+ platform="${BEACON_SMOKE_PLATFORM:-linux/amd64}"
# --pull so the weekly rebuild picks up the base image's security updates under the same tag.
- # linux/amd64 is what the CI runners are, whatever machine publishes. The source label links the
- # GHCR package to the repository, which is what lets a workflow's job token pull a private one;
- # GHCR reads it only from a single image manifest, so no provenance index is pushed around it.
- # Publishing needs BuildKit (buildx), which GitHub's runners have.
+ # The source label links the GHCR package to the repository, which is what lets a workflow's job
+ # token pull and prune it; GHCR reads it only from a single image manifest, so no provenance index
+ # is pushed around it. Publishing needs BuildKit (buildx), which GitHub's runners have.
source="${GITHUB_SERVER_URL:-https://github.com}/${GITHUB_REPOSITORY:-asymptote-labs/agent-beacon}"
- docker build --pull --platform "${BEACON_SMOKE_PLATFORM:-linux/amd64}" --provenance=false \
- --label "org.opencontainers.image.source=$source" -t "$ref" -f "$dockerfile" "$dir" >&2
+ docker build --pull --platform "$platform" --provenance=false \
+ --label "org.opencontainers.image.source=$source" -t "$dated" -t "$ref" -f "$dockerfile" "$dir" >&2
+ boot_check "$dated"
+ # Dated tag first: if the second push fails, CI still pulls the previous image.
+ docker push "$dated" >&2
docker push "$ref" >&2
echo "$ref"
exit 0
diff --git a/packaging/linux/smoke/package-deb.Dockerfile b/packaging/linux/smoke/package-deb.Dockerfile
index dd19531d..d6845cc9 100644
--- a/packaging/linux/smoke/package-deb.Dockerfile
+++ b/packaging/linux/smoke/package-deb.Dockerfile
@@ -3,6 +3,8 @@
# from GHCR (see .github/workflows/smoke-images.yml).
ARG BASE=ubuntu:24.04
FROM ${BASE}
-RUN apt-get update -qq && apt-get install -y -qq systemd systemd-sysv dbus libpam-systemd curl ca-certificates iproute2 binutils >/dev/null
+# --error-on=any: a failed index download fails here, naming the mirror, rather than surfacing later
+# as "Unable to locate package".
+RUN apt-get update -qq --error-on=any && apt-get install -y -qq systemd systemd-sysv dbus libpam-systemd curl ca-certificates iproute2 binutils >/dev/null
STOPSIGNAL SIGRTMIN+3
CMD ["/sbin/init"]
diff --git a/packaging/linux/smoke/prune.sh b/packaging/linux/smoke/prune.sh
new file mode 100755
index 00000000..52f629eb
--- /dev/null
+++ b/packaging/linux/smoke/prune.sh
@@ -0,0 +1,57 @@
+#!/bin/sh
+# Deletes old versions of the smoke image packages on GHCR, keeping what CI and a rollback need.
+#
+# GH_TOKEN=... sh packaging/linux/smoke/prune.sh OWNER NAME...
+#
+# The packages hold versions of more than one recipe: each Dockerfile change publishes under a new
+# digest tag, and a workflow dispatched from a branch publishes that branch's recipe. So versions are
+# judged by the recipe they belong to, never by age alone, which could let a few branch publishes
+# push out the image main pulls:
+#
+# - the recipe in this checkout (the one CI on this ref pulls): its newest KEEP versions stay -- the
+# current image and the dated ones a rollback points at -- and the older ones go;
+# - any other recipe, and an untagged version: kept for GRACE_DAYS, so a pull request that changed
+# a Dockerfile, or a branch that has not caught up, still pulls its image; deleted after that.
+#
+# Run it from main only: on a branch, "the recipe in this checkout" is the branch's, and main's
+# would count as another recipe. A version GHCR refuses to delete (it does for a public version with
+# more than 5,000 downloads) is reported and skipped; storage of a public package costs nothing.
+set -eu
+
+owner="${1:?usage: prune.sh OWNER NAME...}"
+shift
+KEEP="${KEEP:-4}"
+GRACE_DAYS="${GRACE_DAYS:-30}"
+dir="$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)"
+cutoff="$(date -u -d "$GRACE_DAYS days ago" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \
+ || date -u -v-"$GRACE_DAYS"d +%Y-%m-%dT%H:%M:%SZ)"
+tab="$(printf '\t')"
+
+for name in "$@"; do
+ pkg="beacon-smoke-$name"
+ current="$(sh "$dir/image.sh" --digest "$name")"
+ if ! versions="$(gh api --paginate "orgs/$owner/packages/container/$pkg/versions" \
+ --jq '.[] | "\(.created_at)\t\(.id)\t\(.metadata.container.tags | join(" "))"')"; then
+ echo "::warning::$pkg: cannot list versions; not pruned"
+ continue
+ fi
+ kept=0
+ # Newest first, so the first KEEP versions of the current recipe are the ones kept.
+ printf '%s\n' "$versions" | sort -r | while IFS="$tab" read -r created id tags; do
+ [ -n "$id" ] || continue
+ case " $tags " in
+ *" $current "* | *" $current-"*)
+ kept=$((kept + 1))
+ [ "$kept" -gt "$KEEP" ] || continue
+ why="older than the newest $KEEP of recipe $current" ;;
+ *)
+ [ "$created" \< "$cutoff" ] || continue
+ why="another recipe or untagged, older than $GRACE_DAYS days" ;;
+ esac
+ if gh api -X DELETE "orgs/$owner/packages/container/$pkg/versions/$id" >/dev/null; then
+ echo "$pkg: deleted version $id ($created; ${tags:-untagged}): $why"
+ else
+ echo "::warning::$pkg: could not delete version $id ($created; ${tags:-untagged}); left in place"
+ fi
+ done
+done
diff --git a/packaging/linux/smoke/systemd.Dockerfile b/packaging/linux/smoke/systemd.Dockerfile
index 1b932da2..e8bfc89a 100644
--- a/packaging/linux/smoke/systemd.Dockerfile
+++ b/packaging/linux/smoke/systemd.Dockerfile
@@ -2,6 +2,8 @@
# PID 1. Built by image.sh, which CI pulls prebuilt from GHCR (see .github/workflows/smoke-images.yml).
ARG BASE=ubuntu:24.04
FROM ${BASE}
-RUN apt-get update -qq && apt-get install -y -qq systemd systemd-sysv dbus curl ca-certificates iproute2 >/dev/null
+# --error-on=any: a failed index download fails here, naming the mirror, rather than surfacing later
+# as "Unable to locate package".
+RUN apt-get update -qq --error-on=any && apt-get install -y -qq systemd systemd-sysv dbus curl ca-certificates iproute2 >/dev/null
STOPSIGNAL SIGRTMIN+3
CMD ["/sbin/init"]