The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu
Go agent-ai-cliagent-harnessesagent-securitybrowser-extensionciclaude-codecloudcodexcursordetection-engineeringjevknowledge-basememorymobile-device-managementobservabilitysecuritysecurity-information-and-event-managementsecurity-toolstelemetrytraces
ci: boot-check smoke images before publishing, keep rollbacks, prune and report (#1247)
kyle-elliott-asymptotea39afa3563eb
diff --git a/.github/workflows/smoke-images.yml b/.github/workflows/smoke-images.ymlindex 3aec7495..f95a192c 100644--- a/.github/workflows/smoke-images.yml+++ b/.github/workflows/smoke-images.yml@@ -8,8 +8,13 @@ name: Smoke images# - weekly, every image is rebuilt from a freshly pulled base under its existing tag, so the# smoke tests keep running on a patched base without a pull request to bump anything.#-# Until the first publish of a recipe, the smoke scripts build it themselves, as on a developer-# machine, so a pull request that changes a Dockerfile still tests the change.+# An image is published only once it boots, and every publish also carries a dated tag to roll back+# to (see smoke/image.sh). Until the first publish of a recipe, the smoke scripts build it+# themselves, as on a developer machine, so a pull request that changes a Dockerfile still tests it.+#+# Cost: the packages are public and so is this repository, so GHCR storage and transfer and these+# Actions minutes are free. The prune job bounds each package anyway (smoke/prune.sh), so that a+# change of visibility could not leave a year of weekly images to bill for.on:push:branches:@@ -28,6 +33,9 @@ concurrency:group: ${{ github.workflow }}cancel-in-progress: false+env:+ IMAGES: systemd package-deb package-rpm+jobs:publish:runs-on: ubuntu-latest@@ -45,7 +53,65 @@ jobs:run: echo "$GITHUB_TOKEN" | docker login ghcr.io -u "$GITHUB_ACTOR" --password-stdinenv:GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}- - name: Build and push ${{ matrix.image }}+ - name: Build, boot-check and push ${{ matrix.image }}run: |BEACON_SMOKE_REGISTRY="ghcr.io/$(echo "$GITHUB_REPOSITORY_OWNER" | tr '[:upper:]' '[:lower:]')" \sh packaging/linux/smoke/image.sh --publish "${{ matrix.image }}"++ # Deletes versions CI and a rollback no longer need; see smoke/prune.sh for what is kept. Only on+ # main: what it keeps is decided by the Dockerfiles in the checkout, and a branch's are not the ones+ # main's CI pulls. Runs after a failed publish too, since what it keeps does not depend on what+ # was just pushed. A version it cannot delete is a warning, not a failure.+ prune:+ needs: publish+ if: ${{ !cancelled() && github.ref == 'refs/heads/main' }}+ runs-on: ubuntu-latest+ timeout-minutes: 5+ permissions:+ contents: read+ packages: write+ steps:+ - uses: actions/checkout@v7+ - name: Delete versions no longer needed+ env:+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}+ # Word splitting of $IMAGES is the point: one argument per image.+ run: |+ # shellcheck disable=SC2086+ sh packaging/linux/smoke/prune.sh "$GITHUB_REPOSITORY_OWNER" $IMAGES++ # A failed publish on main has no reviewer watching, and GitHub mails a failed scheduled run only+ # to whoever last edited the cron line. One open issue tracks it instead: opened or commented on+ # when a publish fails, closed by the next run that publishes. A run dispatched from a branch+ # reports nothing, since what it publishes is the branch's business.+ report:+ needs: [publish, prune]+ if: ${{ !cancelled() && github.ref == 'refs/heads/main' }}+ runs-on: ubuntu-latest+ timeout-minutes: 5+ permissions:+ issues: write+ env:+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}+ GH_REPO: ${{ github.repository }}+ TITLE: "Smoke images: publish failing"+ RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}+ FAILED: ${{ needs.publish.result != 'success' }}+ steps:+ - name: Open, update or close the tracking issue+ run: |+ set -eu+ open="$(gh issue list --state open --search "in:title \"$TITLE\"" --json number,title \+ --jq "map(select(.title == \"$TITLE\")) | .[0].number // empty")"+ if [ "$FAILED" = true ]; then+ body="The smoke image workflow failed: $RUN_URL++ CI keeps pulling the last published images, so pull requests are not affected until a Dockerfile changes, but the images are not getting base updates. A distribution mirror outage is the usual cause; re-run the workflow once it is over."+ if [ -n "$open" ]; then+ gh issue comment "$open" --body "Failed again: $RUN_URL"+ else+ gh issue create --title "$TITLE" --body "$body"+ fi+ elif [ -n "$open" ]; then+ gh issue close "$open" --comment "Published successfully: $RUN_URL"+ fidiff --git a/CLAUDE.md b/CLAUDE.mdindex 11689293..bf8eca33 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -744,7 +744,7 @@ developer mode.- `terraform` (ubuntu): `terraform fmt -check`, `validate`, and `terraform test` (mocked providers) for every module under `terraform/`, and `validate` for each example.- `cross-build` (ubuntu): builds all five release targets and asserts the Linux binaries are statically linked (no `PT_INTERP`).- `package-smoke` (macOS): `packaging/macos/test-endpoint-scripts.sh`, a `build-pkg.sh` payload check, and `packaging/macos/smoke-endpoint.sh`.-- `linux-systemd-smoke` and `linux-package-smoke` (ubuntu): the real systemd install path, and installing the built `.deb` and `.rpm` in a systemd container. The containers come prebuilt from GHCR (`ghcr.io/asymptote-labs/beacon-smoke-*`), published by `smoke-images.yml` from `packaging/linux/smoke/*.Dockerfile` on a change to `main` and weekly from a fresh base, and tagged by a digest of the Dockerfile; `packaging/linux/smoke/image.sh` pulls the tag and builds locally when it is not published yet. So these jobs never run apt or dnf against a distribution mirror.+- `linux-systemd-smoke` and `linux-package-smoke` (ubuntu): the real systemd install path, and installing the built `.deb` and `.rpm` in a systemd container. The containers come prebuilt from GHCR (`ghcr.io/asymptote-labs/beacon-smoke-*`, public), published by `smoke-images.yml` from `packaging/linux/smoke/*.Dockerfile` on a change to `main` and weekly from a fresh base, and tagged by a digest of the Dockerfile; `packaging/linux/smoke/image.sh` pulls the tag and builds locally when it is not published yet. So these jobs never run apt or dnf against a distribution mirror. A publish moves the tag only after the image boots systemd, and also pushes `<digest>-YYYYMMDD` to roll back to (`image.sh` shows the command). On `main` only, the workflow prunes by recipe (`packaging/linux/smoke/prune.sh`: the newest four versions of the checked-out Dockerfile's digest, and other digests or untagged versions after 30 days) and tracks a failed publish in one open issue titled `Smoke images: publish failing`.- `windows-package-smoke` (windows-2025): `packaging/windows/test-endpoint-scripts.ps1`, then fetching the pinned Vector, building the MSI and installing it.The Go jobs build the embedded hooks binary first (`make build-hooks-current`); `cli/beacon` testsdiff --git a/packaging/linux/smoke/image.sh b/packaging/linux/smoke/image.shindex b69ad389..35aa0d4f 100755--- a/packaging/linux/smoke/image.sh+++ b/packaging/linux/smoke/image.sh@@ -2,7 +2,8 @@# Resolves the container image a Linux smoke test runs in, and prints its reference on stdout.## sh packaging/linux/smoke/image.sh NAME # NAME is a Dockerfile here: systemd, package-deb, package-rpm-# sh packaging/linux/smoke/image.sh --publish NAME # build with a fresh base and push to the registry+# sh packaging/linux/smoke/image.sh --publish NAME # build with a fresh base, boot-check, push to the registry+# sh packaging/linux/smoke/image.sh --digest NAME # print the tag NAME's Dockerfile is published under## The tag is a digest of the Dockerfile, so an image is only ever reused for the recipe it was built# from. With BEACON_SMOKE_REGISTRY set (CI sets ghcr.io/<owner>), the prebuilt image is pulled from@@ -12,16 +13,26 @@# is built here, as it is on a developer machine. BEACON_SMOKE_BASE replaces the base image and always# builds locally, since the registry holds only the default bases.#+# --publish moves the tag CI pulls only after the new image has booted systemd to a running state, so+# a broken base update stays out of every smoke job instead of failing them all until someone+# notices. Each publish also carries a dated tag, <digest>-YYYYMMDD, which is what a rollback points+# the plain tag back at:+#+# docker buildx imagetools create -t ghcr.io/<owner>/beacon-smoke-NAME:<digest> \+# ghcr.io/<owner>/beacon-smoke-NAME:<digest>-<date>+## The build output goes to stderr in full, so a mirror failure reads as one rather than as a bare# `exit code: 100`.set -eu+mode=resolve+case "${1:-}" in+ --publish) mode=publish; shift ;;+ --digest) mode=digest; shift ;;+esacpublish=0-if [ "${1:-}" = "--publish" ]; then- publish=1- shift-fi-name="${1:?usage: image.sh [--publish] NAME}"+[ "$mode" = publish ] && publish=1+name="${1:?usage: image.sh [--publish|--digest] NAME}"dir="$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)"dockerfile="$dir/$name.Dockerfile"[ -f "$dockerfile" ] || { echo "image.sh: no $dockerfile" >&2; exit 1; }@@ -31,21 +42,52 @@ if command -v sha256sum >/dev/null 2>&1; thenelsedigest="$(shasum -a 256 "$dockerfile" | cut -c1-16)"fi+if [ "$mode" = digest ]; then+ echo "$digest"+ exit 0+firepo="beacon-smoke-$name"registry="${BEACON_SMOKE_REGISTRY:-}"base="${BEACON_SMOKE_BASE:-}"+# boot_check IMAGE: systemd comes up as PID 1 in IMAGE and reaches running (or degraded, which a+# container without every kernel facility always is), the state each smoke test waits for first.+boot_check() {+ check="beacon-smoke-check-$$"+ docker rm -f "$check" >/dev/null 2>&1 || true+ docker run -d --name "$check" --platform "$platform" --privileged --cgroupns=host \+ -v /sys/fs/cgroup:/sys/fs/cgroup:rw --tmpfs /run --tmpfs /run/lock "$1" >/dev/null+ i=0 state=+ while [ "$i" -lt 60 ]; do+ state="$(docker exec "$check" systemctl is-system-running 2>/dev/null || true)"+ case "$state" in running|degraded) break ;; esac+ i=$((i+1)); sleep 1+ done+ pid1="$(docker exec "$check" cat /proc/1/comm 2>/dev/null || true)"+ docker rm -f "$check" >/dev/null 2>&1 || true+ case "$state" in running|degraded) ;; *)+ echo "image.sh: $1 did not boot (systemd state '$state', PID 1 '$pid1'); not publishing it" >&2+ return 1 ;;+ esac+ [ "$pid1" = systemd ] || { echo "image.sh: $1 has PID 1 '$pid1', not systemd; not publishing it" >&2; return 1; }+}+if [ "$publish" = 1 ]; then[ -n "$registry" ] || { echo "image.sh: --publish needs BEACON_SMOKE_REGISTRY" >&2; exit 1; }ref="$registry/$repo:$digest"+ dated="$ref-$(date -u +%Y%m%d)"+ # linux/amd64 is what the CI runners are, whatever machine publishes.+ platform="${BEACON_SMOKE_PLATFORM:-linux/amd64}"# --pull so the weekly rebuild picks up the base image's security updates under the same tag.- # linux/amd64 is what the CI runners are, whatever machine publishes. The source label links the- # GHCR package to the repository, which is what lets a workflow's job token pull a private one;- # GHCR reads it only from a single image manifest, so no provenance index is pushed around it.- # Publishing needs BuildKit (buildx), which GitHub's runners have.+ # The source label links the GHCR package to the repository, which is what lets a workflow's job+ # token pull and prune it; GHCR reads it only from a single image manifest, so no provenance index+ # is pushed around it. Publishing needs BuildKit (buildx), which GitHub's runners have.source="${GITHUB_SERVER_URL:-https://github.com}/${GITHUB_REPOSITORY:-asymptote-labs/agent-beacon}"- docker build --pull --platform "${BEACON_SMOKE_PLATFORM:-linux/amd64}" --provenance=false \- --label "org.opencontainers.image.source=$source" -t "$ref" -f "$dockerfile" "$dir" >&2+ docker build --pull --platform "$platform" --provenance=false \+ --label "org.opencontainers.image.source=$source" -t "$dated" -t "$ref" -f "$dockerfile" "$dir" >&2+ boot_check "$dated"+ # Dated tag first: if the second push fails, CI still pulls the previous image.+ docker push "$dated" >&2docker push "$ref" >&2echo "$ref"exit 0diff --git a/packaging/linux/smoke/package-deb.Dockerfile b/packaging/linux/smoke/package-deb.Dockerfileindex dd19531d..d6845cc9 100644--- a/packaging/linux/smoke/package-deb.Dockerfile+++ b/packaging/linux/smoke/package-deb.Dockerfile@@ -3,6 +3,8 @@# from GHCR (see .github/workflows/smoke-images.yml).ARG BASE=ubuntu:24.04FROM ${BASE}-RUN apt-get update -qq && apt-get install -y -qq systemd systemd-sysv dbus libpam-systemd curl ca-certificates iproute2 binutils >/dev/null+# --error-on=any: a failed index download fails here, naming the mirror, rather than surfacing later+# as "Unable to locate package".+RUN apt-get update -qq --error-on=any && apt-get install -y -qq systemd systemd-sysv dbus libpam-systemd curl ca-certificates iproute2 binutils >/dev/nullSTOPSIGNAL SIGRTMIN+3CMD ["/sbin/init"]diff --git a/packaging/linux/smoke/prune.sh b/packaging/linux/smoke/prune.shnew file mode 100755index 00000000..52f629eb--- /dev/null+++ b/packaging/linux/smoke/prune.sh@@ -0,0 +1,57 @@+#!/bin/sh+# Deletes old versions of the smoke image packages on GHCR, keeping what CI and a rollback need.+#+# GH_TOKEN=... sh packaging/linux/smoke/prune.sh OWNER NAME...+#+# The packages hold versions of more than one recipe: each Dockerfile change publishes under a new+# digest tag, and a workflow dispatched from a branch publishes that branch's recipe. So versions are+# judged by the recipe they belong to, never by age alone, which could let a few branch publishes+# push out the image main pulls:+#+# - the recipe in this checkout (the one CI on this ref pulls): its newest KEEP versions stay -- the+# current image and the dated ones a rollback points at -- and the older ones go;+# - any other recipe, and an untagged version: kept for GRACE_DAYS, so a pull request that changed+# a Dockerfile, or a branch that has not caught up, still pulls its image; deleted after that.+#+# Run it from main only: on a branch, "the recipe in this checkout" is the branch's, and main's+# would count as another recipe. A version GHCR refuses to delete (it does for a public version with+# more than 5,000 downloads) is reported and skipped; storage of a public package costs nothing.+set -eu++owner="${1:?usage: prune.sh OWNER NAME...}"+shift+KEEP="${KEEP:-4}"+GRACE_DAYS="${GRACE_DAYS:-30}"+dir="$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)"+cutoff="$(date -u -d "$GRACE_DAYS days ago" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \+ || date -u -v-"$GRACE_DAYS"d +%Y-%m-%dT%H:%M:%SZ)"+tab="$(printf '\t')"++for name in "$@"; do+ pkg="beacon-smoke-$name"+ current="$(sh "$dir/image.sh" --digest "$name")"+ if ! versions="$(gh api --paginate "orgs/$owner/packages/container/$pkg/versions" \+ --jq '.[] | "\(.created_at)\t\(.id)\t\(.metadata.container.tags | join(" "))"')"; then+ echo "::warning::$pkg: cannot list versions; not pruned"+ continue+ fi+ kept=0+ # Newest first, so the first KEEP versions of the current recipe are the ones kept.+ printf '%s\n' "$versions" | sort -r | while IFS="$tab" read -r created id tags; do+ [ -n "$id" ] || continue+ case " $tags " in+ *" $current "* | *" $current-"*)+ kept=$((kept + 1))+ [ "$kept" -gt "$KEEP" ] || continue+ why="older than the newest $KEEP of recipe $current" ;;+ *)+ [ "$created" \< "$cutoff" ] || continue+ why="another recipe or untagged, older than $GRACE_DAYS days" ;;+ esac+ if gh api -X DELETE "orgs/$owner/packages/container/$pkg/versions/$id" >/dev/null; then+ echo "$pkg: deleted version $id ($created; ${tags:-untagged}): $why"+ else+ echo "::warning::$pkg: could not delete version $id ($created; ${tags:-untagged}); left in place"+ fi+ done+donediff --git a/packaging/linux/smoke/systemd.Dockerfile b/packaging/linux/smoke/systemd.Dockerfileindex 1b932da2..e8bfc89a 100644--- a/packaging/linux/smoke/systemd.Dockerfile+++ b/packaging/linux/smoke/systemd.Dockerfile@@ -2,6 +2,8 @@# PID 1. Built by image.sh, which CI pulls prebuilt from GHCR (see .github/workflows/smoke-images.yml).ARG BASE=ubuntu:24.04FROM ${BASE}-RUN apt-get update -qq && apt-get install -y -qq systemd systemd-sysv dbus curl ca-certificates iproute2 >/dev/null+# --error-on=any: a failed index download fails here, naming the mirror, rather than surfacing later+# as "Unable to locate package".+RUN apt-get update -qq --error-on=any && apt-get install -y -qq systemd systemd-sysv dbus curl ca-certificates iproute2 >/dev/nullSTOPSIGNAL SIGRTMIN+3CMD ["/sbin/init"]