Database governance built for humans and agents — controlling changes and access across every major database.
Go cicddata-governancedata-maskingdata-securitydatabase-accessdatabase-governancedevopsdevsecopsflywaygitopsliquibasemongodbmysqloraclepostgresqlschema-migrationssecuritysql-clientsqlservertidb
feat(infra): build the devbox, and point CI at it (#21332)
Danny Xu2ed531acdc7f
diff --git a/.github/workflows/start-runner-vm.yml b/.github/workflows/start-runner-vm.ymlindex 7bd1ce69ff..22edc07d37 100644--- a/.github/workflows/start-runner-vm.yml+++ b/.github/workflows/start-runner-vm.yml@@ -1,13 +1,15 @@-# Starts the github-runner GCE VM, which hosts this repo's self-hosted Actions-# runner (used by jobs with `runs-on: self-hosted`, e.g. backend-tests,-# frontend-tests, golangci-lint, test_link).-name: Start github-runner VM+# Starts the devbox GCE VM, which hosts the org's self-hosted Actions runners+# (used by jobs with `runs-on: self-hosted`, e.g. backend-tests, frontend-tests,+# golangci-lint, test_link). It is a Spot instance, so a preemption stops it and+# this workflow is what brings it back. Design: docs/design/devbox-and-ci-runner.md+name: Start devbox VMon:workflow_dispatch:schedule:- # Three times a day, at 21:00, 02:00 and 06:00 UTC.- - cron: "0 2,6,21 * * *"+ # Every 15 minutes: that is the ceiling on how long a preemption leaves CI+ # without runners. Starting an already-running instance is a no-op.+ - cron: "*/15 * * * *"permissions:contents: read@@ -24,12 +26,12 @@ jobs:- name: Set up Cloud SDKuses: google-github-actions/setup-gcloud@v3- - name: Start github-runner VM if stopped+ - name: Start devbox VM if stoppedrun: |- STATUS=$(gcloud compute instances describe github-runner \- --zone southamerica-east1-b --project bytebase-dev \+ STATUS=$(gcloud compute instances describe devbox \+ --zone northamerica-northeast2-a --project bytebase-dev \--format='value(status)')if [ "$STATUS" != "RUNNING" ]; then- gcloud compute instances start github-runner \- --zone southamerica-east1-b --project bytebase-dev+ gcloud compute instances start devbox \+ --zone northamerica-northeast2-a --project bytebase-devfidiff --git a/docs/design/devbox-and-ci-runner.md b/docs/design/devbox-and-ci-runner.mdindex b83775a796..42de7ec7ed 100644--- a/docs/design/devbox-and-ci-runner.md+++ b/docs/design/devbox-and-ci-runner.md@@ -1,7 +1,7 @@# Combined Dev Box and CI Runner on GCPOne Spot VM with two jobs. It is the SSH target that agentic coding tools drive, and-it hosts three self-hosted GitHub Actions runners for the whole org.+it hosts four self-hosted GitHub Actions runners for the whole org.## Goals@@ -11,14 +11,14 @@ it hosts three self-hosted GitHub Actions runners for the whole org.2. **One startup script, so the box is close to stateless.** Everything on the box isproduced by that script. Only `/home` and a shared toolchain in `/usr/local` areyours to manage; nothing else is set up by hand.-3. **GitHub Actions runners that set themselves up.** Three org-level slots. Each+3. **GitHub Actions runners that set themselves up.** Four org-level slots. Eachre-registers at boot from a secret, so there is no registration state to preserve.4. **A clear split between Local SSD and the persistent disk.** Local SSD holds whatcan be discarded: caches, Docker, temp files, swap. The boot disk holds repos anduncommitted work. It can be lost too, so push your branches.5. **Cache cleanup that will not break a running job.** Leaked temp files are swept- always. Caches are evicted only under disk pressure, cheapest to rebuild first.- The one exception is a disk about to fill, which would fail every job anyway.+ always. Caches are evicted only under disk pressure, and only while the box is+ idle. The one exception is a disk about to fill, which would fail every job anyway.6. **Guest memory and swap visible in Cloud Monitoring.** The hypervisor reports onlyCPU and disk. The Ops Agent reports the rest, so RAM is sized from measurementinstead of a guess.@@ -36,7 +36,7 @@ it hosts three self-hosted GitHub Actions runners for the whole org.| Provisioning | `SPOT`, `instanceTerminationAction=STOP` || Image | `ubuntu-2404-lts-amd64` || Boot disk | 100 GB pd-balanced, `--no-boot-disk-auto-delete` |-| Scratch | 375 GB Local SSD (NVMe) |+| Scratch | 2 x 375 GB Local SSD (NVMe), striped into one 750 GB `/scratch` || External IP | reserved; carries all egress, and inbound SSH || Inbound | SSH only, via the VPC default rule; no tag, no custom rules || Service account | dedicated; reads one secret, writes metrics and logs, nothing else |@@ -46,18 +46,20 @@ it hosts three self-hosted GitHub Actions runners for the whole org.|---|---|---|| 20 vCPU (N2) | $0.003290 / vCPU-hr | $48.03 || 32 GiB RAM (N2) | $0.000441 / GiB-hr | $10.30 |-| 375 GB Local SSD | $0.052800 / GB-mo | $19.80 |+| 750 GB Local SSD | $0.052800 / GB-mo | $39.60 || 100 GB pd-balanced | $0.110 / GB-mo | $11.00 || Reserved external IP | $0.0025 / hr | $1.82 |-| **Total** | | **$90.96** |+| **Total** | | **$110.76** |-Catalog prices, 2026-09-01. vCPU and RAM are changeable on a stopped instance with-`set-machine-type`; Local SSD capacity is fixed at creation.+Catalog prices, re-verified 2026-09-03. vCPU and RAM are changeable on a stopped+instance with `set-machine-type`; Local SSD capacity is fixed at creation.### Re-checking the regionSpot prices are set per family *per region*, and they drift. Re-check before any-rebuild.+rebuild. Swept 2026-09-03 over the Cloud Billing catalog against the 40 regions that+can actually deploy each family: Toronto N2 was cheapest of all 80 region-family+pairs, and stayed cheapest when the sweep was re-run at half the disk.Price the whole VM, not just the CPU. Keep only regions that can actually run thefamily: the catalog prices N1 in 14 regions that cannot run it. Prefer N2 or N2D,@@ -79,7 +81,8 @@ manual — at `github.com/settings/personal-access-tokens/new`:| Organization permissions → **Self-hosted runners** | **Read and write** || Repository permissions | none |-Org-owned tokens may need an org admin to approve the request.+An org-owned token may need an org admin to approve the request. An org owner's own+request self-approves.Developers already have access: the developer group holds Editor on the project,which covers OS Login with `sudo` and acting as the service account.@@ -88,51 +91,14 @@ which covers OS Login with `sudo` and acting as the service account.on the secret, which Editor cannot do. It stops at the first failed step.```bash-set -e-PROJECT=bytebase-dev-ZONE=northamerica-northeast2-a-SA=devbox@${PROJECT}.iam.gserviceaccount.com--gcloud services enable secretmanager.googleapis.com oslogin.googleapis.com \- monitoring.googleapis.com logging.googleapis.com --project=$PROJECT--# Service account; its roles are granted below, nothing more.-gcloud iam service-accounts create devbox --project=$PROJECT \- --display-name="devbox: dev host and CI runners"--# The PAT, readable by that service account and nothing else.-read -rsp 'GitHub PAT: ' GH_PAT && echo-printf '%s' "$GH_PAT" | gcloud secrets create gh-runner-pat \- --project=$PROJECT --replication-policy=automatic --data-file=--unset GH_PAT--gcloud secrets add-iam-policy-binding gh-runner-pat --project=$PROJECT \- --member="serviceAccount:${SA}" --role=roles/secretmanager.secretAccessor--# Metrics and logs from the Ops Agent. Without these the agent gets 403s and goal 6-# is silently unmet -- a scope is not a role.-for r in monitoring.metricWriter logging.logWriter; do- gcloud projects add-iam-policy-binding $PROJECT \- --member="serviceAccount:${SA}" --role=roles/$r-done--# A reserved address. An ephemeral one is released when the instance stops, so every-# preemption would hand the box a new IP and strand the generated SSH entry.-gcloud compute addresses create devbox --project=$PROJECT --region=${ZONE%-*}--# The instance.-gcloud compute instances create devbox --project=$PROJECT --zone=$ZONE \- --address=devbox \- --machine-type=n2-custom-20-32768 \- --provisioning-model=SPOT --instance-termination-action=STOP \- --image-family=ubuntu-2404-lts-amd64 --image-project=ubuntu-os-cloud \- --boot-disk-size=100GB --boot-disk-type=pd-balanced --no-boot-disk-auto-delete \- --local-ssd=interface=NVME \- --service-account=$SA --scopes=cloud-platform \- --metadata=enable-oslogin=TRUE \- --metadata-from-file=startup-script=./startup.sh+./scripts/devbox/create.sh```+It enables the APIs, creates the service account, stores the PAT it prompts for in+Secret Manager, grants that account the secret plus `monitoring.metricWriter` and+`logging.logWriter`, reserves the address, and creates the instance with+`startup.sh` as its startup-script metadata.+The `cloud-platform` scope resolves to the intersection with the account's roles: onesecret, metrics, logs. The scope is not the boundary. The IAM bindings are.@@ -140,24 +106,44 @@ There is no flag day. Runners register at the organization under the automatic`self-hosted` label, with host-qualified names. So this box only adds capacityalongside any existing runner, and you can delete the old one whenever.-One step does have to happen in order. A preemption stops the instance, and-`.github/workflows/start-runner-vm.yml` already restarts a runner box on a schedule.-Repoint its instance name and zone at this one, in two places each, as the last step-of cutover. Do it earlier and it stops restarting the box it serves today. Until then,-start this one by hand.+One step has to happen in order, and it is the last. A preemption stops the instance,+and `.github/workflows/start-runner-vm.yml` restarts it on a schedule. That workflow+now names this instance and zone, in two places each. Repointing it any earlier would+have stopped it restarting the box that was serving CI at the time.-Raise its cadence at the same time. It runs at three fixed hours, so a preemption just-after one of them leaves the box stopped for up to fifteen -- which is not "always on"-for either the dev box or CI. Every fifteen minutes bounds the outage at fifteen-minutes, and starting a running instance is a no-op.+Its cadence changed with it, from three fixed hours to every fifteen minutes. At three+a day, a preemption just after one of them left the box stopped for up to fifteen+hours, which is not "always on" for either the dev box or CI. Fifteen minutes bounds+the outage at fifteen minutes, and starting a running instance is a no-op.### Connect-Run `gcloud compute config-ssh --project=bytebase-dev` once. It writes the `~/.ssh/config` entries, and this-box's alias is `devbox.northamerica-northeast2-a.bytebase-dev`. Desktop agents take-that alias as their SSH host: Claude Desktop under **Add SSH connection**, Codex in-its SSH environment. Each developer gets their own account and home, created by OS-Login on first connect.+```+Host devbox+ HostName devbox+ User <your OS Login name>+ IdentityFile ~/.ssh/google_compute_engine+ ProxyCommand /path/to/gcloud compute start-iap-tunnel devbox 22 --listen-on-stdin --project=bytebase-dev --zone=northamerica-northeast2-a --verbosity=warning+```++Then `ssh devbox`, and the same host in Claude Desktop's **Add SSH connection**.+Run `gcloud compute config-ssh --project=bytebase-dev` once first to register your key;+`gcloud compute os-login describe-profile` prints your account name. Keep this entry+outside the block config-ssh writes.++Three things about that `ProxyCommand`:++- **It tunnels SSH over HTTPS**, so it works on a network that intercepts port 22 --+ a VPN resolving hosts into `198.18.0.0/15` does, and a direct alias then dies at+ `kex_exchange_identification`. Nothing needs opening; the default VPC rule already+ admits IAP's range.+- **Absolute path, not `gcloud`.** GUI apps inherit launchd's minimal `PATH`, so a+ bare `gcloud` resolves in your terminal and fails in Claude Desktop.+- **It needs a live gcloud session.** When it lapses, `ssh devbox` fails on reauth+ rather than on the network; `gcloud auth login` restores it.++Without the tunnel, on a network that permits port 22,+`ssh devbox.northamerica-northeast2-a.bytebase-dev` works with no entry at all.Log in once interactively before pointing an agent at a new account. The cache linksin (b) are made by the profile, and a non-login `bash -c` does not read one -- so an@@ -169,17 +155,18 @@ at all: it comes from `/etc/environment`, which every session reads.## 2. Startup script-Saved as `startup.sh` and passed to the create command above. It runs as root on every-boot and is idempotent. Local SSD is discarded on every stop, so anything living there-is rebuilt. Anything on the boot disk is guarded, so it is not redone.+[`scripts/devbox/startup.sh`](../../scripts/devbox/startup.sh), passed to the create+command above as startup-script metadata. It runs as root on every boot and is+idempotent. Local SSD is discarded on every stop, so anything living there is+rebuilt. Anything on the boot disk is guarded, so it is not redone.### a. Disk layoutTwo kinds of account use the box, and they need opposite things from it.-**Runner accounts** are `runner1`, `runner2` and `runner3`. The script creates them, `nologin`,-one per job slot. Nothing of theirs is worth keeping: software, work trees and caches-all sit on Local SSD, and their home directories are on it too. They are+**Runner accounts** are `runner1` through `runner4`. The script creates them,+`nologin`, one per job slot. Nothing of theirs is worth keeping: software, work trees+and caches all sit on Local SSD, and their home directories are on it too. They aredisposable, and are disposed of on every stop.**Interactive accounts** arrive through OS Login on first connect. Their `/home/<u>`@@ -199,6 +186,15 @@ Local SSD also carries Docker's `data-root`, the swapfile and `/tmp`. Containerstartup and database `fsync` are the heaviest I/O on this box. Docker starts beforethe script runs, so the script points it at Local SSD and restarts it.+There are two partitions because N2 will not attach one at 20 vCPU, and the script+stripes them into a single 750 GB `/scratch`. A reboot keeps Local SSD, and udev+re-assembles the array under a name of its own choosing -- `/dev/md127`, not the+`/dev/md0` it was created as -- so the script asks `mdadm --detail --scan` what is+live before it creates anything. Creating over a live array would discard every+cache on the box, quietly, on every reboot. Stopping by hand needs+`--discard-local-ssd=true`, which gcloud now requires to be explicit; that is what a+preemption does regardless.+**Swap causes hangs; it does not prevent them.** Without swap, exhausting RAM is afast OOM kill. With swap, the box thrashes until the job timeout. So there is 8 GB at`vm.swappiness=10`: enough to evict cold pages, too little to sustain a thrash. And@@ -215,6 +211,11 @@ every session, non-login included. CI jobs bring their own through `setup-go` aninstalls one, and Go defaults to `CGO_ENABLED=1`, so dependencies carrying cgo filesfail to build without it.+`gh` is the one tool the script does install, from upstream rather than apt, which+ships 2.45.0. It lands in `/usr/local/bin` on the boot disk, so a reboot is a no-op+and only a rebuild or a new release refetches. A failed fetch warns rather than+calling `fatal`: a boot must not depend on GitHub being reachable.+### b. Cache pathsFour paths carry every cache. `XDG_CACHE_HOME` relocates the Go **build** cache,@@ -257,12 +258,13 @@ and `node_modules` — which is right for anything inside a working copy.### c. GitHub runners-Three accounts, three job slots, one systemd template. A pull request touching both+Four accounts, four job slots, one systemd template. A pull request touching bothbackend and frontend schedules five self-hosted jobs -- one each from `backend-tests`,-`golangci-lint` and `test_link`, two from `frontend-tests` -- so two of them queue,+`golangci-lint` and `test_link`, two from `frontend-tests` -- so one of them queues,before counting any other repository in the org. That is deliberate: jobs queue, they-do not fail, and 20 vCPU split five ways is thin. A fourth or fifth slot is one more-name in the script's account list if the wait proves worse than the contention.+do not fail. Four ways is five cores and 8 GB a slot, which is what a comparable box+ran two slots on; watch `memory/percent_used` before adding a fifth, which is one more+name in the script's account list.The runner is stateless, so it lives on scratch and is installed by the unit ratherthan by the startup script: an `ExecStartPre` runs `install-runner` as root, before@@ -279,7 +281,7 @@ server-side entry of the same name. Names are host-qualified, so two boxes nevereach other's.Re-registering rather than preserving credentials means the runner holds no identity-on disk. Wipe the boot disk and the same service account brings back the same three+on disk. Wipe the boot disk and the same service account brings back the same fourrunners. Registration is derived state, like `/scratch`.The units carry no `[Install]` section. The script starts them rather than enabling@@ -308,221 +310,6 @@ It runs on a schedule, not at boot. A boot-time check cannot fire on a box thatup for days, and Local SSD survives a guest *reboot*. It is discarded only on stop orpreemption.-### startup.sh--```bash-#!/bin/bash-# GCE startup-script: root, every boot, idempotent. No `set -e`: a failed step must-# not abandon the boot; `journalctl -t devbox-startup` has the exit status.-set -uo pipefail-trap 'logger -t devbox-startup "exited: status $?"' EXIT-# A fatal exit powers the box off. Left RUNNING with no runners, the start workflow-# would never touch it; off, it is restarted and this script retried on the-# workflow's next tick. Five minutes keeps SSH usable for the journal or /home first.-fatal() {- logger -t devbox-startup "FATAL: $1"; systemctl stop docker.socket docker containerd 2>/dev/null- shutdown -h +5 "devbox-startup: $1"; exit 1-}--# Docker down and masked until (a) has mounted /scratch. daemon.json persists, so the-# previous boot's dockerd is already up with its data root on the boot disk, and-# mounting over it would leave it writing there unseen. Masked, nothing can restart it-# -- not apt's postinst, not a connection to a still-listening socket -- until (a) is done.-systemctl mask --now docker.socket docker containerd 2>/dev/null--# ---------- packages -----------export DEBIAN_FRONTEND=noninteractive-echo 'DPkg::Lock::Timeout "300";' > /etc/apt/apt.conf.d/99-lock-timeout # unattended-upgrades holds the lock at boot-PKGS="docker.io cron systemd-oomd build-essential" # build-essential: Go defaults to CGO_ENABLED=1-dpkg --configure -a 2>/dev/null || true # clears a transaction a preemption cut short; apt repairs the rest-apt-get -y -qq -f install $PKGS 2>/dev/null || { apt-get update -qq && apt-get -y -qq -f install $PKGS; } \- || fatal "prerequisite install failed"--# ---------- (a) disk layout -----------mkdir -p /scratch-DEV=$(ls /dev/disk/by-id/google-local-* 2>/dev/null | head -1)-if [[ -n "$DEV" ]] && ! mountpoint -q /scratch; then- # A stop discards Local SSD; a reboot does not. Format only when there is no filesystem.- [[ -n "$(blkid -s TYPE -o value "$DEV")" ]] || mkfs.ext4 -F -m 0 -E lazy_itable_init=0,lazy_journal_init=0,discard "$DEV"- mount -o noatime,discard "$DEV" /scratch-fi-# Everything below assumes it. Falling back to the boot disk would put swap, Docker,-# work trees and caches on the 100 GB disk that holds /home.-mountpoint -q /scratch || fatal "no Local SSD at /scratch"-chmod 1777 /scratch # OS Login accounts create their own subtree-mkdir -p /scratch/tmp && chmod 1777 /scratch/tmp-# Compare inodes, not `mountpoint`: a tmpfs /tmp is a mountpoint too, and would leave temp files in RAM.-[ "$(stat -c '%d:%i' /tmp)" = "$(stat -c '%d:%i' /scratch/tmp)" ] || mount --bind /scratch/tmp /tmp \- || fatal "/tmp not on Local SSD"-swapon --show=NAME --noheadings | grep -q /scratch/swapfile \- || { rm -f /scratch/swapfile && fallocate -l 8G /scratch/swapfile && chmod 600 /scratch/swapfile \- && mkswap /scratch/swapfile >/dev/null && swapon /scratch/swapfile; } \- || logger -t devbox-startup "WARN: no swap active"-sysctl -qw vm.swappiness=10-systemctl enable --now systemd-oomd || logger -t devbox-startup "WARN: systemd-oomd inert"-# On the slice: containers get their own scopes under system.slice, not under the runner unit.-mkdir -p /etc/systemd/system/system.slice.d-printf '[Slice]\nManagedOOMMemoryPressure=kill\n' > /etc/systemd/system/system.slice.d/oomd.conf--# Docker on Local SSD with log rotation. The socket is opened to every account: OS-# Login users appear on first connect and cannot be pre-added to the docker group.-mkdir -p /scratch/docker /scratch/containerd /var/lib/containerd /etc/systemd/system/docker.service.d-echo '{"data-root":"/scratch/docker","log-driver":"json-file","log-opts":{"max-size":"10m","max-file":"3"}}' > /etc/docker/daemon.json-printf '[Service]\nExecStartPost=/bin/chmod 666 /var/run/docker.sock\n' > /etc/systemd/system/docker.service.d/socket.conf-# Engine 29 keeps images in containerd's own store, which data-root does not cover. A-# bind needs no containerd config to track.-mountpoint -q /var/lib/containerd || mount --bind /scratch/containerd /var/lib/containerd \- || fatal "containerd store not on Local SSD"-# Masked until here: an active docker.socket cannot spawn a masked service, so nothing-# can start dockerd on the boot disk between the stop above and this restart. And-# disabled from here on: only this script starts Docker, so a reboot no longer brings-# it up on the boot disk before the script has run.-systemctl unmask docker.socket docker containerd 2>/dev/null-systemctl disable docker.socket docker containerd 2>/dev/null \- || logger -t devbox-startup "WARN: docker autostart still enabled; the mask covers the next boot"-systemctl daemon-reload && systemctl restart containerd docker || fatal "docker would not start"-docker info --format '{{.DockerRootDir}}' 2>/dev/null | grep -q '^/scratch/' \- || fatal "docker not on Local SSD" # daemon.json unwritten or rejected: dockerd starts fine on the boot disk--# ---------- (b) accounts and cache paths -----------for u in runner1 runner2 runner3; do- # Home on scratch: whatever a job writes home-relative is disposable too.- id "$u" &>/dev/null || useradd -M -d "/scratch/$u/home" -s /usr/sbin/nologin "$u"- # runner/ up front: systemd applies WorkingDirectory before ExecStartPre could create it.- mkdir -p "/scratch/$u"/{cache,work,runner,home}- chown "$u:$u" "/scratch/$u" "/scratch/$u"/{cache,work,runner,home} # not -R: contents are the user's own-done--# Interactive accounts appear at first OS Login connect, so the profile provisions them:-# each tool's default cache path becomes a symlink onto scratch. Symlinks, not variables,-# so an agent's `bash -c` -- which sources no profile -- lands there too. PATH goes in-# /etc/environment for the same reason: pam_env applies it to every session.-echo 'PATH="/usr/local/go/bin:/usr/local/node/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"' > /etc/environment-cat > /etc/profile.d/devbox.sh <<'EOF'-u=$(id -un); c=/scratch/$u/cache-if mountpoint -q /scratch && [ -n "$u" ] && [ -n "${HOME:-}" ] && mkdir -p "$c" 2>/dev/null; then- for pair in "$HOME/.cache:$c" "$HOME/go/pkg/mod:$c/go-mod" "$HOME/.npm:$c/npm" "$HOME/.local/share/pnpm:$c/pnpm"; do- link=${pair%%:*}; target=${pair#*:}- mkdir -p "$target" # every login: scratch targets vanish on a stop, the symlink in HOME does not- [ -L "$link" ] || { mkdir -p "$(dirname "$link")" && rm -rf "$link" && ln -s "$target" "$link"; }- done 2>/dev/null-fi-unset u c pair link target-EOF-# Existing accounts keep their symlinks across a stop but not the targets. Remake them,-# so only a brand-new account needs a first interactive login.-for h in /home/*; do- u=${h##*/}; id "$u" &>/dev/null || continue- for d in "" /go-mod /npm /pnpm; do install -d -o "$u" -g "$u" "/scratch/$u/cache$d"; done-done--# ---------- (c) GitHub runners -----------# Stateless, so it lives on scratch and is installed from the unit rather than from-# here: systemd retries that step forever, so a failed download heals itself.-cat > /usr/local/sbin/install-runner <<'EOF'-#!/bin/bash-# <account>. Root, from the unit. Re-runs whole on every retry, deps included.-set -euo pipefail-DIR=/scratch/$1/runner-# Latest release; if the lookup fails, keep what is installed, or fall back to a pin.-V=$(curl -sf --retry 3 --max-time 30 https://api.github.com/repos/actions/runner/releases/latest \- | python3 -c 'import sys,json; print(json.load(sys.stdin)["tag_name"].lstrip("v"))' 2>/dev/null) \- || V=$(cat "$DIR/.installed" 2>/dev/null || echo 2.336.0)-# Version-stamped and written last: a reboot keeps scratch, so a bare marker would pin-# the old release, and an unstamped one would skip a failed dep step.-[[ "$(cat "$DIR/.installed" 2>/dev/null)" == "$V" ]] && exit 0-rm -rf "$DIR" && install -d -o "$1" -g "$1" "$DIR" # clean tree: an old release's files must not linger-curl -fsSL --retry 3 "https://github.com/actions/runner/releases/download/v$V/actions-runner-linux-x64-$V.tar.gz" | tar xz -C "$DIR"-chown -R "$1:$1" "$DIR"-"$DIR"/bin/installdependencies.sh >/dev/null # apt is a no-op once satisfied-echo "$V" > "$DIR/.installed"-EOF--cat > /usr/local/sbin/register-runner <<'EOF'-#!/bin/bash-# <account>. Runs as the account, every boot.-set -euo pipefail-ORG=bytebase-md() { curl -sf -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/$1"; }-PROJECT=$(md project/project-id)-ACCESS=$(md instance/service-accounts/default/token | python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])')-PAT=$(curl -sf -H "Authorization: Bearer $ACCESS" \- "https://secretmanager.googleapis.com/v1/projects/${PROJECT}/secrets/gh-runner-pat/versions/latest:access" \- | python3 -c 'import sys,base64,json; print(base64.b64decode(json.load(sys.stdin)["payload"]["data"]).decode())')-TOKEN=$(curl -sfX POST -H "Authorization: Bearer $PAT" -H "Accept: application/vnd.github+json" \- "https://api.github.com/orgs/${ORG}/actions/runners/registration-token" \- | python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')-cd "/scratch/$1/runner"-rm -f .runner .credentials .credentials_rsakey # config.sh refuses to overwrite an existing config-./config.sh --unattended --replace --url "https://github.com/${ORG}" --token "$TOKEN" \- --name "$(hostname -s)-$1" --work "/scratch/$1/work" # host-qualified name: never collides with another box-EOF-chmod +x /usr/local/sbin/install-runner /usr/local/sbin/register-runner--cat > /etc/systemd/system/[email protected] <<'EOF'-[Unit]-Description=GitHub Actions runner (%i)-Wants=network-online.target-After=network-online.target docker.service-[Service]-User=%i-WorkingDirectory=/scratch/%i/runner-Environment=XDG_CACHE_HOME=/scratch/%i/cache GOMODCACHE=/scratch/%i/cache/go-mod npm_config_cache=/scratch/%i/cache/npm PNPM_CONFIG_STORE_DIR=/scratch/%i/cache/pnpm-ExecStartPre=+/usr/local/sbin/install-runner %i-ExecStartPre=/usr/local/sbin/register-runner %i-ExecStart=/scratch/%i/runner/run.sh-# install-runner fetches ~200 MB and may wait on the apt lock; the 90s default would-# kill it, and the retry restarts the download from zero.-TimeoutStartSec=600-Restart=always-# Registration failure retries forever; at 5s x3 runners that would burn the API quota.-RestartSec=30-EOF--# ---------- (d) cache cleanup -----------cat > /usr/local/sbin/cache-gc <<'EOF'-#!/bin/bash-# Every 30 min. Leaked go-build sandboxes are swept always. Above 85% used, wait for an-# idle box -- unless it is nearly full -- then delete everything disposable. It all-# rebuilds on the next job; a full disk does not.-set -uo pipefail-mountpoint -q /scratch || exit 0 # cron outlives a stop; before (a) has mounted, there is nothing to clean-HIGH=85 CRITICAL=95-find /tmp -maxdepth 1 -name 'go-build*' -type d -mmin +1440 -exec rm -rf {} +-used() { df -P /scratch | awk 'NR==2 {print $5+0}'; }-# Load catches detached work -- tmux, nohup, a container -- that holds neither an SSH-# session nor a Runner.Worker. On 20 vCPU, anything actually working exceeds 2.-idle() { ! pgrep -f Runner.Worker >/dev/null && [[ -z "$(ss -Htn state established '( sport = :22 )')" ]] \- && (( $(cut -d. -f1 /proc/loadavg) < 2 )); }--[[ $(used) -ge $HIGH ]] || exit 0-if ! idle && [[ $(used) -lt $CRITICAL ]]; then logger -t cache-gc "deferred: busy at $(used)%"; exit 0; fi-logger -t cache-gc "cleaning: $(used)% used"-rm -rf /tmp/* /tmp/.[!.]* # /tmp is on scratch; the box is idle-docker ps -q | xargs -r docker rm -f >/dev/null 2>&1 # idle, so a running container is orphaned-docker system prune -af --volumes >/dev/null 2>&1-# Everything on scratch is disposable except the runner installs, which are large and-# would otherwise be re-downloaded on the next boot.-rm -rf /scratch/*/cache /scratch/*/home /scratch/*/work-for u in runner1 runner2 runner3; do- install -d -o "$u" -g "$u" /scratch/$u/{cache,home,work} # the units need them; a profile remakes an interactive account's-done-logger -t cache-gc "cleaned -> $(used)%"-EOF-chmod +x /usr/local/sbin/cache-gc-# PATH first: cron's default omits /usr/sbin, and a missing `ss` would make idle() look true.-printf 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n*/30 * * * * root /usr/local/sbin/cache-gc\n' > /etc/cron.d/devbox--systemctl daemon-reload-systemctl restart actions-runner@{runner1,runner2,runner3}--# Last and time-bounded: observability must not hold up CI capacity.-systemctl is-active --quiet google-cloud-ops-agent \- || { curl -sS --retry 3 --max-time 120 -o /tmp/ops.sh https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh \- && timeout 300 bash /tmp/ops.sh --also-install; } \- || logger -t devbox-startup "WARN: Ops Agent install failed; no memory or swap metrics"-```-## 3. Considered**N2D** — its guaranteed Milan silicon avoids N2's Cascade-Lake-or-Ice-Lake lottery,@@ -544,9 +331,18 @@ with `set-scheduling`.stays up for the union of both duty cycles at the sum of both sizes. Consolidation waschosen for operational simplicity.-**750 GB Local SSD** — another $19.80/mo, against a projected 49% use of 375 GB. Local-SSD is the one capacity fixed at creation, so it is the first thing to revisit if that-projection is wrong.+**375 GB Local SSD** — not offered. N2 at 20 vCPU accepts 0, 2, 4, 8, 16 or 24+partitions, so the smallest non-zero scratch is two, and goal 1 fixes the vCPU count.+Both are striped rather than leaving one idle, since they are paid for either way.+Local SSD is the one capacity fixed at creation, so revisit it before any rebuild.++**A larger persistent disk instead of Local SSD** — slower and dearer at every size+tried. Measured on this box with `fio`, pd-balanced returns the same 12.7k random+read IOPS and 547 MB/s at 100 GB, 500 GB and 850 GB, because a 20 vCPU instance caps+throughput long before per-GB scaling matters. The striped Local SSD returns 360k+IOPS and 1407 MB/s, and costs $0.0528/GB-mo against pd-balanced's $0.110. Buying+persistent disk here would pay twice per GB for a twenty-eighth of the IOPS. What it+buys instead is durability, which is why `/home` is on it and nothing else is.**A firewall rule and network tag** — the default VPC leaves several ports open to`0.0.0.0/0`, but nothing here listens on them. Tags can be added to a running instance,@@ -571,7 +367,8 @@ change any `/home` on the box -- uncommitted work, agent CLI state, `~/.ssh`. Trthis as a shared machine and keep nothing on it you would not put on one.**Pinning Ice Lake** — deferred. `cpuPlatform` after each restart shows what Toronto-hands out. A pin, or a different region, is a stop-and-start away.+hands out; so far, Cascade Lake every time. A pin, or a different region, is a+stop-and-start away.## Reference@@ -579,3 +376,4 @@ hands out. A pin, or a different region, is a stop-and-start away.- Self-hosted runners: https://docs.github.com/en/actions/hosting-your-own-runners- Fine-grained PAT permissions: https://docs.github.com/en/rest/authentication/permissions-required-for-fine-grained-personal-access-tokens- CPU platforms: https://cloud.google.com/compute/docs/cpu-platforms+- IAP TCP forwarding: https://cloud.google.com/iap/docs/using-tcp-forwardingdiff --git a/scripts/devbox/create.sh b/scripts/devbox/create.shnew file mode 100755index 0000000000..e3ca31e6c8--- /dev/null+++ b/scripts/devbox/create.sh@@ -0,0 +1,56 @@+#!/bin/bash+# Creates the devbox: the SSH target for agentic coding tools and the host for the+# org's self-hosted GitHub Actions runners. Design: docs/design/devbox-and-ci-runner.md+#+# Run once, as a project Owner: it sets IAM policy on the project and on the secret,+# which Editor cannot do. It stops at the first failed step and is not idempotent --+# after a partial run, delete what was created before re-running.+#+# Prerequisite, and the one step no API can do: a fine-grained GitHub PAT owned by the+# `bytebase` org, with organization permission "Self-hosted runners: Read and write"+# and nothing else. See the Create section of the design doc.+set -euo pipefail+PROJECT=bytebase-dev+ZONE=northamerica-northeast2-a+SA=devbox@${PROJECT}.iam.gserviceaccount.com+HERE=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)++gcloud services enable secretmanager.googleapis.com oslogin.googleapis.com \+ monitoring.googleapis.com logging.googleapis.com --project=$PROJECT++# Service account; its roles are granted below, nothing more.+gcloud iam service-accounts create devbox --project=$PROJECT \+ --display-name="devbox: dev host and CI runners"++# The PAT, readable by that service account and nothing else.+read -rsp 'GitHub PAT: ' GH_PAT; echo # not `&& echo`: set -e ignores a failed read inside an && list+printf '%s' "$GH_PAT" | gcloud secrets create gh-runner-pat \+ --project=$PROJECT --replication-policy=automatic --data-file=-+unset GH_PAT++gcloud secrets add-iam-policy-binding gh-runner-pat --project=$PROJECT \+ --member="serviceAccount:${SA}" --role=roles/secretmanager.secretAccessor++# Metrics and logs from the Ops Agent. Without these the agent gets 403s and guest+# memory never reaches Cloud Monitoring -- a scope is not a role.+for r in monitoring.metricWriter logging.logWriter; do+ gcloud projects add-iam-policy-binding $PROJECT \+ --member="serviceAccount:${SA}" --role=roles/$r --condition=None+done++# A reserved address. An ephemeral one is released when the instance stops, so every+# preemption would hand the box a new IP and strand the generated SSH entry.+gcloud compute addresses create devbox --project=$PROJECT --region=${ZONE%-*}++# The instance.+gcloud compute instances create devbox --project=$PROJECT --zone=$ZONE \+ --address=devbox \+ --machine-type=n2-custom-20-32768 \+ --provisioning-model=SPOT --instance-termination-action=STOP \+ --image-family=ubuntu-2404-lts-amd64 --image-project=ubuntu-os-cloud \+ --boot-disk-size=100GB --boot-disk-type=pd-balanced --no-boot-disk-auto-delete \+ --local-ssd=interface=NVME --local-ssd=interface=NVME \+ `# two partitions, not one: N2 at 20 vCPU accepts only 0, 2, 4, 8, 16 or 24` \+ --service-account=$SA --scopes=cloud-platform \+ --metadata=enable-oslogin=TRUE \+ --metadata-from-file=startup-script="$HERE/startup.sh"diff --git a/scripts/devbox/startup.sh b/scripts/devbox/startup.shnew file mode 100755index 0000000000..fff8362917--- /dev/null+++ b/scripts/devbox/startup.sh@@ -0,0 +1,238 @@+#!/bin/bash+# GCE startup-script: root, every boot, idempotent. No `set -e`: a failed step must+# not abandon the boot; `journalctl -t devbox-startup` has the exit status.+set -uo pipefail+trap 'logger -t devbox-startup "exited: status $?"' EXIT+# A fatal exit powers the box off. Left RUNNING with no runners, the start workflow+# would never touch it; off, it is restarted and this script retried on the+# workflow's next tick. Five minutes keeps SSH usable for the journal or /home first.+fatal() {+ logger -t devbox-startup "FATAL: $1"; systemctl stop docker.socket docker containerd 2>/dev/null+ shutdown -h +5 "devbox-startup: $1"; exit 1+}++# Docker down and masked until (a) has mounted /scratch. daemon.json persists, so the+# previous boot's dockerd is already up with its data root on the boot disk, and+# mounting over it would leave it writing there unseen. Masked, nothing can restart it+# -- not apt's postinst, not a connection to a still-listening socket -- until (a) is done.+systemctl mask --now docker.socket docker containerd 2>/dev/null++# ---------- packages ----------+export DEBIAN_FRONTEND=noninteractive+echo 'DPkg::Lock::Timeout "300";' > /etc/apt/apt.conf.d/99-lock-timeout # unattended-upgrades holds the lock at boot+PKGS="docker.io cron systemd-oomd build-essential mdadm" # build-essential: Go defaults to CGO_ENABLED=1+dpkg --configure -a 2>/dev/null || true # clears a transaction a preemption cut short; apt repairs the rest+apt-get -y -qq -f install $PKGS 2>/dev/null || { apt-get update -qq && apt-get -y -qq -f install $PKGS; } \+ || fatal "prerequisite install failed"++# ---------- (a) disk layout ----------+mkdir -p /scratch+DEVS=(); for d in /dev/disk/by-id/google-local-*; do [[ -b $d && $d != *-part* ]] && DEVS+=("$d"); done+DEV=${DEVS[0]:-}+# N2 refuses a single Local SSD at 20 vCPU -- 0, 2, 4, 8, 16 or 24 only -- so there are+# always at least two, and they are striped into one /scratch. A reboot keeps Local SSD+# and udev may have re-assembled the array under a name of its choosing, so look for a+# live array, then for an assemblable one, and only then create: `mdadm --create` over+# an existing array discards it.+if (( ${#DEVS[@]} > 1 )); then+ md() { mdadm --detail --scan 2>/dev/null | awk 'NR==1{print $2}'; }+ DEV=$(md)+ [[ -n "$DEV" ]] || { mdadm --assemble --scan >/dev/null 2>&1; DEV=$(md); }+ [[ -n "$DEV" ]] || { mdadm --create /dev/md0 --level=0 --raid-devices="${#DEVS[@]}" \+ --run "${DEVS[@]}" >/dev/null 2>&1 && DEV=/dev/md0; }+fi+if [[ -n "$DEV" ]] && ! mountpoint -q /scratch; then+ # A stop discards Local SSD; a reboot does not. Format only when there is no filesystem.+ [[ -n "$(blkid -s TYPE -o value "$DEV")" ]] || mkfs.ext4 -F -m 0 -E lazy_itable_init=0,lazy_journal_init=0,discard "$DEV"+ mount -o noatime,discard "$DEV" /scratch+fi+# Everything below assumes it. Falling back to the boot disk would put swap, Docker,+# work trees and caches on the 100 GB disk that holds /home.+mountpoint -q /scratch || fatal "no Local SSD at /scratch"+chmod 1777 /scratch # OS Login accounts create their own subtree+mkdir -p /scratch/tmp && chmod 1777 /scratch/tmp+# Compare inodes, not `mountpoint`: a tmpfs /tmp is a mountpoint too, and would leave temp files in RAM.+[ "$(stat -c '%d:%i' /tmp)" = "$(stat -c '%d:%i' /scratch/tmp)" ] || mount --bind /scratch/tmp /tmp \+ || fatal "/tmp not on Local SSD"+swapon --show=NAME --noheadings | grep -q /scratch/swapfile \+ || { rm -f /scratch/swapfile && fallocate -l 8G /scratch/swapfile && chmod 600 /scratch/swapfile \+ && mkswap /scratch/swapfile >/dev/null && swapon /scratch/swapfile; } \+ || logger -t devbox-startup "WARN: no swap active"+sysctl -qw vm.swappiness=10+systemctl enable --now systemd-oomd || logger -t devbox-startup "WARN: systemd-oomd inert"+# On the slice: containers get their own scopes under system.slice, not under the runner unit.+mkdir -p /etc/systemd/system/system.slice.d+printf '[Slice]\nManagedOOMMemoryPressure=kill\n' > /etc/systemd/system/system.slice.d/oomd.conf++# Docker on Local SSD with log rotation. The socket is opened to every account: OS+# Login users appear on first connect and cannot be pre-added to the docker group.+mkdir -p /scratch/docker /scratch/containerd /var/lib/containerd /etc/systemd/system/docker.service.d+echo '{"data-root":"/scratch/docker","log-driver":"json-file","log-opts":{"max-size":"10m","max-file":"3"}}' > /etc/docker/daemon.json+printf '[Service]\nExecStartPost=/bin/chmod 666 /var/run/docker.sock\n' > /etc/systemd/system/docker.service.d/socket.conf+# Engine 29 keeps images in containerd's own store, which data-root does not cover. A+# bind needs no containerd config to track.+mountpoint -q /var/lib/containerd || mount --bind /scratch/containerd /var/lib/containerd \+ || fatal "containerd store not on Local SSD"+# Masked until here: an active docker.socket cannot spawn a masked service, so nothing+# can start dockerd on the boot disk between the stop above and this restart. And+# disabled from here on: only this script starts Docker, so a reboot no longer brings+# it up on the boot disk before the script has run.+systemctl unmask docker.socket docker containerd 2>/dev/null+systemctl disable docker.socket docker containerd 2>/dev/null \+ || logger -t devbox-startup "WARN: docker autostart still enabled; the mask covers the next boot"+systemctl daemon-reload && systemctl restart containerd docker || fatal "docker would not start"+docker info --format '{{.DockerRootDir}}' 2>/dev/null | grep -q '^/scratch/' \+ || fatal "docker not on Local SSD" # daemon.json unwritten or rejected: dockerd starts fine on the boot disk++# ---------- (b) accounts and cache paths ----------+for u in runner1 runner2 runner3 runner4; do+ # Home on scratch: whatever a job writes home-relative is disposable too.+ id "$u" &>/dev/null || useradd -M -d "/scratch/$u/home" -s /usr/sbin/nologin "$u"+ # runner/ up front: systemd applies WorkingDirectory before ExecStartPre could create it.+ mkdir -p "/scratch/$u"/{cache,work,runner,home}+ chown "$u:$u" "/scratch/$u" "/scratch/$u"/{cache,work,runner,home} # not -R: contents are the user's own+done++# Interactive accounts appear at first OS Login connect, so the profile provisions them:+# each tool's default cache path becomes a symlink onto scratch. Symlinks, not variables,+# so an agent's `bash -c` -- which sources no profile -- lands there too. PATH goes in+# /etc/environment for the same reason: pam_env applies it to every session.+echo 'PATH="/usr/local/go/bin:/usr/local/node/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"' > /etc/environment+cat > /etc/profile.d/devbox.sh <<'EOF'+u=$(id -un); c=/scratch/$u/cache+if mountpoint -q /scratch && [ -n "$u" ] && [ -n "${HOME:-}" ] && mkdir -p "$c" 2>/dev/null; then+ for pair in "$HOME/.cache:$c" "$HOME/go/pkg/mod:$c/go-mod" "$HOME/.npm:$c/npm" "$HOME/.local/share/pnpm:$c/pnpm"; do+ link=${pair%%:*}; target=${pair#*:}+ mkdir -p "$target" # every login: scratch targets vanish on a stop, the symlink in HOME does not+ [ -L "$link" ] || { mkdir -p "$(dirname "$link")" && rm -rf "$link" && ln -s "$target" "$link"; }+ done 2>/dev/null+fi+unset u c pair link target+EOF+# Existing accounts keep their symlinks across a stop but not the targets. Remake them,+# so only a brand-new account needs a first interactive login.+for h in /home/*; do+ u=${h##*/}; id "$u" &>/dev/null || continue+ for d in "" /go-mod /npm /pnpm; do install -d -o "$u" -g "$u" "/scratch/$u/cache$d"; done+done++# ---------- gh ----------+# Upstream, not apt: Ubuntu 24.04 ships 2.45.0. /usr/local/bin is on the boot disk, so+# this is a no-op on a reboot and only refetches after a rebuild or a new release.+# Never fatal -- gh is a convenience, and a boot must not turn on GitHub being reachable.+GH=$(curl -sf --retry 3 --max-time 30 https://api.github.com/repos/cli/cli/releases/latest \+ | python3 -c 'import sys,json; print(json.load(sys.stdin)["tag_name"].lstrip("v"))' 2>/dev/null) \+ || GH=$(gh --version 2>/dev/null | awk 'NR==1{print $3}')+if [[ -n "${GH:-}" && "$(gh --version 2>/dev/null | awk 'NR==1{print $3}')" != "$GH" ]]; then+ curl -fsSL --retry 3 "https://github.com/cli/cli/releases/download/v$GH/gh_${GH}_linux_amd64.tar.gz" \+ | tar xz -C /tmp \+ && install -m 0755 "/tmp/gh_${GH}_linux_amd64/bin/gh" /usr/local/bin/gh \+ || logger -t devbox-startup "WARN: gh $GH install failed"+ rm -rf "/tmp/gh_${GH}_linux_amd64"+fi++# ---------- (c) GitHub runners ----------+# Stateless, so it lives on scratch and is installed from the unit rather than from+# here: systemd retries that step forever, so a failed download heals itself.+cat > /usr/local/sbin/install-runner <<'EOF'+#!/bin/bash+# <account>. Root, from the unit. Re-runs whole on every retry, deps included.+set -euo pipefail+DIR=/scratch/$1/runner+# Latest release; if the lookup fails, keep what is installed, or fall back to a pin.+V=$(curl -sf --retry 3 --max-time 30 https://api.github.com/repos/actions/runner/releases/latest \+ | python3 -c 'import sys,json; print(json.load(sys.stdin)["tag_name"].lstrip("v"))' 2>/dev/null) \+ || V=$(cat "$DIR/.installed" 2>/dev/null || echo 2.336.0)+# Version-stamped and written last: a reboot keeps scratch, so a bare marker would pin+# the old release, and an unstamped one would skip a failed dep step.+[[ "$(cat "$DIR/.installed" 2>/dev/null)" == "$V" ]] && exit 0+rm -rf "$DIR" && install -d -o "$1" -g "$1" "$DIR" # clean tree: an old release's files must not linger+curl -fsSL --retry 3 "https://github.com/actions/runner/releases/download/v$V/actions-runner-linux-x64-$V.tar.gz" | tar xz -C "$DIR"+chown -R "$1:$1" "$DIR"+"$DIR"/bin/installdependencies.sh >/dev/null # apt is a no-op once satisfied+echo "$V" > "$DIR/.installed"+EOF++cat > /usr/local/sbin/register-runner <<'EOF'+#!/bin/bash+# <account>. Runs as the account, every boot.+set -euo pipefail+ORG=bytebase+md() { curl -sf -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/$1"; }+PROJECT=$(md project/project-id)+ACCESS=$(md instance/service-accounts/default/token | python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])')+PAT=$(curl -sf -H "Authorization: Bearer $ACCESS" \+ "https://secretmanager.googleapis.com/v1/projects/${PROJECT}/secrets/gh-runner-pat/versions/latest:access" \+ | python3 -c 'import sys,base64,json; print(base64.b64decode(json.load(sys.stdin)["payload"]["data"]).decode())')+TOKEN=$(curl -sfX POST -H "Authorization: Bearer $PAT" -H "Accept: application/vnd.github+json" \+ "https://api.github.com/orgs/${ORG}/actions/runners/registration-token" \+ | python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')+cd "/scratch/$1/runner"+rm -f .runner .credentials .credentials_rsakey # config.sh refuses to overwrite an existing config+./config.sh --unattended --replace --url "https://github.com/${ORG}" --token "$TOKEN" \+ --name "$(hostname -s)-$1" --work "/scratch/$1/work" # host-qualified name: never collides with another box+EOF+chmod +x /usr/local/sbin/install-runner /usr/local/sbin/register-runner++cat > /etc/systemd/system/[email protected] <<'EOF'+[Unit]+Description=GitHub Actions runner (%i)+Wants=network-online.target+After=network-online.target docker.service+[Service]+User=%i+WorkingDirectory=/scratch/%i/runner+Environment=XDG_CACHE_HOME=/scratch/%i/cache GOMODCACHE=/scratch/%i/cache/go-mod npm_config_cache=/scratch/%i/cache/npm PNPM_CONFIG_STORE_DIR=/scratch/%i/cache/pnpm+ExecStartPre=+/usr/local/sbin/install-runner %i+ExecStartPre=/usr/local/sbin/register-runner %i+ExecStart=/scratch/%i/runner/run.sh+# install-runner fetches ~200 MB and may wait on the apt lock; the 90s default would+# kill it, and the retry restarts the download from zero.+TimeoutStartSec=600+Restart=always+# Registration failure retries forever; at 5s x3 runners that would burn the API quota.+RestartSec=30+EOF++# ---------- (d) cache cleanup ----------+cat > /usr/local/sbin/cache-gc <<'EOF'+#!/bin/bash+# Every 30 min. Leaked go-build sandboxes are swept always. Above 85% used, wait for an+# idle box -- unless it is nearly full -- then delete everything disposable. It all+# rebuilds on the next job; a full disk does not.+set -uo pipefail+mountpoint -q /scratch || exit 0 # cron outlives a stop; before (a) has mounted, there is nothing to clean+HIGH=85 CRITICAL=95+find /tmp -maxdepth 1 -name 'go-build*' -type d -mmin +1440 -exec rm -rf {} ++used() { df -P /scratch | awk 'NR==2 {print $5+0}'; }+# Load catches detached work -- tmux, nohup, a container -- that holds neither an SSH+# session nor a Runner.Worker. On 20 vCPU, anything actually working exceeds 2.+idle() { ! pgrep -f Runner.Worker >/dev/null && [[ -z "$(ss -Htn state established '( sport = :22 )')" ]] \+ && (( $(cut -d. -f1 /proc/loadavg) < 2 )); }++[[ $(used) -ge $HIGH ]] || exit 0+if ! idle && [[ $(used) -lt $CRITICAL ]]; then logger -t cache-gc "deferred: busy at $(used)%"; exit 0; fi+logger -t cache-gc "cleaning: $(used)% used"+rm -rf /tmp/* /tmp/.[!.]* # /tmp is on scratch; the box is idle+docker ps -q | xargs -r docker rm -f >/dev/null 2>&1 # idle, so a running container is orphaned+docker system prune -af --volumes >/dev/null 2>&1+# Everything on scratch is disposable except the runner installs, which are large and+# would otherwise be re-downloaded on the next boot.+rm -rf /scratch/*/cache /scratch/*/home /scratch/*/work+for u in runner1 runner2 runner3 runner4; do+ install -d -o "$u" -g "$u" /scratch/$u/{cache,home,work} # the units need them; a profile remakes an interactive account's+done+logger -t cache-gc "cleaned -> $(used)%"+EOF+chmod +x /usr/local/sbin/cache-gc+# PATH first: cron's default omits /usr/sbin, and a missing `ss` would make idle() look true.+printf 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n*/30 * * * * root /usr/local/sbin/cache-gc\n' > /etc/cron.d/devbox++systemctl daemon-reload+systemctl restart actions-runner@{runner1,runner2,runner3,runner4}++# Last and time-bounded: observability must not hold up CI capacity.+systemctl is-active --quiet google-cloud-ops-agent \+ || { curl -sS --retry 3 --max-time 120 -o /tmp/ops.sh https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh \+ && timeout 300 bash /tmp/ops.sh --also-install; } \+ || logger -t devbox-startup "WARN: Ops Agent install failed; no memory or swap metrics"