Skip to content

Database governance built for humans and agents — controlling changes and access across every major database.

feat(infra): build the devbox, and point CI at it (#21332)

Danny Xu2ed531acdc7f
diff --git a/.github/workflows/start-runner-vm.yml b/.github/workflows/start-runner-vm.yml
index 7bd1ce69ff..22edc07d37 100644
--- a/.github/workflows/start-runner-vm.yml
+++ b/.github/workflows/start-runner-vm.yml
@@ -1,13 +1,15 @@
-# Starts the github-runner GCE VM, which hosts this repo's self-hosted Actions
-# runner (used by jobs with `runs-on: self-hosted`, e.g. backend-tests,
-# frontend-tests, golangci-lint, test_link).
-name: Start github-runner VM
+# Starts the devbox GCE VM, which hosts the org's self-hosted Actions runners
+# (used by jobs with `runs-on: self-hosted`, e.g. backend-tests, frontend-tests,
+# golangci-lint, test_link). It is a Spot instance, so a preemption stops it and
+# this workflow is what brings it back. Design: docs/design/devbox-and-ci-runner.md
+name: Start devbox VM
on:
workflow_dispatch:
schedule:
- # Three times a day, at 21:00, 02:00 and 06:00 UTC.
- - cron: "0 2,6,21 * * *"
+ # Every 15 minutes: that is the ceiling on how long a preemption leaves CI
+ # without runners. Starting an already-running instance is a no-op.
+ - cron: "*/15 * * * *"
permissions:
contents: read
@@ -24,12 +26,12 @@ jobs:
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v3
- - name: Start github-runner VM if stopped
+ - name: Start devbox VM if stopped
run: |
- STATUS=$(gcloud compute instances describe github-runner \
- --zone southamerica-east1-b --project bytebase-dev \
+ STATUS=$(gcloud compute instances describe devbox \
+ --zone northamerica-northeast2-a --project bytebase-dev \
--format='value(status)')
if [ "$STATUS" != "RUNNING" ]; then
- gcloud compute instances start github-runner \
- --zone southamerica-east1-b --project bytebase-dev
+ gcloud compute instances start devbox \
+ --zone northamerica-northeast2-a --project bytebase-dev
fi
diff --git a/docs/design/devbox-and-ci-runner.md b/docs/design/devbox-and-ci-runner.md
index b83775a796..42de7ec7ed 100644
--- a/docs/design/devbox-and-ci-runner.md
+++ b/docs/design/devbox-and-ci-runner.md
@@ -1,7 +1,7 @@
# Combined Dev Box and CI Runner on GCP
One Spot VM with two jobs. It is the SSH target that agentic coding tools drive, and
-it hosts three self-hosted GitHub Actions runners for the whole org.
+it hosts four self-hosted GitHub Actions runners for the whole org.
## Goals
@@ -11,14 +11,14 @@ it hosts three self-hosted GitHub Actions runners for the whole org.
2. **One startup script, so the box is close to stateless.** Everything on the box is
produced by that script. Only `/home` and a shared toolchain in `/usr/local` are
yours to manage; nothing else is set up by hand.
-3. **GitHub Actions runners that set themselves up.** Three org-level slots. Each
+3. **GitHub Actions runners that set themselves up.** Four org-level slots. Each
re-registers at boot from a secret, so there is no registration state to preserve.
4. **A clear split between Local SSD and the persistent disk.** Local SSD holds what
can be discarded: caches, Docker, temp files, swap. The boot disk holds repos and
uncommitted work. It can be lost too, so push your branches.
5. **Cache cleanup that will not break a running job.** Leaked temp files are swept
- always. Caches are evicted only under disk pressure, cheapest to rebuild first.
- The one exception is a disk about to fill, which would fail every job anyway.
+ always. Caches are evicted only under disk pressure, and only while the box is
+ idle. The one exception is a disk about to fill, which would fail every job anyway.
6. **Guest memory and swap visible in Cloud Monitoring.** The hypervisor reports only
CPU and disk. The Ops Agent reports the rest, so RAM is sized from measurement
instead of a guess.
@@ -36,7 +36,7 @@ it hosts three self-hosted GitHub Actions runners for the whole org.
| Provisioning | `SPOT`, `instanceTerminationAction=STOP` |
| Image | `ubuntu-2404-lts-amd64` |
| Boot disk | 100 GB pd-balanced, `--no-boot-disk-auto-delete` |
-| Scratch | 375 GB Local SSD (NVMe) |
+| Scratch | 2 x 375 GB Local SSD (NVMe), striped into one 750 GB `/scratch` |
| External IP | reserved; carries all egress, and inbound SSH |
| Inbound | SSH only, via the VPC default rule; no tag, no custom rules |
| Service account | dedicated; reads one secret, writes metrics and logs, nothing else |
@@ -46,18 +46,20 @@ it hosts three self-hosted GitHub Actions runners for the whole org.
|---|---|---|
| 20 vCPU (N2) | $0.003290 / vCPU-hr | $48.03 |
| 32 GiB RAM (N2) | $0.000441 / GiB-hr | $10.30 |
-| 375 GB Local SSD | $0.052800 / GB-mo | $19.80 |
+| 750 GB Local SSD | $0.052800 / GB-mo | $39.60 |
| 100 GB pd-balanced | $0.110 / GB-mo | $11.00 |
| Reserved external IP | $0.0025 / hr | $1.82 |
-| **Total** | | **$90.96** |
+| **Total** | | **$110.76** |
-Catalog prices, 2026-09-01. vCPU and RAM are changeable on a stopped instance with
-`set-machine-type`; Local SSD capacity is fixed at creation.
+Catalog prices, re-verified 2026-09-03. vCPU and RAM are changeable on a stopped
+instance with `set-machine-type`; Local SSD capacity is fixed at creation.
### Re-checking the region
Spot prices are set per family *per region*, and they drift. Re-check before any
-rebuild.
+rebuild. Swept 2026-09-03 over the Cloud Billing catalog against the 40 regions that
+can actually deploy each family: Toronto N2 was cheapest of all 80 region-family
+pairs, and stayed cheapest when the sweep was re-run at half the disk.
Price the whole VM, not just the CPU. Keep only regions that can actually run the
family: the catalog prices N1 in 14 regions that cannot run it. Prefer N2 or N2D,
@@ -79,7 +81,8 @@ manual — at `github.com/settings/personal-access-tokens/new`:
| Organization permissions → **Self-hosted runners** | **Read and write** |
| Repository permissions | none |
-Org-owned tokens may need an org admin to approve the request.
+An org-owned token may need an org admin to approve the request. An org owner's own
+request self-approves.
Developers already have access: the developer group holds Editor on the project,
which covers OS Login with `sudo` and acting as the service account.
@@ -88,51 +91,14 @@ which covers OS Login with `sudo` and acting as the service account.
on the secret, which Editor cannot do. It stops at the first failed step.
```bash
-set -e
-PROJECT=bytebase-dev
-ZONE=northamerica-northeast2-a
-SA=devbox@${PROJECT}.iam.gserviceaccount.com
-
-gcloud services enable secretmanager.googleapis.com oslogin.googleapis.com \
- monitoring.googleapis.com logging.googleapis.com --project=$PROJECT
-
-# Service account; its roles are granted below, nothing more.
-gcloud iam service-accounts create devbox --project=$PROJECT \
- --display-name="devbox: dev host and CI runners"
-
-# The PAT, readable by that service account and nothing else.
-read -rsp 'GitHub PAT: ' GH_PAT && echo
-printf '%s' "$GH_PAT" | gcloud secrets create gh-runner-pat \
- --project=$PROJECT --replication-policy=automatic --data-file=-
-unset GH_PAT
-
-gcloud secrets add-iam-policy-binding gh-runner-pat --project=$PROJECT \
- --member="serviceAccount:${SA}" --role=roles/secretmanager.secretAccessor
-
-# Metrics and logs from the Ops Agent. Without these the agent gets 403s and goal 6
-# is silently unmet -- a scope is not a role.
-for r in monitoring.metricWriter logging.logWriter; do
- gcloud projects add-iam-policy-binding $PROJECT \
- --member="serviceAccount:${SA}" --role=roles/$r
-done
-
-# A reserved address. An ephemeral one is released when the instance stops, so every
-# preemption would hand the box a new IP and strand the generated SSH entry.
-gcloud compute addresses create devbox --project=$PROJECT --region=${ZONE%-*}
-
-# The instance.
-gcloud compute instances create devbox --project=$PROJECT --zone=$ZONE \
- --address=devbox \
- --machine-type=n2-custom-20-32768 \
- --provisioning-model=SPOT --instance-termination-action=STOP \
- --image-family=ubuntu-2404-lts-amd64 --image-project=ubuntu-os-cloud \
- --boot-disk-size=100GB --boot-disk-type=pd-balanced --no-boot-disk-auto-delete \
- --local-ssd=interface=NVME \
- --service-account=$SA --scopes=cloud-platform \
- --metadata=enable-oslogin=TRUE \
- --metadata-from-file=startup-script=./startup.sh
+./scripts/devbox/create.sh
```
+It enables the APIs, creates the service account, stores the PAT it prompts for in
+Secret Manager, grants that account the secret plus `monitoring.metricWriter` and
+`logging.logWriter`, reserves the address, and creates the instance with
+`startup.sh` as its startup-script metadata.
+
The `cloud-platform` scope resolves to the intersection with the account's roles: one
secret, metrics, logs. The scope is not the boundary. The IAM bindings are.
@@ -140,24 +106,44 @@ There is no flag day. Runners register at the organization under the automatic
`self-hosted` label, with host-qualified names. So this box only adds capacity
alongside any existing runner, and you can delete the old one whenever.
-One step does have to happen in order. A preemption stops the instance, and
-`.github/workflows/start-runner-vm.yml` already restarts a runner box on a schedule.
-Repoint its instance name and zone at this one, in two places each, as the last step
-of cutover. Do it earlier and it stops restarting the box it serves today. Until then,
-start this one by hand.
+One step has to happen in order, and it is the last. A preemption stops the instance,
+and `.github/workflows/start-runner-vm.yml` restarts it on a schedule. That workflow
+now names this instance and zone, in two places each. Repointing it any earlier would
+have stopped it restarting the box that was serving CI at the time.
-Raise its cadence at the same time. It runs at three fixed hours, so a preemption just
-after one of them leaves the box stopped for up to fifteen -- which is not "always on"
-for either the dev box or CI. Every fifteen minutes bounds the outage at fifteen
-minutes, and starting a running instance is a no-op.
+Its cadence changed with it, from three fixed hours to every fifteen minutes. At three
+a day, a preemption just after one of them left the box stopped for up to fifteen
+hours, which is not "always on" for either the dev box or CI. Fifteen minutes bounds
+the outage at fifteen minutes, and starting a running instance is a no-op.
### Connect
-Run `gcloud compute config-ssh --project=bytebase-dev` once. It writes the `~/.ssh/config` entries, and this
-box's alias is `devbox.northamerica-northeast2-a.bytebase-dev`. Desktop agents take
-that alias as their SSH host: Claude Desktop under **Add SSH connection**, Codex in
-its SSH environment. Each developer gets their own account and home, created by OS
-Login on first connect.
+```
+Host devbox
+ HostName devbox
+ User <your OS Login name>
+ IdentityFile ~/.ssh/google_compute_engine
+ ProxyCommand /path/to/gcloud compute start-iap-tunnel devbox 22 --listen-on-stdin --project=bytebase-dev --zone=northamerica-northeast2-a --verbosity=warning
+```
+
+Then `ssh devbox`, and the same host in Claude Desktop's **Add SSH connection**.
+Run `gcloud compute config-ssh --project=bytebase-dev` once first to register your key;
+`gcloud compute os-login describe-profile` prints your account name. Keep this entry
+outside the block config-ssh writes.
+
+Three things about that `ProxyCommand`:
+
+- **It tunnels SSH over HTTPS**, so it works on a network that intercepts port 22 --
+ a VPN resolving hosts into `198.18.0.0/15` does, and a direct alias then dies at
+ `kex_exchange_identification`. Nothing needs opening; the default VPC rule already
+ admits IAP's range.
+- **Absolute path, not `gcloud`.** GUI apps inherit launchd's minimal `PATH`, so a
+ bare `gcloud` resolves in your terminal and fails in Claude Desktop.
+- **It needs a live gcloud session.** When it lapses, `ssh devbox` fails on reauth
+ rather than on the network; `gcloud auth login` restores it.
+
+Without the tunnel, on a network that permits port 22,
+`ssh devbox.northamerica-northeast2-a.bytebase-dev` works with no entry at all.
Log in once interactively before pointing an agent at a new account. The cache links
in (b) are made by the profile, and a non-login `bash -c` does not read one -- so an
@@ -169,17 +155,18 @@ at all: it comes from `/etc/environment`, which every session reads.
## 2. Startup script
-Saved as `startup.sh` and passed to the create command above. It runs as root on every
-boot and is idempotent. Local SSD is discarded on every stop, so anything living there
-is rebuilt. Anything on the boot disk is guarded, so it is not redone.
+[`scripts/devbox/startup.sh`](../../scripts/devbox/startup.sh), passed to the create
+command above as startup-script metadata. It runs as root on every boot and is
+idempotent. Local SSD is discarded on every stop, so anything living there is
+rebuilt. Anything on the boot disk is guarded, so it is not redone.
### a. Disk layout
Two kinds of account use the box, and they need opposite things from it.
-**Runner accounts** are `runner1`, `runner2` and `runner3`. The script creates them, `nologin`,
-one per job slot. Nothing of theirs is worth keeping: software, work trees and caches
-all sit on Local SSD, and their home directories are on it too. They are
+**Runner accounts** are `runner1` through `runner4`. The script creates them,
+`nologin`, one per job slot. Nothing of theirs is worth keeping: software, work trees
+and caches all sit on Local SSD, and their home directories are on it too. They are
disposable, and are disposed of on every stop.
**Interactive accounts** arrive through OS Login on first connect. Their `/home/<u>`
@@ -199,6 +186,15 @@ Local SSD also carries Docker's `data-root`, the swapfile and `/tmp`. Container
startup and database `fsync` are the heaviest I/O on this box. Docker starts before
the script runs, so the script points it at Local SSD and restarts it.
+There are two partitions because N2 will not attach one at 20 vCPU, and the script
+stripes them into a single 750 GB `/scratch`. A reboot keeps Local SSD, and udev
+re-assembles the array under a name of its own choosing -- `/dev/md127`, not the
+`/dev/md0` it was created as -- so the script asks `mdadm --detail --scan` what is
+live before it creates anything. Creating over a live array would discard every
+cache on the box, quietly, on every reboot. Stopping by hand needs
+`--discard-local-ssd=true`, which gcloud now requires to be explicit; that is what a
+preemption does regardless.
+
**Swap causes hangs; it does not prevent them.** Without swap, exhausting RAM is a
fast OOM kill. With swap, the box thrashes until the job timeout. So there is 8 GB at
`vm.swappiness=10`: enough to evict cold pages, too little to sustain a thrash. And
@@ -215,6 +211,11 @@ every session, non-login included. CI jobs bring their own through `setup-go` an
installs one, and Go defaults to `CGO_ENABLED=1`, so dependencies carrying cgo files
fail to build without it.
+`gh` is the one tool the script does install, from upstream rather than apt, which
+ships 2.45.0. It lands in `/usr/local/bin` on the boot disk, so a reboot is a no-op
+and only a rebuild or a new release refetches. A failed fetch warns rather than
+calling `fatal`: a boot must not depend on GitHub being reachable.
+
### b. Cache paths
Four paths carry every cache. `XDG_CACHE_HOME` relocates the Go **build** cache,
@@ -257,12 +258,13 @@ and `node_modules` — which is right for anything inside a working copy.
### c. GitHub runners
-Three accounts, three job slots, one systemd template. A pull request touching both
+Four accounts, four job slots, one systemd template. A pull request touching both
backend and frontend schedules five self-hosted jobs -- one each from `backend-tests`,
-`golangci-lint` and `test_link`, two from `frontend-tests` -- so two of them queue,
+`golangci-lint` and `test_link`, two from `frontend-tests` -- so one of them queues,
before counting any other repository in the org. That is deliberate: jobs queue, they
-do not fail, and 20 vCPU split five ways is thin. A fourth or fifth slot is one more
-name in the script's account list if the wait proves worse than the contention.
+do not fail. Four ways is five cores and 8 GB a slot, which is what a comparable box
+ran two slots on; watch `memory/percent_used` before adding a fifth, which is one more
+name in the script's account list.
The runner is stateless, so it lives on scratch and is installed by the unit rather
than by the startup script: an `ExecStartPre` runs `install-runner` as root, before
@@ -279,7 +281,7 @@ server-side entry of the same name. Names are host-qualified, so two boxes never
each other's.
Re-registering rather than preserving credentials means the runner holds no identity
-on disk. Wipe the boot disk and the same service account brings back the same three
+on disk. Wipe the boot disk and the same service account brings back the same four
runners. Registration is derived state, like `/scratch`.
The units carry no `[Install]` section. The script starts them rather than enabling
@@ -308,221 +310,6 @@ It runs on a schedule, not at boot. A boot-time check cannot fire on a box that
up for days, and Local SSD survives a guest *reboot*. It is discarded only on stop or
preemption.
-### startup.sh
-
-```bash
-#!/bin/bash
-# GCE startup-script: root, every boot, idempotent. No `set -e`: a failed step must
-# not abandon the boot; `journalctl -t devbox-startup` has the exit status.
-set -uo pipefail
-trap 'logger -t devbox-startup "exited: status $?"' EXIT
-# A fatal exit powers the box off. Left RUNNING with no runners, the start workflow
-# would never touch it; off, it is restarted and this script retried on the
-# workflow's next tick. Five minutes keeps SSH usable for the journal or /home first.
-fatal() {
- logger -t devbox-startup "FATAL: $1"; systemctl stop docker.socket docker containerd 2>/dev/null
- shutdown -h +5 "devbox-startup: $1"; exit 1
-}
-
-# Docker down and masked until (a) has mounted /scratch. daemon.json persists, so the
-# previous boot's dockerd is already up with its data root on the boot disk, and
-# mounting over it would leave it writing there unseen. Masked, nothing can restart it
-# -- not apt's postinst, not a connection to a still-listening socket -- until (a) is done.
-systemctl mask --now docker.socket docker containerd 2>/dev/null
-
-# ---------- packages ----------
-export DEBIAN_FRONTEND=noninteractive
-echo 'DPkg::Lock::Timeout "300";' > /etc/apt/apt.conf.d/99-lock-timeout # unattended-upgrades holds the lock at boot
-PKGS="docker.io cron systemd-oomd build-essential" # build-essential: Go defaults to CGO_ENABLED=1
-dpkg --configure -a 2>/dev/null || true # clears a transaction a preemption cut short; apt repairs the rest
-apt-get -y -qq -f install $PKGS 2>/dev/null || { apt-get update -qq && apt-get -y -qq -f install $PKGS; } \
- || fatal "prerequisite install failed"
-
-# ---------- (a) disk layout ----------
-mkdir -p /scratch
-DEV=$(ls /dev/disk/by-id/google-local-* 2>/dev/null | head -1)
-if [[ -n "$DEV" ]] && ! mountpoint -q /scratch; then
- # A stop discards Local SSD; a reboot does not. Format only when there is no filesystem.
- [[ -n "$(blkid -s TYPE -o value "$DEV")" ]] || mkfs.ext4 -F -m 0 -E lazy_itable_init=0,lazy_journal_init=0,discard "$DEV"
- mount -o noatime,discard "$DEV" /scratch
-fi
-# Everything below assumes it. Falling back to the boot disk would put swap, Docker,
-# work trees and caches on the 100 GB disk that holds /home.
-mountpoint -q /scratch || fatal "no Local SSD at /scratch"
-chmod 1777 /scratch # OS Login accounts create their own subtree
-mkdir -p /scratch/tmp && chmod 1777 /scratch/tmp
-# Compare inodes, not `mountpoint`: a tmpfs /tmp is a mountpoint too, and would leave temp files in RAM.
-[ "$(stat -c '%d:%i' /tmp)" = "$(stat -c '%d:%i' /scratch/tmp)" ] || mount --bind /scratch/tmp /tmp \
- || fatal "/tmp not on Local SSD"
-swapon --show=NAME --noheadings | grep -q /scratch/swapfile \
- || { rm -f /scratch/swapfile && fallocate -l 8G /scratch/swapfile && chmod 600 /scratch/swapfile \
- && mkswap /scratch/swapfile >/dev/null && swapon /scratch/swapfile; } \
- || logger -t devbox-startup "WARN: no swap active"
-sysctl -qw vm.swappiness=10
-systemctl enable --now systemd-oomd || logger -t devbox-startup "WARN: systemd-oomd inert"
-# On the slice: containers get their own scopes under system.slice, not under the runner unit.
-mkdir -p /etc/systemd/system/system.slice.d
-printf '[Slice]\nManagedOOMMemoryPressure=kill\n' > /etc/systemd/system/system.slice.d/oomd.conf
-
-# Docker on Local SSD with log rotation. The socket is opened to every account: OS
-# Login users appear on first connect and cannot be pre-added to the docker group.
-mkdir -p /scratch/docker /scratch/containerd /var/lib/containerd /etc/systemd/system/docker.service.d
-echo '{"data-root":"/scratch/docker","log-driver":"json-file","log-opts":{"max-size":"10m","max-file":"3"}}' > /etc/docker/daemon.json
-printf '[Service]\nExecStartPost=/bin/chmod 666 /var/run/docker.sock\n' > /etc/systemd/system/docker.service.d/socket.conf
-# Engine 29 keeps images in containerd's own store, which data-root does not cover. A
-# bind needs no containerd config to track.
-mountpoint -q /var/lib/containerd || mount --bind /scratch/containerd /var/lib/containerd \
- || fatal "containerd store not on Local SSD"
-# Masked until here: an active docker.socket cannot spawn a masked service, so nothing
-# can start dockerd on the boot disk between the stop above and this restart. And
-# disabled from here on: only this script starts Docker, so a reboot no longer brings
-# it up on the boot disk before the script has run.
-systemctl unmask docker.socket docker containerd 2>/dev/null
-systemctl disable docker.socket docker containerd 2>/dev/null \
- || logger -t devbox-startup "WARN: docker autostart still enabled; the mask covers the next boot"
-systemctl daemon-reload && systemctl restart containerd docker || fatal "docker would not start"
-docker info --format '{{.DockerRootDir}}' 2>/dev/null | grep -q '^/scratch/' \
- || fatal "docker not on Local SSD" # daemon.json unwritten or rejected: dockerd starts fine on the boot disk
-
-# ---------- (b) accounts and cache paths ----------
-for u in runner1 runner2 runner3; do
- # Home on scratch: whatever a job writes home-relative is disposable too.
- id "$u" &>/dev/null || useradd -M -d "/scratch/$u/home" -s /usr/sbin/nologin "$u"
- # runner/ up front: systemd applies WorkingDirectory before ExecStartPre could create it.
- mkdir -p "/scratch/$u"/{cache,work,runner,home}
- chown "$u:$u" "/scratch/$u" "/scratch/$u"/{cache,work,runner,home} # not -R: contents are the user's own
-done
-
-# Interactive accounts appear at first OS Login connect, so the profile provisions them:
-# each tool's default cache path becomes a symlink onto scratch. Symlinks, not variables,
-# so an agent's `bash -c` -- which sources no profile -- lands there too. PATH goes in
-# /etc/environment for the same reason: pam_env applies it to every session.
-echo 'PATH="/usr/local/go/bin:/usr/local/node/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"' > /etc/environment
-cat > /etc/profile.d/devbox.sh <<'EOF'
-u=$(id -un); c=/scratch/$u/cache
-if mountpoint -q /scratch && [ -n "$u" ] && [ -n "${HOME:-}" ] && mkdir -p "$c" 2>/dev/null; then
- for pair in "$HOME/.cache:$c" "$HOME/go/pkg/mod:$c/go-mod" "$HOME/.npm:$c/npm" "$HOME/.local/share/pnpm:$c/pnpm"; do
- link=${pair%%:*}; target=${pair#*:}
- mkdir -p "$target" # every login: scratch targets vanish on a stop, the symlink in HOME does not
- [ -L "$link" ] || { mkdir -p "$(dirname "$link")" && rm -rf "$link" && ln -s "$target" "$link"; }
- done 2>/dev/null
-fi
-unset u c pair link target
-EOF
-# Existing accounts keep their symlinks across a stop but not the targets. Remake them,
-# so only a brand-new account needs a first interactive login.
-for h in /home/*; do
- u=${h##*/}; id "$u" &>/dev/null || continue
- for d in "" /go-mod /npm /pnpm; do install -d -o "$u" -g "$u" "/scratch/$u/cache$d"; done
-done
-
-# ---------- (c) GitHub runners ----------
-# Stateless, so it lives on scratch and is installed from the unit rather than from
-# here: systemd retries that step forever, so a failed download heals itself.
-cat > /usr/local/sbin/install-runner <<'EOF'
-#!/bin/bash
-# <account>. Root, from the unit. Re-runs whole on every retry, deps included.
-set -euo pipefail
-DIR=/scratch/$1/runner
-# Latest release; if the lookup fails, keep what is installed, or fall back to a pin.
-V=$(curl -sf --retry 3 --max-time 30 https://api.github.com/repos/actions/runner/releases/latest \
- | python3 -c 'import sys,json; print(json.load(sys.stdin)["tag_name"].lstrip("v"))' 2>/dev/null) \
- || V=$(cat "$DIR/.installed" 2>/dev/null || echo 2.336.0)
-# Version-stamped and written last: a reboot keeps scratch, so a bare marker would pin
-# the old release, and an unstamped one would skip a failed dep step.
-[[ "$(cat "$DIR/.installed" 2>/dev/null)" == "$V" ]] && exit 0
-rm -rf "$DIR" && install -d -o "$1" -g "$1" "$DIR" # clean tree: an old release's files must not linger
-curl -fsSL --retry 3 "https://github.com/actions/runner/releases/download/v$V/actions-runner-linux-x64-$V.tar.gz" | tar xz -C "$DIR"
-chown -R "$1:$1" "$DIR"
-"$DIR"/bin/installdependencies.sh >/dev/null # apt is a no-op once satisfied
-echo "$V" > "$DIR/.installed"
-EOF
-
-cat > /usr/local/sbin/register-runner <<'EOF'
-#!/bin/bash
-# <account>. Runs as the account, every boot.
-set -euo pipefail
-ORG=bytebase
-md() { curl -sf -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/$1"; }
-PROJECT=$(md project/project-id)
-ACCESS=$(md instance/service-accounts/default/token | python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])')
-PAT=$(curl -sf -H "Authorization: Bearer $ACCESS" \
- "https://secretmanager.googleapis.com/v1/projects/${PROJECT}/secrets/gh-runner-pat/versions/latest:access" \
- | python3 -c 'import sys,base64,json; print(base64.b64decode(json.load(sys.stdin)["payload"]["data"]).decode())')
-TOKEN=$(curl -sfX POST -H "Authorization: Bearer $PAT" -H "Accept: application/vnd.github+json" \
- "https://api.github.com/orgs/${ORG}/actions/runners/registration-token" \
- | python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
-cd "/scratch/$1/runner"
-rm -f .runner .credentials .credentials_rsakey # config.sh refuses to overwrite an existing config
-./config.sh --unattended --replace --url "https://github.com/${ORG}" --token "$TOKEN" \
- --name "$(hostname -s)-$1" --work "/scratch/$1/work" # host-qualified name: never collides with another box
-EOF
-chmod +x /usr/local/sbin/install-runner /usr/local/sbin/register-runner
-
-cat > /etc/systemd/system/[email protected] <<'EOF'
-[Unit]
-Description=GitHub Actions runner (%i)
-Wants=network-online.target
-After=network-online.target docker.service
-[Service]
-User=%i
-WorkingDirectory=/scratch/%i/runner
-Environment=XDG_CACHE_HOME=/scratch/%i/cache GOMODCACHE=/scratch/%i/cache/go-mod npm_config_cache=/scratch/%i/cache/npm PNPM_CONFIG_STORE_DIR=/scratch/%i/cache/pnpm
-ExecStartPre=+/usr/local/sbin/install-runner %i
-ExecStartPre=/usr/local/sbin/register-runner %i
-ExecStart=/scratch/%i/runner/run.sh
-# install-runner fetches ~200 MB and may wait on the apt lock; the 90s default would
-# kill it, and the retry restarts the download from zero.
-TimeoutStartSec=600
-Restart=always
-# Registration failure retries forever; at 5s x3 runners that would burn the API quota.
-RestartSec=30
-EOF
-
-# ---------- (d) cache cleanup ----------
-cat > /usr/local/sbin/cache-gc <<'EOF'
-#!/bin/bash
-# Every 30 min. Leaked go-build sandboxes are swept always. Above 85% used, wait for an
-# idle box -- unless it is nearly full -- then delete everything disposable. It all
-# rebuilds on the next job; a full disk does not.
-set -uo pipefail
-mountpoint -q /scratch || exit 0 # cron outlives a stop; before (a) has mounted, there is nothing to clean
-HIGH=85 CRITICAL=95
-find /tmp -maxdepth 1 -name 'go-build*' -type d -mmin +1440 -exec rm -rf {} +
-used() { df -P /scratch | awk 'NR==2 {print $5+0}'; }
-# Load catches detached work -- tmux, nohup, a container -- that holds neither an SSH
-# session nor a Runner.Worker. On 20 vCPU, anything actually working exceeds 2.
-idle() { ! pgrep -f Runner.Worker >/dev/null && [[ -z "$(ss -Htn state established '( sport = :22 )')" ]] \
- && (( $(cut -d. -f1 /proc/loadavg) < 2 )); }
-
-[[ $(used) -ge $HIGH ]] || exit 0
-if ! idle && [[ $(used) -lt $CRITICAL ]]; then logger -t cache-gc "deferred: busy at $(used)%"; exit 0; fi
-logger -t cache-gc "cleaning: $(used)% used"
-rm -rf /tmp/* /tmp/.[!.]* # /tmp is on scratch; the box is idle
-docker ps -q | xargs -r docker rm -f >/dev/null 2>&1 # idle, so a running container is orphaned
-docker system prune -af --volumes >/dev/null 2>&1
-# Everything on scratch is disposable except the runner installs, which are large and
-# would otherwise be re-downloaded on the next boot.
-rm -rf /scratch/*/cache /scratch/*/home /scratch/*/work
-for u in runner1 runner2 runner3; do
- install -d -o "$u" -g "$u" /scratch/$u/{cache,home,work} # the units need them; a profile remakes an interactive account's
-done
-logger -t cache-gc "cleaned -> $(used)%"
-EOF
-chmod +x /usr/local/sbin/cache-gc
-# PATH first: cron's default omits /usr/sbin, and a missing `ss` would make idle() look true.
-printf 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n*/30 * * * * root /usr/local/sbin/cache-gc\n' > /etc/cron.d/devbox
-
-systemctl daemon-reload
-systemctl restart actions-runner@{runner1,runner2,runner3}
-
-# Last and time-bounded: observability must not hold up CI capacity.
-systemctl is-active --quiet google-cloud-ops-agent \
- || { curl -sS --retry 3 --max-time 120 -o /tmp/ops.sh https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh \
- && timeout 300 bash /tmp/ops.sh --also-install; } \
- || logger -t devbox-startup "WARN: Ops Agent install failed; no memory or swap metrics"
-```
-
## 3. Considered
**N2D** — its guaranteed Milan silicon avoids N2's Cascade-Lake-or-Ice-Lake lottery,
@@ -544,9 +331,18 @@ with `set-scheduling`.
stays up for the union of both duty cycles at the sum of both sizes. Consolidation was
chosen for operational simplicity.
-**750 GB Local SSD** — another $19.80/mo, against a projected 49% use of 375 GB. Local
-SSD is the one capacity fixed at creation, so it is the first thing to revisit if that
-projection is wrong.
+**375 GB Local SSD** — not offered. N2 at 20 vCPU accepts 0, 2, 4, 8, 16 or 24
+partitions, so the smallest non-zero scratch is two, and goal 1 fixes the vCPU count.
+Both are striped rather than leaving one idle, since they are paid for either way.
+Local SSD is the one capacity fixed at creation, so revisit it before any rebuild.
+
+**A larger persistent disk instead of Local SSD** — slower and dearer at every size
+tried. Measured on this box with `fio`, pd-balanced returns the same 12.7k random
+read IOPS and 547 MB/s at 100 GB, 500 GB and 850 GB, because a 20 vCPU instance caps
+throughput long before per-GB scaling matters. The striped Local SSD returns 360k
+IOPS and 1407 MB/s, and costs $0.0528/GB-mo against pd-balanced's $0.110. Buying
+persistent disk here would pay twice per GB for a twenty-eighth of the IOPS. What it
+buys instead is durability, which is why `/home` is on it and nothing else is.
**A firewall rule and network tag** — the default VPC leaves several ports open to
`0.0.0.0/0`, but nothing here listens on them. Tags can be added to a running instance,
@@ -571,7 +367,8 @@ change any `/home` on the box -- uncommitted work, agent CLI state, `~/.ssh`. Tr
this as a shared machine and keep nothing on it you would not put on one.
**Pinning Ice Lake** — deferred. `cpuPlatform` after each restart shows what Toronto
-hands out. A pin, or a different region, is a stop-and-start away.
+hands out; so far, Cascade Lake every time. A pin, or a different region, is a
+stop-and-start away.
## Reference
@@ -579,3 +376,4 @@ hands out. A pin, or a different region, is a stop-and-start away.
- Self-hosted runners: https://docs.github.com/en/actions/hosting-your-own-runners
- Fine-grained PAT permissions: https://docs.github.com/en/rest/authentication/permissions-required-for-fine-grained-personal-access-tokens
- CPU platforms: https://cloud.google.com/compute/docs/cpu-platforms
+- IAP TCP forwarding: https://cloud.google.com/iap/docs/using-tcp-forwarding
diff --git a/scripts/devbox/create.sh b/scripts/devbox/create.sh
new file mode 100755
index 0000000000..e3ca31e6c8
--- /dev/null
+++ b/scripts/devbox/create.sh
@@ -0,0 +1,56 @@
+#!/bin/bash
+# Creates the devbox: the SSH target for agentic coding tools and the host for the
+# org's self-hosted GitHub Actions runners. Design: docs/design/devbox-and-ci-runner.md
+#
+# Run once, as a project Owner: it sets IAM policy on the project and on the secret,
+# which Editor cannot do. It stops at the first failed step and is not idempotent --
+# after a partial run, delete what was created before re-running.
+#
+# Prerequisite, and the one step no API can do: a fine-grained GitHub PAT owned by the
+# `bytebase` org, with organization permission "Self-hosted runners: Read and write"
+# and nothing else. See the Create section of the design doc.
+set -euo pipefail
+PROJECT=bytebase-dev
+ZONE=northamerica-northeast2-a
+SA=devbox@${PROJECT}.iam.gserviceaccount.com
+HERE=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)
+
+gcloud services enable secretmanager.googleapis.com oslogin.googleapis.com \
+ monitoring.googleapis.com logging.googleapis.com --project=$PROJECT
+
+# Service account; its roles are granted below, nothing more.
+gcloud iam service-accounts create devbox --project=$PROJECT \
+ --display-name="devbox: dev host and CI runners"
+
+# The PAT, readable by that service account and nothing else.
+read -rsp 'GitHub PAT: ' GH_PAT; echo # not `&& echo`: set -e ignores a failed read inside an && list
+printf '%s' "$GH_PAT" | gcloud secrets create gh-runner-pat \
+ --project=$PROJECT --replication-policy=automatic --data-file=-
+unset GH_PAT
+
+gcloud secrets add-iam-policy-binding gh-runner-pat --project=$PROJECT \
+ --member="serviceAccount:${SA}" --role=roles/secretmanager.secretAccessor
+
+# Metrics and logs from the Ops Agent. Without these the agent gets 403s and guest
+# memory never reaches Cloud Monitoring -- a scope is not a role.
+for r in monitoring.metricWriter logging.logWriter; do
+ gcloud projects add-iam-policy-binding $PROJECT \
+ --member="serviceAccount:${SA}" --role=roles/$r --condition=None
+done
+
+# A reserved address. An ephemeral one is released when the instance stops, so every
+# preemption would hand the box a new IP and strand the generated SSH entry.
+gcloud compute addresses create devbox --project=$PROJECT --region=${ZONE%-*}
+
+# The instance.
+gcloud compute instances create devbox --project=$PROJECT --zone=$ZONE \
+ --address=devbox \
+ --machine-type=n2-custom-20-32768 \
+ --provisioning-model=SPOT --instance-termination-action=STOP \
+ --image-family=ubuntu-2404-lts-amd64 --image-project=ubuntu-os-cloud \
+ --boot-disk-size=100GB --boot-disk-type=pd-balanced --no-boot-disk-auto-delete \
+ --local-ssd=interface=NVME --local-ssd=interface=NVME \
+ `# two partitions, not one: N2 at 20 vCPU accepts only 0, 2, 4, 8, 16 or 24` \
+ --service-account=$SA --scopes=cloud-platform \
+ --metadata=enable-oslogin=TRUE \
+ --metadata-from-file=startup-script="$HERE/startup.sh"
diff --git a/scripts/devbox/startup.sh b/scripts/devbox/startup.sh
new file mode 100755
index 0000000000..fff8362917
--- /dev/null
+++ b/scripts/devbox/startup.sh
@@ -0,0 +1,238 @@
+#!/bin/bash
+# GCE startup-script: root, every boot, idempotent. No `set -e`: a failed step must
+# not abandon the boot; `journalctl -t devbox-startup` has the exit status.
+set -uo pipefail
+trap 'logger -t devbox-startup "exited: status $?"' EXIT
+# A fatal exit powers the box off. Left RUNNING with no runners, the start workflow
+# would never touch it; off, it is restarted and this script retried on the
+# workflow's next tick. Five minutes keeps SSH usable for the journal or /home first.
+fatal() {
+ logger -t devbox-startup "FATAL: $1"; systemctl stop docker.socket docker containerd 2>/dev/null
+ shutdown -h +5 "devbox-startup: $1"; exit 1
+}
+
+# Docker down and masked until (a) has mounted /scratch. daemon.json persists, so the
+# previous boot's dockerd is already up with its data root on the boot disk, and
+# mounting over it would leave it writing there unseen. Masked, nothing can restart it
+# -- not apt's postinst, not a connection to a still-listening socket -- until (a) is done.
+systemctl mask --now docker.socket docker containerd 2>/dev/null
+
+# ---------- packages ----------
+export DEBIAN_FRONTEND=noninteractive
+echo 'DPkg::Lock::Timeout "300";' > /etc/apt/apt.conf.d/99-lock-timeout # unattended-upgrades holds the lock at boot
+PKGS="docker.io cron systemd-oomd build-essential mdadm" # build-essential: Go defaults to CGO_ENABLED=1
+dpkg --configure -a 2>/dev/null || true # clears a transaction a preemption cut short; apt repairs the rest
+apt-get -y -qq -f install $PKGS 2>/dev/null || { apt-get update -qq && apt-get -y -qq -f install $PKGS; } \
+ || fatal "prerequisite install failed"
+
+# ---------- (a) disk layout ----------
+mkdir -p /scratch
+DEVS=(); for d in /dev/disk/by-id/google-local-*; do [[ -b $d && $d != *-part* ]] && DEVS+=("$d"); done
+DEV=${DEVS[0]:-}
+# N2 refuses a single Local SSD at 20 vCPU -- 0, 2, 4, 8, 16 or 24 only -- so there are
+# always at least two, and they are striped into one /scratch. A reboot keeps Local SSD
+# and udev may have re-assembled the array under a name of its choosing, so look for a
+# live array, then for an assemblable one, and only then create: `mdadm --create` over
+# an existing array discards it.
+if (( ${#DEVS[@]} > 1 )); then
+ md() { mdadm --detail --scan 2>/dev/null | awk 'NR==1{print $2}'; }
+ DEV=$(md)
+ [[ -n "$DEV" ]] || { mdadm --assemble --scan >/dev/null 2>&1; DEV=$(md); }
+ [[ -n "$DEV" ]] || { mdadm --create /dev/md0 --level=0 --raid-devices="${#DEVS[@]}" \
+ --run "${DEVS[@]}" >/dev/null 2>&1 && DEV=/dev/md0; }
+fi
+if [[ -n "$DEV" ]] && ! mountpoint -q /scratch; then
+ # A stop discards Local SSD; a reboot does not. Format only when there is no filesystem.
+ [[ -n "$(blkid -s TYPE -o value "$DEV")" ]] || mkfs.ext4 -F -m 0 -E lazy_itable_init=0,lazy_journal_init=0,discard "$DEV"
+ mount -o noatime,discard "$DEV" /scratch
+fi
+# Everything below assumes it. Falling back to the boot disk would put swap, Docker,
+# work trees and caches on the 100 GB disk that holds /home.
+mountpoint -q /scratch || fatal "no Local SSD at /scratch"
+chmod 1777 /scratch # OS Login accounts create their own subtree
+mkdir -p /scratch/tmp && chmod 1777 /scratch/tmp
+# Compare inodes, not `mountpoint`: a tmpfs /tmp is a mountpoint too, and would leave temp files in RAM.
+[ "$(stat -c '%d:%i' /tmp)" = "$(stat -c '%d:%i' /scratch/tmp)" ] || mount --bind /scratch/tmp /tmp \
+ || fatal "/tmp not on Local SSD"
+swapon --show=NAME --noheadings | grep -q /scratch/swapfile \
+ || { rm -f /scratch/swapfile && fallocate -l 8G /scratch/swapfile && chmod 600 /scratch/swapfile \
+ && mkswap /scratch/swapfile >/dev/null && swapon /scratch/swapfile; } \
+ || logger -t devbox-startup "WARN: no swap active"
+sysctl -qw vm.swappiness=10
+systemctl enable --now systemd-oomd || logger -t devbox-startup "WARN: systemd-oomd inert"
+# On the slice: containers get their own scopes under system.slice, not under the runner unit.
+mkdir -p /etc/systemd/system/system.slice.d
+printf '[Slice]\nManagedOOMMemoryPressure=kill\n' > /etc/systemd/system/system.slice.d/oomd.conf
+
+# Docker on Local SSD with log rotation. The socket is opened to every account: OS
+# Login users appear on first connect and cannot be pre-added to the docker group.
+mkdir -p /scratch/docker /scratch/containerd /var/lib/containerd /etc/systemd/system/docker.service.d
+echo '{"data-root":"/scratch/docker","log-driver":"json-file","log-opts":{"max-size":"10m","max-file":"3"}}' > /etc/docker/daemon.json
+printf '[Service]\nExecStartPost=/bin/chmod 666 /var/run/docker.sock\n' > /etc/systemd/system/docker.service.d/socket.conf
+# Engine 29 keeps images in containerd's own store, which data-root does not cover. A
+# bind needs no containerd config to track.
+mountpoint -q /var/lib/containerd || mount --bind /scratch/containerd /var/lib/containerd \
+ || fatal "containerd store not on Local SSD"
+# Masked until here: an active docker.socket cannot spawn a masked service, so nothing
+# can start dockerd on the boot disk between the stop above and this restart. And
+# disabled from here on: only this script starts Docker, so a reboot no longer brings
+# it up on the boot disk before the script has run.
+systemctl unmask docker.socket docker containerd 2>/dev/null
+systemctl disable docker.socket docker containerd 2>/dev/null \
+ || logger -t devbox-startup "WARN: docker autostart still enabled; the mask covers the next boot"
+systemctl daemon-reload && systemctl restart containerd docker || fatal "docker would not start"
+docker info --format '{{.DockerRootDir}}' 2>/dev/null | grep -q '^/scratch/' \
+ || fatal "docker not on Local SSD" # daemon.json unwritten or rejected: dockerd starts fine on the boot disk
+
+# ---------- (b) accounts and cache paths ----------
+for u in runner1 runner2 runner3 runner4; do
+ # Home on scratch: whatever a job writes home-relative is disposable too.
+ id "$u" &>/dev/null || useradd -M -d "/scratch/$u/home" -s /usr/sbin/nologin "$u"
+ # runner/ up front: systemd applies WorkingDirectory before ExecStartPre could create it.
+ mkdir -p "/scratch/$u"/{cache,work,runner,home}
+ chown "$u:$u" "/scratch/$u" "/scratch/$u"/{cache,work,runner,home} # not -R: contents are the user's own
+done
+
+# Interactive accounts appear at first OS Login connect, so the profile provisions them:
+# each tool's default cache path becomes a symlink onto scratch. Symlinks, not variables,
+# so an agent's `bash -c` -- which sources no profile -- lands there too. PATH goes in
+# /etc/environment for the same reason: pam_env applies it to every session.
+echo 'PATH="/usr/local/go/bin:/usr/local/node/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"' > /etc/environment
+cat > /etc/profile.d/devbox.sh <<'EOF'
+u=$(id -un); c=/scratch/$u/cache
+if mountpoint -q /scratch && [ -n "$u" ] && [ -n "${HOME:-}" ] && mkdir -p "$c" 2>/dev/null; then
+ for pair in "$HOME/.cache:$c" "$HOME/go/pkg/mod:$c/go-mod" "$HOME/.npm:$c/npm" "$HOME/.local/share/pnpm:$c/pnpm"; do
+ link=${pair%%:*}; target=${pair#*:}
+ mkdir -p "$target" # every login: scratch targets vanish on a stop, the symlink in HOME does not
+ [ -L "$link" ] || { mkdir -p "$(dirname "$link")" && rm -rf "$link" && ln -s "$target" "$link"; }
+ done 2>/dev/null
+fi
+unset u c pair link target
+EOF
+# Existing accounts keep their symlinks across a stop but not the targets. Remake them,
+# so only a brand-new account needs a first interactive login.
+for h in /home/*; do
+ u=${h##*/}; id "$u" &>/dev/null || continue
+ for d in "" /go-mod /npm /pnpm; do install -d -o "$u" -g "$u" "/scratch/$u/cache$d"; done
+done
+
+# ---------- gh ----------
+# Upstream, not apt: Ubuntu 24.04 ships 2.45.0. /usr/local/bin is on the boot disk, so
+# this is a no-op on a reboot and only refetches after a rebuild or a new release.
+# Never fatal -- gh is a convenience, and a boot must not turn on GitHub being reachable.
+GH=$(curl -sf --retry 3 --max-time 30 https://api.github.com/repos/cli/cli/releases/latest \
+ | python3 -c 'import sys,json; print(json.load(sys.stdin)["tag_name"].lstrip("v"))' 2>/dev/null) \
+ || GH=$(gh --version 2>/dev/null | awk 'NR==1{print $3}')
+if [[ -n "${GH:-}" && "$(gh --version 2>/dev/null | awk 'NR==1{print $3}')" != "$GH" ]]; then
+ curl -fsSL --retry 3 "https://github.com/cli/cli/releases/download/v$GH/gh_${GH}_linux_amd64.tar.gz" \
+ | tar xz -C /tmp \
+ && install -m 0755 "/tmp/gh_${GH}_linux_amd64/bin/gh" /usr/local/bin/gh \
+ || logger -t devbox-startup "WARN: gh $GH install failed"
+ rm -rf "/tmp/gh_${GH}_linux_amd64"
+fi
+
+# ---------- (c) GitHub runners ----------
+# Stateless, so it lives on scratch and is installed from the unit rather than from
+# here: systemd retries that step forever, so a failed download heals itself.
+cat > /usr/local/sbin/install-runner <<'EOF'
+#!/bin/bash
+# <account>. Root, from the unit. Re-runs whole on every retry, deps included.
+set -euo pipefail
+DIR=/scratch/$1/runner
+# Latest release; if the lookup fails, keep what is installed, or fall back to a pin.
+V=$(curl -sf --retry 3 --max-time 30 https://api.github.com/repos/actions/runner/releases/latest \
+ | python3 -c 'import sys,json; print(json.load(sys.stdin)["tag_name"].lstrip("v"))' 2>/dev/null) \
+ || V=$(cat "$DIR/.installed" 2>/dev/null || echo 2.336.0)
+# Version-stamped and written last: a reboot keeps scratch, so a bare marker would pin
+# the old release, and an unstamped one would skip a failed dep step.
+[[ "$(cat "$DIR/.installed" 2>/dev/null)" == "$V" ]] && exit 0
+rm -rf "$DIR" && install -d -o "$1" -g "$1" "$DIR" # clean tree: an old release's files must not linger
+curl -fsSL --retry 3 "https://github.com/actions/runner/releases/download/v$V/actions-runner-linux-x64-$V.tar.gz" | tar xz -C "$DIR"
+chown -R "$1:$1" "$DIR"
+"$DIR"/bin/installdependencies.sh >/dev/null # apt is a no-op once satisfied
+echo "$V" > "$DIR/.installed"
+EOF
+
+cat > /usr/local/sbin/register-runner <<'EOF'
+#!/bin/bash
+# <account>. Runs as the account, every boot.
+set -euo pipefail
+ORG=bytebase
+md() { curl -sf -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/$1"; }
+PROJECT=$(md project/project-id)
+ACCESS=$(md instance/service-accounts/default/token | python3 -c 'import sys,json; print(json.load(sys.stdin)["access_token"])')
+PAT=$(curl -sf -H "Authorization: Bearer $ACCESS" \
+ "https://secretmanager.googleapis.com/v1/projects/${PROJECT}/secrets/gh-runner-pat/versions/latest:access" \
+ | python3 -c 'import sys,base64,json; print(base64.b64decode(json.load(sys.stdin)["payload"]["data"]).decode())')
+TOKEN=$(curl -sfX POST -H "Authorization: Bearer $PAT" -H "Accept: application/vnd.github+json" \
+ "https://api.github.com/orgs/${ORG}/actions/runners/registration-token" \
+ | python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
+cd "/scratch/$1/runner"
+rm -f .runner .credentials .credentials_rsakey # config.sh refuses to overwrite an existing config
+./config.sh --unattended --replace --url "https://github.com/${ORG}" --token "$TOKEN" \
+ --name "$(hostname -s)-$1" --work "/scratch/$1/work" # host-qualified name: never collides with another box
+EOF
+chmod +x /usr/local/sbin/install-runner /usr/local/sbin/register-runner
+
+cat > /etc/systemd/system/[email protected] <<'EOF'
+[Unit]
+Description=GitHub Actions runner (%i)
+Wants=network-online.target
+After=network-online.target docker.service
+[Service]
+User=%i
+WorkingDirectory=/scratch/%i/runner
+Environment=XDG_CACHE_HOME=/scratch/%i/cache GOMODCACHE=/scratch/%i/cache/go-mod npm_config_cache=/scratch/%i/cache/npm PNPM_CONFIG_STORE_DIR=/scratch/%i/cache/pnpm
+ExecStartPre=+/usr/local/sbin/install-runner %i
+ExecStartPre=/usr/local/sbin/register-runner %i
+ExecStart=/scratch/%i/runner/run.sh
+# install-runner fetches ~200 MB and may wait on the apt lock; the 90s default would
+# kill it, and the retry restarts the download from zero.
+TimeoutStartSec=600
+Restart=always
+# Registration failure retries forever; at 5s x3 runners that would burn the API quota.
+RestartSec=30
+EOF
+
+# ---------- (d) cache cleanup ----------
+cat > /usr/local/sbin/cache-gc <<'EOF'
+#!/bin/bash
+# Every 30 min. Leaked go-build sandboxes are swept always. Above 85% used, wait for an
+# idle box -- unless it is nearly full -- then delete everything disposable. It all
+# rebuilds on the next job; a full disk does not.
+set -uo pipefail
+mountpoint -q /scratch || exit 0 # cron outlives a stop; before (a) has mounted, there is nothing to clean
+HIGH=85 CRITICAL=95
+find /tmp -maxdepth 1 -name 'go-build*' -type d -mmin +1440 -exec rm -rf {} +
+used() { df -P /scratch | awk 'NR==2 {print $5+0}'; }
+# Load catches detached work -- tmux, nohup, a container -- that holds neither an SSH
+# session nor a Runner.Worker. On 20 vCPU, anything actually working exceeds 2.
+idle() { ! pgrep -f Runner.Worker >/dev/null && [[ -z "$(ss -Htn state established '( sport = :22 )')" ]] \
+ && (( $(cut -d. -f1 /proc/loadavg) < 2 )); }
+
+[[ $(used) -ge $HIGH ]] || exit 0
+if ! idle && [[ $(used) -lt $CRITICAL ]]; then logger -t cache-gc "deferred: busy at $(used)%"; exit 0; fi
+logger -t cache-gc "cleaning: $(used)% used"
+rm -rf /tmp/* /tmp/.[!.]* # /tmp is on scratch; the box is idle
+docker ps -q | xargs -r docker rm -f >/dev/null 2>&1 # idle, so a running container is orphaned
+docker system prune -af --volumes >/dev/null 2>&1
+# Everything on scratch is disposable except the runner installs, which are large and
+# would otherwise be re-downloaded on the next boot.
+rm -rf /scratch/*/cache /scratch/*/home /scratch/*/work
+for u in runner1 runner2 runner3 runner4; do
+ install -d -o "$u" -g "$u" /scratch/$u/{cache,home,work} # the units need them; a profile remakes an interactive account's
+done
+logger -t cache-gc "cleaned -> $(used)%"
+EOF
+chmod +x /usr/local/sbin/cache-gc
+# PATH first: cron's default omits /usr/sbin, and a missing `ss` would make idle() look true.
+printf 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n*/30 * * * * root /usr/local/sbin/cache-gc\n' > /etc/cron.d/devbox
+
+systemctl daemon-reload
+systemctl restart actions-runner@{runner1,runner2,runner3,runner4}
+
+# Last and time-bounded: observability must not hold up CI capacity.
+systemctl is-active --quiet google-cloud-ops-agent \
+ || { curl -sS --retry 3 --max-time 120 -o /tmp/ops.sh https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh \
+ && timeout 300 bash /tmp/ops.sh --also-install; } \
+ || logger -t devbox-startup "WARN: Ops Agent install failed; no memory or swap metrics"