Skip to content

Database governance built for humans and agents — controlling changes and access across every major database.

chore(devbox): run two CI runners instead of four (#21379)

Danny Xub35d323909cb
diff --git a/docs/design/devbox-and-ci-runner.md b/docs/design/devbox-and-ci-runner.md
index 42de7ec7ed..d233bd41e5 100644
--- a/docs/design/devbox-and-ci-runner.md
+++ b/docs/design/devbox-and-ci-runner.md
@@ -1,7 +1,7 @@
# Combined Dev Box and CI Runner on GCP
One Spot VM with two jobs. It is the SSH target that agentic coding tools drive, and
-it hosts four self-hosted GitHub Actions runners for the whole org.
+it hosts two self-hosted GitHub Actions runners for the whole org.
## Goals
@@ -11,7 +11,7 @@ it hosts four self-hosted GitHub Actions runners for the whole org.
2. **One startup script, so the box is close to stateless.** Everything on the box is
produced by that script. Only `/home` and a shared toolchain in `/usr/local` are
yours to manage; nothing else is set up by hand.
-3. **GitHub Actions runners that set themselves up.** Four org-level slots. Each
+3. **GitHub Actions runners that set themselves up.** Two org-level slots. Each
re-registers at boot from a secret, so there is no registration state to preserve.
4. **A clear split between Local SSD and the persistent disk.** Local SSD holds what
can be discarded: caches, Docker, temp files, swap. The boot disk holds repos and
@@ -164,7 +164,7 @@ rebuilt. Anything on the boot disk is guarded, so it is not redone.
Two kinds of account use the box, and they need opposite things from it.
-**Runner accounts** are `runner1` through `runner4`. The script creates them,
+**Runner accounts** are `runner1` and `runner2`. The script creates them,
`nologin`, one per job slot. Nothing of theirs is worth keeping: software, work trees
and caches all sit on Local SSD, and their home directories are on it too. They are
disposable, and are disposed of on every stop.
@@ -258,13 +258,20 @@ and `node_modules` — which is right for anything inside a working copy.
### c. GitHub runners
-Four accounts, four job slots, one systemd template. A pull request touching both
+Two accounts, two job slots, one systemd template. A pull request touching both
backend and frontend schedules five self-hosted jobs -- one each from `backend-tests`,
-`golangci-lint` and `test_link`, two from `frontend-tests` -- so one of them queues,
+`golangci-lint` and `test_link`, two from `frontend-tests` -- so three of them queue,
before counting any other repository in the org. That is deliberate: jobs queue, they
-do not fail. Four ways is five cores and 8 GB a slot, which is what a comparable box
-ran two slots on; watch `memory/percent_used` before adding a fifth, which is one more
-name in the script's account list.
+do not fail. It began as four slots, five cores and 8 GB each, and that oversubscribed
+the box: a Go build or a frontend build uses every core it can find, and four at once
+pushed the load average past 60 on 20 vCPUs; the one runner hang seen so far happened
+under that load. Two ways is ten cores and 16 GB a slot. Watch the load average and
+`memory/percent_used` before adding a third, which is one more name in the script's
+account list.
+
+Removing one is the reverse, with one care: `systemctl stop` cancels a running job.
+Park the runner in a runner group no repository can use, wait for its job to finish,
+then stop it, delete its registration, and delete its account.
The runner is stateless, so it lives on scratch and is installed by the unit rather
than by the startup script: an `ExecStartPre` runs `install-runner` as root, before
@@ -281,7 +288,7 @@ server-side entry of the same name. Names are host-qualified, so two boxes never
each other's.
Re-registering rather than preserving credentials means the runner holds no identity
-on disk. Wipe the boot disk and the same service account brings back the same four
+on disk. Wipe the boot disk and the same service account brings back the same two
runners. Registration is derived state, like `/scratch`.
The units carry no `[Install]` section. The script starts them rather than enabling
diff --git a/scripts/devbox/startup.sh b/scripts/devbox/startup.sh
index fc0736b211..e0db7db714 100755
--- a/scripts/devbox/startup.sh
+++ b/scripts/devbox/startup.sh
@@ -85,7 +85,7 @@ docker info --format '{{.DockerRootDir}}' 2>/dev/null | grep -q '^/scratch/' \
|| fatal "docker not on Local SSD" # daemon.json unwritten or rejected: dockerd starts fine on the boot disk
# ---------- (b) accounts and cache paths ----------
-for u in runner1 runner2 runner3 runner4; do
+for u in runner1 runner2; do
# Home on scratch: whatever a job writes home-relative is disposable too.
id "$u" &>/dev/null || useradd -M -d "/scratch/$u/home" -s /usr/sbin/nologin "$u"
# runner/ up front: systemd applies WorkingDirectory before ExecStartPre could create it.
@@ -224,7 +224,7 @@ docker system prune -af --volumes >/dev/null 2>&1
# Everything on scratch is disposable except the runner installs, which are large and
# would otherwise be re-downloaded on the next boot.
rm -rf /scratch/*/cache /scratch/*/home /scratch/*/work
-for u in runner1 runner2 runner3 runner4; do
+for u in runner1 runner2; do
install -d -o "$u" -g "$u" /scratch/$u/{cache,home,work} # the units need them; a profile remakes an interactive account's
done
logger -t cache-gc "cleaned -> $(used)%"
@@ -234,7 +234,7 @@ chmod +x /usr/local/sbin/cache-gc
printf 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n*/30 * * * * root /usr/local/sbin/cache-gc\n' > /etc/cron.d/devbox
systemctl daemon-reload
-systemctl restart actions-runner@{runner1,runner2,runner3,runner4}
+systemctl restart actions-runner@{runner1,runner2}
# Last and time-bounded: observability must not hold up CI capacity.
systemctl is-active --quiet google-cloud-ops-agent \