Database governance built for humans and agents — controlling changes and access across every major database.
Go cicddata-governancedata-maskingdata-securitydatabase-accessdatabase-governancedevopsdevsecopsflywaygitopsliquibasemongodbmysqloraclepostgresqlschema-migrationssecuritysql-clientsqlservertidb
chore(devbox): run two CI runners instead of four (#21379)
Danny Xub35d323909cb
diff --git a/docs/design/devbox-and-ci-runner.md b/docs/design/devbox-and-ci-runner.mdindex 42de7ec7ed..d233bd41e5 100644--- a/docs/design/devbox-and-ci-runner.md+++ b/docs/design/devbox-and-ci-runner.md@@ -1,7 +1,7 @@# Combined Dev Box and CI Runner on GCPOne Spot VM with two jobs. It is the SSH target that agentic coding tools drive, and-it hosts four self-hosted GitHub Actions runners for the whole org.+it hosts two self-hosted GitHub Actions runners for the whole org.## Goals@@ -11,7 +11,7 @@ it hosts four self-hosted GitHub Actions runners for the whole org.2. **One startup script, so the box is close to stateless.** Everything on the box isproduced by that script. Only `/home` and a shared toolchain in `/usr/local` areyours to manage; nothing else is set up by hand.-3. **GitHub Actions runners that set themselves up.** Four org-level slots. Each+3. **GitHub Actions runners that set themselves up.** Two org-level slots. Eachre-registers at boot from a secret, so there is no registration state to preserve.4. **A clear split between Local SSD and the persistent disk.** Local SSD holds whatcan be discarded: caches, Docker, temp files, swap. The boot disk holds repos and@@ -164,7 +164,7 @@ rebuilt. Anything on the boot disk is guarded, so it is not redone.Two kinds of account use the box, and they need opposite things from it.-**Runner accounts** are `runner1` through `runner4`. The script creates them,+**Runner accounts** are `runner1` and `runner2`. The script creates them,`nologin`, one per job slot. Nothing of theirs is worth keeping: software, work treesand caches all sit on Local SSD, and their home directories are on it too. They aredisposable, and are disposed of on every stop.@@ -258,13 +258,20 @@ and `node_modules` — which is right for anything inside a working copy.### c. GitHub runners-Four accounts, four job slots, one systemd template. A pull request touching both+Two accounts, two job slots, one systemd template. A pull request touching bothbackend and frontend schedules five self-hosted jobs -- one each from `backend-tests`,-`golangci-lint` and `test_link`, two from `frontend-tests` -- so one of them queues,+`golangci-lint` and `test_link`, two from `frontend-tests` -- so three of them queue,before counting any other repository in the org. That is deliberate: jobs queue, they-do not fail. Four ways is five cores and 8 GB a slot, which is what a comparable box-ran two slots on; watch `memory/percent_used` before adding a fifth, which is one more-name in the script's account list.+do not fail. It began as four slots, five cores and 8 GB each, and that oversubscribed+the box: a Go build or a frontend build uses every core it can find, and four at once+pushed the load average past 60 on 20 vCPUs; the one runner hang seen so far happened+under that load. Two ways is ten cores and 16 GB a slot. Watch the load average and+`memory/percent_used` before adding a third, which is one more name in the script's+account list.++Removing one is the reverse, with one care: `systemctl stop` cancels a running job.+Park the runner in a runner group no repository can use, wait for its job to finish,+then stop it, delete its registration, and delete its account.The runner is stateless, so it lives on scratch and is installed by the unit ratherthan by the startup script: an `ExecStartPre` runs `install-runner` as root, before@@ -281,7 +288,7 @@ server-side entry of the same name. Names are host-qualified, so two boxes nevereach other's.Re-registering rather than preserving credentials means the runner holds no identity-on disk. Wipe the boot disk and the same service account brings back the same four+on disk. Wipe the boot disk and the same service account brings back the same tworunners. Registration is derived state, like `/scratch`.The units carry no `[Install]` section. The script starts them rather than enablingdiff --git a/scripts/devbox/startup.sh b/scripts/devbox/startup.shindex fc0736b211..e0db7db714 100755--- a/scripts/devbox/startup.sh+++ b/scripts/devbox/startup.sh@@ -85,7 +85,7 @@ docker info --format '{{.DockerRootDir}}' 2>/dev/null | grep -q '^/scratch/' \|| fatal "docker not on Local SSD" # daemon.json unwritten or rejected: dockerd starts fine on the boot disk# ---------- (b) accounts and cache paths -----------for u in runner1 runner2 runner3 runner4; do+for u in runner1 runner2; do# Home on scratch: whatever a job writes home-relative is disposable too.id "$u" &>/dev/null || useradd -M -d "/scratch/$u/home" -s /usr/sbin/nologin "$u"# runner/ up front: systemd applies WorkingDirectory before ExecStartPre could create it.@@ -224,7 +224,7 @@ docker system prune -af --volumes >/dev/null 2>&1# Everything on scratch is disposable except the runner installs, which are large and# would otherwise be re-downloaded on the next boot.rm -rf /scratch/*/cache /scratch/*/home /scratch/*/work-for u in runner1 runner2 runner3 runner4; do+for u in runner1 runner2; doinstall -d -o "$u" -g "$u" /scratch/$u/{cache,home,work} # the units need them; a profile remakes an interactive account'sdonelogger -t cache-gc "cleaned -> $(used)%"@@ -234,7 +234,7 @@ chmod +x /usr/local/sbin/cache-gcprintf 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n*/30 * * * * root /usr/local/sbin/cache-gc\n' > /etc/cron.d/devboxsystemctl daemon-reload-systemctl restart actions-runner@{runner1,runner2,runner3,runner4}+systemctl restart actions-runner@{runner1,runner2}# Last and time-bounded: observability must not hold up CI capacity.systemctl is-active --quiet google-cloud-ops-agent \