Database governance built for humans and agents — controlling changes and access across every major database.
Go cicddata-governancedata-maskingdata-securitydatabase-accessdatabase-governancedevopsdevsecopsflywaygitopsliquibasemongodbmysqloraclepostgresqlschema-migrationssecuritysql-clientsqlservertidb
chore(infra): restart the devbox from Cloud Scheduler, not a GitHub cron (#21422)
Danny Xude767f5dca4f
diff --git a/.github/workflows/start-runner-vm.yml b/.github/workflows/start-runner-vm.ymldeleted file mode 100644index 22edc07d37..0000000000--- a/.github/workflows/start-runner-vm.yml+++ /dev/null@@ -1,37 +0,0 @@-# Starts the devbox GCE VM, which hosts the org's self-hosted Actions runners-# (used by jobs with `runs-on: self-hosted`, e.g. backend-tests, frontend-tests,-# golangci-lint, test_link). It is a Spot instance, so a preemption stops it and-# this workflow is what brings it back. Design: docs/design/devbox-and-ci-runner.md-name: Start devbox VM--on:- workflow_dispatch:- schedule:- # Every 15 minutes: that is the ceiling on how long a preemption leaves CI- # without runners. Starting an already-running instance is a no-op.- - cron: "*/15 * * * *"--permissions:- contents: read--jobs:- start-vm:- runs-on: ubuntu-24.04- steps:- - name: Authenticate to Google Cloud- uses: google-github-actions/auth@v3- with:- credentials_json: ${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}-- - name: Set up Cloud SDK- uses: google-github-actions/setup-gcloud@v3-- - name: Start devbox VM if stopped- run: |- STATUS=$(gcloud compute instances describe devbox \- --zone northamerica-northeast2-a --project bytebase-dev \- --format='value(status)')- if [ "$STATUS" != "RUNNING" ]; then- gcloud compute instances start devbox \- --zone northamerica-northeast2-a --project bytebase-dev- fidiff --git a/docs/design/devbox-and-ci-runner.md b/docs/design/devbox-and-ci-runner.mdindex d233bd41e5..b121e77356 100644--- a/docs/design/devbox-and-ci-runner.md+++ b/docs/design/devbox-and-ci-runner.md@@ -40,7 +40,7 @@ it hosts two self-hosted GitHub Actions runners for the whole org.| External IP | reserved; carries all egress, and inbound SSH || Inbound | SSH only, via the VPC default rule; no tag, no custom rules || Service account | dedicated; reads one secret, writes metrics and logs, nothing else |-| Availability | always on; preemption stops it, the start workflow restarts it within its polling interval |+| Availability | always on; preemption stops it, and a Cloud Scheduler job starts it again within two minutes, capacity permitting || Component | Rate | $/mo ||---|---|---|@@ -96,8 +96,8 @@ on the secret, which Editor cannot do. It stops at the first failed step.It enables the APIs, creates the service account, stores the PAT it prompts for inSecret Manager, grants that account the secret plus `monitoring.metricWriter` and-`logging.logWriter`, reserves the address, and creates the instance with-`startup.sh` as its startup-script metadata.+`logging.logWriter`, reserves the address, creates the instance with `startup.sh` as+its startup-script metadata, and schedules its restart.The `cloud-platform` scope resolves to the intersection with the account's roles: onesecret, metrics, logs. The scope is not the boundary. The IAM bindings are.@@ -106,15 +106,11 @@ There is no flag day. Runners register at the organization under the automatic`self-hosted` label, with host-qualified names. So this box only adds capacityalongside any existing runner, and you can delete the old one whenever.-One step has to happen in order, and it is the last. A preemption stops the instance,-and `.github/workflows/start-runner-vm.yml` restarts it on a schedule. That workflow-now names this instance and zone, in two places each. Repointing it any earlier would-have stopped it restarting the box that was serving CI at the time.--Its cadence changed with it, from three fixed hours to every fifteen minutes. At three-a day, a preemption just after one of them left the box stopped for up to fifteen-hours, which is not "always on" for either the dev box or CI. Fifteen minutes bounds-the outage at fifteen minutes, and starting a running instance is a no-op.+A preemption stops the instance, and the Cloud Scheduler job `devbox-start` starts it+again: every minute it calls the Compute API's start, a no-op while the box runs, as+the Compute Engine default service account. It lives in `northamerica-northeast1`, as+Cloud Scheduler has no Toronto region. Pause it before stopping the box by hand, and+resume it after.### Connectdiff --git a/scripts/devbox/create.sh b/scripts/devbox/create.shindex e3ca31e6c8..1cf9f95488 100755--- a/scripts/devbox/create.sh+++ b/scripts/devbox/create.sh@@ -16,7 +16,7 @@ SA=devbox@${PROJECT}.iam.gserviceaccount.comHERE=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)gcloud services enable secretmanager.googleapis.com oslogin.googleapis.com \- monitoring.googleapis.com logging.googleapis.com --project=$PROJECT+ monitoring.googleapis.com logging.googleapis.com cloudscheduler.googleapis.com --project=$PROJECT# Service account; its roles are granted below, nothing more.gcloud iam service-accounts create devbox --project=$PROJECT \@@ -54,3 +54,11 @@ gcloud compute instances create devbox --project=$PROJECT --zone=$ZONE \--service-account=$SA --scopes=cloud-platform \--metadata=enable-oslogin=TRUE \--metadata-from-file=startup-script="$HERE/startup.sh"++# Restarts the instance after a preemption; a no-op while it runs. It relies on the+# Compute Engine default account's Editor role. Montréal: no Cloud Scheduler in Toronto.+PN=$(gcloud projects describe $PROJECT --format='value(projectNumber)')+gcloud scheduler jobs create http devbox-start --project=$PROJECT \+ --location=northamerica-northeast1 --schedule='* * * * *' --http-method=POST \+ --uri="https://compute.googleapis.com/compute/v1/projects/$PROJECT/zones/$ZONE/instances/devbox/start" \+ --oauth-service-account-email="${PN}[email protected]"diff --git a/scripts/devbox/startup.sh b/scripts/devbox/startup.shindex e0db7db714..dd0e30145f 100755--- a/scripts/devbox/startup.sh+++ b/scripts/devbox/startup.sh@@ -3,9 +3,9 @@# not abandon the boot; `journalctl -t devbox-startup` has the exit status.set -uo pipefailtrap 'logger -t devbox-startup "exited: status $?"' EXIT-# A fatal exit powers the box off. Left RUNNING with no runners, the start workflow-# would never touch it; off, it is restarted and this script retried on the-# workflow's next tick. Five minutes keeps SSH usable for the journal or /home first.+# A fatal exit powers the box off. Left RUNNING with no runners, the devbox-start job+# would never touch it; off, the job starts it within a minute and this script is+# retried. Five minutes keeps SSH usable for the journal or /home first.fatal() {logger -t devbox-startup "FATAL: $1"; systemctl stop docker.socket docker containerd 2>/dev/nullshutdown -h +5 "devbox-startup: $1"; exit 1