Skip to content

Self-improving AI Employees for everyone

docs: add GCP VM deploy runbook (manual gcloud + Cloudflare path)

trohitge367776d0940
diff --git a/docs-internal/gcp_vm_deploy_runbook.md b/docs-internal/gcp_vm_deploy_runbook.md
new file mode 100644
index 00000000..9cdb83ee
--- /dev/null
+++ b/docs-internal/gcp_vm_deploy_runbook.md
@@ -0,0 +1,362 @@
+# GCP VM Deploy Runbook (released npm package, manual gcloud + cf)
+
+Step-by-step runbook for deploying the **released** `machinaos` npm package on a fresh
+GCP VM behind a Cloudflare-proxied domain, with the single-owner login gate enabled.
+Written to be executable by an AI agent (or a human) with no other context.
+
+This is the manual path — it does NOT use `machina deploy` / Terraform
+(`cli/commands/deploy/`, `cli/terraform/`). It was derived from a real deployment to
+`demo.zeenie.xyz` (June 2026, [email protected]) and encodes every pitfall hit on the way.
+
+## Parameters
+
+Decide these up front; they appear in commands below as `<PLACEHOLDERS>`.
+
+| Placeholder | Example | Notes |
+|---|---|---|
+| `<PROJECT>` | `zeenie` | GCP project id (`gcloud config get-value project`) |
+| `<VM_NAME>` | `demo` | Instance name |
+| `<REGION>` / `<ZONE>` | `asia-south1` / `asia-south1-a` | Mumbai. Any region works |
+| `<IP_NAME>` | `machinaos-india-ip` | Name for the reserved static IP |
+| `<DOMAIN>` | `demo.zeenie.xyz` | Must be a subdomain of a zone in the Cloudflare account |
+| `<SUBDOMAIN>` | `demo` | The record name within the zone |
+| `<OWNER_EMAIL>` | `[email protected]` | Login email for the owner account |
+| `<OWNER_PASSWORD>` | (generated) | >= 8 chars; generate 20-char alphanumeric if not supplied |
+| `<JWT_KEY>` `<SECRET_KEY>` `<ENC_KEY>` | (generated) | Three independent 48-hex secrets |
+
+Generate secrets (run locally):
+
+```bash
+python -c "import secrets,string; a=string.ascii_letters+string.digits; \
+print('PW='+''.join(secrets.choice(a) for _ in range(20))); \
+print('JWT='+secrets.token_hex(24)); print('SEC='+secrets.token_hex(24)); \
+print('ENC='+secrets.token_hex(24))"
+```
+
+## Prerequisites (verify before starting)
+
+1. `gcloud` installed and authenticated: `gcloud config get-value project` returns the
+ project and `gcloud config get-value account` returns an account. On Windows, ignore
+ the harmless `Test-Path ... bundledpython ... is denied` stderr noise — output after
+ it is still valid.
+2. `cf` (Cloudflare CLI, npm package) installed and authenticated:
+ `cf auth whoami` shows `"authenticated": true` with `dns_records:edit` scope.
+3. No Terraform, no ADC needed — plain gcloud credentials suffice.
+
+## Known pitfalls (read first — these each cost a redeploy or a debugging loop)
+
+1. **Debian 12 fails.** The npm package's `postinstall` hard-requires Python 3.12+;
+ Debian 12 ships 3.11 and `npm install -g machinaos` exits 1
+ (`ERROR: Python 3.12+ is required.`). Use **Ubuntu 24.04** (`ubuntu-2404-lts-amd64`
+ in `ubuntu-os-cloud`), which ships Python 3.12.
+2. **The released package has no `machina serve`.** Released CLI commands are only
+ `start | dev | stop | build | clean | doctor | help | version`. The single-port
+ `serve` command and `SERVE_STATIC_CLIENT` backend support exist only in unreleased
+ source. Production shape for the release is the README's `machina start`
+ (static client on **:3000**, backend uvicorn on **:3010**) plus an nginx reverse
+ proxy on :80/:443.
+3. **`MACHINA_OWNER_*` env seeding is NOT honored by the release.** Setting
+ `MACHINA_OWNER_EMAIL/PASSWORD` does nothing in 0.0.88. Instead: with
+ `AUTH_MODE=single`, registration is open until the first user registers and that
+ user becomes owner. Register the owner via `POST /api/auth/register` immediately
+ after first boot (step 7). Registration auto-closes afterwards.
+4. **Unknown env vars are fine, but don't set `PORT=80`.** The released backend does
+ not read `PORT`; nginx owns :80. Keep the env file to the keys listed in step 2.
+5. **Cloudflare proxied + "Full" SSL mode → 521 without origin TLS.** If the zone
+ forces HTTPS and SSL mode is Full, Cloudflare connects to origin :443. Fix: nginx
+ listens on 443 with a **self-signed** cert (Full mode accepts it; only
+ Full-strict would reject) and the GCP firewall must allow **tcp:443** too.
+6. **Startup script must have LF line endings.** When authoring on Windows, strip CRLF
+ before passing to `--metadata-from-file` or bash on the VM fails obscurely.
+7. **GCP startup scripts re-run on every boot.** Keep the script idempotent (it is,
+ below). A stop/start re-provisions harmlessly.
+8. **Renaming a VM requires stopping it** (`gcloud compute instances set-name`), and an
+ *ephemeral* IP would be released on recreate. Reserve the static IP first (step 3)
+ so the DNS record never goes stale.
+9. **PowerShell quoting eats `|` and `\"` in long ssh `--command` strings.** Write a
+ local `.sh` file, `gcloud compute scp` it, then `ssh --command="sudo bash /tmp/x.sh"`.
+10. **WebSocket 403 without a cookie is correct behavior**, not a proxy bug. The
+ backend rejects unauthenticated `/ws/status` upgrades; with a valid login cookie
+ the same handshake returns `101 Switching Protocols`.
+
+## Step 1 — Reserve a static IP
+
+```bash
+gcloud compute addresses create <IP_NAME> --region=<REGION> --project=<PROJECT>
+gcloud compute addresses describe <IP_NAME> --region=<REGION> --project=<PROJECT> --format="value(address)"
+```
+
+Save the returned address as `<STATIC_IP>`.
+
+(If the VM already exists with an ephemeral IP, promote it instead:
+`gcloud compute addresses create <IP_NAME> --region=<REGION> --addresses=<CURRENT_IP>`.)
+
+## Step 2 — Author the startup script (locally)
+
+Write `machinaos-startup.sh` with the placeholders filled in. This is the complete,
+corrected, idempotent script (toolchain + release install + env + nginx with TLS +
+systemd):
+
+```bash
+#!/bin/bash
+# MachinaOs VM startup script: released npm package via `machina start`,
+# nginx on :80/:443 (self-signed TLS for Cloudflare Full mode), login gate on.
+set -euo pipefail
+export DEBIAN_FRONTEND=noninteractive
+
+echo "[machinaos] installing toolchain..."
+apt-get update
+apt-get install -y curl ca-certificates git build-essential pkg-config libffi-dev libssl-dev nginx
+curl -fsSL https://deb.nodesource.com/setup_22.x | bash -
+apt-get install -y nodejs
+corepack enable || true
+curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR=/usr/local/bin sh
+
+echo "[machinaos] installing MachinaOs released package..."
+npm install -g machinaos@latest
+
+echo "[machinaos] writing login-gate env..."
+mkdir -p /etc/machinaos
+cat > /etc/machinaos/machina.env <<'MACHINA_ENV_EOF'
+HOST=0.0.0.0
+DATA_DIR=/var/lib/machinaos
+WORKSPACE_BASE_DIR=workspaces
+VITE_AUTH_ENABLED=true
+AUTH_MODE=single
+JWT_COOKIE_SECURE=true
+JWT_COOKIE_SAMESITE=lax
+TEMPORAL_ENABLED=false
+REDIS_ENABLED=false
+LOG_FORMAT=text
+JWT_SECRET_KEY=<JWT_KEY>
+SECRET_KEY=<SECRET_KEY>
+API_KEY_ENCRYPTION_KEY=<ENC_KEY>
+MACHINA_ENV_EOF
+chmod 600 /etc/machinaos/machina.env
+mkdir -p /var/lib/machinaos
+
+echo "[machinaos] self-signed TLS cert (Cloudflare Full mode accepts it)..."
+mkdir -p /etc/nginx/certs
+if [ ! -f /etc/nginx/certs/origin.crt ]; then
+ openssl req -x509 -nodes -newkey rsa:2048 -days 3650 \
+ -keyout /etc/nginx/certs/origin.key -out /etc/nginx/certs/origin.crt \
+ -subj "/CN=<DOMAIN>"
+fi
+
+echo "[machinaos] nginx reverse proxy (SPA :3000, backend :3010)..."
+cat > /etc/nginx/sites-available/machinaos <<'NGINX_EOF'
+server {
+ listen 80 default_server;
+ listen 443 ssl default_server;
+ ssl_certificate /etc/nginx/certs/origin.crt;
+ ssl_certificate_key /etc/nginx/certs/origin.key;
+ server_name <DOMAIN> _;
+ client_max_body_size 50m;
+
+ location /api/ {
+ proxy_pass http://127.0.0.1:3010;
+ proxy_set_header Host $host;
+ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+ proxy_set_header X-Forwarded-Proto $scheme;
+ proxy_read_timeout 300s;
+ }
+ location /ws/ {
+ proxy_pass http://127.0.0.1:3010;
+ proxy_http_version 1.1;
+ proxy_set_header Upgrade $http_upgrade;
+ proxy_set_header Connection "upgrade";
+ proxy_set_header Host $host;
+ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+ proxy_read_timeout 3600s;
+ proxy_send_timeout 3600s;
+ }
+ location /webhook/ {
+ proxy_pass http://127.0.0.1:3010;
+ proxy_set_header Host $host;
+ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+ }
+ location = /health {
+ proxy_pass http://127.0.0.1:3010;
+ }
+ location / {
+ proxy_pass http://127.0.0.1:3000;
+ proxy_set_header Host $host;
+ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+ }
+}
+NGINX_EOF
+rm -f /etc/nginx/sites-enabled/default
+ln -sf /etc/nginx/sites-available/machinaos /etc/nginx/sites-enabled/machinaos
+nginx -t
+systemctl enable --now nginx
+systemctl reload nginx
+
+echo "[machinaos] installing systemd service (README usage: machina start)..."
+MACHINA_BIN=$(command -v machina)
+cat > /etc/systemd/system/machinaos.service <<SERVICE_EOF
+[Unit]
+Description=MachinaOs (released, machina start)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+EnvironmentFile=/etc/machinaos/machina.env
+WorkingDirectory=/usr/lib/node_modules/machinaos
+ExecStart=$MACHINA_BIN start
+Restart=on-failure
+RestartSec=5
+TimeoutStopSec=30
+
+[Install]
+WantedBy=multi-user.target
+SERVICE_EOF
+
+systemctl daemon-reload
+systemctl enable --now machinaos
+systemctl restart machinaos
+echo "[machinaos] done."
+```
+
+On Windows, force LF endings after writing:
+
+```powershell
+$p = "$env:TEMP\machinaos-startup.sh"
+[IO.File]::WriteAllText($p, [IO.File]::ReadAllText($p).Replace("`r`n","`n"))
+```
+
+## Step 3 — Firewall + VM
+
+```bash
+gcloud compute firewall-rules create machinaos-allow-http \
+ --project=<PROJECT> --direction=INGRESS --action=ALLOW \
+ --rules=tcp:80,tcp:443 --source-ranges=0.0.0.0/0 --target-tags=machinaos
+```
+
+```bash
+gcloud compute instances create <VM_NAME> \
+ --project=<PROJECT> --zone=<ZONE> \
+ --machine-type=e2-standard-2 \
+ --image-family=ubuntu-2404-lts-amd64 --image-project=ubuntu-os-cloud \
+ --boot-disk-size=30GB --boot-disk-type=pd-balanced \
+ --tags=machinaos --address=<STATIC_IP> \
+ --metadata-from-file=startup-script=machinaos-startup.sh
+```
+
+Notes: `e2-standard-2` (2 vCPU / 8 GB) is comfortable; the old `e2-micro` is too small.
+Updating an existing firewall rule uses `--rules=` (not `--allow=`):
+`gcloud compute firewall-rules update machinaos-allow-http --rules="tcp:80,tcp:443"`.
+
+## Step 4 — DNS via cf CLI
+
+The `cf` CLI needs an account + zone context once, and `records create` takes a raw
+JSON `--body` (individual flags like `--type/--name` are NOT supported):
+
+```bash
+cf zones list # find the zone; grab account id from the output
+cf context set account-id <ACCOUNT_ID>
+cf context set zone <ZONE_DOMAIN> # e.g. zeenie.xyz
+cf dns records list # check the name is free
+cf dns records create --body '{"type":"A","name":"<SUBDOMAIN>","content":"<STATIC_IP>","proxied":true,"ttl":1,"comment":"MachinaOs VM <ZONE> (gcloud, machinaos@latest)"}'
+```
+
+Delete a record by id: `cf dns records delete <RECORD_ID>`.
+
+`proxied: true` matches the zone's other app records; Cloudflare terminates public TLS
+and (in Full SSL mode) connects to the origin's self-signed :443.
+
+## Step 5 — Wait for provisioning
+
+Provisioning takes ~5-8 minutes (apt + Node 22 + npm install with Python dep sync).
+Poll until healthy:
+
+```bash
+until curl -sf -m 5 http://<STATIC_IP>/health; do sleep 10; done
+```
+
+If it never comes up, debug in this order:
+
+```bash
+# startup script progress / errors:
+gcloud compute instances get-serial-port-output <VM_NAME> --zone=<ZONE> | grep -E "machinaos|error"
+# service state:
+gcloud compute ssh <VM_NAME> --zone=<ZONE> --quiet --command="systemctl is-active machinaos nginx; sudo journalctl -u machinaos -n 40 --no-pager"
+```
+
+## Step 6 — Register the owner (this IS the login setup)
+
+Because the release ignores `MACHINA_OWNER_*` env (pitfall 3), create the owner via the
+API the moment health is green. First user in `single` mode becomes owner and closes
+registration:
+
+```bash
+curl -s -X POST http://<STATIC_IP>/api/auth/register \
+ -H "Content-Type: application/json" \
+ -d '{"email":"<OWNER_EMAIL>","password":"<OWNER_PASSWORD>","display_name":"Owner"}'
+# expect: {"success":true,"user":{...,"is_owner":true}}
+```
+
+Do this promptly — until it runs, anyone reaching the IP could register as owner.
+(Optionally create the VM with `--allow-cidr` style source-range restriction on the
+firewall first, then widen after registering.)
+
+## Step 7 — Verify end-to-end
+
+All of these must pass before declaring success:
+
+```bash
+# 1. health via domain (Cloudflare -> origin TLS):
+curl -s https://<DOMAIN>/health # 200, {"status":"healthy",...}
+# 2. SPA serves:
+curl -s https://<DOMAIN>/ | grep -o "<title>[^<]*</title>" # <title>MachinaOS</title>
+# 3. auth gate on, registration closed:
+curl -s https://<DOMAIN>/api/auth/status
+# expect: {"auth_enabled":true,"auth_mode":"single","can_register":false,...}
+# 4. login works (200 + HttpOnly machina_token cookie):
+curl -s -i -X POST https://<DOMAIN>/api/auth/login \
+ -H "Content-Type: application/json" \
+ -d '{"email":"<OWNER_EMAIL>","password":"<OWNER_PASSWORD>"}' | head -12
+# 5. WebSocket: 403 without cookie (correct), 101 with it:
+cookie=$(curl -s -i -X POST https://<DOMAIN>/api/auth/login -H "Content-Type: application/json" \
+ -d '{"email":"<OWNER_EMAIL>","password":"<OWNER_PASSWORD>"}' \
+ | grep -i '^set-cookie' | sed 's/^[Ss]et-[Cc]ookie: //' | cut -d';' -f1)
+curl -s -i -N -m 8 -H "Connection: Upgrade" -H "Upgrade: websocket" \
+ -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" -H "Sec-WebSocket-Version: 13" \
+ -H "Cookie: $cookie" https://<DOMAIN>/ws/status | head -3 # HTTP/1.1 101
+```
+
+If `https://<DOMAIN>` returns **521**: origin :443 is unreachable — confirm the
+firewall allows tcp:443 and nginx is listening (`sudo ss -tlnp | grep :443`).
+If **526**: the zone is Full (strict); replace the self-signed cert with a Cloudflare
+Origin CA cert.
+
+## Step 8 — Hygiene
+
+- Delete any local temp scripts containing the secrets after the deploy.
+- Report the URL, owner email, and password to the operator once; the password is not
+ recoverable from the VM (only its bcrypt hash is stored).
+
+## Teardown
+
+```bash
+gcloud compute instances delete <VM_NAME> --zone=<ZONE> --project=<PROJECT> --quiet
+gcloud compute addresses delete <IP_NAME> --region=<REGION> --project=<PROJECT> --quiet
+gcloud compute firewall-rules delete machinaos-allow-http --project=<PROJECT> --quiet
+cf dns records list # find the record id
+cf dns records delete <RECORD_ID>
+```
+
+## Operations cheat sheet
+
+```bash
+# logs:
+gcloud compute ssh <VM_NAME> --zone=<ZONE> --quiet --command="sudo journalctl -u machinaos -f"
+# restart app:
+gcloud compute ssh <VM_NAME> --zone=<ZONE> --quiet --command="sudo systemctl restart machinaos"
+# upgrade to a new release:
+gcloud compute ssh <VM_NAME> --zone=<ZONE> --quiet --command="sudo npm install -g machinaos@latest && sudo systemctl restart machinaos"
+# env (secrets live here):
+# /etc/machinaos/machina.env (chmod 600; service reads it via EnvironmentFile)
+# data:
+# /var/lib/machinaos (DATA_DIR: DBs, workspaces, packages)
+```