Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
three.ws production on Google Cloud — operations runbook
This is the complete operational record of the production platform. If the
machine that performed the migration is gone, this document + gcloud access
to the project is everything needed to operate, deploy, debug, and recover the
site. Written 2026-07-07, the day production moved off Vercel (Vercel disabled
the deployment with 402 DEPLOYMENT_DISABLED; the site was dark until DNS
cutover to Google Cloud the same day).
Related: server/README.md (the server code itself), STRUCTURE.md (surface map), docs/ops/gcp-model-workers.md (the 3D model workers, which were already on Cloud Run before this migration).
Architecture
Namecheap DNS (three.ws)
A @ → 136.68.246.178 (global static IP: compute address "three-ws-ip")
A www → 136.68.246.178
│
Global External Application Load Balancer (EXTERNAL_MANAGED)
:80 forwarding rule "three-ws-http" → target-http-proxy "three-ws-http-proxy"
→ url-map "three-ws-http-redirect" (301 → https)
:443 forwarding rule "three-ws-https" → target-https-proxy "three-ws-https-proxy"
→ ssl-certificate "three-ws-cert" (Google-managed, three.ws + www.three.ws,
auto-renews) → url-map "three-ws-lb"
→ backend-service "three-ws-backend" (Cloud CDN ON, cache mode
USE_ORIGIN_HEADERS)
→ serverless NEG "three-ws-api-neg" (us-central1)
│
Cloud Run service "three-ws-api" (us-central1, min 0, 2 vCPU / 2 Gi, timeout 900s,
allow-unauthenticated, runtime SA three-ws@…)
└─ one Express container (server/index.mjs) serving:
• static frontend from dist/ (baked into the image)
• the vercel.json route table (1,285 rules as of 2026-09-01:
headers/rewrites/redirects/404)
• all api/** handlers with Vercel-parity filesystem routing
│
Data layer (unchanged by the migration — all vendor-neutral HTTP):
Neon Postgres (DATABASE_URL) · Upstash Redis · Cloudflare R2 (S3_* / R2_*)
Cloud Scheduler: one job per vercel.json cron (that array is the count) → GET /api/cron/* with
`Authorization: Bearer $CRON_SECRET`
- GCP project:
aerial-vehicle-466722-p5(billing account "Sperax",01B467-A61905-9A97D2— the $100k credits pool; owner-confirmed 2026-07-07). - Region:
us-central1for everything (same as the model workers). - Image:
us-central1-docker.pkg.dev/aerial-vehicle-466722-p5/cloud-run-source-deploy/three-ws-api:latestBuilt from the repo-rootDockerfile. It runs as the unprivilegednodeuser (not root) and prunes devDependencies after the SDK build (npm prune --omit=dev), so the runtime image carries only production deps. The container never writes to/appat runtime — the persona/ledger disk fallbacks and headless-chromium download both target/tmp— so the root-owned, read-only app tree is expected. Still single-stage: thepython3/make/g++toolchain stays becausecloudbuild.yaml's inline-cache path (BUILDKIT_INLINE_CACHE=1+--cache-from) only restores the final stage's layers, so a multi-stage split would forfeit the cachednpm ciand regress rebuilds from ~3 min to ~12 min. A future move todocker buildxwith a registry cache would unlock the multi-stage drop of the toolchain.
Production database (verified 2026-07-07)
The DB is Neon Postgres, not hosted on Vercel — the migration did not and
could not "move" it; Cloud Run connects to the same instance via DATABASE_URL.
- Production DB: Neon host
ep-muddy-morning-af1v1xpa-pooler.c-2.us-west-2.aws.neon.tech— the live one, verified holding 344 tables / ~676k rows / 13.7k users / 13.3k avatars. This is the valueDATABASE_URLcarries on the Cloud Run service (and it was among the few vars the Vercel pull returned non-empty). - NOT production:
ep-rapid-surf-ak9p7occ(Neon projectwild-river-11025097) appears in old env archives — it is a stale/smaller copy (85 tables / ~127k rows / 224 avatars). Do not point production at it. - ⚠️ DECOMMISSION HAZARD (confirmed 2026-07-08): this Neon project was
provisioned via Vercel's Postgres integration. If it's the Vercel-Managed
Integration (billing inside Vercel), Neon's own docs state that deleting
the database from Vercel's interface removes the underlying Neon project
permanently, and Neon does not offer a self-serve transfer for
Vercel-integrated projects ("projects with Vercel integrations cannot be
transferred" — neon.com/docs/manage/orgs-project-transfer).
There is no dashboard "claim it" button — do not assume one exists.
Two real paths before the Vercel account is closed:
- Open a Neon support ticket asking to detach/transfer
ep-muddy-morninginto a standalone (non-Vercel) Neon org — some users have gotten this handled manually by support even though it isn't self-serve. - Cut over independently: provision a fresh standalone Postgres (Neon-native
org or Cloud SQL), restore into it, swap
DATABASE_URLon the Cloud Run service, verify, then let the old Vercel-linked project go.
- Safety net taken 2026-07-08: full
pg_dump -Fcofneondb(344 tables, 673 MB live / 47 MB compressed) uploaded to Cloudflare R2 ats3://chatty-storage/db-backups/neondb-2026-07-08.dump— durable, off both Neon and Vercel. This is a point-in-time backup for disaster recovery, not a replacement for resolving ownership above; retention/ rotation isn't automated yet.
- Open a Neon support ticket asking to detach/transfer
Service accounts (IMPORTANT — the project's default compute SA was deleted)
Every build and deploy MUST pin these explicitly or it fails with "service account does not exist":
- Build SA:
[email protected](pass via--build-service-account/serviceAccount:in cloudbuild.yaml) - Runtime SA:
[email protected](pass via--service-accounton everygcloud run deploy)
Resolved (verified 2026-07-25): the build SA now holds roles/run.admin (plus
cloudbuild.builds.builder), and gcloud builds submit runs the full
build+push+deploy pipeline end to end. Recent deploys (revisions 00272, 00276)
were created by exactly this path, so no human-authed CLI finish is needed
anymore. If a future submit fails at the deploy step with a permissions error,
re-check this binding first: gcloud projects get-iam-policy aerial-vehicle-466722-p5 --filter="bindings.members:three-ws-build@".
Deploying
# Frontend changed? Build first — dist/ ships from the local build.
# build:gcp = check:conflicts -> check:browser-graph -> check:tdz-bootstrap
# -> ensure:avatar-studio -> build:info:snapshot -> build:lib:full (the agent-3d
# UMD lib) -> build:avatar-sdk -> build:chat -> build (the site `vite build`)
# -> publish:lib -> build:info -> check:dist -> check:pages, IN THAT ORDER. The
# order is load-bearing: build:lib:full and build:avatar-sdk must run BEFORE the
# site build (it resolves avatar-sdk/src/agent.js against the copied bundle), and
# the site `vite build` WIPES dist/, so it must run before publish:lib mirrors the
# lib into dist/. Plain `npm run build` is NOT enough (skips the lib publish, so
# /agent-3d/latest/agent-3d.js 404s and the hero avatar dies). Hand-running
# `build:vercel` as the frontend build is also WRONG — it builds the SDK/lib
# sub-artifacts, not the static HTML pages, so dist/ has no /, /create, etc.
npm run build:gcp
# Build image on Cloud Build (32-vCPU + BuildKit layer cache) + push + deploy.
# Gated on check:dist, check:pages AND db:check so an incomplete dist/, an
# unreachable page, or an out-of-date database can no longer ship. After the
# submit it runs deploy:gcp:sync-crons, which creates a Cloud Scheduler job for
# any cron declared in vercel.json that has none yet (existing jobs are never
# touched), so a new cron starts firing with the code that serves it. Ends with
# a CDN purge and then smoke:prod against the live site.
npm run deploy:gcp
# Or do build + submit + CDN purge in one (from a clean worktree):
# npm run deploy:gcp:full
# Verify the deploy actually landed — /api/version reports the live commit SHA
# and Cloud Run revision (stamped into dist/build-info.json by build:info):
curl -s https://three.ws/api/version | jq '{commitShort, branch, builtAt, revision: .runtime.revision}'
# Sweep every page declared in data/pages.json against the live site. deploy:gcp
# runs this automatically after the purge; run it standalone any time.
npm run smoke:prod
# Tell Bing, Yandex, Seznam and Naver about pages added in the last 7 days so
# they are indexed in minutes. Run only after the deploy is live (IndexNow
# fetches every URL it is given). --days N, --since YYYY-MM-DD, --all, --dry-run.
npm run seo:indexnow
A fresh deploy worktree needs THREE staged artifacts, not just
node_modules.build:gcpshells into two nested projects that carry their own dependency trees, and neither is in git. Hardlinking only the rootnode_modulesleaves both broken, and each one fails several minutes into the build (2026-07-26: two consecutive dead builds, one exit 144, one exit 1):npm run clean:worktrees -- --apply # reclaim old deploy trees before staging a new one npm run prep:worktree # plan only: shows what it would stage or build npm run prep:worktree -- --apply # stage /workspaces/.deploy-wt, artifacts and env included
prep:worktreeis the supported way to stage a deploy tree: it is the sequence below plus the checks a hand-run misses. Use--path <dir>to stage your own tree when a concurrent agent holds the default, and--forceto replace an existing tree (refused when that tree holds uncommitted work). It BUILDS a nested artifact that is absent from the source tree instead of dying on thecp -al, which is what a machine that has never deployed hits first. It copies both env files rather than hardlinking them, and it rejects a--pathon another filesystem up front, becausecp -alcannot hardlink across a device boundary and leaves the tree half-staged when it tries. The equivalent by hand, for reference:git worktree add --detach /workspaces/.deploy-wt HEAD cp -al /workspaces/three.ws/node_modules /workspaces/.deploy-wt/node_modules cp -al /workspaces/three.ws/chat/node_modules /workspaces/.deploy-wt/chat/node_modules cp -al /workspaces/three.ws/character-studio/build /workspaces/.deploy-wt/character-studio/build cp /workspaces/three.ws/.env /workspaces/.deploy-wt/.env cp /workspaces/three.ws/.env.local /workspaces/.deploy-wt/.env.localBoth env files, not just
.env.DATABASE_URLlives in.env.local, and it is what the deploy's migration gate reads; a worktree without it failsdb:checkwithDATABASE_URL is not setand tempts whoever is deploying to skip the gate entirely.Remove the worktree once the deploy lands (
git worktree remove --force /workspaces/.deploy-wt). Nobody did for three days in early August 2026 and ten trees filled the 126 GB disk to 100%. The failure does not name its cause: the nextgit worktree adddies mid-checkout withNo space left on deviceand leaves a half-written tree, and the build after that fails somewhere unrelated.npm run clean:worktrees(plan only;--applyacts) reclaims any linked worktree that is detached, clean, and idle past--min-age-hours(default 2, so a concurrent agent's running build is never deleted). A tree holding uncommitted work is always reported and kept.
character-studio/build—ensure:avatar-studioonly skips the heavy avatar-studio vite build whencharacter-studio/build/index.htmlexists. Without it the build runs for real and gets OOM-killed (exit 144) on a box already hosting concurrent agent builds.chat/node_modules—build:chatrunschat/scripts/ensure-deps.mjs, whosenpm installcan finish "successfully" while still leaving@sveltejs/vite-plugin-svelteunresolvable, sovite builddies withCannot find package '.../@sveltejs/vite-plugin-svelte/index.js'.cp -al(hardlink) only works on the SAME filesystem and is near-instant; plaincp -rof these trees costs minutes. If the destination directory already exists,cp -al src dstnests intodst/build—rm -rfthe destination first.
The CDN purge is synchronous on purpose.
deploy:gcp:purge-cdnused to pass--async, so anything verifying immediately afterwards read stale edge content and reported phantom failures — a cached pre-deploy 404 for a page that the new revision serves fine. Dropping--asynccosts ~3 s and makes every post-deploy check trustworthy. Do not add it back.
Never bypass
check:distby callinggcloud builds submitdirectly. It is the only thing standing between a half-builtdist/and production. When it reportsMISSING: dist/agent-3d/latest/agent-3d.js, that is not a stale or pre-existing failure to work around — it means you ran plainnpm run buildand the agent-3d CDN lib was never published intodist/. Deploying anyway 404s the lib and kills the hero avatar on every page that embeds<agent-3d>. Fix it by runningnpm run build:gcp(or, if the site build already ran,npm run build:lib:full && npm run publish:lib) untilcheck:distis green. This happened on 2026-07-09 (revisionthree-ws-api-00034), where the check was overridden on the assumption its failure was unrelated to the change.
check:pagesguards a different failure: a page that ships unreachable. Every entry indata/pages.jsonfeeds the sitemap,llms.txtandfeatures.json, so it is advertised to crawlers the moment it lands, whether or not anything can serve it.server/index.mjs'sresolveStatic()has no.htmlextension fallback by design, so a page built todist/<slug>.htmlis a hard 404 at/<slug>unlessvercel.jsoncarries a rewrite. That shipped twice in three days:/timeline(fixed by5688277bd, "page shipped without a route entry") and/tracker(advertised 2026-07-23, 404 until a route landed two days later).check:pagesresolves all ~760 declared paths through the real route table and fails the build on any that dead-end. When it fires it tells you which of the two causes applies: a missingvercel.jsonrewrite, or a page that was never built (missingvite.config.jsinput entry).
Database migrations live in api/_lib/migrations/*.sql and are tracked in
the schema_migrations table (filename + sha256). Nothing applies them
automatically — the flow is:
npm run db:status # dry run: list applied/pending against DATABASE_URL
npm run db:migrate # apply pending migrations (do this BEFORE deploy:gcp)
npm run db:check # what deploy:gcp and deploy:gcp:submit run: exits 4 if anything is pending
DATABASE_URLcomes from.env.localand must point at the production Neon DB (the same value the Cloud Run service uses).- Never edit a migration file after it has been applied — the runner detects the sha256 drift and refuses (exit 3). Roll forward with a new file.
- Apply migrations before deploying the code that needs them: migrations are
additive, so old code + new schema is safe; new code + old schema is not.
When that order is inverted the symptom appears far from the cause: on
2026-08-14
/api/healthzreported the EVM agent indexunknownwithcolumn "blocks_behind" does not exist, because the code that reads and writes those columns shipped while20260813210000_erc8004_crawl_head_tracking.sqlwas still pending. Submit throughnpm run deploy:gcp:submit(ordeploy:gcp), never a baregcloud builds submit, sodb:checkgets its say. db:migratestops at the first migration that fails and reports every migration still queued behind it (exit 5). Those are unapplied: the schema is behind the code by exactly that list until the failing file is fixed with a NEW migration and the run drains the rest.
Dependency and import gates. deploy:gcp:submit runs two more checks after
db:check, both added after the 2026-09-18 outage, when two merged dependency
bumps took the paid x402 surface down on import:
npm run audit:deploy # includes: installed node_modules must match package-lock.json
npm run check:api-imports # imports every module under api/, fails on any that throws while loading
The image installs from
package-lock.json, but local tests and every deploy worktree (which hardlinksnode_modules) run whatever is on disk. If the two differ, a greennpm testsays nothing about what production will boot. Whenaudit:deployreports drift, runnpm install, thennpm test.The server imports a handler the first time its route is hit, so a module that throws while loading is not a build error. It is a 500 on every request to that route.
check:api-importsneeds no environment and groups failures by error, so one broken shared module reads as one finding. Scope it with a path while debugging:node scripts/check-api-imports.mjs api/x402.A dependency bump that changes an API is fixed in code or pinned back in
package.jsonand held in.github/dependabot.yml. It is never waved through.Lockfile unchanged → layer cache skips the workspace
npm ci: ~3–5 min.Lockfile changed → full install: ~12 min.
If the pipeline's deploy step fails on IAM (see above), deploy the pushed image directly:
gcloud run deploy three-ws-api \
--image us-central1-docker.pkg.dev/aerial-vehicle-466722-p5/cloud-run-source-deploy/three-ws-api:latest \
--project aerial-vehicle-466722-p5 --region us-central1 \
--allow-unauthenticated --memory 2Gi --cpu 2 --timeout 900 \
--service-account [email protected]
Rollback (instant):
gcloud run revisions list --service three-ws-api --region us-central1 --project aerial-vehicle-466722-p5
gcloud run services update-traffic three-ws-api --region us-central1 \
--project aerial-vehicle-466722-p5 --to-revisions <good-revision>=100
Logs: use the reader CLI; it handles jsonPayload and request-log
entries that a raw textPayload read silently misses. Full guide incl. the
automated triage monitor: gcp-logs.md.
npm run logs # three-ws-api, last hour (vercel logs equivalent)
npm run logs:tail # live tail
npm run logs:errors # ERROR+ across all services, last 6h
npm run triage:gcp # healthz + fleet log sweep -> classified action plan
Build-context gotcha: the upload is governed by the allowlist in
.gcloudignore. If a handler starts reading a directory
at runtime that isn't allowlisted, it will be missing from the image — add the
directory there. animation-sources/ (2.6 GB) is deliberately excluded; the
few endpoints reading it degrade until it moves to object storage.
Not yet created: a Cloud Build GitHub trigger (push-to-deploy like Vercel
had). Requires connecting the nirholas/three.ws repo in Cloud Build and
adding the vite build into the Docker build. This is Cloud Build's own GitHub
integration — not GitHub Actions, which this repo does not use.
/ingest/* (PostHog proxy): on Vercel, vercel.json routes whose dest
is an absolute URL (/ingest/static/* → us-assets.i.posthog.com, /ingest/*
→ us.i.posthog.com) were proxied by the platform itself. server/index.mjs
now replicates this with its own external-dest middleware (mounted before the
body parsers, so POST event bodies stream through unconsumed) — added
2026-07-07 after prod was serving 404s + a MIME-sniffing block on
/ingest/static/array.js. If a future vercel.json route gets an
http(s):// dest and analytics start 404ing again, check that this
middleware's phase1Routes walk still matches it before the API/static
phases.
Crons (Cloud Scheduler)
The schedules in vercel.json crons are mirrored to Cloud Scheduler jobs in
us-central1, mostly named cron--api-cron-<name> (a few older ones use a bare
name, so match by target URI, not by job name, when auditing). Vercel's crons
died with the deployment, so there is no double-fire risk.
The mirror is not automatically enforced: verify it after editing crons.
Declaring a cron in vercel.json does nothing on its own; only the sync script
below creates the job. That gap has bitten repeatedly (on 2026-07-25 five
declared crons had no scheduler job at all and had therefore never run in
production: kol-tracker-refresh, forge-thumbnail-backfill,
forge-off-crown, quality-bench, confirm-pending-purchases; again on
2026-08-14 for retention-rollup and likeness-eval), so audit with the two
checks rather than by eye:
# Which declared crons are missing, paused, or on a different schedule than
# vercel.json says? Needs a live gcloud session; add --json for a report.
# Each MISSING row also says WHY, from production's own answer on that path:
# "deployed, never synced" -> the handler is live and only the job is absent,
# so the sync below fixes it right now.
# "handler not deployed" -> the path 404s, so its job belongs to the deploy
# that ships the handler; syncing now would only
# schedule a 404 every run.
npm run check:cron-drift
# Do the handlers behind them route, load, run, and refuse an anonymous caller?
npm run audit:cron-liveness
# Sync after editing crons in vercel.json. Idempotent create-or-update, and
# CONFIG ONLY: it never changes whether an existing job is running, so it is safe
# to re-run during an incident hold. A job it creates starts ENABLED.
CRON_SECRET=<value> node scripts/create-gcp-scheduler.mjs
# Blanket run-state levers, explicit and mutually exclusive:
CRON_SECRET=<value> node scripts/create-gcp-scheduler.mjs --pause # stop the whole fleet
CRON_SECRET=<value> node scripts/create-gcp-scheduler.mjs --resume # recover from --pause
# List / pause / force-run one job. Note the DOUBLE hyphen after the `cron`
# prefix: the cron path's leading slash becomes a hyphen of its own, and a
# single-hyphen name fails with NOT_FOUND.
gcloud scheduler jobs list --location us-central1 --project aerial-vehicle-466722-p5
gcloud scheduler jobs pause cron--api-cron-economy-tick --location us-central1 --project aerial-vehicle-466722-p5
gcloud scheduler jobs run cron--api-cron-uptime-check --location us-central1 --project aerial-vehicle-466722-p5
CRON_SECRET: was EMPTY on Vercel (every cron guard fail-closed with 503 — Vercel crons had been silently dead). A real secret was generated 2026-07-07 and set in two places: the Cloud Run service env and every Scheduler job's Authorization header. Recover it any time from either place:
gcloud scheduler jobs describe cron--api-cron-uptime-check --location us-central1 \
--project aerial-vehicle-466722-p5 --format="value(httpTarget.headers)"
Environment variables
All service env lives on the Cloud Run service (view/edit):
gcloud run services describe three-ws-api --region us-central1 \
--project aerial-vehicle-466722-p5 --format=yaml | grep -A2 'name:'
gcloud run services update three-ws-api --region us-central1 \
--project aerial-vehicle-466722-p5 --update-env-vars KEY=value
⚠️ THE MIGRATION TRAP — read this before trusting any env export
vercel env pull silently returns EMPTY values for secret-type vars —
144 of 173 keys pulled blank (only integration-injected vars like
DATABASE_URL came through). The plaintexts are not retrievable from Vercel
by CLI at all. Consequences and current state:
- The initial 157-var application contained mostly empty strings.
- The
S3_*group + real values for ~90 keys were recovered from the owner's own env archives (owner-provided dump, 2026-07-07) and staged as an 89-var validated set (deduped, other-project keys pruned, JWKS verified). If the staging machine is lost before application, the set must be rebuilt from the owner's archives — the sources are the owner's saved env dumps; the selection rules are: skipVERCEL*/NX_*/TURBO_*/PG*/POSTGRES_*/NEON_*/DATABASE_URL*/REDIS_URL, skip keys not referenced by this codebase, last-occurrence-wins on duplicates. - Verify state empirically, not from dashboards: an endpoint returning
503 {"error":"not_configured"}means aMissing required env var: Xline in the logs names the exact var.
Vars with no default that silently kill a whole subsystem
CIRCULATION_ENABLED=1 is the master gate for the circulation engine
(api/_lib/circulation.js, driven by pulse-tick via economy-tick) — the
engine that generates the Money Pulse's agent-to-agent activity. It is OFF by
design when unset, so it does not show up as a 503 not_configured or any log
error: every tick just reports a clean skipped: "disabled". This is exactly
how the 2026-07-07 migration killed the pulse for ten days — the var existed
only on Vercel, was never carried to Cloud Run, and nothing alerted. Restored
2026-07-17 (--update-env-vars CIRCULATION_ENABLED=1; the treasury secret
CIRCULATION_TREASURY_SECRET already rides on Secret Manager
wallet-sol-engines-b58). When auditing env completeness, check the
economy section of /api/status for engines reporting disabled, not just
5xx responses.
Known-missing (blocked on owner)
| What | Vars | Impact | Recovery |
|---|---|---|---|
| Ops-alert Telegram push | TELEGRAM_ALERTS_CHAT_ID |
api/_lib/alerts.js degrades to table-only (ops_alerts): no push notification for treasury-low, balance-floor, or breach alerts. This is why the 10-day Money Pulse flatline never pinged anyone. TELEGRAM_BOT_TOKEN survived; only the private ops chat id was lost in the migration. |
Owner supplies the private ops chat/DM id (never the public holders' channel) and applies --update-env-vars TELEGRAM_ALERTS_CHAT_ID=<id>. |
| Collection authority | SOLANA_AGENT_COLLECTION_AUTHORITY_KEY |
Agent NFT collection ops can't sign | Intentionally excluded from scripts/wire-master-wallet.mjs (on-chain update authority must stay its original wallet). Owner holds the key. |
| R2/S3 storage creds | S3_ENDPOINT, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_BUCKET, S3_PUBLIC_DOMAIN |
/api/marketplace, /api/explore, /api/avatars/:id — every route that resolves an asset URL — 503 not_configured |
Real values already sit in the repo's .env.local (never committed). Apply with scripts/gcp/apply-s3-env.sh (needs a human-authed gcloud auth login first — the 89-var apply is still blocked on reauth per above). |
| CoinCommunities API key | CC_API_KEY |
Town chat renders its designed locked state. /api/community/worlds and /api/clash/state + /api/clash/leaderboard no longer die: they fail over to the live pump.fun trending feed (api/_lib/pump-trending.js) and tag the response source: "pump-trending", so the world picker and the Clash arena stay alive (the Clash roster is frozen per round). The Play holders gate shows its unavailable state ("Holder check is offline") and offers a one-click path into the coin's open world, so a gated world is never a dead end. |
Keys are provisioned manually by the CoinCommunities team (their docs say "contact support": no self-serve signup, and the SDK's business/register endpoints are not deployed on api.coin-communities.xyz). Owner outreach to @CoinComms required; set CC_API_KEY (+ optional CC_SERVER_KEY/CC_SERVER_SECRET) on the Cloud Run service when granted. |
| Agent mail to and from the internet | (not a var: DNS records for agents.three.ws) |
Mail between two agents.three.ws mailboxes works (delivered inside the platform). Mail to any outside address fails with send_failed (credits are refunded), and outside senders cannot reach an agent, because the mail provider reports the agents.three.ws domain as failed: none of its records exist at the DNS host. Checked 2026-10-10. |
In the Resend dashboard, open the agents.three.ws domain and add every record it lists at the DNS host for three.ws: the DKIM TXT at resend._domainkey.agents, the SPF MX and TXT at send.agents, the return-path CNAME, and the receiving MX at agents (priority 10). Then press Verify and point the domain's inbound webhook at https://three.ws/api/mail/inbound with the secret in RESEND_WEBHOOK_SECRET. See docs/agent-mail.md. |
| x402 ring payer USDC float | — (not a var: an on-chain balance) | The autonomous loop's payer wallet holds $0 USDC, so every paid call is skipped. Since 2026-07-09 this manifested as ~1 failed settle/minute (broadcast_failed:Simulation failed); the loop now skips paid calls instead. Free + monitoring entries are unaffected. |
Send USDC to the payer wallet (X402_SEED_SOLANA_SECRET_BASE58's pubkey). Floor is X402_RING_PAYER_USDC_FLOOR_ATOMIC (default $5) — fund meaningfully above it or it re-drains. See x402 ring payer float below. |
| x402 prepaid settler | X402_PREPAID_SETTLER_SECRET_BASE58, X402_PREPAID_SETTLE_LIVE, X402_PREPAID_DEPOSIT_WALLET, X402_PREPAID_MIN_SETTLE_ATOMICS, X402_PREPAID_MAX_PAYEES_PER_TX |
Unset settler or X402_PREPAID_SETTLE_LIVE not 1: the settle cron only plans, committed debits accumulate unpaid. |
Owner action: fund the settler (SOL fees plus USDC float), review the planned window, then set the flag with --update-env-vars. See x402-prepaid.md. |
x402 ring payer float — why an empty wallet used to storm Solana
The autonomous loop (api/cron/x402-autonomous-loop.js, driven every minute by
economy-tick) spends real USDC against the platform's own paid endpoints. Two
independent limits apply, and for a long time only one was enforced:
- the daily spend cap (
remainingCap) bounds what the loop is allowed to spend; - the payer's USDC balance bounds what it is able to spend.
With a drained float and cap headroom remaining, each tick probed for a 402,
signed a payment transaction, and POSTed it — and the self-hosted facilitator
failed every one at settle with broadcast_failed:Simulation failed. Observed:
1,665 failed settles against 3 successes over 24h. Nothing was lost (the
transaction fails in simulation, before broadcast, so no fee is burned and no
funds move), but the ring did no useful work and hammered the RPC indefinitely.
readPayerUsdcAtomic() is now read once per tick and treated as a hard spend
ceiling. Its three-way contract is load-bearing, and tests/x402-autonomous-payer-float.test.js
pins it:
| Read result | Meaning | Loop behavior |
|---|---|---|
| a number > 0 | real float | pay up to min(dailyCap, float) |
0 |
genuinely empty (including a wallet with no ATA) | skip every paid call; free + run() monitors still execute |
null |
balance undeterminable (RPC blip) | do not gate — leave the spend path open |
The null-vs-0 distinction matters in both directions: if a transient RPC
failure read as 0, one blip would silently halt the ring; if an empty wallet
read as null, the doomed-payment storm returns. Emptiness is detected
structurally (getAccountInfo → null ⇒ no token account ⇒ no USDC), not by
pattern-matching an RPC error string, whose wording varies by provider.
To resume autonomous spend, fund the payer's USDC float. The loop self-recovers on the next tick — no redeploy, no flag flip.
The sponsor's SOL floor gate has the opposite default. Its balance read can also
come back null, and until 2026-08-28 that read as "not paused": all four paid
RPC lanes were over quota at the moment the sponsor sat at 0.000899 SOL, so the
loop spent three hours signing payments that could not settle (95 attempts, 0
settled). An unreadable balance is not evidence of solvency, so a null read now
defers to sponsorKnownBelowFloor() (api/_lib/x402/self-facilitator.js), which
the settle path itself writes when the chain rejects the sponsor for
rent-exemption. That second opinion costs no RPC call, so the gate still answers
when the RPC does not.
Ops alerts: dashboard-primary (2026-07-12)
Ops alerting used to be Telegram-only, and Telegram-only meant that with no chat
configured every sendOpsAlert() was a silent no-op — 5xx faults,
api/client-errors.js browser reports, ring-leak CRITICALs, and low-balance
warnings all dropped. That is why the x402 payment storm below ran for a day
unnoticed.
Now every sendOpsAlert() always persists to the ops_alerts table
(api/_lib/alerts.js, migration 20260712000000_ops_alerts.sql), keyed by the
alert's stable signature so a recurring condition is one row with a growing
count, not a flood. Read the active feed from the ops_alerts table or the
x-ops-secret-gated ops APIs (/api/ops/health and friends; the gate is
authorizeOps in api/_lib/ops-auth.js: admin session or a dedicated
OPS_SECRET, deliberately never CRON_SECRET, fail-closed in production).
This is the primary sink and needs no third-party service.
Telegram is now an optional extra push, off by default. When both
TELEGRAM_BOT_TOKEN and TELEGRAM_ALERTS_CHAT_ID are set, alerts also push to
that private chat (deduped per signature for 1h, capped at 20/h). It is
deliberately unset in production — the dashboard is the channel. /api/healthz
reports alerts.primary: "ops_alerts" and alerts.telegram_push: "disabled"|"enabled"; a disabled push is expected, not a fault. The
@threewsbot token remains in Secret Manager (telegram-bot-token) for
user-facing bot use (e.g. changelog push to holders); it is not wired to ops
alerts. To turn the extra push on, add @threewsbot to a private ops chat, read
its id from getUpdates, and set TELEGRAM_ALERTS_CHAT_ID (+ point
TELEGRAM_BOT_TOKEN at the secret).
Resolved: x402 sponsor co-signing key (2026-07-09)
/api/healthz reported x402.sponsor_cosign: "missing" and every sponsor-mode
settlement (club-cover, dance-tip) failed with sponsor_key_unconfigured.
Root cause: X402_FEE_PAYER_SOLANA advertised
2wKupLR9q6wXYppw8Gr2NvWxKBUqm4PPJKkQfoxHDBg4 — a third-party facilitator's
shared fee-payer account that three.ws has never controlled, left over from
when Solana settlement went through an external facilitator (see
ARCHITECTURE.md). The self-hosted facilitator cannot co-sign for a key it does
not hold, so it 502'd at settle while passing every other health check.
An earlier revision of the table above described that address as a three.ws ring
wallet that "custodies funds — do NOT regenerate." That was wrong, and it
would have blocked the correct fix indefinitely. Evidence: the account fee-pays
~200 tx/hour for many unrelated co-signer wallets, none of them involving our
payTo, while our own cosign was provably unconfigured. Its balance was never
ours. scripts/audit-service-wallets.mjs has always flagged it correctly and
told operators to override the var.
Fix applied:
- Generated a three.ws-controlled sponsor with
scripts/x402-ring-setup.mjs --roles=sponsor(sponsor role only — the fundedpayTotreasury was never touched):GGf9qBhJDCe1UUz4s4Vxq1uPPvcv7UW7sJTuj2Yo5XQj. - Stored the secret in Secret Manager as
x402-fee-payer-secret-base58(grantedroles/secretmanager.secretAccessortothree-ws@…) and wired it as asecretKeyRef, rather than a plaintext env value on the service. This also removes the "secret only exists on whatever machine ran the script" fragility. - Funded it with 0.05 SOL (floors:
X402_SPONSOR_SOL_FLOOR_LAMPORTS= 0.02 hard refuse-to-settle, 0.03 audit/topup).
gcloud run services update three-ws-api --region us-central1 \
--update-env-vars X402_FEE_PAYER_SOLANA=GGf9qBhJDCe1UUz4s4Vxq1uPPvcv7UW7sJTuj2Yo5XQj \
--update-secrets X402_FEE_PAYER_SECRET_BASE58=x402-fee-payer-secret-base58:latest
Verified by a real $0.01 USDC mainnet sponsor-mode settlement through
POST /api/x402-facilitator/settle (tx
5rmLnLUmWK7jJCjP6RJPLmwT2oiifQMmTnQmhoQKeZcVogxnTMvxB77fpb5BqMyV6VzbawXdPgWz2N7C5kXqqbQ5):
the sponsor signed as fee payer, paid the 10001-lamport fee, and $0.01 landed in
payTo. sponsor_cosign now reports ready.
Resolved: Upstash Redis + full signer wiring (2026-07-09, same day, later)
Owner supplied the Upstash database and the wallet secrets. Applied on revision
three-ws-api-00029-c45:
- Upstash Redis live:
UPSTASH_REDIS_REST_URL(plaintext) +UPSTASH_REDIS_REST_TOKEN(Secret Managerupstash-redis-rest-token).X402_ALLOW_MEMORY_FALLBACKremoved per the escape-hatch's own instructions. healthz cache backend now reportsupstash healthy; money limiters rate-limit for real instead of failing closed. - Signer slots wired per
scripts/wire-master-wallet.mjsASSIGNMENTS, from the three economy wallets (wwwww…ccrUx402 loop,wwwqv…HGUnSOL engines,WwwuGbq…3WwWtreasury face). Secrets live in Secret Manager (wallet-x402-treasury-b58/b64,wallet-sol-engines-b58/b64,wallet-economy-master-b58/b64) assecretKeyRefs — no longer only on one machine.ECONOMY_MASTER_SECRET_BASE58,X402_TREASURY_SECRET_BASE58,X402_SEED_SOLANA_SECRET_BASE58,A2A_PAYER_SOLANA_SECRETwere concurrently set (same wallets) by another session and left as-is.LAUNCHER_MASTER_SECRET_KEY_B64deviates from the map (deployed onX4o2…astML, map sayswwwqv) — left deployed value;LABOR_ESCROWskipped per the script's own live-funds guard. - Proof, full user path (not just the facilitator): real paid calls through
the live endpoints returned 200 + product content —
club-cover $0.01 (tx
54FQ4rAFjfgF9b5pDRCB5dnhWNoTK7RrvJm2EJAuNGQDiMAPcLGptXpWzdBys49KHEASH7JrKb8sHL3AFwZT36gF, pass issued) and dance-tip $0.001 (tx2hSvvR17hpawE1ZjhgHisaEpRzYkh6jyWtWBgsoCKAhunQkhcUCfQQ49aY2QmfsCYNBaMpvVTxSBfky5t7tstUZ6, thriller ticket).scripts/audit-service-wallets.mjsagainst the live revision: all checked wallets configured, funded, and consistent (14/15 signers; only the intentionally-excluded collection authority remains).
Resolved: self-hosted Memorystore Redis, off the Upstash free tier (2026-07-10)
The Upstash free database (saving-titmouse-110599) hit its hard 500k
commands/month cap within hours — that ceiling is far too small for this
platform (the rate limiter alone spends 2–3 commands per request). Redis then
errored on every command, the cache circuit opened, and paid endpoints were only
saved by the durable Postgres rate-limit fallback (commit a1b0b9610). The free
tier does not reset until the 1st, so the fix was to stand up our own Redis on
the GCP credit pool — permanently ending the third-party quota dependency.
Architecture (all in aerial-vehicle-466722-p5, us-central1; app code
UNCHANGED — it still speaks the @upstash/redis REST dialect):
three-ws-api (Cloud Run)
├─ VPC connector "three-ws-vpc" (10.8.0.0/28), egress private-ranges-only
│ → only RFC1918 dests traverse the VPC; all external calls (RPC, LLM
│ APIs) stay on the direct public path (no Cloud NAT needed, no egress
│ bottleneck).
└─ UPSTASH_REDIS_REST_URL = http://10.128.15.228 (internal LB IP)
UPSTASH_REDIS_REST_TOKEN = secret:srh-proxy-token
│
Regional internal Application LB (10.128.15.228:80)
forwarding-rule "three-ws-redis-fr" → target-http-proxy "three-ws-redis-tproxy"
→ url-map "three-ws-redis-urlmap" → backend "three-ws-redis-be"
→ serverless NEG "three-ws-redis-neg"
(proxy-only subnet "three-ws-proxy-only" 10.100.0.0/23; firewall
"three-ws-allow-proxy-subnet" + "three-ws-allow-connector-to-ilb")
│
Cloud Run "three-ws-redis-proxy" (hiett/serverless-redis-http:0.0.10, pinned;
ingress internal-and-cloud-load-balancing; SRH_TOKEN=secret:srh-proxy-token;
SRH_CONNECTION_STRING=redis://10.234.231.139:6379; on VPC connector three-ws-vpc)
│
Memorystore Redis "three-ws-redis" (basic, 1 GB, redis_7_0) @ 10.234.231.139:6379
Why this shape: the org blocks allUsers invocation (domain-restricted
sharing), so a public REST proxy is unreachable (its SRH bearer token collides
with the Google auth token Cloud Run would demand). Fronting the proxy with a
regional internal LB and reaching it over the VPC connector keeps the proxy
off the public internet AND keeps the app's external egress on the fast direct
path. The connector + Redis env are pinned in server/cloudbuild.yaml so a
full-config redeploy can never silently drop Redis back to the capped Upstash
tier.
Verified 2026-07-10 on revision three-ws-api-00052-kbg: /api/healthz
overall ok, cache backend healthy (the client self-labels "upstash" — it is
our Memorystore); proxy access logs show a steady stream of POST 200
/pipeline commands; the old Upstash endpoint stayed capped and untouched
(proving the app moved off it); and a real $0.01 club-cover settlement completed
end to end (tx 4fWXfymUxwS7YRMrkQqHwwgtA1GiEAmnJXsgYaXHo5NvyAqhisGkmU3k7waLD85RsVx5s4mXmB4ymiTuEvzAWnQ8).
Cost ≈ $75/mo on the credit pool (Memorystore ~$35, proxy min-1 ~$13, connector
~$9, internal LB ~$18). To retire: repoint UPSTASH_REDIS_REST_URL/TOKEN at a
managed Upstash paid plan and delete the four three-ws-redis-* LB resources +
proxy + Memorystore + connector. The upstash-redis-rest-token secret and the
saving-titmouse free DB are now unused (kept for rollback).
| What | Vars | Impact | Recovery |
|---|---|---|---|
| OKX X Layer facilitator creds | OKX_API_KEY, OKX_SECRET_KEY, OKX_PASSPHRASE (or, as a no-OKX-account fallback, X402_XLAYER_RELAYER_KEY) |
xlayerSettleable() (api/_lib/x402-xlayer-okx.js) is false without one of these, so no 402 challenge on any endpoint advertises the eip155:196 (X Layer) rail, confirmed live 2026-07-08 via /api/okx/3d/health (payment-rail.settleable:false, facilitator_configured:false) and by inspecting the raw 402 accepts[] on /api/okx/3d/identity-studio and /api/okx/3d/pose-seed (Solana + Base only, no X Layer entry). X402_PAY_TO_XLAYER and X402_ASSET_ADDRESS_XLAYER are set in prod, so this is the only missing piece. This is the exact rail OKX's listing review requires (agent #2632 rejection reason). The credential has since been set: probed live 2026-09-09, all four paid Forge rows advertise eip155:196 and OKX's own agent x402-check reads valid: true on each. Treat this row as the recovery path if that ever regresses; the relisting order is prompts/finish/911-okx-ai-08-forge-relisting.md. |
OKX Web3 developer console (owner) for the three real creds, or generate X402_XLAYER_RELAYER_KEY as a fresh EVM keypair + fund it with OKB gas as a stopgap that needs no OKX account. |
DNS / TLS / CDN
- Registrar + DNS: Namecheap (nameservers
registrar-servers.com). Web records:A @ → 136.68.246.178,A www → 136.68.246.178. Everything else there (privy/world CNAMEs, TXT/DKIM) is mail/verification — untouched by the migration. - TLS: Google-managed cert
three-ws-cert(three.ws, www.three.ws) — went ACTIVE 2026-07-07 ~18:40 UTC, auto-renews. Status:gcloud compute ssl-certificates describe three-ws-cert --global --project aerial-vehicle-466722-p5 - CDN: Cloud CDN on
three-ws-backend,USE_ORIGIN_HEADERS— the vercel.json asset rule (7-day cache on glb/png/woff2/…) is what the CDN honors. - The Cloud Run URL
https://three-ws-api-lp642k3kpa-uc.a.run.appserves the identical site and bypasses the LB — useful for isolating LB vs service issues. www.three.wsis a duplicate host, not a second origin. Its A record and the managed cert both cover it, so it reaches the same backend, but the apex is the only origin the platform declares:server/seo-head.mjspins every canonical URL tohttps://three.wsand the media bucket's CORS read rule allowlists that origin alone. Measured 2026-09-09 on one bucket object: the apex getsAccess-Control-Allow-Origin: https://three.wsback,wwwgets no header at all, so every GLB fetch on a www page was blocked by the browser.server/index.mjsnow redirects the whole www host to the apex (301 on GET/HEAD, 308 on everything else so a write keeps its method and body), andtests/server-canonical-host.test.jslocks in that no other host, including the*.run.appURL above anddev.three.ws, is touched. Do not add a second serving hostname without either widening the bucket policy to match or redirecting it here too.
A truncated asset can sit in the CDN and hang every browser (2026-09-11)
Cloud CDN held one copy of /vendor/mediapipe/wasm/vision_wasm_internal.wasm
that carried content-length: 11153617 (the identity length) with a
brotli-compressed body and no content-encoding header. A client that reads
that response waits forever for eight megabytes that never arrive.
It only reproduced with the exact Accept-Encoding a browser sends:
# healthy
curl -s -o /dev/null -w '%{size_download} %{time_total}\n' \
-H 'Accept-Encoding: br' https://three.ws/vendor/mediapipe/wasm/vision_wasm_internal.wasm
# the bug: bytes stop arriving, the request never completes
curl -s -o /dev/null -w '%{size_download} %{time_total}\n' \
-H 'Accept-Encoding: gzip, deflate, br, zstd' \
https://three.ws/vendor/mediapipe/wasm/vision_wasm_internal.wasm
The Cloud Run origin served the same URL correctly under every encoding, so the bad variant was created and cached at the edge, not by the container. A cache-busting query string also served correctly, which is how you tell a poisoned object from a broken build.
Symptom at the product level: /create/selfie sat on "Processing your
face..." indefinitely and never sent the reconstruction request, because the
capture gates await MediaPipe and MediaPipe's loader has no deadline. Nothing
errored, nothing retried, and no page check caught it: a page sweep reads HTML,
not the largest binary on the site.
Recovery, one path, not a global purge:
gcloud compute url-maps invalidate-cdn-cache three-ws-lb \
--path '/vendor/mediapipe/wasm/vision_wasm_internal.wasm' \
--project aerial-vehicle-466722-p5
Detection: npm run check:asset-encoding walks the largest compressible
files under public/, requests each from the live site with a browser's
Accept-Encoding, reads every body to completion, and fails on a length that
disagrees with the bytes delivered. Run it after a deploy; it is the only check
that would have caught this.
Client-side hardening: src/shared/mediapipe-assets.js now prefers our own
vendored copy unless the origin definitively lacks the file (a probe that
merely times out no longer sends the runtime to a public CDN), shares one probe
across all callers, and bounds every vision-model load with
withVisionDeadline. A stalled load now degrades to the unrefined photo after
15 seconds instead of hanging the flow. npm run verify:selfie -- --stall-vision
reproduces the hostile case end to end.
A rejected object-storage credential silently eats finished avatars (2026-09-11)
The R2 credential stopped verifying at 03:15 UTC on 2026-09-11, the second
such outage in three days (the first ran 05:04-05:19 on 2026-09-09 and was
fixed by rotating the secret). Every signed operation answered
SignatureDoesNotMatch, for reads and writes alike.
Confirm it in one command, against whatever the service is actually running
(the secret is a Secret Manager reference, so describe shows valueFrom):
export S3_ENDPOINT=$(node scripts/read-service-env.mjs '^S3_ENDPOINT$' --raw)
export S3_BUCKET=$(node scripts/read-service-env.mjs '^S3_BUCKET$' --raw)
export S3_PUBLIC_DOMAIN=$(node scripts/read-service-env.mjs '^S3_PUBLIC_DOMAIN$' --raw)
export S3_ACCESS_KEY_ID=$(node scripts/read-service-env.mjs '^S3_ACCESS_KEY_ID$' --raw)
export S3_SECRET_ACCESS_KEY=$(node scripts/read-service-env.mjs '^S3_SECRET_ACCESS_KEY$' --raw)
node -e "const {putObject}=await import('./api/_lib/r2.js');
await putObject({key:'forge/refs/probe.png',body:Buffer.from('89504e470d0a1a0a','hex'),contentType:'image/png'});
console.log('write ok')"
Count the blast radius, which is what tells a credential fault from a code bug:
gcloud logging read 'resource.type="cloud_run_revision"
resource.labels.service_name="three-ws-api" textPayload:"SignatureDoesNotMatch"' \
--freshness=12h --limit=1000 --format='value(timestamp)' \
--project aerial-vehicle-466722-p5 | cut -c1-13 | sort | uniq -c
A clean start hour with nothing before it is a credential that died at that minute. Failures spread evenly across many hours are a different fault.
Symptom at the product level: /create/selfie reconstructed the mesh on
the GPU, took ~35 s, and then said "Avatar finished but could not be saved.
Try again." Retrying could not work, because the prompt was never the problem.
The job row is the ground truth, and it logs nothing:
select job_id, status, result_avatar_id, error from avatar_regen_jobs
order by created_at desc limit 5;
-- status 'done', result_avatar_id NULL, error NULL
error stays NULL on purpose: /api/cron/reconstruct-sweep recovers exactly
the rows matching status = 'done' and error is null, so writing the failure
there would strand the job instead of retrying it. Do not "fix" that NULL.
Why it is no longer fatal. api/_lib/reconstruct-finalize.js treats a
storage-infrastructure fault (isStorageInfrastructureError: credential
rejected or revoked, bucket missing, endpoint unreachable) as a reason to lose
the durable copy, never the reconstruction. The mesh is already public and
lifecycle-free in gs://three-ws-avatar-reconstructions, so that URL becomes
the avatar's storage key: publicUrl() passes an absolute key through,
defaultStorageMode() records r2.present = false, and source_meta carries
servedFrom: 'provider' plus the pendingBucketKey a later re-copy needs. Any
other fault still throws. Same trade api/_lib/image-persist.js makes for
reference images.
POST /api/v1/ai/image cannot make that trade (its contract is a durable https
URL a third party can fetch), so it answers 503 storage_unavailable rather
than the old opaque 502 generation_failed, which told callers to retry a good
prompt and told operators nothing.
Real recovery is a new R2 API token, and it is owner-gated: no Cloudflare
API token exists on this machine or on the Cloud Run service, so a token cannot
be minted from here. Mint one in the Cloudflare dashboard (R2 > Manage API
tokens, Object Read & Write on the bucket in S3_BUCKET), then:
printf %s "<new secret>" | gcloud secrets versions add s3-secret-access-key \
--data-file=- --project aerial-vehicle-466722-p5
# only if the access key id changed too (--update-env-vars MERGES; --set-env-vars REPLACES)
gcloud run services update three-ws-api --region us-central1 \
--update-env-vars S3_ACCESS_KEY_ID=<new id> --project aerial-vehicle-466722-p5
api/_lib/r2.js re-probes a rejected credential every 15 s, so a fixed token
recovers without a redeploy. Avatars written during the outage keep serving from
GCS; find them with storage_mode->'r2'->>'present' = 'false'.
Client geo header (analytics country column)
api/_lib/client-geo.js resolves a visitor's country from an edge geo header
and never from the raw IP. On Vercel that header was x-vercel-ip-country, set
by the edge for free. The GCLB sets no geo header by default, so after the
2026-07-07 migration widget_views.country and widget_chat_threads.country
went null for every row (verified: 49/49 populated in May, 0/27 in August), and
the top_countries panel on /api/widgets/:id/stats renders empty for every
widget.
The reader now prefers x-client-geo-location, then x-client-region. Populate
one on the backend service to restore capture (owner-gated: this is a
production LB change, not a Cloud Run config update):
gcloud compute backend-services update three-ws-backend --global \
--project aerial-vehicle-466722-p5 \
--custom-request-header='X-Client-Region:{client_region}'
{client_region} is the GCLB variable for the ISO 3166-1 alpha-2 code of the
client. Confirm with gcloud compute backend-services describe three-ws-backend --global --format='value(customRequestHeaders)', then check capture:
select count(*) filter (where country is not null), count(*)
from widget_views where created_at > now() - interval '1 day';
Until that header exists the code degrades to null rather than guessing, which
is the intended behavior. Note the LB does not strip an inbound copy of these
headers, so a client can forge one; clientCountry validates every value down
to two ASCII letters so the worst case is a visitor mislabeling its own row.
Verifying after any change
B=https://three.ws
for p in / /3d /changelog /api/healthz /api/feed /api/explore?limit=4; do
curl -s -m 25 -o /dev/null -w "%{http_code} $p\n" "$B$p"; done
curl -s -o /dev/null -w "%{http_code} → %{redirect_url}\n" http://three.ws/ # expect 301 → https
/api/healthz is designed to stay green through dependency outages — read its
JSON body (x402 block, monitor block) for subsystem truth.
What Vercel still holds (decommission checklist)
- The project + its env vars (plaintexts unreadable, but delete only after the ring secrets question is settled — the dashboard may still be useful to a human with owner access).
@vercel/ogremains a code dependency (runs fine on Cloud Run — plain Node).vercel.jsonis now a live config file consumed by server/index.mjs (routes + crons) — do NOT delete it as "Vercel leftovers".- The
deploynpm script still points at Vercel; superseded bydeploy:gcp.
Web domains
Agents register web domains through Google Cloud Domains and serve their public page on them. Product doc: docs/domains.md. Code: api/_lib/cloud-domains.js, api/_lib/domains-service.js, api/_lib/domain-connect.js, api/cron/domain-renewals.js.
- Identity. The API never calls Cloud Domains as its own runtime identity. It impersonates
three-ws-domains@aerial-vehicle-466722-p5.iam.gserviceaccount.comthrough IAM CredentialsgenerateAccessToken; the runtime account holdsroles/iam.serviceAccountTokenCreatoron that one account. No key exists for it. The account holdsroles/domains.admin,roles/dns.adminand the custom rolethreewsDomainConnect(compute.sslCertificates.create/delete/get/list,compute.targetHttpsProxies.get/setSslCertificates/use,compute.globalAddresses.get,compute.globalOperations.get).DOMAINS_ACCESS_TOKENoverrides it for workstation scripts only. - Quota.
domains.googleapis.comallows 300 sensitive requests per day and 100 per minute per project, and 20 registrations. The daily window resets about 07:00 UTC. A 3000/day increase was requested (preferencedomains-sensitive-per-day).GET /api/domains/quotashows the platform's approximate use. - TLD snapshot.
node scripts/refresh-domain-tlds.mjsregeneratesdata/domain-tlds.json(about 150 candidate TLDs probed serially, around 3 minutes and 150 quota calls). Run it right after a quota reset, then commit the file. - Connect procedure (what
connectDomainautomates): Cloud DNS managed zonetw-<sha12>withArecords for the apex andwwwat the named global addressthree-ws-ip(136.68.246.178);configureDnsSettingson the registration to the zone's nameservers; a Google-managed certificatetw-<sha12>created and attached tothree-ws-https-proxyby replacing its certificate list under a Redis lock; the host becomeslivewhen the certificate isACTIVE. The proxy holds at most 15 certificates, one of which isthree-ws-cert; past thatconnectanswerscert_capacity. Express routes alivehost's/to/agents/<id>(server/index.mjs). - Renewals.
/api/cron/domain-renewals(daily, invercel.json) debits credits 14 days before expiry and otherwise switches the name to manual renewal. After adding it, runnpm run deploy:gcp:sync-crons. - Rollback. Detach a certificate with
gcloud compute target-https-proxies update three-ws-https-proxy --ssl-certificates=<remaining list> --global; delete theweb_domain_hostsrow (or set statusfailed) to stop host routing.