Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

56.0 KB
Raw Blame

three.ws production on Google Cloud — operations runbook

This is the complete operational record of the production platform. If the machine that performed the migration is gone, this document + gcloud access to the project is everything needed to operate, deploy, debug, and recover the site. Written 2026-07-07, the day production moved off Vercel (Vercel disabled the deployment with 402 DEPLOYMENT_DISABLED; the site was dark until DNS cutover to Google Cloud the same day).

Related: server/README.md (the server code itself), STRUCTURE.md (surface map), docs/ops/gcp-model-workers.md (the 3D model workers, which were already on Cloud Run before this migration).


Architecture

Namecheap DNS (three.ws)
  A @   → 136.68.246.178      (global static IP: compute address "three-ws-ip")
  A www → 136.68.246.178
        │
Global External Application Load Balancer (EXTERNAL_MANAGED)
  :80  forwarding rule "three-ws-http"  → target-http-proxy  "three-ws-http-proxy"
        → url-map "three-ws-http-redirect" (301 → https)
  :443 forwarding rule "three-ws-https" → target-https-proxy "three-ws-https-proxy"
        → ssl-certificate "three-ws-cert" (Google-managed, three.ws + www.three.ws,
          auto-renews) → url-map "three-ws-lb"
        → backend-service "three-ws-backend" (Cloud CDN ON, cache mode
          USE_ORIGIN_HEADERS)
        → serverless NEG "three-ws-api-neg" (us-central1)
        │
Cloud Run service "three-ws-api" (us-central1, min 0, 2 vCPU / 2 Gi, timeout 900s,
allow-unauthenticated, runtime SA three-ws@…)
  └─ one Express container (server/index.mjs) serving:
       • static frontend from dist/ (baked into the image)
       • the vercel.json route table (1,285 rules as of 2026-09-01:
         headers/rewrites/redirects/404)
       • all api/** handlers with Vercel-parity filesystem routing
        │
Data layer (unchanged by the migration — all vendor-neutral HTTP):
  Neon Postgres (DATABASE_URL) · Upstash Redis · Cloudflare R2 (S3_* / R2_*)
Cloud Scheduler: one job per vercel.json cron (that array is the count) → GET /api/cron/* with
  `Authorization: Bearer $CRON_SECRET`
  • GCP project: aerial-vehicle-466722-p5 (billing account "Sperax", 01B467-A61905-9A97D2 — the $100k credits pool; owner-confirmed 2026-07-07).
  • Region: us-central1 for everything (same as the model workers).
  • Image: us-central1-docker.pkg.dev/aerial-vehicle-466722-p5/cloud-run-source-deploy/three-ws-api:latest Built from the repo-root Dockerfile. It runs as the unprivileged node user (not root) and prunes devDependencies after the SDK build (npm prune --omit=dev), so the runtime image carries only production deps. The container never writes to /app at runtime — the persona/ledger disk fallbacks and headless-chromium download both target /tmp — so the root-owned, read-only app tree is expected. Still single-stage: the python3/make/g++ toolchain stays because cloudbuild.yaml's inline-cache path (BUILDKIT_INLINE_CACHE=1 + --cache-from) only restores the final stage's layers, so a multi-stage split would forfeit the cached npm ci and regress rebuilds from ~3 min to ~12 min. A future move to docker buildx with a registry cache would unlock the multi-stage drop of the toolchain.

Production database (verified 2026-07-07)

The DB is Neon Postgres, not hosted on Vercel — the migration did not and could not "move" it; Cloud Run connects to the same instance via DATABASE_URL.

  • Production DB: Neon host ep-muddy-morning-af1v1xpa-pooler.c-2.us-west-2.aws.neon.tech — the live one, verified holding 344 tables / ~676k rows / 13.7k users / 13.3k avatars. This is the value DATABASE_URL carries on the Cloud Run service (and it was among the few vars the Vercel pull returned non-empty).
  • NOT production: ep-rapid-surf-ak9p7occ (Neon project wild-river-11025097) appears in old env archives — it is a stale/smaller copy (85 tables / ~127k rows / 224 avatars). Do not point production at it.
  • ⚠️ DECOMMISSION HAZARD (confirmed 2026-07-08): this Neon project was provisioned via Vercel's Postgres integration. If it's the Vercel-Managed Integration (billing inside Vercel), Neon's own docs state that deleting the database from Vercel's interface removes the underlying Neon project permanently, and Neon does not offer a self-serve transfer for Vercel-integrated projects ("projects with Vercel integrations cannot be transferred" — neon.com/docs/manage/orgs-project-transfer). There is no dashboard "claim it" button — do not assume one exists. Two real paths before the Vercel account is closed:
    1. Open a Neon support ticket asking to detach/transfer ep-muddy-morning into a standalone (non-Vercel) Neon org — some users have gotten this handled manually by support even though it isn't self-serve.
    2. Cut over independently: provision a fresh standalone Postgres (Neon-native org or Cloud SQL), restore into it, swap DATABASE_URL on the Cloud Run service, verify, then let the old Vercel-linked project go.
    • Safety net taken 2026-07-08: full pg_dump -Fc of neondb (344 tables, 673 MB live / 47 MB compressed) uploaded to Cloudflare R2 at s3://chatty-storage/db-backups/neondb-2026-07-08.dump — durable, off both Neon and Vercel. This is a point-in-time backup for disaster recovery, not a replacement for resolving ownership above; retention/ rotation isn't automated yet.

Service accounts (IMPORTANT — the project's default compute SA was deleted)

Every build and deploy MUST pin these explicitly or it fails with "service account does not exist":

  • Build SA: [email protected] (pass via --build-service-account / serviceAccount: in cloudbuild.yaml)
  • Runtime SA: [email protected] (pass via --service-account on every gcloud run deploy)

Resolved (verified 2026-07-25): the build SA now holds roles/run.admin (plus cloudbuild.builds.builder), and gcloud builds submit runs the full build+push+deploy pipeline end to end. Recent deploys (revisions 00272, 00276) were created by exactly this path, so no human-authed CLI finish is needed anymore. If a future submit fails at the deploy step with a permissions error, re-check this binding first: gcloud projects get-iam-policy aerial-vehicle-466722-p5 --filter="bindings.members:three-ws-build@".


Deploying

# Frontend changed? Build first — dist/ ships from the local build.
# build:gcp = check:conflicts -> check:browser-graph -> check:tdz-bootstrap
# -> ensure:avatar-studio -> build:info:snapshot -> build:lib:full (the agent-3d
# UMD lib) -> build:avatar-sdk -> build:chat -> build (the site `vite build`)
# -> publish:lib -> build:info -> check:dist -> check:pages, IN THAT ORDER. The
# order is load-bearing: build:lib:full and build:avatar-sdk must run BEFORE the
# site build (it resolves avatar-sdk/src/agent.js against the copied bundle), and
# the site `vite build` WIPES dist/, so it must run before publish:lib mirrors the
# lib into dist/. Plain `npm run build` is NOT enough (skips the lib publish, so
# /agent-3d/latest/agent-3d.js 404s and the hero avatar dies). Hand-running
# `build:vercel` as the frontend build is also WRONG — it builds the SDK/lib
# sub-artifacts, not the static HTML pages, so dist/ has no /, /create, etc.
npm run build:gcp

# Build image on Cloud Build (32-vCPU + BuildKit layer cache) + push + deploy.
# Gated on check:dist, check:pages AND db:check so an incomplete dist/, an
# unreachable page, or an out-of-date database can no longer ship. After the
# submit it runs deploy:gcp:sync-crons, which creates a Cloud Scheduler job for
# any cron declared in vercel.json that has none yet (existing jobs are never
# touched), so a new cron starts firing with the code that serves it. Ends with
# a CDN purge and then smoke:prod against the live site.
npm run deploy:gcp

# Or do build + submit + CDN purge in one (from a clean worktree):
#   npm run deploy:gcp:full

# Verify the deploy actually landed — /api/version reports the live commit SHA
# and Cloud Run revision (stamped into dist/build-info.json by build:info):
curl -s https://three.ws/api/version | jq '{commitShort, branch, builtAt, revision: .runtime.revision}'

# Sweep every page declared in data/pages.json against the live site. deploy:gcp
# runs this automatically after the purge; run it standalone any time.
npm run smoke:prod

# Tell Bing, Yandex, Seznam and Naver about pages added in the last 7 days so
# they are indexed in minutes. Run only after the deploy is live (IndexNow
# fetches every URL it is given). --days N, --since YYYY-MM-DD, --all, --dry-run.
npm run seo:indexnow

A fresh deploy worktree needs THREE staged artifacts, not just node_modules. build:gcp shells into two nested projects that carry their own dependency trees, and neither is in git. Hardlinking only the root node_modules leaves both broken, and each one fails several minutes into the build (2026-07-26: two consecutive dead builds, one exit 144, one exit 1):

npm run clean:worktrees -- --apply   # reclaim old deploy trees before staging a new one
npm run prep:worktree                # plan only: shows what it would stage or build
npm run prep:worktree -- --apply     # stage /workspaces/.deploy-wt, artifacts and env included

prep:worktree is the supported way to stage a deploy tree: it is the sequence below plus the checks a hand-run misses. Use --path <dir> to stage your own tree when a concurrent agent holds the default, and --force to replace an existing tree (refused when that tree holds uncommitted work). It BUILDS a nested artifact that is absent from the source tree instead of dying on the cp -al, which is what a machine that has never deployed hits first. It copies both env files rather than hardlinking them, and it rejects a --path on another filesystem up front, because cp -al cannot hardlink across a device boundary and leaves the tree half-staged when it tries. The equivalent by hand, for reference:

git worktree add --detach /workspaces/.deploy-wt HEAD
cp -al /workspaces/three.ws/node_modules                /workspaces/.deploy-wt/node_modules
cp -al /workspaces/three.ws/chat/node_modules           /workspaces/.deploy-wt/chat/node_modules
cp -al /workspaces/three.ws/character-studio/build      /workspaces/.deploy-wt/character-studio/build
cp    /workspaces/three.ws/.env                         /workspaces/.deploy-wt/.env
cp    /workspaces/three.ws/.env.local                   /workspaces/.deploy-wt/.env.local

Both env files, not just .env. DATABASE_URL lives in .env.local, and it is what the deploy's migration gate reads; a worktree without it fails db:check with DATABASE_URL is not set and tempts whoever is deploying to skip the gate entirely.

Remove the worktree once the deploy lands (git worktree remove --force /workspaces/.deploy-wt). Nobody did for three days in early August 2026 and ten trees filled the 126 GB disk to 100%. The failure does not name its cause: the next git worktree add dies mid-checkout with No space left on device and leaves a half-written tree, and the build after that fails somewhere unrelated. npm run clean:worktrees (plan only; --apply acts) reclaims any linked worktree that is detached, clean, and idle past --min-age-hours (default 2, so a concurrent agent's running build is never deleted). A tree holding uncommitted work is always reported and kept.

  • character-studio/build — ensure:avatar-studio only skips the heavy avatar-studio vite build when character-studio/build/index.html exists. Without it the build runs for real and gets OOM-killed (exit 144) on a box already hosting concurrent agent builds.
  • chat/node_modules — build:chat runs chat/scripts/ensure-deps.mjs, whose npm install can finish "successfully" while still leaving @sveltejs/vite-plugin-svelte unresolvable, so vite build dies with Cannot find package '.../@sveltejs/vite-plugin-svelte/index.js'.
  • cp -al (hardlink) only works on the SAME filesystem and is near-instant; plain cp -r of these trees costs minutes. If the destination directory already exists, cp -al src dst nests into dst/build — rm -rf the destination first.

The CDN purge is synchronous on purpose. deploy:gcp:purge-cdn used to pass --async, so anything verifying immediately afterwards read stale edge content and reported phantom failures — a cached pre-deploy 404 for a page that the new revision serves fine. Dropping --async costs ~3 s and makes every post-deploy check trustworthy. Do not add it back.

Never bypass check:dist by calling gcloud builds submit directly. It is the only thing standing between a half-built dist/ and production. When it reports MISSING: dist/agent-3d/latest/agent-3d.js, that is not a stale or pre-existing failure to work around — it means you ran plain npm run build and the agent-3d CDN lib was never published into dist/. Deploying anyway 404s the lib and kills the hero avatar on every page that embeds <agent-3d>. Fix it by running npm run build:gcp (or, if the site build already ran, npm run build:lib:full && npm run publish:lib) until check:dist is green. This happened on 2026-07-09 (revision three-ws-api-00034), where the check was overridden on the assumption its failure was unrelated to the change.

check:pages guards a different failure: a page that ships unreachable. Every entry in data/pages.json feeds the sitemap, llms.txt and features.json, so it is advertised to crawlers the moment it lands, whether or not anything can serve it. server/index.mjs's resolveStatic() has no .html extension fallback by design, so a page built to dist/<slug>.html is a hard 404 at /<slug> unless vercel.json carries a rewrite. That shipped twice in three days: /timeline (fixed by 5688277bd, "page shipped without a route entry") and /tracker (advertised 2026-07-23, 404 until a route landed two days later). check:pages resolves all ~760 declared paths through the real route table and fails the build on any that dead-end. When it fires it tells you which of the two causes applies: a missing vercel.json rewrite, or a page that was never built (missing vite.config.js input entry).

Database migrations live in api/_lib/migrations/*.sql and are tracked in the schema_migrations table (filename + sha256). Nothing applies them automatically — the flow is:

npm run db:status    # dry run: list applied/pending against DATABASE_URL
npm run db:migrate   # apply pending migrations (do this BEFORE deploy:gcp)
npm run db:check     # what deploy:gcp and deploy:gcp:submit run: exits 4 if anything is pending
  • DATABASE_URL comes from .env.local and must point at the production Neon DB (the same value the Cloud Run service uses).
  • Never edit a migration file after it has been applied — the runner detects the sha256 drift and refuses (exit 3). Roll forward with a new file.
  • Apply migrations before deploying the code that needs them: migrations are additive, so old code + new schema is safe; new code + old schema is not. When that order is inverted the symptom appears far from the cause: on 2026-08-14 /api/healthz reported the EVM agent index unknown with column "blocks_behind" does not exist, because the code that reads and writes those columns shipped while 20260813210000_erc8004_crawl_head_tracking.sql was still pending. Submit through npm run deploy:gcp:submit (or deploy:gcp), never a bare gcloud builds submit, so db:check gets its say.
  • db:migrate stops at the first migration that fails and reports every migration still queued behind it (exit 5). Those are unapplied: the schema is behind the code by exactly that list until the failing file is fixed with a NEW migration and the run drains the rest.

Dependency and import gates. deploy:gcp:submit runs two more checks after db:check, both added after the 2026-09-18 outage, when two merged dependency bumps took the paid x402 surface down on import:

npm run audit:deploy        # includes: installed node_modules must match package-lock.json
npm run check:api-imports   # imports every module under api/, fails on any that throws while loading
  • The image installs from package-lock.json, but local tests and every deploy worktree (which hardlinks node_modules) run whatever is on disk. If the two differ, a green npm test says nothing about what production will boot. When audit:deploy reports drift, run npm install, then npm test.

  • The server imports a handler the first time its route is hit, so a module that throws while loading is not a build error. It is a 500 on every request to that route. check:api-imports needs no environment and groups failures by error, so one broken shared module reads as one finding. Scope it with a path while debugging: node scripts/check-api-imports.mjs api/x402.

  • A dependency bump that changes an API is fixed in code or pinned back in package.json and held in .github/dependabot.yml. It is never waved through.

  • Lockfile unchanged → layer cache skips the workspace npm ci: ~3–5 min.

  • Lockfile changed → full install: ~12 min.

  • If the pipeline's deploy step fails on IAM (see above), deploy the pushed image directly:

gcloud run deploy three-ws-api \
  --image us-central1-docker.pkg.dev/aerial-vehicle-466722-p5/cloud-run-source-deploy/three-ws-api:latest \
  --project aerial-vehicle-466722-p5 --region us-central1 \
  --allow-unauthenticated --memory 2Gi --cpu 2 --timeout 900 \
  --service-account [email protected]

Rollback (instant):

gcloud run revisions list --service three-ws-api --region us-central1 --project aerial-vehicle-466722-p5
gcloud run services update-traffic three-ws-api --region us-central1 \
  --project aerial-vehicle-466722-p5 --to-revisions <good-revision>=100

Logs: use the reader CLI; it handles jsonPayload and request-log entries that a raw textPayload read silently misses. Full guide incl. the automated triage monitor: gcp-logs.md.

npm run logs                       # three-ws-api, last hour (vercel logs equivalent)
npm run logs:tail                  # live tail
npm run logs:errors                # ERROR+ across all services, last 6h
npm run triage:gcp                 # healthz + fleet log sweep -> classified action plan

Build-context gotcha: the upload is governed by the allowlist in .gcloudignore. If a handler starts reading a directory at runtime that isn't allowlisted, it will be missing from the image — add the directory there. animation-sources/ (2.6 GB) is deliberately excluded; the few endpoints reading it degrade until it moves to object storage.

Not yet created: a Cloud Build GitHub trigger (push-to-deploy like Vercel had). Requires connecting the nirholas/three.ws repo in Cloud Build and adding the vite build into the Docker build. This is Cloud Build's own GitHub integration — not GitHub Actions, which this repo does not use.

/ingest/* (PostHog proxy): on Vercel, vercel.json routes whose dest is an absolute URL (/ingest/static/* → us-assets.i.posthog.com, /ingest/* → us.i.posthog.com) were proxied by the platform itself. server/index.mjs now replicates this with its own external-dest middleware (mounted before the body parsers, so POST event bodies stream through unconsumed) — added 2026-07-07 after prod was serving 404s + a MIME-sniffing block on /ingest/static/array.js. If a future vercel.json route gets an http(s):// dest and analytics start 404ing again, check that this middleware's phase1Routes walk still matches it before the API/static phases.


Crons (Cloud Scheduler)

The schedules in vercel.json crons are mirrored to Cloud Scheduler jobs in us-central1, mostly named cron--api-cron-<name> (a few older ones use a bare name, so match by target URI, not by job name, when auditing). Vercel's crons died with the deployment, so there is no double-fire risk.

The mirror is not automatically enforced: verify it after editing crons. Declaring a cron in vercel.json does nothing on its own; only the sync script below creates the job. That gap has bitten repeatedly (on 2026-07-25 five declared crons had no scheduler job at all and had therefore never run in production: kol-tracker-refresh, forge-thumbnail-backfill, forge-off-crown, quality-bench, confirm-pending-purchases; again on 2026-08-14 for retention-rollup and likeness-eval), so audit with the two checks rather than by eye:

# Which declared crons are missing, paused, or on a different schedule than
# vercel.json says? Needs a live gcloud session; add --json for a report.
# Each MISSING row also says WHY, from production's own answer on that path:
#   "deployed, never synced" -> the handler is live and only the job is absent,
#                               so the sync below fixes it right now.
#   "handler not deployed"   -> the path 404s, so its job belongs to the deploy
#                               that ships the handler; syncing now would only
#                               schedule a 404 every run.
npm run check:cron-drift

# Do the handlers behind them route, load, run, and refuse an anonymous caller?
npm run audit:cron-liveness
# Sync after editing crons in vercel.json. Idempotent create-or-update, and
# CONFIG ONLY: it never changes whether an existing job is running, so it is safe
# to re-run during an incident hold. A job it creates starts ENABLED.
CRON_SECRET=<value> node scripts/create-gcp-scheduler.mjs

# Blanket run-state levers, explicit and mutually exclusive:
CRON_SECRET=<value> node scripts/create-gcp-scheduler.mjs --pause   # stop the whole fleet
CRON_SECRET=<value> node scripts/create-gcp-scheduler.mjs --resume  # recover from --pause

# List / pause / force-run one job. Note the DOUBLE hyphen after the `cron`
# prefix: the cron path's leading slash becomes a hyphen of its own, and a
# single-hyphen name fails with NOT_FOUND.
gcloud scheduler jobs list --location us-central1 --project aerial-vehicle-466722-p5
gcloud scheduler jobs pause  cron--api-cron-economy-tick --location us-central1 --project aerial-vehicle-466722-p5
gcloud scheduler jobs run    cron--api-cron-uptime-check --location us-central1 --project aerial-vehicle-466722-p5

CRON_SECRET: was EMPTY on Vercel (every cron guard fail-closed with 503 — Vercel crons had been silently dead). A real secret was generated 2026-07-07 and set in two places: the Cloud Run service env and every Scheduler job's Authorization header. Recover it any time from either place:

gcloud scheduler jobs describe cron--api-cron-uptime-check --location us-central1 \
  --project aerial-vehicle-466722-p5 --format="value(httpTarget.headers)"

Environment variables

All service env lives on the Cloud Run service (view/edit):

gcloud run services describe three-ws-api --region us-central1 \
  --project aerial-vehicle-466722-p5 --format=yaml | grep -A2 'name:'
gcloud run services update three-ws-api --region us-central1 \
  --project aerial-vehicle-466722-p5 --update-env-vars KEY=value

⚠️ THE MIGRATION TRAP — read this before trusting any env export

vercel env pull silently returns EMPTY values for secret-type vars — 144 of 173 keys pulled blank (only integration-injected vars like DATABASE_URL came through). The plaintexts are not retrievable from Vercel by CLI at all. Consequences and current state:

  1. The initial 157-var application contained mostly empty strings.
  2. The S3_* group + real values for ~90 keys were recovered from the owner's own env archives (owner-provided dump, 2026-07-07) and staged as an 89-var validated set (deduped, other-project keys pruned, JWKS verified). If the staging machine is lost before application, the set must be rebuilt from the owner's archives — the sources are the owner's saved env dumps; the selection rules are: skip VERCEL*/NX_*/TURBO_*/PG*/POSTGRES_*/ NEON_*/DATABASE_URL*/REDIS_URL, skip keys not referenced by this codebase, last-occurrence-wins on duplicates.
  3. Verify state empirically, not from dashboards: an endpoint returning 503 {"error":"not_configured"} means a Missing required env var: X line in the logs names the exact var.

Vars with no default that silently kill a whole subsystem

CIRCULATION_ENABLED=1 is the master gate for the circulation engine (api/_lib/circulation.js, driven by pulse-tick via economy-tick) — the engine that generates the Money Pulse's agent-to-agent activity. It is OFF by design when unset, so it does not show up as a 503 not_configured or any log error: every tick just reports a clean skipped: "disabled". This is exactly how the 2026-07-07 migration killed the pulse for ten days — the var existed only on Vercel, was never carried to Cloud Run, and nothing alerted. Restored 2026-07-17 (--update-env-vars CIRCULATION_ENABLED=1; the treasury secret CIRCULATION_TREASURY_SECRET already rides on Secret Manager wallet-sol-engines-b58). When auditing env completeness, check the economy section of /api/status for engines reporting disabled, not just 5xx responses.

Known-missing (blocked on owner)

What Vars Impact Recovery
Ops-alert Telegram push TELEGRAM_ALERTS_CHAT_ID api/_lib/alerts.js degrades to table-only (ops_alerts): no push notification for treasury-low, balance-floor, or breach alerts. This is why the 10-day Money Pulse flatline never pinged anyone. TELEGRAM_BOT_TOKEN survived; only the private ops chat id was lost in the migration. Owner supplies the private ops chat/DM id (never the public holders' channel) and applies --update-env-vars TELEGRAM_ALERTS_CHAT_ID=<id>.
Collection authority SOLANA_AGENT_COLLECTION_AUTHORITY_KEY Agent NFT collection ops can't sign Intentionally excluded from scripts/wire-master-wallet.mjs (on-chain update authority must stay its original wallet). Owner holds the key.
R2/S3 storage creds S3_ENDPOINT, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_BUCKET, S3_PUBLIC_DOMAIN /api/marketplace, /api/explore, /api/avatars/:id — every route that resolves an asset URL — 503 not_configured Real values already sit in the repo's .env.local (never committed). Apply with scripts/gcp/apply-s3-env.sh (needs a human-authed gcloud auth login first — the 89-var apply is still blocked on reauth per above).
CoinCommunities API key CC_API_KEY Town chat renders its designed locked state. /api/community/worlds and /api/clash/state + /api/clash/leaderboard no longer die: they fail over to the live pump.fun trending feed (api/_lib/pump-trending.js) and tag the response source: "pump-trending", so the world picker and the Clash arena stay alive (the Clash roster is frozen per round). The Play holders gate shows its unavailable state ("Holder check is offline") and offers a one-click path into the coin's open world, so a gated world is never a dead end. Keys are provisioned manually by the CoinCommunities team (their docs say "contact support": no self-serve signup, and the SDK's business/register endpoints are not deployed on api.coin-communities.xyz). Owner outreach to @CoinComms required; set CC_API_KEY (+ optional CC_SERVER_KEY/CC_SERVER_SECRET) on the Cloud Run service when granted.
Agent mail to and from the internet (not a var: DNS records for agents.three.ws) Mail between two agents.three.ws mailboxes works (delivered inside the platform). Mail to any outside address fails with send_failed (credits are refunded), and outside senders cannot reach an agent, because the mail provider reports the agents.three.ws domain as failed: none of its records exist at the DNS host. Checked 2026-10-10. In the Resend dashboard, open the agents.three.ws domain and add every record it lists at the DNS host for three.ws: the DKIM TXT at resend._domainkey.agents, the SPF MX and TXT at send.agents, the return-path CNAME, and the receiving MX at agents (priority 10). Then press Verify and point the domain's inbound webhook at https://three.ws/api/mail/inbound with the secret in RESEND_WEBHOOK_SECRET. See docs/agent-mail.md.
x402 ring payer USDC float — (not a var: an on-chain balance) The autonomous loop's payer wallet holds $0 USDC, so every paid call is skipped. Since 2026-07-09 this manifested as ~1 failed settle/minute (broadcast_failed:Simulation failed); the loop now skips paid calls instead. Free + monitoring entries are unaffected. Send USDC to the payer wallet (X402_SEED_SOLANA_SECRET_BASE58's pubkey). Floor is X402_RING_PAYER_USDC_FLOOR_ATOMIC (default $5) — fund meaningfully above it or it re-drains. See x402 ring payer float below.
x402 prepaid settler X402_PREPAID_SETTLER_SECRET_BASE58, X402_PREPAID_SETTLE_LIVE, X402_PREPAID_DEPOSIT_WALLET, X402_PREPAID_MIN_SETTLE_ATOMICS, X402_PREPAID_MAX_PAYEES_PER_TX Unset settler or X402_PREPAID_SETTLE_LIVE not 1: the settle cron only plans, committed debits accumulate unpaid. Owner action: fund the settler (SOL fees plus USDC float), review the planned window, then set the flag with --update-env-vars. See x402-prepaid.md.

x402 ring payer float — why an empty wallet used to storm Solana

The autonomous loop (api/cron/x402-autonomous-loop.js, driven every minute by economy-tick) spends real USDC against the platform's own paid endpoints. Two independent limits apply, and for a long time only one was enforced:

  • the daily spend cap (remainingCap) bounds what the loop is allowed to spend;
  • the payer's USDC balance bounds what it is able to spend.

With a drained float and cap headroom remaining, each tick probed for a 402, signed a payment transaction, and POSTed it — and the self-hosted facilitator failed every one at settle with broadcast_failed:Simulation failed. Observed: 1,665 failed settles against 3 successes over 24h. Nothing was lost (the transaction fails in simulation, before broadcast, so no fee is burned and no funds move), but the ring did no useful work and hammered the RPC indefinitely.

readPayerUsdcAtomic() is now read once per tick and treated as a hard spend ceiling. Its three-way contract is load-bearing, and tests/x402-autonomous-payer-float.test.js pins it:

Read result Meaning Loop behavior
a number > 0 real float pay up to min(dailyCap, float)
0 genuinely empty (including a wallet with no ATA) skip every paid call; free + run() monitors still execute
null balance undeterminable (RPC blip) do not gate — leave the spend path open

The null-vs-0 distinction matters in both directions: if a transient RPC failure read as 0, one blip would silently halt the ring; if an empty wallet read as null, the doomed-payment storm returns. Emptiness is detected structurally (getAccountInfo → null ⇒ no token account ⇒ no USDC), not by pattern-matching an RPC error string, whose wording varies by provider.

To resume autonomous spend, fund the payer's USDC float. The loop self-recovers on the next tick — no redeploy, no flag flip.

The sponsor's SOL floor gate has the opposite default. Its balance read can also come back null, and until 2026-08-28 that read as "not paused": all four paid RPC lanes were over quota at the moment the sponsor sat at 0.000899 SOL, so the loop spent three hours signing payments that could not settle (95 attempts, 0 settled). An unreadable balance is not evidence of solvency, so a null read now defers to sponsorKnownBelowFloor() (api/_lib/x402/self-facilitator.js), which the settle path itself writes when the chain rejects the sponsor for rent-exemption. That second opinion costs no RPC call, so the gate still answers when the RPC does not.

Ops alerts: dashboard-primary (2026-07-12)

Ops alerting used to be Telegram-only, and Telegram-only meant that with no chat configured every sendOpsAlert() was a silent no-op — 5xx faults, api/client-errors.js browser reports, ring-leak CRITICALs, and low-balance warnings all dropped. That is why the x402 payment storm below ran for a day unnoticed.

Now every sendOpsAlert() always persists to the ops_alerts table (api/_lib/alerts.js, migration 20260712000000_ops_alerts.sql), keyed by the alert's stable signature so a recurring condition is one row with a growing count, not a flood. Read the active feed from the ops_alerts table or the x-ops-secret-gated ops APIs (/api/ops/health and friends; the gate is authorizeOps in api/_lib/ops-auth.js: admin session or a dedicated OPS_SECRET, deliberately never CRON_SECRET, fail-closed in production). This is the primary sink and needs no third-party service.

Telegram is now an optional extra push, off by default. When both TELEGRAM_BOT_TOKEN and TELEGRAM_ALERTS_CHAT_ID are set, alerts also push to that private chat (deduped per signature for 1h, capped at 20/h). It is deliberately unset in production — the dashboard is the channel. /api/healthz reports alerts.primary: "ops_alerts" and alerts.telegram_push: "disabled"|"enabled"; a disabled push is expected, not a fault. The @threewsbot token remains in Secret Manager (telegram-bot-token) for user-facing bot use (e.g. changelog push to holders); it is not wired to ops alerts. To turn the extra push on, add @threewsbot to a private ops chat, read its id from getUpdates, and set TELEGRAM_ALERTS_CHAT_ID (+ point TELEGRAM_BOT_TOKEN at the secret).

Resolved: x402 sponsor co-signing key (2026-07-09)

/api/healthz reported x402.sponsor_cosign: "missing" and every sponsor-mode settlement (club-cover, dance-tip) failed with sponsor_key_unconfigured.

Root cause: X402_FEE_PAYER_SOLANA advertised 2wKupLR9q6wXYppw8Gr2NvWxKBUqm4PPJKkQfoxHDBg4 — a third-party facilitator's shared fee-payer account that three.ws has never controlled, left over from when Solana settlement went through an external facilitator (see ARCHITECTURE.md). The self-hosted facilitator cannot co-sign for a key it does not hold, so it 502'd at settle while passing every other health check.

An earlier revision of the table above described that address as a three.ws ring wallet that "custodies funds — do NOT regenerate." That was wrong, and it would have blocked the correct fix indefinitely. Evidence: the account fee-pays ~200 tx/hour for many unrelated co-signer wallets, none of them involving our payTo, while our own cosign was provably unconfigured. Its balance was never ours. scripts/audit-service-wallets.mjs has always flagged it correctly and told operators to override the var.

Fix applied:

  • Generated a three.ws-controlled sponsor with scripts/x402-ring-setup.mjs --roles=sponsor (sponsor role only — the funded payTo treasury was never touched): GGf9qBhJDCe1UUz4s4Vxq1uPPvcv7UW7sJTuj2Yo5XQj.
  • Stored the secret in Secret Manager as x402-fee-payer-secret-base58 (granted roles/secretmanager.secretAccessor to three-ws@…) and wired it as a secretKeyRef, rather than a plaintext env value on the service. This also removes the "secret only exists on whatever machine ran the script" fragility.
  • Funded it with 0.05 SOL (floors: X402_SPONSOR_SOL_FLOOR_LAMPORTS = 0.02 hard refuse-to-settle, 0.03 audit/topup).
gcloud run services update three-ws-api --region us-central1 \
  --update-env-vars X402_FEE_PAYER_SOLANA=GGf9qBhJDCe1UUz4s4Vxq1uPPvcv7UW7sJTuj2Yo5XQj \
  --update-secrets X402_FEE_PAYER_SECRET_BASE58=x402-fee-payer-secret-base58:latest

Verified by a real $0.01 USDC mainnet sponsor-mode settlement through POST /api/x402-facilitator/settle (tx 5rmLnLUmWK7jJCjP6RJPLmwT2oiifQMmTnQmhoQKeZcVogxnTMvxB77fpb5BqMyV6VzbawXdPgWz2N7C5kXqqbQ5): the sponsor signed as fee payer, paid the 10001-lamport fee, and $0.01 landed in payTo. sponsor_cosign now reports ready.

Resolved: Upstash Redis + full signer wiring (2026-07-09, same day, later)

Owner supplied the Upstash database and the wallet secrets. Applied on revision three-ws-api-00029-c45:

  • Upstash Redis live: UPSTASH_REDIS_REST_URL (plaintext) + UPSTASH_REDIS_REST_TOKEN (Secret Manager upstash-redis-rest-token). X402_ALLOW_MEMORY_FALLBACK removed per the escape-hatch's own instructions. healthz cache backend now reports upstash healthy; money limiters rate-limit for real instead of failing closed.
  • Signer slots wired per scripts/wire-master-wallet.mjs ASSIGNMENTS, from the three economy wallets (wwwww…ccrU x402 loop, wwwqv…HGUn SOL engines, WwwuGbq…3WwW treasury face). Secrets live in Secret Manager (wallet-x402-treasury-b58/b64, wallet-sol-engines-b58/b64, wallet-economy-master-b58/b64) as secretKeyRefs — no longer only on one machine. ECONOMY_MASTER_SECRET_BASE58, X402_TREASURY_SECRET_BASE58, X402_SEED_SOLANA_SECRET_BASE58, A2A_PAYER_SOLANA_SECRET were concurrently set (same wallets) by another session and left as-is. LAUNCHER_MASTER_SECRET_KEY_B64 deviates from the map (deployed on X4o2…astML, map says wwwqv) — left deployed value; LABOR_ESCROW skipped per the script's own live-funds guard.
  • Proof, full user path (not just the facilitator): real paid calls through the live endpoints returned 200 + product content — club-cover $0.01 (tx 54FQ4rAFjfgF9b5pDRCB5dnhWNoTK7RrvJm2EJAuNGQDiMAPcLGptXpWzdBys49KHEASH7JrKb8sHL3AFwZT36gF, pass issued) and dance-tip $0.001 (tx 2hSvvR17hpawE1ZjhgHisaEpRzYkh6jyWtWBgsoCKAhunQkhcUCfQQ49aY2QmfsCYNBaMpvVTxSBfky5t7tstUZ6, thriller ticket). scripts/audit-service-wallets.mjs against the live revision: all checked wallets configured, funded, and consistent (14/15 signers; only the intentionally-excluded collection authority remains).

Resolved: self-hosted Memorystore Redis, off the Upstash free tier (2026-07-10)

The Upstash free database (saving-titmouse-110599) hit its hard 500k commands/month cap within hours — that ceiling is far too small for this platform (the rate limiter alone spends 2–3 commands per request). Redis then errored on every command, the cache circuit opened, and paid endpoints were only saved by the durable Postgres rate-limit fallback (commit a1b0b9610). The free tier does not reset until the 1st, so the fix was to stand up our own Redis on the GCP credit pool — permanently ending the third-party quota dependency.

Architecture (all in aerial-vehicle-466722-p5, us-central1; app code UNCHANGED — it still speaks the @upstash/redis REST dialect):

three-ws-api (Cloud Run)
  ├─ VPC connector "three-ws-vpc" (10.8.0.0/28), egress private-ranges-only
  │    → only RFC1918 dests traverse the VPC; all external calls (RPC, LLM
  │      APIs) stay on the direct public path (no Cloud NAT needed, no egress
  │      bottleneck).
  └─ UPSTASH_REDIS_REST_URL = http://10.128.15.228   (internal LB IP)
     UPSTASH_REDIS_REST_TOKEN = secret:srh-proxy-token
        │
Regional internal Application LB (10.128.15.228:80)
  forwarding-rule "three-ws-redis-fr" → target-http-proxy "three-ws-redis-tproxy"
  → url-map "three-ws-redis-urlmap" → backend "three-ws-redis-be"
  → serverless NEG "three-ws-redis-neg"
  (proxy-only subnet "three-ws-proxy-only" 10.100.0.0/23; firewall
   "three-ws-allow-proxy-subnet" + "three-ws-allow-connector-to-ilb")
        │
Cloud Run "three-ws-redis-proxy" (hiett/serverless-redis-http:0.0.10, pinned;
  ingress internal-and-cloud-load-balancing; SRH_TOKEN=secret:srh-proxy-token;
  SRH_CONNECTION_STRING=redis://10.234.231.139:6379; on VPC connector three-ws-vpc)
        │
Memorystore Redis "three-ws-redis" (basic, 1 GB, redis_7_0) @ 10.234.231.139:6379

Why this shape: the org blocks allUsers invocation (domain-restricted sharing), so a public REST proxy is unreachable (its SRH bearer token collides with the Google auth token Cloud Run would demand). Fronting the proxy with a regional internal LB and reaching it over the VPC connector keeps the proxy off the public internet AND keeps the app's external egress on the fast direct path. The connector + Redis env are pinned in server/cloudbuild.yaml so a full-config redeploy can never silently drop Redis back to the capped Upstash tier.

Verified 2026-07-10 on revision three-ws-api-00052-kbg: /api/healthz overall ok, cache backend healthy (the client self-labels "upstash" — it is our Memorystore); proxy access logs show a steady stream of POST 200 /pipeline commands; the old Upstash endpoint stayed capped and untouched (proving the app moved off it); and a real $0.01 club-cover settlement completed end to end (tx 4fWXfymUxwS7YRMrkQqHwwgtA1GiEAmnJXsgYaXHo5NvyAqhisGkmU3k7waLD85RsVx5s4mXmB4ymiTuEvzAWnQ8).

Cost ≈ $75/mo on the credit pool (Memorystore ~$35, proxy min-1 ~$13, connector ~$9, internal LB ~$18). To retire: repoint UPSTASH_REDIS_REST_URL/TOKEN at a managed Upstash paid plan and delete the four three-ws-redis-* LB resources + proxy + Memorystore + connector. The upstash-redis-rest-token secret and the saving-titmouse free DB are now unused (kept for rollback).

What Vars Impact Recovery
OKX X Layer facilitator creds OKX_API_KEY, OKX_SECRET_KEY, OKX_PASSPHRASE (or, as a no-OKX-account fallback, X402_XLAYER_RELAYER_KEY) xlayerSettleable() (api/_lib/x402-xlayer-okx.js) is false without one of these, so no 402 challenge on any endpoint advertises the eip155:196 (X Layer) rail, confirmed live 2026-07-08 via /api/okx/3d/health (payment-rail.settleable:false, facilitator_configured:false) and by inspecting the raw 402 accepts[] on /api/okx/3d/identity-studio and /api/okx/3d/pose-seed (Solana + Base only, no X Layer entry). X402_PAY_TO_XLAYER and X402_ASSET_ADDRESS_XLAYER are set in prod, so this is the only missing piece. This is the exact rail OKX's listing review requires (agent #2632 rejection reason). The credential has since been set: probed live 2026-09-09, all four paid Forge rows advertise eip155:196 and OKX's own agent x402-check reads valid: true on each. Treat this row as the recovery path if that ever regresses; the relisting order is prompts/finish/911-okx-ai-08-forge-relisting.md. OKX Web3 developer console (owner) for the three real creds, or generate X402_XLAYER_RELAYER_KEY as a fresh EVM keypair + fund it with OKB gas as a stopgap that needs no OKX account.

DNS / TLS / CDN

  • Registrar + DNS: Namecheap (nameservers registrar-servers.com). Web records: A @ → 136.68.246.178, A www → 136.68.246.178. Everything else there (privy/world CNAMEs, TXT/DKIM) is mail/verification — untouched by the migration.
  • TLS: Google-managed cert three-ws-cert (three.ws, www.three.ws) — went ACTIVE 2026-07-07 ~18:40 UTC, auto-renews. Status: gcloud compute ssl-certificates describe three-ws-cert --global --project aerial-vehicle-466722-p5
  • CDN: Cloud CDN on three-ws-backend, USE_ORIGIN_HEADERS — the vercel.json asset rule (7-day cache on glb/png/woff2/…) is what the CDN honors.
  • The Cloud Run URL https://three-ws-api-lp642k3kpa-uc.a.run.app serves the identical site and bypasses the LB — useful for isolating LB vs service issues.
  • www.three.ws is a duplicate host, not a second origin. Its A record and the managed cert both cover it, so it reaches the same backend, but the apex is the only origin the platform declares: server/seo-head.mjs pins every canonical URL to https://three.ws and the media bucket's CORS read rule allowlists that origin alone. Measured 2026-09-09 on one bucket object: the apex gets Access-Control-Allow-Origin: https://three.ws back, www gets no header at all, so every GLB fetch on a www page was blocked by the browser. server/index.mjs now redirects the whole www host to the apex (301 on GET/HEAD, 308 on everything else so a write keeps its method and body), and tests/server-canonical-host.test.js locks in that no other host, including the *.run.app URL above and dev.three.ws, is touched. Do not add a second serving hostname without either widening the bucket policy to match or redirecting it here too.

A truncated asset can sit in the CDN and hang every browser (2026-09-11)

Cloud CDN held one copy of /vendor/mediapipe/wasm/vision_wasm_internal.wasm that carried content-length: 11153617 (the identity length) with a brotli-compressed body and no content-encoding header. A client that reads that response waits forever for eight megabytes that never arrive.

It only reproduced with the exact Accept-Encoding a browser sends:

# healthy
curl -s -o /dev/null -w '%{size_download} %{time_total}\n' \
  -H 'Accept-Encoding: br' https://three.ws/vendor/mediapipe/wasm/vision_wasm_internal.wasm

# the bug: bytes stop arriving, the request never completes
curl -s -o /dev/null -w '%{size_download} %{time_total}\n' \
  -H 'Accept-Encoding: gzip, deflate, br, zstd' \
  https://three.ws/vendor/mediapipe/wasm/vision_wasm_internal.wasm

The Cloud Run origin served the same URL correctly under every encoding, so the bad variant was created and cached at the edge, not by the container. A cache-busting query string also served correctly, which is how you tell a poisoned object from a broken build.

Symptom at the product level: /create/selfie sat on "Processing your face..." indefinitely and never sent the reconstruction request, because the capture gates await MediaPipe and MediaPipe's loader has no deadline. Nothing errored, nothing retried, and no page check caught it: a page sweep reads HTML, not the largest binary on the site.

Recovery, one path, not a global purge:

gcloud compute url-maps invalidate-cdn-cache three-ws-lb \
  --path '/vendor/mediapipe/wasm/vision_wasm_internal.wasm' \
  --project aerial-vehicle-466722-p5

Detection: npm run check:asset-encoding walks the largest compressible files under public/, requests each from the live site with a browser's Accept-Encoding, reads every body to completion, and fails on a length that disagrees with the bytes delivered. Run it after a deploy; it is the only check that would have caught this.

Client-side hardening: src/shared/mediapipe-assets.js now prefers our own vendored copy unless the origin definitively lacks the file (a probe that merely times out no longer sends the runtime to a public CDN), shares one probe across all callers, and bounds every vision-model load with withVisionDeadline. A stalled load now degrades to the unrefined photo after 15 seconds instead of hanging the flow. npm run verify:selfie -- --stall-vision reproduces the hostile case end to end.

A rejected object-storage credential silently eats finished avatars (2026-09-11)

The R2 credential stopped verifying at 03:15 UTC on 2026-09-11, the second such outage in three days (the first ran 05:04-05:19 on 2026-09-09 and was fixed by rotating the secret). Every signed operation answered SignatureDoesNotMatch, for reads and writes alike.

Confirm it in one command, against whatever the service is actually running (the secret is a Secret Manager reference, so describe shows valueFrom):

export S3_ENDPOINT=$(node scripts/read-service-env.mjs '^S3_ENDPOINT$' --raw)
export S3_BUCKET=$(node scripts/read-service-env.mjs '^S3_BUCKET$' --raw)
export S3_PUBLIC_DOMAIN=$(node scripts/read-service-env.mjs '^S3_PUBLIC_DOMAIN$' --raw)
export S3_ACCESS_KEY_ID=$(node scripts/read-service-env.mjs '^S3_ACCESS_KEY_ID$' --raw)
export S3_SECRET_ACCESS_KEY=$(node scripts/read-service-env.mjs '^S3_SECRET_ACCESS_KEY$' --raw)
node -e "const {putObject}=await import('./api/_lib/r2.js');
await putObject({key:'forge/refs/probe.png',body:Buffer.from('89504e470d0a1a0a','hex'),contentType:'image/png'});
console.log('write ok')"

Count the blast radius, which is what tells a credential fault from a code bug:

gcloud logging read 'resource.type="cloud_run_revision"
  resource.labels.service_name="three-ws-api" textPayload:"SignatureDoesNotMatch"' \
  --freshness=12h --limit=1000 --format='value(timestamp)' \
  --project aerial-vehicle-466722-p5 | cut -c1-13 | sort | uniq -c

A clean start hour with nothing before it is a credential that died at that minute. Failures spread evenly across many hours are a different fault.

Symptom at the product level: /create/selfie reconstructed the mesh on the GPU, took ~35 s, and then said "Avatar finished but could not be saved. Try again." Retrying could not work, because the prompt was never the problem. The job row is the ground truth, and it logs nothing:

select job_id, status, result_avatar_id, error from avatar_regen_jobs
order by created_at desc limit 5;
-- status 'done', result_avatar_id NULL, error NULL

error stays NULL on purpose: /api/cron/reconstruct-sweep recovers exactly the rows matching status = 'done' and error is null, so writing the failure there would strand the job instead of retrying it. Do not "fix" that NULL.

Why it is no longer fatal. api/_lib/reconstruct-finalize.js treats a storage-infrastructure fault (isStorageInfrastructureError: credential rejected or revoked, bucket missing, endpoint unreachable) as a reason to lose the durable copy, never the reconstruction. The mesh is already public and lifecycle-free in gs://three-ws-avatar-reconstructions, so that URL becomes the avatar's storage key: publicUrl() passes an absolute key through, defaultStorageMode() records r2.present = false, and source_meta carries servedFrom: 'provider' plus the pendingBucketKey a later re-copy needs. Any other fault still throws. Same trade api/_lib/image-persist.js makes for reference images.

POST /api/v1/ai/image cannot make that trade (its contract is a durable https URL a third party can fetch), so it answers 503 storage_unavailable rather than the old opaque 502 generation_failed, which told callers to retry a good prompt and told operators nothing.

Real recovery is a new R2 API token, and it is owner-gated: no Cloudflare API token exists on this machine or on the Cloud Run service, so a token cannot be minted from here. Mint one in the Cloudflare dashboard (R2 > Manage API tokens, Object Read & Write on the bucket in S3_BUCKET), then:

printf %s "<new secret>" | gcloud secrets versions add s3-secret-access-key \
  --data-file=- --project aerial-vehicle-466722-p5
# only if the access key id changed too (--update-env-vars MERGES; --set-env-vars REPLACES)
gcloud run services update three-ws-api --region us-central1 \
  --update-env-vars S3_ACCESS_KEY_ID=<new id> --project aerial-vehicle-466722-p5

api/_lib/r2.js re-probes a rejected credential every 15 s, so a fixed token recovers without a redeploy. Avatars written during the outage keep serving from GCS; find them with storage_mode->'r2'->>'present' = 'false'.

Client geo header (analytics country column)

api/_lib/client-geo.js resolves a visitor's country from an edge geo header and never from the raw IP. On Vercel that header was x-vercel-ip-country, set by the edge for free. The GCLB sets no geo header by default, so after the 2026-07-07 migration widget_views.country and widget_chat_threads.country went null for every row (verified: 49/49 populated in May, 0/27 in August), and the top_countries panel on /api/widgets/:id/stats renders empty for every widget.

The reader now prefers x-client-geo-location, then x-client-region. Populate one on the backend service to restore capture (owner-gated: this is a production LB change, not a Cloud Run config update):

gcloud compute backend-services update three-ws-backend --global \
  --project aerial-vehicle-466722-p5 \
  --custom-request-header='X-Client-Region:{client_region}'

{client_region} is the GCLB variable for the ISO 3166-1 alpha-2 code of the client. Confirm with gcloud compute backend-services describe three-ws-backend --global --format='value(customRequestHeaders)', then check capture:

select count(*) filter (where country is not null), count(*)
from widget_views where created_at > now() - interval '1 day';

Until that header exists the code degrades to null rather than guessing, which is the intended behavior. Note the LB does not strip an inbound copy of these headers, so a client can forge one; clientCountry validates every value down to two ASCII letters so the worst case is a visitor mislabeling its own row.


Verifying after any change

B=https://three.ws
for p in / /3d /changelog /api/healthz /api/feed /api/explore?limit=4; do
  curl -s -m 25 -o /dev/null -w "%{http_code}  $p\n" "$B$p"; done
curl -s -o /dev/null -w "%{http_code} → %{redirect_url}\n" http://three.ws/   # expect 301 → https

/api/healthz is designed to stay green through dependency outages — read its JSON body (x402 block, monitor block) for subsystem truth.


What Vercel still holds (decommission checklist)

  • The project + its env vars (plaintexts unreadable, but delete only after the ring secrets question is settled — the dashboard may still be useful to a human with owner access).
  • @vercel/og remains a code dependency (runs fine on Cloud Run — plain Node).
  • vercel.json is now a live config file consumed by server/index.mjs (routes + crons) — do NOT delete it as "Vercel leftovers".
  • The deploy npm script still points at Vercel; superseded by deploy:gcp.

Web domains

Agents register web domains through Google Cloud Domains and serve their public page on them. Product doc: docs/domains.md. Code: api/_lib/cloud-domains.js, api/_lib/domains-service.js, api/_lib/domain-connect.js, api/cron/domain-renewals.js.

  • Identity. The API never calls Cloud Domains as its own runtime identity. It impersonates three-ws-domains@aerial-vehicle-466722-p5.iam.gserviceaccount.com through IAM Credentials generateAccessToken; the runtime account holds roles/iam.serviceAccountTokenCreator on that one account. No key exists for it. The account holds roles/domains.admin, roles/dns.admin and the custom role threewsDomainConnect (compute.sslCertificates.create/delete/get/list, compute.targetHttpsProxies.get/setSslCertificates/use, compute.globalAddresses.get, compute.globalOperations.get). DOMAINS_ACCESS_TOKEN overrides it for workstation scripts only.
  • Quota. domains.googleapis.com allows 300 sensitive requests per day and 100 per minute per project, and 20 registrations. The daily window resets about 07:00 UTC. A 3000/day increase was requested (preference domains-sensitive-per-day). GET /api/domains/quota shows the platform's approximate use.
  • TLD snapshot. node scripts/refresh-domain-tlds.mjs regenerates data/domain-tlds.json (about 150 candidate TLDs probed serially, around 3 minutes and 150 quota calls). Run it right after a quota reset, then commit the file.
  • Connect procedure (what connectDomain automates): Cloud DNS managed zone tw-<sha12> with A records for the apex and www at the named global address three-ws-ip (136.68.246.178); configureDnsSettings on the registration to the zone's nameservers; a Google-managed certificate tw-<sha12> created and attached to three-ws-https-proxy by replacing its certificate list under a Redis lock; the host becomes live when the certificate is ACTIVE. The proxy holds at most 15 certificates, one of which is three-ws-cert; past that connect answers cert_capacity. Express routes a live host's / to /agents/<id> (server/index.mjs).
  • Renewals. /api/cron/domain-renewals (daily, in vercel.json) debits credits 14 days before expiry and otherwise switches the name to manual renewal. After adding it, run npm run deploy:gcp:sync-crons.
  • Rollback. Detach a certificate with gcloud compute target-https-proxies update three-ws-https-proxy --ssl-certificates=<remaining list> --global; delete the web_domain_hosts row (or set status failed) to stop host routing.