Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
LIVE-OPS: $THREE Community Day
One page. Everything an operator needs while the event is running, and nothing else.
Prepared 2026-08-08. Project aerial-vehicle-466722-p5, region us-central1.
The link and the release
Event URL (the canonical $THREE world, from README.md):
https://three.ws/play?coin=FeMbDoX7R1Psc4GEcvJdsbNbZA3bfztcyDCatJVJpump&name=three.ws&symbol=three&image=%2Fapi%2Fimg%3Furl%3Dhttps%253A%252F%252Fipfs.io%252Fipfs%252Fbafybeihe22b5sxr3ihnxt7pregfieyteqvubqhik3j3y4bbx243xlqjw3q%26seed%3DFeMbDoX7R1Psc4GEcvJdsbNbZA3bfztcyDCatJVJpump
Live release as of this writing: c1e600a04 on revision three-ws-api-00364-n65.
The world server runs an image built 2026-08-06 (revision three-ws-multiplayer-00010-g89).
The release SHA above is NOT the event build. The event features (quest line, photo mode, live event board,
/eventlanding page, operator announce, the/populationendpoints) are committed but unshipped. See Ship the release at the bottom. Confirm the SHA withcurl -s https://three.ws/api/versionafter the deploy and update this line.
Health: three commands that prove the site is up
# 1. API alive, and which build is actually serving
curl -s https://three.ws/api/version
# 2. API subsystems (x402, monitor, speech, alerts) - status must be "ok"
curl -s https://three.ws/api/healthz | head -c 400
# 3. The world server itself. If this is not {"ok":true}, /play is dead
# no matter what the site says.
curl -s https://three-ws-multiplayer-93741856042.us-central1.run.app/health
Continuous version, one terminal, quiet unless something changes:
npm run event:watch # every 30s
npm run event:watch -- --interval 15 # tighter during the opening rush
npm run event:watch -- --once # single sweep; exit 1 if anything is down
It watches the API, the live release SHA, the /play page (including that the
game bootstrap is still in the HTML), the world server, and the live player
count. It is plain HTTPS on purpose: it keeps working when gcloud auth has
expired, which it does on its own schedule in this environment.
Logs: two commands for fast triage
export PATH="$HOME/google-cloud-sdk/bin:$PATH"
# Errors across the whole fleet, last 6h
npm run logs:errors
# The world server specifically - join refusals, origin rejections, room churn
gcloud logging read 'resource.type="cloud_run_revision"
resource.labels.service_name="three-ws-multiplayer" severity>=WARNING' \
--freshness=1h --limit=50 --project aerial-vehicle-466722-p5
Deeper sweep when the cause is not obvious: npm run triage:gcp:deep.
Rollback: pre-filled, per service
These three revisions are the known-good pre-event state. They are live right now and all report Ready. After tomorrow's release deploy creates new revisions, these are what you roll back TO.
export PATH="$HOME/google-cloud-sdk/bin:$PATH"
R="--region us-central1 --project aerial-vehicle-466722-p5"
# API + frontend
gcloud run services update-traffic three-ws-api $R \
--to-revisions=three-ws-api-00364-n65=100
# The /play world server (kicks everyone in-world; they auto-reconnect)
gcloud run services update-traffic three-ws-multiplayer $R \
--to-revisions=three-ws-multiplayer-00010-g89=100
# Redis proxy (presence, durable builds)
gcloud run services update-traffic three-ws-redis-proxy $R \
--to-revisions=three-ws-redis-proxy-00004-bwk=100
Then purge the edge or the CDN keeps serving the bad build:
npm run deploy:gcp:purge-cdn
Return to normal afterwards with --to-latest instead of --to-revisions.
Rehearsal result (2026-08-08): every target above exists and reports
Ready=True with 100% of traffic. Traffic was not actually shifted, because
shifting it and back is itself two world-server restarts on the eve of the event;
the commands are the same update-traffic calls that created the current
routing, and the revisions are the ones already serving.
Scale: what is set, and how to raise it
Pre-scaled 2026-08-08 (config-only, pre-approved per docs/ops/gcp-credits-plan.md):
| Service | min | max | concurrency | CPU / memory | changed from |
|---|---|---|---|---|---|
three-ws-api |
6 | 100 | 160 | 2 / 4Gi | min was 2 |
three-ws-multiplayer |
1 | 1 (hard) | 1000 | 4 / 8Gi | was 2 CPU / 4Gi |
three-ws-redis-proxy |
3 | 10 | 80 | default | was min 2 / max 4 |
export PATH="$HOME/google-cloud-sdk/bin:$PATH"
R="--region us-central1 --project aerial-vehicle-466722-p5"
# More API headroom (safe, instant)
gcloud run services update three-ws-api $R --min-instances=12 --max-instances=200
# More world-server headroom. NOTE: raise CPU/RAM or concurrency, never max-instances.
gcloud run services update three-ws-multiplayer $R --cpu=8 --memory=16Gi
gcloud run services update three-ws-multiplayer $R --concurrency=2000
three-ws-multiplayer max-instances must stay 1. Colyseus rooms live in one
process. A second instance is only correct when the room registry and presence
are shared through Redis and each instance is individually addressable so a
seat reservation lands on the box hosting the room. Cloud Run gives every
instance the same URL, so instance 2 would silently split players in the same
world into two invisible crowds. A Memorystore Redis exists
(three-ws-redis, 10.234.231.139:6379) and REDIS_URI would switch the driver
on, but the addressability half is not solved and the eve of an event is not when
to find out. Use CPU, memory, and concurrency instead.
Capacity: measured, not assumed
npm run event:capacity -- --n 400 --ramp 45 --hold 150 against live production,
2026-08-08:
| requested / joined | 400 / 400 (100%) |
| held for the full 150s | 400 |
| join latency | p50 155ms · p95 219ms · max 388ms |
| mid-run drops | 0 |
| moves sent / state patches | 269,633 / 415,418 |
| verdict | PASS on all four thresholds |
The single world instance carried 400 concurrent players walking at 8 Hz with
zero drops. The known cliff is the concurrency ceiling: at 1000 simultaneous
connections Cloud Run wants a second instance and max-instances=1 refuses it.
If the count approaches that, raise --concurrency, not --max-instances.
One caveat baked into the harness: at 400 clients its own event loop stalls, so
the page-latency figure it prints measures the machine running it, not the
origin. It now prints its own loop lag next to that number and says so. An idle
curl during the same run measured /play at a steady 420-490ms.
Known failure modes and what to do
Audits 1-9 did not leave an accepted-risk register, so this is derived from the code's own documented landmines plus what was measured on 2026-08-08.
| If this happens | Why | Do this |
|---|---|---|
/play loads but nobody sees anyone else |
World server restarted or a second instance appeared | curl .../health; confirm max-instances=1; rollback command above |
| Players kicked seconds after joining, phones only | Mobile memory kill, not a network fault | node scripts/play-mobile-repro.mjs reproduces it; it reads as bytes, not as a networking bug |
/walk looks completely offline while /play works |
Someone set PLAY_GATE_MINT/THREE_MINT on the world server. /walk sends no play pass, so every join throws play_pass_required (landmine documented in multiplayer/src/rooms/WalkRoom.js) |
Unset it: gcloud run services update three-ws-multiplayer $R --remove-env-vars=PLAY_GATE_MINT,THREE_MINT |
| NPC / concierge replies slow or absent | Paid LLM tier is degraded today: OpenRouter 402 Payment Required, OpenAI 429. Vertex Gemini is healthy and the free-first chain leads, so players still get answers (measured 8-16s first token after live demotions) |
Nothing during the event. Topping up OpenRouter is an owner spend decision |
| Wallet balances read as zero / holder tier collapses to Member | Helius and Alchemy are both 429 today. 6 free rungs plus the QuickNode reserve carry it (proven by poisoning the primary) |
npm run event:failover to see which rungs are live now |
| Coin image missing on the event link or OG card | 4everland.io gateway times out, gateway.pinata.cloud is slow (~4s). api/img.js races five gateways and takes the first valid one |
Nothing. 4 of 5 gateways served the CID on 2026-08-08 |
| An x402 / boutique payment does not settle | /api/healthz shows 50,442 settle failures with fee_runway_exhausted. This is settle-floor starvation, which looks identical to "the wallets are dry" and has a different fix |
Run the x402-economy-triage agent BEFORE concluding anything about wallet balances |
A deploy dies at npm ci with "package.json and package-lock.json are not in sync" |
Almost certainly NOT a dependency mistake. FROM node:24-slim floats, so a newer bundled npm resolves the tree differently and rejects an untouched lockfile. This took the 2026-08-08 release build down |
The Dockerfile now pins ARG NPM_VERSION. If it recurs, compare npm --version locally against docker run --rm node:24-slim npm --version and set the pin to the version that produced the lockfile |
| Something breaks and nobody is told | Push alerting is thin. TELEGRAM_ALERTS_CHAT_ID is unset in production, so /api/healthz reports telegram_push: "disabled" and platform alerts land only in the ops_alerts table |
Email now works: two uptime checks (site + world server) and two alert policies (uptime failure, three-ws-api 5xx rate) were created 2026-08-08 and page [email protected]. Email is slow, so also keep npm run event:watch open on a screen |
Talk to the whole world mid-event
scripts/announce-play.mjs pushes a message to every player standing in a live
world, right now. It is the host's microphone: use it to call each agenda beat,
to tell people where to go, and to say anything that cannot wait for chat to
scroll. Requires the world server to be running the event build (the endpoint is
/internal/announce; a 404 means the deploy below has not happened yet).
# Every live world, toast only
node scripts/announce-play.mjs "Totem showdown starts in 2 minutes. Get to the plaza."
# With a title: also raises the centre-screen banner for 12s
node scripts/announce-play.mjs \
--title "Wheel hour" \
--detail "Free spin at Fortune's Folly, prizes on every wedge" \
"Wheel hour is open. Head to the plaza wheel."
# Only the $THREE world, and hold the banner 30s
node scripts/announce-play.mjs --coin FeMbDoX7R1Psc4GEcvJdsbNbZA3bfztcyDCatJVJpump \
--duration-ms 30000 --title "Fireworks" "Look up."
It prints how many rooms and players it reached, so 0 player(s) is your cue
that nobody is in the world yet (or that you targeted the wrong coin). The
request is HMAC-signed with MULTIPLAYER_SHARED_SECRET (falling back to
HOLDER_PASS_SECRET, which is in .env), and the server rejects an unsigned or
stale call, so the endpoint is safe to leave exposed. Announcements ride the same
notice channel the game already uses, so they reach every connected client
regardless of which build the player loaded.
Escalation: which tool for which subsystem
| Subsystem | Tool |
|---|---|
| x402 payments, settles, the ring, "wallets are dry" | x402-economy-triage agent. Do not diagnose by hand; settle-floor starvation and capital dispersion look the same from outside |
| Cloud Run, crons, logs, DB migrations, TLS, fleet readiness | gcp-triage skill, or npm run triage:gcp:deep |
| Phones dropping out of the world | node scripts/play-mobile-repro.mjs |
| Any page 404/500 across the site | npm run smoke:prod |
| A third-party provider suspected down | npm run event:failover |
| World capacity under a crowd | npm run event:capacity -- --n <2x current peak> |
Ship the release (owner-gated)
Two services, two commands. Both need explicit owner approval.
# 1. API + frontend (26+ commits ahead of what is live)
npm run deploy:gcp:full
# 2. The world server - required for the event features and /population.
# Its image predates every event commit.
cd multiplayer && ./deploy-cloudrun.sh
deploy-cloudrun.sh now defaults to CPU=4 MEMORY=8Gi so a redeploy cannot
silently undo the pre-scaling above. gcloud run deploy replaces resource limits
on every run, so a hotfix deploy with the old defaults would have halved the
world server's CPU mid-event.
After either deploy: curl -s https://three.ws/api/version, then
npm run event:watch -- --once, then update the release SHA at the top of this
page.
Open items
TELEGRAM_ALERTS_CHAT_IDexists nowhere (not.env, not.env.local, not the Cloud Run service). Every alerting path inapi/_lib/alerts.jsis already wired behind it; it needs one private chat/DM id, which only the owner can create. Then:gcloud run services update three-ws-api --region us-central1 --project aerial-vehicle-466722-p5 --update-env-vars=TELEGRAM_ALERTS_CHAT_ID=<id>(--update-env-vars, never--set-env-vars.) It must be a private chat: alerts carry stack traces, URLs, and IPs.Email alerting now exists; it did not before 2026-08-08. Created that day and wired to the existing verified
[email protected]channel:- uptime check
three.ws API healthzonhttps://three.ws/api/healthz - uptime check
three.ws world serveron the world server's/health - policy "Event: three.ws or the /play world server is DOWN" (uptime failing from more than 2 probe locations)
- policy "Event: three-ws-api 5xx rate elevated" (5xx above 5/s for 5 minutes)
Both policies link back to this page. Email is minutes-slow by nature, so it is the backstop, not the primary:
npm run event:watchis the primary.- uptime check
The release build failed once on 2026-08-08 at the
npm cilayer, root caused to the floating base image (see the failure-modes table). The Dockerfile pin is committed; the build has not been re-run since, so the release still needs a green build before the deploy commands above are known good.gcloudauth expires on its own schedule here. It died mid-session (invalid_grant: reauth related error (invalid_rapt)) and was restored withgcloud auth login --no-launch-browser. Verify with a real call (gcloud run services describe three-ws-api --region us-central1), never withgcloud auth list, which reports an expired account as ACTIVE.