Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
feat(ops): read service env through its Secret Manager references and document the key migration
nirholas40746c22dec0
diff --git a/docs/ops/wallet-key-migration.md b/docs/ops/wallet-key-migration.mdindex 22e65f4d3..10e29b53c 100644--- a/docs/ops/wallet-key-migration.md+++ b/docs/ops/wallet-key-migration.md@@ -146,6 +146,139 @@ almost exactly. Nothing changed in the underlying incident: the customer-fund dedocumented above (re-keying abandons the balance; that call is the owner's) is stillopen, and no funds were moved.+## Where the wallet secrets live now, and how to rotate one (2026-09-02)++The incident above is about a key we could not read any more. This section is about the+opposite failure: a key that was readable by too many people.++`ECONOMY_MASTER_SECRET_BASE58` is the base58 secret key of the economy master wallet+(`WwwuGbqHrwF5RG89KhUbmRWEvjnRH9k5kVM5p7T3WwW`), the funding root every other Solana+engine is topped up from (`api/_lib/economy-master.js`, cron `api/cron/treasury-topup.js`).+It sat on the `three-ws-api` Cloud Run service as a **plaintext literal**, so any principal+holding `run.services.get` on `aerial-vehicle-466722-p5` could read the private key of a+funded mainnet wallet straight out of the service config, without ever touching Secret+Manager. It is now a `secretKeyRef`.++### The end state++| | |+|---|---|+| Secret | `wallet-economy-master-b58` (it already held this exact value, so no new version was minted) |+| Reference | `ECONOMY_MASTER_SECRET_BASE58 = wallet-economy-master-b58:latest` |+| Who can read it | `[email protected]`, granted `roles/secretmanager.secretAccessor` **on that one secret**, never project-wide |+| Landed on | revision `three-ws-api-00405-z6c`, 100% of traffic |++Read the current state at any time with `node scripts/migrate-plaintext-secrets.mjs --verify`.++### The tool, and the Cloud Run gotcha it encodes++`scripts/migrate-plaintext-secrets.mjs` does the whole move: it classifies every env var on+the service, reuses the Secret Manager secret that already holds a value rather than minting+a second copy of it, grants the runtime service account access to that one secret, flips the+service in a single update, and then re-reads the service to prove no plaintext literal+survived and that the new revision serves 100% of traffic. It is a dry run unless you pass+`--apply`, and it never prints, logs, or writes a secret value.++The gotcha it exists to encode: **Cloud Run refuses to retype an env var in place.** An+update that only sets `--update-secrets` on a name that is currently a literal fails with++```+Cannot update environment variable [X] to the given type because it has already been set with a different type.+```++The literal has to be dropped in the SAME update (`--remove-env-vars X --update-secrets+X=<secret>:latest`), which lands as one revision, so the variable is never missing from a+serving container. Two separate updates would ship a revision with the var absent.++### Adding a new version of an existing secret++```sh+# Never pass a secret as a CLI argument; feed it on stdin.+printf %s "<the new value>" | gcloud secrets versions add wallet-economy-master-b58 \+ --data-file=- --project aerial-vehicle-466722-p5+```++The service points at `:latest`, so a new version takes effect on the **next revision**, not+immediately. Force one with a config-only update:++```sh+gcloud run services update three-ws-api --region us-central1 \+ --project aerial-vehicle-466722-p5 \+ --update-secrets ECONOMY_MASTER_SECRET_BASE58=wallet-economy-master-b58:latest+```++### Rotating the master wallet (owner-gated, do not run unattended)++Migrating storage is config work. **Rotating is not**: it generates a new keypair and moves+real mainnet SOL, which is CLAUDE.md stop-and-ask gate 1. An agent prepares this and stops;+the owner runs it, or explicitly approves each transfer.++The old value was a plaintext literal for some window, so it should be treated as exposed to+everyone who has held project viewer access in that period. Whether that warrants a rotation+is a judgment call about who those principals are: it is logged as an owner decision in+`prompts/finish/production-100-OWNER-ACTIONS.md`.++If the owner decides to rotate:++```sh+# 1. Generate the new keypair OFFLINE and keep a durable copy. Nothing here reads+# it back later: a secret version's value is unreadable to anyone without the+# accessor role, and a lost key is a stranded wallet (see the top of this file).+solana-keygen new --no-bip39-passphrase --outfile new-master.json+solana-keygen pubkey new-master.json # the address funds will move TO++# 2. Fund the new wallet and drain the old one. OWNER-GATED, every transfer+# confirmed individually: this is real mainnet SOL leaving a funded wallet.+# Leave the old wallet enough SOL to pay its own transaction fees.++# 3. Publish the new secret as a version of the SAME secret, so nothing else has+# to be re-pointed.+printf %s "<new base58 secret>" | gcloud secrets versions add wallet-economy-master-b58 \+ --data-file=- --project aerial-vehicle-466722-p5++# 4. Update the advertised address in the same update as the new revision, so the+# key and the address it derives to never disagree on a serving container.+gcloud run services update three-ws-api --region us-central1 \+ --project aerial-vehicle-466722-p5 \+ --update-env-vars ECONOMY_MASTER_ADDRESS=<new pubkey> \+ --update-secrets ECONOMY_MASTER_SECRET_BASE58=wallet-economy-master-b58:latest++# 5. Prove the new key is the one production signs with: the treasury sweep writes+# a heartbeat row to economy_master_ledger on every run, carrying the pubkey it+# derived from the loaded secret.+node --env-file=.env.local -e "import('./api/_lib/db.js').then(async(m)=>{const sql=m.sql||m.default;\+ console.log(await sql\`select ts,event,master_pubkey from economy_master_ledger order by ts desc limit 3\`);process.exit(0)})"++# 6. Only once step 5 shows the NEW pubkey and the old wallet is empty, disable the+# old version so a rollback cannot silently resurrect a drained wallet.+gcloud secrets versions disable <old version number> --secret=wallet-economy-master-b58 \+ --project aerial-vehicle-466722-p5+```++Never `--set-env-vars` in any of these: it replaces the whole environment set.++### Verifying a migration did not break anything++Storage moves are invisible until they are not, so both readings below are taken against the+live site, not assumed:++1. **`/api/healthz` before and after.** Compare the `subsystems` block entry by entry. A+ fresh revision resets in-process counters (RPC quota cooldowns, uptime), so expect those+ to move; what must not move is a subsystem going from `ok` to `degraded` or `down`.+2. **A real signing path.** For the master wallet that is the treasury sweep: a row in+ `economy_master_ledger` timestamped after the new revision's `creationTimestamp`, carrying+ the master's pubkey, proves the container read the secret, decoded 64 bytes, and derived+ the right wallet. On 2026-09-02 revision `three-ws-api-00405-z6c` was created at+ `19:15:17Z` and the ledger kept writing through `19:25:23Z` under+ `WwwuGbqHrwF5RG89KhUbmRWEvjnRH9k5kVM5p7T3WwW`.++If a migration does regress the service, roll traffic back first and diagnose after:++```sh+gcloud run services update-traffic three-ws-api --region us-central1 \+ --project aerial-vehicle-466722-p5 --to-revisions <prior revision>=100+```+## Takeaways- **A key rotation must carry the old key forward, or accompany a sweep.** This is nowdiff --git a/scripts/play-photo-mode-check.mjs b/scripts/play-photo-mode-check.mjsindex 6840446b0..e38b39dce 100644--- a/scripts/play-photo-mode-check.mjs+++ b/scripts/play-photo-mode-check.mjs@@ -86,7 +86,7 @@ let failedEarly = false;try {// ── boot the world ────────────────────────────────────────────────────────console.log(at(), `photo-mode check · ${ENGINE} · ${width}x${height} · ${WORLD}`);- await page.goto(WORLD, { waitUntil: 'domcontentloaded', timeout: 120000 });+ await page.goto(WORLD, { waitUntil: 'domcontentloaded', timeout: BOOT_MS });// The HUD is the honest "the world is up" signal: it unhides only once the// scene has a renderer and the room has answered. Asserted as a predicatediff --git a/scripts/read-service-env.mjs b/scripts/read-service-env.mjsnew file mode 100755index 000000000..5a41e3eaf--- /dev/null+++ b/scripts/read-service-env.mjs@@ -0,0 +1,110 @@+#!/usr/bin/env node+// Resolve environment variables off the live three-ws-api Cloud Run service,+// following Secret Manager references.+//+// Production's authoritative env lives on the Cloud Run service, and several+// runbooks tell an operator to read a value out of it with `gcloud run services+// describe`. That stopped working for credentials on 2026-09-02, when every+// credential-bearing var moved into Secret Manager+// (scripts/migrate-plaintext-secrets.mjs): `describe` now shows a+// `valueFrom.secretKeyRef` where the value used to be, and a snippet that reads+// `.value` silently gets an empty string. This resolves both shapes, so a runbook+// command keeps working whether a var is a literal or a reference.+//+// USAGE+// node scripts/read-service-env.mjs '^S3_' # export lines, safe for eval+// node scripts/read-service-env.mjs JWT_SECRET --raw # just the value, no name+// node scripts/read-service-env.mjs '_API_KEY$' --names # names + where each lives+//+// eval "$(node scripts/read-service-env.mjs '^S3_')" # load them into a shell+//+// The pattern is a JavaScript regular expression matched against the variable+// name. With no pattern, every variable is listed in --names form.+//+// This prints credentials to your terminal, which is the point: it is how an+// operator gets a production value onto a machine that needs it. Do not pipe it+// into a file that lives in the repo, and do not paste its output into a commit.++import { execFile } from 'node:child_process';+import { promisify } from 'node:util';+import './lib/gcloud-path.mjs';++const execFileP = promisify(execFile);++const PROJECT = 'aerial-vehicle-466722-p5';+const REGION = 'us-central1';+const SERVICE = 'three-ws-api';++const argv = process.argv.slice(2);+const RAW = argv.includes('--raw');+const NAMES_ONLY = argv.includes('--names');+const pattern = argv.find((a) => !a.startsWith('--'));++function fail(msg) {+ console.error(`\n FAILED: ${msg}\n`);+ process.exit(1);+}++async function gcloud(args) {+ const { stdout } = await execFileP('gcloud', [...args, `--project=${PROJECT}`], {+ maxBuffer: 16 * 1024 * 1024,+ });+ return stdout;+}++async function main() {+ let regex;+ try {+ regex = pattern ? new RegExp(pattern) : /.*/;+ } catch (e) {+ fail(`"${pattern}" is not a valid regular expression: ${e.message}`);+ }++ let svc;+ try {+ svc = JSON.parse(await gcloud(['run', 'services', 'describe', SERVICE, `--region=${REGION}`, '--format=json']));+ } catch (e) {+ fail(`could not read ${SERVICE}: ${(e?.stderr || e?.message || '').trim().split('\n')[0]}`);+ }++ const env = (svc?.spec?.template?.spec?.containers?.[0]?.env || []).filter((e) => regex.test(e.name));+ if (!env.length) fail(`no variable on ${SERVICE} matches ${regex}`);++ if (NAMES_ONLY) {+ for (const e of env) {+ const ref = e.valueFrom?.secretKeyRef;+ console.log(`${e.name}\t${ref ? `secret:${ref.name}:${ref.key}` : 'literal'}`);+ }+ return;+ }++ if (RAW && env.length > 1) {+ fail(`--raw prints one value; ${regex} matches ${env.length} variables (${env.map((e) => e.name).join(', ')})`);+ }++ for (const e of env) {+ let value = e.value;+ if (value === undefined) {+ const ref = e.valueFrom?.secretKeyRef;+ if (!ref?.name) {+ console.error(` ${e.name}: neither a literal nor a readable secret reference, skipped`);+ continue;+ }+ try {+ value = await gcloud(['secrets', 'versions', 'access', ref.key || 'latest', `--secret=${ref.name}`]);+ } catch (err) {+ console.error(+ ` ${e.name}: cannot read ${ref.name}:${ref.key}. Needs roles/secretmanager.secretAccessor on that secret.`,+ );+ process.exitCode = 1;+ continue;+ }+ }+ // Single quotes with the embedded-quote escape, so the line survives eval+ // no matter what the value holds.+ if (RAW) process.stdout.write(value);+ else console.log(`export ${e.name}='${String(value).replace(/'/g, `'\\''`)}'`);+ }+}++main().catch((e) => fail(e?.message || String(e)));diff --git a/workers/okx-chat-bot/README.md b/workers/okx-chat-bot/README.mdindex b0d2fae4b..b7bf4db48 100644--- a/workers/okx-chat-bot/README.md+++ b/workers/okx-chat-bot/README.md@@ -183,6 +183,31 @@ is a stopgap and says so on the wire: every beat carries `host` andcodespace green would rebuild, one level up, the false-green this worker existsto kill.+### The first boot pages for an OTP, unless the state object is seeded++`gs://three-ws-okx-bot-state` holds no snapshot yet, so the first revision boots+logged out and alerts for a human email OTP as `[email protected]`. That round trip+is avoidable: a machine already holding a live session (the codespace stopgap+does) can write the same archive this worker writes, and the new host then+restores an authenticated session and comes up online.++```bash+# On the machine holding the live session, with its daemon STOPPED so the sqlite+# files are quiesced. The worker's SIGTERM path stops it for you.+OKX_BOT_STATE_BUCKET=three-ws-okx-bot-state node -e "+ import('./workers/okx-chat-bot/state.js').then(async ({ snapshotState }) => {+ const { loadConfig } = await import('./workers/okx-chat-bot/config.js');+ console.log(await snapshotState(loadConfig(), { reason: 'seed' }));+ })"+```++Two caveats. The archive carries the wallet keyring and session, so it belongs in+this private bucket and nowhere else, and `snapshotState` needs+`GCP_SERVICE_ACCOUNT_JSON` on a machine that is not already on GCP. And the+bucket has exactly one writer: stop the seeding host before the Cloud Run service+starts, or the two interleave snapshots. Skipping the seed is not a failure, it+just leaves the OTP that the first boot would have asked for anyway.+The service must run `--min-instances=1 --max-instances=1`. This is not acapacity choice: the GCS snapshot has exactly one writer, and concurrentrevisions would interleave snapshots and corrupt the identity.