Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

feat(ops): read service env through its Secret Manager references and document the key migration

nirholas40746c22dec0
diff --git a/docs/ops/wallet-key-migration.md b/docs/ops/wallet-key-migration.md
index 22e65f4d3..10e29b53c 100644
--- a/docs/ops/wallet-key-migration.md
+++ b/docs/ops/wallet-key-migration.md
@@ -146,6 +146,139 @@ almost exactly. Nothing changed in the underlying incident: the customer-fund de
documented above (re-keying abandons the balance; that call is the owner's) is still
open, and no funds were moved.
+## Where the wallet secrets live now, and how to rotate one (2026-09-02)
+
+The incident above is about a key we could not read any more. This section is about the
+opposite failure: a key that was readable by too many people.
+
+`ECONOMY_MASTER_SECRET_BASE58` is the base58 secret key of the economy master wallet
+(`WwwuGbqHrwF5RG89KhUbmRWEvjnRH9k5kVM5p7T3WwW`), the funding root every other Solana
+engine is topped up from (`api/_lib/economy-master.js`, cron `api/cron/treasury-topup.js`).
+It sat on the `three-ws-api` Cloud Run service as a **plaintext literal**, so any principal
+holding `run.services.get` on `aerial-vehicle-466722-p5` could read the private key of a
+funded mainnet wallet straight out of the service config, without ever touching Secret
+Manager. It is now a `secretKeyRef`.
+
+### The end state
+
+| | |
+|---|---|
+| Secret | `wallet-economy-master-b58` (it already held this exact value, so no new version was minted) |
+| Reference | `ECONOMY_MASTER_SECRET_BASE58 = wallet-economy-master-b58:latest` |
+| Who can read it | `[email protected]`, granted `roles/secretmanager.secretAccessor` **on that one secret**, never project-wide |
+| Landed on | revision `three-ws-api-00405-z6c`, 100% of traffic |
+
+Read the current state at any time with `node scripts/migrate-plaintext-secrets.mjs --verify`.
+
+### The tool, and the Cloud Run gotcha it encodes
+
+`scripts/migrate-plaintext-secrets.mjs` does the whole move: it classifies every env var on
+the service, reuses the Secret Manager secret that already holds a value rather than minting
+a second copy of it, grants the runtime service account access to that one secret, flips the
+service in a single update, and then re-reads the service to prove no plaintext literal
+survived and that the new revision serves 100% of traffic. It is a dry run unless you pass
+`--apply`, and it never prints, logs, or writes a secret value.
+
+The gotcha it exists to encode: **Cloud Run refuses to retype an env var in place.** An
+update that only sets `--update-secrets` on a name that is currently a literal fails with
+
+```
+Cannot update environment variable [X] to the given type because it has already been set with a different type.
+```
+
+The literal has to be dropped in the SAME update (`--remove-env-vars X --update-secrets
+X=<secret>:latest`), which lands as one revision, so the variable is never missing from a
+serving container. Two separate updates would ship a revision with the var absent.
+
+### Adding a new version of an existing secret
+
+```sh
+# Never pass a secret as a CLI argument; feed it on stdin.
+printf %s "<the new value>" | gcloud secrets versions add wallet-economy-master-b58 \
+ --data-file=- --project aerial-vehicle-466722-p5
+```
+
+The service points at `:latest`, so a new version takes effect on the **next revision**, not
+immediately. Force one with a config-only update:
+
+```sh
+gcloud run services update three-ws-api --region us-central1 \
+ --project aerial-vehicle-466722-p5 \
+ --update-secrets ECONOMY_MASTER_SECRET_BASE58=wallet-economy-master-b58:latest
+```
+
+### Rotating the master wallet (owner-gated, do not run unattended)
+
+Migrating storage is config work. **Rotating is not**: it generates a new keypair and moves
+real mainnet SOL, which is CLAUDE.md stop-and-ask gate 1. An agent prepares this and stops;
+the owner runs it, or explicitly approves each transfer.
+
+The old value was a plaintext literal for some window, so it should be treated as exposed to
+everyone who has held project viewer access in that period. Whether that warrants a rotation
+is a judgment call about who those principals are: it is logged as an owner decision in
+`prompts/finish/production-100-OWNER-ACTIONS.md`.
+
+If the owner decides to rotate:
+
+```sh
+# 1. Generate the new keypair OFFLINE and keep a durable copy. Nothing here reads
+# it back later: a secret version's value is unreadable to anyone without the
+# accessor role, and a lost key is a stranded wallet (see the top of this file).
+solana-keygen new --no-bip39-passphrase --outfile new-master.json
+solana-keygen pubkey new-master.json # the address funds will move TO
+
+# 2. Fund the new wallet and drain the old one. OWNER-GATED, every transfer
+# confirmed individually: this is real mainnet SOL leaving a funded wallet.
+# Leave the old wallet enough SOL to pay its own transaction fees.
+
+# 3. Publish the new secret as a version of the SAME secret, so nothing else has
+# to be re-pointed.
+printf %s "<new base58 secret>" | gcloud secrets versions add wallet-economy-master-b58 \
+ --data-file=- --project aerial-vehicle-466722-p5
+
+# 4. Update the advertised address in the same update as the new revision, so the
+# key and the address it derives to never disagree on a serving container.
+gcloud run services update three-ws-api --region us-central1 \
+ --project aerial-vehicle-466722-p5 \
+ --update-env-vars ECONOMY_MASTER_ADDRESS=<new pubkey> \
+ --update-secrets ECONOMY_MASTER_SECRET_BASE58=wallet-economy-master-b58:latest
+
+# 5. Prove the new key is the one production signs with: the treasury sweep writes
+# a heartbeat row to economy_master_ledger on every run, carrying the pubkey it
+# derived from the loaded secret.
+node --env-file=.env.local -e "import('./api/_lib/db.js').then(async(m)=>{const sql=m.sql||m.default;\
+ console.log(await sql\`select ts,event,master_pubkey from economy_master_ledger order by ts desc limit 3\`);process.exit(0)})"
+
+# 6. Only once step 5 shows the NEW pubkey and the old wallet is empty, disable the
+# old version so a rollback cannot silently resurrect a drained wallet.
+gcloud secrets versions disable <old version number> --secret=wallet-economy-master-b58 \
+ --project aerial-vehicle-466722-p5
+```
+
+Never `--set-env-vars` in any of these: it replaces the whole environment set.
+
+### Verifying a migration did not break anything
+
+Storage moves are invisible until they are not, so both readings below are taken against the
+live site, not assumed:
+
+1. **`/api/healthz` before and after.** Compare the `subsystems` block entry by entry. A
+ fresh revision resets in-process counters (RPC quota cooldowns, uptime), so expect those
+ to move; what must not move is a subsystem going from `ok` to `degraded` or `down`.
+2. **A real signing path.** For the master wallet that is the treasury sweep: a row in
+ `economy_master_ledger` timestamped after the new revision's `creationTimestamp`, carrying
+ the master's pubkey, proves the container read the secret, decoded 64 bytes, and derived
+ the right wallet. On 2026-09-02 revision `three-ws-api-00405-z6c` was created at
+ `19:15:17Z` and the ledger kept writing through `19:25:23Z` under
+ `WwwuGbqHrwF5RG89KhUbmRWEvjnRH9k5kVM5p7T3WwW`.
+
+If a migration does regress the service, roll traffic back first and diagnose after:
+
+```sh
+gcloud run services update-traffic three-ws-api --region us-central1 \
+ --project aerial-vehicle-466722-p5 --to-revisions <prior revision>=100
+```
+
## Takeaways
- **A key rotation must carry the old key forward, or accompany a sweep.** This is now
diff --git a/scripts/play-photo-mode-check.mjs b/scripts/play-photo-mode-check.mjs
index 6840446b0..e38b39dce 100644
--- a/scripts/play-photo-mode-check.mjs
+++ b/scripts/play-photo-mode-check.mjs
@@ -86,7 +86,7 @@ let failedEarly = false;
try {
// ── boot the world ────────────────────────────────────────────────────────
console.log(at(), `photo-mode check · ${ENGINE} · ${width}x${height} · ${WORLD}`);
- await page.goto(WORLD, { waitUntil: 'domcontentloaded', timeout: 120000 });
+ await page.goto(WORLD, { waitUntil: 'domcontentloaded', timeout: BOOT_MS });
// The HUD is the honest "the world is up" signal: it unhides only once the
// scene has a renderer and the room has answered. Asserted as a predicate
diff --git a/scripts/read-service-env.mjs b/scripts/read-service-env.mjs
new file mode 100755
index 000000000..5a41e3eaf
--- /dev/null
+++ b/scripts/read-service-env.mjs
@@ -0,0 +1,110 @@
+#!/usr/bin/env node
+// Resolve environment variables off the live three-ws-api Cloud Run service,
+// following Secret Manager references.
+//
+// Production's authoritative env lives on the Cloud Run service, and several
+// runbooks tell an operator to read a value out of it with `gcloud run services
+// describe`. That stopped working for credentials on 2026-09-02, when every
+// credential-bearing var moved into Secret Manager
+// (scripts/migrate-plaintext-secrets.mjs): `describe` now shows a
+// `valueFrom.secretKeyRef` where the value used to be, and a snippet that reads
+// `.value` silently gets an empty string. This resolves both shapes, so a runbook
+// command keeps working whether a var is a literal or a reference.
+//
+// USAGE
+// node scripts/read-service-env.mjs '^S3_' # export lines, safe for eval
+// node scripts/read-service-env.mjs JWT_SECRET --raw # just the value, no name
+// node scripts/read-service-env.mjs '_API_KEY$' --names # names + where each lives
+//
+// eval "$(node scripts/read-service-env.mjs '^S3_')" # load them into a shell
+//
+// The pattern is a JavaScript regular expression matched against the variable
+// name. With no pattern, every variable is listed in --names form.
+//
+// This prints credentials to your terminal, which is the point: it is how an
+// operator gets a production value onto a machine that needs it. Do not pipe it
+// into a file that lives in the repo, and do not paste its output into a commit.
+
+import { execFile } from 'node:child_process';
+import { promisify } from 'node:util';
+import './lib/gcloud-path.mjs';
+
+const execFileP = promisify(execFile);
+
+const PROJECT = 'aerial-vehicle-466722-p5';
+const REGION = 'us-central1';
+const SERVICE = 'three-ws-api';
+
+const argv = process.argv.slice(2);
+const RAW = argv.includes('--raw');
+const NAMES_ONLY = argv.includes('--names');
+const pattern = argv.find((a) => !a.startsWith('--'));
+
+function fail(msg) {
+ console.error(`\n FAILED: ${msg}\n`);
+ process.exit(1);
+}
+
+async function gcloud(args) {
+ const { stdout } = await execFileP('gcloud', [...args, `--project=${PROJECT}`], {
+ maxBuffer: 16 * 1024 * 1024,
+ });
+ return stdout;
+}
+
+async function main() {
+ let regex;
+ try {
+ regex = pattern ? new RegExp(pattern) : /.*/;
+ } catch (e) {
+ fail(`"${pattern}" is not a valid regular expression: ${e.message}`);
+ }
+
+ let svc;
+ try {
+ svc = JSON.parse(await gcloud(['run', 'services', 'describe', SERVICE, `--region=${REGION}`, '--format=json']));
+ } catch (e) {
+ fail(`could not read ${SERVICE}: ${(e?.stderr || e?.message || '').trim().split('\n')[0]}`);
+ }
+
+ const env = (svc?.spec?.template?.spec?.containers?.[0]?.env || []).filter((e) => regex.test(e.name));
+ if (!env.length) fail(`no variable on ${SERVICE} matches ${regex}`);
+
+ if (NAMES_ONLY) {
+ for (const e of env) {
+ const ref = e.valueFrom?.secretKeyRef;
+ console.log(`${e.name}\t${ref ? `secret:${ref.name}:${ref.key}` : 'literal'}`);
+ }
+ return;
+ }
+
+ if (RAW && env.length > 1) {
+ fail(`--raw prints one value; ${regex} matches ${env.length} variables (${env.map((e) => e.name).join(', ')})`);
+ }
+
+ for (const e of env) {
+ let value = e.value;
+ if (value === undefined) {
+ const ref = e.valueFrom?.secretKeyRef;
+ if (!ref?.name) {
+ console.error(` ${e.name}: neither a literal nor a readable secret reference, skipped`);
+ continue;
+ }
+ try {
+ value = await gcloud(['secrets', 'versions', 'access', ref.key || 'latest', `--secret=${ref.name}`]);
+ } catch (err) {
+ console.error(
+ ` ${e.name}: cannot read ${ref.name}:${ref.key}. Needs roles/secretmanager.secretAccessor on that secret.`,
+ );
+ process.exitCode = 1;
+ continue;
+ }
+ }
+ // Single quotes with the embedded-quote escape, so the line survives eval
+ // no matter what the value holds.
+ if (RAW) process.stdout.write(value);
+ else console.log(`export ${e.name}='${String(value).replace(/'/g, `'\\''`)}'`);
+ }
+}
+
+main().catch((e) => fail(e?.message || String(e)));
diff --git a/workers/okx-chat-bot/README.md b/workers/okx-chat-bot/README.md
index b0d2fae4b..b7bf4db48 100644
--- a/workers/okx-chat-bot/README.md
+++ b/workers/okx-chat-bot/README.md
@@ -183,6 +183,31 @@ is a stopgap and says so on the wire: every beat carries `host` and
codespace green would rebuild, one level up, the false-green this worker exists
to kill.
+### The first boot pages for an OTP, unless the state object is seeded
+
+`gs://three-ws-okx-bot-state` holds no snapshot yet, so the first revision boots
+logged out and alerts for a human email OTP as `[email protected]`. That round trip
+is avoidable: a machine already holding a live session (the codespace stopgap
+does) can write the same archive this worker writes, and the new host then
+restores an authenticated session and comes up online.
+
+```bash
+# On the machine holding the live session, with its daemon STOPPED so the sqlite
+# files are quiesced. The worker's SIGTERM path stops it for you.
+OKX_BOT_STATE_BUCKET=three-ws-okx-bot-state node -e "
+ import('./workers/okx-chat-bot/state.js').then(async ({ snapshotState }) => {
+ const { loadConfig } = await import('./workers/okx-chat-bot/config.js');
+ console.log(await snapshotState(loadConfig(), { reason: 'seed' }));
+ })"
+```
+
+Two caveats. The archive carries the wallet keyring and session, so it belongs in
+this private bucket and nowhere else, and `snapshotState` needs
+`GCP_SERVICE_ACCOUNT_JSON` on a machine that is not already on GCP. And the
+bucket has exactly one writer: stop the seeding host before the Cloud Run service
+starts, or the two interleave snapshots. Skipping the seed is not a failure, it
+just leaves the OTP that the first boot would have asked for anyway.
+
The service must run `--min-instances=1 --max-instances=1`. This is not a
capacity choice: the GCS snapshot has exactly one writer, and concurrent
revisions would interleave snapshots and corrupt the identity.