Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

docs(ops): write the stranded wallet runbook and refresh the credits and payment outcome pages

nirholas1fd8f641d608
diff --git a/docs/api-reference.md b/docs/api-reference.md
index 5e1f79742..143977c67 100644
--- a/docs/api-reference.md
+++ b/docs/api-reference.md
@@ -4997,7 +4997,10 @@ hand-enumerated, so it can't drift from what's actually deployed), plus the quic
below and links to this page, the OpenAPI spec, and the x402 docs. **Machine-readable spec:**
[three.ws/openapi.json](https://three.ws/openapi.json) — every `/api/v1/x/*` path generated
from the same registry (`api/v1/_providers.js` `providerCatalog()`), tagged `Crypto API
-(aggregator)`. Adding a provider directory in the repo: [`api/v1/README.md`](../api/v1/README.md).
+(aggregator)`. Every paid `/api/x402/*` operation is generated the same way, from the
+service catalog (`api/_lib/service-catalog/services/`), so the document lists all of them
+with their real price, inputs and 402 response rather than the subset someone remembered to
+hand-write. Adding a provider directory in the repo: [`api/v1/README.md`](../api/v1/README.md).
```
GET /api/v1/x # discovery: every provider + endpoint
diff --git a/docs/ops/README.md b/docs/ops/README.md
index 276ebd940..7997303ef 100644
--- a/docs/ops/README.md
+++ b/docs/ops/README.md
@@ -29,6 +29,7 @@ Start with [gcp-production.md](gcp-production.md); it is the complete operationa
| [x402-discovery-listings.md](x402-discovery-listings.md) | Getting paid endpoints listed and ranked on x402scan, the Bazaar, and other directories. |
| [seo-keyword-plan.md](seo-keyword-plan.md) | Verified keyword landscape and content calendar (snapshot dated 2026-07-17). |
| [wallet-key-migration.md](wallet-key-migration.md) | Incident record: stranded pool-agent wallets after the WALLET_ENCRYPTION_KEY migration. |
+| [stranded-wallets.md](stranded-wallets.md) | The standing owner decision on custodial wallets sealed by that rotation: measured list, why recovery is impossible, what crediting or writing off the customer balances costs, and the commands for each. |
| [forge-multiview-migration-handoff.md](forge-multiview-migration-handoff.md) | Historical hand-off for the forge multi-view migrations (June 2026); superseded by gcp-production.md. |
| [examples-repo-export.md](examples-repo-export.md) | How the public `three-ws/examples` satellite repo is assembled and published from this monorepo. |
diff --git a/docs/ops/gcp-credits.md b/docs/ops/gcp-credits.md
index 7b4afc453..8ae2a7b84 100644
--- a/docs/ops/gcp-credits.md
+++ b/docs/ops/gcp-credits.md
@@ -1631,3 +1631,45 @@ found while smoke-testing `/dashboard/spend` for prompt 07, not caused by testin
artifact) or has been silently broken since 2026-07-07, with a stable environment (no
concurrent deploys), before attempting a fix — the evidence above rules out several
easy explanations already.
+
+---
+
+## Animation seeding: accept rate and cost (measured 2026-09-02)
+
+The text-to-motion lane (`model-text2motion`, us-east4) is live and generating.
+Subsystem doc, including every gate threshold and how it was derived:
+[docs/animation-seeding.md](../animation-seeding.md).
+
+**Quality gate calibration.** Against a 60-clip sample of the authored Mixamo library
+the gate accepts **87%**. The rejects are all clips legitimately out of spec for
+generated content rather than gate errors: sliced sub-clips a few frames long, static
+pose assets with zero motion, and one hit reaction that genuinely slides.
+
+**Accept rate on generated clips: 5 of 5** on the corrected gate. That sample is small
+and is stated as such. The number that matters more is how the first gate got there:
+an earlier version tested adjacent **local** quaternions and rejected **100%** of
+generated clips (24 of 24) on a `frame_pop` rule. Inspection showed the sampler emits
+a 180 degree twist about a bone's own axis that the child bone cancels, which is
+invisible on the mesh (a shoulder and forearm flipping together on 39 of 119 frames
+while the hand never moved more than 1.1 cm per frame). The gate now runs forward
+kinematics and judges world-space joint positions. Generated clips score 1.3x to 2.5x
+on world-space continuity against the authored library's median of 1.69 and worst case
+of 5.79, so the generated motion is as smooth as the curated library.
+
+**Why the sample is only 5.** `/api/forge-motion` is a public endpoint rate limited per
+IP. A 40-clip batch generated 2 clips and then took `429 rate_limited` with a
+`retry_after` of 2947 seconds on the remaining 38. Bulk seeding must use the
+in-process transport (`scripts/gcp/seed-motion.mjs` with `GCP_TEXT2MOTION_URL` set),
+which calls the provider directly and is not rate limited.
+
+**Cost.** Wall time is **35 to 45 seconds per clip** end to end (submit to clip JSON
+fetched), measured over 29 real generations. Cost per accepted asset in dollars is not
+recorded here yet: it needs `scripts/gcp/burn-report.mjs`, which needs a working
+`gcloud` credential, and this session's had expired (`Reauthentication failed`). Run
+the burn report after the next seeding batch and record the figure here.
+
+**Catalog position.** 58,544 avatars, 13,929 added in the previous 7 days (roughly
+1,600 a day). The avatar seed cron and its quality gate are live and no longer the
+bottleneck; the earlier figure of 18,622 at ~15 a day in the work order is stale.
+The generated-clip half of the library is still at zero published: the clip library
+serves 2,874 clips and every one is the `mx-` Mixamo import.
diff --git a/docs/ops/payment-outcomes.md b/docs/ops/payment-outcomes.md
index 842bc13e8..b62371b64 100644
--- a/docs/ops/payment-outcomes.md
+++ b/docs/ops/payment-outcomes.md
@@ -24,10 +24,10 @@ a `retry-after` header.
## The response envelope
```json
-{ "ok": true, "degraded": [], "generated_at": "…", "inbound": {…}, "ring_settle": {…}, "sponsor": {…} }
+{ "ok": true, "degraded": [], "generated_at": "…", "inbound": {…}, "ring_settle": {…}, "sponsor": {…}, "stranded_custody": {…} }
```
-The three panels are read independently, so one failing never blanks the others
+The panels are read independently, so one failing never blanks the others
(an RPC outage is exactly when the settle panels matter most). `ok` reports
whether the BOARD rendered, never whether payments are healthy: that verdict is
per panel. A panel that threw is replaced by `{ "error": "…" }`, named in
@@ -35,7 +35,7 @@ per panel. A panel that threw is replaced by `{ "error": "…" }`, named in
convention `/api/ops/health` uses. Alert on `degraded` being non-empty: a board
that cannot see is not a board that is fine.
-## The three panels
+## The panels
### `inbound` (from `x402_audit_log`)
@@ -111,6 +111,39 @@ The fee wallet that pays every settle's SOL fee:
- `runway_status` is the verdict the alert fires on (see below), and the
dashboard card colours from it so the board and the page cannot disagree.
+### `stranded_custody` (custodial key health, snapshot-cached)
+
+Custody the platform has LOST: wallets whose `encrypted_solana_secret` no longer
+decrypts, so their SOL is visible on chain and can never be signed for. It sits
+on this board because it is a payment-outcome fact: the treasury self-heal cannot
+reclaim that SOL and the customers holding it cannot withdraw it. Before this
+panel the only way to see the number was running
+`scripts/audit-custodial-key-health.mjs` by hand, so weeks passed between
+readings. The measurement itself is shared with that script
+(`api/_lib/custodial-key-health.js`), so the CLI and the board cannot disagree.
+
+- `status` is the verdict and `reason` says why:
+ - `stranded` / `sealed`: funded wallets are sealed. `sol_stranded` and its
+ `_platform` / `_customer` split are a real measurement.
+ - `unknown` / `partial_read`: some sealed wallet never got a balance read (an
+ RPC chunk failed), so the totals are a FLOOR, not a measurement. Re-read
+ after the lane recovers rather than quoting the number.
+ - `unknown` / `key_mismatch`: every wallet failed to decrypt. That is one wrong
+ key for this deployment, essentially never a fleet of separately sealed
+ wallets.
+ - `unknown` / `no_decryption_key`: this process holds no custodial key at all
+ (only possible outside production, where secret-box fails closed).
+ - `clear`: nothing funded is sealed.
+- Under either `unknown` key verdict the SOL fields are `null` rather than a
+ number with a caveat attached. An unattributable total is exactly the phantom
+ figure this surface exists to stop publishing.
+- `brief` points at [stranded-wallets.md](stranded-wallets.md), the standing owner
+ decision on the customer balances, whenever customer SOL is confirmed stranded.
+- The panel is snapshot-cached for six hours per instance and concurrent reads
+ share one scan (`cache_age_seconds` reports the snapshot's age). Custody loss is
+ a rotation event, not a live metric, so polling the board never turns into a
+ fleet-wide RPC sweep.
+
### The runway alert
The runway is no longer render-only. `checkRingWallets()` in
@@ -193,11 +226,19 @@ curl -s https://three.ws/api/x402/three-intel | jq '.accepts[].network'
`nothing_reclaimable`: the owner has to send SOL.
4. `replay_stages_24h` nonzero: captured X-PAYMENT headers are being re-sent;
the guard is working if the settles panel is unaffected.
+5. `stranded_custody.status` is `stranded`: capital is gone, not late. No cron
+ run, RPC tier, or deposit reaches it. If `sol_stranded_customer` is above
+ zero, that half is a support obligation and outranks every ops number on this
+ board: read [stranded-wallets.md](stranded-wallets.md). A `key_mismatch`
+ verdict is a deployment config problem instead, and the fix is the key, not
+ the wallets.
## Related
- `api/ops/payment-outcomes.js` (the endpoint),
`api/_lib/ops/x402-settle-health.js` (the sensor),
+ `api/_lib/custodial-key-health.js` (the custody-loss measurement shared with
+ `scripts/audit-custodial-key-health.mjs`),
`api/_lib/x402/wallet-balance-monitor.js` (balances, floors, and the runway
alert), `api/_lib/x402/sponsor-runway.js` (the burn measurement, the verdict,
and the alert copy), `api/_lib/x402/audit-log.js` (the durable ledger the
diff --git a/docs/ops/stranded-wallets.md b/docs/ops/stranded-wallets.md
new file mode 100644
index 000000000..20705d321
--- /dev/null
+++ b/docs/ops/stranded-wallets.md
@@ -0,0 +1,180 @@
+# Stranded custodial wallets: the standing owner decision
+
+Some custodial Solana wallets on three.ws are sealed under an encryption key the
+platform no longer holds. Their SOL is visible on chain, keeps rendering in the
+product, and can never be signed for again. Two of those wallets belong to
+customers who cannot withdraw.
+
+This is the decision brief. It carries the measurement, why recovery is
+impossible, what each option costs, and the exact commands for whichever option
+the owner picks. It does not move money: crediting, contacting, or writing off a
+customer balance is the owner's call.
+
+The incident that created them is recorded in
+[wallet-key-migration.md](wallet-key-migration.md). This file is the standing
+decision that incident left open.
+
+## What happened
+
+The 2026-07 Vercel to Cloud Run migration rotated `WALLET_ENCRYPTION_KEY`. Every
+custodial secret written under the retired value became permanently unopenable
+the moment the new key went live, because nothing kept a copy of the old one.
+
+`api/_lib/secret-box.js` has since gained a retired-key read path
+(`WALLET_ENCRYPTION_KEY_PREVIOUS`, comma separated, tried newest first, and a
+legacy `JWT_SECRET` candidate for v1 records). That path works. It has nothing to
+put in it:
+
+- Secret Manager holds exactly ONE version of `WALLET_ENCRYPTION_KEY`, created on
+ migration day. There is no prior version to roll back to.
+- The migration-era `JWT_SECRET` and the current one were both tried against the
+ sealed ciphertext during the 2026-07 investigation. Neither decrypted.
+- AES-GCM authenticates, so there is no "try harder" path: a wrong key throws
+ rather than returning plausible-but-wrong material. There is no brute force
+ worth attempting against a >=32-character master secret.
+
+Recovery is therefore not slow or expensive. It is impossible. The only open
+questions are what we owe the affected customers and what we do with the
+platform's own sealed dust.
+
+## Measured state
+
+Re-derive before acting. On a machine with the production key:
+
+```bash
+node scripts/audit-custodial-key-health.mjs # human table
+node scripts/audit-custodial-key-health.mjs --json # same numbers, machine readable
+```
+
+The script exits 3 without touching the database when no decryption key is
+configured, because a keyless run can only report 100% undecryptable and says
+nothing about production. The same measurement is served continuously at
+`GET /api/ops/payment-outcomes` under `stranded_custody` (see
+[payment-outcomes.md](payment-outcomes.md)), so an operator without the key can
+still read the current count from the running service.
+
+### Platform wallets (measured 2026-09-02, live)
+
+These two are confirmed sealed by production itself: the treasury-topup reclaim
+leg has attempted them on every run and recorded `secret_undecryptable` in
+`economy_master_ledger` 35,736 and 35,739 times respectively, most recently
+2026-09-02.
+
+| Agent | Agent id | Address | SOL on chain | Owner |
+|---|---|---|---|---|
+| Atlas #22 | `00bf4380-a6dd-4693-80d3-52bf23a8855b` | `6FL9viFy2WrYMWPd3HAQA4Bxm5qxQWoQMn3T9GbcwxEB` | 0.078390963 | `[email protected]` (platform bot) |
+| Echo #22 | `73da6b13-223a-481e-ab4c-4c293a462d62` | `8u5raEaz7Qjm5hRzNxwzXiZtjTkdgQ3Co6G6S5WNxFTs` | 0.064484542 | `[email protected]` (platform bot) |
+
+Platform total: **0.142875505 SOL**. This is the number the treasury dry run used
+to advertise as reclaimable; it no longer does (see "What was already fixed").
+
+### Customer wallets (support obligation)
+
+The last keyed audit (2026-08-09, recorded in
+[wallet-key-migration.md](wallet-key-migration.md)) measured **8 sealed wallets
+holding 0.49 SOL, of which 0.35 SOL belongs to customers**. One of those customer
+agents is confirmed independently by production: it hit the withdraw path and
+`agent_custody_events` recorded `wallet_key_retired` against it.
+
+| Agent | Agent id | Address | SOL on chain (2026-09-02) | Owner |
+|---|---|---|---|---|
+| My First Agent | `5e05f68f-eead-4ef9-b6b4-fc85ea73bbe9` | `GemVS5fT958FKRe5fpgizohUYUKE8cUDueEdmB1bmXnm` | 0.250001 | `[email protected]` (wallet-auth customer) |
+
+The remaining customer balance (about 0.10 SOL) belongs to a second customer
+agent that has not attempted a withdrawal, so production has never named it in a
+log. Run the audit with the production key to print it:
+
+```bash
+node scripts/audit-custodial-key-health.mjs --json \
+ | jq '.top_stranded[] | select(.platform == false)'
+```
+
+That command is the only step in this brief that needs the key, and it is
+read-only.
+
+### What the customer sees today
+
+Not a silent lie, as of the 2026-08 fixes: `GET /api/agents/:id/solana-wallet`
+attempts a local decrypt on the owner's own read and returns
+`signable: false, signable_reason: "key_retired"` beside the balance, and a
+withdrawal attempt answers `409 wallet_key_retired` telling the owner that no
+funds moved, that retrying will not help, and to contact support before creating a
+replacement wallet (provisioning a new one abandons the balance at the old
+address). Nobody is being invited to retry forever. Nobody has been made whole
+either.
+
+## The options
+
+| Option | What it costs | What it settles |
+|---|---|---|
+| **1. Credit the customers from treasury** (recommended) | About 0.35 SOL from the economy master, plus the transfer fees. At the SOL price this is a rounding error against the fee wallet's daily burn. | Ends the obligation completely and quietly. The customer is whole, the balance they see becomes real again, and no support conversation is needed. |
+| **2. Contact the customers first, then credit** | The same 0.35 SOL plus the support round trip. Both accounts are wallet-auth (`*@wallet.local`), so there is no email address on file: contact means an in-product notice, not a mail. | Same outcome as option 1, slower, and it advertises an internal key incident to a user who may not have noticed. |
+| **3. Write the balance off** | Zero SOL. Costs the customer 0.35 SOL of their own money and leaves a permanently unspendable balance rendering in their wallet card. | Nothing. The wallet card keeps showing an amount that cannot move; the obligation stays open indefinitely. |
+
+Recommendation: **option 1**. The amount is trivial against the treasury, it is
+our defect rather than the customer's, and the alternative leaves a broken balance
+on screen forever. The 2026-07 precedent supports it in spirit: when the same
+rotation stranded 12 platform pool wallets holding 1.41 SOL, that money was
+written off because it was ours to lose. Customer money is not.
+
+### If the owner picks option 1 (credit)
+
+Crediting is a fund transfer, so it stops at gate 1 in `CLAUDE.md`: recipient,
+amount, and chain get rendered and confirmed before anything signs. The wallets
+themselves cannot receive a "restore" (they cannot be signed for, so a deposit
+into them is a second write-off). Credit the customer's OTHER agent wallet, or a
+destination the customer names.
+
+1. Confirm the current stranded set and its owners:
+ ```bash
+ node scripts/audit-custodial-key-health.mjs --json \
+ | jq '{customer_sol: .sol.stranded_customer, wallets: [.top_stranded[] | select(.platform == false)]}'
+ ```
+2. For each affected user, pick the destination: another custodial agent wallet
+ they own, or an address they supply. Never the sealed address.
+3. Send from the economy master with the standard confirmation table (recipient,
+ amount, token, chain) and the owner's explicit yes, per gate 1.
+4. Record it: append the credit to [wallet-key-migration.md](wallet-key-migration.md)
+ under the incident's timeline so the ledger and the incident record agree.
+
+### If the owner picks option 3 (write-off)
+
+Nothing needs to run. The wallets stay sealed and the product already tells the
+owner they are unsignable. Note the decision in
+[wallet-key-migration.md](wallet-key-migration.md) and delete the OWNER-ACTIONS
+row so the question stops being re-litigated.
+
+### The platform's own 0.14 SOL, either way
+
+No decision needed: it is already gone, and nothing plans against it any more.
+Leave the two bot wallets alone. Do NOT re-key them with
+`scripts/rekey-stale-launch-wallets.mjs`: that script refuses a wallet holding
+more than 0.01 SOL for a reason (`409 wallet_funds_stranded`), and forcing it
+with `--force-drop-funds` abandons the balance at the old address for nothing.
+
+## What was already fixed, so nobody re-diagnoses it
+
+- **The treasury dry run no longer advertises sealed SOL.** `reclaimIdleAgentSol()`
+ in `api/_lib/economy-sweepback.js` used to return its plan before ever touching
+ a key, so `POST /api/cron/treasury-topup?dry=1` promised about 0.12 SOL that the
+ real leg could never move. Two separate sessions read that plan and concluded the
+ treasury would self-heal. The dry path now runs the same read-only key recovery
+ the real path does: sealed wallets are excluded from `reclaimedSol` and reported
+ in `failed` with `stage: "recover"` and a `secret_undecryptable` reason.
+ Pinned by `tests/economy-reclaim-dryrun-key-gate.test.js`.
+- **The wallet card and the withdraw path tell the truth** (`signable: false`,
+ `409 wallet_key_retired`), as described above.
+- **The audit refuses to guess.** A wallet whose balance was not read is never
+ counted as a confirmed zero, and a keyless run aborts instead of reporting a
+ fleet-wide phantom.
+- **The number is visible without a CLI run.** `stranded_custody` on
+ `/api/ops/payment-outcomes` carries count, SOL, and the platform/customer split,
+ snapshot-cached for six hours.
+
+## Preventing the next one
+
+The rotation runbook in [wallet-key-migration.md](wallet-key-migration.md) is the
+procedure: set `WALLET_ENCRYPTION_KEY_PREVIOUS` to the outgoing key BEFORE the new
+key goes live, keep it there until an audit run reports zero undecryptable
+wallets, and only then drop it. A rotation without that overlap is a one-way door,
+and this brief is what the far side of one looks like.
diff --git a/scripts/audit-docs.mjs b/scripts/audit-docs.mjs
index f8afca16c..dbda94763 100644
--- a/scripts/audit-docs.mjs
+++ b/scripts/audit-docs.mjs
@@ -308,6 +308,7 @@ const UNPUBLISHED_DOCS = new Map([
['nvidia-apps-catalog-listing', 'internal: paste-ready listing copy and portal steps for the NVIDIA Inception catalog'],
['nvidia-apps-catalog-request', 'internal: outbound email asking NVIDIA to publish the catalog listing'],
['nvidia-visibility-map', 'internal: outreach priorities and unclaimed program benefits across every NVIDIA surface'],
+ ['nvidia-ngc-listing', 'internal: prerequisite audit and paste-ready listing copy for the NVIDIA NGC catalog submission'],
['nvidia-forum-browser-digital-human', 'internal: forum post drafted for the NVIDIA Developer Forums, owner-gated until posted'],
['openai-listing-channels', 'internal: submission strategy and a candid post-mortem of our own stalled Cookbook PR'],
['okx-marketplace', 'owner-gated: names a crypto project other than $THREE'],