Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
JavaScript 3d-generationai-agentsanimationavatardigital-humangenerative-aiglbgltfimage-to-3dllmmcpmcp-servermodel-context-protocolsolanatext-to-3dthreejsweb-componentswebglwebxrx402
docs(ops): write the stranded wallet runbook and refresh the credits and payment outcome pages
nirholas1fd8f641d608
diff --git a/docs/api-reference.md b/docs/api-reference.mdindex 5e1f79742..143977c67 100644--- a/docs/api-reference.md+++ b/docs/api-reference.md@@ -4997,7 +4997,10 @@ hand-enumerated, so it can't drift from what's actually deployed), plus the quicbelow and links to this page, the OpenAPI spec, and the x402 docs. **Machine-readable spec:**[three.ws/openapi.json](https://three.ws/openapi.json) — every `/api/v1/x/*` path generatedfrom the same registry (`api/v1/_providers.js` `providerCatalog()`), tagged `Crypto API-(aggregator)`. Adding a provider directory in the repo: [`api/v1/README.md`](../api/v1/README.md).+(aggregator)`. Every paid `/api/x402/*` operation is generated the same way, from the+service catalog (`api/_lib/service-catalog/services/`), so the document lists all of them+with their real price, inputs and 402 response rather than the subset someone remembered to+hand-write. Adding a provider directory in the repo: [`api/v1/README.md`](../api/v1/README.md).```GET /api/v1/x # discovery: every provider + endpointdiff --git a/docs/ops/README.md b/docs/ops/README.mdindex 276ebd940..7997303ef 100644--- a/docs/ops/README.md+++ b/docs/ops/README.md@@ -29,6 +29,7 @@ Start with [gcp-production.md](gcp-production.md); it is the complete operationa| [x402-discovery-listings.md](x402-discovery-listings.md) | Getting paid endpoints listed and ranked on x402scan, the Bazaar, and other directories. || [seo-keyword-plan.md](seo-keyword-plan.md) | Verified keyword landscape and content calendar (snapshot dated 2026-07-17). || [wallet-key-migration.md](wallet-key-migration.md) | Incident record: stranded pool-agent wallets after the WALLET_ENCRYPTION_KEY migration. |+| [stranded-wallets.md](stranded-wallets.md) | The standing owner decision on custodial wallets sealed by that rotation: measured list, why recovery is impossible, what crediting or writing off the customer balances costs, and the commands for each. || [forge-multiview-migration-handoff.md](forge-multiview-migration-handoff.md) | Historical hand-off for the forge multi-view migrations (June 2026); superseded by gcp-production.md. || [examples-repo-export.md](examples-repo-export.md) | How the public `three-ws/examples` satellite repo is assembled and published from this monorepo. |diff --git a/docs/ops/gcp-credits.md b/docs/ops/gcp-credits.mdindex 7b4afc453..8ae2a7b84 100644--- a/docs/ops/gcp-credits.md+++ b/docs/ops/gcp-credits.md@@ -1631,3 +1631,45 @@ found while smoke-testing `/dashboard/spend` for prompt 07, not caused by testinartifact) or has been silently broken since 2026-07-07, with a stable environment (noconcurrent deploys), before attempting a fix — the evidence above rules out severaleasy explanations already.++---++## Animation seeding: accept rate and cost (measured 2026-09-02)++The text-to-motion lane (`model-text2motion`, us-east4) is live and generating.+Subsystem doc, including every gate threshold and how it was derived:+[docs/animation-seeding.md](../animation-seeding.md).++**Quality gate calibration.** Against a 60-clip sample of the authored Mixamo library+the gate accepts **87%**. The rejects are all clips legitimately out of spec for+generated content rather than gate errors: sliced sub-clips a few frames long, static+pose assets with zero motion, and one hit reaction that genuinely slides.++**Accept rate on generated clips: 5 of 5** on the corrected gate. That sample is small+and is stated as such. The number that matters more is how the first gate got there:+an earlier version tested adjacent **local** quaternions and rejected **100%** of+generated clips (24 of 24) on a `frame_pop` rule. Inspection showed the sampler emits+a 180 degree twist about a bone's own axis that the child bone cancels, which is+invisible on the mesh (a shoulder and forearm flipping together on 39 of 119 frames+while the hand never moved more than 1.1 cm per frame). The gate now runs forward+kinematics and judges world-space joint positions. Generated clips score 1.3x to 2.5x+on world-space continuity against the authored library's median of 1.69 and worst case+of 5.79, so the generated motion is as smooth as the curated library.++**Why the sample is only 5.** `/api/forge-motion` is a public endpoint rate limited per+IP. A 40-clip batch generated 2 clips and then took `429 rate_limited` with a+`retry_after` of 2947 seconds on the remaining 38. Bulk seeding must use the+in-process transport (`scripts/gcp/seed-motion.mjs` with `GCP_TEXT2MOTION_URL` set),+which calls the provider directly and is not rate limited.++**Cost.** Wall time is **35 to 45 seconds per clip** end to end (submit to clip JSON+fetched), measured over 29 real generations. Cost per accepted asset in dollars is not+recorded here yet: it needs `scripts/gcp/burn-report.mjs`, which needs a working+`gcloud` credential, and this session's had expired (`Reauthentication failed`). Run+the burn report after the next seeding batch and record the figure here.++**Catalog position.** 58,544 avatars, 13,929 added in the previous 7 days (roughly+1,600 a day). The avatar seed cron and its quality gate are live and no longer the+bottleneck; the earlier figure of 18,622 at ~15 a day in the work order is stale.+The generated-clip half of the library is still at zero published: the clip library+serves 2,874 clips and every one is the `mx-` Mixamo import.diff --git a/docs/ops/payment-outcomes.md b/docs/ops/payment-outcomes.mdindex 842bc13e8..b62371b64 100644--- a/docs/ops/payment-outcomes.md+++ b/docs/ops/payment-outcomes.md@@ -24,10 +24,10 @@ a `retry-after` header.## The response envelope```json-{ "ok": true, "degraded": [], "generated_at": "…", "inbound": {…}, "ring_settle": {…}, "sponsor": {…} }+{ "ok": true, "degraded": [], "generated_at": "…", "inbound": {…}, "ring_settle": {…}, "sponsor": {…}, "stranded_custody": {…} }```-The three panels are read independently, so one failing never blanks the others+The panels are read independently, so one failing never blanks the others(an RPC outage is exactly when the settle panels matter most). `ok` reportswhether the BOARD rendered, never whether payments are healthy: that verdict isper panel. A panel that threw is replaced by `{ "error": "…" }`, named in@@ -35,7 +35,7 @@ per panel. A panel that threw is replaced by `{ "error": "…" }`, named inconvention `/api/ops/health` uses. Alert on `degraded` being non-empty: a boardthat cannot see is not a board that is fine.-## The three panels+## The panels### `inbound` (from `x402_audit_log`)@@ -111,6 +111,39 @@ The fee wallet that pays every settle's SOL fee:- `runway_status` is the verdict the alert fires on (see below), and thedashboard card colours from it so the board and the page cannot disagree.+### `stranded_custody` (custodial key health, snapshot-cached)++Custody the platform has LOST: wallets whose `encrypted_solana_secret` no longer+decrypts, so their SOL is visible on chain and can never be signed for. It sits+on this board because it is a payment-outcome fact: the treasury self-heal cannot+reclaim that SOL and the customers holding it cannot withdraw it. Before this+panel the only way to see the number was running+`scripts/audit-custodial-key-health.mjs` by hand, so weeks passed between+readings. The measurement itself is shared with that script+(`api/_lib/custodial-key-health.js`), so the CLI and the board cannot disagree.++- `status` is the verdict and `reason` says why:+ - `stranded` / `sealed`: funded wallets are sealed. `sol_stranded` and its+ `_platform` / `_customer` split are a real measurement.+ - `unknown` / `partial_read`: some sealed wallet never got a balance read (an+ RPC chunk failed), so the totals are a FLOOR, not a measurement. Re-read+ after the lane recovers rather than quoting the number.+ - `unknown` / `key_mismatch`: every wallet failed to decrypt. That is one wrong+ key for this deployment, essentially never a fleet of separately sealed+ wallets.+ - `unknown` / `no_decryption_key`: this process holds no custodial key at all+ (only possible outside production, where secret-box fails closed).+ - `clear`: nothing funded is sealed.+- Under either `unknown` key verdict the SOL fields are `null` rather than a+ number with a caveat attached. An unattributable total is exactly the phantom+ figure this surface exists to stop publishing.+- `brief` points at [stranded-wallets.md](stranded-wallets.md), the standing owner+ decision on the customer balances, whenever customer SOL is confirmed stranded.+- The panel is snapshot-cached for six hours per instance and concurrent reads+ share one scan (`cache_age_seconds` reports the snapshot's age). Custody loss is+ a rotation event, not a live metric, so polling the board never turns into a+ fleet-wide RPC sweep.+### The runway alertThe runway is no longer render-only. `checkRingWallets()` in@@ -193,11 +226,19 @@ curl -s https://three.ws/api/x402/three-intel | jq '.accepts[].network'`nothing_reclaimable`: the owner has to send SOL.4. `replay_stages_24h` nonzero: captured X-PAYMENT headers are being re-sent;the guard is working if the settles panel is unaffected.+5. `stranded_custody.status` is `stranded`: capital is gone, not late. No cron+ run, RPC tier, or deposit reaches it. If `sol_stranded_customer` is above+ zero, that half is a support obligation and outranks every ops number on this+ board: read [stranded-wallets.md](stranded-wallets.md). A `key_mismatch`+ verdict is a deployment config problem instead, and the fix is the key, not+ the wallets.## Related- `api/ops/payment-outcomes.js` (the endpoint),`api/_lib/ops/x402-settle-health.js` (the sensor),+ `api/_lib/custodial-key-health.js` (the custody-loss measurement shared with+ `scripts/audit-custodial-key-health.mjs`),`api/_lib/x402/wallet-balance-monitor.js` (balances, floors, and the runwayalert), `api/_lib/x402/sponsor-runway.js` (the burn measurement, the verdict,and the alert copy), `api/_lib/x402/audit-log.js` (the durable ledger thediff --git a/docs/ops/stranded-wallets.md b/docs/ops/stranded-wallets.mdnew file mode 100644index 000000000..20705d321--- /dev/null+++ b/docs/ops/stranded-wallets.md@@ -0,0 +1,180 @@+# Stranded custodial wallets: the standing owner decision++Some custodial Solana wallets on three.ws are sealed under an encryption key the+platform no longer holds. Their SOL is visible on chain, keeps rendering in the+product, and can never be signed for again. Two of those wallets belong to+customers who cannot withdraw.++This is the decision brief. It carries the measurement, why recovery is+impossible, what each option costs, and the exact commands for whichever option+the owner picks. It does not move money: crediting, contacting, or writing off a+customer balance is the owner's call.++The incident that created them is recorded in+[wallet-key-migration.md](wallet-key-migration.md). This file is the standing+decision that incident left open.++## What happened++The 2026-07 Vercel to Cloud Run migration rotated `WALLET_ENCRYPTION_KEY`. Every+custodial secret written under the retired value became permanently unopenable+the moment the new key went live, because nothing kept a copy of the old one.++`api/_lib/secret-box.js` has since gained a retired-key read path+(`WALLET_ENCRYPTION_KEY_PREVIOUS`, comma separated, tried newest first, and a+legacy `JWT_SECRET` candidate for v1 records). That path works. It has nothing to+put in it:++- Secret Manager holds exactly ONE version of `WALLET_ENCRYPTION_KEY`, created on+ migration day. There is no prior version to roll back to.+- The migration-era `JWT_SECRET` and the current one were both tried against the+ sealed ciphertext during the 2026-07 investigation. Neither decrypted.+- AES-GCM authenticates, so there is no "try harder" path: a wrong key throws+ rather than returning plausible-but-wrong material. There is no brute force+ worth attempting against a >=32-character master secret.++Recovery is therefore not slow or expensive. It is impossible. The only open+questions are what we owe the affected customers and what we do with the+platform's own sealed dust.++## Measured state++Re-derive before acting. On a machine with the production key:++```bash+node scripts/audit-custodial-key-health.mjs # human table+node scripts/audit-custodial-key-health.mjs --json # same numbers, machine readable+```++The script exits 3 without touching the database when no decryption key is+configured, because a keyless run can only report 100% undecryptable and says+nothing about production. The same measurement is served continuously at+`GET /api/ops/payment-outcomes` under `stranded_custody` (see+[payment-outcomes.md](payment-outcomes.md)), so an operator without the key can+still read the current count from the running service.++### Platform wallets (measured 2026-09-02, live)++These two are confirmed sealed by production itself: the treasury-topup reclaim+leg has attempted them on every run and recorded `secret_undecryptable` in+`economy_master_ledger` 35,736 and 35,739 times respectively, most recently+2026-09-02.++| Agent | Agent id | Address | SOL on chain | Owner |+|---|---|---|---|---|+| Atlas #22 | `00bf4380-a6dd-4693-80d3-52bf23a8855b` | `6FL9viFy2WrYMWPd3HAQA4Bxm5qxQWoQMn3T9GbcwxEB` | 0.078390963 | `[email protected]` (platform bot) |+| Echo #22 | `73da6b13-223a-481e-ab4c-4c293a462d62` | `8u5raEaz7Qjm5hRzNxwzXiZtjTkdgQ3Co6G6S5WNxFTs` | 0.064484542 | `[email protected]` (platform bot) |++Platform total: **0.142875505 SOL**. This is the number the treasury dry run used+to advertise as reclaimable; it no longer does (see "What was already fixed").++### Customer wallets (support obligation)++The last keyed audit (2026-08-09, recorded in+[wallet-key-migration.md](wallet-key-migration.md)) measured **8 sealed wallets+holding 0.49 SOL, of which 0.35 SOL belongs to customers**. One of those customer+agents is confirmed independently by production: it hit the withdraw path and+`agent_custody_events` recorded `wallet_key_retired` against it.++| Agent | Agent id | Address | SOL on chain (2026-09-02) | Owner |+|---|---|---|---|---|+| My First Agent | `5e05f68f-eead-4ef9-b6b4-fc85ea73bbe9` | `GemVS5fT958FKRe5fpgizohUYUKE8cUDueEdmB1bmXnm` | 0.250001 | `[email protected]` (wallet-auth customer) |++The remaining customer balance (about 0.10 SOL) belongs to a second customer+agent that has not attempted a withdrawal, so production has never named it in a+log. Run the audit with the production key to print it:++```bash+node scripts/audit-custodial-key-health.mjs --json \+ | jq '.top_stranded[] | select(.platform == false)'+```++That command is the only step in this brief that needs the key, and it is+read-only.++### What the customer sees today++Not a silent lie, as of the 2026-08 fixes: `GET /api/agents/:id/solana-wallet`+attempts a local decrypt on the owner's own read and returns+`signable: false, signable_reason: "key_retired"` beside the balance, and a+withdrawal attempt answers `409 wallet_key_retired` telling the owner that no+funds moved, that retrying will not help, and to contact support before creating a+replacement wallet (provisioning a new one abandons the balance at the old+address). Nobody is being invited to retry forever. Nobody has been made whole+either.++## The options++| Option | What it costs | What it settles |+|---|---|---|+| **1. Credit the customers from treasury** (recommended) | About 0.35 SOL from the economy master, plus the transfer fees. At the SOL price this is a rounding error against the fee wallet's daily burn. | Ends the obligation completely and quietly. The customer is whole, the balance they see becomes real again, and no support conversation is needed. |+| **2. Contact the customers first, then credit** | The same 0.35 SOL plus the support round trip. Both accounts are wallet-auth (`*@wallet.local`), so there is no email address on file: contact means an in-product notice, not a mail. | Same outcome as option 1, slower, and it advertises an internal key incident to a user who may not have noticed. |+| **3. Write the balance off** | Zero SOL. Costs the customer 0.35 SOL of their own money and leaves a permanently unspendable balance rendering in their wallet card. | Nothing. The wallet card keeps showing an amount that cannot move; the obligation stays open indefinitely. |++Recommendation: **option 1**. The amount is trivial against the treasury, it is+our defect rather than the customer's, and the alternative leaves a broken balance+on screen forever. The 2026-07 precedent supports it in spirit: when the same+rotation stranded 12 platform pool wallets holding 1.41 SOL, that money was+written off because it was ours to lose. Customer money is not.++### If the owner picks option 1 (credit)++Crediting is a fund transfer, so it stops at gate 1 in `CLAUDE.md`: recipient,+amount, and chain get rendered and confirmed before anything signs. The wallets+themselves cannot receive a "restore" (they cannot be signed for, so a deposit+into them is a second write-off). Credit the customer's OTHER agent wallet, or a+destination the customer names.++1. Confirm the current stranded set and its owners:+ ```bash+ node scripts/audit-custodial-key-health.mjs --json \+ | jq '{customer_sol: .sol.stranded_customer, wallets: [.top_stranded[] | select(.platform == false)]}'+ ```+2. For each affected user, pick the destination: another custodial agent wallet+ they own, or an address they supply. Never the sealed address.+3. Send from the economy master with the standard confirmation table (recipient,+ amount, token, chain) and the owner's explicit yes, per gate 1.+4. Record it: append the credit to [wallet-key-migration.md](wallet-key-migration.md)+ under the incident's timeline so the ledger and the incident record agree.++### If the owner picks option 3 (write-off)++Nothing needs to run. The wallets stay sealed and the product already tells the+owner they are unsignable. Note the decision in+[wallet-key-migration.md](wallet-key-migration.md) and delete the OWNER-ACTIONS+row so the question stops being re-litigated.++### The platform's own 0.14 SOL, either way++No decision needed: it is already gone, and nothing plans against it any more.+Leave the two bot wallets alone. Do NOT re-key them with+`scripts/rekey-stale-launch-wallets.mjs`: that script refuses a wallet holding+more than 0.01 SOL for a reason (`409 wallet_funds_stranded`), and forcing it+with `--force-drop-funds` abandons the balance at the old address for nothing.++## What was already fixed, so nobody re-diagnoses it++- **The treasury dry run no longer advertises sealed SOL.** `reclaimIdleAgentSol()`+ in `api/_lib/economy-sweepback.js` used to return its plan before ever touching+ a key, so `POST /api/cron/treasury-topup?dry=1` promised about 0.12 SOL that the+ real leg could never move. Two separate sessions read that plan and concluded the+ treasury would self-heal. The dry path now runs the same read-only key recovery+ the real path does: sealed wallets are excluded from `reclaimedSol` and reported+ in `failed` with `stage: "recover"` and a `secret_undecryptable` reason.+ Pinned by `tests/economy-reclaim-dryrun-key-gate.test.js`.+- **The wallet card and the withdraw path tell the truth** (`signable: false`,+ `409 wallet_key_retired`), as described above.+- **The audit refuses to guess.** A wallet whose balance was not read is never+ counted as a confirmed zero, and a keyless run aborts instead of reporting a+ fleet-wide phantom.+- **The number is visible without a CLI run.** `stranded_custody` on+ `/api/ops/payment-outcomes` carries count, SOL, and the platform/customer split,+ snapshot-cached for six hours.++## Preventing the next one++The rotation runbook in [wallet-key-migration.md](wallet-key-migration.md) is the+procedure: set `WALLET_ENCRYPTION_KEY_PREVIOUS` to the outgoing key BEFORE the new+key goes live, keep it there until an audit run reports zero undecryptable+wallets, and only then drop it. A rotation without that overlap is a one-way door,+and this brief is what the far side of one looks like.diff --git a/scripts/audit-docs.mjs b/scripts/audit-docs.mjsindex f8afca16c..dbda94763 100644--- a/scripts/audit-docs.mjs+++ b/scripts/audit-docs.mjs@@ -308,6 +308,7 @@ const UNPUBLISHED_DOCS = new Map([['nvidia-apps-catalog-listing', 'internal: paste-ready listing copy and portal steps for the NVIDIA Inception catalog'],['nvidia-apps-catalog-request', 'internal: outbound email asking NVIDIA to publish the catalog listing'],['nvidia-visibility-map', 'internal: outreach priorities and unclaimed program benefits across every NVIDIA surface'],+ ['nvidia-ngc-listing', 'internal: prerequisite audit and paste-ready listing copy for the NVIDIA NGC catalog submission'],['nvidia-forum-browser-digital-human', 'internal: forum post drafted for the NVIDIA Developer Forums, owner-gated until posted'],['openai-listing-channels', 'internal: submission strategy and a candid post-mortem of our own stalled Cookbook PR'],['okx-marketplace', 'owner-gated: names a crypto project other than $THREE'],