Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
Production Issues — three.ws
Live tracker for known production issues. When an item is fixed, drop it from this file rather than leaving it here marked ✅ — this file should only contain work that is still open.
The 2026-05 incident batch (20 items, all resolved) and the 2026-06 resolved items were archived under
docs/internal/, which was removed from the repo along with the rest of the internal content in commitb80c1c379. Both archives remain readable in git history:git show b80c1c379^:docs/internal/ISSUES-ARCHIVE-2026-05.md
Check the deploy gap before debugging anything here
curl -s https://three.ws/api/version returns the live commit; compare it to
main. Reading the repo tells you what SHOULD be true in production, not what
is: this list has more than once carried confident claims about live mitigations
that had never been deployed.
Deployed 2026-07-30: production moved d6a7bf2a4 (2026-07-28) to bbe5d2403,
closing a ~70 commit gap. That alone fixed three items that had been sitting
here as code-complete, each verified live afterwards rather than assumed:
ENS resolution, web search running on scraps, and the stale accuracy number the
/fact-check page was publishing. Post-deploy sweep was 465/466 pages
(the one miss was a blog route that landed 6 minutes after the built commit).
Deploys are owner-gated (CLAUDE.md gate 2), so a correct fix can sit unshipped
indefinitely. When an item says "needs deploy", no further engineering is
required. npm run db:status reports all migrations already applied.
Open
Re-verified against the live Cloud Run services on 2026-07-22. The 2026-07-04
log-export batch is now mostly closed: JWT_SECRET and WALLET_ENCRYPTION_KEY
are set on three-ws-api (wallet sign-in and custodial provisioning work),
ADMIN_CODE is set on hyperfy-world (world.three.ws build rights are gated
again), and the reconstruct lane has a real fallback via
GCP_RECONSTRUCTION_URL. What remains requires owner/operator action outside
this repo:
- LLM paid backstops are dead (owner billing decision).
OPENAI_API_KEYis set on the service but the account returnsbilling_not_active429s, so every OpenAI paid backstop is out. The OpenRouter platform key burned its credit on paid-model routing (invisible in metering: llm-pricing records OpenRouter as $0). Traffic survives on the free lanes (Groq/NIM plus the vertex-gemini credits anchor). Action: reactivate OpenAI billing and/or fund the OpenRouter key, or accept free-lane-only service. - Every paid Solana RPC lane is over quota at once (owner action: money).
Re-probed 2026-07-30 with a metered
getBalance(nevergetHealth, which is unmetered and returns ok on an exhausted endpoint): Helius-32429 max usage reached, QuickNode-32003 daily request limit reached, and BOTH Alchemy apps (ALCHEMY_API_KEYand the key insideSOLANA_RPC_FALLBACK_URLS) return429 Monthly capacity limit exceeded(they share one account quota, so the second is not a spare). The dead Alchemy endpoint was also pinned asSOLANA_RPC_URL, so every production Solana call began by failing over. Mitigated in config on 2026-07-30 (current revisionthree-ws-api-00346-x9m):SOLANA_RPC_URLpoints athttps://rpc.magicblock.app/mainnet, withapi.mainnet-beta.solana.com, PublicNode and Leo RPC as the fallbacks. Pick the primary by call SHAPE, not by agetBalanceprobe. All the free lanes passgetBalance,getLatestBlockhashandgetSignatureStatuses12/12, which makes them look interchangeable. They are not: ongetTokenAccountsByOwnerfiltered byprogramId, the call every token and USDC balance reader makes constantly, PublicNode returns HTTP 403blocked parameter: params.1.programIdand Leo RPC returns-32603. Only MagicBlock andmainnet-betaserve it, which is why they hold the top two slots. Promoting PublicNode is actively harmful before commit61f3ae758ships, because that 403 was sized as an auth failure and parked the whole node for 30 minutes on ordinary traffic. The exhausted paid lanes stay in the chain and re-enter rotation on their own when quota resets (QUOTA_COOLDOWN_MSis 6h). The platform is therefore up but throttled, which shows up as intermittent 5xx and slow settles. Action: top up or upgrade a plan, or cut call volume via the ring cadence knobs. Note GCP credits cannot help here: Blockchain Node Engine is Ethereum-only, so there is no GCP-hosted Solana RPC. Partly relieved 2026-08-12: a fresh Helius key on a new free-plan account is now set asHELIUS_API_KEYin the local.env. Probed with metered calls only, it serves 6 of the 7 production call shapes at p50 33 ms, the fastest lane in the chain, andsolanaRpcEndpoints()places it first, ahead of the keyless free tail (the one miss isgetProgramAccounts, which every other lane also refuses). Its Helius-only surfaces answer too: DASgetAssetandgetTokenAccounts,getPriorityFeeEstimate, the/v0parsed transaction history, and the webhooks API. Free plan means the quota is modest, so this restores the DAS and holder surfaces rather than replacing a paid lane. Production still runs the exhausted key until the Cloud Run env update lands:gcloud run services update three-ws-api --region us-central1 --project aerial-vehicle-466722-p5 --update-env-vars HELIUS_API_KEY=<key>(single-key merge, never--set-env-vars).
Carried forward from the retired prompts/x402-catalog/ tracker (campaign
closed 2026-07-28; full history in git):
Fact-check benchmark cannot be re-run from this machine (owner action: one credential). The two halves of this item that were about code and about the deploy are both closed, verified live on 2026-07-30 after the deploy: web search really does lead with Vertex-grounded Google Search now (a live call to
POST /api/x402/fact-checkreturnedsupportedat 0.98 confidence withvertexaisearch.cloud.google.comgrounding sources, not DuckDuckGo scraps), andGET /api/fact-check-benchmarknow returnsran: falseinstead of serving the stale 2026-07-08 run, so/fact-checkrenders its honest "not yet run" state to visitors rather than a misleading 20%. What remains is only the measurement.scripts/fact-check-benchmark.mjstargets the PAID endpointhttps://three.ws/api/x402/fact-check, which answers a small free allowance and then 402s, so a 40-claim run needs a bypass. Neither bypass exists: there is no token with thex402:bypassOAuth scope, andINTERNAL_API_KEY(thex-api-keyservice path inapi/_lib/x402/access-control.js) is unset both locally and on thethree-ws-apiCloud Run service. Action: setFACT_CHECK_BYPASS_TOKEN, or setINTERNAL_API_KEYon the service and pass it asx-api-key. Thennode scripts/fact-check-benchmark.mjs. The runner refuses to publish any run with >10% errored claims, so a degraded run cannot poison the page.Replicate is an unfunded optional image lane (owner action, non-blocking). Narrowed on 2026-07-30: two of this item's three original claims were stale and are struck. ASR works,
NVIDIA_ASR_FUNCTION_IDis set andGET /api/v1/ai/asrreturns 200configured: true, with a real 16kHz WAV round-tripping in 1.5s viariva-asr. Redis is present, not absent:UPSTASH_REDIS_REST_URLpoints at a self-hosted SRH proxy on the VPC, and the ASR quota counter decrementing proves the store is live. OnlyREPLICATE_API_TOKENis genuinely absent, andapi/_lib/ai-image-lanes.jstreats Replicate as one of three lanes with Vertex serving the traffic, so nothing is degraded. Fund it only if a third image lane is wanted.BSC testnet contracts are code-complete but never publicly deployed (owner action; re-homed here 2026-07-30 when the BNB Chain campaign's work orders were retired). Every contract proof in that campaign ran against an anvil fork or a local instance because no funded deployer key exists:
BNB_TESTNET_DEPLOYER_KEYis absent from the shell env, the root.env, andcontracts/.env, and the public faucet is reCAPTCHA-gated, so theGreenfieldVaultandWorldMovesdeploys and every real Greenfield write never happened. The deploy scripts are dry-run verified and unchanged (forge script script/DeployGreenfieldVault.s.sol/script/DeployWorldMoves.s.sol --broadcast, both documented in contracts/DEPLOYMENTS.md). Action: fund a throwaway EOA athttps://www.bnbchain.org/en/testnet-faucet, setBNB_TESTNET_DEPLOYER_KEY(andGREENFIELD_VAULT_OPERATOR_KEY, which falls back to the same key), re-run the two scripts, then setWORLD_MOVES_ADDRESS_TESTNET. No code change is needed: the moment the address exists,/api/bnb/world-configstarts returningdeployed:truefor real visitors and the sender/reader/ghost paths light up as already proven. Full history:prompts/bnb-chain/PROGRESS.md.The x402 sponsor has under a day of runway (owner action: ~1 SOL). Re-measured 2026-07-31: the sponsor (
WwwuGbqHrwF5RG89KhUbmRWEvjnRH9k5kVM5p7T3WwW) holds 0.0318 SOL, barely above its 0.03 floor, against a measured burn of 0.060 SOL/day (7 days offee_lamportsover successful settles): roughly half a day of runway. Do not quote a remembered burn rate here; the previously circulated "1 to 2 SOL/day" was wrong by roughly 10x. ~1 SOL covers about 16 days at the current burn. The number is now live on the payment-outcome board:GET /api/ops/payment-outcomesshows balance vs floor, measured burn, and runway (seedocs/ops/payment-outcomes.md; ships with the next deploy). Never top up per-agent wallets, that strands SOL and kills the rail. Related but already handled: the earlierfee_wallet_below_flooroutage (18:00 to 02:45) was a symptom of item 2, not an independent fault, and cleared with no funding the momentSOLANA_RPC_URLwas repointed. Do not lowerX402_SPONSOR_SOL_FLOOR_LAMPORTS; the shortfall was 0.00018 SOL and the floor was doing its job.x402_settlereports degraded, and what remains is rail faults (closes with item 2). Re-read 2026-07-31 01:10 UTC: the sensor is DOWN at 30.3% (549/1813 paid attempts, 3h) with the faults dominated byhttp_502(1055) andhttp_402(153), so the degradation has deepened since the repoint; the free lanes alone are not holding the rate. Earlier readings: near 78% on 2026-07-30, 95.2% in the hour right after the RPC repoint. Every current failure is RPC-shaped rather than financial:broadcast_failedwith empty simulation logs,Blockhash not found, and malformed-response parse errors. The test that settles this is grouping failures by payment amount, because they are amount-independent and the smallest bucket has the worst ratio, which insufficient funds cannot produce. Free lanes cap the achievable rate at roughly 82-95%, so restoring one paid RPC plan is the fix.The OKX marketplace chat bot is logged out (owner action: one login). Chat for OKX agent #2632 is delivered by a local
okx-a2adaemon plus anonchainoswallet session, both outside this repo. The daemon is staged and running with skills linked; only the wallet session is logged out, and finishing it needs a human (email OTP as[email protected]). Runnpm run okx:botfor the current login URL and the follow-up poll command: exit 0 means online, exit 2 means staged but logged out. A codespace cannot stay up on its own, so an always-on host is the durable fix.
Found by the 2026-07-30 documentation audit, which read the handlers behind every surface it documented. Only the production-affecting findings are listed here; the code-quality items from that pass are not production issues.
Recently closed (do not re-open without new evidence)
Kept briefly because each one was previously mis-stated on this list, and the wrong version is what a future reader would otherwise trust.
Live R2 CORS did not match
scripts/set-r2-cors.mjs(filed 2026-07-30, fixed and closed 2026-09-10). The bucket served one legacy allowlist rule for both reads and writes, so a foreign origin got the bytes with noaccess-control-allow-originand itsOPTIONSpreflight was a bare403. Every prior pass recorded this as blocked on one owner-minted credential, on the finding that the only reachableS3_*pair was object-scoped and403 AccessDeniedon Get/PutBucketCors. That finding no longer holds. The pair resolved off the Cloud Run service withnode scripts/read-service-env.mjs '^S3_(ACCESS_KEY_ID|SECRET_ACCESS_KEY|BUCKET|ENDPOINT|PUBLIC_DOMAIN)$'(which follows the Secret Manager reference thatdescribeshows asvalueFrom) is admin-scoped:--getread the live policy and the apply path wrote the new one. No token was minted. The canonical policy is live, split as the script always intended into a world-openpublic-readrule (GET/HEAD,*) and an origin-lockedbrowser-uploadrule (PUT). Verified 2026-09-10 by rawcurlon a real object and bynode scripts/set-r2-cors.mjs --probe, which now exits 0 with every origin matching expectations: foreign origins read and cannot write, andwww.three.ws,*.app.github.devandlocalhost:5173write again. That last row also retires the first-party breakage filed beside this item, where the www host served the app but had every GLB blocked by the browser; theserver/index.mjswww-to-apex redirect remains correct and ships independently. The/cdn/<key>and/api/glbroutes stay the right advice for callers that cannot set an origin, and were not touched.Object storage rejected our credential (filed and closed 2026-09-09). For part of 2026-09-09
/api/healthzreported theobject_storagesubsystemdownwithcredentialFault: trueand the bucket's ownSignatureDoesNotMatch: the access key id was accepted, so the secret was what was wrong. Uploads answered503 storage_unavailable, both library manifests answered{"total":0}, and forge image generation was 0/30 over six hours.S3_SECRET_ACCESS_KEYonthree-ws-apiis a Secret Manager reference (secret:s3-secret-access-key:latest) and rotating it is owner-gated from an agent machine, so the item was filed for the owner. Re-measured the same day and every symptom is gone:object_storageisok(signed read ok, 216ms),POST /api/forge-uploadreturns200with a presigned URL,/api/avatars/libraryand/api/objects/libraryreturn 107 and 511 entries, andforge_generationisokat 95% (60/63 finished, 6 hours). Nothing in this repo changed: on Cloudflare R2 the secret access key is the API token's SHA-256 digest, and a wrong or newline-padded value fails exactly this way, so if it recurs the fix is that one secret, never the code./api/avatar/optimize?draco=1(closed 2026-08-01). It returned500 transcode_failedwithdraco.createCompressedPrimitive is not a functionbecause the deployed image had resolved an older@gltf-transformthan the pinned^4.4.0. This was always dependency drift in the image, never a code bug. The 2026-08-01 rebuild fixed it, verified live against production:curl "https://three.ws/api/avatar/optimize?src=https://three.ws/avatars/cesium-man.glb&draco=1"returns200with a 129,972-byte GLB stampedglTF-Transform v4.4.0. The two sibling defects filed under the same item also shipped: the silent-expression report (x-render-expressionon/api/avatar/render, pinned bytests/api/avatar-render-expression-report.test.js) and the oversized-source stall (streaming 50 MB cap returning413/504, covered bytests/avatar-optimize-source-cap.test.js).A second, unrelated defect on the same endpoint was found once the 500 was gone and is also closed (2026-08-13):
?draco=1returned a file BIGGER than its input (default.glb 748,088 to 890,160, +19.0%; michelle.glb 849,756 to 974,036, +14.6%) and said nothing about it. Cause: stored avatars are quantized and meshopt-packed, gltf-transform keepsEXT_meshopt_compressionattached after reading, so Draco was layered beside the meshopt payload and re-quantized already-quantized attributes. The pipeline now drops the other mesh-compression scheme and dequantizes before encoding, keeps whichever encoding is actually smaller, and falls back to the original bytes when nothing helped.x-three-ws-optimize/x-three-ws-optimize-refusedreport which happened. Pinned bytests/avatar-optimize-never-inflates.test.js.The autopilot spend default.
POST /api/agents/:id/autopilot/runreadbody?.dry_run === true, so a barePOST {}ran a REAL cycle while the sibling runner on the same wallet (api/agents/wallet-intents.js) readdry_run !== falseand simulated. Nowbody?.dry_run !== false: silence simulates, and spending must ask for itself. The cockpit's Run button relied on the unsafe default, so it now sendsdry_run: falseexplicitly and asks a second time before spending; a Preview cycle button runs the simulation. Pinned bytests/autopilot-run-dry-run-default.test.js(3 of its 6 cases fail against the old expression).The
/pulsemarketplace GMV. This list said "a confirmedtrialrow with a non-zero amount inflates published GMV". That mechanism never occurred: a trial is writtenstatus='trial', kind='trial'and only a'pending'row is ever promoted to'confirmed', so thestatus='confirmed'filter already excluded every trial. Live data agrees: all 10,454skill_purchasesrows arekind='trial', status='trial', and there is not one confirmed purchase yet. The defect was real but latent: the money aggregates filtered on status alone while the counts filtered on status AND kind, so they rested on an invariant nothing enforced, and a trial carries the listing's FULL price inamountdespite nothing being paid. Every aggregate now derives from one sharedpaidRowpredicate,series_7dincluded. Do not re-file this as an observed inflation of a published number; it was never that.Oversized sources on
/api/avatar/optimize. See item 9.Neon storage pressure.
isStoragePressured()reportspressured: falseat 2768MB, andSHOW neon.max_cluster_sizereturns 16TB, so the 8192MB high-water on the service is a deliberate runaway backstop rather than a cap being approached. Worth watching only because the branch grew ~230MB in a day.ASR and Redis. See item 5: both were listed as unconfigured and both work.
World blueprint assets. The
worldsubsystem reported "1 blueprint asset(s) missing" while all 17 distinct assets served 200 to an independent sweep. The sensor swept the blueprint list verbatim (136 entries for 17 files) and treated a single transient HEAD timeout as a hard 404, parking a degraded verdict for the full cache hour. Now it sweeps distinct assets, retries a transient failure once, logs which URL is implicated, and only a 404/410 counts.worldreports ok.Three shadowed agent routes.
GET /api/agents/ens/:name,GET /api/agents/8004/agentandGET /api/agents/8004/searchall answered the agent-profile dispatcher's "agent not found": the broad/api/agents/([^/]+)(?:/.*)?route sat above them and, being pre-filesystem, the filesystem phase never saw them either.api/agents/ens/[name].jshad never served a request, and/demos/erc8004could neither search the registry nor open an agent. Routes added above the catch-all and pinned bytests/vercel-agents-subpath-routes.test.js. Ships with the next deploy.The seed cron's 409 storm. Every payment in a batch was built from identical inputs with a fixed priority fee, so all N serialized to the same bytes, collided on one hashed payment id, and 409'd on all but one (about 40 of 41 lost per tick, the largest single source of 4xx on the fleet). The compute unit price now varies by index. Cover:
tests/x402-seed-tx-uniqueness.test.js.Redis proxy SIGBUS.
three-ws-redis-proxywas crashing on signal 10 at 512Mi, which surfaced as cache SET timeouts and rate-limiter degradation while healthz still readcache: okbecause the memory fallback masked it. Raised to 1Gi.The bridge-down alert carried no information. Its message escaped its own template interpolations, so it logged the literal source text instead of the count and the bridge names.