Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

docs(community): correct the OpenAI forum draft with the partner status, live rig conventions, poll-time failover, and honest OKX listing state

nirholasf27093d1e69b
diff --git a/docs/openai-community-3d-studio-post.md b/docs/openai-community-3d-studio-post.md
index f1c86f9a7..040b4a6ba 100644
--- a/docs/openai-community-3d-studio-post.md
+++ b/docs/openai-community-3d-studio-post.md
@@ -4,7 +4,7 @@ _Forum post for the [OpenAI Developer Community](https://community.openai.com),
---
-Hi all. I build [three.ws](https://three.ws), an open-source (Apache-2.0) browser-native 3D AI platform. For the last few months a big part of my time has gone into making it usable from inside ChatGPT, and I wanted to write up the whole thing properly for this community: what the integration looks like on the OpenAI side (a ChatGPT app over the Apps SDK connector, and a custom GPT over Actions), what actually happens on my side when a prompt comes in (a fleet of GPU workers running open 3D models), the parts that were hard, and where it is all going. It is long. The short version is at the top; the rest is for people who want the details.
+Hi all. I build [three.ws](https://three.ws), an open-source (Apache-2.0) browser-native 3D AI platform. three.ws is a Select Partner in the OpenAI Partner Network; the usual caveat applies, it is not an OpenAI product and not endorsed by OpenAI beyond that designation. For the last few months a big part of my time has gone into making it usable from inside ChatGPT, and I wanted to write up the whole thing properly for this community: what the integration looks like on the OpenAI side (a ChatGPT app over the Apps SDK connector, and a custom GPT over Actions), what actually happens on my side when a prompt comes in (a fleet of GPU workers running open 3D models), the parts that were hard, and where it is all going. It is long. The short version is at the top; the rest is for people who want the details.
**Short version:** type "a low-poly orange fox sitting down" into ChatGPT and about a minute later you get a real, textured, downloadable GLB, an interactive 3D viewer, and a link that puts the model in your room in AR on your phone. Optionally the model gets a humanoid skeleton, skin weights and 52 ARKit blendshapes, and then it walks, waves and talks. All of it is free, no account, no API key. The same pipeline is reachable from Claude, Cursor, curl, or any MCP client, and the whole stack (frontend, API, workers, contracts) is public on GitHub.
@@ -134,18 +134,20 @@ Generation runs on a fleet of single-purpose Python FastAPI services on Google C
Around the mesh generators sit the processing workers:
-- **rembg**: BRIA RMBG-2.0 with U2Net as a CPU fallback, cleaning source photos before reconstruction.
+- **rembg** (CPU): ONNX salient-object models (`isnet-general-use` by default, `u2net`, `u2net_human_seg`, `silueta`), cleaning source photos before reconstruction.
- **remesh** (CPU): repair, simplify (quadric decimation), retopologize (QuadriFlow), UV unwrap (xatlas), and format conversion across GLB, OBJ, FBX, STL, PLY, USDZ and 3MF. Headless Blender handles FBX so a rigged GLB converts with its bone hierarchy, skin weights and blendshapes intact, which is what Unity and Unreal users need.
- **stylize** (CPU, pure geometry, no inference): `voxel`, `brick` (voxels plus studs), `voronoi` (an open strut-and-node lattice shell), `lowpoly` (decimated with hard flat-shaded facets). Source color is preserved per output element where the style allows.
- **segment** (CPU, pure geometry): splits a model into named, separable parts. Connected-component splitting handles physically disjoint shells; the minima rule (region growing that cuts the face-adjacency graph at concave creases, where human perception sees part seams) handles the rest. Each part becomes a named node in the output GLB with a manifest (id, name, region, bbox, centroid, face counts, color) so you can hide, recolor, replace or export it individually.
- **texture** (GPU): full retexture from a prompt by rendering depth from 8 canonical viewpoints, generating coherent views with SDXL plus ControlNet-Depth, rasterizing the mesh into UV space and back-projecting each view through the camera it was rendered with, weighting texels by how squarely each view sees them and refusing the ones the depth buffer says are occluded. A magic-brush mode inpaints only a masked UV region and feathers the seam so untouched texels stay bit-identical across passes.
- **rig** (GPU): Make-It-Animatable. Takes a raw generated mesh and adds a 52-bone Mixamo-named humanoid skeleton, per-vertex skinning weights, and ICT-FaceKit ARKit-52 blendshapes. The rig is grafted into the original GLB bytes, so the materials and PBR textures the generator produced survive untouched. This is what `rig_mesh` and `text_to_avatar` call.
-- **avatar-reconstruction** (GPU): the selfie lane. 1 to 6 face photos, six multi-view renders synthesized with Zero123++, InstantMesh reconstruction, and a newer path that morphs a rigged template head to the person's 468 MediaPipe landmarks and projects the photo onto its skin.
-- **text2motion**, **video2motion**, **video2scene**: text to an animation clip, a phone video to a retargetable motion clip, a video to a scene.
+- **avatar-reconstruction**: the selfie lane. It started as InstantMesh over six Zero123++ multi-view renders; since late July it runs CPU-only and morphs a rigged template head to the person's 468 MediaPipe landmarks, then projects the photo onto its skin. A separate CPU controller orchestrates the "scan yourself" path: it picks a mesh backend by weighted random from the pool, reconstructs, auto-rigs, and uploads.
+- **text2motion** (GPU): samples a motion-diffusion model from a prompt and returns a three.js `AnimationClip` JSON on the canonical skeleton, the same format as the curated library, so it retargets like any other clip. **video2motion** (CPU): a video of a person becomes a retargetable clip; the same capture path produces the lexical clips behind sign-language playback. **video2scene** (GPU): streaming video to a 3D point cloud, wrapping LingBot-Map (a feed-forward geometric context transformer with paged-KV attention).
- **garment-forge**: clothing generation for avatars.
- **longcat**: LongCat-Video-Avatar-1.5 for lip-synced talking-avatar video from a reference image plus audio. Built and tested; it needs an 80 GB-class GPU (about 45 GB resident even at int8), so it does not fit Cloud Run's L4 and runs on a dedicated A100/H100 instance.
-Every lane has a failover chain. A single model being unavailable degrades quality gracefully instead of failing the request, and a submit that fails over never double-charges on the paid rail. Worker directory with each service's contract: [workers/](https://github.com/nirholas/three.ws/tree/main/workers).
+Routing is free-first: a paid lane must never run while a healthy free lane can serve the request. The candidate order is the tier's free default (self-hosted TRELLIS for draft and standard, Hunyuan3D for high), then the per-path free chain (self-hosted TRELLIS, Hunyuan3D, the Hugging Face Spaces chain, NVIDIA NIM's hosted TRELLIS, which is text-only because it rejects user photos), then a paid platform lane only when nothing free is configured and healthy. At high tier a small prompt classifier reorders the pool: hard-surface subjects (vehicles, machines, architecture) hoist TRELLIS above Hunyuan3D, organic subjects keep Hunyuan3D first. Lane health is tempered by per-lane Redis circuit breakers (90 s cooldowns), and the UI disables a down lane with the real upstream reason before the user clicks Generate.
+
+Failover happens at two moments. At submit time, a lane rejecting the job (down, quota, cooldown) makes the orchestrator walk to the next free candidate inside the same request. At poll time, which is the interesting one: a lane can accept a job and die minutes later, and the poll that notices may land on a different server instance. The job handle is an opaque HMAC-signed token encoding provider and upstream task id, so polling is stateless across instances, and a dead job is silently re-dispatched from the stored prompt and reference image, with the old handle bound to the new one in Redis. The client keeps the same job id and just sees `status: running` with a new `backend`. Three hops maximum, then a terminal failure that carries `retryable: true` and an ordered list of lanes for one-click resubmit. A submit that fails over never double-charges on the paid rail; the paid order is verify, submit, settle, so a failed submit never charges at all. A quality gate flags degenerate results (near-empty geometry, texture failures) and synchronous lanes retry once inline. Results are cached for seven days keyed on path, tier, backend, normalized prompt and seed, and every user-facing URL points at durable storage, never at a provider URL that expires in an hour. Worker directory with each service's contract: [workers/](https://github.com/nirholas/three.ws/tree/main/workers).
One practical detail that bit me more than once: most three.ws avatars ship with `EXT_meshopt_compression`, which trimesh cannot decode. Any worker that loads a caller's mesh routes it through a meshopt decode step first (using a pinned `gltfpack` binary shipped in the image). A new mesh-consuming worker inherits that requirement or it fails on the first real input.
@@ -168,8 +170,9 @@ So "optimize this idea, generate it, rig it, and make it wave" is `direct_prompt
Anyone who has shipped 3D characters knows this pain: a model rigged in one ecosystem rarely animates correctly in another. I treated it as an infrastructure problem.
-- **Skeleton canonicalization.** A canonicalizer maps the bone-naming conventions of the major ecosystems (Mixamo, Avaturn, Unreal, VRM/VRoid, VRM 1.0, Daz/Genesis, MakeHuman, Blender `.L/.R` suffixes, plain `shoulderL`-style rigs) onto one canonical humanoid skeleton. There is deliberately no allowlist of supported rigs. A new convention is a bone-name mapping to add plus a test case, not a feature to request.
-- **Retargeting.** A retargeting layer drives a shared clip library (idle, walk, run, gestures, dances, a seated idle for chat mode) onto any canonicalized rig, legs included. A rig that genuinely cannot be skeleton-driven (no skin, a non-humanoid prop) falls back to the default body rather than a bind-pose T-pose.
+- **Skeleton canonicalization.** A pure-JS canonicalizer (runs in Node and the browser) rewrites joint names in an uploaded humanoid GLB to one canonical skeleton and repacks the GLB in place. Conventions it maps today: Mixamo (all three prefix spellings), Blender `.L/.R` and `Armature_` prefixes, Rigify `DEF-/ORG-/MCH-`, CharacterStudio `CH_`, HumanIK/Maya namespaces, the Unreal mannequin (`pelvis`, `clavicle_l`, `calf_l`), VRM 0.x/VRoid `J_Bip_*`, VRM 1.0 camelCase, MMD's Japanese bone names, Daz/Genesis, MakeHuman, SMPL/SMPL-X, Roblox R15/R6, Second Life `mPelvis`-style, anatomical scan-kit names (`humerus.L`, `femur.L`), Reallusion CC3/CC4 twist joints, and plain `shoulderL` rigs. There is deliberately no allowlist. A new convention is a bone-name mapping plus a test case, not a feature to request. Two things fall through on purpose: non-humanoid skeletons (quadrupeds, props), and constraint-driven control rigs (the Auto-Rig-Pro family), whose deform bones hang off tracker nodes rather than forming a chain; Blender's constraints have no glTF equivalent, so mapping that layer would tear the arm apart, and the fix for those uploads is re-rigging.
+- **Fingers are load-bearing.** 30 of the 53 tracks in every library clip address a finger joint. A rig whose hands do not name-map scores around 40 percent coverage, falls under the 50 percent minimum, and gets no animation built at all rather than a twitching partial one. So the finger spellings of every convention above are mapped too, and a sweep script measures coverage per rig.
+- **Retargeting.** The library is 112 pre-baked clips (idle, walk, run, gestures, dances, a seated idle for chat mode, an emote family, a farm-task family). Retargeting rewrites each track's bone name to the target's node, drops tracks the rig lacks, rescales hip translation so root motion lands at the new rig's height, and retargets only true-motion channels (joint rotations and root translation), skipping per-bone position and scale channels that encode the authoring rig's bone lengths and would crush a differently-proportioned avatar into a heap. Bind-pose correction handles the A-pose versus T-pose difference and the Mixamo/FBX up-axis bake on Hips; retargeting the reference rig onto itself yields identity corrections per bone, so a matching rig round-trips byte-for-byte. It binds in `THREE.AnimationMixer` for preview and `THREE.GLTFExporter` for animated-GLB export, and ships standalone as `@three-ws/retarget`. A rig that genuinely cannot be skeleton-driven falls back to the default body rather than a bind-pose T-pose.
- **Slot-based animation manager.** Clips are decoupled from rigs and blended per slot (idle, gesture, locomotion) with crossfades, driven by a state machine that listens to protocol events. A new clip is authored against any rig in Blender, exported, and dropped into the library; the runtime picks it up and the agent can invoke it by name.
- **Asset formats.** Every avatar is a GLB (body, rig, textures in one file); every shared gesture is a format-light clip JSON (a serialized `THREE.AnimationClip`, motion only, retargeted at runtime). The web build runs `gltfpack` for meshopt compression, which cuts a typical avatar by about 90 percent.
- **Rig Doctor** ([three.ws/rig-doctor](https://three.ws/rig-doctor)) reports in seconds whether a given GLB will animate and why, which turned "my model is broken" support threads into a link.
@@ -199,7 +202,7 @@ A generated avatar is not locked into the platform. `<agent-3d>` is a framework-
The frontend philosophy is worth stating: vanilla JavaScript modules built with Vite, three.js for rendering, Rapier compiled to WebAssembly for physics, Draco/KTX2/Meshopt decoders, no heavyweight framework. That is what makes the embed story credible; the components carry no runtime tax onto host pages. The viewer targets every browser with WebGL 2.0 and refuses to boot without it rather than rendering a broken canvas.
-Fifteen packages ship from the repo to npm under the `@three-ws` scope. The 3D-relevant ones: `@three-ws/avatar` (viewer, creator iframe, AR/VR runtime, React bindings), `@three-ws/agent-ui` (an avatar overlay that reacts to buttons, inputs and navigation), `@three-ws/avatar-schema` (JSON Schema plus validator for avatar manifests), `@three-ws/viewer-presets` (light rigs, floor reflections, bloom), `@three-ws/avatar-cli` (scaffold, validate, hash and preview manifests from a shell or CI), `@three-ws/walk` (a corner mascot that walks your page, plus a full-page playground). The MCP servers are in the official MCP registry under `io.github.nirholas`. Load-bearing wire formats are written up as specs: the agent manifest, the `<agent-3d>` element and its `postMessage` host protocol with origin lock, the skill bundle layout, the stage/scene configuration, the memory format.
+Dozens of packages ship from the repo to npm under the `@three-ws` scope. The 3D-relevant ones: `@three-ws/avatar` (viewer, creator iframe, AR/VR runtime, React bindings), `@three-ws/agent-ui` (an avatar overlay that reacts to buttons, inputs and navigation), `@three-ws/avatar-schema` (JSON Schema plus validator for avatar manifests), `@three-ws/viewer-presets` (light rigs, floor reflections, bloom), `@three-ws/avatar-cli` (scaffold, validate, hash and preview manifests from a shell or CI), `@three-ws/walk` (a corner mascot that walks your page, plus a full-page playground). The MCP servers are in the official MCP registry under `io.github.nirholas`. Load-bearing wire formats are written up as specs: the agent manifest, the `<agent-3d>` element and its `postMessage` host protocol with origin lock, the skill bundle layout, the stage/scene configuration, the memory format.
## 6. The agent layer: giving the body a brain, an identity and a wallet
@@ -211,7 +214,7 @@ This is where three.ws goes beyond "a 3D generator". The founding idea is: give
- **Payments (x402).** Agents pay other agents per request in USDC using the HTTP 402 pattern: an unpaid call returns `402` with a machine-readable price, an x402-capable client pays and retries, and settlement happens on-chain only after the call succeeds. The paid twin of Forge (`POST /api/x402/forge`) charges $0.05 draft, $0.15 standard, $0.50 high, polling is free, and retried payments are idempotent. Pay-by-name resolves `@username`, `*.sol` names and raw addresses. There is a SKU catalog, subscriptions, receipts, a bazaar search API, and an A2A (agent-to-agent) bridge that exposes paid MCP tools as x402 endpoints. Any signed-in user can mint `[username].threews.sol` in one atomic Solana transaction with three.ws paying gas.
- **Worlds.** At `/play` every Solana token community gets a persistent multiplayer 3D world derived from its mint address, with proximity voice chat, collaborative voxel building, an in-game economy, vehicles, and a shared deterministic day/night clock so two people in the same world always agree on whether it is night. `/temporary` walks your avatar through six staged environments with heightfield physics and live peers. `/club` is a multiplayer venue with rigged dancers. The multiplayer backend is a Colyseus server.
- **Open inference network.** Anyone can run a node: the node-operator client registers with the coordinator using a signature under its own ed25519 key, claims jobs from a queue, runs a real open model on CPU or CUDA, and returns results signed over a canonical receipt binding job, model and input/output hashes, which the coordinator recomputes before accepting. Paid inference settles through the x402 rail and the platform signing key is published so receipts verify offline.
-- **$THREE.** The platform's token on Solana (contract `FeMbDoX7R1Psc4GEcvJdsbNbZA3bfztcyDCatJVJpump`). Holders follow a public changelog that is pushed to Telegram automatically after every deploy; there is also a small MCP server that treats burning it as an agent primitive.
+- **$THREE.** The platform's token on Solana (Token-2022, contract `FeMbDoX7R1Psc4GEcvJdsbNbZA3bfztcyDCatJVJpump`). It gates the high Forge tier (hold or pay) and the game-ready export, discounts plans, and a micro-buy loop turns settled x402 calls into small market buys. Holders follow a public changelog that is pushed to Telegram automatically after every deploy.
## 7. Infrastructure, honestly
@@ -229,12 +232,14 @@ The repo is one npm-workspaces monorepo. A `STRUCTURE.md` maps every product sur
## 8. Partnerships and where it lives
- **OpenAI**: the ChatGPT app over the Apps SDK connector and the "three.ws 3D Studio" custom GPT in the GPT Store, described above.
-- **Anthropic**: the same pipeline runs from Claude Desktop, Claude Code and Claude artifacts through the MCP servers.
-- **IBM**: three.ws is an IBM Business Partner; Granite models on watsonx.ai run inside the studio, and `@three-ws/ibm-x402-mcp` is the first x402-metered MCP server on IBM Cloud (a few cents of USDC per Granite call, no IBM account required). There is a three.ws user group on IBM Community that held its first in-world meetup at `/play` in August with a peak concurrency of 3,145.
+- **Anthropic**: the same pipeline runs from Claude Desktop, Claude Code and Claude artifacts through the MCP servers. Related: **Spatial MCP**, a CC0 response shape that makes a 3D scene a native MCP result instead of a URL in text; three.ws is the reference implementation and it is renderer-agnostic.
+- **IBM**: three.ws is an IBM Business Partner; Granite models on watsonx.ai run inside the studio, and `@three-ws/ibm-x402-mcp` is an x402-metered MCP server over Granite (a few cents of USDC per Granite call, no IBM account required). There is a three.ws user group on IBM Community that held its first in-world meetup at `/play` in August with a peak concurrency of 3,145.
- **AWS**: AWS Partner (APN Software Path), marketplace SaaS listing in review, engineering write-ups on the AWS Builder Center.
- **Alibaba Cloud**: live marketplace listing.
- **Google Cloud**: production runs there, GPU inference included.
-- **OKX**: three.ws Forge is listed as paid services on the OKX.AI agent marketplace (agent #2632), sold to other agents over A2MCP, with an always-on XMTP chat bot host.
+- **OKX**: three.ws Forge is built as a paid service for the OKX.AI agent marketplace (agent #2632), sold to other agents over an OKX-dialect 402 with an always-on XMTP chat bot host. The listing itself is in resubmission after a first review; the endpoints are reachable directly today.
+- **NVIDIA**: Inception member. NIM-hosted TRELLIS is a free forge lane, NIM is a rung in the LLM chain and the FLUX text-to-image lane, and Audio2Face-3D drives visemes in the voice package.
+- **Quicknode** (startup program, Solana RPC) and **HackerNoon** (news syndication).
- **Solana Mobile**: the Seeker wallet (MWA) is wired into the web app and there is a Solana dApp Store release pipeline.
- **BNB Chain**: indexed on Dappbay under AI Agent Launchpad, AI Data and AI Infra.
- **Pump.fun**: a launcher and a set of 23 free read-only MCP tools for token and Solana data.