Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
Selfie to Avatar: scan your face into a rigged 3D character
Point your camera at your face, hold still for a countdown, and about a minute later you have a rigged, animation-ready 3D avatar in the browser. The scanner captures a single clean frontal photo (side angles optional), gates it for quality on-device, and hands it to a server-side face-reconstruction pipeline that fits your likeness onto a pre-rigged humanoid template. No modeling, no rig setup, no download of desktop tools.
Page: /create/selfie (the working surface) · /scan (redirects here) · /features/scan (overview)
API: POST /api/avatars/reconstruct · GET /api/avatars/regenerate-status · GET /api/avatars/:id
Why it exists
Every other surface on three.ws (agents, the marketplace, Play worlds, talking-avatar video, AR) needs a body to drive. A text prompt gives you a character; a selfie gives you yourself. The scanner is the shortest path from a real face to a shareable, riggable GLB, and it runs entirely from a phone or laptop browser with a camera. The output is not a static bust: it is a humanoid with a skeleton and a full ARKit blendshape set, so it can walk, emote, lip-sync, and be exported the moment it finishes.
/scan is a thin alias kept for links and marketing. On load it runs window.location.replace('/create/selfie' + location.search), because /create/selfie carries the complete flow including the bring-your-own-key path. Document and link to /create/selfie.
How it works
The client half lives in three modules the page imports on demand: src/selfie-capture.js (camera and upload UI), src/selfie-refine.js (assessPhoto / refineSelfie), and src/selfie-pipeline.js (the bridge to the backend). A fourth module, src/face-quality.js, runs a live 468-point face mesh for framing feedback. All three MediaPipe consumers (face mesh, segmenter, face detector) load @mediapipe/tasks-vision from our own bundle and resolve the WASM runtime and model files through src/shared/mediapipe-assets.js, which serves the vendored copy under public/vendor/mediapipe/ first and only then falls back to a CDN, so a blocked cdn.jsdelivr.net no longer disables the scanner. Every quality threshold those modules apply lives in one shared, unit-tested module, src/selfie-gates.js: each number is traced in its header comment to a real reconstruction-pipeline failure mode (the worker's no-face rejection, the 35-degree morph-yaw ceiling, provider blur failures, the dim/backlit band from the robustness benchmark), so the client rejects a shot for the same reasons the backend would.
Two of those numbers are measured rather than reasoned about. The blur floor (BLUR_STDDEV_MIN) is calibrated against six real portrait photographs framed as 720x1280 phone selfies and scored through the module's own math, sharp versus the reconstruction worker's exact robustness.py degradations: sharp faces read 22.8 to 39.4, a Gaussian blur of radius longestEdge/220 reads 10.6 to 15.7, and a radius-10 smear reads 6.3 to 9.4. The floor sits at 12, in the empty band between the blurred and sharp populations. And because a blown-out face raises Laplacian response instead of lowering it, and averages back into the accepted band, neither the blur nor the mean-luma gate can see window glare on its own: grayFaceStats also reports clippedFrac, the share of the face crop pinned to near-white, gated separately by CLIPPED_FRAC_MAX.
- Capture, in the browser.
getUserMedia({ video: { facingMode: 'user', width: { ideal: 1280 }, height: { ideal: 1706 } } })opens the front camera. A real-time wireframe overlay tracks head pose, yaw, centering, blur, luminance, and highlight clipping, and only unlocks the shutter when a face is found; the hint line names the single most actionable fix ("Face the camera straight on.", "Hold steady. The image is blurry.", "Too dark. Find better light.", "Too bright. Move out of direct light or turn away from the window.") until every gate passes. A 3-2-1 countdown fires, thenshoot()draws the current video frame to a canvas and callscanvas.toBlob(blob, 'image/jpeg', 0.92). This is a single still, not a video scan. Camera support is feature-detected; if none exists, the shutter is disabled and an "Upload instead" path takes any image file. A photo shared into three.ws from the Android share sheet (the Seeker app or any installed PWA) is POSTed to/create/share, parked bypublic/share-target-sw.js, and lands here with?shared=1. The iOS app has no service worker, so its share extension parks the photos in the app instead and opens the page with?shared=1&inbox=<id>, and theThreeWsAppnative plugin hands them over once. Either way the capture page reads those files once (src/shared/share-target.js), fills the frontal slot and then left and right, and strips both params so a reload does not look like a fresh share. - Review the still. The countdown leaves ~3 seconds for motion blur, a head turn, or a lighting change to land in the actual capture, so the frozen still is re-scored (
checkImageQuality) against the same gates before you accept it. The review screen shows per-gate chips plus a verdict line; only a missing face disables "Use this" (mirroring the worker's one hard input rejection), everything else is a warning you can accept. - Optional side angles. Under an "Add side angles" disclosure you can capture two extra stills (
left,right, turned ~45 degrees). They raise fidelity and the ETA, but only the frontal is required. Confirmed shots (and the style choice) are mirrored tosessionStorage, so a refresh or accidental back-swipe mid-capture restores every slot instead of starting over; the build phase has its own resume rail (selfie:pendingJobId). - Refine, in the browser.
assessPhotogates the frontal (no face is the only hard block; blur, illustration, or a distant subject are non-fatal warnings).refineSelfieisolates the subject from its background and reframes to a clean head-and-shoulders square, rendered at 1024x1024 with JPEG quality 0.92 (src/selfie-refine.js). The refined frontal is previewed so you see exactly what the engine sees. - Submit. The pipeline (
src/selfie-pipeline.js) downscales each photo to a 1024px longest edge at JPEG quality 0.88, POSTs them as data URIs to/api/avatars/reconstruct, and gets back ajobId. Before any GPU work starts the handler pre-flights the caller's avatar library against its plan ceiling (assertAvatarSlotAvailable) and refuses a full library with402 plan_limit, so nobody waits ninety seconds for a mesh that could never be saved. - Reconstruct, server-side. The reconstruct handler (
api/avatars/_actions.js) selects a provider throughapi/_lib/regen-provider.js. The production lane is GCP: it POSTs{ images, body_type }to the Cloud Run service behindGCP_RECONSTRUCTION_URL, today theworkers/avatar-reconstruction/FastAPI app deployed asavatar-reconstruction. (The multi-backendworkers/avatar-pipeline-controller/speaks the identical contract and can take over that env var, but is not deployed today.) That pipeline (face_texture_transfer_v2) uses MediaPipe FaceLandmarker (468 landmarks, Apache-2.0) to transfer your face texture and geometry onto a fixed-topology, pre-rigged Wolf3D/Ready-Player-Me-style humanoid template carrying a Mixamo-compatible skeleton, 52 ARKit blendshapes, and 15 visemes. Full engine detail is in avatar-reconstruction.md. - Auto-rig fallback. The template is born rigged. If a generic mesh comes back unrigged and a rig lane is configured (the
model-rigworker, a Replicate rig model, or the house-key Tripo lane behindTRIPO_API_KEY),reconstruct-finalize.jschains arerigjob (workers/rigfor themodel-rigworker) and moves the job through ariggingstatus before surfacing the avatar. If rigging is impossible it delivers the static mesh taggedunriggedrather than nothing. - Poll and finish. The client polls
GET /api/avatars/regenerate-status?jobId=...every ~3 seconds (backing off on errors). Onstatus: 'done'it fetches the finished avatar fromGET /api/avatars/:idand shows it in a<model-viewer>. Adonejob with noresultAvatarIdmeans the server's materialize stage threw after the mesh was generated; that stage re-runs on every poll, so the client looks again up to three more times before treating it as a failure. If the materialization lost the quota race (another avatar landed while this one was generating), the server ends the job asfailedwith caller-facing copy rather than leaving it stranded atdone. The poll uses the adapter for the provider recorded on the job row, not whichever provider currently leads the precedence order, because the stored job id is provider-shaped and only that backend can read it. - Register and mint. Materializing the avatar also registers it in the Forge store (
forge_creations,path: 'reconstruct') so galleries, share and embed pages, and provenance treat a capture like any other creation, and fires a best-effort draft agent identity mint: a Metaplex Core asset on Solana, with an ERC-8004 registration behind its own flag. Both are best-effort by design: the avatar is already delivered, so neither can fail the job. - Backstop for a closed tab. The browser poll is what drives finalization, so closing the tab mid-build would otherwise strand the job even though the worker finished.
api/cron/reconstruct-sweep.jsruns the same finalize stages server-side every few minutes, so an abandoned capture still lands in your library. It rescues jobs up to 30 days old, and fails anything older with an honest reason rather than leaving it open forever. A job whose materialization is refused by a full avatar library is closed asfailedwith the same plan-limit copy the status poll relays, instead of being retried every tick.
The text-to-avatar variant
The same /api/avatars/reconstruct endpoint accepts a prompt instead of photos, but a prompt never touches the face pipeline: face-texture transfer keeps only the face and pastes it onto the template body, so a described character came back in the template's own clothes. Instead (api/_lib/prompt-avatar.js):
- The prompt is painted as one square (1:1) full-body reference image on the forge reference-image lane (
generateReferenceImage: Vertex first, then the shared text-to-image ladder), with framing cues appended (single figure, neutral A-pose, head to feet in frame, plain background) that say nothing about style. - That image is reconstructed on a full-figure image-to-3D engine, in plan order: self-hosted TRELLIS.2 (MIT licensed, served everywhere), the self-hosted Hunyuan3D worker (high tier), self-hosted TRELLIS, the Replicate and HuggingFace lanes, then the caller's own Meshy or Tripo key. The Tencent-model lanes (Hunyuan3D and HuggingFace) are dropped from the plan for requests from the EU, the UK and South Korea, and for an unknown country (
api/_lib/forge-territory.jsresolves it server side from the edge geo header). A self-hosted lane whose health probe reads hard down is moved to the back of the plan rather than accepting a job it would fail. - The unchanged finalize tail auto-rigs the bare mesh and materializes the avatar and its agent, exactly as for a selfie. The job records
source: 'prompt', the reference image, and the engine that ran.
One endpoint, two front doors.
Walkthrough
- Open /create/selfie. Sign in if prompted (the job resumes after login).
- Allow camera access. Center your face in the wireframe until the shutter unlocks; good light and a plain background help.
- Let the countdown run and hold still for the capture. Review the frame, or retake.
- Optionally open "Add side angles" and capture a left and right turn for higher fidelity.
- Press the submit button (labeled with the ETA, ~90s frontal-only or ~120s with sides). Watch the build steps: mesh, geometry and textures, auto-rig.
- When it lands, use "Open my avatar" to view it, "Customize in editor" to edit, "Turn this into an agent", or the export buttons (.GLB, .FBX) and share panel.
Examples
The reconstruct endpoint is authenticated. With a session cookie or a bearer token scoped avatars:write:
# Submit a reconstruction from a single frontal data-URI (or a hosted image URL).
curl -X POST 'https://three.ws/api/avatars/reconstruct' \
-H 'content-type: application/json' \
-H 'authorization: Bearer <TOKEN>' \
-d '{
"name": "My selfie avatar",
"photos": ["data:image/jpeg;base64,/9j/4AAQ..."],
"visibility": "private",
"params": { "bodyType": "average", "style": "realistic" }
}'
# -> 202 { "ok": true, "jobId": "…", "status": "queued", "eta": 90, "provider": "gcp" }
# Poll until done, then fetch the rigged GLB.
curl -H 'authorization: Bearer <TOKEN>' \
'https://three.ws/api/avatars/regenerate-status?jobId=<JOBID>'
# -> { "ok": true, "jobId": "…", "status": "done", "resultAvatarId": "<ID>" }
curl -H 'authorization: Bearer <TOKEN>' 'https://three.ws/api/avatars/<ID>'
# -> { "avatar": { "url": "<signed GLB url>", "model_url": null, ... } }
Request body: name (1-120, required), photos (1-6 image URLs or data URIs) or prompt (3-600 chars), optional description, visibility (private | unlisted | public), params, and the BYOK pair provider_key + provider_name. Status values progress queued -> running -> (rigging) -> done | failed.
States and limits
- Auth is required. No session or scoped token returns
401, and the page redirects to/login?next=/create/selfie, resuming the pending job afterward viasessionStorage. The/features/scanmarketing line "no account for first scan" does not match the shipped API; treat sign-in as required. - Cost. On the platform-hosted GCP lane there is no per-use charge in the reconstruct handler; usage is rate-limited (
limits.upload), and over-quota returns402 plan_limit. - Bring your own key. If no platform reconstruction backend is configured, the server returns
402 { code: 'regen_needs_byok', providers: ['meshy','tripo'] }and the page shows a key-entry form. Meshy and Tripo run image-to-3D with your own API key; credit exhaustion surfaces as402 insufficient_credits. - Single frame. Despite "captures several angles automatically" in some copy, the scanner takes one manual still per slot; side angles are an optional manual add.
- Timing. The build screen says "1-2 minutes"; polling gives up after 8 minutes, and a job still reported running 10 minutes after submit (possible when a pending job resumes across a reload) stops with a "taking unusually long" message pointing you to your dashboard, since the avatar may still finish server-side.
- Quality gating. Only a frontal with no detectable face hard-blocks. Blur, illustrations, or multiple faces are surfaced as warnings you can proceed past. If the face detector itself cannot load, the gate steps aside rather than blocking: an unavailable model is not evidence about your photo, and the worker runs its own face pass regardless.
- Vision models are on a deadline. The client-side refinement pass (subject matte, reframe, face box) needs MediaPipe's 11 MB runtime plus a model file. Every wait on one is capped at 15 seconds (
withVisionDeadlinein src/shared/mediapipe-assets.js); past that the raw photo is submitted and the reconstruction proceeds. Before that cap existed, an asset that never finished downloading left the page on "Processing your face..." indefinitely, with the reconstruction request never sent. The runtime is served from three.ws (public/vendor/mediapipe/) and only falls back to a public CDN when our own origin definitively lacks the file. - Error handling. Camera denial distinguishes
NotAllowedError("Camera access was denied... use Upload instead") from other failures. Job failures map to friendly messages (no face, content-safety flag, blur, service unavailable, timeout, out-of-memory). Poll tolerates up to 5 consecutive network errors with backoff before "Lost connection to the avatar engine." Rate limits (429) start a 60-second cooldown with a live countdown. Pending jobs resume across reloads. - A storage outage costs the filing, not the avatar. The last step copies the finished mesh into our object storage before the avatar row is written. When that write fails for an infrastructure reason (credential rejected or revoked, bucket missing, endpoint unreachable), the avatar is still created and points at the reconstruction worker's own public copy in Cloud Storage, which is durable and has no expiry.
storage_mode.r2.presentisfalseon such a row andsource_metacarriesservedFrom: 'provider'with thependingBucketKeya later re-copy needs, so these are findable rather than silent. Any other failure in that step still fails the job. Before this, a rejected storage credential threw away a mesh the GPU had already finished and left the page saying "Avatar finished but could not be saved" against a retry that could not succeed (2026-09-11; see docs/ops/gcp-production.md). - Output. glTF 2.0 GLB with a Mixamo-standard humanoid rig, 52 ARKit blendshapes, and 15 visemes. Private avatars return a short-lived signed
url;model_urlis the public CDN URL once you make it public. - Error copy. The worker classifies a failure as either the caller's input or a service fault. An input rejection ("no face detected in any of the provided photos") or a plan refusal is relayed verbatim, and the status response carries
errorKind: 'input'so the client shows that text as-is instead of rewriting an unrecognised message into generic "try a clearer photo" advice; a service fault is masked into neutral retry copy, because its raw text can carry a vendor name or an operator-only correlation id. Your visibility choice rides along to the Forge store, so a private capture never surfaces in a public gallery.
Verifying it end to end
npm run verify:selfie drives the real page in a real browser: it uploads a
photograph into the frontal slot, submits it, and follows the job until the
avatar engine returns a GLB or fails. Nothing in it calls the API behind the
page's back, so the capture gates, the reconstruct call, the poll loop and the
viewer are all exercised the way a visitor exercises them.
npm run audit:web:login # mint the QA session the reconstruct call needs
npm run verify:selfie # generates a portrait, runs the whole flow
Without --photo it generates one through the platform's own free text-to-image
lane (POST /api/v1/ai/image, five free images per day per IP, no payment),
because the reconstruction worker's only hard input rejection is a frame with no
detectable face. Useful flags:
| Flag | What it does |
|---|---|
--photo=<path> |
Use a photograph you already have instead of generating one |
--origin=http://localhost:3000 |
Run against a dev server; the QA session is replayed onto that origin and /api proxies to production |
--stall-vision |
Hang every request for MediaPipe's runtime, to prove the flow still submits without it |
--timeout=<ms> |
How long to wait for the build (default 10 minutes) |
--out=<dir> |
Where screenshots and run.json land (default scripts/.selfie-verify/) |
The run prints the pipeline's own event trail (submit -> preview -> building -> progress -> done), which is the fastest way to see where a failure happened: no
building event means the reconstruction request was never sent, and the
problem is in the browser rather than in the engine.
--stall-vision is the regression harness for the 2026-09-11 stall. Measured
with the vision runtime completely unreachable: the flow gives up on refinement
after two 15-second deadlines, submits the unrefined photo, and still returns a
rigged avatar (58 seconds end to end, against 12 seconds when the runtime loads
normally).
Related
- Avatar reconstruction pipeline: the full engine: MediaPipe face-texture transfer, geometry morph, the fixed pre-rigged head template, and the Cloud Run worker.
- Avatar pipeline: the distinct Forge text-to-3D generation lane.
- Draft agent mint: the on-chain identity every finished reconstruction gets, the network flags, and the devnet proof script.
- Trait-based avatar builder: customize and dress the avatar after you scan it.
- Talking Avatar Video and Lipsync: drive the finished avatar's mouth.
- Pages: /create/selfie, /create, /features/scan.