Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

17.7 KB
Raw Blame

Selfie → Avatar Reconstruction

This is the subsystem that turns a user's photo into a rigged, animation-ready 3D avatar. A text prompt sent to the same endpoint does NOT use this pipeline: fitting a face onto a template body ignores everything a prompt says about outfit, build, and style, so prompts run the full-body lane instead (a full-body reference image, reconstructed on the self-hosted Hunyuan3D worker, then auto-rigged). The lane plan lives in api/_lib/prompt-avatar.js; both inputs share the job contract, the status poll, and the finalize tail described below. It is distinct from the Forge generation lane in the Avatar Pipeline: the Forge generates an arbitrary mesh from a prompt, whereas reconstruction fits a specific person onto a fixed, pre-rigged template — the same architecture Avaturn and Ready Player Me use, and the reason the output is born-rigged with facial blendshapes instead of a bare mesh that still needs auto-rigging.

  • Worker: workers/avatar-reconstruction/: FastAPI on a CPU-only Cloud Run service (8 vCPU). It ran on an L4 until background removal was measured to gain nothing from CUDA; see the comment in its cloudbuild.yaml.
  • Backend provider: api/_providers/gcp.js, selected by AVATAR_REGEN_PROVIDER=gcp (see api/_lib/regen-provider.js).
  • User entry points: the selfie/upload flow (and, via the full-body lane, the text→avatar prompt flow) in api/avatars/_actions.js (POST /api/avatars/reconstruct).
  • Completion: normally driven by the browser polling /api/avatars/regenerate-status, which runs the finalize stages inline. api/cron/reconstruct-sweep.js is the server-side backstop for when it isn't — see below.

Who finishes the job

A reconstruct job is advanced by whoever polls it. The /create/selfie page polls every few seconds, and each poll pulls provider status and runs the shared finalize stages in reconstruct-finalize.js.

That made the browser tab load-bearing. Close it mid-build and the worker still finished the GLB and stored it durably in GCS, but nothing ever collected it: the job row stayed queued/running forever and the avatar never reached the user's library. Production had 26 jobs stranded that way, the oldest from 2026-05-31, invisible to db-retention (which only prunes terminal rows).

reconstruct-sweep closes that hole. Every 5 minutes it picks up reconstruct jobs quiet for over 3 minutes and runs the same finalize stages the browser poll would have, so an abandoned job still lands in the library. Design notes:

  • 30-day rescue window, far longer than the auto-rig sweep's 6 hours. The worker's GLBs (GCS) and job state (Firestore) are both durable, so a weeks-old abandoned job is still genuinely deliverable to a real user.
  • Past that window, and for jobs whose provider is no longer pollable with platform credentials, rows are failed out with an honest reason — the open set must not grow without bound.
  • BYOK jobs (meshy/tripo) stay browser-driven. They authenticate with the user's own key, which the status poll resolves from request context; a cron has no user context to decrypt it. They age out via the same window.
  • A full avatar library closes the job instead of retrying it. Materialization refused by the plan quota is the owner's to fix, so the sweep fails the job with the same PLAN_LIMIT_JOB_ERROR copy (error_kind = 'input') the status poll relays, rather than re-running the refused step every tick.
  • A provider 404 (gcp_task_missing) is terminal here, so a job whose worker record is genuinely gone resolves instead of being retried forever.

Object storage cannot take the avatar away

The last step of finalize copies the finished GLB from the worker's bucket into ours. That single putObject used to sit between the user and the avatar they had already waited ninety seconds for: when the R2 credential stopped verifying at 03:15 UTC on 2026-09-11, materialize aborted, no avatar row was written, and the page could only say "Avatar finished but could not be saved. Try again." Retrying never worked, because the photo was never the problem.

The mesh was never lost. The reconstruction worker parks it in gs://three-ws-avatar-reconstructions, which is public and lifecycle-free, and that URL is already in the job row. So storeGlbOrKeepProviderUrl in reconstruct-finalize.js now lets a named storage-infrastructure fault (isStorageInfrastructureError: credential rejected or revoked, bucket missing, endpoint unreachable) cost the durable copy in our bucket rather than the user's avatar. The provider's absolute https URL becomes the storage key: publicUrl() passes an absolute key straight through, copyObject() declines to copy one, defaultStorageMode() records r2.present = false instead of claiming bytes that are not there, and source_meta carries servedFrom plus the pendingBucketKey a later re-copy needs. Any other error still throws, so only a fault we can name is allowed to degrade.

Plan quotas are checked twice, on purpose

A reconstruction only meets the caller's plan quota at the very end, inside createAvatar → enforceQuotas. That is roughly ninety seconds of GPU time after the button was pressed (plus a Flux reference image on the text→avatar lane), and a refusal there used to leave the job at done with no resultAvatarId: a state no recovery cron looks at (they filter on queued/running), which re-ran the same doomed materialization on every poll and reported the generic engine-fault copy to the browser. The user was told the engine had a problem, and every retry burned the same spend to land in the same place.

Two checks now bracket the job:

  1. Pre-flight, at submit. POST /api/avatars/reconstruct calls assertAvatarSlotAvailable(userId) (api/_lib/avatars.js) before it resolves a provider or generates a reference image. A full library is refused in milliseconds with 402 plan_limit and an actionable error_description. This is the check that fires in practice.
  2. At materialization, for the race. If another avatar lands while the mesh is generating, the finalize stage still throws. failJobOnPlanLimit then ends the job as failed with error_kind = 'input', so /api/avatars/regenerate-status relays the plan copy verbatim (see REGENERATE.md) instead of masking it as an engine fault.

Only the per-plan avatar count and the total-bytes ceiling can be pre-flighted: per-file size is unknowable until the mesh exists, so plan_limit_size remains a materialization-time refusal.

How it works

The avatar is a fixed-topology Wolf3D/RPM template that ships pre-rigged with a humanoid skeleton and 52 ARKit blendshapes + 15 visemes. Reconstruction never generates a new mesh; it fits the person onto that template in two phases.

Phase 1 — Face texture transfer

face_pipeline.py: MediaPipe FaceLandmarker finds 468 landmarks on the best frontal photo, a thin-plate-spline warp maps the face into the template's skin-UV space, it is composited over the face-oval region, and skin / hair / eye colours are sampled and tinted for consistency. Output: the template body wearing the person's face texture.

Phase 2 — Face geometry morph

Texture alone leaves the shape generic — same jaw, nose, brow and face width as the template, so the avatar looks like a sticker on a mannequin. face_geometry.py closes that gap:

  1. Umeyama similarity-aligns the 468 detected landmarks onto MediaPipe's neutral canonical face model, so the person's face and the reference share a frame.
  2. The residual (person − canonical) on a stable identity subset (face oval, nose, cheeks, brow, jaw — expression-prone eye/lip points excluded) is the person's shape deviation from neutral.
  3. That displacement is carried onto the head's corresponding vertices (a nearest-vertex map precomputed by precompute_uv.py), scaled into head units, and clamped to reject landmark/pose outliers.
  4. A thin-plate spline interpolates the sparse displacements across all 2162 head vertices. TPS passes exactly through its control points, so localised identity (a wider jaw, a longer nose) survives instead of being averaged away — the failure mode of normalised-Gaussian/Shepard weighting. TPS extrapolates freely, so a Gaussian locality mask fades the field to zero off the face and the scalp, ears and neck stay put.

strength and max_displacement_frac default to 0.6 / 0.55, chosen by two sweeps over the 40-face reference set, not by eye: python -m eval.tune_morph (mean ISE, is the shape right?) and python -m eval.robustness (does a degraded photo of the same person still give the same head?). 0.55 sits at the knee of that fidelity-vs-robustness curve, cutting mean ISE 46% against the original 0.75 / 0.18, whose ~1.9 cm displacement ceiling was throttling genuine facial variation rather than rejecting outliers; past 0.65 fidelity flatlines while instability keeps rising. See avatar-fidelity-program.md for why the sweep's own looser optimum was deliberately not shipped.

Because vertex count and order never change, glb_ops.set_head_geometry writes the morphed positions (and recomputed normals) back without disturbing the skinning weights or any of the 67 blendshape morph targets — the rig and every ARKit expression survive intact. This is verified end-to-end in test_face_geometry.py.

Toggle Phase 2 with the GEOMETRY_MORPH env var (default 1). It degrades cleanly to texture-only if anything fails; the shape refinement never fails a job.

Phase 3: projective texturing (the rest of the head)

Phases 1 and 2 both stop at the face. The face-oval polygon Phase 1 composites into is only 10.4% of the head's texels (9.4% of its surface area), because MediaPipe's 468 landmarks are face-only: there are no ear, scalp or neck points, so outside the oval there is no correspondence to warp with. Everything else was template texture personalised by a single global skin tint.

face_projection.py takes the other route. Phase 2 leaves us the head's real 3D shape, which turns the problem from find matching points into find where each surface point lands in the photo:

  1. Fit the camera with a closed-form Umeyama similarity between the head's landmark vertices and MediaPipe's 3D landmarks. This is the only use of landmarks, and they locate the camera rather than establishing surface correspondence.

    This started as cv2.solvePnP and had to be replaced, which is worth recording. landmark_vtx is nearest-vertex and many-to-one: on the shipped template, 468 landmarks collapse onto just 262 distinct head vertices, so hundreds of landmarks share one 3D point while sitting at different 2D positions. That is a contradiction no camera can satisfy. Measured on the reference faces, RANSAC found 25-35% inliers and returned a solution behind the camera on half of them, with the depth sign flipping between runs and coordinate conventions.

    Because MediaPipe reports 3D landmarks, the alignment is 3D-to-3D rather than 2D-to-3D, and Umeyama solves it in closed form with a reflection guard: no RANSAC, no depth-sign ambiguity, nothing that can diverge. Duplicate correspondences are averaged first so a vertex is not weighted by how many landmarks happened to land on it. The result is a weak-perspective (scaled orthographic) camera, which is the right model for a selfie anyway: head depth is centimetres against a camera distance of tens, so foreshortening across the head is small. 10 of 10 reference faces now fit, against 50% failure with PnP.

  2. Rasterize the head in UV space so every texel carries its 3D position and normal.

  3. Project and sample. Each texel's 3D position goes through the camera into the photo. Ears, jawline and neck fall out for free.

Four gates decide whether a texel may be painted at all, because projection will otherwise paint the back of a head with whatever pixel lies behind it. The texel must land inside the photo, face the camera (confidence falls off as cos^1.5 of the view angle), survive a depth-buffer occlusion test that catches the nose shadowing the cheek, and sample a foreground pixel per the rembg alpha. A texel failing any gate keeps its existing colour rather than inventing one. The result blends under the Phase 1 composite, which stays authoritative where the landmark warp applies, and is capped at MAX_BLEND = 0.85 so a slightly-wrong pose degrades to a tint rather than a visibly wrong image.

Measured on the shipped template by eval/measure_projection_coverage.py, photographic coverage of the head rises from 10.5% to 41.1%, a 3.9x increase across the 40-face reference set: the face oval keeps its 10.5% and projection adds 30.6% of ears, jawline, neck and forehead that no landmark describes. All 40 faces fit a camera; per-face coverage ranges 33.9% to 42.1%.

Toggle with PROJECTIVE_TEXTURE (default 1). Like the morph, it degrades cleanly: a pose that will not solve logs and leaves the warp-only skin in place. It runs after the geometry morph, since projecting onto the template and then morphing would slide the texture off the features it was sampled from.

Every buffer read is stride-aware. The head's vertex attributes share one interleaved bufferView, so precompute_uv.py and glb_ops honour bufferView.byteStride on every accessor read and write. A stride-unaware read silently interleaves POSITION with NORMAL, which builds face_uv_map.json on corrupted vertex data and breaks the Phase-1 texture correspondence too.

Licensing — why this stack

Every stage is commercial-clean (Apache-2.0 / MIT / CC0). The dominant constraint in single-image avatar reconstruction is that the best-known 3D face and body models are non-commercial: FLAME and the Basel Face Model (BFM) are academic licences, and the whole SMPL/SMPL-X family is Max-Planck non-commercial. Anything built on them (DECA, EMOCA, MICA, ICON, ECON, PIFuHD, …) inherits that restriction. The Phase-2 morph uses only MediaPipe Face Mesh (Apache-2.0) — no 3DMM — so it is unencumbered.

Roadmap — beating Avaturn on fidelity

The active plan built on this roadmap — tracks, owners, capacity and credit allocation — is avatar-fidelity-program.md. Fidelity claims are scored with the ISE metric in workers/avatar-reconstruction/eval/.

Phase 2 captures gross face identity (proportions, projection) but not fine geometry. The path to and past Avaturn quality, with the verified licence for each link:

Version Upgrade Model Licence verdict
v1 (shipped) Real face-shape morph (sparse) MediaPipe Face Mesh Apache-2.0 — clean
v2 (recommended) Dense identity geometry, fused via register_head_to_target MICA + FLAME, commercial licence from MPI Paid MPI commercial licence — clean once signed. FLAME's fixed topology + expression basis map straight to ARKit
v2 (fallback) Dense identity, no licence fee HRN re-based on FLAME-2023-Open (CC-BY-4.0) Clean, but weeks of GPU R&D to retrain the identity regressor
v2 (texture) SHIPPED Projective texturing off the morphed mesh: fills ears, jawline and neck from the same photo (10.5% to 41.1% of the head) none, geometry we already have Clean; no new model or licence
v3 (texture) Inpaint what no camera saw (scalp, far cheek) Imagen inpaint on Vertex AI GCP, pre-approved
v3 Drop the RPM-template dependency for a fully-owned body Anny (parametric body) + ICT-FaceKit (ARKit-52) + deformation transfer Anny Apache-2.0 + CC0; ICT-FaceKit + DT MIT — clean

Rejected: FaceLift (ICCV'25) — initially floated as the clean v2 model, but verification killed it: its weights are licensed from Adobe under the non-commercial Adobe Research License (Apache code does not extend to the weights), and it outputs 3D Gaussian Splats, not a mesh, so it cannot feed a rigged-GLB pipeline. This is the recurring trap here: the high-quality single-image face models are almost all encumbered by a non-commercial 3DMM/ dataset (FLAME, BFM, SMPL) or a non-commercial weight licence. The dense registration itself (register_head_to_target) is model-agnostic and already validated, so whichever licensed model is chosen drops straight in.

Auto-rigging for the reconstruct-the-whole-body variant would use Make-It-Animatable (Apache) or UniRig (MIT) → a Mixamo-standard skeleton that drives the existing pre-baked clip library. The whole recommended chain runs on one L4 (24 GB); an A100 is only needed for throughput, not any single stage.

Deploy

The worker is already provisioned on Cloud Run (CPU-only, 8 vCPU / 16 GiB, DEVICE=cpu). From a clean tree:

gcloud builds submit \
  --config workers/avatar-reconstruction/cloudbuild.yaml \
  --substitutions _GCS_BUCKET=three-ws-avatar-reconstructions,SHORT_SHA=manual$(date +%s)

The backend needs no change — Phase 2 ships inside the same image and the POST /reconstruct → GET /jobs/:id contract is unchanged. Verify with GET /health ("pipeline": "face_texture_transfer_v2", "geometry_morph": true). Full runbook: docs/ops/gcp-production.md.