Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.
Avatar fidelity program — owning selfie → avatar end to end
Goal: a selfie-to-avatar engine that beats the leading commercial selfie-avatar products on identity fidelity, runs entirely on infrastructure we own, and costs nothing per avatar. Not "good enough to avoid a bill" — better than the thing we would have paid for.
This is the program plan. The subsystem it improves is documented in
avatar-reconstruction.md; the worker lives in
workers/avatar-reconstruction/.
Why we are not buying this
Commercial vendors sell exactly this capability, metered per avatar. Two reasons we build instead:
- It is the product. three.ws is an avatar platform. Renting the step that turns a person into an avatar means the core of the product is someone else's, priced by them, rate-limited by them, and deprecable by them.
- The architecture is already ours. The worker fits a person onto a pre-rigged fixed-topology template — the same approach the major commercial avatar platforms use. The output is born rigged with 52 ARKit blendshapes and 15 visemes. What is missing is fidelity, and fidelity is a compute-and-research problem, which is precisely what ~$100k of Google Cloud credits buys.
Every stage is commercial-clean (Apache-2.0 / MIT / BSD / CC0). Holding that line is a hard constraint, not a preference: the highest-quality published face models are almost all built on a non-commercial 3DMM (FLAME, BFM, SMPL) or carry non-commercial weights. See the licence section in avatar-reconstruction.md for the verdicts, including why FaceLift was evaluated and rejected.
The spine: measure before optimising
Every track below is judged by one number, Identity Shape Error (ISE) —
texture-blind, pose- and scale-invariant, dimensionless. Full definition and
usage: workers/avatar-reconstruction/eval/README.md.
Without it, "better than the commercial benchmark" is an opinion and a research track can burn GPU-months moving a number nobody measured. With it, every change is a diff:
python -m eval.run_eval --compare eval/reports/v1.json eval/reports/v2.json
Rule for this program: no fidelity claim without a report, and no report without naming the reference set it scored.
What actually binds
Not money. Two things:
- L4 GPU quota = 3 for us-central1, shared by six warm GPU services. The fleet is permanently at its ceiling. See ops/gcp-credits-plan.md. An RTX PRO 6000 (Blackwell, 96 GB) is already granted and validated live at 1 instance, but needs a CUDA 12.8 / torch cu128 rebuild — L4 images do not run on it.
- Training-data licensing. The reason to prefer per-subject optimisation over training an identity regressor: a regressor needs a face dataset, and the commercially-usable ones are thin. Optimisation needs only the user's own photo. This is the central design decision of the program.
Tracks
Tracks 1-3 are independent and can run in parallel. Track 0 gates the claims of all of them.
Track 0 — Measurement [done]
The ISE metric, batch runner, revision comparator, parameter tuner and property
tests, plus a 40-face reference set generated by
eval/make_refs.py over a
deterministic grid of the platform's diversity matrix and mirrored to
gs://three-ws-avatar-reconstructions/eval-refs/.
Faces are synthesised rather than collected: no real person's biometrics in a benchmark that gets copied between machines, no licensing terms, and any agent can rebuild the identical set.
Because every reference face is a clean frontal portrait, ISE alone cannot see
what the outlier machinery is for, so the set is paired with
eval/robustness.py:
each face is degraded on the fly (blur, 12° turn, occlusion, dim, 1/6 resolution)
and scored on whether the same person still yields the same head. Fidelity and
stability pull opposite ways, and shipped parameters sit at the knee (Track 1).
The confidently-wrong detection is covered too, by a 16-sample adversarial set
(eval/make_adversarial.py)
spanning non-faces (pet, statue, doll, mask, mannequin), depicted faces (poster,
phone screen, painting, magazine), crowds, and out-of-envelope shots
(near-profile, hard backlight, motion blur, macro), audited by
eval/detection_guard.py.
That audit produced three findings, one of which killed the guard I was about to
build:
A face-plausibility gate cannot work — do not rebuild one. The natural design is to align landmarks to the canonical face and reject a large residual. MediaPipe FaceMesh is a fitted model: it does not decide whether something is a face, it reports where a face would be. A golden retriever scored 0.4758, a better canonical fit than all 40 real humans (0.5035-0.5238). Any threshold separating them rejects real people first.
The catastrophic failure mode does not exist in this architecture. Head
displacement from the template: real faces 0.2713-0.2839, every detected
adversarial input 0.2586-0.2929, and zero ungated adversarial inputs distort
more than a real face. A dog photo yields a slightly unusual human head, not a
monstrous one. Four mechanisms compose to guarantee it: canonical alignment
removes pose and scale, strength damps the residual, the per-point clamp bounds
outliers, and the TPS off-face mask confines deformation to the face. Guarding the
mesh against wrong detections is therefore unnecessary — which is worth knowing
before someone spends a sprint on it.
Yaw was the one thing worth gating. Real frontals peak at 2.7°; the
near-profile shot reads 58.6° and was the sole input pushing the head outside the
real-face band. Past roughly a third of a turn the far side of the face is
self-occluded, so its depth is extrapolation and the morph fits guesswork as bone
structure. face_pipeline.MAX_MORPH_YAW_DEG (35°) now skips the morph and keeps
texture transfer above that, clearing every genuine frontal by ~13x. Photos of
photos sit at 9-18°, already produce in-band heads, and are deliberately not
blocked.
Track 1 — Ship the morph that is already written [done]
face_geometry.py (Phase 2 head-shape morph) had been written, tested and never
deployed: production /health reported face_texture_transfer_v1 and omitted
geometry_morph entirely, so the running image predated the work. Production was
texture-only — a person's face painted on a generic skull. It now reports
face_texture_transfer_v2 with geometry_morph: true.
Deploying it surfaced two things worth more than the deploy:
A latent correctness bug. landmarks_to_array passed MediaPipe's raw
normalised landmarks straight into the morph. MediaPipe divides x by image width
and y by image height, so on any non-square photo the point cloud is
anisotropically stretched; Umeyama fits one uniform scale and cannot undo that,
so the stretch survived alignment and was consumed as facial identity. The
camera's aspect ratio was being baked into the avatar's skull — the same person
at 3:4 and 1:1 got head shapes differing by 3.07% of head size, and 3:4 is
exactly what the selfie page recommends. Fixed by scaling to an isotropic pixel
frame; drift is now 0. Pinned by a test that fails on the old path.
Defaults that had never been measured. A sweep over the 40-face set
(eval/tune_morph.py)
moved strength / max_displacement_frac off 0.75 / 0.18, whose clamp permitted
only ~1.9 cm of travel per control point and was throttling genuine facial
variation rather than rejecting outliers. Every setting in the usable range beat
it, and no face was made worse by the change.
Setting the clamp on fidelity alone is not possible: a benchmark of clean
portraits contains nothing for a clamp to reject, so it will always recommend no
clamping at all. That produced the program's second measurement tool,
eval/robustness.py,
which asks the complementary question — does a degraded photo of the same
person still give the same head? Each face is blurred, turned 12°, occluded,
dimmed and downsampled to 1/6 resolution, and the score is the divergence from
the clean-photo head. The person did not change, so all of it is the pipeline
reacting to photo quality rather than identity.
| clamp | ISE | vs floor | instability |
|---|---|---|---|
| 0.18 (was shipped) | 0.2932 | 26.5% | 0.0027 |
| 0.45 | 0.1684 | 57.8% | 0.0053 |
| 0.55 (shipped) | 0.1591 | 60.1% | 0.0057 |
| 0.65 (sweep optimum) | 0.1587 | 60.2% | 0.0060 |
| 1.00 | 0.1587 | 60.2% | 0.0060 |
Fidelity flatlines past 0.65 while instability keeps climbing, so anything looser is pure downside; across the usable range each step buys ~40-50x more fidelity than it costs in stability. 0.55 is the knee. A first pass shipped 0.45 on physical reasoning before the robustness data existed, which turned out to be 2.3 points of fidelity too conservative — a fair illustration of why the second metric was worth building.
Still unmeasured: all 75 degraded images yielded a face detection, so the catastrophic regime the clamp ultimately guards is untested. A detection failure is safe (the job degrades to texture-only); a confident wrong detection is not, and nothing bounds that yet. That is the remaining Track 0 work.
Cost: negligible. As predicted, the largest fidelity gain per unit of effort in the program, because the work was already paid for.
Track 2 — Full-head photoreal texture (Vertex AI Imagen)
A selfie shows one side of a face under one lighting condition. Cheeks in shadow, ears, the scalp, under the jaw — none of it is in the photo, and today those regions fall back to a tinted template texture. That mismatch is what reads as "avatar" rather than "person" the moment the head turns.
Measure the opportunity first, it is bigger than it looks. Of the head mesh's texels, only 10.4% is the face oval the selfie actually covers (9.4% of its actual surface area). The other ~90%, ears, scalp, neck, under-jaw, is template texture the camera never saw, personalised solely by a global skin-tone tint. That is the real prize here, and it also bounds what the mechanisms below can achieve.
An earlier revision of this doc quoted 19.2% / 80.8%. That was measured against the wrong denominator (it counted atlas texels the head mesh does not use). Re-measured against head-only texels on the shipped template it is 10.4%, so the gap was larger than first reported, not smaller. The number is now produced by
eval/measure_projection_coverage.pyrather than by hand.
A landmark-driven warp cannot reach most of it. MediaPipe's 468 landmarks are
face-only: there are no ear, scalp or neck points, and _warp_face_to_uv masks
to the face-oval polygon accordingly. So warping extra synthesised views through
the existing path improves lateral cheek and jaw fidelity (where a frontal is
foreshortened) but cannot texture the other ~90%, because no correspondence
exists there. Anyone picking this up should not expect the view-warp route to
fill the scalp.
The mechanism that does reach it is projective texturing off the mesh we
already have. SHIPPED. face_projection.py
solves the camera by PnP against the head's landmark vertices, rasterises the
head in UV space so every texel knows its 3D position and normal, then projects
each texel into the photo and samples it. No landmarks are needed for the
surface itself, so ears, jawline and neck fall out naturally.
Four gates keep it honest, because projection will otherwise paint the back of a
head with whatever pixel lies behind it: the texel must land inside the photo,
face the camera, survive a depth-buffer occlusion test, and sample a foreground
pixel. Anything failing a gate keeps its existing colour. The result blends
under the face-oval composite, which stays authoritative where the landmark
warp already applies, and is capped at MAX_BLEND so a slightly-wrong pose
degrades to a tint rather than to a visibly wrong image.
Measured on the shipped Wolf3D head with real faces
(eval/measure_projection_coverage.py), across all 40 reference faces
photographic coverage of the head goes from 10.5% to 41.1%, a 3.9x increase,
and every one of the 40 fits a camera:
| Share of the head's texels | |
|---|---|
| Face-oval warp (what existed before) | 10.5% |
| New surface from projection | 30.6% (range 23.3-31.6%) |
| Total photographic | 41.1% (range 33.9-42.1%) |
The harness reports the oval and the projection separately on purpose, and that is what caught the sign bug below: while the signs were inverted, "new" came out exactly equal to "projected", meaning the projected region and the face oval did not overlap at all. That is impossible for a real camera, since the face is the most camera-facing surface on the head. A single headline coverage number would have looked healthy (it read 47%, higher than the correct 41%) and hidden it.
Two decisions are worth recording because both nearly went the wrong way.
The signs. Depth and view direction were both inverted, so the pipeline believed the front of the head pointed away from the lens. Every gate then rejected the face and painted the back of the head instead. All ten unit tests passed throughout, because each one selects its test region by facing and so adapts to whichever sign is in force. Only an absolute claim catches this, and there is now a test asserting that the surface the landmarks sit on faces the camera (it fails at -0.89 with the old signs). Umeyama's reflection guard forces a proper rotation and can absorb a sign flip in the correspondence, so the fitted frame's handedness cannot be reasoned about from MediaPipe's documented z convention; it has to be measured.
The camera model. The obvious approach, cv2.solvePnP against the head's
landmark vertices, does not work here and the synthetic tests did not catch it:
they generated 2D points by projecting the same 3D points, so the
correspondences were perfect by construction. On real photos they are not.
landmark_vtx is many-to-one, collapsing 468 landmarks onto 262 head vertices,
so hundreds of landmarks share a 3D point while sitting at different 2D
positions. RANSAC found 25-35% inliers and put the head behind the camera on
half the reference faces. The replacement fits a weak-perspective camera by
closed-form Umeyama against MediaPipe's 3D landmarks, after averaging duplicate
correspondences: 10 of 10 reference faces now fit, mean coverage 36.9% with a
range of only 36.2-37.5%. The lesson is that a synthetic test built from the
same generative model it validates proves the arithmetic and nothing about the
data.
The facing falloff. cos^3 with a 0.15 floor stops painting at 58 degrees from the camera, which rejects the ears outright. Ears sit at 70-90 degrees from a frontal camera and are a large share of the surface this exists to reach, so that setting would have left the headline gap unclosed while appearing to address it. The shipped curve is cos^1.5 with a 0.06 floor, which admits them as a light wash. Coverage of no-landmark surface measured 0% before the change and 91% after.
The camera-pose machinery is shared with Track 3.
Already fixed on the way in: the skin tint was being applied to the whole
texture, including the face oval that had just been composited from the user's
photograph — shifting the only region with true photographic colour toward an
average of itself by ~1.4% of range. It is now masked to the regions the photo
could not cover, which is what its own comment always said it was for
(test_face_pipeline.py pins the mask behaviour, including that the tint still
reaches every unprotected pixel — a mask that over-protected would leave a
template-coloured neck against a photographic face, a worse artefact than the
one being fixed).
Two candidate mechanisms for the remainder, and the measurement in Track 3 decides between them.
Do not inpaint the UV atlas directly. A UV layout is not a photograph; a generative image model conditioned on one will produce photographically plausible but UV-incoherent content, and the seams land exactly where the head turns.
Do synthesise additional camera views and warp them through the existing path.
Each view goes through the same landmark-driven TPS warp as the frontal
(_warp_face_to_uv), so coverage of cheeks, ears and jawline is UV-coherent by
construction rather than by hoping the model understood the atlas.
Crucially, the finding that kills multi-view for geometry (Track 3) does not kill it for texture. Rotated synthesised views measure as a different person geometrically, but the same-pose control shows appearance is reproduced faithfully (0.0069 divergence), and texture needs plausible skin, not millimetre-accurate landmarks. A 3/4 view that is 0.03 off in face shape still carries the correct complexion, beard, hairline and ear into the regions a frontal cannot see. The severity of the error differs by an order of magnitude between the two uses.
ISE will not move — it is a geometry metric, texture-blind by design. This track must bring its own: masked SSIM / LPIPS against held-out views, plus a seam-continuity check across the UV boundary. Gate the synthesised view on the same-pose identity check before compositing, so a bad generation is rejected rather than painted on.
Track 3 — Dense identity geometry by per-subject optimisation
The Phase-2 morph is driven by 468 sparse landmarks diffused across 2162 head vertices. It captures gross proportion — face width, jaw, nose projection, brow — and cannot capture fine structure, because the signal is not there.
The unencumbered path to dense identity is analysis-by-synthesis: render the current head, compare to the photo, backpropagate to vertex positions, repeat.
Differentiable rasteriser: PyTorch3D (BSD-3-Clause). Commercially clean. Deliberately not nvdiffrast, whose NVIDIA Source Code License is non-commercial.
Regularisation with a Laplacian smoothness term and a displacement bound, so the head stays on a plausible face manifold and every ARKit blendshape survives. Vertex count and order never change, so
glb_ops.set_head_geometrywrites the result back without touching skinning or morph targets — the propertytest_face_geometry.pyalready pins.Multi-view from one photo — MEASURED AND REJECTED for geometry. The plan was to synthesise 3/4 and profile views with a Vertex image model and fit against all of them at once. It does not survive contact with the identity metric. Asking
gemini-2.5-flash-imagefor the same person turned 30°:comparison identity-shape divergence same-pose regeneration (control) 0.0069 synthesised 3/4 view (19.7° yaw) 0.0396 a different real person 0.0358 (range 0.0268-0.0489) The control matters: at unchanged pose the model reproduces the person ~5x more faithfully than a stranger, so identity preservation is genuinely good. But once the head is rotated, the recoverable face geometry is statistically indistinguishable from a different person. Whether the cause is the model drifting under pose change or MediaPipe's depth being pose-dependent (likely both), the signal is too degraded to fit a skull to. Feeding it to a differentiable fit would optimise confidently toward someone who does not exist.
Do not revive this leg on synthesised views. It needs genuine multi-pose capture — several real frames of the actual user, which the
/scanflow already collects — and the metric above is the acceptance test any such source must pass before it drives geometry.
No third-party weights, no face dataset, no licence question: it optimises against the user's own photo. It is GPU-hungry per avatar, which is exactly the resource we have.
Alternative shortcut, requires owner approval because it is external spend:
a commercial FLAME licence from MPI unlocks MICA for dense identity in one
forward pass. Cheaper in engineering time, adds a licensing dependency, and
register_head_to_target is already model-agnostic so it drops straight in if
the owner wants it. Recommendation: build the optimisation path first, since it
is unencumbered and permanent.
Track 4 — Capacity
Fidelity work is throughput-bound before it is idea-bound: Track 3 is minutes of GPU per avatar against a fleet already at its ceiling.
- Chase the L4 quota preference (
l4-no-zonal-us-central1-8, preferred 16). On a grant ≥ 8, the fleet-wide scale-up commands are pre-approved and listed in ops/gcp-credits-plan.md. - Port the reconstruction worker to the granted RTX PRO 6000 (CUDA 12.8 / torch
cu128; platform minimum 20 CPU / 80 Gi).
workers/model-hunyuan3d/Dockerfile.hunyuan21rtxis the first mover to copy. 96 GB of VRAM makes the Track 3 optimisation loop practical at interactive latency. - Vertex AI custom jobs for any batch experiment, so research load never competes with the serving fleet for the same three L4s.
Track 5 — Fully-owned body (v3)
The reconstruction template is a third-party Wolf3D base — the last such asset in
the pipeline. The replacement already exists in this repo and the roadmap
entry above is out of date: public/avatars/parametric-base.glb
is a CC0 MakeHuman/MPFB2 body (vendored via naver/anny
in avatar-sources/anny/) carrying 122 baked morph
sliders — nose, ears, mouth, eyes, jaw, cheeks, head shape, neck, plus gender,
muscle, weight, age, height, torso, hips, limbs — on a 52-bone mixamorig:*
skeleton, built by scripts/build-parametric-base.mjs
and already live in Avatar Studio's base switcher.
That turns this track from "build a body" into "retarget reconstruction onto the body we ship", and it changes the shape of the whole program: the 122 sliders are a parametric identity basis. Fitting a person becomes solving for slider values, which is a far better-conditioned problem than free-form vertex optimisation — bounded, always on the face manifold, and interpretable. Track 3's differentiable fit should target this basis first and fall back to free-form vertex displacement only for residual detail.
Work: rebuild face_uv_map.json against the parametric base (precompute_uv.py
is already parameterised by template), transfer the ARKit-52 expression set with
ICT-FaceKit (MIT) since the Anny base does not ship visemes, and re-baseline
every ISE report — scores are only comparable within one template.
This track also owns the body_type dead path. POST /reconstruct accepts
male | female | neutral and the selfie page offers the choice, but
face_pipeline._load_template
looks for templates/male.glb and female.glb, neither of which exists, so every
request silently falls back to default.glb. The parameter has never done
anything.
Staging the existing stock bases as those two files is the obvious patch and is the
wrong move. public/avatars/realistic-male.glb is topology-compatible (same 2162-
vertex Wolf3D_Head) but carries only 60 morph targets against default.glb's
67, so it would silently drop seven expressions. realistic-female.glb is a
different head entirely (1531 vertices, mesh names stripped), so it needs its own
precomputed map and a _find_head_prim that does not key on a mesh name — and it
would still leave two more templates to maintain in lockstep.
The parametric base dissolves the problem: body type stops being a discrete template swap and becomes slider values (gender, muscle, weight, height, hips) on one mesh with one skeleton, one UV map and one blendshape set. Fix it here, not by adding template files.
Where the credits go
| Line | Rough monthly | Notes |
|---|---|---|
| Warm GPU serving (current 6-service fleet) | $4-6k | Already running |
| RTX PRO 6000 warm instance | ~$2-3k | Track 3 serving + Track 4 headroom |
| Vertex AI custom training/experiment jobs | $2-5k while active | Track 3 sweeps, Track 2 evaluation |
| Vertex Imagen inference | usage-based, small | Track 2 |
Steady state stays in the range ops/gcp-credits-plan.md already sets out: comfortably over a year of runway on the grant. The standing directive is to spend on quality and never downgrade to conserve credits.
Owner decisions
Everything above proceeds without asking, per the standing GCP approvals. Two items do not:
- Commercial FLAME/MICA licence from MPI — external paid dependency (Track 3 alternative). Not needed if the optimisation path lands.
- Production deploys of the main API/frontend. GPU worker deploys and
config-only
gcloud run services updatechanges are pre-approved.