Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

11.8 KB
Raw Blame

Likeness evaluation

Does the avatar look like the person in the photos? This is the harness that answers it with a number instead of an opinion.

Phase 1 of the roadmap is verified on a likeness score: users complete a capture and mint an agent of themselves "with >=4/5 likeness score". Until this landed, nothing produced that number for a real generation. The platform could assert likeness; it could not measure it.

What was already measured, and why it was not enough

Two quality instruments existed before this one, and neither answers the identity question:

Instrument Asks Blind to
Realism quality bench (api/_lib/quality-bench.js) Is the generated asset photorealistic, coherent, on-brief? Whose face it is. A beautiful render of a stranger scores perfectly.
Identity Shape Error (eval/identity_eval.py) Does the head have this person's face shape? Texture, by design. Offline, against a synthetic reference set, and stores nothing per generation.

The gap between them is the whole product claim. A pipeline that paints a selfie onto a template head can score well on realism and poorly on shape while still reading, to the person looking at it, as them or as not them. That perception is what a face-recognition embedding measures.

The metric

For one finished reconstruction:

  1. The avatar is rendered head-framed at three yaws: head-on (0°), three-quarter (35°), and profile (65°), using the platform's own headshot scene preset so the framing matches what a user actually sees on a thumbnail or share card.
  2. Each render and each input capture is passed through a face detector (YuNet) and an identity embedder (SFace), producing a 128-dimensional L2-normalised vector per face.
  3. Each view is scored as the cosine similarity between its render vector and the best-matching capture vector.
  4. The headline likeness score is the head-on view's cosine, mapped to 1-5.

profile is 65° rather than a true 90° on purpose. A full side-on view presents at most three of the five landmarks the alignment step needs, so no model of this family can measure it. 65° is the steepest turn that still yields a measurable face, and it is well past the point where a frontal-only texture transfer stops covering the head.

Why 1-5 means what it means

The mapping is anchored to the embedding model's own calibration, not to taste:

Cosine Score Meaning
0.000 1.0 No relationship between the faces
0.363 3.0 OpenCV's published SFace decision boundary: the exact point where the model stops calling this the same person
0.682 4.0 The roadmap gate
1.000 5.0 Identical embedding

So 4/5 is a cosine of 0.68, roughly twice the same-identity threshold: not "arguably the same person" but "confidently the same person". Between the anchors the mapping is linear.

What the harness refuses to do

A missing input is reported as a status, never scored as a zero, because these are three different findings that a single low number would hide:

  • captures_unusable: no input photo contained a findable face. A capture-quality problem; nothing can be concluded about the avatar.
  • render_unusable: no render contained a findable face. The reconstruction did not produce something recognisable as a head.
  • ok with a low score: the avatar has a face, and it is not this person's.

budget_exhausted is a fourth: a sweep cut short by its time budget covers fewer views than a complete one, so its score is withheld rather than compared against runs that were not truncated.

Secondary readings

  • turn falloff: head-on cosine minus the worst view's. A pipeline that only holds up frontally is visible here and in no single-view number.
  • capture cohesion: mean pairwise cosine among the input photos. A selfie set whose own photos only agree at 0.5 cannot support a claim about the avatar at 0.6.
  • captures embedded / total: how much of the input the measurement rests on.

The models, and why these two

Both are pulled from OpenCV Zoo at a pinned commit and verified against a hardcoded SHA-256 before they are loaded.

Role Model License Size
Detection + 5 landmarks YuNet (face_detection_yunet_2023mar) MIT 233 KB
Identity embedding SFace (face_recognition_sface_2021dec) Apache-2.0 39 MB

Both licenses are commercially clean, which is the same bar the reconstruction worker holds ("no non-commercial 3DMM": no FLAME, BFM or SMPL weights anywhere in the pipeline). The widely-cited alternative, InsightFace's buffalo_l ArcFace pack, is materially stronger on benchmarks and is released for non-commercial research use only, so it is not available to us at any quality.

They run on onnxruntime-web's WASM backend (MIT) rather than onnxruntime-node. The node binding's installer downloads CUDA provider packages from NuGet during npm install and fails outright on a machine with no GPU tree, which would make the repository uninstallable. WASM has no native install step and behaves identically on a laptop and on Cloud Run.

Weights are not vendored into git. They are fetched once from the pinned URL, hash-verified, and cached under FACE_MODEL_CACHE_DIR (default: the system temp directory).

opencv_zoo stores its weights in Git LFS, so raw.githubusercontent.com serves a 130-byte pointer file rather than the model. The fetch uses github.com/<repo>/raw/<commit>/…, which resolves LFS. The SHA-256 check is what caught the difference.

Running it

Weekly, automatically

/api/cron/likeness-eval runs Mondays at 03:47 UTC, 24 minutes after the realism bench, on the same cadence. It takes finished reconstructions the current scorer version has not measured, scores them, and files the results. It bounds itself to a 280-second budget and skips any subject it cannot finish, so it never collects a Cloud Run 504 the way an unbounded sweep does. A sweep whose mean falls below the 4.0 gate logs at warn level:

gcloud logging read 'resource.type="cloud_run_revision" textPayload:"likeness-eval"' --freshness=7d

On demand

# Score everything the current scorer version has not seen (needs DATABASE_URL)
node --env-file=.env scripts/likeness-eval.mjs --backfill

# Run 10 reconstructions through the real production pipeline and score them
node --env-file=.env scripts/likeness-eval.mjs --live=10 --out=reports/likeness.json

--live signs in with the QA account and, for each subject, renders a headshot of a distinct avatar from the public library, submits that portrait as the reconstruction photo, polls the job to completion, and scores the finished avatar against the exact bytes it was built from. The subjects are therefore synthesised, never real people's photos, for the same reason the reconstruction worker synthesises its reference set.

A live subject is a real generation record, because live mode submits a real reconstruction. Not filing it would not keep it out of the numbers, only defer them: the next sweep would find an unscored creation and count it toward the gate rate like any user's avatar. So it is filed deliberately, under the status benchmark, which the distribution's status = 'ok' filter excludes from every aggregate while the outcome breakdown and the recent table still show it. The work queue keys on a row existing at all rather than on its status, so the subject is never re-picked either. Backfill subjects keep their real status.

Both modes print a cross-subject control: every avatar is also scored against every other subject's captures. A likeness number is unfalsifiable without it, because a scorer that returned 0.8 for any two faces would look identical to a working one on the matched pairs alone. The separation between the two distributions is the evidence the instrument works.

The first measurement (2026-08-14, threews.likeness.sface.v1)

The eleven reconstructions that existed in production when the harness landed, scored in one backfill:

Subjects 11 (10 scored, 1 no_captures)
Likeness score min 1.51, mean 2.19, max 3.04
At or above the 4/5 gate 0 of 10
Recognised as the same person (cosine >= 0.363) 2 of 10
Matched cosine min 0.093, median 0.218, mean 0.217, max 0.376
Cross-subject control min -0.024, median 0.176, mean 0.181, max 0.549
Separation (matched mean minus control mean) 0.035
Views that produced no detectable face 3 of 30, all at 65 degrees
Cost ~80 s per subject

Read the separation line first. A matched pair (an avatar against its own source photo) scores barely above a mismatched one, and the control's best mismatch (0.549) beats every matched pair in the set. Some reconstructions resemble a different subject's photo more than their own.

That is a finding about the pipeline, not about the instrument. The embedder separates faces strongly when the identity is actually present: scored across a 35 degree turn on eleven studio-composed avatars, the same avatar matched itself at mean cosine 0.776 while different avatars sat at 0.443, a separation of 0.33 rather than 0.035. The production reconstruction lane transfers a face texture onto a fixed template head, and by its own measurement only about 10% of the head's texels come from the photograph (avatar fidelity program). A near-noise-floor identity reading is the number that shape of pipeline should produce, and it is consistent with the geometry-side ISE result.

This is the baseline Phase 1 has to move. It is quoted here so the next run has something to be compared against, and it is only comparable within threews.likeness.sface.v1.

The board reads one subject higher than this table, and that discrepancy is what produced the benchmark status described above. Verifying --live submitted a real reconstruction, which registered a generation record like any other, and the next backfill scored it at 2.32 as if a user had made it. Live subjects are now filed as benchmark so they cannot reach the gate rate; the one that landed before the fix is left where it is rather than quietly edited out of a published baseline. The table above stays the dated snapshot it was.

Reading the results

/likeness-bench (internal, admin or x-ops-secret) renders the distribution: the fraction clearing the 4/5 gate, the histogram, mean turn falloff, the outcome breakdown, and the most recent individual scores. The same data is available as JSON:

curl -H "x-ops-secret: $OPS_SECRET" 'https://three.ws/api/likeness-bench?days=30&limit=50'

Scores are keyed on the generation record (forge_creations.id) and stamped with scorer_version. A score is only comparable to another score taken by the same scorer version. When the model, the alignment template or the view angles change, the version changes with them, old rows stay valid, and where scorer_version <> $current is the backfill query.

What is deliberately not stored

No capture URLs, no crops, and no embeddings. A face embedding is biometric data; this table holds the measurement, not the person. The stored report carries per-view statuses and cosines, and records a rejected capture by index and reason only.

Related