Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

18.5 KB
Raw Blame
"""
Selfie → 3D face-shape identity morph (Phase 2: geometry, not just texture).

Phase 1 (face_pipeline.py) only paints the user's face texture onto a generic
Wolf3D head, so the *shape* — face width, jaw, nose projection, brow, cheekbones
— stays that of the template. This module recovers the person's actual face
geometry from a single photo and reshapes the template head to match, which is
the difference between "a face sticker on a mannequin" and an avatar that reads
as the same person (the gap vs Avaturn).

Approach — sparse-to-dense identity transfer, fully commercial-clean:
  1. MediaPipe FaceMesh (Apache-2.0, already a dependency) → 468 3D landmarks.
  2. Umeyama similarity-align the detected landmarks onto MediaPipe's neutral
     canonical face model (shared frame). The residual (person - canonical) on a
     stable identity subset IS the person's face-shape deviation from neutral.
  3. Carry that per-landmark displacement onto the template head's corresponding
     vertices (precomputed nearest-vertex map), scaled into head units.
  4. Interpolate the sparse displacements across all head vertices with a
     thin-plate spline, which passes exactly through the control points so
     localised identity (a wider jaw, a longer nose) survives instead of being
     averaged away — the failure mode of normalised-Gaussian/Shepard weighting.
     A Gaussian off-face mask fades the field to zero away from the face, since
     TPS extrapolates freely, so the neck, scalp and back of the head stay put.
  5. face_pipeline writes the morphed positions back with glb_ops.set_head_geometry,
     which preserves skinning and all 52 ARKit blendshapes + visemes.

Everything below the MediaPipe adapter is pure numpy and is unit-tested with
synthetic landmarks, so the geometry math is verifiable without a GPU or a photo.
"""

from __future__ import annotations

import json
import logging
from pathlib import Path
from typing import Optional

import numpy as np

log = logging.getLogger("face_geometry")

HERE = Path(__file__).parent
FACE_MAP_PATH = HERE / "face_uv_map.json"

# Landmarks that encode stable identity (bone/cartilage structure) rather than
# transient expression. We deliberately exclude the eyeballs/irises (468-477),
# inner lips and tongue: a smile or blink in the selfie must not deform the rest
# pose of the avatar. Face oval, nose bridge/tip, cheeks, brow ridge, jawline.
_STABLE_IDENTITY_INDICES = [
    # face oval / silhouette
    10, 338, 297, 332, 284, 251, 389, 356, 454, 323, 361, 288,
    397, 365, 379, 378, 400, 377, 152, 148, 176, 149, 150, 136,
    172, 58, 132, 93, 234, 127, 162, 21, 54, 103, 67, 109,
    # nose bridge + tip + alae
    6, 197, 195, 5, 4, 1, 19, 94, 2, 98, 327, 129, 358,
    # cheeks / malar
    50, 280, 116, 345, 117, 346, 118, 347, 205, 425,
    # brow ridge
    107, 336, 66, 296, 105, 334, 63, 293,
    # jaw / chin
    32, 262, 211, 431, 170, 395, 208, 428,
]


class FaceMap:
    """Precomputed template↔canonical correspondence (built by precompute_uv.py)."""

    def __init__(self, data: dict):
        self.canonical_norm = np.asarray(data["canonical_norm"], dtype=np.float64)  # (468,3)
        self.landmark_vtx = np.asarray(data["landmark_vtx"], dtype=np.int64)        # (468,)
        self.head_face_scale = float(data["head_face_scale"])
        n = self.canonical_norm.shape[0]
        self.stable_indices = np.asarray(
            [i for i in data.get("stable_indices", _STABLE_IDENTITY_INDICES) if i < n],
            dtype=np.int64,
        )

    @classmethod
    def load(cls, path: Path = FACE_MAP_PATH) -> Optional["FaceMap"]:
        if not path.exists():
            return None
        data = json.loads(path.read_text())
        if "canonical_norm" not in data or "landmark_vtx" not in data:
            # Older texture-only map without geometry fields — geometry morph off.
            return None
        return cls(data)


# ── alignment ────────────────────────────────────────────────────────────────

def umeyama(src: np.ndarray, dst: np.ndarray) -> tuple[float, np.ndarray, np.ndarray]:
    """
    Least-squares similarity transform (scale s, rotation R, translation t) that
    best maps src → dst (both (N,3)). Umeyama 1991. Returns (s, R, t) with
    dst ≈ s * R @ src + t.
    """
    src = np.asarray(src, dtype=np.float64)
    dst = np.asarray(dst, dtype=np.float64)
    mu_s, mu_d = src.mean(0), dst.mean(0)
    sc, dc = src - mu_s, dst - mu_d
    cov = (dc.T @ sc) / src.shape[0]
    U, D, Vt = np.linalg.svd(cov)
    S = np.eye(3)
    if np.linalg.det(U) * np.linalg.det(Vt) < 0:
        S[2, 2] = -1.0  # reflection guard — keep a proper rotation
    R = U @ S @ Vt
    var_s = (sc ** 2).sum() / src.shape[0]
    s = float((D * np.diag(S)).sum() / var_s) if var_s > 0 else 1.0
    t = mu_d - s * (R @ mu_s)
    return s, R, t


def _apply(s: float, R: np.ndarray, t: np.ndarray, pts: np.ndarray) -> np.ndarray:
    return (s * (R @ pts.T)).T + t


# ── the morph ────────────────────────────────────────────────────────────────

def morph_head_to_landmarks(
    base_positions: np.ndarray,
    face_map: FaceMap,
    detected_landmarks: np.ndarray,
    *,
    strength: float = 0.6,
    max_displacement_frac: float = 0.55,
    falloff: float = 1.6,
) -> np.ndarray:
    """
    Reshape the template head to the person's face identity.

    Args:
        base_positions:     (V,3) template head vertices (head/metre space).
        face_map:           precomputed correspondence + scale.
        detected_landmarks: (468,3) MediaPipe landmarks for the selfie. May be
                            (478,3) with irises — extra rows are ignored.
        strength:           0..1 how strongly to move toward the person's shape.
                            <1 keeps the result on a plausible face manifold.
        max_displacement_frac: clamp per-control displacement to this fraction of
                            the face scale — rejects landmark/pose outliers.
                            0.55 ≈ 5.7 cm of travel per control point.
        falloff:            radius of the off-face mask, as a multiple of the
                            median control-point spacing. Larger = the morph
                            reaches further past the face. Governs the scalp,
                            ears and neck only, so it is invisible to ISE (which
                            samples at the control points) — tune it against the
                            back-of-head assertions in test_face_geometry.py, not
                            against the fidelity score.

    Returns:
        (V,3) float32 morphed vertices. Vertex count/order unchanged, so the
        caller can write them back without disturbing skinning or blendshapes.
    """
    base = np.asarray(base_positions, dtype=np.float64)
    canon = face_map.canonical_norm                    # (468,3), unit-cube frame
    idx = face_map.stable_indices
    det = np.asarray(detected_landmarks, dtype=np.float64)[: canon.shape[0]]

    # 1. Similarity-align the person's landmarks onto the neutral canonical face,
    #    using only stable identity points, then measure the residual there.
    s, R, t = umeyama(det[idx], canon[idx])
    det_aligned = _apply(s, R, t, det)                 # person in canonical frame
    residual = det_aligned - canon                     # identity deviation (norm units)

    # 2. Carry to head units and clamp outliers.
    #
    # Defaults (strength 0.6, clamp 0.55) are measured, not guessed — they sit at
    # the knee of a fidelity-vs-robustness curve built from two sweeps over the
    # 40-face reference set:
    #
    #   `eval.tune_morph`  — mean ISE, i.e. is the face shape right?
    #   `eval.robustness`  — does a *degraded photo of the same person* still give
    #                        the same head? (blur, 12° turn, occlusion, dim, 1/6
    #                        resolution)
    #
    #   clamp     ISE   vs floor   instability
    #    0.18  0.2932     26.5%        0.0027
    #    0.45  0.1684     57.8%        0.0053
    #    0.55  0.1591     60.1%        0.0057   ← knee
    #    0.65  0.1587     60.2%        0.0060
    #    1.00  0.1587     60.2%        0.0060
    #
    # Fidelity flatlines past 0.65 while instability keeps rising, so anything
    # looser is pure downside. Across the usable range each step buys ~40-50x more
    # fidelity than it costs in stability, which is why the original 0.18 was the
    # real error: at ~1.9 cm of travel per control point it was throttling genuine
    # facial variation, not rejecting outliers. 0.55 ≈ 5.7 cm.
    #
    # Known limit: all 75 degraded images still yielded a face detection, so the
    # catastrophic regime the clamp ultimately guards — a gross mis-detection, a
    # face found on a background object — remains unmeasured. A detection failure
    # is itself safe (the job falls back to texture-only), but a confident *wrong*
    # detection is not, and nothing here bounds that yet.
    disp = residual[idx] * face_map.head_face_scale * float(strength)  # (K,3)
    clamp = max_displacement_frac * face_map.head_face_scale
    mag = np.linalg.norm(disp, axis=1, keepdims=True)
    over = (mag > clamp).ravel()
    if over.any():
        disp[over] *= (clamp / mag[over])

    # 3. Control points = the template vertices those stable landmarks map to.
    ctrl_vtx = face_map.landmark_vtx[idx]
    ctrl_pos = base[ctrl_vtx]                           # (K,3) in head space

    # Deduplicate control vertices (several landmarks can share a nearest vertex);
    # average their displacements so no vertex is double-weighted.
    uniq, inv = np.unique(ctrl_vtx, return_inverse=True)
    acc = np.zeros((uniq.shape[0], 3)); cnt = np.zeros((uniq.shape[0], 1))
    np.add.at(acc, inv, disp); np.add.at(cnt, inv, 1.0)
    ctrl_pos = base[uniq]
    ctrl_disp = acc / np.maximum(cnt, 1.0)

    # 4. Thin-plate-spline INTERPOLATION of the control displacements over every
    #    head vertex. TPS passes THROUGH the anchors, so localised identity (a
    #    wider jaw, a longer nose) is preserved instead of being averaged away —
    #    the failure mode of Shepard/normalised-Gaussian weighting. TPS can drift
    #    far from the anchors, but the off-face mask zeroes it there, so the neck
    #    and scalp stay put. Same RBF family face_pipeline uses for its UV warp.
    spacing = _median_nn_spacing(ctrl_pos)
    sigma = max(spacing * float(falloff), 1e-4)
    nearest = np.sqrt(_pairwise_sq(base, ctrl_pos).min(1, keepdims=True))
    mask = np.exp(-(nearest ** 2) / (2.0 * (sigma * 2.0) ** 2))

    from scipy.interpolate import RBFInterpolator
    rbf = RBFInterpolator(ctrl_pos, ctrl_disp, kernel="thin_plate_spline", smoothing=1e-3)
    field = rbf(base) * mask

    return (base + field).astype(np.float32)


def register_head_to_target(
    base_positions: np.ndarray,
    face_map: FaceMap,
    target_landmarks: np.ndarray,
    *,
    target_points: Optional[np.ndarray] = None,
    strength: float = 1.0,
    max_displacement_frac: float = 0.4,
    falloff: float = 3.0,
    icp_iterations: int = 4,
) -> np.ndarray:
    """
    Dense non-rigid registration (the v2 path): reshape the template head to a
    high-fidelity target head from MICA/FLAME, a re-based HRN, or any dense face
    model. Model-agnostic — it needs the target's 468 landmark positions (every
    such model exposes a landmark embedding) and, optionally, a dense surface.

    Two stages, because they solve different problems:
      A. Landmark-anchored fit — the 468 target landmarks have KNOWN correspondence
         to the template's control vertices, so this recovers gross shape (face
         width, length, nose/jaw projection) without the tangential "sliding" that
         defeats pure closest-point ICP on a smooth surface.
      B. Optional dense point-to-plane ICP — refines fine surface detail along the
         normal direction, where closest-point matching is well-posed.

    The target is first Umeyama-aligned to the template via the shared landmarks,
    so the caller can pass the model's raw output frame. Vertex count/order are
    preserved, so `glb_ops.set_head_geometry` writes it back with the rig and every
    ARKit blendshape intact.

    Args:
        target_landmarks: (>=468, 3) the target's landmark positions (any frame).
        target_points:    optional (M,3) dense target surface for stage B.

    Returns:
        (V,3) float32 morphed vertices.
    """
    base = np.asarray(base_positions, dtype=np.float64)
    tgt_lm = np.asarray(target_landmarks, dtype=np.float64)[: face_map.canonical_norm.shape[0]]
    face_scale = face_map.head_face_scale
    clamp = max_displacement_frac * face_scale

    # Control vertices and the target landmark that drives each (dedup shared).
    lm_vtx = face_map.landmark_vtx
    uniq, inv = np.unique(lm_vtx, return_inverse=True)
    tgt_per_uniq = np.zeros((uniq.shape[0], 3)); cnt = np.zeros((uniq.shape[0], 1))
    np.add.at(tgt_per_uniq, inv, tgt_lm); np.add.at(cnt, inv, 1.0)
    tgt_ctrl = tgt_per_uniq / np.maximum(cnt, 1.0)         # target pos per control vtx
    base_ctrl = base[uniq]

    # Align the target landmarks onto the template's control vertices (absorbs the
    # model's scale/orientation/translation) so displacements are in head space.
    s, R, t = umeyama(tgt_ctrl, base_ctrl)
    tgt_ctrl = _apply(s, R, t, tgt_ctrl)

    # Off-face mask so only the face tracks the target (scalp/ears/neck frozen).
    sigma = max(_median_nn_spacing(base_ctrl) * float(falloff), 1e-4)
    nearest = np.sqrt(_pairwise_sq(base, base_ctrl).min(1, keepdims=True))
    mask = np.exp(-(nearest ** 2) / (2.0 * (sigma * 2.0) ** 2))

    def interpolate(centers, disp):
        """
        Thin-plate-spline INTERPOLATION of the control displacements over all
        vertices (passes through the anchors exactly — unlike Shepard averaging,
        which smooths them away and dilutes localised shape like a wider jaw),
        faded off-face by the mask. Same RBF family face_pipeline uses for its
        texture warp.
        """
        from scipy.interpolate import RBFInterpolator
        rbf = RBFInterpolator(centers, disp, kernel="thin_plate_spline", smoothing=1e-3)
        return rbf(base) * mask

    # ── Stage A: anchored gross-shape fit (known correspondence, no sliding) ──
    disp = (tgt_ctrl - base_ctrl) * float(strength)
    m = np.linalg.norm(disp, axis=1, keepdims=True)
    over = (m > clamp).ravel()
    if over.any():
        disp[over] *= (clamp / m[over])
    current = base + interpolate(base_ctrl, disp)

    # ── Stage B: dense point-to-point ICP refinement (surface detail) ──
    if target_points is not None and len(target_points) > 0:
        target = np.asarray(target_points, dtype=np.float64)
        target = _apply(s, R, t, target)                  # same frame as landmarks
        driven = np.where(mask.ravel() > 0.3)[0]
        for _ in range(max(0, icp_iterations)):
            d2 = _pairwise_sq(current[driven], target)
            nn = d2.argmin(1)
            step = (target[nn] - current[driven]) * 0.5    # damped for stability
            sm = np.linalg.norm(step, axis=1, keepdims=True)
            so = (sm > clamp * 0.5).ravel()
            if so.any():
                step[so] *= (clamp * 0.5 / sm[so])
            current = current + interpolate(current[driven], step)

    return current.astype(np.float32)


def _pairwise_sq(a: np.ndarray, b: np.ndarray) -> np.ndarray:
    """Squared Euclidean distances, (len(a), len(b))."""
    return ((a[:, None, :] - b[None, :, :]) ** 2).sum(-1)


def _median_nn_spacing(pts: np.ndarray) -> float:
    if pts.shape[0] < 2:
        return 1.0
    d2 = _pairwise_sq(pts, pts)
    np.fill_diagonal(d2, np.inf)
    return float(np.median(np.sqrt(d2.min(1))))


# ── MediaPipe adapter ────────────────────────────────────────────────────────

def estimate_yaw_deg(detected_landmarks: np.ndarray, face_map: FaceMap) -> float:
    """
    Absolute head yaw in degrees, recovered from the same Umeyama alignment the
    morph uses (so it costs one extra arctan and cannot disagree with it).

    Why the morph needs this: a single photo only constrains the geometry it can
    see. Past roughly a third of a turn the far side of the face is self-occluded,
    MediaPipe's depth for those landmarks becomes extrapolation rather than
    measurement, and the morph faithfully fits that noise into the skull.

    Measured on the benchmark: the 40 clean reference faces sit at 0.9° mean /
    2.7° worst, while a near-profile shot reads 58.6° and is the only input —
    adversarial or real — that drives the head outside the displacement band every
    genuine face produces. Photos *of* photos (a poster, a phone screen, a framed
    painting) land in between at 9-18°, which is why the threshold is set well
    above them: those already yield in-band heads and do not need blocking.
    """
    canon = face_map.canonical_norm
    idx = face_map.stable_indices
    det = np.asarray(detected_landmarks, dtype=np.float64)[: canon.shape[0]]
    _, R, _ = umeyama(det[idx], canon[idx])
    return float(abs(np.degrees(np.arctan2(-R[2, 0], np.hypot(R[0, 0], R[1, 0])))))


def landmarks_to_array(landmarks, width: float | None = None, height: float | None = None) -> np.ndarray:
    """
    MediaPipe NormalizedLandmark list → (N,3) array. y is flipped to +up so the
    frame matches the canonical model (image space is y-down).

    **Pass the source image's pixel dimensions.** MediaPipe normalises x by image
    width and y by image *height*, so on any non-square photo the returned frame
    is anisotropic: the same face in a 3:4 portrait and a 1:1 crop yields two
    differently-proportioned point clouds. Umeyama fits a single uniform scale
    and cannot undo that, so the aspect distortion survives alignment and is read
    as facial identity — baking the camera's aspect ratio into the avatar's skull.
    Scaling by (W, H, W) — z shares x's width normalisation — restores an
    isotropic pixel frame where uniform-scale alignment is valid.

    Dimensions are optional only so existing callers keep working; every caller
    that has the image should pass them.
    """
    pts = np.array([[lm.x, -lm.y, lm.z] for lm in landmarks], dtype=np.float64)
    if width and height:
        pts *= np.array([float(width), float(height), float(width)])
    return pts