Skip to content

Type a motion, watch a rigged character perform it: fast video model + GVHMR + live Mixamo retargeting

JavaScript

Prompt2Rig: prompt → H3 Max Turbo clip → GVHMR motion → Mixamo rig

blendi-remade38bd8ca6c212
diff --git a/.env.example b/.env.example
new file mode 100644
index 0000000..06fbab2
--- /dev/null
+++ b/.env.example
@@ -0,0 +1,3 @@
+FAL_KEY=your_fal_key_here
+PORT=5177
+# HQ_URL=http://127.0.0.1:5178
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..caeb919
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,17 @@
+# secrets
+.env
+
+# HQ worker: env, downloaded weights, licensed body model, per-clip cache, logs
+hq/.venv/
+hq/weights/
+hq/body_models/
+hq/cache/
+*.log
+*.pt
+
+# local reference checkout
+vendor/
+
+node_modules/
+__pycache__/
+.DS_Store
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..17139fc
--- /dev/null
+++ b/README.md
@@ -0,0 +1,88 @@
+# Prompt2Rig
+
+**Type a motion, watch a rigged character perform it — in seconds.**
+
+A text prompt goes to a faster-than-real-time video model ([MiniMax H3 Max Turbo](https://fal.ai/models/minimax/h3-max-turbo/image-to-video) on fal), the clip's human motion is recovered with [GVHMR](https://github.com/zju3dv/GVHMR) on a local GPU, and retargeted live onto a Mixamo skeleton in the browser. Inspired by [Motion2MixamoRig](https://github.com/MaxLiu22/Motion2MixamoRig), rebuilt for interactive speed.
+
+![Prompt2Rig demo — boxing combo generated from a prompt and retargeted to X Bot](docs/demo.gif)
+
+## How it works
+
+```mermaid
+flowchart LR
+ P[prompt] --> V["H3 Max Turbo<br/>5s clip · ~0.5s GPU"]
+ I["performer image<br/>or previous clip's last frame"] --> V
+ V --> L["MediaPipe (browser)<br/>instant preview"]
+ V --> H["GVHMR (local GPU)<br/>SMPL motion · ~3.4s"]
+ L --> R[Mixamo rig]
+ H -- cross-fade --> R
+ R --> G[.glb export]
+```
+
+| Stage | Where | Time (5s clip, RTX 4090) |
+|---|---|---|
+| Video generation | fal · H3 Max Turbo | ~2–3s wall (0.5s GPU) |
+| **Live** preview — MediaPipe Pose, retargeted per frame | browser | immediate |
+| **HQ** motion — GVHMR → SMPL rotations → Mixamo | local GPU worker | ~3.4s (0.7× real time) |
+
+The live pass hides the wait; the HQ pass is the real result. GVHMR sees the whole clip at once, outputs joint *rotations* on a fixed-proportion body, and recovers a gravity-aware root path — so there's no jitter, limb stretching, or twisted forearms.
+
+![App UI — generated clip with pose overlay, retargeted character, and take history](docs/app.png)
+
+## Quick start
+
+**Needs:** Node 18+, a [fal](https://fal.ai) API key. For the HQ pass: Windows + NVIDIA GPU, [`uv`](https://docs.astral.sh/uv/), `ffmpeg` on PATH.
+
+```bash
+git clone https://github.com/blendi-remade/prompt2rig && cd prompt2rig
+cp .env.example .env # add your FAL_KEY
+node server.mjs # → http://localhost:5177 (live preview works now)
+```
+
+**HQ pass (optional, recommended):**
+
+```powershell
+hq\setup.ps1 # venv, CUDA PyTorch, GVHMR + ~5 GB checkpoints
+```
+
+Then download **SMPL-X v1.1 (NPZ)** from [smpl-x.is.tue.mpg.de](https://smpl-x.is.tue.mpg.de/) (free registration) and put `SMPLX_NEUTRAL.npz` in `hq/body_models/smplx/`. Start the worker next to the web server:
+
+```powershell
+hq\.venv\Scripts\python hq\worker.py # → 127.0.0.1:5178, loads once and stays warm
+```
+
+## Using it
+
+- **Generate** a prompt or click a preset. By default every clip starts from the same flux-generated *performer* (one person, full body, plain backdrop, static camera — ideal pose-estimation input).
+- **Stream mode** — one continuous performance: each clip continues from the previous clip's last frame, plays once its HQ motion is ready while the next one generates, and short idle clips fill gaps between prompts. First motion ≈ 6s after the first prompt, continuous after that.
+- **⬇ GLB** on any take exports the skinned character with the animation clip (Blender, Unity, three.js).
+- **Load character…** swaps in any Mixamo-rigged `.glb`/`.fbx`; **Use a video file…** runs the live pass on your own footage.
+
+## Project layout
+
+| Path | |
+|---|---|
+| `server.mjs` | zero-dependency server; proxies fal (key stays server-side) and the HQ worker |
+| `public/app.js` | UI, generation queue, live pose loop, playback & blending, stream mode, GLB export |
+| `public/retarget.js` | Mixamo rig wrapper: MediaPipe landmarks → bones, SMPL rotations → bones, grounding |
+| `hq/worker.py` | FastAPI worker around the `gvhmr` package, tuned for single-person generated clips |
+
+<details>
+<summary><b>HQ worker speedups</b> (stock <code>gvhmr.recover</code> 10s → 3.4s per 5s clip)</summary>
+
+- batched single-person YOLO11m instead of frame-by-frame YOLOv8x + BoT-SORT tracking
+- decoded frames reused by the crop stage (OpenCV) instead of a second PyAV decode
+- ViTPose without flip-test (`HQ_POSE_FLIP=1` to re-enable — measured 0.6° mean joint difference)
+- 24→30 fps resample with ffmpeg `ultrafast`; all GPU work on one warmed-up thread
+- left/right keypoint-swap stabilizer from Motion2MixamoRig
+- camera-relative heading so chained clips line up (seams: 1–5° hips, ≤5 cm root)
+
+</details>
+
+## Credits & licenses
+
+- [GVHMR](https://github.com/zju3dv/GVHMR) (ZJU3DV) via the [`gvhmr`](https://pypi.org/project/gvhmr/) package · [Motion2MixamoRig](https://github.com/MaxLiu22/Motion2MixamoRig) (Apache-2.0) for the idea and the keypoint stabilizer
+- [MediaPipe Pose](https://ai.google.dev/edge/mediapipe/solutions/vision/pose_landmarker) · [three.js](https://threejs.org) (X Bot model from its examples) · video/image models on [fal](https://fal.ai)
+- **SMPL-X** is licensed for non-commercial research only and is not included — download it yourself.
+
+Prototype — tested on Windows 11 with an RTX 4090.
diff --git a/docs/app.png b/docs/app.png
new file mode 100644
index 0000000..8480a08
Binary files /dev/null and b/docs/app.png differ
diff --git a/docs/demo.gif b/docs/demo.gif
new file mode 100644
index 0000000..ab9f484
Binary files /dev/null and b/docs/demo.gif differ
diff --git a/hq/requirements.txt b/hq/requirements.txt
new file mode 100644
index 0000000..8b43f60
--- /dev/null
+++ b/hq/requirements.txt
@@ -0,0 +1,7 @@
+# HQ worker deps (installed by hq/setup.ps1 — torch and chumpy need special flags, see there).
+# Motion2MixamoRig pulls in gvhmr[preproc,rtmpose]; we reuse its 2D-pose L/R stabilizer.
+motion2mixamorig @ git+https://github.com/MaxLiu22/Motion2MixamoRig@2835dcdfba30e78a2610360b9b2a1e1121d85159
+gvhmr==1.6.1
+ultralytics>=8.3
+fastapi>=0.110
+uvicorn[standard]>=0.29
diff --git a/hq/setup.ps1 b/hq/setup.ps1
new file mode 100644
index 0000000..8e4da22
--- /dev/null
+++ b/hq/setup.ps1
@@ -0,0 +1,21 @@
+# One-time setup for the HQ worker: Windows + NVIDIA GPU, needs `uv` and `ffmpeg` on PATH.
+# Afterwards put SMPLX_NEUTRAL.npz in hq/body_models/smplx/ (see README).
+$ErrorActionPreference = "Stop"
+Set-Location $PSScriptRoot
+
+uv venv --python 3.11 .venv
+$env:VIRTUAL_ENV = Join-Path $PSScriptRoot ".venv"
+
+# CUDA build of PyTorch first, so gvhmr doesn't pull the CPU wheel from PyPI.
+uv pip install "torch==2.7.1" "torchvision==0.22.1" --index-url https://download.pytorch.org/whl/cu126
+# chumpy (legacy gvhmr dependency) breaks under PEP 517 build isolation.
+uv pip install pip setuptools wheel "numpy>=1.26"
+uv pip install --no-build-isolation "chumpy==0.70"
+uv pip install -r requirements.txt
+
+# GVHMR / HMR2 / ViTPose checkpoints (~5 GB) into hq/weights.
+$env:GVHMR_CHECKPOINTS = Join-Path $PSScriptRoot "weights"
+$env:GVHMR_BODY_MODELS = Join-Path $PSScriptRoot "body_models"
+& .venv\Scripts\gvhmr.exe download demo
+New-Item -ItemType Directory -Force body_models\smplx | Out-Null
+Write-Host "Done. Now add body_models\smplx\SMPLX_NEUTRAL.npz (https://smpl-x.is.tue.mpg.de/, v1.1)."
diff --git a/hq/worker.py b/hq/worker.py
new file mode 100644
index 0000000..5a75143
--- /dev/null
+++ b/hq/worker.py
@@ -0,0 +1,303 @@
+"""Prompt2Rig HQ worker — keeps GVHMR warm on the GPU and turns a clip URL into SMPL motion.
+
+POST /process {"video_url": "..."} → {n, fps, global_orient, body_pose, transl, leg_length, timings}
+GET /health → {ok, device, error?}
+
+Run: hq/.venv/Scripts/python hq/worker.py (listens on 127.0.0.1:5178; setup in README)
+"""
+
+from __future__ import annotations
+
+import hashlib
+import os
+import shutil
+import subprocess
+import time
+import traceback
+import urllib.request
+from pathlib import Path
+
+HQ = Path(__file__).resolve().parent
+os.environ.setdefault("GVHMR_CHECKPOINTS", str(HQ / "weights"))
+os.environ.setdefault("GVHMR_BODY_MODELS", str(HQ / "body_models"))
+CACHE = HQ / "cache"
+SMPLX = Path(os.environ["GVHMR_BODY_MODELS"]) / "smplx" / "SMPLX_NEUTRAL.npz"
+DETECTOR = os.environ.get("HQ_DETECTOR", "fast")
+TRACKER = os.environ.get("HQ_TRACKER", "bytetrack.yaml")
+POSE2D = os.environ.get("HQ_POSE2D", "vitpose")
+# ViTPose flip-test runs every crop twice (mirrored) for a small accuracy gain.
+POSE_FLIP = os.environ.get("HQ_POSE_FLIP", "0") == "1"
+
+import numpy as np # noqa: E402
+import torch # noqa: E402
+
+try: # let onnxruntime-gpu (RTMPose) use the CUDA/cuDNN DLLs that ship with torch
+ import onnxruntime
+
+ onnxruntime.preload_dlls()
+except Exception:
+ pass
+
+from fastapi import FastAPI, HTTPException # noqa: E402
+from pydantic import BaseModel # noqa: E402
+
+app = FastAPI()
+state: dict = {"pipe": None, "error": None, "device": None}
+# All GPU work runs on ONE thread: CUDA/cuBLAS/cuDNN handles are per-thread, so a fresh request
+# thread would otherwise pay their setup (seconds) and miss the warm-up.
+from concurrent.futures import ThreadPoolExecutor # noqa: E402
+
+GPU = ThreadPoolExecutor(max_workers=1, thread_name_prefix="gpu")
+
+
+def log(msg: str) -> None:
+ print(f"[hq {time.strftime('%H:%M:%S')}] {msg}", flush=True)
+
+
+FRAME_CACHE: dict[str, list] = {}
+
+
+def fast_read_video_np(video_path, start_frame=0, end_frame=-1, scale=1.0):
+ """Drop-in for gvhmr's PyAV reader: reuse the detector's decoded frames, resize with OpenCV.
+
+ Same output contract (N, H, W, 3) RGB uint8; bicubic like ffmpeg's default scale filter.
+ """
+ import cv2
+
+ frames = FRAME_CACHE.get(str(Path(video_path).resolve()))
+ if frames is None:
+ cap = cv2.VideoCapture(str(video_path))
+ frames = []
+ while True:
+ ok, img = cap.read()
+ if not ok:
+ break
+ frames.append(img)
+ cap.release()
+ frames = frames[start_frame:(None if end_frame == -1 else end_frame)]
+ if scale != 1.0:
+ h, w = frames[0].shape[:2]
+ size = (int(w * scale), int(h * scale))
+ frames = [cv2.resize(f, size, interpolation=cv2.INTER_CUBIC) for f in frames]
+ return np.ascontiguousarray(np.stack(frames)[..., ::-1])
+
+
+class FastSingleDetector:
+ """Batched YOLO detection keeping the largest person per frame — replaces per-frame tracking.
+
+ Generated clips contain exactly one person, so identity tracking buys nothing; batching the
+ detector is ~5x faster than ultralytics' frame-by-frame `track()`.
+ """
+
+ _inst = None
+
+ @classmethod
+ def get(cls):
+ cls._inst = cls._inst or cls()
+ return cls._inst
+
+ def __init__(self, ckpt: Path = HQ / "weights" / "yolo11m.pt", batch: int = 32):
+ from ultralytics import YOLO
+ from ultralytics.utils.downloads import attempt_download_asset
+
+ ckpt.parent.mkdir(parents=True, exist_ok=True)
+ self.yolo, self.batch = YOLO(attempt_download_asset(str(ckpt))), batch
+
+ def get_one_track(self, video_path) -> torch.Tensor:
+ import cv2
+ from gvhmr.utils.net_utils import moving_average_smooth
+
+ cap = cv2.VideoCapture(str(video_path))
+ frames = []
+ while True:
+ ok, img = cap.read()
+ if not ok:
+ break
+ frames.append(img)
+ cap.release()
+ FRAME_CACHE.clear()
+ FRAME_CACHE[str(Path(video_path).resolve())] = frames
+ boxes = np.zeros((len(frames), 4), np.float32)
+ found = np.zeros(len(frames), bool)
+ for i in range(0, len(frames), self.batch):
+ results = self.yolo.predict(frames[i:i + self.batch], classes=[0], conf=0.3, half=True, device=0, verbose=False)
+ for j, r in enumerate(results):
+ if len(r.boxes):
+ xyxy = r.boxes.xyxy.cpu().numpy()
+ area = (xyxy[:, 2] - xyxy[:, 0]) * (xyxy[:, 3] - xyxy[:, 1])
+ boxes[i + j], found[i + j] = xyxy[area.argmax()], True
+ if not found.any():
+ raise RuntimeError("no person detected in the clip")
+ idx = np.arange(len(frames))
+ for k in range(4): # fill frames with no detection
+ boxes[:, k] = np.interp(idx, idx[found], boxes[found, k])
+ return moving_average_smooth(torch.from_numpy(boxes), window_size=5, dim=0)
+
+
+def load_pipeline() -> None:
+ if not SMPLX.exists():
+ state["error"] = f"SMPL-X missing: put SMPLX_NEUTRAL.npz in {SMPLX.parent}"
+ log(state["error"])
+ return
+ try:
+ import gvhmr
+ import gvhmr.cli.demo as demo
+
+ if DETECTOR == "fast":
+ demo.make_detector = lambda *a, **kw: FastSingleDetector.get() # cfg still names a valid YOLO preset
+ import gvhmr.utils.imgcrop as imgcrop
+
+ imgcrop.read_video_np = fast_read_video_np
+
+ t0 = time.perf_counter()
+ state["pipe"] = gvhmr.pipeline("human-motion-recovery", device="cuda")
+ state["device"] = torch.cuda.get_device_name(0)
+ state["error"] = None
+ log(f"GVHMR loaded on {state['device']} in {time.perf_counter() - t0:.1f}s")
+ warmup()
+ except Exception as e: # noqa: BLE001
+ state["error"] = f"{type(e).__name__}: {e}"
+ traceback.print_exc()
+
+
+def warmup() -> None:
+ """Run one tiny clip so CUDA kernels / ONNX sessions are initialized before the first real request."""
+ clip = CACHE / "_warmup.mp4"
+ if not clip.exists():
+ return
+ try:
+ t0 = time.perf_counter()
+ shutil.rmtree(CACHE / "_warmup", ignore_errors=True) # cached stages would skip the warm-up
+ result, _ = recover(clip, CACHE / "_warmup")
+ leg_length(result.smpl_params_world)
+ log(f"warmup done in {time.perf_counter() - t0:.1f}s")
+ except Exception as e: # noqa: BLE001
+ log(f"warmup failed: {e}")
+
+
+def recover(video: Path, out_dir: Path):
+ from motion2mixamorig.extract import _stabilize_pose_cache
+
+ phases: dict[str, float] = {}
+ last = {"name": None, "t": time.perf_counter()}
+
+ def on_progress(_frac: float, desc: str) -> None:
+ now = time.perf_counter()
+ if desc != last["name"]:
+ if last["name"]:
+ phases[last["name"]] = phases.get(last["name"], 0) + now - last["t"]
+ last.update(name=desc, t=now)
+
+ # Generated clips are single-person, plain background, static camera: a mid-size YOLO with ByteTrack
+ # (no camera-motion compensation) tracks as well as YOLOv8x+BoT-SORT at a fraction of the cost, and
+ # ViTPose shares one decode+crop pass with the HMR2 feature backbone.
+ kwargs = dict(static_camera=True, detector="yolo11m" if DETECTOR == "fast" else DETECTOR, pose2d=POSE2D, set_overrides=[f"++detector.tracker={TRACKER}"] + ([] if POSE_FLIP else ["++pose2d.flip_test=false"]),
+ output_dir=str(out_dir), output_root=str(out_dir.parent), progress=False, progress_callback=on_progress)
+ t0 = time.perf_counter()
+ result = state["pipe"](str(video), **kwargs)
+ t1 = time.perf_counter()
+ if last["name"]:
+ phases[last["name"]] = phases.get(last["name"], 0) + t1 - last["t"]
+ log("phases: " + " ".join(f"{k} {v:.2f}s" for k, v in phases.items()))
+ # RTMPose can swap left/right per frame when the subject turns; fix the 2D cache and redo the
+ # (cheap) 3D lift once if anything changed — same trick Motion2MixamoRig uses.
+ pose_cache = out_dir / "preprocess" / "vitpose.pt"
+ if pose_cache.exists() and _stabilize_pose_cache(pose_cache):
+ (out_dir / "hmr4d_results.pt").unlink(missing_ok=True)
+ result = state["pipe"](str(video), **kwargs)
+ return result, {"recover": t1 - t0, "stabilize": time.perf_counter() - t1}
+
+
+_BODY: dict = {}
+
+
+def leg_length(params) -> float:
+ """Mean hip→ankle length (m) of the recovered body; models cached across calls."""
+ from einops import einsum
+ from gvhmr import PROJ_ROOT
+ from gvhmr.utils.smplx_utils import make_smplx
+
+ if not _BODY:
+ dev = torch.device("cuda")
+ _BODY["smplx"] = make_smplx("supermotion").to(dev)
+ _BODY["x2s"] = torch.load(PROJ_ROOT / "gvhmr/utils/body_model/smplx2smpl_sparse.pt", weights_only=False).to(dev)
+ _BODY["jreg"] = torch.load(PROJ_ROOT / "gvhmr/utils/body_model/smpl_neutral_J_regressor.pt", weights_only=False).to(dev)
+ p = {k: v[:1].cuda() for k, v in params.items() if hasattr(v, "cuda")}
+ with torch.no_grad():
+ verts = _BODY["smplx"](**p).vertices
+ verts = torch.stack([torch.matmul(_BODY["x2s"], v) for v in verts])
+ j = einsum(_BODY["jreg"], verts, "j v, l v i -> l j i")[0].cpu().numpy()
+ seg = lambda a, b: float(np.linalg.norm(j[a] - j[b])) # noqa: E731
+ return (seg(1, 4) + seg(4, 7) + seg(2, 5) + seg(5, 8)) / 2
+
+
+class ProcessIn(BaseModel):
+ video_url: str
+
+
+def health():
+ return {"ok": state["pipe"] is not None, "device": state["device"], "error": state["error"]}
+
+
[email protected]("/process")
+def process(body: ProcessIn):
+ if state["pipe"] is None:
+ GPU.submit(load_pipeline).result()
+ if state["pipe"] is None:
+ raise HTTPException(503, state["error"] or "GVHMR not loaded")
+
+ t0 = time.perf_counter()
+ key = hashlib.sha1(body.video_url.encode()).hexdigest()[:12]
+ out_dir = CACHE / key
+ out_dir.mkdir(parents=True, exist_ok=True)
+ video = out_dir / "input.mp4"
+ if not video.exists():
+ src = out_dir / "source.mp4"
+ with urllib.request.urlopen(body.video_url, timeout=30) as r, open(src, "wb") as f:
+ shutil.copyfileobj(r, f)
+ # GVHMR runs at 30fps and would re-encode a 24fps clip with a slow preset; do it fast here.
+ subprocess.run(["ffmpeg", "-y", "-loglevel", "error", "-i", str(src), "-vf", "fps=30", "-an",
+ "-c:v", "libx264", "-preset", "ultrafast", "-crf", "16", str(video)], check=True)
+ t_dl = time.perf_counter() - t0
+ if not (CACHE / "_warmup.mp4").exists(): # reused to warm CUDA kernels on the next start
+ shutil.copyfile(video, CACHE / "_warmup.mp4")
+
+ to_np = lambda x: (x.detach().cpu().numpy() if hasattr(x, "detach") else np.asarray(x)).astype(np.float32) # noqa: E731
+ def run():
+ result, t = recover(video, out_dir)
+ world = result.smpl_params_world
+ t_ll = time.perf_counter()
+ ll = leg_length(world)
+ t["leg_length"] = time.perf_counter() - t_ll
+ return result, t, world, ll, to_np(result.smpl_params_camera["global_orient"])[0]
+
+ result, t, world, ll, incam_go0 = GPU.submit(run).result()
+
+ go, bp, tr = to_np(world["global_orient"]), to_np(world["body_pose"]), to_np(world["transl"])
+ n = int(go.shape[0])
+
+ from motion2mixamorig.extract import playback_fps
+
+ fps = playback_fps(video, n, fallback=30.0)
+ total = time.perf_counter() - t0
+ log(f"{key}: {n} frames @ {fps:.1f}fps download {t_dl:.2f}s gvhmr {t['recover']:.2f}s "
+ f"stabilize {t['stabilize']:.2f}s legs {t['leg_length']:.2f}s total {total:.2f}s")
+ return {
+ "n": n, "fps": fps, "leg_length": ll,
+ "global_orient": go.reshape(-1).round(5).tolist(),
+ "body_pose": bp.reshape(n, -1)[:, :63].reshape(-1).round(5).tolist(),
+ "transl": tr.reshape(-1).round(5).tolist(),
+ # Frame-0 orientation relative to the camera (OpenCV axes): lets the client face the character
+ # the way the person faces the camera, so chained clips keep a consistent heading.
+ "incam_global_orient0": incam_go0.round(5).tolist(),
+ "elapsed": total, "timings": {"download": t_dl, **t},
+ }
+
+
+if __name__ == "__main__":
+ import uvicorn
+
+ CACHE.mkdir(exist_ok=True)
+ GPU.submit(load_pipeline)
+ uvicorn.run(app, host="127.0.0.1", port=int(os.environ.get("HQ_PORT", 5178)), log_level="warning")
diff --git a/package.json b/package.json
new file mode 100644
index 0000000..b5e26fe
--- /dev/null
+++ b/package.json
@@ -0,0 +1,10 @@
+{
+ "name": "prompt2rig",
+ "version": "0.1.0",
+ "private": true,
+ "type": "module",
+ "description": "Prompt → fast video model (fal H3 Max Turbo) → pose estimation → live retargeting onto a Mixamo-rigged character",
+ "scripts": {
+ "start": "node server.mjs"
+ }
+}
diff --git a/public/app.js b/public/app.js
new file mode 100644
index 0000000..ee1c8c4
--- /dev/null
+++ b/public/app.js
@@ -0,0 +1,610 @@
+// Prompt2Rig — prompt → H3 Max Turbo video → live pose (MediaPipe) → Mixamo rig,
+// with an optional GVHMR "HQ" pass on the local GPU that swaps in when ready.
+import * as THREE from 'three';
+import { OrbitControls } from 'three/addons/controls/OrbitControls.js';
+import { GLTFLoader } from 'three/addons/loaders/GLTFLoader.js';
+import { FBXLoader } from 'three/addons/loaders/FBXLoader.js';
+import { GLTFExporter } from 'three/addons/exporters/GLTFExporter.js';
+import { FilesetResolver, PoseLandmarker, DrawingUtils } from '@mediapipe/tasks-vision';
+import { Rig, OneEuro, Track, trackToClip, blendSnapshots, smplGlobals, aaToQuat } from './retarget.js';
+
+const $ = (s) => document.querySelector(s);
+const XBOT_URL = 'https://cdn.jsdelivr.net/gh/mrdoob/three.js@r169/examples/models/gltf/Xbot.glb';
+const MP_WASM = 'https://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/wasm';
+const MP_MODEL = (v) => `https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_${v}/float16/latest/pose_landmarker_${v}.task`;
+
+const PERFORMER_PROMPT =
+ 'Full-body photograph of one young adult standing in a relaxed neutral pose facing the camera, arms relaxed at the sides, ' +
+ 'wearing a fitted plain light-grey t-shirt, black slim pants and white sneakers. The whole body is visible from head to feet ' +
+ 'with generous empty space around, the person is centered and occupies about half of the frame height. Plain seamless ' +
+ 'light-grey studio backdrop and floor, flat even soft lighting, sharp focus, 35mm photo.';
+
+const PRESETS = [
+ 'does an energetic hip hop dance with big arm movements',
+ 'throws a fast boxing combo of jabs, hooks and an uppercut',
+ 'does jumping jacks',
+ 'walks in a small circle and waves at the camera',
+ 'does a shuffling zombie walk with arms stretched forward',
+ 'celebrates a victory, jumping and pumping both fists',
+ 'does a slow tai chi sequence',
+ 'does a high spinning kick',
+];
+
+const IDLE_PROMPT = 'stands in place in a relaxed idle pose, breathing and gently shifting weight from foot to foot';
+const MAX_IDLE_STREAK = 3;
+// Stream mode = continue-from-last-frame + play only once HQ is ready + idle between prompts.
+const chainOn = () => settings.chain || settings.stream;
+const waitOn = () => settings.hq && hqOnline && (settings.waitHq || settings.stream);
+
+const videoPrompt = (action, fromImage) => fromImage
+ ? `The person ${action}. They perform the full motion while their entire body, including both feet, stays visible in frame the whole time. Static locked-off camera, no camera movement, no zoom, no cuts. Plain studio background.`
+ : `Wide full-body shot of one person wearing a plain grey t-shirt and dark pants in an empty light-grey studio. The person ${action}. Entire body including feet visible at all times. Static locked-off camera, no camera movement, no cuts.`;
+
+// ---- settings ------------------------------------------------------------------
+const DEFAULTS = {
+ stream: false, source: 'performer', chain: false, duration: 5, resolution: '480P', hq: true, waitHq: false,
+ rootMotion: true, poseModel: 'heavy', smoothing: 0.5, showSkel: true, performerUrl: null,
+};
+const settings = (() => { try { return { ...DEFAULTS, ...JSON.parse(localStorage.getItem('p2r') || '{}') }; } catch { return { ...DEFAULTS }; } })();
+const save = () => { try { localStorage.setItem('p2r', JSON.stringify(settings)); } catch {} };
+
+function bindSetting(id, kind = 'value', onChange) {
+ const el = $('#' + id);
+ if (kind === 'checked') el.checked = !!settings[id]; else el.value = settings[id];
+ el.addEventListener('input', () => {
+ settings[id] = kind === 'checked' ? el.checked : (el.type === 'range' ? Number(el.value) : el.value);
+ save(); onChange?.(settings[id]);
+ });
+ onChange?.(settings[id]);
+}
+
+// ---- log / status -------------------------------------------------------------
+function log(msg, cls = '') {
+ const el = $('#log'); const line = document.createElement('div');
+ if (cls) line.className = cls;
+ line.textContent = `${new Date().toLocaleTimeString([], { hour12: false })} ${msg}`;
+ el.appendChild(line); el.scrollTop = el.scrollHeight;
+}
+function pill(id, text, state) { const p = $('#pill-' + id); p.textContent = text; p.className = 'pill ' + (state || ''); }
+function banner(text) { const b = $('#banner'); b.textContent = text || ''; b.classList.toggle('on', !!text); }
+
+async function api(path, body) {
+ const res = await fetch(path, { method: body ? 'POST' : 'GET', headers: { 'Content-Type': 'application/json' }, body: body && JSON.stringify(body) });
+ const json = await res.json().catch(() => ({}));
+ if (!res.ok) throw new Error(json.error || `${path} → ${res.status}`);
+ return json;
+}
+
+// ---- three.js stage -------------------------------------------------------------
+const stage = $('#stage');
+const renderer = new THREE.WebGLRenderer({ antialias: true });
+renderer.setPixelRatio(Math.min(2, devicePixelRatio));
+renderer.shadowMap.enabled = true;
+renderer.shadowMap.type = THREE.PCFSoftShadowMap;
+renderer.outputColorSpace = THREE.SRGBColorSpace;
+stage.appendChild(renderer.domElement);
+
+const scene = new THREE.Scene();
+scene.background = new THREE.Color(0x15171a);
+scene.fog = new THREE.Fog(0x15171a, 8, 22);
+const camera = new THREE.PerspectiveCamera(35, 1, 0.05, 100);
+camera.position.set(0.6, 1.5, 5.2);
+const controls = new OrbitControls(camera, renderer.domElement);
+controls.target.set(0, 0.95, 0);
+controls.enableDamping = true;
+
+scene.add(new THREE.HemisphereLight(0xdfe6ff, 0x1a1c20, 1.6));
+const sun = new THREE.DirectionalLight(0xffffff, 2.2);
+sun.position.set(2.5, 5, 3.5); sun.castShadow = true;
+sun.shadow.mapSize.set(2048, 2048);
+Object.assign(sun.shadow.camera, { left: -3, right: 3, top: 3, bottom: -3, near: 0.5, far: 15 });
+scene.add(sun);
+const floor = new THREE.Mesh(new THREE.PlaneGeometry(60, 60), new THREE.MeshStandardMaterial({ color: 0x1f2226, roughness: 1 }));
+floor.rotation.x = -Math.PI / 2; floor.receiveShadow = true; scene.add(floor);
+const grid = new THREE.GridHelper(60, 120, 0x3a3f46, 0x2a2e33); grid.position.y = 0.002; scene.add(grid);
+
+function resize() {
+ const { clientWidth: w, clientHeight: h } = stage;
+ renderer.setSize(w, h); camera.aspect = w / h; camera.updateProjectionMatrix();
+}
+new ResizeObserver(resize).observe(stage);
+
+// ---- character --------------------------------------------------------------------
+let model = null, rig = null;
+
+async function loadCharacter(src, name = 'X Bot') {
+ pill('rig', 'loading ' + name, 'busy');
+ const isFbx = /\.fbx$/i.test(name);
+ const loader = isFbx ? new FBXLoader() : new GLTFLoader();
+ const obj = await loader.loadAsync(src);
+ const root = isFbx ? obj : obj.scene;
+ if (isFbx) root.scale.setScalar(0.01); // Mixamo FBX is in centimeters
+ root.updateMatrixWorld(true);
+ const box = new THREE.Box3().setFromObject(root);
+ root.position.y -= box.min.y;
+ root.position.x -= (box.min.x + box.max.x) / 2;
+ root.position.z -= (box.min.z + box.max.z) / 2;
+ root.name = 'Character';
+ root.traverse((o) => { if (o.isMesh) { o.castShadow = true; o.frustumCulled = false; } });
+ root.updateMatrixWorld(true);
+ const newRig = new Rig(root);
+ if (model) scene.remove(model);
+ scene.add(root); model = root; rig = newRig;
+ // Takes store bone-order-specific data; re-bake HQ data for the new skeleton.
+ for (const t of takes) { t.live = null; t.liveDone = false; if (t.hqRaw) t.hq = bakeSmpl(t.hqRaw); }
+ pill('rig', name, 'ok');
+ log(`character: ${name} (${rig.order.length} bones, ${rig.height.toFixed(2)} m)`);
+}
+
+// ---- MediaPipe ---------------------------------------------------------------------
+let landmarker = null, vision = null, mpTs = 0;
+async function initPose(variant) {
+ pill('pose', 'pose: loading ' + variant, 'busy');
+ vision ||= await FilesetResolver.forVisionTasks(MP_WASM);
+ const make = (delegate) => PoseLandmarker.createFromOptions(vision, {
+ baseOptions: { modelAssetPath: MP_MODEL(variant), delegate },
+ runningMode: 'VIDEO', numPoses: 1,
+ minPoseDetectionConfidence: 0.5, minPosePresenceConfidence: 0.5, minTrackingConfidence: 0.5,
+ });
+ const old = landmarker; landmarker = null;
+ try { landmarker = await make('GPU'); } catch (e) { log('GPU delegate failed, using CPU: ' + e.message, 'err'); landmarker = await make('CPU'); }
+ old?.close();
+ pill('pose', 'pose: ' + variant, 'ok');
+}
+
+// ---- takes --------------------------------------------------------------------------
+let takeSeq = 0;
+const takes = [];
+class Take {
+ constructor(prompt) {
+ this.id = ++takeSeq; this.prompt = prompt;
+ this.videoUrl = null; this.status = 'generating';
+ this.live = null; this.liveDone = false;
+ this.hq = null; this.hqRaw = null; this.hqStatus = 'off'; this.hqAt = 0;
+ this.rootOffset = null; this.timings = {};
+ this.ready = new Promise((r) => (this._resolveReady = r));
+ }
+ get best() { return this.hq || (this.liveDone ? this.live : null); }
+}
+
+function renderTakes() {
+ const el = $('#takes'); el.innerHTML = '';
+ for (const t of [...takes].reverse()) {
+ const card = document.createElement('div');
+ card.className = 'take' + (t === current ? ' playing' : '');
+ const gen = t.status === 'generating' ? '<span class="pill busy"><span class="spin"></span> generating</span>'
+ : t.status === 'error' ? '<span class="pill bad">failed</span>'
+ : t.status === 'skipped' ? '<span class="pill">skipped</span>'
+ : `<span class="pill">${t.timings.gen?.toFixed(1)}s gen</span>`;
+ const live = t.liveDone ? '<span class="pill ok">live ✓</span>' : '';
+ const hq = { running: '<span class="pill busy"><span class="spin"></span> HQ</span>', done: `<span class="pill ok">HQ ✓ ${t.timings.hq?.toFixed(1)}s</span>`, error: '<span class="pill bad">HQ ✗</span>' }[t.hqStatus] || '';
+ card.innerHTML = `<div class="p" title="${escapeHtml(t.prompt)}">#${t.id} · ${t.auto ? '<i style="color:var(--dim)">idle</i> · ' : ''}${escapeHtml(t.prompt)}</div>
+ <div class="meta">${gen}${live}${hq}</div>
+ <div class="actions"><button class="btn" data-a="play" ${t.videoUrl ? '' : 'disabled'}>▶ Play</button>
+ <button class="btn" data-a="glb" ${t.best ? '' : 'disabled'}>⬇ GLB</button></div>`;
+ card.querySelector('[data-a=play]').onclick = () => playTake(t);
+ card.querySelector('[data-a=glb]').onclick = () => exportTake(t);
+ el.appendChild(card);
+ }
+}
+const escapeHtml = (s) => s.replace(/[&<>"]/g, (c) => ({ '&': '&amp;', '<': '&lt;', '>': '&gt;', '"': '&quot;' }[c]));
+
+// ---- generation ------------------------------------------------------------------------
+let genChain = Promise.resolve();
+let lastGenerated = null;
+
+let idleStreak = 0;
+function submitPrompt(prompt, { auto = false } = {}) {
+ prompt = prompt.trim();
+ if (!prompt) return;
+ if (!auto) {
+ idleStreak = 0;
+ // A real prompt shouldn't wait behind an idle filler that hasn't started playing: drop it,
+ // and chain from the clip before it instead.
+ const idle = lastGenerated;
+ if (settings.stream && idle?.auto && !idle.started && idle !== current && idle.status === 'ready' && !genBusy) {
+ const qi = queue.indexOf(idle); if (qi >= 0) queue.splice(qi, 1);
+ idle.status = 'skipped'; lastGenerated = idle.prevGenerated;
+ log(`#${idle.id} idle filler skipped for your prompt`);
+ }
+ }
+ const take = new Take(prompt);
+ take.auto = auto;
+ takes.push(take); renderTakes();
+ // Serialized: "continue from last frame" needs the previous clip's final frame.
+ genBusy++;
+ genChain = genChain.then(() => generate(take)).catch(() => {}).finally(() => genBusy--);
+}
+let genBusy = 0;
+
+/** Stream mode: keep one clip in the pipeline so the performer never freezes for long. */
+function maybeIdle() {
+ if (!settings.stream || idleStreak >= MAX_IDLE_STREAK) return;
+ const ahead = queue.length + genBusy + takes.filter((t) => t.status === 'ready' && t.hqStatus === 'running' && !t.started).length;
+ if (ahead === 0) { idleStreak++; submitPrompt(IDLE_PROMPT, { auto: true }); }
+}
+
+async function ensurePerformer(force = false) {
+ if (settings.performerUrl && !force) return settings.performerUrl;
+ banner('Creating performer…');
+ const t0 = performance.now();
+ const r = await api('/api/image', { prompt: PERFORMER_PROMPT });
+ settings.performerUrl = r.image_url; save();
+ $('#performer-img').src = r.image_url;
+ log(`performer image in ${((performance.now() - t0) / 1000).toFixed(1)}s`, 'ok');
+ banner('');
+ return r.image_url;
+}
+
+async function generate(take) {
+ const t0 = performance.now();
+ try {
+ let image_url = null, from = 'text';
+ if (chainOn() && lastGenerated?.videoUrl) {
+ image_url = await (lastGenerated.lastFrame ||= lastFrameDataUrl(lastGenerated.videoUrl)); from = `take #${lastGenerated.id} last frame`;
+ } else if (settings.source === 'performer') {
+ image_url = await ensurePerformer(); from = 'performer';
+ }
+ banner(`Generating #${take.id}…`);
+ log(`#${take.id} → H3 Max Turbo (${from}, ${settings.duration}s ${settings.resolution})`);
+ const r = await api('/api/video', {
+ prompt: videoPrompt(take.prompt, !!image_url), image_url,
+ duration: settings.duration, resolution: settings.resolution, aspect_ratio: '16:9',
+ });
+ take.videoUrl = r.video_url; take.status = 'ready';
+ take.timings.gen = (performance.now() - t0) / 1000;
+ log(`#${take.id} video in ${take.timings.gen.toFixed(1)}s (GPU ${r.timings?.inference?.toFixed(2) ?? '?'}s)`, 'ok');
+ take.prevGenerated = lastGenerated;
+ lastGenerated = take;
+ // Grab the final frame now (in parallel with HQ) so the next chained clip can start immediately.
+ if (chainOn()) take.lastFrame = lastFrameDataUrl(take.videoUrl).catch(() => null);
+ banner('');
+ if (settings.hq && hqOnline) runHQ(take);
+ renderTakes();
+ if (waitOn() && !(settings.stream && held)) {
+ // Normally play only once HQ is ready; if the player starves meanwhile, startStarved()
+ // plays it early with the live preview and HQ cross-fades in when it lands.
+ take.awaitingHq = true;
+ if (!current) banner(`#${take.id}: waiting for HQ motion…`);
+ take.hqPromise.finally(() => {
+ banner('');
+ if (take.awaitingHq && take.status !== 'skipped') { take.awaitingHq = false; enqueue(take); }
+ });
+ } else enqueue(take);
+ } catch (e) {
+ take.status = 'error'; renderTakes(); banner('');
+ log(`#${take.id} failed: ${e.message}`, 'err');
+ }
+}
+
+/** Grab a clip's final frame as a JPEG data URI (first frame for the next clip). */
+async function lastFrameDataUrl(url) {
+ const v = document.createElement('video');
+ v.crossOrigin = 'anonymous'; v.muted = true; v.preload = 'auto'; v.src = url;
+ await new Promise((res, rej) => { v.onloadeddata = res; v.onerror = () => rej(new Error('cannot load video for last frame')); });
+ v.currentTime = Math.max(0, v.duration - 0.05);
+ await new Promise((res) => (v.onseeked = res));
+ const c = document.createElement('canvas'); c.width = v.videoWidth; c.height = v.videoHeight;
+ c.getContext('2d').drawImage(v, 0, 0);
+ v.removeAttribute('src'); v.load();
+ return c.toDataURL('image/jpeg', 0.92);
+}
+
+// ---- HQ (GVHMR) --------------------------------------------------------------------------
+let hqOnline = false;
+async function checkHQ() {
+ try { const r = await api('/api/hq/health'); hqOnline = !!r.ok; pill('hq', r.ok ? `HQ: ${r.device || 'ready'}` : 'HQ: ' + (r.error || 'not ready'), r.ok ? 'ok' : 'bad'); }
+ catch { hqOnline = false; pill('hq', 'HQ worker offline', 'bad'); }
+}
+
+function runHQ(take) {
+ const t0 = performance.now();
+ take.hqStatus = 'running'; renderTakes();
+ take.hqPromise = api('/api/hq', { video_url: take.videoUrl })
+ .then((r) => {
+ take.hqRaw = r; take.hq = bakeSmpl(r); take.hqAt = performance.now();
+ take.hqStatus = 'done'; take.timings.hq = (performance.now() - t0) / 1000;
+ log(`#${take.id} HQ motion in ${take.timings.hq.toFixed(1)}s (${r.n} frames, worker ${r.elapsed?.toFixed(1)}s)`, 'hq');
+ })
+ .catch((e) => { take.hqStatus = 'error'; log(`#${take.id} HQ failed: ${e.message}`, 'err'); })
+ .finally(renderTakes);
+}
+
+/** GVHMR SMPL params → Track of Mixamo local rotations + grounded hips path. */
+function bakeSmpl(r) {
+ const { n, fps, global_orient: GO, body_pose: BP, transl: T, leg_length } = r;
+ // Heading: turn GVHMR's world so the character faces the viewer the way the person faces the
+ // camera (from the in-camera orientation). Chained clips share the camera, so headings line up.
+ const fWorld = new THREE.Vector3(0, 0, 1).applyQuaternion(aaToQuat(GO[0], GO[1], GO[2]));
+ let targetYaw = 0;
+ if (r.incam_global_orient0) {
+ const [a, b, c] = r.incam_global_orient0;
+ const fc = new THREE.Vector3(0, 0, 1).applyQuaternion(aaToQuat(a, b, c)); // OpenCV camera axes
+ targetYaw = Math.atan2(fc.x, -fc.z); // → three.js (x, -y, -z)
+ }
+ const yawFix = new THREE.Quaternion().setFromAxisAngle(new THREE.Vector3(0, 1, 0), targetYaw - Math.atan2(fWorld.x, fWorld.z));
+ const scale = rig.legLength / (leg_length || 0.85);
+ const track = new Track(rig.order.length);
+ const lows = [];
+ const d = new THREE.Vector3();
+ for (let f = 0; f < n; f++) {
+ rig.reset();
+ rig.applySmplGlobal(smplGlobals(GO.subarray ? GO.subarray(f * 3, f * 3 + 3) : GO.slice(f * 3, f * 3 + 3), BP.slice(f * 63, f * 63 + 63), yawFix));
+ d.set(T[f * 3] - T[0], T[f * 3 + 1] - T[1], T[f * 3 + 2] - T[2]).applyQuaternion(yawFix).multiplyScalar(scale);
+ const p = rig.rest.Hips.wp.clone().add(d);
+ rig.setHipsWorld(p);
+ lows.push(rig.lowestFoot());
+ track.push(f / fps, rig.snapshot());
+ }
+ // Ground: the typical lowest foot sits on the floor; never sink below it.
+ const sorted = [...lows].sort((a, b) => a - b);
+ const offset = rig.floorY - sorted[Math.floor(sorted.length * 0.08)];
+ track.p.forEach((p, f) => { p[1] += offset + Math.max(0, rig.floorY - (lows[f] + offset)); });
+ rig.reset();
+ return track;
+}
+
+// ---- playback -----------------------------------------------------------------------------
+const video = $('#video');
+const overlay = $('#overlay');
+const octx = overlay.getContext('2d');
+let current = null;
+const queue = [];
+let liveSnap = null;
+let liveState = null;
+const worldFilter = new OneEuro(99);
+const hipFilter = new OneEuro(2);
+let lastRenderedHips = new THREE.Vector3();
+let switchFrom = null, switchAt = 0;
+
+function setSmoothing(s) {
+ // s=0 → raw, s=1 → very smooth
+ worldFilter.minCutoff = hipFilter.minCutoff = THREE.MathUtils.lerp(6, 0.4, s);
+ worldFilter.beta = hipFilter.beta = THREE.MathUtils.lerp(1.2, 0.15, s);
+}
+
+let held = false; // stream mode: clip ended, nothing ready yet → hold the last pose
+function enqueue(take) {
+ queue.push(take);
+ // Switch right away unless the current clip is still on its first (live) pass,
+ // or we're chaining (then the next clip continues from this one's last frame).
+ if (!current || held || (current.liveDone && !chainOn())) playNext();
+}
+
+/** Out of HQ-ready clips: start the next generated one now on the live pass instead of freezing. */
+function startStarved() {
+ const next = takes.find((t) => t.awaitingHq && t.status === 'ready' && !t.started);
+ if (!next) return false;
+ next.awaitingHq = false;
+ log(`#${next.id} starting on live preview (HQ still running)`);
+ enqueue(next);
+ return true;
+}
+
+function playNext() { const t = queue.shift(); if (t) playTake(t); }
+
+function playTake(take) {
+ if (!take.videoUrl) return;
+ if (current && rig) { switchFrom = rig.snapshot(); switchAt = performance.now(); }
+ held = false; take.started = true;
+ if (take.hq) take.hqAt = -1e9; // already HQ: no live→HQ cross-fade
+ const prev = current;
+ current = take;
+ const qi = queue.indexOf(take); if (qi >= 0) queue.splice(qi, 1);
+ if (!take.rootOffset) {
+ take.rootOffset = new THREE.Vector3();
+ if (chainOn() && prev && rig) {
+ take.rootOffset.set(lastRenderedHips.x - rig.rest.Hips.wp.x, 0, lastRenderedHips.z - rig.rest.Hips.wp.z);
+ take.rootOffset.clampLength(0, 2.5);
+ }
+ }
+ if (!take.liveDone) {
+ take.live = new Track(rig.order.length);
+ liveState = { hip0: null, baseY: null, mpp: null, scale: null };
+ worldFilter.reset(); hipFilter.reset(); liveSnap = null;
+ }
+ $('#vid-empty').style.display = 'none';
+ video.style.display = 'block';
+ video.src = take.videoUrl;
+ lastDetectT = -1;
+ video.play().catch(() => {});
+ renderTakes();
+ maybeIdle();
+}
+
+video.addEventListener('ended', () => {
+ if (current && !current.liveDone) {
+ current.liveDone = current.live.length > 3;
+ log(`#${current.id} live pass: ${current.live.length} frames`);
+ renderTakes();
+ }
+ if (queue.length) playNext();
+ else if (settings.stream) { held = true; if (!startStarved()) maybeIdle(); } // freeze on the last frame
+ else { video.currentTime = 0; video.play().catch(() => {}); }
+});
+
+// Per decoded video frame: run pose, draw overlay, record the live track.
+const stats = { frames: 0, detections: 0, world: 0, recorded: 0 };
+let lastDetectT = -1;
+// Called from the render loop: run pose once per new video frame (requestVideoFrameCallback
+// stops firing after the source changes in Chromium, so we poll currentTime instead).
+function detectIfNewFrame() {
+ if (!landmarker || !rig || !current || video.readyState < 2 || video.seeking) return;
+ const mediaTime = video.currentTime;
+ if (mediaTime === lastDetectT) return;
+ if (mediaTime < lastDetectT) liveLoopReset();
+ lastDetectT = mediaTime;
+ try {
+ stats.frames++;
+ const ts = Math.max(mpTs + 1, performance.now()); mpTs = ts;
+ const res = landmarker.detectForVideo(video, ts);
+ stats.detections += res.landmarks?.length ? 1 : 0;
+ stats.world += res.worldLandmarks?.length ? 1 : 0;
+ drawOverlay(res);
+ if (!current.liveDone && res.worldLandmarks?.[0]) {
+ liveSnap = solveLive(res, mediaTime);
+ current.live.push(mediaTime, liveSnap); stats.recorded++;
+ }
+ } catch (e) { console.error(e); }
+}
+function liveLoopReset() { /* video looped or switched: keep filters from smearing across the jump */ worldFilter.reset(); hipFilter.reset(); }
+
+function drawOverlay(res) {
+ overlay.width = video.videoWidth; overlay.height = video.videoHeight;
+ octx.clearRect(0, 0, overlay.width, overlay.height);
+ if (!settings.showSkel || !res.landmarks?.[0]) return;
+ const du = new DrawingUtils(octx);
+ du.drawConnectors(res.landmarks[0], PoseLandmarker.POSE_CONNECTIONS, { color: '#7cf29a', lineWidth: 3 });
+ du.drawLandmarks(res.landmarks[0], { color: '#5ab8ff', fillColor: '#0e0f11', lineWidth: 2, radius: 3 });
+}
+
+const _lm = Array.from({ length: 33 }, () => new THREE.Vector3());
+function solveLive(res, t) {
+ const w = res.worldLandmarks[0], n2 = res.landmarks[0];
+ const raw = new Float32Array(99);
+ // MediaPipe world: x = image right (subject's left when facing camera), y down, z away from camera.
+ w.forEach((p, i) => { raw[i * 3] = p.x; raw[i * 3 + 1] = -p.y; raw[i * 3 + 2] = -p.z; });
+ const f = worldFilter.filter(raw, t);
+ for (let i = 0; i < 33; i++) _lm[i].set(f[i * 3], f[i * 3 + 1], f[i * 3 + 2]);
+ const vis = w.map((p) => p.visibility ?? 1);
+
+ // Root motion from the 2D hip track, converted to meters via torso length.
+ const W = video.videoWidth, H = video.videoHeight;
+ const px = (i) => new THREE.Vector2(n2[i].x * W, n2[i].y * H);
+ const hipPx = px(23).add(px(24)).multiplyScalar(0.5);
+ const shPx = px(11).add(px(12)).multiplyScalar(0.5);
+ const hf = hipFilter.filter([hipPx.x, hipPx.y], t);
+ const midHip = _lm[23].clone().add(_lm[24]).multiplyScalar(0.5);
+ const midSh = _lm[11].clone().add(_lm[12]).multiplyScalar(0.5);
+ const mpp = midSh.distanceTo(midHip) / Math.max(1, shPx.distanceTo(hipPx));
+ const leg = (_lm[23].distanceTo(_lm[25]) + _lm[25].distanceTo(_lm[27]) + _lm[24].distanceTo(_lm[26]) + _lm[26].distanceTo(_lm[28])) / 2;
+ const S = liveState;
+ S.mpp = S.mpp == null ? mpp : S.mpp + 0.05 * (mpp - S.mpp);
+ S.scale = S.scale == null ? rig.legLength / leg : S.scale + 0.05 * (rig.legLength / leg - S.scale);
+ if (!S.hip0) S.hip0 = [hf[0], hf[1]];
+
+ rig.reset();
+ rig.applyLandmarks(_lm, vis);
+ const p = rig.rest.Hips.wp.clone();
+ if (settings.rootMotion) {
+ p.x += (hf[0] - S.hip0[0]) * S.mpp * S.scale;
+ p.y += -(hf[1] - S.hip0[1]) * S.mpp * S.scale;
+ }
+ rig.setHipsWorld(p);
+ if (S.baseY == null) S.baseY = rig.floorY - rig.lowestFoot();
+ p.y += S.baseY; rig.setHipsWorld(p);
+ rig.clampToFloor();
+ return rig.snapshot();
+}
+
+// ---- render loop ----------------------------------------------------------------------------
+const bufA = { q: null, p: new Float32Array(3) }, bufB = { q: null, p: new Float32Array(3) }, bufC = { q: null, p: new Float32Array(3) };
+const camTarget = new THREE.Vector3(0, 0.95, 0);
+
+let lastRaf = 0;
+function frame() {
+ requestAnimationFrame(frame);
+ lastRaf = performance.now();
+ tick();
+}
+// Browsers pause rAF in hidden/occluded views; keep pose + playback running from a timer then.
+setInterval(() => { if (performance.now() - lastRaf > 100) tick(); }, 33);
+
+function tick() {
+ detectIfNewFrame();
+ if (rig && current) {
+ const n = rig.order.length * 4;
+ for (const b of [bufA, bufB, bufC]) if (!b.q || b.q.length !== n) b.q = new Float32Array(n);
+ const t = video.currentTime;
+ const live = current.liveDone ? current.live.sample(t, bufA) : liveSnap;
+ const hq = current.hq ? current.hq.sample(t, bufB) : null;
+ let snap = hq || live, src = hq ? 'HQ · GVHMR' : live ? 'LIVE · MediaPipe' : '—';
+ if (hq && live) {
+ const w = Math.min(1, (performance.now() - current.hqAt) / 600);
+ if (w < 1) { snap = blendSnapshots(live.q, live.p, hq.q, hq.p, w, bufC); src = 'LIVE → HQ'; }
+ }
+ if (snap && switchFrom) {
+ const w = Math.min(1, (performance.now() - switchAt) / 250);
+ if (w < 1) snap = blendSnapshots(switchFrom.q, switchFrom.p, snap.q, snap.p, w, snap === bufC ? bufA : bufC);
+ else switchFrom = null;
+ }
+ if (snap) {
+ rig.applySnapshot(snap);
+ if (current.rootOffset?.lengthSq()) { const h = rig.hipsWorld().add(current.rootOffset); rig.setHipsWorld(h); }
+ rig.hipsWorld(lastRenderedHips);
+ }
+ const hud = $('#hud-src'); hud.textContent = src; hud.className = 'tag ' + (hq ? 'src-hq' : 'src-live');
+ const buf = settings.stream ? ` · ready ${queue.length} gen ${genBusy} hq ${takes.filter((x) => x.hqStatus === 'running').length}${held ? ' · HOLD' : ''}` : '';
+ $('#hud-time').textContent = `#${current.id} ${t.toFixed(2)}s${buf}`;
+ // Ease the orbit target toward the character.
+ camTarget.set(lastRenderedHips.x, 0.95, lastRenderedHips.z);
+ const delta = camTarget.clone().sub(controls.target).multiplyScalar(0.04);
+ controls.target.add(delta); camera.position.add(delta);
+ }
+ controls.update();
+ renderer.render(scene, camera);
+}
+
+// ---- export --------------------------------------------------------------------------------
+async function exportTake(take) {
+ const track = take.best;
+ if (!track) return;
+ const name = `take${take.id}_${take.prompt.replace(/[^a-z0-9]+/gi, '_').slice(0, 40)}${take.hq ? '_hq' : '_live'}`;
+ const clip = trackToClip(rig, track, name);
+ const glb = await new GLTFExporter().parseAsync(model, { binary: true, animations: [clip] });
+ const a = document.createElement('a');
+ a.href = URL.createObjectURL(new Blob([glb], { type: 'model/gltf-binary' }));
+ a.download = name + '.glb'; a.click();
+ setTimeout(() => URL.revokeObjectURL(a.href), 5000);
+ log(`exported ${name}.glb`, 'ok');
+}
+
+// ---- UI wiring --------------------------------------------------------------------------------
+bindSetting('stream', 'checked', (on) => {
+ for (const id of ['chain', 'waitHq']) { const el = $('#' + id); el.disabled = on; el.closest('label').style.opacity = on ? 0.5 : 1; }
+});
+bindSetting('source'); bindSetting('chain', 'checked'); bindSetting('resolution');
+bindSetting('duration', 'value', (v) => ($('#duration-v').textContent = v + 's'));
+bindSetting('hq', 'checked'); bindSetting('waitHq', 'checked'); bindSetting('rootMotion', 'checked');
+bindSetting('showSkel', 'checked');
+bindSetting('smoothing', 'value', setSmoothing);
+bindSetting('poseModel', 'value', (v) => { if (vision) initPose(v); });
+
+const chips = $('#chips');
+for (const p of PRESETS) {
+ const c = document.createElement('span'); c.className = 'chip'; c.textContent = p;
+ c.onclick = () => submitPrompt(p);
+ chips.appendChild(c);
+}
+$('#go').onclick = () => submitPrompt($('#prompt').value);
+$('#prompt').addEventListener('keydown', (e) => { if (e.key === 'Enter' && (e.ctrlKey || e.metaKey)) { e.preventDefault(); submitPrompt($('#prompt').value); } });
+$('#performer-new').onclick = () => ensurePerformer(true).catch((e) => { banner(''); log('performer failed: ' + e.message, 'err'); });
+if (settings.performerUrl) $('#performer-img').src = settings.performerUrl;
+
+$('#btn-char').onclick = () => $('#file-char').click();
+$('#file-char').onchange = (e) => {
+ const f = e.target.files[0]; if (!f) return;
+ const url = URL.createObjectURL(f);
+ loadCharacter(url, f.name).catch((err) => { log('character failed: ' + err.message, 'err'); pill('rig', 'character failed', 'bad'); });
+};
+$('#btn-vid').onclick = () => $('#file-vid').click();
+$('#file-vid').onchange = (e) => {
+ const f = e.target.files[0]; if (!f) return;
+ const take = new Take(`(file) ${f.name}`);
+ take.videoUrl = URL.createObjectURL(f); take.status = 'ready'; take.timings.gen = 0;
+ takes.push(take); renderTakes(); enqueue(take);
+};
+
+// Debug handle for the console.
+window.p2r = {
+ get current() { return current; }, get rig() { return rig; }, takes, video, stats,
+ playUrl(u, hq = true) { const t = new Take('debug: ' + u.slice(-24)); t.videoUrl = u; t.status = 'ready'; takes.push(t); if (hq && hqOnline) runHQ(t); enqueue(t); return t; },
+};
+
+// ---- boot --------------------------------------------------------------------------------------
+resize();
+frame();
+loadCharacter(XBOT_URL, 'X Bot').catch((e) => { log('character failed: ' + e.message, 'err'); pill('rig', 'character failed', 'bad'); });
+initPose(settings.poseModel).catch((e) => { log('pose model failed: ' + e.message, 'err'); pill('pose', 'pose failed', 'bad'); });
+checkHQ(); setInterval(checkHQ, 10000);
+log('ready — type a motion and hit Generate');
diff --git a/public/index.html b/public/index.html
new file mode 100644
index 0000000..71710c3
--- /dev/null
+++ b/public/index.html
@@ -0,0 +1,158 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8" />
+<meta name="viewport" content="width=device-width, initial-scale=1" />
+<title>Prompt2Rig</title>
+<script type="importmap">
+{
+ "imports": {
+ "three": "https://cdn.jsdelivr.net/npm/[email protected]/build/three.module.js",
+ "three/addons/": "https://cdn.jsdelivr.net/npm/[email protected]/examples/jsm/",
+ "@mediapipe/tasks-vision": "https://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/vision_bundle.mjs"
+ }
+}
+</script>
+<style>
+ :root {
+ --bg: #0e0f11; --panel: #16181b; --panel2: #1d2024; --line: #2a2e33; --text: #e7e9ec; --dim: #8b929b;
+ --accent: #7cf29a; --accent2: #5ab8ff; --warn: #ffb454; --err: #ff6b6b;
+ }
+ * { box-sizing: border-box; }
+ html, body { margin: 0; height: 100%; background: var(--bg); color: var(--text); font: 13px/1.4 ui-sans-serif, system-ui, -apple-system, "Segoe UI", sans-serif; }
+ button, input, select, textarea { font: inherit; color: inherit; }
+ .app { display: grid; grid-template-columns: 340px 1fr; grid-template-rows: 1fr 132px; height: 100vh; }
+ aside { grid-row: 1 / 3; background: var(--panel); border-right: 1px solid var(--line); overflow-y: auto; padding: 14px; display: flex; flex-direction: column; gap: 14px; }
+ h1 { font-size: 15px; margin: 0; letter-spacing: .2px; display: flex; align-items: center; gap: 8px; }
+ h1 .dot { width: 9px; height: 9px; border-radius: 50%; background: var(--accent); box-shadow: 0 0 10px var(--accent); }
+ .pills { display: flex; flex-wrap: wrap; gap: 6px; }
+ .pill { font-size: 11px; padding: 2px 8px; border-radius: 99px; background: var(--panel2); border: 1px solid var(--line); color: var(--dim); }
+ .pill.ok { color: var(--accent); border-color: #2f5b3b; } .pill.bad { color: var(--err); border-color: #5b2f2f; } .pill.busy { color: var(--warn); border-color: #5b4a2f; }
+ textarea { width: 100%; min-height: 74px; resize: vertical; background: var(--panel2); border: 1px solid var(--line); border-radius: 8px; padding: 10px; outline: none; }
+ textarea:focus { border-color: var(--accent2); }
+ .go { width: 100%; padding: 10px; border: 0; border-radius: 8px; background: var(--accent); color: #07140b; font-weight: 650; cursor: pointer; }
+ .go:disabled { opacity: .5; cursor: default; }
+ .chips { display: flex; flex-wrap: wrap; gap: 6px; }
+ .chip { font-size: 11.5px; padding: 4px 9px; border-radius: 99px; background: var(--panel2); border: 1px solid var(--line); cursor: pointer; }
+ .chip:hover { border-color: var(--accent2); }
+ .section { display: flex; flex-direction: column; gap: 8px; }
+ .section > label.title { font-size: 11px; text-transform: uppercase; letter-spacing: .8px; color: var(--dim); }
+ .row { display: flex; align-items: center; justify-content: space-between; gap: 8px; }
+ .row select, .row input[type=number] { background: var(--panel2); border: 1px solid var(--line); border-radius: 6px; padding: 4px 6px; }
+ .row input[type=range] { width: 130px; }
+ .check { display: flex; gap: 8px; align-items: flex-start; cursor: pointer; }
+ .check small { display: block; color: var(--dim); }
+ .performer { display: flex; gap: 10px; align-items: center; }
+ .performer img { width: 120px; aspect-ratio: 16/9; object-fit: cover; border-radius: 6px; border: 1px solid var(--line); background: var(--panel2); }
+ .btn { padding: 5px 10px; border-radius: 6px; border: 1px solid var(--line); background: var(--panel2); cursor: pointer; }
+ .btn:hover { border-color: var(--accent2); }
+ .log { font: 11.5px/1.45 ui-monospace, Consolas, monospace; color: var(--dim); background: #0b0c0e; border: 1px solid var(--line); border-radius: 8px; padding: 8px; max-height: 160px; overflow-y: auto; white-space: pre-wrap; }
+ .log .ok { color: var(--accent); } .log .err { color: var(--err); } .log .hq { color: var(--accent2); }
+
+ main { position: relative; overflow: hidden; }
+ #stage { position: absolute; inset: 0; }
+ #stage canvas { display: block; }
+ .vidwrap { position: absolute; top: 14px; left: 14px; width: min(38%, 520px); border-radius: 10px; overflow: hidden; border: 1px solid var(--line); background: #000; box-shadow: 0 10px 30px rgba(0,0,0,.45); }
+ .vidwrap video, .vidwrap canvas { display: block; width: 100%; }
+ .vidwrap canvas { position: absolute; inset: 0; height: 100%; pointer-events: none; }
+ .vidwrap .empty { aspect-ratio: 16/9; display: grid; place-items: center; color: var(--dim); }
+ .hud { position: absolute; right: 14px; top: 14px; display: flex; flex-direction: column; gap: 6px; align-items: flex-end; pointer-events: none; }
+ .hud .tag { font: 12px ui-monospace, Consolas, monospace; background: rgba(14,15,17,.75); border: 1px solid var(--line); border-radius: 6px; padding: 4px 8px; }
+ .hud .src-live { color: var(--warn); } .hud .src-hq { color: var(--accent); }
+ .banner { position: absolute; left: 50%; bottom: 18px; transform: translateX(-50%); background: rgba(14,15,17,.8); border: 1px solid var(--line); border-radius: 99px; padding: 6px 14px; font-size: 13px; display: none; }
+ .banner.on { display: block; }
+
+ footer { background: var(--panel); border-top: 1px solid var(--line); overflow-x: auto; overflow-y: hidden; display: flex; gap: 10px; padding: 10px 12px; }
+ .take { flex: 0 0 230px; background: var(--panel2); border: 1px solid var(--line); border-radius: 8px; padding: 8px; display: flex; flex-direction: column; gap: 6px; }
+ .take.playing { border-color: var(--accent); }
+ .take .p { font-size: 12px; overflow: hidden; display: -webkit-box; -webkit-line-clamp: 2; -webkit-box-orient: vertical; min-height: 33px; }
+ .take .meta { display: flex; gap: 5px; flex-wrap: wrap; }
+ .take .actions { display: flex; gap: 6px; margin-top: auto; }
+ .take .actions .btn { padding: 3px 8px; font-size: 12px; }
+ .spin { display: inline-block; width: 9px; height: 9px; border: 2px solid currentColor; border-right-color: transparent; border-radius: 50%; animation: s .8s linear infinite; vertical-align: -1px; }
+ @keyframes s { to { transform: rotate(360deg); } }
+ input[type=file] { display: none; }
+</style>
+</head>
+<body>
+<div class="app">
+ <aside>
+ <h1><span class="dot"></span> Prompt2Rig</h1>
+ <div class="pills">
+ <span class="pill" id="pill-rig">character…</span>
+ <span class="pill" id="pill-pose">pose model…</span>
+ <span class="pill" id="pill-hq">HQ worker…</span>
+ </div>
+
+ <div class="section">
+ <textarea id="prompt" placeholder="Describe a motion… e.g. does an energetic hip hop dance">does an energetic hip hop dance with big arm movements</textarea>
+ <button class="go" id="go">Generate ▸ <span style="opacity:.6;font-weight:400">(Ctrl+Enter)</span></button>
+ <div class="chips" id="chips"></div>
+ </div>
+
+ <div class="section">
+ <label class="title">Performer (first frame)</label>
+ <div class="performer">
+ <img id="performer-img" alt="" />
+ <div style="display:flex;flex-direction:column;gap:6px">
+ <button class="btn" id="performer-new">New performer</button>
+ <span style="color:var(--dim);font-size:11.5px">flux/schnell → image-to-video keeps framing &amp; body consistent</span>
+ </div>
+ </div>
+ </div>
+
+ <div class="section">
+ <label class="title">Generation</label>
+ <label class="check"><input type="checkbox" id="stream" /><span><b>Stream mode</b><small>One continuous performance: each clip continues from the last, plays once its HQ motion is ready, and idles between prompts.</small></span></label>
+ <div class="row"><span>Source</span>
+ <select id="source"><option value="performer">Performer → video</option><option value="t2v">Text → video</option></select></div>
+ <label class="check"><input type="checkbox" id="chain" /><span>Continue from last frame<small>Each clip starts where the previous ended — a continuous stream you direct with prompts.</small></span></label>
+ <div class="row"><span>Duration <b id="duration-v"></b></span><input type="range" id="duration" min="2" max="10" step="1" /></div>
+ <div class="row"><span>Resolution</span><select id="resolution"><option>480P</option><option>768P</option><option>1080P</option></select></div>
+ </div>
+
+ <div class="section">
+ <label class="title">Motion</label>
+ <label class="check"><input type="checkbox" id="hq" /><span>HQ pass (GVHMR on local GPU)<small>Live MediaPipe preview immediately, swaps to GVHMR motion when ready.</small></span></label>
+ <label class="check"><input type="checkbox" id="waitHq" /><span>Hold playback until HQ is ready<small>Cleanest look; adds a few seconds before each clip.</small></span></label>
+ <label class="check"><input type="checkbox" id="rootMotion" /><span>Root motion</span></label>
+ <div class="row"><span>Live pose model</span><select id="poseModel"><option value="heavy">Heavy</option><option value="full">Full</option></select></div>
+ <div class="row"><span>Live smoothing</span><input type="range" id="smoothing" min="0" max="1" step="0.05" /></div>
+ <label class="check"><input type="checkbox" id="showSkel" /><span>Skeleton overlay on video</span></label>
+ </div>
+
+ <div class="section">
+ <label class="title">Files</label>
+ <div class="row">
+ <button class="btn" id="btn-char">Load character…</button>
+ <button class="btn" id="btn-vid">Use a video file…</button>
+ </div>
+ <input type="file" id="file-char" accept=".glb,.gltf,.fbx" />
+ <input type="file" id="file-vid" accept="video/*" />
+ </div>
+
+ <div class="section" style="flex:1">
+ <label class="title">Log</label>
+ <div class="log" id="log"></div>
+ </div>
+ </aside>
+
+ <main>
+ <div id="stage"></div>
+ <div class="vidwrap" id="vidwrap">
+ <div class="empty" id="vid-empty">Generated video appears here</div>
+ <video id="video" muted playsinline crossorigin="anonymous" style="display:none"></video>
+ <canvas id="overlay"></canvas>
+ </div>
+ <div class="hud">
+ <div class="tag" id="hud-src">—</div>
+ <div class="tag" id="hud-time" style="color:var(--dim)"></div>
+ </div>
+ <div class="banner" id="banner"></div>
+ </main>
+
+ <footer id="takes"></footer>
+</div>
+<script type="module" src="./app.js"></script>
+</body>
+</html>
diff --git a/public/retarget.js b/public/retarget.js
new file mode 100644
index 0000000..f3cd7a3
--- /dev/null
+++ b/public/retarget.js
@@ -0,0 +1,363 @@
+// Retargeting onto a Mixamo skeleton.
+//
+// Two sources drive the same Rig:
+// • MediaPipe world landmarks (live, per frame) → applyLandmarks()
+// • SMPL joint rotations from GVHMR (HQ, baked) → applySmplGlobal()
+// Both end in local bone quaternions + a hips world position, captured with
+// snapshot() so takes can be sampled, blended and exported.
+import * as THREE from 'three';
+
+const V3 = THREE.Vector3;
+const Q = THREE.Quaternion;
+
+export const BONES = [
+ 'Hips', 'Spine', 'Spine1', 'Spine2', 'Neck', 'Head',
+ 'LeftShoulder', 'LeftArm', 'LeftForeArm', 'LeftHand',
+ 'RightShoulder', 'RightArm', 'RightForeArm', 'RightHand',
+ 'LeftUpLeg', 'LeftLeg', 'LeftFoot', 'LeftToeBase',
+ 'RightUpLeg', 'RightLeg', 'RightFoot', 'RightToeBase',
+];
+
+// SMPL body joint index → Mixamo bone (GVHMR's 22 body joints).
+export const SMPL_PARENTS = [-1, 0, 0, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 9, 9, 12, 13, 14, 16, 17, 18, 19];
+export const SMPL_TO_MIXAMO = [
+ 'Hips', 'LeftUpLeg', 'RightUpLeg', 'Spine', 'LeftLeg', 'RightLeg', 'Spine1', 'LeftFoot', 'RightFoot',
+ 'Spine2', 'LeftToeBase', 'RightToeBase', 'Neck', 'LeftShoulder', 'RightShoulder', 'Head',
+ 'LeftArm', 'RightArm', 'LeftForeArm', 'RightForeArm', 'LeftHand', 'RightHand',
+];
+
+// MediaPipe Pose landmark indices.
+const MP = {
+ nose: 0, lEye: 2, rEye: 5, lEar: 7, rEar: 8,
+ lShoulder: 11, rShoulder: 12, lElbow: 13, rElbow: 14, lWrist: 15, rWrist: 16,
+ lPinky: 17, rPinky: 18, lIndex: 19, rIndex: 20,
+ lHip: 23, rHip: 24, lKnee: 25, rKnee: 26, lAnkle: 27, rAnkle: 28,
+ lHeel: 29, rHeel: 30, lFoot: 31, rFoot: 32,
+};
+
+const FOOT_POINTS = ['LeftToe_End', 'RightToe_End', 'LeftToeBase', 'RightToeBase', 'LeftFoot', 'RightFoot'];
+
+const shortName = (n) => n.replace(/^mixamorig\d*[:_]?/i, '');
+const _m4 = new THREE.Matrix4();
+const _q1 = new Q(), _q2 = new Q(), _q3 = new Q();
+const _v1 = new V3(), _v2 = new V3();
+
+/** Orthonormal frame: X along `x`, Y as close to `yHint` as possible, Z = X × Y. */
+export function basisQuat(x, yHint, out = new Q()) {
+ const X = x.clone().normalize();
+ const Z = new V3().crossVectors(X, yHint).normalize();
+ const Y = new V3().crossVectors(Z, X);
+ _m4.makeBasis(X, Y, Z);
+ return out.setFromRotationMatrix(_m4);
+}
+
+const smoothstep = (a, b, x) => { const t = Math.min(1, Math.max(0, (x - a) / (b - a))); return t * t * (3 - 2 * t); };
+
+export class Rig {
+ constructor(model) {
+ this.model = model;
+ this.bones = {};
+ model.traverse((o) => {
+ if (o.isBone) { const s = shortName(o.name); if (!(s in this.bones)) this.bones[s] = o; }
+ });
+ for (const need of ['Hips', 'Spine', 'LeftArm', 'RightArm', 'LeftUpLeg', 'RightUpLeg']) {
+ if (!this.bones[need]) throw new Error(`Not a Mixamo rig: bone "${need}" not found`);
+ }
+ this.order = BONES.filter((n) => this.bones[n]);
+ this.hips = this.bones.Hips;
+
+ model.updateMatrixWorld(true);
+ this.rest = {};
+ for (const [n, b] of Object.entries(this.bones)) {
+ this.rest[n] = {
+ q: b.quaternion.clone(), p: b.position.clone(),
+ wq: b.getWorldQuaternion(new Q()), wp: b.getWorldPosition(new V3()),
+ };
+ }
+ const R = (n) => this.rest[n].wp;
+ const chestTop = this.rest.Neck ? R('Neck') : R('Spine2') || R('Spine');
+ this.restHipBasis = basisQuat(R('LeftUpLeg').clone().sub(R('RightUpLeg')), chestTop.clone().sub(R('Hips')));
+ this.restChestBasis = basisQuat(R('LeftArm').clone().sub(R('RightArm')), chestTop.clone().sub(R('Hips')));
+ this.restForward = new V3(0, 0, 1).applyQuaternion(this.restHipBasis);
+
+ // Floor = lowest point of the skinned mesh at rest; per-bone clearance above it.
+ const box = new THREE.Box3().setFromObject(model);
+ this.floorY = box.min.y;
+ this.height = box.max.y - box.min.y;
+ this.footClearance = {};
+ for (const n of FOOT_POINTS) if (this.rest[n]) this.footClearance[n] = R(n).y - this.floorY;
+
+ this.legLength = R('LeftUpLeg').distanceTo(R('LeftLeg')) + R('LeftLeg').distanceTo(R('LeftFoot'));
+
+ this.child = {
+ LeftArm: 'LeftForeArm', LeftForeArm: 'LeftHand', LeftHand: 'LeftHandMiddle1',
+ RightArm: 'RightForeArm', RightForeArm: 'RightHand', RightHand: 'RightHandMiddle1',
+ LeftUpLeg: 'LeftLeg', LeftLeg: 'LeftFoot', LeftFoot: 'LeftToeBase',
+ RightUpLeg: 'RightLeg', RightLeg: 'RightFoot', RightFoot: 'RightToeBase',
+ };
+ }
+
+ // ---- low-level ------------------------------------------------------------
+ reset() {
+ for (const [n, b] of Object.entries(this.bones)) { b.quaternion.copy(this.rest[n].q); b.position.copy(this.rest[n].p); }
+ this.model.updateMatrixWorld(true);
+ }
+
+ setWorldQuat(name, wq) {
+ const b = this.bones[name];
+ if (!b) return;
+ b.parent.getWorldQuaternion(_q3);
+ b.quaternion.copy(_q3.invert().multiply(wq));
+ b.updateMatrixWorld(true);
+ }
+
+ setHipsWorld(pos) {
+ this.hips.position.copy(this.hips.parent.worldToLocal(pos.clone()));
+ this.hips.updateMatrixWorld(true);
+ }
+
+ hipsWorld(out = new V3()) { return this.hips.getWorldPosition(out); }
+
+ /** Lowest world y of the feet, corrected for each joint's rest height above the floor. */
+ lowestFoot() {
+ let lo = Infinity;
+ for (const [n, c] of Object.entries(this.footClearance)) {
+ lo = Math.min(lo, this.bones[n].getWorldPosition(_v1).y - c);
+ }
+ return lo;
+ }
+
+ /** Raise the hips so no foot goes below the floor. Returns the lift applied. */
+ clampToFloor(extraLift = 0) {
+ const lift = Math.max(0, this.floorY - this.lowestFoot()) + extraLift;
+ if (lift) { const p = this.hipsWorld(); p.y += lift; this.setHipsWorld(p); }
+ return lift;
+ }
+
+ // Pose state as plain arrays: local quats in `order`, plus hips world position.
+ snapshot(out) {
+ const n = this.order.length;
+ out = out || { q: new Float32Array(n * 4), p: new Float32Array(3) };
+ this.order.forEach((name, i) => this.bones[name].quaternion.toArray(out.q, i * 4));
+ this.hipsWorld(_v2).toArray(out.p);
+ return out;
+ }
+
+ applySnapshot(s) {
+ this.order.forEach((name, i) => this.bones[name].quaternion.fromArray(s.q, i * 4));
+ this.model.updateMatrixWorld(true);
+ this.setHipsWorld(_v2.fromArray(s.p));
+ }
+
+ // ---- source A: MediaPipe world landmarks -----------------------------------
+ /**
+ * `lm` = 33 THREE.Vector3 in three.js space (x = subject's left, y up, z toward camera),
+ * hip-centred meters. `vis` = per-landmark visibility.
+ */
+ applyLandmarks(lm, vis) {
+ const P = (i) => lm[i];
+ const mid = (a, b) => P(a).clone().add(P(b)).multiplyScalar(0.5);
+ const ok = (...ids) => ids.every((i) => (vis?.[i] ?? 1) > 0.35);
+ const dir = (a, b) => P(b).clone().sub(P(a));
+
+ const midHip = mid(MP.lHip, MP.rHip);
+ const midSh = mid(MP.lShoulder, MP.rShoulder);
+ const up = midSh.clone().sub(midHip);
+
+ const qHip = basisQuat(dir(MP.rHip, MP.lHip), up);
+ const qChest = basisQuat(dir(MP.rShoulder, MP.lShoulder), up);
+ const dHip = qHip.multiply(_q1.copy(this.restHipBasis).invert());
+ const dChest = qChest.multiply(_q1.copy(this.restChestBasis).invert());
+
+ // Head frame from ears + eyes (eye→ear line is roughly level).
+ const earX = dir(MP.rEar, MP.lEar);
+ const fwd = mid(MP.lEye, MP.rEye).sub(mid(MP.lEar, MP.rEar));
+ const qHead = basisQuat(earX, new V3().crossVectors(fwd, earX));
+ let dHead = qHead.multiply(_q1.copy(this.restChestBasis).invert());
+ if (!ok(MP.lEar, MP.rEar)) dHead = dChest.clone();
+
+ const worldFrom = (delta, name) => delta.clone().multiply(this.rest[name].wq);
+ this.setWorldQuat('Hips', worldFrom(dHip, 'Hips'));
+ this.setWorldQuat('Spine', worldFrom(dHip.clone().slerp(dChest, 0.35), 'Spine'));
+ if (this.bones.Spine1) this.setWorldQuat('Spine1', worldFrom(dHip.clone().slerp(dChest, 0.7), 'Spine1'));
+ if (this.bones.Spine2) this.setWorldQuat('Spine2', worldFrom(dChest, 'Spine2'));
+ if (this.bones.Neck) this.setWorldQuat('Neck', worldFrom(dChest.clone().slerp(dHead, 0.4), 'Neck'));
+ if (this.bones.Head) this.setWorldQuat('Head', worldFrom(dHead, 'Head'));
+
+ const fwdChest = new V3(0, 0, 1).applyQuaternion(dChest).applyQuaternion(this.restChestBasis);
+ const fwdHip = new V3(0, 0, 1).applyQuaternion(dHip).applyQuaternion(this.restHipBasis);
+
+ for (const side of ['Left', 'Right']) {
+ const L = side === 'Left';
+ const sh = L ? MP.lShoulder : MP.rShoulder, el = L ? MP.lElbow : MP.rElbow, wr = L ? MP.lWrist : MP.rWrist;
+ const hp = L ? MP.lHip : MP.rHip, kn = L ? MP.lKnee : MP.rKnee, an = L ? MP.lAnkle : MP.rAnkle, ft = L ? MP.lFoot : MP.rFoot;
+ const ix = L ? MP.lIndex : MP.rIndex, pk = L ? MP.lPinky : MP.rPinky;
+
+ if (ok(sh, el)) this.aimLimb(`${side}Arm`, dir(sh, el), dir(el, wr), this.restForward, fwdChest);
+ if (ok(el, wr)) this.aim(`${side}ForeArm`, dir(el, wr));
+ if (ok(wr, ix, pk)) this.aimHand(`${side}Hand`, mid(ix, pk).sub(P(wr)), dir(pk, ix), 0.7);
+ if (ok(hp, kn)) this.aimLimb(`${side}UpLeg`, dir(hp, kn), dir(kn, an), this.restForward.clone().negate(), fwdHip.clone().negate());
+ if (ok(kn, an)) this.aim(`${side}Leg`, dir(kn, an));
+ if (ok(an, ft)) this.aim(`${side}Foot`, dir(an, ft));
+ }
+ }
+
+ /** Swing the bone so its rest bone-axis points along `target` (keeps inherited twist). */
+ aim(name, target, weight = 1) {
+ const b = this.bones[name], c = this.bones[this.child[name]];
+ if (!b || !c || target.lengthSq() < 1e-8) return;
+ const cur = c.getWorldPosition(_v1).sub(b.getWorldPosition(_v2)).normalize();
+ const delta = new Q().setFromUnitVectors(cur, target.clone().normalize());
+ if (weight < 1) delta.slerp(new Q(), 1 - weight);
+ this.setWorldQuat(name, delta.multiply(b.getWorldQuaternion(_q2)));
+ }
+
+ /**
+ * Upper arm / thigh: aim along `upper`, and use the bend plane (direction the
+ * lower segment folds toward) to recover twist so elbows and knees hinge the
+ * right way. Fades to plain swing when the limb is nearly straight.
+ */
+ aimLimb(name, upper, lower, restBendDir, fallbackBend) {
+ const b = this.bones[name], c = this.bones[this.child[name]];
+ if (!b || !c) return;
+ const u = upper.clone().normalize();
+ const l = lower.clone().normalize();
+ const bend = l.clone().sub(u.clone().multiplyScalar(l.dot(u)));
+ const bendAmt = bend.length();
+
+ // Swing-only solution.
+ this.aim(name, u);
+ if (bendAmt < 0.15) return;
+ const swingLocal = b.quaternion.clone();
+
+ // Bend-plane solution: absolute, from rest frame (bone axis, rest bend dir).
+ const restAxis = this.rest[this.child[name]].wp.clone().sub(this.rest[name].wp).normalize();
+ const restBend = restBendDir.clone().sub(restAxis.clone().multiplyScalar(restBendDir.dot(restAxis)));
+ if (restBend.lengthSq() < 1e-6) return;
+ const qRest = basisQuat(restAxis, restBend);
+ const qNow = basisQuat(u, bend.normalize());
+ const wq = qNow.multiply(qRest.invert()).multiply(this.rest[name].wq);
+ this.setWorldQuat(name, wq);
+ const planeLocal = b.quaternion.clone();
+
+ const w = smoothstep(0.15, 0.45, bendAmt);
+ b.quaternion.copy(swingLocal.slerp(planeLocal, w));
+ b.updateMatrixWorld(true);
+ void fallbackBend;
+ }
+
+ /** Hand: aim at knuckles, roll from the pinky→index line. */
+ aimHand(name, along, across, weight = 1) {
+ const b = this.bones[name];
+ const side = name.startsWith('Left') ? 'Left' : 'Right';
+ const idx = this.rest[`${side}HandIndex1`], pky = this.rest[`${side}HandPinky1`], mid = this.rest[`${side}HandMiddle1`];
+ if (!b || !idx || !pky || !mid) { this.aim(name, along, weight); return; }
+ const before = b.quaternion.clone();
+ const restAlong = mid.wp.clone().sub(this.rest[name].wp);
+ const restAcross = idx.wp.clone().sub(pky.wp);
+ const qRest = basisQuat(restAlong, new V3().crossVectors(restAlong, restAcross).cross(restAlong));
+ const qNow = basisQuat(along, new V3().crossVectors(along, across).cross(along));
+ this.setWorldQuat(name, qNow.multiply(qRest.invert()).multiply(this.rest[name].wq));
+ if (weight < 1) { b.quaternion.copy(before.slerp(b.quaternion, weight)); b.updateMatrixWorld(true); }
+ }
+
+ // ---- source B: SMPL (GVHMR) -------------------------------------------------
+ /** `G` = 22 global joint quaternions (three.js frame), relative to the SMPL T-pose. */
+ applySmplGlobal(G) {
+ for (let j = 0; j < SMPL_TO_MIXAMO.length; j++) {
+ const name = SMPL_TO_MIXAMO[j];
+ if (!this.bones[name] || !G[j]) continue;
+ this.setWorldQuat(name, _q1.copy(G[j]).multiply(this.rest[name].wq));
+ }
+ }
+}
+
+// ---- SMPL forward kinematics ---------------------------------------------------
+export function aaToQuat(x, y, z, out = new Q()) {
+ const a = Math.hypot(x, y, z);
+ if (a < 1e-8) return out.set(0, 0, 0, 1);
+ return out.setFromAxisAngle(_v1.set(x / a, y / a, z / a), a);
+}
+
+/** One frame: global_orient (3) + body_pose (63) → 22 global quaternions. */
+export function smplGlobals(go, bp, yawFix) {
+ const G = new Array(22);
+ G[0] = yawFix.clone().multiply(aaToQuat(go[0], go[1], go[2]));
+ for (let j = 1; j < 22; j++) {
+ const k = (j - 1) * 3;
+ G[j] = G[SMPL_PARENTS[j]].clone().multiply(aaToQuat(bp[k], bp[k + 1], bp[k + 2]));
+ }
+ return G;
+}
+
+// ---- One Euro filter (for live landmarks) --------------------------------------
+export class OneEuro {
+ constructor(n, minCutoff = 1.5, beta = 0.6, dCutoff = 1.0) {
+ Object.assign(this, { n, minCutoff, beta, dCutoff });
+ this.x = null; this.dx = new Float32Array(n); this.t = 0;
+ }
+ reset() { this.x = null; }
+ static alpha(cutoff, dt) { const r = 2 * Math.PI * cutoff * dt; return r / (r + 1); }
+ filter(values, t) {
+ if (!this.x || t <= this.t) { this.x = Float32Array.from(values); this.t = t; this.dx.fill(0); return this.x; }
+ const dt = Math.min(0.2, t - this.t); this.t = t;
+ const ad = OneEuro.alpha(this.dCutoff, dt);
+ for (let i = 0; i < this.n; i++) {
+ const dx = (values[i] - this.x[i]) / dt;
+ this.dx[i] += ad * (dx - this.dx[i]);
+ const a = OneEuro.alpha(this.minCutoff + this.beta * Math.abs(this.dx[i]), dt);
+ this.x[i] += a * (values[i] - this.x[i]);
+ }
+ return this.x;
+ }
+}
+
+// ---- Takes: frame storage, sampling, export -------------------------------------
+export class Track {
+ constructor(boneCount) { this.boneCount = boneCount; this.times = []; this.q = []; this.p = []; }
+ push(t, snap) { this.times.push(t); this.q.push(Float32Array.from(snap.q)); this.p.push(Float32Array.from(snap.p)); }
+ get length() { return this.times.length; }
+ get duration() { return this.times.length ? this.times[this.times.length - 1] : 0; }
+
+ /** Interpolated snapshot at time t (clamped). */
+ sample(t, out) {
+ const n = this.times.length;
+ out = out || { q: new Float32Array(this.boneCount * 4), p: new Float32Array(3) };
+ if (!n) return null;
+ let i = 0;
+ if (t <= this.times[0]) i = 0;
+ else if (t >= this.times[n - 1]) i = n - 1;
+ else { let lo = 0, hi = n - 1; while (hi - lo > 1) { const m = (lo + hi) >> 1; (this.times[m] <= t ? (lo = m) : (hi = m)); } i = lo; }
+ const j = Math.min(n - 1, i + 1);
+ const span = this.times[j] - this.times[i];
+ const a = span > 1e-6 ? Math.min(1, Math.max(0, (t - this.times[i]) / span)) : 0;
+ blendSnapshots(this.q[i], this.p[i], this.q[j], this.p[j], a, out);
+ return out;
+ }
+}
+
+export function blendSnapshots(qa, pa, qb, pb, a, out) {
+ for (let k = 0; k < qa.length; k += 4) {
+ _q1.fromArray(qa, k); _q2.fromArray(qb, k);
+ _q1.slerp(_q2, a).toArray(out.q, k);
+ }
+ for (let k = 0; k < 3; k++) out.p[k] = pa[k] + (pb[k] - pa[k]) * a;
+ return out;
+}
+
+/** Bake a Track into a THREE.AnimationClip (local quats + hips local position). */
+export function trackToClip(rig, track, name) {
+ const times = Float32Array.from(track.times.map((t) => t - track.times[0]));
+ const tracks = rig.order.map((bone, bi) => {
+ const v = new Float32Array(times.length * 4);
+ track.q.forEach((q, f) => v.set(q.subarray(bi * 4, bi * 4 + 4), f * 4));
+ return new THREE.QuaternionKeyframeTrack(`${rig.bones[bone].name}.quaternion`, times, v);
+ });
+ const pos = new Float32Array(times.length * 3);
+ const parentInv = new THREE.Matrix4();
+ rig.reset();
+ parentInv.copy(rig.hips.parent.matrixWorld).invert();
+ track.p.forEach((p, f) => _v1.fromArray(p).applyMatrix4(parentInv).toArray(pos, f * 3));
+ tracks.push(new THREE.VectorKeyframeTrack(`${rig.hips.name}.position`, times, pos));
+ return new THREE.AnimationClip(name, times[times.length - 1] || 0, tracks);
+}
diff --git a/server.mjs b/server.mjs
new file mode 100644
index 0000000..2ea810f
--- /dev/null
+++ b/server.mjs
@@ -0,0 +1,135 @@
+// Prompt2Rig — tiny zero-dependency server.
+// Serves ./public and proxies fal.ai so the API key never reaches the browser.
+import http from 'node:http';
+import fs from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const ROOT = path.dirname(fileURLToPath(import.meta.url));
+const PUBLIC = path.join(ROOT, 'public');
+
+// --- .env -------------------------------------------------------------------
+const envFile = path.join(ROOT, '.env');
+if (fs.existsSync(envFile)) {
+ for (const line of fs.readFileSync(envFile, 'utf8').split(/\r?\n/)) {
+ const m = line.match(/^\s*([A-Z0-9_]+)\s*=\s*(.*)\s*$/);
+ if (m && !process.env[m[1]]) process.env[m[1]] = m[2];
+ }
+}
+const FAL_KEY = process.env.FAL_KEY;
+const PORT = Number(process.env.PORT || 5177);
+const HQ_URL = process.env.HQ_URL || 'http://127.0.0.1:5178';
+if (!FAL_KEY) console.warn('[prompt2rig] FAL_KEY is not set — generation will fail. Put it in .env');
+
+const MODELS = {
+ t2v: 'minimax/h3-max-turbo/text-to-video',
+ i2v: 'minimax/h3-max-turbo/image-to-video',
+ image: 'fal-ai/flux/schnell',
+};
+
+async function falRun(model, input) {
+ const t0 = performance.now();
+ const res = await fetch(`https://fal.run/${model}`, {
+ method: 'POST',
+ headers: { Authorization: `Key ${FAL_KEY}`, 'Content-Type': 'application/json' },
+ body: JSON.stringify(input),
+ });
+ const text = await res.text();
+ let json;
+ try { json = JSON.parse(text); } catch { json = { raw: text }; }
+ if (!res.ok) {
+ const detail = json?.detail ? JSON.stringify(json.detail) : text.slice(0, 500);
+ throw Object.assign(new Error(`fal ${model} → ${res.status}: ${detail}`), { status: res.status });
+ }
+ return { ...json, _elapsed: (performance.now() - t0) / 1000 };
+}
+
+// --- request handling ----------------------------------------------------------
+function readJson(req) {
+ return new Promise((resolve, reject) => {
+ let size = 0; const chunks = [];
+ req.on('data', (c) => { size += c.length; if (size > 25e6) reject(new Error('body too large')); else chunks.push(c); });
+ req.on('end', () => { try { resolve(JSON.parse(Buffer.concat(chunks).toString() || '{}')); } catch (e) { reject(e); } });
+ req.on('error', reject);
+ });
+}
+
+function send(res, status, body) {
+ res.writeHead(status, { 'Content-Type': 'application/json' });
+ res.end(JSON.stringify(body));
+}
+
+const MIME = {
+ '.html': 'text/html; charset=utf-8', '.js': 'text/javascript; charset=utf-8', '.css': 'text/css',
+ '.json': 'application/json', '.glb': 'model/gltf-binary', '.png': 'image/png', '.svg': 'image/svg+xml',
+};
+
+const server = http.createServer(async (req, res) => {
+ const url = new URL(req.url, 'http://localhost');
+ try {
+ if (req.method === 'POST' && url.pathname === '/api/video') {
+ const b = await readJson(req);
+ const mode = b.image_url ? 'i2v' : 't2v';
+ const input = {
+ prompt: String(b.prompt || '').slice(0, 4000),
+ duration: Math.min(15, Math.max(1, Number(b.duration) || 5)),
+ resolution: ['480P', '768P', '1080P'].includes(b.resolution) ? b.resolution : '480P',
+ prompt_expansion_mode: b.prompt_expansion_mode || 'disabled',
+ };
+ if (b.seed != null && b.seed !== '') input.seed = Number(b.seed);
+ if (mode === 'i2v') input.image_url = b.image_url;
+ else input.aspect_ratio = b.aspect_ratio || '16:9';
+ const out = await falRun(MODELS[mode], input);
+ console.log(`[video] ${mode} ${input.duration}s ${input.resolution} in ${out._elapsed.toFixed(1)}s (gpu ${out.timings?.inference?.toFixed?.(2) ?? '?'}s)`);
+ return send(res, 200, { mode, video_url: out.video?.url, timings: out.timings, elapsed: out._elapsed, expanded_prompt: out.expanded_prompt });
+ }
+
+ if (req.method === 'POST' && url.pathname === '/api/image') {
+ const b = await readJson(req);
+ const out = await falRun(MODELS.image, {
+ prompt: String(b.prompt || '').slice(0, 4000),
+ image_size: b.image_size || 'landscape_16_9',
+ num_inference_steps: 4,
+ enable_safety_checker: true,
+ });
+ console.log(`[image] in ${out._elapsed.toFixed(1)}s`);
+ return send(res, 200, { image_url: out.images?.[0]?.url, elapsed: out._elapsed });
+ }
+
+ // HQ pass: forward to the local GVHMR worker (hq/worker.py).
+ if (url.pathname === '/api/hq/health') {
+ try {
+ const r = await fetch(`${HQ_URL}/health`, { signal: AbortSignal.timeout(2000) });
+ return send(res, 200, await r.json());
+ } catch { return send(res, 200, { ok: false, error: 'worker offline' }); }
+ }
+ if (req.method === 'POST' && url.pathname === '/api/hq') {
+ const b = await readJson(req);
+ const r = await fetch(`${HQ_URL}/process`, {
+ method: 'POST', headers: { 'Content-Type': 'application/json' },
+ body: JSON.stringify({ video_url: b.video_url }),
+ });
+ const text = await r.text();
+ res.writeHead(r.status, { 'Content-Type': 'application/json' });
+ return res.end(text);
+ }
+
+ if (req.method === 'GET') {
+ let p = decodeURIComponent(url.pathname);
+ if (p === '/') p = '/index.html';
+ const file = path.normalize(path.join(PUBLIC, p));
+ if (!file.startsWith(PUBLIC) || !fs.existsSync(file) || fs.statSync(file).isDirectory()) {
+ res.writeHead(404); return res.end('not found');
+ }
+ res.writeHead(200, { 'Content-Type': MIME[path.extname(file)] || 'application/octet-stream', 'Cache-Control': 'no-cache' });
+ return fs.createReadStream(file).pipe(res);
+ }
+
+ res.writeHead(405); res.end();
+ } catch (err) {
+ console.error('[error]', err.message);
+ send(res, err.status && err.status < 600 ? err.status : 500, { error: err.message });
+ }
+});
+
+server.listen(PORT, () => console.log(`Prompt2Rig → http://localhost:${PORT}`));