Type a motion, watch a rigged character perform it: fast video model + GVHMR + live Mixamo retargeting
JavaScript
Add FLUX 3 Max video model and a character picker
blendi-remade353d24b0ea44
diff --git a/README.md b/README.mdindex 9308abd..6f4180d 100644--- a/README.md+++ b/README.md@@ -2,7 +2,7 @@**Type a motion, watch a rigged character perform it — in seconds.**-A text prompt goes to a faster-than-real-time video model ([MiniMax H3 Max Turbo](https://fal.ai/models/minimax/h3-max-turbo/image-to-video) on fal), the clip's human motion is recovered with [GVHMR](https://github.com/zju3dv/GVHMR) on a local GPU, and retargeted live onto a Mixamo skeleton in the browser. Inspired by [Motion2MixamoRig](https://github.com/MaxLiu22/Motion2MixamoRig), rebuilt for interactive speed.+A text prompt goes to a fast video model on fal — [MiniMax H3 Max Turbo](https://fal.ai/models/minimax/h3-max-turbo/image-to-video) (faster than real time) or [FLUX 3 Max](https://fal.ai/models/blackforestlabs/flux-3-max/image-to-video) (higher fidelity, ~real time) — the clip's human motion is recovered with [GVHMR](https://github.com/zju3dv/GVHMR) on a local GPU, and retargeted live onto a Mixamo skeleton in the browser. Inspired by [Motion2MixamoRig](https://github.com/MaxLiu22/Motion2MixamoRig), rebuilt for interactive speed.@@ -10,7 +10,7 @@ A text prompt goes to a faster-than-real-time video model ([MiniMax H3 Max Turbo```mermaidflowchart LR- P[prompt] --> V["H3 Max Turbo<br/>5s clip · ~0.5s GPU"]+ P[prompt] --> V["H3 Max Turbo / FLUX 3 Max<br/>5s clip · 0.5–4.5s GPU"]I["performer image<br/>or previous clip's last frame"] --> VV --> L["MediaPipe (browser)<br/>instant preview"]V --> H["GVHMR (local GPU)<br/>SMPL motion · ~3.4s"]@@ -21,9 +21,10 @@ flowchart LR| Stage | Where | Time (5s clip, RTX 4090) ||---|---|---|-| Video generation | fal · H3 Max Turbo | ~2–3s wall (0.5s GPU) |+| Video generation | fal · H3 Max Turbo (480P) | ~2–3s wall (0.5s GPU) |+| | fal · FLUX 3 Max (720p) | ~8–9s wall (4.5s GPU) || **Live** preview — MediaPipe Pose, retargeted per frame | browser | immediate |-| **HQ** motion — GVHMR → SMPL rotations → Mixamo | local GPU worker | ~3.4s (0.7× real time) |+| **HQ** motion — GVHMR → SMPL rotations → Mixamo | local GPU worker | ~3.5–4s (0.7–0.8× real time) |The live pass hides the wait; the HQ pass is the real result. GVHMR sees the whole clip at once, outputs joint *rotations* on a fixed-proportion body, and recovers a gravity-aware root path — so there's no jitter, limb stretching, or twisted forearms.@@ -54,10 +55,12 @@ hq\.venv\Scripts\python hq\worker.py # → 127.0.0.1:5178, loads once and sta## Using it- **Generate** a prompt or click a preset. By default every clip starts from the same flux-generated *performer* (one person, full body, plain backdrop, static camera — ideal pose-estimation input).+- **Video model** — *H3 Max Turbo* for the snappiest loop and Stream mode; *FLUX 3 Max* for more realistic motion (best for single takes — at ~9s per 5s clip it can't keep a chained stream fed yet).+- **Character** — X Bot, Michelle or Soldier (textured Mixamo rigs from the three.js examples), or **Load Mixamo file…** for any Mixamo `.fbx`/`.glb` (download from [mixamo.com](https://www.mixamo.com), T-pose).- **Orientation** — portrait 9:16 (default) or landscape 16:9. Portrait gives the performer ~70% more pixels of height at 480P, which helps pose estimation.- **Stream mode** — one continuous performance: each clip continues from the previous clip's last frame, plays once its HQ motion is ready while the next one generates, and short idle clips fill gaps between prompts. First motion ≈ 6s after the first prompt, continuous after that.- **⬇ GLB** on any take exports the skinned character with the animation clip (Blender, Unity, three.js).-- **Load character…** swaps in any Mixamo-rigged `.glb`/`.fbx`; **Use a video file…** runs the live pass on your own footage.+- **Use a video file…** runs the live pass on your own footage.## Project layout@@ -74,7 +77,8 @@ hq\.venv\Scripts\python hq\worker.py # → 127.0.0.1:5178, loads once and sta- batched single-person YOLO11m instead of frame-by-frame YOLOv8x + BoT-SORT tracking- decoded frames reused by the crop stage (OpenCV) instead of a second PyAV decode- ViTPose without flip-test (`HQ_POSE_FLIP=1` to re-enable — measured 0.6° mean joint difference)-- 24→30 fps resample with ffmpeg `ultrafast`; all GPU work on one warmed-up thread+- 24→30 fps resample with ffmpeg `ultrafast`, long side capped at 854px (GVHMR only sees 256px crops)+- fixed detector batch shape and per-orientation warm-up; all GPU work on one warmed-up thread- left/right keypoint-swap stabilizer from Motion2MixamoRig- camera-relative heading so chained clips line up (seams: 1–5° hips, ≤5 cm root)@@ -83,7 +87,7 @@ hq\.venv\Scripts\python hq\worker.py # → 127.0.0.1:5178, loads once and sta## Credits & licenses- [GVHMR](https://github.com/zju3dv/GVHMR) (ZJU3DV) via the [`gvhmr`](https://pypi.org/project/gvhmr/) package · [Motion2MixamoRig](https://github.com/MaxLiu22/Motion2MixamoRig) (Apache-2.0) for the idea and the keypoint stabilizer-- [MediaPipe Pose](https://ai.google.dev/edge/mediapipe/solutions/vision/pose_landmarker) · [three.js](https://threejs.org) (X Bot model from its examples) · video/image models on [fal](https://fal.ai)+- [MediaPipe Pose](https://ai.google.dev/edge/mediapipe/solutions/vision/pose_landmarker) · [three.js](https://threejs.org) (X Bot, Michelle and Soldier from its examples, loaded from the CDN) · video/image models on [fal](https://fal.ai)- **SMPL-X** is licensed for non-commercial research only and is not included — download it yourself.Prototype — tested on Windows 11 with an RTX 4090.diff --git a/hq/worker.py b/hq/worker.pyindex f580fc5..97259f6 100644--- a/hq/worker.py+++ b/hq/worker.py@@ -120,7 +120,10 @@ class FastSingleDetector:boxes = np.zeros((len(frames), 4), np.float32)found = np.zeros(len(frames), bool)for i in range(0, len(frames), self.batch):- results = self.yolo.predict(frames[i:i + self.batch], classes=[0], conf=0.3, half=True, device=0, verbose=False)+ chunk = frames[i:i + self.batch]+ # Pad to a fixed batch size: every new batch shape costs a slow first CUDA call.+ pad = [chunk[-1]] * (self.batch - len(chunk))+ results = self.yolo.predict(chunk + pad, classes=[0], conf=0.3, half=True, device=0, verbose=False)[:len(chunk)]for j, r in enumerate(results):if len(r.boxes):xyxy = r.boxes.xyxy.cpu().numpy()@@ -172,8 +175,8 @@ def warmup() -> None:leg_length(result.smpl_params_world)if DETECTOR == "fast": # first YOLO call per input shape is slow: cover both orientationsdet = FastSingleDetector.get()- for h, w in ((480, 832), (832, 480), (768, 1360), (1360, 768)):- for n in (det.batch, 27):+ for h, w in ((480, 832), (832, 480), (470, 854), (854, 470)): # H3 480P, FLUX 3 720p (downscaled)+ for n in (det.batch,):det.yolo.predict([np.zeros((h, w, 3), np.uint8)] * n, classes=[0], half=True, device=0, verbose=False)log(f"warmup done in {time.perf_counter() - t0:.1f}s")except Exception as e: # noqa: BLE001@@ -262,7 +265,10 @@ def process(body: ProcessIn):with urllib.request.urlopen(body.video_url, timeout=30) as r, open(src, "wb") as f:shutil.copyfileobj(r, f)# GVHMR runs at 30fps and would re-encode a 24fps clip with a slow preset; do it fast here.- subprocess.run(["ffmpeg", "-y", "-loglevel", "error", "-i", str(src), "-vf", "fps=30", "-an",+ # It only ever sees 256px person crops, so cap the long side at 854px (H3 480P size): 720p++ # sources (FLUX 3) otherwise cost ~2x in decode/crop for no accuracy gain.+ fit = "scale='if(gt(iw,ih),min(854,iw),-2)':'if(gt(iw,ih),-2,min(854,ih))'"+ subprocess.run(["ffmpeg", "-y", "-loglevel", "error", "-i", str(src), "-vf", f"fps=30,{fit}", "-an","-c:v", "libx264", "-preset", "ultrafast", "-crf", "16", str(video)], check=True)t_dl = time.perf_counter() - t0if not (CACHE / "_warmup.mp4").exists(): # reused to warm CUDA kernels on the next startdiff --git a/public/app.js b/public/app.jsindex ae5d4df..6e93cae 100644--- a/public/app.js+++ b/public/app.js@@ -9,7 +9,14 @@ import { FilesetResolver, PoseLandmarker, DrawingUtils } from '@mediapipe/tasks-import { Rig, OneEuro, Track, trackToClip, blendSnapshots, smplGlobals, aaToQuat } from './retarget.js';const $ = (s) => document.querySelector(s);-const XBOT_URL = 'https://cdn.jsdelivr.net/gh/mrdoob/three.js@r169/examples/models/gltf/Xbot.glb';+// Mixamo-rigged characters from the three.js examples (loaded from the CDN, not redistributed here).+const THREE_MODELS = 'https://cdn.jsdelivr.net/gh/mrdoob/three.js@r169/examples/models/gltf/';+const CHARACTERS = {+ xbot: { name: 'X Bot', url: THREE_MODELS + 'Xbot.glb' },+ michelle: { name: 'Michelle', url: THREE_MODELS + 'Michelle.glb' },+ soldier: { name: 'Soldier', url: THREE_MODELS + 'Soldier.glb' },+};+const VIDEO_MODEL_LABEL = { h3: 'H3 Max Turbo', flux3: 'FLUX 3 Max' };const MP_WASM = 'https://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/wasm';const MP_MODEL = (v) => `https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_${v}/float16/latest/pose_landmarker_${v}.task`;@@ -44,6 +51,7 @@ const videoPrompt = (action, fromImage) => fromImageconst DEFAULTS = {stream: false, source: 'performer', chain: false, duration: 5, resolution: '480P', hq: true, waitHq: false,rootMotion: true, poseModel: 'heavy', smoothing: 0.5, showSkel: true, orientation: 'portrait', performerUrls: {},+ character: 'xbot', videoModel: 'h3',};const settings = (() => { try { return { ...DEFAULTS, ...JSON.parse(localStorage.getItem('p2r') || '{}') }; } catch { return { ...DEFAULTS }; } })();const save = () => { try { localStorage.setItem('p2r', JSON.stringify(settings)); } catch {} };@@ -120,6 +128,13 @@ async function loadCharacter(src, name = 'X Bot') {const root = isFbx ? obj : obj.scene;if (isFbx) root.scale.setScalar(0.01); // Mixamo FBX is in centimetersroot.updateMatrixWorld(true);+ // Retargeting assumes the character faces +Z (toward the camera); some exports face -Z.+ const armX = (side) => {+ let b; root.traverse((o) => { if (!b && o.isBone && o.name.replace(/^mixamorig\d*[:_]?/i, '') === side + 'Arm') b = o; });+ return b?.getWorldPosition(new THREE.Vector3()).x;+ };+ const lx = armX('Left'), rx = armX('Right');+ if (lx != null && rx != null && lx < rx) { root.rotation.y += Math.PI; root.updateMatrixWorld(true); }const box = new THREE.Box3().setFromObject(root);root.position.y -= box.min.y;root.position.x -= (box.min.x + box.max.x) / 2;@@ -131,7 +146,9 @@ async function loadCharacter(src, name = 'X Bot') {if (model) scene.remove(model);scene.add(root); model = root; rig = newRig;// Takes store bone-order-specific data; re-bake HQ data for the new skeleton.- for (const t of takes) { t.live = null; t.liveDone = false; if (t.hqRaw) t.hq = bakeSmpl(t.hqRaw); }+ for (const t of takes) { t.live = new Track(rig.order.length); t.liveDone = false; if (t.hqRaw) t.hq = bakeSmpl(t.hqRaw); }+ liveState ||= { hip0: null, baseY: null, mpp: null, scale: null };+ liveSnap = null; switchFrom = null;pill('rig', name, 'ok');log(`character: ${name} (${rig.order.length} bones, ${rig.height.toFixed(2)} m)`);}@@ -247,10 +264,10 @@ async function generate(take) {image_url = await ensurePerformer(); from = 'performer';}banner(`Generating #${take.id}…`);- log(`#${take.id} → H3 Max Turbo (${from}, ${settings.duration}s ${settings.resolution})`);+ log(`#${take.id} → ${VIDEO_MODEL_LABEL[settings.videoModel]} (${from}, ${settings.duration}s ${settings.resolution})`);const r = await api('/api/video', {prompt: videoPrompt(take.prompt, !!image_url), image_url,- duration: settings.duration, resolution: settings.resolution, aspect_ratio: settings.orientation === 'portrait' ? '9:16' : '16:9',+ model: settings.videoModel, duration: settings.duration, resolution: settings.resolution, aspect_ratio: settings.orientation === 'portrait' ? '9:16' : '16:9',});take.videoUrl = r.video_url; take.status = 'ready';take.timings.gen = (performance.now() - t0) / 1000;@@ -566,6 +583,11 @@ bindSetting('stream', 'checked', (on) => {for (const id of ['chain', 'waitHq']) { const el = $('#' + id); el.disabled = on; el.closest('label').style.opacity = on ? 0.5 : 1; }});bindSetting('source'); bindSetting('chain', 'checked'); bindSetting('resolution');+bindSetting('videoModel', 'value', (m) => {+ const d = $('#duration'); d.min = m === 'flux3' ? 5 : 2; // FLUX 3 Max: 5–20s+ if (settings.duration < d.min) { settings.duration = Number(d.min); d.value = d.min; $('#duration-v').textContent = d.min + 's'; save(); }+});+bindSetting('character', 'value', (c) => { if (rig && CHARACTERS[c]) loadCharacter(CHARACTERS[c].url, CHARACTERS[c].name).catch((e) => log('character failed: ' + e.message, 'err')); });bindSetting('duration', 'value', (v) => ($('#duration-v').textContent = v + 's'));bindSetting('hq', 'checked'); bindSetting('waitHq', 'checked'); bindSetting('rootMotion', 'checked');bindSetting('showSkel', 'checked');@@ -614,7 +636,8 @@ window.p2r = {// ---- boot --------------------------------------------------------------------------------------resize();frame();-loadCharacter(XBOT_URL, 'X Bot').catch((e) => { log('character failed: ' + e.message, 'err'); pill('rig', 'character failed', 'bad'); });+const startChar = CHARACTERS[settings.character] || CHARACTERS.xbot;+loadCharacter(startChar.url, startChar.name).catch((e) => { log('character failed: ' + e.message, 'err'); pill('rig', 'character failed', 'bad'); });initPose(settings.poseModel).catch((e) => { log('pose model failed: ' + e.message, 'err'); pill('pose', 'pose failed', 'bad'); });checkHQ(); setInterval(checkHQ, 10000);log('ready — type a motion and hit Generate');diff --git a/public/index.html b/public/index.htmlindex 1ff96f6..764bf84 100644--- a/public/index.html+++ b/public/index.html@@ -108,6 +108,8 @@<div class="section"><label class="title">Generation</label><label class="check"><input type="checkbox" id="stream" /><span><b>Stream mode</b><small>One continuous performance: each clip continues from the last, plays once its HQ motion is ready, and idles between prompts.</small></span></label>+ <div class="row"><span>Video model</span>+ <select id="videoModel"><option value="h3">H3 Max Turbo (fastest)</option><option value="flux3">FLUX 3 Max (best)</option></select></div><div class="row"><span>Orientation</span><select id="orientation"><option value="portrait">Portrait 9:16</option><option value="landscape">Landscape 16:9</option></select></div><div class="row"><span>Source</span>@@ -128,9 +130,11 @@</div><div class="section">- <label class="title">Files</label>+ <label class="title">Character</label>+ <div class="row"><span>Rig</span>+ <select id="character"><option value="xbot">X Bot</option><option value="michelle">Michelle</option><option value="soldier">Soldier</option></select></div><div class="row">- <button class="btn" id="btn-char">Load character…</button>+ <button class="btn" id="btn-char">Load Mixamo file…</button><button class="btn" id="btn-vid">Use a video file…</button></div><input type="file" id="file-char" accept=".glb,.gltf,.fbx" />diff --git a/server.mjs b/server.mjsindex 2ea810f..a6be96e 100644--- a/server.mjs+++ b/server.mjs@@ -21,11 +21,32 @@ const PORT = Number(process.env.PORT || 5177);const HQ_URL = process.env.HQ_URL || 'http://127.0.0.1:5178';if (!FAL_KEY) console.warn('[prompt2rig] FAL_KEY is not set — generation will fail. Put it in .env');-const MODELS = {- t2v: 'minimax/h3-max-turbo/text-to-video',- i2v: 'minimax/h3-max-turbo/image-to-video',- image: 'fal-ai/flux/schnell',+const VIDEO_MODELS = {+ // MiniMax H3 Max Turbo — faster than real time (~0.5s GPU for 5s @ 480P)+ h3: {+ t2v: 'minimax/h3-max-turbo/text-to-video',+ i2v: 'minimax/h3-max-turbo/image-to-video',+ input: (b, i2v) => ({+ duration: Math.min(15, Math.max(1, Number(b.duration) || 5)),+ resolution: ['480P', '768P', '1080P'].includes(b.resolution) ? b.resolution : '480P',+ prompt_expansion_mode: 'disabled',+ ...(i2v ? {} : { aspect_ratio: b.aspect_ratio || '16:9' }),+ }),+ },+ // FLUX 3 Max — higher fidelity, ~real time (5s @ 720p ≈ 4.5s GPU, ~9s wall)+ flux3: {+ t2v: 'blackforestlabs/flux-3-max/text-to-video',+ i2v: 'blackforestlabs/flux-3-max/image-to-video',+ input: (b, i2v) => ({+ duration: Math.min(20, Math.max(5, Math.round(Number(b.duration) || 5))),+ resolution: b.resolution === '1080P' ? '1080p' : '720p',+ generate_audio: false,+ reasoning_level: b.reasoning_level === 'high' ? 'high' : 'low',+ aspect_ratio: i2v ? 'auto' : (b.aspect_ratio || '16:9'),+ }),+ },};+const IMAGE_MODEL = 'fal-ai/flux/schnell';async function falRun(model, input) {const t0 = performance.now();@@ -69,24 +90,19 @@ const server = http.createServer(async (req, res) => {try {if (req.method === 'POST' && url.pathname === '/api/video') {const b = await readJson(req);+ const model = VIDEO_MODELS[b.model] ? b.model : 'h3';const mode = b.image_url ? 'i2v' : 't2v';- const input = {- prompt: String(b.prompt || '').slice(0, 4000),- duration: Math.min(15, Math.max(1, Number(b.duration) || 5)),- resolution: ['480P', '768P', '1080P'].includes(b.resolution) ? b.resolution : '480P',- prompt_expansion_mode: b.prompt_expansion_mode || 'disabled',- };+ const input = { prompt: String(b.prompt || '').slice(0, 4000), ...VIDEO_MODELS[model].input(b, mode === 'i2v') };if (b.seed != null && b.seed !== '') input.seed = Number(b.seed);if (mode === 'i2v') input.image_url = b.image_url;- else input.aspect_ratio = b.aspect_ratio || '16:9';- const out = await falRun(MODELS[mode], input);- console.log(`[video] ${mode} ${input.duration}s ${input.resolution} in ${out._elapsed.toFixed(1)}s (gpu ${out.timings?.inference?.toFixed?.(2) ?? '?'}s)`);- return send(res, 200, { mode, video_url: out.video?.url, timings: out.timings, elapsed: out._elapsed, expanded_prompt: out.expanded_prompt });+ const out = await falRun(VIDEO_MODELS[model][mode], input);+ console.log(`[video] ${model} ${mode} ${input.duration}s ${input.resolution} in ${out._elapsed.toFixed(1)}s (gpu ${out.timings?.inference?.toFixed?.(2) ?? '?'}s)`);+ return send(res, 200, { model, mode, video_url: out.video?.url, timings: out.timings, elapsed: out._elapsed });}if (req.method === 'POST' && url.pathname === '/api/image') {const b = await readJson(req);- const out = await falRun(MODELS.image, {+ const out = await falRun(IMAGE_MODEL, {prompt: String(b.prompt || '').slice(0, 4000),image_size: b.image_size || 'landscape_16_9',num_inference_steps: 4,