Skip to content
Python

feat: added mhr mesh support (inference & rendering).

mohamdevf209f71ab94d
diff --git a/.gitignore b/.gitignore
index 72fb3f4..b1ba90a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -35,6 +35,7 @@ images/
models/*.onnx
models/*.pt
models/*.pth
+models/mhr_assets/
!models/README.md
# Notebook checkpoints
diff --git a/README.md b/README.md
index 7184ef2..8306b89 100644
--- a/README.md
+++ b/README.md
@@ -10,7 +10,10 @@ SAM3D teacher's per-person 70-keypoint outputs from a single 224×224 crop.
InstantHMR ships as a single `.onnx` file. The demo pipeline pairs it with
**RF-DETR** for person detection — both stages are timed independently so
-you always know where the latency is going.
+you always know where the latency is going. Optionally, the demo decodes
+the raw `mhr_params` / `shape_params` outputs through Meta's
+**[MHR body model](https://github.com/facebookresearch/MHR)** to render a
+full dense body mesh in the Rerun viewer.
- **Inputs:** `image (N, 3, 224, 224)`, `cliff_cond (N, 3)`.
- **Outputs:** `mhr_params (204)`, `shape_params (45)`, `cam_trans (3)`,
@@ -30,39 +33,40 @@ you always know where the latency is going.
paths. Use `python install.py` which automatically pulls Torch
cu128 (>=2.7) on Blackwell GPUs.
-> **Note on the body mesh.** InstantHMR does **not** run the MHR mesh
-> decoder — rendering the full body mesh is expensive and we want the
-> default pipeline to stay lightweight. The raw `mhr_params` and
-> `shape_params` are still exposed on every `HMRPrediction` so you can
-> feed them into Meta's MHR TorchScript module from the original
-> [SAM 3D Body release](https://github.com/facebookresearch/sam-3d-body)
-> if you need a mesh.
-
## Install
```bash
git clone <this-repo> instanthmr
cd instanthmr
+```
+
+Choose the path that matches your needs:
-conda create -n instanthmr python=3.11 -y
-conda activate instanthmr
+| Goal | Python | Command |
+|------|--------|---------|
+| Pose estimation only (joints + skeleton) | 3.11 or 3.12 | `python install.py` |
+| + MHR body mesh rendering | **3.12** | `python install.py` then see [MHR body mesh](#mhr-body-mesh-rendering) |
+
+```bash
+# Pose estimation only (Python 3.11+):
+conda create -n instanthmr python=3.11 -y && conda activate instanthmr
# Picks the right torch + onnxruntime wheels for your machine:
-# - Linux + NVIDIA GPU → cu128 (Blackwell, RTX 50-series) or cu124 (Ada/Ampere/Hopper)
+# - Linux + NVIDIA GPU → cu128 (Blackwell/RTX 50) or cu124 (Ada/Ampere/Hopper)
# + onnxruntime-gpu + bundled CUDA / cuDNN runtime libs
# - macOS → stock torch + onnxruntime (CoreML EP included)
# - Linux without GPU → CPU torch + onnxruntime
python install.py
```
-`python install.py --dry-run` prints the pip commands without running
-them. `python install.py --force-cpu` skips GPU detection on Linux. If
-you'd rather manage wheels yourself, `pip install -r requirements.txt`
-still works — it gets the CPU torch + onnxruntime path.
+`python install.py --dry-run` prints the pip commands without running them.
+`python install.py --force-cpu` skips GPU detection on Linux.
+If you'd rather manage wheels yourself, `pip install -r requirements.txt`
+still works — it installs the CPU fallback path.
-The first run downloads the RF-DETR (medium) checkpoint into the rfdetr
-cache directory automatically. The InstantHMR ONNX weights are released
-separately on HuggingFace — see [Model weights](#model-weights) below.
+The first run downloads the RF-DETR (medium) checkpoint automatically.
+The InstantHMR ONNX weights are on HuggingFace — see
+[Model weights](#model-weights) below.
## Run the demo
@@ -81,8 +85,107 @@ python demo.py --camera 0
variant, confidence, max persons, frame skip, `.rrd` recording, …).
The demo opens a Rerun viewer with the source image + 2D skeleton, a live
-detector / HMR / total-latency plot, and the 3D scene with the predicted
-camera frustum.
+RF-DETR / InstantHMR / total-latency plot, and the 3D scene with the
+predicted camera frustum.
+
+## MHR body mesh rendering
+
+The demo can render a full dense body mesh for each detected person by
+running a forward pass through Meta's **MHR (Momentum Human Rig)** body
+model, using the `mhr_params (204,)` and `shape_params (45,)` outputs that
+InstantHMR already produces. Render time is displayed separately in the
+Rerun latency plot and in the console.
+
+### Prerequisites
+
+| Requirement | Notes |
+|-------------|-------|
+| **Python ≥ 3.12** | `pymomentum` has no Python 3.11 wheels; recreate your conda env if needed |
+| NVIDIA GPU / Apple Silicon | CPU works but is much slower |
+| CUDA toolkit | Must match your PyTorch wheel (Linux/Windows) |
+
+### Step 1 — Install in one pip call
+
+Always pass **both** requirement files to a **single** `pip install` call.
+Running them separately causes pip to first choose a CUDA-13 torch for
+`rfdetr`, then downgrade to CUDA-12 torch for `pymomentum`, leaving
+`torchvision` broken.
+
+```bash
+conda create -n instanthmr python=3.12 -y && conda activate instanthmr
+pip install -r requirements.txt -r requirements-mhr.txt
+```
+
+`requirements-mhr.txt` auto-selects the right package for your platform
+(`pymomentum-gpu` on Linux/Windows, `pymomentum-cpu` on macOS).
+CPU-only Linux users: see the comment at the top of `requirements-mhr.txt`.
+
+> **⚠ Wrong `pymomentum` on PyPI**
+> `pip install pymomentum` installs an **unrelated** legacy SMS library
+> (pyMomentum v0.1.x by MomentumAS) — not the Meta geometry library.
+> `requirements-mhr.txt` pulls in the correct package automatically.
+> If you accidentally installed the wrong one first:
+> ```bash
+> pip uninstall pymomentum
+> pip install -r requirements.txt -r requirements-mhr.txt
+> ```
+
+> **Torch/torchvision version mismatch?**
+> ```bash
+> python -c "import torch, torchvision; print(torch.__version__, torchvision.__version__)"
+> # If they differ, wipe both and reinstall together:
+> pip uninstall -y torch torchvision
+> pip install -r requirements.txt -r requirements-mhr.txt
+> ```
+
+### Step 2 — Download the body-model assets
+
+```bash
+curl -OL https://github.com/facebookresearch/MHR/releases/download/v1.0.0/assets.zip
+unzip assets.zip -d models/mhr_assets
+```
+
+This unpacks the body template, blend shapes, skinning weights, and pose
+correctives into `models/mhr_assets/`.
+
+### Step 3 — Run the demo
+
+```bash
+# Single image — LOD 0 (73 639 vertices, highest quality)
+python demo.py --image photo.jpg --mhr-assets models/mhr_assets --mhr-lod 0
+
+# Video file — LOD 1 (18 439 vertices, good quality)
+python demo.py --video clip.mp4 --mhr-assets models/mhr_assets --mhr-lod 1
+
+# Live camera — LOD 3 (4 899 vertices, real-time default)
+python demo.py --camera 0 --mhr-assets models/mhr_assets
+```
+
+### LOD reference
+
+| `--mhr-lod` | Vertices | Faces | Recommended use |
+|:-----------:|----------|-------|-----------------|
+| 0 | 73 639 | — | offline images, max quality |
+| 1 | 18 439 | — | video, powerful GPU |
+| 2 | 10 661 | — | video, mid-range GPU |
+| **3** | **4 899** | — | **live camera (default)** |
+| 4 | 2 461 | — | weak GPU / many persons |
+| 5 | 971 | — | borderline real-time |
+| 6 | 595 | — | debugging / stress test |
+
+### How it works
+
+```
+InstantHMR ONNX
+ └─ mhr_params (204,) ──┐
+ └─ shape_params (45,) ──┤─→ MHR.forward() → vertices (V, 3)
+ └─ cam_trans (3,) ──┘ + cam_trans → camera-space mesh
+```
+
+The 204-dim `mhr_params` encode joint rotations in 6-D representation
+(34 joints × 6); `shape_params (45,)` encode identity blend shapes
+(20 body + 20 head + 5 hand components). Facial expression is set to
+neutral (zeros) since InstantHMR does not regress it.
## Performance tuning
@@ -126,14 +229,27 @@ result = pipeline.predict(image_rgb)
for r in result.persons:
print(r.joints_3d_cam.shape) # (70, 3)
print(r.joints_2d.shape) # (70, 2)
- print(r.mhr_params.shape) # (204,) — for downstream MHR mesh decoding
- print(r.shape_params.shape) # (45,)
+ print(r.mhr_params.shape) # (204,) — MHR pose parameters
+ print(r.shape_params.shape) # (45,) — MHR identity shape parameters
```
The 70 keypoints follow the MHR70 ordering — see
[`instanthmr/skeleton.py`](instanthmr/skeleton.py) for joint names and the
canonical skeleton edge list.
+To run the MHR mesh decoder yourself:
+
+```python
+from instanthmr.mhr_renderer import MHRRenderer
+
+renderer = MHRRenderer(assets_folder="models/mhr_assets", device="cuda", lod=3)
+
+for r in result.persons:
+ verts_local = renderer.forward(r.mhr_params, r.shape_params) # (V, 3)
+ verts_cam = verts_local + r.cam_trans # camera space
+ faces = renderer.faces # (F, 3) int32
+```
+
## Model weights
The InstantHMR ONNX weights are hosted on HuggingFace:
@@ -145,12 +261,11 @@ via `--model`.
## Documentation
-- **[`docs/architecture.md`](docs/architecture.md)** — what the network
- looks like (RepViT backbone, 9-query decoder, CLIFF condition).
-- **[`docs/annotation.md`](docs/annotation.md)** — how the training data
- was generated with the SAM3D teacher.
-- **[`docs/training.md`](docs/training.md)** — how the released checkpoint
- was distilled and exported to ONNX.
+- **[`docs/architecture.md`](docs/architecture.md)** — network design
+ (RepViT backbone, 9-query decoder, CLIFF condition).
+- **[`docs/annotation.md`](docs/annotation.md)** — training data generation
+ with the SAM3D teacher.
+- **[`docs/training.md`](docs/training.md)** — distillation and ONNX export.
## License
diff --git a/demo.py b/demo.py
index 2021a7b..a46efe7 100755
--- a/demo.py
+++ b/demo.py
@@ -4,14 +4,26 @@
Detects persons in a video / image / live camera stream with RF-DETR, runs
InstantHMR on each, and visualises the results in Rerun: 3D joints, camera
pose as a pinhole frustum, the source image with the projected 2D skeleton
-drawn on top, and a live latency plot showing RF-DETR vs InstantHMR cost
-separately.
+drawn on top, and live latency plots for RF-DETR, InstantHMR, MHR mesh
+rendering, and total pipeline cost.
+
+Optionally renders a full MHR body mesh per person by decoding the
+``mhr_params`` and ``shape_params`` outputs through Meta's MHR model:
+
+ python demo.py --image photo.jpg \\
+ --mhr-assets models/mhr_assets
+
+See ``instanthmr/mhr_renderer.py`` for setup instructions (assets download,
+MHR package installation).
Usage
-----
- # Single image
+ # Single image (skeleton only)
python demo.py --image path/to/photo.jpg
+ # Single image with MHR mesh
+ python demo.py --image path/to/photo.jpg --mhr-assets models/mhr_assets
+
# Video file
python demo.py --video path/to/clip.mp4
@@ -92,6 +104,22 @@ def parse_args() -> argparse.Namespace:
"--no-spawn", action="store_true",
help="Don't spawn the Rerun viewer GUI (useful when only saving .rrd).",
)
+ p.add_argument(
+ "--mhr-assets", type=str, default=None, metavar="DIR",
+ help=(
+ "Path to the unpacked MHR assets folder. When provided, runs a "
+ "full MHR forward pass per person and renders the body mesh in "
+ "the 3D scene. See instanthmr/mhr_renderer.py for setup."
+ ),
+ )
+ p.add_argument(
+ "--mhr-lod", type=int, default=3, metavar="N",
+ choices=range(7),
+ help=(
+ "MHR level-of-detail (0=73 639 verts … 6=595 verts). "
+ "Default: 3 (4 899 verts)."
+ ),
+ )
return p.parse_args()
@@ -117,11 +145,51 @@ def build_pipeline(args: argparse.Namespace) -> PosePipeline:
)
-def build_visualizer(args: argparse.Namespace) -> RerunVisualizer:
+def build_mhr_renderer(args: argparse.Namespace):
+ if args.mhr_assets is None:
+ return None
+
+ assets_path = Path(args.mhr_assets)
+ if not assets_path.exists():
+ sys.stderr.write(
+ f"\n[error] MHR assets folder not found: {assets_path}\n"
+ " See instanthmr/mhr_renderer.py for download instructions.\n\n"
+ )
+ sys.exit(1)
+
+ try:
+ from instanthmr.mhr_renderer import MHRRenderer
+ except ImportError as e:
+ sys.stderr.write(
+ f"\n[error] Cannot import MHRRenderer: {e}\n"
+ "\n"
+ " MHR mesh rendering requires Python >= 3.12 and:\n"
+ " pip install -r requirements-mhr.txt\n"
+ "\n"
+ " If you see 'No module named pymomentum', make sure you installed\n"
+ " the CORRECT package (NOT the legacy pyMomentum SMS library):\n"
+ " pip uninstall pymomentum # remove wrong package\n"
+ " pip install pymomentum-gpu # NVIDIA GPU\n"
+ " pip install pymomentum-cpu # macOS / CPU-only\n"
+ "\n"
+ " See README.md §'MHR body mesh' for full setup instructions.\n\n"
+ )
+ sys.exit(1)
+
+ print(f"Loading MHR body model (LOD {args.mhr_lod}) from {assets_path} …")
+ return MHRRenderer(
+ assets_folder=assets_path,
+ device=args.device,
+ lod=args.mhr_lod,
+ )
+
+
+def build_visualizer(args: argparse.Namespace, mhr_renderer=None) -> RerunVisualizer:
return RerunVisualizer(
application_id="instanthmr_demo",
spawn_viewer=not args.no_spawn,
save_path=args.save_rrd,
+ mhr_renderer=mhr_renderer,
)
@@ -129,15 +197,18 @@ def _print_timings(
prefix: str,
detector_ms: float,
hmr_ms: float,
+ render_ms: float,
total_ms: float,
n_persons: int,
) -> None:
fps = 1000.0 / total_ms if total_ms > 0 else 0.0
+ render_part = f" MHR={render_ms:6.1f} ms" if render_ms > 0.0 else ""
print(
f"{prefix} "
f"persons={n_persons:>2d} "
f"RF-DETR={detector_ms:6.1f} ms "
- f"InstantHMR={hmr_ms:6.1f} ms "
+ f"InstantHMR={hmr_ms:6.1f} ms"
+ f"{render_part} "
f"total={total_ms:6.1f} ms ({fps:5.1f} fps)"
)
@@ -158,21 +229,26 @@ def run_image(args: argparse.Namespace) -> None:
rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
pipeline = build_pipeline(args)
- viz = build_visualizer(args)
+ mhr = build_mhr_renderer(args)
+ viz = build_visualizer(args, mhr)
# Warm-up: first call includes lazy CUDA / ORT initialisation, so a real
# latency reading is much more useful from the second call onwards.
_ = pipeline.predict(rgb)
result = pipeline.predict(rgb)
- viz.log_frame(
+ render_ms = viz.log_frame(
rgb, result.persons,
frame_idx=0, timestamp=0.0,
detector_ms=result.detector_ms,
hmr_ms=result.hmr_ms,
total_ms=result.total_ms,
)
- _print_timings("image:", result.detector_ms, result.hmr_ms, result.total_ms, len(result.persons))
+ _print_timings(
+ "image:",
+ result.detector_ms, result.hmr_ms, render_ms, result.total_ms,
+ len(result.persons),
+ )
def run_video(args: argparse.Namespace) -> None:
@@ -189,12 +265,14 @@ def run_video(args: argparse.Namespace) -> None:
print(f"video: {video_path.name}, {total} frames @ {fps:.1f} fps")
pipeline = build_pipeline(args)
- viz = build_visualizer(args)
+ mhr = build_mhr_renderer(args)
+ viz = build_visualizer(args, mhr)
frame_idx = 0
sent = 0
det_times: list[float] = []
hmr_times: list[float] = []
+ render_times: list[float] = []
tot_times: list[float] = []
try:
while True:
@@ -213,7 +291,7 @@ def run_video(args: argparse.Namespace) -> None:
hmr_times.append(result.hmr_ms)
tot_times.append(result.total_ms)
- viz.log_frame(
+ render_ms = viz.log_frame(
rgb, result.persons,
frame_idx=frame_idx,
timestamp=frame_idx / fps,
@@ -221,6 +299,7 @@ def run_video(args: argparse.Namespace) -> None:
hmr_ms=result.hmr_ms,
total_ms=result.total_ms,
)
+ render_times.append(render_ms)
sent += 1
frame_idx += 1
@@ -230,6 +309,7 @@ def run_video(args: argparse.Namespace) -> None:
f" frame {frame_idx:>5d}/{total:<5d}",
float(np.mean(det_times[-30:])),
float(np.mean(hmr_times[-30:])),
+ float(np.mean(render_times[-30:])),
float(np.mean(tot_times[-30:])),
len(result.persons),
)
@@ -241,11 +321,14 @@ def run_video(args: argparse.Namespace) -> None:
if tot_times:
# Skip the first frame from the average — it bears the warm-up cost.
warm = slice(1, None) if len(tot_times) > 1 else slice(None)
+ render_avg = float(np.mean(render_times[warm])) if render_times else 0.0
+ render_part = f" MHR={render_avg:.1f} ms" if render_avg > 0 else ""
print(
"\nsummary "
f"frames={sent:d} "
f"RF-DETR={float(np.mean(det_times[warm])):.1f} ms "
- f"InstantHMR={float(np.mean(hmr_times[warm])):.1f} ms "
+ f"InstantHMR={float(np.mean(hmr_times[warm])):.1f} ms"
+ f"{render_part} "
f"total={float(np.mean(tot_times[warm])):.1f} ms "
f"({1000.0 / float(np.mean(tot_times[warm])):.1f} fps)"
)
@@ -262,11 +345,13 @@ def run_camera(args: argparse.Namespace) -> None:
print(f"camera {args.camera}: {width}x{height} @ {fps:.1f} fps — Ctrl+C to stop")
pipeline = build_pipeline(args)
- viz = build_visualizer(args)
+ mhr = build_mhr_renderer(args)
+ viz = build_visualizer(args, mhr)
frame_idx = 0
det_times: list[float] = []
hmr_times: list[float] = []
+ render_times: list[float] = []
tot_times: list[float] = []
try:
while True:
@@ -281,7 +366,7 @@ def run_camera(args: argparse.Namespace) -> None:
hmr_times.append(result.hmr_ms)
tot_times.append(result.total_ms)
- viz.log_frame(
+ render_ms = viz.log_frame(
rgb, result.persons,
frame_idx=frame_idx,
timestamp=frame_idx / fps,
@@ -289,12 +374,14 @@ def run_camera(args: argparse.Namespace) -> None:
hmr_ms=result.hmr_ms,
total_ms=result.total_ms,
)
+ render_times.append(render_ms)
if frame_idx and frame_idx % 30 == 0:
_print_timings(
f" frame {frame_idx:>5d}",
float(np.mean(det_times[-30:])),
float(np.mean(hmr_times[-30:])),
+ float(np.mean(render_times[-30:])),
float(np.mean(tot_times[-30:])),
len(result.persons),
)
diff --git a/instanthmr/inference.py b/instanthmr/inference.py
index 5bbd62e..a2845c1 100644
--- a/instanthmr/inference.py
+++ b/instanthmr/inference.py
@@ -43,14 +43,12 @@ class HMRPrediction:
focal_length: (2,) [fx, fy] full-frame virtual focal = sqrt(H^2+W^2).
principal_point: (2,) [cx, cy] = full-frame centre.
image_shape: (H, W) of the source frame.
- mhr_params: (204,) raw MHR pose+scale parameters. InstantHMR does
- not run the MHR mesh decoder itself (it is resource-heavy and
- kept out of the default pipeline); these params are exposed so
- you can feed them into Meta's MHR TorchScript module from the
- original SAM 3D Body release if you want a body mesh —
- https://github.com/facebookresearch/sam-3d-body.
- shape_params: (45,) MHR identity shape blendshapes. Same use as
- ``mhr_params`` — pair them and run them through MHR.
+ mhr_params: (204,) MHR pose parameters (34 joints × 6-D rotation).
+ Pass to ``instanthmr.mhr_renderer.MHRRenderer.forward()``
+ together with ``shape_params`` to obtain a full body mesh.
+ shape_params: (45,) MHR identity blend-shape coefficients
+ (20 body + 20 head + 5 hand). Pair with ``mhr_params`` and
+ feed both into ``MHRRenderer.forward()``.
"""
bbox: np.ndarray
diff --git a/instanthmr/mhr_renderer.py b/instanthmr/mhr_renderer.py
new file mode 100644
index 0000000..fc6b79d
--- /dev/null
+++ b/instanthmr/mhr_renderer.py
@@ -0,0 +1,110 @@
+"""MHR (Momentum Human Rig) mesh decoder for InstantHMR.
+
+Takes the 204-dim pose parameters and 45-dim shape parameters produced by
+InstantHMR and decodes them into full body mesh vertices by running a forward
+pass through Meta's MHR body model.
+
+Quick setup (requires Python >= 3.12)::
+
+ pip install -r requirements-mhr.txt
+ curl -OL https://github.com/facebookresearch/MHR/releases/download/v1.0.0/assets.zip
+ unzip assets.zip -d models/mhr_assets
+ python demo.py --image photo.jpg --mhr-assets models/mhr_assets
+
+WARNING — wrong pymomentum package on PyPI
+------------------------------------------
+``pip install pymomentum`` installs an **unrelated** legacy SMS library
+(pyMomentum v0.1.x by MomentumAS). The correct Meta package is::
+
+ pip install pymomentum-gpu # NVIDIA GPU
+ pip install pymomentum-cpu # macOS / CPU-only
+
+See README.md §'MHR body mesh' for the full installation walkthrough.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import numpy as np
+
+
+class MHRRenderer:
+ """Loads and caches the MHR body model, runs per-person forward passes.
+
+ Args:
+ assets_folder: path to the unpacked MHR assets folder (the directory
+ that contains the blend-shape files, template mesh, etc.).
+ device: ``'cuda'``, ``'cpu'``, or a ``torch.device``.
+ lod: level-of-detail mesh resolution:
+ 0 = 73 639 verts (highest), 3 = 4 899, 6 = 595 (lowest).
+ LOD 3 is a good default for live demos.
+ """
+
+ def __init__(
+ self,
+ assets_folder: str | Path = "models/mhr_assets",
+ device: str = "cuda",
+ lod: int = 3,
+ ) -> None:
+ import torch
+ from mhr.mhr import MHR
+
+ self._torch = torch
+ self._device = torch.device(device)
+ self._model = MHR.from_files(
+ folder=Path(assets_folder),
+ device=self._device,
+ lod=lod,
+ )
+
+ faces_raw = self._model.character.mesh.faces
+ if isinstance(faces_raw, torch.Tensor):
+ self._faces: np.ndarray = faces_raw.cpu().numpy().astype(np.int32)
+ else:
+ self._faces = np.asarray(faces_raw, dtype=np.int32)
+
+ @property
+ def faces(self) -> np.ndarray:
+ """Triangle face indices ``(F, 3)`` int32, shared across all frames."""
+ return self._faces
+
+ def forward(
+ self,
+ mhr_params: np.ndarray,
+ shape_params: np.ndarray,
+ ) -> np.ndarray:
+ """Decode one person's parameters into body-space mesh vertices.
+
+ Args:
+ mhr_params: ``(204,)`` float32 — pose parameters from InstantHMR.
+ shape_params: ``(45,)`` float32 — shape parameters from InstantHMR.
+
+ Returns:
+ ``(V, 3)`` float32 vertices in **body-centred** space (metres,
+ Y-down). Add the person's ``cam_trans`` to move to camera space.
+ """
+ torch = self._torch
+ identity = torch.from_numpy(shape_params).unsqueeze(0).to(self._device) # (1, 45)
+ pose = torch.from_numpy(mhr_params).unsqueeze(0).to(self._device) # (1, 204)
+ # MHR also accepts facial expression coefficients (72-D); InstantHMR
+ # does not predict them, so we pass zeros (neutral expression).
+ expr = torch.zeros(1, 72, dtype=torch.float32, device=self._device)
+
+ with torch.no_grad():
+ verts, _ = self._model(identity, pose, expr)
+
+ v = verts[0].cpu().numpy().astype(np.float32) # (V, 3)
+
+ # --- Coordinate-space alignment -----------------------------------
+ # MHR body template (FBX) is Y-up right-handed and in centimetres.
+ # InstantHMR / SAM3D uses Y-down right-handed and metres.
+ #
+ # 1. cm → m
+ v /= 100.0
+ # 2. Y-up → Y-down: negate Y.
+ # Negate Z as well to preserve right-handedness after the Y flip.
+ v[:, 1] *= -1.0
+ v[:, 2] *= -1.0
+
+ return v
diff --git a/instanthmr/visualizer.py b/instanthmr/visualizer.py
index 92f7ffb..e51587e 100644
--- a/instanthmr/visualizer.py
+++ b/instanthmr/visualizer.py
@@ -2,12 +2,15 @@
Logs, per frame:
- ``camera/image`` : annotated RGB frame (2D skeleton + bbox + a
- text panel showing detector / HMR / total ms).
+ text panel showing detector / HMR / render / total ms).
- ``camera`` : pinhole intrinsics so the image becomes a
proper frustum in the 3D scene.
- ``world/persons/...`` : 3D joints + skeleton lines per person.
+ - ``world/persons/.../mesh``: full MHR body mesh when an MHRRenderer is
+ provided (vertices from a true MHR forward pass).
- ``timing/detector_ms`` : scalar plot of RF-DETR latency.
- ``timing/hmr_ms`` : scalar plot of InstantHMR latency.
+ - ``timing/render_ms`` : scalar plot of MHR mesh rendering latency.
- ``timing/total_ms`` : scalar plot of full pipeline latency.
- ``timing/fps`` : scalar plot of effective frames-per-second.
@@ -17,7 +20,8 @@ on construction (pass ``spawn_viewer=False`` to suppress).
from __future__ import annotations
-from typing import Optional, Sequence
+import time
+from typing import TYPE_CHECKING, Optional, Sequence
import cv2
import numpy as np
@@ -25,6 +29,9 @@ import numpy as np
from .inference import HMRPrediction
from .skeleton import edges_for
+if TYPE_CHECKING:
+ from .mhr_renderer import MHRRenderer
+
class RerunVisualizer:
def __init__(
@@ -32,10 +39,12 @@ class RerunVisualizer:
application_id: str = "instanthmr_demo",
spawn_viewer: bool = True,
save_path: str | None = None,
+ mhr_renderer: "MHRRenderer | None" = None,
):
import rerun as rr
self._rr = rr
+ self._mhr_renderer = mhr_renderer
rr.init(application_id, spawn=spawn_viewer)
# SAM3D / MHR convention: right-handed, Y-down camera frame.
@@ -60,7 +69,7 @@ class RerunVisualizer:
detector_ms: Optional[float] = None,
hmr_ms: Optional[float] = None,
total_ms: Optional[float] = None,
- ) -> None:
+ ) -> float:
"""Log one frame of the pipeline to Rerun.
Args:
@@ -72,6 +81,10 @@ class RerunVisualizer:
hmr_ms: InstantHMR latency for this frame, summed across
persons (ms).
total_ms: full pipeline latency for this frame (ms).
+
+ Returns:
+ render_ms: time spent running MHR forward passes + Rerun mesh
+ logging for this frame (0.0 when no MHRRenderer is set).
"""
rr = self._rr
rr.set_time("frame", sequence=frame_idx)
@@ -98,16 +111,27 @@ class RerunVisualizer:
# Always log the image — even when no person was detected — so the
# viewer keeps showing the camera feed and the timing HUD.
+ render_ms = 0.0
if not persons:
- blank = self._draw_overlay(image_rgb, [], detector_ms, hmr_ms, total_ms)
+ blank = self._draw_overlay(image_rgb, [], detector_ms, hmr_ms, render_ms, total_ms)
rr.log("camera/image", rr.Image(blank))
- return
+ rr.log("timing/render_ms", rr.Scalars(0.0))
+ return render_ms
+
+ # --- MHR mesh rendering (timed) ----------------------------------
+ if self._mhr_renderer is not None:
+ t0 = time.perf_counter()
+ for idx, person in enumerate(persons):
+ self._log_person_mesh(idx, person)
+ render_ms = (time.perf_counter() - t0) * 1000.0
+
+ rr.log("timing/render_ms", rr.Scalars(render_ms))
for idx, person in enumerate(persons):
self._log_person(idx, person)
annotated = self._draw_overlay(
- image_rgb, persons, detector_ms, hmr_ms, total_ms,
+ image_rgb, persons, detector_ms, hmr_ms, render_ms, total_ms,
)
rr.log("camera/image", rr.Image(annotated))
@@ -134,6 +158,8 @@ class RerunVisualizer:
self._send_blueprint(first)
self._blueprint_sent = True
+ return render_ms
+
# ------------------------------------------------------------------
# Helpers
# ------------------------------------------------------------------
@@ -159,12 +185,39 @@ class RerunVisualizer:
rr.LineStrips3D(lines, colors=[255, 230, 0], radii=0.004),
)
+ def _log_person_mesh(self, idx: int, person: HMRPrediction) -> None:
+ """Run MHR forward pass and log the resulting mesh vertices via Rerun."""
+ assert self._mhr_renderer is not None
+
+ # Decode mhr_params + shape_params → body-space vertices.
+ verts_local = self._mhr_renderer.forward(
+ person.mhr_params, person.shape_params,
+ ) # (V, 3), body-centred
+
+ # Shift from body space to camera space using InstantHMR's cam_trans.
+ verts_cam = verts_local + person.cam_trans # (V, 3) broadcast
+
+ faces = self._mhr_renderer.faces # (F, 3) int32, shared across frames
+
+ n = len(verts_cam)
+ skin = np.tile(np.array([[200, 160, 130, 140]], dtype=np.uint8), (n, 1))
+
+ self._rr.log(
+ f"world/persons/person_{idx}/mesh",
+ self._rr.Mesh3D(
+ vertex_positions=verts_cam,
+ triangle_indices=faces,
+ vertex_colors=skin,
+ ),
+ )
+
@staticmethod
def _draw_overlay(
image_rgb: np.ndarray,
persons: Sequence[HMRPrediction],
detector_ms: Optional[float],
hmr_ms: Optional[float],
+ render_ms: float,
total_ms: Optional[float],
) -> np.ndarray:
"""Draw per-person bbox + 2D skeleton + a timing HUD onto the frame."""
@@ -201,12 +254,14 @@ class RerunVisualizer:
# ---- Timing HUD (top-left, semi-transparent black panel) ----
lines: list[str] = []
if detector_ms is not None:
- lines.append(f"RF-DETR : {detector_ms:6.1f} ms")
+ lines.append(f"RF-DETR : {detector_ms:6.1f} ms")
if hmr_ms is not None:
lines.append(f"InstantHMR: {hmr_ms:6.1f} ms")
+ if render_ms > 0.0:
+ lines.append(f"MHR mesh : {render_ms:6.1f} ms")
if total_ms is not None:
fps = 1000.0 / total_ms if total_ms > 0 else 0.0
- lines.append(f"Total : {total_ms:6.1f} ms ({fps:5.1f} fps)")
+ lines.append(f"Total : {total_ms:6.1f} ms ({fps:5.1f} fps)")
if lines:
font = cv2.FONT_HERSHEY_SIMPLEX
scale = max(0.5, min(w, h) / 1500.0)
diff --git a/requirements-mhr.txt b/requirements-mhr.txt
new file mode 100644
index 0000000..8951b7e
--- /dev/null
+++ b/requirements-mhr.txt
@@ -0,0 +1,43 @@
+# MHR body mesh rendering — install TOGETHER with requirements.txt
+#
+# IMPORTANT: always install both files in a single pip call so the resolver
+# can pick a torch version compatible with rfdetr, torchvision, AND
+# pymomentum-gpu at once. Installing them separately causes pip to first
+# choose a torch version for rfdetr and then downgrade it for pymomentum,
+# leaving torchvision broken.
+#
+# pip install -r requirements.txt -r requirements-mhr.txt
+#
+# Prerequisites
+# -------------
+# * Python >= 3.12 (pymomentum has no Python 3.11 wheels; recreate the
+# conda env if needed: conda create -n instanthmr python=3.12 -y)
+# * CUDA toolkit on the system path (recommended for real-time use)
+#
+# After installing, also download the MHR body-model assets:
+#
+# curl -OL https://github.com/facebookresearch/MHR/releases/download/v1.0.0/assets.zip
+# unzip assets.zip -d models/mhr_assets
+#
+# Then run:
+#
+# python demo.py --image photo.jpg --mhr-assets models/mhr_assets
+
+# pymomentum — Meta's momentum geometry library (C++ Python bindings).
+# This package brings its own torch constraint; do NOT add a separate
+# torch>=X line here or pip will resolve torch twice and downgrade it.
+#
+# WARNING: "pip install pymomentum" installs an UNRELATED legacy SMS library
+# (pyMomentum v0.1.x by MomentumAS). The correct package names are below.
+#
+# Platform markers handle the selection automatically:
+# Linux / Windows → pymomentum-gpu (NVIDIA GPU)
+# macOS → pymomentum-cpu (Apple Silicon / Intel)
+pymomentum-gpu ; platform_system != "Darwin"
+pymomentum-cpu ; platform_system == "Darwin"
+#
+# CPU-only Linux / Windows: comment out pymomentum-gpu above and uncomment:
+# pymomentum-cpu ; platform_system != "Darwin"
+
+# MHR body model (Meta's Momentum Human Rig)
+mhr @ git+https://github.com/facebookresearch/MHR.git
diff --git a/requirements.txt b/requirements.txt
index 3c3f2ba..1644ca1 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -21,6 +21,10 @@ torchvision
# safe fallback.
onnxruntime>=1.19
+# ----- MHR body mesh rendering -----
+# Optional. Requires Python >= 3.12 and a separate setup step.
+# See requirements-mhr.txt and the "MHR body mesh" section of README.md.
+
# ----- training / annotation (optional, for reproduction) -----
# timm>=1.0
# tqdm