Python
feat: added mhr mesh support (inference & rendering).
mohamdevf209f71ab94d
diff --git a/.gitignore b/.gitignoreindex 72fb3f4..b1ba90a 100644--- a/.gitignore+++ b/.gitignore@@ -35,6 +35,7 @@ images/models/*.onnxmodels/*.ptmodels/*.pth+models/mhr_assets/!models/README.md# Notebook checkpointsdiff --git a/README.md b/README.mdindex 7184ef2..8306b89 100644--- a/README.md+++ b/README.md@@ -10,7 +10,10 @@ SAM3D teacher's per-person 70-keypoint outputs from a single 224×224 crop.InstantHMR ships as a single `.onnx` file. The demo pipeline pairs it with**RF-DETR** for person detection — both stages are timed independently so-you always know where the latency is going.+you always know where the latency is going. Optionally, the demo decodes+the raw `mhr_params` / `shape_params` outputs through Meta's+**[MHR body model](https://github.com/facebookresearch/MHR)** to render a+full dense body mesh in the Rerun viewer.- **Inputs:** `image (N, 3, 224, 224)`, `cliff_cond (N, 3)`.- **Outputs:** `mhr_params (204)`, `shape_params (45)`, `cam_trans (3)`,@@ -30,39 +33,40 @@ you always know where the latency is going.paths. Use `python install.py` which automatically pulls Torchcu128 (>=2.7) on Blackwell GPUs.-> **Note on the body mesh.** InstantHMR does **not** run the MHR mesh-> decoder — rendering the full body mesh is expensive and we want the-> default pipeline to stay lightweight. The raw `mhr_params` and-> `shape_params` are still exposed on every `HMRPrediction` so you can-> feed them into Meta's MHR TorchScript module from the original-> [SAM 3D Body release](https://github.com/facebookresearch/sam-3d-body)-> if you need a mesh.-## Install```bashgit clone <this-repo> instanthmrcd instanthmr+```++Choose the path that matches your needs:-conda create -n instanthmr python=3.11 -y-conda activate instanthmr+| Goal | Python | Command |+|------|--------|---------|+| Pose estimation only (joints + skeleton) | 3.11 or 3.12 | `python install.py` |+| + MHR body mesh rendering | **3.12** | `python install.py` then see [MHR body mesh](#mhr-body-mesh-rendering) |++```bash+# Pose estimation only (Python 3.11+):+conda create -n instanthmr python=3.11 -y && conda activate instanthmr# Picks the right torch + onnxruntime wheels for your machine:-# - Linux + NVIDIA GPU → cu128 (Blackwell, RTX 50-series) or cu124 (Ada/Ampere/Hopper)+# - Linux + NVIDIA GPU → cu128 (Blackwell/RTX 50) or cu124 (Ada/Ampere/Hopper)# + onnxruntime-gpu + bundled CUDA / cuDNN runtime libs# - macOS → stock torch + onnxruntime (CoreML EP included)# - Linux without GPU → CPU torch + onnxruntimepython install.py```-`python install.py --dry-run` prints the pip commands without running-them. `python install.py --force-cpu` skips GPU detection on Linux. If-you'd rather manage wheels yourself, `pip install -r requirements.txt`-still works — it gets the CPU torch + onnxruntime path.+`python install.py --dry-run` prints the pip commands without running them.+`python install.py --force-cpu` skips GPU detection on Linux.+If you'd rather manage wheels yourself, `pip install -r requirements.txt`+still works — it installs the CPU fallback path.-The first run downloads the RF-DETR (medium) checkpoint into the rfdetr-cache directory automatically. The InstantHMR ONNX weights are released-separately on HuggingFace — see [Model weights](#model-weights) below.+The first run downloads the RF-DETR (medium) checkpoint automatically.+The InstantHMR ONNX weights are on HuggingFace — see+[Model weights](#model-weights) below.## Run the demo@@ -81,8 +85,107 @@ python demo.py --camera 0variant, confidence, max persons, frame skip, `.rrd` recording, …).The demo opens a Rerun viewer with the source image + 2D skeleton, a live-detector / HMR / total-latency plot, and the 3D scene with the predicted-camera frustum.+RF-DETR / InstantHMR / total-latency plot, and the 3D scene with the+predicted camera frustum.++## MHR body mesh rendering++The demo can render a full dense body mesh for each detected person by+running a forward pass through Meta's **MHR (Momentum Human Rig)** body+model, using the `mhr_params (204,)` and `shape_params (45,)` outputs that+InstantHMR already produces. Render time is displayed separately in the+Rerun latency plot and in the console.++### Prerequisites++| Requirement | Notes |+|-------------|-------|+| **Python ≥ 3.12** | `pymomentum` has no Python 3.11 wheels; recreate your conda env if needed |+| NVIDIA GPU / Apple Silicon | CPU works but is much slower |+| CUDA toolkit | Must match your PyTorch wheel (Linux/Windows) |++### Step 1 — Install in one pip call++Always pass **both** requirement files to a **single** `pip install` call.+Running them separately causes pip to first choose a CUDA-13 torch for+`rfdetr`, then downgrade to CUDA-12 torch for `pymomentum`, leaving+`torchvision` broken.++```bash+conda create -n instanthmr python=3.12 -y && conda activate instanthmr+pip install -r requirements.txt -r requirements-mhr.txt+```++`requirements-mhr.txt` auto-selects the right package for your platform+(`pymomentum-gpu` on Linux/Windows, `pymomentum-cpu` on macOS).+CPU-only Linux users: see the comment at the top of `requirements-mhr.txt`.++> **⚠ Wrong `pymomentum` on PyPI**+> `pip install pymomentum` installs an **unrelated** legacy SMS library+> (pyMomentum v0.1.x by MomentumAS) — not the Meta geometry library.+> `requirements-mhr.txt` pulls in the correct package automatically.+> If you accidentally installed the wrong one first:+> ```bash+> pip uninstall pymomentum+> pip install -r requirements.txt -r requirements-mhr.txt+> ```++> **Torch/torchvision version mismatch?**+> ```bash+> python -c "import torch, torchvision; print(torch.__version__, torchvision.__version__)"+> # If they differ, wipe both and reinstall together:+> pip uninstall -y torch torchvision+> pip install -r requirements.txt -r requirements-mhr.txt+> ```++### Step 2 — Download the body-model assets++```bash+curl -OL https://github.com/facebookresearch/MHR/releases/download/v1.0.0/assets.zip+unzip assets.zip -d models/mhr_assets+```++This unpacks the body template, blend shapes, skinning weights, and pose+correctives into `models/mhr_assets/`.++### Step 3 — Run the demo++```bash+# Single image — LOD 0 (73 639 vertices, highest quality)+python demo.py --image photo.jpg --mhr-assets models/mhr_assets --mhr-lod 0++# Video file — LOD 1 (18 439 vertices, good quality)+python demo.py --video clip.mp4 --mhr-assets models/mhr_assets --mhr-lod 1++# Live camera — LOD 3 (4 899 vertices, real-time default)+python demo.py --camera 0 --mhr-assets models/mhr_assets+```++### LOD reference++| `--mhr-lod` | Vertices | Faces | Recommended use |+|:-----------:|----------|-------|-----------------|+| 0 | 73 639 | — | offline images, max quality |+| 1 | 18 439 | — | video, powerful GPU |+| 2 | 10 661 | — | video, mid-range GPU |+| **3** | **4 899** | — | **live camera (default)** |+| 4 | 2 461 | — | weak GPU / many persons |+| 5 | 971 | — | borderline real-time |+| 6 | 595 | — | debugging / stress test |++### How it works++```+InstantHMR ONNX+ └─ mhr_params (204,) ──┐+ └─ shape_params (45,) ──┤─→ MHR.forward() → vertices (V, 3)+ └─ cam_trans (3,) ──┘ + cam_trans → camera-space mesh+```++The 204-dim `mhr_params` encode joint rotations in 6-D representation+(34 joints × 6); `shape_params (45,)` encode identity blend shapes+(20 body + 20 head + 5 hand components). Facial expression is set to+neutral (zeros) since InstantHMR does not regress it.## Performance tuning@@ -126,14 +229,27 @@ result = pipeline.predict(image_rgb)for r in result.persons:print(r.joints_3d_cam.shape) # (70, 3)print(r.joints_2d.shape) # (70, 2)- print(r.mhr_params.shape) # (204,) — for downstream MHR mesh decoding- print(r.shape_params.shape) # (45,)+ print(r.mhr_params.shape) # (204,) — MHR pose parameters+ print(r.shape_params.shape) # (45,) — MHR identity shape parameters```The 70 keypoints follow the MHR70 ordering — see[`instanthmr/skeleton.py`](instanthmr/skeleton.py) for joint names and thecanonical skeleton edge list.+To run the MHR mesh decoder yourself:++```python+from instanthmr.mhr_renderer import MHRRenderer++renderer = MHRRenderer(assets_folder="models/mhr_assets", device="cuda", lod=3)++for r in result.persons:+ verts_local = renderer.forward(r.mhr_params, r.shape_params) # (V, 3)+ verts_cam = verts_local + r.cam_trans # camera space+ faces = renderer.faces # (F, 3) int32+```+## Model weightsThe InstantHMR ONNX weights are hosted on HuggingFace:@@ -145,12 +261,11 @@ via `--model`.## Documentation-- **[`docs/architecture.md`](docs/architecture.md)** — what the network- looks like (RepViT backbone, 9-query decoder, CLIFF condition).-- **[`docs/annotation.md`](docs/annotation.md)** — how the training data- was generated with the SAM3D teacher.-- **[`docs/training.md`](docs/training.md)** — how the released checkpoint- was distilled and exported to ONNX.+- **[`docs/architecture.md`](docs/architecture.md)** — network design+ (RepViT backbone, 9-query decoder, CLIFF condition).+- **[`docs/annotation.md`](docs/annotation.md)** — training data generation+ with the SAM3D teacher.+- **[`docs/training.md`](docs/training.md)** — distillation and ONNX export.## Licensediff --git a/demo.py b/demo.pyindex 2021a7b..a46efe7 100755--- a/demo.py+++ b/demo.py@@ -4,14 +4,26 @@Detects persons in a video / image / live camera stream with RF-DETR, runsInstantHMR on each, and visualises the results in Rerun: 3D joints, camerapose as a pinhole frustum, the source image with the projected 2D skeleton-drawn on top, and a live latency plot showing RF-DETR vs InstantHMR cost-separately.+drawn on top, and live latency plots for RF-DETR, InstantHMR, MHR mesh+rendering, and total pipeline cost.++Optionally renders a full MHR body mesh per person by decoding the+``mhr_params`` and ``shape_params`` outputs through Meta's MHR model:++ python demo.py --image photo.jpg \\+ --mhr-assets models/mhr_assets++See ``instanthmr/mhr_renderer.py`` for setup instructions (assets download,+MHR package installation).Usage------ # Single image+ # Single image (skeleton only)python demo.py --image path/to/photo.jpg+ # Single image with MHR mesh+ python demo.py --image path/to/photo.jpg --mhr-assets models/mhr_assets+# Video filepython demo.py --video path/to/clip.mp4@@ -92,6 +104,22 @@ def parse_args() -> argparse.Namespace:"--no-spawn", action="store_true",help="Don't spawn the Rerun viewer GUI (useful when only saving .rrd).",)+ p.add_argument(+ "--mhr-assets", type=str, default=None, metavar="DIR",+ help=(+ "Path to the unpacked MHR assets folder. When provided, runs a "+ "full MHR forward pass per person and renders the body mesh in "+ "the 3D scene. See instanthmr/mhr_renderer.py for setup."+ ),+ )+ p.add_argument(+ "--mhr-lod", type=int, default=3, metavar="N",+ choices=range(7),+ help=(+ "MHR level-of-detail (0=73 639 verts … 6=595 verts). "+ "Default: 3 (4 899 verts)."+ ),+ )return p.parse_args()@@ -117,11 +145,51 @@ def build_pipeline(args: argparse.Namespace) -> PosePipeline:)-def build_visualizer(args: argparse.Namespace) -> RerunVisualizer:+def build_mhr_renderer(args: argparse.Namespace):+ if args.mhr_assets is None:+ return None++ assets_path = Path(args.mhr_assets)+ if not assets_path.exists():+ sys.stderr.write(+ f"\n[error] MHR assets folder not found: {assets_path}\n"+ " See instanthmr/mhr_renderer.py for download instructions.\n\n"+ )+ sys.exit(1)++ try:+ from instanthmr.mhr_renderer import MHRRenderer+ except ImportError as e:+ sys.stderr.write(+ f"\n[error] Cannot import MHRRenderer: {e}\n"+ "\n"+ " MHR mesh rendering requires Python >= 3.12 and:\n"+ " pip install -r requirements-mhr.txt\n"+ "\n"+ " If you see 'No module named pymomentum', make sure you installed\n"+ " the CORRECT package (NOT the legacy pyMomentum SMS library):\n"+ " pip uninstall pymomentum # remove wrong package\n"+ " pip install pymomentum-gpu # NVIDIA GPU\n"+ " pip install pymomentum-cpu # macOS / CPU-only\n"+ "\n"+ " See README.md §'MHR body mesh' for full setup instructions.\n\n"+ )+ sys.exit(1)++ print(f"Loading MHR body model (LOD {args.mhr_lod}) from {assets_path} …")+ return MHRRenderer(+ assets_folder=assets_path,+ device=args.device,+ lod=args.mhr_lod,+ )+++def build_visualizer(args: argparse.Namespace, mhr_renderer=None) -> RerunVisualizer:return RerunVisualizer(application_id="instanthmr_demo",spawn_viewer=not args.no_spawn,save_path=args.save_rrd,+ mhr_renderer=mhr_renderer,)@@ -129,15 +197,18 @@ def _print_timings(prefix: str,detector_ms: float,hmr_ms: float,+ render_ms: float,total_ms: float,n_persons: int,) -> None:fps = 1000.0 / total_ms if total_ms > 0 else 0.0+ render_part = f" MHR={render_ms:6.1f} ms" if render_ms > 0.0 else ""print(f"{prefix} "f"persons={n_persons:>2d} "f"RF-DETR={detector_ms:6.1f} ms "- f"InstantHMR={hmr_ms:6.1f} ms "+ f"InstantHMR={hmr_ms:6.1f} ms"+ f"{render_part} "f"total={total_ms:6.1f} ms ({fps:5.1f} fps)")@@ -158,21 +229,26 @@ def run_image(args: argparse.Namespace) -> None:rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)pipeline = build_pipeline(args)- viz = build_visualizer(args)+ mhr = build_mhr_renderer(args)+ viz = build_visualizer(args, mhr)# Warm-up: first call includes lazy CUDA / ORT initialisation, so a real# latency reading is much more useful from the second call onwards._ = pipeline.predict(rgb)result = pipeline.predict(rgb)- viz.log_frame(+ render_ms = viz.log_frame(rgb, result.persons,frame_idx=0, timestamp=0.0,detector_ms=result.detector_ms,hmr_ms=result.hmr_ms,total_ms=result.total_ms,)- _print_timings("image:", result.detector_ms, result.hmr_ms, result.total_ms, len(result.persons))+ _print_timings(+ "image:",+ result.detector_ms, result.hmr_ms, render_ms, result.total_ms,+ len(result.persons),+ )def run_video(args: argparse.Namespace) -> None:@@ -189,12 +265,14 @@ def run_video(args: argparse.Namespace) -> None:print(f"video: {video_path.name}, {total} frames @ {fps:.1f} fps")pipeline = build_pipeline(args)- viz = build_visualizer(args)+ mhr = build_mhr_renderer(args)+ viz = build_visualizer(args, mhr)frame_idx = 0sent = 0det_times: list[float] = []hmr_times: list[float] = []+ render_times: list[float] = []tot_times: list[float] = []try:while True:@@ -213,7 +291,7 @@ def run_video(args: argparse.Namespace) -> None:hmr_times.append(result.hmr_ms)tot_times.append(result.total_ms)- viz.log_frame(+ render_ms = viz.log_frame(rgb, result.persons,frame_idx=frame_idx,timestamp=frame_idx / fps,@@ -221,6 +299,7 @@ def run_video(args: argparse.Namespace) -> None:hmr_ms=result.hmr_ms,total_ms=result.total_ms,)+ render_times.append(render_ms)sent += 1frame_idx += 1@@ -230,6 +309,7 @@ def run_video(args: argparse.Namespace) -> None:f" frame {frame_idx:>5d}/{total:<5d}",float(np.mean(det_times[-30:])),float(np.mean(hmr_times[-30:])),+ float(np.mean(render_times[-30:])),float(np.mean(tot_times[-30:])),len(result.persons),)@@ -241,11 +321,14 @@ def run_video(args: argparse.Namespace) -> None:if tot_times:# Skip the first frame from the average — it bears the warm-up cost.warm = slice(1, None) if len(tot_times) > 1 else slice(None)+ render_avg = float(np.mean(render_times[warm])) if render_times else 0.0+ render_part = f" MHR={render_avg:.1f} ms" if render_avg > 0 else ""print("\nsummary "f"frames={sent:d} "f"RF-DETR={float(np.mean(det_times[warm])):.1f} ms "- f"InstantHMR={float(np.mean(hmr_times[warm])):.1f} ms "+ f"InstantHMR={float(np.mean(hmr_times[warm])):.1f} ms"+ f"{render_part} "f"total={float(np.mean(tot_times[warm])):.1f} ms "f"({1000.0 / float(np.mean(tot_times[warm])):.1f} fps)")@@ -262,11 +345,13 @@ def run_camera(args: argparse.Namespace) -> None:print(f"camera {args.camera}: {width}x{height} @ {fps:.1f} fps — Ctrl+C to stop")pipeline = build_pipeline(args)- viz = build_visualizer(args)+ mhr = build_mhr_renderer(args)+ viz = build_visualizer(args, mhr)frame_idx = 0det_times: list[float] = []hmr_times: list[float] = []+ render_times: list[float] = []tot_times: list[float] = []try:while True:@@ -281,7 +366,7 @@ def run_camera(args: argparse.Namespace) -> None:hmr_times.append(result.hmr_ms)tot_times.append(result.total_ms)- viz.log_frame(+ render_ms = viz.log_frame(rgb, result.persons,frame_idx=frame_idx,timestamp=frame_idx / fps,@@ -289,12 +374,14 @@ def run_camera(args: argparse.Namespace) -> None:hmr_ms=result.hmr_ms,total_ms=result.total_ms,)+ render_times.append(render_ms)if frame_idx and frame_idx % 30 == 0:_print_timings(f" frame {frame_idx:>5d}",float(np.mean(det_times[-30:])),float(np.mean(hmr_times[-30:])),+ float(np.mean(render_times[-30:])),float(np.mean(tot_times[-30:])),len(result.persons),)diff --git a/instanthmr/inference.py b/instanthmr/inference.pyindex 5bbd62e..a2845c1 100644--- a/instanthmr/inference.py+++ b/instanthmr/inference.py@@ -43,14 +43,12 @@ class HMRPrediction:focal_length: (2,) [fx, fy] full-frame virtual focal = sqrt(H^2+W^2).principal_point: (2,) [cx, cy] = full-frame centre.image_shape: (H, W) of the source frame.- mhr_params: (204,) raw MHR pose+scale parameters. InstantHMR does- not run the MHR mesh decoder itself (it is resource-heavy and- kept out of the default pipeline); these params are exposed so- you can feed them into Meta's MHR TorchScript module from the- original SAM 3D Body release if you want a body mesh —- https://github.com/facebookresearch/sam-3d-body.- shape_params: (45,) MHR identity shape blendshapes. Same use as- ``mhr_params`` — pair them and run them through MHR.+ mhr_params: (204,) MHR pose parameters (34 joints × 6-D rotation).+ Pass to ``instanthmr.mhr_renderer.MHRRenderer.forward()``+ together with ``shape_params`` to obtain a full body mesh.+ shape_params: (45,) MHR identity blend-shape coefficients+ (20 body + 20 head + 5 hand). Pair with ``mhr_params`` and+ feed both into ``MHRRenderer.forward()``."""bbox: np.ndarraydiff --git a/instanthmr/mhr_renderer.py b/instanthmr/mhr_renderer.pynew file mode 100644index 0000000..fc6b79d--- /dev/null+++ b/instanthmr/mhr_renderer.py@@ -0,0 +1,110 @@+"""MHR (Momentum Human Rig) mesh decoder for InstantHMR.++Takes the 204-dim pose parameters and 45-dim shape parameters produced by+InstantHMR and decodes them into full body mesh vertices by running a forward+pass through Meta's MHR body model.++Quick setup (requires Python >= 3.12)::++ pip install -r requirements-mhr.txt+ curl -OL https://github.com/facebookresearch/MHR/releases/download/v1.0.0/assets.zip+ unzip assets.zip -d models/mhr_assets+ python demo.py --image photo.jpg --mhr-assets models/mhr_assets++WARNING — wrong pymomentum package on PyPI+------------------------------------------+``pip install pymomentum`` installs an **unrelated** legacy SMS library+(pyMomentum v0.1.x by MomentumAS). The correct Meta package is::++ pip install pymomentum-gpu # NVIDIA GPU+ pip install pymomentum-cpu # macOS / CPU-only++See README.md §'MHR body mesh' for the full installation walkthrough.+"""++from __future__ import annotations++from pathlib import Path++import numpy as np+++class MHRRenderer:+ """Loads and caches the MHR body model, runs per-person forward passes.++ Args:+ assets_folder: path to the unpacked MHR assets folder (the directory+ that contains the blend-shape files, template mesh, etc.).+ device: ``'cuda'``, ``'cpu'``, or a ``torch.device``.+ lod: level-of-detail mesh resolution:+ 0 = 73 639 verts (highest), 3 = 4 899, 6 = 595 (lowest).+ LOD 3 is a good default for live demos.+ """++ def __init__(+ self,+ assets_folder: str | Path = "models/mhr_assets",+ device: str = "cuda",+ lod: int = 3,+ ) -> None:+ import torch+ from mhr.mhr import MHR++ self._torch = torch+ self._device = torch.device(device)+ self._model = MHR.from_files(+ folder=Path(assets_folder),+ device=self._device,+ lod=lod,+ )++ faces_raw = self._model.character.mesh.faces+ if isinstance(faces_raw, torch.Tensor):+ self._faces: np.ndarray = faces_raw.cpu().numpy().astype(np.int32)+ else:+ self._faces = np.asarray(faces_raw, dtype=np.int32)++ @property+ def faces(self) -> np.ndarray:+ """Triangle face indices ``(F, 3)`` int32, shared across all frames."""+ return self._faces++ def forward(+ self,+ mhr_params: np.ndarray,+ shape_params: np.ndarray,+ ) -> np.ndarray:+ """Decode one person's parameters into body-space mesh vertices.++ Args:+ mhr_params: ``(204,)`` float32 — pose parameters from InstantHMR.+ shape_params: ``(45,)`` float32 — shape parameters from InstantHMR.++ Returns:+ ``(V, 3)`` float32 vertices in **body-centred** space (metres,+ Y-down). Add the person's ``cam_trans`` to move to camera space.+ """+ torch = self._torch+ identity = torch.from_numpy(shape_params).unsqueeze(0).to(self._device) # (1, 45)+ pose = torch.from_numpy(mhr_params).unsqueeze(0).to(self._device) # (1, 204)+ # MHR also accepts facial expression coefficients (72-D); InstantHMR+ # does not predict them, so we pass zeros (neutral expression).+ expr = torch.zeros(1, 72, dtype=torch.float32, device=self._device)++ with torch.no_grad():+ verts, _ = self._model(identity, pose, expr)++ v = verts[0].cpu().numpy().astype(np.float32) # (V, 3)++ # --- Coordinate-space alignment -----------------------------------+ # MHR body template (FBX) is Y-up right-handed and in centimetres.+ # InstantHMR / SAM3D uses Y-down right-handed and metres.+ #+ # 1. cm → m+ v /= 100.0+ # 2. Y-up → Y-down: negate Y.+ # Negate Z as well to preserve right-handedness after the Y flip.+ v[:, 1] *= -1.0+ v[:, 2] *= -1.0++ return vdiff --git a/instanthmr/visualizer.py b/instanthmr/visualizer.pyindex 92f7ffb..e51587e 100644--- a/instanthmr/visualizer.py+++ b/instanthmr/visualizer.py@@ -2,12 +2,15 @@Logs, per frame:- ``camera/image`` : annotated RGB frame (2D skeleton + bbox + a- text panel showing detector / HMR / total ms).+ text panel showing detector / HMR / render / total ms).- ``camera`` : pinhole intrinsics so the image becomes aproper frustum in the 3D scene.- ``world/persons/...`` : 3D joints + skeleton lines per person.+ - ``world/persons/.../mesh``: full MHR body mesh when an MHRRenderer is+ provided (vertices from a true MHR forward pass).- ``timing/detector_ms`` : scalar plot of RF-DETR latency.- ``timing/hmr_ms`` : scalar plot of InstantHMR latency.+ - ``timing/render_ms`` : scalar plot of MHR mesh rendering latency.- ``timing/total_ms`` : scalar plot of full pipeline latency.- ``timing/fps`` : scalar plot of effective frames-per-second.@@ -17,7 +20,8 @@ on construction (pass ``spawn_viewer=False`` to suppress).from __future__ import annotations-from typing import Optional, Sequence+import time+from typing import TYPE_CHECKING, Optional, Sequenceimport cv2import numpy as np@@ -25,6 +29,9 @@ import numpy as npfrom .inference import HMRPredictionfrom .skeleton import edges_for+if TYPE_CHECKING:+ from .mhr_renderer import MHRRenderer+class RerunVisualizer:def __init__(@@ -32,10 +39,12 @@ class RerunVisualizer:application_id: str = "instanthmr_demo",spawn_viewer: bool = True,save_path: str | None = None,+ mhr_renderer: "MHRRenderer | None" = None,):import rerun as rrself._rr = rr+ self._mhr_renderer = mhr_rendererrr.init(application_id, spawn=spawn_viewer)# SAM3D / MHR convention: right-handed, Y-down camera frame.@@ -60,7 +69,7 @@ class RerunVisualizer:detector_ms: Optional[float] = None,hmr_ms: Optional[float] = None,total_ms: Optional[float] = None,- ) -> None:+ ) -> float:"""Log one frame of the pipeline to Rerun.Args:@@ -72,6 +81,10 @@ class RerunVisualizer:hmr_ms: InstantHMR latency for this frame, summed acrosspersons (ms).total_ms: full pipeline latency for this frame (ms).++ Returns:+ render_ms: time spent running MHR forward passes + Rerun mesh+ logging for this frame (0.0 when no MHRRenderer is set)."""rr = self._rrrr.set_time("frame", sequence=frame_idx)@@ -98,16 +111,27 @@ class RerunVisualizer:# Always log the image — even when no person was detected — so the# viewer keeps showing the camera feed and the timing HUD.+ render_ms = 0.0if not persons:- blank = self._draw_overlay(image_rgb, [], detector_ms, hmr_ms, total_ms)+ blank = self._draw_overlay(image_rgb, [], detector_ms, hmr_ms, render_ms, total_ms)rr.log("camera/image", rr.Image(blank))- return+ rr.log("timing/render_ms", rr.Scalars(0.0))+ return render_ms++ # --- MHR mesh rendering (timed) ----------------------------------+ if self._mhr_renderer is not None:+ t0 = time.perf_counter()+ for idx, person in enumerate(persons):+ self._log_person_mesh(idx, person)+ render_ms = (time.perf_counter() - t0) * 1000.0++ rr.log("timing/render_ms", rr.Scalars(render_ms))for idx, person in enumerate(persons):self._log_person(idx, person)annotated = self._draw_overlay(- image_rgb, persons, detector_ms, hmr_ms, total_ms,+ image_rgb, persons, detector_ms, hmr_ms, render_ms, total_ms,)rr.log("camera/image", rr.Image(annotated))@@ -134,6 +158,8 @@ class RerunVisualizer:self._send_blueprint(first)self._blueprint_sent = True+ return render_ms+# ------------------------------------------------------------------# Helpers# ------------------------------------------------------------------@@ -159,12 +185,39 @@ class RerunVisualizer:rr.LineStrips3D(lines, colors=[255, 230, 0], radii=0.004),)+ def _log_person_mesh(self, idx: int, person: HMRPrediction) -> None:+ """Run MHR forward pass and log the resulting mesh vertices via Rerun."""+ assert self._mhr_renderer is not None++ # Decode mhr_params + shape_params → body-space vertices.+ verts_local = self._mhr_renderer.forward(+ person.mhr_params, person.shape_params,+ ) # (V, 3), body-centred++ # Shift from body space to camera space using InstantHMR's cam_trans.+ verts_cam = verts_local + person.cam_trans # (V, 3) broadcast++ faces = self._mhr_renderer.faces # (F, 3) int32, shared across frames++ n = len(verts_cam)+ skin = np.tile(np.array([[200, 160, 130, 140]], dtype=np.uint8), (n, 1))++ self._rr.log(+ f"world/persons/person_{idx}/mesh",+ self._rr.Mesh3D(+ vertex_positions=verts_cam,+ triangle_indices=faces,+ vertex_colors=skin,+ ),+ )+@staticmethoddef _draw_overlay(image_rgb: np.ndarray,persons: Sequence[HMRPrediction],detector_ms: Optional[float],hmr_ms: Optional[float],+ render_ms: float,total_ms: Optional[float],) -> np.ndarray:"""Draw per-person bbox + 2D skeleton + a timing HUD onto the frame."""@@ -201,12 +254,14 @@ class RerunVisualizer:# ---- Timing HUD (top-left, semi-transparent black panel) ----lines: list[str] = []if detector_ms is not None:- lines.append(f"RF-DETR : {detector_ms:6.1f} ms")+ lines.append(f"RF-DETR : {detector_ms:6.1f} ms")if hmr_ms is not None:lines.append(f"InstantHMR: {hmr_ms:6.1f} ms")+ if render_ms > 0.0:+ lines.append(f"MHR mesh : {render_ms:6.1f} ms")if total_ms is not None:fps = 1000.0 / total_ms if total_ms > 0 else 0.0- lines.append(f"Total : {total_ms:6.1f} ms ({fps:5.1f} fps)")+ lines.append(f"Total : {total_ms:6.1f} ms ({fps:5.1f} fps)")if lines:font = cv2.FONT_HERSHEY_SIMPLEXscale = max(0.5, min(w, h) / 1500.0)diff --git a/requirements-mhr.txt b/requirements-mhr.txtnew file mode 100644index 0000000..8951b7e--- /dev/null+++ b/requirements-mhr.txt@@ -0,0 +1,43 @@+# MHR body mesh rendering — install TOGETHER with requirements.txt+#+# IMPORTANT: always install both files in a single pip call so the resolver+# can pick a torch version compatible with rfdetr, torchvision, AND+# pymomentum-gpu at once. Installing them separately causes pip to first+# choose a torch version for rfdetr and then downgrade it for pymomentum,+# leaving torchvision broken.+#+# pip install -r requirements.txt -r requirements-mhr.txt+#+# Prerequisites+# -------------+# * Python >= 3.12 (pymomentum has no Python 3.11 wheels; recreate the+# conda env if needed: conda create -n instanthmr python=3.12 -y)+# * CUDA toolkit on the system path (recommended for real-time use)+#+# After installing, also download the MHR body-model assets:+#+# curl -OL https://github.com/facebookresearch/MHR/releases/download/v1.0.0/assets.zip+# unzip assets.zip -d models/mhr_assets+#+# Then run:+#+# python demo.py --image photo.jpg --mhr-assets models/mhr_assets++# pymomentum — Meta's momentum geometry library (C++ Python bindings).+# This package brings its own torch constraint; do NOT add a separate+# torch>=X line here or pip will resolve torch twice and downgrade it.+#+# WARNING: "pip install pymomentum" installs an UNRELATED legacy SMS library+# (pyMomentum v0.1.x by MomentumAS). The correct package names are below.+#+# Platform markers handle the selection automatically:+# Linux / Windows → pymomentum-gpu (NVIDIA GPU)+# macOS → pymomentum-cpu (Apple Silicon / Intel)+pymomentum-gpu ; platform_system != "Darwin"+pymomentum-cpu ; platform_system == "Darwin"+#+# CPU-only Linux / Windows: comment out pymomentum-gpu above and uncomment:+# pymomentum-cpu ; platform_system != "Darwin"++# MHR body model (Meta's Momentum Human Rig)+mhr @ git+https://github.com/facebookresearch/MHR.gitdiff --git a/requirements.txt b/requirements.txtindex 3c3f2ba..1644ca1 100644--- a/requirements.txt+++ b/requirements.txt@@ -21,6 +21,10 @@ torchvision# safe fallback.onnxruntime>=1.19+# ----- MHR body mesh rendering -----+# Optional. Requires Python >= 3.12 and a separate setup step.+# See requirements-mhr.txt and the "MHR body mesh" section of README.md.+# ----- training / annotation (optional, for reproduction) -----# timm>=1.0# tqdm