Skip to content
Python

feat: reproduce SAM 3D Body's continuous MHR regression head behind --cont-head

mohamdev664c56622912
diff --git a/CLAUDE.md b/CLAUDE.md
index 00bfc22..934b9db 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -96,6 +96,28 @@ or augmentation can be measured in under a minute. Do that.
---
+## What the training labels are
+
+**This is not a distillation setup.** Training is supervised by the released
+ground-truth annotations of `facebook/sam-3d-body-dataset` — the human MHR fits
+Meta used to build SAM 3D Body. The whole cluster corpus (`sam3d_gt_sa1b` /
+`_aic` / `_harmony4d` / `_coco` / `_mpii`) and every number in this repo's result
+tables are of that kind.
+
+Distilling from the `sam-3d-body-dinov3` model is an **option**:
+`tools/annotate_dataset.py` runs it over your own images and is the only thing
+that writes `data/sam3d_distill_mix/`. Use it for imagery the released dataset
+does not cover, and remember it caps the student at the teacher's accuracy.
+
+Two consequences for reading this file and the code:
+
+- **`train_distill_*.py` and `data/sam3d_distill_mix/` are historical names.**
+ They say nothing about which labels a run used.
+- **Where the docs say "teacher" they almost always mean "the training
+ target".** Parameter ranges, oracle ablations and
+ `adapter_j14_h36m_teacher.npz` were all measured against the *dataset's*
+ annotations, not against model inference.
+
## Where to read first
Docs are the source of truth; the code is bigger than any context window.
@@ -108,7 +130,7 @@ Docs are the source of truth; the code is bigger than any context window.
| 4 | `benchmark/README.md` | how published numbers are produced; the 3DPW GT (SMPL forward + H36M regressor), the MHR->J14 adapter, and why **J14+adapter** is the row to quote |
| 5 | `datasets_pipeline/README.md` | how the corpus is built, including the SA-1B visibility mask |
| 6 | `docs/architecture.md` | the student model |
-| 7 | `docs/todo.md` | open defects and opportunities from the 2026-09-06 audit, each with the measurement behind it |
+| 7 | `docs/todo.md` | open work: reproducing the teacher's regression pathway, the crop-resolution rebuild, model selection. Cleared and rewritten 2026-09-12 |
`datasets_pipeline/jeanzay/` and `JEAN_ZAY.md` are in `.git/info/exclude` on
purpose — they carry the login name, jump host and project code. Keep
@@ -143,6 +165,33 @@ augmentation, so it never needed the `m_ident` mask; and the 45 `shape_params`
move the 127-joint skeleton by `0.00e+00 cm`, so they are mesh-only and every
loss and metric in the trainer is blind to them.
+**The teacher's root Euler triple is REVERSED relative to the rig, and its own
+source hides it.** `sam_3d_body/models/heads/mhr_head.py` decodes the global
+rotation with `roma.rotmat_to_euler("ZYX", R)`. roma's *uppercase* letters mean
+extrinsic and it returns the triple in the order of the string, so `"ZYX"` hands
+back `(theta_z, theta_y, theta_x)` -- the reverse of the rig's extrinsic-XYZ
+`model_params[3:6]`. Feeding `(0.3, 0.5, 0.7)` through the teacher's own forward
+returns `(0.7, 0.5, 0.3)`. Measured against `checkpoints/mhr_model.pt` (md5
+8f21d804..., byte-identical to the teacher's `assets/mhr_model.pt`): drive
+`3:6`, rigid-fit the 125 joints it moves, and the recovered rotation matches
+`batch6DFromXYZ` to **0.00000-0.014 deg** while roma's `"ZYX"` of the same
+triple is **36-92 deg away**. Do the fit about the moving sub-skeleton, not the
+centroid and not joint 0 -- the root drives joint-parameter slots 10/11/12, so
+joints 0 and 1 do not move and a centroid Procrustes leaves an 11 mm residual
+that reads as a convention mismatch. Either convention round-trips if encode and
+decode agree, so a wrong choice trains fine and only shows up as a rotation loss
+measuring distance in a permuted space. `instanthmr_distill_train/mhr_cont.py`
+uses the rig's and says so; do not "fix" it to match the teacher's line.
+
+**`--cont-head` adds two files that must be rsynced.**
+`instanthmr_distill_train/mhr_cont.py` and
+`instanthmr_distill_train/assets/mhr_cont_head.npz` (30 KB, built by
+`tools/build_cont_head_assets.py` from the teacher checkpoint plus the corpus).
+Neither existed before 2026-09-12 and the job dies at startup without them.
+Gate any change to that head with `tools/verify_cont_head.py` (24 checks,
+including GT reachability to 0.0003 mm of forward kinematics) *and*
+`tools/ddp_smoke.py --cont-head`.
+
**The student's MHR outputs are unbounded and 77 of them are body SIZE.** The
head is a plain `nn.Linear`. `0:3` root translation (exactly `0.000e+00` in all
24,000 sampled annotations — the person is placed by `cam_trans`), `130:136`
diff --git a/docs/todo.md b/docs/todo.md
index 1c87c35..b61773e 100644
--- a/docs/todo.md
+++ b/docs/todo.md
@@ -1,884 +1,527 @@
-# Open work, from the 2026-09-06 accuracy audit
-
-What the external audit found, what was checked against the code, and what is
-left. The audit itself is `docs/accuracy_audit_2026-09-06.md`; this file is the
-verdict on each item and the queue.
-
-Every claim below was reproduced on real data before being written down. Where
-a number appears, the command that produced it is named.
-
----
-
-## Done (2026-09-06 / 09-07)
-
-### 1. The ONNX exporter dropped the scale bounds — 21 mm, silently
-
-`tools/pth_to_onnx.py` built a default `DistillConfig`, where `bound_scales` is
-off, and loaded with `strict=False`. A `--bound-scales` checkpoint's `scale_lo`
-/ `scale_hi` were reported as "unexpected keys" and discarded, so the exported
-graph had no `tanh` remap while the head still emitted pre-`tanh` values.
-
-**Mechanism.** With `--bound-scales` the last layer no longer emits bone scales.
-It emits a raw number that `tanh` maps into the rig's legal size range. Feed
-those raw numbers to the forward kinematics directly and the skeleton is
-wrong — not catastrophically, because the pre-`tanh` values are small
-(`|u|` mean 0.120, max 0.571 over teacher targets), but consistently.
-
-**Measured**, reconstructing the exact pre-`tanh` values a bounded head must
-emit to reproduce 64 real COCO teacher targets, then running the rig both ways:
-
-```
-MPJPE (export vs correct), raw : 21.0 mm
-MPJPE (export vs correct), centred : 19.9 mm
-PA-MPJPE(export vs correct) : 14.4 mm
-```
-
-On a 41.8 mm PA-MPJPE model that is a third added, with no error raised.
-
-**Fixed by** `T.config_from_checkpoint()` in `train_distill_mhr_only.py`, which
-reads `bound_scales` back out of the weights. Unexpected keys are now a hard
-exit, and the exporter gates on a PyTorch↔ONNX parity check (`2.7e-05` on a
-bounded graph, `3.1e-05` on a legacy one).
-
-### 2. Training and evaluation could use different camera inputs
-
-Three separate holes, failing in opposite ways:
-
-* `benchmark/eval_3dpw_ckpt.py` built a default config, so `cliff_focal` was
- off — **silent**. It also loaded strictly, so a bounded checkpoint raised —
- **loud**, and the only reason no wrong number was ever published.
-* `instanthmr/pipeline.py` never passed a focal to `InstantHMR`.
-* `HMRPrediction.focal_length` always reported `sqrt(H^2+W^2)` even when a real
- focal was supplied, so anything unprojecting or rendering used a fake camera.
-
-**Fixed by** separating *which conditioning form* (from `run_config.json` for
-checkpoints, from ONNX metadata for graphs) from *what the focal is* (the
-dataset's own value in training and both evaluators; `--focal`, else
-`1.05 x diag`, at deployment). See `instanthmr_distill_train/README.md`
-§`--cliff-focal` and `benchmark/README.md`.
-
-**Measured cost of the form mismatch**: 0.13 mm J14+adapter PA-MPJPE on a
-170-frame 3DPW validation probe. Small, because Procrustes removes exactly what
-this corrupts. The `bound_scales` half is where the 14–21 mm lives.
+# Open work
+
+Cleared on 12 September 2026. The previous 884 lines were the 2026-09-06 audit
+backlog; most items had been implemented, superseded, or disproved, and the file
+had stopped being a reliable statement of what is open. Recover it with
+`git show HEAD~1:docs/todo.md` if you need an old entry.
+
+Two rules that kept the old file honest and are worth keeping:
+
+- **State the measurement, not the intuition.** An item is worth writing down
+ once there is a number attached to it — what was measured, on what data, and
+ what the result was.
+- **Delete an item when it lands or dies.** A file of stale entries is worse
+ than an empty one, because it looks authoritative.
+
+Reference material: `benchmark/README.md` (evaluation protocol, conversion
+routes, measured floors, gen-6/7 tables), `instanthmr_distill_train/README.md`
+(label source, presets, size bounds, vertex term), `CLAUDE.md` (repo traps),
+`docs/accuracy_audit_2026-09-06.md` (the original audit, as a dated record).
+
+**Architecture constraint, confirmed 2026-09-12:** real-time inference on
+smartphones is a primary contribution. Keep efficient encoders such as HGNetV2
+and RepViT; DINOv3 is a reference, not a proposed deployment backbone. Evaluate
+architecture changes against the current model's latency, memory and operator
+support on the intended phone runtime. No acceptable slowdown has been
+quantified yet; do not assume a larger model or a repeated decoder is affordable
+from desktop GPU timings alone. Training-only changes should be distinguished
+from additions to the deployed inference path.
+
+The [2026-09-12 architecture audit](architecture_audit_2026-09-12.md) records
+code references and GT probes for the items below. Its accuracy opportunities
+are hypotheses unless accompanied by a matched training ablation.
---
-### 3. The v2 crop-centre update forgot to divide by the zoom — fixed 2026-09-07
-
-`train_distill_mhr_only.py` wrote `cx -= dx * crop_w / 2.0`. The augmentation
-warps the crop with `u' = s*R*(u - c) + c + t`, so inverting it puts the visible
-window's centre at `-t/s`: the shift has to be divided by the zoom. It
-over-corrected when zooming in and under-corrected when zooming out, and it was
-live in every `--preset v2` run (`geom_trans=0.20`, `geom_scale_max=2.0`).
-
-**Fixed behind `--crop-centre-fix`** (config `crop_centre_fix`), default off so
-generation 5 and `--preset baseline` stay bit-identical — verified: every loss
-term matches `git show HEAD` to the last printed digit on 48 real augmented
-samples, under both `baseline/legacy` and `v2/rebalanced`.
-
-**Verified on the fix itself.** Reference is the exact preimage of the crop
-centre under the real `M_total` that `cv2.warpAffine` was called with, over 400
-augmented COCO crops at v2 strength with rotation disabled so the zoom is the
-only variable:
-
-| | box-centre error, mean | p95 | max |
+# 1. Reproduce the teacher's regression pathway exactly
+
+**Status: implemented 2026-09-12, launched as generation 8.** Behind
+`--cont-head`, default off; the default path is verified bit-identical (1544
+state_dict keys, same sha256, forward and all 12 loss terms 0.000e+00 apart on
+48 real augmented samples under both `--losses legacy` and `rebalanced`). Code
+in `instanthmr_distill_train/mhr_cont.py` (the vendored conversions) and
+`ContMHRHead` in `train_distill_mhr_only.py`; asset built by
+`tools/build_cont_head_assets.py`; gated by `tools/verify_cont_head.py`.
+Measurements are in `datasets_pipeline/jeanzay/STATUS.md`, "Generation 8".
+
+**One deviation from §1.3 below, and it is deliberate.** That section says to
+build the root inverse with `roma`'s `"ZYX"`, matching the teacher's literal
+call. Measured against the rig -- `checkpoints/mhr_model.pt`, md5-identical to
+the teacher's own `assets/mhr_model.pt` -- driving `model_params[3:6]` and
+rigid-fitting the 125 joints it moves recovers **extrinsic XYZ**
+(`batch6DFromXYZ`) to 0.00000-0.014 deg, while `roma.euler_to_rotmat("ZYX", .)`
+of the same triple is **36-92 deg away**: roma's uppercase `"ZYX"` returns the
+triple Z-first, so it is the reverse of the rig's. Either convention
+round-trips, so both would train; this one is used because only then is the
+intermediate rotation matrix the body's actual root rotation, which is what
+makes the new chordal loss a physical angular error instead of a distance in a
+permuted space. §1.3's own instruction -- "the two conventions must not be
+silently conflated" -- is the reason.
+
+**Decision taken 2026-09-12:** reproduce the
+teacher's pathway *in full*, not a measurement-guided subset. The point is to
+remove the pathway as a confound, so a partial port would leave exactly the
+question it is meant to settle.
+
+## 1.1 The problem, stated precisely
+
+The student and SAM 3D Body feed MHR the same thing — the rig-native 204-vector —
+but they **regress in different spaces**, and the student's is the harder one.
+
+The teacher's head emits **519** numbers in a continuous space and converts them
+to the 204-vector *inside the head*, before MHR sees anything. Verified in
+`sam_3d_body/models/heads/mhr_head.py`, `forward()` (lines ~284-326) and
+`mhr_forward()` (lines ~198-215):
+
+| teacher head emits | dims | conversion applied inside the head | reaches MHR as |
|---|---|---|---|
-| `crop_centre_fix=False` | 7.73 px | 21.00 px | 34.93 px |
-| `crop_centre_fix=True` | **0.68 px** | 1.12 px | 1.50 px |
-
-The 0.68 px residual is the bbox-rounding floor.
-
-**With rotation on (±30°) a ~31 px residual remains against that reference, and
-it is NOT this bug** — the fix moves it only 33.40 → 30.93 px. The code models
-the in-plane rotation as a camera roll (it rotates `joints_3d`, `cam_trans` and
-the box centre about the image centre together), for which "preimage of the crop
-centre" is the wrong target, so most of that 31 px is the reference, not a
-defect. Expressing the `dx`/`dy` correction in the rotated axes instead changes
-the distance from the box centre to the actual body centre by 63.75 → 63.15 px,
-i.e. nothing. **No second defect demonstrated; see item 11 for what is still
-open.**
-
-## Priority Queue: CVPR Research Roadmap
-
-Active core contributions for the paper, strictly prioritized:
-
-### 1. Kinematically-Factorized Decoder with Anatomical Routing
-
-**Motivation.** In the current architecture (`train_distill_mhr_only.py:924`), a single global token of dimension 512 is projected through `head_global = nn.Linear(512, 252)` to predict root translation/rotation, spine, extremities, bone scales, and identity blendshapes simultaneously. Compressing the entire articulated human tree into one flat vector forces the global token to be an undifferentiated average of the body. Furthermore, the 70 2D SimCC queries act only as an auxiliary output head, with no mechanism for their 2D localized image evidence to inform the 3D kinematic parameters.
+| global rotation, 6D | 6 | `rot6d_to_rotmat` → `roma.rotmat_to_euler("ZYX", ·)` | 3 Euler angles |
+| body pose, continuous | 260 | `compact_cont_to_model_params_body` → 133, then `[:130]` | 130 |
+| shape | 45 | passes through raw | 45 |
+| bone scales, PCA | 28 | `scale_mean + coeffs @ scale_comps` | 68 bone scales |
+| hands (2 x 54) | 108 | `replace_hands_in_pose` into the body block | — |
+| face | 72 | multiplied by 0 in `forward()` | — |
+
+The 260-dim body space decomposes as **23 three-DoF joints x 6D (138)** +
+**58 one-DoF hinges x (sin, cos) (116)** + **6 one-DoF translation channels, raw
+(6)**. Those 6 raw channels are the `*_length/_width_flexible` parameters and
+land at `130:136` of the 204-vector — i.e. **the continuous representation does
+not bound them**, so they still need separate size constraints. Bounds on the
+expanded PCA scales must respect the scale subspace; see §1.3.
+
+The student instead emits the 204-vector directly from one `nn.Linear`:
+`head_global = nn.Linear(d_model, model_params_dim + shape_dim + cam_dim)` =
+204 + 45 + 3 = 252 numbers, all off a single query token
+(`shared_feats[:, 0, :]`). So it regresses raw Euler angles and 68 free bone
+scales where the teacher regresses 6D, sin/cos and 28 PCA coefficients.
+
+**The teacher uses a two-layer MLP and an initial pose estimate**, whereas the
+student uses a single linear layer. Reproduce the MLP and prediction relative
+to that estimate at a width compatible with the mobile budget. Valid continuous
+initialization matters: all-zero 6D vectors and all-zero sin/cos pairs are
+degenerate. Use a valid neutral-pose encoding, including identity rotations.
+
+## 1.2 Why this is not merely cosmetic — measured
+
+Over 6,000 `data/sam3d_gt_coco/annotations/*.npz` ground-truth samples:
+
+| parameter block | range | p99 \|angle\| | fraction beyond 2.8 rad |
+|---|---|---|---|
+| root rotation `3:6` | spans ±π: [−3.187, +3.361] | 3.13 rad | **13.59 %** |
+| local joints `6:130` | max 2.908 | **1.18 rad** | **0.0016 %** |
+
+- **22.32 %** of samples have at least one root axis within 0.34 rad of ±π,
+ near a wrap boundary of the target.
+- The root's middle Euler axis spans the **±π/2 singularity** (max 1.6226 vs
+ π/2 = 1.5708). At gimbal lock the first and third angles become degenerate;
+ the range alone does not prove a sample lies exactly there. **2.25 %** are
+ within 0.1 rad of it, **6.42 %** within 0.2 rad.
+- The 124 local rotation values in `6:130` essentially never wrap
+ (p99 1.18 rad). This measures target angles, not an accuracy gain from changing
+ their representation. The decision remains full reproduction.
+
+**Near-boundary targets do not prove current predictions cross that boundary.**
+Nor do they establish an expected millimetre improvement. That requires paired
+predictions and targets, followed by matched training ablations. Root-orientation
+improvements mostly affect MPJPE and orientation quality; PA alignment removes
+global rotation. Any improvement in PA-MPJPE from fixing root regression is
+indirect, not a guaranteed consequence of the angle histogram.
+
+Zhou et al., *On the Continuity of Rotation Representations* (CVPR 2019) is the
+reference for why a discontinuous target costs accuracy.
+
+## 1.3 What to implement
+
+Head emits 519 pose dims, or 447 if only the identically disabled 72-dim face
+block is omitted (**record that choice**), plus a separate 3-dim camera output.
+It then converts to the 204-vector with the
+teacher's own functions so the two pathways are identical:
+
+- `sam_3d_body.models.modules.rot6d_to_rotmat` + `roma.rotmat_to_euler("ZYX")`
+- `sam_3d_body.models.modules.mhr_utils.compact_cont_to_model_params_body`
+- `scale_mean + coeffs @ scale_comps` (both buffers are already in the
+ teacher's state dict; they must be copied into the student or loaded from the
+ rig)
+- the separate hand conversion and `replace_hands_in_pose`, using the teacher's
+ hand buffers and joint-index mappings
+
+**The separate hand parameter blocks matter even without a hand encoder.** They
+overwrite the body representation's finger channels. Omitting that overwrite is
+not the same pathway. A 339-dim head retaining fingers only through the body
+representation would therefore be a partial port; a dedicated hand-crop encoder
+is not required to reproduce the full head.
+
+**Correct root inverse:** do not apply `batch6DFromXYZ` directly to the stored
+root triple. The teacher decodes with `roma.rotmat_to_euler("ZYX", R)`, so build
+the inverse with the matching convention and concatenate matrix **columns**:
+
+```python
+R = roma.euler_to_rotmat("ZYX", mhr_model_params[:, 3:6])
+root_6d = torch.cat([R[:, :, 0], R[:, :, 1]], dim=-1)
+```
-**Mechanism.**
-- **Factorized anatomical queries.** Replace the single global query with structured kinematic queries:
- - `Q_root`: Camera translation, root orientation, global scale.
- - `Q_torso`: Pelvis, spine joints, neck.
- - `Q_larm` & `Q_rarm`: Shoulder, elbow, wrist joints.
- - `Q_lleg` & `Q_rleg`: Hip, knee, ankle joints.
-- **Anatomical routing via sparse/masked cross-attention.** The 70 2D keypoint queries first locate joints across the backbone feature map. Then, each kinematic limb query performs directed cross-attention specifically over its corresponding 2D joint tokens (e.g., `Q_larm` attends to left shoulder/elbow/wrist tokens, `Q_lleg` to left hip/knee/ankle tokens).
-- **Efficiency.** Replacing 1 query with 6 queries adds $<0.1$ ms to decoder execution, preserving the 200 FPS edge speed while providing clear inductive bias and explainability.
-
-
-### 2. Resolving the 12-Dimensional Kinematic Null Space
-
-**Motivation.** `parameter_transform[:, :204]` has rank 192, not 204. Twelve orthogonal directions move the MHR parameters while leaving 3D joint positions *exactly* unchanged ($<0.0002$ mm) — pairing each `*_flexible` parameter against its `scale_*` partner (`scale_spine_length` vs `spine_length_flexible`, `scale_hip_width` vs `hip_width_flexible`, and spine rotations). Along these directions, geometric losses have zero gradient, but direct parameter losses (`loss_pose`, `loss_scale`) penalize the difference. The student is thus forced to waste model capacity and gradient budget memorizing arbitrary solver noise from the teacher/GT annotations that no image can determine.
-
-**Mechanism.**
-- **Null-space projection of parameter loss.** Compute the singular value decomposition (SVD) of the linear transform $T = U \Sigma V^T$. Project the parameter error $(\hat{\theta} - \theta^*)$ onto the 192-dimensional active row space $V_{\text{active}}$, setting loss to zero along the 12 null directions:
- $$\mathcal{L}_{\text{pose\_proj}} = \| V_{\text{active}}^T (\hat{\theta} - \theta^*) \|_1$$
-- **Eliminating teacher noise.** The student is graded only on parameter variations that alter physical human geometry, completely removing gradient penalties on unobservable null-space drift.
-- **Zero runtime cost.** The projection matrix $V_{\text{active}} V_{\text{active}}^T$ is precomputed offline and applied only during training loss evaluation. Zero changes to ONNX inference or parameter count.
-
-
-### 3. Sensor-Agnostic In-the-Wild Field of View (FoV) Adaptation
-
-**Motivation.** Current inference assumes an uncalibrated default focal length $f = 1.05 \times \sqrt{H^2 + W^2}$ (`demo.py`, `instanthmr/inference.py`). In the wild (e.g. smartphone ultra-wide cameras at $0.6\times$ or telephoto lenses at $2\times$), this fixed assumption produces massive depth errors (1–3 meters) and severe scale distortion due to the perspective depth-scale ambiguity.
-
-**Mechanism.**
-- **FoV estimation query token.** Introduce a dedicated scalar FoV query token or integrate an angular field-of-view head predicting $2\omega_x = 2 \arctan(W / 2f)$.
-- **Decoupled scale & depth supervision.** Supervise predicted FoV on datasets with calibrated intrinsics (3DPW, Harmony4D, EgoExo4D) while training the pose decoder with self-consistent perspective reprojection.
-- **Plug-and-play inference.** In-the-wild deployment automatically infers the camera field of view alongside human pose, removing the need for manual `--focal` flags.
-
-
-### 4. Temporally Promptable Decoder (Temporal Smoothing)
-
-**Motivation.** Monocular frame-by-frame estimators (SAM-3D-Body, NLF, and InstantHMR baseline) exhibit high-frequency acceleration jitter and depth flickering on video sequences. Post-processing filters (1-Euro / Kalman) smooth coordinates at the expense of phase delay (lag) and physical plausibility (foot skating). Making the decoder natively temporal resolves jitter without lag.
-
-**Architecture: Additive residual temporal tokens.**
-- **Base learned temporal queries.** The decoder carries $N$ learnable temporal query tokens $\mathbf{q}_i^{\text{temporal}} \in \mathbb{R}^{d_{\text{model}}}$ (`nn.Parameter(1, N, d_model)`, e.g., $N=5$ or $10$). These tokens are always present and encode baseline canonical human priors and temporal relative slot identities ($t-1, t-2, \dots, t-N$).
-- **Additive pose residual.** Each slot receives an additive modulation from past poses:
- $$\text{Token}_i = \mathbf{q}_i^{\text{temporal}} + \Delta \mathbf{z}_i$$
- where $\Delta \mathbf{z}_i = \text{MLP}(\text{mhr\_params}_{t-i}, \text{cam\_trans}_{t-i})$ when frame $t-i$ is available, and **$\Delta \mathbf{z}_i = \mathbf{0}$** when absent.
-- **Zero-init stability.** Initializing the final linear layer of the projection MLP to zero ensures $\Delta \mathbf{z}_i = \mathbf{0}$ at initialization, allowing fine-tuning to start seamlessly from pretrained static weights without disturbing trunk representations.
-- **Online streaming without lag.** At frame 0, $\Delta \mathbf{z}_{1\dots N} = \mathbf{0}$, operating purely on the base learned tokens. As frames arrive, valid past poses modulate slot $1$, then slot $2$, up to $N$ in an incremental FIFO buffer. No future frames are needed, and tensor dimensions remain 100% static for ONNX and mobile NPU deployment.
-
-**Training strategy & exposure bias prevention:**
-- **Static dataset compatibility.** For static splits (COCO, AIC, MPII, SA-1B), $\Delta \mathbf{z}_i = \mathbf{0}$ across all slots. The model naturally learns to predict solely from visual features when temporal deltas are zero.
-- **Variable buffer warmup.** On video splits (Harmony4D, 3DPW), randomly zero out the last $N - k$ deltas ($k \in [0, N]$) to train the model to operate reliably during startup or after tracking dropouts.
-- **Denoising training against exposure bias.** Corrupt past annotations during training with Gaussian jitter on joint angles ($\sigma \sim 0.05$ rad), translation noise, and random slot zeroing (20-30%).
-
-**Evaluation & metrics:**
-- **3DPW test & Harmony4D test**: Both carry continuous video tracks.
-- **Acceleration error ($E_{accel}$, $\text{m/s}^2$)**: Measures discrete second-order joint acceleration differences against ground truth to quantify jitter.
-- **Rigid bone-length variance ($\sigma_{\text{bone}}$)**: Measures temporal scale consistency along rigid kinematic segments (femur, tibia, humerus) across tracks.
-
-### 5. Per-Topology Converter MLPs, Released as a Community Artifact
-
-**Decision (2026-09-09).** Ship one small feedforward converter per target
-topology — `mhr2smpl`, `mhr2soma` — as standalone weights the community can drop
-in front of any MHR-space estimator. The shared-latent / prompt-token version of
-this idea is *deferred*, not dropped; see the measurement below for what it would
-have to prove first.
-
-**Motivation.** Every SMPL number this repo publishes runs
-`benchmark/eval_smpl_fit.py`, which is 400 Adam iterations per frame. Measured on
-an RTX 4070 at batch 64: **0.17 ms/frame** for the mesh plus Meta's barycentric
-map, **42.24 ms/frame** for the fit, **42.4 ms total — 5x the entire 8.06 ms
-student**. Asking InstantHMR for SMPL output currently costs five model
-inferences and a GPU, which is the whole edge story undone at the last step.
-
-**What to build.** An MLP `(MHR 204 + shape 45) -> (theta 23x6D, beta 10)`, root
-taken from the pipeline's existing closed-form Procrustes. Two things matter:
-
-- **Train it with a geometric loss through the SMPL forward, not by regressing
- the solver's parameters.** Measured on 20,000 pairs (16k train / 4k held out),
- scored as vertex distance to the same barycentric target the solver optimises:
-
- | converter | params | latency | held-out vertex error |
- |---|---|---|---|
- | regress solver params, linear | 37 k | 0.007 ms | 40.71 mm |
- | regress solver params, MLP 2x512 | 467 k | 0.033 ms | 39.18 mm |
- | regress solver params, MLP 3x2048 | 9,208 k | 0.046 ms | 40.10 mm |
- | **geometric loss, linear** | 37 k | 0.007 ms | 27.00 mm |
- | **geometric loss, MLP 2x512** | 467 k | 0.033 ms | **19.64 mm** |
- | **geometric loss, MLP 3x1024** | 2,507 k | 0.051 ms | **19.19 mm** |
- | the 400-iteration Adam solver | — | 42.4 ms | 20.77 mm |
-
- Parameter regression plateaus near 40 mm at *every* capacity from 37 k to
- 9.2 M, because the solver trades theta against a 10-dimensional beta in a way
- that is not a smooth function of the input. Supervise the surface and the same
- network halves its error — and **beats** the solver, because 400 Adam steps
- from a cold start get stuck where an amortised network does not.
-
-- **No images are involved.** Pairs come from MHR parameters through the rig, so
- the whole training set is generated offline in ~15 minutes on one GPU. Nothing
- to relabel, no cluster job, no way to contaminate a benchmark.
-
-**SOMA.** `python -m tools.mhr2soma --input <parquet-dir> --output-npz out.npz`
-consumes MHR parquet "including SAM 3D Body outputs" — the exact source this
-corpus was built from, and `tools/parquet_to_npz.py` copies `model_params`
-through unchanged, so our 204 numbers are byte-identical to what NVIDIA's
-converter expects. NVIDIA maintains the label generator and it runs in the
-direction we need. SOMA-X is Apache-2.0. Budget a day for the install: it pins
-nothing in its docs, wants NVIDIA Warp and a CUDA-matched torch wheel, and
-`pymomentum` already segfaults in `sam3d_video_312` — keep label generation in a
-separate environment.
-
-**Prior art — do not write the converter up as a contribution.** Fast SAM 3D
-Body's contribution 3 is "a learned feedforward MHR-to-SMPL projection module
-that replaces hundreds of iterative optimization steps, accelerating
-cross-topology mesh conversion by over 10,000x" (`competitors/fastsam3d.pdf`).
-Their f_omega is a 3-layer MLP 4500->512->256->76 over 1,500 barycentric-projected
-vertices, and their Eq. 5 is exactly the geometric-plus-parameter loss above.
-The release value here is a *set* of converters covering SOMA as well as SMPL,
-shipped as usable weights — packaging and coverage, not novelty.
-
-**Accuracy is not the reason to do it.** The documented mesh-conversion floor is
-11.44 mm MPJPE / 10.66 mm PA-MPJPE / 13.61 mm PVE, but floors add in quadrature:
-at EMDB-1's 52.00 mm a perfect native head gives sqrt(52.00^2 - 10.66^2) = 50.90,
-a **1.10 mm** gain (1.17 mm on 3DPW). Quote the speed-up, not the floor.
-
-**Why the shared-latent version is deferred.** Its architectural premise is that
-a head reading `z_body` beats a head reading the predicted MHR parameters. Tested
-directly — 8,000 crops across five splits, same target, same head capacity, same
-schedule, only the input differs:
-
-| what the SMPL head reads | dim | params | held-out vertex error |
+The rejected inverse returns `[0.7, 0.5, 0.3]` from `[0.3, 0.5, 0.7]` through
+the teacher's forward conversion. For the body inverse, pass `6:136` plus three
+zero jaw values to `compact_model_params_to_cont_body` (133 inputs). Hand
+targets use their own joint ordering and inverse. These targets all come from
+the existing GT 204-vector; no new annotation pass is needed. Test root rotation
+losses against the rig's actual rotation convention as well as this head-space
+round-trip; the two conventions must not be silently conflated.
+
+**Existing scale bounds cannot simply be applied after PCA expansion.**
+Coordinatewise tanh generally moves the expanded scales outside the teacher's
+scale subspace. Preserve that subspace when designing bounds, for example by
+constraining coefficients with validated limits; still constrain the six
+flexible size channels separately and preserve zero root translation. PCA
+coefficients are not inherently bounded. The checkpoint's 28x68 scale matrix
+has effective float32 rank 24, so use a stable truncated pseudoinverse if
+constructing coefficient targets, not its transpose. The audit's GT projection
+residual is <2.2e-8; retaining the subspace is compatible with sampled GT scales.
+
+Every FK / geometric loss stays where it is and operates on the converted 204,
+with gradients flowing back through the conversion. Prefer a rotation-space
+metric over raw Euler differences for root supervision. Canonical 6D L2 targets
+are possible, but their metric is not physical angular error: multiple 6D
+vectors encode the same rotation after orthogonalization. Changing the output
+representation and changing other loss norms are distinct experiments.
+
+Gate it behind a config flag whose default reproduces today's behaviour, and
+prove bit-identity of the old path (`--preset baseline` and `--losses legacy`
+are controls for runs in flight).
+
+## 1.4 Loss and regularisation differences — configuration matters
+
+The public SAM 3D Body release is **inference-only** — no training code, no loss
+definitions. The following is from the paper (§ *Model Training*), so treat it as
+the teacher's stated design, not as verified code:
+
+> *2D/3D Keypoint Loss: We supervise 2D/3D joint locations using an L1 loss,
+> incorporating learnable per-joint uncertainty to modulate the loss based on
+> prediction confidence. For 3D body and hand keypoints, we normalize them with
+> their respective pelvis and wrist locations before computing the loss.*
+>
+> *Parameter Losses: MHR parameters (pose, shape) are supervised with L2
+> regression losses, and joint limit penalties are imposed to discourage
+> anatomically implausible poses.*
+
+**Already implemented in `--losses rebalanced`, including generation 7:**
+`apply_rebalanced_losses` sets `kp3d_loss="euclid"` and
+`w_keypoints3d=0.35`. This is unsquared Euclidean distance in metres, not
+SmoothL1 and not squared L2. **Do not re-fix the default SmoothL1 behavior as if
+generation 7 still used it.** The legacy/default configuration keeps SmoothL1
+for reproducibility. Switching the rebalanced path to the paper's coordinatewise
+L1 would be a new ablation, not completion of a missing fix.
+
+Deltas against the student today (deferred items are not implementation tasks):
+
+| item | teacher (paper) | student now | action |
|---|---|---|---|
-| the student's predicted MHR params | 249 | 467 k | 62.90 mm |
-| the shared latent `z_body` | 512 | 601 k | 62.84 mm |
-| the *teacher's* MHR params (oracle) | 249 | 467 k | 31.84 mm |
-
-0.07 mm out of 62.9. On this trunk the MHR head is a sufficient statistic for the
-SMPL target — where the head attaches is worth 0.1%, while the quality of the MHR
-prediction is worth 49%. What this does **not** test is whether a trunk *trained*
-against several rigs learns a different `z`; that is the only version of the
-claim still standing, and it needs a real run.
-
-**If we come back to it, the condition to satisfy is about data, not
-architecture.** Derived labels make the multi-head objective a *reweighting* of
-supervision we already have — and the reviewer's ablation is `--w-verts 0.35`,
-which we already run. The heads carry new information only if the labels do, i.e.
-from datasets with native SMPL/SOMA ground truth and no MHR annotation. Locally
-that is 81,006 EMDB frames outside EMDB-1 and 22,405 3DPW-train crops; the real
-prize is BEDLAM / AGORA / RICH. The headline experiment then becomes *does adding
-native-SMPL datasets through `head_SMPL` improve the MHR benchmark?* Before
-spending a generation on it, run the cheap gate: a paired short fine-tune, one arm
-MHR-only, one arm MHR + a second topology head, measured on 3DPW. The trunk is
-already underfitting (~29 mm off a near-perfect teacher on its own training data,
-79% of trunk gradient serving non-benchmarked outputs), so negative transfer is
-the live risk.
-
-**Verification.** Re-run `benchmark/eval_smpl_fit.py` with the MLP substituted for
-the solver; the two reports must agree to well under a millimetre. Note that the
-measurement above used `SMPL_MALE` throughout while the real evaluator is
-gendered — handle that before release.
+| parameter loss type | **L2** on MHR parameters | `SmoothL1`, `pose_beta` 0.05 rad (`_m_beta`) | decide; SmoothL1 at beta 0.05 is nearly L1 in the operating range, not L2 |
+| **joint limit penalty** | present, on the pose angles | absent; existing size bounds are not a pose penalty | do not apply stored pose limits blindly: all 6,000 audited COCO GT samples violate at least one, largely from flexible joints with `[0,0]` limits; any future prior must be annotation-compatible |
+| keypoint loss type | L1 | legacy: SmoothL1; **rebalanced/gen-7: unsquared Euclidean**, already implemented | no correction pending for rebalanced |
+| per-joint weighting | **learnable per-joint uncertainty** | fixed weights only (`finger_weight`, default 1.0, 0.2 in gen-6/7) | deferred with annotation-aware weighting; not current work |
+| 3D keypoint normalisation | pelvis (body) / wrist (hands) before the loss | direct difference in rig-origin coordinates; `fk_scale` is only a warm-up multiplier | deferred by user on 2026-09-12; do not implement now |
+| warm-up on the 3D term | yes | `kp3d_warmup_steps` = 2000 | already present |
+| root vs local split | one continuous body space | `pose_split` separates `loss_pose_root` (beta 1.0) from `loss_pose` (beta 0.05) | revisit once the root is 6D |
+
+**The loss metric is a separate bug from the output space.** Converting fixes the
+network's output manifold; it does not fix the metric. A SmoothL1 in radians on
+the root still scores a prediction at −π+ε against a target at π−ε as a ~2π
+error when it is a tiny rotation. The 13.59 % statistic measures large GT angles,
+not the frequency of this prediction error. Put the root's parameter loss on
+the rotation matrix (Frobenius or a numerically stable geodesic formulation)
+rather than on the Euler triple. Test finite gradients near identity and π.
+
+Current student weight defaults, for the record: `w_pose` 1.0, `w_scale` 0.1,
+`w_shape` 1.0, `w_cam` 0.1, `w_keypoints3d` 2.0, `w_keypoints2d` 10.0,
+`w_3d_joints` 1e-3, `w_reproj` 0.01, `w_simcc` 1.0, `w_verts` 0.0. The teacher's
+λ weights are not published.
+
+For **generation 7**, override that defaults list with `w_shape=0.03`,
+`w_keypoints3d=0.35`, `pose_beta=0.05`, `pose_split=True`,
+`finger_weight=0.2`, and the explicitly requested `w_verts=0.35`.
+
+## 1.5 Verification
+
+1. **Bit-identity of the old path** with the flag off — diff a few dozen
+ augmented samples against `git show HEAD:<file>`, comparing *alternating*
+ calls (the TorchScript MHR forward differs by ~5e-10 on its first invocation
+ in a process).
+2. **Round-trip correctness**: test padded body targets, separate hand targets
+ and the corrected root inverse on real GT. Compare rotations and FK geometry
+ as well as parameters: equivalent Euler triples need not be numerically
+ identical after canonicalization. Verify finite gradients and valid neutral
+ initialization before training.
+3. **The progress metric is the fitting gap, not 3DPW.** The student is roughly
+ twice as far from its labels as it needs to be *on images it has already
+ trained on*, and that — not split, conversion or selection — is where the
+ ~23 mm gap to SOTA lives. Track it directly while changing the regression
+ space.
+
+ **Build the measurement; it does not exist today.** The number quoted in the
+ old todo (**62.90 mm** with the student's predicted MHR params against
+ **31.84 mm** with the GT params, held-out vertex error over 8,000 crops on
+ five splits) came from a throwaway MLP ablation whose code is gone — no
+ `z_body` / `head_SMPL` / `smpl_head` remains in the tree. It was also a
+ by-product of a different question (whether an SMPL head should read the
+ trunk latent or the predicted parameters; answer: 0.07 mm, irrelevant), it
+ carries the MLP's own 31.84 mm floor, and it used `SMPL_MALE` throughout.
+ Do not quote it next to a benchmark number.
+
+ **Prefer a floor-free version:** run the student over a few thousand
+ *training* crops, push the predicted 204-vector and the GT 204-vector through
+ `MHRForwardPass` separately, and compare the 70 keypoints **root-relative,
+ not Procrustes-aligned**. No learned head means no floor; no Procrustes means
+ body-size and proportion errors stay visible. ~20 lines.
+
+ **What exists today and its trap.** The epoch log's `held-in` line is a 2 %
+ slice of the training corpus scored against the GT labels — the right family
+ of metric. But `train_distill_jz.py:900-903` **overwrites** `Mesh_PA_MPJPE`
+ and `Mesh_MPJPE` with the 3DPW numbers before they reach `history.jsonl`
+ (deliberately, so selection keys off 3DPW), so those two fields carry no
+ held-in information. Only `Mesh_PA_MPJPE_body` survives. Current values:
+ g7r_s0 ep169 **34.15** (3DPW-val 39.93), g7h_s0 ep111 **40.52** (43.65),
+ g6vv_s ep252 **43.44** (42.22), g6bv_s ep299 **41.82** (42.24) — note that
+ two of the four runs are *worse* on their own training data than on 3DPW,
+ which is worth explaining on its own.
+4. `python tools/ddp_smoke.py --backbone <name> --w-verts <w>` before submitting
+ — a changed head shape is exactly the class of change that passes every
+ single-process check and then aborts DDP on step 2.
+5. Seed spread is wide; anything under ~2 mm on 3DPW needs 2-3 seeds.
---
-## Audit Backlog & Operational Queue
-
-Ordered by (cost of being wrong) x (cheapness of the fix). Items 3, 4 and 6 are
-defects; 5 and 7 onward are opportunities the audit raised that are not bugs.
-
-### 4. Validation drops the frames the model failed on
-
-`val3dpw.py:179` removes non-finite predictions before scoring — necessary,
-because one `inf` makes numpy's Procrustes SVD raise and that killed a healthy
-20-hour job. But the count is only logged, and the surviving-subset score still
-competes for "best checkpoint" at `train_distill_jz.py:895`. A checkpoint that
-blows up on the hardest frames can therefore be selected over one that merely
-does badly on them.
-
-The distributed half is real but numerically trivial: `all_reduce_mean`
-averages rank means rather than pooling, and the 3DPW set is stride-sharded
-499/498/498/498 out of 1,993, so the bias is under 0.1% unless failures cluster
-on one rank.
+# 2. Crop resolution — the corpus is stored at the network input size
+
+**Status:** open.
+
+`datasets_pipeline/build_split.py --body-size` defaults to **224**, and the
+stored crops are 224x224 (`data/sam3d_gt_*/images/*.png`, verified). The network
+input is also 224. Two consequences:
+
+1. **Upsampling existing crops adds no source detail.** Training at 256 or 288
+ may still change feature sampling density, but evaluate that separately from
+ rebuilding crops with `build_split.py --body-size <N>` from original frames.
+2. **`--preset v2`'s zoom augmentation currently upsamples.** Its zoom ceiling is
+ 2.0, so a zoomed-in sample is interpolated from the 224 crop: up to 2x
+ upsampling. The model is being trained to expect blur that a real crop at that
+ scale would not have. Storing crops at 288-320 and letting augmentation crop
+ *down* into 224 can address this **without changing the network at all**.
+ The current dataset resizes to 224 **before** affine augmentation, so the
+ implementation must instead sample the affine crop directly from the stored
+ higher-resolution source. Changing stored resolution alone is insufficient.
+
+**Keep the efficient encoder and mobile speed constraint.** Test modest input
+increases (256/288) on the current HGNetV2/RepViT, or earlier-stage/stride-16
+features at the same 224 input, as separate options. The current encoders emit
+7x7 final features at 224; stride-16 features would give 14x14. More image tokens
+also cost decoder time and memory. Benchmark the complete exported path on the
+intended phone runtime before promoting either option. Do not replace the
+encoder with DINOv3 to reproduce the teacher's 32x32 feature grid.
+
+For scale: NLF uses EfficientNetV2-S at **256 px** and EfficientNetV2-L at
+**384 px** (NLF paper, *Implementation details*). 384² is 2.94x the pixels of
+224². The student is 40.1 M parameters total / 22.6 M backbone (`g7r_s0`,
+`repvit_m2_3`); `g7h_s0` (`hgnetv2_b4`) is 31.9 M / 13.6 M. NLF-L is therefore
+~3x the student in both parameters and pixel budget — the fair size comparison
+is NLF-**S**, not NLF-L.
+
+Cost to rebuild: the corpus is 4.2 M crops, and `sam3d_gt_harmony4d` has no
+originals on the cluster (staged from the laptop), so a rebuild is not free.
+Decide the target size once and do it in one pass.
+
+## Verification
+
+- Confirm stored crop size and that `build_split.py` writes what was asked.
+- Re-measure 3DPW J14+adapter PA-MPJPE at matched epochs, 2-3 seeds.
+- Check separately whether removing the augmentation upsampling helps at the
+ *same* 224 input — that isolates the blur effect from the resolution effect.
-**Risk is low right now**: `--bound-scales` is exactly what stops the outputs
-going non-finite. **Watch** `⚠️ 3DPW: N non-finite prediction(s) dropped` in
-the epoch log — zero every epoch means this never fires; a rising count is a
-real instability and should be investigated immediately.
-
-**Fix when convenient**: reduce error *sums* and valid *counts* rather than
-rank means, and either mark an epoch with drops ineligible for selection or
-score dropped frames at a fixed penalty so the population is constant.
-
-### 5. Score both routes to 2D keypoints on the same footing
-
-The graph emits 2D keypoints two ways, and only one is measured today:
-
-1. `joints_2d` — the SimCC head. This is what `benchmark/eval_coco.py` scores
- (`p.joints_2d[COCO17_FROM_MHR70]`).
-2. `mhr_params` + `cam_trans` -> forward kinematics -> project with the focal.
- Governed by `loss_reproj`.
-
-The detach experiment is what makes this worth measuring: it showed the two
-routes are genuinely independent. `loss_reproj` was unchanged between the arms
-to within seed noise while the SimCC head collapsed to a point, so the trunk
-kept its image-plane localisation and only the readout lost it. Route 2 may
-therefore be usable when route 1 is not — see
-`instanthmr_distill_train/README.md` §"Tried and rejected".
+---
-The logged columns cannot settle which is better: `loss_reproj` is an L1 with
-fingers down-weighted to 0.2 and `w_reproj = 0.5` folded in, while
-`loss_2d_native` is a smooth-L1 at full finger weight times 10. Different
-norms, different weights, both on training data.
+# 3. Model selection
+
+**Status:** open, low priority relative to items 1 and 2.
+
+Selection is `min` over epochs of 3DPW **validation** J14 PA-MPJPE with the
+adapter applied, from `train_distill_jz.py` via `val3dpw.py` (validation split,
+12 sequences, stride 5, 1,993 person-frames, H36M GT).
+
+**The split choice is correct — keep it.** Selecting on 3DPW test or EMDB would
+turn the only clean numbers in the repo into training metrics.
+
+**Selection noise is not a problem, measured.** The EMA validation curve has std
+**0.08-0.17 mm** over the last 30 epochs, so `min` over ~300 epochs is only
+0.2-0.6 mm optimistic. The val→test gap (g7r_s0: 39.93 → 41.55, **+1.62 mm**) is
+genuine split difference, not a winner's curse. Older runs showing test
+*better* than val by 6.5-7.2 mm (b2_s0, b3_s1, v3_s1) had their validation
+scored by a pre-2026-09-05 `val3dpw.py`; those gaps are not comparable.
+
+Two real defects:
+
+1. **One metric, blind to most of what matters.** PA-MPJPE removes translation,
+ rotation and scale; root-relative MPJPE removes translation. So `cam_trans`
+ drift, body-size error, temporal jitter and identity are all invisible to the
+ selector — and they are what the vid1/vid4/vid5 visual check actually
+ judges. The ranking is not stable across benchmarks either: `g6vv_s` is best
+ on 3DPW and under projected-GT-joint EMDB boxes, and **fourth** under EMDB's
+ own boxes. Add an EMDB column and a jitter number (frame-to-frame
+ acceleration; rigid bone-length CV, GT floor 1.06 %) to the selection.
+2. **`summarize_runs.py` reports a pair no checkpoint achieves.**
+ `datasets_pipeline/jeanzay/summarize_runs.py` reduces every METRICS column
+ with `min` *independently* across epochs, and `_dpw` additionally takes
+ `min(raw, ema)` per epoch. So the `g7r_s0` row "39.93 PA / 69.27 MPJPE" is PA
+ from epoch 169's EMA weights and MPJPE from a different epoch, possibly
+ different weights. Report the pair at the selected epoch; keep per-column
+ records in a separate row if they are wanted.
-**Add a `--from-mhr` path to `eval_coco.py`** that projects the MHR keypoints
-instead of reading `joints_2d`, and report both on the same 17 COCO body
-keypoints. That is the measurement that decides whether the SimCC head still
-earns its place — and note the projected route needs `cam_trans` to be right,
-which PA-MPJPE cannot validate, so it is not automatically the safer choice.
+---
-**If the projected route wins, "remove the 2D head" is still not the change to
-make.** The head is 0.025 of ~10.8 GFLOPs — 0.23%, nothing. The cost is in the
-**70 decoder queries** that feed it: 15.7% of total FLOPs. Dropping those is a
-different architecture that no run has tested, and the standing hypothesis is
-that they *help* the global token via self-attention. Sequence: finish the
-runs, score both routes, then a separate ablation with the 70 queries removed.
-(`--detach-2d-head` is gone; it was tried and rejected, see the trainer README.)
+# 4. Decoder feedback within the smartphone inference budget
+
+**Status:** open experiment, conditional on preserving real-time smartphone
+inference. User accepts an adapted version if full teacher feedback costs too
+much. Do not copy the teacher's six-layer, 1024-wide decoder wholesale.
+
+**Verified difference:** the student predicts MHR once after four decoder
+layers. The teacher predicts provisional geometry between layers, projects
+keypoints, samples image features at their 2D locations, and supplies updated
+2D/3D coordinate embeddings to subsequent layers. It has 70 auxiliary 3D
+tokens, although it has no independent XYZ regression head. Restoring a freely
+regressed XYZ head alone would not reproduce this mechanism.
+
+The expected benefit is better local articulation from inspecting image
+evidence near provisional joints. No generation-7 accuracy gain or phone
+latency has been measured for this change.
+
+## Implementation and verification
+
+- Start with one feedback update between existing decoder layers; reuse the
+ encoder features and keep decoder width/depth fixed for the first ablation.
+ Try a small body-joint subset and the existing cheap skeleton FK before
+ considering full mesh geometry or additional 3D tokens at every layer.
+- A cheaper alternative is intermediate SimCC coordinates driving feature
+ sampling, without FK inside decoding. Label this as a 2D-guided approximation
+ and measure it separately; it is not the teacher's geometry-derived feedback.
+- Intermediate supervision can be training-only, but feature updates used by
+ later layers belong to inference too. Do not assume they can be removed at
+ export without testing the resulting accuracy change.
+- The current parameter-output graph leaves FK outside the neural model.
+ Putting FK or bilinear feature sampling inside decoding changes the export
+ requirements. Check actual phone-backend support and fallbacks, end-to-end
+ latency, peak memory and sustained throughput, alongside matched accuracy.
+ If the change costs too much, simplify or leave it unselected; desktop FPS
+ and FLOP estimates do not establish smartphone feasibility.
-### 5b. Late divergence the loss guard cannot see (`bno_s1`, job 1813482)
+---
-**Open. This killed a healthy run at epoch 92 of 300 and nothing caught it.**
+# 5. Exact 70-landmark readout using a small vertex subset
-`bno_s1` reached 45.77 mm J14 PA at epoch 84 and was dead by 92, aborting on
-`still stuck after 3 EMA rollback(s)`. Its twin `bno_s0`, identical config and
-a different seed, skipped exactly **one** step in 100 epochs.
+**Status:** open, scoped to extending existing subset skinning. Preserve MHR's
+direct pose-parameter prediction and rig/mesh separation. First evaluate as a
+training-loss change; a deployed landmark-readout change is a separate decision
+subject to the smartphone speed constraint.
-This is **not** the generation-4 failure. `--bound-scales` held: the anomalous
-losses are 50-175, not the 4e7 / 8.8e13 of the size-parameter blowup, and the
-skeleton stayed metre-scale. PA-MPJPE removes scale, so 341 mm means the
-**pose** was destroyed — the 124 rotations at `mhr_params[6:130]`, which
-`--bound-scales` deliberately leaves free.
+**What these landmarks are:** the 70 annotation keypoints supervised by
+`loss_3d_native` and used for reprojection, including nose, shoulders, elbows,
+knees, wrists, fingers, toes and surface landmarks such as the acromion. They
+are not the rig's 127 kinematic joints or its joint-angle parameters.
-It is also not the learning rate. The collapse ran at `lr 6.8e-5`, a quarter of
-nominal, well past OneCycle's epoch-30 peak.
+The teacher first predicts pose, scale and shape parameters. MHR computes
+skeletal joints by forward kinematics and skins the mesh using those joints.
+Only then does a fixed checkpoint matrix read out landmark positions:
-**The precursor, nine epochs early, that nobody was watching.** Epochs 75-84 ran
-with `skipped 0/1562` every single epoch — no NaN, no loss over the threshold —
-while the two validation branches disagreed wildly:
+```text
+predicted parameters → FK joints J → skinning → vertices V
+ J + V → fixed mapping → 70 landmark positions
-| epoch | RAW J14 PA | EMA J14 PA | skipped |
-|---|---|---|---|
-| 75 | 53.0 | 47.7 | 0 |
-| 76 | 51.7 | **188.9** | 0 |
-| 77 | 48.4 | 123.4 | 0 |
-| 78 | 49.3 | 69.5 | 0 |
-| 79 | 47.4 | 50.8 | 0 |
-| 80-84 | 47-48 | 45.8-48.7 | 0 |
-| **85** | **268.7** | 47.1 | 67 |
-| 86 | 302.3 | 321.0 | 0 |
-
-RAW is sampled once per epoch; the EMA integrates every step. So the epoch-76
-spike says the weights took a large excursion *inside* an epoch that the
-end-of-epoch snapshot missed entirely and the loss guard never saw. The EMA
-decayed it out over ~3 epochs (half-life `ln2/(1-0.9998)` = 3,466 steps ≈ 2.2
-epochs, which matches), everything looked fine again, and nine epochs later the
-run died for real.
-
-**`EMA / RAW` on the validation metric is a free excursion detector and is not
-monitored.** Add it to the epoch line and to `summarize_runs.py`; a ratio above
-~1.5 with `skipped 0` means the weights are wandering between snapshots.
-
-**Each rollback made things immediately worse.** Both visible rollbacks are
-followed by a markedly worse epoch:
-
-| rollback | epoch after | train total | skipped |
-|---|---|---|---|
-| attempt 2 (end of ep 73) | 74 | 3.19 (was 2.64) | 74 |
-| attempt 3 (end of ep 84) | 85 | **7.99** (was 2.60) | 67 |
-
-Attempt 3 restored a **45.8 mm** EMA at `lr 6.8e-5` and the model was at 268 mm
-one epoch later. Restoring good weights and then destroying them within 1,562
-steps is not a learning-rate problem — the recovery mechanism is a prime
-suspect, not just the failure.
-
-**Ranked hypotheses, and how to test each.**
-
-1. **The Adam restart in the rollback.** `optimizer.state.clear()` zeroes both
- moments, so the very next step is `lr * m̂/(√v̂+ε) = lr * sign(g)` for **all**
- 40M parameters at once — a coordinated jump of norm `lr * √(4e7)` ≈ 0.43 at
- this LR, against the heavily damped updates steady-state Adam produces.
- *Test:* keep the moments, or re-warm the LR over ~200 steps after a
- rollback, and see whether the post-rollback epoch still degrades.
-2. **The rotations are genuinely unbounded.** `--bound-scales` covers `0:3` and
- `130:204` only. `6:130` cannot blow up the skeleton's *size* but can produce
- arbitrary pose. *Test:* log `mhr_params[6:130]` percentiles per epoch; if
- they drift monotonically through epochs 75-84 the wandering is there.
-3. **The guard's trigger is the wrong quantity.** It fires on
- `loss > 100 or non-finite`. Epochs 75-84 satisfied neither while the model
- visited bad regions. *Test:* trigger on gradient norm or on a weight-delta
- norm instead of, or as well as, the loss.
-
-**Two log defects found while diagnosing this, both worth fixing:**
-
-* `bad` is all-reduced with `ReduceOp.MAX` (`train_distill_jz.py:682`) but the
- message prints **rank 0's own** loss. That is why the log shows
- `skipping step 57 (loss 10.8)` against a threshold of 100 — a healthy rank
- vetoed by a sick one. Print the reduced maximum, or the vetoing rank.
-* `📈 epoch` reports RAW and EMA side by side but nothing flags their
- divergence, which was the only visible symptom for nine epochs.
-
-**Ruled out.** The safe-fallback path was *not* masking the collapse: `repaired`
-fired **once** in the entire run, and only one epoch shows a `repaired` count.
-The concern filed as item 6 is real bookkeeping sloppiness but had nothing to do
-with this failure.
-
-### 6. `n_repair` overstates successful updates
-
-`GradScaler.step` silently skips the optimiser when gradients overflow, but
-`train_distill_jz.py:715` / `:793` call `scheduler.step()` and
-`ema.update_parameters()` regardless and count a repair. Consequences are
-cosmetic — re-averaging unchanged weights into the EMA is harmless, and
-stepping the scheduler through a skip is deliberate (freezing the LR during a
-skip storm is what caused the earlier runaway). Worth making explicit so the
-"repaired N" line means what it says.
-
-### 7. Direct parameter supervision reaches 20–60% of samples
-
-At `geom_p=0.8`, `geom_flip_p=0.5`: root rotation (`mhr_params[3:6]`) is
-supervised on the 20% with no spatial augmentation; local angles, bone scales
-and shape on the 60% that were not flipped. **Not a bug** — the dataset rotates
-and mirrors the labels without rotating the parameters, so the masked-out
-targets are genuinely wrong, and those samples still get full 3D geometric
-supervision.
-
-**The opportunity**: transform the targets properly instead of masking — mirror
-the left/right parameter channels for flips, compose the roll into the root
-rotation. Needs the rig's mirror mapping **verified, not assumed**; MHR's Euler
-channels are not SMPL's permutation/sign convention.
-
-### 8. Twelve redundant target dimensions (Elevated to CVPR Priority 2)
-
-`parameter_transform[:, :204]` has rank 192. Moving 10 units along a null
-direction changes the skeleton by `< 0.0002 mm`, yet `loss_pose` / `loss_scale`
-still penalise the difference — 12 dimensions of solver-arbitrary teacher noise
-no image can determine. **See Priority 2 in the CVPR Roadmap above for the active
-null-space projection loss formulation.**
-
-**Measure before acting**: how much teacher variance actually lies along those
-directions, and how much trunk gradient they carry. The audit is explicit that
-redundancy alone does not prove a cost. Do **not** "fix" it by zeroing the
-`*_flexible` parameters — the teacher genuinely uses them
-(`leg_length_flexible` mean 0.32, max 1.64) and zeroing shortens every leg.
-
-### 9. The mesh is invisible to the geometry objective
-
-`MHRForwardPass.get_joints` zeroes the 45 `shape_params` before the forward
-pass; perturbing them by +2 sigma moves the 127-joint skeleton by exactly
-`0.00e+00 cm`. They receive direct parameter loss (`w_shape = 0.03`) and
-nothing else. So every geometric loss and every reported metric is blind to
-body shape.
-
-**Consequence for claims, today**: PA-MPJPE can improve while the mesh gets
-worse. Do not say "better mesh" on the strength of a pose number.
-
-**Half-addressed 2026-09-07 by `--w-verts`** (see item 13): the surface *term* now
-exists and is the only loss whose gradient reaches `shape_params` — measured
-`d(loss)/d(shape)` is `8.9e-03` through vertices and **exactly 0.000e+00**
-through joints. What is still missing is the surface *metric*: nothing in
-`benchmark/` reports PVE, so the claim side of this item is unchanged.
-
-### 10. `fy` for both axes — checked, not a problem here
-
-The conditioning uses one focal where a camera has two. Measured `fx/fy` over
-300 samples per split:
-
-| split | fx/fy |
-|---|---|
-| coco, aic, mpii | 1.00000 exactly |
-| harmony4d | 0.98661 – 1.03656 |
-| 3dpw | 0.99625 – 1.00376 |
-
-The `bno` / `dt` mix is `aic/coco/mpii` only — perfectly isotropic — and 3DPW
-differs by 0.4%. Revisit only if a new split with anisotropic pixels enters the
-mix.
-
-
-### 11. The rotation half of the CLIFF crop-centre correction is unverified
-
-Item 3 fixed the zoom. The rotation is a separate question and it was **not**
-resolved: with `geom_rot_deg=30` a ~31 px box-centre residual remains against a
-preimage-of-crop-centre reference, and the zoom fix moves it by only 2.5 px.
-
-Two readings, and the measurement does not separate them. Either the code is
-right — it treats the in-plane rotation as a camera roll, rotating `joints_3d`,
-`cam_trans` and the box centre about the image centre together, and under that
-model the box is *supposed* to move, so the reference is simply wrong — or there
-is a real error hiding under a reference too loose to see it. What was ruled
-out: expressing the `dx`/`dy` correction in the rotated axes rather than the
-unrotated ones, which changes the box-centre-to-body-centre distance from 63.75
-to 63.15 px, i.e. nothing.
-
-**To settle it** the reference has to come from the camera-roll model itself,
-not from the warp: construct a synthetic camera, roll it by the augmentation
-angle, and check that the conditioning the dataset emits equals the conditioning
-that rolled camera would produce for the same box. Until then, do not "fix" the
-rotation — the current form is not shown to be wrong.
-
-### 12. HGNetv2 vs RepViT — measured 2026-09-08; `hgnetv2_b4` is the challenger
-
-**Verdict: adopt `hgnetv2_b4` as the challenger trunk, not the biggest variant.**
-It beats `repvit_m2_3` on every axis measured, with fewer parameters (17.75 M vs
-22.40 M) and less compute (2.728 vs 4.520 GMAC in the backbone). `hgnetv2_b6`
-fits better on larger data but costs 2.9x the training time, 4.2x the mobile-CPU
-latency and quantises worse, so it is a server-side option, not a phone one.
-
-All numbers below are local, on this corpus and `checkpoints/mhr_model.pt`. The
-cluster run that decides the accuracy question has not been done — see the plan
-at the end.
-
-**Accuracy — the 1-batch overfit test, run with the gen-6 `g6v_s` config**
-(`--preset v2 --losses rebalanced --cliff-focal --bound-scales
---anomaly-safe-fallback --crop-centre-fix`), 1000 steps, best Mesh PA-MPJPE over
-the 70 MHR keypoints. Three seeds, because a single run is inside the documented
-34–49 mm spread.
-
-| backbone | 8 crops, 3 seeds | mean | 64 crops, 2 seeds | mean |
-|---|---|---|---|---|
-| `repvit_m2_3` | 47.0 / 51.3 / 43.6 | **47.3** | 28.4 / 29.0 | **28.7** |
-| `hgnetv2_b2` | 40.3 / 43.1 / 37.6 | **40.3** | 26.7 / 24.7 | **25.7** |
-| `hgnetv2_b4` | 39.1 / 30.3 / 34.6 | **34.7** | 24.9 / 24.2 | **24.6** |
-| `hgnetv2_b6` | 34.0 / 45.6 / 35.0 | **38.2** | 23.8 / 20.5 | **22.2** |
-
-The 8-crop and 64-crop orderings disagree at the top: 8 crops is too small to
-need b6's capacity and its fixed-LR trajectory is noisy there. On 64 crops
-capacity orders monotonically. Both agree that every HGNetv2 variant tested
-fits better than `repvit_m2_3`, including b2 at 2.4x fewer parameters and 4x
-less compute.
-
-**The accuracy margin is mostly the checkpoint, not the topology.** Same test
-from random init instead of ImageNet:
-
-| backbone | ImageNet init | random init (2 seeds) |
-|---|---|---|
-| `repvit_m2_3` | 47.3 mm | 42.3 mm (43.1 / 41.4) |
-| `hgnetv2_b4` | 34.7 mm | 40.7 mm (40.5 / 40.8) |
-
-From scratch the two are 1.6 mm apart, inside seed noise. HGNetv2's timm weights
-are Baidu **SSLD** (ImageNet-22k → 1k distillation); RepViT's are the paper's.
-That is a real, free asset — it is the configuration we would train — but the
-honest claim is "better initialisation", not "better architecture for accuracy".
-Note also that RepViT fit *better* from scratch than pretrained: on an 8-crop
-memorisation task a pretrained trunk has to unlearn, which is one more reason
-this test does not settle generalisation.
-
-**Speed.** Whole student, 224 input, RTX 4070. Training is fwd+bwd+AdamW at
-batch 64; the CPU column is single-thread ONNX fp32, the closest mobile-shaped
-number available without a phone.
-
-| backbone | train B64 | peak GPU | ONNX fp16/CUDA B1 | CPU 1-thread |
-|---|---|---|---|---|
-| `repvit_m2_3` (as shipped) | 193.6 ms | 8549 MiB | 4.76 ms | 115.2 ms |
-| `repvit_m2_3` + `.fuse()` | 142.5 ms | 5347 MiB | 3.55 ms | 110.7 ms |
-| `hgnetv2_b2` | 96.5 ms | 2824 MiB | 3.98 ms | 46.9 ms |
-| `hgnetv2_b4` | **97.4 ms** | **3498 MiB** | **3.00 ms** | **72.5 ms** |
-| `hgnetv2_b6` | 315.2 ms | 9444 MiB | 4.16 ms | 302.3 ms |
-
-**Mechanism, since 2x faster on 40% fewer MACs looks wrong.** `repvit_m2_3`
-spends 105 of its convolutions on depthwise kernels that carry only 0.050 of its
-4.520 GMAC. A depthwise convolution moves a whole feature map to do one multiply
-per element, so it is bandwidth-bound and leaves tensor cores — and an NPU's MAC
-array — idle. HGNetv2 is dense 3x3 convolutions in parallel branches that get
-concatenated: high arithmetic per byte moved, which is what the silicon is built
-for.
-
-**Deployability — this is the axis where it is not close.** Backbone-only ONNX
-op counts, after folding the traced shape ops at batch 1:
-
-| | Conv | `Erf` (GELU) | `ReduceMean`+`Sigmoid` (SE) | `BatchNormalization` | nodes |
-|---|---|---|---|---|---|
-| `repvit_m2_3` | 266 | 55 | 24 + 24 | 51 | 789 |
-| `repvit_m2_3` fused | 215 | 55 | 24 + 24 | 0 | 585 |
-| `hgnetv2_b4` | 80 | **0** | **0** | 0 | **144** |
-
-`hgnetv2_b4`'s trunk is `Conv` / `Relu` / `Concat` / `MaxPool` / `Pad` and
-nothing else — the intersection of every accelerator's op set. `Erf` is absent
-from several NPU op sets, so it falls back to CPU or gets approximated.
-Squeeze-excite needs a *global* spatial reduce, which breaks tile-local dataflow
-and forces a full-feature-map round trip through memory.
-
-int8 post-training quantisation, measured as SQNR on the trunk's own feature map
-(ORT `quantize_static`, QDQ, uint8 activations, 64 real COCO crops to calibrate,
-32 held out). SQNR = `10 log10(var(fp32) / var(fp32 - int8))`; higher is better.
-
-| backbone | per-tensor dB / cos | per-channel dB / cos |
-|---|---|---|
-| `repvit_m2_3` | −0.49 / 0.465 | 0.22 / 0.504 |
-| `repvit_m2_3` fused | −0.38 / 0.492 | 0.58 / 0.588 |
-| `hgnetv2_b0` | 3.50 / 0.732 | 5.89 / 0.847 |
-| `hgnetv2_b2` | 2.22 / 0.646 | 9.95 / 0.939 |
-| `hgnetv2_b4` | 7.92 / 0.918 | **15.14 / 0.984** |
-| `hgnetv2_b6` | 5.66 / 0.889 | 9.55 / 0.954 |
-
-A 14.6 dB gap, and fusing barely moves RepViT. At cosine 0.588 the int8 feature
-map is essentially uncorrelated with the fp32 one — RepViT would need
-quantisation-aware training to ship in int8; `hgnetv2_b4` at 0.984 would not.
-**Mechanism:** HGNetv2's activations are all ReLU, so they are non-negative and
-an unsigned int8 grid covers them exactly. RepViT's depthwise kernels have
-per-channel norms spanning orders of magnitude, and its GELU and SE-gate outputs
-are long-tailed, so one scale cannot cover them. These are deliberately naive
-PTQ settings — min/max calibration, 64 images — because that is what a mobile
-toolchain does first; the absolute values are low for everyone and the *gap* is
-the finding.
-
-**The export path survives unchanged.** Built a synthetic `hgnetv2_b4`
-checkpoint plus `run_config.json` and ran `tools/pth_to_onnx.py` on it with no
-edits: torch↔ONNX parity **5.96e-08** against its 1e-3 gate, `cliff_focal` and
-`bound_scales` read back correctly, metadata stamped. `backbone_feat_dim` is
-informational only — `InstantHMRStudent` reads the real width from
-`backbone.num_features` — so the 2048-channel HGNetv2 map needs no config edit.
-
-**Separate finding, independent of which backbone wins: nothing in this repo
-calls `RepViT.fuse()`.** timm ships the multi-branch training graph and the
-exporter never reparameterises it, so every ONNX we have shipped carries 51
-un-foldable `BatchNormalization` nodes and 204 extra nodes. Fusing is free:
-193.6 → 142.5 ms per training step and 4.76 → 3.55 ms ONNX B1. Worth taking even
-if we stay on RepViT.
-
-**Integration cost: zero code.** `--backbone` already exists in
-`train_distill_jz.py`, `run_config.json` records it, and
-`config_from_checkpoint` reads it back, so `benchmark/eval_3dpw_ckpt.py`,
-`benchmark/eval_emdb_ckpt.py`, `tools/pth_to_onnx.py` and
-`instanthmr/inference.py` all need nothing. Two operational steps only:
-
-```sh
-# 1. warm the timm cache on prepost — compute nodes run HF_HUB_OFFLINE=1
-sbatch --partition=prepost --time=03:00:00 --job-name=warm --output=warm_%j.out \
- --account=vsi@v100 --wrap \
- "source \$WORK/InstantHMR/datasets_pipeline/jeanzay/jz_env.sh && \
- python -u instanthmr_distill_train/train_distill_jz.py --warm-cache \
- --backbone hgnetv2_b4 --data_root \$DATA_ROOT"
-
-# 2. add the flag to EXTRA_TRAIN_ARGS; everything else stays at gen-6
-EXTRA_TRAIN_ARGS="--preset v2 $COMMON6 --backbone hgnetv2_b4"
+teacher: K = W_joint @ J + W_vertex @ V
+student: K ≈ W_fitted @ J
```
-If the proxy refuses `cdn-lfs.hf.co`, the weights are 76 MB and already on the
-laptop — `rsync` `models--timm--hgnetv2_b4.ssld_stage2_ft_in1k` the same way
-`JEAN_ZAY.md` describes for RepViT.
-
-**Memory goes down, so the partition question does not reopen.** `hgnetv2_b4`
-peaks at 3498 MiB at batch 64 against `repvit_m2_3`'s 8549 (synthetic step, no
-MHR forward kinematics and no `--w-verts`), so `v100-16g` stays comfortable and
-CLAUDE.md's "re-add `--constraint=v100-32g` if a bigger backbone pushes past
-~12 GiB" does not trigger. `hgnetv2_b6` at 9444 MiB would need watching.
-
-**Plan — launched 2026-09-10 as generation 7.** Generation 6 answered the
-`--w-verts` question first (`g6bv_s`, baseline preset + surface term, 41.67 mm
-J14 PA), so the trunk is now the only axis moving: **2 seeds per backbone**
-(`repvit_m2_3` and `hgnetv2_b4`) on the **full corpus including SA-1B**,
-everything else held at gen-6's winning arm. Two seeds, not the three written
-here originally — the seed spread is wide enough that two will only resolve a
-gap of a few mm, so read a sub-2 mm result as "not separated". The launch block,
-the mixture and the local verification table are in
-`datasets_pipeline/jeanzay/STATUS.md`, "Generation 7".
-
-Two things SA-1B changes, both already handled but both easy to get wrong:
-
-* **The mixture must be named explicitly.** `sqrt` hands SA-1B 0.692 of every
- batch on the strength of 3.4 M crops. Extend `MIX6` rather than falling back
- to a rule.
-* **`joints_2d_vis` is mandatory.** SA-1B stores unobserved 2D keypoints as
- literal `(0, 0)`; only 8% of its hand keypoints are real. The mask is already
- consumed by `loss_2d_native`, `loss_2d_simcc` and `loss_reproj` and defaults
- to all-ones elsewhere, so nothing else changes — but a folder built without it
- would drag every predicted hand keypoint to the top-left corner. See "The
- SA-1B label trap" in `datasets_pipeline/README.md`.
-
-The reported number stays **J14 PA-MPJPE, adapter-applied, on the H36M GT**. The
-overfit test above measures fitting, not generalisation, and does not substitute
-for it.
-
-### 13. PVE has no metric yet — only a loss
-
-`--w-verts` supervises the surface (it is the fix half of item 9), but nothing
-*reports* it. Competitors quote PVE, so a submission needs it.
-
-The pieces are in place and the awkward part is already solved: the vertex
-subset is drawn from the rig's **own** 18,439-vertex mesh, not from a separate
-LOD asset, so predicted and teacher vertices share topology and vertex ids and
-the correspondence is exact — no cross-topology fitting, which is what makes PVE
-comparisons between rigs contentious. `MHRForwardPass.get_joints_and_vertices`
-returns them.
-
-**What is missing**: 3DPW ships SMPL, and MHR is not SMPL. A published PVE
-number needs an MHR→SMPL surface correspondence with the same care the J14
-adapter needed — float64 and a truncated `lstsq`, fitted on train sequences and
-never refitted on the reported split. Until that exists, PVE can be reported
-against the *teacher* on held-out crops, which measures distillation fidelity
-and is honest as long as it is labelled as such.
-
-**Do not raise `n_verts` for the metric.** 595 vertices is a sampling choice for
-the loss; a reported PVE should use all 18,439, which costs 21.6 ms per batch-64
-forward and is irrelevant at evaluation time.
-
-### 14. Do the tanh size bounds block extreme poses? — measured, they do not
-
-Raised because `vid4.mp4` looked poorly fitted after `--bound-scales` landed.
-**The bound cannot be the cause, and here is why.**
-
-`--bound-scales` squashes two slices through a `tanh`: `mhr_params[0:3]` (root
-translation) and `[130:204]` (the six `*_flexible` size parameters and the 68
-bone scales) — 77 parameters, all of them body SIZE. The 124 local joint
-**rotations** at `[6:130]`, which is what a yoga pose actually is, and the root
-rotation at `[3:6]` are **not bounded at all**.
-
-Measured over 4,000 teacher annotations from each of the six local splits —
-1,848,000 bounded targets in total:
-
-| split | targets outside the bound | worst \|atanh\| | max \|rotation param\| |
-|---|---|---|---|
-| 3dpw | 0.0000% | 0.63 | 2.64 |
-| aic | 0.0000% | 0.90 | 2.94 |
-| coco | 0.0000% | 0.97 | 2.94 |
-| harmony4d | 0.0000% | 0.59 | 2.56 |
-| mpii | 0.0000% | 0.80 | 2.94 |
-| distill_mix | 0.0000% | 1.63 | 2.84 |
-| **all** | **0 / 1,848,000** | **1.63** | |
-
-Not one target is clipped, and the worst sits at `|atanh| = 1.63`, i.e.
-`tanh = 0.926` — a value a linear head reaches without saturating. The one real
-cost is gradient damping, `1 - tanh²`, and on the teacher distribution that is
-0.994x at the median, 0.875x at p99 and 0.733x at worst. Harmless.
-
-**So look elsewhere for the `vid4.mp4` regression.** Two candidates, in order:
-
-1. **A stale ONNX.** `models/instanthmr_mhr_only_ckpt90.onnx` and
- `models/instanthmr.onnx` carry **no `metadata_props` at all**, so
- `instanthmr.inference` cannot know how they were conditioned and falls back
- to the pixel-CLIFF form with a `1.00 x diag` focal. Feeding a focal-aware
- model that vector raises nothing and silently mis-places the person in depth.
- Every 2026-09-07 export (`models/instanthmr_mhr_only.onnx`, `models/gen5/*`)
- does carry `cliff_focal=true, bound_scales=true`. **Re-render `vid4.mp4` with
- a stamped export before treating this as a model regression.**
-2. **The deployment focal fallback.** With no annotation to read a focal from,
- `demo.py` uses `1.05 x diag` for focal-aware graphs. If the real camera is
- far from that, depth is wrong — `--focal PX` overrides it.
-
-
-### 15. Host-RAM OOM from the pair index — fixed 2026-09-08
-
-Four generation-6 jobs (1857969/70/73/75, then 1884647/48/50) were OOM-killed
-between epochs 14 and 33, at 73.9-76.0 GiB per node against a `mem=160000M`
-cgroup. **Host RAM, not CUDA**: `State=OUT_OF_ME+`, `ExitCode 0:125`,
-`Detected N oom_kill events`, `tasks 0-3: Out Of Memory`. The only Python output
-was a dataloader worker dying of `ConnectionResetError` in
-`multiprocessing/resource_sharer.py` — the symptom of the killer, not a cause.
-
-**Cause.** A list of `(PosixPath, PosixPath)` pairs measures **694 bytes per
-pair**, so the 4,231,947-crop index is 2.94 GB per process.
-`build_jz_loaders` sets `persistent_workers=True` with `num_workers=9` per rank,
-so a node holds 4 parents + 36 workers = 40 processes. Fork shares those pages
-copy-on-write, but CPython writes a refcount into an object header to *read* it,
-so each worker converts the shared index into a private copy as it samples —
-gradually, which is why the kill lands hours in rather than at startup.
-
-**Fix.** `PairIndex` stores the index as three contiguous numpy byte arrays and
-rebuilds the two `Path`s on access (~2 us against a ~6 ms `__getitem__`):
-
-| | bytes/pair | 4.23 M crops |
-|---|---|---|
-| `list` of `Path` pairs | 694 | 2.94 GB/process |
-| `PairIndex` | **87** | **0.37 GB/process** |
-
-8.0x smaller, and numpy arrays carry no per-element refcounts so forks stay
-shared. Verified: 961,139 pairs compared, **0 mismatches**, iteration order
-identical, mixture table unchanged (45.0 / 22.5 / 25.0 / 7.5), slices stay
-compact. `SAM3DStudentDataset` copies only real lists, never a `PairIndex`.
-
-**Two things this cost a day to learn.**
-
-*The `--w-verts` arms died first and that was a red herring.* The surface term
-adds ~2 GiB per node (measured +0.108 GiB/rank at construction, +0.16 GiB steady
-state), so at 74 GiB it decided the *ordering* of four deaths, not whether they
-happened. A 400-step local test showed no leak at all: allocated 1.276 /
-reserved 4.250 / peak 4.123 GiB GPU and 2.67 GiB host, flat from step 50 to 400
-and matching the no-verts arm to three decimals. Correlation with a change is
-not causation by it when every arm shares the real cause.
-
-*`sacct` MaxRSS undersamples.* It reports ~74 GiB against a 156 GiB limit and
-the OOM-killer still fires, both because it samples at ~30 s and because RSS
-double-counts pages shared between a rank and its workers. Do not read "half the
-limit" as "safe".
-
-**Related, same session**: `--constraint=v100-32g` is now in
-`52_train_ddp.slurm` itself, not on the sbatch line — the self-chain resubmits
-with only `--job-name`, so a CLI constraint survives exactly one job. And the
-epoch log now reports peak GPU as the max over all ranks; the "9.3 GiB" figure
-that justified running unconstrained was rank 0's alone, while the ranks that
-OOM'd on 16 GB cards were at 14.99 GiB.
+This is **not reconstructing joint angles from mesh vertices**. Angles remain
+network predictions, and joints remain FK outputs. Mesh-surface landmarks
+depend on shape and skinning as well as skeletal pose; replacing them with a
+skeleton-only fit introduces an approximation.
+
+Verified examples from the actual DINOv3 checkpoint's mapping:
+
+| landmark | contributing mesh vertices | contributing skeletal joints |
+|---|---:|---:|
+| nose | 8 | 0 |
+| left elbow | 21 | 0 |
+| left knee | 0 | 1 |
+| right wrist | 0 | 1 |
+| left small-toe tip | 24 | 0 |
+
+On 256 GT poses per split, the student's approximation differs from the
+teacher's mapping by **2.93–3.75 mm mean over the 30 nonfinger landmarks**.
+Those are operator discrepancies on identical GT geometry, not predicted
+benchmark gains. The full audit includes errors against the stored annotations.
+
+**Feasible reuse:** the teacher's first 70 mapping rows reference only **468
+distinct vertices**. `MHRForwardPass._build_vertex_subset` already restricts
+skinning to selected vertices, but currently chooses a farthest-point subset.
+Extend it to accept the exact mapping-supported indices, or their union with
+the current 595 vertex-loss indices. Do not assume the 595 already contain the
+468, or that changing the subset size alone selects the right vertices.
+
+## Implementation and verification
+
+- Load the teacher's fixed mapping and select its first 70 rows. Split its
+ vertex and joint columns, restrict vertex columns to their nonzero support,
+ and remap them to the compact skinning indices. Preserve units and axes.
+- Reuse one FK pass and the existing shape-dependent subset skinning; avoid
+ constructing full vertices solely for this landmark readout. The existing
+ shape-blend work and any other full-topology operations must still be
+ accounted for when profiling; 468 indices are not an end-to-end speed claim.
+- Prove landmark and gradient equivalence against the full reference mapping
+ on real GT and perturbed predictions, including shape changes. Measure the
+ union-subset cost at the actual training batch size.
+- Gate the new loss readout so current runs remain reproducible. A training-only
+ implementation adds no deployed operators, but deployment would retain the
+ approximate landmarks; evaluate accuracy through that retained deployment
+ path as well. If exact deployed landmarks are wanted, measure their phone
+ runtime cost separately and document the changed readout in comparisons.
+- Reuse this subset in decoder feedback only if item 4's latency gate passes;
+ it is not a requirement to run skinning between all decoder layers.
---
-
-### 16. Depth reweighting — tested 2026-09-09, rejected
-
-**Do not re-try this.** The hypothesis was that the depth channel is where the
-remaining 3D error lives, so weighting it harder should help. It does not, and
-the mechanism check says why.
-
-The reasoning that motivated it: measured on 2,048 unaugmented crops, the 3D
-error decomposes into an in-plane part that *equals the model's own 2D-head
-error* — COCO 20.6 mm against 20.5, Harmony4D 32.7 against 31.0 — and a depth
-part (23.9 / 37.5 mm) with no comparable bound. In-plane is saturated at the
-precision of the SimCC branch, which is also why anatomical routing (item 1) has
-nothing to win. But "no bound found" is not "headroom exists", and that was the
-error.
-
-Two interventions, both gated so the default is bit-identical:
-
-* **`depthw`** — `loss_3d_native`'s unsquared Euclidean made anisotropic,
- `sqrt(dx^2 + dy^2 + (lam*dz)^2)`, lam = 2.5.
-* **`fore`** — an explicit term on the out-of-plane extent `|dz|` of 8 rigid
- limbs (femur, tibia, humerus, forearm, both sides) indexed via
- `benchlib.joints.J14_FROM_MHR70`.
-
-Four tests, J14+adapter PA-MPJPE on 3DPW test, paired (same seed, same data
-order, only the loss differs):
-
-| regime | base | `depthw` | `fore` |
-|---|---|---|---|
-| 4,000-step fine-tune from `g6bv_s`, 2 seeds, stride 2 | 44.07 | 44.12 (+0.05) | 44.19 (+0.12) |
-| `hgnetv2_b4` from scratch, 18,000 steps, 2 seeds, stride 4 | 64.06 | 65.09 (+1.03) | 64.25 (+0.19) |
-
-Everything is inside noise (base seed spread 0.55 mm in the fine-tune, 1.32 mm
-from scratch), and every arm is on the wrong side of its control.
-
-**The diagnostic is the actual finding.** Weighting the depth residual 2.5x
-harder moved the measured depth error by **0.3% on one seed and 0.9% the wrong
-way on the other** (1,024 unaugmented COCO crops, in-plane/depth split per
-checkpoint). The loss did not do what it was built to do. That is what you expect
-if the model is already at the information limit of a single 224 crop: in-plane
-is capped by the 2D branch's precision, depth by monocular ambiguity. Both
-channels are saturated, for different reasons.
-
-**Corollary.** No loss and no architecture that re-routes *existing* image
-evidence can move depth. Only new information can — a second view, temporal
-context (item 4), or a known focal (item 3). That is a useful constraint on the
-roadmap, not just a dead end.
-
-*Incidental:* `hgnetv2_b4` from scratch runs at 0.062-0.068 s/it at batch 32 on
-an RTX 4070, 1.6x faster than `repvit_m2_3`'s 0.107, consistent with item 12.
-18,000 steps on the 92k local subset reaches ~64 mm, so this budget is a paired
-A/B instrument only — the absolute number means nothing.
-
----
-## Evaluation hygiene the audit raised
-
-Not code defects, but they gate a submission.
-
-* **Sequence manifests.** The local corpus contains all 3DPW sequences in
- `sam3d_distill_mix`; cluster-trained checkpoints use a different, clean
- corpus, but the loader does not *enforce* the separation. Save a
- source-image / sequence manifest and exclude explicitly — including for any
- offline regressor or adapter fitting.
-* **Adapter provenance.** The fixed teacher-derived J14 adapter is fitted on
- 3DPW-train ground truth. That is target-dataset supervision and must be
- disclosed. Test it on held-out sequences; never refit on the reported split.
- The fitting script's halfway split is not an explicit sequence grouping.
-* **Protocol parity.** The harness uses GT-derived boxes, its own crop
- expansion, a 60%-inside-image filter and its own person-frame list. Using the
- standard SMPL/H36M reference does not by itself make the protocol identical
- to a published one. Report GT vs detected boxes and known vs estimated
- intrinsics separately.
-* **Test-set monitoring.** `--val-3dpw-test` is logged and never used for
- selection, which is right, but human design decisions can still overfit a
- number you watch every epoch.
-* **Metrics.** PA-MPJPE removes translation, rotation and scale; root-relative
- MPJPE removes translation. Neither sees `cam_trans` drift, body-size error or
- jitter. Claims about stability need acceleration and bone-length-variance
- numbers, and low acceleration alone can just mean over-smoothing — compare
- against GT motion and report lag.
-* **Seed spread.** The documented 34–49 mm spread is for the 70-keypoint
- metric, not for J14+adapter. Measure the spread of the metric you quote
- before calling any gap real.
+# 6. Camera-ray conditioning and crop-camera parameterization
+
+**Status:** open, focused on absolute pose and subject placement; keep the
+efficient encoder and benchmark any inference overhead.
+
+**Focal-aware conditioning is already present in generation 7** via
+`--cliff-focal`. A learned projection already embeds the three conditioning
+values, including crop size and offset normalized by focal length, into the
+queries. Do not describe this task as simply adding focal information or adding
+a learnable focal token; neither captures the remaining teacher difference.
+
+Two distinct experiments:
+
+1. **Spatial camera rays.** For each feature-grid position, recover its original
+ image coordinate `(u, v)` through the crop transform and compute its viewing
+ direction, proportional to `((u-cx)/fx, (v-cy)/fy, 1)`. Here `cx, cy` denote
+ the camera principal point, not the person-box centre. The teacher encodes
+ these rays and fuses them with image features. Test a lightweight fusion at
+ the student's existing feature resolution; do not blindly copy the wide
+ teacher module. Rays are derived from camera/crop metadata, not learned
+ focal estimates. Keep augmentations, intrinsics and inference fallback
+ calibration consistent. With ideal centred square crops, much of this is
+ derivable from existing CLIFF plus grid position, so gains are not automatic.
+2. **Separate crop-camera head.** The student directly predicts translation in
+ metres; the teacher predicts crop scale and lateral offsets, then converts
+ them to full-camera translation using focal length and box geometry. Assess
+ that parameterization separately from ray embeddings. Check coordinate/sign
+ conventions, depth stability and train/export/inference consistency.
+
+Measure absolute camera-space error, depth/placement stability and reprojection
+as well as body MPJPE. PA-MPJPE removes translation, rotation and scale and is
+not the main success metric for these changes. Keep the current conditioning
+and camera path available as the matched control.
+
+**Deferred by user, 2026-09-12:** pelvis/wrist-relative supervision and
+annotation-aware weighting (including the related uncertainty-weighting
+proposal) are not current implementation tasks.
diff --git a/instanthmr_distill_train/README.md b/instanthmr_distill_train/README.md
index 1867bb5..11c20a4 100644
--- a/instanthmr_distill_train/README.md
+++ b/instanthmr_distill_train/README.md
@@ -1,4 +1,30 @@
-# InstantHMR Distillation — training scripts
+# InstantHMR — training scripts
+
+## What the labels are — read this first
+
+**Training is supervised by the released ground-truth annotations of
+[`facebook/sam-3d-body-dataset`](https://huggingface.co/datasets/facebook/sam-3d-body-dataset)**
+— the human MHR fits Meta used to build SAM 3D Body. They are **not** teacher
+inference. Every run in this repo's result tables, and the whole cluster corpus
+(`sam3d_gt_sa1b` / `_aic` / `_harmony4d` / `_coco` / `_mpii`), is of this kind.
+
+Distilling from the `facebook/sam-3d-body-dinov3` model is a supported
+**option**, not the default: `tools/annotate_dataset.py` runs it over your own
+images and is the only thing that writes `data/sam3d_distill_mix/`. Reach for it
+to label imagery the released dataset does not cover, and note the cost — a
+student trained on teacher output cannot exceed the teacher, while one trained on
+the ground truth can.
+
+Two naming traps that follow from the history:
+
+- **"distill" in a filename means nothing about the labels.**
+ `train_distill_mhr_only.py`, `train_distill_jz.py` and the
+ `data/sam3d_distill_mix/` directory keep the name from when distillation was
+ the only path. Only `annotate_dataset.py` produces teacher labels.
+- **"teacher" further down this file usually means "the training target".** Where
+ a number was measured against a label — parameter ranges, oracle ablations,
+ the `adapter_j14_h36m_teacher.npz` fit — the label is the dataset's ground
+ truth unless the text says `annotate_dataset.py` explicitly.
## Which script
@@ -402,6 +428,96 @@ Do not expect the benchmark to move; expect the conditioning to stop lying.
The rotation half of the same correction is **not** settled: see `docs/todo.md`
item 11.
+## `--cont-head` — the teacher's continuous regression space (added 2026-09-12)
+
+**What it changes.** Without it, the student emits the rig-native 204-vector
+straight out of one `nn.Linear`: raw Euler angles and 68 free bone scales.
+SAM 3D Body does not. Its head emits a *continuous* parameterisation and
+converts to the 204-vector inside the head, before MHR sees anything. This flag
+reproduces that pathway.
+
+| the head emits | dims | converted, inside the model, to |
+|---|---|---|
+| root rotation, 6D | 6 | 3 extrinsic-XYZ Euler angles |
+| body pose, continuous | 260 | 130 joint angles |
+| identity coefficients | 45 | passed through unchanged |
+| bone-scale PCA coefficients | 28 | 68 bone scales |
+| two hand blocks | 2 x 54 | the 54 finger channels, overwritten |
+
+447 numbers total — the teacher's 519 minus the 72-dim face block its own
+`forward()` multiplies by zero. They come from a two-layer MLP at the student's
+own 512 width on LayerNorm'd pose-token features (`nn.TransformerDecoder` is
+built here with `norm_first=True` layers and no final norm, so its output is
+unnormalised), plus a learned initial estimate. Camera translation moves to its
+own 3-dim head, unchanged in parameterisation.
+
+**Nothing downstream changes.** The model still returns 204 MHR parameters, 45
+identity coefficients, camera translation and 70 2D keypoints; every loss, the
+forward kinematics, `tools/pth_to_onnx.py` and `instanthmr.inference` see the
+contract they already saw. `config_from_checkpoint` reads the flag back out of
+the weights (the `cont_head.*` buffers ARE the flag), so an export needs no
+argument.
+
+**Why 260 for the body.** 23 three-DoF joints x 6D (138) + 58 one-DoF hinges x
+(sin, cos) (116) + 6 raw translation channels (6). Those last six are the
+`*_length/_width_flexible` parameters at `130:136` — body SIZE, not rotation —
+and **the continuous space does not bound them**, so they keep their own tanh
+against the rig limits exactly as under `--bound-scales`.
+
+**The root convention is the rig's, not the teacher's literal call.**
+`mhr_head.forward()` decodes the root with `roma.rotmat_to_euler("ZYX", R)`.
+Measured against `checkpoints/mhr_model.pt` (md5-identical to the teacher's own
+`assets/mhr_model.pt`): driving `model_params[3:6]` and rigid-fitting the 125
+joints it moves recovers **extrinsic XYZ** to 0.00000–0.014 deg, while roma's
+`"ZYX"` of the same triple is **36–92 deg away** — roma returns the triple
+Z-first, so it is the reverse of the rig's. Either convention round-trips and
+both would train; the rig's is used because only then is the intermediate
+rotation matrix the body's *actual* root rotation, which is what makes the
+accompanying chordal loss a physical angular error rather than a distance in a
+permuted space. All of this lives in `mhr_cont.py`, which is a transcription of
+the teacher's own functions verified at **0.000e+00**.
+
+**Bone scales are bounded as coefficients, and that ends the runaway.**
+`--bound-scales` clamps the 68 expanded scales coordinatewise, which is wrong
+for a PCA head: an arbitrary point of the 68-dim box is generally not in the
+24-dimensional column space `scale_comps` spans, so the clamp silently leaves
+the subspace. The bound moves onto the 28 coefficients, symmetric about zero so
+an untrained head sits exactly on `scale_mean`. Measured: feeding the head
+`N(0, 25)` — the regime that produced a 28,300 km skeleton and killed four runs
+at epochs 24–32 — now gives a **2.66 m** skeleton, scales still in-subspace to
+1.2e-07.
+
+**The root loss changes with it** (`root_rot_loss`, set by the same flag) from
+SmoothL1 on the Euler triple to the squared chordal distance
+`||Rp - Rt||_F^2 / 4`, on the rotation the head built rather than one rebuilt
+from the decoded Euler triple — `atan2` in the backward path is unstable, and
+2.25% of corpus roots sit within 0.1 rad of the ±pi/2 gimbal. Divided by the six
+root parameters the old term averaged over, so it is budget-neutral: 0.00003 vs
+0.00003 at 1 deg, 0.00063 vs 0.00063 at 5 deg, and 0.333 vs 0.437 at 179 deg —
+matched where it matters, saturating instead of growing.
+
+**Cost: none worth measuring.** +0.37 M parameters, and ONNX CPU single-thread
+batch 1 goes 102.03 -> 102.49 ms on `repvit_m2_3` and 64.55 -> 64.19 ms on
+`hgnetv2_b4`. The 980 extra graph nodes are all elementwise on 447 numbers.
+
+**Before using it:**
+
+```sh
+python tools/verify_cont_head.py --data_root data # 24 gates, exits non-zero on any failure
+python tools/ddp_smoke.py --backbone <name> --w-verts 0.35 --cont-head --data_root data/sam3d_gt_mpii
+```
+
+`instanthmr_distill_train/assets/mhr_cont_head.npz` (30 KB) carries the
+teacher's scale and hand bases plus the measured coefficient bounds;
+`tools/build_cont_head_assets.py` rebuilds it from the teacher checkpoint and
+the corpus. **It is a new file and must be rsynced**, or the job dies at
+startup.
+
+Default off. With it off every tensor in the trainer is bit-identical to the
+previous commit — verified: 1544 state_dict keys, same sha256, forward and all
+12 loss terms 0.000e+00 apart on 48 real augmented samples, under both
+`--losses legacy` and `--losses rebalanced`.
+
## Tried and rejected: detaching the 2D head (2026-09-06 / 09-07)
**Do not retry this without reading the failure mode.** The flag is gone from
diff --git a/instanthmr_distill_train/assets/mhr_cont_head.npz b/instanthmr_distill_train/assets/mhr_cont_head.npz
new file mode 100644
index 0000000..84c1cdd
Binary files /dev/null and b/instanthmr_distill_train/assets/mhr_cont_head.npz differ
diff --git a/instanthmr_distill_train/mhr_cont.py b/instanthmr_distill_train/mhr_cont.py
new file mode 100644
index 0000000..80f561a
--- /dev/null
+++ b/instanthmr_distill_train/mhr_cont.py
@@ -0,0 +1,231 @@
+"""SAM 3D Body's continuous MHR pose space, vendored for the student.
+
+The teacher's head does not emit the rig-native 204-vector. It emits a
+continuous parameterisation and converts, *inside the head*, before MHR sees
+anything:
+
+ 6 global rotation, 6D -> 3 Euler angles
+ 260 body pose, continuous -> 133 angles, kept as [:130]
+ 45 identity ("shape") -> passed through
+ 28 bone-scale PCA coefficients -> 68 bone scales
+ 108 two 54-dim hand blocks -> overwrite the finger channels
+ (72 face, multiplied by zero in the teacher and omitted here)
+
+The 260 decompose as 23 three-DoF joints x 6D (138) + 58 one-DoF hinges x
+(sin, cos) (116) + 6 raw translation channels (6). Those last six are the
+`*_length/_width_flexible` parameters at `130:136` of the 204-vector -- body
+SIZE, not rotation, and the continuous space does NOT bound them.
+
+Everything here is a transcription of
+`sam_3d_body/models/modules/mhr_utils.py` and `.../geometry_utils.py`, verified
+elementwise against those originals. Two deliberate differences:
+
+* **No `roma`.** It is not a dependency of this repo and `rotmat_to_euler` does
+ not export to ONNX. The one place the teacher uses it -- the global rotation
+ -- is replaced by the rig's own convention; see ROOT CONVENTION below.
+* **Scatters are permutations, not boolean masks.** The teacher writes results
+ into a zero tensor through `x[..., mask] = v`, which traces to a dynamic
+ `index_put` in ONNX. The index sets here partition their output exactly, so
+ the same thing is a `cat` followed by one fixed `index_select`.
+
+ROOT CONVENTION -- measured, and NOT what the teacher's source literally says.
+`mhr_head.forward()` calls `roma.rotmat_to_euler("ZYX", R)`. Measured against
+the rig in `checkpoints/mhr_model.pt` (byte-identical to the teacher's own
+`assets/mhr_model.pt`, md5 8f21d804...): driving `model_params[3:6]` and
+rigid-fitting the 125 joints it moves recovers a rotation that matches
+**extrinsic XYZ** -- `batch6DFromXYZ`, i.e. `Rz(c) @ Ry(b) @ Rx(a)` -- to
+0.00000-0.014 degrees, while `roma.euler_to_rotmat("ZYX", .)` of the same triple
+is 36-92 degrees away. roma's uppercase "ZYX" returns the triple in Z-first
+order, so it is the *reverse* of the rig's: feeding it `(0.3, 0.5, 0.7)` through
+the teacher's own forward returns `(0.7, 0.5, 0.3)`.
+
+Either convention round-trips if encode and decode agree, so both would train.
+This module uses the rig's, because only then is the intermediate rotation
+matrix the body's actual root rotation -- which is what makes a geodesic or
+Frobenius loss on it a physical angular error rather than a distance in a
+permuted space. Reproducing the `"ZYX"` call literally would keep the pathway
+identical in shape and silently make that loss meaningless.
+"""
+
+from __future__ import annotations
+
+import torch
+import torch.nn.functional as F
+
+# --- the teacher's index tables, verbatim -----------------------------------
+# fmt: off
+_IDX_3DOF = [(0, 2, 4), (6, 8, 10), (12, 13, 14), (15, 16, 17), (18, 19, 20), (21, 22, 23), (24, 25, 26), (27, 28, 29), (34, 35, 36), (37, 38, 39), (44, 45, 46), (53, 54, 55), (64, 65, 66), (85, 69, 73), (86, 70, 79), (87, 71, 82), (88, 72, 76), (91, 92, 93), (112, 96, 100), (113, 97, 106), (114, 98, 109), (115, 99, 103), (130, 131, 132)]
+_IDX_1DOF = [1, 3, 5, 7, 9, 11, 30, 31, 32, 33, 40, 41, 42, 43, 47, 48, 49, 50, 51, 52, 56, 57, 58, 59, 60, 61, 62, 63, 67, 68, 74, 75, 77, 78, 80, 81, 83, 84, 89, 90, 94, 95, 101, 102, 104, 105, 107, 108, 110, 111, 116, 117, 118, 119, 120, 121, 122, 123]
+_IDX_TRANS = [124, 125, 126, 127, 128, 129]
+# Degrees of freedom per hand joint, in the hand's own joint order.
+_HAND_DOFS = [3, 1, 1, 3, 1, 1, 3, 1, 1, 3, 1, 1, 2, 3, 1, 1]
+# fmt: on
+
+N_BODY_CONT = 260 # 23*6 + 58*2 + 6
+N_BODY_PARAMS = 133 # the 130 that reach the rig, plus 3 jaw values
+N_HAND_CONT = 54
+N_HAND_PARAMS = 27
+N_ROOT_6D = 6
+N_SHAPE = 45
+N_SCALE_COEFF = 28
+# 6 + 260 + 45 + 28 + 2*54 = 447. The teacher's 519 also carries a 72-dim face
+# block that `forward()` multiplies by zero; omitting it is the only difference.
+N_HEAD_OUT = N_ROOT_6D + N_BODY_CONT + N_SHAPE + N_SCALE_COEFF + 2 * N_HAND_CONT
+
+_N3 = len(_IDX_3DOF) * 3 # 69 three-DoF angles
+_N1 = len(_IDX_1DOF) # 58 hinges
+_NT = len(_IDX_TRANS) # 6 raw translation channels
+
+
+def _inverse_permutation(order: list[int], n: int) -> torch.Tensor:
+ """`out[order[i]] = v[i]` expressed as `out = v[inv]`."""
+ assert sorted(order) == list(range(n)), "index tables must partition the output"
+ inv = [0] * n
+ for i, o in enumerate(order):
+ inv[o] = i
+ return torch.tensor(inv, dtype=torch.long)
+
+
+_BODY_ORDER = [i for t in _IDX_3DOF for i in t] + list(_IDX_1DOF) + list(_IDX_TRANS)
+_BODY_GATHER = torch.tensor(_BODY_ORDER, dtype=torch.long)
+_BODY_SCATTER = _inverse_permutation(_BODY_ORDER, N_BODY_PARAMS)
+
+# Hand masks, as gather indices. 3-DoF joints take 6 continuous dims each and
+# 1-/2-DoF joints take 2 per angle, so the continuous block is 2 x 27.
+_h_c3, _h_c1, _h_p3, _h_p1, _c, _p = [], [], [], [], 0, 0
+for _k in _HAND_DOFS:
+ (_h_c3 if _k == 3 else _h_c1).extend(range(_c, _c + 2 * _k)); _c += 2 * _k
+ (_h_p3 if _k == 3 else _h_p1).extend(range(_p, _p + _k)); _p += _k
+assert _c == N_HAND_CONT and _p == N_HAND_PARAMS
+_HAND_CONT_GATHER = torch.tensor(_h_c3 + _h_c1, dtype=torch.long)
+_HAND_PARAM_SCATTER = _inverse_permutation(_h_p3 + _h_p1, N_HAND_PARAMS)
+_HAND_CONT_SCATTER = _inverse_permutation(_h_c3 + _h_c1, N_HAND_CONT)
+_HAND_PARAM_GATHER = torch.tensor(_h_p3 + _h_p1, dtype=torch.long)
+
+
+# --- rotation primitives ----------------------------------------------------
+def rot6d_to_rotmat(x: torch.Tensor) -> torch.Tensor:
+ """(..., 6) first two rotation-matrix COLUMNS -> (..., 3, 3), Gram-Schmidt."""
+ a1, a2 = x[..., :3], x[..., 3:]
+ b1 = F.normalize(a1, dim=-1)
+ b2 = F.normalize(a2 - (b1 * a2).sum(-1, keepdim=True) * b1, dim=-1)
+ b3 = torch.cross(b1, b2, dim=-1)
+ return torch.stack((b1, b2, b3), dim=-1)
+
+
+def rotmat_to_rot6d(R: torch.Tensor) -> torch.Tensor:
+ """(..., 3, 3) -> (..., 6), the first two COLUMNS. Inverse of the above."""
+ return torch.cat([R[..., :, 0], R[..., :, 1]], dim=-1)
+
+
+def euler_xyz_to_rotmat(r: torch.Tensor) -> torch.Tensor:
+ """(..., 3) extrinsic-XYZ Euler -> (..., 3, 3), i.e. `Rz(c) @ Ry(b) @ Rx(a)`.
+
+ This is the rig's convention for `model_params[3:6]` -- see the module
+ docstring. Transcribed from the teacher's `batch6DFromXYZ(return_9D=True)`.
+ """
+ c, s = torch.cos(r), torch.sin(r)
+ cx, cy, cz = c[..., 0], c[..., 1], c[..., 2]
+ sx, sy, sz = s[..., 0], s[..., 1], s[..., 2]
+ zero = torch.zeros_like(cx)
+ rows = [
+ cy * cz, -cx * sz + sx * sy * cz, sx * sz + cx * sy * cz,
+ cy * sz, cx * cz + sx * sy * sz, -sx * cz + cx * sy * sz,
+ -sy, sx * cy, cx * cy,
+ ]
+ return torch.stack([r_ + zero for r_ in rows], dim=-1).unflatten(-1, (3, 3))
+
+
+def rotmat_to_euler_xyz(R: torch.Tensor) -> torch.Tensor:
+ """(..., 3, 3) -> (..., 3) extrinsic-XYZ Euler. Teacher's `batchXYZfrom6D` tail.
+
+ The gimbal-lock branch is a numeric blend, not control flow, so it traces.
+ """
+ sy = torch.sqrt(R[..., 0, 0] ** 2 + R[..., 1, 0] ** 2)
+ singular = (sy < 1e-6).to(R.dtype)
+ x = torch.atan2(R[..., 2, 1], R[..., 2, 2])
+ y = torch.atan2(-R[..., 2, 0], sy)
+ z = torch.atan2(R[..., 1, 0], R[..., 0, 0])
+ xs = torch.atan2(-R[..., 1, 2], R[..., 1, 1])
+ return torch.stack([x * (1 - singular) + xs * singular,
+ y,
+ z * (1 - singular)], dim=-1)
+
+
+def _cont6d_to_euler(c6: torch.Tensor) -> torch.Tensor:
+ """(..., n, 6) -> (..., n, 3) extrinsic-XYZ, the teacher's `batchXYZfrom6D`."""
+ x = F.normalize(c6[..., :3], dim=-1)
+ z = F.normalize(torch.cross(x, c6[..., 3:], dim=-1), dim=-1)
+ y = torch.cross(z, x, dim=-1)
+ return rotmat_to_euler_xyz(torch.stack([x, y, z], dim=-1))
+
+
+def _euler_to_cont6d(e: torch.Tensor) -> torch.Tensor:
+ """(..., n, 3) extrinsic-XYZ -> (..., n, 6). Teacher's `batch6DFromXYZ`."""
+ return rotmat_to_rot6d(euler_xyz_to_rotmat(e))
+
+
+# --- body -------------------------------------------------------------------
+def cont_to_body_params(cont: torch.Tensor) -> torch.Tensor:
+ """(B, 260) continuous -> (B, 133) MHR angles + 6 raw translation channels."""
+ assert cont.shape[-1] == N_BODY_CONT, cont.shape
+ a, b = 2 * _N3, 2 * _N3 + 2 * _N1
+ e3 = _cont6d_to_euler(cont[..., :a].unflatten(-1, (-1, 6))).flatten(-2, -1)
+ p1 = cont[..., a:b].unflatten(-1, (-1, 2))
+ e1 = torch.atan2(p1[..., 0], p1[..., 1])
+ parts = torch.cat([e3, e1, cont[..., b:]], dim=-1)
+ return parts.index_select(-1, _BODY_SCATTER.to(cont.device))
+
+
+def body_params_to_cont(params: torch.Tensor) -> torch.Tensor:
+ """(B, 133) -> (B, 260). Exact inverse of `cont_to_body_params` up to
+ the 6D representative chosen for each rotation."""
+ assert params.shape[-1] == N_BODY_PARAMS, params.shape
+ g = params.index_select(-1, _BODY_GATHER.to(params.device))
+ c3 = _euler_to_cont6d(g[..., :_N3].unflatten(-1, (-1, 3))).flatten(-2, -1)
+ a1 = g[..., _N3:_N3 + _N1]
+ c1 = torch.stack([a1.sin(), a1.cos()], dim=-1).flatten(-2, -1)
+ return torch.cat([c3, c1, g[..., _N3 + _N1:]], dim=-1)
+
+
+# --- hands ------------------------------------------------------------------
+def cont_to_hand_params(cont: torch.Tensor) -> torch.Tensor:
+ """(B, 54) continuous -> (B, 27) finger angles, in the hand's joint order."""
+ assert cont.shape[-1] == N_HAND_CONT, cont.shape
+ g = cont.index_select(-1, _HAND_CONT_GATHER.to(cont.device))
+ n3 = len(_h_c3)
+ e3 = _cont6d_to_euler(g[..., :n3].unflatten(-1, (-1, 6))).flatten(-2, -1)
+ p1 = g[..., n3:].unflatten(-1, (-1, 2))
+ e1 = torch.atan2(p1[..., 0], p1[..., 1])
+ return torch.cat([e3, e1], dim=-1).index_select(-1, _HAND_PARAM_SCATTER.to(cont.device))
+
+
+def hand_params_to_cont(params: torch.Tensor) -> torch.Tensor:
+ """(B, 27) -> (B, 54)."""
+ assert params.shape[-1] == N_HAND_PARAMS, params.shape
+ g = params.index_select(-1, _HAND_PARAM_GATHER.to(params.device))
+ n3 = len(_h_p3)
+ c3 = _euler_to_cont6d(g[..., :n3].unflatten(-1, (-1, 3))).flatten(-2, -1)
+ a1 = g[..., n3:]
+ c1 = torch.stack([a1.sin(), a1.cos()], dim=-1).flatten(-2, -1)
+ return torch.cat([c3, c1], dim=-1).index_select(-1, _HAND_CONT_SCATTER.to(params.device))
+
+
+# --- neutral initialisation -------------------------------------------------
+def neutral_head_output() -> torch.Tensor:
+ """(1, 447) whose decode is the rig's zero pose.
+
+ All-zeros is NOT neutral and not even valid: a zero 6D vector has no
+ rotation and a zero (sin, cos) pair sends `atan2` to a point where its
+ gradient is 0/0. Identity rotations encode as `[1,0,0, 0,1,0]` and a zero
+ hinge as `(0, 1)`.
+
+ The hand block stays at zero, matching the teacher: the hand conversion
+ consumes `hand_pose_mean + coeffs`, so zero coefficients decode the mean
+ hand pose, which is well away from the `atan2` singularity.
+ """
+ out = torch.zeros(1, N_HEAD_OUT)
+ out[:, :N_ROOT_6D] = torch.tensor([1.0, 0.0, 0.0, 0.0, 1.0, 0.0])
+ out[:, N_ROOT_6D:N_ROOT_6D + N_BODY_CONT] = body_params_to_cont(
+ torch.zeros(1, N_BODY_PARAMS))
+ return out
diff --git a/instanthmr_distill_train/train_distill_jz.py b/instanthmr_distill_train/train_distill_jz.py
index 89c5c0d..1572751 100755
--- a/instanthmr_distill_train/train_distill_jz.py
+++ b/instanthmr_distill_train/train_distill_jz.py
@@ -693,6 +693,7 @@ def train(args, cfg, rank, local_rank, world):
"lr": cfg.lr, "lr_arg": args.lr, "lr_scaling": args.lr_scaling,
"mix": args.mix, "steps_per_epoch": steps_per_epoch,
"backbone": cfg.backbone, "w_reproj": cfg.w_reproj,
+ "cont_head": cfg.cont_head, "root_rot_loss": cfg.root_rot_loss,
"geom_scale_max": cfg.geom_scale_max, "occl_p": cfg.occl_p,
"jpeg_p": cfg.jpeg_p, "cam_loss": cfg.cam_loss,
"losses": args.losses, "w_shape": cfg.w_shape,
@@ -1147,6 +1148,14 @@ def parse_args():
"farthest-point subset of the rig's own 18,439-vertex "
"mesh (default 595, lod6's count). Costs 0.6 ms per "
"batch-64 step against 16.7 ms for the full mesh.")
+ p.add_argument("--cont-head", dest="cont_head", action="store_true",
+ help="Reproduce the teacher's regression pathway: the head "
+ "emits 447 numbers in its continuous space (6D root, "
+ "260-dim body, 28 bone-scale PCA coefficients, two "
+ "54-dim hand blocks) through a two-layer MLP with a "
+ "learned neutral initial estimate, and converts to the "
+ "204-vector inside the model. Implies the rotation-matrix "
+ "root loss. Needs assets/mhr_cont_head.npz.")
p.add_argument("--losses", choices=("legacy", "rebalanced"), default="legacy",
help="Orthogonal to --preset, which only controls the input "
"pipeline and the absolute-pose terms. 'rebalanced' applies "
@@ -1186,6 +1195,9 @@ def build_cfg(args):
cfg.per_dataset_caps = {} # replaced by the weighted sampler
if args.backbone: cfg.backbone = args.backbone
if args.crop_centre_fix: cfg.crop_centre_fix = True
+ if args.cont_head:
+ cfg.cont_head = True
+ cfg.root_rot_loss = True
if args.w_verts is not None: cfg.w_verts = args.w_verts
if args.n_verts is not None: cfg.n_verts = args.n_verts
if args.w_simcc is not None: cfg.w_simcc = args.w_simcc
@@ -1294,6 +1306,8 @@ def main():
f"pose_beta={cfg.pose_beta} kp3d={cfg.kp3d_loss}@{cfg.w_keypoints3d} "
f"finger_w={cfg.finger_weight}"
if args.losses != "legacy" else ""))
+ log(f"head {'continuous (teacher pathway, 447 dims)' if cfg.cont_head else 'linear 204-vector'}"
+ + (f" | root loss = rotation matrix" if cfg.root_rot_loss else ""))
log(f"surface w_verts={cfg.w_verts}"
+ (f" over {cfg.n_verts} mesh vertices" if cfg.w_verts > 0 else " (off)")
+ f" | crop_centre_fix={cfg.crop_centre_fix}")
diff --git a/instanthmr_distill_train/train_distill_mhr_only.py b/instanthmr_distill_train/train_distill_mhr_only.py
index 73b035d..4e5b58d 100644
--- a/instanthmr_distill_train/train_distill_mhr_only.py
+++ b/instanthmr_distill_train/train_distill_mhr_only.py
@@ -65,6 +65,11 @@ from torch.utils.data import Dataset, DataLoader, random_split
from torch.optim.swa_utils import AveragedModel, get_ema_multi_avg_fn
import torchvision.transforms as transforms
+
+try:
+ import mhr_cont as MC
+except ImportError: # imported as a package rather than a script
+ from instanthmr_distill_train import mhr_cont as MC
import torchvision.transforms.functional as F_t
from torchvision.transforms import InterpolationMode
@@ -298,6 +303,25 @@ class DistillConfig:
bound_scales: bool = False
scale_bound_margin: float = 0.5
scale_bounds_path: str = str(PROJECT_ROOT / "assets/mhr_size_bounds.npz")
+
+ # --- the teacher's continuous regression pathway (--cont-head) ----------
+ # Off by default: with it off every tensor in this file is bit-identical to
+ # the previous commit, which generations 6 and 7 depend on as controls.
+ #
+ # On, the single `nn.Linear` that emits the rig-native 204-vector is
+ # replaced by the teacher's own space -- 6D root rotation, a 260-dim
+ # continuous body block, 28 bone-scale PCA coefficients and two 54-dim hand
+ # blocks -- converted to the 204-vector inside the model, so nothing
+ # downstream (losses, FK, export, deployment) sees a different contract.
+ # See instanthmr_distill_train/mhr_cont.py and docs/todo.md item 1.
+ cont_head: bool = False
+ cont_head_path: str = str(PROJECT_ROOT / "assets/mhr_cont_head.npz")
+ # Rotation-matrix loss on the root instead of SmoothL1 on its Euler triple.
+ # Set together with cont_head: a SmoothL1 in radians scores a prediction at
+ # -pi+eps against a target at pi-eps as a ~2pi error when it is a tiny
+ # rotation, and 13.59% of corpus root angles sit beyond 2.8 rad.
+ root_rot_loss: bool = False
+ w_root_rot: float = 1.0
use_amp: bool = True
ema_decay: float = 0.9998
early_stop_patience: int = 150
@@ -423,6 +447,15 @@ def config_from_checkpoint(state, ckpt_path=None, **overrides):
cfg.bound_scales = "scale_lo" in state
prov["bound_scales"] = "checkpoint weights"
+ # --cont-head replaces head_global with a ContMHRHead, so the buffers it
+ # registers ARE the flag -- more reliable than run_config.json, which a
+ # hand-copied checkpoint may not have beside it. Rebuilding the default
+ # head for these weights would fail loudly on the missing keys rather than
+ # silently, but only because the shapes differ; do not rely on that.
+ cfg.cont_head = any(k.startswith("cont_head.") for k in state)
+ cfg.root_rot_loss = cfg.cont_head
+ prov["cont_head"] = "checkpoint weights"
+
run_cfg = {}
if ckpt_path is not None:
rc = Path(ckpt_path).resolve().parent / "run_config.json"
@@ -883,6 +916,148 @@ def get_2d_sincos_pos_embed(embed_dim, grid_size):
pos_embed = torch.cat([pe_y, pe_x], dim=1)
return pos_embed.unsqueeze(0)
+class ContMHRHead(nn.Module):
+ """SAM 3D Body's continuous pose head, emitting the rig-native 204-vector.
+
+ 447 numbers come out of a two-layer MLP and are converted here, so the rest
+ of the model, every loss, the ONNX graph and `instanthmr.inference` all keep
+ the contract they already have: 204 MHR parameters and 45 identity
+ coefficients. (The teacher emits 519; the extra 72 are a face block its own
+ `forward()` multiplies by zero.)
+
+ 6 root rotation, 6D -> 3 extrinsic-XYZ Euler angles
+ 260 body pose, continuous -> 130 angles
+ 45 identity -> passed through
+ 28 bone-scale PCA coefficients -> 68 bone scales
+ 108 two 54-dim hand blocks -> overwrite the 54 finger channels
+
+ Three things are deliberate and each is measured in the file that owns it:
+
+ * **Root translation is emitted as exact zero**, not regressed. It is
+ `0.000e+00` in all 24,000 sampled annotations -- the person is placed by
+ `cam_trans` -- and the teacher's head does the same (`global_trans =
+ torch.zeros_like(...)`).
+
+ * **The bone-scale bound is on the 28 coefficients, not the 68 scales.**
+ Clamping after the PCA expansion is what `--bound-scales` does and it is
+ wrong here: an arbitrary point of the 68-dim box is generally not in the
+ 24-dimensional column space `scale_comps` spans, so a coordinatewise tanh
+ silently leaves the subspace. The coefficient limits come from the corpus
+ (`tools/build_cont_head_assets.py`; GT projection residual 2.1e-08).
+
+ * **The six `*_length/_width_flexible` channels at `130:136` are bounded
+ separately.** The continuous body space carries them raw -- they are
+ joint TRANSLATION channels, not rotations -- so they are exactly as
+ unbounded here as they were before, and they reach 375 m on their own.
+
+ Assembly is one `index_select` over a fixed 136-long gather table rather
+ than three masked writes: `x[..., idx] = v` traces to a scatter, a gather
+ exports everywhere and is its own documentation.
+ """
+
+ def __init__(self, cfg, d_model: int):
+ super().__init__()
+ a = np.load(cfg.cont_head_path)
+ self.register_buffer("scale_mean", torch.from_numpy(a["scale_mean"]).float())
+ self.register_buffer("scale_comps", torch.from_numpy(a["scale_comps"]).float())
+ self.register_buffer("coeff_lo", torch.from_numpy(a["coeff_lo"]).float())
+ self.register_buffer("coeff_hi", torch.from_numpy(a["coeff_hi"]).float())
+ self.register_buffer("hand_mean", torch.from_numpy(a["hand_pose_mean"]).float())
+ self.register_buffer("hand_comps", torch.from_numpy(a["hand_pose_comps"]).float())
+
+ # One gather that builds the 136-vector [trans(3), root(3), body(130)]
+ # with the finger channels already replaced by the hand blocks. Source
+ # layout: [zeros(3), root_euler(3), body(130), left(27), right(27)].
+ g = torch.arange(136)
+ for j, t in enumerate(a["hand_joint_idxs_left"].tolist()):
+ g[t] = 136 + j
+ for j, t in enumerate(a["hand_joint_idxs_right"].tolist()):
+ g[t] = 163 + j
+ assert len(set(g.tolist())) == 136, "hand index sets overlap"
+ self.register_buffer("full_gather", g)
+
+ # Two-layer MLP at the student's own width, on normalised pose-token
+ # features: `nn.TransformerDecoder` is built here with norm_first=True
+ # layers and no final norm, so its output is unnormalised.
+ self.norm = nn.LayerNorm(d_model)
+ self.mlp = nn.Sequential(
+ nn.Linear(d_model, d_model), nn.GELU(), nn.Linear(d_model, MC.N_HEAD_OUT))
+ # Learned initial estimate, added to the MLP output. Initialised to the
+ # neutral encoding: all-zeros is not a valid point of this space -- a
+ # zero 6D vector has no rotation and a zero (sin, cos) pair puts atan2
+ # at a 0/0 gradient.
+ self.init_estimate = nn.Parameter(MC.neutral_head_output())
+ nn.init.normal_(self.mlp[-1].weight, mean=0.0, std=1e-4)
+ nn.init.zeros_(self.mlp[-1].bias)
+
+ self.bound_scales = bool(getattr(cfg, "bound_scales", False))
+ if self.bound_scales:
+ b = np.load(cfg.scale_bounds_path)
+ # mhr_size_bounds.npz stores 0:3 then 130:204; take the six
+ # flexible channels, which are the first six of the second slice.
+ lo = torch.from_numpy(b["lo"]).float()
+ hi = torch.from_numpy(b["hi"]).float()
+ m = torch.clamp(0.25 * (hi - lo), min=float(cfg.scale_bound_margin))
+ lo, hi = lo - m, hi + m
+ starts = b["starts"].tolist()
+ off = 0
+ for st, sp in zip(starts, b["stops"].tolist()):
+ if st == 130:
+ break
+ off += sp - st
+ self.register_buffer("flex_lo", lo[off:off + 6])
+ self.register_buffer("flex_hi", hi[off:off + 6])
+
+ @staticmethod
+ def _to_range(x, lo, hi):
+ return lo + (hi - lo) * 0.5 * (torch.tanh(x) + 1.0)
+
+ def forward(self, feat):
+ """(B, d_model) pose token -> (204) MHR params, (45) identity, (3,3) root R."""
+ return self.decode(self.mlp(self.norm(feat)) + self.init_estimate)
+
+ def decode(self, pred):
+ """(B, 447) continuous -> (204) MHR params, (45) identity, (3,3) root R.
+
+ Split out of `forward` so `tools/verify_cont_head.py` can drive the
+ conversion with hand-built vectors -- a second copy of this arithmetic
+ would be free to drift away from the one that trains.
+ """
+ B = pred.shape[0]
+
+ root_6d = pred[:, :6]
+ body_cont = pred[:, 6:6 + MC.N_BODY_CONT]
+ o = 6 + MC.N_BODY_CONT
+ shape = pred[:, o:o + MC.N_SHAPE]; o += MC.N_SHAPE
+ coeff = pred[:, o:o + MC.N_SCALE_COEFF]; o += MC.N_SCALE_COEFF
+ hand_l = pred[:, o:o + MC.N_HAND_CONT]
+ hand_r = pred[:, o + MC.N_HAND_CONT:o + 2 * MC.N_HAND_CONT]
+
+ R_root = MC.rot6d_to_rotmat(root_6d)
+ root_euler = MC.rotmat_to_euler_xyz(R_root)
+ body = MC.cont_to_body_params(body_cont)[:, :130]
+ if self.bound_scales:
+ # 130:136 are size, not pose, and the continuous space leaves them raw.
+ body = torch.cat(
+ [body[:, :130 - 6],
+ self._to_range(body[:, 130 - 6:], self.flex_lo, self.flex_hi)], dim=1)
+ left = MC.cont_to_hand_params(self.hand_mean + hand_l @ self.hand_comps)
+ right = MC.cont_to_hand_params(self.hand_mean + hand_r @ self.hand_comps)
+
+ zeros3 = torch.zeros(B, 3, dtype=pred.dtype, device=pred.device)
+ src = torch.cat([zeros3, root_euler, body, left, right], dim=1)
+ full136 = src.index_select(1, self.full_gather)
+
+ if self.bound_scales:
+ coeff = self._to_range(coeff, self.coeff_lo, self.coeff_hi)
+ scales = self.scale_mean + coeff @ self.scale_comps
+ # R_root is returned so the root loss can be taken on the rotation the
+ # head actually built. Going back through the Euler triple would put
+ # atan2 in the gradient path, and 2.25% of corpus roots sit within
+ # 0.1 rad of the +-pi/2 gimbal singularity where that is unstable.
+ return torch.cat([full136, scales], dim=1), shape, R_root
+
+
class InstantHMRStudent(nn.Module):
def __init__(self, cfg, pretrained=True):
super().__init__()
@@ -933,8 +1108,16 @@ class InstantHMRStudent(nn.Module):
)
self.transformer = nn.TransformerDecoder(decoder_layer, num_layers=cfg.n_decoder_layers)
- self.global_out_dim = cfg.model_params_dim + cfg.shape_dim + cfg.cam_dim
- self.head_global = nn.Linear(cfg.d_model, self.global_out_dim)
+ # The continuous pathway replaces head_global outright rather than
+ # leaving it unused: an orphaned parameter receives no gradient and
+ # aborts DDP on step 2 (see tools/ddp_smoke.py).
+ self.cont_head = ContMHRHead(cfg, cfg.d_model) \
+ if getattr(cfg, "cont_head", False) else None
+ if self.cont_head is not None:
+ self.head_cam = nn.Linear(cfg.d_model, cfg.cam_dim)
+ else:
+ self.global_out_dim = cfg.model_params_dim + cfg.shape_dim + cfg.cam_dim
+ self.head_global = nn.Linear(cfg.d_model, self.global_out_dim)
self.kp2d_bins = getattr(cfg, 'kp2d_bins', 96)
self.kp2d_range = getattr(cfg, 'kp2d_range', 1.5)
@@ -947,12 +1130,18 @@ class InstantHMRStudent(nn.Module):
"kp2d_bin_centers",
torch.linspace(-self.kp2d_range, self.kp2d_range, self.kp2d_bins))
- nn.init.constant_(self.head_global.bias[-1], 2.0)
- nn.init.constant_(self.head_global.bias[-2], 0.0)
- nn.init.constant_(self.head_global.bias[-3], 0.0)
+ if self.cont_head is not None:
+ # Same camera start as head_global's: 2.0 on depth, 0 on x/y.
+ nn.init.normal_(self.head_cam.weight, mean=0.0, std=1e-4)
+ nn.init.zeros_(self.head_cam.bias)
+ nn.init.constant_(self.head_cam.bias[-1], 2.0)
+ else:
+ nn.init.constant_(self.head_global.bias[-1], 2.0)
+ nn.init.constant_(self.head_global.bias[-2], 0.0)
+ nn.init.constant_(self.head_global.bias[-3], 0.0)
- nn.init.normal_(self.head_global.weight, mean=0.0, std=1e-4)
- nn.init.constant_(self.head_global.bias[:-3], 0.0)
+ nn.init.normal_(self.head_global.weight, mean=0.0, std=1e-4)
+ nn.init.constant_(self.head_global.bias[:-3], 0.0)
nn.init.normal_(self.head_2d_logits.weight, mean=0.0, std=1e-4)
nn.init.constant_(self.head_2d_logits.bias, 0.0)
@@ -998,16 +1187,24 @@ class InstantHMRStudent(nn.Module):
feat_global = shared_feats[:, 0, :]
feat_2d = shared_feats[:, 1 : 1 + self.num_2d, :]
- global_preds = self.head_global(feat_global)
+ if self.cont_head is not None:
+ # Bounds live inside ContMHRHead: the bone scales are clamped as PCA
+ # coefficients so the expansion stays in the teacher's subspace, and
+ # root translation comes out as exact zero rather than regressed.
+ pred_mhr_params, pred_shape_params, pred_root_rotmat = \
+ self.cont_head(feat_global)
+ pred_cam_trans = self.head_cam(feat_global)
+ else:
+ global_preds = self.head_global(feat_global)
- idx_mhr = self.cfg.model_params_dim
- idx_shape = idx_mhr + self.cfg.shape_dim
+ idx_mhr = self.cfg.model_params_dim
+ idx_shape = idx_mhr + self.cfg.shape_dim
- pred_mhr_params = global_preds[:, :idx_mhr]
- pred_shape_params = global_preds[:, idx_mhr : idx_shape]
- pred_cam_trans = global_preds[:, idx_shape :]
+ pred_mhr_params = global_preds[:, :idx_mhr]
+ pred_shape_params = global_preds[:, idx_mhr : idx_shape]
+ pred_cam_trans = global_preds[:, idx_shape :]
- if self.bound_scales:
+ if self.bound_scales and self.cont_head is None:
# tanh(0) = 0 maps a zero-init head to the middle of each interval,
# which is ~0 because they are near-symmetric, so this does not move
# the starting point of a fresh run. Static python loop over two
@@ -1033,13 +1230,18 @@ class InstantHMRStudent(nn.Module):
prob_2d = torch.softmax(logits_2d.float(), dim=-1)
pred_joints_2d = (prob_2d * self.kp2d_bin_centers.float()).sum(dim=-1)
- return {
+ out = {
"mhr_params": pred_mhr_params,
"shape_params": pred_shape_params,
"cam_trans": pred_cam_trans,
"joints_2d": pred_joints_2d,
"joints_2d_logits": logits_2d,
}
+ if self.cont_head is not None:
+ # Training-only. The deploy wrapper names its four outputs
+ # explicitly, so this never reaches the ONNX graph.
+ out["root_rotmat"] = pred_root_rotmat
+ return out
# ============================================================
# Cell 8 — The Distillation Loss Module (Foolproof Edition)
@@ -1199,6 +1401,34 @@ class DistillationLoss(nn.Module):
e = F.smooth_l1_loss(pred, tgt, reduction='none', beta=beta).flatten(1).mean(1)
return self._mmean(e, mask)
+ def root_chordal(self, preds, pred_pose, tgt_pose):
+ """Per-sample squared chordal distance between the two root rotations.
+
+ `||Rp - Rt||_F^2 / 4 == 1 - cos(theta)`, which is `theta^2 / 2` for
+ small errors -- the same curvature SmoothL1 at beta=1.0 has on one
+ angle, so swapping the metric does not silently rescale the term -- and
+ saturates at 2 instead of growing, so a half-turn cannot dominate a
+ batch. Chosen over an `acos` geodesic, whose derivative is unbounded at
+ theta = pi; this one is smooth everywhere (its gradient vanishes at pi,
+ a measure-zero stationary point every chordal rotation loss has).
+
+ Uses the rotation the head actually built when there is one. Rebuilding
+ it from the predicted Euler triple would put `atan2` in the backward
+ path, and the corpus sits near the +-pi/2 gimbal singularity often
+ enough (2.25% within 0.1 rad) for that to matter.
+ """
+ Rp = preds.get("root_rotmat")
+ if Rp is None:
+ Rp = MC.euler_xyz_to_rotmat(pred_pose[:, 3:6])
+ Rt = MC.euler_xyz_to_rotmat(tgt_pose[:, 3:6])
+ # Divided by the 6 root parameters the old term averaged over, so this
+ # swaps the METRIC without also rescaling the loss budget. Measured
+ # without it the chordal term is 6x larger at every angle (0.00015 vs
+ # 0.00003 at 1 deg, 1.99985 vs 0.43736 at 179 deg), which is exactly
+ # that dilution factor -- three of those six parameters are the root
+ # translation, which is identically zero. `w_root_rot` scales from here.
+ return (Rp - Rt.detach()).pow(2).flatten(1).sum(1) * (0.25 / POSE_ROOT_DIM)
+
def m_mse(self, pred, tgt, mask):
e = F.mse_loss(pred, tgt, reduction='none').flatten(1).mean(1)
return self._mmean(e, mask)
@@ -1290,8 +1520,19 @@ class DistillationLoss(nn.Module):
n = pred_pose.shape[1]
# Proportional weights: the split changes the MASK, not the
# root-vs-local balance a single mean over n parameters implied.
- losses['loss_pose_root'] = self._m_beta(
- pred_pose[:, :r], tgt_pose[:, :r], m_ident, 1.0) * self.cfg.w_pose * (r / n)
+ if getattr(self.cfg, "root_rot_loss", False):
+ # Rotation-matrix metric. A SmoothL1 in radians scores a
+ # prediction at -pi+eps against a target at pi-eps as a ~2pi
+ # error when the rotation between them is tiny, and 13.59% of
+ # corpus root angles lie beyond 2.8 rad. Root translation
+ # (0:3) is excluded because it is exactly 0.000e+00 on both
+ # sides under --cont-head and contributes nothing.
+ losses['loss_pose_root'] = (
+ self._mmean(self.root_chordal(preds, pred_pose, tgt_pose), m_ident)
+ * self.cfg.w_pose * (r / n) * float(getattr(self.cfg, "w_root_rot", 1.0)))
+ else:
+ losses['loss_pose_root'] = self._m_beta(
+ pred_pose[:, :r], tgt_pose[:, :r], m_ident, 1.0) * self.cfg.w_pose * (r / n)
losses['loss_pose'] = self._m_beta(
pred_pose[:, r:], tgt_pose[:, r:], m_noflip, beta) * self.cfg.w_pose * ((n - r) / n)
else:
@@ -2489,6 +2730,15 @@ def parse_args():
p.add_argument("--no-export", dest="no_export", action="store_true",
help="Skip the ONNX export / quantization step after training.")
p.add_argument("--gpu", type=int, default=None, help="GPU count (informational).")
+ p.add_argument("--cont-head", dest="cont_head", action="store_true",
+ help="Regress the teacher's continuous space (6D root, 260-dim "
+ "body, 28 scale PCA coefficients, 2x54 hand blocks; 447 "
+ "numbers) and convert to the 204-vector inside the model, "
+ "instead of emitting the 204-vector from one nn.Linear. "
+ "Also switches the root's parameter loss to a rotation "
+ "matrix metric, which is not separable from the change. "
+ "Default off: with it off every tensor here is "
+ "bit-identical to generation 7.")
p.add_argument("--bound-scales", dest="bound_scales", action="store_true",
help="Squash the body-size parameters (root translation and "
"130:204) into the MHR rig's own parameter_limits. "
@@ -2556,6 +2806,9 @@ def main():
if args.anomaly_safe_fallback: cfg.anomaly_safe_fallback = True
if args.scale_bound_margin is not None: cfg.scale_bound_margin = args.scale_bound_margin
if args.kp3d_warmup_steps is not None: cfg.kp3d_warmup_steps = args.kp3d_warmup_steps
+ if getattr(args, "cont_head", False):
+ cfg.cont_head = True
+ cfg.root_rot_loss = True
if args.losses == "rebalanced": apply_rebalanced_losses(cfg)
# An explicit --w_keypoints3d still wins over the preset.
if args.w_keypoints3d is not None: cfg.w_keypoints3d = args.w_keypoints3d
diff --git a/tools/build_cont_head_assets.py b/tools/build_cont_head_assets.py
new file mode 100644
index 0000000..357bbe7
--- /dev/null
+++ b/tools/build_cont_head_assets.py
@@ -0,0 +1,138 @@
+#!/usr/bin/env python3
+"""Build `instanthmr_distill_train/assets/mhr_cont_head.npz` for `--cont-head`.
+
+ python tools/build_cont_head_assets.py --data_root data
+
+Two sources, both fixed:
+
+* **The teacher's own buffers**, lifted from `model.ckpt` of
+ `facebook/sam-3d-body-dinov3` (`head_pose.*`). These are the bone-scale PCA
+ basis and the hand pose basis that the teacher's head uses to expand its 28
+ scale coefficients into 68 bone scales and its 2x54 hand coefficients into
+ 2x27 finger angles. `head_pose` and `head_pose_hand` carry byte-identical
+ copies, so which one is read does not matter.
+
+* **Bounds on the 28 scale coefficients, measured from the corpus.**
+ `--bound-scales` clamps the 68 expanded bone scales coordinatewise with a
+ tanh, which is exactly what a PCA head must NOT do: an arbitrary point in the
+ 68-dim box is generally not in the 28-dim column space of `scale_comps`, so
+ clamping after expansion silently leaves the subspace the basis defines. The
+ bound therefore has to live on the coefficients, and the honest limits are
+ the ones the ground truth actually occupies.
+
+The projection uses a rank-truncated pseudo-inverse. `scale_comps` is (28, 68)
+but its float32 rank is only 24, so four directions are numerically null;
+`scale_comps.T` is NOT its inverse and a plain `pinv` amplifies the null
+directions without bound.
+"""
+
+from __future__ import annotations
+
+import argparse
+import glob
+from pathlib import Path
+
+import numpy as np
+import torch
+
+# Below this fraction of the largest singular value a direction of scale_comps
+# is treated as null. The gap in the spectrum is wide, so the exact cut does not
+# matter -- the script prints the spectrum so you can check.
+RCOND = 1e-5
+# Widen the observed coefficient range by this fraction of its own span, so the
+# bound is not exactly the corpus extremes.
+MARGIN_FRAC = 0.25
+
+
+def teacher_buffers(ckpt: Path | None) -> dict[str, np.ndarray]:
+ if ckpt is None:
+ hits = glob.glob(str(Path.home() / ".cache/huggingface/hub/"
+ "models--facebook--sam-3d-body-dinov3/snapshots/*/model.ckpt"))
+ if not hits:
+ raise SystemExit("teacher model.ckpt not found — pass --teacher-ckpt")
+ ckpt = Path(hits[0])
+ sd = torch.load(ckpt, map_location="cpu", weights_only=False)
+ sd = sd.get("state_dict", sd)
+ keys = ("scale_mean", "scale_comps", "hand_pose_mean", "hand_pose_comps",
+ "hand_joint_idxs_left", "hand_joint_idxs_right")
+ out = {k: sd[f"head_pose.{k}"].numpy() for k in keys}
+ for k in keys: # the hand head's copies must agree
+ assert np.array_equal(out[k], sd[f"head_pose_hand.{k}"].numpy()), k
+ print(f"teacher buffers from {ckpt}")
+ return out
+
+
+def gt_scales(data_root: Path, limit: int) -> np.ndarray:
+ """The 68 bone scales (`model_params[136:204]`) from real annotations."""
+ files = sorted(data_root.glob("*/annotations/*.npz"))[:limit]
+ if not files:
+ files = sorted(data_root.glob("annotations/*.npz"))[:limit]
+ if not files:
+ raise SystemExit(f"no annotations under {data_root}")
+ rows = [np.load(f)["mhr_model_params"][136:204] for f in files]
+ print(f"{len(rows):,} GT samples from {data_root}")
+ return np.stack(rows).astype(np.float64)
+
+
+def main():
+ ap = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("--data_root", default="data")
+ ap.add_argument("--teacher-ckpt", type=Path, default=None)
+ ap.add_argument("--limit", type=int, default=20000)
+ ap.add_argument("--out", type=Path,
+ default=Path("instanthmr_distill_train/assets/mhr_cont_head.npz"))
+ args = ap.parse_args()
+
+ buf = teacher_buffers(args.teacher_ckpt)
+ comps = torch.from_numpy(buf["scale_comps"]).double() # (28, 68)
+ mean = torch.from_numpy(buf["scale_mean"]).double() # (68,)
+
+ U, S, Vt = torch.linalg.svd(comps, full_matrices=False)
+ rank = int((S > RCOND * S[0]).sum())
+ print(f"scale_comps singular values: {S[0]:.4g} ... {S[rank-1]:.4g} "
+ f"| {S[rank]:.3g} ... {S[-1]:.3g} -> rank {rank}/28")
+ Sinv = torch.where(S > RCOND * S[0], 1.0 / S, torch.zeros_like(S))
+ pinv = (Vt.T * Sinv) @ U.T # (68, 28)
+
+ scales = torch.from_numpy(gt_scales(Path(args.data_root), args.limit))
+ coeffs = (scales - mean) @ pinv # (N, 28)
+ recon = mean + coeffs @ comps
+ resid = (recon - scales).abs().max()
+ print(f"GT projection residual (max abs, 68 scales): {resid:.3e}")
+ if resid > 1e-4:
+ raise SystemExit("GT scales do not lie in the teacher's scale subspace — stop")
+
+ lo, hi = coeffs.min(0).values, coeffs.max(0).values
+ m = MARGIN_FRAC * (hi - lo)
+ lo, hi = lo - m, hi + m
+ # Symmetrise about zero. `ContMHRHead` maps the head output into [lo, hi]
+ # with a tanh, so a freshly initialised head sits at the MIDPOINT -- and the
+ # observed coefficient box is not centred (midpoint up to 3.01), which would
+ # start every run at an arbitrary body size instead of `scale_mean`.
+ # Widening to +-max(|lo|, |hi|) keeps every GT coefficient reachable, puts
+ # tanh(0) exactly on the mean body, and still confines the 68 expanded
+ # scales to the 24-dimensional subspace, which is what the bound is for.
+ r = torch.maximum(lo.abs(), hi.abs())
+ lo, hi = -r, r
+ print(f"coefficient bounds: symmetric, widest +-{r.max():.3f}, "
+ f"narrowest +-{r.min():.3f}")
+
+ args.out.parent.mkdir(parents=True, exist_ok=True)
+ np.savez(args.out,
+ scale_mean=buf["scale_mean"].astype(np.float32),
+ scale_comps=buf["scale_comps"].astype(np.float32),
+ scale_pinv=pinv.numpy().astype(np.float32),
+ scale_rank=np.int64(rank),
+ coeff_lo=lo.numpy().astype(np.float32),
+ coeff_hi=hi.numpy().astype(np.float32),
+ hand_pose_mean=buf["hand_pose_mean"].astype(np.float32),
+ hand_pose_comps=buf["hand_pose_comps"].astype(np.float32),
+ hand_joint_idxs_left=buf["hand_joint_idxs_left"].astype(np.int64),
+ hand_joint_idxs_right=buf["hand_joint_idxs_right"].astype(np.int64),
+ n_gt=np.int64(len(scales)))
+ print(f"wrote {args.out}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tools/ddp_smoke.py b/tools/ddp_smoke.py
index 42270d0..2ce5382 100644
--- a/tools/ddp_smoke.py
+++ b/tools/ddp_smoke.py
@@ -23,7 +23,9 @@ step 2.
World size 1 is enough: the Reducer and its unused-parameter check are built and
run identically at any world size, so this reproduces the cluster error exactly,
-on one GPU, in under a minute. It does NOT test NCCL, multi-node bring-up or
+on one GPU, in under a minute. The batch size is small for the same reason --
+see --batch_size; on a login node a large one is killed for memory, not for
+anything this is testing. It does NOT test NCCL, multi-node bring-up or
gradient bucketing across ranks -- only that every parameter participates.
Exit status is 0 when every parameter received a gradient, 1 otherwise, so it
@@ -50,12 +52,22 @@ def main() -> int:
formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--backbone", default=None, help="timm name; default is the config's")
p.add_argument("--data_root", default="data")
- p.add_argument("--batch_size", type=int, default=8)
+ p.add_argument("--batch_size", type=int, default=2,
+ help="Small on purpose. Which parameters receive a gradient "
+ "does not depend on the batch size, and a Jean Zay "
+ "LOGIN node kills a process around 3-4 GB: measured peak "
+ "RSS is 4.49 GB for repvit_m2_3 at batch 8 (killed) "
+ "against 2.45 GB at batch 2. Raise it only if you are on "
+ "a compute node.")
p.add_argument("--losses", choices=("legacy", "rebalanced"), default="rebalanced")
p.add_argument("--w-verts", dest="w_verts", type=float, default=0.0)
p.add_argument("--bound-scales", dest="bound_scales", action="store_true", default=True)
p.add_argument("--cliff-focal", dest="cliff_focal", action="store_true", default=True)
p.add_argument("--crop-centre-fix", dest="crop_centre_fix", action="store_true", default=True)
+ p.add_argument("--cont-head", dest="cont_head", action="store_true",
+ help="Build the teacher's continuous regression head instead "
+ "of the linear 204-vector head. A changed head shape is "
+ "exactly the class of change this exists to gate.")
p.add_argument("--pretrained", action="store_true",
help="Load the timm weights too. Off by default: the head "
"geometry this checks does not depend on them.")
@@ -76,6 +88,9 @@ def main() -> int:
cfg.cliff_focal = args.cliff_focal
cfg.crop_centre_fix = args.crop_centre_fix
cfg.w_verts = args.w_verts
+ if args.cont_head:
+ cfg.cont_head = True
+ cfg.root_rot_loss = True
if args.backbone:
cfg.backbone = args.backbone
@@ -94,6 +109,7 @@ def main() -> int:
model = T.InstantHMRStudent(cfg, pretrained=args.pretrained).to(T.device)
n = sum(q.numel() for q in model.parameters())
print(f"{cfg.backbone}: {n/1e6:.2f} M parameters, "
+ f"head={'continuous' if cfg.cont_head else 'linear'}, "
f"w_verts={cfg.w_verts}, batch {args.batch_size}")
ddp = torch.nn.parallel.DistributedDataParallel(model) # find_unused_parameters=False
diff --git a/tools/pth_to_onnx.py b/tools/pth_to_onnx.py
index 89b174d..104dea0 100644
--- a/tools/pth_to_onnx.py
+++ b/tools/pth_to_onnx.py
@@ -134,6 +134,8 @@ def main() -> None:
f"({prov.get('bound_scales', 'default')})")
print(f" cliff_focal : {getattr(cfg, 'cliff_focal', False)} "
f"({prov.get('cliff_focal', 'default')})")
+ print(f" cont_head : {getattr(cfg, 'cont_head', False)} "
+ f"({prov.get('cont_head', 'default')})")
# SimCC head geometry lives in the checkpoint — mirror it so the graph and
# the bin centres agree with what was actually trained.
diff --git a/tools/verify_cont_head.py b/tools/verify_cont_head.py
new file mode 100644
index 0000000..0744b67
--- /dev/null
+++ b/tools/verify_cont_head.py
@@ -0,0 +1,249 @@
+#!/usr/bin/env python3
+"""Gate `--cont-head` before it costs cluster time.
+
+ python tools/verify_cont_head.py --data_root data
+
+Six checks, all on real ground truth and the real rig:
+
+1. **Reachability.** Every GT 204-vector must be expressible in the head's
+ output space. Invert a real annotation into the 447 numbers, push it back
+ through `ContMHRHead`, and compare parameters *and* forward kinematics. If
+ this fails the head cannot represent its own targets and no amount of
+ training fixes it.
+2. **Neutral initialisation.** A freshly built head, before any training, must
+ decode to the rig's zero pose -- not to whatever all-zero continuous inputs
+ happen to mean, which is a 6D vector with no rotation and an `atan2` at a
+ 0/0 gradient.
+3. **Invariants.** Root translation exactly zero, bone scales inside the
+ teacher's 24-dimensional scale subspace, size channels inside the rig limits
+ -- for arbitrary head outputs, including deliberately extreme ones.
+4. **Finite gradients** through the whole conversion, at the neutral pose, at
+ random poses, and at the two places the geometry is singular: identity and a
+ half turn.
+5. **Root-loss magnitude.** The chordal rotation loss replaces a SmoothL1 on the
+ Euler triple. `docs/todo.md` asks for the gradient comparison explicitly, so
+ it is printed rather than assumed.
+6. **Scale stability** under a large random head output: the failure mode that
+ killed four runs at epochs 24-32 was a bone scale running away, and the PCA
+ parameterisation changes how that can happen.
+
+Exit status is 0 only if every check passes.
+"""
+
+from __future__ import annotations
+
+import argparse
+import glob
+import sys
+from pathlib import Path
+
+import numpy as np
+import torch
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "instanthmr_distill_train"))
+import mhr_cont as MC # noqa: E402
+import train_distill_mhr_only as T # noqa: E402
+
+OK, BAD = " ok ", "*FAIL*"
+_fails = 0
+
+
+def check(cond: bool, label: str, detail: str = "") -> None:
+ global _fails
+ if not cond:
+ _fails += 1
+ print(f"[{OK if cond else BAD}] {label}{(' ' + detail) if detail else ''}")
+
+
+def atanh_to_range(x, lo, hi):
+ """Pre-tanh value that `ContMHRHead._to_range` maps back onto `x`."""
+ span = hi - lo
+ t = torch.where(span > 0, (2.0 * (x - lo) / span.clamp_min(1e-12) - 1.0).clamp(-1 + 1e-6, 1 - 1e-6),
+ torch.zeros_like(x))
+ return torch.atanh(t)
+
+
+def head_vector_from_gt(head, gt204, gt_shape):
+ """The 447 numbers whose decode is `gt204` -- the inverse of ContMHRHead."""
+ B = gt204.shape[0]
+ root6 = MC.rotmat_to_rot6d(MC.euler_xyz_to_rotmat(gt204[:, 3:6]))
+
+ body133 = torch.cat([gt204[:, 6:136], torch.zeros(B, 3, device=gt204.device)], 1)
+ if head.bound_scales: # the head tanh's 130:136
+ body133 = body133.clone()
+ body133[:, 124:130] = atanh_to_range(body133[:, 124:130], head.flex_lo, head.flex_hi)
+ body_cont = MC.body_params_to_cont(body133)
+
+ coeff = (gt204[:, 136:204] - head.scale_mean) @ head.scale_pinv
+ if head.bound_scales:
+ coeff = atanh_to_range(coeff, head.coeff_lo, head.coeff_hi)
+
+ hands = []
+ for idx in (head.hand_idx_left, head.hand_idx_right):
+ hands.append(MC.hand_params_to_cont(gt204[:, idx]) - head.hand_mean)
+ return torch.cat([root6, body_cont, gt_shape, coeff] + hands, dim=1)
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("--data_root", default="data")
+ ap.add_argument("--n", type=int, default=512)
+ ap.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu")
+ args = ap.parse_args()
+ dev = args.device
+
+ cfg = T.DistillConfig()
+ T.cfg = cfg
+ T.apply_rebalanced_losses(cfg)
+ cfg.cont_head = True
+ cfg.root_rot_loss = True
+ cfg.bound_scales = True
+ cfg.w_verts = 0.35
+
+ head = T.ContMHRHead(cfg, cfg.d_model).to(dev).eval()
+ # Extras the inverse needs; they live in the asset, not in the head.
+ a = np.load(cfg.cont_head_path)
+ head.scale_pinv = torch.from_numpy(a["scale_pinv"]).float().to(dev)
+ head.hand_idx_left = torch.from_numpy(a["hand_joint_idxs_left"]).long().to(dev)
+ head.hand_idx_right = torch.from_numpy(a["hand_joint_idxs_right"]).long().to(dev)
+
+ files = sorted(glob.glob(f"{args.data_root}/*/annotations/*.npz"))[:args.n]
+ if not files:
+ files = sorted(glob.glob(f"{args.data_root}/annotations/*.npz"))[:args.n]
+ gt204 = torch.from_numpy(np.stack([np.load(f)["mhr_model_params"] for f in files])).float().to(dev)
+ gt_shape = torch.from_numpy(np.stack([np.load(f)["shape_params"] for f in files])).float().to(dev)
+ print(f"{len(gt204):,} real GT annotations from {args.data_root}, device {dev}\n")
+
+ mhr = T.MHRForwardPass(cfg.mhr_model_path, dev,
+ kp_regressor=np.load(cfg.kp_regressor_path))
+
+ # --- 1. reachability ----------------------------------------------------
+ vec = head_vector_from_gt(head, gt204, gt_shape)
+ check(vec.shape[1] == MC.N_HEAD_OUT, "head output width",
+ f"{vec.shape[1]} dims (6 root + 260 body + 45 shape + 28 scale + 108 hands)")
+ with torch.no_grad():
+ rec, rec_shape, _ = head.decode(vec)
+
+ dp = (rec[:, 6:136] - gt204[:, 6:136]).abs().max()
+ ds = (rec[:, 136:204] - gt204[:, 136:204]).abs().max()
+ check(dp < 2e-3, "GT body angles round-trip", f"max {dp:.2e} rad")
+ check(ds < 2e-3, "GT bone scales round-trip", f"max {ds:.2e}")
+
+ # The ROOT is compared as a rotation, not as three numbers. `atan2` returns
+ # the representative in (-pi, pi], and real annotations go outside it: 5 of
+ # 256 COCO roots have an axis beyond pi, so the round trip legitimately
+ # comes back 2pi away on that axis while encoding the identical rotation.
+ # Comparing the raw triples flags those as a 6.28 rad error -- which is
+ # precisely the defect `--cont-head` exists to remove, since the SmoothL1
+ # this replaces charges 0.96 for such a pair, more than it charges for a
+ # genuine 179 deg error.
+ Rg = MC.euler_xyz_to_rotmat(gt204[:, 3:6])
+ Rr = MC.euler_xyz_to_rotmat(rec[:, 3:6])
+ cos = ((torch.einsum("bii->b", Rg.transpose(1, 2) @ Rr) - 1) / 2).clamp(-1, 1)
+ geo = torch.rad2deg(torch.acos(cos)).max()
+ raw = (rec[:, 3:6] - gt204[:, 3:6]).abs().max(1).values
+ n_wrap = int((raw > 1e-2).sum())
+ check(geo < 0.1, "GT root round-trip, as a rotation", f"max {geo:.2e} deg")
+ print(f" ({n_wrap} of {len(gt204)} roots come back on the other side of "
+ f"+-pi -- same rotation, different Euler representative; the root loss "
+ f"is on the matrix, so this costs nothing)")
+ check((rec_shape - gt_shape).abs().max() < 1e-6, "identity coefficients pass through")
+
+ with torch.no_grad():
+ j_gt = mhr.get_joints(gt204, gt_shape)[..., :3]
+ j_rc = mhr.get_joints(rec, gt_shape)[..., :3]
+ fk = (j_gt - j_rc).norm(dim=-1).max() * 10.0 # 1 rig unit = 10 cm -> mm
+ check(fk < 1.0, "forward kinematics after round-trip", f"max joint error {fk:.4f} mm")
+
+ # --- 2. neutral initialisation -----------------------------------------
+ # The 54 finger channels are NOT expected to be zero: zero hand
+ # coefficients decode to `hand_pose_mean`, which is the teacher's mean hand
+ # pose (up to 0.594 rad) and is the intended resting hand.
+ hand_ch = torch.cat([head.hand_idx_left, head.hand_idx_right])
+ body_ch = torch.tensor([i for i in range(6, 130) if i not in set(hand_ch.tolist())],
+ device=dev)
+ with torch.no_grad():
+ p0, _, R0 = head.decode(MC.neutral_head_output().to(dev))
+ check(p0[:, body_ch].abs().max() < 1e-6, "neutral encoding decodes the zero pose",
+ f"max |body angle|, fingers excluded, {p0[:, body_ch].abs().max():.2e} rad")
+ with torch.no_grad():
+ want = MC.cont_to_hand_params(head.hand_mean[None])
+ check((p0[:, head.hand_idx_right] - want).abs().max() < 1e-6,
+ "finger channels start at the teacher's mean hand pose",
+ f"max |angle| {want.abs().max():.3f} rad")
+ check((R0 - torch.eye(3, device=dev)).abs().max() < 1e-6,
+ "neutral root rotation is identity")
+ check((p0[:, 136:204] - head.scale_mean).abs().max() < 1e-5,
+ "neutral bone scales are exactly scale_mean",
+ f"max {(p0[:, 136:204] - head.scale_mean).abs().max():.2e}")
+ with torch.no_grad():
+ q, _, Rq = head(torch.randn(8, cfg.d_model, device=dev))
+ check((q[:, body_ch]).abs().max() < 1e-2 and (Rq - torch.eye(3, device=dev)).abs().max() < 1e-2,
+ "untrained head starts at that neutral pose",
+ f"max |body angle| {q[:, body_ch].abs().max():.2e} rad, "
+ f"root off identity by {torch.rad2deg(torch.acos((((Rq * torch.eye(3, device=dev)).sum((-1,-2)) - 1) / 2).clamp(-1,1))).max():.3f} deg")
+
+ # --- 3. invariants under arbitrary output ------------------------------
+ torch.manual_seed(0)
+ for scale, tag in ((1.0, "N(0,1)"), (25.0, "N(0,25) -- the runaway regime")):
+ with torch.no_grad():
+ p, _, _ = head.decode(torch.randn(256, MC.N_HEAD_OUT, device=dev) * scale)
+ check(p[:, :3].abs().max() == 0, f"root translation exactly zero [{tag}]")
+ resid = ((p[:, 136:204] - head.scale_mean) @ head.scale_pinv @ head.scale_comps
+ + head.scale_mean - p[:, 136:204]).abs().max()
+ check(resid < 1e-3, f"bone scales stay in the scale subspace [{tag}]",
+ f"residual {resid:.2e}")
+ lo, hi = head.flex_lo.min(), head.flex_hi.max()
+ inb = (p[:, 130:136] >= head.flex_lo - 1e-4).all() and (p[:, 130:136] <= head.flex_hi + 1e-4).all()
+ check(bool(inb), f"size channels 130:136 inside rig limits [{tag}]")
+ with torch.no_grad():
+ j = mhr.get_joints(p, torch.zeros(len(p), 45, device=dev))[..., :3]
+ span = (j.max(1).values - j.min(1).values).norm(dim=-1).max() * 10 / 1000
+ check(span < 5.0, f"skeleton stays human-sized [{tag}]", f"max extent {span:.2f} m")
+
+ # --- 4. finite gradients ------------------------------------------------
+ for tag, v in (("neutral", MC.neutral_head_output().to(dev).repeat(8, 1)),
+ ("random", torch.randn(8, MC.N_HEAD_OUT, device=dev))):
+ v = v.clone().requires_grad_(True)
+ p, s, R = head.decode(v)
+ (p.sum() + s.sum() + R.sum()).backward()
+ check(torch.isfinite(v.grad).all(), f"gradients finite through the decode [{tag}]",
+ f"max |grad| {v.grad.abs().max():.3e}")
+
+ crit = T.DistillationLoss(cfg, mhr)
+ for tag, ang in (("identity", torch.zeros(8, 3)),
+ ("half turn (pi)", torch.tensor([[np.pi, 0., 0.]] * 8)),
+ ("gimbal (pi/2 middle axis)", torch.tensor([[0.3, np.pi / 2, 0.7]] * 8))):
+ six = MC.rotmat_to_rot6d(MC.euler_xyz_to_rotmat(ang.to(dev))).requires_grad_(True)
+ Rp = MC.rot6d_to_rotmat(six)
+ tgt = torch.zeros(8, 204, device=dev)
+ e = crit.root_chordal({"root_rotmat": Rp}, None, tgt)
+ e.sum().backward()
+ check(torch.isfinite(six.grad).all(), f"root loss gradient finite at {tag}",
+ f"loss {e.mean():.4f}, max |grad| {six.grad.abs().max():.3e}")
+
+ # --- 5. root-loss magnitude vs the SmoothL1 it replaces -----------------
+ print("\nroot-loss magnitude (docs/todo.md asks for this explicitly):")
+ err_deg = [1.0, 5.0, 30.0, 90.0, 179.0]
+ print(f" {'error':>10} | {'chordal /6':>10} | {'SmoothL1(beta=1) on the Euler triple':>38}")
+ for d in err_deg:
+ r = torch.deg2rad(torch.tensor(d))
+ A = MC.euler_xyz_to_rotmat(torch.zeros(1, 3))
+ B = MC.euler_xyz_to_rotmat(torch.tensor([[r, 0., 0.]]))
+ ch = ((A - B).pow(2).sum() * 0.25 / 6.0).item()
+ sl = torch.nn.functional.smooth_l1_loss(
+ torch.tensor([[0., 0., 0., r, 0., 0.]]), torch.zeros(1, 6),
+ reduction="none", beta=1.0).mean().item()
+ print(f" {d:8.0f} deg | {ch:10.5f} | {sl:38.5f}")
+
+ print()
+ if _fails:
+ print(f"{_fails} check(s) FAILED")
+ else:
+ print("all checks passed")
+ return 1 if _fails else 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())